From patchwork Tue Sep 8 20:30:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 305877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58C10C433E2 for ; Tue, 8 Sep 2020 20:31:43 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C531B20759 for ; Tue, 8 Sep 2020 20:31:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="JZkFXYxU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C531B20759 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:59124 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kFkHF-0005rn-PF for qemu-devel@archiver.kernel.org; Tue, 08 Sep 2020 16:31:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50210) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kFkGA-0004DJ-M3 for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:35 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:39145) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kFkG7-0007uQ-VK for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1599597029; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Dj8PFivSAcJNd/FlfrotVGRkKpULTf3HJuqDkHyOKcI=; b=JZkFXYxU74o/+fNAaQEpTKePd3L625hk8wTCKD0CW74JN2GufuXccizgb2CSKMgnNkKdJ+ JXrY/IkUTWbwA/H/nsn8wEYGmnLFmT9wX43pctVaZ52LiG9VghNayWLnkNP7tVYNglRr6k M+lTpmOX8YdF9VPsqzeXGPlCZzz0MlI= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-64-htux_8cAMgO8vaMV3kPvkw-1; Tue, 08 Sep 2020 16:30:27 -0400 X-MC-Unique: htux_8cAMgO8vaMV3kPvkw-1 Received: by mail-qt1-f199.google.com with SMTP id t11so347065qtn.8 for ; Tue, 08 Sep 2020 13:30:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Dj8PFivSAcJNd/FlfrotVGRkKpULTf3HJuqDkHyOKcI=; b=WKG8HH+3M8jCHYeHnyRYXSdV3MaENA7p8YEEAJZQnof21Ix4aDUqY/yfukgimP1Xye YipncfVeFCbfbNNdIMjm8w2xIJzJf26c0/dGEyvk4L8j9Ik5G1gJGqdHLF2iKsQZi+74 Lt3SCgDZmSMt+A4PwhcMMF32v7tgSyb3/1NNEA81HjUYHsnMlnhKwlE5naQ64Gqlw6KC X5DpfSTOvSUuGHzl+sAwSWp99OCBnb1mIhAYtPZy2hkLSyjDmVpMxNuD6CugYNDEe0+v QYL2G2Ef7xZUFsV3FG5MXNu3MvBr5KGMaA9HTTLDssNwiw9CrzVr04GVGiDcUTrGipie QxPw== X-Gm-Message-State: AOAM5318hH/GeuLjJGgjdUIzQe2MzuYGBgAmK/wkTX3otwezuCZ709hm 2KvY+4PiogZiOEvUHHQ0qy6rPU32Psy54kxsbfT7enZajbjOZVBNLlRQkiLKwenna3mz9C3wv+6 ccOoVmhnULayv4i8= X-Received: by 2002:a37:a392:: with SMTP id m140mr331073qke.195.1599597026806; Tue, 08 Sep 2020 13:30:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyKZqNGoI2qPVsxlQ52PMAe3LlObIL1VytGC6hZhmsjYNWZh1Ab2xnTyyqn+bHm0i+mZUMu0A== X-Received: by 2002:a37:a392:: with SMTP id m140mr331048qke.195.1599597026507; Tue, 08 Sep 2020 13:30:26 -0700 (PDT) Received: from xz-x1.redhat.com (bras-vprn-toroon474qw-lp130-11-70-53-122-15.dsl.bell.ca. [70.53.122.15]) by smtp.gmail.com with ESMTPSA id o28sm595397qtl.62.2020.09.08.13.30.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 13:30:25 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 1/6] migration: Properly destroy variables on incoming side Date: Tue, 8 Sep 2020 16:30:17 -0400 Message-Id: <20200908203022.341615-2-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200908203022.341615-1-peterx@redhat.com> References: <20200908203022.341615-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0.001 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/08 12:54:58 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Xiaohui Li , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" In migration_incoming_state_destroy(), we've got a few variables that aren't destroyed properly, namely: main_thread_load_event postcopy_pause_sem_dst postcopy_pause_sem_fault rp_mutex Destroy them properly. Signed-off-by: Peter Xu --- migration/migration.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 58a5452471..749d9b145b 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -238,12 +238,15 @@ void migration_incoming_state_destroy(void) mis->postcopy_remote_fds = NULL; } - qemu_event_reset(&mis->main_thread_load_event); - if (mis->socket_address_list) { qapi_free_SocketAddressList(mis->socket_address_list); mis->socket_address_list = NULL; } + + qemu_event_destroy(&mis->main_thread_load_event); + qemu_sem_destroy(&mis->postcopy_pause_sem_dst); + qemu_sem_destroy(&mis->postcopy_pause_sem_fault); + qemu_mutex_destroy(&mis->rp_mutex); } static void migrate_generate_event(int new_state) From patchwork Tue Sep 8 20:30:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 274206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9D4AC43461 for ; Tue, 8 Sep 2020 20:31:45 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8BEDA20759 for ; Tue, 8 Sep 2020 20:31:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="X20GfKU3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8BEDA20759 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:59296 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kFkHI-0005wG-LT for qemu-devel@archiver.kernel.org; Tue, 08 Sep 2020 16:31:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50208) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kFkGA-0004DI-KP for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:35 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:33416 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kFkG8-0007w9-39 for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1599597031; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=//PaBaAJUXYXcmeywaN+dRPykWTKONRk0alo9As/Rfg=; b=X20GfKU3ZBEpHksxGymE87IQcFWwx44GrspkirQ7vbaFZPCuqxCvru4y3fIfhCQcVMXhOH Grfdd49+JTTIqng4CvGZU1W7fpPJoVhVxqGefe2OEtNbe335ddqYyRhqOPS6ywtlL0jpBx 3Mx8//CxZaFhSV1vVrpoaqd935ymojg= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-271-jWFvg0yrPiyL8hWoxr1luw-1; Tue, 08 Sep 2020 16:30:29 -0400 X-MC-Unique: jWFvg0yrPiyL8hWoxr1luw-1 Received: by mail-qk1-f200.google.com with SMTP id m203so92100qke.16 for ; Tue, 08 Sep 2020 13:30:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=//PaBaAJUXYXcmeywaN+dRPykWTKONRk0alo9As/Rfg=; b=caQxmREx07dgl6RHqPjW+Y8pOtcFSN4JM67cNfn3fsSNgkiGC7Yrw/PsELNbZqbygx 02yt7QaBoJNaaBTYmOqz9JYeV+ZegAS57bgTQZ4U/mUW+PD1RNgi+FOhgmkLr3zaeaws nw3RLmwAoIsCuwQHwue/VYJ8Md2EU18IoN3qjC391i9NpYPRkLuysQa17aGLL2EzKdMC 7P2xJy9ZLcw7u/YVqJWj+dGM/a8IGbCkHDhYRF+5P38+GTiaWFgGq7eNuUI8PWux//rU P0ZWy2ZpE+x5TwDSRKIZguxmVoodD2BroHpihxrDhDYLlb0ut0FGhpl1x/fbzVl/fPuh LAGA== X-Gm-Message-State: AOAM533kZAb8W34H4pqV8wmOd346Qj1xi2xSr9tKT7cNKh9ml1Otkiig Ge1JnF16OQsj+LIaLeJKp1U0eUfJZhufLcqmrF2wKCj7l3hImKHT3/ymyuBoFUQU8mK5aITyDBF bO+UAJCvCnn4YCBU= X-Received: by 2002:a37:6283:: with SMTP id w125mr322689qkb.411.1599597028308; Tue, 08 Sep 2020 13:30:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxE4mq0XS/kamV9AaVfM+ICs1Qa16FMeocZoh6z/zXwuL7LBHwJHH07U6YqSojvU25WyZPV2A== X-Received: by 2002:a37:6283:: with SMTP id w125mr322666qkb.411.1599597028038; Tue, 08 Sep 2020 13:30:28 -0700 (PDT) Received: from xz-x1.redhat.com (bras-vprn-toroon474qw-lp130-11-70-53-122-15.dsl.bell.ca. [70.53.122.15]) by smtp.gmail.com with ESMTPSA id o28sm595397qtl.62.2020.09.08.13.30.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 13:30:27 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 2/6] migration: Rework migrate_send_rp_req_pages() function Date: Tue, 8 Sep 2020 16:30:18 -0400 Message-Id: <20200908203022.341615-3-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200908203022.341615-1-peterx@redhat.com> References: <20200908203022.341615-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0.002 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=peterx@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/08 12:55:20 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Xiaohui Li , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" We duplicated the logic of maintaining the last_rb variable at both callers of this function. Pass *rb pointer into the function so that we can avoid duplicating the logic. Also, when we have the rb pointer, it's also easier to remove the original 2nd & 4th parameters, because both of them (name of the ramblock when needed, or the page size) can be fetched from the ramblock pointer too. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 26 ++++++++++++++++++-------- migration/migration.h | 4 ++-- migration/postcopy-ram.c | 24 ++---------------------- 3 files changed, 22 insertions(+), 32 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 749d9b145b..2e6b21544b 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -314,25 +314,35 @@ error: return ret; } -/* Request a range of pages from the source VM at the given - * start address. - * rbname: Name of the RAMBlock to request the page in, if NULL it's the same - * as the last request (a name must have been given previously) +/* Request one page from the source VM at the given start address. + * rb: the RAMBlock to request the page in * Start: Address offset within the RB * Len: Length in bytes required - must be a multiple of pagesize */ -int migrate_send_rp_req_pages(MigrationIncomingState *mis, const char *rbname, - ram_addr_t start, size_t len) +int migrate_send_rp_req_pages(MigrationIncomingState *mis, RAMBlock *rb, + ram_addr_t start) { uint8_t bufc[12 + 1 + 255]; /* start (8), len (4), rbname up to 256 */ size_t msglen = 12; /* start + len */ + size_t len = qemu_ram_pagesize(rb); enum mig_rp_message_type msg_type; + const char *rbname; + int rbname_len; *(uint64_t *)bufc = cpu_to_be64((uint64_t)start); *(uint32_t *)(bufc + 8) = cpu_to_be32((uint32_t)len); - if (rbname) { - int rbname_len = strlen(rbname); + /* + * We maintain the last ramblock that we requested for page. Note that we + * don't need locking because this function will only be called within the + * postcopy ram fault thread. + */ + if (rb != mis->last_rb) { + mis->last_rb = rb; + + rbname = qemu_ram_get_idstr(rb); + rbname_len = strlen(rbname); + assert(rbname_len < 256); bufc[msglen++] = rbname_len; diff --git a/migration/migration.h b/migration/migration.h index ae497bd45a..ca8dc4c773 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -328,8 +328,8 @@ void migrate_send_rp_shut(MigrationIncomingState *mis, uint32_t value); void migrate_send_rp_pong(MigrationIncomingState *mis, uint32_t value); -int migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbname, - ram_addr_t start, size_t len); +int migrate_send_rp_req_pages(MigrationIncomingState *mis, RAMBlock *rb, + ram_addr_t start); void migrate_send_rp_recv_bitmap(MigrationIncomingState *mis, char *block_name); void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value); diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 1bb22f2b6c..11a70441a6 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -684,14 +684,7 @@ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb, qemu_ram_get_idstr(rb), rb_offset); return postcopy_wake_shared(pcfd, client_addr, rb); } - if (rb != mis->last_rb) { - mis->last_rb = rb; - migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), - aligned_rbo, pagesize); - } else { - /* Save some space */ - migrate_send_rp_req_pages(mis, NULL, aligned_rbo, pagesize); - } + migrate_send_rp_req_pages(mis, rb, aligned_rbo); return 0; } @@ -986,20 +979,7 @@ retry: * Send the request to the source - we want to request one * of our host page sizes (which is >= TPS) */ - if (rb != mis->last_rb) { - mis->last_rb = rb; - ret = migrate_send_rp_req_pages(mis, - qemu_ram_get_idstr(rb), - rb_offset, - qemu_ram_pagesize(rb)); - } else { - /* Save some space */ - ret = migrate_send_rp_req_pages(mis, - NULL, - rb_offset, - qemu_ram_pagesize(rb)); - } - + ret = migrate_send_rp_req_pages(mis, rb, rb_offset); if (ret) { /* May be network failure, try to wait for recovery */ if (ret == -EIO && postcopy_pause_fault_thread(mis)) { From patchwork Tue Sep 8 20:30:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 305875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1EE1C43461 for ; Tue, 8 Sep 2020 20:33:15 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9CCCB2080A for ; Tue, 8 Sep 2020 20:33:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bBlSc/di" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9CCCB2080A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:38634 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kFkIk-0000dB-Q7 for qemu-devel@archiver.kernel.org; Tue, 08 Sep 2020 16:33:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50242) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kFkGC-0004EN-D3 for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:37 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:43420) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kFkGA-0007wF-3F for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1599597032; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HAVQfWRfTE2sBR36iK2CCVFgnnD6WcSKf0PKmxqcWhE=; b=bBlSc/di1nIvvJCsjIy4/3ZSkBwIe/p40iUzPxebQeUaYuqCwX0YW2PPfCy3+x+vLu4/cR esgInn7Oimj+c+SxuJqyxTkugZU5iOLZiksBNRVStU9wn6duDaqZfCMjJmAJMQdsahc1wO V9icCxCBffmSyLh+nrfylk4h60WenWw= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-38-HSBytEn_PXWfzrDKMfPwZA-1; Tue, 08 Sep 2020 16:30:30 -0400 X-MC-Unique: HSBytEn_PXWfzrDKMfPwZA-1 Received: by mail-qk1-f198.google.com with SMTP id 6so104641qko.8 for ; Tue, 08 Sep 2020 13:30:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HAVQfWRfTE2sBR36iK2CCVFgnnD6WcSKf0PKmxqcWhE=; b=tEzXq9Ef/NHuo3LAfMsp9hqd/+tAruuCA9ZzaMIIEyCo+s5FVLoh4bkMbEMP6JzOqW 1dONahL8kDpzLlNzlvyObh3yZaSdQHKV+vvOxET1idV+gib4yBHkzzDN1tRiIKlwn34l pnO85C3Hj8k8aJT63fNxj1PtByv3hKz9zOkdQc0YY3gh1cd07AnLjRhCh9y1gQuyJllC 0uLDEkV4NT1w5WKYnaLzu2YCNKyM1ImgMSOd8ZxsaPbn3U+UzeBaqYu0Y9hJC4faBDPC Ntpjnwor8Lq8ejee0MNVlglX9s7bcRbq8aqjYrxEhnwZFyMh+DhOdIzp4dfMNKXzadCZ 3I2w== X-Gm-Message-State: AOAM533+qIG4q/dPIcRksGwgZL3uIKEpJqSJq00nFPSAF72c03B9/+da Xjwgbov+CUVEFmlH4Zaan/2dgBZUKy+cUlGcj+eGg/A2FqlWzmUkfF+kBSIKnN+XbirjIqWYNnO lxyog7edVn59OLeg= X-Received: by 2002:a0c:f194:: with SMTP id m20mr875090qvl.9.1599597029536; Tue, 08 Sep 2020 13:30:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyJx2ZUVuLMLbvCJUcpF+qft+iI9SMnUbNxb4TCSBoIfGbrPHKbJr8Gqb3hOizAlsMXjCNUjQ== X-Received: by 2002:a0c:f194:: with SMTP id m20mr875072qvl.9.1599597029256; Tue, 08 Sep 2020 13:30:29 -0700 (PDT) Received: from xz-x1.redhat.com (bras-vprn-toroon474qw-lp130-11-70-53-122-15.dsl.bell.ca. [70.53.122.15]) by smtp.gmail.com with ESMTPSA id o28sm595397qtl.62.2020.09.08.13.30.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 13:30:28 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 3/6] migration: Pass incoming state into qemu_ufd_copy_ioctl() Date: Tue, 8 Sep 2020 16:30:19 -0400 Message-Id: <20200908203022.341615-4-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200908203022.341615-1-peterx@redhat.com> References: <20200908203022.341615-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0.003 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/08 12:54:58 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Xiaohui Li , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" It'll be used in follow up patches to access more fields out of it. Meanwhile fetch the userfaultfd inside the function. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/postcopy-ram.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 11a70441a6..d333c3fd0e 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -1128,10 +1128,12 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) return 0; } -static int qemu_ufd_copy_ioctl(int userfault_fd, void *host_addr, +static int qemu_ufd_copy_ioctl(MigrationIncomingState *mis, void *host_addr, void *from_addr, uint64_t pagesize, RAMBlock *rb) { + int userfault_fd = mis->userfault_fd; int ret; + if (from_addr) { struct uffdio_copy copy_struct; copy_struct.dst = (uint64_t)(uintptr_t)host_addr; @@ -1185,7 +1187,7 @@ int postcopy_place_page(MigrationIncomingState *mis, void *host, void *from, * which would be slightly cheaper, but we'd have to be careful * of the order of updating our page state. */ - if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, from, pagesize, rb)) { + if (qemu_ufd_copy_ioctl(mis, host, from, pagesize, rb)) { int e = errno; error_report("%s: %s copy host: %p from: %p (size: %zd)", __func__, strerror(e), host, from, pagesize); @@ -1212,7 +1214,7 @@ int postcopy_place_page_zero(MigrationIncomingState *mis, void *host, * but it's not available for everything (e.g. hugetlbpages) */ if (qemu_ram_is_uf_zeroable(rb)) { - if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, NULL, pagesize, rb)) { + if (qemu_ufd_copy_ioctl(mis, host, NULL, pagesize, rb)) { int e = errno; error_report("%s: %s zero host: %p", __func__, strerror(e), host); From patchwork Tue Sep 8 20:30:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 305874 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3926DC43461 for ; Tue, 8 Sep 2020 20:34:16 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D8083208FE for ; Tue, 8 Sep 2020 20:34:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fUqyXlqW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D8083208FE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43470 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kFkJj-0002bt-2z for qemu-devel@archiver.kernel.org; Tue, 08 Sep 2020 16:34:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50246) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kFkGC-0004Ej-El for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:37 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:46323 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kFkGA-0007wW-K7 for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1599597033; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=amKOtoi6vPf/V/x2bL7dd1QXHRAnGLq9RsDduAfD7yI=; b=fUqyXlqWiqH1nSupchumy1DVtoEuwYxcT/bmcy5Acn8FhRR4KNg9ZSst5yY5P7caQE8omq Ad+UFKU3SgU2oT7Uw6O4oA720jW3xxJtREOT9QmglD3ITOdxDW9hgbtLpCitSsuJCxC7WH FXfg40vJ2BqGPIslffJOUFrhotj8vVk= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-49-plPdsnNNPQGkLoTRepDjfA-1; Tue, 08 Sep 2020 16:30:31 -0400 X-MC-Unique: plPdsnNNPQGkLoTRepDjfA-1 Received: by mail-qt1-f200.google.com with SMTP id y53so361972qth.2 for ; Tue, 08 Sep 2020 13:30:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=amKOtoi6vPf/V/x2bL7dd1QXHRAnGLq9RsDduAfD7yI=; b=Xo5y1O3uoKFBYlGBsg4KemDtaJUjCprzKlyJPOSc5loh8das0RU+MJ1/Sig23pit2a X3elsY5n0UQ/sBzV6/oeO+WxAXYsWHh8yrJKbkm4GwkJKQuT23PLIJwdSsghaeObLDdV vlAv3OVLgHNuXJmQZEjrAb/oJWekUUmxYO5sEvYh0btphhbzv1Q+q7GscFtaew213jAL qt8t8X3K8BzWRX3/35p/Qk89mUg+Q07rHp6OMfD3wRaFOs0CXFl1zEzVdpobJ6+IAERj a0oM3ps1R6pBEEBlitlk50PeHoXH2UYhpZ94Vj6TeF62VQ0ancZMFxf7VOcZytIfqN+H GZAA== X-Gm-Message-State: AOAM531zWVVKHIRPlIRM6s5smUVQTN3iquEu9iH0g7ixntkvquP16XTK O3lZdl0CL88UK3RKconr5FGKnvSNPn8Qd5/pdJyBJbiI2HA/vYFqeofmo4u78FLUtkWQQARZ3CZ wNy3zxU5sHk3ZPtQ= X-Received: by 2002:a05:620a:2404:: with SMTP id d4mr341544qkn.284.1599597030879; Tue, 08 Sep 2020 13:30:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzFmUngwy0dJAvKj+aoqnNnBh74yjMOE7qDM9NgQTTO1mdmcgrchSAU9NBOYwlLniDwrjjsAw== X-Received: by 2002:a05:620a:2404:: with SMTP id d4mr341523qkn.284.1599597030615; Tue, 08 Sep 2020 13:30:30 -0700 (PDT) Received: from xz-x1.redhat.com (bras-vprn-toroon474qw-lp130-11-70-53-122-15.dsl.bell.ca. [70.53.122.15]) by smtp.gmail.com with ESMTPSA id o28sm595397qtl.62.2020.09.08.13.30.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 13:30:29 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 4/6] migration: Introduce migrate_send_rp_message_req_pages() Date: Tue, 8 Sep 2020 16:30:20 -0400 Message-Id: <20200908203022.341615-5-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200908203022.341615-1-peterx@redhat.com> References: <20200908203022.341615-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0.002 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=peterx@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/08 12:55:20 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Xiaohui Li , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is another layer wrapper for sending a page request to the source VM. The new migrate_send_rp_message_req_pages() will be used elsewhere in coming patches. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 10 ++++++++-- migration/migration.h | 2 ++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 2e6b21544b..6e06b6f4e6 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -319,8 +319,8 @@ error: * Start: Address offset within the RB * Len: Length in bytes required - must be a multiple of pagesize */ -int migrate_send_rp_req_pages(MigrationIncomingState *mis, RAMBlock *rb, - ram_addr_t start) +int migrate_send_rp_message_req_pages(MigrationIncomingState *mis, + RAMBlock *rb, ram_addr_t start) { uint8_t bufc[12 + 1 + 255]; /* start (8), len (4), rbname up to 256 */ size_t msglen = 12; /* start + len */ @@ -356,6 +356,12 @@ int migrate_send_rp_req_pages(MigrationIncomingState *mis, RAMBlock *rb, return migrate_send_rp_message(mis, msg_type, msglen, bufc); } +int migrate_send_rp_req_pages(MigrationIncomingState *mis, + RAMBlock *rb, ram_addr_t start) +{ + return migrate_send_rp_message_req_pages(mis, rb, start); +} + static bool migration_colo_enabled; bool migration_incoming_colo_enabled(void) { diff --git a/migration/migration.h b/migration/migration.h index ca8dc4c773..f552725305 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -330,6 +330,8 @@ void migrate_send_rp_pong(MigrationIncomingState *mis, uint32_t value); int migrate_send_rp_req_pages(MigrationIncomingState *mis, RAMBlock *rb, ram_addr_t start); +int migrate_send_rp_message_req_pages(MigrationIncomingState *mis, + RAMBlock *rb, ram_addr_t start); void migrate_send_rp_recv_bitmap(MigrationIncomingState *mis, char *block_name); void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value); From patchwork Tue Sep 8 20:30:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 305876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A304FC433E2 for ; Tue, 8 Sep 2020 20:31:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0AEC920759 for ; Tue, 8 Sep 2020 20:31:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RvUY7ZJ8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0AEC920759 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:59410 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kFkHJ-0005z3-0Q for qemu-devel@archiver.kernel.org; Tue, 08 Sep 2020 16:31:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50276) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kFkGE-0004GF-CM for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:39 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:26119 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kFkGC-0007x0-4M for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1599597035; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Qf4Ka93KYqxz3yYxn9kdeuYj9Txj4DgAdPldhUUG8Ks=; b=RvUY7ZJ856NEWQ6Q3Yau1AtqiM1kalAOLBuWSNZWGU2QGNdWVW1OV/rQxEpjSEwsAVdixi e9sW0205WtpeyZnUUsp4x4H1Pbw0xifvRkP5SC0KMx0+5PNcUfKAv6Fe/Em79UH0QTZTXy vF59FZJkF910LLFmhKqB0qzqSTSfw5k= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-238-bKty-REHPCSOguD_I9Lm-g-1; Tue, 08 Sep 2020 16:30:33 -0400 X-MC-Unique: bKty-REHPCSOguD_I9Lm-g-1 Received: by mail-qt1-f199.google.com with SMTP id e11so324786qth.21 for ; Tue, 08 Sep 2020 13:30:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Qf4Ka93KYqxz3yYxn9kdeuYj9Txj4DgAdPldhUUG8Ks=; b=W9WmpSRY5TqHJumC9QUzYDJ/87hU+yq8u3uyc541tUSqBXyjCs1dPrrGLJx79xg6oy wphP52xlaKXl4DdmxTVDDBzFxyIcqATFm4LMuzDSTe01n2L4cGa48IYK+vGvcJMQcPfr YNp5sgQ/ROlsM+qtX1m2odxRuzYndubs+GJ4ir02hwo6sxOVVds0fasqdCc5wfD4nqXY RbntG2/xD+mY+cVgMWq6jZRrsxCBN4ULtPMyw5DMqivsV1Lf0xdykpfQy+Y8YJJO0yvr a44eaj5V9JYQA3ztCS3fWD5A/enkOCyydWu4z7ExJUTZHf80/mFgmxtP9Ey///iffEng jXsQ== X-Gm-Message-State: AOAM530VN+LFnZxv8fCxy+MFVqS9PbbZyENyI8N56mhWVBUsdniU/Abr FRSw9pqCnadqqM1zmZfV1iojM60Wea5TOCCD9FKTQwSOXJKGz1yaqBe4k/clfGWE6+jRbnT45AZ chu/2obMDXf/v59g= X-Received: by 2002:ac8:5057:: with SMTP id h23mr186834qtm.219.1599597032515; Tue, 08 Sep 2020 13:30:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxT7zsrQIDOJ0p9Fn6PTTtzWT0MltZanRSlSTO8dml1wkdVoEykMkjHxX+gMQV6ZC2M2yMD5w== X-Received: by 2002:ac8:5057:: with SMTP id h23mr186801qtm.219.1599597032207; Tue, 08 Sep 2020 13:30:32 -0700 (PDT) Received: from xz-x1.redhat.com (bras-vprn-toroon474qw-lp130-11-70-53-122-15.dsl.bell.ca. [70.53.122.15]) by smtp.gmail.com with ESMTPSA id o28sm595397qtl.62.2020.09.08.13.30.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 13:30:31 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 5/6] migration: Maintain postcopy faulted addresses Date: Tue, 8 Sep 2020 16:30:21 -0400 Message-Id: <20200908203022.341615-6-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200908203022.341615-1-peterx@redhat.com> References: <20200908203022.341615-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0.001 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/08 01:08:25 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Xiaohui Li , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Maintain a list of faulted addresses on the destination host for which we're waiting on. This is implemented using a GTree rather than a real list to make sure even there're plenty of vCPUs/threads that are faulting, the lookup will still be fast with O(log(N)) (because we'll do that after placing each page). It should bring a slight overhead, but ideally that shouldn't be a big problem simply because in most cases the requested page list will be short. Actually we did similar things for postcopy blocktime measurements. This patch didn't use that simply because: (1) blocktime measurement is towards vcpu threads only, but here we need to record all faulted addresses, including main thread and external thread (like, DPDK via vhost-user). (2) blocktime measurement will require UFFD_FEATURE_THREAD_ID, but here we don't want to add that extra dependency on the kernel version since not necessary. E.g., we don't need to know which thread faulted on which page, we also don't care about multiple threads faulting on the same page. But we only care about what addresses are faulted so waiting for a page copying from src. (3) blocktime measurement is not enabled by default. However we need this by default especially for postcopy recover. Another thing to mention is that this patch introduced a new mutex to serialize the receivedmap and the page_requested tree, however that serialization does not cover other procedures like UFFDIO_COPY. Signed-off-by: Peter Xu --- migration/migration.c | 44 +++++++++++++++++++++++++++++++++++++++- migration/migration.h | 19 ++++++++++++++++- migration/postcopy-ram.c | 18 +++++++++++++--- migration/trace-events | 2 ++ 4 files changed, 78 insertions(+), 5 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 6e06b6f4e6..3a12378429 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -143,6 +143,13 @@ static int migration_maybe_pause(MigrationState *s, int new_state); static void migrate_fd_cancel(MigrationState *s); +static gint page_request_addr_cmp(gconstpointer ap, gconstpointer bp) +{ + uint64_t a = (uint64_t) ap, b = (uint64_t) bp; + + return (a > b) - (a < b); +} + void migration_object_init(void) { MachineState *ms = MACHINE(qdev_get_machine()); @@ -165,6 +172,8 @@ void migration_object_init(void) qemu_event_init(¤t_incoming->main_thread_load_event, false); qemu_sem_init(¤t_incoming->postcopy_pause_sem_dst, 0); qemu_sem_init(¤t_incoming->postcopy_pause_sem_fault, 0); + qemu_mutex_init(¤t_incoming->page_request_mutex); + current_incoming->page_requested = g_tree_new(page_request_addr_cmp); if (!migration_object_check(current_migration, &err)) { error_report_err(err); @@ -238,6 +247,11 @@ void migration_incoming_state_destroy(void) mis->postcopy_remote_fds = NULL; } + if (mis->page_requested) { + g_tree_destroy(mis->page_requested); + mis->page_requested = NULL; + } + if (mis->socket_address_list) { qapi_free_SocketAddressList(mis->socket_address_list); mis->socket_address_list = NULL; @@ -247,6 +261,7 @@ void migration_incoming_state_destroy(void) qemu_sem_destroy(&mis->postcopy_pause_sem_dst); qemu_sem_destroy(&mis->postcopy_pause_sem_fault); qemu_mutex_destroy(&mis->rp_mutex); + qemu_mutex_destroy(&mis->page_request_mutex); } static void migrate_generate_event(int new_state) @@ -357,8 +372,35 @@ int migrate_send_rp_message_req_pages(MigrationIncomingState *mis, } int migrate_send_rp_req_pages(MigrationIncomingState *mis, - RAMBlock *rb, ram_addr_t start) + RAMBlock *rb, ram_addr_t start, uint64_t haddr) { + uint64_t aligned = haddr & (-qemu_target_page_size()); + bool received; + + WITH_QEMU_LOCK_GUARD(&mis->page_request_mutex) { + received = ramblock_recv_bitmap_test_byte_offset(rb, start); + if (!received && !g_tree_lookup(mis->page_requested, + (gpointer) aligned)) { + /* + * The page has not been received, and it's not yet in the page + * request list. Queue it. Set the value of element to 1, so that + * things like g_tree_lookup() will return TRUE (1) when found. + */ + g_tree_insert(mis->page_requested, (gpointer) aligned, + (gpointer) 1); + mis->page_requested_count++; + trace_postcopy_page_req_add(aligned, mis->page_requested_count); + } + } + + /* + * If the page is there, skip sending the message. We don't even need the + * lock because as long as the page arrived, it'll be there forever. + */ + if (received) { + return 0; + } + return migrate_send_rp_message_req_pages(mis, rb, start); } diff --git a/migration/migration.h b/migration/migration.h index f552725305..81311dc154 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -103,6 +103,23 @@ struct MigrationIncomingState { /* List of listening socket addresses */ SocketAddressList *socket_address_list; + + /* A tree of pages that we requested to the source VM */ + GTree *page_requested; + /* For debugging purpose only, but would be nice to keep */ + int page_requested_count; + /* + * The mutex helps to maintain the requested pages that we sent to the + * source, IOW, to guarantee coherent between the page_requests tree and + * the per-ramblock receivedmap. Note! This does not guarantee consistency + * of the real page copy procedures (using UFFDIO_[ZERO]COPY). E.g., even + * if one bit in receivedmap is cleared, UFFDIO_COPY could have happened + * for that page already. This is intended so that the mutex won't + * serialize and blocked by slow operations like UFFDIO_* ioctls. However + * this should be enough to make sure the page_requested tree always + * contains valid information. + */ + QemuMutex page_request_mutex; }; MigrationIncomingState *migration_incoming_get_current(void); @@ -329,7 +346,7 @@ void migrate_send_rp_shut(MigrationIncomingState *mis, void migrate_send_rp_pong(MigrationIncomingState *mis, uint32_t value); int migrate_send_rp_req_pages(MigrationIncomingState *mis, RAMBlock *rb, - ram_addr_t start); + ram_addr_t start, uint64_t haddr); int migrate_send_rp_message_req_pages(MigrationIncomingState *mis, RAMBlock *rb, ram_addr_t start); void migrate_send_rp_recv_bitmap(MigrationIncomingState *mis, diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index d333c3fd0e..a30627e838 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -684,7 +684,7 @@ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb, qemu_ram_get_idstr(rb), rb_offset); return postcopy_wake_shared(pcfd, client_addr, rb); } - migrate_send_rp_req_pages(mis, rb, aligned_rbo); + migrate_send_rp_req_pages(mis, rb, aligned_rbo, client_addr); return 0; } @@ -979,7 +979,8 @@ retry: * Send the request to the source - we want to request one * of our host page sizes (which is >= TPS) */ - ret = migrate_send_rp_req_pages(mis, rb, rb_offset); + ret = migrate_send_rp_req_pages(mis, rb, rb_offset, + msg.arg.pagefault.address); if (ret) { /* May be network failure, try to wait for recovery */ if (ret == -EIO && postcopy_pause_fault_thread(mis)) { @@ -1149,10 +1150,21 @@ static int qemu_ufd_copy_ioctl(MigrationIncomingState *mis, void *host_addr, ret = ioctl(userfault_fd, UFFDIO_ZEROPAGE, &zero_struct); } if (!ret) { + qemu_mutex_lock(&mis->page_request_mutex); ramblock_recv_bitmap_set_range(rb, host_addr, pagesize / qemu_target_page_size()); + /* + * If this page resolves a page fault for a previous recorded faulted + * address, take a special note to maintain the requested page list. + */ + if (g_tree_lookup(mis->page_requested, (gconstpointer)host_addr)) { + g_tree_remove(mis->page_requested, (gconstpointer)host_addr); + mis->page_requested_count--; + trace_postcopy_page_req_del((uint64_t)host_addr, + mis->page_requested_count); + } + qemu_mutex_unlock(&mis->page_request_mutex); mark_postcopy_blocktime_end((uintptr_t)host_addr); - } return ret; } diff --git a/migration/trace-events b/migration/trace-events index 4ab0a503d2..b89ce02cb0 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -157,6 +157,7 @@ postcopy_pause_return_path(void) "" postcopy_pause_return_path_continued(void) "" postcopy_pause_continued(void) "" postcopy_start_set_run(void) "" +postcopy_page_req_add(uint64_t addr, int count) "new page req 0x%lx total %d" source_return_path_thread_bad_end(void) "" source_return_path_thread_end(void) "" source_return_path_thread_entry(void) "" @@ -267,6 +268,7 @@ postcopy_ram_incoming_cleanup_blocktime(uint64_t total) "total blocktime %" PRIu postcopy_request_shared_page(const char *sharer, const char *rb, uint64_t rb_offset) "for %s in %s offset 0x%"PRIx64 postcopy_request_shared_page_present(const char *sharer, const char *rb, uint64_t rb_offset) "%s already %s offset 0x%"PRIx64 postcopy_wake_shared(uint64_t client_addr, const char *rb) "at 0x%"PRIx64" in %s" +postcopy_page_req_del(uint64_t addr, int count) "resolved page req 0x%lx total %d" get_mem_fault_cpu_index(int cpu, uint32_t pid) "cpu: %d, pid: %u" From patchwork Tue Sep 8 20:30:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 274204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 772F8C433E2 for ; Tue, 8 Sep 2020 20:33:18 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1EED52080A for ; Tue, 8 Sep 2020 20:33:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VxE6o32E" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EED52080A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:38830 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kFkIn-0000jh-7A for qemu-devel@archiver.kernel.org; Tue, 08 Sep 2020 16:33:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50280) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kFkGE-0004GM-LT for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:39 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:46404 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kFkGC-0007xH-T3 for qemu-devel@nongnu.org; Tue, 08 Sep 2020 16:30:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1599597036; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S9JOV0WBlguYwtR4pnvjIPBz9nBbT8zTYpwDfLLypjc=; b=VxE6o32E4RO+x4wTwEdtuuYifeeu/pxU2/UON+l3H8X0dB/FBZ4YLwiR4QNmzLIoLBAuwR dUVvVy0HALNDo5AAHMtyMqiS9mszt0CbMtkgM6U/l1WOT2nTpA+hlRnQO1GHvUHF2PT5Fc kypCh4wwRMlFk4SpvNrE/RG+hODpnrk= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-388-hmrGZniCPS2hMtAF9qziWA-1; Tue, 08 Sep 2020 16:30:34 -0400 X-MC-Unique: hmrGZniCPS2hMtAF9qziWA-1 Received: by mail-qt1-f200.google.com with SMTP id m13so343822qtu.10 for ; Tue, 08 Sep 2020 13:30:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S9JOV0WBlguYwtR4pnvjIPBz9nBbT8zTYpwDfLLypjc=; b=flENrpfQlGAoB088NV28266KTqSDnEeIQ9mC5NLV0rY0iqA5PLkXsQSTSs3alP96rR pKYjiNBR4gElbruhVsQMd92gfr0qQa1j05CCWVK7jLdFftVO/tXlQ0oh8M+npYM1/8ew 6Dz6y0F5k73nJXjV1Q80ZOZhbmjZziSdnCg2Cyn5ueVmw1JXajQoWFI2EwpPC12grPk9 3sjeF5m7AL7A4QfzI9+99tmglfVBOJlyy/gvdbffvr+mpqSOrxiG+VVcLeSEb0FkKNj0 7pUNTlCsbFDnhiVaA3Pxx0FWqYgXTh4An+2TecLxKJx0HBZVk46Fzgw0wYj6eWUTF29v kZ+Q== X-Gm-Message-State: AOAM533RMx86vQvGZ3o0a0Pet/rNaJbAIMOQSH0tSaLC7fF0SS2gfvtg USbHwDMcJ5V3vOlipggDfGcIY8IkodHN+wrUsyE4yalCFrqYk0CcCWX7BXG0T0c7xgZ4IhTq4d0 kA5SoWvjiAi/O8l0= X-Received: by 2002:ac8:2612:: with SMTP id u18mr172217qtu.363.1599597033826; Tue, 08 Sep 2020 13:30:33 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxI8GBFpKLFYZ4U+8tnTvL2nViOdrM+O8x3xuhDIrC0oimxgf69+zTJdfktOPq+sOtSbZrVIg== X-Received: by 2002:ac8:2612:: with SMTP id u18mr172188qtu.363.1599597033539; Tue, 08 Sep 2020 13:30:33 -0700 (PDT) Received: from xz-x1.redhat.com (bras-vprn-toroon474qw-lp130-11-70-53-122-15.dsl.bell.ca. [70.53.122.15]) by smtp.gmail.com with ESMTPSA id o28sm595397qtl.62.2020.09.08.13.30.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 13:30:32 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 6/6] migration: Sync requested pages after postcopy recovery Date: Tue, 8 Sep 2020 16:30:22 -0400 Message-Id: <20200908203022.341615-7-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200908203022.341615-1-peterx@redhat.com> References: <20200908203022.341615-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0.003 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.120; envelope-from=peterx@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/08 02:10:53 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Xiaohui Li , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" We synchronize the requested pages right after a postcopy recovery happens. This helps to synchronize the prioritized pages on source so that the faulted threads can be served faster. Reported-by: Xiaohui Li Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/savevm.c | 57 ++++++++++++++++++++++++++++++++++++++++++ migration/trace-events | 1 + 2 files changed, 58 insertions(+) diff --git a/migration/savevm.c b/migration/savevm.c index 304d98ff78..a84ce512b9 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2011,6 +2011,49 @@ static int loadvm_postcopy_handle_run(MigrationIncomingState *mis) return LOADVM_QUIT; } +/* We must be with page_request_mutex held */ +static gboolean postcopy_sync_page_req(gpointer key, gpointer value, + gpointer data) +{ + MigrationIncomingState *mis = data; + void *host_addr = (void *) key; + ram_addr_t rb_offset; + RAMBlock *rb; + int ret; + + rb = qemu_ram_block_from_host(host_addr, true, &rb_offset); + if (!rb) { + /* + * This should _never_ happen. However be nice for a migrating VM to + * not crash/assert. Post an error (note: intended to not use *_once + * because we do want to see all the illegal addresses; and this can + * never be triggered by the guest so we're safe) and move on next. + */ + error_report("%s: illegal host addr %p", __func__, host_addr); + /* Try the next entry */ + return FALSE; + } + + ret = migrate_send_rp_message_req_pages(mis, rb, rb_offset); + if (ret) { + /* Please refer to above comment. */ + error_report("%s: send rp message failed for addr %p", + __func__, host_addr); + return FALSE; + } + + trace_postcopy_page_req_sync((uint64_t)host_addr); + + return FALSE; +} + +static void migrate_send_rp_req_pages_pending(MigrationIncomingState *mis) +{ + WITH_QEMU_LOCK_GUARD(&mis->page_request_mutex) { + g_tree_foreach(mis->page_requested, postcopy_sync_page_req, mis); + } +} + static int loadvm_postcopy_handle_resume(MigrationIncomingState *mis) { if (mis->state != MIGRATION_STATUS_POSTCOPY_RECOVER) { @@ -2033,6 +2076,20 @@ static int loadvm_postcopy_handle_resume(MigrationIncomingState *mis) /* Tell source that "we are ready" */ migrate_send_rp_resume_ack(mis, MIGRATION_RESUME_ACK_VALUE); + /* + * After a postcopy recovery, the source should have lost the postcopy + * queue, or potentially the requested pages could have been lost during + * the network down phase. Let's re-sync with the source VM by re-sending + * all the pending pages that we eagerly need, so these threads won't get + * blocked too long due to the recovery. + * + * Without this procedure, the faulted destination VM threads (waiting for + * page requests right before the postcopy is interrupted) can keep hanging + * until the pages are sent by the source during the background copying of + * pages, or another thread faulted on the same address accidentally. + */ + migrate_send_rp_req_pages_pending(mis); + return 0; } diff --git a/migration/trace-events b/migration/trace-events index b89ce02cb0..54a6dd2761 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -49,6 +49,7 @@ vmstate_save(const char *idstr, const char *vmsd_name) "%s, %s" vmstate_load(const char *idstr, const char *vmsd_name) "%s, %s" postcopy_pause_incoming(void) "" postcopy_pause_incoming_continued(void) "" +postcopy_page_req_sync(uint64_t host_addr) "sync page req 0x%"PRIx64 # vmstate.c vmstate_load_field_error(const char *field, int ret) "field \"%s\" load failed, ret = %d"