From patchwork Tue Apr 15 21:05:18 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Huang X-Patchwork-Id: 28426 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-yh0-f70.google.com (mail-yh0-f70.google.com [209.85.213.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id B3F092036D for ; Tue, 15 Apr 2014 21:09:30 +0000 (UTC) Received: by mail-yh0-f70.google.com with SMTP id c41sf37286157yho.9 for ; Tue, 15 Apr 2014 14:09:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id:in-reply-to :references:cc:subject:precedence:list-id:list-unsubscribe:list-post :list-help:list-subscribe:mime-version:sender:errors-to :x-original-sender:x-original-authentication-results:mailing-list :list-archive:content-type:content-transfer-encoding; bh=qL7I8FZ9XSRRnSZ53RKeo8fXqd+iK+I8sGEnncFG6OI=; b=jbrGyrHCTvfcQ0lNYHQ8YNqfenrgrPotDABHPN84tCxvINBz1V84azZHYWIpYRj7is vJ6bNPh253hsmlK1Mbvwr9T7IpCs+MF+JfJTHL+9NWAhLzNQuyAoB+f7yTnJRim6MGiv yHnKR0v23KFVLkIZiMZ+MI76u67bOJ1APpHVdU+7g/4mGZKuEVCgFH6eG7JkZsrK5E0q HFkH0hBUjXA/yzJa2qsf/h5TbXqAAlatvg1bkRNyuiCsX8pZC0Ix0PgSagbT/0IuRLEp HJfPTTFb/LS/ypS5M1zekQcYlOIvIojo05Veby1yieqDhsoxcXbSUhJVXXvqHZ8sCIMI i9JA== X-Gm-Message-State: ALoCoQl0tDEGuBf/6CAJCk4OkOC2dadXh59T1OGpodXbpgFmOUeSXDW9s5pDjc1fe4CCD8NxbCjX X-Received: by 10.236.231.52 with SMTP id k50mr1690433yhq.52.1397596170164; Tue, 15 Apr 2014 14:09:30 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.19.115 with SMTP id 106ls328977qgg.75.gmail; Tue, 15 Apr 2014 14:09:30 -0700 (PDT) X-Received: by 10.221.34.7 with SMTP id sq7mr3061914vcb.5.1397596169996; Tue, 15 Apr 2014 14:09:29 -0700 (PDT) Received: from mail-vc0-f175.google.com (mail-vc0-f175.google.com [209.85.220.175]) by mx.google.com with ESMTPS id tv3si3611883vdc.90.2014.04.15.14.09.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 15 Apr 2014 14:09:29 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.175 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.175; Received: by mail-vc0-f175.google.com with SMTP id lh14so9630865vcb.6 for ; Tue, 15 Apr 2014 14:09:29 -0700 (PDT) X-Received: by 10.221.29.137 with SMTP id ry9mr3122175vcb.6.1397596169859; Tue, 15 Apr 2014 14:09:29 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.221.72 with SMTP id ib8csp262215vcb; Tue, 15 Apr 2014 14:09:29 -0700 (PDT) X-Received: by 10.50.92.98 with SMTP id cl2mr554846igb.34.1397596169037; Tue, 15 Apr 2014 14:09:29 -0700 (PDT) Received: from lists.xen.org (lists.xen.org. [50.57.142.19]) by mx.google.com with ESMTPS id gd1si17515616igd.50.2014.04.15.14.09.28 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 15 Apr 2014 14:09:29 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of xen-devel-bounces@lists.xen.org designates 50.57.142.19 as permitted sender) client-ip=50.57.142.19; Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1WaAaZ-0005Sm-1M; Tue, 15 Apr 2014 21:08:19 +0000 Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1WaAaW-0005OG-LG for xen-devel@lists.xen.org; Tue, 15 Apr 2014 21:08:17 +0000 Received: from [85.158.143.35:56309] by server-3.bemta-4.messagelabs.com id 1F/E2-13602-FBF9D435; Tue, 15 Apr 2014 21:08:15 +0000 X-Env-Sender: w1.huang@samsung.com X-Msg-Ref: server-6.tower-21.messagelabs.com!1397596091!5511383!1 X-Originating-IP: [203.254.224.25] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMjAzLjI1NC4yMjQuMjUgPT4gMzcxMzg3\n X-StarScan-Received: X-StarScan-Version: 6.11.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 1377 invoked from network); 15 Apr 2014 21:08:14 -0000 Received: from mailout2.samsung.com (HELO mailout2.samsung.com) (203.254.224.25) by server-6.tower-21.messagelabs.com with DES-CBC3-SHA encrypted SMTP; 15 Apr 2014 21:08:14 -0000 Received: from epcpsbgm2.samsung.com (epcpsbgm2 [203.254.230.27]) by mailout2.samsung.com (Oracle Communications Messaging Server 7u4-24.01(7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTP id <0N43008AQC1NPAA0@mailout2.samsung.com> for xen-devel@lists.xen.org; Wed, 16 Apr 2014 06:08:11 +0900 (KST) X-AuditID: cbfee61b-b7f456d000006dfd-f4-534d9fba585e Received: from epmmp2 ( [203.254.227.17]) by epcpsbgm2.samsung.com (EPCPMTA) with SMTP id 78.AD.28157.ABF9D435; Wed, 16 Apr 2014 06:08:10 +0900 (KST) Received: from weihp.spa.sarc.sas ([105.140.31.10]) by mmp2.samsung.com (Oracle Communications Messaging Server 7u4-24.01 (7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTPA id <0N43002SLC0INC00@mmp2.samsung.com>; Wed, 16 Apr 2014 06:08:10 +0900 (KST) From: Wei Huang To: xen-devel@lists.xen.org Date: Tue, 15 Apr 2014 16:05:18 -0500 Message-id: <1397595918-30419-14-git-send-email-w1.huang@samsung.com> X-Mailer: git-send-email 1.7.9.5 In-reply-to: <1397595918-30419-1-git-send-email-w1.huang@samsung.com> References: <1397595918-30419-1-git-send-email-w1.huang@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprBLMWRmVeSWpSXmKPExsVy+t9jQd1d832DDR5dtrJY/HgXu8Wb3g4W ixu9t9gspv+5w2bx885FRovX5z4yWyz5uJjFouPfNDYHDo/Xkycwemx/IuJx59oeNo+ju38z efRtWcUYwBrFZZOSmpNZllqkb5fAldG3YSpbwfqTjBUHlhs0MB5rZexi5OSQEDCRuHX0LSuE LSZx4d56ti5GLg4hgemMEn1fbzGDJIQEmpkkfvTpg9hsAmoSpy7+ZwGxRQSkJa59vswI0sAs sIdRomF3H1iDsECIxKMzd9m7GDk4WARUJS6+0QQJ8wq4SnzvmQ8WlhBQkJgzyQYkzAkUntyw mxFiVazEh645TBMYeRcwMqxiFE0tSC4oTkrPNdIrTswtLs1L10vOz93ECA6vZ9I7GFc1WBxi FOBgVOLhnZnjGyzEmlhWXJl7iFGCg1lJhHdDC1CINyWxsiq1KD++qDQntfgQozQHi5I478FW 60AhgfTEktTs1NSC1CKYLBMHp1QDo9CWh+o7j63j5eLzX9ySy+p3M1D8oVLSbMXnFR/us627 0JVfI5vm8zpF4x9X0947bR1VV69Uy6RJXqq2kRbPiv/wbW6b+/4XD29Z5X3t1hTb8NWlwTFq YeCbW8zNR28cTL6yoGbNqjqx09cjTTjXcEnq2z/f9e7dY6d34glfdHe9c7jANCfMTImlOCPR UIu5qDgRAIkSG3IrAgAA Cc: w1.huang@samsung.com, ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, andrew.cooper3@citrix.com, julien.grall@linaro.org, jaeyong.yoo@samsung.com, yjhyun.yoo@samsung.com Subject: [Xen-devel] [RFC v2 6/6] xen/arm: Implement toolstack for xl restore/save and migrate X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: w1.huang@samsung.com X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.175 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Archive: From: Jaeyong Yoo This patch implements xl save/restore operation in xc_arm_migrate.c and make it compilable with existing design. The operation is also used by migration. The overall process of save is the following: 1) save guest parameters (i.e., memory map, console and store pfn, etc) 2) save memory (if it is live migration, perform dirty-page tracing) 3) save hvm states (i.e., gic, timer, vcpu etc) Signed-off-by: Alexey Sokolov Signed-off-by: Wei Huang --- config/arm32.mk | 1 + config/arm64.mk | 1 + tools/libxc/Makefile | 6 +- tools/libxc/xc_arm_migrate.c | 702 ++++++++++++++++++++++++++++++++++++++++++ tools/libxc/xc_dom_arm.c | 4 +- tools/libxl/libxl.h | 3 - tools/misc/Makefile | 4 +- 7 files changed, 714 insertions(+), 7 deletions(-) create mode 100644 tools/libxc/xc_arm_migrate.c diff --git a/config/arm32.mk b/config/arm32.mk index aa79d22..01374c9 100644 --- a/config/arm32.mk +++ b/config/arm32.mk @@ -1,6 +1,7 @@ CONFIG_ARM := y CONFIG_ARM_32 := y CONFIG_ARM_$(XEN_OS) := y +CONFIG_MIGRATE := y CONFIG_XEN_INSTALL_SUFFIX := diff --git a/config/arm64.mk b/config/arm64.mk index 15b57a4..7ac3b65 100644 --- a/config/arm64.mk +++ b/config/arm64.mk @@ -1,6 +1,7 @@ CONFIG_ARM := y CONFIG_ARM_64 := y CONFIG_ARM_$(XEN_OS) := y +CONFIG_MIGRATE := y CONFIG_XEN_INSTALL_SUFFIX := diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile index 2cca2b2..6b90b1c 100644 --- a/tools/libxc/Makefile +++ b/tools/libxc/Makefile @@ -43,8 +43,13 @@ CTRL_SRCS-$(CONFIG_MiniOS) += xc_minios.c GUEST_SRCS-y := GUEST_SRCS-y += xg_private.c xc_suspend.c ifeq ($(CONFIG_MIGRATE),y) +ifeq ($(CONFIG_X86),y) GUEST_SRCS-y += xc_domain_restore.c xc_domain_save.c GUEST_SRCS-y += xc_offline_page.c xc_compression.c +endif +ifeq ($(CONFIG_ARM),y) +GUEST_SRCS-y += xc_arm_migrate.c +endif else GUEST_SRCS-y += xc_nomigrate.c endif @@ -64,7 +69,6 @@ $(patsubst %.c,%.opic,$(ELF_SRCS-y)): CFLAGS += -Wno-pointer-sign GUEST_SRCS-y += xc_dom_core.c xc_dom_boot.c GUEST_SRCS-y += xc_dom_elfloader.c GUEST_SRCS-$(CONFIG_X86) += xc_dom_bzimageloader.c -GUEST_SRCS-$(CONFIG_X86) += xc_dom_decompress_lz4.c GUEST_SRCS-$(CONFIG_ARM) += xc_dom_armzimageloader.c GUEST_SRCS-y += xc_dom_binloader.c GUEST_SRCS-y += xc_dom_compat_linux.c diff --git a/tools/libxc/xc_arm_migrate.c b/tools/libxc/xc_arm_migrate.c new file mode 100644 index 0000000..ab2b94c --- /dev/null +++ b/tools/libxc/xc_arm_migrate.c @@ -0,0 +1,702 @@ +/* + * Copyright (c) 2013, Samsung Electronics + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * this program; if not, write to the Free Software Foundation, Inc., 59 Temple + * Place - Suite 330, Boston, MA 02111-1307 USA. + */ + +#include +#include +#include +#include + +#include +#include +#include +#include "xc_bitops.h" +#include "xg_private.h" + +#define DEF_MAX_ITERS 29 /* limit us to 30 times round loop */ +#define DEF_MAX_FACTOR 3 /* never send more than 3x p2m_size */ +#define DEF_MIN_DIRTY_PER_ITER 50 /* dirty page count to define last iter */ +#define DEF_PROGRESS_RATE 50 /* progress bar update rate */ + +//#define DISABLE_LIVE_MIGRATION + +//#define ARM_MIGRATE_VERBOSE + +/* + * Guest params to save: used HVM params, save flags, memory map + */ +typedef struct guest_params +{ + unsigned long console_pfn; + unsigned long store_pfn; + uint32_t flags; + xen_pfn_t start_gpfn; + xen_pfn_t max_gpfn; + uint32_t max_vcpu_id; +} guest_params_t; + +static int suspend_and_state(int (*suspend)(void*), void *data, + xc_interface *xch, int dom) +{ + xc_dominfo_t info; + if ( !(*suspend)(data) ) + { + ERROR("Suspend request failed"); + return -1; + } + + if ( (xc_domain_getinfo(xch, dom, 1, &info) != 1) || + !info.shutdown || (info.shutdown_reason != SHUTDOWN_suspend) ) + { + ERROR("Domain is not in suspended state after suspend attempt"); + return -1; + } + + return 0; +} + +static int write_exact_handled(xc_interface *xch, int fd, const void *data, + size_t size) +{ + if ( write_exact(fd, data, size) ) + { + ERROR("Write failed, check space"); + return -1; + } + return 0; +} + +/* ============ Memory ============= */ +static int save_memory(xc_interface *xch, int io_fd, uint32_t dom, + struct save_callbacks *callbacks, + uint32_t max_iters, uint32_t max_factor, + guest_params_t *params) +{ + int live = !!(params->flags & XCFLAGS_LIVE); + int debug = !!(params->flags & XCFLAGS_DEBUG); + xen_pfn_t i; + char reportbuf[80]; + int iter = 0; + int last_iter = !live; + int total_dirty_pages_num = 0; + int dirty_pages_on_prev_iter_num = 0; + int count = 0; + char *page = 0; + xen_pfn_t *busy_pages = 0; + int busy_pages_count = 0; + int busy_pages_max = 256; + + DECLARE_HYPERCALL_BUFFER(unsigned long, to_send); + + xen_pfn_t start = params->start_gpfn; + const xen_pfn_t end = params->max_gpfn; + const xen_pfn_t mem_size = end - start; + + if ( debug ) + { + IPRINTF("(save mem) start=%llx end=%llx!\n", (unsigned long long)start, + (unsigned long long)end); + } + + if ( live ) + { + if ( xc_shadow_control(xch, dom, XEN_DOMCTL_SHADOW_OP_ENABLE_LOGDIRTY, + NULL, 0, NULL, 0, NULL) < 0 ) + { + ERROR("Couldn't enable log-dirty mode !\n"); + return -1; + } + + max_iters = max_iters ? : DEF_MAX_ITERS; + max_factor = max_factor ? : DEF_MAX_FACTOR; + + if ( debug ) + IPRINTF("Log-dirty mode enabled, max_iters=%d, max_factor=%d!\n", + max_iters, max_factor); + } + + to_send = xc_hypercall_buffer_alloc_pages(xch, to_send, + NRPAGES(bitmap_size(mem_size))); + if ( !to_send ) + { + ERROR("Couldn't allocate to_send array!\n"); + return -1; + } + + /* send all pages on first iter */ + memset(to_send, 0xff, bitmap_size(mem_size)); + + for ( ; ; ) + { + int dirty_pages_on_current_iter_num = 0; + int frc; + iter++; + + snprintf(reportbuf, sizeof(reportbuf), + "Saving memory: iter %d (last sent %u)", + iter, dirty_pages_on_prev_iter_num); + + xc_report_progress_start(xch, reportbuf, mem_size); + + if ( (iter > 1 && + dirty_pages_on_prev_iter_num < DEF_MIN_DIRTY_PER_ITER) || + (iter == max_iters) || + (total_dirty_pages_num >= mem_size*max_factor) ) + { + if ( debug ) + IPRINTF("Last iteration"); + last_iter = 1; + } + + if ( last_iter ) + { + if ( suspend_and_state(callbacks->suspend, callbacks->data, + xch, dom) ) + { + ERROR("Domain appears not to have suspended"); + return -1; + } + } + if ( live && iter > 1 ) + { + frc = xc_shadow_control(xch, dom, XEN_DOMCTL_SHADOW_OP_CLEAN, + HYPERCALL_BUFFER(to_send), mem_size, + NULL, 0, NULL); + if ( frc != mem_size ) + { + ERROR("Error peeking shadow bitmap"); + xc_hypercall_buffer_free_pages(xch, to_send, + NRPAGES(bitmap_size(mem_size))); + return -1; + } + } + + busy_pages = malloc(sizeof(xen_pfn_t) * busy_pages_max); + + for ( i = start; i < end; ++i ) + { + if ( test_bit(i - start, to_send) ) + { + page = xc_map_foreign_range(xch, dom, PAGE_SIZE, PROT_READ, i); + if ( !page ) + { + /* This page is mapped elsewhere, should be resent later */ + busy_pages[busy_pages_count] = i; + busy_pages_count++; + if ( busy_pages_count >= busy_pages_max ) + { + busy_pages_max += 256; + busy_pages = realloc(busy_pages, sizeof(xen_pfn_t) * + busy_pages_max); + } + continue; + } + + if ( write_exact_handled(xch, io_fd, &i, sizeof(i)) || + write_exact_handled(xch, io_fd, page, PAGE_SIZE) ) + { + munmap(page, PAGE_SIZE); + free(busy_pages); + return -1; + } + count++; + munmap(page, PAGE_SIZE); + + if ( (i % DEF_PROGRESS_RATE) == 0 ) + xc_report_progress_step(xch, i - start, mem_size); + dirty_pages_on_current_iter_num++; + } + } + + while ( busy_pages_count ) + { + /* Send busy pages */ + busy_pages_count--; + i = busy_pages[busy_pages_count]; + if ( test_bit(i - start, to_send) ) + { + page = xc_map_foreign_range(xch, dom, PAGE_SIZE,PROT_READ, i); + if ( !page ) + { + IPRINTF("WARNING: 2nd attempt to save page " + "busy failed pfn=%llx", (unsigned long long)i); + continue; + } + + if ( debug ) + { + IPRINTF("save mem: resend busy page %llx\n", + (unsigned long long)i); + } + + if ( write_exact_handled(xch, io_fd, &i, sizeof(i)) || + write_exact_handled(xch, io_fd, page, PAGE_SIZE) ) + { + munmap(page, PAGE_SIZE); + free(busy_pages); + return -1; + } + count++; + munmap(page, PAGE_SIZE); + dirty_pages_on_current_iter_num++; + } + } + free(busy_pages); + + if ( debug ) + IPRINTF("Dirty pages=%d", dirty_pages_on_current_iter_num); + + xc_report_progress_step(xch, mem_size, mem_size); + + dirty_pages_on_prev_iter_num = dirty_pages_on_current_iter_num; + total_dirty_pages_num += dirty_pages_on_current_iter_num; + + if ( last_iter ) + { + xc_hypercall_buffer_free_pages(xch, to_send, + NRPAGES(bitmap_size(mem_size))); + if ( live ) + { + if ( xc_shadow_control(xch, dom, XEN_DOMCTL_SHADOW_OP_OFF, + NULL, 0, NULL, 0, NULL) < 0 ) + ERROR("Couldn't disable log-dirty mode"); + } + break; + } + } + if ( debug ) + { + IPRINTF("save mem: pages count = %d\n", count); + } + + i = (xen_pfn_t) -1; /* end page marker */ + return write_exact_handled(xch, io_fd, &i, sizeof(i)); +} + +static int restore_memory(xc_interface *xch, int io_fd, uint32_t dom, + guest_params_t *params) +{ + xen_pfn_t end = params->max_gpfn; + xen_pfn_t gpfn; + int debug = !!(params->flags & XCFLAGS_DEBUG); + int count = 0; + char *page; + xen_pfn_t start = params->start_gpfn; + + /* TODO allocate several pages per call */ + for ( gpfn = start; gpfn < end; ++gpfn ) + { + if ( xc_domain_populate_physmap_exact(xch, dom, 1, 0, 0, &gpfn) ) + { + PERROR("Memory allocation for a new domain failed"); + return -1; + } + } + + while ( 1 ) + { + + if ( read_exact(io_fd, &gpfn, sizeof(gpfn)) ) + { + PERROR("GPFN read failed during memory transfer, count=%d", count); + return -1; + } + if ( gpfn == (xen_pfn_t) -1 ) break; /* end page marker */ + + if ( gpfn < start || gpfn >= end ) + { + ERROR("GPFN %llx doesn't belong to RAM address space, count=%d", + (unsigned long long)gpfn, count); + return -1; + } + page = xc_map_foreign_range(xch, dom, PAGE_SIZE, + PROT_READ | PROT_WRITE, gpfn); + if ( !page ) + { + PERROR("xc_map_foreign_range failed, pfn=%llx", gpfn); + return -1; + } + if ( read_exact(io_fd, page, PAGE_SIZE) ) + { + PERROR("Page data read failed during memory transfer, pfn=%llx", + gpfn); + return -1; + } + munmap(page, PAGE_SIZE); + count++; + } + + if ( debug ) + { + IPRINTF("Memory restored, pages count=%d", count); + } + return 0; +} + +/* ============ HVM context =========== */ +static int save_armhvm(xc_interface *xch, int io_fd, uint32_t dom, int debug) +{ + /* HVM: a buffer for holding HVM context */ + uint32_t hvm_buf_size = 0; + uint8_t *hvm_buf = NULL; + uint32_t rec_size; + int retval = -1; + + /* Need another buffer for HVM context */ + hvm_buf_size = xc_domain_hvm_getcontext(xch, dom, 0, 0); + if ( hvm_buf_size == -1 ) + { + ERROR("Couldn't get HVM context size from Xen"); + goto out; + } + hvm_buf = malloc(hvm_buf_size); + + if ( !hvm_buf ) + { + ERROR("Couldn't allocate memory for hvm buffer"); + goto out; + } + + /* Get HVM context from Xen and save it too */ + if ( (rec_size = xc_domain_hvm_getcontext(xch, dom, hvm_buf, + hvm_buf_size)) == -1 ) + { + ERROR("HVM:Could not get hvm buffer"); + goto out; + } + + if ( debug ) + IPRINTF("HVM save size %d %d", hvm_buf_size, rec_size); + + if ( write_exact_handled(xch, io_fd, &rec_size, sizeof(uint32_t)) ) + goto out; + + if ( write_exact_handled(xch, io_fd, hvm_buf, rec_size) ) + { + goto out; + } + + retval = 0; + +out: + if ( hvm_buf ) + free (hvm_buf); + + return retval; +} + +static int restore_armhvm(xc_interface *xch, int io_fd, + uint32_t dom, int debug) +{ + uint32_t rec_size; + uint32_t hvm_buf_size = 0; + uint8_t *hvm_buf = NULL; + int frc = 0; + int retval = -1; + + if ( read_exact(io_fd, &rec_size, sizeof(uint32_t)) ) + { + PERROR("Could not read HVM size"); + goto out; + } + + if ( !rec_size ) + { + ERROR("Zero HVM size"); + goto out; + } + + hvm_buf_size = xc_domain_hvm_getcontext(xch, dom, 0, 0); + if ( hvm_buf_size != rec_size ) + { + ERROR("HVM size for this domain is not the same as stored"); + } + + hvm_buf = malloc(hvm_buf_size); + if ( !hvm_buf ) + { + ERROR("Couldn't allocate memory"); + goto out; + } + + if ( read_exact(io_fd, hvm_buf, hvm_buf_size) ) + { + PERROR("Could not read HVM context"); + goto out; + } + + frc = xc_domain_hvm_setcontext(xch, dom, hvm_buf, hvm_buf_size); + if ( frc ) + { + ERROR("error setting the HVM context"); + goto out; + } + retval = 0; + + if ( debug ) + { + IPRINTF("HVM restore size %d %d", hvm_buf_size, rec_size); + } +out: + if ( hvm_buf ) + free (hvm_buf); + return retval; +} + +/* ================= Console & Xenstore & Memory map =========== */ +static int save_guest_params(xc_interface *xch, int io_fd, + uint32_t dom, uint32_t flags, + guest_params_t *params) +{ + size_t sz = sizeof(guest_params_t); + xc_dominfo_t dom_info; + + params->max_gpfn = xc_domain_maximum_gpfn(xch, dom); + params->start_gpfn = (GUEST_RAM_BASE >> PAGE_SHIFT); + + if ( flags & XCFLAGS_DEBUG ) + { + IPRINTF("Guest param save size: %d ", (int)sz); + } + + if ( xc_get_hvm_param(xch, dom, HVM_PARAM_CONSOLE_PFN, + ¶ms->console_pfn) ) + { + ERROR("Can't get console gpfn"); + return -1; + } + + if ( xc_get_hvm_param(xch, dom, HVM_PARAM_STORE_PFN, ¶ms->store_pfn) ) + { + ERROR("Can't get store gpfn"); + return -1; + } + + if ( xc_domain_getinfo(xch, dom, 1, &dom_info ) < 0) + { + ERROR("Can't get domain info for dom %d", dom); + return -1; + } + params->max_vcpu_id = dom_info.max_vcpu_id; + + params->flags = flags; + + if ( write_exact_handled(xch, io_fd, params, sz) ) + { + return -1; + } + + return 0; +} + +static int restore_guest_params(xc_interface *xch, int io_fd, + uint32_t dom, guest_params_t *params) +{ + size_t sz = sizeof(guest_params_t); + xen_pfn_t nr_pfns; + unsigned int maxmemkb; + + if ( read_exact(io_fd, params, sizeof(guest_params_t)) ) + { + PERROR("Can't read guest params"); + return -1; + } + + nr_pfns = params->max_gpfn - params->start_gpfn; + maxmemkb = (unsigned int) nr_pfns << (PAGE_SHIFT - 10); + + if ( params->flags & XCFLAGS_DEBUG ) + { + IPRINTF("Guest param restore size: %d ", (int)sz); + IPRINTF("Guest memory size: %d MB", maxmemkb >> 10); + } + + if ( xc_domain_setmaxmem(xch, dom, maxmemkb) ) + { + ERROR("Can't set memory map"); + return -1; + } + + /* Set max. number of vcpus as max_vcpu_id + 1 */ + if ( xc_domain_max_vcpus(xch, dom, params->max_vcpu_id + 1) ) + { + ERROR("Can't set max vcpu number for domain"); + return -1; + } + + return 0; +} + +static int set_guest_params(xc_interface *xch, int io_fd, uint32_t dom, + guest_params_t *params, unsigned int console_evtchn, + domid_t console_domid, unsigned int store_evtchn, + domid_t store_domid) +{ + int rc = 0; + + if ( (rc = xc_clear_domain_page(xch, dom, params->console_pfn)) ) + { + ERROR("Can't clear console page"); + return rc; + } + + if ( (rc = xc_clear_domain_page(xch, dom, params->store_pfn)) ) + { + ERROR("Can't clear xenstore page"); + return rc; + } + + if ( (rc = xc_dom_gnttab_hvm_seed(xch, dom, params->console_pfn, + params->store_pfn, console_domid, + store_domid)) ) + { + ERROR("Can't grant console and xenstore pages"); + return rc; + } + + if ( (rc = xc_set_hvm_param(xch, dom, HVM_PARAM_CONSOLE_PFN, + params->console_pfn)) ) + { + ERROR("Can't set console gpfn"); + return rc; + } + + if ( (rc = xc_set_hvm_param(xch, dom, HVM_PARAM_STORE_PFN, + params->store_pfn)) ) + { + ERROR("Can't set xenstore gpfn"); + return rc; + } + + if ( (rc = xc_set_hvm_param(xch, dom, HVM_PARAM_CONSOLE_EVTCHN, + console_evtchn)) ) + { + ERROR("Can't set console event channel"); + return rc; + } + + if ( (rc = xc_set_hvm_param(xch, dom, HVM_PARAM_STORE_EVTCHN, + store_evtchn)) ) + { + ERROR("Can't set xenstore event channel"); + return rc; + } + return 0; +} + +/* ================== Main ============== */ +int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, + uint32_t max_iters, uint32_t max_factor, uint32_t flags, + struct save_callbacks *callbacks, int hvm, + unsigned long vm_generationid_addr) +{ + int debug; + guest_params_t params; + +#ifdef ARM_MIGRATE_VERBOSE + flags |= XCFLAGS_DEBUG; +#endif + +#ifdef DISABLE_LIVE_MIGRATION + flags &= ~(XCFLAGS_LIVE); +#endif + + debug = !!(flags & XCFLAGS_DEBUG); + if ( save_guest_params(xch, io_fd, dom, flags, ¶ms) ) + { + ERROR("Can't save guest params"); + return -1; + } + + if ( save_memory(xch, io_fd, dom, callbacks, max_iters, + max_factor, ¶ms) ) + { + ERROR("Memory not saved"); + return -1; + } + + if ( save_armhvm(xch, io_fd, dom, debug) ) + { + ERROR("HVM not saved"); + return -1; + } + + if ( debug ) + { + IPRINTF("Domain %d saved", dom); + } + return 0; +} + +int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, + unsigned int store_evtchn, unsigned long *store_gpfn, + domid_t store_domid, unsigned int console_evtchn, + unsigned long *console_gpfn, domid_t console_domid, + unsigned int hvm, unsigned int pae, int superpages, + int no_incr_generationid, int checkpointed_stream, + unsigned long *vm_generationid_addr, + struct restore_callbacks *callbacks) +{ + guest_params_t params; + int debug = 1; + + if ( restore_guest_params(xch, io_fd, dom, ¶ms) ) + { + ERROR("Can't restore guest params"); + return -1; + } + debug = !!(params.flags & XCFLAGS_DEBUG); + + if ( restore_memory(xch, io_fd, dom, ¶ms) ) + { + ERROR("Can't restore memory"); + return -1; + } + if ( set_guest_params(xch, io_fd, dom, ¶ms, + console_evtchn, console_domid, + store_evtchn, store_domid) ) + { + ERROR("Can't setup guest params"); + return -1; + } + + /* Setup console and store PFNs to caller */ + *console_gpfn = params.console_pfn; + *store_gpfn = params.store_pfn; + + if ( restore_armhvm(xch, io_fd, dom, debug) ) + { + ERROR("HVM not restored"); + return -1; + } + + if ( debug ) + { + IPRINTF("Domain %d restored", dom); + } + + return 0; +} + +/* + * Local variables: + * mode: C + * c-set-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/tools/libxc/xc_dom_arm.c b/tools/libxc/xc_dom_arm.c index f051515..044a8de 100644 --- a/tools/libxc/xc_dom_arm.c +++ b/tools/libxc/xc_dom_arm.c @@ -335,7 +335,9 @@ int arch_setup_meminit(struct xc_dom_image *dom) modbase += dtb_size; } - return 0; + return xc_domain_setmaxmem(dom->xch, dom->guest_domid, + (dom->total_pages + NR_MAGIC_PAGES) + << (PAGE_SHIFT - 10)); } int arch_setup_bootearly(struct xc_dom_image *dom) diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h index b2c3015..e10f4fb 100644 --- a/tools/libxl/libxl.h +++ b/tools/libxl/libxl.h @@ -441,9 +441,6 @@ * - libxl_domain_resume * - libxl_domain_remus_start */ -#if defined(__arm__) || defined(__aarch64__) -#define LIBXL_HAVE_NO_SUSPEND_RESUME 1 -#endif /* * LIBXL_HAVE_DEVICE_PCI_SEIZE diff --git a/tools/misc/Makefile b/tools/misc/Makefile index 17aeda5..0824100 100644 --- a/tools/misc/Makefile +++ b/tools/misc/Makefile @@ -11,7 +11,7 @@ HDRS = $(wildcard *.h) TARGETS-y := xenperf xenpm xen-tmem-list-parse gtraceview gtracestat xenlockprof xenwatchdogd xencov TARGETS-$(CONFIG_X86) += xen-detect xen-hvmctx xen-hvmcrash xen-lowmemd xen-mfndump -TARGETS-$(CONFIG_MIGRATE) += xen-hptool +TARGETS-$(CONFIG_X86) += xen-hptool TARGETS := $(TARGETS-y) SUBDIRS := $(SUBDIRS-y) @@ -23,7 +23,7 @@ INSTALL_BIN := $(INSTALL_BIN-y) INSTALL_SBIN-y := xen-bugtool xen-python-path xenperf xenpm xen-tmem-list-parse gtraceview \ gtracestat xenlockprof xenwatchdogd xen-ringwatch xencov INSTALL_SBIN-$(CONFIG_X86) += xen-hvmctx xen-hvmcrash xen-lowmemd xen-mfndump -INSTALL_SBIN-$(CONFIG_MIGRATE) += xen-hptool +INSTALL_SBIN-$(CONFIG_X86) += xen-hptool INSTALL_SBIN := $(INSTALL_SBIN-y) INSTALL_PRIVBIN-y := xenpvnetboot