From patchwork Thu Apr 10 16:48:59 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Huang X-Patchwork-Id: 28231 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-yk0-f198.google.com (mail-yk0-f198.google.com [209.85.160.198]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 04466212DC for ; Thu, 10 Apr 2014 16:53:46 +0000 (UTC) Received: by mail-yk0-f198.google.com with SMTP id 9sf8267367ykp.1 for ; Thu, 10 Apr 2014 09:53:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id:in-reply-to :references:cc:subject:precedence:list-id:list-unsubscribe:list-post :list-help:list-subscribe:mime-version:sender:errors-to :x-original-sender:x-original-authentication-results:mailing-list :list-archive:content-type:content-transfer-encoding; bh=7SKQwaWPbYNEFF4TwA0ra8w1aBhMOTuD0/e39kKmTv8=; b=XSB/4JRU00JPHXSaafZjmNYXQZ5a+z1kAsss9tkhk0O2Xjdtuwg6iFY9tzeE2pEiaz 7mUEehdmbhLA/Z7OOyzsaYMem4YMzffzvIL9achnq3sNiYlQ8Y16pl4Wv6dVtT6rpQ4m DQUbrIV6YjSMnZfitfgoI5tcrPwxt/O794ogNL2+9D/TF3WFwFHDabDqqpJ6dnnbfamN 5yNSwBGFXbWj6aJ0TEJ/8DLl0Wv9t+6PG8VSvMBv0znyYaZF4EbyCtrjqk4TjfztJRGC MOitGJx3/f7QprTA6bySrf9kdxXodUE+jI9wQkaZrMUBGEDHTuQ5cOTUbEq9sol8D2Eo AvNA== X-Gm-Message-State: ALoCoQmzUm85xJtCD6A8RFwvF9GurzjG2Pexo5WXqGv6tGxLA4Lht9EggXOOsdZO2cnZCpLDi+E8 X-Received: by 10.236.127.35 with SMTP id c23mr6990592yhi.51.1397148826683; Thu, 10 Apr 2014 09:53:46 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.83.137 with SMTP id j9ls1280710qgd.70.gmail; Thu, 10 Apr 2014 09:53:46 -0700 (PDT) X-Received: by 10.52.23.97 with SMTP id l1mr12567269vdf.11.1397148826542; Thu, 10 Apr 2014 09:53:46 -0700 (PDT) Received: from mail-ve0-f174.google.com (mail-ve0-f174.google.com [209.85.128.174]) by mx.google.com with ESMTPS id h11si792866veh.104.2014.04.10.09.53.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 10 Apr 2014 09:53:46 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.128.174 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.128.174; Received: by mail-ve0-f174.google.com with SMTP id oz11so3695628veb.33 for ; Thu, 10 Apr 2014 09:53:46 -0700 (PDT) X-Received: by 10.58.230.103 with SMTP id sx7mr1270691vec.28.1397148826453; Thu, 10 Apr 2014 09:53:46 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.221.72 with SMTP id ib8csp75290vcb; Thu, 10 Apr 2014 09:53:46 -0700 (PDT) X-Received: by 10.140.92.131 with SMTP id b3mr20793903qge.41.1397148825888; Thu, 10 Apr 2014 09:53:45 -0700 (PDT) Received: from lists.xen.org (lists.xen.org. [50.57.142.19]) by mx.google.com with ESMTPS id o60si2146428qga.110.2014.04.10.09.53.45 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 10 Apr 2014 09:53:45 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of xen-devel-bounces@lists.xen.org designates 50.57.142.19 as permitted sender) client-ip=50.57.142.19; Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1WYICH-0000oX-Du; Thu, 10 Apr 2014 16:51:29 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1WYICG-0000oB-5M for xen-devel@lists.xen.org; Thu, 10 Apr 2014 16:51:28 +0000 Received: from [193.109.254.147:3921] by server-14.bemta-14.messagelabs.com id 27/65-08195-F0CC6435; Thu, 10 Apr 2014 16:51:27 +0000 X-Env-Sender: w1.huang@samsung.com X-Msg-Ref: server-13.tower-27.messagelabs.com!1397148683!7563812!1 X-Originating-IP: [203.254.224.33] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMjAzLjI1NC4yMjQuMzMgPT4gMzY3NDY5\n X-StarScan-Received: X-StarScan-Version: 6.11.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 6339 invoked from network); 10 Apr 2014 16:51:26 -0000 Received: from mailout3.samsung.com (HELO mailout3.samsung.com) (203.254.224.33) by server-13.tower-27.messagelabs.com with DES-CBC3-SHA encrypted SMTP; 10 Apr 2014 16:51:26 -0000 Received: from epcpsbgm1.samsung.com (epcpsbgm1 [203.254.230.26]) by mailout3.samsung.com (Oracle Communications Messaging Server 7u4-24.01(7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTP id <0N3T0065SQTLC270@mailout3.samsung.com> for xen-devel@lists.xen.org; Fri, 11 Apr 2014 01:51:22 +0900 (KST) X-AuditID: cbfee61a-b7fb26d00000724f-20-5346cc097961 Received: from epmmp2 ( [203.254.227.17]) by epcpsbgm1.samsung.com (EPCPMTA) with SMTP id 85.0C.29263.90CC6435; Fri, 11 Apr 2014 01:51:21 +0900 (KST) Received: from weihp.spa.sarc.sas ([105.140.31.10]) by mmp2.samsung.com (Oracle Communications Messaging Server 7u4-24.01 (7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTPA id <0N3T00LLWQT18X00@mmp2.samsung.com>; Fri, 11 Apr 2014 01:51:21 +0900 (KST) From: Wei Huang To: xen-devel@lists.xen.org Date: Thu, 10 Apr 2014 16:48:59 +0000 Message-id: <1397148539-19084-7-git-send-email-w1.huang@samsung.com> X-Mailer: git-send-email 1.8.3.2 In-reply-to: <1397148539-19084-1-git-send-email-w1.huang@samsung.com> References: <1397148539-19084-1-git-send-email-w1.huang@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprGLMWRmVeSWpSXmKPExsVy+t9jQV3OM27BBktOG1u86e1gsbjRe4vN YvqfO2wWP+9cZLR4fe4js8WSj4tZLDr+TWNzYPd4PXkCo8f2JyIed67tYfM4uvs3k0ffllWM AaxRXDYpqTmZZalF+nYJXBm3Dr5jLVhylrHiTG8bcwPj6w7GLkYODgkBE4nnL6W6GDmBTDGJ C/fWs3UxcnEICUxnlJiyoQ/KaWaSaFp3lBmkik1ATeLUxf8sILaIgLTEtc+XGUGKmAUWMkr0 bv4MlhAWCJY4sfA6E4jNIqAq8XRWJ5jNK+AicW35azaIdQoSy76sBRvKKeAq8XLbIVYQWwio 5n7LK9YJjLwLGBlWMYqmFiQXFCel5xrqFSfmFpfmpesl5+duYgSH1jOpHYwrGywOMQpwMCrx 8B5Y5hosxJpYVlyZe4hRgoNZSYSX+4BbsBBvSmJlVWpRfnxRaU5q8SFGaQ4WJXHeA63WgUIC 6YklqdmpqQWpRTBZJg5OqQbGpjy1XcuyFi3JWblr8zetbxobeAJWmjXJumZlHHjT+v9J0FWG xMlpr8z33pAKkLBfHPTxHev/K+LPr+X9Lj5+sL5z//RYAamupnvxlw4xllt6aJdaTZKQljcR 0FboTzl64dS+VUcKzPyudRdWsrzbzfTOoK9Q/izDFPM9FS45Jif09Bgmbw5UYinOSDTUYi4q TgQA7NIr9CkCAAA= Cc: w1.huang@samsung.com, ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, julien.grall@linaro.org, jaeyong.yoo@samsung.com, yjhyun.yoo@samsung.com Subject: [Xen-devel] [PATCH 6/6] xen/arm: Implement toolstack for xl restore/save and migrate X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: w1.huang@samsung.com X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.128.174 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Archive: From: Alexey Sokolov This patch implements xl save/restore operation in xc_arm_migrate.c and make it compilable with existing design. The operation is also used by migration. The overall process of save is the following: 1) save guest parameters (i.e., memory map, console and store pfn, etc) 2) save memory (if it is live migration, perform dirty-page tracing) 3) save hvm states (i.e., gic, timer, vcpu etc) Signed-off-by: Alexey Sokolov Signed-off-by: Wei Huang --- config/arm32.mk | 1 + config/arm64.mk | 1 + tools/libxc/Makefile | 6 +- tools/libxc/xc_arm_migrate.c | 712 +++++++++++++++++++++++++++++++++++++++++++ tools/libxc/xc_dom_arm.c | 4 +- tools/libxl/libxl.h | 3 - tools/misc/Makefile | 4 +- 7 files changed, 724 insertions(+), 7 deletions(-) create mode 100644 tools/libxc/xc_arm_migrate.c diff --git a/config/arm32.mk b/config/arm32.mk index aa79d22..01374c9 100644 --- a/config/arm32.mk +++ b/config/arm32.mk @@ -1,6 +1,7 @@ CONFIG_ARM := y CONFIG_ARM_32 := y CONFIG_ARM_$(XEN_OS) := y +CONFIG_MIGRATE := y CONFIG_XEN_INSTALL_SUFFIX := diff --git a/config/arm64.mk b/config/arm64.mk index 15b57a4..7ac3b65 100644 --- a/config/arm64.mk +++ b/config/arm64.mk @@ -1,6 +1,7 @@ CONFIG_ARM := y CONFIG_ARM_64 := y CONFIG_ARM_$(XEN_OS) := y +CONFIG_MIGRATE := y CONFIG_XEN_INSTALL_SUFFIX := diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile index 2cca2b2..6b90b1c 100644 --- a/tools/libxc/Makefile +++ b/tools/libxc/Makefile @@ -43,8 +43,13 @@ CTRL_SRCS-$(CONFIG_MiniOS) += xc_minios.c GUEST_SRCS-y := GUEST_SRCS-y += xg_private.c xc_suspend.c ifeq ($(CONFIG_MIGRATE),y) +ifeq ($(CONFIG_X86),y) GUEST_SRCS-y += xc_domain_restore.c xc_domain_save.c GUEST_SRCS-y += xc_offline_page.c xc_compression.c +endif +ifeq ($(CONFIG_ARM),y) +GUEST_SRCS-y += xc_arm_migrate.c +endif else GUEST_SRCS-y += xc_nomigrate.c endif @@ -64,7 +69,6 @@ $(patsubst %.c,%.opic,$(ELF_SRCS-y)): CFLAGS += -Wno-pointer-sign GUEST_SRCS-y += xc_dom_core.c xc_dom_boot.c GUEST_SRCS-y += xc_dom_elfloader.c GUEST_SRCS-$(CONFIG_X86) += xc_dom_bzimageloader.c -GUEST_SRCS-$(CONFIG_X86) += xc_dom_decompress_lz4.c GUEST_SRCS-$(CONFIG_ARM) += xc_dom_armzimageloader.c GUEST_SRCS-y += xc_dom_binloader.c GUEST_SRCS-y += xc_dom_compat_linux.c diff --git a/tools/libxc/xc_arm_migrate.c b/tools/libxc/xc_arm_migrate.c new file mode 100644 index 0000000..48c77cb --- /dev/null +++ b/tools/libxc/xc_arm_migrate.c @@ -0,0 +1,712 @@ +/****************************************************************************** + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; + * version 2.1 of the License. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + * + * Copyright (c) 2013, Samsung Electronics + */ + +#include +#include +#include +#include + +#include +#include +#include +#include "xc_bitops.h" +#include "xg_private.h" + +/* + * XXX: Use correct definition for RAM base when the following patch + * xen: arm: 64-bit guest support and domU FDT autogeneration + * will be upstreamed. + */ + +#define DEF_MAX_ITERS 29 /* limit us to 30 times round loop */ +#define DEF_MAX_FACTOR 3 /* never send more than 3x p2m_size */ +#define DEF_MIN_DIRTY_PER_ITER 50 /* dirty page count to define last iter */ +#define DEF_PROGRESS_RATE 50 /* progress bar update rate */ + +/* Enable this macro for debug only: "static" migration instead of live */ +/* +#define DISABLE_LIVE_MIGRATION +*/ + +/* Enable this macro for debug only: additional debug info */ +/* +#define ARM_MIGRATE_VERBOSE +*/ + +/* + * Guest params to save: used HVM params, save flags, memory map + */ +typedef struct guest_params +{ + unsigned long console_pfn; + unsigned long store_pfn; + uint32_t flags; + xen_pfn_t start_gpfn; + xen_pfn_t max_gpfn; + uint32_t max_vcpu_id; +} guest_params_t; + +static int suspend_and_state(int (*suspend)(void*), void *data, + xc_interface *xch, int dom) +{ + xc_dominfo_t info; + if ( !(*suspend)(data) ) + { + ERROR("Suspend request failed"); + return -1; + } + + if ( (xc_domain_getinfo(xch, dom, 1, &info) != 1) || + !info.shutdown || (info.shutdown_reason != SHUTDOWN_suspend) ) + { + ERROR("Domain is not in suspended state after suspend attempt"); + return -1; + } + + return 0; +} + +static int write_exact_handled(xc_interface *xch, int fd, const void *data, + size_t size) +{ + if ( write_exact(fd, data, size) ) + { + ERROR("Write failed, check space"); + return -1; + } + return 0; +} + +/* ============ Memory ============= */ +static int save_memory(xc_interface *xch, int io_fd, uint32_t dom, + struct save_callbacks *callbacks, + uint32_t max_iters, uint32_t max_factor, + guest_params_t *params) +{ + int live = !!(params->flags & XCFLAGS_LIVE); + int debug = !!(params->flags & XCFLAGS_DEBUG); + xen_pfn_t i; + char reportbuf[80]; + int iter = 0; + int last_iter = !live; + int total_dirty_pages_num = 0; + int dirty_pages_on_prev_iter_num = 0; + int count = 0; + char *page = 0; + xen_pfn_t *busy_pages = 0; + int busy_pages_count = 0; + int busy_pages_max = 256; + + DECLARE_HYPERCALL_BUFFER(unsigned long, to_send); + + xen_pfn_t start = params->start_gpfn; + const xen_pfn_t end = params->max_gpfn; + const xen_pfn_t mem_size = end - start; + + if ( debug ) + { + IPRINTF("(save mem) start=%llx end=%llx!\n", (unsigned long long)start, + (unsigned long long)end); + } + + if ( live ) + { + if ( xc_shadow_control(xch, dom, XEN_DOMCTL_SHADOW_OP_ENABLE_LOGDIRTY, + NULL, 0, NULL, 0, NULL) < 0 ) + { + ERROR("Couldn''t enable log-dirty mode !\n"); + return -1; + } + + max_iters = max_iters ? : DEF_MAX_ITERS; + max_factor = max_factor ? : DEF_MAX_FACTOR; + + if ( debug ) + IPRINTF("Log-dirty mode enabled, max_iters=%d, max_factor=%d!\n", + max_iters, max_factor); + } + + to_send = xc_hypercall_buffer_alloc_pages(xch, to_send, + NRPAGES(bitmap_size(mem_size))); + if ( !to_send ) + { + ERROR("Couldn''t allocate to_send array!\n"); + return -1; + } + + /* send all pages on first iter */ + memset(to_send, 0xff, bitmap_size(mem_size)); + + for ( ; ; ) + { + int dirty_pages_on_current_iter_num = 0; + int frc; + iter++; + + snprintf(reportbuf, sizeof(reportbuf), + "Saving memory: iter %d (last sent %u)", + iter, dirty_pages_on_prev_iter_num); + + xc_report_progress_start(xch, reportbuf, mem_size); + + if ( (iter > 1 && + dirty_pages_on_prev_iter_num < DEF_MIN_DIRTY_PER_ITER) || + (iter == max_iters) || + (total_dirty_pages_num >= mem_size*max_factor) ) + { + if ( debug ) + IPRINTF("Last iteration"); + last_iter = 1; + } + + if ( last_iter ) + { + if ( suspend_and_state(callbacks->suspend, callbacks->data, + xch, dom) ) + { + ERROR("Domain appears not to have suspended"); + return -1; + } + } + if ( live && iter > 1 ) + { + frc = xc_shadow_control(xch, dom, XEN_DOMCTL_SHADOW_OP_CLEAN, + HYPERCALL_BUFFER(to_send), mem_size, + NULL, 0, NULL); + if ( frc != mem_size ) + { + ERROR("Error peeking shadow bitmap"); + xc_hypercall_buffer_free_pages(xch, to_send, + NRPAGES(bitmap_size(mem_size))); + return -1; + } + } + + busy_pages = malloc(sizeof(xen_pfn_t) * busy_pages_max); + + for ( i = start; i < end; ++i ) + { + if ( test_bit(i - start, to_send) ) + { + page = xc_map_foreign_range(xch, dom, PAGE_SIZE, PROT_READ, i); + if ( !page ) + { + /* This page is mapped elsewhere, should be resent later */ + busy_pages[busy_pages_count] = i; + busy_pages_count++; + if ( busy_pages_count >= busy_pages_max ) + { + busy_pages_max += 256; + busy_pages = realloc(busy_pages, sizeof(xen_pfn_t) * + busy_pages_max); + } + continue; + } + + if ( write_exact_handled(xch, io_fd, &i, sizeof(i)) || + write_exact_handled(xch, io_fd, page, PAGE_SIZE) ) + { + munmap(page, PAGE_SIZE); + free(busy_pages); + return -1; + } + count++; + munmap(page, PAGE_SIZE); + + if ( (i % DEF_PROGRESS_RATE) == 0 ) + xc_report_progress_step(xch, i - start, mem_size); + dirty_pages_on_current_iter_num++; + } + } + + while ( busy_pages_count ) + { + /* Send busy pages */ + busy_pages_count--; + i = busy_pages[busy_pages_count]; + if ( test_bit(i - start, to_send) ) + { + page = xc_map_foreign_range(xch, dom, PAGE_SIZE,PROT_READ, i); + if ( !page ) + { + IPRINTF("WARNING: 2nd attempt to save page " + "busy failed pfn=%llx", (unsigned long long)i); + continue; + } + + if ( debug ) + { + IPRINTF("save mem: resend busy page %llx\n", + (unsigned long long)i); + } + + if ( write_exact_handled(xch, io_fd, &i, sizeof(i)) || + write_exact_handled(xch, io_fd, page, PAGE_SIZE) ) + { + munmap(page, PAGE_SIZE); + free(busy_pages); + return -1; + } + count++; + munmap(page, PAGE_SIZE); + dirty_pages_on_current_iter_num++; + } + } + free(busy_pages); + + if ( debug ) + IPRINTF("Dirty pages=%d", dirty_pages_on_current_iter_num); + + xc_report_progress_step(xch, mem_size, mem_size); + + dirty_pages_on_prev_iter_num = dirty_pages_on_current_iter_num; + total_dirty_pages_num += dirty_pages_on_current_iter_num; + + if ( last_iter ) + { + xc_hypercall_buffer_free_pages(xch, to_send, + NRPAGES(bitmap_size(mem_size))); + if ( live ) + { + if ( xc_shadow_control(xch, dom, XEN_DOMCTL_SHADOW_OP_OFF, + NULL, 0, NULL, 0, NULL) < 0 ) + ERROR("Couldn''t disable log-dirty mode"); + } + break; + } + } + if ( debug ) + { + IPRINTF("save mem: pages count = %d\n", count); + } + + i = (xen_pfn_t) -1; /* end page marker */ + return write_exact_handled(xch, io_fd, &i, sizeof(i)); +} + +static int restore_memory(xc_interface *xch, int io_fd, uint32_t dom, + guest_params_t *params) +{ + xen_pfn_t end = params->max_gpfn; + xen_pfn_t gpfn; + int debug = !!(params->flags & XCFLAGS_DEBUG); + int count = 0; + char *page; + xen_pfn_t start = params->start_gpfn; + + /* TODO allocate several pages per call */ + for ( gpfn = start; gpfn < end; ++gpfn ) + { + if ( xc_domain_populate_physmap_exact(xch, dom, 1, 0, 0, &gpfn) ) + { + PERROR("Memory allocation for a new domain failed"); + return -1; + } + } + while ( 1 ) + { + + if ( read_exact(io_fd, &gpfn, sizeof(gpfn)) ) + { + PERROR("GPFN read failed during memory transfer, count=%d", count); + return -1; + } + if ( gpfn == (xen_pfn_t) -1 ) break; /* end page marker */ + + if ( gpfn < start || gpfn >= end ) + { + ERROR("GPFN %llx doesn''t belong to RAM address space, count=%d", + (unsigned long long)gpfn, count); + return -1; + } + page = xc_map_foreign_range(xch, dom, PAGE_SIZE, + PROT_READ | PROT_WRITE, gpfn); + if ( !page ) + { + PERROR("xc_map_foreign_range failed, pfn=%llx", gpfn); + return -1; + } + if ( read_exact(io_fd, page, PAGE_SIZE) ) + { + PERROR("Page data read failed during memory transfer, pfn=%llx", + gpfn); + return -1; + } + munmap(page, PAGE_SIZE); + count++; + } + + if ( debug ) + { + IPRINTF("Memory restored, pages count=%d", count); + } + return 0; +} + +/* ============ HVM context =========== */ +static int save_armhvm(xc_interface *xch, int io_fd, uint32_t dom, int debug) +{ + /* HVM: a buffer for holding HVM context */ + uint32_t hvm_buf_size = 0; + uint8_t *hvm_buf = NULL; + uint32_t rec_size; + int retval = -1; + + /* Need another buffer for HVM context */ + hvm_buf_size = xc_domain_hvm_getcontext(xch, dom, 0, 0); + if ( hvm_buf_size == -1 ) + { + ERROR("Couldn''t get HVM context size from Xen"); + goto out; + } + hvm_buf = malloc(hvm_buf_size); + + if ( !hvm_buf ) + { + ERROR("Couldn''t allocate memory for hvm buffer"); + goto out; + } + + /* Get HVM context from Xen and save it too */ + if ( (rec_size = xc_domain_hvm_getcontext(xch, dom, hvm_buf, + hvm_buf_size)) == -1 ) + { + ERROR("HVM:Could not get hvm buffer"); + goto out; + } + + if ( debug ) + IPRINTF("HVM save size %d %d", hvm_buf_size, rec_size); + + if ( write_exact_handled(xch, io_fd, &rec_size, sizeof(uint32_t)) ) + goto out; + + if ( write_exact_handled(xch, io_fd, hvm_buf, rec_size) ) + { + goto out; + } + retval = 0; + +out: + if ( hvm_buf ) + free (hvm_buf); + return retval; +} + +static int restore_armhvm(xc_interface *xch, int io_fd, + uint32_t dom, int debug) +{ + uint32_t rec_size; + uint32_t hvm_buf_size = 0; + uint8_t *hvm_buf = NULL; + int frc = 0; + int retval = -1; + + if ( read_exact(io_fd, &rec_size, sizeof(uint32_t)) ) + { + PERROR("Could not read HVM size"); + goto out; + } + + if ( !rec_size ) + { + ERROR("Zero HVM size"); + goto out; + } + + hvm_buf_size = xc_domain_hvm_getcontext(xch, dom, 0, 0); + if ( hvm_buf_size != rec_size ) + { + ERROR("HVM size for this domain is not the same as stored"); + } + + hvm_buf = malloc(hvm_buf_size); + if ( !hvm_buf ) + { + ERROR("Couldn''t allocate memory"); + goto out; + } + + if ( read_exact(io_fd, hvm_buf, hvm_buf_size) ) + { + PERROR("Could not read HVM context"); + goto out; + } + + frc = xc_domain_hvm_setcontext(xch, dom, hvm_buf, hvm_buf_size); + if ( frc ) + { + ERROR("error setting the HVM context"); + goto out; + } + retval = 0; + + if ( debug ) + { + IPRINTF("HVM restore size %d %d", hvm_buf_size, rec_size); + } +out: + if ( hvm_buf ) + free (hvm_buf); + return retval; +} + +/* ================= Console & Xenstore & Memory map =========== */ +static int save_guest_params(xc_interface *xch, int io_fd, + uint32_t dom, uint32_t flags, + guest_params_t *params) +{ + size_t sz = sizeof(guest_params_t); + xc_dominfo_t dom_info; + + params->max_gpfn = xc_domain_maximum_gpfn(xch, dom); + params->start_gpfn = (GUEST_RAM_BASE >> PAGE_SHIFT); + + if ( flags & XCFLAGS_DEBUG ) + { + IPRINTF("Guest param save size: %d ", (int)sz); + } + + if ( xc_get_hvm_param(xch, dom, HVM_PARAM_CONSOLE_PFN, + ¶ms->console_pfn) ) + { + ERROR("Can''t get console gpfn"); + return -1; + } + + if ( xc_get_hvm_param(xch, dom, HVM_PARAM_STORE_PFN, ¶ms->store_pfn) ) + { + ERROR("Can''t get store gpfn"); + return -1; + } + + if ( xc_domain_getinfo(xch, dom, 1, &dom_info ) < 0) + { + ERROR("Can''t get domain info for dom %d", dom); + return -1; + } + params->max_vcpu_id = dom_info.max_vcpu_id; + + params->flags = flags; + + if ( write_exact_handled(xch, io_fd, params, sz) ) + { + return -1; + } + + return 0; +} + +static int restore_guest_params(xc_interface *xch, int io_fd, + uint32_t dom, guest_params_t *params) +{ + size_t sz = sizeof(guest_params_t); + xen_pfn_t nr_pfns; + unsigned int maxmemkb; + + if ( read_exact(io_fd, params, sizeof(guest_params_t)) ) + { + PERROR("Can''t read guest params"); + return -1; + } + + nr_pfns = params->max_gpfn - params->start_gpfn; + maxmemkb = (unsigned int) nr_pfns << (PAGE_SHIFT - 10); + + if ( params->flags & XCFLAGS_DEBUG ) + { + IPRINTF("Guest param restore size: %d ", (int)sz); + IPRINTF("Guest memory size: %d MB", maxmemkb >> 10); + } + + if ( xc_domain_setmaxmem(xch, dom, maxmemkb) ) + { + ERROR("Can''t set memory map"); + return -1; + } + + /* Set max. number of vcpus as max_vcpu_id + 1 */ + if ( xc_domain_max_vcpus(xch, dom, params->max_vcpu_id + 1) ) + { + ERROR("Can''t set max vcpu number for domain"); + return -1; + } + + return 0; +} + +static int set_guest_params(xc_interface *xch, int io_fd, uint32_t dom, + guest_params_t *params, unsigned int console_evtchn, + domid_t console_domid, unsigned int store_evtchn, + domid_t store_domid) +{ + int rc = 0; + + if ( (rc = xc_clear_domain_page(xch, dom, params->console_pfn)) ) + { + ERROR("Can''t clear console page"); + return rc; + } + + if ( (rc = xc_clear_domain_page(xch, dom, params->store_pfn)) ) + { + ERROR("Can''t clear xenstore page"); + return rc; + } + + if ( (rc = xc_dom_gnttab_hvm_seed(xch, dom, params->console_pfn, + params->store_pfn, console_domid, + store_domid)) ) + { + ERROR("Can''t grant console and xenstore pages"); + return rc; + } + + if ( (rc = xc_set_hvm_param(xch, dom, HVM_PARAM_CONSOLE_PFN, + params->console_pfn)) ) + { + ERROR("Can''t set console gpfn"); + return rc; + } + + if ( (rc = xc_set_hvm_param(xch, dom, HVM_PARAM_STORE_PFN, + params->store_pfn)) ) + { + ERROR("Can''t set xenstore gpfn"); + return rc; + } + + if ( (rc = xc_set_hvm_param(xch, dom, HVM_PARAM_CONSOLE_EVTCHN, + console_evtchn)) ) + { + ERROR("Can''t set console event channel"); + return rc; + } + + if ( (rc = xc_set_hvm_param(xch, dom, HVM_PARAM_STORE_EVTCHN, + store_evtchn)) ) + { + ERROR("Can''t set xenstore event channel"); + return rc; + } + return 0; +} + +/* ================== Main ============== */ +int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, + uint32_t max_iters, uint32_t max_factor, uint32_t flags, + struct save_callbacks *callbacks, int hvm, + unsigned long vm_generationid_addr) +{ + int debug; + guest_params_t params; + +#ifdef ARM_MIGRATE_VERBOSE + flags |= XCFLAGS_DEBUG; +#endif + +#ifdef DISABLE_LIVE_MIGRATION + flags &= ~(XCFLAGS_LIVE); +#endif + + debug = !!(flags & XCFLAGS_DEBUG); + if ( save_guest_params(xch, io_fd, dom, flags, ¶ms) ) + { + ERROR("Can''t save guest params"); + return -1; + } + + if ( save_memory(xch, io_fd, dom, callbacks, max_iters, + max_factor, ¶ms) ) + { + ERROR("Memory not saved"); + return -1; + } + + if ( save_armhvm(xch, io_fd, dom, debug) ) + { + ERROR("HVM not saved"); + return -1; + } + + if ( debug ) + { + IPRINTF("Domain %d saved", dom); + } + return 0; +} + +int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, + unsigned int store_evtchn, unsigned long *store_gpfn, + domid_t store_domid, unsigned int console_evtchn, + unsigned long *console_gpfn, domid_t console_domid, + unsigned int hvm, unsigned int pae, int superpages, + int no_incr_generationid, int checkpointed_stream, + unsigned long *vm_generationid_addr, + struct restore_callbacks *callbacks) +{ + guest_params_t params; + int debug = 1; + + if ( restore_guest_params(xch, io_fd, dom, ¶ms) ) + { + ERROR("Can''t restore guest params"); + return -1; + } + debug = !!(params.flags & XCFLAGS_DEBUG); + + if ( restore_memory(xch, io_fd, dom, ¶ms) ) + { + ERROR("Can''t restore memory"); + return -1; + } + if ( set_guest_params(xch, io_fd, dom, ¶ms, + console_evtchn, console_domid, + store_evtchn, store_domid) ) + { + ERROR("Can''t setup guest params"); + return -1; + } + + /* Setup console and store PFNs to caller */ + *console_gpfn = params.console_pfn; + *store_gpfn = params.store_pfn; + + if ( restore_armhvm(xch, io_fd, dom, debug) ) + { + ERROR("HVM not restored"); + return -1; + } + + if ( debug ) + { + IPRINTF("Domain %d restored", dom); + } + + return 0; +} + +/* + * Local variables: + * mode: C + * c-set-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/tools/libxc/xc_dom_arm.c b/tools/libxc/xc_dom_arm.c index a40e04d..f4d3af4 100644 --- a/tools/libxc/xc_dom_arm.c +++ b/tools/libxc/xc_dom_arm.c @@ -297,7 +297,9 @@ int arch_setup_meminit(struct xc_dom_image *dom) dom->devicetree_seg.vstart, dom->devicetree_seg.vend); } - return 0; + return xc_domain_setmaxmem(dom->xch, dom->guest_domid, + (dom->total_pages + NR_MAGIC_PAGES) + << (PAGE_SHIFT - 10)); } int arch_setup_bootearly(struct xc_dom_image *dom) diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h index b2c3015..e10f4fb 100644 --- a/tools/libxl/libxl.h +++ b/tools/libxl/libxl.h @@ -441,9 +441,6 @@ * - libxl_domain_resume * - libxl_domain_remus_start */ -#if defined(__arm__) || defined(__aarch64__) -#define LIBXL_HAVE_NO_SUSPEND_RESUME 1 -#endif /* * LIBXL_HAVE_DEVICE_PCI_SEIZE diff --git a/tools/misc/Makefile b/tools/misc/Makefile index 17aeda5..0824100 100644 --- a/tools/misc/Makefile +++ b/tools/misc/Makefile @@ -11,7 +11,7 @@ HDRS = $(wildcard *.h) TARGETS-y := xenperf xenpm xen-tmem-list-parse gtraceview gtracestat xenlockprof xenwatchdogd xencov TARGETS-$(CONFIG_X86) += xen-detect xen-hvmctx xen-hvmcrash xen-lowmemd xen-mfndump -TARGETS-$(CONFIG_MIGRATE) += xen-hptool +TARGETS-$(CONFIG_X86) += xen-hptool TARGETS := $(TARGETS-y) SUBDIRS := $(SUBDIRS-y) @@ -23,7 +23,7 @@ INSTALL_BIN := $(INSTALL_BIN-y) INSTALL_SBIN-y := xen-bugtool xen-python-path xenperf xenpm xen-tmem-list-parse gtraceview \ gtracestat xenlockprof xenwatchdogd xen-ringwatch xencov INSTALL_SBIN-$(CONFIG_X86) += xen-hvmctx xen-hvmcrash xen-lowmemd xen-mfndump -INSTALL_SBIN-$(CONFIG_MIGRATE) += xen-hptool +INSTALL_SBIN-$(CONFIG_X86) += xen-hptool INSTALL_SBIN := $(INSTALL_SBIN-y) INSTALL_PRIVBIN-y := xenpvnetboot