From patchwork Tue Jun 13 14:07:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Maydell X-Patchwork-Id: 105355 Delivered-To: patch@linaro.org Received: by 10.140.91.77 with SMTP id y71csp430712qgd; Tue, 13 Jun 2017 07:21:23 -0700 (PDT) X-Received: by 10.80.175.165 with SMTP id h34mr45132edd.87.1497363683260; Tue, 13 Jun 2017 07:21:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1497363683; cv=none; d=google.com; s=arc-20160816; b=tdDsXaMb3leF8pntlX1yD/OSWqz/q/ML4FG5Hyp0zDMCdZeEqbJ681BZOtdGNQx3dQ Z5j/i63q1n5mY1j2CgKpGaS/1w98+IenhwUjnYPKoNqgyrkD3AzH1Rhii/FEfu4xrTFf M4VurBSff0wEOEEEd9ZJXc1JjM4sX8AwpXIJJRY+QH1cFc8OG/x+Kv7kYEN2UaVyQQnJ gv9o3ATfGdamJNH50FroXAdEyIV0BrIKPiUwLpATnTI+swpszZR22Vt/bQ8oadqic53y Fn0nOD4MIk5dz6h63Va0UNDT1kweehumOCOPXzxpwN0KBLIT8HedmO+j4dfKbL5/e6PM bPSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:arc-authentication-results; bh=bbm1SXF0hlcPb9ulCpiwHBIc7MZkzvz5jRzVIMqmVyQ=; b=Jer3Z7CoMryfvDAWdG6LkihfC4SJtuCQhZQzVZ2R40eqhwxQDK4driV/FkCB/CaLPf P35uMOEP4CbOnNzfNxF9o/rGz/guwb3USMfDEY5MyZ0xrtog2ia5TMAaYzPS9/PKDcZj LJk7JKH+oysYX7eSxKlcXkgpqFR8mbK2PZuF5i6wz2p1jrINfFn/BXyyRvluT4wTsPO9 Op9SJu5T5oKHuDSDAwpw1lngyvG2SAp5nTxEsr2yFM+4JV+RY7kPxktv0KvUd74/s2gu yZtjPCC/gdm37f5javxrGRkIzuzDMSRABVK22RDKrccW/TZg7gzCHnfjlLs2tK01rySg vs9Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [208.118.235.17]) by mx.google.com with ESMTPS id g39si28323edb.262.2017.06.13.07.21.22 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 13 Jun 2017 07:21:23 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:43534 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dKmh7-0000oV-Ua for patch@linaro.org; Tue, 13 Jun 2017 10:21:21 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33072) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dKmTY-0005Em-Nc for qemu-devel@nongnu.org; Tue, 13 Jun 2017 10:07:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dKmTX-0006fe-Co for qemu-devel@nongnu.org; Tue, 13 Jun 2017 10:07:20 -0400 Received: from orth.archaic.org.uk ([2001:8b0:1d0::2]:37257) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dKmTX-0006cz-39 for qemu-devel@nongnu.org; Tue, 13 Jun 2017 10:07:19 -0400 Received: from pm215 by orth.archaic.org.uk with local (Exim 4.84_2) (envelope-from ) id 1dKmTV-00015V-Ux for qemu-devel@nongnu.org; Tue, 13 Jun 2017 15:07:17 +0100 From: Peter Maydell To: qemu-devel@nongnu.org Date: Tue, 13 Jun 2017 15:07:04 +0100 Message-Id: <1497362826-21125-16-git-send-email-peter.maydell@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1497362826-21125-1-git-send-email-peter.maydell@linaro.org> References: <1497362826-21125-1-git-send-email-peter.maydell@linaro.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:8b0:1d0::2 Subject: [Qemu-devel] [PULL 15/17] hw/intc/arm_gicv3_its: Implement state save/restore X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: Eric Auger We need to handle both registers and ITS tables. While register handling is standard, ITS table handling is more challenging since the kernel API is devised so that the tables are flushed into guest RAM and not in vmstate buffers. Flushing the ITS tables on device pre_save() is too late since the guest RAM is already saved at this point. Table flushing needs to happen when we are sure the vcpus are stopped and before the last dirty page saving. The right point is RUN_STATE_FINISH_MIGRATE but sometimes the VM gets stopped before migration launch so let's simply flush the tables each time the VM gets stopped. For regular ITS registers we just can use vmstate pre_save() and post_load() callbacks. Signed-off-by: Eric Auger Message-id: 1497023553-18411-3-git-send-email-eric.auger@redhat.com Reviewed-by: Peter Maydell Signed-off-by: Peter Maydell --- include/hw/intc/arm_gicv3_its_common.h | 8 +++ hw/intc/arm_gicv3_its_common.c | 10 ++++ hw/intc/arm_gicv3_its_kvm.c | 105 +++++++++++++++++++++++++++++++++ 3 files changed, 123 insertions(+) -- 2.7.4 diff --git a/include/hw/intc/arm_gicv3_its_common.h b/include/hw/intc/arm_gicv3_its_common.h index 1ba1894..fd1fe64 100644 --- a/include/hw/intc/arm_gicv3_its_common.h +++ b/include/hw/intc/arm_gicv3_its_common.h @@ -28,6 +28,13 @@ #define ITS_TRANS_SIZE 0x10000 #define ITS_SIZE (ITS_CONTROL_SIZE + ITS_TRANS_SIZE) +#define GITS_CTLR 0x0 +#define GITS_IIDR 0x4 +#define GITS_CBASER 0x80 +#define GITS_CWRITER 0x88 +#define GITS_CREADR 0x90 +#define GITS_BASER 0x100 + struct GICv3ITSState { SysBusDevice parent_obj; @@ -43,6 +50,7 @@ struct GICv3ITSState { /* Registers */ uint32_t ctlr; + uint32_t iidr; uint64_t cbaser; uint64_t cwriter; uint64_t creadr; diff --git a/hw/intc/arm_gicv3_its_common.c b/hw/intc/arm_gicv3_its_common.c index 9d67c5c..696c11c 100644 --- a/hw/intc/arm_gicv3_its_common.c +++ b/hw/intc/arm_gicv3_its_common.c @@ -49,6 +49,15 @@ static const VMStateDescription vmstate_its = { .pre_save = gicv3_its_pre_save, .post_load = gicv3_its_post_load, .unmigratable = true, + .fields = (VMStateField[]) { + VMSTATE_UINT32(ctlr, GICv3ITSState), + VMSTATE_UINT32(iidr, GICv3ITSState), + VMSTATE_UINT64(cbaser, GICv3ITSState), + VMSTATE_UINT64(cwriter, GICv3ITSState), + VMSTATE_UINT64(creadr, GICv3ITSState), + VMSTATE_UINT64_ARRAY(baser, GICv3ITSState, 8), + VMSTATE_END_OF_LIST() + }, }; static MemTxResult gicv3_its_trans_read(void *opaque, hwaddr offset, @@ -118,6 +127,7 @@ static void gicv3_its_common_reset(DeviceState *dev) s->cbaser = 0; s->cwriter = 0; s->creadr = 0; + s->iidr = 0; memset(&s->baser, 0, sizeof(s->baser)); gicv3_its_post_load(s, 0); diff --git a/hw/intc/arm_gicv3_its_kvm.c b/hw/intc/arm_gicv3_its_kvm.c index 340c2b0..4cd8f5f 100644 --- a/hw/intc/arm_gicv3_its_kvm.c +++ b/hw/intc/arm_gicv3_its_kvm.c @@ -53,6 +53,33 @@ static int kvm_its_send_msi(GICv3ITSState *s, uint32_t value, uint16_t devid) return kvm_vm_ioctl(kvm_state, KVM_SIGNAL_MSI, &msi); } +/** + * vm_change_state_handler - VM change state callback aiming at flushing + * ITS tables into guest RAM + * + * The tables get flushed to guest RAM whenever the VM gets stopped. + */ +static void vm_change_state_handler(void *opaque, int running, + RunState state) +{ + GICv3ITSState *s = (GICv3ITSState *)opaque; + Error *err = NULL; + int ret; + + if (running) { + return; + } + + ret = kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_ITS_SAVE_TABLES, NULL, true, &err); + if (err) { + error_report_err(err); + } + if (ret < 0 && ret != -EFAULT) { + abort(); + } +} + static void kvm_arm_its_realize(DeviceState *dev, Error **errp) { GICv3ITSState *s = ARM_GICV3_ITS_COMMON(dev); @@ -89,6 +116,8 @@ static void kvm_arm_its_realize(DeviceState *dev, Error **errp) kvm_msi_use_devid = true; kvm_gsi_direct_mapping = false; kvm_msi_via_irqfd_allowed = kvm_irqfds_enabled(); + + qemu_add_vm_change_state_handler(vm_change_state_handler, s); } static void kvm_arm_its_init(Object *obj) @@ -102,6 +131,80 @@ static void kvm_arm_its_init(Object *obj) &error_abort); } +/** + * kvm_arm_its_pre_save - handles the saving of ITS registers. + * ITS tables are flushed into guest RAM separately and earlier, + * through the VM change state handler, since at the moment pre_save() + * is called, the guest RAM has already been saved. + */ +static void kvm_arm_its_pre_save(GICv3ITSState *s) +{ + int i; + + for (i = 0; i < 8; i++) { + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_BASER + i * 8, &s->baser[i], false, + &error_abort); + } + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CTLR, &s->ctlr, false, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CBASER, &s->cbaser, false, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CREADR, &s->creadr, false, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CWRITER, &s->cwriter, false, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_IIDR, &s->iidr, false, &error_abort); +} + +/** + * kvm_arm_its_post_load - Restore both the ITS registers and tables + */ +static void kvm_arm_its_post_load(GICv3ITSState *s) +{ + int i; + + if (!s->iidr) { + return; + } + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_IIDR, &s->iidr, true, &error_abort); + + /* + * must be written before GITS_CREADR since GITS_CBASER write + * access resets GITS_CREADR. + */ + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CBASER, &s->cbaser, true, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CREADR, &s->creadr, true, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CWRITER, &s->cwriter, true, &error_abort); + + + for (i = 0; i < 8; i++) { + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_BASER + i * 8, &s->baser[i], true, + &error_abort); + } + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_ITS_RESTORE_TABLES, NULL, true, + &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CTLR, &s->ctlr, true, &error_abort); +} + static void kvm_arm_its_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); @@ -109,6 +212,8 @@ static void kvm_arm_its_class_init(ObjectClass *klass, void *data) dc->realize = kvm_arm_its_realize; icc->send_msi = kvm_its_send_msi; + icc->pre_save = kvm_arm_its_pre_save; + icc->post_load = kvm_arm_its_post_load; } static const TypeInfo kvm_arm_its_info = {