From patchwork Sat Jun 13 21:36:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 187898 Delivered-To: patch@linaro.org Received: by 2002:a92:cf06:0:0:0:0:0 with SMTP id c6csp1521538ilo; Sat, 13 Jun 2020 14:48:23 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzhy8sFYi5EDPnuWh5yKL/1OEeDiiuc2EGb9Em/i/sAyAdpFRdGd98hN/8grg7y9U7NBZeM X-Received: by 2002:a25:80c3:: with SMTP id c3mr36563300ybm.33.1592084903802; Sat, 13 Jun 2020 14:48:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1592084903; cv=none; d=google.com; s=arc-20160816; b=ye+S6arNbxleoQhhF5KwpYyCwOBeaN2C5tBNPpl4mOp4LeN2plbpFbLm0RjWnFwirv 9abOKXEqXJCVDgjhyK18A3pMBJqII8yRPwQmwmYUKi/ZbbPrjxSbKtiafXj4Fo5fCSQX Ceblzourx3AL6Un+xMNFhwcWb72iMDXV5PnenMzfs3auheG2nFLqg3o3LdynU9jqESeh Ob8jk4z6Ht/5imA+0YZjrdQiUWItgsSJySeGdz3jILV2iMy7mbTZviIdPZZZoqsNUOWl cshGmS1QKGMOvMYWmR4VGvWusENn3JwbrVLWz3r0dCWCxp3NaePSbXdDHWnKBBxNDWhK 7y4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mime-version:references :in-reply-to:message-id:date:subject:to:from; bh=PodGFge1WsRN0LLsPvFhAEc6XhRiRFjSuqskwKT/4d0=; b=P/TtN4oPG19HhrSRSjB+qAex0IV9ucWVxp65Hix8osi0e0pOomgqLYnDRjNmmOanN5 bRbYlCRCfBUprMl0TdQetDdlUhqQVF0hWx1KwZ3AoQ+F2caNCBpPBOAJDUjdSSZVygvs lPrvlLz6L5Fau3GW2lykIaMfw1FO6SOb2dh9L27sHMUeEclZTzrdslq7rQsJhoVFaC9R evj2gvMruzS3mYzBAUpxwhnOO1V1W1/mzlfX5sgZ9b5uU4aUnSQOjykX9p073QG7qxhZ hxoarvgHrz2oQJydfqA8CHw88HCtMZHCSJbussh/Qi9OUxf5XWdw01nD72kyxowaxLS0 R9xA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org" Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id p123si8575074ybg.360.2020.06.13.14.48.23 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sat, 13 Jun 2020 14:48:23 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org" Received: from localhost ([::1]:39476 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jkE0l-0002H6-7b for patch@linaro.org; Sat, 13 Jun 2020 17:48:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59356) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxV-0005jW-I3; Sat, 13 Jun 2020 17:45:01 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:43436 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jkDxS-000380-Eh; Sat, 13 Jun 2020 17:45:01 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 883C7821393FE162372F; Sun, 14 Jun 2020 05:44:55 +0800 (CST) Received: from A190218597.china.huawei.com (10.47.30.60) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.487.0; Sun, 14 Jun 2020 05:44:47 +0800 From: Salil Mehta To: , Subject: [PATCH RFC 06/22] arm/cpuhp: Changes to pre-size GIC with possible vcpus @machine init Date: Sat, 13 Jun 2020 22:36:13 +0100 Message-ID: <20200613213629.21984-7-salil.mehta@huawei.com> X-Mailer: git-send-email 2.8.3 In-Reply-To: <20200613213629.21984-1-salil.mehta@huawei.com> References: <20200613213629.21984-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.47.30.60] X-CFilter-Loop: Reflected Received-SPF: pass client-ip=45.249.212.32; envelope-from=salil.mehta@huawei.com; helo=huawei.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/13 17:44:20 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, drjones@redhat.com, sudeep.holla@arm.com, gshan@redhat.com, mst@redhat.com, jiakernel2@gmail.com, maz@kernel.org, zhukeqian1@huawei.com, david@redhat.com, richard.henderson@linaro.org, linuxarm@huawei.com, eric.auger@redhat.com, james.morse@arm.com, catalin.marinas@arm.com, imammedo@redhat.com, Salil Mehta , pbonzini@redhat.com, mehta.salil.lnk@gmail.com, maran.wilson@oracle.com, will@kernel.org, wangxiongfeng2@huawei.com Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" GIC needs to be pre-sized with possible vcpus at the initialization time. This is necessary because Memory regions and resources associated with GICC/GICR etc cannot be changed (add/del/modified) after VM has inited. Also, GIC_TYPER needs to be initialized with mp_affinity and cpu interface number association. This cannot be changed after GIC has initialized. Once all the cpu interfaces of the GIC has been inited it needs to be ensured that any updations to the GICC during reset only takes place for the present vcpus and not the disabled ones. Therefore, proper checks are required at various places. Co-developed-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 18 +++++++++--------- hw/intc/arm_gicv3_common.c | 8 ++++++-- hw/intc/arm_gicv3_cpuif.c | 6 ++++++ hw/intc/arm_gicv3_kvm.c | 29 ++++++++++++++++++++++++++--- include/hw/arm/virt.h | 2 +- 5 files changed, 48 insertions(+), 15 deletions(-) -- 2.17.1 diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 8040473d30..9e55b20685 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -628,19 +628,19 @@ static void create_v2m(VirtMachineState *vms) static void create_gic(VirtMachineState *vms) { - MachineState *ms = MACHINE(vms); /* We create a standalone GIC */ SysBusDevice *gicbusdev; const char *gictype; int type = vms->gic_version, i; - unsigned int smp_cpus = ms->smp.cpus; + unsigned int max_cpus = vms->max_cpus; + unsigned int smp_cpus = vms->smp_cpus; uint32_t nb_redist_regions = 0; gictype = (type == 3) ? gicv3_class_name() : gic_class_name(); vms->gic = qdev_create(NULL, gictype); qdev_prop_set_uint32(vms->gic, "revision", type); - qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus); + qdev_prop_set_uint32(vms->gic, "num-cpu", max_cpus); /* Note that the num-irq property counts both internal and external * interrupts; there are always 32 of the former (mandated by GIC spec). */ @@ -652,7 +652,7 @@ static void create_gic(VirtMachineState *vms) if (type == 3) { uint32_t redist0_capacity = vms->memmap[VIRT_GIC_REDIST].size / GICV3_REDIST_SIZE; - uint32_t redist0_count = MIN(smp_cpus, redist0_capacity); + uint32_t redist0_count = MIN(max_cpus, redist0_capacity); nb_redist_regions = virt_gicv3_redist_region_count(vms); @@ -665,7 +665,7 @@ static void create_gic(VirtMachineState *vms) vms->memmap[VIRT_HIGH_GIC_REDIST2].size / GICV3_REDIST_SIZE; qdev_prop_set_uint32(vms->gic, "redist-region-count[1]", - MIN(smp_cpus - redist0_count, redist1_capacity)); + MIN(max_cpus - redist0_count, redist1_capacity)); } } else { if (!kvm_irqchip_in_kernel()) { @@ -722,7 +722,7 @@ static void create_gic(VirtMachineState *vms) } else if (vms->virt) { qemu_irq irq = qdev_get_gpio_in(vms->gic, ppibase + ARCH_GIC_MAINT_IRQ); - sysbus_connect_irq(gicbusdev, i + 4 * smp_cpus, irq); + sysbus_connect_irq(gicbusdev, i + 4 * max_cpus, irq); } qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, @@ -730,11 +730,11 @@ static void create_gic(VirtMachineState *vms) + VIRTUAL_PMU_IRQ)); sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); - sysbus_connect_irq(gicbusdev, i + smp_cpus, + sysbus_connect_irq(gicbusdev, i + max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); - sysbus_connect_irq(gicbusdev, i + 2 * smp_cpus, + sysbus_connect_irq(gicbusdev, i + 2 * max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); - sysbus_connect_irq(gicbusdev, i + 3 * smp_cpus, + sysbus_connect_irq(gicbusdev, i + 3 * max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); } diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index 58ef65f589..bfa514444a 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -348,11 +348,15 @@ static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) s->cpu = g_new0(GICv3CPUState, s->num_cpu); for (i = 0; i < s->num_cpu; i++) { - CPUState *cpu = qemu_get_cpu(i); + CPUState *cpu = qemu_get_possible_cpu(i); uint64_t cpu_affid; int last; - s->cpu[i].cpu = cpu; + if (qemu_present_cpu(cpu)) + s->cpu[i].cpu = cpu; + else + s->cpu[i].cpu = NULL; + s->cpu[i].gic = s; /* Store GICv3CPUState in CPUARMState gicv3state pointer */ gicv3_set_gicv3state(cpu, &s->cpu[i]); diff --git a/hw/intc/arm_gicv3_cpuif.c b/hw/intc/arm_gicv3_cpuif.c index 08e000e33c..90d8b0118e 100644 --- a/hw/intc/arm_gicv3_cpuif.c +++ b/hw/intc/arm_gicv3_cpuif.c @@ -779,6 +779,9 @@ void gicv3_cpuif_update(GICv3CPUState *cs) ARMCPU *cpu = ARM_CPU(cs->cpu); CPUARMState *env = &cpu->env; + if (!qemu_present_cpu(cs->cpu)) + return; + g_assert(qemu_mutex_iothread_locked()); trace_gicv3_cpuif_update(gicv3_redist_affid(cs), cs->hppi.irq, @@ -1654,6 +1657,9 @@ static void icc_generate_sgi(CPUARMState *env, GICv3CPUState *cs, for (i = 0; i < s->num_cpu; i++) { GICv3CPUState *ocs = &s->cpu[i]; + if (!qemu_present_cpu(ocs->cpu)) + continue; + if (irm) { /* IRM == 1 : route to all CPUs except self */ if (cs == ocs) { diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index ca43bf87ca..7fe000e53c 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -25,6 +25,7 @@ #include "hw/sysbus.h" #include "qemu/error-report.h" #include "qemu/module.h" +#include "sysemu/cpus.h" #include "sysemu/kvm.h" #include "sysemu/runstate.h" #include "kvm_arm.h" @@ -458,6 +459,17 @@ static void kvm_arm_gicv3_put(GICv3State *s) GICv3CPUState *c = &s->cpu[ncpu]; int num_pri_bits; + /* + * To support hotplug of vcpus we need to make sure all gic cpuif/GICC + * are initialized at machvirt init time. Once the init is done we + * release the ARMCPU object for disabled vcpus but this leg could hit + * during reset of GICC later as well i.e. after init has happened and + * all of the cases we want to make sure we dont acess the GICC for + * the disabled VCPUs. + */ + if (!qemu_present_cpu(c->cpu)) + continue; + kvm_gicc_access(s, ICC_SRE_EL1, ncpu, &c->icc_sre_el1, true); kvm_gicc_access(s, ICC_CTLR_EL1, ncpu, &c->icc_ctlr_el1[GICV3_NS], true); @@ -677,10 +689,21 @@ static void arm_gicv3_icc_reset(CPUARMState *env, const ARMCPRegInfo *ri) return; } + /* + * This shall be called even when vcpu is being hotplugged and other vcpus + * might be running. Host kernel KVM code to handle device access of IOCTLs + * KVM_{GET|SET}_DEVICE_ATTR might fail due to inability to grab vcpu locks + * for all the vcpus. Hence, we need to pause all vcpus to facilitate + * locking within host. + */ + if (!qemu_present_cpu(c->cpu)) + pause_all_vcpus(); /* Initialize to actual HW supported configuration */ kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, KVM_VGIC_ATTR(ICC_CTLR_EL1, c->gicr_typer), &c->icc_ctlr_el1[GICV3_NS], false, &error_abort); + if (!qemu_present_cpu(c->cpu)) + resume_all_vcpus(); c->icc_ctlr_el1[GICV3_S] = c->icc_ctlr_el1[GICV3_NS]; } @@ -788,9 +811,9 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) } for (i = 0; i < s->num_cpu; i++) { - ARMCPU *cpu = ARM_CPU(qemu_get_cpu(i)); - - define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); + CPUState *cs = qemu_get_cpu(i); + if (qemu_present_cpu(cs)) + define_arm_cp_regs(ARM_CPU(cs), gicv3_cpuif_reginfo); } /* Try to create the device via the device control API */ diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 5b8ba64ec2..38a9cad168 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -178,7 +178,7 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms) assert(vms->gic_version == VIRT_GIC_VERSION_3); - return vms->smp_cpus > redist0_capacity ? 2 : 1; + return vms->max_cpus > redist0_capacity ? 2 : 1; } #endif /* QEMU_ARM_VIRT_H */