From patchwork Wed Apr 27 11:22:53 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 66780 Delivered-To: patch@linaro.org Received: by 10.140.93.198 with SMTP id d64csp2127132qge; Wed, 27 Apr 2016 04:23:36 -0700 (PDT) X-Received: by 10.176.68.162 with SMTP id n31mr4280605uan.26.1461756216389; Wed, 27 Apr 2016 04:23:36 -0700 (PDT) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org. [192.237.175.120]) by mx.google.com with ESMTPS id 105si1984308uat.70.2016.04.27.04.23.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Apr 2016 04:23:36 -0700 (PDT) Received-SPF: neutral (google.com: 192.237.175.120 is neither permitted nor denied by best guess record for domain of xen-devel-bounces@lists.xen.org) client-ip=192.237.175.120; Authentication-Results: mx.google.com; spf=neutral (google.com: 192.237.175.120 is neither permitted nor denied by best guess record for domain of xen-devel-bounces@lists.xen.org) smtp.mailfrom=xen-devel-bounces@lists.xen.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1avNYd-0006RM-JH; Wed, 27 Apr 2016 11:23:03 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1avNYc-0006RA-O2 for xen-devel@lists.xen.org; Wed, 27 Apr 2016 11:23:02 +0000 Received: from [85.158.139.211] by server-15.bemta-5.messagelabs.com id A2/B1-04147-611A0275; Wed, 27 Apr 2016 11:23:02 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrHLMWRWlGSWpSXmKPExsVysyfVTVd0oUK 4waENXBZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8bc1f9ZC1ZKVmxb0cDUwHhIpIuRi0NIYCOj xNxp71khnNOMEv/+TgRyODnYBDQl7nz+xARiiwhIS1z7fJkRpIhZoI1Rort/JTNIQlggReLph FNADRwcLAKqElMehYKEeQVcJO4+WMgCYksIyEmcPDaZdQIj5wJGhlWM6sWpRWWpRbpmeklFme kZJbmJmTm6hgamermpxcWJ6ak5iUnFesn5uZsYgf5iAIIdjFMbnA8xSnIwKYnyLmxXCBfiS8p PqcxILM6ILyrNSS0+xCjDwaEkwTtnHlBOsCg1PbUiLTMHGDgwaQkOHiUR3niQNG9xQWJucWY6 ROoUo6KUOG8oSEIAJJFRmgfXBgvWS4yyUsK8jECHCPEUpBblZpagyr9iFOdgVBLmzQaZwpOZV wI3/RXQYiagxZcPyYIsLklESEk1MLqpHlh9MU5xmkJ6/0flCff3FE5IX3aq/0abAcc2o1UpBT Fmt7286y5fm/562hubmpYt5tWX9rvFfmA8/EZidwNP556EpnM3cp059qV8u7N7Vs6Ox6umzXO blc1ZZTNl8x6tDZ3X9GeFt5lMd0rcuf1kQuX+UmMR/msPJ/vf+2HQWKmu4So9JVKJpTgj0VCL uag4EQD2N/OHUQIAAA== X-Env-Sender: julien.grall@arm.com X-Msg-Ref: server-5.tower-206.messagelabs.com!1461756180!36701256!1 X-Originating-IP: [217.140.101.70] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 8.34; banners=-,-,- X-VirusChecked: Checked Received: (qmail 37757 invoked from network); 27 Apr 2016 11:23:00 -0000 Received: from foss.arm.com (HELO foss.arm.com) (217.140.101.70) by server-5.tower-206.messagelabs.com with SMTP; 27 Apr 2016 11:23:00 -0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7ECE43A; Wed, 27 Apr 2016 04:23:00 -0700 (PDT) Received: from e108454-lin.cambridge.arm.com (e108454-lin.cambridge.arm.com [10.1.215.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DFC163F42B; Wed, 27 Apr 2016 04:22:58 -0700 (PDT) From: Julien Grall To: xen-devel@lists.xen.org Date: Wed, 27 Apr 2016 12:22:53 +0100 Message-Id: <1461756173-10300-1-git-send-email-julien.grall@arm.com> X-Mailer: git-send-email 1.9.1 Cc: andre.przywara@arm.com, Julien Grall , sstabellini@kernel.org, wei.liu2@citrix.com, steve.capper@arm.com Subject: [Xen-devel] [for-4.7 v2] xen/arm: Force broadcast of TLB and instruction cache maintenance instructions X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" UP guest may use TLB instructions to flush only on the local CPU. Therefore, TLB flush will not be broadcasted across all the CPUs within the same innershareable domain. When the vCPU is migrated between different CPUs, it may be rescheduled to a previous CPU where the TLB has not been flushed. The TLB may contain stale entries which will result to translate incorrectly a VA to IPA or even cause TLB conflicts. To avoid a such situation, it is possible to set HCR_EL2.FB, which will force the broadcast of TLB and instruction cache maintenance instructions. The performance impact of setting HCR_EL2.FB will depend on how often a guest makes use of local flush instructions. ARM64 Linux kernel is SMP-aware (no possibility to build only for UP). Most of the flush instructions are innershareable. The local flushes are limited to the boot (1 per CPU) and when a task is getting a new ASIC. Therefore the impact of setting HCR.FB for those guests is very limited. ARM32 Linux kernel offers the possibility to be built either for SMP or UP. The number of local flush is very limited in the former kernel whilst the latter will only issue local flushes. Therefore there will be an impact to set HCR.FB for guest kernel only built for UP. Note that the SMP kernel can run in a domain using 1 vCPU and it will still make use of innershareable flush instruction. Looking at other OSes, such as FreeBSD, they are very similar to ARM32 Linux kernel (i.e offering two configuration: SMP and UP). However, nothing prevents an SMP-aware kernel to make more often use of local flush instrutions. In the case that HCR_EL2.FB is not set, Xen would need to: * Flush all the TLBs for the VMID associated to this domain * Invalidate all the entries from the branch predictor * Invalidate all the entries from the instruction cache Those actions would only be needed when the vCPU is migrating between 2 physical CPUs. Whilst this solution would have a negative performance impact on kernels which do not heavily use local flush instructions, this may improve performance for kernels only built for UP system. For now implement the easiest solution (i.e setting HCR_EL2.FB). We can revisit it if the performance impact is too high for UP kernel. Signed-off-by: Julien Grall --- This is a bug fix for Xen 4.7 and should be backported up to Xen 4.4 (first official release for ARM). Without this patch, UP guest will crash if it gets migrated on a physical CPU with stale TLBs for this guest. Changes in v2: - Rework the commit message to include the possible performance impact of setting HCR_EL2.FB. --- xen/arch/arm/traps.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c index 5e865cf..9926a57 100644 --- a/xen/arch/arm/traps.c +++ b/xen/arch/arm/traps.c @@ -124,7 +124,8 @@ void init_traps(void) /* Setup hypervisor traps */ WRITE_SYSREG(HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM| - HCR_TWE|HCR_TWI|HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP, HCR_EL2); + HCR_TWE|HCR_TWI|HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB, + HCR_EL2); isb(); }