From patchwork Wed Apr 17 17:58:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 162415 Delivered-To: patch@linaro.org Received: by 2002:a02:c6d8:0:0:0:0:0 with SMTP id r24csp5795944jan; Wed, 17 Apr 2019 10:59:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqyb9Pr1c1jrnUd6YjcIeJUiQn+y4ZpKnm0x98tn4SwlOzc2jsHKsD2YRnkkBC0OaRgWHoJf X-Received: by 2002:a81:99cc:: with SMTP id q195mr71518036ywg.148.1555523993380; Wed, 17 Apr 2019 10:59:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555523993; cv=none; d=google.com; s=arc-20160816; b=06+PL7IIOsrp5N2Se251/F+WPWRtGo8pSbGmRDGFfBbrTfnLxvMQemdvUqBIfibYhS qBi93dQV+ZEs6DynA5tXrrIwxQBZoMjw4z08iwnJ96HW2ubXw0ArY9Ew1tqarbcKxDOm AhXpr2RZuv4wvfPKfWHf3qCFdWD2qjQBbZ44OOtkJurq0eCkpduR5hFjnFNMaEZwL2Ul FqQLSoxxmaIjZ+m3nf05mfwKlBbiBiTKmcYHP/MkcfXTTcSIbnTmjDdlLanxJAP3yxOj +l5U0gZDi4h9bXFWAbjF2BHdxIrXxYYHG+LFG2Unv1UufCuo8J6DGpADjsZw1iow1caH 50Ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc :list-subscribe:list-help:list-post:list-unsubscribe:list-id :precedence:subject:references:in-reply-to:message-id:date:to:from; bh=ABEPnZqZv6hBnoffSqPmJ8PQpuZlF/4zPnFJJNRnM94=; b=EojwA6/jgWeCavZLbpBG5aeAxSLen02ojl8kCbZfMCisd5CwgNNHFoFpz4areqUzxt 3u6cfORbXWIg17QfTQ61srNBXOGZxuDIInL8zZIa4iGiDl8ZpyrNYuzUkMoOV9bSwMZP UhGYLoWd9y53mrlqseQDtI70ZfMEXm6eDgpkaIQLsNoFODPu7CeUoDLL2msz9AU6NaWL enKzH0hj3zod7GKiMRzluUGV6mrdxqLekJBQixexVknT2GkIMFDs0udq9hu7oxSSQlE0 PbntJKoFgx5zeglrD8AQiRBNauOlcKhw9BAQusRVSxTPgUn9G3AfkC8m1kQj8C7n5yHm MdqA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of xen-devel-bounces@lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org. [192.237.175.120]) by mx.google.com with ESMTPS id k193si38289391ybk.9.2019.04.17.10.59.53 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 17 Apr 2019 10:59:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of xen-devel-bounces@lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of xen-devel-bounces@lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hGopL-0004DC-VT; Wed, 17 Apr 2019 17:58:31 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hGopK-0004CA-Kw for xen-devel@lists.xenproject.org; Wed, 17 Apr 2019 17:58:30 +0000 X-Inumbo-ID: 68525f5a-613a-11e9-845b-c37828ef12d8 Received: from foss.arm.com (unknown [217.140.101.70]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTP id 68525f5a-613a-11e9-845b-c37828ef12d8; Wed, 17 Apr 2019 17:58:29 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5DAEF15AB; Wed, 17 Apr 2019 10:58:29 -0700 (PDT) Received: from e108454-lin.cambridge.arm.com (e108454-lin.cambridge.arm.com [10.1.196.50]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4DCA43F59C; Wed, 17 Apr 2019 10:58:28 -0700 (PDT) From: Julien Grall To: xen-devel@lists.xenproject.org Date: Wed, 17 Apr 2019 18:58:14 +0100 Message-Id: <20190417175815.16905-7-julien.grall@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190417175815.16905-1-julien.grall@arm.com> References: <20190417175815.16905-1-julien.grall@arm.com> Subject: [Xen-devel] [PATCH 6/7] xen/arm: tlbflush: Rework TLB helpers X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Oleksandr_Tyshchenko@epam.com, Julien Grall , sstabellini@kernel.org, Andrii_Anisov@epam.com MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" All the TLBs helpers invalidate all the TLB entries are using the same pattern: DSB SY TLBI ... DSB SY ISB This pattern is following pattern recommended by the Arm Arm to ensure visibility of updates to translation tables (see K10-5610 in ARM DDI 0487.A.k). We have been a bit too eager in Xen and use system-wide DSBs when this can be limited to the inner-shareable domain. Furthermore, the first DSB can be restrict further to only store in the inner-shareable domain. This is because the DSB is here to ensure visibility of the update to translation table walks. Lastly, there are a lack of documentation in most of the TLBs helper. Rather than trying to update the helpers one by one, this patch introduce a per-arch macro to generate the TLB helpers. This will be easier to update the TLBs helper in the future and the documentation. Signed-off-by: Julien Grall Reviewed-by: Andrii Anisov --- xen/include/asm-arm/arm32/flushtlb.h | 73 ++++++++++++++-------------------- xen/include/asm-arm/arm64/flushtlb.h | 76 +++++++++++++++--------------------- 2 files changed, 60 insertions(+), 89 deletions(-) diff --git a/xen/include/asm-arm/arm32/flushtlb.h b/xen/include/asm-arm/arm32/flushtlb.h index b629db61cb..9085e65011 100644 --- a/xen/include/asm-arm/arm32/flushtlb.h +++ b/xen/include/asm-arm/arm32/flushtlb.h @@ -1,59 +1,44 @@ #ifndef __ASM_ARM_ARM32_FLUSHTLB_H__ #define __ASM_ARM_ARM32_FLUSHTLB_H__ -/* Flush local TLBs, current VMID only */ -static inline void flush_guest_tlb_local(void) -{ - dsb(sy); - - WRITE_CP32((uint32_t) 0, TLBIALL); - - dsb(sy); - isb(); +/* + * Every invalidation operation use the following patterns: + * + * DSB ISHST // Ensure prior page-tables updates have completed + * TLBI... // Invalidate the TLB + * DSB ISH // Ensure the TLB invalidation has completed + * ISB // See explanation below + * + * For Xen page-tables the ISB will discard any instructions fetched + * from the old mappings. + * + * For the Stage-2 page-tables the ISB ensures the completion of the DSB + * (and therefore the TLB invalidation) before continuing. So we know + * the TLBs cannot contain an entry for a mapping we may have removed. + */ +#define TLB_HELPER(name, tlbop) \ +static inline void name(void) \ +{ \ + dsb(ishst); \ + WRITE_CP32(0, tlbop); \ + dsb(ish); \ + isb(); \ } -/* Flush inner shareable TLBs, current VMID only */ -static inline void flush_guest_tlb(void) -{ - dsb(sy); - - WRITE_CP32((uint32_t) 0, TLBIALLIS); +/* Flush local TLBs, current VMID only */ +TLB_HELPER(flush_guest_tlb_local, TLBIALL); - dsb(sy); - isb(); -} +/* Flush inner shareable TLBs, current VMID only */ +TLB_HELPER(flush_guest_tlb, TLBIALLIS); /* Flush local TLBs, all VMIDs, non-hypervisor mode */ -static inline void flush_all_guests_tlb_local(void) -{ - dsb(sy); - - WRITE_CP32((uint32_t) 0, TLBIALLNSNH); - - dsb(sy); - isb(); -} +TLB_HELPER(flush_all_guests_tlb_local, TLBIALLNSNH); /* Flush innershareable TLBs, all VMIDs, non-hypervisor mode */ -static inline void flush_all_guests_tlb(void) -{ - dsb(sy); - - WRITE_CP32((uint32_t) 0, TLBIALLNSNHIS); - - dsb(sy); - isb(); -} +TLB_HELPER(flush_all_guests_tlb, TLBIALLNSNHIS); /* Flush all hypervisor mappings from the TLB of the local processor. */ -static inline void flush_xen_tlb_local(void) -{ - asm volatile("dsb;" /* Ensure preceding are visible */ - CMD_CP32(TLBIALLH) - "dsb;" /* Ensure completion of the TLB flush */ - "isb;" - : : : "memory"); -} +TLB_HELPER(flush_xen_tlb_local, TLBIALLH); /* Flush TLB of local processor for address va. */ static inline void __flush_xen_tlb_one_local(vaddr_t va) diff --git a/xen/include/asm-arm/arm64/flushtlb.h b/xen/include/asm-arm/arm64/flushtlb.h index 2fed34b2ec..ceec59542e 100644 --- a/xen/include/asm-arm/arm64/flushtlb.h +++ b/xen/include/asm-arm/arm64/flushtlb.h @@ -1,60 +1,46 @@ #ifndef __ASM_ARM_ARM64_FLUSHTLB_H__ #define __ASM_ARM_ARM64_FLUSHTLB_H__ -/* Flush local TLBs, current VMID only */ -static inline void flush_guest_tlb_local(void) -{ - asm volatile( - "dsb sy;" - "tlbi vmalls12e1;" - "dsb sy;" - "isb;" - : : : "memory"); +/* + * Every invalidation operation use the following patterns: + * + * DSB ISHST // Ensure prior page-tables updates have completed + * TLBI... // Invalidate the TLB + * DSB ISH // Ensure the TLB invalidation has completed + * ISB // See explanation below + * + * For Xen page-tables the ISB will discard any instructions fetched + * from the old mappings. + * + * For the Stage-2 page-tables the ISB ensures the completion of the DSB + * (and therefore the TLB invalidation) before continuing. So we know + * the TLBs cannot contain an entry for a mapping we may have removed. + */ +#define TLB_HELPER(name, tlbop) \ +static inline void name(void) \ +{ \ + asm volatile( \ + "dsb ishst;" \ + "tlbi " # tlbop ";" \ + "dsb ish;" \ + "isb;" \ + : : : "memory"); \ } +/* Flush local TLBs, current VMID only. */ +TLB_HELPER(flush_guest_tlb_local, vmalls12e1); + /* Flush innershareable TLBs, current VMID only */ -static inline void flush_guest_tlb(void) -{ - asm volatile( - "dsb sy;" - "tlbi vmalls12e1is;" - "dsb sy;" - "isb;" - : : : "memory"); -} +TLB_HELPER(flush_guest_tlb, vmalls12e1is); /* Flush local TLBs, all VMIDs, non-hypervisor mode */ -static inline void flush_all_guests_tlb_local(void) -{ - asm volatile( - "dsb sy;" - "tlbi alle1;" - "dsb sy;" - "isb;" - : : : "memory"); -} +TLB_HELPER(flush_all_guests_tlb_local, alle1); /* Flush innershareable TLBs, all VMIDs, non-hypervisor mode */ -static inline void flush_all_guests_tlb(void) -{ - asm volatile( - "dsb sy;" - "tlbi alle1is;" - "dsb sy;" - "isb;" - : : : "memory"); -} +TLB_HELPER(flush_all_guests_tlb, alle1is); /* Flush all hypervisor mappings from the TLB of the local processor. */ -static inline void flush_xen_tlb_local(void) -{ - asm volatile ( - "dsb sy;" /* Ensure visibility of PTE writes */ - "tlbi alle2;" /* Flush hypervisor TLB */ - "dsb sy;" /* Ensure completion of TLB flush */ - "isb;" - : : : "memory"); -} +TLB_HELPER(flush_xen_tlb_local, alle2); /* Flush TLB of local processor for address va. */ static inline void __flush_xen_tlb_one_local(vaddr_t va)