From patchwork Fri Aug 24 15:52:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145083 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1411193ljw; Fri, 24 Aug 2018 08:53:38 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb1xwCxcZVk6yYedrKzdTslYfKRm/JnCGVwl72JFLaXNwd+1WzyfwpPKnXQgcJA377bS8Vp X-Received: by 2002:a62:9042:: with SMTP id a63-v6mr2622426pfe.52.1535126018604; Fri, 24 Aug 2018 08:53:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535126018; cv=none; d=google.com; s=arc-20160816; b=Y/sDuQ90SM6nJa4d+5hJYe/2N1IvXZXpOmw9Wpw57KR6PzUdrn8kbtkj+8qVEZvhYR v5v2pI6mRHD74p0U15mq3sThQftEAb+jtCN43kDPkVfuy4BtguxYOKsXI5ghY+erclMY 9FuhE6zONcLh1Y/39r4Axdsric6nQ9o1xnd0XhgSBPuvv21YvZSkllHlLjE3sm35vNFV j0UJGZ+AxVkpUbDBOGYBKn6iNqb7dGu479B+/P6XjVDEAXzGzo0KwX6dCaTXz4MX5cc+ Ly8r24t0MqmLmZysI0rE3BbU6x3L9YYmHBIeNVJJnP0MyYtbUfhS91wQNyBpM68Z2O78 d90g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Mc6pOOsmGvtQt/y+Ju/BH/GnTOxKCskvlW5ST031YVE=; b=O6vUvsZTXHsoDXO6VFm/SN23hKsvNVPiKr0Ch4g4u64/NSkByEREb9zJgxFR2TjHnW x2LRfH2D07MBakU+tgvoEwg9r3/Glj5MrPaklTSjWyxYhe5giXSvAIvMGcQuFAUTc11e Jm6qrdbbn84N1cfsEZQqgy5+Gn3fr4RdllgiWgobOKeKzZ19AB7ebW2cZCXKisUFMTZ0 nRRA5SZNbFIqDR7r6rqW8YezO9ClG/AiG+U0vKPVDj0ObreSn2ZnvY29OioDyaVYVfyA aFfEL17ioSZ1sxoqr4kj4bf4ThtNQmLqytACMuiYrzmX7o0eYsDHHgyp/LCic2JUwuwL DfUw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g34-v6si7159766pld.244.2018.08.24.08.53.38; Fri, 24 Aug 2018 08:53:38 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727435AbeHXT1v (ORCPT + 32 others); Fri, 24 Aug 2018 15:27:51 -0400 Received: from foss.arm.com ([217.140.101.70]:32806 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726279AbeHXT1v (ORCPT ); Fri, 24 Aug 2018 15:27:51 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E082F15A2; Fri, 24 Aug 2018 08:52:37 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B233F3F763; Fri, 24 Aug 2018 08:52:37 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id A76191AE321F; Fri, 24 Aug 2018 16:52:47 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 01/11] arm64: tlb: Use last-level invalidation in flush_tlb_kernel_range() Date: Fri, 24 Aug 2018 16:52:36 +0100 Message-Id: <1535125966-7666-2-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org flush_tlb_kernel_range() is only ever used to invalidate last-level entries, so we can restrict the scope of the TLB invalidation instruction. Signed-off-by: Will Deacon --- arch/arm64/include/asm/tlbflush.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index a4a1901140ee..7e2a35424ca4 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -199,7 +199,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end dsb(ishst); for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) - __tlbi(vaae1is, addr); + __tlbi(vaale1is, addr); dsb(ish); isb(); } From patchwork Fri Aug 24 15:52:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145073 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1410357ljw; Fri, 24 Aug 2018 08:52:40 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaSrhsUdGhKhqnMggFNsQm4L3aqymJ3t/4tx3bKoJD7zGbpqBLaR04SqNm7rFL5lBQi0EJe X-Received: by 2002:a63:c00b:: with SMTP id h11-v6mr2236528pgg.279.1535125960197; Fri, 24 Aug 2018 08:52:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535125960; cv=none; d=google.com; s=arc-20160816; b=VQXQBp+f0jBeAbUqTxdm6+o9UDuqkkx8lzAKw3xzGeHydjdKuoJiBrYBrkVT9tgA8H cPtBwLjZOmonrdRUCX3E8+svlMal+W0q5nxhaYP+kHLLaD2UYHQl72GoFmeKDN5RKQ0P BoynWnQWJWA4g7mbnuZHfpQw3szNBiAdegj67yOiqy11VE4/1/I5oS8S9x+yIlqIrUUO pXwvVOYWUPLtkUn8eB51ZdXrsw2ybFq/F2fVfGJzGqgpwgJfH9fqgqjEoDlTL0a4asXQ EMdxtHc4AHzg2g3NsC43BkYLuGmIwQFXWL1bFb9tJpmnona1AcYrEF6A1wNP9atqQOVr IXWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=RozllPryD3GkPRtMWEWjoDAYyDfgAVoUGTzvkBtaSfE=; b=NQDty45btUc8vKbxtJvd+/NIDTB/UuApUv5OQTb8G/Vu72rF8JRhk1rBxPOATayrNt tszkGYQnYxzH7LmnZKctMLGQcdXyLYzPFq/XxdfyrqoHlAtHjMP4FYvfsNrxgKLvuGDK aTxusPFhfLivTp1aytUj1RbLw/kqjRh9F9I6FjIHoORc6rRd/Qu2sSzS8ZllfiH+muc1 ncp4/HNxf666TqjbXzu3t8op/vJ0NZ9SPYnTZiCtfas0Su8BhRaO2RQlEQNbm5k8SnuQ MQGlJlPPcAU4uoN8IDDN+jxSQdATkbKOGoPhH9tlIMYvKheV0rycHUylVLD1qvnhLeSB EdmQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m10-v6si7021181pgv.374.2018.08.24.08.52.39; Fri, 24 Aug 2018 08:52:40 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727557AbeHXT1v (ORCPT + 32 others); Fri, 24 Aug 2018 15:27:51 -0400 Received: from foss.arm.com ([217.140.101.70]:32816 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726374AbeHXT1v (ORCPT ); Fri, 24 Aug 2018 15:27:51 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F09FC1650; Fri, 24 Aug 2018 08:52:37 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C25043F7BE; Fri, 24 Aug 2018 08:52:37 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id B7AB01AE3265; Fri, 24 Aug 2018 16:52:47 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 02/11] arm64: tlb: Add DSB ISHST prior to TLBI in __flush_tlb_[kernel_]pgtable() Date: Fri, 24 Aug 2018 16:52:37 +0100 Message-Id: <1535125966-7666-3-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org __flush_tlb_[kernel_]pgtable() rely on set_pXd() having a DSB after writing the new table entry and therefore avoid the barrier prior to the TLBI instruction. In preparation for delaying our walk-cache invalidation on the unmap() path, move the DSB into the TLB invalidation routines. Signed-off-by: Will Deacon --- arch/arm64/include/asm/tlbflush.h | 2 ++ 1 file changed, 2 insertions(+) -- 2.1.4 diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 7e2a35424ca4..e257f8655b84 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -213,6 +213,7 @@ static inline void __flush_tlb_pgtable(struct mm_struct *mm, { unsigned long addr = __TLBI_VADDR(uaddr, ASID(mm)); + dsb(ishst); __tlbi(vae1is, addr); __tlbi_user(vae1is, addr); dsb(ish); @@ -222,6 +223,7 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr) { unsigned long addr = __TLBI_VADDR(kaddr, 0); + dsb(ishst); __tlbi(vaae1is, addr); dsb(ish); } From patchwork Fri Aug 24 15:52:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145081 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1411031ljw; Fri, 24 Aug 2018 08:53:28 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb6AKq0S4NS7HBJFSOQqzZ+i6cagGnrR6ILz5JPdqaYNvh+6hLanQNq0X9/LlZGCtwNO8cl X-Received: by 2002:aa7:800f:: with SMTP id j15-v6mr2621322pfi.174.1535126008094; Fri, 24 Aug 2018 08:53:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535126008; cv=none; d=google.com; s=arc-20160816; b=Jx3IBXNs7rAAYcjtmG1+nKzvxsna/RAfSxRqPp0MLB9zU9jAgW08R8EnB3198F67Yz edyxiBVqQOUA41ZUTYWSU+6TsD2WMxcRhqLMS698S3LiSdIwbo+XzWLW3PkRoEKfL1ix /DhOmRW5PXUgRApif//YDGnF5ceqMXr0QfBYsDp+27IME6sPw80gJSnRqeo4ZWScduEe VSqQyQlLmC7YEM+HAS8RHDrsJgW2pyR3+3iqsU9U0KQb1OS3IwUNdJ/txCl/dztsQ6ok UqsJSfyF3u9u64PliYLzXs0m/f3BknyXALpsnnaC/2bvPI025sLLW1FAcWVAWWFDywRX sYPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=mgLzZPBJn+RP+iAVaiUfifPZNPKh2V3/3J8/kCSjBtc=; b=Ud/sMUSdVlOg3rsITXEnO5Un8MQrwVyT1ApQPoJCPX9QGdheQScEFsCwTcSA1ytZra RSGySBMSm7GV4mXbYmLXCgFJaAETl1bKOdQAvOVlUKYu5z5RdIFFhnPc4GeJcVn0z4wV SG+QzGE95yK0na0suVnJg7/I6UEWCI+4BYAXWK9yRj8cgfdFbUXvCj+H21I09LDZ0qH2 KRjEKqyND4dDm3jsqleqbwexPRyae9VoiCfgAxE3LXtyFAr7ikQ3B5+hw5MRuOS+u03A PuyWS0ujGyY6gaSfbWil0bWT57FrKMokNHvkkrOCSPH+KHZJP3odoFRTE856GSyeU95l Riig== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g10-v6si6762177plt.468.2018.08.24.08.53.27; Fri, 24 Aug 2018 08:53:28 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728150AbeHXT2i (ORCPT + 32 others); Fri, 24 Aug 2018 15:28:38 -0400 Received: from foss.arm.com ([217.140.101.70]:32826 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726871AbeHXT1v (ORCPT ); Fri, 24 Aug 2018 15:27:51 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0A8B41682; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D01ED3F7F3; Fri, 24 Aug 2018 08:52:37 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id C7F1D1AE3307; Fri, 24 Aug 2018 16:52:47 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 03/11] arm64: pgtable: Implement p[mu]d_valid() and check in set_p[mu]d() Date: Fri, 24 Aug 2018 16:52:38 +0100 Message-Id: <1535125966-7666-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Now that our walk-cache invalidation routines imply a DSB before the invalidation, we no longer need one when we are clearing an entry during unmap. Signed-off-by: Will Deacon --- arch/arm64/include/asm/pgtable.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1bdeca8918a6..2ab2031b778c 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -360,6 +360,7 @@ static inline int pmd_protnone(pmd_t pmd) #define pmd_present(pmd) pte_present(pmd_pte(pmd)) #define pmd_dirty(pmd) pte_dirty(pmd_pte(pmd)) #define pmd_young(pmd) pte_young(pmd_pte(pmd)) +#define pmd_valid(pmd) pte_valid(pmd_pte(pmd)) #define pmd_wrprotect(pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd))) #define pmd_mkold(pmd) pte_pmd(pte_mkold(pmd_pte(pmd))) #define pmd_mkwrite(pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd))) @@ -431,7 +432,9 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) { WRITE_ONCE(*pmdp, pmd); - dsb(ishst); + + if (pmd_valid(pmd)) + dsb(ishst); } static inline void pmd_clear(pmd_t *pmdp) @@ -477,11 +480,14 @@ static inline phys_addr_t pmd_page_paddr(pmd_t pmd) #define pud_none(pud) (!pud_val(pud)) #define pud_bad(pud) (!(pud_val(pud) & PUD_TABLE_BIT)) #define pud_present(pud) pte_present(pud_pte(pud)) +#define pud_valid(pud) pte_valid(pud_pte(pud)) static inline void set_pud(pud_t *pudp, pud_t pud) { WRITE_ONCE(*pudp, pud); - dsb(ishst); + + if (pud_valid(pud)) + dsb(ishst); } static inline void pud_clear(pud_t *pudp) From patchwork Fri Aug 24 15:52:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145084 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1411235ljw; Fri, 24 Aug 2018 08:53:41 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaC855vARgwJPgNmm3IndlR5TYz/QAHk06lgcxyAbbdqZEK9jRaqLNC+TnCvt2raIReU1vr X-Received: by 2002:a62:4898:: with SMTP id q24-v6mr2651217pfi.22.1535126021009; Fri, 24 Aug 2018 08:53:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535126021; cv=none; d=google.com; s=arc-20160816; b=BanaIq7u/zdfqhPwZnW4rgBFIv43goYsROHCvLbwz3J/kUtGNAnS8UPHGUF4d3Ig0p KWoHdBcrnwt/KGnsenU0l0pt8dhO5eyne3/CXIVQ7l6nBEw/ORDpCcnAhiBKs6Tt6QY0 bBAmXdU4vrMCU5i/nIURLrTqDGTSto4c2Ss1cS18tGyxqr2jYbPd1mWDUniJiF9u60xW nKcw6B7n/sJaxe1+hxlqCUcfwb2Xrkb6e7qS8x15NiCFNS2cxtHzSUPCEI0HlQZw+i6M s2R3GufqMq4idHfMKoWdjHxYObGtcKRWQ1Nys079f48XOP/3lnyiu8pt/ybCxm2lB87n dzCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=O/F+Wwa3qzWoWLG2ynBUoJAdpqUksovup4kAYC73rqk=; b=ikSoBBQvX1QM+emmd27XwjDSDSmrIF5yqCayDuXRcu5oRRYOzUBtf+Yd/jEmdI80VO 9+suT4KNSPD39Nfm73AHOOnj1Q7v3eDPwl2/VCqOvvw797o3me4q09h9mfJ+aCUeuUyR Hzq1T+1KMb74xvyn4StpXyC5IXtE9wnk1o595CipNBtLw4hL0X53YF7bY4zLuMY3ApLc 8S4lOSGLom4nXyXAd/hLrxI5RVsga9lN3s9IHgplmx3uAgehUEk/iSzRtwuYPORxrCV1 1Bhv+BtSc+mh4/81JhcG8xN11Ai4BCRvHqvXN7JSMTtbSD+Ebn19bHL281YJayRrBIds Ip8w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g34-v6si7159766pld.244.2018.08.24.08.53.40; Fri, 24 Aug 2018 08:53:41 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728183AbeHXT2w (ORCPT + 32 others); Fri, 24 Aug 2018 15:28:52 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:32846 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726896AbeHXT1v (ORCPT ); Fri, 24 Aug 2018 15:27:51 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1B48D1688; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E14263F5BC; Fri, 24 Aug 2018 08:52:37 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id D63031AE3318; Fri, 24 Aug 2018 16:52:47 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 04/11] arm64: tlb: Justify non-leaf invalidation in flush_tlb_range() Date: Fri, 24 Aug 2018 16:52:39 +0100 Message-Id: <1535125966-7666-5-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add a comment to explain why we can't get away with last-level invalidation in flush_tlb_range() Signed-off-by: Will Deacon --- arch/arm64/include/asm/tlbflush.h | 4 ++++ 1 file changed, 4 insertions(+) -- 2.1.4 diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index e257f8655b84..ddbf1718669d 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -182,6 +182,10 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, static inline void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { + /* + * We cannot use leaf-only invalidation here, since we may be invalidating + * table entries as part of collapsing hugepages or moving page tables. + */ __flush_tlb_range(vma, start, end, false); } From patchwork Fri Aug 24 15:52:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145074 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1410386ljw; Fri, 24 Aug 2018 08:52:42 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaXUFSxFK7cKy8jsbAlT++GYpLboeyOVzRkhfxdyE5IAuzuxNxp/hmoNG2i2vJqGR0jHrYN X-Received: by 2002:a17:902:7246:: with SMTP id c6-v6mr2274750pll.38.1535125962639; Fri, 24 Aug 2018 08:52:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535125962; cv=none; d=google.com; s=arc-20160816; b=QZi4Yekzx8dQ3tNHorzlhTRNBrGXrK+yDfAku5ExBZKSeudoVmXoVCzFl61wF5Mzg3 edJgnXiS+HjtD0o6XbHU+2SNBqzOcIS8ze7JQsE9k5+nje8r0mL2gLEQ4zmg8HBqWXQx miYZh1i23qjOK4PuX6Z58bd1ySSBBsUFdDp4C5SGrKII1gex0DG32QzhjvihA2uvKcG9 DprSnLQkTjz+GFoA2JsjGK8bWx89qoBaly7Y2qNIFgcLerps87HyoJHsEsjds9HFgC6S QOZiJzVH1J7xOXEVkXZRjGKfvX7x+9If61JWQl/PN3iz4zo2nsCwdDQp32ipLendV+pF rOUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=WdmmoQ8xGgf5wnEtraTFiDTfm47hxGszZAHH8LfjiK8=; b=MIOfiEVLoWlFreTbnGsEPBB9JgyDm/Q2irn5RSRltrth6krXnfW15PvblA6cU+DgdM OgokZdB4pCS02OQZOB6vTiBghndPUp2VUVoCgWOsWkuTZXAFcxEXLMfULBzpCvJj1Oda zeWtBkx63wcumPkjfqP9qYE4FrCzAySeUA9iK2K736DvtcOuL5vtsnByJshrFYVG8GKv frZB37VxOVqCNe71oavDRNbOBfwWbFFfsfgev/Sf09Lsumwxrbqoqk8H1kVF/OW/UtJn 9WGrreyz2d4BLXhUt15N4UEIcauIy6P08UGlfvK0VDMw9BXhvfy62cR8Lbpc4QNxeYte 9MBA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p63-v6si5829157pgp.651.2018.08.24.08.52.42; Fri, 24 Aug 2018 08:52:42 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727712AbeHXT1w (ORCPT + 32 others); Fri, 24 Aug 2018 15:27:52 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:32858 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726374AbeHXT1v (ORCPT ); Fri, 24 Aug 2018 15:27:51 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BC497168F; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8DC693F5BC; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id E75951AE3326; Fri, 24 Aug 2018 16:52:47 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 05/11] arm64: tlbflush: Allow stride to be specified for __flush_tlb_range() Date: Fri, 24 Aug 2018 16:52:40 +0100 Message-Id: <1535125966-7666-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When we are unmapping intermediate page-table entries or huge pages, we don't need to issue a TLBI instruction for every PAGE_SIZE chunk in the VA range being unmapped. Allow the invalidation stride to be passed to __flush_tlb_range(), and adjust our "just nuke the ASID" heuristic to take this into account. Signed-off-by: Will Deacon --- arch/arm64/include/asm/tlb.h | 2 +- arch/arm64/include/asm/tlbflush.h | 11 +++++++---- 2 files changed, 8 insertions(+), 5 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a3233167be60..1e1f68ce28f4 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -53,7 +53,7 @@ static inline void tlb_flush(struct mmu_gather *tlb) * the __(pte|pmd|pud)_free_tlb() functions, so last level * TLBI is sufficient here. */ - __flush_tlb_range(&vma, tlb->start, tlb->end, true); + __flush_tlb_range(&vma, tlb->start, tlb->end, PAGE_SIZE, true); } static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index ddbf1718669d..1f77d08e638b 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -153,12 +153,15 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, - bool last_level) + unsigned long stride, bool last_level) { unsigned long asid = ASID(vma->vm_mm); unsigned long addr; - if ((end - start) > MAX_TLB_RANGE) { + /* Convert the stride into units of 4k */ + stride >>= 12; + + if ((end - start) > (MAX_TLB_RANGE * stride)) { flush_tlb_mm(vma->vm_mm); return; } @@ -167,7 +170,7 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, end = __TLBI_VADDR(end, asid); dsb(ishst); - for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) { + for (addr = start; addr < end; addr += stride) { if (last_level) { __tlbi(vale1is, addr); __tlbi_user(vale1is, addr); @@ -186,7 +189,7 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, * We cannot use leaf-only invalidation here, since we may be invalidating * table entries as part of collapsing hugepages or moving page tables. */ - __flush_tlb_range(vma, start, end, false); + __flush_tlb_range(vma, start, end, PAGE_SIZE, false); } static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end) From patchwork Fri Aug 24 15:52:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145080 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1410915ljw; Fri, 24 Aug 2018 08:53:19 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaMH+/TJ2L3N8MGeBcHYx8eiK6u3fYdw2jpuMH/8A8ksZ7zELdecU4QMlPoJzlr2EY9Boog X-Received: by 2002:a62:49d5:: with SMTP id r82-v6mr2564341pfi.245.1535125999660; Fri, 24 Aug 2018 08:53:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535125999; cv=none; d=google.com; s=arc-20160816; b=KWGi8YDjk3e8g87DjNBSRqfH3es62QKCKvKvlgFkznnvhj/hzzXlF7ecpbc0moTtiS tG+8VwAaihuFw9q+j5CGPZ7KjGNfObU6+Yw5AFQV8iFhPizlKHSD5ZSL4n5/ZZLq3jFL LyHl+x1wJBgsLvunokf2kEBLZ62hqx2QYO98ZHnpKsHke+M64mGjjLmvWhankIEZyhq9 qXRM2SM3V0vSZaJjK9CIiJv42557Jf9ZAvTWABYiodFTVIxu319B2QLozvpYYrt7IS9E OcfJDwrDP0tWV4jytUGr2mXhDwum9geYwIJruvqmzfmWr7tMGGSnTQcbMc3dG554EXIf 1uaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=7vykjle0yOPkLV1is8B9sNa/1dZFWFfaIAu1AcfOKwU=; b=EVt4mt5Q2xWFkRY1UWtd/l6T26S1x2NaUBF4XPb18sbcbDSlM512N9W7NsBiHYop+6 CSeQYcxBinmpKnbnkOBvB01+JYnpZ4ym7GK6xA65KJGhLuEj1q16hcN2+EM9y0jsTmgK j3L3orr9lVx3X8rVgFSD47H2StSNGUBynn11Mp2by8I7YVrOiyPQYUtZvFxU5nfYQ5+l olN3vAV9QtpyCnJo3M7bRw0cGDVzL29weS/JI+4h0gMRMj5RkPZUjTVNgMslAYjggIk7 vqnERXg75I4kCQSqzBt6oG0oOsSiQHA9mWNWtPmN2rPt3vF+ruzEVgHu1EIq7BSkyCgM d+6w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h2-v6si6663834plh.145.2018.08.24.08.53.19; Fri, 24 Aug 2018 08:53:19 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728135AbeHXT2b (ORCPT + 32 others); Fri, 24 Aug 2018 15:28:31 -0400 Received: from foss.arm.com ([217.140.101.70]:32866 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727565AbeHXT1w (ORCPT ); Fri, 24 Aug 2018 15:27:52 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C590E1713; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 96D163F763; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 0B9D51AE3333; Fri, 24 Aug 2018 16:52:47 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 06/11] arm64: tlb: Remove redundant !CONFIG_HAVE_RCU_TABLE_FREE code Date: Fri, 24 Aug 2018 16:52:41 +0100 Message-Id: <1535125966-7666-7-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org If there's one thing the RCU-based table freeing doesn't need, it's more ifdeffery. Remove the redundant !CONFIG_HAVE_RCU_TABLE_FREE code, since this option is unconditionally selected in our Kconfig. Signed-off-by: Will Deacon --- arch/arm64/include/asm/tlb.h | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 1e1f68ce28f4..bd00017d529a 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -22,16 +22,10 @@ #include #include -#ifdef CONFIG_HAVE_RCU_TABLE_FREE - -#define tlb_remove_entry(tlb, entry) tlb_remove_table(tlb, entry) static inline void __tlb_remove_table(void *_table) { free_page_and_swap_cache((struct page *)_table); } -#else -#define tlb_remove_entry(tlb, entry) tlb_remove_page(tlb, entry) -#endif /* CONFIG_HAVE_RCU_TABLE_FREE */ static void tlb_flush(struct mmu_gather *tlb); @@ -61,7 +55,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { __flush_tlb_pgtable(tlb->mm, addr); pgtable_page_dtor(pte); - tlb_remove_entry(tlb, pte); + tlb_remove_table(tlb, pte); } #if CONFIG_PGTABLE_LEVELS > 2 @@ -69,7 +63,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { __flush_tlb_pgtable(tlb->mm, addr); - tlb_remove_entry(tlb, virt_to_page(pmdp)); + tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif @@ -78,7 +72,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { __flush_tlb_pgtable(tlb->mm, addr); - tlb_remove_entry(tlb, virt_to_page(pudp)); + tlb_remove_table(tlb, virt_to_page(pudp)); } #endif From patchwork Fri Aug 24 15:52:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145079 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1410842ljw; Fri, 24 Aug 2018 08:53:14 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbCwHwFND5123CjqXZyHKqvrswpjN1O8MrMhSPCvCErS105wwEIN2EnvD9KJCs7LvSpHdkP X-Received: by 2002:a17:902:4a0c:: with SMTP id w12-v6mr2222533pld.289.1535125994382; Fri, 24 Aug 2018 08:53:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535125994; cv=none; d=google.com; s=arc-20160816; b=v/zoLnLSgoUrtzUXpZcvFSaIGTQuTjumfrcQv0GGsNFrunc+1V3ncPFRlfLhPTulzT Bkgiv2Phul4G1510IA9HJCmhrRCefoOVcjMfOV5Zlm8bah+LCIw7YjXvwjjvJmWxm7iZ 49I9wxFgg7qmptDBc//6Jrb/HY9nGTrAHik73r1u0nAnao7IRwO1M7hUYNCuTi9P8Cp1 tKFwq64cycyScajlISlh6lH1z9m4Wk3s7JbbzEwq6HyqqKwVO9hVKk6HfX9fqMzyU5lT 61N/mt17wRVlB1AsOhD9REzBnT2RNMDMOpcAFStNCnWmmL3yB7bj9UdZpT10Npw6ticY n8WQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=GrU7vRm1KhZZEJeulQACNDdskQuSpQ6gO5QbomWSYOU=; b=MWXpEyilDtjIoJy3pfksTbJKAyxtT3pckDUWGF6eUlPnu+z0ZpFBOlt5bsu7brySDX TaYMrR6LkiKNt2Cj5BVZYe9c0AjH/L8o+hf4KICFg7wpOqVyY/TGp2PkwIJAp7c0oTdi Wxrykn5BVz4zHbX9TRgkitoFhzxEw/ELzOkCirICuZehMSRl1JvG9iu1CEe9cIiuTO5e ihx9p0epmFBxtDTTpPA/NhWlbnx7ISxgujFm4WCa1oTT/NprNi6yoQ7DeEnqkiY7sppH VqHw7qxitelO7c3l+VRRUJPN2nqKc86bT74bqneS/ktb0lV04V5mAqpSQMlX62hbfSMa BBmQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h2-v6si6663834plh.145.2018.08.24.08.53.14; Fri, 24 Aug 2018 08:53:14 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728114AbeHXT2Z (ORCPT + 32 others); Fri, 24 Aug 2018 15:28:25 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:32874 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727566AbeHXT1w (ORCPT ); Fri, 24 Aug 2018 15:27:52 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D297D174E; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9FD2E3F7BE; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 1BE9D1AE33A3; Fri, 24 Aug 2018 16:52:48 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 07/11] asm-generic/tlb: Guard with #ifdef CONFIG_MMU Date: Fri, 24 Aug 2018 16:52:42 +0100 Message-Id: <1535125966-7666-8-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The inner workings of the mmu_gather-based TLB invalidation mechanism are not relevant to nommu configurations, so guard them with an ifdef. This allows us to implement future functions using static inlines without breaking the build. Signed-off-by: Will Deacon --- include/asm-generic/tlb.h | 3 +++ 1 file changed, 3 insertions(+) -- 2.1.4 diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b3353e21f3b3..859f897d3d53 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -20,6 +20,8 @@ #include #include +#ifdef CONFIG_MMU + #ifdef CONFIG_HAVE_RCU_TABLE_FREE /* * Semi RCU freeing of the page directories. @@ -312,4 +314,5 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_migrate_finish(mm) do {} while (0) +#endif /* CONFIG_MMU */ #endif /* _ASM_GENERIC__TLB_H */ From patchwork Fri Aug 24 15:52:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145076 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1410411ljw; Fri, 24 Aug 2018 08:52:44 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbnomL+fwLYO3jRjBxabSDRZnB66jb2bijnCtP8Vhy0p0VwW0I1fuR1gSyVclnuG2jppyFI X-Received: by 2002:a63:5706:: with SMTP id l6-v6mr2355695pgb.118.1535125964683; Fri, 24 Aug 2018 08:52:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535125964; cv=none; d=google.com; s=arc-20160816; b=vPUnl8SJLhTxwk9esGwP0QV9/AlY6nq/qAA+/PQENdxZCgbktmq154M3HWIh3x3fS9 gy6f+vu4sRImAn8KbveoenE4UeEAtDgoIF8lhChs0OVTlBRym2GDAhMqoiYGIvvVm3z7 DPpN38BbDU/vSMa7+B4pAU5d+5MS37N/ploregXy+75boxB16FeSAu4Sukar4/XDtEXa zVo+/Hkln1kFSwLASsbqyJid2olZ6UQ64mG9j73DVs69mc5OpDqWdraBhNgwKDhip52d JoxN4euj+OIN4EflU61/PTRn3kPd4et1mEz0ai236Yv1HKhXgNH6nDghXRa30zLAVXGB i2Pw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=yCfibgqVcJKkMihTC97KVlbXJOk1cx2tlzA5BAYAvEc=; b=N1eIf6wr2LfBKXB/pSbkTp+OpMKlVW8sEGCgYugYinVZu6k4qGketyf7cHsUrBkBnh i0SowHFXTvDoSOIgKjcy4Oc1reurvfXAzhQJDhIlNEBWISKh4pnUDXpoMWRETGm8v0mn AnXe1DHYI8wPBUg3xJzRY4RrMVn4RoFmbBd1PUkao7FTr65xvv7gRXOS6wTFyzRAZu8q WkISy5K2FP8aalrryaXwFdPruF2TOst/NJU12EPerC4fLEhdgFKeYdsta7mf1plE2CTv kQdiFMzqRtQup1xP8dm+YOxKuDWOB6rmw6y+vLaMVsrvEmDQ6EHkBE8cmnJ9d2hfCbat FG5A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x11-v6si6546317plr.247.2018.08.24.08.52.44; Fri, 24 Aug 2018 08:52:44 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727957AbeHXT1z (ORCPT + 32 others); Fri, 24 Aug 2018 15:27:55 -0400 Received: from foss.arm.com ([217.140.101.70]:32882 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727592AbeHXT1w (ORCPT ); Fri, 24 Aug 2018 15:27:52 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D72BA19BF; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A97903F7F3; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 2C3291AE33B8; Fri, 24 Aug 2018 16:52:48 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 08/11] asm-generic/tlb: Track freeing of page-table directories in struct mmu_gather Date: Fri, 24 Aug 2018 16:52:43 +0100 Message-Id: <1535125966-7666-9-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Peter Zijlstra Some architectures require different TLB invalidation instructions depending on whether it is only the last-level of page table being changed, or whether there are also changes to the intermediate (directory) entries higher up the tree. Add a new bit to the flags bitfield in struct mmu_gather so that the architecture code can operate accordingly if it's the intermediate levels being invalidated. Signed-off-by: Peter Zijlstra Signed-off-by: Will Deacon --- include/asm-generic/tlb.h | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 859f897d3d53..a5caf90264e6 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -99,12 +99,22 @@ struct mmu_gather { #endif unsigned long start; unsigned long end; - /* we are in the middle of an operation to clear - * a full mm and can make some optimizations */ - unsigned int fullmm : 1, - /* we have performed an operation which - * requires a complete flush of the tlb */ - need_flush_all : 1; + /* + * we are in the middle of an operation to clear + * a full mm and can make some optimizations + */ + unsigned int fullmm : 1; + + /* + * we have performed an operation which + * requires a complete flush of the tlb + */ + unsigned int need_flush_all : 1; + + /* + * we have removed page directories + */ + unsigned int freed_tables : 1; struct mmu_gather_batch *active; struct mmu_gather_batch local; @@ -139,6 +149,7 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) tlb->start = TASK_SIZE; tlb->end = 0; } + tlb->freed_tables = 0; } static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) @@ -280,6 +291,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pte_free_tlb(tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __pte_free_tlb(tlb, ptep, address); \ } while (0) #endif @@ -287,7 +299,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #ifndef pmd_free_tlb #define pmd_free_tlb(tlb, pmdp, address) \ do { \ - __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0) #endif @@ -297,6 +310,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pud_free_tlb(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __pud_free_tlb(tlb, pudp, address); \ } while (0) #endif @@ -306,7 +320,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #ifndef p4d_free_tlb #define p4d_free_tlb(tlb, pudp, address) \ do { \ - __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __p4d_free_tlb(tlb, pudp, address); \ } while (0) #endif From patchwork Fri Aug 24 15:52:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145077 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1410627ljw; Fri, 24 Aug 2018 08:53:00 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZweJJYqxDR+y/n6/hEsvDauWPZYWcs0K2U3sXyPOpkSFWn5XONVrBpLo4I38c0iqvKw6k7 X-Received: by 2002:a62:83ca:: with SMTP id h193-v6mr2588133pfe.123.1535125980209; Fri, 24 Aug 2018 08:53:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535125980; cv=none; d=google.com; s=arc-20160816; b=N24fQ4DQ0Q5L9Jnx4gMKtOCAtid4CJZrqwgKp1xTpp1f7rTV3hAUCyl7UHWMwwZzBv vGZolQgwgn1iXFe/iocooMAezVCYQ+xaHORAu6H0wix1NOW7pVYfHH33hSHSu1lf0GBB U+llSmrDxEc+/rut72lnuHTr02RxKxAx8Da9E4qqP7THxM7Z4uGJQXTf2KSy/YQPQibD I38pn8F4iA4TYyaeayc2ZO/hxccFwkRe99jPGiwxh2QQzFnLsYa9feJy9waLqQx3FW33 UMbY1VXhsJ/GhqNMcK3b8WjzA5K03zvxrULi0YTDN2Gp+a80GJgEdYAHIyrLYCPWxedp tZZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=dk/EKGepgL0z5A9hRjkk9ipN1K0IzqwFKPaRUTcQyic=; b=j/cM2JEXXoks1uVyw1f1uWRO2Rx5UN/gSRZjn4do9mTsXt/2hnQ4Hzagos82gL0IXD 0daNn4dE/wkpx2EbcdmAIJzFZuuyGHenNt/+5hokZiP8TmfNDBJPEkmozz8NDya9K7Qg fGJ+gqP65KdVNwt18ePNP6HEmw4hFAph+IsShbFJxTYXCTVazoC+Z0kMB8nnIQx5JZfj zeuPfnzfY4EjVTeEKhJ9D/XsOu+/Gp3cpNsw1G2dOAhAmmtJAQ4BtRZA1/8ZwNiG5nJl yFkWftMbzkd+Wl6hJamLBmuRlNstQ4/sFtgaImA64hWpC7JygiDjsCdsXooc+lcWTkHB WUcQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r206-v6si7404272pgr.17.2018.08.24.08.52.59; Fri, 24 Aug 2018 08:53:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728048AbeHXT2L (ORCPT + 32 others); Fri, 24 Aug 2018 15:28:11 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:32890 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727597AbeHXT1w (ORCPT ); Fri, 24 Aug 2018 15:27:52 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DFFD91A25; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B28103F93D; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 3E84B1AE33CF; Fri, 24 Aug 2018 16:52:48 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 09/11] asm-generic/tlb: Track which levels of the page tables have been cleared Date: Fri, 24 Aug 2018 16:52:44 +0100 Message-Id: <1535125966-7666-10-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org It is common for architectures with hugepage support to require only a single TLB invalidation operation per hugepage during unmap(), rather than iterating through the mapping at a PAGE_SIZE increment. Currently, however, the level in the page table where the unmap() operation occurs is not stored in the mmu_gather structure, therefore forcing architectures to issue additional TLB invalidation operations or to give up and over-invalidate by e.g. invalidating the entire TLB. Ideally, we could add an interval rbtree to the mmu_gather structure, which would allow us to associate the correct mapping granule with the various sub-mappings within the range being invalidated. However, this is costly in terms of book-keeping and memory management, so instead we approximate by keeping track of the page table levels that are cleared and provide a means to query the smallest granule required for invalidation. Signed-off-by: Will Deacon --- include/asm-generic/tlb.h | 53 ++++++++++++++++++++++++++++++++++++++++------- mm/memory.c | 4 +++- 2 files changed, 48 insertions(+), 9 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index a5caf90264e6..081b105d7215 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -116,6 +116,14 @@ struct mmu_gather { */ unsigned int freed_tables : 1; + /* + * at which levels have we cleared entries? + */ + unsigned int cleared_ptes : 1; + unsigned int cleared_pmds : 1; + unsigned int cleared_puds : 1; + unsigned int cleared_p4ds : 1; + struct mmu_gather_batch *active; struct mmu_gather_batch local; struct page *__pages[MMU_GATHER_BUNDLE]; @@ -150,6 +158,10 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) tlb->end = 0; } tlb->freed_tables = 0; + tlb->cleared_ptes = 0; + tlb->cleared_pmds = 0; + tlb->cleared_puds = 0; + tlb->cleared_p4ds = 0; } static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) @@ -199,6 +211,20 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, } #endif +static inline unsigned long tlb_get_unmap_granule(struct mmu_gather *tlb) +{ + if (tlb->cleared_ptes) + return PAGE_SIZE; + if (tlb->cleared_pmds) + return PMD_SIZE; + if (tlb->cleared_puds) + return PUD_SIZE; + if (tlb->cleared_p4ds) + return P4D_SIZE; + + return PAGE_SIZE; +} + /* * In the case of tlb vma handling, we can optimise these away in the * case where we're doing a full MM flush. When we're doing a munmap, @@ -232,13 +258,19 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_tlb_entry(tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->cleared_ptes = 1; \ __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) -#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ - do { \ - __tlb_adjust_range(tlb, address, huge_page_size(h)); \ - __tlb_remove_tlb_entry(tlb, ptep, address); \ +#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ + do { \ + unsigned long _sz = huge_page_size(h); \ + __tlb_adjust_range(tlb, address, _sz); \ + if (_sz == PMD_SIZE) \ + tlb->cleared_pmds = 1; \ + else if (_sz == PUD_SIZE) \ + tlb->cleared_puds = 1; \ + __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) /** @@ -252,6 +284,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_pmd_tlb_entry(tlb, pmdp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PMD_SIZE); \ + tlb->cleared_pmds = 1; \ __tlb_remove_pmd_tlb_entry(tlb, pmdp, address); \ } while (0) @@ -266,6 +299,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_pud_tlb_entry(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PUD_SIZE); \ + tlb->cleared_puds = 1; \ __tlb_remove_pud_tlb_entry(tlb, pudp, address); \ } while (0) @@ -291,7 +325,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pte_free_tlb(tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_pmds = 1; \ __pte_free_tlb(tlb, ptep, address); \ } while (0) #endif @@ -300,7 +335,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pmd_free_tlb(tlb, pmdp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_puds = 1; \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0) #endif @@ -310,7 +346,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pud_free_tlb(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_p4ds = 1; \ __pud_free_tlb(tlb, pudp, address); \ } while (0) #endif @@ -321,7 +358,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define p4d_free_tlb(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ __p4d_free_tlb(tlb, pudp, address); \ } while (0) #endif diff --git a/mm/memory.c b/mm/memory.c index 83aef222f11b..69982d1d9b7a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -267,8 +267,10 @@ void arch_tlb_finish_mmu(struct mmu_gather *tlb, { struct mmu_gather_batch *batch, *next; - if (force) + if (force) { + __tlb_reset_range(tlb); __tlb_adjust_range(tlb, start, end - start); + } tlb_flush_mmu(tlb); From patchwork Fri Aug 24 15:52:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145078 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1410718ljw; Fri, 24 Aug 2018 08:53:06 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbbU2eo/4FDI9yAbJA7kHQtQBF7ZV/w4JH2UOScqvf+8yv602ztFLXzfQUM/PaRRAcMT3v5 X-Received: by 2002:a63:b19:: with SMTP id 25-v6mr2289461pgl.301.1535125985863; Fri, 24 Aug 2018 08:53:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535125985; cv=none; d=google.com; s=arc-20160816; b=a09is4g5pp+wtdB8CohddwDvgA9TZS87cm6bw9mqfsekpfAk+W6FF65Q1WhTavbkhP TdJ5bx/YB051ZBfgUInqQoYoVB//3ddr9o1xWj5rxf613YrsUx9TkNyuFxcdTkRlPyMQ q2BhQshBHXnRYIm68AzniRzrIfjiakJsRk7JfCX73gmmvoq++mko6Hdm6Ovx+VkD74Bq 7wBlRrYO9/WeMGHfatz8q6FuAUBVrE737BjZipN143jfpAqGsxRP+Tycm3MP8y6Twp2X WCWioKII7EWA2f/S3vh2ARt7szMMOMuwaO4L3wXVgpnxfkee6rQkgKOgAbWa2nnz0ijx IM2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=SuSTYfW56h5OPXzEN7E9R6K4q9ZQKOw8hFmFbAmIwyc=; b=RY+d9ai62fq2l5ezeAbx6t0YUM+09Av4MNY/A/VBZNW354wP/U30VMmGWMawo1Vuh0 Ty1GiODrPaDQWBW4ZgmHgOpwBGemX6braNfjUiR3ZmY/3TC8WbLChjTu4ffbDpSrSRH8 sLEOzwssUEfypZ7QIzewUaw2A4Sq/1ep5V671q4sgZEXEuBtfnxaReLkaEo3sh7KpbkO 4W65GvbaM3dgVtI8jMnpUx+FdKJ8FR2wZKuOemRNaTaRhEAbPZ9uQh0bh+YKeKL7Alhq +HWGgn2u38KCH33uRaHwDu6oJPb01dNVpIIojCLxbq8dJ4o0tIeESOqGIeWd9y0gGcZP l4PA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r206-v6si7404272pgr.17.2018.08.24.08.53.05; Fri, 24 Aug 2018 08:53:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728086AbeHXT2R (ORCPT + 32 others); Fri, 24 Aug 2018 15:28:17 -0400 Received: from foss.arm.com ([217.140.101.70]:32902 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727613AbeHXT1w (ORCPT ); Fri, 24 Aug 2018 15:27:52 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E8EC01AC1; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id BB12A3F994; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 50CF51AE33DD; Fri, 24 Aug 2018 16:52:48 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 10/11] arm64: tlb: Adjust stride and type of TLBI according to mmu_gather Date: Fri, 24 Aug 2018 16:52:45 +0100 Message-Id: <1535125966-7666-11-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Now that the core mmu_gather code keeps track of both the levels of page table cleared and also whether or not these entries correspond to intermediate entries, we can use this in our tlb_flush() callback to reduce the number of invalidations we issue as well as their scope. Signed-off-by: Will Deacon --- arch/arm64/include/asm/tlb.h | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index bd00017d529a..baca8dff6884 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -34,20 +34,21 @@ static void tlb_flush(struct mmu_gather *tlb); static inline void tlb_flush(struct mmu_gather *tlb) { struct vm_area_struct vma = TLB_FLUSH_VMA(tlb->mm, 0); + bool last_level = !tlb->freed_tables; + unsigned long stride = tlb_get_unmap_granule(tlb); /* - * The ASID allocator will either invalidate the ASID or mark - * it as used. + * If we're tearing down the address space then we only care about + * invalidating the walk-cache, since the ASID allocator won't + * reallocate our ASID without invalidating the entire TLB. */ - if (tlb->fullmm) + if (tlb->fullmm) { + if (!last_level) + flush_tlb_mm(tlb->mm); return; + } - /* - * The intermediate page table levels are already handled by - * the __(pte|pmd|pud)_free_tlb() functions, so last level - * TLBI is sufficient here. - */ - __flush_tlb_range(&vma, tlb->start, tlb->end, PAGE_SIZE, true); + __flush_tlb_range(&vma, tlb->start, tlb->end, stride, last_level); } static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, From patchwork Fri Aug 24 15:52:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 145075 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1410395ljw; Fri, 24 Aug 2018 08:52:43 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbGa9DeDXuj3IHSMpisG7mIRRMtgF/GT9xEzEd7XMn0GhDq8SIbI37dbTGBglkS5v5dbcyz X-Received: by 2002:a62:7088:: with SMTP id l130-v6mr2585051pfc.144.1535125963139; Fri, 24 Aug 2018 08:52:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535125963; cv=none; d=google.com; s=arc-20160816; b=yWUKfrGd0dTp4qUev4imfrGGXJTq4pPQgLmFCwHAMg2qHPOyxJhOYwmEVB02tNNtif pqAy+lPUSSXc0l3HCBhYHxwUzS+9ZTbl8SctH7AoDmbuwETPVHC3nsVWW9m15Ff1ouyT iX7Z0XDhknSwjCYD1Js3cV6yW+sR21rr38DuJSZZj8uTRl2EyIZZLYFrZ1grca42uhG9 afnQo6LuHjxHAnb8jRtxgsUhGN1iYJQwRAcsSiZQknTYz2UO1CVHjp7iYTXGn+7hTTb3 FdBOvptZ76Lu1ktGepb8E/IRr30XAHX6NmcJFbX6L7iv2BADsjtLM48sGOvmaz4RR2kH zQjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=AUM4EXrKH8SY7z1d8V/QVUkV+WfTR8znL8jv7IfUU1U=; b=p9QBVwN0CuB4rwQ6Hdk1k0OpjJwI/sIwRppWwUY9G0/DMe37e4pUY3jiO91EdzvipN o/6xTSE3pxEttfzoET6apQ/rspSC15fzXIQyRWtMcgNH0eNnZynxg5crmf5coAGWerhT M85cbBNRVERjOKhnjC8RZEuPIyD4OIAiVqwwBF2InpOmpy5zpIanZrgtnJhC7qmJ695G /t057M62p4okk2oOixPPlXIq2YmNtuigu5Te9BnKtM8n0Q9pkqZDpUxiQgFHi37s9xza 5wCuIKYsEY3WAlKMONUy5UGSiFhIr2ITo2PXiD2Kj0JuRqRlN+9O0Q3UeJYFQ7smlsHP VO0g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p63-v6si5829157pgp.651.2018.08.24.08.52.42; Fri, 24 Aug 2018 08:52:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727836AbeHXT1y (ORCPT + 32 others); Fri, 24 Aug 2018 15:27:54 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:32904 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727615AbeHXT1w (ORCPT ); Fri, 24 Aug 2018 15:27:52 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F1E391AED; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C42433F5BC; Fri, 24 Aug 2018 08:52:38 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 623591AE33E9; Fri, 24 Aug 2018 16:52:48 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [RFC PATCH 11/11] arm64: tlb: Avoid synchronous TLBIs when freeing page tables Date: Fri, 24 Aug 2018 16:52:46 +0100 Message-Id: <1535125966-7666-12-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> References: <1535125966-7666-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org By selecting HAVE_RCU_TABLE_INVALIDATE, we can rely on tlb_flush() being called if we fail to batch table pages for freeing. This in turn allows us to postpone walk-cache invalidation until tlb_finish_mmu(), which avoids lots of unnecessary DSBs and means we can shoot down the ASID if the range is large enough. Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/tlb.h | 3 --- arch/arm64/include/asm/tlbflush.h | 11 ----------- 3 files changed, 1 insertion(+), 14 deletions(-) -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 29e75b47becd..89059ee1eccc 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -142,6 +142,7 @@ config ARM64 select HAVE_PERF_USER_STACK_DUMP select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RCU_TABLE_FREE + select HAVE_RCU_TABLE_INVALIDATE select HAVE_RSEQ select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index baca8dff6884..4f65614b3141 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -54,7 +54,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - __flush_tlb_pgtable(tlb->mm, addr); pgtable_page_dtor(pte); tlb_remove_table(tlb, pte); } @@ -63,7 +62,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - __flush_tlb_pgtable(tlb->mm, addr); tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif @@ -72,7 +70,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - __flush_tlb_pgtable(tlb->mm, addr); tlb_remove_table(tlb, virt_to_page(pudp)); } #endif diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 1f77d08e638b..2064ba97845f 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -215,17 +215,6 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end * Used to invalidate the TLB (walk caches) corresponding to intermediate page * table levels (pgd/pud/pmd). */ -static inline void __flush_tlb_pgtable(struct mm_struct *mm, - unsigned long uaddr) -{ - unsigned long addr = __TLBI_VADDR(uaddr, ASID(mm)); - - dsb(ishst); - __tlbi(vae1is, addr); - __tlbi_user(vae1is, addr); - dsb(ish); -} - static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr) { unsigned long addr = __TLBI_VADDR(kaddr, 0);