From patchwork Tue Oct 6 17:46:23 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 54558 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-lb0-f197.google.com (mail-lb0-f197.google.com [209.85.217.197]) by patches.linaro.org (Postfix) with ESMTPS id 8ABA022F05 for ; Tue, 6 Oct 2015 17:49:35 +0000 (UTC) Received: by lbwr8 with SMTP id r8sf50908521lbw.0 for ; Tue, 06 Oct 2015 10:49:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:subject:date:message-id :in-reply-to:references:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:cc:mime-version :content-type:content-transfer-encoding:sender:errors-to :x-original-sender:x-original-authentication-results:mailing-list; bh=dIV4mVH2l9ncrkUNhi6Lfsv281IVx5GzjIv78+PaRu8=; b=Oeax1hU4fEuhQo6YeEhg/AZUXbwRFfboDw1QjlCx6TGolSe+cu7PVBOUEgf/YcrDdc gEb2VhLPkK6miDlaZdg/I1kaGhECkuZ99PBxQBG5DWK9cXrUL3H0262XBZAImj41kBqR k603y1vM7iPfopvflb1GFxmGrdtP7A54RtYhXgT0+7bl53gYaTRzu6GADWeCJVA92nbY GZM+4oSDt0wG8BN0kG5g5OMxUeYOljI0GDIw2ATVkowKH5JgZeyCOEs0+EXTl3Z1NfBU BMBzbrfCqUFoZCdmexnIEVJTi1zNsEAWqGCt29bibYwIhIzdUa7ZjlERDTZH+lSqVX0P Xh2g== X-Gm-Message-State: ALoCoQmlGpCCqBVNZfl1PThFNXRE1VJkaNptcsMd7l1oI3dPYEZp6wu79CaTkAqxlh3Dg2YGel3y X-Received: by 10.112.138.170 with SMTP id qr10mr6827017lbb.4.1444153774137; Tue, 06 Oct 2015 10:49:34 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.25.161.20 with SMTP id k20ls539708lfe.88.gmail; Tue, 06 Oct 2015 10:49:33 -0700 (PDT) X-Received: by 10.112.13.161 with SMTP id i1mr15312022lbc.96.1444153773944; Tue, 06 Oct 2015 10:49:33 -0700 (PDT) Received: from mail-la0-f46.google.com (mail-la0-f46.google.com. [209.85.215.46]) by mx.google.com with ESMTPS id w11si21808295lfd.40.2015.10.06.10.49.33 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Oct 2015 10:49:33 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.46 as permitted sender) client-ip=209.85.215.46; Received: by labzv5 with SMTP id zv5so149106063lab.1 for ; Tue, 06 Oct 2015 10:49:33 -0700 (PDT) X-Received: by 10.25.28.73 with SMTP id c70mr8093161lfc.76.1444153773724; Tue, 06 Oct 2015 10:49:33 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.59.35 with SMTP id w3csp1985741lbq; Tue, 6 Oct 2015 10:49:33 -0700 (PDT) X-Received: by 10.180.104.38 with SMTP id gb6mr18338748wib.86.1444153773065; Tue, 06 Oct 2015 10:49:33 -0700 (PDT) Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id fy5si25163815wib.14.2015.10.06.10.49.32 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Oct 2015 10:49:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) client-ip=2001:1868:205::9; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1ZjWLE-0000Eb-I5; Tue, 06 Oct 2015 17:47:56 +0000 Received: from cam-admin0.cambridge.arm.com ([217.140.96.50]) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1ZjWKF-0007p2-By for linux-arm-kernel@lists.infradead.org; Tue, 06 Oct 2015 17:46:58 +0000 Received: from edgewater-inn.cambridge.arm.com (edgewater-inn.cambridge.arm.com [10.1.203.122]) by cam-admin0.cambridge.arm.com (8.12.6/8.12.6) with ESMTP id t96HkWWr001368; Tue, 6 Oct 2015 18:46:32 +0100 (BST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 3DF041AE1009; Tue, 6 Oct 2015 18:46:32 +0100 (BST) From: Will Deacon To: catalin.marinas@arm.com Subject: [PATCH v2 03/10] arm64: flush: use local TLB and I-cache invalidation Date: Tue, 6 Oct 2015 18:46:23 +0100 Message-Id: <1444153590-24173-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1444153590-24173-1-git-send-email-will.deacon@arm.com> References: <1444153590-24173-1-git-send-email-will.deacon@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20151006_104655_870681_E8281FA9 X-CRM114-Status: GOOD ( 12.78 ) X-Spam-Score: -6.9 (------) X-Spam-Report: SpamAssassin version 3.4.0 on bombadil.infradead.org summary: Content analysis details: (-6.9 points) pts rule name description ---- ---------------------- -------------------------------------------------- -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high trust [217.140.96.50 listed in list.dnswl.org] -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay domain -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , Cc: Will Deacon , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: will.deacon@arm.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.46 as permitted sender) smtp.mailfrom=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 There are a number of places where a single CPU is running with a private page-table and we need to perform maintenance on the TLB and I-cache in order to ensure correctness, but do not require the operation to be broadcast to other CPUs. This patch adds local variants of tlb_flush_all and __flush_icache_all to support these use-cases and updates the callers respectively. __local_flush_icache_all also implies an isb, since it is intended to be used synchronously. Reviewed-by: Catalin Marinas Signed-off-by: Will Deacon Acked-by: David Daney Acked-by: Ard Biesheuvel --- arch/arm64/include/asm/cacheflush.h | 7 +++++++ arch/arm64/include/asm/tlbflush.h | 8 ++++++++ arch/arm64/kernel/efi.c | 4 ++-- arch/arm64/kernel/smp.c | 2 +- arch/arm64/kernel/suspend.c | 2 +- arch/arm64/mm/context.c | 4 ++-- arch/arm64/mm/mmu.c | 2 +- 7 files changed, 22 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/cacheflush.h index c75b8d027eb1..54efedaf331f 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -115,6 +115,13 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *, #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 extern void flush_dcache_page(struct page *); +static inline void __local_flush_icache_all(void) +{ + asm("ic iallu"); + dsb(nsh); + isb(); +} + static inline void __flush_icache_all(void) { asm("ic ialluis"); diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 7bd2da021658..96f944e75dc4 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -63,6 +63,14 @@ * only require the D-TLB to be invalidated. * - kaddr - Kernel virtual memory address */ +static inline void local_flush_tlb_all(void) +{ + dsb(nshst); + asm("tlbi vmalle1"); + dsb(nsh); + isb(); +} + static inline void flush_tlb_all(void) { dsb(ishst); diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 13671a9cf016..4d12926ea40d 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -344,9 +344,9 @@ static void efi_set_pgd(struct mm_struct *mm) else cpu_switch_mm(mm->pgd, mm); - flush_tlb_all(); + local_flush_tlb_all(); if (icache_is_aivivt()) - __flush_icache_all(); + __local_flush_icache_all(); } void efi_virtmap_load(void) diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index dbdaacddd9a5..fdd4d4dbd64f 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -152,7 +152,7 @@ asmlinkage void secondary_start_kernel(void) * point to zero page to avoid speculatively fetching new entries. */ cpu_set_reserved_ttbr0(); - flush_tlb_all(); + local_flush_tlb_all(); cpu_set_default_tcr_t0sz(); preempt_disable(); diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c index 8297d502217e..3c5e4e6dcf68 100644 --- a/arch/arm64/kernel/suspend.c +++ b/arch/arm64/kernel/suspend.c @@ -90,7 +90,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) else cpu_switch_mm(mm->pgd, mm); - flush_tlb_all(); + local_flush_tlb_all(); /* * Restore per-cpu offset before any kernel diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index d70ff14dbdbd..48b53fb381af 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -48,9 +48,9 @@ static void flush_context(void) { /* set the reserved TTBR0 before flushing the TLB */ cpu_set_reserved_ttbr0(); - flush_tlb_all(); + local_flush_tlb_all(); if (icache_is_aivivt()) - __flush_icache_all(); + __local_flush_icache_all(); } static void set_mm_context(struct mm_struct *mm, unsigned int asid) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 9211b8527f25..71a310478c9e 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -456,7 +456,7 @@ void __init paging_init(void) * point to zero page to avoid speculatively fetching new entries. */ cpu_set_reserved_ttbr0(); - flush_tlb_all(); + local_flush_tlb_all(); cpu_set_default_tcr_t0sz(); }