From patchwork Mon Nov 10 13:56:17 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 40487 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ee0-f70.google.com (mail-ee0-f70.google.com [74.125.83.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 166D2218DE for ; Mon, 10 Nov 2014 13:56:32 +0000 (UTC) Received: by mail-ee0-f70.google.com with SMTP id b57sf5560448eek.9 for ; Mon, 10 Nov 2014 05:56:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:date:from:to:cc:subject:message-id :references:mime-version:in-reply-to:user-agent:sender:precedence :list-id:x-original-sender:x-original-authentication-results :mailing-list:list-post:list-help:list-archive:list-unsubscribe :content-type:content-disposition; bh=K/p0t8S3psh0GZLkyeXalteLdl5gvaGGbMnpq/OZoyE=; b=g86qIvpEFHbFdsuSl+N79FViiqIvJ7vbwTY9OloUGki67cP8+AFU1upglblK1qjol7 cgg0vkKcxV4iymB1zDzIK7yMjoP1vHLwWnEGKKQZzpXPzss81TkAD+GAoCDa2EAT4TPH B0ID0YGq/ZJ7aViJN4uLG2o+5mC7+EfRuHBK1TOteGOXYhK6LxKfZrdOGN4iEsqopb0q /dF4PcQotg5wz3hvIn5fSNwxUd7TLUB0Q6kVjbTvy6maDycH6hZ4G/zQWR35yBRrW5y7 Qk0JiMZaUANRacd3KsXup2rSBV7ug5ZZC8s+euOC+SQLc9q+64fwqmO5WoUReoL+jMoU 74lA== X-Gm-Message-State: ALoCoQnHjesWkjis8Oj+B9N9900Vhn5c4F86LBwiUFUXieRPkZO5MbHG5YoPPvH7WUxWXqHOqD/W X-Received: by 10.152.42.171 with SMTP id p11mr5355211lal.4.1415627791028; Mon, 10 Nov 2014 05:56:31 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.43.166 with SMTP id x6ls441313lal.93.gmail; Mon, 10 Nov 2014 05:56:30 -0800 (PST) X-Received: by 10.112.63.133 with SMTP id g5mr28011279lbs.33.1415627790398; Mon, 10 Nov 2014 05:56:30 -0800 (PST) Received: from mail-la0-f51.google.com (mail-la0-f51.google.com. [209.85.215.51]) by mx.google.com with ESMTPS id 6si27040320laj.120.2014.11.10.05.56.30 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 10 Nov 2014 05:56:30 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.51 as permitted sender) client-ip=209.85.215.51; Received: by mail-la0-f51.google.com with SMTP id q1so7765999lam.10 for ; Mon, 10 Nov 2014 05:56:30 -0800 (PST) X-Received: by 10.112.12.35 with SMTP id v3mr2724064lbb.80.1415627790265; Mon, 10 Nov 2014 05:56:30 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.184.201 with SMTP id ew9csp90105lbc; Mon, 10 Nov 2014 05:56:28 -0800 (PST) X-Received: by 10.66.100.136 with SMTP id ey8mr17290219pab.93.1415627787522; Mon, 10 Nov 2014 05:56:27 -0800 (PST) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b14si16510744pat.196.2014.11.10.05.56.26 for ; Mon, 10 Nov 2014 05:56:27 -0800 (PST) Received-SPF: none (google.com: linux-kernel-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753043AbaKJN4Z (ORCPT + 25 others); Mon, 10 Nov 2014 08:56:25 -0500 Received: from foss-mx-na.foss.arm.com ([217.140.108.86]:55521 "EHLO foss-mx-na.foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752582AbaKJN4Y (ORCPT ); Mon, 10 Nov 2014 08:56:24 -0500 Received: from foss-smtp-na-1.foss.arm.com (unknown [10.80.61.8]) by foss-mx-na.foss.arm.com (Postfix) with ESMTP id 7291666; Mon, 10 Nov 2014 07:56:20 -0600 (CST) Received: from collaborate-mta1.arm.com (highbank-bc01-b06.austin.arm.com [10.112.81.134]) by foss-smtp-na-1.foss.arm.com (Postfix) with ESMTP id E45565FAD7; Mon, 10 Nov 2014 07:56:17 -0600 (CST) Received: from arm.com (edgewater-inn.cambridge.arm.com [10.1.203.36]) by collaborate-mta1.arm.com (Postfix) with ESMTPS id CCD9013F78C; Mon, 10 Nov 2014 07:56:16 -0600 (CST) Date: Mon, 10 Nov 2014 13:56:17 +0000 From: Will Deacon To: Catalin Marinas Cc: Linus Torvalds , Peter Zijlstra , Linux Kernel Mailing List , Russell King - ARM Linux , Benjamin Herrenschmidt Subject: Re: [RFC PATCH 1/2] zap_pte_range: update addr when forcing flush after TLB batching faiure Message-ID: <20141110135617.GC23942@arm.com> References: <20141029194738.GA29911@arm.com> <20141104142955.GA24816@e104818-lin.cambridge.arm.com> <20141106135740.GC19702@e104818-lin.cambridge.arm.com> <20141106183808.GH19702@e104818-lin.cambridge.arm.com> <20141107165003.GG29148@e104818-lin.cambridge.arm.com> MIME-Version: 1.0 In-Reply-To: <20141107165003.GG29148@e104818-lin.cambridge.arm.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: will.deacon@arm.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.51 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Content-Disposition: inline On Fri, Nov 07, 2014 at 04:50:04PM +0000, Catalin Marinas wrote: > On Thu, Nov 06, 2014 at 09:29:54PM +0000, Linus Torvalds wrote: > > That's fine. That makes sense. In fact, how about adding "granularity" > > to the mmu_gather structure, and then doing:\ > > > > - in __tlb_reset_range(), setting it to ~0ul > > > > - add "granularity" to __tlb_adjust_range(), and make it do something like > > > > if (!tlb->fullmm) { > > tlb->granularity = min(tlb->granularity, granularity); > > tlb->start = min(tlb->start, address); > > tlb->end = max(tlb->end, address+1); > > } > > > > and then the TLB flush logic would basically do > > > > address = tlb->start; > > do { > > flush(address); > > if (address + tlb->granularity < address) > > break; > > address = address + tlb->granularity; > > } while (address < tlb->end); > > > > or something like that. > > Indeed. We'll come up with a patch after Will's clean-up. My clean-up is the patch I sent previously, plus the removal of need_flush. Incremental diff for the latter part below. We drop a set of need_flush from tlb_remove_table, but I can't figure out why it was there in the first place (need_flush was already set by pXd_free_tlb). Will --->8 --- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a9c9df0f60ff..c028fe37456f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -39,7 +39,7 @@ static inline void tlb_flush(struct mmu_gather *tlb) { if (tlb->fullmm) { flush_tlb_mm(tlb->mm); - } else if (tlb->end > 0) { + } else { struct vm_area_struct vma = { .vm_mm = tlb->mm, }; flush_tlb_range(&vma, tlb->start, tlb->end); } diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 340bc5c5ca2d..08848050922e 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -96,10 +96,9 @@ struct mmu_gather { #endif unsigned long start; unsigned long end; - unsigned int need_flush : 1, /* Did free PTEs */ /* we are in the middle of an operation to clear * a full mm and can make some optimizations */ - fullmm : 1, + unsigned int fullmm : 1, /* we have performed an operation which * requires a complete flush of the tlb */ need_flush_all : 1; @@ -131,10 +130,8 @@ static inline void tlb_remove_page(struct mmu_gather *tlb, struct page *page) static inline void __tlb_adjust_range(struct mmu_gather *tlb, unsigned long address) { - if (!tlb->fullmm) { - tlb->start = min(tlb->start, address); - tlb->end = max(tlb->end, address + PAGE_SIZE); - } + tlb->start = min(tlb->start, address); + tlb->end = max(tlb->end, address + PAGE_SIZE); } static inline void __tlb_reset_range(struct mmu_gather *tlb) @@ -154,7 +151,7 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) #define __tlb_end_vma(tlb, vma) \ do { \ - if (!tlb->fullmm) { \ + if (!tlb->fullmm && tlb->end) { \ tlb_flush(tlb); \ __tlb_reset_range(tlb); \ } \ @@ -171,13 +168,12 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) /** * tlb_remove_tlb_entry - remember a pte unmapping for later tlb invalidation. * - * Record the fact that pte's were really umapped in ->need_flush, so we can - * later optimise away the tlb invalidate. This helps when userspace is - * unmapping already-unmapped pages, which happens quite a lot. + * Record the fact that pte's were really unmapped by updating the range, + * so we can later optimise away the tlb invalidate. This helps when + * userspace is unmapping already-unmapped pages, which happens quite a lot. */ #define tlb_remove_tlb_entry(tlb, ptep, address) \ do { \ - tlb->need_flush = 1; \ __tlb_adjust_range(tlb, address); \ __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) @@ -192,14 +188,12 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) #define tlb_remove_pmd_tlb_entry(tlb, pmdp, address) \ do { \ - tlb->need_flush = 1; \ __tlb_adjust_range(tlb, address); \ __tlb_remove_pmd_tlb_entry(tlb, pmdp, address); \ } while (0) #define pte_free_tlb(tlb, ptep, address) \ do { \ - tlb->need_flush = 1; \ __tlb_adjust_range(tlb, address); \ __pte_free_tlb(tlb, ptep, address); \ } while (0) @@ -207,7 +201,6 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) #ifndef __ARCH_HAS_4LEVEL_HACK #define pud_free_tlb(tlb, pudp, address) \ do { \ - tlb->need_flush = 1; \ __tlb_adjust_range(tlb, address); \ __pud_free_tlb(tlb, pudp, address); \ } while (0) @@ -215,7 +208,6 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) #define pmd_free_tlb(tlb, pmdp, address) \ do { \ - tlb->need_flush = 1; \ __tlb_adjust_range(tlb, address); \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0) diff --git a/mm/memory.c b/mm/memory.c index 0bc940e41ec9..8b1c1d2e7c67 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -220,7 +220,6 @@ void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, unsigned long /* Is it from 0 to ~0? */ tlb->fullmm = !(start | (end+1)); tlb->need_flush_all = 0; - tlb->need_flush = 0; tlb->local.next = NULL; tlb->local.nr = 0; tlb->local.max = ARRAY_SIZE(tlb->__pages); @@ -236,7 +235,9 @@ void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, unsigned long static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) { - tlb->need_flush = 0; + if (!tlb->end) + return; + tlb_flush(tlb); #ifdef CONFIG_HAVE_RCU_TABLE_FREE tlb_table_flush(tlb); @@ -257,8 +258,6 @@ static void tlb_flush_mmu_free(struct mmu_gather *tlb) void tlb_flush_mmu(struct mmu_gather *tlb) { - if (!tlb->need_flush) - return; tlb_flush_mmu_tlbonly(tlb); tlb_flush_mmu_free(tlb); } @@ -293,7 +292,7 @@ int __tlb_remove_page(struct mmu_gather *tlb, struct page *page) { struct mmu_gather_batch *batch; - VM_BUG_ON(!tlb->need_flush); + VM_BUG_ON(!tlb->end); batch = tlb->active; batch->pages[batch->nr++] = page; @@ -360,8 +359,6 @@ void tlb_remove_table(struct mmu_gather *tlb, void *table) { struct mmu_table_batch **batch = &tlb->batch; - tlb->need_flush = 1; - /* * When there's less then two users of this mm there cannot be a * concurrent page-table walk.