From patchwork Thu May 23 15:31:20 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Capper X-Patchwork-Id: 17155 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-gh0-f200.google.com (mail-gh0-f200.google.com [209.85.160.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 47CB82395B for ; Thu, 23 May 2013 15:32:43 +0000 (UTC) Received: by mail-gh0-f200.google.com with SMTP id 10sf3833396ghy.11 for ; Thu, 23 May 2013 08:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-beenthere:x-forwarded-to:x-forwarded-for :delivered-to:from:to:cc:subject:date:message-id:x-mailer :in-reply-to:references:x-gm-message-state:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-google-group-id:list-post:list-help:list-archive:list-unsubscribe; bh=uazyZPybZOVnUDZRZh7l92oDDEYCCzgh9f/0wLEWkIc=; b=gBjRQWara0YWoRy3FtkudEeOLNjRn3zZDlogVV/sxTVWc16x0EcnLWnLKq9lBXgFdE RAFvAkDt9+V0Dh0dZaLu0wVX8J0sggHpCqM8PkqRUTa89iZTIHQ7QwMDSllIpJuBe8Yp FW+hG16L600Z9U59r6fojFv3KmNJbY+Il4quiVedl5Kdu+5b07fOwcVfXLsGquec0NlL sZp9rm0hHM6FIg7OM1/crOzSFN2edtdT/+gPl7rF2BV7yrGitKduWKjN+Sd02vZpFWvr l07bXEstq5u3ojQa3gZTbk67yIiVsNizJR4bgyI5tgX0ZTy9tO3ZZ8DEXXjqtjSfvay4 y0gg== X-Received: by 10.236.198.168 with SMTP id v28mr6234469yhn.10.1369323107988; Thu, 23 May 2013 08:31:47 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.49.52.74 with SMTP id r10ls848156qeo.85.gmail; Thu, 23 May 2013 08:31:47 -0700 (PDT) X-Received: by 10.52.164.163 with SMTP id yr3mr4752773vdb.76.1369323107771; Thu, 23 May 2013 08:31:47 -0700 (PDT) Received: from mail-ve0-x231.google.com (mail-ve0-x231.google.com [2607:f8b0:400c:c01::231]) by mx.google.com with ESMTPS id b16si6648174vcq.46.2013.05.23.08.31.47 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 23 May 2013 08:31:47 -0700 (PDT) Received-SPF: neutral (google.com: 2607:f8b0:400c:c01::231 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=2607:f8b0:400c:c01::231; Received: by mail-ve0-f177.google.com with SMTP id ox1so2498822veb.8 for ; Thu, 23 May 2013 08:31:47 -0700 (PDT) X-Received: by 10.52.69.200 with SMTP id g8mr4783364vdu.4.1369323107606; Thu, 23 May 2013 08:31:47 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.220.126.138 with SMTP id c10csp57339vcs; Thu, 23 May 2013 08:31:47 -0700 (PDT) X-Received: by 10.180.212.3 with SMTP id ng3mr25963219wic.22.1369323106598; Thu, 23 May 2013 08:31:46 -0700 (PDT) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [2a00:1450:400c:c00::22e]) by mx.google.com with ESMTPS id em9si14734805wic.35.2013.05.23.08.31.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 23 May 2013 08:31:46 -0700 (PDT) Received-SPF: neutral (google.com: 2a00:1450:400c:c00::22e is neither permitted nor denied by best guess record for domain of steve.capper@linaro.org) client-ip=2a00:1450:400c:c00::22e; Received: by mail-wg0-f46.google.com with SMTP id l18so2117611wgh.13 for ; Thu, 23 May 2013 08:31:46 -0700 (PDT) X-Received: by 10.180.74.207 with SMTP id w15mr25722009wiv.19.1369323101658; Thu, 23 May 2013 08:31:41 -0700 (PDT) Received: from localhost.localdomain (marmot.wormnet.eu. [188.246.204.87]) by mx.google.com with ESMTPSA id b11sm36416484wiv.10.2013.05.23.08.31.40 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 23 May 2013 08:31:41 -0700 (PDT) From: Steve Capper To: linux-arm-kernel@lists.infradead.org Cc: will.deacon@arm.com, patches@linaro.org, Catalin Marinas , Steve Capper Subject: [PATCH v2 4/4] ARM: mm: Transparent huge page support for LPAE systems. Date: Thu, 23 May 2013 16:31:20 +0100 Message-Id: <1369323080-9673-5-git-send-email-steve.capper@linaro.org> X-Mailer: git-send-email 1.7.2.5 In-Reply-To: <1369323080-9673-1-git-send-email-steve.capper@linaro.org> References: <1369323080-9673-1-git-send-email-steve.capper@linaro.org> X-Gm-Message-State: ALoCoQmXNHtKJKxiDQXdSZu6xXIQQ9AHSbkvtQLL2BP00D31949Z3ip2XW5fW51tU+en7ZcAd/1v X-Original-Sender: steve.capper@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 2607:f8b0:400c:c01::231 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , From: Catalin Marinas The patch adds support for THP (transparent huge pages) to LPAE systems. When this feature is enabled, the kernel tries to map anonymous pages as 2MB sections where possible. Signed-off-by: Catalin Marinas [steve.capper@linaro.org: symbolic constants used, value of PMD_SECT_SPLITTING adjusted, tlbflush.h included in pgtable.h, added PROT_NONE support.] Signed-off-by: Steve Capper Reviewed-by: Will Deacon --- arch/arm/Kconfig | 4 ++ arch/arm/include/asm/pgtable-3level-hwdef.h | 2 + arch/arm/include/asm/pgtable-3level.h | 60 +++++++++++++++++++++++++++++ arch/arm/include/asm/pgtable.h | 3 ++ arch/arm/include/asm/tlb.h | 6 +++ arch/arm/include/asm/tlbflush.h | 2 + arch/arm/mm/fsr-3level.c | 2 +- 7 files changed, 78 insertions(+), 1 deletion(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 860f034..f07a462 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1711,6 +1711,10 @@ config SYS_SUPPORTS_HUGETLBFS def_bool y depends on ARM_LPAE +config HAVE_ARCH_TRANSPARENT_HUGEPAGE + def_bool y + depends on ARM_LPAE + source "mm/Kconfig" config FORCE_MAX_ZONEORDER diff --git a/arch/arm/include/asm/pgtable-3level-hwdef.h b/arch/arm/include/asm/pgtable-3level-hwdef.h index 42df407..f088c86 100644 --- a/arch/arm/include/asm/pgtable-3level-hwdef.h +++ b/arch/arm/include/asm/pgtable-3level-hwdef.h @@ -42,6 +42,8 @@ */ #define PMD_SECT_BUFFERABLE (_AT(pmdval_t, 1) << 2) #define PMD_SECT_CACHEABLE (_AT(pmdval_t, 1) << 3) +#define PMD_SECT_USER (_AT(pmdval_t, 1) << 6) /* AP[1] */ +#define PMD_SECT_RDONLY (_AT(pmdval_t, 1) << 7) /* AP[2] */ #define PMD_SECT_S (_AT(pmdval_t, 3) << 8) #define PMD_SECT_AF (_AT(pmdval_t, 1) << 10) #define PMD_SECT_nG (_AT(pmdval_t, 1) << 11) diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h index d1bcd82..54733e5 100644 --- a/arch/arm/include/asm/pgtable-3level.h +++ b/arch/arm/include/asm/pgtable-3level.h @@ -87,6 +87,11 @@ #define L_PTE_SPECIAL (_AT(pteval_t, 1) << 56) /* unused */ #define L_PTE_NONE (_AT(pteval_t, 1) << 57) /* PROT_NONE */ +#define PMD_SECT_VALID (_AT(pmdval_t, 1) << 0) +#define PMD_SECT_DIRTY (_AT(pmdval_t, 1) << 55) +#define PMD_SECT_SPLITTING (_AT(pmdval_t, 1) << 56) +#define PMD_SECT_NONE (_AT(pmdval_t, 1) << 57) + /* * To be used in assembly code with the upper page attributes. */ @@ -196,6 +201,61 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr) #define pte_huge(pte) (pte_val(pte) && !(pte_val(pte) & PTE_TABLE_BIT)) #define pte_mkhuge(pte) (__pte(pte_val(pte) & ~PTE_TABLE_BIT)) +#define pmd_young(pmd) (pmd_val(pmd) & PMD_SECT_AF) + +#define __HAVE_ARCH_PMD_WRITE +#define pmd_write(pmd) (!(pmd_val(pmd) & PMD_SECT_RDONLY)) + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define pmd_trans_huge(pmd) (pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT)) +#define pmd_trans_splitting(pmd) (pmd_val(pmd) & PMD_SECT_SPLITTING) +#endif + +#define PMD_BIT_FUNC(fn,op) \ +static inline pmd_t pmd_##fn(pmd_t pmd) { pmd_val(pmd) op; return pmd; } + +PMD_BIT_FUNC(wrprotect, |= PMD_SECT_RDONLY); +PMD_BIT_FUNC(mkold, &= ~PMD_SECT_AF); +PMD_BIT_FUNC(mksplitting, |= PMD_SECT_SPLITTING); +PMD_BIT_FUNC(mkwrite, &= ~PMD_SECT_RDONLY); +PMD_BIT_FUNC(mkdirty, |= PMD_SECT_DIRTY); +PMD_BIT_FUNC(mkyoung, |= PMD_SECT_AF); + +#define pmd_mkhuge(pmd) (__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT)) + +#define pmd_pfn(pmd) (((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT) +#define pfn_pmd(pfn,prot) (__pmd(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))) +#define mk_pmd(page,prot) pfn_pmd(page_to_pfn(page),prot) + +/* represent a notpresent pmd by zero, this is used by pmdp_invalidate */ +#define pmd_mknotpresent(pmd) (__pmd(0)) + +static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) +{ + const pmdval_t mask = PMD_SECT_USER | PMD_SECT_XN | PMD_SECT_RDONLY | + PMD_SECT_VALID | PMD_SECT_NONE; + pmd_val(pmd) = (pmd_val(pmd) & ~mask) | (pgprot_val(newprot) & mask); + return pmd; +} + +static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, + pmd_t *pmdp, pmd_t pmd) +{ + BUG_ON(addr >= TASK_SIZE); + + /* create a faulting entry if PROT_NONE protected */ + if (pmd_val(pmd) & PMD_SECT_NONE) + pmd_val(pmd) &= ~PMD_SECT_VALID; + + *pmdp = __pmd(pmd_val(pmd) | PMD_SECT_nG); + flush_pmd_entry(pmdp); +} + +static inline int has_transparent_hugepage(void) +{ + return 1; +} + #endif /* __ASSEMBLY__ */ #endif /* _ASM_PGTABLE_3LEVEL_H */ diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h index 9bcd262..eaedce7 100644 --- a/arch/arm/include/asm/pgtable.h +++ b/arch/arm/include/asm/pgtable.h @@ -24,6 +24,9 @@ #include #include + +#include + #ifdef CONFIG_ARM_LPAE #include #else diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 99a1951..bdc62da 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -223,6 +223,12 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, #endif } +static inline void +tlb_remove_pmd_tlb_entry(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) +{ + tlb_add_flush(tlb, addr); +} + #define pte_free_tlb(tlb, ptep, addr) __pte_free_tlb(tlb, ptep, addr) #define pmd_free_tlb(tlb, pmdp, addr) __pmd_free_tlb(tlb, pmdp, addr) #define pud_free_tlb(tlb, pudp, addr) pud_free((tlb)->mm, pudp) diff --git a/arch/arm/include/asm/tlbflush.h b/arch/arm/include/asm/tlbflush.h index a3625d1..c374592 100644 --- a/arch/arm/include/asm/tlbflush.h +++ b/arch/arm/include/asm/tlbflush.h @@ -535,6 +535,8 @@ static inline void update_mmu_cache(struct vm_area_struct *vma, } #endif +#define update_mmu_cache_pmd(vma, address, pmd) do { } while (0) + #endif #endif /* CONFIG_MMU */ diff --git a/arch/arm/mm/fsr-3level.c b/arch/arm/mm/fsr-3level.c index e115fc7..ab4409a 100644 --- a/arch/arm/mm/fsr-3level.c +++ b/arch/arm/mm/fsr-3level.c @@ -9,7 +9,7 @@ static struct fsr_info fsr_info[] = { { do_page_fault, SIGSEGV, SEGV_MAPERR, "level 3 translation fault" }, { do_bad, SIGBUS, 0, "reserved access flag fault" }, { do_bad, SIGSEGV, SEGV_ACCERR, "level 1 access flag fault" }, - { do_bad, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 access flag fault" }, { do_bad, SIGBUS, 0, "reserved permission fault" }, { do_bad, SIGSEGV, SEGV_ACCERR, "level 1 permission fault" },