From patchwork Mon Aug 7 18:35:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109582 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1951882qge; Mon, 7 Aug 2017 11:37:40 -0700 (PDT) X-Received: by 10.98.131.141 with SMTP id h135mr1504015pfe.271.1502131060659; Mon, 07 Aug 2017 11:37:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131060; cv=none; d=google.com; s=arc-20160816; b=RU/jzulDfL8qRxchz56F4vmMUsIrj16GNktx9bmWv1KN1bVwutqWAtza24zgN5Ks42 zvQQg2dNPRGUTSde0FQTAuhBVHiTi0bxe9lM2QLQs8/pMGMbwHvB3yjqgiLcZqb8/m6V fn+8/ijpSYIqLriqUN7NMRCXasO4vQtXaIv3y02Qhv5Lw2tQdohOGlab8bSZx8ThFUCX 4UhHJT1Qb53CfJElv9ZjAHooB9LIq2ddhDkJum4g+ZGp+R37Qq6IlB36w/2Sa6F+KEE7 wkHgluFFhvoaCeejhyAHLf39hylp4Uq2Xjio9KukWArcd53c4pAizbYm4peYW7xv/exZ Umcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=J1QoV70zeK2xWn4Y/Ff1f+Pq9wWI5i3bZr/DV9XUGys=; b=N60snvfkWs7RB0JTmMsEl9gFVHdPFu9wSQJwMDEQI4vVJ838ChiuvTeVNrpPBnlKN+ 30d7qqCkpEg1S6bQyZyzdiHlJ130Cbh+R+SHmkiL3oI0hIhsWaONiP5vU4N1KLlCnRJ/ 5b+pzCVUUXfEuqjnKvbyCnYNEwJX5U6C2pXEKDE+LThEfT52XpgYeDGqW91RdFhHOE1t f4p1Sq1PvBq7w4gtuwO6n+TOqZNUxLxvOyUciwMCal8n1f7IFuzE8kvgxaG0+qA3STpw csnihtlJdMPV++/P4rLR0LNcACsy8Aj30U3adKg5Dg9gTqyd1qywrKlZrdLJ9KhHZtvc +MvA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 186si5049288pfu.378.2017.08.07.11.37.40; Mon, 07 Aug 2017 11:37:40 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751974AbdHGShi (ORCPT + 25 others); Mon, 7 Aug 2017 14:37:38 -0400 Received: from foss.arm.com ([217.140.101.70]:52482 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751426AbdHGShh (ORCPT ); Mon, 7 Aug 2017 14:37:37 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0317715B2; Mon, 7 Aug 2017 11:37:37 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E4E493F577; Mon, 7 Aug 2017 11:37:34 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 01/14] arm64: remove __die()'s stack dump Date: Mon, 7 Aug 2017 19:35:52 +0100 Message-Id: <1502130965-18710-2-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Our __die() implementation tries to dump the stack memory, in addition to a backtrace, which is problematic. For contemporary 16K stacks, this can be a lot of data, which can take a long time to dump, and can push other useful context out of the kernel's printk ringbuffer (and/or a user's scrollback buffer on an attached console). Additionally, the code implicitly assumes that the SP is on the task's stack, and tries to dump everything between the SP and the highest task stack address. When the SP points at an IRQ stack (or is corrupted), this makes the kernel attempt to dump vast amounts of VA space. With vmap'd stacks, this may result in erroneous accesses to peripherals. This patch removes the memory dump, leaving us to rely on the backtrace, and other means of dumping stack memory such as kdump. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: Laura Abbott Cc: James Morse Cc: Will Deacon --- arch/arm64/kernel/traps.c | 2 -- 1 file changed, 2 deletions(-) -- 1.9.1 diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index c2a81bf..9633773 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -237,8 +237,6 @@ static int __die(const char *str, int err, struct pt_regs *regs) end_of_stack(tsk)); if (!user_mode(regs)) { - dump_mem(KERN_EMERG, "Stack: ", regs->sp, - THREAD_SIZE + (unsigned long)task_stack_page(tsk)); dump_backtrace(regs, tsk); dump_instr(KERN_EMERG, regs); } From patchwork Mon Aug 7 18:35:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109583 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952159qge; Mon, 7 Aug 2017 11:37:56 -0700 (PDT) X-Received: by 10.98.194.88 with SMTP id l85mr1583347pfg.252.1502131076643; Mon, 07 Aug 2017 11:37:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131076; cv=none; d=google.com; s=arc-20160816; b=jgcpQPU7H1OYXz6a3CIU7vKt4CnzGv0bVfag306kFhOL5iFt94pmuIVOe+mMm5XO+/ 9HSrtaHWuPtO18KtJISb6RKjapkiIBOWGo68nrrEgaUiyN1qUaQ1AbLyQILbrgx4P6nw dnaJVaDmKDEUdinuTz1k4HYbzblZFjrtRgrSzwzcQy7pPQnB0prSwZRbuRVJIPhyDZz4 3B1YFqNNP3/f1GJ2AEr0KALVa0OuW74c1gMqh7YFgAcuVjJTyQ+O3v7Dig60Gc8C3A3i IJfRVLtZQ9yJconGtci0gd2fuQKeyIdsr3gzNhbStyIkU8VKCUHYyEM3MzANqJAE0ScG Ut+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=F463WU8StH6ifPP6iNkZUVzkM6XvhR43pwP5TFXKUtw=; b=MD1NSp2Mvf/k+QdZv3UpL695+BnRsx9mCxofSCzfTDKfQm+OuEJNrt/dUVlg8SOC6F CI3rfX9VnquzQkZNK+zn7lNfYX3vPTtoFMkSLrV9cwCu4zbdt5pvl6mvPvriccPY3Rry 6sdN8Fn6dE+FcKPz24uWFVvAIbcR9CUxkAcQBMoMDNjra7lH0W2XXXlQna6iUieRdibB c53o+bXBcv2FyhRTa8TiQKeMdQnlWoKQpb/T4ep0QQDzIm4aVrnTRVm6gXIbTtMHClga E93I17K0yMB5aVOreHRUV9xecKlBgZD+Ibickq2lk2HmH7gIcjpv4KlCz6kFKV4/RSfH kUXA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l14si3174992pfb.161.2017.08.07.11.37.56; Mon, 07 Aug 2017 11:37:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752009AbdHGShx (ORCPT + 25 others); Mon, 7 Aug 2017 14:37:53 -0400 Received: from foss.arm.com ([217.140.101.70]:52506 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751508AbdHGShw (ORCPT ); Mon, 7 Aug 2017 14:37:52 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 633E015BE; Mon, 7 Aug 2017 11:37:52 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 508AA3F577; Mon, 7 Aug 2017 11:37:50 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 02/14] fork: allow arch-override of VMAP stack alignment Date: Mon, 7 Aug 2017 19:35:53 +0100 Message-Id: <1502130965-18710-3-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In some cases, an architecture might wish its stacks to be aligned to a boundary larger than THREAD_SIZE. For example, using an alignment of double THREAD_SIZE can allow for stack overflows smaller than THREAD_SIZE to be detected by checking a single bit of the stack pointer. This patch allows architectures to override the alignment of VMAP'd stacks, by defining THREAD_ALIGN. Where not defined, this defaults to THREAD_SIZE, as is the case today. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon Cc: linux-kernel@vger.kernel.org --- kernel/fork.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -- 1.9.1 diff --git a/kernel/fork.c b/kernel/fork.c index 17921b0..696d692 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -217,7 +217,10 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node) return s->addr; } - stack = __vmalloc_node_range(THREAD_SIZE, THREAD_SIZE, +#ifndef THREAD_ALIGN +#define THREAD_ALIGN THREAD_SIZE +#endif + stack = __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN, VMALLOC_START, VMALLOC_END, THREADINFO_GFP, PAGE_KERNEL, From patchwork Mon Aug 7 18:35:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109584 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952544qge; Mon, 7 Aug 2017 11:38:19 -0700 (PDT) X-Received: by 10.99.3.198 with SMTP id 189mr1400780pgd.49.1502131099373; Mon, 07 Aug 2017 11:38:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131099; cv=none; d=google.com; s=arc-20160816; b=u3bb8FqG2MYAH5GK3lVD7f4ZeeRpg7tO71KETQPN3Lix+sQNF4gLNXBGyKUUqj5FKk GDKYvxtMycqeK4xuP3p5iZbyVsGMaAmFRQbBuGQvXEIylgmrdoBPhKEwDkKc1zE6QWAB MPPn1vC/oyrUREdzft/79n22QBPiFV7iozO+Wvj0OEDo+avFE3z7TwDZVBi3m4WM1SwO fvZVURLtH7rE606fOgfvhgEfqs36226mRls2LnNkCW9Wf3V2acM+sS0mMROgi1X+FqH9 lquU2X6iifNP/9UTAcQR7FlLabHglJAJSFD6aJga47knt2hpFH8W5guKyNvvDq9h5qUs UlPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Yo3rPzh4yZfGN5NUqBLCmudZL0zbNHrh2Nj3s7Oyebc=; b=DVc1yUXpoTE9nhBa9FfxRz2D5iLmwVZqjwpo7OGApcEk90QNHXY3+sxUGTFgmqdAsA 7LtU2WCArzAItRozvwPASICJYH/Ip7WMn/fqYMFs7x2iPkd/Ze2heJe0tQLYabjeXAqs pRvnW7yH4BIgp21d0VX4hxKrRAOsY1QTUekBmWd5p3+M3wjC+NOkd7w+hMkwLbiyO0Hj R2GWcByfwJBz3jAyua0fs9b4y1cqsMLxrrCE8VTd70QDol6eXYUTN4arGu/rTHsUBMcH ySpudS+OGG91KQZouKakNyivskV8PjoXUgIIM6E2nDKZJDNQ1dqYUE9vVSG0sr2Cs+JW BvQg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l14si3174992pfb.161.2017.08.07.11.38.19; Mon, 07 Aug 2017 11:38:19 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752048AbdHGSiQ (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:16 -0400 Received: from foss.arm.com ([217.140.101.70]:52528 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751900AbdHGSiP (ORCPT ); Mon, 7 Aug 2017 14:38:15 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1E8F71610; Mon, 7 Aug 2017 11:38:15 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0C2263F577; Mon, 7 Aug 2017 11:38:12 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 03/14] arm64: kernel: remove {THREAD,IRQ_STACK}_START_SP Date: Mon, 7 Aug 2017 19:35:54 +0100 Message-Id: <1502130965-18710-4-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Ard Biesheuvel For historical reasons, we leave the top 16 bytes of our task and IRQ stacks unused, a practice used to ensure that the SP can always be masked to find the base of the current stack (historically, where thread_info could be found). However, this is not necessary, as: * When an exception is taken from a task stack, we decrement the SP by S_FRAME_SIZE and stash the exception registers before we compare the SP against the task stack. In such cases, the SP must be at least S_FRAME_SIZE below the limit, and can be safely masked to determine whether the task stack is in use. * When transitioning to an IRQ stack, we'll place a dummy frame onto the IRQ stack before enabling asynchronous exceptions, or executing code we expect to trigger faults. Thus, if an exception is taken from the IRQ stack, the SP must be at least 16 bytes below the limit. * We no longer mask the SP to find the thread_info, which is now found via sp_el0. Note that historically, the offset was critical to ensure that cpu_switch_to() found the correct stack for new threads that hadn't yet executed ret_from_fork(). Given that, this initial offset serves no purpose, and can be removed. This brings us in-line with other architectures (e.g. x86) which do not rely on this masking. Signed-off-by: Ard Biesheuvel [Mark: rebase, kill THREAD_START_SP, commit msg additions] Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/irq.h | 5 ++--- arch/arm64/include/asm/processor.h | 2 +- arch/arm64/include/asm/thread_info.h | 1 - arch/arm64/kernel/entry.S | 2 +- arch/arm64/kernel/smp.c | 2 +- 5 files changed, 5 insertions(+), 7 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h index 8ba89c4..1ebe202 100644 --- a/arch/arm64/include/asm/irq.h +++ b/arch/arm64/include/asm/irq.h @@ -2,7 +2,6 @@ #define __ASM_IRQ_H #define IRQ_STACK_SIZE THREAD_SIZE -#define IRQ_STACK_START_SP THREAD_START_SP #ifndef __ASSEMBLER__ @@ -26,9 +25,9 @@ static inline int nr_legacy_irqs(void) static inline bool on_irq_stack(unsigned long sp) { unsigned long low = (unsigned long)raw_cpu_ptr(irq_stack); - unsigned long high = low + IRQ_STACK_START_SP; + unsigned long high = low + IRQ_STACK_SIZE; - return (low <= sp && sp <= high); + return (low <= sp && sp < high); } static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 64c9e78..6687dd2 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -159,7 +159,7 @@ extern struct task_struct *cpu_switch_to(struct task_struct *prev, struct task_struct *next); #define task_pt_regs(p) \ - ((struct pt_regs *)(THREAD_START_SP + task_stack_page(p)) - 1) + ((struct pt_regs *)(THREAD_SIZE + task_stack_page(p)) - 1) #define KSTK_EIP(tsk) ((unsigned long)task_pt_regs(tsk)->pc) #define KSTK_ESP(tsk) user_stack_pointer(task_pt_regs(tsk)) diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 46c3b93..b29ab0e 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -30,7 +30,6 @@ #endif #define THREAD_SIZE 16384 -#define THREAD_START_SP (THREAD_SIZE - 16) #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 4ddb8d7..1c0f787 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -268,7 +268,7 @@ alternative_else_nop_endif cbnz x25, 9998f adr_this_cpu x25, irq_stack, x26 - mov x26, #IRQ_STACK_START_SP + mov x26, #IRQ_STACK_SIZE add x26, x25, x26 /* switch to the irq stack */ diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index dc66e6e..f13ddb2 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -154,7 +154,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) * page tables. */ secondary_data.task = idle; - secondary_data.stack = task_stack_page(idle) + THREAD_START_SP; + secondary_data.stack = task_stack_page(idle) + THREAD_SIZE; update_cpu_boot_status(CPU_MMU_OFF); __flush_dcache_area(&secondary_data, sizeof(secondary_data)); From patchwork Mon Aug 7 18:35:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109585 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952633qge; Mon, 7 Aug 2017 11:38:25 -0700 (PDT) X-Received: by 10.99.173.6 with SMTP id g6mr1413063pgf.1.1502131105407; Mon, 07 Aug 2017 11:38:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131105; cv=none; d=google.com; s=arc-20160816; b=sDC+YVFiPhV3Tk2z9rLGwTVNdXIixhWyKy8NhmZxd2DJVjTQ7yyq5sUFJs0Ow5ZjEi UPY4DpD5Sc/dpo1qxNO8yrB3kQn7yRdBC+SAz5dW5lbyfdTsnahrf3/OVqwpgAaXi5XN u190OlGPvZVqaUEopqdqiXJY8MgZwYWGs82lYNALse9blFHPHDuDaLraZHwTN5mCn0DC snBW1H1i+Rj6vchNAqDpBlebZMDzzLwAGaYis9KvuonxPJsQyJdQDg6FWx187mMjAoRU Q/i0/lPVmU5qcRcDugpYiGyK2ABr30BshApAFeRirPQaSbuINKaS/TXXPADFKN8PXDpf GlZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xZxIgwWSIG+gEpPyUVmKMU91P/VJjpt3tlccCECSW+k=; b=yJKISsgVl4HGTPASD+hTNs407uxySm+W+brL87+Ynpoxx9IpDGqGy2QUe9Bkm5n4mN EQ9nQiryhSFf5oB22PnvwCvEZ99w2GutU2wtBwRTmvlG5o3jaB2LijfBUHhMZd9TkHaR 7E81mEDlu25xPNETFt+HASXtVOOTNAlrJa3huGbIcwXru1h5AIotPNsUlcFDa2YgMRyG IFbDiTgP5qAORmPQPDk3m18drb04eaYGctnM/bw7DxvEAXlIMQxoC1ylXkczkthDOefO OD4mnPjm2pT5WH36808i9OXSwP2/r6PJKDkiACaDWUo6mNnqt0BiKjvX9v2XJ0cyzqV5 b49A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b3si4879932pgc.821.2017.08.07.11.38.25; Mon, 07 Aug 2017 11:38:25 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752075AbdHGSiV (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:21 -0400 Received: from foss.arm.com ([217.140.101.70]:52564 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751900AbdHGSiS (ORCPT ); Mon, 7 Aug 2017 14:38:18 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0325A165C; Mon, 7 Aug 2017 11:38:18 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E30133F577; Mon, 7 Aug 2017 11:38:15 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 04/14] arm64: factor out PAGE_* and CONT_* definitions Date: Mon, 7 Aug 2017 19:35:55 +0100 Message-Id: <1502130965-18710-5-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Some headers rely on PAGE_* definitions from , but cannot include this due to potential circular includes. For example, a number of definitions in rely on PAGE_SHIFT, and includes . This requires users of these definitions to include both headers, which is fragile and error-prone. This patch ameliorates matters by moving the basic definitions out to a new header, . Both and are updated to include this, avoiding this fragility, and avoiding the possibility of circular include dependencies. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/memory.h | 1 + arch/arm64/include/asm/page-def.h | 34 ++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/page.h | 12 +----------- 3 files changed, 36 insertions(+), 11 deletions(-) create mode 100644 arch/arm64/include/asm/page-def.h -- 1.9.1 diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 32f827233..77d55dc 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -25,6 +25,7 @@ #include #include #include +#include #include /* diff --git a/arch/arm64/include/asm/page-def.h b/arch/arm64/include/asm/page-def.h new file mode 100644 index 0000000..01591a2 --- /dev/null +++ b/arch/arm64/include/asm/page-def.h @@ -0,0 +1,34 @@ +/* + * Based on arch/arm/include/asm/page.h + * + * Copyright (C) 1995-2003 Russell King + * Copyright (C) 2017 ARM Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ +#ifndef __ASM_PAGE_DEF_H +#define __ASM_PAGE_DEF_H + +#include + +/* PAGE_SHIFT determines the page size */ +/* CONT_SHIFT determines the number of pages which can be tracked together */ +#define PAGE_SHIFT CONFIG_ARM64_PAGE_SHIFT +#define CONT_SHIFT CONFIG_ARM64_CONT_SHIFT +#define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT) +#define PAGE_MASK (~(PAGE_SIZE-1)) + +#define CONT_SIZE (_AC(1, UL) << (CONT_SHIFT + PAGE_SHIFT)) +#define CONT_MASK (~(CONT_SIZE-1)) + +#endif /* __ASM_PAGE_DEF_H */ diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index 8472c6d..60d02c8 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -19,17 +19,7 @@ #ifndef __ASM_PAGE_H #define __ASM_PAGE_H -#include - -/* PAGE_SHIFT determines the page size */ -/* CONT_SHIFT determines the number of pages which can be tracked together */ -#define PAGE_SHIFT CONFIG_ARM64_PAGE_SHIFT -#define CONT_SHIFT CONFIG_ARM64_CONT_SHIFT -#define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT) -#define PAGE_MASK (~(PAGE_SIZE-1)) - -#define CONT_SIZE (_AC(1, UL) << (CONT_SHIFT + PAGE_SHIFT)) -#define CONT_MASK (~(CONT_SIZE-1)) +#include #ifndef __ASSEMBLY__ From patchwork Mon Aug 7 18:35:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109586 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952647qge; Mon, 7 Aug 2017 11:38:26 -0700 (PDT) X-Received: by 10.99.95.200 with SMTP id t191mr1361743pgb.237.1502131105941; Mon, 07 Aug 2017 11:38:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131105; cv=none; d=google.com; s=arc-20160816; b=F+eKe9EuKE/AUJE06mD9Enpa3z0Eem2kveuxNLwuA337wH0BS83B+lVhuJqRxC1ERS EemHcuGaWpJzTdQP7nmFk2nN8jmgLjhnof8Lz/7kmhV6Euv73gfyLbrXtOYJdnQ1Aurf 7GNTKRET3kn7d7Aqu1PsSgMJ+swJYLUy2hotbLPZMpRVqZ0Wj0Nx+L3FgJqtZ3sVOIsB p6+ksIit3KbdFN+U23A6NggaHHl35vS5HzJS++2OeirwFCjW1DmHU7uzxoOkn15HmVfK zO5TxjNUxfGF4pLFS8bppAnf4rNqY2i5Kq9rsV0tnYssX3/jIBXOJm/eDp46cYdLB37C IyAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=2XsXVAq47T0IHR4AfXhnYMH+4vq/hKa7nsX+P2Jtqy0=; b=KZkp4K3HrHi6/P1AImZQY6voOmBPkGgxJxYxrrIiLIedP2EI655qtmpL/bTk87sd4S Yrs6qxbAQ3A17WH0pOTy8WPK7XZustevd4i2QSIcftSd6eooT1y56k0TjVRr0E92WO0m wtORW1FykJPr7EX33siZYvUY2vtWKpO954dRllH2Uf35hTasgBLCJMYMaTaptQg9tsr0 +zAit5mgokcx2gQos6gBlUUBkyoJwfb/L7+yqqi5bZIKFJO0OZeiUcE1ik9y/o/8NwCm d1WvH6UavekqvbczF0a0pPYhm1DvfArNjEd4QmIsYBnsIOLzfTAkuZU9eSFokeh4qPMe EKhw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b3si4879932pgc.821.2017.08.07.11.38.25; Mon, 07 Aug 2017 11:38:25 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752101AbdHGSiX (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:23 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52584 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752069AbdHGSiV (ORCPT ); Mon, 7 Aug 2017 14:38:21 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AD46E165D; Mon, 7 Aug 2017 11:38:20 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id BDE323F577; Mon, 7 Aug 2017 11:38:18 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 05/14] arm64: clean up THREAD_* definitions Date: Mon, 7 Aug 2017 19:35:56 +0100 Message-Id: <1502130965-18710-6-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently we define THREAD_SIZE and THREAD_SIZE order separately, with the latter dependent on particular CONFIG_ARM64_*K_PAGES definitions. This is somewhat opaque, and will get in the way of future modifications to THREAD_SIZE. This patch cleans this up, defining both in terms of a common THREAD_SHIFT, and using PAGE_SHIFT to calculate THREAD_SIZE_ORDER, rather than using a number of definitions dependent on config symbols. Subsequent patches will make use of this to alter the stack size used in some configurations. At the same time, these are moved into , which will avoid circular include issues in subsequent patches. To ensure that existing code isn't adversely affected, is updated to transitively include these definitions. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/memory.h | 8 ++++++++ arch/arm64/include/asm/thread_info.h | 9 +-------- 2 files changed, 9 insertions(+), 8 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 77d55dc..8ab4774 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -102,6 +102,14 @@ #define KASAN_SHADOW_SIZE (0) #endif +#define THREAD_SHIFT 14 + +#if THREAD_SHIFT >= PAGE_SHIFT +#define THREAD_SIZE_ORDER (THREAD_SHIFT - PAGE_SHIFT) +#endif + +#define THREAD_SIZE (UL(1) << THREAD_SHIFT) + /* * Memory types available. */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index b29ab0e..aa04b73 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -23,18 +23,11 @@ #include -#ifdef CONFIG_ARM64_4K_PAGES -#define THREAD_SIZE_ORDER 2 -#elif defined(CONFIG_ARM64_16K_PAGES) -#define THREAD_SIZE_ORDER 0 -#endif - -#define THREAD_SIZE 16384 - #ifndef __ASSEMBLY__ struct task_struct; +#include #include #include From patchwork Mon Aug 7 18:35:57 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109587 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952692qge; Mon, 7 Aug 2017 11:38:30 -0700 (PDT) X-Received: by 10.98.245.20 with SMTP id n20mr1584215pfh.46.1502131110030; Mon, 07 Aug 2017 11:38:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131110; cv=none; d=google.com; s=arc-20160816; b=g0QIEoB487OT9wpbXSD24F+cXFr3nKdHzubrUW7hex3+K09VE1erwZyz2TGPjZ8ZCh hHmE2XtLDnkKMwfPFiZ/S9hS1T4VyX8qK0lRda7D65BiLxinlJIGvyS8GELEVa5jaz36 llBJweIjizGgJS90n+4PTQhaC0MFee0RhWsS1Zhe/EggQcnUUXsB8GSpWecUcCIRtagN UvsY0IlUql328eukA4ihZJtslpeZjqgXkt2Red20NHZw9Klub/aUGVNyhucBe4dDz/05 +q5Rw9+vtf/0VuEVjA/aBILwjAdvRddE/QE/uP20uWlXHuEXRsvFEeggOZYjEjecqzG+ f8tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=rpxYGujq7JZgNvn32iBo2ddGsB4dn4smHUwCuhgxPEw=; b=a1FeCL7nuXzvDqqOflrXTkQ4X93ZdSKOMxdZbl00pPqSRkZIYzZsZcJqV09plJWI28 e0wKAgNVXlVuiDXyLq8grvuEtsez7xyD8hpzJVbhP9i4rFdPr8aNnKrK/JJR9t2r58LQ YN6e1tHC5O2ajwU7goKT0UP1awiLx4ujEjD8qArEAJq5vSLsE5T9A6HaoEM3A6pJuCG2 3VYGEFt68mkozW7x7mWUuDoSDpEFNkCt9pBA4tcJG5Q8UmEwxbuXIR3vV8dImHiLAQ9E eoWzYlxyfNXzDQPhSvDO3wKfUW25uRBkuTLoZL9lDufKaFYTd0QpOWoDyeM055o6GqBq zsjQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b3si4879932pgc.821.2017.08.07.11.38.29; Mon, 07 Aug 2017 11:38:30 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752123AbdHGSi1 (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:27 -0400 Received: from foss.arm.com ([217.140.101.70]:52610 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752069AbdHGSiX (ORCPT ); Mon, 7 Aug 2017 14:38:23 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7C850168F; Mon, 7 Aug 2017 11:38:23 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 69F033F577; Mon, 7 Aug 2017 11:38:21 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 06/14] arm64: clean up irq stack definitions Date: Mon, 7 Aug 2017 19:35:57 +0100 Message-Id: <1502130965-18710-7-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Before we add yet another stack to the kernel, it would be nice to ensure that we consistently organise stack definitions and related helper functions. This patch moves the basic IRQ stack defintions to to live with their task stack counterparts. Helpers used for unwinding are moved into , where subsequent patches will add helpers for other stacks. Includes are fixed up accordingly. This patch is a pure refactoring -- there should be no functional changes as a result of this patch. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/irq.h | 24 ------------------------ arch/arm64/include/asm/memory.h | 2 ++ arch/arm64/include/asm/stacktrace.h | 25 ++++++++++++++++++++++++- arch/arm64/kernel/ptrace.c | 1 + 4 files changed, 27 insertions(+), 25 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h index 1ebe202..5e6f772 100644 --- a/arch/arm64/include/asm/irq.h +++ b/arch/arm64/include/asm/irq.h @@ -1,20 +1,12 @@ #ifndef __ASM_IRQ_H #define __ASM_IRQ_H -#define IRQ_STACK_SIZE THREAD_SIZE - #ifndef __ASSEMBLER__ -#include -#include - #include -#include struct pt_regs; -DECLARE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack); - extern void set_handle_irq(void (*handle_irq)(struct pt_regs *)); static inline int nr_legacy_irqs(void) @@ -22,21 +14,5 @@ static inline int nr_legacy_irqs(void) return 0; } -static inline bool on_irq_stack(unsigned long sp) -{ - unsigned long low = (unsigned long)raw_cpu_ptr(irq_stack); - unsigned long high = low + IRQ_STACK_SIZE; - - return (low <= sp && sp < high); -} - -static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) -{ - unsigned long low = (unsigned long)task_stack_page(tsk); - unsigned long high = low + THREAD_SIZE; - - return (low <= sp && sp < high); -} - #endif /* !__ASSEMBLER__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 8ab4774..1fc2453 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -110,6 +110,8 @@ #define THREAD_SIZE (UL(1) << THREAD_SHIFT) +#define IRQ_STACK_SIZE THREAD_SIZE + /* * Memory types available. */ diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index 3bebab3..000e2418 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -16,7 +16,12 @@ #ifndef __ASM_STACKTRACE_H #define __ASM_STACKTRACE_H -struct task_struct; +#include +#include +#include + +#include +#include struct stackframe { unsigned long fp; @@ -31,4 +36,22 @@ extern void walk_stackframe(struct task_struct *tsk, struct stackframe *frame, int (*fn)(struct stackframe *, void *), void *data); extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); +DECLARE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack); + +static inline bool on_irq_stack(unsigned long sp) +{ + unsigned long low = (unsigned long)raw_cpu_ptr(irq_stack); + unsigned long high = low + IRQ_STACK_SIZE; + + return (low <= sp && sp < high); +} + +static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) +{ + unsigned long low = (unsigned long)task_stack_page(tsk); + unsigned long high = low + THREAD_SIZE; + + return (low <= sp && sp < high); +} + #endif /* __ASM_STACKTRACE_H */ diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c index baf0838..a9f8715 100644 --- a/arch/arm64/kernel/ptrace.c +++ b/arch/arm64/kernel/ptrace.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include From patchwork Mon Aug 7 18:35:58 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109595 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1954248qge; Mon, 7 Aug 2017 11:40:11 -0700 (PDT) X-Received: by 10.99.38.193 with SMTP id m184mr1386179pgm.322.1502131211346; Mon, 07 Aug 2017 11:40:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131211; cv=none; d=google.com; s=arc-20160816; b=doDD+vjkCDkus//UTO2gaxXZbsudtESeELWfYoaiJVzxZl4ItEHFQEU8ybcMkubwxf 3jsTSuEPvnCii5vGjvIqE/gCNZ1lmDOBBnVAJItvtylGhz1VsGJgeNfjLQ/4vvh9IuZ4 c6pXGRLTTnfg4l+X7eNe6K1KiNAjftJvJZ561d0K62fGMFLszHrEsmg96ZfDrLZ6a0Ej 72RMSutNPXtLUVaD3xILszpohssW7l7xkJhqSqy4ujWQOAO+KgosDcGD2F5kqrpjDd7y XVW46eNHNXAXoGVqgNEtEvLowu2Gk4SWZ0XdJ9Jtl1nuHgfRpCKPE/yE5qHG4u71DIUA axHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=HgCoibxZ1lZ9gSRAUQlmkEOdmHtP4bBo3iPD/JAEm/8=; b=ykBWHqAz7LOeagjGCK2zEKLuU8jIyfgIVSblve6k10f1EcwxFu7Z/vY0S/dSyOLYvI iszROeykFzSelrJQPEqUIai85j7sOgv2VgMF2P5yjZU4Jwoe1H/JEI/SIKdHMbnOCtJU mERXdPlkOrdSGYCsBrEF2LtddTlivJs7dv2OIkY9ySWY1W/FfK19BjE+j16owsDhAdpg KOc0LoLBFeA9jX9EmVvKNuOZC63AnzEh90y7dTM/Wu48YF8kCpW69bWoSnpYHQDOR19J v/EwfziZJuSrKirgNOO3X+F5VR1tNDf3sVedIRRxaNU+mG8nPkj+K3U1fxaLgppo5KhY GzOw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id c66si2595376pga.312.2017.08.07.11.40.11; Mon, 07 Aug 2017 11:40:11 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752330AbdHGSkI (ORCPT + 25 others); Mon, 7 Aug 2017 14:40:08 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52630 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752107AbdHGSi0 (ORCPT ); Mon, 7 Aug 2017 14:38:26 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 207E416A0; Mon, 7 Aug 2017 11:38:26 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 319073F577; Mon, 7 Aug 2017 11:38:24 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 07/14] arm64: move SEGMENT_ALIGN to Date: Mon, 7 Aug 2017 19:35:58 +0100 Message-Id: <1502130965-18710-8-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently we define SEGMENT_ALIGN directly in our vmlinux.lds.S. This is unfortunate, as the EFI stub currently open-codes the same number, and in future we'll want to fiddle with this. This patch moves the definition to our , where it can be used by both vmlinux.lds.S and the EFI stub code. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/memory.h | 19 +++++++++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 16 ---------------- 2 files changed, 19 insertions(+), 16 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 1fc2453..7fa6ad4 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -113,6 +113,25 @@ #define IRQ_STACK_SIZE THREAD_SIZE /* + * Alignment of kernel segments (e.g. .text, .data). + */ +#if defined(CONFIG_DEBUG_ALIGN_RODATA) +/* + * 4 KB granule: 1 level 2 entry + * 16 KB granule: 128 level 3 entries, with contiguous bit + * 64 KB granule: 32 level 3 entries, with contiguous bit + */ +#define SEGMENT_ALIGN SZ_2M +#else +/* + * 4 KB granule: 16 level 3 entries, with contiguous bit + * 16 KB granule: 4 level 3 entries, without contiguous bit + * 64 KB granule: 1 level 3 entry + */ +#define SEGMENT_ALIGN SZ_64K +#endif + +/* * Memory types available. */ #define MT_DEVICE_nGnRnE 0 diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 987a00e..7156538 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -72,22 +72,6 @@ PECOFF_FILE_ALIGNMENT = 0x200; #define PECOFF_EDATA_PADDING #endif -#if defined(CONFIG_DEBUG_ALIGN_RODATA) -/* - * 4 KB granule: 1 level 2 entry - * 16 KB granule: 128 level 3 entries, with contiguous bit - * 64 KB granule: 32 level 3 entries, with contiguous bit - */ -#define SEGMENT_ALIGN SZ_2M -#else -/* - * 4 KB granule: 16 level 3 entries, with contiguous bit - * 16 KB granule: 4 level 3 entries, without contiguous bit - * 64 KB granule: 1 level 3 entry - */ -#define SEGMENT_ALIGN SZ_64K -#endif - SECTIONS { /* From patchwork Mon Aug 7 18:35:59 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109588 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952753qge; Mon, 7 Aug 2017 11:38:34 -0700 (PDT) X-Received: by 10.84.195.131 with SMTP id j3mr1584805pld.147.1502131114656; Mon, 07 Aug 2017 11:38:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131114; cv=none; d=google.com; s=arc-20160816; b=l2vSKiyugb78jVNrLTghWcjj35MaodQSNBteJRzIOQfqf7VS4pyst3gc0YJnq9nje8 VeCouFibFTS5bhn3vFYFHj8UlareNeGHIfFED2Cc0xAq1yUrB+YKzLyrRmM3tzYXhiZs YQa12OfPJom5aYGnjgUEpUpbhQJSssPQXkTLx4F119ROaG9M63ZP2lgoiE2g3Dc5xNss t0mu93mZTc9HD7iMLRhh0gVvJxo8b4mcsgdu8ML2+0iOEzillseRCRV2JEt7UjHTFMwc pWXiSWL1T3oXcypi4PVRNL4edzHmQ4V41zqNhApWeQutYWVfIxtKg/9NrCjafmnznQBz X8Lg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=f4W+MVt+MWmgN7H5Ph3KbbnDHwzDRKdW0Fw6JePBmSc=; b=GIL/+sWRQYir3J76bzj5l4ww2OGeovKBG7Zrj5a5HhLE/vYdXc8IVnQDPgPmklb3b3 fn0jAy1+xBUJIw7QYaKQWDea6gzQ3bC/j5JmQF3Ii15aVoX7wlix5goHJ6sDXCPwwxJa Wo9Iq7ms1LZ9IWrDSBrv8pHQvC8ZIalIMBOL+1Fl+f2/717OndoCBanKzuklaEsiRJn6 2XuzL7q9tTKYMnRDtrXIVY2luD04gFGXvtSYg+aglf9494QB1QsT6VbVrLn2PZoE6OWG EggvWKJkOIWOSNRTuqZC7b6tYDPk/eE8mmo43gMp7wyaBt4vuGT+bIRokp5iAueEn2/O kBiw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 5si5029386pfi.554.2017.08.07.11.38.34; Mon, 07 Aug 2017 11:38:34 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752150AbdHGSib (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:31 -0400 Received: from foss.arm.com ([217.140.101.70]:52642 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752069AbdHGSi3 (ORCPT ); Mon, 7 Aug 2017 14:38:29 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0CD651715; Mon, 7 Aug 2017 11:38:29 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E4F3B3F577; Mon, 7 Aug 2017 11:38:26 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 08/14] efi/arm64: add EFI_KIMG_ALIGN Date: Mon, 7 Aug 2017 19:35:59 +0100 Message-Id: <1502130965-18710-9-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The EFI stub is intimately coupled with the kernel, and takes advantage of this by relocating the kernel at a weaker alignment than the documented boot protocol mandates. However, it does so by assuming it can align the kernel to the segment alignment, and assumes that this is 64K. In subsequent patches, we'll have to consider other details to determine this de-facto alignment constraint. This patch adds a new EFI_KIMG_ALIGN definition that will track the kernel's de-facto alignment requirements. Subsequent patches will modify this as required. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Matt Fleming Cc: Will Deacon --- arch/arm64/include/asm/efi.h | 3 +++ drivers/firmware/efi/libstub/arm64-stub.c | 6 ++++-- 2 files changed, 7 insertions(+), 2 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index 8f3043a..0e8cc3b 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -48,6 +49,8 @@ */ #define EFI_FDT_ALIGN SZ_2M /* used by allocate_new_fdt_and_exit_boot() */ +#define EFI_KIMG_ALIGN SEGMENT_ALIGN + /* on arm64, the FDT may be located anywhere in system RAM */ static inline unsigned long efi_get_max_fdt_addr(unsigned long dram_base) { diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index b4c2589..af6ae95 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c @@ -11,6 +11,7 @@ */ #include #include +#include #include #include @@ -81,9 +82,10 @@ efi_status_t handle_kernel_image(efi_system_table_t *sys_table_arg, /* * If CONFIG_DEBUG_ALIGN_RODATA is not set, produce a * displacement in the interval [0, MIN_KIMG_ALIGN) that - * is a multiple of the minimal segment alignment (SZ_64K) + * doesn't violate this kernel's de-facto alignment + * constraints. */ - u32 mask = (MIN_KIMG_ALIGN - 1) & ~(SZ_64K - 1); + u32 mask = (MIN_KIMG_ALIGN - 1) & ~(EFI_KIMG_ALIGN - 1); u32 offset = !IS_ENABLED(CONFIG_DEBUG_ALIGN_RODATA) ? (phys_seed >> 32) & mask : TEXT_OFFSET; From patchwork Mon Aug 7 18:36:00 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109589 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952818qge; Mon, 7 Aug 2017 11:38:39 -0700 (PDT) X-Received: by 10.98.210.70 with SMTP id c67mr1612808pfg.6.1502131119639; Mon, 07 Aug 2017 11:38:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131119; cv=none; d=google.com; s=arc-20160816; b=yRKxmlF8C+Xwav2IeRWI08jBIZ8fuQVFaFgwmJzU+H8wj32eQo3OgE8XXCRO75flzF +CVC9MMTVJwo6ukkaWxR1KE4ycJOR6TXE3iq1UhSfDg8WgRrcO1bfBUgGrrGPDkqT0+f iCMAn+CP4S4ju83+rKpWIkDOFJMLW0IzfbYnE2FodIFUaOI2VnKHjY+FQJK+VcGBlSPU YXuQE6pVaFefTwrXpmiPGiswSAgzjzjqH/XtiM0ZAhvHIK9dSqhdrtmN97XklJ3Ec8lK PrYD4QENVcknobf+s5rfVSeTTxrcegis0wVJnYoX1/P5pcr4A7TygVFX28SxqJnC3hb5 Q7jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=zvG8oXOIbEDvS1aFAx4TRiciEmSSZyyPxSIUEBM5C/4=; b=dfhWEDgVRx55nUD6qWCGeQT5wWQYkDvxkswvcotVdO21m7JGCKwzdx7cHaATot217j yYYq/fxbq/CfCS/4y5uPr8Im6nEsbIv79dtxSsg8uuLqn/6+eE0p7P+XKEnc2ZuENB2O KEuIMRo/ft8DSeDc6kleuksM+RhX6ttMLgaHJqwNonwcIFx04oVhMlmlFS0WO77W0Bvw AUCC+exccip3xpNz8iRRShkWYarEsYuYPY4uPUM8fy8pkvI9IqsCrvWyY4ID7nueXRyk pbJIdUtkz6v5zuVs9YBFAFOCtOAj0POue5nM8Spf8is+/IgfLNlm9zl93cLQ1o7rhAUX WOkA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q5si5011114pfj.271.2017.08.07.11.38.39; Mon, 07 Aug 2017 11:38:39 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752171AbdHGSif (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:35 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52680 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752010AbdHGSic (ORCPT ); Mon, 7 Aug 2017 14:38:32 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E01CC174E; Mon, 7 Aug 2017 11:38:31 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C3C053F577; Mon, 7 Aug 2017 11:38:29 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 09/14] arm64: factor out entry stack manipulation Date: Mon, 7 Aug 2017 19:36:00 +0100 Message-Id: <1502130965-18710-10-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In subsequent patches, we will detect stack overflow in our exception entry code, by verifying the SP after it has been decremented to make space for the exception regs. This verification code is small, and we can minimize its impact by placing it directly in the vectors. To avoid redundant modification of the SP, we also need to move the initial decrement of the SP into the vectors. As a preparatory step, this patch introduces kernel_ventry, which performs this decrement, and updates the entry code accordingly. Subsequent patches will fold SP verification into kernel_ventry. There should be no functional change as a result of this patch. Signed-off-by: Ard Biesheuvel [Mark: turn into prep patch, expand commit msg] Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/kernel/entry.S | 47 ++++++++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 21 deletions(-) -- 1.9.1 diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 1c0f787..bd3b6de 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -69,8 +69,13 @@ #define BAD_FIQ 2 #define BAD_ERROR 3 - .macro kernel_entry, el, regsize = 64 + .macro kernel_ventry label + .align 7 sub sp, sp, #S_FRAME_SIZE + b \label + .endm + + .macro kernel_entry, el, regsize = 64 .if \regsize == 32 mov w0, w0 // zero upper 32 bits of x0 .endif @@ -315,31 +320,31 @@ tsk .req x28 // current thread_info .align 11 ENTRY(vectors) - ventry el1_sync_invalid // Synchronous EL1t - ventry el1_irq_invalid // IRQ EL1t - ventry el1_fiq_invalid // FIQ EL1t - ventry el1_error_invalid // Error EL1t + kernel_ventry el1_sync_invalid // Synchronous EL1t + kernel_ventry el1_irq_invalid // IRQ EL1t + kernel_ventry el1_fiq_invalid // FIQ EL1t + kernel_ventry el1_error_invalid // Error EL1t - ventry el1_sync // Synchronous EL1h - ventry el1_irq // IRQ EL1h - ventry el1_fiq_invalid // FIQ EL1h - ventry el1_error_invalid // Error EL1h + kernel_ventry el1_sync // Synchronous EL1h + kernel_ventry el1_irq // IRQ EL1h + kernel_ventry el1_fiq_invalid // FIQ EL1h + kernel_ventry el1_error_invalid // Error EL1h - ventry el0_sync // Synchronous 64-bit EL0 - ventry el0_irq // IRQ 64-bit EL0 - ventry el0_fiq_invalid // FIQ 64-bit EL0 - ventry el0_error_invalid // Error 64-bit EL0 + kernel_ventry el0_sync // Synchronous 64-bit EL0 + kernel_ventry el0_irq // IRQ 64-bit EL0 + kernel_ventry el0_fiq_invalid // FIQ 64-bit EL0 + kernel_ventry el0_error_invalid // Error 64-bit EL0 #ifdef CONFIG_COMPAT - ventry el0_sync_compat // Synchronous 32-bit EL0 - ventry el0_irq_compat // IRQ 32-bit EL0 - ventry el0_fiq_invalid_compat // FIQ 32-bit EL0 - ventry el0_error_invalid_compat // Error 32-bit EL0 + kernel_ventry el0_sync_compat // Synchronous 32-bit EL0 + kernel_ventry el0_irq_compat // IRQ 32-bit EL0 + kernel_ventry el0_fiq_invalid_compat // FIQ 32-bit EL0 + kernel_ventry el0_error_invalid_compat // Error 32-bit EL0 #else - ventry el0_sync_invalid // Synchronous 32-bit EL0 - ventry el0_irq_invalid // IRQ 32-bit EL0 - ventry el0_fiq_invalid // FIQ 32-bit EL0 - ventry el0_error_invalid // Error 32-bit EL0 + kernel_ventry el0_sync_invalid // Synchronous 32-bit EL0 + kernel_ventry el0_irq_invalid // IRQ 32-bit EL0 + kernel_ventry el0_fiq_invalid // FIQ 32-bit EL0 + kernel_ventry el0_error_invalid // Error 32-bit EL0 #endif END(vectors) From patchwork Mon Aug 7 18:36:01 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109590 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952840qge; Mon, 7 Aug 2017 11:38:41 -0700 (PDT) X-Received: by 10.84.177.131 with SMTP id x3mr1633008plb.280.1502131121591; Mon, 07 Aug 2017 11:38:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131121; cv=none; d=google.com; s=arc-20160816; b=lFlttdM6P7GB5CqRN4V33BKy1hEb7X1xMSRt/61pa5tOB0MWCd15sOixE5Y2AoPQYT FLXAZQpmclesO/vRJU39slWdDiJaMT4GNw+0tYyaw0RpWM29P2CV1XGBaTtKR1ejjoKL YJhbLlsoQlwAzO1A4dvGdTH7wIDJVMAPETwSxlB/HuvfN2LG6RUEaiG7ExDoabfhrUf6 zn8vKjov1q9VP0b6kcJCcXF+/4wkrJl/WfYDX91XUWGWCyAa5YWV8jFq0bBimrk9uGha GQMf4myqyWuewcMeUf2vW60HRWnqnkanLF0Sd1JXA45tQbm6sT3hYohYTKDE2OevOmp2 JsqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=UAxKEONwm0A/qq0dUNWzKMWxikXjDQNpuPl7WOGGyN4=; b=UpDp97x4rLIZRb8xZFPCt+0U+1md6zjtlogZGN8UhbsmJukhKu8AOVuWI6ssaYnboc +ctrZ/SIgqfbOHTHPL7/7NtUdyGVIG0NH34BU/ZBk7GwyeuU0LVSL/2bDZz6jhXDSKVE HIaADEj4QGtILTOBTBbWvbw3oIr/xnXzawV6WK5w6fq5Vb3th5HY3zCx2TmSGr6YO4DQ zlPTXh19seLlYl0DMRYUq7elkpdvzI5U5wsWhgXDm6vJ5mIUqx6i9CTgx3EkW91oJvI8 weEqNYFon8CqwP9qK+VkEI5B3VsHm+qg7QEpehOvnHrVxS8CyhjKERJpOxrnWbjf0rrQ lxtA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q5si5011114pfj.271.2017.08.07.11.38.41; Mon, 07 Aug 2017 11:38:41 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752012AbdHGSih (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:37 -0400 Received: from foss.arm.com ([217.140.101.70]:52686 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752168AbdHGSif (ORCPT ); Mon, 7 Aug 2017 14:38:35 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BDB9C1991; Mon, 7 Aug 2017 11:38:34 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AA67D3F577; Mon, 7 Aug 2017 11:38:32 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 10/14] arm64: assembler: allow adr_this_cpu to use the stack pointer Date: Mon, 7 Aug 2017 19:36:01 +0100 Message-Id: <1502130965-18710-11-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Ard Biesheuvel Given that adr_this_cpu already requires a temp register in addition to the destination register, tweak the instruction sequence so that sp may be used as well. This will simplify switching to per-cpu stacks in subsequent patches. While this limits the range of adr_this_cpu, to +/-4GiB, we don't currently use adr_this_cpu in modules, and this is not problematic for the main kernel image. Signed-off-by: Ard Biesheuvel [Mark: add more commit text] Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/assembler.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 610a420..4775af5 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -235,7 +235,8 @@ * @tmp: scratch register */ .macro adr_this_cpu, dst, sym, tmp - adr_l \dst, \sym + adrp \tmp, \sym + add \dst, \tmp, #:lo12:\sym mrs \tmp, tpidr_el1 add \dst, \dst, \tmp .endm From patchwork Mon Aug 7 18:36:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109591 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952871qge; Mon, 7 Aug 2017 11:38:43 -0700 (PDT) X-Received: by 10.98.69.207 with SMTP id n76mr1569158pfi.28.1502131123845; Mon, 07 Aug 2017 11:38:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131123; cv=none; d=google.com; s=arc-20160816; b=hMT7hAE867clQm/ultAwV3PP1lqTFa0ODfqFja2cCZy+dusI8fk8b7LA7e7aSa9Dwc jPCd7xwB3CW1rAboDq4DjnjXXNbQ6Rf+KBiol8CBQGeFbckJ3iJoaIJihCz3nPBICdaA J91XE+Ya+jsXrHo0IUsNcgOoJjJY62qU2BN+ZEFpzstfNe43YkL3p34umFUjUt9SAjhs nheRO18Tc/86/uOLG4lVauCl3fjpaLY/Q35oCLq54Z0hutsSLghpzgnfOvW2tnHtpgRg X9LBqCKOUTkZAzMrOPyX3gDLN8TKIQPVwXlUyN9RcalY1sJobjMVY482uAlRj7V4Dwr0 jsHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=GZ9B9PYcPSHOZtzWw+EqpyPGwfmf21RI5fnE0s8g3Gs=; b=S4vcrd2K8sHcfC/ZZ7ZGN00x4p9Ux0c69opW++KV9vYoveqFJKz3TsIpSgRTmcEBqO fXf4MZth+sptYCbHGUDyZhUayMC/7aFJZKP/RGkCapXTtSCinLIR6J9AvABNOMdCddj2 aA2Kher58DugABzX6V5RYEdU7/H/8R57RLVHxY9OWPu3v+lQobPSjwKHCdbvNlcbUi5B 9D/cC50c9XKNKVJeXlOiPYh8RfUBk8Q4U/GN8R8GOAPkHV3OCJ/wubizVPglmcQV5mhg hnjMzpciGd2RZTAKpASKAZ6M5PZVeshsItBlizTg/qi4fg78HNgP7eZI7gLCEwUm7Z8a jQuA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g6si5609578plk.775.2017.08.07.11.38.43; Mon, 07 Aug 2017 11:38:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752207AbdHGSik (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:40 -0400 Received: from foss.arm.com ([217.140.101.70]:52706 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752168AbdHGSii (ORCPT ); Mon, 7 Aug 2017 14:38:38 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ACD14199B; Mon, 7 Aug 2017 11:38:37 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9108F3F577; Mon, 7 Aug 2017 11:38:35 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 11/14] arm64: use an irq stack pointer Date: Mon, 7 Aug 2017 19:36:02 +0100 Message-Id: <1502130965-18710-12-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We allocate our IRQ stacks using a percpu array. This allows us to generate our IRQ stack pointers with adr_this_cpu, but bloats the kernel Image with the boot CPU's IRQ stack. Additionally, these are packed with other percpu variables, and aren't guaranteed to have guard pages. When we enable VMAP_STACK we'll want to vmap our IRQ stacks also, in order to provide guard pages and to permit more stringent alignment requirements. Doing so will require that we use a percpu pointer to each IRQ stack, rather than allocating a percpu IRQ stack in the kernel image. This patch updates our IRQ stack code to use a percpu pointer to the base of each IRQ stack. This will allow us to change the way the stack is allocated with minimal changes elsewhere. In some cases we may try to backtrace before the IRQ stack pointers are initialised, so on_irq_stack() is updated to account for this. In testing with cyclictest, there was no measureable difference between using adr_this_cpu (for irq_stack) and ldr_this_cpu (for irq_stack_ptr) in the IRQ entry path. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/stacktrace.h | 7 +++++-- arch/arm64/kernel/entry.S | 2 +- arch/arm64/kernel/irq.c | 10 ++++++++++ 3 files changed, 16 insertions(+), 3 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index 000e2418..4c68d8a 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -36,13 +36,16 @@ extern void walk_stackframe(struct task_struct *tsk, struct stackframe *frame, int (*fn)(struct stackframe *, void *), void *data); extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); -DECLARE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack); +DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); static inline bool on_irq_stack(unsigned long sp) { - unsigned long low = (unsigned long)raw_cpu_ptr(irq_stack); + unsigned long low = (unsigned long)raw_cpu_read(irq_stack_ptr); unsigned long high = low + IRQ_STACK_SIZE; + if (!low) + return false; + return (low <= sp && sp < high); } diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index bd3b6de..e5aa866 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -272,7 +272,7 @@ alternative_else_nop_endif and x25, x25, #~(THREAD_SIZE - 1) cbnz x25, 9998f - adr_this_cpu x25, irq_stack, x26 + ldr_this_cpu x25, irq_stack_ptr, x26 mov x26, #IRQ_STACK_SIZE add x26, x25, x26 diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index 2386b26..5141282 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -32,6 +32,7 @@ /* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */ DEFINE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack) __aligned(16); +DEFINE_PER_CPU(unsigned long *, irq_stack_ptr); int arch_show_interrupts(struct seq_file *p, int prec) { @@ -50,8 +51,17 @@ void __init set_handle_irq(void (*handle_irq)(struct pt_regs *)) handle_arch_irq = handle_irq; } +static void init_irq_stacks(void) +{ + int cpu; + + for_each_possible_cpu(cpu) + per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu); +} + void __init init_IRQ(void) { + init_irq_stacks(); irqchip_init(); if (!handle_arch_irq) panic("No interrupt controller found."); From patchwork Mon Aug 7 18:36:03 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109592 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952905qge; Mon, 7 Aug 2017 11:38:45 -0700 (PDT) X-Received: by 10.84.231.135 with SMTP id g7mr1606222plk.405.1502131125364; Mon, 07 Aug 2017 11:38:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131125; cv=none; d=google.com; s=arc-20160816; b=lD6pV9Uwz4vxUIShKh4ihbEPAM3wlG1oiY2gFrofo+sJKFc9n9KBs9T5EmVn63cyvc p2xg3ipqeyInBro95WEFHidI7mknOhiHr9sz4CaXXN52xa3fpoLzVcnWT+HOkCAbHADK v1pE3di6iIc71rHqJUtFq/hCNdgi6MQNIVZ7O+COWOGzxk9TbNbSfCoYV2oFDWYFxozN BFp3PkBV0+1/b4cTvfji5jPpcuDVWmr9ocsw79GvjRtZuMWpPRQY7mA48sN6AdeGfZeg 83b95iT7CEnQdva9igf68Zir2dLx1bhFl1S9954wKRgrPUj1RRCNWj/Mx9PXV8goscAB Tpxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=NeG4PQTODK1lNwSgI6qnp/s7JYl6vSSjq2bmHxXqwGY=; b=EqzmOuYaXZ1E537C8feiTon4RAL1h7zL0rMsKU+hsskGOess3cBk+WYJu+yM9lAkMQ 2dgXlB4q+3aHxHRdSrQjGK5hwaZcWTprNoFFr2UdmpBXME1gZIyMnnjVHJNFtCmLXMe5 D2vzksDJHGb/UVl84uwXxW5Kx4cI5Y3yB6OQGFR5wOUVvBSkq/Yv/gV/bWtYfANgc7Es DiMW4131tfzm0kuQMRt8Q+BJTXnRDkcime0jhNbO9d2OFzzfMHQnki9cqsJKntQz5OEu ILV3DqFXrSBhvPfasOUK7JDBF8OqeFZoTb+ezrla1yBVSum2mPDU5XWpz/qr3PVmZ+bZ wacA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o11si4965564pgc.125.2017.08.07.11.38.45; Mon, 07 Aug 2017 11:38:45 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752228AbdHGSim (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:42 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52726 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752202AbdHGSik (ORCPT ); Mon, 7 Aug 2017 14:38:40 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5F2721A09; Mon, 7 Aug 2017 11:38:40 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6FCD63F577; Mon, 7 Aug 2017 11:38:38 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 12/14] arm64: add basic VMAP_STACK support Date: Mon, 7 Aug 2017 19:36:03 +0100 Message-Id: <1502130965-18710-13-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This path enables arm64 to be built with vmap'd task and IRQ stacks. As vmap'd stacks are mapped at page granularity, stacks must be a multiple of PAGE_SIZE. This means that a 64K page kernel must use stacks of at least 64K in size. To minimize the increase in Image size, IRQ stacks are dynamically allocated at boot time, rather than embedding the boot CPU's IRQ stack in the kernel image. This patch was co-authored by Ard Biesheuvel and Mark Rutland. Signed-off-by: Ard Biesheuvel Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/efi.h | 7 ++++++- arch/arm64/include/asm/memory.h | 23 ++++++++++++++++++++++- arch/arm64/kernel/irq.c | 30 ++++++++++++++++++++++++++++-- arch/arm64/kernel/vmlinux.lds.S | 2 +- 5 files changed, 58 insertions(+), 5 deletions(-) -- 1.9.1 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index dfd9086..d66f9db 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -75,6 +75,7 @@ config ARM64 select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE + select HAVE_ARCH_VMAP_STACK select HAVE_ARM_SMCCC select HAVE_EBPF_JIT select HAVE_C_RECORDMCOUNT diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index 0e8cc3b..2b1e5de 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -49,7 +49,12 @@ */ #define EFI_FDT_ALIGN SZ_2M /* used by allocate_new_fdt_and_exit_boot() */ -#define EFI_KIMG_ALIGN SEGMENT_ALIGN +/* + * In some configurations (e.g. VMAP_STACK && 64K pages), stacks built into the + * kernel need greater alignment than we require the segments to be padded to. + */ +#define EFI_KIMG_ALIGN \ + (SEGMENT_ALIGN > THREAD_ALIGN ? SEGMENT_ALIGN : THREAD_ALIGN) /* on arm64, the FDT may be located anywhere in system RAM */ static inline unsigned long efi_get_max_fdt_addr(unsigned long dram_base) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 7fa6ad4..c5cd2c5 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -102,7 +102,17 @@ #define KASAN_SHADOW_SIZE (0) #endif -#define THREAD_SHIFT 14 +#define MIN_THREAD_SHIFT 14 + +/* + * VMAP'd stacks are allocated at page granularity, so we must ensure that such + * stacks are a multiple of page size. + */ +#if defined(CONFIG_VMAP_STACK) && (MIN_THREAD_SHIFT < PAGE_SHIFT) +#define THREAD_SHIFT PAGE_SHIFT +#else +#define THREAD_SHIFT MIN_THREAD_SHIFT +#endif #if THREAD_SHIFT >= PAGE_SHIFT #define THREAD_SIZE_ORDER (THREAD_SHIFT - PAGE_SHIFT) @@ -110,6 +120,17 @@ #define THREAD_SIZE (UL(1) << THREAD_SHIFT) +/* + * By aligning VMAP'd stacks to 2 * THREAD_SIZE, we can detect overflow by + * checking sp & (1 << THREAD_SHIFT), which we can do cheaply in the entry + * assembly. + */ +#ifdef CONFIG_VMAP_STACK +#define THREAD_ALIGN (2 * THREAD_SIZE) +#else +#define THREAD_ALIGN THREAD_SIZE +#endif + #define IRQ_STACK_SIZE THREAD_SIZE /* diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index 5141282..713561e 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -23,15 +23,15 @@ #include #include +#include #include #include #include #include +#include unsigned long irq_err_count; -/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */ -DEFINE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack) __aligned(16); DEFINE_PER_CPU(unsigned long *, irq_stack_ptr); int arch_show_interrupts(struct seq_file *p, int prec) @@ -51,6 +51,31 @@ void __init set_handle_irq(void (*handle_irq)(struct pt_regs *)) handle_arch_irq = handle_irq; } +#ifdef CONFIG_VMAP_STACK +static void init_irq_stacks(void) +{ + int cpu; + unsigned long *p; + + for_each_possible_cpu(cpu) { + /* + * To ensure that VMAP'd stack overflow detection works + * correctly, the IRQ stacks need to have the same + * alignment as other stacks. + */ + p = __vmalloc_node_range(IRQ_STACK_SIZE, THREAD_ALIGN, + VMALLOC_START, VMALLOC_END, + THREADINFO_GFP, PAGE_KERNEL, + 0, cpu_to_node(cpu), + __builtin_return_address(0)); + + per_cpu(irq_stack_ptr, cpu) = p; + } +} +#else +/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */ +DEFINE_PER_CPU_ALIGNED(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack); + static void init_irq_stacks(void) { int cpu; @@ -58,6 +83,7 @@ static void init_irq_stacks(void) for_each_possible_cpu(cpu) per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu); } +#endif void __init init_IRQ(void) { diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7156538..fe56c26 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -176,7 +176,7 @@ SECTIONS _data = .; _sdata = .; - RW_DATA_SECTION(L1_CACHE_BYTES, PAGE_SIZE, THREAD_SIZE) + RW_DATA_SECTION(L1_CACHE_BYTES, PAGE_SIZE, THREAD_ALIGN) /* * Data written with the MMU off but read with the MMU on requires From patchwork Mon Aug 7 18:36:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109593 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1952975qge; Mon, 7 Aug 2017 11:38:49 -0700 (PDT) X-Received: by 10.84.218.136 with SMTP id r8mr1601396pli.111.1502131129845; Mon, 07 Aug 2017 11:38:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131129; cv=none; d=google.com; s=arc-20160816; b=U6De8cpsgaDLAPwto2heuV0L0kwNFnzxHSaYTjBQ6r3VL4BvAa1c7sMjnTCajYgDQx 0vLcJJUI0mEhdtyi1U/K2FeRf7s0EsLUGN/jyna9barNvl/eXwWnJZ7R4oB73+LDaEbR ZW/Ye68n+KCO1kDHUjGqsLnOi3U8hi+Mkb1j6Ea+qtqfmTUmdvwuW4x9uDCIXUSFITSg 87Mmj4nbR8CcpAny+/py1zFS++OKBGwWI0lJWRPzFGmzraUS3M9eBRvCxoWerOU6nYNI WOnEj9/Jv6WI/AbaFqHxK3gUahNFvqR+GGnQes/S5uiBIrx057153wrW5efldqBwTAf1 +7+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=fwdU1Ev1PcwI6UO6/HQOFFQq7oSstlvEZbWRcJy4Na0=; b=x4FTF6NBqPD5hMkcWnG7vYFtFt+MMhu2uEtle8vKmLGq+KvaDsxfWKe2+sxGOUY8HX Yi89vUnI2U9KTuhBm9HUAZtEMQnbPeVQy11KAYOQZ/DyBGcIB1ivs074A8hbp1kVfpWB RrY45/ERfM1hLgjDyY57OU86t3ZxyrWurRR4lCPkvAGCVwXilOgq+qD+iNDStOlsR/Ce BJLx/rU4AQL6Kpd4j6Sch9g+B1GMNJUBxrza04X7RDxYk95iPFIVI+DGvdD7GRhCkxc1 icHdtUf4LijsxLqp2XGyZ3zURKJsI/1Y0iNzCHqSPjSsoZNJAtsCl8DL+itSXTxNndxQ Ubrw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b34si5637596pld.726.2017.08.07.11.38.49; Mon, 07 Aug 2017 11:38:49 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752253AbdHGSir (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:47 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52748 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752202AbdHGSin (ORCPT ); Mon, 7 Aug 2017 14:38:43 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 47AA91A25; Mon, 7 Aug 2017 11:38:43 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2B68F3F577; Mon, 7 Aug 2017 11:38:41 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 13/14] arm64: add on_accessible_stack() Date: Mon, 7 Aug 2017 19:36:04 +0100 Message-Id: <1502130965-18710-14-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Both unwind_frame() and dump_backtrace() try to check whether a stack address is sane to access, with very similar logic. Both will need updating in order to handle overflow stacks. Factor out this logic into a helper, so that we can avoid further duplication when we add overflow stacks. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/stacktrace.h | 16 ++++++++++++++++ arch/arm64/kernel/stacktrace.c | 7 +------ arch/arm64/kernel/traps.c | 3 +-- 3 files changed, 18 insertions(+), 8 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index 4c68d8a..92ddb6d 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -57,4 +57,20 @@ static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) return (low <= sp && sp < high); } +/* + * We can only safely access per-cpu stacks from current in a non-preemptible + * context. + */ +static inline bool on_accessible_stack(struct task_struct *tsk, unsigned long sp) +{ + if (on_task_stack(tsk, sp)) + return true; + if (tsk != current || preemptible()) + return false; + if (on_irq_stack(sp)) + return true; + + return false; +} + #endif /* __ASM_STACKTRACE_H */ diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c index 54f3463..d9b80eb 100644 --- a/arch/arm64/kernel/stacktrace.c +++ b/arch/arm64/kernel/stacktrace.c @@ -50,12 +50,7 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) if (!tsk) tsk = current; - /* - * Switching between stacks is valid when tracing current and in - * non-preemptible context. - */ - if (!(tsk == current && !preemptible() && on_irq_stack(fp)) && - !on_task_stack(tsk, fp)) + if (!on_accessible_stack(tsk, fp)) return -EINVAL; frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp)); diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 9633773..d01c598 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -193,8 +193,7 @@ void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk) if (in_entry_text(frame.pc)) { stack = frame.fp - offsetof(struct pt_regs, stackframe); - if (on_task_stack(tsk, stack) || - (tsk == current && !preemptible() && on_irq_stack(stack))) + if (on_accessible_stack(tsk, stack)) dump_mem("", "Exception stack", stack, stack + sizeof(struct pt_regs)); } From patchwork Mon Aug 7 18:36:05 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 109594 Delivered-To: patch@linaro.org Received: by 10.140.95.78 with SMTP id h72csp1953057qge; Mon, 7 Aug 2017 11:38:56 -0700 (PDT) X-Received: by 10.99.119.206 with SMTP id s197mr1404967pgc.439.1502131136220; Mon, 07 Aug 2017 11:38:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502131136; cv=none; d=google.com; s=arc-20160816; b=RM2JWo6V/JD6sSLhkXDtn4xvCNhlfcZbCt+Gp7sopuR4cSGHOVffgdBtVEI7albwOI jAoDqAtIwBqDGbbIYAxFaIpxjnKef/Q8rqnU0y8ujWTTUTzg7MvtB6Q1viVpnSeNSSIx I+F0NqxEQDNs+YwqrgNEi3+PjR7A4orisGM0KC74OWEBahFlSaj16LWWq8SzT9FIB4jX tD2e5C1bJZ2cw7Iq1OJebpLb1f61YH8CaidxSIBeyFXsy3Th4KwhXujUv4xut7QLMJTE lFTtoAv+cbTFUGghxcwSg1tff27zrECGiizZA4gnjSyiWm8nDApt8W6tJ7myTXVYaRVZ dN/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=n9ITr7PyeBdviPaZw7h8nVAL8WwHZZL+EyD7PJPXolY=; b=p9su3CVqk5cI/XneParmVPKHEWrZooP4PGSQ5pnCbRCpulybSGpz9T9+ySEVNpaNIg YcKIsyGMe36aS+DXMFdWJqCoUcUjfmnaWwZg695JTzGg2hzNy/5AKJHM/NgGUoZNb5NL imj0SkDrxuP95R4Rt4ylcBzkJmwjkEODhrSAzgxWiy+ElJV091c1slsyesAftu8w6O8K IcWZ5BHY4e7ST0NyqSvyD2YGUiyvcB7pUhGtqBUpPnZivuNJoz0JHJCW5oLT7Om0AuQb XB71ckJPaG8dLhAOT1qArQ2z4oW/IhZvBJbhSrRCdQlnata+njEPMwEuqSblF12QUivh 7I3A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l5si4869772pgc.954.2017.08.07.11.38.55; Mon, 07 Aug 2017 11:38:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752272AbdHGSiv (ORCPT + 25 others); Mon, 7 Aug 2017 14:38:51 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52766 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752231AbdHGSiq (ORCPT ); Mon, 7 Aug 2017 14:38:46 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 10D341A3B; Mon, 7 Aug 2017 11:38:46 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id F29C93F577; Mon, 7 Aug 2017 11:38:43 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCH 14/14] arm64: add VMAP_STACK overflow detection Date: Mon, 7 Aug 2017 19:36:05 +0100 Message-Id: <1502130965-18710-15-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> References: <1502130965-18710-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch adds stack overflow detection to arm64, usable when vmap'd stacks are in use. Overflow is detected in a small preamble executed for each exception entry, which checks whether there is enough space on the current stack for the general purpose registers to be saved. If there is not enough space, the overflow handler is invoked on a per-cpu overflow stack. This approach preserves the original exception information in ESR_EL1 (and where appropriate, FAR_EL1). Task and IRQ stacks are aligned to double their size, enabling overflow to be detected with a single bit test. For example, a 16K stack is aligned to 32K, ensuring that bit 14 of the SP must be zero. On an overflow (or underflow), this bit is flipped. Thus, overflow (of less than the size of the stack) can be detected by testing whether this bit is set. The overflow check is performed before any attempt is made to access the stack, avoiding recursive faults (and the loss of exception information these would entail). As logical operations cannot be performed on the SP directly, the SP is temporarily swapped with a general purpose register using arithmetic operations to enable the test to be performed. This gives us a useful error message on stack overflow, as can be trigger with the LKDTM overflow test: root@ribbensteg:/sys/kernel/debug/provoke-crash# echo OVERFLOW > DIRECT [ 116.249161] lkdtm: Performing direct entry OVERFLOW [ 116.254048] Insufficient stack space to handle exception! [ 116.254059] CPU: 4 PID: 2269 Comm: bash Not tainted 4.13.0-rc3-00020-g307fec7 #197 [ 116.266913] Hardware name: ARM Juno development board (r1) (DT) [ 116.272783] task: ffff800976bf0e00 task.stack: ffff00000d540000 [ 116.278660] PC is at recursive_loop+0x10/0x50 [ 116.282981] LR is at recursive_loop+0x34/0x50 [ 116.287300] pc : [] lr : [] pstate: 40000145 [ 116.294633] sp : ffff00000d53ff30 [ 116.297916] x29: ffff00000d540350 x28: ffff800976bf0e00 [ 116.303188] x27: ffff000008981000 x26: ffff000008f701f8 [ 116.308458] x25: ffff00000d543eb8 x24: ffff00000d543eb8 [ 116.313729] x23: ffff000008f6ff30 x22: 0000000000000009 [ 116.318999] x21: ffff800975c43000 x20: ffff000008f6ff80 [ 116.324269] x19: 0000000000000013 x18: 0000000000000010 [ 116.329539] x17: 0000ffffb24cf6a4 x16: ffff0000081fbc40 [ 116.334820] x15: 0000000000000006 x14: ffff000088fc637f [ 116.340099] x13: ffff000008fc638d x12: ffff000008ec2460 [ 116.345379] x11: ffff00000d543a30 x10: 0000000005f5e0ff [ 116.350659] x9 : 00000000ffffffd0 x8 : ffff00000d540770 [ 116.355939] x7 : 1313131313131313 x6 : 000000000000019c [ 116.361218] x5 : 0000000000000000 x4 : 0000000000000000 [ 116.366497] x3 : 0000000000000000 x2 : 0000000000000400 [ 116.371777] x1 : 0000000000000013 x0 : 0000000000000012 [ 116.377058] Task stack: [0xffff00000d540000..0xffff00000d544000] [ 116.383366] IRQ stack: [0xffff000008020000..0xffff000008024000] [ 116.389675] Overflow stack: [0xffff80097ffa54e0..0xffff80097ffa64e0] [ 116.395984] ESR: 0x96000047 -- DABT (current EL) [ 116.400569] FAR: 0xffff00000d53ff30 [ 116.404036] Kernel panic - not syncing: kernel stack overflow [ 116.409744] CPU: 4 PID: 2269 Comm: bash Not tainted 4.13.0-rc3-00020-g307fec7 #197 [ 116.417268] Hardware name: ARM Juno development board (r1) (DT) [ 116.423146] Call trace: [ 116.425587] [] dump_backtrace+0x0/0x268 [ 116.430955] [] show_stack+0x14/0x20 [ 116.435976] [] dump_stack+0x98/0xb8 [ 116.440997] [] panic+0x118/0x28c [ 116.445758] [] nmi_panic+0x6c/0x70 [ 116.450693] [] handle_bad_stack+0x118/0x128 [ 116.456401] Exception stack(0xffff80097ffa63a0 to 0xffff80097ffa64e0) [ 116.462799] 63a0: 0000000000000012 0000000000000013 0000000000000400 0000000000000000 [ 116.470585] 63c0: 0000000000000000 0000000000000000 000000000000019c 1313131313131313 [ 116.478372] 63e0: ffff00000d540770 00000000ffffffd0 0000000005f5e0ff ffff00000d543a30 [ 116.486157] 6400: ffff000008ec2460 ffff000008fc638d ffff000088fc637f 0000000000000006 [ 116.493943] 6420: ffff0000081fbc40 0000ffffb24cf6a4 0000000000000010 0000000000000013 [ 116.501730] 6440: ffff000008f6ff80 ffff800975c43000 0000000000000009 ffff000008f6ff30 [ 116.509516] 6460: ffff00000d543eb8 ffff00000d543eb8 ffff000008f701f8 ffff000008981000 [ 116.517302] 6480: ffff800976bf0e00 ffff00000d540350 ffff00000859779c ffff00000d53ff30 [ 116.525087] 64a0: ffff000008597778 0000000040000145 0000000000000000 0000000000000000 [ 116.532874] 64c0: 0001000000000000 0000000000000000 ffff00000d540350 ffff000008597778 [ 116.540660] [] __bad_stack+0x88/0x8c [ 116.545767] [] recursive_loop+0x10/0x50 [ 116.551132] [] recursive_loop+0x34/0x50 [ 116.556497] [] recursive_loop+0x34/0x50 [ 116.561862] [] recursive_loop+0x34/0x50 [ 116.567228] [] recursive_loop+0x34/0x50 [ 116.572592] [] recursive_loop+0x34/0x50 [ 116.577957] [] recursive_loop+0x34/0x50 [ 116.583322] [] recursive_loop+0x34/0x50 [ 116.588687] [] recursive_loop+0x34/0x50 [ 116.594051] [] recursive_loop+0x34/0x50 [ 116.599416] [] recursive_loop+0x34/0x50 [ 116.604781] [] recursive_loop+0x34/0x50 [ 116.610146] [] recursive_loop+0x34/0x50 [ 116.615511] [] recursive_loop+0x34/0x50 [ 116.620876] [] lkdtm_OVERFLOW+0x14/0x20 [ 116.626241] [] lkdtm_do_action+0x1c/0x24 [ 116.631693] [] direct_entry+0xe0/0x168 [ 116.636974] [] full_proxy_write+0x60/0xa8 [ 116.642511] [] __vfs_write+0x1c/0x118 [ 116.647704] [] vfs_write+0x9c/0x1a8 [ 116.652723] [] SyS_write+0x44/0xa0 [ 116.657655] Exception stack(0xffff00000d543ec0 to 0xffff00000d544000) [ 116.664053] 3ec0: 0000000000000001 000000001952d808 0000000000000009 0000000000000000 [ 116.671838] 3ee0: 0000000000000000 0000000000000000 0000ffffb24d6c6c 0dfefefefeff07ff [ 116.679624] 3f00: 0000000000000040 fefefefefefefeff 0000000019555b28 0000000000000008 [ 116.687411] 3f20: 0000000000000000 0000000000000018 ffffffffffffffff 00000ca9b8000000 [ 116.695196] 3f40: 0000000000000000 0000ffffb24cf6a4 0000ffffd8d00e40 0000000000000009 [ 116.702983] 3f60: 000000001952d808 0000ffffb25ad178 0000000000000009 0000000000000000 [ 116.710768] 3f80: 0000000000000001 00000000004c9c98 00000000004ca628 00000000004ed000 [ 116.718554] 3fa0: 00000000004ea8e0 0000ffffd8d00fe0 0000ffffb24d674c 0000ffffd8d00fe0 [ 116.726340] 3fc0: 0000ffffb2524fec 0000000060000000 0000000000000001 0000000000000040 [ 116.734125] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000ffffb2524fec [ 116.741912] [] el0_svc_naked+0x24/0x28 [ 116.747189] [<0000ffffb2524fec>] 0xffffb2524fec [ 116.751695] SMP: stopping secondary CPUs [ 116.755909] Kernel Offset: disabled [ 116.759375] CPU features: 0x002086 [ 116.762753] Memory Limit: none [ 116.765795] ---[ end Kernel panic - not syncing: kernel stack overflow This patch was co-authored by Ard Biesheuvel and Mark Rutland. Signed-off-by: Ard Biesheuvel Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/memory.h | 2 ++ arch/arm64/include/asm/stacktrace.h | 18 +++++++++++ arch/arm64/kernel/entry.S | 59 +++++++++++++++++++++++++++++++++++++ arch/arm64/kernel/traps.c | 39 ++++++++++++++++++++++++ 4 files changed, 118 insertions(+) -- 1.9.1 diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index c5cd2c5..1a025b7 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -133,6 +133,8 @@ #define IRQ_STACK_SIZE THREAD_SIZE +#define OVERFLOW_STACK_SIZE SZ_4K + /* * Alignment of kernel segments (e.g. .text, .data). */ diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index 92ddb6d..ee19563 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -57,6 +57,22 @@ static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) return (low <= sp && sp < high); } +#ifdef CONFIG_VMAP_STACK +DECLARE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack); + +#define OVERFLOW_STACK_PTR() ((unsigned long)this_cpu_ptr(overflow_stack) + OVERFLOW_STACK_SIZE) + +static inline bool on_overflow_stack(unsigned long sp) +{ + unsigned long low = (unsigned long)this_cpu_ptr(overflow_stack); + unsigned long high = low + OVERFLOW_STACK_SIZE; + + return (low <= sp && sp < high); +} +#else +static inline bool on_overflow_stack(unsigned long sp) { return false; } +#endif + /* * We can only safely access per-cpu stacks from current in a non-preemptible * context. @@ -69,6 +85,8 @@ static inline bool on_accessible_stack(struct task_struct *tsk, unsigned long sp return false; if (on_irq_stack(sp)) return true; + if (on_overflow_stack(sp)) + return true; return false; } diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index e5aa866..44a27c3 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -72,6 +72,37 @@ .macro kernel_ventry label .align 7 sub sp, sp, #S_FRAME_SIZE +#ifdef CONFIG_VMAP_STACK + add sp, sp, x0 // sp' = sp + x0 + sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp + tbnz x0, #THREAD_SHIFT, 0f + sub x0, sp, x0 // sp' - x0' = (sp + x0) - sp = x0 + sub sp, sp, x0 // sp' - x0 = (sp + x0) - x0 = sp + b \label + + /* Stash the original SP value in tpidr_el0 */ +0: msr tpidr_el0, x0 + + /* Recover the original x0 value and stash it in tpidrro_el0 */ + sub x0, sp, x0 + msr tpidrro_el0, x0 + + /* Switch to the overflow stack */ + adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0 + + /* + * Check whether we were already on the overflow stack. This may happen + * after panic() re-enables interrupts. + */ + mrs x0, tpidr_el0 // sp of interrupted context + sub x0, sp, x0 // delta with top of overflow stack + tst x0, #~(OVERFLOW_STACK_SIZE - 1) // within range? + b.ne __bad_stack // no? -> bad stack pointer + + /* We were already on the overflow stack. Restore sp/x0 and carry on. */ + sub sp, sp, x0 + mrs x0, tpidrro_el0 +#endif b \label .endm @@ -348,6 +379,34 @@ ENTRY(vectors) #endif END(vectors) +#ifdef CONFIG_VMAP_STACK + /* + * We detected an overflow in kernel_ventry, which switched to the + * overflow stack. Stash the exception regs, and head to our overflow + * handler. + */ +__bad_stack: + /* Restore the original x0 value */ + mrs x0, tpidrro_el0 + + /* + * Store the original GPRs to the new stack. The orginial SP (minus + * S_FRAME_SIZE) was stashed in tpidr_el0 by kernel_ventry. + */ + sub sp, sp, #S_FRAME_SIZE + kernel_entry 1 + mrs x0, tpidr_el0 + add x0, x0, #S_FRAME_SIZE + str x0, [sp, #S_SP] + + /* Stash the regs for handle_bad_stack */ + mov x0, sp + + /* Time to die */ + bl handle_bad_stack + ASM_BUG() +#endif /* CONFIG_VMAP_STACK */ + /* * Invalid mode handlers */ diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index d01c598..2c80a11 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include @@ -41,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -666,6 +668,43 @@ asmlinkage void bad_el0_sync(struct pt_regs *regs, int reason, unsigned int esr) force_sig_info(info.si_signo, &info, current); } +#ifdef CONFIG_VMAP_STACK + +DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack) + __aligned(16); + +asmlinkage void handle_bad_stack(struct pt_regs *regs) +{ + unsigned long tsk_stk = (unsigned long)current->stack; + unsigned long irq_stk = (unsigned long)this_cpu_read(irq_stack_ptr); + unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack); + unsigned int esr = read_sysreg(esr_el1); + unsigned long far = read_sysreg(far_el1); + + console_verbose(); + pr_emerg("Insufficient stack space to handle exception!"); + + __show_regs(regs); + + pr_emerg("Task stack: [0x%016lx..0x%016lx]\n", + tsk_stk, tsk_stk + THREAD_SIZE); + pr_emerg("IRQ stack: [0x%016lx..0x%016lx]\n", + irq_stk, irq_stk + THREAD_SIZE); + pr_emerg("Overflow stack: [0x%016lx..0x%016lx]\n", + ovf_stk, ovf_stk + OVERFLOW_STACK_SIZE); + + pr_emerg("ESR: 0x%08x -- %s\n", esr, esr_get_class_string(esr)); + pr_emerg("FAR: 0x%016lx\n", far); + + /* + * We use nmi_panic to limit the potential for recusive overflows, and + * to get a better stack trace. + */ + nmi_panic(NULL, "kernel stack overflow"); + cpu_park_loop(); +} +#endif + void __pte_error(const char *file, int line, unsigned long val) { pr_err("%s:%d: bad pte %016lx.\n", file, line, val);