From patchwork Tue Aug 15 12:50:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110143 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439082obb; Tue, 15 Aug 2017 05:52:35 -0700 (PDT) X-Received: by 10.84.229.8 with SMTP id b8mr30363856plk.93.1502801555522; Tue, 15 Aug 2017 05:52:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801555; cv=none; d=google.com; s=arc-20160816; b=Hvz2RBHx0E7rXz+LXGjaEwCnv+nuptQO3UbS+4c5YJD6vxg8XKkUH2721ioV/Iy3Hk fGNCgfcEwiQpcXnaeN6rsXcbEOFMc94r5ikbD+SFUKvO57ndR2xkMwUfmvnkPiTtgGCj FhF5/nbwo8yQ3f7q/yHqr9v0GSw/Na01uDpmRZ9uuprt7VaR57AYPtW8VXCfhc9V9El1 EY3OuKy7oq1BPW6qkaIQ7ntlNuT9bnsEzZzFYaECuuex3O0OQOdHtFK8DHuGDwvYitOK 9fxWFjb1vnmBF7w5PYIIzpVNh+pGneiHKizvIazbYVYxsqDwdEUMFP5WijEIfPXAJAVH 3CVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=J1QoV70zeK2xWn4Y/Ff1f+Pq9wWI5i3bZr/DV9XUGys=; b=EYKdOq2C09eXZAcTbUsc3mAIXgpPwmjc+LSOwLI90yZnFh9EA4hGpzWMmiWGY0mziz 0/wQs9EbpIrOBtjtJlZB6SCeoH/80YqD/5qvrACinuBFAK36YDWfVAL+fLD3bOUkMfVp Q76HkHoPFEMv/1t2ORPNdmaW4K1SPxewr42UHlDV+CcpwmupQcelodnxawO7Hpn1v5zK T3dyLmDOKZIlIbcF58bEOPAoPJ5etXq7ArEQphKA6rD0b28usMh+nAdeYes1WGpv3IrQ VsYXpcsCmeDjfqxLR4DxiCjPZRKKuW8FPrHau2a5dO59h2VQTG/p8rAnQ2bAJJ94Q1Vd u4Qg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q73si5508468pfa.255.2017.08.15.05.52.35; Tue, 15 Aug 2017 05:52:35 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752663AbdHOMwc (ORCPT + 25 others); Tue, 15 Aug 2017 08:52:32 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51390 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752632AbdHOMwb (ORCPT ); Tue, 15 Aug 2017 08:52:31 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BF02F13D5; Tue, 15 Aug 2017 05:52:30 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AC1F93F540; Tue, 15 Aug 2017 05:52:28 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 01/14] arm64: remove __die()'s stack dump Date: Tue, 15 Aug 2017 13:50:36 +0100 Message-Id: <1502801449-29246-2-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Our __die() implementation tries to dump the stack memory, in addition to a backtrace, which is problematic. For contemporary 16K stacks, this can be a lot of data, which can take a long time to dump, and can push other useful context out of the kernel's printk ringbuffer (and/or a user's scrollback buffer on an attached console). Additionally, the code implicitly assumes that the SP is on the task's stack, and tries to dump everything between the SP and the highest task stack address. When the SP points at an IRQ stack (or is corrupted), this makes the kernel attempt to dump vast amounts of VA space. With vmap'd stacks, this may result in erroneous accesses to peripherals. This patch removes the memory dump, leaving us to rely on the backtrace, and other means of dumping stack memory such as kdump. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: Laura Abbott Cc: James Morse Cc: Will Deacon --- arch/arm64/kernel/traps.c | 2 -- 1 file changed, 2 deletions(-) -- 1.9.1 diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index c2a81bf..9633773 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -237,8 +237,6 @@ static int __die(const char *str, int err, struct pt_regs *regs) end_of_stack(tsk)); if (!user_mode(regs)) { - dump_mem(KERN_EMERG, "Stack: ", regs->sp, - THREAD_SIZE + (unsigned long)task_stack_page(tsk)); dump_backtrace(regs, tsk); dump_instr(KERN_EMERG, regs); } From patchwork Tue Aug 15 12:50:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110144 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439181obb; Tue, 15 Aug 2017 05:52:41 -0700 (PDT) X-Received: by 10.99.122.94 with SMTP id j30mr27176922pgn.188.1502801561552; Tue, 15 Aug 2017 05:52:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801561; cv=none; d=google.com; s=arc-20160816; b=opiRAzkTrAMY1bwnwmQgeZn9I6KukKJb5w8mx5TTch9DjbK+1RYLgysKK++5D2d/ki 0eg2J9fxFwPUWq3YcxdS3YUwOt1zojMy0KLeFUXVdfdE3L8xgCNFoxrLP0KeHHNA2yn2 MtGjhdWiGHZRmdfEpgB6KBYclm7oRbnijL4VCssjpYZ7i12dDmccfk8aMZ5eAjj89Y2w kvkrPBnvkrv260xuZa7KPKiqut58l65umv37vv9iUeYRiBkDVYqpfdf4z1Ps4NZZUSDF e26QWl49vU18SFOee/CAc5z/3guvfnivwcmaGhNegsYnQlPvEnxMHAkdTrx9I0opLVKl vcKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=F463WU8StH6ifPP6iNkZUVzkM6XvhR43pwP5TFXKUtw=; b=bIgkSG8jbIREvOpxhyHX1vbDDCDBjHKy7q7lejoVxnNHhpsMV0MEFg2zISIxVNnNYw a0UC5KMIet2iLYV0tYgzNew1DgJ7XxyUmkhpWYVHwNm11TjfPY2Ae1xQ1XTttfq6QQGY U9xQeLnlWjvGlrqVR79SmRLIYxgYjf7i1h5TIQGnOY50r0Q3gV8LBgYqspQpuskQ61Ge D2Oy6dyUVy3HIVegAWqzJLbYl5Q6Gv88lLKl6KkU3IRQBtiTjmczy1ea98kl9YVyeKGT FIuUrS7CIBpGYZ/XihjOAHL/hLliNL5/8bruiubaYqc6cmURomiKShjG27Lja6UtAwE1 bZww== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k10si6288545pln.658.2017.08.15.05.52.41; Tue, 15 Aug 2017 05:52:41 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752687AbdHOMwi (ORCPT + 25 others); Tue, 15 Aug 2017 08:52:38 -0400 Received: from foss.arm.com ([217.140.101.70]:51426 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752664AbdHOMwg (ORCPT ); Tue, 15 Aug 2017 08:52:36 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 263031596; Tue, 15 Aug 2017 05:52:36 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 13CBD3F540; Tue, 15 Aug 2017 05:52:33 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 02/14] fork: allow arch-override of VMAP stack alignment Date: Tue, 15 Aug 2017 13:50:37 +0100 Message-Id: <1502801449-29246-3-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In some cases, an architecture might wish its stacks to be aligned to a boundary larger than THREAD_SIZE. For example, using an alignment of double THREAD_SIZE can allow for stack overflows smaller than THREAD_SIZE to be detected by checking a single bit of the stack pointer. This patch allows architectures to override the alignment of VMAP'd stacks, by defining THREAD_ALIGN. Where not defined, this defaults to THREAD_SIZE, as is the case today. Signed-off-by: Mark Rutland Cc: Andy Lutomirski Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon Cc: linux-kernel@vger.kernel.org --- kernel/fork.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -- 1.9.1 diff --git a/kernel/fork.c b/kernel/fork.c index 17921b0..696d692 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -217,7 +217,10 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node) return s->addr; } - stack = __vmalloc_node_range(THREAD_SIZE, THREAD_SIZE, +#ifndef THREAD_ALIGN +#define THREAD_ALIGN THREAD_SIZE +#endif + stack = __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN, VMALLOC_START, VMALLOC_END, THREADINFO_GFP, PAGE_KERNEL, From patchwork Tue Aug 15 12:50:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110145 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439238obb; Tue, 15 Aug 2017 05:52:46 -0700 (PDT) X-Received: by 10.84.217.70 with SMTP id e6mr32319546plj.414.1502801566000; Tue, 15 Aug 2017 05:52:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801565; cv=none; d=google.com; s=arc-20160816; b=ld6DEkppgFNbKT9YjJNe5d5oRc8hzpQBLfwhZlTM18+3jgSEXfyjNlVIT4p+iF+E6a mbUeGiWVgNRxWh5DhEEexh70gLc0ssrXK8Iai3V9swNPf2qtS4JBpZPjV1m9XG8w1pGM oJenrAkyBYWMkqsutqUeJP5WoD4m/wxCNU7+6oeDaqTygckWxCPdvONDlLYa/TvgP8ih 7Y0PNXebhX4jEhFQ9+XVYGT50IfdPn6l36BsdQKN1Fzj4YTSba16Wj8vkUsarHCssS4g lcxDXxnlzDs1I8DksXAuVk3wHwA+UgqaGdNT8tsvcZQHLMH2r5UxaH3OjqTnXyRQzwfK DOwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=U+XUULveGlPYNfeWwsPhk+JDrrty+kkL9PjgxULl+w4=; b=eH8i9N7Rirqf3lnH1IrlDsS/EOjsadqOFX2ZgCecW61wHfejCHik+5o0wsBSBWovqD h5mGgsTi0IlfVQn80ZZxxeWFzj6fZQ2FQUWWg+ixKGqawW0RXsY4mptpmkiSSiJvAIRL IL2Z4rl9UiAmnYbqfPtxztYCTra0BAOAKFF1p+H//rzXAZb7TQBVfPJ7VzzYztdLzJwv lOON23THpL3CMoEod0LPSXZWhGPLZ47Diw7LeoMpl/Y09blAXkljuNgekUE+3pXBpQPl BUO3Kjhw1eZrv6AG8yu1OzOWjSoHhNMm7haybSkABrQCXB0LqcpQFtZLBgAjS+//osqP mHsA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k10si6288545pln.658.2017.08.15.05.52.45; Tue, 15 Aug 2017 05:52:45 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752716AbdHOMwl (ORCPT + 25 others); Tue, 15 Aug 2017 08:52:41 -0400 Received: from foss.arm.com ([217.140.101.70]:51450 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752664AbdHOMwj (ORCPT ); Tue, 15 Aug 2017 08:52:39 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 66A3E15BF; Tue, 15 Aug 2017 05:52:39 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 547C33F540; Tue, 15 Aug 2017 05:52:37 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 03/14] arm64: kernel: remove {THREAD,IRQ_STACK}_START_SP Date: Tue, 15 Aug 2017 13:50:38 +0100 Message-Id: <1502801449-29246-4-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Ard Biesheuvel For historical reasons, we leave the top 16 bytes of our task and IRQ stacks unused, a practice used to ensure that the SP can always be masked to find the base of the current stack (historically, where thread_info could be found). However, this is not necessary, as: * When an exception is taken from a task stack, we decrement the SP by S_FRAME_SIZE and stash the exception registers before we compare the SP against the task stack. In such cases, the SP must be at least S_FRAME_SIZE below the limit, and can be safely masked to determine whether the task stack is in use. * When transitioning to an IRQ stack, we'll place a dummy frame onto the IRQ stack before enabling asynchronous exceptions, or executing code we expect to trigger faults. Thus, if an exception is taken from the IRQ stack, the SP must be at least 16 bytes below the limit. * We no longer mask the SP to find the thread_info, which is now found via sp_el0. Note that historically, the offset was critical to ensure that cpu_switch_to() found the correct stack for new threads that hadn't yet executed ret_from_fork(). Given that, this initial offset serves no purpose, and can be removed. This brings us in-line with other architectures (e.g. x86) which do not rely on this masking. Signed-off-by: Ard Biesheuvel [Mark: rebase, kill THREAD_START_SP, commit msg additions] Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/irq.h | 5 ++--- arch/arm64/include/asm/processor.h | 2 +- arch/arm64/include/asm/thread_info.h | 1 - arch/arm64/kernel/entry.S | 2 +- arch/arm64/kernel/smp.c | 2 +- 5 files changed, 5 insertions(+), 7 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h index 8ba89c4..1ebe202 100644 --- a/arch/arm64/include/asm/irq.h +++ b/arch/arm64/include/asm/irq.h @@ -2,7 +2,6 @@ #define __ASM_IRQ_H #define IRQ_STACK_SIZE THREAD_SIZE -#define IRQ_STACK_START_SP THREAD_START_SP #ifndef __ASSEMBLER__ @@ -26,9 +25,9 @@ static inline int nr_legacy_irqs(void) static inline bool on_irq_stack(unsigned long sp) { unsigned long low = (unsigned long)raw_cpu_ptr(irq_stack); - unsigned long high = low + IRQ_STACK_START_SP; + unsigned long high = low + IRQ_STACK_SIZE; - return (low <= sp && sp <= high); + return (low <= sp && sp < high); } static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 64c9e78..6687dd2 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -159,7 +159,7 @@ extern struct task_struct *cpu_switch_to(struct task_struct *prev, struct task_struct *next); #define task_pt_regs(p) \ - ((struct pt_regs *)(THREAD_START_SP + task_stack_page(p)) - 1) + ((struct pt_regs *)(THREAD_SIZE + task_stack_page(p)) - 1) #define KSTK_EIP(tsk) ((unsigned long)task_pt_regs(tsk)->pc) #define KSTK_ESP(tsk) user_stack_pointer(task_pt_regs(tsk)) diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 46c3b93..b29ab0e 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -30,7 +30,6 @@ #endif #define THREAD_SIZE 16384 -#define THREAD_START_SP (THREAD_SIZE - 16) #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 612a077..f31c7b2 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -272,7 +272,7 @@ alternative_else_nop_endif cbnz x25, 9998f adr_this_cpu x25, irq_stack, x26 - mov x26, #IRQ_STACK_START_SP + mov x26, #IRQ_STACK_SIZE add x26, x25, x26 /* switch to the irq stack */ diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index dc66e6e..f13ddb24 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -154,7 +154,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) * page tables. */ secondary_data.task = idle; - secondary_data.stack = task_stack_page(idle) + THREAD_START_SP; + secondary_data.stack = task_stack_page(idle) + THREAD_SIZE; update_cpu_boot_status(CPU_MMU_OFF); __flush_dcache_area(&secondary_data, sizeof(secondary_data)); From patchwork Tue Aug 15 12:50:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110146 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439302obb; Tue, 15 Aug 2017 05:52:50 -0700 (PDT) X-Received: by 10.99.9.67 with SMTP id 64mr27684469pgj.12.1502801569964; Tue, 15 Aug 2017 05:52:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801569; cv=none; d=google.com; s=arc-20160816; b=KsKJkKEOda0TbYhSkwNEe3nZLKL5O2seqa/0V/JiKdRsDRHK1fcMNhbUKfQTKhbapo nrB1l+h+zWbkdPF2Qxq1ilf60ncYMSjt/KVqq2q/hNCnGIyZ7LHJsPEPa+TOc8oiuDMd D75nAliLmx6cJ2YDLQlJ8VgBZT35RucS2QAQGJbF9zTahgFrWxIn86rTOudq/8hBS0GK KEZ9tTVGXQ7+EfSB63sVSKj5yryhKN4yMWnuBdem1cIAP575FrR5+q+qRtmzXPhu3UZL y5lL9SA2GJ4lVLElOWlC1vV7/G/qSj0XspLOBsnXNAdkPKL/CYz5+M1wwcLG5mHMXEFS kQzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xZxIgwWSIG+gEpPyUVmKMU91P/VJjpt3tlccCECSW+k=; b=njCdu1BTdDboNypT+BDox7lacBh/+N3H6wb65pZhXJ8ql/z6aCy+1lz+gl6bXdq0HA cVFTQtfKTk84tMyWyYtsVLz2wieaNgfGjciP8hHPzT9mWKdLFF6uRgqpJ670yYfdrKaD qMJCggj/u/6157vtg4COfblZeIOT53Ff+LKaGYQNMwBLhaN83tYWQLM98BwUhm+E6w19 1vN5DZ/ifyETAS317IC8qLrltMbzl6ezPvx2RquzdiA69JvH0Y8t7XeOCqrigMOJFIJG IX64VknTbuzrZ9G3oIHtw7SfjGX0jcL2RVY1Rn3IrPt39oEKE8Lmu87vexqcTafi8ucb +hiw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y100si6239345plh.900.2017.08.15.05.52.49; Tue, 15 Aug 2017 05:52:49 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752751AbdHOMwr (ORCPT + 25 others); Tue, 15 Aug 2017 08:52:47 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51466 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752664AbdHOMwn (ORCPT ); Tue, 15 Aug 2017 08:52:43 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AB085164F; Tue, 15 Aug 2017 05:52:42 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9897C3F540; Tue, 15 Aug 2017 05:52:40 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 04/14] arm64: factor out PAGE_* and CONT_* definitions Date: Tue, 15 Aug 2017 13:50:39 +0100 Message-Id: <1502801449-29246-5-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Some headers rely on PAGE_* definitions from , but cannot include this due to potential circular includes. For example, a number of definitions in rely on PAGE_SHIFT, and includes . This requires users of these definitions to include both headers, which is fragile and error-prone. This patch ameliorates matters by moving the basic definitions out to a new header, . Both and are updated to include this, avoiding this fragility, and avoiding the possibility of circular include dependencies. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/memory.h | 1 + arch/arm64/include/asm/page-def.h | 34 ++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/page.h | 12 +----------- 3 files changed, 36 insertions(+), 11 deletions(-) create mode 100644 arch/arm64/include/asm/page-def.h -- 1.9.1 diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 32f827233..77d55dc 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -25,6 +25,7 @@ #include #include #include +#include #include /* diff --git a/arch/arm64/include/asm/page-def.h b/arch/arm64/include/asm/page-def.h new file mode 100644 index 0000000..01591a2 --- /dev/null +++ b/arch/arm64/include/asm/page-def.h @@ -0,0 +1,34 @@ +/* + * Based on arch/arm/include/asm/page.h + * + * Copyright (C) 1995-2003 Russell King + * Copyright (C) 2017 ARM Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ +#ifndef __ASM_PAGE_DEF_H +#define __ASM_PAGE_DEF_H + +#include + +/* PAGE_SHIFT determines the page size */ +/* CONT_SHIFT determines the number of pages which can be tracked together */ +#define PAGE_SHIFT CONFIG_ARM64_PAGE_SHIFT +#define CONT_SHIFT CONFIG_ARM64_CONT_SHIFT +#define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT) +#define PAGE_MASK (~(PAGE_SIZE-1)) + +#define CONT_SIZE (_AC(1, UL) << (CONT_SHIFT + PAGE_SHIFT)) +#define CONT_MASK (~(CONT_SIZE-1)) + +#endif /* __ASM_PAGE_DEF_H */ diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index 8472c6d..60d02c8 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -19,17 +19,7 @@ #ifndef __ASM_PAGE_H #define __ASM_PAGE_H -#include - -/* PAGE_SHIFT determines the page size */ -/* CONT_SHIFT determines the number of pages which can be tracked together */ -#define PAGE_SHIFT CONFIG_ARM64_PAGE_SHIFT -#define CONT_SHIFT CONFIG_ARM64_CONT_SHIFT -#define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT) -#define PAGE_MASK (~(PAGE_SIZE-1)) - -#define CONT_SIZE (_AC(1, UL) << (CONT_SHIFT + PAGE_SHIFT)) -#define CONT_MASK (~(CONT_SIZE-1)) +#include #ifndef __ASSEMBLY__ From patchwork Tue Aug 15 12:50:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110156 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5441992obb; Tue, 15 Aug 2017 05:55:43 -0700 (PDT) X-Received: by 10.84.232.11 with SMTP id h11mr32006415plk.126.1502801743512; Tue, 15 Aug 2017 05:55:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801743; cv=none; d=google.com; s=arc-20160816; b=PMBbd9V+tFMscxsDnrqW5k25JFWenylRORUp+I5GAurhTvTSnQx2/cZL44/qK6zqXb uxlZYsd+vdcZq0i+gWPzTqXDd4O65fVcP/BJxwpu5XPO1JTS4pCsvzUbRCpjMeRxtHTs JJ4BttqZ4yfxYtVmO7sfVxumQc+lEIA8pqfU8/Gql8m3rpRT2mzvyBkEAV3tY63xNxna aT9MtCZcoLzUvqwZnBwvoq+Ugxc0FE5RIR+77eG1Kkr4tAHbjCD2LaafcJtrlKsMrOJE HVQ2d3ijhjNzf+Ei33UFLJ9R0lqs/vBrPXH7ZqnxyTvMMdV/dJbfvUFj8HiBj9Bdf++2 FBeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=zoJfAKFCtf47xzIccC4zHTzYlbWQVIlYS4lQ0vV6nF4=; b=ENxK1gGHGgHXBCNoNCiTtAiLwGPuO9Cn9doF6cgRmjizXrIAxlM+S+g3P54NFJ03ZQ 7kJXz+4EAhLu5lSgW5yOP99fwcmRgdPhe2rUixO6DtSXTGQhWB412ZphfmJ0zM0yUI+c 8fkeDFB/5WFIzB87MJidsUDfZhVwxVW5u3vJAV0U+S1Dlad+00SCd9r4KELUVTUzdH4G sASmJ0UgkXiCi+vZr/3V71DaBrAHrf+fsKO6b/NKCsc7TqYbiKmtVWtkUO/Xb78TozUY IGhaLC2lQiXpStX5TMQwjn2MX4uSFFwt+o8HuVQJxUOT9+ot1RuIaOO4EV64Zx9djI1/ Xh6w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f4si6124033plm.562.2017.08.15.05.55.43; Tue, 15 Aug 2017 05:55:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752913AbdHOMzl (ORCPT + 25 others); Tue, 15 Aug 2017 08:55:41 -0400 Received: from foss.arm.com ([217.140.101.70]:51486 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752717AbdHOMwq (ORCPT ); Tue, 15 Aug 2017 08:52:46 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7DBBC1650; Tue, 15 Aug 2017 05:52:45 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8ED423F540; Tue, 15 Aug 2017 05:52:43 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 05/14] arm64: clean up THREAD_* definitions Date: Tue, 15 Aug 2017 13:50:40 +0100 Message-Id: <1502801449-29246-6-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently we define THREAD_SIZE and THREAD_SIZE_ORDER separately, with the latter dependent on particular CONFIG_ARM64_*K_PAGES definitions. This is somewhat opaque, and will get in the way of future modifications to THREAD_SIZE. This patch cleans this up, defining both in terms of a common THREAD_SHIFT, and using PAGE_SHIFT to calculate THREAD_SIZE_ORDER, rather than using a number of definitions dependent on config symbols. Subsequent patches will make use of this to alter the stack size used in some configurations. At the same time, these are moved into , which will avoid circular include issues in subsequent patches. To ensure that existing code isn't adversely affected, is updated to transitively include these definitions. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/memory.h | 8 ++++++++ arch/arm64/include/asm/thread_info.h | 9 +-------- 2 files changed, 9 insertions(+), 8 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 77d55dc..8ab4774 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -102,6 +102,14 @@ #define KASAN_SHADOW_SIZE (0) #endif +#define THREAD_SHIFT 14 + +#if THREAD_SHIFT >= PAGE_SHIFT +#define THREAD_SIZE_ORDER (THREAD_SHIFT - PAGE_SHIFT) +#endif + +#define THREAD_SIZE (UL(1) << THREAD_SHIFT) + /* * Memory types available. */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index b29ab0e..aa04b73 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -23,18 +23,11 @@ #include -#ifdef CONFIG_ARM64_4K_PAGES -#define THREAD_SIZE_ORDER 2 -#elif defined(CONFIG_ARM64_16K_PAGES) -#define THREAD_SIZE_ORDER 0 -#endif - -#define THREAD_SIZE 16384 - #ifndef __ASSEMBLY__ struct task_struct; +#include #include #include From patchwork Tue Aug 15 12:50:41 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110147 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439387obb; Tue, 15 Aug 2017 05:52:55 -0700 (PDT) X-Received: by 10.99.43.5 with SMTP id r5mr26594629pgr.313.1502801575048; Tue, 15 Aug 2017 05:52:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801575; cv=none; d=google.com; s=arc-20160816; b=GTN6VjXur3BgGV1DIJuV2CDvPMbQ/H8L24ZH387ll7koiLWhDaiTsEWqNK4hr/9/SB HAHLfSk1GsrY1aISyakwfPk3RJLWEFhpJY0h0EihTPUVouQIQcWlqSKVYbeqaxT9PFnx gkgfAjPMmDoxlSQyCa5geUgLfEg3vul7bX1kE44i0pRcyPjmo/FjFUVJk5AeOGpw7Wox 6n+cGCoiiicKzXmtRWnMMPUEGOAVjY4h0KYyqFxQT2c2GIyt5ovo5JgoyhJ6zjZ/wSDO 9NzSFAlkyLoipEwCb2WRAkgGk9w6Pcm86GXCCrcF9PSkFC0T69cHVIq0ZHHLU7aYuv2a Ijwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=rpxYGujq7JZgNvn32iBo2ddGsB4dn4smHUwCuhgxPEw=; b=xD3IeiPN7FsGM9phGU6SUe+caU0KT2Wnn231kfv8rYd+FmfTXcH0Geliq54y94ULH7 TNg1SWWOddBIg5d72KhY9Qi92Ijm+l/iJ03BHcU4LZt1Y2pA8+3c2UJomokNjCkWnWU9 L7I5+auNM4uclcZNgP0PSmg14E2cgJrk/yw2DMVGOOgnhOEOhF6w6ihRREpdLbRmNim7 ZHIl9BmqA/+IwDiOKWVGJiww+5DPXBaZb0YyK6ivem97OwIN+BLto/gtY4P82bQp0qyY r86u0cbb56s59hZBGc4YdHtTTpzdx1+aSaTe7OOIX6BqndjsYhPdLIqrw+GanRBioSnW XHyg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u85si5497400pfi.252.2017.08.15.05.52.54; Tue, 15 Aug 2017 05:52:55 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752768AbdHOMwv (ORCPT + 25 others); Tue, 15 Aug 2017 08:52:51 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51492 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752133AbdHOMws (ORCPT ); Tue, 15 Aug 2017 08:52:48 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8C11E165D; Tue, 15 Aug 2017 05:52:48 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 79A6D3F540; Tue, 15 Aug 2017 05:52:46 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 06/14] arm64: clean up irq stack definitions Date: Tue, 15 Aug 2017 13:50:41 +0100 Message-Id: <1502801449-29246-7-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Before we add yet another stack to the kernel, it would be nice to ensure that we consistently organise stack definitions and related helper functions. This patch moves the basic IRQ stack defintions to to live with their task stack counterparts. Helpers used for unwinding are moved into , where subsequent patches will add helpers for other stacks. Includes are fixed up accordingly. This patch is a pure refactoring -- there should be no functional changes as a result of this patch. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/irq.h | 24 ------------------------ arch/arm64/include/asm/memory.h | 2 ++ arch/arm64/include/asm/stacktrace.h | 25 ++++++++++++++++++++++++- arch/arm64/kernel/ptrace.c | 1 + 4 files changed, 27 insertions(+), 25 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h index 1ebe202..5e6f772 100644 --- a/arch/arm64/include/asm/irq.h +++ b/arch/arm64/include/asm/irq.h @@ -1,20 +1,12 @@ #ifndef __ASM_IRQ_H #define __ASM_IRQ_H -#define IRQ_STACK_SIZE THREAD_SIZE - #ifndef __ASSEMBLER__ -#include -#include - #include -#include struct pt_regs; -DECLARE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack); - extern void set_handle_irq(void (*handle_irq)(struct pt_regs *)); static inline int nr_legacy_irqs(void) @@ -22,21 +14,5 @@ static inline int nr_legacy_irqs(void) return 0; } -static inline bool on_irq_stack(unsigned long sp) -{ - unsigned long low = (unsigned long)raw_cpu_ptr(irq_stack); - unsigned long high = low + IRQ_STACK_SIZE; - - return (low <= sp && sp < high); -} - -static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) -{ - unsigned long low = (unsigned long)task_stack_page(tsk); - unsigned long high = low + THREAD_SIZE; - - return (low <= sp && sp < high); -} - #endif /* !__ASSEMBLER__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 8ab4774..1fc2453 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -110,6 +110,8 @@ #define THREAD_SIZE (UL(1) << THREAD_SHIFT) +#define IRQ_STACK_SIZE THREAD_SIZE + /* * Memory types available. */ diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index 3bebab3..000e2418 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -16,7 +16,12 @@ #ifndef __ASM_STACKTRACE_H #define __ASM_STACKTRACE_H -struct task_struct; +#include +#include +#include + +#include +#include struct stackframe { unsigned long fp; @@ -31,4 +36,22 @@ extern void walk_stackframe(struct task_struct *tsk, struct stackframe *frame, int (*fn)(struct stackframe *, void *), void *data); extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); +DECLARE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack); + +static inline bool on_irq_stack(unsigned long sp) +{ + unsigned long low = (unsigned long)raw_cpu_ptr(irq_stack); + unsigned long high = low + IRQ_STACK_SIZE; + + return (low <= sp && sp < high); +} + +static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) +{ + unsigned long low = (unsigned long)task_stack_page(tsk); + unsigned long high = low + THREAD_SIZE; + + return (low <= sp && sp < high); +} + #endif /* __ASM_STACKTRACE_H */ diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c index baf0838..a9f8715 100644 --- a/arch/arm64/kernel/ptrace.c +++ b/arch/arm64/kernel/ptrace.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include From patchwork Tue Aug 15 12:50:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110148 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439500obb; Tue, 15 Aug 2017 05:53:02 -0700 (PDT) X-Received: by 10.99.109.140 with SMTP id i134mr27230561pgc.368.1502801581947; Tue, 15 Aug 2017 05:53:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801581; cv=none; d=google.com; s=arc-20160816; b=giW4yAks7ZQGhq6NcqOYpZnM4US0SIX4il32+5MedlKQ+noUgS86TXrBUSDGHHQSWt +99J552+ToI5gq165tg2kx6bldgH+NuUAlVeiCtD7EWTIc5YUlWJ4/dHW+NIJmU299uP M8wdOquqc0zvyXLHI+DhHlo3/nVCG0PfcjuOJ4hgKQ/B6o+jR5x2qeLm5qckM6dHmQ4t PeXSsHQhIgu+JZDmM9kNw0YYybRy2y8895dEDElVMUTNDebBVu92Ty2sJkNvQI7u70Bs vdxjBrpL/2FOFrELQ3EbEYD+lse/dn0Qofm6NsIZvIo3DpOTN1GA7o6GmvRUQe1TF3Yp ocCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=HgCoibxZ1lZ9gSRAUQlmkEOdmHtP4bBo3iPD/JAEm/8=; b=udokWcPMfxF552PDfQMcuPdWqL/tWt2e9cdUQOQQkU8Li/2jltQxRO7GzR8Bg5bzgk JI19w0NPsXBg0ECMeptXaercjx23zuxgvarTcC6AyVelzelB5mHKprRfiGBqkLvlrmjK +vWn044SXCKgQ2j3VXXZtmgBnvYRd+QxXSGp2ZRNh7m4Jk9X2WtdJu155UDL6jYSj+DE NkDnMtpzSA6HQC1YrqqXQKt4MDJVu2UtjUmOx3vgESrwJXJ548OSwZLMRa0frPbh2uPY udf8PgaKvQ6TowUzAc0qq/Rb5AFy2IVhtAYABQ8ryocJqN9s0tAREbDXzsLIrzf2vkD0 rEQg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u85si5497400pfi.252.2017.08.15.05.53.01; Tue, 15 Aug 2017 05:53:01 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752805AbdHOMw7 (ORCPT + 25 others); Tue, 15 Aug 2017 08:52:59 -0400 Received: from foss.arm.com ([217.140.101.70]:51522 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752756AbdHOMwv (ORCPT ); Tue, 15 Aug 2017 08:52:51 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 594A41688; Tue, 15 Aug 2017 05:52:51 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 67C1D3F540; Tue, 15 Aug 2017 05:52:49 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 07/14] arm64: move SEGMENT_ALIGN to Date: Tue, 15 Aug 2017 13:50:42 +0100 Message-Id: <1502801449-29246-8-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently we define SEGMENT_ALIGN directly in our vmlinux.lds.S. This is unfortunate, as the EFI stub currently open-codes the same number, and in future we'll want to fiddle with this. This patch moves the definition to our , where it can be used by both vmlinux.lds.S and the EFI stub code. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/memory.h | 19 +++++++++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 16 ---------------- 2 files changed, 19 insertions(+), 16 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 1fc2453..7fa6ad4 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -113,6 +113,25 @@ #define IRQ_STACK_SIZE THREAD_SIZE /* + * Alignment of kernel segments (e.g. .text, .data). + */ +#if defined(CONFIG_DEBUG_ALIGN_RODATA) +/* + * 4 KB granule: 1 level 2 entry + * 16 KB granule: 128 level 3 entries, with contiguous bit + * 64 KB granule: 32 level 3 entries, with contiguous bit + */ +#define SEGMENT_ALIGN SZ_2M +#else +/* + * 4 KB granule: 16 level 3 entries, with contiguous bit + * 16 KB granule: 4 level 3 entries, without contiguous bit + * 64 KB granule: 1 level 3 entry + */ +#define SEGMENT_ALIGN SZ_64K +#endif + +/* * Memory types available. */ #define MT_DEVICE_nGnRnE 0 diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 987a00e..7156538 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -72,22 +72,6 @@ PECOFF_FILE_ALIGNMENT = 0x200; #define PECOFF_EDATA_PADDING #endif -#if defined(CONFIG_DEBUG_ALIGN_RODATA) -/* - * 4 KB granule: 1 level 2 entry - * 16 KB granule: 128 level 3 entries, with contiguous bit - * 64 KB granule: 32 level 3 entries, with contiguous bit - */ -#define SEGMENT_ALIGN SZ_2M -#else -/* - * 4 KB granule: 16 level 3 entries, with contiguous bit - * 16 KB granule: 4 level 3 entries, without contiguous bit - * 64 KB granule: 1 level 3 entry - */ -#define SEGMENT_ALIGN SZ_64K -#endif - SECTIONS { /* From patchwork Tue Aug 15 12:50:43 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110154 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5441408obb; Tue, 15 Aug 2017 05:55:03 -0700 (PDT) X-Received: by 10.98.82.197 with SMTP id g188mr28560793pfb.34.1502801703674; Tue, 15 Aug 2017 05:55:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801703; cv=none; d=google.com; s=arc-20160816; b=0JpNxSEOocr2zFw/tXTBaF7ed4qLRd75KAebE1w6kAmk0Sp7E1l7GLi94RmwUkAkmy MoyGDlMz3CRelarmB1N28ktarg54sDleyHd3daMBIwreYbpAr0sE+rWMAJ1JOGh1fsO2 RdBy7uk2dzDNat4VlRFHvySrMnCfTjzO1Ml9G5tyTciMsj6TasUbh1H4KHIVxNRR1FiK ijaFJEX3G8jVDLNpnmBhZt6fDF1jpYPPPWW5D63nE64w/VWeuRPkIUwEiCg1CsNILZld tvs2Ksx/SIG+oVPrc6xJs+OYp7fSvyVLpoQtW7RfMtWPsKlvMD3fr6+P2fdqjtta6PRp xzWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=f4W+MVt+MWmgN7H5Ph3KbbnDHwzDRKdW0Fw6JePBmSc=; b=L76MY4Wkd+iC3MxrA7fgPfDRICWI0oFlhNHqWqiDGwGbKMCFTbC1JKR3VqWhKWwNAT H8pNSSO/51m15h0zqdvxUC5Wp0EWvguCkPpJL9hlc6fSRndh4l099oM8yzXZ2wPIIdzW MGKbOSPBIi7542pYmU4twpjm/4z4rw4fuRBMjQ6jAPnEzfhRCqBIV58HsBCEgq1GEhIk VhbmbFw0uI9+RmfKZyBqxmzFpimRcRmKcA/SCsBD50Dk3mFBOqtjQN1GgRivHE07tl5r UnRmNtkGoVrYnunKPReV3nhmhvWfPE6DnfJKGCCdY8FE1pDgS+tLrGPzC+w5Ljz4rnWG fz1g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i64si5469001pfj.648.2017.08.15.05.55.03; Tue, 15 Aug 2017 05:55:03 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752788AbdHOMw5 (ORCPT + 25 others); Tue, 15 Aug 2017 08:52:57 -0400 Received: from foss.arm.com ([217.140.101.70]:51528 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752769AbdHOMwy (ORCPT ); Tue, 15 Aug 2017 08:52:54 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4A6E6168F; Tue, 15 Aug 2017 05:52:54 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2F1DC3F540; Tue, 15 Aug 2017 05:52:52 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 08/14] efi/arm64: add EFI_KIMG_ALIGN Date: Tue, 15 Aug 2017 13:50:43 +0100 Message-Id: <1502801449-29246-9-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The EFI stub is intimately coupled with the kernel, and takes advantage of this by relocating the kernel at a weaker alignment than the documented boot protocol mandates. However, it does so by assuming it can align the kernel to the segment alignment, and assumes that this is 64K. In subsequent patches, we'll have to consider other details to determine this de-facto alignment constraint. This patch adds a new EFI_KIMG_ALIGN definition that will track the kernel's de-facto alignment requirements. Subsequent patches will modify this as required. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Matt Fleming Cc: Will Deacon --- arch/arm64/include/asm/efi.h | 3 +++ drivers/firmware/efi/libstub/arm64-stub.c | 6 ++++-- 2 files changed, 7 insertions(+), 2 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index 8f3043a..0e8cc3b 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -48,6 +49,8 @@ */ #define EFI_FDT_ALIGN SZ_2M /* used by allocate_new_fdt_and_exit_boot() */ +#define EFI_KIMG_ALIGN SEGMENT_ALIGN + /* on arm64, the FDT may be located anywhere in system RAM */ static inline unsigned long efi_get_max_fdt_addr(unsigned long dram_base) { diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index b4c2589..af6ae95 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c @@ -11,6 +11,7 @@ */ #include #include +#include #include #include @@ -81,9 +82,10 @@ efi_status_t handle_kernel_image(efi_system_table_t *sys_table_arg, /* * If CONFIG_DEBUG_ALIGN_RODATA is not set, produce a * displacement in the interval [0, MIN_KIMG_ALIGN) that - * is a multiple of the minimal segment alignment (SZ_64K) + * doesn't violate this kernel's de-facto alignment + * constraints. */ - u32 mask = (MIN_KIMG_ALIGN - 1) & ~(SZ_64K - 1); + u32 mask = (MIN_KIMG_ALIGN - 1) & ~(EFI_KIMG_ALIGN - 1); u32 offset = !IS_ENABLED(CONFIG_DEBUG_ALIGN_RODATA) ? (phys_seed >> 32) & mask : TEXT_OFFSET; From patchwork Tue Aug 15 12:50:44 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110155 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5441427obb; Tue, 15 Aug 2017 05:55:05 -0700 (PDT) X-Received: by 10.84.233.141 with SMTP id l13mr31519787plk.333.1502801705276; Tue, 15 Aug 2017 05:55:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801705; cv=none; d=google.com; s=arc-20160816; b=DrSdPFeUrE628oSAWXTNVLMkAKoIZjG+D3g5e5vVpxooz+5L5wspiFvry8kyQEGD1L ItpgBdfrZoJggtaWWykAqJVh9aD/r8Rj9EXOTW1txDdABqqf+gCG0LZ8Wcl1bkxLBiWK HFG7evSF/KShLzl+wQQERrJt9hlKyI52CdewMYOvx8aqJnpSUNXEmwFj0t8QmWopbCdS z7w9ROtEPCCAlh63Ftln5QtF6KMM27deD3+XgKAJ8r7Klzodj5QaGfkRGjk/55ayhxDZ oowtbkOtXe6yBvhNhZrEEBz4m/Aikjs+WAVqmVjEECaA15w/y8oxMajJl6/J2e32POi6 d5dg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=AP4s6SaW4th44rKiCCixbngg7iQYyZ0zacp1gDoZmrs=; b=uNLF5AjDk7O26A0SgAIthSBaLj3mspSe1sX6nq0cDx9t4h/DFb07k8ULFU9PfLVsTu thcM1Hb/TO/VX8uZUnSTGLJv4dSR6IOuF7pjda7NHn8QxV8JPU1DvqWRahAUrY0z/6fV BG26csKZ2wQMokkzRyLCk7apw3kh3vNAhL+a32HozHid0g9R8IKToBZ+7D5rJuIEUkTm Nb4p9Hx1sxUHcg9ufUx+c5ajs1CLByPdEm0Z0g/YDyhF+EJFBgchNUvsbdQmtRBHfD8l C31UB4SgkbG5qkx4GYcBFIDAdZtqSzePLqhCkUu8/iYzBzWc6BLOKfp4JptECB5+i0Cn hAxw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f4si6124033plm.562.2017.08.15.05.55.04; Tue, 15 Aug 2017 05:55:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753053AbdHOMzB (ORCPT + 25 others); Tue, 15 Aug 2017 08:55:01 -0400 Received: from foss.arm.com ([217.140.101.70]:51570 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752769AbdHOMw5 (ORCPT ); Tue, 15 Aug 2017 08:52:57 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1461213D5; Tue, 15 Aug 2017 05:52:57 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 023943F540; Tue, 15 Aug 2017 05:52:54 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 09/14] arm64: factor out entry stack manipulation Date: Tue, 15 Aug 2017 13:50:44 +0100 Message-Id: <1502801449-29246-10-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In subsequent patches, we will detect stack overflow in our exception entry code, by verifying the SP after it has been decremented to make space for the exception regs. This verification code is small, and we can minimize its impact by placing it directly in the vectors. To avoid redundant modification of the SP, we also need to move the initial decrement of the SP into the vectors. As a preparatory step, this patch introduces kernel_ventry, which performs this decrement, and updates the entry code accordingly. Subsequent patches will fold SP verification into kernel_ventry. There should be no functional change as a result of this patch. Signed-off-by: Ard Biesheuvel [Mark: turn into prep patch, expand commit msg] Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/kernel/entry.S | 47 ++++++++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 21 deletions(-) -- 1.9.1 diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index f31c7b2..58eba94 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -69,8 +69,13 @@ #define BAD_FIQ 2 #define BAD_ERROR 3 - .macro kernel_entry, el, regsize = 64 + .macro kernel_ventry label + .align 7 sub sp, sp, #S_FRAME_SIZE + b \label + .endm + + .macro kernel_entry, el, regsize = 64 .if \regsize == 32 mov w0, w0 // zero upper 32 bits of x0 .endif @@ -319,31 +324,31 @@ tsk .req x28 // current thread_info .align 11 ENTRY(vectors) - ventry el1_sync_invalid // Synchronous EL1t - ventry el1_irq_invalid // IRQ EL1t - ventry el1_fiq_invalid // FIQ EL1t - ventry el1_error_invalid // Error EL1t + kernel_ventry el1_sync_invalid // Synchronous EL1t + kernel_ventry el1_irq_invalid // IRQ EL1t + kernel_ventry el1_fiq_invalid // FIQ EL1t + kernel_ventry el1_error_invalid // Error EL1t - ventry el1_sync // Synchronous EL1h - ventry el1_irq // IRQ EL1h - ventry el1_fiq_invalid // FIQ EL1h - ventry el1_error_invalid // Error EL1h + kernel_ventry el1_sync // Synchronous EL1h + kernel_ventry el1_irq // IRQ EL1h + kernel_ventry el1_fiq_invalid // FIQ EL1h + kernel_ventry el1_error_invalid // Error EL1h - ventry el0_sync // Synchronous 64-bit EL0 - ventry el0_irq // IRQ 64-bit EL0 - ventry el0_fiq_invalid // FIQ 64-bit EL0 - ventry el0_error_invalid // Error 64-bit EL0 + kernel_ventry el0_sync // Synchronous 64-bit EL0 + kernel_ventry el0_irq // IRQ 64-bit EL0 + kernel_ventry el0_fiq_invalid // FIQ 64-bit EL0 + kernel_ventry el0_error_invalid // Error 64-bit EL0 #ifdef CONFIG_COMPAT - ventry el0_sync_compat // Synchronous 32-bit EL0 - ventry el0_irq_compat // IRQ 32-bit EL0 - ventry el0_fiq_invalid_compat // FIQ 32-bit EL0 - ventry el0_error_invalid_compat // Error 32-bit EL0 + kernel_ventry el0_sync_compat // Synchronous 32-bit EL0 + kernel_ventry el0_irq_compat // IRQ 32-bit EL0 + kernel_ventry el0_fiq_invalid_compat // FIQ 32-bit EL0 + kernel_ventry el0_error_invalid_compat // Error 32-bit EL0 #else - ventry el0_sync_invalid // Synchronous 32-bit EL0 - ventry el0_irq_invalid // IRQ 32-bit EL0 - ventry el0_fiq_invalid // FIQ 32-bit EL0 - ventry el0_error_invalid // Error 32-bit EL0 + kernel_ventry el0_sync_invalid // Synchronous 32-bit EL0 + kernel_ventry el0_irq_invalid // IRQ 32-bit EL0 + kernel_ventry el0_fiq_invalid // FIQ 32-bit EL0 + kernel_ventry el0_error_invalid // Error 32-bit EL0 #endif END(vectors) From patchwork Tue Aug 15 12:50:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110149 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439593obb; Tue, 15 Aug 2017 05:53:06 -0700 (PDT) X-Received: by 10.84.132.13 with SMTP id 13mr31989306ple.42.1502801586645; Tue, 15 Aug 2017 05:53:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801586; cv=none; d=google.com; s=arc-20160816; b=e8qR16juhRIQT8P5QhfsGzs7L3stKXx5LE44umVvNWPkWq1BRiriZSFXlY3/RW9PzO fVqGTm7hcNVbBIzeORb/GrjHE+CYJmrUrATZ8Djlra5Cp9rV5YbFUYwC1xC4cl/VsJAX ceHoUDaU9Fr6+gmlAaE81DT2FxI2/pPPvfUK6CQyx+tgFSppk8aP7cFu4VigSA2Nb5uX FPK0Hp8Qk0cun43kMDxwFf+jnlVd6gfUK4bnCZxzTKmOJLpp6UykBHuzOCl9tpvKA0SL zotOT/T5j53JILKPiLIW8QYFRlwYS+KthI/7vzW8UlXOYWWrJ+xfTcxcLpDWcGR4lEbS dIMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=fpRpZkbtKY+BDSp7ztD9RhxkFVKMiZh6KAbEjsiKDWU=; b=vT/nKIPaRQte0YXDB9W0bgUvIuY4oQCKJ9qJ0sq/z7DMGESbQmdPajPSuUJeBVzDCZ cMrE4rmgtHijSGkznoQ2RfoK0s/cqicX/QH+zZ7KDdmOo9zUYq/NHXoRgkrPbKFetIQT hrc6Dz8xhbvNT1H1TXF1+4du6wr3Rhf79dA1sEr+g8kJMi9hRxhudzsaGXHUYa1nbPV9 QGDdcI7zV/qWqfB6xDC5sEi7fKdUpQzQdCpkDFiHXfDcGZB3yE3u4TZcGHbz2JEQ5yN8 8YOLR67u+s5j42RobuMiMnmVmJn8UDJsRKCRgPBkeyy3XLRJZPb/YYy7GYs/LgSfH+jn mnYA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b68si5504932pfc.322.2017.08.15.05.53.06; Tue, 15 Aug 2017 05:53:06 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752833AbdHOMxB (ORCPT + 25 others); Tue, 15 Aug 2017 08:53:01 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51580 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752756AbdHOMxA (ORCPT ); Tue, 15 Aug 2017 08:53:00 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D0EC3169F; Tue, 15 Aug 2017 05:52:59 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id BDD0A3F540; Tue, 15 Aug 2017 05:52:57 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 10/14] arm64: assembler: allow adr_this_cpu to use the stack pointer Date: Tue, 15 Aug 2017 13:50:45 +0100 Message-Id: <1502801449-29246-11-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Ard Biesheuvel Given that adr_this_cpu already requires a temp register in addition to the destination register, tweak the instruction sequence so that sp may be used as well. This will simplify switching to per-cpu stacks in subsequent patches. While this limits the range of adr_this_cpu, to +/-4GiB, we don't currently use adr_this_cpu in modules, and this is not problematic for the main kernel image. Signed-off-by: Ard Biesheuvel [Mark: add more commit text] Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/assembler.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 610a420..2f2bd51 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -230,12 +230,18 @@ .endm /* - * @dst: Result of per_cpu(sym, smp_processor_id()) + * @dst: Result of per_cpu(sym, smp_processor_id()), can be SP for + * non-module code * @sym: The name of the per-cpu variable * @tmp: scratch register */ .macro adr_this_cpu, dst, sym, tmp +#ifndef MODULE + adrp \tmp, \sym + add \dst, \tmp, #:lo12:\sym +#else adr_l \dst, \sym +#endif mrs \tmp, tpidr_el1 add \dst, \dst, \tmp .endm From patchwork Tue Aug 15 12:50:46 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110150 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439608obb; Tue, 15 Aug 2017 05:53:07 -0700 (PDT) X-Received: by 10.98.160.146 with SMTP id p18mr28716307pfl.322.1502801587872; Tue, 15 Aug 2017 05:53:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801587; cv=none; d=google.com; s=arc-20160816; b=Z2caEYv1k14GHSe3xdfI6FHUBMiUPmXan8A1sNmX/X2FjdBItEpBXTPvlDWr5ssayd d3gstmY4+2I46qD/n+joWKU30Yjpkf1NSZ4Shvx7ZYt2Kn83HSdLKyDTM9o0BRV3DI+h PLvHe0dA0yYGEUd8wAmxplgASUOFhr9fE6rnKS/8FNBuk09vTS9HPfP4VCPMosrPl6o5 L3pibQkBMZw1cIdVhjsq/VTtZncWld1Ji3SffE1Vy52WzJpg4ZhwTADAI1GEyjvF5fvD 1pwf5PLcEXz5yMaqS7R7XJDneZs2sZINJ7GNzdNzwVA8arlksTMsnUUGTuJBja2fYcMY D8Pw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=z7xMWS7KOL3/NKi2FkdpMmBh0y7x0vffhggJ6JY98Sw=; b=a6ak+WxzPkx5TkiCMAwk2RMwlhUxNSixQDJtfiMq5r3LCegGIkYyGw/oJBRCuH9fuH xzcxljY+kM5t1sbrTvNNZjb4wIg8qoNZO6FDsKyQq6+JjM8aJ18h+lesF+AYTpxdIAGP HAUiLBh7vwjxGbcLQwsknPGtk3BfD6rhvkO5CB6w/ZdXPUIn82KL3kEsrM7WcuG/P9VW dzlgBIA0yKwyH7h5FBvd1DbbT1tB2vOHdk1J+Yz4kwpByeAdeb7DdIvuLlpj9MrvtOX/ dD87eEUqTBFycP0XL6cPBe2D+gpkFgmiTMa+LF+aWCI5r+037r2daBE4frFAlu4EN7LM d4Hg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g6si6158954plm.924.2017.08.15.05.53.07; Tue, 15 Aug 2017 05:53:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752851AbdHOMxE (ORCPT + 25 others); Tue, 15 Aug 2017 08:53:04 -0400 Received: from foss.arm.com ([217.140.101.70]:51592 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752756AbdHOMxC (ORCPT ); Tue, 15 Aug 2017 08:53:02 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8F7A916A3; Tue, 15 Aug 2017 05:53:02 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7D4653F540; Tue, 15 Aug 2017 05:53:00 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 11/14] arm64: use an irq stack pointer Date: Tue, 15 Aug 2017 13:50:46 +0100 Message-Id: <1502801449-29246-12-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We allocate our IRQ stacks using a percpu array. This allows us to generate our IRQ stack pointers with adr_this_cpu, but bloats the kernel Image with the boot CPU's IRQ stack. Additionally, these are packed with other percpu variables, and aren't guaranteed to have guard pages. When we enable VMAP_STACK we'll want to vmap our IRQ stacks also, in order to provide guard pages and to permit more stringent alignment requirements. Doing so will require that we use a percpu pointer to each IRQ stack, rather than allocating a percpu IRQ stack in the kernel image. This patch updates our IRQ stack code to use a percpu pointer to the base of each IRQ stack. This will allow us to change the way the stack is allocated with minimal changes elsewhere. In some cases we may try to backtrace before the IRQ stack pointers are initialised, so on_irq_stack() is updated to account for this. In testing with cyclictest, there was no measureable difference between using adr_this_cpu (for irq_stack) and ldr_this_cpu (for irq_stack_ptr) in the IRQ entry path. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/stacktrace.h | 7 +++++-- arch/arm64/kernel/entry.S | 2 +- arch/arm64/kernel/irq.c | 10 ++++++++++ 3 files changed, 16 insertions(+), 3 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index 000e2418..4c68d8a 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -36,13 +36,16 @@ extern void walk_stackframe(struct task_struct *tsk, struct stackframe *frame, int (*fn)(struct stackframe *, void *), void *data); extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); -DECLARE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack); +DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); static inline bool on_irq_stack(unsigned long sp) { - unsigned long low = (unsigned long)raw_cpu_ptr(irq_stack); + unsigned long low = (unsigned long)raw_cpu_read(irq_stack_ptr); unsigned long high = low + IRQ_STACK_SIZE; + if (!low) + return false; + return (low <= sp && sp < high); } diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 58eba94..5234886 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -276,7 +276,7 @@ alternative_else_nop_endif and x25, x25, #~(THREAD_SIZE - 1) cbnz x25, 9998f - adr_this_cpu x25, irq_stack, x26 + ldr_this_cpu x25, irq_stack_ptr, x26 mov x26, #IRQ_STACK_SIZE add x26, x25, x26 diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index 2386b26..5141282 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -32,6 +32,7 @@ /* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */ DEFINE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack) __aligned(16); +DEFINE_PER_CPU(unsigned long *, irq_stack_ptr); int arch_show_interrupts(struct seq_file *p, int prec) { @@ -50,8 +51,17 @@ void __init set_handle_irq(void (*handle_irq)(struct pt_regs *)) handle_arch_irq = handle_irq; } +static void init_irq_stacks(void) +{ + int cpu; + + for_each_possible_cpu(cpu) + per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu); +} + void __init init_IRQ(void) { + init_irq_stacks(); irqchip_init(); if (!handle_arch_irq) panic("No interrupt controller found."); From patchwork Tue Aug 15 12:50:47 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110151 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439703obb; Tue, 15 Aug 2017 05:53:13 -0700 (PDT) X-Received: by 10.99.122.94 with SMTP id j30mr27178083pgn.188.1502801593376; Tue, 15 Aug 2017 05:53:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801593; cv=none; d=google.com; s=arc-20160816; b=g8xAm7pSw/P44i8+bPwZ8lN8CDQ4oiZ+60JQaAbOilsLcx2ZS9oe9YFcmU99M3sydX cAtD/bnfvrW6OoCGkwzlF7xSrBW/q4QJx38aYvS56dmauul6CaHUPjMtsDjOAvppk5JL 6DUjyor48oZjXDhIiOYlrV5eJdbYguUceJcewQHegVEI3oumUjOE++1/Z+EzJshJwdqd 9ynURQpX3/vTV92Q587WfkQ1AqG3/5XAa76QYYH+jjqv94JQr2+LIyBY/EIz/9wVjzQ5 CiBxkngq+y7ldxoS/nNR25GJZAtCZZpN7EpVSfVTJ0UbkoNyo3gJSGsG/1zNeEL8eJBv HPNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=gy1LB7CPJW/aI9u9/+B18CXyhQgLyzmOu1sITbqEY/w=; b=XQ9o4b/G2DfjZd32g24mv+3G23lYyl0RRrN33KfI8cvaX/a4NUWEPuqqGq0ttjI+25 7RGRjloJdqktslVtKgDX4BvkkdHVnupNwTqPKK/O2REK89H/H4+6zTs5K3FzgEOpVxuc qWYqu09HpH5KN3pdMGOB5FL2Mywtvfu/mBb0okA7e+qgQT+691Q3tzYvxBKnqYULwZc6 0zDGOhfKVZpZ9Rpc4AFE1G+XP+LWLZcoZHfKCI5+HDYA6m1zvz/9p6Eu4t15sdMzUjRF MH3U5BiuT+KVsfdc2NDrUnz/2aK2+Kb/RuL0AalwqKhJWFQVB+2it2YX++rPwMJRmcir c6uQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g6si6158954plm.924.2017.08.15.05.53.13; Tue, 15 Aug 2017 05:53:13 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752871AbdHOMxJ (ORCPT + 25 others); Tue, 15 Aug 2017 08:53:09 -0400 Received: from foss.arm.com ([217.140.101.70]:51628 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752756AbdHOMxF (ORCPT ); Tue, 15 Aug 2017 08:53:05 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 58C1C1596; Tue, 15 Aug 2017 05:53:05 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3CCD93F540; Tue, 15 Aug 2017 05:53:03 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 12/14] arm64: add basic VMAP_STACK support Date: Tue, 15 Aug 2017 13:50:47 +0100 Message-Id: <1502801449-29246-13-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch enables arm64 to be built with vmap'd task and IRQ stacks. As vmap'd stacks are mapped at page granularity, stacks must be a multiple of PAGE_SIZE. This means that a 64K page kernel must use stacks of at least 64K in size. To minimize the increase in Image size, IRQ stacks are dynamically allocated at boot time, rather than embedding the boot CPU's IRQ stack in the kernel image. This patch was co-authored by Ard Biesheuvel and Mark Rutland. Signed-off-by: Ard Biesheuvel Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/efi.h | 7 ++++++- arch/arm64/include/asm/memory.h | 23 ++++++++++++++++++++++- arch/arm64/kernel/irq.c | 30 ++++++++++++++++++++++++++++-- arch/arm64/kernel/vmlinux.lds.S | 2 +- 5 files changed, 58 insertions(+), 5 deletions(-) -- 1.9.1 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index dfd9086..d66f9db 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -75,6 +75,7 @@ config ARM64 select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE + select HAVE_ARCH_VMAP_STACK select HAVE_ARM_SMCCC select HAVE_EBPF_JIT select HAVE_C_RECORDMCOUNT diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index 0e8cc3b..2b1e5de 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -49,7 +49,12 @@ */ #define EFI_FDT_ALIGN SZ_2M /* used by allocate_new_fdt_and_exit_boot() */ -#define EFI_KIMG_ALIGN SEGMENT_ALIGN +/* + * In some configurations (e.g. VMAP_STACK && 64K pages), stacks built into the + * kernel need greater alignment than we require the segments to be padded to. + */ +#define EFI_KIMG_ALIGN \ + (SEGMENT_ALIGN > THREAD_ALIGN ? SEGMENT_ALIGN : THREAD_ALIGN) /* on arm64, the FDT may be located anywhere in system RAM */ static inline unsigned long efi_get_max_fdt_addr(unsigned long dram_base) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 7fa6ad4..c5cd2c5 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -102,7 +102,17 @@ #define KASAN_SHADOW_SIZE (0) #endif -#define THREAD_SHIFT 14 +#define MIN_THREAD_SHIFT 14 + +/* + * VMAP'd stacks are allocated at page granularity, so we must ensure that such + * stacks are a multiple of page size. + */ +#if defined(CONFIG_VMAP_STACK) && (MIN_THREAD_SHIFT < PAGE_SHIFT) +#define THREAD_SHIFT PAGE_SHIFT +#else +#define THREAD_SHIFT MIN_THREAD_SHIFT +#endif #if THREAD_SHIFT >= PAGE_SHIFT #define THREAD_SIZE_ORDER (THREAD_SHIFT - PAGE_SHIFT) @@ -110,6 +120,17 @@ #define THREAD_SIZE (UL(1) << THREAD_SHIFT) +/* + * By aligning VMAP'd stacks to 2 * THREAD_SIZE, we can detect overflow by + * checking sp & (1 << THREAD_SHIFT), which we can do cheaply in the entry + * assembly. + */ +#ifdef CONFIG_VMAP_STACK +#define THREAD_ALIGN (2 * THREAD_SIZE) +#else +#define THREAD_ALIGN THREAD_SIZE +#endif + #define IRQ_STACK_SIZE THREAD_SIZE /* diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index 5141282..713561e 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -23,15 +23,15 @@ #include #include +#include #include #include #include #include +#include unsigned long irq_err_count; -/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */ -DEFINE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack) __aligned(16); DEFINE_PER_CPU(unsigned long *, irq_stack_ptr); int arch_show_interrupts(struct seq_file *p, int prec) @@ -51,6 +51,31 @@ void __init set_handle_irq(void (*handle_irq)(struct pt_regs *)) handle_arch_irq = handle_irq; } +#ifdef CONFIG_VMAP_STACK +static void init_irq_stacks(void) +{ + int cpu; + unsigned long *p; + + for_each_possible_cpu(cpu) { + /* + * To ensure that VMAP'd stack overflow detection works + * correctly, the IRQ stacks need to have the same + * alignment as other stacks. + */ + p = __vmalloc_node_range(IRQ_STACK_SIZE, THREAD_ALIGN, + VMALLOC_START, VMALLOC_END, + THREADINFO_GFP, PAGE_KERNEL, + 0, cpu_to_node(cpu), + __builtin_return_address(0)); + + per_cpu(irq_stack_ptr, cpu) = p; + } +} +#else +/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */ +DEFINE_PER_CPU_ALIGNED(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack); + static void init_irq_stacks(void) { int cpu; @@ -58,6 +83,7 @@ static void init_irq_stacks(void) for_each_possible_cpu(cpu) per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu); } +#endif void __init init_IRQ(void) { diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7156538..fe56c26 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -176,7 +176,7 @@ SECTIONS _data = .; _sdata = .; - RW_DATA_SECTION(L1_CACHE_BYTES, PAGE_SIZE, THREAD_SIZE) + RW_DATA_SECTION(L1_CACHE_BYTES, PAGE_SIZE, THREAD_ALIGN) /* * Data written with the MMU off but read with the MMU on requires From patchwork Tue Aug 15 12:50:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110153 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5440218obb; Tue, 15 Aug 2017 05:53:46 -0700 (PDT) X-Received: by 10.99.176.4 with SMTP id h4mr26753296pgf.300.1502801626218; Tue, 15 Aug 2017 05:53:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801626; cv=none; d=google.com; s=arc-20160816; b=fkc2T3MbJKcbbG40UFVs/EhrpGuPz4Ha6PRzuFiZGKUi9SlBhGG1HF4XrzSgmDy2M5 EOr33y0wZWfqPK2Kc6kEJ5ZisxBzpy46obKrXz7dvqjKFD7wD3xQH84MAu50fD1gPjlS YA5Sq7waddSvp03ZtLNp788oo5AO5vY58XfpKRjKnht4MqD0MXFUXZtnI3hJH2j3CNF7 VTDNRTXvinrxiExUHqlemUwbORgnsPnU/VcPd1d1vBEJI7HIuJRf8IMoXWyjzQXVRaDV tu7YFYG1nRWlfZagsaLZbfSI8cPIDMBIcu+aWXx0SxroOwQ6MXxoSyN+4LFY8Sr447oG 10ww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=NXnSjGXKxLst34agjQNzMCrXR12YpUbRfgTjX4wZwxs=; b=LDPS0HXzC0C/21M/vmtqYdkKHG94G7geSqWUUXHPVwqLUqwV6Ik43oo3RWzAY/v7s4 JAu43mzh09sgEwgK68vDlh+8am0XloJbP7u9lp0hAX6KTLQnzQ8jOsBKsGL1IbEpWFQp HY9jmH1DuqW3rgorU2LVIj5b2XWBGICZ4LHX/OCUZb46lNxVF+BZboMCyNDJ7hhk1oE0 lY9Nc8HKR6OzQScRFVe8WVxDg+2Zsz+LKqaEvB2jMQ/rcwdJcd2823kVsOjlU83viqqL 6f8Dl8m2EQC23fj7+AiJl4AThUmi7YxRy/WPjfJ946hDVi1zoYH4BxLnTT67dS7AhWTL h/7w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t12si5534414pfj.329.2017.08.15.05.53.45; Tue, 15 Aug 2017 05:53:46 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752992AbdHOMxn (ORCPT + 25 others); Tue, 15 Aug 2017 08:53:43 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51638 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752852AbdHOMxI (ORCPT ); Tue, 15 Aug 2017 08:53:08 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3BFDC15BF; Tue, 15 Aug 2017 05:53:08 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1FAFC3F540; Tue, 15 Aug 2017 05:53:05 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 13/14] arm64: add on_accessible_stack() Date: Tue, 15 Aug 2017 13:50:48 +0100 Message-Id: <1502801449-29246-14-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Both unwind_frame() and dump_backtrace() try to check whether a stack address is sane to access, with very similar logic. Both will need updating in order to handle overflow stacks. Factor out this logic into a helper, so that we can avoid further duplication when we add overflow stacks. Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/stacktrace.h | 16 ++++++++++++++++ arch/arm64/kernel/stacktrace.c | 7 +------ arch/arm64/kernel/traps.c | 3 +-- 3 files changed, 18 insertions(+), 8 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index 4c68d8a..92ddb6d 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -57,4 +57,20 @@ static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) return (low <= sp && sp < high); } +/* + * We can only safely access per-cpu stacks from current in a non-preemptible + * context. + */ +static inline bool on_accessible_stack(struct task_struct *tsk, unsigned long sp) +{ + if (on_task_stack(tsk, sp)) + return true; + if (tsk != current || preemptible()) + return false; + if (on_irq_stack(sp)) + return true; + + return false; +} + #endif /* __ASM_STACKTRACE_H */ diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c index 35588ca..3144584 100644 --- a/arch/arm64/kernel/stacktrace.c +++ b/arch/arm64/kernel/stacktrace.c @@ -50,12 +50,7 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) if (!tsk) tsk = current; - /* - * Switching between stacks is valid when tracing current and in - * non-preemptible context. - */ - if (!(tsk == current && !preemptible() && on_irq_stack(fp)) && - !on_task_stack(tsk, fp)) + if (!on_accessible_stack(tsk, fp)) return -EINVAL; frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp)); diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 9633773..d01c598 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -193,8 +193,7 @@ void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk) if (in_entry_text(frame.pc)) { stack = frame.fp - offsetof(struct pt_regs, stackframe); - if (on_task_stack(tsk, stack) || - (tsk == current && !preemptible() && on_irq_stack(stack))) + if (on_accessible_stack(tsk, stack)) dump_mem("", "Exception stack", stack, stack + sizeof(struct pt_regs)); } From patchwork Tue Aug 15 12:50:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 110152 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp5439795obb; Tue, 15 Aug 2017 05:53:18 -0700 (PDT) X-Received: by 10.99.127.14 with SMTP id a14mr27338557pgd.112.1502801598305; Tue, 15 Aug 2017 05:53:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502801598; cv=none; d=google.com; s=arc-20160816; b=R/yv18+QiTVYtDVetxiTNJ39GfSK13n7M2HCXsTYGUzUC0GqBIYhMvt3Tc7meI+y77 KNsAz28lBUpVzqGDwV8JC3Gp7FotCZ9hRrSqqipJCj60FIFbmEhLGpYJGGg1uTpb58FS s+0xOXZIuxwBArFSGjqWQ+txKztTTxCIrX3OP6WL79thKxo7dGOs/FhEbFcPw++/BLGK dgrmAI+u4ekIhSgfh25AOX/iaCEguVOdwAzpAJv4LRMV45i4Z7kbQINonTSWul7RjAPB xj2VJMoYtAcDf7JEy68zuKCGwFrUVCJ+XiZDn1tOeRxCnRagyPBxcm3u1EOzSCLxboOm u1ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=fwwalDwMfdEDpVm69qwKTHrMVIXDP/aTG+rI7LdhWtM=; b=NAtdXIV6C95rdA42mOniG9xqgb7S9vpx4TC9bx87MQp6so56yf9aRV2DqMZrMbkvEn b92AcaSIjM1FNaAt7FMGWHNHR30qAIDgS/G+EKo2MfLuaqKVE5vS4bkJK4uDBYwqBBmZ mxdi6iAoyamzO6L6KR6e/PbvwupRcAbdv3SU3UFTwPk/tdaFssdUc1vb3IfMXmsAFIFI 5d55Kdudx+0t8ZlSAdvR81na1cz9DzDSdlo1Gc3bxXMLuPpifcV+i88yvS2y/rbx0oZK PiuzCQZ6vWPmmwUC+XWLFFLpzTrWu6Wtu3/zrPqXgPYL0GUttLF1Y9Du1az2cAIbxxeY 9Oiw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a15si5510743pfl.347.2017.08.15.05.53.17; Tue, 15 Aug 2017 05:53:18 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752893AbdHOMxP (ORCPT + 25 others); Tue, 15 Aug 2017 08:53:15 -0400 Received: from foss.arm.com ([217.140.101.70]:51658 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752440AbdHOMxL (ORCPT ); Tue, 15 Aug 2017 08:53:11 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1CC8D16A0; Tue, 15 Aug 2017 05:53:11 -0700 (PDT) Received: from leverpostej.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0A1283F540; Tue, 15 Aug 2017 05:53:08 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: ard.biesheuvel@linaro.org, catalin.marinas@arm.com, james.morse@arm.com, labbott@redhat.com, linux-kernel@vger.kernel.org, luto@amacapital.net, mark.rutland@arm.com, matt@codeblueprint.co.uk, will.deacon@arm.com, kernel-hardening@lists.openwall.com, keescook@chromium.org Subject: [PATCHv2 14/14] arm64: add VMAP_STACK overflow detection Date: Tue, 15 Aug 2017 13:50:49 +0100 Message-Id: <1502801449-29246-15-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> References: <1502801449-29246-1-git-send-email-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch adds stack overflow detection to arm64, usable when vmap'd stacks are in use. Overflow is detected in a small preamble executed for each exception entry, which checks whether there is enough space on the current stack for the general purpose registers to be saved. If there is not enough space, the overflow handler is invoked on a per-cpu overflow stack. This approach preserves the original exception information in ESR_EL1 (and where appropriate, FAR_EL1). Task and IRQ stacks are aligned to double their size, enabling overflow to be detected with a single bit test. For example, a 16K stack is aligned to 32K, ensuring that bit 14 of the SP must be zero. On an overflow (or underflow), this bit is flipped. Thus, overflow (of less than the size of the stack) can be detected by testing whether this bit is set. The overflow check is performed before any attempt is made to access the stack, avoiding recursive faults (and the loss of exception information these would entail). As logical operations cannot be performed on the SP directly, the SP is temporarily swapped with a general purpose register using arithmetic operations to enable the test to be performed. This gives us a useful error message on stack overflow, as can be trigger with the LKDTM overflow test: [ 305.388749] lkdtm: Performing direct entry OVERFLOW [ 305.395444] Insufficient stack space to handle exception! [ 305.395482] ESR: 0x96000047 -- DABT (current EL) [ 305.399890] FAR: 0xffff00000a5e7f30 [ 305.401315] Task stack: [0xffff00000a5e8000..0xffff00000a5ec000] [ 305.403815] IRQ stack: [0xffff000008000000..0xffff000008004000] [ 305.407035] Overflow stack: [0xffff80003efce4e0..0xffff80003efcf4e0] [ 305.409622] CPU: 0 PID: 1219 Comm: sh Not tainted 4.13.0-rc3-00021-g9636aea #5 [ 305.412785] Hardware name: linux,dummy-virt (DT) [ 305.415756] task: ffff80003d051c00 task.stack: ffff00000a5e8000 [ 305.419221] PC is at recursive_loop+0x10/0x48 [ 305.421637] LR is at recursive_loop+0x38/0x48 [ 305.423768] pc : [] lr : [] pstate: 40000145 [ 305.428020] sp : ffff00000a5e7f50 [ 305.430469] x29: ffff00000a5e8350 x28: ffff80003d051c00 [ 305.433191] x27: ffff000008981000 x26: ffff000008f80400 [ 305.439012] x25: ffff00000a5ebeb8 x24: ffff00000a5ebeb8 [ 305.440369] x23: ffff000008f80138 x22: 0000000000000009 [ 305.442241] x21: ffff80003ce65000 x20: ffff000008f80188 [ 305.444552] x19: 0000000000000013 x18: 0000000000000006 [ 305.446032] x17: 0000ffffa2601280 x16: ffff0000081fe0b8 [ 305.448252] x15: ffff000008ff546d x14: 000000000047a4c8 [ 305.450246] x13: ffff000008ff7872 x12: 0000000005f5e0ff [ 305.452953] x11: ffff000008ed2548 x10: 000000000005ee8d [ 305.454824] x9 : ffff000008545380 x8 : ffff00000a5e8770 [ 305.457105] x7 : 1313131313131313 x6 : 00000000000000e1 [ 305.459285] x5 : 0000000000000000 x4 : 0000000000000000 [ 305.461781] x3 : 0000000000000000 x2 : 0000000000000400 [ 305.465119] x1 : 0000000000000013 x0 : 0000000000000012 [ 305.467724] Kernel panic - not syncing: kernel stack overflow [ 305.470561] CPU: 0 PID: 1219 Comm: sh Not tainted 4.13.0-rc3-00021-g9636aea #5 [ 305.473325] Hardware name: linux,dummy-virt (DT) [ 305.475070] Call trace: [ 305.476116] [] dump_backtrace+0x0/0x378 [ 305.478991] [] show_stack+0x14/0x20 [ 305.481237] [] dump_stack+0x98/0xb8 [ 305.483294] [] panic+0x118/0x280 [ 305.485673] [] nmi_panic+0x6c/0x70 [ 305.486216] [] handle_bad_stack+0x118/0x128 [ 305.486612] Exception stack(0xffff80003efcf3a0 to 0xffff80003efcf4e0) [ 305.487334] f3a0: 0000000000000012 0000000000000013 0000000000000400 0000000000000000 [ 305.488025] f3c0: 0000000000000000 0000000000000000 00000000000000e1 1313131313131313 [ 305.488908] f3e0: ffff00000a5e8770 ffff000008545380 000000000005ee8d ffff000008ed2548 [ 305.489403] f400: 0000000005f5e0ff ffff000008ff7872 000000000047a4c8 ffff000008ff546d [ 305.489759] f420: ffff0000081fe0b8 0000ffffa2601280 0000000000000006 0000000000000013 [ 305.490256] f440: ffff000008f80188 ffff80003ce65000 0000000000000009 ffff000008f80138 [ 305.490683] f460: ffff00000a5ebeb8 ffff00000a5ebeb8 ffff000008f80400 ffff000008981000 [ 305.491051] f480: ffff80003d051c00 ffff00000a5e8350 ffff00000859f358 ffff00000a5e7f50 [ 305.491444] f4a0: ffff00000859f330 0000000040000145 0000000000000000 0000000000000000 [ 305.492008] f4c0: 0001000000000000 0000000000000000 ffff00000a5e8350 ffff00000859f330 [ 305.493063] [] __bad_stack+0x88/0x8c [ 305.493396] [] recursive_loop+0x10/0x48 [ 305.493731] [] recursive_loop+0x38/0x48 [ 305.494088] [] recursive_loop+0x38/0x48 [ 305.494425] [] recursive_loop+0x38/0x48 [ 305.494649] [] recursive_loop+0x38/0x48 [ 305.494898] [] recursive_loop+0x38/0x48 [ 305.495205] [] recursive_loop+0x38/0x48 [ 305.495453] [] recursive_loop+0x38/0x48 [ 305.495708] [] recursive_loop+0x38/0x48 [ 305.496000] [] recursive_loop+0x38/0x48 [ 305.496302] [] recursive_loop+0x38/0x48 [ 305.496644] [] recursive_loop+0x38/0x48 [ 305.496894] [] recursive_loop+0x38/0x48 [ 305.497138] [] recursive_loop+0x38/0x48 [ 305.497325] [] lkdtm_OVERFLOW+0x14/0x20 [ 305.497506] [] lkdtm_do_action+0x1c/0x28 [ 305.497786] [] direct_entry+0xe0/0x170 [ 305.498095] [] full_proxy_write+0x60/0xa8 [ 305.498387] [] __vfs_write+0x1c/0x128 [ 305.498679] [] vfs_write+0xa0/0x1b0 [ 305.498926] [] SyS_write+0x44/0xa0 [ 305.499182] Exception stack(0xffff00000a5ebec0 to 0xffff00000a5ec000) [ 305.499429] bec0: 0000000000000001 000000001c4cf5e0 0000000000000009 000000001c4cf5e0 [ 305.499674] bee0: 574f4c465245564f 0000000000000000 0000000000000000 8000000080808080 [ 305.499904] bf00: 0000000000000040 0000000000000038 fefefeff1b4bc2ff 7f7f7f7f7f7fff7f [ 305.500189] bf20: 0101010101010101 0000000000000000 000000000047a4c8 0000000000000038 [ 305.500712] bf40: 0000000000000000 0000ffffa2601280 0000ffffc63f6068 00000000004b5000 [ 305.501241] bf60: 0000000000000001 000000001c4cf5e0 0000000000000009 000000001c4cf5e0 [ 305.501791] bf80: 0000000000000020 0000000000000000 00000000004b5000 000000001c4cc458 [ 305.502314] bfa0: 0000000000000000 0000ffffc63f7950 000000000040a3c4 0000ffffc63f70e0 [ 305.502762] bfc0: 0000ffffa2601268 0000000080000000 0000000000000001 0000000000000040 [ 305.503207] bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 305.503680] [] el0_svc_naked+0x24/0x28 [ 305.504720] Kernel Offset: disabled [ 305.505189] CPU features: 0x002082 [ 305.505473] Memory Limit: none [ 305.506181] ---[ end Kernel panic - not syncing: kernel stack overflow This patch was co-authored by Ard Biesheuvel and Mark Rutland. Signed-off-by: Ard Biesheuvel Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Laura Abbott Cc: Will Deacon --- arch/arm64/include/asm/memory.h | 2 ++ arch/arm64/include/asm/stacktrace.h | 16 +++++++++ arch/arm64/kernel/entry.S | 70 +++++++++++++++++++++++++++++++++++++ arch/arm64/kernel/traps.c | 39 +++++++++++++++++++++ 4 files changed, 127 insertions(+) -- 1.9.1 diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index c5cd2c5..1a025b7 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -133,6 +133,8 @@ #define IRQ_STACK_SIZE THREAD_SIZE +#define OVERFLOW_STACK_SIZE SZ_4K + /* * Alignment of kernel segments (e.g. .text, .data). */ diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h index 92ddb6d..6ad3077 100644 --- a/arch/arm64/include/asm/stacktrace.h +++ b/arch/arm64/include/asm/stacktrace.h @@ -57,6 +57,20 @@ static inline bool on_task_stack(struct task_struct *tsk, unsigned long sp) return (low <= sp && sp < high); } +#ifdef CONFIG_VMAP_STACK +DECLARE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack); + +static inline bool on_overflow_stack(unsigned long sp) +{ + unsigned long low = (unsigned long)raw_cpu_ptr(overflow_stack); + unsigned long high = low + OVERFLOW_STACK_SIZE; + + return (low <= sp && sp < high); +} +#else +static inline bool on_overflow_stack(unsigned long sp) { return false; } +#endif + /* * We can only safely access per-cpu stacks from current in a non-preemptible * context. @@ -69,6 +83,8 @@ static inline bool on_accessible_stack(struct task_struct *tsk, unsigned long sp return false; if (on_irq_stack(sp)) return true; + if (on_overflow_stack(sp)) + return true; return false; } diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 5234886..3ef6e22 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -72,6 +72,48 @@ .macro kernel_ventry label .align 7 sub sp, sp, #S_FRAME_SIZE +#ifdef CONFIG_VMAP_STACK + /* + * Test whether the SP has overflowed, without corrupting a GPR. + * Task and IRQ stacks are aligned to (1 << THREAD_SHIFT). + */ + add sp, sp, x0 // sp' = sp + x0 + sub x0, sp, x0 // x0' = sp' - x0 = (sp + x0) - x0 = sp + tbnz x0, #THREAD_SHIFT, 0f + sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 + sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp + b \label + +0: + /* + * Either we've just detected an overflow, or we've taken an exception + * while on the overflow stack. Either way, we won't return to + * userspace, and can clobber EL0 registers to free up GPRs. + */ + + /* Stash the original SP (minus S_FRAME_SIZE) in tpidr_el0. */ + msr tpidr_el0, x0 + + /* Recover the original x0 value and stash it in tpidrro_el0 */ + sub x0, sp, x0 + msr tpidrro_el0, x0 + + /* Switch to the overflow stack */ + adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0 + + /* + * Check whether we were already on the overflow stack. This may happen + * after panic() re-enables interrupts. + */ + mrs x0, tpidr_el0 // sp of interrupted context + sub x0, sp, x0 // delta with top of overflow stack + tst x0, #~(OVERFLOW_STACK_SIZE - 1) // within range? + b.ne __bad_stack // no? -> bad stack pointer + + /* We were already on the overflow stack. Restore sp/x0 and carry on. */ + sub sp, sp, x0 + mrs x0, tpidrro_el0 +#endif b \label .endm @@ -352,6 +394,34 @@ ENTRY(vectors) #endif END(vectors) +#ifdef CONFIG_VMAP_STACK + /* + * We detected an overflow in kernel_ventry, which switched to the + * overflow stack. Stash the exception regs, and head to our overflow + * handler. + */ +__bad_stack: + /* Restore the original x0 value */ + mrs x0, tpidrro_el0 + + /* + * Store the original GPRs to the new stack. The orginal SP (minus + * S_FRAME_SIZE) was stashed in tpidr_el0 by kernel_ventry. + */ + sub sp, sp, #S_FRAME_SIZE + kernel_entry 1 + mrs x0, tpidr_el0 + add x0, x0, #S_FRAME_SIZE + str x0, [sp, #S_SP] + + /* Stash the regs for handle_bad_stack */ + mov x0, sp + + /* Time to die */ + bl handle_bad_stack + ASM_BUG() +#endif /* CONFIG_VMAP_STACK */ + /* * Invalid mode handlers */ diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index d01c598..2d59180 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include @@ -41,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -666,6 +668,43 @@ asmlinkage void bad_el0_sync(struct pt_regs *regs, int reason, unsigned int esr) force_sig_info(info.si_signo, &info, current); } +#ifdef CONFIG_VMAP_STACK + +DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack) + __aligned(16); + +asmlinkage void handle_bad_stack(struct pt_regs *regs) +{ + unsigned long tsk_stk = (unsigned long)current->stack; + unsigned long irq_stk = (unsigned long)this_cpu_read(irq_stack_ptr); + unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack); + unsigned int esr = read_sysreg(esr_el1); + unsigned long far = read_sysreg(far_el1); + + console_verbose(); + pr_emerg("Insufficient stack space to handle exception!"); + + pr_emerg("ESR: 0x%08x -- %s\n", esr, esr_get_class_string(esr)); + pr_emerg("FAR: 0x%016lx\n", far); + + pr_emerg("Task stack: [0x%016lx..0x%016lx]\n", + tsk_stk, tsk_stk + THREAD_SIZE); + pr_emerg("IRQ stack: [0x%016lx..0x%016lx]\n", + irq_stk, irq_stk + THREAD_SIZE); + pr_emerg("Overflow stack: [0x%016lx..0x%016lx]\n", + ovf_stk, ovf_stk + OVERFLOW_STACK_SIZE); + + __show_regs(regs); + + /* + * We use nmi_panic to limit the potential for recusive overflows, and + * to get a better stack trace. + */ + nmi_panic(NULL, "kernel stack overflow"); + cpu_park_loop(); +} +#endif + void __pte_error(const char *file, int line, unsigned long val) { pr_err("%s:%d: bad pte %016lx.\n", file, line, val);