From patchwork Mon Jul 2 18:11:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140850 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp249758ljj; Mon, 2 Jul 2018 11:14:58 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfqI+Mzs9Gc9u4yD3TB0hmpCi1VCXXoIBlKpA4w51UB/F5whWI4ovCu0YT6My2h1u8ZYTDV X-Received: by 2002:a62:170c:: with SMTP id 12-v6mr26419584pfx.139.1530555297958; Mon, 02 Jul 2018 11:14:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530555297; cv=none; d=google.com; s=arc-20160816; b=dnQQ3T5DEGVGsrq9Xr30v78hIemRn7x24y/RI3aeK8Dcx8s/zZqKPTh9pXocu9jG/u olBt/rx77pEk/udM3rT1VUshyZ4ZBsekcr1dsIHLN6PUqfGYpXVPvJSCPm86+MENQ3nF LDoYEcVJ4RuO+EUn2GCJf4e/C4KksjaYfbsQkbvxDIsXesVRWd8POZO/0F+/K+JiZYNO BDnTsaBabVo8lO+mlx4iKIcUlWC+aouO1ioqtbMo6qJVvV6taR3kSHRK9gUoPRAGQQdA yJzo1OfyFbjDmf10q6PjrkC1W8k4aG8KuHPD3LGHe0umrZoNOTNjhVBW6QsHG9LTEv4n ZXNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=bYIEjgYsFCspjdyZn23Ek0/9Xld2X4/AQrtzD7Bk1Ig=; b=XlYU7t7rNAQ/h9rm9nsl6UkTRjzzT0bAOVNxkDhqAoVC27L71tYQt0gTAP0tHs5XPJ QTXRNYsZ7xqEO9XYHAHs8U6vVp5ZJbHhhvnliGKXbOEXs2p0dMZxWlZRYYTiv/NqazwR 8jX+sURNDb+tlVC27HEFH3pj2IEjieHDWafTQ9noyv9kcIZyDz4VVby0U6vVKT8lRtCa CVckDNhROO4RX8LFlahm2elYJFI1hs8oh7IDfLJqK6vZNJNZZ2w9+ZOQmSZKBlUKowir sqWPVPZoJfQF71NtXwSeX7ZUJqxy5UWQTZ6IP9PdGJrn2sQpwndfky9L6HSLLLfutfww KcZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FkyoYASw; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 6-v6si16787191plb.409.2018.07.02.11.14.57; Mon, 02 Jul 2018 11:14:57 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FkyoYASw; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753355AbeGBSOz (ORCPT + 31 others); Mon, 2 Jul 2018 14:14:55 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:35747 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753328AbeGBSLz (ORCPT ); Mon, 2 Jul 2018 14:11:55 -0400 Received: by mail-wr0-f193.google.com with SMTP id h40-v6so5338872wrh.2 for ; Mon, 02 Jul 2018 11:11:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bYIEjgYsFCspjdyZn23Ek0/9Xld2X4/AQrtzD7Bk1Ig=; b=FkyoYASwtLuwL6Ys+RF95vTBeo00E/h6AYLejkixx4YIeQU0cTYq08FQDO/RhDyvl0 pjVysDdtmPuDQb5zTxUNpxwi9jKuDKYg4aUvAGVXc5v5b5+ozJKscnU+swtD2RuB2VXr MfR7HAW5r8vkudT3oYUv0kfmrzgEd+c5uvu1w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bYIEjgYsFCspjdyZn23Ek0/9Xld2X4/AQrtzD7Bk1Ig=; b=X5kZDyAdnlBBQFkv5LDd0wF3zKlroEVAxw1iyyxKg3CQzgGPQXAj34G1yyUzBKX1xs 2YETIgjS0MwS/5/SbXWhkzvbBzbd3hKqANiM+GTDpWvgI43F2BRx44887huYWtf+fRc0 rrJOVGJ7TLZw1PfptJQxq3ONTFsFTx7/n1tAvelWYxxzIPiTn5bRDF2cvILnE6eqD84I hgWXLjLW7GTcx6Wfd2XWViY59bcAEb1tizgOupcKWtxGKr2b7t3czT7LJAO882XWmicZ AcNkzoJrR7Cvq3a87jrRp0VVwbECWZSh8WlqKm7c6bGdvOQ7P7mzuJyIvD/g5YY//yki wZkA== X-Gm-Message-State: APt69E0Q6gy3blRGAyB3sGCBjn7RSSVtcU7AkV1KNX3LECvgGy8JX5iY winHotm1KZlEcaT6jEE49TjtWw== X-Received: by 2002:adf:de84:: with SMTP id w4-v6mr8884955wrl.270.1530555114143; Mon, 02 Jul 2018 11:11:54 -0700 (PDT) Received: from localhost.localdomain (151.21.90.92.rev.sfr.net. [92.90.21.151]) by smtp.gmail.com with ESMTPSA id 189-v6sm10582822wmd.17.2018.07.02.11.11.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jul 2018 11:11:53 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org Cc: Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v2 1/8] kernel/jump_label: abstract jump_entry member accessors Date: Mon, 2 Jul 2018 20:11:38 +0200 Message-Id: <20180702181145.4799-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180702181145.4799-1-ard.biesheuvel@linaro.org> References: <20180702181145.4799-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation of allowing architectures to use relative references in jump_label entries [which can dramatically reduce the memory footprint], introduce abstractions for references to the 'code', 'target' and 'key' members of struct jump_entry. Signed-off-by: Ard Biesheuvel --- include/linux/jump_label.h | 34 +++++++++++++++++ kernel/jump_label.c | 40 ++++++++------------ 2 files changed, 49 insertions(+), 25 deletions(-) -- 2.17.1 diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index b46b541c67c4..4603a1c88e48 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -119,6 +119,40 @@ struct static_key { #ifdef HAVE_JUMP_LABEL #include + +#ifndef __ASSEMBLY__ + +static inline unsigned long jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline unsigned long jump_entry_target(const struct jump_entry *entry) +{ + return entry->target; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#endif #endif #ifndef __ASSEMBLY__ diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 01ebdf1f9f40..8a3ac4f5f490 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -38,10 +38,10 @@ static int jump_label_cmp(const void *a, const void *b) const struct jump_entry *jea = a; const struct jump_entry *jeb = b; - if (jea->key < jeb->key) + if (jump_entry_key(jea) < jump_entry_key(jeb)) return -1; - if (jea->key > jeb->key) + if (jump_entry_key(jea) > jump_entry_key(jeb)) return 1; return 0; @@ -261,8 +261,8 @@ EXPORT_SYMBOL_GPL(jump_label_rate_limit); static int addr_conflict(struct jump_entry *entry, void *start, void *end) { - if (entry->code <= (unsigned long)end && - entry->code + JUMP_LABEL_NOP_SIZE > (unsigned long)start) + if (jump_entry_code(entry) <= (unsigned long)end && + jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE > (unsigned long)start) return 1; return 0; @@ -321,16 +321,6 @@ static inline void static_key_set_linked(struct static_key *key) key->type |= JUMP_TYPE_LINKED; } -static inline struct static_key *jump_entry_key(struct jump_entry *entry) -{ - return (struct static_key *)((unsigned long)entry->key & ~1UL); -} - -static bool jump_entry_branch(struct jump_entry *entry) -{ - return (unsigned long)entry->key & 1UL; -} - /*** * A 'struct static_key' uses a union such that it either points directly * to a table of 'struct jump_entry' or to a linked list of modules which in @@ -355,7 +345,7 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool enabled = static_key_enabled(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return enabled ^ branch; @@ -370,12 +360,12 @@ static void __jump_label_update(struct static_key *key, * An entry->code of 0 indicates an entry which has been * disabled because it was in an init text area. */ - if (entry->code) { - if (kernel_text_address(entry->code)) + if (!jump_entry_is_init(entry)) { + if (kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); else WARN_ONCE(1, "can't patch jump_label at %pS", - (void *)(unsigned long)entry->code); + (void *)jump_entry_code(entry)); } } } @@ -430,8 +420,8 @@ void __init jump_label_invalidate_initmem(void) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (init_section_contains((void *)(unsigned long)iter->code, 1)) - iter->code = 0; + if (init_section_contains((void *)jump_entry_code(iter), 1)) + jump_entry_set_init(iter); } } @@ -441,7 +431,7 @@ static enum jump_label_type jump_label_init_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool type = static_key_type(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return type ^ branch; @@ -565,7 +555,7 @@ static int jump_label_add_module(struct module *mod) continue; key = iterk; - if (within_module(iter->key, mod)) { + if (within_module((unsigned long)key, mod)) { static_key_set_entries(key, iter); continue; } @@ -615,7 +605,7 @@ static void jump_label_del_module(struct module *mod) key = jump_entry_key(iter); - if (within_module(iter->key, mod)) + if (within_module((unsigned long)key, mod)) continue; /* No memory during module load */ @@ -659,8 +649,8 @@ static void jump_label_invalidate_module_init(struct module *mod) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(iter->code, mod)) - iter->code = 0; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_init(iter); } } From patchwork Mon Jul 2 18:11:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140839 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp246954ljj; Mon, 2 Jul 2018 11:12:07 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfEs6XnALxqr1tWdYWZ5f7QaWsqiGLvP4DoO8jLo4+X1Kiz6mPwatLsHPlQfV6RSRdWaagQ X-Received: by 2002:a62:b29c:: with SMTP id z28-v6mr21753952pfl.8.1530555127366; Mon, 02 Jul 2018 11:12:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530555127; cv=none; d=google.com; s=arc-20160816; b=wnlt5v2s13mXAkR2u6uqFNZuRZvQ3fmJGDy22JGmURhG0dPwG4cZLuReObbDNs/Gqb DfB8H+xrNRdwP/ydlQyCE8UCGVpC0W7TdFWDPvNbh21v4Y4B5yFMQM8o6oHY52DBJayp K123QKpC4nIZBUeD1LyWPDQT30YqsM0KiFjUT0z203JH/fB8Wri/a7Pxl0qNWoBvpdAj vSmzxlu4H9/K8eSIKmHA4djbbREsSowZx0/NBP0zcwTBB5sSPH5jzo0qlsOF2jKAvbZe 5cpto767Bjyx1ZOQSSXmnyBn8hhm4Jp9GrHURyJw1FYXc8pNZBJGMJ0JLxhYlVss6Ubg uZVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=sN+HT57t+gSH0ShF8Th3JKe4zmSu8oRHrYwUlNvmdrc=; b=JouXMgpVKc7hntbzheCsZsEE/QB3z+v4ohQdygIzKCMaKXpTBYaBxrWNdkLNaqJDD2 5e8pogpVww+DsHlO0CPeMWDvQs195wd7d1SeK62vPioEETvOq2Po+XeVZEDQcmyzO17z HuT6YpBpSOLM6Sx/47SBJ6ba1mYH/Vd7YZKW7bEm1Rckcr7otUAc/IFx9IgrLDBxFjjR pp5oMvNTxS3c2123tiFA98h3CAsU/4uPWVbobaJyKo6E9CGnIftnFzMb2G4Z/z8jLd9a uZsNc4CvCJ4JMSPYeQ2fCi23cDnaTwx82ufFDjKDY2OHFHQuC26qFabUcN5CcnBfE8k4 B27w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=L12Ji9yF; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b18-v6si14440380pgs.417.2018.07.02.11.12.07; Mon, 02 Jul 2018 11:12:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=L12Ji9yF; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753383AbeGBSMC (ORCPT + 31 others); Mon, 2 Jul 2018 14:12:02 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:37072 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753348AbeGBSL5 (ORCPT ); Mon, 2 Jul 2018 14:11:57 -0400 Received: by mail-wr0-f193.google.com with SMTP id q10-v6so1707126wrd.4 for ; Mon, 02 Jul 2018 11:11:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sN+HT57t+gSH0ShF8Th3JKe4zmSu8oRHrYwUlNvmdrc=; b=L12Ji9yFkOczS8LxdqLIaOyOwP4xEPBvHP1UgNpl+z3lLalzVbfYOgTI0GKhzcnrls UZIePyE850IJ5e69rkXOVAhCKUpEBVKr0gkM64ZjPQ/ZdROpXtXWMMfaSm7qlG5/qQWo draOD5nv3m+K4RcQdV/waomYm1A6BDXnXjz2s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sN+HT57t+gSH0ShF8Th3JKe4zmSu8oRHrYwUlNvmdrc=; b=WbDaOSrD0F4PEPoi4BxQxX0zFVEfymZ95lJqVuBskyelnSeDDsip6ADDxu9u8wZMTA /HqV7+526jW+T8tCvHNCUr1eTE+qrFL3WhOExob7Uv8yokqmkNEjwvFjT9DWSi4sHa0O C9lYK62xEwGn2jfRWSetey2LHGdyT6BGTB4GhEoaqpFtN9QMYzSgDWzX+OOKc1YMNUBZ uoJSs0z59XK/HkL8kehWc2tEAD1KXiTS7WmWxzvnaWBItc0hFEQZZYBmZz6l3sMBurIK pA4twDVG1LQHtMI/NW6T5XmEyEt+4cgngeTg5g2V2kD1j6vZ9MpcoO27LWmxUznOzhky ZNDg== X-Gm-Message-State: APt69E3YdcBhETmui9dNqV4vPuTJJSTQM8YGVrpyTo1uYUK/ve7Mmbbr rRACgteOiglyOkY2xAqXs7oauw== X-Received: by 2002:adf:83a6:: with SMTP id 35-v6mr20405339wre.13.1530555116389; Mon, 02 Jul 2018 11:11:56 -0700 (PDT) Received: from localhost.localdomain (151.21.90.92.rev.sfr.net. [92.90.21.151]) by smtp.gmail.com with ESMTPSA id 189-v6sm10582822wmd.17.2018.07.02.11.11.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jul 2018 11:11:55 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org Cc: Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v2 2/8] kernel/jump_label: implement generic support for relative references Date: Mon, 2 Jul 2018 20:11:39 +0200 Message-Id: <20180702181145.4799-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180702181145.4799-1-ard.biesheuvel@linaro.org> References: <20180702181145.4799-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org To reduce the size taken up by absolute references in jump label entries themselves and the associated relocation records in the .init segment, add support for emitting them as relative references instead. Note that this requires some extra care in the sorting routine, given that the offsets change when entries are moved around in the jump_entry table. Signed-off-by: Ard Biesheuvel --- arch/Kconfig | 3 +++ include/linux/jump_label.h | 28 ++++++++++++++++++++ kernel/jump_label.c | 22 ++++++++++++++- 3 files changed, 52 insertions(+), 1 deletion(-) -- 2.17.1 diff --git a/arch/Kconfig b/arch/Kconfig index 2b8b70820002..22fa3792626e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -348,6 +348,9 @@ config HAVE_PERF_USER_STACK_DUMP config HAVE_ARCH_JUMP_LABEL bool +config HAVE_ARCH_JUMP_LABEL_RELATIVE + bool + config HAVE_RCU_TABLE_FREE bool diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 4603a1c88e48..871826fd0c3b 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -121,6 +121,32 @@ struct static_key { #include #ifndef __ASSEMBLY__ +#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE + +struct jump_entry { + s32 code; + s32 target; + long key; // key may be far away from the core kernel under KASLR +}; + +static inline unsigned long jump_entry_code(const struct jump_entry *entry) +{ + return (unsigned long)&entry->code + entry->code; +} + +static inline unsigned long jump_entry_target(const struct jump_entry *entry) +{ + return (unsigned long)&entry->target + entry->target; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + long offset = entry->key & ~1L; + + return (struct static_key *)((unsigned long)&entry->key + offset); +} + +#else static inline unsigned long jump_entry_code(const struct jump_entry *entry) { @@ -137,6 +163,8 @@ static inline struct static_key *jump_entry_key(const struct jump_entry *entry) return (struct static_key *)((unsigned long)entry->key & ~1UL); } +#endif + static inline bool jump_entry_is_branch(const struct jump_entry *entry) { return (unsigned long)entry->key & 1UL; diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 8a3ac4f5f490..d424e1d22d63 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -47,14 +47,34 @@ static int jump_label_cmp(const void *a, const void *b) return 0; } +static void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + static void jump_label_sort_entries(struct jump_entry *start, struct jump_entry *stop) { unsigned long size; + void *swapfn = NULL; + + if (IS_ENABLED(CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE)) + swapfn = jump_label_swap; size = (((unsigned long)stop - (unsigned long)start) / sizeof(struct jump_entry)); - sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL); + sort(start, size, sizeof(struct jump_entry), jump_label_cmp, swapfn); } static void jump_label_update(struct static_key *key); From patchwork Mon Jul 2 18:11:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140849 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp249301ljj; Mon, 2 Jul 2018 11:14:35 -0700 (PDT) X-Google-Smtp-Source: ADUXVKITJDL3zK06UYhqEU2RUZnp9fbe973oqrXgEPvfnbE6w8OEilleThLb5ckVz1GpT69QG7ld X-Received: by 2002:a17:902:6acc:: with SMTP id i12-v6mr26861240plt.278.1530555275299; Mon, 02 Jul 2018 11:14:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530555275; cv=none; d=google.com; s=arc-20160816; b=vKgX62a7AKa0OneAB8fzvR2BC//1Ic/rhhkv7J/eY3kZPrxCbsyDF3klY0nkxKEKo2 eP2M0egb8sNxQbDcRoSEnxLj3CHczPtNMtzbe5itFjQKDif41rMQ/AMXywsPVRaJ5Svu Wi20r+uX6R+p1eRi5fjbEnqsNYqFtpUDrjqYXkducrpzE+6R8yIZVmNR1nj0uu7lOYan Uy8Tb8CsUaZaUCzyucrUwTLDSO5JsyZNM9LveT0rlHG/fTWdq9GYCp2uKZBK77IONS5n 62r6WTHIGVFQneRlSd8L3kt1sXK6xCLtPHhpH8cDU0yZCQ6MnhdcBFgSjYSbmkgXKPdH JcEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=o8m2ubatinriCgE+fyFRDo1C7fsKjnxpNpjlgZpUC+Y=; b=ROsuiRKXiZrNPEsPlvyDfXmwW92tBmPnn4qhQDCg21oW70YNRBlaQAnKTMVIwpXkQl dTz4YJp1+tK6vUfXBhLrxrR6JWTLsL6o5M/29N6bTTFEMMN7t1xiGtOS+CFnUWNN1CHL ezHibVEv4uaz0kzDICFTRsoMqh9DNmfPQxbft4zOEtO3Vo2aUgzRyordV03Mai7V+XYV WHHo3xzblxyENabJ8Jf3HozVDtwcdpYrS5QSuBdvK0JSbA/klG4dJ98eeEVUyy56mfAQ bw6Z8Xaj2j4kJ0mr5DQFjJJEPUjiEFajvM1B6lBxDvcGhi8XEqF92QeESfH156NMQKxY JyXg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OAn2Wztv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 6-v6si16787191plb.409.2018.07.02.11.14.35; Mon, 02 Jul 2018 11:14:35 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OAn2Wztv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932180AbeGBSOe (ORCPT + 31 others); Mon, 2 Jul 2018 14:14:34 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:39012 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753323AbeGBSMA (ORCPT ); Mon, 2 Jul 2018 14:12:00 -0400 Received: by mail-wr0-f193.google.com with SMTP id b8-v6so16422671wro.6 for ; Mon, 02 Jul 2018 11:11:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=o8m2ubatinriCgE+fyFRDo1C7fsKjnxpNpjlgZpUC+Y=; b=OAn2Wztv72vqFNhayw1xHb7skMG8qhK6hKdeatCA5L6fGMcqW864ln6UrWAfAFCxKA j0Jte8u75R9lvgw8tbhVaSEiZ7nP3WDPw83Fl3bQzEhb96VlBDxh/8q2r24x4ItyMlyy v7kiVMHld00DBkkIwClWjSvXm3DyM2TrLJc9E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=o8m2ubatinriCgE+fyFRDo1C7fsKjnxpNpjlgZpUC+Y=; b=h6fnFWQCq7xUAvhaNKTe/yK0FSHxYwU9m2E64vnJ+B0NmPmYpdntmjlnhr8BtG+CrN vgIVmL9jrc7ie+4gi8JiJlh2PD/5b5kmVV3MGqVyKx2GC9r7eW/LDvKbo+9d08rNyJSe x57coe/U69qquFPF3iw2Dp4YF9V5nyxKiXefEwlq3X6PWEst5RsLUGUp0VLCU6V8OQlT rWmU4FCjJYGOrBD2AVBAbHWIe3tFIjUQ5/VYxTm4EJI852URTci9vZwPfrdc9cmmnD3U Mwdj5hZmS04M5gTXbHFlcew0pd+DvFuuIGp7lMeVoUU3ZrNijBDsJe79ZTxupqdPN1Wb 8APQ== X-Gm-Message-State: APt69E0lPJVB7Vh/+8KfBl0umVNzZGQ2RyKSz8djcGyzCEdW3IbdhKxF Pg+Rs9n2j1CK+qn7YkG14WLybQ== X-Received: by 2002:adf:d181:: with SMTP id h1-v6mr20077520wri.198.1530555118810; Mon, 02 Jul 2018 11:11:58 -0700 (PDT) Received: from localhost.localdomain (151.21.90.92.rev.sfr.net. [92.90.21.151]) by smtp.gmail.com with ESMTPSA id 189-v6sm10582822wmd.17.2018.07.02.11.11.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jul 2018 11:11:58 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org Cc: Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v2 3/8] arm64/kernel: jump_label: switch to relative references Date: Mon, 2 Jul 2018 20:11:40 +0200 Message-Id: <20180702181145.4799-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180702181145.4799-1-ard.biesheuvel@linaro.org> References: <20180702181145.4799-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On a randomly chosen distro kernel build for arm64, vmlinux.o shows the following sections, containing jump label entries, and the associated RELA relocation records, respectively: ... [38088] __jump_table PROGBITS 0000000000000000 00e19f30 000000000002ea10 0000000000000000 WA 0 0 8 [38089] .rela__jump_table RELA 0000000000000000 01fd8bb0 000000000008be30 0000000000000018 I 38178 38088 8 ... In other words, we have 190 KB worth of 'struct jump_entry' instances, and 573 KB worth of RELA entries to relocate each entry's code, target and key members. This means the RELA section occupies 10% of the .init segment, and the two sections combined represent 5% of vmlinux's entire memory footprint. So let's switch from 64-bit absolute references to 32-bit relative references for the code and target field, and a 64-bit relative reference for the 'key' field (which may reside in another module or the core kernel, which may be more than 4 GB way on arm64 when running with KASLR enable): this reduces the size of the __jump_table by 33%, and gets rid of the RELA section entirely. Acked-by: Will Deacon Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/jump_label.h | 36 +++++++++----------- arch/arm64/kernel/jump_label.c | 6 ++-- 3 files changed, 20 insertions(+), 23 deletions(-) -- 2.17.1 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1940c6405d04..d17aa9614e69 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -91,6 +91,7 @@ config ARM64 select HAVE_ARCH_BITREVERSE select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 1b5e0e843c3a..bb6d15b34668 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -26,13 +26,15 @@ #define JUMP_LABEL_NOP_SIZE AARCH64_INSN_SIZE -static __always_inline bool arch_static_branch(struct static_key *key, bool branch) +static __always_inline bool arch_static_branch(struct static_key *key, + bool branch) { - asm goto("1: nop\n\t" - ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" - ".popsection\n\t" + asm goto("1: nop \n\t" + " .pushsection __jump_table, \"aw\" \n\t" + " .align 3 \n\t" + " .long 1b - ., %l[l_yes] - . \n\t" + " .quad %c0 - . \n\t" + " .popsection \n\t" : : "i"(&((char *)key)[branch]) : : l_yes); return false; @@ -40,13 +42,15 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran return true; } -static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch) +static __always_inline bool arch_static_branch_jump(struct static_key *key, + bool branch) { - asm goto("1: b %l[l_yes]\n\t" - ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" - ".popsection\n\t" + asm goto("1: b %l[l_yes] \n\t" + " .pushsection __jump_table, \"aw\" \n\t" + " .align 3 \n\t" + " .long 1b - ., %l[l_yes] - . \n\t" + " .quad %c0 - . \n\t" + " .popsection \n\t" : : "i"(&((char *)key)[branch]) : : l_yes); return false; @@ -54,13 +58,5 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool return true; } -typedef u64 jump_label_t; - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index c2dd1ad3e648..903d17023d77 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -25,12 +25,12 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - void *addr = (void *)entry->code; + void *addr = (void *)jump_entry_code(entry); u32 insn; if (type == JUMP_LABEL_JMP) { - insn = aarch64_insn_gen_branch_imm(entry->code, - entry->target, + insn = aarch64_insn_gen_branch_imm(jump_entry_code(entry), + jump_entry_target(entry), AARCH64_INSN_BRANCH_NOLINK); } else { insn = aarch64_insn_gen_nop(); From patchwork Mon Jul 2 18:11:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140840 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp247001ljj; Mon, 2 Jul 2018 11:12:10 -0700 (PDT) X-Google-Smtp-Source: AAOMgpd2sGLUE8FWD/p7ZEEQzrut86Ojgf68bWys2qQKz6QQ4xPClrrVrvocuUSQTefG0HvxEMAh X-Received: by 2002:aa7:808f:: with SMTP id v15-v6mr9276553pff.38.1530555130415; Mon, 02 Jul 2018 11:12:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530555130; cv=none; d=google.com; s=arc-20160816; b=tP72GJxKirpsk2wpuOrK25tM5IByn+7JfZPp5e4DqDo0AsXztdG87IZqdEnesnMunk cL7nxsxuZp/zGcYGwaEe59q2WmPWkXunDC5mC86wOy8Q++39hjG/v6pUaFwB1pfj5V+U 3OFAzS6NrNvtTD6o7F/IejA+o1/mtjQTNBrmK55aBbg4f3yYo2cBX07xLah9oN4b0BbL 5TooSif/OZPqQX7kAaCM527hExZIJ1qWW7tsdj7eLHCB+/Os2V1WGNJ8oMq1ym5UB4vW QmsuJLotpLos615ETYVMIRME1ipMpvYRSMHDj3/RA0DZGWPed+92IMO94CVou4Arm0G6 Acuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=6nUyftJQHbn4ADCDRiHDYhO9k2WDA48zRvsOuVLWFCc=; b=LiIPbqbDW5n+7PYPzUQDZBkom+YCHE9Hi4QhxahRPoEheANcMuhkM8Knw/M3tZ/wKu XSGK0r6cV49A59X5ZREPCJS3mkMBfk9BR/RcLADeUmIsokC5V5Ck+7f1F+G3fV28CytX nIGiUKiHS5jzCPTAJa1Ca9OC0MCo/a7xbxy3ta87woDV3v+enMV4hPYzvOQD/QLEAMzJ h9Ys1/uhJl2rdm3Id/Wi8omPz07K/Ckgk4IRpWQF2oIZBsoRyUrk+zsbliB5f8krdqoL 3VuZ+l14n9wiT6dFZdRcYvTAyjl5hQt8gJaV10JjWkujTWCrDgFNdoCkMC7muL5hSPyK x8kw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MH9B829V; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q65-v6si11104938pga.283.2018.07.02.11.12.10; Mon, 02 Jul 2018 11:12:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MH9B829V; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753408AbeGBSMI (ORCPT + 31 others); Mon, 2 Jul 2018 14:12:08 -0400 Received: from mail-wr0-f196.google.com ([209.85.128.196]:32779 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753348AbeGBSMD (ORCPT ); Mon, 2 Jul 2018 14:12:03 -0400 Received: by mail-wr0-f196.google.com with SMTP id k7-v6so12933931wrq.0 for ; Mon, 02 Jul 2018 11:12:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6nUyftJQHbn4ADCDRiHDYhO9k2WDA48zRvsOuVLWFCc=; b=MH9B829VsTLGEaymnuo3lvDs2YlPFP+tCJ4WAiPdSGavKSbuibs/57DY2V9s64nv+N Wu5/hzTyQX4OgnsjUsQWylcvIfyPpRPM9ImXRgbF/Vb7Wo4c3RDzobRfDcLQNlFTvAD/ CtHAKlxbhonAXmdbn45KBkVYQlAigW8Nd+8/0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6nUyftJQHbn4ADCDRiHDYhO9k2WDA48zRvsOuVLWFCc=; b=lhPa5z6jjIW2ecI0Fd/lKeCuqszsob0JyGsjzMzWuSNSy5oAe5ws3HVQDAeM/ZYA9i osF8wQshF4FnOrFtzDVhwm6Z8WX4bNEWRRlOMD/7bsk6zAuUeTWAsQYjH3EZxWGMvKYT noh3w2tLf9lTQ//s9Ec4UEcPqQ0mKqA2hvZNXPaVgW3DPkpP6Wycb+dSvr12Lx6XkExT pvDZhccisdUaSVOMhvb0qsTdlUisjBWxBI9FWZ6wCeU2jqvvwPW5Jo8PetbjQ3MqUw7a IopNMmvg6BclOt9cppJzunsA8uRlJTUGgqIkp/ASTPo4+UoQmU2hjLv6uKN38pQz4XYD 5T8g== X-Gm-Message-State: APt69E0mohtSR+B41t5k93SxrL+WfnzSKq0WRicnM7McMNeWHEvp0/f6 N+GjXyk8W4P3wRL8FiRlVhVuSQ== X-Received: by 2002:adf:f50e:: with SMTP id q14-v6mr76852wro.241.1530555121138; Mon, 02 Jul 2018 11:12:01 -0700 (PDT) Received: from localhost.localdomain (151.21.90.92.rev.sfr.net. [92.90.21.151]) by smtp.gmail.com with ESMTPSA id 189-v6sm10582822wmd.17.2018.07.02.11.11.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jul 2018 11:12:00 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org Cc: Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v2 4/8] x86: add support for 64-bit place relative relocations Date: Mon, 2 Jul 2018 20:11:41 +0200 Message-Id: <20180702181145.4799-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180702181145.4799-1-ard.biesheuvel@linaro.org> References: <20180702181145.4799-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add support for R_X86_64_PC64 relocations, which operate on 64-bit quantities holding a relative symbol reference. This allows jump table entries to be emitted in a way that makes them invariant under runtime relocation, which means that no metadata needs to be emitted into the kernel image to describe such data structures, resulting in a size reduction. Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/elf.h | 1 + arch/x86/kernel/machine_kexec_64.c | 4 ++++ arch/x86/kernel/module.c | 6 ++++++ arch/x86/tools/relocs.c | 10 ++++++++++ 4 files changed, 21 insertions(+) -- 2.17.1 diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h index 0d157d2a1e2a..d3925d684296 100644 --- a/arch/x86/include/asm/elf.h +++ b/arch/x86/include/asm/elf.h @@ -62,6 +62,7 @@ typedef struct user_fxsr_struct elf_fpxregset_t; #define R_X86_64_PC16 13 /* 16 bit sign extended pc relative */ #define R_X86_64_8 14 /* Direct 8 bit sign extended */ #define R_X86_64_PC8 15 /* 8 bit sign extended pc relative */ +#define R_X86_64_PC64 24 /* Place relative 64-bit signed */ #define R_X86_64_NUM 16 diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c index 4c8acdfdc5a7..6638d1edb2be 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -496,6 +496,10 @@ int arch_kexec_apply_relocations_add(struct purgatory_info *pi, value -= (u64)address; *(u32 *)location = value; break; + case R_X86_64_PC64: + value -= (u64)address; + *(u64 *)location = value; + break; default: pr_err("Unknown rela relocation: %llu\n", ELF64_R_TYPE(rel[i].r_info)); diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index f58336af095c..b052e883dd8c 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -201,6 +201,12 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, goto overflow; #endif break; + case R_X86_64_PC64: + if (*(u64 *)loc != 0) + goto invalid_relocation; + val -= (u64)loc; + *(u64 *)loc = val; + break; default: pr_err("%s: Unknown rela relocation: %llu\n", me->name, ELF64_R_TYPE(rel[i].r_info)); diff --git a/arch/x86/tools/relocs.c b/arch/x86/tools/relocs.c index 220e97841e49..a4075bc37e8f 100644 --- a/arch/x86/tools/relocs.c +++ b/arch/x86/tools/relocs.c @@ -195,6 +195,7 @@ static const char *rel_type(unsigned type) #if ELF_BITS == 64 REL_TYPE(R_X86_64_NONE), REL_TYPE(R_X86_64_64), + REL_TYPE(R_X86_64_PC64), REL_TYPE(R_X86_64_PC32), REL_TYPE(R_X86_64_GOT32), REL_TYPE(R_X86_64_PLT32), @@ -781,6 +782,15 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym, add_reloc(&relocs32neg, offset); break; + case R_X86_64_PC64: + /* + * Only used by jump labels + */ + if (is_percpu_sym(sym, symname)) + die("Invalid R_X86_64_PC64 relocation against per-CPU symbol %s\n", + symname); + break; + case R_X86_64_32: case R_X86_64_32S: case R_X86_64_64: From patchwork Mon Jul 2 18:11:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140848 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp248954ljj; Mon, 2 Jul 2018 11:14:13 -0700 (PDT) X-Google-Smtp-Source: AAOMgpcz/TVoZ6RwrZN+tKHuT8TrFBHhu4KmMtb+U8mjU3pkS9kgyLC6IpcDUEN+ZSzDJWI9N5I1 X-Received: by 2002:a62:98d6:: with SMTP id d83-v6mr17842733pfk.186.1530555253005; Mon, 02 Jul 2018 11:14:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530555252; cv=none; d=google.com; s=arc-20160816; b=jH92wHNzIiqDxT8HaqvDAE9EcnDe7ktzm0byJCW/x6NNEMDrLFU697MQuTYzM5biiK ZBWAE2yWuE4cAEwv1fQrd2te7kKUPZb4UxnbH2qAXIyuhQMsEcW8FzF1OTr+ygu1URrP W0JfjR7b6Ie4xew0u26qCC7RfBK8xRfV4AMStGxz+BxMtdS4f/L2cDbmqAg1zz3ce20K wBCDnToZt540zuVuYwb3aAEb/A7Wgqn29ZXM8Bnj++QS5NI4B8+4qIWt2rvn8EoeFC57 yuZCvkB2Skbet9aM2e4vcLHPQbcxPYw17ng1ZXeRpHGxA3gLwqWO86RLVWp1s1UHcjAL dwAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=w4lxjj61CeIJtbmnFVjG2GWM9frrLYdO8XaMvDkK3nI=; b=cP6gwpWx1WHmhQgDil96AbuzZh0agWFs3qZ7vKw09mNxYB/jScVhI35oTawHInzgRd VkajA1BgpbkdlPn9mBhyDM4zs1oxr9GxYyHhnWdqciuNu3galMo5/61EqCqx9ADRYvqA Y6fy/7BJKPI+C5GJn4BSrP++Zyw1H8mw8qe6JskCoFI5y8etdZzxHfgsNxZEv1rUf6ls ROqLheG2L5W0e45YTRyYumyriQwbnWJdBx6O5vtJDYaIZM0fEep8ScAv9kZvECPre60N X5nSrZspFymCwihLbUV0FGlW44/D0tLQtT6kV8iCO8idOSAFFt9hAzLdYGsBq8sIDQSG yPxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GymrCStW; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 6-v6si16787191plb.409.2018.07.02.11.14.12; Mon, 02 Jul 2018 11:14:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GymrCStW; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932148AbeGBSOK (ORCPT + 31 others); Mon, 2 Jul 2018 14:14:10 -0400 Received: from mail-wr0-f194.google.com ([209.85.128.194]:44074 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753384AbeGBSME (ORCPT ); Mon, 2 Jul 2018 14:12:04 -0400 Received: by mail-wr0-f194.google.com with SMTP id p12-v6so16417898wrn.11 for ; Mon, 02 Jul 2018 11:12:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=w4lxjj61CeIJtbmnFVjG2GWM9frrLYdO8XaMvDkK3nI=; b=GymrCStWjyo2SvoZDUrwyeQXJkBuT4se9Fg+tlEcaSo8Aym536nvFRSHOd7XtAmMya 1cL8Wd/1/asNFKwEGUHVvxVPJM+2IlGnMiza3uAWHP5OVn+C/2H3QjewoJJxBO7xHlhd OpHAV/O6SWI/9zM3HEWvMLl+wwzB48KZ/wPpM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=w4lxjj61CeIJtbmnFVjG2GWM9frrLYdO8XaMvDkK3nI=; b=VJwSMMdzxJnEMMEYtkTPBIvXFqIgefpV3UKSGs2UPziUPE0l38sw+ynk4shM8bnmP6 XppepRi7mocWGhN3BD+7HsOD0oOl8csYRwpYvcksdXjcT9WVwg9O8e//MeagY3C2LW6g ClAM0oORmUx0pLV7ig+sOlfoxXM3uLb4W6CY60RhvFnX1koff4uhmBdDGxPyYSfV57Gm OP1gWv4XdgGqeHb80nx3htLmUaRGN63HzoYG7T3n5aDBLLto0oZo77Qr8kjRrCAs3eve zlEUb7l+od0+799vBVslKNzNxQhhJ6la5/A7lMGzo+2wlnisoD3I0ODGHx8JYDdrajcx HExQ== X-Gm-Message-State: APt69E0hCe+Uy6dHdJcNQg+I/WIL3aPc08lhJYrBwU3nqRCC/eWOTkQe zv7PNhbHEw7yPbnhVudwyOqVHQ== X-Received: by 2002:adf:8e49:: with SMTP id n67-v6mr19861946wrb.131.1530555123423; Mon, 02 Jul 2018 11:12:03 -0700 (PDT) Received: from localhost.localdomain (151.21.90.92.rev.sfr.net. [92.90.21.151]) by smtp.gmail.com with ESMTPSA id 189-v6sm10582822wmd.17.2018.07.02.11.12.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jul 2018 11:12:02 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org Cc: Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v2 5/8] x86: jump_label: switch to jump_entry accessors Date: Mon, 2 Jul 2018 20:11:42 +0200 Message-Id: <20180702181145.4799-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180702181145.4799-1-ard.biesheuvel@linaro.org> References: <20180702181145.4799-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation of switching x86 to use place-relative references for the code, target and key members of struct jump_entry, replace direct references to the struct members with invocations of the new accessors. This will allow us to make the switch by modifying the accessors only. This incorporates a cleanup of __jump_label_transform() proposed by Peter. Cc: Peter Zijlstra Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/jump_label.c | 62 ++++++++------------ 1 file changed, 25 insertions(+), 37 deletions(-) -- 2.17.1 diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e56c95be2808..023768222a94 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -42,52 +42,37 @@ static void __jump_label_transform(struct jump_entry *entry, void *(*poker)(void *, const void *, size_t), int init) { - union jump_code_union code; + union jump_code_union jmp; const unsigned char default_nop[] = { STATIC_KEY_INIT_NOP }; const unsigned char *ideal_nop = ideal_nops[NOP_ATOMIC5]; + const void *expect, *code; + int line; + + jmp.jump = 0xe9; + jmp.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); if (type == JUMP_LABEL_JMP) { if (init) { - /* - * Jump label is enabled for the first time. - * So we expect a default_nop... - */ - if (unlikely(memcmp((void *)entry->code, default_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + expect = default_nop; line = __LINE__; } else { - /* - * ...otherwise expect an ideal_nop. Otherwise - * something went horribly wrong. - */ - if (unlikely(memcmp((void *)entry->code, ideal_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + expect = ideal_nop; line = __LINE__; } - code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); + code = &jmp.code; } else { - /* - * We are disabling this jump label. If it is not what - * we think it is, then something must have gone wrong. - * If this is the first initialization call, then we - * are converting the default nop to the ideal nop. - */ if (init) { - if (unlikely(memcmp((void *)entry->code, default_nop, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + expect = default_nop; line = __LINE__; } else { - code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); - if (unlikely(memcmp((void *)entry->code, &code, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + expect = &jmp.code; line = __LINE__; } - memcpy(&code, ideal_nops[NOP_ATOMIC5], JUMP_LABEL_NOP_SIZE); + + code = ideal_nop; } + if (memcmp((void *)jump_entry_code(entry), expect, JUMP_LABEL_NOP_SIZE)) + bug_at((void *)jump_entry_code(entry), line); + /* * Make text_poke_bp() a default fallback poker. * @@ -96,11 +81,14 @@ static void __jump_label_transform(struct jump_entry *entry, * always nop being the 'currently valid' instruction * */ - if (poker) - (*poker)((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE); - else - text_poke_bp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE, - (void *)entry->code + JUMP_LABEL_NOP_SIZE); + if (poker) { + (*poker)((void *)jump_entry_code(entry), code, + JUMP_LABEL_NOP_SIZE); + return; + } + + text_poke_bp((void *)jump_entry_code(entry), code, JUMP_LABEL_NOP_SIZE, + (void *)jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); } void arch_jump_label_transform(struct jump_entry *entry, From patchwork Mon Jul 2 18:11:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140847 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp248356ljj; Mon, 2 Jul 2018 11:13:39 -0700 (PDT) X-Google-Smtp-Source: ADUXVKLKqMKk2cSH5UthesxxmuwRyPt7uO8dAyZe2nmQ8D5itoYo7EQQA5wzkpSTN5HhXs4XyCHI X-Received: by 2002:a63:2d45:: with SMTP id t66-v6mr22288661pgt.381.1530555219630; Mon, 02 Jul 2018 11:13:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530555219; cv=none; d=google.com; s=arc-20160816; b=PBnysxXuT4c7nup2fCph5HVjfTPSgmFRUW/UVp8027Q2/jus9sOkoCYctp0whs/Uy3 izAtrpjhmOp3VqMhQsDxnHaslAgDV3GXnb0JXfK4T4PvUl2rlQNHAOnHetlCM2/Pb5S7 lXQxJrH+OZSG7dmSQTjS62O2meJYU6wNp5POBNNAjgGjalhyuCua4AHUjRe9lMumMdV3 PPDnwG2X4qO1qHogc13ynA5OxzZa8VmDNmzdMlmU3zThTUg987sxyV5fuOMqPyoFRfoT oj6sOZH2HxHOsnLU1DJSdcD7qRvXpvGHnWC/jkx7748dOX9vhXMQy59XQVfEjQAFS6lH PtBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=L7ZgdBdHSh65TnBtMV8VfmyEs0kgJc5Pfx1hmdMEX3g=; b=G4BWSHTZ35QKWpUyx8JuHRt+XeBils7aICwvt1Dqx38i0bKUW1BIBj1s0ezRsLUtTb ZNNxrOoFPVXzXTvcugbCsmgDwLvUBnXjRCot9SsAyL+sK+s5eRXbkKR34QtM9BkikVlt BR9h0Eut8L2NluoQ0PcrPotluXRtFrIciNf43ysbotZpwL7Xr1mX0DGxmCcoXTO8HFSN yt6sSy6QLKIF6/msn9FcqN4EdkO5jAr2ERRu2fLNIPOFGQNwMHNnIHBvE/3vk9I3X+w7 lO/2zr/yzjZhPYTaE5+S47t6t85mH9dtmHr8dKzrY50oAYA26q0q6XUPrAaJKpjggeOr jigg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FLdBsiKw; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b59-v6si16421362plb.107.2018.07.02.11.13.39; Mon, 02 Jul 2018 11:13:39 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FLdBsiKw; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753346AbeGBSNi (ORCPT + 31 others); Mon, 2 Jul 2018 14:13:38 -0400 Received: from mail-wr0-f195.google.com ([209.85.128.195]:45533 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753392AbeGBSMH (ORCPT ); Mon, 2 Jul 2018 14:12:07 -0400 Received: by mail-wr0-f195.google.com with SMTP id u7-v6so16423599wrn.12 for ; Mon, 02 Jul 2018 11:12:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=L7ZgdBdHSh65TnBtMV8VfmyEs0kgJc5Pfx1hmdMEX3g=; b=FLdBsiKwQ9szv78kIWndu3Hz8IIvvO7FXMKscslgC2tgl9L9/n9Hb1iUaVlXKY02uI 4x8+w+oT5OV/+Q6RthuEO13DIgB+cWxmieYol+xyO0Rt4R6vyLUuhD/MvjMMmDSnYrpy NPI5FJkO5YJq4yOsj0IscLAF76B10J09kuSgM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=L7ZgdBdHSh65TnBtMV8VfmyEs0kgJc5Pfx1hmdMEX3g=; b=Z51n03UnxmRfqizmSSg5+6+uL5UrCzW+y2g2aD/1RgnWHdTMS4ZipMhO5Azm1G5BiO Lyh5oiymqRkpozFmd6XUzK8OzHp/MbRUcAGu/TpMyKylMJrbsbaJnzdeyCxrMAgA2Dr3 y+BhO2dE6+9m/pHstPGTzwab9fRIaGqQY1OucBS7mFE3YfbeByyo9om9kpdGyfX65Y7Y Rq1wZGHbNg2xT+SQM/1k7Wds4fXX5LfV95vpYe6fmOGJZZJ09d+lEI6fh4mcczb0/zDE lB/Qo8YIRuESzTE1a9yjul3a8THTlm9bLxKsEo6yjQO2JTE1E8XF2ZTWtPysgp2Lqmpm M/sQ== X-Gm-Message-State: APt69E36rSUHmjW+OfVl+7j8eLJ38kUgyWi/Q0CxjYaQjZw8BsXVuG+O 2zF5heooUwInT0cQA3tY+VX3EA== X-Received: by 2002:adf:b90a:: with SMTP id k10-v6mr3724078wrf.169.1530555125951; Mon, 02 Jul 2018 11:12:05 -0700 (PDT) Received: from localhost.localdomain (151.21.90.92.rev.sfr.net. [92.90.21.151]) by smtp.gmail.com with ESMTPSA id 189-v6sm10582822wmd.17.2018.07.02.11.12.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jul 2018 11:12:05 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org Cc: Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v2 6/8] x86/kernel: jump_table: use relative references Date: Mon, 2 Jul 2018 20:11:43 +0200 Message-Id: <20180702181145.4799-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180702181145.4799-1-ard.biesheuvel@linaro.org> References: <20180702181145.4799-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Similar to the arm64 case, 64-bit x86 can benefit from using relative references rather than absolute ones when emitting struct jump_entry instances. Not only does this reduce the memory footprint of the entries themselves by 33%, it also removes the need for carrying relocation metadata on relocatable builds (i.e., for KASLR) which saves a fair chunk of .init space as well (although the savings are not as dramatic as on arm64) Signed-off-by: Ard Biesheuvel --- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 24 +++++++------------- tools/objtool/special.c | 4 ++-- 3 files changed, 11 insertions(+), 18 deletions(-) -- 2.17.1 diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index e10a3542db7e..dd71258ec1cc 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -118,6 +118,7 @@ config X86 select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if X86_64 || X86_PAE select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 8c0de4282659..21efc9d07ed9 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -37,7 +37,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" ".pushsection __jump_table, \"aw\" \n\t" _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".long 1b - ., %l[l_yes] - . \n\t" + _ASM_PTR "%c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -53,7 +54,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool "2:\n\t" ".pushsection __jump_table, \"aw\" \n\t" _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".long 1b - ., %l[l_yes] - . \n\t" + _ASM_PTR "%c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -62,18 +64,6 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool return true; } -#ifdef CONFIG_X86_64 -typedef u64 jump_label_t; -#else -typedef u32 jump_label_t; -#endif - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #else /* __ASSEMBLY__ */ .macro STATIC_JUMP_IF_TRUE target, key, def @@ -88,7 +78,8 @@ struct jump_entry { .endif .pushsection __jump_table, "aw" _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + .long .Lstatic_jump_\@ - ., \target - . + _ASM_PTR \key - . .popsection .endm @@ -104,7 +95,8 @@ struct jump_entry { .endif .pushsection __jump_table, "aw" _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 + .long .Lstatic_jump_\@ - ., \target - . + _ASM_PTR \key + 1 - . .popsection .endm diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 84f001d52322..50af4e1274b3 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -30,9 +30,9 @@ #define EX_ORIG_OFFSET 0 #define EX_NEW_OFFSET 4 -#define JUMP_ENTRY_SIZE 24 +#define JUMP_ENTRY_SIZE 16 #define JUMP_ORIG_OFFSET 0 -#define JUMP_NEW_OFFSET 8 +#define JUMP_NEW_OFFSET 4 #define ALT_ENTRY_SIZE 13 #define ALT_ORIG_OFFSET 0 From patchwork Mon Jul 2 18:11:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140841 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp247140ljj; Mon, 2 Jul 2018 11:12:18 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJu0DGGEOmpmtQDsViC1CnHOPHKNGeV2GB72SsoPAwOQm+1eHEAx8hw7S6DlahNS4tWCv3I X-Received: by 2002:a17:902:b418:: with SMTP id x24-v6mr26837743plr.2.1530555138137; Mon, 02 Jul 2018 11:12:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530555138; cv=none; d=google.com; s=arc-20160816; b=Ttil/r2Cz8Ldb7+4X7MVQT+9iYIX+rLFI5TVEyF3iKYS7GmMBsG+Je6wauktyaUrQU 111ti/xB1dapRMNuOzC8QTNj9hO4w9OniO8OxPb7DWndV+97ah4JfGMhPaLg3hx1Kufu +GRTCJ9JSJI/4dR02sWOm8N7+0CFMyNRwbc5xXCtP5pQx78XX663X0cccG66K/fzfZ2o AYfCM67k818olP53LGuQpNlUqPlgvU2K4b0PqCegJOFs4KR1rRY5tZ/WfTrEPxsXKcDD DQ8q/L8YTgAhU+1vT5GJcDjDWe25cli7Plt6rMN1oYsq6/LGsUGI2ZvXQLLzW5e//VJO Da2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=cadlqJXdwe8CrO/JHhgh75R+ewEWBI40jWFpP2FbnG8=; b=G9u+7uNBibk77AzK8vMPuQ1BVvZ+MCXOQh9Xkd7JkMoukUvc0f7Ny6wi/XgmEfCR7f MFcKA5FsMOpB25Bt2pWg5uBPpZ2uHOMGk3XewTSbYvPIjzleLZjKemGzgv4/DEh+Umid YfrjvvqeEidtEZdwvUG0L881rQT1vKUSW5mAjCIrFp4KnHp++bgGEbXY4RjyIWsqrSdc 652hNUoronCnrBWFpKPMK2AnvaSrFShlAYRTfWqv0cK3CYrJvz19lKzOvv2ylwqhXs2x N/6tFklYq6GMYFEL3BlPM/nfQzwcsQD8sLW8KlYZPC/EZZPKEAOJ6+5yxNFxSm8BttLR pFHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=K+NImYqv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q65-v6si11104938pga.283.2018.07.02.11.12.17; Mon, 02 Jul 2018 11:12:18 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=K+NImYqv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753437AbeGBSMO (ORCPT + 31 others); Mon, 2 Jul 2018 14:12:14 -0400 Received: from mail-wr0-f196.google.com ([209.85.128.196]:35882 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753409AbeGBSMJ (ORCPT ); Mon, 2 Jul 2018 14:12:09 -0400 Received: by mail-wr0-f196.google.com with SMTP id f16-v6so16430178wrm.3 for ; Mon, 02 Jul 2018 11:12:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=cadlqJXdwe8CrO/JHhgh75R+ewEWBI40jWFpP2FbnG8=; b=K+NImYqveGE4IO8GbXZTULp648/OGzvx2TZX0mMwc+QsX0v0pZ6Xl02YW9+hOXl54c 43oAykSex1GmOB4cPy9RJWZKPu+eNEsUEk3EpbRlZiwGF6/vKaBTwWryWz7NWjLRiEcL 0K2tAYhRTfnZwqqVuDeVZG5bFsMxetHEPuBX4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=cadlqJXdwe8CrO/JHhgh75R+ewEWBI40jWFpP2FbnG8=; b=igt8mBLXg2KRcBez4ouJXlPTb3DacC+c7ihXzNC12DbBfggWlF95xCXsSrHX2B/s9j CSh/HJWzQz676yzVAnWpAHnVFRZjgHOY1jobo2yNHaXYX2lzw+PE9hL1oXiMYblnfYJ5 inlllIyOBt0wClBhyXeWJfoIfD2ytRTeX2P1UORn7jrPPnpReTQP7nc60mlLTu53A8N4 CL5JX7aSyBxdxGxxWuHvQf8iheKUcn8Sy0abN8UmwNfjOujijGkBxTu3H33IjB0Rz+35 3X9oP+M3Con7zHye7FUCpf4Qz+kPqpRQ/+vVsAqo50RppiQxiV5TjkKpCR4A0UHv6Nzt XkFw== X-Gm-Message-State: APt69E0QgHidrKZO3wTXJbLGaeMgaXoYb4T3M7zJJUYbuParX7+c/nJt N4PIlFFDwzJP1faLKdsEu4H/lg== X-Received: by 2002:adf:c70e:: with SMTP id k14-v6mr19184697wrg.16.1530555128491; Mon, 02 Jul 2018 11:12:08 -0700 (PDT) Received: from localhost.localdomain (151.21.90.92.rev.sfr.net. [92.90.21.151]) by smtp.gmail.com with ESMTPSA id 189-v6sm10582822wmd.17.2018.07.02.11.12.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jul 2018 11:12:07 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org Cc: Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v2 7/8] jump_label: annotate entries that operate on __init code earlier Date: Mon, 2 Jul 2018 20:11:44 +0200 Message-Id: <20180702181145.4799-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180702181145.4799-1-ard.biesheuvel@linaro.org> References: <20180702181145.4799-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Jump table entries are mostly read-only, with the exception of the init and module loader code that defuses entries that point into init code when the code being referred to is freed. For robustness, it would be better to move these entries into the ro_after_init section, but clearing the 'code' member of each jump table entry referring to init code at module load time races with the module_enable_ro() call that remaps the ro_after_init section read only, so we'd like to do it earlier. So given that whether such an entry refers to init code can be decided much earlier, we can pull this check forward. Since we may still need the code entry at this point, let's switch to setting a low bit in the 'key' member just like we do to annotate the default state of a jump table entry. Signed-off-by: Ard Biesheuvel --- include/linux/jump_label.h | 11 ++--- init/main.c | 1 - kernel/jump_label.c | 48 ++++++-------------- 3 files changed, 18 insertions(+), 42 deletions(-) -- 2.17.1 Reviewed-by: Kees Cook diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 871826fd0c3b..feee8abc96be 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -141,7 +141,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - long offset = entry->key & ~1L; + long offset = entry->key & ~3L; return (struct static_key *)((unsigned long)&entry->key + offset); } @@ -160,7 +160,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + return (struct static_key *)((unsigned long)entry->key & ~3UL); } #endif @@ -172,12 +172,12 @@ static inline bool jump_entry_is_branch(const struct jump_entry *entry) static inline bool jump_entry_is_init(const struct jump_entry *entry) { - return entry->code == 0; + return (unsigned long)entry->key & 2UL; } static inline void jump_entry_set_init(struct jump_entry *entry) { - entry->code = 0; + entry->key |= 2; } #endif @@ -213,7 +213,6 @@ extern struct jump_entry __start___jump_table[]; extern struct jump_entry __stop___jump_table[]; extern void jump_label_init(void); -extern void jump_label_invalidate_initmem(void); extern void jump_label_lock(void); extern void jump_label_unlock(void); extern void arch_jump_label_transform(struct jump_entry *entry, @@ -261,8 +260,6 @@ static __always_inline void jump_label_init(void) static_key_initialized = true; } -static inline void jump_label_invalidate_initmem(void) {} - static __always_inline bool static_key_false(struct static_key *key) { if (unlikely(static_key_count(key) > 0)) diff --git a/init/main.c b/init/main.c index e59a01f163d6..d1a6b8a896e5 100644 --- a/init/main.c +++ b/init/main.c @@ -1062,7 +1062,6 @@ static int __ref kernel_init(void *unused) /* need to finish all async __init code before freeing the memory */ async_synchronize_full(); ftrace_free_init_mem(); - jump_label_invalidate_initmem(); free_initmem(); mark_readonly(); system_state = SYSTEM_RUNNING; diff --git a/kernel/jump_label.c b/kernel/jump_label.c index d424e1d22d63..7cdd49aeaf6a 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -373,14 +373,15 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) static void __jump_label_update(struct static_key *key, struct jump_entry *entry, - struct jump_entry *stop) + struct jump_entry *stop, + bool init) { for (; (entry < stop) && (jump_entry_key(entry) == key); entry++) { /* * An entry->code of 0 indicates an entry which has been * disabled because it was in an init text area. */ - if (!jump_entry_is_init(entry)) { + if (init || !jump_entry_is_init(entry)) { if (kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); else @@ -420,6 +421,9 @@ void __init jump_label_init(void) if (jump_label_type(iter) == JUMP_LABEL_NOP) arch_jump_label_transform_static(iter, JUMP_LABEL_NOP); + if (init_section_contains((void *)jump_entry_code(iter), 1)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -432,19 +436,6 @@ void __init jump_label_init(void) cpus_read_unlock(); } -/* Disable any jump label entries in __init/__exit code */ -void __init jump_label_invalidate_initmem(void) -{ - struct jump_entry *iter_start = __start___jump_table; - struct jump_entry *iter_stop = __stop___jump_table; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (init_section_contains((void *)jump_entry_code(iter), 1)) - jump_entry_set_init(iter); - } -} - #ifdef CONFIG_MODULES static enum jump_label_type jump_label_init_type(struct jump_entry *entry) @@ -524,7 +515,8 @@ static void __jump_label_mod_update(struct static_key *key) stop = __stop___jump_table; else stop = m->jump_entries + m->num_jump_entries; - __jump_label_update(key, mod->entries, stop); + __jump_label_update(key, mod->entries, stop, + m->state == MODULE_STATE_COMING); } } @@ -570,6 +562,9 @@ static int jump_label_add_module(struct module *mod) for (iter = iter_start; iter < iter_stop; iter++) { struct static_key *iterk; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -605,7 +600,7 @@ static int jump_label_add_module(struct module *mod) /* Only update if we've changed from our initial state */ if (jump_label_type(iter) != jump_label_init_type(iter)) - __jump_label_update(key, iter, iter_stop); + __jump_label_update(key, iter, iter_stop, true); } return 0; @@ -661,19 +656,6 @@ static void jump_label_del_module(struct module *mod) } } -/* Disable any jump label entries in module init code */ -static void jump_label_invalidate_module_init(struct module *mod) -{ - struct jump_entry *iter_start = mod->jump_entries; - struct jump_entry *iter_stop = iter_start + mod->num_jump_entries; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(jump_entry_code(iter), mod)) - jump_entry_set_init(iter); - } -} - static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data) @@ -695,9 +677,6 @@ jump_label_module_notify(struct notifier_block *self, unsigned long val, case MODULE_STATE_GOING: jump_label_del_module(mod); break; - case MODULE_STATE_LIVE: - jump_label_invalidate_module_init(mod); - break; } jump_label_unlock(); @@ -767,7 +746,8 @@ static void jump_label_update(struct static_key *key) entry = static_key_entries(key); /* if there are no users, entry can be NULL */ if (entry) - __jump_label_update(key, entry, stop); + __jump_label_update(key, entry, stop, + system_state < SYSTEM_RUNNING); } #ifdef CONFIG_STATIC_KEYS_SELFTEST From patchwork Mon Jul 2 18:11:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140846 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp248092ljj; Mon, 2 Jul 2018 11:13:23 -0700 (PDT) X-Google-Smtp-Source: AAOMgpewMYLY/oJn1DZjYm5y607mH+mN4VDoe1jwXs3YtO/VVGW6oG0TK2W3t96EEgf/r64oo2DY X-Received: by 2002:a62:6a01:: with SMTP id f1-v6mr9848594pfc.156.1530555203566; Mon, 02 Jul 2018 11:13:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530555203; cv=none; d=google.com; s=arc-20160816; b=caY/ZJHNiL5pwncZwWgcWfvhKEX41wAwV6QbFPD4hl7mcD/sbrc/csPurWQ3VZ4iqw kAC7gRr8suvH/0oI9UAhf7vOZyRB6FXCGsc8YVLQa1tl6GXHHmBjT1W+qytd8JTziKZP CiykFzepPYgyQatpHf2CqD6ZdluQ8G9jCq/fX7xavRVbNHALPrLCHZdc/idqXghKq2ws u5cdhbuPewsUnMsxI6N+SSNnnfDDT6p4XxETtLRNZUj4Wp5BJA7khebZAc7KtI3HvEM8 ny5qWhXPkPoPp4VwFe1h4i1pueagesLEcjYT6TQaB/XuRKfUSHQ3NUbzElQtHTSSqqa/ PulA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=7N9XKGd0sxRVNc0we3lJWgc+AZzZcg0kB6rVsSIgvFo=; b=hHuUrJEFbiBsIJj6t5aX5nKFod+sAIynldp85YnROGYCYQTYNtufzE3vFobRBu2Wvj agdy8OxzlBG4kO5DLkC7P2sjENskfg87WkQWtDRPzpEopdsK16IA88bx81xf/5mlffK+ 4NzQasAMqBuSsUaV1151GBUF8FsluqvaseMGE3hVWoiPL9cjClQrN8Yqz5t0voRrGyq2 5l1Y3cMLZwg9Mr4DHHqzF3gHUta9gpsI3HrdT0NBoEei25+QNsBNroyo/mN+sTKjdkSU Pew2gmbi8WiG6KAXGg14bupoOHUA+XREn7jMTSGLPwsMZud/HRyC36RJW4URCvyi4ir9 Z0qw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="YU/k/IND"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b59-v6si16421362plb.107.2018.07.02.11.13.23; Mon, 02 Jul 2018 11:13:23 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="YU/k/IND"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932128AbeGBSNW (ORCPT + 31 others); Mon, 2 Jul 2018 14:13:22 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:33821 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753411AbeGBSMM (ORCPT ); Mon, 2 Jul 2018 14:12:12 -0400 Received: by mail-wm0-f66.google.com with SMTP id l15-v6so8434961wmc.1 for ; Mon, 02 Jul 2018 11:12:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7N9XKGd0sxRVNc0we3lJWgc+AZzZcg0kB6rVsSIgvFo=; b=YU/k/INDi7ij+ctOedglsRe1RQX9OZkobjcjBTQZxe0TrWVakVSSHk/gzOjSe48e8y VBuXRqlAK8daj7ebbxM1MvemtyICuuPWJNG4ZL6wx6Tc13wGbhRJOAY/cmCBEVVBm7Si L4a5xr2wOAY9UkUzCayl09/NATos5WTWpEJ5w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=7N9XKGd0sxRVNc0we3lJWgc+AZzZcg0kB6rVsSIgvFo=; b=JLf0u5xGZf+qEvzM7rpTbyw0WqnkJuWhtmrVnGu+Ly/WSWkUeMx3CnTWoNbTLx2VWb iG7a332u7bo9FWfFf9UAL1FVeYbPUKRdyIC5I1jxqh745rgbfiLDw4/PFJoeANaGtltg TJQW0uodcylLkhJt40bR7VANJ2HDwuP67600EpabWWbMvtpgpUqLC/oCWgesMk12vyWm nw/VlsuPZ4OWC/cEOnkIq5kh28i0GKM4XCY+q9yOdfJMCGiQhOLXooERUXc3qRGs98AE PkimaB5YBFJ7dYRHOBbrUr23egrVFGt0R4Y/LDtwxa4blhp8/crC1hHQvAy9tc+W5lZa 6pZw== X-Gm-Message-State: APt69E3VDnqupBuTI+fJefWS3Y6c5aYnHl7YmDg1bslnB3zSEOByNd3J NTVNgbgErWlCULa2kAYdZt3v2A== X-Received: by 2002:a1c:b6d6:: with SMTP id g205-v6mr9662447wmf.17.1530555130943; Mon, 02 Jul 2018 11:12:10 -0700 (PDT) Received: from localhost.localdomain (151.21.90.92.rev.sfr.net. [92.90.21.151]) by smtp.gmail.com with ESMTPSA id 189-v6sm10582822wmd.17.2018.07.02.11.12.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jul 2018 11:12:10 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org Cc: Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v2 8/8] jump_table: move entries into ro_after_init region Date: Mon, 2 Jul 2018 20:11:45 +0200 Message-Id: <20180702181145.4799-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180702181145.4799-1-ard.biesheuvel@linaro.org> References: <20180702181145.4799-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The __jump_table sections emitted into the core kernel and into each module consist of statically initialized references into other parts of the code, and with the exception of entries that point into init code, which are defused at post-init time, these data structures are never modified. So let's move them into the ro_after_init section, to prevent them from being corrupted inadvertently by buggy code, or deliberately by an attacker. Signed-off-by: Ard Biesheuvel --- arch/arm/kernel/vmlinux-xip.lds.S | 1 + arch/s390/kernel/vmlinux.lds.S | 1 + include/asm-generic/vmlinux.lds.h | 11 +++++++---- kernel/module.c | 9 +++++++++ 4 files changed, 18 insertions(+), 4 deletions(-) -- 2.17.1 Reviewed-by: Kees Cook Acked-by: Jessica Yu diff --git a/arch/arm/kernel/vmlinux-xip.lds.S b/arch/arm/kernel/vmlinux-xip.lds.S index 3593d5c1acd2..763c41068ecc 100644 --- a/arch/arm/kernel/vmlinux-xip.lds.S +++ b/arch/arm/kernel/vmlinux-xip.lds.S @@ -118,6 +118,7 @@ SECTIONS RW_DATA_SECTION(L1_CACHE_BYTES, PAGE_SIZE, THREAD_SIZE) .data.ro_after_init : AT(ADDR(.data.ro_after_init) - LOAD_OFFSET) { *(.data..ro_after_init) + JUMP_TABLE_DATA } _edata = .; diff --git a/arch/s390/kernel/vmlinux.lds.S b/arch/s390/kernel/vmlinux.lds.S index f0414f52817b..a7cf61e46f88 100644 --- a/arch/s390/kernel/vmlinux.lds.S +++ b/arch/s390/kernel/vmlinux.lds.S @@ -67,6 +67,7 @@ SECTIONS __start_ro_after_init = .; .data..ro_after_init : { *(.data..ro_after_init) + JUMP_TABLE_DATA } EXCEPTION_TABLE(16) . = ALIGN(PAGE_SIZE); diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index e373e2e10f6a..ed6befa4c47b 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -256,10 +256,6 @@ STRUCT_ALIGN(); \ *(__tracepoints) \ /* implement dynamic printk debug */ \ - . = ALIGN(8); \ - __start___jump_table = .; \ - KEEP(*(__jump_table)) \ - __stop___jump_table = .; \ . = ALIGN(8); \ __start___verbose = .; \ KEEP(*(__verbose)) \ @@ -303,6 +299,12 @@ . = __start_init_task + THREAD_SIZE; \ __end_init_task = .; +#define JUMP_TABLE_DATA \ + . = ALIGN(8); \ + __start___jump_table = .; \ + KEEP(*(__jump_table)) \ + __stop___jump_table = .; + /* * Allow architectures to handle ro_after_init data on their * own by defining an empty RO_AFTER_INIT_DATA. @@ -311,6 +313,7 @@ #define RO_AFTER_INIT_DATA \ __start_ro_after_init = .; \ *(.data..ro_after_init) \ + JUMP_TABLE_DATA \ __end_ro_after_init = .; #endif diff --git a/kernel/module.c b/kernel/module.c index 7cb82e0fcac0..0d4e320e41cd 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -3349,6 +3349,15 @@ static struct module *layout_and_allocate(struct load_info *info, int flags) * Note: ro_after_init sections also have SHF_{WRITE,ALLOC} set. */ ndx = find_sec(info, ".data..ro_after_init"); + if (ndx) + info->sechdrs[ndx].sh_flags |= SHF_RO_AFTER_INIT; + /* + * Mark the __jump_table section as ro_after_init as well: these data + * structures are never modified, with the exception of entries that + * refer to code in the __init section, which are annotated as such + * at module load time. + */ + ndx = find_sec(info, "__jump_table"); if (ndx) info->sechdrs[ndx].sh_flags |= SHF_RO_AFTER_INIT;