From patchwork Wed Sep 19 06:51:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147026 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp343913ljw; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) X-Google-Smtp-Source: ANB0VdY+FnDDI1KpgFQQS17x2k0y6qVotWr5clyXm88cmgFvOB6HCIZSrqBWLkjQE8qtblFTivO7 X-Received: by 2002:a17:902:20c6:: with SMTP id v6-v6mr33427272plg.228.1537339963506; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339963; cv=none; d=google.com; s=arc-20160816; b=dph0zf510dCo5hHKkBTy1jHTtG4AaPC7AsBhv1dSV2DYcC9Mr9ybZVqTKmg/negrVK 3sUUb00R/vyM8huuKi8Zf0tNoaU45U4U4U1bsN8247m7UzrnLrPfRNMLq5/Z7hpIZS8o 8NtPufAy/y907HTh+03OIzVehVuUhhf6Z4XK7disMVMfq2dNyWBNYAPubD8AEhSraQzd UUHDQrF2B3QXyDAYTQ4DCH6Io2QFmr7Hretn1Tk9q2cfElA/hvsAPHTN/1dzpk9r6zBj QTsaYEzSw9nmXievSq3W53EufjcCXEIHGm2HrcQ+I4cdVLLw+2psy1HYb2O4DWSP4trx Ibzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=Je2400ni9A5s80feHs0HBjY1gZ6UX9x0bE0fIx90rAE=; b=PZENtDR23ouCiDHVgzjJ1TZK+bX64TTSzFNDnxbJiaHM3e62X+W+CIZ9p7jUgVSHvy Z7W0z5f0nXjUAzxfYcmxZ49F2mVzW7xQ+RIVk5QqC8IDr1pNQrprjpjZ9R/lUkIyJ0wr 1kfniCBua4fqtL3bP0J45Rj7uZOCge4FM/h4mNY+7dI+FZPqxMSYx++qdsPgw6/shZHt 523qj0COFUYb+ez2ZK0Kb9jZXU4o1wpSoJKeGj+0k3efBZAwvxbKBnoONrR/v9Wr2afI vArG0ee5XSvoExGIwZaf2x69yGyr7atIQXBlu8y+l7w1Hu4EXONekwi/+89+6+1BXsuI Oc7Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ePc7YC7A; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x6-v6si19632022pge.100.2018.09.18.23.52.43; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ePc7YC7A; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731121AbeISM3J (ORCPT + 32 others); Wed, 19 Sep 2018 08:29:09 -0400 Received: from mail-it0-f66.google.com ([209.85.214.66]:51180 "EHLO mail-it0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731086AbeISM3I (ORCPT ); Wed, 19 Sep 2018 08:29:08 -0400 Received: by mail-it0-f66.google.com with SMTP id j81-v6so6864116ite.0 for ; Tue, 18 Sep 2018 23:52:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Je2400ni9A5s80feHs0HBjY1gZ6UX9x0bE0fIx90rAE=; b=ePc7YC7AmNfXSr4FpXbzsfuPGom2b9uu9QTkOtVTTpvbQnmlm+Ab5Fda4DSvDkiNSW 2n930+omZG2krqTyIvZHbqPMRfYLPEPCf7CqbnlVZak11lPxt0+JZGVX0RaAkWHUW1Qg 5Yzihq5i7+V3yXSxUwy87y990PW7TLphcsVXA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Je2400ni9A5s80feHs0HBjY1gZ6UX9x0bE0fIx90rAE=; b=KBZ/KOps0zmg9KeEhtNB24GjbwIP+8hojRzxb5cuBmxmuIx/LFw9ckycBadMPR3HQ7 2ly2lGxuzA2PiGiv3fSaf1v/xS69MgZoCZDVaBQEhNTuafbwHQdbOMAHa5MCVQ8182+t TSM9XHv+KZeT6OnT93oCJeguzhQki3sGGo1UtCM6c1UE20Q8w5mTC1FacjEMCHzfLWeI ZcadGw0yNc8izVRUQeElF5ZV+zsH0ZskPg+W5AV5u8uWgIJfRNT8AjTpbSMs/HZwBrJq HRmuQhFSOdzswKvQ8NK1UveL9JYKbb+YRHg2RFxeS9mTkHpc4JgkoaOQV152lNRjzCdi yttQ== X-Gm-Message-State: APzg51CIbMXXgm110npmvZvuXCe7zjQH6cJAPnl+F5liquWXv6UGYF8Z Zw+0J0dYkvZ0RuGFFZvVfyU19Q8sD8rDFQ== X-Received: by 2002:a24:6401:: with SMTP id t1-v6mr21062079itc.135.1537339958595; Tue, 18 Sep 2018 23:52:38 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:37 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 7/9] jump_label: annotate entries that operate on __init code earlier Date: Tue, 18 Sep 2018 23:51:42 -0700 Message-Id: <20180919065144.25010-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Jump table entries are mostly read-only, with the exception of the init and module loader code that defuses entries that point into init code when the code being referred to is freed. For robustness, it would be better to move these entries into the ro_after_init section, but clearing the 'code' member of each jump table entry referring to init code at module load time races with the module_enable_ro() call that remaps the ro_after_init section read only, so we'd like to do it earlier. So given that whether such an entry refers to init code can be decided much earlier, we can pull this check forward. Since we may still need the code entry at this point, let's switch to setting a low bit in the 'key' member just like we do to annotate the default state of a jump table entry. Reviewed-by: Kees Cook Signed-off-by: Ard Biesheuvel --- include/linux/jump_label.h | 11 ++--- init/main.c | 1 - kernel/jump_label.c | 48 ++++++-------------- 3 files changed, 18 insertions(+), 42 deletions(-) -- 2.17.1 diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 2768a925bafa..5df6a621e464 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -141,7 +141,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - long offset = entry->key & ~1L; + long offset = entry->key & ~3L; return (struct static_key *)((unsigned long)&entry->key + offset); } @@ -160,7 +160,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + return (struct static_key *)((unsigned long)entry->key & ~3UL); } #endif @@ -172,12 +172,12 @@ static inline bool jump_entry_is_branch(const struct jump_entry *entry) static inline bool jump_entry_is_init(const struct jump_entry *entry) { - return entry->code == 0; + return (unsigned long)entry->key & 2UL; } static inline void jump_entry_set_init(struct jump_entry *entry) { - entry->code = 0; + entry->key |= 2; } #endif @@ -213,7 +213,6 @@ extern struct jump_entry __start___jump_table[]; extern struct jump_entry __stop___jump_table[]; extern void jump_label_init(void); -extern void jump_label_invalidate_initmem(void); extern void jump_label_lock(void); extern void jump_label_unlock(void); extern void arch_jump_label_transform(struct jump_entry *entry, @@ -261,8 +260,6 @@ static __always_inline void jump_label_init(void) static_key_initialized = true; } -static inline void jump_label_invalidate_initmem(void) {} - static __always_inline bool static_key_false(struct static_key *key) { if (unlikely(static_key_count(key) > 0)) diff --git a/init/main.c b/init/main.c index 18f8f0140fa0..a664246450d1 100644 --- a/init/main.c +++ b/init/main.c @@ -1064,7 +1064,6 @@ static int __ref kernel_init(void *unused) /* need to finish all async __init code before freeing the memory */ async_synchronize_full(); ftrace_free_init_mem(); - jump_label_invalidate_initmem(); free_initmem(); mark_readonly(); diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 898a1d0c38dc..e8cf3ff3149c 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -373,14 +373,15 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) static void __jump_label_update(struct static_key *key, struct jump_entry *entry, - struct jump_entry *stop) + struct jump_entry *stop, + bool init) { for (; (entry < stop) && (jump_entry_key(entry) == key); entry++) { /* * An entry->code of 0 indicates an entry which has been * disabled because it was in an init text area. */ - if (!jump_entry_is_init(entry)) { + if (init || !jump_entry_is_init(entry)) { if (kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); else @@ -420,6 +421,9 @@ void __init jump_label_init(void) if (jump_label_type(iter) == JUMP_LABEL_NOP) arch_jump_label_transform_static(iter, JUMP_LABEL_NOP); + if (init_section_contains((void *)jump_entry_code(iter), 1)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -432,19 +436,6 @@ void __init jump_label_init(void) cpus_read_unlock(); } -/* Disable any jump label entries in __init/__exit code */ -void __init jump_label_invalidate_initmem(void) -{ - struct jump_entry *iter_start = __start___jump_table; - struct jump_entry *iter_stop = __stop___jump_table; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (init_section_contains((void *)jump_entry_code(iter), 1)) - jump_entry_set_init(iter); - } -} - #ifdef CONFIG_MODULES static enum jump_label_type jump_label_init_type(struct jump_entry *entry) @@ -524,7 +515,8 @@ static void __jump_label_mod_update(struct static_key *key) stop = __stop___jump_table; else stop = m->jump_entries + m->num_jump_entries; - __jump_label_update(key, mod->entries, stop); + __jump_label_update(key, mod->entries, stop, + m->state == MODULE_STATE_COMING); } } @@ -570,6 +562,9 @@ static int jump_label_add_module(struct module *mod) for (iter = iter_start; iter < iter_stop; iter++) { struct static_key *iterk; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -605,7 +600,7 @@ static int jump_label_add_module(struct module *mod) /* Only update if we've changed from our initial state */ if (jump_label_type(iter) != jump_label_init_type(iter)) - __jump_label_update(key, iter, iter_stop); + __jump_label_update(key, iter, iter_stop, true); } return 0; @@ -661,19 +656,6 @@ static void jump_label_del_module(struct module *mod) } } -/* Disable any jump label entries in module init code */ -static void jump_label_invalidate_module_init(struct module *mod) -{ - struct jump_entry *iter_start = mod->jump_entries; - struct jump_entry *iter_stop = iter_start + mod->num_jump_entries; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(jump_entry_code(iter), mod)) - jump_entry_set_init(iter); - } -} - static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data) @@ -695,9 +677,6 @@ jump_label_module_notify(struct notifier_block *self, unsigned long val, case MODULE_STATE_GOING: jump_label_del_module(mod); break; - case MODULE_STATE_LIVE: - jump_label_invalidate_module_init(mod); - break; } jump_label_unlock(); @@ -767,7 +746,8 @@ static void jump_label_update(struct static_key *key) entry = static_key_entries(key); /* if there are no users, entry can be NULL */ if (entry) - __jump_label_update(key, entry, stop); + __jump_label_update(key, entry, stop, + system_state < SYSTEM_RUNNING); } #ifdef CONFIG_STATIC_KEYS_SELFTEST