From patchwork Wed Sep 19 06:51:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147020 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp343663ljw; Tue, 18 Sep 2018 23:52:21 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYdLMh78XtSqejOqOdD84hCm0Azao5527D28ptkyx8R2gULsAukXgHGLbOjqdXAdE/23Cec X-Received: by 2002:a63:5ec1:: with SMTP id s184-v6mr31523862pgb.26.1537339941837; Tue, 18 Sep 2018 23:52:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339941; cv=none; d=google.com; s=arc-20160816; b=q0DykOLoZfYwxjeZBL5GhXnMQ3MC3FSj2P1cVDEgUOsVSs3+vos/0yP2AHR2JXB66C tfQEZosbMLsRjTrM1ZplftSmxipnWDP729I3mgjmixu/nkOeVOTcmy6qGLfZw5pxsH5t 3QKRFfluMwnO0pKT4Vo177M9wmCYaOG2zrRO2RvxpyId2HKyhRM7q4/8Hx4FehMQsD6v 43KSujCZmW4UUTNtaZtoI7H9OQajteR/zWzkyddKtME7ZwH1EL1bN2EGFCyTyMbtNmZ0 mB6B0DZkwVziG1Q1J0pomwp5qib9fkdjA+dfM1Q5OwxNx4o1Ji/oaGGCuOXm/uhO7zcM Jz1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=FZJNEVNflrAxT+1Z0Hv5uIC0eQ5VzZc2FwvWytlu3MQ=; b=bgvugxsf2PWyFTP9ZL3aHrSqfvD5XCfe7xr6cf0HFdmJOAre0W/wfGrtCFuUF3HFSY /LPB5HPzGp2zc7c/kslt+c0yopfpqpscXLML/fPN1G59nUpT6QlAfcYFL1ATSdJa5zwQ GFZ6jeAMGWv5kNbXaxVYJhkw4xuVTCKk6tthZqZge/I/HjCUDbf1Y7uQ5iZSmiiydJNM TU+dFVwRa9Cw8+62+9mjaTMeWTvutgyTgoInwfohLJfU/jkHaHeSFOxA5QJH+DVS8YoC Sw8avjD20VDwkZDrVbzjn/kuuGT647wRZ3Kh/F1C1A5PuGfFHUep6dBjNAM9T2L9AXW8 UetQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EhK3obIX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b13-v6si20418573pgh.255.2018.09.18.23.52.21; Tue, 18 Sep 2018 23:52:21 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EhK3obIX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730942AbeISM2m (ORCPT + 32 others); Wed, 19 Sep 2018 08:28:42 -0400 Received: from mail-io1-f67.google.com ([209.85.166.67]:43766 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728297AbeISM2m (ORCPT ); Wed, 19 Sep 2018 08:28:42 -0400 Received: by mail-io1-f67.google.com with SMTP id y10-v6so3588947ioa.10 for ; Tue, 18 Sep 2018 23:52:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FZJNEVNflrAxT+1Z0Hv5uIC0eQ5VzZc2FwvWytlu3MQ=; b=EhK3obIXXfTsziRS4vydKb2tw/b8QbSumbZwfBdocf8KWlnGbvytD6YkoFzpwgpz8o YW5z7ju0gy9iwK67mFRVf1NQlXfH+qCIOW+LgDoyAhmatpvTE7mGtXg7VqGeuSes5EKK N6syiiph9gKTA+xyTGgM1CO1Gvw/wJNtuCdgc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FZJNEVNflrAxT+1Z0Hv5uIC0eQ5VzZc2FwvWytlu3MQ=; b=bMBaXinzDWcQQ34yQ9nOKUvOH4qRG7Y7LnHpKx1K+0+lyUYpw33Nl/yGRkF77o+H8d LsmIMJyvXPN+BZQb6cypZoZEqBXZzzxaQSUjZwFlKH/IN0AlonD6P3nrFxZNf89gCPjq JYMNKteBogBQMtied5wwYDzGSRy7y6N8MPRKWne48WGEyO/rsIbF/eI0xls5kIH6h8Ud 9lds8QAlfX4a/YXHCHwOInWqArK8Dg7yRu1XBsQFxLuiq+pzN4I+Dnv+mRTbQTxmesQD wy55yexrLBgxMtPs8HknjqibOe5vw6MeDlyM4l1LFurZpRzz/rELhLfjfJCSCH8FLOYq dHYA== X-Gm-Message-State: APzg51CfyMY6Zev93XztpZaMmS4a3vHCle9Ch0DE+H/qblI9vsXOTiKo R0X4Q6DSckiVtfM4bod3rZNRZkifimM6cw== X-Received: by 2002:a6b:9d95:: with SMTP id g143-v6mr28303195ioe.46.1537339931935; Tue, 18 Sep 2018 23:52:11 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:11 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 1/9] kernel/jump_label: abstract jump_entry member accessors Date: Tue, 18 Sep 2018 23:51:36 -0700 Message-Id: <20180919065144.25010-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation of allowing architectures to use relative references in jump_label entries [which can dramatically reduce the memory footprint], introduce abstractions for references to the 'code' and 'key' members of struct jump_entry. Signed-off-by: Ard Biesheuvel --- include/linux/jump_label.h | 34 +++++++++++++++++ kernel/jump_label.c | 40 ++++++++------------ 2 files changed, 49 insertions(+), 25 deletions(-) -- 2.17.1 diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 1a0b6f17a5d6..2eadff9b3b90 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -119,6 +119,40 @@ struct static_key { #ifdef HAVE_JUMP_LABEL #include + +#ifndef __ASSEMBLY__ + +static inline unsigned long jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline unsigned long jump_entry_target(const struct jump_entry *entry) +{ + return entry->target; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#endif #endif #ifndef __ASSEMBLY__ diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 2e62503bea0d..834e43de0daf 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -38,10 +38,10 @@ static int jump_label_cmp(const void *a, const void *b) const struct jump_entry *jea = a; const struct jump_entry *jeb = b; - if (jea->key < jeb->key) + if (jump_entry_key(jea) < jump_entry_key(jeb)) return -1; - if (jea->key > jeb->key) + if (jump_entry_key(jea) > jump_entry_key(jeb)) return 1; return 0; @@ -261,8 +261,8 @@ EXPORT_SYMBOL_GPL(jump_label_rate_limit); static int addr_conflict(struct jump_entry *entry, void *start, void *end) { - if (entry->code <= (unsigned long)end && - entry->code + JUMP_LABEL_NOP_SIZE > (unsigned long)start) + if (jump_entry_code(entry) <= (unsigned long)end && + jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE > (unsigned long)start) return 1; return 0; @@ -321,16 +321,6 @@ static inline void static_key_set_linked(struct static_key *key) key->type |= JUMP_TYPE_LINKED; } -static inline struct static_key *jump_entry_key(struct jump_entry *entry) -{ - return (struct static_key *)((unsigned long)entry->key & ~1UL); -} - -static bool jump_entry_branch(struct jump_entry *entry) -{ - return (unsigned long)entry->key & 1UL; -} - /*** * A 'struct static_key' uses a union such that it either points directly * to a table of 'struct jump_entry' or to a linked list of modules which in @@ -355,7 +345,7 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool enabled = static_key_enabled(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return enabled ^ branch; @@ -370,12 +360,12 @@ static void __jump_label_update(struct static_key *key, * An entry->code of 0 indicates an entry which has been * disabled because it was in an init text area. */ - if (entry->code) { - if (kernel_text_address(entry->code)) + if (!jump_entry_is_init(entry)) { + if (kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); else WARN_ONCE(1, "can't patch jump_label at %pS", - (void *)(unsigned long)entry->code); + (void *)jump_entry_code(entry)); } } } @@ -430,8 +420,8 @@ void __init jump_label_invalidate_initmem(void) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (init_section_contains((void *)(unsigned long)iter->code, 1)) - iter->code = 0; + if (init_section_contains((void *)jump_entry_code(iter), 1)) + jump_entry_set_init(iter); } } @@ -441,7 +431,7 @@ static enum jump_label_type jump_label_init_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool type = static_key_type(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return type ^ branch; @@ -565,7 +555,7 @@ static int jump_label_add_module(struct module *mod) continue; key = iterk; - if (within_module(iter->key, mod)) { + if (within_module((unsigned long)key, mod)) { static_key_set_entries(key, iter); continue; } @@ -615,7 +605,7 @@ static void jump_label_del_module(struct module *mod) key = jump_entry_key(iter); - if (within_module(iter->key, mod)) + if (within_module((unsigned long)key, mod)) continue; /* No memory during module load */ @@ -659,8 +649,8 @@ static void jump_label_invalidate_module_init(struct module *mod) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(iter->code, mod)) - iter->code = 0; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_init(iter); } } From patchwork Wed Sep 19 06:51:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147021 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp343666ljw; Tue, 18 Sep 2018 23:52:22 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYqtoS8ERq5H8FqiyaDiX4R/BjMYMJLrByqUaxpxUlMg8xe0YIp7biQw4H9ZSmcLcmogDKp X-Received: by 2002:a17:902:5a4e:: with SMTP id f14-v6mr33293131plm.311.1537339942214; Tue, 18 Sep 2018 23:52:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339942; cv=none; d=google.com; s=arc-20160816; b=wjfgSbtek99nMXKDwjvzZLtEL36xDNjOGl/2mKSXsSifmEjU9ffJs7N0nWaEl1O/p8 hPQRE88zNjw7hADV9OBteCyeaSX6JUmmUNjjFzg2PIS1j3gL4BY7uqnwB/y6bl5lszMw zrOjcy4uDfr33/B2mGjw4oY9QMrfVvF7AKV4kPWUGGyQ9Ukxehk7IF7vlzRRchEFbCQi SA5YjOeiaG9nsdL44xqbkS+K/5C+B79J2So0Q99fH7KDwLQdi8GKHvk2slcg51ootTJm 4In2pCd0fYdAHTHldX7CFvVh7L3C/WwjqNgbNC24WX2Jk7nGDn1k7oSHLZihwXDNOfKC QWzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=Ve/tzdX0J+Uuck6FHDdYfUVN2t3T9LHXWJmCGGfmOhA=; b=z0fR7X3twFcHeqPWe77/sEpVT0dw5+KxuoMvRqo1rE1Q1kMcj0A0TF2mb1M+X9Jbq7 wGUKI7c4i7+fZjQddRFjmzBo5fv+XZ8Gf6joAHuDGDPebUyC+uaiDf4GVpsBZInIYWfP QUzmjxYJt8EfRs9g2YQdhi2xjAfTBE8pOXP86WoJFbBnSkCFE0BzKJ6JiJGukkjRohv1 dNNNVrgnsIP9h2i/8YDMSasmqyzrNNmJ2VG/EL6v6LA7JrNfgHJ++mxEACS+poEAPl5h mR7oyyNGnlGcwnPIjUkCZz97h43RqnnsQPGW/gDAGV43UBP/vFD/RqgB+NRdZoEnBSSU tieA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XSBs8g1F; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b13-v6si20418573pgh.255.2018.09.18.23.52.21; Tue, 18 Sep 2018 23:52:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XSBs8g1F; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730973AbeISM2q (ORCPT + 32 others); Wed, 19 Sep 2018 08:28:46 -0400 Received: from mail-it0-f67.google.com ([209.85.214.67]:34817 "EHLO mail-it0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730951AbeISM2q (ORCPT ); Wed, 19 Sep 2018 08:28:46 -0400 Received: by mail-it0-f67.google.com with SMTP id 139-v6so6401236itf.0 for ; Tue, 18 Sep 2018 23:52:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ve/tzdX0J+Uuck6FHDdYfUVN2t3T9LHXWJmCGGfmOhA=; b=XSBs8g1FXx8+MYOqu4x8BUS5rCmHRqZYI9aGmmBiXo0QwJ2kIR6WDnc9UUL/Bu+m+y zVeou4wFWxg8G6IoGz4AeO3MicrYY+zjoYV6Eiy/DHCjJnu8y++FAsajVh0ji7RNYmex cJ5/VtZNVe4avLVH3jZwR66wSdQsdSIkkVsjg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ve/tzdX0J+Uuck6FHDdYfUVN2t3T9LHXWJmCGGfmOhA=; b=PqQAkswS+5zRwTHVLK5m6xSSsACA4XeeHBUk/8BG6b9GjZrjrBNwGA2ytEpbBV6+pr X8J77y863Y0rzxVm+nsfhb3oU9+zBTjUgF4xGPqGtX4PS+5GVaa62a0sFlpwRIyMGGcH 6hMKb+q81EE6ODCRSRuPLmdO8FITO4ONsu8CZakWFXW9FQ3NNiqiceEYby5KxJoBVPUB QDCDVNfSY158XQmr/YZ1I+E6frq3v/d1mm5JGuKBIefNeMlk8x+weVZn4RpfnB62ow45 nNFBjAT3mtB5OV0YURHqaXLW++QWl+qmEx/Rq/FawmU8RhN9VkHQHQ7s+3p7qiDd25OR zoPQ== X-Gm-Message-State: APzg51CUkCw7A0Uw0XyUlk+HyLMirZ36xHyqlwZHkmFoT6wRxZE+IrgS 4dcxyXcMSSSxazgsfpBVXHo0o+NkKQCJoQ== X-Received: by 2002:a24:6e0f:: with SMTP id w15-v6mr20900860itc.77.1537339936033; Tue, 18 Sep 2018 23:52:16 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:15 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 2/9] kernel/jump_label: implement generic support for relative references Date: Tue, 18 Sep 2018 23:51:37 -0700 Message-Id: <20180919065144.25010-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org To reduce the size taken up by absolute references in jump label entries themselves and the associated relocation records in the .init segment, add support for emitting them as relative references instead. Note that this requires some extra care in the sorting routine, given that the offsets change when entries are moved around in the jump_entry table. Signed-off-by: Ard Biesheuvel --- arch/Kconfig | 3 +++ include/linux/jump_label.h | 28 ++++++++++++++++++++ kernel/jump_label.c | 22 ++++++++++++++- 3 files changed, 52 insertions(+), 1 deletion(-) -- 2.17.1 diff --git a/arch/Kconfig b/arch/Kconfig index 6801123932a5..9d329608913e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -359,6 +359,9 @@ config HAVE_PERF_USER_STACK_DUMP config HAVE_ARCH_JUMP_LABEL bool +config HAVE_ARCH_JUMP_LABEL_RELATIVE + bool + config HAVE_RCU_TABLE_FREE bool diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 2eadff9b3b90..2768a925bafa 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -121,6 +121,32 @@ struct static_key { #include #ifndef __ASSEMBLY__ +#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE + +struct jump_entry { + s32 code; + s32 target; + long key; // key may be far away from the core kernel under KASLR +}; + +static inline unsigned long jump_entry_code(const struct jump_entry *entry) +{ + return (unsigned long)&entry->code + entry->code; +} + +static inline unsigned long jump_entry_target(const struct jump_entry *entry) +{ + return (unsigned long)&entry->target + entry->target; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + long offset = entry->key & ~1L; + + return (struct static_key *)((unsigned long)&entry->key + offset); +} + +#else static inline unsigned long jump_entry_code(const struct jump_entry *entry) { @@ -137,6 +163,8 @@ static inline struct static_key *jump_entry_key(const struct jump_entry *entry) return (struct static_key *)((unsigned long)entry->key & ~1UL); } +#endif + static inline bool jump_entry_is_branch(const struct jump_entry *entry) { return (unsigned long)entry->key & 1UL; diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 834e43de0daf..898a1d0c38dc 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -47,14 +47,34 @@ static int jump_label_cmp(const void *a, const void *b) return 0; } +static void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + static void jump_label_sort_entries(struct jump_entry *start, struct jump_entry *stop) { unsigned long size; + void *swapfn = NULL; + + if (IS_ENABLED(CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE)) + swapfn = jump_label_swap; size = (((unsigned long)stop - (unsigned long)start) / sizeof(struct jump_entry)); - sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL); + sort(start, size, sizeof(struct jump_entry), jump_label_cmp, swapfn); } static void jump_label_update(struct static_key *key); From patchwork Wed Sep 19 06:51:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147022 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp343738ljw; Tue, 18 Sep 2018 23:52:27 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYUEcOPSsJzoozfR6bJ3r1p+6EpSPNrurJaC0NiUaocDJUowkJ3Gblduy8DMGL42jZNT1NY X-Received: by 2002:a17:902:c7:: with SMTP id a65-v6mr32533408pla.264.1537339947047; Tue, 18 Sep 2018 23:52:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339947; cv=none; d=google.com; s=arc-20160816; b=J002xX1Zl/pKjS4Gpi6sf2cyU5vEnuyYvUvRerlabB1MiUM0hx08wv9/aLiJsfNo1d kYD23QNtJBwCBwfKUTP6YDKWsNxpAyzyTURKQHM29+cgphmhLGSGarLDsLueTvzUBBPA vooHMR75m17dKYQujXckt+qvw+CRY473Kc/QlADpvy6Qy2W3pXKjvphvBNMpZkaTcVkr 0IbuOcRxmX4ZsTFzmnq8JcdGpAqYE/XAbQMqViYQbMhU0YxfaM2El1Q0rG5+8fe90pIL E+0Er1aBOkiMjoYWC4R9izFUXCUD8RCRAjgxfj+p87NRSfUk32wGz48hFIyaXi3a0FuJ K77Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=TGyPYi2VRIFEPcRhb8tIzZMhU5s7IYbBe3KULe7Oy1M=; b=G9EGpyGNWN1EMbSzIPOf2kWCW/8GYSqxQPBzyuoYq5Q5BzJkHIaBN72IeHK6Z3TQ7D 1GZzAqkyomT9Db9uCvGSlgQacGdZ+xkFVRr4vAv7zQzNdTbYFM6Dk3PmpLfL8+gfCnAy wwhxX/puKY+ptpWYuUJ01zVpqavsTYZhvAr3eXY7S6SiNhglBdC32TggOycHwLi++Xom Guhvm68XbSItP1wM+vM9HMmSCt+B99pCp4HqhzBlIulLFM3eWKmHf2GDXllPKT/fr/j5 /wom7aihal0ydeTRmPJhOfVg3aUkEz3DglaNDl8c8UI8h1AU8dMWs/D8BfTBDtHyaF7i eevw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RX689+zq; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h3-v6si19826303plb.249.2018.09.18.23.52.26; Tue, 18 Sep 2018 23:52:27 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RX689+zq; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730997AbeISM2v (ORCPT + 32 others); Wed, 19 Sep 2018 08:28:51 -0400 Received: from mail-it0-f65.google.com ([209.85.214.65]:37522 "EHLO mail-it0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730951AbeISM2u (ORCPT ); Wed, 19 Sep 2018 08:28:50 -0400 Received: by mail-it0-f65.google.com with SMTP id h20-v6so6387970itf.2 for ; Tue, 18 Sep 2018 23:52:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TGyPYi2VRIFEPcRhb8tIzZMhU5s7IYbBe3KULe7Oy1M=; b=RX689+zqYWhkAwVV303gAnGbrJ9CzrBF+KgU2p+y7qbKoXHWycb1HUUPkpNb3+j9xw nYaYaat5echYppzD0W+cO6BYfEwuoSP6FJyKDc6DONSA8nY79Jtkstw4r8B3d+GPZ+r6 W5Ip07ogGOZ8mv9fOKV/zmGUvyGDwwqxr+AXY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TGyPYi2VRIFEPcRhb8tIzZMhU5s7IYbBe3KULe7Oy1M=; b=M3GYe26YUsTnRH0+TgKJHuVRSlMI7h/ZfQN4OLMt6qtys3fHUzfvWt/CwRFTGd79Yi Np1Yw4TZkbAXrnukOiNT5pwSe91qwffpCZoh1/b+cEIv+XVoe7CoUDXV+RUI+ukXdp5a +Q5bJlkg+Ee8++OjwvKO0osxBZHFseK+0TtEGVyUKVVO3rc+C81u2S74saT8STUTqrDM XLlItlMRb33WneZXfFSe5keBxj6F5ud5EMlTZJu7V7bTnC8iCgqeuSVxYU67+oCRtKQr EvDpOc6DzhFsv1TIE8PCnOzW1LypBVVEXoKcuOJLKGqji+vQQHlGlf1jXgFe41bVQ2Zz PfGw== X-Gm-Message-State: APzg51D4jxT+A3UPv+T5HkV19JEWcRDrHmvCMVSG5OZWLFMB2pGV0x24 Qqs7KNJ13B1VWO8MxfqjQR0brWEmY0oD2w== X-Received: by 2002:a02:8647:: with SMTP id e65-v6mr30385365jai.62.1537339940402; Tue, 18 Sep 2018 23:52:20 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:19 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 3/9] arm64/kernel: jump_label: switch to relative references Date: Tue, 18 Sep 2018 23:51:38 -0700 Message-Id: <20180919065144.25010-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On a randomly chosen distro kernel build for arm64, vmlinux.o shows the following sections, containing jump label entries, and the associated RELA relocation records, respectively: ... [38088] __jump_table PROGBITS 0000000000000000 00e19f30 000000000002ea10 0000000000000000 WA 0 0 8 [38089] .rela__jump_table RELA 0000000000000000 01fd8bb0 000000000008be30 0000000000000018 I 38178 38088 8 ... In other words, we have 190 KB worth of 'struct jump_entry' instances, and 573 KB worth of RELA entries to relocate each entry's code, target and key members. This means the RELA section occupies 10% of the .init segment, and the two sections combined represent 5% of vmlinux's entire memory footprint. So let's switch from 64-bit absolute references to 32-bit relative references for the code and target field, and a 64-bit relative reference for the 'key' field (which may reside in another module or the core kernel, which may be more than 4 GB way on arm64 when running with KASLR enable): this reduces the size of the __jump_table by 33%, and gets rid of the RELA section entirely. Acked-by: Will Deacon Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/jump_label.h | 38 ++++++++++---------- arch/arm64/kernel/jump_label.c | 6 ++-- 3 files changed, 22 insertions(+), 23 deletions(-) -- 2.17.1 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1b1a0e95c751..16e1e2c1e77b 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -104,6 +104,7 @@ config ARM64 select HAVE_ARCH_BITREVERSE select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 7e2b3e360086..472023498d71 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -26,13 +26,16 @@ #define JUMP_LABEL_NOP_SIZE AARCH64_INSN_SIZE -static __always_inline bool arch_static_branch(struct static_key *key, bool branch) +static __always_inline bool arch_static_branch(struct static_key *key, + bool branch) { - asm_volatile_goto("1: nop\n\t" - ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" - ".popsection\n\t" + asm_volatile_goto( + "1: nop \n\t" + " .pushsection __jump_table, \"aw\" \n\t" + " .align 3 \n\t" + " .long 1b - ., %l[l_yes] - . \n\t" + " .quad %c0 - . \n\t" + " .popsection \n\t" : : "i"(&((char *)key)[branch]) : : l_yes); return false; @@ -40,13 +43,16 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran return true; } -static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch) +static __always_inline bool arch_static_branch_jump(struct static_key *key, + bool branch) { - asm_volatile_goto("1: b %l[l_yes]\n\t" - ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" - ".popsection\n\t" + asm_volatile_goto( + "1: b %l[l_yes] \n\t" + " .pushsection __jump_table, \"aw\" \n\t" + " .align 3 \n\t" + " .long 1b - ., %l[l_yes] - . \n\t" + " .quad %c0 - . \n\t" + " .popsection \n\t" : : "i"(&((char *)key)[branch]) : : l_yes); return false; @@ -54,13 +60,5 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool return true; } -typedef u64 jump_label_t; - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index e0756416e567..646b9562ee64 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -25,12 +25,12 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - void *addr = (void *)entry->code; + void *addr = (void *)jump_entry_code(entry); u32 insn; if (type == JUMP_LABEL_JMP) { - insn = aarch64_insn_gen_branch_imm(entry->code, - entry->target, + insn = aarch64_insn_gen_branch_imm(jump_entry_code(entry), + jump_entry_target(entry), AARCH64_INSN_BRANCH_NOLINK); } else { insn = aarch64_insn_gen_nop(); From patchwork Wed Sep 19 06:51:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147023 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp343753ljw; Tue, 18 Sep 2018 23:52:28 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb/aGPrXGzncQabu7Ggmg0nS9Wg/XPMPRp4bl5jq4zQ9/ebmPw+gInCGSoTTivbeZUN5OYD X-Received: by 2002:a17:902:9307:: with SMTP id bc7-v6mr32489070plb.225.1537339948326; Tue, 18 Sep 2018 23:52:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339948; cv=none; d=google.com; s=arc-20160816; b=WPK5k03NYtbWiXiVf57Tx9BKMqyanOdoPoeUzNa/ip5sW0Ty2SddcSpvrzF53Ot1lP RBKdCLfM8EEzt4ngM7VyChJ5DOIkw2aGkAAodYi612iR+cXBElvndSIe/7fU7WmwiAEO lruVeEq5ONEL/mVu87HAITjLtBXfVMp0dp8GajlgnszRcLxBDqFQIUKd15iS6QZ361tw c1s9Pm+l4JVDXRnob6I9lPO1/NFtbVAc0dLklJZYSA3HojtIzgj/Uh1QgmoxwtKDKEIp R4MXAfkpol009TArnWGsIbJrAoV/HS9kHPu+cdMW9DgA+724oBe093o54lCeaCwMtMir Bhdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=vjlNGgKK+hdQ2oIOvWFpRpQO96t5oT9NlwIzOhMdM8o=; b=uPHjWqpKx08aYpEziYyaD/YdjQSng13M5KefX7XH0L4C8GrdnZRJrcpyPwuovhtgw7 hvGgundnAPB8tqtM4qoidCEndugYk1NdKbPnlGKhVPUtbs+SP30ZCuvmQh1TnIPnk2/U d+evZqUY1+nS5mChHTg5hn9fqR33yeMtO0MTX+1+/HuRZDC4sDXL2IlqvLATLdNzoOZN ricYLlbL6J/vTQ2iJrZ1ltLY8N4XJGG6gkKSGh8BYq7qNoX2hL8ogLvxGNF4EdXVU7UZ CVejcAtyJYToXzm17YE3BqsMfnbZ+azQ67S0ezCJg6IHmjuRYQWzASU2FkwOz+WsOFts LtuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=DhDtIlRo; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y25-v6si21275650pga.39.2018.09.18.23.52.28; Tue, 18 Sep 2018 23:52:28 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=DhDtIlRo; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731030AbeISM2y (ORCPT + 32 others); Wed, 19 Sep 2018 08:28:54 -0400 Received: from mail-it0-f65.google.com ([209.85.214.65]:52726 "EHLO mail-it0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731001AbeISM2y (ORCPT ); Wed, 19 Sep 2018 08:28:54 -0400 Received: by mail-it0-f65.google.com with SMTP id h3-v6so6840711ita.2 for ; Tue, 18 Sep 2018 23:52:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vjlNGgKK+hdQ2oIOvWFpRpQO96t5oT9NlwIzOhMdM8o=; b=DhDtIlRo84pAR/lLfZZdTtSpY4IbWrE2M7Q3fbmY0W8iqGZiJJMONls0dIy7dmO7yE VpiQJ9v47+5DZd1eceatrZ7qN+PeX+XwJhjOIasK6oYUVfSk2VmSzZffnSJbDgZ302fQ 3dX12Q4QmLBeuTPo7bvGVMxg5fjcnMLlZmt1I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vjlNGgKK+hdQ2oIOvWFpRpQO96t5oT9NlwIzOhMdM8o=; b=G3ebBzyz9gtFnhk9OHRbcj+FsSWhYMkIMjYLVGa+BqA+7gOp/rjUdLZtVBqvGVDLk5 +xF09XJ3bicRgT4yJa1oM6DqLOw/HJf6MMYMHYqYc36Sl44k2m+ow+hA9sqVu5kbEPS2 QlPPv3BZW7uNVQr7o+qNtst3nIF5c4x6Zh0jO2Rrz88I+Mavh03S/BC/XH5vo7ezGggg uRM34voxUpZqRL78rallOGVPQXgbXJ7AsxUsun4EBIFzYm6jJK8deewzeJqfXu0NvslV bWspTpyjFHqXTJ0FPCd1dAwMJp+dg1bV8rWyokhcsSWhm13xetRegP3EN/y6xq/+YOqU hmdA== X-Gm-Message-State: APzg51B2ZXuO2ryjkLW/v1EZLbjgsaefJq3igylb8wcQIU/1F18yJcrP 3vcDL6LuQvECpd9jzK6progWoj9Sa03Qjg== X-Received: by 2002:a24:4a83:: with SMTP id k125-v6mr19923669itb.121.1537339944194; Tue, 18 Sep 2018 23:52:24 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:23 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 4/9] x86: add support for 64-bit place relative relocations Date: Tue, 18 Sep 2018 23:51:39 -0700 Message-Id: <20180919065144.25010-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add support for R_X86_64_PC64 relocations, which operate on 64-bit quantities holding a relative symbol reference. Also remove the definition of R_X86_64_NUM: given that it is currently unused, it is unclear what the new value should be. Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/elf.h | 3 +-- arch/x86/kernel/module.c | 6 ++++++ arch/x86/tools/relocs.c | 10 ++++++++++ arch/x86/um/asm/elf.h | 3 +-- 4 files changed, 18 insertions(+), 4 deletions(-) -- 2.17.1 diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h index 0d157d2a1e2a..a357031d85b5 100644 --- a/arch/x86/include/asm/elf.h +++ b/arch/x86/include/asm/elf.h @@ -62,8 +62,7 @@ typedef struct user_fxsr_struct elf_fpxregset_t; #define R_X86_64_PC16 13 /* 16 bit sign extended pc relative */ #define R_X86_64_8 14 /* Direct 8 bit sign extended */ #define R_X86_64_PC8 15 /* 8 bit sign extended pc relative */ - -#define R_X86_64_NUM 16 +#define R_X86_64_PC64 24 /* Place relative 64-bit signed */ /* * These are used to set parameters in the core dumps. diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index f58336af095c..b052e883dd8c 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -201,6 +201,12 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, goto overflow; #endif break; + case R_X86_64_PC64: + if (*(u64 *)loc != 0) + goto invalid_relocation; + val -= (u64)loc; + *(u64 *)loc = val; + break; default: pr_err("%s: Unknown rela relocation: %llu\n", me->name, ELF64_R_TYPE(rel[i].r_info)); diff --git a/arch/x86/tools/relocs.c b/arch/x86/tools/relocs.c index 3a6c8ebc8032..0b08067c45f3 100644 --- a/arch/x86/tools/relocs.c +++ b/arch/x86/tools/relocs.c @@ -196,6 +196,7 @@ static const char *rel_type(unsigned type) #if ELF_BITS == 64 REL_TYPE(R_X86_64_NONE), REL_TYPE(R_X86_64_64), + REL_TYPE(R_X86_64_PC64), REL_TYPE(R_X86_64_PC32), REL_TYPE(R_X86_64_GOT32), REL_TYPE(R_X86_64_PLT32), @@ -782,6 +783,15 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym, add_reloc(&relocs32neg, offset); break; + case R_X86_64_PC64: + /* + * Only used by jump labels + */ + if (is_percpu_sym(sym, symname)) + die("Invalid R_X86_64_PC64 relocation against per-CPU symbol %s\n", + symname); + break; + case R_X86_64_32: case R_X86_64_32S: case R_X86_64_64: diff --git a/arch/x86/um/asm/elf.h b/arch/x86/um/asm/elf.h index 548197212a45..413f3519d9a1 100644 --- a/arch/x86/um/asm/elf.h +++ b/arch/x86/um/asm/elf.h @@ -116,8 +116,7 @@ do { \ #define R_X86_64_PC16 13 /* 16 bit sign extended pc relative */ #define R_X86_64_8 14 /* Direct 8 bit sign extended */ #define R_X86_64_PC8 15 /* 8 bit sign extended pc relative */ - -#define R_X86_64_NUM 16 +#define R_X86_64_PC64 24 /* Place relative 64-bit signed */ /* * This is used to ensure we don't load something for the wrong architecture. From patchwork Wed Sep 19 06:51:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147024 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp343806ljw; Tue, 18 Sep 2018 23:52:32 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZjnrWwy+v4N5ilba0L3EjYh3uY3SdQKmy4/1eYgxfRhQIrZgXa9yaxKywDeOHFxvn1Ja7C X-Received: by 2002:a62:5543:: with SMTP id j64-v6mr34042075pfb.188.1537339952303; Tue, 18 Sep 2018 23:52:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339952; cv=none; d=google.com; s=arc-20160816; b=Usyr/NuWBnBkjzwqhHH7kzeXBD+ajoqI/NKdr/Up5sAiMx62EnOz0udbCc6BPqKeEo fesAzSLHb3fWyx28/F2L4s4YTIk+NTfJlFdEAjsykXGoETxqJ27GrJBZosehmZYPGqhK 1NsiPQjZq+Q+VId2f3IF7RtzVU9Ogs4m/AD3+xfRAYDYTBlkoRfFF6Sz998h/M3HcYvs Ee3A/n+0kntBTuqq2f/NOY1bza0MfLD+dvLLonBuwSUJO/8BCWe2zT5ujd1Qu1ZSfPLx LEi0sN2QYJdZFfDaMT/KZZsuQRBn6/mnG1yYA+I3QnQldbBHolHGM+yIz8T5hKXBMcPB ncWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=NQAhy2Ey7Mb6Ox0BCHyj7wjUntgBhLLd89LZtlZjBHU=; b=FnvDWas7KWLQNO2R+oypbVjl8mUlRfRnAs40wGiLTxFL62w1+xpUbyYPXX5gzmdqIj tBRQgMWQfeD0r0Dm4RUQy0jTGoBchxxSy+axbz9zBaZ7lYsX1Cr8VN3rFPDeqEqmlxNi XF0YlGL6s5S6nIAZgNNIx/b70tNP7bC6T9JkmCgPQYKMS+772LYJND0b4Azs4/j2KRJj 7EX1fes6AJltc95e2UkpChj3ESrFIOGpSb81C6BsZBp0fSqYpTJ6XARFUec+52SkvkQg ZIStWUy4AEbbtAPa5BX9/4xkgkuDz8a8w0Gi9E0tY8CMAbg8HSnRbb/ezCtBhZx9AF4B PtVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NsN3AGnv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t64-v6si19375292pgd.176.2018.09.18.23.52.31; Tue, 18 Sep 2018 23:52:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NsN3AGnv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731059AbeISM27 (ORCPT + 32 others); Wed, 19 Sep 2018 08:28:59 -0400 Received: from mail-it0-f65.google.com ([209.85.214.65]:34502 "EHLO mail-it0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731001AbeISM26 (ORCPT ); Wed, 19 Sep 2018 08:28:58 -0400 Received: by mail-it0-f65.google.com with SMTP id x79-v6so16348118ita.1 for ; Tue, 18 Sep 2018 23:52:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NQAhy2Ey7Mb6Ox0BCHyj7wjUntgBhLLd89LZtlZjBHU=; b=NsN3AGnv7VsZoqzNrVLUZ4ZX6yXb4cVP5GslK5jPZ0Y4tbBxC1B7e1XNykxHAR/3nT Xwjbvtf7nZ33/GqY6GYPE+PxBEbnXSd7i1+38SmgzfPHMcuuuVMJTKvhYZemWDB1m4QH Ja5JsrtYPbYr70dgGUwnDQM80n6TlQiQHp9AQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NQAhy2Ey7Mb6Ox0BCHyj7wjUntgBhLLd89LZtlZjBHU=; b=W/cmJwP9OVsmkPg4cepxH6Q3gdvRZ/3h2nhAOPVRcWRwr1yYfjzbOtOtyqmF5FSrZz sYBYyDX5ABZwekI7HdXEZzGK0YryohKqGyMDtgeT+J3mUmE0hAkNLeU7lG4PmPBKEugQ kKo+wFvE6qiM3uC3ELirYU5afT9oegiYCQelIeJ3D5ljLN9S8Y7WW6Ygdl5Cfy2EN3Ly V38AIqT9QIv+9EH54jv+DNOigOV4JrfIggu5EXfx3wVvAvf31qtm/uw+Rx7H3nT2ljBm sq5PjfgjHT+EUKJ/znN7xcWIG2zNRFm9lnyPBW2wTiQpO9gWxlsJM2hSM90YrgEhviey w1WQ== X-Gm-Message-State: APzg51B2lJ8gw8xS2njBXwIpGXP6HagWoq4SX5hL5dxiW6MjraS0w0AS cUrpNC8vyLczP0jvBcsEq6uh4vdSX5JxJA== X-Received: by 2002:a24:1ad2:: with SMTP id 201-v6mr20493825iti.18.1537339948813; Tue, 18 Sep 2018 23:52:28 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:28 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 5/9] x86: jump_label: switch to jump_entry accessors Date: Tue, 18 Sep 2018 23:51:40 -0700 Message-Id: <20180919065144.25010-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation of switching x86 to use place-relative references for the code, target and key members of struct jump_entry, replace direct references to the struct members with invocations of the new accessors. This will allow us to make the switch by modifying the accessors only. This incorporates a cleanup of __jump_label_transform() proposed by Peter. Cc: Peter Zijlstra Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/jump_label.c | 62 ++++++++------------ 1 file changed, 25 insertions(+), 37 deletions(-) -- 2.17.1 diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index eeea935e9bb5..aac0c1f7e354 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -42,55 +42,40 @@ static void __ref __jump_label_transform(struct jump_entry *entry, void *(*poker)(void *, const void *, size_t), int init) { - union jump_code_union code; + union jump_code_union jmp; const unsigned char default_nop[] = { STATIC_KEY_INIT_NOP }; const unsigned char *ideal_nop = ideal_nops[NOP_ATOMIC5]; + const void *expect, *code; + int line; + + jmp.jump = 0xe9; + jmp.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); if (early_boot_irqs_disabled) poker = text_poke_early; if (type == JUMP_LABEL_JMP) { if (init) { - /* - * Jump label is enabled for the first time. - * So we expect a default_nop... - */ - if (unlikely(memcmp((void *)entry->code, default_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + expect = default_nop; line = __LINE__; } else { - /* - * ...otherwise expect an ideal_nop. Otherwise - * something went horribly wrong. - */ - if (unlikely(memcmp((void *)entry->code, ideal_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + expect = ideal_nop; line = __LINE__; } - code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); + code = &jmp.code; } else { - /* - * We are disabling this jump label. If it is not what - * we think it is, then something must have gone wrong. - * If this is the first initialization call, then we - * are converting the default nop to the ideal nop. - */ if (init) { - if (unlikely(memcmp((void *)entry->code, default_nop, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + expect = default_nop; line = __LINE__; } else { - code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); - if (unlikely(memcmp((void *)entry->code, &code, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + expect = &jmp.code; line = __LINE__; } - memcpy(&code, ideal_nops[NOP_ATOMIC5], JUMP_LABEL_NOP_SIZE); + + code = ideal_nop; } + if (memcmp((void *)jump_entry_code(entry), expect, JUMP_LABEL_NOP_SIZE)) + bug_at((void *)jump_entry_code(entry), line); + /* * Make text_poke_bp() a default fallback poker. * @@ -99,11 +84,14 @@ static void __ref __jump_label_transform(struct jump_entry *entry, * always nop being the 'currently valid' instruction * */ - if (poker) - (*poker)((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE); - else - text_poke_bp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE, - (void *)entry->code + JUMP_LABEL_NOP_SIZE); + if (poker) { + (*poker)((void *)jump_entry_code(entry), code, + JUMP_LABEL_NOP_SIZE); + return; + } + + text_poke_bp((void *)jump_entry_code(entry), code, JUMP_LABEL_NOP_SIZE, + (void *)jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); } void arch_jump_label_transform(struct jump_entry *entry, From patchwork Wed Sep 19 06:51:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147025 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp343845ljw; Tue, 18 Sep 2018 23:52:37 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZ4ytiXnUZAz/Nn58G9fjmL2JCpKLebgOirs/ZyyPUoy3cLafP6S2sKRebrJGzFHaR0Yo1S X-Received: by 2002:a17:902:e004:: with SMTP id ca4-v6mr32106873plb.252.1537339957147; Tue, 18 Sep 2018 23:52:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339957; cv=none; d=google.com; s=arc-20160816; b=GT3HefypF0C9MC8a1pYqflKFNqNkl/AiW8oLHbT3occJNQqdKXYCury+NzgzvIMqBC znQeMjdJEAsX/YKYQ6xf9wtiumMWGcIpTph4KASqgbF92p7JQ6FoFNOhPnIuxj7vVDBv VHoJB4hE2UhTD4aXVj7r1Xcrtk7xchvdv2Y9AtZ7GrBxDlAVh0rQNJLvivwlg73RkJ8z ZNXNEANz8uB+X9GYi/K8OWtpIjF98aD908q44lsvSR0YyGhTi7Ila39qQakLWGOh8UOf xvUmpBzD5tnF/nKksrS2yn0qYA2Tp78CRMtllKV0NTcnIHnmnDIiaZWRlC9HvzGNlID6 NqJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=XGRY1T3h+gs2S6eNeT9ZaVSn/BnvjE5DrD6FeDRoeQ4=; b=IG65i4FVs7PwEJmWc0Ldtd43VPkN6TiYboDp+cYE/sZCgR6K0oEpzP5Fq69Av4JN1Z mxWAygAIIFKdORp7UavLNiUTUTo8GrwNMepgJOhK6AZrK4L9REUX/k132qXnpXOsEKJY qCRKntojkd6AdWN+3bVRip2P5Hsj8h+IYidUGV24R9be+simuqGmAuCTlzA+ndc+hdTe 2knC1UQE1m7c4zI4tPNS8TRmnnHsmxfGfNE7Mt4zdFrGuKWrYndp0SZE7or6NtE0EoYV 1U31nZAIA7Le2DcoTw5EW4SI0NQs7p9fv3lAMghLza4BV8j686bGOZyb2i5k+Sq/M7UF dS2Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="NB7mskp/"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t64-v6si19375292pgd.176.2018.09.18.23.52.36; Tue, 18 Sep 2018 23:52:37 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="NB7mskp/"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731082AbeISM3E (ORCPT + 32 others); Wed, 19 Sep 2018 08:29:04 -0400 Received: from mail-io1-f68.google.com ([209.85.166.68]:45074 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731062AbeISM3D (ORCPT ); Wed, 19 Sep 2018 08:29:03 -0400 Received: by mail-io1-f68.google.com with SMTP id e12-v6so3584993iok.12 for ; Tue, 18 Sep 2018 23:52:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XGRY1T3h+gs2S6eNeT9ZaVSn/BnvjE5DrD6FeDRoeQ4=; b=NB7mskp/qv9FTKKiv/NIH3QElhQgT74KLCGGPtWIXEbKuGh+Mn7QgIY8P0rP2earA6 HGPls0WytLUlXWc6UX60x6rMJgjdziPfxccBg+oM7l/vTE9ydrroiYYiRC+mZAFx3Pl3 mkSPKyvp48KYUfPkN/+fCiXjZWlfxNVTGFUGg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XGRY1T3h+gs2S6eNeT9ZaVSn/BnvjE5DrD6FeDRoeQ4=; b=nhEVl0+Imvl2ss0frVxWykOEsNL4mm9DszTvNqXzLG6SBPShmGDhhsD9A48WyrLCvP FO6aHI890kojMK8tSQqdV++u/4qVHXffWBMngMPWxB9txa/lq1H08iukh9sXtsBDGr+7 Ly8MdN/d4D60jd6BwILfqxheprLKfNToGtBGSjFsgYvMZx1Jc4SEiDsz6/GZ81leO0Rf GIMyThYgtvEujrdzXAPybkbq4bE9gmaBVDB/Qu3AEkzTtsdQveFOqm5Mew19e7R577Rj 9NNvyf44bdp6nD33kExpn87oP6yjHTD/X72aHU+BeWXjvqVPu8WwwYzJgN0MtUcDMVXH Td1A== X-Gm-Message-State: APzg51Bj7oxrM3GcAzTlL69SySxhQ6Ts76SGBqyL+r0SA5ffjWiigQgb LxeDDSVKkHmMch7k/qpFZ+A/QLrc5g82Fw== X-Received: by 2002:a5e:db07:: with SMTP id q7-v6mr28850533iop.81.1537339953658; Tue, 18 Sep 2018 23:52:33 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:33 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 6/9] x86/kernel: jump_table: use relative references Date: Tue, 18 Sep 2018 23:51:41 -0700 Message-Id: <20180919065144.25010-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Similar to the arm64 case, 64-bit x86 can benefit from using relative references rather than absolute ones when emitting struct jump_entry instances. Not only does this reduce the memory footprint of the entries themselves by 33%, it also removes the need for carrying relocation metadata on relocatable builds (i.e., for KASLR) which saves a fair chunk of .init space as well (although the savings are not as dramatic as on arm64) Signed-off-by: Ard Biesheuvel --- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 24 +++++++------------- tools/objtool/special.c | 4 ++-- 3 files changed, 11 insertions(+), 18 deletions(-) -- 2.17.1 diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1a0be022f91d..603f374f871d 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -119,6 +119,7 @@ config X86 select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if X86_64 || X86_PAE select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 8c0de4282659..21efc9d07ed9 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -37,7 +37,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" ".pushsection __jump_table, \"aw\" \n\t" _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".long 1b - ., %l[l_yes] - . \n\t" + _ASM_PTR "%c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -53,7 +54,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool "2:\n\t" ".pushsection __jump_table, \"aw\" \n\t" _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".long 1b - ., %l[l_yes] - . \n\t" + _ASM_PTR "%c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -62,18 +64,6 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool return true; } -#ifdef CONFIG_X86_64 -typedef u64 jump_label_t; -#else -typedef u32 jump_label_t; -#endif - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #else /* __ASSEMBLY__ */ .macro STATIC_JUMP_IF_TRUE target, key, def @@ -88,7 +78,8 @@ struct jump_entry { .endif .pushsection __jump_table, "aw" _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + .long .Lstatic_jump_\@ - ., \target - . + _ASM_PTR \key - . .popsection .endm @@ -104,7 +95,8 @@ struct jump_entry { .endif .pushsection __jump_table, "aw" _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 + .long .Lstatic_jump_\@ - ., \target - . + _ASM_PTR \key + 1 - . .popsection .endm diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 84f001d52322..50af4e1274b3 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -30,9 +30,9 @@ #define EX_ORIG_OFFSET 0 #define EX_NEW_OFFSET 4 -#define JUMP_ENTRY_SIZE 24 +#define JUMP_ENTRY_SIZE 16 #define JUMP_ORIG_OFFSET 0 -#define JUMP_NEW_OFFSET 8 +#define JUMP_NEW_OFFSET 4 #define ALT_ENTRY_SIZE 13 #define ALT_ORIG_OFFSET 0 From patchwork Wed Sep 19 06:51:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147026 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp343913ljw; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) X-Google-Smtp-Source: ANB0VdY+FnDDI1KpgFQQS17x2k0y6qVotWr5clyXm88cmgFvOB6HCIZSrqBWLkjQE8qtblFTivO7 X-Received: by 2002:a17:902:20c6:: with SMTP id v6-v6mr33427272plg.228.1537339963506; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339963; cv=none; d=google.com; s=arc-20160816; b=dph0zf510dCo5hHKkBTy1jHTtG4AaPC7AsBhv1dSV2DYcC9Mr9ybZVqTKmg/negrVK 3sUUb00R/vyM8huuKi8Zf0tNoaU45U4U4U1bsN8247m7UzrnLrPfRNMLq5/Z7hpIZS8o 8NtPufAy/y907HTh+03OIzVehVuUhhf6Z4XK7disMVMfq2dNyWBNYAPubD8AEhSraQzd UUHDQrF2B3QXyDAYTQ4DCH6Io2QFmr7Hretn1Tk9q2cfElA/hvsAPHTN/1dzpk9r6zBj QTsaYEzSw9nmXievSq3W53EufjcCXEIHGm2HrcQ+I4cdVLLw+2psy1HYb2O4DWSP4trx Ibzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=Je2400ni9A5s80feHs0HBjY1gZ6UX9x0bE0fIx90rAE=; b=PZENtDR23ouCiDHVgzjJ1TZK+bX64TTSzFNDnxbJiaHM3e62X+W+CIZ9p7jUgVSHvy Z7W0z5f0nXjUAzxfYcmxZ49F2mVzW7xQ+RIVk5QqC8IDr1pNQrprjpjZ9R/lUkIyJ0wr 1kfniCBua4fqtL3bP0J45Rj7uZOCge4FM/h4mNY+7dI+FZPqxMSYx++qdsPgw6/shZHt 523qj0COFUYb+ez2ZK0Kb9jZXU4o1wpSoJKeGj+0k3efBZAwvxbKBnoONrR/v9Wr2afI vArG0ee5XSvoExGIwZaf2x69yGyr7atIQXBlu8y+l7w1Hu4EXONekwi/+89+6+1BXsuI Oc7Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ePc7YC7A; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x6-v6si19632022pge.100.2018.09.18.23.52.43; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ePc7YC7A; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731121AbeISM3J (ORCPT + 32 others); Wed, 19 Sep 2018 08:29:09 -0400 Received: from mail-it0-f66.google.com ([209.85.214.66]:51180 "EHLO mail-it0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731086AbeISM3I (ORCPT ); Wed, 19 Sep 2018 08:29:08 -0400 Received: by mail-it0-f66.google.com with SMTP id j81-v6so6864116ite.0 for ; Tue, 18 Sep 2018 23:52:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Je2400ni9A5s80feHs0HBjY1gZ6UX9x0bE0fIx90rAE=; b=ePc7YC7AmNfXSr4FpXbzsfuPGom2b9uu9QTkOtVTTpvbQnmlm+Ab5Fda4DSvDkiNSW 2n930+omZG2krqTyIvZHbqPMRfYLPEPCf7CqbnlVZak11lPxt0+JZGVX0RaAkWHUW1Qg 5Yzihq5i7+V3yXSxUwy87y990PW7TLphcsVXA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Je2400ni9A5s80feHs0HBjY1gZ6UX9x0bE0fIx90rAE=; b=KBZ/KOps0zmg9KeEhtNB24GjbwIP+8hojRzxb5cuBmxmuIx/LFw9ckycBadMPR3HQ7 2ly2lGxuzA2PiGiv3fSaf1v/xS69MgZoCZDVaBQEhNTuafbwHQdbOMAHa5MCVQ8182+t TSM9XHv+KZeT6OnT93oCJeguzhQki3sGGo1UtCM6c1UE20Q8w5mTC1FacjEMCHzfLWeI ZcadGw0yNc8izVRUQeElF5ZV+zsH0ZskPg+W5AV5u8uWgIJfRNT8AjTpbSMs/HZwBrJq HRmuQhFSOdzswKvQ8NK1UveL9JYKbb+YRHg2RFxeS9mTkHpc4JgkoaOQV152lNRjzCdi yttQ== X-Gm-Message-State: APzg51CIbMXXgm110npmvZvuXCe7zjQH6cJAPnl+F5liquWXv6UGYF8Z Zw+0J0dYkvZ0RuGFFZvVfyU19Q8sD8rDFQ== X-Received: by 2002:a24:6401:: with SMTP id t1-v6mr21062079itc.135.1537339958595; Tue, 18 Sep 2018 23:52:38 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:37 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 7/9] jump_label: annotate entries that operate on __init code earlier Date: Tue, 18 Sep 2018 23:51:42 -0700 Message-Id: <20180919065144.25010-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Jump table entries are mostly read-only, with the exception of the init and module loader code that defuses entries that point into init code when the code being referred to is freed. For robustness, it would be better to move these entries into the ro_after_init section, but clearing the 'code' member of each jump table entry referring to init code at module load time races with the module_enable_ro() call that remaps the ro_after_init section read only, so we'd like to do it earlier. So given that whether such an entry refers to init code can be decided much earlier, we can pull this check forward. Since we may still need the code entry at this point, let's switch to setting a low bit in the 'key' member just like we do to annotate the default state of a jump table entry. Reviewed-by: Kees Cook Signed-off-by: Ard Biesheuvel --- include/linux/jump_label.h | 11 ++--- init/main.c | 1 - kernel/jump_label.c | 48 ++++++-------------- 3 files changed, 18 insertions(+), 42 deletions(-) -- 2.17.1 diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 2768a925bafa..5df6a621e464 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -141,7 +141,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - long offset = entry->key & ~1L; + long offset = entry->key & ~3L; return (struct static_key *)((unsigned long)&entry->key + offset); } @@ -160,7 +160,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + return (struct static_key *)((unsigned long)entry->key & ~3UL); } #endif @@ -172,12 +172,12 @@ static inline bool jump_entry_is_branch(const struct jump_entry *entry) static inline bool jump_entry_is_init(const struct jump_entry *entry) { - return entry->code == 0; + return (unsigned long)entry->key & 2UL; } static inline void jump_entry_set_init(struct jump_entry *entry) { - entry->code = 0; + entry->key |= 2; } #endif @@ -213,7 +213,6 @@ extern struct jump_entry __start___jump_table[]; extern struct jump_entry __stop___jump_table[]; extern void jump_label_init(void); -extern void jump_label_invalidate_initmem(void); extern void jump_label_lock(void); extern void jump_label_unlock(void); extern void arch_jump_label_transform(struct jump_entry *entry, @@ -261,8 +260,6 @@ static __always_inline void jump_label_init(void) static_key_initialized = true; } -static inline void jump_label_invalidate_initmem(void) {} - static __always_inline bool static_key_false(struct static_key *key) { if (unlikely(static_key_count(key) > 0)) diff --git a/init/main.c b/init/main.c index 18f8f0140fa0..a664246450d1 100644 --- a/init/main.c +++ b/init/main.c @@ -1064,7 +1064,6 @@ static int __ref kernel_init(void *unused) /* need to finish all async __init code before freeing the memory */ async_synchronize_full(); ftrace_free_init_mem(); - jump_label_invalidate_initmem(); free_initmem(); mark_readonly(); diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 898a1d0c38dc..e8cf3ff3149c 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -373,14 +373,15 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) static void __jump_label_update(struct static_key *key, struct jump_entry *entry, - struct jump_entry *stop) + struct jump_entry *stop, + bool init) { for (; (entry < stop) && (jump_entry_key(entry) == key); entry++) { /* * An entry->code of 0 indicates an entry which has been * disabled because it was in an init text area. */ - if (!jump_entry_is_init(entry)) { + if (init || !jump_entry_is_init(entry)) { if (kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); else @@ -420,6 +421,9 @@ void __init jump_label_init(void) if (jump_label_type(iter) == JUMP_LABEL_NOP) arch_jump_label_transform_static(iter, JUMP_LABEL_NOP); + if (init_section_contains((void *)jump_entry_code(iter), 1)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -432,19 +436,6 @@ void __init jump_label_init(void) cpus_read_unlock(); } -/* Disable any jump label entries in __init/__exit code */ -void __init jump_label_invalidate_initmem(void) -{ - struct jump_entry *iter_start = __start___jump_table; - struct jump_entry *iter_stop = __stop___jump_table; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (init_section_contains((void *)jump_entry_code(iter), 1)) - jump_entry_set_init(iter); - } -} - #ifdef CONFIG_MODULES static enum jump_label_type jump_label_init_type(struct jump_entry *entry) @@ -524,7 +515,8 @@ static void __jump_label_mod_update(struct static_key *key) stop = __stop___jump_table; else stop = m->jump_entries + m->num_jump_entries; - __jump_label_update(key, mod->entries, stop); + __jump_label_update(key, mod->entries, stop, + m->state == MODULE_STATE_COMING); } } @@ -570,6 +562,9 @@ static int jump_label_add_module(struct module *mod) for (iter = iter_start; iter < iter_stop; iter++) { struct static_key *iterk; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -605,7 +600,7 @@ static int jump_label_add_module(struct module *mod) /* Only update if we've changed from our initial state */ if (jump_label_type(iter) != jump_label_init_type(iter)) - __jump_label_update(key, iter, iter_stop); + __jump_label_update(key, iter, iter_stop, true); } return 0; @@ -661,19 +656,6 @@ static void jump_label_del_module(struct module *mod) } } -/* Disable any jump label entries in module init code */ -static void jump_label_invalidate_module_init(struct module *mod) -{ - struct jump_entry *iter_start = mod->jump_entries; - struct jump_entry *iter_stop = iter_start + mod->num_jump_entries; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(jump_entry_code(iter), mod)) - jump_entry_set_init(iter); - } -} - static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data) @@ -695,9 +677,6 @@ jump_label_module_notify(struct notifier_block *self, unsigned long val, case MODULE_STATE_GOING: jump_label_del_module(mod); break; - case MODULE_STATE_LIVE: - jump_label_invalidate_module_init(mod); - break; } jump_label_unlock(); @@ -767,7 +746,8 @@ static void jump_label_update(struct static_key *key) entry = static_key_entries(key); /* if there are no users, entry can be NULL */ if (entry) - __jump_label_update(key, entry, stop); + __jump_label_update(key, entry, stop, + system_state < SYSTEM_RUNNING); } #ifdef CONFIG_STATIC_KEYS_SELFTEST From patchwork Wed Sep 19 06:51:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147027 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp343941ljw; Tue, 18 Sep 2018 23:52:46 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZfZImhF0HHpwitxVnosEj5OAPDLtCgXfZ5+fe9rreOgm5TDAZ47aGDz/ZpHyXil8SKDr20 X-Received: by 2002:a62:586:: with SMTP id 128-v6mr34671587pff.80.1537339966217; Tue, 18 Sep 2018 23:52:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339966; cv=none; d=google.com; s=arc-20160816; b=ahtJcCNHvN4aSPfy6eS5wKucnnSpH9DyBxry0k3UrvE0V1A3XvPuTIa87/cIerGoqI PaAQlwG3hk4L5r3LyoTOFCIQB1qYfDAdLarV6M28d48tdndqQKuDSZWJbQygAad+CVOn MZyL+W7Jdmn+bcJ36CMjvzH1P99Yy9JuZsRAYD9YD/nXr+1RwYA1/QI/2HaSueNRg5n0 XoW0igOohTkugF9oRwXRNQ5tO2cCImdsRY8KMWLIOI9f8pJ0Zft5WE25Z82BM1n5Vfkb cx9uc7EqzzGT+GxYJZwA9lFUrEdnvf6MeyFqUOPPbn44pAr48HSWKIsecI55qYsAoaWr PJPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=6WcczBvykg2Z58bAiHqwLTs7BS22B+N9kwtSqthViSE=; b=JJEghgtEYOcJNwnUt3SmPBfW7fsAsRrnQsfsKAj6uu8ezLZYEEgrwgLXJzwLShNpRN 7P6p1mXmXHfug5WexrKvud7GC69G0ttKYoKxOmlzl8/luHK6CrWZxwNPIhbNKGncmF2x QZosRre0TRz1S1S14fD2w4TQq3STUgR7sKTYCWnvCPtHKIo0J9CsLtRKVnHF9yhhKHnm w6IfX2/bgRVLG4pZjTyIrloT3/nZdxeLHLaPi0EThaeBl9V2KHfBpdVAGGO+4vhb4OMU 4/6I8kseajH02sY0FBNdIiLLFeZgL5oLOb7g55PEZnEK/ri2DM6L1jFfjJvwxFfZJRbN XqoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Focb2vMv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x6-v6si19632022pge.100.2018.09.18.23.52.45; Tue, 18 Sep 2018 23:52:46 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Focb2vMv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731139AbeISM3N (ORCPT + 32 others); Wed, 19 Sep 2018 08:29:13 -0400 Received: from mail-it0-f65.google.com ([209.85.214.65]:51186 "EHLO mail-it0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731086AbeISM3N (ORCPT ); Wed, 19 Sep 2018 08:29:13 -0400 Received: by mail-it0-f65.google.com with SMTP id j81-v6so6864335ite.0 for ; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6WcczBvykg2Z58bAiHqwLTs7BS22B+N9kwtSqthViSE=; b=Focb2vMv0Nwcyy8JpgHJ+eawzudhCpL77NjYEXyCv5owfpRIMB9U+XM+EUUsK4ZAgM FzzcDvaN97C+D4oF8Abvfa1yIacasMqfez3ah7v1BBLBqkVVyQMiR4oCyqB/pi8iYLns qqs7M3GyoO3OUXvzfKgFJjlZ98/QYCN5z09n0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6WcczBvykg2Z58bAiHqwLTs7BS22B+N9kwtSqthViSE=; b=Sv+8DEOUIv8GoGfRZuu0qWT0wzNTZJGNRX9kVsRaANo8zJHNmTetKpE64Lv86N/Mk/ AMMz3RH2IA2x6gv4VYzAkYpATSDtL9GOq0l455vaNGDPyDt7LRFnifdnEvRTdoXTrT/q SbV6HxWuIT3HH9kcBfZER28ER60vLVVUv/i7VvJ+KHJligVxEThnMNWteNKUgfaPG4Vq gkF4FqAnLkuiJy+sORYqrmsmEO6XMyYOkShbQHkfr/vYbrsn3hFEWgP7DdNQgRZxgi4W wpLiHCGRFgrv23ez14aUqrBD943U7n54uFjJndwafjlpIM0dhIqE7CeEUsXM5sGtSfBw FJlw== X-Gm-Message-State: APzg51Cd1t2ixLfd8vsKkCDKjFuRHRfgAdDmKtpXmBcPrd9hmkHjMOLY YFI2HkyLiNt+j6SC6qCCbEweEyYiIq8KBw== X-Received: by 2002:a24:1355:: with SMTP id 82-v6mr20940938itz.74.1537339963109; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:42 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 8/9] jump_table: move entries into ro_after_init region Date: Tue, 18 Sep 2018 23:51:43 -0700 Message-Id: <20180919065144.25010-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The __jump_table sections emitted into the core kernel and into each module consist of statically initialized references into other parts of the code, and with the exception of entries that point into init code, which are defused at post-init time, these data structures are never modified. So let's move them into the ro_after_init section, to prevent them from being corrupted inadvertently by buggy code, or deliberately by an attacker. Reviewed-by: Kees Cook Acked-by: Jessica Yu Signed-off-by: Ard Biesheuvel --- arch/s390/kernel/vmlinux.lds.S | 1 + include/asm-generic/vmlinux.lds.h | 11 +++++++---- kernel/module.c | 9 +++++++++ 3 files changed, 17 insertions(+), 4 deletions(-) -- 2.17.1 diff --git a/arch/s390/kernel/vmlinux.lds.S b/arch/s390/kernel/vmlinux.lds.S index b43f8d33a369..4042bbf3f9ad 100644 --- a/arch/s390/kernel/vmlinux.lds.S +++ b/arch/s390/kernel/vmlinux.lds.S @@ -66,6 +66,7 @@ SECTIONS *(.data..ro_after_init) } EXCEPTION_TABLE(16) + JUMP_TABLE_DATA . = ALIGN(PAGE_SIZE); __end_ro_after_init = .; diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 7b75ff6e2fce..f09ee3c544bc 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -253,10 +253,6 @@ STRUCT_ALIGN(); \ *(__tracepoints) \ /* implement dynamic printk debug */ \ - . = ALIGN(8); \ - __start___jump_table = .; \ - KEEP(*(__jump_table)) \ - __stop___jump_table = .; \ . = ALIGN(8); \ __start___verbose = .; \ KEEP(*(__verbose)) \ @@ -300,6 +296,12 @@ . = __start_init_task + THREAD_SIZE; \ __end_init_task = .; +#define JUMP_TABLE_DATA \ + . = ALIGN(8); \ + __start___jump_table = .; \ + KEEP(*(__jump_table)) \ + __stop___jump_table = .; + /* * Allow architectures to handle ro_after_init data on their * own by defining an empty RO_AFTER_INIT_DATA. @@ -308,6 +310,7 @@ #define RO_AFTER_INIT_DATA \ __start_ro_after_init = .; \ *(.data..ro_after_init) \ + JUMP_TABLE_DATA \ __end_ro_after_init = .; #endif diff --git a/kernel/module.c b/kernel/module.c index 6746c85511fe..49a405891587 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -3315,6 +3315,15 @@ static struct module *layout_and_allocate(struct load_info *info, int flags) * Note: ro_after_init sections also have SHF_{WRITE,ALLOC} set. */ ndx = find_sec(info, ".data..ro_after_init"); + if (ndx) + info->sechdrs[ndx].sh_flags |= SHF_RO_AFTER_INIT; + /* + * Mark the __jump_table section as ro_after_init as well: these data + * structures are never modified, with the exception of entries that + * refer to code in the __init section, which are annotated as such + * at module load time. + */ + ndx = find_sec(info, "__jump_table"); if (ndx) info->sechdrs[ndx].sh_flags |= SHF_RO_AFTER_INIT; From patchwork Wed Sep 19 06:51:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 147028 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp344010ljw; Tue, 18 Sep 2018 23:52:50 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbvPg66kVcuKy7VdtFz8NmKX/j7cfhqd93YeVxv43LQ/pCaPGPoDOz/XI5xVI2ySkWZzl6W X-Received: by 2002:a63:2a0b:: with SMTP id q11-v6mr27875892pgq.36.1537339970336; Tue, 18 Sep 2018 23:52:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537339970; cv=none; d=google.com; s=arc-20160816; b=qdhh+Z6fYMBVerp96OPu8h2LZAoJgR08YYiKiHFru7o8ouFgyqupjsG1bND4Oc+ZzZ Kwg3zKYBMDTUxJagUov3zu5QVVnMbfvhFfHbQgRntbiHWQsdKzS2huIPnqGcQCUnOqer dXqN/jRtdwnqy9oTymNcHoBOPzBnrcjcUfj0GdlP7aBR6rhBISsw/y32lT3eZw2iEQzN vfJsyUqaWCJXj7XDzKu2SAHutx/0w32WIK94HB13dFV9WyLlBIWyncfsoJJ3s4G0DUUd ioPcqYMKJ76ACICTHTbguiO5aE3km1pFsnOkdKH6nKNQzUXTCRRy5kl+Qblpw5sxt1WH Gysg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=iA34PvoXnv/5jnydRzA1GyetmcM2pytY3HL8SIbi1DA=; b=sfNGtQ0AArO+dT4B/eKcpdsMGsfAmW5xlBPd8wSsOG7Qzi8SBokolypgpUCX3FEJ+P fMTH6iEVmebIcbZBx8kvZfc40tf2lp3qNrP6jk5o3jZHnvJUaytNzCF/K+KFIdpFyPtD PgyWjhzHJRT9KSXtQfC33/NUZ0t2Qp3N7WxlhmEnRY9WsEZxnOZvWqvxj1Mom1L/oTBc 5TbnzTD0in7kSSMWnnP79AJhS8eDnx1JUehiBK5OFsu6Dg+y51sna4NC7jJwTXPeGmCC x24P8ME+Sk+wj56ZYGDBw7zcxGjekFFtzEsqwhwowb0BmVUUbodOZTNI+XOT/+kTULOK PDzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hX3BTpM6; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p3-v6si20138736pld.329.2018.09.18.23.52.50; Tue, 18 Sep 2018 23:52:50 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hX3BTpM6; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731157AbeISM3R (ORCPT + 32 others); Wed, 19 Sep 2018 08:29:17 -0400 Received: from mail-it0-f66.google.com ([209.85.214.66]:53974 "EHLO mail-it0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731142AbeISM3Q (ORCPT ); Wed, 19 Sep 2018 08:29:16 -0400 Received: by mail-it0-f66.google.com with SMTP id p79-v6so6840240itp.3 for ; Tue, 18 Sep 2018 23:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iA34PvoXnv/5jnydRzA1GyetmcM2pytY3HL8SIbi1DA=; b=hX3BTpM6OFVDYHhbaduRyiEvKVa1h5kwWbYpjAoXCspqeB/8w7+LnC7RvhgfoKauDh K/qeYeiNjLC3g0sGJS48hHaypqf2IsTsjQVJVmwnZUT1yHVeaa9UqpQsRT936/9JiXi2 dZmpGMvJAVhcNCShHhhW5p2Y0X6JZN8AwlEvA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iA34PvoXnv/5jnydRzA1GyetmcM2pytY3HL8SIbi1DA=; b=f8yyqeeYKsk0RMmS6I2YwgVs8ffLW746d7DXyLYqmKyWsM4vpR9CYZZc9/Y5etG+Kh VUqxCDEAOoMdFH2CB3NgBvVumXmiZMmNpL7kwAwuUSMA4x+Ag/wxfq5SlYPbyyVsuIdM hUHZlQy4C3/inWzNoy+9OqR3STSTdETATDIHtbZgk+IwHE7JKfQkNp/Fmj0/x07kmehx lJQoatDYho9Tv0a7BGZl84sgOq72DLizd9+ao1wmGxPfEYda8qWlVd4uGh0LPL9NqwNz j4xKVcGaCVOL1CfvFqTmd2O/5OfrjEvqko4n/UvYL2TLjIDiPzoKTI+6ZqmzkItVG3/h 7iJA== X-Gm-Message-State: APzg51CYVLE+La4WgtntG5gP0lfOo1z1EYznJ9nMenWR7kcuvsdPLPch 37dOI2ehW5wA1E5d11nmj4+3xTKwDj1iFQ== X-Received: by 2002:a24:cf06:: with SMTP id y6-v6mr19881811itf.5.1537339966590; Tue, 18 Sep 2018 23:52:46 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:45 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, Ard Biesheuvel , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Thomas Gleixner , Catalin Marinas , Ingo Molnar , Steven Rostedt , Martin Schwidefsky , Jessica Yu , Peter Zijlstra Subject: [PATCH v3 9/9] s390/jump_label: switch to relative references Date: Tue, 18 Sep 2018 23:51:44 -0700 Message-Id: <20180919065144.25010-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Heiko Carstens Enable support for relative references in jump_label entries. Signed-off-by: Heiko Carstens Signed-off-by: Ard Biesheuvel --- arch/s390/Kconfig | 1 + arch/s390/include/asm/jump_label.h | 40 ++++++++------------ arch/s390/kernel/jump_label.c | 11 +++--- 3 files changed, 23 insertions(+), 29 deletions(-) -- 2.17.1 diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 9a9c7a6fe925..12dc4c1af96c 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -120,6 +120,7 @@ config S390 select HAVE_ALIGNED_STRUCT_PAGE if SLUB select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select CPU_NO_EFFICIENT_FFS if !HAVE_MARCH_Z9_109_FEATURES select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_SOFT_DIRTY diff --git a/arch/s390/include/asm/jump_label.h b/arch/s390/include/asm/jump_label.h index 40f651292aa7..e2d3e6c43395 100644 --- a/arch/s390/include/asm/jump_label.h +++ b/arch/s390/include/asm/jump_label.h @@ -14,41 +14,33 @@ * We use a brcl 0,2 instruction for jump labels at compile time so it * can be easily distinguished from a hotpatch generated instruction. */ -static __always_inline bool arch_static_branch(struct static_key *key, bool branch) +static inline bool arch_static_branch(struct static_key *key, bool branch) { - asm_volatile_goto("0: brcl 0,"__stringify(JUMP_LABEL_NOP_OFFSET)"\n" - ".pushsection __jump_table, \"aw\"\n" - ".balign 8\n" - ".quad 0b, %l[label], %0\n" - ".popsection\n" - : : "X" (&((char *)key)[branch]) : : label); - + asm_volatile_goto("0: brcl 0,"__stringify(JUMP_LABEL_NOP_OFFSET)"\n" + ".pushsection __jump_table,\"aw\"\n" + ".balign 8\n" + ".long 0b-.,%l[label]-.\n" + ".quad %0-.\n" + ".popsection\n" + : : "X" (&((char *)key)[branch]) : : label); return false; label: return true; } -static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch) +static inline bool arch_static_branch_jump(struct static_key *key, bool branch) { - asm_volatile_goto("0: brcl 15, %l[label]\n" - ".pushsection __jump_table, \"aw\"\n" - ".balign 8\n" - ".quad 0b, %l[label], %0\n" - ".popsection\n" - : : "X" (&((char *)key)[branch]) : : label); - + asm_volatile_goto("0: brcl 15,%l[label]\n" + ".pushsection __jump_table,\"aw\"\n" + ".balign 8\n" + ".long 0b-.,%l[label]-.\n" + ".quad %0-.\n" + ".popsection\n" + : : "X" (&((char *)key)[branch]) : : label); return false; label: return true; } -typedef unsigned long jump_label_t; - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/s390/kernel/jump_label.c b/arch/s390/kernel/jump_label.c index 43f8430fb67d..50a1798604a8 100644 --- a/arch/s390/kernel/jump_label.c +++ b/arch/s390/kernel/jump_label.c @@ -33,13 +33,13 @@ static void jump_label_make_branch(struct jump_entry *entry, struct insn *insn) { /* brcl 15,offset */ insn->opcode = 0xc0f4; - insn->offset = (entry->target - entry->code) >> 1; + insn->offset = (jump_entry_target(entry) - jump_entry_code(entry)) >> 1; } static void jump_label_bug(struct jump_entry *entry, struct insn *expected, struct insn *new) { - unsigned char *ipc = (unsigned char *)entry->code; + unsigned char *ipc = (unsigned char *)jump_entry_code(entry); unsigned char *ipe = (unsigned char *)expected; unsigned char *ipn = (unsigned char *)new; @@ -59,6 +59,7 @@ static void __jump_label_transform(struct jump_entry *entry, enum jump_label_type type, int init) { + void *code = (void *)jump_entry_code(entry); struct insn old, new; if (type == JUMP_LABEL_JMP) { @@ -69,13 +70,13 @@ static void __jump_label_transform(struct jump_entry *entry, jump_label_make_nop(entry, &new); } if (init) { - if (memcmp((void *)entry->code, &orignop, sizeof(orignop))) + if (memcmp(code, &orignop, sizeof(orignop))) jump_label_bug(entry, &orignop, &new); } else { - if (memcmp((void *)entry->code, &old, sizeof(old))) + if (memcmp(code, &old, sizeof(old))) jump_label_bug(entry, &old, &new); } - s390_kernel_write((void *)entry->code, &new, sizeof(new)); + s390_kernel_write(code, &new, sizeof(new)); } static int __sm_arch_jump_label_transform(void *data)