From patchwork Wed Jun 27 16:06:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140361 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1069124ljj; Wed, 27 Jun 2018 09:06:16 -0700 (PDT) X-Google-Smtp-Source: ADUXVKI9IMqp7juYAZB/nhfwcqvFxopFAEsu6qaefUGdEqjmNQn4Xe0V/xcX7zJzshfIkHt0/axM X-Received: by 2002:a63:4b1f:: with SMTP id y31-v6mr5827007pga.14.1530115576049; Wed, 27 Jun 2018 09:06:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530115576; cv=none; d=google.com; s=arc-20160816; b=hRVLbf+SUoaSN33VE55hdS4LS7mklnw2/ZOaYy0hLLFUCtRQB6YNll3/3qvH6HLy/n wgihKCxuhoyrzPq5NKg+/BSB5SkOLUtpEBcIsDQtltuqPZp0N8ys33DC9jolYdWkxcth HtmW7hkagefVryVU6qkXNFaUJBpUa3nxtaT5AlrVBRwoKOO3/3xP0sS/g/+TGs1TbzAM Zjh2y8CQG7WypzDOKD/XEeprYDIwq8Qd1NZ60Ce5fmIx0AMkD8zZlq+kV/Hdl1qI0qtF bd4N1+ctJ8AqoT4xXhpzGfDmAfV7qoKQld74rqAPZKFs/hswCinvhDs236tUmMvgLh8+ 65ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=B883F5dmVQG/nVCIvlNlD4L/gAxRvgth0kf0bW+7srI=; b=Mu+x3ue3M6F30wBWP+ZvkGM05b8qqzagnXmAFXG9vHWMmBpfV9J+5Q5/heOiZqf4hl ttbkCcSLJ7APIrGK5dD/kkpGPbgKcY1OV80mruOOBGCYsQO9JV5+GiNUbmqmRfMe+nN5 8r3HMowFTIosg/vXmgdmB/1KAXTEITHxyViIS1W6Ujy9vwTjD/OiZN5BoNH740uWGEm3 jnBYXtcj7NaJXHayBWPpFE4WouzEj35pkMexYco4irqNDC8Un3AviOY9A8188kUWmEwD UbzIuyT9WYjAps/s8dQNeOvdc8FE1gE5ynF7uawpZh2sbWGHDow0dBOPX2Q4Gvw4gv8u pJPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EtvViJLK; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o3-v6si3752232pgc.381.2018.06.27.09.06.15; Wed, 27 Jun 2018 09:06:16 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EtvViJLK; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935063AbeF0QGN (ORCPT + 31 others); Wed, 27 Jun 2018 12:06:13 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:39275 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934909AbeF0QGK (ORCPT ); Wed, 27 Jun 2018 12:06:10 -0400 Received: by mail-wm0-f66.google.com with SMTP id p11-v6so6049881wmc.4 for ; Wed, 27 Jun 2018 09:06:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=B883F5dmVQG/nVCIvlNlD4L/gAxRvgth0kf0bW+7srI=; b=EtvViJLKEsaJc1tUpkX0Rq3AVPZIsvTnuvmrDdheEm3Cuzmo+BdVFP1GP+C72Vhw/n SCeEF4dtSkdT6YmgeP8veplRNpA8xMUnT5Tlqug5UeQRyKREdWG/heVBFAABnvcyR3/x llqh1uRcU4ZMXTp/TU1Hvqd22ggTAs9RFmXSA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=B883F5dmVQG/nVCIvlNlD4L/gAxRvgth0kf0bW+7srI=; b=U6YfIMZJD4bCUg4JrixzH3TNTYKXw5JPhaO/vYp8/PSQEuPchAr6CtjlICVAYU9fF/ lbrCuMyLzV1+OEFMg2M/RsH2IssSJlNXYgTS6HzWH/IJUL9kx8s2DHbT5VC9dnk2suOF K2/y4ve5Sg2ZEzlFflL9ENIdjc79YPToD0TG5elOvvsNUfsAshAwApB01Z6rDXiPMaaI AiN2qauPn/ter9k0P72IgPXrHJx1noKn62V1NZ+JACmUTQwyEWGs/tUJg78vhdPR9mfa sPD9uAgWhvVPBYX2GjpfHS5Q4QBHXEBrurYS5rTfn+S9tp3NBLIC1H8hVhNwaK9m/knw 1/Dg== X-Gm-Message-State: APt69E10b8J3uQhfgsIjLv2whomF3xjEgtqJw2jtoA1QWyl7/8sNmFq6 AFnnePeGGAsB+RMhYmXKFqcg1Ctm4zs= X-Received: by 2002:a1c:5a08:: with SMTP id o8-v6mr5247383wmb.61.1530115568787; Wed, 27 Jun 2018 09:06:08 -0700 (PDT) Received: from rev02.home ([2a01:cb1d:112:6f00:4d04:66e:dea8:5b23]) by smtp.gmail.com with ESMTPSA id s2-v6sm7055000wrn.75.2018.06.27.09.06.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Jun 2018 09:06:08 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Arnd Bergmann , Steven Rostedt , Peter Zijlstra Subject: [PATCH 1/5] kernel/jump_label: abstract jump_entry member accessors Date: Wed, 27 Jun 2018 18:06:00 +0200 Message-Id: <20180627160604.8154-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180627160604.8154-1-ard.biesheuvel@linaro.org> References: <20180627160604.8154-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation of allowing architectures to use relative references in jump_label entries [which can dramatically reduce the memory footprint], introduce abstractions for references to the 'code' and 'key' members of struct jump_entry. Signed-off-by: Ard Biesheuvel --- include/linux/jump_label.h | 36 ++++++++++++++++++++ kernel/jump_label.c | 36 ++++++++------------ 2 files changed, 50 insertions(+), 22 deletions(-) -- 2.11.0 diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index b46b541c67c4..86ec0652d3b1 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -119,6 +119,42 @@ struct static_key { #ifdef HAVE_JUMP_LABEL #include + +#ifndef __ASSEMBLY__ + +struct jump_entry; /* defined by the architecture */ + +static inline unsigned long jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline unsigned long jump_entry_target(const struct jump_entry *entry) +{ + return entry->target; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#endif #endif #ifndef __ASSEMBLY__ diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 01ebdf1f9f40..c3524c9b3004 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -38,10 +38,12 @@ static int jump_label_cmp(const void *a, const void *b) const struct jump_entry *jea = a; const struct jump_entry *jeb = b; - if (jea->key < jeb->key) + if ((unsigned long)jump_entry_key(jea) < + (unsigned long)jump_entry_key(jeb)) return -1; - if (jea->key > jeb->key) + if ((unsigned long)jump_entry_key(jea) > + (unsigned long)jump_entry_key(jeb)) return 1; return 0; @@ -261,8 +263,8 @@ EXPORT_SYMBOL_GPL(jump_label_rate_limit); static int addr_conflict(struct jump_entry *entry, void *start, void *end) { - if (entry->code <= (unsigned long)end && - entry->code + JUMP_LABEL_NOP_SIZE > (unsigned long)start) + if (jump_entry_code(entry) <= (unsigned long)end && + jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE > (unsigned long)start) return 1; return 0; @@ -321,16 +323,6 @@ static inline void static_key_set_linked(struct static_key *key) key->type |= JUMP_TYPE_LINKED; } -static inline struct static_key *jump_entry_key(struct jump_entry *entry) -{ - return (struct static_key *)((unsigned long)entry->key & ~1UL); -} - -static bool jump_entry_branch(struct jump_entry *entry) -{ - return (unsigned long)entry->key & 1UL; -} - /*** * A 'struct static_key' uses a union such that it either points directly * to a table of 'struct jump_entry' or to a linked list of modules which in @@ -355,7 +347,7 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool enabled = static_key_enabled(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return enabled ^ branch; @@ -370,8 +362,8 @@ static void __jump_label_update(struct static_key *key, * An entry->code of 0 indicates an entry which has been * disabled because it was in an init text area. */ - if (entry->code) { - if (kernel_text_address(entry->code)) + if (!jump_entry_is_module_init(entry)) { + if (kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); else WARN_ONCE(1, "can't patch jump_label at %pS", @@ -441,7 +433,7 @@ static enum jump_label_type jump_label_init_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool type = static_key_type(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return type ^ branch; @@ -565,7 +557,7 @@ static int jump_label_add_module(struct module *mod) continue; key = iterk; - if (within_module(iter->key, mod)) { + if (within_module((unsigned long)key, mod)) { static_key_set_entries(key, iter); continue; } @@ -615,7 +607,7 @@ static void jump_label_del_module(struct module *mod) key = jump_entry_key(iter); - if (within_module(iter->key, mod)) + if (within_module((unsigned long)key, mod)) continue; /* No memory during module load */ @@ -659,8 +651,8 @@ static void jump_label_invalidate_module_init(struct module *mod) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(iter->code, mod)) - iter->code = 0; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_module_init(iter); } } From patchwork Wed Jun 27 16:06:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140366 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1070849ljj; Wed, 27 Jun 2018 09:07:36 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKp5YQhOOvuKY7Hh+uVVPFq9+QrflWIlVNtjV0ywkpnUNEJP8F5jZpuZagCKv9a6Npc5rUn X-Received: by 2002:a65:5683:: with SMTP id v3-v6mr5606755pgs.176.1530115656745; Wed, 27 Jun 2018 09:07:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530115656; cv=none; d=google.com; s=arc-20160816; b=YTxx7Se7bNi1vyk3PPB54GpgfOBxDo0TEwvjt6LHERTSgmkGUmNXCF1IlY6IMzK5N2 ta5cqcsWNQ+eBApQGQXz52u/nUElLWSuEDRKqbR7Fic+RRpS25WZblTk7N05stdsGm+i vLTm4f7Q8iL01/wcRb9QG5/6IdKEcdM0GHRa5221iRpXFr+aRSLHdne5IjUkkHkqxINa BHEaL/ScSCGhM644G9YSSy+SSEgyHYHmet6HX7Tav8JRTuILp72AHqL5c7eS9bkLxnSt f1kUlHAPPqznRV8tEvPpW/yPwi7hXpofnYipV7VW2wKGrlKO64FKN8Rlg5+p/qbZ7eiq Opfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=4S8kZQAvHa84YpWVvxunUgaXp0tuBTi7qepgqm8/+QE=; b=bQlJWoCfGp7jtObY2a1vOdzM5TMm+EYyik0i1c6oEdT0WfEbG9ygvHkqruWMGJ1vtk 4Fa4q6XIzDmBJz578BfLwxLvsGLQs7NcoVYse8ewsMn2ZHwsHqzUUBHcpZ5jIUtCKnZD iOFvrQ4Xt8GxJYOpDHGJR+wmtJP9gmFumJcWY/UxbYmugudhQnZIuv33Bpti+JEDQycO /cSbsGpbdwUqNp6MrTgZjQT/4Er1v77dNGLVLDKuUqPE7vf/VAzSu+xl2TjIapcki/2a UmW9b+IAcJbpt9hAf/Wlbu2E8c0vZn9cEBuW0l8kUu9QmRMPhQdjksPy6KbIG0OOaIBE 4xqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=bNVV4EdS; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id c81-v6si4300879pfj.138.2018.06.27.09.07.36; Wed, 27 Jun 2018 09:07:36 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=bNVV4EdS; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935118AbeF0QHe (ORCPT + 31 others); Wed, 27 Jun 2018 12:07:34 -0400 Received: from mail-wm0-f68.google.com ([74.125.82.68]:51973 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934916AbeF0QGL (ORCPT ); Wed, 27 Jun 2018 12:06:11 -0400 Received: by mail-wm0-f68.google.com with SMTP id w137-v6so6477111wmw.1 for ; Wed, 27 Jun 2018 09:06:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=4S8kZQAvHa84YpWVvxunUgaXp0tuBTi7qepgqm8/+QE=; b=bNVV4EdSWbYIG3AcgXqoPrgzHv5IuSt1y79hMVSTqlv4GSE4DswGLrww8OA0uUBI4+ hh5hIMZh3Z4SrdgXjLmiRdIWEOb7HOj5fFV/VFoCLlL0JUQYEYCtPuvMJBX+JQBVx9/G tXDhyhXI8X7QnidEH3SaViXefT2ypauu6Asy8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=4S8kZQAvHa84YpWVvxunUgaXp0tuBTi7qepgqm8/+QE=; b=CNjNu8RTQN7s1LlKnevB4oZ2WBjl3rd1ISekPzCzmOqVreRYaWYJio/Fv6BmkX48y2 QACvsMy3hXjtXflQYqSO+91eSY1zQqGy30WiofdMMoToRrlCaUmapVFxl/586kamvJo/ nVUQPwrhBqSkCDmQyJpnyLv63+XLuvaUivpk5tm38jwxsS6LQ36bMYzbnI/IqLa8+g2V 2caT6sp/5CXukE1r3B6PucnwuqeqVO1PT1ybrgUltSi9r0kyM7HK4ab8KW2XYgVYxFu4 cQesd2IK7ibLNXJFaS+L8SlT025fzilRsjCnU4B4N26M6JuYkB7rKcmz21MpQjK2KVxZ pZWA== X-Gm-Message-State: APt69E21Dhq9o6KV1e3FXChPxmy5aUCzFV4/HN+7n7msBNcY8PX5lIbw g+Yku/jUNnrl2a10EJsw5nGzCjAb/rQ= X-Received: by 2002:a1c:8c55:: with SMTP id o82-v6mr5246007wmd.60.1530115570114; Wed, 27 Jun 2018 09:06:10 -0700 (PDT) Received: from rev02.home ([2a01:cb1d:112:6f00:4d04:66e:dea8:5b23]) by smtp.gmail.com with ESMTPSA id s2-v6sm7055000wrn.75.2018.06.27.09.06.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Jun 2018 09:06:09 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Arnd Bergmann , Steven Rostedt , Peter Zijlstra Subject: [PATCH 2/5] kernel/jump_label: implement generic support for relative references Date: Wed, 27 Jun 2018 18:06:01 +0200 Message-Id: <20180627160604.8154-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180627160604.8154-1-ard.biesheuvel@linaro.org> References: <20180627160604.8154-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org To reduce the size taken up by absolute references in jump label entries themselves and the associated relocation records in the .init segment, add support for emitting them as 32-bit relative references instead. Note that this requires some extra care in the sorting routine, given that the offsets change when entries are moved around in the jump_entry table. Signed-off-by: Ard Biesheuvel --- arch/Kconfig | 3 +++ include/linux/jump_label.h | 28 ++++++++++++++++++++ kernel/jump_label.c | 20 +++++++++++++- 3 files changed, 50 insertions(+), 1 deletion(-) -- 2.11.0 diff --git a/arch/Kconfig b/arch/Kconfig index 2b8b70820002..22fa3792626e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -348,6 +348,9 @@ config HAVE_PERF_USER_STACK_DUMP config HAVE_ARCH_JUMP_LABEL bool +config HAVE_ARCH_JUMP_LABEL_RELATIVE + bool + config HAVE_RCU_TABLE_FREE bool diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 86ec0652d3b1..aa203dffe72c 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -121,6 +121,32 @@ struct static_key { #include #ifndef __ASSEMBLY__ +#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE + +struct jump_entry { + int code; + int target; + int key; +}; + +static inline unsigned long jump_entry_code(const struct jump_entry *entry) +{ + return (unsigned long)&entry->code + entry->code; +} + +static inline unsigned long jump_entry_target(const struct jump_entry *entry) +{ + return (unsigned long)&entry->target + entry->target; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); +} + +#else struct jump_entry; /* defined by the architecture */ @@ -139,6 +165,8 @@ static inline struct static_key *jump_entry_key(const struct jump_entry *entry) return (struct static_key *)((unsigned long)entry->key & ~1UL); } +#endif + static inline bool jump_entry_is_branch(const struct jump_entry *entry) { return (unsigned long)entry->key & 1UL; diff --git a/kernel/jump_label.c b/kernel/jump_label.c index c3524c9b3004..285eff13ecd1 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -49,6 +49,22 @@ static int jump_label_cmp(const void *a, const void *b) return 0; } +static void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + static void jump_label_sort_entries(struct jump_entry *start, struct jump_entry *stop) { @@ -56,7 +72,9 @@ jump_label_sort_entries(struct jump_entry *start, struct jump_entry *stop) size = (((unsigned long)stop - (unsigned long)start) / sizeof(struct jump_entry)); - sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL); + sort(start, size, sizeof(struct jump_entry), jump_label_cmp, + IS_ENABLED(CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE) ? jump_label_swap + : NULL); } static void jump_label_update(struct static_key *key); From patchwork Wed Jun 27 16:06:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140364 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1069936ljj; Wed, 27 Jun 2018 09:06:53 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfZnDnM+EnxRBoLDYtDTV2j5378ToXp1Bb9sNAqRsLQ1D97TRMpbs3y0TTz9wyOoYaQDkOw X-Received: by 2002:a62:993:: with SMTP id 19-v6mr6484140pfj.105.1530115612839; Wed, 27 Jun 2018 09:06:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530115612; cv=none; d=google.com; s=arc-20160816; b=lDF+rF7zKRUQneMt5QoaQrLuGSyZ4a/MMpo/wgjaLlAO2ucAOE/v0q51xW7Be02yhR 9AmwKD4apF9OpGcADDoZRw3Z8ncJXOHDdStvufaUASx28ESSltespdGh9LGcoToDxgZ8 AzlblfbEWxC7Tkb10y5SPWosTyS2uYSq7wzRAwb2z208hxuVX/Fos+LcGVBR5HZ6EAEJ xBFRxPeSw2MTHWWX0FZufWsfK8tISmLkfgD5MmYIZx0bAaVb+ad/GJk88VvpLZKLISCr hRXOMaICGz1P3DP1inmrYlaO+pzjm0FNI05/7PTqdiS+8D7N37HpEcgkcYdlsFbZLb8B gLug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=bihw6AeKi0fURk5bIT2bV23QoQQq88ujufV07nqCrwI=; b=NmQ9G+SN8RuV4a4AB8yQGmhE7yzKFUsgXBCHLjN8LlrxdfqIFO96snqHFQgSiqfPXe nkRENv3LvitwTrzMTsCKNVEkgNn7OJGiflvc7tFGEH6bjE4zI85EyGQAN3I0eFTCKnXU i0doJH9kgeg8hMo8zTi+WK0Q4kgBoif5HZzoReJQq41YJWD2vcVkuwLFBw6Z6pQyOHC8 dq5camAyOwA858EbbU+qhkOZauRsycqBCgXekD6h8RJAWJNllGfC/X8WueVtvvYSmDZi G4BMWSivl+1z0ik4RZQUpM1ONnE47Z8YAEyPLsifkQexPakSYc2obUdxZuNa+lzeeIok 1rzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LonVhfcj; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x15-v6si3786635pgq.619.2018.06.27.09.06.52; Wed, 27 Jun 2018 09:06:52 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LonVhfcj; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935092AbeF0QGS (ORCPT + 31 others); Wed, 27 Jun 2018 12:06:18 -0400 Received: from mail-wr0-f194.google.com ([209.85.128.194]:44565 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S935050AbeF0QGN (ORCPT ); Wed, 27 Jun 2018 12:06:13 -0400 Received: by mail-wr0-f194.google.com with SMTP id p12-v6so2585645wrn.11 for ; Wed, 27 Jun 2018 09:06:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bihw6AeKi0fURk5bIT2bV23QoQQq88ujufV07nqCrwI=; b=LonVhfcj1q+ksxpCU58rrgajQcfmWDqT+Bll21pKsV9uhxz5VJ8XnjkVJnjuU/IGur vIxc5ptaVzTPUzbcCQZ1cBBYd3q9hfKITxtMAe5i4QIKTHVnT6oXhezpETTv7r7ZbvhB fD4+TxebhU3/GkEZPgvGqpWKNzYbVtIHbv9kQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bihw6AeKi0fURk5bIT2bV23QoQQq88ujufV07nqCrwI=; b=JdlJQmfdanbZZwivI8QlBOmpAIeNl8dKN2JpTIXLpiLykt9h9Kotsk5tcE4rtWXCse ARNFS7owzDpXLu2o0PtUCI2TwVt+a+/Xjqv0nMhNXPOA/QSlp2XvhEGzyVClAPFlttlN cSrDGBEYV7RJImXtHCVr3uERbvEd37qVzDKCr/UR5DA7F92MnVImUd/sn8XpReOeEQGw pUrC4k5UxC1VLVHdS4D/yRMydNsWWYH8TpF5eRU2pA0a7dp0HUJfMlp+l6RZPYcFv0ja AYw3EY2z6Pc+7Dk8bweA9RlxDo5vdYfuVxmlNBUcmI5g6kdV55X4+QXregkta128tWQy nJLQ== X-Gm-Message-State: APt69E0dDAKw8D12r1wUlYc+rpIoDCk9wtk17WwPnx1ea9YBiJEr6IYa JcGQ48mK5KTrRoiMrAWJ82LpEBT7U4k= X-Received: by 2002:adf:9883:: with SMTP id w3-v6mr6056390wrb.9.1530115571547; Wed, 27 Jun 2018 09:06:11 -0700 (PDT) Received: from rev02.home ([2a01:cb1d:112:6f00:4d04:66e:dea8:5b23]) by smtp.gmail.com with ESMTPSA id s2-v6sm7055000wrn.75.2018.06.27.09.06.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Jun 2018 09:06:10 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Arnd Bergmann , Steven Rostedt , Peter Zijlstra Subject: [PATCH 3/5] arm64/kernel: jump_label: switch to relative references Date: Wed, 27 Jun 2018 18:06:02 +0200 Message-Id: <20180627160604.8154-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180627160604.8154-1-ard.biesheuvel@linaro.org> References: <20180627160604.8154-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On a randomly chosen distro kernel build for arm64, vmlinux.o shows the following sections, containing jump label entries, and the associated RELA relocation records, respectively: ... [38088] __jump_table PROGBITS 0000000000000000 00e19f30 000000000002ea10 0000000000000000 WA 0 0 8 [38089] .rela__jump_table RELA 0000000000000000 01fd8bb0 000000000008be30 0000000000000018 I 38178 38088 8 ... In other words, we have 190 KB worth of 'struct jump_entry' instances, and 573 KB worth of RELA entries to relocate each entry's code, target and key members. This means the RELA section occupies 10% of the .init segment, and the two sections combined represent 5% of vmlinux's entire memory footprint. So let's switch from 64-bit absolute references to 32-bit relative references: this reduces the size of the __jump_table by 50%, and gets rid of the RELA section entirely. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/jump_label.h | 16 ++++------------ arch/arm64/kernel/jump_label.c | 6 +++--- 3 files changed, 8 insertions(+), 15 deletions(-) -- 2.11.0 Acked-by: Will Deacon diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1940c6405d04..d17aa9614e69 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -91,6 +91,7 @@ config ARM64 select HAVE_ARCH_BITREVERSE select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 1b5e0e843c3a..5e3e31df3805 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -30,8 +30,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm goto("1: nop\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -44,8 +44,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool { asm goto("1: b %l[l_yes]\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -54,13 +54,5 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool return true; } -typedef u64 jump_label_t; - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index c2dd1ad3e648..903d17023d77 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -25,12 +25,12 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - void *addr = (void *)entry->code; + void *addr = (void *)jump_entry_code(entry); u32 insn; if (type == JUMP_LABEL_JMP) { - insn = aarch64_insn_gen_branch_imm(entry->code, - entry->target, + insn = aarch64_insn_gen_branch_imm(jump_entry_code(entry), + jump_entry_target(entry), AARCH64_INSN_BRANCH_NOLINK); } else { insn = aarch64_insn_gen_nop(); From patchwork Wed Jun 27 16:06:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140362 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1069281ljj; Wed, 27 Jun 2018 09:06:23 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJx32RQvo84XMNhXq4zVpqt+6ycjPxAlvUf9AF1Fjynu4O5XFoH5TH1BwJWvCAnlVuYsinO X-Received: by 2002:a17:902:ba8a:: with SMTP id k10-v6mr6799473pls.338.1530115583267; Wed, 27 Jun 2018 09:06:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530115583; cv=none; d=google.com; s=arc-20160816; b=KJv4sFVt9PKdXPB2adksXFNCfgTDppEgku2SlqcqDFbiyk7NTLEwyQFj3JN3RDPORf F5X/2l3DnNfxQZSxf6hVcSO/RriMZYUEmR40A907A3ECYWi0syTDM8LrQcuSsh1etq/B Ep7K2aIZAYfsbjYF+95TU+PshfiuHlLyZlMbFjQyMNVDnoF3JJazB1rcLfIEZ3l28mMu iGFur0i8yVv0YJAnzo5HIUEeGzneCnbQpZtkMCnIODC2AypbEhPjk2XpxOtgJ5LGyIH+ 3p2LGB+zvWw5KIwuzVXRPxZEVnMFMjfklf7xU4tnLtbHrI4ajQG07llXQ+E+ShZodrXI /iXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=esdLUiP7Nq3pkGb9sS91qfLIwlUoKTrbJ1pPFJuzVN8=; b=fny9jkwc2mcNLUuVmeS7DVAbT3DTVZ/ACCeF3+XKipjCNR/MQqFSu1mgGoucKbN2DJ CqET+gZI022UYseuo5HRRy6H7IeTgsvGGh8T00xxu5hqnTBaLLMdpC41qTDZvCZ7YBud a0t9AOk1qz4xit4Fh8XCYOybf/P0/UpfhtDm6fSCReRrzbfusnu49DBwHQxJEFhStWnk 9JdtHpJyJVZeAqzltDakxyFFhgUudtq0V+712vliq0fZ32vLhimumvFUb0uK1fU4wiex fAFABIf+8qSLgSDs6Q60dSSEARvEPM61V7POti4kfRrs88QHRfnA0S+C5l6DO8hnDy1m nSuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=CFOKClWp; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a10-v6si4089041pff.304.2018.06.27.09.06.22; Wed, 27 Jun 2018 09:06:23 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=CFOKClWp; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935110AbeF0QGU (ORCPT + 31 others); Wed, 27 Jun 2018 12:06:20 -0400 Received: from mail-wm0-f67.google.com ([74.125.82.67]:54208 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934909AbeF0QGO (ORCPT ); Wed, 27 Jun 2018 12:06:14 -0400 Received: by mail-wm0-f67.google.com with SMTP id b188-v6so321937wme.3 for ; Wed, 27 Jun 2018 09:06:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=esdLUiP7Nq3pkGb9sS91qfLIwlUoKTrbJ1pPFJuzVN8=; b=CFOKClWpw5aVyAdTvFohT5qLEXHmaYuvG7el2Mj0dHB2mwVKTW2MFGWsyrkUpBMILS ChNE9yCoBxh78nTLsP/Bm0nc0tkOTSwx0MzIoHQUFzm4XeBhhuUnXwWjroQ8AF6w5i08 SAaSJr5sUWwDHqD9Aiox6yQObmW1VDz2iY1gs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=esdLUiP7Nq3pkGb9sS91qfLIwlUoKTrbJ1pPFJuzVN8=; b=bUsstkuIKUimNZQcn5pAyenJIi9vHyRio1sPDAYK8nQyHFaChX1USPIzcrOeOfwse0 YB895HVEbXwDBNkjI1FzvjZan+7h13KtAVOx1nJMbBrNe7uBgwzI81czpLlENYSQvOYc RgJP8q2BJaEOmy+cxUwP9yIXTvXOSVgIf2GeXQOJe0QRry+3CKVvscUXVLSQbstfRYG1 adCvl9euZcU4CzSUCF2r6BF2ENQ4i13L3f1OmZYLBtg7dF2g8JAPqcf049Kla9sAd+qk FjTLyD8jghAhfkNxTv94hyafPPT2VW+WWZBok8oB2zi+FUJSdpcy9istgJY0r56LZQ/7 3Qew== X-Gm-Message-State: APt69E1MMol4rwZhe1IeAF72bnhyfxZq1grO/TafMiSP+0hKdY1Ze7WJ 4DvA3glObRpqyTwuxBo9Z3kWI4UlAss= X-Received: by 2002:a1c:568a:: with SMTP id k132-v6mr2709334wmb.89.1530115573151; Wed, 27 Jun 2018 09:06:13 -0700 (PDT) Received: from rev02.home ([2a01:cb1d:112:6f00:4d04:66e:dea8:5b23]) by smtp.gmail.com with ESMTPSA id s2-v6sm7055000wrn.75.2018.06.27.09.06.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Jun 2018 09:06:12 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Arnd Bergmann , Steven Rostedt , Peter Zijlstra Subject: [PATCH 4/5] x86: jump_label: switch to jump_entry accessors Date: Wed, 27 Jun 2018 18:06:03 +0200 Message-Id: <20180627160604.8154-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180627160604.8154-1-ard.biesheuvel@linaro.org> References: <20180627160604.8154-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation of switching x86 to use place-relative references for the code, target and key members of struct jump_entry, replace direct references to the struct members with invocations of the new accessors. This will allow us to make the switch by modifying the accessors only. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/jump_label.c | 43 ++++++++++++-------- 1 file changed, 26 insertions(+), 17 deletions(-) -- 2.11.0 diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e56c95be2808..d64296092ef5 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -52,22 +52,24 @@ static void __jump_label_transform(struct jump_entry *entry, * Jump label is enabled for the first time. * So we expect a default_nop... */ - if (unlikely(memcmp((void *)entry->code, default_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { /* * ...otherwise expect an ideal_nop. Otherwise * something went horribly wrong. */ - if (unlikely(memcmp((void *)entry->code, ideal_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + ideal_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); } else { /* * We are disabling this jump label. If it is not what @@ -76,14 +78,18 @@ static void __jump_label_transform(struct jump_entry *entry, * are converting the default nop to the ideal nop. */ if (init) { - if (unlikely(memcmp((void *)entry->code, default_nop, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); - if (unlikely(memcmp((void *)entry->code, &code, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); + if (unlikely(memcmp((void *)jump_entry_code(entry), + &code, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } memcpy(&code, ideal_nops[NOP_ATOMIC5], JUMP_LABEL_NOP_SIZE); } @@ -97,10 +103,13 @@ static void __jump_label_transform(struct jump_entry *entry, * */ if (poker) - (*poker)((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE); + (*poker)((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE); else - text_poke_bp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE, - (void *)entry->code + JUMP_LABEL_NOP_SIZE); + text_poke_bp((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE, + (void *)jump_entry_code(entry) + + JUMP_LABEL_NOP_SIZE); } void arch_jump_label_transform(struct jump_entry *entry, From patchwork Wed Jun 27 16:06:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 140365 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1070143ljj; Wed, 27 Jun 2018 09:07:03 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKWfiK3Dqsv5DkumgiNc2TVnpyOMhBqizM53OK6DQVHnEKKAENsjKvEhJuw2q353bC0q4t5 X-Received: by 2002:a17:902:3081:: with SMTP id v1-v6mr6958475plb.266.1530115623712; Wed, 27 Jun 2018 09:07:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530115623; cv=none; d=google.com; s=arc-20160816; b=YVOz0DyTBbjgmWYtM1aBcrGHbboh5MtCUl2UUYhNMQbDw2QpY42R+UeDl2uTqd/Y35 8Maq7igyqOH1PXJcEYE3Hh9/dCXxCR/T4qNqZow7QbRY6bd8q6UJQFseAETOxyp4F6+m GSNxiLYBF9TsIctxHxPgAtNBd9DH4exghFEIzonOq8Tpufs5JnqRW4/r3e0lmooWe53L Cq5azhEaiigJBXl60m5HcV87ddMx1hjNnrb9cKdLVjFQ5JX64TLuDQxBDTe84eNxdK4z IeL4Qt6fHetj8EZiWXnkm2WD3tj+1oe0c1EABlz7aqYQfDXt6Dph9Zatb1QlThG238aN y6CQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=3yRjPsLxkB46VvOLi+86pNrf3CoOWXbFse0Dmkmvc3s=; b=FcMTWsD49EmAdAudkgfjIH/L+0VCGeEphvx25B7oR2GuN7s9wbmQSqcXJolv85cNy+ p+dzNS4yxfRDx3hz7hA8bSfYuhfthjtm529vBj/jvYtAWceuYQYaBGGzZCjR1qaR78ii 3kjRVVyAxn7E4t0Dy+o9nQPkWqWazkkMpLzrKBEhM4NVz/5yW3mNTSKjR0s4wTEWDTK8 aMRKwO0tXxwBAAuvrLTQBGTw46py+boUlXZpeLKzBb7PHzpWXBv/h78NGTrFfJ0oUEUI VXsderPjHfl8fJWDunsZ0NeXfzVK3vJetkvsGCT2oLip657HS/qoC5EJJjN0suT6W2c6 W25w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="dJd/8Yyx"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j5-v6si4113271pfh.3.2018.06.27.09.07.03; Wed, 27 Jun 2018 09:07:03 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="dJd/8Yyx"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965682AbeF0QHB (ORCPT + 31 others); Wed, 27 Jun 2018 12:07:01 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:33955 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S935065AbeF0QGQ (ORCPT ); Wed, 27 Jun 2018 12:06:16 -0400 Received: by mail-wm0-f66.google.com with SMTP id l15-v6so18434174wmc.1 for ; Wed, 27 Jun 2018 09:06:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3yRjPsLxkB46VvOLi+86pNrf3CoOWXbFse0Dmkmvc3s=; b=dJd/8Yyxskiuc9hVUV5Gfebg3Y4+xz/jCiFF1KxcKQSG52XBlC+lTLyMk2p8oPrGG3 KmsT5bX8P+tY1kJZmQGtm6EgEYtf29u8fShb3/iUI1R2lnezr+6CZn00weWsHco3D7F/ +LKJ4oW6Qi4mWcz25rt1dZmJvl26KrnPeURcI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3yRjPsLxkB46VvOLi+86pNrf3CoOWXbFse0Dmkmvc3s=; b=gqra1e9LSbACLQARq8hNiI0QPl5HzsDTdq+gJWFbgYBy6Z+mWP5OhfJ9+4okCeW0n/ V4GAeWmN5LgnLBCqQfJQLyef8vNMj8hXZdepQGxzh0JrsQaXIt+5LhQgukznLTwWjzQ5 tAIRndZEL3R75vmLlqIhflyzAdNfVK9mazeaAflG+/UfIVCxsM/mf7Jic8nQ5fFvekCv WgC6ISxGGgDq2Krs8yUQ4rmckfJjnHvlllhFBR7SliCBgDw07yZyu2QMWoOtLdIZ6IVr dYtpFAUJPE6neaO5lkx/ofTdUMg9VcAxZ2c8Qi+ncdim+cdEBXL+bzOGFJS7tB55JyKT cHzA== X-Gm-Message-State: APt69E3csB+3dlGot0j2FklVIVc2p3OyhSsjaYIqyqIU6GtCZNkdOiwv VqStsIFGdThouXzRx6NiqnnC++su/bg= X-Received: by 2002:a1c:9c4c:: with SMTP id f73-v6mr5286896wme.141.1530115574543; Wed, 27 Jun 2018 09:06:14 -0700 (PDT) Received: from rev02.home ([2a01:cb1d:112:6f00:4d04:66e:dea8:5b23]) by smtp.gmail.com with ESMTPSA id s2-v6sm7055000wrn.75.2018.06.27.09.06.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Jun 2018 09:06:13 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Arnd Bergmann , Steven Rostedt , Peter Zijlstra Subject: [PATCH 5/5] x86/kernel: jump_table: use relative references Date: Wed, 27 Jun 2018 18:06:04 +0200 Message-Id: <20180627160604.8154-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180627160604.8154-1-ard.biesheuvel@linaro.org> References: <20180627160604.8154-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Similar to the arm64 case, 64-bit x86 can benefit from using 32-bit relative references rather than 64-bit absolute ones when emitting struct jump_entry instances. Not only does this reduce the memory footprint of the entries themselves by 50%, it also removes the need for carrying relocation metadata on relocatable builds (i.e., for KASLR) which saves a fair chunk of .init space as well (although the savings are not as dramatic as on arm64) Signed-off-by: Ard Biesheuvel --- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 28 ++++++-------------- tools/objtool/special.c | 4 +-- 3 files changed, 11 insertions(+), 22 deletions(-) -- 2.11.0 diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index e10a3542db7e..dd71258ec1cc 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -118,6 +118,7 @@ config X86 select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if X86_64 || X86_PAE select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 8c0de4282659..d0f1f25b41d5 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -36,8 +36,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran asm_volatile_goto("1:" ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -52,8 +52,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool ".byte 0xe9\n\t .long %l[l_yes] - 2f\n\t" "2:\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -62,18 +62,6 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool return true; } -#ifdef CONFIG_X86_64 -typedef u64 jump_label_t; -#else -typedef u32 jump_label_t; -#endif - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #else /* __ASSEMBLY__ */ .macro STATIC_JUMP_IF_TRUE target, key, def @@ -87,8 +75,8 @@ struct jump_entry { .byte STATIC_KEY_INIT_NOP .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key - . .popsection .endm @@ -103,8 +91,8 @@ struct jump_entry { .Lstatic_jump_after_\@: .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key + 1 - . .popsection .endm diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 84f001d52322..98ae55b39037 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -30,9 +30,9 @@ #define EX_ORIG_OFFSET 0 #define EX_NEW_OFFSET 4 -#define JUMP_ENTRY_SIZE 24 +#define JUMP_ENTRY_SIZE 12 #define JUMP_ORIG_OFFSET 0 -#define JUMP_NEW_OFFSET 8 +#define JUMP_NEW_OFFSET 4 #define ALT_ENTRY_SIZE 13 #define ALT_ORIG_OFFSET 0