From patchwork Wed Dec 27 08:50:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122764 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1920191qgn; Wed, 27 Dec 2017 00:51:36 -0800 (PST) X-Google-Smtp-Source: ACJfBosHdhzTXo7PFjMEhqzx1FQtIjvmA1qWN7nsUwKfjQtcCiDlW/x6AD7MTPruCO+u1UfAG2K5 X-Received: by 10.84.247.2 with SMTP id n2mr27666110pll.268.1514364696622; Wed, 27 Dec 2017 00:51:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364696; cv=none; d=google.com; s=arc-20160816; b=p8DwrekSQSo6AwmHpNimPOf7H6442dYvSWi7+qRfRVpHYWkVorb3/mRaFiHV59SlVS LYHPH7oi/f1G+Tz+6thxmhiqXAtA9Z4qsgH2YkZAynKOkZQIbZSXv7xNfULmzdZ+1CyK 8jKmY49ykwRD9a6PDxkZn9HUjz61niuJJCuioaBGoaMdT8zaSUm2tM8ONviB0HJ+e3bZ dgI7W7ep2zxYQ8zj2WYRMyttNxteRJdQMKGBY1exqica9jjw97Qzg4JUxqXgqV3AhHWB ABf0xIW2pBd0UciQY/7TC8YeH/nf2/17nv6NZqqHb1iH3rWk+tZq78J9fP0kGfInDFwM xovg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=nmovwwxZQQa2ugEDLnXxoU69CsqWUnhsDto5qVfQljI=; b=o5lRHVM3uD1s4Df9T1HjmjwlTNX7iuKEPQefhNsGK5JTGHj5sdcoH7J27nu9Jnkl6q GgEJdBruYoTfY39xcWhsBDHDm0r8xGVjYsgQJar+JhLpQYi1DrUjUkbSlqvO20LpaCNC FpFGQRFa5entE9Rdjtr9vPfjgXQGOiOgCzwFSnJCkCm6tT9Iq7xhZPB/of2+X4TBpxWE ns2baB4kUv9vL8Qr+uedk4PgpfIhMTepYODOYlbcymdMzzvsUtkC25neKKI70wkwIpT8 LsMH0r1b0wNh0bdw7Mv07fsYTv+h6t0AYywuciGrn4pLMlNQlAktS2OD21Ts8IhvwzYj FrVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WxlwI5eY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 31si1458344pla.359.2017.12.27.00.51.36; Wed, 27 Dec 2017 00:51:36 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WxlwI5eY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751964AbdL0Ivd (ORCPT + 28 others); Wed, 27 Dec 2017 03:51:33 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:32891 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751934AbdL0Iv3 (ORCPT ); Wed, 27 Dec 2017 03:51:29 -0500 Received: by mail-wm0-f67.google.com with SMTP id g130so38648674wme.0 for ; Wed, 27 Dec 2017 00:51:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nmovwwxZQQa2ugEDLnXxoU69CsqWUnhsDto5qVfQljI=; b=WxlwI5eYdKaO3NmcGBbfQz+LK6UKVWOFoMQ+Jyib+BEmRa2fOBlwIMLzNism5cTraH jwShntvWLc1MLEshXs0fqjK4udpfG/RyCibzZzw0w/no0a/0bNsUe8LbcE4bd6Bxc4nm APC55TMa+SgyxGz/NqB151Pg5zBZdpBRBlez4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nmovwwxZQQa2ugEDLnXxoU69CsqWUnhsDto5qVfQljI=; b=UKZ2KDct0eWgOx4jo2h7qZWCZlAANBVcLF0nyIzCSp8+3VuUJqMmokLRyExonOu0Tu egXusYfQh2u7E9WMIFYNuoQVaLItzzc//BtI+VAD3KH70j/g+Y/EzOxK6LTAReIA96Kq 3keJcyMy1LSGLP/t9ITEBIdxc4IBsGJGB3YZB0pShzIjl9uWsKw5XNCWdm3HWbtbyyeD Q7WJUVHcK1YL1dlgGQasqNMIF5K8S/WIAcianzL9/UeVCXOoakjj+B+Ymr0vwgx5mFXV Divx/zd7d0IYpHEpcGV0M4S2h9bmxQTzz+V0BvfKjTMvVfEQ3xDyJrGx+eweKw4ovXen GEqQ== X-Gm-Message-State: AKGB3mLQ8+cTnpWf26cDN1iAcUQxb6jhOXNGq0PdvPRWKNQEWAMURpd8 4yf9lJAmfqr+TkHpLs+LlYA+SL9aL+k= X-Received: by 10.28.106.16 with SMTP id f16mr13613978wmc.58.1514364687433; Wed, 27 Dec 2017 00:51:27 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.51.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:51:26 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v6 7/8] arm64/kernel: jump_label: use relative references Date: Wed, 27 Dec 2017 08:50:32 +0000 Message-Id: <20171227085033.22389-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On a randomly chosen distro kernel build for arm64, vmlinux.o shows the following sections, containing jump label entries, and the associated RELA relocation records, respectively: ... [38088] __jump_table PROGBITS 0000000000000000 00e19f30 000000000002ea10 0000000000000000 WA 0 0 8 [38089] .rela__jump_table RELA 0000000000000000 01fd8bb0 000000000008be30 0000000000000018 I 38178 38088 8 ... In other words, we have 190 KB worth of 'struct jump_entry' instances, and 573 KB worth of RELA entries to relocate each entry's code, target and key members. This means the RELA section occupies 10% of the .init segment, and the two sections combined represent 5% of vmlinux's entire memory footprint. So let's switch from 64-bit absolute references to 32-bit relative references: this reduces the size of the __jump_table by 50%, and gets rid of the RELA section entirely. Note that this requires some extra care in the sorting routine, given that the offsets change when entries are moved around in the jump_entry table. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/jump_label.h | 27 ++++++++++++-------- arch/arm64/kernel/jump_label.c | 22 +++++++++++++--- 2 files changed, 36 insertions(+), 13 deletions(-) -- 2.11.0 diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 9d6e46355c89..5cec68616125 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -30,8 +30,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm goto("1: nop\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -44,8 +44,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool { asm goto("1: b %l[l_yes]\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -57,19 +57,26 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool typedef u64 jump_label_t; struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (jump_label_t)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (jump_label_t)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -87,7 +94,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index c2dd1ad3e648..2b8e459e91f7 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -25,12 +25,12 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - void *addr = (void *)entry->code; + void *addr = (void *)jump_entry_code(entry); u32 insn; if (type == JUMP_LABEL_JMP) { - insn = aarch64_insn_gen_branch_imm(entry->code, - entry->target, + insn = aarch64_insn_gen_branch_imm(jump_entry_code(entry), + jump_entry_target(entry), AARCH64_INSN_BRANCH_NOLINK); } else { insn = aarch64_insn_gen_nop(); @@ -50,4 +50,20 @@ void arch_jump_label_transform_static(struct jump_entry *entry, */ } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif /* HAVE_JUMP_LABEL */