From patchwork Tue Jan 2 20:05:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123214 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4760901edv; Tue, 2 Jan 2018 12:06:30 -0800 (PST) X-Google-Smtp-Source: ACJfBosoJC0RgfUETe+l45MJ8EEDixF6UTaMLWhTii3StfVWF2H7bOVtRwUaMdWL+axWPtUePP7k X-Received: by 10.101.66.12 with SMTP id c12mr16841950pgq.63.1514923589384; Tue, 02 Jan 2018 12:06:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923589; cv=none; d=google.com; s=arc-20160816; b=sXm/vUP605a3taF0Ot+cEzAnaG5Mwo7yb/+J96D3E0/RANksr/aFpHKVDkZGwR2uby hhjM5+ke7KNWzyDR6o1S1OQ9SKJB4djtyPgbfOXGMvyV61xGe6FBBLlSIAiJdwu2yGxX Vm3FR8ArETfPCbhZjtn7auflMKAdx6rcAgtGCfa3l/8K1J28MfFkpJxQbZbgay04+Iyp qP5DF7JWxEw9b52A8ZYg1yXPOaqpxczIkIMhp3fRVS17nOHz1mVFeSAzklGJzUoET9ws V1TJbeqFEq/WXQTnhyD6t7EpUmt07nSG7XaEsegoWUBbV3Yp3Vw9Kb4hdDXIYsGhMfE0 xhnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=8wbbwFOWuQMFP0SKy8UHj3NriHdOJ7kBnwcJ7OnyKjo=; b=gG9lNcegbsE7hMZCzoVFBj7e+qAh0VTA7ScKr+gE/8xe+6xw4c8WzwF5apy7tGqb44 +uEYGikUeJs9DEkaSpfoI9mk5hRh//4LCX7y+dVdat4oNyrOqSQf7hPbx31aFEVTUxSf 44Dm8oSGl4LsWBczyQ5uoJAEq/sGKCHiz1QaEMy/MyzFL9cN9OiVGtQN2w3c3155f6fN uBtuqGwVcWczDBkMzovX+8C/dP/S98nQySEfXjhHVJFz2AH4Qc/QYgjzJiaRArNYQSaA IL8I72LM3JO6bETm2ZOH/h7y4TLk8WhO/rbX90HdRkup2z+BFSleZW/7CbK0dXIB5GPI SV9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=DYml98nl; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z8si2242279pfh.232.2018.01.02.12.06.28; Tue, 02 Jan 2018 12:06:29 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=DYml98nl; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751059AbeABUGY (ORCPT + 28 others); Tue, 2 Jan 2018 15:06:24 -0500 Received: from mail-wr0-f194.google.com ([209.85.128.194]:40266 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751014AbeABUGT (ORCPT ); Tue, 2 Jan 2018 15:06:19 -0500 Received: by mail-wr0-f194.google.com with SMTP id p17so25843299wre.7 for ; Tue, 02 Jan 2018 12:06:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8wbbwFOWuQMFP0SKy8UHj3NriHdOJ7kBnwcJ7OnyKjo=; b=DYml98nl5C9dxpX8PaphCwO1AwOOVT37wBSgEpvuWvz1+8lzAEk4lAznRm97Y/6LxX Y4BE3AmmwB7ErSHppbzjssrLWql23m3K278dd7UPXIhPBJLssy7YFhsS3HVTTffQ0BvS xMwTDlpUaHy6gjzAk7kS/8JHmrEGXAcyxkyKE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8wbbwFOWuQMFP0SKy8UHj3NriHdOJ7kBnwcJ7OnyKjo=; b=GvEGA/kt6HDPWqBqCZiRr0OD+ZCz83cIKtkoklLbSmmbFQkWv6Kov1ZKs+839CvB4p CFIzveSmgHhEEZ4ii8K1px/A71HPNEflmc769cEnnXbpDJk4Dqhqc1dowL8g3wfM+tfx 9T/BK16H8vWuzrI7LwWKtqQSYBtbE4KwBoru5pHzAtXxPAw1OGzN9IeiPxoAIjnnZynx aUqklRSELIB9aiSS94h1vKX4O/wdpu308m4OA8KqWUNNeiTEmEVEjqvqpoICetF9wT3D 7R0qSj0Ajn92NuAKSznPryE2SpTYNuQfrmaIGBUWKD1zBtDAq9z+CSrskmq+TrziPuFs Ltyw== X-Gm-Message-State: AKGB3mLfnwS8VSFOjJUKncNciXdOwNHUk0uKClxE+RfjiOc8M/kkNa3u 8vyDVAoI1F6VIFoLA4vl6tv+/BkGmjc= X-Received: by 10.223.151.146 with SMTP id s18mr20037055wrb.180.1514923577962; Tue, 02 Jan 2018 12:06:17 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:17 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v7 01/10] arch: enable relative relocations for arm64, power and x86 Date: Tue, 2 Jan 2018 20:05:40 +0000 Message-Id: <20180102200549.22984-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Before updating certain subsystems to use place relative 32-bit relocations in special sections, to save space and reduce the number of absolute relocations that need to be processed at runtime by relocatable kernels, introduce the Kconfig symbol and define it for some architectures that should be able to support and benefit from it. Cc: Catalin Marinas Cc: Will Deacon Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: Michael Ellerman Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin" Cc: x86@kernel.org Signed-off-by: Ard Biesheuvel --- arch/Kconfig | 10 ++++++++++ arch/arm64/Kconfig | 1 + arch/powerpc/Kconfig | 1 + arch/x86/Kconfig | 1 + 4 files changed, 13 insertions(+) -- 2.11.0 diff --git a/arch/Kconfig b/arch/Kconfig index 400b9e1b2f27..dbc036a7bd1b 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -959,4 +959,14 @@ config REFCOUNT_FULL against various use-after-free conditions that can be used in security flaw exploits. +config HAVE_ARCH_PREL32_RELOCATIONS + bool + help + May be selected by an architecture if it supports place-relative + 32-bit relocations, both in the toolchain and in the module loader, + in which case relative references can be used in special sections + for PCI fixup, initcalls etc which are only half the size on 64 bit + architectures, and don't require runtime relocation on relocatable + kernels. + source "kernel/gcov/Kconfig" diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c9a7e9e1414f..66c7b9ab2a3d 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -89,6 +89,7 @@ config ARM64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index c51e6ce42e7a..e172478e2ae7 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -177,6 +177,7 @@ config PPC select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select ARCH_HAS_STRICT_KERNEL_RWX if ((PPC_BOOK3S_64 || PPC32) && !RELOCATABLE && !HIBERNATION) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index d4fc98c50378..9f2bb853aedb 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -115,6 +115,7 @@ config X86 select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT select HAVE_ARCH_COMPAT_MMAP_BASES if MMU && COMPAT + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE From patchwork Tue Jan 2 20:05:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123215 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4761041edv; Tue, 2 Jan 2018 12:06:38 -0800 (PST) X-Google-Smtp-Source: ACJfBotqLT7FWwxaA+vKN0cz35RT4VreWlv6P1vyW+G72zuq52JOU6cBEI8obRahGliLCNFkK2m9 X-Received: by 10.159.198.10 with SMTP id f10mr45807868plo.356.1514923598016; Tue, 02 Jan 2018 12:06:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923598; cv=none; d=google.com; s=arc-20160816; b=Eu4cx5LwXrA8mY25leRyJsfiXj8EI/ozjtYTFaUH6b1UaTzkH9/lhTnrC9AmsP/7Iu NVBCSywjihUHxZ6fqJWoBeKbEZYNLu4CpQOwu2iO3W4qMGGMlPVnEZ9ryJcocXIlBoxI K3H8YFsN0cS3Fv/vuelX7rST7Nt7KN+l5COOvu0S9b5nRg2IAWLbR9padwXDHPf4dgSR +TfkmHu20Pdd0ISlB2tLLy6b0y/3sJat1/80WicUYFeaW9YePrD5ZkxLdEGJrOe33Fxu lzCfWXbpH2AOrwzWJwPGRw8DZO9/12+bm2qRh1vf3NcIKbLhsoSD43OXjVtaxGuUXT6c FlaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=9pf2nmv48uHqsDbsl+hfOY5WaUyvrm1nnNGc9/9m6Qg=; b=T7qXe5l6sp2B+R2a5vhmEHXuZcgYPK6Pc/nz/dZeuwmgbiIkj9Eijx9v9agDzEfM76 94bf/OAHYjQP/1MfJcP3iF4Rz9V4jWcSGwyZGlQcrSurShTWKiwhGEpLmZUMXORrApwG fUzCaU7ll119QlSLo9Ke0yDdJzspwOTRJ44YYNljqOyQft6a1sw3/GMHkXIRoKZMdJCD JUlualbWmoo13cQWju1oq3xzbPTX6qRVUeaJI4djo6K38CWXXOLth0kt2XMHZpr98v9F tu2ia3DruyOFgmrsVZQWYU0Egb3KzrQi+XD1Pt0zA7xh0Y34ZEEoLqJL3I/c8yETSDjJ QyLA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JvakjjXX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o14si4476037pgq.631.2018.01.02.12.06.37; Tue, 02 Jan 2018 12:06:38 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JvakjjXX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751099AbeABUGa (ORCPT + 28 others); Tue, 2 Jan 2018 15:06:30 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:41910 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751061AbeABUG0 (ORCPT ); Tue, 2 Jan 2018 15:06:26 -0500 Received: by mail-wm0-f67.google.com with SMTP id g75so63337433wme.0 for ; Tue, 02 Jan 2018 12:06:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9pf2nmv48uHqsDbsl+hfOY5WaUyvrm1nnNGc9/9m6Qg=; b=JvakjjXXh2ftac/uQdStpmRnyog8dVy1B1dV3/FBRYhclGcwN1Nf1ULl77WmER9ry4 qNvpqUIkOxaXMPtj6B3oJYPN87PgdaiNOi0rNpCB5XAS3BaMGwa31IA7QPoscn3u7sb0 uOQNLx6iterfuYqWKriCvDY/4G6QygREzc7s0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9pf2nmv48uHqsDbsl+hfOY5WaUyvrm1nnNGc9/9m6Qg=; b=Newx9+Mz59U4kOGNBIBAMoHew4qWP2J4iyr7F07UhtO8ZWmwYTClRXkggSxBSzoupe T5wtjKKBppP0YHhG1NikrDDJD6ZKZjmiVGO1uTxm69T/e/P4ufCd4rs6E/EFPcHqyjo2 sPziKBTlgYOLMkF0U9sCfpKCLhpRaaTrylVRhLdpoSd6rg4tWGyXCdoqHo5T2Z6EDsYh 2eiAness/vBtIAMatyQcM/xxGkrHrSxgP/dAcscMs9Q4iKwLkl5/zuPK4NyYOJX1GlDa AVLOO2pz+HXGhpkb7QFBW7ZAZAQI8v+u10MJoemcPcbzcuIlHdcH17u+8zdRNt58pGjc UhQA== X-Gm-Message-State: AKGB3mLiQqKAsKCrwh92FdOr54GQe9pLFUFPmn/WIkSOmuWtLsVYsree gHxvjaA2NaCHUFVNwMHnZ/i2uI4plZQ= X-Received: by 10.28.97.194 with SMTP id v185mr11849434wmb.21.1514923584444; Tue, 02 Jan 2018 12:06:24 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:23 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, matt@codeblueprint.co.uk Subject: [PATCH v7 02/10] module: allow symbol exports to be disabled Date: Tue, 2 Jan 2018 20:05:41 +0000 Message-Id: <20180102200549.22984-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org To allow existing C code to be incorporated into the decompressor or the UEFI stub, introduce a CPP macro that turns all EXPORT_SYMBOL_xxx declarations into nops, and #define it in places where such exports are undesirable. Note that this gets rid of a rather dodgy redefine of linux/export.h's header guard. Cc: matt@codeblueprint.co.uk Cc: keescook@chromium.org Cc: jeyu@kernel.org Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/kaslr.c | 5 +---- drivers/firmware/efi/libstub/Makefile | 3 ++- include/linux/export.h | 9 +++++++++ 3 files changed, 12 insertions(+), 5 deletions(-) -- 2.11.0 diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c index 8199a6187251..3a2a6d7049e4 100644 --- a/arch/x86/boot/compressed/kaslr.c +++ b/arch/x86/boot/compressed/kaslr.c @@ -23,11 +23,8 @@ * _ctype[] in lib/ctype.c is needed by isspace() of linux/ctype.h. * While both lib/ctype.c and lib/cmdline.c will bring EXPORT_SYMBOL * which is meaningless and will cause compiling error in some cases. - * So do not include linux/export.h and define EXPORT_SYMBOL(sym) - * as empty. */ -#define _LINUX_EXPORT_H -#define EXPORT_SYMBOL(sym) +#define __DISABLE_EXPORTS #include "misc.h" #include "error.h" diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index adaa4a964f0c..312bd0b64a61 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -20,7 +20,8 @@ cflags-$(CONFIG_EFI_ARMSTUB) += -I$(srctree)/scripts/dtc/libfdt KBUILD_CFLAGS := $(cflags-y) -DDISABLE_BRANCH_PROFILING \ -D__NO_FORTIFY \ $(call cc-option,-ffreestanding) \ - $(call cc-option,-fno-stack-protector) + $(call cc-option,-fno-stack-protector) \ + -D__DISABLE_EXPORTS GCOV_PROFILE := n KASAN_SANITIZE := n diff --git a/include/linux/export.h b/include/linux/export.h index 1a1dfdb2a5c6..6dba2fb08f77 100644 --- a/include/linux/export.h +++ b/include/linux/export.h @@ -83,6 +83,15 @@ extern struct module __this_module; */ #define __EXPORT_SYMBOL(sym, sec) === __KSYM_##sym === +#elif defined(__DISABLE_EXPORTS) + +/* + * Allow symbol exports to be disabled completely so that C code may + * be reused in other execution contexts such as the UEFI stub or the + * decompressor. + */ +#define __EXPORT_SYMBOL(sym, sec) + #elif defined(CONFIG_TRIM_UNUSED_KSYMS) #include From patchwork Tue Jan 2 20:05:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123216 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4761115edv; Tue, 2 Jan 2018 12:06:42 -0800 (PST) X-Google-Smtp-Source: ACJfBotDeom9GTcXhA0QnMZuEYHwY113a/e9F/JN36qsdEuPTi0r66zhk+TnApHmYEbDI2HUHh8n X-Received: by 10.159.216.139 with SMTP id s11mr45613567plp.266.1514923602063; Tue, 02 Jan 2018 12:06:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923602; cv=none; d=google.com; s=arc-20160816; b=dukVPnUroBgUA/D22plEvKVDr+pr5YbZK98zQXAtLrCkURtxd+1nbN3vkZ1hb6rgiG /ZMGC1wdgASbbMrF6qY5u4PN824oNyGXIJ6VrnKCHKvuuJ90x3rd3KdF0buolBsE0Qfy j15zZ48QmghyyScwIrKnga+DC1uL7grNJC1+m8YVMYXCDiXRKkv17PbSf4I/oEV4/z43 eNtmGPUQIcz8Tg4tNHXZqrRiZEfYx01+b41f9/GEWPZwq0CYMRQFcSxghEdiV/DjL3N5 kShK4MIuLL1rkPVm9B8UqgIBsTBCfcBhzvSoPnxfp8F6sp+goHv8+B3uYR35ifMvqCU+ OCjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=YeY2Xm5F7YqCntxBJlOwEmhyquMbh7oq6A2LuKFDBk0=; b=ziQVTsTXEo/f7NfxU5WXFyX/97BnrWFOgK7OfHjXwDuOZ6U8DDR/zrH+oxfYtIq16p 1vecdbt0xo/foe/mY0twFxf+N5WD/d1zWyTl86IvaaZzNSJYOxnr+Mu3bdrg4hQllPg3 NmERj86agxBaxMiQpX2iYHrl2e1jc3ItbBvCzYfaqTw+mT6jZG6mUu+2XsQpI+QsA4Ac N5oSxSskBbWrHBdnmcjE5Y0dkW7I8Emf9j/H8vNBr7uxyDjw+VaJDP/D20rT6ac+ry26 Q50U5Vf7aZWYwtRFZJM9ezLE4fDvyyfZAVIQpwH1ZxLFjyU1K8UM9jyWpAO1O44j78vT ZQ3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=I7jWcnRZ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r25si8528038pgv.7.2018.01.02.12.06.41; Tue, 02 Jan 2018 12:06:42 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=I7jWcnRZ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751141AbeABUGg (ORCPT + 28 others); Tue, 2 Jan 2018 15:06:36 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:45959 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751023AbeABUGb (ORCPT ); Tue, 2 Jan 2018 15:06:31 -0500 Received: by mail-wm0-f68.google.com with SMTP id 9so62960756wme.4 for ; Tue, 02 Jan 2018 12:06:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YeY2Xm5F7YqCntxBJlOwEmhyquMbh7oq6A2LuKFDBk0=; b=I7jWcnRZxSQJ+r+NtiSwxZap+e4+EqIntXlM8zv+adVYkjRBnxpAYSFrC0dqGsT1QF 8rDSC94FmIB9N8KAC8sCCuRCErl1CHAnd3c/Uj+mcD10ddgBJm92tB35pktDiPzqQesh +AnUz+oxl5F+2nDDr5snXku1SfLcXf+ApxaMw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YeY2Xm5F7YqCntxBJlOwEmhyquMbh7oq6A2LuKFDBk0=; b=S8NGlBkj0QBGT6ARtaSVKyIuQwZMM4niBsa3UajBqjxuTcOPU4qmnBdsTMeTK/K8Kh b8KXAUlqRhlxM23x4lAS5AsZL+o1GWqhu31ffe5+9nyH5B5xw2b/4p5ejQpftSNEtWIu pG+tXECZz9flvMBnAtHkQHE/kavW7TuqqNXq4cRDeWEN6//40wZcKXLb/6LZjSgxxkzJ R5+c/oSclDahzahgrR2r/ETvIXrlKFRJdQx5kC/85rcbw38NN5q4rnN1+Wontzlx8g1K xxP3gfwNFZMsaTNu/AQe7b5LDiPsOVvr+F/K2i7FT9ZpZJZm6mGTSJ95l6gJUuY1KAxy foOw== X-Gm-Message-State: AKGB3mJvxGP+yX/b65Y8CwyerWJuFPi//Ynm2HXsc+5TmqUnnwPXuT3g gARNe2/hGTLjI8urD8eEbVi9PSBTc/8= X-Received: by 10.28.129.69 with SMTP id c66mr38913551wmd.29.1514923590192; Tue, 02 Jan 2018 12:06:30 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:29 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, Ingo Molnar Subject: [PATCH v7 03/10] module: use relative references for __ksymtab entries Date: Tue, 2 Jan 2018 20:05:42 +0000 Message-Id: <20180102200549.22984-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org An ordinary arm64 defconfig build has ~64 KB worth of __ksymtab entries, each consisting of two 64-bit fields containing absolute references, to the symbol itself and to a char array containing its name, respectively. When we build the same configuration with KASLR enabled, we end up with an additional ~192 KB of relocations in the .init section, i.e., one 24 byte entry for each absolute reference, which all need to be processed at boot time. Given how the struct kernel_symbol that describes each entry is completely local to module.c (except for the references emitted by EXPORT_SYMBOL() itself), we can easily modify it to contain two 32-bit relative references instead. This reduces the size of the __ksymtab section by 50% for all 64-bit architectures, and gets rid of the runtime relocations entirely for architectures implementing KASLR, either via standard PIE linking (arm64) or using custom host tools (x86). Note that the binary search involving __ksymtab contents relies on each section being sorted by symbol name. This is implemented based on the input section names, not the names in the ksymtab entries, so this patch does not interfere with that. Given that the use of place-relative relocations requires support both in the toolchain and in the module loader, we cannot enable this feature for all architectures. So make it dependent on whether CONFIG_HAVE_ARCH_PREL32_RELOCATIONS is defined. Cc: Arnd Bergmann Cc: Andrew Morton Cc: Ingo Molnar Cc: Kees Cook Cc: Thomas Garnier Cc: Nicolas Pitre Acked-by: Jessica Yu Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/Kbuild | 1 + arch/x86/include/asm/export.h | 5 --- include/asm-generic/export.h | 12 ++++- include/linux/compiler.h | 10 +++++ include/linux/export.h | 46 +++++++++++++++----- kernel/module.c | 33 +++++++++++--- 6 files changed, 83 insertions(+), 24 deletions(-) -- 2.11.0 diff --git a/arch/x86/include/asm/Kbuild b/arch/x86/include/asm/Kbuild index 5d6a53fd7521..3e8a88dcaa1d 100644 --- a/arch/x86/include/asm/Kbuild +++ b/arch/x86/include/asm/Kbuild @@ -9,5 +9,6 @@ generated-y += xen-hypercalls.h generic-y += clkdev.h generic-y += dma-contiguous.h generic-y += early_ioremap.h +generic-y += export.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h diff --git a/arch/x86/include/asm/export.h b/arch/x86/include/asm/export.h deleted file mode 100644 index 2a51d66689c5..000000000000 --- a/arch/x86/include/asm/export.h +++ /dev/null @@ -1,5 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifdef CONFIG_64BIT -#define KSYM_ALIGN 16 -#endif -#include diff --git a/include/asm-generic/export.h b/include/asm-generic/export.h index 719db1968d81..97ce606459ae 100644 --- a/include/asm-generic/export.h +++ b/include/asm-generic/export.h @@ -5,12 +5,10 @@ #define KSYM_FUNC(x) x #endif #ifdef CONFIG_64BIT -#define __put .quad #ifndef KSYM_ALIGN #define KSYM_ALIGN 8 #endif #else -#define __put .long #ifndef KSYM_ALIGN #define KSYM_ALIGN 4 #endif @@ -25,6 +23,16 @@ #define KSYM(name) name #endif +.macro __put, val, name +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + .long \val - ., \name - . +#elif defined(CONFIG_64BIT) + .quad \val, \name +#else + .long \val, \name +#endif +.endm + /* * note on .section use: @progbits vs %progbits nastiness doesn't matter, * since we immediately emit into those sections anyway. diff --git a/include/linux/compiler.h b/include/linux/compiler.h index 52e611ab9a6c..79db4aa87d75 100644 --- a/include/linux/compiler.h +++ b/include/linux/compiler.h @@ -327,4 +327,14 @@ static __always_inline void __write_once_size(volatile void *p, void *res, int s compiletime_assert(__native_word(t), \ "Need native word sized stores/loads for atomicity.") +/* + * Force the compiler to emit 'sym' as a symbol, so that we can reference + * it from inline assembler. Necessary in case 'sym' could be inlined + * otherwise, or eliminated entirely due to lack of references that are + * visible to the compiler. + */ +#define __ADDRESSABLE(sym) \ + static void * const __attribute__((section(".discard"), used)) \ + __PASTE(__addressable_##sym, __LINE__) = (void *)&sym; + #endif /* __LINUX_COMPILER_H */ diff --git a/include/linux/export.h b/include/linux/export.h index 6dba2fb08f77..4744cf4736b0 100644 --- a/include/linux/export.h +++ b/include/linux/export.h @@ -24,12 +24,6 @@ #define VMLINUX_SYMBOL_STR(x) __VMLINUX_SYMBOL_STR(x) #ifndef __ASSEMBLY__ -struct kernel_symbol -{ - unsigned long value; - const char *name; -}; - #ifdef MODULE extern struct module __this_module; #define THIS_MODULE (&__this_module) @@ -60,17 +54,47 @@ extern struct module __this_module; #define __CRC_SYMBOL(sym, sec) #endif +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#include +/* + * Emit the ksymtab entry as a pair of relative references: this reduces + * the size by half on 64-bit architectures, and eliminates the need for + * absolute relocations that require runtime processing on relocatable + * kernels. + */ +#define __KSYMTAB_ENTRY(sym, sec) \ + __ADDRESSABLE(sym) \ + asm(" .section \"___ksymtab" sec "+" #sym "\", \"a\" \n" \ + " .balign 8 \n" \ + VMLINUX_SYMBOL_STR(__ksymtab_##sym) ": \n" \ + " .long " VMLINUX_SYMBOL_STR(sym) "- . \n" \ + " .long " VMLINUX_SYMBOL_STR(__kstrtab_##sym) "- .\n" \ + " .previous \n") + +struct kernel_symbol { + int value_offset; + int name_offset; +}; +#else +#define __KSYMTAB_ENTRY(sym, sec) \ + static const struct kernel_symbol __ksymtab_##sym \ + __attribute__((section("___ksymtab" sec "+" #sym), used)) \ + = { (unsigned long)&sym, __kstrtab_##sym } + +struct kernel_symbol { + unsigned long value; + const char *name; +}; +#endif + /* For every exported symbol, place a struct in the __ksymtab section */ #define ___EXPORT_SYMBOL(sym, sec) \ extern typeof(sym) sym; \ __CRC_SYMBOL(sym, sec) \ static const char __kstrtab_##sym[] \ - __attribute__((section("__ksymtab_strings"), aligned(1))) \ + __attribute__((section("__ksymtab_strings"), used, aligned(1))) \ = VMLINUX_SYMBOL_STR(sym); \ - static const struct kernel_symbol __ksymtab_##sym \ - __used \ - __attribute__((section("___ksymtab" sec "+" #sym), used)) \ - = { (unsigned long)&sym, __kstrtab_##sym } + __KSYMTAB_ENTRY(sym, sec) #if defined(__KSYM_DEPS__) diff --git a/kernel/module.c b/kernel/module.c index dea01ac9cb74..d3a908ffc42c 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -549,12 +549,31 @@ static bool check_symbol(const struct symsearch *syms, return true; } +static unsigned long kernel_symbol_value(const struct kernel_symbol *sym) +{ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + return (unsigned long)&sym->value_offset + sym->value_offset; +#else + return sym->value; +#endif +} + +static const char *kernel_symbol_name(const struct kernel_symbol *sym) +{ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + return (const char *)((unsigned long)&sym->name_offset + + sym->name_offset); +#else + return sym->name; +#endif +} + static int cmp_name(const void *va, const void *vb) { const char *a; const struct kernel_symbol *b; a = va; b = vb; - return strcmp(a, b->name); + return strcmp(a, kernel_symbol_name(b)); } static bool find_symbol_in_section(const struct symsearch *syms, @@ -2198,7 +2217,7 @@ void *__symbol_get(const char *symbol) sym = NULL; preempt_enable(); - return sym ? (void *)sym->value : NULL; + return sym ? (void *)kernel_symbol_value(sym) : NULL; } EXPORT_SYMBOL_GPL(__symbol_get); @@ -2228,10 +2247,12 @@ static int verify_export_symbols(struct module *mod) for (i = 0; i < ARRAY_SIZE(arr); i++) { for (s = arr[i].sym; s < arr[i].sym + arr[i].num; s++) { - if (find_symbol(s->name, &owner, NULL, true, false)) { + if (find_symbol(kernel_symbol_name(s), &owner, NULL, + true, false)) { pr_err("%s: exports duplicate symbol %s" " (owned by %s)\n", - mod->name, s->name, module_name(owner)); + mod->name, kernel_symbol_name(s), + module_name(owner)); return -ENOEXEC; } } @@ -2280,7 +2301,7 @@ static int simplify_symbols(struct module *mod, const struct load_info *info) ksym = resolve_symbol_wait(mod, info, name); /* Ok if resolved. */ if (ksym && !IS_ERR(ksym)) { - sym[i].st_value = ksym->value; + sym[i].st_value = kernel_symbol_value(ksym); break; } @@ -2540,7 +2561,7 @@ static int is_exported(const char *name, unsigned long value, ks = lookup_symbol(name, __start___ksymtab, __stop___ksymtab); else ks = lookup_symbol(name, mod->syms, mod->syms + mod->num_syms); - return ks != NULL && ks->value == value; + return ks != NULL && kernel_symbol_value(ks) == value; } /* As per nm */ From patchwork Tue Jan 2 20:05:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123217 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4761215edv; Tue, 2 Jan 2018 12:06:48 -0800 (PST) X-Google-Smtp-Source: ACJfBouts38i+nXF2cDknSDjp4QEaghOYafn4uZV8K/xGEuzBNCbzbkQZxwVq9wFglu16esWVtXW X-Received: by 10.98.36.199 with SMTP id k68mr45735651pfk.236.1514923608336; Tue, 02 Jan 2018 12:06:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923608; cv=none; d=google.com; s=arc-20160816; b=ZT2fOXErMxJQuZYKJ3uFznSmKcc8/1SwiidmPYvm4pNvE5JpNL8v91Nzacay9crlsr m5kRNwe7JMoXphVuhEuboZfoqogDG8uUpjc6OBi3lUeLjz774bcrlulnaJhOMn9nSSDa edIuVNpOfqg0j/7CgT25upDIE5W3mz1X1F6XHZQEPHDWKv5TBJ9N3Nsw/v9kqzXChmQ2 BrlzRnyyk70NNPcsXR3A4/bsGavKTXcLWsYXLRe+ksw6tjOSW9SdmBFVei5lCuhqM+cL GxSt2k0o5TAZgEZZa6UXo1JxFWbAtOjCwmpZJOXnJO+SMeKtyTj6Cxc51Wvi0GdJtE9a vxtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=/EOip1AIYHq0lLmjVJhDl5soNaKDqq6XOAGGb6F1q1o=; b=OHP0tBtcaWFiQrO3lIUw8GorQXcRy2J2iuXBxodMNrRgOvy1oR6w8uEIwhPAWnGECs 3tSFIzk6dqvfQZWcSTanouvlOOrCSLQlkFOrvPinX1T8m3Xt/osKw/vNsqGm5KOH884/ lGEeII1gtsRxZBtTDqbHILAd9f3q/7kqatqSxp2GfK4lu6qVl6IVtzDI75W6wqCnuk+S OGPUDQF0oQo9NjWiG1RlJbuF76nv19ZhDfANf2d4KpNiXH9DRiPNCyWp9ogVgZRJtAhu 8s7bTNrpBuJQWTuqKHffsYR241CKg37bAsP4pTJ2l/QHamvOV9rigFsfjJXwLbQiQKHs p21w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MZO1fAVB; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e61si7515995plb.524.2018.01.02.12.06.47; Tue, 02 Jan 2018 12:06:48 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MZO1fAVB; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751170AbeABUGp (ORCPT + 28 others); Tue, 2 Jan 2018 15:06:45 -0500 Received: from mail-wr0-f195.google.com ([209.85.128.195]:33536 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750787AbeABUGh (ORCPT ); Tue, 2 Jan 2018 15:06:37 -0500 Received: by mail-wr0-f195.google.com with SMTP id v21so38855767wrc.0 for ; Tue, 02 Jan 2018 12:06:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/EOip1AIYHq0lLmjVJhDl5soNaKDqq6XOAGGb6F1q1o=; b=MZO1fAVB5d8JRhoU/dxiG9mAm1vs47AzcIZm005LKWPWqltWMORB0h8IrpNjc6h1yM 3w8IlJB11um83+6PPjZLrCjdybt+umM2Irsf7x405cLMxNqpTDhTjxDr8h/PKA1Z+vnf 9C6rc9lcQI9Zpfy3xhy/B3RqgGlocDfqmj+8I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/EOip1AIYHq0lLmjVJhDl5soNaKDqq6XOAGGb6F1q1o=; b=fWqiJQ1ZaxWFU5FzHSMtHjAopoRxv//A4jZL34OOIwsh5NilMayxAa/WvoBWD2lSBr 72I9ZjiLKhAiNQP7MTlBZWruUvtZvjR9u2I9k8+ySWlryVQnWkVmIcjMXPZB7pM/a5U9 BSMR06FzWP5179gKhxzPqEKOqzbzlvv51KOwA8Lr/3imbIjv4NoMZT8tMi1KIQmr5Eli Jwts7cOJH876A6BAyb+7UngBkMU8g9zb/eMGdT8MKVrdRCGLscc1b+R5ZFD/56Q9pB03 iKLfAkaapw4+viwquHug7WnnGJD30Ko6bKiAJlmc8R3dtGofCtG4EnovF328iAzNBll5 a2ZQ== X-Gm-Message-State: AKGB3mKDb6rxdnFyRqNOtJoEpfeEJFPZpkat+MUeE78pAWGDXW/wpsw2 iYShNjP8FT6HNb7tSwdUdsihQ6K6E0c= X-Received: by 10.223.138.195 with SMTP id z3mr46074806wrz.152.1514923595658; Tue, 02 Jan 2018 12:06:35 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:34 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v7 04/10] init: allow initcall tables to be emitted using relative references Date: Tue, 2 Jan 2018 20:05:43 +0000 Message-Id: <20180102200549.22984-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Allow the initcall tables to be emitted using relative references that are only half the size on 64-bit architectures and don't require fixups at runtime on relocatable kernels. Cc: Petr Mladek Cc: Sergey Senozhatsky Cc: Steven Rostedt Cc: James Morris Cc: "Serge E. Hallyn" Signed-off-by: Ard Biesheuvel --- include/linux/init.h | 44 +++++++++++++++----- init/main.c | 32 +++++++------- kernel/printk/printk.c | 4 +- security/security.c | 4 +- 4 files changed, 53 insertions(+), 31 deletions(-) -- 2.11.0 diff --git a/include/linux/init.h b/include/linux/init.h index ea1b31101d9e..cef8e817e5a5 100644 --- a/include/linux/init.h +++ b/include/linux/init.h @@ -109,8 +109,24 @@ typedef int (*initcall_t)(void); typedef void (*exitcall_t)(void); -extern initcall_t __con_initcall_start[], __con_initcall_end[]; -extern initcall_t __security_initcall_start[], __security_initcall_end[]; +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +typedef int initcall_entry_t; + +static inline initcall_t initcall_from_entry(initcall_entry_t *entry) +{ + return (initcall_t)((unsigned long)entry + *entry); +} +#else +typedef initcall_t initcall_entry_t; + +static inline initcall_t initcall_from_entry(initcall_entry_t *entry) +{ + return *entry; +} +#endif + +extern initcall_entry_t __con_initcall_start[], __con_initcall_end[]; +extern initcall_entry_t __security_initcall_start[], __security_initcall_end[]; /* Used for contructor calls. */ typedef void (*ctor_fn_t)(void); @@ -160,9 +176,20 @@ extern bool initcall_debug; * as KEEP() in the linker script. */ -#define __define_initcall(fn, id) \ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#define ___define_initcall(fn, id, __sec) \ + __ADDRESSABLE(fn) \ + asm(".section \"" #__sec ".init\", \"a\" \n" \ + "__initcall_" #fn #id ": \n" \ + ".long " VMLINUX_SYMBOL_STR(fn) " - . \n" \ + ".previous \n"); +#else +#define ___define_initcall(fn, id, __sec) \ static initcall_t __initcall_##fn##id __used \ - __attribute__((__section__(".initcall" #id ".init"))) = fn; + __attribute__((__section__(#__sec ".init"))) = fn; +#endif + +#define __define_initcall(fn, id) ___define_initcall(fn, id, .initcall##id) /* * Early initcalls run before initializing SMP. @@ -201,13 +228,8 @@ extern bool initcall_debug; #define __exitcall(fn) \ static exitcall_t __exitcall_##fn __exit_call = fn -#define console_initcall(fn) \ - static initcall_t __initcall_##fn \ - __used __section(.con_initcall.init) = fn - -#define security_initcall(fn) \ - static initcall_t __initcall_##fn \ - __used __section(.security_initcall.init) = fn +#define console_initcall(fn) ___define_initcall(fn,, .con_initcall) +#define security_initcall(fn) ___define_initcall(fn,, .security_initcall) struct obs_kernel_param { const char *str; diff --git a/init/main.c b/init/main.c index a8100b954839..d81487cc126d 100644 --- a/init/main.c +++ b/init/main.c @@ -848,18 +848,18 @@ int __init_or_module do_one_initcall(initcall_t fn) } -extern initcall_t __initcall_start[]; -extern initcall_t __initcall0_start[]; -extern initcall_t __initcall1_start[]; -extern initcall_t __initcall2_start[]; -extern initcall_t __initcall3_start[]; -extern initcall_t __initcall4_start[]; -extern initcall_t __initcall5_start[]; -extern initcall_t __initcall6_start[]; -extern initcall_t __initcall7_start[]; -extern initcall_t __initcall_end[]; - -static initcall_t *initcall_levels[] __initdata = { +extern initcall_entry_t __initcall_start[]; +extern initcall_entry_t __initcall0_start[]; +extern initcall_entry_t __initcall1_start[]; +extern initcall_entry_t __initcall2_start[]; +extern initcall_entry_t __initcall3_start[]; +extern initcall_entry_t __initcall4_start[]; +extern initcall_entry_t __initcall5_start[]; +extern initcall_entry_t __initcall6_start[]; +extern initcall_entry_t __initcall7_start[]; +extern initcall_entry_t __initcall_end[]; + +static initcall_entry_t *initcall_levels[] __initdata = { __initcall0_start, __initcall1_start, __initcall2_start, @@ -885,7 +885,7 @@ static char *initcall_level_names[] __initdata = { static void __init do_initcall_level(int level) { - initcall_t *fn; + initcall_entry_t *fn; strcpy(initcall_command_line, saved_command_line); parse_args(initcall_level_names[level], @@ -895,7 +895,7 @@ static void __init do_initcall_level(int level) NULL, &repair_env_string); for (fn = initcall_levels[level]; fn < initcall_levels[level+1]; fn++) - do_one_initcall(*fn); + do_one_initcall(initcall_from_entry(fn)); } static void __init do_initcalls(void) @@ -926,10 +926,10 @@ static void __init do_basic_setup(void) static void __init do_pre_smp_initcalls(void) { - initcall_t *fn; + initcall_entry_t *fn; for (fn = __initcall_start; fn < __initcall0_start; fn++) - do_one_initcall(*fn); + do_one_initcall(initcall_from_entry(fn)); } /* diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c index b9006617710f..0516005261c7 100644 --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -2611,7 +2611,7 @@ EXPORT_SYMBOL(unregister_console); */ void __init console_init(void) { - initcall_t *call; + initcall_entry_t *call; /* Setup the default TTY line discipline. */ n_tty_init(); @@ -2622,7 +2622,7 @@ void __init console_init(void) */ call = __con_initcall_start; while (call < __con_initcall_end) { - (*call)(); + initcall_from_entry(call)(); call++; } } diff --git a/security/security.c b/security/security.c index 1cd8526cb0b7..f648eeff06de 100644 --- a/security/security.c +++ b/security/security.c @@ -45,10 +45,10 @@ static __initdata char chosen_lsm[SECURITY_NAME_MAX + 1] = static void __init do_security_initcalls(void) { - initcall_t *call; + initcall_entry_t *call; call = __security_initcall_start; while (call < __security_initcall_end) { - (*call) (); + initcall_from_entry(call)(); call++; } } From patchwork Tue Jan 2 20:05:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123224 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4763377edv; Tue, 2 Jan 2018 12:08:58 -0800 (PST) X-Google-Smtp-Source: ACJfBotwoDo2FeL/Dpje4gZqdLc7NRdUXzlc1rpmh6ZrcL4DFvifuSdFTBpQy0Hy9fFWPkUWnE1X X-Received: by 10.101.98.149 with SMTP id f21mr8136436pgv.53.1514923738241; Tue, 02 Jan 2018 12:08:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923738; cv=none; d=google.com; s=arc-20160816; b=NTgn6+63W9NZlM8LCNnTvUPG63Pc8e124bsAUFG542wsjkA66HHKo2GyZWqAwb8A9B MnRejWbHHtLFoHp7rBLYrMP0+F8ynbBZlhMe+rtwKbVJbfSntYE+Y+ADyqLZq170Z5PK GsCNVJLNQZKX1+NOtoDRMM42Wcq1YOf2Ztz4TI5hvI1IHHlS3qJPZZPccD1z0pWLmLQI VaJSEp8joHzUx2kHXR10LxI3EuEupW/pOlBnF20adlCKFUT66XwFf07rEHDpKd0VcChz QhDHD5BWqFo/CsayJ9MBgGe9Uj/HD749OQzIJy9O1MkVtupiXUtCKgM6mnnTkS8vzKzG 20FQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=mUgwtmE8XO4Hwp0PbFiB7DRxpjhYfUHYwRE5RAZQ5hI=; b=K8VAN29caXZNUGHd+nn0Xl2AJxIL691N/JaKNNwjJWT9hZGHZw6oj3+UQ4L60bxa/T Re3K1UlQVyCgid1vxmgfY1L25atrCQSV6d78mjmfvp0cKEYYgaqAr4dCFFaWItpYhB3o wfkV4iPCR937T/2tEo9dN5rLv563oGBbBfIg5Osm0if0pMyKkTlCxPYEYSNmYLpcqRhs BIsc1WWQwmgsdk9ImlRsEfRs/jviGQw4igXUbSQ4rW7tsohNRvIGjmYmZZ2Fq8fNmKBU ENPo2AmyLgd9muRTwPpuZ8DCo6brVKKEl+Qzfg5YRmDRNNZeVmmgfoprGzDYJO328ZM0 NKdw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XNQ7JrGE; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k2si18669613pgs.12.2018.01.02.12.08.57; Tue, 02 Jan 2018 12:08:58 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XNQ7JrGE; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751372AbeABUIz (ORCPT + 28 others); Tue, 2 Jan 2018 15:08:55 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:43663 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751149AbeABUGn (ORCPT ); Tue, 2 Jan 2018 15:06:43 -0500 Received: by mail-wr0-f193.google.com with SMTP id w68so37151929wrc.10 for ; Tue, 02 Jan 2018 12:06:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mUgwtmE8XO4Hwp0PbFiB7DRxpjhYfUHYwRE5RAZQ5hI=; b=XNQ7JrGEjWPy3PKFnPYalcrQuqtv85mlGCtacZb9rZGxQtaUrqUYOpVSk04jltsOTo fgM5xlh15AjaahM/p+1MDFZVbDp/gPRs3DX0Lgg14Wv04BBRNOWjAx6nzkyW26YW1qt+ 925exYfDt17jeoetEnjl9dHLTCAyudN7VOq2M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mUgwtmE8XO4Hwp0PbFiB7DRxpjhYfUHYwRE5RAZQ5hI=; b=AW59YdtJHqbCRaR3GqhgMsP+Vp4me2vBOJQyGpODFDuR2sVQQuPmSzVl88rerNUTLD tktvo/dJrRpqL4JcDbSqiHshEvviFpUEFpxdyaiSugyiBTrQYQUgUVG6bjHajhHDfaUe hp/FsAHDPi1AfLCwqOEtqVTjgyQpqTObCLhO0MKcixsLOURx8HznjiJbADf3YHaQD9QE jci6fTESZBfJK/xHpJpXlyeEyyFTlOWdfhLVYnelHfXD2Nqq9L2aiviarr2WDwRfsRbT RAD44YR4JRSTEvAN7TOI8S8qrfNYWoEsZ15dyvhPLdajoPJUVQi3pqGWn7grsfSCbWB5 KqsQ== X-Gm-Message-State: AKGB3mLmd1uYFZpZdFL8sMCCxx4vYoyPkjZCx+brlSGzumocuA5dPjDh 7/pzu80wzZIywb2zL8XntsNWRbYgm3E= X-Received: by 10.223.179.215 with SMTP id x23mr3214340wrd.137.1514923601860; Tue, 02 Jan 2018 12:06:41 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:41 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v7 05/10] PCI: Add support for relative addressing in quirk tables Date: Tue, 2 Jan 2018 20:05:44 +0000 Message-Id: <20180102200549.22984-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Allow the PCI quirk tables to be emitted in a way that avoids absolute references to the hook functions. This reduces the size of the entries, and, more importantly, makes them invariant under runtime relocation (e.g., for KASLR) Acked-by: Bjorn Helgaas Signed-off-by: Ard Biesheuvel --- drivers/pci/quirks.c | 13 ++++++++++--- include/linux/pci.h | 20 ++++++++++++++++++++ 2 files changed, 30 insertions(+), 3 deletions(-) -- 2.11.0 diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 10684b17d0bd..b6d51b4d5ce1 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3556,9 +3556,16 @@ static void pci_do_fixups(struct pci_dev *dev, struct pci_fixup *f, f->vendor == (u16) PCI_ANY_ID) && (f->device == dev->device || f->device == (u16) PCI_ANY_ID)) { - calltime = fixup_debug_start(dev, f->hook); - f->hook(dev); - fixup_debug_report(dev, calltime, f->hook); + void (*hook)(struct pci_dev *dev); +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + hook = (void *)((unsigned long)&f->hook_offset + + f->hook_offset); +#else + hook = f->hook; +#endif + calltime = fixup_debug_start(dev, hook); + hook(dev); + fixup_debug_report(dev, calltime, hook); } } diff --git a/include/linux/pci.h b/include/linux/pci.h index c170c9250c8b..086c3965710b 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1792,7 +1792,11 @@ struct pci_fixup { u16 device; /* You can use PCI_ANY_ID here of course */ u32 class; /* You can use PCI_ANY_ID here too */ unsigned int class_shift; /* should be 0, 8, 16 */ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + int hook_offset; +#else void (*hook)(struct pci_dev *dev); +#endif }; enum pci_fixup_pass { @@ -1806,12 +1810,28 @@ enum pci_fixup_pass { pci_fixup_suspend_late, /* pci_device_suspend_late() */ }; +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#define __DECLARE_PCI_FIXUP_SECTION(sec, name, vendor, device, class, \ + class_shift, hook) \ + __ADDRESSABLE(hook) \ + asm(".section " #sec ", \"a\" \n" \ + ".balign 16 \n" \ + ".short " #vendor ", " #device " \n" \ + ".long " #class ", " #class_shift " \n" \ + ".long " VMLINUX_SYMBOL_STR(hook) " - . \n" \ + ".previous \n"); +#define DECLARE_PCI_FIXUP_SECTION(sec, name, vendor, device, class, \ + class_shift, hook) \ + __DECLARE_PCI_FIXUP_SECTION(sec, name, vendor, device, class, \ + class_shift, hook) +#else /* Anonymous variables would be nice... */ #define DECLARE_PCI_FIXUP_SECTION(section, name, vendor, device, class, \ class_shift, hook) \ static const struct pci_fixup __PASTE(__pci_fixup_##name,__LINE__) __used \ __attribute__((__section__(#section), aligned((sizeof(void *))))) \ = { vendor, device, class, class_shift, hook }; +#endif #define DECLARE_PCI_FIXUP_CLASS_EARLY(vendor, device, class, \ class_shift, hook) \ From patchwork Tue Jan 2 20:05:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123218 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4761342edv; Tue, 2 Jan 2018 12:06:55 -0800 (PST) X-Google-Smtp-Source: ACJfBosZdD2I9W1RHOFbz2QB3qL1tuN7J6BQgXMkovQUnm3pzsjBHEYAoCVBDZLgjEdh1Oye4pav X-Received: by 10.98.152.147 with SMTP id d19mr45668237pfk.95.1514923615194; Tue, 02 Jan 2018 12:06:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923615; cv=none; d=google.com; s=arc-20160816; b=Gw0zbwLbo/GhOxn4/3P4JdVT4RRQRnlWyRfBkDhOV/e6BNYGV9ruh6sr5QblqcqKl8 2bG4zKJ6j+t4ArzYQ0pMo7DXg+AOC5AImM0M27KMmqlD2uVJogyh9G782VZnkhtwx8V4 yTQ2TQVsTY1Qw2mZzpkI7S+qW+lRXe/Ftj77EWLKEfdsfuHD+V1BQARwIvj/aHXX1gUt MK0+mmwlRlq2GR8/OOAJovMttJwxx8jiEFI+JfxjEhiUJ8ZglrOi20gp3f98zmxPR0vh CW6wfyrpj7KvFkyH5RXC29SWHLJsPYy1sHRJnUZQsLHlcJyNZvK8rZZrMra24e4MbqpR nUgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=wKZwhkbHcAnKwdhVFtFBF5H65qthlsMyQbywDqRcj68=; b=fc/Tv34DxQ/NQpUiLYkG7NrTqkBi1aPMLADxMOsGgJEN05663nOj+KBIGO3M3yvma6 cAPnweB/7HdVqbTmgj1+WVXy5Fgs5vJfc1GxisoW2Vr6//FYNolX6VYEpAAX7RbCYva7 YS/+7TYu3C6s9znGXskw/vkFoK/f+uWZAgchb/PXY2Y9UHSSODzTY8XQZAkg7IcADg66 af4N1T/h+XapfGSGP/ITxs0s7oWphENfMHhx4Gb0ySOA7q2qN1ydcorBWsIQ6+VEVxSR n1HMDjQsKpNzH24Xwhn/jSXduB0JhqNF6dvgbfypMommi1aoDmnNLLPHNT/3/13U7g5k KaMw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OTP3P2Q0; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e61si7515995plb.524.2018.01.02.12.06.54; Tue, 02 Jan 2018 12:06:55 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OTP3P2Q0; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751190AbeABUGx (ORCPT + 28 others); Tue, 2 Jan 2018 15:06:53 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:37942 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751008AbeABUGt (ORCPT ); Tue, 2 Jan 2018 15:06:49 -0500 Received: by mail-wr0-f196.google.com with SMTP id y9so7456929wrb.5 for ; Tue, 02 Jan 2018 12:06:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wKZwhkbHcAnKwdhVFtFBF5H65qthlsMyQbywDqRcj68=; b=OTP3P2Q0+30/oFxHDyVu3M4GplpoAAATn7kHNcGIe9IokkX7drMcUm7be/KmZtGjZD WsazG+awbUfxpOYDUSYbOhoZlElVtzJpO670EGPlB/IE9QlNm18N9TnV/I2WtyXjIXNO mkB3y33bzP1G9PF+9/zy9mGfxkST/iRm29Obo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wKZwhkbHcAnKwdhVFtFBF5H65qthlsMyQbywDqRcj68=; b=RH/6LY/6serXwC3OGEnur3oyLLZtzmu/7J/4k+VKn/FL+i2Wa3wSZIgV7NfnWgN3BB +VApvmklVIJAqAaYJTY64chyY3pOBifZX98ImZ+11+CDE554rm4zEsCBUqG+0hI/WbZt VStHKcZY0p5MwDm+7RdCP33QsQabEP8UcjxdvMqq4673UQve0rARv2VpXIcq/gbKqdnZ JDvLkunGUgFiPAvfvaotRbyvQL3J7UFLEybsJygrRbjP/hVXtQq0X5yAfvEcBD6Er8OH vjwIbH/9EM4fl+AFj6Y0iSzJ9APCiQrLaLbw1eGIrH+2PPiONT1l7qU3jjkB3RQmGv5d GqxQ== X-Gm-Message-State: AKGB3mIMy6+8BXEvXcdF5ZrZqEUaE4/O6654prxiGnQ2eQD2BxRA45mT +Nlgq4oznaaE+C66owVvIHQLfgJN4AQ= X-Received: by 10.223.138.195 with SMTP id z3mr46075191wrz.152.1514923607843; Tue, 02 Jan 2018 12:06:47 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:47 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v7 06/10] kernel: tracepoints: add support for relative references Date: Tue, 2 Jan 2018 20:05:45 +0000 Message-Id: <20180102200549.22984-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org To avoid the need for relocating absolute references to tracepoint structures at boot time when running relocatable kernels (which may take a disproportionate amount of space), add the option to emit these tables as relative references instead. Cc: Ingo Molnar Acked-by: Steven Rostedt (VMware) Signed-off-by: Ard Biesheuvel --- include/linux/tracepoint.h | 19 ++++++-- kernel/tracepoint.c | 50 +++++++++++--------- 2 files changed, 42 insertions(+), 27 deletions(-) -- 2.11.0 diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index a26ffbe09e71..d02bf1a695e8 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -228,6 +228,19 @@ extern void syscall_unregfunc(void); return static_key_false(&__tracepoint_##name.key); \ } +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#define __TRACEPOINT_ENTRY(name) \ + asm(" .section \"__tracepoints_ptrs\", \"a\" \n" \ + " .balign 4 \n" \ + " .long " VMLINUX_SYMBOL_STR(__tracepoint_##name) " - .\n" \ + " .previous \n") +#else +#define __TRACEPOINT_ENTRY(name) \ + static struct tracepoint * const __tracepoint_ptr_##name __used \ + __attribute__((section("__tracepoints_ptrs"))) = \ + &__tracepoint_##name +#endif + /* * We have no guarantee that gcc and the linker won't up-align the tracepoint * structures, so we create an array of pointers that will be used for iteration @@ -237,11 +250,9 @@ extern void syscall_unregfunc(void); static const char __tpstrtab_##name[] \ __attribute__((section("__tracepoints_strings"))) = #name; \ struct tracepoint __tracepoint_##name \ - __attribute__((section("__tracepoints"))) = \ + __attribute__((section("__tracepoints"), used)) = \ { __tpstrtab_##name, STATIC_KEY_INIT_FALSE, reg, unreg, NULL };\ - static struct tracepoint * const __tracepoint_ptr_##name __used \ - __attribute__((section("__tracepoints_ptrs"))) = \ - &__tracepoint_##name; + __TRACEPOINT_ENTRY(name); #define DEFINE_TRACE(name) \ DEFINE_TRACE_FN(name, NULL, NULL); diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 685c50ae6300..05649fef106c 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -327,6 +327,28 @@ int tracepoint_probe_unregister(struct tracepoint *tp, void *probe, void *data) } EXPORT_SYMBOL_GPL(tracepoint_probe_unregister); +static void for_each_tracepoint_range(struct tracepoint * const *begin, + struct tracepoint * const *end, + void (*fct)(struct tracepoint *tp, void *priv), + void *priv) +{ + if (!begin) + return; + + if (IS_ENABLED(CONFIG_HAVE_ARCH_PREL32_RELOCATIONS)) { + const int *iter; + + for (iter = (const int *)begin; iter < (const int *)end; iter++) + fct((struct tracepoint *)((unsigned long)iter + *iter), + priv); + } else { + struct tracepoint * const *iter; + + for (iter = begin; iter < end; iter++) + fct(*iter, priv); + } +} + #ifdef CONFIG_MODULES bool trace_module_has_bad_taint(struct module *mod) { @@ -391,15 +413,9 @@ EXPORT_SYMBOL_GPL(unregister_tracepoint_module_notifier); * Ensure the tracer unregistered the module's probes before the module * teardown is performed. Prevents leaks of probe and data pointers. */ -static void tp_module_going_check_quiescent(struct tracepoint * const *begin, - struct tracepoint * const *end) +static void tp_module_going_check_quiescent(struct tracepoint *tp, void *priv) { - struct tracepoint * const *iter; - - if (!begin) - return; - for (iter = begin; iter < end; iter++) - WARN_ON_ONCE((*iter)->funcs); + WARN_ON_ONCE(tp->funcs); } static int tracepoint_module_coming(struct module *mod) @@ -450,8 +466,9 @@ static void tracepoint_module_going(struct module *mod) * Called the going notifier before checking for * quiescence. */ - tp_module_going_check_quiescent(mod->tracepoints_ptrs, - mod->tracepoints_ptrs + mod->num_tracepoints); + for_each_tracepoint_range(mod->tracepoints_ptrs, + mod->tracepoints_ptrs + mod->num_tracepoints, + tp_module_going_check_quiescent, NULL); break; } } @@ -503,19 +520,6 @@ static __init int init_tracepoints(void) __initcall(init_tracepoints); #endif /* CONFIG_MODULES */ -static void for_each_tracepoint_range(struct tracepoint * const *begin, - struct tracepoint * const *end, - void (*fct)(struct tracepoint *tp, void *priv), - void *priv) -{ - struct tracepoint * const *iter; - - if (!begin) - return; - for (iter = begin; iter < end; iter++) - fct(*iter, priv); -} - /** * for_each_kernel_tracepoint - iteration on all kernel tracepoints * @fct: callback From patchwork Tue Jan 2 20:05:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123219 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4761525edv; Tue, 2 Jan 2018 12:07:04 -0800 (PST) X-Google-Smtp-Source: ACJfBotTRm1nNWsEuhWgTTCLE66T29PvPpmGI89XdxLK39UDr38E6BOvaMZBaRNVk1VbnVy+4ZsQ X-Received: by 10.98.162.20 with SMTP id m20mr46597846pff.6.1514923624548; Tue, 02 Jan 2018 12:07:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923624; cv=none; d=google.com; s=arc-20160816; b=s1Xm6+bMZQhouOwoHugfBg2UHYHi0aZe/geV06gIEvEFpgW93l2xmiiKhbhgmlQF3+ sB6KHIQ8U10RXdTUDMk++etLYU6UVratBHJoUQZ9Z26PAjhAKGSWnH8fhKbMqBWh0gN6 4lwWRZlkVCiF/K0Vfdnp6xQcIm+W9jq9l18gdculZWUYBsaHR+1DxxdHSsHRGPNJds9L Q/fBzJ9NbbvrQrdaZHA+epPLIE7XYf0gsCV5CrqF9DnjSa4Mwsk8piGZmqDFbH3ra5QZ KUzOTU1UPi2MKlZQGorU1G5vcM+gebXYkHGLFr2X2hdDvXKvv0WPuUgnWchQOa10QHxR uDuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=vJVJOehG9ztW1uyWcobdBB+SDC9ShNv9yhy7/gSRq5c=; b=JrR4o2TIqxUUE3xAm3ip+AYxqR4ONGHwmQHelrUcjbOqhNil3yvdExZEGTYTNb5fQm gHDhBxvikoaa/aaxDxcxdu5c80GsSC8Z8omtN40WvsnvanaZtOueDYrR3i5E1A/ALGun yBSFULrSY+FWnOytaOhSaEIhgcRcPvdfqAODIAtopWx9E7ClSGhEUpz7ZPv39Dwpo/cS oY5x1kR73NIkViAOhKDJzKJZvZf3YIR11Gva2e9beLTunNfQubkmi/0adF1kfRMhTpdc 0yNtX22pHmsAe6IvGOmaLC2/EvU5EMFN437IeNcg4vDaSej1gJ7G4J9iRJh7WZe/68LC UbHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jeEb6ZB0; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y10si1657982pgp.662.2018.01.02.12.07.04; Tue, 02 Jan 2018 12:07:04 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jeEb6ZB0; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751208AbeABUHB (ORCPT + 28 others); Tue, 2 Jan 2018 15:07:01 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:42437 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751008AbeABUG5 (ORCPT ); Tue, 2 Jan 2018 15:06:57 -0500 Received: by mail-wr0-f196.google.com with SMTP id w107so23755888wrb.9 for ; Tue, 02 Jan 2018 12:06:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vJVJOehG9ztW1uyWcobdBB+SDC9ShNv9yhy7/gSRq5c=; b=jeEb6ZB0z2/zzzc3KaVGNrlfvf6s5dpCO+sLMLIshA7WAe6n5wRQLi93dLFKC0U/t1 RtaYo5PCH71xiMnB7L/+HADgXxGrsLlzKVHDKbSjZPJnUWWsvvpW9Z8YPBuR7MaKdEci dlLKq+Zd88nyl8pO4RVNbuEc9E8VLOxWlUvT0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vJVJOehG9ztW1uyWcobdBB+SDC9ShNv9yhy7/gSRq5c=; b=WzXVBw4wsUtHwCdzl3zSAJyKiI+m173AUM3cbGBtQ4F2FQfzA/YbznU4mgHxeW+T0C BlSbw3g7VVl461sBJGussPe0C3C3JAMB3BBtcQKNHi2HQDeEi3ORzWfIIRn/rLIz4uvO TfdID9pMlRa1aR1iNd0uetHmH0GI4Dzwm2mN4Tv8UcJ0cc1bBveJqPGCCQKs8iDbJEwf gIoul9VX1/ix2b971z5Mo6dp/Np0HbYuUQy0Qk4C/SZxS1I98TrCRIOgC3XrWmbOFAcR Sb3pQY9s18Yz4aydIbp+dc20I4EySjuZ/LF+QpbBPHRM8UiemFNoiYv9DOWGZ6IoXyXg XYdA== X-Gm-Message-State: AKGB3mJoLWW9PiDNtjR4/Tz5kwU3aDlK6Xz4d5o5T7snPqFMzLLMV0cJ Ct5JW9+5Z1Xlz3gmzd8GScD9ZMMtuX8= X-Received: by 10.223.185.85 with SMTP id b21mr42063291wrg.264.1514923615474; Tue, 02 Jan 2018 12:06:55 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:54 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v7 07/10] kernel/jump_label: abstract jump_entry member accessors Date: Tue, 2 Jan 2018 20:05:46 +0000 Message-Id: <20180102200549.22984-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation of allowing architectures to use relative references in jump_label entries [which can dramatically reduce the memory footprint], introduce abstractions for references to the 'code' and 'key' members of struct jump_entry. Signed-off-by: Ard Biesheuvel --- arch/arm/include/asm/jump_label.h | 27 ++++++++++++++ arch/arm64/include/asm/jump_label.h | 27 ++++++++++++++ arch/mips/include/asm/jump_label.h | 27 ++++++++++++++ arch/powerpc/include/asm/jump_label.h | 27 ++++++++++++++ arch/s390/include/asm/jump_label.h | 27 ++++++++++++++ arch/sparc/include/asm/jump_label.h | 27 ++++++++++++++ arch/tile/include/asm/jump_label.h | 27 ++++++++++++++ arch/x86/include/asm/jump_label.h | 27 ++++++++++++++ kernel/jump_label.c | 38 +++++++++----------- 9 files changed, 232 insertions(+), 22 deletions(-) -- 2.11.0 diff --git a/arch/arm/include/asm/jump_label.h b/arch/arm/include/asm/jump_label.h index e12d7d096fc0..7b05b404063a 100644 --- a/arch/arm/include/asm/jump_label.h +++ b/arch/arm/include/asm/jump_label.h @@ -45,5 +45,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 1b5e0e843c3a..9d6e46355c89 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -62,5 +62,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/mips/include/asm/jump_label.h b/arch/mips/include/asm/jump_label.h index e77672539e8e..70df9293dc49 100644 --- a/arch/mips/include/asm/jump_label.h +++ b/arch/mips/include/asm/jump_label.h @@ -66,5 +66,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif /* _ASM_MIPS_JUMP_LABEL_H */ diff --git a/arch/powerpc/include/asm/jump_label.h b/arch/powerpc/include/asm/jump_label.h index 9a287e0ac8b1..412b2699c9f6 100644 --- a/arch/powerpc/include/asm/jump_label.h +++ b/arch/powerpc/include/asm/jump_label.h @@ -59,6 +59,33 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #else #define ARCH_STATIC_BRANCH(LABEL, KEY) \ 1098: nop; \ diff --git a/arch/s390/include/asm/jump_label.h b/arch/s390/include/asm/jump_label.h index 40f651292aa7..1ecfd46835d9 100644 --- a/arch/s390/include/asm/jump_label.h +++ b/arch/s390/include/asm/jump_label.h @@ -50,5 +50,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/sparc/include/asm/jump_label.h b/arch/sparc/include/asm/jump_label.h index 94eb529dcb77..18e893687f7c 100644 --- a/arch/sparc/include/asm/jump_label.h +++ b/arch/sparc/include/asm/jump_label.h @@ -48,5 +48,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/tile/include/asm/jump_label.h b/arch/tile/include/asm/jump_label.h index cde7573f397b..86acaa6ff33d 100644 --- a/arch/tile/include/asm/jump_label.h +++ b/arch/tile/include/asm/jump_label.h @@ -55,4 +55,31 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* _ASM_TILE_JUMP_LABEL_H */ diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 8c0de4282659..009ff2699d07 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -74,6 +74,33 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #else /* __ASSEMBLY__ */ .macro STATIC_JUMP_IF_TRUE target, key, def diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 8594d24e4adc..4f44db58d981 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -37,10 +37,12 @@ static int jump_label_cmp(const void *a, const void *b) const struct jump_entry *jea = a; const struct jump_entry *jeb = b; - if (jea->key < jeb->key) + if ((unsigned long)jump_entry_key(jea) < + (unsigned long)jump_entry_key(jeb)) return -1; - if (jea->key > jeb->key) + if ((unsigned long)jump_entry_key(jea) > + (unsigned long)jump_entry_key(jeb)) return 1; return 0; @@ -53,7 +55,8 @@ jump_label_sort_entries(struct jump_entry *start, struct jump_entry *stop) size = (((unsigned long)stop - (unsigned long)start) / sizeof(struct jump_entry)); - sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL); + sort(start, size, sizeof(struct jump_entry), jump_label_cmp, + jump_label_swap); } static void jump_label_update(struct static_key *key); @@ -254,8 +257,8 @@ EXPORT_SYMBOL_GPL(jump_label_rate_limit); static int addr_conflict(struct jump_entry *entry, void *start, void *end) { - if (entry->code <= (unsigned long)end && - entry->code + JUMP_LABEL_NOP_SIZE > (unsigned long)start) + if (jump_entry_code(entry) <= (unsigned long)end && + jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE > (unsigned long)start) return 1; return 0; @@ -314,16 +317,6 @@ static inline void static_key_set_linked(struct static_key *key) key->type |= JUMP_TYPE_LINKED; } -static inline struct static_key *jump_entry_key(struct jump_entry *entry) -{ - return (struct static_key *)((unsigned long)entry->key & ~1UL); -} - -static bool jump_entry_branch(struct jump_entry *entry) -{ - return (unsigned long)entry->key & 1UL; -} - /*** * A 'struct static_key' uses a union such that it either points directly * to a table of 'struct jump_entry' or to a linked list of modules which in @@ -348,7 +341,7 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool enabled = static_key_enabled(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return enabled ^ branch; @@ -364,7 +357,8 @@ static void __jump_label_update(struct static_key *key, * kernel_text_address() verifies we are not in core kernel * init code, see jump_label_invalidate_module_init(). */ - if (entry->code && kernel_text_address(entry->code)) + if (!jump_entry_is_module_init(entry) && + kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); } } @@ -417,7 +411,7 @@ static enum jump_label_type jump_label_init_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool type = static_key_type(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return type ^ branch; @@ -541,7 +535,7 @@ static int jump_label_add_module(struct module *mod) continue; key = iterk; - if (within_module(iter->key, mod)) { + if (within_module((unsigned long)key, mod)) { static_key_set_entries(key, iter); continue; } @@ -591,7 +585,7 @@ static void jump_label_del_module(struct module *mod) key = jump_entry_key(iter); - if (within_module(iter->key, mod)) + if (within_module((unsigned long)key, mod)) continue; /* No memory during module load */ @@ -634,8 +628,8 @@ static void jump_label_invalidate_module_init(struct module *mod) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(iter->code, mod)) - iter->code = 0; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_module_init(iter); } } From patchwork Tue Jan 2 20:05:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123220 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4761722edv; Tue, 2 Jan 2018 12:07:15 -0800 (PST) X-Google-Smtp-Source: ACJfBovn4uoRlcNEB8HvHfcKfh8PiFsMCT5TePStZBSbU86UKSGBFN9HK9vqdveWEEbR8mTY14tG X-Received: by 10.98.204.10 with SMTP id a10mr46927460pfg.18.1514923635507; Tue, 02 Jan 2018 12:07:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923635; cv=none; d=google.com; s=arc-20160816; b=AwvboGg1tP1QcdgJLfT/eGQDgWLDZdzKGwuZZY+tftkTHe2BAgXrloqBRvt9yhhiY3 p590MuKSfi9Qa2DrVdJV+AtSALToq8HE1g4EeZxo8Jm1pTDjIg7iz3+yFR/HYF787tyZ yjuF7qrGvtIlQr1MQJulPKE7v25waNmIvDwbsT8Mhn3r0DiUr+fK6KFILYmO9FHMrXVb XaO4f6ZSFEhf0SR8OMjDbj0s1FTQx84WASe8qZLDrBQX6Q9xSxhXItr0BfLzZtnhg/7a Eg5TEbHpaDIAl1PCGHq87FEAlXOO1iqvHJELRC/ZWzX2towYXWksT12z4ffA4fGN8T6N Ic9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=RtgdmqqE7QnNDS0Od8ctH/UMcFkuxkhGpZ1L+nRoK9Y=; b=gJ4aQfu2bbULtKjDsIqPPto1HJQ9F6CzzsLynWJ+ltMdEEtm4QgBYJT9aW4N4Z+Cc5 C/ceHeyxbliT/QXD5b13mAc0umZrpB9PyANDzsE9RT8gfGyBkdpsibfnMg+EPzJXNLYT zpOvnx58LBYj6EkvM03PkqJcO22jvb57xdN1DZgJDDYxXmIieoap164+rkE0ND/y+sCi jjJIl/NJYTF2WTcexvAiqP6/2mwmy2jB4Wi+TPGGdg1272Q6ZX+AqTzn4j7LVnDPTvaP ea5P+1YLF6pQBqVIR9mZRtXgKXiWXbJkQED28RotTSQmMpu/AhjUjiFHAhMG6ZaWT3ns a87g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ZXOsFAJi; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q17si34367969pfj.31.2018.01.02.12.07.14; Tue, 02 Jan 2018 12:07:15 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ZXOsFAJi; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751244AbeABUHK (ORCPT + 28 others); Tue, 2 Jan 2018 15:07:10 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:35190 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751211AbeABUHD (ORCPT ); Tue, 2 Jan 2018 15:07:03 -0500 Received: by mail-wr0-f193.google.com with SMTP id l19so37219715wrc.2 for ; Tue, 02 Jan 2018 12:07:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RtgdmqqE7QnNDS0Od8ctH/UMcFkuxkhGpZ1L+nRoK9Y=; b=ZXOsFAJioLobdvr9YpnYOCxgnNq32bEakvz+d8oHqbR+rWFbkbaVPtVyawqeeowoT1 Dzbj11m2Bi/29hzMeufYCDd4u72LhfE0tCsxWkG9HEh2S9NwOvXB6vLZxGTzpuOUsPI5 3M8pDezCKUo4tMXGIDfUWJ0EZBHr/1jyyzjGc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RtgdmqqE7QnNDS0Od8ctH/UMcFkuxkhGpZ1L+nRoK9Y=; b=A5levprkxsGP3pK4iNhyjpHLbjRcip8t/TDO3pi1qKW3olyW46jyTG3wVVt2IZ1TVG C6+0eFArSvY5yNazItrSKd6Y97TQhqCWGwz6JST13MtFhTMMHENFbj+UOo4j6qTu3c5z yr3UFSc7VpkzUMtpoY+vNxsyX79EejqsDgE7PZBESZ315eBHaOi+Z3EVjslnTxP+TSlm /3yMyXFJYAr/8/0O6kslJifVXdCX6Uz7dFAt23cF+cbAlSUXUTwXV1HYwtHRxCqy6nug 9mF8dtX9xsBAIPlE1lc29jjOEfA2m7IAZbZmQnbv2v9oQA2tYWaRNGDr44ylhG3k1s6c s0FQ== X-Gm-Message-State: AKGB3mKZ32b10JOzmW7yIvuA3YE9w2QUYsQxGWugG3Kq2yq+rGByVug9 xUahPpLyyYbxkObdyAFiLCHNUZ9LxcA= X-Received: by 10.223.186.13 with SMTP id o13mr15747922wrg.43.1514923622162; Tue, 02 Jan 2018 12:07:02 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:07:01 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v7 08/10] arm64/kernel: jump_label: use relative references Date: Tue, 2 Jan 2018 20:05:47 +0000 Message-Id: <20180102200549.22984-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On a randomly chosen distro kernel build for arm64, vmlinux.o shows the following sections, containing jump label entries, and the associated RELA relocation records, respectively: ... [38088] __jump_table PROGBITS 0000000000000000 00e19f30 000000000002ea10 0000000000000000 WA 0 0 8 [38089] .rela__jump_table RELA 0000000000000000 01fd8bb0 000000000008be30 0000000000000018 I 38178 38088 8 ... In other words, we have 190 KB worth of 'struct jump_entry' instances, and 573 KB worth of RELA entries to relocate each entry's code, target and key members. This means the RELA section occupies 10% of the .init segment, and the two sections combined represent 5% of vmlinux's entire memory footprint. So let's switch from 64-bit absolute references to 32-bit relative references: this reduces the size of the __jump_table by 50%, and gets rid of the RELA section entirely. Note that this requires some extra care in the sorting routine, given that the offsets change when entries are moved around in the jump_entry table. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/jump_label.h | 27 ++++++++++++-------- arch/arm64/kernel/jump_label.c | 22 +++++++++++++--- 2 files changed, 36 insertions(+), 13 deletions(-) -- 2.11.0 diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 9d6e46355c89..8f82adeb7b0b 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -30,8 +30,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm goto("1: nop\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -44,8 +44,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool { asm goto("1: b %l[l_yes]\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -57,19 +57,26 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool typedef u64 jump_label_t; struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (unsigned long)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (unsigned long)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -87,7 +94,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index c2dd1ad3e648..2b8e459e91f7 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -25,12 +25,12 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - void *addr = (void *)entry->code; + void *addr = (void *)jump_entry_code(entry); u32 insn; if (type == JUMP_LABEL_JMP) { - insn = aarch64_insn_gen_branch_imm(entry->code, - entry->target, + insn = aarch64_insn_gen_branch_imm(jump_entry_code(entry), + jump_entry_target(entry), AARCH64_INSN_BRANCH_NOLINK); } else { insn = aarch64_insn_gen_nop(); @@ -50,4 +50,20 @@ void arch_jump_label_transform_static(struct jump_entry *entry, */ } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif /* HAVE_JUMP_LABEL */ From patchwork Tue Jan 2 20:05:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123221 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4761848edv; Tue, 2 Jan 2018 12:07:21 -0800 (PST) X-Google-Smtp-Source: ACJfBou7HnPvqQx6ODe+p9hX/53JNnfKqR+dIJhMZ6rJcPhEmIhV9djfLIfiSy7gwwmwNJP/UGUt X-Received: by 10.99.145.68 with SMTP id l65mr9070831pge.204.1514923641254; Tue, 02 Jan 2018 12:07:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923641; cv=none; d=google.com; s=arc-20160816; b=Mjg9+PcRzX/Z1wCO/ZMrzHTqAzHlBAMTf0n7J3Jm0qSj2fc1m+mZgFwJ1VoeW0egkB np7YRoQQv3003pSQsH2Ctoq9Jta/TfianlJ/HY3GrHH4c5Ze7eDt6ISNOKSIT8nS6+MO 2LIoPddRykMk6PcEgFKPSplRuUFSAoLklkWp+dZLRLOoGMicG22ev1BbKyLJOqgOsX7/ Jue/2pfqxP3xKM0huuhh1PFcNkFM052UFn/r+XR4Cyxil6nJzWeBn5e6GZJxxmkcTXaR z40ZacOLkqA2DtgCGbcpiyKHySD7kiImUmyUIVtnjxDHMuiiUZRxlfaU8lp6RgQZ/9+j WGTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=3xW6hB18H7nCj1B8KM19meDl2NukzrBIHv9vfi9d94c=; b=mo1pMdxI0mqSAxg/jg5AHjZOm0YFIAoAh6Db6/HwYwXe1BHA/beAM2zEGyu/T7J9+J E+sWpN/HsCBOvikwDGvW4L9bZjOIBIiYd8kwRVqqwfVRSFosFc3wya4KXEaFC4VYfAG8 aHcvrnZKcoxCdOkTWkWvbokTV5FhJfYkiki7Poin9v3QP3ZLx1ZiJOiLObIVxkgT4Fpo CoQby1XwXG0ng7wn5SyrSaq5wmB9Nnn+MURSI0rArO21PDTYlCsoxgvH1KE1Vlw5jdp1 5dJ/R+TGPh1/l6Pt4bEMcDBYOB87HnDF0/AgdE6ngz7xim8Y2VA2ec4pRh1dJR0aPfjJ l5xQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IlsjC4yq; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p14si34181770pli.392.2018.01.02.12.07.20; Tue, 02 Jan 2018 12:07:21 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IlsjC4yq; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751266AbeABUHN (ORCPT + 28 others); Tue, 2 Jan 2018 15:07:13 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:34244 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751234AbeABUHL (ORCPT ); Tue, 2 Jan 2018 15:07:11 -0500 Received: by mail-wr0-f196.google.com with SMTP id 36so16291021wrh.1 for ; Tue, 02 Jan 2018 12:07:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3xW6hB18H7nCj1B8KM19meDl2NukzrBIHv9vfi9d94c=; b=IlsjC4yqku/Bgvxd3dSYA80TsqwMeZzyABWwlbxe827nworF3Z2Xw2NJJ9GC5u1EVj TlmJPcYtP0p5eP2s5u9UB06rVtkb1x+/J7q2wqmbSLsSc3bgNNB1+bkZWxdCYlujbUoP GxFZ5Jka238l0BCJThgTwy43cQ6Jj89D4IPmg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3xW6hB18H7nCj1B8KM19meDl2NukzrBIHv9vfi9d94c=; b=TUd05nRPCGFQbvPwAgmidrAxF990jTUM9HeynmWW1BC2SG9WvJOV/uo++XknQuneMf CvLJBVvfW1QAyq6cXZgqGLLeQwbBjBR4l3sfVTJ5005Uz9W/UWCmaaFQXRAQxp32Gnmg thpuVwi2QTdZBSjTjIHwKQipSQef/fn2DVf7QPymmU63g1OPfcKh4d9Lxlh7ZJGIeky8 m2DYd416FsQNrvCA0bKdhAbf6YEAB6Db0N1GUSjLgEWwTKfadVYS//wc6doIsvtOzgDS lZEQmrzBfyY4yGd9SoqvU6X9mFGrW4cX+ov2qgRaCOLIY6gF6JWpP2y4N4RnyxTPEwDN tzAQ== X-Gm-Message-State: AKGB3mJYJCMyzztO+4rqo9NARtePUCMDCaLXKxPc7BHdWa6uL1zKOgZH Fu2PXXksNQjm3CMpsi5rGJqXBT1A/nY= X-Received: by 10.223.177.143 with SMTP id q15mr14341520wra.42.1514923629540; Tue, 02 Jan 2018 12:07:09 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.07.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:07:08 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v7 09/10] x86: jump_label: switch to jump_entry accessors Date: Tue, 2 Jan 2018 20:05:48 +0000 Message-Id: <20180102200549.22984-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation of switching x86 to use place-relative references for the code, target and key members of struct jump_entry, replace direct references to the struct member with invocations of the new accessors. This will allow us to make the switch by modifying the accessors only. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/jump_label.c | 43 ++++++++++++-------- 1 file changed, 26 insertions(+), 17 deletions(-) -- 2.11.0 diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e56c95be2808..d64296092ef5 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -52,22 +52,24 @@ static void __jump_label_transform(struct jump_entry *entry, * Jump label is enabled for the first time. * So we expect a default_nop... */ - if (unlikely(memcmp((void *)entry->code, default_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { /* * ...otherwise expect an ideal_nop. Otherwise * something went horribly wrong. */ - if (unlikely(memcmp((void *)entry->code, ideal_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + ideal_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); } else { /* * We are disabling this jump label. If it is not what @@ -76,14 +78,18 @@ static void __jump_label_transform(struct jump_entry *entry, * are converting the default nop to the ideal nop. */ if (init) { - if (unlikely(memcmp((void *)entry->code, default_nop, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); - if (unlikely(memcmp((void *)entry->code, &code, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); + if (unlikely(memcmp((void *)jump_entry_code(entry), + &code, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } memcpy(&code, ideal_nops[NOP_ATOMIC5], JUMP_LABEL_NOP_SIZE); } @@ -97,10 +103,13 @@ static void __jump_label_transform(struct jump_entry *entry, * */ if (poker) - (*poker)((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE); + (*poker)((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE); else - text_poke_bp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE, - (void *)entry->code + JUMP_LABEL_NOP_SIZE); + text_poke_bp((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE, + (void *)jump_entry_code(entry) + + JUMP_LABEL_NOP_SIZE); } void arch_jump_label_transform(struct jump_entry *entry, From patchwork Tue Jan 2 20:05:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 123222 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp4762003edv; Tue, 2 Jan 2018 12:07:30 -0800 (PST) X-Google-Smtp-Source: ACJfBouwH1NHE9Gte40rCdzPGNitR/4XCLFmUz0oi6NhfMpcd8i9uxSRFcbEEzViy7RvbgXnUqx3 X-Received: by 10.99.96.69 with SMTP id u66mr41460298pgb.355.1514923650414; Tue, 02 Jan 2018 12:07:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514923650; cv=none; d=google.com; s=arc-20160816; b=OZf80qEu+QJnrThzvYhQI/m6QcZHuseVGg6GVsXQvIWs0EydsMj5F+M16VSGJNclvC 7nbAnHBmlcOgse6+b5SpuSece5g3vx1KMOishGhR4aWd3j82APSuI+CcOufZTNvzqr0P KpZ/IzH/Toorw0lcy5PkJbbbmHQNeQyl9P57dFHHq9y4xuB9zDrMCj/3Wl/fey4UYRXv iWnlb9nGTRzpO6cOYHUaiewjZh2uXpw0hPpEn+VWJvQqz1QnoIGpCVfmr234XIqu9RqP jNY1d5dsL7iDLt7AHZ9Ch4IPK+1MsUGdoFmOP+XXW4epNA1yhe2kMeLFXTIbO0bMQhQK yIng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=y3cZHthDjrZdZe+7+thZcDtJZzDXcVyA3ettxQyn2/I=; b=uUO9VRgaaMS1P1teJv/TNcOikdjOPO0Sp2ml5zwQMu3KRRzonfvYeJU8GkvunU9aJL /knZEHfhEFcPValhuOup3Zocve9sutmbO04dcGKkWDaBJ55cIY/rxh78GgL+AeB1ci+S JSkF5EcAi62OtzDsPUh4SWSlYSbksLbjj8UnvcHIO/y1pfWcq0gIHWEBubsPlrKYAXAd TyZOjuXU4tGHdcMzEwVBCZm8NSoa/DGyX2ayEec/IFfHcxrnxxlcRhsk/VfGjnswCyqx B1vOv5RBAIE7f5Za353SNf2vsQVkeES90FxRdNMWGPUXndHaggqPFzxsoB1xuT17o4S5 GgNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=U4JVdCRA; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b70si34618169pfk.47.2018.01.02.12.07.30; Tue, 02 Jan 2018 12:07:30 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=U4JVdCRA; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751311AbeABUHW (ORCPT + 28 others); Tue, 2 Jan 2018 15:07:22 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:41008 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751273AbeABUHR (ORCPT ); Tue, 2 Jan 2018 15:07:17 -0500 Received: by mail-wm0-f68.google.com with SMTP id g75so63340693wme.0 for ; Tue, 02 Jan 2018 12:07:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=y3cZHthDjrZdZe+7+thZcDtJZzDXcVyA3ettxQyn2/I=; b=U4JVdCRAWxIu8asYC7VR7yKgASQ4tUAP3aRQgb65Z/v9h8Of/kX68XF7IQrKEQqRbL mILpXc3g0V1XmE0jvCO2bKvYotrOisM9fdYrmv1LQvUXHa3V9ocBPN5rZ/3uvp1owdcE JyUIUE3jrgb01ODtZJbvYgN0b++pSR4Jo52yo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=y3cZHthDjrZdZe+7+thZcDtJZzDXcVyA3ettxQyn2/I=; b=ASgka48SqfBCHh1IZXvwStVWrQyEE/R8hy2zEKyOo7Itn46Xcr/XXA6tAi6Xp3C0Ei bpPC5O3PN0royir8+F1+/ydJDQp+0O93uPpqvJ05yiVhBl++nV3Ws1sLeKLCpGbQNM7c 5eQWNaM32FuFIuXDCtqW/Kgx5q/TtEZZHHPekaFvcabig4zDhm2m+GHwzb5KBQFw8gcQ aH2Fn145vlmWThZVO/pBzlBxiJ6KTa9IZDSe2NF5mLmQ8dO2EkP5kZJnYHJMHAf/sUw7 Gilm44ZbFYP+Q/xk9pkJnHhoce3HPYd7E3gsqqE9gHmm8laEFFjjw4Tw1+hjeucG/JhJ 2TxA== X-Gm-Message-State: AKGB3mLVD3qhOJ3ovndAvGylpv1rIxC03E52Z0z9f0bDaRGracuEaPuI xy8pzwR88/q0GwHMBlVBsAB4xRBrS/4= X-Received: by 10.28.74.147 with SMTP id n19mr40211357wmi.134.1514923636103; Tue, 02 Jan 2018 12:07:16 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.07.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:07:15 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v7 10/10] x86/kernel: jump_table: use relative references Date: Tue, 2 Jan 2018 20:05:49 +0000 Message-Id: <20180102200549.22984-11-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Similar to the arm64 case, 64-bit x86 can benefit from using 32-bit relative references rather than 64-bit absolute ones when emitting struct jump_entry instances. Not only does this reduce the memory footprint of the entries themselves by 50%, it also removes the need for carrying relocation metadata on relocatable builds (i.e., for KASLR) which saves a fair chunk of .init space as well (although the savings are not as dramatic as on arm64) Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/jump_label.h | 35 ++++++++++++-------- arch/x86/kernel/jump_label.c | 16 +++++++++ tools/objtool/special.c | 4 +-- 3 files changed, 39 insertions(+), 16 deletions(-) -- 2.11.0 diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 009ff2699d07..35fc2c5ec846 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -36,8 +36,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran asm_volatile_goto("1:" ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -52,8 +52,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool ".byte 0xe9\n\t .long %l[l_yes] - 2f\n\t" "2:\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -69,19 +69,26 @@ typedef u32 jump_label_t; #endif struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (unsigned long)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (unsigned long)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -99,7 +106,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #else /* __ASSEMBLY__ */ @@ -114,8 +121,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .byte STATIC_KEY_INIT_NOP .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key - . .popsection .endm @@ -130,8 +137,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .Lstatic_jump_after_\@: .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key + 1 - . .popsection .endm diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index d64296092ef5..cc5034b42335 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -149,4 +149,20 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, __jump_label_transform(entry, type, text_poke_early, 1); } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 84f001d52322..98ae55b39037 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -30,9 +30,9 @@ #define EX_ORIG_OFFSET 0 #define EX_NEW_OFFSET 4 -#define JUMP_ENTRY_SIZE 24 +#define JUMP_ENTRY_SIZE 12 #define JUMP_ORIG_OFFSET 0 -#define JUMP_NEW_OFFSET 8 +#define JUMP_NEW_OFFSET 4 #define ALT_ENTRY_SIZE 13 #define ALT_ORIG_OFFSET 0