From patchwork Wed Dec 27 08:50:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122766 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1921167qgn; Wed, 27 Dec 2017 00:52:53 -0800 (PST) X-Google-Smtp-Source: ACJfBoveFk2q5067bOFdsHK9S/1/GXV9eYo5eSMiZ58M1Olc2AYT0lFvoyqTtCBSGL505iEKWCBl X-Received: by 10.98.35.144 with SMTP id q16mr27743145pfj.13.1514364773659; Wed, 27 Dec 2017 00:52:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364773; cv=none; d=google.com; s=arc-20160816; b=J/H03gYB3J8wH3snZ1IGz6W0l/PuTFQmvCCPjprKp3/N8R0maPKVYTHNTZj2zBVvJf Be9GAtVxAQk2KXDvhZzyi1/DZzL1sAFJ2oF7U42OVPWTcp3vw9r4/bsGNunhVELY+VZm FjFozRf7el0o8oeGcNZjBeggjjBSSySryqIjjgYa/Lbazn6g2gJwyBIm0W5MKVktxw2G e1qHttSUIgBwjq9gCBsBoU3mlDKW8qbe3bPJvXT0PhFoJQ2fxqvjr5+Qw20WjcWYcijT sTe1GK16xfPdfLlVNKLYqyYiKyckgQmvU0tzgNum+E31WPebkUTACcVsNv4j/tty38yS pGEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=hET4AABNBeNsSpkOZUI8roSkdCikZPdlup845D/iO+8=; b=zo6FJeZHHB60QMyYP5etywOU+iX+rmZoy5ptOi3AMzBHFSaXkUFZTueJsAIBtYIGi8 Dj4pw3f1u2JuCx+WBY8hGrUbA1Vn7lmG92gdh0lRginzcsCwMz9RPeU35Iynp4SXIdz5 tCZ4YWHTeWVMchT+vf2EvEC2GY/2GIi4LEQYB+fWTnY+ha/QSpsmRPA+vyghgHbFhwsx 38ePem4DdL89dyYqPfwyqIsKs22gXmmKps0JsSfB1Klt9v1h+Ajeb1DGEdFujFgJW4lO LxEcvitPaxG4aPINgra8ZNDALbZ8c0QjgUQE7SW917jipi/tguK4X+msc+2ou8i2gDvY Lceg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RVum98bZ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f5si4750157pgn.653.2017.12.27.00.52.53; Wed, 27 Dec 2017 00:52:53 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RVum98bZ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751798AbdL0IvA (ORCPT + 28 others); Wed, 27 Dec 2017 03:51:00 -0500 Received: from mail-wr0-f195.google.com ([209.85.128.195]:40638 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751426AbdL0Iuy (ORCPT ); Wed, 27 Dec 2017 03:50:54 -0500 Received: by mail-wr0-f195.google.com with SMTP id p17so12481764wre.7 for ; Wed, 27 Dec 2017 00:50:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hET4AABNBeNsSpkOZUI8roSkdCikZPdlup845D/iO+8=; b=RVum98bZoyQ3h4KK/u30IZ0+svrbkkfzw05MN0fWbBwGf2z0TtCwCLZXsmRTSK2oN9 ibrUYYLKxYBMNZ1xp76hh3n7X2+06M02o2pUswQgzgXY4Y8HhgqogNZjLjmrhYgIX3Sp HJgH8YIhuAmiRCU5ACk0UdxtyspUl40Szk//8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hET4AABNBeNsSpkOZUI8roSkdCikZPdlup845D/iO+8=; b=FWoOc3JqoZPPlOQMlzbUTJ5YAz6a1CxgfjP2N3H/kqCo0MQ5gT5ZXB1RGIcCeXzGis nJhMn4Xuyu/6mPBlzdjc9oAT3n26uHvzJ2tl7W7LIOGOKQdHLHeMuujhs03RQfoCb0qA O7NN7Sq46bVfZ+zIRIkZ28Rz5xscCjxYuNbVWrpRcuJ89JhnmEnTCe9oG2VMNpoXeRXf kovD9VvPwYZHram2juEVLiRn22533xMcakiluXg606uzOo5NxlSzNk+NOAuDR0MEKqCB Q32ParBNI1c8XUfBqWM/D6Rwp7q+B3mNKmpIObT+QGOSTNXU0ICCcRr1ZOPWEsEUMJRj 3oiA== X-Gm-Message-State: AKGB3mIFeCLJiCjFhS6wXe/idyKGzH0GxYUD9zpyXDod4LfDvBnQF8Al OG1VtxGzAZlK9k3iIume1FEWLh54K+E= X-Received: by 10.223.169.8 with SMTP id u8mr29546944wrc.232.1514364652850; Wed, 27 Dec 2017 00:50:52 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.50.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:50:52 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v6 1/8] arch: enable relative relocations for arm64, power, x86, s390 and x86 Date: Wed, 27 Dec 2017 08:50:26 +0000 Message-Id: <20171227085033.22389-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Before updating certain subsystems to use place relative 32-bit relocations in special sections, to save space and reduce the number of absolute relocations that need to be processed at runtime by relocatable kernels, introduce the Kconfig symbol and define it for some architectures that should be able to support and benefit from it. Cc: Catalin Marinas Cc: Will Deacon Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: Michael Ellerman Cc: Martin Schwidefsky Cc: Heiko Carstens Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin" Cc: x86@kernel.org Signed-off-by: Ard Biesheuvel --- arch/Kconfig | 10 ++++++++++ arch/arm64/Kconfig | 1 + arch/arm64/kernel/vmlinux.lds.S | 2 +- arch/powerpc/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/x86/Kconfig | 1 + 6 files changed, 15 insertions(+), 1 deletion(-) -- 2.11.0 diff --git a/arch/Kconfig b/arch/Kconfig index 400b9e1b2f27..dbc036a7bd1b 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -959,4 +959,14 @@ config REFCOUNT_FULL against various use-after-free conditions that can be used in security flaw exploits. +config HAVE_ARCH_PREL32_RELOCATIONS + bool + help + May be selected by an architecture if it supports place-relative + 32-bit relocations, both in the toolchain and in the module loader, + in which case relative references can be used in special sections + for PCI fixup, initcalls etc which are only half the size on 64 bit + architectures, and don't require runtime relocation on relocatable + kernels. + source "kernel/gcov/Kconfig" diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c9a7e9e1414f..66c7b9ab2a3d 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -89,6 +89,7 @@ config ARM64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7da3e5c366a0..49ae5b43fe2b 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -156,7 +156,7 @@ SECTIONS CON_INITCALL SECURITY_INITCALL INIT_RAM_FS - *(.init.rodata.* .init.bss) /* from the EFI stub */ + *(.init.rodata.* .init.bss .init.discard.*) /* EFI stub */ } .exit.data : { ARM_EXIT_KEEP(EXIT_DATA) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index c51e6ce42e7a..e172478e2ae7 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -177,6 +177,7 @@ config PPC select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select ARCH_HAS_STRICT_KERNEL_RWX if ((PPC_BOOK3S_64 || PPC32) && !RELOCATABLE && !HIBERNATION) diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 829c67986db7..ed29d1ebecd9 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -129,6 +129,7 @@ config S390 select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_JUMP_LABEL select CPU_NO_EFFICIENT_FFS if !HAVE_MARCH_Z9_109_FEATURES + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_SOFT_DIRTY select HAVE_ARCH_TRACEHOOK diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index d4fc98c50378..9f2bb853aedb 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -115,6 +115,7 @@ config X86 select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT select HAVE_ARCH_COMPAT_MMAP_BASES if MMU && COMPAT + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE From patchwork Wed Dec 27 08:50:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122767 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1921251qgn; Wed, 27 Dec 2017 00:53:01 -0800 (PST) X-Google-Smtp-Source: ACJfBot9F56y20/jA72hNtNceMxzh6hw6n9pY+lK0gzvkqrhTFzKVsKH8QD3wG4Kve0cSicfS/kS X-Received: by 10.99.109.2 with SMTP id i2mr25077012pgc.9.1514364781049; Wed, 27 Dec 2017 00:53:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364781; cv=none; d=google.com; s=arc-20160816; b=QkgcxflNLlrSMTCa/zu9EQB3suKpyHnX/MbJOmFel/37AYRHpX4lfkdrclU9ZmUq/o mjvUo5Qrq9kMCaO88lgXXsQTBuTKrPxr4SZDERRGGpw3nKYDZ44HsW8VSB4h98v2mF4k I8R5WkTvRi+PPt0sz0+3/EmXeIUTWUGA7LKAi4qwDSBx1LdQd0YcbiygqoDkvXSavUuw Gwj0t7TktjvwlztCktLusAbWA5zhXzVHoL+Hx0nESox+xqwnZyZd7ZCNGY+nnEDitYf1 3aYGjSBnqznugiDqb1hYGzCadEKBUQ6u144KMwbOh2ZRWR7oddJ8SteRP9O91lUW5jlG KNww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=hwNmpUBJll0CBM4WpZWcFTd/iFpXr+kE+wqybgwXamo=; b=g0yKm8upC6LVSqf9YBiZYUSjqnSS2RMhqxbS6gihHBH+7mtklc77eQA7vCjXdb5L4p lIjB1b4qd55ynV265N8rSEU8xk1T5SVAvhcqgz75Up0NMCUVKyQOluE5uR9oOCj9Aym9 TD67o/O7Vj5ZP+RmfHeOWVJHInDqxh3rvmt8EM/k93upt3i05/eDeSnf9a43XK2jeBKO wxicJoFkFv8dpVXSTm1mrIzBQlNwgScZluc40+hJjd9ZgIeqJqi3xvHL+lepzL9iihXp ElVNySF7YgF9A0v9yadoRkBPSf6ibRud4YNyQz2LGUr3TmsSqsqmiC6c56tNVBX7u+3e pQGg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YZQk+3+2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w12si24081420plz.760.2017.12.27.00.53.00; Wed, 27 Dec 2017 00:53:01 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YZQk+3+2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751998AbdL0Iw6 (ORCPT + 28 others); Wed, 27 Dec 2017 03:52:58 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:38479 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751539AbdL0Iu7 (ORCPT ); Wed, 27 Dec 2017 03:50:59 -0500 Received: by mail-wr0-f196.google.com with SMTP id o2so36218911wro.5 for ; Wed, 27 Dec 2017 00:50:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hwNmpUBJll0CBM4WpZWcFTd/iFpXr+kE+wqybgwXamo=; b=YZQk+3+2mVncYlBRUI7czm9/CijwgrR43rx7aTrHNaR72l6DzK2cM5M1c9RMweTWLn SQRbrEjrSsRZZLFN2/3IadhqSHkVqyFCFjg1JaEFrn0LMtg00oPthkDajS+L3JQBElDh YA4a0IIrTlVbSuicCnWlBIgzsqfOlI/FPQEGo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hwNmpUBJll0CBM4WpZWcFTd/iFpXr+kE+wqybgwXamo=; b=YctB5Elm2byGBpcZOdsfm5XB9W2cqth8BTPVn6KhEe2nEmNE7VO9L66uS040+9cjM6 wXZJJCPKvsyMec4WjMlZxMOuCktW9Apw48aYyAn85TC7rrHT+7ablIqADuhnXiLjh0Cb udOJJ36P5EuyQSmuicEiEiH4i+zMY85ucaiFrMhGCVUlDhAzTH9SoCtGSGiVzKgh32PL Ul/O1i+lI7aH0BT+2TFeRyxgW4TauPb/R3hlbmxVMgiL4RUNHpsKPcArHEEQda9IfdIU swrwhb3QqeBlEop5+Oj8Tj+T9PEXOOKZftDVPNyAN8lMMktFZqhLg7oq2brN32/gtx7u JTgQ== X-Gm-Message-State: AKGB3mKFN3A8gP2e7HAWsEJkWGvWk2gRLsg++XC2u0xrIdtcILLbiXUU qgsnHbnI1b8yOfxpDAb+Kuof5pwa3OU= X-Received: by 10.223.139.196 with SMTP id w4mr26262405wra.51.1514364657648; Wed, 27 Dec 2017 00:50:57 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.50.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:50:56 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, Ingo Molnar Subject: [PATCH v6 2/8] module: use relative references for __ksymtab entries Date: Wed, 27 Dec 2017 08:50:27 +0000 Message-Id: <20171227085033.22389-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org An ordinary arm64 defconfig build has ~64 KB worth of __ksymtab entries, each consisting of two 64-bit fields containing absolute references, to the symbol itself and to a char array containing its name, respectively. When we build the same configuration with KASLR enabled, we end up with an additional ~192 KB of relocations in the .init section, i.e., one 24 byte entry for each absolute reference, which all need to be processed at boot time. Given how the struct kernel_symbol that describes each entry is completely local to module.c (except for the references emitted by EXPORT_SYMBOL() itself), we can easily modify it to contain two 32-bit relative references instead. This reduces the size of the __ksymtab section by 50% for all 64-bit architectures, and gets rid of the runtime relocations entirely for architectures implementing KASLR, either via standard PIE linking (arm64) or using custom host tools (x86). Note that the binary search involving __ksymtab contents relies on each section being sorted by symbol name. This is implemented based on the input section names, not the names in the ksymtab entries, so this patch does not interfere with that. Given that the use of place-relative relocations requires support both in the toolchain and in the module loader, we cannot enable this feature for all architectures. So make it dependent on whether CONFIG_HAVE_ARCH_PREL32_RELOCATIONS is defined. Cc: Arnd Bergmann Cc: Andrew Morton Cc: Ingo Molnar Cc: Kees Cook Cc: Thomas Garnier Cc: Nicolas Pitre Acked-by: Jessica Yu Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/Kbuild | 1 + arch/x86/include/asm/export.h | 5 --- include/asm-generic/export.h | 12 ++++- include/linux/compiler.h | 11 +++++ include/linux/export.h | 46 +++++++++++++++----- kernel/module.c | 33 +++++++++++--- 6 files changed, 84 insertions(+), 24 deletions(-) -- 2.11.0 diff --git a/arch/x86/include/asm/Kbuild b/arch/x86/include/asm/Kbuild index 5d6a53fd7521..3e8a88dcaa1d 100644 --- a/arch/x86/include/asm/Kbuild +++ b/arch/x86/include/asm/Kbuild @@ -9,5 +9,6 @@ generated-y += xen-hypercalls.h generic-y += clkdev.h generic-y += dma-contiguous.h generic-y += early_ioremap.h +generic-y += export.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h diff --git a/arch/x86/include/asm/export.h b/arch/x86/include/asm/export.h deleted file mode 100644 index 2a51d66689c5..000000000000 --- a/arch/x86/include/asm/export.h +++ /dev/null @@ -1,5 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifdef CONFIG_64BIT -#define KSYM_ALIGN 16 -#endif -#include diff --git a/include/asm-generic/export.h b/include/asm-generic/export.h index 719db1968d81..97ce606459ae 100644 --- a/include/asm-generic/export.h +++ b/include/asm-generic/export.h @@ -5,12 +5,10 @@ #define KSYM_FUNC(x) x #endif #ifdef CONFIG_64BIT -#define __put .quad #ifndef KSYM_ALIGN #define KSYM_ALIGN 8 #endif #else -#define __put .long #ifndef KSYM_ALIGN #define KSYM_ALIGN 4 #endif @@ -25,6 +23,16 @@ #define KSYM(name) name #endif +.macro __put, val, name +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + .long \val - ., \name - . +#elif defined(CONFIG_64BIT) + .quad \val, \name +#else + .long \val, \name +#endif +.endm + /* * note on .section use: @progbits vs %progbits nastiness doesn't matter, * since we immediately emit into those sections anyway. diff --git a/include/linux/compiler.h b/include/linux/compiler.h index 52e611ab9a6c..fe752d365334 100644 --- a/include/linux/compiler.h +++ b/include/linux/compiler.h @@ -327,4 +327,15 @@ static __always_inline void __write_once_size(volatile void *p, void *res, int s compiletime_assert(__native_word(t), \ "Need native word sized stores/loads for atomicity.") +/* + * Force the compiler to emit 'sym' as a symbol, so that we can reference + * it from inline assembler. Necessary in case 'sym' could be inlined + * otherwise, or eliminated entirely due to lack of references that are + * visibile to the compiler. + */ +#define __ADDRESSABLE(sym) \ + static void *__attribute__((section(".discard.text"), used)) \ + __PASTE(__discard_##sym, __LINE__)(void) \ + { return (void *)&sym; } \ + #endif /* __LINUX_COMPILER_H */ diff --git a/include/linux/export.h b/include/linux/export.h index 1a1dfdb2a5c6..5112d0c41512 100644 --- a/include/linux/export.h +++ b/include/linux/export.h @@ -24,12 +24,6 @@ #define VMLINUX_SYMBOL_STR(x) __VMLINUX_SYMBOL_STR(x) #ifndef __ASSEMBLY__ -struct kernel_symbol -{ - unsigned long value; - const char *name; -}; - #ifdef MODULE extern struct module __this_module; #define THIS_MODULE (&__this_module) @@ -60,17 +54,47 @@ extern struct module __this_module; #define __CRC_SYMBOL(sym, sec) #endif +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#include +/* + * Emit the ksymtab entry as a pair of relative references: this reduces + * the size by half on 64-bit architectures, and eliminates the need for + * absolute relocations that require runtime processing on relocatable + * kernels. + */ +#define __KSYMTAB_ENTRY(sym, sec) \ + __ADDRESSABLE(sym) \ + asm(" .section \"___ksymtab" sec "+" #sym "\", \"a\" \n" \ + " .balign 8 \n" \ + VMLINUX_SYMBOL_STR(__ksymtab_##sym) ": \n" \ + " .long " VMLINUX_SYMBOL_STR(sym) "- . \n" \ + " .long " VMLINUX_SYMBOL_STR(__kstrtab_##sym) "- .\n" \ + " .previous \n") + +struct kernel_symbol { + signed int value_offset; + signed int name_offset; +}; +#else +#define __KSYMTAB_ENTRY(sym, sec) \ + static const struct kernel_symbol __ksymtab_##sym \ + __attribute__((section("___ksymtab" sec "+" #sym), used)) \ + = { (unsigned long)&sym, __kstrtab_##sym } + +struct kernel_symbol { + unsigned long value; + const char *name; +}; +#endif + /* For every exported symbol, place a struct in the __ksymtab section */ #define ___EXPORT_SYMBOL(sym, sec) \ extern typeof(sym) sym; \ __CRC_SYMBOL(sym, sec) \ static const char __kstrtab_##sym[] \ - __attribute__((section("__ksymtab_strings"), aligned(1))) \ + __attribute__((section("__ksymtab_strings"), used, aligned(1))) \ = VMLINUX_SYMBOL_STR(sym); \ - static const struct kernel_symbol __ksymtab_##sym \ - __used \ - __attribute__((section("___ksymtab" sec "+" #sym), used)) \ - = { (unsigned long)&sym, __kstrtab_##sym } + __KSYMTAB_ENTRY(sym, sec) #if defined(__KSYM_DEPS__) diff --git a/kernel/module.c b/kernel/module.c index dea01ac9cb74..d3a908ffc42c 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -549,12 +549,31 @@ static bool check_symbol(const struct symsearch *syms, return true; } +static unsigned long kernel_symbol_value(const struct kernel_symbol *sym) +{ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + return (unsigned long)&sym->value_offset + sym->value_offset; +#else + return sym->value; +#endif +} + +static const char *kernel_symbol_name(const struct kernel_symbol *sym) +{ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + return (const char *)((unsigned long)&sym->name_offset + + sym->name_offset); +#else + return sym->name; +#endif +} + static int cmp_name(const void *va, const void *vb) { const char *a; const struct kernel_symbol *b; a = va; b = vb; - return strcmp(a, b->name); + return strcmp(a, kernel_symbol_name(b)); } static bool find_symbol_in_section(const struct symsearch *syms, @@ -2198,7 +2217,7 @@ void *__symbol_get(const char *symbol) sym = NULL; preempt_enable(); - return sym ? (void *)sym->value : NULL; + return sym ? (void *)kernel_symbol_value(sym) : NULL; } EXPORT_SYMBOL_GPL(__symbol_get); @@ -2228,10 +2247,12 @@ static int verify_export_symbols(struct module *mod) for (i = 0; i < ARRAY_SIZE(arr); i++) { for (s = arr[i].sym; s < arr[i].sym + arr[i].num; s++) { - if (find_symbol(s->name, &owner, NULL, true, false)) { + if (find_symbol(kernel_symbol_name(s), &owner, NULL, + true, false)) { pr_err("%s: exports duplicate symbol %s" " (owned by %s)\n", - mod->name, s->name, module_name(owner)); + mod->name, kernel_symbol_name(s), + module_name(owner)); return -ENOEXEC; } } @@ -2280,7 +2301,7 @@ static int simplify_symbols(struct module *mod, const struct load_info *info) ksym = resolve_symbol_wait(mod, info, name); /* Ok if resolved. */ if (ksym && !IS_ERR(ksym)) { - sym[i].st_value = ksym->value; + sym[i].st_value = kernel_symbol_value(ksym); break; } @@ -2540,7 +2561,7 @@ static int is_exported(const char *name, unsigned long value, ks = lookup_symbol(name, __start___ksymtab, __stop___ksymtab); else ks = lookup_symbol(name, mod->syms, mod->syms + mod->num_syms); - return ks != NULL && ks->value == value; + return ks != NULL && kernel_symbol_value(ks) == value; } /* As per nm */ From patchwork Wed Dec 27 08:50:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122760 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1919874qgn; Wed, 27 Dec 2017 00:51:14 -0800 (PST) X-Google-Smtp-Source: ACJfBovQWGMU3dDSjwvc2KkDJUSKwRhNCXluN+SEFxpUUr5LviGMZAoEWSTLvR/sRjS/0E/JCdWa X-Received: by 10.101.101.203 with SMTP id y11mr11340223pgv.387.1514364674480; Wed, 27 Dec 2017 00:51:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364674; cv=none; d=google.com; s=arc-20160816; b=qC2brAUzhptnFflzACsD/sKBEoD8V/GKPr4MvWWZsxIA29kf1kb7QMJMgOYA1MECvk 5bCTYpOXUmG85tgGQ9M/gpRtCQBU7ovmktOGUdBFCv82wJ/Dbnk2mvYrpA69PRYA/6Ro Ui4Qc3DWvlDOkI4qnCHIjYFIpQfrzXBoPNUTSnNiMyWED7HGxInbviOEC7XCFmW4izBo QVUGcFb06O6CG0t9oeBtMgu5BXjzBLbAdaLoivPVd7807LBK8FEVdeI8m5YZSQmB7P8X fhZBVITZX8T+3YZZlhVh3c6T6mJ6nXTBo0qTGYltNHiBDHCqvZPZhCxAzMr4+Kgse5oz eVSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=lwLLrS7mueZ/zWj6YdhKL1lPIayLUTMzei7H77VCkQk=; b=Lqvrd4sDrI3LC8KrgFu5Y+FLxbmLs5SELMyQTjC0Xr6uGXca3304RsPA9M37QHjJ2k BreUikF2REoWKVedtqmvq0jLk48fmlIl8r89yH+ZTN9sNm6ApASXVN2fzXXorzqnYP2f RPwSUIrOmnWP+sBhRowtgDMBcRHSwInpXK1IqdzcgZFSn7Hcg2TozicnVXoFDI+Pcbwl 2qgIgQ1B399UMzV+cjttX1rIPEhvxQXFuVHmqZjul/OIZKZW108I7YgcTmetPap4Xwqp LUYes5dM659vpGWyBG0gojrfIEk42k8S15C1YPM6dQf2lD3NIkjATLJwYUN1n+NaF4cw 6+gg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QtJm4zXV; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n11si18413086plg.13.2017.12.27.00.51.14; Wed, 27 Dec 2017 00:51:14 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QtJm4zXV; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751862AbdL0IvK (ORCPT + 28 others); Wed, 27 Dec 2017 03:51:10 -0500 Received: from mail-wm0-f66.google.com ([74.125.82.66]:36653 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751833AbdL0IvG (ORCPT ); Wed, 27 Dec 2017 03:51:06 -0500 Received: by mail-wm0-f66.google.com with SMTP id b76so38343658wmg.1 for ; Wed, 27 Dec 2017 00:51:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=lwLLrS7mueZ/zWj6YdhKL1lPIayLUTMzei7H77VCkQk=; b=QtJm4zXVhuLpsXWoyJuUQkhle/5dXy0JI2WcpXmhBquEjre0C3BUPD5ch/f18mczrf DiLO3OML2FGRfJ8GotRLc7hdIbjGDa2Qbkl4gX/eiHwmkj4/1YkM4Ig4gS7IqJTnbFTL WjUFiGZM9AzmnZmYJOL1Yzn18Jf3xY0BnuVrs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=lwLLrS7mueZ/zWj6YdhKL1lPIayLUTMzei7H77VCkQk=; b=fZxrjOcQ4Bh0iS4H2ZGv+jlgEXuz/gPsrU+WV+U07D63iU160NJA2D6o7Lr6e+ajpy 1MXLlewvXdlAYGBYoEV98vqB4wIIdYgTUvmX0Y8L5jfKoinMo7Xp6+4iDZv20dRFaeyJ AFrnPNB9bRqkeDIzktZGurtYsTxYUtuS2K6ANnwY4fxOb4vA15U4m1Bte/j/XQosmls9 sqIv9xAQkAHjAvLftI909Kk75JnZoVROtBicbUY4hphYv8mkbg2teYiQJVGPauyZqTkL CTZs40wWBCWNJBJyOXb0oXX8WLpYrtbjLiRy4XtT1gGPLZdU/fKnkRuCWhquGC3uvvLd cAww== X-Gm-Message-State: AKGB3mIzpBhmjEp8W/Yo5Kmt9XeJ87FMc3UHiu30iCx2mpIoXRFw79/m 11vYcnG6TivTR83g6Gw6as+vlm+m6xA= X-Received: by 10.28.160.23 with SMTP id j23mr21304907wme.54.1514364664371; Wed, 27 Dec 2017 00:51:04 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.50.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:51:03 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v6 3/8] init: allow initcall tables to be emitted using relative references Date: Wed, 27 Dec 2017 08:50:28 +0000 Message-Id: <20171227085033.22389-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Allow the initcall tables to be emitted using relative references that are only half the size on 64-bit architectures and don't require fixups at runtime on relocatable kernels. Cc: Petr Mladek Cc: Sergey Senozhatsky Cc: Steven Rostedt Cc: James Morris Cc: "Serge E. Hallyn" Signed-off-by: Ard Biesheuvel --- include/linux/init.h | 44 +++++++++++++++----- init/main.c | 32 +++++++------- kernel/printk/printk.c | 4 +- security/security.c | 4 +- 4 files changed, 53 insertions(+), 31 deletions(-) -- 2.11.0 diff --git a/include/linux/init.h b/include/linux/init.h index ea1b31101d9e..125bbea99c6b 100644 --- a/include/linux/init.h +++ b/include/linux/init.h @@ -109,8 +109,24 @@ typedef int (*initcall_t)(void); typedef void (*exitcall_t)(void); -extern initcall_t __con_initcall_start[], __con_initcall_end[]; -extern initcall_t __security_initcall_start[], __security_initcall_end[]; +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +typedef signed int initcall_entry_t; + +static inline initcall_t initcall_from_entry(initcall_entry_t *entry) +{ + return (initcall_t)((unsigned long)entry + *entry); +} +#else +typedef initcall_t initcall_entry_t; + +static inline initcall_t initcall_from_entry(initcall_entry_t *entry) +{ + return *entry; +} +#endif + +extern initcall_entry_t __con_initcall_start[], __con_initcall_end[]; +extern initcall_entry_t __security_initcall_start[], __security_initcall_end[]; /* Used for contructor calls. */ typedef void (*ctor_fn_t)(void); @@ -160,9 +176,20 @@ extern bool initcall_debug; * as KEEP() in the linker script. */ -#define __define_initcall(fn, id) \ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#define ___define_initcall(fn, id, __sec) \ + __ADDRESSABLE(fn) \ + asm(".section \"" #__sec ".init\", \"a\" \n" \ + "__initcall_" #fn #id ": \n" \ + ".long " VMLINUX_SYMBOL_STR(fn) " - . \n" \ + ".previous \n"); +#else +#define ___define_initcall(fn, id, __sec) \ static initcall_t __initcall_##fn##id __used \ - __attribute__((__section__(".initcall" #id ".init"))) = fn; + __attribute__((__section__(#__sec ".init"))) = fn; +#endif + +#define __define_initcall(fn, id) ___define_initcall(fn, id, .initcall##id) /* * Early initcalls run before initializing SMP. @@ -201,13 +228,8 @@ extern bool initcall_debug; #define __exitcall(fn) \ static exitcall_t __exitcall_##fn __exit_call = fn -#define console_initcall(fn) \ - static initcall_t __initcall_##fn \ - __used __section(.con_initcall.init) = fn - -#define security_initcall(fn) \ - static initcall_t __initcall_##fn \ - __used __section(.security_initcall.init) = fn +#define console_initcall(fn) ___define_initcall(fn,, .con_initcall) +#define security_initcall(fn) ___define_initcall(fn,, .security_initcall) struct obs_kernel_param { const char *str; diff --git a/init/main.c b/init/main.c index 7b606fc48482..2cbe3c2804ab 100644 --- a/init/main.c +++ b/init/main.c @@ -845,18 +845,18 @@ int __init_or_module do_one_initcall(initcall_t fn) } -extern initcall_t __initcall_start[]; -extern initcall_t __initcall0_start[]; -extern initcall_t __initcall1_start[]; -extern initcall_t __initcall2_start[]; -extern initcall_t __initcall3_start[]; -extern initcall_t __initcall4_start[]; -extern initcall_t __initcall5_start[]; -extern initcall_t __initcall6_start[]; -extern initcall_t __initcall7_start[]; -extern initcall_t __initcall_end[]; - -static initcall_t *initcall_levels[] __initdata = { +extern initcall_entry_t __initcall_start[]; +extern initcall_entry_t __initcall0_start[]; +extern initcall_entry_t __initcall1_start[]; +extern initcall_entry_t __initcall2_start[]; +extern initcall_entry_t __initcall3_start[]; +extern initcall_entry_t __initcall4_start[]; +extern initcall_entry_t __initcall5_start[]; +extern initcall_entry_t __initcall6_start[]; +extern initcall_entry_t __initcall7_start[]; +extern initcall_entry_t __initcall_end[]; + +static initcall_entry_t *initcall_levels[] __initdata = { __initcall0_start, __initcall1_start, __initcall2_start, @@ -882,7 +882,7 @@ static char *initcall_level_names[] __initdata = { static void __init do_initcall_level(int level) { - initcall_t *fn; + initcall_entry_t *fn; strcpy(initcall_command_line, saved_command_line); parse_args(initcall_level_names[level], @@ -892,7 +892,7 @@ static void __init do_initcall_level(int level) NULL, &repair_env_string); for (fn = initcall_levels[level]; fn < initcall_levels[level+1]; fn++) - do_one_initcall(*fn); + do_one_initcall(initcall_from_entry(fn)); } static void __init do_initcalls(void) @@ -923,10 +923,10 @@ static void __init do_basic_setup(void) static void __init do_pre_smp_initcalls(void) { - initcall_t *fn; + initcall_entry_t *fn; for (fn = __initcall_start; fn < __initcall0_start; fn++) - do_one_initcall(*fn); + do_one_initcall(initcall_from_entry(fn)); } /* diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c index b9006617710f..0516005261c7 100644 --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -2611,7 +2611,7 @@ EXPORT_SYMBOL(unregister_console); */ void __init console_init(void) { - initcall_t *call; + initcall_entry_t *call; /* Setup the default TTY line discipline. */ n_tty_init(); @@ -2622,7 +2622,7 @@ void __init console_init(void) */ call = __con_initcall_start; while (call < __con_initcall_end) { - (*call)(); + initcall_from_entry(call)(); call++; } } diff --git a/security/security.c b/security/security.c index 1cd8526cb0b7..f648eeff06de 100644 --- a/security/security.c +++ b/security/security.c @@ -45,10 +45,10 @@ static __initdata char chosen_lsm[SECURITY_NAME_MAX + 1] = static void __init do_security_initcalls(void) { - initcall_t *call; + initcall_entry_t *call; call = __security_initcall_start; while (call < __security_initcall_end) { - (*call) (); + initcall_from_entry(call)(); call++; } } From patchwork Wed Dec 27 08:50:29 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122761 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1919948qgn; Wed, 27 Dec 2017 00:51:20 -0800 (PST) X-Google-Smtp-Source: ACJfBouhLzJJfjD6+5HE+D5y/frAIybrj2f+C0tCfaAMipcDOvEydf5XqWnie/EXlOLsS1Sq6oQv X-Received: by 10.98.252.7 with SMTP id e7mr27659988pfh.12.1514364680150; Wed, 27 Dec 2017 00:51:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364680; cv=none; d=google.com; s=arc-20160816; b=Kxfms+0LTScBZDkUK1mjb1u8Kq9RHFWkJhn+41jHR9aAiOyhfjUqLgyf1hNZKcNqzn xpqoVapCftK80Zr5OK9LGXWE/IOcIgknPTalY9b7AKuR29i2m7hcoBzsHjCYcZ9+nJ9Y huwHpNlJLM2eyT6KQcXvBKbsYtSFgFB75SoJOdfq26NMZWvyVtOEkJ3ahg18WpMGN15T MtHXfxZRgZEIqsgT/NwCGzhKUAxgNfXue3p4PcezQfjnZyCibTnaa13ARUMsyALdgORy f+3SYHGc0Yg/sy0egP1Ldit6DEaQek1u4cZBd8Wt6E09ypXE6igUfMLUImd5jav9FVjq JA8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=MNuvKMvZr3GVfTz4MvnzswRbek+xsbzeG112C9Cor2w=; b=ciRVMrkBIFMGpxRXg0iMwc8ROuFGYxPhY60TaVTOmidkKeYegyStAAfo91ePqsKm15 zHYA9NZpGjrsXT2S/4L0/PuW8IuMCEgxdnMAKhM40g1AMzecY9DccUGoFGY/im8JiKo7 g8iqtkYCQnCIIeW7eH30ce3gsSBwLceEAD4edK9XVhqSfNGLhAc9gnXrfw0q3PuFhWOr AAAwPiZQ36q0GaJD1F1QS/Ez4zM0RCMDDBKXzDg7c0VHbnwu/ZYd6PLbXLkMcGuCiU2Y FmEs9OqFmsC9ydo8J3kv1FGyXtbLLYg4l1izCXi96xEK+bBwTfaKyJ4oGj5B5KTd5YSN byYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AJ/zr34a; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f1si24887036pln.118.2017.12.27.00.51.19; Wed, 27 Dec 2017 00:51:20 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AJ/zr34a; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751887AbdL0IvQ (ORCPT + 28 others); Wed, 27 Dec 2017 03:51:16 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:45768 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751867AbdL0IvM (ORCPT ); Wed, 27 Dec 2017 03:51:12 -0500 Received: by mail-wr0-f193.google.com with SMTP id o15so4022592wrf.12 for ; Wed, 27 Dec 2017 00:51:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MNuvKMvZr3GVfTz4MvnzswRbek+xsbzeG112C9Cor2w=; b=AJ/zr34aK0sT46fTFsfCFR1G30dXqEZ4pyPPilfvxKN86s3tJsT9l08DRTgEjFquhT JRk6X/cAvjr272ahonxRSd8HJ3zdzM93Lqw+0YW3ftpE5HIFcpceZ42qkfgsPKlN5E1z Khsmi+imR9JlPH8hPy3YgBFyJyeMNX7h8Otkk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MNuvKMvZr3GVfTz4MvnzswRbek+xsbzeG112C9Cor2w=; b=GlHr7T3jpLX1gSGL8p7g+6ZPqAWNoxmAsNRgFzGKq5GKiA1Xt/kFBnI7sLxQMs7AjE 8fxqyY6hSoSs8KEiDRXoDLLMiXv3JAeTmibatm/PRt7ZD4eITY259VQeDDNqteWR+3jI KGYkn2ptJfLYjQuDUGkk02AKye/A8v2C5vtc5yakTMsbOm8LOiAq4DWqGyjJ1MCD4bJe aPR046oN6ChZCPJ2LuUD063+ImwApcFShEgLU3gv9NR37BuIDb7FiLcPbscmLvPbAv+t Vk1OVrXyZjMbp9x9ysJI+P1YTw1Wn3QZBcO5UAEb3fUTf9/oDVdidLAdSYGnrNoVlcZl EwpA== X-Gm-Message-State: AKGB3mKXjOOCgH3LFpN94iXmf1+Wf6txm87wNjLYN8hw2wndK0h5dEr8 f1em3hDhUhU4qBrlfV5XfxrUVJW7+oU= X-Received: by 10.223.128.9 with SMTP id 9mr19838598wrk.70.1514364670694; Wed, 27 Dec 2017 00:51:10 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.51.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:51:09 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v6 4/8] PCI: Add support for relative addressing in quirk tables Date: Wed, 27 Dec 2017 08:50:29 +0000 Message-Id: <20171227085033.22389-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Allow the PCI quirk tables to be emitted in a way that avoids absolute references to the hook functions. This reduces the size of the entries, and, more importantly, makes them invariant under runtime relocation (e.g., for KASLR) Acked-by: Bjorn Helgaas Signed-off-by: Ard Biesheuvel --- drivers/pci/quirks.c | 13 ++++++++++--- include/linux/pci.h | 20 ++++++++++++++++++++ 2 files changed, 30 insertions(+), 3 deletions(-) -- 2.11.0 diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 10684b17d0bd..b6d51b4d5ce1 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3556,9 +3556,16 @@ static void pci_do_fixups(struct pci_dev *dev, struct pci_fixup *f, f->vendor == (u16) PCI_ANY_ID) && (f->device == dev->device || f->device == (u16) PCI_ANY_ID)) { - calltime = fixup_debug_start(dev, f->hook); - f->hook(dev); - fixup_debug_report(dev, calltime, f->hook); + void (*hook)(struct pci_dev *dev); +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + hook = (void *)((unsigned long)&f->hook_offset + + f->hook_offset); +#else + hook = f->hook; +#endif + calltime = fixup_debug_start(dev, hook); + hook(dev); + fixup_debug_report(dev, calltime, hook); } } diff --git a/include/linux/pci.h b/include/linux/pci.h index c170c9250c8b..e8c34afb5d4a 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1792,7 +1792,11 @@ struct pci_fixup { u16 device; /* You can use PCI_ANY_ID here of course */ u32 class; /* You can use PCI_ANY_ID here too */ unsigned int class_shift; /* should be 0, 8, 16 */ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + signed int hook_offset; +#else void (*hook)(struct pci_dev *dev); +#endif }; enum pci_fixup_pass { @@ -1806,12 +1810,28 @@ enum pci_fixup_pass { pci_fixup_suspend_late, /* pci_device_suspend_late() */ }; +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#define __DECLARE_PCI_FIXUP_SECTION(sec, name, vendor, device, class, \ + class_shift, hook) \ + __ADDRESSABLE(hook) \ + asm(".section " #sec ", \"a\" \n" \ + ".balign 16 \n" \ + ".short " #vendor ", " #device " \n" \ + ".long " #class ", " #class_shift " \n" \ + ".long " VMLINUX_SYMBOL_STR(hook) " - . \n" \ + ".previous \n"); +#define DECLARE_PCI_FIXUP_SECTION(sec, name, vendor, device, class, \ + class_shift, hook) \ + __DECLARE_PCI_FIXUP_SECTION(sec, name, vendor, device, class, \ + class_shift, hook) +#else /* Anonymous variables would be nice... */ #define DECLARE_PCI_FIXUP_SECTION(section, name, vendor, device, class, \ class_shift, hook) \ static const struct pci_fixup __PASTE(__pci_fixup_##name,__LINE__) __used \ __attribute__((__section__(#section), aligned((sizeof(void *))))) \ = { vendor, device, class, class_shift, hook }; +#endif #define DECLARE_PCI_FIXUP_CLASS_EARLY(vendor, device, class, \ class_shift, hook) \ From patchwork Wed Dec 27 08:50:30 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122762 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1920073qgn; Wed, 27 Dec 2017 00:51:27 -0800 (PST) X-Google-Smtp-Source: ACJfBoursDmquIESOi3dL/9tKNUzTv03GhbnxpYpi67wMLiwuHZshy7DkRjNyaUyfBCUsQguuToG X-Received: by 10.159.252.76 with SMTP id t12mr28212132plz.323.1514364687669; Wed, 27 Dec 2017 00:51:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364687; cv=none; d=google.com; s=arc-20160816; b=o5lgoeq3x4cdQgIB+gwL49ZoXSp3BbGhEPLVAXJdj4JF2X6TuKoSZ417rzqm7mkebu u/lBmSXQZv8nyK5wfZ83m6Pjjen/ULp0dTvFLqB3E3BJ81sW/DFvvHdKCcQqrdvlE4QP mQB/KNdNFA53pzigX68cWEtEgLkG0YBWHshE81MMpiBXtTngZr2ZE4y1VLRr/EXvh6CJ pM7kKpdnoEyfRZHExBswKWsIql7x/337PBO+9lEaQ+GpFuziBEnEf/gUHfbt5osvtQu1 Fx5r9+b+cVGe+yYrRAflbYsU40i/KZLKwaPjeS5Xow0mFhB+Hv90W8QlZH8TGg/5bks0 rf8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=FeRUkMrB+h9OzAG6tsXZHgHhFJKOYm9CVFNsKuS7D+M=; b=ZpznLs7k8/BzabqeMpphuM5CBXqPszo7bH5AOEJeIa8V24R4JHUFzIuC0K/sRdStRQ Ky0VG7GnpjxGn7MvajsASmebmeOjGAwueGodWoa2LgI0Yu+SGeZjM5kPY1+TAXp7NuNE A8BfaMywQzK3D6/N916cXT4R8wameNAMiXlSz76TnYUBWe6/Sgt6qr5TkvRZLw6ubcv1 D0UtDJ+EwSHD0mzWSKAb6faJdGCZgAjhxAWYgxG3kU8ZBZd+GwzaVbo/PoZCFyJODYpP DeHbRJjJYtGWX1ul/L5jO7UTac6qNTkOfjkB2xZISFXF7JkZRCtRpP6Stp4VczF2uhNU ZRPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=C0bXnKbk; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a17si16507541pgf.378.2017.12.27.00.51.27; Wed, 27 Dec 2017 00:51:27 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=C0bXnKbk; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751911AbdL0IvW (ORCPT + 28 others); Wed, 27 Dec 2017 03:51:22 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:46375 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751891AbdL0IvS (ORCPT ); Wed, 27 Dec 2017 03:51:18 -0500 Received: by mail-wr0-f193.google.com with SMTP id g17so26988237wrd.13 for ; Wed, 27 Dec 2017 00:51:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FeRUkMrB+h9OzAG6tsXZHgHhFJKOYm9CVFNsKuS7D+M=; b=C0bXnKbkb3eJ2wpcGYBYzKmS7OKKVzqXvhGL+TE/Uy7dsuLNclfxKdtFgpHGQgKloY Zs4iWMSBuMsqiYFGCw7m1R4DbRiqKG1NW7f82TFyG7d1QwH3ewhZOSt0KdSM89TLjK2S iuQ8d9gZaTRU5K3xKVmzRGBFzahnqD72bpvu0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FeRUkMrB+h9OzAG6tsXZHgHhFJKOYm9CVFNsKuS7D+M=; b=T4M/W0OfA6o1sG8TYXRom9DC9xnb16VjbGeA7Ia/86bicZ9p8z0feIYYK0gIBLOR6A 9CbXgLAKkvCvJKTi6LOvo0qx/vLlPx2bdZibhCdeZxy1XGdHFRBif7RcdJmTErqbN763 XHnZsCiYhzTEQjoXRcPjKRl9AyC0MC0br+wDuf3o6L+sqJMplMTDSzFYbe90qd6nk4+j OCNbwFn3ftfsfLrS5GTpNHZoI51huEza7pUgYEWaN4nXurteVHfifhLaWljHKqdNCKg5 +yarxF6zIEGvC0eT2emMNdRp6r27xyGdXOxJ0szepHjZ0HGEKBYStTLSfKd3kHLtpucJ NeMw== X-Gm-Message-State: AKGB3mLFLvMcrzBBdR9AmTK6QRmACau2PdZOPOwB5sGW/0jvT6hKipOZ W+q+GwQQebmMjB0TB5zhmVUAH2i2sYg= X-Received: by 10.223.136.118 with SMTP id e51mr28244488wre.21.1514364676546; Wed, 27 Dec 2017 00:51:16 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.51.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:51:15 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v6 5/8] kernel: tracepoints: add support for relative references Date: Wed, 27 Dec 2017 08:50:30 +0000 Message-Id: <20171227085033.22389-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org To avoid the need for relocating absolute references to tracepoint structures at boot time when running relocatable kernels (which may take a disproportionate amount of space), add the option to emit these tables as relative references instead. Cc: Steven Rostedt Cc: Ingo Molnar Signed-off-by: Ard Biesheuvel --- include/linux/tracepoint.h | 19 ++++++-- kernel/tracepoint.c | 50 +++++++++++--------- 2 files changed, 42 insertions(+), 27 deletions(-) -- 2.11.0 Acked-by: Steven Rostedt (VMware) diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index a26ffbe09e71..d02bf1a695e8 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -228,6 +228,19 @@ extern void syscall_unregfunc(void); return static_key_false(&__tracepoint_##name.key); \ } +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#define __TRACEPOINT_ENTRY(name) \ + asm(" .section \"__tracepoints_ptrs\", \"a\" \n" \ + " .balign 4 \n" \ + " .long " VMLINUX_SYMBOL_STR(__tracepoint_##name) " - .\n" \ + " .previous \n") +#else +#define __TRACEPOINT_ENTRY(name) \ + static struct tracepoint * const __tracepoint_ptr_##name __used \ + __attribute__((section("__tracepoints_ptrs"))) = \ + &__tracepoint_##name +#endif + /* * We have no guarantee that gcc and the linker won't up-align the tracepoint * structures, so we create an array of pointers that will be used for iteration @@ -237,11 +250,9 @@ extern void syscall_unregfunc(void); static const char __tpstrtab_##name[] \ __attribute__((section("__tracepoints_strings"))) = #name; \ struct tracepoint __tracepoint_##name \ - __attribute__((section("__tracepoints"))) = \ + __attribute__((section("__tracepoints"), used)) = \ { __tpstrtab_##name, STATIC_KEY_INIT_FALSE, reg, unreg, NULL };\ - static struct tracepoint * const __tracepoint_ptr_##name __used \ - __attribute__((section("__tracepoints_ptrs"))) = \ - &__tracepoint_##name; + __TRACEPOINT_ENTRY(name); #define DEFINE_TRACE(name) \ DEFINE_TRACE_FN(name, NULL, NULL); diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 685c50ae6300..05649fef106c 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -327,6 +327,28 @@ int tracepoint_probe_unregister(struct tracepoint *tp, void *probe, void *data) } EXPORT_SYMBOL_GPL(tracepoint_probe_unregister); +static void for_each_tracepoint_range(struct tracepoint * const *begin, + struct tracepoint * const *end, + void (*fct)(struct tracepoint *tp, void *priv), + void *priv) +{ + if (!begin) + return; + + if (IS_ENABLED(CONFIG_HAVE_ARCH_PREL32_RELOCATIONS)) { + const int *iter; + + for (iter = (const int *)begin; iter < (const int *)end; iter++) + fct((struct tracepoint *)((unsigned long)iter + *iter), + priv); + } else { + struct tracepoint * const *iter; + + for (iter = begin; iter < end; iter++) + fct(*iter, priv); + } +} + #ifdef CONFIG_MODULES bool trace_module_has_bad_taint(struct module *mod) { @@ -391,15 +413,9 @@ EXPORT_SYMBOL_GPL(unregister_tracepoint_module_notifier); * Ensure the tracer unregistered the module's probes before the module * teardown is performed. Prevents leaks of probe and data pointers. */ -static void tp_module_going_check_quiescent(struct tracepoint * const *begin, - struct tracepoint * const *end) +static void tp_module_going_check_quiescent(struct tracepoint *tp, void *priv) { - struct tracepoint * const *iter; - - if (!begin) - return; - for (iter = begin; iter < end; iter++) - WARN_ON_ONCE((*iter)->funcs); + WARN_ON_ONCE(tp->funcs); } static int tracepoint_module_coming(struct module *mod) @@ -450,8 +466,9 @@ static void tracepoint_module_going(struct module *mod) * Called the going notifier before checking for * quiescence. */ - tp_module_going_check_quiescent(mod->tracepoints_ptrs, - mod->tracepoints_ptrs + mod->num_tracepoints); + for_each_tracepoint_range(mod->tracepoints_ptrs, + mod->tracepoints_ptrs + mod->num_tracepoints, + tp_module_going_check_quiescent, NULL); break; } } @@ -503,19 +520,6 @@ static __init int init_tracepoints(void) __initcall(init_tracepoints); #endif /* CONFIG_MODULES */ -static void for_each_tracepoint_range(struct tracepoint * const *begin, - struct tracepoint * const *end, - void (*fct)(struct tracepoint *tp, void *priv), - void *priv) -{ - struct tracepoint * const *iter; - - if (!begin) - return; - for (iter = begin; iter < end; iter++) - fct(*iter, priv); -} - /** * for_each_kernel_tracepoint - iteration on all kernel tracepoints * @fct: callback From patchwork Wed Dec 27 08:50:31 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122763 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1920158qgn; Wed, 27 Dec 2017 00:51:34 -0800 (PST) X-Google-Smtp-Source: ACJfBouec0PANs1KcsFqOomDmYBA12bhsfJfJY/MGIeNCYuoRITRwbWliQk0nPNzl7luNxM07mcW X-Received: by 10.84.128.47 with SMTP id 44mr27717320pla.114.1514364694811; Wed, 27 Dec 2017 00:51:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364694; cv=none; d=google.com; s=arc-20160816; b=rbmDSiNLSDXQIjfySlO+Lcj1C8lWlxgTeZZjVDOfQ/my0hssj7JtiWS7YexQcD3dX8 Ab9UBp5Qpzz9H43WJHMQ0WXZB68Cr/g7n5PU/Ixm8QZmnxNEFmQHZzE/91Uv3uXD92qr GR/72xs5QCQjXy4m7NSiHa/4ubmxocopujlm6GgqqJ8QBOhggKA0B+yGP4WnVh5QLnnQ ZHEP89otJBhb4pNWDqE29V9Pi8VWlTAQvQ/22sWqciGGU1l4Bet5VC8D0R8jwQAWgMoZ M9mMav8qcnKF3cATDko3AicUp5AiKFL/O+UR0ZDJ8nOwe44UFW7N9rhOUZE4XLtVe8ZR ZZMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=AUOuipd04iXjbyQ7tRijtnVIvzezHl+DybY8/NUX2wA=; b=PFM9qBz34nHKi4F3dGvNQT3BdJbH1MhOK85u3mFxKVL+fdaffM8OGS4FyxsNz33H11 07RbHbW0z0HKra8kDCc9SNAafM9FfCIB5vVF3uffIJ/Mz6snZVYDQOuo27DHVmatiNXa WKTaJ/bF0hsla6fPLN4yWSLMds8g/2RvozDp41cmlYcKCrZtTmagjGe7Q2RfHJifmoTr 1horYvycSUj505uyOfG9hsnV3SRWBhhhAAttSVf7kwIkG3mbZzhmFV6HqQl446J6fGp2 vDs2H7BZmlRb5knQOF9A78EXIJLYYljRv6egEdfuZ5P1yRKbiI41dOQas/k2ExX6VcMX A9pQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Su8E/6dM; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 31si1458344pla.359.2017.12.27.00.51.34; Wed, 27 Dec 2017 00:51:34 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Su8E/6dM; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751931AbdL0Iv2 (ORCPT + 28 others); Wed, 27 Dec 2017 03:51:28 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:34234 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751915AbdL0IvX (ORCPT ); Wed, 27 Dec 2017 03:51:23 -0500 Received: by mail-wm0-f67.google.com with SMTP id y82so38517104wmg.1 for ; Wed, 27 Dec 2017 00:51:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AUOuipd04iXjbyQ7tRijtnVIvzezHl+DybY8/NUX2wA=; b=Su8E/6dM4Vy7DKySqRJ3l4jpuAH4x7hmWnhUdVPEHgsUx1cAjV/QWCVdf3KmuWxfXJ if75rf7tsEcy9R3gldRubinvQYj/oP7SCk7bThryphMiYE9Vjb6cnNRsNzfE2a+YSFKM iq/frQeSsGa0mkmKE0McBrZfcqMxwzJ9WY4Do= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AUOuipd04iXjbyQ7tRijtnVIvzezHl+DybY8/NUX2wA=; b=WqQLNzi112uHCSL5J6cpUkDysTvudwcNVkRJjmxgS5r93/ctuuK/nf8ZrcJUwJEkr+ RFeh4S134Y1ahpIEpSfdjMDGa8SqgtGKgSFxxDPL1udDhKU7JpkSEqD29fRopeRVGPwB RtHKGYvPFQ/g7yYNU0qaSAWpx5+hRGi7rJXTXpRMayq8xuti8X/j+2wQl7LZLUfxEsz9 wuQLrMGHRHnoAtrsCOchGQ2i69jz9Azbyzeb7d7XX8QTSIaBHBBBohZxN7R439bmA3WM iLpubou3V84Obyptpmm9FP8LRfoYjyybnqEwDiiukZssmLkCVikhY0iCUmqKNDip2T2N 5+1A== X-Gm-Message-State: AKGB3mLorVhOAhH/hwydT0uoO8WH+nX1crvV7nVLH+1Rni8WeRY+ZDBZ raD6l318N0YABTAFdhb7kfDXALbv3GA= X-Received: by 10.28.220.215 with SMTP id t206mr22256701wmg.75.1514364681622; Wed, 27 Dec 2017 00:51:21 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.51.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:51:20 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v6 6/8] kernel/jump_label: abstract jump_entry member accessors Date: Wed, 27 Dec 2017 08:50:31 +0000 Message-Id: <20171227085033.22389-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In preparation of allowing architectures to use relative references in jump_label entries [which can dramatically reduce the memory footprint], introduce abstractions for references to the 'code' and 'key' members of struct jump_entry. Signed-off-by: Ard Biesheuvel --- arch/arm/include/asm/jump_label.h | 27 ++++++++++++++ arch/arm64/include/asm/jump_label.h | 27 ++++++++++++++ arch/mips/include/asm/jump_label.h | 27 ++++++++++++++ arch/powerpc/include/asm/jump_label.h | 27 ++++++++++++++ arch/s390/include/asm/jump_label.h | 20 +++++++++++ arch/sparc/include/asm/jump_label.h | 27 ++++++++++++++ arch/tile/include/asm/jump_label.h | 27 ++++++++++++++ arch/x86/include/asm/jump_label.h | 27 ++++++++++++++ kernel/jump_label.c | 38 +++++++++----------- 9 files changed, 225 insertions(+), 22 deletions(-) -- 2.11.0 diff --git a/arch/arm/include/asm/jump_label.h b/arch/arm/include/asm/jump_label.h index e12d7d096fc0..7b05b404063a 100644 --- a/arch/arm/include/asm/jump_label.h +++ b/arch/arm/include/asm/jump_label.h @@ -45,5 +45,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 1b5e0e843c3a..9d6e46355c89 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -62,5 +62,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/mips/include/asm/jump_label.h b/arch/mips/include/asm/jump_label.h index e77672539e8e..70df9293dc49 100644 --- a/arch/mips/include/asm/jump_label.h +++ b/arch/mips/include/asm/jump_label.h @@ -66,5 +66,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif /* _ASM_MIPS_JUMP_LABEL_H */ diff --git a/arch/powerpc/include/asm/jump_label.h b/arch/powerpc/include/asm/jump_label.h index 9a287e0ac8b1..412b2699c9f6 100644 --- a/arch/powerpc/include/asm/jump_label.h +++ b/arch/powerpc/include/asm/jump_label.h @@ -59,6 +59,33 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #else #define ARCH_STATIC_BRANCH(LABEL, KEY) \ 1098: nop; \ diff --git a/arch/s390/include/asm/jump_label.h b/arch/s390/include/asm/jump_label.h index 40f651292aa7..3d4a08e9514b 100644 --- a/arch/s390/include/asm/jump_label.h +++ b/arch/s390/include/asm/jump_label.h @@ -50,5 +50,25 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline jump_label_t jump_entry_key(const struct jump_entry *entry) +{ + return entry->key; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/sparc/include/asm/jump_label.h b/arch/sparc/include/asm/jump_label.h index 94eb529dcb77..18e893687f7c 100644 --- a/arch/sparc/include/asm/jump_label.h +++ b/arch/sparc/include/asm/jump_label.h @@ -48,5 +48,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/tile/include/asm/jump_label.h b/arch/tile/include/asm/jump_label.h index cde7573f397b..86acaa6ff33d 100644 --- a/arch/tile/include/asm/jump_label.h +++ b/arch/tile/include/asm/jump_label.h @@ -55,4 +55,31 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* _ASM_TILE_JUMP_LABEL_H */ diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 8c0de4282659..009ff2699d07 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -74,6 +74,33 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #else /* __ASSEMBLY__ */ .macro STATIC_JUMP_IF_TRUE target, key, def diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 8594d24e4adc..4f44db58d981 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -37,10 +37,12 @@ static int jump_label_cmp(const void *a, const void *b) const struct jump_entry *jea = a; const struct jump_entry *jeb = b; - if (jea->key < jeb->key) + if ((unsigned long)jump_entry_key(jea) < + (unsigned long)jump_entry_key(jeb)) return -1; - if (jea->key > jeb->key) + if ((unsigned long)jump_entry_key(jea) > + (unsigned long)jump_entry_key(jeb)) return 1; return 0; @@ -53,7 +55,8 @@ jump_label_sort_entries(struct jump_entry *start, struct jump_entry *stop) size = (((unsigned long)stop - (unsigned long)start) / sizeof(struct jump_entry)); - sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL); + sort(start, size, sizeof(struct jump_entry), jump_label_cmp, + jump_label_swap); } static void jump_label_update(struct static_key *key); @@ -254,8 +257,8 @@ EXPORT_SYMBOL_GPL(jump_label_rate_limit); static int addr_conflict(struct jump_entry *entry, void *start, void *end) { - if (entry->code <= (unsigned long)end && - entry->code + JUMP_LABEL_NOP_SIZE > (unsigned long)start) + if (jump_entry_code(entry) <= (unsigned long)end && + jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE > (unsigned long)start) return 1; return 0; @@ -314,16 +317,6 @@ static inline void static_key_set_linked(struct static_key *key) key->type |= JUMP_TYPE_LINKED; } -static inline struct static_key *jump_entry_key(struct jump_entry *entry) -{ - return (struct static_key *)((unsigned long)entry->key & ~1UL); -} - -static bool jump_entry_branch(struct jump_entry *entry) -{ - return (unsigned long)entry->key & 1UL; -} - /*** * A 'struct static_key' uses a union such that it either points directly * to a table of 'struct jump_entry' or to a linked list of modules which in @@ -348,7 +341,7 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool enabled = static_key_enabled(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return enabled ^ branch; @@ -364,7 +357,8 @@ static void __jump_label_update(struct static_key *key, * kernel_text_address() verifies we are not in core kernel * init code, see jump_label_invalidate_module_init(). */ - if (entry->code && kernel_text_address(entry->code)) + if (!jump_entry_is_module_init(entry) && + kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); } } @@ -417,7 +411,7 @@ static enum jump_label_type jump_label_init_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool type = static_key_type(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return type ^ branch; @@ -541,7 +535,7 @@ static int jump_label_add_module(struct module *mod) continue; key = iterk; - if (within_module(iter->key, mod)) { + if (within_module((unsigned long)key, mod)) { static_key_set_entries(key, iter); continue; } @@ -591,7 +585,7 @@ static void jump_label_del_module(struct module *mod) key = jump_entry_key(iter); - if (within_module(iter->key, mod)) + if (within_module((unsigned long)key, mod)) continue; /* No memory during module load */ @@ -634,8 +628,8 @@ static void jump_label_invalidate_module_init(struct module *mod) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(iter->code, mod)) - iter->code = 0; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_module_init(iter); } } From patchwork Wed Dec 27 08:50:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122764 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1920191qgn; Wed, 27 Dec 2017 00:51:36 -0800 (PST) X-Google-Smtp-Source: ACJfBosHdhzTXo7PFjMEhqzx1FQtIjvmA1qWN7nsUwKfjQtcCiDlW/x6AD7MTPruCO+u1UfAG2K5 X-Received: by 10.84.247.2 with SMTP id n2mr27666110pll.268.1514364696622; Wed, 27 Dec 2017 00:51:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364696; cv=none; d=google.com; s=arc-20160816; b=p8DwrekSQSo6AwmHpNimPOf7H6442dYvSWi7+qRfRVpHYWkVorb3/mRaFiHV59SlVS LYHPH7oi/f1G+Tz+6thxmhiqXAtA9Z4qsgH2YkZAynKOkZQIbZSXv7xNfULmzdZ+1CyK 8jKmY49ykwRD9a6PDxkZn9HUjz61niuJJCuioaBGoaMdT8zaSUm2tM8ONviB0HJ+e3bZ dgI7W7ep2zxYQ8zj2WYRMyttNxteRJdQMKGBY1exqica9jjw97Qzg4JUxqXgqV3AhHWB ABf0xIW2pBd0UciQY/7TC8YeH/nf2/17nv6NZqqHb1iH3rWk+tZq78J9fP0kGfInDFwM xovg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=nmovwwxZQQa2ugEDLnXxoU69CsqWUnhsDto5qVfQljI=; b=o5lRHVM3uD1s4Df9T1HjmjwlTNX7iuKEPQefhNsGK5JTGHj5sdcoH7J27nu9Jnkl6q GgEJdBruYoTfY39xcWhsBDHDm0r8xGVjYsgQJar+JhLpQYi1DrUjUkbSlqvO20LpaCNC FpFGQRFa5entE9Rdjtr9vPfjgXQGOiOgCzwFSnJCkCm6tT9Iq7xhZPB/of2+X4TBpxWE ns2baB4kUv9vL8Qr+uedk4PgpfIhMTepYODOYlbcymdMzzvsUtkC25neKKI70wkwIpT8 LsMH0r1b0wNh0bdw7Mv07fsYTv+h6t0AYywuciGrn4pLMlNQlAktS2OD21Ts8IhvwzYj FrVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WxlwI5eY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 31si1458344pla.359.2017.12.27.00.51.36; Wed, 27 Dec 2017 00:51:36 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WxlwI5eY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751964AbdL0Ivd (ORCPT + 28 others); Wed, 27 Dec 2017 03:51:33 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:32891 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751934AbdL0Iv3 (ORCPT ); Wed, 27 Dec 2017 03:51:29 -0500 Received: by mail-wm0-f67.google.com with SMTP id g130so38648674wme.0 for ; Wed, 27 Dec 2017 00:51:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nmovwwxZQQa2ugEDLnXxoU69CsqWUnhsDto5qVfQljI=; b=WxlwI5eYdKaO3NmcGBbfQz+LK6UKVWOFoMQ+Jyib+BEmRa2fOBlwIMLzNism5cTraH jwShntvWLc1MLEshXs0fqjK4udpfG/RyCibzZzw0w/no0a/0bNsUe8LbcE4bd6Bxc4nm APC55TMa+SgyxGz/NqB151Pg5zBZdpBRBlez4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nmovwwxZQQa2ugEDLnXxoU69CsqWUnhsDto5qVfQljI=; b=UKZ2KDct0eWgOx4jo2h7qZWCZlAANBVcLF0nyIzCSp8+3VuUJqMmokLRyExonOu0Tu egXusYfQh2u7E9WMIFYNuoQVaLItzzc//BtI+VAD3KH70j/g+Y/EzOxK6LTAReIA96Kq 3keJcyMy1LSGLP/t9ITEBIdxc4IBsGJGB3YZB0pShzIjl9uWsKw5XNCWdm3HWbtbyyeD Q7WJUVHcK1YL1dlgGQasqNMIF5K8S/WIAcianzL9/UeVCXOoakjj+B+Ymr0vwgx5mFXV Divx/zd7d0IYpHEpcGV0M4S2h9bmxQTzz+V0BvfKjTMvVfEQ3xDyJrGx+eweKw4ovXen GEqQ== X-Gm-Message-State: AKGB3mLQ8+cTnpWf26cDN1iAcUQxb6jhOXNGq0PdvPRWKNQEWAMURpd8 4yf9lJAmfqr+TkHpLs+LlYA+SL9aL+k= X-Received: by 10.28.106.16 with SMTP id f16mr13613978wmc.58.1514364687433; Wed, 27 Dec 2017 00:51:27 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.51.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:51:26 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v6 7/8] arm64/kernel: jump_label: use relative references Date: Wed, 27 Dec 2017 08:50:32 +0000 Message-Id: <20171227085033.22389-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On a randomly chosen distro kernel build for arm64, vmlinux.o shows the following sections, containing jump label entries, and the associated RELA relocation records, respectively: ... [38088] __jump_table PROGBITS 0000000000000000 00e19f30 000000000002ea10 0000000000000000 WA 0 0 8 [38089] .rela__jump_table RELA 0000000000000000 01fd8bb0 000000000008be30 0000000000000018 I 38178 38088 8 ... In other words, we have 190 KB worth of 'struct jump_entry' instances, and 573 KB worth of RELA entries to relocate each entry's code, target and key members. This means the RELA section occupies 10% of the .init segment, and the two sections combined represent 5% of vmlinux's entire memory footprint. So let's switch from 64-bit absolute references to 32-bit relative references: this reduces the size of the __jump_table by 50%, and gets rid of the RELA section entirely. Note that this requires some extra care in the sorting routine, given that the offsets change when entries are moved around in the jump_entry table. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/jump_label.h | 27 ++++++++++++-------- arch/arm64/kernel/jump_label.c | 22 +++++++++++++--- 2 files changed, 36 insertions(+), 13 deletions(-) -- 2.11.0 diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 9d6e46355c89..5cec68616125 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -30,8 +30,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm goto("1: nop\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -44,8 +44,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool { asm goto("1: b %l[l_yes]\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -57,19 +57,26 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool typedef u64 jump_label_t; struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (jump_label_t)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (jump_label_t)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -87,7 +94,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index c2dd1ad3e648..2b8e459e91f7 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -25,12 +25,12 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - void *addr = (void *)entry->code; + void *addr = (void *)jump_entry_code(entry); u32 insn; if (type == JUMP_LABEL_JMP) { - insn = aarch64_insn_gen_branch_imm(entry->code, - entry->target, + insn = aarch64_insn_gen_branch_imm(jump_entry_code(entry), + jump_entry_target(entry), AARCH64_INSN_BRANCH_NOLINK); } else { insn = aarch64_insn_gen_nop(); @@ -50,4 +50,20 @@ void arch_jump_label_transform_static(struct jump_entry *entry, */ } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif /* HAVE_JUMP_LABEL */ From patchwork Wed Dec 27 08:50:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122765 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1920285qgn; Wed, 27 Dec 2017 00:51:44 -0800 (PST) X-Google-Smtp-Source: ACJfBouKUjseEOBxotv55QHFQp/+1fHZBF/Hf4ljUNIBfiZB7Fgq5YLJyvrKIH4nBxDBTJVE5suM X-Received: by 10.159.207.145 with SMTP id z17mr27598023plo.269.1514364704097; Wed, 27 Dec 2017 00:51:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514364704; cv=none; d=google.com; s=arc-20160816; b=E2g4PsVsqGC8ia9/s2kJ7PtY+wip6WYqH1q8HYE1mcYxRv/g/O8iVWSUq5FoC27Wf/ Evd+0lzzVLZZjZ4NqnxbQbwfrXqripHxH6bPdSwPI8QvXWHeW7hzDU+gVyjGrJwc5KHb U+VakR28UOVVwkqUo8/9oqaqiRWpaigUN6aLXvtm5qgeYFoc1GVReBtPKOwxh1o/LZzt joohd4hj7FHuyn4KPxfl0RuHbokGW5ZXta8V7JRlqo8K4bMGFliJwVVkdjc8Qz+o+K8p oX/NdxlwGTPVfcRJsfbK+6uBsjKonUaxqc9LHQLsUSLTRbmq5q6tnI6RhlgqGgvF4Oy1 hVsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=zBo/6EncMhQE+u7uWHtusHrsjwb/Xhu+WleYCGKp7Zw=; b=zElWAjGbKJvErNldfH350LpsmzZavWYaZeW5i2aQ41tRUI2comiihLs1m/Q6v+FgJZ MPiUTJKF6Qyi0RnV3xwkpDjBEBu2Hd/C45KMtuKeDYqQqJnaF9OHo4AIS+qSDjgvPy5J JmdX2xwTmf0ixSMNpDmKmfNTn7e3YnGp23iYTxRs+MUd3EzAUw5rPZMxwI0gcIlGdzYj GtSgVwEvMltVdgrhARjU5ObrsCkvgf9bhhSDKDAU+kHT+p74bdAS8+g3wQNenYF8n9Pe NUvRk0qKPNVfG+irBqWxOgApr4Tlt09YMkxqNeIDLO+jzwkL3litIbE8gunOTJCAtRYu BYCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YDuC1/ee; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j4si24477080pfc.22.2017.12.27.00.51.43; Wed, 27 Dec 2017 00:51:44 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YDuC1/ee; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751983AbdL0Ivj (ORCPT + 28 others); Wed, 27 Dec 2017 03:51:39 -0500 Received: from mail-wm0-f65.google.com ([74.125.82.65]:46979 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751934AbdL0Ive (ORCPT ); Wed, 27 Dec 2017 03:51:34 -0500 Received: by mail-wm0-f65.google.com with SMTP id r78so38448234wme.5 for ; Wed, 27 Dec 2017 00:51:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zBo/6EncMhQE+u7uWHtusHrsjwb/Xhu+WleYCGKp7Zw=; b=YDuC1/eeZUOl9wiL7YxX5dzXK0PSaQTy3t4NObHTF4xOFkUGsRmXQq8KAG7NRYYRGH yvFc+Kxhj7WKoU4/JjOhY3SOKcgu16yIUO32QBwK2XFsI/VBDg+H0zfroSAdow5nrDgc Pp3jK9/mizaywl6ACAMCQbDUwGaVDmnL3yJEw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zBo/6EncMhQE+u7uWHtusHrsjwb/Xhu+WleYCGKp7Zw=; b=JQMkRB6wsyljEACsWNqbL3htQZmQxCcJoTBoy6gzj5Sbs5iN1qljJOV4wbMQxiZZOR yRCzAjRPPMSfLrPgizfwBiUZ8f/ftj5d7N3xjb8tRNAfmRPM0aRgH4NG+ImZM5BTsGpg k0Sq+024+BCEujY1SGx4TRAeSR8DdagUVTqNfrBMo0GWB7KNQnzZEFm1/kHlTUs8tYqw PWWl2+4OtwcIhyuSXE0Y0CG1PUMvBCmozo1/NsAyRWN9oeHp7swsEcOcoFICjip0oC8u 499fbCZpXVdgrP/AjOrAjpDEmGBaL+XMJ7RZz/OJacaiEdIwBmxxVOL2sKc64X47RvYR wYzA== X-Gm-Message-State: AKGB3mKRIcU/+7+4TJ2Y3IDQaEuTCtHhdlDg5kZXNAtMvRtggzAMWyze r+SgH/3IOeMvh8tyAbwts815pWsDMyk= X-Received: by 10.28.4.146 with SMTP id 140mr21020116wme.38.1514364692449; Wed, 27 Dec 2017 00:51:32 -0800 (PST) Received: from localhost.localdomain ([105.137.110.132]) by smtp.gmail.com with ESMTPSA id q74sm32677226wmg.22.2017.12.27.00.51.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Dec 2017 00:51:31 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , "H. Peter Anvin" , Ralf Baechle , Arnd Bergmann , Heiko Carstens , Kees Cook , Will Deacon , Michael Ellerman , Thomas Garnier , Thomas Gleixner , "Serge E. Hallyn" , Bjorn Helgaas , Benjamin Herrenschmidt , Russell King , Paul Mackerras , Catalin Marinas , "David S. Miller" , Petr Mladek , Ingo Molnar , James Morris , Andrew Morton , Nicolas Pitre , Josh Poimboeuf , Steven Rostedt , Martin Schwidefsky , Sergey Senozhatsky , Linus Torvalds , Jessica Yu , linux-arm-kernel@lists.infradead.org, linux-mips@linux-mips.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v6 8/8] x86/kernel: jump_table: use relative references Date: Wed, 27 Dec 2017 08:50:33 +0000 Message-Id: <20171227085033.22389-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171227085033.22389-1-ard.biesheuvel@linaro.org> References: <20171227085033.22389-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Similar to the arm64 case, 64-bit x86 can benefit from using 32-bit relative references rather than 64-bit absolute ones when emitting struct jump_entry instances. Not only does this reduce the memory footprint of the entries themselves by 50%, it also removes the need for carrying relocation metadata on relocatable builds (i.e., for KASLR) which saves a fair chunk of .init space as well (although the savings are not as dramatic as on arm64) Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/jump_label.h | 35 +++++++----- arch/x86/kernel/jump_label.c | 59 ++++++++++++++------ tools/objtool/special.c | 4 +- 3 files changed, 65 insertions(+), 33 deletions(-) -- 2.11.0 diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 009ff2699d07..91c01af96907 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -36,8 +36,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran asm_volatile_goto("1:" ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -52,8 +52,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool ".byte 0xe9\n\t .long %l[l_yes] - 2f\n\t" "2:\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -69,19 +69,26 @@ typedef u32 jump_label_t; #endif struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (jump_label_t)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (jump_label_t)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -99,7 +106,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #else /* __ASSEMBLY__ */ @@ -114,8 +121,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .byte STATIC_KEY_INIT_NOP .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key - . .popsection .endm @@ -130,8 +137,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .Lstatic_jump_after_\@: .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key - . + 1 .popsection .endm diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e56c95be2808..cc5034b42335 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -52,22 +52,24 @@ static void __jump_label_transform(struct jump_entry *entry, * Jump label is enabled for the first time. * So we expect a default_nop... */ - if (unlikely(memcmp((void *)entry->code, default_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { /* * ...otherwise expect an ideal_nop. Otherwise * something went horribly wrong. */ - if (unlikely(memcmp((void *)entry->code, ideal_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + ideal_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); } else { /* * We are disabling this jump label. If it is not what @@ -76,14 +78,18 @@ static void __jump_label_transform(struct jump_entry *entry, * are converting the default nop to the ideal nop. */ if (init) { - if (unlikely(memcmp((void *)entry->code, default_nop, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); - if (unlikely(memcmp((void *)entry->code, &code, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); + if (unlikely(memcmp((void *)jump_entry_code(entry), + &code, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } memcpy(&code, ideal_nops[NOP_ATOMIC5], JUMP_LABEL_NOP_SIZE); } @@ -97,10 +103,13 @@ static void __jump_label_transform(struct jump_entry *entry, * */ if (poker) - (*poker)((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE); + (*poker)((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE); else - text_poke_bp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE, - (void *)entry->code + JUMP_LABEL_NOP_SIZE); + text_poke_bp((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE, + (void *)jump_entry_code(entry) + + JUMP_LABEL_NOP_SIZE); } void arch_jump_label_transform(struct jump_entry *entry, @@ -140,4 +149,20 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, __jump_label_transform(entry, type, text_poke_early, 1); } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 84f001d52322..98ae55b39037 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -30,9 +30,9 @@ #define EX_ORIG_OFFSET 0 #define EX_NEW_OFFSET 4 -#define JUMP_ENTRY_SIZE 24 +#define JUMP_ENTRY_SIZE 12 #define JUMP_ORIG_OFFSET 0 -#define JUMP_NEW_OFFSET 8 +#define JUMP_NEW_OFFSET 4 #define ALT_ENTRY_SIZE 13 #define ALT_ORIG_OFFSET 0