From patchwork Tue Oct 29 16:58:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 178046 Delivered-To: patch@linaro.org Received: by 2002:a92:409a:0:0:0:0:0 with SMTP id d26csp67218ill; Tue, 29 Oct 2019 09:58:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqwlsKCipRKQp135du2supmxofPO03eDd+MOgFDyMofpyMhDJuLA0TFbfXBege67C1YrFRpx X-Received: by 2002:a50:ab50:: with SMTP id t16mr27012858edc.171.1572368330617; Tue, 29 Oct 2019 09:58:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572368330; cv=none; d=google.com; s=arc-20160816; b=zBUdu0ss/FhJ/wA9KJbs4HKgIttnhGyPD/Kh8SG2Ek7Zx7ExYtXduchm8wFzRisLyK 3XS+4JhyLwvwrLdQnSIWQLD/EAjIgkZ400z987opcRxhaVlwyC6kJKCFNVt7I0DLNwTE UpVFh1QLcW7PAEhaD7Re1WeXrYO1JaX0k7UFQQeTrLUynAtes+oarYXIddooik09LunJ zfAzem/ggJ/Yd1/F0ngPnDU7qAIq/WaPOL76KNCRtrfO4jO59sL8aVatvjAqdvKzguDn IISB3bBd43yL3DyiUAsHJPBQp4AbshOGXF77LOspT9JooPNglrgjw+sPk9VOdu6Wy+CV p65w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=QdkSYnKrCuYhJ3tUiAND7/60PlI0VedHoAaUzR1hZaU=; b=pRU1bbg6hhivVuttKJ1WqZGhLK+TOV7u1GI1xPtOh/gOsbFo2HxERzyMARy2VeHtnb BGEPDgVdSVYXAHVoW6He9/3eFXKgiRgf6ygrclkCd87NtsAGppBWDkZvEmInLG5O2tM9 X2kTl9QTy1O3a1L6eXDqK583FhB5kzAm1LqQaK90u3Xoroht36fcfDdYC2LrZkdpzP90 gou/hWFfDP2FJ8XmJNenB0vB9cdVmMkbFnlX78k2Tnui9c5U/gYjLJBvUe3V/zeETXLN 95Ck1JJOD0mlY7uJmyBmLhCirwd1sI81fg+Qvv5Q6XaW0kz+PDxI9ObMq2TtpAab4cxK g/eQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l12si6020128edk.60.2019.10.29.09.58.50; Tue, 29 Oct 2019 09:58:50 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390637AbfJ2Q6q (ORCPT + 26 others); Tue, 29 Oct 2019 12:58:46 -0400 Received: from foss.arm.com ([217.140.110.172]:54558 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727279AbfJ2Q6p (ORCPT ); Tue, 29 Oct 2019 12:58:45 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1F44C337; Tue, 29 Oct 2019 09:58:45 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C11D13F71F; Tue, 29 Oct 2019 09:58:42 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Steven Rostedt Cc: linux-kernel@vger.kernel.org, amit.kachhap@arm.com, catalin.marinas@arm.com, deller@gmx.de, duwe@suse.de, James.Bottomley@HansenPartnership.com, james.morse@arm.com, jeyu@kernel.org, jpoimboe@redhat.com, jthierry@redhat.com, linux-parisc@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, svens@stackframe.org, takahiro.akashi@linaro.org, will@kernel.org Subject: [PATCHv2 1/8] ftrace: add ftrace_init_nop() Date: Tue, 29 Oct 2019 16:58:25 +0000 Message-Id: <20191029165832.33606-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191029165832.33606-1-mark.rutland@arm.com> References: <20191029165832.33606-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Architectures may need to perform special initialization of ftrace callsites, and today they do so by special-casing ftrace_make_nop() when the expected branch address is MCOUNT_ADDR. In some cases (e.g. for patchable-function-entry), we don't have an mcount-like symbol and don't want a synthetic MCOUNT_ADDR, but we may need to perform some initialization of callsites. To make it possible to separate initialization from runtime modification, and to handle cases without an mcount-like symbol, this patch adds an optional ftrace_init_nop() function that architectures can implement, which does not pass a branch address. Where an architecture does not provide ftrace_init_nop(), we will fall back to the existing behaviour of calling ftrace_make_nop() with MCOUNT_ADDR. At the same time, ftrace_code_disable() is renamed to ftrace_nop_initialize() to make it clearer that it is intended to intialize a callsite into a disabled state, and is not for disabling a callsite that has been runtime enabled. The kerneldoc description of rec arguments is updated to cover non-mcount callsites. Signed-off-by: Mark Rutland Reviewed-by: Ard Biesheuvel Cc: Ingo Molnar Cc: Steven Rostedt Cc: Torsten Duwe --- include/linux/ftrace.h | 35 ++++++++++++++++++++++++++++++++--- kernel/trace/ftrace.c | 6 +++--- 2 files changed, 35 insertions(+), 6 deletions(-) -- 2.11.0 Reviewed-by: Miroslav Benes Reviewed-by: Steven Rostedt (VMware) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 8a8cb3c401b2..9867d90d635e 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -499,7 +499,7 @@ static inline int ftrace_disable_ftrace_graph_caller(void) { return 0; } /** * ftrace_make_nop - convert code into nop * @mod: module structure if called by module load initialization - * @rec: the mcount call site record + * @rec: the call site record (e.g. mcount/fentry) * @addr: the address that the call site should be calling * * This is a very sensitive operation and great care needs @@ -520,9 +520,38 @@ static inline int ftrace_disable_ftrace_graph_caller(void) { return 0; } extern int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr); + +/** + * ftrace_init_nop - initialize a nop call site + * @mod: module structure if called by module load initialization + * @rec: the call site record (e.g. mcount/fentry) + * + * This is a very sensitive operation and great care needs + * to be taken by the arch. The operation should carefully + * read the location, check to see if what is read is indeed + * what we expect it to be, and then on success of the compare, + * it should write to the location. + * + * The code segment at @rec->ip should contain the contents created by + * the compiler + * + * Return must be: + * 0 on success + * -EFAULT on error reading the location + * -EINVAL on a failed compare of the contents + * -EPERM on error writing to the location + * Any other value will be considered a failure. + */ +#ifndef ftrace_init_nop +static inline int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) +{ + return ftrace_make_nop(mod, rec, MCOUNT_ADDR); +} +#endif + /** * ftrace_make_call - convert a nop call site into a call to addr - * @rec: the mcount call site record + * @rec: the call site record (e.g. mcount/fentry) * @addr: the address that the call site should call * * This is a very sensitive operation and great care needs @@ -545,7 +574,7 @@ extern int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr); #ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS /** * ftrace_modify_call - convert from one addr to another (no nop) - * @rec: the mcount call site record + * @rec: the call site record (e.g. mcount/fentry) * @old_addr: the address expected to be currently called to * @addr: the address to change to * diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index f296d89be757..5259d4dea675 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2494,14 +2494,14 @@ struct dyn_ftrace *ftrace_rec_iter_record(struct ftrace_rec_iter *iter) } static int -ftrace_code_disable(struct module *mod, struct dyn_ftrace *rec) +ftrace_nop_initialize(struct module *mod, struct dyn_ftrace *rec) { int ret; if (unlikely(ftrace_disabled)) return 0; - ret = ftrace_make_nop(mod, rec, MCOUNT_ADDR); + ret = ftrace_init_nop(mod, rec); if (ret) { ftrace_bug_type = FTRACE_BUG_INIT; ftrace_bug(ret, rec); @@ -2943,7 +2943,7 @@ static int ftrace_update_code(struct module *mod, struct ftrace_page *new_pgs) * to the NOP instructions. */ if (!__is_defined(CC_USING_NOP_MCOUNT) && - !ftrace_code_disable(mod, p)) + !ftrace_nop_initialize(mod, p)) break; update_cnt++; From patchwork Tue Oct 29 16:58:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 178047 Delivered-To: patch@linaro.org Received: by 2002:a92:409a:0:0:0:0:0 with SMTP id d26csp67304ill; Tue, 29 Oct 2019 09:58:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqwuN9RhFHKtX0U95IL4+ZO5gqxVvJOBmMwwn757gFNQLRnxWLyHl9qNCtBLH5/xwBxnnop1 X-Received: by 2002:a17:906:1949:: with SMTP id b9mr4304289eje.277.1572368336055; Tue, 29 Oct 2019 09:58:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572368336; cv=none; d=google.com; s=arc-20160816; b=UT0ub3uRJyxiQpWR3McgebcJNe6iK0E6wNSA/LysDNWWme2YHEZRBFzal50wsZU6z/ ZDNKelOd/DX56vaixUHJKlQTLueXW25GkQYdvKJ2/2qrjwnb3DUQKYgVAi2WPoMZEbCk Qpyz+NwxyyU7T/tdPs41cmbpXGpSkYrG107LvGzg6iFFUthnq/a3uhPscz4GSNTqcTTM qy/ohX9kpZrIbomrZDvrbO0xXz/IGg3kBxtfO2IwKBr7RAogQ42TLug5NefMDH/+T1xY KRrS8q6Ved2bgKMnpwHlwYJycga/KR/Wc8O09ejv1aKHbPBp/BgKq/3ohX7gPfcXrn/o FdEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=6Za8xN0gSpEbamg3mBOyVEiaxZF8KL6fU8wq9s4D91E=; b=HXdmNo2Y4xi9+cLdff1uJ+YtNTQEkLm1w/vbxB+wNUqbyGDcQIlu5ahb71tjtk9QcX 33rfzBhvKaNwfAw8vVZoNTfNqYHiGY2EyZEVV2kLVJgMU7zrT9Q7Be4E4qyfqL35l+Gp Y/fGvUh9hFL5WjGF9WK9jCY3I6papB/6ifUwc0HJNbek0GEuAB0RWGIZLDSAKT6biXwH EZOnp9GDrj9VhLHwSjIT1jLvAjIoYzkQMWAIh7B//o4RYFg1maSeD2KliesUrcfd1HE4 +QoQChsycajH9gISgkjul0BFyP83LfXtW7zQt4uD9abvACuzzbYZMJwKoz+0XdBBFgvi yi+w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i9si4497735ejz.368.2019.10.29.09.58.55; Tue, 29 Oct 2019 09:58:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390661AbfJ2Q6v (ORCPT + 26 others); Tue, 29 Oct 2019 12:58:51 -0400 Received: from foss.arm.com ([217.140.110.172]:54582 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727279AbfJ2Q6v (ORCPT ); Tue, 29 Oct 2019 12:58:51 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7A35A4AC; Tue, 29 Oct 2019 09:58:50 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0D7133F71F; Tue, 29 Oct 2019 09:58:47 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org, Jessica Yu , Helge Deller , "James E.J. Bottomley" Cc: linux-kernel@vger.kernel.org, amit.kachhap@arm.com, catalin.marinas@arm.com, duwe@suse.de, james.morse@arm.com, jpoimboe@redhat.com, jthierry@redhat.com, linux-parisc@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, rostedt@goodmis.org, svens@stackframe.org, takahiro.akashi@linaro.org, will@kernel.org Subject: [PATCHv2 2/8] module/ftrace: handle patchable-function-entry Date: Tue, 29 Oct 2019 16:58:26 +0000 Message-Id: <20191029165832.33606-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191029165832.33606-1-mark.rutland@arm.com> References: <20191029165832.33606-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When using patchable-function-entry, the compiler will record the callsites into a section named "__patchable_function_entries" rather than "__mcount_loc". Let's abstract this difference behind a new FTRACE_CALLSITE_SECTION, so that architectures don't have to handle this explicitly (e.g. with custom module linker scripts). As parisc currently handles this explicitly, it is fixed up accordingly, with its custom linker script removed. Since FTRACE_CALLSITE_SECTION is only defined when DYNAMIC_FTRACE is selected, the parisc module loading code is updated to only use the definition in that case. When DYNAMIC_FTRACE is not selected, modules shouldn't have this section, so this removes some redundant work in that case. I built parisc generic-{32,64}bit_defconfig with DYNAMIC_FTRACE enabled, and verified that the section made it into the .ko files for modules. Signed-off-by: Mark Rutland Reviewed-by: Ard Biesheuvel Cc: Helge Deller Cc: Ingo Molnar Cc: James E.J. Bottomley Cc: Jessica Yu Cc: Steven Rostedt Cc: Sven Schnelle Cc: linux-parisc@vger.kernel.org --- arch/parisc/Makefile | 1 - arch/parisc/kernel/module.c | 10 +++++++--- arch/parisc/kernel/module.lds | 7 ------- include/linux/ftrace.h | 5 +++++ kernel/module.c | 2 +- 5 files changed, 13 insertions(+), 12 deletions(-) delete mode 100644 arch/parisc/kernel/module.lds -- 2.11.0 Acked-by: Steven Rostedt (VMware) Acked-by: Helge Deller Reviewed-by: Miroslav Benes diff --git a/arch/parisc/Makefile b/arch/parisc/Makefile index 36b834f1c933..dca8f2de8cf5 100644 --- a/arch/parisc/Makefile +++ b/arch/parisc/Makefile @@ -60,7 +60,6 @@ KBUILD_CFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY=1 \ -DFTRACE_PATCHABLE_FUNCTION_SIZE=$(NOP_COUNT) CC_FLAGS_FTRACE := -fpatchable-function-entry=$(NOP_COUNT),$(shell echo $$(($(NOP_COUNT)-1))) -KBUILD_LDS_MODULE += $(srctree)/arch/parisc/kernel/module.lds endif OBJCOPY_FLAGS =-O binary -R .note -R .comment -S diff --git a/arch/parisc/kernel/module.c b/arch/parisc/kernel/module.c index ac5f34993b53..1c50093e2ebe 100644 --- a/arch/parisc/kernel/module.c +++ b/arch/parisc/kernel/module.c @@ -43,6 +43,7 @@ #include #include #include +#include #include #include #include @@ -862,7 +863,7 @@ int module_finalize(const Elf_Ehdr *hdr, const char *strtab = NULL; const Elf_Shdr *s; char *secstrings; - int err, symindex = -1; + int symindex = -1; Elf_Sym *newptr, *oldptr; Elf_Shdr *symhdr = NULL; #ifdef DEBUG @@ -946,11 +947,13 @@ int module_finalize(const Elf_Ehdr *hdr, /* patch .altinstructions */ apply_alternatives(aseg, aseg + s->sh_size, me->name); +#ifdef CONFIG_DYNAMIC_FTRACE /* For 32 bit kernels we're compiling modules with * -ffunction-sections so we must relocate the addresses in the - *__mcount_loc section. + * ftrace callsite section. */ - if (symindex != -1 && !strcmp(secname, "__mcount_loc")) { + if (symindex != -1 && !strcmp(secname, FTRACE_CALLSITE_SECTION)) { + int err; if (s->sh_type == SHT_REL) err = apply_relocate((Elf_Shdr *)sechdrs, strtab, symindex, @@ -962,6 +965,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (err) return err; } +#endif } return 0; } diff --git a/arch/parisc/kernel/module.lds b/arch/parisc/kernel/module.lds deleted file mode 100644 index 1a9a92aca5c8..000000000000 --- a/arch/parisc/kernel/module.lds +++ /dev/null @@ -1,7 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ - -SECTIONS { - __mcount_loc : { - *(__patchable_function_entries) - } -} diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 9867d90d635e..9141f2263286 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -738,6 +738,11 @@ static inline unsigned long get_lock_parent_ip(void) #ifdef CONFIG_FTRACE_MCOUNT_RECORD extern void ftrace_init(void); +#ifdef CC_USING_PATCHABLE_FUNCTION_ENTRY +#define FTRACE_CALLSITE_SECTION "__patchable_function_entries" +#else +#define FTRACE_CALLSITE_SECTION "__mcount_loc" +#endif #else static inline void ftrace_init(void) { } #endif diff --git a/kernel/module.c b/kernel/module.c index ff2d7359a418..acf7962936c4 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -3222,7 +3222,7 @@ static int find_module_sections(struct module *mod, struct load_info *info) #endif #ifdef CONFIG_FTRACE_MCOUNT_RECORD /* sechdrs[0].sh_size is always zero */ - mod->ftrace_callsites = section_objs(info, "__mcount_loc", + mod->ftrace_callsites = section_objs(info, FTRACE_CALLSITE_SECTION, sizeof(*mod->ftrace_callsites), &mod->num_ftrace_callsites); #endif From patchwork Tue Oct 29 16:58:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 178048 Delivered-To: patch@linaro.org Received: by 2002:a92:409a:0:0:0:0:0 with SMTP id d26csp67455ill; Tue, 29 Oct 2019 09:59:04 -0700 (PDT) X-Google-Smtp-Source: APXvYqwinC4ZDyNEFpOBbYI6Yo4aZa2ErOhXSiAqDxYplhLRmGkc2igkDDIvi9WImQhE0q/CXHvd X-Received: by 2002:a05:6402:1ad0:: with SMTP id ba16mr18096587edb.107.1572368344770; Tue, 29 Oct 2019 09:59:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572368344; cv=none; d=google.com; s=arc-20160816; b=RU7HLSHw1wjoQOKgkNIFYmGo0b1EARmodQI8DyGMNQAQ0rcPM8nK/eZ44VofdHhAzJ bnwwc7krr0H2WMuHMLZDrmyhyo6NVlLfH1ESHfp5uXEkfrPwdhEu6JcOxzWAIT8SapdH 91Z3WPAi/b0hBoqfC6Uz4cJzXVLvLy0+eyzapKJ51QGGhns2YkA7oer4X8NyUNAcnyT1 Q6tX+7j20ubdZqREckd6w33y3DxZWpPS3AeYBRwbqgq/2RzV5FKyLhKWR21zMuZxBiWa py6PjWFwJPuY1HSYKx5KZPDWPFmJnvS0rEzVIAEJcaGv8dYTIZ2lkgrCAqzghuHFiL8F QZBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=L/trl+ls8G0PlpDrS/qP/gBJZEbSL+as8I1pnXDEWU8=; b=lKP5gxBU8mXVWaxaBvKtyBeJ+TT3IN0FBHZ3TdnEP+0C/58XYhd6+cAcZI5NvwBl6n NpIPsFqcCi9AJGaYlxEnNXonxqjPKXDrjDMOy7QmfNHV26I7m+Ye1Ar4+KIdNF1FgcJZ 57ZjgGwdNHcrcrstTurobUSakQlflvM5v4affdh+7e64zUHNlOdRhy5R9rjtwBA3KbLy 0VQ9hFN1cAuR5fplr5Nlw+PD6iWRAoKlJGr+NYbFVj2RjUKe3OX+bysbQ55h2xis5Ap8 10VURhNhhbQN3KOygHjLnz8/HOtaPHuMDKo+h+H3MRqn8HLEV9mHmbJDT+U0Wvb5ZRD7 KysA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x24si1609975ejf.1.2019.10.29.09.59.04; Tue, 29 Oct 2019 09:59:04 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390694AbfJ2Q7A (ORCPT + 26 others); Tue, 29 Oct 2019 12:59:00 -0400 Received: from foss.arm.com ([217.140.110.172]:54620 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390643AbfJ2Q7A (ORCPT ); Tue, 29 Oct 2019 12:59:00 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 41AB94B2; Tue, 29 Oct 2019 09:58:59 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D887C3F71F; Tue, 29 Oct 2019 09:58:56 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, amit.kachhap@arm.com, catalin.marinas@arm.com, deller@gmx.de, duwe@suse.de, James.Bottomley@HansenPartnership.com, james.morse@arm.com, jeyu@kernel.org, jpoimboe@redhat.com, jthierry@redhat.com, linux-parisc@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, rostedt@goodmis.org, svens@stackframe.org, takahiro.akashi@linaro.org, will@kernel.org Subject: [PATCHv2 3/8] arm64: module: rework special section handling Date: Tue, 29 Oct 2019 16:58:27 +0000 Message-Id: <20191029165832.33606-4-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191029165832.33606-1-mark.rutland@arm.com> References: <20191029165832.33606-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When we load a module, we have to perform some special work for a couple of named sections. To do this, we iterate over all of the module's sections, and perform work for each section we recognize. To make it easier to handle the unexpected absence of a section, and to make the section-specific logic easer to read, let's factor the section search into a helper. Similar is already done in the core module loader, and other architectures (and ideally we'd unify these in future). If we expect a module to have an ftrace trampoline section, but it doesn't have one, we'll now reject loading the module. When ARM64_MODULE_PLTS is selected, any correctly built module should have one (and this is assumed by arm64's ftrace PLT code) and the absence of such a section implies something has gone wrong at build time. Subsequent patches will make use of the new helper. Signed-off-by: Mark Rutland Reviewed-by: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Will Deacon --- arch/arm64/kernel/module.c | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) -- 2.11.0 Reviewed-by: Miroslav Benes diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index 03ff15bffbb6..763a86d52fef 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -470,22 +470,39 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, return -ENOEXEC; } -int module_finalize(const Elf_Ehdr *hdr, - const Elf_Shdr *sechdrs, - struct module *me) +static const Elf_Shdr *find_section(const Elf_Ehdr *hdr, + const Elf_Shdr *sechdrs, + const char *name) { const Elf_Shdr *s, *se; const char *secstrs = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; for (s = sechdrs, se = sechdrs + hdr->e_shnum; s < se; s++) { - if (strcmp(".altinstructions", secstrs + s->sh_name) == 0) - apply_alternatives_module((void *)s->sh_addr, s->sh_size); + if (strcmp(name, secstrs + s->sh_name) == 0) + return s; + } + + return NULL; +} + +int module_finalize(const Elf_Ehdr *hdr, + const Elf_Shdr *sechdrs, + struct module *me) +{ + const Elf_Shdr *s; + + s = find_section(hdr, sechdrs, ".altinstructions"); + if (s) + apply_alternatives_module((void *)s->sh_addr, s->sh_size); + #ifdef CONFIG_ARM64_MODULE_PLTS - if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE) && - !strcmp(".text.ftrace_trampoline", secstrs + s->sh_name)) - me->arch.ftrace_trampoline = (void *)s->sh_addr; -#endif + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE)) { + s = find_section(hdr, sechdrs, ".text.ftrace_trampoline"); + if (!s) + return -ENOEXEC; + me->arch.ftrace_trampoline = (void *)s->sh_addr; } +#endif return 0; } From patchwork Tue Oct 29 16:58:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 178049 Delivered-To: patch@linaro.org Received: by 2002:a92:409a:0:0:0:0:0 with SMTP id d26csp67616ill; Tue, 29 Oct 2019 09:59:12 -0700 (PDT) X-Google-Smtp-Source: APXvYqwlMg5NddIz0PY16SxEGjcb5kmg+c4uioh/ciYL/+iB3t1owx5PA+UIUMhNkeQsvvB21gVn X-Received: by 2002:aa7:da19:: with SMTP id r25mr15258856eds.143.1572368352485; Tue, 29 Oct 2019 09:59:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572368352; cv=none; d=google.com; s=arc-20160816; b=lw6KYM7AsZHAUPxH/CIG4xjsILPczxAtIeIsr/5Xx9B2ZAlWRRIqTjGrCvyCyC0wS0 owI7lhTW2+kM/vzwChKxuceSuZGKbxaGI4EfytmQM6KftIGOHBdRpsT4QVB32CFkawZH eGP6n16/JnNYRRlHIp+Vg8L/Ys7TixzjAhIqnShTrjN5624eizfVABIXWlDvgeCMdBfo UD3zqES2Agv1FIBCwzjcGuMKY8ax7Br+ilwEcUGT4WHPnvteK/Yby32+o1Mn2NC2FAmO yeOvMUaUpTu90Blf5XWtFYcFM16kfThRp5FNM1khXrE/A1uhI6K07AQbS5fh4/vWVAJ0 24AQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=JxQJvqDj876F6t9mW1ZUsv8YZHwYQZuQOHHn1UUwH4A=; b=hcIEuTm+vFb6SueVcS7ZQb+X6JIW6Ver9PeZo86zxhVpj/s0JKbEwBz1RTuVMsY3X0 rw/12ittsUC4GDFy2nERcGvE+BpUntquaOt1fnGtLnmI9zB/yGZDLICxQ9nfsr2jVgKQ KfMYPKJ4IzBuK1jMgN7kFHHwUQEKZzRngH60/kWJcut6Ase2/5u69s7l37zT8G0K6dJc C0tRPx0p+oYUeMqMbVt0NyHsBRJq6qk/9UaEcK2Nsw6gKq639rB93VMQETdrc4514vRV oe202kYIEWnvfvKQR8RxR3tohvgyd1+ktsiGkSWCqckmy3aD6x8yPGgoKfPXxy4XdWRM TLBw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j11si10387572edj.43.2019.10.29.09.59.12; Tue, 29 Oct 2019 09:59:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390722AbfJ2Q7I (ORCPT + 26 others); Tue, 29 Oct 2019 12:59:08 -0400 Received: from foss.arm.com ([217.140.110.172]:54646 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390643AbfJ2Q7I (ORCPT ); Tue, 29 Oct 2019 12:59:08 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7969A4F5; Tue, 29 Oct 2019 09:59:07 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 26C623F71F; Tue, 29 Oct 2019 09:59:05 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, amit.kachhap@arm.com, catalin.marinas@arm.com, deller@gmx.de, duwe@suse.de, James.Bottomley@HansenPartnership.com, james.morse@arm.com, jeyu@kernel.org, jpoimboe@redhat.com, jthierry@redhat.com, linux-parisc@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, rostedt@goodmis.org, svens@stackframe.org, takahiro.akashi@linaro.org, will@kernel.org Subject: [PATCHv2 4/8] arm64: module/ftrace: intialize PLT at load time Date: Tue, 29 Oct 2019 16:58:28 +0000 Message-Id: <20191029165832.33606-5-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191029165832.33606-1-mark.rutland@arm.com> References: <20191029165832.33606-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently we lazily-initialize a module's ftrace PLT at runtime when we install the first ftrace call. To do so we have to apply a number of sanity checks, transiently mark the module text as RW, and perform an IPI as part of handling Neoverse-N1 erratum #1542419. We only expect the ftrace trampoline to point at ftrace_caller() (AKA FTRACE_ADDR), so let's simplify all of this by intializing the PLT at module load time, before the module loader marks the module RO and performs the intial I-cache maintenance for the module. Thus we can rely on the module having been correctly intialized, and can simplify the runtime work necessary to install an ftrace call in a module. This will also allow for the removal of module_disable_ro(). Tested by forcing ftrace_make_call() to use the module PLT, and then loading up a module after setting up ftrace with: | echo ":mod:" > set_ftrace_filter; | echo function > current_tracer; | modprobe Since FTRACE_ADDR is only defined when CONFIG_DYNAMIC_FTRACE is selected, we wrap its use along with most of module_init_ftrace_plt() with ifdeffery rather than using IS_ENABLED(). Signed-off-by: Mark Rutland Reviewed-by: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Peter Zijlstra Cc: Will Deacon --- arch/arm64/kernel/ftrace.c | 55 ++++++++++++---------------------------------- arch/arm64/kernel/module.c | 32 +++++++++++++++++---------- 2 files changed, 35 insertions(+), 52 deletions(-) -- 2.11.0 diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index 06e56b470315..822718eafdb4 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -73,10 +73,22 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) if (offset < -SZ_128M || offset >= SZ_128M) { #ifdef CONFIG_ARM64_MODULE_PLTS - struct plt_entry trampoline, *dst; struct module *mod; /* + * There is only one ftrace trampoline per module. For now, + * this is not a problem since on arm64, all dynamic ftrace + * invocations are routed via ftrace_caller(). This will need + * to be revisited if support for multiple ftrace entry points + * is added in the future, but for now, the pr_err() below + * deals with a theoretical issue only. + */ + if (addr != FTRACE_ADDR) { + pr_err("ftrace: far branches to multiple entry points unsupported inside a single module\n"); + return -EINVAL; + } + + /* * On kernels that support module PLTs, the offset between the * branch instruction and its target may legally exceed the * range of an ordinary relative 'bl' opcode. In this case, we @@ -93,46 +105,7 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) if (WARN_ON(!mod)) return -EINVAL; - /* - * There is only one ftrace trampoline per module. For now, - * this is not a problem since on arm64, all dynamic ftrace - * invocations are routed via ftrace_caller(). This will need - * to be revisited if support for multiple ftrace entry points - * is added in the future, but for now, the pr_err() below - * deals with a theoretical issue only. - * - * Note that PLTs are place relative, and plt_entries_equal() - * checks whether they point to the same target. Here, we need - * to check if the actual opcodes are in fact identical, - * regardless of the offset in memory so use memcmp() instead. - */ - dst = mod->arch.ftrace_trampoline; - trampoline = get_plt_entry(addr, dst); - if (memcmp(dst, &trampoline, sizeof(trampoline))) { - if (plt_entry_is_initialized(dst)) { - pr_err("ftrace: far branches to multiple entry points unsupported inside a single module\n"); - return -EINVAL; - } - - /* point the trampoline to our ftrace entry point */ - module_disable_ro(mod); - *dst = trampoline; - module_enable_ro(mod, true); - - /* - * Ensure updated trampoline is visible to instruction - * fetch before we patch in the branch. Although the - * architecture doesn't require an IPI in this case, - * Neoverse-N1 erratum #1542419 does require one - * if the TLB maintenance in module_enable_ro() is - * skipped due to rodata_enabled. It doesn't seem worth - * it to make it conditional given that this is - * certainly not a fast-path. - */ - flush_icache_range((unsigned long)&dst[0], - (unsigned long)&dst[1]); - } - addr = (unsigned long)dst; + addr = (unsigned long)mod->arch.ftrace_trampoline; #else /* CONFIG_ARM64_MODULE_PLTS */ return -EINVAL; #endif /* CONFIG_ARM64_MODULE_PLTS */ diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index 763a86d52fef..5f5bc3b94da7 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -485,24 +486,33 @@ static const Elf_Shdr *find_section(const Elf_Ehdr *hdr, return NULL; } +int module_init_ftrace_plt(const Elf_Ehdr *hdr, + const Elf_Shdr *sechdrs, + struct module *mod) +{ +#if defined(CONFIG_ARM64_MODULE_PLTS) && defined(CONFIG_DYNAMIC_FTRACE) + const Elf_Shdr *s; + struct plt_entry *plt; + + s = find_section(hdr, sechdrs, ".text.ftrace_trampoline"); + if (!s) + return -ENOEXEC; + + plt = (void *)s->sh_addr; + *plt = get_plt_entry(FTRACE_ADDR, plt); + mod->arch.ftrace_trampoline = plt; +#endif + return 0; +} + int module_finalize(const Elf_Ehdr *hdr, const Elf_Shdr *sechdrs, struct module *me) { const Elf_Shdr *s; - s = find_section(hdr, sechdrs, ".altinstructions"); if (s) apply_alternatives_module((void *)s->sh_addr, s->sh_size); -#ifdef CONFIG_ARM64_MODULE_PLTS - if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE)) { - s = find_section(hdr, sechdrs, ".text.ftrace_trampoline"); - if (!s) - return -ENOEXEC; - me->arch.ftrace_trampoline = (void *)s->sh_addr; - } -#endif - - return 0; + return module_init_ftrace_plt(hdr, sechdrs, me); } From patchwork Tue Oct 29 16:58:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 178050 Delivered-To: patch@linaro.org Received: by 2002:a92:409a:0:0:0:0:0 with SMTP id d26csp67774ill; Tue, 29 Oct 2019 09:59:19 -0700 (PDT) X-Google-Smtp-Source: APXvYqxyO/QzMSuYAtvt2VntisMutDWRN0J/V8uKBFMcm378q2P6cviadgfSpwbJOtQgyOPkDUTe X-Received: by 2002:a17:906:c444:: with SMTP id ck4mr4353864ejb.110.1572368359557; Tue, 29 Oct 2019 09:59:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572368359; cv=none; d=google.com; s=arc-20160816; b=XLgjO9lfFZeOF6LCrgaVi9CiH24QbJv20Rpsu6oaYsUtspqgtXH+Z0u6Je6fsBUwem KIotLYG/npLMWyuy89DilgvaV12MeByysPd1HrnfYUpPMkbLQr88RZ/NCH+AQl273KL9 u6LfTuc1hrtJ98qnzxPo5ljybP27CtIWTfY4Ibas9BckNOPRrMV/J4m893S51hG7uiyA Xbraqq/RvCNtMPcrvIgRWwNAeUh602mJj+WK+/k3KC2YRO1wwmYQYjQXJPnS4GHiiPtG GZuFFY6aCb9y9r7dk46ZRSHlHZh9Ju8ZGpdAdQ8gezmB205d6/NlTP7tDGDGkWhe/+yh HoDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=9xcDQxgIeUO7PEas6WFKM7Y3z/kg7VpUIHTXDug4Rhc=; b=ErfaXHVMrGhNJW/RJmlDpBh2mFuHKhW2TCJECrXJgv+qktI2MaE0piEIpJS/M1L07v XkdKjJ3z7zLknHtm4N1UivG5TPwu+HN4W+YkMRTJ+ZAiV7SGzdiIDc6rQh7FqzeYuPhU Nc6II7TGGfHmn9lHD2iDH25usGsIeMCuZjgFXoTF8Bt3ErQ3fXy+A3k7oDJ0BLMJUTux 72zHbNOGNMsvBit64w2XW5iLWDlZbApn9TyEuMRLZjJOkZxJA+cuChGbfPYsI9jcs+Ya u2h9XOxtqKdUKp5qA+WfhxEB3Gii/+m5BcqstCFJ4wFaT0Zl4OiI7aTcjxq2OJaRnfyd 5XVA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p21si1955415edc.305.2019.10.29.09.59.19; Tue, 29 Oct 2019 09:59:19 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390742AbfJ2Q7P (ORCPT + 26 others); Tue, 29 Oct 2019 12:59:15 -0400 Received: from foss.arm.com ([217.140.110.172]:54672 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390606AbfJ2Q7O (ORCPT ); Tue, 29 Oct 2019 12:59:14 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 228AE55D; Tue, 29 Oct 2019 09:59:14 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A3C513F71F; Tue, 29 Oct 2019 09:59:11 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, amit.kachhap@arm.com, catalin.marinas@arm.com, deller@gmx.de, duwe@suse.de, James.Bottomley@HansenPartnership.com, james.morse@arm.com, jeyu@kernel.org, jpoimboe@redhat.com, jthierry@redhat.com, linux-parisc@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, rostedt@goodmis.org, svens@stackframe.org, takahiro.akashi@linaro.org, will@kernel.org Subject: [PATCHv2 5/8] arm64: insn: add encoder for MOV (register) Date: Tue, 29 Oct 2019 16:58:29 +0000 Message-Id: <20191029165832.33606-6-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191029165832.33606-1-mark.rutland@arm.com> References: <20191029165832.33606-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org For FTRACE_WITH_REGS, we're going to want to generate a MOV (register) instruction as part of the callsite intialization. As MOV (register) is an alias for ORR (shifted register), we can generate this with aarch64_insn_gen_logical_shifted_reg(), but it's somewhat verbose and difficult to read in-context. Add a aarch64_insn_gen_move_reg() wrapper for this case so that we can write callers in a more straightforward way. Signed-off-by: Mark Rutland Reviewed-by: Ard Biesheuvel Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/include/asm/insn.h | 3 +++ arch/arm64/kernel/insn.c | 13 +++++++++++++ 2 files changed, 16 insertions(+) -- 2.11.0 diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index 39e7780bedd6..bb313dde58a4 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -440,6 +440,9 @@ u32 aarch64_insn_gen_logical_shifted_reg(enum aarch64_insn_register dst, int shift, enum aarch64_insn_variant variant, enum aarch64_insn_logic_type type); +u32 aarch64_insn_gen_move_reg(enum aarch64_insn_register dst, + enum aarch64_insn_register src, + enum aarch64_insn_variant variant); u32 aarch64_insn_gen_logical_immediate(enum aarch64_insn_logic_type type, enum aarch64_insn_variant variant, enum aarch64_insn_register Rn, diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c index d801a7094076..513b29c3e735 100644 --- a/arch/arm64/kernel/insn.c +++ b/arch/arm64/kernel/insn.c @@ -1268,6 +1268,19 @@ u32 aarch64_insn_gen_logical_shifted_reg(enum aarch64_insn_register dst, return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_6, insn, shift); } +/* + * MOV (register) is architecturally an alias of ORR (shifted register) where + * MOV <*d>, <*m> is equivalent to ORR <*d>, <*ZR>, <*m> + */ +u32 aarch64_insn_gen_move_reg(enum aarch64_insn_register dst, + enum aarch64_insn_register src, + enum aarch64_insn_variant variant) +{ + return aarch64_insn_gen_logical_shifted_reg(dst, AARCH64_INSN_REG_ZR, + src, 0, variant, + AARCH64_INSN_LOGIC_ORR); +} + u32 aarch64_insn_gen_adr(unsigned long pc, unsigned long addr, enum aarch64_insn_register reg, enum aarch64_insn_adr_type type) From patchwork Tue Oct 29 16:58:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 178051 Delivered-To: patch@linaro.org Received: by 2002:a92:409a:0:0:0:0:0 with SMTP id d26csp67832ill; Tue, 29 Oct 2019 09:59:23 -0700 (PDT) X-Google-Smtp-Source: APXvYqy2NQww4ccI0deE12MW9uiKQGEYV3hTXi7aq45f/GCBEq7t6wp6IjkhWumwGFWqHtoGQp2h X-Received: by 2002:a05:6402:1006:: with SMTP id c6mr27070390edu.2.1572368362999; Tue, 29 Oct 2019 09:59:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572368362; cv=none; d=google.com; s=arc-20160816; b=yyPrkd0QZhxrtESpStLhsQJkX4R5810Afp3HBPNT06IbNcqhzfup4E8Cl17j60qJUI DWGFD1tKeHUJQNdoLwKzZpUn9VoqxZHSHhP5XWLdwWVuBHdQjSg3yT11IfpEokJfnBU2 z/YQKN4TpYXorVQCe1Z9adrJIN363/NIJKBm7b+p/3ppabYIDRTx6wTgUL5aH93F6B5g zDDWxT1pN95TNEkNfSWghB6MCL/OvxfY7+w1g32SqthLD+5gUvU+hs+KuMGidsOb4Qbd ZJ/t9NCJxn7ZhTFYdjOC8QNAeHk/WeN34k9bPSun3CF40fYEm2rH2NTn0hzwPPHK6iet bWyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=IJatL9YMLUvc7tpetcCS5Eh36sF1G1jVOUCUEnpxnlQ=; b=eavfEEtUPjnvdML+t3+dZuiC5iD89R9VNMJ7wSbCFM0ShoWA/5CE3jnQEMghAo2x3h I458/Tl1epMwrhIItbzs+aiubC8XEDTpm0zKri1R+euQIuRt3lO/crUzKjqwNc1yhhtw OGP1/StcFiPaJalwfxY6cs5ZGwBzc9eGddW0QL1lxl2iMrDepQLRkYoXgQbueMjr/Kff itE4PdhctWAg/R2OXcJwd7EghfUASlhvmQu8Yeltt0xY/Fnn/kzepUJkw0Y5JDd0G6OM nUG+sKGkWCnv9/eAhNGhOXB9/GQ/GijUGk2O68EvFNxnCRZOT6zPWXTgigUiUHVVe1Zj 2Lmg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n19si8185442ejk.264.2019.10.29.09.59.22; Tue, 29 Oct 2019 09:59:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390758AbfJ2Q7T (ORCPT + 26 others); Tue, 29 Oct 2019 12:59:19 -0400 Received: from foss.arm.com ([217.140.110.172]:54704 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390606AbfJ2Q7S (ORCPT ); Tue, 29 Oct 2019 12:59:18 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4E566625; Tue, 29 Oct 2019 09:59:18 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E621F3F71F; Tue, 29 Oct 2019 09:59:15 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, amit.kachhap@arm.com, catalin.marinas@arm.com, deller@gmx.de, duwe@suse.de, James.Bottomley@HansenPartnership.com, james.morse@arm.com, jeyu@kernel.org, jpoimboe@redhat.com, jthierry@redhat.com, linux-parisc@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, rostedt@goodmis.org, svens@stackframe.org, takahiro.akashi@linaro.org, will@kernel.org Subject: [PATCHv2 6/8] arm64: asm-offsets: add S_FP Date: Tue, 29 Oct 2019 16:58:30 +0000 Message-Id: <20191029165832.33606-7-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191029165832.33606-1-mark.rutland@arm.com> References: <20191029165832.33606-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org So that assembly code can more easily manipulate the FP (x29) within a pt_regs, add an S_FP asm-offsets definition. Signed-off-by: Mark Rutland Reviewed-by: Ard Biesheuvel Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/kernel/asm-offsets.c | 1 + 1 file changed, 1 insertion(+) -- 2.11.0 diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 214685760e1c..a5bdce8af65b 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -56,6 +56,7 @@ int main(void) DEFINE(S_X24, offsetof(struct pt_regs, regs[24])); DEFINE(S_X26, offsetof(struct pt_regs, regs[26])); DEFINE(S_X28, offsetof(struct pt_regs, regs[28])); + DEFINE(S_FP, offsetof(struct pt_regs, regs[29])); DEFINE(S_LR, offsetof(struct pt_regs, regs[30])); DEFINE(S_SP, offsetof(struct pt_regs, sp)); DEFINE(S_PSTATE, offsetof(struct pt_regs, pstate)); From patchwork Tue Oct 29 16:58:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 178052 Delivered-To: patch@linaro.org Received: by 2002:a92:409a:0:0:0:0:0 with SMTP id d26csp67950ill; Tue, 29 Oct 2019 09:59:29 -0700 (PDT) X-Google-Smtp-Source: APXvYqwXbyjyaIiyMXLrPP9/mxDHoKOWmfi91S1MjSPQSCuqgUwVWDit8K32iJyO427i4vhw4+Tn X-Received: by 2002:aa7:d714:: with SMTP id t20mr27436511edq.129.1572368368950; Tue, 29 Oct 2019 09:59:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572368368; cv=none; d=google.com; s=arc-20160816; b=yLcbaNqnSoWYHBkeBzBLPZZd0GlcrRZUiOxgLCKfGTwyOvyY+e3wJcgQ7NzkGfE5z0 hdGvozbg0A63RTyKQkbs6E6UCxq1+PiTVap/zOSInnRK6/Y7V5CpeLiKvw18yEpJEP6p uD2vU3bR4Js3glvMyH5OOoVylZVzRNAd3yw6oELQV8J+3JFUUop2jBfPARx4w06uPFYc LsnA2OA/N/MwuCh39PRiODyxT3Iwp4Y55RK8BSzl3+xzQOCsp2q3FvAv5WniIFruw9c/ 8cir19c8nLdjUENSUVywFUxT2BjHPn3Xw8beH2h6iJTylsCxwHdrbiRAhi5GOmD4Yxhx ZutA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=f5UXOQip++4xNKe6jUimCpDWkUIGJ08vjyDdjSahysg=; b=x54nI3KUkgG+ep7X59wdHwmPlJU14dtl6yZDSvVEgL8v2Xs4k+QGVhY2XGM6UcwWfq pcH6TRY5JfWJZ0TiEZIdFy/SCbi6c1ujt4b5Ne08OCFDyHg+ylYG6GGm24pFfpmEznW+ Yl1HB0W16FxwiYM+clIwpQ5oIvt3VMelD5UTswNr26MSlLqpsKrB+3tbyPZQjKphKhOg yL+6O+Lc2QrGHuW6XI2UcuZhO+s0fymydn+UCk5QSxt0hhV7EeWEObEAKXwxAH1Yso1c nw+l4v7l9UMoh1mvY5Ipfwcmi6MdIfS7ZvSOAnAZxGQ8AMyE0qQW6klfaJ3/UhgUswyx yAGg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k24si8793965ejz.417.2019.10.29.09.59.28; Tue, 29 Oct 2019 09:59:28 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390780AbfJ2Q7Z (ORCPT + 26 others); Tue, 29 Oct 2019 12:59:25 -0400 Received: from foss.arm.com ([217.140.110.172]:54736 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390765AbfJ2Q7Y (ORCPT ); Tue, 29 Oct 2019 12:59:24 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E1DB368D; Tue, 29 Oct 2019 09:59:23 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 598613F71F; Tue, 29 Oct 2019 09:59:21 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, amit.kachhap@arm.com, catalin.marinas@arm.com, deller@gmx.de, duwe@suse.de, James.Bottomley@HansenPartnership.com, james.morse@arm.com, jeyu@kernel.org, jpoimboe@redhat.com, jthierry@redhat.com, linux-parisc@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, rostedt@goodmis.org, svens@stackframe.org, takahiro.akashi@linaro.org, will@kernel.org, Torsten Duwe Subject: [PATCHv2 7/8] arm64: implement ftrace with regs Date: Tue, 29 Oct 2019 16:58:31 +0000 Message-Id: <20191029165832.33606-8-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191029165832.33606-1-mark.rutland@arm.com> References: <20191029165832.33606-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Torsten Duwe This patch implements FTRACE_WITH_REGS for arm64, which allows a traced function's arguments (and some other registers) to be captured into a struct pt_regs, allowing these to be inspected and/or modified. This is a building block for live-patching, where a function's arguments may be forwarded to another function. This is also necessary to enable ftrace and in-kernel pointer authentication at the same time, as it allows the LR value to be captured and adjusted prior to signing. Using GCC's -fpatchable-function-entry=N option, we can have the compiler insert a configurable number of NOPs between the function entry point and the usual prologue. This also ensures functions are AAPCS compliant (e.g. disabling inter-procedural register allocation). For example, with -fpatchable-function-entry=2, GCC 8.1.0 compiles the following: | unsigned long bar(void); | | unsigned long foo(void) | { | return bar() + 1; | } ... to: | : | nop | nop | stp x29, x30, [sp, #-16]! | mov x29, sp | bl 0 | add x0, x0, #0x1 | ldp x29, x30, [sp], #16 | ret This patch builds the kernel with -fpatchable-function-entry=2, prefixing each function with two NOPs. To trace a function, we replace these NOPs with a sequence that saves the LR into a GPR, then calls an ftrace entry assembly function which saves this and other relevant registers: | mov x9, x30 | bl Since patchable functions are AAPCS compliant (and the kernel does not use x18 as a platform register), x9-x18 can be safely clobbered in the patched sequence and the ftrace entry code. There are now two ftrace entry functions, ftrace_regs_entry (which saves all GPRs), and ftrace_entry (which saves the bare minimum). A PLT is allocated for each within modules. Signed-off-by: Torsten Duwe [Mark: rework asm, comments, PLTs, initialization, commit message] Signed-off-by: Mark Rutland Reviewed-by: Ard Biesheuvel Cc: AKASHI Takahiro Cc: Amit Daniel Kachhap Cc: Catalin Marinas Cc: Josh Poimboeuf Cc: Julien Thierry Cc: Will Deacon --- arch/arm64/Kconfig | 2 + arch/arm64/Makefile | 5 ++ arch/arm64/include/asm/ftrace.h | 23 +++++++ arch/arm64/include/asm/module.h | 2 +- arch/arm64/kernel/entry-ftrace.S | 140 +++++++++++++++++++++++++++++++++++++-- arch/arm64/kernel/ftrace.c | 84 +++++++++++++++++++---- arch/arm64/kernel/module-plts.c | 3 +- arch/arm64/kernel/module.c | 18 +++-- 8 files changed, 252 insertions(+), 25 deletions(-) -- 2.11.0 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 950a56b71ff0..0ffb8596b8a1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -143,6 +143,8 @@ config ARM64 select HAVE_DEBUG_KMEMLEAK select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE + select HAVE_DYNAMIC_FTRACE_WITH_REGS \ + if $(cc-option,-fpatchable-function-entry=2) select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_FAST_GUP select HAVE_FTRACE_MCOUNT_RECORD diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 2c0238ce0551..1fbe24d4fdb6 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -95,6 +95,11 @@ ifeq ($(CONFIG_ARM64_MODULE_PLTS),y) KBUILD_LDS_MODULE += $(srctree)/arch/arm64/kernel/module.lds endif +ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_REGS),y) + KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY + CC_FLAGS_FTRACE := -fpatchable-function-entry=2 +endif + # Default value head-y := arch/arm64/kernel/head.o diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index d48667b04c41..91fa4baa1a93 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -11,9 +11,20 @@ #include #define HAVE_FUNCTION_GRAPH_FP_TEST + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS +#define ARCH_SUPPORTS_FTRACE_OPS 1 +#else #define MCOUNT_ADDR ((unsigned long)_mcount) +#endif + +/* The BL at the callsite's adjusted rec->ip */ #define MCOUNT_INSN_SIZE AARCH64_INSN_SIZE +#define FTRACE_PLT_IDX 0 +#define FTRACE_REGS_PLT_IDX 1 +#define NR_FTRACE_PLTS 2 + /* * Currently, gcc tends to save the link register after the local variables * on the stack. This causes the max stack tracer to report the function @@ -44,12 +55,24 @@ extern void return_to_handler(void); static inline unsigned long ftrace_call_adjust(unsigned long addr) { /* + * Adjust addr to point at the BL in the callsite. + * See ftrace_init_nop() for the callsite sequence. + */ + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) + return addr + AARCH64_INSN_SIZE; + /* * addr is the address of the mcount call instruction. * recordmcount does the necessary offset calculation. */ return addr; } +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS +struct dyn_ftrace; +int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); +#define ftrace_init_nop ftrace_init_nop +#endif + #define ftrace_return_address(n) return_address(n) /* diff --git a/arch/arm64/include/asm/module.h b/arch/arm64/include/asm/module.h index f80e13cbf8ec..1e93de68c044 100644 --- a/arch/arm64/include/asm/module.h +++ b/arch/arm64/include/asm/module.h @@ -21,7 +21,7 @@ struct mod_arch_specific { struct mod_plt_sec init; /* for CONFIG_DYNAMIC_FTRACE */ - struct plt_entry *ftrace_trampoline; + struct plt_entry *ftrace_trampolines; }; #endif diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index 33d003d80121..94720388957f 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -7,10 +7,137 @@ */ #include +#include #include #include #include +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS +/* + * Due to -fpatchable-function-entry=2, the compiler has placed two NOPs before + * the regular function prologue. For an enabled callsite, ftrace_init_nop() and + * ftrace_make_call() have patched those NOPs to: + * + * MOV X9, LR + * BL + * + * ... where is either ftrace_caller or ftrace_regs_caller. + * + * Each instrumented function follows the AAPCS, so here x0-x8 and x19-x30 are + * live, and x9-x18 are safe to clobber. + * + * We save the callsite's context into a pt_regs before invoking and ftrace + * callbacks. So that we can get a sensible backtrace, we create a stack record + * for the callsite and the ftrace entry assembly. This is not sufficient for + * reliable stacktrace: until we create the callsite stack record, its caller + * is missing from the LR and existing chain of frame records. + */ + .macro ftrace_regs_entry, allregs=0 + /* Make room for pt_regs, plus a callee frame */ + sub sp, sp, #(S_FRAME_SIZE + 16) + + /* Save function arguments (and x9 for simplicity) */ + stp x0, x1, [sp, #S_X0] + stp x2, x3, [sp, #S_X2] + stp x4, x5, [sp, #S_X4] + stp x6, x7, [sp, #S_X6] + stp x8, x9, [sp, #S_X8] + + /* Optionally save the callee-saved registers, always save the FP */ + .if \allregs == 1 + stp x10, x11, [sp, #S_X10] + stp x12, x13, [sp, #S_X12] + stp x14, x15, [sp, #S_X14] + stp x16, x17, [sp, #S_X16] + stp x18, x19, [sp, #S_X18] + stp x20, x21, [sp, #S_X20] + stp x22, x23, [sp, #S_X22] + stp x24, x25, [sp, #S_X24] + stp x26, x27, [sp, #S_X26] + stp x28, x29, [sp, #S_X28] + .else + str x29, [sp, #S_FP] + .endif + + /* Save the callsite's SP and LR */ + add x10, sp, #(S_FRAME_SIZE + 16) + stp x9, x10, [sp, #S_LR] + + /* Save the PC after the ftrace callsite */ + str x30, [sp, #S_PC] + + /* Create a frame record for the callsite above pt_regs */ + stp x29, x9, [sp, #S_FRAME_SIZE] + add x29, sp, #S_FRAME_SIZE + + /* Create our frame record within pt_regs. */ + stp x29, x30, [sp, #S_STACKFRAME] + add x29, sp, #S_STACKFRAME + .endm + +ENTRY(ftrace_regs_caller) + ftrace_regs_entry 1 + b ftrace_common +ENDPROC(ftrace_regs_caller) + +ENTRY(ftrace_caller) + ftrace_regs_entry 0 + b ftrace_common +ENDPROC(ftrace_caller) + +ENTRY(ftrace_common) + sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn) + mov x1, x9 // parent_ip (callsite's LR) + ldr_l x2, function_trace_op // op + mov x3, sp // regs + +GLOBAL(ftrace_call) + bl ftrace_stub + +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +GLOBAL(ftrace_graph_call) // ftrace_graph_caller(); + nop // If enabled, this will be replaced + // "b ftrace_graph_caller" +#endif + +/* + * At the callsite x0-x8 and x19-x30 were live. Any C code will have preserved + * x19-x29 per the AAPCS, and we created frame records upon entry, so we need + * to restore x0-x8, x29, and x30. + */ +ftrace_common_return: + /* Restore function arguments */ + ldp x0, x1, [sp] + ldp x2, x3, [sp, #S_X2] + ldp x4, x5, [sp, #S_X4] + ldp x6, x7, [sp, #S_X6] + ldr x8, [sp, #S_X8] + + /* Restore the callsite's FP, LR, PC */ + ldr x29, [sp, #S_FP] + ldr x30, [sp, #S_LR] + ldr x9, [sp, #S_PC] + + /* Restore the callsite's SP */ + add sp, sp, #S_FRAME_SIZE + 16 + + ret x9 +ENDPROC(ftrace_common) + +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +ENTRY(ftrace_graph_caller) + ldr x0, [sp, #S_PC] + sub x0, x0, #AARCH64_INSN_SIZE // ip (callsite's BL insn) + add x1, sp, #S_LR // parent_ip (callsite's LR) + ldr x2, [sp, #S_FRAME_SIZE] // parent fp (callsite's FP) + bl prepare_ftrace_return + b ftrace_common_return +ENDPROC(ftrace_graph_caller) +#else +#endif + +#else /* CONFIG_DYNAMIC_FTRACE_WITH_REGS */ + /* * Gcc with -pg will put the following code in the beginning of each function: * mov x0, x30 @@ -160,11 +287,6 @@ GLOBAL(ftrace_graph_call) // ftrace_graph_caller(); mcount_exit ENDPROC(ftrace_caller) -#endif /* CONFIG_DYNAMIC_FTRACE */ - -ENTRY(ftrace_stub) - ret -ENDPROC(ftrace_stub) #ifdef CONFIG_FUNCTION_GRAPH_TRACER /* @@ -184,7 +306,15 @@ ENTRY(ftrace_graph_caller) mcount_exit ENDPROC(ftrace_graph_caller) +#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ +#endif /* CONFIG_DYNAMIC_FTRACE */ +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_REGS */ + +ENTRY(ftrace_stub) + ret +ENDPROC(ftrace_stub) +#ifdef CONFIG_FUNCTION_GRAPH_TRACER /* * void return_to_handler(void) * diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index 822718eafdb4..aea652c33a38 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -62,6 +62,19 @@ int ftrace_update_ftrace_func(ftrace_func_t func) return ftrace_modify_code(pc, 0, new, false); } +#ifdef CONFIG_ARM64_MODULE_PLTS +static struct plt_entry *get_ftrace_plt(struct module *mod, unsigned long addr) +{ + struct plt_entry *plt = mod->arch.ftrace_trampolines; + + if (addr == FTRACE_ADDR) + return &plt[FTRACE_PLT_IDX]; + if (addr == FTRACE_REGS_ADDR && IS_ENABLED(CONFIG_FTRACE_WITH_REGS)) + return &plt[FTRACE_REGS_PLT_IDX]; + return NULL; +} +#endif + /* * Turn on the call to ftrace_caller() in instrumented function */ @@ -74,19 +87,7 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) if (offset < -SZ_128M || offset >= SZ_128M) { #ifdef CONFIG_ARM64_MODULE_PLTS struct module *mod; - - /* - * There is only one ftrace trampoline per module. For now, - * this is not a problem since on arm64, all dynamic ftrace - * invocations are routed via ftrace_caller(). This will need - * to be revisited if support for multiple ftrace entry points - * is added in the future, but for now, the pr_err() below - * deals with a theoretical issue only. - */ - if (addr != FTRACE_ADDR) { - pr_err("ftrace: far branches to multiple entry points unsupported inside a single module\n"); - return -EINVAL; - } + struct plt_entry *plt; /* * On kernels that support module PLTs, the offset between the @@ -105,7 +106,13 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) if (WARN_ON(!mod)) return -EINVAL; - addr = (unsigned long)mod->arch.ftrace_trampoline; + plt = get_ftrace_plt(mod, addr); + if (!plt) { + pr_err("ftrace: no module PLT for %ps\n", (void *)addr); + return -EINVAL; + } + + addr = (unsigned long)plt; #else /* CONFIG_ARM64_MODULE_PLTS */ return -EINVAL; #endif /* CONFIG_ARM64_MODULE_PLTS */ @@ -117,6 +124,55 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) return ftrace_modify_code(pc, old, new, true); } +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS +int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, + unsigned long addr) +{ + unsigned long pc = rec->ip; + u32 old, new; + + old = aarch64_insn_gen_branch_imm(pc, old_addr, + AARCH64_INSN_BRANCH_LINK); + new = aarch64_insn_gen_branch_imm(pc, addr, AARCH64_INSN_BRANCH_LINK); + + return ftrace_modify_code(pc, old, new, true); +} + +/* + * The compiler has inserted two NOPs before the regular function prologue. + * All instrumented functions follow the AAPCS, so x0-x8 and x19-x30 are live, + * and x9-x18 are free for our use. + * + * At runtime we want to be able to swing a single NOP <-> BL to enable or + * disable the ftrace call. The BL requires us to save the original LR value, + * so here we insert a over the first NOP so the instructions + * before the regular prologue are: + * + * | Compiled | Disabled | Enabled | + * +----------+------------+------------+ + * | NOP | MOV X9, LR | MOV X9, LR | + * | NOP | NOP | BL | + * + * The LR value will be recovered by ftrace_regs_entry, and restored into LR + * before returning to the regular function prologue. When a function is not + * being traced, the MOV is not harmful given x9 is not live per the AAPCS. + * + * Note: ftrace_process_locs() has pre-adjusted rec->ip to be the address of + * the BL. + */ +int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) +{ + unsigned long pc = rec->ip - AARCH64_INSN_SIZE; + u32 old, new; + + old = aarch64_insn_gen_nop(); + new = aarch64_insn_gen_move_reg(AARCH64_INSN_REG_9, + AARCH64_INSN_REG_LR, + AARCH64_INSN_VARIANT_64BIT); + return ftrace_modify_code(pc, old, new, true); +} +#endif + /* * Turn off the call to ftrace_caller() in instrumented function */ diff --git a/arch/arm64/kernel/module-plts.c b/arch/arm64/kernel/module-plts.c index b182442b87a3..65b08a74aec6 100644 --- a/arch/arm64/kernel/module-plts.c +++ b/arch/arm64/kernel/module-plts.c @@ -4,6 +4,7 @@ */ #include +#include #include #include #include @@ -330,7 +331,7 @@ int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs, tramp->sh_type = SHT_NOBITS; tramp->sh_flags = SHF_EXECINSTR | SHF_ALLOC; tramp->sh_addralign = __alignof__(struct plt_entry); - tramp->sh_size = sizeof(struct plt_entry); + tramp->sh_size = NR_FTRACE_PLTS * sizeof(struct plt_entry); } return 0; diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index 5f5bc3b94da7..00d21a420c60 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -486,21 +486,31 @@ static const Elf_Shdr *find_section(const Elf_Ehdr *hdr, return NULL; } +static inline void __init_plt(struct plt_entry *plt, unsigned long addr) +{ + *plt = get_plt_entry(addr, plt); +} + int module_init_ftrace_plt(const Elf_Ehdr *hdr, const Elf_Shdr *sechdrs, struct module *mod) { #if defined(CONFIG_ARM64_MODULE_PLTS) && defined(CONFIG_DYNAMIC_FTRACE) const Elf_Shdr *s; - struct plt_entry *plt; + struct plt_entry *plts; s = find_section(hdr, sechdrs, ".text.ftrace_trampoline"); if (!s) return -ENOEXEC; - plt = (void *)s->sh_addr; - *plt = get_plt_entry(FTRACE_ADDR, plt); - mod->arch.ftrace_trampoline = plt; + plts = (void *)s->sh_addr; + + __init_plt(&plts[FTRACE_PLT_IDX], FTRACE_ADDR); + + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) + __init_plt(&plts[FTRACE_REGS_PLT_IDX], FTRACE_REGS_ADDR); + + mod->arch.ftrace_trampolines = plts; #endif return 0; } From patchwork Tue Oct 29 16:58:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 178053 Delivered-To: patch@linaro.org Received: by 2002:a92:409a:0:0:0:0:0 with SMTP id d26csp68016ill; Tue, 29 Oct 2019 09:59:32 -0700 (PDT) X-Google-Smtp-Source: APXvYqw/oTwndAXw3P3fHcsa2d9WI4DpweJdtLQUbgd8wcNZsJG8d7LHe7Vwc0CTWvofHLi5sfxs X-Received: by 2002:a05:6402:1490:: with SMTP id e16mr12924216edv.64.1572368372351; Tue, 29 Oct 2019 09:59:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572368372; cv=none; d=google.com; s=arc-20160816; b=PAKzsmLnpw2NsXN831p1+VlU1n6fpbP03WfoeeHuvAC4TN3YaT7d4k8zb6IZ/NLh+h 66yOZMhtW1fiVi7shPZ+TcE/rVpkqgxhxHjhlOGyESC1SYs7gxp/yd9Qfr+7emkydpQF sTuuYgfZ+h/1cz56qmIKAswH//H5wUMBPu5pA5iDfQchPxRjEDvemeV3fRWSq2Ft9pvp Ri9WfsmUYp6hsOCrRhmxiisuAlTrJAwXj1wi2iEPO9N80h6xCgjVcPhyh2wLoqUQWGAm tFrAp1iw7/zu6FYlbd/ifOyEAvaPxNPaTVjS0MYAkliTbRG2IqQ7IqDFGfNFr8Qoowea C9Rw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=38ZImR0bc2gO1AuEhd4HyZN5wcTOmBsJrMsK6iG3DY8=; b=jTKT3iNIZSY3ejR212HEAeOHP3vASrXNu7/pomr5oNLMGNrciib3F/cqxdCky9Fu8f NUlE9BO5g5yzDGhMlgGE6YxIFzEtDjiWeZfXl0KhuoPKc/NzFYsGYdH/vGK3BSRre4IU h5EGAUKUVK61djHDPnSwgTACtN7ml9ckg4Q7OT2ZTXVK+mkDVTjTMGQuIwE3vZ5pDoiH i0zhqe9s2qKqY71EyQOar1kAipbOfZR5BnoAM7sCn2o9wsqV+geJ4VS29lMbXB/CYaAJ UluSwvj15GCR1nNcrWCoPBMkq7kQysXWG7WR7RQYP2I9KAskjOws5VSDsHwhd+StLI1c f+zA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y64si10931516edy.434.2019.10.29.09.59.32; Tue, 29 Oct 2019 09:59:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403777AbfJ2Q72 (ORCPT + 26 others); Tue, 29 Oct 2019 12:59:28 -0400 Received: from foss.arm.com ([217.140.110.172]:54760 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390802AbfJ2Q72 (ORCPT ); Tue, 29 Oct 2019 12:59:28 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5F8347AD; Tue, 29 Oct 2019 09:59:27 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0B57C3F71F; Tue, 29 Oct 2019 09:59:24 -0700 (PDT) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, amit.kachhap@arm.com, catalin.marinas@arm.com, deller@gmx.de, duwe@suse.de, James.Bottomley@HansenPartnership.com, james.morse@arm.com, jeyu@kernel.org, jpoimboe@redhat.com, jthierry@redhat.com, linux-parisc@vger.kernel.org, mark.rutland@arm.com, mingo@redhat.com, peterz@infradead.org, rostedt@goodmis.org, svens@stackframe.org, takahiro.akashi@linaro.org, will@kernel.org Subject: [PATCHv2 8/8] arm64: ftrace: minimize ifdeffery Date: Tue, 29 Oct 2019 16:58:32 +0000 Message-Id: <20191029165832.33606-9-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20191029165832.33606-1-mark.rutland@arm.com> References: <20191029165832.33606-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Now that we no longer refer to mod->arch.ftrace_trampolines in the body of ftrace_make_call(), we can use IS_ENABLED() rather than ifdeffery, and make the code easier to follow. Likewise in ftrace_make_nop(). Let's do so. Signed-off-by: Mark Rutland Reviewed-by: Ard Biesheuvel Cc: Catalin Marinas Cc: Will Deacon --- arch/arm64/kernel/ftrace.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) -- 2.11.0 diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index aea652c33a38..8618faa82e6d 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -62,18 +62,18 @@ int ftrace_update_ftrace_func(ftrace_func_t func) return ftrace_modify_code(pc, 0, new, false); } -#ifdef CONFIG_ARM64_MODULE_PLTS static struct plt_entry *get_ftrace_plt(struct module *mod, unsigned long addr) { +#ifdef CONFIG_ARM64_MODULE_PLTS struct plt_entry *plt = mod->arch.ftrace_trampolines; if (addr == FTRACE_ADDR) return &plt[FTRACE_PLT_IDX]; if (addr == FTRACE_REGS_ADDR && IS_ENABLED(CONFIG_FTRACE_WITH_REGS)) return &plt[FTRACE_REGS_PLT_IDX]; +#endif return NULL; } -#endif /* * Turn on the call to ftrace_caller() in instrumented function @@ -85,10 +85,12 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) long offset = (long)pc - (long)addr; if (offset < -SZ_128M || offset >= SZ_128M) { -#ifdef CONFIG_ARM64_MODULE_PLTS struct module *mod; struct plt_entry *plt; + if (!IS_ENABLED(CONFIG_ARM64_MODULE_PLTS)) + return -EINVAL; + /* * On kernels that support module PLTs, the offset between the * branch instruction and its target may legally exceed the @@ -113,9 +115,6 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) } addr = (unsigned long)plt; -#else /* CONFIG_ARM64_MODULE_PLTS */ - return -EINVAL; -#endif /* CONFIG_ARM64_MODULE_PLTS */ } old = aarch64_insn_gen_nop(); @@ -185,9 +184,11 @@ int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, long offset = (long)pc - (long)addr; if (offset < -SZ_128M || offset >= SZ_128M) { -#ifdef CONFIG_ARM64_MODULE_PLTS u32 replaced; + if (!IS_ENABLED(CONFIG_ARM64_MODULE_PLTS)) + return -EINVAL; + /* * 'mod' is only set at module load time, but if we end up * dealing with an out-of-range condition, we can assume it @@ -218,9 +219,6 @@ int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, return -EINVAL; validate = false; -#else /* CONFIG_ARM64_MODULE_PLTS */ - return -EINVAL; -#endif /* CONFIG_ARM64_MODULE_PLTS */ } else { old = aarch64_insn_gen_branch_imm(pc, addr, AARCH64_INSN_BRANCH_LINK);