From patchwork Wed Nov 8 18:41:01 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Bellasi X-Patchwork-Id: 118327 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5638007qgn; Wed, 8 Nov 2017 10:41:11 -0800 (PST) X-Google-Smtp-Source: ABhQp+Qb9tPBdYJ14lGOy5IDlQC7MeJ0yZtQPBFzkoTeitEwf8y7N15+Yv+Sdpoh4e3pYUyTYnp+ X-Received: by 10.159.207.147 with SMTP id z19mr1243633plo.441.1510166471900; Wed, 08 Nov 2017 10:41:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510166471; cv=none; d=google.com; s=arc-20160816; b=A3lsA1Y/T2r9N0MwoTbcvd83DTDZiY+7pT9IiAG4BRtKLafgSlo/7I/jetTgHuifNR lCb8jdpBxn+aLhfKQVu7tWV1OXQ9qOqwFFRs0ytbY41lo5TtlmmyhwY88QemuYeA7+oA +Qd5Ue8F8a2vd0OF79qIS7gxYbCDIdXkauPtdqMCuQYC6hf+HMxnj+XbwoF0xsXqEc4G +i4VgdipH9rUUI0CL7PJ/8ktLsC7llLAE7Xz1Ph0bgl0lLziZh6pHu4OmGEJ/muKzfuk RCp03XU4joPqsrnFACRj6bDTWzgsNdQMsrTG+hcnl3RHk+RD6dUho3iahT6YUGhsQCPr IW9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :arc-authentication-results; bh=SxaJ/jLrcR4z3c+GWQ7taV7ruE68LE85b5Iz+GrVKss=; b=tAKW7aB7uFT4vzTjlxMltwb5Bmp5AmAGLm4+G5fD+bxpgYFwYd4U2U92KwXWcPwMv8 mWEaKuqyMjYIrjAYyvPzyh7QtdUatvFO3bHdPVhlgtXzqvEUioto7nJuCV6AwdYIXaom PW9Cm0Dq1dU34jYfHyQgQ8cBNiSKBcBEMACL+fI6epjUMGhRSEL7kxM5nvH8wE/kw8Yp fdkuIgU0eQfZFFn4IpQobQNYdJB5a1GRnLxcatnhfN8eAqHb9X6igWo+0WYGuwmoxBqp bbroDeuFph6d1j1J6g2Thod0OOemCyemcgoOOTIk3xMKV5+qofSCsvmmvNQqIaR/5iqq xN8g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g207si4770790pfb.8.2017.11.08.10.41.11; Wed, 08 Nov 2017 10:41:11 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752211AbdKHSlK (ORCPT + 25 others); Wed, 8 Nov 2017 13:41:10 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:37502 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751839AbdKHSlJ (ORCPT ); Wed, 8 Nov 2017 13:41:09 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EAD7F14; Wed, 8 Nov 2017 10:41:08 -0800 (PST) Received: from e110439-lin.cambridge.arm.com (e110439-lin.cambridge.arm.com [10.1.210.68]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 27F1F3F3DF; Wed, 8 Nov 2017 10:41:07 -0800 (PST) From: Patrick Bellasi To: linux-kernel@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Juri Lelli , Vincent Guittot , Morten Rasmussen , Dietmar Eggemann , Chris Redpath Subject: [PATCH v2] sched: fix sched_feat for !SCHED_DEBUG builds Date: Wed, 8 Nov 2017 18:41:01 +0000 Message-Id: <20171108184101.16006-1-patrick.bellasi@arm.com> X-Mailer: git-send-email 2.14.1 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When the kernel is compiled with !SCHED_DEBUG support, we expect that all SCHED_FEAT are turned into compile time constants being propagated to support compiler optimizations. Specifically, we expect that code blocks like this: if (sched_feat(FEATURE_NAME) [&& ]) { /* FEATURE CODE */ } are turned into dead-code in case FEATURE_NAME defaults to FALSE, and thus being removed by the compiler from the finale image. For this mechanism to properly work it's required for the compiler to have full access, from each translation unit, to whatever is the value defined by the sched_feat macro. This macro is defined as: #define sched_feat(x) (sysctl_sched_features & (1UL << __SCHED_FEAT_##x)) and thus, the compiler can optimize that code only if the value of sysctl_sched_features is visible within each translation unit. Since: 029632fbb sched: Make separate sched*.c translation units the scheduler code has been split into separate translation units however the definition of sysctl_sched_features is part of kernel/sched/core.c while, for all the other scheduler modules, it is visible only via kernel/sched/sched.h as an: extern const_debug unsigned int sysctl_sched_features Unfortunately, an extern reference does not allow the compiler to apply constants propagation. Thus, on !SCHED_DEBUG kernel we still end up with code to load a memory reference and (eventually) doing an unconditional jump of a chunk of code. This mechanism is unavoidable when sched_features can be turned on and off at run-time. However, this is not the case for "production" kernels compiled with !SCHED_DEBUG. In this case, sysctl_sched_features is just a constant value which cannot be changed at run-time and thus memory loads and jumps can be avoided altogether. This patch fixes the case of !SCHED_DEBUG kernel by declaring a local version of the sysctl_sched_features constant for each translation unit. This will ultimately allow the compiler to perform constants propagation and dead-code pruning. Tests have been done, with !SCHED_DEBUG on a v4.14-rc8 with and without the patch, by running 30 iterations of: perf bench sched messaging --pipe --thread --group 4 --loop 50000 on a 40 cores Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz using the powersave governor to rule out variations due to frequency scaling. Statistics on the reported completion time: count mean std min 99% max v4.14-rc8 30.0 15.7831 0.176032 15.442 16.01226 16.014 v4.14-rc8+patch 30.0 15.5033 0.189681 15.232 15.93938 15.962 show a 1.8% speedup on average completion time and 0.5% speedup in the 99 percentile. Signed-off-by: Patrick Bellasi Signed-off-by: Chris Redpath Reviewed-by: Dietmar Eggemann Reviewed-by: Brendan Jackman Cc: Ingo Molnar Cc: Peter Zijlstra Cc: linux-kernel@vger.kernel.org --- kernel/sched/core.c | 9 ++++++--- kernel/sched/sched.h | 25 ++++++++++++++++++++++--- 2 files changed, 28 insertions(+), 6 deletions(-) -- 2.14.1 diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d17c5da523a0..0edb08716555 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -42,18 +42,21 @@ DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +#if defined(CONFIG_SCHED_DEBUG) && defined(HAVE_JUMP_LABEL) /* * Debugging: various feature bits + * + * If SCHED_DEBUG is disabled, each compilation unit has its own copy of + * sysctl_sched_features, defined in sched.h, to allow constants propagation + * at compile time and compiler optimization based on features default. */ - #define SCHED_FEAT(name, enabled) \ (1UL << __SCHED_FEAT_##name) * enabled | - const_debug unsigned int sysctl_sched_features = #include "features.h" 0; - #undef SCHED_FEAT +#endif /* * Number of tasks to iterate in a single balance run. diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 3b448ba82225..2b8647a3f61f 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1219,8 +1219,6 @@ static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu) # define const_debug const #endif -extern const_debug unsigned int sysctl_sched_features; - #define SCHED_FEAT(name, enabled) \ __SCHED_FEAT_##name , @@ -1232,6 +1230,13 @@ enum { #undef SCHED_FEAT #if defined(CONFIG_SCHED_DEBUG) && defined(HAVE_JUMP_LABEL) + +/* + * To support run-time toggling of sched features, all the translation units + * (but core.c) reference the sysctl_sched_features defined in core.c. + */ +extern const_debug unsigned int sysctl_sched_features; + #define SCHED_FEAT(name, enabled) \ static __always_inline bool static_branch_##name(struct static_key *key) \ { \ @@ -1239,13 +1244,27 @@ static __always_inline bool static_branch_##name(struct static_key *key) \ } #include "features.h" - #undef SCHED_FEAT extern struct static_key sched_feat_keys[__SCHED_FEAT_NR]; #define sched_feat(x) (static_branch_##x(&sched_feat_keys[__SCHED_FEAT_##x])) + #else /* !(SCHED_DEBUG && HAVE_JUMP_LABEL) */ + +/* + * Each translation unit has its own copy of sysctl_sched_features to allow + * constants propagation at compile time and compiler optimization based on + * features default. + */ +#define SCHED_FEAT(name, enabled) \ + (1UL << __SCHED_FEAT_##name) * enabled | +static const_debug __maybe_unused unsigned int sysctl_sched_features = +#include "features.h" + 0; +#undef SCHED_FEAT + #define sched_feat(x) (sysctl_sched_features & (1UL << __SCHED_FEAT_##x)) + #endif /* SCHED_DEBUG && HAVE_JUMP_LABEL */ extern struct static_key_false sched_numa_balancing;