From patchwork Wed Nov 22 14:35:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 119492 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp253826qgn; Wed, 22 Nov 2017 06:36:19 -0800 (PST) X-Google-Smtp-Source: AGs4zMa4T+0fkjf7ahPy1XKWaah0BiPkudfIPahqSc3KZJHxKH5LffZSMHDwY5lmOHSVTCKNsBIU X-Received: by 10.98.204.198 with SMTP id j67mr2929507pfk.41.1511361379472; Wed, 22 Nov 2017 06:36:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511361379; cv=none; d=google.com; s=arc-20160816; b=egSQL14eKkmTKRaZmTGxIDUZiQKbPbBnMnNGDKCJpS/7EQMOd6Y7tzFbFNIFYB+Af3 DqVg2PfMt7OuXs1fWxJnP7I40mX+k6xk5Q4R1cm4M1MPYDeSCPWsuPn7apzGBj/QmQQw aw23tD/f2GaAokJBDsAGc6YCT7oh2kMRlaD3uE+4DMoJ3hrIjNGdHOKUj8RS093rxOGa hkQhoqFJbSnp5hBSNj0FQGhHnMGPt/D79LQUE/1kBrvFm4j4Jt44IEC7dPL91agKudOH A7BxaTUUAZaymMmrwaVA+oZE/fL5WUJS5pRacX7DPGREecYvK61zhYpyP4MK6Rz7/SOc 7JRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=CqTr3BxTOITSGBwBI+/wOqMZwCHUhGBTHILVcQDr2cE=; b=Vtcy0mcFVClCPshnQr+Hpk3C9DtI2yzpoLkBgmg0sq2tiTcnhkc4YBDJzr3kiUiK4K 3rVnW0e3A9rz4wpsyt4TjcIMLH41ycr5Uzi8eI8hIJn16LcYahE0kwPNwdZOQE+Kkckl lRhqnmVyUNSZwc1dTFlCdJ2Io8M7KLuG8zLm9siK4WYDvFV2mobKGyoFPFUoo1Ha7Kvb qUKieL1+MOCjTPlnCjQan/rQyq7tZIQYj/8S5rsx7UP+WDuD8bUSNhVc0DgGZa4TjB16 +I3fKvCalDKun1bjw4/MbDA94K0IoZYN0+hKX9VkE6TkmV6RxJUB4vDa4rSr31OT6i3R L2KA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NFYiGJnk; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k15si7396758pgc.476.2017.11.22.06.36.19; Wed, 22 Nov 2017 06:36:19 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NFYiGJnk; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751595AbdKVOgQ (ORCPT + 28 others); Wed, 22 Nov 2017 09:36:16 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:38747 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751460AbdKVOgL (ORCPT ); Wed, 22 Nov 2017 09:36:11 -0500 Received: by mail-wm0-f67.google.com with SMTP id 128so10620298wmo.3 for ; Wed, 22 Nov 2017 06:36:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CqTr3BxTOITSGBwBI+/wOqMZwCHUhGBTHILVcQDr2cE=; b=NFYiGJnkjaMpW8dtUpnvZCZGipaM2c7tXfCKalF0YHCX+n5/VP7mWdb2g2jJMTY8cX kZMnuaI37bTLCCyZu5rKW6S1A2bw1UYiOh99AEs7d3GGFsMOLiIe6KQ2oevjiJhMhncF PhT2sTcSc8lBdHWrSyd+ZrbS5i3KUkV1W+WGc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=CqTr3BxTOITSGBwBI+/wOqMZwCHUhGBTHILVcQDr2cE=; b=CDh1oQb1kIxz5ZEBPwmwI0wrei584TCuTNA2/tInhkellXjaqC53zrALLB+yDWOg88 WMnT76PWRNJ8YvADhT85KYzLe+PG5fet7jwoEWgFtRO8yCFNHFmxQAeWR/BdGbLYGtBN mC4AmJDQRmdrnD/Kcpgt0awB+XPIIE5K7VnleHKenUcJKjGDQL4mLZf80IyYYJjuPX54 7xmldm4qz9OlTU8tfHzFWLbah4sl1X1eE8m7KMBZOSddKNahdDKRf9Eo89dIT5spnEb+ lDq1HNL05THPXmkm/HnsPH1/bDkfy0TwIIoGQ7JFI04umaeP4oRiC/1PV0xRX2xoN5iy vj8w== X-Gm-Message-State: AJaThX6KGY7lnI9COFtSdtCV6Benfa394YaSEg7CdsdkBNJm6K4JKkP6 JcEq1OmcqnYOyH6z/E0DuDvgz0tY8BY= X-Received: by 10.28.136.67 with SMTP id k64mr4163353wmd.133.1511361369768; Wed, 22 Nov 2017 06:36:09 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:f445:1e6c:5367:6c59]) by smtp.gmail.com with ESMTPSA id v18sm7931087wrv.37.2017.11.22.06.36.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 22 Nov 2017 06:36:09 -0800 (PST) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org, rjw@rjwysocki.net Cc: juri.lelli@arm.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, Vincent Guittot Subject: [PATCH v3 1/3] sched/pelt: Move pelt related code in a dedicated file Date: Wed, 22 Nov 2017 15:35:53 +0100 Message-Id: <1511361355-10715-2-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1511361355-10715-1-git-send-email-vincent.guittot@linaro.org> References: <1511361355-10715-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We want to track rt_rq's utilization as a part of the estimation of the whole rq's utilization. This is necessary because rt tasks can steal utilization to cfs tasks and make them lighter than they are. As we want to use the same load tracking mecanism for both and prevent useless dependency between cfs and rt code, pelt code is moved in a dedicated file. Signed-off-by: Vincent Guittot --- kernel/sched/Makefile | 2 +- kernel/sched/fair.c | 308 +------------------------------------------------- kernel/sched/pelt.c | 308 ++++++++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/pelt.h | 17 +++ kernel/sched/sched.h | 20 ++++ 5 files changed, 347 insertions(+), 308 deletions(-) create mode 100644 kernel/sched/pelt.c create mode 100644 kernel/sched/pelt.h -- 2.7.4 diff --git a/kernel/sched/Makefile b/kernel/sched/Makefile index e2f9d4f..5a6d1c1 100644 --- a/kernel/sched/Makefile +++ b/kernel/sched/Makefile @@ -19,7 +19,7 @@ endif obj-y += core.o loadavg.o clock.o cputime.o obj-y += idle_task.o fair.o rt.o deadline.o obj-y += wait.o wait_bit.o swait.o completion.o idle.o -obj-$(CONFIG_SMP) += cpupri.o cpudeadline.o topology.o stop_task.o +obj-$(CONFIG_SMP) += cpupri.o cpudeadline.o topology.o stop_task.o pelt.o obj-$(CONFIG_SCHED_AUTOGROUP) += autogroup.o obj-$(CONFIG_SCHEDSTATS) += stats.o obj-$(CONFIG_SCHED_DEBUG) += debug.o diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0989676..b88550e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -270,9 +270,6 @@ static inline struct rq *rq_of(struct cfs_rq *cfs_rq) return cfs_rq->rq; } -/* An entity is a task if it doesn't "own" a runqueue */ -#define entity_is_task(se) (!se->my_q) - static inline struct task_struct *task_of(struct sched_entity *se) { SCHED_WARN_ON(!entity_is_task(se)); @@ -434,7 +431,6 @@ static inline struct rq *rq_of(struct cfs_rq *cfs_rq) return container_of(cfs_rq, struct rq, cfs); } -#define entity_is_task(se) 1 #define for_each_sched_entity(se) \ for (; se; se = NULL) @@ -707,7 +703,7 @@ static u64 sched_vslice(struct cfs_rq *cfs_rq, struct sched_entity *se) } #ifdef CONFIG_SMP - +#include "pelt.h" #include "sched-pelt.h" static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu); @@ -2723,19 +2719,6 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) } while (0) #ifdef CONFIG_SMP -/* - * XXX we want to get rid of these helpers and use the full load resolution. - */ -static inline long se_weight(struct sched_entity *se) -{ - return scale_load_down(se->load.weight); -} - -static inline long se_runnable(struct sched_entity *se) -{ - return scale_load_down(se->runnable_weight); -} - static inline void enqueue_runnable_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) { @@ -3038,289 +3021,6 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq) } #ifdef CONFIG_SMP -/* - * Approximate: - * val * y^n, where y^32 ~= 0.5 (~1 scheduling period) - */ -static u64 decay_load(u64 val, u64 n) -{ - unsigned int local_n; - - if (unlikely(n > LOAD_AVG_PERIOD * 63)) - return 0; - - /* after bounds checking we can collapse to 32-bit */ - local_n = n; - - /* - * As y^PERIOD = 1/2, we can combine - * y^n = 1/2^(n/PERIOD) * y^(n%PERIOD) - * With a look-up table which covers y^n (n= LOAD_AVG_PERIOD)) { - val >>= local_n / LOAD_AVG_PERIOD; - local_n %= LOAD_AVG_PERIOD; - } - - val = mul_u64_u32_shr(val, runnable_avg_yN_inv[local_n], 32); - return val; -} - -static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3) -{ - u32 c1, c2, c3 = d3; /* y^0 == 1 */ - - /* - * c1 = d1 y^p - */ - c1 = decay_load((u64)d1, periods); - - /* - * p-1 - * c2 = 1024 \Sum y^n - * n=1 - * - * inf inf - * = 1024 ( \Sum y^n - \Sum y^n - y^0 ) - * n=0 n=p - */ - c2 = LOAD_AVG_MAX - decay_load(LOAD_AVG_MAX, periods) - 1024; - - return c1 + c2 + c3; -} - -#define cap_scale(v, s) ((v)*(s) >> SCHED_CAPACITY_SHIFT) - -/* - * Accumulate the three separate parts of the sum; d1 the remainder - * of the last (incomplete) period, d2 the span of full periods and d3 - * the remainder of the (incomplete) current period. - * - * d1 d2 d3 - * ^ ^ ^ - * | | | - * |<->|<----------------->|<--->| - * ... |---x---|------| ... |------|-----x (now) - * - * p-1 - * u' = (u + d1) y^p + 1024 \Sum y^n + d3 y^0 - * n=1 - * - * = u y^p + (Step 1) - * - * p-1 - * d1 y^p + 1024 \Sum y^n + d3 y^0 (Step 2) - * n=1 - */ -static __always_inline u32 -accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, - unsigned long load, unsigned long runnable, int running) -{ - unsigned long scale_freq, scale_cpu; - u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */ - u64 periods; - - scale_freq = arch_scale_freq_capacity(NULL, cpu); - scale_cpu = arch_scale_cpu_capacity(NULL, cpu); - - delta += sa->period_contrib; - periods = delta / 1024; /* A period is 1024us (~1ms) */ - - /* - * Step 1: decay old *_sum if we crossed period boundaries. - */ - if (periods) { - sa->load_sum = decay_load(sa->load_sum, periods); - sa->runnable_load_sum = - decay_load(sa->runnable_load_sum, periods); - sa->util_sum = decay_load((u64)(sa->util_sum), periods); - - /* - * Step 2 - */ - delta %= 1024; - contrib = __accumulate_pelt_segments(periods, - 1024 - sa->period_contrib, delta); - } - sa->period_contrib = delta; - - contrib = cap_scale(contrib, scale_freq); - if (load) - sa->load_sum += load * contrib; - if (runnable) - sa->runnable_load_sum += runnable * contrib; - if (running) - sa->util_sum += contrib * scale_cpu; - - return periods; -} - -/* - * We can represent the historical contribution to runnable average as the - * coefficients of a geometric series. To do this we sub-divide our runnable - * history into segments of approximately 1ms (1024us); label the segment that - * occurred N-ms ago p_N, with p_0 corresponding to the current period, e.g. - * - * [<- 1024us ->|<- 1024us ->|<- 1024us ->| ... - * p0 p1 p2 - * (now) (~1ms ago) (~2ms ago) - * - * Let u_i denote the fraction of p_i that the entity was runnable. - * - * We then designate the fractions u_i as our co-efficients, yielding the - * following representation of historical load: - * u_0 + u_1*y + u_2*y^2 + u_3*y^3 + ... - * - * We choose y based on the with of a reasonably scheduling period, fixing: - * y^32 = 0.5 - * - * This means that the contribution to load ~32ms ago (u_32) will be weighted - * approximately half as much as the contribution to load within the last ms - * (u_0). - * - * When a period "rolls over" and we have new u_0`, multiplying the previous - * sum again by y is sufficient to update: - * load_avg = u_0` + y*(u_0 + u_1*y + u_2*y^2 + ... ) - * = u_0 + u_1*y + u_2*y^2 + ... [re-labeling u_i --> u_{i+1}] - */ -static __always_inline int -___update_load_sum(u64 now, int cpu, struct sched_avg *sa, - unsigned long load, unsigned long runnable, int running) -{ - u64 delta; - - delta = now - sa->last_update_time; - /* - * This should only happen when time goes backwards, which it - * unfortunately does during sched clock init when we swap over to TSC. - */ - if ((s64)delta < 0) { - sa->last_update_time = now; - return 0; - } - - /* - * Use 1024ns as the unit of measurement since it's a reasonable - * approximation of 1us and fast to compute. - */ - delta >>= 10; - if (!delta) - return 0; - - sa->last_update_time += delta << 10; - - /* - * running is a subset of runnable (weight) so running can't be set if - * runnable is clear. But there are some corner cases where the current - * se has been already dequeued but cfs_rq->curr still points to it. - * This means that weight will be 0 but not running for a sched_entity - * but also for a cfs_rq if the latter becomes idle. As an example, - * this happens during idle_balance() which calls - * update_blocked_averages() - */ - if (!load) - runnable = running = 0; - - /* - * Now we know we crossed measurement unit boundaries. The *_avg - * accrues by two steps: - * - * Step 1: accumulate *_sum since last_update_time. If we haven't - * crossed period boundaries, finish. - */ - if (!accumulate_sum(delta, cpu, sa, load, runnable, running)) - return 0; - - return 1; -} - -static __always_inline void -___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runnable) -{ - u32 divider = LOAD_AVG_MAX - 1024 + sa->period_contrib; - - /* - * Step 2: update *_avg. - */ - sa->load_avg = div_u64(load * sa->load_sum, divider); - sa->runnable_load_avg = div_u64(runnable * sa->runnable_load_sum, divider); - sa->util_avg = sa->util_sum / divider; -} - -/* - * sched_entity: - * - * task: - * se_runnable() == se_weight() - * - * group: [ see update_cfs_group() ] - * se_weight() = tg->weight * grq->load_avg / tg->load_avg - * se_runnable() = se_weight(se) * grq->runnable_load_avg / grq->load_avg - * - * load_sum := runnable_sum - * load_avg = se_weight(se) * runnable_avg - * - * runnable_load_sum := runnable_sum - * runnable_load_avg = se_runnable(se) * runnable_avg - * - * XXX collapse load_sum and runnable_load_sum - * - * cfq_rs: - * - * load_sum = \Sum se_weight(se) * se->avg.load_sum - * load_avg = \Sum se->avg.load_avg - * - * runnable_load_sum = \Sum se_runnable(se) * se->avg.runnable_load_sum - * runnable_load_avg = \Sum se->avg.runable_load_avg - */ - -static int -__update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se) -{ - if (entity_is_task(se)) - se->runnable_weight = se->load.weight; - - if (___update_load_sum(now, cpu, &se->avg, 0, 0, 0)) { - ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); - return 1; - } - - return 0; -} - -static int -__update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se) -{ - if (entity_is_task(se)) - se->runnable_weight = se->load.weight; - - if (___update_load_sum(now, cpu, &se->avg, !!se->on_rq, !!se->on_rq, - cfs_rq->curr == se)) { - - ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); - return 1; - } - - return 0; -} - -static int -__update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) -{ - if (___update_load_sum(now, cpu, &cfs_rq->avg, - scale_load_down(cfs_rq->load.weight), - scale_load_down(cfs_rq->runnable_weight), - cfs_rq->curr != NULL)) { - - ___update_load_avg(&cfs_rq->avg, 1, 1); - return 1; - } - - return 0; -} - #ifdef CONFIG_FAIR_GROUP_SCHED /** * update_tg_load_avg - update the tg's load avg @@ -3831,12 +3531,6 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf); #else /* CONFIG_SMP */ -static inline int -update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq) -{ - return 0; -} - #define UPDATE_TG 0x0 #define SKIP_AGE_LOAD 0x0 #define DO_ATTACH 0x0 diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c new file mode 100644 index 0000000..da6d84f --- /dev/null +++ b/kernel/sched/pelt.c @@ -0,0 +1,308 @@ +/* + * Per Entity Load Tracking + * + * Copyright (C) 2007 Red Hat, Inc., Ingo Molnar + * + * Interactivity improvements by Mike Galbraith + * (C) 2007 Mike Galbraith + * + * Various enhancements by Dmitry Adamushko. + * (C) 2007 Dmitry Adamushko + * + * Group scheduling enhancements by Srivatsa Vaddagiri + * Copyright IBM Corporation, 2007 + * Author: Srivatsa Vaddagiri + * + * Scaled math optimizations by Thomas Gleixner + * Copyright (C) 2007, Thomas Gleixner + * + * Adaptive scheduling granularity, math enhancements by Peter Zijlstra + * Copyright (C) 2007 Red Hat, Inc., Peter Zijlstra + * + * Move PELT related code from fair.c into this pelt.c file + * Author: Vincent Guittot + */ + +#include +#include "sched.h" +#include "sched-pelt.h" + +/* + * Approximate: + * val * y^n, where y^32 ~= 0.5 (~1 scheduling period) + */ +static u64 decay_load(u64 val, u64 n) +{ + unsigned int local_n; + + if (unlikely(n > LOAD_AVG_PERIOD * 63)) + return 0; + + /* after bounds checking we can collapse to 32-bit */ + local_n = n; + + /* + * As y^PERIOD = 1/2, we can combine + * y^n = 1/2^(n/PERIOD) * y^(n%PERIOD) + * With a look-up table which covers y^n (n= LOAD_AVG_PERIOD)) { + val >>= local_n / LOAD_AVG_PERIOD; + local_n %= LOAD_AVG_PERIOD; + } + + val = mul_u64_u32_shr(val, runnable_avg_yN_inv[local_n], 32); + return val; +} + +static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3) +{ + u32 c1, c2, c3 = d3; /* y^0 == 1 */ + + /* + * c1 = d1 y^p + */ + c1 = decay_load((u64)d1, periods); + + /* + * p-1 + * c2 = 1024 \Sum y^n + * n=1 + * + * inf inf + * = 1024 ( \Sum y^n - \Sum y^n - y^0 ) + * n=0 n=p + */ + c2 = LOAD_AVG_MAX - decay_load(LOAD_AVG_MAX, periods) - 1024; + + return c1 + c2 + c3; +} + +#define cap_scale(v, s) ((v)*(s) >> SCHED_CAPACITY_SHIFT) + +/* + * Accumulate the three separate parts of the sum; d1 the remainder + * of the last (incomplete) period, d2 the span of full periods and d3 + * the remainder of the (incomplete) current period. + * + * d1 d2 d3 + * ^ ^ ^ + * | | | + * |<->|<----------------->|<--->| + * ... |---x---|------| ... |------|-----x (now) + * + * p-1 + * u' = (u + d1) y^p + 1024 \Sum y^n + d3 y^0 + * n=1 + * + * = u y^p + (Step 1) + * + * p-1 + * d1 y^p + 1024 \Sum y^n + d3 y^0 (Step 2) + * n=1 + */ +static __always_inline u32 +accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, + unsigned long load, unsigned long runnable, int running) +{ + unsigned long scale_freq, scale_cpu; + u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */ + u64 periods; + + scale_freq = arch_scale_freq_capacity(NULL, cpu); + scale_cpu = arch_scale_cpu_capacity(NULL, cpu); + + delta += sa->period_contrib; + periods = delta / 1024; /* A period is 1024us (~1ms) */ + + /* + * Step 1: decay old *_sum if we crossed period boundaries. + */ + if (periods) { + sa->load_sum = decay_load(sa->load_sum, periods); + sa->runnable_load_sum = + decay_load(sa->runnable_load_sum, periods); + sa->util_sum = decay_load((u64)(sa->util_sum), periods); + + /* + * Step 2 + */ + delta %= 1024; + contrib = __accumulate_pelt_segments(periods, + 1024 - sa->period_contrib, delta); + } + sa->period_contrib = delta; + + contrib = cap_scale(contrib, scale_freq); + if (load) + sa->load_sum += load * contrib; + if (runnable) + sa->runnable_load_sum += runnable * contrib; + if (running) + sa->util_sum += contrib * scale_cpu; + + return periods; +} + +/* + * We can represent the historical contribution to runnable average as the + * coefficients of a geometric series. To do this we sub-divide our runnable + * history into segments of approximately 1ms (1024us); label the segment that + * occurred N-ms ago p_N, with p_0 corresponding to the current period, e.g. + * + * [<- 1024us ->|<- 1024us ->|<- 1024us ->| ... + * p0 p1 p2 + * (now) (~1ms ago) (~2ms ago) + * + * Let u_i denote the fraction of p_i that the entity was runnable. + * + * We then designate the fractions u_i as our co-efficients, yielding the + * following representation of historical load: + * u_0 + u_1*y + u_2*y^2 + u_3*y^3 + ... + * + * We choose y based on the with of a reasonably scheduling period, fixing: + * y^32 = 0.5 + * + * This means that the contribution to load ~32ms ago (u_32) will be weighted + * approximately half as much as the contribution to load within the last ms + * (u_0). + * + * When a period "rolls over" and we have new u_0`, multiplying the previous + * sum again by y is sufficient to update: + * load_avg = u_0` + y*(u_0 + u_1*y + u_2*y^2 + ... ) + * = u_0 + u_1*y + u_2*y^2 + ... [re-labeling u_i --> u_{i+1}] + */ +static __always_inline int +___update_load_sum(u64 now, int cpu, struct sched_avg *sa, + unsigned long load, unsigned long runnable, int running) +{ + u64 delta; + + delta = now - sa->last_update_time; + /* + * This should only happen when time goes backwards, which it + * unfortunately does during sched clock init when we swap over to TSC. + */ + if ((s64)delta < 0) { + sa->last_update_time = now; + return 0; + } + + /* + * Use 1024ns as the unit of measurement since it's a reasonable + * approximation of 1us and fast to compute. + */ + delta >>= 10; + if (!delta) + return 0; + + sa->last_update_time += delta << 10; + + /* + * running is a subset of runnable (weight) so running can't be set if + * runnable is clear. But there are some corner cases where the current + * se has been already dequeued but cfs_rq->curr still points to it. + * This means that weight will be 0 but not running for a sched_entity + * but also for a cfs_rq if the latter becomes idle. As an example, + * this happens during idle_balance() which calls + * update_blocked_averages() + */ + if (!load) + runnable = running = 0; + + /* + * Now we know we crossed measurement unit boundaries. The *_avg + * accrues by two steps: + * + * Step 1: accumulate *_sum since last_update_time. If we haven't + * crossed period boundaries, finish. + */ + if (!accumulate_sum(delta, cpu, sa, load, runnable, running)) + return 0; + + return 1; +} + +static __always_inline void +___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runnable) +{ + u32 divider = LOAD_AVG_MAX - 1024 + sa->period_contrib; + + /* + * Step 2: update *_avg. + */ + sa->load_avg = div_u64(load * sa->load_sum, divider); + sa->runnable_load_avg = div_u64(runnable * sa->runnable_load_sum, divider); + sa->util_avg = sa->util_sum / divider; +} + +/* + * sched_entity: + * + * task: + * se_runnable() == se_weight() + * + * group: [ see update_cfs_group() ] + * se_weight() = tg->weight * grq->load_avg / tg->load_avg + * se_runnable() = se_weight(se) * grq->runnable_load_avg / grq->load_avg + * + * load_sum := runnable_sum + * load_avg = se_weight(se) * runnable_avg + * + * runnable_load_sum := runnable_sum + * runnable_load_avg = se_runnable(se) * runnable_avg + * + * XXX collapse load_sum and runnable_load_sum + * + * cfq_rs: + * + * load_sum = \Sum se_weight(se) * se->avg.load_sum + * load_avg = \Sum se->avg.load_avg + * + * runnable_load_sum = \Sum se_runnable(se) * se->avg.runnable_load_sum + * runnable_load_avg = \Sum se->avg.runable_load_avg + */ + +int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se) +{ + if (entity_is_task(se)) + se->runnable_weight = se->load.weight; + + if (___update_load_sum(now, cpu, &se->avg, 0, 0, 0)) { + ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); + return 1; + } + + return 0; +} + +int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + if (entity_is_task(se)) + se->runnable_weight = se->load.weight; + + if (___update_load_sum(now, cpu, &se->avg, !!se->on_rq, !!se->on_rq, + cfs_rq->curr == se)) { + + ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); + return 1; + } + + return 0; +} + +int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) +{ + if (___update_load_sum(now, cpu, &cfs_rq->avg, + scale_load_down(cfs_rq->load.weight), + scale_load_down(cfs_rq->runnable_weight), + cfs_rq->curr != NULL)) { + + ___update_load_avg(&cfs_rq->avg, 1, 1); + return 1; + } + + return 0; +} diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h new file mode 100644 index 0000000..c312d8c --- /dev/null +++ b/kernel/sched/pelt.h @@ -0,0 +1,17 @@ +#ifdef CONFIG_SMP + +int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se); +int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se); +int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); + +#else + +static inline int +update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq) +{ + return 0; +} + +#endif + + diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 45ab0bf..6fefef6 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -528,6 +528,7 @@ struct rt_rq { unsigned long rt_nr_total; int overloaded; struct plist_head pushable_tasks; + #endif /* CONFIG_SMP */ int rt_queued; @@ -602,7 +603,26 @@ struct dl_rq { u64 bw_ratio; }; +#ifdef CONFIG_FAIR_GROUP_SCHED +/* An entity is a task if it doesn't "own" a runqueue */ +#define entity_is_task(se) (!se->my_q) +#else +#define entity_is_task(se) 1 +#endif + #ifdef CONFIG_SMP +/* + * XXX we want to get rid of these helpers and use the full load resolution. + */ +static inline long se_weight(struct sched_entity *se) +{ + return scale_load_down(se->load.weight); +} + +static inline long se_runnable(struct sched_entity *se) +{ + return scale_load_down(se->runnable_weight); +} static inline bool sched_asym_prefer(int a, int b) { From patchwork Wed Nov 22 14:35:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 119494 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp254087qgn; Wed, 22 Nov 2017 06:36:34 -0800 (PST) X-Google-Smtp-Source: AGs4zMYKzAAYuNxOhOYBQvZ0mFn+hVOaHCpZbB/kSxyiaJAAV8zNNgaCYw83Mlcibjpx45j7BhW3 X-Received: by 10.98.33.203 with SMTP id o72mr19242063pfj.163.1511361394432; Wed, 22 Nov 2017 06:36:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511361394; cv=none; d=google.com; s=arc-20160816; b=R3mO9zXa4hEdFFnwdpsSgnTT4/Ntkug/ut1ijCgnXzKRFXJQ3Pm/cxaw0qEcfMBgKp Kpf3byvy3p0ksbZRyRR57haxJY8Xz5iMcZeI2IEew7PT7BxP047r88dv1odZ2/1JAZZE pPCuw6vVRCsL+EGfQ2OTTdF/eEJwr57iDN0DHyb7bqSb5+yC6G7OS1/yK0u2bMYsZY5O NozPDMasuDAh51tN0cvoHjVR2Jryxy2nJlFLU1dRhl7vUFqOLeIQiHJzOCloxUu8BX5F 9xpMeY9pyRZ0pmfRgq0pLUu2VYnq5OggBzJJh+tnndilAsaJH1PtX85gtsb91T75zodC LeXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=JpfvcPHa35i386+8TxxK0xdOEQR6x4zGL6DkJiJrAD8=; b=AesZS+ndWnCzLnpF5oD8ye21V9aeLjNB56OoUaPBx806r9bDqaAtV3O82MvigO/7Um zZDr8vxt6TxhIXu3LtVkJGppTbRWiQjJw3wRn/YbUmf3nyundqBGERz3OhF7noAY22tn XyRQ3g8cEgUKsYfp4cIyvGTTLdhv7HSj00HBXFi7vBuxceAv8GHSjS4CzUWr2F8nFUkg fFgWx3E7AZlid1Ms3gIX4eRADQhfJTSVEEoJ26Dt877Js4b/Kkkb8uvn7YwzLkUskj90 TBo4CV+1gaVjApR/GYDnX7pWv+73mzboGNf9gShOWxKfqxnPac5LpmUSWIa0rrMGHOiX BMVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=h1tEIdoj; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k10si13407686pgq.750.2017.11.22.06.36.33; Wed, 22 Nov 2017 06:36:34 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=h1tEIdoj; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751677AbdKVOgb (ORCPT + 28 others); Wed, 22 Nov 2017 09:36:31 -0500 Received: from mail-wr0-f194.google.com ([209.85.128.194]:33799 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751274AbdKVOgM (ORCPT ); Wed, 22 Nov 2017 09:36:12 -0500 Received: by mail-wr0-f194.google.com with SMTP id k18so9770562wre.1 for ; Wed, 22 Nov 2017 06:36:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=JpfvcPHa35i386+8TxxK0xdOEQR6x4zGL6DkJiJrAD8=; b=h1tEIdoj19IY2Iwzy2LwqqNhktP+r7G+4K3UJWGmTf6/0yAPE+4+inA2hK3HZUGb0d HCDxktkpHlZumXZVLa3Dh3bn5uFFn+4W7Up2KwVoY8cP7kkztT8U5OSSoPloGsrjNg93 RpK0jy2uFuLFStU+lln/Xaj39k57pt8HF+ors= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=JpfvcPHa35i386+8TxxK0xdOEQR6x4zGL6DkJiJrAD8=; b=r4lgXQiU5pQwgktKA32gG1CWX6VcuJwFmijTjB4btmxve7Go/m4yuO4Wb0wTQ7Kd2w kUyB+CPxKO+Kfite40rLBQnyiRjLbcrpQJCe4AP3E7CshRWBY/7vvLExY7bWGyN6FrSl jmY8M3ReuhhEOlWPeHB4OP0RO0CTajuLhg++Oz5i7V/uzjGXWIzXZHG7cZoM8/QOeCP1 Zvb4CR4lSG/NB19/bxEtVZL0fasdHliLPGK8jcqQ70hCgUupcMRiWVzkwB/JQTNK+l6w FajOuCmIsTeeJJNyTwj995PDRiE0GJBQA84ygJkrDnqW9zA+PVH9IEf/PLDLd4bugvh+ qY5Q== X-Gm-Message-State: AJaThX6KMxmxxklLpeuEg+4TxZgWHL6JtuzEwiKuof4+ZVaKRfKEsHkp geOM0RrQdej/rOO7O6DGHVIHfA== X-Received: by 10.223.176.150 with SMTP id i22mr19150221wra.257.1511361370875; Wed, 22 Nov 2017 06:36:10 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:f445:1e6c:5367:6c59]) by smtp.gmail.com with ESMTPSA id v18sm7931087wrv.37.2017.11.22.06.36.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 22 Nov 2017 06:36:10 -0800 (PST) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org, rjw@rjwysocki.net Cc: juri.lelli@arm.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, Vincent Guittot Subject: [PATCH v3 2/3] sched/rt: add rt_rq utilization tracking Date: Wed, 22 Nov 2017 15:35:54 +0100 Message-Id: <1511361355-10715-3-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1511361355-10715-1-git-send-email-vincent.guittot@linaro.org> References: <1511361355-10715-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org schedutil governor relies on cfs_rq's util_avg to choose the OPP when cfs tasks are running. When the CPU is overloaded by cfs and rt tasks, cfs tasks are preempted by rt tasks and in this case util_avg reflects the remaining capacity that is used by cfs task but not what cfs want to use. In such case, schedutil can select a lower OPP whereas the CPU is overloaded. In order to have a more accurate view of the utilization of the CPU, we track the utilization that is "stolen" by RT tasks. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 2 ++ kernel/sched/pelt.c | 23 +++++++++++++++++++++++ kernel/sched/pelt.h | 7 +++++++ kernel/sched/rt.c | 9 +++++++++ kernel/sched/sched.h | 1 + 5 files changed, 42 insertions(+) -- 2.7.4 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b88550e..57d486a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7029,6 +7029,7 @@ static void update_blocked_averages(int cpu) if (cfs_rq_is_decayed(cfs_rq)) list_del_leaf_cfs_rq(cfs_rq); } + update_rt_rq_load_avg(rq_clock_task(rq), cpu, &rq->rt, 0); rq_unlock_irqrestore(rq, &rf); } @@ -7088,6 +7089,7 @@ static inline void update_blocked_averages(int cpu) rq_lock_irqsave(rq, &rf); update_rq_clock(rq); update_cfs_rq_load_avg(cfs_rq_clock_task(cfs_rq), cfs_rq); + update_rt_rq_load_avg(rq_clock_task(rq), cpu, &rq->rt, 0); rq_unlock_irqrestore(rq, &rf); } diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index da6d84f..c8b5d23 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -306,3 +306,26 @@ int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) return 0; } + +/* + * rt_rq: + * + * util_sum = \Sum se->avg.util_sum but se->avg.util_sum is not tracked + * util_sum = cpu_scale * load_sum + * runnable_load_sum = load_sum + * + */ + +int update_rt_rq_load_avg(u64 now, int cpu, struct rt_rq *rt_rq, int running) +{ + if (___update_load_sum(now, cpu, &rt_rq->avg, + running, + running, + running)) { + + ___update_load_avg(&rt_rq->avg, 1, 1); + return 1; + } + + return 0; +} diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h index c312d8c..78a2107 100644 --- a/kernel/sched/pelt.h +++ b/kernel/sched/pelt.h @@ -3,6 +3,7 @@ int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se); int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se); int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); +int update_rt_rq_load_avg(u64 now, int cpu, struct rt_rq *rt_rq, int running); #else @@ -12,6 +13,12 @@ update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq) return 0; } +static inline int +update_rt_rq_load_avg(u64 now, int cpu, struct rt_rq *rt_rq, int running) +{ + return 0; +} + #endif diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index d8c43d7..2ddcc27 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -9,6 +9,8 @@ #include #include +#include "pelt.h" + int sched_rr_timeslice = RR_TIMESLICE; int sysctl_sched_rr_timeslice = (MSEC_PER_SEC / HZ) * RR_TIMESLICE; @@ -1569,6 +1571,10 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) queue_push_tasks(rq); + if (p) + update_rt_rq_load_avg(rq_clock_task(rq), cpu_of(rq), rt_rq, + rq->curr->sched_class == &rt_sched_class); + return p; } @@ -1576,6 +1582,8 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p) { update_curr_rt(rq); + update_rt_rq_load_avg(rq_clock_task(rq), cpu_of(rq), &rq->rt, 1); + /* * The previous task needs to be made eligible for pushing * if it is still active @@ -2283,6 +2291,7 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued) struct sched_rt_entity *rt_se = &p->rt; update_curr_rt(rq); + update_rt_rq_load_avg(rq_clock_task(rq), cpu_of(rq), &rq->rt, 1); watchdog(rq, p); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 6fefef6..c7bd5dd 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -529,6 +529,7 @@ struct rt_rq { int overloaded; struct plist_head pushable_tasks; + struct sched_avg avg; #endif /* CONFIG_SMP */ int rt_queued; From patchwork Wed Nov 22 14:35:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 119493 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp254059qgn; Wed, 22 Nov 2017 06:36:33 -0800 (PST) X-Google-Smtp-Source: AGs4zMYnDOUOgR5RXLVAjNnQmYoM7p37MLaaTpzaOk0/2LA543xQTTzGBSzmlVC7hycrzpYsoxv9 X-Received: by 10.99.126.78 with SMTP id o14mr21046070pgn.159.1511361392915; Wed, 22 Nov 2017 06:36:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511361392; cv=none; d=google.com; s=arc-20160816; b=CRy7+SRTH9r3jiZyeTh+KcbBPEg0FEIcfZAt+rAXHS879BVtoeqYwp9mpLQN3VR1wV /m5ZRTXap/OdRbnwCVgjEjLyA4RKLZyX4UoYr2+2oD0zbx/jnnPy8YqQGZJaSE4OBTUH 0hXgsS7RF+9//TNwSZvO6HCxljUSQ8gzSARtCP40EReqLI0XelS3B8oMlGSefw/wt+It Kn2tazuayii8oyJvtERaWmAy2ug7EbxEU009pfSaXwpmImF5EsOBjFfP/7QvdnTujDzt GtwLmuS1k5rQjZUlabrGYeeM8jXZ3z1Hr7kDJlixR267Jzs9/8UeHHxKkjvsKVdZYTxh 9DzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=IAUQ9Qmuei7/n87obcKNnGkKXEbQTDehiueOWWs0zv4=; b=nH6h+BzYiDRHhNmYS9monAnAeFEFXIeP3/iun6RY78NXBHCTP9Z2jxUNmj2g2cj8Cl 7RcOk6IEZz2co3fRoSyIdwjnJpd9Mvs/x/wPvxyQo6XFHOYPDL/Aoofaaun43A3VIyeJ JEpNqidDO5gRbwBQ/3zxC+9HSxUAgx9hOlCd+IPRJGKez+y6FhaJd6B9Y52t3TMl/fLY Rl+wrTn1WmbBxwSs4B1KNV2/KwkYq/RTym76QzAiXyNcZGtHoYwMauTXsR1ik/TMdYSZ 0ZDtwKTwQJKW+2NtFhqjcdcWOv0oIojP6nqhomPY5UI6O4dq6wc3i7437MplCLGk7ZYF HRmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ffLJwGCQ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n18si14856590pfh.268.2017.11.22.06.36.32; Wed, 22 Nov 2017 06:36:32 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ffLJwGCQ; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751644AbdKVOga (ORCPT + 28 others); Wed, 22 Nov 2017 09:36:30 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:41573 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751510AbdKVOgN (ORCPT ); Wed, 22 Nov 2017 09:36:13 -0500 Received: by mail-wr0-f193.google.com with SMTP id z14so14715002wrb.8 for ; Wed, 22 Nov 2017 06:36:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IAUQ9Qmuei7/n87obcKNnGkKXEbQTDehiueOWWs0zv4=; b=ffLJwGCQKGH+WknWgmG8tm6YV/GbIlQnZjJcyo6E+gt8k5kBfZmyIxOSGCvOxaWNMC r5Wp5MX+mUgU9IBmj9P4OHggZE/p91OpA+jNdu1yvBL84kh2zur3KJKWobPaU+5FOG6+ jAfYO83WdmjNMZG362tKFichYG8OZHIBOpo7w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IAUQ9Qmuei7/n87obcKNnGkKXEbQTDehiueOWWs0zv4=; b=J/F8pWETUX+qWI7PR+KMTMXadQ0sy5j1ztXXfh2TzG6iURRNLT7EYIzyitC/a5TKYd GcF7MF87RPuHYkMgnAYLvdmN2DeL1NJisyYmFHeUou/AXJm3anPY62nXrpL1H33U05Xz MguCdub/qt+9omZYG7mPHqmaCsIWXvVHUCGewz5zYu1YcGHYZKy5+tOfdO4ZsZIXvPVO Kp9RxgO/VaJvxIvjRLNd4HiB/cy8hQ1paWBFDQPCtx6s8zV86pKWhHq8TRzLZfk3Hfky /niMSqJy4WyBOx0/cqOuqB8lURSwQnqlLN0UcTglPsL4S897PB30/WV3J1CB/x2e/w8n fdXQ== X-Gm-Message-State: AJaThX45VA2vsffepI72r1ivl4MBRd2kzi8urN3yKp2FLSgeg8rLIAG3 X0JwqHGZRZ7R6aUiMIQ+kJP5VA== X-Received: by 10.223.129.35 with SMTP id 32mr16802013wrm.271.1511361371976; Wed, 22 Nov 2017 06:36:11 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:f445:1e6c:5367:6c59]) by smtp.gmail.com with ESMTPSA id v18sm7931087wrv.37.2017.11.22.06.36.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 22 Nov 2017 06:36:11 -0800 (PST) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org, rjw@rjwysocki.net Cc: juri.lelli@arm.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, Vincent Guittot Subject: [PATCH 3/3] cpufreq/schedutil: add rt utilization tracking Date: Wed, 22 Nov 2017 15:35:55 +0100 Message-Id: <1511361355-10715-4-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1511361355-10715-1-git-send-email-vincent.guittot@linaro.org> References: <1511361355-10715-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org add both cfs and rt utilization when selecting an OPP as rt can preempt and steal cfs's running time Signed-off-by: Vincent Guittot --- kernel/sched/cpufreq_schedutil.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.7.4 diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 9209d83..d06a41e 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -183,7 +183,7 @@ static void sugov_get_util(unsigned long *util, unsigned long *max, int cpu) cfs_max = arch_scale_cpu_capacity(NULL, cpu); - *util = min(rq->cfs.avg.util_avg, cfs_max); + *util = min(rq->cfs.avg.util_avg + rq->rt.avg.util_avg, cfs_max); *max = cfs_max; }