From patchwork Thu Jun 28 15:45:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140460 Delivered-To: patch@linaro.org Received: by 2002:a50:86eb:0:0:0:0:0 with SMTP id 40-v6csp1954831edu; Thu, 28 Jun 2018 08:46:53 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKT7ZrqSv9jRZ1NT15TJJtyL4IGiAOag1HO0Cn7gvuDqba6iWpPeAfaARiNgtFnLL/URXZg X-Received: by 2002:a17:902:7c8b:: with SMTP id y11-v6mr10700046pll.222.1530200813279; Thu, 28 Jun 2018 08:46:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200813; cv=none; d=google.com; s=arc-20160816; b=Cjd5jz+UVmLlVZm+jOVkJTLmogQSVX25GC5WFiOk77QuGOhzel0kzfzepWAPHA0ZWq s5/gHm/wRz3nLeVHHEn5r7uwrLIO/vbqfRZwimYH+SOGOlKNOYpBi9QbU3uBD4X75te3 /rTEOUEUT6HD5tlaE/LDu6X/CabDGU1OWWjyEXC+NfvLe4a7JPSEYRiIh1kAtysVlk3S 5lX5iEv0v7cuELu3aIhwjP0QWHiq5IlO5qM09nDR8uzr6SPhhnJ+RUKtXBGaSVdMBnp4 aAwBeNamoaZTq09a4oL22blEJSOwyfn4npTH1UQV4uj87GScNGbWW9qyGhK2GM/sJqO5 d10Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=kdbaVO8MqOUS2uy/Xi5v5kMbhesXvCPgjGvHJVmNWfY=; b=FYfMXptZ1FIijA0WIZJ/CZxf9A44Hsdx6FIC9r0e/7gWTb1a5KN31LR90pnQrScoH8 q5olaiBPIWRQ0OV524IaAysuz4kMJg1L51JZQAcVCIIqpS0xzXh+5ruMlnvY31yg6g9h 37OFT6ZFgms4DfCq3pq1uLHPBHVwtoMrw8RkD+C/h7pGwC/kWDe6qBZaXmdcA9U6J+Su jbKAH6x9/WoWdEnaz2B+C0isffPCzhPCqvKw3Q01dC83XjJAj4hNVbwl+lbzELtlTL3x ZZ6yoy3n05aqDXpHqbLI2XLEtsRA8W7dDLqxElmOEcjjGbTlkRGKKPnw9fwKYtGer1RP bA4A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LFPE7BFT; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w4-v6si5879949pgr.549.2018.06.28.08.46.52; Thu, 28 Jun 2018 08:46:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LFPE7BFT; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S967318AbeF1Pp3 (ORCPT + 31 others); Thu, 28 Jun 2018 11:45:29 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:36385 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965462AbeF1PpY (ORCPT ); Thu, 28 Jun 2018 11:45:24 -0400 Received: by mail-wr0-f193.google.com with SMTP id f16-v6so6010522wrm.3 for ; Thu, 28 Jun 2018 08:45:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kdbaVO8MqOUS2uy/Xi5v5kMbhesXvCPgjGvHJVmNWfY=; b=LFPE7BFTrZZQJG9/rH40dcU7Z69wCqiyNx8ZqcA9VbukNSs7WAvHmiWDVhBgFIlzW4 xU2XrVl5dEZ4yvVW+yghMbYGOR2wjx6hcNBPhynKn9lusubG8owKWKLYle4sX7HWBP2T fQFEeW931C/rHN2wg+Baoy4Y7CMp1m4uDOvyI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=kdbaVO8MqOUS2uy/Xi5v5kMbhesXvCPgjGvHJVmNWfY=; b=TtfRuBDGO6ZUZmNbjEgeQPitHINB0SDEWb73E/zvTZQUBt74M6YLWCS7CEmNQz/kXh BLZOPjmLsZhHqMgK1VZc6VWMK9hQ46OMntojK37hgyKMpYiNn1KLYR+z42XdooYVnvI3 ZfjqHqmO1FHJbgN9e5btt3TjNx60klfVhxt6WVVgvYKbZ395Pgpax2CcDMUbT8+Cwiay c2T7AMy4FftBa9wxPtvbC8dN2WR4nXgj8UCBMLZmdXq03e3iCEMgo4Jbl9aQ9VZuGE6O +qxw2baHZH2U3HO3g96zapkIOCKvfLQm2Sx4uVHxZFf5sVM22nxiKhiDlvhLrSkgouE1 RIxA== X-Gm-Message-State: APt69E1dQAk8a3z5N2oE1cVtAI4Tce9rOUE+6m2uQaVlAOM5TZc6NVrb fiGbmPZDwO6whxTyjp8kUm3E0w== X-Received: by 2002:a5d:4141:: with SMTP id c1-v6mr8769534wrq.129.1530200722754; Thu, 28 Jun 2018 08:45:22 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:22 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 01/11] sched/pelt: Move pelt related code in a dedicated file Date: Thu, 28 Jun 2018 17:45:04 +0200 Message-Id: <1530200714-4504-2-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We want to track rt_rq's utilization as a part of the estimation of the whole rq's utilization. This is necessary because rt tasks can steal utilization to cfs tasks and make them lighter than they are. As we want to use the same load tracking mecanism for both and prevent useless dependency between cfs and rt code, pelt code is moved in a dedicated file. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/Makefile | 2 +- kernel/sched/fair.c | 333 +------------------------------------------------- kernel/sched/pelt.c | 311 ++++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/pelt.h | 43 +++++++ kernel/sched/sched.h | 19 +++ 5 files changed, 375 insertions(+), 333 deletions(-) create mode 100644 kernel/sched/pelt.c create mode 100644 kernel/sched/pelt.h -- 2.7.4 diff --git a/kernel/sched/Makefile b/kernel/sched/Makefile index d9a02b3..7fe1834 100644 --- a/kernel/sched/Makefile +++ b/kernel/sched/Makefile @@ -20,7 +20,7 @@ obj-y += core.o loadavg.o clock.o cputime.o obj-y += idle.o fair.o rt.o deadline.o obj-y += wait.o wait_bit.o swait.o completion.o -obj-$(CONFIG_SMP) += cpupri.o cpudeadline.o topology.o stop_task.o +obj-$(CONFIG_SMP) += cpupri.o cpudeadline.o topology.o stop_task.o pelt.o obj-$(CONFIG_SCHED_AUTOGROUP) += autogroup.o obj-$(CONFIG_SCHEDSTATS) += stats.o obj-$(CONFIG_SCHED_DEBUG) += debug.o diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 4cc1441..bdab9ed 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -255,9 +255,6 @@ static inline struct rq *rq_of(struct cfs_rq *cfs_rq) return cfs_rq->rq; } -/* An entity is a task if it doesn't "own" a runqueue */ -#define entity_is_task(se) (!se->my_q) - static inline struct task_struct *task_of(struct sched_entity *se) { SCHED_WARN_ON(!entity_is_task(se)); @@ -419,7 +416,6 @@ static inline struct rq *rq_of(struct cfs_rq *cfs_rq) return container_of(cfs_rq, struct rq, cfs); } -#define entity_is_task(se) 1 #define for_each_sched_entity(se) \ for (; se; se = NULL) @@ -692,7 +688,7 @@ static u64 sched_vslice(struct cfs_rq *cfs_rq, struct sched_entity *se) } #ifdef CONFIG_SMP - +#include "pelt.h" #include "sched-pelt.h" static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu); @@ -2749,19 +2745,6 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) } while (0) #ifdef CONFIG_SMP -/* - * XXX we want to get rid of these helpers and use the full load resolution. - */ -static inline long se_weight(struct sched_entity *se) -{ - return scale_load_down(se->load.weight); -} - -static inline long se_runnable(struct sched_entity *se) -{ - return scale_load_down(se->runnable_weight); -} - static inline void enqueue_runnable_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) { @@ -3062,314 +3045,6 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq, int flags) } #ifdef CONFIG_SMP -/* - * Approximate: - * val * y^n, where y^32 ~= 0.5 (~1 scheduling period) - */ -static u64 decay_load(u64 val, u64 n) -{ - unsigned int local_n; - - if (unlikely(n > LOAD_AVG_PERIOD * 63)) - return 0; - - /* after bounds checking we can collapse to 32-bit */ - local_n = n; - - /* - * As y^PERIOD = 1/2, we can combine - * y^n = 1/2^(n/PERIOD) * y^(n%PERIOD) - * With a look-up table which covers y^n (n= LOAD_AVG_PERIOD)) { - val >>= local_n / LOAD_AVG_PERIOD; - local_n %= LOAD_AVG_PERIOD; - } - - val = mul_u64_u32_shr(val, runnable_avg_yN_inv[local_n], 32); - return val; -} - -static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3) -{ - u32 c1, c2, c3 = d3; /* y^0 == 1 */ - - /* - * c1 = d1 y^p - */ - c1 = decay_load((u64)d1, periods); - - /* - * p-1 - * c2 = 1024 \Sum y^n - * n=1 - * - * inf inf - * = 1024 ( \Sum y^n - \Sum y^n - y^0 ) - * n=0 n=p - */ - c2 = LOAD_AVG_MAX - decay_load(LOAD_AVG_MAX, periods) - 1024; - - return c1 + c2 + c3; -} - -/* - * Accumulate the three separate parts of the sum; d1 the remainder - * of the last (incomplete) period, d2 the span of full periods and d3 - * the remainder of the (incomplete) current period. - * - * d1 d2 d3 - * ^ ^ ^ - * | | | - * |<->|<----------------->|<--->| - * ... |---x---|------| ... |------|-----x (now) - * - * p-1 - * u' = (u + d1) y^p + 1024 \Sum y^n + d3 y^0 - * n=1 - * - * = u y^p + (Step 1) - * - * p-1 - * d1 y^p + 1024 \Sum y^n + d3 y^0 (Step 2) - * n=1 - */ -static __always_inline u32 -accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, - unsigned long load, unsigned long runnable, int running) -{ - unsigned long scale_freq, scale_cpu; - u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */ - u64 periods; - - scale_freq = arch_scale_freq_capacity(cpu); - scale_cpu = arch_scale_cpu_capacity(NULL, cpu); - - delta += sa->period_contrib; - periods = delta / 1024; /* A period is 1024us (~1ms) */ - - /* - * Step 1: decay old *_sum if we crossed period boundaries. - */ - if (periods) { - sa->load_sum = decay_load(sa->load_sum, periods); - sa->runnable_load_sum = - decay_load(sa->runnable_load_sum, periods); - sa->util_sum = decay_load((u64)(sa->util_sum), periods); - - /* - * Step 2 - */ - delta %= 1024; - contrib = __accumulate_pelt_segments(periods, - 1024 - sa->period_contrib, delta); - } - sa->period_contrib = delta; - - contrib = cap_scale(contrib, scale_freq); - if (load) - sa->load_sum += load * contrib; - if (runnable) - sa->runnable_load_sum += runnable * contrib; - if (running) - sa->util_sum += contrib * scale_cpu; - - return periods; -} - -/* - * We can represent the historical contribution to runnable average as the - * coefficients of a geometric series. To do this we sub-divide our runnable - * history into segments of approximately 1ms (1024us); label the segment that - * occurred N-ms ago p_N, with p_0 corresponding to the current period, e.g. - * - * [<- 1024us ->|<- 1024us ->|<- 1024us ->| ... - * p0 p1 p2 - * (now) (~1ms ago) (~2ms ago) - * - * Let u_i denote the fraction of p_i that the entity was runnable. - * - * We then designate the fractions u_i as our co-efficients, yielding the - * following representation of historical load: - * u_0 + u_1*y + u_2*y^2 + u_3*y^3 + ... - * - * We choose y based on the with of a reasonably scheduling period, fixing: - * y^32 = 0.5 - * - * This means that the contribution to load ~32ms ago (u_32) will be weighted - * approximately half as much as the contribution to load within the last ms - * (u_0). - * - * When a period "rolls over" and we have new u_0`, multiplying the previous - * sum again by y is sufficient to update: - * load_avg = u_0` + y*(u_0 + u_1*y + u_2*y^2 + ... ) - * = u_0 + u_1*y + u_2*y^2 + ... [re-labeling u_i --> u_{i+1}] - */ -static __always_inline int -___update_load_sum(u64 now, int cpu, struct sched_avg *sa, - unsigned long load, unsigned long runnable, int running) -{ - u64 delta; - - delta = now - sa->last_update_time; - /* - * This should only happen when time goes backwards, which it - * unfortunately does during sched clock init when we swap over to TSC. - */ - if ((s64)delta < 0) { - sa->last_update_time = now; - return 0; - } - - /* - * Use 1024ns as the unit of measurement since it's a reasonable - * approximation of 1us and fast to compute. - */ - delta >>= 10; - if (!delta) - return 0; - - sa->last_update_time += delta << 10; - - /* - * running is a subset of runnable (weight) so running can't be set if - * runnable is clear. But there are some corner cases where the current - * se has been already dequeued but cfs_rq->curr still points to it. - * This means that weight will be 0 but not running for a sched_entity - * but also for a cfs_rq if the latter becomes idle. As an example, - * this happens during idle_balance() which calls - * update_blocked_averages() - */ - if (!load) - runnable = running = 0; - - /* - * Now we know we crossed measurement unit boundaries. The *_avg - * accrues by two steps: - * - * Step 1: accumulate *_sum since last_update_time. If we haven't - * crossed period boundaries, finish. - */ - if (!accumulate_sum(delta, cpu, sa, load, runnable, running)) - return 0; - - return 1; -} - -static __always_inline void -___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runnable) -{ - u32 divider = LOAD_AVG_MAX - 1024 + sa->period_contrib; - - /* - * Step 2: update *_avg. - */ - sa->load_avg = div_u64(load * sa->load_sum, divider); - sa->runnable_load_avg = div_u64(runnable * sa->runnable_load_sum, divider); - sa->util_avg = sa->util_sum / divider; -} - -/* - * When a task is dequeued, its estimated utilization should not be update if - * its util_avg has not been updated at least once. - * This flag is used to synchronize util_avg updates with util_est updates. - * We map this information into the LSB bit of the utilization saved at - * dequeue time (i.e. util_est.dequeued). - */ -#define UTIL_AVG_UNCHANGED 0x1 - -static inline void cfs_se_util_change(struct sched_avg *avg) -{ - unsigned int enqueued; - - if (!sched_feat(UTIL_EST)) - return; - - /* Avoid store if the flag has been already set */ - enqueued = avg->util_est.enqueued; - if (!(enqueued & UTIL_AVG_UNCHANGED)) - return; - - /* Reset flag to report util_avg has been updated */ - enqueued &= ~UTIL_AVG_UNCHANGED; - WRITE_ONCE(avg->util_est.enqueued, enqueued); -} - -/* - * sched_entity: - * - * task: - * se_runnable() == se_weight() - * - * group: [ see update_cfs_group() ] - * se_weight() = tg->weight * grq->load_avg / tg->load_avg - * se_runnable() = se_weight(se) * grq->runnable_load_avg / grq->load_avg - * - * load_sum := runnable_sum - * load_avg = se_weight(se) * runnable_avg - * - * runnable_load_sum := runnable_sum - * runnable_load_avg = se_runnable(se) * runnable_avg - * - * XXX collapse load_sum and runnable_load_sum - * - * cfq_rs: - * - * load_sum = \Sum se_weight(se) * se->avg.load_sum - * load_avg = \Sum se->avg.load_avg - * - * runnable_load_sum = \Sum se_runnable(se) * se->avg.runnable_load_sum - * runnable_load_avg = \Sum se->avg.runable_load_avg - */ - -static int -__update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se) -{ - if (entity_is_task(se)) - se->runnable_weight = se->load.weight; - - if (___update_load_sum(now, cpu, &se->avg, 0, 0, 0)) { - ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); - return 1; - } - - return 0; -} - -static int -__update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se) -{ - if (entity_is_task(se)) - se->runnable_weight = se->load.weight; - - if (___update_load_sum(now, cpu, &se->avg, !!se->on_rq, !!se->on_rq, - cfs_rq->curr == se)) { - - ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); - cfs_se_util_change(&se->avg); - return 1; - } - - return 0; -} - -static int -__update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) -{ - if (___update_load_sum(now, cpu, &cfs_rq->avg, - scale_load_down(cfs_rq->load.weight), - scale_load_down(cfs_rq->runnable_weight), - cfs_rq->curr != NULL)) { - - ___update_load_avg(&cfs_rq->avg, 1, 1); - return 1; - } - - return 0; -} - #ifdef CONFIG_FAIR_GROUP_SCHED /** * update_tg_load_avg - update the tg's load avg @@ -4045,12 +3720,6 @@ util_est_dequeue(struct cfs_rq *cfs_rq, struct task_struct *p, bool task_sleep) #else /* CONFIG_SMP */ -static inline int -update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq) -{ - return 0; -} - #define UPDATE_TG 0x0 #define SKIP_AGE_LOAD 0x0 #define DO_ATTACH 0x0 diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c new file mode 100644 index 0000000..e6ecbb2 --- /dev/null +++ b/kernel/sched/pelt.c @@ -0,0 +1,311 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Per Entity Load Tracking + * + * Copyright (C) 2007 Red Hat, Inc., Ingo Molnar + * + * Interactivity improvements by Mike Galbraith + * (C) 2007 Mike Galbraith + * + * Various enhancements by Dmitry Adamushko. + * (C) 2007 Dmitry Adamushko + * + * Group scheduling enhancements by Srivatsa Vaddagiri + * Copyright IBM Corporation, 2007 + * Author: Srivatsa Vaddagiri + * + * Scaled math optimizations by Thomas Gleixner + * Copyright (C) 2007, Thomas Gleixner + * + * Adaptive scheduling granularity, math enhancements by Peter Zijlstra + * Copyright (C) 2007 Red Hat, Inc., Peter Zijlstra + * + * Move PELT related code from fair.c into this pelt.c file + * Author: Vincent Guittot + */ + +#include +#include "sched.h" +#include "sched-pelt.h" +#include "pelt.h" + +/* + * Approximate: + * val * y^n, where y^32 ~= 0.5 (~1 scheduling period) + */ +static u64 decay_load(u64 val, u64 n) +{ + unsigned int local_n; + + if (unlikely(n > LOAD_AVG_PERIOD * 63)) + return 0; + + /* after bounds checking we can collapse to 32-bit */ + local_n = n; + + /* + * As y^PERIOD = 1/2, we can combine + * y^n = 1/2^(n/PERIOD) * y^(n%PERIOD) + * With a look-up table which covers y^n (n= LOAD_AVG_PERIOD)) { + val >>= local_n / LOAD_AVG_PERIOD; + local_n %= LOAD_AVG_PERIOD; + } + + val = mul_u64_u32_shr(val, runnable_avg_yN_inv[local_n], 32); + return val; +} + +static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3) +{ + u32 c1, c2, c3 = d3; /* y^0 == 1 */ + + /* + * c1 = d1 y^p + */ + c1 = decay_load((u64)d1, periods); + + /* + * p-1 + * c2 = 1024 \Sum y^n + * n=1 + * + * inf inf + * = 1024 ( \Sum y^n - \Sum y^n - y^0 ) + * n=0 n=p + */ + c2 = LOAD_AVG_MAX - decay_load(LOAD_AVG_MAX, periods) - 1024; + + return c1 + c2 + c3; +} + +#define cap_scale(v, s) ((v)*(s) >> SCHED_CAPACITY_SHIFT) + +/* + * Accumulate the three separate parts of the sum; d1 the remainder + * of the last (incomplete) period, d2 the span of full periods and d3 + * the remainder of the (incomplete) current period. + * + * d1 d2 d3 + * ^ ^ ^ + * | | | + * |<->|<----------------->|<--->| + * ... |---x---|------| ... |------|-----x (now) + * + * p-1 + * u' = (u + d1) y^p + 1024 \Sum y^n + d3 y^0 + * n=1 + * + * = u y^p + (Step 1) + * + * p-1 + * d1 y^p + 1024 \Sum y^n + d3 y^0 (Step 2) + * n=1 + */ +static __always_inline u32 +accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, + unsigned long load, unsigned long runnable, int running) +{ + unsigned long scale_freq, scale_cpu; + u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */ + u64 periods; + + scale_freq = arch_scale_freq_capacity(cpu); + scale_cpu = arch_scale_cpu_capacity(NULL, cpu); + + delta += sa->period_contrib; + periods = delta / 1024; /* A period is 1024us (~1ms) */ + + /* + * Step 1: decay old *_sum if we crossed period boundaries. + */ + if (periods) { + sa->load_sum = decay_load(sa->load_sum, periods); + sa->runnable_load_sum = + decay_load(sa->runnable_load_sum, periods); + sa->util_sum = decay_load((u64)(sa->util_sum), periods); + + /* + * Step 2 + */ + delta %= 1024; + contrib = __accumulate_pelt_segments(periods, + 1024 - sa->period_contrib, delta); + } + sa->period_contrib = delta; + + contrib = cap_scale(contrib, scale_freq); + if (load) + sa->load_sum += load * contrib; + if (runnable) + sa->runnable_load_sum += runnable * contrib; + if (running) + sa->util_sum += contrib * scale_cpu; + + return periods; +} + +/* + * We can represent the historical contribution to runnable average as the + * coefficients of a geometric series. To do this we sub-divide our runnable + * history into segments of approximately 1ms (1024us); label the segment that + * occurred N-ms ago p_N, with p_0 corresponding to the current period, e.g. + * + * [<- 1024us ->|<- 1024us ->|<- 1024us ->| ... + * p0 p1 p2 + * (now) (~1ms ago) (~2ms ago) + * + * Let u_i denote the fraction of p_i that the entity was runnable. + * + * We then designate the fractions u_i as our co-efficients, yielding the + * following representation of historical load: + * u_0 + u_1*y + u_2*y^2 + u_3*y^3 + ... + * + * We choose y based on the with of a reasonably scheduling period, fixing: + * y^32 = 0.5 + * + * This means that the contribution to load ~32ms ago (u_32) will be weighted + * approximately half as much as the contribution to load within the last ms + * (u_0). + * + * When a period "rolls over" and we have new u_0`, multiplying the previous + * sum again by y is sufficient to update: + * load_avg = u_0` + y*(u_0 + u_1*y + u_2*y^2 + ... ) + * = u_0 + u_1*y + u_2*y^2 + ... [re-labeling u_i --> u_{i+1}] + */ +static __always_inline int +___update_load_sum(u64 now, int cpu, struct sched_avg *sa, + unsigned long load, unsigned long runnable, int running) +{ + u64 delta; + + delta = now - sa->last_update_time; + /* + * This should only happen when time goes backwards, which it + * unfortunately does during sched clock init when we swap over to TSC. + */ + if ((s64)delta < 0) { + sa->last_update_time = now; + return 0; + } + + /* + * Use 1024ns as the unit of measurement since it's a reasonable + * approximation of 1us and fast to compute. + */ + delta >>= 10; + if (!delta) + return 0; + + sa->last_update_time += delta << 10; + + /* + * running is a subset of runnable (weight) so running can't be set if + * runnable is clear. But there are some corner cases where the current + * se has been already dequeued but cfs_rq->curr still points to it. + * This means that weight will be 0 but not running for a sched_entity + * but also for a cfs_rq if the latter becomes idle. As an example, + * this happens during idle_balance() which calls + * update_blocked_averages() + */ + if (!load) + runnable = running = 0; + + /* + * Now we know we crossed measurement unit boundaries. The *_avg + * accrues by two steps: + * + * Step 1: accumulate *_sum since last_update_time. If we haven't + * crossed period boundaries, finish. + */ + if (!accumulate_sum(delta, cpu, sa, load, runnable, running)) + return 0; + + return 1; +} + +static __always_inline void +___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runnable) +{ + u32 divider = LOAD_AVG_MAX - 1024 + sa->period_contrib; + + /* + * Step 2: update *_avg. + */ + sa->load_avg = div_u64(load * sa->load_sum, divider); + sa->runnable_load_avg = div_u64(runnable * sa->runnable_load_sum, divider); + sa->util_avg = sa->util_sum / divider; +} + +/* + * sched_entity: + * + * task: + * se_runnable() == se_weight() + * + * group: [ see update_cfs_group() ] + * se_weight() = tg->weight * grq->load_avg / tg->load_avg + * se_runnable() = se_weight(se) * grq->runnable_load_avg / grq->load_avg + * + * load_sum := runnable_sum + * load_avg = se_weight(se) * runnable_avg + * + * runnable_load_sum := runnable_sum + * runnable_load_avg = se_runnable(se) * runnable_avg + * + * XXX collapse load_sum and runnable_load_sum + * + * cfq_rq: + * + * load_sum = \Sum se_weight(se) * se->avg.load_sum + * load_avg = \Sum se->avg.load_avg + * + * runnable_load_sum = \Sum se_runnable(se) * se->avg.runnable_load_sum + * runnable_load_avg = \Sum se->avg.runable_load_avg + */ + +int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se) +{ + if (entity_is_task(se)) + se->runnable_weight = se->load.weight; + + if (___update_load_sum(now, cpu, &se->avg, 0, 0, 0)) { + ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); + return 1; + } + + return 0; +} + +int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + if (entity_is_task(se)) + se->runnable_weight = se->load.weight; + + if (___update_load_sum(now, cpu, &se->avg, !!se->on_rq, !!se->on_rq, + cfs_rq->curr == se)) { + + ___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); + cfs_se_util_change(&se->avg); + return 1; + } + + return 0; +} + +int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) +{ + if (___update_load_sum(now, cpu, &cfs_rq->avg, + scale_load_down(cfs_rq->load.weight), + scale_load_down(cfs_rq->runnable_weight), + cfs_rq->curr != NULL)) { + + ___update_load_avg(&cfs_rq->avg, 1, 1); + return 1; + } + + return 0; +} diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h new file mode 100644 index 0000000..9cac73e --- /dev/null +++ b/kernel/sched/pelt.h @@ -0,0 +1,43 @@ +#ifdef CONFIG_SMP + +int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se); +int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se); +int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); + +/* + * When a task is dequeued, its estimated utilization should not be update if + * its util_avg has not been updated at least once. + * This flag is used to synchronize util_avg updates with util_est updates. + * We map this information into the LSB bit of the utilization saved at + * dequeue time (i.e. util_est.dequeued). + */ +#define UTIL_AVG_UNCHANGED 0x1 + +static inline void cfs_se_util_change(struct sched_avg *avg) +{ + unsigned int enqueued; + + if (!sched_feat(UTIL_EST)) + return; + + /* Avoid store if the flag has been already set */ + enqueued = avg->util_est.enqueued; + if (!(enqueued & UTIL_AVG_UNCHANGED)) + return; + + /* Reset flag to report util_avg has been updated */ + enqueued &= ~UTIL_AVG_UNCHANGED; + WRITE_ONCE(avg->util_est.enqueued, enqueued); +} + +#else + +static inline int +update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq) +{ + return 0; +} + +#endif + + diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 6601baf..db6878e 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -666,7 +666,26 @@ struct dl_rq { u64 bw_ratio; }; +#ifdef CONFIG_FAIR_GROUP_SCHED +/* An entity is a task if it doesn't "own" a runqueue */ +#define entity_is_task(se) (!se->my_q) +#else +#define entity_is_task(se) 1 +#endif + #ifdef CONFIG_SMP +/* + * XXX we want to get rid of these helpers and use the full load resolution. + */ +static inline long se_weight(struct sched_entity *se) +{ + return scale_load_down(se->load.weight); +} + +static inline long se_runnable(struct sched_entity *se) +{ + return scale_load_down(se->runnable_weight); +} static inline bool sched_asym_prefer(int a, int b) { From patchwork Thu Jun 28 15:45:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140454 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2342928ljj; Thu, 28 Jun 2018 08:45:36 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKj1tqrp1ifFPJrExWpm5oONMtxpebxa1q12g5VdbUgNxBwoLclHLpgX0nAm/fC5TFduOXU X-Received: by 2002:a65:5348:: with SMTP id w8-v6mr9507730pgr.247.1530200736561; Thu, 28 Jun 2018 08:45:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200736; cv=none; d=google.com; s=arc-20160816; b=dfRixawgi7HQK7TCyxFFPtGqAW4xsZ9zRxySdp+BmFgJ0OXQXJLCViUcAXygW7/y28 QbwLnRkKwRTLMkypI3bRmaZHlrgpPWFNpbjVrOKKF7AbzdP90tz/MZXHomJvW07K68xP 7NA7JLkk8UwtvhYIJVVtqphFIZkJ6HEESnbqQ+RKL5gsm8cYbHPP04JKj5AGcRs383Hm BBeq5mP4Somu7ahN4zTO3C9X1Ully3VkRVRkZ1cn+KLeCA+5WrREjfS0l7cZizf1rOlI Uk2A8VcOT2frlu2u9DNBH8jwNr6xZXW2DasuQEpjQnU18B6wq8tPfJXEfYyWbkch2FYY wt6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=40iPUFhsx7XM6LB0p73GtKzFTWtCVEeSOv4I54ZfXIE=; b=jz+IMlOSxD670vSdlLSACP7kUG4+keuPS5B4HASb4pn7WMpnmG7VHaLT/3ZRFpMzYV aOnWh0HiXJXSodT4QYOWYjVizFS9XGFoWodvEU58R3c4iXJsgjLv4x7oRJCMBY6fAqmf VJ0K+I1UzLxhfOHFObn4DEXqdpdHiPRP0p8xG+ksOovzmUxypesPrdTuVpyPWOsWCSql ShR1LGQ9TQIPc+UYJ7eC0xfstxXKDJ/56IOWb20oXk7p8+llu1cTHIs3dYdUBQFIKyLu I6Oykdfpr4qQcdFvDSQpBM/QbdP4KJd8oTkm5Wl+iU5Vn/ERV++ytADtBl9jVxiz47ly Zt5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Gv2mNDwh; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x66-v6si7273659pfx.67.2018.06.28.08.45.36; Thu, 28 Jun 2018 08:45:36 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Gv2mNDwh; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1030213AbeF1Ppd (ORCPT + 31 others); Thu, 28 Jun 2018 11:45:33 -0400 Received: from mail-wr0-f195.google.com ([209.85.128.195]:37887 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932316AbeF1PpZ (ORCPT ); Thu, 28 Jun 2018 11:45:25 -0400 Received: by mail-wr0-f195.google.com with SMTP id k6-v6so6009549wrp.4 for ; Thu, 28 Jun 2018 08:45:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=40iPUFhsx7XM6LB0p73GtKzFTWtCVEeSOv4I54ZfXIE=; b=Gv2mNDwh3lU9Ld5fyu0O/sXrMTZBQDsNekOPwau0a6lFKKGSqzGeEsE01F5w8tNy5b 0JOwdISDQss/e/nM633nn7Od18kiWovlzy2YIrPUeMsfc5TnqaOYN+A8l6uTvlN/iLi0 G3flSo1y9vxrEgLo0m2Rdqe3vzLW4FVs2rDD8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=40iPUFhsx7XM6LB0p73GtKzFTWtCVEeSOv4I54ZfXIE=; b=HgBtL/sFznIq3ADbHksjJWp2la6Q4rxHWeEcD36CayFkHlLzAiCgFpF2TyJceya0aR gu0Dn4WfsGUuLKgc82EZPacHPNrnTGhAvkcbYCRnC4vtWOFEfSb/QXre0KIXAmeZHpqZ Lv0iYG+I0xhKElZd5q02WRCD0BpH7eC8aeX4/Q9BAn/aNuH2NpT0xe3z+ycFNoC2fCdn C5AOeOIgH31dL2dGr6dc5Qe+DKrUU9sRMpBaVNoqMiXbTivSvzXRF6eDMqRlk/zHPNvz JYf10HMbTG97q9uaCMEgveFEBt3o97AbP3gv+2G75St7qFrCeoaQ2bLvVXe3t+1C80r/ xo/A== X-Gm-Message-State: APt69E3mjfmdwv3j6+3208B8ze1+JVQKblcax1YF++dJL6yT1FK1S8g+ TiwqwUFRc0YCRGYAY/xqcHEKuw== X-Received: by 2002:adf:ab95:: with SMTP id s21-v6mr9617322wrc.90.1530200724460; Thu, 28 Jun 2018 08:45:24 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:23 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 02/11] sched/rt: add rt_rq utilization tracking Date: Thu, 28 Jun 2018 17:45:05 +0200 Message-Id: <1530200714-4504-3-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org schedutil governor relies on cfs_rq's util_avg to choose the OPP when cfs tasks are running. When the CPU is overloaded by cfs and rt tasks, cfs tasks are preempted by rt tasks and in this case util_avg reflects the remaining capacity but not what cfs want to use. In such case, schedutil can select a lower OPP whereas the CPU is overloaded. In order to have a more accurate view of the utilization of the CPU, we track the utilization of rt tasks. Only util_avg is correctly tracked but not load_avg and runnable_load_avg which are useless for rt_rq. rt_rq uses rq_clock_task and cfs_rq uses cfs_rq_clock_task but they are the same at the root group level, so the PELT windows of the util_sum are aligned. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 15 ++++++++++++++- kernel/sched/pelt.c | 25 +++++++++++++++++++++++++ kernel/sched/pelt.h | 7 +++++++ kernel/sched/rt.c | 13 +++++++++++++ kernel/sched/sched.h | 7 +++++++ 5 files changed, 66 insertions(+), 1 deletion(-) -- 2.7.4 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index bdab9ed..328bedc 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7289,6 +7289,14 @@ static inline bool cfs_rq_has_blocked(struct cfs_rq *cfs_rq) return false; } +static inline bool rt_rq_has_blocked(struct rq *rq) +{ + if (READ_ONCE(rq->avg_rt.util_avg)) + return true; + + return false; +} + #ifdef CONFIG_FAIR_GROUP_SCHED static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) @@ -7348,6 +7356,10 @@ static void update_blocked_averages(int cpu) if (cfs_rq_has_blocked(cfs_rq)) done = false; } + update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); + /* Don't need periodic decay once load/util_avg are null */ + if (rt_rq_has_blocked(rq)) + done = false; #ifdef CONFIG_NO_HZ_COMMON rq->last_blocked_load_update_tick = jiffies; @@ -7413,9 +7425,10 @@ static inline void update_blocked_averages(int cpu) rq_lock_irqsave(rq, &rf); update_rq_clock(rq); update_cfs_rq_load_avg(cfs_rq_clock_task(cfs_rq), cfs_rq); + update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); #ifdef CONFIG_NO_HZ_COMMON rq->last_blocked_load_update_tick = jiffies; - if (!cfs_rq_has_blocked(cfs_rq)) + if (!cfs_rq_has_blocked(cfs_rq) && !rt_rq_has_blocked(rq)) rq->has_blocked_load = 0; #endif rq_unlock_irqrestore(rq, &rf); diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index e6ecbb2..a00b1ba 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -309,3 +309,28 @@ int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) return 0; } + +/* + * rt_rq: + * + * util_sum = \Sum se->avg.util_sum but se->avg.util_sum is not tracked + * util_sum = cpu_scale * load_sum + * runnable_load_sum = load_sum + * + * load_avg and runnable_load_avg are not supported and meaningless. + * + */ + +int update_rt_rq_load_avg(u64 now, struct rq *rq, int running) +{ + if (___update_load_sum(now, rq->cpu, &rq->avg_rt, + running, + running, + running)) { + + ___update_load_avg(&rq->avg_rt, 1, 1); + return 1; + } + + return 0; +} diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h index 9cac73e..b2983b7 100644 --- a/kernel/sched/pelt.h +++ b/kernel/sched/pelt.h @@ -3,6 +3,7 @@ int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se); int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se); int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); +int update_rt_rq_load_avg(u64 now, struct rq *rq, int running); /* * When a task is dequeued, its estimated utilization should not be update if @@ -38,6 +39,12 @@ update_cfs_rq_load_avg(u64 now, struct cfs_rq *cfs_rq) return 0; } +static inline int +update_rt_rq_load_avg(u64 now, struct rq *rq, int running) +{ + return 0; +} + #endif diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 47556b0..0e3e57a 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -5,6 +5,8 @@ */ #include "sched.h" +#include "pelt.h" + int sched_rr_timeslice = RR_TIMESLICE; int sysctl_sched_rr_timeslice = (MSEC_PER_SEC / HZ) * RR_TIMESLICE; @@ -1572,6 +1574,14 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) rt_queue_push_tasks(rq); + /* + * If prev task was rt, put_prev_task() has already updated the + * utilization. We only care of the case where we start to schedule a + * rt task + */ + if (rq->curr->sched_class != &rt_sched_class) + update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); + return p; } @@ -1579,6 +1589,8 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p) { update_curr_rt(rq); + update_rt_rq_load_avg(rq_clock_task(rq), rq, 1); + /* * The previous task needs to be made eligible for pushing * if it is still active @@ -2308,6 +2320,7 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued) struct sched_rt_entity *rt_se = &p->rt; update_curr_rt(rq); + update_rt_rq_load_avg(rq_clock_task(rq), rq, 1); watchdog(rq, p); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index db6878e..f2b12b0 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -592,6 +592,7 @@ struct rt_rq { unsigned long rt_nr_total; int overloaded; struct plist_head pushable_tasks; + #endif /* CONFIG_SMP */ int rt_queued; @@ -847,6 +848,7 @@ struct rq { u64 rt_avg; u64 age_stamp; + struct sched_avg avg_rt; u64 idle_stamp; u64 avg_idle; @@ -2205,4 +2207,9 @@ static inline unsigned long cpu_util_cfs(struct rq *rq) return util; } + +static inline unsigned long cpu_util_rt(struct rq *rq) +{ + return rq->avg_rt.util_avg; +} #endif From patchwork Thu Jun 28 15:45:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140464 Delivered-To: patch@linaro.org Received: by 2002:a50:86eb:0:0:0:0:0 with SMTP id 40-v6csp1955924edu; Thu, 28 Jun 2018 08:48:07 -0700 (PDT) X-Google-Smtp-Source: AAOMgpcazmUw4Tg+BCnDoSVmi406J/qBBiB6uWhBpyOWaqOLCcpeZf+LrK8193xxDal9BgGZnDR6 X-Received: by 2002:aa7:87d0:: with SMTP id i16-v6mr6805668pfo.82.1530200887135; Thu, 28 Jun 2018 08:48:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200887; cv=none; d=google.com; s=arc-20160816; b=Jl8ckn8oHi4nBPhMb66hnORJPVQNvMGblWbgwpKHjPpGmQRm+YYBIuMszKGn1QC5hr 5CIDe9UOnWSYXx2T4/mxBU4Fb4uvtMTax6k2ziyJIps4XItTnUtpmZw1ciJj+L4JWlmd 5GdEwzxvo33wpkFbOquH4LKIHRD/LF68T31tE48fj/QtPgOHRvtX8blBZpTKbUcU+zof 5nly4SwCi7visAiyDmhyylasm2f1oCYZbddCPgdBlQ1Pec06QOyaJ3gIoBPGRXIE3HWL 0QEy+2t6lU9Y4zH/2AH+14uEPnn3U+O63bUw7hI9ZFBh1aDhwX2/rK14+6Gd0HFGb/k4 LpFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=OcmgxRiLyrkswXgHU3qseRb+IEdrpHSwiSXvn0FFVBQ=; b=rZzQHyOhuKSECjNYo6UMruv6j8vH3jOfOCWRNJALB16UvlBhKLfcNLiKBciDya/CzG OtwG76KY+F6WLlG7uw9arX5a9cb6l9yagYfJ5/L1Y5Ij+aseZxVssKSuRgkGdSge3EE0 yJhYmSwVUaurJncY3lvvAlh4UIdxiksLShXLlnFiXCKgfAxGsY0XDinNnYGXzCbXcSSF CHYShU/1D5Uz9T5f/9jQkxWo3C1eo64M+5AvKbXcex8PzG02kjHv1lRu5CH4HrjlbO9n 3Kn7lz4vQsG5IZZgw9wJ/huiVr64Ro9ZuFIvEdie8weGqSSeBrWNch026GFIrfkuER/D Aw9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D5FL72Lu; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d8-v6si5915471pgq.162.2018.06.28.08.48.06; Thu, 28 Jun 2018 08:48:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D5FL72Lu; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934870AbeF1PsE (ORCPT + 31 others); Thu, 28 Jun 2018 11:48:04 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:54121 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S966977AbeF1Pp1 (ORCPT ); Thu, 28 Jun 2018 11:45:27 -0400 Received: by mail-wm0-f66.google.com with SMTP id b188-v6so4123217wme.3 for ; Thu, 28 Jun 2018 08:45:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=OcmgxRiLyrkswXgHU3qseRb+IEdrpHSwiSXvn0FFVBQ=; b=D5FL72LukmNDASiDvkVM2BNLpT29ZErQuBdjkjtpiDANecf6RKnvSj/55e8trsql2d YC3xB3Nq8THBfHcYpugL8md7xu6lx4TGunH2EVc1UGj8IoyyPfMgX52sudu9MnoUU9xX ONhWRPtn8lTE7405oNyv+KBaPKygsizeOKcEk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=OcmgxRiLyrkswXgHU3qseRb+IEdrpHSwiSXvn0FFVBQ=; b=PsRYhlGfWy3bo+eiwq6eogCMBzWqWlE1Nqm0upxLdFQmbthZoTdYaZRqlTyBBnDzzs EIRrVX5PNkbIq8IH7+4hbBEl6Qw9PGk5aTOkmSCVUOKPzTSAWCJ/8poDthUfyAmqy3tG 3R6d0YfRRJq2cWp7nGh0+vErlJLsK7wlj0xDm0jGX78t0EGRSK7x6dwcsz1PchkSJBpl FWjf9xjBv3/6tzGum/azqaq4gzRt84fPQReWAJjKkWYO2MAyPqmtWWyMK7ZAWlrwvC4q Y12XAD97bWvlOfk7bbl5DclZ4MpnOLFN95zGIZAAZk4EsJyixN3x/kxZFgPNaSFKhbb7 P2Qw== X-Gm-Message-State: APt69E1utYG2GC8x2JfXPV//e/qe3ErLEjTFoFllwvkAiUPOOD+ZPH0d 0YN4CL5+GwJA9byoy6x16X+31g== X-Received: by 2002:a1c:818e:: with SMTP id c136-v6mr8740170wmd.107.1530200726039; Thu, 28 Jun 2018 08:45:26 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:25 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 03/11] cpufreq/schedutil: use rt utilization tracking Date: Thu, 28 Jun 2018 17:45:06 +0200 Message-Id: <1530200714-4504-4-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add both cfs and rt utilization when selecting an OPP for cfs tasks as rt can preempt and steal cfs's running time. rt util_avg is used to take into account the utilization of rt tasks on the CPU when selecting OPP. If a rt task migrate, the rt utilization will not migrate but will decay over time. On an overloaded CPU, cfs utilization reflects the remaining utilization avialable on CPU. When rt task migrates, the cfs utilization will increase when tasks will start to use the newly available capacity. At the same pace, rt utilization will decay and both variations will compensate each other to keep unchanged overall utilization and will prevent any OPP drop. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/cpufreq_schedutil.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) -- 2.7.4 Acked-by: Viresh Kumar diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 3cde464..9c5e92e 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -56,6 +56,7 @@ struct sugov_cpu { /* The fields below are only needed when sharing a policy: */ unsigned long util_cfs; unsigned long util_dl; + unsigned long util_rt; unsigned long max; /* The field below is for single-CPU policies only: */ @@ -186,15 +187,21 @@ static void sugov_get_util(struct sugov_cpu *sg_cpu) sg_cpu->max = arch_scale_cpu_capacity(NULL, sg_cpu->cpu); sg_cpu->util_cfs = cpu_util_cfs(rq); sg_cpu->util_dl = cpu_util_dl(rq); + sg_cpu->util_rt = cpu_util_rt(rq); } static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) { struct rq *rq = cpu_rq(sg_cpu->cpu); + unsigned long util; if (rq->rt.rt_nr_running) return sg_cpu->max; + util = sg_cpu->util_dl; + util += sg_cpu->util_cfs; + util += sg_cpu->util_rt; + /* * Utilization required by DEADLINE must always be granted while, for * FAIR, we use blocked utilization of IDLE CPUs as a mechanism to @@ -205,7 +212,7 @@ static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) * util_cfs + util_dl as requested freq. However, cpufreq is not yet * ready for such an interface. So, we only do the latter for now. */ - return min(sg_cpu->max, (sg_cpu->util_dl + sg_cpu->util_cfs)); + return min(sg_cpu->max, util); } /** From patchwork Thu Jun 28 15:45:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140463 Delivered-To: patch@linaro.org Received: by 2002:a50:86eb:0:0:0:0:0 with SMTP id 40-v6csp1955506edu; Thu, 28 Jun 2018 08:47:37 -0700 (PDT) X-Google-Smtp-Source: ADUXVKLeFfaw2uWqYv8i1k8DjGwaGCToYEPw3V1eA/Z87yQQuqbxjS/QCxdkmUWt3HbFevVpYulH X-Received: by 2002:a17:902:8d98:: with SMTP id v24-v6mr11088338plo.250.1530200857326; Thu, 28 Jun 2018 08:47:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200857; cv=none; d=google.com; s=arc-20160816; b=jAedg3R1wHRCLxBLFdjoiam5bkcyiyPS0VZ6a4aqvt6B0nStUwHA+6CUN2hYqh2pBo j38TZclh8LambFLe3AQ3JhG8btDVX7Pe0MI4GxDROsViB3nvGJxtXyjlVUMOPalR+RIT IX11LJ/aSCOk88seCw7Ukh4VDq6U1KLV3zp9eIqB8aja246zJS8cCFL3RrL+69MVDr4z 7KrmRgNOjXAf6+xmuToJgmuy9plRox0adS4EN4U7Pj6H+BLkN7zKjMcMmLYGLdhpr9d7 yt2ARNdslA7rhV8LhOTQZPjw64M1cNs6DNZY9N15sWvtBRylIWKvCPzDYyN0UVAuTFUi 4aqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=IhPXEvdETPjS/XrwEpztFR59irmHT4gle9wfp3C4/H8=; b=T9VvD2el0pl8mwJ0zx4087viS3geioTiUR8mqylZjw4y5FL5Y0DjYM5/kies2RGzU6 6aMGGhYL8NIuFh8ZB84+JIfSTPPgXAO99/GPIBZMH1TrPsWaEFsEZhSIskMzMgbqrm6+ qOH5R6C7vjWIjjM9xqBBK3PoZQdt+ZvXOKFbjecPIlCib7Ir146S2ftrirLMYYHn+T6C cgqN2M+iWvyf9bfpBys/2pQ4+jWrOpSVZCaYkkR0In2lsy0glRAlI+/6Kj+wa5OPIysO zQKtbCfBERJu3gGIIZ4TndLNC8iqhROxrC4Pm6bK4Pe7Eb7jo1wGJocf02xjTkS6S5j3 P+fQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jMylM3Sm; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t6-v6si7059412plo.508.2018.06.28.08.47.36; Thu, 28 Jun 2018 08:47:37 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=jMylM3Sm; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934735AbeF1Prf (ORCPT + 31 others); Thu, 28 Jun 2018 11:47:35 -0400 Received: from mail-wr0-f195.google.com ([209.85.128.195]:40292 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S967006AbeF1Pp2 (ORCPT ); Thu, 28 Jun 2018 11:45:28 -0400 Received: by mail-wr0-f195.google.com with SMTP id l2-v6so5423754wro.7 for ; Thu, 28 Jun 2018 08:45:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IhPXEvdETPjS/XrwEpztFR59irmHT4gle9wfp3C4/H8=; b=jMylM3SmtzQYOB7WHvsl0+vWSB4Mk2WBbrl8EnV4gN9vfaAemPus3Bob0eTO9+EJ3r wiz1URLF2nvxZk0dUpsxjR7pogsFXOsVfwaWSo5+rRVTTBWboqTYfdCuj61zbda+Z/+N oEnYgkRhIwuwgA9pAQ7FJGSVYTGmKYDs/Y3EA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IhPXEvdETPjS/XrwEpztFR59irmHT4gle9wfp3C4/H8=; b=NdNUFa4XrzTWwlBUaG1XJT3qnMo3UPOXSQuILMnNSzt85RT5hMD2+5Nf0Asyqa1XOr AH3acbh/H2zPc0VhV6Z8KQ+gyia2Q4rPVz1wlJGatflo2SF3a1UfWe8xjUcVXH+mx8Wm 8bubVBP7LGxr8andba+WMPE0zQFfzncF9C1GScUJ32WN+1JkMSd64G9mUoqWp9VEYtKM 5iMO793y8oRUmv0z7cf7T1qu+JGYfX7xFNzO4LB89qAcWsaByaoLFRiiUHd0HUSywVab tDt1PiHvGds7XtiXFibQYrq0/Yd9LXI3w8/1xbp7TKNZAGFETemGEE+4V6G7DhpIhAN+ uP6g== X-Gm-Message-State: APt69E2AsQAShLkMxehnNaUYNDWOswWwcmCu8k+8EYbMEV1xzy2QP7PY Z2ahwjlP2YkZvsHNv/N4RNQsVw== X-Received: by 2002:adf:b2f4:: with SMTP id g107-v6mr9579165wrd.53.1530200727293; Thu, 28 Jun 2018 08:45:27 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:26 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 04/11] sched/dl: add dl_rq utilization tracking Date: Thu, 28 Jun 2018 17:45:07 +0200 Message-Id: <1530200714-4504-5-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Similarly to what happens with rt tasks, cfs tasks can be preempted by dl tasks and the cfs's utilization might no longer describes the real utilization level. Current dl bandwidth reflects the requirements to meet deadline when tasks are enqueued but not the current utilization of the dl sched class. We track dl class utilization to estimate the system utilization. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/deadline.c | 6 ++++++ kernel/sched/fair.c | 11 ++++++++--- kernel/sched/pelt.c | 23 +++++++++++++++++++++++ kernel/sched/pelt.h | 6 ++++++ kernel/sched/sched.h | 1 + 5 files changed, 44 insertions(+), 3 deletions(-) -- 2.7.4 diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index fbfc3f1..f4de2698 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -16,6 +16,7 @@ * Fabio Checconi */ #include "sched.h" +#include "pelt.h" struct dl_bandwidth def_dl_bandwidth; @@ -1761,6 +1762,9 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) deadline_queue_push_tasks(rq); + if (rq->curr->sched_class != &dl_sched_class) + update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); + return p; } @@ -1768,6 +1772,7 @@ static void put_prev_task_dl(struct rq *rq, struct task_struct *p) { update_curr_dl(rq); + update_dl_rq_load_avg(rq_clock_task(rq), rq, 1); if (on_dl_rq(&p->dl) && p->nr_cpus_allowed > 1) enqueue_pushable_dl_task(rq, p); } @@ -1784,6 +1789,7 @@ static void task_tick_dl(struct rq *rq, struct task_struct *p, int queued) { update_curr_dl(rq); + update_dl_rq_load_avg(rq_clock_task(rq), rq, 1); /* * Even when we have runtime, update_curr_dl() might have resulted in us * not being the leftmost task anymore. In that case NEED_RESCHED will diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 328bedc..ffce4b2 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7289,11 +7289,14 @@ static inline bool cfs_rq_has_blocked(struct cfs_rq *cfs_rq) return false; } -static inline bool rt_rq_has_blocked(struct rq *rq) +static inline bool others_rqs_have_blocked(struct rq *rq) { if (READ_ONCE(rq->avg_rt.util_avg)) return true; + if (READ_ONCE(rq->avg_dl.util_avg)) + return true; + return false; } @@ -7357,8 +7360,9 @@ static void update_blocked_averages(int cpu) done = false; } update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); + update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); /* Don't need periodic decay once load/util_avg are null */ - if (rt_rq_has_blocked(rq)) + if (others_rqs_have_blocked(rq)) done = false; #ifdef CONFIG_NO_HZ_COMMON @@ -7426,9 +7430,10 @@ static inline void update_blocked_averages(int cpu) update_rq_clock(rq); update_cfs_rq_load_avg(cfs_rq_clock_task(cfs_rq), cfs_rq); update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); + update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); #ifdef CONFIG_NO_HZ_COMMON rq->last_blocked_load_update_tick = jiffies; - if (!cfs_rq_has_blocked(cfs_rq) && !rt_rq_has_blocked(rq)) + if (!cfs_rq_has_blocked(cfs_rq) && !others_rqs_have_blocked(rq)) rq->has_blocked_load = 0; #endif rq_unlock_irqrestore(rq, &rf); diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index a00b1ba..8b78b63 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -334,3 +334,26 @@ int update_rt_rq_load_avg(u64 now, struct rq *rq, int running) return 0; } + +/* + * dl_rq: + * + * util_sum = \Sum se->avg.util_sum but se->avg.util_sum is not tracked + * util_sum = cpu_scale * load_sum + * runnable_load_sum = load_sum + * + */ + +int update_dl_rq_load_avg(u64 now, struct rq *rq, int running) +{ + if (___update_load_sum(now, rq->cpu, &rq->avg_dl, + running, + running, + running)) { + + ___update_load_avg(&rq->avg_dl, 1, 1); + return 1; + } + + return 0; +} diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h index b2983b7..0e4f912 100644 --- a/kernel/sched/pelt.h +++ b/kernel/sched/pelt.h @@ -4,6 +4,7 @@ int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se); int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se); int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); int update_rt_rq_load_avg(u64 now, struct rq *rq, int running); +int update_dl_rq_load_avg(u64 now, struct rq *rq, int running); /* * When a task is dequeued, its estimated utilization should not be update if @@ -45,6 +46,11 @@ update_rt_rq_load_avg(u64 now, struct rq *rq, int running) return 0; } +static inline int +update_dl_rq_load_avg(u64 now, struct rq *rq, int running) +{ + return 0; +} #endif diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index f2b12b0..7d7d4f4 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -849,6 +849,7 @@ struct rq { u64 rt_avg; u64 age_stamp; struct sched_avg avg_rt; + struct sched_avg avg_dl; u64 idle_stamp; u64 avg_idle; From patchwork Thu Jun 28 15:45:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140462 Delivered-To: patch@linaro.org Received: by 2002:a50:86eb:0:0:0:0:0 with SMTP id 40-v6csp1954951edu; Thu, 28 Jun 2018 08:47:00 -0700 (PDT) X-Google-Smtp-Source: ADUXVKI6SDQaQRi+6qHvOdgWhjIq3XsM52/eJHUPyXEx54aurOGn1SrzXGe/pdqheQMTgo4I1Pre X-Received: by 2002:a65:4e86:: with SMTP id b6-v6mr9478978pgs.322.1530200819942; Thu, 28 Jun 2018 08:46:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200819; cv=none; d=google.com; s=arc-20160816; b=ePzw3evXX+cniyiamxYL5AzUmfhbT27+WfSiD340ALXURrcPS039JrUEXfl8WlgvyO e2T4NZOrIG8CABrEfi33xcSyrDxFQ8TTY+HEsDW+RVT9DB8Y+bcyOJa56A7WjbDJ1ZWA E2wzLYq5CstPpMkqfe/9i7IzBQrjb5Du5JFq2vZNQD7cYSR6mw8ibWmpkd/j4SsS10t7 hrZp/EOQ+ceYnZatRCDK0jj4j8LX+kQJP60/kH0x1Ov6nnYGfMudouiISJ6+x17cw8r9 UPmvIn+GYQOZxBggY5WyRD9mBsjlclaiviqzThZBXMCrANPX9lRi3yOU0A2iaWAkS82t juyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=YT5me/HnRjYE9yp/0nmuxMBvwSfbkC95sqdLOxFzYCo=; b=C1dy2W/nT6bu17ACU/i0WmhK73u8Ymb6ez8wzCin3TDRw2LCvWh4uooaxoBEWTWk7m YDivZmDH707J08XovQ8YMOm7AJGe4y8I3+t4PE53IbRACAan2GuWuyToDJSf2jJd4iWm oFAzSEWNJzkhyZxrUe6rxF1LdcaWc79uWo5YdmDyXObND5k8nTRmL6y5YxB2HOerLjjH ZAKmDVUEfuPqWk/n180iGH/QSSZ+qV2UhECAaAvtMUAvh1Bqnr4ZVk7iomgr400W28id W5fNqOMHSbiSF10hy7c2GxGLTRGqgmK48uzB1waR16ncrhw8s9QzBo9tJoXeuezlhP6k Nshg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LUQuXuLi; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t6-v6si7059412plo.508.2018.06.28.08.46.59; Thu, 28 Jun 2018 08:46:59 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=LUQuXuLi; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754131AbeF1Pq4 (ORCPT + 31 others); Thu, 28 Jun 2018 11:46:56 -0400 Received: from mail-wr0-f196.google.com ([209.85.128.196]:36868 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S967314AbeF1Ppa (ORCPT ); Thu, 28 Jun 2018 11:45:30 -0400 Received: by mail-wr0-f196.google.com with SMTP id k6-v6so6009749wrp.4 for ; Thu, 28 Jun 2018 08:45:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YT5me/HnRjYE9yp/0nmuxMBvwSfbkC95sqdLOxFzYCo=; b=LUQuXuLinYCDUNKo+4+Q7SVtwB/719UuWQM1z0wXEsOyEU7vN7h3s5xvp4wrz6C9Mx EDvrhjFaOPGXeqebieRzVOO1PERkEhSxQUxHT9QvOV+HK34Bij0VqR1zZM27hDHSoPYR twIAjZb6QbBc0pdEoZyqFgUdrTBwWZv8ggJHA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YT5me/HnRjYE9yp/0nmuxMBvwSfbkC95sqdLOxFzYCo=; b=TD5LpxKykOssMjTOZp+J/eEFc+kwAlbWn4b32ba7ci6cOq5Wsiyc7DRFZVegX1Qq3r aT1t5gFikPh+5DHxpmzwqbQmS7uyLsdYbwItjnF3YSUZDrHoI/MoCiYLt3481ZU5c3tU eyHLXRCqTtXCmR2jbad+SebdIBkW38Yr64DHeTWQpNC3umyPWQkCe6e92JA3e2rWtpqE GbmQ5Q8P43J4d3nj2dqzzEXE3OKEsV+a0ex4j5M4W3DwMgYszzShXPcZaYKk4i86OmOo HaqrC+Q8SmpMH/sc7/e+JdSgVg6NrwTDdquri65AeVKFP3VjWvqEz5ReUhAJYYY9xPUe bWHg== X-Gm-Message-State: APt69E2l1U2tzJEC4roGE/1IJgvnVGpGl+ucCFhxUwzzaiTZit/dX2Rj z2o7ezxbVbstIi7UBoHjgs2BGQ== X-Received: by 2002:adf:f708:: with SMTP id r8-v6mr4372177wrp.85.1530200728655; Thu, 28 Jun 2018 08:45:28 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:27 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 05/11] cpufreq/schedutil: use dl utilization tracking Date: Thu, 28 Jun 2018 17:45:08 +0200 Message-Id: <1530200714-4504-6-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Now that we have both the dl class bandwidth requirement and the dl class utilization, we can detect when CPU is fully used so we should run at max. Otherwise, we keep using the dl bandwidth requirement to define the utilization of the CPU Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/cpufreq_schedutil.c | 23 +++++++++++++++++------ kernel/sched/sched.h | 7 ++++++- 2 files changed, 23 insertions(+), 7 deletions(-) -- 2.7.4 Acked-by: Viresh Kumar diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 9c5e92e..edfbfc1 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -56,6 +56,7 @@ struct sugov_cpu { /* The fields below are only needed when sharing a policy: */ unsigned long util_cfs; unsigned long util_dl; + unsigned long bw_dl; unsigned long util_rt; unsigned long max; @@ -187,6 +188,7 @@ static void sugov_get_util(struct sugov_cpu *sg_cpu) sg_cpu->max = arch_scale_cpu_capacity(NULL, sg_cpu->cpu); sg_cpu->util_cfs = cpu_util_cfs(rq); sg_cpu->util_dl = cpu_util_dl(rq); + sg_cpu->bw_dl = cpu_bw_dl(rq); sg_cpu->util_rt = cpu_util_rt(rq); } @@ -198,20 +200,29 @@ static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) if (rq->rt.rt_nr_running) return sg_cpu->max; - util = sg_cpu->util_dl; - util += sg_cpu->util_cfs; + util = sg_cpu->util_cfs; util += sg_cpu->util_rt; + if ((util + sg_cpu->util_dl) >= sg_cpu->max) + return sg_cpu->max; + /* - * Utilization required by DEADLINE must always be granted while, for - * FAIR, we use blocked utilization of IDLE CPUs as a mechanism to - * gracefully reduce the frequency when no tasks show up for longer + * As there is still idle time on the CPU, we need to compute the + * utilization level of the CPU. + * + * Bandwidth required by DEADLINE must always be granted while, for + * FAIR and RT, we use blocked utilization of IDLE CPUs as a mechanism + * to gracefully reduce the frequency when no tasks show up for longer * periods of time. * * Ideally we would like to set util_dl as min/guaranteed freq and * util_cfs + util_dl as requested freq. However, cpufreq is not yet * ready for such an interface. So, we only do the latter for now. */ + + /* Add DL bandwidth requirement */ + util += sg_cpu->bw_dl; + return min(sg_cpu->max, util); } @@ -367,7 +378,7 @@ static inline bool sugov_cpu_is_busy(struct sugov_cpu *sg_cpu) { return false; } */ static inline void ignore_dl_rate_limit(struct sugov_cpu *sg_cpu, struct sugov_policy *sg_policy) { - if (cpu_util_dl(cpu_rq(sg_cpu->cpu)) > sg_cpu->util_dl) + if (cpu_bw_dl(cpu_rq(sg_cpu->cpu)) > sg_cpu->bw_dl) sg_policy->need_freq_update = true; } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 7d7d4f4..ef5d6aa 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2192,11 +2192,16 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) {} #endif #ifdef CONFIG_CPU_FREQ_GOV_SCHEDUTIL -static inline unsigned long cpu_util_dl(struct rq *rq) +static inline unsigned long cpu_bw_dl(struct rq *rq) { return (rq->dl.running_bw * SCHED_CAPACITY_SCALE) >> BW_SHIFT; } +static inline unsigned long cpu_util_dl(struct rq *rq) +{ + return READ_ONCE(rq->avg_dl.util_avg); +} + static inline unsigned long cpu_util_cfs(struct rq *rq) { unsigned long util = READ_ONCE(rq->cfs.avg.util_avg); From patchwork Thu Jun 28 15:45:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140461 Delivered-To: patch@linaro.org Received: by 2002:a50:86eb:0:0:0:0:0 with SMTP id 40-v6csp1954880edu; Thu, 28 Jun 2018 08:46:56 -0700 (PDT) X-Google-Smtp-Source: ADUXVKISzElupBvpeyV8H4p/Cio1DXR1TTJ7W5betKQLO+HmHf4kXxd3gqPhtk7esh38r0z485zw X-Received: by 2002:a63:6383:: with SMTP id x125-v6mr9257207pgb.127.1530200816070; Thu, 28 Jun 2018 08:46:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200816; cv=none; d=google.com; s=arc-20160816; b=FSjmk5vgE3oycCccd2Wlla7EhsJKID5i7h/Gd9Wi4oIcAqi8rvOZHZ6ezkPASoXoCG 38/BLL92zgr20eYBIneVj72viISMFc/RPZibkXQ/YoTKp/+n+GKJP5wEjtFqdoaXg3vg EJFSxJqcy/e78rfI+ZkBz+hLocdzFy8OcXnU+889fwthTBITgVVkO3WQwbPQnSxhn4dy rL41wDlpF5X6HOHaXDht9Fuqu5cpogyEztMZSVA3S9m+jlIPs/e9GmkbrNcxWVBVHFFL wAfEA0/3Ej09gLfWR3BqnX1KU2NEXCuAcleLTzj/q4LSTGlffn9Qt7uSJvhPjDAgfTGb FM5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=khbfR0A1jc9cKblSJKeoc1wGMCANZRoaQHs1pMgMrS0=; b=yVK02MwbYrzjREYo6F5gSCKIMXnqyNV7Oa3qBb/88nR4qTABurdZnzcFBVdSvwGwC7 czoYjtQbqj8VEVlBMI+6IZYpsnuArn2c3AHXnhlOcYRtwg39LPiFLwKW8nItMnpZHGzg R5vGhM7s7hs/gFkoN/J8ClFs1k1J21KzARK2bR6HTWRu2eJ0CcdXYb4cdTuwdU+4THJd Av4ACQ89N7zw2D6+j1NJn3oAqjEOgEzT7yC1x0n38TJ06tdznIhXoYlH/VKAsfjxqhz0 DyQ7Wf6f6iBuAUzBtM5ym2nHp0AXtxzw5mhpCeUBtTDT9iumBKRw6lTkLIzG93y4cI0a dDMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="EnY9KF/A"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l13-v6si6553458pff.261.2018.06.28.08.46.55; Thu, 28 Jun 2018 08:46:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="EnY9KF/A"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935527AbeF1Pqw (ORCPT + 31 others); Thu, 28 Jun 2018 11:46:52 -0400 Received: from mail-wm0-f65.google.com ([74.125.82.65]:36611 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S967319AbeF1Ppb (ORCPT ); Thu, 28 Jun 2018 11:45:31 -0400 Received: by mail-wm0-f65.google.com with SMTP id u18-v6so9602254wmc.1 for ; Thu, 28 Jun 2018 08:45:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=khbfR0A1jc9cKblSJKeoc1wGMCANZRoaQHs1pMgMrS0=; b=EnY9KF/AmhXIp6ruDeq4SkXzGchRIRzF1759AjT3m3rjhJPCmhGvZjOMAxaeJ2NOTx FxfMOULgBvxNmmurjAiJERwozC++180LRCvJlMeNNiDIMzg1HK2J5CYbqB1F1b8iB+ge kgLCNfhQHx/BzbiyaVl4lgovMS+/v/jUlet28= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=khbfR0A1jc9cKblSJKeoc1wGMCANZRoaQHs1pMgMrS0=; b=OgfIt3/jnPvE12oDUEEB3K4pbjM2cvjq2Tv2TPtvsXf8XgvSpQbUgwinqR92zqhgXN gZLgL2hkX676bre5BoofdgjEJf6UUtAwNQNe6DNe4Vy+Xcxt9ALCzFJRFUx1jLVG8N5E 3vK11qvh3IlkHlG4TZplnxKhT04lXhNTSANrD63LmJ+bekooQVtOtQduH1/MPhhrPNRi qG+97quLhq7glNnKF4ZPXbtbbLb4LUzma6MR3TTV3LrbFFWRebSAkJLknJBYxy8w0Of0 Y6LTvq+HrFsMEyeOiE9Xt3vpfAVfIrjrk3/ssOhrHmNPosXCBRkKfwtj/XNfBsEc4Boh XXww== X-Gm-Message-State: APt69E3lfrqlxMOuvsE/gsdZqY/znRug/c0scu1io4n01YLQAVyYtCiX q/eziLi0ro3jKJutIbkq7ziRVQ== X-Received: by 2002:a1c:afd4:: with SMTP id y203-v6mr8286102wme.55.1530200729927; Thu, 28 Jun 2018 08:45:29 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:29 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 06/11] sched/irq: add irq utilization tracking Date: Thu, 28 Jun 2018 17:45:09 +0200 Message-Id: <1530200714-4504-7-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org interrupt and steal time are the only remaining activities tracked by rt_avg. Like for sched classes, we can use PELT to track their average utilization of the CPU. But unlike sched class, we don't track when entering/leaving interrupt; Instead, we take into account the time spent under interrupt context when we update rqs' clock (rq_clock_task). This also means that we have to decay the normal context time and account for interrupt time during the update. That's also important to note that because rq_clock == rq_clock_task + interrupt time and rq_clock_task is used by a sched class to compute its utilization, the util_avg of a sched class only reflects the utilization of the time spent in normal context and not of the whole time of the CPU. The utilization of interrupt gives an more accurate level of utilization of CPU. The CPU utilization is : avg_irq + (1 - avg_irq / max capacity) * /Sum avg_rq Most of the time, avg_irq is small and neglictible so the use of the approximation CPU utilization = /Sum avg_rq was enough Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/core.c | 4 +++- kernel/sched/fair.c | 13 ++++++++++--- kernel/sched/pelt.c | 40 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/pelt.h | 16 ++++++++++++++++ kernel/sched/sched.h | 3 +++ 5 files changed, 72 insertions(+), 4 deletions(-) -- 2.7.4 diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 78d8fac..e5263a4 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -18,6 +18,8 @@ #include "../workqueue_internal.h" #include "../smpboot.h" +#include "pelt.h" + #define CREATE_TRACE_POINTS #include @@ -186,7 +188,7 @@ static void update_rq_clock_task(struct rq *rq, s64 delta) #if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) if ((irq_delta + steal) && sched_feat(NONTASK_CAPACITY)) - sched_rt_avg_update(rq, irq_delta + steal); + update_irq_load_avg(rq, irq_delta + steal); #endif } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ffce4b2..d2758e3 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7289,7 +7289,7 @@ static inline bool cfs_rq_has_blocked(struct cfs_rq *cfs_rq) return false; } -static inline bool others_rqs_have_blocked(struct rq *rq) +static inline bool others_have_blocked(struct rq *rq) { if (READ_ONCE(rq->avg_rt.util_avg)) return true; @@ -7297,6 +7297,11 @@ static inline bool others_rqs_have_blocked(struct rq *rq) if (READ_ONCE(rq->avg_dl.util_avg)) return true; +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) + if (READ_ONCE(rq->avg_irq.util_avg)) + return true; +#endif + return false; } @@ -7361,8 +7366,9 @@ static void update_blocked_averages(int cpu) } update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); + update_irq_load_avg(rq, 0); /* Don't need periodic decay once load/util_avg are null */ - if (others_rqs_have_blocked(rq)) + if (others_have_blocked(rq)) done = false; #ifdef CONFIG_NO_HZ_COMMON @@ -7431,9 +7437,10 @@ static inline void update_blocked_averages(int cpu) update_cfs_rq_load_avg(cfs_rq_clock_task(cfs_rq), cfs_rq); update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); + update_irq_load_avg(rq, 0); #ifdef CONFIG_NO_HZ_COMMON rq->last_blocked_load_update_tick = jiffies; - if (!cfs_rq_has_blocked(cfs_rq) && !others_rqs_have_blocked(rq)) + if (!cfs_rq_has_blocked(cfs_rq) && !others_have_blocked(rq)) rq->has_blocked_load = 0; #endif rq_unlock_irqrestore(rq, &rf); diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index 8b78b63..ead6d8b 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -357,3 +357,43 @@ int update_dl_rq_load_avg(u64 now, struct rq *rq, int running) return 0; } + +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) +/* + * irq: + * + * util_sum = \Sum se->avg.util_sum but se->avg.util_sum is not tracked + * util_sum = cpu_scale * load_sum + * runnable_load_sum = load_sum + * + */ + +int update_irq_load_avg(struct rq *rq, u64 running) +{ + int ret = 0; + /* + * We know the time that has been used by interrupt since last update + * but we don't when. Let be pessimistic and assume that interrupt has + * happened just before the update. This is not so far from reality + * because interrupt will most probably wake up task and trig an update + * of rq clock during which the metric si updated. + * We start to decay with normal context time and then we add the + * interrupt context time. + * We can safely remove running from rq->clock because + * rq->clock += delta with delta >= running + */ + ret = ___update_load_sum(rq->clock - running, rq->cpu, &rq->avg_irq, + 0, + 0, + 0); + ret += ___update_load_sum(rq->clock, rq->cpu, &rq->avg_irq, + 1, + 1, + 1); + + if (ret) + ___update_load_avg(&rq->avg_irq, 1, 1); + + return ret; +} +#endif diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h index 0e4f912..d2894db 100644 --- a/kernel/sched/pelt.h +++ b/kernel/sched/pelt.h @@ -6,6 +6,16 @@ int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); int update_rt_rq_load_avg(u64 now, struct rq *rq, int running); int update_dl_rq_load_avg(u64 now, struct rq *rq, int running); +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) +int update_irq_load_avg(struct rq *rq, u64 running); +#else +static inline int +update_irq_load_avg(struct rq *rq, u64 running) +{ + return 0; +} +#endif + /* * When a task is dequeued, its estimated utilization should not be update if * its util_avg has not been updated at least once. @@ -51,6 +61,12 @@ update_dl_rq_load_avg(u64 now, struct rq *rq, int running) { return 0; } + +static inline int +update_irq_load_avg(struct rq *rq, u64 running) +{ + return 0; +} #endif diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index ef5d6aa..377be2b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -850,6 +850,9 @@ struct rq { u64 age_stamp; struct sched_avg avg_rt; struct sched_avg avg_dl; +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) + struct sched_avg avg_irq; +#endif u64 idle_stamp; u64 avg_idle; From patchwork Thu Jun 28 15:45:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140459 Delivered-To: patch@linaro.org Received: by 2002:a50:86eb:0:0:0:0:0 with SMTP id 40-v6csp1954567edu; Thu, 28 Jun 2018 08:46:37 -0700 (PDT) X-Google-Smtp-Source: ADUXVKL2eYeAyOYKlPqFcfIUMknjn4R5Iy+Mm/iZ/sEC178+NX9e3G7mKI+R+DW7xGpVhLv55ld4 X-Received: by 2002:a17:902:7898:: with SMTP id q24-v6mr10977701pll.254.1530200797283; Thu, 28 Jun 2018 08:46:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200797; cv=none; d=google.com; s=arc-20160816; b=sBf3JK7ra4ecHcNNu7jlxovlmU57mu387WaQ46WqpaCOlQbsjzBuQq6insrfs5oYgv 1Agq3XUo44PmY/dggzSiETAMBHmd4Rle92JCiFNyPcD0sp25q/HhPLK7oZKkQ4Vp0ZIu kuX4oDWBFa5lNjV7wKlyV83ckvcsK3XzZFCpQHvkNvCpOzOUFz7Z0G9gpAextgHn5CBm aenKmm1SHiCo2ku27y6z0twf9Q7ezbfqbtQE8/lrm2/grPCZavFrth28PZ4R81O+FEMT S+kcFqSzrlRe1DbbFG/+s0i0jtjbo3skpA/k4mX3JxyLg1qm9iILXRcO5l23J2LJRKSf 5DJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=LJM9mfokBEkFWGXIXh91vliCCTSGjKg+K/zcNJNEiLM=; b=IGVJVbzgn24ZF+2JC/+rFQWaL7CjVhCppyuq3Cdmh99jT8zV1229GayK3hGco9V3j0 ggi2GKhpXW4L24MafxzG3DJLhAPmipCglRK/CB9AeaNzijxcSiE0GFoceHRDiqv7BufD nhE8H6EmG3TLSRyf2WcnSYMcJS+haY9My0c0cbHk0tOEWHUDAK+rWa+SxKVvzaz7De4j WfL+CV6so6P5QF4x6EShrckCnvQGzFOUe7wkf1bfQfaXUHddZ34LNjg/leAzZ7CjVYRB PxRAtHGbF6U4A+JSq6VdnNwNWjo8jvrigQ8bZj0E9tXe+HguIpVUgsYvSQtcv71yQ9ri 8mHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YAG7P0jM; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 188-v6si6190198pgj.169.2018.06.28.08.46.36; Thu, 28 Jun 2018 08:46:37 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YAG7P0jM; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935514AbeF1Pqf (ORCPT + 31 others); Thu, 28 Jun 2018 11:46:35 -0400 Received: from mail-wr0-f194.google.com ([209.85.128.194]:41778 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1030200AbeF1Ppc (ORCPT ); Thu, 28 Jun 2018 11:45:32 -0400 Received: by mail-wr0-f194.google.com with SMTP id h10-v6so5995385wrq.8 for ; Thu, 28 Jun 2018 08:45:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LJM9mfokBEkFWGXIXh91vliCCTSGjKg+K/zcNJNEiLM=; b=YAG7P0jMqLC0N+FEFJYFNzId+EuXU0Ba3HtU1Cqmz9rJ464yvAcOstX81PPfvj41YV /QHVNC5zm+ME+7McKZZizCWTyiBbRhvI3s3ODDyQIpLmuyH7lh60dyk/dD1ormCGZRTM 3MP5SBvqEodIYQI89BeEiU8VWnApAp6O0MzI0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LJM9mfokBEkFWGXIXh91vliCCTSGjKg+K/zcNJNEiLM=; b=BtcvuIoj/jmh6uQQrW37OUmLp58GCbl+4r5K+ZTFd7OnV8q1I1DzjXrizyegm6uGeo kIbwZvJKPWBMu2TmStYZC+qQy1eR6QcQc9GFl/TC9R/aaVK7ldBauJmDIyjssFPFKvMM kPPeJfvhyQvtgPcX1W33rknh43gqEdn6ntE/bZmf92gYYeJmBLicvQGZQyY+qNJaQzzh M+4qageeqSbX+64+JN7QOk9+Q8qtGqT890/DLkpmOfiloYWpMyuAcDKh+0CX+HrjKvOE l6suN1QcRhGd1Kf772eRJ93HKwpId0j/M5VDEoXewhWDjHv5sW3CR6l2KNnx39UITUx5 v2Bw== X-Gm-Message-State: APt69E1kZhK8Y9Kc1Aib+NXummdUENy25lQDc/Ok4p32+HP1xlFriFat 8j4bF8DCOXmPbr9rYqxulTy+EQ== X-Received: by 2002:adf:b782:: with SMTP id s2-v6mr8931779wre.247.1530200731383; Thu, 28 Jun 2018 08:45:31 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:30 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 07/11] cpufreq/schedutil: take into account interrupt Date: Thu, 28 Jun 2018 17:45:10 +0200 Message-Id: <1530200714-4504-8-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The time spent under interrupt can be significant but it is not reflected in the utilization of CPU when deciding to choose an OPP. Now that we have access to this metric, schedutil can take it into account when selecting the OPP for a CPU. rqs utilization don't see the time spend under interrupt context and report their value in the normal context time window. We need to compensate this when adding interrupt utilization The CPU utilization is : irq util_avg + (1 - irq util_avg / max capacity ) * /Sum rq util_avg A test with iperf on hikey (octo arm64) gives: iperf -c server_address -r -t 5 w/o patch w/ patch Tx 276 Mbits/sec 304 Mbits/sec +10% Rx 299 Mbits/sec 328 Mbits/sec +09% 8 iterations stdev is lower than 1% Only WFI idle state is enable (shallowest diel state) Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/cpufreq_schedutil.c | 25 +++++++++++++++++++++---- kernel/sched/sched.h | 13 +++++++++++++ 2 files changed, 34 insertions(+), 4 deletions(-) -- 2.7.4 Acked-by: Viresh Kumar diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index edfbfc1..b77bfef 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -58,6 +58,7 @@ struct sugov_cpu { unsigned long util_dl; unsigned long bw_dl; unsigned long util_rt; + unsigned long util_irq; unsigned long max; /* The field below is for single-CPU policies only: */ @@ -190,21 +191,30 @@ static void sugov_get_util(struct sugov_cpu *sg_cpu) sg_cpu->util_dl = cpu_util_dl(rq); sg_cpu->bw_dl = cpu_bw_dl(rq); sg_cpu->util_rt = cpu_util_rt(rq); + sg_cpu->util_irq = cpu_util_irq(rq); } static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) { struct rq *rq = cpu_rq(sg_cpu->cpu); - unsigned long util; + unsigned long util, max = sg_cpu->max; if (rq->rt.rt_nr_running) return sg_cpu->max; + if (unlikely(sg_cpu->util_irq >= max)) + return max; + + /* Sum rq utilization */ util = sg_cpu->util_cfs; util += sg_cpu->util_rt; - if ((util + sg_cpu->util_dl) >= sg_cpu->max) - return sg_cpu->max; + /* + * Interrupt time is not seen by rqs utilization nso we can compare + * them with the CPU capacity + */ + if ((util + sg_cpu->util_dl) >= max) + return max; /* * As there is still idle time on the CPU, we need to compute the @@ -220,10 +230,17 @@ static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) * ready for such an interface. So, we only do the latter for now. */ + /* Weight rqs utilization to normal context window */ + util *= (max - sg_cpu->util_irq); + util /= max; + + /* Add interrupt utilization */ + util += sg_cpu->util_irq; + /* Add DL bandwidth requirement */ util += sg_cpu->bw_dl; - return min(sg_cpu->max, util); + return min(max, util); } /** diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 377be2b..9438e68 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2221,4 +2221,17 @@ static inline unsigned long cpu_util_rt(struct rq *rq) { return rq->avg_rt.util_avg; } + +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) +static inline unsigned long cpu_util_irq(struct rq *rq) +{ + return rq->avg_irq.util_avg; +} +#else +static inline unsigned long cpu_util_irq(struct rq *rq) +{ + return 0; +} + +#endif #endif From patchwork Thu Jun 28 15:45:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140455 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2343015ljj; Thu, 28 Jun 2018 08:45:42 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJF3f8M5n7aqYE8gX3evrRX9Z+YTtBU8dIcBnsffuHZAukb86J6/4DlQKM3TQnO1L82AN0D X-Received: by 2002:a63:7f16:: with SMTP id a22-v6mr9280607pgd.255.1530200742784; Thu, 28 Jun 2018 08:45:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200742; cv=none; d=google.com; s=arc-20160816; b=QipOqTsX6zQZJhYpt0SKyDwP+/t33terig8KSuJm9/rEmG8NadU+yqrtR12OKpjQi1 JDtUiy401cdth1GBN0qwv8COjgxLYtF/2slS6fc+QwW+fSDeUBmSh6qWaGd5tDDPzVvb v9yvS8tS0pfkD/loHAjQ0tuvxGTu5OuGwq2NlOM8ODgu5TMBY47jgzaZ70WjtU12Sju9 2F00ft5NRz40PsWBXCNOkmNOpCJveT4JVJ2WFGbMCqnk64VB8MrR0mzJ7DWtdO7HnPDq a4Ue/YGDB7F5T66TSWHT+4F08xZ/2ySOGgP3i1LVs/Mm5ojAeZ9Siv5eSGuMSSMlPh6W nsmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=T44Fp1bkuBlEc/StRgz1812IqYNO9JGIyE7AOt7QI4c=; b=o/uZWMxuf8k9Z6spQRlKqhSJeKHX3v47jEbnJirIMbpEDtBDbY01ZWYYMDL7OHsG9Z CivzWS5KWE/Ar8RFxJwN+hj5hgeVrMQBoD+FXlD5mOPtzXc+UI5mTH69z0aukH9k1r4q a4t67QvXxJ8llrHXNRva53RUtfzec/kkTnSQR2jkHdlDQbGXD9TAHR2pr+M+yHvpSm2p 7Paa4uMwUrtiovVsarcXs+kc1Q/uY4gaRQJm8vMFR/7bhixik3hIRvjv6YMl3cnHfSqr r6EJDualVZ7pP/Ik0GEKMXNtYUyCDZs46xfoVMqgN1v60eAlLCpylcZ6iv+24QO7SXLw S8Wg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cC9zEnGD; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q76-v6si6501234pgq.597.2018.06.28.08.45.42; Thu, 28 Jun 2018 08:45:42 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cC9zEnGD; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1030229AbeF1Ppi (ORCPT + 31 others); Thu, 28 Jun 2018 11:45:38 -0400 Received: from mail-wr0-f195.google.com ([209.85.128.195]:36875 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932316AbeF1Ppf (ORCPT ); Thu, 28 Jun 2018 11:45:35 -0400 Received: by mail-wr0-f195.google.com with SMTP id k6-v6so6010042wrp.4 for ; Thu, 28 Jun 2018 08:45:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=T44Fp1bkuBlEc/StRgz1812IqYNO9JGIyE7AOt7QI4c=; b=cC9zEnGDp61FSn/uuolUneADfnI8vlKJaUYl+Iwhpecy9RmDJsq41UKOT96A4Ac4M/ iFpaa9bdxlp285udHk64AVALTqrnkN1dAg5LJo5ZkYNfbyZecLKjXhHIbwJZXIH973x4 PKvgqwTm4jwDy5I5NWKSLp7qfzYW6AqK/AfqE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=T44Fp1bkuBlEc/StRgz1812IqYNO9JGIyE7AOt7QI4c=; b=crj14VBdBYAfMCh568rjD5GdxIxgzVpxT35CkS8SBLnhN9ECU1RMk1s+BCH9UhVMCv EadyJ6U3MKJ4XddnLw+O5boPdB25C2QfhGv6wwibys/HTxZ0f4nj+gLKEfWjgqIWVzPG A2enCk56PWKcSMwdMVC0U9ae5DadzMAmgn+vEAsmbHabpe6CDmWSIPzvBk7Er2FklFRx TxzeMt6GzyUqdZF2rdhVSSs5LPxUfIP81j/CpLJD6t+dyR9mc45KxkYv6k9M4MHTkMN/ c1sTEO+tnPOWzmzOa/38+ehr5bB36l6upY3LY8hd7YU8j8Nc2PyT1gZP/CjHVsgWznSe 8SRQ== X-Gm-Message-State: APt69E2fA84mYi9lk29XoCeW7Wbz8zbKBcyH/s9doKDUXQF3GvSJLkoq s8B2JVfuhoj814k6CYRjZBD4ew== X-Received: by 2002:adf:ff88:: with SMTP id j8-v6mr9547540wrr.137.1530200733686; Thu, 28 Jun 2018 08:45:33 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:32 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 08/11] sched: schedutil: remove sugov_aggregate_util() Date: Thu, 28 Jun 2018 17:45:11 +0200 Message-Id: <1530200714-4504-9-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org There is no reason why sugov_get_util() and sugov_aggregate_util() were in fact separate functions. Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Peter Zijlstra (Intel) [rebased after adding irq tracking and fixed some compilation errors] Signed-off-by: Vincent Guittot --- kernel/sched/cpufreq_schedutil.c | 44 ++++++++++++++-------------------------- kernel/sched/sched.h | 2 +- 2 files changed, 16 insertions(+), 30 deletions(-) -- 2.7.4 Acked-by: Viresh Kumar diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index b77bfef..d04f941 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -53,12 +53,7 @@ struct sugov_cpu { unsigned int iowait_boost_max; u64 last_update; - /* The fields below are only needed when sharing a policy: */ - unsigned long util_cfs; - unsigned long util_dl; unsigned long bw_dl; - unsigned long util_rt; - unsigned long util_irq; unsigned long max; /* The field below is for single-CPU policies only: */ @@ -182,38 +177,31 @@ static unsigned int get_next_freq(struct sugov_policy *sg_policy, return cpufreq_driver_resolve_freq(policy, freq); } -static void sugov_get_util(struct sugov_cpu *sg_cpu) +static unsigned long sugov_get_util(struct sugov_cpu *sg_cpu) { struct rq *rq = cpu_rq(sg_cpu->cpu); + unsigned long util, irq, max; - sg_cpu->max = arch_scale_cpu_capacity(NULL, sg_cpu->cpu); - sg_cpu->util_cfs = cpu_util_cfs(rq); - sg_cpu->util_dl = cpu_util_dl(rq); - sg_cpu->bw_dl = cpu_bw_dl(rq); - sg_cpu->util_rt = cpu_util_rt(rq); - sg_cpu->util_irq = cpu_util_irq(rq); -} - -static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) -{ - struct rq *rq = cpu_rq(sg_cpu->cpu); - unsigned long util, max = sg_cpu->max; + sg_cpu->max = max = arch_scale_cpu_capacity(NULL, sg_cpu->cpu); + sg_cpu->bw_dl = cpu_bw_dl(rq); if (rq->rt.rt_nr_running) - return sg_cpu->max; + return max; + + irq = cpu_util_irq(rq); - if (unlikely(sg_cpu->util_irq >= max)) + if (unlikely(irq >= max)) return max; /* Sum rq utilization */ - util = sg_cpu->util_cfs; - util += sg_cpu->util_rt; + util = cpu_util_cfs(rq); + util += cpu_util_rt(rq); /* * Interrupt time is not seen by rqs utilization nso we can compare * them with the CPU capacity */ - if ((util + sg_cpu->util_dl) >= max) + if ((util + cpu_util_dl(rq)) >= max) return max; /* @@ -231,11 +219,11 @@ static unsigned long sugov_aggregate_util(struct sugov_cpu *sg_cpu) */ /* Weight rqs utilization to normal context window */ - util *= (max - sg_cpu->util_irq); + util *= (max - irq); util /= max; /* Add interrupt utilization */ - util += sg_cpu->util_irq; + util += irq; /* Add DL bandwidth requirement */ util += sg_cpu->bw_dl; @@ -418,9 +406,8 @@ static void sugov_update_single(struct update_util_data *hook, u64 time, busy = sugov_cpu_is_busy(sg_cpu); - sugov_get_util(sg_cpu); + util = sugov_get_util(sg_cpu); max = sg_cpu->max; - util = sugov_aggregate_util(sg_cpu); sugov_iowait_apply(sg_cpu, time, &util, &max); next_f = get_next_freq(sg_policy, util, max); /* @@ -459,9 +446,8 @@ static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu, u64 time) struct sugov_cpu *j_sg_cpu = &per_cpu(sugov_cpu, j); unsigned long j_util, j_max; - sugov_get_util(j_sg_cpu); + j_util = sugov_get_util(j_sg_cpu); j_max = j_sg_cpu->max; - j_util = sugov_aggregate_util(j_sg_cpu); sugov_iowait_apply(j_sg_cpu, time, &j_util, &j_max); if (j_util * max > j_max * util) { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9438e68..59a633d 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2219,7 +2219,7 @@ static inline unsigned long cpu_util_cfs(struct rq *rq) static inline unsigned long cpu_util_rt(struct rq *rq) { - return rq->avg_rt.util_avg; + return READ_ONCE(rq->avg_rt.util_avg); } #if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) From patchwork Thu Jun 28 15:45:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140458 Delivered-To: patch@linaro.org Received: by 2002:a50:86eb:0:0:0:0:0 with SMTP id 40-v6csp1954348edu; Thu, 28 Jun 2018 08:46:22 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJfyAS1bBNvL8Wh7FXcnosKETFY8/bL0OghUv90cxgNJHyxhIatHxBO4WsnZlXedArxr/gM X-Received: by 2002:a63:91c8:: with SMTP id l191-v6mr9088540pge.180.1530200782473; Thu, 28 Jun 2018 08:46:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200782; cv=none; d=google.com; s=arc-20160816; b=wcWnCfhBTBdIVhC1mo/q52dDZFf6icdQ/UWR4EzTwBV+Usyq02FwrtffOxgBDHY2CC gLG8SCx73PI7pGzLLfgcLTUiYZyFRA6rm4b22iWjb4TbbrrlVK+pTTxh4FF5yGmkALQF WtdXNq80VAMVD75+HHgWxC0rdYgk1mrNNjbqEOb2qAttz/vDktH13krOO4KEbReWMFht u0qhUjbFoPn0rXQpKQOBCYubiMr0Z/0iGZEKh4vUj6WNaYAGPd/YS5ZFzqYmAEBqhj7f U0W9R7rJ+HoQ+BJT1a72NjxWDfnbl9o6xvMi/NXx1zBfs8HmJMFenaXcQKfn/TnKjj2s NKaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=Zl1T6q56msdYU0ByNc7HaXHBLSwGLVg5J9bQL90yjrE=; b=JU6E7N1z3sUJhpiBbK/bL125tMebnag2SV9sqURTZ7G4iXv5EiqsAIT/9G9jLEJf0D /kj/+qWde1cC+sOF8Rv/QsswGJXvpHQNMAlCrQiEnojjfxJhNJb+lwJMGbXZ5i5xTDiJ Vdb49s8XFKPi9TMIJf2gUNOxoW3G+DDbM9RdujKHZ7M0w1o0WlKDRQOXqGRBF2VJFQWp ZM7r/mi1XTGT5wy2GkOTGZMFXD4A7z2dhhE6IBRWGagKuzEkcYpKTkpPgGgqiOyHwTZ3 cMtEiSjLD9TcvnaIHk4lNeR5KVOqM9Rktj9Xe4ZHLa4Sml5sDeC4N5X+/gBDor62tc+E pCVA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="HB6/D75V"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o25-v6si1397984pfi.279.2018.06.28.08.46.22; Thu, 28 Jun 2018 08:46:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="HB6/D75V"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935495AbeF1PqU (ORCPT + 31 others); Thu, 28 Jun 2018 11:46:20 -0400 Received: from mail-wm0-f65.google.com ([74.125.82.65]:34877 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S935010AbeF1Ppg (ORCPT ); Thu, 28 Jun 2018 11:45:36 -0400 Received: by mail-wm0-f65.google.com with SMTP id z137-v6so9613949wmc.0 for ; Thu, 28 Jun 2018 08:45:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Zl1T6q56msdYU0ByNc7HaXHBLSwGLVg5J9bQL90yjrE=; b=HB6/D75VP9llFmXijrF8Pi7+2MV2uJ7hICGfVk38x//OXSMU0RET6k3syL9yeJDf/Z u8YjjmdH9gXzGpBoZJ/+xe5gN5PrEmEFsfTZ/fv7QoVyxpAtexLig86MLkk9yVQSWMv5 5Zt5Sr6ZuJ7YavA+Z9z0Wyjkb7sv+oENNPIGE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Zl1T6q56msdYU0ByNc7HaXHBLSwGLVg5J9bQL90yjrE=; b=nyUcRk1k3R1G1W4nFDFn1DcMigpCjqk01KKaJwQ9ZN62gEHzDiWY46E/klZYhktBxp xM9d1p85mPFoVQzuSmD1V5YRkWSE2apDk11p/K3OEdAGRHaylUCT+e7VuGEfaLH/pDWX Mbn/o8AvuTW/AUi+QqSzTJLwtcYjz6VXlqNKSZc3aG56wu1/wL1bW0acpIewZsxbhcQo 6MoAfHr8TPBsgVNLy2yxxY36NXhp6RCU0TIgdjhAWLiq6WuBxnPundELUM0HARm8xasU xa9dvkuvWdJMHKyxGm/o6/ugxcbAZnUGRO5c/5pPS47uhDBTPYuhKUyFd05JA5mEn919 svUw== X-Gm-Message-State: APt69E3vaIgr08qBGoAFaBdt9j/UcAE4WcZECS+w45KtXoU0xPiVqMqy lDNoY/qA3MxIuu2CSghkZMQ1Fg== X-Received: by 2002:a1c:b801:: with SMTP id i1-v6mr8440469wmf.30.1530200734938; Thu, 28 Jun 2018 08:45:34 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:34 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 09/11] sched: use pelt for scale_rt_capacity() Date: Thu, 28 Jun 2018 17:45:12 +0200 Message-Id: <1530200714-4504-10-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The utilization of the CPU by rt, dl and interrupts are now tracked with PELT so we can use these metrics instead of rt_avg to evaluate the remaining capacity available for cfs class. scale_rt_capacity() behavior has been changed and now returns the remaining capacity available for cfs instead of a scaling factor because rt, dl and interrupt provide now absolute utilization value. The same formula as schedutil is used: irq util_avg + (1 - irq util_avg / max capacity ) * /Sum rq util_avg but the implementation is different because it doesn't return the same value and doesn't benefit of the same optimization Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/deadline.c | 2 -- kernel/sched/fair.c | 41 +++++++++++++++++++---------------------- kernel/sched/pelt.c | 2 +- kernel/sched/rt.c | 2 -- 4 files changed, 20 insertions(+), 27 deletions(-) -- 2.7.4 diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index f4de2698..68b8a9f 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -1180,8 +1180,6 @@ static void update_curr_dl(struct rq *rq) curr->se.exec_start = now; cgroup_account_cputime(curr, delta_exec); - sched_rt_avg_update(rq, delta_exec); - if (dl_entity_is_special(dl_se)) return; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d2758e3..ce0dcbf 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7550,39 +7550,36 @@ static inline int get_sd_load_idx(struct sched_domain *sd, static unsigned long scale_rt_capacity(int cpu) { struct rq *rq = cpu_rq(cpu); - u64 total, used, age_stamp, avg; - s64 delta; - - /* - * Since we're reading these variables without serialization make sure - * we read them once before doing sanity checks on them. - */ - age_stamp = READ_ONCE(rq->age_stamp); - avg = READ_ONCE(rq->rt_avg); - delta = __rq_clock_broken(rq) - age_stamp; + unsigned long max = arch_scale_cpu_capacity(NULL, cpu); + unsigned long used, irq, free; - if (unlikely(delta < 0)) - delta = 0; +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) + irq = READ_ONCE(rq->avg_irq.util_avg); - total = sched_avg_period() + delta; + if (unlikely(irq >= max)) + return 1; +#endif - used = div_u64(avg, total); + used = READ_ONCE(rq->avg_rt.util_avg); + used += READ_ONCE(rq->avg_dl.util_avg); - if (likely(used < SCHED_CAPACITY_SCALE)) - return SCHED_CAPACITY_SCALE - used; + if (unlikely(used >= max)) + return 1; - return 1; + free = max - used; +#if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) + free *= (max - irq); + free /= max; +#endif + return free; } static void update_cpu_capacity(struct sched_domain *sd, int cpu) { - unsigned long capacity = arch_scale_cpu_capacity(sd, cpu); + unsigned long capacity = scale_rt_capacity(cpu); struct sched_group *sdg = sd->groups; - cpu_rq(cpu)->cpu_capacity_orig = capacity; - - capacity *= scale_rt_capacity(cpu); - capacity >>= SCHED_CAPACITY_SHIFT; + cpu_rq(cpu)->cpu_capacity_orig = arch_scale_cpu_capacity(sd, cpu); if (!capacity) capacity = 1; diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c index ead6d8b..35475c0 100644 --- a/kernel/sched/pelt.c +++ b/kernel/sched/pelt.c @@ -237,7 +237,7 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runna */ sa->load_avg = div_u64(load * sa->load_sum, divider); sa->runnable_load_avg = div_u64(runnable * sa->runnable_load_sum, divider); - sa->util_avg = sa->util_sum / divider; + WRITE_ONCE(sa->util_avg, sa->util_sum / divider); } /* diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 0e3e57a..2a881bd 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -970,8 +970,6 @@ static void update_curr_rt(struct rq *rq) curr->se.exec_start = now; cgroup_account_cputime(curr, delta_exec); - sched_rt_avg_update(rq, delta_exec); - if (!rt_bandwidth_enabled()) return; From patchwork Thu Jun 28 15:45:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140457 Delivered-To: patch@linaro.org Received: by 2002:a50:86eb:0:0:0:0:0 with SMTP id 40-v6csp1954129edu; Thu, 28 Jun 2018 08:46:08 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJX9FwieChYXVdhmArsypJf18JhPMNGD3WEugoJVUMZF6UCXdGjUYckEWjS+6jyc5h4NEHN X-Received: by 2002:a63:3444:: with SMTP id b65-v6mr9232491pga.396.1530200767958; Thu, 28 Jun 2018 08:46:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200767; cv=none; d=google.com; s=arc-20160816; b=xN5HN1rDeaM6Muaklkc8ol2PJ+5yifPQdweLjpGO233n0ca5msuLVnyHTiXBwtAIYd ONTJ9fugmRzoTL4iNlw2BM5e3iFSNZaqSPOUF0U7mo3pEBmGvDOUtAaw/xouGIRgBlVk fmAimagQonuM+qq90n3f3BKRH0Ic75LymsOcVRZMqsdSzAYdc9+HRGPVrC81sR8p9yBh KlE/z7HB1gD9P1b8g10wkPDi5XTE0wnGyY9EYWTaKhLkRHIwDLM4RUlQUedKognqYLnA W0GleL/mVLnddf1zSU/8ZURRgph5bJj9jsPCKciN8F/D2+GRXT1edkKPNgnouPPLIxwE mIfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=D5Qny5qPkCUDDQBYNsLYT5W10m4bYLoJk8RUhx4horE=; b=NUcW5uzowPbfJ5sZWKFV8jZZWSyrJW3WyxR9Gffqd7eMb16PD+R4LLJmMOzxzh+WAh OkdFNq3oEGTjCgZatdw/+1DP3s+LUwxsPMvCVRZz4Uz/W/yGX6PZmSuSAmuO30wC6TkE 2GWjvHbdQ2p/qIfR2ZcyoqLquf0d5QRB2iKze+gfefE5erlPRVmYvsN/hS9/bGnBdEY6 NynzpqDQYlPyBDKEVXNtQ8bsY4TqwH/B2OwLpEXp08mrJWaTPA6rDIorZeu54N/9GPsA XJWY/TiVdDS1TPVpdAFhGmuWed8BiNynE0KfCHKilPhqrkCtUcgUMVAv4YR7qs69WvNR +TsA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AeOeLXaO; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f71-v6si7076382pfc.316.2018.06.28.08.46.07; Thu, 28 Jun 2018 08:46:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AeOeLXaO; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935474AbeF1PqG (ORCPT + 31 others); Thu, 28 Jun 2018 11:46:06 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:40739 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1030214AbeF1Pph (ORCPT ); Thu, 28 Jun 2018 11:45:37 -0400 Received: by mail-wm0-f66.google.com with SMTP id z13-v6so9590943wma.5 for ; Thu, 28 Jun 2018 08:45:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=D5Qny5qPkCUDDQBYNsLYT5W10m4bYLoJk8RUhx4horE=; b=AeOeLXaO0HkF+rsvuOrXrJbqXOHKiqi5FRP39VMPdR/9PGh+DLz3QaklmEWl59nMb5 bX8+O7tYJqwVyDzzxBSCi1OevO0lrp7r/mTSBSjGoXbq32mjSm1RavQNkwokRZ8pEYd3 2lAzCn9oVtH2s806qcXKtjPy5LzTI5R8hOCFM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=D5Qny5qPkCUDDQBYNsLYT5W10m4bYLoJk8RUhx4horE=; b=sUNvprNl2Ib13Tz2llczbf0drup3o9CFsOVyG2V7TRUkCp3u/WJXw17vXBWLQSU8qj +p8aOc3aDSnOC8+hIrm8aqZP4WmfjVmc3CRs/qo2HPaaBUEBPZCLRnlCyOsHNKFP4X4T XW1qX6SVZWI/c/CFbUp0R1uo3/vPmnUzVlHCQJOuYmHbHrYrcI3uQrWkNWtv738iwG0y BLvPmLmszRYeI6xcCXEajdZNjreA03WuduzG+XnDc26kMYv4orsChvOJqtImYOu2YGRN jhS70nOqyvLKcCYSL6PyHvIz3YZp04gppT8fJLPPKOI8cOQIh0fZnoh5DMSfWmWjRGug mgsQ== X-Gm-Message-State: APt69E1T3QuOvy95KrvTmL0r+0x6uQ6x5MIqe/eoCawekxTnAce0iFbL 6CgluZ7fxxSJI5rbWlJ8Xl9ILw== X-Received: by 2002:a1c:5d0e:: with SMTP id r14-v6mr8384763wmb.152.1530200736285; Thu, 28 Jun 2018 08:45:36 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:35 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar Subject: [PATCH 10/11] sched: remove rt_avg code Date: Thu, 28 Jun 2018 17:45:13 +0200 Message-Id: <1530200714-4504-11-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org rt_avg is no more used anywhere so we can remove all related code Cc: Ingo Molnar Cc: Peter Zijlstra Signed-off-by: Vincent Guittot --- kernel/sched/core.c | 26 -------------------------- kernel/sched/fair.c | 2 -- kernel/sched/sched.h | 17 ----------------- 3 files changed, 45 deletions(-) -- 2.7.4 diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e5263a4..e9aae7f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -652,23 +652,6 @@ bool sched_can_stop_tick(struct rq *rq) return true; } #endif /* CONFIG_NO_HZ_FULL */ - -void sched_avg_update(struct rq *rq) -{ - s64 period = sched_avg_period(); - - while ((s64)(rq_clock(rq) - rq->age_stamp) > period) { - /* - * Inline assembly required to prevent the compiler - * optimising this loop into a divmod call. - * See __iter_div_u64_rem() for another example of this. - */ - asm("" : "+rm" (rq->age_stamp)); - rq->age_stamp += period; - rq->rt_avg /= 2; - } -} - #endif /* CONFIG_SMP */ #if defined(CONFIG_RT_GROUP_SCHED) || (defined(CONFIG_FAIR_GROUP_SCHED) && \ @@ -5719,13 +5702,6 @@ void set_rq_offline(struct rq *rq) } } -static void set_cpu_rq_start_time(unsigned int cpu) -{ - struct rq *rq = cpu_rq(cpu); - - rq->age_stamp = sched_clock_cpu(cpu); -} - /* * used to mark begin/end of suspend/resume: */ @@ -5843,7 +5819,6 @@ static void sched_rq_cpu_starting(unsigned int cpu) int sched_cpu_starting(unsigned int cpu) { - set_cpu_rq_start_time(cpu); sched_rq_cpu_starting(cpu); sched_tick_start(cpu); return 0; @@ -6111,7 +6086,6 @@ void __init sched_init(void) #ifdef CONFIG_SMP idle_thread_set_boot_cpu(); - set_cpu_rq_start_time(smp_processor_id()); #endif init_sched_fair_class(); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ce0dcbf..7ddb13a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5322,8 +5322,6 @@ static void cpu_load_update(struct rq *this_rq, unsigned long this_load, this_rq->cpu_load[i] = (old_load * (scale - 1) + new_load) >> i; } - - sched_avg_update(this_rq); } /* Used instead of source_load when we know the type == 0 */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 59a633d..c71ea81 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -846,8 +846,6 @@ struct rq { struct list_head cfs_tasks; - u64 rt_avg; - u64 age_stamp; struct sched_avg avg_rt; struct sched_avg avg_dl; #if defined(CONFIG_IRQ_TIME_ACCOUNTING) || defined(CONFIG_PARAVIRT_TIME_ACCOUNTING) @@ -1712,11 +1710,6 @@ extern const_debug unsigned int sysctl_sched_time_avg; extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; -static inline u64 sched_avg_period(void) -{ - return (u64)sysctl_sched_time_avg * NSEC_PER_MSEC / 2; -} - #ifdef CONFIG_SCHED_HRTICK /* @@ -1753,8 +1746,6 @@ unsigned long arch_scale_freq_capacity(int cpu) #endif #ifdef CONFIG_SMP -extern void sched_avg_update(struct rq *rq); - #ifndef arch_scale_cpu_capacity static __always_inline unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu) @@ -1765,12 +1756,6 @@ unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu) return SCHED_CAPACITY_SCALE; } #endif - -static inline void sched_rt_avg_update(struct rq *rq, u64 rt_delta) -{ - rq->rt_avg += rt_delta * arch_scale_freq_capacity(cpu_of(rq)); - sched_avg_update(rq); -} #else #ifndef arch_scale_cpu_capacity static __always_inline @@ -1779,8 +1764,6 @@ unsigned long arch_scale_cpu_capacity(void __always_unused *sd, int cpu) return SCHED_CAPACITY_SCALE; } #endif -static inline void sched_rt_avg_update(struct rq *rq, u64 rt_delta) { } -static inline void sched_avg_update(struct rq *rq) { } #endif struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf) From patchwork Thu Jun 28 15:45:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 140456 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2343078ljj; Thu, 28 Jun 2018 08:45:45 -0700 (PDT) X-Google-Smtp-Source: AAOMgpcDxNMRbTxGLyr3Rrd1HVnkwbBWLfawAnsAsGp749n8dFluXk8FPsaPOK8YqjQ7hqr3vywQ X-Received: by 2002:a62:211a:: with SMTP id h26-v6mr10630141pfh.133.1530200745844; Thu, 28 Jun 2018 08:45:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530200745; cv=none; d=google.com; s=arc-20160816; b=x2kDaLvBmcOR8uyjVHVSMwIq5Z4oBiGRoyll+dAnE22FmQByEmkWe7axzU1z/BORP6 /sqmpYKD7G3VabObw2Ely2UCSvaTzBlirfYl1Y/3X37DPbRhRI5bQugbhZ22WEotexcL OXLZUlF/1ixxVj5ekN8eVLy4aM9VNinmVd8pjxiVHE1+ZQBxeL9Q5qz4H3+RiTNSMSWj fhxHXO6QEMdV5yCTNyOygzRJ3QI/liyIFdqur+qSJlPQSIYpwh4a9c5kUxe/zWnpGqVU sAdCN8dUBJZcOGMDRxxnv4bndA13B+rVw164LeUD+SIccoQEKHdHs9s1HOzwxeptsFWO QkaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=JJYZdlqUxsEPbpnWuEmMYmdnaFoz3y9OMFnGg+MT9Uw=; b=ja1I7SrZS4UJbkjM/YaDcJpTAzUG31hZomKtnX6p9zUYPFIaIWwQoish/BJ5jdws8f 3PNMA+5zhyJRKeBtdx9zYoSKSf4xlNGAvwLoaNcZmJE5a9xRsN7bl+fT7vbZRPa5casj CD7boZx8aSp2V+b11LFiUguIOx6pFMleqQHW9WbfkDHJoNCzNHeFI7SVKO05rsuHY6a8 Ul3ZsenaIx3zEVAIT4t5TiBivS0pURUg1wRkXBvLg8DL/RA+auPSMK38GIVA2ENQSbnl mcJVjJCjJ3ulwKRjWWLyKGbpGoJVP85nnApH79jVqtAlJZ2nu40TaxuU2l5y8POcJCJB UFyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WJGMcBNq; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q76-v6si6501234pgq.597.2018.06.28.08.45.45; Thu, 28 Jun 2018 08:45:45 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WJGMcBNq; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1030245AbeF1Ppn (ORCPT + 31 others); Thu, 28 Jun 2018 11:45:43 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:46569 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1030230AbeF1Ppj (ORCPT ); Thu, 28 Jun 2018 11:45:39 -0400 Received: by mail-wr0-f193.google.com with SMTP id t6-v6so5987972wrq.13 for ; Thu, 28 Jun 2018 08:45:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=JJYZdlqUxsEPbpnWuEmMYmdnaFoz3y9OMFnGg+MT9Uw=; b=WJGMcBNqZMU7HTVsVxDfPiQLDJVd5BAZLNqCo4GszX70olJDn87dGpOPzCdn8a6RxO wyJBCKOO87/Xvs+CVuBeWAcNtNJLfKmo0FOr9YouWH+ol4xccWG3Az5JwVghw7FhOp5a k/Mj/SlXXPUc8asrxXzva7cFdVdSYL6ByO+/s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=JJYZdlqUxsEPbpnWuEmMYmdnaFoz3y9OMFnGg+MT9Uw=; b=gSe1DQCF8oSkyrtnT2np5TH30f7LwGZFxUIpgBxGmfvi1O1e5yLP+w9vU8Kd7oz+1d MQg8fOJZrNGNGrlyCB8ieVRlBTIsjKvlI/sgQYoruPAEmXOXRiD9ePS0pJxLjxVe2f7s xZfgjqYtLaVa2fAWdIBudu9qzbnaugY3KRdExYedX32/q1wiYT1heEvzUBvWbgXeIv7O 32kLRrvExUr7W2soeGhgFUCJqHRpwelc/VgE1Xt4pQ6k4UYYnh4L5XdDzXx5T4u+1WrQ c93Bf/uO8BBsL6NDtAQFWdT8GTunGMOdGe8DCUm1rbFnmXFtwZV3+OBcGb5o/GES/4QB UNaA== X-Gm-Message-State: APt69E3KsRjU3LfzI0yh93qXsLB/EFc6vKQYfx3EmwoDrYbLTu8QVBpL CGHMfIzte0Tfzdoyp28oAz/CBA== X-Received: by 2002:adf:9c12:: with SMTP id f18-v6mr9353815wrc.40.1530200737983; Thu, 28 Jun 2018 08:45:37 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:21c3:ec41:bec9:c38]) by smtp.gmail.com with ESMTPSA id i4-v6sm6202115wrq.28.2018.06.28.08.45.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Jun 2018 08:45:37 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: rjw@rjwysocki.net, juri.lelli@redhat.com, dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, viresh.kumar@linaro.org, valentin.schneider@arm.com, patrick.bellasi@arm.com, joel@joelfernandes.org, daniel.lezcano@linaro.org, quentin.perret@arm.com, luca.abeni@santannapisa.it, claudio@evidence.eu.com, Vincent Guittot , Ingo Molnar , Kees Cook , "Luis R. Rodriguez" Subject: [PATCH 11/11] proc/sched: remove unused sched_time_avg_ms Date: Thu, 28 Jun 2018 17:45:14 +0200 Message-Id: <1530200714-4504-12-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> References: <1530200714-4504-1-git-send-email-vincent.guittot@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org /proc/sys/kernel/sched_time_avg_ms entry is not used anywhere. Remove it Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Kees Cook Cc: "Luis R. Rodriguez" Signed-off-by: Vincent Guittot --- include/linux/sched/sysctl.h | 1 - kernel/sched/core.c | 8 -------- kernel/sched/sched.h | 1 - kernel/sysctl.c | 8 -------- 4 files changed, 18 deletions(-) -- 2.7.4 Reviewed-by: Luis R. Rodriguez diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h index 1c1a151..913488d 100644 --- a/include/linux/sched/sysctl.h +++ b/include/linux/sched/sysctl.h @@ -40,7 +40,6 @@ extern unsigned int sysctl_numa_balancing_scan_size; #ifdef CONFIG_SCHED_DEBUG extern __read_mostly unsigned int sysctl_sched_migration_cost; extern __read_mostly unsigned int sysctl_sched_nr_migrate; -extern __read_mostly unsigned int sysctl_sched_time_avg; int sched_proc_update_handler(struct ctl_table *table, int write, void __user *buffer, size_t *length, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e9aae7f..6935691 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -48,14 +48,6 @@ const_debug unsigned int sysctl_sched_features = const_debug unsigned int sysctl_sched_nr_migrate = 32; /* - * period over which we average the RT time consumption, measured - * in ms. - * - * default: 1s - */ -const_debug unsigned int sysctl_sched_time_avg = MSEC_PER_SEC; - -/* * period over which we measure -rt task CPU usage in us. * default: 1s */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index c71ea81..47b9175 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1706,7 +1706,6 @@ extern void deactivate_task(struct rq *rq, struct task_struct *p, int flags); extern void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags); -extern const_debug unsigned int sysctl_sched_time_avg; extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 2d9837c..f22f76b 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -368,14 +368,6 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = proc_dointvec, }, - { - .procname = "sched_time_avg_ms", - .data = &sysctl_sched_time_avg, - .maxlen = sizeof(unsigned int), - .mode = 0644, - .proc_handler = proc_dointvec_minmax, - .extra1 = &one, - }, #ifdef CONFIG_SCHEDSTATS { .procname = "sched_schedstats",