From patchwork Mon Apr 10 09:18:29 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 97066 Delivered-To: patch@linaro.org Received: by 10.182.246.10 with SMTP id xs10csp1266400obc; Mon, 10 Apr 2017 02:18:53 -0700 (PDT) X-Received: by 10.99.52.202 with SMTP id b193mr53855726pga.131.1491815933500; Mon, 10 Apr 2017 02:18:53 -0700 (PDT) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r186si13088297pgr.193.2017.04.10.02.18.53; Mon, 10 Apr 2017 02:18:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752807AbdDJJSj (ORCPT + 24 others); Mon, 10 Apr 2017 05:18:39 -0400 Received: from mail-wr0-f175.google.com ([209.85.128.175]:33180 "EHLO mail-wr0-f175.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751968AbdDJJSi (ORCPT ); Mon, 10 Apr 2017 05:18:38 -0400 Received: by mail-wr0-f175.google.com with SMTP id l28so14924040wre.0 for ; Mon, 10 Apr 2017 02:18:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=jq7rUAO4H6k1VAIIWrc9aCgh36HZd1lEgG6RP/ohVgk=; b=DwFwog0uoFIV71pcqk7QZejayFUbxGRxUcQVxKS5k4YshoU8um3x9Yjb3YjrfO33R7 WmaK5sbUz5cjDVyyPzqQYkn1RhTeMo52ri5031XZ7UfT3gnWR9s6L5y/VcpIJ0qrRsVw JDzuhL/1xdFAorfoLhtw/4GcDiEN4FxF0ZS+Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=jq7rUAO4H6k1VAIIWrc9aCgh36HZd1lEgG6RP/ohVgk=; b=Q8n+vDIA2RnddTXoIkpd5RcKZBUVMwpX1a8iIxRE4r9ZgV+CWT0slElDJIC/yTWfc0 ooHZp6qcaTajs9zN+e+Cr8Sdh4QEfL2wnXBAEDHhicQKLNLGz60OZGd7PTtPvH0K8R8I EA1wcnlgOvwG36asCTZotlJzKgj/n+iuoMOpNnZ1EZk8Evf94Efi0mAiYGOmcswgC4Sm Q3FEB0z37+X6LFstJvVuT36Da22WFq05tCxaGHnD40Sd3qqatJSp1ooBLXeIo5BPBn0Y /Uaai2OqD2kfiU/fKKdE8H606Yx310uSwT3qql2sYZy3qkNvoyblvH01/51xioXcM4yy Ob7g== X-Gm-Message-State: AFeK/H1kz2VrQ4omzU5aO9a037zaH2vvBjxZhKZWPC8I7QWo9daW+rlGIGihr3TWGybMbm7j X-Received: by 10.223.176.13 with SMTP id f13mr29204664wra.124.1491815916740; Mon, 10 Apr 2017 02:18:36 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:a906:3da3:1c98:59b7]) by smtp.gmail.com with ESMTPSA id y11sm16525264wrd.22.2017.04.10.02.18.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 10 Apr 2017 02:18:36 -0700 (PDT) From: Vincent Guittot To: peterz@infradead.org, mingo@kernel.org, linux-kernel@vger.kernel.org Cc: dietmar.eggemann@arm.com, Morten.Rasmussen@arm.com, yuyang.du@intel.com, pjt@google.com, bsegall@google.com, Vincent Guittot Subject: [PATCH v2] sched/fair: update scale invariance of PELT Date: Mon, 10 Apr 2017 11:18:29 +0200 Message-Id: <1491815909-13345-1-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.7.4 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The current implementation of load tracking invariance scales the contribution with current frequency and uarch performance (only for utilization) of the CPU. One main result of this formula is that the figures are capped by current capacity of CPU. Another one is that the load_avg is not invariant because not scaled with uarch. The util_avg of a periodic task that runs r time slots every p time slots varies in the range : U * (1-y^r)/(1-y^p) * y^i < Utilization < U * (1-y^r)/(1-y^p) with U is the max util_avg value = SCHED_CAPACITY_SCALE At a lower capacity, the range becomes: U * C * (1-y^r')/(1-y^p) * y^i' < Utilization < U * C * (1-y^r')/(1-y^p) with C reflecting the compute capacity ratio between current capacity and max capacity. so C tries to compensate changes in (1-y^r') but it can't be accurate. Instead of scaling the contribution value of PELT algo, we should scale the running time. The PELT signal aims to track the amount of computation of tasks and/or rq so it seems more correct to scale the running time to reflect the effective amount of computation done since the last update. In order to be fully invariant, we need to apply the same amount of running time and idle time whatever the current capacity. Because running at lower capacity implies that the task will run longer, we have to track the amount of "stolen" idle time and to apply it when task becomes idle. But once we have reached the maximum utilization value (SCHED_CAPACITY_SCALE), it means that the task is seen as an always-running task whatever the capacity of the cpu (even at max compute capacity). In this case, we can discard the "stolen" idle times which becomes meaningless. In order to cope with rounding effect of PELT algo we take a margin and consider task with utilization greater than 1000 (vs 1024 max) as an always-running task. Then, we can use the same algorithm for both utilization and load and simplify __update_load_avg now that the load of a task doesn't have to be capped by CPU uarch. The responsivness of PELT is improved when CPU is not running at max capacity with this new algorithm. I have put below some examples of duration to reach some typical load values according to the capacity of the CPU with current implementation and with this patch. Util (%) max capacity half capacity(mainline) half capacity(w/ patch) 972 (95%) 138ms not reachable 276ms 486 (47.5%) 30ms 138ms 60ms 256 (25%) 13ms 32ms 26ms On my hikey (octo ARM platform) with schedutil governor, the time to reach max OPP when starting from a null utilization, decreases from 223ms with current scale invariance down to 121ms with the new algorithm. For this test, i have enable arch_scale_freq for arm64. Signed-off-by: Vincent Guittot --- Update since v1: - rebase on latest tip/sched/core which includes "Optimize __update_sched_avg()" + patch : https://lkml.org/lkml/2017/3/31/308 include/linux/sched.h | 1 + kernel/sched/fair.c | 53 ++++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 47 insertions(+), 7 deletions(-) -- 2.7.4 diff --git a/include/linux/sched.h b/include/linux/sched.h index d67eee8..ca9d00f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -313,6 +313,7 @@ struct load_weight { */ struct sched_avg { u64 last_update_time; + u64 stolen_idle_time; u64 load_sum; u32 util_sum; u32 period_contrib; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1e5f580..b6f4253 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -734,6 +734,7 @@ void init_entity_runnable_average(struct sched_entity *se) struct sched_avg *sa = &se->avg; sa->last_update_time = 0; + sa->stolen_idle_time = 0; /* * sched_avg's period_contrib should be strictly less then 1024, so * we give it 1023 to make sure it is almost a period (1024us), and @@ -2819,15 +2820,12 @@ static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3) * n=1 */ static __always_inline u32 -accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, +accumulate_sum(u64 delta, struct sched_avg *sa, unsigned long weight, int running, struct cfs_rq *cfs_rq) { - unsigned long scale_freq, scale_cpu; u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */ u64 periods; - scale_freq = arch_scale_freq_capacity(NULL, cpu); - scale_cpu = arch_scale_cpu_capacity(NULL, cpu); delta += sa->period_contrib; periods = delta / 1024; /* A period is 1024us (~1ms) */ @@ -2852,19 +2850,54 @@ accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, } sa->period_contrib = delta; - contrib = cap_scale(contrib, scale_freq); if (weight) { sa->load_sum += weight * contrib; if (cfs_rq) cfs_rq->runnable_load_sum += weight * contrib; } if (running) - sa->util_sum += contrib * scale_cpu; + sa->util_sum += contrib << SCHED_CAPACITY_SHIFT; return periods; } /* + * Scale the time to reflect the effective amount of computation done during + * this delta time. + */ +static __always_inline u64 +scale_time(u64 delta, int cpu, struct sched_avg *sa, + unsigned long weight, int running) +{ + if (running) { + sa->stolen_idle_time += delta; + /* + * scale the elapsed time to reflect the real amount of + * computation + */ + delta = cap_scale(delta, arch_scale_freq_capacity(NULL, cpu)); + delta = cap_scale(delta, arch_scale_cpu_capacity(NULL, cpu)); + + /* + * Track the amount of stolen idle time due to running at + * lower capacity + */ + sa->stolen_idle_time -= delta; + } else if (!weight) { + if (sa->util_sum < (LOAD_AVG_MAX * 1000)) { + /* + * Add the idle time stolen by running at lower compute + * capacity + */ + delta += sa->stolen_idle_time; + } + sa->stolen_idle_time = 0; + } + + return delta; +} + +/* * We can represent the historical contribution to runnable average as the * coefficients of a geometric series. To do this we sub-divide our runnable * history into segments of approximately 1ms (1024us); label the segment that @@ -2918,13 +2951,19 @@ ___update_load_avg(u64 now, int cpu, struct sched_avg *sa, sa->last_update_time = now; /* + * Scale time to reflect the amount a computation effectively done + * during the time slot at current capacity + */ + delta = scale_time(delta, cpu, sa, weight, running); + + /* * Now we know we crossed measurement unit boundaries. The *_avg * accrues by two steps: * * Step 1: accumulate *_sum since last_update_time. If we haven't * crossed period boundaries, finish. */ - if (!accumulate_sum(delta, cpu, sa, weight, running, cfs_rq)) + if (!accumulate_sum(delta, sa, weight, running, cfs_rq)) return 0; /*