From patchwork Fri Aug 7 09:50:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juri Lelli X-Patchwork-Id: 257112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3E79C433E0 for ; Fri, 7 Aug 2020 09:51:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8B35F22CAE for ; Fri, 7 Aug 2020 09:51:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GUxBZ1Ub" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728257AbgHGJvb (ORCPT ); Fri, 7 Aug 2020 05:51:31 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:24440 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728219AbgHGJv1 (ORCPT ); Fri, 7 Aug 2020 05:51:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1596793885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b/HF9RVTQ+ZTIh7ud0STiu49WCNPHw9YdlpSFuMKv58=; b=GUxBZ1UbxP72qFOWl/Yd90zbe78sFS+sFKgFD2KIj07QJg8f5subPsFD+TKIHikkysjjZF PPGYHBekTxXslSUUqjRccsupO717LgLG92QeO3tciwebmS0Xe+zsGN5mlTh+BEBOKB0nLI v8qMf1vvvCmH3Cn5SWyrWMqRTUpNRQs= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-276-B0bmQOASPki7Sa2aZDliww-1; Fri, 07 Aug 2020 05:51:22 -0400 X-MC-Unique: B0bmQOASPki7Sa2aZDliww-1 Received: by mail-wr1-f69.google.com with SMTP id m7so534067wrb.20 for ; Fri, 07 Aug 2020 02:51:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=b/HF9RVTQ+ZTIh7ud0STiu49WCNPHw9YdlpSFuMKv58=; b=JmNVuDGo3zrAlpOtQoO+yVYETTtu2fCpU6dZNoIYyXH3hyjy+iS/YG+PqAxsUu+FU+ bwP1xKyWiLW7wToiyJL5Xy/TxC+f4yZ/iZ8GtG70SYalT9r3exgIjCiChUFSfmg714As ZIUEui6QQPZRBbxT16CNQE9IUxmjjnzuD9Z+ewKKV0WgPIVpJ0wliviCSCw9hBeOkKVa lRCRRN3pQ+aUc0VtQcXffPNtPefaBu94lVQ1k0t9NQbTSG8nxYOtYJhCEXBBpKwcYcGj 7u8l6oJw10AQ7P1mjmiKTwqYJiVrCPEeSR8j0IHFQLi0HHc8bL3gdERJJGqpKj4N3E6J Td8g== X-Gm-Message-State: AOAM53060gGB+FsIpgA9mN84LQH7Lcn94HyZbKkdBUctAunaFao+gLg6 7s0NW+LCaLmT9ifmWTY5k8txekaHSAAgcYOm7GPE1uC4HJKfztoHMXjcY2K6WgWVHPw3N5pGyer +p2qRh7wlgKaYneRBdAAXaEyRES0= X-Received: by 2002:adf:eb89:: with SMTP id t9mr11182824wrn.65.1596793880896; Fri, 07 Aug 2020 02:51:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxVWq6bepkCG4H0mARF8/cqSTRwKYKa0bwk+v75Q1KrkoLYz1ALxZjp2ycz7XCelnea3DAEGA== X-Received: by 2002:adf:eb89:: with SMTP id t9mr11182803wrn.65.1596793880638; Fri, 07 Aug 2020 02:51:20 -0700 (PDT) Received: from localhost.localdomain.com ([151.29.36.84]) by smtp.gmail.com with ESMTPSA id h14sm9388396wml.30.2020.08.07.02.51.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Aug 2020 02:51:19 -0700 (PDT) From: Juri Lelli To: peterz@infradead.org, mingo@redhat.com Cc: rostedt@goodmis.org, tglx@linutronix.de, linux-kernel@vger.kernel.org, luca.abeni@santannapisa.it, tommaso.cucinotta@santannapisa.it, alessio.balsini@gmail.com, bristot@redhat.com, dietmar.eggemann@arm.com, linux-rt-users@vger.kernel.org, mtosatti@redhat.com, williams@redhat.com, valentin.schneider@arm.com Subject: [RFC PATCH v2 1/6] sched: Unify runtime accounting across classes Date: Fri, 7 Aug 2020 11:50:46 +0200 Message-Id: <20200807095051.385985-2-juri.lelli@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200807095051.385985-1-juri.lelli@redhat.com> References: <20200807095051.385985-1-juri.lelli@redhat.com> MIME-Version: 1.0 Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org From: Peter Zijlstra All classes use sched_entity::exec_start to track runtime and have copies of the exact same code around to compute runtime. Collapse all that. Signed-off-by: Peter Zijlstra (Intel) --- include/linux/sched.h | 2 +- kernel/sched/deadline.c | 17 +++----------- kernel/sched/fair.c | 50 +++++++++++++++++++++++++++++++--------- kernel/sched/rt.c | 17 +++----------- kernel/sched/sched.h | 2 ++ kernel/sched/stop_task.c | 16 +------------ 6 files changed, 49 insertions(+), 55 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index a6bf77c346876..f7b9ba04970bc 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -424,7 +424,7 @@ struct sched_statistics { u64 block_start; u64 block_max; - u64 exec_max; + s64 exec_max; u64 slice_max; u64 nr_migrations_cold; diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 3862a28cd05d0..2ece83b5991f5 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -1221,9 +1221,8 @@ static void update_curr_dl(struct rq *rq) { struct task_struct *curr = rq->curr; struct sched_dl_entity *dl_se = &curr->dl; - u64 delta_exec, scaled_delta_exec; + s64 delta_exec, scaled_delta_exec; int cpu = cpu_of(rq); - u64 now; if (!dl_task(curr) || !on_dl_rq(dl_se)) return; @@ -1236,23 +1235,13 @@ static void update_curr_dl(struct rq *rq) * natural solution, but the full ramifications of this * approach need further study. */ - now = rq_clock_task(rq); - delta_exec = now - curr->se.exec_start; - if (unlikely((s64)delta_exec <= 0)) { + delta_exec = update_curr_common(rq); + if (unlikely(delta_exec <= 0)) { if (unlikely(dl_se->dl_yielded)) goto throttle; return; } - schedstat_set(curr->se.statistics.exec_max, - max(curr->se.statistics.exec_max, delta_exec)); - - curr->se.sum_exec_runtime += delta_exec; - account_group_exec_runtime(curr, delta_exec); - - curr->se.exec_start = now; - cgroup_account_cputime(curr, delta_exec); - if (dl_entity_is_special(dl_se)) return; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 2ba8f230feb9a..10a230d85104a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -836,30 +836,58 @@ static void update_tg_load_avg(struct cfs_rq *cfs_rq, int force) } #endif /* CONFIG_SMP */ +static s64 update_curr_se(struct rq *rq, struct sched_entity *curr) +{ + u64 now = rq_clock_task(rq); + s64 delta_exec; + + delta_exec = now - curr->exec_start; + if (unlikely(delta_exec <= 0)) + return delta_exec; + + curr->exec_start = now; + curr->sum_exec_runtime += delta_exec; + + schedstat_set(curr->statistics.exec_max, + max(delta_exec, curr->statistics.exec_max)); + + return delta_exec; +} + +/* + * Used by other classes to account runtime. + */ +s64 update_curr_common(struct rq *rq) +{ + struct task_struct *curr = rq->curr; + s64 delta_exec; + + delta_exec = update_curr_se(rq, &curr->se); + if (unlikely(delta_exec <= 0)) + return delta_exec; + + account_group_exec_runtime(curr, delta_exec); + cgroup_account_cputime(curr, delta_exec); + + return delta_exec; +} + /* * Update the current task's runtime statistics. */ static void update_curr(struct cfs_rq *cfs_rq) { struct sched_entity *curr = cfs_rq->curr; - u64 now = rq_clock_task(rq_of(cfs_rq)); - u64 delta_exec; + s64 delta_exec; if (unlikely(!curr)) return; - delta_exec = now - curr->exec_start; - if (unlikely((s64)delta_exec <= 0)) + delta_exec = update_curr_se(rq_of(cfs_rq), curr); + if (unlikely(delta_exec <= 0)) return; - curr->exec_start = now; - - schedstat_set(curr->statistics.exec_max, - max(delta_exec, curr->statistics.exec_max)); - - curr->sum_exec_runtime += delta_exec; schedstat_add(cfs_rq->exec_clock, delta_exec); - curr->vruntime += calc_delta_fair(delta_exec, curr); update_min_vruntime(cfs_rq); diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index f215eea6a9661..196171fbf5978 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -997,26 +997,15 @@ static void update_curr_rt(struct rq *rq) { struct task_struct *curr = rq->curr; struct sched_rt_entity *rt_se = &curr->rt; - u64 delta_exec; - u64 now; + s64 delta_exec; if (curr->sched_class != &rt_sched_class) return; - now = rq_clock_task(rq); - delta_exec = now - curr->se.exec_start; - if (unlikely((s64)delta_exec <= 0)) + delta_exec = update_curr_common(rq); + if (unlikely(delta_exec < 0)) return; - schedstat_set(curr->se.statistics.exec_max, - max(curr->se.statistics.exec_max, delta_exec)); - - curr->se.sum_exec_runtime += delta_exec; - account_group_exec_runtime(curr, delta_exec); - - curr->se.exec_start = now; - cgroup_account_cputime(curr, delta_exec); - if (!rt_bandwidth_enabled()) return; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 3fd283892761d..963c16fc27500 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1769,6 +1769,8 @@ extern const u32 sched_prio_to_wmult[40]; #define RETRY_TASK ((void *)-1UL) +extern s64 update_curr_common(struct rq *rq); + struct sched_class { #ifdef CONFIG_UCLAMP_TASK diff --git a/kernel/sched/stop_task.c b/kernel/sched/stop_task.c index 394bc8126a1e5..1eb1e336e18e7 100644 --- a/kernel/sched/stop_task.c +++ b/kernel/sched/stop_task.c @@ -62,21 +62,7 @@ static void yield_task_stop(struct rq *rq) static void put_prev_task_stop(struct rq *rq, struct task_struct *prev) { - struct task_struct *curr = rq->curr; - u64 delta_exec; - - delta_exec = rq_clock_task(rq) - curr->se.exec_start; - if (unlikely((s64)delta_exec < 0)) - delta_exec = 0; - - schedstat_set(curr->se.statistics.exec_max, - max(curr->se.statistics.exec_max, delta_exec)); - - curr->se.sum_exec_runtime += delta_exec; - account_group_exec_runtime(curr, delta_exec); - - curr->se.exec_start = rq_clock_task(rq); - cgroup_account_cputime(curr, delta_exec); + update_curr_common(rq); } /* From patchwork Fri Aug 7 09:50:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juri Lelli X-Patchwork-Id: 257110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22B60C433E3 for ; Fri, 7 Aug 2020 09:51:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E3EE42177B for ; Fri, 7 Aug 2020 09:51:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="jIXlkLlw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728234AbgHGJva (ORCPT ); Fri, 7 Aug 2020 05:51:30 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:36045 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728216AbgHGJv1 (ORCPT ); Fri, 7 Aug 2020 05:51:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1596793885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zjAjRBZqRUxOSD09O8un2EvyPI/eihDhik4FjIAp3oQ=; b=jIXlkLlw55FP7kdAYyMxixzWpuh+5+j9F3A97xC0mZpseD9UIrXVqX+brH9KUv3zAwZFwz RlgFzAcVylBWaDSgqrASsHzhdcSZemn5t+w+xlMKY8a7rz2/slM45H7MHC9T+7VqOaeQ1s JjLLmORuKvcHI5beHXJIVvuhA2w2hhk= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-280-IKUvwA4vMciWk9gBc_EYyw-1; Fri, 07 Aug 2020 05:51:24 -0400 X-MC-Unique: IKUvwA4vMciWk9gBc_EYyw-1 Received: by mail-wm1-f71.google.com with SMTP id h7so573808wmm.7 for ; Fri, 07 Aug 2020 02:51:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zjAjRBZqRUxOSD09O8un2EvyPI/eihDhik4FjIAp3oQ=; b=sSXBV5jG12Wmj11hLqkDCUaQH+sciUqS3zVuq3DOCzuE8bS1uKtZcWh8EcDLAUqpZM RakfYaXG68Y+IbbcXutEZk9tiRqbMXMEOTj6OZq7TVY7ypiHiJu280E7s3wi0BLbtfhr ooD5M7vdY8WzRXWVP31R4FNLMYpE9JJcsu+1wc+K4lV2NOB6R+DeO/RYT0TgQDyrBBqB BDA0Flz31jED64CoMu2J5COCEaMYucAje6uRN8VN6Y37MOI6vZ9HvYr7InTuc0ZZSGl6 CzQbP828TEw9xoMAD8bN9gixYtrIBSdfxdafcGCeztQLUPdCZwrBseGnK3yRw925O7Yh mIoA== X-Gm-Message-State: AOAM531SSyc5ZruR1gSobmxi08HDnVGclfeNHDrvw3UHgJ1w0gXYE2y2 hTkVRc0azJrC0LZlmZ2WXspRs+DmyiN2aceiLULq/NxyoONOVlDloPRM6JnBWMJ4uSE2XcVnT4d DpQrgBwdiJKvABODP7Cz29kginHc= X-Received: by 2002:adf:9c8c:: with SMTP id d12mr11112935wre.369.1596793882750; Fri, 07 Aug 2020 02:51:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzkip5rLEQTz9TI9POQCDYsEa8Kk+JzuIIjQ//8hNa+MzW7XcP0R8ZojIy+dEPc7Ef8cI3L/g== X-Received: by 2002:adf:9c8c:: with SMTP id d12mr11112927wre.369.1596793882525; Fri, 07 Aug 2020 02:51:22 -0700 (PDT) Received: from localhost.localdomain.com ([151.29.36.84]) by smtp.gmail.com with ESMTPSA id h14sm9388396wml.30.2020.08.07.02.51.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Aug 2020 02:51:21 -0700 (PDT) From: Juri Lelli To: peterz@infradead.org, mingo@redhat.com Cc: rostedt@goodmis.org, tglx@linutronix.de, linux-kernel@vger.kernel.org, luca.abeni@santannapisa.it, tommaso.cucinotta@santannapisa.it, alessio.balsini@gmail.com, bristot@redhat.com, dietmar.eggemann@arm.com, linux-rt-users@vger.kernel.org, mtosatti@redhat.com, williams@redhat.com, valentin.schneider@arm.com Subject: [RFC PATCH v2 2/6] sched/deadline: Collect sched_dl_entity initialization Date: Fri, 7 Aug 2020 11:50:47 +0200 Message-Id: <20200807095051.385985-3-juri.lelli@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200807095051.385985-1-juri.lelli@redhat.com> References: <20200807095051.385985-1-juri.lelli@redhat.com> MIME-Version: 1.0 Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org From: Peter Zijlstra Create a single function that initializes a sched_dl_entity. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 5 +---- kernel/sched/deadline.c | 22 +++++++++++++++------- kernel/sched/sched.h | 5 +---- 3 files changed, 17 insertions(+), 15 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 12e1f3a2cabc6..6b36bf82b53c2 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3081,10 +3081,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) memset(&p->se.statistics, 0, sizeof(p->se.statistics)); #endif - RB_CLEAR_NODE(&p->dl.rb_node); - init_dl_task_timer(&p->dl); - init_dl_inactive_task_timer(&p->dl); - __dl_clear_params(p); + init_dl_entity(&p->dl); INIT_LIST_HEAD(&p->rt.run_list); p->rt.timeout = 0; diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 2ece83b5991f5..8d909bdb9a119 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -219,6 +219,8 @@ static void dl_change_utilization(struct task_struct *p, u64 new_bw) __add_rq_bw(new_bw, &rq->dl); } +static void __dl_clear_params(struct sched_dl_entity *dl_se); + /* * The utilization of a task cannot be immediately removed from * the rq active utilization (running_bw) when the task blocks. @@ -317,7 +319,7 @@ static void task_non_contending(struct task_struct *p) sub_rq_bw(&p->dl, &rq->dl); raw_spin_lock(&dl_b->lock); __dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p))); - __dl_clear_params(p); + __dl_clear_params(dl_se); raw_spin_unlock(&dl_b->lock); } @@ -1123,7 +1125,7 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer) return HRTIMER_NORESTART; } -void init_dl_task_timer(struct sched_dl_entity *dl_se) +static void init_dl_task_timer(struct sched_dl_entity *dl_se) { struct hrtimer *timer = &dl_se->dl_timer; @@ -1335,7 +1337,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) raw_spin_lock(&dl_b->lock); __dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p))); raw_spin_unlock(&dl_b->lock); - __dl_clear_params(p); + __dl_clear_params(dl_se); goto unlock; } @@ -1351,7 +1353,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) return HRTIMER_NORESTART; } -void init_dl_inactive_task_timer(struct sched_dl_entity *dl_se) +static void init_dl_inactive_task_timer(struct sched_dl_entity *dl_se) { struct hrtimer *timer = &dl_se->inactive_timer; @@ -2741,10 +2743,8 @@ bool __checkparam_dl(const struct sched_attr *attr) /* * This function clears the sched_dl_entity static params. */ -void __dl_clear_params(struct task_struct *p) +static void __dl_clear_params(struct sched_dl_entity *dl_se) { - struct sched_dl_entity *dl_se = &p->dl; - dl_se->dl_runtime = 0; dl_se->dl_deadline = 0; dl_se->dl_period = 0; @@ -2759,6 +2759,14 @@ void __dl_clear_params(struct task_struct *p) dl_se->dl_overrun = 0; } +void init_dl_entity(struct sched_dl_entity *dl_se) +{ + RB_CLEAR_NODE(&dl_se->rb_node); + init_dl_task_timer(dl_se); + init_dl_inactive_task_timer(dl_se); + __dl_clear_params(dl_se); +} + bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr) { struct sched_dl_entity *dl_se = &p->dl; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 963c16fc27500..62304d4de99cc 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -255,8 +255,6 @@ struct rt_bandwidth { unsigned int rt_period_active; }; -void __dl_clear_params(struct task_struct *p); - /* * To keep the bandwidth of -deadline tasks and groups under control * we need some place where: @@ -1939,8 +1937,7 @@ extern void init_rt_bandwidth(struct rt_bandwidth *rt_b, u64 period, u64 runtime extern struct dl_bandwidth def_dl_bandwidth; extern void init_dl_bandwidth(struct dl_bandwidth *dl_b, u64 period, u64 runtime); -extern void init_dl_task_timer(struct sched_dl_entity *dl_se); -extern void init_dl_inactive_task_timer(struct sched_dl_entity *dl_se); +extern void init_dl_entity(struct sched_dl_entity *dl_se); #define BW_SHIFT 20 #define BW_UNIT (1 << BW_SHIFT) From patchwork Fri Aug 7 09:50:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juri Lelli X-Patchwork-Id: 257111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6684C433E0 for ; Fri, 7 Aug 2020 09:51:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B46B82177B for ; Fri, 7 Aug 2020 09:51:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ccII5+/Z" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728300AbgHGJvg (ORCPT ); Fri, 7 Aug 2020 05:51:36 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:46125 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727820AbgHGJvf (ORCPT ); Fri, 7 Aug 2020 05:51:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1596793892; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z8tgJTs+X+8vEZKUl5jq8+JAws3cxDj6P8K3XwwrUrQ=; b=ccII5+/Za3BwGxsRqqUMJVeGj8lLrijfSF9+VgxgsRGyQRaDZkotKa6pB9GwgnqoXLb6/A 3LNgTAAo0HVC5UXOONvy0otPaNEdRJbCTSll89aVixeeyyvb8D2geVliXSc6Zn2rmccAuM h//O5c28n1C49arfmBLiOJ06aFlY7qI= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-438-C29s-3rNOPCiTuVjJ9zamA-1; Fri, 07 Aug 2020 05:51:31 -0400 X-MC-Unique: C29s-3rNOPCiTuVjJ9zamA-1 Received: by mail-wr1-f71.google.com with SMTP id b13so535322wrq.19 for ; Fri, 07 Aug 2020 02:51:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Z8tgJTs+X+8vEZKUl5jq8+JAws3cxDj6P8K3XwwrUrQ=; b=bXDjtq5E+Hfs2QY5gV5HKYr5OZ3bIM6oZL+c6M/xkbzFy6U0mBxkEu57RBaNvTqeDQ n/lgT+Kh/4NlEhDjdgaDiF4t3nZdofLkhGc2DxihBqbhREQU2UeFLMx5d6Wq9U1/N0WV zAjYkGT9nomMpNsliWHFYEnIKwjHprxaK0M1FhUTYQzGbNdjBJpEuV0YdalPvV6tbXTv 8SjaNeBe3TkfrU1kB21L2DQmsEg63ArVelQwCUF99GcOsUjzqtDWzC4rG52raEbXwbFA mP2QCZ0VOF1C3E2E3kuV0GKHUuOZJ8ZvEb6B6KxEwmd+b4TjPj0gMJ2hD4C5VgKuIWfP RIcg== X-Gm-Message-State: AOAM532HbDIFD53VoML8RYp2AEygxQeMKRgdoMD9WH8igP/Q65f7lIfL bOhEvZnnNmLczPsv1vXTVy3WxY9VQpByLS+WxBWwWYzzlcoiHvmU+bMa0mY5tft42UBJXABSVFG xuXtoWURqG0FcbajXEYujvqmMmNI= X-Received: by 2002:a05:6000:12c1:: with SMTP id l1mr10782164wrx.270.1596793888628; Fri, 07 Aug 2020 02:51:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzoF/IgfG/wrZaT1KmhKuf9rOoHeqn6nVwIPqkddHT7tcqpp9p7N0WOqR+WS+JPz4qdSbwJ7A== X-Received: by 2002:a05:6000:12c1:: with SMTP id l1mr10782157wrx.270.1596793888413; Fri, 07 Aug 2020 02:51:28 -0700 (PDT) Received: from localhost.localdomain.com ([151.29.36.84]) by smtp.gmail.com with ESMTPSA id h14sm9388396wml.30.2020.08.07.02.51.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Aug 2020 02:51:27 -0700 (PDT) From: Juri Lelli To: peterz@infradead.org, mingo@redhat.com Cc: rostedt@goodmis.org, tglx@linutronix.de, linux-kernel@vger.kernel.org, luca.abeni@santannapisa.it, tommaso.cucinotta@santannapisa.it, alessio.balsini@gmail.com, bristot@redhat.com, dietmar.eggemann@arm.com, linux-rt-users@vger.kernel.org, mtosatti@redhat.com, williams@redhat.com, valentin.schneider@arm.com Subject: [RFC PATCH v2 5/6] sched/fair: Add trivial fair server Date: Fri, 7 Aug 2020 11:50:50 +0200 Message-Id: <20200807095051.385985-6-juri.lelli@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200807095051.385985-1-juri.lelli@redhat.com> References: <20200807095051.385985-1-juri.lelli@redhat.com> MIME-Version: 1.0 Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org From: Peter Zijlstra Use deadline servers to service fair tasks. This patch adds a fair_server deadline entity which acts as a container for fair entities and can be used to fix starvation when higher priority (wrt fair) tasks are monopolizing CPU(s). Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 1 + kernel/sched/fair.c | 29 +++++++++++++++++++++++++++++ kernel/sched/sched.h | 4 ++++ 3 files changed, 34 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7c471961fd0b8..6537637139c63 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7170,6 +7170,7 @@ void __init sched_init(void) #endif /* CONFIG_SMP */ hrtick_rq_init(rq); atomic_set(&rq->nr_iowait, 0); + fair_server_init(rq); } set_load_weight(&init_task, false); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5130239c0e1e5..6a97ee2a4e26d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5514,6 +5514,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) */ util_est_enqueue(&rq->cfs, p); + if (!rq->cfs.h_nr_running) + dl_server_start(&rq->fair_server); + /* * If in_iowait is set, the code below may not trigger any cpufreq * utilization updates, so do it here explicitly with the IOWAIT flag @@ -5666,6 +5669,9 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) rq->next_balance = jiffies; dequeue_throttle: + if (!rq->cfs.h_nr_running) + dl_server_stop(&rq->fair_server); + util_est_dequeue(&rq->cfs, p, task_sleep); hrtick_update(rq); } @@ -7151,6 +7157,29 @@ static struct task_struct *__pick_next_task_fair(struct rq *rq) return pick_next_task_fair(rq, NULL, NULL); } +static bool fair_server_has_tasks(struct sched_dl_entity *dl_se) +{ + return !!dl_se->rq->cfs.nr_running; +} + +static struct task_struct *fair_server_pick(struct sched_dl_entity *dl_se) +{ + return pick_next_task_fair(dl_se->rq, NULL, NULL); +} + +void fair_server_init(struct rq *rq) +{ + struct sched_dl_entity *dl_se = &rq->fair_server; + + init_dl_entity(dl_se); + + dl_se->dl_runtime = TICK_NSEC; + dl_se->dl_deadline = 20 * TICK_NSEC; + dl_se->dl_period = 20 * TICK_NSEC; + + dl_server_init(dl_se, rq, fair_server_has_tasks, fair_server_pick); +} + /* * Account for a descheduled task: */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index f035cd8ccd224..bf8c9c07705c9 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -375,6 +375,8 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq, dl_server_has_tasks_f has_tasks, dl_server_pick_f pick); +extern void fair_server_init(struct rq *); + #ifdef CONFIG_CGROUP_SCHED #include @@ -959,6 +961,8 @@ struct rq { struct rt_rq rt; struct dl_rq dl; + struct sched_dl_entity fair_server; + #ifdef CONFIG_FAIR_GROUP_SCHED /* list of leaf cfs_rq on this CPU: */ struct list_head leaf_cfs_rq_list;