From patchwork Mon Nov 26 11:20:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 152000 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp5537219ljp; Mon, 26 Nov 2018 03:24:56 -0800 (PST) X-Google-Smtp-Source: AFSGD/UQbpa8qDl9B6o0vzhr8+SWae1AEgTGJfnUcOmoFchfUO7G00+/80CXFv5HDmWuTcsxg3kx X-Received: by 2002:a17:902:b20e:: with SMTP id t14mr16742973plr.128.1543231496717; Mon, 26 Nov 2018 03:24:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543231496; cv=none; d=google.com; s=arc-20160816; b=SBTd8egOs+YmAFjYprXHPzg6nPU7dEZj+kDw7IZFxGDud+Iie4Q56xuHzbIhxkPQIK K009ds6A3oTYa8EsVOEqgEoVYGUWMIMW14HMPmPiA9tHV3fDT/gktfVGqLz/PHepqzSj 2Ba2VSQOvniTWC+6UcMd3fjxdvpjifcJV5xSdalW1B7i1TzX8Xylanb+BS+fIo1GNj1D L23jyhLoOKYFpbJZzSdRqwqhChr9hXm/xbmGRpVeGzSx0Og+iE7a+Y0XoRX0WIpEO4Hw yGUFKTv54Yq5kJUxzDIbw8M8BBssuazfAERKVdB8+po84NBRySVRQPePcoPr/MBYMfgB 0itQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KGAn6Dmctb3M6Js+r6sAQXS0roAHSwBAADu8U/GYcmU=; b=Ffe85MkRXjXRurloOOa4TgewNiPl4XEFYLkQTBVpe8mzLaJ//WWKoyESf8Ph5wARfq M+KrYQ/0pY+a7PHdfCmf1dJn51APyBvvt2iTmvaxvPZzreGhg6MKksDWCmqlPbqkY/h5 iP4HxspCvFNHqvbnrAONuxALniF8QhysNFWPUMkAGdY7wXIVsoyed63RvDFFpEr4RxQ1 qhCwSXfpslpSb+MG8AfTetJAROXhPCODKM4EiFbv4OKH8vgdqdp1FNLyJPg1yhhQ7kCw CEHqnjGi6zHvU9YaJNLNZxeD+spWtlGG622I746btyYdpNkcbSVrVqBMjxrYBlgeEh4i fkTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gHDqpwN8; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g10si47041319pll.428.2018.11.26.03.24.54; Mon, 26 Nov 2018 03:24:56 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=gHDqpwN8; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729147AbeKZWO4 (ORCPT + 32 others); Mon, 26 Nov 2018 17:14:56 -0500 Received: from mail-pl1-f196.google.com ([209.85.214.196]:45198 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726199AbeKZWOz (ORCPT ); Mon, 26 Nov 2018 17:14:55 -0500 Received: by mail-pl1-f196.google.com with SMTP id a14so13707630plm.12 for ; Mon, 26 Nov 2018 03:21:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KGAn6Dmctb3M6Js+r6sAQXS0roAHSwBAADu8U/GYcmU=; b=gHDqpwN8gfy+sg2OO6B8R2/87Nf70jXbqV44Tu0hDF8DYqUwWDFDOpGRO3eoeWKN8I WFffhwH06yIZc7nqlD9E3zZ0u/54PKd1gFZtfhrtva5PgOpdVAWgbrtOnwOADWzkY2/K z/KoOSpvz6sByaWf72FpqSETJilulwenM9swY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KGAn6Dmctb3M6Js+r6sAQXS0roAHSwBAADu8U/GYcmU=; b=q3+ivKb0rO7gKFSoOgEqK2mggSubxxjkYUdBfLvgLWpuVLLOmKjOsT8RozxcQ9fb9v AYdQY7qikQFzmcWuKpC+AyxpJ/xLT4qL3A9s1J54Y0TFI3ISRf3xRkN59hSwhBqWZ06o dN7Tz2sSSYYgBEh51/k/C5qB6ca1v61ARjFG+wqbeELAB+iEF0EqTQqR6z31TIfruxKQ Sea9XN+6KFniPnIu/XPyYkCLHcw6eOyiuj8Y2P6cgzl9dqgVh4GCHkE2TxLJWebMqCEv Fd+2/kSrEWMZ92+sOy2jt7s5UceS0MBqesvFM3C0yJpvu6t6tA31veLAHZ9QrX2kTLil dDQw== X-Gm-Message-State: AA+aEWbXz18TpFd81hYWn7GqJJLJdP7EgmWcPjtzk+TO9cgT8OQ7X956 EylDwztnqTopx9VczcXEOKhXVA== X-Received: by 2002:a17:902:481:: with SMTP id e1-v6mr27043023ple.132.1543231266326; Mon, 26 Nov 2018 03:21:06 -0800 (PST) Received: from localhost ([122.172.88.116]) by smtp.gmail.com with ESMTPSA id c4sm46780551pfm.151.2018.11.26.03.21.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 03:21:05 -0800 (PST) From: Viresh Kumar To: Ingo Molnar , Peter Zijlstra Cc: Viresh Kumar , linux-kernel@vger.kernel.org, Vincent Guittot , tkjos@google.com, Daniel Lezcano , quentin.perret@linaro.org, chris.redpath@arm.com, Dietmar.Eggemann@arm.com Subject: [RFC][PATCH 1/2] sched: Start tracking SCHED_IDLE tasks count in cfs_rq Date: Mon, 26 Nov 2018 16:50:23 +0530 Message-Id: X-Mailer: git-send-email 2.19.1.568.g152ad8e3369a In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Start tracking how many tasks with SCHED_IDLE policy are present in each cfs_rq. This will be used by later commits. Signed-off-by: Viresh Kumar --- kernel/sched/fair.c | 14 ++++++++++++-- kernel/sched/sched.h | 2 ++ 2 files changed, 14 insertions(+), 2 deletions(-) -- 2.19.1.568.g152ad8e3369a diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e30dea59d215..ad0b09ddddc0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4453,7 +4453,7 @@ static void throttle_cfs_rq(struct cfs_rq *cfs_rq) struct rq *rq = rq_of(cfs_rq); struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); struct sched_entity *se; - long task_delta, dequeue = 1; + long task_delta, idle_task_delta, dequeue = 1; bool empty; se = cfs_rq->tg->se[cpu_of(rq_of(cfs_rq))]; @@ -4464,6 +4464,7 @@ static void throttle_cfs_rq(struct cfs_rq *cfs_rq) rcu_read_unlock(); task_delta = cfs_rq->h_nr_running; + idle_task_delta = cfs_rq->idle_h_nr_running; for_each_sched_entity(se) { struct cfs_rq *qcfs_rq = cfs_rq_of(se); /* throttled entity or throttle-on-deactivate */ @@ -4473,6 +4474,7 @@ static void throttle_cfs_rq(struct cfs_rq *cfs_rq) if (dequeue) dequeue_entity(qcfs_rq, se, DEQUEUE_SLEEP); qcfs_rq->h_nr_running -= task_delta; + qcfs_rq->idle_h_nr_running -= idle_task_delta; if (qcfs_rq->load.weight) dequeue = 0; @@ -4512,7 +4514,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); struct sched_entity *se; int enqueue = 1; - long task_delta; + long task_delta, idle_task_delta; se = cfs_rq->tg->se[cpu_of(rq)]; @@ -4532,6 +4534,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) return; task_delta = cfs_rq->h_nr_running; + idle_task_delta = cfs_rq->idle_h_nr_running; for_each_sched_entity(se) { if (se->on_rq) enqueue = 0; @@ -4540,6 +4543,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) if (enqueue) enqueue_entity(cfs_rq, se, ENQUEUE_WAKEUP); cfs_rq->h_nr_running += task_delta; + cfs_rq->idle_h_nr_running += idle_task_delta; if (cfs_rq_throttled(cfs_rq)) break; @@ -5092,6 +5096,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) { struct cfs_rq *cfs_rq; struct sched_entity *se = &p->se; + int idle_h_nr_running = unlikely(task_has_idle_policy(p)) ? 1 : 0; /* * The code below (indirectly) updates schedutil which looks at @@ -5124,6 +5129,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (cfs_rq_throttled(cfs_rq)) break; cfs_rq->h_nr_running++; + cfs_rq->idle_h_nr_running += idle_h_nr_running; flags = ENQUEUE_WAKEUP; } @@ -5131,6 +5137,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) for_each_sched_entity(se) { cfs_rq = cfs_rq_of(se); cfs_rq->h_nr_running++; + cfs_rq->idle_h_nr_running += idle_h_nr_running; if (cfs_rq_throttled(cfs_rq)) break; @@ -5157,6 +5164,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) struct cfs_rq *cfs_rq; struct sched_entity *se = &p->se; int task_sleep = flags & DEQUEUE_SLEEP; + int idle_h_nr_running = unlikely(task_has_idle_policy(p)) ? 1 : 0; for_each_sched_entity(se) { cfs_rq = cfs_rq_of(se); @@ -5171,6 +5179,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (cfs_rq_throttled(cfs_rq)) break; cfs_rq->h_nr_running--; + cfs_rq->idle_h_nr_running -= idle_h_nr_running; /* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) { @@ -5190,6 +5199,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) for_each_sched_entity(se) { cfs_rq = cfs_rq_of(se); cfs_rq->h_nr_running--; + cfs_rq->idle_h_nr_running -= idle_h_nr_running; if (cfs_rq_throttled(cfs_rq)) break; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index e0e052a50fcd..86a388c506ac 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -488,6 +488,8 @@ struct cfs_rq { unsigned long runnable_weight; unsigned int nr_running; unsigned int h_nr_running; + /* h_nr_running for SCHED_IDLE tasks */ + unsigned int idle_h_nr_running; u64 exec_clock; u64 min_vruntime; From patchwork Mon Nov 26 11:20:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 152001 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp5537249ljp; Mon, 26 Nov 2018 03:24:58 -0800 (PST) X-Google-Smtp-Source: AFSGD/XuruS4KkWVFoG7hgLC8UsWKm+zvNCELsBtKn8C0Hpo44o1mvGEwBw/LY+BA3waZEY4+MIp X-Received: by 2002:a17:902:b20e:: with SMTP id t14mr16743068plr.128.1543231498350; Mon, 26 Nov 2018 03:24:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543231498; cv=none; d=google.com; s=arc-20160816; b=q3RdJqluxO0SlB9o0ZB2vAfcNtyNFe4I30oRC9Nv2Ky4aCOVw8jQcjIdOf6GvQda55 Sd+Ii2CkmJuvg1/Dsdh6qKr2JHRdg9xKxOge1By5Xx1sQoT8bbjkmf6In6cinVSIxC3G aBwkUpEcczOAPkaEAKyO1309kWs3gsEitwY5Rx/iTOu1RKCatRG8fRbKu/FUxYlWBGK9 bPlDMyP9U3NDrbmSHJNuqZI5wbnWD9Ssn1MuDSYFRsLSoAB1WbLLDUxtgUhEM+VJSz/X nk9OH+Otqze110CmoyCkmHWW+y03fsqLk6lYMHLqKM5abcPuKVZRxkBGMlIPtpmGwyFa Nrgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cXjCpnaegYAXM+GZRUdmlkOBI/edddeoKcnlEDVaygI=; b=0hBFy6mGcMk6fAYIlD/oTUAAOgRr1oInVVSeMoqYU9o/p10QDAxFJKJP1ZxXLm5GMs qqH0RYIqWNZ2jauaY/wLmLWFPXn1DU8mjNntoii2XfFvAlZbfOack1EQ/jz7Vj0hrqAD hO0h38RhksS7NmTdKyyugTZZP+9f76a8s/M6QXxA0SGToSKsaHdB5kQXsPvCt+9eop7x wSpwQ8mDnd1egV2BuoB4fZ/QMBdGfQFK5Q9dEEaxDU0e2DGgoXNddy1pPCh7FQHNeNYO 2LsM1lYWzOLgp+YM47Ltw/EjtJCkaGt5EbO2FsBy2+YrPFQnUmzsVcerkFazg9P1ZME+ SZxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=CnWJzWkq; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g10si47041319pll.428.2018.11.26.03.24.56; Mon, 26 Nov 2018 03:24:58 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=CnWJzWkq; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729447AbeKZWO7 (ORCPT + 32 others); Mon, 26 Nov 2018 17:14:59 -0500 Received: from mail-pf1-f194.google.com ([209.85.210.194]:41602 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726199AbeKZWO7 (ORCPT ); Mon, 26 Nov 2018 17:14:59 -0500 Received: by mail-pf1-f194.google.com with SMTP id b7so6420961pfi.8 for ; Mon, 26 Nov 2018 03:21:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cXjCpnaegYAXM+GZRUdmlkOBI/edddeoKcnlEDVaygI=; b=CnWJzWkqXjXQgN1QMfMyd331KGRYduvoL4211649yMNrd/dK5H/zOvmt17dYi5mkDN PkpZcwhLV+mqBgZ2MiOnFDMYAp/dTKnYmN7MpkZR9R6BjFg4r1sUcdjnBiPsubdI8kyq HzbNYKy29QEL/YYhnmu57/r7moNW30Dw4W9c4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cXjCpnaegYAXM+GZRUdmlkOBI/edddeoKcnlEDVaygI=; b=Zzo0nzKpV5K2VG1x1GqyncbwA9prNh61mdZ3KheBctd30xz0T19e4zJaAqAeEBU2ba mgAzCs0Aqp4xha/QWshzcttswB4eu/oV8Px4iY/0G5XtvTCtKsOzv39g4/9H9XfC9Jpt JCA8rp4Of9a7fp5/WO9YyOjeuVRsjhkf+FTR5H55T/n7V0FYOCGlflES+G9/y/AOjULn wSpU2Ck25Ve7k8Q0qT7hmA4RBOAlpZqiyYSU9DwEXzFvH9OBCZJyKwivA0Dr9aQCp4cT EfuAtlMg9VX5YhWy7hdINphHTVadKVl73XLqa8Xs/eTYA0iJq7BNTkp2cP72fqmsgtu9 TVig== X-Gm-Message-State: AA+aEWZVOaU/OhpJuBwuQXG/DykLf0FG1q05ujIvqC5LaZHIBizu309z A346Kzf6QUpUbzw07ovqwYzuuw== X-Received: by 2002:a63:5a08:: with SMTP id o8mr24314999pgb.185.1543231269682; Mon, 26 Nov 2018 03:21:09 -0800 (PST) Received: from localhost ([122.172.88.116]) by smtp.gmail.com with ESMTPSA id a73sm51209pfa.7.2018.11.26.03.21.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Nov 2018 03:21:08 -0800 (PST) From: Viresh Kumar To: Ingo Molnar , Peter Zijlstra Cc: Viresh Kumar , linux-kernel@vger.kernel.org, Vincent Guittot , tkjos@google.com, Daniel Lezcano , quentin.perret@linaro.org, chris.redpath@arm.com, Dietmar.Eggemann@arm.com Subject: [RFC][PATCH 2/2] sched: Enqueue tasks on a cpu with only SCHED_IDLE tasks Date: Mon, 26 Nov 2018 16:50:24 +0530 Message-Id: <71a578a0c5b39169fe74ad378ee41eaf546844ac.1543229820.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.19.1.568.g152ad8e3369a In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The scheduler tries to schedule a newly wakeup task on an idle CPU to make sure the new task gets chance to run as soon as possible, for performance reasons. The SCHED_IDLE scheduling policy is used for tasks which have the lowest priority and there is no hurry in running them. If all the tasks currently enqueued on a CPU have their policy set to SCHED_IDLE, then any new task (non SCHED_IDLE) enqueued on that CPU should normally get a chance to run immediately. This patch takes advantage of this to save power in some cases by avoiding waking up an idle CPU (which may be in some deep idle state) and enqueuing the new task on a CPU which only has SCHED_IDLE tasks. Signed-off-by: Viresh Kumar --- kernel/sched/core.c | 23 ++++++++++++++++++++ kernel/sched/fair.c | 50 +++++++++++++++++++++++++++++++------------- kernel/sched/sched.h | 3 +++ 3 files changed, 62 insertions(+), 14 deletions(-) -- 2.19.1.568.g152ad8e3369a diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 3d87a28da378..176eed77b18e 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4020,6 +4020,29 @@ int available_idle_cpu(int cpu) return 1; } +/* CPU only has SCHED_IDLE tasks enqueued */ +int cpu_only_has_sched_idle_tasks(int cpu) +{ + struct rq *rq = cpu_rq(cpu); + + return unlikely(rq->nr_running && + rq->nr_running == rq->cfs.idle_h_nr_running); +} + +int available_sched_idle_cpu(int cpu) +{ + if (vcpu_is_preempted(cpu)) + return 0; + + if (idle_cpu(cpu)) + return 1; + + if (cpu_only_has_sched_idle_tasks(cpu)) + return 1; + + return 0; +} + /** * idle_task - return the idle task for a given CPU. * @cpu: the processor in question. diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ad0b09ddddc0..3a029c740d51 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5623,9 +5623,10 @@ wake_affine_idle(int this_cpu, int prev_cpu, int sync) * on one CPU. */ if (available_idle_cpu(this_cpu) && cpus_share_cache(this_cpu, prev_cpu)) - return available_idle_cpu(prev_cpu) ? prev_cpu : this_cpu; + return available_sched_idle_cpu(prev_cpu) ? prev_cpu : this_cpu; - if (sync && cpu_rq(this_cpu)->nr_running == 1) + if ((sync && cpu_rq(this_cpu)->nr_running == 1) || + cpu_only_has_sched_idle_tasks(this_cpu)) return this_cpu; return nr_cpumask_bits; @@ -5888,6 +5889,9 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this latest_idle_timestamp = rq->idle_stamp; shallowest_idle_cpu = i; } + } else if (cpu_only_has_sched_idle_tasks(i) && !vcpu_is_preempted(i)) { + /* Prefer CPU with only SCHED_IDLE tasks */ + return i; } else if (shallowest_idle_cpu == -1) { load = weighted_cpuload(cpu_rq(i)); if (load < min_load) { @@ -6049,7 +6053,7 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int */ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int target) { - int cpu; + int cpu, last_idle_cpu = -1; if (!static_branch_likely(&sched_smt_present)) return -1; @@ -6057,11 +6061,18 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t for_each_cpu(cpu, cpu_smt_mask(target)) { if (!cpumask_test_cpu(cpu, &p->cpus_allowed)) continue; - if (available_idle_cpu(cpu)) - return cpu; + if (!vcpu_is_preempted(cpu)) { + if (idle_cpu(cpu)) { + /* Prefer CPU with only SCHED_IDLE tasks */ + last_idle_cpu = cpu; + continue; + } + if (cpu_only_has_sched_idle_tasks(cpu)) + return cpu; + } } - return -1; + return last_idle_cpu; } #else /* CONFIG_SCHED_SMT */ @@ -6089,7 +6100,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t u64 avg_cost, avg_idle; u64 time, cost; s64 delta; - int cpu, nr = INT_MAX; + int cpu, nr = INT_MAX, last_idle_cpu = -1; this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc)); if (!this_sd) @@ -6116,12 +6127,23 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t time = local_clock(); for_each_cpu_wrap(cpu, sched_domain_span(sd), target) { - if (!--nr) - return -1; + if (!--nr) { + if (last_idle_cpu == -1) + return -1; + cpu = last_idle_cpu; + break; + } if (!cpumask_test_cpu(cpu, &p->cpus_allowed)) continue; - if (available_idle_cpu(cpu)) - break; + if (!vcpu_is_preempted(cpu)) { + if (idle_cpu(cpu)) { + /* Prefer CPU with only SCHED_IDLE tasks */ + last_idle_cpu = cpu; + continue; + } + if (cpu_only_has_sched_idle_tasks(cpu)) + break; + } } time = local_clock() - time; @@ -6140,13 +6162,13 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) struct sched_domain *sd; int i, recent_used_cpu; - if (available_idle_cpu(target)) + if (available_sched_idle_cpu(target)) return target; /* * If the previous CPU is cache affine and idle, don't be stupid: */ - if (prev != target && cpus_share_cache(prev, target) && available_idle_cpu(prev)) + if (prev != target && cpus_share_cache(prev, target) && available_sched_idle_cpu(prev)) return prev; /* Check a recently used CPU as a potential idle candidate: */ @@ -6154,7 +6176,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (recent_used_cpu != prev && recent_used_cpu != target && cpus_share_cache(recent_used_cpu, target) && - available_idle_cpu(recent_used_cpu) && + available_sched_idle_cpu(recent_used_cpu) && cpumask_test_cpu(p->recent_used_cpu, &p->cpus_allowed)) { /* * Replace recent_used_cpu with prev as it is a potential diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 86a388c506ac..ecd016c64ee2 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1828,6 +1828,9 @@ extern void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags); extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; +extern int cpu_only_has_sched_idle_tasks(int cpu); +extern int available_sched_idle_cpu(int cpu); + #ifdef CONFIG_SCHED_HRTICK /*