From patchwork Thu Apr 25 17:23:23 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 16402 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-gg0-f200.google.com (mail-gg0-f200.google.com [209.85.161.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id CD1B12395D for ; Thu, 25 Apr 2013 17:25:58 +0000 (UTC) Received: by mail-gg0-f200.google.com with SMTP id w6sf4751988ggk.7 for ; Thu, 25 Apr 2013 10:25:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-beenthere:x-received:received-spf :x-received:x-forwarded-to:x-forwarded-for:delivered-to:x-received :received-spf:x-received:from:to:cc:subject:date:message-id:x-mailer :in-reply-to:references:x-gm-message-state:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-google-group-id:list-post:list-help:list-archive:list-unsubscribe; bh=klz2dsGIAIwh5nsvmJR1d9jfPHjRtVCcjDzw7klQogM=; b=LlQlDp2mfbxnbm9AS7BgZCentKih7zyz9QPM6hz8PzfrjLn9iduQvASFr733yduSmm NUpz2N+wJ5L6bEej19MTkn5SgP+jobA5j2/vgkYYWoaTwU9+12HLo78AW73gMv4IKwXn Z5wOy7AJI+2BSfIDCloZ2Z6cQ1csiCJq0L9QvUPGHKJhgAS6sZKBF8hnrj0tGJCm2Z2/ M9DV4deq+ApnCZ8GajS1HsOsLPkDOM6F0a8tCYU/gSIP+Ot6zKqA2jXCWDlHV2IaUniK 5oT4DqrmpfCThho+aCSn60tysfN2Qbp+3P/DKrsYUDtfLhPn8TOeUQLBCslylYB5XTXu opcw== X-Received: by 10.236.130.103 with SMTP id j67mr11203709yhi.18.1366910705062; Thu, 25 Apr 2013 10:25:05 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.49.127.137 with SMTP id ng9ls1584136qeb.42.gmail; Thu, 25 Apr 2013 10:25:04 -0700 (PDT) X-Received: by 10.52.97.131 with SMTP id ea3mr23263706vdb.71.1366910704843; Thu, 25 Apr 2013 10:25:04 -0700 (PDT) Received: from mail-vc0-f176.google.com (mail-vc0-f176.google.com [209.85.220.176]) by mx.google.com with ESMTPS id p19si3652803vcw.89.2013.04.25.10.25.04 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 25 Apr 2013 10:25:04 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.176; Received: by mail-vc0-f176.google.com with SMTP id hf12so3052870vcb.35 for ; Thu, 25 Apr 2013 10:25:04 -0700 (PDT) X-Received: by 10.52.71.4 with SMTP id q4mr23279086vdu.8.1366910704758; Thu, 25 Apr 2013 10:25:04 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.58.127.98 with SMTP id nf2csp32664veb; Thu, 25 Apr 2013 10:25:04 -0700 (PDT) X-Received: by 10.180.198.49 with SMTP id iz17mr11562190wic.19.1366910703741; Thu, 25 Apr 2013 10:25:03 -0700 (PDT) Received: from mail-wg0-x22d.google.com (mail-wg0-x22d.google.com [2a00:1450:400c:c00::22d]) by mx.google.com with ESMTPS id e9si2742934wik.28.2013.04.25.10.25.03 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 25 Apr 2013 10:25:03 -0700 (PDT) Received-SPF: neutral (google.com: 2a00:1450:400c:c00::22d is neither permitted nor denied by best guess record for domain of vincent.guittot@linaro.org) client-ip=2a00:1450:400c:c00::22d; Received: by mail-wg0-f45.google.com with SMTP id l18so1578428wgh.0 for ; Thu, 25 Apr 2013 10:25:03 -0700 (PDT) X-Received: by 10.194.62.18 with SMTP id u18mr28855421wjr.53.1366910703285; Thu, 25 Apr 2013 10:25:03 -0700 (PDT) Received: from localhost.localdomain (LPuteaux-156-14-44-212.w82-127.abo.wanadoo.fr. [82.127.83.212]) by mx.google.com with ESMTPSA id q13sm12311485wie.8.2013.04.25.10.25.00 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 25 Apr 2013 10:25:02 -0700 (PDT) From: Vincent Guittot To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linaro-kernel@lists.linaro.org, peterz@infradead.org, mingo@kernel.org, linux@arm.linux.org.uk, pjt@google.com, santosh.shilimkar@ti.com, Morten.Rasmussen@arm.com, chander.kashyap@linaro.org, cmetcalf@tilera.com, tony.luck@intel.com, alex.shi@intel.com, preeti@linux.vnet.ibm.com Cc: paulmck@linux.vnet.ibm.com, tglx@linutronix.de, len.brown@intel.com, arjan@linux.intel.com, amit.kucheria@linaro.org, corbet@lwn.net, l.majewski@samsung.com, Vincent Guittot Subject: [PATCH 07/14] sched: agressively pack at wake/fork/exec Date: Thu, 25 Apr 2013 19:23:23 +0200 Message-Id: <1366910611-20048-8-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1366910611-20048-1-git-send-email-vincent.guittot@linaro.org> References: <1366910611-20048-1-git-send-email-vincent.guittot@linaro.org> X-Gm-Message-State: ALoCoQlwC6UqJCQBBLDJHhTNyQmnRkfW20uro72PY2KYRkrQXneYJ9/aVA+JeoV6yBYU0RAM5wXy X-Original-Sender: vincent.guittot@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , According to the packing policy, the scheduler can pack tasks at different step: -SCHED_PACKING_NONE level: we don't pack any task. -SCHED_PACKING_DEFAULT: we only pack small tasks at wake up when system is not busy. -SCHED_PACKING_FULL: we pack tasks at wake up until a CPU becomes full. During a fork or a exec, we assume that the new task is a full running one and we look for an idle CPU close to the buddy CPU. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 47 ++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 98166aa..874f330 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3259,13 +3259,16 @@ static struct sched_group * find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu, int load_idx) { - struct sched_group *idlest = NULL, *group = sd->groups; + struct sched_group *idlest = NULL, *group = sd->groups, *buddy = NULL; unsigned long min_load = ULONG_MAX, this_load = 0; int imbalance = 100 + (sd->imbalance_pct-100)/2; + int buddy_cpu = per_cpu(sd_pack_buddy, this_cpu); + int get_buddy = ((sysctl_sched_packing_mode == SCHED_PACKING_FULL) && + !(sd->flags & SD_SHARE_POWERDOMAIN) && (buddy_cpu != -1)); do { unsigned long load, avg_load; - int local_group; + int local_group, buddy_group = 0; int i; /* Skip over this group if it has no CPUs allowed */ @@ -3276,6 +3279,11 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, local_group = cpumask_test_cpu(this_cpu, sched_group_cpus(group)); + if (get_buddy) { + buddy_group = cpumask_test_cpu(buddy_cpu, + sched_group_cpus(group)); + } + /* Tally up the load of all CPUs in the group */ avg_load = 0; @@ -3287,6 +3295,9 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, load = target_load(i, load_idx); avg_load += load; + + if ((buddy_group) && idle_cpu(i)) + buddy = group; } /* Adjust by relative CPU power of the group */ @@ -3300,6 +3311,9 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, } } while (group = group->next, group != sd->groups); + if (buddy) + return buddy; + if (!idlest || 100*this_load < imbalance*min_load) return NULL; return idlest; @@ -3402,6 +3416,21 @@ static bool is_buddy_busy(int cpu) return (sum > (period / (rq->nr_running + 2))); } +static bool is_buddy_full(int cpu) +{ + struct rq *rq = cpu_rq(cpu); + u32 sum = rq->avg.runnable_avg_sum; + u32 period = rq->avg.runnable_avg_period; + + sum = min(sum, period); + + /* + * A full buddy is a CPU with a sum greater or equal to period + * We keep a margin of 2.4% + */ + return (sum * 1024 >= period * 1000); +} + static bool is_light_task(struct task_struct *p) { /* A light task runs less than 20% in average */ @@ -3413,6 +3442,9 @@ static int check_pack_buddy(int cpu, struct task_struct *p) { int buddy = per_cpu(sd_pack_buddy, cpu); + if (sysctl_sched_packing_mode == SCHED_PACKING_NONE) + return false; + /* No pack buddy for this CPU */ if (buddy == -1) return false; @@ -3421,14 +3453,19 @@ static int check_pack_buddy(int cpu, struct task_struct *p) if (!cpumask_test_cpu(buddy, tsk_cpus_allowed(p))) return false; + /* We agressively pack at wake up */ + if ((sysctl_sched_packing_mode == SCHED_PACKING_FULL) + && !is_buddy_full(buddy)) + return true; /* * If the task is a small one and the buddy is not overloaded, * we use buddy cpu */ - if (!is_light_task(p) || is_buddy_busy(buddy)) - return false; + if (is_light_task(p) && !is_buddy_busy(buddy)) + return true; + + return false; - return true; } /*