From patchwork Thu Apr 25 17:23:26 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 16405 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wi0-f198.google.com (mail-wi0-f198.google.com [209.85.212.198]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 21B8C2395D for ; Thu, 25 Apr 2013 17:26:07 +0000 (UTC) Received: by mail-wi0-f198.google.com with SMTP id c10sf4422151wiw.1 for ; Thu, 25 Apr 2013 10:25:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-beenthere:x-received:received-spf :x-received:x-forwarded-to:x-forwarded-for:delivered-to:x-received :received-spf:x-received:from:to:cc:subject:date:message-id:x-mailer :in-reply-to:references:x-gm-message-state:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-google-group-id:list-post:list-help:list-archive:list-unsubscribe; bh=DQ7dlCEEWgMvAPEVAM3pk7bF0N9g2AM5MdsrslAgBnM=; b=Y3EyzTemV27Gug4OpNo6imWZZ4T6KYi5dPDYst2pW1G9VeSlsHtU4He6c/z8CWr0eA CoobfTV6Dzkbdl/B0HoOpfYdWfQuZD7o4qy9ccEH9VDfw1jPGyJdADgRcZZHL4QSaDxN lsGgFjLGlTov/rj7d1eeWqBvYgXepz6WDq1JzK012tJmudrVBncNq0YdaQETQqt8qWsp c5GHFo/eFoTfq4AmpaNlbcn1NXzvv65Fv/Dhekj919EYbaYNO6xIXw6J6XyvrQLlJLxG cWyNY4oxU7WGDN/2if3OYn151OOlqig/pAXzUUfDDfb/wsVNNPp1H3zzKBI9fH38pWmQ DqGQ== X-Received: by 10.180.87.193 with SMTP id ba1mr1190003wib.0.1366910712816; Thu, 25 Apr 2013 10:25:12 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.180.108.212 with SMTP id hm20ls978310wib.16.gmail; Thu, 25 Apr 2013 10:25:12 -0700 (PDT) X-Received: by 10.194.235.169 with SMTP id un9mr4895606wjc.1.1366910712602; Thu, 25 Apr 2013 10:25:12 -0700 (PDT) Received: from mail-ve0-x229.google.com (mail-ve0-x229.google.com [2607:f8b0:400c:c01::229]) by mx.google.com with ESMTPS id b10si3842843wiw.43.2013.04.25.10.25.12 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 25 Apr 2013 10:25:12 -0700 (PDT) Received-SPF: neutral (google.com: 2607:f8b0:400c:c01::229 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=2607:f8b0:400c:c01::229; Received: by mail-ve0-f169.google.com with SMTP id jz10so1420194veb.28 for ; Thu, 25 Apr 2013 10:25:11 -0700 (PDT) X-Received: by 10.220.189.9 with SMTP id dc9mr14072676vcb.8.1366910711447; Thu, 25 Apr 2013 10:25:11 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.58.127.98 with SMTP id nf2csp32670veb; Thu, 25 Apr 2013 10:25:10 -0700 (PDT) X-Received: by 10.180.108.106 with SMTP id hj10mr5662678wib.0.1366910710589; Thu, 25 Apr 2013 10:25:10 -0700 (PDT) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [2a00:1450:400c:c05::22f]) by mx.google.com with ESMTPS id e2si2736911wic.58.2013.04.25.10.25.10 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 25 Apr 2013 10:25:10 -0700 (PDT) Received-SPF: neutral (google.com: 2a00:1450:400c:c05::22f is neither permitted nor denied by best guess record for domain of vincent.guittot@linaro.org) client-ip=2a00:1450:400c:c05::22f; Received: by mail-wi0-f175.google.com with SMTP id h11so8964730wiv.8 for ; Thu, 25 Apr 2013 10:25:10 -0700 (PDT) X-Received: by 10.180.39.207 with SMTP id r15mr39140379wik.16.1366910710128; Thu, 25 Apr 2013 10:25:10 -0700 (PDT) Received: from localhost.localdomain (LPuteaux-156-14-44-212.w82-127.abo.wanadoo.fr. [82.127.83.212]) by mx.google.com with ESMTPSA id q13sm12311485wie.8.2013.04.25.10.25.08 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 25 Apr 2013 10:25:09 -0700 (PDT) From: Vincent Guittot To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linaro-kernel@lists.linaro.org, peterz@infradead.org, mingo@kernel.org, linux@arm.linux.org.uk, pjt@google.com, santosh.shilimkar@ti.com, Morten.Rasmussen@arm.com, chander.kashyap@linaro.org, cmetcalf@tilera.com, tony.luck@intel.com, alex.shi@intel.com, preeti@linux.vnet.ibm.com Cc: paulmck@linux.vnet.ibm.com, tglx@linutronix.de, len.brown@intel.com, arjan@linux.intel.com, amit.kucheria@linaro.org, corbet@lwn.net, l.majewski@samsung.com, Vincent Guittot Subject: [PATCH 10/14] sched: update the buddy CPU Date: Thu, 25 Apr 2013 19:23:26 +0200 Message-Id: <1366910611-20048-11-git-send-email-vincent.guittot@linaro.org> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1366910611-20048-1-git-send-email-vincent.guittot@linaro.org> References: <1366910611-20048-1-git-send-email-vincent.guittot@linaro.org> X-Gm-Message-State: ALoCoQnVZ7cu45SWXvq1i+DgC9cGTiBBrcyL+qk8EEBG4m4uPBo/uktHFpnUYD3hndaxae2GYkdo X-Original-Sender: vincent.guittot@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 2607:f8b0:400c:c01::229 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Periodically updates the buddy of a CPU according to the current activity of the system. A CPU is its own buddy if it participates to the packing effort. Otherwise, it points to a CPU that participates to the packing effort. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 86 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 234ecdd..28f8ea7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -174,11 +174,17 @@ void sched_init_granularity(void) #ifdef CONFIG_SMP +static unsigned long power_of(int cpu) +{ + return cpu_rq(cpu)->cpu_power; +} + /* * Save the id of the optimal CPU that should be used to pack small tasks * The value -1 is used when no buddy has been found */ DEFINE_PER_CPU(int, sd_pack_buddy); +DEFINE_PER_CPU(struct sched_domain *, sd_pack_domain); /* * Look for the best buddy CPU that can be used to pack small tasks @@ -237,6 +243,68 @@ void update_packing_domain(int cpu) } pr_debug("CPU%d packing on CPU%d\n", cpu, id); + per_cpu(sd_pack_domain, cpu) = sd; + per_cpu(sd_pack_buddy, cpu) = id; +} + +void update_packing_buddy(int cpu, int activity) +{ + struct sched_domain *sd = per_cpu(sd_pack_domain, cpu); + struct sched_group *sg, *pack, *tmp; + int id = cpu; + + if (!sd) + return; + + /* + * The sched_domain of a CPU points on the local sched_group + * and this CPU of this local group is a good candidate + */ + pack = sg = sd->groups; + + /* loop the sched groups to find the best one */ + for (tmp = sg->next; tmp != sg; tmp = tmp->next) { + if ((tmp->sgp->power * pack->group_weight) > + (pack->sgp->power_available * tmp->group_weight)) + continue; + + if (((tmp->sgp->power * pack->group_weight) == + (pack->sgp->power * tmp->group_weight)) + && (cpumask_first(sched_group_cpus(tmp)) >= id)) + continue; + + /* we have found a better group */ + pack = tmp; + + /* Take the 1st CPU of the new group */ + id = cpumask_first(sched_group_cpus(pack)); + } + + if ((cpu == id) || (activity <= power_of(id))) { + per_cpu(sd_pack_buddy, cpu) = id; + return; + } + + for (tmp = pack; activity > 0; tmp = tmp->next) { + if (tmp->sgp->power > activity) { + id = cpumask_first(sched_group_cpus(tmp)); + activity -= power_of(id); + if (cpu == id) + activity = 0; + while ((activity > 0) && (id < nr_cpu_ids)) { + id = cpumask_next(id, sched_group_cpus(tmp)); + activity -= power_of(id); + if (cpu == id) + activity = 0; + } + } else if (cpumask_test_cpu(cpu, sched_group_cpus(tmp))) { + id = cpu; + activity = 0; + } else { + activity -= tmp->sgp->power; + } + } + per_cpu(sd_pack_buddy, cpu) = id; } @@ -3014,11 +3082,6 @@ static unsigned long target_load(int cpu, int type) return max(rq->cpu_load[type-1], total); } -static unsigned long power_of(int cpu) -{ - return cpu_rq(cpu)->cpu_power; -} - static unsigned long cpu_avg_load_per_task(int cpu) { struct rq *rq = cpu_rq(cpu); @@ -4740,6 +4803,22 @@ static bool update_sd_pick_busiest(struct lb_env *env, return false; } +static void update_plb_buddy(int cpu, int *balance, struct sd_lb_stats *sds, + struct sched_domain *sd) +{ + int buddy; + + if (sysctl_sched_packing_mode != SCHED_PACKING_FULL) + return; + + /* Update my buddy */ + if (sd == per_cpu(sd_pack_domain, cpu)) + update_packing_buddy(cpu, sds->total_activity); + + /* Get my new buddy */ + buddy = per_cpu(sd_pack_buddy, cpu); +} + /** * update_sd_lb_stats - Update sched_domain's statistics for load balancing. * @env: The load balancing environment. @@ -4807,6 +4886,8 @@ static inline void update_sd_lb_stats(struct lb_env *env, sg = sg->next; } while (sg != env->sd->groups); + + update_plb_buddy(env->dst_cpu, balance, sds, env->sd); } /**