From patchwork Wed Jan 13 23:07:16 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: gary.robertson@linaro.org X-Patchwork-Id: 59692 Delivered-To: patch@linaro.org Received: by 10.112.130.2 with SMTP id oa2csp3655217lbb; Wed, 13 Jan 2016 15:08:05 -0800 (PST) X-Received: by 10.140.101.130 with SMTP id u2mr1050292qge.71.1452726485419; Wed, 13 Jan 2016 15:08:05 -0800 (PST) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id a192si3821128qkb.70.2016.01.13.15.08.05; Wed, 13 Jan 2016 15:08:05 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dkim=neutral (body hash did not verify) header.i=@linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id EEDB0617BC; Wed, 13 Jan 2016 23:08:04 +0000 (UTC) Authentication-Results: lists.linaro.org; dkim=fail reason="verification failed; unprotected key" header.d=linaro.org header.i=@linaro.org header.b=dB1WXhxF; dkim-adsp=none (unprotected policy); dkim-atps=neutral X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, T_DKIM_INVALID, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 46F44617BF; Wed, 13 Jan 2016 23:07:38 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id F1CBB6176C; Wed, 13 Jan 2016 23:07:32 +0000 (UTC) Received: from mail-ob0-f178.google.com (mail-ob0-f178.google.com [209.85.214.178]) by lists.linaro.org (Postfix) with ESMTPS id D76E66176C for ; Wed, 13 Jan 2016 23:07:31 +0000 (UTC) Received: by mail-ob0-f178.google.com with SMTP id py5so100824717obc.2 for ; Wed, 13 Jan 2016 15:07:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9w3DKS1pC9RHkqPc3jdCULLYDTWnCF7c7Bjy2FXbXhw=; b=dB1WXhxFNUx5+vki1bR88A1hM/ou5bucT1r/Qv4tT0YBx8Vt1OgoR4//qHIUjgp8Z+ 1BH6G6KBctdocHgrJzULIOUnP0+UUjTORTG+qY+U8NHY6yXH4ZSbLrIJgB1q7U+KUe+v 787gkj6EK5VUqIxkshXURnQoSg8Ox91O2kQD4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9w3DKS1pC9RHkqPc3jdCULLYDTWnCF7c7Bjy2FXbXhw=; b=Gpevl8tFtR3EwlkfeYOEAjqJmA1Dp9dSjZ67iRmbQ17fTGzbgSFHprKrMm4ERjquiZ NknKBLCYwW2USih3MkR2haPJLH572RV4ZLJhEmrBGw9HN4wErOREk6usqAKwPa+uhzW7 MCPuqe+ZXJEMx/kA0t3Ua5DEqEAY8o9bR0SgLIBOXtMQpukjyJanUpyzD5Nu39Sw7SU4 574MNMg/Rdb2/GEPU/KpFQrVlCkqcg4zVk6MWRSFRXYa0NYYtLHQrx9Qz0bloHsonHj6 CCPTF2j/KKs3HUPvuHqtymn1icdCRxXGOh7gcsyzcmv+60QmwZa0/f47JG9ee7aoI30s +3vg== X-Gm-Message-State: ALoCoQnlafnLXTB3aI4CpSdbU/SZcN1YsgyL6smxzvEAs4iCZm7PLsYm4Bg+CC+dGc2gVWDSy+mj9K7fRCrvCeLBJt64BsVE0A== X-Received: by 10.60.62.243 with SMTP id b19mr706669oes.28.1452726451417; Wed, 13 Jan 2016 15:07:31 -0800 (PST) Received: from honkintosh.cybertech.lan (65-120-133-114.dia.static.qwest.net. [65.120.133.114]) by smtp.googlemail.com with ESMTPSA id kg7sm1804489obb.27.2016.01.13.15.07.30 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 13 Jan 2016 15:07:30 -0800 (PST) From: "Gary S. Robertson" To: mike.holmes@linaro.org, bill.fischofer@linaro.org, maxim.uvarov@linaro.org, anders.roxell@linaro.org, stuart.haslam@linaro.org, petri.savolainen@linaro.org Date: Wed, 13 Jan 2016 17:07:16 -0600 Message-Id: <1452726436-12095-2-git-send-email-gary.robertson@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1452726436-12095-1-git-send-email-gary.robertson@linaro.org> References: <1452726436-12095-1-git-send-email-gary.robertson@linaro.org> Cc: lng-odp@lists.linaro.org Subject: [lng-odp] [API_NEXT RFC 01/01] linux-generic: add support for initial cpumasks X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" These code changes depend on the addition of control and worker cpumasks to the ODP initialization parameter data structure, and implement the change in behavior suggested with that patch. They serve as the 'glue' between the input of the new ODP API initial cpuset masks and the use of those new cpumasks by the ODP application or instance. Specifically: if neither of the new cpumasks are populated prior to calling odp_init_global(), then the behavior for allocation of control and worker cpumasks is unchanged from its current (pre-patch) state. However, if the cpumasks are populated prior to calling odp_init_global() then that routine saves their contents into global variables for later reference. Then when odp_cpumask_default_control() or odp_cpumask_default_worker() are called they build the caller's cpumasks based on the saved contents of the pre-populated cpuset input. The 'hooks' implemented here will enable a helper function to parse cpuset specification options from the command line, initialize the new cpuset arguments as specified by those command line options, and enforce the proper usage of those specified cpusets by the linux-generic reference platform implementation. Alternatively some system CPU resource provisioning utility could be used to monitor available CPU resources dynamically and to orchestrate their use for ODP applications by manipulating the new cpuset arguments as new ODP applications or instances were started. diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index 49e23d9..47b63bd 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -38,6 +38,8 @@ struct odp_global_data_s { odp_log_func_t log_fn; odp_abort_func_t abort_fn; odp_system_info_t system_info; + odp_cpumask_t control_cpus; + odp_cpumask_t worker_cpus; }; extern struct odp_global_data_s odp_global_data; diff --git a/platform/linux-generic/odp_cpumask_task.c b/platform/linux-generic/odp_cpumask_task.c index 47cf6ea..57b9849 100644 --- a/platform/linux-generic/odp_cpumask_task.c +++ b/platform/linux-generic/odp_cpumask_task.c @@ -18,10 +18,28 @@ int odp_cpumask_default_worker(odp_cpumask_t *mask, int num) int ret, cpu, i; cpu_set_t cpuset; - ret = pthread_getaffinity_np(pthread_self(), - sizeof(cpu_set_t), &cpuset); - if (ret != 0) - ODP_ABORT("failed to read CPU affinity value\n"); + CPU_ZERO(&cpuset); + + /* + * If the available worker cpumask was specified prior to global init, + * then allocate worker CPUs from that cpumask. + */ + if (odp_cpumask_count(&odp_global_data.worker_cpus)) { + for (i = 0; i < CPU_SETSIZE; i++) + if (odp_cpumask_isset(&odp_global_data.worker_cpus, i)) + CPU_SET(i, &cpuset); + } else { + /* + * No initial worker cpumask was specified... implying that + * CPU isolation is not needed for this ODP instance. + * So allocate worker CPUs from the available CPUs defined + * by the default OS environment. + */ + ret = pthread_getaffinity_np(pthread_self(), + sizeof(cpu_set_t), &cpuset); + if (ret != 0) + ODP_ABORT("failed to read CPU affinity value\n"); + } odp_cpumask_zero(mask); @@ -29,7 +47,7 @@ int odp_cpumask_default_worker(odp_cpumask_t *mask, int num) * If no user supplied number or it's too large, then attempt * to use all CPUs */ - if (0 == num || CPU_SETSIZE < num) + if (0 == num || CPU_COUNT(&cpuset) < num) num = CPU_COUNT(&cpuset); /* build the mask, allocating down from highest numbered CPU */ @@ -49,10 +67,26 @@ int odp_cpumask_default_worker(odp_cpumask_t *mask, int num) int odp_cpumask_default_control(odp_cpumask_t *mask, int num ODP_UNUSED) { + int cpu_count = 0; + odp_cpumask_t all_cpus; + odp_cpumask_zero(mask); - /* By default all control threads on CPU 0 */ - odp_cpumask_set(mask, 0); - return 1; + + /* + * If the available control cpumask was specified prior to global init, + * then allocate control CPUs from that cpumask. + */ + if ((cpu_count = odp_cpumask_count(&odp_global_data.control_cpus))) { + odp_cpumask_copy(mask, &odp_global_data.control_cpus); + } + + /* CPU 0 must always be usable for control threads */ + if (!odp_cpumask_isset(mask, 0)) { + odp_cpumask_set(mask, 0); + cpu_count++; + } + + return cpu_count; } int odp_cpumask_all_available(odp_cpumask_t *mask) diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index 70a3dbe..7578b84 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -22,6 +22,23 @@ int odp_init_global(const odp_init_t *params, odp_global_data.log_fn = params->log_fn; if (params->abort_fn != NULL) odp_global_data.abort_fn = params->abort_fn; + /* + * Save the control and worker cpumask contents + * in a globally accessible data structure + * so odp_cpumask_default_control(), + * odp_cpumask_default_worker(), and any + * isolation support logic may reference them later. + */ + if (!odp_cpumask_count(¶ms->control_cpus)) + odp_cpumask_zero(&odp_global_data.control_cpus); + else + odp_cpumask_copy(&odp_global_data.control_cpus, + ¶ms->control_cpus); + if (!odp_cpumask_count(¶ms->worker_cpus)) + odp_cpumask_zero(&odp_global_data.worker_cpus); + else + odp_cpumask_copy(&odp_global_data.worker_cpus, + ¶ms->worker_cpus); } if (odp_time_global_init()) {