From patchwork Tue Sep 11 14:00:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 146454 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp3677593ljw; Tue, 11 Sep 2018 07:03:21 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbZOpl61+6DN/Mg6Gf7Lul+SweCH9YsBZkUAEsU64h/rUuORYjH+IXO9XdZWvknDaht765m X-Received: by 2002:ac8:1b5b:: with SMTP id p27-v6mr19390988qtk.99.1536674601327; Tue, 11 Sep 2018 07:03:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536674601; cv=none; d=google.com; s=arc-20160816; b=LXW8bHzx4Vcb7f9qKkiayCdLLNNX1rj1Tx34vsafxk6x6C42Ijj2kH28WVaOnIuIHN 5ab4n2AHwdw56h1PMc0t8PtvH6uEzcAHLNVehLej+2PD1jeqwKI7u0/bttNSul8i8Vu6 bMhrzdtUFQ3/7E7Ta+wAokvHrroJfALZU2l4OY0C+RXhtRqXe7y6LsD7WU8vzddrTgBK 4FX7QI7bQ5ar7WzRX4kmEGGQPqv+WwWwIEg+AYCX7C3GRAYNKgUBYr4CEs89OhjG+DV/ 4LhaHuKg1D5WFa4+i+lIxzgotbndo9fITrHHwxNBrKXfWxTVzByoAhHH3tlhIktqpJyq 5H6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to; bh=QaTxxBD2E3U2D0puVhdLE9rAfff0Rydffm59g1dqGNo=; b=H0WsqvLtUV4KvR7NZg8ZqhYAwjTBfM70z71P7K/lKFtk65q76Tj/yaEe5oUdE28RDS K2NCFQ5tHMzdnWhXygvmeVXSDVSFAVskdnaBD4s02JtSd9ZjBNAw0Ox9UP0kRMf/j3/T ZqWCrLLhLV8HLNb2EBDzysXDAf1yuDcgeQ1E6ur1FPTWRkImfUeiJLeGcBz6t2yev3I6 8LC5CAAu2oM0h90m6eDwhCxGSlf7xjgLTtoUVhnRt5dov71+k5vF+CZKqTe9cCx1ydgf TOpeFRaR8Qg+LwOQnLrH1yCFYDsdk4iVE1hXQwFzvPr9CfLrMwO/z5JsYkgI+3ZIvSmt Yj7A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id i19-v6si205628qvc.125.2018.09.11.07.03.18; Tue, 11 Sep 2018 07:03:21 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 08D6861C52; Tue, 11 Sep 2018 14:03:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 4357561C0D; Tue, 11 Sep 2018 14:01:54 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 3733661C13; Tue, 11 Sep 2018 14:01:48 +0000 (UTC) Received: from forward102j.mail.yandex.net (forward102j.mail.yandex.net [5.45.198.243]) by lists.linaro.org (Postfix) with ESMTPS id 2C48F61C20 for ; Tue, 11 Sep 2018 14:00:28 +0000 (UTC) Received: from mxback12j.mail.yandex.net (mxback12j.mail.yandex.net [IPv6:2a02:6b8:0:1619::87]) by forward102j.mail.yandex.net (Yandex) with ESMTP id 4727856063D6 for ; Tue, 11 Sep 2018 17:00:22 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback12j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id ZIAFwPzxPu-0MsSaCNm; Tue, 11 Sep 2018 17:00:22 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id mBghUNGHde-0IS4tYw9; Tue, 11 Sep 2018 17:00:18 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 11 Sep 2018 14:00:11 +0000 Message-Id: <1536674416-8465-2-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> References: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 699 Subject: [lng-odp] [PATCH v1 1/6] linux-gen: queue: remove extra checks X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Remove unnecessary checks from critical sections of scheduled queue enqueue and dequeue operations. Parallelism improves when the number of instructions and (potential) cache misses decreases when holding the lock. Signed-off-by: Petri Savolainen --- /** Email created from pull request 699 (psavol:master-sched-optim-clean-ups) ** https://github.com/Linaro/odp/pull/699 ** Patch: https://github.com/Linaro/odp/pull/699.patch ** Base sha: 33fbc04b6373960ec3f84de4e7e7b34c49d71508 ** Merge commit sha: 32d7a11f22e6f2e1e378b653993c5377d4116d8f **/ platform/linux-generic/odp_queue_basic.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/platform/linux-generic/odp_queue_basic.c b/platform/linux-generic/odp_queue_basic.c index 7e8b7e34d..61cf8a56c 100644 --- a/platform/linux-generic/odp_queue_basic.c +++ b/platform/linux-generic/odp_queue_basic.c @@ -681,12 +681,6 @@ static inline int _sched_queue_enq_multi(odp_queue_t handle, LOCK(queue); - if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { - UNLOCK(queue); - ODP_ERR("Bad queue status\n"); - return -1; - } - num_enq = ring_st_enq_multi(ring_st, queue->s.ring_data, queue->s.ring_mask, buf_idx, num); @@ -712,7 +706,7 @@ static inline int _sched_queue_enq_multi(odp_queue_t handle, int sched_queue_deq(uint32_t queue_index, odp_event_t ev[], int max_num, int update_status) { - int num_deq; + int num_deq, status; ring_st_t *ring_st; queue_entry_t *queue = qentry_from_index(queue_index); int status_sync = sched_fn->status_sync; @@ -722,7 +716,9 @@ int sched_queue_deq(uint32_t queue_index, odp_event_t ev[], int max_num, LOCK(queue); - if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { + status = queue->s.status; + + if (odp_unlikely(status < QUEUE_STATUS_READY)) { /* Bad queue, or queue has been destroyed. * Scheduler finalizes queue destroy after this. */ UNLOCK(queue); @@ -734,10 +730,10 @@ int sched_queue_deq(uint32_t queue_index, odp_event_t ev[], int max_num, if (num_deq == 0) { /* Already empty queue */ - if (update_status && queue->s.status == QUEUE_STATUS_SCHED) { + if (update_status && status == QUEUE_STATUS_SCHED) { queue->s.status = QUEUE_STATUS_NOTSCHED; - if (status_sync) + if (odp_unlikely(status_sync)) sched_fn->unsched_queue(queue->s.index); } @@ -746,7 +742,7 @@ int sched_queue_deq(uint32_t queue_index, odp_event_t ev[], int max_num, return 0; } - if (status_sync && queue->s.type == ODP_QUEUE_TYPE_SCHED) + if (odp_unlikely(status_sync)) sched_fn->save_context(queue->s.index); UNLOCK(queue); From patchwork Tue Sep 11 14:00:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 146455 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp3678784ljw; Tue, 11 Sep 2018 07:04:19 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaTIYfvyrC7ER6aEgsG1XWWGbmTNA5vT7ONHi4ECodgPrXDnVVBxROwQNwGwmP3OB1uL7iW X-Received: by 2002:a37:18d9:: with SMTP id 86-v6mr19208950qky.350.1536674659492; Tue, 11 Sep 2018 07:04:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536674659; cv=none; d=google.com; s=arc-20160816; b=A5Ny5Ctwdmd0bmINGRhTv52ChqemH4WEcT+UJePAOZm+AMYtsvDgwkqBGECIkxIoRW UHFr0H3NA1uzPUO3qWzXjQwrBmbsYD2n43eipaIRY92jjndd8+yz2LpCr6xldh8+ZK22 EBmMqpTL7xSRq8RNgbUnSmlYb3a6lOtEdNwF8t/0xU7j4zH9X9cBF6apOPaEn+kHF9AM GPhJyNE+nA/W2mhSqWPsgRjQ2Pa+jJY3qBcExjoZ4dxUZtZO7VQTR6ydw3z3w/rXCih8 dtcuiTeVqPqHeTYcEKNYmXpAdd4W5H7cRVU94+CdrBltjaaPIJSaBnGb4RZ8dkbNV57X Ni3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to; bh=iIjJMiJZ7GeRN4TXwE4YRCJqSB90JX2MS32iTReLuYo=; b=un94aVFi6+A84tdtJADkWMRzzFwNfO/E+SI1icQMW+jHQG4htUGjPK7yVjd+P4GH+b qNSNFCHhDCqvt5+XhO4SslvA0DW2Olx144Qm/Oo2evMW1wX423LhmHyMqIz9u+k9UdCy jN/i4DAPLG7BsDgJha/epfC3QdpYp8LLZvf7e0lUz9EYv8o92UbNLW7Z5JZMbHdX6t/K 6XXaYMqWix09CTGfg+8j9jbB3iEqeT+www3O+5J7T8bqp8VjsG19r1QvYGGXlUllKQ5C AnL3x7xxPz4t0XPIy1Rt4F1N74MbOCjeTi44vQmzn7WqmL3EEcjyAmg90XTe9Jr0rOhc 09fw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id t41-v6si4544555qth.214.2018.09.11.07.04.19; Tue, 11 Sep 2018 07:04:19 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 33C4D61C1D; Tue, 11 Sep 2018 14:04:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id F401761C05; Tue, 11 Sep 2018 14:02:14 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id AB9F061C13; Tue, 11 Sep 2018 14:01:50 +0000 (UTC) Received: from forward104j.mail.yandex.net (forward104j.mail.yandex.net [5.45.198.247]) by lists.linaro.org (Postfix) with ESMTPS id 7FE0B61C1B for ; Tue, 11 Sep 2018 14:00:26 +0000 (UTC) Received: from mxback12j.mail.yandex.net (mxback12j.mail.yandex.net [IPv6:2a02:6b8:0:1619::87]) by forward104j.mail.yandex.net (Yandex) with ESMTP id 1961744597 for ; Tue, 11 Sep 2018 17:00:25 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback12j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id MO4JEmAnXG-0PsGF7ol; Tue, 11 Sep 2018 17:00:25 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id mBghUNGHde-0MSqMe8F; Tue, 11 Sep 2018 17:00:22 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 11 Sep 2018 14:00:12 +0000 Message-Id: <1536674416-8465-3-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> References: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 699 Subject: [lng-odp] [PATCH v1 2/6] linux-gen: sched: clean up local data struct X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Move stash variables into a struct. Use only 16 bits for thread id, which is enough for 64k threads. Signed-off-by: Petri Savolainen --- /** Email created from pull request 699 (psavol:master-sched-optim-clean-ups) ** https://github.com/Linaro/odp/pull/699 ** Patch: https://github.com/Linaro/odp/pull/699.patch ** Base sha: 33fbc04b6373960ec3f84de4e7e7b34c49d71508 ** Merge commit sha: 32d7a11f22e6f2e1e378b653993c5377d4116d8f **/ platform/linux-generic/odp_schedule_basic.c | 62 ++++++++++++--------- 1 file changed, 35 insertions(+), 27 deletions(-) diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index e329a8e8c..89c0a5c42 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -84,6 +84,10 @@ ODP_STATIC_ASSERT(CHECK_IS_POWER2(ODP_CONFIG_QUEUES), ODP_STATIC_ASSERT(CHECK_IS_POWER2(MAX_RING_SIZE), "Ring_size_is_not_power_of_two"); +/* Thread ID is saved into uint16_t variable */ +ODP_STATIC_ASSERT(ODP_THREAD_COUNT_MAX < (64 * 1024), + "Max_64k_threads_supported"); + /* Mask of queues per priority */ typedef uint8_t pri_mask_t; @@ -118,19 +122,22 @@ ODP_STATIC_ASSERT(sizeof(lock_called_t) == sizeof(uint32_t), "Lock_called_values_do_not_fit_in_uint32"); /* Scheduler local data */ -typedef struct { - int thr; - uint16_t stash_num; - uint16_t stash_index; +typedef struct ODP_ALIGNED_CACHE { + uint16_t thr; + uint16_t pause; uint16_t grp_round; uint16_t spread_round; - uint32_t stash_qi; - odp_queue_t stash_queue; - odp_event_t stash_ev[BURST_SIZE_MAX]; + + struct { + uint16_t num_ev; + uint16_t ev_index; + uint32_t qi; + odp_queue_t queue; + odp_event_t ev[BURST_SIZE_MAX]; + } stash; uint32_t grp_epoch; uint16_t num_grp; - uint16_t pause; uint8_t grp[NUM_SCHED_GRPS]; uint8_t spread_tbl[SPREAD_TBL_SIZE]; uint8_t grp_weight[GRP_WEIGHT_TBL_SIZE]; @@ -304,8 +311,8 @@ static void sched_local_init(void) memset(&sched_local, 0, sizeof(sched_local_t)); sched_local.thr = odp_thread_id(); - sched_local.stash_queue = ODP_QUEUE_INVALID; - sched_local.stash_qi = PRIO_QUEUE_EMPTY; + sched_local.stash.queue = ODP_QUEUE_INVALID; + sched_local.stash.qi = PRIO_QUEUE_EMPTY; sched_local.ordered.src_queue = NULL_INDEX; spread = prio_spread_index(sched_local.thr); @@ -445,7 +452,7 @@ static int schedule_init_local(void) static int schedule_term_local(void) { - if (sched_local.stash_num) { + if (sched_local.stash.num_ev) { ODP_ERR("Locally pre-scheduled events exist.\n"); return -1; } @@ -618,9 +625,9 @@ static void schedule_pktio_start(int pktio_index, int num_pktin, static void schedule_release_atomic(void) { - uint32_t qi = sched_local.stash_qi; + uint32_t qi = sched_local.stash.qi; - if (qi != PRIO_QUEUE_EMPTY && sched_local.stash_num == 0) { + if (qi != PRIO_QUEUE_EMPTY && sched_local.stash.num_ev == 0) { int grp = sched->queue[qi].grp; int prio = sched->queue[qi].prio; int spread = sched->queue[qi].spread; @@ -629,7 +636,7 @@ static void schedule_release_atomic(void) /* Release current atomic queue */ ring_enq(ring, sched->ring_mask, qi); - sched_local.stash_qi = PRIO_QUEUE_EMPTY; + sched_local.stash.qi = PRIO_QUEUE_EMPTY; } } @@ -717,7 +724,8 @@ static void schedule_release_ordered(void) queue_index = sched_local.ordered.src_queue; - if (odp_unlikely((queue_index == NULL_INDEX) || sched_local.stash_num)) + if (odp_unlikely((queue_index == NULL_INDEX) || + sched_local.stash.num_ev)) return; release_ordered(); @@ -735,10 +743,10 @@ static inline int copy_from_stash(odp_event_t out_ev[], unsigned int max) { int i = 0; - while (sched_local.stash_num && max) { - out_ev[i] = sched_local.stash_ev[sched_local.stash_index]; - sched_local.stash_index++; - sched_local.stash_num--; + while (sched_local.stash.num_ev && max) { + out_ev[i] = sched_local.stash.ev[sched_local.stash.ev_index]; + sched_local.stash.ev_index++; + sched_local.stash.num_ev--; max--; i++; } @@ -889,7 +897,7 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], int pktin; unsigned int max_deq = max_burst; int stashed = 1; - odp_event_t *ev_tbl = sched_local.stash_ev; + odp_event_t *ev_tbl = sched_local.stash.ev; if (id >= num_spread) id = 0; @@ -984,7 +992,7 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], } else if (queue_is_atomic(qi)) { /* Hold queue during atomic access */ - sched_local.stash_qi = qi; + sched_local.stash.qi = qi; } else { /* Continue scheduling the queue */ ring_enq(ring, ring_mask, qi); @@ -993,12 +1001,12 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], handle = queue_from_index(qi); if (stashed) { - sched_local.stash_num = num; - sched_local.stash_index = 0; - sched_local.stash_queue = handle; + sched_local.stash.num_ev = num; + sched_local.stash.ev_index = 0; + sched_local.stash.queue = handle; ret = copy_from_stash(out_ev, max_num); } else { - sched_local.stash_num = 0; + sched_local.stash.num_ev = 0; ret = num; } @@ -1025,11 +1033,11 @@ static inline int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], uint16_t spread_round, grp_round; uint32_t epoch; - if (sched_local.stash_num) { + if (sched_local.stash.num_ev) { ret = copy_from_stash(out_ev, max_num); if (out_queue) - *out_queue = sched_local.stash_queue; + *out_queue = sched_local.stash.queue; return ret; } From patchwork Tue Sep 11 14:00:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 146453 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp3676122ljw; Tue, 11 Sep 2018 07:02:07 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYp4Hzr9dcsvM3NPhCi7p897P9Ie8hlyjfZPiIgkwH4I+ieXYLCpeXDVtjFQ6EHj3v9i1Iq X-Received: by 2002:a37:5f03:: with SMTP id t3-v6mr18582289qkb.193.1536674527562; Tue, 11 Sep 2018 07:02:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536674527; cv=none; d=google.com; s=arc-20160816; b=GKzUd5sJdGmkB8uzHjSDtD5sngJLyWQMh9smxpNWZucndl6u/o4u60iUT8YXYIYvUH GTs3iA9F3H/e1Wdir/Gb9lyMgasjLdAxzvQiRueyPKwv6dPI/V9eeS6MB4oEKqk1Fe+T yOescUZDVWlsaHKNfB4HWdySc9iDbHWE0isYb00GA0VMtPamIkqSJ/uFs1TqlrdLfKBS s8nbCVz/B1uqRWC0gk5+I0klth1taSN/jplDi7/RFp5ywoX6dbVgi+AaeXYNx/GiFvHr Ds9lRllQNduWngBsabrg67xyzF92qwsfWfX+1tE8O74SAeKXGGnELfVCpVNczIOxwS7E 0NSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to; bh=qKt6mKX8d5wJQqEhIYmq+l6I/0k2yXPqUp/V/mm2Nx4=; b=jtmQdlUWgiXJuXoClqMox4MmyGPudANDNgm34QPxjcpONeCBe/OSbX6QrDTIIkI4uL bH5Q1BTsUr9i0DjxIAH9D6VUBh1AhK+zb18EFtzzeAhLSbwSSOzanIy7Ey0Hy6s6Hj2R +tEHHC58E2frzzSC+7YujNYg4Yy7uZxje8I1GGOGV8SlQn/PSzSHyxlkUWIFVi1756GL tkYuxU7g6O/Y8tCixwMGk5UmPPptjmUJUq0/aT3XcTvmuG7kpd4vQzBnXX4odh2KZ3Kt As8BELSbovNr2b6cLHa53Z0j1wGgUCSAkhyPb+nwu9kRKgZBpQkRT2Jpr9Cn5JGuxfqE 785Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id z4-v6si12051675qkd.401.2018.09.11.07.02.06; Tue, 11 Sep 2018 07:02:07 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id CCFC061C11; Tue, 11 Sep 2018 14:02:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id C86AB60604; Tue, 11 Sep 2018 14:01:51 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 6DAB961C0C; Tue, 11 Sep 2018 14:01:47 +0000 (UTC) Received: from forward103j.mail.yandex.net (forward103j.mail.yandex.net [5.45.198.246]) by lists.linaro.org (Postfix) with ESMTPS id 45E1861C1D for ; Tue, 11 Sep 2018 14:00:27 +0000 (UTC) Received: from mxback18j.mail.yandex.net (mxback18j.mail.yandex.net [IPv6:2a02:6b8:0:1619::94]) by forward103j.mail.yandex.net (Yandex) with ESMTP id 4781734C79D6 for ; Tue, 11 Sep 2018 17:00:26 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback18j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id H8muSj5zqk-0Q388mY9; Tue, 11 Sep 2018 17:00:26 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id mBghUNGHde-0PSSGdMB; Tue, 11 Sep 2018 17:00:25 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 11 Sep 2018 14:00:13 +0000 Message-Id: <1536674416-8465-4-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> References: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 699 Subject: [lng-odp] [PATCH v1 3/6] linux-gen: sched: single variable for sync context status X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Use single thread local variable to keep track if a synchronization context is held and the type of the context (atomic or ordered). Performance is improved as sync context status is located on single (the first) cache line of sched_local_t. Signed-off-by: Petri Savolainen --- /** Email created from pull request 699 (psavol:master-sched-optim-clean-ups) ** https://github.com/Linaro/odp/pull/699 ** Patch: https://github.com/Linaro/odp/pull/699.patch ** Base sha: 33fbc04b6373960ec3f84de4e7e7b34c49d71508 ** Merge commit sha: 32d7a11f22e6f2e1e378b653993c5377d4116d8f **/ platform/linux-generic/odp_schedule_basic.c | 130 +++++++++++--------- 1 file changed, 72 insertions(+), 58 deletions(-) diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index 89c0a5c42..46ae7f1c1 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -30,6 +30,9 @@ #include #include +/* No synchronization context */ +#define NO_SYNC_CONTEXT ODP_SCHED_SYNC_PARALLEL + /* Number of priority levels */ #define NUM_PRIO 8 @@ -124,7 +127,8 @@ ODP_STATIC_ASSERT(sizeof(lock_called_t) == sizeof(uint32_t), /* Scheduler local data */ typedef struct ODP_ALIGNED_CACHE { uint16_t thr; - uint16_t pause; + uint8_t pause; + uint8_t sync_ctx; uint16_t grp_round; uint16_t spread_round; @@ -241,9 +245,6 @@ static sched_global_t *sched; /* Thread local scheduler context */ static __thread sched_local_t sched_local; -/* Function prototypes */ -static inline void schedule_release_context(void); - static int read_config_file(sched_global_t *sched) { const char *str; @@ -311,6 +312,7 @@ static void sched_local_init(void) memset(&sched_local, 0, sizeof(sched_local_t)); sched_local.thr = odp_thread_id(); + sched_local.sync_ctx = NO_SYNC_CONTEXT; sched_local.stash.queue = ODP_QUEUE_INVALID; sched_local.stash.qi = PRIO_QUEUE_EMPTY; sched_local.ordered.src_queue = NULL_INDEX; @@ -450,17 +452,6 @@ static int schedule_init_local(void) return 0; } -static int schedule_term_local(void) -{ - if (sched_local.stash.num_ev) { - ODP_ERR("Locally pre-scheduled events exist.\n"); - return -1; - } - - schedule_release_context(); - return 0; -} - static inline void grp_update_mask(int grp, const odp_thrmask_t *new_mask) { odp_thrmask_copy(&sched->sched_grp[grp].mask, new_mask); @@ -565,14 +556,9 @@ static int schedule_init_queue(uint32_t queue_index, return 0; } -static inline int queue_is_atomic(uint32_t queue_index) +static inline uint8_t sched_sync_type(uint32_t queue_index) { - return sched->queue[queue_index].sync == ODP_SCHED_SYNC_ATOMIC; -} - -static inline int queue_is_ordered(uint32_t queue_index) -{ - return sched->queue[queue_index].sync == ODP_SCHED_SYNC_ORDERED; + return sched->queue[queue_index].sync; } static void schedule_destroy_queue(uint32_t queue_index) @@ -584,7 +570,7 @@ static void schedule_destroy_queue(uint32_t queue_index) sched->queue[queue_index].prio = 0; sched->queue[queue_index].spread = 0; - if (queue_is_ordered(queue_index) && + if ((sched_sync_type(queue_index) == ODP_SCHED_SYNC_ORDERED) && odp_atomic_load_u64(&sched->order[queue_index].ctx) != odp_atomic_load_u64(&sched->order[queue_index].next_ctx)) ODP_ERR("queue reorder incomplete\n"); @@ -623,21 +609,26 @@ static void schedule_pktio_start(int pktio_index, int num_pktin, } } -static void schedule_release_atomic(void) +static inline void release_atomic(void) { - uint32_t qi = sched_local.stash.qi; + uint32_t qi = sched_local.stash.qi; + int grp = sched->queue[qi].grp; + int prio = sched->queue[qi].prio; + int spread = sched->queue[qi].spread; + ring_t *ring = &sched->prio_q[grp][prio][spread].ring; - if (qi != PRIO_QUEUE_EMPTY && sched_local.stash.num_ev == 0) { - int grp = sched->queue[qi].grp; - int prio = sched->queue[qi].prio; - int spread = sched->queue[qi].spread; - ring_t *ring = &sched->prio_q[grp][prio][spread].ring; + /* Release current atomic queue */ + ring_enq(ring, sched->ring_mask, qi); - /* Release current atomic queue */ - ring_enq(ring, sched->ring_mask, qi); + /* We don't hold sync context anymore */ + sched_local.sync_ctx = NO_SYNC_CONTEXT; +} - sched_local.stash.qi = PRIO_QUEUE_EMPTY; - } +static void schedule_release_atomic(void) +{ + if (sched_local.sync_ctx == ODP_SCHED_SYNC_ATOMIC && + sched_local.stash.num_ev == 0) + release_atomic(); } static inline int ordered_own_turn(uint32_t queue_index) @@ -709,9 +700,11 @@ static inline void release_ordered(void) } sched_local.ordered.lock_called.all = 0; - sched_local.ordered.src_queue = NULL_INDEX; sched_local.ordered.in_order = 0; + /* We don't hold sync context anymore */ + sched_local.sync_ctx = NO_SYNC_CONTEXT; + ordered_stash_release(); /* Next thread can continue processing */ @@ -720,23 +713,26 @@ static inline void release_ordered(void) static void schedule_release_ordered(void) { - uint32_t queue_index; - - queue_index = sched_local.ordered.src_queue; - - if (odp_unlikely((queue_index == NULL_INDEX) || + if (odp_unlikely((sched_local.sync_ctx != ODP_SCHED_SYNC_ORDERED) || sched_local.stash.num_ev)) return; release_ordered(); } -static inline void schedule_release_context(void) +static int schedule_term_local(void) { - if (sched_local.ordered.src_queue != NULL_INDEX) - release_ordered(); - else + if (sched_local.stash.num_ev) { + ODP_ERR("Locally pre-scheduled events exist.\n"); + return -1; + } + + if (sched_local.sync_ctx == ODP_SCHED_SYNC_ATOMIC) schedule_release_atomic(); + else if (sched_local.sync_ctx == ODP_SCHED_SYNC_ORDERED) + schedule_release_ordered(); + + return 0; } static inline int copy_from_stash(odp_event_t out_ev[], unsigned int max) @@ -758,13 +754,22 @@ static int schedule_ord_enq_multi(odp_queue_t dst_queue, void *buf_hdr[], int num, int *ret) { int i; - uint32_t stash_num = sched_local.ordered.stash_num; - queue_entry_t *dst_qentry = qentry_from_handle(dst_queue); - uint32_t src_queue = sched_local.ordered.src_queue; + uint32_t stash_num; + queue_entry_t *dst_qentry; + uint32_t src_queue; - if ((src_queue == NULL_INDEX) || sched_local.ordered.in_order) + /* This check is done for every queue enqueue operation, also for plain + * queues. Return fast when not holding a scheduling context. */ + if (odp_likely(sched_local.sync_ctx != ODP_SCHED_SYNC_ORDERED)) return 0; + if (sched_local.ordered.in_order) + return 0; + + src_queue = sched_local.ordered.src_queue; + stash_num = sched_local.ordered.stash_num; + dst_qentry = qentry_from_handle(dst_queue); + if (ordered_own_turn(src_queue)) { /* Own turn, so can do enqueue directly. */ sched_local.ordered.in_order = 1; @@ -891,7 +896,7 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], for (i = 0; i < num_spread;) { int num; - int ordered; + uint8_t sync_ctx, ordered; odp_queue_t handle; ring_t *ring; int pktin; @@ -921,7 +926,8 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], continue; } - ordered = queue_is_ordered(qi); + sync_ctx = sched_sync_type(qi); + ordered = (sync_ctx == ODP_SCHED_SYNC_ORDERED); /* When application's array is larger than max burst * size, output all events directly there. Also, ordered @@ -989,10 +995,12 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], /* Continue scheduling ordered queues */ ring_enq(ring, ring_mask, qi); + sched_local.sync_ctx = sync_ctx; - } else if (queue_is_atomic(qi)) { + } else if (sync_ctx == ODP_SCHED_SYNC_ATOMIC) { /* Hold queue during atomic access */ sched_local.stash.qi = qi; + sched_local.sync_ctx = sync_ctx; } else { /* Continue scheduling the queue */ ring_enq(ring, ring_mask, qi); @@ -1042,7 +1050,11 @@ static inline int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], return ret; } - schedule_release_context(); + /* Release schedule context */ + if (sched_local.sync_ctx == ODP_SCHED_SYNC_ATOMIC) + release_atomic(); + else if (sched_local.sync_ctx == ODP_SCHED_SYNC_ORDERED) + release_ordered(); if (odp_unlikely(sched_local.pause)) return 0; @@ -1141,14 +1153,10 @@ static int schedule_multi(odp_queue_t *out_queue, uint64_t wait, static inline void order_lock(void) { - uint32_t queue_index; - - queue_index = sched_local.ordered.src_queue; - - if (queue_index == NULL_INDEX) + if (sched_local.sync_ctx != ODP_SCHED_SYNC_ORDERED) return; - wait_for_order(queue_index); + wait_for_order(sched_local.ordered.src_queue); } static void order_unlock(void) @@ -1160,6 +1168,9 @@ static void schedule_order_lock(uint32_t lock_index) odp_atomic_u64_t *ord_lock; uint32_t queue_index; + if (sched_local.sync_ctx != ODP_SCHED_SYNC_ORDERED) + return; + queue_index = sched_local.ordered.src_queue; ODP_ASSERT(queue_index != NULL_INDEX && @@ -1187,6 +1198,9 @@ static void schedule_order_unlock(uint32_t lock_index) odp_atomic_u64_t *ord_lock; uint32_t queue_index; + if (sched_local.sync_ctx != ODP_SCHED_SYNC_ORDERED) + return; + queue_index = sched_local.ordered.src_queue; ODP_ASSERT(queue_index != NULL_INDEX && From patchwork Tue Sep 11 14:00:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 146456 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp3679900ljw; Tue, 11 Sep 2018 07:05:14 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYbRJGj5y4xadLgwZW8yfk5KX/XGDq4Wdi76E1ajXzVte4cfd+EtXggkMnRaGWLa8O0pA+1 X-Received: by 2002:ac8:6a05:: with SMTP id t5-v6mr20208875qtr.249.1536674713910; Tue, 11 Sep 2018 07:05:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536674713; cv=none; d=google.com; s=arc-20160816; b=pOTiRXykf0cuAOeuc0nE5n56W5P5Xx5KpqJ9N3PpxKACvUIysnEXwcgh9/F9RCAUSh CpSzCsLXsXWPoOfLhIA0qf1GR57/Fgh+BcF3+kC0lhCSHbWCtjPaIG+oKlqbs8+uIy/f 7g0y/nLtIPkwXELbbDBjukVMEbIFsOS8XjTCPJDX9oIQ8X0woPO0qx889oNxprlHC4/G geTzbT9pgUbdEran768Bm1qXlnvBWH/Cqo/kele1C78l0E/FdRd0zNTDYenVzCzi1WQB 5R8zz/UfRN6jsXKPfK7wl9OE3SmEL55ot8uYNRi//HP+2SVhVaTIzlrpI5AZkbP8bVRr nO5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to; bh=9jSbNTXbe3JNm3+vlCc8gsHxlEgiovbdXjHTo3QXsSE=; b=D7FjIOMLbPjKh/cDHnqR+sJCBTWHxbtGTdz51WSNh0ydIaBWX4CY+V8VtrpjOzp7zD cS8EV7kK1JJtTL0pwQt2kgqYG+K3nzsViZLWd+sfJHWSU3EHrs8fxt4Nhg/AD3AsmZ21 oY0vhMwoyHp6eFs0PFK0ZvaFg+bUgWDa9U4YNaPV1zg+GWwgNKBPZsZZcJOXBVgcR+pL XOYxhju5jTUV9w7vzCGyen/alp36BQqqcFa5lQvU2ROYBX9P1GKf32TKHHWCTSFcs9Dn MWBV1/bfeP5DvN5ET6TSuO9es1vjBeIWZEVTMW+DKiu7vs2AGLOY5xs2FQ0KvY2Hn1GK 31dg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id k2-v6si13624102qvb.251.2018.09.11.07.05.13; Tue, 11 Sep 2018 07:05:13 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 94CF661C20; Tue, 11 Sep 2018 14:05:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 47A2861C21; Tue, 11 Sep 2018 14:02:18 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 2DDC461C1B; Tue, 11 Sep 2018 14:01:54 +0000 (UTC) Received: from forward102o.mail.yandex.net (forward102o.mail.yandex.net [37.140.190.182]) by lists.linaro.org (Postfix) with ESMTPS id 9AC7F61C0D for ; Tue, 11 Sep 2018 14:00:31 +0000 (UTC) Received: from mxback15j.mail.yandex.net (mxback15j.mail.yandex.net [IPv6:2a02:6b8:0:1619::91]) by forward102o.mail.yandex.net (Yandex) with ESMTP id A03655A0089F for ; Tue, 11 Sep 2018 17:00:27 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback15j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id TiCk8qKKKa-0RR8r4s5; Tue, 11 Sep 2018 17:00:27 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id mBghUNGHde-0QSqumoJ; Tue, 11 Sep 2018 17:00:26 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 11 Sep 2018 14:00:14 +0000 Message-Id: <1536674416-8465-5-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> References: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 699 Subject: [lng-odp] [PATCH v1 4/6] linux-gen: sched: remove queue_destroy_finalize callback X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Scheduled queue dequeue function calls directly the scheduler queue destroy callback. Sched_queue_deq() usage is simpler when the extra round of callbacks is removed. Signed-off-by: Petri Savolainen --- /** Email created from pull request 699 (psavol:master-sched-optim-clean-ups) ** https://github.com/Linaro/odp/pull/699 ** Patch: https://github.com/Linaro/odp/pull/699.patch ** Base sha: 33fbc04b6373960ec3f84de4e7e7b34c49d71508 ** Merge commit sha: 32d7a11f22e6f2e1e378b653993c5377d4116d8f **/ .../include/odp_queue_basic_internal.h | 1 - platform/linux-generic/odp_queue_basic.c | 20 +++------ platform/linux-generic/odp_schedule_basic.c | 11 +---- platform/linux-generic/odp_schedule_iquery.c | 35 +++++++-------- platform/linux-generic/odp_schedule_sp.c | 43 ++++++++++--------- 5 files changed, 47 insertions(+), 63 deletions(-) diff --git a/platform/linux-generic/include/odp_queue_basic_internal.h b/platform/linux-generic/include/odp_queue_basic_internal.h index 46b747955..41ca424c7 100644 --- a/platform/linux-generic/include/odp_queue_basic_internal.h +++ b/platform/linux-generic/include/odp_queue_basic_internal.h @@ -113,7 +113,6 @@ static inline queue_entry_t *qentry_from_handle(odp_queue_t handle) void queue_spsc_init(queue_entry_t *queue, uint32_t queue_size); /* Functions for schedulers */ -void sched_queue_destroy_finalize(uint32_t queue_index); void sched_queue_set_status(uint32_t queue_index, int status); int sched_queue_deq(uint32_t queue_index, odp_event_t ev[], int num, int update_status); diff --git a/platform/linux-generic/odp_queue_basic.c b/platform/linux-generic/odp_queue_basic.c index 61cf8a56c..3f00cc118 100644 --- a/platform/linux-generic/odp_queue_basic.c +++ b/platform/linux-generic/odp_queue_basic.c @@ -353,19 +353,6 @@ static odp_queue_t queue_create(const char *name, return handle; } -void sched_queue_destroy_finalize(uint32_t queue_index) -{ - queue_entry_t *queue = qentry_from_index(queue_index); - - LOCK(queue); - - if (queue->s.status == QUEUE_STATUS_DESTROYED) { - queue->s.status = QUEUE_STATUS_FREE; - sched_fn->destroy_queue(queue_index); - } - UNLOCK(queue); -} - void sched_queue_set_status(uint32_t queue_index, int status) { queue_entry_t *queue = qentry_from_index(queue_index); @@ -720,7 +707,12 @@ int sched_queue_deq(uint32_t queue_index, odp_event_t ev[], int max_num, if (odp_unlikely(status < QUEUE_STATUS_READY)) { /* Bad queue, or queue has been destroyed. - * Scheduler finalizes queue destroy after this. */ + * Inform scheduler about a destroyed queue. */ + if (queue->s.status == QUEUE_STATUS_DESTROYED) { + queue->s.status = QUEUE_STATUS_FREE; + sched_fn->destroy_queue(queue_index); + } + UNLOCK(queue); return -1; } diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index 46ae7f1c1..6ed1f8b49 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -402,11 +402,6 @@ static int schedule_init_global(void) return 0; } -static inline void queue_destroy_finalize(uint32_t qi) -{ - sched_queue_destroy_finalize(qi); -} - static int schedule_term_global(void) { int ret = 0; @@ -427,9 +422,6 @@ static int schedule_term_global(void) num = sched_queue_deq(qi, events, 1, 1); - if (num < 0) - queue_destroy_finalize(qi); - if (num > 0) ODP_ERR("Queue not empty\n"); } @@ -944,10 +936,9 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], num = sched_queue_deq(qi, ev_tbl, max_deq, !pktin); - if (num < 0) { + if (odp_unlikely(num < 0)) { /* Destroyed queue. Continue scheduling the same * priority queue. */ - sched_queue_destroy_finalize(qi); continue; } diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index 7dde77844..f76942ff3 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -209,6 +209,7 @@ struct sched_thread_local { * in the same priority level. */ odp_rwlock_t lock; + int r_locked; queue_index_sparse_t indexes[NUM_SCHED_PRIO]; sparse_bitmap_iterator_t iterators[NUM_SCHED_PRIO]; @@ -292,9 +293,7 @@ static int schedule_term_global(void) if (sched->availables[i]) count = sched_queue_deq(i, events, 1, 1); - if (count < 0) - sched_queue_destroy_finalize(i); - else if (count > 0) + if (count > 0) ODP_ERR("Queue (%d) not empty\n", i); } @@ -526,7 +525,14 @@ static void destroy_sched_queue(uint32_t queue_index) return; } + if (thread_local.r_locked) + odp_rwlock_read_unlock(&thread_local.lock); + __destroy_sched_queue(G, queue_index); + + if (thread_local.r_locked) + odp_rwlock_read_lock(&thread_local.lock); + odp_rwlock_write_unlock(&G->lock); if (sched->queues[queue_index].sync == ODP_SCHED_SYNC_ORDERED && @@ -614,9 +620,6 @@ static int schedule_pktio_stop(int pktio, int pktin ODP_UNUSED) return remains; } -#define DO_SCHED_LOCK() odp_rwlock_read_lock(&thread_local.lock) -#define DO_SCHED_UNLOCK() odp_rwlock_read_unlock(&thread_local.lock) - static inline bool do_schedule_prio(int prio); static inline int pop_cache_events(odp_event_t ev[], unsigned int max) @@ -720,7 +723,9 @@ static int do_schedule(odp_queue_t *out_queue, if (odp_unlikely(thread_local.pause)) return count; - DO_SCHED_LOCK(); + odp_rwlock_read_lock(&thread_local.lock); + thread_local.r_locked = 1; + /* Schedule events */ for (prio = 0; prio < NUM_SCHED_PRIO; prio++) { /* Round robin iterate the interested queue @@ -732,11 +737,14 @@ static int do_schedule(odp_queue_t *out_queue, count = pop_cache_events(out_ev, max_num); assign_queue_handle(out_queue); - DO_SCHED_UNLOCK(); + + odp_rwlock_read_unlock(&thread_local.lock); + thread_local.r_locked = 0; return count; } - DO_SCHED_UNLOCK(); + odp_rwlock_read_unlock(&thread_local.lock); + thread_local.r_locked = 0; /* Poll packet input when there are no events */ pktio_poll_input(); @@ -1536,14 +1544,7 @@ static inline int consume_queue(int prio, unsigned int queue_index) count = sched_queue_deq(queue_index, cache->stash, max, 1); - if (count < 0) { - DO_SCHED_UNLOCK(); - sched_queue_destroy_finalize(queue_index); - DO_SCHED_LOCK(); - return 0; - } - - if (count == 0) + if (count <= 0) return 0; cache->top = &cache->stash[0]; diff --git a/platform/linux-generic/odp_schedule_sp.c b/platform/linux-generic/odp_schedule_sp.c index 7932e1860..8ddd1e94e 100644 --- a/platform/linux-generic/odp_schedule_sp.c +++ b/platform/linux-generic/odp_schedule_sp.c @@ -223,12 +223,21 @@ static int init_local(void) static int term_global(void) { + odp_event_t event; int qi, ret = 0; for (qi = 0; qi < NUM_QUEUE; qi++) { + int report = 1; + if (sched_global->queue_cmd[qi].s.init) { - /* todo: dequeue until empty ? */ - sched_queue_destroy_finalize(qi); + while (sched_queue_deq(qi, &event, 1, 1) > 0) { + if (report) { + ODP_ERR("Queue not empty\n"); + report = 0; + } + odp_event_free(event); + } + } } @@ -564,28 +573,20 @@ static int schedule_multi(odp_queue_t *from, uint64_t wait, qi = cmd->s.index; num = sched_queue_deq(qi, events, 1, 1); - if (num > 0) { - sched_local.cmd = cmd; - - if (from) - *from = queue_from_index(qi); - - return num; - } - - if (num < 0) { - /* Destroyed queue */ - sched_queue_destroy_finalize(qi); + if (num <= 0) { + /* Destroyed or empty queue. Remove empty queue from + * scheduling. A dequeue operation to on an already + * empty queue moves it to NOTSCHED state and + * sched_queue() will be called on next enqueue. */ continue; } - if (num == 0) { - /* Remove empty queue from scheduling. A dequeue - * operation to on an already empty queue moves - * it to NOTSCHED state and sched_queue() will - * be called on next enqueue. */ - continue; - } + sched_local.cmd = cmd; + + if (from) + *from = queue_from_index(qi); + + return num; } } From patchwork Tue Sep 11 14:00:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 146457 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp3680261ljw; Tue, 11 Sep 2018 07:05:32 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZONexkhT+GM1pJbPhQk0Sw2IaOn53RH+vbq2gnLmPKf5Bbb+YP+qXBlT+AW48stbdGRF4N X-Received: by 2002:aed:2a13:: with SMTP id c19-v6mr19858244qtd.147.1536674731921; Tue, 11 Sep 2018 07:05:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536674731; cv=none; d=google.com; s=arc-20160816; b=BBDwvaUsXen/tRN3qkr/IrFA/sFzlT1DgDffXMG8TvNc41m99TCuJ6ELwwGXgqwya+ GDIFXQ7JXfg4+UDZyl+QY+BWe8sjy1+vv53sO/fLb0m64qwHd2sD5/97S5aJlUxdT2uj fjdffwPk01C78vZxmKig/7iOXzOfsfYUZWJYKnpGkLX4P43r0vzjGMk6Ug/G3GCJyuNg PlJJlpkfFtGltHDREcOyvdc1ICeWQCMCAen6Mq2s+B0YaxGOxp6xkBWufgreUm8axZ2P zZhxwVlPn9TN/MhuNMb/+zsIQqV9BxWPHFQ2oinhgMJ6ruFHyuADcrI9GYnIZ/cr7ssE mzNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to; bh=SS4g/QRXYFr215gSnfJe7699YRSnNHsmfNxPzgJ7ozY=; b=ZA+u4MzoiWwGSSUSn/ToWhKfrkIA/pHTM5ahlKIl8uQ9n2JhGd3rECEq1G8y8fpdy9 n+aX7qceZpSbN4ApawmOEzvZtJgfzGwWYD8Bz1rto2hAgicWJRVgF1t8+Kl1hgKf77+j QnwRp1SoewZGOqYsBzF9M6UJjk4CK0zACLrKq4VE26UscgNV8sFI5HHkYc1DQTNHmwDR Cxb3lC35o+HtbQuojXDpsSTYhoLkpB2Kc2dPqhj+7Jk4XR0QAVU4f1xoTVMpaOKMXidz oO98Coax7EWi1+4eD9lmD30BAFyh5qijsCg7KDH1MuhJxdkRPEHCsrQHMsjtHD4/VZRX pDLQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id p33-v6si5067804qtd.201.2018.09.11.07.05.31; Tue, 11 Sep 2018 07:05:31 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 869F661C20; Tue, 11 Sep 2018 14:05:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 8DA0261C28; Tue, 11 Sep 2018 14:02:21 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id BACCE61C20; Tue, 11 Sep 2018 14:01:54 +0000 (UTC) Received: from forward106j.mail.yandex.net (forward106j.mail.yandex.net [5.45.198.249]) by lists.linaro.org (Postfix) with ESMTPS id 1375D61C05 for ; Tue, 11 Sep 2018 14:00:33 +0000 (UTC) Received: from mxback19j.mail.yandex.net (mxback19j.mail.yandex.net [IPv6:2a02:6b8:0:1619::95]) by forward106j.mail.yandex.net (Yandex) with ESMTP id C8A8D1805613 for ; Tue, 11 Sep 2018 17:00:29 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback19j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id 2F46waNuvL-0T0K8Tpc; Tue, 11 Sep 2018 17:00:29 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id mBghUNGHde-0RSq5fFL; Tue, 11 Sep 2018 17:00:27 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 11 Sep 2018 14:00:15 +0000 Message-Id: <1536674416-8465-6-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> References: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 699 Subject: [lng-odp] [PATCH v1 5/6] linux-gen: sched: stash ring pointer X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Save ring pointer into stash to avoid table lookups when releasing the atomic context. Signed-off-by: Petri Savolainen --- /** Email created from pull request 699 (psavol:master-sched-optim-clean-ups) ** https://github.com/Linaro/odp/pull/699 ** Patch: https://github.com/Linaro/odp/pull/699.patch ** Base sha: 33fbc04b6373960ec3f84de4e7e7b34c49d71508 ** Merge commit sha: 32d7a11f22e6f2e1e378b653993c5377d4116d8f **/ platform/linux-generic/odp_schedule_basic.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index 6ed1f8b49..77fee74dd 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -137,6 +137,7 @@ typedef struct ODP_ALIGNED_CACHE { uint16_t ev_index; uint32_t qi; odp_queue_t queue; + ring_t *ring; odp_event_t ev[BURST_SIZE_MAX]; } stash; @@ -604,10 +605,7 @@ static void schedule_pktio_start(int pktio_index, int num_pktin, static inline void release_atomic(void) { uint32_t qi = sched_local.stash.qi; - int grp = sched->queue[qi].grp; - int prio = sched->queue[qi].prio; - int spread = sched->queue[qi].spread; - ring_t *ring = &sched->prio_q[grp][prio][spread].ring; + ring_t *ring = sched_local.stash.ring; /* Release current atomic queue */ ring_enq(ring, sched->ring_mask, qi); @@ -990,8 +988,9 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], } else if (sync_ctx == ODP_SCHED_SYNC_ATOMIC) { /* Hold queue during atomic access */ - sched_local.stash.qi = qi; - sched_local.sync_ctx = sync_ctx; + sched_local.stash.qi = qi; + sched_local.stash.ring = ring; + sched_local.sync_ctx = sync_ctx; } else { /* Continue scheduling the queue */ ring_enq(ring, ring_mask, qi); From patchwork Tue Sep 11 14:00:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 146458 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp3681249ljw; Tue, 11 Sep 2018 07:06:18 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZGTtNmYApnf3+/hT5XKkaFr2uQiXqNwePuQyjYlqQtXZDo/QF3zbVZOWJg50LbZ3AH+YFr X-Received: by 2002:ac8:c0e:: with SMTP id k14-v6mr19735849qti.20.1536674778621; Tue, 11 Sep 2018 07:06:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536674778; cv=none; d=google.com; s=arc-20160816; b=I5oRi2Ve4DdsehmZUwcQEaCNZduL/xxO1NK/A5wEfbH4n8qM8OrM/kwGxZjXMrJ3gu SBNPiGEkzpSiPoq+NfQfkIoKd2tjEnO7y2R5Tpr3eYm8iqyd06l9bWsg0AGfQdwkms3E zE8mnTH9v+WZ6yLXMZuC/TM4cQXwOf19KFLgBXLLR1g+Qqkk+OGRChPnUeOVTgaa6tCD w1vqmKJrGuzXofiTkGocIwlQGuanYFN0c1SaS18qOHQb1TMywcnqO6A3fiHVf7wfEeqI EHglrt5Qlnzuodg9wTylDbGwgY+F/xMXH0VvvKX+/Z7MMMFLuP3lmtEb+XLCWT8HOSoY 4Ndw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to; bh=HO3NVhjyQqXLCnwAhiYlzS7fK3B98wPRygQz/vHl5/8=; b=PVCgqzgzP2WMutpD9a5Rq3gSB/SSajfZKijMB0J10+DIPgPZmoxG6R5rR/p8HNTGyO fpntPFdgYC5lMC2/o/GrZyz8MR6NHCJPsjmxpHoPX4t+UjoWy+Rxpq8fFKqAGVMnTRMr 6Y9UGnn3XmPruNE88mNoT0Zs5QDiXQooPZOKE4IbdY+krfwPu1und0UrV0jawtUiL7zQ LaHdMxQ1nqpNEhrbH67wqOGmf2YANQ4rGVJ8fSt5xfoDHead/QaQk7rX7HUa2/7ePQvL aAMtJ2LYq+hLEcjagNLoOPPdxsH+FQjcfg4ulG5Sx2d9of0iElZ+HMS4UP+hujd3fFLm K+Iw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id j3-v6si4002957qvj.26.2018.09.11.07.06.18; Tue, 11 Sep 2018 07:06:18 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 4A4B261C27; Tue, 11 Sep 2018 14:06:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id CFF7661C2D; Tue, 11 Sep 2018 14:02:27 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id E00B961C1D; Tue, 11 Sep 2018 14:01:54 +0000 (UTC) Received: from forward103o.mail.yandex.net (forward103o.mail.yandex.net [37.140.190.177]) by lists.linaro.org (Postfix) with ESMTPS id D089961C27 for ; Tue, 11 Sep 2018 14:00:34 +0000 (UTC) Received: from mxback16j.mail.yandex.net (mxback16j.mail.yandex.net [IPv6:2a02:6b8:0:1619::92]) by forward103o.mail.yandex.net (Yandex) with ESMTP id 4BA385888814 for ; Tue, 11 Sep 2018 17:00:33 +0300 (MSK) Received: from smtp3p.mail.yandex.net (smtp3p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:8]) by mxback16j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id nobP0hMtzz-0WjCcsEM; Tue, 11 Sep 2018 17:00:32 +0300 Received: by smtp3p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id mBghUNGHde-0TSqgfno; Tue, 11 Sep 2018 17:00:30 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 11 Sep 2018 14:00:16 +0000 Message-Id: <1536674416-8465-7-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> References: <1536674416-8465-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 699 Subject: [lng-odp] [PATCH v1 6/6] linux-gen: ring: change ring_deq return value X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Return number of data values dequeued (0 or 1) instead of the data value. This improves error tolerance as there's no data value reserved to indicate empty ring. Also CPU may speculate further before the actual data value is actually needed. Signed-off-by: Petri Savolainen --- /** Email created from pull request 699 (psavol:master-sched-optim-clean-ups) ** https://github.com/Linaro/odp/pull/699 ** Patch: https://github.com/Linaro/odp/pull/699.patch ** Base sha: 33fbc04b6373960ec3f84de4e7e7b34c49d71508 ** Merge commit sha: 32d7a11f22e6f2e1e378b653993c5377d4116d8f **/ platform/linux-generic/include/odp_ring_internal.h | 10 ++++------ platform/linux-generic/odp_schedule_basic.c | 8 +++----- platform/linux-generic/odp_schedule_iquery.c | 5 ++--- platform/linux-generic/odp_schedule_sp.c | 3 +-- 4 files changed, 10 insertions(+), 16 deletions(-) diff --git a/platform/linux-generic/include/odp_ring_internal.h b/platform/linux-generic/include/odp_ring_internal.h index 97673bef4..9a637afb3 100644 --- a/platform/linux-generic/include/odp_ring_internal.h +++ b/platform/linux-generic/include/odp_ring_internal.h @@ -18,9 +18,6 @@ extern "C" { #include #include -/* Ring empty, not a valid data value. */ -#define RING_EMPTY ((uint32_t)-1) - /* Ring of uint32_t data * * Ring stores head and tail counters. Ring indexes are formed from these @@ -59,7 +56,7 @@ static inline void ring_init(ring_t *ring) } /* Dequeue data from the ring head */ -static inline uint32_t ring_deq(ring_t *ring, uint32_t mask) +static inline uint32_t ring_deq(ring_t *ring, uint32_t mask, uint32_t *data) { uint32_t head, tail, new_head; @@ -73,7 +70,7 @@ static inline uint32_t ring_deq(ring_t *ring, uint32_t mask) tail = odp_atomic_load_acq_u32(&ring->w_tail); if (head == tail) - return RING_EMPTY; + return 0; new_head = head + 1; @@ -83,7 +80,8 @@ static inline uint32_t ring_deq(ring_t *ring, uint32_t mask) /* Read data. CAS acquire-release ensures that data read * does not move above from here. */ - return ring->data[new_head & mask]; + *data = ring->data[new_head & mask]; + return 1; } /* Dequeue multiple data from the ring head. Num is smaller than ring size. */ diff --git a/platform/linux-generic/odp_schedule_basic.c b/platform/linux-generic/odp_schedule_basic.c index 77fee74dd..a285edc3c 100644 --- a/platform/linux-generic/odp_schedule_basic.c +++ b/platform/linux-generic/odp_schedule_basic.c @@ -416,8 +416,7 @@ static int schedule_term_global(void) ring_t *ring = &sched->prio_q[grp][i][j].ring; uint32_t qi; - while ((qi = ring_deq(ring, ring_mask)) != - RING_EMPTY) { + while (ring_deq(ring, ring_mask, &qi)) { odp_event_t events[1]; int num; @@ -907,10 +906,9 @@ static inline int do_schedule_grp(odp_queue_t *out_queue, odp_event_t out_ev[], /* Get queue index from the priority queue */ ring = &sched->prio_q[grp][prio][id].ring; - qi = ring_deq(ring, ring_mask); - /* Priority queue empty */ - if (qi == RING_EMPTY) { + if (ring_deq(ring, ring_mask, &qi) == 0) { + /* Priority queue empty */ i++; id++; continue; diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index f76942ff3..e1ef10c46 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -271,7 +271,7 @@ static int schedule_init_global(void) ring_init(&queue->ring); for (k = 0; k < PKTIO_RING_SIZE; k++) - queue->cmd_index[k] = RING_EMPTY; + queue->cmd_index[k] = -1; } for (i = 0; i < NUM_PKTIO_CMD; i++) @@ -668,9 +668,8 @@ static inline void pktio_poll_input(void) for (i = 0; i < PKTIO_CMD_QUEUES; i++, hash = (hash + 1) % PKTIO_CMD_QUEUES) { ring = &sched->pktio_poll.queues[hash].ring; - index = ring_deq(ring, PKTIO_RING_MASK); - if (odp_unlikely(index == RING_EMPTY)) + if (odp_unlikely(ring_deq(ring, PKTIO_RING_MASK, &index) == 0)) continue; cmd = &sched->pktio_poll.commands[index]; diff --git a/platform/linux-generic/odp_schedule_sp.c b/platform/linux-generic/odp_schedule_sp.c index 8ddd1e94e..6b9431b69 100644 --- a/platform/linux-generic/odp_schedule_sp.c +++ b/platform/linux-generic/odp_schedule_sp.c @@ -401,9 +401,8 @@ static inline sched_cmd_t *rem_head(int group, int prio) int pktio; prio_queue = &sched_global->prio_queue[group][prio]; - ring_idx = ring_deq(&prio_queue->ring, RING_MASK); - if (ring_idx == RING_EMPTY) + if (ring_deq(&prio_queue->ring, RING_MASK, &ring_idx) == 0) return NULL; pktio = index_from_ring_idx(&index, ring_idx);