From patchwork Tue Sep 5 10:00:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 111645 Delivered-To: patch@linaro.org Received: by 10.140.94.166 with SMTP id g35csp2618195qge; Tue, 5 Sep 2017 03:03:40 -0700 (PDT) X-Google-Smtp-Source: ADKCNb7TzqGkQIN+1JyrqXreNjMONAIDcIa49pjT2+LEhAdeb1nrri2g6DNFlsdcWgqINHmAEhCc X-Received: by 10.55.16.74 with SMTP id a71mr4922082qkh.170.1504605820058; Tue, 05 Sep 2017 03:03:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504605820; cv=none; d=google.com; s=arc-20160816; b=oe4khK9XNIpQ1U1etAXFDNsSHiWp0EaEi+/fZ/mURaqg7gUZh8mzcKVGbaW8lqB6NV IkYIApGmuCvoMHIJK8hq6eQlhpHhubat0ihBwMsv+ANxKAg22bXi2nzee4QJ+3hFwRQv lHBjLKW7fuKbeT+PzSbKqr/s4N/IH1S6Sc+hPp1tveCJbA6zeIFmx1fLtUWBox+Qkwb6 YlUE0MgArM630XN5XZX+s/p1nitTu84/nlTGcGl2MLs6yN2Y5lKENBp1pbLX8JRyot+M TXR0UVlBw+26ReMaD5WnWhyl49CuYG0bYl+lQBvTKjfsQZyuaUsrX0HlSuEE4HMeA28Q 9vjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=GlBoH0d2UccS31d2z9GgFtg4p0YkUThmEU/lcspHjpQ=; b=LQ7g73NjniMyU6BedyU3SG/RGgtYu507PRHHEej4oyPIN9IRsTv6UpUDON6UEbP9h9 EHQLhTFYtZYh3gbkHeY2QesUXBpZ5mhouoyygqEy72VutSMVtThLR4SwEMBlefugvXQ/ HWb+jqvaBSAAqLQusATyn3e+qbemYxpQUeQOgLxlJLP0bFwmoI1RzZ96Dmmaou99cvOY Ful+ySKeQ6OAv1EmbvdzAa17rTC67/1cbomjbcde81FCH8ViO9v2jscjkioPAooNZg+b eKZ3ZtRG1hhAChQgygaRe8LfKxMvmobEUrquEO6j2GXX6XZzXOuKOKjhZOcnGurHSegZ CNiw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id k66si41951qkb.272.2017.09.05.03.03.39; Tue, 05 Sep 2017 03:03:40 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id A042060D75; Tue, 5 Sep 2017 10:03:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 0B24C60D6E; Tue, 5 Sep 2017 10:00:39 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 6433760CA9; Tue, 5 Sep 2017 10:00:25 +0000 (UTC) Received: from forward101p.mail.yandex.net (forward101p.mail.yandex.net [77.88.28.101]) by lists.linaro.org (Postfix) with ESMTPS id 2637C60C92 for ; Tue, 5 Sep 2017 10:00:15 +0000 (UTC) Received: from mxback6o.mail.yandex.net (mxback6o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::20]) by forward101p.mail.yandex.net (Yandex) with ESMTP id 9BCBE6A845C3 for ; Tue, 5 Sep 2017 13:00:13 +0300 (MSK) Received: from smtp4p.mail.yandex.net (smtp4p.mail.yandex.net [2a02:6b8:0:1402::15:6]) by mxback6o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id hZqThusCJY-0DN0FuSV; Tue, 05 Sep 2017 13:00:13 +0300 Received: by smtp4p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id jmFoBzoBIV-0C0qiEap; Tue, 05 Sep 2017 13:00:12 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 5 Sep 2017 13:00:06 +0300 Message-Id: <1504605607-27572-2-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504605607-27572-1-git-send-email-odpbot@yandex.ru> References: <1504605607-27572-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 160 Subject: [lng-odp] [PATCH API-NEXT v1 1/2] api: schedule: add schedule order unlock lock api X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Balasubramanian Manoharan Adds odp_schedule_order_unlock_lock() api. This API combines schedule order unlock and lock into a single api for performance optimization in HW. Signed-off-by: Balasubramanian Manoharan --- /** Email created from pull request 160 (bala-manoharan:api_sched_order_lock) ** https://github.com/Linaro/odp/pull/160 ** Patch: https://github.com/Linaro/odp/pull/160.patch ** Base sha: 91c0b58fc87ba0431241818758cea94438cd5498 ** Merge commit sha: ac7f94d8a80fbad8a99ab45d7ea089bc3166e776 **/ include/odp/api/spec/schedule.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/include/odp/api/spec/schedule.h b/include/odp/api/spec/schedule.h index 8244746d..0997315e 100644 --- a/include/odp/api/spec/schedule.h +++ b/include/odp/api/spec/schedule.h @@ -368,6 +368,25 @@ void odp_schedule_order_lock(unsigned lock_index); void odp_schedule_order_unlock(unsigned lock_index); /** + * Release ordered context lock and acquires new lock + * + * This call is valid only when holding an ordered synchronization context. + * Release a previously locked ordered context lock and acquires + * a new ordered context lock. + * This call is valid only when there is a single ordered context lock active. + * Results are undefined if this call is made when multiple ordered context + * locks are acquired in nested fashion. + * + * @param lock_index Index of the ordered lock in the current context to be + * acquired. Previously acquired ordered lock is released. + * Must be in the range 0...odp_queue_lock_count() - 1. + * + * @see odp_schedule_order_lock(), odp_schedule_order_unlock() + * + */ +void odp_schedule_order_unlock_lock(uint32_t lock_index); + +/** * @} */ From patchwork Tue Sep 5 10:00:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 111644 Delivered-To: patch@linaro.org Received: by 10.140.94.166 with SMTP id g35csp2616226qge; Tue, 5 Sep 2017 03:02:03 -0700 (PDT) X-Google-Smtp-Source: ADKCNb6K+TkXWarzyKtBgkBlrSV0ejdWpZnl9zdTho6r9qvBh5kSjCzG8a11Lg2U1JPR0xiNLCnx X-Received: by 10.200.35.193 with SMTP id r1mr4424571qtr.95.1504605723677; Tue, 05 Sep 2017 03:02:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504605723; cv=none; d=google.com; s=arc-20160816; b=rahGgE/CpbNmahiqlBnM5eAba2we73xD/YT7qvTsx+/kyRc97f6RjPB9N2iaX3g5xz UsNeK3IBceXVkNPMPQ+K9I2XpcS+/iBo+42DL7P0MXmYOtBxW4HN6KNYZQ9t//1bihzY ks2KyQ1kMf8PE8yxtGWHOaFaaNGtEpGdJfRTmzH5dlxipjbNQE+OgngR9Kuer9QSlr42 EsODrNgrGRb3G6yWIGFislQgR1bWN9yR4eEH6ltKDixMvSNMyAwMY1bEzKbK5HcH+RB6 EBsIU7t23Ypl/jgfLWf88mscyJbhmELZq+CtCuptXIeOu3Tzt7eCGeJrozeq+MzS0wAr a15A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=aQxAyUnxkgLw6tPgYb7xjeRuOiK9FCJeigVCwwCrg0o=; b=IsYP3O1gXeywgZyIpRV7U1dW7lL5yu9tEFT/Kfl+p4s8ZPuDESWu8FQgP1bIh4up1q 7b+N3wLwAkC8tGJzgCtQDZnNw0Vlnx8qgjl6o/ezEAkDDlkM6BUd8440m4O3IFWeo347 c6a+/rRJNIAofTqdlnJL/Lw7HXNzMd51XhNEE7lO+mO4YvmeaWmEH18b1B8xlj0UnnA9 wGKOjSbntfSyHIdS8iY/7R0cvJFyD2P8g918JY6cQS5yjuZmZJ5wCBpJnzTUCvLpCsDb FGLs+haR4QE6c1FzZeS9JacaxKkpLV9ggPI8XgrK7swGAmDO+4r7n/kCrv+v26wWxZ+i mBPA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id d35si317351qtb.128.2017.09.05.03.02.03; Tue, 05 Sep 2017 03:02:03 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 64DCC60D87; Tue, 5 Sep 2017 10:02:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id A2C5160D3A; Tue, 5 Sep 2017 10:00:32 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 2BA1F60CC1; Tue, 5 Sep 2017 10:00:24 +0000 (UTC) Received: from forward101j.mail.yandex.net (forward101j.mail.yandex.net [5.45.198.241]) by lists.linaro.org (Postfix) with ESMTPS id A630060CA9 for ; Tue, 5 Sep 2017 10:00:15 +0000 (UTC) Received: from mxback9g.mail.yandex.net (mxback9g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:170]) by forward101j.mail.yandex.net (Yandex) with ESMTP id 31CE01244683 for ; Tue, 5 Sep 2017 13:00:14 +0300 (MSK) Received: from smtp4p.mail.yandex.net (smtp4p.mail.yandex.net [2a02:6b8:0:1402::15:6]) by mxback9g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id 2ZbUr579I7-0EcqLoI2; Tue, 05 Sep 2017 13:00:14 +0300 Received: by smtp4p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id jmFoBzoBIV-0D04Lfpe; Tue, 05 Sep 2017 13:00:13 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 5 Sep 2017 13:00:07 +0300 Message-Id: <1504605607-27572-3-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504605607-27572-1-git-send-email-odpbot@yandex.ru> References: <1504605607-27572-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 160 Subject: [lng-odp] [PATCH API-NEXT v1 2/2] linux-generic: api schedule unlock lock X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Balasubramanian Manoharan Signed-off-by: Balasubramanian Manoharan --- /** Email created from pull request 160 (bala-manoharan:api_sched_order_lock) ** https://github.com/Linaro/odp/pull/160 ** Patch: https://github.com/Linaro/odp/pull/160.patch ** Base sha: 91c0b58fc87ba0431241818758cea94438cd5498 ** Merge commit sha: ac7f94d8a80fbad8a99ab45d7ea089bc3166e776 **/ platform/linux-generic/include/odp_schedule_if.h | 1 + .../include/odp_schedule_scalable_ordered.h | 1 + platform/linux-generic/odp_schedule.c | 24 +++++++++++++++++++++- platform/linux-generic/odp_schedule_if.c | 5 +++++ platform/linux-generic/odp_schedule_iquery.c | 24 +++++++++++++++++++++- platform/linux-generic/odp_schedule_scalable.c | 19 +++++++++++++++++ platform/linux-generic/odp_schedule_sp.c | 8 +++++++- 7 files changed, 79 insertions(+), 3 deletions(-) diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h index 657993b1..b0db67ab 100644 --- a/platform/linux-generic/include/odp_schedule_if.h +++ b/platform/linux-generic/include/odp_schedule_if.h @@ -95,6 +95,7 @@ typedef struct { odp_schedule_group_info_t *); void (*schedule_order_lock)(unsigned); void (*schedule_order_unlock)(unsigned); + void (*schedule_order_unlock_lock)(unsigned); } schedule_api_t; diff --git a/platform/linux-generic/include/odp_schedule_scalable_ordered.h b/platform/linux-generic/include/odp_schedule_scalable_ordered.h index 1c365a2b..cdc2ab49 100644 --- a/platform/linux-generic/include/odp_schedule_scalable_ordered.h +++ b/platform/linux-generic/include/odp_schedule_scalable_ordered.h @@ -79,6 +79,7 @@ typedef struct reorder_window { uint32_t tail; uint32_t turn; uint32_t olock[CONFIG_QUEUE_MAX_ORD_LOCKS]; + uint32_t lock_acquired; uint16_t lock_count; /* Reorder contexts in this window */ reorder_context_t *ring[RWIN_SIZE]; diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 5b940762..48a31290 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -214,6 +214,8 @@ typedef struct { /* Array of ordered locks */ odp_atomic_u64_t lock[CONFIG_QUEUE_MAX_ORD_LOCKS]; + odp_atomic_u64_t lock_acquired; + } order_context_t ODP_ALIGNED_CACHE; typedef struct { @@ -1103,6 +1105,7 @@ static void order_unlock(void) static void schedule_order_lock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; + odp_atomic_u64_t *lock_acquired; uint32_t queue_index; queue_index = sched_local.ordered.src_queue; @@ -1112,6 +1115,7 @@ static void schedule_order_lock(unsigned lock_index) !sched_local.ordered.lock_called.u8[lock_index]); ord_lock = &sched->order[queue_index].lock[lock_index]; + lock_acquired = &sched->order[queue_index].lock_acquired; /* Busy loop to synchronize ordered processing */ while (1) { @@ -1121,6 +1125,7 @@ static void schedule_order_lock(unsigned lock_index) if (lock_seq == sched_local.ordered.ctx) { sched_local.ordered.lock_called.u8[lock_index] = 1; + odp_atomic_store_rel_u64(lock_acquired, lock_index); return; } odp_cpu_pause(); @@ -1130,6 +1135,7 @@ static void schedule_order_lock(unsigned lock_index) static void schedule_order_unlock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; + odp_atomic_u64_t *lock_acquired; uint32_t queue_index; queue_index = sched_local.ordered.src_queue; @@ -1138,12 +1144,27 @@ static void schedule_order_unlock(unsigned lock_index) lock_index <= sched->queue[queue_index].order_lock_count); ord_lock = &sched->order[queue_index].lock[lock_index]; + lock_acquired = &sched->order[queue_index].lock_acquired; ODP_ASSERT(sched_local.ordered.ctx == odp_atomic_load_u64(ord_lock)); + odp_atomic_store_rel_u64(lock_acquired, + sched->queue[queue_index].order_lock_count + 1); odp_atomic_store_rel_u64(ord_lock, sched_local.ordered.ctx + 1); } +static void schedule_order_unlock_lock(unsigned lock_index) +{ + odp_atomic_u64_t *lock_acquired; + uint32_t queue_index; + + queue_index = sched_local.ordered.src_queue; + + lock_acquired = &sched->order[queue_index].lock_acquired; + schedule_order_unlock(odp_atomic_load_u64(lock_acquired)); + schedule_order_lock(lock_index); +} + static void schedule_pause(void) { sched_local.pause = 1; @@ -1429,5 +1450,6 @@ const schedule_api_t schedule_default_api = { .schedule_group_thrmask = schedule_group_thrmask, .schedule_group_info = schedule_group_info, .schedule_order_lock = schedule_order_lock, - .schedule_order_unlock = schedule_order_unlock + .schedule_order_unlock = schedule_order_unlock, + .schedule_order_unlock_lock = schedule_order_unlock_lock }; diff --git a/platform/linux-generic/odp_schedule_if.c b/platform/linux-generic/odp_schedule_if.c index e56e3722..858c1949 100644 --- a/platform/linux-generic/odp_schedule_if.c +++ b/platform/linux-generic/odp_schedule_if.c @@ -129,3 +129,8 @@ void odp_schedule_order_unlock(unsigned lock_index) { return sched_api->schedule_order_unlock(lock_index); } + +void odp_schedule_order_unlock_lock(uint32_t lock_index) +{ + sched_api->schedule_order_unlock_lock(lock_index); +} diff --git a/platform/linux-generic/odp_schedule_iquery.c b/platform/linux-generic/odp_schedule_iquery.c index b81e5dab..d8154358 100644 --- a/platform/linux-generic/odp_schedule_iquery.c +++ b/platform/linux-generic/odp_schedule_iquery.c @@ -135,6 +135,8 @@ typedef struct { /* Array of ordered locks */ odp_atomic_u64_t lock[CONFIG_QUEUE_MAX_ORD_LOCKS]; + odp_atomic_u64_t lock_acquired; + } order_context_t ODP_ALIGNED_CACHE; typedef struct { @@ -1255,6 +1257,7 @@ static void order_unlock(void) static void schedule_order_lock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; + odp_atomic_u64_t *lock_acquired; uint32_t queue_index; queue_index = thread_local.ordered.src_queue; @@ -1264,6 +1267,7 @@ static void schedule_order_lock(unsigned lock_index) !thread_local.ordered.lock_called.u8[lock_index]); ord_lock = &sched->order[queue_index].lock[lock_index]; + lock_acquired = &sched->order[queue_index].lock_acquired; /* Busy loop to synchronize ordered processing */ while (1) { @@ -1273,6 +1277,7 @@ static void schedule_order_lock(unsigned lock_index) if (lock_seq == thread_local.ordered.ctx) { thread_local.ordered.lock_called.u8[lock_index] = 1; + odp_atomic_store_rel_u64(lock_acquired, lock_index); return; } odp_cpu_pause(); @@ -1282,6 +1287,7 @@ static void schedule_order_lock(unsigned lock_index) static void schedule_order_unlock(unsigned lock_index) { odp_atomic_u64_t *ord_lock; + odp_atomic_u64_t *lock_acquired; uint32_t queue_index; queue_index = thread_local.ordered.src_queue; @@ -1290,12 +1296,27 @@ static void schedule_order_unlock(unsigned lock_index) lock_index <= sched->queues[queue_index].lock_count); ord_lock = &sched->order[queue_index].lock[lock_index]; + lock_acquired = &sched->order[queue_index].lock_acquired; ODP_ASSERT(thread_local.ordered.ctx == odp_atomic_load_u64(ord_lock)); + odp_atomic_store_rel_u64(lock_acquired, + sched->queues[queue_index].lock_count + 1); odp_atomic_store_rel_u64(ord_lock, thread_local.ordered.ctx + 1); } +static void schedule_order_unlock_lock(unsigned lock_index) +{ + uint32_t queue_index; + odp_atomic_u64_t *lock_acquired; + + queue_index = thread_local.ordered.src_queue; + + lock_acquired = &sched->order[queue_index].lock_acquired; + schedule_order_unlock(odp_atomic_load_u64(lock_acquired)); + schedule_order_lock(lock_index); +} + static unsigned schedule_max_ordered_locks(void) { return CONFIG_QUEUE_MAX_ORD_LOCKS; @@ -1368,7 +1389,8 @@ const schedule_api_t schedule_iquery_api = { .schedule_group_thrmask = schedule_group_thrmask, .schedule_group_info = schedule_group_info, .schedule_order_lock = schedule_order_lock, - .schedule_order_unlock = schedule_order_unlock + .schedule_order_unlock = schedule_order_unlock, + .schedule_order_unlock_lock = schedule_order_unlock_lock }; static void thread_set_interest(sched_thread_local_t *thread, diff --git a/platform/linux-generic/odp_schedule_scalable.c b/platform/linux-generic/odp_schedule_scalable.c index 765326e8..0baedb06 100644 --- a/platform/linux-generic/odp_schedule_scalable.c +++ b/platform/linux-generic/odp_schedule_scalable.c @@ -1007,6 +1007,8 @@ static void schedule_order_lock(unsigned lock_index) monitor32(&rctx->rwin->olock[lock_index], __ATOMIC_ACQUIRE) != rctx->sn) doze(); + atomic_store_release(&rctx->rwin->lock_acquired, lock_index, false); + } } @@ -1025,9 +1027,25 @@ static void schedule_order_unlock(unsigned lock_index) atomic_store_release(&rctx->rwin->olock[lock_index], rctx->sn + 1, /*readonly=*/false); + atomic_store_release(&rctx->rwin->lock_acquired, + rctx->rwin->lock_count + 1, false); rctx->olock_flags |= 1U << lock_index; } +static void schedule_order_unlock_lock(unsigned lock_index) +{ + struct reorder_context *rctx; + + rctx = sched_ts->rctx; + if (odp_unlikely(rctx == NULL || rctx->rwin == NULL)) { + ODP_ERR("Invalid call to odp_schedule_order_unlock_lock\n"); + return; + } + schedule_order_unlock(__atomic_load_n(&rctx->rwin->lock_acquired, + __ATOMIC_ACQUIRE)); + schedule_order_lock(lock_index); +} + static void schedule_release_atomic(void) { sched_scalable_thread_state_t *ts; @@ -1978,4 +1996,5 @@ const schedule_api_t schedule_scalable_api = { .schedule_group_info = schedule_group_info, .schedule_order_lock = schedule_order_lock, .schedule_order_unlock = schedule_order_unlock, + .schedule_order_unlock_lock = schedule_order_unlock_lock, }; diff --git a/platform/linux-generic/odp_schedule_sp.c b/platform/linux-generic/odp_schedule_sp.c index 05241275..d4dfbcaf 100644 --- a/platform/linux-generic/odp_schedule_sp.c +++ b/platform/linux-generic/odp_schedule_sp.c @@ -819,6 +819,11 @@ static void schedule_order_unlock(unsigned lock_index) (void)lock_index; } +static void schedule_order_unlock_lock(unsigned lock_index) +{ + (void)lock_index; +} + static void order_lock(void) { } @@ -868,5 +873,6 @@ const schedule_api_t schedule_sp_api = { .schedule_group_thrmask = schedule_group_thrmask, .schedule_group_info = schedule_group_info, .schedule_order_lock = schedule_order_lock, - .schedule_order_unlock = schedule_order_unlock + .schedule_order_unlock = schedule_order_unlock, + .schedule_order_unlock_lock = schedule_order_unlock_lock };