From patchwork Mon Aug 24 04:06:36 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bill Fischofer X-Patchwork-Id: 52619 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-la0-f69.google.com (mail-la0-f69.google.com [209.85.215.69]) by patches.linaro.org (Postfix) with ESMTPS id 3F02220AF6 for ; Mon, 24 Aug 2015 04:11:50 +0000 (UTC) Received: by labia3 with SMTP id ia3sf10995234lab.3 for ; Sun, 23 Aug 2015 21:11:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:delivered-to:from:to:date :message-id:in-reply-to:references:subject:precedence:list-id :list-unsubscribe:list-archive:list-post:list-help:list-subscribe :mime-version:content-type:content-transfer-encoding:errors-to :sender:x-original-sender:x-original-authentication-results :mailing-list; bh=xP1YXja/gN65Ic4DKaw8ZM2sz6rHbF3RLmi6b9fmjdM=; b=VhWnHVIuawnXV4NhPOYdFvinT6djBHQbxWqbeCj3roRtI1GngI0LDiJq5A47wAz/mz 6zDw4ZNGllFLP/544wBANpI+a7HRfBs8Epaf0foO/DXKW31Vf2x1MqAfA2XQegR0CjuN zqwfyHBHA2+OVxDFdIgyEYpKz12F8d6P6SA9RHDBWUrd4Liw4gr7ooURFmviJ+5See5N FRS2AH3DsQpOgVMCokoR/JUHfrpAzYsDz+6P5UpiAT/gCa363Boa6EXxOIeUsY/2DpeI GtgaTirDFbyapdPp1LFhVKTeWDCyJJSnKvM04wmg8eZkeWKAQbR+Mn9aRKZPkV9T+PTS /qZQ== X-Gm-Message-State: ALoCoQnkN42vjTRs+hPgztYA6Nl+m2olWO0tYIYjs35jq5xZyWw5n9opMeIs7C1jVU26Q2ZowdqR X-Received: by 10.152.4.69 with SMTP id i5mr6164250lai.10.1440389509231; Sun, 23 Aug 2015 21:11:49 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.23.103 with SMTP id l7ls529488laf.35.gmail; Sun, 23 Aug 2015 21:11:49 -0700 (PDT) X-Received: by 10.112.168.100 with SMTP id zv4mr18075478lbb.117.1440389509085; Sun, 23 Aug 2015 21:11:49 -0700 (PDT) Received: from mail-lb0-f169.google.com (mail-lb0-f169.google.com. [209.85.217.169]) by mx.google.com with ESMTPS id kx10si12121012lac.157.2015.08.23.21.11.49 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 23 Aug 2015 21:11:49 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.169 as permitted sender) client-ip=209.85.217.169; Received: by lbbtg9 with SMTP id tg9so72537495lbb.1 for ; Sun, 23 Aug 2015 21:11:49 -0700 (PDT) X-Received: by 10.112.67.65 with SMTP id l1mr18542473lbt.86.1440389508992; Sun, 23 Aug 2015 21:11:48 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.162.200 with SMTP id yc8csp2208193lbb; Sun, 23 Aug 2015 21:11:47 -0700 (PDT) X-Received: by 10.140.239.135 with SMTP id k129mr52434638qhc.55.1440389507137; Sun, 23 Aug 2015 21:11:47 -0700 (PDT) Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id 65si26332863qhw.91.2015.08.23.21.11.46; Sun, 23 Aug 2015 21:11:47 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Received: by lists.linaro.org (Postfix, from userid 109) id 4E82B61CBB; Mon, 24 Aug 2015 04:11:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252.ec2.internal X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from ip-10-142-244-252.ec2.internal (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 8802061C89; Mon, 24 Aug 2015 04:07:29 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 9969461C65; Mon, 24 Aug 2015 04:07:22 +0000 (UTC) Received: from mail-oi0-f52.google.com (mail-oi0-f52.google.com [209.85.218.52]) by lists.linaro.org (Postfix) with ESMTPS id DA94661B80 for ; Mon, 24 Aug 2015 04:06:48 +0000 (UTC) Received: by oieu205 with SMTP id u205so2577135oie.0 for ; Sun, 23 Aug 2015 21:06:48 -0700 (PDT) X-Received: by 10.202.89.133 with SMTP id n127mr18148866oib.46.1440389208340; Sun, 23 Aug 2015 21:06:48 -0700 (PDT) Received: from Ubuntu15.localdomain (cpe-24-28-70-239.austin.res.rr.com. [24.28.70.239]) by smtp.gmail.com with ESMTPSA id ru8sm9436956oeb.8.2015.08.23.21.06.47 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 23 Aug 2015 21:06:47 -0700 (PDT) From: Bill Fischofer To: lng-odp@lists.linaro.org Date: Sun, 23 Aug 2015 23:06:36 -0500 Message-Id: <1440389196-28814-9-git-send-email-bill.fischofer@linaro.org> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1440389196-28814-1-git-send-email-bill.fischofer@linaro.org> References: <1440389196-28814-1-git-send-email-bill.fischofer@linaro.org> X-Topics: patch Subject: [lng-odp] [API-NEXT PATCH 8/8] linux-generic: schedule: implement ordered locks X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: bill.fischofer@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.169 as permitted sender) smtp.mailfrom=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Signed-off-by: Bill Fischofer --- .../include/odp/plat/schedule_types.h | 2 -- .../linux-generic/include/odp_buffer_internal.h | 3 ++ .../linux-generic/include/odp_queue_internal.h | 2 ++ platform/linux-generic/odp_queue.c | 39 ++++++++++++++++++++++ platform/linux-generic/odp_schedule.c | 7 ++++ 5 files changed, 51 insertions(+), 2 deletions(-) diff --git a/platform/linux-generic/include/odp/plat/schedule_types.h b/platform/linux-generic/include/odp/plat/schedule_types.h index f13bfab..3665fec 100644 --- a/platform/linux-generic/include/odp/plat/schedule_types.h +++ b/platform/linux-generic/include/odp/plat/schedule_types.h @@ -52,8 +52,6 @@ typedef int odp_schedule_group_t; #define ODP_SCHED_GROUP_NAME_LEN 32 -typedef int odp_schedule_olock_t; - /** * @} */ diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index ca4d314..6badeba 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -140,7 +140,10 @@ typedef struct odp_buffer_hdr_t { void *addr[ODP_BUFFER_MAX_SEG]; /* block addrs */ uint64_t order; /* sequence for ordered queues */ queue_entry_t *origin_qe; /* ordered queue origin */ + union { queue_entry_t *target_qe; /* ordered queue target */ + uint64_t sync; /* for ordered synchronization */ + }; } odp_buffer_hdr_t; /** @internal Compile time assert that the diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h index aab36be..4cee9b6 100644 --- a/platform/linux-generic/include/odp_queue_internal.h +++ b/platform/linux-generic/include/odp_queue_internal.h @@ -122,6 +122,7 @@ int queue_sched_atomic(odp_queue_t handle); int release_order(queue_entry_t *origin_qe, uint64_t order, odp_pool_t pool, int enq_called); void get_sched_order(queue_entry_t **origin_qe, uint64_t *order); +void get_sched_sync(queue_entry_t **origin_qe, uint64_t **sync); void sched_enq_called(void); void sched_order_resolved(odp_buffer_hdr_t *buf_hdr); @@ -193,6 +194,7 @@ static inline void reorder_enq(queue_entry_t *queue, static inline void order_release(queue_entry_t *origin_qe, int count) { origin_qe->s.order_out += count; + odp_atomic_fetch_add_u64(&origin_qe->s.sync_out, count); } static inline int reorder_deq(queue_entry_t *queue, diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 6b5eee4..8a673d0 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -626,6 +626,7 @@ odp_buffer_hdr_t *queue_deq(queue_entry_t *queue) if (queue_is_ordered(queue)) { buf_hdr->origin_qe = queue; buf_hdr->order = queue->s.order_in++; + buf_hdr->sync = odp_atomic_fetch_inc_u64(&queue->s.sync_in); buf_hdr->flags.sustain = 0; } else { buf_hdr->origin_qe = NULL; @@ -673,6 +674,8 @@ int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num) if (queue_is_ordered(queue)) { buf_hdr[i]->origin_qe = queue; buf_hdr[i]->order = queue->s.order_in++; + buf_hdr[i]->sync = + odp_atomic_fetch_inc_u64(&queue->s.sync_in); buf_hdr[i]->flags.sustain = 0; } else { buf_hdr[i]->origin_qe = NULL; @@ -985,3 +988,39 @@ int release_order(queue_entry_t *origin_qe, uint64_t order, UNLOCK(&origin_qe->s.lock); return 0; } + +void odp_schedule_order_lock(void) +{ + queue_entry_t *origin_qe; + uint64_t *sync; + + get_sched_sync(&origin_qe, &sync); + if (!origin_qe) + return; + + /* Wait until we are in order. Note that sync_out will be incremented + * both by unlocks as well as order resolution, so we're OK if only + * some events in the ordered flow need to lock. + */ + while (*sync > odp_atomic_load_u64(&origin_qe->s.sync_out)) + odp_spin(); +} + +void odp_schedule_order_unlock(void) +{ + queue_entry_t *origin_qe; + uint64_t *sync; + + get_sched_sync(&origin_qe, &sync); + if (!origin_qe) + return; + + /* Get a new sync order for reusability, and release the lock. Note + * that this must be done in this sequence to prevent race conditions + * where the next waiter could lock and unlock before we're able to + * get a new sync order since that would cause order inversion on + * subsequent locks we may perform in this ordered context. + */ + *sync = odp_atomic_fetch_inc_u64(&origin_qe->s.sync_in); + odp_atomic_fetch_inc_u64(&origin_qe->s.sync_out); +} diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index f816460..4087730 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -84,6 +84,7 @@ typedef struct { queue_entry_t *qe; queue_entry_t *origin_qe; uint64_t order; + uint64_t sync; odp_pool_t pool; int enq_called; int num; @@ -795,6 +796,12 @@ void get_sched_order(queue_entry_t **origin_qe, uint64_t *order) *order = sched_local.order; } +void get_sched_sync(queue_entry_t **origin_qe, uint64_t **sync) +{ + *origin_qe = sched_local.origin_qe; + *sync = &sched_local.sync; +} + void sched_order_resolved(odp_buffer_hdr_t *buf_hdr) { if (buf_hdr)