From patchwork Fri Jul 31 02:41:45 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bill Fischofer X-Patchwork-Id: 51734 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-la0-f72.google.com (mail-la0-f72.google.com [209.85.215.72]) by patches.linaro.org (Postfix) with ESMTPS id F1D0E22A24 for ; Fri, 31 Jul 2015 02:44:28 +0000 (UTC) Received: by labby2 with SMTP id by2sf655957lab.3 for ; Thu, 30 Jul 2015 19:44:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:delivered-to:from:to:date :message-id:in-reply-to:references:subject:precedence:list-id :list-unsubscribe:list-archive:list-post:list-help:list-subscribe :mime-version:content-type:content-transfer-encoding:errors-to :sender:x-original-sender:x-original-authentication-results :mailing-list; bh=Eb1i5RHBpYIg9lD/M1m60Swf324IW+so1QoIPoJKBAo=; b=kufR3qFI8fBxTigssM1mCny+0kf8wePRPyqIwwq1h1Vf0/vtmfJL0ogFitZiauWkSO aT0SWqPnyUNueY45Hz/7A50JLVWNc1z3pJmSEg8OXE8fIwwdYlJ+Ps+ZrXbnB/CHr7QK +tlgNnoEtWDqIvmzEr3wOgFcOqWZk7fDu0eQiJwL4Pyr+jCHOtHnm+igB+0FxntnTPy9 Qr/hmRLkJFJAGrKTVUA/ESovG2hj38yBG19fPc3qgHY1TM7fCkx5+13nn277mXTaUwla VwGT+8/gLqJ7uzb3H1F2aYEmsxzHF0StZmZYk9tzb+u0nGTvmlxF8+5e7ktDSksUM8ts GizQ== X-Gm-Message-State: ALoCoQlV/0ZvXyR9O6uZP9LZ6WsYfd3acVxtoA6L1byeg99v8DbAT3d8f4foHgpEmZxrZMaPV/wq X-Received: by 10.152.115.196 with SMTP id jq4mr156523lab.1.1438310667656; Thu, 30 Jul 2015 19:44:27 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.10.74 with SMTP id g10ls255088lab.92.gmail; Thu, 30 Jul 2015 19:44:27 -0700 (PDT) X-Received: by 10.152.44.230 with SMTP id h6mr405337lam.21.1438310667488; Thu, 30 Jul 2015 19:44:27 -0700 (PDT) Received: from mail-lb0-f181.google.com (mail-lb0-f181.google.com. [209.85.217.181]) by mx.google.com with ESMTPS id ld20si2325733lbb.61.2015.07.30.19.44.26 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Jul 2015 19:44:26 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.181 as permitted sender) client-ip=209.85.217.181; Received: by lbqc9 with SMTP id c9so12923430lbq.1 for ; Thu, 30 Jul 2015 19:44:26 -0700 (PDT) X-Received: by 10.152.22.99 with SMTP id c3mr450978laf.32.1438310666644; Thu, 30 Jul 2015 19:44:26 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.7.198 with SMTP id l6csp100617lba; Thu, 30 Jul 2015 19:44:25 -0700 (PDT) X-Received: by 10.55.19.168 with SMTP id 40mr678844qkt.37.1438310664905; Thu, 30 Jul 2015 19:44:24 -0700 (PDT) Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id t77si3922159qga.36.2015.07.30.19.44.23; Thu, 30 Jul 2015 19:44:24 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Received: by lists.linaro.org (Postfix, from userid 109) id 3B1A5620CC; Fri, 31 Jul 2015 02:44:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252.ec2.internal X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from ip-10-142-244-252.ec2.internal (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 1A407620D7; Fri, 31 Jul 2015 02:42:40 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 1C90E620CD; Fri, 31 Jul 2015 02:42:32 +0000 (UTC) Received: from mail-pa0-f53.google.com (mail-pa0-f53.google.com [209.85.220.53]) by lists.linaro.org (Postfix) with ESMTPS id DF74B620C0 for ; Fri, 31 Jul 2015 02:42:02 +0000 (UTC) Received: by pacan13 with SMTP id an13so33359700pac.1 for ; Thu, 30 Jul 2015 19:42:02 -0700 (PDT) X-Received: by 10.66.186.195 with SMTP id fm3mr1178825pac.91.1438310522237; Thu, 30 Jul 2015 19:42:02 -0700 (PDT) Received: from localhost.localdomain (205.158.164.101.ptr.us.xo.net. [205.158.164.101]) by smtp.gmail.com with ESMTPSA id xs13sm4561776pac.3.2015.07.30.19.42.01 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 30 Jul 2015 19:42:01 -0700 (PDT) From: Bill Fischofer To: lng-odp@lists.linaro.org Date: Thu, 30 Jul 2015 19:41:45 -0700 Message-Id: <1438310507-10750-5-git-send-email-bill.fischofer@linaro.org> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1438310507-10750-1-git-send-email-bill.fischofer@linaro.org> References: <1438310507-10750-1-git-send-email-bill.fischofer@linaro.org> X-Topics: patch Subject: [lng-odp] [API-NEXT PATCHv3 4/6] linux-generic: queue: implement ordered queues X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: bill.fischofer@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.181 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Signed-off-by: Bill Fischofer --- .../linux-generic/include/odp_buffer_internal.h | 5 + .../linux-generic/include/odp_queue_internal.h | 4 + .../linux-generic/include/odp_schedule_internal.h | 2 +- platform/linux-generic/odp_pool.c | 3 + platform/linux-generic/odp_queue.c | 144 ++++++++++++++++++++- platform/linux-generic/odp_schedule.c | 2 - 6 files changed, 151 insertions(+), 9 deletions(-) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index ae799dd..c459fce 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -103,6 +103,8 @@ typedef union odp_buffer_bits_t { /* forward declaration */ struct odp_buffer_hdr_t; +union queue_entry_u; +typedef union queue_entry_u queue_entry_t; /* Common buffer header */ typedef struct odp_buffer_hdr_t { @@ -131,6 +133,9 @@ typedef struct odp_buffer_hdr_t { uint32_t segcount; /* segment count */ uint32_t segsize; /* segment size */ void *addr[ODP_BUFFER_MAX_SEG]; /* block addrs */ + uint64_t order; /* sequence for ordered queues */ + queue_entry_t *origin_qe; /* ordered queue origin */ + queue_entry_t *target_qe; /* ordered queue target */ } odp_buffer_hdr_t; /** @internal Compile time assert that the diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h index 61d0c43..9cca552 100644 --- a/platform/linux-generic/include/odp_queue_internal.h +++ b/platform/linux-generic/include/odp_queue_internal.h @@ -77,6 +77,10 @@ struct queue_entry_s { odp_pktio_t pktin; odp_pktio_t pktout; char name[ODP_QUEUE_NAME_LEN]; + uint64_t order_in; + uint64_t order_out; + odp_buffer_hdr_t *reorder_head; + odp_buffer_hdr_t *reorder_tail; }; typedef union queue_entry_u { diff --git a/platform/linux-generic/include/odp_schedule_internal.h b/platform/linux-generic/include/odp_schedule_internal.h index 4c6577d..6ea90fb 100644 --- a/platform/linux-generic/include/odp_schedule_internal.h +++ b/platform/linux-generic/include/odp_schedule_internal.h @@ -15,6 +15,7 @@ extern "C" { #include +#include #include #include #include @@ -28,7 +29,6 @@ static inline int schedule_queue(const queue_entry_t *qe) return odp_queue_enq(qe->s.pri_queue, qe->s.cmd_ev); } - int schedule_pktio_start(odp_pktio_t pktio, int prio); diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 14221fd..30d4b2b 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -514,6 +514,9 @@ odp_buffer_t buffer_alloc(odp_pool_t pool_hdl, size_t size) /* By default, buffers inherit their pool's zeroization setting */ buf->buf.flags.zeroized = pool->s.flags.zeroized; + /* By default, buffers are not associated with an ordered queue */ + buf->buf.origin_qe = NULL; + if (buf->buf.type == ODP_EVENT_PACKET) packet_init(pool, &buf->pkt, size); diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 4a0df11..4d0e1b4 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -27,11 +27,13 @@ #define LOCK(a) odp_ticketlock_lock(a) #define UNLOCK(a) odp_ticketlock_unlock(a) #define LOCK_INIT(a) odp_ticketlock_init(a) +#define LOCK_TRY(a) odp_ticketlock_trylock(a) #else #include #define LOCK(a) odp_spinlock_lock(a) #define UNLOCK(a) odp_spinlock_unlock(a) #define LOCK_INIT(a) odp_spinlock_init(a) +#define LOCK_TRY(a) odp_spinlock_trylock(a) #endif #include @@ -89,6 +91,9 @@ static void queue_init(queue_entry_t *queue, const char *name, queue->s.head = NULL; queue->s.tail = NULL; + queue->s.reorder_head = NULL; + queue->s.reorder_tail = NULL; + queue->s.pri_queue = ODP_QUEUE_INVALID; queue->s.cmd_ev = ODP_EVENT_INVALID; } @@ -329,14 +334,76 @@ odp_queue_t odp_queue_lookup(const char *name) int queue_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr) { int sched = 0; + queue_entry_t *origin_qe = buf_hdr->origin_qe; + + /* Need two locks for enq operations from ordered queues */ + if (origin_qe) { + LOCK(&origin_qe->s.lock); + while (!LOCK_TRY(&queue->s.lock)) { + UNLOCK(&origin_qe->s.lock); + LOCK(&origin_qe->s.lock); + } + if (odp_unlikely(origin_qe->s.status < QUEUE_STATUS_READY)) { + UNLOCK(&queue->s.lock); + UNLOCK(&origin_qe->s.lock); + ODP_ERR("Bad origin queue status\n"); + return -1; + } + } else { + LOCK(&queue->s.lock); + } - LOCK(&queue->s.lock); if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { UNLOCK(&queue->s.lock); + if (origin_qe) + UNLOCK(&origin_qe->s.lock); ODP_ERR("Bad queue status\n"); return -1; } + /* We can only complete the enq if we're in order */ + if (origin_qe) { + if (buf_hdr->order > origin_qe->s.order_out) { + odp_buffer_hdr_t *reorder_buf = + origin_qe->s.reorder_head; + + if (!reorder_buf) { + buf_hdr->next = NULL; + origin_qe->s.reorder_head = buf_hdr; + origin_qe->s.reorder_tail = buf_hdr; + } else { + odp_buffer_hdr_t *reorder_prev = NULL; + + while (buf_hdr->order > reorder_buf->order) { + reorder_prev = reorder_buf; + reorder_buf = reorder_buf->next; + if (!reorder_buf) + break; + } + + buf_hdr->next = reorder_buf; + if (reorder_prev) + reorder_prev->next = buf_hdr; + else + origin_qe->s.reorder_head = buf_hdr; + + if (!reorder_buf) + origin_qe->s.reorder_tail = buf_hdr; + } + + buf_hdr->target_qe = queue; + + /* This enq can't complete until order is restored, so + * we're done here. + */ + UNLOCK(&queue->s.lock); + UNLOCK(&origin_qe->s.lock); + return 0; + } + + origin_qe->s.order_out++; + } + if (queue->s.head == NULL) { /* Empty queue */ queue->s.head = buf_hdr; @@ -352,7 +419,48 @@ int queue_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr) queue->s.status = QUEUE_STATUS_SCHED; sched = 1; /* retval: schedule queue */ } - UNLOCK(&queue->s.lock); + + /* + * If we came from an ordered queue, check to see if our successful + * enq has unblocked other buffers in the origin's reorder queue. + */ + if (origin_qe) { + odp_buffer_hdr_t *reorder_buf = origin_qe->s.reorder_head; + odp_buffer_hdr_t *reorder_prev; + uint32_t release_count = 0; + + while (reorder_buf && + reorder_buf->target_qe == queue && + reorder_buf->order <= + origin_qe->s.order_out + release_count) { + release_count++; + reorder_prev = reorder_buf; + reorder_buf = reorder_buf->next; + } + + /* Add released buffers to the queue as well */ + if (release_count > 0) { + queue->s.tail->next = origin_qe->s.reorder_head; + queue->s.tail = reorder_prev; + origin_qe->s.reorder_head = reorder_prev->next; + reorder_prev->next = NULL; + origin_qe->s.order_out += release_count; + } + + /* Now handle unblocked buffers destined for other queues */ + if (reorder_buf && + reorder_buf->order <= origin_qe->s.order_out) { + UNLOCK(&origin_qe->s.lock); + UNLOCK(&queue->s.lock); + if (schedule_enq(reorder_buf->target_qe, origin_qe)) + ODP_ABORT("schedule_enq failed\n"); + } else { + UNLOCK(&origin_qe->s.lock); + UNLOCK(&queue->s.lock); + } + } else { + UNLOCK(&queue->s.lock); + } /* Add queue to scheduling */ if (sched && schedule_queue(queue)) @@ -364,14 +472,26 @@ int queue_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr) int queue_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num) { int sched = 0; - int i; + int i, j; odp_buffer_hdr_t *tail; - for (i = 0; i < num - 1; i++) - buf_hdr[i]->next = buf_hdr[i+1]; + for (i = 0; i < num; i++) { + /* If any buffer is coming from an ordered queue, enqueue them + * individually since in the general case each might originate + * from a different ordered queue. If any of these fail, the + * return code tells the caller how many succeeded. + */ + if (buf_hdr[i]->origin_qe) { + for (j = 0; j < num; j++) { + if (queue_enq(queue, buf_hdr[j])) + return j; + } + return num; + } + buf_hdr[i]->next = i == num - 1 ? NULL : buf_hdr[i + 1]; + } tail = buf_hdr[num-1]; - buf_hdr[num-1]->next = NULL; LOCK(&queue->s.lock); if (odp_unlikely(queue->s.status < QUEUE_STATUS_READY)) { @@ -449,6 +569,12 @@ odp_buffer_hdr_t *queue_deq(queue_entry_t *queue) buf_hdr = queue->s.head; queue->s.head = buf_hdr->next; buf_hdr->next = NULL; + if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED) { + buf_hdr->origin_qe = queue; + buf_hdr->order = queue->s.order_in++; + } else { + buf_hdr->origin_qe = NULL; + } if (queue->s.head == NULL) { /* Queue is now empty */ @@ -489,6 +615,12 @@ int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num) buf_hdr[i] = hdr; hdr = hdr->next; buf_hdr[i]->next = NULL; + if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED) { + buf_hdr[i]->origin_qe = queue; + buf_hdr[i]->order = queue->s.order_in++; + } else { + buf_hdr[i]->origin_qe = NULL; + } } queue->s.head = hdr; diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 20dd850..df90df3 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -411,8 +411,6 @@ static inline int copy_events(odp_event_t out_ev[], unsigned int max) /* * Schedule queues - * - * TODO: SYNC_ORDERED not implemented yet */ static int schedule(odp_queue_t *out_queue, odp_event_t out_ev[], unsigned int max_num, unsigned int max_deq)