From patchwork Sun Aug 9 01:55:10 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bill Fischofer X-Patchwork-Id: 52078 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wi0-f197.google.com (mail-wi0-f197.google.com [209.85.212.197]) by patches.linaro.org (Postfix) with ESMTPS id 9E73720539 for ; Sun, 9 Aug 2015 02:03:23 +0000 (UTC) Received: by wilj18 with SMTP id j18sf27742215wil.0 for ; Sat, 08 Aug 2015 19:03:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:delivered-to:from:to:date :message-id:in-reply-to:references:subject:precedence:list-id :list-unsubscribe:list-archive:list-post:list-help:list-subscribe :mime-version:content-type:content-transfer-encoding:errors-to :sender:x-original-sender:x-original-authentication-results :mailing-list; bh=+BIVo+knW6f6Kv9ArIjzCxfR8wHrlpaoDzMm9yAOSB4=; b=VJnwpRqO+cR1c4Sx2VNpVkXfZZvuKMvU+7pF+tYWzfDqNmnbfmsMG/DLOxMGTLRNbw WG0+Oy8KfArRf4Jxg3pDSiUpXNq6XOVaMr8sbX5mjs1pEbRMSVOnHMOZbaLR6zMNtLrB REROAiQWxn5hOuDFAMvah0aBqBzDTHPnGdezRWoo2O1CCdmCgB40+8RSwxOZbpPeibul vT98Q8WbkzCrHgs4mVPO26GvmdVsf9492w9hM9Pp5/pr2CgqdluKoCh3EbzZ3WHGmVDm JjWB+Ng8gmEAGRzHqRJ+Qdk5rZ6ghhb1E5d+YQHgyk9wCEuIF13PHVLglCdxCm0krACv ZNlQ== X-Gm-Message-State: ALoCoQkogK09C6ydKEu7qIIxzuKaLu/SOKbnpIOeMbxCEzao92lfc7JrOkxtrMr/VM4fEV0BLVgG X-Received: by 10.112.139.137 with SMTP id qy9mr4261848lbb.17.1439085802952; Sat, 08 Aug 2015 19:03:22 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.36.201 with SMTP id s9ls198540laj.40.gmail; Sat, 08 Aug 2015 19:03:22 -0700 (PDT) X-Received: by 10.152.45.9 with SMTP id i9mr14662135lam.105.1439085802550; Sat, 08 Aug 2015 19:03:22 -0700 (PDT) Received: from mail-lb0-f173.google.com (mail-lb0-f173.google.com. [209.85.217.173]) by mx.google.com with ESMTPS id kg7si11149677lbc.176.2015.08.08.19.03.22 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 08 Aug 2015 19:03:22 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.173 as permitted sender) client-ip=209.85.217.173; Received: by lbbpo9 with SMTP id po9so78930983lbb.2 for ; Sat, 08 Aug 2015 19:03:22 -0700 (PDT) X-Received: by 10.112.199.133 with SMTP id jk5mr15168637lbc.32.1439085802253; Sat, 08 Aug 2015 19:03:22 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.7.198 with SMTP id l6csp1188610lba; Sat, 8 Aug 2015 19:03:21 -0700 (PDT) X-Received: by 10.55.22.75 with SMTP id g72mr27674271qkh.27.1439085800673; Sat, 08 Aug 2015 19:03:20 -0700 (PDT) Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id h52si26898535qgf.43.2015.08.08.19.03.20; Sat, 08 Aug 2015 19:03:20 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Received: by lists.linaro.org (Postfix, from userid 109) id DE92461EDB; Sun, 9 Aug 2015 02:03:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252.ec2.internal X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from ip-10-142-244-252.ec2.internal (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 0B8AD61EF9; Sun, 9 Aug 2015 02:02:03 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 6EC1C61EF9; Sun, 9 Aug 2015 01:55:32 +0000 (UTC) Received: from mail-oi0-f52.google.com (mail-oi0-f52.google.com [209.85.218.52]) by lists.linaro.org (Postfix) with ESMTPS id D7CFE61E06 for ; Sun, 9 Aug 2015 01:55:28 +0000 (UTC) Received: by oiev193 with SMTP id v193so42370863oie.3 for ; Sat, 08 Aug 2015 18:55:28 -0700 (PDT) X-Received: by 10.202.69.130 with SMTP id s124mr13161459oia.70.1439085328331; Sat, 08 Aug 2015 18:55:28 -0700 (PDT) Received: from localhost.localdomain (cpe-24-28-70-239.austin.res.rr.com. [24.28.70.239]) by smtp.gmail.com with ESMTPSA id d184sm9449191oif.9.2015.08.08.18.55.27 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 08 Aug 2015 18:55:27 -0700 (PDT) From: Bill Fischofer To: lng-odp@lists.linaro.org Date: Sat, 8 Aug 2015 20:55:10 -0500 Message-Id: <1439085315-6957-9-git-send-email-bill.fischofer@linaro.org> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1439085315-6957-1-git-send-email-bill.fischofer@linaro.org> References: <1439085315-6957-1-git-send-email-bill.fischofer@linaro.org> X-Topics: patch Subject: [lng-odp] [API-NEXT PATCHv11 08/13] linux-generic: queue: modify structs for ordered queue support X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: bill.fischofer@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.173 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Add additional fields to internal buffer and queue structs to enable support for ordered queues. Signed-off-by: Bill Fischofer --- .../linux-generic/include/odp_buffer_internal.h | 15 ++- .../linux-generic/include/odp_queue_internal.h | 121 +++++++++++++++++++++ .../linux-generic/include/odp_schedule_internal.h | 2 +- 3 files changed, 136 insertions(+), 2 deletions(-) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index ae799dd..6badeba 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -103,16 +103,23 @@ typedef union odp_buffer_bits_t { /* forward declaration */ struct odp_buffer_hdr_t; +union queue_entry_u; +typedef union queue_entry_u queue_entry_t; /* Common buffer header */ typedef struct odp_buffer_hdr_t { - struct odp_buffer_hdr_t *next; /* next buf in a list */ + struct odp_buffer_hdr_t *next; /* next buf in a list--keep 1st */ + union { /* Multi-use secondary link */ + struct odp_buffer_hdr_t *prev; + struct odp_buffer_hdr_t *link; + }; odp_buffer_bits_t handle; /* handle */ union { uint32_t all; struct { uint32_t zeroized:1; /* Zeroize buf data on free */ uint32_t hdrdata:1; /* Data is in buffer hdr */ + uint32_t sustain:1; /* Sustain order */ }; } flags; int16_t allocator; /* allocating thread id */ @@ -131,6 +138,12 @@ typedef struct odp_buffer_hdr_t { uint32_t segcount; /* segment count */ uint32_t segsize; /* segment size */ void *addr[ODP_BUFFER_MAX_SEG]; /* block addrs */ + uint64_t order; /* sequence for ordered queues */ + queue_entry_t *origin_qe; /* ordered queue origin */ + union { + queue_entry_t *target_qe; /* ordered queue target */ + uint64_t sync; /* for ordered synchronization */ + }; } odp_buffer_hdr_t; /** @internal Compile time assert that the diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h index 61d0c43..11bc837 100644 --- a/platform/linux-generic/include/odp_queue_internal.h +++ b/platform/linux-generic/include/odp_queue_internal.h @@ -23,6 +23,7 @@ extern "C" { #include #include #include +#include #define USE_TICKETLOCK @@ -77,6 +78,12 @@ struct queue_entry_s { odp_pktio_t pktin; odp_pktio_t pktout; char name[ODP_QUEUE_NAME_LEN]; + uint64_t order_in; + uint64_t order_out; + odp_buffer_hdr_t *reorder_head; + odp_buffer_hdr_t *reorder_tail; + odp_atomic_u64_t sync_in; + odp_atomic_u64_t sync_out; }; typedef union queue_entry_u { @@ -93,6 +100,10 @@ odp_buffer_hdr_t *queue_deq(queue_entry_t *queue); int queue_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num); int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num); +int queue_pktout_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr); +int queue_pktout_enq_multi(queue_entry_t *queue, + odp_buffer_hdr_t *buf_hdr[], int num); + int queue_enq_dummy(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr); int queue_enq_multi_dummy(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num); @@ -137,6 +148,116 @@ static inline int queue_prio(queue_entry_t *qe) return qe->s.param.sched.prio; } +static inline void reorder_enq(queue_entry_t *queue, + queue_entry_t *origin_qe, + odp_buffer_hdr_t *buf_hdr) +{ + odp_buffer_hdr_t *reorder_buf = origin_qe->s.reorder_head; + odp_buffer_hdr_t *reorder_prev = + (odp_buffer_hdr_t *)&origin_qe->s.reorder_head; + + while (reorder_buf && buf_hdr->order >= reorder_buf->order) { + reorder_prev = reorder_buf; + reorder_buf = reorder_buf->next; + } + + buf_hdr->next = reorder_buf; + reorder_prev->next = buf_hdr; + + if (!reorder_buf) + origin_qe->s.reorder_tail = buf_hdr; + + buf_hdr->target_qe = queue; +} + +static inline void order_release(queue_entry_t *origin_qe, int count) +{ + origin_qe->s.order_out += count; +} + +static inline void reorder_deq(queue_entry_t *queue, + queue_entry_t *origin_qe, + odp_buffer_hdr_t **reorder_buf_return, + odp_buffer_hdr_t **reorder_prev_return, + odp_buffer_hdr_t **placeholder_buf_return, + uint32_t *release_count_return, + uint32_t *placeholder_count_return) +{ + odp_buffer_hdr_t *reorder_buf = origin_qe->s.reorder_head; + odp_buffer_hdr_t *reorder_prev = NULL; + odp_buffer_hdr_t *placeholder_buf = NULL; + odp_buffer_hdr_t *next_buf; + uint32_t release_count = 0; + uint32_t placeholder_count = 0; + + while (reorder_buf && + reorder_buf->order <= origin_qe->s.order_out + + release_count + placeholder_count) { + /* + * Elements on the reorder list fall into one of + * three categories: + * + * 1. Those destined for the same queue. These + * can be enq'd now if they were waiting to + * be unblocked by this enq. + * + * 2. Those representing placeholders for events + * whose ordering was released by a prior + * odp_schedule_release_ordered() call. These + * can now just be freed. + * + * 3. Those representing events destined for another + * queue. These cannot be consolidated with this + * enq since they have a different target. + * + * Detecting an element with an order sequence gap, an + * element in category 3, or running out of elements + * stops the scan. + */ + next_buf = reorder_buf->next; + + if (odp_likely(reorder_buf->target_qe == queue)) { + /* promote any chain */ + odp_buffer_hdr_t *reorder_link = + reorder_buf->link; + + if (reorder_link) { + reorder_buf->next = reorder_link; + reorder_buf->link = NULL; + while (reorder_link->next) + reorder_link = reorder_link->next; + reorder_link->next = next_buf; + reorder_prev = reorder_link; + } else { + reorder_prev = reorder_buf; + } + + if (!reorder_buf->flags.sustain) + release_count++; + reorder_buf = next_buf; + } else if (!reorder_buf->target_qe) { + if (reorder_prev) + reorder_prev->next = next_buf; + else + origin_qe->s.reorder_head = next_buf; + + reorder_buf->next = placeholder_buf; + placeholder_buf = reorder_buf; + + reorder_buf = next_buf; + placeholder_count++; + } else { + break; + } + } + + *reorder_buf_return = reorder_buf; + *reorder_prev_return = reorder_prev; + *placeholder_buf_return = placeholder_buf; + *release_count_return = release_count; + *placeholder_count_return = placeholder_count; +} + void queue_destroy_finalize(queue_entry_t *qe); #ifdef __cplusplus diff --git a/platform/linux-generic/include/odp_schedule_internal.h b/platform/linux-generic/include/odp_schedule_internal.h index 4c6577d..6ea90fb 100644 --- a/platform/linux-generic/include/odp_schedule_internal.h +++ b/platform/linux-generic/include/odp_schedule_internal.h @@ -15,6 +15,7 @@ extern "C" { #include +#include #include #include #include @@ -28,7 +29,6 @@ static inline int schedule_queue(const queue_entry_t *qe) return odp_queue_enq(qe->s.pri_queue, qe->s.cmd_ev); } - int schedule_pktio_start(odp_pktio_t pktio, int prio);