From patchwork Tue Sep 12 13:00:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112314 Delivered-To: patch@linaro.org Received: by 10.140.106.117 with SMTP id d108csp5170659qgf; Tue, 12 Sep 2017 06:11:34 -0700 (PDT) X-Google-Smtp-Source: AOwi7QAaDq2DgogpCHmssA3QF1wkWDeaYgpSvqFeB1doJOVYcIGZLwNGvKLmiywrZHyme6ayo5VG X-Received: by 10.55.184.1 with SMTP id i1mr6159461qkf.236.1505221894399; Tue, 12 Sep 2017 06:11:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1505221894; cv=none; d=google.com; s=arc-20160816; b=m/T6ds0/UkjcLVFOw79AQ+JSYV1nRxma17XAbkVRYIIoMVc1pp3nhxFET/UJRyPgo2 taYDiY5xo52Vzd9A9fHs8ZhlUx2VLYHMQg6YcrhGi9XrF1PbCX1lsxw6XZCAOM/cZ3KU Fkn8GjIcC3F2N6ihb/UTREoskU0vZ9GT1rhnlDYDAUWeJMoUcPggb2+D38AjQAaBr2hY hwDq0r7wDEU6PZkcl1idYsg+LE2WaZdgkkoxF75tFh5GzyC4WmNWZAs363b8zjAn5OaF sUhyQTCrJE8Ks1TD0G/Ox3JQ9Dm346tAseyvseU479sylVgkCTw6UEFXjrqgd2oCEmbs GCRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=o+7hAKCt5/5Wsk4hznBGmn1CSxssQOYrNWoK6Oy54TQ=; b=M5w7HyxcrDt0e8XZ1i0aoewbquhQjktHc9B0zLDr8OyuGOTB6diCREwmSVkKO6GKBq skbmp8fufUvV85LXoCWtc5E7aWYTfcVXzO/x3fuSf2TmCLoZjtHffSbAFlFI7u8WcjXN GmhWp0wVldWzt+xfmA3aBV3M3tQIhS0PCODFD6VLvbm6qfLJFZUsG/J+4mGeLApM/NnQ n7FZo9X/0jrHhIjrciXN4FFwS8ppjrz2Hqha8kQHwU/iouP6UTQ7ijospx/mTLuL3hl0 GqRyBvuzOuI82bYwHbyf8C7ovaS9pMp1I7Odn4lsRrt5C8EFA8I9wdeXpQaAOVm1WLs4 AUIw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id s2si12180582qti.449.2017.09.12.06.11.34; Tue, 12 Sep 2017 06:11:34 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 0D9F660DCC; Tue, 12 Sep 2017 13:11:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 1945262CFA; Tue, 12 Sep 2017 13:02:28 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 84EBB62C6C; Tue, 12 Sep 2017 13:01:59 +0000 (UTC) Received: from forward105j.mail.yandex.net (forward105j.mail.yandex.net [5.45.198.248]) by lists.linaro.org (Postfix) with ESMTPS id B674F61BC5 for ; Tue, 12 Sep 2017 13:00:52 +0000 (UTC) Received: from mxback15g.mail.yandex.net (mxback15g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:94]) by forward105j.mail.yandex.net (Yandex) with ESMTP id 1C7CB184CD1 for ; Tue, 12 Sep 2017 16:00:51 +0300 (MSK) Received: from smtp1p.mail.yandex.net (smtp1p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:6]) by mxback15g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id umei6JWMNv-0jdWu6RH; Tue, 12 Sep 2017 16:00:45 +0300 Received: by smtp1p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id eAiz36RVVp-0iHeUpHZ; Tue, 12 Sep 2017 16:00:44 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 12 Sep 2017 16:00:08 +0300 Message-Id: <1505221212-27163-8-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1505221212-27163-1-git-send-email-odpbot@yandex.ru> References: <1505221212-27163-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v2 7/11] linux-gen: packet: implement static references X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Implemented static references with a reference counter. Counter is zero when not used (reference API not used). Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: fb3f36cec108ce9c55241d9f0e66d4832a552b8a ** Merge commit sha: bab9e010b5432ba0f2ff0651313a85a6a1b563c2 **/ .../linux-generic/include/odp_buffer_internal.h | 21 +++-- .../linux-generic/include/odp_packet_internal.h | 5 +- platform/linux-generic/odp_packet.c | 92 +++++++++++++++++++--- platform/linux-generic/odp_pool.c | 2 + 4 files changed, 102 insertions(+), 18 deletions(-) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 0c873d2d..cd067a08 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -62,11 +62,16 @@ struct odp_buffer_hdr_t { /* Last header of the segment list */ void *last_seg; - /* Initial buffer data pointer and length */ + /* Initial buffer data pointer */ uint8_t *base_data; - uint8_t *buf_end; - /* --- 40 bytes --- */ + /* Reference count */ + odp_atomic_u32_t ref_cnt; + + /* Event type. Maybe different than pool type (crypto compl event) */ + int8_t event_type; + + /* --- 37 bytes --- */ /* Segments */ seg_entry_t seg[CONFIG_PACKET_MAX_SEGS]; @@ -93,14 +98,17 @@ struct odp_buffer_hdr_t { /* Pool pointer */ void *pool_ptr; + /* Initial buffer tail pointer */ + uint8_t *buf_end; + /* User area pointer */ void *uarea_addr; /* User area size */ uint32_t uarea_size; - /* Event type. Maybe different than pool type (crypto compl event) */ - int8_t event_type; + /* Max data size */ + uint32_t size; /* ipc mapped process can not walk over pointers, * offset has to be used */ @@ -110,9 +118,6 @@ struct odp_buffer_hdr_t { * inlining */ odp_pool_t pool_hdl; - /* Max data size */ - uint32_t size; - /* Data or next header */ uint8_t data[0]; } ODP_ALIGNED_CACHE; diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index d8e5766c..d8d4584a 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -116,11 +116,13 @@ typedef struct { packet_parser_t p; + uint32_t frame_len; + odp_pktio_t input; - uint32_t frame_len; uint32_t headroom; uint32_t tailroom; + uint32_t shared_len; /* * Members below are not initialized by packet_init() @@ -217,6 +219,7 @@ static inline void packet_init(odp_packet_hdr_t *pkt_hdr, uint32_t len) * segment occupied by the allocated length. */ pkt_hdr->frame_len = len; + pkt_hdr->shared_len = 0; pkt_hdr->headroom = CONFIG_PACKET_HEADROOM; pkt_hdr->tailroom = CONFIG_PACKET_MAX_SEG_LEN - seg_len + CONFIG_PACKET_TAILROOM; diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index 94fdffab..cde77bb5 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -491,6 +491,36 @@ static inline void copy_buf_hdr(odp_packet_hdr_t *pkt_hdr, int first, int num, } } +static inline void packet_free_multi(odp_buffer_hdr_t *hdr[], int num) +{ + int i; + uint32_t ref_cnt; + int num_ref = 0; + + for (i = 0; i < num; i++) { + /* Zero when reference API has not been used */ + ref_cnt = odp_atomic_load_u32(&hdr[i]->ref_cnt); + + if (odp_unlikely(ref_cnt)) { + ref_cnt = odp_atomic_fetch_dec_u32(&hdr[i]->ref_cnt); + + if (ref_cnt > 1) { + num_ref++; + continue; + } + } + + /* Skip references and pack to be freed headers to array head */ + if (odp_unlikely(num_ref)) + hdr[i - num_ref] = hdr[i]; + } + + num -= num_ref; + + if (odp_likely(num)) + buffer_free_multi(hdr, num); +} + static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) { seg_entry_t *seg; @@ -503,7 +533,7 @@ static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) buf_hdr[i] = seg->hdr; } - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); } static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, @@ -560,7 +590,7 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, pkt_hdr = new_hdr; - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); } else { /* Free last 'num' bufs. * First, find the last remaining header. */ @@ -575,7 +605,7 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, buf_hdr[i] = seg->hdr; } - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); /* Head segment remains, no need to copy or update majority * of the metadata. */ @@ -699,7 +729,7 @@ void odp_packet_free(odp_packet_t pkt) int num_seg = pkt_hdr->buf_hdr.segcount; if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1)) - buffer_free_multi((odp_buffer_hdr_t **)&hdl, 1); + packet_free_multi((odp_buffer_hdr_t **)&hdl, 1); else free_all_segments(pkt_hdr, num_seg); } @@ -707,7 +737,7 @@ void odp_packet_free(odp_packet_t pkt) void odp_packet_free_multi(const odp_packet_t pkt[], int num) { if (CONFIG_PACKET_MAX_SEGS == 1) { - buffer_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); + packet_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); } else { odp_buffer_hdr_t *buf_hdr[num * CONFIG_PACKET_MAX_SEGS]; int i; @@ -728,7 +758,7 @@ void odp_packet_free_multi(const odp_packet_t pkt[], int num) bufs += num_seg - 1; } - buffer_free_multi(buf_hdr, bufs); + packet_free_multi(buf_hdr, bufs); } } @@ -1937,9 +1967,37 @@ uint64_t odp_packet_seg_to_u64(odp_packet_seg_t hdl) return _odp_pri(hdl); } +static inline void buffer_ref_inc(odp_buffer_hdr_t *buf_hdr) +{ + /* First count increment after alloc */ + if (odp_atomic_load_u32(&buf_hdr->ref_cnt) == 0) + odp_atomic_store_u32(&buf_hdr->ref_cnt, 2); + else + odp_atomic_inc_u32(&buf_hdr->ref_cnt); +} + +static inline void packet_ref_inc(odp_packet_hdr_t *pkt_hdr) +{ + seg_entry_t *seg; + int i; + int seg_count = pkt_hdr->buf_hdr.segcount; + odp_packet_hdr_t *hdr = pkt_hdr; + uint8_t idx = 0; + + for (i = 0; i < seg_count; i++) { + seg = seg_entry_next(&hdr, &idx); + buffer_ref_inc(seg->hdr); + } +} + odp_packet_t odp_packet_ref_static(odp_packet_t pkt) { - return odp_packet_copy(pkt, odp_packet_pool(pkt)); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + + packet_ref_inc(pkt_hdr); + pkt_hdr->shared_len = packet_len(pkt_hdr); + + return pkt; } odp_packet_t odp_packet_ref(odp_packet_t pkt, uint32_t offset) @@ -2001,14 +2059,30 @@ odp_packet_t odp_packet_ref_pkt(odp_packet_t pkt, uint32_t offset, int odp_packet_has_ref(odp_packet_t pkt) { - (void)pkt; + odp_buffer_hdr_t *buf_hdr; + seg_entry_t *seg; + int i; + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + int seg_count = pkt_hdr->buf_hdr.segcount; + odp_packet_hdr_t *hdr = pkt_hdr; + uint8_t idx = 0; + + for (i = 0; i < seg_count; i++) { + seg = seg_entry_next(&hdr, &idx); + buf_hdr = seg->hdr; + + if (odp_atomic_load_u32(&buf_hdr->ref_cnt) > 1) + return 1; + } return 0; } uint32_t odp_packet_unshared_len(odp_packet_t pkt) { - return odp_packet_len(pkt); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + + return packet_len(pkt_hdr) - pkt_hdr->shared_len; } /* Include non-inlined versions of API functions */ diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 47a39f5b..2f65cb20 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -277,6 +277,8 @@ static void init_buffers(pool_t *pool) buf_hdr->seg[0].data = &data[offset]; buf_hdr->seg[0].len = pool->data_size; + odp_atomic_init_u32(&buf_hdr->ref_cnt, 0); + /* Store base values for fast init */ buf_hdr->base_data = buf_hdr->seg[0].data; buf_hdr->buf_end = &data[offset + pool->data_size +