From patchwork Fri Sep 8 14:00:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112084 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp1665999ybm; Fri, 8 Sep 2017 07:05:33 -0700 (PDT) X-Google-Smtp-Source: AOwi7QCPDWmJ7i4k1xmFF2HZx+afTbp2ImiVAOSu3uKkqEzwY49Y4E0ZHfADaxGshnbmmVsAiAT2 X-Received: by 10.55.130.198 with SMTP id e189mr3691222qkd.155.1504879533550; Fri, 08 Sep 2017 07:05:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879533; cv=none; d=google.com; s=arc-20160816; b=Ps64bEvR9qoXIZ4kWIgoxdo6atmAp0dpepTlef0MG4xzhEfUPqetVunjSarfUMll3p YNh5jcRgseFgP+s6U26vKLZDE0a60A61AUXxZFVnhHYAX+3/XExIkucjvZm5kV6I/Y3h Neg2CTwl8ch8rbfdAj35jC0CW352dmOBdJxtZk2b8K7zRr0wY4TN9dWS5dB2Nh9qdthc hn7mykx3k4hwWMpspPf3ZmY81SiE0pptL48W+wdbEB7Ds/+BN0L1DizbTLKD6+TN+glL o7vSveXRCBaIa+FK7NZP5Us47sdEujRWBM2WTeDEGanFvov+ldIuZ2o2HQfPLHkgm3Yd o/tA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=o/OODECPkqGYPWbl1SMX+eVex6dHzlFZA9ATngQWWTI=; b=dgkhtZZlW/wiQ6c3TXxvUGCEBOxX/2lJCfhYxvtKbUdlnEo3uu8Th9QwD5k3iqaoG7 8I54fLBDGyxfdF1yqeUOJHrxv+foz7NgxxApRGWgxcnhntxqKumk2JDthHKz5Jffao1V iEnhFJbvAONzd5hbNdiRLM9TmUJmFW/OGB5Jzm9AXASziCkXDdOuO1ntG1X+oFv0x0Q0 57NomE8ljR+Yg0DZ3YWeIxbLXIuuYaf/MUs3fUFUBboMNhw1tm7LovIJFjqtoJiy531/ tUDFHkN7dmoEPGTcxUCLY+d6MJ9pa2rECcoDORc8Pi6P8rL8zeBv9L7mlf1/Gld3gLKY YsjA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id q2si2109986qke.372.2017.09.08.07.05.33; Fri, 08 Sep 2017 07:05:33 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 22E1665BEF; Fri, 8 Sep 2017 14:05:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 5951B644CB; Fri, 8 Sep 2017 14:00:58 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id CAAAD608E3; Fri, 8 Sep 2017 14:00:29 +0000 (UTC) Received: from forward100j.mail.yandex.net (forward100j.mail.yandex.net [5.45.198.240]) by lists.linaro.org (Postfix) with ESMTPS id A7AC8609DB for ; Fri, 8 Sep 2017 14:00:20 +0000 (UTC) Received: from mxback9g.mail.yandex.net (mxback9g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:170]) by forward100j.mail.yandex.net (Yandex) with ESMTP id 7F2005D81861 for ; Fri, 8 Sep 2017 17:00:18 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback9g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id Tqy5Zaqrn5-0IcqkQUw; Fri, 08 Sep 2017 17:00:18 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0HKSn8tT; Fri, 08 Sep 2017 17:00:17 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:02 +0300 Message-Id: <1504879212-3689-2-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 1/11] linux-gen: packet: roll back to copy based references X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen In order to re-implement segmentation for the new reference implementation roll back to the copy based reference implementation, which have been proved to work OK. This zero copy reference implementation introduced multiple issues: * segment pointers and length were not set correctly, but were left to NULL * reference counting has a race condition which causes random assert failures also with non-reference packets * there's larger than expected performance drop with non-reference packets * the first segment of a reference was zero bytes long, which is possible by the API but is unusual for users Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ .../include/odp/api/plat/packet_inlines.h | 40 +- .../include/odp/api/plat/packet_inlines_api.h | 16 + .../include/odp/api/plat/packet_types.h | 8 - .../linux-generic/include/odp_packet_internal.h | 98 +--- platform/linux-generic/odp_packet.c | 620 ++++++--------------- 5 files changed, 204 insertions(+), 578 deletions(-) diff --git a/platform/linux-generic/include/odp/api/plat/packet_inlines.h b/platform/linux-generic/include/odp/api/plat/packet_inlines.h index 06b049fcb..d0cf13901 100644 --- a/platform/linux-generic/include/odp/api/plat/packet_inlines.h +++ b/platform/linux-generic/include/odp/api/plat/packet_inlines.h @@ -50,19 +50,7 @@ static inline uint32_t _odp_packet_seg_len(odp_packet_t pkt) /** @internal Inline function @param pkt @return */ static inline uint32_t _odp_packet_len(odp_packet_t pkt) { - uint32_t pkt_len = _odp_pkt_get(pkt, uint32_t, frame_len); - void *ref_nxt = _odp_pkt_get(pkt, void *, ref_hdr); - void *ref_pkt = (void *)pkt; - - while (ref_nxt) { - pkt_len += _odp_pkt_get(ref_pkt, uint32_t, ref_len) - - _odp_pkt_get(ref_pkt, uint32_t, ref_offset); - - ref_pkt = ref_nxt; - ref_nxt = _odp_pkt_get(ref_nxt, void *, ref_hdr); - } - - return pkt_len; + return _odp_pkt_get(pkt, uint32_t, frame_len); } /** @internal Inline function @param pkt @return */ @@ -90,6 +78,12 @@ static inline odp_pktio_t _odp_packet_input(odp_packet_t pkt) } /** @internal Inline function @param pkt @return */ +static inline int _odp_packet_num_segs(odp_packet_t pkt) +{ + return _odp_pkt_get(pkt, uint8_t, segcount); +} + +/** @internal Inline function @param pkt @return */ static inline void *_odp_packet_user_ptr(odp_packet_t pkt) { return _odp_pkt_get(pkt, void *, user_ptr); @@ -128,8 +122,7 @@ static inline void *_odp_packet_head(odp_packet_t pkt) /** @internal Inline function @param pkt @return */ static inline int _odp_packet_is_segmented(odp_packet_t pkt) { - return _odp_pkt_get(pkt, uint8_t, segcount) > 1 || - _odp_pkt_get(pkt, void *, ref_hdr) != NULL; + return _odp_pkt_get(pkt, uint8_t, segcount) > 1; } /** @internal Inline function @param pkt @return */ @@ -140,6 +133,23 @@ static inline odp_packet_seg_t _odp_packet_first_seg(odp_packet_t pkt) return _odp_packet_seg_from_ndx(0); } +/** @internal Inline function @param pkt @return */ +static inline odp_packet_seg_t _odp_packet_last_seg(odp_packet_t pkt) +{ + return _odp_packet_seg_from_ndx(_odp_packet_num_segs(pkt) - 1); +} + +/** @internal Inline function @param pkt @param seg @return */ +static inline odp_packet_seg_t _odp_packet_next_seg(odp_packet_t pkt, + odp_packet_seg_t seg) +{ + if (odp_unlikely(_odp_packet_seg_to_ndx(seg) >= + _odp_packet_seg_to_ndx(_odp_packet_last_seg(pkt)))) + return ODP_PACKET_SEG_INVALID; + + return seg + 1; +} + /** @internal Inline function @param pkt @param offset @param len */ static inline void _odp_packet_prefetch(odp_packet_t pkt, uint32_t offset, uint32_t len) diff --git a/platform/linux-generic/include/odp/api/plat/packet_inlines_api.h b/platform/linux-generic/include/odp/api/plat/packet_inlines_api.h index f818f820e..233bc8761 100644 --- a/platform/linux-generic/include/odp/api/plat/packet_inlines_api.h +++ b/platform/linux-generic/include/odp/api/plat/packet_inlines_api.h @@ -48,6 +48,11 @@ _ODP_INLINE odp_pktio_t odp_packet_input(odp_packet_t pkt) return _odp_packet_input(pkt); } +_ODP_INLINE int odp_packet_num_segs(odp_packet_t pkt) +{ + return _odp_packet_num_segs(pkt); +} + _ODP_INLINE void *odp_packet_user_ptr(odp_packet_t pkt) { return _odp_packet_user_ptr(pkt); @@ -88,6 +93,17 @@ _ODP_INLINE odp_packet_seg_t odp_packet_first_seg(odp_packet_t pkt) return _odp_packet_first_seg(pkt); } +_ODP_INLINE odp_packet_seg_t odp_packet_last_seg(odp_packet_t pkt) +{ + return _odp_packet_last_seg(pkt); +} + +_ODP_INLINE odp_packet_seg_t odp_packet_next_seg(odp_packet_t pkt, + odp_packet_seg_t seg) +{ + return _odp_packet_next_seg(pkt, seg); +} + _ODP_INLINE void odp_packet_prefetch(odp_packet_t pkt, uint32_t offset, uint32_t len) { diff --git a/platform/linux-generic/include/odp/api/plat/packet_types.h b/platform/linux-generic/include/odp/api/plat/packet_types.h index 1554c3af3..41468becf 100644 --- a/platform/linux-generic/include/odp/api/plat/packet_types.h +++ b/platform/linux-generic/include/odp/api/plat/packet_types.h @@ -84,14 +84,6 @@ typedef struct _odp_packet_inline_offset_t { /** @internal field offset */ size_t tailroom; /** @internal field offset */ - size_t unshared_len; - /** @internal field offset */ - size_t ref_hdr; - /** @internal field offset */ - size_t ref_offset; - /** *internal field offset */ - size_t ref_len; - /** @internal field offset */ size_t pool; /** @internal field offset */ size_t input; diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index d923ee233..f77987c8d 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -19,7 +19,6 @@ extern "C" { #include #include -#include #include #include #include @@ -107,7 +106,7 @@ typedef struct { * packet_init(). Because of this any new fields added must be reviewed for * initialization requirements. */ -typedef struct odp_packet_hdr_t { +typedef struct { /* common buffer header */ odp_buffer_hdr_t buf_hdr; @@ -123,19 +122,6 @@ typedef struct odp_packet_hdr_t { uint32_t headroom; uint32_t tailroom; - /* Fields used to support packet references */ - uint32_t unshared_len; - /* Next pkt_hdr in reference chain */ - struct odp_packet_hdr_t *ref_hdr; - /* Offset into next pkt_hdr that ref was created at */ - uint32_t ref_offset; - /* frame_len in next pkt_hdr at time ref was created. This - * allows original offset to be maintained when base pkt len - * is changed */ - uint32_t ref_len; - /* Incremented on refs, decremented on frees. */ - odp_atomic_u32_t ref_count; - /* * Members below are not initialized by packet_init() */ @@ -171,50 +157,6 @@ static inline odp_packet_hdr_t *odp_packet_hdr(odp_packet_t pkt) return (odp_packet_hdr_t *)(uintptr_t)pkt; } -static inline odp_packet_hdr_t *packet_last_hdr(odp_packet_t pkt, - uint32_t *offset) -{ - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); - odp_packet_hdr_t *prev_hdr = pkt_hdr; - uint32_t ref_offset = 0; - - while (pkt_hdr->ref_hdr) { - ref_offset = pkt_hdr->ref_offset; - prev_hdr = pkt_hdr; - pkt_hdr = pkt_hdr->ref_hdr; - } - - if (offset) { - if (prev_hdr != pkt_hdr) - ref_offset += pkt_hdr->frame_len - prev_hdr->ref_len; - *offset = ref_offset; - } - - return pkt_hdr; -} - -static inline odp_packet_hdr_t *packet_prev_hdr(odp_packet_hdr_t *pkt_hdr, - odp_packet_hdr_t *cur_hdr, - uint32_t *offset) -{ - uint32_t ref_offset = 0; - odp_packet_hdr_t *prev_hdr = pkt_hdr; - - while (pkt_hdr->ref_hdr != cur_hdr) { - ref_offset = pkt_hdr->ref_offset; - prev_hdr = pkt_hdr; - pkt_hdr = pkt_hdr->ref_hdr; - } - - if (offset) { - if (prev_hdr != pkt_hdr) - ref_offset += pkt_hdr->frame_len - prev_hdr->ref_len; - *offset = ref_offset; - } - - return pkt_hdr; -} - static inline odp_packet_t packet_handle(odp_packet_hdr_t *pkt_hdr) { return (odp_packet_t)pkt_hdr; @@ -267,10 +209,6 @@ static inline void packet_init(odp_packet_hdr_t *pkt_hdr, uint32_t len) CONFIG_PACKET_TAILROOM; pkt_hdr->input = ODP_PKTIO_INVALID; - - /* By default packet has no references */ - pkt_hdr->unshared_len = len; - pkt_hdr->ref_hdr = NULL; } static inline void copy_packet_parser_metadata(odp_packet_hdr_t *src_hdr, @@ -295,49 +233,17 @@ static inline void pull_tail(odp_packet_hdr_t *pkt_hdr, uint32_t len) pkt_hdr->tailroom += len; pkt_hdr->frame_len -= len; - pkt_hdr->unshared_len -= len; pkt_hdr->buf_hdr.seg[last].len -= len; } static inline uint32_t packet_len(odp_packet_hdr_t *pkt_hdr) { - uint32_t pkt_len = pkt_hdr->frame_len; - odp_packet_hdr_t *ref_hdr = pkt_hdr->ref_hdr; - - while (ref_hdr) { - pkt_len += (pkt_hdr->ref_len - pkt_hdr->ref_offset); - pkt_hdr = ref_hdr; - ref_hdr = ref_hdr->ref_hdr; - } - - return pkt_len; -} - -static inline uint32_t packet_ref_count(odp_packet_hdr_t *pkt_hdr) -{ - /* Breach the atomic type to do a peek at the ref count. This - * is used to bypass atomic operations if ref_count == 1 for - * performance reasons. - */ - return pkt_hdr->ref_count.v; -} - -static inline void packet_ref_count_set(odp_packet_hdr_t *pkt_hdr, uint32_t n) -{ - /* Only used during init when there are no other possible - * references to this pkt, so avoid the "atomic" overhead by - * a controlled breach of the atomic type here. This saves - * over 10% of the pathlength in routines like packet_alloc(). - */ - pkt_hdr->ref_count.v = n; + return pkt_hdr->frame_len; } static inline void packet_set_len(odp_packet_hdr_t *pkt_hdr, uint32_t len) { - ODP_ASSERT(packet_ref_count(pkt_hdr) == 1); - pkt_hdr->frame_len = len; - pkt_hdr->unshared_len = len; } /* Forward declarations */ diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index 1fc715ca7..ac0571243 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -35,10 +35,6 @@ const _odp_packet_inline_offset_t _odp_packet_inline ODP_ALIGNED_CACHE = { .frame_len = offsetof(odp_packet_hdr_t, frame_len), .headroom = offsetof(odp_packet_hdr_t, headroom), .tailroom = offsetof(odp_packet_hdr_t, tailroom), - .unshared_len = offsetof(odp_packet_hdr_t, unshared_len), - .ref_hdr = offsetof(odp_packet_hdr_t, ref_hdr), - .ref_offset = offsetof(odp_packet_hdr_t, ref_offset), - .ref_len = offsetof(odp_packet_hdr_t, ref_len), .pool = offsetof(odp_packet_hdr_t, buf_hdr.pool_hdl), .input = offsetof(odp_packet_hdr_t, input), .segcount = offsetof(odp_packet_hdr_t, buf_hdr.segcount), @@ -63,16 +59,6 @@ static inline odp_buffer_t buffer_handle(odp_packet_hdr_t *pkt_hdr) return (odp_buffer_t)pkt_hdr; } -static inline void packet_ref_inc(odp_packet_hdr_t *pkt_hdr) -{ - odp_atomic_inc_u32(&pkt_hdr->ref_count); -} - -static inline uint32_t packet_ref_dec(odp_packet_hdr_t *pkt_hdr) -{ - return odp_atomic_fetch_dec_u32(&pkt_hdr->ref_count); -} - static inline odp_packet_hdr_t *buf_to_packet_hdr(odp_buffer_t buf) { return (odp_packet_hdr_t *)buf_hdl_to_hdr(buf); @@ -94,13 +80,12 @@ static inline uint32_t packet_seg_len(odp_packet_hdr_t *pkt_hdr, return pkt_hdr->buf_hdr.seg[seg_idx].len; } -static inline uint8_t *packet_seg_data(odp_packet_hdr_t *pkt_hdr, - uint32_t seg_idx) +static inline void *packet_seg_data(odp_packet_hdr_t *pkt_hdr, uint32_t seg_idx) { return pkt_hdr->buf_hdr.seg[seg_idx].data; } -static inline uint32_t packet_last_seg(odp_packet_hdr_t *pkt_hdr) +static inline int packet_last_seg(odp_packet_hdr_t *pkt_hdr) { if (CONFIG_PACKET_MAX_SEGS == 1) return 0; @@ -155,7 +140,6 @@ static inline void push_head(odp_packet_hdr_t *pkt_hdr, uint32_t len) { pkt_hdr->headroom -= len; pkt_hdr->frame_len += len; - pkt_hdr->unshared_len += len; pkt_hdr->buf_hdr.seg[0].data -= len; pkt_hdr->buf_hdr.seg[0].len += len; } @@ -164,7 +148,6 @@ static inline void pull_head(odp_packet_hdr_t *pkt_hdr, uint32_t len) { pkt_hdr->headroom += len; pkt_hdr->frame_len -= len; - pkt_hdr->unshared_len -= len; pkt_hdr->buf_hdr.seg[0].data += len; pkt_hdr->buf_hdr.seg[0].len -= len; } @@ -175,7 +158,6 @@ static inline void push_tail(odp_packet_hdr_t *pkt_hdr, uint32_t len) pkt_hdr->tailroom -= len; pkt_hdr->frame_len += len; - pkt_hdr->unshared_len += len; pkt_hdr->buf_hdr.seg[last].len += len; } @@ -203,10 +185,6 @@ static inline void packet_seg_copy_md(odp_packet_hdr_t *dst, dst->buf_hdr.uarea_addr = src->buf_hdr.uarea_addr; dst->buf_hdr.uarea_size = src->buf_hdr.uarea_size; - /* reference related metadata */ - dst->ref_len = src->ref_len; - dst->unshared_len = src->unshared_len; - /* segmentation data is not copied: * buf_hdr.seg[] * buf_hdr.segcount @@ -221,15 +199,7 @@ static inline void *packet_map(odp_packet_hdr_t *pkt_hdr, int seg = 0; int seg_count = pkt_hdr->buf_hdr.segcount; - /* Special processing for references */ - while (offset >= pkt_hdr->frame_len && pkt_hdr->ref_hdr) { - offset -= (pkt_hdr->frame_len - pkt_hdr->ref_offset); - offset += (pkt_hdr->ref_hdr->frame_len - pkt_hdr->ref_len); - pkt_hdr = pkt_hdr->ref_hdr; - seg_count = pkt_hdr->buf_hdr.segcount; - } - - if (odp_unlikely(offset > pkt_hdr->frame_len)) + if (odp_unlikely(offset >= pkt_hdr->frame_len)) return NULL; if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || seg_count == 1)) { @@ -271,9 +241,6 @@ void packet_parse_reset(odp_packet_hdr_t *pkt_hdr) pkt_hdr->p.l2_offset = 0; pkt_hdr->p.l3_offset = ODP_PACKET_OFFSET_INVALID; pkt_hdr->p.l4_offset = ODP_PACKET_OFFSET_INVALID; - - /* Ensure dummy pkt hdrs used in I/O recv classification are valid */ - pkt_hdr->ref_hdr = NULL; } static inline void init_segments(odp_packet_hdr_t *pkt_hdr[], int num) @@ -286,7 +253,6 @@ static inline void init_segments(odp_packet_hdr_t *pkt_hdr[], int num) hdr->buf_hdr.seg[0].data = hdr->buf_hdr.base_data; hdr->buf_hdr.seg[0].len = BASE_LEN; - packet_ref_count_set(hdr, 1); /* Link segments */ if (CONFIG_PACKET_MAX_SEGS != 1) { @@ -296,7 +262,6 @@ static inline void init_segments(odp_packet_hdr_t *pkt_hdr[], int num) for (i = 1; i < num; i++) { odp_buffer_hdr_t *buf_hdr; - packet_ref_count_set(pkt_hdr[i], 1); buf_hdr = &pkt_hdr[i]->buf_hdr; hdr->buf_hdr.seg[i].hdr = buf_hdr; hdr->buf_hdr.seg[i].data = buf_hdr->base_data; @@ -414,10 +379,9 @@ static inline odp_packet_hdr_t *add_segments(odp_packet_hdr_t *pkt_hdr, new_hdr->buf_hdr.seg[0].len = seg_len; packet_seg_copy_md(new_hdr, pkt_hdr); - new_hdr->frame_len = pkt_hdr->frame_len + len; - new_hdr->unshared_len = pkt_hdr->unshared_len + len; - new_hdr->headroom = pool->headroom + offset; - new_hdr->tailroom = pkt_hdr->tailroom; + new_hdr->frame_len = pkt_hdr->frame_len + len; + new_hdr->headroom = pool->headroom + offset; + new_hdr->tailroom = pkt_hdr->tailroom; pkt_hdr = new_hdr; } else { @@ -430,9 +394,8 @@ static inline odp_packet_hdr_t *add_segments(odp_packet_hdr_t *pkt_hdr, last = packet_last_seg(pkt_hdr); pkt_hdr->buf_hdr.seg[last].len = seg_len; - pkt_hdr->frame_len += len; - pkt_hdr->unshared_len += len; - pkt_hdr->tailroom = pool->tailroom + offset; + pkt_hdr->frame_len += len; + pkt_hdr->tailroom = pool->tailroom + offset; } return pkt_hdr; @@ -440,20 +403,13 @@ static inline odp_packet_hdr_t *add_segments(odp_packet_hdr_t *pkt_hdr, static inline void free_bufs(odp_packet_hdr_t *pkt_hdr, int first, int num) { - int i, nfree; + int i; odp_buffer_hdr_t *buf_hdr[num]; - for (i = 0, nfree = 0; i < num; i++) { - odp_packet_hdr_t *hdr = pkt_hdr->buf_hdr.seg[first + i].hdr; - - if (packet_ref_count(hdr) == 1 || packet_ref_dec(hdr) == 1) { - ODP_ASSERT((packet_ref_count_set(hdr, 0), 1)); - buf_hdr[nfree++] = &hdr->buf_hdr; - } - } + for (i = 0; i < num; i++) + buf_hdr[i] = pkt_hdr->buf_hdr.seg[first + i].hdr; - if (nfree > 0) - buffer_free_multi(buf_hdr, nfree); + buffer_free_multi(buf_hdr, num); } static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, @@ -464,19 +420,11 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, if (head) { odp_packet_hdr_t *new_hdr; - int i, nfree; + int i; odp_buffer_hdr_t *buf_hdr[num]; - for (i = 0, nfree = 0; i < num; i++) { - new_hdr = pkt_hdr->buf_hdr.seg[i].hdr; - - if (packet_ref_count(new_hdr) == 1 || - packet_ref_dec(new_hdr) == 1) { - ODP_ASSERT((packet_ref_count_set(new_hdr, 0), - 1)); - buf_hdr[nfree++] = &new_hdr->buf_hdr; - } - } + for (i = 0; i < num; i++) + buf_hdr[i] = pkt_hdr->buf_hdr.seg[i].hdr; /* First remaining segment is the new packet descriptor */ new_hdr = pkt_hdr->buf_hdr.seg[num].hdr; @@ -485,17 +433,15 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, packet_seg_copy_md(new_hdr, pkt_hdr); /* Tailroom not changed */ - new_hdr->tailroom = pkt_hdr->tailroom; - new_hdr->headroom = seg_headroom(new_hdr, 0); - new_hdr->frame_len = pkt_hdr->frame_len - free_len; - new_hdr->unshared_len = pkt_hdr->unshared_len - free_len; + new_hdr->tailroom = pkt_hdr->tailroom; + new_hdr->headroom = seg_headroom(new_hdr, 0); + new_hdr->frame_len = pkt_hdr->frame_len - free_len; pull_head(new_hdr, pull_len); pkt_hdr = new_hdr; - if (nfree > 0) - buffer_free_multi(buf_hdr, nfree); + buffer_free_multi(buf_hdr, num); } else { /* Free last 'num' bufs */ free_bufs(pkt_hdr, num_remain, num); @@ -504,7 +450,6 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, * of the metadata. */ pkt_hdr->buf_hdr.segcount = num_remain; pkt_hdr->frame_len -= free_len; - pkt_hdr->unshared_len -= free_len; pkt_hdr->tailroom = seg_tailroom(pkt_hdr, num_remain - 1); pull_tail(pkt_hdr, pull_len); @@ -611,80 +556,48 @@ int odp_packet_alloc_multi(odp_pool_t pool_hdl, uint32_t len, return num; } -static inline void packet_free(odp_packet_hdr_t *pkt_hdr) +void odp_packet_free(odp_packet_t pkt) { - odp_packet_hdr_t *ref_hdr; - odp_buffer_hdr_t *buf_hdr; - uint32_t ref_count; - int num_seg; - - do { - buf_hdr = &pkt_hdr->buf_hdr; - ref_count = packet_ref_count(pkt_hdr); - num_seg = pkt_hdr->buf_hdr.segcount; - ref_hdr = pkt_hdr->ref_hdr; - ODP_ASSERT(ref_count >= 1); - - if (odp_likely((CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1) && - ref_count == 1)) { - ODP_ASSERT((packet_ref_count_set(pkt_hdr, 0), 1)); - buffer_free_multi(&buf_hdr, 1); - } else { - free_bufs(pkt_hdr, 0, num_seg); - } + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + odp_buffer_t hdl = buffer_handle(pkt_hdr); - pkt_hdr = ref_hdr; - } while (pkt_hdr); -} + int num_seg = pkt_hdr->buf_hdr.segcount; -void odp_packet_free(odp_packet_t pkt) -{ - packet_free(packet_hdr(pkt)); + if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1)) + buffer_free_multi((odp_buffer_hdr_t **)&hdl, 1); + else + free_bufs(pkt_hdr, 0, num_seg); } void odp_packet_free_multi(const odp_packet_t pkt[], int num) { - odp_packet_hdr_t *pkt_hdr, *ref_hdr, *hdr; - int nbufs = num * CONFIG_PACKET_MAX_SEGS * 2; - odp_buffer_hdr_t *buf_hdr[nbufs]; - int num_seg; - int i, j; - uint32_t ref_count; - int nfree = 0; + if (CONFIG_PACKET_MAX_SEGS == 1) { + buffer_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); + } else { + odp_buffer_hdr_t *buf_hdr[num * CONFIG_PACKET_MAX_SEGS]; + int i; + int j; + int bufs = 0; - for (i = 0; i < num; i++) { - pkt_hdr = packet_hdr(pkt[i]); + for (i = 0; i < num; i++) { + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt[i]); + int num_seg = pkt_hdr->buf_hdr.segcount; + odp_buffer_hdr_t *hdr = &pkt_hdr->buf_hdr; - do { - num_seg = pkt_hdr->buf_hdr.segcount; - ref_hdr = pkt_hdr->ref_hdr; + buf_hdr[bufs] = hdr; + bufs++; - /* Make sure we have enough space for this pkt's segs */ - if (nfree + num_seg > nbufs) { - buffer_free_multi(buf_hdr, nfree); - nfree = 0; - } + if (odp_likely(num_seg == 1)) + continue; - for (j = 0; j < num_seg; j++) { - hdr = pkt_hdr->buf_hdr.seg[j].hdr; - ref_count = packet_ref_count(hdr); - ODP_ASSERT(ref_count >= 1); - - if (ref_count == 1 || - packet_ref_dec(hdr) == 1) { - ODP_ASSERT - ((packet_ref_count_set(hdr, 0), - 1)); - buf_hdr[nfree++] = &hdr->buf_hdr; - } + for (j = 1; j < num_seg; j++) { + buf_hdr[bufs] = hdr->seg[j].hdr; + bufs++; } + } - pkt_hdr = ref_hdr; - } while (pkt_hdr); + buffer_free_multi(buf_hdr, bufs); } - - if (nfree > 0) - buffer_free_multi(buf_hdr, nfree); } int odp_packet_reset(odp_packet_t pkt, uint32_t len) @@ -696,9 +609,6 @@ int odp_packet_reset(odp_packet_t pkt, uint32_t len) if (odp_unlikely(len > (pool->max_seg_len * num))) return -1; - if (pkt_hdr->ref_hdr) - packet_free(pkt_hdr->ref_hdr); - reset_seg(pkt_hdr, 0, num); packet_init(pkt_hdr, len); @@ -732,47 +642,13 @@ odp_event_t odp_packet_to_event(odp_packet_t pkt) uint32_t odp_packet_buf_len(odp_packet_t pkt) { odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - uint32_t buf_len = 0; - - do { - buf_len += pkt_hdr->buf_hdr.size * pkt_hdr->buf_hdr.segcount; - pkt_hdr = pkt_hdr->ref_hdr; - } while (pkt_hdr); - return buf_len; -} - -uint32_t odp_packet_unshared_len(odp_packet_t pkt) -{ - odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - uint32_t pkt_len = 0, offset = 0; - - if (packet_ref_count(pkt_hdr) == 1) - pkt_hdr->unshared_len = pkt_hdr->frame_len; - - do { - if (packet_ref_count(pkt_hdr) > 1) { - if (offset == 0) - pkt_len += pkt_hdr->unshared_len; - break; - } - - pkt_len += pkt_hdr->frame_len - offset; - offset = pkt_hdr->ref_offset; - - if (pkt_hdr->ref_hdr) - offset += (pkt_hdr->ref_hdr->frame_len - - pkt_hdr->ref_len); - - pkt_hdr = pkt_hdr->ref_hdr; - } while (pkt_hdr); - - return pkt_len; + return pkt_hdr->buf_hdr.size * pkt_hdr->buf_hdr.segcount; } void *odp_packet_tail(odp_packet_t pkt) { - odp_packet_hdr_t *pkt_hdr = packet_last_hdr(pkt, NULL); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); return packet_tail(pkt_hdr); } @@ -968,13 +844,6 @@ int odp_packet_extend_head(odp_packet_t *pkt, uint32_t len, segs = pkt_hdr->buf_hdr.segcount; if (odp_unlikely((segs + num) > CONFIG_PACKET_MAX_SEGS)) { - /* Corner case: fail request if packet has - * references since we cannot shuffle segments - * since another thread may be accessing them - * concurrently */ - if (packet_ref_count(pkt_hdr) > 1) - return -1; - /* Cannot directly add new segments */ odp_packet_hdr_t *new_hdr; int new_segs = 0; @@ -1026,7 +895,6 @@ int odp_packet_extend_head(odp_packet_t *pkt, uint32_t len, pkt_hdr->buf_hdr.segcount = segs; pkt_hdr->frame_len = frame_len; - pkt_hdr->unshared_len = frame_len; pkt_hdr->headroom = offset + pool->headroom; pkt_hdr->tailroom = pool->tailroom; @@ -1052,16 +920,11 @@ int odp_packet_extend_head(odp_packet_t *pkt, uint32_t len, push_head(pkt_hdr, len); } - if (data_ptr || seg_len) { - uint32_t seg_ln = 0; - void *data = packet_map(pkt_hdr, 0, &seg_ln, NULL); - - if (data_ptr) - *data_ptr = data; + if (data_ptr) + *data_ptr = packet_data(pkt_hdr); - if (seg_len) - *seg_len = seg_ln; - } + if (seg_len) + *seg_len = packet_first_seg_len(pkt_hdr); return ret; } @@ -1073,8 +936,6 @@ void *odp_packet_pull_head(odp_packet_t pkt, uint32_t len) if (len > pkt_hdr->frame_len) return NULL; - ODP_ASSERT(len <= pkt_hdr->unshared_len); - pull_head(pkt_hdr, len); return packet_data(pkt_hdr); } @@ -1082,35 +943,15 @@ void *odp_packet_pull_head(odp_packet_t pkt, uint32_t len) int odp_packet_trunc_head(odp_packet_t *pkt, uint32_t len, void **data_ptr, uint32_t *seg_len_out) { - odp_packet_hdr_t *pkt_hdr = packet_hdr(*pkt), *nxt_hdr; + odp_packet_hdr_t *pkt_hdr = packet_hdr(*pkt); uint32_t seg_len = packet_first_seg_len(pkt_hdr); - int ret = 0; - if (len > packet_len(pkt_hdr)) + if (len > pkt_hdr->frame_len) return -1; - ODP_ASSERT(len <= odp_packet_unshared_len(*pkt)); - - /* Special processing for references */ - while (len >= pkt_hdr->frame_len && pkt_hdr->ref_hdr) { - ODP_ASSERT(packet_ref_count(pkt_hdr) == 1); - nxt_hdr = pkt_hdr->ref_hdr; - len -= pkt_hdr->frame_len; - len += pkt_hdr->ref_offset + - (nxt_hdr->frame_len - pkt_hdr->ref_len); - pkt_hdr->ref_hdr = NULL; - packet_free(pkt_hdr); - pkt_hdr = nxt_hdr; - seg_len = packet_first_seg_len(pkt_hdr); - *pkt = packet_handle(pkt_hdr); - ret = 1; - } - - if (CONFIG_PACKET_MAX_SEGS == 1 || - len < seg_len || - pkt_hdr->buf_hdr.segcount == 1) { + if (len < seg_len) { pull_head(pkt_hdr, len); - } else { + } else if (CONFIG_PACKET_MAX_SEGS != 1) { int num = 0; uint32_t pull_len = 0; @@ -1125,29 +966,23 @@ int odp_packet_trunc_head(odp_packet_t *pkt, uint32_t len, *pkt = packet_handle(pkt_hdr); } - if (data_ptr || seg_len_out) { - void *data_head = packet_map(pkt_hdr, 0, &seg_len, NULL); + if (data_ptr) + *data_ptr = packet_data(pkt_hdr); - if (data_ptr) - *data_ptr = data_head; + if (seg_len_out) + *seg_len_out = packet_first_seg_len(pkt_hdr); - if (seg_len_out) - *seg_len_out = seg_len; - } - - return ret; + return 0; } void *odp_packet_push_tail(odp_packet_t pkt, uint32_t len) { - odp_packet_hdr_t *pkt_hdr = packet_last_hdr(pkt, NULL); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); void *old_tail; if (len > pkt_hdr->tailroom) return NULL; - ODP_ASSERT(packet_ref_count(pkt_hdr) == 1); - old_tail = packet_tail(pkt_hdr); push_tail(pkt_hdr, len); @@ -1157,14 +992,12 @@ void *odp_packet_push_tail(odp_packet_t pkt, uint32_t len) int odp_packet_extend_tail(odp_packet_t *pkt, uint32_t len, void **data_ptr, uint32_t *seg_len_out) { - odp_packet_hdr_t *pkt_hdr = packet_last_hdr(*pkt, NULL); + odp_packet_hdr_t *pkt_hdr = packet_hdr(*pkt); uint32_t frame_len = pkt_hdr->frame_len; uint32_t tailroom = pkt_hdr->tailroom; uint32_t tail_off = frame_len; int ret = 0; - ODP_ASSERT(packet_ref_count(pkt_hdr) == 1); - if (len > tailroom) { pool_t *pool = pkt_hdr->buf_hdr.pool_ptr; int num; @@ -1255,7 +1088,6 @@ void *odp_packet_pull_tail(odp_packet_t pkt, uint32_t len) if (len > packet_last_seg_len(pkt_hdr)) return NULL; - ODP_ASSERT(packet_ref_count(pkt_hdr) == 1); pull_tail(pkt_hdr, len); return packet_tail(pkt_hdr); @@ -1266,34 +1098,17 @@ int odp_packet_trunc_tail(odp_packet_t *pkt, uint32_t len, { int last; uint32_t seg_len; - uint32_t offset; - odp_packet_hdr_t *first_hdr = packet_hdr(*pkt); - odp_packet_hdr_t *pkt_hdr, *prev_hdr; + odp_packet_hdr_t *pkt_hdr = packet_hdr(*pkt); - if (len > packet_len(first_hdr)) + if (len > pkt_hdr->frame_len) return -1; - pkt_hdr = packet_last_hdr(*pkt, &offset); - - /* Special processing for references */ - while (len >= pkt_hdr->frame_len - offset && first_hdr->ref_hdr) { - len -= (pkt_hdr->frame_len - offset); - prev_hdr = packet_prev_hdr(first_hdr, pkt_hdr, &offset); - ODP_ASSERT(packet_ref_count(prev_hdr) == 1); - prev_hdr->ref_hdr = NULL; - packet_free(pkt_hdr); - pkt_hdr = prev_hdr; - } - - ODP_ASSERT(packet_ref_count(pkt_hdr) == 1); last = packet_last_seg(pkt_hdr); seg_len = packet_seg_len(pkt_hdr, last); - if (CONFIG_PACKET_MAX_SEGS == 1 || - len < seg_len || - pkt_hdr->buf_hdr.segcount == 1) { + if (len < seg_len) { pull_tail(pkt_hdr, len); - } else { + } else if (CONFIG_PACKET_MAX_SEGS != 1) { int num = 0; uint32_t pull_len = 0; @@ -1440,46 +1255,6 @@ void odp_packet_ts_set(odp_packet_t pkt, odp_time_t timestamp) pkt_hdr->p.input_flags.timestamp = 1; } -int odp_packet_num_segs(odp_packet_t pkt) -{ - odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - uint32_t segcount = 0, i; - uint32_t seg_offset = 0, offset; - - do { - segcount += pkt_hdr->buf_hdr.segcount - seg_offset; - offset = pkt_hdr->ref_offset; - pkt_hdr = pkt_hdr->ref_hdr; - if (pkt_hdr) { - for (i = 0, seg_offset = 0; - i < pkt_hdr->buf_hdr.segcount; - i++, seg_offset++) { - if (offset < pkt_hdr->buf_hdr.seg[i].len) - break; - offset -= pkt_hdr->buf_hdr.seg[i].len; - } - } - } while (pkt_hdr); - - return segcount; -} - -odp_packet_seg_t odp_packet_last_seg(odp_packet_t pkt) -{ - return _odp_packet_seg_from_ndx(odp_packet_num_segs(pkt) - 1); -} - -odp_packet_seg_t odp_packet_next_seg(odp_packet_t pkt, odp_packet_seg_t seg) -{ - odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - - if (odp_unlikely(_odp_packet_seg_to_ndx(seg) >= - packet_last_seg(pkt_hdr))) - return ODP_PACKET_SEG_INVALID; - - return seg + 1; -} - /* * * Segment level @@ -1490,53 +1265,23 @@ odp_packet_seg_t odp_packet_next_seg(odp_packet_t pkt, odp_packet_seg_t seg) void *odp_packet_seg_data(odp_packet_t pkt, odp_packet_seg_t seg) { odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - uint32_t seg_offset = 0, offset = 0, i; - uint32_t seg_ndx = _odp_packet_seg_to_ndx(seg); - - while (seg_ndx >= pkt_hdr->buf_hdr.segcount - seg_offset && - pkt_hdr->ref_hdr) { - seg_ndx -= (pkt_hdr->buf_hdr.segcount - seg_offset); - offset = pkt_hdr->ref_offset; - pkt_hdr = pkt_hdr->ref_hdr; - for (i = 0, seg_offset = 0; - i < pkt_hdr->buf_hdr.segcount; - i++, seg_offset++) { - if (offset < pkt_hdr->buf_hdr.seg[i].len) - break; - offset -= pkt_hdr->buf_hdr.seg[i].len; - } - } - if (odp_unlikely(seg_ndx + seg_offset >= pkt_hdr->buf_hdr.segcount)) + if (odp_unlikely(_odp_packet_seg_to_ndx(seg) >= + pkt_hdr->buf_hdr.segcount)) return NULL; - return packet_seg_data(pkt_hdr, seg_ndx + seg_offset) + offset; + return packet_seg_data(pkt_hdr, _odp_packet_seg_to_ndx(seg)); } uint32_t odp_packet_seg_data_len(odp_packet_t pkt, odp_packet_seg_t seg) { odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - uint32_t seg_offset = 0, offset = 0, i; - uint32_t seg_ndx = _odp_packet_seg_to_ndx(seg); - - while (seg_ndx >= pkt_hdr->buf_hdr.segcount - seg_offset && - pkt_hdr->ref_hdr) { - seg_ndx -= (pkt_hdr->buf_hdr.segcount - seg_offset); - offset = pkt_hdr->ref_offset; - pkt_hdr = pkt_hdr->ref_hdr; - for (i = 0, seg_offset = 0; - i < pkt_hdr->buf_hdr.segcount; - i++, seg_offset++) { - if (offset < pkt_hdr->buf_hdr.seg[i].len) - break; - offset -= pkt_hdr->buf_hdr.seg[i].len; - } - } - if (odp_unlikely(seg_ndx + seg_offset >= pkt_hdr->buf_hdr.segcount)) + if (odp_unlikely(_odp_packet_seg_to_ndx(seg) >= + pkt_hdr->buf_hdr.segcount)) return 0; - return packet_seg_len(pkt_hdr, seg_ndx + seg_offset) - offset; + return packet_seg_len(pkt_hdr, _odp_packet_seg_to_ndx(seg)); } /* @@ -1557,8 +1302,6 @@ int odp_packet_add_data(odp_packet_t *pkt_ptr, uint32_t offset, uint32_t len) if (offset > pktlen) return -1; - ODP_ASSERT(odp_packet_unshared_len(*pkt_ptr) >= offset); - newpkt = odp_packet_alloc(pool->pool_hdl, pktlen + len); if (newpkt == ODP_PACKET_INVALID) @@ -1582,15 +1325,13 @@ int odp_packet_rem_data(odp_packet_t *pkt_ptr, uint32_t offset, uint32_t len) { odp_packet_t pkt = *pkt_ptr; odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - uint32_t pktlen = packet_len(pkt_hdr); + uint32_t pktlen = pkt_hdr->frame_len; pool_t *pool = pkt_hdr->buf_hdr.pool_ptr; odp_packet_t newpkt; - if (offset + len > pktlen) + if (offset > pktlen || offset + len > pktlen) return -1; - ODP_ASSERT(odp_packet_unshared_len(*pkt_ptr) >= offset + len); - newpkt = odp_packet_alloc(pool->pool_hdl, pktlen - len); if (newpkt == ODP_PACKET_INVALID) @@ -1624,8 +1365,6 @@ int odp_packet_align(odp_packet_t *pkt, uint32_t offset, uint32_t len, if (align > ODP_CACHE_LINE_SIZE) return -1; - ODP_ASSERT(odp_packet_has_ref(*pkt) == 0); - if (seglen >= len) { misalign = align <= 1 ? 0 : ROUNDUP_ALIGN(uaddr, align) - uaddr; @@ -1665,13 +1404,10 @@ int odp_packet_concat(odp_packet_t *dst, odp_packet_t src) uint32_t dst_len = dst_hdr->frame_len; uint32_t src_len = src_hdr->frame_len; - ODP_ASSERT(packet_ref_count(dst_hdr) == 1); - /* Do a copy if resulting packet would be out of segments or packets - * are from different pools or src is a reference. */ + * are from different pools. */ if (odp_unlikely((dst_segs + src_segs) > CONFIG_PACKET_MAX_SEGS) || - odp_unlikely(dst_pool != src_pool) || - odp_unlikely(packet_ref_count(src_hdr) > 1)) { + odp_unlikely(dst_pool != src_pool)) { if (odp_packet_extend_tail(dst, src_len, NULL, NULL) >= 0) { (void)odp_packet_copy_from_pkt(*dst, dst_len, src, 0, src_len); @@ -1686,9 +1422,8 @@ int odp_packet_concat(odp_packet_t *dst, odp_packet_t src) add_all_segs(dst_hdr, src_hdr); - dst_hdr->frame_len = dst_len + src_len; - dst_hdr->unshared_len = dst_len + src_len; - dst_hdr->tailroom = src_hdr->tailroom; + dst_hdr->frame_len = dst_len + src_len; + dst_hdr->tailroom = src_hdr->tailroom; /* Data was not moved in memory */ return 0; @@ -1701,7 +1436,6 @@ int odp_packet_split(odp_packet_t *pkt, uint32_t len, odp_packet_t *tail) if (len >= pktlen || tail == NULL) return -1; - ODP_ASSERT(odp_packet_unshared_len(*pkt) >= len); *tail = odp_packet_copy_part(*pkt, len, pktlen - len, odp_packet_pool(*pkt)); @@ -1712,109 +1446,6 @@ int odp_packet_split(odp_packet_t *pkt, uint32_t len, odp_packet_t *tail) } /* - * References - */ - -static inline void packet_ref(odp_packet_hdr_t *pkt_hdr) -{ - uint32_t i; - odp_packet_hdr_t *hdr; - - do { - for (i = 0; i < pkt_hdr->buf_hdr.segcount; i++) { - hdr = pkt_hdr->buf_hdr.seg[i].hdr; - packet_ref_inc(hdr); - } - - pkt_hdr = pkt_hdr->ref_hdr; - } while (pkt_hdr); -} - -static inline odp_packet_t packet_splice(odp_packet_hdr_t *pkt_hdr, - uint32_t offset, - odp_packet_hdr_t *ref_hdr) -{ - /* Catch attempted references to stale handles in debug builds */ - ODP_ASSERT(packet_ref_count(pkt_hdr) > 0); - - /* Splicing is from the last section of src pkt */ - while (ref_hdr->ref_hdr) - ref_hdr = ref_hdr->ref_hdr; - - /* Find section where splice begins */ - while (offset >= pkt_hdr->frame_len && pkt_hdr->ref_hdr) { - offset -= (pkt_hdr->frame_len - pkt_hdr->ref_offset); - offset += (pkt_hdr->ref_hdr->frame_len - pkt_hdr->ref_len); - pkt_hdr = pkt_hdr->ref_hdr; - } - - ref_hdr->ref_hdr = pkt_hdr; - ref_hdr->ref_offset = offset; - ref_hdr->ref_len = pkt_hdr->frame_len; - - if (packet_ref_count(pkt_hdr) == 1 || offset < pkt_hdr->unshared_len) - pkt_hdr->unshared_len = offset; - - packet_ref(pkt_hdr); - return packet_handle(ref_hdr); -} - -odp_packet_t odp_packet_ref_static(odp_packet_t pkt) -{ - odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - - packet_ref(pkt_hdr); - pkt_hdr->unshared_len = 0; - return pkt; -} - -odp_packet_t odp_packet_ref(odp_packet_t pkt, uint32_t offset) -{ - odp_packet_t hdr; - odp_packet_hdr_t *pkt_hdr; - - if (pkt == ODP_PACKET_INVALID) - return ODP_PACKET_INVALID; - - pkt_hdr = packet_hdr(pkt); - if (offset >= packet_len(pkt_hdr)) - return ODP_PACKET_INVALID; - - hdr = odp_packet_alloc(odp_packet_pool(pkt), 0); - - if (hdr == ODP_PACKET_INVALID) - return ODP_PACKET_INVALID; - - return packet_splice(pkt_hdr, offset, packet_hdr(hdr)); -} - -odp_packet_t odp_packet_ref_pkt(odp_packet_t pkt, uint32_t offset, - odp_packet_t hdr) -{ - odp_packet_hdr_t *pkt_hdr; - - if (pkt == ODP_PACKET_INVALID || - hdr == ODP_PACKET_INVALID || - pkt == hdr) - return ODP_PACKET_INVALID; - - ODP_ASSERT(odp_packet_has_ref(hdr) == 0); - - pkt_hdr = packet_hdr(pkt); - if (offset >= packet_len(pkt_hdr)) - return ODP_PACKET_INVALID; - - return packet_splice(pkt_hdr, offset, packet_hdr(hdr)); -} - -int odp_packet_has_ref(odp_packet_t pkt) -{ - odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - - return pkt_hdr->ref_hdr != NULL || packet_ref_count(pkt_hdr) > 1; -} - -/* * * Copy * ******************************************************** @@ -1823,7 +1454,8 @@ int odp_packet_has_ref(odp_packet_t pkt) odp_packet_t odp_packet_copy(odp_packet_t pkt, odp_pool_t pool) { - uint32_t pktlen = odp_packet_len(pkt); + odp_packet_hdr_t *srchdr = packet_hdr(pkt); + uint32_t pktlen = srchdr->frame_len; odp_packet_t newpkt = odp_packet_alloc(pool, pktlen); if (newpkt != ODP_PACKET_INVALID) { @@ -1862,7 +1494,7 @@ int odp_packet_copy_to_mem(odp_packet_t pkt, uint32_t offset, uint8_t *dstaddr = (uint8_t *)dst; odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - if (offset + len > packet_len(pkt_hdr)) + if (offset + len > pkt_hdr->frame_len) return -1; while (len > 0) { @@ -1886,11 +1518,9 @@ int odp_packet_copy_from_mem(odp_packet_t pkt, uint32_t offset, const uint8_t *srcaddr = (const uint8_t *)src; odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - if (offset + len > packet_len(pkt_hdr)) + if (offset + len > pkt_hdr->frame_len) return -1; - ODP_ASSERT(odp_packet_unshared_len(pkt) >= offset + len); - while (len > 0) { mapaddr = packet_map(pkt_hdr, offset, &seglen, NULL); cpylen = len > seglen ? seglen : len; @@ -1916,12 +1546,10 @@ int odp_packet_copy_from_pkt(odp_packet_t dst, uint32_t dst_offset, uint32_t src_seglen = 0; /* GCC */ int overlap; - if (dst_offset + len > packet_len(dst_hdr) || - src_offset + len > packet_len(src_hdr)) + if (dst_offset + len > dst_hdr->frame_len || + src_offset + len > src_hdr->frame_len) return -1; - ODP_ASSERT(odp_packet_unshared_len(dst) >= dst_offset + len); - overlap = (dst_hdr == src_hdr && ((dst_offset <= src_offset && dst_offset + len >= src_offset) || @@ -2054,7 +1682,7 @@ void odp_packet_print(odp_packet_t pkt) len += snprintf(&str[len], n - len, " l4_offset %" PRIu32 "\n", hdr->p.l4_offset); len += snprintf(&str[len], n - len, - " frame_len %" PRIu32 "\n", packet_len(hdr)); + " frame_len %" PRIu32 "\n", hdr->frame_len); len += snprintf(&str[len], n - len, " input %" PRIu64 "\n", odp_pktio_to_u64(hdr->input)); @@ -2458,6 +2086,80 @@ uint64_t odp_packet_seg_to_u64(odp_packet_seg_t hdl) return _odp_pri(hdl); } +odp_packet_t odp_packet_ref_static(odp_packet_t pkt) +{ + return odp_packet_copy(pkt, odp_packet_pool(pkt)); +} + +odp_packet_t odp_packet_ref(odp_packet_t pkt, uint32_t offset) +{ + odp_packet_t new; + int ret; + + new = odp_packet_copy(pkt, odp_packet_pool(pkt)); + + if (new == ODP_PACKET_INVALID) { + ODP_ERR("copy failed\n"); + return ODP_PACKET_INVALID; + } + + ret = odp_packet_trunc_head(&new, offset, NULL, NULL); + + if (ret < 0) { + ODP_ERR("trunk_head failed\n"); + odp_packet_free(new); + return ODP_PACKET_INVALID; + } + + return new; +} + +odp_packet_t odp_packet_ref_pkt(odp_packet_t pkt, uint32_t offset, + odp_packet_t hdr) +{ + odp_packet_t new; + int ret; + + new = odp_packet_copy(pkt, odp_packet_pool(pkt)); + + if (new == ODP_PACKET_INVALID) { + ODP_ERR("copy failed\n"); + return ODP_PACKET_INVALID; + } + + if (offset) { + ret = odp_packet_trunc_head(&new, offset, NULL, NULL); + + if (ret < 0) { + ODP_ERR("trunk_head failed\n"); + odp_packet_free(new); + return ODP_PACKET_INVALID; + } + } + + ret = odp_packet_concat(&hdr, new); + + if (ret < 0) { + ODP_ERR("concat failed\n"); + odp_packet_free(new); + return ODP_PACKET_INVALID; + } + + return hdr; +} + +int odp_packet_has_ref(odp_packet_t pkt) +{ + (void)pkt; + + return 0; +} + +uint32_t odp_packet_unshared_len(odp_packet_t pkt) +{ + return odp_packet_len(pkt); +} + /* Include non-inlined versions of API functions */ #if ODP_ABI_COMPAT == 1 #include From patchwork Fri Sep 8 14:00:03 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112081 Delivered-To: patch@linaro.org Received: by 10.140.106.117 with SMTP id d108csp361064qgf; Fri, 8 Sep 2017 07:01:41 -0700 (PDT) X-Google-Smtp-Source: AOwi7QChP1igaPdscORAPbPdxORRL3pleAnod/bHmZ4V/pRl6oP9SvLeL8zMBq9WTX8zpThaBynI X-Received: by 10.200.56.184 with SMTP id f53mr3872039qtc.139.1504879301347; Fri, 08 Sep 2017 07:01:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879301; cv=none; d=google.com; s=arc-20160816; b=YK5KUuQQ6yb+42Ubeb+zqb24ybgQ5LofYKh6FTlYj0DCmzaDqeQ8H1JlUfkTZ6E8MS caPU3j0dddS16POFq9oBoYa9xm3dV4DAYjnqLZM9bllgm+GayPdHGoG50r5tyTs8gvR0 PpTYtduGd4vtvq6x4HCx9h4DUToFFxZr3AUU7khg1OeLDxRl00DBvhfqsJ+Oe7JXpA03 7k2MPGVLgCTnGOpQdt6K+qlzI+5RrK/igIiC/MCtBuDlCKta2Gjl4enIRAHmeiIe6sqc mUT3a9n4ND7Av0OUaKZr3EzBSI0lwT//C8XKmDgWHacmYr780LHp+neZIYpvMoUpjhkK 4hfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=94PfLEyfio6dvoYZU3lNnThGBjH/x18j6bWt7foh/9w=; b=MpIsfuRl6vcdZgetfZtxlj/w+ZWj2WbEF7WbNijiMEyNGigfiGQ2vW8qM9sZFdQTWu zvHOtB6gmzL51xvRUEdvccx9/oTfXJhQoYK1wbE9goFb+VV+lAFGUhlKPBCHQV4iO1Qv /0YSKhoNizhSbHVOGU1Qz5MiuymJoOBduD3DvmvNlt7qzba5QcGHMtNB8c9BRxn/4YFp dbjBanCceF3mhqL+EdDc5HFDJNiECKXgdYFjimeY3QwchmBYG4a3W1+Tq58fmIfmzzv0 UBAwXhd6nP/ycgmw255n+okN4MBepkKRmJF3E+aeMLfZJWZeK98CuolfU/TVfMDnDI5F kUfA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id 12si2183416qtz.236.2017.09.08.07.01.41; Fri, 08 Sep 2017 07:01:41 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 1972060D6B; Fri, 8 Sep 2017 14:01:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 2668762CE2; Fri, 8 Sep 2017 14:00:34 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id C714660A4D; Fri, 8 Sep 2017 14:00:22 +0000 (UTC) Received: from forward102o.mail.yandex.net (forward102o.mail.yandex.net [37.140.190.182]) by lists.linaro.org (Postfix) with ESMTPS id 79E8C60730 for ; Fri, 8 Sep 2017 14:00:20 +0000 (UTC) Received: from mxback6g.mail.yandex.net (mxback6g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:167]) by forward102o.mail.yandex.net (Yandex) with ESMTP id EC9535A05004 for ; Fri, 8 Sep 2017 17:00:18 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback6g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id VDhoSyg9oA-0InaA8Td; Fri, 08 Sep 2017 17:00:18 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0IK8Uv7M; Fri, 08 Sep 2017 17:00:18 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:03 +0300 Message-Id: <1504879212-3689-3-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 2/11] validation: packet: relax small concat test X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Limit number of consecutive small concat to fraction of the number of packets in the pool. An implementation linking packets together (no copy) cannot be expected to support one byte concats up to max packet size. Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ test/common_plat/validation/api/packet/packet.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/common_plat/validation/api/packet/packet.c b/test/common_plat/validation/api/packet/packet.c index 28bc12b0d..898dac878 100644 --- a/test/common_plat/validation/api/packet/packet.c +++ b/test/common_plat/validation/api/packet/packet.c @@ -1376,7 +1376,7 @@ void packet_test_concat_small(void) int ret; uint8_t *data; uint32_t i; - uint32_t len = 32000; + uint32_t len = PACKET_POOL_NUM / 4; uint8_t buf[len]; CU_ASSERT_FATAL(odp_pool_capability(&capa) == 0); From patchwork Fri Sep 8 14:00:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112085 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp1667386ybm; Fri, 8 Sep 2017 07:06:30 -0700 (PDT) X-Google-Smtp-Source: ADKCNb4xDbiTWXhKDdOpIna2Rldjgzs814igwVjfT0BScA7wSgZltSsE5dq5PdC0D8YJ5MH4/RNJ X-Received: by 10.55.66.22 with SMTP id p22mr4061457qka.30.1504879469216; Fri, 08 Sep 2017 07:04:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879469; cv=none; d=google.com; s=arc-20160816; b=kOLvg1sQ5peW54s5CQjwnAwFdjctujwvwOOTPWZTP+mctZ/qZPyrQT7eL3c8lE8K4T mKuCNAUl/a9UbeO57kcEiPkEviynOvEV9lGXOjXWmof8ujxHS0JFgIjpfnY+OR1jkI8U jEZGTKa6EQPyyOCuLz1HsoJSV4S5u3qp2z6KThWGi4u/ZJ3BsxUAiG1bOA9AVebboGiM fijLbWk463U6eCkxB9l687isA/YklKMJ74GhJVVvZfJOV+4qEAQr0xiHdRviPC6EmFBC DN395dcxHqUuRxagBQMuZRsutbHzE/O0Wg/kWVX9eZDykW6uzm0B0VMCSqpzt3qo7sMK YsPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=StAAwBbxmrvTmruWVMA5QAPZQNlk/59p6UgJ22X63nc=; b=OlcSEISmRDgmCb59yQku9HjR2zGv11cRy+UzbOfBSAwi52ewVLA6lC+G/+OxGCn1K/ 2TW7YCRzKRn44FBE1XU+SNFWYmK0RuKdOtGqM6ECBOu8LMwrCk/6R5iQ3qTPmSmfSqRz mAHncJRtHLOht9OcDA5zkHVC5SIvZQjRePfJRgeTDE+w8M/IoAOsnoSL3Af0yfwNvK2w wa1IoAz2u1PpqQaNsLQYTy4J3ElkL+oF/Zdn7Lkl2c7mXvemjPz4J8FIZvWHkt0Rldcc 8AGl6JEkzvdjfFhHNJo++LSaHSAeYuhpHFKxOHn/J5lGC9gXSHrRMeTsaimyTgHr3i1e kA2g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id e9si2177518qka.106.2017.09.08.07.04.24; Fri, 08 Sep 2017 07:04:29 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id D8E2D65C0F; Fri, 8 Sep 2017 14:04:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 09EC9644C9; Fri, 8 Sep 2017 14:00:51 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id C423462CE2; Fri, 8 Sep 2017 14:00:28 +0000 (UTC) Received: from forward106p.mail.yandex.net (forward106p.mail.yandex.net [77.88.28.109]) by lists.linaro.org (Postfix) with ESMTPS id 91E7D608E3 for ; Fri, 8 Sep 2017 14:00:20 +0000 (UTC) Received: from mxback6g.mail.yandex.net (mxback6g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:167]) by forward106p.mail.yandex.net (Yandex) with ESMTP id 55D742D8487A for ; Fri, 8 Sep 2017 17:00:19 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback6g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id x9s7SpWMyy-0JnOvSXW; Fri, 08 Sep 2017 17:00:19 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0JKeWshZ; Fri, 08 Sep 2017 17:00:19 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:04 +0300 Message-Id: <1504879212-3689-4-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 3/11] validation: packet: unshared len with static reference X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen When static references are used all data is shared, and thus unshared length is zero. When there's no references, all data is unique to the packet (unshared len == len). Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ test/common_plat/validation/api/packet/packet.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/test/common_plat/validation/api/packet/packet.c b/test/common_plat/validation/api/packet/packet.c index 898dac878..c40dab985 100644 --- a/test/common_plat/validation/api/packet/packet.c +++ b/test/common_plat/validation/api/packet/packet.c @@ -2226,6 +2226,14 @@ void packet_test_ref(void) if (odp_packet_has_ref(base_pkt) == 1) { /* CU_ASSERT needs braces */ CU_ASSERT(odp_packet_has_ref(ref_pkt[0]) == 1); + + CU_ASSERT(odp_packet_unshared_len(base_pkt) == 0); + CU_ASSERT(odp_packet_unshared_len(ref_pkt[0]) == 0); + } else { + CU_ASSERT(odp_packet_unshared_len(base_pkt) == + odp_packet_len(base_pkt)); + CU_ASSERT(odp_packet_unshared_len(ref_pkt[0]) == + odp_packet_len(ref_pkt[0])); } CU_ASSERT(odp_packet_len(ref_pkt[0]) == odp_packet_len(base_pkt)); From patchwork Fri Sep 8 14:00:05 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112082 Delivered-To: patch@linaro.org Received: by 10.140.106.117 with SMTP id d108csp362467qgf; Fri, 8 Sep 2017 07:02:32 -0700 (PDT) X-Google-Smtp-Source: AOwi7QAaOe60R0xfHwfzuviWJaezjhOaRFlY8J09Mb1jDk/3+whJJ1fkMbYjsIXS4qPE1/DgQG02 X-Received: by 10.55.139.65 with SMTP id n62mr3778236qkd.94.1504879352703; Fri, 08 Sep 2017 07:02:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879352; cv=none; d=google.com; s=arc-20160816; b=EUdCO94oEV+8rvkuLPQwbKRK/sKg6dqIt/wKKCZJTP3S8TC8Dd/8PiBQFu/04PcA42 sgGUTvnXMZ1WmVaH8B5JmGWXlPwDK29w9ZwclADC2AJNg27D/EO4yLZ1IplUhPjzbNeB z5PQDJtRF2DZhcpp9f25csCWp4t40xnjaNrhYHg3XIo4NG5qy5o1ytFVYTJQADlsirff kGELFL8vrk84KoEI8CrziSxx7JqxJbuKOEP8tWBcokNAljo8Nnzc0ZkyYw+R5HpB03hG ZU0xEC9vVKvzN23JCrxcbEeYs8SKKegFLdTf8uNzgs51DzyWscE4hZ3KsrP7hVHPEqXz fMsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=B535GJtR4g1GDU5O/z4exFzkG8M4zSqo9i1mHfDfjoA=; b=svCAkiw98pPwUOr1V/TYwSVeNSjEwqfY5j1eAej2nDGiKbbhmGKH8lqTpk0e0wg8uq Ai12pHuMoAUqXVA2P2tlngZ7c3X8NEYpQbPCbozytEets4WPcvZWExqNhFiFYlqbIVvG 8x3iY/xQuP5Lv+Xz5KzKPOj4sowKQATIZvOqqyzoAl5aDeB3FusQQ0+As6UNeYh/5klX vJbU/X2wcXkSjXKNsVeZMxJ09ttWHnlRbBzGo1sBukmH+Rxgxfzdq7Ff10ZfSFbQDVeZ FulP+wtMf5vj2Cj+qSLapD9wR2itbFJaLvBJfbbocqu55kVpRPmCqsOndWSjoEdEf5bz B6vg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id w7si2140313qte.424.2017.09.08.07.02.32; Fri, 08 Sep 2017 07:02:32 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 2121365BFF; Fri, 8 Sep 2017 14:02:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 72A9F63DB8; Fri, 8 Sep 2017 14:00:36 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 54B8260AF7; Fri, 8 Sep 2017 14:00:24 +0000 (UTC) Received: from forward103j.mail.yandex.net (forward103j.mail.yandex.net [5.45.198.246]) by lists.linaro.org (Postfix) with ESMTPS id 8A3B960C3E for ; Fri, 8 Sep 2017 14:00:21 +0000 (UTC) Received: from mxback10j.mail.yandex.net (mxback10j.mail.yandex.net [IPv6:2a02:6b8:0:1619::113]) by forward103j.mail.yandex.net (Yandex) with ESMTP id CF43434C1F4D for ; Fri, 8 Sep 2017 17:00:19 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback10j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id ppWuTuWqOf-0JJKC2Qj; Fri, 08 Sep 2017 17:00:19 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0JK4Z2WQ; Fri, 08 Sep 2017 17:00:19 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:05 +0300 Message-Id: <1504879212-3689-5-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 4/11] validation: packet: check reference packet content X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Check that reference and base packets contain the same data. Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ test/common_plat/validation/api/packet/packet.c | 99 ++++++++++++++++++++++++- 1 file changed, 96 insertions(+), 3 deletions(-) diff --git a/test/common_plat/validation/api/packet/packet.c b/test/common_plat/validation/api/packet/packet.c index c40dab985..ffcdc3fcd 100644 --- a/test/common_plat/validation/api/packet/packet.c +++ b/test/common_plat/validation/api/packet/packet.c @@ -2072,14 +2072,107 @@ void packet_test_ref(void) { odp_packet_t base_pkt, segmented_base_pkt, hdr_pkt[4], ref_pkt[4], refhdr_pkt[4], hdr_cpy; + odp_packet_t pkt, pkt2, pkt3, ref, ref2; uint32_t pkt_len, segmented_pkt_len, hdr_len[4], offset[4], hr[4], base_hr, ref_len[4]; - int i; + int i, ret; + odp_pool_t pool; + + /* Create references and compare data */ + pool = odp_packet_pool(test_packet); + + pkt = odp_packet_copy(test_packet, pool); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID) + ref = odp_packet_ref_static(pkt); + CU_ASSERT_FATAL(ref != ODP_PACKET_INVALID) + packet_compare_data(pkt, ref); + odp_packet_free(ref); + odp_packet_free(pkt); + + pkt = odp_packet_copy(test_packet, pool); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID) + ref = odp_packet_ref(pkt, 0); + CU_ASSERT_FATAL(ref != ODP_PACKET_INVALID) + packet_compare_data(pkt, ref); + odp_packet_free(ref); + odp_packet_free(pkt); + + pkt = odp_packet_copy(test_packet, pool); + pkt3 = odp_packet_copy(test_packet, pool); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID) + CU_ASSERT_FATAL(pkt3 != ODP_PACKET_INVALID) + ret = odp_packet_concat(&pkt3, pkt); + CU_ASSERT_FATAL(ret >= 0); + + pkt = odp_packet_copy(test_packet, pool); + pkt2 = odp_packet_copy(test_packet, pool); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID) + CU_ASSERT_FATAL(pkt2 != ODP_PACKET_INVALID) + ref = odp_packet_ref_pkt(pkt, 0, pkt2); + CU_ASSERT_FATAL(ref != ODP_PACKET_INVALID) + packet_compare_data(pkt3, ref); + odp_packet_free(ref); + odp_packet_free(pkt); + odp_packet_free(pkt3); + + /* Do the same for segmented packets */ + pool = odp_packet_pool(segmented_test_packet); + + pkt = odp_packet_copy(segmented_test_packet, pool); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID) + ref = odp_packet_ref_static(pkt); + CU_ASSERT_FATAL(ref != ODP_PACKET_INVALID) + packet_compare_data(pkt, ref); + odp_packet_free(ref); + odp_packet_free(pkt); + + pkt = odp_packet_copy(segmented_test_packet, pool); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID) + ref = odp_packet_ref(pkt, 0); + CU_ASSERT_FATAL(ref != ODP_PACKET_INVALID) + packet_compare_data(pkt, ref); + odp_packet_free(ref); + odp_packet_free(pkt); + + /* Avoid to create too large packets with concat */ + pool = odp_packet_pool(test_packet); + + pkt = odp_packet_copy(test_packet, pool); + pkt2 = odp_packet_copy(test_packet, pool); + pkt3 = odp_packet_copy(test_packet, pool); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID) + CU_ASSERT_FATAL(pkt2 != ODP_PACKET_INVALID) + CU_ASSERT_FATAL(pkt3 != ODP_PACKET_INVALID) + ret = odp_packet_concat(&pkt3, pkt2); + CU_ASSERT_FATAL(ret >= 0); + ret = odp_packet_concat(&pkt3, pkt); + CU_ASSERT_FATAL(ret >= 0); + + pkt = odp_packet_copy(test_packet, pool); + pkt2 = odp_packet_copy(test_packet, pool); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID) + CU_ASSERT_FATAL(pkt2 != ODP_PACKET_INVALID) + ref = odp_packet_ref_pkt(pkt, 0, pkt2); + CU_ASSERT_FATAL(ref != ODP_PACKET_INVALID) + pkt2 = odp_packet_copy(test_packet, pool); + CU_ASSERT_FATAL(pkt2 != ODP_PACKET_INVALID) + ref2 = odp_packet_ref_pkt(ref, 0, pkt2); + CU_ASSERT_FATAL(ref2 != ODP_PACKET_INVALID) + packet_compare_data(pkt3, ref2); + + /* Try print function on a reference */ + odp_packet_print(ref2); + + odp_packet_free(ref); + odp_packet_free(ref2); + odp_packet_free(pkt); + odp_packet_free(pkt3); + /* Test has_ref, unshared_len, lengths, etc */ base_pkt = odp_packet_copy(test_packet, odp_packet_pool(test_packet)); - base_hr = odp_packet_headroom(base_pkt); - pkt_len = odp_packet_len(test_packet); CU_ASSERT_FATAL(base_pkt != ODP_PACKET_INVALID); + base_hr = odp_packet_headroom(base_pkt); + pkt_len = odp_packet_len(test_packet); segmented_base_pkt = odp_packet_copy(segmented_test_packet, From patchwork Fri Sep 8 14:00:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112087 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp1671061ybm; Fri, 8 Sep 2017 07:08:51 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBBUO5OQcdTWrZRo9f4fJkgJ8Gmu8oqyT56puyYC1EcxBfyC6yaAeT6wRFo7CUlClYNnJeW X-Received: by 10.55.24.22 with SMTP id j22mr3643339qkh.310.1504879731306; Fri, 08 Sep 2017 07:08:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879731; cv=none; d=google.com; s=arc-20160816; b=Qg+OSkBLcHv1YOKjQh8sBqiwQgWAxtOgbB149jMR5S0Kdo5sY10r1j/eP3rCX+/6Ib PAxVX7EQDHlpz2uZl9UTwfgCR8nnYn53CnUJYbnWFlnwIUN7cWFu0lzJwDNe4dDV5ZEG P5/Olzi4BTjkQfGtZ0OvsDu+VPp4Qh0OSI5OSa6s6kCulKjY/DXJ+sNGK90FjeAHGmon eeupOPZxdZIIzuUHBOP36OAjQjbJNMx8HOKgk0RciZPZysEdEVWfNmke5GRyAqJeztUg gesu7JhMQ+EWTlOv80sLaZrwzSpNQkVZcYl1pJR0z8dANr4HCHlEI/NBuv/gcHclPrz6 79dg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=kYE3lftV4r4t+7l7SuB38jTf0RjhNBmnK35e/w9lV+E=; b=a6wGaKTPQbsn+oE4t+HNSIVOY8BxHRiPUGv95H8LuswMKcgv8valqlsLJ5a0Q724RQ Y7bZm6WDObkexLy7rk6ExvGgtDuiyzr/SIFAKdY2c8WRnVqpcCr5NJDDB40g0ii9qREi bsqSh8Bnifi1Fqta9HFzPmM8tO4BYAxHmXYieQCP1d++jiz+CNhk4zkBfHhdXpLm/U4m n48h4ybUjATbmRiPBaziRG0YI9ARsb7I0oJc8CVPMOGwMncPjfMjh0kSx+yUbWD0DhDT SM11cqrHYrh5FjTzf49H/edN9FFLdhh9SGSG5VtLjjuYApfgFSzZs3mFKIksOwqWRKCd VqSw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id y14si2157393qtc.525.2017.09.08.07.08.50; Fri, 08 Sep 2017 07:08:51 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id C3134644D8; Fri, 8 Sep 2017 14:08:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id E9583644DC; Fri, 8 Sep 2017 14:01:06 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 36F4562DF3; Fri, 8 Sep 2017 14:00:33 +0000 (UTC) Received: from forward105o.mail.yandex.net (forward105o.mail.yandex.net [37.140.190.183]) by lists.linaro.org (Postfix) with ESMTPS id 8CA0160722 for ; Fri, 8 Sep 2017 14:00:23 +0000 (UTC) Received: from mxback3o.mail.yandex.net (mxback3o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::1d]) by forward105o.mail.yandex.net (Yandex) with ESMTP id 19E454441866 for ; Fri, 8 Sep 2017 17:00:22 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback3o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id 0HZeSTvtoy-0MsGw2lr; Fri, 08 Sep 2017 17:00:22 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0JKWJk1u; Fri, 08 Sep 2017 17:00:19 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:06 +0300 Message-Id: <1504879212-3689-6-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 5/11] linux-gen: packet: re-implement packet segmentation X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Removed limitation to fixed number of segments (CONFIG_PACKET_MAX_SEGS) per packet. Packet headers carrying segment tables are now linked together with 'next_seg' pointer. Last header containing a segment table is pointed by 'last_seg' pointer. Other than the first segment are not referred directly but with seg_entry_ functions. Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ .../linux-generic/include/odp_buffer_internal.h | 25 +- .../linux-generic/include/odp_packet_internal.h | 19 +- platform/linux-generic/odp_packet.c | 641 ++++++++------------- platform/linux-generic/odp_pool.c | 3 + 4 files changed, 283 insertions(+), 405 deletions(-) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index e20d1c8d3..309094658 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -35,6 +35,12 @@ extern "C" { #define BUFFER_BURST_SIZE CONFIG_BURST_SIZE +typedef struct seg_entry_t { + void *hdr; + uint8_t *data; + uint32_t len; +} seg_entry_t; + /* Common buffer header */ struct odp_buffer_hdr_t { /* Buffer index in the pool */ @@ -54,15 +60,20 @@ struct odp_buffer_hdr_t { uint8_t burst_num; uint8_t burst_first; - /* Segment count */ - uint8_t segcount; + /* Number of seg[] entries used */ + uint8_t num_seg; + + /* Total segment count */ + uint32_t segcount; + + /* Next header which continues the segment list */ + void *next_seg; + + /* Last header of the segment list */ + void *last_seg; /* Segments */ - struct { - void *hdr; - uint8_t *data; - uint32_t len; - } seg[CONFIG_PACKET_MAX_SEGS]; + seg_entry_t seg[CONFIG_PACKET_MAX_SEGS]; /* Next buf in a list */ struct odp_buffer_hdr_t *next; diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index f77987c8d..d8e5766ce 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -172,6 +172,16 @@ static inline odp_packet_t packet_from_buf_hdr(odp_buffer_hdr_t *buf_hdr) return (odp_packet_t)(odp_packet_hdr_t *)buf_hdr; } +static inline seg_entry_t *seg_entry_last(odp_packet_hdr_t *hdr) +{ + odp_packet_hdr_t *last; + uint8_t last_seg; + + last = hdr->buf_hdr.last_seg; + last_seg = last->buf_hdr.num_seg - 1; + return &last->buf_hdr.seg[last_seg]; +} + /** * Initialize packet */ @@ -184,10 +194,13 @@ static inline void packet_init(odp_packet_hdr_t *pkt_hdr, uint32_t len) seg_len = len; pkt_hdr->buf_hdr.seg[0].len = len; } else { + seg_entry_t *last; + seg_len = len - ((num - 1) * CONFIG_PACKET_MAX_SEG_LEN); /* Last segment data length */ - pkt_hdr->buf_hdr.seg[num - 1].len = seg_len; + last = seg_entry_last(pkt_hdr); + last->len = seg_len; } pkt_hdr->p.input_flags.all = 0; @@ -229,11 +242,11 @@ static inline void copy_packet_cls_metadata(odp_packet_hdr_t *src_hdr, static inline void pull_tail(odp_packet_hdr_t *pkt_hdr, uint32_t len) { - int last = pkt_hdr->buf_hdr.segcount - 1; + seg_entry_t *last = seg_entry_last(pkt_hdr); pkt_hdr->tailroom += len; pkt_hdr->frame_len -= len; - pkt_hdr->buf_hdr.seg[last].len -= len; + last->len -= len; } static inline uint32_t packet_len(odp_packet_hdr_t *pkt_hdr) diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index ac0571243..94fdffab3 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -74,15 +74,72 @@ static inline odp_buffer_t packet_to_buffer(odp_packet_t pkt) return (odp_buffer_t)pkt; } +static inline seg_entry_t *seg_entry(odp_packet_hdr_t *hdr, + uint32_t seg_idx) +{ + uint32_t idx = 0; + uint8_t num_seg = hdr->buf_hdr.num_seg; + + while (odp_unlikely(idx + num_seg - 1 < seg_idx)) { + idx += num_seg; + hdr = hdr->buf_hdr.next_seg; + num_seg = hdr->buf_hdr.num_seg; + } + + idx = seg_idx - idx; + + return &hdr->buf_hdr.seg[idx]; +} + +static inline void seg_entry_find_idx(odp_packet_hdr_t **p_hdr, + uint8_t *p_idx, + uint32_t find_idx) +{ + odp_packet_hdr_t *hdr = *p_hdr; + uint32_t idx = 0; + uint8_t num_seg = hdr->buf_hdr.num_seg; + + while (odp_unlikely(idx + num_seg - 1 < find_idx)) { + idx += num_seg; + hdr = hdr->buf_hdr.next_seg; + num_seg = hdr->buf_hdr.num_seg; + } + + idx = find_idx - idx; + *p_hdr = hdr; + *p_idx = idx; +} + +static inline seg_entry_t *seg_entry_next(odp_packet_hdr_t **cur_hdr, + uint8_t *cur_idx) +{ + odp_packet_hdr_t *hdr = *cur_hdr; + uint8_t idx = *cur_idx; + uint8_t num_seg = hdr->buf_hdr.num_seg; + + if (idx == num_seg - 1) { + *cur_hdr = hdr->buf_hdr.next_seg; + *cur_idx = 0; + } else { + *cur_idx = idx + 1; + } + + return &hdr->buf_hdr.seg[idx]; +} + static inline uint32_t packet_seg_len(odp_packet_hdr_t *pkt_hdr, uint32_t seg_idx) { - return pkt_hdr->buf_hdr.seg[seg_idx].len; + seg_entry_t *seg = seg_entry(pkt_hdr, seg_idx); + + return seg->len; } static inline void *packet_seg_data(odp_packet_hdr_t *pkt_hdr, uint32_t seg_idx) { - return pkt_hdr->buf_hdr.seg[seg_idx].data; + seg_entry_t *seg = seg_entry(pkt_hdr, seg_idx); + + return seg->data; } static inline int packet_last_seg(odp_packet_hdr_t *pkt_hdr) @@ -95,14 +152,7 @@ static inline int packet_last_seg(odp_packet_hdr_t *pkt_hdr) static inline uint32_t packet_first_seg_len(odp_packet_hdr_t *pkt_hdr) { - return packet_seg_len(pkt_hdr, 0); -} - -static inline uint32_t packet_last_seg_len(odp_packet_hdr_t *pkt_hdr) -{ - int last = packet_last_seg(pkt_hdr); - - return packet_seg_len(pkt_hdr, last); + return pkt_hdr->buf_hdr.seg[0].len; } static inline void *packet_data(odp_packet_hdr_t *pkt_hdr) @@ -112,26 +162,27 @@ static inline void *packet_data(odp_packet_hdr_t *pkt_hdr) static inline void *packet_tail(odp_packet_hdr_t *pkt_hdr) { - int last = packet_last_seg(pkt_hdr); - uint32_t seg_len = pkt_hdr->buf_hdr.seg[last].len; + seg_entry_t *last_seg = seg_entry_last(pkt_hdr); - return pkt_hdr->buf_hdr.seg[last].data + seg_len; + return last_seg->data + last_seg->len; } -static inline uint32_t seg_headroom(odp_packet_hdr_t *pkt_hdr, int seg) +static inline uint32_t seg_headroom(odp_packet_hdr_t *pkt_hdr, int seg_idx) { - odp_buffer_hdr_t *hdr = pkt_hdr->buf_hdr.seg[seg].hdr; + seg_entry_t *seg = seg_entry(pkt_hdr, seg_idx); + odp_buffer_hdr_t *hdr = seg->hdr; uint8_t *base = hdr->base_data; - uint8_t *head = pkt_hdr->buf_hdr.seg[seg].data; + uint8_t *head = seg->data; return CONFIG_PACKET_HEADROOM + (head - base); } -static inline uint32_t seg_tailroom(odp_packet_hdr_t *pkt_hdr, int seg) +static inline uint32_t seg_tailroom(odp_packet_hdr_t *pkt_hdr, int seg_idx) { - uint32_t seg_len = pkt_hdr->buf_hdr.seg[seg].len; - odp_buffer_hdr_t *hdr = pkt_hdr->buf_hdr.seg[seg].hdr; - uint8_t *tail = pkt_hdr->buf_hdr.seg[seg].data + seg_len; + seg_entry_t *seg = seg_entry(pkt_hdr, seg_idx); + + odp_buffer_hdr_t *hdr = seg->hdr; + uint8_t *tail = seg->data + seg->len; return hdr->buf_end - tail; } @@ -154,11 +205,11 @@ static inline void pull_head(odp_packet_hdr_t *pkt_hdr, uint32_t len) static inline void push_tail(odp_packet_hdr_t *pkt_hdr, uint32_t len) { - int last = packet_last_seg(pkt_hdr); + seg_entry_t *last_seg = seg_entry_last(pkt_hdr); pkt_hdr->tailroom -= len; pkt_hdr->frame_len += len; - pkt_hdr->buf_hdr.seg[last].len += len; + last_seg->len += len; } /* Copy all metadata for segmentation modification. Segment data and lengths @@ -188,6 +239,9 @@ static inline void packet_seg_copy_md(odp_packet_hdr_t *dst, /* segmentation data is not copied: * buf_hdr.seg[] * buf_hdr.segcount + * buf_hdr.num_seg + * buf_hdr.next_seg + * buf_hdr.last_seg */ } @@ -196,7 +250,7 @@ static inline void *packet_map(odp_packet_hdr_t *pkt_hdr, { void *addr; uint32_t len; - int seg = 0; + int seg_id = 0; int seg_count = pkt_hdr->buf_hdr.segcount; if (odp_unlikely(offset >= pkt_hdr->frame_len)) @@ -207,10 +261,14 @@ static inline void *packet_map(odp_packet_hdr_t *pkt_hdr, len = pkt_hdr->buf_hdr.seg[0].len - offset; } else { int i; + seg_entry_t *seg = NULL; uint32_t seg_start = 0, seg_end = 0; + odp_packet_hdr_t *hdr = pkt_hdr; + uint8_t idx = 0; for (i = 0; i < seg_count; i++) { - seg_end += pkt_hdr->buf_hdr.seg[i].len; + seg = seg_entry_next(&hdr, &idx); + seg_end += seg->len; if (odp_likely(offset < seg_end)) break; @@ -218,16 +276,16 @@ static inline void *packet_map(odp_packet_hdr_t *pkt_hdr, seg_start = seg_end; } - addr = pkt_hdr->buf_hdr.seg[i].data + (offset - seg_start); - len = pkt_hdr->buf_hdr.seg[i].len - (offset - seg_start); - seg = i; + addr = seg->data + (offset - seg_start); + len = seg->len - (offset - seg_start); + seg_id = i; } if (seg_len) *seg_len = len; if (seg_idx) - *seg_idx = seg; + *seg_idx = seg_id; return addr; } @@ -243,10 +301,43 @@ void packet_parse_reset(odp_packet_hdr_t *pkt_hdr) pkt_hdr->p.l4_offset = ODP_PACKET_OFFSET_INVALID; } +static inline void link_segments(odp_packet_hdr_t *pkt_hdr[], int num) +{ + int cur, i; + odp_packet_hdr_t *hdr; + odp_packet_hdr_t *head = pkt_hdr[0]; + + cur = 0; + + while (1) { + hdr = pkt_hdr[cur]; + + for (i = 0; i < CONFIG_PACKET_MAX_SEGS; i++) { + odp_buffer_hdr_t *buf_hdr; + + buf_hdr = &pkt_hdr[cur]->buf_hdr; + hdr->buf_hdr.seg[i].hdr = buf_hdr; + hdr->buf_hdr.seg[i].data = buf_hdr->base_data; + hdr->buf_hdr.seg[i].len = BASE_LEN; + cur++; + + if (cur == num) { + /* Last segment */ + hdr->buf_hdr.num_seg = i + 1; + hdr->buf_hdr.next_seg = NULL; + head->buf_hdr.last_seg = &hdr->buf_hdr; + return; + } + } + + hdr->buf_hdr.num_seg = CONFIG_PACKET_MAX_SEGS; + hdr->buf_hdr.next_seg = pkt_hdr[cur]; + } +} + static inline void init_segments(odp_packet_hdr_t *pkt_hdr[], int num) { odp_packet_hdr_t *hdr; - int i; /* First segment is the packet descriptor */ hdr = pkt_hdr[0]; @@ -258,30 +349,33 @@ static inline void init_segments(odp_packet_hdr_t *pkt_hdr[], int num) if (CONFIG_PACKET_MAX_SEGS != 1) { hdr->buf_hdr.segcount = num; + /* Defaults for single segment packet */ + hdr->buf_hdr.num_seg = 1; + hdr->buf_hdr.next_seg = NULL; + hdr->buf_hdr.last_seg = &hdr->buf_hdr; + if (odp_unlikely(num > 1)) { - for (i = 1; i < num; i++) { - odp_buffer_hdr_t *buf_hdr; + link_segments(pkt_hdr, num); - buf_hdr = &pkt_hdr[i]->buf_hdr; - hdr->buf_hdr.seg[i].hdr = buf_hdr; - hdr->buf_hdr.seg[i].data = buf_hdr->base_data; - hdr->buf_hdr.seg[i].len = BASE_LEN; - } } } } static inline void reset_seg(odp_packet_hdr_t *pkt_hdr, int first, int num) { - odp_buffer_hdr_t *hdr; + odp_packet_hdr_t *hdr = pkt_hdr; void *base; int i; + seg_entry_t *seg; + uint8_t idx; + + seg_entry_find_idx(&hdr, &idx, first); - for (i = first; i < first + num; i++) { - hdr = pkt_hdr->buf_hdr.seg[i].hdr; - base = hdr->base_data; - pkt_hdr->buf_hdr.seg[i].len = BASE_LEN; - pkt_hdr->buf_hdr.seg[i].data = base; + for (i = 0; i < num; i++) { + base = hdr->buf_hdr.base_data; + seg = seg_entry_next(&hdr, &idx); + seg->len = BASE_LEN; + seg->data = base; } } @@ -309,31 +403,11 @@ static inline int num_segments(uint32_t len) static inline void add_all_segs(odp_packet_hdr_t *to, odp_packet_hdr_t *from) { - int i; - int n = to->buf_hdr.segcount; - int num = from->buf_hdr.segcount; - - for (i = 0; i < num; i++) { - to->buf_hdr.seg[n + i].hdr = from->buf_hdr.seg[i].hdr; - to->buf_hdr.seg[n + i].data = from->buf_hdr.seg[i].data; - to->buf_hdr.seg[n + i].len = from->buf_hdr.seg[i].len; - } - - to->buf_hdr.segcount = n + num; -} - -static inline void copy_num_segs(odp_packet_hdr_t *to, odp_packet_hdr_t *from, - int first, int num) -{ - int i; - - for (i = 0; i < num; i++) { - to->buf_hdr.seg[i].hdr = from->buf_hdr.seg[first + i].hdr; - to->buf_hdr.seg[i].data = from->buf_hdr.seg[first + i].data; - to->buf_hdr.seg[i].len = from->buf_hdr.seg[first + i].len; - } + odp_packet_hdr_t *last = to->buf_hdr.last_seg; - to->buf_hdr.segcount = num; + last->buf_hdr.next_seg = from; + to->buf_hdr.last_seg = from->buf_hdr.last_seg; + to->buf_hdr.segcount += from->buf_hdr.segcount; } static inline odp_packet_hdr_t *alloc_segments(pool_t *pool, int num) @@ -385,14 +459,14 @@ static inline odp_packet_hdr_t *add_segments(odp_packet_hdr_t *pkt_hdr, pkt_hdr = new_hdr; } else { - int last; + seg_entry_t *last_seg; /* add into the tail */ add_all_segs(pkt_hdr, new_hdr); /* adjust last segment length */ - last = packet_last_seg(pkt_hdr); - pkt_hdr->buf_hdr.seg[last].len = seg_len; + last_seg = seg_entry_last(pkt_hdr); + last_seg->len = seg_len; pkt_hdr->frame_len += len; pkt_hdr->tailroom = pool->tailroom + offset; @@ -401,13 +475,33 @@ static inline odp_packet_hdr_t *add_segments(odp_packet_hdr_t *pkt_hdr, return pkt_hdr; } -static inline void free_bufs(odp_packet_hdr_t *pkt_hdr, int first, int num) +static inline void copy_buf_hdr(odp_packet_hdr_t *pkt_hdr, int first, int num, + odp_buffer_hdr_t *buf_hdr[]) +{ + seg_entry_t *seg; + int i; + uint8_t idx; + odp_packet_hdr_t *hdr = pkt_hdr; + + seg_entry_find_idx(&hdr, &idx, first); + + for (i = 0; i < num; i++) { + seg = seg_entry_next(&hdr, &idx); + buf_hdr[i] = seg->hdr; + } +} + +static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) { + seg_entry_t *seg; int i; odp_buffer_hdr_t *buf_hdr[num]; + uint8_t idx = 0; - for (i = 0; i < num; i++) - buf_hdr[i] = pkt_hdr->buf_hdr.seg[first + i].hdr; + for (i = 0; i < num; i++) { + seg = seg_entry_next(&pkt_hdr, &idx); + buf_hdr[i] = seg->hdr; + } buffer_free_multi(buf_hdr, num); } @@ -416,20 +510,45 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, int num, uint32_t free_len, uint32_t pull_len, int head) { + seg_entry_t *seg; + int i; int num_remain = pkt_hdr->buf_hdr.segcount - num; + odp_packet_hdr_t *hdr = pkt_hdr; + odp_packet_hdr_t *last_hdr = pkt_hdr->buf_hdr.last_seg; + uint8_t idx; + uint8_t num_seg; + odp_buffer_hdr_t *buf_hdr[num]; if (head) { odp_packet_hdr_t *new_hdr; - int i; - odp_buffer_hdr_t *buf_hdr[num]; - for (i = 0; i < num; i++) - buf_hdr[i] = pkt_hdr->buf_hdr.seg[i].hdr; + idx = 0; + for (i = 0; i < num; i++) { + seg = seg_entry_next(&hdr, &idx); + buf_hdr[i] = seg->hdr; + } + + /* The first remaining header is the new packet descriptor. + * Copy remaining segments from the last to-be-removed header + * to the new header. */ + new_hdr = hdr->buf_hdr.seg[idx].hdr; + num_seg = hdr->buf_hdr.num_seg - idx; + + new_hdr->buf_hdr.next_seg = hdr->buf_hdr.next_seg; + + if (hdr == last_hdr) + new_hdr->buf_hdr.last_seg = new_hdr; + else + new_hdr->buf_hdr.last_seg = last_hdr; + + new_hdr->buf_hdr.num_seg = num_seg; + new_hdr->buf_hdr.segcount = num_remain; - /* First remaining segment is the new packet descriptor */ - new_hdr = pkt_hdr->buf_hdr.seg[num].hdr; + for (i = 0; i < num_seg; i++) { + seg = seg_entry_next(&hdr, &idx); + new_hdr->buf_hdr.seg[i] = *seg; + } - copy_num_segs(new_hdr, pkt_hdr, num, num_remain); packet_seg_copy_md(new_hdr, pkt_hdr); /* Tailroom not changed */ @@ -443,11 +562,27 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, buffer_free_multi(buf_hdr, num); } else { - /* Free last 'num' bufs */ - free_bufs(pkt_hdr, num_remain, num); + /* Free last 'num' bufs. + * First, find the last remaining header. */ + seg_entry_find_idx(&hdr, &idx, num_remain - 1); + last_hdr = hdr; + num_seg = idx + 1; + + seg_entry_next(&hdr, &idx); + + for (i = 0; i < num; i++) { + seg = seg_entry_next(&hdr, &idx); + buf_hdr[i] = seg->hdr; + } + + buffer_free_multi(buf_hdr, num); /* Head segment remains, no need to copy or update majority * of the metadata. */ + last_hdr->buf_hdr.num_seg = num_seg; + last_hdr->buf_hdr.next_seg = NULL; + + pkt_hdr->buf_hdr.last_seg = last_hdr; pkt_hdr->buf_hdr.segcount = num_remain; pkt_hdr->frame_len -= free_len; pkt_hdr->tailroom = seg_tailroom(pkt_hdr, num_remain - 1); @@ -566,7 +701,7 @@ void odp_packet_free(odp_packet_t pkt) if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1)) buffer_free_multi((odp_buffer_hdr_t **)&hdl, 1); else - free_bufs(pkt_hdr, 0, num_seg); + free_all_segments(pkt_hdr, num_seg); } void odp_packet_free_multi(const odp_packet_t pkt[], int num) @@ -576,7 +711,6 @@ void odp_packet_free_multi(const odp_packet_t pkt[], int num) } else { odp_buffer_hdr_t *buf_hdr[num * CONFIG_PACKET_MAX_SEGS]; int i; - int j; int bufs = 0; for (i = 0; i < num; i++) { @@ -590,10 +724,8 @@ void odp_packet_free_multi(const odp_packet_t pkt[], int num) if (odp_likely(num_seg == 1)) continue; - for (j = 1; j < num_seg; j++) { - buf_hdr[bufs] = hdr->seg[j].hdr; - bufs++; - } + copy_buf_hdr(pkt_hdr, 1, num_seg - 1, &buf_hdr[bufs]); + bufs += num_seg - 1; } buffer_free_multi(buf_hdr, bufs); @@ -664,166 +796,6 @@ void *odp_packet_push_head(odp_packet_t pkt, uint32_t len) return packet_data(pkt_hdr); } -static inline uint32_t pack_seg_head(odp_packet_hdr_t *pkt_hdr, int seg) -{ - odp_buffer_hdr_t *hdr = pkt_hdr->buf_hdr.seg[seg].hdr; - uint32_t len = pkt_hdr->buf_hdr.seg[seg].len; - uint8_t *src = pkt_hdr->buf_hdr.seg[seg].data; - uint8_t *dst = hdr->base_data; - - if (dst != src) { - memmove(dst, src, len); - pkt_hdr->buf_hdr.seg[seg].data = dst; - } - - return len; -} - -static inline uint32_t pack_seg_tail(odp_packet_hdr_t *pkt_hdr, int seg) -{ - odp_buffer_hdr_t *hdr = pkt_hdr->buf_hdr.seg[seg].hdr; - uint32_t len = pkt_hdr->buf_hdr.seg[seg].len; - uint8_t *src = pkt_hdr->buf_hdr.seg[seg].data; - uint8_t *dst = hdr->base_data + BASE_LEN - len; - - if (dst != src) { - memmove(dst, src, len); - pkt_hdr->buf_hdr.seg[seg].data = dst; - } - - return len; -} - -static inline uint32_t fill_seg_head(odp_packet_hdr_t *pkt_hdr, int dst_seg, - int src_seg, uint32_t max_len) -{ - uint32_t len = pkt_hdr->buf_hdr.seg[src_seg].len; - uint8_t *src = pkt_hdr->buf_hdr.seg[src_seg].data; - uint32_t offset = pkt_hdr->buf_hdr.seg[dst_seg].len; - uint8_t *dst = pkt_hdr->buf_hdr.seg[dst_seg].data + offset; - - if (len > max_len) - len = max_len; - - memmove(dst, src, len); - - pkt_hdr->buf_hdr.seg[dst_seg].len += len; - pkt_hdr->buf_hdr.seg[src_seg].len -= len; - pkt_hdr->buf_hdr.seg[src_seg].data += len; - - if (pkt_hdr->buf_hdr.seg[src_seg].len == 0) { - odp_buffer_hdr_t *hdr = pkt_hdr->buf_hdr.seg[src_seg].hdr; - - pkt_hdr->buf_hdr.seg[src_seg].data = hdr->base_data; - } - - return len; -} - -static inline uint32_t fill_seg_tail(odp_packet_hdr_t *pkt_hdr, int dst_seg, - int src_seg, uint32_t max_len) -{ - uint32_t src_len = pkt_hdr->buf_hdr.seg[src_seg].len; - uint8_t *src = pkt_hdr->buf_hdr.seg[src_seg].data; - uint8_t *dst = pkt_hdr->buf_hdr.seg[dst_seg].data; - uint32_t len = src_len; - - if (len > max_len) - len = max_len; - - src += src_len - len; - dst -= len; - - memmove(dst, src, len); - - pkt_hdr->buf_hdr.seg[dst_seg].data -= len; - pkt_hdr->buf_hdr.seg[dst_seg].len += len; - pkt_hdr->buf_hdr.seg[src_seg].len -= len; - - if (pkt_hdr->buf_hdr.seg[src_seg].len == 0) { - odp_buffer_hdr_t *hdr = pkt_hdr->buf_hdr.seg[src_seg].hdr; - - pkt_hdr->buf_hdr.seg[src_seg].data = hdr->base_data; - } - - return len; -} - -static inline int move_data_to_head(odp_packet_hdr_t *pkt_hdr, int segs) -{ - int dst_seg, src_seg; - uint32_t len, free_len; - uint32_t moved = 0; - - for (dst_seg = 0; dst_seg < segs; dst_seg++) { - len = pack_seg_head(pkt_hdr, dst_seg); - moved += len; - - if (len == BASE_LEN) - continue; - - free_len = BASE_LEN - len; - - for (src_seg = dst_seg + 1; CONFIG_PACKET_MAX_SEGS > 1 && - src_seg < segs; src_seg++) { - len = fill_seg_head(pkt_hdr, dst_seg, src_seg, - free_len); - moved += len; - - if (len == free_len) { - /* dst seg is full */ - break; - } - - /* src seg is empty */ - free_len -= len; - } - - if (moved == pkt_hdr->frame_len) - break; - } - - /* last segment which have data */ - return dst_seg; -} - -static inline int move_data_to_tail(odp_packet_hdr_t *pkt_hdr, int segs) -{ - int dst_seg, src_seg; - uint32_t len, free_len; - uint32_t moved = 0; - - for (dst_seg = segs - 1; dst_seg >= 0; dst_seg--) { - len = pack_seg_tail(pkt_hdr, dst_seg); - moved += len; - - if (len == BASE_LEN) - continue; - - free_len = BASE_LEN - len; - - for (src_seg = dst_seg - 1; src_seg >= 0; src_seg--) { - len = fill_seg_tail(pkt_hdr, dst_seg, src_seg, - free_len); - moved += len; - - if (len == free_len) { - /* dst seg is full */ - break; - } - - /* src seg is empty */ - free_len -= len; - } - - if (moved == pkt_hdr->frame_len) - break; - } - - /* first segment which have data */ - return dst_seg; -} - int odp_packet_extend_head(odp_packet_t *pkt, uint32_t len, void **data_ptr, uint32_t *seg_len) { @@ -835,87 +807,23 @@ int odp_packet_extend_head(odp_packet_t *pkt, uint32_t len, if (len > headroom) { pool_t *pool = pkt_hdr->buf_hdr.pool_ptr; int num; - int segs; + void *ptr; if (odp_unlikely((frame_len + len) > pool->max_len)) return -1; - num = num_segments(len - headroom); - segs = pkt_hdr->buf_hdr.segcount; - - if (odp_unlikely((segs + num) > CONFIG_PACKET_MAX_SEGS)) { - /* Cannot directly add new segments */ - odp_packet_hdr_t *new_hdr; - int new_segs = 0; - int free_segs = 0; - uint32_t offset; - - num = num_segments(frame_len + len); - - if (num > segs) { - /* Allocate additional segments */ - new_segs = num - segs; - new_hdr = alloc_segments(pool, new_segs); - - if (new_hdr == NULL) - return -1; - - } else if (num < segs) { - free_segs = segs - num; - } - - /* Pack all data to packet tail */ - move_data_to_tail(pkt_hdr, segs); - reset_seg(pkt_hdr, 0, segs); - - if (new_segs) { - add_all_segs(new_hdr, pkt_hdr); - packet_seg_copy_md(new_hdr, pkt_hdr); - segs += new_segs; - - pkt_hdr = new_hdr; - *pkt = packet_handle(pkt_hdr); - } else if (CONFIG_PACKET_MAX_SEGS > 1 && free_segs) { - new_hdr = pkt_hdr->buf_hdr.seg[free_segs].hdr; - packet_seg_copy_md(new_hdr, pkt_hdr); - - /* Free extra segs */ - free_bufs(pkt_hdr, 0, free_segs); - - segs -= free_segs; - pkt_hdr = new_hdr; - *pkt = packet_handle(pkt_hdr); - } - - frame_len += len; - offset = (segs * BASE_LEN) - frame_len; - - pkt_hdr->buf_hdr.seg[0].data += offset; - pkt_hdr->buf_hdr.seg[0].len -= offset; - - pkt_hdr->buf_hdr.segcount = segs; - pkt_hdr->frame_len = frame_len; - pkt_hdr->headroom = offset + pool->headroom; - pkt_hdr->tailroom = pool->tailroom; - - /* Data was moved */ - ret = 1; - } else { - void *ptr; + num = num_segments(len - headroom); + push_head(pkt_hdr, headroom); + ptr = add_segments(pkt_hdr, pool, len - headroom, num, 1); - push_head(pkt_hdr, headroom); - ptr = add_segments(pkt_hdr, pool, len - headroom, - num, 1); - - if (ptr == NULL) { - /* segment alloc failed, rollback changes */ - pull_head(pkt_hdr, headroom); - return -1; - } - - *pkt = packet_handle(ptr); - pkt_hdr = ptr; + if (ptr == NULL) { + /* segment alloc failed, rollback changes */ + pull_head(pkt_hdr, headroom); + return -1; } + + *pkt = packet_handle(ptr); + pkt_hdr = ptr; } else { push_head(pkt_hdr, len); } @@ -1001,75 +909,19 @@ int odp_packet_extend_tail(odp_packet_t *pkt, uint32_t len, if (len > tailroom) { pool_t *pool = pkt_hdr->buf_hdr.pool_ptr; int num; - int segs; + void *ptr; if (odp_unlikely((frame_len + len) > pool->max_len)) return -1; - num = num_segments(len - tailroom); - segs = pkt_hdr->buf_hdr.segcount; - - if (odp_unlikely((segs + num) > CONFIG_PACKET_MAX_SEGS)) { - /* Cannot directly add new segments */ - odp_packet_hdr_t *new_hdr; - int new_segs = 0; - int free_segs = 0; - uint32_t offset; - - num = num_segments(frame_len + len); - - if (num > segs) { - /* Allocate additional segments */ - new_segs = num - segs; - new_hdr = alloc_segments(pool, new_segs); - - if (new_hdr == NULL) - return -1; - - } else if (num < segs) { - free_segs = segs - num; - } - - /* Pack all data to packet head */ - move_data_to_head(pkt_hdr, segs); - reset_seg(pkt_hdr, 0, segs); + num = num_segments(len - tailroom); + push_tail(pkt_hdr, tailroom); + ptr = add_segments(pkt_hdr, pool, len - tailroom, num, 0); - if (new_segs) { - /* Add new segs */ - add_all_segs(pkt_hdr, new_hdr); - segs += new_segs; - } else if (free_segs) { - /* Free extra segs */ - free_bufs(pkt_hdr, segs - free_segs, free_segs); - - segs -= free_segs; - } - - frame_len += len; - offset = (segs * BASE_LEN) - frame_len; - - pkt_hdr->buf_hdr.seg[segs - 1].len -= offset; - - pkt_hdr->buf_hdr.segcount = segs; - pkt_hdr->frame_len = frame_len; - pkt_hdr->headroom = pool->headroom; - pkt_hdr->tailroom = offset + pool->tailroom; - - /* Data was moved */ - ret = 1; - } else { - void *ptr; - - push_tail(pkt_hdr, tailroom); - - ptr = add_segments(pkt_hdr, pool, len - tailroom, - num, 0); - - if (ptr == NULL) { - /* segment alloc failed, rollback changes */ - pull_tail(pkt_hdr, tailroom); - return -1; - } + if (ptr == NULL) { + /* segment alloc failed, rollback changes */ + pull_tail(pkt_hdr, tailroom); + return -1; } } else { push_tail(pkt_hdr, len); @@ -1084,8 +936,9 @@ int odp_packet_extend_tail(odp_packet_t *pkt, uint32_t len, void *odp_packet_pull_tail(odp_packet_t pkt, uint32_t len) { odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + seg_entry_t *last_seg = seg_entry_last(pkt_hdr); - if (len > packet_last_seg_len(pkt_hdr)) + if (len > last_seg->len) return NULL; pull_tail(pkt_hdr, len); @@ -1098,13 +951,15 @@ int odp_packet_trunc_tail(odp_packet_t *pkt, uint32_t len, { int last; uint32_t seg_len; + seg_entry_t *last_seg; odp_packet_hdr_t *pkt_hdr = packet_hdr(*pkt); if (len > pkt_hdr->frame_len) return -1; - last = packet_last_seg(pkt_hdr); - seg_len = packet_seg_len(pkt_hdr, last); + last = packet_last_seg(pkt_hdr); + last_seg = seg_entry_last(pkt_hdr); + seg_len = last_seg->len; if (len < seg_len) { pull_tail(pkt_hdr, len); @@ -1397,17 +1252,13 @@ int odp_packet_concat(odp_packet_t *dst, odp_packet_t src) { odp_packet_hdr_t *dst_hdr = packet_hdr(*dst); odp_packet_hdr_t *src_hdr = packet_hdr(src); - int dst_segs = dst_hdr->buf_hdr.segcount; - int src_segs = src_hdr->buf_hdr.segcount; pool_t *dst_pool = dst_hdr->buf_hdr.pool_ptr; pool_t *src_pool = src_hdr->buf_hdr.pool_ptr; uint32_t dst_len = dst_hdr->frame_len; uint32_t src_len = src_hdr->frame_len; - /* Do a copy if resulting packet would be out of segments or packets - * are from different pools. */ - if (odp_unlikely((dst_segs + src_segs) > CONFIG_PACKET_MAX_SEGS) || - odp_unlikely(dst_pool != src_pool)) { + /* Do a copy if packets are from different pools. */ + if (odp_unlikely(dst_pool != src_pool)) { if (odp_packet_extend_tail(dst, src_len, NULL, NULL) >= 0) { (void)odp_packet_copy_from_pkt(*dst, dst_len, src, 0, src_len); diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 01edff8b5..47a39f5b5 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -268,6 +268,9 @@ static void init_buffers(pool_t *pool) /* Show user requested size through API */ buf_hdr->uarea_size = pool->params.pkt.uarea_size; buf_hdr->segcount = 1; + buf_hdr->num_seg = 1; + buf_hdr->next_seg = NULL; + buf_hdr->last_seg = buf_hdr; /* Pointer to data start (of the first segment) */ buf_hdr->seg[0].hdr = buf_hdr; From patchwork Fri Sep 8 14:00:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112083 Delivered-To: patch@linaro.org Received: by 10.140.106.117 with SMTP id d108csp364129qgf; Fri, 8 Sep 2017 07:03:36 -0700 (PDT) X-Google-Smtp-Source: AOwi7QCR4UoOfwcMMzOcf2Em8s37PeYgSieSb17TH09hiK1oFSFasOsst02exrVcHDPATrWOztSP X-Received: by 10.237.34.66 with SMTP id o2mr3914038qtc.69.1504879415972; Fri, 08 Sep 2017 07:03:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879415; cv=none; d=google.com; s=arc-20160816; b=i2OjyYUyHscKv2kXXK+tpuEtwfh1L4J80HGiiBW0yDtW8MVLqIBt355yBdzybY0Uqp kdC9FFP9RX5s4fThcFTJEFfPUF7eeL34koZGKup2BSUm/08OA6sFGVtRjvI9HeKJ2zU7 F7b8EkH+MjjIf1zpAsPgZVNCalXKVpzBdJEwFaSS95g03CPhWM8pPX5wsJm2pAgO7Bgz pFFdj9MCiQ7A1/OGn2b35DH8MIH1Ugqg3CHMPmoTEge6C8YN/435vR6NAoP4YVBS1WOE 5Te3QEBh4AZ7IpCd+wrIGBuyGrUvmCd09XqgABOOJ/rICmf79EVdqRykFkWkfhDuRj6H C2fg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=3sQ3KDm5DKVxKwmLU7jyau0PUUmWNOiAEbsCJJsBgO0=; b=YZZNEsli8Y69ACVTzaujkbe2qcgL6MlMheB16eiPcje6cUojYfAGq3R2Vg5j1CaGKE 3mwhUfkq37aKiydvZrGyYLNXBr4sn1vBfpIvpYIFIbH+Ribqk1AvMB97kCpWJDkqkyyw 2oXxYXO5bE5f/FDLzqpdsyPfIM6q+VIvnzcoiAI/PNRf22qzX3qfjlhnbdjiErldRtbx wH8Pe+YdrOxr3HT/hllEHQtyEJLmtseEoJ1EmR7xyM2MgLzsaH9oZG12baE90Js0H57H JW6By3E+XLxoceJIi036n44uZ5VLGlNgSmy5JOzZn+7BfbldplMfnlcJZLQNZ9TAdOWu 7tcQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id u46si2231027qtk.309.2017.09.08.07.03.31; Fri, 08 Sep 2017 07:03:35 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 6367365C0D; Fri, 8 Sep 2017 14:03:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id C1472644B2; Fri, 8 Sep 2017 14:00:47 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 9AEEE62DF3; Fri, 8 Sep 2017 14:00:28 +0000 (UTC) Received: from forward101o.mail.yandex.net (forward101o.mail.yandex.net [37.140.190.181]) by lists.linaro.org (Postfix) with ESMTPS id 38DB360730 for ; Fri, 8 Sep 2017 14:00:24 +0000 (UTC) Received: from mxback7o.mail.yandex.net (mxback7o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::21]) by forward101o.mail.yandex.net (Yandex) with ESMTP id B22AF1344F5E for ; Fri, 8 Sep 2017 17:00:22 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback7o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id N6aDBQ9uib-0MV4Oi7j; Fri, 08 Sep 2017 17:00:22 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0MK86C52; Fri, 08 Sep 2017 17:00:22 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:07 +0300 Message-Id: <1504879212-3689-7-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 6/11] linux-gen: buffer: optimize header layout X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Pack most often used buffer header fields (including seg[0]) into the first cache line. Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ .../linux-generic/include/odp_buffer_internal.h | 43 ++++++++++++---------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 309094658..0c873d2d2 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -43,41 +43,46 @@ typedef struct seg_entry_t { /* Common buffer header */ struct odp_buffer_hdr_t { - /* Buffer index in the pool */ - uint32_t index; - /* Initial buffer data pointer and length */ - uint8_t *base_data; - uint8_t *buf_end; + /* Buffer index in the pool */ + uint32_t index; - /* Max data size */ - uint32_t size; + /* Total segment count */ + uint16_t segcount; /* Pool type */ int8_t type; - /* Burst counts */ - uint8_t burst_num; - uint8_t burst_first; - /* Number of seg[] entries used */ uint8_t num_seg; - /* Total segment count */ - uint32_t segcount; - /* Next header which continues the segment list */ void *next_seg; /* Last header of the segment list */ void *last_seg; + /* Initial buffer data pointer and length */ + uint8_t *base_data; + uint8_t *buf_end; + + /* --- 40 bytes --- */ + /* Segments */ seg_entry_t seg[CONFIG_PACKET_MAX_SEGS]; + /* Burst counts */ + uint8_t burst_num; + uint8_t burst_first; + /* Next buf in a list */ struct odp_buffer_hdr_t *next; + /* Burst table */ + struct odp_buffer_hdr_t *burst[BUFFER_BURST_SIZE]; + + /* --- Mostly read only data --- */ + /* User context pointer or u64 */ union { uint64_t buf_u64; @@ -85,6 +90,9 @@ struct odp_buffer_hdr_t { const void *buf_cctx; /* const alias for ctx */ }; + /* Pool pointer */ + void *pool_ptr; + /* User area pointer */ void *uarea_addr; @@ -94,9 +102,6 @@ struct odp_buffer_hdr_t { /* Event type. Maybe different than pool type (crypto compl event) */ int8_t event_type; - /* Burst table */ - struct odp_buffer_hdr_t *burst[BUFFER_BURST_SIZE]; - /* ipc mapped process can not walk over pointers, * offset has to be used */ uint64_t ipc_data_offset; @@ -105,8 +110,8 @@ struct odp_buffer_hdr_t { * inlining */ odp_pool_t pool_hdl; - /* Pool pointer */ - void *pool_ptr; + /* Max data size */ + uint32_t size; /* Data or next header */ uint8_t data[0]; From patchwork Fri Sep 8 14:00:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112089 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp1675645ybm; Fri, 8 Sep 2017 07:11:53 -0700 (PDT) X-Google-Smtp-Source: ADKCNb5BpsCEbhbgAopmEl0BePAxybg6sDuLhRrMVJmwoKYPU1baW9Iy6b9T1Th7jLDmLmKsT1JD X-Received: by 10.55.111.1 with SMTP id k1mr1284338qkc.15.1504879913585; Fri, 08 Sep 2017 07:11:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879913; cv=none; d=google.com; s=arc-20160816; b=0QGwU34EPiHkY9so/wTFpfK/OcNzOMXU+hdaDoIBDI55AiL3+nXJfe433pjC67C+z9 ukmsn6a0IWjYvMHEJtKzDYL9hrFRTZLqhZFG+iuuQWlKIi9p0cZygYRheCROumx/bAai CIyuOuGnvXLmLtZdUrWhnRqGuj5RXdFC5/Wdhz46bS5QxWZU6JuOjBGZW2tuO/Iqtlrn tJpvtdgIMM2nqYAwkVFBJgVHIpddbbVF2bbwF2URUDqiNckB7ytHmeQAaScW+d6vvnsT 8GN0VCHGIqGflp2BJbM1mL5XHhCCaID03hyUdC0aITTK4wiFJLg1ppUFMDLE1tODLcmM yueg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=oClz0WnbpPB8IhaUjSNABgelCA+rv4rRkVg1p5Zyj7w=; b=dNi5s3sNTtHUHV7PR8/VlnuVq5h7IdsktlG3g1MmbJufPQZKeHntZ2fVFSGRA4UWNV iPUabdOj5Hq4OSUWCJBOnQLB7qnC6kooZtO+iPnqp3I2yQH8sAZYLVukutRt20bKisyi FYq0m48p3uauO3J6hbTH14YuaVbPMpxF/qTxGKJQlKrLAkrPudm1tNR+KQCKTNsSxn46 f2+kwxZsSIPEFJ5xamidsoqUC6IQ3CinIk6iwb/cW4N5NVWoucNeX2W6b3Jw5mq0l7cN dU/YUFPfBh5F1kybqyhdbY/PGrjpU8yWYPZl3MFDos1xnW6P989DFaFjcuH/iOE3bU1Y FHUA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id o19si2209590qtc.69.2017.09.08.07.11.53; Fri, 08 Sep 2017 07:11:53 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 3DA8260903; Fri, 8 Sep 2017 14:11:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 8ACE760C2D; Fri, 8 Sep 2017 14:01:23 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 1E5E4644B0; Fri, 8 Sep 2017 14:00:37 +0000 (UTC) Received: from forward101j.mail.yandex.net (forward101j.mail.yandex.net [5.45.198.241]) by lists.linaro.org (Postfix) with ESMTPS id AE6EF60A4D for ; Fri, 8 Sep 2017 14:00:24 +0000 (UTC) Received: from mxback10g.mail.yandex.net (mxback10g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:171]) by forward101j.mail.yandex.net (Yandex) with ESMTP id 638101243241 for ; Fri, 8 Sep 2017 17:00:23 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback10g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id A8OIoLaJra-0NamURvZ; Fri, 08 Sep 2017 17:00:23 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0MKePN9G; Fri, 08 Sep 2017 17:00:22 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:08 +0300 Message-Id: <1504879212-3689-8-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 7/11] linux-gen: packet: implement static references X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Implemented static references with a reference counter. Counter is zero when not used (reference API not used). Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ .../linux-generic/include/odp_buffer_internal.h | 21 +++-- .../linux-generic/include/odp_packet_internal.h | 5 +- platform/linux-generic/odp_packet.c | 92 +++++++++++++++++++--- platform/linux-generic/odp_pool.c | 2 + 4 files changed, 102 insertions(+), 18 deletions(-) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 0c873d2d2..cd067a08b 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -62,11 +62,16 @@ struct odp_buffer_hdr_t { /* Last header of the segment list */ void *last_seg; - /* Initial buffer data pointer and length */ + /* Initial buffer data pointer */ uint8_t *base_data; - uint8_t *buf_end; - /* --- 40 bytes --- */ + /* Reference count */ + odp_atomic_u32_t ref_cnt; + + /* Event type. Maybe different than pool type (crypto compl event) */ + int8_t event_type; + + /* --- 37 bytes --- */ /* Segments */ seg_entry_t seg[CONFIG_PACKET_MAX_SEGS]; @@ -93,14 +98,17 @@ struct odp_buffer_hdr_t { /* Pool pointer */ void *pool_ptr; + /* Initial buffer tail pointer */ + uint8_t *buf_end; + /* User area pointer */ void *uarea_addr; /* User area size */ uint32_t uarea_size; - /* Event type. Maybe different than pool type (crypto compl event) */ - int8_t event_type; + /* Max data size */ + uint32_t size; /* ipc mapped process can not walk over pointers, * offset has to be used */ @@ -110,9 +118,6 @@ struct odp_buffer_hdr_t { * inlining */ odp_pool_t pool_hdl; - /* Max data size */ - uint32_t size; - /* Data or next header */ uint8_t data[0]; } ODP_ALIGNED_CACHE; diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index d8e5766ce..d8d4584a7 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -116,11 +116,13 @@ typedef struct { packet_parser_t p; + uint32_t frame_len; + odp_pktio_t input; - uint32_t frame_len; uint32_t headroom; uint32_t tailroom; + uint32_t shared_len; /* * Members below are not initialized by packet_init() @@ -217,6 +219,7 @@ static inline void packet_init(odp_packet_hdr_t *pkt_hdr, uint32_t len) * segment occupied by the allocated length. */ pkt_hdr->frame_len = len; + pkt_hdr->shared_len = 0; pkt_hdr->headroom = CONFIG_PACKET_HEADROOM; pkt_hdr->tailroom = CONFIG_PACKET_MAX_SEG_LEN - seg_len + CONFIG_PACKET_TAILROOM; diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index 94fdffab3..cde77bb5f 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -491,6 +491,36 @@ static inline void copy_buf_hdr(odp_packet_hdr_t *pkt_hdr, int first, int num, } } +static inline void packet_free_multi(odp_buffer_hdr_t *hdr[], int num) +{ + int i; + uint32_t ref_cnt; + int num_ref = 0; + + for (i = 0; i < num; i++) { + /* Zero when reference API has not been used */ + ref_cnt = odp_atomic_load_u32(&hdr[i]->ref_cnt); + + if (odp_unlikely(ref_cnt)) { + ref_cnt = odp_atomic_fetch_dec_u32(&hdr[i]->ref_cnt); + + if (ref_cnt > 1) { + num_ref++; + continue; + } + } + + /* Skip references and pack to be freed headers to array head */ + if (odp_unlikely(num_ref)) + hdr[i - num_ref] = hdr[i]; + } + + num -= num_ref; + + if (odp_likely(num)) + buffer_free_multi(hdr, num); +} + static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) { seg_entry_t *seg; @@ -503,7 +533,7 @@ static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) buf_hdr[i] = seg->hdr; } - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); } static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, @@ -560,7 +590,7 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, pkt_hdr = new_hdr; - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); } else { /* Free last 'num' bufs. * First, find the last remaining header. */ @@ -575,7 +605,7 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, buf_hdr[i] = seg->hdr; } - buffer_free_multi(buf_hdr, num); + packet_free_multi(buf_hdr, num); /* Head segment remains, no need to copy or update majority * of the metadata. */ @@ -699,7 +729,7 @@ void odp_packet_free(odp_packet_t pkt) int num_seg = pkt_hdr->buf_hdr.segcount; if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1)) - buffer_free_multi((odp_buffer_hdr_t **)&hdl, 1); + packet_free_multi((odp_buffer_hdr_t **)&hdl, 1); else free_all_segments(pkt_hdr, num_seg); } @@ -707,7 +737,7 @@ void odp_packet_free(odp_packet_t pkt) void odp_packet_free_multi(const odp_packet_t pkt[], int num) { if (CONFIG_PACKET_MAX_SEGS == 1) { - buffer_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); + packet_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); } else { odp_buffer_hdr_t *buf_hdr[num * CONFIG_PACKET_MAX_SEGS]; int i; @@ -728,7 +758,7 @@ void odp_packet_free_multi(const odp_packet_t pkt[], int num) bufs += num_seg - 1; } - buffer_free_multi(buf_hdr, bufs); + packet_free_multi(buf_hdr, bufs); } } @@ -1937,9 +1967,37 @@ uint64_t odp_packet_seg_to_u64(odp_packet_seg_t hdl) return _odp_pri(hdl); } +static inline void buffer_ref_inc(odp_buffer_hdr_t *buf_hdr) +{ + /* First count increment after alloc */ + if (odp_atomic_load_u32(&buf_hdr->ref_cnt) == 0) + odp_atomic_store_u32(&buf_hdr->ref_cnt, 2); + else + odp_atomic_inc_u32(&buf_hdr->ref_cnt); +} + +static inline void packet_ref_inc(odp_packet_hdr_t *pkt_hdr) +{ + seg_entry_t *seg; + int i; + int seg_count = pkt_hdr->buf_hdr.segcount; + odp_packet_hdr_t *hdr = pkt_hdr; + uint8_t idx = 0; + + for (i = 0; i < seg_count; i++) { + seg = seg_entry_next(&hdr, &idx); + buffer_ref_inc(seg->hdr); + } +} + odp_packet_t odp_packet_ref_static(odp_packet_t pkt) { - return odp_packet_copy(pkt, odp_packet_pool(pkt)); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + + packet_ref_inc(pkt_hdr); + pkt_hdr->shared_len = packet_len(pkt_hdr); + + return pkt; } odp_packet_t odp_packet_ref(odp_packet_t pkt, uint32_t offset) @@ -2001,14 +2059,30 @@ odp_packet_t odp_packet_ref_pkt(odp_packet_t pkt, uint32_t offset, int odp_packet_has_ref(odp_packet_t pkt) { - (void)pkt; + odp_buffer_hdr_t *buf_hdr; + seg_entry_t *seg; + int i; + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + int seg_count = pkt_hdr->buf_hdr.segcount; + odp_packet_hdr_t *hdr = pkt_hdr; + uint8_t idx = 0; + + for (i = 0; i < seg_count; i++) { + seg = seg_entry_next(&hdr, &idx); + buf_hdr = seg->hdr; + + if (odp_atomic_load_u32(&buf_hdr->ref_cnt) > 1) + return 1; + } return 0; } uint32_t odp_packet_unshared_len(odp_packet_t pkt) { - return odp_packet_len(pkt); + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + + return packet_len(pkt_hdr) - pkt_hdr->shared_len; } /* Include non-inlined versions of API functions */ diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 47a39f5b5..2f65cb20c 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -277,6 +277,8 @@ static void init_buffers(pool_t *pool) buf_hdr->seg[0].data = &data[offset]; buf_hdr->seg[0].len = pool->data_size; + odp_atomic_init_u32(&buf_hdr->ref_cnt, 0); + /* Store base values for fast init */ buf_hdr->base_data = buf_hdr->seg[0].data; buf_hdr->buf_end = &data[offset + pool->data_size + From patchwork Fri Sep 8 14:00:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112091 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp1678166ybm; Fri, 8 Sep 2017 07:13:44 -0700 (PDT) X-Google-Smtp-Source: AOwi7QDutFbacV+xlijhoCpx7irB9WsMfUyMLJDd7t0Gw9aycsui4VXAWaxJr7pCaq4dr24OBdiy X-Received: by 10.55.97.208 with SMTP id v199mr3846741qkb.165.1504880024482; Fri, 08 Sep 2017 07:13:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504880024; cv=none; d=google.com; s=arc-20160816; b=zyhJmcottXbgd1u/8oGPWLeuOcWAZMA8SK/zsRDMv1wtQDzc8SZbynfxf41Jy0MVP3 DezKJLFmUq+EYCd2EKCYnFyaQ+9V6BVa4ORNybtqtguRcSQzZdw3XP0BvlWwyqNLOydx 6vQivftwakb7sBgRLPUnBWJlSFbUJ2KO2W2aE13ud/YIktuXcOZ94lrh0Q8Orm50HCRm qp0slUyebD6biTh2skVjPspdhcuKDoSEoHX5DdqkdKSDnH4EjBpf2/cwdtcT9NGm7MPp Dh6l8ZtTPVwgQqft6BSyaM8iThqcY+kmyF5Mfo/piY4iiMOuuo+X0g7BJfo7FKzZ4TTO xB8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=G/J6uGwyNjc1bBT8JBShvH7Yc+/yHPLNhp+xufoiGfU=; b=iOQFixSyBh17AIT0s/39BTmwH1fUr7dMMvJYkMLM9W+xl8QjnR4NlkW/tpFpQXhN+G 3iB3gQpaGq/I9o+tJuOhBIp2S7W/VHf02nXE7QDVVpJtlAb4+0yj4j7eTNP656LQ+fJr R0lLNfecMVuXKh1cFYawVe+IsW7F9CNCPHV3pgGI9ezmI+iYH4Dl9ZE6eOMhbq5/wVQM 6cbFMdyiGjfAkbHLA3Nol7xRSPrblm30pk/04W8hqqECRFD8AjfuqBYexqhOxIO1s3mM HQKLELVcqc9gKJLLnOq/kXMFQ1uu7h4MZUKPFgLQ2clBpSDl6fSkFGvJ2xc28eVTRlqu +zrA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id j127si2229088qkd.99.2017.09.08.07.13.44; Fri, 08 Sep 2017 07:13:44 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 1F18460639; Fri, 8 Sep 2017 14:13:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 2D97F644C6; Fri, 8 Sep 2017 14:01:29 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 6E2B1644B0; Fri, 8 Sep 2017 14:00:39 +0000 (UTC) Received: from forward103o.mail.yandex.net (forward103o.mail.yandex.net [37.140.190.177]) by lists.linaro.org (Postfix) with ESMTPS id 4204262C5D for ; Fri, 8 Sep 2017 14:00:28 +0000 (UTC) Received: from mxback1o.mail.yandex.net (mxback1o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::1b]) by forward103o.mail.yandex.net (Yandex) with ESMTP id CE8495883611 for ; Fri, 8 Sep 2017 17:00:23 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback1o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id sZbXk5bH8N-0NomMdh0; Fri, 08 Sep 2017 17:00:23 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0NK4fwJ3; Fri, 08 Sep 2017 17:00:23 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:09 +0300 Message-Id: <1504879212-3689-9-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 8/11] linux-gen: packet: enhance segment prints X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Print segment addresses and reference counts in addition to segment lengths. Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ platform/linux-generic/odp_packet.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index cde77bb5f..9c50ca2bb 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -1536,11 +1536,13 @@ int _odp_packet_cmp_data(odp_packet_t pkt, uint32_t offset, * ******************************************************** * */ - void odp_packet_print(odp_packet_t pkt) { odp_packet_seg_t seg; - int max_len = 512; + seg_entry_t *seg_entry; + odp_packet_hdr_t *seg_hdr; + uint8_t idx; + int max_len = 1024; char str[max_len]; int len = 0; int n = max_len - 1; @@ -1576,12 +1578,22 @@ void odp_packet_print(odp_packet_t pkt) len += snprintf(&str[len], n - len, " num_segs %i\n", odp_packet_num_segs(pkt)); + seg_hdr = hdr; + idx = 0; seg = odp_packet_first_seg(pkt); while (seg != ODP_PACKET_SEG_INVALID) { + odp_buffer_hdr_t *buf_hdr; + + seg_entry = seg_entry_next(&seg_hdr, &idx); + buf_hdr = seg_entry->hdr; + len += snprintf(&str[len], n - len, - " seg_len %" PRIu32 "\n", - odp_packet_seg_data_len(pkt, seg)); + " seg_len %-4" PRIu32 " seg_data %p ", + odp_packet_seg_data_len(pkt, seg), + odp_packet_seg_data(pkt, seg)); + len += snprintf(&str[len], n - len, "ref_cnt %u\n", + odp_atomic_load_u32(&buf_hdr->ref_cnt)); seg = odp_packet_next_seg(pkt, seg); } From patchwork Fri Sep 8 14:00:10 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112086 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp1669459ybm; Fri, 8 Sep 2017 07:07:52 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBtbHGS72/BlULsKIvkNWdfHxYNfW3JlaAsy3OmZUFUL093MplZMm4FSGSbmc4+/ic8FU75 X-Received: by 10.233.237.7 with SMTP id c7mr3884829qkg.69.1504879672258; Fri, 08 Sep 2017 07:07:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879672; cv=none; d=google.com; s=arc-20160816; b=Ku5jC9mOL3tSr8YIC94vbCroOTULGN3UY4kr85Eg7v1Aly7qWmiIXBwyV3uMZa56BK LHeVlMcWIPFmDJ1H3bFy+7U8K9acKUPCWuvz0FL5+JHvk//cwvXRdegLQs/TwfEU5084 RW7ZqTrR5RLE8YEb78W4L10RxL5Iox9t/ssOxeKGxGQXl+1lM7R8vS2aUaG+IMjt6yum MHjXD5SuaAoglglajfm++vNGuNFCFv7XldBJd6iAvlOEs+lpB+3fGJPuRiVzwek5WI8e uHqzWtSNPlgzMY3C2FSV9VwUhiXADoP5BwPloKtrSC/hWOpJnJPUbcoxegpEXxmTa91e babg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=uakf0ZCJE3YObGMmnB5EEz+efJ7F3CrIUgOruvsnJuI=; b=gE/FqQsWMxv1XoMaQ2txaDwvOxNzz4XXodKsfbZYZ8LAZDNkXf6EA1/sfJmURUaXPj j1S+r9QKtHzPDO2A9Jx6c5KqwkUXz1GBct2Yx5qvcYfLl5eUJwEpBHoC2FWcQkrlAqTU PM2XDmeeDgqcL0l3JRyG9A+6m8f3A3dZSLgytla6BV76qqMQyAg0IoQLjAs8HbVvvKMV rbV9kcO3Q6OzKxRGjSoHotHyNKArWpeeNaFrs5vMx4jOyoD+0n6hpTVkBNltoLSBxu7C Fvn4mAb4weJiT0p1iagP53ZlXyQRRiC+1kr/A0/bMKeBst1q8Aq4h5p5LOBvNW4mUXzg PAXA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id q14si2079789qtg.327.2017.09.08.07.07.51; Fri, 08 Sep 2017 07:07:52 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id D0D7A62D0E; Fri, 8 Sep 2017 14:07:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id A8F2C644DD; Fri, 8 Sep 2017 14:01:00 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id D66AF62DF3; Fri, 8 Sep 2017 14:00:30 +0000 (UTC) Received: from forward105o.mail.yandex.net (forward105o.mail.yandex.net [37.140.190.183]) by lists.linaro.org (Postfix) with ESMTPS id 6609F60AF7 for ; Fri, 8 Sep 2017 14:00:25 +0000 (UTC) Received: from mxback18j.mail.yandex.net (mxback18j.mail.yandex.net [IPv6:2a02:6b8:0:1619::94]) by forward105o.mail.yandex.net (Yandex) with ESMTP id 486F4444411D for ; Fri, 8 Sep 2017 17:00:24 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback18j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id giCqXIUUSQ-0Oj8V7Ts; Fri, 08 Sep 2017 17:00:24 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0NKS0xAB; Fri, 08 Sep 2017 17:00:23 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:10 +0300 Message-Id: <1504879212-3689-10-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 9/11] linux-gen: packet: compress packet offset table X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Use uint16_t (instead of size_t) for offsets and thus decrease size of the offset table into one fourth. Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ .../include/odp/api/plat/packet_types.h | 28 +++++++++++----------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/platform/linux-generic/include/odp/api/plat/packet_types.h b/platform/linux-generic/include/odp/api/plat/packet_types.h index 41468becf..96379ebf9 100644 --- a/platform/linux-generic/include/odp/api/plat/packet_types.h +++ b/platform/linux-generic/include/odp/api/plat/packet_types.h @@ -74,33 +74,33 @@ typedef enum { /** @internal Packet header field offsets for inline functions */ typedef struct _odp_packet_inline_offset_t { /** @internal field offset */ - size_t data; + uint16_t data; /** @internal field offset */ - size_t seg_len; + uint16_t seg_len; /** @internal field offset */ - size_t frame_len; + uint16_t frame_len; /** @internal field offset */ - size_t headroom; + uint16_t headroom; /** @internal field offset */ - size_t tailroom; + uint16_t tailroom; /** @internal field offset */ - size_t pool; + uint16_t pool; /** @internal field offset */ - size_t input; + uint16_t input; /** @internal field offset */ - size_t segcount; + uint16_t segcount; /** @internal field offset */ - size_t user_ptr; + uint16_t user_ptr; /** @internal field offset */ - size_t user_area; + uint16_t user_area; /** @internal field offset */ - size_t user_area_size; + uint16_t user_area_size; /** @internal field offset */ - size_t flow_hash; + uint16_t flow_hash; /** @internal field offset */ - size_t timestamp; + uint16_t timestamp; /** @internal field offset */ - size_t input_flags; + uint16_t input_flags; } _odp_packet_inline_offset_t; From patchwork Fri Sep 8 14:00:11 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112088 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp1673879ybm; Fri, 8 Sep 2017 07:10:43 -0700 (PDT) X-Google-Smtp-Source: AOwi7QAAYzd6o1ICswFD5U25WLVgkD6lAFwf+HukvFa9LqzzHiV19rNxjyBaBfmDFuEPRz/Azkt6 X-Received: by 10.55.146.198 with SMTP id u189mr3762222qkd.317.1504879843739; Fri, 08 Sep 2017 07:10:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879843; cv=none; d=google.com; s=arc-20160816; b=vP9OHtIgGHsBgsBL26Eng0Ej46VMm8LtBXhmv8Z4Hli7fR9Jp1z3d4cbCgzcM3C+Zf NSKRC6VAvevn+i0KNC0vuHTEhYO0Ukhs8QvYWi9SttYfk3+wiVRC9ZrClPKbjAwGixs0 5OXrYlBnPKepWHAeiftwQ6DVQcVg8Mdtb3dZTYXUFCQPg1RoFQzsHpB0cYVMAqI+Q7bC 0Mr+Qv0gZ3wwEEOsLuSMgsH8ylOc3G6Zvm+1JafSbw5lpFsou9nGjAX0UA/QW5TrdwNa 8v8y6lNEDuDCA6Rqf5KQo5ySgKcNL0+4q/A5/ZwZZddKwaYcThbZGnq5pY9wvHStNsR3 9GSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=ybkDeANm/cpk16NNsIVADcVFyywoiFM6N12Pg6GIxwg=; b=m1HZ2CPt/2maJCLq3aNj4g0VDN48iEYxxQ00MS4VranRNoBp7MnfMHloqhCwHaGhDR t8E+jtHowC0vwu6P4+JWQbBJ0qhw2QCoU7Qvn8D/UqVT8cg2FUYAdmcDPDtSe9/5Sq7l +RzJ4viuI9txAKnlWTVvRA6aWW41PT7/tZEVQkEk3KyaE/nlJwriZa9b/ACAa1Xg1beX a3oDMwIDmrJha0eFOnGffPF/gpuovAV9udXEZT8kf89LoR2vX1jO9uV4gcpDmGRCm8yk k/AeSItkio2zkJQ2Khv5MwYcCRyIF7SHtVjemakhX+++LuuwxXkzoGQb9vBy33ULjjE2 aMhA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id g68si2131700qkc.270.2017.09.08.07.10.42; Fri, 08 Sep 2017 07:10:43 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id BB0AC644E9; Fri, 8 Sep 2017 14:10:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 3A888608E6; Fri, 8 Sep 2017 14:01:15 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 0EA05644A0; Fri, 8 Sep 2017 14:00:35 +0000 (UTC) Received: from forward105o.mail.yandex.net (forward105o.mail.yandex.net [37.140.190.183]) by lists.linaro.org (Postfix) with ESMTPS id EEEF560C3E for ; Fri, 8 Sep 2017 14:00:25 +0000 (UTC) Received: from mxback11g.mail.yandex.net (mxback11g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:90]) by forward105o.mail.yandex.net (Yandex) with ESMTP id DB3164444C51 for ; Fri, 8 Sep 2017 17:00:24 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback11g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id QW0HiK3mZy-0O0G15LD; Fri, 08 Sep 2017 17:00:24 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0OKSkY48; Fri, 08 Sep 2017 17:00:24 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:11 +0300 Message-Id: <1504879212-3689-11-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 10/11] linux-gen: packet: implement dynamic references X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Use link headers to implement dynamic references. These are segment headers which first segment (seg[0]) does not point to itself but to a referenced segment. Link headers enable long chains of references (new dynamic references from previous references). Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ platform/linux-generic/odp_packet.c | 314 +++++++++++++++++++++++++++--------- 1 file changed, 242 insertions(+), 72 deletions(-) diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index 9c50ca2bb..0279bc373 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -127,6 +127,43 @@ static inline seg_entry_t *seg_entry_next(odp_packet_hdr_t **cur_hdr, return &hdr->buf_hdr.seg[idx]; } +static inline void seg_entry_find_offset(odp_packet_hdr_t **p_hdr, + uint8_t *p_idx, + uint32_t *seg_offset, + uint32_t *seg_idx, + uint32_t offset) +{ + int i; + odp_packet_hdr_t *hdr, *cur_hdr; + uint8_t idx, cur_idx; + seg_entry_t *seg = NULL; + uint32_t seg_start = 0, seg_end = 0; + int seg_count; + + hdr = *p_hdr; + cur_hdr = hdr; + idx = 0; + cur_idx = 0; + seg_count = hdr->buf_hdr.segcount; + + for (i = 0; i < seg_count; i++) { + cur_hdr = hdr; + cur_idx = idx; + seg = seg_entry_next(&hdr, &idx); + seg_end += seg->len; + + if (odp_likely(offset < seg_end)) + break; + + seg_start = seg_end; + } + + *p_hdr = cur_hdr; + *p_idx = cur_idx; + *seg_offset = offset - seg_start; + *seg_idx = i; +} + static inline uint32_t packet_seg_len(odp_packet_hdr_t *pkt_hdr, uint32_t seg_idx) { @@ -456,6 +493,7 @@ static inline odp_packet_hdr_t *add_segments(odp_packet_hdr_t *pkt_hdr, new_hdr->frame_len = pkt_hdr->frame_len + len; new_hdr->headroom = pool->headroom + offset; new_hdr->tailroom = pkt_hdr->tailroom; + new_hdr->shared_len = pkt_hdr->shared_len; pkt_hdr = new_hdr; } else { @@ -475,20 +513,11 @@ static inline odp_packet_hdr_t *add_segments(odp_packet_hdr_t *pkt_hdr, return pkt_hdr; } -static inline void copy_buf_hdr(odp_packet_hdr_t *pkt_hdr, int first, int num, - odp_buffer_hdr_t *buf_hdr[]) +static inline int seg_is_link(void *hdr) { - seg_entry_t *seg; - int i; - uint8_t idx; - odp_packet_hdr_t *hdr = pkt_hdr; - - seg_entry_find_idx(&hdr, &idx, first); + odp_packet_hdr_t *pkt_hdr = hdr; - for (i = 0; i < num; i++) { - seg = seg_entry_next(&hdr, &idx); - buf_hdr[i] = seg->hdr; - } + return pkt_hdr != pkt_hdr->buf_hdr.seg[0].hdr; } static inline void packet_free_multi(odp_buffer_hdr_t *hdr[], int num) @@ -510,9 +539,14 @@ static inline void packet_free_multi(odp_buffer_hdr_t *hdr[], int num) } } + /* Reset link header back to normal header */ + if (odp_unlikely(seg_is_link(hdr[i]))) + hdr[i]->seg[0].hdr = hdr[i]; + /* Skip references and pack to be freed headers to array head */ if (odp_unlikely(num_ref)) hdr[i - num_ref] = hdr[i]; + } num -= num_ref; @@ -523,14 +557,36 @@ static inline void packet_free_multi(odp_buffer_hdr_t *hdr[], int num) static inline void free_all_segments(odp_packet_hdr_t *pkt_hdr, int num) { - seg_entry_t *seg; int i; - odp_buffer_hdr_t *buf_hdr[num]; - uint8_t idx = 0; + odp_buffer_hdr_t *buf_hdr[num + 1]; - for (i = 0; i < num; i++) { - seg = seg_entry_next(&pkt_hdr, &idx); - buf_hdr[i] = seg->hdr; + if (odp_likely(pkt_hdr->buf_hdr.num_seg == num)) { + for (i = 0; i < num; i++) + buf_hdr[i] = pkt_hdr->buf_hdr.seg[i].hdr; + + if (odp_unlikely(seg_is_link(pkt_hdr))) { + buf_hdr[num] = &pkt_hdr->buf_hdr; + num++; + } + } else { + seg_entry_t *seg; + odp_buffer_hdr_t *link_hdr[num]; + uint8_t idx = 0; + int links = 0; + + for (i = 0; i < num; i++) { + /* Free also link headers */ + if (odp_unlikely(idx == 0 && seg_is_link(pkt_hdr))) { + link_hdr[links] = &pkt_hdr->buf_hdr; + links++; + } + + seg = seg_entry_next(&pkt_hdr, &idx); + buf_hdr[i] = seg->hdr; + } + + if (odp_unlikely(links)) + packet_free_multi(link_hdr, links); } packet_free_multi(buf_hdr, num); @@ -548,14 +604,24 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, uint8_t idx; uint8_t num_seg; odp_buffer_hdr_t *buf_hdr[num]; + odp_buffer_hdr_t *link_hdr[num]; + odp_packet_hdr_t *tmp_hdr; + int links = 0; if (head) { odp_packet_hdr_t *new_hdr; idx = 0; for (i = 0; i < num; i++) { + tmp_hdr = hdr; seg = seg_entry_next(&hdr, &idx); buf_hdr[i] = seg->hdr; + + /* Free link headers, if those become empty */ + if (odp_unlikely(idx == 0 && seg_is_link(tmp_hdr))) { + link_hdr[links] = &tmp_hdr->buf_hdr; + links++; + } } /* The first remaining header is the new packet descriptor. @@ -583,13 +649,23 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, /* Tailroom not changed */ new_hdr->tailroom = pkt_hdr->tailroom; - new_hdr->headroom = seg_headroom(new_hdr, 0); - new_hdr->frame_len = pkt_hdr->frame_len - free_len; + + /* Link header does not have headroom */ + if (seg_is_link(new_hdr)) + new_hdr->headroom = 0; + else + new_hdr->headroom = seg_headroom(new_hdr, 0); + + new_hdr->frame_len = pkt_hdr->frame_len - free_len; + new_hdr->shared_len = pkt_hdr->shared_len; pull_head(new_hdr, pull_len); pkt_hdr = new_hdr; + if (odp_unlikely(links)) + packet_free_multi(link_hdr, links); + packet_free_multi(buf_hdr, num); } else { /* Free last 'num' bufs. @@ -601,10 +677,20 @@ static inline odp_packet_hdr_t *free_segments(odp_packet_hdr_t *pkt_hdr, seg_entry_next(&hdr, &idx); for (i = 0; i < num; i++) { + tmp_hdr = hdr; seg = seg_entry_next(&hdr, &idx); buf_hdr[i] = seg->hdr; + + /* Free link headers, if those become empty */ + if (odp_unlikely(idx == 0 && seg_is_link(tmp_hdr))) { + link_hdr[links] = &tmp_hdr->buf_hdr; + links++; + } } + if (odp_unlikely(links)) + packet_free_multi(link_hdr, links); + packet_free_multi(buf_hdr, num); /* Head segment remains, no need to copy or update majority @@ -724,42 +810,56 @@ int odp_packet_alloc_multi(odp_pool_t pool_hdl, uint32_t len, void odp_packet_free(odp_packet_t pkt) { odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - odp_buffer_t hdl = buffer_handle(pkt_hdr); - int num_seg = pkt_hdr->buf_hdr.segcount; - if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1)) - packet_free_multi((odp_buffer_hdr_t **)&hdl, 1); - else + if (odp_likely(CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1)) { + odp_buffer_hdr_t *buf_hdr[2]; + int num = 1; + + buf_hdr[0] = &pkt_hdr->buf_hdr; + + if (odp_unlikely(seg_is_link(pkt_hdr))) { + buf_hdr[1] = pkt_hdr->buf_hdr.seg[0].hdr; + num++; + } + + packet_free_multi(buf_hdr, num); + } else { free_all_segments(pkt_hdr, num_seg); + } } void odp_packet_free_multi(const odp_packet_t pkt[], int num) { - if (CONFIG_PACKET_MAX_SEGS == 1) { - packet_free_multi((odp_buffer_hdr_t **)(uintptr_t)pkt, num); - } else { - odp_buffer_hdr_t *buf_hdr[num * CONFIG_PACKET_MAX_SEGS]; - int i; - int bufs = 0; - - for (i = 0; i < num; i++) { - odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt[i]); - int num_seg = pkt_hdr->buf_hdr.segcount; - odp_buffer_hdr_t *hdr = &pkt_hdr->buf_hdr; + odp_buffer_hdr_t *buf_hdr[num]; + odp_buffer_hdr_t *buf_hdr2[num]; + int i; + int links = 0; + int num_freed = 0; - buf_hdr[bufs] = hdr; - bufs++; + for (i = 0; i < num; i++) { + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt[i]); + int num_seg = pkt_hdr->buf_hdr.segcount; - if (odp_likely(num_seg == 1)) - continue; + if (odp_unlikely(num_seg > 1)) { + free_all_segments(pkt_hdr, num_seg); + num_freed++; + continue; + } - copy_buf_hdr(pkt_hdr, 1, num_seg - 1, &buf_hdr[bufs]); - bufs += num_seg - 1; + if (odp_unlikely(seg_is_link(pkt_hdr))) { + buf_hdr2[links] = pkt_hdr->buf_hdr.seg[0].hdr; + links++; } - packet_free_multi(buf_hdr, bufs); + buf_hdr[i - num_freed] = &pkt_hdr->buf_hdr; } + + if (odp_unlikely(links)) + packet_free_multi(buf_hdr2, links); + + if (odp_likely(num - num_freed)) + packet_free_multi(buf_hdr, num - num_freed); } int odp_packet_reset(odp_packet_t pkt, uint32_t len) @@ -1584,7 +1684,9 @@ void odp_packet_print(odp_packet_t pkt) while (seg != ODP_PACKET_SEG_INVALID) { odp_buffer_hdr_t *buf_hdr; + odp_packet_hdr_t *tmp_hdr; + tmp_hdr = seg_hdr; seg_entry = seg_entry_next(&seg_hdr, &idx); buf_hdr = seg_entry->hdr; @@ -1592,8 +1694,16 @@ void odp_packet_print(odp_packet_t pkt) " seg_len %-4" PRIu32 " seg_data %p ", odp_packet_seg_data_len(pkt, seg), odp_packet_seg_data(pkt, seg)); - len += snprintf(&str[len], n - len, "ref_cnt %u\n", + len += snprintf(&str[len], n - len, "ref_cnt %u ", odp_atomic_load_u32(&buf_hdr->ref_cnt)); + if (seg_is_link(tmp_hdr)) { + uint32_t ref; + + ref = odp_atomic_load_u32(&tmp_hdr->buf_hdr.ref_cnt); + len += snprintf(&str[len], n - len, "L(%u)\n", ref); + } else { + len += snprintf(&str[len], n - len, "\n"); + } seg = odp_packet_next_seg(pkt, seg); } @@ -2014,59 +2124,116 @@ odp_packet_t odp_packet_ref_static(odp_packet_t pkt) odp_packet_t odp_packet_ref(odp_packet_t pkt, uint32_t offset) { - odp_packet_t new; - int ret; + odp_packet_t ref; + odp_packet_hdr_t *link_hdr; + odp_packet_hdr_t *next_hdr; + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + odp_packet_hdr_t *hdr = pkt_hdr; + seg_entry_t *seg; + uint32_t seg_idx = 0; + uint8_t idx = 0; + uint32_t seg_offset = 0; + int i, num_copy, segcount; + uint32_t len; - new = odp_packet_copy(pkt, odp_packet_pool(pkt)); + if (offset >= packet_len(pkt_hdr)) { + ODP_ERR("offset too large\n"); + return ODP_PACKET_INVALID; + } - if (new == ODP_PACKET_INVALID) { - ODP_ERR("copy failed\n"); + /* Allocate link segment */ + if (packet_alloc(pkt_hdr->buf_hdr.pool_ptr, 0, 1, 1, &ref) != 1) { + ODP_ERR("segment alloc failed\n"); return ODP_PACKET_INVALID; } - ret = odp_packet_trunc_head(&new, offset, NULL, NULL); + link_hdr = packet_hdr(ref); - if (ret < 0) { - ODP_ERR("trunk_head failed\n"); - odp_packet_free(new); - return ODP_PACKET_INVALID; + seg_entry_find_offset(&hdr, &idx, &seg_offset, &seg_idx, offset); + num_copy = hdr->buf_hdr.num_seg - idx; + segcount = pkt_hdr->buf_hdr.segcount; + + if (seg_is_link(hdr)) + buffer_ref_inc((odp_buffer_hdr_t *)hdr); + + seg = seg_entry_next(&hdr, &idx); + link_hdr->buf_hdr.num_seg = 1; + link_hdr->buf_hdr.seg[0].hdr = seg->hdr; + link_hdr->buf_hdr.seg[0].data = seg->data + seg_offset; + link_hdr->buf_hdr.seg[0].len = seg->len - seg_offset; + buffer_ref_inc(seg->hdr); + + for (i = 1; i < num_copy; i++) { + if (idx == 0 && seg_is_link(hdr)) + buffer_ref_inc((odp_buffer_hdr_t *)hdr); + + seg = seg_entry_next(&hdr, &idx); + + link_hdr->buf_hdr.num_seg++; + link_hdr->buf_hdr.seg[i].hdr = seg->hdr; + link_hdr->buf_hdr.seg[i].data = seg->data; + link_hdr->buf_hdr.seg[i].len = seg->len; + buffer_ref_inc(seg->hdr); + } + + next_hdr = hdr; + + /* Increment ref count for remaining segments */ + for (i = seg_idx + num_copy; i < segcount; i++) { + if (idx == 0 && seg_is_link(hdr)) + buffer_ref_inc((odp_buffer_hdr_t *)hdr); + + seg = seg_entry_next(&hdr, &idx); + buffer_ref_inc(seg->hdr); } - return new; + len = pkt_hdr->frame_len - offset; + link_hdr->buf_hdr.next_seg = next_hdr; + link_hdr->buf_hdr.last_seg = pkt_hdr->buf_hdr.last_seg; + link_hdr->buf_hdr.segcount = segcount - seg_idx; + link_hdr->frame_len = len; + link_hdr->tailroom = pkt_hdr->tailroom; + link_hdr->shared_len = len; + + /* Link header does not have headroom */ + link_hdr->headroom = 0; + + if (pkt_hdr->shared_len < len) + pkt_hdr->shared_len = len; + + return ref; + } odp_packet_t odp_packet_ref_pkt(odp_packet_t pkt, uint32_t offset, odp_packet_t hdr) { - odp_packet_t new; + odp_packet_t ref; int ret; + odp_packet_hdr_t *new_hdr; + odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); + uint32_t len = packet_len(pkt_hdr); - new = odp_packet_copy(pkt, odp_packet_pool(pkt)); + ref = odp_packet_ref(pkt, offset); - if (new == ODP_PACKET_INVALID) { - ODP_ERR("copy failed\n"); + if (ref == ODP_PACKET_INVALID) { + ODP_ERR("reference create failed\n"); return ODP_PACKET_INVALID; } - if (offset) { - ret = odp_packet_trunc_head(&new, offset, NULL, NULL); - - if (ret < 0) { - ODP_ERR("trunk_head failed\n"); - odp_packet_free(new); - return ODP_PACKET_INVALID; - } - } - - ret = odp_packet_concat(&hdr, new); + ret = odp_packet_concat(&hdr, ref); if (ret < 0) { ODP_ERR("concat failed\n"); - odp_packet_free(new); + odp_packet_free(ref); return ODP_PACKET_INVALID; } + new_hdr = packet_hdr(hdr); + new_hdr->shared_len = len - offset; + return hdr; + } int odp_packet_has_ref(odp_packet_t pkt) @@ -2094,7 +2261,10 @@ uint32_t odp_packet_unshared_len(odp_packet_t pkt) { odp_packet_hdr_t *pkt_hdr = packet_hdr(pkt); - return packet_len(pkt_hdr) - pkt_hdr->shared_len; + if (odp_packet_has_ref(pkt)) + return packet_len(pkt_hdr) - pkt_hdr->shared_len; + + return packet_len(pkt_hdr); } /* Include non-inlined versions of API functions */ From patchwork Fri Sep 8 14:00:12 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 112090 Delivered-To: patch@linaro.org Received: by 10.37.128.210 with SMTP id c18csp1677065ybm; Fri, 8 Sep 2017 07:12:53 -0700 (PDT) X-Google-Smtp-Source: AOwi7QD0m3nIxcEfqr5TYm+ikk+AS4zeODKA3omFaU6IPgGBCFh+nJFExHTFYNh8alEK+KwrCMUR X-Received: by 10.55.97.203 with SMTP id v194mr3665458qkb.305.1504879973258; Fri, 08 Sep 2017 07:12:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504879973; cv=none; d=google.com; s=arc-20160816; b=UpCSG1tVUCqWjwg96qgACy+xGiaKVu36eWpaoYPaeGditiQRY+EBevJfChsupR8ti8 qfzE9jin7zZmzn6N0ShJk7YRwQ5BldJYZsKroeLgzmn+5t2T6jSVibk9gKuHnC3wWKKi bglamZnPlMrP4wb/5+YaXBDDiEbnnIaDBCA06/3ngtfAMjOmYgWBVDoBQPJXwBi36pJf 7vMR1SRP+YyjK33Rp5JWHA2+HYNhS1UPPsG4r41b9cdlXq+UZUhidkSDCVPWzmCa60Pz IxnmLmBSuy31DU2JJbweX9CnJVwP8OMpijckVATgRLZeYdngcUnVRN9XmF44bNVE6/fV PAzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=YDXWRxlW99Pd6vN0mmkNos+UWCfTfo89GVyQjf20BgA=; b=m2bDtWCXYaA/iRhkoaSuV3bBAVvL1ocd8Kl24LsiPWby6wFGFip9ZiNTZKC7J3743N ZBLRxCjO/sW46QnRnNxo0AKG3ChvT/WcD2+q5/cen18PqX+8a/himb9ndzXOC/BVF3CJ kJu09Jhs/628kxtbPp37PK2DM4Dg/HB85i/eLEvx1k0C4AKTasW6GNGYCWREqESCAd5/ HZyTJLRNQTN3i7V7VQnslEWas54dOkxnHQvLAQtDaA8/L9Koe1bysGCZTQeBgrLKTAI8 iZ1D834oViQFasCaakpTDQhGaEt6eyO0wgOhjNXwNBMW0AXZRhxYHvFYlSh2DZCYn6Po 5ZLA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id i6si2075758qkc.424.2017.09.08.07.12.52; Fri, 08 Sep 2017 07:12:53 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id E0FB660C2D; Fri, 8 Sep 2017 14:12:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, RCVD_IN_SORBS_SPAM, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id D395560A55; Fri, 8 Sep 2017 14:01:26 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 1E6FB644B0; Fri, 8 Sep 2017 14:00:38 +0000 (UTC) Received: from forward105p.mail.yandex.net (forward105p.mail.yandex.net [77.88.28.108]) by lists.linaro.org (Postfix) with ESMTPS id 0AD6560C4C for ; Fri, 8 Sep 2017 14:00:27 +0000 (UTC) Received: from mxback8o.mail.yandex.net (mxback8o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::22]) by forward105p.mail.yandex.net (Yandex) with ESMTP id 74F53408505D for ; Fri, 8 Sep 2017 17:00:25 +0300 (MSK) Received: from smtp2p.mail.yandex.net (smtp2p.mail.yandex.net [2a02:6b8:0:1472:2741:0:8b6:7]) by mxback8o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id xApzrBoIRU-0PKicVJd; Fri, 08 Sep 2017 17:00:25 +0300 Received: by smtp2p.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IwYlQdvFwh-0OKqY48t; Fri, 08 Sep 2017 17:00:25 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Fri, 8 Sep 2017 17:00:12 +0300 Message-Id: <1504879212-3689-12-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> References: <1504879212-3689-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 170 Subject: [lng-odp] [PATCH v1 11/11] linux-gen: packet: optimize header layout X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Petri Savolainen Pack most often used data into the first and the last cache line of buffer header, and the first cache line of packet header. Reduce offsets and head-/tailroom fields to 16 bits, since those are in maximum as large as a segment, usually much less. Pack header fields into correct alignments (when possible), so that holes are avoided. These changes reduce packet header size by one cache line. Signed-off-by: Petri Savolainen --- /** Email created from pull request 170 (psavol:master-packet-ref-rework) ** https://github.com/Linaro/odp/pull/170 ** Patch: https://github.com/Linaro/odp/pull/170.patch ** Base sha: 360c1d34b52a2356619b9290811862b9de41de00 ** Merge commit sha: e19cce86ab00ccd959d2c3c82c0786ec6da19ba7 **/ include/odp/arch/default/api/abi/packet.h | 2 +- .../include/odp/api/plat/packet_inlines.h | 4 ++-- .../linux-generic/include/odp/api/plat/packet_types.h | 2 +- platform/linux-generic/include/odp_buffer_internal.h | 16 ++++++++-------- platform/linux-generic/include/odp_packet_internal.h | 19 ++++++++++++------- platform/linux-generic/odp_pool.c | 3 +++ 6 files changed, 27 insertions(+), 19 deletions(-) diff --git a/include/odp/arch/default/api/abi/packet.h b/include/odp/arch/default/api/abi/packet.h index 4aac75b9e..15cf081bf 100644 --- a/include/odp/arch/default/api/abi/packet.h +++ b/include/odp/arch/default/api/abi/packet.h @@ -28,7 +28,7 @@ typedef _odp_abi_packet_seg_t *odp_packet_seg_t; #define ODP_PACKET_INVALID ((odp_packet_t)0xffffffff) #define ODP_PACKET_SEG_INVALID ((odp_packet_seg_t)0xffffffff) -#define ODP_PACKET_OFFSET_INVALID (0x0fffffff) +#define ODP_PACKET_OFFSET_INVALID 0xffff typedef enum { ODP_PACKET_GREEN = 0, diff --git a/platform/linux-generic/include/odp/api/plat/packet_inlines.h b/platform/linux-generic/include/odp/api/plat/packet_inlines.h index d0cf13901..6874d1496 100644 --- a/platform/linux-generic/include/odp/api/plat/packet_inlines.h +++ b/platform/linux-generic/include/odp/api/plat/packet_inlines.h @@ -56,13 +56,13 @@ static inline uint32_t _odp_packet_len(odp_packet_t pkt) /** @internal Inline function @param pkt @return */ static inline uint32_t _odp_packet_headroom(odp_packet_t pkt) { - return _odp_pkt_get(pkt, uint32_t, headroom); + return _odp_pkt_get(pkt, uint16_t, headroom); } /** @internal Inline function @param pkt @return */ static inline uint32_t _odp_packet_tailroom(odp_packet_t pkt) { - return _odp_pkt_get(pkt, uint32_t, tailroom); + return _odp_pkt_get(pkt, uint16_t, tailroom); } /** @internal Inline function @param pkt @return */ diff --git a/platform/linux-generic/include/odp/api/plat/packet_types.h b/platform/linux-generic/include/odp/api/plat/packet_types.h index 96379ebf9..68c66312b 100644 --- a/platform/linux-generic/include/odp/api/plat/packet_types.h +++ b/platform/linux-generic/include/odp/api/plat/packet_types.h @@ -36,7 +36,7 @@ typedef ODP_HANDLE_T(odp_packet_t); #define ODP_PACKET_INVALID _odp_cast_scalar(odp_packet_t, 0) -#define ODP_PACKET_OFFSET_INVALID (0x0fffffff) +#define ODP_PACKET_OFFSET_INVALID 0xffff typedef uint8_t odp_packet_seg_t; diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index cd067a08b..5d40303b3 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -65,13 +65,10 @@ struct odp_buffer_hdr_t { /* Initial buffer data pointer */ uint8_t *base_data; - /* Reference count */ - odp_atomic_u32_t ref_cnt; - - /* Event type. Maybe different than pool type (crypto compl event) */ - int8_t event_type; + /* Pool pointer */ + void *pool_ptr; - /* --- 37 bytes --- */ + /* --- 40 bytes --- */ /* Segments */ seg_entry_t seg[CONFIG_PACKET_MAX_SEGS]; @@ -95,8 +92,11 @@ struct odp_buffer_hdr_t { const void *buf_cctx; /* const alias for ctx */ }; - /* Pool pointer */ - void *pool_ptr; + /* Reference count */ + odp_atomic_u32_t ref_cnt; + + /* Event type. Maybe different than pool type (crypto compl event) */ + int8_t event_type; /* Initial buffer tail pointer */ uint8_t *buf_end; diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h index d8d4584a7..d76d7bf08 100644 --- a/platform/linux-generic/include/odp_packet_internal.h +++ b/platform/linux-generic/include/odp_packet_internal.h @@ -88,9 +88,14 @@ typedef struct { error_flags_t error_flags; output_flags_t output_flags; - uint32_t l2_offset; /**< offset to L2 hdr, e.g. Eth */ - uint32_t l3_offset; /**< offset to L3 hdr, e.g. IPv4, IPv6 */ - uint32_t l4_offset; /**< offset to L4 hdr (TCP, UDP, SCTP, also ICMP) */ + /* offset to L2 hdr, e.g. Eth */ + uint16_t l2_offset; + + /* offset to L3 hdr, e.g. IPv4, IPv6 */ + uint16_t l3_offset; + + /* offset to L4 hdr (TCP, UDP, SCTP, also ICMP) */ + uint16_t l4_offset; } packet_parser_t; /* Packet extra data length */ @@ -116,14 +121,14 @@ typedef struct { packet_parser_t p; - uint32_t frame_len; - odp_pktio_t input; - uint32_t headroom; - uint32_t tailroom; + uint32_t frame_len; uint32_t shared_len; + uint16_t headroom; + uint16_t tailroom; + /* * Members below are not initialized by packet_init() */ diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 2f65cb20c..2a0a35e40 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -41,6 +41,9 @@ ODP_STATIC_ASSERT(CONFIG_POOL_CACHE_SIZE > (2 * CACHE_BURST), ODP_STATIC_ASSERT(CONFIG_PACKET_SEG_LEN_MIN >= 256, "ODP Segment size must be a minimum of 256 bytes"); +ODP_STATIC_ASSERT(CONFIG_PACKET_SEG_SIZE < 0xffff, + "Segment size must be less than 64k (16 bit offsets)"); + /* Thread local variables */ typedef struct pool_local_t { pool_cache_t *cache[ODP_CONFIG_POOLS];