From patchwork Wed Jul 20 13:42:46 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elo, Matias \(Nokia - FI/Espoo\)" X-Patchwork-Id: 72426 Delivered-To: patch@linaro.org Received: by 10.140.29.52 with SMTP id a49csp663641qga; Wed, 20 Jul 2016 06:45:59 -0700 (PDT) X-Received: by 10.37.65.204 with SMTP id o195mr12871656yba.87.1469022359312; Wed, 20 Jul 2016 06:45:59 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id g10si1752027qkb.4.2016.07.20.06.45.58; Wed, 20 Jul 2016 06:45:59 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE dis=NONE) header.from=nokia.com Received: by lists.linaro.org (Postfix, from userid 109) id DB29F688F3; Wed, 20 Jul 2016 13:45:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAD_ENC_HEADER,BAYES_00, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 6835B6869A; Wed, 20 Jul 2016 13:44:36 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id C69AA68437; Wed, 20 Jul 2016 13:44:24 +0000 (UTC) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0107.outbound.protection.outlook.com [104.47.2.107]) by lists.linaro.org (Postfix) with ESMTPS id E503D6869E for ; Wed, 20 Jul 2016 13:43:23 +0000 (UTC) Received: from HE1PR0701CA0037.eurprd07.prod.outlook.com (10.165.214.175) by DB4PR07MB0493.eurprd07.prod.outlook.com (10.242.195.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.544.10; Wed, 20 Jul 2016 13:43:20 +0000 Received: from DB3FFO11FD027.protection.gbl (2a01:111:f400:7e04::177) by HE1PR0701CA0037.outlook.office365.com (2603:10a6:3:5::47) with Microsoft SMTP Server (TLS) id 15.1.539.14 via Frontend Transport; Wed, 20 Jul 2016 13:43:19 +0000 Received-SPF: Pass (protection.outlook.com: domain of nokia.com designates 131.228.2.240 as permitted sender) receiver=protection.outlook.com; client-ip=131.228.2.240; helo=fihe3nok0734.emea.nsn-net.net; Received: from fihe3nok0734.emea.nsn-net.net (131.228.2.240) by DB3FFO11FD027.mail.protection.outlook.com (10.47.217.58) with Microsoft SMTP Server (TLS) id 15.1.534.7 via Frontend Transport; Wed, 20 Jul 2016 13:43:20 +0000 Received: from fihe3nok0734.emea.nsn-net.net (localhost [127.0.0.1]) by fihe3nok0734.emea.nsn-net.net (8.14.9/8.14.5) with ESMTP id u6KDgmGp030644 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 20 Jul 2016 16:42:48 +0300 Received: from 10.144.19.15 ([10.144.104.111]) by fihe3nok0734.emea.nsn-net.net (8.14.9/8.14.5) with ESMTP id u6KDgmZJ030641 (version=TLSv1/SSLv3 cipher=AES128-SHA256 bits=128 verify=NOT); Wed, 20 Jul 2016 16:42:48 +0300 X-HPESVCS-Source-Ip: 10.144.104.111 From: Matias Elo To: Date: Wed, 20 Jul 2016 16:42:46 +0300 Message-ID: <1469022168-28941-1-git-send-email-matias.elo@nokia.com> X-Mailer: git-send-email 2.7.4 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:131.228.2.240; IPV:NLI; CTRY:FI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(7916002)(2980300002)(438002)(189002)(199003)(19580395003)(356003)(19580405001)(8676002)(81156014)(33646002)(8936002)(11100500001)(4326007)(575784001)(50226002)(81166006)(92566002)(16796002)(68736007)(50986999)(5003940100001)(450100001)(189998001)(107886002)(110136002)(97736004)(229853001)(2351001)(50466002)(106466001)(6806005)(87936001)(2906002)(47776003)(48376002)(77096005)(7846002)(36756003)(4001430100002)(305945005)(586003)(42882005); DIR:OUT; SFP:1102; SCL:1; SRVR:DB4PR07MB0493; H:fihe3nok0734.emea.nsn-net.net; FPR:; SPF:Pass; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; DB3FFO11FD027; 1:7qat3AGmJx/HZ0d9q6AnwX4aUk/y9Fr7MJ1BW/XycBl5VV4/yZI09xttvJJapQznlbimRl9nkPhzx55aPwxtk5lShsabqRK7reX+UOGjpPQ0WmF9zs6oYehjyzA4bIIBDQCDJ6pgHfhdgiMaPP3NbsJ3N3wlP8+vZNmIF7BonaPulNPYdhlPT4XB+WVDPFYS9+ATfNr5m6PG8MbMZNO3F+623ZP1k/IkO8BKXEs48j6pgqJ4c19H85K7WWixYT/xnQh6sDO/9ORLaghIPovtmMg9ETFlzLNDbxWhLUfcL2TV6p4os2rkGIJ9rg/QwCcRX8QmGE3r3oh/fOXyDN2eGIdeg/gO3wFf/rAEX89iYFgTj1g0M2BzKaTVcREBx6aGEQRwY2dw5Kf/UIYgOv4hBfv7SD7wN2gjHqzGmC5ITJicYGNL7IccqzlVtS2xI9xzUSowPr2DcykmtBhawGoQFC02vTJtKzYipl2F1uFvsfww99MdH/QFcEbsKQ73ADfCnpZ0KGhpocIbnJbFyKg7bQ== MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: f48b2b4b-d713-4db1-0f43-08d3b0a3cfe7 X-Microsoft-Exchange-Diagnostics: 1; DB4PR07MB0493; 2:Id/bXYQGWMajANQqVsQgnucPtYAtZxeMHkfmjZY1+BeY1qQSYbha46FQabRTwDzayX/tm1/9mG8zUOAPDhYs0QEpqNk6i2mauoi/jV+YaBsY4LqrxHA5UKm947pVJgOc1ZF6rOFzZ1pk7/1/1lRnoBRTy087OIntfAX7rRqz776qKFYpf6vQHSUg/usSMJ3V; 3:Kd0VZVPLgOefivvV7Np4SuK3Sb/YekI/oGsVSP3IXOsSsXYkU3TY/CBnhpTLZHlpU7kCV8CfZLrnyLEfGKxqMEY5UklNCwBERvpmU8Dp0Mn3Gp7qY+ayD8bOYrnszDEUgRAEMYOsnoBgfJHSeK7BK8lsumeTphzzRv+P9Paqbf/rW+VkQC9wTTHLzuisJjbJmQlVuNt4i04zirzeNxwTf5nl+tTrj1vFTb0I5q2Q28bC3oOuTUrUL/3nPfNnYEzmhMpz5XdyjDUKYjVsbvsZCw==; 25:fG475kqKCOdD7qkW0MMcjGY8GsklwQWjx2Q5QkRz6qIrbzLj4j4/wutxcftoXCyKd8pDnQlVnX3DJhlAianOkwo0g3fjlQsviuLHf+aZHYD6pY0rRFUFLwtsy1msmTOmjs93+ctU/tTZbLj2KNn/n+14f/tVik3XJz741HWuhME9pTP2yK7YLhIFkMvESLNA1D292BbztFbPsGGR1QcacipzsnwhgoNevzKk7BRdVibNNusyGvKAkcnSK1W5yR7i8ccbPc08wAT7fBBp6v0Tlh4MWWebkZ71eglklbG/xSMD9T/ft8MsiFUwV11GBCATsexmDJFNcpwlUyhjUyIHt8yRjg9BuH6fOYyZ6sQWSKFg5fFcDz9VF35Iaf7z//NOiw4Cf/gOiBKUpt1ziTpQ3tBlzKWOTI+4N8z72F3wysM= X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501002); SRVR:DB4PR07MB0493; X-Microsoft-Exchange-Diagnostics: 1; DB4PR07MB0493; 31:UiG2oXfM5WgrRQE2Xi6Yx3AQSDpnE7SyTKmX0hOR3UndV0txKxjXUGmeLphbuP6V5CJeiDSqI7bTQoUt16QUhdds0fWt6KjI2q6u1/3FfuyDRVsOyb/uSOAkciMuI35CPNU1cz5RG8L5OrHOf323DViGLItfcwsygVljMn4cs0gf0YUOlYd4PpsLu6G4V2trnwF1M4MvYBvHyzdYbTl/FQ==; 20:QScabfGRbymcHD2+WsemeM7oYWMlS0a5eMNr5IvREznN7LELafJdGvgDAgzgJacAom27Fl9YTQZr99vk4YxIUtUrULBHq/45EIghEdFn838sCiLmULya8SV6LFLK8q5AP+S8MUYSruwpBJ/lgAtSg8G4NSPi1yrwEv6lOvebBkIYh+qCnZTJSKQX2Pw4i2ziJrFhhC46WLEg9lrDEb83Fx4mz9AUlhmXiaqINtrbc4svTsAfNSDIjMuUJO7DAepD0ptx1j2Oaf+nwUaAX2lfGmp8hOmS5TG8kLJkC+4i9RKdkQF2vwczau1tmpM25z6w8LWxw3Lkf6TVa5HgcVIE4YVje1RjFPvDlI7GvLLySQT/saRnA3f/KK7ppQmEndqPbQTZ76gyf0J9hwpUqEGXLzmKPkMQ0GG+8aXz8aF0G3h8fRuoodGdrNYKeslHqH6U3QsecbfSlNJbELkizLQrIpFQenNaLW7Iu6/BbKfeu9aR4IfQlmFAa0L87aO2ZPdeuMdNPkqWdclYh5rs850QwBEidb35tcvucP29mp3NQfWvmH2YiVAnCGJfYmzUUbQighrq6GILnvYK1csweyIVQ0VJxJEA/1QROJznidWNLe8= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(82608151540597); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(13018025)(8121501046)(5005006)(13016025)(10201501046)(3002001)(6055026); SRVR:DB4PR07MB0493; BCL:0; PCL:0; RULEID:; SRVR:DB4PR07MB0493; X-Microsoft-Exchange-Diagnostics: 1; DB4PR07MB0493; 4:0x+UwaApdQWhLAZSRZR17GEJAmEAqpokD1W/9FN8eqRr/VH1kKi+HYsjcj/SD+zu40YXpHZHfCFbQcE8nu/81xoXjIiDQSvdfYcqVCbJbUtp1u1X3a+F8WDR8BB3MqL7yZK+Af/q+QPkOUS72Sjv/RA0CBLfXuv1SlypM8cZjwtGyK/P8wdGIlXorfUwb82FaeXZ9ZtKJbtORSTvAk1on3Wh33ERrC1LpM+AYw12AytK5ep6Kew67eej5OiqKxrTXDqjqGJkWoOGxI6hD/u62rEYUMIUzqtmjL3w8ZjCT1aFzv3Gvvua4ZCSaSGRon9tJTAutBoIQh5MY4hVKjQ+0sqOQ2QCWL9fu49RUQDhnkNYCPgkV9aKQLMUhD8xcNd42DqwaaXFqVCVTAdHRSIYrM5DsWEi9Q7MBeJqJ5lCHMKAkpyeGifzdB+lg5HlrSFtRpodlfW/rmv6HIIxUJ0MDQ== X-Forefront-PRVS: 000947967F X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DB4PR07MB0493; 23:Sg7D4auBLYi95EsiOlcwJ1KLq0PBCQYn9TZt6jmVv?= =?us-ascii?Q?ImPI5XtxIG7cj0w3gOebij+mGPspxTlIZxd2+vjYXqd/JHgYEgPH3oJ3bguc?= =?us-ascii?Q?CQW1ynO6H0RvYxRpXKPv4s4dGz/6FWxZsgIVQsvovWx9iwdMauJhijQ1wIqG?= =?us-ascii?Q?oVquDizDal73iODN4ciuCjhQD/73wqZEPSL7S97Q76adzTAV8Tchbd5u4k2W?= =?us-ascii?Q?WnNLtvf57N5mZrNUVwI2gWma+CMjvnY/1QQs4cJcGedycXnoLs7y4JJYegLM?= =?us-ascii?Q?lmCc1ypts1ohtQoajieoEFhAE/tn2y/8IEwNeD4hLpjCADbwC6NsyQ23SLPm?= =?us-ascii?Q?5sIwUkWbRNCWR38eTGCQcuTlJUZ6di18NN/xlu3QDkIWORR1ynPx0bCIPw7m?= =?us-ascii?Q?QUxMpUMwhoN/lklyyd9pfXmH2SgtiYh4k1z00GFhTocblgrkU9h7nuPdpLDI?= =?us-ascii?Q?VsfiUqn5BmCLozezqxjpev3v5ezXRYQcimPQaE9iZiVS3/wJHLisDgPp3KYb?= =?us-ascii?Q?QwbFfUSnnCKPU47sag9p85rgyZppv55eC0oprkjqJEqlfjRDnMX6y2+TVEMw?= =?us-ascii?Q?aBk/uixzgTU4JvZM9oJp+4W1I8DYdwaSlVan54JP5IdHJS9hQkYi2gX6WMyA?= =?us-ascii?Q?CbbYaefUBGhzCuVZCyZmEZ0M9CfuMCb/XfSH+S2JAWyKKKvm1EDzlLYExYLO?= =?us-ascii?Q?vuEti+zkqU3mB8IeiaitmwHZpesWUV/E2LKKVFv24WFF+09KQOMAA0fBM0iF?= =?us-ascii?Q?W/R4lqJ+V+YddHpq5ushBjDYuTEzBnoaGEK9HmOvf9SvS4Sv46wVgXROqE1o?= =?us-ascii?Q?kfi8AO9LHfGukydduvkriqSg/Bdgrofd19SC2jlm4JzvDLh/q2e2GIEE9BUW?= =?us-ascii?Q?HAu5obQawWlzKCpkQ09E3vwa9rI/d0vTz7XaDfDYcvvs3exj8KrU6o7WS27l?= =?us-ascii?Q?dlINd8nqkez2FnSS06icieBLSHZISs08vcdKEnmxIKBl3vqRgVYfzgMQEfZV?= =?us-ascii?Q?R7v61Vg2f9PJeRj/V3Tgnaabz8JkwPXEwJcL8kI8Eze+s+0t0U8sK22ekOQj?= =?us-ascii?Q?ZYyPJkwF7bBd265gWBWwmqQGHjEUleoER/vFOTjgWXx4r6jTQVdy94LItbiU?= =?us-ascii?Q?SHLOxhy5O8=3D?= X-Microsoft-Exchange-Diagnostics: 1; DB4PR07MB0493; 6:YgS7GJJjBD2azvgsGqoN2xfOt1TD+zHYwxXcsvByCktxywTkFwpKwuAxANW6Px+tZYsvjhEbcxCh/zn37AfwHfmgl2Qmx+OtQ2xeeuZcEjYRixamjzhnooXWs+FYB1xMQGBmAZfS67RGl03o5QEEANBfx6GzsE9TjsCDVcvvCUl3cxv84FwtQCwSFezdKcspG4WoOmtx4rMJWvAHF8ekQj5iDbPQuXrNuvKbq5Po6x8aQV5W3L5wiugld3MdhfR6ZhpAGCiH2Fd8eozIZ5I1e/vott57cpLBz3seY17JTfI53KLsjWc4azDXEbb0Bx/bSk5r/0IjFDtozzQn90d/Uw==; 5:57tsLSsBLrhv3noRYIWSCsvH1bKgZDZIDxUoffN3epDGkys5obZpB+K0otjgBejkRRJAspVv59AaAttm7TaWCqMKhrAkxqOzElTT11x3Ef1fboQ5AJpJ52XJNbXI/gver06IUvQQHFOw1YGXYW3HDQ==; 24:eCc5eQrCkTj9YUJLyYta1e2ONpLe1LAJAPOXOHnqJJ5ulMvDDhWaMsrnFOxPOrMTOT9hHxlbThjtkyWyjlIXwLEY1lvj8/ByEnEU3FWWDpA=; 7:FnVrTJiMvIU6YIWjvsxQQoH42qm6DrRCCtBB5kK9YAh2DQs9P/9sGve+iMCcGMwdJGhSXrbO+A20d2e0NfQYwcMl5iE+lnLUoAXZ0B/ErP3GzP41sN4FdE4qB6rkeGz2PsTizN8PxGZzTQbwmPWhYPtDNNdiWgrsIsWjUOL4O3QkbXoIwHsWOHzM3KCEl8P/PgJP2KfWft1rx8EWDvv3393dQ3lo0cg2i1ywcjmFCYMJ+gHllvPQg41hJNiB5W6U SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: nokia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jul 2016 13:43:20.1528 (UTC) X-MS-Exchange-CrossTenant-Id: 5d471751-9675-428d-917b-70f44f9630b0 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5d471751-9675-428d-917b-70f44f9630b0; Ip=[131.228.2.240]; Helo=[fihe3nok0734.emea.nsn-net.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB4PR07MB0493 X-Topics: patch Cc: Petri Savolainen Subject: [lng-odp] [PATCH v2 1/3] linux-gen: pool: optimize thread local buffer cache X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" Optimize local buffer cache performance which is critical to many use cases - including packet IO. Main parts of the optimization are: * Local cache implemented as an array of buf_hdr pointers, instead of a linked list (which causes a lot of cache misses) * Alloc and free N buffers per operation All above steps are needed to demonstrate the performance upgrade. Some related pool functions (get_buf(), ret_buf(), etc) were moved from pool header to c source file, since those were actual local to the c source file. Also some unused pool variables are removed also. Signed-off-by: Petri Savolainen Signed-off-by: Matias Elo --- V2: - Split pktio modifications into a separate patch (Bill) - Improve performance by adding separate functions for single buffer alloc/free operations .../linux-generic/include/odp_buffer_inlines.h | 26 +- .../linux-generic/include/odp_buffer_internal.h | 5 +- platform/linux-generic/include/odp_internal.h | 2 - platform/linux-generic/include/odp_pool_internal.h | 143 +------ platform/linux-generic/odp_buffer.c | 3 - platform/linux-generic/odp_packet.c | 5 +- platform/linux-generic/odp_pool.c | 473 +++++++++++++++++---- 7 files changed, 426 insertions(+), 231 deletions(-) -- 2.7.4 diff --git a/platform/linux-generic/include/odp_buffer_inlines.h b/platform/linux-generic/include/odp_buffer_inlines.h index 3f4d9fd..2b1ab42 100644 --- a/platform/linux-generic/include/odp_buffer_inlines.h +++ b/platform/linux-generic/include/odp_buffer_inlines.h @@ -56,30 +56,12 @@ static inline odp_buffer_hdr_t *odp_buf_to_hdr(odp_buffer_t buf) (pool->pool_mdata_addr + (index * ODP_CACHE_LINE_SIZE)); } -static inline uint32_t odp_buffer_refcount(odp_buffer_hdr_t *buf) +static inline uint32_t pool_id_from_buf(odp_buffer_t buf) { - return odp_atomic_load_u32(&buf->ref_count); -} + odp_buffer_bits_t handle; -static inline uint32_t odp_buffer_incr_refcount(odp_buffer_hdr_t *buf, - uint32_t val) -{ - return odp_atomic_fetch_add_u32(&buf->ref_count, val) + val; -} - -static inline uint32_t odp_buffer_decr_refcount(odp_buffer_hdr_t *buf, - uint32_t val) -{ - uint32_t tmp; - - tmp = odp_atomic_fetch_sub_u32(&buf->ref_count, val); - - if (tmp < val) { - odp_atomic_fetch_add_u32(&buf->ref_count, val - tmp); - return 0; - } else { - return tmp - val; - } + handle.handle = buf; + return handle.pool_id; } static inline odp_buffer_hdr_t *validate_buf(odp_buffer_t buf) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index f21364c..7b0ef8b 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -114,7 +114,6 @@ struct odp_buffer_hdr_t { union { uint32_t all; struct { - uint32_t zeroized:1; /* Zeroize buf data on free */ uint32_t hdrdata:1; /* Data is in buffer hdr */ uint32_t sustain:1; /* Sustain order */ }; @@ -123,7 +122,6 @@ struct odp_buffer_hdr_t { int8_t type; /* buffer type */ odp_event_type_t event_type; /* for reuse as event */ uint32_t size; /* max data size */ - odp_atomic_u32_t ref_count; /* reference count */ odp_pool_t pool_hdl; /* buffer pool handle */ union { uint64_t buf_u64; /* user u64 */ @@ -174,6 +172,9 @@ typedef struct { odp_buffer_t buffer_alloc(odp_pool_t pool, size_t size); int buffer_alloc_multi(odp_pool_t pool_hdl, size_t size, odp_buffer_t buf[], int num); +void buffer_free(uint32_t pool_id, const odp_buffer_t buf); +void buffer_free_multi(uint32_t pool_id, + const odp_buffer_t buf[], int num_free); int seg_alloc_head(odp_buffer_hdr_t *buf_hdr, int segcount); void seg_free_head(odp_buffer_hdr_t *buf_hdr, int segcount); int seg_alloc_tail(odp_buffer_hdr_t *buf_hdr, int segcount); diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index d12f850..8bad450 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -119,8 +119,6 @@ int odp_tm_term_global(void); int _odp_int_name_tbl_init_global(void); int _odp_int_name_tbl_term_global(void); -void _odp_flush_caches(void); - int cpuinfo_parser(FILE *file, system_info_t *sysinfo); uint64_t odp_cpu_hz_current(int id); diff --git a/platform/linux-generic/include/odp_pool_internal.h b/platform/linux-generic/include/odp_pool_internal.h index 3317bd0..d6717ff 100644 --- a/platform/linux-generic/include/odp_pool_internal.h +++ b/platform/linux-generic/include/odp_pool_internal.h @@ -51,15 +51,25 @@ typedef struct _odp_buffer_pool_init_t { void *buf_init_arg; /**< Argument to be passed to buf_init() */ } _odp_buffer_pool_init_t; /**< Type of buffer initialization struct */ +#define POOL_MAX_LOCAL_CHUNKS 4 +#define POOL_CHUNK_SIZE 32 +#define POOL_MAX_LOCAL_BUFS (POOL_MAX_LOCAL_CHUNKS * POOL_CHUNK_SIZE) + +struct local_cache_s { + uint64_t bufallocs; /* Local buffer alloc count */ + uint64_t buffrees; /* Local buffer free count */ + + uint32_t num_buf; + odp_buffer_hdr_t *buf[POOL_MAX_LOCAL_BUFS]; +}; + /* Local cache for buffer alloc/free acceleration */ typedef struct local_cache_t { union { - struct { - odp_buffer_hdr_t *buf_freelist; /* The local cache */ - uint64_t bufallocs; /* Local buffer alloc count */ - uint64_t buffrees; /* Local buffer free count */ - }; - uint8_t pad[ODP_CACHE_LINE_SIZE_ROUNDUP(sizeof(uint64_t))]; + struct local_cache_s s; + + uint8_t pad[ODP_CACHE_LINE_SIZE_ROUNDUP( + sizeof(struct local_cache_s))]; }; } local_cache_t; @@ -214,127 +224,6 @@ static inline void ret_blk(struct pool_entry_s *pool, void *block) odp_atomic_inc_u64(&pool->poolstats.blkfrees); } -static inline odp_buffer_hdr_t *get_buf(struct pool_entry_s *pool) -{ - odp_buffer_hdr_t *myhead; - POOL_LOCK(&pool->buf_lock); - - myhead = pool->buf_freelist; - - if (odp_unlikely(myhead == NULL)) { - POOL_UNLOCK(&pool->buf_lock); - odp_atomic_inc_u64(&pool->poolstats.bufempty); - } else { - pool->buf_freelist = myhead->next; - POOL_UNLOCK(&pool->buf_lock); - uint64_t bufcount = - odp_atomic_fetch_sub_u32(&pool->bufcount, 1) - 1; - - /* Check for low watermark condition */ - if (bufcount == pool->buf_low_wm && !pool->buf_low_wm_assert) { - pool->buf_low_wm_assert = 1; - odp_atomic_inc_u64(&pool->poolstats.buf_low_wm_count); - } - - odp_atomic_inc_u64(&pool->poolstats.bufallocs); - } - - return (void *)myhead; -} - -static inline void ret_buf(struct pool_entry_s *pool, odp_buffer_hdr_t *buf) -{ - if (!buf->flags.hdrdata && buf->type != ODP_EVENT_BUFFER) { - while (buf->segcount > 0) { - if (buffer_is_secure(buf) || pool_is_secure(pool)) - memset(buf->addr[buf->segcount - 1], - 0, buf->segsize); - ret_blk(pool, buf->addr[--buf->segcount]); - } - buf->size = 0; - } - - buf->allocator = ODP_FREEBUF; /* Mark buffer free */ - POOL_LOCK(&pool->buf_lock); - buf->next = pool->buf_freelist; - pool->buf_freelist = buf; - POOL_UNLOCK(&pool->buf_lock); - - uint64_t bufcount = odp_atomic_fetch_add_u32(&pool->bufcount, 1) + 1; - - /* Check if low watermark condition should be deasserted */ - if (bufcount == pool->buf_high_wm && pool->buf_low_wm_assert) { - pool->buf_low_wm_assert = 0; - odp_atomic_inc_u64(&pool->poolstats.buf_high_wm_count); - } - - odp_atomic_inc_u64(&pool->poolstats.buffrees); -} - -static inline void *get_local_buf(local_cache_t *buf_cache, - struct pool_entry_s *pool, - size_t totsize) -{ - odp_buffer_hdr_t *buf = buf_cache->buf_freelist; - - if (odp_likely(buf != NULL)) { - buf_cache->buf_freelist = buf->next; - - if (odp_unlikely(buf->size < totsize)) { - intmax_t needed = totsize - buf->size; - - do { - void *blk = get_blk(pool); - if (odp_unlikely(blk == NULL)) { - ret_buf(pool, buf); - buf_cache->buffrees--; - return NULL; - } - buf->addr[buf->segcount++] = blk; - needed -= pool->seg_size; - } while (needed > 0); - - buf->size = buf->segcount * pool->seg_size; - } - - buf_cache->bufallocs++; - } - - return buf; -} - -static inline void ret_local_buf(local_cache_t *buf_cache, - odp_buffer_hdr_t *buf) -{ - buf->allocator = ODP_FREEBUF; - buf->next = buf_cache->buf_freelist; - buf_cache->buf_freelist = buf; - - buf_cache->buffrees++; -} - -static inline void flush_cache(local_cache_t *buf_cache, - struct pool_entry_s *pool) -{ - odp_buffer_hdr_t *buf = buf_cache->buf_freelist; - uint32_t flush_count = 0; - - while (buf != NULL) { - odp_buffer_hdr_t *next = buf->next; - ret_buf(pool, buf); - buf = next; - flush_count++; - } - - odp_atomic_add_u64(&pool->poolstats.bufallocs, buf_cache->bufallocs); - odp_atomic_add_u64(&pool->poolstats.buffrees, - buf_cache->buffrees - flush_count); - - buf_cache->buf_freelist = NULL; - buf_cache->bufallocs = 0; - buf_cache->buffrees = 0; -} - static inline odp_pool_t pool_index_to_handle(uint32_t pool_id) { return _odp_cast_scalar(odp_pool_t, pool_id); diff --git a/platform/linux-generic/odp_buffer.c b/platform/linux-generic/odp_buffer.c index e7e4d58..ce2fdba 100644 --- a/platform/linux-generic/odp_buffer.c +++ b/platform/linux-generic/odp_buffer.c @@ -67,9 +67,6 @@ int odp_buffer_snprint(char *str, uint32_t n, odp_buffer_t buf) len += snprintf(&str[len], n-len, " size %" PRIu32 "\n", hdr->size); len += snprintf(&str[len], n-len, - " ref_count %" PRIu32 "\n", - odp_atomic_load_u32(&hdr->ref_count)); - len += snprintf(&str[len], n-len, " type %i\n", hdr->type); return len; diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index 0e319d2..474fa81 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -972,10 +972,7 @@ int _odp_packet_copy_md_to_packet(odp_packet_t srcpkt, odp_packet_t dstpkt) srchdr->buf_hdr.uarea_size ? dsthdr->buf_hdr.uarea_size : srchdr->buf_hdr.uarea_size); - odp_atomic_store_u32( - &dsthdr->buf_hdr.ref_count, - odp_atomic_load_u32( - &srchdr->buf_hdr.ref_count)); + copy_packet_parser_metadata(srchdr, dsthdr); /* Metadata copied, but return indication of whether the packet diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 419f03f..0a427ed 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -57,8 +57,15 @@ static const char SHM_DEFAULT_NAME[] = "odp_buffer_pools"; /* Pool entry pointers (for inlining) */ void *pool_entry_ptr[ODP_CONFIG_POOLS]; -/* Cache thread id locally for local cache performance */ -static __thread int local_id; +/* Thread local variables */ +typedef struct pool_local_t { + local_cache_t *cache[ODP_CONFIG_POOLS]; + int thr_id; +} pool_local_t; + +static __thread pool_local_t local; + +static void flush_cache(local_cache_t *buf_cache, struct pool_entry_s *pool); int odp_pool_init_global(void) { @@ -111,7 +118,19 @@ int odp_pool_init_global(void) int odp_pool_init_local(void) { - local_id = odp_thread_id(); + pool_entry_t *pool; + int i; + int thr_id = odp_thread_id(); + + memset(&local, 0, sizeof(pool_local_t)); + + for (i = 0; i < ODP_CONFIG_POOLS; i++) { + pool = get_pool_entry(i); + local.cache[i] = &pool->s.local_cache[thr_id]; + local.cache[i]->s.num_buf = 0; + } + + local.thr_id = thr_id; return 0; } @@ -144,7 +163,14 @@ int odp_pool_term_global(void) int odp_pool_term_local(void) { - _odp_flush_caches(); + int i; + + for (i = 0; i < ODP_CONFIG_POOLS; i++) { + pool_entry_t *pool = get_pool_entry(i); + + flush_cache(local.cache[i], &pool->s); + } + return 0; } @@ -179,10 +205,53 @@ int odp_pool_capability(odp_pool_capability_t *capa) return 0; } -/** +static inline odp_buffer_hdr_t *get_buf(struct pool_entry_s *pool) +{ + odp_buffer_hdr_t *myhead; + + POOL_LOCK(&pool->buf_lock); + + myhead = pool->buf_freelist; + + if (odp_unlikely(myhead == NULL)) { + POOL_UNLOCK(&pool->buf_lock); + odp_atomic_inc_u64(&pool->poolstats.bufempty); + } else { + pool->buf_freelist = myhead->next; + POOL_UNLOCK(&pool->buf_lock); + + odp_atomic_fetch_sub_u32(&pool->bufcount, 1); + odp_atomic_inc_u64(&pool->poolstats.bufallocs); + } + + return (void *)myhead; +} + +static inline void ret_buf(struct pool_entry_s *pool, odp_buffer_hdr_t *buf) +{ + if (!buf->flags.hdrdata && buf->type != ODP_EVENT_BUFFER) { + while (buf->segcount > 0) { + if (buffer_is_secure(buf) || pool_is_secure(pool)) + memset(buf->addr[buf->segcount - 1], + 0, buf->segsize); + ret_blk(pool, buf->addr[--buf->segcount]); + } + buf->size = 0; + } + + buf->allocator = ODP_FREEBUF; /* Mark buffer free */ + POOL_LOCK(&pool->buf_lock); + buf->next = pool->buf_freelist; + pool->buf_freelist = buf; + POOL_UNLOCK(&pool->buf_lock); + + odp_atomic_fetch_add_u32(&pool->bufcount, 1); + odp_atomic_inc_u64(&pool->poolstats.buffrees); +} + +/* * Pool creation */ - odp_pool_t _pool_create(const char *name, odp_pool_param_t *params, uint32_t shmflags) @@ -208,9 +277,6 @@ odp_pool_t _pool_create(const char *name, /* Restriction for v1.0: All non-packet buffers are unsegmented */ int unseg = 1; - /* Restriction for v1.0: No zeroization support */ - const int zeroized = 0; - uint32_t blk_size, buf_stride, buf_num, blk_num, seg_len = 0; uint32_t buf_align = params->type == ODP_POOL_BUFFER ? params->buf.align : 0; @@ -350,7 +416,6 @@ odp_pool_t _pool_create(const char *name, POOL_UNLOCK(&pool->s.lock); pool->s.flags.unsegmented = unseg; - pool->s.flags.zeroized = zeroized; pool->s.seg_size = unseg ? blk_size : seg_len; pool->s.blk_size = blk_size; @@ -383,9 +448,7 @@ odp_pool_t _pool_create(const char *name, /* Iniitalize buffer metadata */ tmp->allocator = ODP_FREEBUF; tmp->flags.all = 0; - tmp->flags.zeroized = zeroized; tmp->size = 0; - odp_atomic_init_u32(&tmp->ref_count, 0); tmp->type = params->type; tmp->event_type = params->type; tmp->pool_hdl = pool->s.pool_hdl; @@ -503,6 +566,41 @@ int odp_pool_info(odp_pool_t pool_hdl, odp_pool_info_t *info) return 0; } +static inline void get_local_cache_bufs(local_cache_t *buf_cache, uint32_t idx, + odp_buffer_hdr_t *buf_hdr[], + uint32_t num) +{ + uint32_t i; + + for (i = 0; i < num; i++) { + buf_hdr[i] = buf_cache->s.buf[idx + i]; + odp_prefetch(buf_hdr[i]); + odp_prefetch_store(buf_hdr[i]); + } +} + +static void flush_cache(local_cache_t *buf_cache, struct pool_entry_s *pool) +{ + uint32_t flush_count = 0; + uint32_t num; + + while ((num = buf_cache->s.num_buf)) { + odp_buffer_hdr_t *buf; + + buf = buf_cache->s.buf[num - 1]; + ret_buf(pool, buf); + flush_count++; + buf_cache->s.num_buf--; + } + + odp_atomic_add_u64(&pool->poolstats.bufallocs, buf_cache->s.bufallocs); + odp_atomic_add_u64(&pool->poolstats.buffrees, + buf_cache->s.buffrees - flush_count); + + buf_cache->s.bufallocs = 0; + buf_cache->s.buffrees = 0; +} + int odp_pool_destroy(odp_pool_t pool_hdl) { uint32_t pool_id = pool_handle_to_index(pool_hdl); @@ -621,71 +719,207 @@ void seg_free_tail(odp_buffer_hdr_t *buf_hdr, int segcount) buf_hdr->size = buf_hdr->segcount * pool->s.seg_size; } -odp_buffer_t buffer_alloc(odp_pool_t pool_hdl, size_t size) +static inline int get_local_bufs(local_cache_t *buf_cache, + odp_buffer_hdr_t *buf_hdr[], uint32_t max_num) +{ + uint32_t num_buf = buf_cache->s.num_buf; + uint32_t num = num_buf; + + if (odp_unlikely(num_buf == 0)) + return 0; + + if (odp_likely(max_num < num)) + num = max_num; + + get_local_cache_bufs(buf_cache, num_buf - num, buf_hdr, num); + buf_cache->s.num_buf -= num; + buf_cache->s.bufallocs += num; + + return num; +} + +static inline void ret_local_buf(local_cache_t *buf_cache, uint32_t idx, + odp_buffer_hdr_t *buf) +{ + buf_cache->s.buf[idx] = buf; + buf_cache->s.num_buf++; + buf_cache->s.buffrees++; +} + +static inline void ret_local_bufs(local_cache_t *buf_cache, uint32_t idx, + odp_buffer_hdr_t *buf[], int num_buf) +{ + int i; + + for (i = 0; i < num_buf; i++) + buf_cache->s.buf[idx + i] = buf[i]; + + buf_cache->s.num_buf += num_buf; + buf_cache->s.buffrees += num_buf; +} + +int buffer_alloc_multi(odp_pool_t pool_hdl, size_t size, + odp_buffer_t buf[], int max_num) { uint32_t pool_id = pool_handle_to_index(pool_hdl); pool_entry_t *pool = get_pool_entry(pool_id); uintmax_t totsize = pool->s.headroom + size + pool->s.tailroom; - odp_anybuf_t *buf; + odp_buffer_hdr_t *buf_tbl[max_num]; + odp_buffer_hdr_t *buf_hdr; + int num, i; + intmax_t needed; + void *blk; /* Reject oversized allocation requests */ if ((pool->s.flags.unsegmented && totsize > pool->s.seg_size) || (!pool->s.flags.unsegmented && totsize > pool->s.seg_size * ODP_BUFFER_MAX_SEG)) - return ODP_BUFFER_INVALID; + return 0; /* Try to satisfy request from the local cache */ - buf = (odp_anybuf_t *) - (void *)get_local_buf(&pool->s.local_cache[local_id], - &pool->s, totsize); + num = get_local_bufs(local.cache[pool_id], buf_tbl, max_num); /* If cache is empty, satisfy request from the pool */ - if (odp_unlikely(buf == NULL)) { - buf = (odp_anybuf_t *)(void *)get_buf(&pool->s); + if (odp_unlikely(num < max_num)) { + for (; num < max_num; num++) { + buf_hdr = get_buf(&pool->s); - if (odp_unlikely(buf == NULL)) + if (odp_unlikely(buf_hdr == NULL)) + goto pool_empty; + + /* Get blocks for this buffer, if pool uses + * application data */ + if (buf_hdr->size < totsize) { + uint32_t segcount; + + needed = totsize - buf_hdr->size; + do { + blk = get_blk(&pool->s); + if (odp_unlikely(blk == NULL)) { + ret_buf(&pool->s, buf_hdr); + goto pool_empty; + } + + segcount = buf_hdr->segcount++; + buf_hdr->addr[segcount] = blk; + needed -= pool->s.seg_size; + } while (needed > 0); + buf_hdr->size = buf_hdr->segcount * + pool->s.seg_size; + } + + buf_tbl[num] = buf_hdr; + } + } + +pool_empty: + for (i = 0; i < num; i++) { + buf_hdr = buf_tbl[i]; + + /* Mark buffer as allocated */ + buf_hdr->allocator = local.thr_id; + + /* By default, buffers are not associated with + * an ordered queue */ + buf_hdr->origin_qe = NULL; + + buf[i] = odp_hdr_to_buf(buf_hdr); + + /* Add more segments if buffer from local cache is too small */ + if (odp_unlikely(buf_hdr->size < totsize)) { + needed = totsize - buf_hdr->size; + do { + blk = get_blk(&pool->s); + if (odp_unlikely(blk == NULL)) { + int j; + + ret_buf(&pool->s, buf_hdr); + buf_hdr = NULL; + local.cache[pool_id]->s.buffrees--; + + /* move remaining bufs up one step + * and update loop counters */ + num--; + for (j = i; j < num; j++) + buf_tbl[j] = buf_tbl[j + 1]; + + i--; + break; + } + needed -= pool->s.seg_size; + buf_hdr->addr[buf_hdr->segcount++] = blk; + buf_hdr->size = buf_hdr->segcount * + pool->s.seg_size; + } while (needed > 0); + } + } + + return num; +} + +odp_buffer_t buffer_alloc(odp_pool_t pool_hdl, size_t size) +{ + uint32_t pool_id = pool_handle_to_index(pool_hdl); + pool_entry_t *pool = get_pool_entry(pool_id); + uintmax_t totsize = pool->s.headroom + size + pool->s.tailroom; + odp_buffer_hdr_t *buf_hdr; + intmax_t needed; + void *blk; + + /* Reject oversized allocation requests */ + if ((pool->s.flags.unsegmented && totsize > pool->s.seg_size) || + (!pool->s.flags.unsegmented && + totsize > pool->s.seg_size * ODP_BUFFER_MAX_SEG)) + return 0; + + /* Try to satisfy request from the local cache. If cache is empty, + * satisfy request from the pool */ + if (odp_unlikely(!get_local_bufs(local.cache[pool_id], &buf_hdr, 1))) { + buf_hdr = get_buf(&pool->s); + + if (odp_unlikely(buf_hdr == NULL)) return ODP_BUFFER_INVALID; /* Get blocks for this buffer, if pool uses application data */ - if (buf->buf.size < totsize) { - intmax_t needed = totsize - buf->buf.size; + if (buf_hdr->size < totsize) { + needed = totsize - buf_hdr->size; do { - uint8_t *blk = get_blk(&pool->s); - if (blk == NULL) { - ret_buf(&pool->s, &buf->buf); + blk = get_blk(&pool->s); + if (odp_unlikely(blk == NULL)) { + ret_buf(&pool->s, buf_hdr); return ODP_BUFFER_INVALID; } - buf->buf.addr[buf->buf.segcount++] = blk; + buf_hdr->addr[buf_hdr->segcount++] = blk; needed -= pool->s.seg_size; } while (needed > 0); - buf->buf.size = buf->buf.segcount * pool->s.seg_size; + buf_hdr->size = buf_hdr->segcount * pool->s.seg_size; } } - /* Mark buffer as allocated */ - buf->buf.allocator = local_id; + buf_hdr->allocator = local.thr_id; - /* By default, buffers inherit their pool's zeroization setting */ - buf->buf.flags.zeroized = pool->s.flags.zeroized; + /* By default, buffers are not associated with + * an ordered queue */ + buf_hdr->origin_qe = NULL; - /* By default, buffers are not associated with an ordered queue */ - buf->buf.origin_qe = NULL; - - return odp_hdr_to_buf(&buf->buf); -} - -int buffer_alloc_multi(odp_pool_t pool_hdl, size_t size, - odp_buffer_t buf[], int num) -{ - int count; - - for (count = 0; count < num; ++count) { - buf[count] = buffer_alloc(pool_hdl, size); - if (buf[count] == ODP_BUFFER_INVALID) - break; + /* Add more segments if buffer from local cache is too small */ + if (odp_unlikely(buf_hdr->size < totsize)) { + needed = totsize - buf_hdr->size; + do { + blk = get_blk(&pool->s); + if (odp_unlikely(blk == NULL)) { + ret_buf(&pool->s, buf_hdr); + buf_hdr = NULL; + local.cache[pool_id]->s.buffrees--; + return ODP_BUFFER_INVALID; + } + buf_hdr->addr[buf_hdr->segcount++] = blk; + needed -= pool->s.seg_size; + } while (needed > 0); + buf_hdr->size = buf_hdr->segcount * pool->s.seg_size; } - return count; + return odp_hdr_to_buf(buf_hdr); } odp_buffer_t odp_buffer_alloc(odp_pool_t pool_hdl) @@ -701,35 +935,132 @@ int odp_buffer_alloc_multi(odp_pool_t pool_hdl, odp_buffer_t buf[], int num) return buffer_alloc_multi(pool_hdl, buf_size, buf, num); } -void odp_buffer_free(odp_buffer_t buf) +static void multi_pool_free(odp_buffer_hdr_t *buf_hdr[], int num_buf) { - odp_buffer_hdr_t *buf_hdr = odp_buf_to_hdr(buf); - pool_entry_t *pool = odp_buf_to_pool(buf_hdr); + uint32_t pool_id, num; + local_cache_t *buf_cache; + pool_entry_t *pool; + int i, j, idx; + for (i = 0; i < num_buf; i++) { + pool_id = pool_handle_to_index(buf_hdr[i]->pool_hdl); + buf_cache = local.cache[pool_id]; + num = buf_cache->s.num_buf; + + if (num < POOL_MAX_LOCAL_BUFS) { + ret_local_buf(buf_cache, num, buf_hdr[i]); + continue; + } + + idx = POOL_MAX_LOCAL_BUFS - POOL_CHUNK_SIZE; + pool = get_pool_entry(pool_id); + + /* local cache full, return a chunk */ + for (j = 0; j < POOL_CHUNK_SIZE; j++) { + odp_buffer_hdr_t *tmp; + + tmp = buf_cache->s.buf[idx + i]; + ret_buf(&pool->s, tmp); + } + + num = POOL_MAX_LOCAL_BUFS - POOL_CHUNK_SIZE; + buf_cache->s.num_buf = num; + ret_local_buf(buf_cache, num, buf_hdr[i]); + } +} + +void buffer_free_multi(uint32_t pool_id, + const odp_buffer_t buf[], int num_free) +{ + local_cache_t *buf_cache = local.cache[pool_id]; + uint32_t num; + int i, idx; + pool_entry_t *pool; + odp_buffer_hdr_t *buf_hdr[num_free]; + int multi_pool = 0; + + for (i = 0; i < num_free; i++) { + uint32_t id; + + buf_hdr[i] = odp_buf_to_hdr(buf[i]); + ODP_ASSERT(buf_hdr[i]->allocator != ODP_FREEBUF); + buf_hdr[i]->allocator = ODP_FREEBUF; + id = pool_handle_to_index(buf_hdr[i]->pool_hdl); + multi_pool |= (pool_id != id); + } + + if (odp_unlikely(multi_pool)) { + multi_pool_free(buf_hdr, num_free); + return; + } + + num = buf_cache->s.num_buf; + + if (odp_likely((num + num_free) < POOL_MAX_LOCAL_BUFS)) { + ret_local_bufs(buf_cache, num, buf_hdr, num_free); + return; + } + + pool = get_pool_entry(pool_id); + + /* Return at least one chunk into the global pool */ + if (odp_unlikely(num_free > POOL_CHUNK_SIZE)) { + for (i = 0; i < num_free; i++) + ret_buf(&pool->s, buf_hdr[i]); + + return; + } + + idx = num - POOL_CHUNK_SIZE; + for (i = 0; i < POOL_CHUNK_SIZE; i++) + ret_buf(&pool->s, buf_cache->s.buf[idx + i]); + + num -= POOL_CHUNK_SIZE; + buf_cache->s.num_buf = num; + ret_local_bufs(buf_cache, num, buf_hdr, num_free); +} + +void buffer_free(uint32_t pool_id, const odp_buffer_t buf) +{ + local_cache_t *buf_cache = local.cache[pool_id]; + uint32_t num; + int i; + pool_entry_t *pool; + odp_buffer_hdr_t *buf_hdr; + + buf_hdr = odp_buf_to_hdr(buf); ODP_ASSERT(buf_hdr->allocator != ODP_FREEBUF); + buf_hdr->allocator = ODP_FREEBUF; - if (odp_unlikely(pool->s.buf_low_wm_assert || pool->s.blk_low_wm_assert)) - ret_buf(&pool->s, buf_hdr); - else - ret_local_buf(&pool->s.local_cache[local_id], buf_hdr); + num = buf_cache->s.num_buf; + + if (odp_likely((num + 1) < POOL_MAX_LOCAL_BUFS)) { + ret_local_bufs(buf_cache, num, &buf_hdr, 1); + return; + } + + pool = get_pool_entry(pool_id); + + num -= POOL_CHUNK_SIZE; + for (i = 0; i < POOL_CHUNK_SIZE; i++) + ret_buf(&pool->s, buf_cache->s.buf[num + i]); + + buf_cache->s.num_buf = num; + ret_local_bufs(buf_cache, num, &buf_hdr, 1); +} + +void odp_buffer_free(odp_buffer_t buf) +{ + uint32_t pool_id = pool_id_from_buf(buf); + + buffer_free(pool_id, buf); } void odp_buffer_free_multi(const odp_buffer_t buf[], int num) { - int i; + uint32_t pool_id = pool_id_from_buf(buf[0]); - for (i = 0; i < num; ++i) - odp_buffer_free(buf[i]); -} - -void _odp_flush_caches(void) -{ - int i; - - for (i = 0; i < ODP_CONFIG_POOLS; i++) { - pool_entry_t *pool = get_pool_entry(i); - flush_cache(&pool->s.local_cache[local_id], &pool->s); - } + buffer_free_multi(pool_id, buf, num); } void odp_pool_print(odp_pool_t pool_hdl) @@ -774,7 +1105,6 @@ void odp_pool_print(odp_pool_t pool_hdl) pool->s.quiesced ? "quiesced" : "active"); ODP_DBG(" pool opts %s, %s, %s\n", pool->s.flags.unsegmented ? "unsegmented" : "segmented", - pool->s.flags.zeroized ? "zeroized" : "non-zeroized", pool->s.flags.predefined ? "predefined" : "created"); ODP_DBG(" pool base %p\n", pool->s.pool_base_addr); ODP_DBG(" pool size %zu (%zu pages)\n", @@ -817,10 +1147,11 @@ void odp_pool_print(odp_pool_t pool_hdl) ODP_DBG(" blk low wm count %lu\n", blklowmct); } - odp_pool_t odp_buffer_pool(odp_buffer_t buf) { - return odp_buf_to_hdr(buf)->pool_hdl; + uint32_t pool_id = pool_id_from_buf(buf); + + return pool_index_to_handle(pool_id); } void odp_pool_param_init(odp_pool_param_t *params)