From patchwork Fri Jul 22 10:04:54 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Elo, Matias \(Nokia - FI/Espoo\)" X-Patchwork-Id: 72609 Delivered-To: patch@linaro.org Received: by 10.140.29.52 with SMTP id a49csp916677qga; Fri, 22 Jul 2016 03:18:05 -0700 (PDT) X-Received: by 10.55.169.212 with SMTP id s203mr3827831qke.39.1469182685077; Fri, 22 Jul 2016 03:18:05 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id j1si8211016qkf.317.2016.07.22.03.18.04; Fri, 22 Jul 2016 03:18:05 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE dis=NONE) header.from=nokia.com Received: by lists.linaro.org (Postfix, from userid 109) id ACB706815A; Fri, 22 Jul 2016 10:18:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAD_ENC_HEADER,BAYES_00, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id AB1C46679B; Fri, 22 Jul 2016 10:17:52 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 44F0861685; Fri, 22 Jul 2016 10:17:49 +0000 (UTC) Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10134.outbound.protection.outlook.com [40.107.1.134]) by lists.linaro.org (Postfix) with ESMTPS id A88DD6156B for ; Fri, 22 Jul 2016 10:17:46 +0000 (UTC) Received: from DB6PR0701CA0026.eurprd07.prod.outlook.com (10.168.7.164) by DB4PR07MB0495.eurprd07.prod.outlook.com (10.242.195.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.544.10; Fri, 22 Jul 2016 10:05:55 +0000 Received: from AM1FFO11FD031.protection.gbl (2a01:111:f400:7e00::112) by DB6PR0701CA0026.outlook.office365.com (2603:10a6:4:3::36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.544.10 via Frontend Transport; Fri, 22 Jul 2016 10:05:55 +0000 Received-SPF: Pass (protection.outlook.com: domain of nokia.com designates 131.228.2.241 as permitted sender) receiver=protection.outlook.com; client-ip=131.228.2.241; helo=fihe3nok0735.emea.nsn-net.net; Received: from fihe3nok0735.emea.nsn-net.net (131.228.2.241) by AM1FFO11FD031.mail.protection.outlook.com (10.174.64.220) with Microsoft SMTP Server (TLS) id 15.1.534.7 via Frontend Transport; Fri, 22 Jul 2016 10:05:55 +0000 Received: from fihe3nok0735.emea.nsn-net.net (localhost [127.0.0.1]) by fihe3nok0735.emea.nsn-net.net (8.14.9/8.14.5) with ESMTP id u6MA5W1O013409 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 22 Jul 2016 13:05:32 +0300 Received: from 10.144.19.15 ([10.144.104.111]) by fihe3nok0735.emea.nsn-net.net (8.14.9/8.14.5) with ESMTP id u6MA5MWS013147 (version=TLSv1/SSLv3 cipher=AES128-SHA256 bits=128 verify=NOT); Fri, 22 Jul 2016 13:05:32 +0300 X-HPESVCS-Source-Ip: 10.144.104.111 From: Matias Elo To: Date: Fri, 22 Jul 2016 13:04:54 +0300 Message-ID: <1469181897-19168-1-git-send-email-matias.elo@nokia.com> X-Mailer: git-send-email 2.7.4 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:131.228.2.241; IPV:NLI; CTRY:FI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(7916002)(2980300002)(438002)(199003)(189002)(2906002)(68736007)(50466002)(48376002)(356003)(586003)(305945005)(4326007)(4001430100002)(8676002)(7846002)(47776003)(11100500001)(19580405001)(5003940100001)(575784001)(16796002)(50986999)(450100001)(6806005)(92566002)(87936001)(19580395003)(33646002)(229853001)(2351001)(8936002)(50226002)(36756003)(106466001)(77096005)(189998001)(97736004)(110136002)(81156014)(81166006)(107886002)(42882005); DIR:OUT; SFP:1102; SCL:1; SRVR:DB4PR07MB0495; H:fihe3nok0735.emea.nsn-net.net; FPR:; SPF:Pass; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; AM1FFO11FD031; 1:VGT1GryVMKFTyUODbKHxJPIZKnnsGjd60ajXS2ZVh6YYOMOxzMv9UWD4SFwlus7T9zPxG5HnvJhik6N0oc8T6g/hD1eCLjHZQF2hhYvNkf5SeqYDJ6hq0X/TA/Q53XEhujTRGx77N9F6qdcm/uUiIUQ3sPlXyvZVC2VkItFnf28Ue5wUcMf/87G/KW5i6JXaQDlalgl9C3Ql+w65ptcFmY5rix6UOtKanVHaknyitTvS7+gwmnkNesnq4I+OO1/86MaK1JAvE5LkMKcL4XU0YlsiANFwVURTmlwresHxT0zTxydk3y+i90Ng4RYpeMYHOAN+I5Y92pxq4xJwMd2pknul8n1E1ZfV2mQPLatHFzx6mK2kQMLdSG5+PQ470HdXtHgk6oX31SsiorSPoKuJyTxJosbuBUbUvxBxGDTbAKXiUOu+Fp55vt4sQWfimkPGN4BkZiilaBJEcriLbMSGoabcZcmRpJoubHdoG7QyR1C6XUyaoYBBdDXIUun8d+S4 MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 311b9972-f676-4410-8e2f-08d3b217c5a6 X-Microsoft-Exchange-Diagnostics: 1; DB4PR07MB0495; 2:Izh2fYzsE8ARvVfRWlTT/fQ+OCfhvSpNrPsuMlsh3MUH22dzsy67irziIlZGCi1pgV2IteLMGIbi1aJeflt+3DKbT2/b4kvtD7GYvogBigrdml9d433+OdcA0K2odecfW19x4VxxE/QbQKd7ZlhJkJcaRsppsJb5eoqh97XgSoYaTnJwOX4uQaJNAs7cS1q9; 3:xfxYsOxwGKddbJps9uXRPwXsgXBSmGTyx+65e3xGeS28I7V+43Zaupj+OpwozbHhCk+zea986JoI9nmS7INk3OyzEpI2CT7l9XX+BLbSs4PpGA3WgnRwSgbaAigz99+1et3xosl2HNkvZnCduOZNL4eJ88lnaTMMWxdCvoPKqx6XOVUzfksODYzIXoYNfpkhsbihtCD9neRpqTwn8rBn15OB8Y+ZdLhQZKXnjYdhoncBBkpvVeK1k1zkQH3r31SeYSaqFQ5cQ+NTXU1S/NYRmA== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501002); SRVR:DB4PR07MB0495; X-Microsoft-Exchange-Diagnostics: 1; DB4PR07MB0495; 25:Ca/TNEVqgpqR6RJ4kS6OlPfUk7GQpB4r5vtJR2Sen/VzdQl94LZNpmawbz1ylU+KjukTOV13TIaDn3e0Lv45RUV5GLoV/Rf8nK+N+8OOIgQ/djiHpzgC6sCJr74bgZOa7KOd0i4Y2/QUV7Gqo/5Bu6XyF15hqRpbLAxjDRSChnT6s2VWLA8C34p4d+bRaSAxcqNcMePuX43EcCVJ8g9YyerbWneFOgjhTzgoBzGkVvna5Te7omO7MWXqLsi2ddrvwSHsxbhKw263OrHEOHQC4Ut1B2svGg4fz2U9jrmkDK41PxjtNS2lSCYmAUeVDfPPwYMlQb8TkrIWCeo+NdjTkjqMfEsPSIlSRbv9CNG6gpJgbJBwlF0LGZfCePBuMas6rUjZIHsFDi9bht9CBeX9ma9+KdLsFSjm/3gxqJJME/mIjtEnP0MAwnSrf5lzWNmlS1Y3pRXV/g8FSTzTEJD1VUHQQDy5m7hLbM2r/maN1FXiE0jPAAZeO5TRt2CkXoaQizf/rgNaXIRt9hkzRbzG0rr+o0bgC22on4L1G1OpEwu1iPt9hOMuPE8YSeGhi3QB0j/VP0somb9jywNaxR7u9lOm2Oh8W2cmhCfaiHm3eYvAEumfLlw30Y6xmwjfLbfhuTqhpgCzQ0Z6ZEygm2MqqtEevwoTgDW3+F2vCqfaf/z0djzbvldpXhhZVm8KuIPaGQs69j/w6pcty2IL2TLXvsmSpwrQMxrqAJzKTDwZqJTppVHByfKHyKWC9KSqzy9cyhmmFGWeL1FaWkKvXKzJfw== X-Microsoft-Exchange-Diagnostics: 1; DB4PR07MB0495; 31:/pBminSibspa9flXKMm2KqsGjKxATAKtGeZ61dNlOmt5wc9ZBTTdaI1RkGM9XrKQt7mXjxuyH4pMPSjo/2vHOstHQZv9z21T+5HuC3ULHqcIkqptWnokcPgf2WxvG6ZspE3ZdtHuZIuDAoQzbHUAlWvOM+/ob5V4DX9d/6FyANrTaByrTKOPhZm+ijJbHrSvSqMjkK9a2A2y7DkhWrNBnQ==; 20:mnXDxcPiJ3pcuLYWCCo/HBXLzH0piAZExlgKUpzCZ2eJGAcgiZKx4qff3liLCXjnT9k8pw2Pm7GK7wmrmxH1Zg8PvHLZkQ3RZnombzqUh1KaIagKAX4XJahoG5M6zeaGNo4M2Lfwol1U3wMdh0XU8jUjrkhwtW75/FceWKalYeBQU4Bx44Ri+zws9vmwCOnkDqd0YbqGJy/F8MSK9rwf4jJvo23N8FTPyd5hc6ndiyixDYhNjyjWP8jtQziq0jJuovI5+778ocYoMloDvMB4XQYQb+CDsVpp5+O3WZzWAg1CWK0lXXO9DVhzD0GfM0ItIXSe6xdfWcnck0J59UT9oRGiQQmnbyEOdjFfme+OfEqebwfheWk70C/dr6mKbKpLM26TjxWhAXZ+MMdTefCz/IZfNiuALNedDQAiz/1S8Oc8dRo+nVN/ELu+RlQlXqRb+xnmqK4YdJYLbQVNUYYpd/F99gjBYI3L/hVCKTvrThQoV/cb78LVGEAzcFScxUPXuOAXMoCyE6IC4yK6W9+yYnmnPBy7uLwDSswXf64Tcb88oPKjx/h0uFUsh4PInMCamsu4NxhYhHCgJRKbXT3gcVg9K9B+H55x8BsoVt2NwOw= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(82608151540597); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(13018025)(13016025)(8121501046)(5005006)(10201501046)(3002001)(6055026); SRVR:DB4PR07MB0495; BCL:0; PCL:0; RULEID:; SRVR:DB4PR07MB0495; X-Microsoft-Exchange-Diagnostics: 1; DB4PR07MB0495; 4:eSiz2PoqYXmHyFMO7B00P7Wm3aBsrEiEuGahbvYWsEUfwdPtHybpzRMg+eY93kULF8orl+5ZBLOQhuCL4c6CNoDLeoNmoDaNuiIF2tlefKLLJ43uc2SzVITN4UmNeoOmEn8VkuhOxtm1tW/BLtwOMcZAPpMo+WqhpYSl/30HypNDoz0+JHW6vuACw3RG5QH00C0/xjZAk0w+Jv+oVs2XBL1Ha9aUUtUb2vdrL/Sw65LlvU7JOsyxy1EEV+cDj+ktZOcN9TcLB8EAjInTf5olsJ2Ir9J/UHZQeR2p7YvAeZ0p4uhcAnbGiKliUHVt1PCS7OODYSQ+dLMmKgIvPAL/Mqxx4fhtq39J8fvABg8d3uz6SRs4pGIDmPEE20N8CVgDOTplhjMF+OPha6QVO/AbWKrYcZlUbf0T8ZUUjOdfA2dQgGdswQ2LBE/11bAiihXD4qO2ajfRkkPPdCXhMWiQRw== X-Forefront-PRVS: 0011612A55 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DB4PR07MB0495; 23:1ky5DnO8EQqR5hDCoE0eI0x3Xxap/JAMC5P4SIyhU?= =?us-ascii?Q?zFCuGiD7xlxNMVqaFjGOKJibWHsbdmTahDuhliCxeW0RINv5+nirI9MneDlU?= =?us-ascii?Q?3FlQFa4LCCGLEurs2mTsL7oBu0VmDA4Nqap7DvgDVH9W4Y6v2fLcy44krx9k?= =?us-ascii?Q?etHhscwwutCyhrUWPtKQmpA06vHwoFT3s+eaeJxytU+ttLCVVdtqwJI4jqry?= =?us-ascii?Q?61EAvYBlPwhKVSflHyt5FH98dSYcYpFa4HrMBW9DDnNLxPXxV0j/dXbaW1ah?= =?us-ascii?Q?ZPhhRr90NND6lLABBRVXtGYQXWVTYnu80rHMSKDE5B9K0O5H6Mema8AZnCO5?= =?us-ascii?Q?cAqpp5UA4FvvLii9kQcLSsjyroCs0ekUetm/mKu9r5uCu9mWcQ8kQ2OtTm/r?= =?us-ascii?Q?Y0s1XZ1u5rVBJr69L0kNntfOHfE2Fq5Ahbu0xrDtHuBoM3PvW9FwnHGQWOdR?= =?us-ascii?Q?LksO2dHOfVXs5J0UCclf4Q9rxhmYHhW83e0wcKrUBTB+rA4FY3+zwhcmIT78?= =?us-ascii?Q?XKwM5A6xoE3oDl+X3c5OocTGoJCY/NQF9sQAMZmIy9AbLK6PEsr5B0fd3csG?= =?us-ascii?Q?pqy8zGz3XfPtKZFm1y9B8ZzRZQW7NL25ZHHCOPi/+U7W/lQEVYQ9/MePb86h?= =?us-ascii?Q?zRAAVsABNnvkpHC5KjtZy4FpW1ieNFj/4mak5cToKX1D11JuF3pb+MR2Gy12?= =?us-ascii?Q?RwKAMACX3IyLErsOMlrPExrPLqW5MAHaivJ7hd8EvaLQO4sg3Qda9iBpW4YV?= =?us-ascii?Q?FXUeFTn83cJcX2MeP0k41b4XjXMl7NmKTQAWpYbWXtWV9uHJivsZc+5vfBWX?= =?us-ascii?Q?4WOVloP3UrISHSP6pUVaXvIFD7KxWg1RZYesGAoEmaCGkmeusJ9coXhgf34B?= =?us-ascii?Q?T/Q/gcY5Evd+EhFb/RFzR6yCJnf7W5/YRaMoR2kRhURAwq1EZ4C9ASlorsMt?= =?us-ascii?Q?chPphPceyjhWx61qYkRRl3o6pMQwL5dVrPn/6L/XXqSmJQozFYXGq64znE+n?= =?us-ascii?Q?9Mldpaz/PmzhmyeSczkjwiVO9fRyq+P/VP4L7Xncw1t71pIUijSdmyhYP4uQ?= =?us-ascii?Q?Wja2IcIg+B38Nymd9XQKV7F33AEkFsc5a14gb9NokHSl3IhB/q1JY2sP8wIi?= =?us-ascii?Q?ApLnFVvUvM=3D?= X-Microsoft-Exchange-Diagnostics: 1; DB4PR07MB0495; 6:vj6n/+dxjDsGV27+uDxziZHg+DwFWbCnPUeXlhe1x6+fjyZYh19pWTvfq5HPCzg9/wOmvHz3fpRv3DpfUtOpHAKn+ee0yvge/RTWl2tznnY0+faBQeZNEtbuIz4PRJk3mBhEpjPnL/grDOwT5yyawMfbnfkPLfBHk8e7wxcB7XI9T6iX9O5iBOLW5yb3c00qttyp9uw9Hus4yaNxlTQVNeH4lahIUX2BCJqZDSZ9y8FpEwgVxls3xpdoBPWOaXmQWNWIOy5ZpQaKMHcPXz4w4zpyZe2niWpS2W12brPDS9wJ6eowAEtW5Ky1002SkXs36+wTR30vM57jwh+gC8NUHA==; 5:tciCx8Wq7U0v8RdfefM2gwlTzZFIXpQv0tm/sZbkzEEY+YLh5bKUmdgBxo5g5+xfMfeRWlQSSPEHlC3wG9YounlTuGN7OCXPfEXI7kAkkghmYoJMVNENux0wn4cPfxKakPZTWx4Sq9rgnIVvalxR3Q==; 24:xGFQsfANVmslY8r4pIW6vNg4Zq84CXmzPG+yvonKAk7qvyWq+s49gZz0ZdX3J0DAYNoayiJe60pcFpTUi/pemFEUCBBPGTrnx2zzF2PUjTQ=; 7:rS0Jj3JB1nu+Wn/QRxgmvifvf+r9cfvVX6PWWJwhNs9DJiiZTORB0tBC83CeabFM5wKR50iXm7Jfhcj0F38YdIAD10J9QHjqRCUTeT4kMd2D0LuZ1dvimNxs8LdLN91/wTj6VLzUv3iDtN4EdWbewZKV2Zno/BrVG6K5yuYcnvTGrVvznQ8YIipe+gDyfJ/kLVj2UFaXoWTUg+K5yYeTT7OavKwzljUeH5VQSCW2YDNhRonbeSAxkJXKAr6/BNPg SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: nokia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jul 2016 10:05:55.7830 (UTC) X-MS-Exchange-CrossTenant-Id: 5d471751-9675-428d-917b-70f44f9630b0 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5d471751-9675-428d-917b-70f44f9630b0; Ip=[131.228.2.241]; Helo=[fihe3nok0735.emea.nsn-net.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB4PR07MB0495 X-Topics: patch Cc: Petri Savolainen Subject: [lng-odp] [PATCH v3 1/4] linux-gen: pool: optimize thread local buffer cache X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" Optimize local buffer cache performance which is critical to many use cases - including packet IO. Main parts of the optimization are: * Local cache implemented as an array of buf_hdr pointers, instead of a linked list (which causes a lot of cache misses) * Alloc and free N buffers per operation All above steps are needed to demonstrate the performance upgrade. Some related pool functions (get_buf(), ret_buf(), etc) were moved from pool header to c source file, since those were actual local to the c source file. Also some unused pool variables are removed also. Signed-off-by: Petri Savolainen Signed-off-by: Matias Elo --- V2: - Split pktio modifications into a separate patch (Bill) - Improve performance by adding separate functions for single buffer alloc/free operations .../linux-generic/include/odp_buffer_inlines.h | 26 +- .../linux-generic/include/odp_buffer_internal.h | 5 +- platform/linux-generic/include/odp_internal.h | 2 - platform/linux-generic/include/odp_pool_internal.h | 143 +------ platform/linux-generic/odp_buffer.c | 3 - platform/linux-generic/odp_packet.c | 5 +- platform/linux-generic/odp_pool.c | 473 +++++++++++++++++---- 7 files changed, 426 insertions(+), 231 deletions(-) -- 2.7.4 diff --git a/platform/linux-generic/include/odp_buffer_inlines.h b/platform/linux-generic/include/odp_buffer_inlines.h index 3f4d9fd..2b1ab42 100644 --- a/platform/linux-generic/include/odp_buffer_inlines.h +++ b/platform/linux-generic/include/odp_buffer_inlines.h @@ -56,30 +56,12 @@ static inline odp_buffer_hdr_t *odp_buf_to_hdr(odp_buffer_t buf) (pool->pool_mdata_addr + (index * ODP_CACHE_LINE_SIZE)); } -static inline uint32_t odp_buffer_refcount(odp_buffer_hdr_t *buf) +static inline uint32_t pool_id_from_buf(odp_buffer_t buf) { - return odp_atomic_load_u32(&buf->ref_count); -} + odp_buffer_bits_t handle; -static inline uint32_t odp_buffer_incr_refcount(odp_buffer_hdr_t *buf, - uint32_t val) -{ - return odp_atomic_fetch_add_u32(&buf->ref_count, val) + val; -} - -static inline uint32_t odp_buffer_decr_refcount(odp_buffer_hdr_t *buf, - uint32_t val) -{ - uint32_t tmp; - - tmp = odp_atomic_fetch_sub_u32(&buf->ref_count, val); - - if (tmp < val) { - odp_atomic_fetch_add_u32(&buf->ref_count, val - tmp); - return 0; - } else { - return tmp - val; - } + handle.handle = buf; + return handle.pool_id; } static inline odp_buffer_hdr_t *validate_buf(odp_buffer_t buf) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index f21364c..7b0ef8b 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -114,7 +114,6 @@ struct odp_buffer_hdr_t { union { uint32_t all; struct { - uint32_t zeroized:1; /* Zeroize buf data on free */ uint32_t hdrdata:1; /* Data is in buffer hdr */ uint32_t sustain:1; /* Sustain order */ }; @@ -123,7 +122,6 @@ struct odp_buffer_hdr_t { int8_t type; /* buffer type */ odp_event_type_t event_type; /* for reuse as event */ uint32_t size; /* max data size */ - odp_atomic_u32_t ref_count; /* reference count */ odp_pool_t pool_hdl; /* buffer pool handle */ union { uint64_t buf_u64; /* user u64 */ @@ -174,6 +172,9 @@ typedef struct { odp_buffer_t buffer_alloc(odp_pool_t pool, size_t size); int buffer_alloc_multi(odp_pool_t pool_hdl, size_t size, odp_buffer_t buf[], int num); +void buffer_free(uint32_t pool_id, const odp_buffer_t buf); +void buffer_free_multi(uint32_t pool_id, + const odp_buffer_t buf[], int num_free); int seg_alloc_head(odp_buffer_hdr_t *buf_hdr, int segcount); void seg_free_head(odp_buffer_hdr_t *buf_hdr, int segcount); int seg_alloc_tail(odp_buffer_hdr_t *buf_hdr, int segcount); diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index d12f850..8bad450 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -119,8 +119,6 @@ int odp_tm_term_global(void); int _odp_int_name_tbl_init_global(void); int _odp_int_name_tbl_term_global(void); -void _odp_flush_caches(void); - int cpuinfo_parser(FILE *file, system_info_t *sysinfo); uint64_t odp_cpu_hz_current(int id); diff --git a/platform/linux-generic/include/odp_pool_internal.h b/platform/linux-generic/include/odp_pool_internal.h index 3317bd0..d6717ff 100644 --- a/platform/linux-generic/include/odp_pool_internal.h +++ b/platform/linux-generic/include/odp_pool_internal.h @@ -51,15 +51,25 @@ typedef struct _odp_buffer_pool_init_t { void *buf_init_arg; /**< Argument to be passed to buf_init() */ } _odp_buffer_pool_init_t; /**< Type of buffer initialization struct */ +#define POOL_MAX_LOCAL_CHUNKS 4 +#define POOL_CHUNK_SIZE 32 +#define POOL_MAX_LOCAL_BUFS (POOL_MAX_LOCAL_CHUNKS * POOL_CHUNK_SIZE) + +struct local_cache_s { + uint64_t bufallocs; /* Local buffer alloc count */ + uint64_t buffrees; /* Local buffer free count */ + + uint32_t num_buf; + odp_buffer_hdr_t *buf[POOL_MAX_LOCAL_BUFS]; +}; + /* Local cache for buffer alloc/free acceleration */ typedef struct local_cache_t { union { - struct { - odp_buffer_hdr_t *buf_freelist; /* The local cache */ - uint64_t bufallocs; /* Local buffer alloc count */ - uint64_t buffrees; /* Local buffer free count */ - }; - uint8_t pad[ODP_CACHE_LINE_SIZE_ROUNDUP(sizeof(uint64_t))]; + struct local_cache_s s; + + uint8_t pad[ODP_CACHE_LINE_SIZE_ROUNDUP( + sizeof(struct local_cache_s))]; }; } local_cache_t; @@ -214,127 +224,6 @@ static inline void ret_blk(struct pool_entry_s *pool, void *block) odp_atomic_inc_u64(&pool->poolstats.blkfrees); } -static inline odp_buffer_hdr_t *get_buf(struct pool_entry_s *pool) -{ - odp_buffer_hdr_t *myhead; - POOL_LOCK(&pool->buf_lock); - - myhead = pool->buf_freelist; - - if (odp_unlikely(myhead == NULL)) { - POOL_UNLOCK(&pool->buf_lock); - odp_atomic_inc_u64(&pool->poolstats.bufempty); - } else { - pool->buf_freelist = myhead->next; - POOL_UNLOCK(&pool->buf_lock); - uint64_t bufcount = - odp_atomic_fetch_sub_u32(&pool->bufcount, 1) - 1; - - /* Check for low watermark condition */ - if (bufcount == pool->buf_low_wm && !pool->buf_low_wm_assert) { - pool->buf_low_wm_assert = 1; - odp_atomic_inc_u64(&pool->poolstats.buf_low_wm_count); - } - - odp_atomic_inc_u64(&pool->poolstats.bufallocs); - } - - return (void *)myhead; -} - -static inline void ret_buf(struct pool_entry_s *pool, odp_buffer_hdr_t *buf) -{ - if (!buf->flags.hdrdata && buf->type != ODP_EVENT_BUFFER) { - while (buf->segcount > 0) { - if (buffer_is_secure(buf) || pool_is_secure(pool)) - memset(buf->addr[buf->segcount - 1], - 0, buf->segsize); - ret_blk(pool, buf->addr[--buf->segcount]); - } - buf->size = 0; - } - - buf->allocator = ODP_FREEBUF; /* Mark buffer free */ - POOL_LOCK(&pool->buf_lock); - buf->next = pool->buf_freelist; - pool->buf_freelist = buf; - POOL_UNLOCK(&pool->buf_lock); - - uint64_t bufcount = odp_atomic_fetch_add_u32(&pool->bufcount, 1) + 1; - - /* Check if low watermark condition should be deasserted */ - if (bufcount == pool->buf_high_wm && pool->buf_low_wm_assert) { - pool->buf_low_wm_assert = 0; - odp_atomic_inc_u64(&pool->poolstats.buf_high_wm_count); - } - - odp_atomic_inc_u64(&pool->poolstats.buffrees); -} - -static inline void *get_local_buf(local_cache_t *buf_cache, - struct pool_entry_s *pool, - size_t totsize) -{ - odp_buffer_hdr_t *buf = buf_cache->buf_freelist; - - if (odp_likely(buf != NULL)) { - buf_cache->buf_freelist = buf->next; - - if (odp_unlikely(buf->size < totsize)) { - intmax_t needed = totsize - buf->size; - - do { - void *blk = get_blk(pool); - if (odp_unlikely(blk == NULL)) { - ret_buf(pool, buf); - buf_cache->buffrees--; - return NULL; - } - buf->addr[buf->segcount++] = blk; - needed -= pool->seg_size; - } while (needed > 0); - - buf->size = buf->segcount * pool->seg_size; - } - - buf_cache->bufallocs++; - } - - return buf; -} - -static inline void ret_local_buf(local_cache_t *buf_cache, - odp_buffer_hdr_t *buf) -{ - buf->allocator = ODP_FREEBUF; - buf->next = buf_cache->buf_freelist; - buf_cache->buf_freelist = buf; - - buf_cache->buffrees++; -} - -static inline void flush_cache(local_cache_t *buf_cache, - struct pool_entry_s *pool) -{ - odp_buffer_hdr_t *buf = buf_cache->buf_freelist; - uint32_t flush_count = 0; - - while (buf != NULL) { - odp_buffer_hdr_t *next = buf->next; - ret_buf(pool, buf); - buf = next; - flush_count++; - } - - odp_atomic_add_u64(&pool->poolstats.bufallocs, buf_cache->bufallocs); - odp_atomic_add_u64(&pool->poolstats.buffrees, - buf_cache->buffrees - flush_count); - - buf_cache->buf_freelist = NULL; - buf_cache->bufallocs = 0; - buf_cache->buffrees = 0; -} - static inline odp_pool_t pool_index_to_handle(uint32_t pool_id) { return _odp_cast_scalar(odp_pool_t, pool_id); diff --git a/platform/linux-generic/odp_buffer.c b/platform/linux-generic/odp_buffer.c index e7e4d58..ce2fdba 100644 --- a/platform/linux-generic/odp_buffer.c +++ b/platform/linux-generic/odp_buffer.c @@ -67,9 +67,6 @@ int odp_buffer_snprint(char *str, uint32_t n, odp_buffer_t buf) len += snprintf(&str[len], n-len, " size %" PRIu32 "\n", hdr->size); len += snprintf(&str[len], n-len, - " ref_count %" PRIu32 "\n", - odp_atomic_load_u32(&hdr->ref_count)); - len += snprintf(&str[len], n-len, " type %i\n", hdr->type); return len; diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c index 0e319d2..474fa81 100644 --- a/platform/linux-generic/odp_packet.c +++ b/platform/linux-generic/odp_packet.c @@ -972,10 +972,7 @@ int _odp_packet_copy_md_to_packet(odp_packet_t srcpkt, odp_packet_t dstpkt) srchdr->buf_hdr.uarea_size ? dsthdr->buf_hdr.uarea_size : srchdr->buf_hdr.uarea_size); - odp_atomic_store_u32( - &dsthdr->buf_hdr.ref_count, - odp_atomic_load_u32( - &srchdr->buf_hdr.ref_count)); + copy_packet_parser_metadata(srchdr, dsthdr); /* Metadata copied, but return indication of whether the packet diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 419f03f..0a427ed 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -57,8 +57,15 @@ static const char SHM_DEFAULT_NAME[] = "odp_buffer_pools"; /* Pool entry pointers (for inlining) */ void *pool_entry_ptr[ODP_CONFIG_POOLS]; -/* Cache thread id locally for local cache performance */ -static __thread int local_id; +/* Thread local variables */ +typedef struct pool_local_t { + local_cache_t *cache[ODP_CONFIG_POOLS]; + int thr_id; +} pool_local_t; + +static __thread pool_local_t local; + +static void flush_cache(local_cache_t *buf_cache, struct pool_entry_s *pool); int odp_pool_init_global(void) { @@ -111,7 +118,19 @@ int odp_pool_init_global(void) int odp_pool_init_local(void) { - local_id = odp_thread_id(); + pool_entry_t *pool; + int i; + int thr_id = odp_thread_id(); + + memset(&local, 0, sizeof(pool_local_t)); + + for (i = 0; i < ODP_CONFIG_POOLS; i++) { + pool = get_pool_entry(i); + local.cache[i] = &pool->s.local_cache[thr_id]; + local.cache[i]->s.num_buf = 0; + } + + local.thr_id = thr_id; return 0; } @@ -144,7 +163,14 @@ int odp_pool_term_global(void) int odp_pool_term_local(void) { - _odp_flush_caches(); + int i; + + for (i = 0; i < ODP_CONFIG_POOLS; i++) { + pool_entry_t *pool = get_pool_entry(i); + + flush_cache(local.cache[i], &pool->s); + } + return 0; } @@ -179,10 +205,53 @@ int odp_pool_capability(odp_pool_capability_t *capa) return 0; } -/** +static inline odp_buffer_hdr_t *get_buf(struct pool_entry_s *pool) +{ + odp_buffer_hdr_t *myhead; + + POOL_LOCK(&pool->buf_lock); + + myhead = pool->buf_freelist; + + if (odp_unlikely(myhead == NULL)) { + POOL_UNLOCK(&pool->buf_lock); + odp_atomic_inc_u64(&pool->poolstats.bufempty); + } else { + pool->buf_freelist = myhead->next; + POOL_UNLOCK(&pool->buf_lock); + + odp_atomic_fetch_sub_u32(&pool->bufcount, 1); + odp_atomic_inc_u64(&pool->poolstats.bufallocs); + } + + return (void *)myhead; +} + +static inline void ret_buf(struct pool_entry_s *pool, odp_buffer_hdr_t *buf) +{ + if (!buf->flags.hdrdata && buf->type != ODP_EVENT_BUFFER) { + while (buf->segcount > 0) { + if (buffer_is_secure(buf) || pool_is_secure(pool)) + memset(buf->addr[buf->segcount - 1], + 0, buf->segsize); + ret_blk(pool, buf->addr[--buf->segcount]); + } + buf->size = 0; + } + + buf->allocator = ODP_FREEBUF; /* Mark buffer free */ + POOL_LOCK(&pool->buf_lock); + buf->next = pool->buf_freelist; + pool->buf_freelist = buf; + POOL_UNLOCK(&pool->buf_lock); + + odp_atomic_fetch_add_u32(&pool->bufcount, 1); + odp_atomic_inc_u64(&pool->poolstats.buffrees); +} + +/* * Pool creation */ - odp_pool_t _pool_create(const char *name, odp_pool_param_t *params, uint32_t shmflags) @@ -208,9 +277,6 @@ odp_pool_t _pool_create(const char *name, /* Restriction for v1.0: All non-packet buffers are unsegmented */ int unseg = 1; - /* Restriction for v1.0: No zeroization support */ - const int zeroized = 0; - uint32_t blk_size, buf_stride, buf_num, blk_num, seg_len = 0; uint32_t buf_align = params->type == ODP_POOL_BUFFER ? params->buf.align : 0; @@ -350,7 +416,6 @@ odp_pool_t _pool_create(const char *name, POOL_UNLOCK(&pool->s.lock); pool->s.flags.unsegmented = unseg; - pool->s.flags.zeroized = zeroized; pool->s.seg_size = unseg ? blk_size : seg_len; pool->s.blk_size = blk_size; @@ -383,9 +448,7 @@ odp_pool_t _pool_create(const char *name, /* Iniitalize buffer metadata */ tmp->allocator = ODP_FREEBUF; tmp->flags.all = 0; - tmp->flags.zeroized = zeroized; tmp->size = 0; - odp_atomic_init_u32(&tmp->ref_count, 0); tmp->type = params->type; tmp->event_type = params->type; tmp->pool_hdl = pool->s.pool_hdl; @@ -503,6 +566,41 @@ int odp_pool_info(odp_pool_t pool_hdl, odp_pool_info_t *info) return 0; } +static inline void get_local_cache_bufs(local_cache_t *buf_cache, uint32_t idx, + odp_buffer_hdr_t *buf_hdr[], + uint32_t num) +{ + uint32_t i; + + for (i = 0; i < num; i++) { + buf_hdr[i] = buf_cache->s.buf[idx + i]; + odp_prefetch(buf_hdr[i]); + odp_prefetch_store(buf_hdr[i]); + } +} + +static void flush_cache(local_cache_t *buf_cache, struct pool_entry_s *pool) +{ + uint32_t flush_count = 0; + uint32_t num; + + while ((num = buf_cache->s.num_buf)) { + odp_buffer_hdr_t *buf; + + buf = buf_cache->s.buf[num - 1]; + ret_buf(pool, buf); + flush_count++; + buf_cache->s.num_buf--; + } + + odp_atomic_add_u64(&pool->poolstats.bufallocs, buf_cache->s.bufallocs); + odp_atomic_add_u64(&pool->poolstats.buffrees, + buf_cache->s.buffrees - flush_count); + + buf_cache->s.bufallocs = 0; + buf_cache->s.buffrees = 0; +} + int odp_pool_destroy(odp_pool_t pool_hdl) { uint32_t pool_id = pool_handle_to_index(pool_hdl); @@ -621,71 +719,207 @@ void seg_free_tail(odp_buffer_hdr_t *buf_hdr, int segcount) buf_hdr->size = buf_hdr->segcount * pool->s.seg_size; } -odp_buffer_t buffer_alloc(odp_pool_t pool_hdl, size_t size) +static inline int get_local_bufs(local_cache_t *buf_cache, + odp_buffer_hdr_t *buf_hdr[], uint32_t max_num) +{ + uint32_t num_buf = buf_cache->s.num_buf; + uint32_t num = num_buf; + + if (odp_unlikely(num_buf == 0)) + return 0; + + if (odp_likely(max_num < num)) + num = max_num; + + get_local_cache_bufs(buf_cache, num_buf - num, buf_hdr, num); + buf_cache->s.num_buf -= num; + buf_cache->s.bufallocs += num; + + return num; +} + +static inline void ret_local_buf(local_cache_t *buf_cache, uint32_t idx, + odp_buffer_hdr_t *buf) +{ + buf_cache->s.buf[idx] = buf; + buf_cache->s.num_buf++; + buf_cache->s.buffrees++; +} + +static inline void ret_local_bufs(local_cache_t *buf_cache, uint32_t idx, + odp_buffer_hdr_t *buf[], int num_buf) +{ + int i; + + for (i = 0; i < num_buf; i++) + buf_cache->s.buf[idx + i] = buf[i]; + + buf_cache->s.num_buf += num_buf; + buf_cache->s.buffrees += num_buf; +} + +int buffer_alloc_multi(odp_pool_t pool_hdl, size_t size, + odp_buffer_t buf[], int max_num) { uint32_t pool_id = pool_handle_to_index(pool_hdl); pool_entry_t *pool = get_pool_entry(pool_id); uintmax_t totsize = pool->s.headroom + size + pool->s.tailroom; - odp_anybuf_t *buf; + odp_buffer_hdr_t *buf_tbl[max_num]; + odp_buffer_hdr_t *buf_hdr; + int num, i; + intmax_t needed; + void *blk; /* Reject oversized allocation requests */ if ((pool->s.flags.unsegmented && totsize > pool->s.seg_size) || (!pool->s.flags.unsegmented && totsize > pool->s.seg_size * ODP_BUFFER_MAX_SEG)) - return ODP_BUFFER_INVALID; + return 0; /* Try to satisfy request from the local cache */ - buf = (odp_anybuf_t *) - (void *)get_local_buf(&pool->s.local_cache[local_id], - &pool->s, totsize); + num = get_local_bufs(local.cache[pool_id], buf_tbl, max_num); /* If cache is empty, satisfy request from the pool */ - if (odp_unlikely(buf == NULL)) { - buf = (odp_anybuf_t *)(void *)get_buf(&pool->s); + if (odp_unlikely(num < max_num)) { + for (; num < max_num; num++) { + buf_hdr = get_buf(&pool->s); - if (odp_unlikely(buf == NULL)) + if (odp_unlikely(buf_hdr == NULL)) + goto pool_empty; + + /* Get blocks for this buffer, if pool uses + * application data */ + if (buf_hdr->size < totsize) { + uint32_t segcount; + + needed = totsize - buf_hdr->size; + do { + blk = get_blk(&pool->s); + if (odp_unlikely(blk == NULL)) { + ret_buf(&pool->s, buf_hdr); + goto pool_empty; + } + + segcount = buf_hdr->segcount++; + buf_hdr->addr[segcount] = blk; + needed -= pool->s.seg_size; + } while (needed > 0); + buf_hdr->size = buf_hdr->segcount * + pool->s.seg_size; + } + + buf_tbl[num] = buf_hdr; + } + } + +pool_empty: + for (i = 0; i < num; i++) { + buf_hdr = buf_tbl[i]; + + /* Mark buffer as allocated */ + buf_hdr->allocator = local.thr_id; + + /* By default, buffers are not associated with + * an ordered queue */ + buf_hdr->origin_qe = NULL; + + buf[i] = odp_hdr_to_buf(buf_hdr); + + /* Add more segments if buffer from local cache is too small */ + if (odp_unlikely(buf_hdr->size < totsize)) { + needed = totsize - buf_hdr->size; + do { + blk = get_blk(&pool->s); + if (odp_unlikely(blk == NULL)) { + int j; + + ret_buf(&pool->s, buf_hdr); + buf_hdr = NULL; + local.cache[pool_id]->s.buffrees--; + + /* move remaining bufs up one step + * and update loop counters */ + num--; + for (j = i; j < num; j++) + buf_tbl[j] = buf_tbl[j + 1]; + + i--; + break; + } + needed -= pool->s.seg_size; + buf_hdr->addr[buf_hdr->segcount++] = blk; + buf_hdr->size = buf_hdr->segcount * + pool->s.seg_size; + } while (needed > 0); + } + } + + return num; +} + +odp_buffer_t buffer_alloc(odp_pool_t pool_hdl, size_t size) +{ + uint32_t pool_id = pool_handle_to_index(pool_hdl); + pool_entry_t *pool = get_pool_entry(pool_id); + uintmax_t totsize = pool->s.headroom + size + pool->s.tailroom; + odp_buffer_hdr_t *buf_hdr; + intmax_t needed; + void *blk; + + /* Reject oversized allocation requests */ + if ((pool->s.flags.unsegmented && totsize > pool->s.seg_size) || + (!pool->s.flags.unsegmented && + totsize > pool->s.seg_size * ODP_BUFFER_MAX_SEG)) + return 0; + + /* Try to satisfy request from the local cache. If cache is empty, + * satisfy request from the pool */ + if (odp_unlikely(!get_local_bufs(local.cache[pool_id], &buf_hdr, 1))) { + buf_hdr = get_buf(&pool->s); + + if (odp_unlikely(buf_hdr == NULL)) return ODP_BUFFER_INVALID; /* Get blocks for this buffer, if pool uses application data */ - if (buf->buf.size < totsize) { - intmax_t needed = totsize - buf->buf.size; + if (buf_hdr->size < totsize) { + needed = totsize - buf_hdr->size; do { - uint8_t *blk = get_blk(&pool->s); - if (blk == NULL) { - ret_buf(&pool->s, &buf->buf); + blk = get_blk(&pool->s); + if (odp_unlikely(blk == NULL)) { + ret_buf(&pool->s, buf_hdr); return ODP_BUFFER_INVALID; } - buf->buf.addr[buf->buf.segcount++] = blk; + buf_hdr->addr[buf_hdr->segcount++] = blk; needed -= pool->s.seg_size; } while (needed > 0); - buf->buf.size = buf->buf.segcount * pool->s.seg_size; + buf_hdr->size = buf_hdr->segcount * pool->s.seg_size; } } - /* Mark buffer as allocated */ - buf->buf.allocator = local_id; + buf_hdr->allocator = local.thr_id; - /* By default, buffers inherit their pool's zeroization setting */ - buf->buf.flags.zeroized = pool->s.flags.zeroized; + /* By default, buffers are not associated with + * an ordered queue */ + buf_hdr->origin_qe = NULL; - /* By default, buffers are not associated with an ordered queue */ - buf->buf.origin_qe = NULL; - - return odp_hdr_to_buf(&buf->buf); -} - -int buffer_alloc_multi(odp_pool_t pool_hdl, size_t size, - odp_buffer_t buf[], int num) -{ - int count; - - for (count = 0; count < num; ++count) { - buf[count] = buffer_alloc(pool_hdl, size); - if (buf[count] == ODP_BUFFER_INVALID) - break; + /* Add more segments if buffer from local cache is too small */ + if (odp_unlikely(buf_hdr->size < totsize)) { + needed = totsize - buf_hdr->size; + do { + blk = get_blk(&pool->s); + if (odp_unlikely(blk == NULL)) { + ret_buf(&pool->s, buf_hdr); + buf_hdr = NULL; + local.cache[pool_id]->s.buffrees--; + return ODP_BUFFER_INVALID; + } + buf_hdr->addr[buf_hdr->segcount++] = blk; + needed -= pool->s.seg_size; + } while (needed > 0); + buf_hdr->size = buf_hdr->segcount * pool->s.seg_size; } - return count; + return odp_hdr_to_buf(buf_hdr); } odp_buffer_t odp_buffer_alloc(odp_pool_t pool_hdl) @@ -701,35 +935,132 @@ int odp_buffer_alloc_multi(odp_pool_t pool_hdl, odp_buffer_t buf[], int num) return buffer_alloc_multi(pool_hdl, buf_size, buf, num); } -void odp_buffer_free(odp_buffer_t buf) +static void multi_pool_free(odp_buffer_hdr_t *buf_hdr[], int num_buf) { - odp_buffer_hdr_t *buf_hdr = odp_buf_to_hdr(buf); - pool_entry_t *pool = odp_buf_to_pool(buf_hdr); + uint32_t pool_id, num; + local_cache_t *buf_cache; + pool_entry_t *pool; + int i, j, idx; + for (i = 0; i < num_buf; i++) { + pool_id = pool_handle_to_index(buf_hdr[i]->pool_hdl); + buf_cache = local.cache[pool_id]; + num = buf_cache->s.num_buf; + + if (num < POOL_MAX_LOCAL_BUFS) { + ret_local_buf(buf_cache, num, buf_hdr[i]); + continue; + } + + idx = POOL_MAX_LOCAL_BUFS - POOL_CHUNK_SIZE; + pool = get_pool_entry(pool_id); + + /* local cache full, return a chunk */ + for (j = 0; j < POOL_CHUNK_SIZE; j++) { + odp_buffer_hdr_t *tmp; + + tmp = buf_cache->s.buf[idx + i]; + ret_buf(&pool->s, tmp); + } + + num = POOL_MAX_LOCAL_BUFS - POOL_CHUNK_SIZE; + buf_cache->s.num_buf = num; + ret_local_buf(buf_cache, num, buf_hdr[i]); + } +} + +void buffer_free_multi(uint32_t pool_id, + const odp_buffer_t buf[], int num_free) +{ + local_cache_t *buf_cache = local.cache[pool_id]; + uint32_t num; + int i, idx; + pool_entry_t *pool; + odp_buffer_hdr_t *buf_hdr[num_free]; + int multi_pool = 0; + + for (i = 0; i < num_free; i++) { + uint32_t id; + + buf_hdr[i] = odp_buf_to_hdr(buf[i]); + ODP_ASSERT(buf_hdr[i]->allocator != ODP_FREEBUF); + buf_hdr[i]->allocator = ODP_FREEBUF; + id = pool_handle_to_index(buf_hdr[i]->pool_hdl); + multi_pool |= (pool_id != id); + } + + if (odp_unlikely(multi_pool)) { + multi_pool_free(buf_hdr, num_free); + return; + } + + num = buf_cache->s.num_buf; + + if (odp_likely((num + num_free) < POOL_MAX_LOCAL_BUFS)) { + ret_local_bufs(buf_cache, num, buf_hdr, num_free); + return; + } + + pool = get_pool_entry(pool_id); + + /* Return at least one chunk into the global pool */ + if (odp_unlikely(num_free > POOL_CHUNK_SIZE)) { + for (i = 0; i < num_free; i++) + ret_buf(&pool->s, buf_hdr[i]); + + return; + } + + idx = num - POOL_CHUNK_SIZE; + for (i = 0; i < POOL_CHUNK_SIZE; i++) + ret_buf(&pool->s, buf_cache->s.buf[idx + i]); + + num -= POOL_CHUNK_SIZE; + buf_cache->s.num_buf = num; + ret_local_bufs(buf_cache, num, buf_hdr, num_free); +} + +void buffer_free(uint32_t pool_id, const odp_buffer_t buf) +{ + local_cache_t *buf_cache = local.cache[pool_id]; + uint32_t num; + int i; + pool_entry_t *pool; + odp_buffer_hdr_t *buf_hdr; + + buf_hdr = odp_buf_to_hdr(buf); ODP_ASSERT(buf_hdr->allocator != ODP_FREEBUF); + buf_hdr->allocator = ODP_FREEBUF; - if (odp_unlikely(pool->s.buf_low_wm_assert || pool->s.blk_low_wm_assert)) - ret_buf(&pool->s, buf_hdr); - else - ret_local_buf(&pool->s.local_cache[local_id], buf_hdr); + num = buf_cache->s.num_buf; + + if (odp_likely((num + 1) < POOL_MAX_LOCAL_BUFS)) { + ret_local_bufs(buf_cache, num, &buf_hdr, 1); + return; + } + + pool = get_pool_entry(pool_id); + + num -= POOL_CHUNK_SIZE; + for (i = 0; i < POOL_CHUNK_SIZE; i++) + ret_buf(&pool->s, buf_cache->s.buf[num + i]); + + buf_cache->s.num_buf = num; + ret_local_bufs(buf_cache, num, &buf_hdr, 1); +} + +void odp_buffer_free(odp_buffer_t buf) +{ + uint32_t pool_id = pool_id_from_buf(buf); + + buffer_free(pool_id, buf); } void odp_buffer_free_multi(const odp_buffer_t buf[], int num) { - int i; + uint32_t pool_id = pool_id_from_buf(buf[0]); - for (i = 0; i < num; ++i) - odp_buffer_free(buf[i]); -} - -void _odp_flush_caches(void) -{ - int i; - - for (i = 0; i < ODP_CONFIG_POOLS; i++) { - pool_entry_t *pool = get_pool_entry(i); - flush_cache(&pool->s.local_cache[local_id], &pool->s); - } + buffer_free_multi(pool_id, buf, num); } void odp_pool_print(odp_pool_t pool_hdl) @@ -774,7 +1105,6 @@ void odp_pool_print(odp_pool_t pool_hdl) pool->s.quiesced ? "quiesced" : "active"); ODP_DBG(" pool opts %s, %s, %s\n", pool->s.flags.unsegmented ? "unsegmented" : "segmented", - pool->s.flags.zeroized ? "zeroized" : "non-zeroized", pool->s.flags.predefined ? "predefined" : "created"); ODP_DBG(" pool base %p\n", pool->s.pool_base_addr); ODP_DBG(" pool size %zu (%zu pages)\n", @@ -817,10 +1147,11 @@ void odp_pool_print(odp_pool_t pool_hdl) ODP_DBG(" blk low wm count %lu\n", blklowmct); } - odp_pool_t odp_buffer_pool(odp_buffer_t buf) { - return odp_buf_to_hdr(buf)->pool_hdl; + uint32_t pool_id = pool_id_from_buf(buf); + + return pool_index_to_handle(pool_id); } void odp_pool_param_init(odp_pool_param_t *params)