From patchwork Fri Nov 11 06:08:19 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Forrest Shi X-Patchwork-Id: 81788 Delivered-To: patch@linaro.org Received: by 10.140.97.165 with SMTP id m34csp1130199qge; Thu, 10 Nov 2016 23:17:18 -0800 (PST) X-Received: by 10.237.62.89 with SMTP id m25mr1781070qtf.119.1478848638831; Thu, 10 Nov 2016 23:17:18 -0800 (PST) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id z187si5941368qka.18.2016.11.10.23.17.18; Thu, 10 Nov 2016 23:17:18 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=pass (p=NONE dis=NONE) header.from=linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 4F37B60ED2; Fri, 11 Nov 2016 07:17:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAD_ENC_HEADER,BAYES_00, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id EEBA760EF4; Fri, 11 Nov 2016 07:17:00 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 7EB8460EF7; Fri, 11 Nov 2016 07:16:57 +0000 (UTC) Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0122.outbound.protection.outlook.com [104.47.38.122]) by lists.linaro.org (Postfix) with ESMTPS id D1B0660ED2 for ; Fri, 11 Nov 2016 07:16:54 +0000 (UTC) Received: from BN6PR03CA0015.namprd03.prod.outlook.com (10.168.230.153) by BN6PR03MB2978.namprd03.prod.outlook.com (10.175.126.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.707.6; Fri, 11 Nov 2016 07:16:51 +0000 Received: from BY2FFO11FD036.protection.gbl (2a01:111:f400:7c0c::194) by BN6PR03CA0015.outlook.office365.com (2603:10b6:404:23::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.707.6 via Frontend Transport; Fri, 11 Nov 2016 07:16:50 +0000 Received-SPF: Neutral (protection.outlook.com: 192.88.168.50 is neither permitted nor denied by domain of freescale.com) Received: from tx30smr01.am.freescale.net (192.88.168.50) by BY2FFO11FD036.mail.protection.outlook.com (10.1.14.221) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.707.3 via Frontend Transport; Fri, 11 Nov 2016 07:16:49 +0000 X-IncomingTopHeaderMarker: OriginalChecksum:; UpperCasedChecksum:; SizeAsReceived:505; Count:8 Received: from localhost (rock.ap.freescale.net [10.193.20.106]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id uAB7Gkt6009970; Fri, 11 Nov 2016 00:16:47 -0700 From: To: Date: Fri, 11 Nov 2016 14:08:19 +0800 Message-ID: <1478844499-20434-1-git-send-email-forrest.shi@linaro.org> X-Mailer: git-send-email 1.8.4 X-IncomingHeaderCount: 8 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:192.88.168.50; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(7916002)(2980300002)(199003)(189002)(8936002)(4326007)(8676002)(50466002)(2906002)(68736007)(48376002)(2876002)(50226002)(2351001)(6916009)(105606002)(106466001)(104016004)(305945005)(42882006)(110136003)(57986006)(7846002)(5660300001)(6666003)(36756003)(76506005)(81166006)(81156014)(626004)(97736004)(50986999)(356003)(77096005)(586003)(47776003)(86152002)(33646002)(5003940100001)(575784001)(87936001)(92566002)(189998001)(579004); DIR:OUT; SFP:1102; SCL:1; SRVR:BN6PR03MB2978; H:tx30smr01.am.freescale.net; FPR:; SPF:Neutral; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD036; 1:6B657z9YW4r6/h09mKmpAgtR/PYP22+i58vAaQxz+PYzwlyTrZduqAuiDquWQWQauA/4URIjMQwstH8qSZ+HbGI5LxvItkymD33sW4vb/p5L5PbRjszLrWGBw0lWdJLAaHIHDMCJgo5I2FHufMufTKtQnDmDUZUklc47h987IR7gAMvzFkUeWMLBmgRQjV+0ABJABVRR30EdnwDsoFL5LxDucjiKn01jpoMp19MDI7vbMwthdE4sWxGJGz4j1LnesW5WvQ4YKuE8mhU3O/SSgbumDDFnXvnI5wh8Ls8ydGOoxCzTz5S+NZ0z0O83D6oO3opF5X8oImHRlqf63Hcb2n0ZzBv7cQWhWLk2Bhvpdmh0N2coG0gSSY3XP0yYEPftHWXo41NdYsJVGvbMDGoUUD2pTzsZS6yxVsJm5mqa+8mb/scvqJGQ1cOk2XF0aNEzbcTAER1d3xSh+AswdkPjz7b7AkQTCg799kFXspui/D7c4eUXon++l322szrP8Tqb/MsLceMqFh28JoBAbzndlbxDgraO7Tb/39U5E2iceEumNNcrpilqulIPvwDVE3GQQ0RN3WyOWv2cJ4k2beIf00td1yGWEqd2X39Rfejq8Nrcl0lUYB07uazqyw47WExNGmtwrZC1vXMZzF3hm6CYeQ== MIME-Version: 1.0 X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2978; 2:V2rxBuByVChMSmh9UlQzzqZGnUpbJ6qOZtVWrW64yFdG09Knpbu99DhqOO2wqw1b8byBJvYHC27jeL0IBXpPkTlrsgqG9w+duzya2cTQ2IDgcNXTZnw/6CTqhgtboT7+Z94+Af18V2GG2OVZTq9y4rGIaMKJZeedKryzdOQ7nCc=; 3:Y+YnPRkIsPAaOBgwEO3YRD0esIFhpTpDGs1G3a83/S/doVGz1smShz9jV4ybzPY9smgbHJYepZPVHaz6FfmBKdA0MSHDkw7zXYeuEoZ3XUBLZyzLJIFsoomBygMyiNxEIn9ioIbeiQRSdsgoQ6kn8SplJDJV2dMDzVaManYCPlH4Z+iAqW1O6SmkDyA1Ii15n5f9puRE5zK/54GIZVMAVa7hBjCJ5yEjQsvuFf78H6KIYvozZEPsp0QYO2z4uywM1aoGUxgeydYigXQYuxPcuw== X-MS-Office365-Filtering-Correlation-Id: 4e7683f6-30bd-43c5-83c1-08d40a02b4d3 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001); SRVR:BN6PR03MB2978; X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2978; 25:/I8L5vL3GEHB+EaR0X+Ial5KHWq+2JeE8ae93hkNuVIHHR2fT5DsIiqer+YxKD2pCMEDE3hwze5/4VXyazfuJR0yBbYcDNsy1Y981E2hxVN/hfW8tgb9xk8pv9UsURErvxj56+25ncdFZVwTD0v70Gw7Slg1JJXoRliKnmlMu5D37ojs12sfAYt9vvek2OX+ClOwLKU2e0Ud/IZEgyohTGCUeNEZIh3ck6h9Ee+FHsAQO9xhvLhTjCj9dY0vsrS2nOtK570qFAmGjgTOzAnaCT7hREJm8ZXZmVZ9s9sg45jRb3OOCvLyS6haJtmvsBGwoBI/bdMoz93HkjFjfqyt/3BXU4JtAAQ0ZDb2e3DiXJCWhgN/byZ+0nhkgKMi8HLVwba/7QGcqjWuDlelLMkKzwrs+aBeJnuiEFbssW8obhenBYXFC2PGsks7c3Ha4HJHo2U1ilSVA8SxYvp9pALxhapq/KWMTPXFhUZKl1ts2Utv+nvfSfSgUqyj4L/fjRESxg4cRN4U5oX9mYlbv9Nb849T8fuRgNyR+EH656V0/G5TlMRXq903SPyIU5042OYH6yZp1RmcVkzchpzvz5WFzJZixafRQ9fBf+s+IIsXevMLLQvwB56tsQD++mUR3R4e0rGb3poWtG+gCTwdVSb/dI8GeA6lOk/uAHOyHsiQJew8j6OmnLVxuX0FA9aBGEwYC3UK15bXrSlMQTZcxjwvgiIwehMde2DDFPRmwNiJwevH/STmIx8UE3YSOOpt4CX/m5Z0CLoNDV62bcoVXqsjTI+hXFbLByut/qqLwVIH+P8= X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2978; 31:ZXtCl3o7gudu+zUhExCyTeCY0sre5GPqfj+4GMWIzcB7zMXiHvUpUsdGRfyOPaHIWnTWupEkf7KhkqKI+qj9fUxeGL3z+Ar6d1fnl12bsFoL6Df3oFPsP9S8x87e3HIqxwELtdoLfGL7OgA3ZZW7BLAumuB4Q/5aK6r0uqd45VwyeQzDiHAMQ2bTc2jibZaE/mE0NF/mg6vmzQ+fAJ483tDGfsDOqd3fNqmXlyIgySWIpR0sx5JyD6Z3X05T2Xem5OMdvg1rl4x0bN8fWHLqXVAohJbe5Orvj6KloYbA3PA=; 20:oCVBWO2vpbHyjmtNY/SZWdX8NmmqZn2YXO7Kl7FYipI0n1Wan1f2ZFZs7X144oy/oM0arnYB1sIq78nHq5i+8sS4D+B/YBf07sUBDe7MJhK7X7IA2aV1vnJQGK6ypEGimi//1ALq+EXIwJBRW4/BJAvvX8KVi1YNi5HCdnU9aDVgDimjY+SvsCfXksbQpFLDMyoX4wK84qSD5OFHv0LK61OJ9e5+NS4ENfIzeByBnfbGT9H+hYytR+f2t8aSa4shveemLZLtDt3M72nWnW7NpWUN0zDn7mu0E3pLLv3B+6L8OkkFy5tMaHCyzAR/gc1uiO/OOMDR7ZPEF0DeFacWHCWh/VgG5YllM7yhTuWkneYHk9urW7eB55gvMkVSu4u3fWivpDi7XV/7XXJdzUMOm1XkaskVdTzlqXU+8y36z71icmmnN7JCDEr1j08hJS9Fol7AVNno5EUQM3gYX2VIXpiGjxKW93w4ay3it/PpBjnNEZRTlUzRBPGrR1WctSjfPSmyIKI6nnp4678tRg2QjXEwXH8F797jlNfPnYC7vzSsGaWjIk2QLhXORYVs0FNxJ83pTyRwyseLjItFd8atx6hMnD6qfly9qYN+sw4xYjQ= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(6045074)(601004)(2401047)(13018025)(8121501046)(13017025)(13015025)(13024025)(13023025)(5005006)(3002001)(10201501046)(6055026)(6046074)(6072074); SRVR:BN6PR03MB2978; BCL:0; PCL:0; RULEID:; SRVR:BN6PR03MB2978; X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2978; 4:toiezcVhgwOR46rFWArXWuy5LI/MXARpedV0tYKEMPZk13RFF+52acblt2/FMGxSOBIxWkMEdidDXG6Q0ETPa4cQBojLKVE/BVtJJzvki+DK7yL4q+atcArL9r5tzduGxjFc/rtEvHmy+yvF7xPCYIovB2J+cQ8Vj1+kzEIphl4zWEHXNCNetyBGfrBzWxmfbetH08JfZ239cRWPryCBz3fmGSadnzzCM8dpEz6Pwo9wQHk0De1fE1L5LiBFpqiY9uDfDXfH1lqnmOV0t/ppz6vggM4ZtoKGZARvIPdZhQfaoY/ho2XSXKIZRpR9nk7J9kqzWjGVMRvdhQuFP4+nXb1vNAwQyNDlVIg+upOiThteSxHo9ftg9SumlqfBfDZLLIAB9vG7nCcUtBNWF7YYNiufB8JUTde5IgEIWRdPmbDgDm+C4Hs3bvn2JuWfJFg3KGrYWozJ7UENuETTTUBNLzCXQpPxFBHiOZfpfi68H/y1FbgffU7IrBL9/pCs+0GSS4AU3vDakxKtLlAZRAqlSRFtAvMhaK4CjMq49RhHoY4rjtTstDr+b91hzJ0rIyH+ X-Forefront-PRVS: 012349AD1C X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN6PR03MB2978; 23:NkPzNoBkn2MMeobNLoA5HvQEn5bC8p6NADxn516SS?= =?us-ascii?Q?YjvOp3tqTq1yDReJzHthxyFHXHcl6vK6sgW+5tgxmBHGm3MIYeFzLyuXzdsf?= =?us-ascii?Q?gHzydLa8DRUhBgv5UpH+WlXRJt51+Kse7f+ZvXKwle6Zb8oHdND1DKOAHf6G?= =?us-ascii?Q?yqXWEbNTzHxOZomgJqumGVvdlebpAhyBGh4F5Gx3j/PBS+KOg8M0tAvCEj2B?= =?us-ascii?Q?dpBPIlKL9QT1He1Gq18r/LSwF+pkBFOUOu/Dfd7vkcBewKLADraC4RETbohm?= =?us-ascii?Q?XfQ8i2ks+XnXSKC6r/WQy6+YwGEGCmyWnAZMjoQkjhgN9S9FfeEBDnMBaOr3?= =?us-ascii?Q?H5sdbAxuTA/eeOyMS0vOMOHndqwgY0FoxyZjARlR+ssgzd1I+cXKhthpbqIA?= =?us-ascii?Q?VYoKNh6/f+WO/fGMNjHcBpJCGqSY5UWt6LmdxGEnv00nHSoBsdS8DZQh8EY1?= =?us-ascii?Q?ke6OKAKXTJxmz8UwUO5/HHdeEVqiZflG61GcQ2I2X8opzhY0oqMPO5NscjUK?= =?us-ascii?Q?oB5CMReCZ9kCKThw2ADgvTRrxRUCbVKXkN/vIxdXqNVHMS2zcHssunkDcyxf?= =?us-ascii?Q?HCanTZqjINprMvPmiHUwBYh4T/riokBsNWmjBcRIwI4o6iOtCaRkofH2l1Vz?= =?us-ascii?Q?hgynzo1XQl6EpfHCVwUrSo9yAx+XzXFQQXJpcednsmyhgxx+KTpx2HiPGHBK?= =?us-ascii?Q?rAN80egrDJNsDnc4FhS78G0R6vtYqOppTMJEf3XbbgkJymmRsDq7CX31cTy9?= =?us-ascii?Q?8+WN1i9v+VblnnGS7YBUNrk1Cn43ySDkwUc3QNgsstJz/S388/ADOAOYOrCE?= =?us-ascii?Q?CdVHv4Mal/imFxmNNVESMzEMHTSEfnmP0NupV0ZG7u598fMi0MK+PsthSiZ0?= =?us-ascii?Q?EvvEap3l1kt3Y88GlWfSqYyjs0p0Vx7Yd7NOOVCchndrCvaorZN8OU/kVcR2?= =?us-ascii?Q?P3SCel7Op55Z9oC0FI7wKWuTsR1MfGqC0AUU0jPDen1sb7L2P+bNIW0eU2cf?= =?us-ascii?Q?Eei4sQ2Vg7LNNrrmszAUSHTIqOjp8ovNPdmRre5JjstB+YZif45I/dk6YDP7?= =?us-ascii?Q?fbtpiIqJ8lS2kFgxg6pVT0RsUAn?= X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2978; 6:+F10Rw3eDO6EH+G23qtr0+JTD+0aVbmIzcUah3kDrQoCfXsXcyjQZh8kSbqSCF9qIFEbDgdoGQfI96ctLSYV+AwZSbBOOaB7AifWuHV/e+NXUhDc1N5w2AEsBUysblMScEwT+CQvwH7rOkcBdZfRPgP1TyC+A+8uxgclFlethx/+v7xl8a9d1IUTMjLZfqrk5KH1NbaYS80ySWglGXvhdpkuGRhDPBn7FMuNhB0LSsUKmz0wIYK+TM1DJo9ySfYMv+Rf2foVaVwPAlCbbP+EZKmKrxOD/svcUiPsjK/RoUU9Sb+xVQ+DNrZpLL4/WSoMLA8g4YAnGyWw1btuy9tbME8iyZ99eVyEpvZFP5UHwySpJ1+plglETAGUnwSyk5vJ; 5:zCK8r3m7VZ4BNYwNRcKD5i4LLgAiJO1gWEhVUMfeWfZT8W56rnxC0YHMQ7zr7JjnI1NVUB+TdAsGrmAM2Ildwn8ubf1eZFxiQTelsvVTHkIbhvHXOZ67eJpA2fs+G7ixP4eJy8aeobdlTntgamHQgw==; 24:nnYhzF6VCqFg08v5UmOx7dFe1sKYra8A0f25bpJ3W4dToLwXEN7o9dXVOAxBEh25jE3WQAAa+exkY698xrbmzqBX4bYQwD6CMqLCMpGKf30= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2978; 7:dMe3pcyU1ffpMFhvFvERUbA3r+YtfAAMk7Sx9iocyibzOuW+cr0uhB6AkQmoxTpPSX8WMWgTjI4YjxdDXctEepHIMBFi+gD4xSvLcZqm/0EGCKUZmhbifr+BREd9ODppQH8cUKqnBruRUk3h37cKRJsevz4DpuMSwlNAz+KAGRH5UPowpmq10MKzIAQa+Zl2Yc7m0YBPvb2rXuFaBfhBwTps8/BYxtx4Og00L8ZnXCPUmPcu8GUf8XDWwVXAY/x3wO3UugE8wlqUA9Uww+s+a52HGW4Y+tKxGrdIRc+cl9CSHgQ/r4qZxk0vvQx00VHycjxrengonJprcFCfE6fAS2FVdU1xGC+F1YJdRhqIdUw= X-OriginatorOrg: freescale.onmicrosoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Nov 2016 07:16:49.7475 (UTC) X-MS-Exchange-CrossTenant-Id: 710a03f5-10f6-4d38-9ff4-a80b81da590d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=710a03f5-10f6-4d38-9ff4-a80b81da590d; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR03MB2978 Subject: [lng-odp] [RFC] example: traffic_mgmt: enhancement with multiple pktios X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Xuelin Shi This patch made the following changes: - receiving packets from multiple pktios other than generating packets - identifying TM user by ip with individual class of service - print the user service while starting the program - pirnt the packets counts every 10 seconds for each user Signed-off-by: Xuelin Shi --- example/traffic_mgmt/Makefile.am | 7 +- example/traffic_mgmt/odp_traffic_mgmt.c | 476 +++++++++--------- example/traffic_mgmt/odp_traffic_mgmt.h | 47 ++ example/traffic_mgmt/odp_traffic_pktio.c | 794 +++++++++++++++++++++++++++++++ 4 files changed, 1091 insertions(+), 233 deletions(-) create mode 100644 example/traffic_mgmt/odp_traffic_mgmt.h create mode 100644 example/traffic_mgmt/odp_traffic_pktio.c -- 1.8.3.1 diff --git a/example/traffic_mgmt/Makefile.am b/example/traffic_mgmt/Makefile.am index c8ff797..d2c7929 100644 --- a/example/traffic_mgmt/Makefile.am +++ b/example/traffic_mgmt/Makefile.am @@ -2,8 +2,9 @@ include $(top_srcdir)/example/Makefile.inc bin_PROGRAMS = odp_traffic_mgmt$(EXEEXT) odp_traffic_mgmt_LDFLAGS = $(AM_LDFLAGS) -static -odp_traffic_mgmt_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example +odp_traffic_mgmt_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example -I${top_srcdir}/test -noinst_HEADERS = $(top_srcdir)/example/example_debug.h +noinst_HEADERS = $(top_srcdir)/example/example_debug.h \ + $(top_srcdir)/example/traffic_mgmt/odp_traffic_mgmt.h -dist_odp_traffic_mgmt_SOURCES = odp_traffic_mgmt.c +dist_odp_traffic_mgmt_SOURCES = odp_traffic_mgmt.c odp_traffic_pktio.c diff --git a/example/traffic_mgmt/odp_traffic_mgmt.c b/example/traffic_mgmt/odp_traffic_mgmt.c index c4f5356..f0fe198 100644 --- a/example/traffic_mgmt/odp_traffic_mgmt.c +++ b/example/traffic_mgmt/odp_traffic_mgmt.c @@ -9,16 +9,22 @@ #define _GNU_SOURCE #include +#include #include +#include #include #include #include #include +#include +#include "odp_traffic_mgmt.h" + +#define TM_USER_IP_START 0x01010101 #define NUM_SVC_CLASSES 4 #define USERS_PER_SVC_CLASS 2 -#define APPS_PER_USER 2 -#define TM_QUEUES_PER_APP 2 +#define APPS_PER_USER 1 +#define TM_QUEUES_PER_APP 1 #define NUM_USERS (USERS_PER_SVC_CLASS * NUM_SVC_CLASSES) #define NUM_TM_QUEUES (NUM_USERS * APPS_PER_USER * TM_QUEUES_PER_APP) #define TM_QUEUES_PER_USER (TM_QUEUES_PER_APP * APPS_PER_USER) @@ -49,11 +55,6 @@ typedef struct { odp_tm_wred_t wred_profiles[ODP_NUM_PACKET_COLORS]; } profile_set_t; -static const odp_init_t ODP_INIT_PARAMS = { - .log_fn = odp_override_log, - .abort_fn = odp_override_abort -}; - static profile_params_set_t COMPANY_PROFILE_PARAMS = { .shaper_params = { .commit_bps = 50 * MBPS, .commit_burst = 1000000, @@ -161,8 +162,8 @@ static profile_params_set_t COS2_PROFILE_PARAMS = { }, .threshold_params = { - .max_pkts = 1000, .enable_max_pkts = TRUE, - .max_bytes = 100000, .enable_max_bytes = TRUE + .max_pkts = 1000, .enable_max_pkts = FALSE, + .max_bytes = 100000, .enable_max_bytes = FALSE }, .wred_params = { @@ -194,8 +195,8 @@ static profile_params_set_t COS3_PROFILE_PARAMS = { }, .threshold_params = { - .max_pkts = 400, .enable_max_pkts = TRUE, - .max_bytes = 60000, .enable_max_bytes = TRUE + .max_pkts = 400, .enable_max_pkts = FALSE, + .max_bytes = 60000, .enable_max_bytes = FALSE }, .wred_params = { @@ -226,19 +227,46 @@ static profile_set_t APP_PROFILE_SETS[NUM_SVC_CLASSES][APPS_PER_USER]; static odp_tm_t odp_tm_test; -static odp_pool_t odp_pool; - static odp_tm_queue_t queue_num_tbls[NUM_SVC_CLASSES][TM_QUEUES_PER_CLASS + 1]; static uint32_t next_queue_nums[NUM_SVC_CLASSES]; static uint8_t random_buf[RANDOM_BUF_LEN]; static uint32_t next_rand_byte; -static odp_atomic_u32_t atomic_pkts_into_tm; -static odp_atomic_u32_t atomic_pkts_from_tm; +typedef struct { + struct { + odp_atomic_u64_t tm_pkts_in; + odp_atomic_u64_t tm_bytes_in; + odp_atomic_u64_t tm_pkts_out; + odp_atomic_u64_t tm_bytes_out; + } s; /* statistics info for each user of tm */ + uint64_t max_pkts_limit; /* max packets allowed for this user */ + uint64_t max_bytes_limit; /* max bytes allowed for this user */ + odp_bool_t enable_pkts_limit; /* max packets limit is valid */ + odp_bool_t enable_bytes_limit; /* max bytes limit is valid */ + uint32_t ipaddr; /* user ipaddr */ + int svc; /* service class */ +} tm_user_t; + +/* statistics value info, used for param pass */ +typedef struct { + uint64_t pkts_in; + uint64_t pkts_out; + uint64_t bytes_in; + uint64_t bytes_out; +} stat_info_t; + +static tm_user_t tm_users[NUM_USERS]; + +static profile_params_set_t *cos_profile_params[NUM_SVC_CLASSES] = { + &COS0_PROFILE_PARAMS, + &COS1_PROFILE_PARAMS, + &COS2_PROFILE_PARAMS, + &COS3_PROFILE_PARAMS +}; -static uint32_t g_num_pkts_to_send = 1000; -static uint8_t g_print_tm_stats = TRUE; +static odp_atomic_u64_t atomic_pkts_into_tm; +static odp_atomic_u64_t atomic_pkts_from_tm; static void tester_egress_fcn(odp_packet_t odp_pkt); @@ -308,17 +336,15 @@ static uint32_t create_profile_set(profile_params_set_t *profile_params_set, return err_cnt; } -/* Returns the number of errors encountered. */ - static uint32_t init_profile_sets(void) { uint32_t class_shaper_scale, class_threshold_scale, user_shaper_scale; uint32_t user_threshold_scale, err_cnt, app_idx; - class_shaper_scale = TM_QUEUES_PER_CLASS / 2; - class_threshold_scale = TM_QUEUES_PER_CLASS; - user_shaper_scale = TM_QUEUES_PER_USER / 2; - user_threshold_scale = TM_QUEUES_PER_USER; + class_shaper_scale = 1; + class_threshold_scale = 1; + user_shaper_scale = 1; + user_threshold_scale = 1; err_cnt = 0; err_cnt += create_profile_set(&COMPANY_PROFILE_PARAMS, @@ -384,7 +410,7 @@ static int config_example_user(odp_tm_node_t cos_tm_node, profile_set_t *profile_set; uint32_t app_idx, queue_idx, svc_class_queue_num; char user_name[64]; - int rc; + int rc, numq = 0; profile_set = &USER_PROFILE_SETS[svc_class]; @@ -429,6 +455,7 @@ static int config_example_user(odp_tm_node_t cos_tm_node, if (rc < 0) return rc; + numq++; svc_class_queue_num = next_queue_nums[svc_class]++; queue_num_tbls[svc_class][svc_class_queue_num + 1] = tm_queue; @@ -440,11 +467,13 @@ static int config_example_user(odp_tm_node_t cos_tm_node, static int config_company_node(const char *company_name) { + profile_params_set_t *param; odp_tm_node_params_t tm_node_params; profile_set_t *profile_set; odp_tm_node_t company_tm_node, cos_tm_node; uint32_t cos_idx, user_idx; char cos_node_name[64]; + int idx = 0; profile_set = &COMPANY_PROFILE_SET; odp_tm_node_params_init(&tm_node_params); @@ -484,12 +513,29 @@ static int config_company_node(const char *company_name) &tm_node_params); odp_tm_node_connect(cos_tm_node, company_tm_node); - for (user_idx = 0; user_idx < USERS_PER_SVC_CLASS; user_idx++) + param = cos_profile_params[cos_idx]; + for (user_idx = 0; user_idx < USERS_PER_SVC_CLASS; user_idx++) { config_example_user(cos_tm_node, cos_idx, cos_idx * 256 + user_idx); + tm_users[idx].svc = cos_idx; + tm_users[idx].ipaddr = TM_USER_IP_START + idx; + + tm_users[idx].max_bytes_limit = + param->threshold_params.max_bytes; + tm_users[idx].max_pkts_limit = + param->threshold_params.max_pkts; + + if (param->threshold_params.enable_max_pkts) + tm_users[idx].enable_pkts_limit = TRUE; + + if (param->threshold_params.enable_max_bytes) + tm_users[idx].enable_bytes_limit = TRUE; + idx++; + } } odp_tm_node_connect(company_tm_node, ODP_TM_ROOT); + return 0; } @@ -500,6 +546,9 @@ static int create_and_config_tm(void) odp_tm_egress_t egress; uint32_t level, err_cnt; + if (NUM_USERS < NUM_SVC_CLASSES * USERS_PER_SVC_CLASS) + return 1; + odp_tm_requirements_init(&requirements); odp_tm_egress_init(&egress); @@ -532,20 +581,8 @@ static int create_and_config_tm(void) __func__, err_cnt); config_company_node("TestCompany"); - return err_cnt; -} - -static uint32_t random_8(void) -{ - uint32_t rand8; - - if (RANDOM_BUF_LEN <= next_rand_byte) { - odp_random_data(random_buf, RANDOM_BUF_LEN, 1); - next_rand_byte = 0; - } - rand8 = random_buf[next_rand_byte++]; - return rand8; + return err_cnt; } static uint32_t random_16(void) @@ -562,232 +599,211 @@ static uint32_t random_16(void) return (((uint16_t)byte1) << 8) | ((uint16_t)byte2); } -static uint32_t pkt_service_class(void) +static inline uint32_t pkt_to_user(odp_packet_t pkt) { - uint32_t rand8; - - /* Make most of the traffic use service class 3 to increase the amount - * of delayed traffic so as to stimulate more interesting behaviors. - */ - rand8 = random_8(); - switch (rand8) { - case 0 ... 24: return 0; - case 25 ... 49: return 1; - case 50 ... 150: return 2; - case 151 ... 255: return 3; - default: return 3; - } + odph_ipv4hdr_t *ip; + uint32_t idx; + + ip = odp_packet_l3_ptr(pkt, NULL); + idx = odp_be_to_cpu_32(ip->src_addr) - TM_USER_IP_START; + + return idx & (NUM_USERS - 1); } -static odp_packet_t make_odp_packet(uint16_t pkt_len) +/** + * find the packet from which user and eligible for which service. + * this should be a lookup table implementation. Here for simplicity, + * only check last byte of ip src addr to classify the users. + */ +static inline uint32_t pkt_service_class(odp_packet_t pkt) { - odp_packet_t odp_pkt; - uint8_t rand8a, rand8b, pkt_color, drop_eligible; - - rand8a = random_8(); - rand8b = random_8(); - pkt_color = (rand8a < 224) ? 0 : ((rand8a < 248) ? 1 : 2); - drop_eligible = (rand8b < 240) ? 1 : 0; - odp_pkt = odp_packet_alloc(odp_pool, pkt_len); - if (odp_pkt == ODP_PACKET_INVALID) { - printf("%s odp_packet_alloc failure *******\n", __func__); - return 0; - } + uint32_t idx; + tm_user_t *u; + + idx = pkt_to_user(pkt); + u = &tm_users[idx]; - odp_packet_color_set(odp_pkt, pkt_color); - odp_packet_drop_eligible_set(odp_pkt, drop_eligible); - odp_packet_shaper_len_adjust_set(odp_pkt, 24); - return odp_pkt; + return u->svc; } -void tester_egress_fcn(odp_packet_t odp_pkt ODP_UNUSED) +void tester_egress_fcn(odp_packet_t odp_pkt) { - odp_atomic_inc_u32(&atomic_pkts_from_tm); + tm_user_t *u; + uint32_t idx = pkt_to_user(odp_pkt); + + odp_atomic_inc_u64(&atomic_pkts_from_tm); + u = &tm_users[idx]; + odp_atomic_inc_u64(&u->s.tm_pkts_out); + odp_atomic_add_u64(&u->s.tm_bytes_out, odp_packet_len(odp_pkt)); + + /* not forwarding, need to free, otherwise packet pool will be full */ + odp_packet_free(odp_pkt); } -static int traffic_generator(uint32_t pkts_to_send) +int tm_send_packet(odp_packet_t pkt) { - odp_pool_param_t pool_params; odp_tm_queue_t tm_queue; - odp_packet_t pkt; - odp_bool_t tm_is_idle; - uint32_t svc_class, queue_num, pkt_len, pkts_into_tm; - uint32_t pkts_from_tm, pkt_cnt, millisecs, odp_tm_enq_errs; + uint32_t svc_class, queue_num; int rc; + tm_user_t *u; + uint32_t idx; + uint64_t bytes_in; - memset(&pool_params, 0, sizeof(odp_pool_param_t)); - pool_params.type = ODP_POOL_PACKET; - pool_params.pkt.num = pkts_to_send + 10; - pool_params.pkt.len = 1600; - pool_params.pkt.seg_len = 0; - pool_params.pkt.uarea_size = 0; - - odp_pool = odp_pool_create("MyPktPool", &pool_params); - odp_tm_enq_errs = 0; - - pkt_cnt = 0; - while (pkt_cnt < pkts_to_send) { - svc_class = pkt_service_class(); - queue_num = random_16() & (TM_QUEUES_PER_CLASS - 1); - tm_queue = queue_num_tbls[svc_class][queue_num + 1]; - pkt_len = ((uint32_t)((random_8() & 0x7F) + 2)) * 32; - pkt_len = MIN(pkt_len, 1500); - pkt = make_odp_packet(pkt_len); - - pkt_cnt++; - rc = odp_tm_enq(tm_queue, pkt); - if (rc < 0) { - odp_tm_enq_errs++; - continue; - } + idx = pkt_to_user(pkt); + u = &tm_users[idx]; - odp_atomic_inc_u32(&atomic_pkts_into_tm); - } + if (u->enable_pkts_limit && + u->max_pkts_limit <= odp_atomic_load_u64(&u->s.tm_pkts_in)) + return -1; - printf("%s odp_tm_enq_errs=%u\n", __func__, odp_tm_enq_errs); - - /* Wait until the main traffic mgmt worker thread is idle and has no - * outstanding events (i.e. no timers, empty work queue, etc), but - * not longer than 60 seconds. - */ - for (millisecs = 0; millisecs < 600000; millisecs++) { - usleep(100); - tm_is_idle = odp_tm_is_idle(odp_tm_test); - if (tm_is_idle) - break; - } + bytes_in = odp_atomic_load_u64(&u->s.tm_bytes_in); + if (u->enable_bytes_limit && + u->max_bytes_limit <= bytes_in + odp_packet_len(pkt)) + return -1; + + svc_class = pkt_service_class(pkt); + queue_num = random_16() & (TM_QUEUES_PER_CLASS - 1); + tm_queue = queue_num_tbls[svc_class][queue_num + 1]; - if (!tm_is_idle) - printf("%s WARNING stopped waiting for the TM system " - "to be IDLE!\n", __func__); - - /* Wait for up to 2 seconds for pkts_from_tm to match pkts_into_tm. */ - for (millisecs = 0; millisecs < 2000; millisecs++) { - usleep(1000); - pkts_into_tm = odp_atomic_load_u32(&atomic_pkts_into_tm); - pkts_from_tm = odp_atomic_load_u32(&atomic_pkts_from_tm); - if (pkts_into_tm <= pkts_from_tm) - break; + rc = odp_tm_enq(tm_queue, pkt); + if (rc > 0) { + odp_atomic_inc_u64(&atomic_pkts_into_tm); + odp_atomic_inc_u64(&u->s.tm_pkts_in); + odp_atomic_add_u64(&u->s.tm_bytes_in, odp_packet_len(pkt)); } - return 0; + return rc; } -static int process_cmd_line_options(uint32_t argc, char *argv[]) +int tm_config_and_init(void) { - uint32_t arg_idx; - char *arg; - - arg_idx = 1; - while (arg_idx < argc) { - arg = argv[arg_idx++]; - if (!arg) { - return -1; - } else if (arg[0] == '-') { - switch (arg[1]) { - case 'n': - if (argc <= arg_idx) - return -1; - g_num_pkts_to_send = - atoi(argv[arg_idx++]); - break; - - case 'q': - g_print_tm_stats = FALSE; - break; - - default: - printf("Unrecognized cmd line option '%s'\n", - arg); - return -1; - } - } else { - /* Currently all cmd line options are '-' flag based. */ - return -1; + int i, rc; + tm_user_t *u; + + rc = create_and_config_tm(); + if (!rc) { + odp_random_data(random_buf, RANDOM_BUF_LEN, 1); + next_rand_byte = 0; + + odp_atomic_init_u64(&atomic_pkts_into_tm, 0); + odp_atomic_init_u64(&atomic_pkts_from_tm, 0); + + for (i = 0; i < NUM_USERS; i++) { + u = &tm_users[i]; + odp_atomic_init_u64(&u->s.tm_pkts_in, 0); + odp_atomic_init_u64(&u->s.tm_pkts_out, 0); + odp_atomic_init_u64(&u->s.tm_bytes_in, 0); + odp_atomic_init_u64(&u->s.tm_bytes_out, 0); } } - return 0; + return rc; } -static void signal_handler(int signal) +static void tm_print_stat_impl(int interval, stat_info_t *prev) { - size_t num_stack_frames; - const char *signal_name; - void *bt_array[128]; - - switch (signal) { - case SIGILL: - signal_name = "SIGILL"; break; - case SIGFPE: - signal_name = "SIGFPE"; break; - case SIGSEGV: - signal_name = "SIGSEGV"; break; - case SIGTERM: - signal_name = "SIGTERM"; break; - case SIGBUS: - signal_name = "SIGBUS"; break; - default: - signal_name = "UNKNOWN"; break; + int i; + + printf("\nTM toal pkts_in=%" PRIu64 ", pkts_out=%" PRIu64 "\n", + odp_atomic_load_u64(&atomic_pkts_into_tm), + odp_atomic_load_u64(&atomic_pkts_from_tm)); + + printf("----------------------\n"); + for (i = 0; i < NUM_USERS; i++) { + uint64_t bps; + tm_user_t *u; + stat_info_t si, *prev_si; + + u = &tm_users[i]; + prev_si = &prev[i]; + si.pkts_in = odp_atomic_load_u64(&u->s.tm_pkts_in); + si.pkts_out = odp_atomic_load_u64(&u->s.tm_pkts_out); + si.bytes_in = odp_atomic_load_u64(&u->s.tm_bytes_in); + si.bytes_out = odp_atomic_load_u64(&u->s.tm_bytes_out); + bps = (si.bytes_out - prev_si->bytes_out) * 8 / interval; + *prev_si = si; + + printf("user %d: pkts_in=%" PRIu64 ", pkts_out=%" PRIu64 + ", bytes_in=%" PRIu64 ", bytes_out=%" PRIu64 + ", bps=%" PRIu64 "\n", i, si.pkts_in, si.pkts_out, + si.bytes_in, si.bytes_out, bps); } - num_stack_frames = backtrace(bt_array, 100); - printf("Received signal=%u (%s) exiting.", signal, signal_name); - backtrace_symbols_fd(bt_array, num_stack_frames, fileno(stderr)); - fflush(NULL); - sync(); - abort(); + printf("\n"); } -int main(int argc, char *argv[]) +void tm_print_stat(int duration, int interval) { - struct sigaction signal_action; - struct rlimit rlimit; - uint32_t pkts_into_tm, pkts_from_tm; - odp_instance_t instance; - int rc; - - memset(&signal_action, 0, sizeof(signal_action)); - signal_action.sa_handler = signal_handler; - sigfillset(&signal_action.sa_mask); - sigaction(SIGILL, &signal_action, NULL); - sigaction(SIGFPE, &signal_action, NULL); - sigaction(SIGSEGV, &signal_action, NULL); - sigaction(SIGTERM, &signal_action, NULL); - sigaction(SIGBUS, &signal_action, NULL); - - getrlimit(RLIMIT_CORE, &rlimit); - rlimit.rlim_cur = rlimit.rlim_max; - setrlimit(RLIMIT_CORE, &rlimit); - - rc = odp_init_global(&instance, &ODP_INIT_PARAMS, NULL); - if (rc != 0) { - printf("Error: odp_init_global() failed, rc = %d\n", rc); - abort(); + int i; + int elapsed = 0; + int loop_forever = (duration == 0); + stat_info_t prev[NUM_USERS]; + + for (i = 0; i < NUM_USERS; i++) { + tm_user_t *u; + stat_info_t *si; + + u = &tm_users[i]; + si = &prev[i]; + si->pkts_in = odp_atomic_load_u64(&u->s.tm_pkts_in); + si->pkts_out = odp_atomic_load_u64(&u->s.tm_pkts_out); + si->bytes_in = odp_atomic_load_u64(&u->s.tm_bytes_in); + si->bytes_out = odp_atomic_load_u64(&u->s.tm_bytes_out); } - rc = odp_init_local(instance, ODP_THREAD_CONTROL); - if (rc != 0) { - printf("Error: odp_init_local() failed, rc = %d\n", rc); - abort(); - } - - if (process_cmd_line_options(argc, argv) < 0) - return -1; - - create_and_config_tm(); - - odp_random_data(random_buf, RANDOM_BUF_LEN, 1); - next_rand_byte = 0; - - odp_atomic_init_u32(&atomic_pkts_into_tm, 0); - odp_atomic_init_u32(&atomic_pkts_from_tm, 0); - - traffic_generator(g_num_pkts_to_send); - pkts_into_tm = odp_atomic_load_u32(&atomic_pkts_into_tm); - pkts_from_tm = odp_atomic_load_u32(&atomic_pkts_from_tm); - printf("pkts_into_tm=%u pkts_from_tm=%u\n", pkts_into_tm, pkts_from_tm); + do { + sleep(interval); + tm_print_stat_impl(interval, &prev[0]); + elapsed += interval; + } while (loop_forever || (elapsed < duration)); +} - odp_tm_stats_print(odp_tm_test); - return 0; +void tm_print_user_cos(void) +{ + int i; + char buf[80]; + + printf("\nClass Of Service\n" + "----------------------\n"); + + for (i = 0; i < NUM_SVC_CLASSES; i++) { + profile_params_set_t *p; + char *b = buf; + int n; + + p = cos_profile_params[i]; + snprintf(buf, 32, "COS%d:", i); + printf("%-12s%-16scommit bps=%" PRIu64 ", burst=%d\n", + buf, "shaper: ", p->shaper_params.commit_bps, + p->shaper_params.commit_burst); + printf("%-28speak bps=%" PRIu64 ", burst=%d\n", "", + p->shaper_params.peak_bps, + p->shaper_params.peak_burst); + + n = snprintf(buf, 80, "%-12s%-16s", "", "threshold: "); + b = buf + n; + if (p->threshold_params.enable_max_pkts) { + n = snprintf(b, 80, "max pkts=%" PRIu64, + p->threshold_params.max_pkts); + b += n; + } + if (p->threshold_params.enable_max_bytes) { + n = snprintf(b, 80, ", bytes=%" PRIu64, + p->threshold_params.max_bytes); + b += n; + } + printf("%s\n", buf); + } + printf("\nTM Users\n" + "--%6s----%6s--------%3s--\n", "userid", "ipaddr", "cos"); + for (i = 0; i < NUM_USERS; i++) { + uint8_t *p; + tm_user_t *u; + + u = &tm_users[i]; + p = (uint8_t *)&u->ipaddr; + snprintf(buf, 16, "%d.%d.%d.%d", p[3], p[2], p[1], p[0]); + printf("%6d %10s %8d\n", i, buf, u->svc); + } + printf("\n"); } diff --git a/example/traffic_mgmt/odp_traffic_mgmt.h b/example/traffic_mgmt/odp_traffic_mgmt.h new file mode 100644 index 0000000..9081ca1 --- /dev/null +++ b/example/traffic_mgmt/odp_traffic_mgmt.h @@ -0,0 +1,47 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ODP_TRAFFIC_MODULE_H_ +#define _ODP_TRAFFIC_MODULE_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * Config and initialize the tm system. + * + * @return 0 if success else -1 + */ +int tm_config_and_init(void); + +/** + * Print tm user stastics for each interval seconds + * + * @param duration how many seconds this function will run + * @param interval how many seconds for each print + */ +void tm_print_stat(int duration, int interval); + +/** + * Print tm service information for a user + */ +void tm_print_user_cos(void); + +/** + * Send packets to traffic management system + * + * @param pkt the packet will be sent + * + * @return 1 if success else <= 0 + */ +int tm_send_packet(odp_packet_t pkt); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/example/traffic_mgmt/odp_traffic_pktio.c b/example/traffic_mgmt/odp_traffic_pktio.c new file mode 100644 index 0000000..a1ed45a --- /dev/null +++ b/example/traffic_mgmt/odp_traffic_pktio.c @@ -0,0 +1,794 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * @example odp_traffic_mgmt.c + */ + +/** enable strtok */ +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "odp_traffic_mgmt.h" + +/** @def MAX_WORKERS + * @brief Maximum number of worker threads + */ +#define MAX_WORKERS 32 + +/** @def SHM_PKT_POOL_SIZE + * @brief Size of the shared memory block + */ +#define SHM_PKT_POOL_SIZE 8192 + +/** @def SHM_PKT_POOL_BUF_SIZE + * @brief Buffer size of the packet pool buffer + */ +#define SHM_PKT_POOL_BUF_SIZE 1856 + +/** @def MAX_PKT_BURST + * @brief Maximum number of packet in a burst + */ +#define MAX_PKT_BURST 32 + +/** Maximum number of pktio queues per interface */ +#define MAX_QUEUES 32 + +/** Maximum number of pktio interfaces */ +#define MAX_PKTIOS 8 + +/** Default seconds to run */ +#define DEFAULT_RUN_SECONDS 60 + +/** Get rid of path in filename - only for unix-type paths using '/' */ +#define NO_PATH(file_name) (strrchr((file_name), '/') ? \ + strrchr((file_name), '/') + 1 : (file_name)) +/** + * Parsed command line application arguments + */ +typedef struct { + int cpu_count; + int duration; /**< Number of seconds to run */ + int if_count; /**< Number of interfaces to be used */ + int num_workers; /**< Number of worker threads */ + char **if_names; /**< Array of pointers to interface names */ + char *if_str; /**< Storage for interface names */ + int error_check; /**< Check packet errors */ +} appl_args_t; + +static int exit_threads; /**< Break workers loop if set to 1 */ + +static const odp_init_t ODP_INIT_PARAMS = { + .log_fn = odp_override_log, + .abort_fn = odp_override_abort +}; + +/** + * Statistics + */ +typedef union { + struct { + /** Number of packets received */ + odp_atomic_u64_t packets; + odp_atomic_u64_t bytes; + /** Packets dropped due to receive error */ + odp_atomic_u64_t rx_drops; + odp_atomic_u64_t rx_drop_bytes; + /** Packets dropped due to enqueue traffic management error */ + odp_atomic_u64_t tm_drops; + odp_atomic_u64_t tm_drop_bytes; + } s; + + uint8_t padding[ODP_CACHE_LINE_SIZE]; +} stats_t ODP_ALIGNED_CACHE; + +/** + * Thread specific arguments + */ +typedef struct thread_args_t { + uint64_t pkts; + + int thr_idx; + int num_pktio; + + struct { + odp_pktio_t rx_pktio; + odp_pktin_queue_t pktin; + odp_queue_t rx_queue; + int rx_idx; + int rx_queue_idx; + } pktio[MAX_PKTIOS]; +} thread_args_t; + +/** + * Grouping of all global data + */ +typedef struct { + /** Application (parsed) arguments */ + appl_args_t appl; + /** Thread specific arguments */ + thread_args_t thread[MAX_WORKERS]; + + /** Table of pktio handles */ + struct { + odp_pktio_t pktio; + odp_pktin_queue_t pktin[MAX_QUEUES]; + odp_queue_t rx_q[MAX_QUEUES]; + int num_rx_thr; + int num_rx_queue; + int next_rx_queue; + } pktios[MAX_PKTIOS]; +} args_t; + +/** Global pointer to args */ +static args_t *gbl_args; +/** Global barrier to synchronize main and workers */ +static odp_barrier_t barrier; + +/** + * Drop packets which input parsing marked as containing errors. + * + * Frees packets with error and modifies pkt_tbl[] to only contain packets with + * no detected errors. + * + * @param pkt_tbl Array of packets + * @param num Number of packets in pkt_tbl[] + * + * @return Number of packets dropped + */ +static inline int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned num) +{ + odp_packet_t pkt; + unsigned dropped = 0; + unsigned i, j; + + for (i = 0, j = 0; i < num; ++i) { + pkt = pkt_tbl[i]; + + if (odp_unlikely(odp_packet_has_error(pkt))) { + odp_packet_free(pkt); /* Drop */ + dropped++; + } else if (odp_unlikely(i != j++)) { + pkt_tbl[j - 1] = pkt; + } + } + + return dropped; +} + +/** + * Packet IO worker thread accessing IO resources directly + * + * @param arg thread arguments of type 'thread_args_t *' + */ +static int run_worker_direct_mode(void *arg) +{ + int thr; + int pkts, i; + odp_packet_t pkt_tbl[MAX_PKT_BURST]; + int num_pktio; + odp_pktin_queue_t pktin; + int pktio = 0; + thread_args_t *thr_args = arg; + + thr = odp_thread_id(); + + num_pktio = thr_args->num_pktio; + pktin = thr_args->pktio[pktio].pktin; + + printf("[%02i] num pktios %i, PKTIN_DIRECT\n", thr, num_pktio); + + /* Loop packets */ + while (!exit_threads) { + if (num_pktio > 1) { + pktin = thr_args->pktio[pktio].pktin; + pktio++; + if (pktio == num_pktio) + pktio = 0; + } + + pkts = odp_pktin_recv(pktin, pkt_tbl, MAX_PKT_BURST); + if (odp_unlikely(pkts <= 0)) + continue; + + if (gbl_args->appl.error_check) { + int rx_drops; + + /* Drop packets with errors */ + rx_drops = drop_err_pkts(pkt_tbl, pkts); + + if (odp_unlikely(rx_drops)) { + if (pkts == rx_drops) + continue; + + pkts -= rx_drops; + } + } + + for (i = 0; i < pkts; i++) { + /* try to send packets to Traffic Management System */ + if (tm_send_packet(pkt_tbl[i]) <= 0) + odp_packet_free(pkt_tbl[i]); + } + } + + /* Make sure that latest stat writes are visible to other threads */ + odp_mb_full(); + + return 0; +} + +/** + * Create a pktio handle, optionally associating a default input queue. + * + * @param dev Name of device to open + * @param index Pktio index + * @param pool Pool to associate with device for packet RX/TX + * + * @retval 0 on success + * @retval -1 on failure + */ +static int create_pktio(const char *dev, int idx, int num_rx, odp_pool_t pool) +{ + odp_pktio_t pktio; + odp_pktio_param_t pktio_param; + odp_pktio_capability_t capa; + odp_pktin_queue_param_t pktin_param; + odp_pktio_op_mode_t mode_rx; + + odp_pktio_param_init(&pktio_param); + + pktio = odp_pktio_open(dev, pool, &pktio_param); + if (pktio == ODP_PKTIO_INVALID) { + LOG_ERR("Error: failed to open %s\n", dev); + return -1; + } + + printf("created pktio %" PRIu64 " (%s)\n", + odp_pktio_to_u64(pktio), dev); + + if (odp_pktio_capability(pktio, &capa)) { + LOG_ERR("Error: capability query failed %s\n", dev); + return -1; + } + + odp_pktin_queue_param_init(&pktin_param); + + mode_rx = ODP_PKTIO_OP_MT_UNSAFE; + + if (num_rx > (int)capa.max_input_queues) { + printf("Sharing %i input queues between %i workers\n", + capa.max_input_queues, num_rx); + num_rx = capa.max_input_queues; + mode_rx = ODP_PKTIO_OP_MT; + } + + pktin_param.hash_enable = 1; + pktin_param.hash_proto.proto.ipv4_udp = 1; + pktin_param.num_queues = num_rx; + pktin_param.op_mode = mode_rx; + + if (odp_pktin_queue_config(pktio, &pktin_param)) { + LOG_ERR("Error: input queue config failed %s\n", dev); + return -1; + } + + if (odp_pktin_queue(pktio, gbl_args->pktios[idx].pktin, + num_rx) != num_rx) { + LOG_ERR("Error: pktin queue query failed %s\n", dev); + return -1; + } + + printf("created %i input on (%s)\n", num_rx, dev); + + gbl_args->pktios[idx].num_rx_queue = num_rx; + gbl_args->pktios[idx].pktio = pktio; + + return 0; +} + +/* + * Bind worker threads to interfaces and calculate number of queues needed + * + * less workers (N) than interfaces (M) + * - assign each worker to process every Nth interface + * - workers process inequal number of interfaces, when M is not divisible by N + * - needs only single queue per interface + * otherwise + * - assign an interface to every Mth worker + * - interfaces are processed by inequal number of workers, when N is not + * divisible by M + * - tries to configure a queue per worker per interface + * - shares queues, if interface capability does not allows a queue per worker + */ +static void bind_workers(void) +{ + int if_count, num_workers; + int rx_idx, thr, pktio; + thread_args_t *thr_args; + + if_count = gbl_args->appl.if_count; + num_workers = gbl_args->appl.num_workers; + + if (if_count > num_workers) { + thr = 0; + + for (rx_idx = 0; rx_idx < if_count; rx_idx++) { + thr_args = &gbl_args->thread[thr]; + pktio = thr_args->num_pktio; + thr_args->pktio[pktio].rx_idx = rx_idx; + thr_args->num_pktio++; + + gbl_args->pktios[rx_idx].num_rx_thr++; + + thr++; + if (thr >= num_workers) + thr = 0; + } + } else { + rx_idx = 0; + + for (thr = 0; thr < num_workers; thr++) { + thr_args = &gbl_args->thread[thr]; + pktio = thr_args->num_pktio; + thr_args->pktio[pktio].rx_idx = rx_idx; + thr_args->num_pktio++; + + gbl_args->pktios[rx_idx].num_rx_thr++; + + rx_idx++; + if (rx_idx >= if_count) + rx_idx = 0; + } + } +} + +/* + * Bind queues to threads and fill in missing thread arguments (handles) + */ +static void bind_queues(void) +{ + int num_workers; + int thr, pktio; + + num_workers = gbl_args->appl.num_workers; + + for (thr = 0; thr < num_workers; thr++) { + int rx_idx; + thread_args_t *thr_args = &gbl_args->thread[thr]; + int num = thr_args->num_pktio; + + for (pktio = 0; pktio < num; pktio++) { + int rx_queue; + + rx_idx = thr_args->pktio[pktio].rx_idx; + rx_queue = gbl_args->pktios[rx_idx].next_rx_queue; + + thr_args->pktio[pktio].rx_queue_idx = rx_queue; + thr_args->pktio[pktio].pktin = + gbl_args->pktios[rx_idx].pktin[rx_queue]; + thr_args->pktio[pktio].rx_queue = + gbl_args->pktios[rx_idx].rx_q[rx_queue]; + thr_args->pktio[pktio].rx_pktio = + gbl_args->pktios[rx_idx].pktio; + + rx_queue++; + + if (rx_queue >= gbl_args->pktios[rx_idx].num_rx_queue) + rx_queue = 0; + + gbl_args->pktios[rx_idx].next_rx_queue = rx_queue; + } + } +} + +/** + * Prinf usage information + */ +static void usage(char *progname) +{ + printf("\n" + "OpenDataPlane traffic management application.\n" + "\n" + "Usage: %s OPTIONS\n" + " E.g. %s -i eth0,eth1 -c 2\n" + " In the above example,\n" + " two threads will be used for receiving pkts from eth0 and eth1\n" + "\n" + "Mandatory OPTIONS:\n" + " -i, --interface Eth interfaces (comma-separated, no spaces)\n" + " Interface count min 1, max %i\n" + "\n" + "Optional OPTIONS:\n" + " -c, --count CPU count.\n" + " -t, --time seconds to run.\n" + " -e, --error_check 0: Don't check packet errors (default)\n" + " 1: Check packet errors\n" + " -h, --help Display help and exit.\n\n" + "\n", NO_PATH(progname), NO_PATH(progname), MAX_PKTIOS + ); +} + +/** + * Parse and store the command line arguments + * + * @param argc argument count + * @param argv[] argument vector + * @param appl_args Store application arguments here + */ +static void parse_args(int argc, char *argv[], appl_args_t *appl_args) +{ + int opt; + int long_index; + char *token; + size_t len; + int i; + static const struct option longopts[] = { + {"count", required_argument, NULL, 'c'}, + {"time", required_argument, NULL, 't'}, + {"interface", required_argument, NULL, 'i'}, + {"error_check", required_argument, NULL, 'e'}, + {"help", no_argument, NULL, 'h'}, + {NULL, 0, NULL, 0} + }; + + static const char *shortopts = "+c:t:i:e:h"; + + /* let helper collect its own arguments (e.g. --odph_proc) */ + odph_parse_options(argc, argv, shortopts, longopts); + + appl_args->error_check = 0; /* don't check packet errors by default */ + + opterr = 0; /* do not issue errors on helper options */ + + while (1) { + opt = getopt_long(argc, argv, shortopts, longopts, &long_index); + + if (opt == -1) + break; /* No more options */ + + switch (opt) { + case 'c': + appl_args->cpu_count = atoi(optarg); + break; + case 't': + appl_args->duration = atoi(optarg); + break; + case 'i': + len = strlen(optarg); + if (len == 0) { + usage(argv[0]); + exit(EXIT_FAILURE); + } + len += 1; /* add room for '\0' */ + + appl_args->if_str = malloc(len); + if (appl_args->if_str == NULL) { + usage(argv[0]); + exit(EXIT_FAILURE); + } + + /* count the number of tokens separated by ',' */ + strcpy(appl_args->if_str, optarg); + for (token = strtok(appl_args->if_str, ","), i = 0; + token != NULL; + token = strtok(NULL, ","), i++) + ; + + appl_args->if_count = i; + + if (appl_args->if_count < 1 || + appl_args->if_count > MAX_PKTIOS) { + usage(argv[0]); + exit(EXIT_FAILURE); + } + + /* allocate storage for the if names */ + appl_args->if_names = + calloc(appl_args->if_count, sizeof(char *)); + + /* store the if names (reset names string) */ + strcpy(appl_args->if_str, optarg); + for (token = strtok(appl_args->if_str, ","), i = 0; + token != NULL; token = strtok(NULL, ","), i++) { + appl_args->if_names[i] = token; + } + break; + case 'e': + appl_args->error_check = atoi(optarg); + break; + case 'h': + usage(argv[0]); + exit(EXIT_SUCCESS); + break; + default: + break; + } + } + + if (appl_args->if_count == 0) { + usage(argv[0]); + exit(EXIT_FAILURE); + } + + optind = 1; /* reset 'extern optind' from the getopt lib */ +} + +/** + * Print system and application info + */ +static void print_info(char *progname, appl_args_t *appl_args) +{ + int i; + + printf("\n" + "ODP system info\n" + "---------------\n" + "ODP API version: %s\n" + "ODP impl name: %s\n" + "CPU model: %s\n" + "CPU freq (hz): %" PRIu64 "\n" + "Cache line size: %i\n" + "CPU count: %i\n" + "\n", + odp_version_api_str(), odp_version_impl_name(), + odp_cpu_model_str(), odp_cpu_hz_max(), + odp_sys_cache_line_size(), odp_cpu_count()); + + printf("Running ODP appl: \"%s\"\n" + "-----------------\n" + "IF-count: %i\n" + "Using IFs: ", + progname, appl_args->if_count); + for (i = 0; i < appl_args->if_count; ++i) + printf(" %s", appl_args->if_names[i]); + printf("\n" + "Mode: PKTIN_DIRECT, "); + + printf("\n\n"); + fflush(NULL); +} + +static void gbl_args_init(args_t *args) +{ + int pktio, queue; + + memset(args, 0, sizeof(args_t)); + + for (pktio = 0; pktio < MAX_PKTIOS; pktio++) { + args->pktios[pktio].pktio = ODP_PKTIO_INVALID; + + for (queue = 0; queue < MAX_QUEUES; queue++) + args->pktios[pktio].rx_q[queue] = ODP_QUEUE_INVALID; + } +} + +static void signal_handler(int signal) +{ + size_t num_stack_frames; + const char *signal_name; + void *bt_array[128]; + + switch (signal) { + case SIGILL: + signal_name = "SIGILL"; break; + case SIGFPE: + signal_name = "SIGFPE"; break; + case SIGSEGV: + signal_name = "SIGSEGV"; break; + case SIGTERM: + signal_name = "SIGTERM"; break; + case SIGBUS: + signal_name = "SIGBUS"; break; + default: + signal_name = "UNKNOWN"; break; + } + + num_stack_frames = backtrace(bt_array, 100); + printf("Received signal=%u (%s) exiting.", signal, signal_name); + backtrace_symbols_fd(bt_array, num_stack_frames, fileno(stderr)); + fflush(NULL); + sync(); + abort(); +} + +int main(int argc, char *argv[]) +{ + odph_odpthread_t thread_tbl[MAX_WORKERS]; + odp_pool_t pool; + int i; + int cpu; + int num_workers; + odp_shm_t shm; + odp_cpumask_t cpumask; + char cpumaskstr[ODP_CPUMASK_STR_SIZE]; + odp_pool_param_t params; + int ret; + int if_count; + int duration; + int (*thr_run_func)(void *); + odp_instance_t instance; + struct sigaction signal_action; + + memset(&signal_action, 0, sizeof(signal_action)); + signal_action.sa_handler = signal_handler; + sigfillset(&signal_action.sa_mask); + sigaction(SIGILL, &signal_action, NULL); + sigaction(SIGFPE, &signal_action, NULL); + sigaction(SIGSEGV, &signal_action, NULL); + sigaction(SIGTERM, &signal_action, NULL); + sigaction(SIGBUS, &signal_action, NULL); + + /* Init ODP before calling anything else */ + if (odp_init_global(&instance, NULL, NULL)) { + LOG_ERR("Error: ODP global init failed.\n"); + exit(EXIT_FAILURE); + } + + /* Init this thread */ + if (odp_init_local(instance, ODP_THREAD_CONTROL)) { + LOG_ERR("Error: ODP local init failed.\n"); + exit(EXIT_FAILURE); + } + + /* Reserve memory for args from shared mem */ + shm = odp_shm_reserve("shm_args", sizeof(args_t), + ODP_CACHE_LINE_SIZE, 0); + gbl_args = odp_shm_addr(shm); + + if (gbl_args == NULL) { + LOG_ERR("Error: shared mem alloc failed.\n"); + exit(EXIT_FAILURE); + } + gbl_args_init(gbl_args); + + /* Parse and store the application arguments */ + parse_args(argc, argv, &gbl_args->appl); + + /* Print both system and application information */ + print_info(NO_PATH(argv[0]), &gbl_args->appl); + + /* Default to system CPU count unless user specified */ + num_workers = MAX_WORKERS; + if (gbl_args->appl.cpu_count) + num_workers = gbl_args->appl.cpu_count; + + /* Get default worker cpumask */ + num_workers = odp_cpumask_default_worker(&cpumask, num_workers); + (void)odp_cpumask_to_str(&cpumask, cpumaskstr, sizeof(cpumaskstr)); + + gbl_args->appl.num_workers = num_workers; + + for (i = 0; i < num_workers; i++) + gbl_args->thread[i].thr_idx = i; + + if_count = gbl_args->appl.if_count; + + printf("num worker threads: %i\n", num_workers); + printf("first CPU: %i\n", odp_cpumask_first(&cpumask)); + printf("cpu mask: %s\n", cpumaskstr); + + /* Create packet pool */ + odp_pool_param_init(¶ms); + params.pkt.seg_len = SHM_PKT_POOL_BUF_SIZE; + params.pkt.len = SHM_PKT_POOL_BUF_SIZE; + params.pkt.num = SHM_PKT_POOL_SIZE; + params.type = ODP_POOL_PACKET; + + pool = odp_pool_create("packet pool", ¶ms); + + if (pool == ODP_POOL_INVALID) { + LOG_ERR("Error: packet pool create failed.\n"); + exit(EXIT_FAILURE); + } + odp_pool_print(pool); + + bind_workers(); + + for (i = 0; i < if_count; ++i) { + const char *dev = gbl_args->appl.if_names[i]; + int num_rx; + + /* A queue per assigned worker */ + num_rx = gbl_args->pktios[i].num_rx_thr; + + if (create_pktio(dev, i, num_rx, pool)) + exit(EXIT_FAILURE); + } + + gbl_args->pktios[i].pktio = ODP_PKTIO_INVALID; + + bind_queues(); + + if (tm_config_and_init()) { + LOG_ERR("Error: tm system initialization failed.\n"); + exit(EXIT_FAILURE); + } + tm_print_user_cos(); + + memset(thread_tbl, 0, sizeof(thread_tbl)); + + odp_barrier_init(&barrier, num_workers + 1); + + thr_run_func = run_worker_direct_mode; + + /* Create worker threads */ + cpu = odp_cpumask_first(&cpumask); + for (i = 0; i < num_workers; ++i) { + odp_cpumask_t thd_mask; + odph_odpthread_params_t thr_params; + + memset(&thr_params, 0, sizeof(thr_params)); + thr_params.start = thr_run_func; + thr_params.arg = &gbl_args->thread[i]; + thr_params.thr_type = ODP_THREAD_WORKER; + thr_params.instance = instance; + + odp_cpumask_zero(&thd_mask); + odp_cpumask_set(&thd_mask, cpu); + odph_odpthreads_create(&thread_tbl[i], &thd_mask, + &thr_params); + cpu = odp_cpumask_next(&cpumask, cpu); + } + + /* Start packet receive and transmit */ + for (i = 0; i < if_count; ++i) { + odp_pktio_t pktio; + uint8_t mac[ODPH_ETHADDR_LEN]; + char buf[32]; + const char *dev; + + pktio = gbl_args->pktios[i].pktio; + ret = odp_pktio_start(pktio); + if (ret) { + LOG_ERR("Error: unable to start %s\n", + gbl_args->appl.if_names[i]); + exit(EXIT_FAILURE); + } else { + dev = gbl_args->appl.if_names[i]; + odp_pktio_mac_addr(pktio, mac, ODPH_ETHADDR_LEN); + sprintf(buf, "%02x:%02x:%02x:%02x:%02x:%02x", + mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]); + printf("start pktio: %s, mac %s\n", dev, buf); + } + } + + /* Print packets count every 10 seconds */ + duration = gbl_args->appl.duration; + if (duration < 10) + duration = DEFAULT_RUN_SECONDS; + tm_print_stat(duration, 10); + exit_threads = 1; + + /* Master thread waits for other threads to exit */ + for (i = 0; i < num_workers; ++i) + odph_odpthreads_join(&thread_tbl[i]); + + free(gbl_args->appl.if_names); + free(gbl_args->appl.if_str); + + printf("Exit %d\n\n", ret); + return ret; +}