From patchwork Wed Aug 31 01:54:15 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Forrest Shi X-Patchwork-Id: 75032 Delivered-To: patch@linaro.org Received: by 10.140.29.52 with SMTP id a49csp107969qga; Tue, 30 Aug 2016 20:02:18 -0700 (PDT) X-Received: by 10.55.95.2 with SMTP id t2mr8820821qkb.144.1472612538624; Tue, 30 Aug 2016 20:02:18 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id q29si1533047qte.113.2016.08.30.20.02.17; Tue, 30 Aug 2016 20:02:18 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=pass (p=NONE dis=NONE) header.from=linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 76754609BD; Wed, 31 Aug 2016 03:02:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAD_ENC_HEADER,BAYES_00, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 03BA7615FC; Wed, 31 Aug 2016 03:01:14 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 7E0B761600; Wed, 31 Aug 2016 03:01:07 +0000 (UTC) Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0121.outbound.protection.outlook.com [104.47.40.121]) by lists.linaro.org (Postfix) with ESMTPS id C1B85607E9 for ; Wed, 31 Aug 2016 02:57:25 +0000 (UTC) Received: from BN6PR03CA0069.namprd03.prod.outlook.com (10.173.137.31) by BN6PR03MB2420.namprd03.prod.outlook.com (10.168.223.10) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.599.9; Wed, 31 Aug 2016 02:57:22 +0000 Received: from BN1BFFO11FD021.protection.gbl (2a01:111:f400:7c10::1:146) by BN6PR03CA0069.outlook.office365.com (2603:10b6:404:4c::31) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.599.9 via Frontend Transport; Wed, 31 Aug 2016 02:57:22 +0000 Received-SPF: Neutral (protection.outlook.com: 192.88.168.50 is neither permitted nor denied by domain of freescale.com) Received: from tx30smr01.am.freescale.net (192.88.168.50) by BN1BFFO11FD021.mail.protection.outlook.com (10.58.144.84) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.587.6 via Frontend Transport; Wed, 31 Aug 2016 02:57:23 +0000 Received: from localhost (rock.ap.freescale.net [10.193.20.106]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id u7V2vLmu027325; Tue, 30 Aug 2016 19:57:22 -0700 From: To: , Date: Wed, 31 Aug 2016 09:54:15 +0800 Message-ID: <1472608455-31488-3-git-send-email-forrest.shi@linaro.org> X-Mailer: git-send-email 1.8.4 In-Reply-To: <1472608455-31488-1-git-send-email-forrest.shi@linaro.org> References: <1472608455-31488-1-git-send-email-forrest.shi@linaro.org> X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:192.88.168.50; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(7916002)(2980300002)(199003)(189002)(57986006)(87936001)(86152002)(356003)(5660300001)(81156014)(47776003)(76176999)(5001770100001)(68736007)(81166006)(626004)(36756003)(305945005)(77096005)(104016004)(106466001)(8936002)(8676002)(2950100001)(50986999)(19580395003)(586003)(2876002)(48376002)(50466002)(19580405001)(11100500001)(33646002)(76506005)(92566002)(229853001)(189998001)(4326007)(5003940100001)(97736004)(2906002)(105606002)(7846002)(50226002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN6PR03MB2420; H:tx30smr01.am.freescale.net; FPR:; SPF:Neutral; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BN1BFFO11FD021; 1:fJDloUiQcf799IKeUgFuUlHfBahMHG9swnXDtxWUfaY+qRANe3P7/L6cFjt72Qolhtn9GwL+KLnEt8cutFRuFvKk+j1q/AQFwcLzadeDY9jh/iSOGvTX8ynfC851vfoarpTE4SiJNtK7eRdFCGKl5cOLjxQYBjEzypHzlmSomFDi5V5hEFiexfyym2hH+5uMhEg6tMILvNEgFM9qwrluLlu20at3zuf+ntoMOIId03x6ZM48r3rBveMX2GgUep6NOW5eC67K4LdNIsXo1pYTaUx7F/J2uMJkhjKSpjeJTEFZq6nrdSszGQrInMf+4BZX1bIr++7vaDZsmAdXenfuKe4njZzvSTpJk9dRzjWoJf0C4nvPsbW81qDIzrGHzQPgRGcGn4cfWG/LQ9wFUK+f+drUKjN9a4k6d8IjkspLUihCdtfITFd/xGrINR+ZHi38mJegYC0dVURkmp9ZAQCZfqvQYu3xGcSKn17b6qp1TYwUGN5D8wENYPT4ck5D6/hbvgPG8OI0TWvGTeA53sE/7eWJ4kvWdQmeeHfuvjWj8m9vNOVGNU7g96KMEQFEpZ2Qd7hdRuIyMzJnMAt/34CXIQ== MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: c9c1716c-80be-4371-e640-08d3d14a88b0 X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2420; 2:1Hih8SdwnrJjQieWLIUgcQkCCPjCcqgclQL1qhWAgPVzdWWiV93sl1SF/rO2zcRYTBaii7wypmMYUAO8iZ50ssRbDp9cMoxOlPpPvhR92X495IkhSFBmFPq2c/UPvWRToBJlGQRhxrE5dKipwVG13A4kMHlUUV+NZiCYZQaIfysznhRdP8+qJEjmgWqkMUIQ; 3:UbA3VEsH2UtnujpsU2hJnUj3dtRc0EwVFG+zgIAaDQj0v647IysA2oRH3y9o+2F2ShKcVhxuvYU/ojquO8Ab/8obwhD9IMhSBUntYM+b/AoHI1I0l+VyJ17jobrJUAHeQtz1ZetXyioxTn+v4JgAa1yYkdGKVxt49P8iVIFKKLZj90SgHWILPWyJljrv/5jk2Hf/oWsMlAhbiuTykADrORgnHYQguafFStvQWZtI9vA= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN6PR03MB2420; X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2420; 25:exmR5K1SG6WwWS1CDy0HpAZ68lzEXfV4U7ruMZHZlC5SRu6spWoxO3E1tH8EihJGoh9+QBfEROVsyfnP9g5PRoxEr4z+UpwLTwwFihemyN4eiKijfRp6uczc2/eMIn4IF8uEDBFhP0cAl50ihYjls8AzOMa7Z7JpPpTtshjdSTtXGP/AQt019+IqTLE+i9JqWtYwdLGEl5wJeDAuis2KNH0Hb1HHvnQsUrhrn0W2VwXteSVklAHXy3xkZ2v4MQLWPXA61FGMAws/exVfIcnkYl/ZlzdQcWdvBE60pRFaB4+tv8vMV+bw/CTbvUtRuKj70IZNrBshCV+w+e9h3bhGFMd6SRA8CuMDyNJmpp9EIQ8dc3yhFzjf3oqAMSP0Rzk2y5+BJ3URXwCEg6Xhgs0CXW7vfKXZBG16CDU5QZmijBIQB9aM/9d9v44B4w2RpvJD7869xIRLzA92FkjF0Aq+1OTjFNSPBVxDcsE+gjjKNeojUoWEgaRypp3BFiAQS1NAe+lowmrbxUNH3j9GKpsuFZMRs76GfMeYA+eNIimJe3qOJRKgXTNis7TaE0JoPJ5l7vUhqFJdutudCkn5CeydRzfx4R4rSHYuQu/MRUJlCLxB9xIKFu0l5hwVZe+VwPUEZdZs6mTAZtgfFwiL3niJasGXZfm/0ysNIo3kRUsvrAoRYbiXo+2zFUixABwoCD9FP2qOxw4kb4s32fBhI5YSCdJl86aawPfU2nOWxWQCiUngZdznOl2OY5/tGdHeiNWkK/KbKuFevOQ0Khx046j+RQ== X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2420; 31:yublNV6UxyCUrsLNfJXkIJvCtW15J+4O9ARNMGgP8b1B+p7LnAPZY9GsivEY7Pe+aoqgYeQgfWhfaefY5Q2i2UBNy5VM3U8nnwcHmY+KXzNELmTn+ARcqRblvNIUn9JYnGMr9PNCRcBT/KOwjpSMu6bWuVGdpj1/UZu+aj1xSX80eN0wceTH6mwuZLB1n7K2Skk3/xBXolLOZOYG8iBGCjYsJCBo4LbmV4qR1jindiw=; 20:ri+sQSKU4Y26pogg3JQQiw6Lg0hvfYDcvrWoj61hUy/prF6GOIx9gn0B7RMK/ZdN4tdtK4eNvldYC+UZCglo2R8DXt2KFTQUG4oG1Mwus0kT/ha/TJeHSHJvXQIgat/eV7z2E+cNhh+J2qP1ME72G2LGZUCQqMZ6RNbx4M8mtHhLUZv1Yzj97I+lWWhHiEYHoBoD7gsNJ+LkIryLSb7wMuO9eaojbAriV8RvvV3pMzK7m/YU6KOZftmijwyNUtmVpRtNQtRSqRecDx8hIbiVBkjGJoeW5EHdXv1P3ZpDa3U8BfV2bTc6WhEqnxEC3cZT0hHUo85xh+NWF/uqpUJF0FIRzg1YsJkiqbKEel/0QDrTN3qMs1wE8r3TTcsjUIce3gbq7f+8gVgUKWBg4F9k+rmL+07QX9zQWagCewxyzuFvlX+5Q764bom2J5V5yqOrSBCeLfuxYz+cP1zPuziQubfMv7RtKN2EWQDWiZKr5rUv/8R77jDWJsrEryOmVFXDvaQAWWBIZgYWyuh0C1nw0WYqq0catZp2lWF1T5V7W5uMEq2WfdzBfbLgFoBbZoL9EnOgtq6FmMVVoMv3bdqM0Rk3oL1Be4a2CYBWDQ2Emsc= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(13015025)(13017025)(13023025)(13024025)(13018025)(8121501046)(5005006)(10201501046)(3002001)(6055026); SRVR:BN6PR03MB2420; BCL:0; PCL:0; RULEID:; SRVR:BN6PR03MB2420; X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2420; 4:rZTckgQyFcgwHn9YWng4mKZgcbGgSMKZuMRxtwqnLLhmJEZcFz0LKskJVLKu/pWnV6pNBL8NvS04oHE90GRGc3xQBEAOjFP5fjptUs3fD/TEtymRmFz1atHtbFt+ZCrlPGM1qjrSQfvVDL0pDERoZwFhRCr3cREscsTpcEenc21Z4N7r+lg5qCjPFVQW+J0rZr4LR7CkWE5etw1YwU3nJYkRMi3Kj6vh3jYe+5qWqCytWh4YfIoSn0Bn8ZotN8xQMHIQQNa6cx9lrTL4vEKF0QKAUgjThvDXdpKY4dEBNhIqNmUhveCQoDjGCVOJArvOhgCJPkEx3+OFcAH/P4QIRTD2aZReGkHch8f4X9+CRnBwIfBAULyjQE+kVvjaBE7D5SRdkb+FcYnTInbLlBZRR/9ArxeMXZifa8fsjQh0838u/ZOIL6Nxobq+I3KJRip8qVZ3yGLIspelanA/zuRi4aFxgLK0KTcWNmqufGi6UpKjL2d+mf5vUbFXf/G+QbJt X-Forefront-PRVS: 00514A2FE6 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN6PR03MB2420; 23:UZrx3PLZVeuSPU6n8aeTRAi1CXRGXNsN9kKTYouq1?= =?us-ascii?Q?Mrww0QWzbfwaBivofzT1JKQ4kdvAfta1SM0LMieS4KZg8Q9H3oUR0FaPRlH/?= =?us-ascii?Q?um8m6miHJcd95JyM5mHchHF3LqMJWy10fzrUz3dKFAzvdGVgkzsqdmgkHE4G?= =?us-ascii?Q?BKJoHBahqHycAu7Of37ITei60hYj9LJBcTRdLEajv6HDbkxY55sIFbvaEcU6?= =?us-ascii?Q?tjCTd0Apxrd22fWaaJ94FPh1BrxwK0Vnc1PAJSbkYr1dKQ1B1VLJTPZFNP4N?= =?us-ascii?Q?VS43cYI7/DKW+jnRqSpsKb0IajtaT9wkDuicRw6m1qQQTwoam/BJ8Q66nmRc?= =?us-ascii?Q?9LS8XK8FVelrMbuCtHwvQrBKGe/z2tebIuOfX/o+csvLMPdkaTOOb0EqV3iE?= =?us-ascii?Q?ryUWzvlqgef1qn+TnSv71DjWSY1kyaiwhDfEu72ywsNi0ufCLPyShV094v83?= =?us-ascii?Q?2R2wd21qBGgcmp6UKiNuTmf8AEb/7SDjnGMjXK6trq37bZZUF+p6XNNFbI7w?= =?us-ascii?Q?OZQQMzzcRsH3fROrdeIYkOYye/I4g4uZgUKaDA5qfV07W0L8bSnvIS8kPaFJ?= =?us-ascii?Q?FC/7DVR23cT1CcCWDKuKdD2G+Ma+PkNz9pwr82Kznx0V/+uMd+KeBx1PCLDz?= =?us-ascii?Q?8bWvDW0dVKlcuEF8HLVRE51N6aGyW6ugelUZ8AzubGgGMb+w08TBQoSRrq9s?= =?us-ascii?Q?Xl0mxOIoRZE8QewBONV/gd0CUDUNjb5XN1y4cb/AY2V5h9Jn8e6XiaiLIw/j?= =?us-ascii?Q?zt/K1j42BtEun2n+NTQDnsSjKYkxEl5i9AvFQvSu6IW78Day4+ZO3IDTyUej?= =?us-ascii?Q?qIeTH8oLWfNINTFVA9LnCfYrOP+wkmZEfciThe0XtG7Hk7Kj59hZ2Rf7jvDt?= =?us-ascii?Q?VomeMNbkF5jYtX/13tWVk4O8S0DwlEl5HBN+p+yGTHkzufOeBI0ee897Uv95?= =?us-ascii?Q?2ORtVcRTugPgVEWSXJaNewojERRUJZs3JHm/Q3wxZtoSS79gjgqRzY7onVUV?= =?us-ascii?Q?/88w3Y1Q8lyiaoKQWxEpBTa5bt7E4N/AaeISxcWa6W90UJE2sqB5YE6UvLik?= =?us-ascii?Q?7GpGCZYt1QFCB1aOaOJ68SONsLtimlz/H1qclvnB1Pwp2kf8w=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN6PR03MB2420; 6:yLvHb7ShCJpkOcJokxwLefZ0JFrxbW24qYvFKo4gRZ9ouzY3H5ahsAVrAywjWN4sZY3U+FRJScKXbim3cJkS/Do2CQoWs+1PDXWelx8hN/K3/hn7H+JetxwjA/auFZmz+gmohL0dHtmpCMeSZz/Sa98NKYlgO80m/j2nWjANrSxHyUw7exWIjMPaI6PSb0duQl2M7HvmQf3PmjotnfDjabhCYCEavBuKe5OyZiSmDDXsyYM3ESk9Gujw0y4PwmOJ/mtdb7rzE5Z6wbo+V1NAN9+ndWrzxJmzKSRxyxrV+VBW4pOKGBZQdaiRyGItKDYNfJAaisPK6QuySbUUxqpn2g==; 5:WZUgmYiGhb+1P2LIXjIvKpkIRERhEhboWxaqCskWoHDLBradg3gy7GWcv1v7SSOeRf/MKq9mGgMZqxmbseHhhwh4Iwg+fZMIlTA/aDmRpEhBc1+ag+1CmVNpOlZeLzlqy2k3qivYOPO4vNTNbvsGeQ==; 24:dhGctsxeOlWm8vm5lC5rSyCg1pxaqmqRWrX6vO3GwAHPNbbPCE2GR4/ruY0CMkBTcTio/bSfZtYscMkbu3R7041L29amrqyM5fPAXGqwgZE=; 7:AUkETPHN+W/eCwq/oYF7oeCoPGTxtAsaljlB41uVR3BGnOCSjVpkoGo9vzEzkdg+FAYAoT2MphZdZJWsW5gU+Td1px+NUcWvRuSaj96gFFmpdrKnueGTBJ+HPX3xnNVj2iFjlVhrZlj1q0ftQL6x48YoHhFn/E3DZ0Tz3e+bL4zAKIi/wIhV4nwPTkGeSXsWNtYJGRholCc3jMfDmZG/QuujlW/Ts1WkTuHSZwnnYFD0WtYtgZOT/QP66yMdqfzD SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: freescale.onmicrosoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Aug 2016 02:57:23.8579 (UTC) X-MS-Exchange-CrossTenant-Id: 710a03f5-10f6-4d38-9ff4-a80b81da590d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=710a03f5-10f6-4d38-9ff4-a80b81da590d; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR03MB2420 X-Topics: patch Subject: [lng-odp] [PATCH 3/3] example/l3fwd: enhance flow lookup performance in hash X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Xuelin Shi increase the hash size to accomodate more entries. pre-compute the hash keys of possible ip destinations in a subnet. Signed-off-by: Xuelin Shi --- example/l3fwd/odp_l3fwd.c | 3 - example/l3fwd/odp_l3fwd_db.c | 160 ++++++++++++++++++++++++++++++------------- example/l3fwd/odp_l3fwd_db.h | 35 ++++++---- 3 files changed, 131 insertions(+), 67 deletions(-) -- 2.7.4 diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c index 0998614..44778b0 100644 --- a/example/l3fwd/odp_l3fwd.c +++ b/example/l3fwd/odp_l3fwd.c @@ -1061,9 +1061,6 @@ int main(int argc, char **argv) for (i = 0; i < nb_worker; i++) odph_odpthreads_join(&thread_tbl[i]); - /* release resource on exit */ - destroy_fwd_db(); - /* if_names share a single buffer, so only one free */ free(args->if_names[0]); diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c index b3a237f..4731237 100644 --- a/example/l3fwd/odp_l3fwd_db.c +++ b/example/l3fwd/odp_l3fwd_db.c @@ -150,7 +150,7 @@ char *mac_addr_str(char *b, odph_ethaddr_t *mac) */ typedef struct flow_entry_s { ipv4_tuple5_t key; /**< match key */ - struct flow_entry_s *next; /**< next entry on the list */ + struct flow_entry_s *next; /**< next entry in the bucket */ fwd_db_entry_t *fwd_entry; /**< entry info in db */ } flow_entry_t; @@ -158,59 +158,96 @@ typedef struct flow_entry_s { * Flow cache table bucket */ typedef struct flow_bucket_s { - odp_spinlock_t lock; /**< Bucket lock*/ - flow_entry_t *next; /**< Pointer to first flow entry in bucket*/ + odp_rwlock_t lock; /**< Bucket lock*/ + flow_entry_t *next; /**< First flow entry in bucket*/ } flow_bucket_t; /** * Flow hash table, fast lookup cache */ typedef struct flow_table_s { - flow_bucket_t *bucket; - uint32_t count; + odp_rwlock_t flow_lock; /**< flow table lock*/ + flow_entry_t *flows; /**< flow store */ + flow_bucket_t *bucket; /**< bucket store */ + uint32_t bkt_cnt; + uint32_t flow_cnt; + uint32_t next_flow; /**< next available flow in the store */ } flow_table_t; static flow_table_t fwd_lookup_cache; -void init_fwd_hash_cache(void) +static void create_fwd_hash_cache(void) { odp_shm_t hash_shm; flow_bucket_t *bucket; - uint32_t bucket_count; + flow_entry_t *flows; + uint32_t bucket_count, flow_count, size; uint32_t i; - bucket_count = FWD_DEF_BUCKET_COUNT; + flow_count = FWD_MAX_FLOW_COUNT; + bucket_count = flow_count / FWD_DEF_BUCKET_ENTRIES; /*Reserve memory for Routing hash table*/ - hash_shm = odp_shm_reserve("route_table", - sizeof(flow_bucket_t) * bucket_count, - ODP_CACHE_LINE_SIZE, 0); + size = sizeof(flow_bucket_t) * bucket_count + + sizeof(flow_entry_t) * flow_count; + hash_shm = odp_shm_reserve("flow_table", size, ODP_CACHE_LINE_SIZE, 0); bucket = odp_shm_addr(hash_shm); if (!bucket) { - EXAMPLE_ERR("Error: shared mem alloc failed.\n"); - exit(-1); + /* Try the second time with small request */ + flow_count /= 4; + bucket_count = flow_count / FWD_DEF_BUCKET_ENTRIES; + size = sizeof(flow_bucket_t) * bucket_count + + sizeof(flow_entry_t) * flow_count; + hash_shm = odp_shm_reserve("flow_table", size, + ODP_CACHE_LINE_SIZE, 0); + bucket = odp_shm_addr(hash_shm); + if (!bucket) { + EXAMPLE_ERR("Error: shared mem alloc failed.\n"); + exit(-1); + } } + size = sizeof(flow_bucket_t) * bucket_count; + flows = (flow_entry_t *)((char *)bucket + size); + fwd_lookup_cache.bucket = bucket; - fwd_lookup_cache.count = bucket_count; + fwd_lookup_cache.bkt_cnt = bucket_count; + fwd_lookup_cache.flows = flows; + fwd_lookup_cache.flow_cnt = flow_count; - /*Initialize Locks*/ + /*Initialize bucket locks*/ for (i = 0; i < bucket_count; i++) { bucket = &fwd_lookup_cache.bucket[i]; - odp_spinlock_init(&bucket->lock); + odp_rwlock_init(&bucket->lock); bucket->next = NULL; } + + memset(flows, 0, sizeof(flow_entry_t) * flow_count); + odp_rwlock_init(&fwd_lookup_cache.flow_lock); + fwd_lookup_cache.next_flow = 0; +} + +static inline flow_entry_t *get_new_flow(void) +{ + uint32_t next; + flow_entry_t *flow = NULL; + + odp_rwlock_write_lock(&fwd_lookup_cache.flow_lock); + next = fwd_lookup_cache.next_flow; + if (next < fwd_lookup_cache.flow_cnt) { + flow = &fwd_lookup_cache.flows[next]; + fwd_lookup_cache.next_flow++; + } + odp_rwlock_write_unlock(&fwd_lookup_cache.flow_lock); + + return flow; } static inline int match_key_flow(ipv4_tuple5_t *key, flow_entry_t *flow) { - if (key->src_ip == flow->key.src_ip && - key->dst_ip == flow->key.dst_ip && - key->src_port == flow->key.src_port && - key->dst_port == flow->key.dst_port && - key->proto == flow->key.proto) + if (key->hi64 == flow->key.hi64 && key->lo64 == flow->key.lo64) return 1; return 0; @@ -221,10 +258,12 @@ flow_entry_t *lookup_fwd_cache(ipv4_tuple5_t *key, flow_bucket_t *bucket) { flow_entry_t *rst; + odp_rwlock_read_lock(&bucket->lock); for (rst = bucket->next; rst != NULL; rst = rst->next) { if (match_key_flow(key, rst)) break; } + odp_rwlock_read_unlock(&bucket->lock); return rst; } @@ -239,22 +278,58 @@ flow_entry_t *insert_fwd_cache(ipv4_tuple5_t *key, if (!entry) return NULL; - flow = malloc(sizeof(flow_entry_t)); + flow = get_new_flow(); + if (!flow) + return NULL; + flow->key = *key; flow->fwd_entry = entry; - odp_spinlock_lock(&bucket->lock); - if (!bucket->next) { - bucket->next = flow; - } else { + odp_rwlock_write_lock(&bucket->lock); + if (bucket->next) flow->next = bucket->next; - bucket->next = flow; - } - odp_spinlock_unlock(&bucket->lock); + bucket->next = flow; + odp_rwlock_write_unlock(&bucket->lock); return flow; } +void init_fwd_hash_cache(void) +{ + fwd_db_entry_t *entry; + flow_entry_t *flow; + flow_bucket_t *bucket; + uint64_t hash; + uint32_t i, nb_hosts; + ipv4_tuple5_t key; + + create_fwd_hash_cache(); + + /** + * warm up the lookup cache with possible hosts. + * with millions flows, save significant time during runtime. + */ + memset(&key, 0, sizeof(key)); + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { + nb_hosts = 1 << (32 - entry->subnet.depth); + for (i = 0; i < nb_hosts; i++) { + key.dst_ip = entry->subnet.addr + i; + hash = l3fwd_calc_hash(&key); + hash &= fwd_lookup_cache.bkt_cnt - 1; + bucket = &fwd_lookup_cache.bucket[hash]; + flow = lookup_fwd_cache(&key, bucket); + if (flow) + return; + + flow = insert_fwd_cache(&key, bucket, entry); + if (!flow) + goto out; + } + } +out: + return; +} + /** Global pointer to fwd db */ fwd_db_t *fwd_db; @@ -382,35 +457,22 @@ void dump_fwd_db(void) printf("\n"); } -void destroy_fwd_db(void) -{ - flow_bucket_t *bucket; - flow_entry_t *flow, *tmp; - uint32_t i; - - for (i = 0; i < fwd_lookup_cache.count; i++) { - bucket = &fwd_lookup_cache.bucket[i]; - flow = bucket->next; - odp_spinlock_lock(&bucket->lock); - while (flow) { - tmp = flow->next; - free(flow); - flow = tmp; - } - odp_spinlock_unlock(&bucket->lock); - } -} - fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key) { fwd_db_entry_t *entry; flow_entry_t *flow; flow_bucket_t *bucket; uint64_t hash; + ipv4_tuple5_t newkey; + + newkey.hi64 = 0; + newkey.lo64 = 0; + newkey.dst_ip = key->dst_ip; + key = &newkey; /* first find in cache */ hash = l3fwd_calc_hash(key); - hash &= fwd_lookup_cache.count - 1; + hash &= fwd_lookup_cache.bkt_cnt - 1; bucket = &fwd_lookup_cache.bucket[hash]; flow = lookup_fwd_cache(key, bucket); if (flow) diff --git a/example/l3fwd/odp_l3fwd_db.h b/example/l3fwd/odp_l3fwd_db.h index d9f6d61..1b24f1a 100644 --- a/example/l3fwd/odp_l3fwd_db.h +++ b/example/l3fwd/odp_l3fwd_db.h @@ -19,14 +19,14 @@ extern "C" { #define MAX_STRING 32 /** - * Default number of flows + * Max number of flows */ -#define FWD_DEF_FLOW_COUNT 100000 +#define FWD_MAX_FLOW_COUNT (1 << 22) /** - * Default Hash bucket number + * Default hash entries in a bucket */ -#define FWD_DEF_BUCKET_COUNT (FWD_DEF_FLOW_COUNT / 8) +#define FWD_DEF_BUCKET_ENTRIES 4 /** * IP address range (subnet) @@ -40,12 +40,22 @@ typedef struct ip_addr_range_s { * TCP/UDP flow */ typedef struct ipv4_tuple5_s { - uint32_t src_ip; - uint32_t dst_ip; - uint16_t src_port; - uint16_t dst_port; - uint8_t proto; -} ipv4_tuple5_t; + union { + struct { + int32_t src_ip; + int32_t dst_ip; + int16_t src_port; + int16_t dst_port; + int8_t proto; + int8_t pad1; + int16_t pad2; + }; + struct { + int64_t hi64; + int64_t lo64; + }; + }; +} ipv4_tuple5_t ODP_ALIGNED_CACHE; /** * Forwarding data base entry @@ -116,11 +126,6 @@ void dump_fwd_db_entry(fwd_db_entry_t *entry); void dump_fwd_db(void); /** - * Destroy the forwarding database - */ -void destroy_fwd_db(void); - -/** * Find a matching forwarding database entry * * @param key ipv4 tuple