From patchwork Fri Aug 26 06:04:35 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuelin Shi X-Patchwork-Id: 74786 Delivered-To: patch@linaro.org Received: by 10.140.29.52 with SMTP id a49csp293402qga; Fri, 26 Aug 2016 04:52:13 -0700 (PDT) X-Received: by 10.55.103.145 with SMTP id b139mr2616648qkc.15.1472212333539; Fri, 26 Aug 2016 04:52:13 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id o76si13993063qkl.195.2016.08.26.04.52.13; Fri, 26 Aug 2016 04:52:13 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE dis=NONE) header.from=nxp.com Received: by lists.linaro.org (Postfix, from userid 109) id ED23260844; Fri, 26 Aug 2016 11:51:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAD_ENC_HEADER,BAYES_00, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id D213E617AE; Fri, 26 Aug 2016 11:49:04 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 9075261787; Fri, 26 Aug 2016 07:09:09 +0000 (UTC) Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0138.outbound.protection.outlook.com [104.47.33.138]) by lists.linaro.org (Postfix) with ESMTPS id 5CC1C61787 for ; Fri, 26 Aug 2016 07:09:07 +0000 (UTC) Received: from BLUPR0301CA0019.namprd03.prod.outlook.com (10.162.113.157) by MWHPR03MB2622.namprd03.prod.outlook.com (10.168.206.148) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.587.9; Fri, 26 Aug 2016 07:09:05 +0000 Received: from BN1AFFO11FD009.protection.gbl (2a01:111:f400:7c10::134) by BLUPR0301CA0019.outlook.office365.com (2a01:111:e400:5259::29) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.599.9 via Frontend Transport; Fri, 26 Aug 2016 07:09:04 +0000 Received-SPF: Neutral (protection.outlook.com: 192.88.168.50 is neither permitted nor denied by domain of freescale.com) Received: from tx30smr01.am.freescale.net (192.88.168.50) by BN1AFFO11FD009.mail.protection.outlook.com (10.58.52.69) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.587.6 via Frontend Transport; Fri, 26 Aug 2016 07:09:05 +0000 Received: from localhost (rock.ap.freescale.net [10.193.20.106]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id u7Q793oX017530; Fri, 26 Aug 2016 00:09:04 -0700 From: Xuelin Shi To: Date: Fri, 26 Aug 2016 14:04:35 +0800 Message-ID: <1472191475-4466-2-git-send-email-xuelin.shi@nxp.com> X-Mailer: git-send-email 1.8.4 In-Reply-To: <1472191475-4466-1-git-send-email-xuelin.shi@nxp.com> References: <1472191475-4466-1-git-send-email-xuelin.shi@nxp.com> X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:192.88.168.50; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(7916002)(2980300002)(199003)(189002)(76506005)(106466001)(2906002)(105606002)(229853001)(81166006)(57986006)(356003)(7846002)(5660300001)(2351001)(305945005)(189998001)(76176999)(50986999)(8676002)(19580395003)(81156014)(92566002)(50226002)(19580405001)(2950100001)(36756003)(86362001)(104016004)(33646002)(8936002)(48376002)(586003)(68736007)(50466002)(87936001)(97736004)(110136002)(4326007)(5003940100001)(47776003)(77096005)(626004); DIR:OUT; SFP:1102; SCL:1; SRVR:MWHPR03MB2622; H:tx30smr01.am.freescale.net; FPR:; SPF:Neutral; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BN1AFFO11FD009; 1:umPjsFGFnEauhFLFiWLgZ+/DzoXH4yQgCa0wb2zJVzDVhNPOssMDx+DDoDeTe9tigne6tfte3uD04UGaTIUpmXUzPhc39AQEwLIg1R4MjrHO5r5uCX0sx+1jpfW0cOb2RzpWC6qYBm+y+ZbT/A1k22G4cN3NGO4gmJ0AVhmUpE6dQp6LJb1eeoyQcuSuS29nUmPP+lTvMmm362795Vmn6y3wqIs5O2L7opmoeHyO8UKlIpAmPJhh2s28Z2esAWO2a31HKy2Ckr9TTnNNNyETLTw0lmVvhzRYEAmgnTgaPGa9+yvxMENTYE3k9N7qLrgS4w83HrAtgHoe157xo0Mys0QUlyRh91Qu89o/phvL1H6KeKzLsx2/pn/Gk6+DWEuLjvu9l6jmdsKRAEpNijtTt0yaMOjzscITznnms4CmrSP5IbAsbkG7I0wWJM65Bul2mFT0Og1gLBGacfANrrn3l179F7YM4fR8dSqD31wj11RMr3YonZ8Ups5HsJBN8odm8IOdXxPxYARF2TvKgNXtxXC69mvovUkXcizNpiOeQg16lDmgzluD5TSTmovD5QEBOpe8Oqcm6Ui3hMYgTzVFofGbXN7mBqKkT063LI3paN8= MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 8c25587f-9db2-43e3-4401-08d3cd7fde01 X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB2622; 2:Wh9/iePKYKxv2S6qU09WYlQmoOypKTpz0wtplyZaHBT2iZctEr7F5aOC4ttVxioMlMYl3ar4lkkoMNh1VSaXPJBh4W7C+Vh78JhtRewj11jIe35Ig8gCBs/IBVoB29vizgSbYLFo5VTblbqqhaVJKmVNCp7xmGo+pbk8PFuAWhYeAAh3ER5ahX1BrHbUp7/3; 3:9DpqjBbCls1LXXmZ7jDdt9ZBUQtP8TYukvFhT+MuWLuFX0bZmGO73gZQNDf7cgXzZ2+XocpgVI3j7ZB906YxTnJMnb+lkS1ugS4iLMb8t7OPQrMrLqINCOKHKShN5cMZGinzXIcjNu26RZuYrrF+1ivRN2IDg4HIz+ovg0kdmZprEoYVzbyz1a9R58dqv71984V/p5lYuJOSXjlN/w1EWmyzPeKHOtaM4dC119x/PPE= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:MWHPR03MB2622; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB2622; 25:tX++jH1qG9Oc2g5eihNe9OyWZ22VoUCjMWEJ3qZJMMSFmD8xByAyEqhpGZuXUye+ckaciFsUsFNTLaIPym1Vm3BZLk65WAoZq4Cgii6jpxDoq/aHTXGRJqh4CfHcddtZlDczYa2i4Vmo+69w56OcLJLD0D6sqFJQIqGp7K1OlO4E7kqQhuL6UAfxangJRHoicrXFfr2by6m2adXvD5PC2727GdTmfB6SrXqC5wBupllrInxuFuhCrJ/WilDCHQzTfFqndoEuGN1+20aAvLOkszUcyZk59cIzUz/r68wROK7Hta7sBzyBX5+YHezNw7dF2luZIiB1y4RUxDmLwQ0jVcgxiahEcHSNkIwltqCi/9Hy/99RaFvZ8mpwpOD46p1qjk/ZXovSTZzFwJWzIsq0E6vkFj50pcBtcS/uwcFIRx119qL1hnkxMaxBjgh/sh9BqJMgqsOFpD/3UVk6zL8WuDdVWGMpURSqfAW6Ntei+KDl5iMUjw1uieCc6gsZx+Td2GEWpDPRqJQ9zDSSU4scxIxFC5TUwAjtcRmBKAbLONvvPJdvlYCO2RKsBsOSOYI/uOEOTDdD/4XSh9WGngxqHNZ4jpAl+QWReAhanjrbZw7M2m4vV4ZqtNlyuFoX5yR+ZthptLhlUDH1EtCdLyWOvfFuJ8YMRl6oBbceB+QiK87aj3SsQKfNKKYfYYGHZBE7HHQtBELAMy8xmi8n/lMA2O491zddIBmCR+wRlHbUag2i2u6Kxleu4NDtC32YXFkf4osAbTddvxdA6S1HLlgVACVKKJl/wRm9RMDBU7zDjoM= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB2622; 31:5NWbeSXVEzVVaQ6qi/PTxijkCfjd45UEoNqwPDUsealyy1/FsyV9h/qEgzxNYqe3HVO5MWVkcId2M3hbRqswkaw+xyhUNCwEQb7Se3jbLmh23NM3WdgYcDYgEyNYK9By/FaHYAg3Fs8PClIMMuvilEHFrZS3qbVekWA7giXb6CvWSSNGXN7s+J8Y8uvMmrJlz8bbSvxpcX9L2WcSHSnS+G2ygwWtdThsVJa/br/Dms4=; 20:FM8HgCaT2gDUHfa0UAXA08iQH/O1NHRhpComsC1Z2kjmJzu+/0dHjnCIKEdfy3bP10rntJJqDfgkwfDDoL8p2g97fLmYDg9cn0ChP1hZSL7Nqy9d4a2J0AdXHt0z3n1uZyO4I+CLGDYwdnf9yzK2UPzg9fGgukG72PFAp+0IHcn9QwZr/Nhn96w96q8tc84XKQLV1F62sVBsW+j8L4K/U0ExGh45OkPq27xywV/VG7fO2IlJAohiLtiRY/P6eYydXD97F3JCYUKQXnRP7oZL6EhR3msIOSQmD5KG83hTiqPG7eauwfFjzKcPpsOjMTwImQIqNY+snmv4lRUI2PNwRpSbG/pgy3/T+E7QJ4lfzmRLcZFw9n9Cb4WmtXnuGZJ+DfWMvRcgvOGz3HfvMT1TJeZjrzYaWUsoeuTi2KhFjoWCYjfsYvyML8QVhq2+Pmlg0wJVtFfZ8K/gBR7Qi692jbUDS2vcIhqYLuHNO8k0+WPaVb+gFzczEKitwROGyaxRTn2itdKIbbUAFyiCXhnMrl1EcVDcNx3eiyMBYttj0ooXLNbiImQGotC7/i5PrlQET3caOrOAPkZTNhhR1FUlt8AK/B9byPFtwBYZYG47ZNU= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(13024025)(13015025)(13018025)(13023025)(13017025)(5005006)(8121501046)(10201501046)(3002001)(6055026); SRVR:MWHPR03MB2622; BCL:0; PCL:0; RULEID:; SRVR:MWHPR03MB2622; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB2622; 4:PsedIJ2C+942IyvMTDhME3x9N9cWDRyNEc7EtHqs7DX/Lo1tK3ISe/iwhhroO//gK+r4D9Vk2AzoomyO2RDf9in5mMJIO9YfkmFWujvVDVu8Igx7U+Bp/ECyqOZ+Qj+j/Wu+yyZnjmx1yq0El0dkp3KycnFnO4EhC/uetPEX8h3+76KvLUt3szQ4+61J1whHQf8i5ZLZMYlzAkDFOpdOIMfLkZ9xH7waKc65eninioducp2Du0cBkXAeQYzGumGOivP4/hM2tN65ga32eZ3ksEqBYSMWl8Y7T8IfEdwnCArbwtXw07AHwMFudmedrKlCne9QkDteyQZK1sNLCBmlDpWGxO0fn8Yh5BwXAC33JaBFnZ+TbIqhBeABQfNQEc1m0T/qc2uDpn/rnC664lKIOu3fYQ2R4GCSZ17W/X2LqMMXFTZnkekcgu3brNdPh/LzVVgyxdAEfz2fo315b991p6cL5MYvr2vT56naj4PNjWaSNpXVhFPTMlNiGEMcO2UH X-Forefront-PRVS: 00462943DE X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB2622; 23:0/2E8UiucpijvlpO2e1u+r2tK+Ux+JYg9WKqEzKJd?= =?us-ascii?Q?vhQs6/L+CQBpT7B561glANQfgQ/pfIVj50ZmdPSN9s+NySdjaOYcaUaJasRB?= =?us-ascii?Q?29D13glVft5A1hdczXIlY9AmXAo337KYSZkcUCenW6odnSKTfoQmQJ9hIVBK?= =?us-ascii?Q?2+nGfk9qahSSR5oFIt1wYHuC/AeDEz7/8biLB5R2pK1YAKc9Qfl2/6Afd7mX?= =?us-ascii?Q?p1R8k/wApza9E35aDYtY8k8P8pLbpJTV7HkUEXl7V6aRV4qjM79smt47KlzJ?= =?us-ascii?Q?MPaxoAAdK1Sudx98TQOGsHxKEZxeSpwTURQi4ZsXR0fIbzxq1OPnh5PzvUbS?= =?us-ascii?Q?1fzd//fb1QDlBoTaIfW7yaabSEMhm4XGUqM8CindaHGTy1j3nm9+HwxxRwfL?= =?us-ascii?Q?6J4rIXn+Mx+eA6HHCD5v8X9FKMK0Fh2FTQOeKUXVufQ7LsWtNfmYhP0Q++vk?= =?us-ascii?Q?KWO4hCajFFNh4bLmaZyoOjiZhLGoY3rvCznfFLSJIe288rjCc/M0DnqWGHdp?= =?us-ascii?Q?AkJpbPdQO3ZriO71jupxii+Ktcd5KfqqkFn7OiQgM2vDsYfu3+LeE0muzT2K?= =?us-ascii?Q?S5RZhXdur7V/BMhi7JRXguVOflloKSEoYl3NDnXuKEeCJWyJg+5g8vZyfB4l?= =?us-ascii?Q?fasQUOr7XaqvrE7/OIqMoD3dHsP5s/9AYMaOG5l0Tyo+sgFYSvEjUmzB7nbe?= =?us-ascii?Q?HKB7fSIALAu1s+HuiWGvr8FItIKym6+1AWNA1zyB2XMSD8ou9pWIuHJ6jip9?= =?us-ascii?Q?+LzKP+TfvLc+yySkWoiyCEB2Vtwpaks8alhYEGLUYEQN/mMaEHZg8WFy33/n?= =?us-ascii?Q?hp6GpnlaXA4jEHjqwjKDPctIL/X5RznDzjQnNUT82FCriLiMLNPdJs2Cg8H5?= =?us-ascii?Q?IubIO3LJYuuElKUdF0XvaTOd/+jCtQ6JJme1RmEGaZGMuqJoamTDXgTmllDd?= =?us-ascii?Q?qT2MKfECjR9qLYuO3u/utOgNwNnO1qP4f3A78LbQN3cRL7o724vygMQBnQQH?= =?us-ascii?Q?6pyxmwm4rWO/iiSt5dq586LfKyuglmXZu2fusiAeNB64XovKKWg4w3XQql8C?= =?us-ascii?Q?Zp37WEQYLu2xk1hZFhkwYHZFJ61?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB2622; 6:eqSZTlpe8d3yGq3M7Z3GxWD9abpUg2Wyom9jJSjvzyC2sxWohSdf/oglcgKA4Ikig8laGo6931kp1V/DLaMvifbwdH6g79BhkFD53G44Kx4j3lNHTw0WRmTuHH22wYBn6zR8k0sOkYWSO5gij1UMDHj17yBTPhEGKmmIQVPuKiMNPdRn3MURs3Wxd69dGnObbbPUFhqWkUp2uuNlK0BpEIXBY6z3atY8Gy37LxROTsd0caaP7bHS9B/L9qmzqb3DZ1LvOYVvIqn8QDVQvSxfhnYez4ZXoGENZEYs7gSqcMubijRcMSW4dOPFquBxkQ5wXvOgl4KjXe20+nFxBM28lg==; 5:0v8LyGymZFojkTknJo539dn5EbC3F9ucZvunnd2kLVQ75L59bKGrzjVXcUs5Bdyfw8BXlLqYWTPX0zvE5+28/hSjWQTUvzPQ8hFGxtenb878ONseRYf312Vx2sH7b3Y7shtygiVknQ4cxWrVvdb7eg==; 24:0B9fTvqkXa5GQrlfh4K8mzVIiRnlNpAgY4dV7nzuGBS54tGI4BIt6yWYrbHQB+q6CZAIT/kisCFzPRcOT2VZFMLmjrh59rNGa0zcM19UW6o=; 7:nqKnFfYbhMmFSYtseCeVePBjfEQEgJ4jtDpqXq4DOsxbqP6wo2esw/Fl4OzYyNA1ssoH91gJZgdbShuwAaKrd6thC2T9G9Xdd/mIPtgx0/dm2ZS1MmVx5llfgEhWPPjcfmfN7CBNqtEjoAighBriBSErBEd+XbWCP6R0goXDA24IGYM5rJjsHfFJUQ4ZJVHX1J92Ybpomef9RkHjTVfOqL59B8dMaCLC8NT0SwT850oMYoU+z+aj9CvsPCgF5dZn SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: freescale.onmicrosoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2016 07:09:05.6197 (UTC) X-MS-Exchange-CrossTenant-Id: 710a03f5-10f6-4d38-9ff4-a80b81da590d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=710a03f5-10f6-4d38-9ff4-a80b81da590d; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB2622 X-Mailman-Approved-At: Fri, 26 Aug 2016 11:49:02 +0000 X-Topics: patch Subject: [lng-odp] [PATCH] example/l3fwd: enhance flow lookup performance in hash X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Xuelin Shi increase the hash size. pre-compute the hash keys of possible ip destinations in a subnet. Signed-off-by: Xuelin Shi --- example/l3fwd/odp_l3fwd.c | 3 - example/l3fwd/odp_l3fwd_db.c | 160 ++++++++++++++++++++++++++++++------------- example/l3fwd/odp_l3fwd_db.h | 35 ++++++---- 3 files changed, 131 insertions(+), 67 deletions(-) -- 1.8.3.1 diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c index efe0382..be71997 100644 --- a/example/l3fwd/odp_l3fwd.c +++ b/example/l3fwd/odp_l3fwd.c @@ -1061,9 +1061,6 @@ int main(int argc, char **argv) for (i = 0; i < nb_worker; i++) odph_odpthreads_join(&thread_tbl[i]); - /* release resource on exit */ - destroy_fwd_db(); - /* if_names share a single buffer, so only one free */ free(args->if_names[0]); diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c index b3a237f..4731237 100644 --- a/example/l3fwd/odp_l3fwd_db.c +++ b/example/l3fwd/odp_l3fwd_db.c @@ -150,7 +150,7 @@ char *mac_addr_str(char *b, odph_ethaddr_t *mac) */ typedef struct flow_entry_s { ipv4_tuple5_t key; /**< match key */ - struct flow_entry_s *next; /**< next entry on the list */ + struct flow_entry_s *next; /**< next entry in the bucket */ fwd_db_entry_t *fwd_entry; /**< entry info in db */ } flow_entry_t; @@ -158,59 +158,96 @@ typedef struct flow_entry_s { * Flow cache table bucket */ typedef struct flow_bucket_s { - odp_spinlock_t lock; /**< Bucket lock*/ - flow_entry_t *next; /**< Pointer to first flow entry in bucket*/ + odp_rwlock_t lock; /**< Bucket lock*/ + flow_entry_t *next; /**< First flow entry in bucket*/ } flow_bucket_t; /** * Flow hash table, fast lookup cache */ typedef struct flow_table_s { - flow_bucket_t *bucket; - uint32_t count; + odp_rwlock_t flow_lock; /**< flow table lock*/ + flow_entry_t *flows; /**< flow store */ + flow_bucket_t *bucket; /**< bucket store */ + uint32_t bkt_cnt; + uint32_t flow_cnt; + uint32_t next_flow; /**< next available flow in the store */ } flow_table_t; static flow_table_t fwd_lookup_cache; -void init_fwd_hash_cache(void) +static void create_fwd_hash_cache(void) { odp_shm_t hash_shm; flow_bucket_t *bucket; - uint32_t bucket_count; + flow_entry_t *flows; + uint32_t bucket_count, flow_count, size; uint32_t i; - bucket_count = FWD_DEF_BUCKET_COUNT; + flow_count = FWD_MAX_FLOW_COUNT; + bucket_count = flow_count / FWD_DEF_BUCKET_ENTRIES; /*Reserve memory for Routing hash table*/ - hash_shm = odp_shm_reserve("route_table", - sizeof(flow_bucket_t) * bucket_count, - ODP_CACHE_LINE_SIZE, 0); + size = sizeof(flow_bucket_t) * bucket_count + + sizeof(flow_entry_t) * flow_count; + hash_shm = odp_shm_reserve("flow_table", size, ODP_CACHE_LINE_SIZE, 0); bucket = odp_shm_addr(hash_shm); if (!bucket) { - EXAMPLE_ERR("Error: shared mem alloc failed.\n"); - exit(-1); + /* Try the second time with small request */ + flow_count /= 4; + bucket_count = flow_count / FWD_DEF_BUCKET_ENTRIES; + size = sizeof(flow_bucket_t) * bucket_count + + sizeof(flow_entry_t) * flow_count; + hash_shm = odp_shm_reserve("flow_table", size, + ODP_CACHE_LINE_SIZE, 0); + bucket = odp_shm_addr(hash_shm); + if (!bucket) { + EXAMPLE_ERR("Error: shared mem alloc failed.\n"); + exit(-1); + } } + size = sizeof(flow_bucket_t) * bucket_count; + flows = (flow_entry_t *)((char *)bucket + size); + fwd_lookup_cache.bucket = bucket; - fwd_lookup_cache.count = bucket_count; + fwd_lookup_cache.bkt_cnt = bucket_count; + fwd_lookup_cache.flows = flows; + fwd_lookup_cache.flow_cnt = flow_count; - /*Initialize Locks*/ + /*Initialize bucket locks*/ for (i = 0; i < bucket_count; i++) { bucket = &fwd_lookup_cache.bucket[i]; - odp_spinlock_init(&bucket->lock); + odp_rwlock_init(&bucket->lock); bucket->next = NULL; } + + memset(flows, 0, sizeof(flow_entry_t) * flow_count); + odp_rwlock_init(&fwd_lookup_cache.flow_lock); + fwd_lookup_cache.next_flow = 0; +} + +static inline flow_entry_t *get_new_flow(void) +{ + uint32_t next; + flow_entry_t *flow = NULL; + + odp_rwlock_write_lock(&fwd_lookup_cache.flow_lock); + next = fwd_lookup_cache.next_flow; + if (next < fwd_lookup_cache.flow_cnt) { + flow = &fwd_lookup_cache.flows[next]; + fwd_lookup_cache.next_flow++; + } + odp_rwlock_write_unlock(&fwd_lookup_cache.flow_lock); + + return flow; } static inline int match_key_flow(ipv4_tuple5_t *key, flow_entry_t *flow) { - if (key->src_ip == flow->key.src_ip && - key->dst_ip == flow->key.dst_ip && - key->src_port == flow->key.src_port && - key->dst_port == flow->key.dst_port && - key->proto == flow->key.proto) + if (key->hi64 == flow->key.hi64 && key->lo64 == flow->key.lo64) return 1; return 0; @@ -221,10 +258,12 @@ flow_entry_t *lookup_fwd_cache(ipv4_tuple5_t *key, flow_bucket_t *bucket) { flow_entry_t *rst; + odp_rwlock_read_lock(&bucket->lock); for (rst = bucket->next; rst != NULL; rst = rst->next) { if (match_key_flow(key, rst)) break; } + odp_rwlock_read_unlock(&bucket->lock); return rst; } @@ -239,22 +278,58 @@ flow_entry_t *insert_fwd_cache(ipv4_tuple5_t *key, if (!entry) return NULL; - flow = malloc(sizeof(flow_entry_t)); + flow = get_new_flow(); + if (!flow) + return NULL; + flow->key = *key; flow->fwd_entry = entry; - odp_spinlock_lock(&bucket->lock); - if (!bucket->next) { - bucket->next = flow; - } else { + odp_rwlock_write_lock(&bucket->lock); + if (bucket->next) flow->next = bucket->next; - bucket->next = flow; - } - odp_spinlock_unlock(&bucket->lock); + bucket->next = flow; + odp_rwlock_write_unlock(&bucket->lock); return flow; } +void init_fwd_hash_cache(void) +{ + fwd_db_entry_t *entry; + flow_entry_t *flow; + flow_bucket_t *bucket; + uint64_t hash; + uint32_t i, nb_hosts; + ipv4_tuple5_t key; + + create_fwd_hash_cache(); + + /** + * warm up the lookup cache with possible hosts. + * with millions flows, save significant time during runtime. + */ + memset(&key, 0, sizeof(key)); + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { + nb_hosts = 1 << (32 - entry->subnet.depth); + for (i = 0; i < nb_hosts; i++) { + key.dst_ip = entry->subnet.addr + i; + hash = l3fwd_calc_hash(&key); + hash &= fwd_lookup_cache.bkt_cnt - 1; + bucket = &fwd_lookup_cache.bucket[hash]; + flow = lookup_fwd_cache(&key, bucket); + if (flow) + return; + + flow = insert_fwd_cache(&key, bucket, entry); + if (!flow) + goto out; + } + } +out: + return; +} + /** Global pointer to fwd db */ fwd_db_t *fwd_db; @@ -382,35 +457,22 @@ void dump_fwd_db(void) printf("\n"); } -void destroy_fwd_db(void) -{ - flow_bucket_t *bucket; - flow_entry_t *flow, *tmp; - uint32_t i; - - for (i = 0; i < fwd_lookup_cache.count; i++) { - bucket = &fwd_lookup_cache.bucket[i]; - flow = bucket->next; - odp_spinlock_lock(&bucket->lock); - while (flow) { - tmp = flow->next; - free(flow); - flow = tmp; - } - odp_spinlock_unlock(&bucket->lock); - } -} - fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key) { fwd_db_entry_t *entry; flow_entry_t *flow; flow_bucket_t *bucket; uint64_t hash; + ipv4_tuple5_t newkey; + + newkey.hi64 = 0; + newkey.lo64 = 0; + newkey.dst_ip = key->dst_ip; + key = &newkey; /* first find in cache */ hash = l3fwd_calc_hash(key); - hash &= fwd_lookup_cache.count - 1; + hash &= fwd_lookup_cache.bkt_cnt - 1; bucket = &fwd_lookup_cache.bucket[hash]; flow = lookup_fwd_cache(key, bucket); if (flow) diff --git a/example/l3fwd/odp_l3fwd_db.h b/example/l3fwd/odp_l3fwd_db.h index d9f6d61..1b24f1a 100644 --- a/example/l3fwd/odp_l3fwd_db.h +++ b/example/l3fwd/odp_l3fwd_db.h @@ -19,14 +19,14 @@ extern "C" { #define MAX_STRING 32 /** - * Default number of flows + * Max number of flows */ -#define FWD_DEF_FLOW_COUNT 100000 +#define FWD_MAX_FLOW_COUNT (1 << 22) /** - * Default Hash bucket number + * Default hash entries in a bucket */ -#define FWD_DEF_BUCKET_COUNT (FWD_DEF_FLOW_COUNT / 8) +#define FWD_DEF_BUCKET_ENTRIES 4 /** * IP address range (subnet) @@ -40,12 +40,22 @@ typedef struct ip_addr_range_s { * TCP/UDP flow */ typedef struct ipv4_tuple5_s { - uint32_t src_ip; - uint32_t dst_ip; - uint16_t src_port; - uint16_t dst_port; - uint8_t proto; -} ipv4_tuple5_t; + union { + struct { + int32_t src_ip; + int32_t dst_ip; + int16_t src_port; + int16_t dst_port; + int8_t proto; + int8_t pad1; + int16_t pad2; + }; + struct { + int64_t hi64; + int64_t lo64; + }; + }; +} ipv4_tuple5_t ODP_ALIGNED_CACHE; /** * Forwarding data base entry @@ -116,11 +126,6 @@ void dump_fwd_db_entry(fwd_db_entry_t *entry); void dump_fwd_db(void); /** - * Destroy the forwarding database - */ -void destroy_fwd_db(void); - -/** * Find a matching forwarding database entry * * @param key ipv4 tuple