From patchwork Fri Nov 9 16:39:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Honnappa Nagarahalli X-Patchwork-Id: 150687 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp73428ljp; Fri, 9 Nov 2018 08:40:25 -0800 (PST) X-Google-Smtp-Source: AJdET5czdK4wuRvHtEPybXo0AyeaH/5bN0igOHRTQ506OIHhp9C02b2AEXyBCYoIblqU6PL83VVA X-Received: by 2002:a50:9354:: with SMTP id n20-v6mr3039145eda.114.1541781625338; Fri, 09 Nov 2018 08:40:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1541781625; cv=none; d=google.com; s=arc-20160816; b=aqGhyk3e+pLNxL5o8/yqXmBeSl88vuadBEtWsShufTy+TUb4MdKAZgVyH5G2PLFS/F u1FPoj8B6fTP8CIIwxqeoe4XyNCYf+hLbEq8sFui+L/SNw4eIr3JAbt6od569EZsx/JT hb+z0OGREdFbvnPHneVUg9MzydF8qRV0ecwi7y+gSUGNvayMk7GXudICQoawSZnWq3qM 8aAzLLtix72ls7Uo/CcLqsw4DLzxWmOP7jdkr2LjcynP0kZKEaEzP/sOAGXgVBFx2UgX KyKnWoZxSHfGnbPY2+4PYe3KM9Yj/LmMeliHUePJKAUR7FNOCSEj0OsT7ocyzgC7Eitx qfvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:cc:to:from; bh=dIewlATsH+yAS//24PToxRRkO+JBYXplsYNQYXokp+w=; b=dDFfDHUaQb86wlWD46n2sIMrfDDJ4TmbECOou/x3hKRopOPZBnym4TNRLB1yZLu4EP yb+HxL2RiwlX4jwNxIH9xTL4rjVKt7iRuz40lRZKif5ZAUxJ3yswG3ij4IC3HaFAAfq3 pnV0nZ8JDi8Zmf+irZoHFhRO2AbZlPeW6gvuqqSx5tgsl1clHkqhrOj/PWDJ/qn1oLap CN4HHricSnWVNZ0fa5yvlqg+QtgN4m5cdjixdkepfhRK+AhLQJBK9ZOEtHBh5pq1/WhS qplJlup2B35D3gpU+lJx2cEe9/yPAdQuWBGmkK9NCAcIL1ZLkuVw3pERD6l1E1BgWB9u iC/w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org Return-Path: Received: from dpdk.org (dpdk.org. [92.243.14.124]) by mx.google.com with ESMTP id d33-v6si5100413edd.393.2018.11.09.08.40.25; Fri, 09 Nov 2018 08:40:25 -0800 (PST) Received-SPF: pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) client-ip=92.243.14.124; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 67D915424; Fri, 9 Nov 2018 17:39:52 +0100 (CET) Received: from foss.arm.com (usa-sjc-mx-foss1.foss.arm.com [217.140.101.70]) by dpdk.org (Postfix) with ESMTP id 860F94C94 for ; Fri, 9 Nov 2018 17:39:45 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D4C381682; Fri, 9 Nov 2018 08:39:44 -0800 (PST) Received: from qc2400f-1.austin.arm.com (qc2400f-1.austin.arm.com [10.118.12.122]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 535733F5BD; Fri, 9 Nov 2018 08:39:44 -0800 (PST) From: Honnappa Nagarahalli To: bruce.richardson@intel.com, pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org, jerin.jacob@caviumnetworks.com, hemant.agrawal@nxp.com, chaozhu@linux.vnet.ibm.com, yipeng1.wang@intel.com, dharmik.thakkar@arm.com, gavin.hu@arm.com, honnappa.nagarahalli@arm.com, nd@arm.com Date: Fri, 9 Nov 2018 10:39:17 -0600 Message-Id: <20181109163917.16845-5-honnappa.nagarahalli@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181109163917.16845-1-honnappa.nagarahalli@arm.com> References: <20181109163917.16845-1-honnappa.nagarahalli@arm.com> Subject: [dpdk-dev] [PATCH 4/4] hash: separate lf and rw lock lookup code paths X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The lock-free algorithm has caused significant lookup performance regression for certain use cases. The regression is attributed to the use of non-relaxed memory orderings. 2 versions of the lookup functions are created. One that uses the RW lock and the one that is lock-free. This restores the performance regression caused for use cases that used RW lock version of the lookup function. Fixes: e605a1d36 ("hash: add lock-free r/w concurrency") Cc: honnappa.nagarahalli@arm.com Suggested-by: Jerin Jacob Signed-off-by: Honnappa Nagarahalli Reviewed-by: Ola Liljedahl Reviewed-by: Gavin Hu --- lib/librte_hash/rte_cuckoo_hash.c | 44 ++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 10 deletions(-) -- 2.17.1 diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c index 9390dc5e4..7e1a9ac96 100644 --- a/lib/librte_hash/rte_cuckoo_hash.c +++ b/lib/librte_hash/rte_cuckoo_hash.c @@ -1129,10 +1129,11 @@ rte_hash_add_key_data(const struct rte_hash *h, const void *key, void *data) return ret; } -/* Search one bucket to find the match key */ +/* Search one bucket to find the match key - uses rw lock */ static inline int32_t -search_one_bucket(const struct rte_hash *h, const void *key, uint16_t sig, - void **data, const struct rte_hash_bucket *bkt) +search_one_bucket_l(const struct rte_hash *h, const void *key, + uint16_t sig, void **data, + const struct rte_hash_bucket *bkt) { int i; struct rte_hash_key *k, *keys = h->key_store; @@ -1191,8 +1192,8 @@ search_one_bucket_lf(const struct rte_hash *h, const void *key, uint16_t sig, } static inline int32_t -__rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, - hash_sig_t sig, void **data) +__rte_hash_lookup_with_hash_l(const struct rte_hash *h, const void *key, + hash_sig_t sig, void **data) { uint32_t prim_bucket_idx, sec_bucket_idx; struct rte_hash_bucket *bkt, *cur_bkt; @@ -1207,7 +1208,7 @@ __rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, /* Check if key is in primary location */ bkt = &h->buckets[prim_bucket_idx]; - ret = search_one_bucket(h, key, short_sig, data, bkt); + ret = search_one_bucket_l(h, key, short_sig, data, bkt); if (ret != -1) { __hash_rw_reader_unlock(h); return ret; @@ -1217,7 +1218,7 @@ __rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, /* Check if key is in secondary location */ FOR_EACH_BUCKET(cur_bkt, bkt) { - ret = search_one_bucket(h, key, short_sig, + ret = search_one_bucket_l(h, key, short_sig, data, cur_bkt); if (ret != -1) { __hash_rw_reader_unlock(h); @@ -1291,6 +1292,16 @@ __rte_hash_lookup_with_hash_lf(const struct rte_hash *h, const void *key, return -ENOENT; } +static inline int32_t +__rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, + hash_sig_t sig, void **data) +{ + if (h->readwrite_concur_lf_support) + return __rte_hash_lookup_with_hash_lf(h, key, sig, data); + else + return __rte_hash_lookup_with_hash_l(h, key, sig, data); +} + int32_t rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, hash_sig_t sig) @@ -1592,7 +1603,7 @@ compare_signatures(uint32_t *prim_hash_matches, uint32_t *sec_hash_matches, #define PREFETCH_OFFSET 4 static inline void -__rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, +__rte_hash_lookup_bulk_l(const struct rte_hash *h, const void **keys, int32_t num_keys, int32_t *positions, uint64_t *hit_mask, void *data[]) { @@ -1762,10 +1773,10 @@ __rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, next_bkt = secondary_bkt[i]->next; FOR_EACH_BUCKET(cur_bkt, next_bkt) { if (data != NULL) - ret = search_one_bucket(h, keys[i], + ret = search_one_bucket_l(h, keys[i], sig[i], &data[i], cur_bkt); else - ret = search_one_bucket(h, keys[i], + ret = search_one_bucket_l(h, keys[i], sig[i], NULL, cur_bkt); if (ret != -1) { positions[i] = ret; @@ -2005,6 +2016,19 @@ __rte_hash_lookup_bulk_lf(const struct rte_hash *h, const void **keys, *hit_mask = hits; } +static inline void +__rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, + int32_t num_keys, int32_t *positions, + uint64_t *hit_mask, void *data[]) +{ + if (h->readwrite_concur_lf_support) + return __rte_hash_lookup_bulk_lf(h, keys, num_keys, + positions, hit_mask, data); + else + return __rte_hash_lookup_bulk_l(h, keys, num_keys, + positions, hit_mask, data); +} + int rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, uint32_t num_keys, int32_t *positions)