| Message ID | 1495098540-8303-1-git-send-email-jianbo.liu@linaro.org |
|---|---|
| State | New |
| Headers | show |
18/05/2017 11:09, Jianbo Liu: > Implement the same hash functions with crc32 on arm platform. > > Signed-off-by: Jianbo Liu <jianbo.liu@linaro.org> > --- > examples/ip_pipeline/pipeline/hash_func.h | 2 + > examples/ip_pipeline/pipeline/hash_func_arm64.h | 245 ++++++++++++++++++++++++ > 2 files changed, 247 insertions(+) > create mode 100644 examples/ip_pipeline/pipeline/hash_func_arm64.h I don't understand why this code is in an example. We have some CRC code in librte_hash, librte_net and ip_pipeline. Cristian, Jianbo, does it make sense to move these functions somewhere else?
> -----Original Message----- > From: Thomas Monjalon [mailto:thomas@monjalon.net] > Sent: Monday, July 3, 2017 10:12 PM > To: Jianbo Liu <jianbo.liu@linaro.org>; Dumitrescu, Cristian > <cristian.dumitrescu@intel.com> > Cc: dev@dpdk.org; jerin.jacob@caviumnetworks.com; > ashwin.sekhar@caviumnetworks.com > Subject: Re: [dpdk-dev] [PATCH] examples/ip_pipeline: use crc32 in hash > functions for arm64 > > 18/05/2017 11:09, Jianbo Liu: > > Implement the same hash functions with crc32 on arm platform. > > > > Signed-off-by: Jianbo Liu <jianbo.liu@linaro.org> > > --- > > examples/ip_pipeline/pipeline/hash_func.h | 2 + > > examples/ip_pipeline/pipeline/hash_func_arm64.h | 245 > ++++++++++++++++++++++++ > > 2 files changed, 247 insertions(+) > > create mode 100644 examples/ip_pipeline/pipeline/hash_func_arm64.h > > I don't understand why this code is in an example. > We have some CRC code in librte_hash, librte_net and ip_pipeline. > Cristian, Jianbo, > does it make sense to move these functions somewhere else? > I think example apps are a great way to propose new hash functions. IMO we should encourage the definition/exploration of new hash functions in our example apps. These functions are examples of how fast hash functions can be built using special CPU instructions. They have much better performance than e.g. jhash, but their properties are largely unknown, as no rigorous study on their properties (such as uniform distribution) has been conducted. I have seen them providing good performance for the data set I have been using, but I have no extensive data to support their maturity level. If somebody is willing to invest the effort in proving them, I would be more than happy to see them moved to a library like librte_hash. Pablo as maintainer has the choice (I think it is not the first time we discuss bout these hash funcs :) ) As mentioned in one of our deprecation notices, I am actively working (not ready for 17.8 unfortunately) to add a key mask parameter to these functions, so more work on these hash functions is likely to take place.
04/07/2017 01:19, Dumitrescu, Cristian: > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > 18/05/2017 11:09, Jianbo Liu: > > > Implement the same hash functions with crc32 on arm platform. > > > > > > Signed-off-by: Jianbo Liu <jianbo.liu@linaro.org> > > > --- > > > examples/ip_pipeline/pipeline/hash_func.h | 2 + > > > examples/ip_pipeline/pipeline/hash_func_arm64.h | 245 > > ++++++++++++++++++++++++ > > > 2 files changed, 247 insertions(+) > > > create mode 100644 examples/ip_pipeline/pipeline/hash_func_arm64.h > > > > I don't understand why this code is in an example. > > We have some CRC code in librte_hash, librte_net and ip_pipeline. > > Cristian, Jianbo, > > does it make sense to move these functions somewhere else? > > > > I think example apps are a great way to propose new hash functions. IMO we should encourage the > definition/exploration of new hash functions in our example apps. > > These functions are examples of how fast hash functions can be built using special CPU instructions. > They have much better performance than e.g. jhash, but their properties are largely unknown, as no > rigorous study on their properties (such as uniform distribution) has been conducted. I have seen them > providing good performance for the data set I have been using, but I have no extensive data to support > their maturity level. > > If somebody is willing to invest the effort in proving them, I would be more than happy to see them > moved to a library like librte_hash. Pablo as maintainer has the choice (I think it is not the first time we > discuss bout these hash funcs :) ) > > As mentioned in one of our deprecation notices, I am actively working (not ready for 17.8 unfortunately) > to add a key mask parameter to these functions, so more work on these hash functions is likely to take place. OK thanks for the explanation. I still think we do not need to prove hash for integrating them. I would be interested to read Pablo's opinion.
> -----Original Message----- > From: Thomas Monjalon [mailto:thomas@monjalon.net] > Sent: Tuesday, July 4, 2017 12:26 AM > To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>; De Lara Guarch, > Pablo <pablo.de.lara.guarch@intel.com> > Cc: dev@dpdk.org; Jianbo Liu <jianbo.liu@linaro.org>; > jerin.jacob@caviumnetworks.com; ashwin.sekhar@caviumnetworks.com > Subject: Re: [dpdk-dev] [PATCH] examples/ip_pipeline: use crc32 in hash > functions for arm64 > > 04/07/2017 01:19, Dumitrescu, Cristian: > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > > 18/05/2017 11:09, Jianbo Liu: > > > > Implement the same hash functions with crc32 on arm platform. > > > > > > > > Signed-off-by: Jianbo Liu <jianbo.liu@linaro.org> > > > > --- > > > > examples/ip_pipeline/pipeline/hash_func.h | 2 + > > > > examples/ip_pipeline/pipeline/hash_func_arm64.h | 245 > > > ++++++++++++++++++++++++ > > > > 2 files changed, 247 insertions(+) create mode 100644 > > > > examples/ip_pipeline/pipeline/hash_func_arm64.h > > > > > > I don't understand why this code is in an example. > > > We have some CRC code in librte_hash, librte_net and ip_pipeline. > > > Cristian, Jianbo, > > > does it make sense to move these functions somewhere else? > > > > > > > I think example apps are a great way to propose new hash functions. > > IMO we should encourage the definition/exploration of new hash > functions in our example apps. > > > > These functions are examples of how fast hash functions can be built > using special CPU instructions. > > They have much better performance than e.g. jhash, but their > > properties are largely unknown, as no rigorous study on their > > properties (such as uniform distribution) has been conducted. I have > > seen them providing good performance for the data set I have been > using, but I have no extensive data to support their maturity level. > > > > If somebody is willing to invest the effort in proving them, I would > > be more than happy to see them moved to a library like librte_hash. > > Pablo as maintainer has the choice (I think it is not the first time > > we discuss bout these hash funcs :) ) > > > > As mentioned in one of our deprecation notices, I am actively working > > (not ready for 17.8 unfortunately) to add a key mask parameter to these > functions, so more work on these hash functions is likely to take place. > > OK thanks for the explanation. > I still think we do not need to prove hash for integrating them. > I would be interested to read Pablo's opinion. If these functions are used as hash functions, I would place them in rte_hash. The case where we placed the CRC function in librte_net was because that was not used as a hash function, so it made sense to me placing it there, but in this case, it looks like it is, so I think rte_hash is a valid place (although someone would need to integrate it with the existing CRC hash function in that library). Thanks, Pablo
On 4 July 2017 at 21:55, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com> wrote: > > >> -----Original Message----- >> From: Thomas Monjalon [mailto:thomas@monjalon.net] >> Sent: Tuesday, July 4, 2017 12:26 AM >> To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>; De Lara Guarch, >> Pablo <pablo.de.lara.guarch@intel.com> >> Cc: dev@dpdk.org; Jianbo Liu <jianbo.liu@linaro.org>; >> jerin.jacob@caviumnetworks.com; ashwin.sekhar@caviumnetworks.com >> Subject: Re: [dpdk-dev] [PATCH] examples/ip_pipeline: use crc32 in hash >> functions for arm64 >> >> 04/07/2017 01:19, Dumitrescu, Cristian: >> > From: Thomas Monjalon [mailto:thomas@monjalon.net] >> > > 18/05/2017 11:09, Jianbo Liu: >> > > > Implement the same hash functions with crc32 on arm platform. >> > > > >> > > > Signed-off-by: Jianbo Liu <jianbo.liu@linaro.org> >> > > > --- >> > > > examples/ip_pipeline/pipeline/hash_func.h | 2 + >> > > > examples/ip_pipeline/pipeline/hash_func_arm64.h | 245 >> > > ++++++++++++++++++++++++ >> > > > 2 files changed, 247 insertions(+) create mode 100644 >> > > > examples/ip_pipeline/pipeline/hash_func_arm64.h >> > > >> > > I don't understand why this code is in an example. >> > > We have some CRC code in librte_hash, librte_net and ip_pipeline. >> > > Cristian, Jianbo, >> > > does it make sense to move these functions somewhere else? >> > > >> > >> > I think example apps are a great way to propose new hash functions. >> > IMO we should encourage the definition/exploration of new hash >> functions in our example apps. >> > >> > These functions are examples of how fast hash functions can be built >> using special CPU instructions. >> > They have much better performance than e.g. jhash, but their >> > properties are largely unknown, as no rigorous study on their >> > properties (such as uniform distribution) has been conducted. I have >> > seen them providing good performance for the data set I have been >> using, but I have no extensive data to support their maturity level. >> > >> > If somebody is willing to invest the effort in proving them, I would >> > be more than happy to see them moved to a library like librte_hash. >> > Pablo as maintainer has the choice (I think it is not the first time >> > we discuss bout these hash funcs :) ) >> > >> > As mentioned in one of our deprecation notices, I am actively working >> > (not ready for 17.8 unfortunately) to add a key mask parameter to these >> functions, so more work on these hash functions is likely to take place. >> >> OK thanks for the explanation. >> I still think we do not need to prove hash for integrating them. >> I would be interested to read Pablo's opinion. > > If these functions are used as hash functions, I would place them in rte_hash. > > The case where we placed the CRC function in librte_net was because that > was not used as a hash function, so it made sense to me placing it there, > but in this case, it looks like it is, so I think rte_hash is a valid place > (although someone would need to integrate it with the existing CRC hash function in that library). > I think Cristian explanation justified using the special hash functions here. And they may have better performance than the standard functions in the library. Thanks! Jianbo
Hi Jianbo, <snip>... > >> > As mentioned in one of our deprecation notices, I am actively working > >> > (not ready for 17.8 unfortunately) to add a key mask parameter to these > >> functions, so more work on these hash functions is likely to take place. > >> The above mentioned rework has finally happened, thank you for your patience. Are you able to rework your patch to accommodate the mask-based hash functions to target the 18.02 release? Regards, Cristian
On 24 October 2017 at 23:38, Dumitrescu, Cristian <cristian.dumitrescu@intel.com> wrote: > Hi Jianbo, > > <snip>... > >> >> > As mentioned in one of our deprecation notices, I am actively working >> >> > (not ready for 17.8 unfortunately) to add a key mask parameter to these >> >> functions, so more work on these hash functions is likely to take place. >> >> > > The above mentioned rework has finally happened, thank you for your patience. > > Are you able to rework your patch to accommodate the mask-based hash functions to target the 18.02 release? > Sure, I will send a new version. Thanks! Jianbo
diff --git a/examples/ip_pipeline/pipeline/hash_func.h b/examples/ip_pipeline/pipeline/hash_func.h index 9db7173..990bbee 100644 --- a/examples/ip_pipeline/pipeline/hash_func.h +++ b/examples/ip_pipeline/pipeline/hash_func.h @@ -335,6 +335,8 @@ #define hash_default_key56 hash_crc_key56 #define hash_default_key64 hash_crc_key64 +#elif defined(RTE_ARCH_ARM64) +#include "hash_func_arm64.h" #else #define hash_default_key8 hash_xor_key8 diff --git a/examples/ip_pipeline/pipeline/hash_func_arm64.h b/examples/ip_pipeline/pipeline/hash_func_arm64.h new file mode 100644 index 0000000..f6a45d9 --- /dev/null +++ b/examples/ip_pipeline/pipeline/hash_func_arm64.h @@ -0,0 +1,245 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2017 Linaro Limited. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ +#ifndef __HASH_FUNC_ARM64_H__ +#define __HASH_FUNC_ARM64_H__ + +#define _CRC32CX(crc, val) \ + __asm__("crc32cx %w[c], %w[c], %x[v]":[c] "+r" (crc):[v] "r" (val)) + +static inline uint64_t +hash_crc_key8(void *key, __rte_unused uint32_t key_size, uint64_t seed) +{ + uint64_t *k = key; + uint32_t crc0; + + crc0 = seed; + _CRC32CX(crc0, k[0]); + + return crc0; +} + +static inline uint64_t +hash_crc_key16(void *key, __rte_unused uint32_t key_size, uint64_t seed) +{ + uint64_t *k = key, k0; + uint32_t crc0, crc1; + + k0 = k[0]; + + crc0 = k0; + _CRC32CX(crc0, seed); + crc1 = k0 >> 32; + _CRC32CX(crc1, k[1]); + + crc0 ^= crc1; + + return crc0; +} + +static inline uint64_t +hash_crc_key24(void *key, __rte_unused uint32_t key_size, uint64_t seed) +{ + uint64_t *k = key, k0, k2; + uint32_t crc0, crc1; + + k0 = k[0]; + k2 = k[2]; + + crc0 = k0; + _CRC32CX(crc0, seed); + crc1 = k0 >> 32; + _CRC32CX(crc1, k[1]); + + _CRC32CX(crc0, k2); + + crc0 ^= crc1; + + return crc0; +} + +static inline uint64_t +hash_crc_key32(void *key, __rte_unused uint32_t key_size, uint64_t seed) +{ + uint64_t *k = key, k0, k2; + uint32_t crc0, crc1, crc2, crc3; + + k0 = k[0]; + k2 = k[2]; + + crc0 = k0; + _CRC32CX(crc0, seed); + crc1 = k0 >> 32; + _CRC32CX(crc1, k[1]); + + crc2 = k2; + _CRC32CX(crc2, k[3]); + crc3 = k2 >> 32; + + _CRC32CX(crc0, crc1); + _CRC32CX(crc2, crc3); + + crc0 ^= crc2; + + return crc0; +} + +static inline uint64_t +hash_crc_key40(void *key, __rte_unused uint32_t key_size, uint64_t seed) +{ + uint64_t *k = key, k0, k2; + uint32_t crc0, crc1, crc2, crc3; + + k0 = k[0]; + k2 = k[2]; + + crc0 = k0; + _CRC32CX(crc0, seed); + crc1 = k0 >> 32; + _CRC32CX(crc1, k[1]); + + crc2 = k2; + _CRC32CX(crc2, k[3]); + crc3 = k2 >> 32; + _CRC32CX(crc3, k[4]); + + _CRC32CX(crc0, crc1); + _CRC32CX(crc2, crc3); + + crc0 ^= crc2; + + return crc0; +} + +static inline uint64_t +hash_crc_key48(void *key, __rte_unused uint32_t key_size, uint64_t seed) +{ + uint64_t *k = key, k0, k2, k5; + uint32_t crc0, crc1, crc2, crc3; + + k0 = k[0]; + k2 = k[2]; + k5 = k[5]; + + crc0 = k0; + _CRC32CX(crc0, seed); + crc1 = k0 >> 32; + _CRC32CX(crc1, k[1]); + + crc2 = k2; + _CRC32CX(crc2, k[3]); + crc3 = k2 >> 32; + _CRC32CX(crc3, k[4]); + + _CRC32CX(crc0, ((uint64_t)crc1 << 32) ^ crc2); + _CRC32CX(crc3, k5); + + crc0 ^= crc3; + + return crc0; +} + +static inline uint64_t +hash_crc_key56(void *key, __rte_unused uint32_t key_size, uint64_t seed) +{ + uint64_t *k = key, k0, k2, k5; + uint32_t crc0, crc1, crc2, crc3, crc4, crc5; + + k0 = k[0]; + k2 = k[2]; + k5 = k[5]; + + crc0 = k0; + _CRC32CX(crc0, seed); + crc1 = k0 >> 32; + _CRC32CX(crc1, k[1]); + + crc2 = k2; + _CRC32CX(crc2, k[3]); + crc3 = k2 >> 32; + _CRC32CX(crc3, k[4]); + + crc4 = k5; + _CRC32CX(crc4, k[6]); + crc5 = k5 >> 32; + + _CRC32CX(crc0, ((uint64_t)crc1 << 32) ^ crc2); + _CRC32CX(crc3, ((uint64_t)crc4 << 32) ^ crc5); + + crc0 ^= crc3; + + return crc0; +} + +static inline uint64_t +hash_crc_key64(void *key, __rte_unused uint32_t key_size, uint64_t seed) +{ + uint64_t *k = key, k0, k2, k5; + uint32_t crc0, crc1, crc2, crc3, crc4, crc5; + + k0 = k[0]; + k2 = k[2]; + k5 = k[5]; + + crc0 = k0; + _CRC32CX(crc0, seed); + crc1 = k0 >> 32; + _CRC32CX(crc1, k[1]); + + crc2 = k2; + _CRC32CX(crc2, k[3]); + crc3 = k2 >> 32; + _CRC32CX(crc3, k[4]); + + crc4 = k5; + _CRC32CX(crc4, k[6]); + crc5 = k5 >> 32; + _CRC32CX(crc5, k[7]); + + _CRC32CX(crc0, ((uint64_t)crc1 << 32) ^ crc2); + _CRC32CX(crc3, ((uint64_t)crc4 << 32) ^ crc5); + + crc0 ^= crc3; + + return crc0; +} + +#define hash_default_key8 hash_crc_key8 +#define hash_default_key16 hash_crc_key16 +#define hash_default_key24 hash_crc_key24 +#define hash_default_key32 hash_crc_key32 +#define hash_default_key40 hash_crc_key40 +#define hash_default_key48 hash_crc_key48 +#define hash_default_key56 hash_crc_key56 +#define hash_default_key64 hash_crc_key64 + +#endif
Implement the same hash functions with crc32 on arm platform. Signed-off-by: Jianbo Liu <jianbo.liu@linaro.org> --- examples/ip_pipeline/pipeline/hash_func.h | 2 + examples/ip_pipeline/pipeline/hash_func_arm64.h | 245 ++++++++++++++++++++++++ 2 files changed, 247 insertions(+) create mode 100644 examples/ip_pipeline/pipeline/hash_func_arm64.h -- 1.8.3.1