| Message ID | 20201217185516.26969-1-ardb@kernel.org |
|---|---|
| State | Accepted |
| Commit | 15deb4333cd6d4e1e3216582e4c531ec40a6b060 |
| Headers | show |
| Series | [1/2] crypto: arm64/aes-ce - really hide slower algos when faster ones are enabled | expand |
On Thu, Dec 17, 2020 at 07:55:15PM +0100, Ard Biesheuvel wrote: > Commit 69b6f2e817e5b ("crypto: arm64/aes-neon - limit exposed routines if > faster driver is enabled") intended to hide modes from the plain NEON > driver that are also implemented by the faster bit sliced NEON one if > both are enabled. However, the defined() CPP function does not detect > if the bit sliced NEON driver is enabled as a module. So instead, let's > use IS_ENABLED() here. > > Fixes: 69b6f2e817e5b ("crypto: arm64/aes-neon - limit exposed routines if ...") > Signed-off-by: Ard Biesheuvel <ardb@kernel.org> > --- > arch/arm64/crypto/aes-glue.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) Patch applied. Thanks. -- Email: Herbert Xu <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Thu, Dec 17, 2020 at 07:55:16PM +0100, Ard Biesheuvel wrote: > Counter mode is a stream cipher chaining mode that is typically used > with inputs that are of arbitrarily length, and so a tail block which > is smaller than a full AES block is rule rather than exception. > > The current ctr(aes) implementation for arm64 always makes a separate > call into the assembler routine to process this tail block, which is > suboptimal, given that it requires reloading of the AES round keys, > and prevents us from handling this tail block using the 5-way stride > that we use for better performance on deep pipelines. > > So let's update the assembler routine so it can handle any input size, > and uses NEON permutation instructions and overlapping loads and stores > to handle the tail block. This results in a ~16% speedup for 1420 byte > blocks on cores with deep pipelines such as ThunderX2. > > Signed-off-by: Ard Biesheuvel <ardb@kernel.org> > --- > arch/arm64/crypto/aes-glue.c | 46 +++--- > arch/arm64/crypto/aes-modes.S | 165 +++++++++++++------- > 2 files changed, 137 insertions(+), 74 deletions(-) Patch applied. Thanks. -- Email: Herbert Xu <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 34b8a89197be..cafb5b96be0e 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -55,7 +55,7 @@ MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions"); #define aes_mac_update neon_aes_mac_update MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 NEON"); #endif -#if defined(USE_V8_CRYPTO_EXTENSIONS) || !defined(CONFIG_CRYPTO_AES_ARM64_BS) +#if defined(USE_V8_CRYPTO_EXTENSIONS) || !IS_ENABLED(CONFIG_CRYPTO_AES_ARM64_BS) MODULE_ALIAS_CRYPTO("ecb(aes)"); MODULE_ALIAS_CRYPTO("cbc(aes)"); MODULE_ALIAS_CRYPTO("ctr(aes)"); @@ -650,7 +650,7 @@ static int __maybe_unused xts_decrypt(struct skcipher_request *req) } static struct skcipher_alg aes_algs[] = { { -#if defined(USE_V8_CRYPTO_EXTENSIONS) || !defined(CONFIG_CRYPTO_AES_ARM64_BS) +#if defined(USE_V8_CRYPTO_EXTENSIONS) || !IS_ENABLED(CONFIG_CRYPTO_AES_ARM64_BS) .base = { .cra_name = "__ecb(aes)", .cra_driver_name = "__ecb-aes-" MODE,
Commit 69b6f2e817e5b ("crypto: arm64/aes-neon - limit exposed routines if faster driver is enabled") intended to hide modes from the plain NEON driver that are also implemented by the faster bit sliced NEON one if both are enabled. However, the defined() CPP function does not detect if the bit sliced NEON driver is enabled as a module. So instead, let's use IS_ENABLED() here. Fixes: 69b6f2e817e5b ("crypto: arm64/aes-neon - limit exposed routines if ...") Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- arch/arm64/crypto/aes-glue.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)