From patchwork Mon Nov 3 16:50:01 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 40048 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wg0-f70.google.com (mail-wg0-f70.google.com [74.125.82.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 3A2B521894 for ; Mon, 3 Nov 2014 16:50:17 +0000 (UTC) Received: by mail-wg0-f70.google.com with SMTP id x13sf6476024wgg.9 for ; Mon, 03 Nov 2014 08:50:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:sender:precedence:list-id:x-original-sender :x-original-authentication-results:mailing-list:list-post:list-help :list-archive:list-unsubscribe; bh=JTNobNUW1ZHg3JE9IHruYkksrkqu5Rz1trLfwAPVTH0=; b=G8iIR6QjnENpGVEBPdOqHo7LT6j/jYFwaeDqkQEM0gQclS1PJWTVSd4tHRld50PStg nTZ+k2Zx4lO5u2iSPEDhxaDDDbtRK26pr3lXTI/zQbzEwJKT4QGFB13YkKTKWQ5JCtIw yJVGGNwm2M4f8Zuj25WcMdibV+ybPOXHRvlkOkRfofTHN8Sbwn1hQqZxXyGtwl18FoA5 7ObtLBGLcZDK3zrB2PYNWLBKTUxkfXatTrUK6sSosrpMMQz+I8o5D/nTU81e7ib+m5vQ e1JUcDbg3rYRmkdiIjTrzdqArxOz0PW6SE3foGIGRusyzZPWiUYZbsEAa3NqUX9cTPW0 i00w== X-Gm-Message-State: ALoCoQkk0prkw++V8w16ulFHDZ3FMGbpYt9Y5RisjDqLxJ0m5aVyVZix79OTGcdC7EnURu770X+M X-Received: by 10.180.218.100 with SMTP id pf4mr1277537wic.4.1415033416438; Mon, 03 Nov 2014 08:50:16 -0800 (PST) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.120.4 with SMTP id ky4ls797918lab.23.gmail; Mon, 03 Nov 2014 08:50:16 -0800 (PST) X-Received: by 10.112.85.106 with SMTP id g10mr51775448lbz.38.1415033416246; Mon, 03 Nov 2014 08:50:16 -0800 (PST) Received: from mail-lb0-f181.google.com (mail-lb0-f181.google.com. [209.85.217.181]) by mx.google.com with ESMTPS id jj7si33201986lbc.65.2014.11.03.08.50.16 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Nov 2014 08:50:16 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.181 as permitted sender) client-ip=209.85.217.181; Received: by mail-lb0-f181.google.com with SMTP id l4so2887293lbv.40 for ; Mon, 03 Nov 2014 08:50:16 -0800 (PST) X-Received: by 10.112.218.101 with SMTP id pf5mr16082075lbc.46.1415033416133; Mon, 03 Nov 2014 08:50:16 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.141.34 with SMTP id rl2csp20074lbb; Mon, 3 Nov 2014 08:50:14 -0800 (PST) X-Received: by 10.68.135.33 with SMTP id pp1mr4433561pbb.120.1415033413649; Mon, 03 Nov 2014 08:50:13 -0800 (PST) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id cr2si15748885pbc.11.2014.11.03.08.50.13 for ; Mon, 03 Nov 2014 08:50:13 -0800 (PST) Received-SPF: none (google.com: linux-crypto-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752866AbaKCQuL (ORCPT ); Mon, 3 Nov 2014 11:50:11 -0500 Received: from mail-wg0-f45.google.com ([74.125.82.45]:59092 "EHLO mail-wg0-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752446AbaKCQuL (ORCPT ); Mon, 3 Nov 2014 11:50:11 -0500 Received: by mail-wg0-f45.google.com with SMTP id x12so11645535wgg.18 for ; Mon, 03 Nov 2014 08:50:09 -0800 (PST) X-Received: by 10.194.250.68 with SMTP id za4mr11327227wjc.83.1415033409475; Mon, 03 Nov 2014 08:50:09 -0800 (PST) Received: from ards-macbook-pro.local (cag06-7-83-153-85-71.fbx.proxad.net. [83.153.85.71]) by mx.google.com with ESMTPSA id ex2sm5773918wib.19.2014.11.03.08.50.08 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Nov 2014 08:50:08 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org, linux-arm-kernel@lists.infradead.org, herbert@gondor.apana.org.au, will.deacon@arm.com, catalin.marinas@arm.com, jussi.kivilinna@iki.fi Cc: Ard Biesheuvel Subject: [PATCH v2] arm64/crypto: use crypto instructions to generate AES key schedule Date: Mon, 3 Nov 2014 17:50:01 +0100 Message-Id: <1415033401-4722-1-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.3.2 Sender: linux-crypto-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.181 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , This patch implements the AES key schedule generation using ARMv8 Crypto Instructions. It replaces the table based C implementation in aes_generic.ko, which means we can drop the dependency on that module. Tested-by: Steve Capper Acked-by: Steve Capper Signed-off-by: Ard Biesheuvel --- v2: grammar/whitespace fixes, acked/tested-by added arch/arm64/crypto/Kconfig | 5 +- arch/arm64/crypto/aes-ce-ccm-glue.c | 4 +- arch/arm64/crypto/aes-ce-cipher.c | 112 +++++++++++++++++++++++++++++++++++- arch/arm64/crypto/aes-ce-setkey.h | 5 ++ arch/arm64/crypto/aes-glue.c | 18 ++++-- 5 files changed, 133 insertions(+), 11 deletions(-) create mode 100644 arch/arm64/crypto/aes-ce-setkey.h diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 5562652c5316..a38b02ce5f9a 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -27,20 +27,19 @@ config CRYPTO_AES_ARM64_CE tristate "AES core cipher using ARMv8 Crypto Extensions" depends on ARM64 && KERNEL_MODE_NEON select CRYPTO_ALGAPI - select CRYPTO_AES config CRYPTO_AES_ARM64_CE_CCM tristate "AES in CCM mode using ARMv8 Crypto Extensions" depends on ARM64 && KERNEL_MODE_NEON select CRYPTO_ALGAPI - select CRYPTO_AES + select CRYPTO_AES_ARM64_CE select CRYPTO_AEAD config CRYPTO_AES_ARM64_CE_BLK tristate "AES in ECB/CBC/CTR/XTS modes using ARMv8 Crypto Extensions" depends on ARM64 && KERNEL_MODE_NEON select CRYPTO_BLKCIPHER - select CRYPTO_AES + select CRYPTO_AES_ARM64_CE select CRYPTO_ABLK_HELPER config CRYPTO_AES_ARM64_NEON_BLK diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index 9e6cdde9b43d..0ac73b838fa3 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -16,6 +16,8 @@ #include #include +#include "aes-ce-setkey.h" + static int num_rounds(struct crypto_aes_ctx *ctx) { /* @@ -48,7 +50,7 @@ static int ccm_setkey(struct crypto_aead *tfm, const u8 *in_key, struct crypto_aes_ctx *ctx = crypto_aead_ctx(tfm); int ret; - ret = crypto_aes_expand_key(ctx, in_key, key_len); + ret = ce_aes_expandkey(ctx, in_key, key_len); if (!ret) return 0; diff --git a/arch/arm64/crypto/aes-ce-cipher.c b/arch/arm64/crypto/aes-ce-cipher.c index 2075e1acae6b..7f96e8af6a5a 100644 --- a/arch/arm64/crypto/aes-ce-cipher.c +++ b/arch/arm64/crypto/aes-ce-cipher.c @@ -14,6 +14,8 @@ #include #include +#include "aes-ce-setkey.h" + MODULE_DESCRIPTION("Synchronous AES cipher using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); @@ -124,6 +126,114 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const src[]) kernel_neon_end(); } +/* + * aes_sub() - use the aese instruction to perform the AES sbox substitution + * on each byte in 'input' + */ +static u32 aes_sub(u32 input) +{ + u32 ret; + + __asm__("dup v1.4s, %w[in] ;" + "movi v0.16b, #0 ;" + "aese v0.16b, v1.16b ;" + "umov %w[out], v0.4s[0] ;" + + : [out] "=r"(ret) + : [in] "r"(input) + : "v0","v1"); + + return ret; +} + +int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, + unsigned int key_len) +{ + /* + * The AES key schedule round constants + */ + static u8 const rcon[] = { + 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, + }; + + u32 kwords = key_len / sizeof(u32); + struct aes_block *key_enc, *key_dec; + int i, j; + + if (key_len != AES_KEYSIZE_128 && + key_len != AES_KEYSIZE_192 && + key_len != AES_KEYSIZE_256) + return -EINVAL; + + memcpy(ctx->key_enc, in_key, key_len); + ctx->key_length = key_len; + + kernel_neon_begin_partial(2); + for (i = 0; i < sizeof(rcon); i++) { + u32 *rki = ctx->key_enc + (i * kwords); + u32 *rko = rki + kwords; + + rko[0] = ror32(aes_sub(rki[kwords - 1]), 8) ^ rcon[i] ^ rki[0]; + rko[1] = rko[0] ^ rki[1]; + rko[2] = rko[1] ^ rki[2]; + rko[3] = rko[2] ^ rki[3]; + + if (key_len == AES_KEYSIZE_192) { + if (i >= 7) + break; + rko[4] = rko[3] ^ rki[4]; + rko[5] = rko[4] ^ rki[5]; + } else if (key_len == AES_KEYSIZE_256) { + if (i >= 6) + break; + rko[4] = aes_sub(rko[3]) ^ rki[4]; + rko[5] = rko[4] ^ rki[5]; + rko[6] = rko[5] ^ rki[6]; + rko[7] = rko[6] ^ rki[7]; + } + } + + /* + * Generate the decryption keys for the Equivalent Inverse Cipher. + * This involves reversing the order of the round keys, and applying + * the Inverse Mix Columns transformation on all but the first and + * the last one. + */ + key_enc = (struct aes_block *)ctx->key_enc; + key_dec = (struct aes_block *)ctx->key_dec; + j = num_rounds(ctx); + + key_dec[0] = key_enc[j]; + for (i = 1, j--; j > 0; i++, j--) + __asm__("ld1 {v0.16b}, %[in] ;" + "aesimc v1.16b, v0.16b ;" + "st1 {v1.16b}, %[out] ;" + + : [out] "=Q"(key_dec[i]) + : [in] "Q"(key_enc[j]) + : "v0","v1"); + key_dec[i] = key_enc[0]; + + kernel_neon_end(); + return 0; +} +EXPORT_SYMBOL(ce_aes_expandkey); + +int ce_aes_setkey(struct crypto_tfm *tfm, const u8 *in_key, + unsigned int key_len) +{ + struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm); + int ret; + + ret = ce_aes_expandkey(ctx, in_key, key_len); + if (!ret) + return 0; + + tfm->crt_flags |= CRYPTO_TFM_RES_BAD_KEY_LEN; + return -EINVAL; +} +EXPORT_SYMBOL(ce_aes_setkey); + static struct crypto_alg aes_alg = { .cra_name = "aes", .cra_driver_name = "aes-ce", @@ -135,7 +245,7 @@ static struct crypto_alg aes_alg = { .cra_cipher = { .cia_min_keysize = AES_MIN_KEY_SIZE, .cia_max_keysize = AES_MAX_KEY_SIZE, - .cia_setkey = crypto_aes_set_key, + .cia_setkey = ce_aes_setkey, .cia_encrypt = aes_cipher_encrypt, .cia_decrypt = aes_cipher_decrypt } diff --git a/arch/arm64/crypto/aes-ce-setkey.h b/arch/arm64/crypto/aes-ce-setkey.h new file mode 100644 index 000000000000..f08a6471d034 --- /dev/null +++ b/arch/arm64/crypto/aes-ce-setkey.h @@ -0,0 +1,5 @@ + +int ce_aes_setkey(struct crypto_tfm *tfm, const u8 *in_key, + unsigned int key_len); +int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, + unsigned int key_len); diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 79cd911ef88c..801aae32841f 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -16,9 +16,13 @@ #include #include +#include "aes-ce-setkey.h" + #ifdef USE_V8_CRYPTO_EXTENSIONS #define MODE "ce" #define PRIO 300 +#define aes_setkey ce_aes_setkey +#define aes_expandkey ce_aes_expandkey #define aes_ecb_encrypt ce_aes_ecb_encrypt #define aes_ecb_decrypt ce_aes_ecb_decrypt #define aes_cbc_encrypt ce_aes_cbc_encrypt @@ -30,6 +34,8 @@ MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions"); #else #define MODE "neon" #define PRIO 200 +#define aes_setkey crypto_aes_set_key +#define aes_expandkey crypto_aes_expand_key #define aes_ecb_encrypt neon_aes_ecb_encrypt #define aes_ecb_decrypt neon_aes_ecb_decrypt #define aes_cbc_encrypt neon_aes_cbc_encrypt @@ -79,10 +85,10 @@ static int xts_set_key(struct crypto_tfm *tfm, const u8 *in_key, struct crypto_aes_xts_ctx *ctx = crypto_tfm_ctx(tfm); int ret; - ret = crypto_aes_expand_key(&ctx->key1, in_key, key_len / 2); + ret = aes_expandkey(&ctx->key1, in_key, key_len / 2); if (!ret) - ret = crypto_aes_expand_key(&ctx->key2, &in_key[key_len / 2], - key_len / 2); + ret = aes_expandkey(&ctx->key2, &in_key[key_len / 2], + key_len / 2); if (!ret) return 0; @@ -288,7 +294,7 @@ static struct crypto_alg aes_algs[] = { { .min_keysize = AES_MIN_KEY_SIZE, .max_keysize = AES_MAX_KEY_SIZE, .ivsize = AES_BLOCK_SIZE, - .setkey = crypto_aes_set_key, + .setkey = aes_setkey, .encrypt = ecb_encrypt, .decrypt = ecb_decrypt, }, @@ -306,7 +312,7 @@ static struct crypto_alg aes_algs[] = { { .min_keysize = AES_MIN_KEY_SIZE, .max_keysize = AES_MAX_KEY_SIZE, .ivsize = AES_BLOCK_SIZE, - .setkey = crypto_aes_set_key, + .setkey = aes_setkey, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, }, @@ -324,7 +330,7 @@ static struct crypto_alg aes_algs[] = { { .min_keysize = AES_MIN_KEY_SIZE, .max_keysize = AES_MAX_KEY_SIZE, .ivsize = AES_BLOCK_SIZE, - .setkey = crypto_aes_set_key, + .setkey = aes_setkey, .encrypt = ctr_encrypt, .decrypt = ctr_encrypt, },