From patchwork Thu Feb 10 23:28:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nathan Huckleberry X-Patchwork-Id: 541544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92075C433FE for ; Thu, 10 Feb 2022 23:28:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345685AbiBJX2k (ORCPT ); Thu, 10 Feb 2022 18:28:40 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:41260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345624AbiBJX2k (ORCPT ); Thu, 10 Feb 2022 18:28:40 -0500 Received: from mail-ua1-x949.google.com (mail-ua1-x949.google.com [IPv6:2607:f8b0:4864:20::949]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7711F5F5B for ; Thu, 10 Feb 2022 15:28:39 -0800 (PST) Received: by mail-ua1-x949.google.com with SMTP id z4-20020a9f3704000000b0030bb302b19dso3550700uad.11 for ; Thu, 10 Feb 2022 15:28:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=hMLABJ99eOAvi/mzHi8TOvXnAJT7y1QJHMEUiSzJRyA=; b=h5r1dLJAVwLcHRThB7P468t18WUMTHgtU7vjJI1al922TynvB0jopSwpau6Qg84bpJ xkfE46S1MjW5s3BUuIWmwDl5DVq+uDkBY+hKdZ8wXO6dXz8UnZzYZzkXZVRQXgVXxNEC qNLA1lZtGxZ74PUUUtW43StMwcMKdg+7I8GC0prG1gm0LvNOSz72DvUgwTN9oa6C11zz /6p1ME3wmYUs9KATcOPfo7AUWZGzWuzqGD3AHZHVX7YkGOJ+TP55Q0iy3d5nGpurNMpT d74Pc51aicWjhjbQlLa0os1qoYsolql/MuiuicANEpRtGdt5HET716/yrv484q5dN6t5 WYqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=hMLABJ99eOAvi/mzHi8TOvXnAJT7y1QJHMEUiSzJRyA=; b=IHk0Q99BfBfSSDM/5B+gCrAJHodOp66B8GSh8Z058q2XisKcTpM8xn6qmHY78wlshM gnfBcYD/KO081uGhJwy2fwcHecDmG+FOgNVjxmaH42HWz1dPFhKUkHmPxg1ZyOzwrkxd r29y4JHMQJJw4JDcuw/BTOPjZk42GfGTulR5V0aztVrCozydXSxwV7A7TN31b9MfUaDQ N8VEFML2b22OfvDmHUZB9N4c2q7TRyE3JIhiXJzxU1SCB1SCG1p52gIhxZ9nZWbLQoPE OxFxjLAk/kMN7baS8AHvoKwD5d9LIBfKUAawDVrfyPeuytkUaSJZ7RbA/ho0mDW0Lb6+ 7tug== X-Gm-Message-State: AOAM531d8podT7j+4R0valjKYNUhTHQLODJ4c20ZM+skglOu+4p38ZEs LPGfbO0avusg+idxWTfCiBKpmHlVVrLNljhNIKwzSZczv97bbq73qvv5fE2i2IuQJ2cVV8jl5uC P3gI3faVao/ZYCKTdjTNAHq1jUEzMktREIil6ty8mJ+LAE8+RqM5xpTHV24aRUnspF1A= X-Google-Smtp-Source: ABdhPJxbcp3VuiZ0f+3SeDdptIhMvEd+qOnhM68C86J6K1lZEWSMcPJQQrTBoGwLq2P0zYvkIUc8CKd7ow== X-Received: from nhuck.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:39cc]) (user=nhuck job=sendgmr) by 2002:a05:6122:7c7:: with SMTP id l7mr3487351vkr.9.1644535718486; Thu, 10 Feb 2022 15:28:38 -0800 (PST) Date: Thu, 10 Feb 2022 23:28:07 +0000 In-Reply-To: <20220210232812.798387-1-nhuck@google.com> Message-Id: <20220210232812.798387-3-nhuck@google.com> Mime-Version: 1.0 References: <20220210232812.798387-1-nhuck@google.com> X-Mailer: git-send-email 2.35.1.265.g69c8d7142f-goog Subject: [RFC PATCH v2 2/7] crypto: polyval - Add POLYVAL support From: Nathan Huckleberry To: linux-crypto@vger.kernel.org Cc: Herbert Xu , "David S. Miller" , linux-arm-kernel@lists.infradead.org, Paul Crowley , Eric Biggers , Sami Tolvanen , Ard Biesheuvel , Nathan Huckleberry Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add support for POLYVAL, an ε-Δ-universal hash function similar to GHASH. POLYVAL is used as a component to implement HCTR2 mode. POLYVAL is implemented as an shash algorithm. The implementation is modified from ghash-generic.c. More information on POLYVAL can be found in the HCTR2 paper: https://eprint.iacr.org/2021/1441.pdf Signed-off-by: Nathan Huckleberry --- Changes since v1: * Added comments explaining the GHASH/POLYVAL trick * Fixed a bug on non-block-multiple messages crypto/Kconfig | 8 ++ crypto/Makefile | 1 + crypto/polyval-generic.c | 199 +++++++++++++++++++++++++++ crypto/tcrypt.c | 4 + crypto/testmgr.c | 6 + crypto/testmgr.h | 284 +++++++++++++++++++++++++++++++++++++++ include/crypto/polyval.h | 22 +++ 7 files changed, 524 insertions(+) create mode 100644 crypto/polyval-generic.c create mode 100644 include/crypto/polyval.h diff --git a/crypto/Kconfig b/crypto/Kconfig index 8543f34fa200..0c61d03530a6 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -760,6 +760,14 @@ config CRYPTO_GHASH GHASH is the hash function used in GCM (Galois/Counter Mode). It is not a general-purpose cryptographic hash function. +config CRYPTO_POLYVAL + tristate + select CRYPTO_GF128MUL + select CRYPTO_HASH + help + POLYVAL is the hash function used in HCTR2. It is not a general-purpose + cryptographic hash function. + config CRYPTO_POLY1305 tristate "Poly1305 authenticator algorithm" select CRYPTO_HASH diff --git a/crypto/Makefile b/crypto/Makefile index 6b3fe3df1489..561f901a91d4 100644 --- a/crypto/Makefile +++ b/crypto/Makefile @@ -169,6 +169,7 @@ UBSAN_SANITIZE_jitterentropy.o = n jitterentropy_rng-y := jitterentropy.o jitterentropy-kcapi.o obj-$(CONFIG_CRYPTO_TEST) += tcrypt.o obj-$(CONFIG_CRYPTO_GHASH) += ghash-generic.o +obj-$(CONFIG_CRYPTO_POLYVAL) += polyval-generic.o obj-$(CONFIG_CRYPTO_USER_API) += af_alg.o obj-$(CONFIG_CRYPTO_USER_API_HASH) += algif_hash.o obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o diff --git a/crypto/polyval-generic.c b/crypto/polyval-generic.c new file mode 100644 index 000000000000..e8d7e34e5355 --- /dev/null +++ b/crypto/polyval-generic.c @@ -0,0 +1,199 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * POLYVAL: hash function for HCTR2. + * + * Copyright (c) 2007 Nokia Siemens Networks - Mikko Herranen + * Copyright (c) 2009 Intel Corp. + * Author: Huang Ying + * Copyright 2021 Google LLC + */ + +/* + * Code based on crypto/ghash-generic.c + * + * POLYVAL is a keyed hash function similar to GHASH. POLYVAL uses a + * different modulus for finite field multiplication which makes hardware + * accelerated implementations on little-endian machines faster. + * + * Like GHASH, POLYVAL is not a cryptographic hash function and should + * not be used outside of crypto modes explicitly designed to use POLYVAL. + * + * This implementation uses a convenient trick involving the GHASH and POLYVAL + * fields. This trick allows multiplication in the POLYVAL field to be + * implemented by using multiplication in the GHASH field as a subroutine. An + * element of the POLYVAL field can be converted to an element of the GHASH + * field by computing x*REVERSE(a), where REVERSE reverses the byte-ordering of + * a. Similarly, an element of the GHASH field can be converted back to the + * POLYVAL field by computing REVERSE(x^{-1}*a). + * + * By using this trick, we do not need to implement the POLYVAL field for the + * generic implementation. + * + * Warning: this generic implementation is not intended to be used in practice + * and is not constant time. For practical use, a hardware accelerated + * implementation of POLYVAL should be used instead. + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct polyval_tfm_ctx { + struct gf128mul_4k *gf128; +}; + +static int polyval_init(struct shash_desc *desc) +{ + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + + memset(dctx, 0, sizeof(*dctx)); + + return 0; +} + +static void reverse_block(u8 block[POLYVAL_BLOCK_SIZE]) +{ + u64 *p1 = (u64 *)block; + u64 *p2 = (u64 *)&block[8]; + u64 a = get_unaligned(p1); + u64 b = get_unaligned(p2); + + put_unaligned(swab64(a), p2); + put_unaligned(swab64(b), p1); +} + +static int polyval_setkey(struct crypto_shash *tfm, + const u8 *key, unsigned int keylen) +{ + struct polyval_tfm_ctx *ctx = crypto_shash_ctx(tfm); + be128 k; + + if (keylen != POLYVAL_BLOCK_SIZE) + return -EINVAL; + + gf128mul_free_4k(ctx->gf128); + + BUILD_BUG_ON(sizeof(k) != POLYVAL_BLOCK_SIZE); + // avoid violating alignment rules + memcpy(&k, key, POLYVAL_BLOCK_SIZE); + + reverse_block((u8 *)&k); + gf128mul_x_lle(&k, &k); + + ctx->gf128 = gf128mul_init_4k_lle(&k); + memzero_explicit(&k, POLYVAL_BLOCK_SIZE); + + if (!ctx->gf128) + return -ENOMEM; + + return 0; +} + +static int polyval_update(struct shash_desc *desc, + const u8 *src, unsigned int srclen) +{ + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + const struct polyval_tfm_ctx *ctx = crypto_shash_ctx(desc->tfm); + u8 *dst = dctx->buffer; + u8 *pos; + u8 tmp[POLYVAL_BLOCK_SIZE]; + int n; + + if (dctx->bytes) { + n = min(srclen, dctx->bytes); + pos = dst + dctx->bytes - 1; + + dctx->bytes -= n; + srclen -= n; + + while (n--) + *pos-- ^= *src++; + + if (!dctx->bytes) + gf128mul_4k_lle((be128 *)dst, ctx->gf128); + } + + while (srclen >= POLYVAL_BLOCK_SIZE) { + memcpy(tmp, src, POLYVAL_BLOCK_SIZE); + reverse_block(tmp); + crypto_xor(dst, tmp, POLYVAL_BLOCK_SIZE); + gf128mul_4k_lle((be128 *)dst, ctx->gf128); + src += POLYVAL_BLOCK_SIZE; + srclen -= POLYVAL_BLOCK_SIZE; + } + + if (srclen) { + dctx->bytes = POLYVAL_BLOCK_SIZE - srclen; + pos = dst + POLYVAL_BLOCK_SIZE - 1; + while (srclen--) + *pos-- ^= *src++; + } + + return 0; +} + +static int polyval_final(struct shash_desc *desc, u8 *dst) +{ + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + const struct polyval_tfm_ctx *ctx = crypto_shash_ctx(desc->tfm); + u8 *buf = dctx->buffer; + + if (dctx->bytes) + gf128mul_4k_lle((be128 *)buf, ctx->gf128); + dctx->bytes = 0; + + reverse_block(buf); + memcpy(dst, buf, POLYVAL_BLOCK_SIZE); + + return 0; +} + +static void polyval_exit_tfm(struct crypto_tfm *tfm) +{ + struct polyval_tfm_ctx *ctx = crypto_tfm_ctx(tfm); + + gf128mul_free_4k(ctx->gf128); +} + +static struct shash_alg polyval_alg = { + .digestsize = POLYVAL_DIGEST_SIZE, + .init = polyval_init, + .update = polyval_update, + .final = polyval_final, + .setkey = polyval_setkey, + .descsize = sizeof(struct polyval_desc_ctx), + .base = { + .cra_name = "polyval", + .cra_driver_name = "polyval-generic", + .cra_priority = 100, + .cra_blocksize = POLYVAL_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct polyval_tfm_ctx), + .cra_module = THIS_MODULE, + .cra_exit = polyval_exit_tfm, + }, +}; + +static int __init polyval_mod_init(void) +{ + return crypto_register_shash(&polyval_alg); +} + +static void __exit polyval_mod_exit(void) +{ + crypto_unregister_shash(&polyval_alg); +} + +subsys_initcall(polyval_mod_init); +module_exit(polyval_mod_exit); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("POLYVAL hash function"); +MODULE_ALIAS_CRYPTO("polyval"); +MODULE_ALIAS_CRYPTO("polyval-generic"); diff --git a/crypto/tcrypt.c b/crypto/tcrypt.c index b3a23dbf5b14..ced7467bb481 100644 --- a/crypto/tcrypt.c +++ b/crypto/tcrypt.c @@ -1730,6 +1730,10 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb) ret += tcrypt_test("ccm(sm4)"); break; + case 57: + ret += tcrypt_test("polyval"); + break; + case 100: ret += tcrypt_test("hmac(md5)"); break; diff --git a/crypto/testmgr.c b/crypto/testmgr.c index d2b42ff0b04a..3e54d17fe644 100644 --- a/crypto/testmgr.c +++ b/crypto/testmgr.c @@ -5245,6 +5245,12 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(poly1305_tv_template) } + }, { + .alg = "polyval", + .test = alg_test_hash, + .suite = { + .hash = __VECS(polyval_tv_template) + } }, { .alg = "rfc3686(ctr(aes))", .test = alg_test_skcipher, diff --git a/crypto/testmgr.h b/crypto/testmgr.h index e1ebbb3c4d4c..da3736e51982 100644 --- a/crypto/testmgr.h +++ b/crypto/testmgr.h @@ -33346,4 +33346,288 @@ static const struct cipher_testvec aes_xctr_tv_template[] = { }, }; +/* + * Test vectors generated using https://github.com/google/hctr2 + */ +static const struct hash_testvec polyval_tv_template[] = { + { + .key = "\x31\x07\x28\xd9\x91\x1f\x1f\x38" + "\x37\xb2\x43\x16\xc3\xfa\xb9\xa0", + .plaintext = "\x65\x78\x61\x6d\x70\x6c\x65\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x48\x65\x6c\x6c\x6f\x20\x77\x6f" + "\x72\x6c\x64\x00\x00\x00\x00\x00" + "\x38\x00\x00\x00\x00\x00\x00\x00" + "\x58\x00\x00\x00\x00\x00\x00\x00", + .digest = "\xad\x7f\xcf\x0b\x51\x69\x85\x16" + "\x62\x67\x2f\x3c\x5f\x95\x13\x8f", + .psize = 48, + .ksize = 16, + }, + { + .key = "\xd9\xb3\x60\x27\x96\x94\x94\x1a" + "\xc5\xdb\xc6\x98\x7a\xda\x73\x77", + .plaintext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .digest = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .psize = 16, + .ksize = 16, + }, + { + .key = "\xd9\xb3\x60\x27\x96\x94\x94\x1a" + "\xc5\xdb\xc6\x98\x7a\xda\x73\x77", + .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x40\x00\x00\x00\x00\x00\x00\x00", + .digest = "\xeb\x93\xb7\x74\x09\x62\xc5\xe4" + "\x9d\x2a\x90\xa7\xdc\x5c\xec\x74", + .psize = 32, + .ksize = 16, + }, + { + .key = "\xd9\xb3\x60\x27\x96\x94\x94\x1a" + "\xc5\xdb\xc6\x98\x7a\xda\x73\x77", + .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x60\x00\x00\x00\x00\x00\x00\x00", + .digest = "\x48\xeb\x6c\x6c\x5a\x2d\xbe\x4a" + "\x1d\xde\x50\x8f\xee\x06\x36\x1b", + .psize = 32, + .ksize = 16, + }, + { + .key = "\xd9\xb3\x60\x27\x96\x94\x94\x1a" + "\xc5\xdb\xc6\x98\x7a\xda\x73\x77", + .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x02\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x01\x00\x00\x00\x00\x00\x00", + .digest = "\xce\x6e\xdc\x9a\x50\xb3\x6d\x9a" + "\x98\x98\x6b\xbf\x6a\x26\x1c\x3b", + .psize = 48, + .ksize = 16, + }, + { + .key = "\xd9\xb3\x60\x27\x96\x94\x94\x1a" + "\xc5\xdb\xc6\x98\x7a\xda\x73\x77", + .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x02\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x03\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x80\x01\x00\x00\x00\x00\x00\x00", + .digest = "\x81\x38\x87\x46\xbc\x22\xd2\x6b" + "\x2a\xbc\x3d\xcb\x15\x75\x42\x22", + .psize = 64, + .ksize = 16, + }, + { + .key = "\xd9\xb3\x60\x27\x96\x94\x94\x1a" + "\xc5\xdb\xc6\x98\x7a\xda\x73\x77", + .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x02\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x03\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x04\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x02\x00\x00\x00\x00\x00\x00", + .digest = "\x1e\x39\xb6\xd3\x34\x4d\x34\x8f" + "\x60\x44\xf8\x99\x35\xd1\xcf\x78", + .psize = 80, + .ksize = 16, + }, + { + .key = "\xd9\xb3\x60\x27\x96\x94\x94\x1a" + "\xc5\xdb\xc6\x98\x7a\xda\x73\x77", + .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x02\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x03\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x08\x00\x00\x00\x00\x00\x00\x00" + "\x00\x01\x00\x00\x00\x00\x00\x00", + .digest = "\x2c\xe7\xda\xaf\x7c\x89\x49\x08" + "\x22\x05\x12\x55\xb1\x2e\xca\x6b", + .psize = 64, + .ksize = 16, + }, + { + .key = "\xd9\xb3\x60\x27\x96\x94\x94\x1a" + "\xc5\xdb\xc6\x98\x7a\xda\x73\x77", + .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x02\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x03\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x04\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x08\x00\x00\x00\x00\x00\x00\x00" + "\x80\x01\x00\x00\x00\x00\x00\x00", + .digest = "\x9c\xa9\x87\x71\x5d\x69\xc1\x78" + "\x67\x11\xdf\xcd\x22\xf8\x30\xfc", + .psize = 80, + .ksize = 16, + }, + { + .key = "\xd9\xb3\x60\x27\x96\x94\x94\x1a" + "\xc5\xdb\xc6\x98\x7a\xda\x73\x77", + .plaintext = "\x01\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x02\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x03\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x04\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x05\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x08\x00\x00\x00\x00\x00\x00\x00" + "\x00\x02\x00\x00\x00\x00\x00\x00", + .digest = "\xff\xcd\x05\xd5\x77\x0f\x34\xad" + "\x92\x67\xf0\xa5\x99\x94\xb1\x5a", + .psize = 96, + .ksize = 16, + }, + { + .key = "\x03\x6e\xe1\xfe\x2d\x79\x26\xaf" + "\x68\x89\x80\x95\xe5\x4e\x7b\x3c", + .plaintext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .digest = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .psize = 16, + .ksize = 16, + }, + { + .key = "\x37\x24\xf5\x5f\x1d\x22\xac\x0a" + "\xb8\x30\xda\x0b\x6a\x99\x5d\x74", + .plaintext = "\x75\x76\xf7\x02\x8e\xc6\xeb\x5e" + "\xa7\xe2\x98\x34\x2a\x94\xd4\xb2" + "\x02\xb3\x70\xef\x97\x68\xec\x65" + "\x61\xc4\xfe\x6b\x7e\x72\x96\xfa" + "\x85\x9c\x21\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\xe4\x2a\x3c\x02\xc2\x5b\x64\x86" + "\x9e\x14\x6d\x7b\x23\x39\x87\xbd" + "\xdf\xc2\x40\x87\x1d\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x18\x01\x00\x00\x00\x00\x00\x00" + "\xa8\x00\x00\x00\x00\x00\x00\x00", + .digest = "\x4c\xbb\xa0\x90\xf0\x3f\x7d\x11" + "\x88\xea\x55\x74\x9f\xa6\xc7\xbd", + .psize = 96, + .ksize = 16, + }, + { + .key = "\x90\xcc\xac\xee\xba\xd7\xd4\x68" + "\x98\xa6\x79\x70\xdf\x66\x15\x6c", + .plaintext = "", + .digest = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .psize = 0, + .ksize = 16, + }, + { + .key = "\x89\xc9\x4b\xde\x40\xa6\xf9\x62" + "\x58\x04\x51\x26\xb4\xb1\x14\xe4", + .plaintext = "", + .digest = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .psize = 0, + .ksize = 16, + }, + { + .key = "\x37\xbe\x68\x16\x50\xb9\x4e\xb0" + "\x47\xde\xe2\xbd\xde\xe4\x48\x09", + .plaintext = "\x87\xfc\x68\x9f\xff\xf2\x4a\x1e" + "\x82\x3b\x73\x8f\xc1\xb2\x1b\x7a" + "\x6c\x4f\x81\xbc\x88\x9b\x6c\xa3" + "\x9c\xc2\xa5\xbc\x14\x70\x4c\x9b" + "\x0c\x9f\x59\x92\x16\x4b\x91\x3d" + "\x18\x55\x22\x68\x12\x8c\x63\xb2" + "\x51\xcb\x85\x4b\xd2\xae\x0b\x1c" + "\x5d\x28\x9d\x1d\xb1\xc8\xf0\x77" + "\xe9\xb5\x07\x4e\x06\xc8\xee\xf8" + "\x1b\xed\x72\x2a\x55\x7d\x16\xc9" + "\xf2\x54\xe7\xe9\xe0\x44\x5b\x33" + "\xb1\x49\xee\xff\x43\xfb\x82\xcd" + "\x4a\x70\x78\x81\xa4\x34\x36\xe8" + "\x4c\x28\x54\xa6\x6c\xc3\x6b\x78" + "\xe7\xc0\x5d\xc6\x5d\x81\xab\x70" + "\x08\x86\xa1\xfd\xf4\x77\x55\xfd" + "\xa3\xe9\xe2\x1b\xdf\x99\xb7\x80" + "\xf9\x0a\x4f\x72\x4a\xd3\xaf\xbb" + "\xb3\x3b\xeb\x08\x58\x0f\x79\xce" + "\xa5\x99\x05\x12\x34\xd4\xf4\x86" + "\x37\x23\x1d\xc8\x49\xc0\x92\xae" + "\xa6\xac\x9b\x31\x55\xed\x15\xc6" + "\x05\x17\x37\x8d\x90\x42\xe4\x87" + "\x89\x62\x88\x69\x1c\x6a\xfd\xe3" + "\x00\x2b\x47\x1a\x73\xc1\x51\xc2" + "\xc0\x62\x74\x6a\x9e\xb2\xe5\x21" + "\xbe\x90\xb5\xb0\x50\xca\x88\x68" + "\xe1\x9d\x7a\xdf\x6c\xb7\xb9\x98" + "\xee\x28\x62\x61\x8b\xd1\x47\xf9" + "\x04\x7a\x0b\x5d\xcd\x2b\x65\xf5" + "\x12\xa3\xfe\x1a\xaa\x2c\x78\x42" + "\xb8\xbe\x7d\x74\xeb\x59\xba\xba", + .digest = "\xae\x11\xd4\x60\x2a\x5f\x9e\x42" + "\x89\x04\xc2\x34\x8d\x55\x94\x0a", + .psize = 256, + .ksize = 16, + }, + { + .key = "\xc8\x53\xde\xaa\xb1\x4b\x6b\xd5" + "\x88\xd6\x4c\xe9\xba\x35\x3d\x5a", + .plaintext = "\xc1\xeb\xba\x8d\xb7\x20\x09\xe0" + "\x28\x4f\x29\xf3\xd8\x26\x50\x40" + "\xd9\x06\xa8\xa8\xc0\xbe\xf0\xfb" + "\x75\x7c\x02\x86\x16\x83\x9d\x65" + "\x8f\x5e\xc4\x58\xed\x6a\xb3\x10" + "\xd2\xf7\x23\xc2\x4a\xb0\x00\x6a" + "\x01\x7c\xf7\xf7\x69\x42\xb2\x12" + "\xb0\xeb\x65\x07\xd7\x8e\x2d\x27" + "\x67\xa2\x57\xf0\x49\x0f\x3f\x0e" + "\xc9\xf7\x1b\xe0\x5b\xdd\x87\xfb" + "\x89\xd1\xfa\xb1\x46\xaf\xa2\x93" + "\x01\x65\xb6\x6f\xbe\x29\x7d\x9f" + "\xfa\xf5\x58\xc6\xb5\x92\x55\x25" + "\x4c\xb5\x0c\xc2\x61\x9f\xc4\xb1" + "\x7f\xe3\x61\x18\x3f\x8c\xb2\xd6" + "\xfd\x9f\xd8\xe5\x3d\x03\x05\xa2" + "\x5d\x1a\xa8\xf0\x04\x41\xea\xa6" + "\x07\x67\x86\x00\xe8\x86\xfc\xb1" + "\xc3\x15\x3e\xc8\x84\x2e\x5e\x5f" + "\x7b\x75\x6a\xc4\x48\xb4\xee\x5f" + "\xe9\x76\xdf\xe6\x1a\xd4\x15\x92" + "\x23\x03\x06\xc1\x2d\x0f\x94\xcb" + "\xe6\x5e\x18\xa6\x3b\x38\x1f\xc2" + "\x28\x73\x8a\xbd\x3a\x6f\xb0\x95" + "\x0f\x1c\xc7\xdf\x10\x0b\x2a\x7d" + "\xf9\x6b\xe1\x4a\xfb\xe1\x07\xc9" + "\x69\x7b\x27\x65\xc0\x08\x49\xc0" + "\xf3\x0b\x5b\xa6\x8b\xf7\x1a\xfe" + "\xe3\x9f\x87\x1d\x68\x07\xf4\x53" + "\x8d\x54\xe9\x3f\xd5\x02\x3a\x09" + "\x72\xa9\x84\xdc\x25\xd3\xad\xdb" + "\x4e\x45\x4f\x7f\xe8\x02\x69\x45", + .digest = "\x7b\x4f\x29\xb3\x0b\x4d\x2b\xa3" + "\x40\xc8\x56\x5a\x0a\xcf\xbd\x9b", + .psize = 256, + .ksize = 16, + }, +}; + #endif /* _CRYPTO_TESTMGR_H */ diff --git a/include/crypto/polyval.h b/include/crypto/polyval.h new file mode 100644 index 000000000000..fd0c6e124b65 --- /dev/null +++ b/include/crypto/polyval.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Common values for the Polyval hash algorithm + * + * Copyright 2021 Google LLC + */ + +#ifndef _CRYPTO_POLYVAL_H +#define _CRYPTO_POLYVAL_H + +#include +#include + +#define POLYVAL_BLOCK_SIZE 16 +#define POLYVAL_DIGEST_SIZE 16 + +struct polyval_desc_ctx { + u8 buffer[POLYVAL_BLOCK_SIZE]; + u32 bytes; +}; + +#endif From patchwork Thu Feb 10 23:28:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nathan Huckleberry X-Patchwork-Id: 541543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F0C5C4332F for ; Thu, 10 Feb 2022 23:28:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345650AbiBJX2u (ORCPT ); Thu, 10 Feb 2022 18:28:50 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:41322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345690AbiBJX2o (ORCPT ); Thu, 10 Feb 2022 18:28:44 -0500 Received: from mail-vk1-xa49.google.com (mail-vk1-xa49.google.com [IPv6:2607:f8b0:4864:20::a49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C8C915F5B for ; Thu, 10 Feb 2022 15:28:44 -0800 (PST) Received: by mail-vk1-xa49.google.com with SMTP id w78-20020a1fad51000000b0032823aa16b6so996302vke.7 for ; Thu, 10 Feb 2022 15:28:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=LFn5HsQG6yE4p0N4NsfZ9UVbRekmJnl7KohBN08JwfM=; b=MqunerhqEQY9+Vidv+33xbfQP6h0LUfsliuqwFe5UCWlAJIQvm9mWUgQPr5rJlt6nX B8oivp6F3fRLW2rdD1fPPelbQDkU+QkCAuYm+QS0RfMOXLobiCGgxJx8QkyB22iKhnpy Z/gxNWtHVcCTQb5Gy+uEECOHKL1wXQIP3cwLq+Yj07ftJWRqXr2gDYPGpLiTMsA6YfDG vs3jnIJTqPNEYYJv0f7f9xEYFbiukxUSStYmx/u3cU987yxTzXlpWlOf9+p8i2fL7kMB 8IbunbtPrcYLZg4bE2EkkayGPE87+gw5eBHJmEwx2XzMz9igy5NBjNLzqYNZDETCPPyq MWug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=LFn5HsQG6yE4p0N4NsfZ9UVbRekmJnl7KohBN08JwfM=; b=qbFsHE+Vsv0fCPhwDJnUplWaL5xfdA9cZpC4mcBtlZ7z3o+PAx+L/Z8u40q9KfWq9O q0aeVdiDNymSUU1h3ZJoxgeCPkaRHQmUe4dT8bwlIEehYky1+HkC7cxPn61P48ooIBUP z5wyxzBArZcTlfoKIBfZihhzb6KnjFlEaQmoPLwfejiXWcd5zmr2b6SuJ8TKZGXikr2F R/zkKREiK6X/94xX1KeEvFhcQtDYRvrSbQNiN9KpmIyhstN86DK40P8GX+k1HwAu1eJO nQfPa9MLWBZU1Hw2G8szO+YCz23T2ey9uuXYqX9qGvVN/fenEUwRROCu01+5/sIdi+kV GRbg== X-Gm-Message-State: AOAM5302Xw/n902+dNnsIkfwu3l5d3H8WsfaqcMC7chccXpddT74kSzZ zokhNHSZj9oHPl4qZuiXTc8SXzn8SPhDmFNGiJru8FC7olHz5bYFKuqnpYW0S0bBrC3o8iPOvFg x+FyJ0FtNINJhUhSzJtj57EEq+Y2bhhHpo7pWGs5CS6Y24RR/vhdKcbBzV4phoLqZ9XM= X-Google-Smtp-Source: ABdhPJznK+CUJbs5EjuSA9KUzmEcqeaxALCXxOk9kaOi3iurlmCd8wfGHt7j8HzJjBdocTRDIjlGmWAG9w== X-Received: from nhuck.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:39cc]) (user=nhuck job=sendgmr) by 2002:ab0:59e2:: with SMTP id k31mr3083066uad.117.1644535723875; Thu, 10 Feb 2022 15:28:43 -0800 (PST) Date: Thu, 10 Feb 2022 23:28:10 +0000 In-Reply-To: <20220210232812.798387-1-nhuck@google.com> Message-Id: <20220210232812.798387-6-nhuck@google.com> Mime-Version: 1.0 References: <20220210232812.798387-1-nhuck@google.com> X-Mailer: git-send-email 2.35.1.265.g69c8d7142f-goog Subject: [RFC PATCH v2 5/7] crypto: arm64/aes-xctr: Add accelerated implementation of XCTR From: Nathan Huckleberry To: linux-crypto@vger.kernel.org Cc: Herbert Xu , "David S. Miller" , linux-arm-kernel@lists.infradead.org, Paul Crowley , Eric Biggers , Sami Tolvanen , Ard Biesheuvel , Nathan Huckleberry Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add hardware accelerated version of XCTR for ARM64 CPUs with ARMv8 Crypto Extension support. This XCTR implementation is based on the CTR implementation in aes-modes.S. More information on XCTR can be found in the HCTR2 paper: Length-preserving encryption with HCTR2: https://eprint.iacr.org/2021/1441.pdf Signed-off-by: Nathan Huckleberry --- Changes since v1: * Added STRIDE back to aes-glue.c arch/arm64/crypto/Kconfig | 4 +- arch/arm64/crypto/aes-glue.c | 72 ++++++++++++++++++- arch/arm64/crypto/aes-modes.S | 130 ++++++++++++++++++++++++++++++++++ 3 files changed, 202 insertions(+), 4 deletions(-) diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 2a965aa0188d..897f9a4b5b67 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -84,13 +84,13 @@ config CRYPTO_AES_ARM64_CE_CCM select CRYPTO_LIB_AES config CRYPTO_AES_ARM64_CE_BLK - tristate "AES in ECB/CBC/CTR/XTS modes using ARMv8 Crypto Extensions" + tristate "AES in ECB/CBC/CTR/XTS/XCTR modes using ARMv8 Crypto Extensions" depends on KERNEL_MODE_NEON select CRYPTO_SKCIPHER select CRYPTO_AES_ARM64_CE config CRYPTO_AES_ARM64_NEON_BLK - tristate "AES in ECB/CBC/CTR/XTS modes using NEON instructions" + tristate "AES in ECB/CBC/CTR/XTS/XCTR modes using NEON instructions" depends on KERNEL_MODE_NEON select CRYPTO_SKCIPHER select CRYPTO_LIB_AES diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 561dd2332571..dd04f3c5b0f1 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -24,6 +24,7 @@ #ifdef USE_V8_CRYPTO_EXTENSIONS #define MODE "ce" #define PRIO 300 +#define STRIDE 5 #define aes_expandkey ce_aes_expandkey #define aes_ecb_encrypt ce_aes_ecb_encrypt #define aes_ecb_decrypt ce_aes_ecb_decrypt @@ -34,13 +35,15 @@ #define aes_essiv_cbc_encrypt ce_aes_essiv_cbc_encrypt #define aes_essiv_cbc_decrypt ce_aes_essiv_cbc_decrypt #define aes_ctr_encrypt ce_aes_ctr_encrypt +#define aes_xctr_encrypt ce_aes_xctr_encrypt #define aes_xts_encrypt ce_aes_xts_encrypt #define aes_xts_decrypt ce_aes_xts_decrypt #define aes_mac_update ce_aes_mac_update -MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions"); +MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS/XCTR using ARMv8 Crypto Extensions"); #else #define MODE "neon" #define PRIO 200 +#define STRIDE 4 #define aes_ecb_encrypt neon_aes_ecb_encrypt #define aes_ecb_decrypt neon_aes_ecb_decrypt #define aes_cbc_encrypt neon_aes_cbc_encrypt @@ -50,16 +53,18 @@ MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions"); #define aes_essiv_cbc_encrypt neon_aes_essiv_cbc_encrypt #define aes_essiv_cbc_decrypt neon_aes_essiv_cbc_decrypt #define aes_ctr_encrypt neon_aes_ctr_encrypt +#define aes_xctr_encrypt neon_aes_xctr_encrypt #define aes_xts_encrypt neon_aes_xts_encrypt #define aes_xts_decrypt neon_aes_xts_decrypt #define aes_mac_update neon_aes_mac_update -MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 NEON"); +MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS/XCTR using ARMv8 NEON"); #endif #if defined(USE_V8_CRYPTO_EXTENSIONS) || !IS_ENABLED(CONFIG_CRYPTO_AES_ARM64_BS) MODULE_ALIAS_CRYPTO("ecb(aes)"); MODULE_ALIAS_CRYPTO("cbc(aes)"); MODULE_ALIAS_CRYPTO("ctr(aes)"); MODULE_ALIAS_CRYPTO("xts(aes)"); +MODULE_ALIAS_CRYPTO("xctr(aes)"); #endif MODULE_ALIAS_CRYPTO("cts(cbc(aes))"); MODULE_ALIAS_CRYPTO("essiv(cbc(aes),sha256)"); @@ -89,6 +94,10 @@ asmlinkage void aes_cbc_cts_decrypt(u8 out[], u8 const in[], u32 const rk[], asmlinkage void aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int bytes, u8 ctr[]); +asmlinkage void aes_xctr_encrypt(u8 out[], u8 const in[], u32 const rk[], + int rounds, int bytes, u8 ctr[], u8 finalbuf[], + int byte_ctr); + asmlinkage void aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[], int rounds, int bytes, u32 const rk2[], u8 iv[], int first); @@ -442,6 +451,49 @@ static int __maybe_unused essiv_cbc_decrypt(struct skcipher_request *req) return err ?: cbc_decrypt_walk(req, &walk); } +static int __maybe_unused xctr_encrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); + int err, rounds = 6 + ctx->key_length / 4; + struct skcipher_walk walk; + unsigned int byte_ctr = 0; + + err = skcipher_walk_virt(&walk, req, false); + + while (walk.nbytes > 0) { + const u8 *src = walk.src.virt.addr; + unsigned int nbytes = walk.nbytes; + u8 *dst = walk.dst.virt.addr; + u8 buf[AES_BLOCK_SIZE]; + unsigned int tail; + + if (unlikely(nbytes < AES_BLOCK_SIZE)) + src = memcpy(buf, src, nbytes); + else if (nbytes < walk.total) + nbytes &= ~(AES_BLOCK_SIZE - 1); + + kernel_neon_begin(); + aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, + walk.iv, buf, byte_ctr); + kernel_neon_end(); + + tail = nbytes % (STRIDE * AES_BLOCK_SIZE); + if (tail > 0 && tail < AES_BLOCK_SIZE) + /* + * The final partial block could not be returned using + * an overlapping store, so it was passed via buf[] + * instead. + */ + memcpy(dst + nbytes - tail, buf, tail); + byte_ctr += nbytes; + + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); + } + + return err; +} + static int __maybe_unused ctr_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); @@ -669,6 +721,22 @@ static struct skcipher_alg aes_algs[] = { { .setkey = skcipher_aes_setkey, .encrypt = ctr_encrypt, .decrypt = ctr_encrypt, +}, { + .base = { + .cra_name = "xctr(aes)", + .cra_driver_name = "xctr-aes-" MODE, + .cra_priority = PRIO, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct crypto_aes_ctx), + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .chunksize = AES_BLOCK_SIZE, + .setkey = skcipher_aes_setkey, + .encrypt = xctr_encrypt, + .decrypt = xctr_encrypt, }, { .base = { .cra_name = "xts(aes)", diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index dc35eb0245c5..3b5c7a5c21e4 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -479,6 +479,136 @@ ST5( mov v3.16b, v4.16b ) b .Lctrout AES_FUNC_END(aes_ctr_encrypt) + /* + * aes_xctr_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, + * int bytes, u8 const ctr[], u8 finalbuf[], int + * byte_ctr) + */ + +AES_FUNC_START(aes_xctr_encrypt) + stp x29, x30, [sp, #-16]! + mov x29, sp + + enc_prepare w3, x2, x12 + ld1 {vctr.16b}, [x5] + + umov x12, vctr.d[0] /* keep ctr in reg */ + lsr x7, x7, #4 + add x11, x7, #1 + +.LxctrloopNx: + add w7, w4, #15 + sub w4, w4, #MAX_STRIDE << 4 + lsr w7, w7, #4 + mov w8, #MAX_STRIDE + cmp w7, w8 + csel w7, w7, w8, lt + add x11, x11, x7 + + mov v0.16b, vctr.16b + mov v1.16b, vctr.16b + mov v2.16b, vctr.16b + mov v3.16b, vctr.16b +ST5( mov v4.16b, vctr.16b ) + + sub x7, x11, #MAX_STRIDE + eor x7, x12, x7 + ins v0.d[0], x7 + sub x7, x11, #MAX_STRIDE - 1 + sub x8, x11, #MAX_STRIDE - 2 + eor x7, x7, x12 + sub x9, x11, #MAX_STRIDE - 3 + mov v1.d[0], x7 + eor x8, x8, x12 + eor x9, x9, x12 +ST5( sub x10, x11, #MAX_STRIDE - 4) + mov v2.d[0], x8 + eor x10, x10, x12 + mov v3.d[0], x9 +ST5( mov v4.d[0], x10 ) + tbnz w4, #31, .Lxctrtail + ld1 {v5.16b-v7.16b}, [x1], #48 +ST4( bl aes_encrypt_block4x ) +ST5( bl aes_encrypt_block5x ) + eor v0.16b, v5.16b, v0.16b +ST4( ld1 {v5.16b}, [x1], #16 ) + eor v1.16b, v6.16b, v1.16b +ST5( ld1 {v5.16b-v6.16b}, [x1], #32 ) + eor v2.16b, v7.16b, v2.16b + eor v3.16b, v5.16b, v3.16b +ST5( eor v4.16b, v6.16b, v4.16b ) + st1 {v0.16b-v3.16b}, [x0], #64 +ST5( st1 {v4.16b}, [x0], #16 ) + cbz w4, .Lxctrout + b .LxctrloopNx + +.Lxctrout: + ldp x29, x30, [sp], #16 + ret + +.Lxctrtail: + /* XOR up to MAX_STRIDE * 16 - 1 bytes of in/output with v0 ... v3/v4 */ + mov x17, #16 + ands x13, x4, #0xf + csel x13, x13, x17, ne + +ST5( cmp w4, #64 - (MAX_STRIDE << 4)) +ST5( csel x14, x17, xzr, gt ) + cmp w4, #48 - (MAX_STRIDE << 4) + csel x15, x17, xzr, gt + cmp w4, #32 - (MAX_STRIDE << 4) + csel x16, x17, xzr, gt + cmp w4, #16 - (MAX_STRIDE << 4) + ble .Lxctrtail1x + +ST5( mov v4.d[0], x10 ) + + adr_l x12, .Lcts_permute_table + add x12, x12, x13 + +ST5( ld1 {v5.16b}, [x1], x14 ) + ld1 {v6.16b}, [x1], x15 + ld1 {v7.16b}, [x1], x16 + +ST4( bl aes_encrypt_block4x ) +ST5( bl aes_encrypt_block5x ) + + ld1 {v8.16b}, [x1], x13 + ld1 {v9.16b}, [x1] + ld1 {v10.16b}, [x12] + +ST4( eor v6.16b, v6.16b, v0.16b ) +ST4( eor v7.16b, v7.16b, v1.16b ) +ST4( tbl v3.16b, {v3.16b}, v10.16b ) +ST4( eor v8.16b, v8.16b, v2.16b ) +ST4( eor v9.16b, v9.16b, v3.16b ) + +ST5( eor v5.16b, v5.16b, v0.16b ) +ST5( eor v6.16b, v6.16b, v1.16b ) +ST5( tbl v4.16b, {v4.16b}, v10.16b ) +ST5( eor v7.16b, v7.16b, v2.16b ) +ST5( eor v8.16b, v8.16b, v3.16b ) +ST5( eor v9.16b, v9.16b, v4.16b ) + +ST5( st1 {v5.16b}, [x0], x14 ) + st1 {v6.16b}, [x0], x15 + st1 {v7.16b}, [x0], x16 + add x13, x13, x0 + st1 {v9.16b}, [x13] // overlapping stores + st1 {v8.16b}, [x0] + b .Lxctrout + +.Lxctrtail1x: + // use finalbuf if less than a full block + csel x0, x0, x6, eq + ld1 {v5.16b}, [x1] +ST5( mov v3.16b, v4.16b ) + encrypt_block v3, w3, x2, x8, w7 + eor v5.16b, v5.16b, v3.16b + st1 {v5.16b}, [x0] + b .Lxctrout +AES_FUNC_END(aes_xctr_encrypt) + /* * aes_xts_encrypt(u8 out[], u8 const in[], u8 const rk1[], int rounds, From patchwork Thu Feb 10 23:28:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nathan Huckleberry X-Patchwork-Id: 541542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36171C433EF for ; Thu, 10 Feb 2022 23:28:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345649AbiBJX2u (ORCPT ); Thu, 10 Feb 2022 18:28:50 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:41358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345661AbiBJX2t (ORCPT ); Thu, 10 Feb 2022 18:28:49 -0500 Received: from mail-vs1-xe49.google.com (mail-vs1-xe49.google.com [IPv6:2607:f8b0:4864:20::e49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C9E35F6B for ; Thu, 10 Feb 2022 15:28:48 -0800 (PST) Received: by mail-vs1-xe49.google.com with SMTP id b12-20020a67fe8c000000b0031a490e8b9dso644536vsr.15 for ; Thu, 10 Feb 2022 15:28:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=aI+ZZN/XgFx9qNVpwESSjNwjrEMAlNEJZKE9wM7Vg8M=; b=szJN+mEd6nLTGyWwqNr7fYO0/2rktvhHXQztqi/tX90V+UH1y5L56DnLAVXvuM+dZj KZ6aBrbekNOSCcuB1MNM/4UXyLMoVCAw5jnWNZiXlpmrpCn2K00MUoT5nJ6k/BdBQ+1B 4tU4JvSvK2ie5atiyYMBkZY2iIxFQQzy7gH6OSY0toUs9PDHSIrr1susNs1Te5EXhJwN EgB6sVHg8+YTEoi2lzbS563wEaReTAkyWSF4zBxnEPhjKHqE4bU0uDPNWEutBEBHyvkq +Q9pmtChWIia2FBJ8PltcLBnY2ZSWjwBk40Wqx3OokYV0FZ7RwBL30C5YbZbBZFvjRPw pj8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=aI+ZZN/XgFx9qNVpwESSjNwjrEMAlNEJZKE9wM7Vg8M=; b=mBBw44sTzlPI7K6bXXlMh+/Vj+yjhXaRTS6aM6sEgOscZf9i74uVlm6wtx8SivHV8K uSfaCsEa3IIABHCCy2aA8fLKn5fbmMnonTMpqpf3uEFDzzLN7w/D2rNyQc+NAR7ZkRc2 HkT+6hEqF3SNbLaqUs8ndGOFV3+AZgU1l/q3sGAgV23gUvqfP2Bli7vlU+kEBog2oNH4 8jGaECglvChfQLlXfTEPf9Tuc80cBa5z43ORvsFFdrM4ZjyHyOQ4Iy9DeJHjx/3cWCoR ZWykfidF8GJ8rzhlDPXbq1jTiBL3itKER63jsaZEWH8fG9ZAgh4+gvQ+L33FJWljwJG1 goKg== X-Gm-Message-State: AOAM533/KnetsFWDmePZmHj5UeNINiA/uI6a2RRZGUQirT/DCjTKk+k3 Ya1TY8JpvnR80LG11Du6Wh7CmGyKbPYk4bfNbLmQ3eDNEjnjTG0fuwtfMSN8bFG5MZ//a7i8d2X gGRx0hU3TVsCA3JkeLRHNGwtEdKOTW8NYtqZ/YlLmM3os6cAPybN95J00fR5/4rfRBQE= X-Google-Smtp-Source: ABdhPJzCBY0mrmEviuzKIse40qgMrfXkf3Pj774y8cx6D6m7AiAAbw5HhZBEYCXE4J1+fE0/KjzeZZYC5w== X-Received: from nhuck.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:39cc]) (user=nhuck job=sendgmr) by 2002:a67:fb0b:: with SMTP id d11mr2991237vsr.35.1644535727137; Thu, 10 Feb 2022 15:28:47 -0800 (PST) Date: Thu, 10 Feb 2022 23:28:12 +0000 In-Reply-To: <20220210232812.798387-1-nhuck@google.com> Message-Id: <20220210232812.798387-8-nhuck@google.com> Mime-Version: 1.0 References: <20220210232812.798387-1-nhuck@google.com> X-Mailer: git-send-email 2.35.1.265.g69c8d7142f-goog Subject: [RFC PATCH v2 7/7] crypto: arm64/polyval: Add PMULL accelerated implementation of POLYVAL From: Nathan Huckleberry To: linux-crypto@vger.kernel.org Cc: Herbert Xu , "David S. Miller" , linux-arm-kernel@lists.infradead.org, Paul Crowley , Eric Biggers , Sami Tolvanen , Ard Biesheuvel , Nathan Huckleberry Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add hardware accelerated version of POLYVAL for ARM64 CPUs with Crypto Extension support. This implementation is accelerated using PMULL instructions to perform the finite field computations. For added efficiency, 8 blocks of the message are processed simultaneously by precomputing the first 8 powers of the key. Karatsuba multiplication is used instead of Schoolbook multiplication because it was found to be slightly faster on ARM64 CPUs. Montgomery reduction must be used instead of Barrett reduction due to the difference in modulus between POLYVAL's field and other finite fields. Signed-off-by: Nathan Huckleberry --- Changes since v1: * Rename C, D, E to LO, MI, HI * Add comments explaining POLYVAL * Fix bug in handling of non-block-multiple messages * Wrap shash in ahash (as done in ghash) arch/arm64/crypto/Kconfig | 7 + arch/arm64/crypto/Makefile | 3 + arch/arm64/crypto/polyval-ce-core.S | 405 ++++++++++++++++++++++++++++ arch/arm64/crypto/polyval-ce-glue.c | 365 +++++++++++++++++++++++++ 4 files changed, 780 insertions(+) create mode 100644 arch/arm64/crypto/polyval-ce-core.S create mode 100644 arch/arm64/crypto/polyval-ce-glue.c diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index 897f9a4b5b67..f7fbe8637e5c 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -60,6 +60,13 @@ config CRYPTO_GHASH_ARM64_CE select CRYPTO_GF128MUL select CRYPTO_LIB_AES +config CRYPTO_POLYVAL_ARM64_CE + tristate "POLYVAL using ARMv8 Crypto Extensions (for HCTR2)" + depends on KERNEL_MODE_NEON + select CRYPTO_CRYPTD + select CRYPTO_HASH + select CRYPTO_POLYVAL + config CRYPTO_CRCT10DIF_ARM64_CE tristate "CRCT10DIF digest algorithm using PMULL instructions" depends on KERNEL_MODE_NEON && CRC_T10DIF diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile index 09a805cc32d7..53f9af962b86 100644 --- a/arch/arm64/crypto/Makefile +++ b/arch/arm64/crypto/Makefile @@ -26,6 +26,9 @@ sm4-ce-y := sm4-ce-glue.o sm4-ce-core.o obj-$(CONFIG_CRYPTO_GHASH_ARM64_CE) += ghash-ce.o ghash-ce-y := ghash-ce-glue.o ghash-ce-core.o +obj-$(CONFIG_CRYPTO_POLYVAL_ARM64_CE) += polyval-ce.o +polyval-ce-y := polyval-ce-glue.o polyval-ce-core.o + obj-$(CONFIG_CRYPTO_CRCT10DIF_ARM64_CE) += crct10dif-ce.o crct10dif-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o diff --git a/arch/arm64/crypto/polyval-ce-core.S b/arch/arm64/crypto/polyval-ce-core.S new file mode 100644 index 000000000000..3b2a5adf8987 --- /dev/null +++ b/arch/arm64/crypto/polyval-ce-core.S @@ -0,0 +1,405 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2021 Google LLC + * + * Use of this source code is governed by an MIT-style + * license that can be found in the LICENSE file or at + * https://opensource.org/licenses/MIT. + */ +/* + * This is an efficient implementation of POLYVAL using ARMv8 Crypto Extension + * instructions. It works on 8 blocks at a time, by precomputing the first 8 + * keys powers h^8, ..., h^1 in the POLYVAL finite field. This precomputation + * allows us to split finite field multiplication into two steps. + * + * In the first step, we consider h^i, m_i as normal polynomials of degree less + * than 128. We then compute p(x) = h^8m_0 + ... + h^1m_7 where multiplication + * is simply polynomial multiplication. + * + * In the second step, we compute the reduction of p(x) modulo the finite field + * modulus g(x) = x^128 + x^127 + x^126 + x^121 + 1. + * + * This two step process is equivalent to computing h^8m_0 + ... + h^1m_7 where + * multiplication is finite field multiplication. The advantage is that the + * two-step process only requires 1 finite field reduction for every 8 + * polynomial multiplications. Further parallelism is gained by interleaving the + * multiplications and polynomial reductions. + */ + +#include +#define NUM_PRECOMPUTE_POWERS 8 + +BLOCKS_LEFT .req x2 +OP1 .req x9 +KEY_START .req x10 +EXTRA_BYTES .req x11 +IND .req x12 +TMP .req x13 +PARTIAL_LEFT .req x14 + +M0 .req v0 +M1 .req v1 +M2 .req v2 +M3 .req v3 +M4 .req v4 +M5 .req v5 +M6 .req v6 +M7 .req v7 +KEY8 .req v8 +KEY7 .req v9 +KEY6 .req v10 +KEY5 .req v11 +KEY4 .req v12 +KEY3 .req v13 +KEY2 .req v14 +KEY1 .req v15 +PL .req v16 +PH .req v17 +T .req v18 +V .req v19 +LO .req v20 +MI .req v21 +HI .req v22 +SUM .req v23 +GSTAR .req v24 + + .text + .align 4 + + .arch armv8-a+crypto + .align 4 + +.Lgstar: + .quad 0xc200000000000000, 0xc200000000000000 + +/* + * Computes the product of two 128-bit polynomials in X and Y and XORs the + * components of the 256-bit product into LO, MI, HI. + * + * The multiplication produces four parts: + * LOW: The polynomial given by performing carryless multiplication of X_L and + * Y_L + * MID: The polynomial given by performing carryless multiplication of (X_L ^ + * X_H) and (Y_L ^ Y_H) + * HIGH: The polynomial given by performing carryless multiplication of X_H + * and Y_H + * + * We compute: + * LO ^= LOW + * MI ^= MID + * HI ^= HIGH + * + * Later, the 256-bit result can be extracted as: + * [HI_H : HI_L ^ HI_H ^ MI_H ^ LO_H :: LO_H ^ HI_L ^ MI_L ^ LO_L : LO_L] + * This step is done when computing the polynomial reduction for efficiency + * reasons. + */ +.macro karatsuba1 X Y + X .req \X + Y .req \Y + ext v25.16b, X.16b, Y.16b, #8 + eor v25.16b, v25.16b, X.16b + ext v26.16b, Y.16b, Y.16b, #8 + eor v26.16b, v26.16b, Y.16b + pmull v26.1q, v25.1d, v26.1d + pmull2 v25.1q, X.2d, Y.2d + pmull X.1q, X.1d, Y.1d + eor HI.16b, HI.16b, v26.16b + eor LO.16b, LO.16b, v25.16b + eor MI.16b, MI.16b, X.16b + .unreq X + .unreq Y +.endm + +/* + * Computes the 256-bit polynomial represented by LO, HI, MI. Stores + * the result in PL, PH. + * [PH :: PL] = [HI_H : HI_L ^ HI_H ^ MI_H ^ LO_H :: LO_H ^ HI_L ^ MI_L ^ LO_L + * : LO_L] + */ +.macro karatsuba2 + ext v4.16b, MI.16b, LO.16b, #8 + eor HI.16b, HI.16b, v4.16b //[HI1 ^ LO0 : HI0 ^ MI1] + eor v4.16b, LO.16b, MI.16b //[LO1 ^ MI1 : LO0 ^ MI0] + //[LO0 ^ LO1 ^ MI1 ^ HI1 : MI1 ^ LO0 ^ MI0 ^ HI0] + eor v4.16b, HI.16b, v4.16b + ext LO.16b, LO.16b, LO.16b, #8 // [LO0 : LO1] + ext MI.16b, MI.16b, MI.16b, #8 // [MI0 : MI1] + ext PH.16b, v4.16b, LO.16b, #8 //[LO1 : LO1 ^ MI1 ^ HI1 ^ LO0] + ext PL.16b, MI.16b, v4.16b, #8 //[MI1 ^ LO0 ^ MI0 ^ HI0 : MI0] +.endm + +/* + * Computes the 128-bit reduction of PL, PH. Stores the result in PH. + * + * This macro computes p(x) mod g(x) where p(x) is in montgomery form and g(x) = + * x^128 + x^127 + x^126 + x^121 + 1. + * + * The montgomery form of a polynomial p(x) is p(x)x^{128}. Montgomery reduction + * works by simultaneously dividing by x^{128} and computing the modular + * reduction. + * + * Suppose we wish to reduce the montgomery form of p(x) = [P_3 : P_2 : P_1 : + * P_0] where P_i is a polynomial of degree at most 64 represented as 64-bits. + * Thus we would like to compute: + * p(x) / x^{128} mod g(x) + * = (P_3*x^{192} + P_2*x^{128} + P_1*x^{64} + P_0) / x^{128} mod g(x) + * + * We would like to divide by x^{128} efficiently. Since P_3*x^{128}, + * P_2*x^{128} are multiples of x^{128}, we can simply bitshift right by 128. + * + * We now focus on dividing P_1*x^{64} + P_0 by x^{128}. We do this by making + * P_1*x^{64} + P_0 divisble by x^{128} then bitshifting. To add divisibility, + * we consider the polynomials mod x^{128}. + * + * Let c(x) = P_1*x^{64} + P_0. + * + * Now let m(x) = c(x) mod x^{128} + * and + * Let z(x) = [c(x) + m(x)g(x)] / x^{128} + * + * First notice that: + * c(x) + m(x)g(x) = c(x) mod g(x). + * Furthermore, g(x) mod x^{128} = 1, so we have + * c(x) + m(x)g(x) = c(x) + c(x) = 0 (mod x^{128}). + * + * Thus c(x) + m(x)g(x) is divisible by x^{128} and is equivalent to c(x) mod + * g(x). + * + * In practice we use a slight modification of this idea, by using g*(x) = + * x^{63} + x^{62} + x^{57}. This is because we can only multiply 64-bit + * polynomials. Notice that g(x) = x^128 + g*(x)*x^{64} + 1 + * + * We do this by substituting g(x) = x^{128} + g*(x)x^{64} + 1 + * z(x) = [c(x) + c(x)*(x^{128} + g*(x)*x^{64} + 1)] / x^{128} + * = [P_1*x^{192} + P_0*x^{128} + P_1*g*(x)x^{128} + P_0*g*(x)*x^{64} z(x)] + * / x^{128} + * = P_1*x^{64} + P_0 + P_1*g*(x) + P_0*g*(x)*x^{-64} + * + * The only difficulty left in this expression is P_0*g*(x)x^{-64}. + * Let t(x) = P_0*g*(x) = [T_1 : T_0] + * Notice that we can repeat the above process: + * g(x) mod x^{64} = 1 + * m'(x) = t(x) mod x^{64} + * z'(x) = [t(x) + m'(x)g(x)] / x^64 + * Thus we get + * z'(x) = [t(x) + (x^{128} + g*(x)x^{64} + 1)T_0] / x^64 + * = T_1 + T_0*x^{64} + g*(x)*T_0 + * + * Recall that this is only the reduction for [P_1*x^{64} + P_0] / x^{64}. The + * full computation we need to make is: + * p(x) / x^{128} = P_3*x^{64} + P_2 + P_1*x^{64} + P_0 + P_1*g*(x) + + * T_1 + T_0*x^{64} + g*(x)*T_0 + * + * Thus we have: + * t(x) = g*(x) * P_0 = [T_1 : T_0] + * v(x) = g*(x) * (T_0 ^ P_1) = [V_1 : V_0] + * p(x) / x^{128} mod g(x) = [P_3 ^ P_1 ^ V_1 ^ T_0 : P_2 ^ P_0 ^ V_0 ^ T_1] + */ +.macro montgomery_reduction + pmull T.1q, GSTAR.1d, PL.1d + ext T.16b, T.16b, T.16b, #8 + eor PL.16b, PL.16b, T.16b + pmull2 V.1q, GSTAR.2d, PL.2d + eor V.16b, PL.16b, V.16b + eor PH.16b, PH.16b, V.16b +.endm + +/* + * Compute Polyval on 8 blocks. + * + * If reduce is set, performs interleaved montgomery reduction + * on the last full_stride iteration's PL, PH. + * + * Sets PL, PH. + */ +.macro full_stride reduce + .set reduce, \reduce + eor LO.16b, LO.16b, LO.16b + eor MI.16b, MI.16b, MI.16b + eor HI.16b, HI.16b, HI.16b + + ld1 {M0.16b, M1.16b, M2.16b, M3.16b}, [x0], #64 + ld1 {M4.16b, M5.16b, M6.16b, M7.16b}, [x0], #64 + + karatsuba1 M7 KEY1 + .if(reduce) + pmull T.1q, GSTAR.1d, PL.1d + .endif + + karatsuba1 M6 KEY2 + .if(reduce) + ext T.16b, T.16b, T.16b, #8 + .endif + + karatsuba1 M5 KEY3 + .if(reduce) + eor PL.16b, PL.16b, T.16b + .endif + + karatsuba1 M4 KEY4 + .if(reduce) + pmull2 V.1q, GSTAR.2d, PL.2d + .endif + + karatsuba1 M3 KEY5 + .if(reduce) + eor V.16b, PL.16b, V.16b + .endif + + karatsuba1 M2 KEY6 + .if(reduce) + eor PH.16b, PH.16b, V.16b + .endif + + karatsuba1 M1 KEY7 + .if(reduce) + mov SUM.16b, PH.16b + .endif + eor M0.16b, M0.16b, SUM.16b + + karatsuba1 M0 KEY8 + + karatsuba2 +.endm + +/* + * Handle any extra blocks before + * full_stride loop. + */ +.macro partial_stride + eor LO.16b, LO.16b, LO.16b + eor MI.16b, MI.16b, MI.16b + eor HI.16b, HI.16b, HI.16b + add KEY_START, x1, #(NUM_PRECOMPUTE_POWERS << 4) + sub KEY_START, KEY_START, PARTIAL_LEFT, lsl #4 + ld1 {v0.16b}, [KEY_START] + mov v1.16b, SUM.16b + karatsuba1 v0 v1 + karatsuba2 + montgomery_reduction + mov SUM.16b, PH.16b + eor LO.16b, LO.16b, LO.16b + eor MI.16b, MI.16b, MI.16b + eor HI.16b, HI.16b, HI.16b + mov IND, XZR +.LloopPartial: + cmp IND, PARTIAL_LEFT + bge .LloopExitPartial + + sub TMP, IND, PARTIAL_LEFT + + cmp TMP, #-4 + bgt .Lgt4Partial + ld1 {M0.16b, M1.16b, M2.16b, M3.16b}, [x0], #64 + // Clobber key registers + ld1 {KEY8.16b, KEY7.16b, KEY6.16b, KEY5.16b}, [KEY_START], #64 + karatsuba1 M0 KEY8 + karatsuba1 M1 KEY7 + karatsuba1 M2 KEY6 + karatsuba1 M3 KEY5 + add IND, IND, #4 + b .LoutPartial + +.Lgt4Partial: + cmp TMP, #-3 + bgt .Lgt3Partial + ld1 {M0.16b, M1.16b, M2.16b}, [x0], #48 + // Clobber key registers + ld1 {KEY8.16b, KEY7.16b, KEY6.16b}, [KEY_START], #48 + karatsuba1 M0 KEY8 + karatsuba1 M1 KEY7 + karatsuba1 M2 KEY6 + add IND, IND, #3 + b .LoutPartial + +.Lgt3Partial: + cmp TMP, #-2 + bgt .Lgt2Partial + ld1 {M0.16b, M1.16b}, [x0], #32 + // Clobber key registers + ld1 {KEY8.16b, KEY7.16b}, [KEY_START], #32 + karatsuba1 M0 KEY8 + karatsuba1 M1 KEY7 + add IND, IND, #2 + b .LoutPartial + +.Lgt2Partial: + ld1 {M0.16b}, [x0], #16 + // Clobber key registers + ld1 {KEY8.16b}, [KEY_START], #16 + karatsuba1 M0 KEY8 + add IND, IND, #1 +.LoutPartial: + b .LloopPartial +.LloopExitPartial: + karatsuba2 + montgomery_reduction + eor SUM.16b, SUM.16b, PH.16b +.endm + +/* + * Perform montgomery multiplication in GF(2^128) and store result in op1. + * + * Computes op1*op2*x^{-128} mod x^128 + x^127 + x^126 + x^121 + 1 + * If op1, op2 are in montgomery form, this computes the montgomery + * form of op1*op2. + * + * void pmull_polyval_mul(u8 *op1, const u8 *op2); + */ +SYM_FUNC_START(pmull_polyval_mul) + adr TMP, .Lgstar + ld1 {GSTAR.2d}, [TMP] + eor LO.16b, LO.16b, LO.16b + eor MI.16b, MI.16b, MI.16b + eor HI.16b, HI.16b, HI.16b + ld1 {v0.16b}, [x0] + ld1 {v1.16b}, [x1] + karatsuba1 v0 v1 + karatsuba2 + montgomery_reduction + st1 {PH.16b}, [x0] + ret +SYM_FUNC_END(pmull_polyval_mul) + +/* + * Perform polynomial evaluation as specified by POLYVAL. If nblocks = k, this + * routine multiplies the value stored at accumulator by h^k and XORs the + * evaluated polynomial into it. + * + * Computes h^k * accumulator + h^kM_0 + ... + h^1M_{k-1} (No constant term) + * + * x0 (OP1) - pointer to message blocks + * x1 - pointer to precomputed key struct + * x2 - number of blocks to hash + * x3 - location to XOR with evaluated polynomial + * + * void pmull_polyval_update(const u8 *in, const struct polyval_ctx *ctx, + * size_t nblocks, u8 *accumulator); + */ +SYM_FUNC_START(pmull_polyval_update) + adr TMP, .Lgstar + ld1 {GSTAR.2d}, [TMP] + ld1 {SUM.16b}, [x3] + ands PARTIAL_LEFT, BLOCKS_LEFT, #7 + beq .LskipPartial + partial_stride +.LskipPartial: + subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS + blt .LstrideLoopExit + ld1 {KEY8.16b, KEY7.16b, KEY6.16b, KEY5.16b}, [x1], #64 + ld1 {KEY4.16b, KEY3.16b, KEY2.16b, KEY1.16b}, [x1], #64 + full_stride 0 + subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS + blt .LstrideLoopExitReduce +.LstrideLoop: + full_stride 1 + subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS + bge .LstrideLoop +.LstrideLoopExitReduce: + montgomery_reduction + mov SUM.16b, PH.16b +.LstrideLoopExit: + st1 {SUM.16b}, [x3] + ret +SYM_FUNC_END(pmull_polyval_update) diff --git a/arch/arm64/crypto/polyval-ce-glue.c b/arch/arm64/crypto/polyval-ce-glue.c new file mode 100644 index 000000000000..ca92e027b4ec --- /dev/null +++ b/arch/arm64/crypto/polyval-ce-glue.c @@ -0,0 +1,365 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Accelerated POLYVAL implementation with ARMv8 Crypto Extension + * instructions. This file contains glue code. + * + * Copyright (c) 2007 Nokia Siemens Networks - Mikko Herranen + * Copyright (c) 2009 Intel Corp. + * Author: Huang Ying + * Copyright 2021 Google LLC + */ +/* + * Glue code based on ghash-clmulni-intel_glue.c. + * + * This implementation of POLYVAL uses montgomery multiplication accelerated by + * ARMv8 Crypto Extension instructions to implement the finite field operations. + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define POLYVAL_BLOCK_SIZE 16 +#define POLYVAL_DIGEST_SIZE 16 +#define NUM_PRECOMPUTE_POWERS 8 + +struct polyval_async_ctx { + struct cryptd_ahash *cryptd_tfm; +}; + +struct polyval_ctx { + be128 key_powers[NUM_PRECOMPUTE_POWERS]; +}; + +struct polyval_desc_ctx { + u8 buffer[POLYVAL_BLOCK_SIZE]; + u32 bytes; +}; + +asmlinkage void pmull_polyval_update(const u8 *in, const struct polyval_ctx + *ctx, size_t nblocks, u8 *accumulator); +asmlinkage void pmull_polyval_mul(u8 *op1, const u8 *op2); + +static int polyval_init(struct shash_desc *desc) +{ + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + + memset(dctx, 0, sizeof(*dctx)); + + return 0; +} + +static int polyval_setkey(struct crypto_shash *tfm, + const u8 *key, unsigned int keylen) +{ + struct polyval_ctx *ctx = crypto_shash_ctx(tfm); + int i; + + if (keylen != POLYVAL_BLOCK_SIZE) + return -EINVAL; + + BUILD_BUG_ON(sizeof(u128) != POLYVAL_BLOCK_SIZE); + + memcpy(&ctx->key_powers[NUM_PRECOMPUTE_POWERS-1], key, sizeof(be128)); + + kernel_neon_begin(); + for (i = NUM_PRECOMPUTE_POWERS-2; i >= 0; i--) { + memcpy(&ctx->key_powers[i], key, sizeof(be128)); + pmull_polyval_mul((u8 *)&ctx->key_powers[i], + (u8 *)&ctx->key_powers[i+1]); + } + kernel_neon_end(); + + return 0; +} + +static int polyval_update(struct shash_desc *desc, + const u8 *src, unsigned int srclen) +{ + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + struct polyval_ctx *ctx = crypto_shash_ctx(desc->tfm); + u8 *dst = dctx->buffer; + u8 *pos; + unsigned int nblocks; + unsigned int n; + + kernel_neon_begin(); + if (dctx->bytes) { + n = min(srclen, dctx->bytes); + pos = dst + POLYVAL_BLOCK_SIZE - dctx->bytes; + + dctx->bytes -= n; + srclen -= n; + + while (n--) + *pos++ ^= *src++; + + if (!dctx->bytes) + pmull_polyval_mul(dst, + (u8 *)&ctx->key_powers[NUM_PRECOMPUTE_POWERS-1]); + } + + nblocks = srclen/POLYVAL_BLOCK_SIZE; + pmull_polyval_update(src, ctx, nblocks, dst); + srclen -= nblocks*POLYVAL_BLOCK_SIZE; + kernel_neon_end(); + + if (srclen) { + dctx->bytes = POLYVAL_BLOCK_SIZE - srclen; + src += nblocks*POLYVAL_BLOCK_SIZE; + pos = dst; + while (srclen--) + *pos++ ^= *src++; + } + + return 0; +} + +static int polyval_final(struct shash_desc *desc, u8 *dst) +{ + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + struct polyval_ctx *ctx = crypto_shash_ctx(desc->tfm); + u8 *buf = dctx->buffer; + + if (dctx->bytes) { + kernel_neon_begin(); + pmull_polyval_mul(buf, + (u8 *)&ctx->key_powers[NUM_PRECOMPUTE_POWERS-1]); + kernel_neon_end(); + } + + dctx->bytes = 0; + memcpy(dst, buf, POLYVAL_BLOCK_SIZE); + + return 0; +} + +static struct shash_alg polyval_alg = { + .digestsize = POLYVAL_DIGEST_SIZE, + .init = polyval_init, + .update = polyval_update, + .final = polyval_final, + .setkey = polyval_setkey, + .descsize = sizeof(struct polyval_desc_ctx), + .base = { + .cra_name = "__polyval", + .cra_driver_name = "__polyval-ce", + .cra_priority = 0, + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = POLYVAL_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct polyval_ctx), + .cra_module = THIS_MODULE, + }, +}; + +static int polyval_async_init(struct ahash_request *req) +{ + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm); + struct ahash_request *cryptd_req = ahash_request_ctx(req); + struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; + struct shash_desc *desc = cryptd_shash_desc(cryptd_req); + struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm); + + desc->tfm = child; + return crypto_shash_init(desc); +} + +static int polyval_async_update(struct ahash_request *req) +{ + struct ahash_request *cryptd_req = ahash_request_ctx(req); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm); + struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; + struct shash_desc *desc; + + if (!crypto_simd_usable() || + (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) { + memcpy(cryptd_req, req, sizeof(*req)); + ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base); + return crypto_ahash_update(cryptd_req); + } + desc = cryptd_shash_desc(cryptd_req); + + return shash_ahash_update(req, desc); +} + +static int polyval_async_final(struct ahash_request *req) +{ + struct ahash_request *cryptd_req = ahash_request_ctx(req); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm); + struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; + struct shash_desc *desc; + + if (!crypto_simd_usable() || + (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) { + memcpy(cryptd_req, req, sizeof(*req)); + ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base); + return crypto_ahash_final(cryptd_req); + } + desc = cryptd_shash_desc(cryptd_req); + + return crypto_shash_final(desc, req->result); +} + +static int polyval_async_import(struct ahash_request *req, const void *in) +{ + struct ahash_request *cryptd_req = ahash_request_ctx(req); + struct shash_desc *desc = cryptd_shash_desc(cryptd_req); + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + + polyval_async_init(req); + memcpy(dctx, in, sizeof(*dctx)); + return 0; + +} + +static int polyval_async_export(struct ahash_request *req, void *out) +{ + struct ahash_request *cryptd_req = ahash_request_ctx(req); + struct shash_desc *desc = cryptd_shash_desc(cryptd_req); + struct polyval_desc_ctx *dctx = shash_desc_ctx(desc); + + memcpy(out, dctx, sizeof(*dctx)); + return 0; + +} + +static int polyval_async_digest(struct ahash_request *req) +{ + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm); + struct ahash_request *cryptd_req = ahash_request_ctx(req); + struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm; + struct shash_desc *desc; + struct crypto_shash *child; + + if (!crypto_simd_usable() || + (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) { + memcpy(cryptd_req, req, sizeof(*req)); + ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base); + return crypto_ahash_digest(cryptd_req); + } + desc = cryptd_shash_desc(cryptd_req); + child = cryptd_ahash_child(cryptd_tfm); + + desc->tfm = child; + return shash_ahash_digest(req, desc); +} + +static int polyval_async_setkey(struct crypto_ahash *tfm, const u8 *key, + unsigned int keylen) +{ + struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm); + struct crypto_ahash *child = &ctx->cryptd_tfm->base; + + crypto_ahash_clear_flags(child, CRYPTO_TFM_REQ_MASK); + crypto_ahash_set_flags(child, crypto_ahash_get_flags(tfm) + & CRYPTO_TFM_REQ_MASK); + return crypto_ahash_setkey(child, key, keylen); +} + +static int polyval_async_init_tfm(struct crypto_tfm *tfm) +{ + struct cryptd_ahash *cryptd_tfm; + struct polyval_async_ctx *ctx = crypto_tfm_ctx(tfm); + + cryptd_tfm = cryptd_alloc_ahash("__polyval-ce", + CRYPTO_ALG_INTERNAL, + CRYPTO_ALG_INTERNAL); + if (IS_ERR(cryptd_tfm)) + return PTR_ERR(cryptd_tfm); + ctx->cryptd_tfm = cryptd_tfm; + crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm), + sizeof(struct ahash_request) + + crypto_ahash_reqsize(&cryptd_tfm->base)); + + return 0; +} + +static void polyval_async_exit_tfm(struct crypto_tfm *tfm) +{ + struct polyval_async_ctx *ctx = crypto_tfm_ctx(tfm); + + cryptd_free_ahash(ctx->cryptd_tfm); +} + +static struct ahash_alg polyval_async_alg = { + .init = polyval_async_init, + .update = polyval_async_update, + .final = polyval_async_final, + .setkey = polyval_async_setkey, + .digest = polyval_async_digest, + .export = polyval_async_export, + .import = polyval_async_import, + .halg = { + .digestsize = POLYVAL_DIGEST_SIZE, + .statesize = sizeof(struct polyval_desc_ctx), + .base = { + .cra_name = "polyval", + .cra_driver_name = "polyval-clmulni", + .cra_priority = 200, + .cra_ctxsize = sizeof(struct polyval_async_ctx), + .cra_flags = CRYPTO_ALG_ASYNC, + .cra_blocksize = POLYVAL_BLOCK_SIZE, + .cra_module = THIS_MODULE, + .cra_init = polyval_async_init_tfm, + .cra_exit = polyval_async_exit_tfm, + }, + }, +}; + +static int __init polyval_ce_mod_init(void) +{ + int err; + + if (!cpu_have_named_feature(ASIMD)) + return -ENODEV; + + if (!cpu_have_named_feature(PMULL)) + return -ENODEV; + + err = crypto_register_shash(&polyval_alg); + if (err) + goto err_out; + err = crypto_register_ahash(&polyval_async_alg); + if (err) + goto err_shash; + + return 0; + +err_shash: + crypto_unregister_shash(&polyval_alg); +err_out: + return err; +} + +static void __exit polyval_ce_mod_exit(void) +{ + crypto_unregister_ahash(&polyval_async_alg); + crypto_unregister_shash(&polyval_alg); +} + +static const struct cpu_feature polyval_cpu_feature[] = { + { cpu_feature(PMULL) }, { } +}; +MODULE_DEVICE_TABLE(cpu, polyval_cpu_feature); + +module_init(polyval_ce_mod_init); +module_exit(polyval_ce_mod_exit); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("POLYVAL hash function accelerated by ARMv8 Crypto Extension"); +MODULE_ALIAS_CRYPTO("polyval");