From patchwork Sun Sep 29 17:38:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 174680 Delivered-To: patch@linaro.org Received: by 2002:a92:7e96:0:0:0:0:0 with SMTP id q22csp6171705ill; Sun, 29 Sep 2019 10:39:12 -0700 (PDT) X-Google-Smtp-Source: APXvYqyrsxESbvOJ0/yJkf/rRk20t9VYn3fvED/PHj8F7H2p3bAFt4odWxsawvVQMqjWeP2x9Tje X-Received: by 2002:a05:6402:1501:: with SMTP id f1mr15544057edw.76.1569778752778; Sun, 29 Sep 2019 10:39:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1569778752; cv=none; d=google.com; s=arc-20160816; b=QIdSSbVKy7abTkGB5m8dZH5lv7pAn1mV4Cvl5dUpycBilgMkxdBFVRxLET0AO9BxgE boBaNNrsUtVbsZ3PV3GdVlltSoU1rnbknAWfIDxijgNjUGnXcWo9VTV7wjf8TJD+oIAQ 98d2bRnRnUzIkrLj4/pNvHYqeUkvcqdnN1g7LxL+lWCqiHt270xhpekw46t6rNsmOtta Ca48kWXAnfVx4e8/i97h8pBxBrYV+f3hR+YBm9MtsSOFjhx6IvTVTX/P19IitSltcKM2 CLTyjI5z4XfYiC7VxIw0qxvsydU63iYFVHz+QbmFvis6YVAkBJD4swyVJwB5CfETWScM BVwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=X/vLL5kgt+leeMxxc/cNujK7VVt1jHGGLQNhA6i9258=; b=VVupLnXsYfYY6vM64PPIdj3OakZoFf3RdLr/6irrC4P4tKA+K0VnqsB8OUTCWFPUIM ZRJLB1PhCUXjHR9v5MxIY3tZYK5ZPpAUeCYoupYpSBMbjrtiD0RNuh7YEa1sIw28qbgA oSX5mfElD1dVZkIxOpMbXB+yiCH77Erwzh7z0mIoYQlHenXggkiu70mlIPok9X0ggtQF 3d/u93TWw3ekoGaCAxu+puJQHpnOrv7g2swyaM+zxsBNUBj34SRiRT+1ksHAoHXXLZza irYkYhPD58ECuFWjdn9cZf4RERt1TH9FYr/649kVO6d8uMf8KCLLEtaXW9IH7NySUXnf usWQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YdD0dwSX; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a54si5497520edc.333.2019.09.29.10.39.12; Sun, 29 Sep 2019 10:39:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=YdD0dwSX; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729225AbfI2RjK (ORCPT + 3 others); Sun, 29 Sep 2019 13:39:10 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:35129 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729361AbfI2RjK (ORCPT ); Sun, 29 Sep 2019 13:39:10 -0400 Received: by mail-wr1-f65.google.com with SMTP id v8so8465478wrt.2 for ; Sun, 29 Sep 2019 10:39:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=X/vLL5kgt+leeMxxc/cNujK7VVt1jHGGLQNhA6i9258=; b=YdD0dwSXEfmRjBxzhTLjM1ewhfF0rOBkxdm9VQD186pTNSpHTPnlZsmfvX2N45wHXh 7TIfyUzKkKaMa59tLyegY4yy+fjfKrwAEwUC9scd9Dl2oel72IhYtar9ZyhYWoe804y+ 6pY2K0+CSybGc6Wb/gPBDhxmRkmmRuXlzP1H7Sanvj/mgyQ6fVvpJnZAwp+UmrG6L/Ci GKWMmxIRNxs132yh3KsyIsXc1zAB4cN8RtkQmXjF2F6l0wPCYGBrx//9IDU7ksgW6rwy I0ceq+4fX/6xC+TmK7EnnPcmUpfus4BcKne0fX7bOap4rv5LuJW7mXjWlhG0a95adhO8 VmZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=X/vLL5kgt+leeMxxc/cNujK7VVt1jHGGLQNhA6i9258=; b=lXwLJyoNEMkJZcQOqnzquZzOrGQkhO7dE+PJ55bCsJCoWbj2Vk8WOh+m4UFXlPkAge fxiLGBZaA99k78jsVzMzOif5lUtqxqGaO/hU6vScnjV3NeFqSb9jW8TO9yM+lJi3Lmg1 1NqeBF82aiYk3tS+oCDqG8C19drNiLCIZQcvuFwIIPFp/3AuPdTygS8BgzSSivAEwYy/ JlnvRwKSZ27lCPyKc32tHHY4xAO9OjUNh6bYfX8u8SdPBa2DCv3QCa9f81An/VPrTF+y Afg3DcOKLv2pXHF4Z/MyhPBIImEmO5VY4AsOWb0E1INxCk1AF4WVleFgSsu0uRrMAnn1 shxA== X-Gm-Message-State: APjAAAX02ihA8NWnH3DwrzuvPhIbyi/DmgteGPeoQDa3KqzBwWt0V0b6 aUw197kLJAGiqiApIAoHnFKLgnd6QIZGEQms X-Received: by 2002:adf:cf0e:: with SMTP id o14mr10234224wrj.277.1569778748055; Sun, 29 Sep 2019 10:39:08 -0700 (PDT) Received: from e123331-lin.nice.arm.com (bar06-5-82-246-156-241.fbx.proxad.net. [82.246.156.241]) by smtp.gmail.com with ESMTPSA id q192sm17339779wme.23.2019.09.29.10.39.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Sep 2019 10:39:07 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Herbert Xu , David Miller , Greg KH , Linus Torvalds , "Jason A . Donenfeld" , Samuel Neves , Dan Carpenter , Arnd Bergmann , Eric Biggers , Andy Lutomirski , Will Deacon , Marc Zyngier , Catalin Marinas , Martin Willi Subject: [RFC PATCH 06/20] crypto: x86/poly1305 - expose existing driver as poly1305 library Date: Sun, 29 Sep 2019 19:38:36 +0200 Message-Id: <20190929173850.26055-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190929173850.26055-1-ard.biesheuvel@linaro.org> References: <20190929173850.26055-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Implement the init/update/final Poly1305 library routines in the accelerated SIMD driver for x86 so they are accessible to users of the Poly1305 library interface. Signed-off-by: Ard Biesheuvel --- arch/x86/crypto/poly1305_glue.c | 60 +++++++++++++++----- crypto/Kconfig | 2 + 2 files changed, 48 insertions(+), 14 deletions(-) -- 2.17.1 diff --git a/arch/x86/crypto/poly1305_glue.c b/arch/x86/crypto/poly1305_glue.c index b43b93c95e79..d3cc92996f58 100644 --- a/arch/x86/crypto/poly1305_glue.c +++ b/arch/x86/crypto/poly1305_glue.c @@ -85,18 +85,11 @@ static unsigned int poly1305_simd_blocks(struct poly1305_desc_ctx *dctx, return srclen; } -static int poly1305_simd_update(struct shash_desc *desc, - const u8 *src, unsigned int srclen) +static int poly1305_simd_do_update(struct poly1305_desc_ctx *dctx, + const u8 *src, unsigned int srclen) { - struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); unsigned int bytes; - /* kernel_fpu_begin/end is costly, use fallback for small updates */ - if (srclen <= 288 || !crypto_simd_usable()) - return crypto_poly1305_update(desc, src, srclen); - - kernel_fpu_begin(); - if (unlikely(dctx->buflen)) { bytes = min(srclen, POLY1305_BLOCK_SIZE - dctx->buflen); memcpy(dctx->buf + dctx->buflen, src, bytes); @@ -117,8 +110,6 @@ static int poly1305_simd_update(struct shash_desc *desc, srclen = bytes; } - kernel_fpu_end(); - if (unlikely(srclen)) { dctx->buflen = srclen; memcpy(dctx->buf, src, srclen); @@ -127,6 +118,47 @@ static int poly1305_simd_update(struct shash_desc *desc, return 0; } +static int poly1305_simd_update(struct shash_desc *desc, + const u8 *src, unsigned int srclen) +{ + struct poly1305_desc_ctx *dctx = shash_desc_ctx(desc); + int ret; + + /* kernel_fpu_begin/end is costly, use fallback for small updates */ + if (srclen <= 288 || !crypto_simd_usable()) + return crypto_poly1305_update(desc, src, srclen); + + kernel_fpu_begin(); + ret = poly1305_simd_do_update(dctx, src, srclen); + kernel_fpu_end(); + + return ret; +} + +void poly1305_init(struct poly1305_desc_ctx *desc, const u8 *key) +{ + poly1305_init_generic(desc, key); +} +EXPORT_SYMBOL(poly1305_init); + +void poly1305_update(struct poly1305_desc_ctx *dctx, const u8 *src, + unsigned int nbytes) +{ + if (nbytes <= 288 || !crypto_simd_usable()) + return poly1305_update_generic(dctx, src, nbytes); + + kernel_fpu_begin(); + poly1305_simd_do_update(dctx, src, nbytes); + kernel_fpu_end(); +} +EXPORT_SYMBOL(poly1305_update); + +void poly1305_final(struct poly1305_desc_ctx *desc, u8 *digest) +{ + poly1305_final_generic(desc, digest); +} +EXPORT_SYMBOL(poly1305_final); + static struct shash_alg alg = { .digestsize = POLY1305_DIGEST_SIZE, .init = crypto_poly1305_init, @@ -151,9 +183,9 @@ static int __init poly1305_simd_mod_init(void) boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_AVX2) && cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL); - alg.descsize = sizeof(struct poly1305_desc_ctx) + 5 * sizeof(u32); - if (poly1305_use_avx2) - alg.descsize += 10 * sizeof(u32); + alg.descsize = sizeof(struct poly1305_desc_ctx); + if (!poly1305_use_avx2) + alg.descsize -= 10 * sizeof(u32); return crypto_register_shash(&alg); } diff --git a/crypto/Kconfig b/crypto/Kconfig index f40e8dca57d1..6a952a61675b 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -659,6 +659,7 @@ config CRYPTO_ARCH_HAVE_LIB_POLY1305 config CRYPTO_LIB_POLY1305_RSIZE int + default 4 if X86_64 default 1 config CRYPTO_LIB_POLY1305 @@ -680,6 +681,7 @@ config CRYPTO_POLY1305_X86_64 tristate "Poly1305 authenticator algorithm (x86_64/SSE2/AVX2)" depends on X86 && 64BIT select CRYPTO_POLY1305 + select CRYPTO_ARCH_HAVE_LIB_POLY1305 help Poly1305 authenticator algorithm, RFC7539.