From patchwork Fri Jan 25 09:36:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 156569 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp140474jaa; Fri, 25 Jan 2019 01:36:38 -0800 (PST) X-Google-Smtp-Source: ALg8bN71zKXc31g+1+1saK/Ny74gmWkm8+2pPACz0lTEwlDXcz5xUntxM3xDA5kMoXD+B9Z2JmxL X-Received: by 2002:a63:ef04:: with SMTP id u4mr9362284pgh.197.1548408998519; Fri, 25 Jan 2019 01:36:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1548408998; cv=none; d=google.com; s=arc-20160816; b=PnSo8pLFZ1bWA0hqkbphK1pjriJ+bJbEA93aMK2SeE2oPtvKBOpIoMGBS2t5lgfvPo VIFP55W3raRlCc4IkH7VVj+snZDIiqkk4HajoT+SmWl35E7RGqGD6ELyi4KdNCawcmIr eVH/a8wXm3GxuXInVumi5Z0oTLzgjbAagn0SmY7ez3++PRlflmsztwXxGs7Qdkonn50i rEXyDzxVGPprIUgA7KhIZS+V/ljGyDBtgy/+Vu7xamu41GPTPys8vqtLHjdHc9MaZ7el 7x7lcOmGfdofDykXBkz9BCsrryFZYBcHv+IFJF2556mvPlMLeX8HOCMlLXqk+8PMtcOv x6yg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=pHeXQyjHBGPZ2p98q1m4Ogx2QHTHJsPCH+kq0wtIQtM=; b=lM0Q/6Pxc6CQ2MbVW4x/2TPYpxgtfeb+YzS23Z/uiWgy4D+BGjrJrivVvrBOVDwIdE F+OH5LZ86vOO/uGcSgc6gR5EmCgDuZLzIQggv+UAkgQceH9WBREw5GZHmoUMNPC+6SHh 2FwfAZGGc+AZpp0+AqmBLvNrlTGFTCB37ptaFpFNU0WZc54RfM9yNH1Sb3sz0D7B23+N EfFPhNkbwsgdQ7heBrNeOAh2GIPBBtT/1hXs7Dps6YRJmbJ8aWfMNTfSS9OpTsxzaiPO qotHHGg0zRT93ObMfyqudEY8XfHx2V8onF/7lCiVs/ZL3ckJKNtCRiSlf3wPhj+EBY9r A/fg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=a1O8FZXP; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v141si26360336pfc.260.2019.01.25.01.36.38; Fri, 25 Jan 2019 01:36:38 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=a1O8FZXP; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727097AbfAYJgh (ORCPT + 3 others); Fri, 25 Jan 2019 04:36:37 -0500 Received: from mail-wm1-f66.google.com ([209.85.128.66]:35497 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726269AbfAYJgh (ORCPT ); Fri, 25 Jan 2019 04:36:37 -0500 Received: by mail-wm1-f66.google.com with SMTP id t200so6002166wmt.0 for ; Fri, 25 Jan 2019 01:36:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=pHeXQyjHBGPZ2p98q1m4Ogx2QHTHJsPCH+kq0wtIQtM=; b=a1O8FZXPbRSdLVvC+7SFfRYY57XdY8Bns4xivd2uaq2ijEzntpWiiG0LV9zF3oXew8 Q7p9mE8EEOaxKjBWigr8x12ITPZUa+hrdzO5mtHaFfeMae3XFZQ9VpdPwxF0QqrspM5/ 0ESVJjKZ1JAUhzV5qFFT4/HtrsEnBTXdxwchA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=pHeXQyjHBGPZ2p98q1m4Ogx2QHTHJsPCH+kq0wtIQtM=; b=qbql5E7Pe216dbmQciwcEVhvX2H80MO6GmA/EpNXVvv9cBk1mGBZp1caZ0+MLlegXD R1IqGn/Iskh1Vt3WCZXx6yQF7X6uDxGWIZA5etypBZcZWgFCp6Y5ELI7Dpiy+eCOMnx6 eHxL+mfu+Lk8DrktL0gvpNf6Y/y1qvFPxKykr4qiwkrPvRIKCDlKsDixrc17WORaMo3i BeVextW1fJzpdb0BUXUOi+m7O4XvGzGOvBESxBT5QIRc9IjR4F+3AvkVAlPti8gq9sWz +abwfHrJ740qVSF3NSZIECAReDfSOcOvBPVXeYGaKfLQbLVW45Aw64j62jEGq4jEiiBU I/dA== X-Gm-Message-State: AJcUukepovySLJyabOQbmRuxQGFDWSuqFra48PKD0+E4jos9VR0G05H0 XzNMqqKoEfZNZvtna9rjKIoddhs42OqG5g== X-Received: by 2002:a1c:1383:: with SMTP id 125mr6346548wmt.71.1548408994700; Fri, 25 Jan 2019 01:36:34 -0800 (PST) Received: from dogfood.home ([2a01:cb1d:112:6f00:bcd4:806e:230a:673e]) by smtp.gmail.com with ESMTPSA id w80sm92323955wme.38.2019.01.25.01.36.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 25 Jan 2019 01:36:33 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 1/2] crypto: arm64/crct10dif - register PMULL variants as separate algos Date: Fri, 25 Jan 2019 10:36:26 +0100 Message-Id: <20190125093627.14288-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190125093627.14288-1-ard.biesheuvel@linaro.org> References: <20190125093627.14288-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The arm64 CRC-T10DIF implementation either uses 8-bit or 64-bit polynomial multiplication instructions, since the latter are faster but not mandatory in the architecture. Since that prevents us from testing both implementations on the same system, let's expose both implementations to the crypto API, with the priorities reflecting that the P64 version is the preferred one if available. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/crct10dif-ce-glue.c | 54 +++++++++++++++----- 1 file changed, 42 insertions(+), 12 deletions(-) -- 2.17.1 diff --git a/arch/arm64/crypto/crct10dif-ce-glue.c b/arch/arm64/crypto/crct10dif-ce-glue.c index 567c24f3d224..242757cc6da9 100644 --- a/arch/arm64/crypto/crct10dif-ce-glue.c +++ b/arch/arm64/crypto/crct10dif-ce-glue.c @@ -25,8 +25,6 @@ asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 buf[], u64 len); asmlinkage u16 crc_t10dif_pmull_p8(u16 init_crc, const u8 buf[], u64 len); -static u16 (*crc_t10dif_pmull)(u16 init_crc, const u8 buf[], u64 len); - static int crct10dif_init(struct shash_desc *desc) { u16 *crc = shash_desc_ctx(desc); @@ -35,14 +33,30 @@ static int crct10dif_init(struct shash_desc *desc) return 0; } -static int crct10dif_update(struct shash_desc *desc, const u8 *data, +static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data, + unsigned int length) +{ + u16 *crc = shash_desc_ctx(desc); + + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { + kernel_neon_begin(); + *crc = crc_t10dif_pmull_p8(*crc, data, length); + kernel_neon_end(); + } else { + *crc = crc_t10dif_generic(*crc, data, length); + } + + return 0; +} + +static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data, unsigned int length) { u16 *crc = shash_desc_ctx(desc); if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { kernel_neon_begin(); - *crc = crc_t10dif_pmull(*crc, data, length); + *crc = crc_t10dif_pmull_p64(*crc, data, length); kernel_neon_end(); } else { *crc = crc_t10dif_generic(*crc, data, length); @@ -59,10 +73,22 @@ static int crct10dif_final(struct shash_desc *desc, u8 *out) return 0; } -static struct shash_alg crc_t10dif_alg = { +static struct shash_alg crc_t10dif_alg[] = {{ .digestsize = CRC_T10DIF_DIGEST_SIZE, .init = crct10dif_init, - .update = crct10dif_update, + .update = crct10dif_update_pmull_p8, + .final = crct10dif_final, + .descsize = CRC_T10DIF_DIGEST_SIZE, + + .base.cra_name = "crct10dif", + .base.cra_driver_name = "crct10dif-arm64-neon", + .base.cra_priority = 100, + .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, + .base.cra_module = THIS_MODULE, +}, { + .digestsize = CRC_T10DIF_DIGEST_SIZE, + .init = crct10dif_init, + .update = crct10dif_update_pmull_p64, .final = crct10dif_final, .descsize = CRC_T10DIF_DIGEST_SIZE, @@ -71,21 +97,25 @@ static struct shash_alg crc_t10dif_alg = { .base.cra_priority = 200, .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, .base.cra_module = THIS_MODULE, -}; +}}; static int __init crc_t10dif_mod_init(void) { if (elf_hwcap & HWCAP_PMULL) - crc_t10dif_pmull = crc_t10dif_pmull_p64; + return crypto_register_shashes(crc_t10dif_alg, + ARRAY_SIZE(crc_t10dif_alg)); else - crc_t10dif_pmull = crc_t10dif_pmull_p8; - - return crypto_register_shash(&crc_t10dif_alg); + /* only register the first array element */ + return crypto_register_shash(crc_t10dif_alg); } static void __exit crc_t10dif_mod_exit(void) { - crypto_unregister_shash(&crc_t10dif_alg); + if (elf_hwcap & HWCAP_PMULL) + crypto_unregister_shashes(crc_t10dif_alg, + ARRAY_SIZE(crc_t10dif_alg)); + else + crypto_unregister_shash(crc_t10dif_alg); } module_cpu_feature_match(ASIMD, crc_t10dif_mod_init);