From patchwork Thu Jan 24 18:27:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 156505 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp2284421jaa; Thu, 24 Jan 2019 10:27:23 -0800 (PST) X-Google-Smtp-Source: ALg8bN5gp1gWy1pgeYrKIrl87FB7/0Tbt7KOvxsjepAlLA+hh3f8heQIXg4jcuf0Jj1NLrXCIkp/ X-Received: by 2002:a63:8c0d:: with SMTP id m13mr6987188pgd.422.1548354443028; Thu, 24 Jan 2019 10:27:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1548354443; cv=none; d=google.com; s=arc-20160816; b=zNopygkoseHODWGBTGDjzNNf7x0QOjU34BT/nKaF/R6mtcxlSYXXyKc0ZDUB/xxrEL cANW6Z7cEt51Qe08H2MnByg8Mdop3ePa0OLm1fpEb+Hl2qJfUhqO3qBZoBIikUIspJG3 c5M7nSzHRokIvbn7Xx84s9j1MVK9CNwcTBgxj44XZZdUCYaykkZ8qm7oGmZjaW5qKIoG g1fmP1ORBklTGbZXgZmIWx55EdVYxAp9NMb/L7l99dQDaFA/e757Rva+mCOUm6B+VQLi Jy/7LB0Eifb/SJ8z/JbVhQK+xDfeEE7BNQr5/3qBqZKcfvdWEs/0y8gPB47y83TKDRx+ SyCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=usyANgO+md6MAxAYC+xPQ71SM8cvmj1kbE42YVBL7tM=; b=0/ZVq1ZIhz9NPoAywImAOrzJLxRy1UT//MieOX1gOn6ZMUoM0IjvZ6CVxXwcKnZD8N +5sqtIiiv3qvoDlZv22qEXOt57/aSzrRfVyRdZgVZa1OnMqvaE01LzIYjQcGx0WQ54ag vBFONR/R6gX1mDWQPrQsJxXg5Qx644pLGsE6RyBMbyNJ8RmjFHOveRnfgIaI25TP+VdP T6TWwQrANp3k20YQH62j0eSvA3kXqNY4Qm8btPqYj9ECwQjWuZ3mXtQxP0FPkDjsolGc jdV69ZvY7DCJ6+QIUU4wZC2duwjf/lSQuiwGdT7HdtdWuoVY8FwDvBil/SZfe+z55jvf /Qxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hl6687nM; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n4si22707825pgm.487.2019.01.24.10.27.22; Thu, 24 Jan 2019 10:27:23 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hl6687nM; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726026AbfAXS1W (ORCPT + 3 others); Thu, 24 Jan 2019 13:27:22 -0500 Received: from mail-wm1-f65.google.com ([209.85.128.65]:39538 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725909AbfAXS1W (ORCPT ); Thu, 24 Jan 2019 13:27:22 -0500 Received: by mail-wm1-f65.google.com with SMTP id y8so4106117wmi.4 for ; Thu, 24 Jan 2019 10:27:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=usyANgO+md6MAxAYC+xPQ71SM8cvmj1kbE42YVBL7tM=; b=hl6687nMxC/Lvjr/tM2DjrjI/c8ecYNEFmGYcpWIz77p8QVJs+H+jeO7eVzqdxvccp mTMjUSf5HKVBt5Zic8+nAiGOYWobjoBpdi5d5nsFkkGsb42uTDGMuzhsyH06T4Yk6AE3 jnTYDN0edqWfUgdVxi/XmW+dAlLOU3jED3uRQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=usyANgO+md6MAxAYC+xPQ71SM8cvmj1kbE42YVBL7tM=; b=cn8va3o4vYbEUpyUCxWLzPXWRhgZrTx7NXyvlVNj7+5RDGRfUkAth7kaK6frJGRe7s r9Q2cOE+PFJEQUGZ3ciVxtuNGrPRHesrX/uO/h0kpYTEGVSjHehvTp6y9+zJmiVGQaxT USisBf494T3jn46tCGCJguRkNbBihsNkaHxmEQz/YY/fx/e9e4Dkup4y/OQedSzeNg3l Anvb8KVlEO2NHb8GvrQGwB3JGsIyChvaZcGiXomxXUDliAqFMwmd8m6SlQdHgJOruyiL 4Zj37+ZSQaqB4StJNrq4C2jMaaIR18bqKIrv4OKbOkCkNJPtW9vEz4tZlb7AUNj/kG8V x/OQ== X-Gm-Message-State: AJcUukfz+Wgz3Wtnv6RqlSGJIdRoOyAdtv3Ar99copnNqaLX5vBq6wDy sHZVYW47xMXUubE9sFYRtmGqQwN1n5rpag== X-Received: by 2002:a1c:e044:: with SMTP id x65mr3685792wmg.86.1548354439745; Thu, 24 Jan 2019 10:27:19 -0800 (PST) Received: from dogfood.home ([2a01:cb1d:112:6f00:bcd4:806e:230a:673e]) by smtp.gmail.com with ESMTPSA id 202sm73247138wmt.8.2019.01.24.10.27.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Jan 2019 10:27:18 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 1/2] crypto: arm/crct10dif - revert to C code for short inputs Date: Thu, 24 Jan 2019 19:27:11 +0100 Message-Id: <20190124182712.7142-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190124182712.7142-1-ard.biesheuvel@linaro.org> References: <20190124182712.7142-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The SIMD routine ported from x86 used to have a special code path for inputs < 16 bytes, which got lost somewhere along the way. Instead, the current glue code aligns the input pointer to permit the NEON routine to use special versions of the vld1 instructions that assume 16 byte alignment, but this could result in inputs of less than 16 bytes to be passed in. This not only fails the new extended tests that Eric has implemented, it also results in the code reading before the input pointer, which could potentially result in crashes when dealing with less than 16 bytes of input at the start of a page which is preceded by an unmapped page. So update the glue code to only invoke the NEON routine if the input is more than 16 bytes. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/crct10dif-ce-core.S | 20 ++++++++--------- arch/arm/crypto/crct10dif-ce-glue.c | 23 +++++--------------- 2 files changed, 16 insertions(+), 27 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/crct10dif-ce-core.S b/arch/arm/crypto/crct10dif-ce-core.S index ce45ba0c0687..3fd13d7c842c 100644 --- a/arch/arm/crypto/crct10dif-ce-core.S +++ b/arch/arm/crypto/crct10dif-ce-core.S @@ -124,10 +124,10 @@ ENTRY(crc_t10dif_pmull) vext.8 q10, qzr, q0, #4 // receive the initial 64B data, xor the initial crc value - vld1.64 {q0-q1}, [arg2, :128]! - vld1.64 {q2-q3}, [arg2, :128]! - vld1.64 {q4-q5}, [arg2, :128]! - vld1.64 {q6-q7}, [arg2, :128]! + vld1.64 {q0-q1}, [arg2]! + vld1.64 {q2-q3}, [arg2]! + vld1.64 {q4-q5}, [arg2]! + vld1.64 {q6-q7}, [arg2]! CPU_LE( vrev64.8 q0, q0 ) CPU_LE( vrev64.8 q1, q1 ) CPU_LE( vrev64.8 q2, q2 ) @@ -150,7 +150,7 @@ CPU_LE( vrev64.8 q7, q7 ) veor.8 q0, q0, q10 adr ip, rk3 - vld1.64 {q10}, [ip, :128] // xmm10 has rk3 and rk4 + vld1.64 {q10}, [ip] // xmm10 has rk3 and rk4 // // we subtract 256 instead of 128 to save one instruction from the loop @@ -167,7 +167,7 @@ CPU_LE( vrev64.8 q7, q7 ) _fold_64_B_loop: .macro fold64, reg1, reg2 - vld1.64 {q11-q12}, [arg2, :128]! + vld1.64 {q11-q12}, [arg2]! vmull.p64 q8, \reg1\()h, d21 vmull.p64 \reg1, \reg1\()l, d20 @@ -203,13 +203,13 @@ CPU_LE( vrev64.8 q12, q12 ) // constants adr ip, rk9 - vld1.64 {q10}, [ip, :128]! + vld1.64 {q10}, [ip]! .macro fold16, reg, rk vmull.p64 q8, \reg\()l, d20 vmull.p64 \reg, \reg\()h, d21 .ifnb \rk - vld1.64 {q10}, [ip, :128]! + vld1.64 {q10}, [ip]! .endif veor.8 q7, q7, q8 veor.8 q7, q7, \reg @@ -238,7 +238,7 @@ _16B_reduction_loop: vmull.p64 q7, d15, d21 veor.8 q7, q7, q8 - vld1.64 {q0}, [arg2, :128]! + vld1.64 {q0}, [arg2]! CPU_LE( vrev64.8 q0, q0 ) vswp d0, d1 veor.8 q7, q7, q0 @@ -335,7 +335,7 @@ _less_than_128: vmov.i8 q0, #0 vmov s3, arg1_low32 // get the initial crc value - vld1.64 {q7}, [arg2, :128]! + vld1.64 {q7}, [arg2]! CPU_LE( vrev64.8 q7, q7 ) vswp d14, d15 veor.8 q7, q7, q0 diff --git a/arch/arm/crypto/crct10dif-ce-glue.c b/arch/arm/crypto/crct10dif-ce-glue.c index d428355cf38d..14c19c70a841 100644 --- a/arch/arm/crypto/crct10dif-ce-glue.c +++ b/arch/arm/crypto/crct10dif-ce-glue.c @@ -35,26 +35,15 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data, unsigned int length) { u16 *crc = shash_desc_ctx(desc); - unsigned int l; - if (!may_use_simd()) { - *crc = crc_t10dif_generic(*crc, data, length); + if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) { + kernel_neon_begin(); + *crc = crc_t10dif_pmull(*crc, data, length); + kernel_neon_end(); } else { - if (unlikely((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE)) { - l = min_t(u32, length, CRC_T10DIF_PMULL_CHUNK_SIZE - - ((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE)); - - *crc = crc_t10dif_generic(*crc, data, l); - - length -= l; - data += l; - } - if (length > 0) { - kernel_neon_begin(); - *crc = crc_t10dif_pmull(*crc, data, length); - kernel_neon_end(); - } + *crc = crc_t10dif_generic(*crc, data, length); } + return 0; }