From patchwork Thu May 1 15:51:24 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 29520 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-yh0-f69.google.com (mail-yh0-f69.google.com [209.85.213.69]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 6AA39203F3 for ; Thu, 1 May 2014 15:51:44 +0000 (UTC) Received: by mail-yh0-f69.google.com with SMTP id b6sf12101455yha.4 for ; Thu, 01 May 2014 08:51:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:in-reply-to:references:sender:precedence:list-id :x-original-sender:x-original-authentication-results:mailing-list :list-post:list-help:list-archive:list-unsubscribe; bh=rM8uqrff35Qo3gl+WcY/8h73Cs6o0MokvW7zO+pj3F4=; b=bmM5TKbm26vDWRDw78rxaznX3BjYqVW9+kw3ejvLxAHr+JQiLvPqp+Db33fi55T0EQ 9Hw6eijcJjuSPzV/FY0gi9/DaspFl5gD/Mb6WAiySj0iXdIJCzmEHxIajtsWVAYxMoSV cBQyeGVVoxQiPVjbYMYpAyHcbkHMaYezOzbqPz4us0munvM2dQHH3luzd7k5UvpPD4hO AHTyOThbJswjCcpL8t3NEP+CSkOuQxsONFnaihLL3Rg9avFifPtlYWywDL5xlNEsLSa3 aAfAC5wE5l+cAXOwuQpLFLXUAAmOLLkTKqpJfISyYk+0t/p8H406E0vTsDij4Pb8OaZs R6BQ== X-Gm-Message-State: ALoCoQnhj2vDZw6BmJ3WbYKNnDQZIrhrzlKp8fT27xZaHnpD+tE2vrmtva70s/7O5HaXI9cV+8rx X-Received: by 10.224.36.137 with SMTP id t9mr5933698qad.4.1398959504136; Thu, 01 May 2014 08:51:44 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.101.4 with SMTP id t4ls1133501qge.1.gmail; Thu, 01 May 2014 08:51:43 -0700 (PDT) X-Received: by 10.58.195.202 with SMTP id ig10mr510312vec.33.1398959503961; Thu, 01 May 2014 08:51:43 -0700 (PDT) Received: from mail-ve0-f172.google.com (mail-ve0-f172.google.com [209.85.128.172]) by mx.google.com with ESMTPS id pd4si6126484veb.105.2014.05.01.08.51.43 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 01 May 2014 08:51:43 -0700 (PDT) Received-SPF: none (google.com: patch+caf_=patchwork-forward=linaro.org@linaro.org does not designate permitted sender hosts) client-ip=209.85.128.172; Received: by mail-ve0-f172.google.com with SMTP id jy13so4022641veb.17 for ; Thu, 01 May 2014 08:51:43 -0700 (PDT) X-Received: by 10.58.24.231 with SMTP id x7mr460027vef.71.1398959503889; Thu, 01 May 2014 08:51:43 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.221.72 with SMTP id ib8csp34105vcb; Thu, 1 May 2014 08:51:43 -0700 (PDT) X-Received: by 10.50.143.34 with SMTP id sb2mr3952853igb.48.1398959502645; Thu, 01 May 2014 08:51:42 -0700 (PDT) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w2si3639369igw.7.2014.05.01.08.51.42; Thu, 01 May 2014 08:51:42 -0700 (PDT) Received-SPF: none (google.com: linux-crypto-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752629AbaEAPvj (ORCPT ); Thu, 1 May 2014 11:51:39 -0400 Received: from mail-wi0-f177.google.com ([209.85.212.177]:34127 "EHLO mail-wi0-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752891AbaEAPvj (ORCPT ); Thu, 1 May 2014 11:51:39 -0400 Received: by mail-wi0-f177.google.com with SMTP id cc10so969753wib.4 for ; Thu, 01 May 2014 08:51:37 -0700 (PDT) X-Received: by 10.194.189.116 with SMTP id gh20mr9550582wjc.41.1398959497918; Thu, 01 May 2014 08:51:37 -0700 (PDT) Received: from ards-macbook-pro.local (cag06-7-83-153-85-71.fbx.proxad.net. [83.153.85.71]) by mx.google.com with ESMTPSA id kp5sm41727776wjb.30.2014.05.01.08.51.36 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 01 May 2014 08:51:37 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-crypto@vger.kernel.org Cc: catalin.marinas@arm.com, will.deacon@arm.com, steve.capper@linaro.org, Ard Biesheuvel Subject: [PATCH resend 13/15] arm64/crypto: add voluntary preemption to Crypto Extensions SHA1 Date: Thu, 1 May 2014 17:51:24 +0200 Message-Id: <1398959486-8222-4-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1398959486-8222-1-git-send-email-ard.biesheuvel@linaro.org> References: <1398959486-8222-1-git-send-email-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ard.biesheuvel@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: patch+caf_=patchwork-forward=linaro.org@linaro.org does not designate permitted sender hosts) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , The Crypto Extensions based SHA1 implementation uses the NEON register file, and hence runs with preemption disabled. This patch adds a TIF_NEED_RESCHED check to its inner loop so we at least give up the CPU voluntarily when we are running in process context and have been tagged for preemption by the scheduler. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha1-ce-core.S | 19 ++++++++-------- arch/arm64/crypto/sha1-ce-glue.c | 49 +++++++++++++++++++++++++++++++--------- 2 files changed, 48 insertions(+), 20 deletions(-) diff --git a/arch/arm64/crypto/sha1-ce-core.S b/arch/arm64/crypto/sha1-ce-core.S index bd4af29f2722..831e332f9331 100644 --- a/arch/arm64/crypto/sha1-ce-core.S +++ b/arch/arm64/crypto/sha1-ce-core.S @@ -66,8 +66,8 @@ .word 0x5a827999, 0x6ed9eba1, 0x8f1bbcdc, 0xca62c1d6 /* - * void sha1_ce_transform(int blocks, u8 const *src, u32 *state, - * u8 *head, long bytes) + * int sha1_ce_transform(int blocks, u8 const *src, u32 *state, + * u8 *head, long bytes, struct thread_info *ti) */ ENTRY(sha1_ce_transform) /* load round constants */ @@ -127,7 +127,13 @@ CPU_LE( rev32 v11.16b, v11.16b ) add dgbv.2s, dgbv.2s, dg1v.2s add dgav.4s, dgav.4s, dg0v.4s - cbnz w0, 0b + cbz w0, 4f + b_if_no_resched x5, x8, 0b + + /* store new state */ +3: str dga, [x2] + str dgb, [x2, #16] + ret /* * Final block: add padding and total bit count. @@ -135,7 +141,7 @@ CPU_LE( rev32 v11.16b, v11.16b ) * size was not a round multiple of the block size, and the padding is * handled by the C code. */ - cbz x4, 3f +4: cbz x4, 3b movi v9.2d, #0 mov x8, #0x80000000 movi v10.2d, #0 @@ -145,9 +151,4 @@ CPU_LE( rev32 v11.16b, v11.16b ) mov v11.d[0], xzr mov v11.d[1], x7 b 2b - - /* store new state */ -3: str dga, [x2] - str dgb, [x2, #16] - ret ENDPROC(sha1_ce_transform) diff --git a/arch/arm64/crypto/sha1-ce-glue.c b/arch/arm64/crypto/sha1-ce-glue.c index 6fe83f37a750..69850a163668 100644 --- a/arch/arm64/crypto/sha1-ce-glue.c +++ b/arch/arm64/crypto/sha1-ce-glue.c @@ -20,8 +20,8 @@ MODULE_DESCRIPTION("SHA1 secure hash using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); -asmlinkage void sha1_ce_transform(int blocks, u8 const *src, u32 *state, - u8 *head, long bytes); +asmlinkage int sha1_ce_transform(int blocks, u8 const *src, u32 *state, + u8 *head, long bytes, struct thread_info *ti); static int sha1_init(struct shash_desc *desc) { @@ -42,6 +42,7 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, sctx->count += len; if ((partial + len) >= SHA1_BLOCK_SIZE) { + struct thread_info *ti = NULL; int blocks; if (partial) { @@ -52,16 +53,30 @@ static int sha1_update(struct shash_desc *desc, const u8 *data, len -= p; } + /* + * Pass current's thread info pointer to sha1_ce_transform() + * below if we want it to play nice under preemption. + */ + if ((IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) || + IS_ENABLED(CONFIG_PREEMPT)) && !in_interrupt()) + ti = current_thread_info(); + blocks = len / SHA1_BLOCK_SIZE; len %= SHA1_BLOCK_SIZE; - kernel_neon_begin_partial(16); - sha1_ce_transform(blocks, data, sctx->state, - partial ? sctx->buffer : NULL, 0); - kernel_neon_end(); + do { + int rem; + + kernel_neon_begin_partial(16); + rem = sha1_ce_transform(blocks, data, sctx->state, + partial ? sctx->buffer : NULL, + 0, ti); + kernel_neon_end(); - data += blocks * SHA1_BLOCK_SIZE; - partial = 0; + data += (blocks - rem) * SHA1_BLOCK_SIZE; + blocks = rem; + partial = 0; + } while (unlikely(ti && blocks > 0)); } if (len) memcpy(sctx->buffer + partial, data, len); @@ -94,6 +109,7 @@ static int sha1_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { struct sha1_state *sctx = shash_desc_ctx(desc); + struct thread_info *ti = NULL; __be32 *dst = (__be32 *)out; int blocks; int i; @@ -111,9 +127,20 @@ static int sha1_finup(struct shash_desc *desc, const u8 *data, */ blocks = len / SHA1_BLOCK_SIZE; - kernel_neon_begin_partial(16); - sha1_ce_transform(blocks, data, sctx->state, NULL, len); - kernel_neon_end(); + if ((IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) || + IS_ENABLED(CONFIG_PREEMPT)) && !in_interrupt()) + ti = current_thread_info(); + + do { + int rem; + + kernel_neon_begin_partial(16); + rem = sha1_ce_transform(blocks, data, sctx->state, + NULL, len, ti); + kernel_neon_end(); + data += (blocks - rem) * SHA1_BLOCK_SIZE; + blocks = rem; + } while (unlikely(ti && blocks > 0)); for (i = 0; i < SHA1_DIGEST_SIZE / sizeof(__be32); i++) put_unaligned_be32(sctx->state[i], dst++);