From patchwork Tue Dec 26 10:29:25 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122742 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp782728qgn; Tue, 26 Dec 2017 02:34:57 -0800 (PST) X-Google-Smtp-Source: ACJfBov/H496C2CDgnkboM1Y6MFv5iRbdzXp07maucDF9qpio5+aAGj1MHeQ9LGQs8MNEI+KOnOg X-Received: by 10.159.241.1 with SMTP id q1mr25163600plr.383.1514284497371; Tue, 26 Dec 2017 02:34:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514284497; cv=none; d=google.com; s=arc-20160816; b=yGS/hV7dsVj2OLN96IhxEyrgSiJLIw9PECOD0GzZEvmOpJg3rK+/tz/D+Sc9434eZe Y8m4wI+sGSu+GuBI5Sogfi1y/b2HJfEl8TAOnv6j60+eKz84zV3S1GfvQ9DECzCGVnqL bW+trdvYLGAIHwQs3I2J9DHoSW3ZaDOVTADzp5H8Y14DWaaAI8q7cOoi1WwqusjZFIta 6dUtXVlL92xZwGQJSR/cavMc0ST76LL+ZeJvV3OcTcinWvDoVn/c9/nFAJqrNp60uVuP z9U1yoF0x5Ma/I3SJdFEnY4lxJv7XtNaJDlFWzQxmth04L2fOridPGesTxiVNrpD3qIO 05tA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=IhTik8BmpN/IjyhBdu2dYZZ72sUFaQrQeiEa5qMgGQk=; b=mHm2oFDoKMiEF3+89E87hDBHCagAIS4bH4hTZDwHJh5+1YsDyAiwzknAAJCviFmaSx kLfN0Bzyujaeki7KE2yhLDekk6Kh39dnx/WfEHwFY4CseRW12bHcRFTpUADvFZiJI2z1 zRxS8WeM5jdoUcKo7JBdOJSeg3fPh3yMfnleq4LisXsKY5rpNRhF7B2tpl7/XafxH7jg evuRCManl3KLN1yOfO2J3pQg4XQ+uckMDE4A8q3LFcxmVC9hBvtFGPwLo98rVtK3tQVn V938Vm9opgtUquaoQ6J48uAu7QnAuv5LLc9MuppcCuSswmAFSL/NaIL3eC3a9N7AvIVc JzpQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PFo46X6T; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u85si22845532pfi.278.2017.12.26.02.34.57; Tue, 26 Dec 2017 02:34:57 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PFo46X6T; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751441AbdLZKey (ORCPT + 28 others); Tue, 26 Dec 2017 05:34:54 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:44620 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751055AbdLZKay (ORCPT ); Tue, 26 Dec 2017 05:30:54 -0500 Received: by mail-wm0-f68.google.com with SMTP id t8so34450180wmc.3 for ; Tue, 26 Dec 2017 02:30:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IhTik8BmpN/IjyhBdu2dYZZ72sUFaQrQeiEa5qMgGQk=; b=PFo46X6TkvWOtXe7FM+ZEv5Ex7Ujw3V4qVQa1uSh6NYG1FDY/3YWOv5QjWaGVswklk Q243Kg8U7v7NeE8HznjJd5Hu1HdP7y1VR9eJ3n6R98F4YsrGjs3aBDQ6FmJAurKjlvgX C9hJvreBD79fZq2bsV4l51/WbUJkwBt5uIwyw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IhTik8BmpN/IjyhBdu2dYZZ72sUFaQrQeiEa5qMgGQk=; b=dzZwydc4Mo+wKZojzPetzRlOf703qSMrRA/rSIISvfj4cXYcSY1TMseiGQOzbBu54v 8mlzTIB3RdKDyV2s3v5sHghs9tDEHQq6S0xMdEKhA5nC5G+8Or0rkkUInUOlds8xtSHB o+Fj0+5453YLLNrdEAH4xvEb9ReOjAmzBzIWPaGFTIdmPWYA5kqWXiNM0n1vSdj9Nc3B o8xxKcrUlYrz+bcZllmF4n2K2fDXL/iEHaI7WiU/bz/betRJA5HhFgcEMXnAnpRQJJ10 XLR0QNe20HjtODbchR4dgaR+ItxMQ4A2BOPdoP4rsVw6NCj1l2UmdlmlECp0C5MOatk9 QmvQ== X-Gm-Message-State: AKGB3mLhRLw/wvt7mhP5db0ImD4auM+VYyb53e7czFmK4lK2zPR32BOu qzpfvG/42LBehDclYMYde6lSdF3eJYA= X-Received: by 10.28.15.201 with SMTP id 192mr18925702wmp.97.1514284253276; Tue, 26 Dec 2017 02:30:53 -0800 (PST) Received: from localhost.localdomain ([160.171.216.245]) by smtp.gmail.com with ESMTPSA id l142sm13974036wmb.43.2017.12.26.02.30.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Dec 2017 02:30:52 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v4 05/20] crypto: arm64/chacha20 - move kernel mode neon en/disable into loop Date: Tue, 26 Dec 2017 10:29:25 +0000 Message-Id: <20171226102940.26908-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171226102940.26908-1-ard.biesheuvel@linaro.org> References: <20171226102940.26908-1-ard.biesheuvel@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When kernel mode NEON was first introduced on arm64, the preserve and restore of the userland NEON state was completely unoptimized, and involved saving all registers on each call to kernel_neon_begin(), and restoring them on each call to kernel_neon_end(). For this reason, the NEON crypto code that was introduced at the time keeps the NEON enabled throughout the execution of the crypto API methods, which may include calls back into the crypto API that could result in memory allocation or other actions that we should avoid when running with preemption disabled. Since then, we have optimized the kernel mode NEON handling, which now restores lazily (upon return to userland), and so the preserve action is only costly the first time it is called after entering the kernel. So let's put the kernel_neon_begin() and kernel_neon_end() calls around the actual invocations of the NEON crypto code, and run the remainder of the code with kernel mode NEON disabled (and preemption enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/chacha20-neon-glue.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) -- 2.11.0 diff --git a/arch/arm64/crypto/chacha20-neon-glue.c b/arch/arm64/crypto/chacha20-neon-glue.c index cbdb75d15cd0..727579c93ded 100644 --- a/arch/arm64/crypto/chacha20-neon-glue.c +++ b/arch/arm64/crypto/chacha20-neon-glue.c @@ -37,12 +37,19 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, u8 buf[CHACHA20_BLOCK_SIZE]; while (bytes >= CHACHA20_BLOCK_SIZE * 4) { + kernel_neon_begin(); chacha20_4block_xor_neon(state, dst, src); + kernel_neon_end(); bytes -= CHACHA20_BLOCK_SIZE * 4; src += CHACHA20_BLOCK_SIZE * 4; dst += CHACHA20_BLOCK_SIZE * 4; state[12] += 4; } + + if (!bytes) + return; + + kernel_neon_begin(); while (bytes >= CHACHA20_BLOCK_SIZE) { chacha20_block_xor_neon(state, dst, src); bytes -= CHACHA20_BLOCK_SIZE; @@ -55,6 +62,7 @@ static void chacha20_doneon(u32 *state, u8 *dst, const u8 *src, chacha20_block_xor_neon(state, buf, buf); memcpy(dst, buf, bytes); } + kernel_neon_end(); } static int chacha20_neon(struct skcipher_request *req) @@ -68,11 +76,10 @@ static int chacha20_neon(struct skcipher_request *req) if (!may_use_simd() || req->cryptlen <= CHACHA20_BLOCK_SIZE) return crypto_chacha20_crypt(req); - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); crypto_chacha20_init(state, ctx, walk.iv); - kernel_neon_begin(); while (walk.nbytes > 0) { unsigned int nbytes = walk.nbytes; @@ -83,7 +90,6 @@ static int chacha20_neon(struct skcipher_request *req) nbytes); err = skcipher_walk_done(&walk, walk.nbytes - nbytes); } - kernel_neon_end(); return err; }