From patchwork Tue Dec 26 10:29:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 122726 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp778923qgn; Tue, 26 Dec 2017 02:30:55 -0800 (PST) X-Google-Smtp-Source: ACJfBosu7rVIwsnASFYWTCgGiedcfAxziGDdGtQVkV7YP7xfxpCrkTzUWxD8Rgw0XktyPPoR/x0W X-Received: by 10.98.93.92 with SMTP id r89mr25162492pfb.1.1514284255833; Tue, 26 Dec 2017 02:30:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514284255; cv=none; d=google.com; s=arc-20160816; b=CGN4F/qxhaN65GO4+vxlHsI5NuIF7qZUtOvQsu2f+rMtBcG969I92ckEE80/GrKdD8 /GDNDbViKMoyJbgMwXZAXdp9LCUvbgt+KGOFo5M/K9ZOSjJMRN9hv9BShc0+0PN6l+pd HbJDTQYUuSu3iMFanTTdNj07+E68U/vUorweXaqKzxvG0r0fVL/DsfhgRDcK9RvHrhAA bn3ZiJs5oqYtm2RQaE2P1nZyI7mdpT2YjQgu/0YzaRO9EDRUmj60WUfzE21BE6NiQLxW nrSIH0+t119xp9TvMDPXZmR5WvkJxITnpO+mCMnuDGw3eQ1jHgpcta4hiwZf6NwW86z+ J3eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=NYW3GNv1Fb5c3EF6hTCqbc8ovx6cH6VeyEIJ7wuOJjw=; b=ywfeVv0c0OKD8sglBSY8sUwZS9xScTwp+DbeQabJXRIfJ2H3rz4Y3+BgHckPyDwPRl uZWYxO09XwQxO9ojLljDO6oofs6aS8PbVlvL/TiY3X6nSsM0ECmAk3Et58Ghnr6ksvHL H9GYfGti3b2leDhIOVFQboDN50nlk699mzd/ozDk3gLbrh2X9cTyFoYaVUGxb2Vx+UnF kCbBTE929NABkGdT/z8gQ+pbHE0gNWpBSE0Ed8GMHYIaTIf4MiliT+CJfl/c8gc6r0MA slsJ9Q50fMC95RLl5WlpBAq26Uo2dhFUiJPrag4MYxsf22X+Qli61A42ZdqKyNLIBzvb F4Lg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=KEIHobJX; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t10si22769770plh.272.2017.12.26.02.30.55; Tue, 26 Dec 2017 02:30:55 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=KEIHobJX; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751061AbdLZKax (ORCPT + 4 others); Tue, 26 Dec 2017 05:30:53 -0500 Received: from mail-wr0-f193.google.com ([209.85.128.193]:35605 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750951AbdLZKav (ORCPT ); Tue, 26 Dec 2017 05:30:51 -0500 Received: by mail-wr0-f193.google.com with SMTP id l19so21938529wrc.2 for ; Tue, 26 Dec 2017 02:30:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=siDqjuKDSU5MVSI0X6LMwRScVHyvLRBVTI5yU5S7QOs=; b=KEIHobJXG6FYvyR/9hLKknW0+Ad10cG9dg+LQvtDn/FrotN5pWwP96ACwF0O0Jpu+X KRRW8Mx+Xrf1Sm6Zw2j8F57zTPDgrrHVnOi4o9aPLMHVJFnMuL5fVl0g3wZ+tWkkksLf Gmqv3UI+tnaaEoFIEd8e42jvjZ4Re2NCpC9zw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=siDqjuKDSU5MVSI0X6LMwRScVHyvLRBVTI5yU5S7QOs=; b=JfW3/XmVd5UDrQLuSgOsnXlMdBOWoUTCih+IVtV/MUaxZoGpeMNGY4HzfgwASbUe3R uyxfix4t14jwlpju7SuIZarO6Q0OGC+gOpLj22+W8zQrHPt1+ezlwqzKfRNfU2fDFjgk UsWMZes8pf6uOIlUS89k7NIsMoNvV8Mj8Ob8NDh6oteK/wYQW3j9/BhkbtXR6m4KMBs1 RqNoU4JW8JW2lfpcwHrYq3uzohsbuReyfB4D9qCrcff+h+refMSTIhH2bL8BHnQFTQPZ QiZeK13fqOcgMNA3+crQuaH4enn47TUp0Sm+3txl2BMQ4cwv5siC35KCFaekRI57qBZ/ 24OQ== X-Gm-Message-State: AKGB3mIKKMEtX4c7ygOLGUEo+59zzGD1HjNg/GktkhEqpL+dBUnra1tz YrfPAcBHTUnWlXsJBHrE/qhjHA== X-Received: by 10.223.184.200 with SMTP id c8mr24658523wrg.268.1514284250087; Tue, 26 Dec 2017 02:30:50 -0800 (PST) Received: from localhost.localdomain ([160.171.216.245]) by smtp.gmail.com with ESMTPSA id l142sm13974036wmb.43.2017.12.26.02.30.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Dec 2017 02:30:49 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v4 04/20] crypto: arm64/aes-bs - move kernel mode neon en/disable into loop Date: Tue, 26 Dec 2017 10:29:24 +0000 Message-Id: <20171226102940.26908-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171226102940.26908-1-ard.biesheuvel@linaro.org> References: <20171226102940.26908-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org When kernel mode NEON was first introduced on arm64, the preserve and restore of the userland NEON state was completely unoptimized, and involved saving all registers on each call to kernel_neon_begin(), and restoring them on each call to kernel_neon_end(). For this reason, the NEON crypto code that was introduced at the time keeps the NEON enabled throughout the execution of the crypto API methods, which may include calls back into the crypto API that could result in memory allocation or other actions that we should avoid when running with preemption disabled. Since then, we have optimized the kernel mode NEON handling, which now restores lazily (upon return to userland), and so the preserve action is only costly the first time it is called after entering the kernel. So let's put the kernel_neon_begin() and kernel_neon_end() calls around the actual invocations of the NEON crypto code, and run the remainder of the code with kernel mode NEON disabled (and preemption enabled) Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-neonbs-glue.c | 36 +++++++++----------- 1 file changed, 17 insertions(+), 19 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c index 9d823c77ec84..e7a95a566462 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -99,9 +99,8 @@ static int __ecb_crypt(struct skcipher_request *req, struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -109,12 +108,13 @@ static int __ecb_crypt(struct skcipher_request *req, blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, ctx->rounds, blocks); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -158,19 +158,19 @@ static int cbc_encrypt(struct skcipher_request *req) struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; /* fall back to the non-bitsliced NEON implementation */ + kernel_neon_begin(); neon_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->enc, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -181,9 +181,8 @@ static int cbc_decrypt(struct skcipher_request *req) struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -191,13 +190,14 @@ static int cbc_decrypt(struct skcipher_request *req) blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -229,9 +229,8 @@ static int ctr_encrypt(struct skcipher_request *req) u8 buf[AES_BLOCK_SIZE]; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes > 0) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; u8 *final = (walk.total % AES_BLOCK_SIZE) ? buf : NULL; @@ -242,8 +241,10 @@ static int ctr_encrypt(struct skcipher_request *req) final = NULL; } + kernel_neon_begin(); aesbs_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, ctx->rounds, blocks, walk.iv, final); + kernel_neon_end(); if (final) { u8 *dst = walk.dst.virt.addr + blocks * AES_BLOCK_SIZE; @@ -258,8 +259,6 @@ static int ctr_encrypt(struct skcipher_request *req) err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); - return err; } @@ -304,12 +303,11 @@ static int __xts_crypt(struct skcipher_request *req, struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); kernel_neon_begin(); - - neon_aes_ecb_encrypt(walk.iv, walk.iv, ctx->twkey, - ctx->key.rounds, 1); + neon_aes_ecb_encrypt(walk.iv, walk.iv, ctx->twkey, ctx->key.rounds, 1); + kernel_neon_end(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -318,13 +316,13 @@ static int __xts_crypt(struct skcipher_request *req, blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); - return err; }