From patchwork Mon Dec 4 12:26:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120518 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp4365689qgn; Mon, 4 Dec 2017 04:27:25 -0800 (PST) X-Google-Smtp-Source: AGs4zMYWtktVQmJ0ks4trI47hjKi8mwDURklkRyJ/Umnqbs4BDzjlvZuNuPB6qyPV4UXpM0gS+F4 X-Received: by 10.84.202.194 with SMTP id q2mr10858186plh.220.1512390445577; Mon, 04 Dec 2017 04:27:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512390445; cv=none; d=google.com; s=arc-20160816; b=O3r871rDCj7A5R903M/oqGQLVM30F+umkVHLnJ8bOvNpIOD5j56Vg55f/QzPKHqNQB JAa5dtA6EM3bmxrEGUXZiPk6KmGbQrp1/cfqKGFLPBb/itsJJGlIhv/pSS5Ja9qhE7FD k5f9Jsvlu3KjWD7qYDD+iffP+j6AChHjkz33CzOOBRmEVSaUtup4n11rE1qJ816dY3Ow L4uTL26lzmWGDq1LpfwU+BqkLOSjK0LgWyAhOCyYUdLIg3aasslUs5arj7pg9tmx5m6I 44fuv2ZXSlFW++mYbpw2vBFOy3mk2AFKTueKP7wSGtQ56e396PKw5zn7nxrmy2h2dv09 ZELQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=ByEfG8S6mziH6dHyDov6YH26KAHAKkbH9eW3AIOrYSU=; b=nAiyJYpt4osbidZYspuKlAFd0qSt0GCxwSfKYD0x08meVo91An56NvdKgAdsooASGc op3moxb8UowEfAMdriHgr7FE8viZsqargCmW4UyH6YgxobyJ7jV41FgGQQp6Cs+UhDaK +OpexjIWbzI/HNYRVrgtdMOM7qWC2/7LY5qO8atQER9OVfY/MGc2aIfcIcO+/gTyl0vR R+K4ya+Zp9M02Uavi3/Hl93G4pMpn0jISTkJnSc3N8cxlCVScfJqil0EagQ5wi+QkzeP fmdDG7NyaLKVq076aVr+gBAxb8aEchwBNlRXU9S8U5als/Lntc0BJbttZ/1/mtWMJaKy 40VQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=OQrM+6Mo; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u144si9409634pgb.226.2017.12.04.04.27.25; Mon, 04 Dec 2017 04:27:25 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=OQrM+6Mo; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753174AbdLDM1X (ORCPT + 4 others); Mon, 4 Dec 2017 07:27:23 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:34855 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753187AbdLDM1U (ORCPT ); Mon, 4 Dec 2017 07:27:20 -0500 Received: by mail-wm0-f67.google.com with SMTP id f9so13986454wmh.0 for ; Mon, 04 Dec 2017 04:27:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=I6Gnvs3Ig+XMQwO7jGoODHjYQvJyRj6RsM7VfBN0Rns=; b=OQrM+6MoWQd1oJ4iyICVrf082S9bL5keCMp6OFqusSDnyLle/sJ/49lQR+hFUrj09I t/4KiGHMCbbBKNHhWrIq8z5zPLr9VsW0XDF03Ap8UkExZM9+FTAdxw9D6wI2z3aF+zf/ iWqmkDEQDasn52BHq9772/D7C3nPn50RzIdl4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=I6Gnvs3Ig+XMQwO7jGoODHjYQvJyRj6RsM7VfBN0Rns=; b=GUiqS0Gc0GhC9lfg3GUulrEi5uTnreIpYxN3TMoAmBhMGiLS9Gkxwsbm0VU+nsr9XI /LyU/K9Sdk2UZxmAAaFPHOKZ/vYt0XVrRHs5O4WB0lEuboXarIClUMDq+kD35Hav6ROW xt2ZAGA8hOO3LeLqSi1Ir7x/hSRoyUUcgFe+OOugHVpkgEWYjtvcoK0TgpLSsHwkCTDz D2rRrWLOimK8SFi1fJE6Z7AesUpzK5E8gRYoBAiZPgRtLpQjMsy2oy04ODCw7yI2qki1 9ZZPhIFWHlKuMp+ciAF8wJ/ld1+9zpzCoHE1BloHkNgv66kfEshvhgau/Un4GQOc4nbq RupQ== X-Gm-Message-State: AJaThX4ymqSQAyQOJn48dL2Gsc5Ip9UXmg5kNL6u4UhTwrexIS8UKjSC 4WnWCeIXSIck04FNYktna6sNAg== X-Received: by 10.28.131.73 with SMTP id f70mr7439697wmd.153.1512390439577; Mon, 04 Dec 2017 04:27:19 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id a8sm7665839wmh.41.2017.12.04.04.27.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 04 Dec 2017 04:27:18 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v2 08/19] crypto: arm64/aes-blk - add 4 way interleave to CBC encrypt path Date: Mon, 4 Dec 2017 12:26:34 +0000 Message-Id: <20171204122645.31535-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171204122645.31535-1-ard.biesheuvel@linaro.org> References: <20171204122645.31535-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org CBC encryption is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC encrypt which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 31 ++++++++++++++++---- 1 file changed, 25 insertions(+), 6 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 27a235b2ddee..e86535a1329d 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -94,17 +94,36 @@ AES_ENDPROC(aes_ecb_decrypt) */ AES_ENTRY(aes_cbc_encrypt) - ld1 {v0.16b}, [x5] /* get iv */ + ld1 {v4.16b}, [x5] /* get iv */ enc_prepare w3, x2, x6 -.Lcbcencloop: - ld1 {v1.16b}, [x1], #16 /* get next pt block */ - eor v0.16b, v0.16b, v1.16b /* ..and xor with iv */ +.Lcbcencloop4x: + subs w4, w4, #4 + bmi .Lcbcenc1x + ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ + eor v0.16b, v0.16b, v4.16b /* ..and xor with iv */ encrypt_block v0, w3, x2, x6, w7 - st1 {v0.16b}, [x0], #16 + eor v1.16b, v1.16b, v0.16b + encrypt_block v1, w3, x2, x6, w7 + eor v2.16b, v2.16b, v1.16b + encrypt_block v2, w3, x2, x6, w7 + eor v3.16b, v3.16b, v2.16b + encrypt_block v3, w3, x2, x6, w7 + st1 {v0.16b-v3.16b}, [x0], #64 + mov v4.16b, v3.16b + b .Lcbcencloop4x +.Lcbcenc1x: + adds w4, w4, #4 + beq .Lcbcencout +.Lcbcencloop: + ld1 {v0.16b}, [x1], #16 /* get next pt block */ + eor v4.16b, v4.16b, v0.16b /* ..and xor with iv */ + encrypt_block v4, w3, x2, x6, w7 + st1 {v4.16b}, [x0], #16 subs w4, w4, #1 bne .Lcbcencloop - st1 {v0.16b}, [x5] /* return iv */ +.Lcbcencout: + st1 {v4.16b}, [x5] /* return iv */ ret AES_ENDPROC(aes_cbc_encrypt)