From patchwork Sat Mar 10 15:21:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 131298 Delivered-To: patch@linaro.org Received: by 10.46.66.2 with SMTP id p2csp2244651lja; Sat, 10 Mar 2018 07:22:58 -0800 (PST) X-Google-Smtp-Source: AG47ELvw4JpeQwZ0/LHjWWakByFV2dy6EjAueweyrvEJipb5cxfC4hzm5awGyubt6IQB3/rOl7yr X-Received: by 2002:a17:902:b785:: with SMTP id e5-v6mr1703604pls.354.1520695378084; Sat, 10 Mar 2018 07:22:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520695378; cv=none; d=google.com; s=arc-20160816; b=e91LDzX5JwQI+NzX3cBHbc3ghP8Gng8E3WXljkb/Wa/8HdnO4mUEYsbFQj9cC1D4fv IA+d4ZVdc+CPDyE9EsMVpab6jpi8szklWjod9OYv2GdH8R7tOh5BLNJxquqHKnTultys rIl+OhWCahkgae3jwbfTaosyAfRoYZPYHzK6g8fA0FEcMRGIxAQvHK6oCMpe2Mt3KoJZ 0GL3LdWglc1sDwQEmivpT5uS4q/erUPW5OVSiAq3ApdWNzkeqr10btK+Q+vsy6SRgBYM GVa8O7xgl6TWeducPN9t5OOhpG8BdchrOCbq7vNiTrfcn57arck204GMA0xiG/PZX8t/ dhfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=SWaTAqImlUy+j9GIerQtkvTvfHo8IM7KHMAA2NnX1Ag=; b=Ud+IszENUiVt1mjRoG4IyJ2ihAGo57KzQjRIr3h+OL8t4sK9QzA5C9Jd+oLnBecLIS 7o9EwXzLUYuVy+B6zFug3aVLLv456AH1ny9FaKU5XeIt59NnysSJoKcRbBMPtkoJ1q/P BmcUxx352vZ1aVyreYL8NtyKD1TBI3iiw9LlAPVgW6VunrwQKAAE8MnP0RSPN4lA2sA8 IhkPcFuMPNpHhEriDjZn+F1HZ2ORX4JUS+VSJqggdGw2/nA6ex+q5hQuir90kWkKKNwk UWB38kLeMuWLK8cXh4qMxk+VZxQ4MLWMoqh2QblK0VEX3pr+HFY0PprPai7mKTCzZlbl hohA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GPqtIFW2; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q16si2824191pfg.221.2018.03.10.07.22.57; Sat, 10 Mar 2018 07:22:58 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GPqtIFW2; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932321AbeCJPW4 (ORCPT + 1 other); Sat, 10 Mar 2018 10:22:56 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:40210 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932308AbeCJPWy (ORCPT ); Sat, 10 Mar 2018 10:22:54 -0500 Received: by mail-wm0-f67.google.com with SMTP id t6so8750195wmt.5 for ; Sat, 10 Mar 2018 07:22:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SWaTAqImlUy+j9GIerQtkvTvfHo8IM7KHMAA2NnX1Ag=; b=GPqtIFW2E2k9LNStc3Z9Oro8B0dz4DGxsgTQLwqMAi4I34CzMjbs2U9ZToAW2lXiaP an07aC0SUfD7RTP5RmhTkjNLlX2MISFzXHf/pWtM6J3X3hQQh4qhKbLKnSd+pq7RJw8X JkhVAziEZUiliixX6YJKMmxe8o6z1u/hWQJ2Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SWaTAqImlUy+j9GIerQtkvTvfHo8IM7KHMAA2NnX1Ag=; b=EtbX9+G2LCEJMVN4kKWH0kwRLQuT0YbjPnRWMWMDjLx0s+KBVXNOme+ryDj2Rtd7N3 gqEKoPi/adrsE3eWBJ9VTFGywNKmc0i4oD+r6+FZe00ViG/xT0edhex5gdSlgwC0pv/O sKnDqtLW2JbZgt8kLSTBqmwbc75rFNEsT/BruOg+pFpIAWRYtquqTufqQagurIGMHKho RBy08mXKd3yMgYdnhVSZeJ1Pe6KOD3A2W/bS3W+HX2vvcT95HY1ic6gCegseOVNWYV0Q GIP914MOZAPWGguj4rburhg0loNi5UOXluHyHpXlI/4CBpxvbupB3rpl2HWzDRJBwL9M gnVQ== X-Gm-Message-State: AElRT7G1Bsf97AwwzIO0bvb5KCDkTMMo2kXvSt+Q5zs6LinNxWS7/8Ch Bb42aD5pse2pyTMUWydfH/03brCufZA= X-Received: by 10.28.69.197 with SMTP id l66mr1359264wmi.34.1520695373023; Sat, 10 Mar 2018 07:22:53 -0800 (PST) Received: from localhost.localdomain ([105.148.128.186]) by smtp.gmail.com with ESMTPSA id m9sm7027531wrf.13.2018.03.10.07.22.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 07:22:51 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v5 08/23] crypto: arm64/aes-blk - add 4 way interleave to CBC-MAC encrypt path Date: Sat, 10 Mar 2018 15:21:53 +0000 Message-Id: <20180310152208.10369-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180310152208.10369-1-ard.biesheuvel@linaro.org> References: <20180310152208.10369-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org CBC MAC is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC MAC which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 23 ++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) -- 2.15.1 diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index e86535a1329d..a68412e1e3a4 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -395,8 +395,28 @@ AES_ENDPROC(aes_xts_decrypt) AES_ENTRY(aes_mac_update) ld1 {v0.16b}, [x4] /* get dg */ enc_prepare w2, x1, x7 - cbnz w5, .Lmacenc + cbz w5, .Lmacloop4x + encrypt_block v0, w2, x1, x7, w8 + +.Lmacloop4x: + subs w3, w3, #4 + bmi .Lmac1x + ld1 {v1.16b-v4.16b}, [x0], #64 /* get next pt block */ + eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */ + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v2.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v3.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v4.16b + cmp w3, wzr + csinv x5, x6, xzr, eq + cbz w5, .Lmacout + encrypt_block v0, w2, x1, x7, w8 + b .Lmacloop4x +.Lmac1x: + add w3, w3, #4 .Lmacloop: cbz w3, .Lmacout ld1 {v1.16b}, [x0], #16 /* get next pt block */ @@ -406,7 +426,6 @@ AES_ENTRY(aes_mac_update) csinv x5, x6, xzr, eq cbz w5, .Lmacout -.Lmacenc: encrypt_block v0, w2, x1, x7, w8 b .Lmacloop