From patchwork Wed Aug 21 14:32:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171956 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055166ily; Wed, 21 Aug 2019 07:33:05 -0700 (PDT) X-Google-Smtp-Source: APXvYqxwwPpuH38uwa2k3rMQ8q+Z5wQ2NlzFCRL0R1WbRLJ9dLhqu1bnJyVf/zjD6nmOkOU06J91 X-Received: by 2002:aa7:8b51:: with SMTP id i17mr36351600pfd.33.1566397985112; Wed, 21 Aug 2019 07:33:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397985; cv=none; d=google.com; s=arc-20160816; b=U3xd28EQCrXtBHGvzKQbvFpe60Zbf2KD770GZBu5vgtSudmYJcIHvT3D/5pw80UY4Y 36r4AapsF/AbGFUiMsiCbL1el/CVng2jc1VkusLioQxRlEStjRyw03FTSZUg/4ImFGxL 5rBPk5DJfADnXfhQxNwi2NC70ZhIwykAt867l4WZN1cOspvi4SM01f1hGKxHVGKblDi+ Bz0dX0gGuhmLcUe1LSMcWCvpKiSHmcFPEnCXjRjUMFxNSnal0tgFyBNnFy7aCfASH6CS 87P/9extAzeltbZ1u46T2RKTxHPdhFa4OTsa/WhJKAPbAO2gkmvBEFhK6t8dbkYLp0fL Vfnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=Fx4v5+pxRWU+v+BvHaZkcgPS3e4c7qTcjz3iyucazFw=; b=oEZW/gG2ZBaO8DwHnLMonvEQmevUGKA3ldkSrBZI9S9o2Wi+HLgCbfSkuc6ggrDFu5 /Tv1tR8IOEPlCQQePGna46qT68l7pTj0EJfgvN0Bmac2qJhaIJ2S0kYe77bIRT6129wl Swn4oxua829AHlEY/i0TR2YEfxUyizZDm8fmuiesuogvA/auffRXv5QHGG/rlPq79/Cm sZe22EBxBGLokzyM6gENObnP0YskRZJnUZuIWa8ITMYUhqxPxKVLySZptOVCruADHH7P LeG1jJpBvFpfQfQ7PnMUH9rSp55lb5LOvlwCaZZu+xLHvhfRF8TjL2bHXu5botSGSiUy CCqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IvQZirvy; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.04; Wed, 21 Aug 2019 07:33:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IvQZirvy; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729168AbfHUOdE (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:04 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:43926 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728502AbfHUOdE (ORCPT ); Wed, 21 Aug 2019 10:33:04 -0400 Received: by mail-wr1-f68.google.com with SMTP id y8so2250988wrn.10 for ; Wed, 21 Aug 2019 07:33:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Fx4v5+pxRWU+v+BvHaZkcgPS3e4c7qTcjz3iyucazFw=; b=IvQZirvyJq+P+RO21JYYEshDe40ZRa0jFKH1cYxsPk7TjxNNCj7KKaBNe4aDRU1Cxf mQ+dceQbd3wVjhIg4672OGMrtz5Ylrk9qmjE8yl45zFDyly7En2pYw1juxGWNRImmijy iQhTGH4ohQuVsDANA+r/+FOhrXSs9l/46QPqPXo523RfEA3kWJocvQgkcLDdecAKtFFt 7732r1Kb/JkmNrvWbKNvbac1KGGUlCq0h7MOJoWU2J7c/bYb2r7MSIjNJzNu6b+DxtCq qMt8dEAzroTsMUOg5SFunX7+iTcsinCc9am7R5Lkas4DFrhRbfSCZqtpXxVynDFecTeg Hfsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Fx4v5+pxRWU+v+BvHaZkcgPS3e4c7qTcjz3iyucazFw=; b=CQ/1SDoZraz+VIup2ds/FTfy+VueDRHlNQuxnJg4jQ0g+sloRBPqz3BRH7YU6W+lR8 y9H8+UXxsJ7t3k1jKb3THqk3V9AcXXwdWOLGZ4Qbvm0E7Ie8gWJwRm8jWvIKq++oqDEV qUc+lli9kf6GIQq9QTbGsiEy8GLgU+vDQh7KJvkd8FjnPOg4vsF0bfdjL9nixxXw/TXn iU22ItT+CJ85yhlTW5EKOA9cLfGSc5c58vbOAaw5Qn60Cb8qhhevLQXqDxKjbrTkCrSW Zm5bFWHp0ut2QTxQUfSTKRApZe+RVaUtZczyqMl20QFNUDYpj7OqdNHfJLecZabU6Nlf EOBw== X-Gm-Message-State: APjAAAULHAKwaPPuCtZn49bk60VcHwyfssb3ae0+Xuhw7RFGWqEErWA5 CIufVbujI1OAAqpm8s38g7EosaVzbgyZlA== X-Received: by 2002:adf:ef48:: with SMTP id c8mr254746wrp.103.1566397981312; Wed, 21 Aug 2019 07:33:01 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.32.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:00 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 01/17] crypto: arm/aes - fix round key prototypes Date: Wed, 21 Aug 2019 17:32:37 +0300 Message-Id: <20190821143253.30209-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The AES round keys are arrays of u32s in native endianness now, so update the function prototypes accordingly. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-core.S | 18 ++++----- arch/arm/crypto/aes-ce-glue.c | 40 ++++++++++---------- 2 files changed, 29 insertions(+), 29 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/aes-ce-core.S b/arch/arm/crypto/aes-ce-core.S index 425000232d49..1e0d45183590 100644 --- a/arch/arm/crypto/aes-ce-core.S +++ b/arch/arm/crypto/aes-ce-core.S @@ -154,9 +154,9 @@ ENDPROC(aes_decrypt_3x) .endm /* - * aes_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, + * aes_ecb_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, * int blocks) - * aes_ecb_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, + * aes_ecb_decrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, * int blocks) */ ENTRY(ce_aes_ecb_encrypt) @@ -212,9 +212,9 @@ ENTRY(ce_aes_ecb_decrypt) ENDPROC(ce_aes_ecb_decrypt) /* - * aes_cbc_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, + * aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, * int blocks, u8 iv[]) - * aes_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, + * aes_cbc_decrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, * int blocks, u8 iv[]) */ ENTRY(ce_aes_cbc_encrypt) @@ -272,7 +272,7 @@ ENTRY(ce_aes_cbc_decrypt) ENDPROC(ce_aes_cbc_decrypt) /* - * aes_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, + * aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, * int blocks, u8 ctr[]) */ ENTRY(ce_aes_ctr_encrypt) @@ -349,10 +349,10 @@ ENTRY(ce_aes_ctr_encrypt) ENDPROC(ce_aes_ctr_encrypt) /* - * aes_xts_encrypt(u8 out[], u8 const in[], u8 const rk1[], int rounds, - * int blocks, u8 iv[], u8 const rk2[], int first) - * aes_xts_decrypt(u8 out[], u8 const in[], u8 const rk1[], int rounds, - * int blocks, u8 iv[], u8 const rk2[], int first) + * aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[], int rounds, + * int blocks, u8 iv[], u32 const rk2[], int first) + * aes_xts_decrypt(u8 out[], u8 const in[], u32 const rk1[], int rounds, + * int blocks, u8 iv[], u32 const rk2[], int first) */ .macro next_tweak, out, in, const, tmp diff --git a/arch/arm/crypto/aes-ce-glue.c b/arch/arm/crypto/aes-ce-glue.c index a7265d0a7063..75d2ff03a63e 100644 --- a/arch/arm/crypto/aes-ce-glue.c +++ b/arch/arm/crypto/aes-ce-glue.c @@ -25,25 +25,25 @@ MODULE_LICENSE("GPL v2"); asmlinkage u32 ce_aes_sub(u32 input); asmlinkage void ce_aes_invert(void *dst, void *src); -asmlinkage void ce_aes_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], +asmlinkage void ce_aes_ecb_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks); -asmlinkage void ce_aes_ecb_decrypt(u8 out[], u8 const in[], u8 const rk[], +asmlinkage void ce_aes_ecb_decrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks); -asmlinkage void ce_aes_cbc_encrypt(u8 out[], u8 const in[], u8 const rk[], +asmlinkage void ce_aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks, u8 iv[]); -asmlinkage void ce_aes_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[], +asmlinkage void ce_aes_cbc_decrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks, u8 iv[]); -asmlinkage void ce_aes_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[], +asmlinkage void ce_aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks, u8 ctr[]); -asmlinkage void ce_aes_xts_encrypt(u8 out[], u8 const in[], u8 const rk1[], +asmlinkage void ce_aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[], int rounds, int blocks, u8 iv[], - u8 const rk2[], int first); -asmlinkage void ce_aes_xts_decrypt(u8 out[], u8 const in[], u8 const rk1[], + u32 const rk2[], int first); +asmlinkage void ce_aes_xts_decrypt(u8 out[], u8 const in[], u32 const rk1[], int rounds, int blocks, u8 iv[], - u8 const rk2[], int first); + u32 const rk2[], int first); struct aes_block { u8 b[AES_BLOCK_SIZE]; @@ -182,7 +182,7 @@ static int ecb_encrypt(struct skcipher_request *req) kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { ce_aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - (u8 *)ctx->key_enc, num_rounds(ctx), blocks); + ctx->key_enc, num_rounds(ctx), blocks); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } kernel_neon_end(); @@ -202,7 +202,7 @@ static int ecb_decrypt(struct skcipher_request *req) kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { ce_aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - (u8 *)ctx->key_dec, num_rounds(ctx), blocks); + ctx->key_dec, num_rounds(ctx), blocks); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } kernel_neon_end(); @@ -222,7 +222,7 @@ static int cbc_encrypt(struct skcipher_request *req) kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { ce_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - (u8 *)ctx->key_enc, num_rounds(ctx), blocks, + ctx->key_enc, num_rounds(ctx), blocks, walk.iv); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } @@ -243,7 +243,7 @@ static int cbc_decrypt(struct skcipher_request *req) kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { ce_aes_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - (u8 *)ctx->key_dec, num_rounds(ctx), blocks, + ctx->key_dec, num_rounds(ctx), blocks, walk.iv); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } @@ -263,7 +263,7 @@ static int ctr_encrypt(struct skcipher_request *req) kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { ce_aes_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - (u8 *)ctx->key_enc, num_rounds(ctx), blocks, + ctx->key_enc, num_rounds(ctx), blocks, walk.iv); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } @@ -278,8 +278,8 @@ static int ctr_encrypt(struct skcipher_request *req) */ blocks = -1; - ce_aes_ctr_encrypt(tail, NULL, (u8 *)ctx->key_enc, - num_rounds(ctx), blocks, walk.iv); + ce_aes_ctr_encrypt(tail, NULL, ctx->key_enc, num_rounds(ctx), + blocks, walk.iv); crypto_xor_cpy(tdst, tsrc, tail, nbytes); err = skcipher_walk_done(&walk, 0); } @@ -324,8 +324,8 @@ static int xts_encrypt(struct skcipher_request *req) kernel_neon_begin(); for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) { ce_aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - (u8 *)ctx->key1.key_enc, rounds, blocks, - walk.iv, (u8 *)ctx->key2.key_enc, first); + ctx->key1.key_enc, rounds, blocks, walk.iv, + ctx->key2.key_enc, first); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } kernel_neon_end(); @@ -346,8 +346,8 @@ static int xts_decrypt(struct skcipher_request *req) kernel_neon_begin(); for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) { ce_aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - (u8 *)ctx->key1.key_dec, rounds, blocks, - walk.iv, (u8 *)ctx->key2.key_enc, first); + ctx->key1.key_dec, rounds, blocks, walk.iv, + ctx->key2.key_enc, first); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } kernel_neon_end(); From patchwork Wed Aug 21 14:32:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171957 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055217ily; Wed, 21 Aug 2019 07:33:07 -0700 (PDT) X-Google-Smtp-Source: APXvYqwnNkzTq+DFZ1Cvi4GjI4oJdchjfn8gtU/0eoWM9EX7GDPThDdQTc+euPhG+hfrbMLqwJ8p X-Received: by 2002:a63:d002:: with SMTP id z2mr30046615pgf.364.1566397987591; Wed, 21 Aug 2019 07:33:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397987; cv=none; d=google.com; s=arc-20160816; b=Yc0W6BEQMu2d6XqViJX1t2eIc6KpUNCmWn9tnluR+8Q/zhgayokpvuz7yr1TOcktw2 TXVyEbsKiKulJg6spKY12q50RL3B8lq872X4oUZjUR3YRAq4JAJYxqMYHrBR6rx1GoeO GZRvZrSg74A34SYwa/zTWWOhdpHZdSD8Wol3jSDYi0MlFn9Lp0hMBkxjASyPgB7wlXaw qJgcFnqBj8yvT1aCff9GL8tWO+OBzxM2tPCHwndeq11OoOqMWTeASfqciVc77k6cnPGd r344I/AvMr8CphyQSn8Z7xdYJ3wxZAMNP+yYTKGnSNN5UvUSxj9bIBV/6FRx78+PFN7C 8X1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=iouKzcU6vzz3J8Cw/jp3uuc+QHFS3Cf59HMMjSOOWbs=; b=riDtCAkE8xY3pX1khPGavOselX/V6pPYxOZh3Ue8a/iHHooDId87Zevb+9sVNhDQ3r eoRAvalXhNyzl7AS41j6exxS8hArQF40ZraGLkW1jDd2Jo9k0XpC3Jyp3emasejneEA3 sxQeGmZy6sikkJr16hcMZP8bzWzAFj/VzqjvSs4jCOwS+JYJE34jhSELyAzKUYhj0epm 4jg61RVmJdq5Hwk3MP/RU/nKg+msNqmpoFoHRRaM4M2ctYQfv6ZL/iai1kThuhYgKt01 ehEvaw341jbOJUQzM3djksXkocHJlOzXiKJT5vWg2kcaS3I5i2sew/VbPiE4tOdgxDxW yggQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GaAOKLbS; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.07; Wed, 21 Aug 2019 07:33:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GaAOKLbS; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728502AbfHUOdH (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:07 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:40882 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728980AbfHUOdG (ORCPT ); Wed, 21 Aug 2019 10:33:06 -0400 Received: by mail-wr1-f66.google.com with SMTP id c3so2257291wrd.7 for ; Wed, 21 Aug 2019 07:33:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iouKzcU6vzz3J8Cw/jp3uuc+QHFS3Cf59HMMjSOOWbs=; b=GaAOKLbSlqipMH664RibEBtkwaszXHKiJcIXNSxoGR2IKNaSLiCl+ZQRMV+pUHajNM AJJGpuZNdY4qtB5wtVvjrkXhsOBUVERYoOMnQ3A3WghV0M3auzU0mkoI3O1tUNxvmfH8 RRpLRD3JDtzP0ra4wG/pa9IxxmpCrcBH7aEDeHXWx8rI1XWAwnysIif0iMEpc1dXK6fK bAh0oAQ5UFL0AZWkdymwGxM627A5XO03Ne1NYsPIfilIWQmZT8GXOo8TVwHm36AJ9Lft iWDLLQZ4Eohc+acV+HCGe7G89gyGs+GB2o7ahnEDKSFDAb6Gk0cMENLZeZehtxIEii6o xFQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iouKzcU6vzz3J8Cw/jp3uuc+QHFS3Cf59HMMjSOOWbs=; b=NkPievD5lD46PHBK3rNzBBmIoVbKKhPdySYHleAzCliIsFIUKQxn7cKhHkHjQ/T5nl BG8UdFMGjLBAk8R8tIizTkBoyl3Qi5q+2MnFWh5DoYtxRC4xfOX1OaDRiwU7PobGl5+6 QyrYnVWA4eEGR0JitzlZP2dh3ehHWbdDNFVtWWgXWcplRTZSnFRYZGEh5pEuFtfpxTkZ tzHzHaVlMyvug6JmfUtJ5H2aVDtypwgYunMJWSre2CXX/7ByF2rWscIkxdwtECOysK+3 rzyW9o3K0IgBTQ3EmcV3f9QWhdwHuKOeN5uJ8KCGoAuG/n7vSaRzneVXQdVCSekdPuK1 domQ== X-Gm-Message-State: APjAAAUR1l2X3l2mckwZjKp8/VwSqUGalneYlJ5A1L8BDOx6k0WAEl9+ ji3EBTbMfB4QVQQ2EaAhs/uP7go8N4sVaQ== X-Received: by 2002:adf:fe85:: with SMTP id l5mr38995702wrr.5.1566397982854; Wed, 21 Aug 2019 07:33:02 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:02 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 02/17] crypto: arm/aes-ce - yield the SIMD unit between scatterwalk steps Date: Wed, 21 Aug 2019 17:32:38 +0300 Message-Id: <20190821143253.30209-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Reduce the scope of the kernel_neon_begin/end regions so that the SIMD unit is released (and thus preemption re-enabled) if the crypto operation cannot be completed in a single scatterwalk step. This avoids scheduling blackouts due to preemption being enabled for unbounded periods, resulting in a more responsive system. After this change, we can also permit the cipher_walk infrastructure to sleep, so set the 'atomic' parameter to skcipher_walk_virt() to false as well. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-glue.c | 47 ++++++++++---------- arch/arm/crypto/aes-neonbs-glue.c | 22 ++++----- 2 files changed, 34 insertions(+), 35 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/aes-ce-glue.c b/arch/arm/crypto/aes-ce-glue.c index 75d2ff03a63e..486e862ae34a 100644 --- a/arch/arm/crypto/aes-ce-glue.c +++ b/arch/arm/crypto/aes-ce-glue.c @@ -177,15 +177,15 @@ static int ecb_encrypt(struct skcipher_request *req) unsigned int blocks; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { + kernel_neon_begin(); ce_aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key_enc, num_rounds(ctx), blocks); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -197,15 +197,15 @@ static int ecb_decrypt(struct skcipher_request *req) unsigned int blocks; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { + kernel_neon_begin(); ce_aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key_dec, num_rounds(ctx), blocks); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -217,16 +217,16 @@ static int cbc_encrypt(struct skcipher_request *req) unsigned int blocks; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { + kernel_neon_begin(); ce_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key_enc, num_rounds(ctx), blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -238,16 +238,16 @@ static int cbc_decrypt(struct skcipher_request *req) unsigned int blocks; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { + kernel_neon_begin(); ce_aes_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key_dec, num_rounds(ctx), blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -258,13 +258,14 @@ static int ctr_encrypt(struct skcipher_request *req) struct skcipher_walk walk; int err, blocks; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { + kernel_neon_begin(); ce_aes_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key_enc, num_rounds(ctx), blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } if (walk.nbytes) { @@ -278,13 +279,13 @@ static int ctr_encrypt(struct skcipher_request *req) */ blocks = -1; + kernel_neon_begin(); ce_aes_ctr_encrypt(tail, NULL, ctx->key_enc, num_rounds(ctx), blocks, walk.iv); + kernel_neon_end(); crypto_xor_cpy(tdst, tsrc, tail, nbytes); err = skcipher_walk_done(&walk, 0); } - kernel_neon_end(); - return err; } @@ -319,17 +320,16 @@ static int xts_encrypt(struct skcipher_request *req) struct skcipher_walk walk; unsigned int blocks; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) { + kernel_neon_begin(); ce_aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key1.key_enc, rounds, blocks, walk.iv, ctx->key2.key_enc, first); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); - return err; } @@ -341,17 +341,16 @@ static int xts_decrypt(struct skcipher_request *req) struct skcipher_walk walk; unsigned int blocks; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) { + kernel_neon_begin(); ce_aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key1.key_dec, rounds, blocks, walk.iv, ctx->key2.key_enc, first); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); - return err; } diff --git a/arch/arm/crypto/aes-neonbs-glue.c b/arch/arm/crypto/aes-neonbs-glue.c index 45cd9818791e..9000d0796d5e 100644 --- a/arch/arm/crypto/aes-neonbs-glue.c +++ b/arch/arm/crypto/aes-neonbs-glue.c @@ -90,9 +90,8 @@ static int __ecb_crypt(struct skcipher_request *req, struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -100,12 +99,13 @@ static int __ecb_crypt(struct skcipher_request *req, blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, ctx->rounds, blocks); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -159,9 +159,8 @@ static int cbc_decrypt(struct skcipher_request *req) struct skcipher_walk walk; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -169,13 +168,14 @@ static int cbc_decrypt(struct skcipher_request *req) blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -223,9 +223,8 @@ static int ctr_encrypt(struct skcipher_request *req) u8 buf[AES_BLOCK_SIZE]; int err; - err = skcipher_walk_virt(&walk, req, true); + err = skcipher_walk_virt(&walk, req, false); - kernel_neon_begin(); while (walk.nbytes > 0) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; u8 *final = (walk.total % AES_BLOCK_SIZE) ? buf : NULL; @@ -236,8 +235,10 @@ static int ctr_encrypt(struct skcipher_request *req) final = NULL; } + kernel_neon_begin(); aesbs_ctr_encrypt(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, ctx->rounds, blocks, walk.iv, final); + kernel_neon_end(); if (final) { u8 *dst = walk.dst.virt.addr + blocks * AES_BLOCK_SIZE; @@ -252,7 +253,6 @@ static int ctr_encrypt(struct skcipher_request *req) err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } @@ -329,7 +329,6 @@ static int __xts_crypt(struct skcipher_request *req, crypto_cipher_encrypt_one(ctx->tweak_tfm, walk.iv, walk.iv); - kernel_neon_begin(); while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -337,12 +336,13 @@ static int __xts_crypt(struct skcipher_request *req, blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + kernel_neon_begin(); fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, ctx->key.rounds, blocks, walk.iv); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - kernel_neon_end(); return err; } From patchwork Wed Aug 21 14:32:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171959 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055268ily; Wed, 21 Aug 2019 07:33:10 -0700 (PDT) X-Google-Smtp-Source: APXvYqx9gULaUIwLxHCA6AVrauJ5wJrIKGea5HWXc+bKOjR0EThaZ4ecIQ0kWz9qyhfCLemXoxvw X-Received: by 2002:a62:80cb:: with SMTP id j194mr37030274pfd.183.1566397990074; Wed, 21 Aug 2019 07:33:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397990; cv=none; d=google.com; s=arc-20160816; b=KOt3nzNbxnDsMIqLHzjPchcxIFGVOJ3QVUYKJfPu9UxK+r6x7nKS3436441LqlsIMk RgQvaJ+02WPb3Fv4/dTH+EyOBUrZ60XK6+v9S7TDOeEzVSUMghSGSgWGt/v8qFMIlV/i bZV7cvg9e6rMX3CEBipSj6dsWk4jbdSPzUgLfjjM/f4fB00O/ziP+Rzdn4WbtWxIqyaJ Znmp/NZLXKIzwYHqlQ02/4NWjHk9kyGs3rxuA814LbXQ1C89/k9Hjp9Mt1tOhpP/2Z8B NZBZK3gWoAVfC2/OoCbnqd6x+huImoQ3E/HsxDwW7Bp9UEdBbAVMfDpOaMvGGF4mfNK4 3OyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=u8c7yjt2nGuO09VOjn5Nf28VuKqSVZEi1e4/jRpqUrQ=; b=P7gMOct4C3rsMd0NSSu2X8B1jsBAivBuCbYV5ZHsmcF6gH2ZUiR853hepo3Nf1kWPb EY6SIsl4PcYpbvGrZ2AMn1xUAMitg6IAPAXDSmLLcbHH75HOQwMvQX5gzp7+Dd62LV7Y UFYT5hBQBM/rMjlcf04nz23yod9JP2jdNOz/uaLpc+G3NYjpTxbfjcBOqbx+UC5L31bv qx/ftexXsCLCKklPNIQnTMmxT0eNRnJq15F1hkl1fONVO2JteHSz8ZVn03roWPdtNNqI 3qHaVcmmfsItyMY9o1D0jRjnwagtYHAA2j6E7JlB+ehGuAppD9TBZw9cjGFf1Hcebv9z P+6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Tt3jNYo1; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.09; Wed, 21 Aug 2019 07:33:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Tt3jNYo1; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729293AbfHUOdJ (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:09 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:46571 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727949AbfHUOdJ (ORCPT ); Wed, 21 Aug 2019 10:33:09 -0400 Received: by mail-wr1-f67.google.com with SMTP id z1so2244949wru.13 for ; Wed, 21 Aug 2019 07:33:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=u8c7yjt2nGuO09VOjn5Nf28VuKqSVZEi1e4/jRpqUrQ=; b=Tt3jNYo1W2sgzxG/ODJSO8GuD8jRA0Nyf0XqdWAdJNt4d2CkxA/PW0iwIqFWjS8lGv BF+uZMI3F56mfP0Q7Ti+XtYnK9SS2z1LcYgm5v4RQjY9gPV2Mqq5BujRdaKcTQ2R6EgK GxBr6yKJPOs5PdSYxv+gUxdVYMPnX38ZI6Z5s1xCBO+5HifW04OWHLqHWgXv3wONdyYR CgDZWNDuzhmQ+koyoqRz3knf42Ln06quEx7yhiKuGjMar6Rz+bDElCxHJHeVlG8aGu8H FvgmRle0cAwFL7EUA7D4ilHj3g3i5+15TkQIZm05kThqXc1gAmXqNqWUh+sj+/UgcPnj rrxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=u8c7yjt2nGuO09VOjn5Nf28VuKqSVZEi1e4/jRpqUrQ=; b=AUMghgSqFJP5BXhDcg2qmuZvNQ2BJMlt7O0dtTem+HZdQ1JAa40WJbNAm5ibuoNBe3 Q/ot5Wz2W4ri4O7UerUj20vpRd6FChGVhN3yhQn9D9vrjc5ntNlG8LgJzn6tG9KOwvcy JZKbKEL4Tz6w5+iJ27VfQxQc5t3XbcO+WqyYbz3wIsptnds+Ug2MaTUiFvaI+OBjMb2M xj30O/0dcMR6Ewf1n/6FdzFgAsqc73x6+Wx7xaSGpj94d9vydFS6KAOD3iW6+MBunCoI lMCLbKbNzS1y+/Btnnk6HlxXmLfwF5NYn07+6J82yk9DYLMsiWHugxP5VXR6s6Gc7KhV mHkg== X-Gm-Message-State: APjAAAXujtfF0vHTIZbgbNU+hV7xROxRm3Pw+beQnTb44c0xb1KuPO3q +vmxmi7aJ+RgzAPyAUrorPdUdKoBR0RTJg== X-Received: by 2002:adf:ce04:: with SMTP id p4mr41771147wrn.227.1566397984578; Wed, 21 Aug 2019 07:33:04 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:03 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 03/17] crypto: arm/aes-ce - switch to 4x interleave Date: Wed, 21 Aug 2019 17:32:39 +0300 Message-Id: <20190821143253.30209-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When the ARM AES instruction based crypto driver was introduced, there were no known implementations that could benefit from a 4-way interleave, and so a 3-way interleave was used instead. Since we have sufficient space in the SIMD register file, let's switch to a 4-way interleave to align with the 64-bit driver, and to ensure that we can reach optimum performance when running under emulation on high end 64-bit cores. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-core.S | 263 +++++++++++--------- 1 file changed, 144 insertions(+), 119 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/aes-ce-core.S b/arch/arm/crypto/aes-ce-core.S index 1e0d45183590..a3ca4ac2d7bb 100644 --- a/arch/arm/crypto/aes-ce-core.S +++ b/arch/arm/crypto/aes-ce-core.S @@ -44,46 +44,56 @@ veor q0, q0, \key3 .endm - .macro enc_dround_3x, key1, key2 + .macro enc_dround_4x, key1, key2 enc_round q0, \key1 enc_round q1, \key1 enc_round q2, \key1 + enc_round q3, \key1 enc_round q0, \key2 enc_round q1, \key2 enc_round q2, \key2 + enc_round q3, \key2 .endm - .macro dec_dround_3x, key1, key2 + .macro dec_dround_4x, key1, key2 dec_round q0, \key1 dec_round q1, \key1 dec_round q2, \key1 + dec_round q3, \key1 dec_round q0, \key2 dec_round q1, \key2 dec_round q2, \key2 + dec_round q3, \key2 .endm - .macro enc_fround_3x, key1, key2, key3 + .macro enc_fround_4x, key1, key2, key3 enc_round q0, \key1 enc_round q1, \key1 enc_round q2, \key1 + enc_round q3, \key1 aese.8 q0, \key2 aese.8 q1, \key2 aese.8 q2, \key2 + aese.8 q3, \key2 veor q0, q0, \key3 veor q1, q1, \key3 veor q2, q2, \key3 + veor q3, q3, \key3 .endm - .macro dec_fround_3x, key1, key2, key3 + .macro dec_fround_4x, key1, key2, key3 dec_round q0, \key1 dec_round q1, \key1 dec_round q2, \key1 + dec_round q3, \key1 aesd.8 q0, \key2 aesd.8 q1, \key2 aesd.8 q2, \key2 + aesd.8 q3, \key2 veor q0, q0, \key3 veor q1, q1, \key3 veor q2, q2, \key3 + veor q3, q3, \key3 .endm .macro do_block, dround, fround @@ -114,8 +124,9 @@ * transforms. These should preserve all registers except q0 - q2 and ip * Arguments: * q0 : first in/output block - * q1 : second in/output block (_3x version only) - * q2 : third in/output block (_3x version only) + * q1 : second in/output block (_4x version only) + * q2 : third in/output block (_4x version only) + * q3 : fourth in/output block (_4x version only) * q8 : first round key * q9 : secound round key * q14 : final round key @@ -136,16 +147,16 @@ aes_decrypt: ENDPROC(aes_decrypt) .align 6 -aes_encrypt_3x: +aes_encrypt_4x: add ip, r2, #32 @ 3rd round key - do_block enc_dround_3x, enc_fround_3x -ENDPROC(aes_encrypt_3x) + do_block enc_dround_4x, enc_fround_4x +ENDPROC(aes_encrypt_4x) .align 6 -aes_decrypt_3x: +aes_decrypt_4x: add ip, r2, #32 @ 3rd round key - do_block dec_dround_3x, dec_fround_3x -ENDPROC(aes_decrypt_3x) + do_block dec_dround_4x, dec_fround_4x +ENDPROC(aes_decrypt_4x) .macro prepare_key, rk, rounds add ip, \rk, \rounds, lsl #4 @@ -163,17 +174,17 @@ ENTRY(ce_aes_ecb_encrypt) push {r4, lr} ldr r4, [sp, #8] prepare_key r2, r3 -.Lecbencloop3x: - subs r4, r4, #3 +.Lecbencloop4x: + subs r4, r4, #4 bmi .Lecbenc1x vld1.8 {q0-q1}, [r1]! - vld1.8 {q2}, [r1]! - bl aes_encrypt_3x + vld1.8 {q2-q3}, [r1]! + bl aes_encrypt_4x vst1.8 {q0-q1}, [r0]! - vst1.8 {q2}, [r0]! - b .Lecbencloop3x + vst1.8 {q2-q3}, [r0]! + b .Lecbencloop4x .Lecbenc1x: - adds r4, r4, #3 + adds r4, r4, #4 beq .Lecbencout .Lecbencloop: vld1.8 {q0}, [r1]! @@ -189,17 +200,17 @@ ENTRY(ce_aes_ecb_decrypt) push {r4, lr} ldr r4, [sp, #8] prepare_key r2, r3 -.Lecbdecloop3x: - subs r4, r4, #3 +.Lecbdecloop4x: + subs r4, r4, #4 bmi .Lecbdec1x vld1.8 {q0-q1}, [r1]! - vld1.8 {q2}, [r1]! - bl aes_decrypt_3x + vld1.8 {q2-q3}, [r1]! + bl aes_decrypt_4x vst1.8 {q0-q1}, [r0]! - vst1.8 {q2}, [r0]! - b .Lecbdecloop3x + vst1.8 {q2-q3}, [r0]! + b .Lecbdecloop4x .Lecbdec1x: - adds r4, r4, #3 + adds r4, r4, #4 beq .Lecbdecout .Lecbdecloop: vld1.8 {q0}, [r1]! @@ -236,38 +247,40 @@ ENDPROC(ce_aes_cbc_encrypt) ENTRY(ce_aes_cbc_decrypt) push {r4-r6, lr} ldrd r4, r5, [sp, #16] - vld1.8 {q6}, [r5] @ keep iv in q6 + vld1.8 {q15}, [r5] @ keep iv in q15 prepare_key r2, r3 -.Lcbcdecloop3x: - subs r4, r4, #3 +.Lcbcdecloop4x: + subs r4, r4, #4 bmi .Lcbcdec1x vld1.8 {q0-q1}, [r1]! - vld1.8 {q2}, [r1]! - vmov q3, q0 - vmov q4, q1 - vmov q5, q2 - bl aes_decrypt_3x - veor q0, q0, q6 - veor q1, q1, q3 - veor q2, q2, q4 - vmov q6, q5 + vld1.8 {q2-q3}, [r1]! + vmov q4, q0 + vmov q5, q1 + vmov q6, q2 + vmov q7, q3 + bl aes_decrypt_4x + veor q0, q0, q15 + veor q1, q1, q4 + veor q2, q2, q5 + veor q3, q3, q6 + vmov q15, q7 vst1.8 {q0-q1}, [r0]! - vst1.8 {q2}, [r0]! - b .Lcbcdecloop3x + vst1.8 {q2-q3}, [r0]! + b .Lcbcdecloop4x .Lcbcdec1x: - adds r4, r4, #3 + adds r4, r4, #4 beq .Lcbcdecout - vmov q15, q14 @ preserve last round key + vmov q6, q14 @ preserve last round key .Lcbcdecloop: vld1.8 {q0}, [r1]! @ get next ct block veor q14, q15, q6 @ combine prev ct with last key - vmov q6, q0 + vmov q15, q0 bl aes_decrypt vst1.8 {q0}, [r0]! subs r4, r4, #1 bne .Lcbcdecloop .Lcbcdecout: - vst1.8 {q6}, [r5] @ keep iv in q6 + vst1.8 {q15}, [r5] @ keep iv in q15 pop {r4-r6, pc} ENDPROC(ce_aes_cbc_decrypt) @@ -278,46 +291,52 @@ ENDPROC(ce_aes_cbc_decrypt) ENTRY(ce_aes_ctr_encrypt) push {r4-r6, lr} ldrd r4, r5, [sp, #16] - vld1.8 {q6}, [r5] @ load ctr + vld1.8 {q7}, [r5] @ load ctr prepare_key r2, r3 - vmov r6, s27 @ keep swabbed ctr in r6 + vmov r6, s31 @ keep swabbed ctr in r6 rev r6, r6 cmn r6, r4 @ 32 bit overflow? bcs .Lctrloop -.Lctrloop3x: - subs r4, r4, #3 +.Lctrloop4x: + subs r4, r4, #4 bmi .Lctr1x add r6, r6, #1 - vmov q0, q6 - vmov q1, q6 + vmov q0, q7 + vmov q1, q7 rev ip, r6 add r6, r6, #1 - vmov q2, q6 + vmov q2, q7 vmov s7, ip rev ip, r6 add r6, r6, #1 + vmov q3, q7 vmov s11, ip - vld1.8 {q3-q4}, [r1]! - vld1.8 {q5}, [r1]! - bl aes_encrypt_3x - veor q0, q0, q3 - veor q1, q1, q4 - veor q2, q2, q5 + rev ip, r6 + add r6, r6, #1 + vmov s15, ip + vld1.8 {q4-q5}, [r1]! + vld1.8 {q6}, [r1]! + vld1.8 {q15}, [r1]! + bl aes_encrypt_4x + veor q0, q0, q4 + veor q1, q1, q5 + veor q2, q2, q6 + veor q3, q3, q15 rev ip, r6 vst1.8 {q0-q1}, [r0]! - vst1.8 {q2}, [r0]! - vmov s27, ip - b .Lctrloop3x + vst1.8 {q2-q3}, [r0]! + vmov s31, ip + b .Lctrloop4x .Lctr1x: - adds r4, r4, #3 + adds r4, r4, #4 beq .Lctrout .Lctrloop: - vmov q0, q6 + vmov q0, q7 bl aes_encrypt adds r6, r6, #1 @ increment BE ctr rev ip, r6 - vmov s27, ip + vmov s31, ip bcs .Lctrcarry .Lctrcarrydone: @@ -329,7 +348,7 @@ ENTRY(ce_aes_ctr_encrypt) bne .Lctrloop .Lctrout: - vst1.8 {q6}, [r5] @ return next CTR value + vst1.8 {q7}, [r5] @ return next CTR value pop {r4-r6, pc} .Lctrtailblock: @@ -337,7 +356,7 @@ ENTRY(ce_aes_ctr_encrypt) b .Lctrout .Lctrcarry: - .irp sreg, s26, s25, s24 + .irp sreg, s30, s29, s28 vmov ip, \sreg @ load next word of ctr rev ip, ip @ ... to handle the carry adds ip, ip, #1 @@ -368,8 +387,8 @@ ENDPROC(ce_aes_ctr_encrypt) .quad 1, 0x87 ce_aes_xts_init: - vldr d14, .Lxts_mul_x - vldr d15, .Lxts_mul_x + 8 + vldr d30, .Lxts_mul_x + vldr d31, .Lxts_mul_x + 8 ldrd r4, r5, [sp, #16] @ load args ldr r6, [sp, #28] @@ -390,48 +409,51 @@ ENTRY(ce_aes_xts_encrypt) bl ce_aes_xts_init @ run shared prologue prepare_key r2, r3 - vmov q3, q0 + vmov q4, q0 teq r6, #0 @ start of a block? - bne .Lxtsenc3x + bne .Lxtsenc4x -.Lxtsencloop3x: - next_tweak q3, q3, q7, q6 -.Lxtsenc3x: - subs r4, r4, #3 +.Lxtsencloop4x: + next_tweak q4, q4, q15, q10 +.Lxtsenc4x: + subs r4, r4, #4 bmi .Lxtsenc1x - vld1.8 {q0-q1}, [r1]! @ get 3 pt blocks - vld1.8 {q2}, [r1]! - next_tweak q4, q3, q7, q6 - veor q0, q0, q3 - next_tweak q5, q4, q7, q6 - veor q1, q1, q4 - veor q2, q2, q5 - bl aes_encrypt_3x - veor q0, q0, q3 - veor q1, q1, q4 - veor q2, q2, q5 - vst1.8 {q0-q1}, [r0]! @ write 3 ct blocks - vst1.8 {q2}, [r0]! - vmov q3, q5 + vld1.8 {q0-q1}, [r1]! @ get 4 pt blocks + vld1.8 {q2-q3}, [r1]! + next_tweak q5, q4, q15, q10 + veor q0, q0, q4 + next_tweak q6, q5, q15, q10 + veor q1, q1, q5 + next_tweak q7, q6, q15, q10 + veor q2, q2, q6 + veor q3, q3, q7 + bl aes_encrypt_4x + veor q0, q0, q4 + veor q1, q1, q5 + veor q2, q2, q6 + veor q3, q3, q7 + vst1.8 {q0-q1}, [r0]! @ write 4 ct blocks + vst1.8 {q2-q3}, [r0]! + vmov q4, q7 teq r4, #0 beq .Lxtsencout - b .Lxtsencloop3x + b .Lxtsencloop4x .Lxtsenc1x: - adds r4, r4, #3 + adds r4, r4, #4 beq .Lxtsencout .Lxtsencloop: vld1.8 {q0}, [r1]! - veor q0, q0, q3 + veor q0, q0, q4 bl aes_encrypt - veor q0, q0, q3 + veor q0, q0, q4 vst1.8 {q0}, [r0]! subs r4, r4, #1 beq .Lxtsencout - next_tweak q3, q3, q7, q6 + next_tweak q4, q4, q15, q6 b .Lxtsencloop .Lxtsencout: - vst1.8 {q3}, [r5] + vst1.8 {q4}, [r5] pop {r4-r6, pc} ENDPROC(ce_aes_xts_encrypt) @@ -441,49 +463,52 @@ ENTRY(ce_aes_xts_decrypt) bl ce_aes_xts_init @ run shared prologue prepare_key r2, r3 - vmov q3, q0 + vmov q4, q0 teq r6, #0 @ start of a block? - bne .Lxtsdec3x + bne .Lxtsdec4x -.Lxtsdecloop3x: - next_tweak q3, q3, q7, q6 -.Lxtsdec3x: - subs r4, r4, #3 +.Lxtsdecloop4x: + next_tweak q4, q4, q15, q10 +.Lxtsdec4x: + subs r4, r4, #4 bmi .Lxtsdec1x - vld1.8 {q0-q1}, [r1]! @ get 3 ct blocks - vld1.8 {q2}, [r1]! - next_tweak q4, q3, q7, q6 - veor q0, q0, q3 - next_tweak q5, q4, q7, q6 - veor q1, q1, q4 - veor q2, q2, q5 - bl aes_decrypt_3x - veor q0, q0, q3 - veor q1, q1, q4 - veor q2, q2, q5 - vst1.8 {q0-q1}, [r0]! @ write 3 pt blocks - vst1.8 {q2}, [r0]! - vmov q3, q5 + vld1.8 {q0-q1}, [r1]! @ get 4 ct blocks + vld1.8 {q2-q3}, [r1]! + next_tweak q5, q4, q15, q10 + veor q0, q0, q4 + next_tweak q6, q5, q15, q10 + veor q1, q1, q5 + next_tweak q7, q6, q15, q10 + veor q2, q2, q6 + veor q3, q3, q7 + bl aes_decrypt_4x + veor q0, q0, q4 + veor q1, q1, q5 + veor q2, q2, q6 + veor q3, q3, q7 + vst1.8 {q0-q1}, [r0]! @ write 4 pt blocks + vst1.8 {q2-q3}, [r0]! + vmov q4, q7 teq r4, #0 beq .Lxtsdecout - b .Lxtsdecloop3x + b .Lxtsdecloop4x .Lxtsdec1x: - adds r4, r4, #3 + adds r4, r4, #4 beq .Lxtsdecout .Lxtsdecloop: vld1.8 {q0}, [r1]! - veor q0, q0, q3 + veor q0, q0, q4 add ip, r2, #32 @ 3rd round key bl aes_decrypt - veor q0, q0, q3 + veor q0, q0, q4 vst1.8 {q0}, [r0]! subs r4, r4, #1 beq .Lxtsdecout - next_tweak q3, q3, q7, q6 + next_tweak q4, q4, q15, q6 b .Lxtsdecloop .Lxtsdecout: - vst1.8 {q3}, [r5] + vst1.8 {q4}, [r5] pop {r4-r6, pc} ENDPROC(ce_aes_xts_decrypt) From patchwork Wed Aug 21 14:32:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171958 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055247ily; Wed, 21 Aug 2019 07:33:09 -0700 (PDT) X-Google-Smtp-Source: APXvYqzhbvTCfraHgGzOmeVpIoOBZkr4JahhvbpGkFk9EkTKVitaqzFEjy77RT9HH808ogATTDja X-Received: by 2002:a63:f926:: with SMTP id h38mr29259536pgi.80.1566397989017; Wed, 21 Aug 2019 07:33:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397989; cv=none; d=google.com; s=arc-20160816; b=l+orTMyJsmht0h1QwraKN/4Z3oby9ndcA2r/xPCGij/nlaPDPuRJsKPApljPwHB/s6 f0PJV39Fn1lC5o0WR0EsSQ8DeAb2OBakeZ/Jh40TB2tiDd3atN/45ZRNz7ndFviZOjoC vSdJsnyptJ40uazqvTU8+11lu/+nCjHYYnxDs3euX1julW/Z+9DgeEoNiG1zOgygR6/O 5pdBVUjBQJwvobRCjk19w+VWQa8BYmwEDQRWo5CZXpiR74Ea16Qm1X7CQ32cwbtoL4z2 XJv9Lqcs1OrKU02xqeUUPPDGjkuBM2/Xgy+0wtIHxzxFBhGLxnFAFE22TkYakl17/LUu EuZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=aI11eOfvUHWceFRi+8iF4u67GyVnjyFicxzOIJeHmqs=; b=cn3RzrgD2BxtiemTgiP4x78Q9iElHzMJlMY5WLMN1xijw+VDcGNPeiNRU9fQJ1v62f 60fepTSjBDOJBF8AE17ZopIcidXgMsJ3dyneAHkf2sVhlu3K/D94kkHw+eTYTWV5nV23 gkTV8mSotZsXNn74e1wgBuFUIJlE8t6S6XETClKr3Q7xAvYKA4exGu7HPzo/0P7dz519 hdIAmmm0pk7l6Y06tpS0gpn6Od1SekNWMLuEV7gTpEf/r+pBnRQ5EVMrXmqj9nNA9Jg6 KS3+IuZXa6sIfsqZioa1OwI/B1nCYpoTrdITQ5N+Oau0aSSshvLg49ewZ6i1kPFEzjE3 oqSw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ddEZtJwc; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.08; Wed, 21 Aug 2019 07:33:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ddEZtJwc; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729271AbfHUOdI (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:08 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:43936 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728980AbfHUOdI (ORCPT ); Wed, 21 Aug 2019 10:33:08 -0400 Received: by mail-wr1-f68.google.com with SMTP id y8so2251248wrn.10 for ; Wed, 21 Aug 2019 07:33:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aI11eOfvUHWceFRi+8iF4u67GyVnjyFicxzOIJeHmqs=; b=ddEZtJwcsiZncsOlztMjOo2Rph0rOlKCbabKpMj8fxxRifPIZGzpXtAq9jKbNvCmQv lQ+T6aHvHHpbOU4Q/qfnKuQeu9uHBaUm2nVZUdfB9qcd8HqdPpGNQxayvjQtnoZdi4gw AWVHiGxGwGH37pDaL44Rldv5YGrpb+9y4Ecu6fpYcOTKvUDcuOQTxLU1UghkDh0G/B4Z 1v5orHWU2Oy1NBuTXu6XUDFsabccXaKux4OJdB67pSrXEDWzUUV8qFE5nSOeuemlfVzq f79bk06E6gL458kS6RXktd+rV6C+1iqqpK0NxMQS25jqT38WWhYb8pdfllSXUtQhdTzb zeOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aI11eOfvUHWceFRi+8iF4u67GyVnjyFicxzOIJeHmqs=; b=rmhIcXWMwqT5715U9qtJdPr50sl1O8m6joBJcqpqrRGEFsSsFuMjtu99SheS0AzzRY Yz4hQTG6fka+D7zkMoaw6KhYxZ6yt6x685e3B9SthPnbkDg6xYx48O20/rVCt7s/5Yki g8f9t+2/RDHSsmyDROdzns1vfQZwiwMLLX6IeCQj83ZYR30P9m0IQr17eKawDUQI4JoU 32v0kI6T0MGmWLkRczGvGJpJEre/wOJkBh9EruaGODawMEUEkuOsKWIODx+GsNbDckU5 wy3tU5Q02orYjhu7jPRs8niUf7dXDKJ0klj5nWB6mqECpAHVNF7omHClUOtj/+SVTZ1x NvGA== X-Gm-Message-State: APjAAAVabuRtx+04IJRGMHgdBB37RX/9ezJTeq/pzJ5yKYXOIV0mOFYk dpyhncKkRUwok/ZtsOg9frFwxy+02W6fyg== X-Received: by 2002:adf:d4c6:: with SMTP id w6mr42017542wrk.98.1566397986351; Wed, 21 Aug 2019 07:33:06 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:05 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 04/17] crypto: arm/aes-ce - replace tweak mask literal with composition Date: Wed, 21 Aug 2019 17:32:40 +0300 Message-Id: <20190821143253.30209-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Replace the vector load from memory sequence with a simple instruction sequence to compose the tweak vector directly. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-core.S | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/aes-ce-core.S b/arch/arm/crypto/aes-ce-core.S index a3ca4ac2d7bb..bb6ec1844370 100644 --- a/arch/arm/crypto/aes-ce-core.S +++ b/arch/arm/crypto/aes-ce-core.S @@ -382,13 +382,10 @@ ENDPROC(ce_aes_ctr_encrypt) veor \out, \out, \tmp .endm - .align 3 -.Lxts_mul_x: - .quad 1, 0x87 - ce_aes_xts_init: - vldr d30, .Lxts_mul_x - vldr d31, .Lxts_mul_x + 8 + vmov.i32 d30, #0x87 @ compose tweak mask vector + vmovl.u32 q15, d30 + vshr.u64 d30, d31, #7 ldrd r4, r5, [sp, #16] @ load args ldr r6, [sp, #28] From patchwork Wed Aug 21 14:32:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171960 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055292ily; Wed, 21 Aug 2019 07:33:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqwrjic2jxQ6y9cybp//Dj3AVmZ7uYTNWhpBksU+yl23S7SggZSCfD//yPZCvqqczTFRgLtP X-Received: by 2002:a63:9e54:: with SMTP id r20mr30285314pgo.64.1566397991122; Wed, 21 Aug 2019 07:33:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397991; cv=none; d=google.com; s=arc-20160816; b=UMekxEkoLRHrTsZRwPrFbNTzJu/AeUyrKPpqTjWttKDvcb/N+pfvmP3jifxhxxEL0w JBPyHEaVwp/4cIhd0s4ENE8OljU2CESeUzknHIiVa/CyZPyH1ajryYfSoq7e7PXvIr8i vW2gvAZruIZCxfQFIyl7nUIPicdqSqPKSm6jKg2bALgR579eTXBeN56iyFE3kkhF0a4M Z21M30e/zUbuRUsXQ0JnyinF8tUjFZRrCWxRiXluJd6qr+XwqEfc3xTRoI/JGZDMtz7r 5k9WMWK4LWZFX8BskUhdZC+OhtodCm1sYVzLbFyBchaNmwqouyQj8w0VDACbLy5IiFOr HmKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=vb3yeqqCuXjboCdKiVDdUqWWVOWb7cOxAkDE/WzQmrQ=; b=cEf28LY2bpjbVsC5LybsnRoBuFbJcnUd0q8YcIrNYEMxcguYUpZwDaPg1IVL/P7Ppk 0PhVMEeRJjdaPKPSQ8ru13m0CqYoVopXyyiGXXw54G8/k7vLsmRUaMMYfiIkCbXuoJH+ QaAA4bsNl/NOnkNEswNt+BVcADqIUCTTL1+o1X8d/jM16KJc6msZfUNq3iyjVubCw/lz ZPVfHlhBA0LNCwTvMOX5eRr903kswCee7mp9cHr93SwlhbrCegizQMvsIvkz09H2bFui rC+1o9NTLKnRGDW3JIOi9eT5nzqvMNyAG/XTgPrG6Kl9hZvW+kaxAlub6g/JY0ItwYkD vmWQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=W+2rpNaU; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.10; Wed, 21 Aug 2019 07:33:11 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=W+2rpNaU; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727949AbfHUOdK (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:10 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:35656 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728980AbfHUOdK (ORCPT ); Wed, 21 Aug 2019 10:33:10 -0400 Received: by mail-wm1-f66.google.com with SMTP id l2so2394221wmg.0 for ; Wed, 21 Aug 2019 07:33:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vb3yeqqCuXjboCdKiVDdUqWWVOWb7cOxAkDE/WzQmrQ=; b=W+2rpNaUkXMImvHppqPtNDzaji/3lCx3sOC5o5KgOYCyHJxvSkVigtjsqzcEM86oOW Wo8T3gLwXICLt5yJKQJnRs+fYCUf/aXFSqOHxoRgQaY0EpX8FJG2N+dlKw8EAwq3jOfR 7OsLvKl51bgd3PlXkIOaWz0IM4dfrHpYKErZg/Bi7SS8Lcdkz9NpHt4q5EGUtH5KwUrD HzVm5HyHlimIP2D9PACbCijc8bM6/Y0Nh1Tv9/EbQeCQJOfUQn7jAP0ZPloaY3IHTMuT jOhIClaDtJLmytlqEhcIOkChA0Hc+ONOy25zCsrDGUMDiuFDo8vyuihSa6vrP2wqR/gl dQzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vb3yeqqCuXjboCdKiVDdUqWWVOWb7cOxAkDE/WzQmrQ=; b=O649VN9BW6sJhJgCUMVou7H/CBz7iKUIXYgcgiu0BEpGJ2xgiVE2fIZGkuBrarElck CmX8QET0KGt7eAhaPEjd6xA5HeFClB5ZfjOiFnSA/pNwMdcfGlJpbNToaVDVebfi9a3P +MIVW3VqFEc9kHEHXV8bb0FKSH63jEdMWXw/uiriqbMYyAoqxr4oCnc7ACra0vCirPl2 q8FMopOCXoVsoOr9K0iCj72bBmkuMOnjyrQHduuh13a6cWhdNMgYSXW9h2CyJ4Q1Vvgr YStHOp9jeOJAve5M5qbH9mtblYDNUnzmyOSo2F7NQNGR5sl8UEKV+E2GWZ2/DGX7eenW oAmQ== X-Gm-Message-State: APjAAAX1+6ipSnqLM9NvtFlsFayM43i44pUbgGi0TJHwVXVHkxmyM0PF ued0J6q+IkUltrT7IO69QtyYJ2VlH+Y9RQ== X-Received: by 2002:a1c:d185:: with SMTP id i127mr365113wmg.63.1566397987951; Wed, 21 Aug 2019 07:33:07 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:07 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 05/17] crypto: arm/aes-neonbs - replace tweak mask literal with composition Date: Wed, 21 Aug 2019 17:32:41 +0300 Message-Id: <20190821143253.30209-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Replace the vector load from memory sequence with a simple instruction sequence to compose the tweak vector directly. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-neonbs-core.S | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/aes-neonbs-core.S b/arch/arm/crypto/aes-neonbs-core.S index d3eab76b6e1b..bb75918e4984 100644 --- a/arch/arm/crypto/aes-neonbs-core.S +++ b/arch/arm/crypto/aes-neonbs-core.S @@ -887,10 +887,6 @@ ENDPROC(aesbs_ctr_encrypt) veor \out, \out, \tmp .endm - .align 4 -.Lxts_mul_x: - .quad 1, 0x87 - /* * aesbs_xts_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, * int blocks, u8 iv[]) @@ -899,7 +895,9 @@ ENDPROC(aesbs_ctr_encrypt) */ __xts_prepare8: vld1.8 {q14}, [r7] // load iv - __ldr q15, .Lxts_mul_x // load tweak mask + vmov.i32 d30, #0x87 // compose tweak mask vector + vmovl.u32 q15, d30 + vshr.u64 d30, d31, #7 vmov q12, q14 __adr ip, 0f From patchwork Wed Aug 21 14:32:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171961 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055350ily; Wed, 21 Aug 2019 07:33:13 -0700 (PDT) X-Google-Smtp-Source: APXvYqydwUi7vsa7o/P/+srgqg1sJiWEU3cVOtGWUmtwaKhzQuyDOd/xvGtRmsVYqyCPNKsGm1+5 X-Received: by 2002:a65:64c5:: with SMTP id t5mr30321473pgv.168.1566397993147; Wed, 21 Aug 2019 07:33:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397993; cv=none; d=google.com; s=arc-20160816; b=xW08sspWdFD8h3wK06aeqWa/sMQzvXjSH4cqSCpJG4EdJzoU3r+G0dwspnxMRY5uZC nUShZapD+JOOPBGB8BGYHUdnDuk7gylPOmUAaSeASWbdzFzUYPQawzSoY3DOK8QLpl+k JY5YUUQ96Wb1XWj5IfK8O2aYjXgatJB5BUaWKxKYHBMyMwLe7pIYw0Z/VwLsE7851bKW bE2pZ+VXy0ghc3vpmAtdeEpM2Aw8CJQh0aPckMhepqLj0SPkrXw9xQSJDmIokKbQGds6 +pKs/JjUXOl7s6QC83W4HH+C9NYq0CtEuyejP6zCD4w3ckkLLHAdeF/sR4iHQRYllJBN 8cKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=u8SkUbJP7rMNf0OKuHQI1V2U/tlAU0T6PVWrsyU6ges=; b=Z4x3VxKWfDxyuzavMmUSb7pRoVoNEZou+ZX2yiNz3tT45kJU902+FBQkjYo/+EjRrK YsRZEqfXBimftGs6MY3Fs2NtD/dPgD+MkZ7oI5by9QZCWhrVnkP9oJrrtYjFYWVW/+3K Pqs7wDj5T/l7SVh5bff9ckMtNDGWbYN3rKVVUUFtS9KKmpHjH+pLPjh/mZGXPOnPDDU5 ckBJZHnDydU5pqN3WW7el459R1SVXXv5vAg5F+R/t04R8LF+LhVezocESUGla6DuJnB7 XqMJR4vV+eg+5RAzE9Y8h+QwGF74faMyvPNT2MdozwaxwJwmglec2G8qkLfO7TG+rGVP zojA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=agOKtmBy; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.12; Wed, 21 Aug 2019 07:33:13 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=agOKtmBy; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728980AbfHUOdM (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:12 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:36526 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728822AbfHUOdM (ORCPT ); Wed, 21 Aug 2019 10:33:12 -0400 Received: by mail-wr1-f67.google.com with SMTP id r3so2277266wrt.3 for ; Wed, 21 Aug 2019 07:33:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=u8SkUbJP7rMNf0OKuHQI1V2U/tlAU0T6PVWrsyU6ges=; b=agOKtmByQmfCXzPEkiehriCi58nAeqL7qeZj/0YiY7LK3xoamzSEPbd2SSrZxSRulu kAVHl25gcca2E3u0rJV0B7ghzDbobQEHdnpPaGnpGry9EdnXOQIyg0pem8VMSOPbTD02 f2vgDT9T3PwWXUsGeRS5rrnZiZX61txuVBm4Oe7w307HfZ8W5fVxCioisG7qIVrIc3i3 vYjat4r5r6ARM7af82GCZFdNpE1Sdyt9XRTY3ISQ3NtKOmYVf5inRglLFloGUr6KjZL4 MH82DfHcvQfMUD074HqlPLOmawL9+vR4EZqsAD7yjz2FgPMKSSMvsxvvI9SCVq9XUBm+ mrDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=u8SkUbJP7rMNf0OKuHQI1V2U/tlAU0T6PVWrsyU6ges=; b=Qy4a6LM49EILb5P1AeA9nj7H2BctgpA0yRt9fP3IiyN8UnzmtakuDWb8g2BV/g8Vuy ilG6maF9VYp2nZdYrP/WEzB0MqOrpe6VZ9zXfodcxJGKPo0y1UnVtYJqVA83pPBjBCRw E8DkkSu6b36W9D7Y8jszOF8YB5wYpBbbGy4w0hwIstFNwRLB5KyI3Ka98yFjR06UP+B3 rtVOcxoG5Wm3NeIhW/JVZUZPTGqSVqMoqJ/HmZUNuA1KBiEYKFUJbvg+Bd0j8wgT4lBA c/vaug1rpufyMFJswoQcoZgDk1zURPBruMIpWcB4u7NhLeeGdthH/ZC8JA6P7B4XfuBZ 8y6g== X-Gm-Message-State: APjAAAVdkbuaQ+qaQtlPoCL4pOfZtyGCi3N4urv4H12FbDXxCIykJgAk c1oRps+81Zul06cgSul7hsTLZSRYmvL7XQ== X-Received: by 2002:adf:ef48:: with SMTP id c8mr255514wrp.103.1566397989579; Wed, 21 Aug 2019 07:33:09 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:08 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 06/17] crypto: arm64/aes-neonbs - replace tweak mask literal with composition Date: Wed, 21 Aug 2019 17:32:42 +0300 Message-Id: <20190821143253.30209-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Replace the vector load from memory sequence with a simple instruction sequence to compose the tweak vector directly. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-neonbs-core.S | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) -- 2.17.1 diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S index cf10ff8878a3..65982039fa36 100644 --- a/arch/arm64/crypto/aes-neonbs-core.S +++ b/arch/arm64/crypto/aes-neonbs-core.S @@ -730,11 +730,6 @@ ENDPROC(aesbs_cbc_decrypt) eor \out\().16b, \out\().16b, \tmp\().16b .endm - .align 4 -.Lxts_mul_x: -CPU_LE( .quad 1, 0x87 ) -CPU_BE( .quad 0x87, 1 ) - /* * aesbs_xts_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, * int blocks, u8 iv[]) @@ -806,7 +801,9 @@ ENDPROC(__xts_crypt8) mov x23, x4 mov x24, x5 -0: ldr q30, .Lxts_mul_x +0: movi v30.2s, #0x1 + movi v25.2s, #0x87 + uzp1 v30.4s, v30.4s, v25.4s ld1 {v25.16b}, [x24] 99: adr x7, \do8 From patchwork Wed Aug 21 14:32:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171963 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055426ily; Wed, 21 Aug 2019 07:33:16 -0700 (PDT) X-Google-Smtp-Source: APXvYqyrYp8DTCbBas30SrzuO5OGCfd1MP9qv+XwJIyTTnKUWTDqgooF8UBarzMp/MT5pssU/+Zs X-Received: by 2002:a63:1d2:: with SMTP id 201mr30254331pgb.307.1566397996689; Wed, 21 Aug 2019 07:33:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397996; cv=none; d=google.com; s=arc-20160816; b=QNEffWizNukBai/5LxH6ycAwB666Qn8aGw4OPB7+g1hvoeg8iNJ8sGlbBCDx6d5PYM 7yuYM6kb+ej9Vsq2jRRq3O5RRtA2R0T7nY465vgUyrwonPbgu7JLq1njmMHyWwPedRtt FWUsGIRhiJKwOSg07BWep/2QTDdSkj4OHrLxIv8TITmqFF/zcaK7JhgD0Wva/ryF/Es5 7Sg6Lqyd6zNe2asDfYxXMPhUPclicxftjybeJrCXf0zRQNlU2eesa/fCCMO9WoWrmRQU BquqktYciERqXYErRy4iFsgAs/9+aD2ukLpMmbqwKiMNZgdtCk6XpLclVKV4eCEqWcm+ g6FQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=LWz788qVRg8VvOU54DLOD0vDhiGV6LnBrRgMkVFGK/8=; b=nvrJMSQF4IFvP/95cXEpTUuukOH6yhWpJuHhei4Su39IyKnZLeKB24eAiFGlWvIL1T 4ymMOIjXP363x5429grsFLTXOQdNaFTH02x9ioPAlxsNiI1YNidijPmfTlBleuJuuFwW MzUWJZg4s/6PrH74dkKMMPwZsZJX605xZW8lsXwObti1B2yBwvMWN3+VcT2IJ4NMgn4D EoX2qCTISllRHB1Naqnx3SsTy8ddaidwk9YCTmql0G62QJ+oPx4ZIRwWG22mqJA8MpVk /r+yD0YqH8nQMzaeEbwTIpc9faA+TuK4ET4REtj7inYoJNcnjLtPwjt4kAuTlTGHX8F6 /BHg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kh0lP5zW; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.16; Wed, 21 Aug 2019 07:33:16 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kh0lP5zW; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728822AbfHUOdP (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:15 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:33529 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728763AbfHUOdP (ORCPT ); Wed, 21 Aug 2019 10:33:15 -0400 Received: by mail-wm1-f68.google.com with SMTP id p77so4916062wme.0 for ; Wed, 21 Aug 2019 07:33:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LWz788qVRg8VvOU54DLOD0vDhiGV6LnBrRgMkVFGK/8=; b=kh0lP5zWQCmIej/pzRW7hJEdUZP13UY2aQP5NxCMeJcCIz3aryKYV+Kuimjonw6ytk 0/mOIRXkileb2okTU3kRfbQQGphAeLvbF8djfn+IIfpneCRXjaBYa0MQSFPEXZ19WyM5 S1fUeJAGJht4SYzXBaXxMHL/Si/1+PuN8TYGXSOdq+DDpWq6YGSor6Qf20AU4ZR7kFxv KDIDuMTDINYG7GUmG9gvslINTNaQSCW6UFqHoQWuskWuUol0ZP+roC81Qcyxa/tSqCcz 1wi7N8yT9mVMr0TTTr2rCf93YJXtAR8ET1UvWRU4/sx4HD91kp5U41Cew/Y+sV4e2t7T dosw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LWz788qVRg8VvOU54DLOD0vDhiGV6LnBrRgMkVFGK/8=; b=lmpku8CQ812SRylPIUZoHcKE1x5qn8MgCy3j6FjFteOL+Gz8qWT/yVHgr5wl+k5VKf w1jDwh4Ohfqwmfqn3/5tOq5BJ12B5OuUoxpP7Z6hp8qNn82oEY6BF31vKgqMT6gnyyIw 9cyYxidQ6/M9n+WeZJz5/c2UjWJTHnYuA72Xfb7soGUlnZ3KkgsWY3Sa3/RRGOOweQJ2 KjysWuy0hFVbLw2OTyPucnyeqhnDbhqO4WCZnDGyKStzDpO3OkEEJEsnNCggu63SCjw2 PNs66phlnJQsa1ZfLA/4J51sTdkM8lQK2udQm+DpM0eO0k7tuOfBiG+SWtZuLsN1fFtP /M4Q== X-Gm-Message-State: APjAAAVt/Yz7UVGBiZ6WhU5l4K2d2ps3ChSwza3pG/DVrSqnaZDkCmRX swJzz14AVWVoXJ6iQzJmdzjDKfT2WHIJyQ== X-Received: by 2002:a1c:6145:: with SMTP id v66mr428480wmb.42.1566397991198; Wed, 21 Aug 2019 07:33:11 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:10 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 07/17] crypto: arm64/aes-neon - limit exposed routines if faster driver is enabled Date: Wed, 21 Aug 2019 17:32:43 +0300 Message-Id: <20190821143253.30209-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The pure NEON AES implementation predates the bit-slicing one, and is generally slower, unless the algorithm in question can only execute sequentially. So advertising the skciphers that the bit-slicing driver implements as well serves no real purpose, and we can just disable them. Note that the bit-slicing driver also has a link time dependency on the pure NEON driver, for CBC encryption and for XTS tweak calculation, so we still need both drivers on systems that do not implement the Crypto Extensions. At the same time, expose those modaliases for the AES instruction based driver. This is necessary since otherwise, we may end up loading the wrong driver when any of the skciphers are instantiated before the CPU capability based module loading has completed. Finally, add the missing modalias for cts(cbc(aes)) so requests for this algorithm will autoload the correct module. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-glue.c | 112 +++++++++++--------- 1 file changed, 59 insertions(+), 53 deletions(-) -- 2.17.1 diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index ca0c84d56cba..4154bb93a85b 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -54,15 +54,18 @@ MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions"); #define aes_xts_decrypt neon_aes_xts_decrypt #define aes_mac_update neon_aes_mac_update MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 NEON"); +#endif +#if defined(USE_V8_CRYPTO_EXTENSIONS) || !defined(CONFIG_CRYPTO_AES_ARM64_BS) MODULE_ALIAS_CRYPTO("ecb(aes)"); MODULE_ALIAS_CRYPTO("cbc(aes)"); -MODULE_ALIAS_CRYPTO("essiv(cbc(aes),sha256)"); MODULE_ALIAS_CRYPTO("ctr(aes)"); MODULE_ALIAS_CRYPTO("xts(aes)"); +#endif +MODULE_ALIAS_CRYPTO("cts(cbc(aes))"); +MODULE_ALIAS_CRYPTO("essiv(cbc(aes),sha256)"); MODULE_ALIAS_CRYPTO("cmac(aes)"); MODULE_ALIAS_CRYPTO("xcbc(aes)"); MODULE_ALIAS_CRYPTO("cbcmac(aes)"); -#endif MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); @@ -144,8 +147,8 @@ static int skcipher_aes_setkey(struct crypto_skcipher *tfm, const u8 *in_key, return ret; } -static int xts_set_key(struct crypto_skcipher *tfm, const u8 *in_key, - unsigned int key_len) +static int __maybe_unused xts_set_key(struct crypto_skcipher *tfm, + const u8 *in_key, unsigned int key_len) { struct crypto_aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); int ret; @@ -165,8 +168,9 @@ static int xts_set_key(struct crypto_skcipher *tfm, const u8 *in_key, return -EINVAL; } -static int essiv_cbc_set_key(struct crypto_skcipher *tfm, const u8 *in_key, - unsigned int key_len) +static int __maybe_unused essiv_cbc_set_key(struct crypto_skcipher *tfm, + const u8 *in_key, + unsigned int key_len) { struct crypto_aes_essiv_cbc_ctx *ctx = crypto_skcipher_ctx(tfm); SHASH_DESC_ON_STACK(desc, ctx->hash); @@ -190,7 +194,7 @@ static int essiv_cbc_set_key(struct crypto_skcipher *tfm, const u8 *in_key, return -EINVAL; } -static int ecb_encrypt(struct skcipher_request *req) +static int __maybe_unused ecb_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); @@ -210,7 +214,7 @@ static int ecb_encrypt(struct skcipher_request *req) return err; } -static int ecb_decrypt(struct skcipher_request *req) +static int __maybe_unused ecb_decrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); @@ -248,7 +252,7 @@ static int cbc_encrypt_walk(struct skcipher_request *req, return err; } -static int cbc_encrypt(struct skcipher_request *req) +static int __maybe_unused cbc_encrypt(struct skcipher_request *req) { struct skcipher_walk walk; int err; @@ -277,7 +281,7 @@ static int cbc_decrypt_walk(struct skcipher_request *req, return err; } -static int cbc_decrypt(struct skcipher_request *req) +static int __maybe_unused cbc_decrypt(struct skcipher_request *req) { struct skcipher_walk walk; int err; @@ -404,7 +408,7 @@ static int cts_cbc_decrypt(struct skcipher_request *req) return skcipher_walk_done(&walk, 0); } -static int essiv_cbc_init_tfm(struct crypto_skcipher *tfm) +static int __maybe_unused essiv_cbc_init_tfm(struct crypto_skcipher *tfm) { struct crypto_aes_essiv_cbc_ctx *ctx = crypto_skcipher_ctx(tfm); @@ -415,14 +419,14 @@ static int essiv_cbc_init_tfm(struct crypto_skcipher *tfm) return 0; } -static void essiv_cbc_exit_tfm(struct crypto_skcipher *tfm) +static void __maybe_unused essiv_cbc_exit_tfm(struct crypto_skcipher *tfm) { struct crypto_aes_essiv_cbc_ctx *ctx = crypto_skcipher_ctx(tfm); crypto_free_shash(ctx->hash); } -static int essiv_cbc_encrypt(struct skcipher_request *req) +static int __maybe_unused essiv_cbc_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_essiv_cbc_ctx *ctx = crypto_skcipher_ctx(tfm); @@ -444,7 +448,7 @@ static int essiv_cbc_encrypt(struct skcipher_request *req) return err ?: cbc_encrypt_walk(req, &walk); } -static int essiv_cbc_decrypt(struct skcipher_request *req) +static int __maybe_unused essiv_cbc_decrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_essiv_cbc_ctx *ctx = crypto_skcipher_ctx(tfm); @@ -520,7 +524,7 @@ static void ctr_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst) local_irq_restore(flags); } -static int ctr_encrypt_sync(struct skcipher_request *req) +static int __maybe_unused ctr_encrypt_sync(struct skcipher_request *req) { if (!crypto_simd_usable()) return crypto_ctr_encrypt_walk(req, ctr_encrypt_one); @@ -528,7 +532,7 @@ static int ctr_encrypt_sync(struct skcipher_request *req) return ctr_encrypt(req); } -static int xts_encrypt(struct skcipher_request *req) +static int __maybe_unused xts_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); @@ -550,7 +554,7 @@ static int xts_encrypt(struct skcipher_request *req) return err; } -static int xts_decrypt(struct skcipher_request *req) +static int __maybe_unused xts_decrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); @@ -573,6 +577,7 @@ static int xts_decrypt(struct skcipher_request *req) } static struct skcipher_alg aes_algs[] = { { +#if defined(USE_V8_CRYPTO_EXTENSIONS) || !defined(CONFIG_CRYPTO_AES_ARM64_BS) .base = { .cra_name = "__ecb(aes)", .cra_driver_name = "__ecb-aes-" MODE, @@ -603,42 +608,6 @@ static struct skcipher_alg aes_algs[] = { { .setkey = skcipher_aes_setkey, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, -}, { - .base = { - .cra_name = "__cts(cbc(aes))", - .cra_driver_name = "__cts-cbc-aes-" MODE, - .cra_priority = PRIO, - .cra_flags = CRYPTO_ALG_INTERNAL, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto_aes_ctx), - .cra_module = THIS_MODULE, - }, - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .ivsize = AES_BLOCK_SIZE, - .walksize = 2 * AES_BLOCK_SIZE, - .setkey = skcipher_aes_setkey, - .encrypt = cts_cbc_encrypt, - .decrypt = cts_cbc_decrypt, - .init = cts_cbc_init_tfm, -}, { - .base = { - .cra_name = "__essiv(cbc(aes),sha256)", - .cra_driver_name = "__essiv-cbc-aes-sha256-" MODE, - .cra_priority = PRIO + 1, - .cra_flags = CRYPTO_ALG_INTERNAL, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto_aes_essiv_cbc_ctx), - .cra_module = THIS_MODULE, - }, - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .ivsize = AES_BLOCK_SIZE, - .setkey = essiv_cbc_set_key, - .encrypt = essiv_cbc_encrypt, - .decrypt = essiv_cbc_decrypt, - .init = essiv_cbc_init_tfm, - .exit = essiv_cbc_exit_tfm, }, { .base = { .cra_name = "__ctr(aes)", @@ -688,6 +657,43 @@ static struct skcipher_alg aes_algs[] = { { .setkey = xts_set_key, .encrypt = xts_encrypt, .decrypt = xts_decrypt, +}, { +#endif + .base = { + .cra_name = "__cts(cbc(aes))", + .cra_driver_name = "__cts-cbc-aes-" MODE, + .cra_priority = PRIO, + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct crypto_aes_ctx), + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .walksize = 2 * AES_BLOCK_SIZE, + .setkey = skcipher_aes_setkey, + .encrypt = cts_cbc_encrypt, + .decrypt = cts_cbc_decrypt, + .init = cts_cbc_init_tfm, +}, { + .base = { + .cra_name = "__essiv(cbc(aes),sha256)", + .cra_driver_name = "__essiv-cbc-aes-sha256-" MODE, + .cra_priority = PRIO + 1, + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct crypto_aes_essiv_cbc_ctx), + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .setkey = essiv_cbc_set_key, + .encrypt = essiv_cbc_encrypt, + .decrypt = essiv_cbc_decrypt, + .init = essiv_cbc_init_tfm, + .exit = essiv_cbc_exit_tfm, } }; static int cbcmac_setkey(struct crypto_shash *tfm, const u8 *in_key, From patchwork Wed Aug 21 14:32:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171962 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055405ily; Wed, 21 Aug 2019 07:33:16 -0700 (PDT) X-Google-Smtp-Source: APXvYqxaE3j5WpXrOtfkYIbGxWE6GBu3xiR1pckP+tgmyV0XZ9aGvjgjKu6JFrrAMe3UgP8+pXBL X-Received: by 2002:a17:902:1e6:: with SMTP id b93mr33685421plb.295.1566397996319; Wed, 21 Aug 2019 07:33:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397996; cv=none; d=google.com; s=arc-20160816; b=tciBMJRJFCSCFui/0LLa6tA4NizJGYSyZZKblmXophBAIo7zIRC1so9wNGcBEotgOX G4PwXlpVLB+dswgnSDec4KqXWgewvtpUh02e83nhJzizB6eHFr+wT96H/sAcz0WGvs0O 14yQoJvorRwRv/2A05WTaKE+Qk9wS41gjx4jjU8EuyYSx0kxqgTd6R/GU1E802jUiBRH BCRBBZeyLED57zGrY3AHFQM3a5bHW/+V3z1GYbsHr2y7Nk/Hc40q2qDmep0H5lZVMm3C ltc5sSmih6tvfz5OZJN7VYth8+yhOwtMeXXZUiHL/ePk7tTKuvVrcGEIXUfS/Jh7R2t4 k9uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=RP1SFo8o3z1+1lccKhX0WLT22cd3vbFmL6iK1Nrc258=; b=drVcBTfOSGyMkC2//r4F6jM6rK0YeajNKeD8MXvXj8QPUPelnnoQbadRGvvhVzfUFM 2PU55CSxKK/FxRDBcLSX5Ktb8YwLvU2+uCeMzwcnMSrdnugUMgVk3c6i0+tQiSDW+hhG MKHEO1YaLDRIAOX4mdfMf5mr8oAiUJ87Pd2xzcRAm1XStgoKKWR3UXtKXKOxwzZc+9ha NyrOMLs1qPZB87Pvh7sscA1G3uLqd5n1zS6TbvE1j9RXIuTLneftzqATdHKxWMw5qTCd jq9RM1zYQciDIVabqD05+cumlYXI8D1Mh05xy2GtNN38P05PsPFqKoM9DBzhN3uEjbGX ryyg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OYSQlSzf; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.16; Wed, 21 Aug 2019 07:33:16 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OYSQlSzf; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729073AbfHUOdP (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:15 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:40611 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728822AbfHUOdP (ORCPT ); Wed, 21 Aug 2019 10:33:15 -0400 Received: by mail-wm1-f67.google.com with SMTP id c5so2176791wmb.5 for ; Wed, 21 Aug 2019 07:33:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RP1SFo8o3z1+1lccKhX0WLT22cd3vbFmL6iK1Nrc258=; b=OYSQlSzfG5P/PXfk3dZ5BKvjslEuFcuOODfcf6mrtIUupYQEsoCe1sRvpl3Piy4ReG h4LoZQWBmOgHmANRAuM3DwHLO5K9CYQVvHEwahV3V0do2pZssPXr8sRvM3QTNTfVGjbm h9AKjwDRsj/Ow3GobQ08/HRL0RzmrMTS762tNHMLwg2BiKtD8habB9WtI/hv0qN7q7de 7RDwGA3F8r7Ygzz3nL/8od9fBIgkgBKkXS9f2//ffCkFFkpbnfmkrwA7qSGKtBoH2g6i S1nDfQ9vsYkQaGZpdxAkkRHqggLFfsAZMcSUwZK338ZReAq7q89e2hOIEfqSifi91/69 8nJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RP1SFo8o3z1+1lccKhX0WLT22cd3vbFmL6iK1Nrc258=; b=RSScXLRGil4yVLy/eqHgZB5c3I4d8JUw0a82Qz7ydkR0ixl93wGCtQbt8yTBK5lBmX PjkYstea+3Yk/1L75KjDCZmVQ+Y50LSSYGLwSL+OjVFgFkZYc4CiPYw4Gsw2f5/59dIu GlzHfn56kyegdCu51x4tZR2SCliFXDlq+giN4FnsFHQCL0OhbIQXnDBaECVBHUL/y1i2 hySp5vkHF/+lFlODonKCiI0LqCKUtYfeOfjBvREG0rvrRFCCvw4kvhOB51EteUA/TCfL G8KQspY+PevgGyGcKbUHi9KUXO1E0Y2t6M8PfMKKuyBbMi2LLyTL0CxVmCddCUSVYjue DujQ== X-Gm-Message-State: APjAAAUWbTLYHEXUTLp/jc93Q3pWA98d0S+CswJMDksn6qbGXFbktVsm Xdf/ME/AbmEPHGlqlKoBQsrAkgQQ/4f0Rw== X-Received: by 2002:a1c:1f4e:: with SMTP id f75mr326557wmf.137.1566397992768; Wed, 21 Aug 2019 07:33:12 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:12 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 08/17] crypto: skcipher - add the ability to abort a skcipher walk Date: Wed, 21 Aug 2019 17:32:44 +0300 Message-Id: <20190821143253.30209-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org After starting a skcipher walk, the only way to ensure that all resources it has tied up are released is to complete it. In some cases, it will be useful to be able to abort a walk cleanly after it has started, so add this ability to the skcipher walk API. Signed-off-by: Ard Biesheuvel --- crypto/skcipher.c | 3 +++ include/crypto/internal/skcipher.h | 5 +++++ 2 files changed, 8 insertions(+) -- 2.17.1 Tested-by: Ard Biesheuvel diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 5d836fc3df3e..973ab1c7dcca 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -140,6 +140,9 @@ int skcipher_walk_done(struct skcipher_walk *walk, int err) goto already_advanced; } + if (unlikely(!n)) + goto finish; + scatterwalk_advance(&walk->in, n); scatterwalk_advance(&walk->out, n); already_advanced: diff --git a/include/crypto/internal/skcipher.h b/include/crypto/internal/skcipher.h index d68faa5759ad..bc488173531f 100644 --- a/include/crypto/internal/skcipher.h +++ b/include/crypto/internal/skcipher.h @@ -148,6 +148,11 @@ int skcipher_walk_aead_decrypt(struct skcipher_walk *walk, struct aead_request *req, bool atomic); void skcipher_walk_complete(struct skcipher_walk *walk, int err); +static inline void skcipher_walk_abort(struct skcipher_walk *walk) +{ + skcipher_walk_done(walk, walk->nbytes); +} + static inline void ablkcipher_request_complete(struct ablkcipher_request *req, int err) { From patchwork Wed Aug 21 14:32:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171964 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055452ily; Wed, 21 Aug 2019 07:33:17 -0700 (PDT) X-Google-Smtp-Source: APXvYqwmCaAJZoGzeku4gDqCpYWZ61gIE6lVqSwAOyH/U58fcyIIc6pjpAf1JGEDmTND/O2jBma3 X-Received: by 2002:a17:90a:220a:: with SMTP id c10mr297965pje.33.1566397997710; Wed, 21 Aug 2019 07:33:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397997; cv=none; d=google.com; s=arc-20160816; b=R05DxFi9+lCY9rUaySSIg2LZdZqeCm1rHQPN0HOUGzAnMit2fS6fyumR6KpLUO4Ds9 iSMNT1J0F0RDwsdb0G8H38a+oRtLErUSgyHXiT+LCplVilkgyNPYNkzzwts3225fndoq PABopFsuZ/5Y+ZZ144JGtM6ESTeHcM6Vfx6Lrk8vaScxVY7r+tEgOThgZLinxQ4lE8Ts sXbBUxhWb4dOfswL55xr1fo96RZp0Np3AggAzTvBwW5mNCZcjytNrcuG109hV2qo4A0p YZNPBvijHBOIgLmckQKFZRqFxgce6xU/nZ2baiZrWk+FSfI2E9bscEYeTmb7x1WfC/iX aO7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=gkbLgubyXxiJMMRMzAmpKCmMYVcmKbpOZX5ob6E8mqs=; b=eK0P0afEnL+wiqyJlTwILTzZBmkbixhTJKPsR2VU9HGOfQuHwxOPZZMss9kNkdn9+f fzAJ5biV7YA9m2Fj8BQ2WRw8YsX+xRSNYSC/zEHGCCpoo/stUn1c6kUq7FFO2plVO/t7 oewnuKXFefrY9OBjiIadXkz58ZcU4m+2Jbq14xcdxorcoEEtxrYHElqvkjkrfp7gUxHa 8nzisosNfxkuzx66EkgMBRUWtTHL8Qk+YpwRnwziWqZUiPN6EOFg1MIFBq3TpUCqMBXO +eTWNa9i32P8QJjV/GAMm4nimWIweh4/Db5No4pQ91OQssdEC2MZhVaJ9USKVKGrFxNf xC4Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=szJfzLYA; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.17; Wed, 21 Aug 2019 07:33:17 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=szJfzLYA; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728984AbfHUOdR (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:17 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:35667 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728964AbfHUOdQ (ORCPT ); Wed, 21 Aug 2019 10:33:16 -0400 Received: by mail-wm1-f66.google.com with SMTP id l2so2394581wmg.0 for ; Wed, 21 Aug 2019 07:33:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gkbLgubyXxiJMMRMzAmpKCmMYVcmKbpOZX5ob6E8mqs=; b=szJfzLYALpagYlStlGrcldxHDC71KgrrVq/2Lvma8CKThhHW29eHiIOAyilSpqSYI1 lpVUdhvz3EyHWhXAgAM0gShZ06fTET0OCdcEqmt1X8vrfmJRns9YcuTk4YgZGRIRn7Ph Z+Wmg4XzICMidAwoPKXCfJZYOuNRQBjb7+L5vmYDhkc2Rjh/NyrBl+ivQ5LY3gxMBO2m ItYZWOuim7xv1hpPmiV5F+61E5t4gld4t0Y+72TH4NoQfz/mWVerUSEW7I6NY/EzvWSn 3qstcIz3JOFBYZJXko3tGp94lnDWN8ETVy0MW6IHerNDjvEw9i6t/JsVh7Mf+qZMylV1 H3Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gkbLgubyXxiJMMRMzAmpKCmMYVcmKbpOZX5ob6E8mqs=; b=SIiK7GJ0KZtGhHkLCF7iKtqtAHDEczloer8DI3jovsrwZ7oPJK7+OZLlWmTU5bqVgw Lrylpw4bADflSCIeUlUTiZfl9a87XW+B6A86Gi9YPzHGFwMM00rG+qMrcApjcLYVGEfL cHpgPMd6QI8PwZLTAWYLm1G/sCgKQc1a6w+fBgtq1xeSpna2vQuBoR3AEY0pU97xMhzx 72QkySh2qPC56utLH/PwJM0ZLtpLVpMk2/HDGdbhhYhtL7POYBsubtqK5jD56jkQO+P4 xvzLI8HJ0Rw7BIqGNXS9zWb/Tp+7S0kMeAhzWCuVSJfZWWqrphQhH1ZFm30qwI6622Np wBug== X-Gm-Message-State: APjAAAVj1pAStI+6u28rexbBcI//A5XWMiex1C2FGOg2DwHRxq6uQu5c oV2u3zWCM73WMsoY457SLEWeGjkqNWnZfw== X-Received: by 2002:a1c:20c3:: with SMTP id g186mr364851wmg.15.1566397994370; Wed, 21 Aug 2019 07:33:14 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:13 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 09/17] crypto: arm64/aes-cts-cbc-ce - performance tweak Date: Wed, 21 Aug 2019 17:32:45 +0300 Message-Id: <20190821143253.30209-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Optimize away one of the tbl instructions in the decryption path, which turns out to be unnecessary. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) -- 2.17.1 diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 2879f030a749..38cd5a2091a8 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -293,12 +293,11 @@ AES_ENTRY(aes_cbc_cts_decrypt) ld1 {v5.16b}, [x5] /* get iv */ dec_prepare w3, x2, x6 - tbl v2.16b, {v1.16b}, v4.16b decrypt_block v0, w3, x2, x6, w7 - eor v2.16b, v2.16b, v0.16b + tbl v2.16b, {v0.16b}, v3.16b + eor v2.16b, v2.16b, v1.16b tbx v0.16b, {v1.16b}, v4.16b - tbl v2.16b, {v2.16b}, v3.16b decrypt_block v0, w3, x2, x6, w7 eor v0.16b, v0.16b, v5.16b /* xor with iv */ From patchwork Wed Aug 21 14:32:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171965 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055516ily; Wed, 21 Aug 2019 07:33:20 -0700 (PDT) X-Google-Smtp-Source: APXvYqxa83U2ZYF7hA4B27uCaxdEO/x0E375kRyLmTI9bxzQScbIFtDBBXf6eAyzyp57yyZhdbuj X-Received: by 2002:aa7:93cf:: with SMTP id y15mr35665817pff.251.1566397999948; Wed, 21 Aug 2019 07:33:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566397999; cv=none; d=google.com; s=arc-20160816; b=0hC08k4TiyQTSrMz1Bruc//Gn7nuFb9rXxb85d8NcfdNtFMxBGnSGATCzwGsEh2kRY K4NXUamsfw5Hy5npH/yG72IE//AT3Hw8fCpfmj1R6sxSbBf/VeazE5YsYOMSb9fEM5yY B2grUzskkP/tcRlZvwwxOWCdNuqPGQ354FhJWmJLB2NvuRWiM12LxqhhRWHLdJK7mn/a dw+3mYeel5Mlr7ydNg88lJcxozDvBmfbYXy362kkW9N/im/F6MvLJdsa0HbBfGax699B G+EalTafIUqLAULHq3MemkV9yBP/QWjbAa7jH4qbvM5sPsf4bLBOVBqsXAFwrpgmFNTL NSMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=4rOqC3sDxYEvPZjvQzEBSOK6QwtrxcRA7eQQZQ1weWg=; b=c77ne2RyXt/WrhHVL5bJBwrMfZQw7DPMuSw5m6llPNOgAlyn3fIGkrAPD/zq4z4bC0 KHgY4BYWGnu5iMTQ+ryKYMRh6cB1KqVHKXCqNz7atKbukazCBahErbhiBktx1ilr0/wD xjJceOVsysQqf7KG2GWU4+hhZ7GWwTw2rmMVHaETpFU54L5iscidZVaCIlSi15oqsygk miEqdNA3mSPeuiAaRoFWLODfpLtFHY0j09v4sxpg7Ln2pUnmPDZlk7roDhAIwkUPT3fR A9D2pCldMRLiOA3Ln0E9bSOx3uNGHoNxUd5+EolAJXpQdLSdFZDqU6X2I6KgLr1kGWyc +qTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EwWBwtST; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j16si15915283pfh.0.2019.08.21.07.33.19; Wed, 21 Aug 2019 07:33:19 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EwWBwtST; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728546AbfHUOdT (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:19 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:55975 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728763AbfHUOdT (ORCPT ); Wed, 21 Aug 2019 10:33:19 -0400 Received: by mail-wm1-f66.google.com with SMTP id f72so2364837wmf.5 for ; Wed, 21 Aug 2019 07:33:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=4rOqC3sDxYEvPZjvQzEBSOK6QwtrxcRA7eQQZQ1weWg=; b=EwWBwtST+N+pDX259VYc6hMCiCiRFsj5huNKdJ61973ObppP+x7c4HczzHMCQk40Hl 2I15aox8qgvkaf5hIp8L603o7BXcA7qCTxRlkp5+b48IXNJj/shtNJ3AbCLYrILoKQkZ tb5wqj5RwMR2WLttSe13OMyZWGabjDhhGvrTfjgJGutKHlsTdJE+J1h16F2ozOMRT/1U xB1I9mQEVFC527Qc/VnmSGKotjqPcrnJoa1tzu9m3UnQcw9rD2t4e+RS9gR/ldyGGD67 /Kj7vE/OaljhXwag4MS6PwIUw4WvXGPDh1EyaWwbHN6ByHPiT4wSpBa9hkYfjWKtE8BM 6CcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=4rOqC3sDxYEvPZjvQzEBSOK6QwtrxcRA7eQQZQ1weWg=; b=PNnZ31jLEb6HQIxAsIT6mgBGMPU5xC7d7SH2rQa+RSv/yIBGVgzQQdxCK/K5wYMINh JRcbitCzg+xUkZPG7KGVcpi5eYQsmQtqEa2feTdnDJTG5fDpQX9Y9RVSn3ocpHyt8QkE VCtucdC66GuRGeqryh7J8VFz1Q+UDOp67lXDLJfbjyaKY9yMRDvO9AW8RieE0C9F7W9n oktqrbB54vKeFLSB37uig+5e6wzivlwlXHq/2gdQ9+UW+m0Q3V3d+3TH1rzqlj2iW9eh GkkFmrajjCrqO3PfcZbIPKS5hiMpfBZC26jLJ94ahP3M44cB7hxHaaxceiEZXd0fdni8 TrYQ== X-Gm-Message-State: APjAAAWwP2IAESVonoFlzt1xNTzs0UYCvfoVLErkzczEOlC+68ELHIkx 0jHnAPwl2Oeo43RZ8SHH2F4zwltYVvou9Q== X-Received: by 2002:a7b:c4c6:: with SMTP id g6mr423241wmk.52.1566397996004; Wed, 21 Aug 2019 07:33:16 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:15 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 10/17] crypto: arm64/aes-cts-cbc - move request context data to the stack Date: Wed, 21 Aug 2019 17:32:46 +0300 Message-Id: <20190821143253.30209-11-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Since the CTS-CBC code completes synchronously, there is no point in keeping part of the scratch data it uses in the request context, so move it to the stack instead. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-glue.c | 61 +++++++++----------- 1 file changed, 26 insertions(+), 35 deletions(-) -- 2.17.1 diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 4154bb93a85b..5ee980c5a5c2 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -107,12 +107,6 @@ asmlinkage void aes_mac_update(u8 const in[], u32 const rk[], int rounds, int blocks, u8 dg[], int enc_before, int enc_after); -struct cts_cbc_req_ctx { - struct scatterlist sg_src[2]; - struct scatterlist sg_dst[2]; - struct skcipher_request subreq; -}; - struct crypto_aes_xts_ctx { struct crypto_aes_ctx key1; struct crypto_aes_ctx __aligned(8) key2; @@ -292,23 +286,20 @@ static int __maybe_unused cbc_decrypt(struct skcipher_request *req) return cbc_decrypt_walk(req, &walk); } -static int cts_cbc_init_tfm(struct crypto_skcipher *tfm) -{ - crypto_skcipher_set_reqsize(tfm, sizeof(struct cts_cbc_req_ctx)); - return 0; -} - static int cts_cbc_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); - struct cts_cbc_req_ctx *rctx = skcipher_request_ctx(req); int err, rounds = 6 + ctx->key_length / 4; int cbc_blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; struct scatterlist *src = req->src, *dst = req->dst; + struct scatterlist sg_src[2], sg_dst[2]; + struct skcipher_request subreq; struct skcipher_walk walk; - skcipher_request_set_tfm(&rctx->subreq, tfm); + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, skcipher_request_flags(req), + NULL, NULL); if (req->cryptlen <= AES_BLOCK_SIZE) { if (req->cryptlen < AES_BLOCK_SIZE) @@ -317,31 +308,30 @@ static int cts_cbc_encrypt(struct skcipher_request *req) } if (cbc_blocks > 0) { - skcipher_request_set_crypt(&rctx->subreq, req->src, req->dst, + skcipher_request_set_crypt(&subreq, req->src, req->dst, cbc_blocks * AES_BLOCK_SIZE, req->iv); - err = skcipher_walk_virt(&walk, &rctx->subreq, false) ?: - cbc_encrypt_walk(&rctx->subreq, &walk); + err = skcipher_walk_virt(&walk, &subreq, false) ?: + cbc_encrypt_walk(&subreq, &walk); if (err) return err; if (req->cryptlen == AES_BLOCK_SIZE) return 0; - dst = src = scatterwalk_ffwd(rctx->sg_src, req->src, - rctx->subreq.cryptlen); + dst = src = scatterwalk_ffwd(sg_src, req->src, subreq.cryptlen); if (req->dst != req->src) - dst = scatterwalk_ffwd(rctx->sg_dst, req->dst, - rctx->subreq.cryptlen); + dst = scatterwalk_ffwd(sg_dst, req->dst, + subreq.cryptlen); } /* handle ciphertext stealing */ - skcipher_request_set_crypt(&rctx->subreq, src, dst, + skcipher_request_set_crypt(&subreq, src, dst, req->cryptlen - cbc_blocks * AES_BLOCK_SIZE, req->iv); - err = skcipher_walk_virt(&walk, &rctx->subreq, false); + err = skcipher_walk_virt(&walk, &subreq, false); if (err) return err; @@ -357,13 +347,16 @@ static int cts_cbc_decrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); - struct cts_cbc_req_ctx *rctx = skcipher_request_ctx(req); int err, rounds = 6 + ctx->key_length / 4; int cbc_blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; struct scatterlist *src = req->src, *dst = req->dst; + struct scatterlist sg_src[2], sg_dst[2]; + struct skcipher_request subreq; struct skcipher_walk walk; - skcipher_request_set_tfm(&rctx->subreq, tfm); + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, skcipher_request_flags(req), + NULL, NULL); if (req->cryptlen <= AES_BLOCK_SIZE) { if (req->cryptlen < AES_BLOCK_SIZE) @@ -372,31 +365,30 @@ static int cts_cbc_decrypt(struct skcipher_request *req) } if (cbc_blocks > 0) { - skcipher_request_set_crypt(&rctx->subreq, req->src, req->dst, + skcipher_request_set_crypt(&subreq, req->src, req->dst, cbc_blocks * AES_BLOCK_SIZE, req->iv); - err = skcipher_walk_virt(&walk, &rctx->subreq, false) ?: - cbc_decrypt_walk(&rctx->subreq, &walk); + err = skcipher_walk_virt(&walk, &subreq, false) ?: + cbc_decrypt_walk(&subreq, &walk); if (err) return err; if (req->cryptlen == AES_BLOCK_SIZE) return 0; - dst = src = scatterwalk_ffwd(rctx->sg_src, req->src, - rctx->subreq.cryptlen); + dst = src = scatterwalk_ffwd(sg_src, req->src, subreq.cryptlen); if (req->dst != req->src) - dst = scatterwalk_ffwd(rctx->sg_dst, req->dst, - rctx->subreq.cryptlen); + dst = scatterwalk_ffwd(sg_dst, req->dst, + subreq.cryptlen); } /* handle ciphertext stealing */ - skcipher_request_set_crypt(&rctx->subreq, src, dst, + skcipher_request_set_crypt(&subreq, src, dst, req->cryptlen - cbc_blocks * AES_BLOCK_SIZE, req->iv); - err = skcipher_walk_virt(&walk, &rctx->subreq, false); + err = skcipher_walk_virt(&walk, &subreq, false); if (err) return err; @@ -675,7 +667,6 @@ static struct skcipher_alg aes_algs[] = { { .setkey = skcipher_aes_setkey, .encrypt = cts_cbc_encrypt, .decrypt = cts_cbc_decrypt, - .init = cts_cbc_init_tfm, }, { .base = { .cra_name = "__essiv(cbc(aes),sha256)", From patchwork Wed Aug 21 14:32:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171967 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055649ily; Wed, 21 Aug 2019 07:33:25 -0700 (PDT) X-Google-Smtp-Source: APXvYqyp7CaI/iVDhp0CqreAwGDEIHjVRR8Q45NQZlkhhsvZ18zG8wGDIOHMk133BR10rDBwRRph X-Received: by 2002:a63:784c:: with SMTP id t73mr30403019pgc.268.1566398004831; Wed, 21 Aug 2019 07:33:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566398004; cv=none; d=google.com; s=arc-20160816; b=XiF+rpZikODrOstGYgICiJFTbEXWI9SR4QAG1Pccl39RGi+DbaxcTHepmbesYgb3zB whyEF/QrIZ7AQm0WS5ctUcgJAlrI2nWQyBn72AgburmOITRYhTIKJqIW1vO1vYSOL2i0 CzcNsvqkyBKvNNlw7Ag9ALV1QEBQtdKaB/L6zJfUY+FtBZsI3wELQqrnesZuA3s/avP5 uFq+wgR5VTiTElLb+mKUjlHuHPp31jl5kXR3RNPA2DNU1YtkE1V9ZeWZcmdrZsyNGY9q nf7+F+VZmjpqfCtUCZYcS356Zaxgfz/9FFSyR/NgIpwi0zVL0nbVpPIZ1BgFoiyUv9Xa ee9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=lztbyRYUuu5cMp563Zlls3XdyH0KZSCJz0DwSh7fXts=; b=Es/BMB/Byf/dp/PAXv8fPSINqBkHa4NH5AKaRoGuYF50eZ89MA6Kl98jNeWFm+0vkO 7n93mVeXnadZKOHrlqKH8yiFFp+FUocv12mNZyUh8+WuPwJiQUsLM+wDqENlabpGq+0D SA7oE37brPzbaSyPyaEulZsqnVriHjCYbfw58ZlYlYC26refUz8oZl7zEIEW7H997+G3 GUb/Gv+wvAgqCX2fzh7j14hFxFsnhLhAZYy2MTotWZm3BXhxBLF1y1NCw2Tg4kXS2B6/ SGraVreWHlL8DWfuDFCM+Hkb+MPntX3BeRpi6Q4qd81VtGp207X0hwX3C5bC7w2pcX2N cG7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AZj37Mwb; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q13si14616343pgr.514.2019.08.21.07.33.24; Wed, 21 Aug 2019 07:33:24 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AZj37Mwb; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729011AbfHUOdX (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:23 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:39528 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728763AbfHUOdW (ORCPT ); Wed, 21 Aug 2019 10:33:22 -0400 Received: by mail-wm1-f67.google.com with SMTP id i63so2381044wmg.4 for ; Wed, 21 Aug 2019 07:33:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=lztbyRYUuu5cMp563Zlls3XdyH0KZSCJz0DwSh7fXts=; b=AZj37MwbCaqyqq/V9/vMcceDFC/L/NYKHrv/WBzJjhkpE6DBMyo0T/ZINQeoCmgxMT oMVyQ6t5g6NosETRFqncBWEoc3haQar9hXndW81bovcunShgneer4VMQVsUyylQJHYIj oplc0ZpoAsN9RhRZTG9asF6VYfcZNctu9vx5hrVAoE6knnwbU0mTHbTYwOcNj0eAP3G/ PIwHUJHZTC1d/5vCq6DMlP4TvGG8ZovVY9VEjQBodg/bQPzseOV4rYujwG0GdhO/xUXu 2oS3LwwiJhQ62SN/xH91HWUMbZL6w7k/smFybBJwCkO/ksKTsvR2Zf2zZI4iRM8IgwCo h1Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=lztbyRYUuu5cMp563Zlls3XdyH0KZSCJz0DwSh7fXts=; b=kgBLdjQYCtEXEUOPbX4Yi2tzPoAfi9oJ4vHYpr6oKZ0uIKGmcPf/jtGTEOK6PSa15J 419yxkWLDTYwqM95FiFwMxL1Y4edVg4J8D8tk3GEZvk6Znyj9LDOBSHLVizcqeke3iVj SptRK5CZ3mK+FEEsB0jtyhRBHdsbku8celtCEMtNdZ1+AEeMt0JU0QMSDW28cgdLjzH6 ZgS8zn/jDDzoi6llLktCh6XLyU/HbHbkn/XrbDOxQ5TldvobLZrDIKDJtQ/StBueZxPa N96c848DfivysWnun7PhVy5ttVVLhINjpFmbFPC9qOXaZqTcGRZo4zWwWW/jM0eF4Oob gyNA== X-Gm-Message-State: APjAAAXgfCzMrleoLdnoAIu30Sgn593oFk1B9j+q7AndTHkGvOnKsHry 7gH2KiIxgpbZSUdgxLrvI9n4Q6nAoVzmHQ== X-Received: by 2002:a05:600c:228e:: with SMTP id 14mr361348wmf.101.1566397998234; Wed, 21 Aug 2019 07:33:18 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:17 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 11/17] crypto: arm64/aes - implement support for XTS ciphertext stealing Date: Wed, 21 Aug 2019 17:32:47 +0300 Message-Id: <20190821143253.30209-12-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add the missing support for ciphertext stealing in the implementation of AES-XTS, which is part of the XTS specification but was omitted up until now due to lack of a need for it. The asm helpers are updated so they can deal with any input size, as long as the last full block and the final partial block are presented at the same time. The glue code is updated so that the common case of operating on a sector or page is mostly as before. When CTS is needed, the walk is split up into two pieces, unless the entire input is covered by a single step. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-glue.c | 126 ++++++++++++++++++-- arch/arm64/crypto/aes-modes.S | 99 ++++++++++++--- 2 files changed, 195 insertions(+), 30 deletions(-) -- 2.17.1 diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 5ee980c5a5c2..eecb74fd2f61 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -90,10 +90,10 @@ asmlinkage void aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks, u8 ctr[]); asmlinkage void aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[], - int rounds, int blocks, u32 const rk2[], u8 iv[], + int rounds, int bytes, u32 const rk2[], u8 iv[], int first); asmlinkage void aes_xts_decrypt(u8 out[], u8 const in[], u32 const rk1[], - int rounds, int blocks, u32 const rk2[], u8 iv[], + int rounds, int bytes, u32 const rk2[], u8 iv[], int first); asmlinkage void aes_essiv_cbc_encrypt(u8 out[], u8 const in[], u32 const rk1[], @@ -529,21 +529,71 @@ static int __maybe_unused xts_encrypt(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); int err, first, rounds = 6 + ctx->key1.key_length / 4; + int tail = req->cryptlen % AES_BLOCK_SIZE; + struct scatterlist sg_src[2], sg_dst[2]; + struct skcipher_request subreq; + struct scatterlist *src, *dst; struct skcipher_walk walk; - unsigned int blocks; + + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; err = skcipher_walk_virt(&walk, req, false); - for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) { + if (unlikely(tail > 0 && walk.nbytes < walk.total)) { + int xts_blocks = DIV_ROUND_UP(req->cryptlen, + AES_BLOCK_SIZE) - 2; + + skcipher_walk_abort(&walk); + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, + skcipher_request_flags(req), + NULL, NULL); + skcipher_request_set_crypt(&subreq, req->src, req->dst, + xts_blocks * AES_BLOCK_SIZE, + req->iv); + req = &subreq; + err = skcipher_walk_virt(&walk, req, false); + } else { + tail = 0; + } + + for (first = 1; walk.nbytes >= AES_BLOCK_SIZE; first = 0) { + int nbytes = walk.nbytes; + + if (walk.nbytes < walk.total) + nbytes &= ~(AES_BLOCK_SIZE - 1); + kernel_neon_begin(); aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, blocks, + ctx->key1.key_enc, rounds, nbytes, ctx->key2.key_enc, walk.iv, first); kernel_neon_end(); - err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); } - return err; + if (err || likely(!tail)) + return err; + + dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); + + skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, + req->iv); + + err = skcipher_walk_virt(&walk, &subreq, false); + if (err) + return err; + + kernel_neon_begin(); + aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); + kernel_neon_end(); + + return skcipher_walk_done(&walk, 0); } static int __maybe_unused xts_decrypt(struct skcipher_request *req) @@ -551,21 +601,72 @@ static int __maybe_unused xts_decrypt(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); int err, first, rounds = 6 + ctx->key1.key_length / 4; + int tail = req->cryptlen % AES_BLOCK_SIZE; + struct scatterlist sg_src[2], sg_dst[2]; + struct skcipher_request subreq; + struct scatterlist *src, *dst; struct skcipher_walk walk; - unsigned int blocks; + + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; err = skcipher_walk_virt(&walk, req, false); - for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) { + if (unlikely(tail > 0 && walk.nbytes < walk.total)) { + int xts_blocks = DIV_ROUND_UP(req->cryptlen, + AES_BLOCK_SIZE) - 2; + + skcipher_walk_abort(&walk); + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, + skcipher_request_flags(req), + NULL, NULL); + skcipher_request_set_crypt(&subreq, req->src, req->dst, + xts_blocks * AES_BLOCK_SIZE, + req->iv); + req = &subreq; + err = skcipher_walk_virt(&walk, req, false); + } else { + tail = 0; + } + + for (first = 1; walk.nbytes >= AES_BLOCK_SIZE; first = 0) { + int nbytes = walk.nbytes; + + if (walk.nbytes < walk.total) + nbytes &= ~(AES_BLOCK_SIZE - 1); + kernel_neon_begin(); aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, blocks, + ctx->key1.key_dec, rounds, nbytes, ctx->key2.key_enc, walk.iv, first); kernel_neon_end(); - err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); } - return err; + if (err || likely(!tail)) + return err; + + dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); + + skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, + req->iv); + + err = skcipher_walk_virt(&walk, &subreq, false); + if (err) + return err; + + + kernel_neon_begin(); + aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); + kernel_neon_end(); + + return skcipher_walk_done(&walk, 0); } static struct skcipher_alg aes_algs[] = { { @@ -646,6 +747,7 @@ static struct skcipher_alg aes_algs[] = { { .min_keysize = 2 * AES_MIN_KEY_SIZE, .max_keysize = 2 * AES_MAX_KEY_SIZE, .ivsize = AES_BLOCK_SIZE, + .walksize = 2 * AES_BLOCK_SIZE, .setkey = xts_set_key, .encrypt = xts_encrypt, .decrypt = xts_decrypt, diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 38cd5a2091a8..f2c2ba739f36 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -413,10 +413,10 @@ AES_ENDPROC(aes_ctr_encrypt) /* + * aes_xts_encrypt(u8 out[], u8 const in[], u8 const rk1[], int rounds, + * int bytes, u8 const rk2[], u8 iv[], int first) * aes_xts_decrypt(u8 out[], u8 const in[], u8 const rk1[], int rounds, - * int blocks, u8 const rk2[], u8 iv[], int first) - * aes_xts_decrypt(u8 out[], u8 const in[], u8 const rk1[], int rounds, - * int blocks, u8 const rk2[], u8 iv[], int first) + * int bytes, u8 const rk2[], u8 iv[], int first) */ .macro next_tweak, out, in, tmp @@ -451,7 +451,7 @@ AES_ENTRY(aes_xts_encrypt) .LxtsencloopNx: next_tweak v4, v4, v8 .LxtsencNx: - subs w4, w4, #4 + subs w4, w4, #64 bmi .Lxtsenc1x ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ next_tweak v5, v4, v8 @@ -468,33 +468,66 @@ AES_ENTRY(aes_xts_encrypt) eor v2.16b, v2.16b, v6.16b st1 {v0.16b-v3.16b}, [x0], #64 mov v4.16b, v7.16b - cbz w4, .Lxtsencout + cbz w4, .Lxtsencret xts_reload_mask v8 b .LxtsencloopNx .Lxtsenc1x: - adds w4, w4, #4 + adds w4, w4, #64 beq .Lxtsencout + subs w4, w4, #16 + bmi .LxtsencctsNx .Lxtsencloop: - ld1 {v1.16b}, [x1], #16 - eor v0.16b, v1.16b, v4.16b + ld1 {v0.16b}, [x1], #16 +.Lxtsencctsout: + eor v0.16b, v0.16b, v4.16b encrypt_block v0, w3, x2, x8, w7 eor v0.16b, v0.16b, v4.16b - st1 {v0.16b}, [x0], #16 - subs w4, w4, #1 - beq .Lxtsencout + cbz w4, .Lxtsencout + subs w4, w4, #16 next_tweak v4, v4, v8 + bmi .Lxtsenccts + st1 {v0.16b}, [x0], #16 b .Lxtsencloop .Lxtsencout: + st1 {v0.16b}, [x0] +.Lxtsencret: st1 {v4.16b}, [x6] ldp x29, x30, [sp], #16 ret -AES_ENDPROC(aes_xts_encrypt) +.LxtsencctsNx: + mov v0.16b, v3.16b + sub x0, x0, #16 +.Lxtsenccts: + adr_l x8, .Lcts_permute_table + + add x1, x1, w4, sxtw /* rewind input pointer */ + add w4, w4, #16 /* # bytes in final block */ + add x9, x8, #32 + add x8, x8, x4 + sub x9, x9, x4 + add x4, x0, x4 /* output address of final block */ + + ld1 {v1.16b}, [x1] /* load final block */ + ld1 {v2.16b}, [x8] + ld1 {v3.16b}, [x9] + + tbl v2.16b, {v0.16b}, v2.16b + tbx v0.16b, {v1.16b}, v3.16b + st1 {v2.16b}, [x4] /* overlapping stores */ + mov w4, wzr + b .Lxtsencctsout +AES_ENDPROC(aes_xts_encrypt) AES_ENTRY(aes_xts_decrypt) stp x29, x30, [sp, #-16]! mov x29, sp + /* subtract 16 bytes if we are doing CTS */ + sub w8, w4, #0x10 + tst w4, #0xf + csel w4, w4, w8, eq + ld1 {v4.16b}, [x6] xts_load_mask v8 cbz w7, .Lxtsdecnotfirst @@ -509,7 +542,7 @@ AES_ENTRY(aes_xts_decrypt) .LxtsdecloopNx: next_tweak v4, v4, v8 .LxtsdecNx: - subs w4, w4, #4 + subs w4, w4, #64 bmi .Lxtsdec1x ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ next_tweak v5, v4, v8 @@ -530,22 +563,52 @@ AES_ENTRY(aes_xts_decrypt) xts_reload_mask v8 b .LxtsdecloopNx .Lxtsdec1x: - adds w4, w4, #4 + adds w4, w4, #64 beq .Lxtsdecout + subs w4, w4, #16 .Lxtsdecloop: - ld1 {v1.16b}, [x1], #16 - eor v0.16b, v1.16b, v4.16b + ld1 {v0.16b}, [x1], #16 + bmi .Lxtsdeccts +.Lxtsdecctsout: + eor v0.16b, v0.16b, v4.16b decrypt_block v0, w3, x2, x8, w7 eor v0.16b, v0.16b, v4.16b st1 {v0.16b}, [x0], #16 - subs w4, w4, #1 - beq .Lxtsdecout + cbz w4, .Lxtsdecout + subs w4, w4, #16 next_tweak v4, v4, v8 b .Lxtsdecloop .Lxtsdecout: st1 {v4.16b}, [x6] ldp x29, x30, [sp], #16 ret + +.Lxtsdeccts: + adr_l x8, .Lcts_permute_table + + add x1, x1, w4, sxtw /* rewind input pointer */ + add w4, w4, #16 /* # bytes in final block */ + add x9, x8, #32 + add x8, x8, x4 + sub x9, x9, x4 + add x4, x0, x4 /* output address of final block */ + + next_tweak v5, v4, v8 + + ld1 {v1.16b}, [x1] /* load final block */ + ld1 {v2.16b}, [x8] + ld1 {v3.16b}, [x9] + + eor v0.16b, v0.16b, v5.16b + decrypt_block v0, w3, x2, x8, w7 + eor v0.16b, v0.16b, v5.16b + + tbl v2.16b, {v0.16b}, v2.16b + tbx v0.16b, {v1.16b}, v3.16b + + st1 {v2.16b}, [x4] /* overlapping stores */ + mov w4, wzr + b .Lxtsdecctsout AES_ENDPROC(aes_xts_decrypt) /* From patchwork Wed Aug 21 14:32:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171966 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055623ily; Wed, 21 Aug 2019 07:33:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqw+DRacTeE8O5krkCbOU3FI9I/nMoOdodS5E2TYIOh5oG1Yw1RpLZZd8apGxzoFlYFIbLlu X-Received: by 2002:a17:902:2f05:: with SMTP id s5mr34346280plb.170.1566398004112; Wed, 21 Aug 2019 07:33:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566398004; cv=none; d=google.com; s=arc-20160816; b=oH0DPGSO2YGpXFQKUpdV53JwUyOwf9O45adwW+a79zrNJALwwp2nVolev9OAPxzUeH ggw+1SEUyn7lDxobhybNuogEDrysdfqHc28Jk7/M5WkPXZw20q8s8wmJMj0kZM4+JWU4 a1XN7S4Hu3A0fpUulxLf8gMaxW/fSmExiYCEXW4YMINYT4lS846ilUs1dLwBFhAigGT1 N1JPPJw8lq9GUfpE5CWUBDxLDrf8OrHZEp/cQzNbI1FKAxMeAlLE1XxD53OaWZwyJOMl o/agnbfstKin5yMLuspGOJPmrdtLIqBWyLUC3NLms9JXo8w4LcSQyX5f5F1u80xjlRZq nOlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=inENxCG3FHp1A7FxE00V5o6CRhf+tePDkDNu7R3bkwg=; b=o/EENRM0GfrFAfE7UAjV2MyGN1mRmw+hTvAFP6Uov5vrTpzhwaEnaEHQM9b3X7o7oC Jkau+sLbr4QFZ9tzoRBh3s0RJOCFnSuQFgpMSHs97P0Hr6vKQg03m448T4Hp+aleQ+MJ QBHCxX6jwNjVFR7SCmttSO8xxtwv2N6dtwTOxwCz31VYXiCzPAtB6X1LwVNZioWJESXm IdxllMd00thhbzPVRCcY7zG1wf9mrWIfLhJZaXteNpEB0V+XmWTMx3fOB2/fqiK2CZjx SQqJccg3BO4vTm6BE74oOxM/JlyIMXlkSWd6FsE/XpbNBAcRS4lFgN8j5cLO69qKcVQY eamg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JCts070z; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q13si14616343pgr.514.2019.08.21.07.33.23; Wed, 21 Aug 2019 07:33:24 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JCts070z; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728964AbfHUOdW (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:22 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:55987 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729011AbfHUOdW (ORCPT ); Wed, 21 Aug 2019 10:33:22 -0400 Received: by mail-wm1-f67.google.com with SMTP id f72so2365119wmf.5 for ; Wed, 21 Aug 2019 07:33:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=inENxCG3FHp1A7FxE00V5o6CRhf+tePDkDNu7R3bkwg=; b=JCts070zoVcAudvUMZh6M+fun9PKjMe9ETdFJWpoUTq6tF4DseUW86ZWryLTErFtGs nx2qJa85uGK4emAP6Oxq0hA67Vmg8cZAoKYlN6V7dVGIIGgEjsmSYMJa2tvFVZUS4kXo Hoh8bL0HnxgrjDvnZPgWNRZVzrphoh8yzHg/AE2g2orLeUmP/5JB6u4VrKrM8t8aXolZ BQaYDZXmp3HgwOmh05uFB+K9NyfsrXAYNq5wnOp/fUIP7zvacgvmRIHjVrTgVhTSxrFL yDBZjwaVm9YiiyT8pnKPNt1fIANx3OAeeSNW6Ye+VS3jq1kv/vy6sBvoRqE12GrrczTO y0Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=inENxCG3FHp1A7FxE00V5o6CRhf+tePDkDNu7R3bkwg=; b=GHf9PqLz5MkbJVgI8xdT9/XjnKp4EHVZFneG1c/4Zj3P4e9UsG2tzqLvxy5vjHH7kC GrDMwxCPqn1Y9hdFw3OG02TVvwefKDHq377s5ObxZQQAP5i+QgBSrw6lxN+WmR/1nI+3 xTRFJiSp2EnYRe0hhv013DUDWtqssz+dkQJdLoGG05T9l/2OKG1oMfGDQxefEfVh0q0B rx0/bbS4AqLFyjBHOH4m0fkyICrBlscy71tomJA7ZpRmCC6ldgrbrjA6YTr3HIy0emAb 4lNoMyhjemqfX0FkvUAzV8A7qQVzlEfMmYhjjuzzSiRUAwDNghiKD5kRHTiKm1wtpyiy k0GQ== X-Gm-Message-State: APjAAAWXm6vMR3dbdPSjP46bdde9XNFCowKa7v2LkERYpsshLYKfMW8/ 3Mp00B9wWaY2PYljDg/MLrBLR2LgNRTAGg== X-Received: by 2002:a1c:c706:: with SMTP id x6mr405732wmf.104.1566397999771; Wed, 21 Aug 2019 07:33:19 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:19 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 12/17] crypto: arm64/aes-neonbs - implement ciphertext stealing for XTS Date: Wed, 21 Aug 2019 17:32:48 +0300 Message-Id: <20190821143253.30209-13-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Update the AES-XTS implementation based on NEON instructions so that it can deal with inputs whose size is not a multiple of the cipher block size. This is part of the original XTS specification, but was never implemented before in the Linux kernel. Since the bit slicing driver is only faster if it can operate on at least 7 blocks of input at the same time, let's reuse the alternate path we are adding for CTS to process any data tail whose size is not a multiple of 128 bytes. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce.S | 3 + arch/arm64/crypto/aes-glue.c | 2 + arch/arm64/crypto/aes-modes.S | 3 + arch/arm64/crypto/aes-neon.S | 5 + arch/arm64/crypto/aes-neonbs-glue.c | 111 +++++++++++++++++--- 5 files changed, 110 insertions(+), 14 deletions(-) -- 2.17.1 diff --git a/arch/arm64/crypto/aes-ce.S b/arch/arm64/crypto/aes-ce.S index 00bd2885feaa..c132c49c89a8 100644 --- a/arch/arm64/crypto/aes-ce.S +++ b/arch/arm64/crypto/aes-ce.S @@ -21,6 +21,9 @@ .macro xts_reload_mask, tmp .endm + .macro xts_cts_skip_tw, reg, lbl + .endm + /* preload all round keys */ .macro load_round_keys, rounds, rk cmp \rounds, #12 diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index eecb74fd2f61..327ac8d1489e 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -1073,5 +1073,7 @@ module_cpu_feature_match(AES, aes_init); module_init(aes_init); EXPORT_SYMBOL(neon_aes_ecb_encrypt); EXPORT_SYMBOL(neon_aes_cbc_encrypt); +EXPORT_SYMBOL(neon_aes_xts_encrypt); +EXPORT_SYMBOL(neon_aes_xts_decrypt); #endif module_exit(aes_exit); diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index f2c2ba739f36..131618389f1f 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -442,6 +442,7 @@ AES_ENTRY(aes_xts_encrypt) cbz w7, .Lxtsencnotfirst enc_prepare w3, x5, x8 + xts_cts_skip_tw w7, .LxtsencNx encrypt_block v4, w3, x5, x8, w7 /* first tweak */ enc_switch_key w3, x2, x8 b .LxtsencNx @@ -530,10 +531,12 @@ AES_ENTRY(aes_xts_decrypt) ld1 {v4.16b}, [x6] xts_load_mask v8 + xts_cts_skip_tw w7, .Lxtsdecskiptw cbz w7, .Lxtsdecnotfirst enc_prepare w3, x5, x8 encrypt_block v4, w3, x5, x8, w7 /* first tweak */ +.Lxtsdecskiptw: dec_prepare w3, x2, x8 b .LxtsdecNx diff --git a/arch/arm64/crypto/aes-neon.S b/arch/arm64/crypto/aes-neon.S index 0cac5df6c901..22d9b110cf78 100644 --- a/arch/arm64/crypto/aes-neon.S +++ b/arch/arm64/crypto/aes-neon.S @@ -19,6 +19,11 @@ xts_load_mask \tmp .endm + /* special case for the neon-bs driver calling into this one for CTS */ + .macro xts_cts_skip_tw, reg, lbl + tbnz \reg, #1, \lbl + .endm + /* multiply by polynomial 'x' in GF(2^8) */ .macro mul_by_x, out, in, temp, const sshr \temp, \in, #7 diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-neonbs-glue.c index bafd2ebef8f1..ea873b8904c4 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -45,6 +46,12 @@ asmlinkage void neon_aes_ecb_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks); asmlinkage void neon_aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks, u8 iv[]); +asmlinkage void neon_aes_xts_encrypt(u8 out[], u8 const in[], + u32 const rk1[], int rounds, int bytes, + u32 const rk2[], u8 iv[], int first); +asmlinkage void neon_aes_xts_decrypt(u8 out[], u8 const in[], + u32 const rk1[], int rounds, int bytes, + u32 const rk2[], u8 iv[], int first); struct aesbs_ctx { u8 rk[13 * (8 * AES_BLOCK_SIZE) + 32]; @@ -64,6 +71,7 @@ struct aesbs_ctr_ctx { struct aesbs_xts_ctx { struct aesbs_ctx key; u32 twkey[AES_MAX_KEYLENGTH_U32]; + struct crypto_aes_ctx cts; }; static int aesbs_setkey(struct crypto_skcipher *tfm, const u8 *in_key, @@ -270,6 +278,10 @@ static int aesbs_xts_setkey(struct crypto_skcipher *tfm, const u8 *in_key, return err; key_len /= 2; + err = aes_expandkey(&ctx->cts, in_key, key_len); + if (err) + return err; + err = aes_expandkey(&rk, in_key + key_len, key_len); if (err) return err; @@ -302,48 +314,119 @@ static int ctr_encrypt_sync(struct skcipher_request *req) return ctr_encrypt(req); } -static int __xts_crypt(struct skcipher_request *req, +static int __xts_crypt(struct skcipher_request *req, bool encrypt, void (*fn)(u8 out[], u8 const in[], u8 const rk[], int rounds, int blocks, u8 iv[])) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct aesbs_xts_ctx *ctx = crypto_skcipher_ctx(tfm); + int tail = req->cryptlen % (8 * AES_BLOCK_SIZE); + struct scatterlist sg_src[2], sg_dst[2]; + struct skcipher_request subreq; + struct scatterlist *src, *dst; struct skcipher_walk walk; - int err; + int nbytes, err; + int first = 1; + u8 *out, *in; + + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; + + /* ensure that the cts tail is covered by a single step */ + if (unlikely(tail > 0 && tail < AES_BLOCK_SIZE)) { + int xts_blocks = DIV_ROUND_UP(req->cryptlen, + AES_BLOCK_SIZE) - 2; + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, + skcipher_request_flags(req), + NULL, NULL); + skcipher_request_set_crypt(&subreq, req->src, req->dst, + xts_blocks * AES_BLOCK_SIZE, + req->iv); + req = &subreq; + } else { + tail = 0; + } err = skcipher_walk_virt(&walk, req, false); if (err) return err; - kernel_neon_begin(); - neon_aes_ecb_encrypt(walk.iv, walk.iv, ctx->twkey, ctx->key.rounds, 1); - kernel_neon_end(); - while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; - if (walk.nbytes < walk.total) + if (walk.nbytes < walk.total || walk.nbytes % AES_BLOCK_SIZE) blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + out = walk.dst.virt.addr; + in = walk.src.virt.addr; + nbytes = walk.nbytes; + kernel_neon_begin(); - fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, - ctx->key.rounds, blocks, walk.iv); + if (likely(blocks > 6)) { /* plain NEON is faster otherwise */ + if (first) + neon_aes_ecb_encrypt(walk.iv, walk.iv, + ctx->twkey, + ctx->key.rounds, 1); + first = 0; + + fn(out, in, ctx->key.rk, ctx->key.rounds, blocks, + walk.iv); + + out += blocks * AES_BLOCK_SIZE; + in += blocks * AES_BLOCK_SIZE; + nbytes -= blocks * AES_BLOCK_SIZE; + } + + if (walk.nbytes == walk.total && nbytes > 0) + goto xts_tail; + kernel_neon_end(); - err = skcipher_walk_done(&walk, - walk.nbytes - blocks * AES_BLOCK_SIZE); + skcipher_walk_done(&walk, nbytes); } - return err; + + if (err || likely(!tail)) + return err; + + /* handle ciphertext stealing */ + dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); + + skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, + req->iv); + + err = skcipher_walk_virt(&walk, req, false); + if (err) + return err; + + out = walk.dst.virt.addr; + in = walk.src.virt.addr; + nbytes = walk.nbytes; + + kernel_neon_begin(); +xts_tail: + if (encrypt) + neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, ctx->key.rounds, + nbytes, ctx->twkey, walk.iv, first ?: 2); + else + neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, ctx->key.rounds, + nbytes, ctx->twkey, walk.iv, first ?: 2); + kernel_neon_end(); + + return skcipher_walk_done(&walk, 0); } static int xts_encrypt(struct skcipher_request *req) { - return __xts_crypt(req, aesbs_xts_encrypt); + return __xts_crypt(req, true, aesbs_xts_encrypt); } static int xts_decrypt(struct skcipher_request *req) { - return __xts_crypt(req, aesbs_xts_decrypt); + return __xts_crypt(req, false, aesbs_xts_decrypt); } static struct skcipher_alg aes_algs[] = { { From patchwork Wed Aug 21 14:32:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171968 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055678ily; Wed, 21 Aug 2019 07:33:26 -0700 (PDT) X-Google-Smtp-Source: APXvYqyniAojKHU96yXxWgyD5MiBKPuRP9C3OLml8skwtU/Ship7RvwHhgJaUrUJHF5Gis0z3FyP X-Received: by 2002:a17:90a:a4c5:: with SMTP id l5mr287724pjw.49.1566398006542; Wed, 21 Aug 2019 07:33:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566398006; cv=none; d=google.com; s=arc-20160816; b=uZt0YD/GLulLevkitJCF22dQFPw6kaNKC2x6e7NNx6trVfg+A9iWmb22FHGh2WFKi2 5tehNcl9/+G90ImaXwu0gmoUccIw7ihR4l8P31PiwiJD1qe72zj52uzIBYAgRU7lzc5E KKRi46ZmjAeRCtQghWSidSrxGbh4PWp8dc2U5NY6XGeLcJYtpwGTyX4v7QTn3aQrUoDq wf8EBbDE57PAKmC+pnPVG7dH39Qe/ikUzukpwIzjpvcGAhtk/QED7FdY9UkoX6+s7s+5 8gNKGTf43u0qAdTej02DtS9WQcgET0QXK+DtjXYd67Hk/x+828iAWgGLjPrZcJ61EJLI EeBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=ss0hT+6FcmRtVnmYdaRt5cn9RrJ1dCb8Db0Ibbel/hM=; b=qYtgJb1mDIDnPvdHfPK7XLitb6mtMIV1qd1nrxpDN/As3aep9IMHJTztpZkoNLqmd3 RTMKUNvDaSix8dcQiNLGivww8R4T0pA6dx3t6JhtEtkMuSLLNklpEkTi5hMeGxAONVWr Sz8N/DvdESwFpyAUZf4ZoCKWhzA0zQKmKi+IGPuZwnWGP6Xl18b6romMoI6v5i29eIHG d7tGp2TbrTAC+sWGOT+/qUNSP3fuGvqCbgV7lnlcSoZCY9/ZtdRT3K4PhJpBQxL9zxKe +PUUx5LRUsuu7N3EtLweLSiiiUVqzotv6xh/PEwQVgocEEmH0HCdWRxjYqEzIQc8RAER iidA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=K5pEavGV; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q13si14616343pgr.514.2019.08.21.07.33.26; Wed, 21 Aug 2019 07:33:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=K5pEavGV; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729309AbfHUOdZ (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:25 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:36788 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729308AbfHUOdZ (ORCPT ); Wed, 21 Aug 2019 10:33:25 -0400 Received: by mail-wm1-f66.google.com with SMTP id g67so2389583wme.1 for ; Wed, 21 Aug 2019 07:33:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ss0hT+6FcmRtVnmYdaRt5cn9RrJ1dCb8Db0Ibbel/hM=; b=K5pEavGVOe8xcJ2yhaSYpEJJApLDk9mF3VxseRT7bqmrCWUm6jFf1uzqpqitVXyjfP B/Lg4kXPAGKVzdA0RLss4Mb3x+xclF1vYlpj9FJ5cmIBIh3Mo5WI2KhsXvyoFo/Zhk+s U1CSRbE6XZs9dZ5FEM/XsOds/ZBjlqux7UOy4R9fflqPPhRkBduYMZgxYwNFtzjgDNaE 0u/06+0LAJD+oYwLOTg+P8vNAOrTe3DeKxAN2SL2q6HHR5n3LX2rWA7+6WPAFbyd3ntd Z5dAk8ghzKZZcJmb0vIpJGdy53XdQbK1uO7P/IGGKISJCRrvqslYTNIhNIf6VEf45E1z wZnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ss0hT+6FcmRtVnmYdaRt5cn9RrJ1dCb8Db0Ibbel/hM=; b=r/Dik7RF4T4L4GO56e2JfdZSexgcyfT27C8NzJriJc68BxZChshm4pn3Ictd7Hpkkb iQ+9y6vbwRvecORSRkcMOyzN3irSx/6ThU5KQunPQaioAvaiT2FHg44TZyzonzIc9VYw Puf9iAkaDRtrs1DfGMp7C/C+WE+iAmTX5dtcFER0egvQbIXS49ZGiiaiGPQoQYj+tLrM YrFYQDXGYVpBBDLiJpNxsbmm3VOGHdxhJjA7yoh64aRcP8sFmgh2SmypereNiyCuE6OP b4gToTLVI2OZssHloKvlS7RT1y+RtiDM+RDZYHv3WJSdhz6IoisM+wxNWwHS7NJRY3If wrMA== X-Gm-Message-State: APjAAAVP1C+p3Ho2zN3dUvErQCJQcrzINHI8hY55d63lMAcFdIWMHkXY XWsFLeQGJw/DWT6FZw9u8PKoR+PoNEULEw== X-Received: by 2002:a1c:6145:: with SMTP id v66mr429554wmb.42.1566398001760; Wed, 21 Aug 2019 07:33:21 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:21 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 13/17] crypto: arm/aes-ce - implement ciphertext stealing for XTS Date: Wed, 21 Aug 2019 17:32:49 +0300 Message-Id: <20190821143253.30209-14-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Update the AES-XTS implementation based on AES instructions so that it can deal with inputs whose size is not a multiple of the cipher block size. This is part of the original XTS specification, but was never implemented before in the Linux kernel. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-core.S | 103 ++++++++++++++-- arch/arm/crypto/aes-ce-glue.c | 128 ++++++++++++++++++-- 2 files changed, 208 insertions(+), 23 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/aes-ce-core.S b/arch/arm/crypto/aes-ce-core.S index bb6ec1844370..763e51604ab6 100644 --- a/arch/arm/crypto/aes-ce-core.S +++ b/arch/arm/crypto/aes-ce-core.S @@ -369,9 +369,9 @@ ENDPROC(ce_aes_ctr_encrypt) /* * aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[], int rounds, - * int blocks, u8 iv[], u32 const rk2[], int first) + * int bytes, u8 iv[], u32 const rk2[], int first) * aes_xts_decrypt(u8 out[], u8 const in[], u32 const rk1[], int rounds, - * int blocks, u8 iv[], u32 const rk2[], int first) + * int bytes, u8 iv[], u32 const rk2[], int first) */ .macro next_tweak, out, in, const, tmp @@ -414,7 +414,7 @@ ENTRY(ce_aes_xts_encrypt) .Lxtsencloop4x: next_tweak q4, q4, q15, q10 .Lxtsenc4x: - subs r4, r4, #4 + subs r4, r4, #64 bmi .Lxtsenc1x vld1.8 {q0-q1}, [r1]! @ get 4 pt blocks vld1.8 {q2-q3}, [r1]! @@ -434,24 +434,58 @@ ENTRY(ce_aes_xts_encrypt) vst1.8 {q2-q3}, [r0]! vmov q4, q7 teq r4, #0 - beq .Lxtsencout + beq .Lxtsencret b .Lxtsencloop4x .Lxtsenc1x: - adds r4, r4, #4 + adds r4, r4, #64 beq .Lxtsencout + subs r4, r4, #16 + bmi .LxtsencctsNx .Lxtsencloop: vld1.8 {q0}, [r1]! +.Lxtsencctsout: veor q0, q0, q4 bl aes_encrypt veor q0, q0, q4 - vst1.8 {q0}, [r0]! - subs r4, r4, #1 + teq r4, #0 beq .Lxtsencout + subs r4, r4, #16 next_tweak q4, q4, q15, q6 + bmi .Lxtsenccts + vst1.8 {q0}, [r0]! b .Lxtsencloop .Lxtsencout: + vst1.8 {q0}, [r0] +.Lxtsencret: vst1.8 {q4}, [r5] pop {r4-r6, pc} + +.LxtsencctsNx: + vmov q0, q3 + sub r0, r0, #16 +.Lxtsenccts: + movw ip, :lower16:.Lcts_permute_table + movt ip, :upper16:.Lcts_permute_table + + add r1, r1, r4 @ rewind input pointer + add r4, r4, #16 @ # bytes in final block + add lr, ip, #32 + add ip, ip, r4 + sub lr, lr, r4 + add r4, r0, r4 @ output address of final block + + vld1.8 {q1}, [r1] @ load final partial block + vld1.8 {q2}, [ip] + vld1.8 {q3}, [lr] + + vtbl.8 d4, {d0-d1}, d4 + vtbl.8 d5, {d0-d1}, d5 + vtbx.8 d0, {d2-d3}, d6 + vtbx.8 d1, {d2-d3}, d7 + + vst1.8 {q2}, [r4] @ overlapping stores + mov r4, #0 + b .Lxtsencctsout ENDPROC(ce_aes_xts_encrypt) @@ -462,13 +496,17 @@ ENTRY(ce_aes_xts_decrypt) prepare_key r2, r3 vmov q4, q0 + /* subtract 16 bytes if we are doing CTS */ + tst r4, #0xf + subne r4, r4, #0x10 + teq r6, #0 @ start of a block? bne .Lxtsdec4x .Lxtsdecloop4x: next_tweak q4, q4, q15, q10 .Lxtsdec4x: - subs r4, r4, #4 + subs r4, r4, #64 bmi .Lxtsdec1x vld1.8 {q0-q1}, [r1]! @ get 4 ct blocks vld1.8 {q2-q3}, [r1]! @@ -491,22 +529,55 @@ ENTRY(ce_aes_xts_decrypt) beq .Lxtsdecout b .Lxtsdecloop4x .Lxtsdec1x: - adds r4, r4, #4 + adds r4, r4, #64 beq .Lxtsdecout + subs r4, r4, #16 .Lxtsdecloop: vld1.8 {q0}, [r1]! + bmi .Lxtsdeccts +.Lxtsdecctsout: veor q0, q0, q4 - add ip, r2, #32 @ 3rd round key bl aes_decrypt veor q0, q0, q4 vst1.8 {q0}, [r0]! - subs r4, r4, #1 + teq r4, #0 beq .Lxtsdecout + subs r4, r4, #16 next_tweak q4, q4, q15, q6 b .Lxtsdecloop .Lxtsdecout: vst1.8 {q4}, [r5] pop {r4-r6, pc} + +.Lxtsdeccts: + movw ip, :lower16:.Lcts_permute_table + movt ip, :upper16:.Lcts_permute_table + + add r1, r1, r4 @ rewind input pointer + add r4, r4, #16 @ # bytes in final block + add lr, ip, #32 + add ip, ip, r4 + sub lr, lr, r4 + add r4, r0, r4 @ output address of final block + + next_tweak q5, q4, q15, q6 + + vld1.8 {q1}, [r1] @ load final partial block + vld1.8 {q2}, [ip] + vld1.8 {q3}, [lr] + + veor q0, q0, q5 + bl aes_decrypt + veor q0, q0, q5 + + vtbl.8 d4, {d0-d1}, d4 + vtbl.8 d5, {d0-d1}, d5 + vtbx.8 d0, {d2-d3}, d6 + vtbx.8 d1, {d2-d3}, d7 + + vst1.8 {q2}, [r4] @ overlapping stores + mov r4, #0 + b .Lxtsdecctsout ENDPROC(ce_aes_xts_decrypt) /* @@ -532,3 +603,13 @@ ENTRY(ce_aes_invert) vst1.32 {q0}, [r0] bx lr ENDPROC(ce_aes_invert) + + .section ".rodata", "a" + .align 6 +.Lcts_permute_table: + .byte 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff + .byte 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff + .byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7 + .byte 0x8, 0x9, 0xa, 0xb, 0xc, 0xd, 0xe, 0xf + .byte 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff + .byte 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff diff --git a/arch/arm/crypto/aes-ce-glue.c b/arch/arm/crypto/aes-ce-glue.c index 486e862ae34a..c215792a2494 100644 --- a/arch/arm/crypto/aes-ce-glue.c +++ b/arch/arm/crypto/aes-ce-glue.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -39,10 +40,10 @@ asmlinkage void ce_aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks, u8 ctr[]); asmlinkage void ce_aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[], - int rounds, int blocks, u8 iv[], + int rounds, int bytes, u8 iv[], u32 const rk2[], int first); asmlinkage void ce_aes_xts_decrypt(u8 out[], u8 const in[], u32 const rk1[], - int rounds, int blocks, u8 iv[], + int rounds, int bytes, u8 iv[], u32 const rk2[], int first); struct aes_block { @@ -317,20 +318,71 @@ static int xts_encrypt(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); int err, first, rounds = num_rounds(&ctx->key1); + int tail = req->cryptlen % AES_BLOCK_SIZE; + struct scatterlist sg_src[2], sg_dst[2]; + struct skcipher_request subreq; + struct scatterlist *src, *dst; struct skcipher_walk walk; - unsigned int blocks; + + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; err = skcipher_walk_virt(&walk, req, false); - for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) { + if (unlikely(tail > 0 && walk.nbytes < walk.total)) { + int xts_blocks = DIV_ROUND_UP(req->cryptlen, + AES_BLOCK_SIZE) - 2; + + skcipher_walk_abort(&walk); + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, + skcipher_request_flags(req), + NULL, NULL); + skcipher_request_set_crypt(&subreq, req->src, req->dst, + xts_blocks * AES_BLOCK_SIZE, + req->iv); + req = &subreq; + err = skcipher_walk_virt(&walk, req, false); + } else { + tail = 0; + } + + for (first = 1; walk.nbytes >= AES_BLOCK_SIZE; first = 0) { + int nbytes = walk.nbytes; + + if (walk.nbytes < walk.total) + nbytes &= ~(AES_BLOCK_SIZE - 1); + kernel_neon_begin(); ce_aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, blocks, walk.iv, + ctx->key1.key_enc, rounds, nbytes, walk.iv, ctx->key2.key_enc, first); kernel_neon_end(); - err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); } - return err; + + if (err || likely(!tail)) + return err; + + dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); + + skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, + req->iv); + + err = skcipher_walk_virt(&walk, req, false); + if (err) + return err; + + kernel_neon_begin(); + ce_aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, walk.nbytes, walk.iv, + ctx->key2.key_enc, first); + kernel_neon_end(); + + return skcipher_walk_done(&walk, 0); } static int xts_decrypt(struct skcipher_request *req) @@ -338,20 +390,71 @@ static int xts_decrypt(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); int err, first, rounds = num_rounds(&ctx->key1); + int tail = req->cryptlen % AES_BLOCK_SIZE; + struct scatterlist sg_src[2], sg_dst[2]; + struct skcipher_request subreq; + struct scatterlist *src, *dst; struct skcipher_walk walk; - unsigned int blocks; + + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; err = skcipher_walk_virt(&walk, req, false); - for (first = 1; (blocks = (walk.nbytes / AES_BLOCK_SIZE)); first = 0) { + if (unlikely(tail > 0 && walk.nbytes < walk.total)) { + int xts_blocks = DIV_ROUND_UP(req->cryptlen, + AES_BLOCK_SIZE) - 2; + + skcipher_walk_abort(&walk); + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, + skcipher_request_flags(req), + NULL, NULL); + skcipher_request_set_crypt(&subreq, req->src, req->dst, + xts_blocks * AES_BLOCK_SIZE, + req->iv); + req = &subreq; + err = skcipher_walk_virt(&walk, req, false); + } else { + tail = 0; + } + + for (first = 1; walk.nbytes >= AES_BLOCK_SIZE; first = 0) { + int nbytes = walk.nbytes; + + if (walk.nbytes < walk.total) + nbytes &= ~(AES_BLOCK_SIZE - 1); + kernel_neon_begin(); ce_aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, blocks, walk.iv, + ctx->key1.key_dec, rounds, nbytes, walk.iv, ctx->key2.key_enc, first); kernel_neon_end(); - err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); } - return err; + + if (err || likely(!tail)) + return err; + + dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); + + skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, + req->iv); + + err = skcipher_walk_virt(&walk, req, false); + if (err) + return err; + + kernel_neon_begin(); + ce_aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, walk.nbytes, walk.iv, + ctx->key2.key_enc, first); + kernel_neon_end(); + + return skcipher_walk_done(&walk, 0); } static struct skcipher_alg aes_algs[] = { { @@ -426,6 +529,7 @@ static struct skcipher_alg aes_algs[] = { { .min_keysize = 2 * AES_MIN_KEY_SIZE, .max_keysize = 2 * AES_MAX_KEY_SIZE, .ivsize = AES_BLOCK_SIZE, + .walksize = 2 * AES_BLOCK_SIZE, .setkey = xts_set_key, .encrypt = xts_encrypt, .decrypt = xts_decrypt, From patchwork Wed Aug 21 14:32:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171969 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055699ily; Wed, 21 Aug 2019 07:33:27 -0700 (PDT) X-Google-Smtp-Source: APXvYqyBHcOVK7Att/nDn64+0WFDYrWBxnch4mM8tydkiH1/mmnzFehEisaJiWTiJJ4UO3z6km8I X-Received: by 2002:a62:383:: with SMTP id 125mr1912742pfd.248.1566398007036; Wed, 21 Aug 2019 07:33:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566398007; cv=none; d=google.com; s=arc-20160816; b=eP996wa+P5ee4Yq9j/cMVYJnwUsLT+c6TMYbtilgp1BRJAJocfp7Dh2NaIfrLsvJ8c JNS5qZRAIRuwqupElFRRrc2vhNruPhZFByaX/f0It7Fs8/DFJXZOkpNjAtw8oRzanmz4 Wn/m8XIz0smLY0UiR4vcfE4CDwqlR6cJruCG5rz+g6Cw45oM3ttxOKW7Smpfx4K8xxLe 2efoBLvT6kEpBcvHck4+l9wLb4hL+PuydXSKCCbnOTiJKbweoTYhs6ZHajFRzBjhAUS1 Kr4O6YG4AIVw1Ty4+cZolice4gXN1N91uljdR5rtA9wcrkM2Y1hzrfaMmKQCNxLyL7Fy Fw7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=1Bgaz1dGwPYBh9D5oDVzcqGBYRtI51XEZb5wgMPv0t8=; b=U7GUdXtfV3J8eL4BLoFueOlyHsqcC3/Uf9M9umSU7krEMJZrYtPTKb5ZZGerk2h34u FtTH8A8/Gv25pF/VdPKMOMXhjONpL9MxFw0DTytBmNGd4PchVGiqHP5BC+gUIjECekUm hLheBwa4a4nRY3IBqQPKqQzMLxeNDoLzbLaCvg321BaX5lc033eTpKoieSzsnVPIq6pD hhcPHUZhuDzChIQFs4l809Yfdwv2yk2OZdqEY57e54ONnruK3d4O+GsJKAPiJx77vez1 yPT0TOCHMU27sxPuio7QgbC2wu32uYj2i3ExiHd3MO0esLVYa++PTl74n7PjBBwwSaZL EzuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=McKiPujK; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q13si14616343pgr.514.2019.08.21.07.33.26; Wed, 21 Aug 2019 07:33:27 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=McKiPujK; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729308AbfHUOd0 (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:26 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:33124 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728763AbfHUOd0 (ORCPT ); Wed, 21 Aug 2019 10:33:26 -0400 Received: by mail-wr1-f67.google.com with SMTP id u16so2294868wrr.0 for ; Wed, 21 Aug 2019 07:33:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1Bgaz1dGwPYBh9D5oDVzcqGBYRtI51XEZb5wgMPv0t8=; b=McKiPujK4M2XLA9lKDIgmo7yIJIL7+Pj6OKhyllOWeUqpt/dCFYZFjbq9gJogIH6AV 5B1lJ61pyyatFHuBUjOBcEgb/zzho4PtiEs1g4IFN5u30M5rG3bfHXXefHhXhfcLKHvI jwQq+4OFME4ON5SrKtDRWtTz/0FxR8psab2GUrkX/KHbDi8z3hNmafR6Sof9tXm83qGS ddoCTtc99QJWzUEiLViSQBz5fVn7Zk5lN5q3QSUltgu187RAtydKtSPThY20J20MOPMZ dHtkccY1N4xw1FLN0S0P6XFWWhuItfqhtdpwA/jwTjhQMtiLdLpLo05R/ubVZSaIVB4Z 00fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1Bgaz1dGwPYBh9D5oDVzcqGBYRtI51XEZb5wgMPv0t8=; b=XIEKwJEZxudR3M9+LnXQf7z8GlRuC/gbFQayhdbJMOgE/21mG2K8RLeLVwDLY9cVsF o4w7PdUgdO/XjGjTcwYgiOp0OwRHfkfrFIPjWt6D/g1l09GBA3EBH6ZOdLNgiEfz21WO mBlQQCzpTEP7mfnCOk8MXH55KHS9vFbdHqxHH1lq4k3IdDLtiul3fRnDoIkSr4rjmsI2 R8TpWg3Dn9+38VzXtLIRTPXIetOExiTw6mH+pVAkt9RBw2ese6qMxU15BjAlYVk43GMh YRIAziuvVLXqfvP9WFAY2RCIb2W57CeYiNWjpqURaNNywkiTs5EyGZXBSZDVbglu1QY7 Qn7g== X-Gm-Message-State: APjAAAX+Yb3rEVMz/A6BqCMY5SUbXru9o0gCKcEOnf12O1F3l6TdMkn4 5c+Qy/qfjl0wS6Etb/UhDZ+mpm1lVmXspg== X-Received: by 2002:adf:ee4f:: with SMTP id w15mr41427297wro.337.1566398003483; Wed, 21 Aug 2019 07:33:23 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:22 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 14/17] crypto: arm/aes-neonbs - implement ciphertext stealing for XTS Date: Wed, 21 Aug 2019 17:32:50 +0300 Message-Id: <20190821143253.30209-15-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Update the AES-XTS implementation based on NEON instructions so that it can deal with inputs whose size is not a multiple of the cipher block size. This is part of the original XTS specification, but was never implemented before in the Linux kernel. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-neonbs-core.S | 16 +++-- arch/arm/crypto/aes-neonbs-glue.c | 69 +++++++++++++++++--- 2 files changed, 72 insertions(+), 13 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/aes-neonbs-core.S b/arch/arm/crypto/aes-neonbs-core.S index bb75918e4984..cfaed4e67535 100644 --- a/arch/arm/crypto/aes-neonbs-core.S +++ b/arch/arm/crypto/aes-neonbs-core.S @@ -889,9 +889,9 @@ ENDPROC(aesbs_ctr_encrypt) /* * aesbs_xts_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, - * int blocks, u8 iv[]) + * int blocks, u8 iv[], int reorder_last_tweak) * aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, - * int blocks, u8 iv[]) + * int blocks, u8 iv[], int reorder_last_tweak) */ __xts_prepare8: vld1.8 {q14}, [r7] // load iv @@ -944,17 +944,25 @@ __xts_prepare8: vld1.8 {q7}, [r1]! next_tweak q14, q12, q15, q13 - veor q7, q7, q12 +THUMB( itt le ) + W(cmple) r8, #0 + ble 1f +0: veor q7, q7, q12 vst1.8 {q12}, [r4, :128] -0: vst1.8 {q14}, [r7] // store next iv + vst1.8 {q14}, [r7] // store next iv bx lr + +1: vswp q12, q14 + b 0b ENDPROC(__xts_prepare8) .macro __xts_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7 push {r4-r8, lr} mov r5, sp // preserve sp ldrd r6, r7, [sp, #24] // get blocks and iv args + ldr r8, [sp, #32] // reorder final tweak? + rsb r8, r8, #1 sub ip, sp, #128 // make room for 8x tweak bic ip, ip, #0xf // align sp to 16 bytes mov sp, ip diff --git a/arch/arm/crypto/aes-neonbs-glue.c b/arch/arm/crypto/aes-neonbs-glue.c index 9000d0796d5e..e85839a8aaeb 100644 --- a/arch/arm/crypto/aes-neonbs-glue.c +++ b/arch/arm/crypto/aes-neonbs-glue.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -37,9 +38,9 @@ asmlinkage void aesbs_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, int blocks, u8 ctr[], u8 final[]); asmlinkage void aesbs_xts_encrypt(u8 out[], u8 const in[], u8 const rk[], - int rounds, int blocks, u8 iv[]); + int rounds, int blocks, u8 iv[], int); asmlinkage void aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[], - int rounds, int blocks, u8 iv[]); + int rounds, int blocks, u8 iv[], int); struct aesbs_ctx { int rounds; @@ -53,6 +54,7 @@ struct aesbs_cbc_ctx { struct aesbs_xts_ctx { struct aesbs_ctx key; + struct crypto_cipher *cts_tfm; struct crypto_cipher *tweak_tfm; }; @@ -291,6 +293,9 @@ static int aesbs_xts_setkey(struct crypto_skcipher *tfm, const u8 *in_key, return err; key_len /= 2; + err = crypto_cipher_setkey(ctx->cts_tfm, in_key, key_len); + if (err) + return err; err = crypto_cipher_setkey(ctx->tweak_tfm, in_key + key_len, key_len); if (err) return err; @@ -302,7 +307,13 @@ static int xts_init(struct crypto_tfm *tfm) { struct aesbs_xts_ctx *ctx = crypto_tfm_ctx(tfm); + ctx->cts_tfm = crypto_alloc_cipher("aes", 0, 0); + if (IS_ERR(ctx->cts_tfm)) + return PTR_ERR(ctx->cts_tfm); + ctx->tweak_tfm = crypto_alloc_cipher("aes", 0, 0); + if (IS_ERR(ctx->tweak_tfm)) + crypto_free_cipher(ctx->cts_tfm); return PTR_ERR_OR_ZERO(ctx->tweak_tfm); } @@ -312,17 +323,34 @@ static void xts_exit(struct crypto_tfm *tfm) struct aesbs_xts_ctx *ctx = crypto_tfm_ctx(tfm); crypto_free_cipher(ctx->tweak_tfm); + crypto_free_cipher(ctx->cts_tfm); } -static int __xts_crypt(struct skcipher_request *req, +static int __xts_crypt(struct skcipher_request *req, bool encrypt, void (*fn)(u8 out[], u8 const in[], u8 const rk[], - int rounds, int blocks, u8 iv[])) + int rounds, int blocks, u8 iv[], int)) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct aesbs_xts_ctx *ctx = crypto_skcipher_ctx(tfm); + int tail = req->cryptlen % AES_BLOCK_SIZE; + struct skcipher_request subreq; + u8 buf[2 * AES_BLOCK_SIZE]; struct skcipher_walk walk; int err; + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; + + if (unlikely(tail)) { + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, + skcipher_request_flags(req), + NULL, NULL); + skcipher_request_set_crypt(&subreq, req->src, req->dst, + req->cryptlen - tail, req->iv); + req = &subreq; + } + err = skcipher_walk_virt(&walk, req, true); if (err) return err; @@ -331,30 +359,53 @@ static int __xts_crypt(struct skcipher_request *req, while (walk.nbytes >= AES_BLOCK_SIZE) { unsigned int blocks = walk.nbytes / AES_BLOCK_SIZE; + int reorder_last_tweak = !encrypt && tail > 0; - if (walk.nbytes < walk.total) + if (walk.nbytes < walk.total) { blocks = round_down(blocks, walk.stride / AES_BLOCK_SIZE); + reorder_last_tweak = 0; + } kernel_neon_begin(); fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->key.rk, - ctx->key.rounds, blocks, walk.iv); + ctx->key.rounds, blocks, walk.iv, reorder_last_tweak); kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } - return err; + if (err || likely(!tail)) + return err; + + /* handle ciphertext stealing */ + scatterwalk_map_and_copy(buf, req->dst, req->cryptlen - AES_BLOCK_SIZE, + AES_BLOCK_SIZE, 0); + memcpy(buf + AES_BLOCK_SIZE, buf, tail); + scatterwalk_map_and_copy(buf, req->src, req->cryptlen, tail, 0); + + crypto_xor(buf, req->iv, AES_BLOCK_SIZE); + + if (encrypt) + crypto_cipher_encrypt_one(ctx->cts_tfm, buf, buf); + else + crypto_cipher_decrypt_one(ctx->cts_tfm, buf, buf); + + crypto_xor(buf, req->iv, AES_BLOCK_SIZE); + + scatterwalk_map_and_copy(buf, req->dst, req->cryptlen - AES_BLOCK_SIZE, + AES_BLOCK_SIZE + tail, 1); + return 0; } static int xts_encrypt(struct skcipher_request *req) { - return __xts_crypt(req, aesbs_xts_encrypt); + return __xts_crypt(req, true, aesbs_xts_encrypt); } static int xts_decrypt(struct skcipher_request *req) { - return __xts_crypt(req, aesbs_xts_decrypt); + return __xts_crypt(req, false, aesbs_xts_decrypt); } static struct skcipher_alg aes_algs[] = { { From patchwork Wed Aug 21 14:32:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171971 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055764ily; Wed, 21 Aug 2019 07:33:30 -0700 (PDT) X-Google-Smtp-Source: APXvYqzl4epPck9Dg4HiyuHMUZ9YEtIe8EVcgJLdpx2lzeKoZKmIrrS20g4FcUnsP40e77tzfwvi X-Received: by 2002:a17:902:2bc7:: with SMTP id l65mr21442514plb.119.1566398010207; Wed, 21 Aug 2019 07:33:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566398010; cv=none; d=google.com; s=arc-20160816; b=IINbo7afOtVpj3aBwGAuSlT52jowvJleZlEsfUM5cJWNLDD026GT4aZJ1tATOZkbCg HG6rtw4ioPS5uOsSWiM+r5W4S+WNbMqQ7LNL0g/jVpWSX/1FXbgXkQxgvFSOn3ctNBWh qBH+Wz3rQR8rk9n/LvZq+pMbzggspFi6QRs7WhYdn0ThWAc1T7vrCDuSbtE/GwXTm6cO HIsRHWxXXk/qn7MtmvcRtCl+6NEjM3w23D9CqNTB6uzQYR7CBZALDmPz3VVTuj4aPdgu q8zthwmQFd6hXg55UxgwWasmrbHliGp1m/j/VoFXmv1baSbN73+v6n6+L2YBpR2Q1qU3 YK8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=Hp23yxcEr25HscpkTDg71i1OPHKTlvstuNQTz096mYk=; b=sgEMc5sqHVyZQOQLwb+VvKi3+AQTssJ22X6tPvRpSOdmyHt1wu3Y2r/4mCSK8KgiR1 tY7txM9PHPdWB4g0xomKuK8MyLkjHJ6bunGHNR0b1ushyij/Roub5il2VZSG7/04VwTa 6himXl3UQi3fsQHFNWyL+Oh39goZx+Otk1D4AjM7NzX6q4DFEmKCPvtBMvPaMcaYnMA7 6N0mQSX1fMcOuggj3zQhncRPEXc9KxWsZM45oICkps3PwJNPhN9349OryFo9rISw5/2s tU3AKPETLGWBGpwj3a85uG3YgsaHgk725MpU4BNnn3sgd/KhlTtaBNLcnBjUULEtN6s9 pJDw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IZHnHCuD; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q13si14616343pgr.514.2019.08.21.07.33.30; Wed, 21 Aug 2019 07:33:30 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IZHnHCuD; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729313AbfHUOd3 (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:29 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:39106 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728763AbfHUOd3 (ORCPT ); Wed, 21 Aug 2019 10:33:29 -0400 Received: by mail-wr1-f68.google.com with SMTP id t16so2272424wra.6 for ; Wed, 21 Aug 2019 07:33:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Hp23yxcEr25HscpkTDg71i1OPHKTlvstuNQTz096mYk=; b=IZHnHCuDr0MnRcraaSf2WWBr1lyW3xoaETta+/qC9crDFRcCFKIEbFgj2Tax0to+l9 QU8tHDpB/N1+hT73pXihZ/H0V50hqJ1/ruOSWjT4k1T4TFuZ11Z35VppXgqFuIJSECxJ v4uLCkZgv1dBVS0TWs7VOsWxjrAwrs+bEfyErdMcV3FCxEY7rZox1fcRS5XQF6w+1PZa lrOMxk5EDL8mVykogB6NRZgedQYXhs4XgRaiLumem/CNsaAgXjBkeQBmGXxHIf7Qg6S6 HOJYK2w/VO0aPYqeMFaNotIpzQMs3BofyzzuS2UM/U1PraVTP3xIfVqOx442jY7OsZrz fElg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Hp23yxcEr25HscpkTDg71i1OPHKTlvstuNQTz096mYk=; b=euH395RVap6cUd+XIHS56N7FCMaLnPoy2Sx1J7XDDiL0oT2KY1X505FoSFQfFiUsry X/yUpR+Q4g/7v0GYwuRBS/bef3VKDAsxTACwdaGARPYPCUDTJ1thD6r/Hg4BwB+5X46q Cs41WoIpdJoLEerEPK8/h82tE9LYI6bvW9+lpkUVnakL76e5/rOqlAbRVFk9v3eaWPpQ +9thQaLGv4qVJ3S2U9SVEdEEQRuzzXN2GXoeTKlIm2wkuNqjt96AQvJVMiquwYvGE7z1 6OLyeNkRsSklCvstxtnBpBgL1KoYaV9gXG3ixik+lTUlTDxAsiVhl5rjq51XhDwpu07B gptw== X-Gm-Message-State: APjAAAW7ajgnMa8Jz5U29esxK2mp6oquLNVE3F4h1tUb0b6X6nM6Z3C+ gZ5weok4NE+mD/oZPZ0AUmRxwWFEkqwSog== X-Received: by 2002:adf:d4c6:: with SMTP id w6mr42019243wrk.98.1566398005615; Wed, 21 Aug 2019 07:33:25 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:24 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 15/17] crypto: arm/aes-ce - implement ciphertext stealing for CBC Date: Wed, 21 Aug 2019 17:32:51 +0300 Message-Id: <20190821143253.30209-16-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Instead of relying on the CTS template to wrap the accelerated CBC skcipher, implement the ciphertext stealing part directly. Signed-off-by: Ard Biesheuvel --- arch/arm/crypto/aes-ce-core.S | 85 +++++++++ arch/arm/crypto/aes-ce-glue.c | 188 ++++++++++++++++++-- 2 files changed, 256 insertions(+), 17 deletions(-) -- 2.17.1 diff --git a/arch/arm/crypto/aes-ce-core.S b/arch/arm/crypto/aes-ce-core.S index 763e51604ab6..b978cdf133af 100644 --- a/arch/arm/crypto/aes-ce-core.S +++ b/arch/arm/crypto/aes-ce-core.S @@ -284,6 +284,91 @@ ENTRY(ce_aes_cbc_decrypt) pop {r4-r6, pc} ENDPROC(ce_aes_cbc_decrypt) + + /* + * ce_aes_cbc_cts_encrypt(u8 out[], u8 const in[], u32 const rk[], + * int rounds, int bytes, u8 const iv[]) + * ce_aes_cbc_cts_decrypt(u8 out[], u8 const in[], u32 const rk[], + * int rounds, int bytes, u8 const iv[]) + */ + +ENTRY(ce_aes_cbc_cts_encrypt) + push {r4-r6, lr} + ldrd r4, r5, [sp, #16] + + movw ip, :lower16:.Lcts_permute_table + movt ip, :upper16:.Lcts_permute_table + sub r4, r4, #16 + add lr, ip, #32 + add ip, ip, r4 + sub lr, lr, r4 + vld1.8 {q5}, [ip] + vld1.8 {q6}, [lr] + + add ip, r1, r4 + vld1.8 {q0}, [r1] @ overlapping loads + vld1.8 {q3}, [ip] + + vld1.8 {q1}, [r5] @ get iv + prepare_key r2, r3 + + veor q0, q0, q1 @ xor with iv + bl aes_encrypt + + vtbl.8 d4, {d0-d1}, d10 + vtbl.8 d5, {d0-d1}, d11 + vtbl.8 d2, {d6-d7}, d12 + vtbl.8 d3, {d6-d7}, d13 + + veor q0, q0, q1 + bl aes_encrypt + + add r4, r0, r4 + vst1.8 {q2}, [r4] @ overlapping stores + vst1.8 {q0}, [r0] + + pop {r4-r6, pc} +ENDPROC(ce_aes_cbc_cts_encrypt) + +ENTRY(ce_aes_cbc_cts_decrypt) + push {r4-r6, lr} + ldrd r4, r5, [sp, #16] + + movw ip, :lower16:.Lcts_permute_table + movt ip, :upper16:.Lcts_permute_table + sub r4, r4, #16 + add lr, ip, #32 + add ip, ip, r4 + sub lr, lr, r4 + vld1.8 {q5}, [ip] + vld1.8 {q6}, [lr] + + add ip, r1, r4 + vld1.8 {q0}, [r1] @ overlapping loads + vld1.8 {q1}, [ip] + + vld1.8 {q3}, [r5] @ get iv + prepare_key r2, r3 + + bl aes_decrypt + + vtbl.8 d4, {d0-d1}, d10 + vtbl.8 d5, {d0-d1}, d11 + vtbx.8 d0, {d2-d3}, d12 + vtbx.8 d1, {d2-d3}, d13 + + veor q1, q1, q2 + bl aes_decrypt + veor q0, q0, q3 @ xor with iv + + add r4, r0, r4 + vst1.8 {q1}, [r4] @ overlapping stores + vst1.8 {q0}, [r0] + + pop {r4-r6, pc} +ENDPROC(ce_aes_cbc_cts_decrypt) + + /* * aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, * int blocks, u8 ctr[]) diff --git a/arch/arm/crypto/aes-ce-glue.c b/arch/arm/crypto/aes-ce-glue.c index c215792a2494..cdb1a07e7ad0 100644 --- a/arch/arm/crypto/aes-ce-glue.c +++ b/arch/arm/crypto/aes-ce-glue.c @@ -35,6 +35,10 @@ asmlinkage void ce_aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks, u8 iv[]); asmlinkage void ce_aes_cbc_decrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks, u8 iv[]); +asmlinkage void ce_aes_cbc_cts_encrypt(u8 out[], u8 const in[], u32 const rk[], + int rounds, int bytes, u8 const iv[]); +asmlinkage void ce_aes_cbc_cts_decrypt(u8 out[], u8 const in[], u32 const rk[], + int rounds, int bytes, u8 const iv[]); asmlinkage void ce_aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[], int rounds, int blocks, u8 ctr[]); @@ -210,48 +214,182 @@ static int ecb_decrypt(struct skcipher_request *req) return err; } -static int cbc_encrypt(struct skcipher_request *req) +static int cbc_encrypt_walk(struct skcipher_request *req, + struct skcipher_walk *walk) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); - struct skcipher_walk walk; unsigned int blocks; - int err; + int err = 0; - err = skcipher_walk_virt(&walk, req, false); - - while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { + while ((blocks = (walk->nbytes / AES_BLOCK_SIZE))) { kernel_neon_begin(); - ce_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ce_aes_cbc_encrypt(walk->dst.virt.addr, walk->src.virt.addr, ctx->key_enc, num_rounds(ctx), blocks, - walk.iv); + walk->iv); kernel_neon_end(); - err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); + err = skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; } -static int cbc_decrypt(struct skcipher_request *req) +static int cbc_encrypt(struct skcipher_request *req) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); struct skcipher_walk walk; - unsigned int blocks; int err; err = skcipher_walk_virt(&walk, req, false); + if (err) + return err; + return cbc_encrypt_walk(req, &walk); +} - while ((blocks = (walk.nbytes / AES_BLOCK_SIZE))) { +static int cbc_decrypt_walk(struct skcipher_request *req, + struct skcipher_walk *walk) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); + unsigned int blocks; + int err = 0; + + while ((blocks = (walk->nbytes / AES_BLOCK_SIZE))) { kernel_neon_begin(); - ce_aes_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ce_aes_cbc_decrypt(walk->dst.virt.addr, walk->src.virt.addr, ctx->key_dec, num_rounds(ctx), blocks, - walk.iv); + walk->iv); kernel_neon_end(); - err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); + err = skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; } +static int cbc_decrypt(struct skcipher_request *req) +{ + struct skcipher_walk walk; + int err; + + err = skcipher_walk_virt(&walk, req, false); + if (err) + return err; + return cbc_decrypt_walk(req, &walk); +} + +static int cts_cbc_encrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); + int cbc_blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; + struct scatterlist *src = req->src, *dst = req->dst; + struct scatterlist sg_src[2], sg_dst[2]; + struct skcipher_request subreq; + struct skcipher_walk walk; + int err; + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, skcipher_request_flags(req), + NULL, NULL); + + if (req->cryptlen <= AES_BLOCK_SIZE) { + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; + cbc_blocks = 1; + } + + if (cbc_blocks > 0) { + skcipher_request_set_crypt(&subreq, req->src, req->dst, + cbc_blocks * AES_BLOCK_SIZE, + req->iv); + + err = skcipher_walk_virt(&walk, &subreq, false) ?: + cbc_encrypt_walk(&subreq, &walk); + if (err) + return err; + + if (req->cryptlen == AES_BLOCK_SIZE) + return 0; + + dst = src = scatterwalk_ffwd(sg_src, req->src, subreq.cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, + subreq.cryptlen); + } + + /* handle ciphertext stealing */ + skcipher_request_set_crypt(&subreq, src, dst, + req->cryptlen - cbc_blocks * AES_BLOCK_SIZE, + req->iv); + + err = skcipher_walk_virt(&walk, &subreq, false); + if (err) + return err; + + kernel_neon_begin(); + ce_aes_cbc_cts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_enc, num_rounds(ctx), walk.nbytes, + walk.iv); + kernel_neon_end(); + + return skcipher_walk_done(&walk, 0); +} + +static int cts_cbc_decrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm); + int cbc_blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; + struct scatterlist *src = req->src, *dst = req->dst; + struct scatterlist sg_src[2], sg_dst[2]; + struct skcipher_request subreq; + struct skcipher_walk walk; + int err; + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, skcipher_request_flags(req), + NULL, NULL); + + if (req->cryptlen <= AES_BLOCK_SIZE) { + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; + cbc_blocks = 1; + } + + if (cbc_blocks > 0) { + skcipher_request_set_crypt(&subreq, req->src, req->dst, + cbc_blocks * AES_BLOCK_SIZE, + req->iv); + + err = skcipher_walk_virt(&walk, &subreq, false) ?: + cbc_decrypt_walk(&subreq, &walk); + if (err) + return err; + + if (req->cryptlen == AES_BLOCK_SIZE) + return 0; + + dst = src = scatterwalk_ffwd(sg_src, req->src, subreq.cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, + subreq.cryptlen); + } + + /* handle ciphertext stealing */ + skcipher_request_set_crypt(&subreq, src, dst, + req->cryptlen - cbc_blocks * AES_BLOCK_SIZE, + req->iv); + + err = skcipher_walk_virt(&walk, &subreq, false); + if (err) + return err; + + kernel_neon_begin(); + ce_aes_cbc_cts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_dec, num_rounds(ctx), walk.nbytes, + walk.iv); + kernel_neon_end(); + + return skcipher_walk_done(&walk, 0); +} + static int ctr_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); @@ -486,6 +624,22 @@ static struct skcipher_alg aes_algs[] = { { .setkey = ce_aes_setkey, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, +}, { + .base.cra_name = "__cts(cbc(aes))", + .base.cra_driver_name = "__cts-cbc-aes-ce", + .base.cra_priority = 300, + .base.cra_flags = CRYPTO_ALG_INTERNAL, + .base.cra_blocksize = AES_BLOCK_SIZE, + .base.cra_ctxsize = sizeof(struct crypto_aes_ctx), + .base.cra_module = THIS_MODULE, + + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .walksize = 2 * AES_BLOCK_SIZE, + .setkey = ce_aes_setkey, + .encrypt = cts_cbc_encrypt, + .decrypt = cts_cbc_decrypt, }, { .base.cra_name = "__ctr(aes)", .base.cra_driver_name = "__ctr-aes-ce", From patchwork Wed Aug 21 14:32:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171970 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055750ily; Wed, 21 Aug 2019 07:33:29 -0700 (PDT) X-Google-Smtp-Source: APXvYqzeAaYbfsbxpb8OxyV4p7MwPSi9V9UZ/K4svlN4HKpVqyKIYxx8CcsibSyCxAhRlBuZzpu1 X-Received: by 2002:a63:cb4f:: with SMTP id m15mr29238404pgi.100.1566398009814; Wed, 21 Aug 2019 07:33:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566398009; cv=none; d=google.com; s=arc-20160816; b=WngQSbZh2iyYREU26ZsaVdEzrZ7+OAzrXBSBguRomibXAVjFGYdpo2ftOZ002KZc8Z MpqSssDDK2bZxiwazHeQM7eKcrYq0GcS2/sQcwzu8JtnNENkQbkOy4s9u2N/7u4PNDhQ xuze+M5iKWtRmDQRmbKA7mFXSxJocESUq8nNvVtpNI6EqAfosv2PUd6eFZUhxeMIzkOU 9yGalWSWHOXs/gxxkRuwhj34UIS0J1T+LDShlF5tVhnB2Rn5trwVAsozXbVVN2KBbEXd 25ZuZN3WAecbdw4MY0YQHULVeo00PcdoP8/fS1VUkzB4m3EuMKlMoXzfLECoNlArSlTI f9cQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=AGMVgjDEG83Kdzuq7B1f+yZ7wOUASWkpWpxVlI6zo6E=; b=Gw2fbBN4cFBoMuLGWOFJC3c5Zvi9XNK3GVDMhN70aWkoJK0Q4r8VAb6JzAJzXQDVnA ZywQNTxbe8lDjj+a1bldNCL/eO/Og9rj50ivlP5PMLvEDE8Pq/h0XXP9L14CXVgv9TT1 ao0lgNqFEA3D8pI3fG4QaR1AMVPZjP0lqN/Kk9lRXbre+zJqhKhEfmkCoWQhQgWDcw7d g9y/wYeGskPX3Cu/dYo4NLHn6e6IPs2yHlAJA3gfAkjkeaHR5X9x/yp6F9upCDM4/9WQ 1FauPOgCOem3S3QaaUjlhTwwhathcqGRi/4p7Gs4LRXYFZYxdpLqLkJ50QoqXwG93hQ9 oVtg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=V0KpIxUN; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q13si14616343pgr.514.2019.08.21.07.33.29; Wed, 21 Aug 2019 07:33:29 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=V0KpIxUN; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729290AbfHUOd3 (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:29 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:33543 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729313AbfHUOd2 (ORCPT ); Wed, 21 Aug 2019 10:33:28 -0400 Received: by mail-wm1-f65.google.com with SMTP id p77so4916545wme.0 for ; Wed, 21 Aug 2019 07:33:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AGMVgjDEG83Kdzuq7B1f+yZ7wOUASWkpWpxVlI6zo6E=; b=V0KpIxUNnxq2SWesypTV5fLlGjhpvZUbWWoHb+hh3I5OT99iej3LuJEGVnC9US6MP+ hKgcQISjxem0S/3CkRl72YfQnW871dZOOrcztgNQhagMTyHTj9bX0THRTXdYDROswzDJ TC0iCpjDsmolTvckXNhOs1tAQh3NgWJd03EHcC3clhFbh/hcRrXkOrMr9iiJsMYaoA8C NIKGUvhOtDPB7ObZFapVS1M97f98jUijpIC5EJROMo8i0j97f8VNyDQpzyTbrCYJnmdC nL06VXat/F0rRhKAnAToC6hhlsaoBxu4A9O6qeuyTndcEAYzgkjEv4tdryF+49gciuJO HliA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AGMVgjDEG83Kdzuq7B1f+yZ7wOUASWkpWpxVlI6zo6E=; b=mJ42c4kh1ERRW41GCFzKVP6xuZPj2jyHe0OjN2VkwnJvySYpUg8dA6Vcpy50hM3ucr I92gmktbA0N61FYCiULC3MspwLNuoCKRRNA3i1SFcR6GFyj/NpumvR9RkO9C7HAKtKd2 wkSBI6fUHowuAHvf+0SaXq9xMtnkwt42ebkkHra5o0/hyfzYbagGBGXse43r721WVGQk zdG1hrAhmmd01O4hJpPolOmtdULGlEzI9L+MNB6GSA5c/iI+2EcN8RiPMa2zUZZlRuKX 2jnHFJxhFRAoDLDdSSf4Z/iovGXcRf/RiT+cIkzqew/2MQhXDDZh/NXTajDRfsjfwNj7 v4ew== X-Gm-Message-State: APjAAAXv+PoIxItZeVK1viH/13g4ByImz+y4fuVVsPS80HuImryQPesu 6Mf5MR/+YBBCIRHf6EJM7swMjylhcC9PIQ== X-Received: by 2002:a1c:18a:: with SMTP id 132mr375595wmb.15.1566398007331; Wed, 21 Aug 2019 07:33:27 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:26 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Subject: [PATCH 16/17] crypto: testmgr - add test vectors for XTS ciphertext stealing Date: Wed, 21 Aug 2019 17:32:52 +0300 Message-Id: <20190821143253.30209-17-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Import the AES-XTS test vectors from IEEE publication P1619/D16 that exercise the ciphertext stealing part of the XTS algorithm, which we haven't supported in the Linux kernel implementation up till now. Tested-by: Pascal van Leeuwen Signed-off-by: Ard Biesheuvel --- crypto/testmgr.h | 60 ++++++++++++++++++++ 1 file changed, 60 insertions(+) -- 2.17.1 diff --git a/crypto/testmgr.h b/crypto/testmgr.h index 154052d07818..b88a1ba87b58 100644 --- a/crypto/testmgr.h +++ b/crypto/testmgr.h @@ -15291,6 +15291,66 @@ static const struct cipher_testvec aes_xts_tv_template[] = { "\xc4\xf3\x6f\xfd\xa9\xfc\xea\x70" "\xb9\xc6\xe6\x93\xe1\x48\xc1\x51", .len = 512, + }, { /* XTS-AES 15 */ + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\xbf\xbe\xbd\xbc\xbb\xba\xb9\xb8" + "\xb7\xb6\xb5\xb4\xb3\xb2\xb1\xb0", + .klen = 32, + .iv = "\x9a\x78\x56\x34\x12\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10", + .ctext = "\x6c\x16\x25\xdb\x46\x71\x52\x2d" + "\x3d\x75\x99\x60\x1d\xe7\xca\x09" + "\xed", + .len = 17, + }, { /* XTS-AES 16 */ + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\xbf\xbe\xbd\xbc\xbb\xba\xb9\xb8" + "\xb7\xb6\xb5\xb4\xb3\xb2\xb1\xb0", + .klen = 32, + .iv = "\x9a\x78\x56\x34\x12\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11", + .ctext = "\xd0\x69\x44\x4b\x7a\x7e\x0c\xab" + "\x09\xe2\x44\x47\xd2\x4d\xeb\x1f" + "\xed\xbf", + .len = 18, + }, { /* XTS-AES 17 */ + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\xbf\xbe\xbd\xbc\xbb\xba\xb9\xb8" + "\xb7\xb6\xb5\xb4\xb3\xb2\xb1\xb0", + .klen = 32, + .iv = "\x9a\x78\x56\x34\x12\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12", + .ctext = "\xe5\xdf\x13\x51\xc0\x54\x4b\xa1" + "\x35\x0b\x33\x63\xcd\x8e\xf4\xbe" + "\xed\xbf\x9d", + .len = 19, + }, { /* XTS-AES 18 */ + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\xbf\xbe\xbd\xbc\xbb\xba\xb9\xb8" + "\xb7\xb6\xb5\xb4\xb3\xb2\xb1\xb0", + .klen = 32, + .iv = "\x9a\x78\x56\x34\x12\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13", + .ctext = "\x9d\x84\xc8\x13\xf7\x19\xaa\x2c" + "\x7b\xe3\xf6\x61\x71\xc7\xc5\xc2" + "\xed\xbf\x9d\xac", + .len = 20, } }; From patchwork Wed Aug 21 14:32:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 171972 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp1055880ily; Wed, 21 Aug 2019 07:33:34 -0700 (PDT) X-Google-Smtp-Source: APXvYqwyrqN1Fjybt3JJ1ijlqAELtv6cG2HnPaucno0EXoF4FNI+7wDtAf9KplrnD4EoTK+JPmLT X-Received: by 2002:a65:52c5:: with SMTP id z5mr29608136pgp.118.1566398014317; Wed, 21 Aug 2019 07:33:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566398014; cv=none; d=google.com; s=arc-20160816; b=HPFl8qB1u3YAltClBGquD+sA2VsdxlxAr1/Rvv9ACuWx7IED2UVfGCVhHXqAiZUkq1 gn8TxSqG9guMFzyO/82f4ivUWjART/TDfYjGiV39YbSyIXKnoeII4sh1SVmwAjTDe6Hk 7gE0YonoYAF7TB6kwySM/fuOsmVM6OtWQG2K1Thy8ff6zg4bwuzL+z0aizwvR3vrgV4b S8xSJpOFJZcXrhGG9pvc/ZFFAlzSuHNjWkAN6rme6vaqX0wkfAGebvXGQqqct4h9aASW 0EeW+R4WtYO5BgMZl08Egb6q5+jtYSnPMkFWNZnnTkSpmCoT11g+9jUjYm4iInErpNCS buJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=sC43Om6eeDsRfZHSH+k26Uj2fDnNk/GMWHykpA6ile0=; b=rNuRlNb8Cdt+Sbq12tBAcV5ahTCsr0DRkQVZwJwudkFxzH9jMi1Rk51jFAyf8eRlRY 8kamliIAvv1W2YtLvjOdh61sHc3y1YqYzPiF8voDv95tVy8zqCacjVfYbJ+aU5JMU8yB UtQcJtq+QMxWQ1zjbDOKihacKR1nZR2PNldduqA0rhE1BPckao1pLYTdTH6sKPwqGk1w +bDig2Bu+Gx0fg86RaMnu6R4BfSaNOGThHPf4aFJc2xWRZYfboQzAWVD0wWaNRt3BQn2 ntkTi/4qK4txx0QVTLIkujPW8KYDFv4PBraEKkcZ4cGybvT/5IJEyXuY8O24o4LWtGpJ Cu2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KHIyiJn3; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q13si14616343pgr.514.2019.08.21.07.33.34; Wed, 21 Aug 2019 07:33:34 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KHIyiJn3; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729318AbfHUOdd (ORCPT + 3 others); Wed, 21 Aug 2019 10:33:33 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:38026 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728763AbfHUOdd (ORCPT ); Wed, 21 Aug 2019 10:33:33 -0400 Received: by mail-wm1-f68.google.com with SMTP id m125so2374422wmm.3 for ; Wed, 21 Aug 2019 07:33:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sC43Om6eeDsRfZHSH+k26Uj2fDnNk/GMWHykpA6ile0=; b=KHIyiJn3OtTRIQvm1PYKU/9kDRYLCxV2hveKRFmLfNkGD5bO7sbwOhpfv/BB9DtMLK m1mPaG4jbae6HTYi4I7lzFpPPS1JNgSy7EaU4KsezWT0ys34L3JVfgytJgTmsxHjFkaM 1n+EPl9Zf1G7pzB8M/wGmOrH4MLeCy0b+y6rJzAszfAQk5yttEaULGJx9Il8AT8IVgdM McFI1njQs6FAzjr/21NjR86yU5Gt4iE06GWSdsksAqMCYvImIHH0dezH0etMccLfRAcp unkaql93n7bkCaTywHV5vZXpvvvUKSr8x2c+H7KgtmsVu6HgkFkY/1sgkRgR4AtcvypM johg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sC43Om6eeDsRfZHSH+k26Uj2fDnNk/GMWHykpA6ile0=; b=Hu0aReWhDdkmTu8qcGkWn9aUq6XJc+JPUpY1zDCwNNnMCde1rpaiEgvMLtfcn9RYmn b0cV8OXSlomfO65a3DGu9WaknANIiY+hshrRGbvp/7Hzup3kQaH3JI0PxQNkt/2Lg20x 3Iw+MfbcOks/biMdPsxVBNFG3uHakacgSiv8E042wAXfbqJYKIclyghd++aBDNlrteSE g9ujSBYRT2lL287P1zlgw0WIGmbVGtg23Z//5umMCpb2bfSCp2HSTgQ/FyevKk8yXdfG QRPuG6SVwgpNDZV60XgIm67xmD4ArAzKalfvdsGsjGe0fNMFljVxpnP45zDUj7mhImhh LP5Q== X-Gm-Message-State: APjAAAVFZggYMFJSXHlonJ/pzBsQwlg7fvIdeFR2tMk37wh7sOx+pMOc MmBZEzT2nhMcZFq2CxEwsSzLmXkI9zbzdg== X-Received: by 2002:a05:600c:228e:: with SMTP id 14mr362429wmf.101.1566398009231; Wed, 21 Aug 2019 07:33:29 -0700 (PDT) Received: from mba13.lan (adsl-103.109.242.1.tellas.gr. [109.242.1.103]) by smtp.gmail.com with ESMTPSA id 16sm181427wmx.45.2019.08.21.07.33.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2019 07:33:28 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, ebiggers@kernel.org, Pascal van Leeuwen , Ard Biesheuvel Subject: [PATCH 17/17] crypto: testmgr - Add additional AES-XTS vectors for covering CTS Date: Wed, 21 Aug 2019 17:32:53 +0300 Message-Id: <20190821143253.30209-18-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190821143253.30209-1-ard.biesheuvel@linaro.org> References: <20190821143253.30209-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Pascal van Leeuwen This patch adds test vectors for AES-XTS that cover data inputs that are not a multiple of 16 bytes and therefore require cipher text stealing (CTS) to be applied. Vectors were added to cover all possible alignments combined with various interesting (i.e. for vector implementations working on 3,4,5 or 8 AES blocks in parallel) lengths. This code was kindly donated to the public domain by the author. Link: https://lore.kernel.org/linux-crypto/MN2PR20MB29739591E1A3E54E7A8A8E18CAC00@MN2PR20MB2973.namprd20.prod.outlook.com/ Signed-off-by: Ard Biesheuvel --- crypto/testmgr.h | 308 ++++++++++++++++++++ 1 file changed, 308 insertions(+) -- 2.17.1 diff --git a/crypto/testmgr.h b/crypto/testmgr.h index b88a1ba87b58..717b9fcb9bfa 100644 --- a/crypto/testmgr.h +++ b/crypto/testmgr.h @@ -15351,6 +15351,314 @@ static const struct cipher_testvec aes_xts_tv_template[] = { "\x7b\xe3\xf6\x61\x71\xc7\xc5\xc2" "\xed\xbf\x9d\xac", .len = 20, + /* Additional vectors to increase CTS coverage */ + }, { /* 1 block + 21 bytes */ + .key = "\xa1\x34\x0e\x49\x38\xfd\x8b\xf6" + "\x45\x60\x67\x07\x0f\x50\xa8\x2b" + "\xa8\xf1\xfe\x7e\xf4\xf0\x47\xcd" + "\xfd\x91\x78\xf9\x14\x8b\x7d\x27" + "\x0e\xdc\xca\xe6\xf4\xfc\xd7\x4f" + "\x19\x8c\xd0\xe6\x9e\x2f\xf8\x75" + "\xb5\xe2\x48\x00\x4f\x07\xd9\xa1" + "\x42\xbc\x9d\xfc\x17\x98\x00\x48", + .klen = 64, + .iv = "\xcb\x35\x47\x5a\x7a\x06\x28\xb9" + "\x80\xf5\xa7\xe6\x8a\x23\x42\xf8", + .ptext = "\x04\x52\xc8\x7f\xb0\x5a\x12\xc5" + "\x96\x47\x6b\xf4\xbc\x2e\xdb\x74" + "\xd2\x20\x24\x32\xe5\x84\xb6\x25" + "\x4c\x2f\x96\xc7\x55\x9c\x90\x6f" + "\x0e\x96\x94\x68\xf4", + .ctext = "\x6a\x2d\x57\xb8\x72\x49\x10\x6b" + "\x5b\x5a\xc9\x92\xab\x59\x79\x36" + "\x7a\x01\x95\xf7\xdd\xcb\x3f\xbf" + "\xb2\xe3\x7e\x35\xe3\x11\x04\x68" + "\x28\xc3\x70\x6a\xe1", + .len = 37, + }, { /* 3 blocks + 22 bytes */ + .key = "\xf7\x87\x75\xdf\x36\x20\xe7\xcb" + "\x20\x5d\x49\x96\x81\x3d\x1d\x80" + "\xc7\x18\x7e\xbf\x2a\x0f\x79\xba" + "\x06\xb5\x4b\x63\x03\xfb\xb8\x49" + "\x93\x2d\x85\x5b\x95\x1f\x78\xea" + "\x7c\x1e\xf5\x5d\x02\xc6\xec\xb0" + "\xf0\xaa\x3d\x0a\x04\xe1\x67\x80" + "\x2a\xbe\x4e\x73\xc9\x11\xcc\x6c", + .klen = 64, + .iv = "\xeb\xba\x55\x24\xfc\x8f\x25\x7c" + "\x66\xf9\x04\x03\xcc\xb1\xf4\x84", + .ptext = "\x40\x75\x1b\x72\x2a\xc8\xbf\xef" + "\x0c\x92\x3e\x19\xc5\x09\x07\x38" + "\x4d\x87\x5c\xb8\xd6\x4f\x1a\x39" + "\x8c\xee\xa5\x22\x41\x12\xe1\x22" + "\xb5\x4b\xd7\xeb\x02\xfa\xaa\xf8" + "\x94\x47\x04\x5d\x8a\xb5\x40\x12" + "\x04\x62\x3d\xe4\x19\x8a\xeb\xb3" + "\xf9\xa3\x7d\xb6\xeb\x57\xf9\xb8" + "\x7f\xa8\xfa\x2d\x75\x2d", + .ctext = "\x46\x6d\xe5\x35\x5d\x22\x42\x33" + "\xf7\xb8\xfb\xc0\xcb\x18\xad\xa4" + "\x75\x6c\xc6\x38\xbb\xd4\xa1\x32" + "\x00\x05\x06\xd9\xc9\x17\xd9\x4f" + "\x1a\xf6\x24\x64\x27\x8a\x4a\xad" + "\x88\xa0\x86\xb7\xf9\x33\xaf\xa8" + "\x0e\x83\xd8\x0e\x88\xa2\x81\x79" + "\x65\x2e\x3e\x84\xaf\xa1\x46\x7d" + "\xa9\x91\xf8\x17\x82\x8d", + .len = 70, + }, { /* 4 blocks + 23 bytes */ + .key = "\x48\x09\xab\x48\xd6\xca\x7d\xb1" + "\x90\xa0\x00\xd8\x33\x8a\x20\x79" + "\x7c\xbc\x0c\x0c\x5f\x41\xbc\xbc" + "\x82\xaf\x41\x81\x23\x93\xcb\xc7" + "\x61\x7b\x83\x13\x16\xb1\x3e\x7c" + "\xcc\xae\xda\xca\x78\xc7\xab\x18" + "\x69\xb6\x58\x3e\x5c\x19\x5f\xed" + "\x7b\xcf\x70\xb9\x76\x00\xd8\xc9", + .klen = 64, + .iv = "\x2e\x20\x36\xf4\xa3\x22\x5d\xd8" + "\x38\x49\x82\xbf\x6c\x56\xd9\x3b", + .ptext = "\x79\x3c\x73\x99\x65\x21\xe1\xb9" + "\xa0\xfd\x22\xb2\x57\xc0\x7f\xf4" + "\x7f\x97\x36\xaf\xf8\x8d\x73\xe1" + "\x0d\x85\xe9\xd5\x3d\x82\xb3\x49" + "\x89\x25\x30\x1f\x0d\xca\x5c\x95" + "\x64\x31\x02\x17\x11\x08\x8f\x32" + "\xbc\x37\x23\x4f\x03\x98\x91\x4a" + "\x50\xe2\x58\xa8\x9b\x64\x09\xe0" + "\xce\x99\xc9\xb0\xa8\x21\x73\xb7" + "\x2d\x4b\x19\xba\x81\x83\x99\xce" + "\xa0\x7a\xd0\x9f\x27\xf6\x8a", + .ctext = "\xf9\x12\x76\x21\x06\x1e\xe4\x4b" + "\xf9\x94\x38\x29\x0f\xee\xcb\x13" + "\xa3\xc3\x50\xe3\xc6\x29\x9d\xcf" + "\x6f\x6a\x0a\x25\xab\x44\xf6\xe4" + "\x71\x29\x75\x3b\x07\x1c\xfc\x1a" + "\x75\xd4\x84\x58\x7f\xc4\xf3\xf7" + "\x8f\x7c\x7a\xdc\xa2\xa3\x95\x38" + "\x15\xdf\x3b\x9c\xdd\x24\xb4\x0b" + "\xa8\x97\xfa\x5f\xee\x58\x00\x0d" + "\x23\xc9\x8d\xee\xc2\x3f\x27\xd8" + "\xd4\x43\xa5\xf8\x25\x71\x3f", + .len = 87, + }, { /* 5 blocks + 24 bytes */ + .key = "\x8c\xf4\x4c\xe5\x91\x8f\x72\xe9" + "\x2f\xf8\xc0\x3c\x87\x76\x16\xa4" + "\x20\xab\x66\x39\x34\x10\xd6\x91" + "\xf1\x99\x2c\xf1\xd6\xc3\xda\x38" + "\xed\x2a\x4c\x80\xf4\xa5\x56\x28" + "\x1a\x1c\x79\x72\x6c\x93\x08\x86" + "\x8f\x8a\xaa\xcd\xf1\x8c\xca\xe7" + "\x0a\xe8\xee\x0c\x1c\xc2\xa8\xea", + .klen = 64, + .iv = "\x9a\x9e\xbc\xe4\xc9\xf3\xef\x9f" + "\xff\x82\x0e\x22\x8f\x80\x42\x76", + .ptext = "\xc1\xde\x66\x1a\x7e\x60\xd3\x3b" + "\x66\xd6\x29\x86\x99\xc6\xd7\xc8" + "\x29\xbf\x00\x57\xab\x21\x06\x24" + "\xd0\x92\xef\xe6\xb5\x1e\x20\xb9" + "\xb7\x7b\xd7\x18\x88\xf8\xd7\xe3" + "\x90\x61\xcd\x73\x2b\xa1\xb5\xc7" + "\x33\xef\xb5\xf2\x45\xf6\x92\x53" + "\x91\x98\xf8\x5a\x20\x75\x4c\xa8" + "\xf1\xf6\x01\x26\xbc\xba\x4c\xac" + "\xcb\xc2\x6d\xb6\x2c\x3c\x38\x61" + "\xe3\x98\x7f\x3e\x98\xbd\xec\xce" + "\xc0\xb5\x74\x23\x43\x24\x7b\x7e" + "\x3f\xed\xcb\xda\x88\x67\x6f\x9a", + .ctext = "\xeb\xdc\x6a\xb7\xd9\x5f\xa7\xfc" + "\x48\x75\x10\xef\xca\x65\xdc\x88" + "\xd0\x23\xde\x17\x5f\x3b\x61\xa2" + "\x15\x13\x81\x81\xf8\x57\x8b\x2a" + "\xe2\xc8\x49\xd1\xba\xed\xd6\xcb" + "\xed\x6f\x26\x69\x9b\xd2\xd2\x91" + "\x4e\xd7\x81\x20\x66\x38\x0c\x62" + "\x60\xcd\x01\x36\x97\x22\xf0\x5c" + "\xcf\x53\xc6\x58\xf5\x8b\x48\x0c" + "\xa5\x50\xc2\x73\xf9\x70\x60\x09" + "\x22\x69\xf3\x71\x74\x5d\xc9\xa0" + "\x9c\x79\xf9\xc4\x87\xac\xd7\x4b" + "\xac\x3c\xc6\xda\x81\x7a\xdd\x14", + .len = 104, + }, { /* 8 blocks + 25 bytes */ + .key = "\x70\x18\x09\x93\x10\x3a\x0c\xa9" + "\x02\x0b\x11\x10\xae\x34\x98\xdb" + "\x10\xb5\xee\x8c\x49\xbc\x52\x8e" + "\x4b\xf7\x0a\x36\x16\x8a\xf7\x06" + "\xb5\x94\x52\x54\xb9\xc1\x4d\x20" + "\xa2\xf0\x6e\x19\x7f\x67\x1e\xaa" + "\x94\x6c\xee\x54\x19\xfc\x96\x95" + "\x04\x85\x00\x53\x7c\x39\x5f\xeb", + .klen = 64, + .iv = "\x36\x87\x8f\x9d\x74\xe9\x52\xfb" + "\xe1\x76\x16\x99\x61\x86\xec\x8f", + .ptext = "\x95\x08\xee\xfe\x87\xb2\x4f\x93" + "\x01\xee\xf3\x77\x0d\xbb\xfb\x26" + "\x3e\xb3\x34\x20\xee\x51\xd6\x40" + "\xb1\x64\xae\xd9\xfd\x71\x8f\x93" + "\xa5\x85\xff\x74\xcc\xd3\xfd\x5e" + "\xc2\xfc\x49\xda\xa8\x3a\x94\x29" + "\xa2\x59\x90\x34\x26\xbb\xa0\x34" + "\x5d\x47\x33\xf2\xa8\x77\x90\x98" + "\x8d\xfd\x38\x60\x23\x1e\x50\xa1" + "\x67\x4d\x8d\x09\xe0\x7d\x30\xe3" + "\xdd\x39\x91\xd4\x70\x68\xbb\x06" + "\x4e\x11\xb2\x26\x0a\x85\x73\xf6" + "\x37\xb6\x15\xd0\x77\xee\x43\x7b" + "\x77\x13\xe9\xb9\x84\x2b\x34\xab" + "\x49\xc1\x27\x91\x2e\xa3\xca\xe5" + "\xa7\x79\x45\xba\x36\x97\x49\x44" + "\xf7\x57\x9b\xd7\xac\xb3\xfd\x6a" + "\x1c\xd1\xfc\x1c\xdf\x6f\x94\xac" + "\x95\xf4\x50\x7a\xc8\xc3\x8c\x60" + "\x3c", + .ctext = "\xb6\xc8\xf9\x5d\x35\x5a\x0a\x33" + "\x2b\xd3\x5a\x18\x09\x1c\x1b\x0b" + "\x2a\x0e\xde\xf6\x0d\x04\xa6\xb3" + "\xa8\xe8\x1b\x86\x29\x58\x75\x56" + "\xab\xab\xbf\xbe\x1f\xb4\xc4\xf3" + "\xde\x1a\xb0\x87\x69\xac\x5b\x0c" + "\x1b\xb7\xc7\x24\xa4\x47\xe7\x81" + "\x2c\x0a\x82\xf9\x18\x5d\xe6\x09" + "\xe3\x65\x36\x54\x3d\x8a\x3a\x64" + "\x34\xf4\x34\x7f\x26\x3c\x1e\x3b" + "\x5a\x13\xdf\x7f\xa8\x2d\x81\xce" + "\xfa\xad\xd0\xb1\xca\xfa\xc3\x55" + "\x94\xc8\xb8\x16\x7e\xff\x44\x88" + "\xb4\x47\x4b\xfe\xda\x60\x68\x2e" + "\xfc\x70\xb5\xe3\xf3\xe9\x46\x22" + "\x1d\x98\x66\x09\x0f\xed\xbb\x20" + "\x7b\x8c\x2a\xff\x45\x62\xde\x9b" + "\x20\x2e\x6c\xb4\xe4\x26\x03\x72" + "\x8a\xb4\x19\xc9\xb1\xcf\x9d\x86" + "\xa3", + .len = 153, + }, { /* 0 blocks + 26 bytes */ + .key = "\x5a\x38\x3f\x9c\x0c\x53\x17\x6c" + "\x60\x72\x23\x26\xba\xfe\xa1\xb7" + "\x03\xa8\xfe\xa0\x7c\xff\x78\x4c" + "\x7d\x84\x2f\x24\x84\x77\xec\x6f" + "\x88\xc8\x36\xe2\xcb\x52\x3c\xb4" + "\x39\xac\x37\xfa\x41\x8b\xc4\x59" + "\x24\x03\xe1\x51\xc9\x54\x7d\xb7" + "\xa3\xde\x91\x44\x8d\x16\x97\x22", + .klen = 64, + .iv = "\xfb\x7f\x3d\x60\x26\x0a\x3a\x3d" + "\xa5\xa3\x45\xf2\x24\x67\xfa\x6e", + .ptext = "\xfb\x56\x97\x65\x7c\xd8\x6c\x3c" + "\x5d\xd3\xea\xa6\xa4\x83\xf7\x9d" + "\x9d\x89\x2c\x85\xb8\xd9\xd4\xf0" + "\x1a\xad", + .ctext = "\xc9\x9b\x4b\xf2\xf7\x0f\x23\xfe" + "\xc3\x93\x88\xa1\xb3\x88\xab\xd6" + "\x26\x78\x82\xa6\x6b\x0b\x76\xad" + "\x21\x5e", + .len = 26, + }, { /* 0 blocks + 27 bytes */ + .key = "\xc0\xcf\x57\xa2\x3c\xa2\x4b\xf6" + "\x5d\x36\x7b\xd7\x1d\x16\xc3\x2f" + "\x50\xc6\x0a\xb2\xfd\xe8\x24\xfc" + "\x33\xcf\x73\xfd\xe0\xe9\xa5\xd1" + "\x98\xfc\xd6\x16\xdd\xfd\x6d\xab" + "\x44\xbc\x37\x9d\xab\x5b\x1d\xf2" + "\x6f\x5d\xbe\x6b\x14\x14\xc7\x74" + "\xbb\x91\x24\x4b\x52\xcb\x78\x31", + .klen = 64, + .iv = "\x5c\xc1\x3d\xb6\xa1\x6a\x2d\x1f" + "\xee\x75\x19\x4b\x04\xfa\xe1\x7e", + .ptext = "\x02\x95\x3a\xab\xac\x3b\xcd\xcd" + "\x63\xc7\x4c\x7c\xe5\x75\xee\x03" + "\x94\xc7\xff\xe8\xe0\xe9\x86\x2a" + "\xd3\xc7\xe4", + .ctext = "\x8e\x84\x76\x8b\xc1\x47\x55\x15" + "\x5e\x51\xb3\xe2\x3f\x72\x4d\x20" + "\x09\x3f\x4f\xb1\xce\xf4\xb0\x14" + "\xf6\xa7\xb3", + .len = 27, + }, { /* 0 blocks + 28 bytes */ + .key = "\x0b\x5b\x1d\xc8\xb1\x3f\x8f\xcd" + "\x87\xd2\x58\x28\x36\xc6\x34\xfb" + "\x04\xe8\xf1\xb7\x91\x30\xda\x75" + "\x66\x4a\x72\x90\x09\x39\x02\x19" + "\x62\x2d\xe9\x24\x95\x0e\x87\x43" + "\x4c\xc7\x96\xe4\xc9\x31\x6a\x13" + "\x16\x10\xef\x34\x9b\x98\x19\xf1" + "\x8b\x14\x38\x3f\xf8\x75\xcc\x76", + .klen = 64, + .iv = "\x0c\x2c\x55\x2c\xda\x40\xe1\xab" + "\xa6\x34\x66\x7a\xa4\xa3\xda\x90", + .ptext = "\xbe\x84\xd3\xfe\xe6\xb4\x29\x67" + "\xfd\x29\x78\x41\x3d\xe9\x81\x4e" + "\x3c\xf9\xf4\xf5\x3f\xd8\x0e\xcd" + "\x63\x73\x65\xf3", + .ctext = "\xd0\xa0\x16\x5f\xf9\x85\xd0\x63" + "\x9b\x81\xa1\x15\x93\xb3\x62\x36" + "\xec\x93\x0e\x14\x07\xf2\xa9\x38" + "\x80\x33\xc0\x20", + .len = 28, + }, { /* 0 blocks + 29 bytes */ + .key = "\xdc\x4c\xdc\x20\xb1\x34\x89\xa4" + "\xd0\xb6\x77\x05\xea\x0c\xcc\x68" + "\xb1\xd6\xf7\xfd\xa7\x0a\x5b\x81" + "\x2d\x4d\xa3\x65\xd0\xab\xa1\x02" + "\x85\x4b\x33\xea\x51\x16\x50\x12" + "\x3b\x25\xba\x13\xba\x7c\xbb\x3a" + "\xe4\xfd\xb3\x9c\x88\x8b\xb8\x30" + "\x7a\x97\xcf\x95\x5d\x69\x7b\x1d", + .klen = 64, + .iv = "\xe7\x69\xed\xd2\x54\x5d\x4a\x29" + "\xb2\xd7\x60\x90\xa0\x0b\x0d\x3a", + .ptext = "\x37\x22\x11\x62\xa0\x74\x92\x62" + "\x40\x4e\x2b\x0a\x8b\xab\xd8\x28" + "\x8a\xd2\xeb\xa5\x8e\xe1\x42\xc8" + "\x49\xef\x9a\xec\x1b", + .ctext = "\x7c\x66\x72\x6b\xe3\xc3\x57\x71" + "\x37\x13\xce\x1f\x6b\xff\x13\x87" + "\x65\xa7\xa1\xc5\x23\x7f\xca\x40" + "\x82\xbf\x2f\xc0\x2a", + .len = 29, + }, { /* 0 blocks + 30 bytes */ + .key = "\x72\x9a\xf5\x53\x55\xdd\x0f\xef" + "\xfc\x75\x6f\x03\x88\xc8\xba\x88" + "\xb7\x65\x89\x5d\x03\x86\x21\x22" + "\xb8\x42\x87\xd9\xa9\x83\x9e\x9c" + "\xca\x28\xa1\xd2\xb6\xd0\xa6\x6c" + "\xf8\x57\x42\x7c\x73\xfc\x7b\x0a" + "\xbc\x3c\x57\x7b\x5a\x39\x61\x55" + "\xb7\x25\xe9\xf1\xc4\xbb\x04\x28", + .klen = 64, + .iv = "\x8a\x38\x22\xba\xea\x5e\x1d\xa4" + "\x31\x18\x12\x5c\x56\x0c\x12\x50", + .ptext = "\x06\xfd\xbb\xa9\x2e\x56\x05\x5f" + "\xf2\xa7\x36\x76\x26\xd3\xb3\x49" + "\x7c\xe2\xe3\xbe\x1f\x65\xd2\x17" + "\x65\xe2\xb3\x0e\xb1\x93", + .ctext = "\xae\x1f\x19\x7e\x3b\xb3\x65\xcb" + "\x14\x70\x6b\x3c\xa0\x63\x95\x94" + "\x56\x52\xe1\xb4\x14\xca\x21\x13" + "\xb5\x03\x3f\xfe\xc9\x9f", + .len = 30, + }, { /* 0 blocks + 31 bytes */ + .key = "\xce\x06\x45\x53\x25\x81\xd2\xb2" + "\xdd\xc9\x57\xfe\xbb\xf6\x83\x07" + "\x28\xd8\x2a\xff\x53\xf8\x57\xc6" + "\x63\x50\xd4\x3e\x2a\x54\x37\x51" + "\x07\x3b\x23\x63\x3c\x31\x57\x0d" + "\xd3\x59\x20\xf2\xd0\x85\xac\xc5" + "\x3f\xa1\x74\x90\x0a\x3f\xf4\x10" + "\x12\xf0\x1b\x2b\xef\xcb\x86\x74", + .klen = 64, + .iv = "\x6d\x3e\x62\x94\x75\x43\x74\xea" + "\xed\x4a\xa6\xde\xba\x55\x83\x38", + .ptext = "\x6a\xe6\xa3\x66\x7e\x78\xef\x42" + "\x8b\x28\x08\x24\xda\xd4\xd6\x42" + "\x3d\xb6\x48\x7e\x51\xa6\x92\x65" + "\x98\x86\x26\x98\x37\x42\xa5", + .ctext = "\x64\xc6\xfc\x60\x21\x87\x7a\xf5" + "\xc3\x1d\xba\x41\x3c\x9c\x8c\xe8" + "\x2d\x93\xf0\x02\x95\x6d\xfe\x8d" + "\x68\x17\x05\x75\xc0\xd3\xa8", + .len = 31, } };