From patchwork Thu Oct 13 18:40:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Markus Stockhausen X-Patchwork-Id: 615821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 654A7C4332F for ; Thu, 13 Oct 2022 19:12:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229771AbiJMTMI (ORCPT ); Thu, 13 Oct 2022 15:12:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230078AbiJMTLu (ORCPT ); Thu, 13 Oct 2022 15:11:50 -0400 Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 710C9E4E5E for ; Thu, 13 Oct 2022 12:11:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1665688306; bh=vMnMvN2Vormu2vCNvhs5YyBOVabvMXVZ700gLozC4kQ=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date:In-Reply-To:References; b=PBVVImJ2SPsKu3K2n37MF+scFMXpnXb6Fsqs07GTaeBmpk2KAwdVmV4mKXkc1Q10L wEZohrk7a8xooxRz+4dPHDF0i3R1hYa9LuLMQvb6snNUd39UYw5BVv3yxYFOVgdcJC RuQp9EDPDBQFuFUjXCVC9xB3ekgVIq2Hhp2inc74= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from fedora.willemsstb.de ([94.31.86.22]) by mail.gmx.net (mrgmx105 [212.227.17.174]) with ESMTPSA (Nemesis) id 1Mqb1c-1pVbIb45Ro-00mZ5Z; Thu, 13 Oct 2022 20:40:30 +0200 From: Markus Stockhausen To: linux-crypto@vger.kernel.org Cc: Markus Stockhausen Subject: [PATCH 4/6] crypto/realtek: skcipher algorithms Date: Thu, 13 Oct 2022 20:40:24 +0200 Message-Id: <20221013184026.63826-5-markus.stockhausen@gmx.de> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221013184026.63826-1-markus.stockhausen@gmx.de> References: <20221013184026.63826-1-markus.stockhausen@gmx.de> MIME-Version: 1.0 X-Provags-ID: V03:K1:XIUA76as5s1cvVn6eFdlCZXpqlrL/PP/F87PWSz3ajQhILeIZec tlKMdqUe+iR1MK55PICJeaeS92zHqRyjv4LTwdfGwmninDQRk2JDu1N80a9w7aVsv5gJbrv pEpgj3ZQ6aukppuqh4VQpJ9YFLcKxdhyERAq8HfYNy9Ohxu+H6Vv62Zj7qNbSFTIMiIQ1lo PwcZtJpPzMrch0zlCqIuA== X-UI-Out-Filterresults: notjunk:1;V03:K0:W3Mlv0hDdEo=:lobCYeDYgZXvnRrwDpS39S VK1xl4ppM8prn54irghqfn630hAnoPBttnfZvuyJWDmffLzqwfYpG3hwp7KqVB4l2l3EFz2Q7 ImgvCQFYKhRTpEsStOe+pKUjK8Yzut+cKJERMrZJB/kvhbfNLyD5s9Bo5xQ4EPe9//HbVMKov v7HNSmu7T14CBvVQHxq16LNNNqE8vElo0R8mc4doNgeMiFUNiZ+Pjc32gThHXyXWjix+l17dN DLAqEuAlqxjEtcuU4ezZnAxF5irdnKU9q3F6JI+S8s/QYnGlw+16iAknxFePHIsqzZHR7F5aE CSZlfcMXO+JTrx+ZPMe2xlY+4vL4ErFFDYNq1PfTgn3oRPHfLtGXAS+qU4o03BkuiHbaG10kO xVGvwpFzdnzc4rOw4Y2aHfJUC5D61rzGwK8HzHFzJiUqLxgHeGz1WGpyRlImgqmqVubEZlX8p QI1DC5Pm1iqrNmKs02jO9zcVKzy7gvK8Q7Rk1oFC3Wtyn+XZ6LJBpb8PUyYIKgHxknIwDjsRG j7FQBlgdZfSSMpSjCOvExPVida/bXQF4kIgH+OXAlW7Pc07XCb0grh4U7mU0x8p9lNlxPURBo FkZzZe/mW5uUUlShI0ObqQpugE8+FOPru8QMsIeFnf1Gx0FpJgF5/fkNFgDN0OlHRoCjIA67j CKZmvfMB7FxgfuKltH+WuS6jt0JC/hGbJFC7pzk1wR9K+qhXzBNanyoXnZ/ycXvKmB9/RA1sc GNrFlqrG0Qf+vTHo2XCPNw/v0zkXKT3iUfj3pdBTfdNTiNOIv0Uxa25zg9QTE4UOdR7jUf/23 bRF6qeGoCpT1YazJZGX/4SLmapobBKYejXAbfY3xgihOuTLQlpc2d35MAZBk43P0wLf/bS0b3 958dGqlOLY8C7iFO6vlX+55CMHah8dM1HYF4McxpmjjpSf3LvYA4gMmipcWQfYsEFMUYbV1D2 VT4uiiDygA3IIwizEGU87XnWlnEWBgQWefMPVsAs7KP0hiRjHytSCEAaMWtovYLx49mXLoub7 OL+LzIjIeqECFGSgDAWsmbdxFDQmT22fVqMQ/U2BCJiccZaTHpen0xZ8tLss556MzBR6Xn6XC OSO1DVA6FJqpUXma+zvUDVQgwXIHkKTVurjgdSD0DCAK08c0K9+3EVGNHsNS39gaMSedfc4Mp x24tp78Ngzg2KYsY+Qu1KFSYg6n2VhD+UP1KvA19pe3tg4ZXt50shJOCZcCOA8VvonHz0XQpa io905aR/wVb7waXMDg/Cz3/gHaqIUGbZqC5SaaThR5Xd+dTCLizx7OsINa2hL1WxQ7CURLv1j nYHC3KtsW2PX/LKhJKmf32pWkQ1akK4n1Mz2H69z9KACEoN5ZOSD2OOPzRmDBvvMswn8jZjCK Zej0xJu3VBY3OU+GfbT28xZCHgOeL27noUQEMcYQqxMIInCZDAuKOPMFSGnB1AINKYwPLrdO8 +zNtI0JzJ0Em3mP+7pV1u3xENl57tdjDDffrkANZoV6Wlgi/7DNGM4lbksKYdtwA4eMSq21Ze i5lrrRJG4Co7FzCaWZ46zfjjzqkUpLyfnr59NCXc23Bpu Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add ecb(aes), cbc(aes) and ctr(aes) skcipher algorithms for new Realtek crypto device. Signed-off-by: Markus Stockhausen --- .../crypto/realtek/realtek_crypto_skcipher.c | 361 ++++++++++++++++++ 1 file changed, 361 insertions(+) create mode 100644 drivers/crypto/realtek/realtek_crypto_skcipher.c -- 2.37.3 diff --git a/drivers/crypto/realtek/realtek_crypto_skcipher.c b/drivers/crypto/realtek/realtek_crypto_skcipher.c new file mode 100644 index 000000000000..6e2cde77b4d4 --- /dev/null +++ b/drivers/crypto/realtek/realtek_crypto_skcipher.c @@ -0,0 +1,361 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Crypto acceleration support for Realtek crypto engine. Based on ideas from + * Rockchip & SafeXcel driver plus Realtek OpenWrt RTK. + * + * Copyright (c) 2022, Markus Stockhausen + */ + +#include +#include + +#include "realtek_crypto.h" + +static inline void rtcr_inc_iv(u8 *iv, int cnt) +{ + u32 *ctr = (u32 *)iv + 4; + u32 old, new, carry = cnt; + + /* avoid looping with crypto_inc() */ + do { + old = be32_to_cpu(*--ctr); + new = old + carry; + *ctr = cpu_to_be32(new); + carry = (new < old) && (ctr > (u32 *)iv) ? 1 : 0; + } while (carry); +} + +static inline void rtcr_cut_skcipher_len(int *reqlen, int opmode, u8 *iv) +{ + int len = min(*reqlen, RTCR_MAX_REQ_SIZE); + + if (opmode & RTCR_SRC_OP_CRYPT_CTR) { + /* limit data as engine does not wrap around cleanly */ + u32 ctr = be32_to_cpu(*((u32 *)iv + 3)); + int blocks = min(~ctr, 0x3fffu) + 1; + + len = min(blocks * AES_BLOCK_SIZE, len); + } + + *reqlen = len; +} + +static inline void rtcr_max_skcipher_len(int *reqlen, struct scatterlist **sg, + int *sgoff, int *sgcnt) +{ + int len, cnt, sgnoff, sgmax = RTCR_MAX_SG_SKCIPHER, datalen, maxlen = *reqlen; + struct scatterlist *sgn; + +redo: + datalen = cnt = 0; + sgnoff = *sgoff; + sgn = *sg; + + while (sgn && (datalen < maxlen) && (cnt < sgmax)) { + cnt++; + len = min((int)sg_dma_len(sgn) - sgnoff, maxlen - datalen); + datalen += len; + if (len + sgnoff < sg_dma_len(sgn)) { + sgnoff = sgnoff + len; + break; + } + sgn = sg_next(sgn); + sgnoff = 0; + if (unlikely((cnt == sgmax) && (datalen < AES_BLOCK_SIZE))) { + /* expand search to get at least one block */ + sgmax = AES_BLOCK_SIZE; + maxlen = min(maxlen, AES_BLOCK_SIZE); + } + } + + if (unlikely((datalen < maxlen) && (datalen & (AES_BLOCK_SIZE - 1)))) { + /* recalculate to get aligned size */ + maxlen = datalen & ~(AES_BLOCK_SIZE - 1); + goto redo; + } + + *sg = sgn; + *sgoff = sgnoff; + *sgcnt = cnt; + *reqlen = datalen; +} + +static int rtcr_process_skcipher(struct skcipher_request *sreq, int opmode) +{ + char *dataout, *iv, ivbk[AES_BLOCK_SIZE], datain[AES_BLOCK_SIZE]; + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(sreq); + struct rtcr_skcipher_ctx *sctx = crypto_skcipher_ctx(tfm); + int totallen = sreq->cryptlen, sgoff = 0, dgoff = 0; + int padlen, sgnoff, sgcnt, reqlen, ret, fblen; + struct rtcr_crypto_dev *cdev = sctx->cdev; + struct scatterlist *sg = sreq->src, *sgn; + int idx, srcidx, dstidx, len, datalen; + + if (!totallen) + return 0; + + if ((totallen & (AES_BLOCK_SIZE - 1)) && (!(opmode & RTCR_SRC_OP_CRYPT_CTR))) + return -EINVAL; + +redo: + sgnoff = sgoff; + sgn = sg; + datalen = totallen; + + /* limit input so that engine can process it */ + rtcr_cut_skcipher_len(&datalen, opmode, sreq->iv); + rtcr_max_skcipher_len(&datalen, &sgn, &sgnoff, &sgcnt); + + /* CTR padding */ + padlen = (AES_BLOCK_SIZE - datalen) & (AES_BLOCK_SIZE - 1); + reqlen = datalen + padlen; + + fblen = 0; + if (sgcnt > RTCR_MAX_SG_SKCIPHER) { + /* single AES block with too many SGs */ + fblen = datalen; + sg_pcopy_to_buffer(sg, sgcnt, datain, datalen, sgoff); + } + + if ((opmode & RTCR_SRC_OP_CRYPT_CBC) && + (!(opmode & RTCR_SRC_OP_KAM_ENC))) { + /* CBC decryption IV might get overwritten */ + sg_pcopy_to_buffer(sg, sgcnt, ivbk, AES_BLOCK_SIZE, + sgoff + datalen - AES_BLOCK_SIZE); + } + + /* Get free space in the ring */ + if (padlen || (datalen + dgoff > sg_dma_len(sreq->dst))) { + len = datalen; + } else { + len = RTCR_WB_LEN_SG_DIRECT; + dataout = sg_virt(sreq->dst) + dgoff; + } + + ret = rtcr_alloc_ring(cdev, 2 + (fblen ? 1 : sgcnt) + (padlen ? 1 : 0), + &srcidx, &dstidx, len, &dataout); + if (ret) + return ret; + + /* Write back any uncommitted data to memory */ + if (dataout == sg_virt(sreq->src) + sgoff) { + dma_map_sg(cdev->dev, sg, sgcnt, DMA_BIDIRECTIONAL); + } else { + dma_sync_single_for_device(cdev->dev, virt_to_phys(dataout), + reqlen, DMA_BIDIRECTIONAL); + if (fblen) + dma_sync_single_for_device(cdev->dev, virt_to_phys(datain), + reqlen, DMA_TO_DEVICE); + else + dma_map_sg(cdev->dev, sg, sgcnt, DMA_TO_DEVICE); + } + + if (sreq->iv) + dma_sync_single_for_device(cdev->dev, virt_to_phys(sreq->iv), + AES_BLOCK_SIZE, DMA_TO_DEVICE); + /* + * Feed input data into the rings. Start with destination ring and fill + * source ring afterwards. Ensure that the owner flag of the first source + * ring is the last that becomes visible to the engine. + */ + rtcr_add_dst_to_ring(cdev, dstidx, dataout, reqlen, sreq->dst, dgoff); + + idx = rtcr_inc_src_idx(srcidx, 1); + rtcr_add_src_to_ring(cdev, idx, sreq->iv, AES_BLOCK_SIZE, reqlen); + + if (fblen) { + idx = rtcr_inc_src_idx(idx, 1); + rtcr_add_src_to_ring(cdev, idx, (void *)datain, fblen, reqlen); + } + + datalen -= fblen; + while (datalen) { + len = min((int)sg_dma_len(sg) - sgoff, datalen); + + idx = rtcr_inc_src_idx(idx, 1); + rtcr_add_src_to_ring(cdev, idx, sg_virt(sg) + sgoff, len, reqlen); + + datalen -= len; + sg = sg_next(sg); + sgoff = 0; + } + + if (padlen) { + idx = rtcr_inc_src_idx(idx, 1); + rtcr_add_src_to_ring(cdev, idx, (void *)empty_zero_page, padlen, reqlen); + } + + rtcr_add_src_pad_to_ring(cdev, idx, reqlen); + rtcr_add_src_skcipher_to_ring(cdev, srcidx, opmode, reqlen, sctx); + + /* Off we go */ + rtcr_kick_engine(cdev); + if (rtcr_wait_for_request(cdev, dstidx)) + return -EINVAL; + + /* Handle IV feedback as engine does not provide it */ + if (opmode & RTCR_SRC_OP_CRYPT_CTR) { + rtcr_inc_iv(sreq->iv, reqlen / AES_BLOCK_SIZE); + } else if (opmode & RTCR_SRC_OP_CRYPT_CBC) { + iv = opmode & RTCR_SRC_OP_KAM_ENC ? + dataout + reqlen - AES_BLOCK_SIZE : ivbk; + memcpy(sreq->iv, iv, AES_BLOCK_SIZE); + } + + sg = sgn; + sgoff = sgnoff; + dgoff += reqlen; + totallen -= min(reqlen, totallen); + + if (totallen) + goto redo; + + return 0; +} + +static int rtcr_skcipher_encrypt(struct skcipher_request *sreq) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(sreq); + struct rtcr_skcipher_ctx *sctx = crypto_skcipher_ctx(tfm); + int opmode = sctx->opmode | RTCR_SRC_OP_KAM_ENC; + + return rtcr_process_skcipher(sreq, opmode); +} + +static int rtcr_skcipher_decrypt(struct skcipher_request *sreq) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(sreq); + struct rtcr_skcipher_ctx *sctx = crypto_skcipher_ctx(tfm); + int opmode = sctx->opmode; + + opmode |= sctx->opmode & RTCR_SRC_OP_CRYPT_CTR ? + RTCR_SRC_OP_KAM_ENC : RTCR_SRC_OP_KAM_DEC; + + return rtcr_process_skcipher(sreq, opmode); +} + +static int rtcr_skcipher_setkey(struct crypto_skcipher *cipher, + const u8 *key, unsigned int keylen) +{ + struct crypto_tfm *tfm = crypto_skcipher_tfm(cipher); + struct rtcr_skcipher_ctx *sctx = crypto_tfm_ctx(tfm); + struct rtcr_crypto_dev *cdev = sctx->cdev; + struct crypto_aes_ctx kctx; + int p, i; + + if (aes_expandkey(&kctx, key, keylen)) + return -EINVAL; + + sctx->keylen = keylen; + sctx->opmode = (sctx->opmode & ~RTCR_SRC_OP_CIPHER_MASK) | + RTCR_SRC_OP_CIPHER_FROM_KEY(keylen); + + memcpy(sctx->key_enc, key, keylen); + /* decryption key is derived from expanded key */ + p = ((keylen / 4) + 6) * 4; + for (i = 0; i < 8; i++) { + sctx->key_dec[i] = cpu_to_le32(kctx.key_enc[p + i]); + if (i == 3) + p -= keylen == AES_KEYSIZE_256 ? 8 : 6; + } + + dma_sync_single_for_device(cdev->dev, virt_to_phys(sctx->key_enc), + 2 * AES_KEYSIZE_256, DMA_TO_DEVICE); + + return 0; +} + +static int rtcr_skcipher_cra_init(struct crypto_tfm *tfm) +{ + struct rtcr_skcipher_ctx *sctx = crypto_tfm_ctx(tfm); + struct rtcr_alg_template *tmpl; + + tmpl = container_of(tfm->__crt_alg, struct rtcr_alg_template, + alg.skcipher.base); + + sctx->cdev = tmpl->cdev; + sctx->opmode = tmpl->opmode; + + return 0; +} + +static void rtcr_skcipher_cra_exit(struct crypto_tfm *tfm) +{ + void *ctx = crypto_tfm_ctx(tfm); + + memzero_explicit(ctx, tfm->__crt_alg->cra_ctxsize); +} + +struct rtcr_alg_template rtcr_skcipher_ecb_aes = { + .type = RTCR_ALG_SKCIPHER, + .opmode = RTCR_SRC_OP_MS_CRYPTO | RTCR_SRC_OP_CRYPT_ECB, + .alg.skcipher = { + .setkey = rtcr_skcipher_setkey, + .encrypt = rtcr_skcipher_encrypt, + .decrypt = rtcr_skcipher_decrypt, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .base = { + .cra_name = "ecb(aes)", + .cra_driver_name = "realtek-ecb-aes", + .cra_priority = 300, + .cra_flags = CRYPTO_ALG_ASYNC, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct rtcr_skcipher_ctx), + .cra_alignmask = 0, + .cra_init = rtcr_skcipher_cra_init, + .cra_exit = rtcr_skcipher_cra_exit, + .cra_module = THIS_MODULE, + }, + }, +}; + +struct rtcr_alg_template rtcr_skcipher_cbc_aes = { + .type = RTCR_ALG_SKCIPHER, + .opmode = RTCR_SRC_OP_MS_CRYPTO | RTCR_SRC_OP_CRYPT_CBC, + .alg.skcipher = { + .setkey = rtcr_skcipher_setkey, + .encrypt = rtcr_skcipher_encrypt, + .decrypt = rtcr_skcipher_decrypt, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .base = { + .cra_name = "cbc(aes)", + .cra_driver_name = "realtek-cbc-aes", + .cra_priority = 300, + .cra_flags = CRYPTO_ALG_ASYNC, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct rtcr_skcipher_ctx), + .cra_alignmask = 0, + .cra_init = rtcr_skcipher_cra_init, + .cra_exit = rtcr_skcipher_cra_exit, + .cra_module = THIS_MODULE, + }, + }, +}; + +struct rtcr_alg_template rtcr_skcipher_ctr_aes = { + .type = RTCR_ALG_SKCIPHER, + .opmode = RTCR_SRC_OP_MS_CRYPTO | RTCR_SRC_OP_CRYPT_CTR, + .alg.skcipher = { + .setkey = rtcr_skcipher_setkey, + .encrypt = rtcr_skcipher_encrypt, + .decrypt = rtcr_skcipher_decrypt, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .base = { + .cra_name = "ctr(aes)", + .cra_driver_name = "realtek-ctr-aes", + .cra_priority = 300, + .cra_flags = CRYPTO_ALG_ASYNC, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct rtcr_skcipher_ctx), + .cra_alignmask = 0, + .cra_init = rtcr_skcipher_cra_init, + .cra_exit = rtcr_skcipher_cra_exit, + .cra_module = THIS_MODULE, + }, + }, +};