From patchwork Thu Oct 13 18:40:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Markus Stockhausen X-Patchwork-Id: 614890 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FDFFC433FE for ; Thu, 13 Oct 2022 19:05:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229717AbiJMTFd (ORCPT ); Thu, 13 Oct 2022 15:05:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229695AbiJMTFc (ORCPT ); Thu, 13 Oct 2022 15:05:32 -0400 Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5305E84E71 for ; Thu, 13 Oct 2022 12:05:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1665687927; bh=JkkGbOrcjjRuW4KeCwf2Ign9inZRbe+hMZq8NVTB7kg=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date:In-Reply-To:References; b=Qm9aFpFYgy7p197Q+UrN/TU754jinUunD+ZFGBe0nC4skVB9955Xz9p7usCxixXAi x4NS7nyfhyDuC/pOG8Uc+NxaAq20FYaab3fmXs7MQe/at7LaZ+iupzxnyrtZrZiS7O hnaucdOrxnyjQGWDDx6NBg+AZY83I/2U/KSbKxy8= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from fedora.willemsstb.de ([94.31.86.22]) by mail.gmx.net (mrgmx105 [212.227.17.174]) with ESMTPSA (Nemesis) id 1MC30P-1or3Ls3Bjp-00CNGS; Thu, 13 Oct 2022 20:40:29 +0200 From: Markus Stockhausen To: linux-crypto@vger.kernel.org Cc: Markus Stockhausen Subject: [PATCH 3/6] crypto/realtek: hash algorithms Date: Thu, 13 Oct 2022 20:40:23 +0200 Message-Id: <20221013184026.63826-4-markus.stockhausen@gmx.de> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221013184026.63826-1-markus.stockhausen@gmx.de> References: <20221013184026.63826-1-markus.stockhausen@gmx.de> MIME-Version: 1.0 X-Provags-ID: V03:K1:vqqXUhAwW8s6PJVujFOZYQlQtOlxCvHlT+I7IFCRMozC5IdEmEg yKJ4gATiAl35HUd+xWsv2fkoobtRZXkGqnK3FLr96R/dOxCVEz7v5EW7lNE21Bl0i03NBf3 ZreSM51/biDU+tL6PXY9FW1TS0CF3P0b6atdWb7FTV3l62y33+0q3P9xp55258Qe1FN93Bc WBf/r++tEmhfrfsV+ylXg== X-UI-Out-Filterresults: notjunk:1;V03:K0:U+AaJdy7nwQ=:SNxAxgkrcafBHLOwqMqTVW QHRUSr+zaR5rx9ycZC2NDzz0Ah1P9zNRA2Y3uDm5ZWZ7ZaRjBdC/JeQ2FXSFhnJbi/OcuThNb 7h5RAAMKrXxD56nsyJxLAuRDGXVVywEFfO1O1NtedU9YeYwkMB+wVXBRzMSQxdV/msL9GmYzK nUuQgnObDVY6oWaff8k1nHN+kSgO6c6l74WRm5TKmvrtWd3USelJGM8e/D1VQMNA4mEys5zi6 iizoo3+QfiJNt6j/2gpe/XQkKe0oxjJyEIzsS/4AsdiAlwIV9gd9MMe6kaZPEm96e6tHiqrVV meeuOFkWHPfhGw4lRyix5KjKKw3p9Ry/txLf8MnnfrCtTvMnt4PWOPfTY9hWL1Iht/pw7grjy PaKAoXA7BGNY98a58X+O+8f2AVDNk3nrrsuiOngBrUHSQwRSk4NtSuKRIzvh8biIefVqDzvS5 ttfHTUTVLcoRDeLqsbEbOfGfEL7BmTiXM14syltVsnGprIEG+zU+UoNkTohubilJI+Py3giIm WPn6J2DjwUL8GXeqWp5E8GffK3BTm/PMufU+saCAZwlX4aih5uqV5+vCowFO4dwBringbd0Vn lFnLuGLz0Km+U49x6W6b9AnXUuyumqeE5JmktY0e+i65r3UaUZ9chcrzZDxhNO4hBNx10rM0f BywsEHJE9FrAf/v5tYmhuXXUZ/u3rJLWPrjSWhd0y8oS3ImfWY4LibvugYITaBMAfz4sNQxPB JD0pwa+eaC4LKFcTeGm/h/eh1xRYUKrlfFpAB2y0qAOH6zPNShKkBMw56TmWP8a8gzwPXx6bD FAGl1jATRPMoAac2b6lyTN801+3OlZq6S4V/AwbDHu1GrPwq/nYJTDHIq/vOY/NNAUSLhqVbb 2pCE/cL8RNoVPcYvjmvSLTHMsJAROtITe5+O+fListpUhBagDbLQFSZmdPy5c5Sf591hk3xYJ eK+HDs7EYzhxh7N73lZuRzMGA8ptjkdORa06qsIXo26YYXHfjIkNq3OZ6ZFnMn8EiTPXVZ0Sr +d1YSKieEwRApAL1jj7k2EictY2hwfn35PiXXtTTHdYpFmni5fJBJ51W3uVpzdmCC8JchpZZV Xz1o3YZKhFS62wYTVvSCUwh6alK5KjSc0We9r08Hz4j4Ghs7MQDg5yNf+oBmS203edpA3sEfA GRreTtriqqTxDNic6UaE90ZQcJZ1gxy1YF3voLkWJXQ5W0ccc2LpGQnMvKNV+w5L0kVWcV+S9 FCl3Ct9XAA7Qp+vC7EzE3KTP8lgSQvwuCMmig4ubqwsCOR7aIxScCcup6TEeMWGRL9luhYX7b TdG2xb6IQAYGOTkU6JIh1xrwf+G+WU8rUeeHUzarmbr9tjWCv3dzZ3jxhJG1Gz2XsT6QgONAs tAUJLA8PECTl/8HbE/CLljMzf3IpgrNxuNza+oxI3bqNfmisKJE7HgyqLkMQ7s9QCHWf8c9zJ n+zaHLgCCKbwNagmBLulNBasfEzVEKyWel/22vZH8x6qot7fShlAgs2gagGPb8zcgDzvd9YZ6 qThdbE3C1SkVqX34eeFN1hnOBcF922rzGq5ODdQuNbOE3 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add md5/sha1 hash algorithms for new Realtek crypto device. Signed-off-by: Markus Stockhausen --- drivers/crypto/realtek/realtek_crypto_ahash.c | 406 ++++++++++++++++++ 1 file changed, 406 insertions(+) create mode 100644 drivers/crypto/realtek/realtek_crypto_ahash.c -- 2.37.3 diff --git a/drivers/crypto/realtek/realtek_crypto_ahash.c b/drivers/crypto/realtek/realtek_crypto_ahash.c new file mode 100644 index 000000000000..eba83451fac1 --- /dev/null +++ b/drivers/crypto/realtek/realtek_crypto_ahash.c @@ -0,0 +1,406 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Crypto acceleration support for Realtek crypto engine. Based on ideas from + * Rockchip & SafeXcel driver plus Realtek OpenWrt RTK. + * + * Copyright (c) 2022, Markus Stockhausen + */ + +#include +#include + +#include "realtek_crypto.h" + +static inline struct ahash_request *fallback_request_ctx(struct ahash_request *areq) +{ + char *p = (char *)ahash_request_ctx(areq); + + return (struct ahash_request *)(p + offsetof(struct rtcr_ahash_req, vector)); +} + +static inline void *fallback_export_state(void *export) +{ + char *p = (char *)export; + + return (void *)(p + offsetof(struct rtcr_ahash_req, vector)); +} + +static int rtcr_process_hash(struct ahash_request *areq, int opmode) +{ + unsigned int len, nextbuflen, datalen, padlen, reqlen; + struct rtcr_ahash_req *hreq = ahash_request_ctx(areq); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq); + struct rtcr_ahash_ctx *hctx = crypto_ahash_ctx(tfm); + int sgcnt = hreq->state & RTCR_REQ_SG_MASK; + struct rtcr_crypto_dev *cdev = hctx->cdev; + struct scatterlist *sg = areq->src; + int idx, srcidx, dstidx, ret; + u64 pad[RTCR_HASH_PAD_SIZE]; + char *ppad; + + /* Quick checks if processing is really needed */ + if (unlikely(!areq->nbytes) && !(opmode & RTCR_HASH_FINAL)) + return 0; + + if (hreq->buflen + areq->nbytes < 64 && !(opmode & RTCR_HASH_FINAL)) { + hreq->buflen += sg_pcopy_to_buffer(areq->src, sg_nents(areq->src), + hreq->buf + hreq->buflen, + areq->nbytes, 0); + return 0; + } + + /* calculate required parts of the request */ + datalen = (opmode & RTCR_HASH_UPDATE) ? areq->nbytes : 0; + if (opmode & RTCR_HASH_FINAL) { + nextbuflen = 0; + padlen = 64 - ((hreq->buflen + datalen) & 63); + if (padlen < 9) + padlen += 64; + hreq->totallen += hreq->buflen + datalen; + + memset(pad, 0, sizeof(pad) - sizeof(u64)); + ppad = (char *)&pad[RTCR_HASH_PAD_SIZE] - padlen; + *ppad = 0x80; + pad[RTCR_HASH_PAD_SIZE - 1] = hreq->state & RTCR_REQ_MD5 ? + cpu_to_le64(hreq->totallen << 3) : + cpu_to_be64(hreq->totallen << 3); + } else { + nextbuflen = (hreq->buflen + datalen) & 63; + padlen = 0; + datalen -= nextbuflen; + hreq->totallen += hreq->buflen + datalen; + } + reqlen = hreq->buflen + datalen + padlen; + + /* Write back any uncommitted data to memory. */ + if (hreq->buflen) + dma_sync_single_for_device(cdev->dev, virt_to_phys(hreq->buf), + hreq->buflen, DMA_TO_DEVICE); + if (padlen) + dma_sync_single_for_device(cdev->dev, virt_to_phys(ppad), + padlen, DMA_TO_DEVICE); + if (datalen) + dma_map_sg(cdev->dev, sg, sgcnt, DMA_TO_DEVICE); + + /* Get free space in the ring */ + sgcnt = 1 + (hreq->buflen ? 1 : 0) + (datalen ? sgcnt : 0) + (padlen ? 1 : 0); + + ret = rtcr_alloc_ring(cdev, sgcnt, &srcidx, &dstidx, RTCR_WB_LEN_HASH, NULL); + if (ret) + return ret; + /* + * Feed input data into the rings. Start with destination ring and fill + * source ring afterwards. Ensure that the owner flag of the first source + * ring is the last that becomes visible to the engine. + */ + rtcr_add_dst_to_ring(cdev, dstidx, NULL, 0, hreq->vector, 0); + + idx = srcidx; + if (hreq->buflen) { + idx = rtcr_inc_src_idx(idx, 1); + rtcr_add_src_to_ring(cdev, idx, hreq->buf, hreq->buflen, reqlen); + } + + while (datalen) { + len = min(sg_dma_len(sg), datalen); + + idx = rtcr_inc_src_idx(idx, 1); + rtcr_add_src_to_ring(cdev, idx, sg_virt(sg), len, reqlen); + + datalen -= len; + if (datalen) + sg = sg_next(sg); + } + + if (padlen) { + idx = rtcr_inc_src_idx(idx, 1); + rtcr_add_src_to_ring(cdev, idx, ppad, padlen, reqlen); + } + + rtcr_add_src_pad_to_ring(cdev, idx, reqlen); + rtcr_add_src_ahash_to_ring(cdev, srcidx, hctx->opmode, reqlen); + + /* Off we go */ + rtcr_kick_engine(cdev); + if (rtcr_wait_for_request(cdev, dstidx)) + return -EINVAL; + + hreq->state |= RTCR_REQ_FB_ACT; + hreq->buflen = nextbuflen; + + if (nextbuflen) + sg_pcopy_to_buffer(sg, sg_nents(sg), hreq->buf, nextbuflen, len); + if (padlen) + memcpy(areq->result, hreq->vector, crypto_ahash_digestsize(tfm)); + + return 0; +} + +static void rtcr_check_request(struct ahash_request *areq, int opmode) +{ + struct rtcr_ahash_req *hreq = ahash_request_ctx(areq); + struct scatterlist *sg = areq->src; + int reqlen, sgcnt, sgmax; + + if (hreq->state & RTCR_REQ_FB_ACT) + return; + + if (reqlen > RTCR_MAX_REQ_SIZE) { + hreq->state |= RTCR_REQ_FB_ACT; + return; + } + + sgcnt = 0; + sgmax = RTCR_MAX_SG_AHASH - (hreq->buflen ? 1 : 0); + reqlen = areq->nbytes; + if (!(opmode & RTCR_HASH_FINAL)) { + reqlen -= (hreq->buflen + reqlen) & 63; + sgmax--; + } + + while (reqlen > 0) { + reqlen -= sg_dma_len(sg); + sgcnt++; + sg = sg_next(sg); + } + + if (sgcnt > sgmax) + hreq->state |= RTCR_REQ_FB_ACT; + else + hreq->state = (hreq->state & ~RTCR_REQ_SG_MASK) | sgcnt; +} + +static bool rtcr_check_fallback(struct ahash_request *areq) +{ + struct ahash_request *freq = fallback_request_ctx(areq); + struct rtcr_ahash_req *hreq = ahash_request_ctx(areq); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq); + struct rtcr_ahash_ctx *hctx = crypto_ahash_ctx(tfm); + union rtcr_fallback_state state; + + if (!(hreq->state & RTCR_REQ_FB_ACT)) + return false; + + if (!(hreq->state & RTCR_REQ_FB_RDY)) { + /* Convert state to generic fallback state */ + if (hreq->state & RTCR_REQ_MD5) { + memcpy(state.md5.hash, hreq->vector, MD5_DIGEST_SIZE); + if (hreq->totallen) + cpu_to_le32_array(state.md5.hash, 4); + memcpy(state.md5.block, hreq->buf, SHA1_BLOCK_SIZE); + state.md5.byte_count = hreq->totallen + (u64)hreq->buflen; + } else { + memcpy(state.sha1.state, hreq->vector, SHA1_DIGEST_SIZE); + memcpy(state.sha1.buffer, &hreq->buf, SHA1_BLOCK_SIZE); + state.sha1.count = hreq->totallen + (u64)hreq->buflen; + } + } + + ahash_request_set_tfm(freq, hctx->fback); + ahash_request_set_crypt(freq, areq->src, areq->result, areq->nbytes); + + if (!(hreq->state & RTCR_REQ_FB_RDY)) { + crypto_ahash_import(freq, &state); + hreq->state |= RTCR_REQ_FB_RDY; + } + + return true; +} + +static int rtcr_ahash_init(struct ahash_request *areq) +{ + struct rtcr_ahash_req *hreq = ahash_request_ctx(areq); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(areq); + int ds = crypto_ahash_digestsize(tfm); + + memset(hreq, 0, sizeof(*hreq)); + + hreq->vector[0] = SHA1_H0; + hreq->vector[1] = SHA1_H1; + hreq->vector[2] = SHA1_H2; + hreq->vector[3] = SHA1_H3; + hreq->vector[4] = SHA1_H4; + + hreq->state |= (ds == MD5_DIGEST_SIZE) ? RTCR_REQ_MD5 : RTCR_REQ_SHA1; + + return 0; +} + +static int rtcr_ahash_update(struct ahash_request *areq) +{ + struct ahash_request *freq = fallback_request_ctx(areq); + + rtcr_check_request(areq, RTCR_HASH_UPDATE); + if (rtcr_check_fallback(areq)) + return crypto_ahash_update(freq); + return rtcr_process_hash(areq, RTCR_HASH_UPDATE); +} + +static int rtcr_ahash_final(struct ahash_request *areq) +{ + struct ahash_request *freq = fallback_request_ctx(areq); + + if (rtcr_check_fallback(areq)) + return crypto_ahash_final(freq); + + return rtcr_process_hash(areq, RTCR_HASH_FINAL); +} + +static int rtcr_ahash_finup(struct ahash_request *areq) +{ + struct ahash_request *freq = fallback_request_ctx(areq); + + rtcr_check_request(areq, RTCR_HASH_FINAL | RTCR_HASH_UPDATE); + if (rtcr_check_fallback(areq)) + return crypto_ahash_finup(freq); + + return rtcr_process_hash(areq, RTCR_HASH_FINAL | RTCR_HASH_UPDATE); +} + +static int rtcr_ahash_digest(struct ahash_request *areq) +{ + struct ahash_request *freq = fallback_request_ctx(areq); + int ret; + + ret = rtcr_ahash_init(areq); + if (ret) + return ret; + + rtcr_check_request(areq, RTCR_HASH_FINAL | RTCR_HASH_UPDATE); + if (rtcr_check_fallback(areq)) + return crypto_ahash_digest(freq); + + return rtcr_process_hash(areq, RTCR_HASH_FINAL | RTCR_HASH_UPDATE); +} + +static int rtcr_ahash_import(struct ahash_request *areq, const void *in) +{ + const void *fexp = (const void *)fallback_export_state((void *)in); + struct ahash_request *freq = fallback_request_ctx(areq); + struct rtcr_ahash_req *hreq = ahash_request_ctx(areq); + const struct rtcr_ahash_req *hexp = in; + + hreq->state = hexp->state; + if (hreq->state & RTCR_REQ_FB_ACT) + hreq->state |= RTCR_REQ_FB_RDY; + + if (rtcr_check_fallback(areq)) + return crypto_ahash_import(freq, fexp); + + memcpy(hreq, hexp, sizeof(struct rtcr_ahash_req)); + + return 0; +} + +static int rtcr_ahash_export(struct ahash_request *areq, void *out) +{ + struct ahash_request *freq = fallback_request_ctx(areq); + struct rtcr_ahash_req *hreq = ahash_request_ctx(areq); + void *fexp = fallback_export_state(out); + struct rtcr_ahash_req *hexp = out; + + if (rtcr_check_fallback(areq)) { + hexp->state = hreq->state; + return crypto_ahash_export(freq, fexp); + } + + memcpy(hexp, hreq, sizeof(struct rtcr_ahash_req)); + + return 0; +} + +static int rtcr_ahash_cra_init(struct crypto_tfm *tfm) +{ + struct crypto_ahash *ahash = __crypto_ahash_cast(tfm); + struct rtcr_ahash_ctx *hctx = crypto_tfm_ctx(tfm); + struct rtcr_crypto_dev *cdev = hctx->cdev; + struct rtcr_alg_template *tmpl; + + tmpl = container_of(__crypto_ahash_alg(tfm->__crt_alg), + struct rtcr_alg_template, alg.ahash); + + hctx->cdev = tmpl->cdev; + hctx->opmode = tmpl->opmode; + hctx->fback = crypto_alloc_ahash(crypto_tfm_alg_name(tfm), 0, + CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK); + + if (IS_ERR(hctx->fback)) { + dev_err(cdev->dev, "could not allocate fallback for %s\n", + crypto_tfm_alg_name(tfm)); + return PTR_ERR(hctx->fback); + } + + crypto_ahash_set_reqsize(ahash, max(sizeof(struct rtcr_ahash_req), + offsetof(struct rtcr_ahash_req, vector) + + sizeof(struct ahash_request) + + crypto_ahash_reqsize(hctx->fback))); + + return 0; +} + +static void rtcr_ahash_cra_exit(struct crypto_tfm *tfm) +{ + struct rtcr_ahash_ctx *hctx = crypto_tfm_ctx(tfm); + + crypto_free_ahash(hctx->fback); +} + +struct rtcr_alg_template rtcr_ahash_md5 = { + .type = RTCR_ALG_AHASH, + .opmode = RTCR_SRC_OP_MS_HASH | RTCR_SRC_OP_HASH_MD5, + .alg.ahash = { + .init = rtcr_ahash_init, + .update = rtcr_ahash_update, + .final = rtcr_ahash_final, + .finup = rtcr_ahash_finup, + .export = rtcr_ahash_export, + .import = rtcr_ahash_import, + .digest = rtcr_ahash_digest, + .halg = { + .digestsize = MD5_DIGEST_SIZE, + /* statesize calculated during initialization */ + .base = { + .cra_name = "md5", + .cra_driver_name = "realtek-md5", + .cra_priority = 300, + .cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize = SHA1_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct rtcr_ahash_ctx), + .cra_alignmask = 0, + .cra_init = rtcr_ahash_cra_init, + .cra_exit = rtcr_ahash_cra_exit, + .cra_module = THIS_MODULE, + } + } + } +}; + +struct rtcr_alg_template rtcr_ahash_sha1 = { + .type = RTCR_ALG_AHASH, + .opmode = RTCR_SRC_OP_MS_HASH | RTCR_SRC_OP_HASH_SHA1, + .alg.ahash = { + .init = rtcr_ahash_init, + .update = rtcr_ahash_update, + .final = rtcr_ahash_final, + .finup = rtcr_ahash_finup, + .export = rtcr_ahash_export, + .import = rtcr_ahash_import, + .digest = rtcr_ahash_digest, + .halg = { + .digestsize = SHA1_DIGEST_SIZE, + /* statesize calculated during initialization */ + .base = { + .cra_name = "sha1", + .cra_driver_name = "realtek-sha1", + .cra_priority = 300, + .cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize = SHA1_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct rtcr_ahash_ctx), + .cra_alignmask = 0, + .cra_init = rtcr_ahash_cra_init, + .cra_exit = rtcr_ahash_cra_exit, + .cra_module = THIS_MODULE, + } + } + } +};