From patchwork Thu May 30 03:11:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jia Jie Ho X-Patchwork-Id: 800273 Received: from CHN02-SH0-obe.outbound.protection.partner.outlook.cn (mail-sh0chn02on2110.outbound.protection.partner.outlook.cn [139.219.146.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61908C2E9; Thu, 30 May 2024 07:45:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=139.219.146.110 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717055129; cv=fail; b=Tj9qIiIN0LFwyA6taB0QG7lHSv+aLIOR7cLBLUALyBGKvqqTkhYjhvhCSaCOP8dK376juoNuorw/6M8edwbwKeSj0Bu5oLo9BD9JbUansYY5bpiraMK5nHrxJVIHUGokGy5m2qzZBs3d/BmGswtP10G9aGMX+bEfZTGmUyyKHco= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717055129; c=relaxed/simple; bh=5ZXG4d8/9BsBkqkELCWTjOlvcrcuY/2dOM8Yvnm/xn4=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Q4FWBG6gUSpm1ccDllTb4Woxsa0MEWbHC1TMXTLzguOwfY9NM5ZOeCmCrTwx7qIGToHrLXCda+Z8tRP8tX9F0nT6z0Hn9Va3QMZjhGKSJQjWQjN/aJWXNoEtOfeHeV1BH8ypdsHDtefNdvsWjdpxZXjU41IajF4pQeO7Ap/PDao= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com; spf=pass smtp.mailfrom=starfivetech.com; arc=fail smtp.client-ip=139.219.146.110 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=starfivetech.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j73gtZD89HstbEeHXLvKbw36zHLAlp8YMSksoGW7mllXfL3kv/fysfl8EXLhxVnMdtnvyJHm8tD9NFZ/hgWj3iPLyW67mMLMyPE70uLCE9Yup6EQJp0c1vxAiXEMdSiTuAfYsNvKosT3MAQyMhNRCWbZUOFPeHhRI9cE/ACZmMv6cjFEGNhI7WhEHIFY78/r60C53wqyY0HVD/zgP1uPmeSXIam/j925M8muwS+5/fc3UeqkO3ZD99ewV332vPyzx410BJeFuUsufD5+AxmiI2LcBbMKWBlJwUn1x787FV4hB1P+u3gl1Li/4Qcea6kXcYj+RLuxq7V952JWU7W2hQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+LdsRqePpXLkY7j9IOxnswrsRGbzGB7Ma6sDy5vmhn8=; b=k1JjDxcM6ZKQVi0OoRyNNTHNbEGGb3lhohM4qCEQNXXB1WQT39n6j3hH3Og1tCt9SGtpFNCgfEX3WExPdh/mhHlbFvoTLua3PSFoqNOHkAq2QUmasn8fbCxjBgo6m746QPxAnwv5zPcs/3koBU/8N6Nu6KYE8aLqf9K2vcaiyyRNiDe/Q58yjmbPimTODtebgTGcRKBzK85pm4L+dsk2H4i/zcr1MsOZffWsi8EspRXRZem0AjbQ51PPaEOTF3Bht5MrMr/jfvAK9KdVJaXO5xtUZSTlloiKJzgUYYu1E6WXTCtjVtpLkBcDh/nGBZt7NnTmOguVXwAOKya+A6tOCw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=starfivetech.com; dmarc=pass action=none header.from=starfivetech.com; dkim=pass header.d=starfivetech.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=starfivetech.com; Received: from NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn (2406:e500:c510:10::10) by NT0PR01MB1054.CHNPR01.prod.partner.outlook.cn (2406:e500:c510:2::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.36; Thu, 30 May 2024 03:11:26 +0000 Received: from NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn ([fe80::e3b:43f8:2e6d:ecce]) by NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn ([fe80::e3b:43f8:2e6d:ecce%7]) with mapi id 15.20.7587.037; Thu, 30 May 2024 03:11:26 +0000 From: Jia Jie Ho To: Herbert Xu , "David S . Miller" , Eugeniy Paltsev , Vinod Koul , linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v5 1/3] dmaengine: dw-axi-dmac: Support hardware quirks Date: Thu, 30 May 2024 11:11:10 +0800 Message-ID: <20240530031112.4952-2-jiajie.ho@starfivetech.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240530031112.4952-1-jiajie.ho@starfivetech.com> References: <20240530031112.4952-1-jiajie.ho@starfivetech.com> X-ClientProxiedBy: ZQ0PR01CA0027.CHNPR01.prod.partner.outlook.cn (2406:e500:c550:2::17) To NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn (2406:e500:c510:10::10) Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: NT0PR01MB1182:EE_|NT0PR01MB1054:EE_ X-MS-Office365-Filtering-Correlation-Id: 76fc530e-90a2-4067-6083-08dc80563131 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; ARA:13230031|366007|41320700004|1800799015|52116005|38350700005; X-Microsoft-Antispam-Message-Info: PKLjfg5kY63EVoSvCdSO3f1aw0VOszBM+FuwyqG3KG3/kDGZAs/VjaJ+cgtuacBq/0Pji4yKkEy2iZNzrPEVSlhpLMlk7RqFcH+wdV8R9q84q70Sqb6RsmLEWVM2XsPsBlWl3tj4Siakq2bbj96SsOLVpYyBSv8jNfpbEO7RsuUgqhWQ4BLdV84eERXT3dc+uNgK4Mrky6GJl4BeQWeK5cfzlTYhXrsCS+DNXNaVoIes6pkI0iKikxAVIHePlOS9F42tMjzXjCPnqMH3dyWS5wODE1qbaeCHcyY1Jw9OzhHL167NqWoTLxjt+e3K/De9ILWSgYbeUweOqfCEfOyQ0uaXkJAZxY3EJtP2FmiyzK0SP4PAUizlB02Vj0GL+gz6rgX64MIjfO5s233mGrjN0eAzxU1vH/FqOJEhHPiWik3Ijs9lYDR9p5ohuZmYmZ7dLnpMigWzjWK7KfMttRBHRt16r3vYgSYgywD9oFljEgVFtVe+Th5SXJfRcFTF3CFLT8ay8kVHYLAFS3j8PpXgoaYIS1IduLVw8jmAXh1LXHILo4qzrx6Wp96XwR/2gXCISeTu6RqhJcUxD8Mxlv2v4kfBpLZPw7pZsLUfboYxiELNxkStly1W93JO85iFehaK X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn; PTR:; CAT:NONE; SFS:(13230031)(366007)(41320700004)(1800799015)(52116005)(38350700005); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: z4rVO9ZasdO4HMPYFkxhVCime2dM790OJBBAo0TsUho8RY3gIiCcwT5J5xuJrQqaDG7l73PuWedn2oZ6W08GWez+mN+gi7V5KcYrO8sK6SUVPGMUpUrQYeNqrdIZEKPxEtHBtdlws1svCoitg/J+MrwPqg9rB1ZN+PZaumFJ1hQ/rwK6xPuN2KHUCpuSxpB0k4dpTJNGh+6ri+UfD874p1yiksqe9bRNgwZMOPkwJ+WmSigHUzKdrIAfoS89ytXflfBNr/1EE8HZVpD6veFPQIvfzcjBH3W9qTZ/vAl/7PtyDtUUo7wp0/o8vz/iCh+7P+BRRok5MLT+VCl6dYAMt0rs8IYaDF8e+8MIuvdvDsOyOZ4fvY5HJPN3DYISebry/XzMtzHkEHiOagwvjA0XS9apcARBPJVVf5eE06DqKWKdImo9zXxqyzbdQMXtdhLIdV+VfnDNyYCWI7FcD73PR2gId7k6n25l9J8pQBD+zAE/nKamt/XtccTbm2mW4IEp3LwEvtlBW0ZoDzeVBBHMeAyvgmWGfHT7jF6TmUtpbgWrb1agD7A2ewtqXSenXx3C1WbIkSrnwlLuN0or0gSxmxLSMMwaFLwh6T8rxXrgib6gPmKP3blc/To5Hj6jcNJpr5w0BIkK6jwIfxagjUpqq7JO3K32Heu2pT/7/xSPKho53USeTjbg4T4FFuT/UqG/FqpdAAgyOazn0S+RhruXH+c9ZXCWitN4un99ssWi6lw/6e9wI6J/UtLZ2KGV69cuSsi+3EaF6GwjGei3RnDqjb+Dp0oYYmljYvYwNoNtBM03wONI8CI3mp2KLs1F/uIBdSuRlw9Wwz2QZOZXdO+M8BTaY7qFaEcNTq93VeOQ5BIWr68ugGfcSThYOhKaqYzTRCTX4f0+Uz9ja0WpfLi7x8AhK1nH1/0QzJJCGAfdhwU7HA26yKZl2lxQnYJc6WMb+CATnFNH+9PBT2pZlsMiHxY9UCDPoLMM4SaKSxo9qlevJr2JVWPrD+35EQaXblup48DErAo92FZsgaOnZnXwEiZdm7M7fS45UqPYng/n8K8fmz7rCs+IIWEoJznxxqpvCShQzfe/C+EB5kYps3UJi1+ztDdXzH5kv0iVHlpVViz4RDatzvLRpVQEqhw96DMRTpJHd/DoeAaMxejXT+Te87NqwD7ZRvIkVIOIQ3hz9AJEozQxDYG1X3D6lCSM8LSinP7y0NmGWVhZ8qzJtBysQhmOOaBJorhNC9ROu4Xfq0DbFc4OfbqTiyUSah1VK9c4GwaKa6m/T628oubwk0DOaHmOqfADdXLyji1G3DhYPuKmBZv7e0VChN2hCQvAnFy4iszq6HIMvFVmqmX531GObS282xtm0nKAXLJ24S8EdN5lX9nBtxeqlyG/HJOAX3qqKS8dCAKuewomK7ytC6w8C15f07EPSzGrYMb5GlTr5wP/m5uRW3MdT1SyT4XZwtEpsgM3bK4ISQQFLgIFBik4mBVODf+hDv7U1HX1fv/yBcjFQkHTFFvLQ01yFA5GBamDNsmPpk0oAwyvD26qyXKAy2nHIg+aq1oVlyc+Tw+rpafnwIbGRBUQLNGC216J+Jwy2aYD8rsR9O4xEnl4UK8EvQ== X-OriginatorOrg: starfivetech.com X-MS-Exchange-CrossTenant-Network-Message-Id: 76fc530e-90a2-4067-6083-08dc80563131 X-MS-Exchange-CrossTenant-AuthSource: NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2024 03:11:26.2786 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 06fe3fa3-1221-43d3-861b-5a4ee687a85c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: A38h/i4rk5PpYclyLRCSHU8pHJimFgVd3tQ8kpYIGJI0HH8aCyQBhUJmV0WfOkXBlVZdtsd7o4h4UIbUf3oPVK+B4knRJ4s/HsBvAQpGCcw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: NT0PR01MB1054 Adds separate dma hardware descriptor setup for JH8100 hardware quirks. JH8100 engine uses AXI1 master for data transfer but current dma driver is hardcoded to use AXI0 only. The FIFO offset needs to be incremented due to hardware limitations. Signed-off-by: Jia Jie Ho --- .../dma/dw-axi-dmac/dw-axi-dmac-platform.c | 32 ++++++++++++++++--- drivers/dma/dw-axi-dmac/dw-axi-dmac.h | 2 ++ include/linux/dma/dw_axi.h | 11 +++++++ 3 files changed, 40 insertions(+), 5 deletions(-) create mode 100644 include/linux/dma/dw_axi.h diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c index 18ce7d64958b..1a8f2e140848 100644 --- a/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c +++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac-platform.c @@ -648,6 +648,7 @@ static void set_desc_dest_master(struct axi_dma_hw_desc *hw_desc, static int dw_axi_dma_set_hw_desc(struct axi_dma_chan *chan, struct axi_dma_hw_desc *hw_desc, + struct axi_dma_desc *desc, dma_addr_t mem_addr, size_t len) { unsigned int data_width = BIT(chan->chip->dw->hdata->m_data_width); @@ -656,6 +657,8 @@ static int dw_axi_dma_set_hw_desc(struct axi_dma_chan *chan, dma_addr_t device_addr; size_t axi_block_ts; size_t block_ts; + bool hw_quirks = chan->quirks & DWAXIDMAC_STARFIVE_SM_ALGO; + u32 val; u32 ctllo, ctlhi; u32 burst_len, src_burst_trans_len, dst_burst_trans_len; @@ -676,7 +679,8 @@ static int dw_axi_dma_set_hw_desc(struct axi_dma_chan *chan, device_addr = chan->config.dst_addr; ctllo = reg_width << CH_CTL_L_DST_WIDTH_POS | mem_width << CH_CTL_L_SRC_WIDTH_POS | - DWAXIDMAC_CH_CTL_L_NOINC << CH_CTL_L_DST_INC_POS | + (hw_quirks ? DWAXIDMAC_CH_CTL_L_INC << CH_CTL_L_DST_INC_POS : + DWAXIDMAC_CH_CTL_L_NOINC << CH_CTL_L_DST_INC_POS) | DWAXIDMAC_CH_CTL_L_INC << CH_CTL_L_SRC_INC_POS; block_ts = len >> mem_width; break; @@ -686,7 +690,8 @@ static int dw_axi_dma_set_hw_desc(struct axi_dma_chan *chan, ctllo = reg_width << CH_CTL_L_SRC_WIDTH_POS | mem_width << CH_CTL_L_DST_WIDTH_POS | DWAXIDMAC_CH_CTL_L_INC << CH_CTL_L_DST_INC_POS | - DWAXIDMAC_CH_CTL_L_NOINC << CH_CTL_L_SRC_INC_POS; + (hw_quirks ? DWAXIDMAC_CH_CTL_L_INC << CH_CTL_L_SRC_INC_POS : + DWAXIDMAC_CH_CTL_L_NOINC << CH_CTL_L_SRC_INC_POS); block_ts = len >> reg_width; break; default: @@ -739,6 +744,17 @@ static int dw_axi_dma_set_hw_desc(struct axi_dma_chan *chan, set_desc_src_master(hw_desc); + if (hw_quirks) { + if (chan->direction == DMA_MEM_TO_DEV) { + set_desc_dest_master(hw_desc, desc); + } else { + /* Select AXI1 for src master */ + val = le32_to_cpu(hw_desc->lli->ctl_lo); + val |= CH_CTL_L_SRC_MAST; + hw_desc->lli->ctl_lo = cpu_to_le32(val); + } + } + hw_desc->len = len; return 0; } @@ -815,8 +831,8 @@ dw_axi_dma_chan_prep_cyclic(struct dma_chan *dchan, dma_addr_t dma_addr, for (i = 0; i < total_segments; i++) { hw_desc = &desc->hw_desc[i]; - status = dw_axi_dma_set_hw_desc(chan, hw_desc, src_addr, - segment_len); + status = dw_axi_dma_set_hw_desc(chan, hw_desc, NULL, + src_addr, segment_len); if (status < 0) goto err_desc_get; @@ -898,7 +914,8 @@ dw_axi_dma_chan_prep_slave_sg(struct dma_chan *dchan, struct scatterlist *sgl, do { hw_desc = &desc->hw_desc[loop++]; - status = dw_axi_dma_set_hw_desc(chan, hw_desc, mem, segment_len); + status = dw_axi_dma_set_hw_desc(chan, hw_desc, desc, + mem, segment_len); if (status < 0) goto err_desc_get; @@ -1036,8 +1053,13 @@ static int dw_axi_dma_chan_slave_config(struct dma_chan *dchan, struct dma_slave_config *config) { struct axi_dma_chan *chan = dchan_to_axi_dma_chan(dchan); + struct dw_axi_peripheral_config *periph = config->peripheral_config; memcpy(&chan->config, config, sizeof(*config)); + if (config->peripheral_size == sizeof(*periph)) + chan->quirks = periph->quirks; + else + chan->quirks = 0; return 0; } diff --git a/drivers/dma/dw-axi-dmac/dw-axi-dmac.h b/drivers/dma/dw-axi-dmac/dw-axi-dmac.h index d76d3d88ceb6..6a68c40c1bf3 100644 --- a/drivers/dma/dw-axi-dmac/dw-axi-dmac.h +++ b/drivers/dma/dw-axi-dmac/dw-axi-dmac.h @@ -14,6 +14,7 @@ #include #include #include +#include #include #include "../virt-dma.h" @@ -50,6 +51,7 @@ struct axi_dma_chan { struct dma_slave_config config; enum dma_transfer_direction direction; bool cyclic; + u32 quirks; /* these other elements are all protected by vc.lock */ bool is_paused; }; diff --git a/include/linux/dma/dw_axi.h b/include/linux/dma/dw_axi.h new file mode 100644 index 000000000000..fd49152869a4 --- /dev/null +++ b/include/linux/dma/dw_axi.h @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __LINUX_DMA_DW_AXI_H +#define __LINUX_DMA_DW_AXI_H + +#include + +struct dw_axi_peripheral_config { +#define DWAXIDMAC_STARFIVE_SM_ALGO BIT(0) + u32 quirks; +}; +#endif /* __LINUX_DMA_DW_AXI_H */ From patchwork Thu May 30 03:11:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jia Jie Ho X-Patchwork-Id: 800670 Received: from CHN02-SH0-obe.outbound.protection.partner.outlook.cn (mail-sh0chn02on2110.outbound.protection.partner.outlook.cn [139.219.146.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 979D11761BF; Thu, 30 May 2024 08:46:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=139.219.146.110 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717058805; cv=fail; b=pnpM/WOjZKmAxPKie52gPFE+7celwH5H3lh0BiULsNXbqqmt2wYkz9Jp3XdNo1xNgKTuwzo83yQ1rNW8C/CpfCQJAZmiIBB9FDLAs3YBTfDB/vuY9Riqn6xhRw//m3wOyN4S+L/KncNbjndMInjK+jM/QQo/neRKAvGcWU70t8E= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717058805; c=relaxed/simple; bh=dapI3b5PaHMoIjGWC0IErBk4j8KmA8iAQem7w5RHM8Q=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=nYtd5YfT9kEtWjYtx6HIJyQuwhgk8MyJWna1L7376oJjwkNVVSuG9fTF8AKAIOasd7mvpOO5PDFjCUUlKhr+mCOCJIGbi31lyJ89w/InUJGpE45MlWnW4LIpro5eZgKlEGUvLK+is2/ZVz5emccZV4fdlo+b6mvw9ODRnEF5+qk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com; spf=pass smtp.mailfrom=starfivetech.com; arc=fail smtp.client-ip=139.219.146.110 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=starfivetech.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IJmCeUDtk9s4aOKwKwQZyM7tPanyzoq9iMO2m5cKmxgO7+aeKYYwXLSuhW9im2FnT1EqDcU8hHygWHXOugunB6MOh0gRK/ZAQuEmbMoB2DtqFa7Xi7gVssBOSdVyNsZXaqs3q63SmlKZnLbrkRoi1Vf1OZBnb4cgHt0zrTOAgk84N7HT76uB8pPA4aIodzHJU0iZ4xyO9j+X3Cm0gLDlHRZUXiGfAsROMoeVMhgQjmE2fOM+/MudzVz9phf2wFix7LU7VCOoapkCD6hBr6ivMd0IVSDhp2HJRS+1NqWQ5NxK8yfdAaV1a/4nqxcdAXKlF9xZcyX0PJYuoa8VG+sWCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=17vlXwX0vAm2ykk21rIX6088jcmuyw1PiyRbe6nJUMk=; b=YMKpyH7cNkfDd2wWnNltFjbp9MCNfSOKMVn58GDgpa/fcIHPOvAk1ug077dBqusXtWyxh4R7H7My7MJMr3R+gjt8GdYsoPsf6EumZ7G2Ha8w2bj2JAjx2BsLdT3dYyUv0yxrnYnrdXu3B9w6obCYX9s/S2qdoJ6fAyazV8l906zxzF6ma8Ld1Cj2OxdAOKxb6DQt27O+ZjWmKdb7nB1xWaqp/dORlCQxGsAzRpGywX+RIlR6ms3dRHceqex/VieVzSz7r+iwW78enf7bZ2i74uEDofGnc13o+na8Q+jgkM3bVt9FVHm+WMvvAz3nzXVeKppW5NBfSUkmFxOeY7yDWA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=starfivetech.com; dmarc=pass action=none header.from=starfivetech.com; dkim=pass header.d=starfivetech.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=starfivetech.com; Received: from NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn (2406:e500:c510:10::10) by NT0PR01MB1054.CHNPR01.prod.partner.outlook.cn (2406:e500:c510:2::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.36; Thu, 30 May 2024 03:11:28 +0000 Received: from NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn ([fe80::e3b:43f8:2e6d:ecce]) by NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn ([fe80::e3b:43f8:2e6d:ecce%7]) with mapi id 15.20.7587.037; Thu, 30 May 2024 03:11:28 +0000 From: Jia Jie Ho To: Herbert Xu , "David S . Miller" , Eugeniy Paltsev , Vinod Koul , linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v5 2/3] crypto: starfive: Add sm3 support for JH8100 Date: Thu, 30 May 2024 11:11:11 +0800 Message-ID: <20240530031112.4952-3-jiajie.ho@starfivetech.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240530031112.4952-1-jiajie.ho@starfivetech.com> References: <20240530031112.4952-1-jiajie.ho@starfivetech.com> X-ClientProxiedBy: ZQ0PR01CA0027.CHNPR01.prod.partner.outlook.cn (2406:e500:c550:2::17) To NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn (2406:e500:c510:10::10) Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: NT0PR01MB1182:EE_|NT0PR01MB1054:EE_ X-MS-Office365-Filtering-Correlation-Id: f01d970c-d301-40ff-6adb-08dc80563244 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; ARA:13230031|366007|41320700004|1800799015|52116005|38350700005; X-Microsoft-Antispam-Message-Info: vt03g4cISiYEhf43EzolKGEX3nZvIsZr2ANytpdWane35g50BMzoxfOTpn3OWPs1pb+LQJCTdxfPsc5hx4AJyzRQcewJxfC4TwP/bxIrFEmBcKGKozIOhWRflAebYnzlzoqQoTgypOZUWe27NnYMzLYTX8l6AzXVlvt6UzDYG6wk7fSxkTrbqXnv4dj/ShWVIXsj527XIUnVBcR6iuPoxdRiHu27nAmCzu/YWNdHStWqFdU6BP+44K1Kzw9mXiQQS1NXaP7nSBTjULibJMFgNPNDnpIZ8o7embCqYikRA82Mu86rlTn54UPkepEJ6j6Tb2s+L+waSRf+ALqyGsOESt0ybZWWO21Qa6aWRpCQRW8S+vDXXInkR3CdufnVypgy9Dvg53JyzXxkwrTLFuiN9Ur7QyGjvf2xpupIznD47UksszRv8WoLrTRja2jce6pG5iOEKU0ZYyO3f2ZcsUrH/RlZLg20DB2BF+4q4uUmC5Tf5A6p8TRHBfSbI6UoJ/Tdc6WA0Pq0VNu2fL4bK2m13B0pWASQWpZW/kzJ/27geZxGAy2w5XdieVNJisnhHZEanGOj3Zb+QsSnFtlCvKtI0te5NsOmlIriUjOoeaDdgzjZZoHG1XZau7SU8BD/PxR+ X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn; PTR:; CAT:NONE; SFS:(13230031)(366007)(41320700004)(1800799015)(52116005)(38350700005); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qjLVRx2CXlJq/FV+Nzivlr2OPyNGRR5xCktfHYdlb7UUsd5N1l3LuGzvr/7oNH6NDNyX/sOo9Obg68RbFV2nySYWAojy9UQZWngQbKf3HIfznrzoCgl0vvVLQYFj+P9IQP/eLdeSIZ1UIBfdqTQuJ4rC+y0QVzfPU96yFTD24dh3P4m7zS/n/br46luGlQN8i6Jk3RXEsafhFQyDq0IleomKRx/jqcBhHPSL/x6mBOz3MSvMsKzkVcWBhNDdp9VN2E06MZ+q3rGp0+TRVQ87Hbg4H6tIjYlgr6Hb043TRundqKGJmwPnDSRRa2SA0yGMENf8ErSzMrUxKJBbimPFzh2HfVpRw18lpROlRRhlGYrqbvvBFU6n33g/itCJcC+fNFnjDRKnZ2Kif4a9JRd0850bdnGxhHVrdGaYTDqqJDX+fc8onpluLd5ITe5dnXwLBvX/sJbgOoaqvpiI/rf9m09VeSadUI4urz2Ne5yhg1vGmXpn2LZR/7dOVF5qPyqcxqoCBeH4dq8hkWeWKVVaVNQ1akU2ewCOM+FzA8flljkKVMEGXuenc4Z5Ypu06+Ra5ysKxzFhXg+PWLaw3VdoXs08vP01PPyLMTv+VnL4Gk5CDzkA5KDh2qSfVG+liw2+QXWKAtitMg4PW084hrzZ6Gu0dH7yFWM01yjSb1vRNd4yqhq9/r1YwUhY4bxJia5EEzxf+K0+Vwkx8HOZSAjvYdyVYkAOriFgDT/XihTk4S3BXsfdy2vFoWSoHWuervA0f45926DqMUDolVAZpNF9alunXXOigrik9qWXqEciGjGKk9kVsFJfuf835ecaetP1AL2kFYUA9f6yaneWtPL3gySqmJ1MeevufFJ5A6Dq4SkZ/2VZJzvxwcgomgdQX8sigETET9XUHZz+7izE4x9pxcMw0GGUPXgJMYi4yvZSworMaS7v0QDTrskW6dSdFF2Pw6Oe4ceeet9Z/Kaq7pV+J2MqhUQCh0+ZFgI/CekrkLUiPogJGi0fckmJX2S/DiO5VrHb1xhTZKApoRy1AacaHkfVlyHqRCn3qi1e9Pd8xLtEejEI1Gd8LOMZJHDubFM6phPUMIFhvOcoHfPCIc2dW0w5NM6lBQ61XDiF0eP+sfUsZ7ID2xb+j+aUgpvo4wyxLOlujMDfEyeoa8UjWZU/QTsJmqufpbOwUQcfj7mS2qqooIvy4zyWlZ/LKOAUp4paxXeLjv5ONYhZ8WjBFqbyyxMAhCcNOokpi6R7NkMBuH70TiWAEhVB1ulnDPrJS1eZ93jx5PMgE+RcrAXQqNLV3Ge8XxGQX3w7LbY6c0+t5Ec672bHWKwv0JnB/eqB3XJXmPSl7UtqcRwMNl8swPLFfDyg6ql+bPwlGjKPyB2ulCJwgqa6+ZpbTuqqm/8rt+NJaWJJ409BmrVMiM0Fzt7NgsN3RDewSgtObF7whVdaxi1g99ksrmZsJTm0sO1I0+BacVj0tLkY/bokoD685ivSpWPx+gxlRX8egqh5LY3q9zQ12dk4Xe/WeFaByNXUcL2Q6k9jKyTWbr4W/L3VfA9ASl/QAW9KdOe2tGbRAfUEJcUxeCU2FJLyyzfGdR5tZwY0NOWFdVqBP4fnL4c6ZEk83g== X-OriginatorOrg: starfivetech.com X-MS-Exchange-CrossTenant-Network-Message-Id: f01d970c-d301-40ff-6adb-08dc80563244 X-MS-Exchange-CrossTenant-AuthSource: NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2024 03:11:28.0984 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 06fe3fa3-1221-43d3-861b-5a4ee687a85c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AFkluPg5LEbSTtyqzf0rmsS/q2yLPxktI5CJNUHOjW1XejHShjH8KkPtdrMrxFRV2tB32c2CFDS6QGlCaJYc9pi5Skydq8Jc+9Nd0fmVaMY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: NT0PR01MB1054 Add driver support for SM3 hash/HMAC for JH8100 SoC. JH8100 contains a separate SM algo engine and new dedicated dma that supports 64-bit address access. Signed-off-by: Jia Jie Ho --- drivers/crypto/starfive/Kconfig | 25 +- drivers/crypto/starfive/Makefile | 5 +- drivers/crypto/starfive/jh7110-aes.c | 3 + drivers/crypto/starfive/jh7110-cryp.c | 28 +- drivers/crypto/starfive/jh7110-cryp.h | 67 +++- drivers/crypto/starfive/jh7110-hash.c | 45 ++- drivers/crypto/starfive/jh8100-sm3.c | 544 ++++++++++++++++++++++++++ 7 files changed, 696 insertions(+), 21 deletions(-) create mode 100644 drivers/crypto/starfive/jh8100-sm3.c diff --git a/drivers/crypto/starfive/Kconfig b/drivers/crypto/starfive/Kconfig index 0fe389e9f932..e6bf02d0ed1f 100644 --- a/drivers/crypto/starfive/Kconfig +++ b/drivers/crypto/starfive/Kconfig @@ -5,7 +5,7 @@ config CRYPTO_DEV_JH7110 tristate "StarFive JH7110 cryptographic engine driver" depends on (SOC_STARFIVE && AMBA_PL08X) || COMPILE_TEST - depends on HAS_DMA + depends on HAS_DMA && !CRYPTO_DEV_JH8100 select CRYPTO_ENGINE select CRYPTO_HMAC select CRYPTO_SHA256 @@ -24,3 +24,26 @@ config CRYPTO_DEV_JH7110 skciphers, AEAD and hash functions. If you choose 'M' here, this module will be called jh7110-crypto. + +config CRYPTO_DEV_JH8100 + tristate "StarFive JH8100 cryptographic engine drivers" + depends on (SOC_STARFIVE && DW_AXI_DMAC) || COMPILE_TEST + depends on HAS_DMA + select CRYPTO_ENGINE + select CRYPTO_HMAC + select CRYPTO_SHA256 + select CRYPTO_SHA512 + select CRYPTO_SM3_GENERIC + select CRYPTO_RSA + select CRYPTO_AES + select CRYPTO_CCM + select CRYPTO_GCM + select CRYPTO_CBC + select CRYPTO_ECB + select CRYPTO_CTR + help + Support for StarFive JH8100 crypto hardware acceleration engine. + This module provides additional support for SM2 signature verification, + SM3 hash/hmac functions and SM4 skcipher. + + If you choose 'M' here, this module will be called jh8100-crypto. diff --git a/drivers/crypto/starfive/Makefile b/drivers/crypto/starfive/Makefile index 8c137afe58ad..867ce035af19 100644 --- a/drivers/crypto/starfive/Makefile +++ b/drivers/crypto/starfive/Makefile @@ -1,4 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CRYPTO_DEV_JH7110) += jh7110-crypto.o -jh7110-crypto-objs := jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-aes.o +jh7110-crypto-objs := jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-aes.o jh8100-sm3.o + +obj-$(CONFIG_CRYPTO_DEV_JH8100) += jh8100-crypto.o +jh8100-crypto-objs := jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-aes.o jh8100-sm3.o diff --git a/drivers/crypto/starfive/jh7110-aes.c b/drivers/crypto/starfive/jh7110-aes.c index 86a1a1fa9f8f..45440a6a29d4 100644 --- a/drivers/crypto/starfive/jh7110-aes.c +++ b/drivers/crypto/starfive/jh7110-aes.c @@ -413,6 +413,9 @@ static void starfive_aes_dma_done(void *param) static void starfive_aes_dma_init(struct starfive_cryp_dev *cryp) { + memset(&cryp->cfg_in, 0, sizeof(struct dma_slave_config)); + memset(&cryp->cfg_out, 0, sizeof(struct dma_slave_config)); + cryp->cfg_in.direction = DMA_MEM_TO_DEV; cryp->cfg_in.src_addr_width = DMA_SLAVE_BUSWIDTH_16_BYTES; cryp->cfg_in.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES; diff --git a/drivers/crypto/starfive/jh7110-cryp.c b/drivers/crypto/starfive/jh7110-cryp.c index e4dfed7ee0b0..19bbcaaec18d 100644 --- a/drivers/crypto/starfive/jh7110-cryp.c +++ b/drivers/crypto/starfive/jh7110-cryp.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -99,6 +100,8 @@ static int starfive_cryp_probe(struct platform_device *pdev) if (!cryp) return -ENOMEM; + cryp->type = (uintptr_t)of_device_get_match_data(&pdev->dev); + platform_set_drvdata(pdev, cryp); cryp->dev = &pdev->dev; @@ -153,7 +156,7 @@ static int starfive_cryp_probe(struct platform_device *pdev) if (ret) goto err_algs_aes; - ret = starfive_hash_register_algs(); + ret = starfive_hash_register_algs(cryp); if (ret) goto err_algs_hash; @@ -161,10 +164,18 @@ static int starfive_cryp_probe(struct platform_device *pdev) if (ret) goto err_algs_rsa; + if (cryp->type == STARFIVE_CRYPTO_JH8100) { + ret = starfive_sm3_register_algs(); + if (ret) + goto err_algs_sm3; + } + return 0; +err_algs_sm3: + starfive_rsa_unregister_algs(); err_algs_rsa: - starfive_hash_unregister_algs(); + starfive_hash_unregister_algs(cryp); err_algs_hash: starfive_aes_unregister_algs(); err_algs_aes: @@ -190,9 +201,12 @@ static void starfive_cryp_remove(struct platform_device *pdev) struct starfive_cryp_dev *cryp = platform_get_drvdata(pdev); starfive_aes_unregister_algs(); - starfive_hash_unregister_algs(); + starfive_hash_unregister_algs(cryp); starfive_rsa_unregister_algs(); + if (cryp->type == STARFIVE_CRYPTO_JH8100) + starfive_sm3_unregister_algs(); + crypto_engine_stop(cryp->engine); crypto_engine_exit(cryp->engine); @@ -208,7 +222,13 @@ static void starfive_cryp_remove(struct platform_device *pdev) } static const struct of_device_id starfive_dt_ids[] __maybe_unused = { - { .compatible = "starfive,jh7110-crypto", .data = NULL}, + { + .compatible = "starfive,jh7110-crypto", + .data = (const void *)STARFIVE_CRYPTO_JH7110, + }, { + .compatible = "starfive,jh8100-crypto", + .data = (const void *)STARFIVE_CRYPTO_JH8100, + }, {}, }; MODULE_DEVICE_TABLE(of, starfive_dt_ids); diff --git a/drivers/crypto/starfive/jh7110-cryp.h b/drivers/crypto/starfive/jh7110-cryp.h index eeb4e2b9655f..60ac752352c8 100644 --- a/drivers/crypto/starfive/jh7110-cryp.h +++ b/drivers/crypto/starfive/jh7110-cryp.h @@ -19,18 +19,34 @@ #define STARFIVE_DMA_IN_LEN_OFFSET 0x10 #define STARFIVE_DMA_OUT_LEN_OFFSET 0x14 +#define STARFIVE_SM_ALG_CR_OFFSET 0x4000 +#define STARFIVE_SM_IE_MASK_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x4) +#define STARFIVE_SM_IE_FLAG_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x8) +#define STARFIVE_SM_DMA_IN_LEN_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0xc) +#define STARFIVE_SM_DMA_OUT_LEN_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x10) +#define STARFIVE_SM_ALG_FIFO_IN_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x20) +#define STARFIVE_SM_ALG_FIFO_OUT_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x28) + #define STARFIVE_IE_MASK_AES_DONE 0x1 #define STARFIVE_IE_MASK_HASH_DONE 0x4 #define STARFIVE_IE_MASK_PKA_DONE 0x8 #define STARFIVE_IE_FLAG_AES_DONE 0x1 #define STARFIVE_IE_FLAG_HASH_DONE 0x4 #define STARFIVE_IE_FLAG_PKA_DONE 0x8 +#define STARFIVE_SM_IE_MASK_SM3_DONE 0x2 +#define STARFIVE_SM_IE_FLAG_SM3_DONE 0x2 #define STARFIVE_MSG_BUFFER_SIZE SZ_16K #define MAX_KEY_SIZE SHA512_BLOCK_SIZE #define STARFIVE_AES_IV_LEN AES_BLOCK_SIZE #define STARFIVE_AES_CTR_LEN AES_BLOCK_SIZE +enum starfive_crypto_type { + STARFIVE_CRYPTO_UNKNOWN = 0, + STARFIVE_CRYPTO_JH7110, + STARFIVE_CRYPTO_JH8100, +}; + union starfive_aes_csr { u32 v; struct { @@ -68,6 +84,20 @@ union starfive_aes_csr { }; }; +union starfive_sm_alg_cr { + u32 v; + struct { + u32 start :1; + u32 sm4_dma_en :1; + u32 sm3_dma_en :1; + u32 rsvd_0 :1; + u32 alg_done :1; + u32 rsvd_1 :3; + u32 clear :1; + u32 rsvd_2 :23; + }; +}; + union starfive_hash_csr { u32 v; struct { @@ -133,6 +163,33 @@ union starfive_pka_casr { }; }; +union starfive_sm3_csr { + u32 v; + struct { + u32 start :1; + u32 reset :1; + u32 ie :1; + u32 firstb :1; +#define STARFIVE_SM3_MODE 0x0 + u32 mode :3; + u32 rsvd_0 :1; + u32 final :1; + u32 rsvd_1 :2; +#define STARFIVE_SM3_HMAC_FLAGS 0x800 + u32 hmac :1; + u32 rsvd_2 :1; +#define STARFIVE_SM3_KEY_DONE BIT(13) + u32 key_done :1; + u32 key_flag :1; +#define STARFIVE_SM3_HMAC_DONE BIT(15) + u32 hmac_done :1; +#define STARFIVE_SM3_BUSY BIT(16) + u32 busy :1; + u32 hashdone :1; + u32 rsvd_3 :14; + }; +}; + struct starfive_rsa_key { u8 *n; u8 *e; @@ -178,7 +235,7 @@ struct starfive_cryp_dev { struct clk *hclk; struct clk *ahb; struct reset_control *rst; - + enum starfive_crypto_type type; void __iomem *base; phys_addr_t phys_base; @@ -211,6 +268,7 @@ struct starfive_cryp_request_ctx { union starfive_hash_csr hash; union starfive_pka_cacr pka; union starfive_aes_csr aes; + union starfive_sm3_csr sm3; } csr; struct scatterlist *in_sg; @@ -226,12 +284,15 @@ struct starfive_cryp_request_ctx { struct starfive_cryp_dev *starfive_cryp_find_dev(struct starfive_cryp_ctx *ctx); -int starfive_hash_register_algs(void); -void starfive_hash_unregister_algs(void); +int starfive_hash_register_algs(struct starfive_cryp_dev *cryp); +void starfive_hash_unregister_algs(struct starfive_cryp_dev *cryp); int starfive_rsa_register_algs(void); void starfive_rsa_unregister_algs(void); int starfive_aes_register_algs(void); void starfive_aes_unregister_algs(void); + +int starfive_sm3_register_algs(void); +void starfive_sm3_unregister_algs(void); #endif diff --git a/drivers/crypto/starfive/jh7110-hash.c b/drivers/crypto/starfive/jh7110-hash.c index 2c60a1047bc3..9673cdbfb554 100644 --- a/drivers/crypto/starfive/jh7110-hash.c +++ b/drivers/crypto/starfive/jh7110-hash.c @@ -110,6 +110,8 @@ static void starfive_hash_dma_callback(void *param) static void starfive_hash_dma_init(struct starfive_cryp_dev *cryp) { + memset(&cryp->cfg_in, 0, sizeof(struct dma_slave_config)); + cryp->cfg_in.src_addr_width = DMA_SLAVE_BUSWIDTH_16_BYTES; cryp->cfg_in.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES; cryp->cfg_in.src_maxburst = cryp->dma_maxburst; @@ -515,12 +517,6 @@ static int starfive_sha512_init_tfm(struct crypto_ahash *hash) STARFIVE_HASH_SHA512, 0); } -static int starfive_sm3_init_tfm(struct crypto_ahash *hash) -{ - return starfive_hash_init_tfm(hash, "sm3-generic", - STARFIVE_HASH_SM3, 0); -} - static int starfive_hmac_sha224_init_tfm(struct crypto_ahash *hash) { return starfive_hash_init_tfm(hash, "hmac(sha224-generic)", @@ -545,13 +541,19 @@ static int starfive_hmac_sha512_init_tfm(struct crypto_ahash *hash) STARFIVE_HASH_SHA512, 1); } +static int starfive_sm3_init_tfm(struct crypto_ahash *hash) +{ + return starfive_hash_init_tfm(hash, "sm3-generic", + STARFIVE_HASH_SM3, 0); +} + static int starfive_hmac_sm3_init_tfm(struct crypto_ahash *hash) { return starfive_hash_init_tfm(hash, "hmac(sm3-generic)", STARFIVE_HASH_SM3, 1); } -static struct ahash_engine_alg algs_sha2_sm3[] = { +static struct ahash_engine_alg algs_sha2[] = { { .base.init = starfive_hash_init, .base.update = starfive_hash_update, @@ -780,7 +782,11 @@ static struct ahash_engine_alg algs_sha2_sm3[] = { .op = { .do_one_request = starfive_hash_one_request, }, -}, { +}, +}; + +static struct ahash_engine_alg algs_sm3[] = { +{ .base.init = starfive_hash_init, .base.update = starfive_hash_update, .base.final = starfive_hash_final, @@ -840,12 +846,27 @@ static struct ahash_engine_alg algs_sha2_sm3[] = { }, }; -int starfive_hash_register_algs(void) +int starfive_hash_register_algs(struct starfive_cryp_dev *cryp) { - return crypto_engine_register_ahashes(algs_sha2_sm3, ARRAY_SIZE(algs_sha2_sm3)); + int ret; + + ret = crypto_engine_register_ahashes(algs_sha2, ARRAY_SIZE(algs_sha2)); + if (ret) + return ret; + + if (cryp->type == STARFIVE_CRYPTO_JH7110) { + ret = crypto_engine_register_ahashes(algs_sm3, ARRAY_SIZE(algs_sm3)); + if (ret) + crypto_engine_unregister_ahashes(algs_sha2, ARRAY_SIZE(algs_sha2)); + } + + return ret; } -void starfive_hash_unregister_algs(void) +void starfive_hash_unregister_algs(struct starfive_cryp_dev *cryp) { - crypto_engine_unregister_ahashes(algs_sha2_sm3, ARRAY_SIZE(algs_sha2_sm3)); + crypto_engine_unregister_ahashes(algs_sha2, ARRAY_SIZE(algs_sha2)); + + if (cryp->type == STARFIVE_CRYPTO_JH7110) + crypto_engine_unregister_ahashes(algs_sm3, ARRAY_SIZE(algs_sm3)); } diff --git a/drivers/crypto/starfive/jh8100-sm3.c b/drivers/crypto/starfive/jh8100-sm3.c new file mode 100644 index 000000000000..4c7685d25851 --- /dev/null +++ b/drivers/crypto/starfive/jh8100-sm3.c @@ -0,0 +1,544 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * SM3 Hash function and HMAC support for StarFive driver + * + * Copyright (c) 2022 - 2023 StarFive Technology + * + */ + +#include +#include +#include +#include +#include "jh7110-cryp.h" +#include +#include +#include +#include + +#define STARFIVE_SM3_REGS_OFFSET 0x4200 +#define STARFIVE_SM3_CSR (STARFIVE_SM3_REGS_OFFSET + 0x0) +#define STARFIVE_SM3_WDR (STARFIVE_SM3_REGS_OFFSET + 0x4) +#define STARFIVE_SM3_RDR (STARFIVE_SM3_REGS_OFFSET + 0x8) +#define STARFIVE_SM3_WSR (STARFIVE_SM3_REGS_OFFSET + 0xC) +#define STARFIVE_SM3_WLEN3 (STARFIVE_SM3_REGS_OFFSET + 0x10) +#define STARFIVE_SM3_WLEN2 (STARFIVE_SM3_REGS_OFFSET + 0x14) +#define STARFIVE_SM3_WLEN1 (STARFIVE_SM3_REGS_OFFSET + 0x18) +#define STARFIVE_SM3_WLEN0 (STARFIVE_SM3_REGS_OFFSET + 0x1C) +#define STARFIVE_SM3_WKR (STARFIVE_SM3_REGS_OFFSET + 0x20) +#define STARFIVE_SM3_WKLEN (STARFIVE_SM3_REGS_OFFSET + 0x24) + +#define STARFIVE_SM3_BUFLEN SHA512_BLOCK_SIZE +#define STARFIVE_SM3_RESET 0x2 + +static inline int starfive_sm3_wait_busy(struct starfive_cryp_dev *cryp) +{ + u32 status; + + return readl_relaxed_poll_timeout(cryp->base + STARFIVE_SM3_CSR, status, + !(status & STARFIVE_SM3_BUSY), 10, 100000); +} + +static inline int starfive_sm3_wait_hmac_done(struct starfive_cryp_dev *cryp) +{ + u32 status; + + return readl_relaxed_poll_timeout(cryp->base + STARFIVE_SM3_CSR, status, + (status & STARFIVE_SM3_HMAC_DONE), 10, 100000); +} + +static inline int starfive_sm3_wait_key_done(struct starfive_cryp_dev *cryp) +{ + u32 status; + + return readl_relaxed_poll_timeout(cryp->base + STARFIVE_SM3_CSR, status, + (status & STARFIVE_SM3_KEY_DONE), 10, 100000); +} + +static int starfive_sm3_hmac_key(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_request_ctx *rctx = ctx->rctx; + struct starfive_cryp_dev *cryp = ctx->cryp; + int klen = ctx->keylen, loop; + unsigned int *key = (unsigned int *)ctx->key; + unsigned char *cl; + + writel(ctx->keylen, cryp->base + STARFIVE_SM3_WKLEN); + + rctx->csr.sm3.hmac = 1; + rctx->csr.sm3.key_flag = 1; + + writel(rctx->csr.sm3.v, cryp->base + STARFIVE_SM3_CSR); + + for (loop = 0; loop < klen / sizeof(unsigned int); loop++, key++) + writel(*key, cryp->base + STARFIVE_SM3_WKR); + + if (klen & 0x3) { + cl = (unsigned char *)key; + for (loop = 0; loop < (klen & 0x3); loop++, cl++) + writeb(*cl, cryp->base + STARFIVE_SM3_WKR); + } + + if (starfive_sm3_wait_key_done(cryp)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, + "starfive_sm3_wait_key_done error\n"); + + return 0; +} + +static void starfive_sm3_start(struct starfive_cryp_dev *cryp) +{ + union starfive_sm3_csr csr; + + csr.v = readl(cryp->base + STARFIVE_SM3_CSR); + csr.firstb = 0; + csr.final = 1; + writel(csr.v, cryp->base + STARFIVE_SM3_CSR); +} + +static void starfive_sm3_dma_callback(void *param) +{ + struct starfive_cryp_dev *cryp = param; + + complete(&cryp->dma_done); +} + +static void starfive_sm3_dma_init(struct starfive_cryp_dev *cryp) +{ + struct dw_axi_peripheral_config periph_conf = {}; + + memset(&cryp->cfg_in, 0, sizeof(struct dma_slave_config)); + periph_conf.quirks = DWAXIDMAC_STARFIVE_SM_ALGO; + + cryp->cfg_in.direction = DMA_MEM_TO_DEV; + cryp->cfg_in.src_addr_width = DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_in.dst_addr_width = DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_in.src_maxburst = cryp->dma_maxburst; + cryp->cfg_in.dst_maxburst = cryp->dma_maxburst; + cryp->cfg_in.dst_addr = cryp->phys_base + STARFIVE_SM_ALG_FIFO_IN_OFFSET; + cryp->cfg_in.peripheral_config = &periph_conf; + cryp->cfg_in.peripheral_size = sizeof(struct dw_axi_peripheral_config); + + dmaengine_slave_config(cryp->tx, &cryp->cfg_in); + + init_completion(&cryp->dma_done); +} + +static int starfive_sm3_dma_xfer(struct starfive_cryp_dev *cryp, + struct scatterlist *sg) +{ + struct dma_async_tx_descriptor *in_desc; + union starfive_sm_alg_cr alg_cr; + int ret = 0; + + alg_cr.v = 0; + alg_cr.start = 1; + alg_cr.sm3_dma_en = 1; + writel(alg_cr.v, cryp->base + STARFIVE_SM_ALG_CR_OFFSET); + + writel(sg_dma_len(sg), cryp->base + STARFIVE_SM_DMA_IN_LEN_OFFSET); + sg_dma_len(sg) = ALIGN(sg_dma_len(sg), sizeof(u32)); + + in_desc = dmaengine_prep_slave_sg(cryp->tx, sg, 1, DMA_MEM_TO_DEV, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + if (!in_desc) { + ret = -EINVAL; + goto end; + } + + reinit_completion(&cryp->dma_done); + in_desc->callback = starfive_sm3_dma_callback; + in_desc->callback_param = cryp; + + dmaengine_submit(in_desc); + dma_async_issue_pending(cryp->tx); + + if (!wait_for_completion_timeout(&cryp->dma_done, + msecs_to_jiffies(1000))) + ret = -ETIMEDOUT; + +end: + alg_cr.v = 0; + alg_cr.clear = 1; + writel(alg_cr.v, cryp->base + STARFIVE_SM_ALG_CR_OFFSET); + + return ret; +} + +static int starfive_sm3_copy_hash(struct ahash_request *req) +{ + struct starfive_cryp_request_ctx *rctx = ahash_request_ctx(req); + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(crypto_ahash_reqtfm(req)); + int count, *data; + int mlen; + + if (!req->result) + return 0; + + mlen = rctx->digsize / sizeof(u32); + data = (u32 *)req->result; + + for (count = 0; count < mlen; count++) + data[count] = readl(ctx->cryp->base + STARFIVE_SM3_RDR); + + return 0; +} + +static void starfive_sm3_done_task(struct starfive_cryp_dev *cryp) +{ + int err = cryp->err; + + if (!err) + err = starfive_sm3_copy_hash(cryp->req.hreq); + + crypto_finalize_hash_request(cryp->engine, cryp->req.hreq, err); +} + +static int starfive_sm3_one_request(struct crypto_engine *engine, void *areq) +{ + struct ahash_request *req = + container_of(areq, struct ahash_request, base); + struct starfive_cryp_ctx *ctx = + crypto_ahash_ctx(crypto_ahash_reqtfm(req)); + struct starfive_cryp_dev *cryp = ctx->cryp; + struct starfive_cryp_request_ctx *rctx = ctx->rctx; + struct scatterlist *tsg; + int ret, src_nents, i; + + rctx->csr.sm3.v = 0; + rctx->csr.sm3.reset = 1; + writel(rctx->csr.sm3.v, cryp->base + STARFIVE_SM3_CSR); + + if (starfive_sm3_wait_busy(cryp)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, "Error resetting hardware.\n"); + + cryp->err = 0; + rctx->csr.sm3.v = 0; + rctx->csr.sm3.mode = ctx->hash_mode; + + if (ctx->is_hmac) { + ret = starfive_sm3_hmac_key(ctx); + if (ret) + return ret; + } else { + rctx->csr.sm3.start = 1; + rctx->csr.sm3.firstb = 1; + writel(rctx->csr.sm3.v, cryp->base + STARFIVE_SM3_CSR); + } + + /* No input message, get digest and end. */ + if (!rctx->total) + goto hash_start; + + starfive_sm3_dma_init(cryp); + + for_each_sg(rctx->in_sg, tsg, rctx->in_sg_len, i) { + src_nents = dma_map_sg(cryp->dev, tsg, 1, DMA_TO_DEVICE); + if (src_nents == 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg error\n"); + + ret = starfive_sm3_dma_xfer(cryp, tsg); + dma_unmap_sg(cryp->dev, tsg, 1, DMA_TO_DEVICE); + if (ret) + return ret; + } + +hash_start: + starfive_sm3_start(cryp); + + if (starfive_sm3_wait_busy(cryp)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, "Error generating digest.\n"); + + if (ctx->is_hmac) + cryp->err = starfive_sm3_wait_hmac_done(cryp); + + starfive_sm3_done_task(cryp); + + return 0; +} + +static void starfive_sm3_set_ahash(struct ahash_request *req, + struct starfive_cryp_ctx *ctx, + struct starfive_cryp_request_ctx *rctx) +{ + ahash_request_set_tfm(&rctx->ahash_fbk_req, ctx->ahash_fbk); + ahash_request_set_callback(&rctx->ahash_fbk_req, + req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP, + req->base.complete, req->base.data); + ahash_request_set_crypt(&rctx->ahash_fbk_req, req->src, + req->result, req->nbytes); +} + +static int starfive_sm3_init(struct ahash_request *req) +{ + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct starfive_cryp_request_ctx *rctx = ahash_request_ctx(req); + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(tfm); + + starfive_sm3_set_ahash(req, ctx, rctx); + + return crypto_ahash_init(&rctx->ahash_fbk_req); +} + +static int starfive_sm3_update(struct ahash_request *req) +{ + struct starfive_cryp_request_ctx *rctx = ahash_request_ctx(req); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(tfm); + + starfive_sm3_set_ahash(req, ctx, rctx); + + return crypto_ahash_update(&rctx->ahash_fbk_req); +} + +static int starfive_sm3_final(struct ahash_request *req) +{ + struct starfive_cryp_request_ctx *rctx = ahash_request_ctx(req); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(tfm); + + starfive_sm3_set_ahash(req, ctx, rctx); + + return crypto_ahash_final(&rctx->ahash_fbk_req); +} + +static int starfive_sm3_finup(struct ahash_request *req) +{ + struct starfive_cryp_request_ctx *rctx = ahash_request_ctx(req); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(tfm); + + starfive_sm3_set_ahash(req, ctx, rctx); + + return crypto_ahash_finup(&rctx->ahash_fbk_req); +} + +static int starfive_sm3_digest(struct ahash_request *req) +{ + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(tfm); + struct starfive_cryp_request_ctx *rctx = ahash_request_ctx(req); + struct starfive_cryp_dev *cryp = ctx->cryp; + + memset(rctx, 0, sizeof(struct starfive_cryp_request_ctx)); + + cryp->req.hreq = req; + rctx->total = req->nbytes; + rctx->in_sg = req->src; + rctx->blksize = crypto_tfm_alg_blocksize(crypto_ahash_tfm(tfm)); + rctx->digsize = crypto_ahash_digestsize(tfm); + rctx->in_sg_len = sg_nents_for_len(rctx->in_sg, rctx->total); + ctx->rctx = rctx; + + return crypto_transfer_hash_request_to_engine(cryp->engine, req); +} + +static int starfive_sm3_export(struct ahash_request *req, void *out) +{ + struct starfive_cryp_request_ctx *rctx = ahash_request_ctx(req); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(tfm); + + ahash_request_set_tfm(&rctx->ahash_fbk_req, ctx->ahash_fbk); + ahash_request_set_callback(&rctx->ahash_fbk_req, + req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP, + req->base.complete, req->base.data); + + return crypto_ahash_export(&rctx->ahash_fbk_req, out); +} + +static int starfive_sm3_import(struct ahash_request *req, const void *in) +{ + struct starfive_cryp_request_ctx *rctx = ahash_request_ctx(req); + struct crypto_ahash *tfm = crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(tfm); + + ahash_request_set_tfm(&rctx->ahash_fbk_req, ctx->ahash_fbk); + ahash_request_set_callback(&rctx->ahash_fbk_req, + req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP, + req->base.complete, req->base.data); + + return crypto_ahash_import(&rctx->ahash_fbk_req, in); +} + +static int starfive_sm3_init_algo(struct crypto_ahash *hash, + const char *alg_name, + bool is_hmac) +{ + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(hash); + + ctx->cryp = starfive_cryp_find_dev(ctx); + if (!ctx->cryp) + return -ENODEV; + + ctx->ahash_fbk = crypto_alloc_ahash(alg_name, 0, + CRYPTO_ALG_NEED_FALLBACK); + + if (IS_ERR(ctx->ahash_fbk)) + return dev_err_probe(ctx->cryp->dev, PTR_ERR(ctx->ahash_fbk), + "starfive-sm3: Could not load fallback driver.\n"); + + crypto_ahash_set_statesize(hash, crypto_ahash_statesize(ctx->ahash_fbk)); + crypto_ahash_set_reqsize(hash, sizeof(struct starfive_cryp_request_ctx) + + crypto_ahash_reqsize(ctx->ahash_fbk)); + + ctx->keylen = 0; + ctx->hash_mode = STARFIVE_SM3_MODE; + ctx->is_hmac = is_hmac; + + return 0; +} + +static void starfive_sm3_exit_tfm(struct crypto_ahash *hash) +{ + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(hash); + + crypto_free_ahash(ctx->ahash_fbk); +} + +static int starfive_sm3_long_setkey(struct starfive_cryp_ctx *ctx, + const u8 *key, unsigned int keylen) +{ + struct crypto_wait wait; + struct ahash_request *req; + struct scatterlist sg; + struct crypto_ahash *ahash_tfm; + struct starfive_cryp_dev *cryp = ctx->cryp; + u8 *buf; + int ret; + + ahash_tfm = crypto_alloc_ahash("sm3-starfive", 0, 0); + if (IS_ERR(ahash_tfm)) + return PTR_ERR(ahash_tfm); + + req = ahash_request_alloc(ahash_tfm, GFP_KERNEL); + if (!req) { + ret = -ENOMEM; + goto err_free_ahash; + } + + crypto_init_wait(&wait); + ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, &wait); + crypto_ahash_clear_flags(ahash_tfm, ~0); + + buf = devm_kzalloc(cryp->dev, keylen + STARFIVE_SM3_BUFLEN, GFP_KERNEL); + if (!buf) { + ret = -ENOMEM; + goto err_free_req; + } + + memcpy(buf, key, keylen); + sg_init_one(&sg, buf, keylen); + ahash_request_set_crypt(req, &sg, ctx->key, keylen); + + ret = crypto_wait_req(crypto_ahash_digest(req), &wait); + +err_free_req: + ahash_request_free(req); +err_free_ahash: + crypto_free_ahash(ahash_tfm); + return ret; +} + +static int starfive_sm3_setkey(struct crypto_ahash *hash, + const u8 *key, unsigned int keylen) +{ + struct starfive_cryp_ctx *ctx = crypto_ahash_ctx(hash); + unsigned int digestsize = crypto_ahash_digestsize(hash); + unsigned int blocksize = crypto_ahash_blocksize(hash); + + crypto_ahash_setkey(ctx->ahash_fbk, key, keylen); + + if (keylen <= blocksize) { + memcpy(ctx->key, key, keylen); + ctx->keylen = keylen; + return 0; + } + + ctx->keylen = digestsize; + + return starfive_sm3_long_setkey(ctx, key, keylen); +} + +static int starfive_sm3_init_tfm(struct crypto_ahash *hash) +{ + return starfive_sm3_init_algo(hash, "sm3-generic", 0); +} + +static int starfive_hmac_sm3_init_tfm(struct crypto_ahash *hash) +{ + return starfive_sm3_init_algo(hash, "hmac(sm3-generic)", 1); +} + +static struct ahash_engine_alg algs_sm3[] = { +{ + .base.init = starfive_sm3_init, + .base.update = starfive_sm3_update, + .base.final = starfive_sm3_final, + .base.finup = starfive_sm3_finup, + .base.digest = starfive_sm3_digest, + .base.export = starfive_sm3_export, + .base.import = starfive_sm3_import, + .base.init_tfm = starfive_sm3_init_tfm, + .base.exit_tfm = starfive_sm3_exit_tfm, + .base.halg = { + .digestsize = SM3_DIGEST_SIZE, + .statesize = sizeof(struct sm3_state), + .base = { + .cra_name = "sm3", + .cra_driver_name = "sm3-starfive", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_ASYNC | + CRYPTO_ALG_TYPE_AHASH | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize = SM3_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct starfive_cryp_ctx), + .cra_module = THIS_MODULE, + } + }, + .op = { + .do_one_request = starfive_sm3_one_request, + }, +}, { + .base.init = starfive_sm3_init, + .base.update = starfive_sm3_update, + .base.final = starfive_sm3_final, + .base.finup = starfive_sm3_finup, + .base.digest = starfive_sm3_digest, + .base.export = starfive_sm3_export, + .base.import = starfive_sm3_import, + .base.init_tfm = starfive_hmac_sm3_init_tfm, + .base.exit_tfm = starfive_sm3_exit_tfm, + .base.setkey = starfive_sm3_setkey, + .base.halg = { + .digestsize = SM3_DIGEST_SIZE, + .statesize = sizeof(struct sm3_state), + .base = { + .cra_name = "hmac(sm3)", + .cra_driver_name = "sm3-hmac-starfive", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_ASYNC | + CRYPTO_ALG_TYPE_AHASH | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize = SM3_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct starfive_cryp_ctx), + .cra_module = THIS_MODULE, + } + }, + .op = { + .do_one_request = starfive_sm3_one_request, + }, +}, +}; + +int starfive_sm3_register_algs(void) +{ + return crypto_engine_register_ahashes(algs_sm3, ARRAY_SIZE(algs_sm3)); +} + +void starfive_sm3_unregister_algs(void) +{ + crypto_engine_unregister_ahashes(algs_sm3, ARRAY_SIZE(algs_sm3)); +} From patchwork Thu May 30 03:11:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jia Jie Ho X-Patchwork-Id: 800274 Received: from CHN02-SH0-obe.outbound.protection.partner.outlook.cn (mail-sh0chn02on2091.outbound.protection.partner.outlook.cn [139.219.146.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34CAB1B947; Thu, 30 May 2024 03:47:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=139.219.146.91 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717040884; cv=fail; b=ankEZDDM8o1+vlgFaWDHxCmW8GhtP8SwATiM4v5TlGIytj0xsXnw3HTpuXfdV9g9Stzq5h4QGIHNpx1lRebqvQcoopL7Wek1DkbF/TkIIWPkRrPBnE4DjaChOJvNahIkImKHS2dWPDuVJZmA3sf9VyygI+vAvRZFHvGzI3aXWcQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717040884; c=relaxed/simple; bh=IO4SKfHlUBzexslaueOylAeVmeNLO+uDLwWMn/4q9aQ=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=HQpnVCZqbtR5FrcLPmGLltCGg/KbokwTE/xatLuKOiKJOLNgt8mf8d2myK5NlDTCC+quqMjsO2X91iHrTrhOHz+zggABPrP1imIBnryo7TMebYtOC0rWVR95sOktY+9SWIjz6Nged2LlYWL1gpmDWnbzxwCJj/T47T+segu3EZ4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com; spf=pass smtp.mailfrom=starfivetech.com; arc=fail smtp.client-ip=139.219.146.91 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=starfivetech.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GOibluYpef2A69KwYJ7uZUrCUnLsvCJ8xyIuvPr8BPPAgwqaGpBO94FLZ49hfXPaQFIYOSDHirZNMtPU73dqP5WLpG4cUiG6J4gU+lpJ0Kfa7SLbJnnTN1H7sj4SdC5XX1xl5NWRITCTaQemnbeFEqPwSNXG2NfBltbfUuRihQBzK0cqk8PE62Jf3DfjL3vsIax0TV+N7UpMujiG2xrIpYNSNvWxHicMwzx/VHUllD2IvpLE/0HDhe83zIkmr8/r0gp2Fc1xIq40zjcGrDh/3D95U9k4SGRwi81Nn3fbdWkStQZDmTZZjMjGpMmcoq6GofJ5jSLrTjxaeddm01K5tQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P946W6R22krBX5mPxyVBE3TMcDHsrgQ37/EJcG/W9FE=; b=n7/v55CwrMAsCMyptrb4xpvcx3/IZddfvj6GblCIjF2+OSlTn82gcEOPqInB4CzQgxjuL0cFW+W23vlbCG11p2DXDGpcomwCHCIAnNlcc4asLnN0EzkqNhIumDldpMqOXx7igP6dRShLfFkXftPld6/vZEUnwO3ZgVgmn/rYXzKhE28a7vt5WVXZhyDNdfgHAi3kwNLCFsLvfHd/0P4RGYOp19obofcv5SurpQueQBtmlPbCgkwdMcmndrTGIpYOHb2M721arRzpuo5NUfel0s4tQ/Z9sH+jlU6ElahSxIKsp5l1Yjd2KV0jqYrHoMDBgYHhM9H35gHP6ri3nHXW3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=starfivetech.com; dmarc=pass action=none header.from=starfivetech.com; dkim=pass header.d=starfivetech.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=starfivetech.com; Received: from NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn (2406:e500:c510:10::10) by NT0PR01MB1054.CHNPR01.prod.partner.outlook.cn (2406:e500:c510:2::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.36; Thu, 30 May 2024 03:11:30 +0000 Received: from NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn ([fe80::e3b:43f8:2e6d:ecce]) by NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn ([fe80::e3b:43f8:2e6d:ecce%7]) with mapi id 15.20.7587.037; Thu, 30 May 2024 03:11:29 +0000 From: Jia Jie Ho To: Herbert Xu , "David S . Miller" , Eugeniy Paltsev , Vinod Koul , linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v5 3/3] crypto: starfive: Add sm4 support for JH8100 Date: Thu, 30 May 2024 11:11:12 +0800 Message-ID: <20240530031112.4952-4-jiajie.ho@starfivetech.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240530031112.4952-1-jiajie.ho@starfivetech.com> References: <20240530031112.4952-1-jiajie.ho@starfivetech.com> X-ClientProxiedBy: ZQ0PR01CA0027.CHNPR01.prod.partner.outlook.cn (2406:e500:c550:2::17) To NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn (2406:e500:c510:10::10) Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: NT0PR01MB1182:EE_|NT0PR01MB1054:EE_ X-MS-Office365-Filtering-Correlation-Id: e6310733-4004-49fe-e36a-08dc80563357 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; ARA:13230031|366007|41320700004|1800799015|52116005|38350700005; X-Microsoft-Antispam-Message-Info: aaXqndxMqf/M0YVLCSoEqQomhgQsrbDL1ZCs8jkCF/j13enfutZVzwlgur5N6Q0pGvdpMyE+dIWQqMTb+9E2MUzPRraISoSNG4RE/ySBQdH9OMwoUUUWLS5U1TklBJQ30X/rAsvFFTZ0g8Y4e3O/QRD/mWG5kF2qcmOyx+LvGNwFbp+UP08x5rMW0jtUxmiiwfZSPuExtizXKfN2+T4bWVSB0DdjKa/EsJBcaYu9x2QYwbmtHqb9EJY4R+0twhL2iKua657x9zJAnH/Fuji7xxasSDuArhkx45pJ9dMojBncgZQ8qlfJrQjWwQE7IIikJtbfzGIP0G4C6YBlqAlllarXPQ44qNp7Hrj9HtFYXC9sBuQavmyQYfo9kX36kOpsKJfXp0UjkgPkAJTV9AaABBLQjlGQ+2R279+F15yaVEJTllMjEpHZwc/xu7h6J5Q7XCm9pa382CVoZ/nfs2IpI5sGvdsGbymF8hw3f40SMfdz10cHndUpHdwXYjs4LcHco02bjRPXFq2Td+m3h5ZQXtI4+B7V3qU3YcEgStFjlSNjiX3mLB+fPEDBN/hI+2dF5Bmbi2ZWsXzXA8/D34tbQS1vZjaRsJ4hVmyO9mcSXWCgeo1cxHOGkNq7pXEXT2mH X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn; PTR:; CAT:NONE; SFS:(13230031)(366007)(41320700004)(1800799015)(52116005)(38350700005); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: VGjGr5zvl0Gls4RT8GpDD4ULFkC2q1A3abXnZIBJw/zvn+8ut0zq2u8Dkh3oDCqSp0LfXiWakqMN98LKjwk61hdxrEpUXtsBwxmFZadd9/4oZ1LtK17DbJPOrt0RsDz0OcV+jDRRYBEMvChU2a/s7EOixOutW+DnzTx0e4Tzt4hYR20Sc/xIdAUrofJ8M8jXG3rysddBwS8NtAm37ly6duTs5Tmsuh4uI+gFa6fptc3PfQlx4kjxklX1VKftdLb1P+bFX9MuDDAbZRfB2s4Kj+ghRRIFkUCLDzDn+29U7UPiWcgYqhzpb3cVycZ9egr32tcW7dReTU9qOvTGnmVPmKGOC/O3Abc7fOGqiX6QHrrsybJ88n1p0aPF6UcqKy/+Jmgf6vrrz/Q4sZ6ULLMz6/qr6DFE5tLVFoj+rmhsTZvBIJ2MkqG6wB8twWMGJwx/kcD18WlxSLG8+nzgIcHjCT0GnDiTvNwM3wjOLLVz4VQfJoikPDWhxi5SIL3R1PchWXnYoQSbQ0s+Ed/dzC/rMzAvmW9rJu3EF96oXtr1qhkAgT25e/6IryPjij0Z5+VpJ8YvxlTdenzHf67zB+gFIc/0HVV8k43Y+v2TKGqVlyOsqPAPOuN7NhNEfDZhx55c/nLjXPysQPTOdUgt18/P5RzMOroOngPgx3VysAD2P9cP1VqXLgIRO5W/0fRNIrqKlP2i+PeeuqUaPbAatB8BypzMSts+rlMX+Wk7L0ytGtCpWXEtMgVu6+aOJlsyE32IqeohggE5pIGj0opamhOheLjAsoOqr2IN1JJfrcMJIUB0oC/abv8N/bAyFYcIkZBvPQxnZtKZqU5hY19PAUg3stf6uNKXU//i8wBs9afnZP/Y2LqqDHOQJVBldbpjdC9Ye3QnMSFnecQ6gBtbO8KOUt8Yxcda09ZNoJO5nmd+eZ9E42Vga8QA7d686Z0RnXHWl4TEVCbNo8slMWRYnJV2Rn+d701tAXhjdEQvM9jZOI3Tsjj+3ogixSrW5ZCmtKtSs2YvAo6EsCpVydHpxu3agcsAmAAIdErPCLk0f62rwW+8fON7KgMz1efqYsTlQwxZHEqKPg3Kg8TjVHvbLfprngDMDyII4nJwzjHU7+gjA8Dn0275hUMm7F8RJUQnyaGnlNtVm335W2Lk4Mwvz1k1STBuAQFQqqTZJAUZ/KnKz3pKGQF/fotUMLrD7N93feg+1n7mDp6vC2LkSaI1asylCCv3IaiOG98VkkMYpLmF3X8/oNs4LozEa0vKzK2f/rqYlaL5Q4M1i2unz8yQrx0ULeiONl3aDAJOFnQGXIDlOHIFsaMRTDxHT9EQBAnv1DyP7dyTwZ2FuK31/C93BPmRCGw7XfvN1IvQsVJx58yK/hilV8bsUehhgHOwGrpgwIzz7Pcp9IQX0UrAelX7E/h1+SHE14SVG1Ll4ZsKSfjaRpQc8xP1gRGTwta7cKVF6sh2FTPKTa235mK9xvllmZkPQTUqkTyIzfEgFRsRW52YhOKDI9X2bbDHkOPNzdI7e3oyiLb5OTUKnMMCdq1S3M3M4noRVE2yWTCBw37h31NvyFbtW83kB2hbmApQQmLEhOlMfVCg6+TXkiif/URctslyHA== X-OriginatorOrg: starfivetech.com X-MS-Exchange-CrossTenant-Network-Message-Id: e6310733-4004-49fe-e36a-08dc80563357 X-MS-Exchange-CrossTenant-AuthSource: NT0PR01MB1182.CHNPR01.prod.partner.outlook.cn X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2024 03:11:29.9374 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 06fe3fa3-1221-43d3-861b-5a4ee687a85c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8Sw7Elwi9jaxiD5Xgj/z79bUebOZ6H6FyB6kV/2Yzr5taNi9HozX7XIJxlLwaqRaozZCU3Xa96qwnENbDd3mphRbhITmbScbBntTfGQwCYg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: NT0PR01MB1054 Add driver support for sm4 skcipher and aead for StarFive JH8100 SoC. Signed-off-by: Jia Jie Ho --- drivers/crypto/starfive/Kconfig | 1 + drivers/crypto/starfive/Makefile | 4 +- drivers/crypto/starfive/jh7110-cryp.c | 10 +- drivers/crypto/starfive/jh7110-cryp.h | 39 + drivers/crypto/starfive/jh8100-sm4.c | 1123 +++++++++++++++++++++++++ 5 files changed, 1174 insertions(+), 3 deletions(-) create mode 100644 drivers/crypto/starfive/jh8100-sm4.c diff --git a/drivers/crypto/starfive/Kconfig b/drivers/crypto/starfive/Kconfig index e6bf02d0ed1f..740bb70c5607 100644 --- a/drivers/crypto/starfive/Kconfig +++ b/drivers/crypto/starfive/Kconfig @@ -34,6 +34,7 @@ config CRYPTO_DEV_JH8100 select CRYPTO_SHA256 select CRYPTO_SHA512 select CRYPTO_SM3_GENERIC + select CRYPTO_SM4_GENERIC select CRYPTO_RSA select CRYPTO_AES select CRYPTO_CCM diff --git a/drivers/crypto/starfive/Makefile b/drivers/crypto/starfive/Makefile index 867ce035af19..0a4476085716 100644 --- a/drivers/crypto/starfive/Makefile +++ b/drivers/crypto/starfive/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CRYPTO_DEV_JH7110) += jh7110-crypto.o -jh7110-crypto-objs := jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-aes.o jh8100-sm3.o +jh7110-crypto-objs := jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-aes.o jh8100-sm3.o jh8100-sm4.o obj-$(CONFIG_CRYPTO_DEV_JH8100) += jh8100-crypto.o -jh8100-crypto-objs := jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-aes.o jh8100-sm3.o +jh8100-crypto-objs := jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-aes.o jh8100-sm3.o jh8100-sm4.o diff --git a/drivers/crypto/starfive/jh7110-cryp.c b/drivers/crypto/starfive/jh7110-cryp.c index 19bbcaaec18d..cdc34b6e1e81 100644 --- a/drivers/crypto/starfive/jh7110-cryp.c +++ b/drivers/crypto/starfive/jh7110-cryp.c @@ -168,10 +168,16 @@ static int starfive_cryp_probe(struct platform_device *pdev) ret = starfive_sm3_register_algs(); if (ret) goto err_algs_sm3; + + ret = starfive_sm4_register_algs(); + if (ret) + goto err_algs_sm4; } return 0; +err_algs_sm4: + starfive_sm3_unregister_algs(); err_algs_sm3: starfive_rsa_unregister_algs(); err_algs_rsa: @@ -204,8 +210,10 @@ static void starfive_cryp_remove(struct platform_device *pdev) starfive_hash_unregister_algs(cryp); starfive_rsa_unregister_algs(); - if (cryp->type == STARFIVE_CRYPTO_JH8100) + if (cryp->type == STARFIVE_CRYPTO_JH8100) { starfive_sm3_unregister_algs(); + starfive_sm4_unregister_algs(); + } crypto_engine_stop(cryp->engine); crypto_engine_exit(cryp->engine); diff --git a/drivers/crypto/starfive/jh7110-cryp.h b/drivers/crypto/starfive/jh7110-cryp.h index 60ac752352c8..ca35062fff60 100644 --- a/drivers/crypto/starfive/jh7110-cryp.h +++ b/drivers/crypto/starfive/jh7110-cryp.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -190,6 +191,40 @@ union starfive_sm3_csr { }; }; +union starfive_sm4_csr { + u32 v; + struct { + u32 cmode :1; + u32 rsvd_0 :1; + u32 ie :1; + u32 sm4rst :1; + u32 rsvd_1 :1; +#define STARFIVE_SM4_DONE BIT(5) + u32 sm4done :1; +#define STARFIVE_SM4_KEY_DONE BIT(6) + u32 krdy :1; + u32 busy :1; + u32 vsm4_start :1; + u32 delay_sm4 :1; +#define STARFIVE_SM4_CCM_START BIT(10) + u32 ccm_start :1; +#define STARFIVE_SM4_GCM_START BIT(11) + u32 gcm_start :1; + u32 rsvd_2 :4; +#define STARFIVE_SM4_MODE_XFB_1 0x0 +#define STARFIVE_SM4_MODE_XFB_128 0x5 + u32 stmode :3; + u32 rsvd_3 :2; +#define STARFIVE_SM4_MODE_ECB 0x0 +#define STARFIVE_SM4_MODE_CBC 0x1 +#define STARFIVE_SM4_MODE_CTR 0x4 +#define STARFIVE_SM4_MODE_CCM 0x5 +#define STARFIVE_SM4_MODE_GCM 0x6 + u32 mode :3; + u32 rsvd_4 :8; + }; +}; + struct starfive_rsa_key { u8 *n; u8 *e; @@ -269,6 +304,7 @@ struct starfive_cryp_request_ctx { union starfive_pka_cacr pka; union starfive_aes_csr aes; union starfive_sm3_csr sm3; + union starfive_sm4_csr sm4; } csr; struct scatterlist *in_sg; @@ -295,4 +331,7 @@ void starfive_aes_unregister_algs(void); int starfive_sm3_register_algs(void); void starfive_sm3_unregister_algs(void); + +int starfive_sm4_register_algs(void); +void starfive_sm4_unregister_algs(void); #endif diff --git a/drivers/crypto/starfive/jh8100-sm4.c b/drivers/crypto/starfive/jh8100-sm4.c new file mode 100644 index 000000000000..195ba95fab7e --- /dev/null +++ b/drivers/crypto/starfive/jh8100-sm4.c @@ -0,0 +1,1123 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * StarFive SM4 acceleration driver + * + * Copyright (c) 2022 - 2023 StarFive Technology + */ + +#include +#include +#include +#include +#include +#include "jh7110-cryp.h" +#include +#include + +#define STARFIVE_SM4_REGS_OFFSET 0x4100 +#define STARFIVE_SM4_SM4DIO0R (STARFIVE_SM4_REGS_OFFSET + 0x0) +#define STARFIVE_SM4_KEY0 (STARFIVE_SM4_REGS_OFFSET + 0x4) +#define STARFIVE_SM4_KEY1 (STARFIVE_SM4_REGS_OFFSET + 0x8) +#define STARFIVE_SM4_KEY2 (STARFIVE_SM4_REGS_OFFSET + 0xC) +#define STARFIVE_SM4_KEY3 (STARFIVE_SM4_REGS_OFFSET + 0x10) +#define STARFIVE_SM4_IV0 (STARFIVE_SM4_REGS_OFFSET + 0x14) +#define STARFIVE_SM4_IV1 (STARFIVE_SM4_REGS_OFFSET + 0x18) +#define STARFIVE_SM4_IV2 (STARFIVE_SM4_REGS_OFFSET + 0x1c) +#define STARFIVE_SM4_IV3 (STARFIVE_SM4_REGS_OFFSET + 0x20) +#define STARFIVE_SM4_CSR (STARFIVE_SM4_REGS_OFFSET + 0x24) +#define STARFIVE_SM4_NONCE0 (STARFIVE_SM4_REGS_OFFSET + 0x30) +#define STARFIVE_SM4_NONCE1 (STARFIVE_SM4_REGS_OFFSET + 0x34) +#define STARFIVE_SM4_NONCE2 (STARFIVE_SM4_REGS_OFFSET + 0x38) +#define STARFIVE_SM4_NONCE3 (STARFIVE_SM4_REGS_OFFSET + 0x3c) +#define STARFIVE_SM4_ALEN0 (STARFIVE_SM4_REGS_OFFSET + 0x40) +#define STARFIVE_SM4_ALEN1 (STARFIVE_SM4_REGS_OFFSET + 0x44) +#define STARFIVE_SM4_MLEN0 (STARFIVE_SM4_REGS_OFFSET + 0x48) +#define STARFIVE_SM4_MLEN1 (STARFIVE_SM4_REGS_OFFSET + 0x4c) +#define STARFIVE_SM4_IVLEN (STARFIVE_SM4_REGS_OFFSET + 0x50) + +#define FLG_MODE_MASK GENMASK(2, 0) +#define FLG_ENCRYPT BIT(4) + +/* Misc */ +#define CCM_B0_ADATA 0x40 +#define SM4_BLOCK_32 (SM4_BLOCK_SIZE / sizeof(u32)) + +static inline int starfive_sm4_wait_done(struct starfive_cryp_dev *cryp) +{ + u32 status; + + return readl_relaxed_poll_timeout(cryp->base + STARFIVE_SM4_CSR, status, + status & STARFIVE_SM4_DONE, 10, 100000); +} + +static inline int starfive_sm4_wait_keydone(struct starfive_cryp_dev *cryp) +{ + u32 status; + + return readl_relaxed_poll_timeout(cryp->base + STARFIVE_SM4_CSR, status, + status & STARFIVE_SM4_KEY_DONE, 10, 100000); +} + +static inline int is_encrypt(struct starfive_cryp_dev *cryp) +{ + return cryp->flags & FLG_ENCRYPT; +} + +static int starfive_sm4_aead_write_key(struct starfive_cryp_ctx *ctx, u32 hw_mode) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + unsigned int value; + u32 *key = (u32 *)ctx->key; + + writel(key[0], cryp->base + STARFIVE_SM4_KEY0); + writel(key[1], cryp->base + STARFIVE_SM4_KEY1); + writel(key[2], cryp->base + STARFIVE_SM4_KEY2); + writel(key[3], cryp->base + STARFIVE_SM4_KEY3); + + value = readl(ctx->cryp->base + STARFIVE_SM4_CSR); + + if (hw_mode == STARFIVE_SM4_MODE_GCM) + value |= STARFIVE_SM4_GCM_START; + else + value |= STARFIVE_SM4_CCM_START; + + writel(value, cryp->base + STARFIVE_SM4_CSR); + + if (starfive_sm4_wait_keydone(cryp)) + return -ETIMEDOUT; + + return 0; +} + +static inline void starfive_sm4_set_alen(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + + writel(upper_32_bits(cryp->assoclen), cryp->base + STARFIVE_SM4_ALEN0); + writel(lower_32_bits(cryp->assoclen), cryp->base + STARFIVE_SM4_ALEN1); +} + +static inline void starfive_sm4_set_mlen(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + + writel(upper_32_bits(cryp->total_in), cryp->base + STARFIVE_SM4_MLEN0); + writel(lower_32_bits(cryp->total_in), cryp->base + STARFIVE_SM4_MLEN1); +} + +static inline int starfive_sm4_ccm_check_iv(const u8 *iv) +{ + /* 2 <= L <= 8, so 1 <= L' <= 7. */ + if (iv[0] < 1 || iv[0] > 7) + return -EINVAL; + + return 0; +} + +static inline void starfive_sm4_write_iv(struct starfive_cryp_ctx *ctx, u32 *iv) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + + writel(iv[0], cryp->base + STARFIVE_SM4_IV0); + writel(iv[1], cryp->base + STARFIVE_SM4_IV1); + writel(iv[2], cryp->base + STARFIVE_SM4_IV2); + writel(iv[3], cryp->base + STARFIVE_SM4_IV3); +} + +static inline void starfive_sm4_get_iv(struct starfive_cryp_dev *cryp, u32 *iv) +{ + iv[0] = readl(cryp->base + STARFIVE_SM4_IV0); + iv[1] = readl(cryp->base + STARFIVE_SM4_IV1); + iv[2] = readl(cryp->base + STARFIVE_SM4_IV2); + iv[3] = readl(cryp->base + STARFIVE_SM4_IV3); +} + +static inline void starfive_sm4_write_nonce(struct starfive_cryp_ctx *ctx, u32 *nonce) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + + writel(nonce[0], cryp->base + STARFIVE_SM4_NONCE0); + writel(nonce[1], cryp->base + STARFIVE_SM4_NONCE1); + writel(nonce[2], cryp->base + STARFIVE_SM4_NONCE2); + writel(nonce[3], cryp->base + STARFIVE_SM4_NONCE3); +} + +static int starfive_sm4_write_key(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + u32 *key = (u32 *)ctx->key; + + writel(key[0], cryp->base + STARFIVE_SM4_KEY0); + writel(key[1], cryp->base + STARFIVE_SM4_KEY1); + writel(key[2], cryp->base + STARFIVE_SM4_KEY2); + writel(key[3], cryp->base + STARFIVE_SM4_KEY3); + + if (starfive_sm4_wait_keydone(cryp)) + return -ETIMEDOUT; + + return 0; +} + +static int starfive_sm4_ccm_init(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + u8 iv[SM4_BLOCK_SIZE], b0[SM4_BLOCK_SIZE]; + unsigned int textlen; + + memcpy(iv, cryp->req.areq->iv, SM4_BLOCK_SIZE); + memset(iv + SM4_BLOCK_SIZE - 1 - iv[0], 0, iv[0] + 1); + + /* Build B0 */ + memcpy(b0, iv, SM4_BLOCK_SIZE); + + b0[0] |= (8 * ((cryp->authsize - 2) / 2)); + + if (cryp->assoclen) + b0[0] |= CCM_B0_ADATA; + + textlen = cryp->total_in; + + b0[SM4_BLOCK_SIZE - 2] = textlen >> 8; + b0[SM4_BLOCK_SIZE - 1] = textlen & 0xFF; + + starfive_sm4_write_nonce(ctx, (u32 *)b0); + + return 0; +} + +static int starfive_sm4_hw_init(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_request_ctx *rctx = ctx->rctx; + struct starfive_cryp_dev *cryp = ctx->cryp; + u32 hw_mode; + int ret = 0; + + /* reset */ + rctx->csr.sm4.v = 0; + rctx->csr.sm4.sm4rst = 1; + writel(rctx->csr.sm4.v, cryp->base + STARFIVE_SM4_CSR); + + /* csr setup */ + hw_mode = cryp->flags & FLG_MODE_MASK; + + rctx->csr.sm4.v = 0; + rctx->csr.sm4.mode = hw_mode; + rctx->csr.sm4.cmode = !is_encrypt(cryp); + rctx->csr.sm4.stmode = STARFIVE_SM4_MODE_XFB_1; + + if (cryp->side_chan) { + rctx->csr.sm4.delay_sm4 = 1; + rctx->csr.sm4.vsm4_start = 1; + } + + writel(rctx->csr.sm4.v, cryp->base + STARFIVE_SM4_CSR); + + switch (hw_mode) { + case STARFIVE_SM4_MODE_GCM: + starfive_sm4_set_alen(ctx); + starfive_sm4_set_mlen(ctx); + writel(GCM_AES_IV_SIZE, cryp->base + STARFIVE_SM4_IVLEN); + ret = starfive_sm4_aead_write_key(ctx, hw_mode); + if (ret) + return ret; + + starfive_sm4_write_iv(ctx, (void *)cryp->req.areq->iv); + break; + case STARFIVE_SM4_MODE_CCM: + starfive_sm4_set_alen(ctx); + starfive_sm4_set_mlen(ctx); + starfive_sm4_ccm_init(ctx); + ret = starfive_sm4_aead_write_key(ctx, hw_mode); + if (ret) + return ret; + break; + case STARFIVE_SM4_MODE_CBC: + case STARFIVE_SM4_MODE_CTR: + starfive_sm4_write_iv(ctx, (void *)cryp->req.sreq->iv); + ret = starfive_sm4_write_key(ctx); + if (ret) + return ret; + break; + case STARFIVE_SM4_MODE_ECB: + ret = starfive_sm4_write_key(ctx); + if (ret) + return ret; + break; + default: + return -EINVAL; + } + + return 0; +} + +static int starfive_sm4_read_authtag(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + struct starfive_cryp_request_ctx *rctx = ctx->rctx; + int i; + + if ((cryp->flags & FLG_MODE_MASK) == STARFIVE_SM4_MODE_GCM) { + cryp->tag_out[0] = readl(cryp->base + STARFIVE_SM4_NONCE0); + cryp->tag_out[1] = readl(cryp->base + STARFIVE_SM4_NONCE1); + cryp->tag_out[2] = readl(cryp->base + STARFIVE_SM4_NONCE2); + cryp->tag_out[3] = readl(cryp->base + STARFIVE_SM4_NONCE3); + } else { + for (i = 0; i < SM4_BLOCK_32; i++) + cryp->tag_out[i] = readl(cryp->base + STARFIVE_SM4_SM4DIO0R); + } + + if (is_encrypt(cryp)) { + scatterwalk_map_and_copy(cryp->tag_out, rctx->out_sg, + cryp->total_in, cryp->authsize, 1); + } else { + if (crypto_memneq(cryp->tag_in, cryp->tag_out, cryp->authsize)) + return dev_err_probe(cryp->dev, -EBADMSG, + "Failed tag verification\n"); + } + + return 0; +} + +static void starfive_sm4_finish_req(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + int err = 0; + + if (cryp->authsize) + err = starfive_sm4_read_authtag(ctx); + + if ((cryp->flags & FLG_MODE_MASK) == STARFIVE_SM4_MODE_CBC || + (cryp->flags & FLG_MODE_MASK) == STARFIVE_SM4_MODE_CTR) + starfive_sm4_get_iv(cryp, (void *)cryp->req.sreq->iv); + + if (cryp->authsize) + crypto_finalize_aead_request(cryp->engine, cryp->req.areq, err); + else + crypto_finalize_skcipher_request(cryp->engine, cryp->req.sreq, + err); +} + +static int starfive_sm4_gcm_write_adata(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + struct starfive_cryp_request_ctx *rctx = ctx->rctx; + u32 *buffer; + int total_len, loop; + + total_len = ALIGN(cryp->assoclen, SM4_BLOCK_SIZE) / sizeof(unsigned int); + buffer = (u32 *)rctx->adata; + + for (loop = 0; loop < total_len; loop += 4) { + writel(*buffer, cryp->base + STARFIVE_SM4_NONCE0); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_NONCE1); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_NONCE2); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_NONCE3); + buffer++; + + if (starfive_sm4_wait_done(cryp)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, + "Timeout processing gcm aad block"); + } + + return 0; +} + +static int starfive_sm4_ccm_write_adata(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp = ctx->cryp; + struct starfive_cryp_request_ctx *rctx = ctx->rctx; + u32 *buffer; + int total_len, loop; + + buffer = (u32 *)rctx->adata; + total_len = ALIGN(cryp->assoclen + 2, SM4_BLOCK_SIZE) / sizeof(unsigned int); + + for (loop = 0; loop < total_len; loop += 4) { + writel(*buffer, cryp->base + STARFIVE_SM4_SM4DIO0R); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_SM4DIO0R); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_SM4DIO0R); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_SM4DIO0R); + buffer++; + + if (starfive_sm4_wait_done(cryp)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, + "Timeout processing ccm aad block"); + } + + return 0; +} + +static void starfive_sm4_dma_done(void *param) +{ + struct starfive_cryp_dev *cryp = param; + + complete(&cryp->dma_done); +} + +static void starfive_sm4_dma_init(struct starfive_cryp_dev *cryp) +{ + struct dw_axi_peripheral_config periph_conf = {}; + + memset(&cryp->cfg_in, 0, sizeof(struct dma_slave_config)); + memset(&cryp->cfg_out, 0, sizeof(struct dma_slave_config)); + + periph_conf.quirks = DWAXIDMAC_STARFIVE_SM_ALGO; + + cryp->cfg_in.direction = DMA_MEM_TO_DEV; + cryp->cfg_in.src_addr_width = DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_in.dst_addr_width = DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_in.src_maxburst = cryp->dma_maxburst; + cryp->cfg_in.dst_maxburst = cryp->dma_maxburst; + cryp->cfg_in.dst_addr = cryp->phys_base + STARFIVE_SM_ALG_FIFO_IN_OFFSET; + cryp->cfg_in.peripheral_config = &periph_conf; + cryp->cfg_in.peripheral_size = sizeof(struct dw_axi_peripheral_config); + + dmaengine_slave_config(cryp->tx, &cryp->cfg_in); + + cryp->cfg_out.direction = DMA_DEV_TO_MEM; + cryp->cfg_out.src_addr_width = DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_out.dst_addr_width = DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_out.src_maxburst = cryp->dma_maxburst; + cryp->cfg_out.dst_maxburst = cryp->dma_maxburst; + cryp->cfg_out.src_addr = cryp->phys_base + STARFIVE_SM_ALG_FIFO_OUT_OFFSET; + cryp->cfg_out.peripheral_config = &periph_conf; + cryp->cfg_out.peripheral_size = sizeof(struct dw_axi_peripheral_config); + + dmaengine_slave_config(cryp->rx, &cryp->cfg_out); + + init_completion(&cryp->dma_done); +} + +static int starfive_sm4_dma_xfer(struct starfive_cryp_dev *cryp, + struct scatterlist *src, + struct scatterlist *dst, + int len) +{ + struct dma_async_tx_descriptor *in_desc, *out_desc; + union starfive_sm_alg_cr alg_cr; + int ret = 0, in_save, out_save; + + alg_cr.v = 0; + alg_cr.start = 1; + alg_cr.sm4_dma_en = 1; + writel(alg_cr.v, cryp->base + STARFIVE_SM_ALG_CR_OFFSET); + + in_save = sg_dma_len(src); + out_save = sg_dma_len(dst); + + writel(ALIGN(len, SM4_BLOCK_SIZE), cryp->base + STARFIVE_SM_DMA_IN_LEN_OFFSET); + writel(ALIGN(len, SM4_BLOCK_SIZE), cryp->base + STARFIVE_SM_DMA_OUT_LEN_OFFSET); + + sg_dma_len(src) = ALIGN(len, SM4_BLOCK_SIZE); + sg_dma_len(dst) = ALIGN(len, SM4_BLOCK_SIZE); + + out_desc = dmaengine_prep_slave_sg(cryp->rx, dst, 1, DMA_DEV_TO_MEM, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + if (!out_desc) { + ret = -EINVAL; + goto dma_err; + } + + out_desc->callback = starfive_sm4_dma_done; + out_desc->callback_param = cryp; + + reinit_completion(&cryp->dma_done); + dmaengine_submit(out_desc); + dma_async_issue_pending(cryp->rx); + + in_desc = dmaengine_prep_slave_sg(cryp->tx, src, 1, DMA_MEM_TO_DEV, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + if (!in_desc) { + ret = -EINVAL; + goto dma_err; + } + + dmaengine_submit(in_desc); + dma_async_issue_pending(cryp->tx); + + if (!wait_for_completion_timeout(&cryp->dma_done, + msecs_to_jiffies(1000))) + ret = -ETIMEDOUT; + +dma_err: + sg_dma_len(src) = in_save; + sg_dma_len(dst) = out_save; + + alg_cr.v = 0; + alg_cr.clear = 1; + writel(alg_cr.v, cryp->base + STARFIVE_SM_ALG_CR_OFFSET); + + return ret; +} + +static int starfive_sm4_map_sg(struct starfive_cryp_dev *cryp, + struct scatterlist *src, + struct scatterlist *dst) +{ + struct scatterlist *stsg, *dtsg; + struct scatterlist _src[2], _dst[2]; + unsigned int remain = cryp->total_in; + unsigned int len, src_nents, dst_nents; + int ret; + + if (src == dst) { + for (stsg = src, dtsg = dst; remain > 0; + stsg = sg_next(stsg), dtsg = sg_next(dtsg)) { + src_nents = dma_map_sg(cryp->dev, stsg, 1, DMA_BIDIRECTIONAL); + if (src_nents == 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg error\n"); + + dst_nents = src_nents; + + len = min(sg_dma_len(stsg), remain); + + ret = starfive_sm4_dma_xfer(cryp, stsg, dtsg, len); + dma_unmap_sg(cryp->dev, stsg, 1, DMA_BIDIRECTIONAL); + if (ret) + return ret; + + remain -= len; + } + } else { + for (stsg = src, dtsg = dst;;) { + src_nents = dma_map_sg(cryp->dev, stsg, 1, DMA_TO_DEVICE); + if (src_nents == 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg src error\n"); + + dst_nents = dma_map_sg(cryp->dev, dtsg, 1, DMA_FROM_DEVICE); + if (dst_nents == 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg dst error\n"); + + len = min(sg_dma_len(stsg), sg_dma_len(dtsg)); + len = min(len, remain); + + ret = starfive_sm4_dma_xfer(cryp, stsg, dtsg, len); + dma_unmap_sg(cryp->dev, stsg, 1, DMA_TO_DEVICE); + dma_unmap_sg(cryp->dev, dtsg, 1, DMA_FROM_DEVICE); + if (ret) + return ret; + + remain -= len; + if (remain == 0) + break; + + if (sg_dma_len(stsg) - len) { + stsg = scatterwalk_ffwd(_src, stsg, len); + dtsg = sg_next(dtsg); + } else if (sg_dma_len(dtsg) - len) { + dtsg = scatterwalk_ffwd(_dst, dtsg, len); + stsg = sg_next(stsg); + } else { + stsg = sg_next(stsg); + dtsg = sg_next(dtsg); + } + } + } + + return 0; +} + +static int starfive_sm4_do_one_req(struct crypto_engine *engine, void *areq) +{ + struct skcipher_request *req = + container_of(areq, struct skcipher_request, base); + struct starfive_cryp_ctx *ctx = + crypto_skcipher_ctx(crypto_skcipher_reqtfm(req)); + struct starfive_cryp_dev *cryp = ctx->cryp; + struct starfive_cryp_request_ctx *rctx = skcipher_request_ctx(req); + int ret; + + cryp->req.sreq = req; + cryp->total_in = req->cryptlen; + cryp->total_out = req->cryptlen; + cryp->assoclen = 0; + cryp->authsize = 0; + + rctx->in_sg = req->src; + rctx->out_sg = req->dst; + + ctx->rctx = rctx; + + ret = starfive_sm4_hw_init(ctx); + if (ret) + return ret; + + if (!cryp->total_in) + goto finish_req; + + starfive_sm4_dma_init(cryp); + + ret = starfive_sm4_map_sg(cryp, rctx->in_sg, rctx->out_sg); + if (ret) + return ret; + +finish_req: + starfive_sm4_finish_req(ctx); + + return 0; +} + +static int starfive_sm4_init_tfm(struct crypto_skcipher *tfm, + const char *alg_name) +{ + struct starfive_cryp_ctx *ctx = crypto_skcipher_ctx(tfm); + + ctx->cryp = starfive_cryp_find_dev(ctx); + if (!ctx->cryp) + return -ENODEV; + + ctx->skcipher_fbk = crypto_alloc_skcipher(alg_name, 0, + CRYPTO_ALG_NEED_FALLBACK); + if (IS_ERR(ctx->skcipher_fbk)) + return dev_err_probe(ctx->cryp->dev, PTR_ERR(ctx->skcipher_fbk), + "%s() failed to allocate fallback for %s\n", + __func__, alg_name); + + crypto_skcipher_set_reqsize(tfm, sizeof(struct starfive_cryp_request_ctx) + + crypto_skcipher_reqsize(ctx->skcipher_fbk)); + + return 0; +} + +static void starfive_sm4_exit_tfm(struct crypto_skcipher *tfm) +{ + struct starfive_cryp_ctx *ctx = crypto_skcipher_ctx(tfm); + + crypto_free_skcipher(ctx->skcipher_fbk); +} + +static int starfive_sm4_aead_do_one_req(struct crypto_engine *engine, void *areq) +{ + struct aead_request *req = + container_of(areq, struct aead_request, base); + struct starfive_cryp_ctx *ctx = + crypto_aead_ctx(crypto_aead_reqtfm(req)); + struct starfive_cryp_dev *cryp = ctx->cryp; + struct starfive_cryp_request_ctx *rctx = aead_request_ctx(req); + struct scatterlist _dst[2], _src[2]; + int ret; + + cryp->req.areq = req; + cryp->assoclen = req->assoclen; + cryp->authsize = crypto_aead_authsize(crypto_aead_reqtfm(req)); + + if (is_encrypt(cryp)) { + cryp->total_in = req->cryptlen; + cryp->total_out = req->cryptlen; + } else { + cryp->total_in = req->cryptlen - cryp->authsize; + cryp->total_out = cryp->total_in; + scatterwalk_map_and_copy(cryp->tag_in, req->src, + cryp->total_in + cryp->assoclen, + cryp->authsize, 0); + } + + if (cryp->assoclen) { + if ((cryp->flags & FLG_MODE_MASK) == STARFIVE_SM4_MODE_CCM) { + rctx->adata = kzalloc(cryp->assoclen + 2 + SM4_BLOCK_SIZE, GFP_KERNEL); + if (!rctx->adata) + return -ENOMEM; + + /* Append 2 bytes zeroes at the start of ccm aad */ + rctx->adata[0] = 0; + rctx->adata[1] = 0; + + sg_copy_to_buffer(req->src, + sg_nents_for_len(req->src, cryp->assoclen), + &rctx->adata[2], cryp->assoclen); + } else { + rctx->adata = kzalloc(cryp->assoclen + SM4_BLOCK_SIZE, GFP_KERNEL); + if (!rctx->adata) + return dev_err_probe(cryp->dev, -ENOMEM, + "Failed to alloc memory for adata"); + + sg_copy_to_buffer(req->src, + sg_nents_for_len(req->src, cryp->assoclen), + rctx->adata, cryp->assoclen); + } + } + + rctx->in_sg = scatterwalk_ffwd(_src, req->src, cryp->assoclen); + if (req->src == req->dst) + rctx->out_sg = rctx->in_sg; + else + rctx->out_sg = scatterwalk_ffwd(_dst, req->dst, cryp->assoclen); + + if (cryp->total_in) + sg_zero_buffer(rctx->in_sg, sg_nents(rctx->in_sg), + sg_dma_len(rctx->in_sg) - cryp->total_in, + cryp->total_in); + + ctx->rctx = rctx; + + ret = starfive_sm4_hw_init(ctx); + if (ret) + return ret; + + if (!cryp->assoclen) + goto write_text; + + if ((cryp->flags & FLG_MODE_MASK) == STARFIVE_SM4_MODE_CCM) + ret = starfive_sm4_ccm_write_adata(ctx); + else + ret = starfive_sm4_gcm_write_adata(ctx); + + kfree(rctx->adata); + + if (ret) + return ret; + +write_text: + if (!cryp->total_in) + goto finish_req; + + starfive_sm4_dma_init(cryp); + + ret = starfive_sm4_map_sg(cryp, rctx->in_sg, rctx->out_sg); + if (ret) + return ret; + +finish_req: + starfive_sm4_finish_req(ctx); + return 0; +} + +static int starfive_sm4_aead_init_tfm(struct crypto_aead *tfm, + const char *alg_name) +{ + struct starfive_cryp_ctx *ctx = crypto_aead_ctx(tfm); + + ctx->cryp = starfive_cryp_find_dev(ctx); + if (!ctx->cryp) + return -ENODEV; + + ctx->aead_fbk = crypto_alloc_aead(alg_name, 0, + CRYPTO_ALG_NEED_FALLBACK); + if (IS_ERR(ctx->aead_fbk)) + return dev_err_probe(ctx->cryp->dev, PTR_ERR(ctx->aead_fbk), + "%s() failed to allocate fallback for %s\n", + __func__, alg_name); + + crypto_aead_set_reqsize(tfm, sizeof(struct starfive_cryp_request_ctx) + + crypto_aead_reqsize(ctx->aead_fbk)); + + return 0; +} + +static void starfive_sm4_aead_exit_tfm(struct crypto_aead *tfm) +{ + struct starfive_cryp_ctx *ctx = crypto_aead_ctx(tfm); + + crypto_free_aead(ctx->aead_fbk); +} + +static bool starfive_sm4_check_unaligned(struct starfive_cryp_dev *cryp, + struct scatterlist *src, + struct scatterlist *dst) +{ + struct scatterlist *tsg; + int i; + + for_each_sg(src, tsg, sg_nents(src), i) + if (!IS_ALIGNED(tsg->length, SM4_BLOCK_SIZE) && + !sg_is_last(tsg)) + return true; + + if (src != dst) + for_each_sg(dst, tsg, sg_nents(dst), i) + if (!IS_ALIGNED(tsg->length, SM4_BLOCK_SIZE) && + !sg_is_last(tsg)) + return true; + + return false; +} + +static int starfive_sm4_do_fallback(struct skcipher_request *req, bool enc) +{ + struct starfive_cryp_ctx *ctx = + crypto_skcipher_ctx(crypto_skcipher_reqtfm(req)); + struct skcipher_request *subreq = skcipher_request_ctx(req); + + skcipher_request_set_tfm(subreq, ctx->skcipher_fbk); + skcipher_request_set_callback(subreq, req->base.flags, + req->base.complete, + req->base.data); + skcipher_request_set_crypt(subreq, req->src, req->dst, + req->cryptlen, req->iv); + + return enc ? crypto_skcipher_encrypt(subreq) : + crypto_skcipher_decrypt(subreq); +} + +static int starfive_sm4_crypt(struct skcipher_request *req, unsigned long flags) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct starfive_cryp_ctx *ctx = crypto_skcipher_ctx(tfm); + struct starfive_cryp_dev *cryp = ctx->cryp; + unsigned int blocksize_align = crypto_skcipher_blocksize(tfm) - 1; + + cryp->flags = flags; + + if ((cryp->flags & FLG_MODE_MASK) == STARFIVE_SM4_MODE_ECB || + (cryp->flags & FLG_MODE_MASK) == STARFIVE_SM4_MODE_CBC) + if (req->cryptlen & blocksize_align) + return -EINVAL; + + if (starfive_sm4_check_unaligned(cryp, req->src, req->dst)) + return starfive_sm4_do_fallback(req, is_encrypt(cryp)); + + return crypto_transfer_skcipher_request_to_engine(cryp->engine, req); +} + +static int starfive_sm4_aead_do_fallback(struct aead_request *req, bool enc) +{ + struct starfive_cryp_ctx *ctx = + crypto_aead_ctx(crypto_aead_reqtfm(req)); + struct aead_request *subreq = aead_request_ctx(req); + + aead_request_set_tfm(subreq, ctx->aead_fbk); + aead_request_set_callback(subreq, req->base.flags, + req->base.complete, + req->base.data); + aead_request_set_crypt(subreq, req->src, req->dst, + req->cryptlen, req->iv); + aead_request_set_ad(subreq, req->assoclen); + + return enc ? crypto_aead_encrypt(subreq) : + crypto_aead_decrypt(subreq); +} + +static int starfive_sm4_aead_crypt(struct aead_request *req, unsigned long flags) +{ + struct starfive_cryp_ctx *ctx = crypto_aead_ctx(crypto_aead_reqtfm(req)); + struct starfive_cryp_dev *cryp = ctx->cryp; + struct scatterlist *src, *dst, _src[2], _dst[2]; + + cryp->flags = flags; + + /* sm4-ccm does not support tag verification for non-aligned text, + * use fallback for ccm decryption instead. + */ + if (((cryp->flags & FLG_MODE_MASK) == STARFIVE_SM4_MODE_CCM) && + !is_encrypt(cryp)) + return starfive_sm4_aead_do_fallback(req, 0); + + src = scatterwalk_ffwd(_src, req->src, req->assoclen); + + if (req->src == req->dst) + dst = src; + else + dst = scatterwalk_ffwd(_dst, req->dst, req->assoclen); + + if (starfive_sm4_check_unaligned(cryp, src, dst)) + return starfive_sm4_aead_do_fallback(req, is_encrypt(cryp)); + + return crypto_transfer_aead_request_to_engine(cryp->engine, req); +} + +static int starfive_sm4_setkey(struct crypto_skcipher *tfm, const u8 *key, + unsigned int keylen) +{ + struct starfive_cryp_ctx *ctx = crypto_skcipher_ctx(tfm); + + if (!key || !keylen) + return -EINVAL; + + if (keylen != SM4_KEY_SIZE) + return -EINVAL; + + memcpy(ctx->key, key, keylen); + ctx->keylen = keylen; + + return crypto_skcipher_setkey(ctx->skcipher_fbk, key, keylen); +} + +static int starfive_sm4_aead_setkey(struct crypto_aead *tfm, const u8 *key, + unsigned int keylen) +{ + struct starfive_cryp_ctx *ctx = crypto_aead_ctx(tfm); + + if (!key || !keylen) + return -EINVAL; + + if (keylen != SM4_KEY_SIZE) + return -EINVAL; + + memcpy(ctx->key, key, keylen); + ctx->keylen = keylen; + + return crypto_aead_setkey(ctx->aead_fbk, key, keylen); +} + +static int starfive_sm4_gcm_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + struct starfive_cryp_ctx *ctx = crypto_aead_ctx(tfm); + int ret; + + ret = crypto_gcm_check_authsize(authsize); + if (ret) + return ret; + + return crypto_aead_setauthsize(ctx->aead_fbk, authsize); +} + +static int starfive_sm4_ccm_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + struct starfive_cryp_ctx *ctx = crypto_aead_ctx(tfm); + + switch (authsize) { + case 4: + case 6: + case 8: + case 10: + case 12: + case 14: + case 16: + break; + default: + return -EINVAL; + } + + return crypto_aead_setauthsize(ctx->aead_fbk, authsize); +} + +static int starfive_sm4_ecb_encrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_ECB | FLG_ENCRYPT); +} + +static int starfive_sm4_ecb_decrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_ECB); +} + +static int starfive_sm4_cbc_encrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_CBC | FLG_ENCRYPT); +} + +static int starfive_sm4_cbc_decrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_CBC); +} + +static int starfive_sm4_ctr_encrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_CTR | FLG_ENCRYPT); +} + +static int starfive_sm4_ctr_decrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_CTR); +} + +static int starfive_sm4_gcm_encrypt(struct aead_request *req) +{ + return starfive_sm4_aead_crypt(req, STARFIVE_SM4_MODE_GCM | FLG_ENCRYPT); +} + +static int starfive_sm4_gcm_decrypt(struct aead_request *req) +{ + return starfive_sm4_aead_crypt(req, STARFIVE_SM4_MODE_GCM); +} + +static int starfive_sm4_ccm_encrypt(struct aead_request *req) +{ + int ret; + + ret = starfive_sm4_ccm_check_iv(req->iv); + if (ret) + return ret; + + return starfive_sm4_aead_crypt(req, STARFIVE_SM4_MODE_CCM | FLG_ENCRYPT); +} + +static int starfive_sm4_ccm_decrypt(struct aead_request *req) +{ + int ret; + + ret = starfive_sm4_ccm_check_iv(req->iv); + if (ret) + return ret; + + return starfive_sm4_aead_crypt(req, STARFIVE_SM4_MODE_CCM); +} + +static int starfive_sm4_ecb_init_tfm(struct crypto_skcipher *tfm) +{ + return starfive_sm4_init_tfm(tfm, "ecb(sm4-generic)"); +} + +static int starfive_sm4_cbc_init_tfm(struct crypto_skcipher *tfm) +{ + return starfive_sm4_init_tfm(tfm, "cbc(sm4-generic)"); +} + +static int starfive_sm4_ctr_init_tfm(struct crypto_skcipher *tfm) +{ + return starfive_sm4_init_tfm(tfm, "ctr(sm4-generic)"); +} + +static int starfive_sm4_ccm_aead_init_tfm(struct crypto_aead *tfm) +{ + return starfive_sm4_aead_init_tfm(tfm, "ccm_base(ctr(sm4-generic),cbcmac(sm4-generic))"); +} + +static int starfive_sm4_gcm_aead_init_tfm(struct crypto_aead *tfm) +{ + return starfive_sm4_aead_init_tfm(tfm, "gcm_base(ctr(sm4-generic),ghash-generic)"); +} + +static struct skcipher_engine_alg skcipher_sm4[] = { +{ + .base.init = starfive_sm4_ecb_init_tfm, + .base.exit = starfive_sm4_exit_tfm, + .base.setkey = starfive_sm4_setkey, + .base.encrypt = starfive_sm4_ecb_encrypt, + .base.decrypt = starfive_sm4_ecb_decrypt, + .base.min_keysize = SM4_KEY_SIZE, + .base.max_keysize = SM4_KEY_SIZE, + .base.base = { + .cra_name = "ecb(sm4)", + .cra_driver_name = "starfive-ecb-sm4", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize = SM4_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct starfive_cryp_ctx), + .cra_alignmask = 0xf, + .cra_module = THIS_MODULE, + }, + .op = { + .do_one_request = starfive_sm4_do_one_req, + }, +}, { + .base.init = starfive_sm4_ctr_init_tfm, + .base.exit = starfive_sm4_exit_tfm, + .base.setkey = starfive_sm4_setkey, + .base.encrypt = starfive_sm4_ctr_encrypt, + .base.decrypt = starfive_sm4_ctr_decrypt, + .base.min_keysize = SM4_KEY_SIZE, + .base.max_keysize = SM4_KEY_SIZE, + .base.ivsize = SM4_BLOCK_SIZE, + .base.base = { + .cra_name = "ctr(sm4)", + .cra_driver_name = "starfive-ctr-sm4", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct starfive_cryp_ctx), + .cra_alignmask = 0xf, + .cra_module = THIS_MODULE, + }, + .op = { + .do_one_request = starfive_sm4_do_one_req, + }, +}, { + .base.init = starfive_sm4_cbc_init_tfm, + .base.exit = starfive_sm4_exit_tfm, + .base.setkey = starfive_sm4_setkey, + .base.encrypt = starfive_sm4_cbc_encrypt, + .base.decrypt = starfive_sm4_cbc_decrypt, + .base.min_keysize = SM4_KEY_SIZE, + .base.max_keysize = SM4_KEY_SIZE, + .base.ivsize = SM4_BLOCK_SIZE, + .base.base = { + .cra_name = "cbc(sm4)", + .cra_driver_name = "starfive-cbc-sm4", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize = SM4_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct starfive_cryp_ctx), + .cra_alignmask = 0xf, + .cra_module = THIS_MODULE, + }, + .op = { + .do_one_request = starfive_sm4_do_one_req, + }, +}, +}; + +static struct aead_engine_alg aead_sm4[] = { +{ + .base.setkey = starfive_sm4_aead_setkey, + .base.setauthsize = starfive_sm4_gcm_setauthsize, + .base.encrypt = starfive_sm4_gcm_encrypt, + .base.decrypt = starfive_sm4_gcm_decrypt, + .base.init = starfive_sm4_gcm_aead_init_tfm, + .base.exit = starfive_sm4_aead_exit_tfm, + .base.ivsize = GCM_AES_IV_SIZE, + .base.maxauthsize = SM4_BLOCK_SIZE, + .base.base = { + .cra_name = "gcm(sm4)", + .cra_driver_name = "starfive-gcm-sm4", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct starfive_cryp_ctx), + .cra_alignmask = 0xf, + .cra_module = THIS_MODULE, + }, + .op = { + .do_one_request = starfive_sm4_aead_do_one_req, + }, +}, { + .base.setkey = starfive_sm4_aead_setkey, + .base.setauthsize = starfive_sm4_ccm_setauthsize, + .base.encrypt = starfive_sm4_ccm_encrypt, + .base.decrypt = starfive_sm4_ccm_decrypt, + .base.init = starfive_sm4_ccm_aead_init_tfm, + .base.exit = starfive_sm4_aead_exit_tfm, + .base.ivsize = SM4_BLOCK_SIZE, + .base.maxauthsize = SM4_BLOCK_SIZE, + .base.base = { + .cra_name = "ccm(sm4)", + .cra_driver_name = "starfive-ccm-sm4", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct starfive_cryp_ctx), + .cra_alignmask = 0xf, + .cra_module = THIS_MODULE, + }, + .op = { + .do_one_request = starfive_sm4_aead_do_one_req, + }, +}, +}; + +int starfive_sm4_register_algs(void) +{ + int ret; + + ret = crypto_engine_register_skciphers(skcipher_sm4, ARRAY_SIZE(skcipher_sm4)); + if (ret) + return ret; + + ret = crypto_engine_register_aeads(aead_sm4, ARRAY_SIZE(aead_sm4)); + if (ret) + crypto_engine_unregister_skciphers(skcipher_sm4, ARRAY_SIZE(skcipher_sm4)); + + return ret; +} + +void starfive_sm4_unregister_algs(void) +{ + crypto_engine_unregister_aeads(aead_sm4, ARRAY_SIZE(aead_sm4)); + crypto_engine_unregister_skciphers(skcipher_sm4, ARRAY_SIZE(skcipher_sm4)); +}