From patchwork Tue Nov 5 14:00:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tero Kristo X-Patchwork-Id: 178498 Delivered-To: patch@linaro.org Received: by 2002:ac9:3c86:0:0:0:0:0 with SMTP id w6csp709140ocf; Tue, 5 Nov 2019 06:01:29 -0800 (PST) X-Google-Smtp-Source: APXvYqw164Tib1EJEnGZMjHsASBCqiWZHCgSW3Nts+4tV83VbaCzX/lJo+QYH0vihMbQdkASEfN+ X-Received: by 2002:a17:906:ecad:: with SMTP id qh13mr7483533ejb.25.1572962489092; Tue, 05 Nov 2019 06:01:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1572962489; cv=none; d=google.com; s=arc-20160816; b=zKSu4CxzaD0BC2c9YT2rNfUWAjfXVn/RHKmWUuwFfle69XRf/9IoX1JIrdyxK3bKm1 asE4vCLwNKSwa8kGwYRNB5Y4v4J9NacG9AGIdO70MJLr5JmYcSjShqJL6Gr/+LOfycXx 70dl7wehgN2dcQNkU7SWhmJPnwWvm0X4JeLrgCpiCzU3rZvtikzKRHRG2kfNx7uOq0b8 P7b3l38/K0QXGKvnsNnqbbeEcMZDdfH/iFoTJcz5jqvyTQcH+4wKaWgbP+bKrvlF2myS 3F6CohcQp793vBQai11aZewbQh20PqVOMpfjPHz5LAN7ZJh8/qryB7Vu6X3mRikuZrJU oLog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=5LuvH1G1CxJXjnQPeRbfhDmv2ndruQP/Xg9DUfak+xk=; b=e6rUpSjGyCFqFHZD2Cqh5qZdF0t+LGYHNgBJtzm1Y/uVB/xOUnD7mE0eaGJ4na/Qs4 rP4lYodGVjxgYyqI9i1BqFu8NTJBcPZ3g+tpdaG8tKAfvW8dyTQT0BDf1DK9oS/7a2o1 lbylysQUGYGLcDHGMP+ko6xJRajKN/61jB0+fZFNjZgwPyghiY7zY4vf/GqpDaD55veO rxiqnLHEkjh2P3XRJL63I+SR3Xz1pdSQARTmqqOCDlqWxJ3kYqHSNlOIV4BxKX5qdPvC OuDsNENV4yY58gG1OWI3JtBwzmPmf/KDpByFSKcOAAc2RZeHv3MUPXpP8Nh7pyTarrW2 +AAw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=niojAQcb; spf=pass (google.com: best guess record for domain of linux-omap-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-omap-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id dk11si13340421ejb.97.2019.11.05.06.01.28; Tue, 05 Nov 2019 06:01:29 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-omap-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@ti.com header.s=ti-com-17Q1 header.b=niojAQcb; spf=pass (google.com: best guess record for domain of linux-omap-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-omap-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=ti.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389294AbfKEOB2 (ORCPT + 5 others); Tue, 5 Nov 2019 09:01:28 -0500 Received: from fllv0016.ext.ti.com ([198.47.19.142]:44666 "EHLO fllv0016.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389290AbfKEOB2 (ORCPT ); Tue, 5 Nov 2019 09:01:28 -0500 Received: from lelv0265.itg.ti.com ([10.180.67.224]) by fllv0016.ext.ti.com (8.15.2/8.15.2) with ESMTP id xA5E1JmZ043337; Tue, 5 Nov 2019 08:01:19 -0600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1572962479; bh=5LuvH1G1CxJXjnQPeRbfhDmv2ndruQP/Xg9DUfak+xk=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=niojAQcbK7t1Ye8eAGH6QDDfvEMsfaPK8HqWSrDMoYjRPgfw2svW1CBTWMfHqdRuw iP6i0mzD3dOK5DzCnRVRBRnNUF8nBqCrVZfU/v09NB0hFCfzr1HfqgxaIgchsURMgP wAYWstDhWpN1df2uk2op+VVq95PP8MauFSQ7l3Pg= Received: from DFLE112.ent.ti.com (dfle112.ent.ti.com [10.64.6.33]) by lelv0265.itg.ti.com (8.15.2/8.15.2) with ESMTPS id xA5E1JP2007694 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 5 Nov 2019 08:01:19 -0600 Received: from DFLE112.ent.ti.com (10.64.6.33) by DFLE112.ent.ti.com (10.64.6.33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1713.5; Tue, 5 Nov 2019 08:01:04 -0600 Received: from fllv0040.itg.ti.com (10.64.41.20) by DFLE112.ent.ti.com (10.64.6.33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1713.5 via Frontend Transport; Tue, 5 Nov 2019 08:01:04 -0600 Received: from sokoban.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by fllv0040.itg.ti.com (8.15.2/8.15.2) with ESMTP id xA5E1FFe068289; Tue, 5 Nov 2019 08:01:17 -0600 From: Tero Kristo To: , , CC: , Subject: [PATCHv2 01/22] crypto: omap-sham: split up data to multiple sg elements with huge data Date: Tue, 5 Nov 2019 16:00:50 +0200 Message-ID: <20191105140111.20285-2-t-kristo@ti.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191105140111.20285-1-t-kristo@ti.com> References: <20191105140111.20285-1-t-kristo@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: linux-omap-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-omap@vger.kernel.org When using huge data amount, allocating free pages fails as the kernel isn't able to process get_free_page requests larger than MAX_ORDER. Also, the DMA subsystem has an inherent limitation that data size larger than some 2MB can't be handled properly. In these cases, split up the data instead to smaller requests so that the kernel can allocate the data, and also so that the DMA driver can handle the separate SG elements. Signed-off-by: Tero Kristo Tested-by: Bin Liu --- drivers/crypto/omap-sham.c | 81 ++++++++++++++++++++++++++++++-------- 1 file changed, 64 insertions(+), 17 deletions(-) -- 2.17.1 -- Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c index ac80bc6af093..2e9435577cea 100644 --- a/drivers/crypto/omap-sham.c +++ b/drivers/crypto/omap-sham.c @@ -112,6 +112,8 @@ #define FLAGS_BE32_SHA1 8 #define FLAGS_SGS_COPIED 9 #define FLAGS_SGS_ALLOCED 10 +#define FLAGS_HUGE 11 + /* context flags */ #define FLAGS_FINUP 16 @@ -136,6 +138,8 @@ #define BUFLEN SHA512_BLOCK_SIZE #define OMAP_SHA_DMA_THRESHOLD 256 +#define OMAP_SHA_MAX_DMA_LEN (1024 * 2048) + struct omap_sham_dev; struct omap_sham_reqctx { @@ -689,21 +693,20 @@ static int omap_sham_copy_sg_lists(struct omap_sham_reqctx *ctx, set_bit(FLAGS_SGS_ALLOCED, &ctx->dd->flags); + ctx->offset += new_len - ctx->bufcnt; ctx->bufcnt = 0; return 0; } static int omap_sham_copy_sgs(struct omap_sham_reqctx *ctx, - struct scatterlist *sg, int bs, int new_len) + struct scatterlist *sg, int bs, + unsigned int new_len) { int pages; void *buf; - int len; - - len = new_len + ctx->bufcnt; - pages = get_order(ctx->total); + pages = get_order(new_len); buf = (void *)__get_free_pages(GFP_ATOMIC, pages); if (!buf) { @@ -715,14 +718,14 @@ static int omap_sham_copy_sgs(struct omap_sham_reqctx *ctx, memcpy(buf, ctx->dd->xmit_buf, ctx->bufcnt); scatterwalk_map_and_copy(buf + ctx->bufcnt, sg, ctx->offset, - ctx->total - ctx->bufcnt, 0); + min(new_len, ctx->total) - ctx->bufcnt, 0); sg_init_table(ctx->sgl, 1); - sg_set_buf(ctx->sgl, buf, len); + sg_set_buf(ctx->sgl, buf, new_len); ctx->sg = ctx->sgl; set_bit(FLAGS_SGS_COPIED, &ctx->dd->flags); ctx->sg_len = 1; + ctx->offset += new_len - ctx->bufcnt; ctx->bufcnt = 0; - ctx->offset = 0; return 0; } @@ -741,7 +744,7 @@ static int omap_sham_align_sgs(struct scatterlist *sg, if (!sg || !sg->length || !nbytes) return 0; - new_len = nbytes; + new_len = nbytes - offset; if (offset) list_ok = false; @@ -751,6 +754,9 @@ static int omap_sham_align_sgs(struct scatterlist *sg, else new_len = (new_len - 1) / bs * bs; + if (!new_len) + return 0; + if (nbytes != new_len) list_ok = false; @@ -794,10 +800,17 @@ static int omap_sham_align_sgs(struct scatterlist *sg, } } + if (new_len > OMAP_SHA_MAX_DMA_LEN) { + new_len = OMAP_SHA_MAX_DMA_LEN; + aligned = false; + } + if (!aligned) return omap_sham_copy_sgs(rctx, sg, bs, new_len); else if (!list_ok) return omap_sham_copy_sg_lists(rctx, sg, bs, new_len); + else + rctx->offset += new_len; rctx->sg_len = n; rctx->sg = sg; @@ -821,7 +834,12 @@ static int omap_sham_prepare_request(struct ahash_request *req, bool update) else nbytes = 0; - rctx->total = nbytes + rctx->bufcnt; + rctx->total = nbytes + rctx->bufcnt - rctx->offset; + + dev_dbg(rctx->dd->dev, + "%s: nbytes=%d, bs=%d, total=%d, offset=%d, bufcnt=%d\n", + __func__, nbytes, bs, rctx->total, rctx->offset, + rctx->bufcnt); if (!rctx->total) return 0; @@ -847,12 +865,15 @@ static int omap_sham_prepare_request(struct ahash_request *req, bool update) xmit_len = rctx->total; + if (xmit_len > OMAP_SHA_MAX_DMA_LEN) + xmit_len = OMAP_SHA_MAX_DMA_LEN; + if (!IS_ALIGNED(xmit_len, bs)) { if (final) xmit_len = DIV_ROUND_UP(xmit_len, bs) * bs; else xmit_len = xmit_len / bs * bs; - } else if (!final) { + } else if (!final && rctx->total == xmit_len) { xmit_len -= bs; } @@ -880,7 +901,7 @@ static int omap_sham_prepare_request(struct ahash_request *req, bool update) rctx->sg_len = 1; } - if (hash_later) { + if (hash_later && hash_later <= rctx->buflen) { int offset = 0; if (hash_later > req->nbytes) { @@ -901,6 +922,9 @@ static int omap_sham_prepare_request(struct ahash_request *req, bool update) rctx->bufcnt = 0; } + if (hash_later > rctx->buflen) + set_bit(FLAGS_HUGE, &rctx->dd->flags); + if (!final) rctx->total = xmit_len; @@ -998,10 +1022,11 @@ static int omap_sham_update_req(struct omap_sham_dev *dd) struct ahash_request *req = dd->req; struct omap_sham_reqctx *ctx = ahash_request_ctx(req); int err; - bool final = ctx->flags & BIT(FLAGS_FINUP); + bool final = (ctx->flags & BIT(FLAGS_FINUP)) && + !(dd->flags & BIT(FLAGS_HUGE)); - dev_dbg(dd->dev, "update_req: total: %u, digcnt: %d, finup: %d\n", - ctx->total, ctx->digcnt, (ctx->flags & BIT(FLAGS_FINUP)) != 0); + dev_dbg(dd->dev, "update_req: total: %u, digcnt: %d, final: %d", + ctx->total, ctx->digcnt, final); if (ctx->total < get_block_size(ctx) || ctx->total < dd->fallback_sz) @@ -1024,6 +1049,9 @@ static int omap_sham_final_req(struct omap_sham_dev *dd) struct omap_sham_reqctx *ctx = ahash_request_ctx(req); int err = 0, use_dma = 1; + if (dd->flags & BIT(FLAGS_HUGE)) + return 0; + if ((ctx->total <= get_block_size(ctx)) || dd->polling_mode) /* * faster to handle last block with cpu or @@ -1083,7 +1111,7 @@ static void omap_sham_finish_req(struct ahash_request *req, int err) if (test_bit(FLAGS_SGS_COPIED, &dd->flags)) free_pages((unsigned long)sg_virt(ctx->sg), - get_order(ctx->sg->length + ctx->bufcnt)); + get_order(ctx->sg->length)); if (test_bit(FLAGS_SGS_ALLOCED, &dd->flags)) kfree(ctx->sg); @@ -1092,6 +1120,21 @@ static void omap_sham_finish_req(struct ahash_request *req, int err) dd->flags &= ~(BIT(FLAGS_SGS_ALLOCED) | BIT(FLAGS_SGS_COPIED)); + if (dd->flags & BIT(FLAGS_HUGE)) { + dd->flags &= ~(BIT(FLAGS_CPU) | BIT(FLAGS_DMA_READY) | + BIT(FLAGS_OUTPUT_READY) | BIT(FLAGS_HUGE)); + omap_sham_prepare_request(req, ctx->op == OP_UPDATE); + if (ctx->op == OP_UPDATE || (dd->flags & BIT(FLAGS_HUGE))) { + err = omap_sham_update_req(dd); + if (err != -EINPROGRESS && + (ctx->flags & BIT(FLAGS_FINUP))) + err = omap_sham_final_req(dd); + } else if (ctx->op == OP_FINAL) { + omap_sham_final_req(dd); + } + return; + } + if (!err) { dd->pdata->copy_hash(req, 1); if (test_bit(FLAGS_FINAL, &dd->flags)) @@ -1107,6 +1150,8 @@ static void omap_sham_finish_req(struct ahash_request *req, int err) pm_runtime_mark_last_busy(dd->dev); pm_runtime_put_autosuspend(dd->dev); + ctx->offset = 0; + if (req->base.complete) req->base.complete(&req->base, err); } @@ -1158,7 +1203,7 @@ static int omap_sham_handle_queue(struct omap_sham_dev *dd, /* request has changed - restore hash */ dd->pdata->copy_hash(req, 0); - if (ctx->op == OP_UPDATE) { + if (ctx->op == OP_UPDATE || (dd->flags & BIT(FLAGS_HUGE))) { err = omap_sham_update_req(dd); if (err != -EINPROGRESS && (ctx->flags & BIT(FLAGS_FINUP))) /* no final() after finup() */ @@ -1730,6 +1775,8 @@ static void omap_sham_done_task(unsigned long data) struct omap_sham_dev *dd = (struct omap_sham_dev *)data; int err = 0; + dev_dbg(dd->dev, "%s: flags=%lx\n", __func__, dd->flags); + if (!test_bit(FLAGS_BUSY, &dd->flags)) { omap_sham_handle_queue(dd, NULL); return;