From patchwork Fri May 6 13:16:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhenwei pi X-Patchwork-Id: 571110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82469C433FE for ; Fri, 6 May 2022 13:21:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347327AbiEFNYt (ORCPT ); Fri, 6 May 2022 09:24:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1392346AbiEFNYr (ORCPT ); Fri, 6 May 2022 09:24:47 -0400 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9660063522 for ; Fri, 6 May 2022 06:20:59 -0700 (PDT) Received: by mail-pj1-x102a.google.com with SMTP id o69so7005822pjo.3 for ; Fri, 06 May 2022 06:20:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gxtlpe27hVHFlryxYz6UvDrK5GqQy2lJ9Lv/VyAYRQ0=; b=fiaSZ0sj1HdZGngOQgS/RiRT4UNti1mz7ikVsFlAyWALRFeEc7iepZNpIM/QjRea3K lAt+FO1K0nuwhkAlvjgyszJ+Tx9ek9JXWcl+QEYIFoXDtO70fICmKRCTc/XGRc4AUCfM IwnAiBXnilyNTO42THJk6aaZG3wDsGTvR0+xCGoRpB/LkTt5kBtFyqWspE2lnZR4b/5f tEPxVc0awSuL/3yEvAAoLRUhJ99CJH4FtTIJKLUJFqtpf1GSDH/f7iIBHwJbv8t9xex0 wn0hYF7z4PUFHqZQNsJIpZTbu+gbDL/FDcgAU6/dZ8mSrtmf/Y8AU8V6vgMSGRkZNYmQ ZDLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gxtlpe27hVHFlryxYz6UvDrK5GqQy2lJ9Lv/VyAYRQ0=; b=ox+Ay2eZMWAaX6VW0K3NYwYAThonzHcbch7NFvQktHp+rohy20rlHBCs6QT7erHapC E6j5q9Bg7F3+0QdaCBduSVoIom75rNAsnJ2YMY7dwD6eSPjAwaRzukIaCbdsfP7OmjLt IhSGdPzPeKrtQIhwp0CItVRwdcg5HR8qjBS3Wfpy/0nGK21PYZlXGsYmcMsVzAY/bcLh L1se4efIwVSf26xH2TXkyz3epb6yRN5nN3isHRkx1qsq1L6iaJlyzDPjCasNIoH5EUx2 gBYTwvS4N+jzMc4aPayTgvBhQ4Riqs2CARwbIYPW5s0AYPPTkenmSpVuYJyVlZs8D22y EHFA== X-Gm-Message-State: AOAM5306tYjuZEhlPHxP5Yruy/kYhdBsNFRmT6EqPo3ufMEhhUY5ipH1 mNqB8k9pZifbmXqeBaAvjZzv7Q== X-Google-Smtp-Source: ABdhPJyFt3NgotMDg2Mh6FuA61wITtjBFZcoJC9kmxO1nloy3urOufFraEJ444VvH58hfCHln8NrKA== X-Received: by 2002:a17:90b:245:b0:1dc:1597:20e with SMTP id fz5-20020a17090b024500b001dc1597020emr4127591pjb.151.1651843258997; Fri, 06 May 2022 06:20:58 -0700 (PDT) Received: from always-x1.www.tendawifi.com ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id i22-20020a63e916000000b003c14af50643sm3256986pgh.91.2022.05.06.06.20.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 May 2022 06:20:58 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, pizhenwei@bytedance.com, davem@davemloft.net Subject: [PATCH v6 1/5] virtio-crypto: change code style Date: Fri, 6 May 2022 21:16:23 +0800 Message-Id: <20220506131627.180784-2-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220506131627.180784-1-pizhenwei@bytedance.com> References: <20220506131627.180784-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Use temporary variable to make code easy to read and maintain. /* Pad cipher's parameters */ vcrypto->ctrl.u.sym_create_session.op_type = cpu_to_le32(VIRTIO_CRYPTO_SYM_OP_CIPHER); vcrypto->ctrl.u.sym_create_session.u.cipher.para.algo = vcrypto->ctrl.header.algo; vcrypto->ctrl.u.sym_create_session.u.cipher.para.keylen = cpu_to_le32(keylen); vcrypto->ctrl.u.sym_create_session.u.cipher.para.op = cpu_to_le32(op); --> sym_create_session = &ctrl->u.sym_create_session; sym_create_session->op_type = cpu_to_le32(VIRTIO_CRYPTO_SYM_OP_CIPHER); sym_create_session->u.cipher.para.algo = ctrl->header.algo; sym_create_session->u.cipher.para.keylen = cpu_to_le32(keylen); sym_create_session->u.cipher.para.op = cpu_to_le32(op); The new style shows more obviously: - the variable we want to operate. - an assignment statement in a single line. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Reviewed-by: Gonglei Signed-off-by: zhenwei pi --- .../virtio/virtio_crypto_akcipher_algs.c | 40 ++++++----- .../virtio/virtio_crypto_skcipher_algs.c | 72 +++++++++---------- 2 files changed, 59 insertions(+), 53 deletions(-) diff --git a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c index f3ec9420215e..20901a263fc8 100644 --- a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c @@ -106,23 +106,27 @@ static int virtio_crypto_alg_akcipher_init_session(struct virtio_crypto_akcipher unsigned int inlen; int err; unsigned int num_out = 0, num_in = 0; + struct virtio_crypto_op_ctrl_req *ctrl; + struct virtio_crypto_session_input *input; pkey = kmemdup(key, keylen, GFP_ATOMIC); if (!pkey) return -ENOMEM; spin_lock(&vcrypto->ctrl_lock); - memcpy(&vcrypto->ctrl.header, header, sizeof(vcrypto->ctrl.header)); - memcpy(&vcrypto->ctrl.u, para, sizeof(vcrypto->ctrl.u)); - vcrypto->input.status = cpu_to_le32(VIRTIO_CRYPTO_ERR); + ctrl = &vcrypto->ctrl; + memcpy(&ctrl->header, header, sizeof(ctrl->header)); + memcpy(&ctrl->u, para, sizeof(ctrl->u)); + input = &vcrypto->input; + input->status = cpu_to_le32(VIRTIO_CRYPTO_ERR); - sg_init_one(&outhdr_sg, &vcrypto->ctrl, sizeof(vcrypto->ctrl)); + sg_init_one(&outhdr_sg, ctrl, sizeof(*ctrl)); sgs[num_out++] = &outhdr_sg; sg_init_one(&key_sg, pkey, keylen); sgs[num_out++] = &key_sg; - sg_init_one(&inhdr_sg, &vcrypto->input, sizeof(vcrypto->input)); + sg_init_one(&inhdr_sg, input, sizeof(*input)); sgs[num_out + num_in++] = &inhdr_sg; err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto, GFP_ATOMIC); @@ -134,12 +138,12 @@ static int virtio_crypto_alg_akcipher_init_session(struct virtio_crypto_akcipher !virtqueue_is_broken(vcrypto->ctrl_vq)) cpu_relax(); - if (le32_to_cpu(vcrypto->input.status) != VIRTIO_CRYPTO_OK) { + if (le32_to_cpu(input->status) != VIRTIO_CRYPTO_OK) { err = -EINVAL; goto out; } - ctx->session_id = le64_to_cpu(vcrypto->input.session_id); + ctx->session_id = le64_to_cpu(input->session_id); ctx->session_valid = true; err = 0; @@ -149,7 +153,7 @@ static int virtio_crypto_alg_akcipher_init_session(struct virtio_crypto_akcipher if (err < 0) pr_err("virtio_crypto: Create session failed status: %u\n", - le32_to_cpu(vcrypto->input.status)); + le32_to_cpu(input->status)); return err; } @@ -161,23 +165,27 @@ static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_akciphe struct virtio_crypto *vcrypto = ctx->vcrypto; unsigned int num_out = 0, num_in = 0, inlen; int err; + struct virtio_crypto_op_ctrl_req *ctrl; + struct virtio_crypto_inhdr *ctrl_status; spin_lock(&vcrypto->ctrl_lock); if (!ctx->session_valid) { err = 0; goto out; } - vcrypto->ctrl_status.status = VIRTIO_CRYPTO_ERR; - vcrypto->ctrl.header.opcode = cpu_to_le32(VIRTIO_CRYPTO_AKCIPHER_DESTROY_SESSION); - vcrypto->ctrl.header.queue_id = 0; + ctrl_status = &vcrypto->ctrl_status; + ctrl_status->status = VIRTIO_CRYPTO_ERR; + ctrl = &vcrypto->ctrl; + ctrl->header.opcode = cpu_to_le32(VIRTIO_CRYPTO_AKCIPHER_DESTROY_SESSION); + ctrl->header.queue_id = 0; - destroy_session = &vcrypto->ctrl.u.destroy_session; + destroy_session = &ctrl->u.destroy_session; destroy_session->session_id = cpu_to_le64(ctx->session_id); - sg_init_one(&outhdr_sg, &vcrypto->ctrl, sizeof(vcrypto->ctrl)); + sg_init_one(&outhdr_sg, ctrl, sizeof(*ctrl)); sgs[num_out++] = &outhdr_sg; - sg_init_one(&inhdr_sg, &vcrypto->ctrl_status.status, sizeof(vcrypto->ctrl_status.status)); + sg_init_one(&inhdr_sg, &ctrl_status->status, sizeof(ctrl_status->status)); sgs[num_out + num_in++] = &inhdr_sg; err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto, GFP_ATOMIC); @@ -189,7 +197,7 @@ static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_akciphe !virtqueue_is_broken(vcrypto->ctrl_vq)) cpu_relax(); - if (vcrypto->ctrl_status.status != VIRTIO_CRYPTO_OK) { + if (ctrl_status->status != VIRTIO_CRYPTO_OK) { err = -EINVAL; goto out; } @@ -201,7 +209,7 @@ static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_akciphe spin_unlock(&vcrypto->ctrl_lock); if (err < 0) { pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%llx\n", - vcrypto->ctrl_status.status, destroy_session->session_id); + ctrl_status->status, destroy_session->session_id); } return err; diff --git a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c index a618c46a52b8..e3c5bc8d6112 100644 --- a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c @@ -123,6 +123,9 @@ static int virtio_crypto_alg_skcipher_init_session( int op = encrypt ? VIRTIO_CRYPTO_OP_ENCRYPT : VIRTIO_CRYPTO_OP_DECRYPT; int err; unsigned int num_out = 0, num_in = 0; + struct virtio_crypto_op_ctrl_req *ctrl; + struct virtio_crypto_session_input *input; + struct virtio_crypto_sym_create_session_req *sym_create_session; /* * Avoid to do DMA from the stack, switch to using @@ -135,24 +138,22 @@ static int virtio_crypto_alg_skcipher_init_session( spin_lock(&vcrypto->ctrl_lock); /* Pad ctrl header */ - vcrypto->ctrl.header.opcode = - cpu_to_le32(VIRTIO_CRYPTO_CIPHER_CREATE_SESSION); - vcrypto->ctrl.header.algo = cpu_to_le32(alg); + ctrl = &vcrypto->ctrl; + ctrl->header.opcode = cpu_to_le32(VIRTIO_CRYPTO_CIPHER_CREATE_SESSION); + ctrl->header.algo = cpu_to_le32(alg); /* Set the default dataqueue id to 0 */ - vcrypto->ctrl.header.queue_id = 0; + ctrl->header.queue_id = 0; - vcrypto->input.status = cpu_to_le32(VIRTIO_CRYPTO_ERR); + input = &vcrypto->input; + input->status = cpu_to_le32(VIRTIO_CRYPTO_ERR); /* Pad cipher's parameters */ - vcrypto->ctrl.u.sym_create_session.op_type = - cpu_to_le32(VIRTIO_CRYPTO_SYM_OP_CIPHER); - vcrypto->ctrl.u.sym_create_session.u.cipher.para.algo = - vcrypto->ctrl.header.algo; - vcrypto->ctrl.u.sym_create_session.u.cipher.para.keylen = - cpu_to_le32(keylen); - vcrypto->ctrl.u.sym_create_session.u.cipher.para.op = - cpu_to_le32(op); - - sg_init_one(&outhdr, &vcrypto->ctrl, sizeof(vcrypto->ctrl)); + sym_create_session = &ctrl->u.sym_create_session; + sym_create_session->op_type = cpu_to_le32(VIRTIO_CRYPTO_SYM_OP_CIPHER); + sym_create_session->u.cipher.para.algo = ctrl->header.algo; + sym_create_session->u.cipher.para.keylen = cpu_to_le32(keylen); + sym_create_session->u.cipher.para.op = cpu_to_le32(op); + + sg_init_one(&outhdr, ctrl, sizeof(*ctrl)); sgs[num_out++] = &outhdr; /* Set key */ @@ -160,7 +161,7 @@ static int virtio_crypto_alg_skcipher_init_session( sgs[num_out++] = &key_sg; /* Return status and session id back */ - sg_init_one(&inhdr, &vcrypto->input, sizeof(vcrypto->input)); + sg_init_one(&inhdr, input, sizeof(*input)); sgs[num_out + num_in++] = &inhdr; err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, @@ -180,20 +181,18 @@ static int virtio_crypto_alg_skcipher_init_session( !virtqueue_is_broken(vcrypto->ctrl_vq)) cpu_relax(); - if (le32_to_cpu(vcrypto->input.status) != VIRTIO_CRYPTO_OK) { + if (le32_to_cpu(input->status) != VIRTIO_CRYPTO_OK) { spin_unlock(&vcrypto->ctrl_lock); pr_err("virtio_crypto: Create session failed status: %u\n", - le32_to_cpu(vcrypto->input.status)); + le32_to_cpu(input->status)); kfree_sensitive(cipher_key); return -EINVAL; } if (encrypt) - ctx->enc_sess_info.session_id = - le64_to_cpu(vcrypto->input.session_id); + ctx->enc_sess_info.session_id = le64_to_cpu(input->session_id); else - ctx->dec_sess_info.session_id = - le64_to_cpu(vcrypto->input.session_id); + ctx->dec_sess_info.session_id = le64_to_cpu(input->session_id); spin_unlock(&vcrypto->ctrl_lock); @@ -211,30 +210,30 @@ static int virtio_crypto_alg_skcipher_close_session( struct virtio_crypto *vcrypto = ctx->vcrypto; int err; unsigned int num_out = 0, num_in = 0; + struct virtio_crypto_op_ctrl_req *ctrl; + struct virtio_crypto_inhdr *ctrl_status; spin_lock(&vcrypto->ctrl_lock); - vcrypto->ctrl_status.status = VIRTIO_CRYPTO_ERR; + ctrl_status = &vcrypto->ctrl_status; + ctrl_status->status = VIRTIO_CRYPTO_ERR; /* Pad ctrl header */ - vcrypto->ctrl.header.opcode = - cpu_to_le32(VIRTIO_CRYPTO_CIPHER_DESTROY_SESSION); + ctrl = &vcrypto->ctrl; + ctrl->header.opcode = cpu_to_le32(VIRTIO_CRYPTO_CIPHER_DESTROY_SESSION); /* Set the default virtqueue id to 0 */ - vcrypto->ctrl.header.queue_id = 0; + ctrl->header.queue_id = 0; - destroy_session = &vcrypto->ctrl.u.destroy_session; + destroy_session = &ctrl->u.destroy_session; if (encrypt) - destroy_session->session_id = - cpu_to_le64(ctx->enc_sess_info.session_id); + destroy_session->session_id = cpu_to_le64(ctx->enc_sess_info.session_id); else - destroy_session->session_id = - cpu_to_le64(ctx->dec_sess_info.session_id); + destroy_session->session_id = cpu_to_le64(ctx->dec_sess_info.session_id); - sg_init_one(&outhdr, &vcrypto->ctrl, sizeof(vcrypto->ctrl)); + sg_init_one(&outhdr, ctrl, sizeof(*ctrl)); sgs[num_out++] = &outhdr; /* Return status and session id back */ - sg_init_one(&status_sg, &vcrypto->ctrl_status.status, - sizeof(vcrypto->ctrl_status.status)); + sg_init_one(&status_sg, &ctrl_status->status, sizeof(ctrl_status->status)); sgs[num_out + num_in++] = &status_sg; err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, @@ -249,11 +248,10 @@ static int virtio_crypto_alg_skcipher_close_session( !virtqueue_is_broken(vcrypto->ctrl_vq)) cpu_relax(); - if (vcrypto->ctrl_status.status != VIRTIO_CRYPTO_OK) { + if (ctrl_status->status != VIRTIO_CRYPTO_OK) { spin_unlock(&vcrypto->ctrl_lock); pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%llx\n", - vcrypto->ctrl_status.status, - destroy_session->session_id); + ctrl_status->status, destroy_session->session_id); return -EINVAL; } From patchwork Fri May 6 13:16:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhenwei pi X-Patchwork-Id: 570266 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F4C0C433EF for ; Fri, 6 May 2022 13:21:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1389697AbiEFNYt (ORCPT ); Fri, 6 May 2022 09:24:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1392376AbiEFNYs (ORCPT ); Fri, 6 May 2022 09:24:48 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 772FB689B3 for ; Fri, 6 May 2022 06:21:04 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id iq10so7036151pjb.0 for ; Fri, 06 May 2022 06:21:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LuuE2YXgICRMZBxIWmCO8QH07jpKhjA+nCktxwJxhy0=; b=zDu0JncO02gniMUhgCo2AOT/a854YgJ83RVH2ExbaB8xBgT9YyyGQLaMzuiDB79626 tdhiytn4aLJmDFEdhZR1niwWcRX5kEQHbetBPXA7yBi7DKdRSsw4ImjQea7KhULfzuCt clDz1WdwGoAs78QWCaf1vTPSbIupZJDeWlcRKQUMOyeyZoA2eQmVGNQo8tgHoWWYrBd5 frrCStUs3BVIJM1Pux+TmjKJRmTU0oEVsVUFBSHnBvTkHdNRrk3SF70797RXi8vmkvbF kefzUrjKE4QjNyh3AGrhrZOZMPGV1rJ7/F8qFKBaWrib6CA1tozyCqEtcmoQ64sN201G xwGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LuuE2YXgICRMZBxIWmCO8QH07jpKhjA+nCktxwJxhy0=; b=2MbseX70Ys/zdKlDoLm2xBH4TzNdX0h8dVTIVQFwWcRAWFy8PG4D580pkdgavSr9qm kgUOwxHf8rSUvYQiqQuHlyhaNAdQr4SITqWvKx21E854DDN6IPt1xhSRJ7B28KPcsfgh 2U5ejQbjW+l9CInRhHbwjVOrgF0zLaCCAYoeRFllGysl/spaGhrOydNchn4/6AbGA6th ZAMf5Cm+p4fL/5H62p5xNFEQtFnv3nPP1A27JkdJHhpGNSwJh9ek/rlkWa9/gHCTtAzO rHGtI+Ilp+DCIZAss20ruWOPKPUX/EDYZ8+k1GEmsIgwLQYoJTIchHnxuC0Qjm8gxO27 MwxA== X-Gm-Message-State: AOAM530gnIDqoYRYCG077kOelrGjC2L2KCyuwyIhDs+XbJ6LKLXdPhcn SiXFjCBEu6caWuG8fQdQj+byERrLlrU3SQ== X-Google-Smtp-Source: ABdhPJwS+mJJPnbUz3tYsnVCvyHmo96PYiuATSal7Jfaoa+DrL6qhFha/++mpmBROvDNNHbr1MYYJA== X-Received: by 2002:a17:902:ed83:b0:15c:e82a:e84f with SMTP id e3-20020a170902ed8300b0015ce82ae84fmr3661958plj.96.1651843263937; Fri, 06 May 2022 06:21:03 -0700 (PDT) Received: from always-x1.www.tendawifi.com ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id i22-20020a63e916000000b003c14af50643sm3256986pgh.91.2022.05.06.06.20.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 May 2022 06:21:03 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, pizhenwei@bytedance.com, davem@davemloft.net, kernel test robot , Dan Carpenter Subject: [PATCH v6 2/5] virtio-crypto: use private buffer for control request Date: Fri, 6 May 2022 21:16:24 +0800 Message-Id: <20220506131627.180784-3-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220506131627.180784-1-pizhenwei@bytedance.com> References: <20220506131627.180784-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Originally, all of the control requests share a single buffer( ctrl & input & ctrl_status fields in struct virtio_crypto), this allows queue depth 1 only, the performance of control queue gets limited by this design. In this patch, each request allocates request buffer dynamically, and free buffer after request, so the scope protected by ctrl_lock also get optimized here. It's possible to optimize control queue depth in the next step. A necessary comment is already in code, still describe it again: /* * Note: there are padding fields in request, clear them to zero before * sending to host to avoid to divulge any information. * Ex, virtio_crypto_ctrl_request::ctrl::u::destroy_session::padding[48] */ So use kzalloc to allocate buffer of struct virtio_crypto_ctrl_request. Potentially dereferencing uninitialized variables: Reported-by: kernel test robot Reported-by: Dan Carpenter Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Reviewed-by: Gonglei Signed-off-by: zhenwei pi --- .../virtio/virtio_crypto_akcipher_algs.c | 57 ++++++++++++------- drivers/crypto/virtio/virtio_crypto_common.h | 17 ++++-- .../virtio/virtio_crypto_skcipher_algs.c | 50 ++++++++++------ 3 files changed, 79 insertions(+), 45 deletions(-) diff --git a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c index 20901a263fc8..698ea57e2649 100644 --- a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c @@ -108,16 +108,22 @@ static int virtio_crypto_alg_akcipher_init_session(struct virtio_crypto_akcipher unsigned int num_out = 0, num_in = 0; struct virtio_crypto_op_ctrl_req *ctrl; struct virtio_crypto_session_input *input; + struct virtio_crypto_ctrl_request *vc_ctrl_req; pkey = kmemdup(key, keylen, GFP_ATOMIC); if (!pkey) return -ENOMEM; - spin_lock(&vcrypto->ctrl_lock); - ctrl = &vcrypto->ctrl; + vc_ctrl_req = kzalloc(sizeof(*vc_ctrl_req), GFP_KERNEL); + if (!vc_ctrl_req) { + err = -ENOMEM; + goto out; + } + + ctrl = &vc_ctrl_req->ctrl; memcpy(&ctrl->header, header, sizeof(ctrl->header)); memcpy(&ctrl->u, para, sizeof(ctrl->u)); - input = &vcrypto->input; + input = &vc_ctrl_req->input; input->status = cpu_to_le32(VIRTIO_CRYPTO_ERR); sg_init_one(&outhdr_sg, ctrl, sizeof(*ctrl)); @@ -129,16 +135,22 @@ static int virtio_crypto_alg_akcipher_init_session(struct virtio_crypto_akcipher sg_init_one(&inhdr_sg, input, sizeof(*input)); sgs[num_out + num_in++] = &inhdr_sg; + spin_lock(&vcrypto->ctrl_lock); err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto, GFP_ATOMIC); - if (err < 0) + if (err < 0) { + spin_unlock(&vcrypto->ctrl_lock); goto out; + } virtqueue_kick(vcrypto->ctrl_vq); while (!virtqueue_get_buf(vcrypto->ctrl_vq, &inlen) && !virtqueue_is_broken(vcrypto->ctrl_vq)) cpu_relax(); + spin_unlock(&vcrypto->ctrl_lock); if (le32_to_cpu(input->status) != VIRTIO_CRYPTO_OK) { + pr_err("virtio_crypto: Create session failed status: %u\n", + le32_to_cpu(input->status)); err = -EINVAL; goto out; } @@ -148,13 +160,9 @@ static int virtio_crypto_alg_akcipher_init_session(struct virtio_crypto_akcipher err = 0; out: - spin_unlock(&vcrypto->ctrl_lock); + kfree(vc_ctrl_req); kfree_sensitive(pkey); - if (err < 0) - pr_err("virtio_crypto: Create session failed status: %u\n", - le32_to_cpu(input->status)); - return err; } @@ -167,15 +175,18 @@ static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_akciphe int err; struct virtio_crypto_op_ctrl_req *ctrl; struct virtio_crypto_inhdr *ctrl_status; + struct virtio_crypto_ctrl_request *vc_ctrl_req; - spin_lock(&vcrypto->ctrl_lock); - if (!ctx->session_valid) { - err = 0; - goto out; - } - ctrl_status = &vcrypto->ctrl_status; + if (!ctx->session_valid) + return 0; + + vc_ctrl_req = kzalloc(sizeof(*vc_ctrl_req), GFP_KERNEL); + if (!vc_ctrl_req) + return -ENOMEM; + + ctrl_status = &vc_ctrl_req->ctrl_status; ctrl_status->status = VIRTIO_CRYPTO_ERR; - ctrl = &vcrypto->ctrl; + ctrl = &vc_ctrl_req->ctrl; ctrl->header.opcode = cpu_to_le32(VIRTIO_CRYPTO_AKCIPHER_DESTROY_SESSION); ctrl->header.queue_id = 0; @@ -188,16 +199,22 @@ static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_akciphe sg_init_one(&inhdr_sg, &ctrl_status->status, sizeof(ctrl_status->status)); sgs[num_out + num_in++] = &inhdr_sg; + spin_lock(&vcrypto->ctrl_lock); err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto, GFP_ATOMIC); - if (err < 0) + if (err < 0) { + spin_unlock(&vcrypto->ctrl_lock); goto out; + } virtqueue_kick(vcrypto->ctrl_vq); while (!virtqueue_get_buf(vcrypto->ctrl_vq, &inlen) && !virtqueue_is_broken(vcrypto->ctrl_vq)) cpu_relax(); + spin_unlock(&vcrypto->ctrl_lock); if (ctrl_status->status != VIRTIO_CRYPTO_OK) { + pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%llx\n", + ctrl_status->status, destroy_session->session_id); err = -EINVAL; goto out; } @@ -206,11 +223,7 @@ static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_akciphe ctx->session_valid = false; out: - spin_unlock(&vcrypto->ctrl_lock); - if (err < 0) { - pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%llx\n", - ctrl_status->status, destroy_session->session_id); - } + kfree(vc_ctrl_req); return err; } diff --git a/drivers/crypto/virtio/virtio_crypto_common.h b/drivers/crypto/virtio/virtio_crypto_common.h index e693d4ee83a6..2422237ec4e6 100644 --- a/drivers/crypto/virtio/virtio_crypto_common.h +++ b/drivers/crypto/virtio/virtio_crypto_common.h @@ -13,6 +13,7 @@ #include #include #include +#include /* Internal representation of a data virtqueue */ @@ -65,11 +66,6 @@ struct virtio_crypto { /* Maximum size of per request */ u64 max_size; - /* Control VQ buffers: protected by the ctrl_lock */ - struct virtio_crypto_op_ctrl_req ctrl; - struct virtio_crypto_session_input input; - struct virtio_crypto_inhdr ctrl_status; - unsigned long status; atomic_t ref_count; struct list_head list; @@ -85,6 +81,17 @@ struct virtio_crypto_sym_session_info { __u64 session_id; }; +/* + * Note: there are padding fields in request, clear them to zero before + * sending to host to avoid to divulge any information. + * Ex, virtio_crypto_ctrl_request::ctrl::u::destroy_session::padding[48] + */ +struct virtio_crypto_ctrl_request { + struct virtio_crypto_op_ctrl_req ctrl; + struct virtio_crypto_session_input input; + struct virtio_crypto_inhdr ctrl_status; +}; + struct virtio_crypto_request; typedef void (*virtio_crypto_data_callback) (struct virtio_crypto_request *vc_req, int len); diff --git a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c index e3c5bc8d6112..6aaf0869b211 100644 --- a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c @@ -126,6 +126,7 @@ static int virtio_crypto_alg_skcipher_init_session( struct virtio_crypto_op_ctrl_req *ctrl; struct virtio_crypto_session_input *input; struct virtio_crypto_sym_create_session_req *sym_create_session; + struct virtio_crypto_ctrl_request *vc_ctrl_req; /* * Avoid to do DMA from the stack, switch to using @@ -136,15 +137,20 @@ static int virtio_crypto_alg_skcipher_init_session( if (!cipher_key) return -ENOMEM; - spin_lock(&vcrypto->ctrl_lock); + vc_ctrl_req = kzalloc(sizeof(*vc_ctrl_req), GFP_KERNEL); + if (!vc_ctrl_req) { + err = -ENOMEM; + goto out; + } + /* Pad ctrl header */ - ctrl = &vcrypto->ctrl; + ctrl = &vc_ctrl_req->ctrl; ctrl->header.opcode = cpu_to_le32(VIRTIO_CRYPTO_CIPHER_CREATE_SESSION); ctrl->header.algo = cpu_to_le32(alg); /* Set the default dataqueue id to 0 */ ctrl->header.queue_id = 0; - input = &vcrypto->input; + input = &vc_ctrl_req->input; input->status = cpu_to_le32(VIRTIO_CRYPTO_ERR); /* Pad cipher's parameters */ sym_create_session = &ctrl->u.sym_create_session; @@ -164,12 +170,12 @@ static int virtio_crypto_alg_skcipher_init_session( sg_init_one(&inhdr, input, sizeof(*input)); sgs[num_out + num_in++] = &inhdr; + spin_lock(&vcrypto->ctrl_lock); err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto, GFP_ATOMIC); if (err < 0) { spin_unlock(&vcrypto->ctrl_lock); - kfree_sensitive(cipher_key); - return err; + goto out; } virtqueue_kick(vcrypto->ctrl_vq); @@ -180,13 +186,13 @@ static int virtio_crypto_alg_skcipher_init_session( while (!virtqueue_get_buf(vcrypto->ctrl_vq, &tmp) && !virtqueue_is_broken(vcrypto->ctrl_vq)) cpu_relax(); + spin_unlock(&vcrypto->ctrl_lock); if (le32_to_cpu(input->status) != VIRTIO_CRYPTO_OK) { - spin_unlock(&vcrypto->ctrl_lock); pr_err("virtio_crypto: Create session failed status: %u\n", le32_to_cpu(input->status)); - kfree_sensitive(cipher_key); - return -EINVAL; + err = -EINVAL; + goto out; } if (encrypt) @@ -194,10 +200,11 @@ static int virtio_crypto_alg_skcipher_init_session( else ctx->dec_sess_info.session_id = le64_to_cpu(input->session_id); - spin_unlock(&vcrypto->ctrl_lock); - + err = 0; +out: + kfree(vc_ctrl_req); kfree_sensitive(cipher_key); - return 0; + return err; } static int virtio_crypto_alg_skcipher_close_session( @@ -212,12 +219,16 @@ static int virtio_crypto_alg_skcipher_close_session( unsigned int num_out = 0, num_in = 0; struct virtio_crypto_op_ctrl_req *ctrl; struct virtio_crypto_inhdr *ctrl_status; + struct virtio_crypto_ctrl_request *vc_ctrl_req; - spin_lock(&vcrypto->ctrl_lock); - ctrl_status = &vcrypto->ctrl_status; + vc_ctrl_req = kzalloc(sizeof(*vc_ctrl_req), GFP_KERNEL); + if (!vc_ctrl_req) + return -ENOMEM; + + ctrl_status = &vc_ctrl_req->ctrl_status; ctrl_status->status = VIRTIO_CRYPTO_ERR; /* Pad ctrl header */ - ctrl = &vcrypto->ctrl; + ctrl = &vc_ctrl_req->ctrl; ctrl->header.opcode = cpu_to_le32(VIRTIO_CRYPTO_CIPHER_DESTROY_SESSION); /* Set the default virtqueue id to 0 */ ctrl->header.queue_id = 0; @@ -236,28 +247,31 @@ static int virtio_crypto_alg_skcipher_close_session( sg_init_one(&status_sg, &ctrl_status->status, sizeof(ctrl_status->status)); sgs[num_out + num_in++] = &status_sg; + spin_lock(&vcrypto->ctrl_lock); err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto, GFP_ATOMIC); if (err < 0) { spin_unlock(&vcrypto->ctrl_lock); - return err; + goto out; } virtqueue_kick(vcrypto->ctrl_vq); while (!virtqueue_get_buf(vcrypto->ctrl_vq, &tmp) && !virtqueue_is_broken(vcrypto->ctrl_vq)) cpu_relax(); + spin_unlock(&vcrypto->ctrl_lock); if (ctrl_status->status != VIRTIO_CRYPTO_OK) { - spin_unlock(&vcrypto->ctrl_lock); pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%llx\n", ctrl_status->status, destroy_session->session_id); return -EINVAL; } - spin_unlock(&vcrypto->ctrl_lock); - return 0; + err = 0; +out: + kfree(vc_ctrl_req); + return err; } static int virtio_crypto_alg_skcipher_init_sessions( From patchwork Fri May 6 13:16:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhenwei pi X-Patchwork-Id: 571109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8258C433EF for ; Fri, 6 May 2022 13:21:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1392427AbiEFNZM (ORCPT ); Fri, 6 May 2022 09:25:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1392380AbiEFNYx (ORCPT ); Fri, 6 May 2022 09:24:53 -0400 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 588306473E for ; Fri, 6 May 2022 06:21:09 -0700 (PDT) Received: by mail-pl1-x634.google.com with SMTP id s14so7417491plk.8 for ; Fri, 06 May 2022 06:21:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AvEGrrZmHBTew/Xb4dK3Q06lksETBCrs8O5XD+mD/UI=; b=6g3JtXGtxQKKmxovklJPc/q6dNF9+SE3MKOKxNNA8l2/ZUrR4Ac7+r6i7AthCGaQuH NpLy5HHa32eYeMtpyaHea/tDd2zqApF2OI80Sxi/haMKCgKAHGNfmcOEdrh73OI89TCz zfQOH63iCRApNbkwbrWNk4q5LKYcCl3tqekrFg5pYa0fOQVEKisx5BWpeJ6+bR0JLTTM Ybwl8Bswn0mPkMlZq2RgKrox1/2nAa1gHJ65pZvr7x7Epaw4hpiKICZ1tP1vz/Hg3p7r R5J0/J8pAXlGAxX+91ihK2OxdCU/p8jd4/6RIhdzUuf9/v45jeOsvis5pruMtgiTnMzA 6mqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AvEGrrZmHBTew/Xb4dK3Q06lksETBCrs8O5XD+mD/UI=; b=XDRTNm9wVa5fRk7dzjjPAVlydr+/mwz/2nHw10EIooE3C4EnkBH9WpzjkD98U4T+wg gW0DaMlBKftvRINySz7y4YsWztdTcYeqmj4lMBVUWpjR+zd2UijTa/cKghMP+DpINb1u LM/nrap9D+Ia0N8rUKnq4W/7ZJqAdGmOq83Vn7YpPe3PATj19glonKIyt4ZMagNkedda E7WC8p0twBYe1UQ7m1eODvM0/Gb0NoEu2DTWNN2GEjW4pUAzWCvYctAIQfTPjJRRN9xv uormFlp85WUwoJhNUS0LYen/3gYHHZS/mLNlcQzraYIGt/CFpbuaJ64CYBbQ5VO4p7YM r/Xw== X-Gm-Message-State: AOAM531eoITTKiF7BvVAAs0pltPRK1A4fVx4QrfJv8BlC2Jfu03SPsC5 1MwoWEBqJC6f9zzvWgPsW/Wlpg== X-Google-Smtp-Source: ABdhPJzFOyIcQJuc+gZF/agXu7BMN8K++eSd8bhAEO+cLSiWlxz4cjcBI+Ut1GwL+jy8Oii9UJHvTQ== X-Received: by 2002:a17:903:32d0:b0:15e:8cbc:fd39 with SMTP id i16-20020a17090332d000b0015e8cbcfd39mr3730198plr.95.1651843268856; Fri, 06 May 2022 06:21:08 -0700 (PDT) Received: from always-x1.www.tendawifi.com ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id i22-20020a63e916000000b003c14af50643sm3256986pgh.91.2022.05.06.06.21.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 May 2022 06:21:08 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, pizhenwei@bytedance.com, davem@davemloft.net Subject: [PATCH v6 3/5] virtio-crypto: wait ctrl queue instead of busy polling Date: Fri, 6 May 2022 21:16:25 +0800 Message-Id: <20220506131627.180784-4-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220506131627.180784-1-pizhenwei@bytedance.com> References: <20220506131627.180784-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Originally, after submitting request into virtio crypto control queue, the guest side polls the result from the virt queue. This works like following: CPU0 CPU1 ... CPUx CPUy | | | | \ \ / / \--------spin_lock(&vcrypto->ctrl_lock)-------/ | virtqueue add & kick | busy poll virtqueue | spin_unlock(&vcrypto->ctrl_lock) ... There are two problems: 1, The queue depth is always 1, the performance of a virtio crypto device gets limited. Multi user processes share a single control queue, and hit spin lock race from control queue. Test on Intel Platinum 8260, a single worker gets ~35K/s create/close session operations, and 8 workers get ~40K/s operations with 800% CPU utilization. 2, The control request is supposed to get handled immediately, but in the current implementation of QEMU(v6.2), the vCPU thread kicks another thread to do this work, the latency also gets unstable. Tracking latency of virtio_crypto_alg_akcipher_close_session in 5s: usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 7 | | 4 -> 7 : 72 | | 8 -> 15 : 186485 |************************| 16 -> 31 : 687 | | 32 -> 63 : 5 | | 64 -> 127 : 3 | | 128 -> 255 : 1 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 0 | | 8192 -> 16383 : 2 | | This means that a CPU may hold vcrypto->ctrl_lock as long as 8192~16383us. To improve the performance of control queue, a request on control queue waits completion instead of busy polling to reduce lock racing, and gets completed by control queue callback. CPU0 CPU1 ... CPUx CPUy | | | | \ \ / / \--------spin_lock(&vcrypto->ctrl_lock)-------/ | virtqueue add & kick | ---------spin_unlock(&vcrypto->ctrl_lock)------ / / \ \ | | | | wait wait wait wait Test this patch, the guest side get ~200K/s operations with 300% CPU utilization. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Reviewed-by: Gonglei Signed-off-by: zhenwei pi --- .../virtio/virtio_crypto_akcipher_algs.c | 29 ++--------- drivers/crypto/virtio/virtio_crypto_common.h | 4 ++ drivers/crypto/virtio/virtio_crypto_core.c | 52 ++++++++++++++++++- .../virtio/virtio_crypto_skcipher_algs.c | 34 ++---------- 4 files changed, 64 insertions(+), 55 deletions(-) diff --git a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c index 698ea57e2649..382ccec9ab12 100644 --- a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c @@ -103,7 +103,6 @@ static int virtio_crypto_alg_akcipher_init_session(struct virtio_crypto_akcipher struct scatterlist outhdr_sg, key_sg, inhdr_sg, *sgs[3]; struct virtio_crypto *vcrypto = ctx->vcrypto; uint8_t *pkey; - unsigned int inlen; int err; unsigned int num_out = 0, num_in = 0; struct virtio_crypto_op_ctrl_req *ctrl; @@ -135,18 +134,9 @@ static int virtio_crypto_alg_akcipher_init_session(struct virtio_crypto_akcipher sg_init_one(&inhdr_sg, input, sizeof(*input)); sgs[num_out + num_in++] = &inhdr_sg; - spin_lock(&vcrypto->ctrl_lock); - err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto, GFP_ATOMIC); - if (err < 0) { - spin_unlock(&vcrypto->ctrl_lock); + err = virtio_crypto_ctrl_vq_request(vcrypto, sgs, num_out, num_in, vc_ctrl_req); + if (err < 0) goto out; - } - - virtqueue_kick(vcrypto->ctrl_vq); - while (!virtqueue_get_buf(vcrypto->ctrl_vq, &inlen) && - !virtqueue_is_broken(vcrypto->ctrl_vq)) - cpu_relax(); - spin_unlock(&vcrypto->ctrl_lock); if (le32_to_cpu(input->status) != VIRTIO_CRYPTO_OK) { pr_err("virtio_crypto: Create session failed status: %u\n", @@ -171,7 +161,7 @@ static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_akciphe struct scatterlist outhdr_sg, inhdr_sg, *sgs[2]; struct virtio_crypto_destroy_session_req *destroy_session; struct virtio_crypto *vcrypto = ctx->vcrypto; - unsigned int num_out = 0, num_in = 0, inlen; + unsigned int num_out = 0, num_in = 0; int err; struct virtio_crypto_op_ctrl_req *ctrl; struct virtio_crypto_inhdr *ctrl_status; @@ -199,18 +189,9 @@ static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_akciphe sg_init_one(&inhdr_sg, &ctrl_status->status, sizeof(ctrl_status->status)); sgs[num_out + num_in++] = &inhdr_sg; - spin_lock(&vcrypto->ctrl_lock); - err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto, GFP_ATOMIC); - if (err < 0) { - spin_unlock(&vcrypto->ctrl_lock); + err = virtio_crypto_ctrl_vq_request(vcrypto, sgs, num_out, num_in, vc_ctrl_req); + if (err < 0) goto out; - } - - virtqueue_kick(vcrypto->ctrl_vq); - while (!virtqueue_get_buf(vcrypto->ctrl_vq, &inlen) && - !virtqueue_is_broken(vcrypto->ctrl_vq)) - cpu_relax(); - spin_unlock(&vcrypto->ctrl_lock); if (ctrl_status->status != VIRTIO_CRYPTO_OK) { pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%llx\n", diff --git a/drivers/crypto/virtio/virtio_crypto_common.h b/drivers/crypto/virtio/virtio_crypto_common.h index 2422237ec4e6..59a4c0259456 100644 --- a/drivers/crypto/virtio/virtio_crypto_common.h +++ b/drivers/crypto/virtio/virtio_crypto_common.h @@ -90,6 +90,7 @@ struct virtio_crypto_ctrl_request { struct virtio_crypto_op_ctrl_req ctrl; struct virtio_crypto_session_input input; struct virtio_crypto_inhdr ctrl_status; + struct completion compl; }; struct virtio_crypto_request; @@ -141,5 +142,8 @@ int virtio_crypto_skcipher_algs_register(struct virtio_crypto *vcrypto); void virtio_crypto_skcipher_algs_unregister(struct virtio_crypto *vcrypto); int virtio_crypto_akcipher_algs_register(struct virtio_crypto *vcrypto); void virtio_crypto_akcipher_algs_unregister(struct virtio_crypto *vcrypto); +int virtio_crypto_ctrl_vq_request(struct virtio_crypto *vcrypto, struct scatterlist *sgs[], + unsigned int out_sgs, unsigned int in_sgs, + struct virtio_crypto_ctrl_request *vc_ctrl_req); #endif /* _VIRTIO_CRYPTO_COMMON_H */ diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c index c6f482db0bc0..60490ffa3df1 100644 --- a/drivers/crypto/virtio/virtio_crypto_core.c +++ b/drivers/crypto/virtio/virtio_crypto_core.c @@ -22,6 +22,56 @@ virtcrypto_clear_request(struct virtio_crypto_request *vc_req) } } +static void virtio_crypto_ctrlq_callback(struct virtio_crypto_ctrl_request *vc_ctrl_req) +{ + complete(&vc_ctrl_req->compl); +} + +static void virtcrypto_ctrlq_callback(struct virtqueue *vq) +{ + struct virtio_crypto *vcrypto = vq->vdev->priv; + struct virtio_crypto_ctrl_request *vc_ctrl_req; + unsigned long flags; + unsigned int len; + + spin_lock_irqsave(&vcrypto->ctrl_lock, flags); + do { + virtqueue_disable_cb(vq); + while ((vc_ctrl_req = virtqueue_get_buf(vq, &len)) != NULL) { + spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); + virtio_crypto_ctrlq_callback(vc_ctrl_req); + spin_lock_irqsave(&vcrypto->ctrl_lock, flags); + } + if (unlikely(virtqueue_is_broken(vq))) + break; + } while (!virtqueue_enable_cb(vq)); + spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); +} + +int virtio_crypto_ctrl_vq_request(struct virtio_crypto *vcrypto, struct scatterlist *sgs[], + unsigned int out_sgs, unsigned int in_sgs, + struct virtio_crypto_ctrl_request *vc_ctrl_req) +{ + int err; + unsigned long flags; + + init_completion(&vc_ctrl_req->compl); + + spin_lock_irqsave(&vcrypto->ctrl_lock, flags); + err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, out_sgs, in_sgs, vc_ctrl_req, GFP_ATOMIC); + if (err < 0) { + spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); + return err; + } + + virtqueue_kick(vcrypto->ctrl_vq); + spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); + + wait_for_completion(&vc_ctrl_req->compl); + + return 0; +} + static void virtcrypto_dataq_callback(struct virtqueue *vq) { struct virtio_crypto *vcrypto = vq->vdev->priv; @@ -73,7 +123,7 @@ static int virtcrypto_find_vqs(struct virtio_crypto *vi) goto err_names; /* Parameters for control virtqueue */ - callbacks[total_vqs - 1] = NULL; + callbacks[total_vqs - 1] = virtcrypto_ctrlq_callback; names[total_vqs - 1] = "controlq"; /* Allocate/initialize parameters for data virtqueues */ diff --git a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c index 6aaf0869b211..e553ccadbcbc 100644 --- a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c @@ -118,7 +118,6 @@ static int virtio_crypto_alg_skcipher_init_session( int encrypt) { struct scatterlist outhdr, key_sg, inhdr, *sgs[3]; - unsigned int tmp; struct virtio_crypto *vcrypto = ctx->vcrypto; int op = encrypt ? VIRTIO_CRYPTO_OP_ENCRYPT : VIRTIO_CRYPTO_OP_DECRYPT; int err; @@ -170,23 +169,9 @@ static int virtio_crypto_alg_skcipher_init_session( sg_init_one(&inhdr, input, sizeof(*input)); sgs[num_out + num_in++] = &inhdr; - spin_lock(&vcrypto->ctrl_lock); - err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, - num_in, vcrypto, GFP_ATOMIC); - if (err < 0) { - spin_unlock(&vcrypto->ctrl_lock); + err = virtio_crypto_ctrl_vq_request(vcrypto, sgs, num_out, num_in, vc_ctrl_req); + if (err < 0) goto out; - } - virtqueue_kick(vcrypto->ctrl_vq); - - /* - * Trapping into the hypervisor, so the request should be - * handled immediately. - */ - while (!virtqueue_get_buf(vcrypto->ctrl_vq, &tmp) && - !virtqueue_is_broken(vcrypto->ctrl_vq)) - cpu_relax(); - spin_unlock(&vcrypto->ctrl_lock); if (le32_to_cpu(input->status) != VIRTIO_CRYPTO_OK) { pr_err("virtio_crypto: Create session failed status: %u\n", @@ -212,7 +197,6 @@ static int virtio_crypto_alg_skcipher_close_session( int encrypt) { struct scatterlist outhdr, status_sg, *sgs[2]; - unsigned int tmp; struct virtio_crypto_destroy_session_req *destroy_session; struct virtio_crypto *vcrypto = ctx->vcrypto; int err; @@ -247,19 +231,9 @@ static int virtio_crypto_alg_skcipher_close_session( sg_init_one(&status_sg, &ctrl_status->status, sizeof(ctrl_status->status)); sgs[num_out + num_in++] = &status_sg; - spin_lock(&vcrypto->ctrl_lock); - err = virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, - num_in, vcrypto, GFP_ATOMIC); - if (err < 0) { - spin_unlock(&vcrypto->ctrl_lock); + err = virtio_crypto_ctrl_vq_request(vcrypto, sgs, num_out, num_in, vc_ctrl_req); + if (err < 0) goto out; - } - virtqueue_kick(vcrypto->ctrl_vq); - - while (!virtqueue_get_buf(vcrypto->ctrl_vq, &tmp) && - !virtqueue_is_broken(vcrypto->ctrl_vq)) - cpu_relax(); - spin_unlock(&vcrypto->ctrl_lock); if (ctrl_status->status != VIRTIO_CRYPTO_OK) { pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%llx\n", From patchwork Fri May 6 13:16:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhenwei pi X-Patchwork-Id: 570265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2800C433FE for ; Fri, 6 May 2022 13:21:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1388427AbiEFNZO (ORCPT ); Fri, 6 May 2022 09:25:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1392437AbiEFNZM (ORCPT ); Fri, 6 May 2022 09:25:12 -0400 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13B1269722 for ; Fri, 6 May 2022 06:21:13 -0700 (PDT) Received: by mail-pj1-x1031.google.com with SMTP id gj17-20020a17090b109100b001d8b390f77bso10831671pjb.1 for ; Fri, 06 May 2022 06:21:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5edDtm3ClDj2Ih4shfC7v+UW75qeVSKJFNdIatinSe4=; b=Uk0uXOW5WRqRK2VTpgFyRJQIfmCllW+gzKZ2xs8e38Byyi/sOBdnh8zgRA2lcnrGCp MbpJ6070kITrxcF0rmmRClG32eCpQhdvZnLOw7NLyTPxN17z9kfcUDGoISZ+d6HUt5GC GVJC143IRGMeIeiMMdCszdGtCr5lKVFZAUq5kJjXPyWZJcE/9kdtn0fobMio59+w9sa2 V/jA6HOPPwfzAulbD2NmBLMojuItt8BegWaY3H53J4ZqG4FkyZNtgmc4blc4AX3OYc9I JFHSNhwzkd2cerBOdCNcSJ/O1sx3tYQCe2VhoYJcoIR2cwtxRpscHgemetzFMfVdAioF n3kQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5edDtm3ClDj2Ih4shfC7v+UW75qeVSKJFNdIatinSe4=; b=oo0wlrOb98oMz1QRJ2Mf1GXLKtu7KEl/k4klGyvqThl8SAZcm5jyrMYOGp1XsrHSRc aYBGc2iAr7a3+Gl5Qf5YIOb2BN1wUJXVnYMZKP2TqwGCB4eL592XPI57FbcVo7LgEOhG U5X0peyu52TT6qC6yf7joTcX04Nlm+jV/N5XTxlOnK2GvUWiTgSsf/YLHuqTH0lACEdA gVTfpyHL1sYYjY4n/N2vlw+GXcDaFOMU1YuBe/CawNoC3cvCtH3Sh++qVmT4TaO06Cnt XD0Q/i7ivjRrdfUJXrjwcLPTAds+BJSlyuppQATa0w/UgywanK8xbtQXJIvWUWerplk7 c4jQ== X-Gm-Message-State: AOAM530qquGEYILvJEJec4dAI+OodUPHQvN7SvXJ646fU3q/MkmbDlQm P4XoRBxIs4OBXzHl/Dqik0P8CQ== X-Google-Smtp-Source: ABdhPJwmOLNxQkSjP7G9cCAfAFBzJustpWCljWo7emBIKpuY9vdl/bdTDNTkpES1Me8znMuFza7RlQ== X-Received: by 2002:a17:90b:1c04:b0:1dc:4dfd:5a43 with SMTP id oc4-20020a17090b1c0400b001dc4dfd5a43mr12535457pjb.160.1651843273430; Fri, 06 May 2022 06:21:13 -0700 (PDT) Received: from always-x1.www.tendawifi.com ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id i22-20020a63e916000000b003c14af50643sm3256986pgh.91.2022.05.06.06.21.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 May 2022 06:21:12 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, pizhenwei@bytedance.com, davem@davemloft.net Subject: [PATCH v6 4/5] virtio-crypto: adjust dst_len at ops callback Date: Fri, 6 May 2022 21:16:26 +0800 Message-Id: <20220506131627.180784-5-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220506131627.180784-1-pizhenwei@bytedance.com> References: <20220506131627.180784-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: lei he For some akcipher operations(eg, decryption of pkcs1pad(rsa)), the length of returned result maybe less than akcipher_req->dst_len, we need to recalculate the actual dst_len through the virt-queue protocol. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Reviewed-by: Gonglei Signed-off-by: lei he Signed-off-by: zhenwei pi --- drivers/crypto/virtio/virtio_crypto_akcipher_algs.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c index 382ccec9ab12..2a60d0525cde 100644 --- a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c @@ -90,9 +90,12 @@ static void virtio_crypto_dataq_akcipher_callback(struct virtio_crypto_request * } akcipher_req = vc_akcipher_req->akcipher_req; - if (vc_akcipher_req->opcode != VIRTIO_CRYPTO_AKCIPHER_VERIFY) + if (vc_akcipher_req->opcode != VIRTIO_CRYPTO_AKCIPHER_VERIFY) { + /* actuall length maybe less than dst buffer */ + akcipher_req->dst_len = len - sizeof(vc_req->status); sg_copy_from_buffer(akcipher_req->dst, sg_nents(akcipher_req->dst), vc_akcipher_req->dst_buf, akcipher_req->dst_len); + } virtio_crypto_akcipher_finalize_req(vc_akcipher_req, akcipher_req, error); } From patchwork Fri May 6 13:16:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhenwei pi X-Patchwork-Id: 570264 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30950C433EF for ; Fri, 6 May 2022 13:21:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1392480AbiEFNZc (ORCPT ); Fri, 6 May 2022 09:25:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1392451AbiEFNZN (ORCPT ); Fri, 6 May 2022 09:25:13 -0400 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4D566973F for ; Fri, 6 May 2022 06:21:18 -0700 (PDT) Received: by mail-pj1-x1033.google.com with SMTP id a15-20020a17090ad80f00b001dc2e23ad84so10798363pjv.4 for ; Fri, 06 May 2022 06:21:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=snAq8vOceBJQXNtHaQrKZKIzrWkN2Ytbsg/b/TMNKmk=; b=RcbJboymTfiVvIkRHCgMojUfl9bLvh+A2nG+XBtgmx0u4q3MuN7jucM7/yuVKMzlwh UnMU2h/tV8bvG7ApiCvCmir6sodCeLyONGmHHscGkU0RofG2yjBLERzmHarv3zKNdn3c W1qDNbeb8vuDdLXbyGwpLxh4as5TGCG+DtP5oGtMuiJwcz9+Ocnj6sZ9xOBb211wgbSW lsRhEZHd8CZ93wsB7gNlHbWHuwdNKtEIzUYtmdB/cxqBRweoDwfSlHs4aj/ZF7ZTmad+ TDCC2YheRyFnYFVnvg1YVHCxdu5HESG/g1d4BiK1Ixu56OjuAOLXVlpoBk9MzY19DPQI qzKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=snAq8vOceBJQXNtHaQrKZKIzrWkN2Ytbsg/b/TMNKmk=; b=0dECydkt6sIh7H5eXTfVkUXHtYrH8wblMzSsjqjZ40Cwqz1tjPvH5T80UQDy3D6S1o NyUPXqlHiRqhLFQLoEXTr0I6TP7br2PUscp/FKfgdAtDwH0FWVdrSuXGyyon2xu3OnyP PWwPEdLYP6kQD6rFF64p16NTw3PU0GgokPRYdYhcJFcUX73ybWPi/SEDP1yXQVbtuunw H6PA2/X7G8Bt5ZFznX3bZEFz4NB7JGbGiOHksXL+e7wiedzCcN9sv1SDznQEytvaZwTw XR+yecjwqEDT5Q/50JYdOxlMwrcZqVGbbhhyvy2gXV5qcqIMhds/OdEyS9OLw7y9eSjq rBXg== X-Gm-Message-State: AOAM530bDO3VSABVUBJ4oHvIg4gGouc8AF2i0c7Zy5JMnXyi7kuVG4jP Xm24t2gpMVeSSXq+GYDuOgYwgPM0FTOyJA== X-Google-Smtp-Source: ABdhPJyFm1OdK877ATfQ8sTTUFDzZf6kwJX19vO8QKfl9vfzuJo5UneDBXKKQgQprvUGJ1o25mQvVQ== X-Received: by 2002:a17:902:8605:b0:15d:10dc:1c6f with SMTP id f5-20020a170902860500b0015d10dc1c6fmr3781372plo.4.1651843277875; Fri, 06 May 2022 06:21:17 -0700 (PDT) Received: from always-x1.www.tendawifi.com ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id i22-20020a63e916000000b003c14af50643sm3256986pgh.91.2022.05.06.06.21.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 May 2022 06:21:17 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, pizhenwei@bytedance.com, davem@davemloft.net Subject: [PATCH v6 5/5] virtio-crypto: enable retry for virtio-crypto-dev Date: Fri, 6 May 2022 21:16:27 +0800 Message-Id: <20220506131627.180784-6-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220506131627.180784-1-pizhenwei@bytedance.com> References: <20220506131627.180784-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: lei he Enable retry for virtio-crypto-dev, so that crypto-engine can process cipher-requests parallelly. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Reviewed-by: Gonglei Signed-off-by: lei he Signed-off-by: zhenwei pi --- drivers/crypto/virtio/virtio_crypto_core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/virtio/virtio_crypto_core.c index 60490ffa3df1..1198bd306365 100644 --- a/drivers/crypto/virtio/virtio_crypto_core.c +++ b/drivers/crypto/virtio/virtio_crypto_core.c @@ -144,7 +144,8 @@ static int virtcrypto_find_vqs(struct virtio_crypto *vi) spin_lock_init(&vi->data_vq[i].lock); vi->data_vq[i].vq = vqs[i]; /* Initialize crypto engine */ - vi->data_vq[i].engine = crypto_engine_alloc_init(dev, 1); + vi->data_vq[i].engine = crypto_engine_alloc_init_and_set(dev, true, NULL, true, + virtqueue_get_vring_size(vqs[i])); if (!vi->data_vq[i].engine) { ret = -ENOMEM; goto err_engine;