From patchwork Tue Mar 28 15:35:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 667991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75FB0C761AF for ; Tue, 28 Mar 2023 15:35:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233400AbjC1Pf4 (ORCPT ); Tue, 28 Mar 2023 11:35:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233215AbjC1Pfu (ORCPT ); Tue, 28 Mar 2023 11:35:50 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA3A9C3; Tue, 28 Mar 2023 08:35:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680017747; x=1711553747; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kO38bfPMogxI1tGFhan6UK8YLGWBnl59qm86GfdzocM=; b=RVMxGRtDoL90VBS8feoVxBgewC9oFoqiHVZV6YzQxzIJFXup5PDavxfV fUEnvimvq5nXNdopIDlnjqva+JNCJspbOtVQus35n9QjgURIGdvTtI2I4 4zDSRwodUNe6JeCuFR7BZffE4/pdWIYV5RvowoTshuhotZN6lCfTprJMz VFsHm0gn+N8Kiz0lFMNkBG4w9hIMj2Ing+NuzNgwqMjzK5c7qVPeepuKk PCE9/y7MBeEORb1EZ1FbzHn4FNhs2M2AhiS65VNG4yBoTkahZ9LrHlXU9 rHKLD9/cAk/7FVt5C7c/npqxDoqOQb5RWGqsUWPpUDiBL0pDjmANlDbj+ Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="342192034" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="342192034" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:35:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="683948280" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="683948280" Received: from sdwarak1-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.intel.com) ([10.212.127.200]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:35:44 -0700 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, giovanni.cabiddu@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v2 02/15] dmaengine: idxd: add external module driver support for dsa_bus_type Date: Tue, 28 Mar 2023 10:35:22 -0500 Message-Id: <20230328153535.126223-3-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230328153535.126223-1-tom.zanussi@linux.intel.com> References: <20230328153535.126223-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Dave Jiang Add support to allow an external driver to be registered to the dsa_bus_type and also auto-loaded. Signed-off-by: Dave Jiang Signed-off-by: Tom Zanussi --- drivers/dma/idxd/bus.c | 6 ++++++ drivers/dma/idxd/idxd.h | 3 +++ 2 files changed, 9 insertions(+) diff --git a/drivers/dma/idxd/bus.c b/drivers/dma/idxd/bus.c index 6f84621053c6..0c9e689a2e77 100644 --- a/drivers/dma/idxd/bus.c +++ b/drivers/dma/idxd/bus.c @@ -67,11 +67,17 @@ static void idxd_config_bus_remove(struct device *dev) idxd_drv->remove(idxd_dev); } +static int idxd_bus_uevent(const struct device *dev, struct kobj_uevent_env *env) +{ + return add_uevent_var(env, "MODALIAS=" IDXD_DEVICES_MODALIAS_FMT, 0); +} + struct bus_type dsa_bus_type = { .name = "dsa", .match = idxd_config_bus_match, .probe = idxd_config_bus_probe, .remove = idxd_config_bus_remove, + .uevent = idxd_bus_uevent, }; EXPORT_SYMBOL_GPL(dsa_bus_type); diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h index 505118fc19de..719f9f1662ad 100644 --- a/drivers/dma/idxd/idxd.h +++ b/drivers/dma/idxd/idxd.h @@ -587,6 +587,9 @@ static inline int idxd_wq_driver_name_match(struct idxd_wq *wq, struct device *d return (strncmp(wq->driver_name, dev->driver->name, strlen(dev->driver->name)) == 0); } +#define MODULE_ALIAS_IDXD_DEVICE(type) MODULE_ALIAS("idxd:t" __stringify(type) "*") +#define IDXD_DEVICES_MODALIAS_FMT "idxd:t%d" + int __must_check __idxd_driver_register(struct idxd_device_driver *idxd_drv, struct module *module, const char *mod_name); #define idxd_driver_register(driver) \ From patchwork Tue Mar 28 15:35:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 667990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB61DC6FD18 for ; Tue, 28 Mar 2023 15:36:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233344AbjC1PgA (ORCPT ); Tue, 28 Mar 2023 11:36:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233465AbjC1Pf4 (ORCPT ); Tue, 28 Mar 2023 11:35:56 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B860AF37; Tue, 28 Mar 2023 08:35:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680017751; x=1711553751; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7/ydwUXLdHl1bpDjApixDXbPYE6quD+r8XqadnXkUFI=; b=bDkjerHrxMihrwqGzS73ByYgn7ccTUlGQ0pvX+cZn29PQq7uHPPN/TIo s6+1mUKXSpUCG7ro3SyIHCc9xBOlgMTxlQmAPk0STIjxfODlwLZrUSJOU UqtkXjKWxWldPLxCQJFzqk/zbgGimTL93EDy0/X4GtqVZAv+2gKFe4V5i JjVv1HI4qI14zKRsVnWcDfSQ938ABxrzFusXHVzvBYnotvEatbSN1CdEO UrMrJ6z/gBh0YeroioWr2VUr6blffDqreda73Kq6eh55jxMnLvUOpXhMj GPYCl+Cv+L7zgl/+bo/wTGpXeOOsxiahLwh9BN3jHaWUV6/k9jJzCjEVr g==; X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="342192062" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="342192062" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:35:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="683948318" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="683948318" Received: from sdwarak1-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.intel.com) ([10.212.127.200]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:35:49 -0700 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, giovanni.cabiddu@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v2 04/15] dmaengine: idxd: Export descriptor management functions Date: Tue, 28 Mar 2023 10:35:24 -0500 Message-Id: <20230328153535.126223-5-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230328153535.126223-1-tom.zanussi@linux.intel.com> References: <20230328153535.126223-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org To allow idxd sub-drivers to access the descriptor management functions, export them. Signed-off-by: Tom Zanussi --- drivers/dma/idxd/submit.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/dma/idxd/submit.c b/drivers/dma/idxd/submit.c index c01db23e3333..9d9ec0b76ccd 100644 --- a/drivers/dma/idxd/submit.c +++ b/drivers/dma/idxd/submit.c @@ -61,6 +61,7 @@ struct idxd_desc *idxd_alloc_desc(struct idxd_wq *wq, enum idxd_op_type optype) return __get_desc(wq, idx, cpu); } +EXPORT_SYMBOL_NS_GPL(idxd_alloc_desc, IDXD); void idxd_free_desc(struct idxd_wq *wq, struct idxd_desc *desc) { @@ -69,6 +70,7 @@ void idxd_free_desc(struct idxd_wq *wq, struct idxd_desc *desc) desc->cpu = -1; sbitmap_queue_clear(&wq->sbq, desc->id, cpu); } +EXPORT_SYMBOL_NS_GPL(idxd_free_desc, IDXD); static struct idxd_desc *list_abort_desc(struct idxd_wq *wq, struct idxd_irq_entry *ie, struct idxd_desc *desc) @@ -215,3 +217,4 @@ int idxd_submit_desc(struct idxd_wq *wq, struct idxd_desc *desc) percpu_ref_put(&wq->wq_active); return 0; } +EXPORT_SYMBOL_GPL(idxd_submit_desc); From patchwork Tue Mar 28 15:35:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 667989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06D5AC6FD18 for ; Tue, 28 Mar 2023 15:36:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234102AbjC1Pgi (ORCPT ); Tue, 28 Mar 2023 11:36:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234051AbjC1PgS (ORCPT ); Tue, 28 Mar 2023 11:36:18 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C926BF76A; Tue, 28 Mar 2023 08:36:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680017761; x=1711553761; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=T24PujgQYZHqfnkCbqb1TyiF+r+EUpp8fid20iaYFLk=; b=DBxUY6EBc41zeEjSH0SFJnyOaloIbPZrzKFXIGJadV2LSj1/tTPxZpO3 Mm0ebYNX8sxskIlSIXkip5+VJC7fm9Q1mXLVAGkftu0wdhm9yQkrTh5Pl wWgAslNbt17HrT/Bu4bWIAlnO1AFJMPB3NQnEKFDbsE9iTySAfKRAjFTz 3dS84pJe+/1gBc5dkeCnsEp4nUy8N4HV64Bsd2Bf2JqjQuhfJopp7paXx zej0uNiSWqWRKA94RT21O1+u4npDKr3aPsfbVwwmOJK6yripbY2AZ5GlL hvLgLgPDa5MWva9f3bNCzDvnA27ZymAAUdWqR6pmy1PAAtm+UkpwYMUOY A==; X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="342192144" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="342192144" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:35:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="683948364" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="683948364" Received: from sdwarak1-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.intel.com) ([10.212.127.200]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:35:54 -0700 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, giovanni.cabiddu@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v2 06/15] dmaengine: idxd: Add private_data to struct idxd_wq Date: Tue, 28 Mar 2023 10:35:26 -0500 Message-Id: <20230328153535.126223-7-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230328153535.126223-1-tom.zanussi@linux.intel.com> References: <20230328153535.126223-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add a void * to idxd_wqs for user-defined context data. Signed-off-by: Tom Zanussi --- drivers/dma/idxd/idxd.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h index 719f9f1662ad..e55213b5153e 100644 --- a/drivers/dma/idxd/idxd.h +++ b/drivers/dma/idxd/idxd.h @@ -216,6 +216,8 @@ struct idxd_wq { u32 max_batch_size; char driver_name[WQ_NAME_SIZE + 1]; + + void *private_data; }; struct idxd_engine { From patchwork Tue Mar 28 15:35:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 667988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23D6EC76196 for ; Tue, 28 Mar 2023 15:36:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234073AbjC1Pg5 (ORCPT ); Tue, 28 Mar 2023 11:36:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234075AbjC1Pgc (ORCPT ); Tue, 28 Mar 2023 11:36:32 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7292AFF02; Tue, 28 Mar 2023 08:36:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680017764; x=1711553764; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tSQWE8qApYPZYzmXqNYhdO2OwDP1sElKiL1XcGoadoo=; b=VCdNoomviF8ArggBq/Y+QxcaZcKSzinIKKK75xLa2+VSJdgmy7X04ttf KVMuQP9Ec4fGufEtZsNXYi2ez/RVjFET98JP36UfTVYYRvs1eWIFd+tY7 wFsewaFBF+cqbhqdqKwQjtO1ChLyE++n4l0g2VlFNGzOKb58cLcLj+31Y NnQE9zzUF7aoJ3n+N2EVsVwb+u/+xi/4tnKle0q0b80MQNcS1AhCNeCpH Qwy2GNEnuOvs1H+L96CDpSNmtPYURbAcscMVqz0/ax3Rv94UMqaO/lWEY ZxXR6E7MSOyeUT+DU16RdVeD1wxpmekSGI/2YMriYaLXAfB/59M1az06C g==; X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="342192153" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="342192153" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:36:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="683948419" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="683948419" Received: from sdwarak1-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.intel.com) ([10.212.127.200]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:35:59 -0700 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, giovanni.cabiddu@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v2 08/15] crypto: iaa - Add IAA Compression Accelerator Documentation Date: Tue, 28 Mar 2023 10:35:28 -0500 Message-Id: <20230328153535.126223-9-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230328153535.126223-1-tom.zanussi@linux.intel.com> References: <20230328153535.126223-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Because the IAA Compression Accelerator requires significant user setup in order to be used properly, this adds documentation on the iaa_crypto driver including setup, usage, and examples. Signed-off-by: Tom Zanussi --- .../driver-api/crypto/iaa/iaa-crypto.rst | 581 ++++++++++++++++++ Documentation/driver-api/crypto/iaa/index.rst | 20 + Documentation/driver-api/crypto/index.rst | 20 + Documentation/driver-api/index.rst | 1 + 4 files changed, 622 insertions(+) create mode 100644 Documentation/driver-api/crypto/iaa/iaa-crypto.rst create mode 100644 Documentation/driver-api/crypto/iaa/index.rst create mode 100644 Documentation/driver-api/crypto/index.rst diff --git a/Documentation/driver-api/crypto/iaa/iaa-crypto.rst b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst new file mode 100644 index 000000000000..442068778fb7 --- /dev/null +++ b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst @@ -0,0 +1,581 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================================= +IAA Compression Accelerator Crypto Driver +========================================= + +Tom Zanussi + +The IAA crypto driver supports compression/decompression compatible +with the DEFLATE compression standard described in RFC 1951, which is +the compression/decompression algorithm exported by this module. + +The IAA hardware spec can be found here: + + https://cdrdv2.intel.com/v1/dl/getContent/721858 + +The iaa_crypto driver is designed to work as a layer underneath +higher-level compression devices such as zswap. + +Users can select IAA compress/decompress acceleration by specifying +'iaa_crypto' as the compression algorithm to use by whatever facility +allows compression algorithms to be selected. + +For example, a zswap device can select iaa_crypto via:: + + # echo iaa_crypto > /sys/module/zswap/parameters/compressor + + +Config options and other setup +============================== + +The IAA crypto driver is available via menuconfig using the following +path:: + + Cryptographic API -> Hardware crypto devices -> Support for Intel(R) IAA Compression Accelerator + +In the configuration file the option called CONFIG_CRYPTO_DEV_IAA_CRYPTO. + +The IAA crypto driver also supports statistics, which are available +via menuconfig using the following path:: + + Cryptographic API -> Hardware crypto devices -> Support for Intel(R) IAA Compression -> Enable Intel(R) IAA Compression Accelerator Statistics + +In the configuration file the option called CONFIG_CRYPTO_DEV_IAA_CRYPTO_STATS. + +The following config options should also be enabled:: + + CONFIG_IRQ_REMAP=y + CONFIG_INTEL_IOMMU=y + CONFIG_INTEL_IOMMU_SVM=y + CONFIG_PCI_ATS=y + CONFIG_PCI_PRI=y + CONFIG_PCI_PASID=y + CONFIG_INTEL_IDXD=m + CONFIG_INTEL_IDXD_SVM=y + +IAA is one of the first Intel accelerator IPs that can work in +conjunction with the Intel IOMMU. There are multiple modes that exist +for testing. Based on IOMMU configuration, there are 3 modes:: + + - Scalable + - Legacy + - No IOMMU + + +Scalable mode +------------- + +Scalable mode supports Shared Virtual Memory (SVM or SVA). It is +entered when using the kernel boot commandline:: + + intel_iommu=on,sm_on + +with VT-d turned on in BIOS. + +With scalable mode, both shared and dedicated workqueues are available +for use. + +For scalable mode, the following BIOS settings should be enabled:: + + Socket Configuration > IIO Configuration > Intel VT for Directed I/O (VT-d) > Intel VT for Directed I/O + + Socket Configuration > IIO Configuration > PCIe ENQCMD > ENQCMDS + + +Legacy mode +----------- + +Legacy mode is entered when using the kernel boot commandline:: + + intel_iommu=off, intel_iommu=on + +or VT-d is not turned on in BIOS. + +If you have booted into Linux and not sure if VT-d is on, do a "dmesg +| grep -i dmar". If you don't see a number of DMAR devices enumerated, +most likely VT-d is not on. + +With legacy mode, only dedicated workqueues are available for use. + + +No IOMMU mode +------------- + +No IOMMU mode is entered when using the kernel boot commandline:: + + iommu=off. + +With no IOMMU mode, only dedicated workqueues are available for use. + + +Usage +===== + +accel-config +------------ + +Unlike typical drivers, the iaa_crypto driver does not enable the +device on driver load. Due to complexity and configurability of the +accelerator devices, it was a design decision to have the user +configure the device and manually enable the desired devices and +workqueues. + +The userspace tool to help doing that is called accel-config. Using +accel-config to configure device or loading a previously saved config +is highly recommended. The device can be controlled via sysfs +directly but comes with the warning that do this ONLY if you know +exactly what you are doing. This document will not cover the sysfs +interface but assumes you will be using accel-config. + +The accel-config tool along with instructions for building it can be +found here: + + https://github.com/intel/idxd-config/#readme + + +Typical usage +------------- + +In order for the iaa_crypto module to actually do any +compression/decompression work on behalf of a facility, one or more +IAA workqueues need to be bound to the iaa_crypto driver. + +For instance, here's an example of configuring an IAA workqueue and +binding it to the iaa_crypto driver (note that device names are +specified as 'iax' rather than 'iaa' - this is because upstream still +has the old 'iax' device naming in place) :: + + # configure wq1.0 + + accel-config config-wq --group-id=0 --mode=dedicated --type=kernel --name="iaa_crypto" --device_name="crypto" iax1/wq1.0 + + # enable IAA device iax1 + + accel-config enable-device iax1 + + # enable wq1.0 on IAX device iax1 + + accel-config enable-wq iax1/wq1.0 + +Whenever a new workqueue is bound to or unbound from the iaa_crypto +driver, the available workqueues are 'rebalanced' such that work +submitted from a particular CPU is given to the most appropriate +workqueue available. Current best practice is to configure and bind +at least one workqueue for each IAA device, but as long as there is at +least one workqueue configured and bound to any IAA device in the +system, the iaa_crypto driver will work, albeit most likely not as +efficiently. + +The IAA crypto algorigthms is operational and compression and +decompression operations are fully enabled following the successful +binding of the first IAA workqueue to the iaa_crypto driver, and +similarly, the IAA crypto algorithm is automatically unregistered when +there are no IAA workqueues bound to the driver, following their +removal. + +As a result, the iaa_crypto crypto algorithm and thus the IAA hardware +are only available when one or more workques are bound to the +iaa_crypto driver. + +Driver attributes +----------------- + +There are a few user-configurable driver attributes that can be used +to configure various modes of operation. They're listed below, along +with their default values. To set any of these attributes, echo the +appropriate values to the attribute file located under +/sys/bus/dsa/drivers/crypto/ + + - verify_compress + + Toggle compression verification. If set, each compress will be + internally decompressed and the contents verified, returning error + codes if unsuccessful. This can be toggled with 0/1: + + echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress + + The default setting is '1' - verify all compresses. + + - compression_mode + + Select compression mode to be used by all compresses and + decompresses. There are a number ofcompression modes available, + each identified by a unique string. These can be selected by + echoing the string to the 'compression_mode' driver attribute. + Currently, there are only two compression modes available, + 'canned' and 'fixed' modes. + + The 'fixed' compression mode implements the compression scheme + specified by RFC 1951. + + The 'canned' compression mode implements a good general-purpose + compression scheme whose tables were generated from statistics + derived from a wide variety of SPEC17 workloads. It provides much + better overall characteristics than the existing deflate-1951 + tables implemented by 'fixed'. + + Either 'fixed' or 'canned' modes can be chosen as the mode to be used + for compression/decompression via the iaa_crypto compression_mode + iaa_crypto driver attribute: + + echo "canned" > /sys/bus/dsa/drivers/crypto/compression_mode + + The default setting is 'fixed'. + + - sync_mode + + Select mode to be used to wait for completion of each compresses + and decompress operation. + + The crypto async interface support implemented by iaa_crypto + provides an implementation that satisfies the interface but does + so in a synchronous manner - it fills and submits the IDXD + descriptor and then loops around waiting for it to complete before + returning. This isn't a problem at the moment, since all existing + callers (e.g. zswap) wrap any asynchronous callees in a + synchronous wrapper anyway. + + The iaa_crypto driver does however provide true asynchronous + support for callers that can make use of it. In this mode, it + fills and submits the IDXD descriptor, then returns immediately + with -EINPROGRESS. The caller can then either poll for completion + itself, which requires specific code in the caller which currently + nothing in the upstream kernel implements, or go to sleep and wait + for an interrupt signaling completion. This latter mode is + supported by current users in the kernel such as zswap via + synchronous wrappers. Although it is supported this mode is + significantly slower than the synchronous mode that does the + polling in the iaa_crypto driver previously mentioned. + + This mode can be enabled by writing 'async_irq' to the sync_mode + iaa_crypto driver attribute: + + echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode + + Async mode without interrupts (caller must poll) can be enabled by + writing 'async' to it: + + echo async > /sys/bus/dsa/drivers/crypto/sync_mode + + The mode that does the polling in the iaa_crypto driver can be + enabled by writing 'sync' to it: + + echo sync > /sys/bus/dsa/drivers/crypto/sync_mode + + The default mode is 'sync'. + + +Statistics +========== + +If the optional debugfs statistics support is enabled, the IAA crypto +driver will generate statistics which can be accessed in debugfs at:: + + # ls -al /sys/kernel/debug/iaa-crypto/ + total 0 + drwxr-xr-x 2 root root 0 Mar 3 09:35 . + drwx------ 47 root root 0 Mar 3 09:35 .. + -rw-r--r-- 1 root root 0 Mar 3 09:35 max_acomp_delay_ns + -rw-r--r-- 1 root root 0 Mar 3 09:35 max_adecomp_delay_ns + -rw-r--r-- 1 root root 0 Mar 3 09:35 max_comp_delay_ns + -rw-r--r-- 1 root root 0 Mar 3 09:35 max_decomp_delay_ns + -rw-r--r-- 1 root root 0 Mar 3 09:35 stats_reset + -rw-r--r-- 1 root root 0 Mar 3 09:35 total_comp_bytes_out + -rw-r--r-- 1 root root 0 Mar 3 09:35 total_comp_calls + -rw-r--r-- 1 root root 0 Mar 3 09:35 total_decomp_bytes_in + -rw-r--r-- 1 root root 0 Mar 3 09:35 total_decomp_calls + -rw-r--r-- 1 root root 0 Mar 3 09:35 wq_stats + +Most of the above statisticss are self-explanatory. The wq_stats file +shows per-wq stats, a set for each iaa device and wq in addition to +some global stats:: + + # cat wq_stats + global stats: + total_comp_calls: 100 + total_decomp_calls: 100 + total_comp_bytes_out: 22800 + total_decomp_bytes_in: 22800 + total_completion_einval_errors: 0 + total_completion_timeout_errors: 0 + total_completion_comp_buf_overflow_errors: 0 + + iaa device: + id: 1 + n_wqs: 1 + comp_calls: 0 + comp_bytes: 0 + decomp_calls: 0 + decomp_bytes: 0 + wqs: + name: iaa_crypto + comp_calls: 0 + comp_bytes: 0 + decomp_calls: 0 + decomp_bytes: 0 + + iaa device: + id: 3 + n_wqs: 1 + comp_calls: 0 + comp_bytes: 0 + decomp_calls: 0 + decomp_bytes: 0 + wqs: + name: iaa_crypto + comp_calls: 0 + comp_bytes: 0 + decomp_calls: 0 + decomp_bytes: 0 + + iaa device: + id: 5 + n_wqs: 1 + comp_calls: 100 + comp_bytes: 22800 + decomp_calls: 100 + decomp_bytes: 22800 + wqs: + name: iaa_crypto + comp_calls: 100 + comp_bytes: 22800 + decomp_calls: 100 + decomp_bytes: 22800 + +Writing 0 to 'stats_reset' resets all the stats, including the +per-device and per-wq stats:: + + # echo 0 > stats_reset + # cat wq_stats + global stats: + total_comp_calls: 0 + total_decomp_calls: 0 + total_comp_bytes_out: 0 + total_decomp_bytes_in: 0 + total_completion_einval_errors: 0 + total_completion_timeout_errors: 0 + total_completion_comp_buf_overflow_errors: 0 + ... + + +Use cases +========= + +Simple zswap test +----------------- + +For this example, the kernel should be configured according to the +dedicated mode options described above, and zswap should be enabled as +well:: + + CONFIG_ZSWAP=y + +This is a simple test that uses iaa_compress as the compressor for a +swap (zswap) device. It sets up the zswap device and then uses the +memory_memadvise program listed below to forcibly swap out and in a +specified number of pages, demonstrating both compress and decompress. + +The zswap test expects the work queues for each IAA device on the +system to be configured properly as a kernel workqueue with a +workqueue driver_name of "crypto". + +The first step is to make sure the iaa_crypto module is loaded:: + + modprobe iaa_crypto + +Following that the IAA device(s) should be configured and enabled. + +The zswap test expects the work queues for each IAA device on the +system to be configured properly as a kernel workqueue with a +workqueue driver_name of "crypto". + +The below script automatically does that:: + + #!/bin/bash + + echo "IAA devices:" + lspci -d:0cfe + echo "# IAA devices:" + lspci -d:0cfe | wc -l + + # + # count iaa instances + # + iaa_dev_id="0cfe" + num_iaa=$(lspci -d:${iaa_dev_id} | wc -l) + echo "Found ${num_iaa} IAA instances" + + # + # disable iaa wqs and devices + # + echo "Disable IAA" + + for ((i = 1; i < ${num_iaa} * 2; i += 2)); do + echo disable wq iax${i}/wq${i}.0 + accel-config disable-wq iax${i}/wq${i}.0 + echo disable iaa iax${i} + accel-config disable-device iax${i} + done + + echo "End Disable IAA" + + # + # configure iaa wqs and devices + # + echo "Configure IAA" + for ((i = 1; i < ${num_iaa} * 2; i += 2)); do + accel-config config-wq --group-id=0 --mode=dedicated --size=128 --priority=10 --type=kernel --name="iaa_crypto" --driver_name="crypto" iax${i}/wq${i} + done + + echo "End Configure IAA" + + # + # enable iaa wqs and devices + # + echo "Enable IAA" + + for ((i = 1; i < ${num_iaa} * 2; i += 2)); do + echo enable iaa iaa${i} + accel-config enable-device iaa${i} + echo enable wq iaa${i}/wq${i}.0 + accel-config enable-wq iaa${i}/wq${i}.0 + done + + echo "End Enable IAA" + +When the workqueues are bound to the iaa_crypto driver, you should +see something similar to the following in dmesg output if you've +enabled debug output (echo -n 'module iaa_crypto +p' > +/sys/kernel/debug/dynamic_debug/control):: + + [ 60.752344] idxd 0000:f6:02.0: add_iaa_wq: added wq 000000004068d14d to iaa 00000000c9585ba2, n_wq 1 + [ 60.752346] iaa_crypto: rebalance_wq_table: nr_nodes=2, nr_cpus 160, nr_iaa 8, cpus_per_iaa 20 + [ 60.752347] iaa_crypto: rebalance_wq_table: iaa=0 + [ 60.752349] idxd 0000:6a:02.0: request_iaa_wq: getting wq from iaa_device 0000000042d7bc52 (0) + [ 60.752350] idxd 0000:6a:02.0: request_iaa_wq: returning unused wq 00000000c8bb4452 (0) from iaa device 0000000042d7bc52 (0) + [ 60.752352] iaa_crypto: rebalance_wq_table: assigned wq for cpu=0, node=0 = wq 00000000c8bb4452 + [ 60.752354] iaa_crypto: rebalance_wq_table: iaa=0 + [ 60.752355] idxd 0000:6a:02.0: request_iaa_wq: getting wq from iaa_device 0000000042d7bc52 (0) + [ 60.752356] idxd 0000:6a:02.0: request_iaa_wq: returning unused wq 00000000c8bb4452 (0) from iaa device 0000000042d7bc52 (0) + [ 60.752358] iaa_crypto: rebalance_wq_table: assigned wq for cpu=1, node=0 = wq 00000000c8bb4452 + [ 60.752359] iaa_crypto: rebalance_wq_table: iaa=0 + [ 60.752360] idxd 0000:6a:02.0: request_iaa_wq: getting wq from iaa_device 0000000042d7bc52 (0) + [ 60.752361] idxd 0000:6a:02.0: request_iaa_wq: returning unused wq 00000000c8bb4452 (0) from iaa device 0000000042d7bc52 (0) + [ 60.752362] iaa_crypto: rebalance_wq_table: assigned wq for cpu=2, node=0 = wq 00000000c8bb4452 + [ 60.752364] iaa_crypto: rebalance_wq_table: iaa=0 + . + . + . + +Once the workqueues and devices have been enabled, the iaa_crypto +algorithm is enabled and available. When the iaa_crypto algorithm has +been successfully enabled, you should see the following dmesg output:: + + [ 64.893759] iaa_crypto: iaa_crypto_enable: iaa_crypto now ENABLED + +Now run the following zswap-specific setup commands:: + + echo 0 > /sys/module/zswap/parameters/enabled + echo 50 > /sys/module/zswap/parameters/max_pool_percent + echo iaa_crypto > /sys/module/zswap/parameters/compressor + echo zsmalloc > /sys/module/zswap/parameters/zpool + echo 1 > /sys/module/zswap/parameters/enabled + echo 0 > /sys/module/zswap/parameters/same_filled_pages_enabled + + echo 100 > /proc/sys/vm/swappiness + echo never > /sys/kernel/mm/transparent_hugepage/enabled + echo 1 > /proc/sys/vm/overcommit_memory + +Finally, you can now run the zswap workload you want to measure. For +example, using the code below, the following command will swap in and +out 100 pages:: + + ./memory_madvise 100 + + Allocating 100 pages to swap in/out + Swapping out 100 pages + Swapping in 100 pages + Swapped out and in 100 pages + +You should see something like the following in the dmesg output if +you've enabled debug output (echo -n 'module iaa_crypto +p' > +/sys/kernel/debug/dynamic_debug/control):: + + [ 404.202972] idxd 0000:e7:02.0: iaa_comp_acompress: dma_map_sg, src_addr 223925c000, nr_sgs 1, req->src 00000000ee7cb5e6, req->slen 4096, sg_dma_len(sg) 4096 + [ 404.202973] idxd 0000:e7:02.0: iaa_comp_acompress: dma_map_sg, dst_addr 21dadf8000, nr_sgs 1, req->dst 000000008d6acea8, req->dlen 4096, sg_dma_len(sg) 8192 + [ 404.202975] idxd 0000:e7:02.0: iaa_compress: desc->src1_addr 223925c000, desc->src1_size 4096, desc->dst_addr 21dadf8000, desc->max_dst_size 4096, desc->src2_addr 2203543000, desc->src2_size 1568 + [ 404.202981] idxd 0000:e7:02.0: iaa_compress_verify: (verify) desc->src1_addr 21dadf8000, desc->src1_size 228, desc->dst_addr 223925c000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + [ 409.203227] idxd 0000:e7:02.0: iaa_comp_adecompress: dma_map_sg, src_addr 21ddd8b100, nr_sgs 1, req->src 0000000084adab64, req->slen 228, sg_dma_len(sg) 228 + [ 409.203235] idxd 0000:e7:02.0: iaa_comp_adecompress: dma_map_sg, dst_addr 21ee3dc000, nr_sgs 1, req->dst 000000004e2990d0, req->dlen 4096, sg_dma_len(sg) 4096 + [ 409.203239] idxd 0000:e7:02.0: iaa_decompress: desc->src1_addr 21ddd8b100, desc->src1_size 228, desc->dst_addr 21ee3dc000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + [ 409.203254] idxd 0000:e7:02.0: iaa_comp_adecompress: dma_map_sg, src_addr 21ddd8b100, nr_sgs 1, req->src 0000000084adab64, req->slen 228, sg_dma_len(sg) 228 + [ 409.203256] idxd 0000:e7:02.0: iaa_comp_adecompress: dma_map_sg, dst_addr 21f1551000, nr_sgs 1, req->dst 000000004e2990d0, req->dlen 4096, sg_dma_len(sg) 4096 + [ 409.203257] idxd 0000:e7:02.0: iaa_decompress: desc->src1_addr 21ddd8b100, desc->src1_size 228, desc->dst_addr 21f1551000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + +memory_madvise.c (gcc -o memory_memadvise memory_madvise.c):: + + #include + #include + #include + #include + #include + #include + + #ifndef MADV_PAGEOUT + #define MADV_PAGEOUT 21 /* force pages out immediately */ + #endif + + #define PG_SZ 4096 + + int main(int argc, char **argv) + { + int i, nr_pages = 1; + int64_t *dump_ptr; + char *addr, *a; + int loop = 1; + + if (argc > 1) + nr_pages = atoi(argv[1]); + + printf("Allocating %d pages to swap in/out\n", nr_pages); + + /* allocate pages */ + addr = mmap(NULL, nr_pages * PG_SZ, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); + *addr = 1; + + /* initialize data in page to all '*' chars */ + memset(addr, '*', nr_pages * PG_SZ); + + printf("Swapping out %d pages\n", nr_pages); + + /* Tell kernel to swap it out */ + madvise(addr, nr_pages * PG_SZ, MADV_PAGEOUT); + + while (loop > 0) { + /* Wait for swap out to finish */ + sleep(5); + + a = addr; + + printf("Swapping in %d pages\n", nr_pages); + + /* Access the page ... this will swap it back in again */ + for (i = 0; i < nr_pages; i++) { + if (a[0] != '*') { + printf("Bad data from decompress!!!!!\n"); + + dump_ptr = (int64_t *)a; + for (int j = 0; j < 100; j++) { + printf(" page %d data: %#llx\n", i, *dump_ptr); + dump_ptr++; + } + } + + a += PG_SZ; + } + + loop --; + } + + printf("Swapped out and in %d pages\n", nr_pages); diff --git a/Documentation/driver-api/crypto/iaa/index.rst b/Documentation/driver-api/crypto/iaa/index.rst new file mode 100644 index 000000000000..aa6837e27264 --- /dev/null +++ b/Documentation/driver-api/crypto/iaa/index.rst @@ -0,0 +1,20 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================= +IAA (Intel Analytics Accelerator) +================================= + +IAA provides hardware compression and decompression via the crypto +API. + +.. toctree:: + :maxdepth: 1 + + iaa-crypto + +.. only:: subproject and html + + Indices + ======= + + * :ref:`genindex` diff --git a/Documentation/driver-api/crypto/index.rst b/Documentation/driver-api/crypto/index.rst new file mode 100644 index 000000000000..fb9709b98bea --- /dev/null +++ b/Documentation/driver-api/crypto/index.rst @@ -0,0 +1,20 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============== +Crypto Drivers +============== + +Documentation for crypto drivers that may need more involved setup and +configuration. + +.. toctree:: + :maxdepth: 1 + + iaa/index + +.. only:: subproject and html + + Indices + ======= + + * :ref:`genindex` diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index ff9aa1afdc62..2ad1237d5d5f 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -113,6 +113,7 @@ available subsections can be seen below. xillybus zorro hte/index + crypto/index .. only:: subproject and html From patchwork Tue Mar 28 15:35:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 667987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C7FFC77B6D for ; Tue, 28 Mar 2023 15:37:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234091AbjC1PhP (ORCPT ); Tue, 28 Mar 2023 11:37:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234041AbjC1Pgg (ORCPT ); Tue, 28 Mar 2023 11:36:36 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4DD181040C; Tue, 28 Mar 2023 08:36:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680017770; x=1711553770; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ObaQS2pwjlIdSbSrb0xThYdsWIAaK38OpfGK6YL3zRc=; b=ZIv/wZCmoXylMch4yUqpHuHT4Vx0N8aNfNNzwehyfp8MjfrlQMfj4Oe8 fiiXZJA/VjWmnIk9kRntNw8Mdp88/ajGF2sYE1fPWp4q0XJ7Py9HXISEX lnoqGHeqWo1JXzO9WOfT+DflCm4LZC2DbFfvH+SBzJfUNmGi0l+ab5igu M7njhm8gwQIAQ76QP77QhALrlBg6iiB5XyYyOXao/qGFyRV62pQGDXCew 5AxoV2okAwMajcaXfmXtwD2WAeENbGkckKfvjgREPabhc5vJ5FjEdtyYA y3cRz5mlB/WbxbycbbPnDkd0WvUgto0xnCGHSik3jqmY+EHfGYif0uFs0 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="342192182" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="342192182" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:36:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="683948451" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="683948451" Received: from sdwarak1-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.intel.com) ([10.212.127.200]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:36:05 -0700 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, giovanni.cabiddu@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v2 10/15] crypto: iaa - Add per-cpu workqueue table with rebalancing Date: Tue, 28 Mar 2023 10:35:30 -0500 Message-Id: <20230328153535.126223-11-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230328153535.126223-1-tom.zanussi@linux.intel.com> References: <20230328153535.126223-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The iaa compression/decompression algorithms in later patches need a way to retrieve an appropriate IAA workqueue depending on how close the associated IAA device is to the current cpu. For this purpose, add a per-cpu array of workqueues such that an appropriate workqueue can be retrieved by simply accessing the per-cpu array. Whenever a new workqueue is bound to or unbound from the iaa_crypto driver, the available workqueues are 'rebalanced' such that work submitted from a particular CPU is given to the most appropriate workqueue available. There currently isn't any way for the user to tweak the way this is done internally - if necessary, knobs can be added later for that purpose. Current best practice is to configure and bind at least one workqueue for each IAA device, but as long as there is at least one workqueue configured and bound to any IAA device in the system, the iaa_crypto driver will work, albeit most likely not as efficiently. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi --- drivers/crypto/intel/iaa/iaa_crypto.h | 7 + drivers/crypto/intel/iaa/iaa_crypto_main.c | 244 +++++++++++++++++++++ 2 files changed, 251 insertions(+) diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h index 5d1fff7f4b8e..c25546fa87f7 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto.h +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -27,4 +27,11 @@ struct iaa_device { struct list_head wqs; }; +struct wq_table_entry { + struct idxd_wq **wqs; + int max_wqs; + int n_wqs; + int cur_wq; +}; + #endif diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index c35a778d53a6..cf728cf8798f 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -22,10 +22,92 @@ /* number of iaa instances probed */ static unsigned int nr_iaa; +static unsigned int nr_cpus; +static unsigned int nr_nodes; +static unsigned int nr_cpus_per_node; + +/* Number of physical cpus sharing each iaa instance */ +static unsigned int cpus_per_iaa; + +/* Per-cpu lookup table for balanced wqs */ +static struct wq_table_entry __percpu *wq_table; + +static void wq_table_add(int cpu, struct idxd_wq *wq) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + if (WARN_ON(entry->n_wqs == entry->max_wqs)) + return; + + entry->wqs[entry->n_wqs++] = wq; + + pr_debug("%s: added iaa wq %d.%d to idx %d of cpu %d\n", __func__, + entry->wqs[entry->n_wqs - 1]->idxd->id, + entry->wqs[entry->n_wqs - 1]->id, entry->n_wqs - 1, cpu); +} + +static void wq_table_free_entry(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + kfree(entry->wqs); + memset(entry, 0, sizeof(*entry)); +} + +static void wq_table_clear_entry(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + entry->n_wqs = 0; + entry->cur_wq = 0; + memset(entry->wqs, 0, entry->max_wqs * sizeof(struct idxd_wq *)); +} static LIST_HEAD(iaa_devices); static DEFINE_MUTEX(iaa_devices_lock); +/* + * Given a cpu, find the closest IAA instance. The idea is to try to + * choose the most appropriate IAA instance for a caller and spread + * available workqueues around to clients. + */ +static inline int cpu_to_iaa(int cpu) +{ + int node, n_cpus = 0, test_cpu, iaa = 0; + int nr_iaa_per_node, nr_cores_per_iaa; + const struct cpumask *node_cpus; + + if (!nr_nodes) + return 0; + + nr_iaa_per_node = nr_iaa / nr_nodes; + if (!nr_iaa_per_node) + return 0; + + nr_cores_per_iaa = nr_cpus_per_node / nr_iaa_per_node; + + for_each_online_node(node) { + node_cpus = cpumask_of_node(node); + if (!cpumask_test_cpu(cpu, node_cpus)) + continue; + + for_each_cpu(test_cpu, node_cpus) { + if ((n_cpus % nr_cpus_per_node) == 0) + iaa = node * nr_iaa_per_node; + + if (test_cpu == cpu) + return iaa; + + n_cpus++; + + if ((n_cpus % cpus_per_iaa) == 0) + iaa++; + } + } + + return -1; +} + static struct iaa_device *iaa_device_alloc(void) { struct iaa_device *iaa_device; @@ -141,6 +223,53 @@ static void del_iaa_wq(struct iaa_device *iaa_device, struct idxd_wq *wq) } } +static void clear_wq_table(void) +{ + int cpu; + + for (cpu = 0; cpu < nr_cpus; cpu++) + wq_table_clear_entry(cpu); + + pr_debug("cleared wq table\n"); +} + +static void free_wq_table(void) +{ + int cpu; + + for (cpu = 0; cpu < nr_cpus; cpu++) + wq_table_free_entry(cpu); + + free_percpu(wq_table); + + pr_debug("freed wq table\n"); +} + +static int alloc_wq_table(int max_wqs) +{ + struct wq_table_entry *entry; + int cpu; + + wq_table = alloc_percpu(struct wq_table_entry); + if (!wq_table) + return -ENOMEM; + + for (cpu = 0; cpu < nr_cpus; cpu++) { + entry = per_cpu_ptr(wq_table, cpu); + entry->wqs = kzalloc(GFP_KERNEL, max_wqs * sizeof(struct wq *)); + if (!entry->wqs) { + free_wq_table(); + return -ENOMEM; + } + + entry->max_wqs = max_wqs; + } + + pr_debug("initialized wq table\n"); + + return 0; +} + static int save_iaa_wq(struct idxd_wq *wq) { struct iaa_device *iaa_device, *found = NULL; @@ -195,6 +324,8 @@ static int save_iaa_wq(struct idxd_wq *wq) return -EINVAL; idxd_wq_get(wq); + + cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa; out: return 0; } @@ -210,6 +341,97 @@ static void remove_iaa_wq(struct idxd_wq *wq) break; } } + + if (nr_iaa) + cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa; + else + cpus_per_iaa = 0; +} + +static int wq_table_add_wqs(int iaa, int cpu) +{ + struct iaa_device *iaa_device, *found_device = NULL; + int ret = 0, cur_iaa = 0, n_wqs_added = 0; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + + list_for_each_entry(iaa_device, &iaa_devices, list) { + idxd = iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + if (cur_iaa != iaa) { + cur_iaa++; + continue; + } + + found_device = iaa_device; + dev_dbg(dev, "getting wq from iaa_device %d, cur_iaa %d\n", + found_device->idxd->id, cur_iaa); + break; + } + + if (!found_device) { + found_device = list_first_entry_or_null(&iaa_devices, + struct iaa_device, list); + if (!found_device) { + pr_debug("couldn't find any iaa devices with wqs!\n"); + ret = -EINVAL; + goto out; + } + cur_iaa = 0; + + idxd = found_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + dev_dbg(dev, "getting wq from only iaa_device %d, cur_iaa %d\n", + found_device->idxd->id, cur_iaa); + } + + list_for_each_entry(iaa_wq, &found_device->wqs, list) { + wq_table_add(cpu, iaa_wq->wq); + pr_debug("rebalance: added wq for cpu=%d: iaa wq %d.%d\n", + cpu, iaa_wq->wq->idxd->id, iaa_wq->wq->id); + n_wqs_added++; + }; + + if (!n_wqs_added) { + pr_debug("couldn't find any iaa wqs!\n"); + ret = -EINVAL; + goto out; + } +out: + return ret; +} + +static void rebalance_wq_table(void) +{ + int cpu, iaa; + + if (nr_iaa == 0) + return; + + clear_wq_table(); + + pr_debug("rebalance: nr_nodes=%d, nr_cpus %d, nr_iaa %d, cpus_per_iaa %d\n", + nr_nodes, nr_cpus, nr_iaa, cpus_per_iaa); + + for (cpu = 0; cpu < nr_cpus; cpu++) { + iaa = cpu_to_iaa(cpu); + pr_debug("rebalance: cpu=%d iaa=%d\n", cpu, iaa); + + if (WARN_ON(iaa == -1)) { + pr_debug("rebalance (cpu_to_iaa(%d)) failed!\n", cpu); + return; + } + + if (WARN_ON(wq_table_add_wqs(iaa, cpu))) { + pr_debug("could not add any wqs for iaa %d to cpu %d!\n", iaa, cpu); + return; + } + } } static int iaa_crypto_probe(struct idxd_dev *idxd_dev) @@ -218,6 +440,7 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) struct idxd_device *idxd = wq->idxd; struct idxd_driver_data *data = idxd->data; struct device *dev = &idxd_dev->conf_dev; + bool first_wq = false; int ret = 0; if (idxd->state != IDXD_DEV_ENABLED) @@ -248,10 +471,19 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) mutex_lock(&iaa_devices_lock); + if (list_empty(&iaa_devices)) { + ret = alloc_wq_table(wq->idxd->max_wqs); + if (ret) + goto err_alloc; + first_wq = true; + } + ret = save_iaa_wq(wq); if (ret) goto err_save; + rebalance_wq_table(); + mutex_unlock(&iaa_devices_lock); out: mutex_unlock(&wq->wq_lock); @@ -259,6 +491,10 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) return ret; err_save: + if (first_wq) + free_wq_table(); +err_alloc: + mutex_unlock(&iaa_devices_lock); drv_disable_wq(wq); err: wq->type = IDXD_WQT_NONE; @@ -277,6 +513,10 @@ static void iaa_crypto_remove(struct idxd_dev *idxd_dev) remove_iaa_wq(wq); drv_disable_wq(wq); + rebalance_wq_table(); + + if (nr_iaa == 0) + free_wq_table(); mutex_unlock(&iaa_devices_lock); mutex_unlock(&wq->wq_lock); @@ -298,6 +538,10 @@ static int __init iaa_crypto_init_module(void) { int ret = 0; + nr_cpus = num_online_cpus(); + nr_nodes = num_online_nodes(); + nr_cpus_per_node = boot_cpu_data.x86_max_cores; + ret = idxd_driver_register(&iaa_crypto_driver); if (ret) { pr_debug("IAA wq sub-driver registration failed\n"); From patchwork Tue Mar 28 15:35:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 667986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F288FC76196 for ; Tue, 28 Mar 2023 15:37:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234083AbjC1Phw (ORCPT ); Tue, 28 Mar 2023 11:37:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233370AbjC1PhQ (ORCPT ); Tue, 28 Mar 2023 11:37:16 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1433C658; Tue, 28 Mar 2023 08:36:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680017791; x=1711553791; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bAvcM2wnoFYRdEaCetQ1olQ6t4qAY5Yvo/LcHgDRG6c=; b=SXHh2jmH9l/ojbVd2IauCHj40i7BVmA4RUjpwWzaXZpvWIIDmKP6JdlI Y5uMvTHMU18r9gzWn0qWjLCxy7YxVw1rU9W6DmOeLW/Y4I2e2xNcjQWDe 6HSW/qUeF98fhKb0OXPtWfD7n+B0DIbgitthVi232Abp4Rvnw3n/nVbUf EHGfgonCB38yu2r/YsLheYUmAH1vXfV36krzrQDJI1g0cvAps+mq0KFGH +g2CXnTqdI+vJCSWKFJOQ/jnlsPAFYqpYZnouWkyjidLPm8d5oRr5VxyT Rvno9X6rL+95uViH8t4kyAiH9Bxjw+UPnYfhyaVxwKASIue8zcuUUPvGf Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="342192239" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="342192239" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:36:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="683948510" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="683948510" Received: from sdwarak1-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.intel.com) ([10.212.127.200]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:36:13 -0700 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, giovanni.cabiddu@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v2 13/15] crypto: iaa - Add support for default IAA 'canned' compression mode Date: Tue, 28 Mar 2023 10:35:33 -0500 Message-Id: <20230328153535.126223-14-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230328153535.126223-1-tom.zanussi@linux.intel.com> References: <20230328153535.126223-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add support for a 'canned' compression mode using the IAA compression mode in-kernel API. The IAA 'canned' compression mode is added alongside the existing 'fixed' compression mode and is named simply 'canned'. It implements a good general-purpose compression scheme whose tables were generated from statistics derived from a wide variety of SPEC17 workloads. It provides much better overall characteristics than the existing deflate-1951 tables implemented by 'fixed'. Either 'fixed' or 'canned' modes can be chosen as the mode to be used for compression/decompression via the iaa_crypto compression_mode iaa_crypto driver attribute: To choose 'fixed' mode: echo "fixed" > /sys/bus/dsa/drivers/crypto/compression_mode To choose 'canned' mode: echo "canned" > /sys/bus/dsa/drivers/crypto/compression_mode [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi --- crypto/testmgr.c | 10 ++ crypto/testmgr.h | 72 ++++++++++++ drivers/crypto/intel/iaa/Makefile | 2 +- drivers/crypto/intel/iaa/iaa_crypto.h | 2 + .../crypto/intel/iaa/iaa_crypto_comp_canned.c | 110 ++++++++++++++++++ drivers/crypto/intel/iaa/iaa_crypto_main.c | 43 ++++++- 6 files changed, 236 insertions(+), 3 deletions(-) create mode 100644 drivers/crypto/intel/iaa/iaa_crypto_comp_canned.c diff --git a/crypto/testmgr.c b/crypto/testmgr.c index c91e93ece20b..e10bc9581d86 100644 --- a/crypto/testmgr.c +++ b/crypto/testmgr.c @@ -5235,6 +5235,16 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(hmac_streebog512_tv_template) } + }, { + .alg = "iaa-canned-deflate", + .test = alg_test_comp, + .fips_allowed = 1, + .suite = { + .comp = { + .comp = __VECS(iaa_canned_deflate_comp_tv_template), + .decomp = __VECS(iaa_canned_deflate_decomp_tv_template) + } + } }, { .alg = "jitterentropy_rng", .fips_allowed = 1, diff --git a/crypto/testmgr.h b/crypto/testmgr.h index f10bfb9d9973..9a26fcc3364f 100644 --- a/crypto/testmgr.h +++ b/crypto/testmgr.h @@ -35782,6 +35782,78 @@ static const struct comp_testvec zlib_deflate_decomp_tv_template[] = { }, }; +static const struct comp_testvec iaa_canned_deflate_comp_tv_template[] = { + { + .inlen = 70, + .outlen = 37, + .input = "Join us now and share the software " + "Join us now and share the software ", + .output = "\x6d\x23\x43\x23\xa4\x71\x31\xd2" + "\x88\xc8\x61\x52\x75\x84\x56\x1a" + "\x13\xa2\x8e\xd6\x49\x63\x43\x74" + "\xd2\x98\xc8\xe0\xd8\x61\x58\x69" + "\xcb\x71\x01\xe5\x7f", + }, { + .inlen = 191, + .outlen = 128, + .input = "This document describes a compression method based on the DEFLATE" + "compression algorithm. This document defines the application of " + "the DEFLATE algorithm to the IP Payload Compression Protocol.", + .output = "\xdd\x42\x42\x63\xa4\xda\x48\x4d" + "\x5c\xb8\x2e\x22\x56\xaa\xd5\xc5" + "\x68\xa2\x43\x83\x74\x31\x52\xb5" + "\x54\x13\x19\x1e\x15\xad\x8b\x89" + "\x09\x8d\x8c\x90\x86\xeb\x62\x43" + "\x22\xb5\xd2\x20\x75\x8c\x4e\x2b" + "\x05\x3d\x36\x44\x27\xf5\x69\xe5" + "\xdb\xde\xbb\x5b\x2b\x7d\x37\x75" + "\xd8\xc0\xc8\xe8\xd0\xd8\x90\x70" + "\x7b\xa9\x54\x1c\x38\x38\x34\x02" + "\xc2\xe2\x8e\xea\xa8\xa8\xb0\x50" + "\x8d\x3a\x16\xf7\x88\x0c\xd6\x8f" + "\x95\x1f\x40\x1a\x1b\x29\x34\xb4" + "\xf1\x97\xfa\xab\x87\x87\x45\xaa" + "\xb5\xd2\x96\x7a\x03\xf9\x47\x47" + "\xc6\x46\x6a\x22\xc3\xec\xff\x07", + }, +}; + +static const struct comp_testvec iaa_canned_deflate_decomp_tv_template[] = { + { + .inlen = 128, + .outlen = 191, + .input = "\xdd\x42\x42\x63\xa4\xda\x48\x4d" + "\x5c\xb8\x2e\x22\x56\xaa\xd5\xc5" + "\x68\xa2\x43\x83\x74\x31\x52\xb5" + "\x54\x13\x19\x1e\x15\xad\x8b\x89" + "\x09\x8d\x8c\x90\x86\xeb\x62\x43" + "\x22\xb5\xd2\x20\x75\x8c\x4e\x2b" + "\x05\x3d\x36\x44\x27\xf5\x69\xe5" + "\xdb\xde\xbb\x5b\x2b\x7d\x37\x75" + "\xd8\xc0\xc8\xe8\xd0\xd8\x90\x70" + "\x7b\xa9\x54\x1c\x38\x38\x34\x02" + "\xc2\xe2\x8e\xea\xa8\xa8\xb0\x50" + "\x8d\x3a\x16\xf7\x88\x0c\xd6\x8f" + "\x95\x1f\x40\x1a\x1b\x29\x34\xb4" + "\xf1\x97\xfa\xab\x87\x87\x45\xaa" + "\xb5\xd2\x96\x7a\x03\xf9\x47\x47" + "\xc6\x46\x6a\x22\xc3\xec\xff\x07", + .output = "This document describes a compression method based on the DEFLATE" + "compression algorithm. This document defines the application of " + "the DEFLATE algorithm to the IP Payload Compression Protocol.", + }, { + .inlen = 37, + .outlen = 70, + .input = "\x6d\x23\x43\x23\xa4\x71\x31\xd2" + "\x88\xc8\x61\x52\x75\x84\x56\x1a" + "\x13\xa2\x8e\xd6\x49\x63\x43\x74" + "\xd2\x98\xc8\xe0\xd8\x61\x58\x69" + "\xcb\x71\x01\xe5\x7f", + .output = "Join us now and share the software " + "Join us now and share the software ", + }, +}; + /* * LZO test vectors (null-terminated strings). */ diff --git a/drivers/crypto/intel/iaa/Makefile b/drivers/crypto/intel/iaa/Makefile index cc87feffd059..ff6ab1d0bc13 100644 --- a/drivers/crypto/intel/iaa/Makefile +++ b/drivers/crypto/intel/iaa/Makefile @@ -7,4 +7,4 @@ ccflags-y += -I $(srctree)/drivers/dma/idxd -DDEFAULT_SYMBOL_NAMESPACE=IDXD obj-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO) := iaa_crypto.o -iaa_crypto-y := iaa_crypto_main.o iaa_crypto_comp_fixed.o +iaa_crypto-y := iaa_crypto_main.o iaa_crypto_comp_canned.o iaa_crypto_comp_fixed.o diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h index 7ed3d8de250d..3a04a7bdea8e 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto.h +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -112,6 +112,8 @@ struct aecs_decomp_table_record { u32 decomp_state[1220]; } __packed; +int iaa_aecs_init_canned(void); +void iaa_aecs_cleanup_canned(void); int iaa_aecs_init_fixed(void); void iaa_aecs_cleanup_fixed(void); diff --git a/drivers/crypto/intel/iaa/iaa_crypto_comp_canned.c b/drivers/crypto/intel/iaa/iaa_crypto_comp_canned.c new file mode 100644 index 000000000000..aff0899ffb9e --- /dev/null +++ b/drivers/crypto/intel/iaa/iaa_crypto_comp_canned.c @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ + +#include "idxd.h" +#include "iaa_crypto.h" + +#define IAA_AECS_ALIGN 32 + +/* + * These tables were generated from statistics derived from a wide + * variety of SPEC17 workloads and implement a good general-purpose + * compression scheme called simply 'canned'. + */ + +static const u32 canned_ll_iaa[286] = { +0x28002, 0x38024, 0x40066, 0x40067, 0x40068, 0x48144, 0x40069, 0x48145, +0x4006a, 0x48146, 0x4006b, 0x48147, 0x48148, 0x48149, 0x4814a, 0x4814b, +0x4006c, 0x4814c, 0x4814d, 0x4814e, 0x4814f, 0x48150, 0x48151, 0x48152, +0x4006d, 0x48153, 0x48154, 0x48155, 0x48156, 0x48157, 0x48158, 0x48159, +0x38025, 0x4815a, 0x4815b, 0x4815c, 0x4815d, 0x4815e, 0x4815f, 0x48160, +0x4006e, 0x48161, 0x48162, 0x48163, 0x48164, 0x48165, 0x4006f, 0x48166, +0x38026, 0x38027, 0x40070, 0x40071, 0x40072, 0x40073, 0x40074, 0x40075, +0x38028, 0x40076, 0x40077, 0x48167, 0x40078, 0x40079, 0x4007a, 0x38029, +0x3802a, 0x4007b, 0x48168, 0x48169, 0x4007c, 0x4816a, 0x4007d, 0x4816b, +0x4007e, 0x4816c, 0x4816d, 0x4816e, 0x4816f, 0x48170, 0x48171, 0x48172, +0x4007f, 0x48173, 0x48174, 0x48175, 0x48176, 0x48177, 0x48178, 0x48179, +0x40080, 0x4817a, 0x4817b, 0x4817c, 0x4817d, 0x4817e, 0x4817f, 0x48180, +0x40081, 0x3802b, 0x40082, 0x3802c, 0x3802d, 0x3802e, 0x40083, 0x48181, +0x40084, 0x40085, 0x48182, 0x48183, 0x40086, 0x40087, 0x40088, 0x40089, +0x4008a, 0x48184, 0x4008b, 0x4008c, 0x4008d, 0x4008e, 0x48185, 0x48186, +0x4008f, 0x48187, 0x48188, 0x48189, 0x4818a, 0x4818b, 0x4818c, 0x4818d, +0x40090, 0x4818e, 0x4818f, 0x48190, 0x48191, 0x48192, 0x48193, 0x48194, +0x40091, 0x48195, 0x48196, 0x48197, 0x48198, 0x48199, 0x4819a, 0x4819b, +0x40092, 0x4819c, 0x4819d, 0x4819e, 0x4819f, 0x481a0, 0x481a1, 0x481a2, +0x40093, 0x481a3, 0x481a4, 0x481a5, 0x481a6, 0x481a7, 0x481a8, 0x481a9, +0x40094, 0x481aa, 0x481ab, 0x481ac, 0x481ad, 0x481ae, 0x481af, 0x481b0, +0x481b1, 0x481b2, 0x481b3, 0x481b4, 0x481b5, 0x481b6, 0x481b7, 0x481b8, +0x40095, 0x481b9, 0x481ba, 0x481bb, 0x481bc, 0x481bd, 0x481be, 0x481bf, +0x40096, 0x481c0, 0x481c1, 0x481c2, 0x481c3, 0x481c4, 0x481c5, 0x40097, +0x40098, 0x481c6, 0x481c7, 0x481c8, 0x481c9, 0x481ca, 0x481cb, 0x481cc, +0x40099, 0x481cd, 0x481ce, 0x481cf, 0x481d0, 0x481d1, 0x481d2, 0x481d3, +0x4009a, 0x481d4, 0x481d5, 0x481d6, 0x481d7, 0x481d8, 0x481d9, 0x481da, +0x481db, 0x481dc, 0x481dd, 0x481de, 0x481df, 0x481e0, 0x481e1, 0x481e2, +0x4009b, 0x481e3, 0x481e4, 0x481e5, 0x481e6, 0x481e7, 0x481e8, 0x481e9, +0x481ea, 0x481eb, 0x481ec, 0x481ed, 0x481ee, 0x481ef, 0x481f0, 0x481f1, +0x4009c, 0x481f2, 0x481f3, 0x481f4, 0x481f5, 0x481f6, 0x481f7, 0x481f8, +0x481f9, 0x481fa, 0x481fb, 0x503fe, 0x481fc, 0x481fd, 0x481fe, 0x4009d, +0x503ff, 0x20000, 0x28003, 0x30010, 0x28004, 0x28005, 0x28006, 0x4009e, +0x4009f, 0x3802f, 0x38030, 0x30011, 0x400a0, 0x38031, 0x38032, 0x400a1, +0x28007, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, +0x0, 0x0, 0x0, 0x0, 0x0, 0x0, +}; + +static const u32 canned_d_iaa[30] = { +0x3807e, 0x20004, 0x481fe, 0x18000, 0x400fe, 0x18001, 0x3003c, 0x20005, +0x20006, 0x28016, 0x20007, 0x28017, 0x20008, 0x28018, 0x28019, 0x20009, +0x2000a, 0x2801a, 0x2801b, 0x2801c, 0x2801d, 0x3003d, 0x3003e, 0x481ff, +0x0, 0x0, 0x0, 0x0, 0x0, 0x0, +}; + +#define CANNED_HEADER_SIZE (71) + +static const u8 canned_header[] = { +0x85, 0xd7, 0x05, 0x40, 0x54, 0x4d, 0x10, 0x06, +0x80, 0x3d, 0x40, 0x44, 0x2c, 0x6c, 0xec, 0xb3, +0xb1, 0xf0, 0x8e, 0x3c, 0x10, 0xb8, 0x43, 0x11, +0xb1, 0xb1, 0xb1, 0x39, 0xee, 0x0e, 0x41, 0x29, +0x09, 0xc5, 0xc6, 0xc6, 0xc6, 0xc6, 0xc6, 0x6e, +0xc5, 0xc6, 0x04, 0x1b, 0x1b, 0xbb, 0x15, 0xbb, +0x15, 0x3b, 0x7e, 0x95, 0x7f, 0xf6, 0xed, 0x2e, +0xdc, 0x3d, 0xee, 0xd8, 0x1b, 0x3f, 0xbe, 0x37, +0xb3, 0xb3, 0xb3, 0x3b, 0xb3, 0xf1, 0x9e, +}; + +#define HEADER_SIZE_IN_BITS 568 + +#define CEIL(a, b) (((a) + ((b) - 1)) / (b)) + +int iaa_aecs_init_canned(void) +{ + u16 gen_decomp_table_flags; + unsigned int slen; + int ret; + + slen = CEIL(HEADER_SIZE_IN_BITS, 8); + + gen_decomp_table_flags = 0x1; + gen_decomp_table_flags |= 1 << 9; // suppress output + gen_decomp_table_flags |= (((slen * 8) - HEADER_SIZE_IN_BITS) << 6); + + ret = add_iaa_compression_mode("canned", + canned_ll_iaa, + sizeof(canned_ll_iaa), + canned_d_iaa, + sizeof(canned_d_iaa), + canned_header, + sizeof(canned_header), + gen_decomp_table_flags, + NULL, NULL); + + if (!ret) + pr_debug("IAA canned compression mode initialized\n"); + + return ret; +} + +void iaa_aecs_cleanup_canned(void) +{ + remove_iaa_compression_mode("canned"); +} diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index d0b1ac45cc51..b6d41642f0dd 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -119,11 +119,16 @@ static DRIVER_ATTR_RW(verify_compress); static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; static int active_compression_mode; +static bool canned_mode = true; + static ssize_t compression_mode_show(struct device_driver *driver, char *buf) { int ret = 0; - ret = sprintf(buf, "%s\n", "fixed"); + if (canned_mode) + ret = sprintf(buf, "%s\n", "canned"); + else + ret = sprintf(buf, "%s\n", "fixed"); return ret; } @@ -382,6 +387,12 @@ int set_iaa_compression_mode(const char *name) ret = 0; } + if (ret == 0 && !strcmp(name, "canned")) + canned_mode = true; + + if (ret == 0 && !strcmp(name, "fixed")) + canned_mode = false; + return ret; } @@ -1109,6 +1120,7 @@ static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_device *idxd; struct iaa_wq *iaa_wq; struct pci_dev *pdev; + dma_addr_t src2_addr; struct device *dev; int ret = 0; @@ -1139,6 +1151,13 @@ static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, desc->max_dst_size = slen; desc->completion_addr = idxd_desc->compl_dma; + if (canned_mode) { + src2_addr = iaa_wq->iaa_device->active_compression_mode->aecs_decomp_table_dma_addr; + desc->src2_addr = (u64)src2_addr; + desc->src2_size = 1088; + desc->flags |= IDXD_OP_FLAG_RD_SRC2_AECS; + } + dev_dbg(dev, "(verify) compression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," @@ -1190,6 +1209,7 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_device *idxd; struct iaa_wq *iaa_wq; struct pci_dev *pdev; + dma_addr_t src2_addr; struct device *dev; int ret = 0; @@ -1219,6 +1239,13 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, desc->src1_size = slen; desc->completion_addr = idxd_desc->compl_dma; + if (canned_mode) { + src2_addr = iaa_wq->iaa_device->active_compression_mode->aecs_decomp_table_dma_addr; + desc->src2_addr = (u64)src2_addr; + desc->src2_size = 1088; + desc->flags |= IDXD_OP_FLAG_RD_SRC2_AECS; + } + dev_dbg(dev, "%s: decompression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," @@ -1496,7 +1523,6 @@ static struct acomp_alg iaa_acomp_deflate = { .decompress = iaa_comp_adecompress, .dst_free = sgl_free, .base = { - .cra_name = "deflate", .cra_driver_name = "iaa_crypto", .cra_module = THIS_MODULE, .cra_priority = IAA_ALG_PRIORITY, @@ -1507,6 +1533,11 @@ static int iaa_register_compression_device(void) { int ret; + if (canned_mode) + strcpy(iaa_acomp_deflate.base.cra_name, "iaa-canned-deflate"); + else + strcpy(iaa_acomp_deflate.base.cra_name, "deflate"); + ret = crypto_register_acomp(&iaa_acomp_deflate); if (ret) pr_err("deflate algorithm acomp registration failed (%d)\n", ret); @@ -1674,8 +1705,15 @@ static int __init iaa_crypto_init_module(void) goto err_attr_create; } + ret = iaa_aecs_init_canned(); + if (ret < 0) { + pr_debug("IAA canned compression mode init failed\n"); + goto err_compression_mode; + } + ret = iaa_aecs_init_fixed(); if (ret < 0) { + iaa_aecs_cleanup_canned(); pr_debug("IAA fixed compression mode init failed\n"); goto err_compression_mode; } @@ -1702,6 +1740,7 @@ static void __exit iaa_crypto_cleanup_module(void) driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_verify_compress); idxd_driver_unregister(&iaa_crypto_driver); + iaa_aecs_cleanup_canned(); iaa_aecs_cleanup_fixed(); pr_debug("cleaned up\n"); From patchwork Tue Mar 28 15:35:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 667985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BE9EC6FD18 for ; Tue, 28 Mar 2023 15:38:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234027AbjC1Pij (ORCPT ); Tue, 28 Mar 2023 11:38:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233192AbjC1Pht (ORCPT ); Tue, 28 Mar 2023 11:37:49 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34ACC11162; Tue, 28 Mar 2023 08:36:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680017809; x=1711553809; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ySf76nZG5BSVEABaWShXg0z5KKr8JGvJzU73vkpnzAg=; b=BCnwbE8DoJ5/T1pnLnKy6qHasRAjRNBgpXLFbY/nMDviT1Zc1EvDFfV3 OdUil8HaOV8BWJE57KD4Nvi4MAse4NuT0F8ZVplzgKdyFmWB4X6EaXXYS Hqz+Nlz9ELSN8E9FXju8k1D1thtZun7cyvYXEuNRrcrhgbVdoDF6IW47J n0mzahGQwirk8jEYSWj2jh3G/TjcGfWyMkhNN364ehCG/NN9Mk3vMtc4N 2R3iBAIMfY+kIcs8BkNfadurlVdR3YXIkV+conWuj94/W/tf0NgPnZqH7 PUq+hE81JjXXdl59trHLs78WU8SHpuvh7Whr+TjxtmlzMmtLesFmpOag8 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="342192250" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="342192250" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:36:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="683948523" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="683948523" Received: from sdwarak1-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.intel.com) ([10.212.127.200]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:36:17 -0700 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, giovanni.cabiddu@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v2 14/15] crypto: iaa - Add irq support for the crypto async interface Date: Tue, 28 Mar 2023 10:35:34 -0500 Message-Id: <20230328153535.126223-15-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230328153535.126223-1-tom.zanussi@linux.intel.com> References: <20230328153535.126223-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The existing iaa crypto async support provides an implementation that satisfies the interface but does so in a synchronous manner - it fills and submits the IDXD descriptor and then waits for it to complete before returning. This isn't a problem at the moment, since all existing callers (e.g. zswap) wrap any asynchronous callees in a synchronous wrapper anyway. This change makes the iaa crypto async implementation truly asynchronous: it fills and submits the IDXD descriptor, then returns immediately with -EINPROGRESS. It also sets the descriptor's 'request completion irq' bit and sets up a callback with the IDXD driver which is called when the operation completes and the irq fires. The existing callers such as zswap use synchronous wrappers to deal with -EINPROGRESS and so work as expected without any changes. This mode can be enabled by writing 'async_irq' to the sync_mode iaa_crypto driver attribute: echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode Async mode without interrupts (caller must poll) can be enabled by writing 'async' to it: echo async > /sys/bus/dsa/drivers/crypto/sync_mode The default sync mode can be enabled by writing 'sync' to it: echo sync > /sys/bus/dsa/drivers/crypto/sync_mode Signed-off-by: Tom Zanussi --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 224 ++++++++++++++++++++- 1 file changed, 222 insertions(+), 2 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index b6d41642f0dd..7a2c0ed80108 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -116,6 +116,102 @@ static ssize_t verify_compress_store(struct device_driver *driver, } static DRIVER_ATTR_RW(verify_compress); +/* + * The iaa crypto driver supports three 'sync' methods determining how + * compressions and decompressions are performed: + * + * - sync: the compression or decompression completes before + * returning. This is the mode used by the async crypto + * interface when the sync mode is set to 'sync' and by + * the sync crypto interface regardless of setting. + * + * - async: the compression or decompression is submitted and returns + * immediately. Completion interrupts are not used so + * the caller is responsible for polling the descriptor + * for completion. This mode is applicable to only the + * async crypto interface and is ignored for anything + * else. + * + * - async_irq: the compression or decompression is submitted and + * returns immediately. Completion interrupts are + * enabled so the caller can wait for the completion and + * yield to other threads. When the compression or + * decompression completes, the completion is signaled + * and the caller awakened. This mode is applicable to + * only the async crypto interface and is ignored for + * anything else. + * + * These modes can be set using the iaa_crypto sync_mode driver + * attribute. + */ + +/* Use async mode */ +static bool async_mode; +/* Use interrupts */ +static bool use_irq; + +/** + * set_iaa_sync_mode - Set IAA sync mode + * @name: The name of the sync mode + * + * Make the IAA sync mode named @name the current sync mode used by + * compression/decompression. + */ + +static int set_iaa_sync_mode(const char *name) +{ + int ret = 0; + + if (sysfs_streq(name, "sync")) { + async_mode = false; + use_irq = false; + } else if (sysfs_streq(name, "async")) { + async_mode = true; + use_irq = false; + } else if (sysfs_streq(name, "async_irq")) { + async_mode = true; + use_irq = true; + } else { + ret = -EINVAL; + } + + return ret; +} + +static ssize_t sync_mode_show(struct device_driver *driver, char *buf) +{ + int ret = 0; + + if (!async_mode && !use_irq) + ret = sprintf(buf, "%s\n", "sync"); + else if (async_mode && !use_irq) + ret = sprintf(buf, "%s\n", "async"); + else if (async_mode && use_irq) + ret = sprintf(buf, "%s\n", "async_irq"); + + return ret; +} + +static ssize_t sync_mode_store(struct device_driver *driver, + const char *buf, size_t count) +{ + int ret = -EBUSY; + + mutex_lock(&iaa_devices_lock); + + if (iaa_crypto_enabled) + goto out; + + ret = set_iaa_sync_mode(buf); + if (ret == 0) + ret = count; +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +} +static DRIVER_ATTR_RW(sync_mode); + static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; static int active_compression_mode; @@ -1033,6 +1129,72 @@ static inline int check_completion(struct device *dev, return ret; } +static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, + struct idxd_wq *wq, + dma_addr_t src_addr, unsigned int slen, + dma_addr_t dst_addr, unsigned int *dlen, + u32 compression_crc); + +static void iaa_desc_complete(struct idxd_desc *idxd_desc, + enum idxd_complete_type comp_type, + bool free_desc, void *__ctx, + u32 *status) +{ + struct crypto_ctx *ctx = __ctx; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + int ret, err = 0; + + iaa_wq = idxd_desc->wq->private_data; + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + dev_dbg(dev, "%s: compression mode %s," + " ctx->src_addr %llx, ctx->dst_addr %llx\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + ctx->src_addr, ctx->dst_addr); + + ret = check_completion(dev, idxd_desc->iax_completion, + ctx->compress, false); + if (ret) { + dev_dbg(dev, "%s: check_completion failed ret=%d\n", __func__, ret); + err = -EIO; + goto err; + } + + ctx->req->dlen = idxd_desc->iax_completion->output_size; + + if (ctx->compress && iaa_verify_compress) { + u32 compression_crc; + + compression_crc = idxd_desc->iax_completion->crc; + dma_sync_sg_for_device(dev, ctx->req->dst, 1, DMA_FROM_DEVICE); + dma_sync_sg_for_device(dev, ctx->req->src, 1, DMA_TO_DEVICE); + ret = iaa_compress_verify(ctx->tfm, ctx->req, iaa_wq->wq, ctx->src_addr, + ctx->req->slen, ctx->dst_addr, &ctx->req->dlen, + compression_crc); + if (ret) { + dev_dbg(dev, "%s: compress verify failed ret=%d\n", __func__, ret); + err = -EIO; + } + } +err: + if (ctx->req->base.complete) + acomp_request_complete(ctx->req, err); + + dma_unmap_sg(dev, ctx->req->dst, sg_nents(ctx->req->dst), DMA_FROM_DEVICE); + dma_unmap_sg(dev, ctx->req->src, sg_nents(ctx->req->src), DMA_TO_DEVICE); + + if (ret != 0) + dev_dbg(dev, "asynchronous compress failed ret=%d\n", ret); + + if (free_desc) + idxd_free_desc(idxd_desc->wq, idxd_desc); +} + static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_wq *wq, dma_addr_t src_addr, unsigned int slen, @@ -1075,6 +1237,22 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, desc->src2_size = sizeof(struct aecs_comp_table_record); desc->completion_addr = idxd_desc->compl_dma; + if (use_irq) { + desc->flags |= IDXD_OP_FLAG_RCI; + + idxd_desc->crypto.req = req; + idxd_desc->crypto.tfm = tfm; + idxd_desc->crypto.src_addr = src_addr; + idxd_desc->crypto.dst_addr = dst_addr; + idxd_desc->crypto.compress = true; + + dev_dbg(dev, "%s use_async_irq: compression mode %s," + " src_addr %llx, dst_addr %llx\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + src_addr, dst_addr); + } else if (async_mode && !disable_async) + req->base.data = idxd_desc; + dev_dbg(dev, "%s: compression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," @@ -1089,6 +1267,12 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, goto err; } + if (async_mode && !disable_async) { + ret = -EINPROGRESS; + dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); + goto out; + } + ret = check_completion(dev, idxd_desc->iax_completion, true, false); if (ret) { dev_dbg(dev, "check_completion failed ret=%d\n", ret); @@ -1099,7 +1283,8 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, *compression_crc = idxd_desc->iax_completion->crc; - idxd_free_desc(wq, idxd_desc); + if (!async_mode) + idxd_free_desc(wq, idxd_desc); out: return ret; err: @@ -1246,6 +1431,22 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, desc->flags |= IDXD_OP_FLAG_RD_SRC2_AECS; } + if (use_irq && !disable_async) { + desc->flags |= IDXD_OP_FLAG_RCI; + + idxd_desc->crypto.req = req; + idxd_desc->crypto.tfm = tfm; + idxd_desc->crypto.src_addr = src_addr; + idxd_desc->crypto.dst_addr = dst_addr; + idxd_desc->crypto.compress = false; + + dev_dbg(dev, "%s: use_async_irq compression mode %s," + " src_addr %llx, dst_addr %llx\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + src_addr, dst_addr); + } else if (async_mode && !disable_async) + req->base.data = idxd_desc; + dev_dbg(dev, "%s: decompression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," @@ -1260,6 +1461,12 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, goto err; } + if (async_mode && !disable_async) { + ret = -EINPROGRESS; + dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); + goto out; + } + ret = check_completion(dev, idxd_desc->iax_completion, false, false); if (ret) { dev_dbg(dev, "check_completion failed ret=%d\n", ret); @@ -1268,7 +1475,8 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, *dlen = idxd_desc->iax_completion->output_size; - idxd_free_desc(wq, idxd_desc); + if (!async_mode) + idxd_free_desc(wq, idxd_desc); out: return ret; err: @@ -1675,6 +1883,7 @@ static struct idxd_device_driver iaa_crypto_driver = { .remove = iaa_crypto_remove, .name = IDXD_SUBDRIVER_NAME, .type = dev_types, + .desc_complete = iaa_desc_complete, }; static int __init iaa_crypto_init_module(void) @@ -1691,6 +1900,13 @@ static int __init iaa_crypto_init_module(void) goto out; } + ret = driver_create_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); + if (ret) { + pr_debug("IAA sync mode attr creation failed\n"); + goto err_attr_create; + } + ret = driver_create_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); if (ret) { @@ -1723,6 +1939,8 @@ static int __init iaa_crypto_init_module(void) return ret; err_compression_mode: + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); driver_remove_file(&iaa_crypto_driver.drv, @@ -1735,6 +1953,8 @@ static int __init iaa_crypto_init_module(void) static void __exit iaa_crypto_cleanup_module(void) { + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); driver_remove_file(&iaa_crypto_driver.drv,