From patchwork Thu May 8 19:41:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888698 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC733289823; Thu, 8 May 2025 19:41:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733300; cv=none; b=QjfK1KIWoUDlndLbxsUm310OPk4sewLLs/m4M/tCM0bQ/E72cxSsxTGfj4i8MebdQ454IXBlyRJ+Ua2Sphb4PGgpvFCtgHiriZ8HiMscLy34nIO6UHQueX1ZCjeiunbv2LR1mJ8oeAMSW5AaNd/R8smv2uATp3zYScuPhkWMXj0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733300; c=relaxed/simple; bh=xDsbv3ls5X3YM0JiXefAsh4WsqCsb4fO+p9sSZmG6EM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dJI1+rAau/uBKbC1sFMu7rFI8hCFv1JbVw7rlEWCFK0cLUFRxD2erV1M1RnmwRGieRNv2zTBgdeUCcGSYijejcrSu0eK9L3UCqlbJHrtsq4o9UmvWZqbLWZ3Ai4eIi7NVMzcM0BuvFPgVkO5cUSOKhScr7bAhHn/lud8qNZcAKQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=gQ2Gnb1h; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="gQ2Gnb1h" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733298; x=1778269298; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xDsbv3ls5X3YM0JiXefAsh4WsqCsb4fO+p9sSZmG6EM=; b=gQ2Gnb1hUL3npkzn74R+QiZi8Ww/1GTFKMEBR/qB08KZfK199tMMgp5E GIjPr4HogXBG2wv4CL2Hmc7zuLJ1FBxL7PE4ierOhJu4os7WhvbMld/CC YyI4vBw2egTqwWxm3rw3RWMGNH3qTTRUjtx0+dcwIG86GJrY6o2fU2aQE vQCYu8SXllqxK6GRT+ZWfkGHlsk/LA+OVOHTvZIZepIHLKDQsJu4rOX7r iXkFhCShwjAEw+ndoRsSk84HSfjoKr1Wo3VwCV9C2j359NS9Wj3Pa+8LX pge6+eVFgW0xY+YaItRdHky7kqJVjMkMNAm/Ysi8ynK0ZTrkdMcQtlTDz g==; X-CSE-ConnectionGUID: LTG/vPOEQG+sVbPjsO0D2g== X-CSE-MsgGUID: bBY8jFyZQXOAmY+cEU2Ckw== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454556" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454556" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:36 -0700 X-CSE-ConnectionGUID: 2muSIeo1Rn2aIK2qd/9l4w== X-CSE-MsgGUID: cBScj3p6ToCtMmPHlUaIeA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880804" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:35 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 01/19] crypto: acomp - Remove request chaining Date: Thu, 8 May 2025 12:41:16 -0700 Message-Id: <20250508194134.28392-2-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Request chaining requires the user to do too much book keeping. Remove it from acomp. Signed-off-by: Herbert Xu --- crypto/acompress.c | 117 ++++++++-------------------- crypto/scompress.c | 18 +---- include/crypto/acompress.h | 14 ---- include/crypto/internal/acompress.h | 5 -- 4 files changed, 35 insertions(+), 119 deletions(-) diff --git a/crypto/acompress.c b/crypto/acompress.c index f7a3fbe5447e..82fb3c04e68f 100644 --- a/crypto/acompress.c +++ b/crypto/acompress.c @@ -161,7 +161,6 @@ static void acomp_save_req(struct acomp_req *req, crypto_completion_t cplt) state->data = req->base.data; req->base.complete = cplt; req->base.data = state; - state->req0 = req; } static void acomp_restore_req(struct acomp_req *req) @@ -172,23 +171,20 @@ static void acomp_restore_req(struct acomp_req *req) req->base.data = state->data; } -static void acomp_reqchain_virt(struct acomp_req_chain *state, int err) +static void acomp_reqchain_virt(struct acomp_req *req) { - struct acomp_req *req = state->cur; + struct acomp_req_chain *state = &req->chain; unsigned int slen = req->slen; unsigned int dlen = req->dlen; - req->base.err = err; - state = &req->chain; - if (state->flags & CRYPTO_ACOMP_REQ_SRC_VIRT) acomp_request_set_src_dma(req, state->src, slen); else if (state->flags & CRYPTO_ACOMP_REQ_SRC_FOLIO) - acomp_request_set_src_folio(req, state->sfolio, state->soff, slen); + acomp_request_set_src_folio(req, state->sfolio, req->soff, slen); if (state->flags & CRYPTO_ACOMP_REQ_DST_VIRT) acomp_request_set_dst_dma(req, state->dst, dlen); else if (state->flags & CRYPTO_ACOMP_REQ_DST_FOLIO) - acomp_request_set_dst_folio(req, state->dfolio, state->doff, dlen); + acomp_request_set_dst_folio(req, state->dfolio, req->doff, dlen); } static void acomp_virt_to_sg(struct acomp_req *req) @@ -213,7 +209,6 @@ static void acomp_virt_to_sg(struct acomp_req *req) size_t off = req->soff; state->sfolio = folio; - state->soff = off; sg_init_table(&state->ssg, 1); sg_set_page(&state->ssg, folio_page(folio, off / PAGE_SIZE), slen, off % PAGE_SIZE); @@ -233,7 +228,6 @@ static void acomp_virt_to_sg(struct acomp_req *req) size_t off = req->doff; state->dfolio = folio; - state->doff = off; sg_init_table(&state->dsg, 1); sg_set_page(&state->dsg, folio_page(folio, off / PAGE_SIZE), dlen, off % PAGE_SIZE); @@ -241,8 +235,7 @@ static void acomp_virt_to_sg(struct acomp_req *req) } } -static int acomp_do_nondma(struct acomp_req_chain *state, - struct acomp_req *req) +static int acomp_do_nondma(struct acomp_req *req, bool comp) { u32 keep = CRYPTO_ACOMP_REQ_SRC_VIRT | CRYPTO_ACOMP_REQ_SRC_NONDMA | @@ -259,7 +252,7 @@ static int acomp_do_nondma(struct acomp_req_chain *state, fbreq->slen = req->slen; fbreq->dlen = req->dlen; - if (state->op == crypto_acomp_reqtfm(req)->compress) + if (comp) err = crypto_acomp_compress(fbreq); else err = crypto_acomp_decompress(fbreq); @@ -268,114 +261,70 @@ static int acomp_do_nondma(struct acomp_req_chain *state, return err; } -static int acomp_do_one_req(struct acomp_req_chain *state, - struct acomp_req *req) +static int acomp_do_one_req(struct acomp_req *req, bool comp) { - state->cur = req; - if (acomp_request_isnondma(req)) - return acomp_do_nondma(state, req); + return acomp_do_nondma(req, comp); acomp_virt_to_sg(req); - return state->op(req); + return comp ? crypto_acomp_reqtfm(req)->compress(req) : + crypto_acomp_reqtfm(req)->decompress(req); } -static int acomp_reqchain_finish(struct acomp_req *req0, int err, u32 mask) +static int acomp_reqchain_finish(struct acomp_req *req, int err) { - struct acomp_req_chain *state = req0->base.data; - struct acomp_req *req = state->cur; - struct acomp_req *n; - - acomp_reqchain_virt(state, err); - - if (req != req0) - list_add_tail(&req->base.list, &req0->base.list); - - list_for_each_entry_safe(req, n, &state->head, base.list) { - list_del_init(&req->base.list); - - req->base.flags &= mask; - req->base.complete = acomp_reqchain_done; - req->base.data = state; - - err = acomp_do_one_req(state, req); - - if (err == -EINPROGRESS) { - if (!list_empty(&state->head)) - err = -EBUSY; - goto out; - } - - if (err == -EBUSY) - goto out; - - acomp_reqchain_virt(state, err); - list_add_tail(&req->base.list, &req0->base.list); - } - - acomp_restore_req(req0); - -out: + acomp_reqchain_virt(req); + acomp_restore_req(req); return err; } static void acomp_reqchain_done(void *data, int err) { - struct acomp_req_chain *state = data; - crypto_completion_t compl = state->compl; + struct acomp_req *req = data; + crypto_completion_t compl; - data = state->data; + compl = req->chain.compl; + data = req->chain.data; - if (err == -EINPROGRESS) { - if (!list_empty(&state->head)) - return; + if (err == -EINPROGRESS) goto notify; - } - err = acomp_reqchain_finish(state->req0, err, - CRYPTO_TFM_REQ_MAY_BACKLOG); - if (err == -EBUSY) - return; + err = acomp_reqchain_finish(req, err); notify: compl(data, err); } -static int acomp_do_req_chain(struct acomp_req *req, - int (*op)(struct acomp_req *req)) +static int acomp_do_req_chain(struct acomp_req *req, bool comp) { - struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); - struct acomp_req_chain *state; int err; - if (crypto_acomp_req_chain(tfm) || - (!acomp_request_chained(req) && acomp_request_issg(req))) - return op(req); - acomp_save_req(req, acomp_reqchain_done); - state = req->base.data; - state->op = op; - state->src = NULL; - INIT_LIST_HEAD(&state->head); - list_splice_init(&req->base.list, &state->head); - - err = acomp_do_one_req(state, req); + err = acomp_do_one_req(req, comp); if (err == -EBUSY || err == -EINPROGRESS) - return -EBUSY; + return err; - return acomp_reqchain_finish(req, err, ~0); + return acomp_reqchain_finish(req, err); } int crypto_acomp_compress(struct acomp_req *req) { - return acomp_do_req_chain(req, crypto_acomp_reqtfm(req)->compress); + struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); + + if (crypto_acomp_req_chain(tfm) || acomp_request_issg(req)) + crypto_acomp_reqtfm(req)->compress(req); + return acomp_do_req_chain(req, true); } EXPORT_SYMBOL_GPL(crypto_acomp_compress); int crypto_acomp_decompress(struct acomp_req *req) { - return acomp_do_req_chain(req, crypto_acomp_reqtfm(req)->decompress); + struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); + + if (crypto_acomp_req_chain(tfm) || acomp_request_issg(req)) + crypto_acomp_reqtfm(req)->decompress(req); + return acomp_do_req_chain(req, false); } EXPORT_SYMBOL_GPL(crypto_acomp_decompress); diff --git a/crypto/scompress.c b/crypto/scompress.c index 5762fcc63b51..c1ce12564299 100644 --- a/crypto/scompress.c +++ b/crypto/scompress.c @@ -293,28 +293,14 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) return ret; } -static int scomp_acomp_chain(struct acomp_req *req, int dir) -{ - struct acomp_req *r2; - int err; - - err = scomp_acomp_comp_decomp(req, dir); - req->base.err = err; - - list_for_each_entry(r2, &req->base.list, base.list) - r2->base.err = scomp_acomp_comp_decomp(r2, dir); - - return err; -} - static int scomp_acomp_compress(struct acomp_req *req) { - return scomp_acomp_chain(req, 1); + return scomp_acomp_comp_decomp(req, 1); } static int scomp_acomp_decompress(struct acomp_req *req) { - return scomp_acomp_chain(req, 0); + return scomp_acomp_comp_decomp(req, 0); } static void crypto_exit_scomp_ops_async(struct crypto_tfm *tfm) diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index c497c73baf13..267d557daeb1 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -52,10 +52,6 @@ struct acomp_req; struct folio; struct acomp_req_chain { - struct list_head head; - struct acomp_req *req0; - struct acomp_req *cur; - int (*op)(struct acomp_req *req); crypto_completion_t compl; void *data; struct scatterlist ssg; @@ -68,8 +64,6 @@ struct acomp_req_chain { u8 *dst; struct folio *dfolio; }; - size_t soff; - size_t doff; u32 flags; }; @@ -349,8 +343,6 @@ static inline void acomp_request_set_callback(struct acomp_req *req, req->base.data = data; req->base.flags &= keep; req->base.flags |= flgs & ~keep; - - crypto_reqchain_init(&req->base); } /** @@ -558,12 +550,6 @@ static inline void acomp_request_set_dst_folio(struct acomp_req *req, req->base.flags |= CRYPTO_ACOMP_REQ_DST_FOLIO; } -static inline void acomp_request_chain(struct acomp_req *req, - struct acomp_req *head) -{ - crypto_request_chain(&req->base, &head->base); -} - /** * crypto_acomp_compress() -- Invoke asynchronous compress operation * diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h index aaf59f3236fa..b69d818d7e68 100644 --- a/include/crypto/internal/acompress.h +++ b/include/crypto/internal/acompress.h @@ -98,11 +98,6 @@ void crypto_unregister_acomp(struct acomp_alg *alg); int crypto_register_acomps(struct acomp_alg *algs, int count); void crypto_unregister_acomps(struct acomp_alg *algs, int count); -static inline bool acomp_request_chained(struct acomp_req *req) -{ - return crypto_request_chained(&req->base); -} - static inline bool acomp_request_issg(struct acomp_req *req) { return !(req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT | From patchwork Thu May 8 19:41:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888696 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4EF928B3FF; Thu, 8 May 2025 19:41:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733302; cv=none; b=Eljn2kUXkDn+411lZEL/wgOxOZBkcYR0WMOy5F+gXARD1aHftNiX95j0Sa+db3nt6B3ykA1i1c9mGZ5xOXptGubNS3GImKPwVi7TBhzGBGUssV8DYP++xp7c1vU3sBfAjzQq49NFWCJuASaV1456gna4sWB2JhBezgaJrxv0clM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733302; c=relaxed/simple; bh=6wbRhOV0GFWlmqPXQS6DTk+b5GGSil7xU+GDXQbyLR0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=smj0MugZ1kY/CwOLlriasr0MDY9FyORs2eTit7wu17bQIidwCZnuFzsyhEfJzlj8sSD2RWuRJEK5NlxZyGwZfzAcYlKY28Y1HR2oQ9a24qVw+q51mEgRe0XzhCImnWhSRvKB/IxHwBcbaTAyaFMYEIcRotPHmq2d91vV6vQIfmg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=G7cAgXfv; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="G7cAgXfv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733301; x=1778269301; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6wbRhOV0GFWlmqPXQS6DTk+b5GGSil7xU+GDXQbyLR0=; b=G7cAgXfvha+wZliockLplGlKJ0UKtBNeRm+rduUW/JW/3orMqCSkbhzL hAUFCFTHG/HyAMVQ5LiD+5amibosvNlT9rOh1T4y5P90N71g8bNh3v5Dp hBgRu9c0LsZvyNioKmgJ7qbGXxmRFppNO1sy1bVcRl9awWLW3fJUEjvyT UpEzKlEb7Zl882ou6I6b5IRvSX6aQKyOR3kKai2C6yxccmxNMLP3pyL8t 0Hb2nXzIjTcwMf7RZ8H0m050AxrklGhKULl6B8kOyd7r47nMRnNyMa+pc 8xgNBq3qkegqOuY7xHgeSil7fBLdJW8MV9EjOqNztnPpxNkAHR2lK5EFd g==; X-CSE-ConnectionGUID: VPHJ6jOeR+W8kHjqJtbEmw== X-CSE-MsgGUID: a6cMcUVQQ7mROxl1mN7/uQ== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454570" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454570" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:36 -0700 X-CSE-ConnectionGUID: 7MUR0bThTVCKOt+PlSoAVg== X-CSE-MsgGUID: iEe5hw7uS4+ZnqZrvIKNDQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880808" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:35 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 02/19] crypto: acomp - Reinstate non-chained crypto_acomp_[de]compress(). Date: Thu, 8 May 2025 12:41:17 -0700 Message-Id: <20250508194134.28392-3-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This reverts the request chaining implementations of crypto_acomp_[de]compress() introduced in commit b67a02600372 ("crypto: acomp - Add request chaining and virtual addresses") since request chaining has been removed from acomp subsequently in commit 64929fe8c0a4 ("crypto: acomp - Remove request chaining"). This patch restores the implementations of crypto_acomp_[de]compress() from prior to commit b67a02600372. Signed-off-by: Kanchana P Sridhar --- crypto/acompress.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/crypto/acompress.c b/crypto/acompress.c index 82fb3c04e68f..d08e0fe8cd9e 100644 --- a/crypto/acompress.c +++ b/crypto/acompress.c @@ -310,21 +310,13 @@ static int acomp_do_req_chain(struct acomp_req *req, bool comp) int crypto_acomp_compress(struct acomp_req *req) { - struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); - - if (crypto_acomp_req_chain(tfm) || acomp_request_issg(req)) - crypto_acomp_reqtfm(req)->compress(req); - return acomp_do_req_chain(req, true); + return crypto_acomp_reqtfm(req)->compress(req); } EXPORT_SYMBOL_GPL(crypto_acomp_compress); int crypto_acomp_decompress(struct acomp_req *req) { - struct crypto_acomp *tfm = crypto_acomp_reqtfm(req); - - if (crypto_acomp_req_chain(tfm) || acomp_request_issg(req)) - crypto_acomp_reqtfm(req)->decompress(req); - return acomp_do_req_chain(req, false); + return crypto_acomp_reqtfm(req)->decompress(req); } EXPORT_SYMBOL_GPL(crypto_acomp_decompress); From patchwork Thu May 8 19:41:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888697 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4123628A729; Thu, 8 May 2025 19:41:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733301; cv=none; b=o/Jm7/V8l4xvf3Fb+c7i2X/U0dR9xHnfPUxqweHuzYn5ihHv7uJBS3pg0K7cRcEeAQcvHkVDIOL+/pUB/4sA8xXhvOYeqsU7Y6ribuFyT/xfeSazYMFsw9siOFb7ygGVtL9/Uxdrbvb4iWBNp0/ICcEf2cNKdUk/4HuhOyhJT9k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733301; c=relaxed/simple; bh=sJFxWyA6AJF6CcAZ5xFoncDGNwqA2S5ppE8eb6ZDAWI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dbKVDSLGbVtTZw29covT5i/MBkvjsKOtOGgXnZ/i339LbC38AkuhP33JEF89jnBzzXZ7V+GeCDqPerzuU5IqKRvtm0b/VZC6ee1USh1NVvA6KpbaNKjb/nREWX2jnkTFZR+yV807amRoPKfH6YBqqIYHkvgHZ2SuywVIfrHSA4o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=i/3txU3q; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="i/3txU3q" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733300; x=1778269300; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sJFxWyA6AJF6CcAZ5xFoncDGNwqA2S5ppE8eb6ZDAWI=; b=i/3txU3q9C6LrLGFntYsu+aEUrCaRWkLOxbNkfESAmvV9WQ0EWRQyEcI XPpdES2NAaseHgbJvE7T+5RdDseXJkSavk0OvgfU42QY54QhIY+TgLs3S Bp6pFwuTvTwKcQUokx6dQXkMdMxemVEqyUhz+P3XFoOCKQnjshdf/VzVJ crxz+RfLP/tdN03LCxG2vXpa3NHN24MAlyz1Bzasvw0Hzcrvjm2gr3PdW kD9XFH/owemMroZVn22yg9kT41EytV7a22uGg9HTINY7gUKNjKj9dXeoW D5vKj2l2oAZOoZ+BNhQZTnGFp7Dn7LijsA0J/Z0L6QlhEF8EOsZyb9yuW g==; X-CSE-ConnectionGUID: 50Z/c8HGQvGlk5nEEYEaDA== X-CSE-MsgGUID: jYKLDgceRv2X1L09zlHGUA== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454603" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454603" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:38 -0700 X-CSE-ConnectionGUID: LAb/b0l+REKDfRVJBNReeQ== X-CSE-MsgGUID: CFUP26gyRf6tnxsLmGwIbQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880829" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:37 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 04/19] crypto: scomp - Fix off-by-one bug when calculating last page Date: Thu, 8 May 2025 12:41:19 -0700 Message-Id: <20250508194134.28392-5-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Fix off-by-one bug in the last page calculation for src and dst. Reported-by: Nhat Pham Fixes: 2d3553ecb4e3 ("crypto: scomp - Remove support for some non-trivial SG lists") Signed-off-by: Herbert Xu --- crypto/scompress.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/crypto/scompress.c b/crypto/scompress.c index c1ce12564299..1ed52b9740c5 100644 --- a/crypto/scompress.c +++ b/crypto/scompress.c @@ -215,8 +215,8 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) spage = nth_page(spage, soff / PAGE_SIZE); soff = offset_in_page(soff); - n = slen / PAGE_SIZE; - n += (offset_in_page(slen) + soff - 1) / PAGE_SIZE; + n = (slen - 1) / PAGE_SIZE; + n += (offset_in_page(slen - 1) + soff) / PAGE_SIZE; if (PageHighMem(nth_page(spage, n)) && size_add(soff, slen) > PAGE_SIZE) break; @@ -243,9 +243,9 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) dpage = nth_page(dpage, doff / PAGE_SIZE); doff = offset_in_page(doff); - n = dlen / PAGE_SIZE; - n += (offset_in_page(dlen) + doff - 1) / PAGE_SIZE; - if (PageHighMem(dpage + n) && + n = (dlen - 1) / PAGE_SIZE; + n += (offset_in_page(dlen - 1) + doff) / PAGE_SIZE; + if (PageHighMem(nth_page(dpage, n)) && size_add(doff, dlen) > PAGE_SIZE) break; dst = kmap_local_page(dpage) + doff; From patchwork Thu May 8 19:41:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888695 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABA4A28BA9F; Thu, 8 May 2025 19:41:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733304; cv=none; b=PoZNGhzYqp5J+kxdQa3UBn9ssuKJqOqzNZV5p5U4mCO+0qNJAjNa0yfXX0ZrQ7CGqHA10pGfAjCP6v9in8Ncnkq45QGmR7ZusRgLeYceaubJNybDmxZtryLETT53MYkbObxHHUCznQG0ayK7GehpOHwVyHK9ItNyBQ/vrlt2beU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733304; c=relaxed/simple; bh=WKyBT52U1L3mEjci9iHKwXn3qQ9NOODMueb2KPC9q4M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=leV1X8X65ckQLbSoXRxF2SVWxP03zreHiwHBg8CA8ATCxWTwIXmWt0W4Uw/Julv/tWCjJyvMtA/aICb2JHL8mL/zBMQfZVQQCbjIK+MItM3mm/74bvWhuwVbuy30g0UETnHJmYLrLsUw63Ea5Qh1fW2SwX0E7aXaY4TPn1sd6mQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=fonUzAIx; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="fonUzAIx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733303; x=1778269303; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WKyBT52U1L3mEjci9iHKwXn3qQ9NOODMueb2KPC9q4M=; b=fonUzAIx+umAU871/jhnh10aTCI+5vB4+l1AiUR6ZC95KKZ5ifqKm9GK 0b3M8psn6lMGsJ+4OsgowsD2jADZ6cHjb/RkiBTnsIHjso8y280RCNEtX D3R8OZUk0BwpK9C6sE5YyxsI//tPhbGpnLWO59otLroeuSpxT05p5KOhx cVDWImLacBNxsEvBFlDDVBiyDhw8m4js8OA1yUo1TPZ2Zwa0zBqdRr1vy foB511m3LeKu0e/FKCDRuvHC+VMrdsCcNISCVVzoZvOhItiRnyI2kyFId kF7RQpowq9q0KSWWY5IUvRPF0vi7mZVm0YeJyr6y5+zf53RCWuF/EQalo w==; X-CSE-ConnectionGUID: GdD20rMwQ3CPyF2AnQDvPQ== X-CSE-MsgGUID: 6XvzeoN0QtenjoMVC+wCLw== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454657" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454657" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:40 -0700 X-CSE-ConnectionGUID: NmiaWf8zRE+xGxVwN4rSJA== X-CSE-MsgGUID: tagljZ4IRdSbiyzYXTkNlw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880851" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:39 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 07/19] crypto: iaa - Define and use req->data instead of req->base.data. Date: Thu, 8 May 2025 12:41:22 -0700 Message-Id: <20250508194134.28392-8-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Since req->base.data is for the user and not for the driver, we define a "void *data" in struct acomp_req for use by driver code. At present, iaa_crypto saves the "struct idxd_desc *idxd_desc" that is allocated in iaa_[de]compress(), in req->data. When batching is introduced in subsequent patches, we will need to support an async "submit-poll" mechanism to achieve parallelism using IAA hardware. To accomplish this, we will submit the descriptors for each request in the batch in iaa_[de]compress(), and return -EINPROGRESS. The polling function will retrieve the descriptor from req->data to check the request's completion status. Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 12 +++++++----- include/crypto/acompress.h | 2 ++ 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 2f2dc6987cc6..0b821b8b4264 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1718,7 +1718,7 @@ static void iaa_desc_complete(struct idxd_desc *idxd_desc, iaa_wq_put(idxd_desc->wq); } -static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, +static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_wq *wq, dma_addr_t src_addr, unsigned int slen, dma_addr_t dst_addr, unsigned int *dlen, @@ -1778,8 +1778,9 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, " src_addr %llx, dst_addr %llx\n", __func__, active_compression_mode->name, src_addr, dst_addr); - } else if (ctx->async_mode) - req->base.data = idxd_desc; + } else if (ctx->async_mode) { + req->data = idxd_desc; + } dev_dbg(dev, "%s: compression mode %s," " desc->src1_addr %llx, desc->src1_size %d," @@ -1889,8 +1890,9 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, " src_addr %llx, dst_addr %llx\n", __func__, active_compression_mode->name, src_addr, dst_addr); - } else if (ctx->async_mode && !disable_async) - req->base.data = idxd_desc; + } else if (ctx->async_mode && !disable_async) { + req->data = idxd_desc; + } dev_dbg(dev, "%s: decompression mode %s," " desc->src1_addr %llx, desc->src1_size %d," diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index 267d557daeb1..01389fd7055f 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -81,6 +81,7 @@ struct acomp_req_chain { * @doff: Destination folio offset * @slen: Size of the input buffer * @dlen: Size of the output buffer and number of bytes produced + * @data: Private API code data, do not use * @chain: Private API code data, do not use * @__ctx: Start of private context data */ @@ -101,6 +102,7 @@ struct acomp_req { unsigned int slen; unsigned int dlen; + void *data; struct acomp_req_chain chain; void *__ctx[] CRYPTO_MINALIGN_ATTR; From patchwork Thu May 8 19:41:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888693 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1B9B28C5A5; Thu, 8 May 2025 19:41:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733308; cv=none; b=m1EUEeuZw6STRrhxiVVk5Yv9FkID/cHG7M88h/B5SMKV780p9yOW/8LEOd742NEox5cdYrjnGlYbnXr33/CTuHwI+2crnZGebdAVcRqOksTH2z54O2/Yx7TqDCyx46MFpF8tPssqCIZYeKpqz5RNqns1xbEzXeAdmENgs7OVGz4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733308; c=relaxed/simple; bh=+XwXzM2JtnnapecA98UfJihsUQ5zhZpmnmCn8je5v3o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t+iRhe1JqHvEtNg4NvCiyxMSGoYThNCpfNKwLOW1U5HqgmIGdnAlw2rrHX0Gm0jGupiaHnn14kWnqaMEA/s0O6qjWOGb6bbko0ZfAIfzYiccQko1kALqLDS79EsTzB//pHyGJx17Vo+PKF2TEygHAV2/bFqTWAXf+xQLyZS2exg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Wgz7OsDE; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Wgz7OsDE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733305; x=1778269305; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+XwXzM2JtnnapecA98UfJihsUQ5zhZpmnmCn8je5v3o=; b=Wgz7OsDEyWDXYG39NkP0Wjjc+DV5UOAYYYUh1A7CDe2CxkisKzx2qw7F A9EjsF9E6vTUUEAMolaPMMt+iZ9DX7RaDu4YtnOoOD4DBlIyw5G32UTyz dJ2KmBz3Tc46HbELhfrh6QADLcREVLWlc9x1wdFl57fPVhj9Zl81n/3Fi ZW6YC9ofKrn3WdWWnuQ579As0aLtdGUGxo2KPu0NIH6UQf9EBjulzeGcZ wP7qvWBDYcIm45VmHuP0YxRyOtmjqjLjMN+ScvPg7nurGEKfbwjVAt+4/ 75rCy+iPcBtdaJYU7vhFv6VLcn9O0epuh7L3da+cOV0sT8L2t1TUeyz+E g==; X-CSE-ConnectionGUID: YfSylAHAQ/+5Q7vc0Gh7gw== X-CSE-MsgGUID: 1+qxDSloSriKKwP9W8MXVQ== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454671" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454671" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:40 -0700 X-CSE-ConnectionGUID: XudDAXLOQoe5R/ppOk4syA== X-CSE-MsgGUID: +pgws5TQSj6wgmnR1IHgvA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880854" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:39 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 08/19] crypto: iaa - Descriptor allocation timeouts with mitigations in iaa_crypto. Date: Thu, 8 May 2025 12:41:23 -0700 Message-Id: <20250508194134.28392-9-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch modifies the descriptor allocation from blocking to non-blocking with bounded retries or "timeouts". This is necessary to prevent task blocked errors in high contention scenarios, for instance, when the platform has only 1 IAA device enabled. With 1 IAA device enabled per package on a dual-package Sapphire Rapids with 56 cores/package, there are 112 logical cores mapped to this single IAA device. In this scenario, the task blocked errors can occur because idxd_alloc_desc() is called with IDXD_OP_BLOCK. With batching, multiple descriptors will need to be allocated per batch. Any process that is able to do so, can cause contention for allocating descriptors for all other processes that share the use of the same sbitmap_queue. Under IDXD_OP_BLOCK, this can cause compress/decompress jobs to stall in stress test scenarios (e.g. zswap_store() of 2M folios). In order to make the iaa_crypto driver be more fail-safe, this commit implements the following: 1) Change compress/decompress descriptor allocations to be non-blocking with retries ("timeouts"). 2) Return compress error to zswap if descriptor allocation with timeouts fails during compress ops. zswap_store() will return an error and the folio gets stored in the backing swap device. 3) Fallback to software decompress if descriptor allocation with timeouts fails during decompress ops. With these fixes, there are no task blocked errors seen under stress testing conditions, and no performance degradation observed. This patch also simplifies the success/error return paths in iaa_[de]compress() and iaa_compress_verify(). Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto.h | 3 + drivers/crypto/intel/iaa/iaa_crypto_main.c | 84 ++++++++++++---------- 2 files changed, 48 insertions(+), 39 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h index 549ac98a9366..b4a94da2c315 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto.h +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -21,6 +21,9 @@ #define IAA_COMPLETION_TIMEOUT 1000000 +#define IAA_ALLOC_DESC_COMP_TIMEOUT 1000 +#define IAA_ALLOC_DESC_DECOMP_TIMEOUT 500 + #define IAA_ANALYTICS_ERROR 0x0a #define IAA_ERROR_DECOMP_BUF_OVERFLOW 0x0b #define IAA_ERROR_COMP_BUF_OVERFLOW 0x19 diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 0b821b8b4264..7dab340c4a34 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1416,6 +1416,7 @@ static int deflate_generic_decompress(struct acomp_req *req) ACOMP_REQUEST_ON_STACK(fbreq, crypto_acomp_reqtfm(req)); int ret; + req->dlen = PAGE_SIZE; acomp_request_set_callback(fbreq, 0, NULL, NULL); acomp_request_set_params(fbreq, req->src, req->dst, req->slen, req->dlen); @@ -1536,7 +1537,8 @@ static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *ctx = crypto_tfm_ctx(tfm); struct iaa_device *iaa_device; - struct idxd_desc *idxd_desc; + struct idxd_desc *idxd_desc = ERR_PTR(-EAGAIN); + u16 alloc_desc_retries = 0; struct iax_hw_desc *desc; struct idxd_device *idxd; struct iaa_wq *iaa_wq; @@ -1552,7 +1554,11 @@ static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, active_compression_mode = get_iaa_device_compression_mode(iaa_device, ctx->mode); - idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + while ((idxd_desc == ERR_PTR(-EAGAIN)) && (alloc_desc_retries++ < IAA_ALLOC_DESC_DECOMP_TIMEOUT)) { + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK); + cpu_relax(); + } + if (IS_ERR(idxd_desc)) { dev_dbg(dev, "idxd descriptor allocation failed\n"); dev_dbg(dev, "iaa compress failed: ret=%ld\n", @@ -1604,14 +1610,10 @@ static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, goto err; } - idxd_free_desc(wq, idxd_desc); -out: - return ret; err: idxd_free_desc(wq, idxd_desc); - dev_dbg(dev, "iaa compress failed: ret=%d\n", ret); - goto out; + return ret; } static void iaa_desc_complete(struct idxd_desc *idxd_desc, @@ -1727,7 +1729,8 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *ctx = crypto_tfm_ctx(tfm); struct iaa_device *iaa_device; - struct idxd_desc *idxd_desc; + struct idxd_desc *idxd_desc = ERR_PTR(-EAGAIN); + u16 alloc_desc_retries = 0; struct iax_hw_desc *desc; struct idxd_device *idxd; struct iaa_wq *iaa_wq; @@ -1743,7 +1746,11 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, active_compression_mode = get_iaa_device_compression_mode(iaa_device, ctx->mode); - idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + while ((idxd_desc == ERR_PTR(-EAGAIN)) && (alloc_desc_retries++ < IAA_ALLOC_DESC_COMP_TIMEOUT)) { + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK); + cpu_relax(); + } + if (IS_ERR(idxd_desc)) { dev_dbg(dev, "idxd descriptor allocation failed\n"); dev_dbg(dev, "iaa compress failed: ret=%ld\n", PTR_ERR(idxd_desc)); @@ -1820,15 +1827,10 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, *compression_crc = idxd_desc->iax_completion->crc; - if (!ctx->async_mode) - idxd_free_desc(wq, idxd_desc); -out: - return ret; err: idxd_free_desc(wq, idxd_desc); - dev_dbg(dev, "iaa compress failed: ret=%d\n", ret); - - goto out; +out: + return ret; } static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, @@ -1840,7 +1842,8 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *ctx = crypto_tfm_ctx(tfm); struct iaa_device *iaa_device; - struct idxd_desc *idxd_desc; + struct idxd_desc *idxd_desc = ERR_PTR(-EAGAIN); + u16 alloc_desc_retries = 0; struct iax_hw_desc *desc; struct idxd_device *idxd; struct iaa_wq *iaa_wq; @@ -1856,12 +1859,18 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, active_compression_mode = get_iaa_device_compression_mode(iaa_device, ctx->mode); - idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + while ((idxd_desc == ERR_PTR(-EAGAIN)) && (alloc_desc_retries++ < IAA_ALLOC_DESC_DECOMP_TIMEOUT)) { + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK); + cpu_relax(); + } + if (IS_ERR(idxd_desc)) { dev_dbg(dev, "idxd descriptor allocation failed\n"); dev_dbg(dev, "iaa decompress failed: ret=%ld\n", PTR_ERR(idxd_desc)); - return PTR_ERR(idxd_desc); + ret = PTR_ERR(idxd_desc); + idxd_desc = NULL; + goto fallback_software_decomp; } desc = idxd_desc->iax_hw; @@ -1905,7 +1914,7 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, ret = idxd_submit_desc(wq, idxd_desc); if (ret) { dev_dbg(dev, "submit_desc failed ret=%d\n", ret); - goto err; + goto fallback_software_decomp; } /* Update stats */ @@ -1919,40 +1928,37 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, } ret = check_completion(dev, idxd_desc->iax_completion, false, false); + +fallback_software_decomp: if (ret) { - dev_dbg(dev, "%s: check_completion failed ret=%d\n", __func__, ret); - if (idxd_desc->iax_completion->status == IAA_ANALYTICS_ERROR) { + dev_dbg(dev, "%s: desc allocation/submission/check_completion failed ret=%d\n", __func__, ret); + if (idxd_desc && idxd_desc->iax_completion->status == IAA_ANALYTICS_ERROR) { pr_warn("%s: falling back to deflate-generic decompress, " "analytics error code %x\n", __func__, idxd_desc->iax_completion->error_code); - ret = deflate_generic_decompress(req); - if (ret) { - dev_dbg(dev, "%s: deflate-generic failed ret=%d\n", - __func__, ret); - goto err; - } - } else { + } + + ret = deflate_generic_decompress(req); + + if (ret) { + pr_err("%s: iaa decompress failed: deflate-generic fallback to software decompress error ret=%d\n", __func__, ret); goto err; } } else { req->dlen = idxd_desc->iax_completion->output_size; + + /* Update stats */ + update_total_decomp_bytes_in(slen); + update_wq_decomp_bytes(wq, slen); } *dlen = req->dlen; - if (!ctx->async_mode || disable_async) +err: + if (idxd_desc) idxd_free_desc(wq, idxd_desc); - - /* Update stats */ - update_total_decomp_bytes_in(slen); - update_wq_decomp_bytes(wq, slen); out: return ret; -err: - idxd_free_desc(wq, idxd_desc); - dev_dbg(dev, "iaa decompress failed: ret=%d\n", ret); - - goto out; } static int iaa_comp_acompress(struct acomp_req *req) From patchwork Thu May 8 19:41:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888694 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1C2C28C840; Thu, 8 May 2025 19:41:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733306; cv=none; b=V/R7mFoI0MOn+KamemyZfltxYKLXHRw1O5Z/4aDkN0K/l/zVMTcbMKVDhNAqH6heFUiYYKo75dK9NyDshslrkJGBurgszEHmRtIf3VIyjpD9deVXFGvm2l/pCc5+gTdWVLMf6nD7etogRSfmiDt5CUotK9wovArCJaimnjQ4Yvo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733306; c=relaxed/simple; bh=ZmGjviUpJ+c+Is95YFL01k3D2zJB4AA9yt94lFpXChk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WS5ISc1+qaU2m0HeL88pTbx+ToPy6TJoRsCt0uAFMNh/yN1ykzJCc0sxyRgJZBEIRFcC7EflTIGTX/pClZJUiCBG0IqVe3pDhE2+OSDS8yDSddiLldxKU0yB4m86L9dkdFYrnzzeyapcDg2lJv5CJBPnyfu1rY0wOBkH6skTSmw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=noBU2dtt; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="noBU2dtt" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733305; x=1778269305; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZmGjviUpJ+c+Is95YFL01k3D2zJB4AA9yt94lFpXChk=; b=noBU2dttrDLSoq+yrUys3gK0xtpiXhBSVTDs9kIpJtj7Bnf+N60kiAJH 5o/iAVwSglRZsU0jdXiIv50+hWjdurl4d9mjZgxJow8G56lq9LfF4Tgfz JkSUcXAvVlR99I6hSgn+xI0WT4wAaHqqoPeq7qwlbHxB8FBpJny5NXhp5 JmDBwIX/NqrpgpLseyC/fWFn7SVqD50w3ukOZD+Zgc+LXgjl7vWt9rX8M tSTCwA3S7rm9XcSOanPt42lyuq4hwJxlWH7G2rnxQGtDvDZ2gKOxxd/g9 W65MG3SOIp3Kwx4wAECOcDliNu74IvEJjUq3Uiy9PG6u/+qMwPad+SObL A==; X-CSE-ConnectionGUID: mNJm+4sSQIe1iXXyDaMzeA== X-CSE-MsgGUID: QfR7oW8aTNCyUnmhnbTmaA== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454685" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454685" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:41 -0700 X-CSE-ConnectionGUID: ldnIJyYrSDuSrtxXPYKUDw== X-CSE-MsgGUID: Er7M6ZiTQe+dALHn2pB2wQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880857" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:40 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 09/19] crypto: iaa - CRYPTO_ACOMP_REQ_POLL acomp_req flag for sequential vs. parallel. Date: Thu, 8 May 2025 12:41:24 -0700 Message-Id: <20250508194134.28392-10-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The purpose of this commit is to allow kernel users of iaa_crypto, such as zswap, to be able to invoke the crypto_acomp_compress() API in fully synchronous mode for non-batching use cases (i.e. today's status-quo), where zswap calls crypto_wait_req(crypto_acomp_compress(req), wait); and to non-instrusively invoke the fully asynchronous batch compress/decompress API that will be introduced in subsequent patches. Both use cases need to reuse same code paths in the driver to interface with hardware: the CRYPTO_ACOMP_REQ_POLL flag allows this shared code to determine whether we need to process an acomp_req synchronously/asynchronously. The idea is to simplify the crypto_acomp sequential/batching interfaces for use by zswap. Thus, regardless of the iaa_crypto driver's 'sync_mode' setting, it can still be forced to use synchronous mode by turning off the CRYPTO_ACOMP_REQ_POLL flag in req->base.flags (the default to support sequential use cases in zswap today). IAA batching functionality will be implemented in subsequent patches, that will set the CRYPTO_ACOMP_REQ_POLL flag for the acomp_reqs in a batch. This enables the iaa_crypto driver to implement true async "submit-polling" for parallel compressions and decompressions in the IAA hardware accelerator. In other words, all three of the following need to be true for a request to be processed in fully async submit-poll mode: 1) async_mode should be "true" 2) use_irq should be "false" 3) req->base.flags & CRYPTO_ACOMP_REQ_POLL should be "true" Subsequent patches will: - Set (1) and (2) as iaa_crypto defaults once async submit-poll is implemented. - Enable (3) for iaa_crypto batching, and clear the CRYPTO_ACOMP_REQ_POLL flags before exiting from the batching routines since the assumption is that the acomp_reqs are created/managed by a higher level kernel user such as zswap, and are reused for both, sequential and batching use cases from zswap's perspective. This patch also removes "disable_async" from iaa_decompress(). Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 15 +++++++-------- include/crypto/acompress.h | 6 ++++++ 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 7dab340c4a34..52fe68606f4d 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1785,7 +1785,7 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, " src_addr %llx, dst_addr %llx\n", __func__, active_compression_mode->name, src_addr, dst_addr); - } else if (ctx->async_mode) { + } else if (ctx->async_mode && (req->base.flags & CRYPTO_ACOMP_REQ_POLL)) { req->data = idxd_desc; } @@ -1807,7 +1807,7 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, update_total_comp_calls(); update_wq_comp_calls(wq); - if (ctx->async_mode) { + if (ctx->async_mode && (req->base.flags & CRYPTO_ACOMP_REQ_POLL)) { ret = -EINPROGRESS; dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); goto out; @@ -1836,8 +1836,7 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_wq *wq, dma_addr_t src_addr, unsigned int slen, - dma_addr_t dst_addr, unsigned int *dlen, - bool disable_async) + dma_addr_t dst_addr, unsigned int *dlen) { struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *ctx = crypto_tfm_ctx(tfm); @@ -1886,7 +1885,7 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, desc->src1_size = slen; desc->completion_addr = idxd_desc->compl_dma; - if (ctx->use_irq && !disable_async) { + if (ctx->use_irq) { desc->flags |= IDXD_OP_FLAG_RCI; idxd_desc->crypto.req = req; @@ -1899,7 +1898,7 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, " src_addr %llx, dst_addr %llx\n", __func__, active_compression_mode->name, src_addr, dst_addr); - } else if (ctx->async_mode && !disable_async) { + } else if (ctx->async_mode && (req->base.flags & CRYPTO_ACOMP_REQ_POLL)) { req->data = idxd_desc; } @@ -1921,7 +1920,7 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, update_total_decomp_calls(); update_wq_decomp_calls(wq); - if (ctx->async_mode && !disable_async) { + if (ctx->async_mode && (req->base.flags & CRYPTO_ACOMP_REQ_POLL)) { ret = -EINPROGRESS; dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); goto out; @@ -2127,7 +2126,7 @@ static int iaa_comp_adecompress(struct acomp_req *req) req->dst, req->dlen, sg_dma_len(req->dst)); ret = iaa_decompress(tfm, req, wq, src_addr, req->slen, - dst_addr, &req->dlen, false); + dst_addr, &req->dlen); if (ret == -EINPROGRESS) return ret; diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index 01389fd7055f..939e51d122b0 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -20,6 +20,12 @@ #include #include +/* + * If set, the driver must have a way to submit the req, then + * poll its completion status for success/error. + */ +#define CRYPTO_ACOMP_REQ_POLL 0x00000001 + /* Set this bit if source is virtual address instead of SG list. */ #define CRYPTO_ACOMP_REQ_SRC_VIRT 0x00000002 From patchwork Thu May 8 19:41:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888692 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A93028CF67; Thu, 8 May 2025 19:41:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733309; cv=none; b=gVq9YNA+fpzKeeajb8MLhI4LlUq6l5rxEFTzot4TqJ9B+lnsiZbBKN1mAGKX1Ef01avdXz+bMaLl9G+q3jRSjTkqtxW/Bc8nkegWtToQXuB7wcQz3mTuAsIBYJRvh6hEHLxfQsjnEm3XXvod0AICp6rTtHExiFyvlrhfU7VniWY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733309; c=relaxed/simple; bh=zk8J++ATRu4+E5dh9QqEjlY39chTwGmrGHh2DWD3/3c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Vqz04m2tgDiG5fkYhBpao3VCGg1yt6tHugRHUondTtnB+Vnf81zhtjVJ+08gHnr6ytwKDycBDmNCMob7z9urstVYhoW9W4nSrLd/SyWefFCUjVq4AMGvAjFkT82eCiChKp+kLY9pnDOaKcdU7uKIEShUGTjROBOipiBBh1U+aUM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=cw8Wr4lF; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="cw8Wr4lF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733308; x=1778269308; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zk8J++ATRu4+E5dh9QqEjlY39chTwGmrGHh2DWD3/3c=; b=cw8Wr4lFtKW9KXWre9AsKM0UdNZvjDgFLVOdlAubbXvUDOzoMFCIy4mH uIFbKHn1SRF04XlB6iYD+ib6ld6WD27JQgWSqb+DzD7lRkUdoiCmIJ4GG s8etx8cN5Qe4i0VDQEWEjRV4WM/UlX7wA6q5iST1LK/LXD0N/nvcKHac2 9uSMkAPniHcZHqlb201IhZtVWNy2l3S1+4ReyerMrPRUqKoaThRLMM0cc 7kFi3zmXtn/OiGZ65+hCgRZ+L4SflVjdMXtGTqWZ28ZZfMxW24rQvCByq J4OaiIQzajy+J4i9K7qrTskzXcKZcQpvToGJolryvl4MbMCobbQ/ZUNE2 g==; X-CSE-ConnectionGUID: MTQElj6DQU2xplp1O3Z7og== X-CSE-MsgGUID: 6PEbxrArR5aJ1f8bmqMq/Q== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454729" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454729" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:43 -0700 X-CSE-ConnectionGUID: tI5CF4s/RR2uTvPat/iG5Q== X-CSE-MsgGUID: wwdVPb4rQnyFdInDQ0eDQw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880869" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:42 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 12/19] crypto: iaa - Enable async mode and make it the default. Date: Thu, 8 May 2025 12:41:27 -0700 Message-Id: <20250508194134.28392-13-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch enables the 'async' sync_mode in the driver. Further, it sets the default sync_mode to 'async', which makes it easier for IAA hardware acceleration in the iaa_crypto driver to be loaded by default in the most efficient/recommended 'async' mode for parallel compressions/decompressions, namely, asynchronous submission of descriptors, followed by polling for job completions. Earlier, the "sync" mode used to be the default. The iaa_crypto driver documentation has been updated with these changes. This way, anyone who wants to use IAA for zswap/zram can do so after building the kernel, and without having to go through these steps to use async mode: 1) disable all the IAA device/wq bindings that happen at boot time 2) rmmod iaa_crypto 3) modprobe iaa_crypto 4) echo async > /sys/bus/dsa/drivers/crypto/sync_mode 5) re-run initialization of the IAA devices and wqs Signed-off-by: Kanchana P Sridhar --- Documentation/driver-api/crypto/iaa/iaa-crypto.rst | 11 ++--------- drivers/crypto/intel/iaa/iaa_crypto_main.c | 4 ++-- 2 files changed, 4 insertions(+), 11 deletions(-) diff --git a/Documentation/driver-api/crypto/iaa/iaa-crypto.rst b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst index 949bfa1ef624..8e0e98d50972 100644 --- a/Documentation/driver-api/crypto/iaa/iaa-crypto.rst +++ b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst @@ -272,7 +272,7 @@ The available attributes are: echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode Async mode without interrupts (caller must poll) can be enabled by - writing 'async' to it (please see Caveat):: + writing 'async' to it:: echo async > /sys/bus/dsa/drivers/crypto/sync_mode @@ -281,14 +281,7 @@ The available attributes are: echo sync > /sys/bus/dsa/drivers/crypto/sync_mode - The default mode is 'sync'. - - Caveat: since the only mechanism that iaa_crypto currently implements - for async polling without interrupts is via the 'sync' mode as - described earlier, writing 'async' to - '/sys/bus/dsa/drivers/crypto/sync_mode' will internally enable the - 'sync' mode. This is to ensure correct iaa_crypto behavior until true - async polling without interrupts is enabled in iaa_crypto. + The default mode is 'async'. - g_comp_wqs_per_iaa diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index d577f555d6ab..cfd4f5ead67b 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -115,7 +115,7 @@ static bool iaa_verify_compress = true; */ /* Use async mode */ -static bool async_mode; +static bool async_mode = true; /* Use interrupts */ static bool use_irq; @@ -169,7 +169,7 @@ static int set_iaa_sync_mode(const char *name) async_mode = false; use_irq = false; } else if (sysfs_streq(name, "async")) { - async_mode = false; + async_mode = true; use_irq = false; } else if (sysfs_streq(name, "async_irq")) { async_mode = true; From patchwork Thu May 8 19:41:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888691 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 440F228D846; Thu, 8 May 2025 19:41:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733311; cv=none; b=eOknh3Pz2OjOgP3N8WS0rJRDw97JQwUPaKDFNAxhSOVgJKle8VKcGv9H5kiW0kRlhGEFIMEPW9NfbLrGvQVq2WWIuznsoqQGwKWapjpzuPbJBCxBbhkCUxbh0actTXbznB/tjTdT0lQQTg6rVZHgqg0tt7vKYkmNy/rBAJOcFig= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733311; c=relaxed/simple; bh=l0g+LAnddWvzoTX2shNX+O/DFhTRbmW793v8KGvd668=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=StermUy0MyBHT6AnQlN1BzqaJfYmrICBS7nV1jCmiAcnrdPuvENBC2b/fKC45i2eDrn1skK8PSRku4NI38J0g14R1fVwXHM/BuoAXGxrL72iWePDMCBkp47diMsfXY0lRioUHHMHdRLaGAOw5mZgFTiQvBTrj5y3Lt080GplyVQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=RpG+K0mn; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="RpG+K0mn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733309; x=1778269309; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=l0g+LAnddWvzoTX2shNX+O/DFhTRbmW793v8KGvd668=; b=RpG+K0mnvrVz1Wto8V5yc5sB2UCEMFmfqRZDD+w6J8CpqVyJclKdQ6wg yulI1iAR6Uyvl2sK0icxcLf+i1WbchGeanYZiwzcvYszHKWIsDY9QOhoO Ot0mUzU1eyqLvWXjm+nNB0lXmeTass4gp0d2F5ZckO04b4rryh331/igT hkyRGajYLR/eh6tL+kRZZT67hlvAWLGoT3eypq56LK6zV+mzjGxe/IhBS lljZ+qXIBF9OELjCBhXTL7/X3gDI57MlaCj+cj4um9n8Weaz1whxFUPCB lGZ8E5t5QX1ahf2Qp+G4JP20k5o3gg1NU+yH5N7YMOFdl/rnY9do/g+ns w==; X-CSE-ConnectionGUID: 04JKs2cuTSiXJd632p0Fsg== X-CSE-MsgGUID: LSpIGn7sSSmvdWXgiiKj/g== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454759" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454759" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:44 -0700 X-CSE-ConnectionGUID: /clEK4/uTbivhaIBWrz6NQ== X-CSE-MsgGUID: c1sVrN5QSga1BE5y1P/+VQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880878" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:43 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 14/19] mm: zswap: Move the CPU hotplug procedures under "pool functions". Date: Thu, 8 May 2025 12:41:29 -0700 Message-Id: <20250508194134.28392-15-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch merely moves zswap_cpu_comp_prepare() and zswap_cpu_comp_dead() to be in the "pool functions" section because these functions are invoked upon pool creation/deletion. Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 188 ++++++++++++++++++++++++++--------------------------- 1 file changed, 94 insertions(+), 94 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 455e9425c5f5..358dad3e612a 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -248,6 +248,100 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp) **********************************/ static void __zswap_pool_empty(struct percpu_ref *ref); +static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) +{ + struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); + struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); + struct crypto_acomp *acomp = NULL; + struct acomp_req *req = NULL; + u8 *buffer = NULL; + int ret; + + buffer = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); + if (!buffer) { + ret = -ENOMEM; + goto fail; + } + + acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu)); + if (IS_ERR(acomp)) { + pr_err("could not alloc crypto acomp %s : %ld\n", + pool->tfm_name, PTR_ERR(acomp)); + ret = PTR_ERR(acomp); + goto fail; + } + + req = acomp_request_alloc(acomp); + if (!req) { + pr_err("could not alloc crypto acomp_request %s\n", + pool->tfm_name); + ret = -ENOMEM; + goto fail; + } + + /* + * Only hold the mutex after completing allocations, otherwise we may + * recurse into zswap through reclaim and attempt to hold the mutex + * again resulting in a deadlock. + */ + mutex_lock(&acomp_ctx->mutex); + crypto_init_wait(&acomp_ctx->wait); + + /* + * if the backend of acomp is async zip, crypto_req_done() will wakeup + * crypto_wait_req(); if the backend of acomp is scomp, the callback + * won't be called, crypto_wait_req() will return without blocking. + */ + acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, &acomp_ctx->wait); + + acomp_ctx->buffer = buffer; + acomp_ctx->acomp = acomp; + acomp_ctx->is_sleepable = acomp_is_async(acomp); + acomp_ctx->req = req; + mutex_unlock(&acomp_ctx->mutex); + return 0; + +fail: + if (acomp) + crypto_free_acomp(acomp); + kfree(buffer); + return ret; +} + +static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) +{ + struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); + struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); + struct acomp_req *req; + struct crypto_acomp *acomp; + u8 *buffer; + + if (IS_ERR_OR_NULL(acomp_ctx)) + return 0; + + mutex_lock(&acomp_ctx->mutex); + req = acomp_ctx->req; + acomp = acomp_ctx->acomp; + buffer = acomp_ctx->buffer; + acomp_ctx->req = NULL; + acomp_ctx->acomp = NULL; + acomp_ctx->buffer = NULL; + mutex_unlock(&acomp_ctx->mutex); + + /* + * Do the actual freeing after releasing the mutex to avoid subtle + * locking dependencies causing deadlocks. + */ + if (!IS_ERR_OR_NULL(req)) + acomp_request_free(req); + if (!IS_ERR_OR_NULL(acomp)) + crypto_free_acomp(acomp); + kfree(buffer); + + return 0; +} + static struct zswap_pool *zswap_pool_create(char *type, char *compressor) { struct zswap_pool *pool; @@ -818,100 +912,6 @@ static void zswap_entry_free(struct zswap_entry *entry) /********************************* * compressed storage functions **********************************/ -static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) -{ - struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); - struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); - struct crypto_acomp *acomp = NULL; - struct acomp_req *req = NULL; - u8 *buffer = NULL; - int ret; - - buffer = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); - if (!buffer) { - ret = -ENOMEM; - goto fail; - } - - acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu)); - if (IS_ERR(acomp)) { - pr_err("could not alloc crypto acomp %s : %ld\n", - pool->tfm_name, PTR_ERR(acomp)); - ret = PTR_ERR(acomp); - goto fail; - } - - req = acomp_request_alloc(acomp); - if (!req) { - pr_err("could not alloc crypto acomp_request %s\n", - pool->tfm_name); - ret = -ENOMEM; - goto fail; - } - - /* - * Only hold the mutex after completing allocations, otherwise we may - * recurse into zswap through reclaim and attempt to hold the mutex - * again resulting in a deadlock. - */ - mutex_lock(&acomp_ctx->mutex); - crypto_init_wait(&acomp_ctx->wait); - - /* - * if the backend of acomp is async zip, crypto_req_done() will wakeup - * crypto_wait_req(); if the backend of acomp is scomp, the callback - * won't be called, crypto_wait_req() will return without blocking. - */ - acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, - crypto_req_done, &acomp_ctx->wait); - - acomp_ctx->buffer = buffer; - acomp_ctx->acomp = acomp; - acomp_ctx->is_sleepable = acomp_is_async(acomp); - acomp_ctx->req = req; - mutex_unlock(&acomp_ctx->mutex); - return 0; - -fail: - if (acomp) - crypto_free_acomp(acomp); - kfree(buffer); - return ret; -} - -static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) -{ - struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); - struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); - struct acomp_req *req; - struct crypto_acomp *acomp; - u8 *buffer; - - if (IS_ERR_OR_NULL(acomp_ctx)) - return 0; - - mutex_lock(&acomp_ctx->mutex); - req = acomp_ctx->req; - acomp = acomp_ctx->acomp; - buffer = acomp_ctx->buffer; - acomp_ctx->req = NULL; - acomp_ctx->acomp = NULL; - acomp_ctx->buffer = NULL; - mutex_unlock(&acomp_ctx->mutex); - - /* - * Do the actual freeing after releasing the mutex to avoid subtle - * locking dependencies causing deadlocks. - */ - if (!IS_ERR_OR_NULL(req)) - acomp_request_free(req); - if (!IS_ERR_OR_NULL(acomp)) - crypto_free_acomp(acomp); - kfree(buffer); - - return 0; -} - static struct crypto_acomp_ctx *acomp_ctx_get_cpu_lock(struct zswap_pool *pool) { struct crypto_acomp_ctx *acomp_ctx; From patchwork Thu May 8 19:41:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888690 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0FBB28D8D0; Thu, 8 May 2025 19:41:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733312; cv=none; b=nb1YXM7b/KVzvtDC9C/K7O3zPAVKYhLMQjv+FFFgeeHC6hJKH0YgyfCj4v7wIxGWvnRh+lY3gN9lMAqGWLpK2pBYfAFX4UWxmQQ9KUkP06dht4fnL52J/43SKdwkYrLBOQcLBs7QrCH16i2GQN3S1R8Gk7td+ZY49A9u0i7PIJc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733312; c=relaxed/simple; bh=AvunHsRrxVpuS1Dp2nlscgwVBY2zU8yiTAXdQyduiSQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KQ77WBnua0FXN6s9/pJfsk5BlOgzud8e4aflewHytW3A7QXfWNlsYXOJyp21e4mVlzAXntr0EuCcw9NA7X3bp2Ex5du7Yw+DlKJW5bpHkYLv1cRSVZdV6LHxBELvK/O/AFpfz/Byh6DD07GsFP4c0nBpupU8CAmeJZv23NTLtgw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=X7qHk/v4; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="X7qHk/v4" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733310; x=1778269310; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=AvunHsRrxVpuS1Dp2nlscgwVBY2zU8yiTAXdQyduiSQ=; b=X7qHk/v4XURfyzCFPvap7QRORKbGpShO1GEoaDUS4G5pA1EumRwaTsVe tXEkFklsjY5nBNB/Uz7GRLCPnxsyu1rEJwDiff+tkWk451BIwSz5t6FAk qJ7J1s3HE+OjmTrcYayx1/dY9gpkxQxSfyrWffi/53RIyTB9dArU0iFS6 RtB2mpnWveVJBy8D4XnVAFT48lKLse0RLBg4UQAR3/s7AFtK0ok7+I9wZ ZKLgMXaX7keKb25/0oDmoYXkb3y00FyZ7QD/A/yxOi0E7hCFY6lixKLDb 9NjZoBRI1xrBaa24qS3lHuy6sBfcii95DjsTgdts0QmCGyCO/jmhSi6un Q==; X-CSE-ConnectionGUID: 1qJUffAlQM6NnlL0Ah/aTA== X-CSE-MsgGUID: AiSyATG7QUK23ZIxmCWDlA== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454773" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454773" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:45 -0700 X-CSE-ConnectionGUID: wb0+0GtoRdqrgRyWLpAIJw== X-CSE-MsgGUID: 4d8ukS6XRsyBS0vsA9qDyw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880882" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:44 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 15/19] mm: zswap: Per-CPU acomp_ctx resources exist from pool creation to deletion. Date: Thu, 8 May 2025 12:41:30 -0700 Message-Id: <20250508194134.28392-16-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch simplifies the zswap_pool's per-CPU acomp_ctx resource management. Similar to the per-CPU acomp_ctx itself, the per-CPU acomp_ctx's resources' (acomp, ref, buffer) lifetime will also be from pool creation to pool deletion. These resources will persist through CPU hotplug operations. The zswap_cpu_comp_dead() teardown callback has been deleted from the call to cpuhp_setup_state_multi(CPUHP_MM_ZSWP_POOL_PREPARE). As a result, CPU offline hotplug operations will be no-ops as far as the acomp_ctx resources are concerned. The main benefit of using the CPU hotplug multi state instance startup callback to allocate the acomp_ctx resources is that it prevents the cores from being offlined until the multi state instance addition call returns. From Documentation/core-api/cpu_hotplug.rst: "The node list add/remove operations and the callback invocations are serialized against CPU hotplug operations." Furthermore, zswap_[de]compress() cannot contend with zswap_cpu_comp_prepare() because: - During pool creation/deletion, the pool is not in the zswap_pools list. - During CPU hot[un]plug, the CPU is not yet online, as Yosry pointed out. zswap_cpu_comp_prepare() will be executed on a control CPU, since CPUHP_MM_ZSWP_POOL_PREPARE is in the PREPARE section of "enum cpuhp_state". Thanks Yosry for sharing this observation! In both these cases, any recursions into zswap reclaim from zswap_cpu_comp_prepare() will be handled by the old pool. The above two observations enable the following simplifications: 1) zswap_cpu_comp_prepare(): CPU cannot be offlined. Reclaim cannot use the pool. Considerations for mutex init/locking and handling subsequent CPU hotplug online-offlines: Should we lock the mutex of current CPU's acomp_ctx from start to end? It doesn't seem like this is required. The CPU hotplug operations acquire a "cpuhp_state_mutex" before proceeding, hence they are serialized against CPU hotplug operations. If the process gets migrated while zswap_cpu_comp_prepare() is running, it will complete on the new CPU. In case of failures, we pass the acomp_ctx pointer obtained at the start of zswap_cpu_comp_prepare() to acomp_ctx_dealloc(), which again, can only undergo migration. There appear to be no contention scenarios that might cause inconsistent values of acomp_ctx's members. Hence, it seems there is no need for mutex_lock(&acomp_ctx->mutex) in zswap_cpu_comp_prepare(). Since the pool is not yet on zswap_pools list, we don't need to initialize the per-CPU acomp_ctx mutex in zswap_pool_create(). This has been restored to occur in zswap_cpu_comp_prepare(). zswap_cpu_comp_prepare() checks upfront if acomp_ctx->acomp is valid. If so, it returns success. This should handle any CPU hotplug online-offline transitions after pool creation is done. 2) CPU offline vis-a-vis zswap ops: Let's suppose the process is migrated to another CPU before the current CPU is dysfunctional. If zswap_[de]compress() holds the acomp_ctx->mutex lock of the offlined CPU, that mutex will be released once it completes on the new CPU. Since there is no teardown callback, there is no possibility of UAF. 3) Pool creation/deletion and process migration to another CPU: - During pool creation/deletion, the pool is not in the zswap_pools list. Hence it cannot contend with zswap ops on that CPU. However, the process can get migrated. Pool creation --> zswap_cpu_comp_prepare() --> process migrated: * CPU offline: no-op. * zswap_cpu_comp_prepare() continues to run on the new CPU to finish allocating acomp_ctx resources for the offlined CPU. Pool deletion --> acomp_ctx_dealloc() --> process migrated: * CPU offline: no-op. * acomp_ctx_dealloc() continues to run on the new CPU to finish de-allocating acomp_ctx resources for the offlined CPU. 4) Pool deletion vis-a-vis CPU onlining: To prevent possibility of race conditions between acomp_ctx_dealloc() freeing the acomp_ctx resources and the initial check for a valid acomp_ctx->acomp in zswap_cpu_comp_prepare(), we need to delete the multi state instance right after it is added, in zswap_pool_create(). Summary of changes based on the above: -------------------------------------- 1) Zero-initialization of pool->acomp_ctx in zswap_pool_create() to simplify and share common code for different error handling/cleanup related to the acomp_ctx. 2) Remove the node list instance right after node list add function call in zswap_pool_create(). This prevents race conditions between CPU onlining after initial pool creation, and acomp_ctx_dealloc() freeing the acomp_ctx resources. 3) zswap_pool_destroy() will call acomp_ctx_dealloc() to de-allocate the per-CPU acomp_ctx resources. 4) Changes to zswap_cpu_comp_prepare(): a) Check if acomp_ctx->acomp is valid at the beginning and return, because the acomp_ctx is already initialized. b) Move the mutex_init to happen in this procedure, before it returns. c) All error conditions handled by calling acomp_ctx_dealloc(). 5) New procedure acomp_ctx_dealloc() for common error/cleanup code. 6) No more multi state instance teardown callback. CPU offlining is a no-op as far as acomp_ctx resources are concerned. 7) Delete acomp_ctx_get_cpu_lock()/acomp_ctx_put_unlock(). Directly call mutex_lock(&acomp_ctx->mutex)/mutex_unlock(&acomp_ctx->mutex) in zswap_[de]compress(). The per-CPU memory cost of not deleting the acomp_ctx resources upon CPU offlining, and only deleting them when the pool is destroyed, is as follows, on x86_64: IAA with batching: 64.8 KB Software compressors: 8.2 KB Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 193 +++++++++++++++++++++++++---------------------------- 1 file changed, 92 insertions(+), 101 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 358dad3e612a..238f92e63a22 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -248,43 +248,65 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp) **********************************/ static void __zswap_pool_empty(struct percpu_ref *ref); +/* + * The per-cpu pool->acomp_ctx is zero-initialized on allocation. This makes + * it easy for different error conditions/cleanup related to the acomp_ctx + * to be handled by acomp_ctx_dealloc(): + * - Errors during zswap_cpu_comp_prepare(). + * - Partial success/error of cpuhp_state_add_instance() call in + * zswap_pool_create(). Only some cores could have executed + * zswap_cpu_comp_prepare(), not others. + * - Cleanup acomp_ctx resources on all cores in zswap_pool_destroy(). + */ +static void acomp_ctx_dealloc(struct crypto_acomp_ctx *acomp_ctx) +{ + if (IS_ERR_OR_NULL(acomp_ctx)) + return; + + if (!IS_ERR_OR_NULL(acomp_ctx->req)) + acomp_request_free(acomp_ctx->req); + if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) + crypto_free_acomp(acomp_ctx->acomp); + kfree(acomp_ctx->buffer); +} + static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) { struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); - struct crypto_acomp *acomp = NULL; - struct acomp_req *req = NULL; - u8 *buffer = NULL; - int ret; + int ret = -ENOMEM; - buffer = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); - if (!buffer) { - ret = -ENOMEM; - goto fail; - } + /* + * The per-CPU pool->acomp_ctx is zero-initialized on allocation. + * Even though we delete the multi state instance right after successful + * addition of the instance in zswap_pool_create(), we cannot eliminate + * the possibility of the CPU going through offline-online transitions. + * If this does happen, we check if the acomp_ctx has already been + * initialized, and return. + */ + if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) + return 0; - acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu)); - if (IS_ERR(acomp)) { + acomp_ctx->buffer = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); + if (!acomp_ctx->buffer) + return ret; + + acomp_ctx->acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu)); + if (IS_ERR(acomp_ctx->acomp)) { pr_err("could not alloc crypto acomp %s : %ld\n", - pool->tfm_name, PTR_ERR(acomp)); - ret = PTR_ERR(acomp); + pool->tfm_name, PTR_ERR(acomp_ctx->acomp)); + ret = PTR_ERR(acomp_ctx->acomp); goto fail; } + acomp_ctx->is_sleepable = acomp_is_async(acomp_ctx->acomp); - req = acomp_request_alloc(acomp); - if (!req) { + acomp_ctx->req = acomp_request_alloc(acomp_ctx->acomp); + if (!acomp_ctx->req) { pr_err("could not alloc crypto acomp_request %s\n", pool->tfm_name); - ret = -ENOMEM; goto fail; } - /* - * Only hold the mutex after completing allocations, otherwise we may - * recurse into zswap through reclaim and attempt to hold the mutex - * again resulting in a deadlock. - */ - mutex_lock(&acomp_ctx->mutex); crypto_init_wait(&acomp_ctx->wait); /* @@ -292,56 +314,17 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) * crypto_wait_req(); if the backend of acomp is scomp, the callback * won't be called, crypto_wait_req() will return without blocking. */ - acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + acomp_request_set_callback(acomp_ctx->req, CRYPTO_TFM_REQ_MAY_BACKLOG, crypto_req_done, &acomp_ctx->wait); - acomp_ctx->buffer = buffer; - acomp_ctx->acomp = acomp; - acomp_ctx->is_sleepable = acomp_is_async(acomp); - acomp_ctx->req = req; - mutex_unlock(&acomp_ctx->mutex); + mutex_init(&acomp_ctx->mutex); return 0; fail: - if (acomp) - crypto_free_acomp(acomp); - kfree(buffer); + acomp_ctx_dealloc(acomp_ctx); return ret; } -static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) -{ - struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); - struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); - struct acomp_req *req; - struct crypto_acomp *acomp; - u8 *buffer; - - if (IS_ERR_OR_NULL(acomp_ctx)) - return 0; - - mutex_lock(&acomp_ctx->mutex); - req = acomp_ctx->req; - acomp = acomp_ctx->acomp; - buffer = acomp_ctx->buffer; - acomp_ctx->req = NULL; - acomp_ctx->acomp = NULL; - acomp_ctx->buffer = NULL; - mutex_unlock(&acomp_ctx->mutex); - - /* - * Do the actual freeing after releasing the mutex to avoid subtle - * locking dependencies causing deadlocks. - */ - if (!IS_ERR_OR_NULL(req)) - acomp_request_free(req); - if (!IS_ERR_OR_NULL(acomp)) - crypto_free_acomp(acomp); - kfree(buffer); - - return 0; -} - static struct zswap_pool *zswap_pool_create(char *type, char *compressor) { struct zswap_pool *pool; @@ -375,19 +358,43 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) strscpy(pool->tfm_name, compressor, sizeof(pool->tfm_name)); - pool->acomp_ctx = alloc_percpu(*pool->acomp_ctx); + /* Many things rely on the zero-initialization. */ + pool->acomp_ctx = alloc_percpu_gfp(*pool->acomp_ctx, + GFP_KERNEL | __GFP_ZERO); if (!pool->acomp_ctx) { pr_err("percpu alloc failed\n"); goto error; } - for_each_possible_cpu(cpu) - mutex_init(&per_cpu_ptr(pool->acomp_ctx, cpu)->mutex); - + /* + * This is serialized against CPU hotplug operations. Hence, cores + * cannot be offlined until this finishes. + * In case of errors, we need to goto "ref_fail" instead of "error" + * because there is no teardown callback registered anymore, for + * cpuhp_state_add_instance() to de-allocate resources as it rolls back + * state on cores before the CPU on which error was encountered. + */ ret = cpuhp_state_add_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); + + /* + * We only needed the multi state instance add operation to invoke the + * startup callback for all cores without cores getting offlined. Since + * the acomp_ctx resources will now only be de-allocated when the pool + * is destroyed, we can safely remove the multi state instance. This + * minimizes (but does not eliminate) the possibility of + * zswap_cpu_comp_prepare() being invoked again due to a CPU + * offline-online transition. Removing the instance also prevents race + * conditions between CPU onlining after initial pool creation, and + * acomp_ctx_dealloc() freeing the acomp_ctx resources. + * Note that we delete the instance before checking the error status of + * the node list add operation because we want the instance removal even + * in case of errors in the former. + */ + cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); + if (ret) - goto error; + goto ref_fail; /* being the current pool takes 1 ref; this func expects the * caller to always add the new pool as the current pool @@ -403,7 +410,8 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) return pool; ref_fail: - cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); + for_each_possible_cpu(cpu) + acomp_ctx_dealloc(per_cpu_ptr(pool->acomp_ctx, cpu)); error: if (pool->acomp_ctx) free_percpu(pool->acomp_ctx); @@ -457,9 +465,13 @@ static struct zswap_pool *__zswap_pool_create_fallback(void) static void zswap_pool_destroy(struct zswap_pool *pool) { + int cpu; + zswap_pool_debug("destroying", pool); - cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); + for_each_possible_cpu(cpu) + acomp_ctx_dealloc(per_cpu_ptr(pool->acomp_ctx, cpu)); + free_percpu(pool->acomp_ctx); zpool_destroy_pool(pool->zpool); @@ -912,31 +924,6 @@ static void zswap_entry_free(struct zswap_entry *entry) /********************************* * compressed storage functions **********************************/ -static struct crypto_acomp_ctx *acomp_ctx_get_cpu_lock(struct zswap_pool *pool) -{ - struct crypto_acomp_ctx *acomp_ctx; - - for (;;) { - acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - mutex_lock(&acomp_ctx->mutex); - if (likely(acomp_ctx->req)) - return acomp_ctx; - /* - * It is possible that we were migrated to a different CPU after - * getting the per-CPU ctx but before the mutex was acquired. If - * the old CPU got offlined, zswap_cpu_comp_dead() could have - * already freed ctx->req (among other things) and set it to - * NULL. Just try again on the new CPU that we ended up on. - */ - mutex_unlock(&acomp_ctx->mutex); - } -} - -static void acomp_ctx_put_unlock(struct crypto_acomp_ctx *acomp_ctx) -{ - mutex_unlock(&acomp_ctx->mutex); -} - static bool zswap_compress(struct page *page, struct zswap_entry *entry, struct zswap_pool *pool) { @@ -949,7 +936,10 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, gfp_t gfp; u8 *dst; - acomp_ctx = acomp_ctx_get_cpu_lock(pool); + acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); + + mutex_lock(&acomp_ctx->mutex); + dst = acomp_ctx->buffer; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); @@ -997,7 +987,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, else if (alloc_ret) zswap_reject_alloc_fail++; - acomp_ctx_put_unlock(acomp_ctx); + mutex_unlock(&acomp_ctx->mutex); return comp_ret == 0 && alloc_ret == 0; } @@ -1009,7 +999,8 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio) int decomp_ret, dlen; u8 *src, *obj; - acomp_ctx = acomp_ctx_get_cpu_lock(entry->pool); + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(&acomp_ctx->mutex); obj = zpool_obj_read_begin(zpool, entry->handle, acomp_ctx->buffer); /* @@ -1033,7 +1024,7 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio) dlen = acomp_ctx->req->dlen; zpool_obj_read_end(zpool, entry->handle, obj); - acomp_ctx_put_unlock(acomp_ctx); + mutex_unlock(&acomp_ctx->mutex); if (!decomp_ret && dlen == PAGE_SIZE) return true; @@ -1849,7 +1840,7 @@ static int zswap_setup(void) ret = cpuhp_setup_state_multi(CPUHP_MM_ZSWP_POOL_PREPARE, "mm/zswap_pool:prepare", zswap_cpu_comp_prepare, - zswap_cpu_comp_dead); + NULL); if (ret) goto hp_fail; From patchwork Thu May 8 19:41:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 888689 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3187928DEE9; Thu, 8 May 2025 19:41:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733314; cv=none; b=JNjitYvtexqvCI+qAuY6TTLMWWCEB+HN5emXFZGcpdwAijXoMI/NFSvGyZYn5T5S45ztiH1VT9x3EDo3MOB2QJXgTzZ9mjMpaLTfmpMLPnqbJN2Pqgqt8FtEMzZuwVl0oASrPlIy+sitthLdOYDbDLjjQapoMHKkBW3M8c7TedY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746733314; c=relaxed/simple; bh=DDaOA19vkB8fzpz/hF+6UEVXZdzAdZ5S5Yw7TXl9byM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=I5Po24ZX2WK6XYmyYpZD8G/m6Vjru9C6NMcYVDhdWngNKLIxohru/jsNYFPrsiDyIdD5TMn51Meu1NMNVFF5kEo0xFD2Xb3ZCg/6mVZyuITKQ6jBBZ/JQjZryfrIQCxa+vI2uVHMM7AyGecLyxycueHoJi+s3zgfpNjBFOs5Npg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ODGkHamn; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ODGkHamn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733312; x=1778269312; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DDaOA19vkB8fzpz/hF+6UEVXZdzAdZ5S5Yw7TXl9byM=; b=ODGkHamnoey7Yz+MvE4Vrwj6SexS3vKe7xfGqrfLyfpDfYdGqqbfawT9 +sQaR4aZHCaHG2H3PbChBjMvM67YApI4VHYFoNYyGYzcLmTJa5RBNp70i 4pOySF8m21lVpFftLYO2k4lHayeCUiTYa2oSlLdNKv2zKfejypqS5a7k1 YfIyFwclk4UjIvusvekL0aLdUOP/kH+ARiY12xZtYKpJLSOtGWJBZ2RBN Vp/1IKKjHmi3Q1qPukW1RNJm/W/ShBXoEEVrsLsn3IUnrYPMMoMXTA5I5 8ePeL99E6kfDSA19nRnefW8FMzjqIAXhmp0EIvPupX+aMfkmr11Og9VAp w==; X-CSE-ConnectionGUID: cmKvsuZxS3qdWCchxb1Apw== X-CSE-MsgGUID: +aRNChBKQ8W7Q1VhYZW+Jg== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454816" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454816" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:47 -0700 X-CSE-ConnectionGUID: KBdUNAPPQ12ix1jCngkBtQ== X-CSE-MsgGUID: foBNvo3jQTaFrKk9Zi2jBw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880893" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:46 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 18/19] mm: zswap: zswap_store() will process a folio in batches. Date: Thu, 8 May 2025 12:41:33 -0700 Message-Id: <20250508194134.28392-19-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch modifies zswap_store() to store a batch of pages at a time, instead of storing one page at a time. It does this by calling a new procedure zswap_store_pages() with "batch_size" pages. If the folio is of order-0, the batch_size is 1. If zswap_store() is processing a large folio: - If the compressor supports batching, the batch_size will be the pool->nr_reqs. - If the compressor does not support batching, the batch_size will be ZSWAP_MAX_BATCH_SIZE. zswap_store_pages() implements all the computes done earlier in zswap_store_page() for a single-page, for multiple pages in a folio, namely the "batch". zswap_store_pages() starts by allocating all zswap entries required to store the batch. Next, it calls zswap_compress() to sequentially compress each page in the batch. Finally, it adds the batch's zswap entries to the xarray and LRU, charges zswap memory and increments zswap stats. The error handling and cleanup required for all failure scenarios that can occur while storing a batch in zswap are consolidated to a single "store_pages_failed" label in zswap_store_pages(). Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 199 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 130 insertions(+), 69 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 2273dbfd460f..1d6795704350 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1518,81 +1518,125 @@ static void shrink_worker(struct work_struct *w) * main API **********************************/ -static bool zswap_store_page(struct page *page, - struct obj_cgroup *objcg, - struct zswap_pool *pool) +/* + * Store multiple pages in @folio, starting from the page at index @start up to + * the page at index @end-1. + */ +static bool zswap_store_pages(struct folio *folio, + long start, + long end, + struct obj_cgroup *objcg, + struct zswap_pool *pool) { - swp_entry_t page_swpentry = page_swap_entry(page); - struct zswap_entry *entry, *old; - - /* allocate entry */ - entry = zswap_entry_cache_alloc(GFP_KERNEL, page_to_nid(page)); - if (!entry) { - zswap_reject_kmemcache_fail++; - return false; - } + struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE]; + int node_id = folio_nid(folio); + u8 i, store_fail_idx = 0, nr_pages = end - start; - if (!zswap_compress(page, entry, pool)) - goto compress_failed; + for (i = 0; i < nr_pages; ++i) { + entries[i] = zswap_entry_cache_alloc(GFP_KERNEL, node_id); - old = xa_store(swap_zswap_tree(page_swpentry), - swp_offset(page_swpentry), - entry, GFP_KERNEL); - if (xa_is_err(old)) { - int err = xa_err(old); + if (unlikely(!entries[i])) { + zswap_reject_kmemcache_fail++; + /* + * While handling this error, we only need to call + * zswap_entry_cache_free() for entries[0 .. i-1]. + */ + nr_pages = i; + goto store_pages_failed; + } - WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); - zswap_reject_alloc_fail++; - goto store_failed; + /* + * Initialize the handle to an error value. This facilitates + * having a consolidated failure handling + * 'goto store_pages_failed' that can inspect the value of the + * handle to determine whether zpool memory needs to be + * de-allocated. + */ + entries[i]->handle = (unsigned long)ERR_PTR(-EINVAL); } - /* - * We may have had an existing entry that became stale when - * the folio was redirtied and now the new version is being - * swapped out. Get rid of the old. - */ - if (old) - zswap_entry_free(old); + for (i = 0; i < nr_pages; ++i) { + struct page *page = folio_page(folio, start + i); - /* - * The entry is successfully compressed and stored in the tree, there is - * no further possibility of failure. Grab refs to the pool and objcg, - * charge zswap memory, and increment zswap_stored_pages. - * The opposite actions will be performed by zswap_entry_free() - * when the entry is removed from the tree. - */ - zswap_pool_get(pool); - if (objcg) { - obj_cgroup_get(objcg); - obj_cgroup_charge_zswap(objcg, entry->length); + if (!zswap_compress(page, entries[i], pool)) + goto store_pages_failed; } - atomic_long_inc(&zswap_stored_pages); - /* - * We finish initializing the entry while it's already in xarray. - * This is safe because: - * - * 1. Concurrent stores and invalidations are excluded by folio lock. - * - * 2. Writeback is excluded by the entry not being on the LRU yet. - * The publishing order matters to prevent writeback from seeing - * an incoherent entry. - */ - entry->pool = pool; - entry->swpentry = page_swpentry; - entry->objcg = objcg; - entry->referenced = true; - if (entry->length) { - INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); + for (i = 0; i < nr_pages; ++i) { + swp_entry_t page_swpentry = page_swap_entry(folio_page(folio, start + i)); + struct zswap_entry *old, *entry = entries[i]; + + old = xa_store(swap_zswap_tree(page_swpentry), + swp_offset(page_swpentry), + entry, GFP_KERNEL); + if (unlikely(xa_is_err(old))) { + int err = xa_err(old); + + WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); + zswap_reject_alloc_fail++; + /* + * Entries up to this point have been stored in the + * xarray. zswap_store() will erase them from the xarray + * and call zswap_entry_free(). Local cleanup in + * 'store_pages_failed' only needs to happen for + * entries from [@i to @nr_pages). + */ + store_fail_idx = i; + goto store_pages_failed; + } + + /* + * We may have had an existing entry that became stale when + * the folio was redirtied and now the new version is being + * swapped out. Get rid of the old. + */ + if (unlikely(old)) + zswap_entry_free(old); + + /* + * The entry is successfully compressed and stored in the tree, there is + * no further possibility of failure. Grab refs to the pool and objcg, + * charge zswap memory, and increment zswap_stored_pages. + * The opposite actions will be performed by zswap_entry_free() + * when the entry is removed from the tree. + */ + zswap_pool_get(pool); + if (objcg) { + obj_cgroup_get(objcg); + obj_cgroup_charge_zswap(objcg, entry->length); + } + atomic_long_inc(&zswap_stored_pages); + + /* + * We finish initializing the entry while it's already in xarray. + * This is safe because: + * + * 1. Concurrent stores and invalidations are excluded by folio lock. + * + * 2. Writeback is excluded by the entry not being on the LRU yet. + * The publishing order matters to prevent writeback from seeing + * an incoherent entry. + */ + entry->pool = pool; + entry->swpentry = page_swpentry; + entry->objcg = objcg; + entry->referenced = true; + if (likely(entry->length)) { + INIT_LIST_HEAD(&entry->lru); + zswap_lru_add(&zswap_list_lru, entry); + } } return true; -store_failed: - zpool_free(pool->zpool, entry->handle); -compress_failed: - zswap_entry_cache_free(entry); +store_pages_failed: + for (i = store_fail_idx; i < nr_pages; ++i) { + if (!IS_ERR_VALUE(entries[i]->handle)) + zpool_free(pool->zpool, entries[i]->handle); + + zswap_entry_cache_free(entries[i]); + } + return false; } @@ -1603,8 +1647,9 @@ bool zswap_store(struct folio *folio) struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; struct zswap_pool *pool; + unsigned int batch_size; bool ret = false; - long index; + long start, end; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1638,10 +1683,26 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } - for (index = 0; index < nr_pages; ++index) { - struct page *page = folio_page(folio, index); + /* + * If a large folio is being swapped out and the zswap compressor + * supports batching, i.e., has multiple acomp requests, the folio will + * be compressed in batches of @pool->nr_reqs. If the compressor has + * only one acomp request, the folio will be compressed in batches of + * ZSWAP_MAX_BATCH_SIZE pages, where each page in the batch is + * compressed sequentially. We see better performance by processing the + * folio in batches of ZSWAP_MAX_BATCH_SIZE, due to cache locality of + * working set structures such as the array of zswap_entry's for the + * batch. + */ + batch_size = (nr_pages > 1) ? ((pool->nr_reqs > 1) ? + pool->nr_reqs : ZSWAP_MAX_BATCH_SIZE) + : 1; + + /* Store the folio in batches of "batch_size" pages. */ + for (start = 0; start < nr_pages; start += batch_size) { + end = min(start + batch_size, nr_pages); - if (!zswap_store_page(page, objcg, pool)) + if (!zswap_store_pages(folio, start, end, objcg, pool)) goto put_pool; } @@ -1671,9 +1732,9 @@ bool zswap_store(struct folio *folio) struct zswap_entry *entry; struct xarray *tree; - for (index = 0; index < nr_pages; ++index) { - tree = swap_zswap_tree(swp_entry(type, offset + index)); - entry = xa_erase(tree, offset + index); + for (start = 0; start < nr_pages; ++start) { + tree = swap_zswap_tree(swp_entry(type, offset + start)); + entry = xa_erase(tree, offset + start); if (entry) zswap_entry_free(entry); }