From patchwork Wed Sep 30 06:53:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 252930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8A4BC2D0A8 for ; Wed, 30 Sep 2020 06:49:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 92EA22076E for ; Wed, 30 Sep 2020 06:49:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VTD6SCyZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728315AbgI3Gto (ORCPT ); Wed, 30 Sep 2020 02:49:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725823AbgI3Gtk (ORCPT ); Wed, 30 Sep 2020 02:49:40 -0400 Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75B89C061755; Tue, 29 Sep 2020 23:49:40 -0700 (PDT) Received: by mail-pg1-x542.google.com with SMTP id d13so490383pgl.6; Tue, 29 Sep 2020 23:49:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=981H7COrDglg2y2OsdhN4Ry4swXs13dJrOliSNOnl9g=; b=VTD6SCyZPXt/5nIAOEqALhGagetdiFPpYbIX86qlGxbocho1y+f/mpB80lXEz+tm7C WPqAJDixQU0DnytgSTNpm86PRzC4bTyHZxzdH07dQaxcBmAC98P4JRlPQytUlDnujCYw VCsdX6QF1B02CM23+AJV8X7BVrHIXixUbS0yeeRHhIkH1zwu1ShforEv739Ju+sB6gmL 31LGEumu0hEvVb7seUGzaCvZaOBZYnzRUnxV4oYPoH3oI0utmkBBYtWHKCu+Jf5K4clj sHMtq1TgNfc44BNLwIcegLpJB66wA/8eIYjaBs0sYHw5Mv6rRakjsuEcif2kLmaWxV7L Dhfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=981H7COrDglg2y2OsdhN4Ry4swXs13dJrOliSNOnl9g=; b=X+qvdSMGFqPED8yy4uKjBWd/6n/QGi3XJV0jDiin0F5PmlRSYCAnOxutKfSCi7d3jc AMy9KhWfLKX/i5eMVcIFA3LqKXRJ6HuaaNvSyLRHmqHrYsg1cw6FwZoMWjcb8lkSHRWr GnvBY7VIcDFQru6dQHcUta85qz/84CgMyGaxXJPIHAdIMB/vxyfEP6xy2knELyjqoO9N rA379L9uOe+z/8yOemuqDYBsQQS8j4U/YzucKDWe9i5CYs4lYYOCZiGw9Ek01Wc2X5+k XjK0aLOLOsomDZZRFAyAz/l+q86P+BJ9lpSvIfjM08MZcfIHiR7ZSLqxfkcPt9x6MtTD nCAA== X-Gm-Message-State: AOAM532qomhp4OBJYAt/glq+Ei1hrjzVwO/LLqpCeL8LZ0bVWdhXWP9P 122stvZ3nVfauoD20oJ1ilc= X-Google-Smtp-Source: ABdhPJw9Vmlwv8qFlV+Os7mUuWY82swGcHqlIj/BIfXkC1jOiYDEGvYsgv1t43sX4UNIWOFDU2oM8Q== X-Received: by 2002:a65:5bcf:: with SMTP id o15mr1054550pgr.126.1601448579954; Tue, 29 Sep 2020 23:49:39 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id 190sm1100865pfy.22.2020.09.29.23.49.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Sep 2020 23:49:39 -0700 (PDT) From: Nick Terrell To: Herbert Xu Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, squashfs-devel@lists.sourceforge.net, linux-f2fs-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org, Kernel Team , Nick Terrell , Nick Terrell , Chris Mason , Petr Malat , Johannes Weiner , Niket Agarwal , Yann Collet , Christoph Hellwig Subject: [PATCH v4 2/9] lib: zstd: Add decompress_sources.h for decompress_unzstd Date: Tue, 29 Sep 2020 23:53:11 -0700 Message-Id: <20200930065318.3326526-3-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200930065318.3326526-1-nickrterrell@gmail.com> References: <20200930065318.3326526-1-nickrterrell@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Nick Terrell Adds decompress_sources.h which includes every .c file necessary for zstd decompression. This is used in decompress_unzstd.c so the internal structure of the library isn't exposed. This allows us to upgrade the zstd library version without modifying any callers. Instead we just need to update decompress_sources.h. Signed-off-by: Nick Terrell --- lib/decompress_unzstd.c | 6 +----- lib/zstd/decompress_sources.h | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 5 deletions(-) create mode 100644 lib/zstd/decompress_sources.h diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c index dbc290af26b4..a79f705f236d 100644 --- a/lib/decompress_unzstd.c +++ b/lib/decompress_unzstd.c @@ -68,11 +68,7 @@ #ifdef STATIC # define UNZSTD_PREBOOT # include "xxhash.c" -# include "zstd/entropy_common.c" -# include "zstd/fse_decompress.c" -# include "zstd/huf_decompress.c" -# include "zstd/zstd_common.c" -# include "zstd/decompress.c" +# include "zstd/decompress_sources.h" #endif #include diff --git a/lib/zstd/decompress_sources.h b/lib/zstd/decompress_sources.h new file mode 100644 index 000000000000..ccb4960ea0cd --- /dev/null +++ b/lib/zstd/decompress_sources.h @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0-only + +/* + * This file includes every .c file needed for decompression. + * It is used by lib/decompress_unzstd.c to include the decompression + * source into the translation-unit, so it can be used for kernel + * decompression. + */ + +#include "entropy_common.c" +#include "fse_decompress.c" +#include "huf_decompress.c" +#include "zstd_common.c" +#include "decompress.c" From patchwork Wed Sep 30 06:53:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 252929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67DF6C2D0A8 for ; Wed, 30 Sep 2020 06:50:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E4F632076E for ; Wed, 30 Sep 2020 06:50:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bx6G1V8+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728411AbgI3GuQ (ORCPT ); Wed, 30 Sep 2020 02:50:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725798AbgI3GuK (ORCPT ); Wed, 30 Sep 2020 02:50:10 -0400 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EA3CC061755; Tue, 29 Sep 2020 23:50:06 -0700 (PDT) Received: by mail-pf1-x441.google.com with SMTP id x22so480735pfo.12; Tue, 29 Sep 2020 23:50:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2E/IwoJzArQOI0tLeIWcAV63W6gwJ92+ohjyeEw8pg8=; b=bx6G1V8+FQodu/hCJFLTVBYsNHoLZUsLaIOHSJ8Ppa+7aPpJsde92+pD3gGT7yAT10 O0CxG8HBpLf16k8DkjDHiLNm97yrhL6DnuRC4QTEvmnr28g09+aqi/jN+qY6G6NLpDXU hebvfuDkxNuS5xnEeGJs3alkM4qRJbmlgnBZqR4Dm4E27s8VWCzav/msUs7y/o/ogxSO +UMTZnXSNOkxV1HWUUVvFbqou7YqhHXZAJeK+eKX8SPW028odqJNDnECZhiNvaMORu9H 6QJCeMkj5qjkMb8nZXogigMgpXnX4zCqBLGdDFbjEo6/YnBbYhArVNvk+xVWyQA3yIB+ sk4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2E/IwoJzArQOI0tLeIWcAV63W6gwJ92+ohjyeEw8pg8=; b=s4aLpv9qcIzsdGlxLcbEJFjwnA8jq/qVC7MXV7OFenKKa6HAUuTzY5wkO1naC+5ywL eS2fn05RVHri+foly3hSD4JSiOo6AiXnlo9SFApC91Jqw//BEL1kJHoX3wY3mtVImJA9 fvI9A9HDQHBzNvpGp7mVz70zP4fVpOUzOihNMnnvA2uih4/fLT0b7W8rfukHapg1yPiu fjUyX0CRz/6oqRpqoTq2hOexhvgWBCwdhagzXE4NnoEyN3/8akqqAI6xeqsPLkR1KCv/ uGuTm8mQE7b8Cplp20ZYniNefMLfxc1ifYULkiBAAT5S3VlD+6fx9gbnBFQ0zP4a2b9I 4ocQ== X-Gm-Message-State: AOAM531BO4Z07Nq09VNTQ7X71ihEkXjj/pJBSdosRvEVmWBhFi4srtko 89+rRGBfa9JMpDu2gleVyGg= X-Google-Smtp-Source: ABdhPJzb4+pKVFANraBU2qyubhVYWvRaXnT+h66sMPI6vDGQZCJME36TZc6rDrWAPVbJKtEviE/Hvg== X-Received: by 2002:a63:4142:: with SMTP id o63mr1029677pga.337.1601448605631; Tue, 29 Sep 2020 23:50:05 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id 190sm1100865pfy.22.2020.09.29.23.50.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Sep 2020 23:50:05 -0700 (PDT) From: Nick Terrell To: Herbert Xu Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, squashfs-devel@lists.sourceforge.net, linux-f2fs-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org, Kernel Team , Nick Terrell , Nick Terrell , Chris Mason , Petr Malat , Johannes Weiner , Niket Agarwal , Yann Collet , Christoph Hellwig Subject: [PATCH v4 5/9] btrfs: zstd: Switch to the zstd-1.4.6 API Date: Tue, 29 Sep 2020 23:53:14 -0700 Message-Id: <20200930065318.3326526-6-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200930065318.3326526-1-nickrterrell@gmail.com> References: <20200930065318.3326526-1-nickrterrell@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Nick Terrell Move away from the compatibility wrapper to the zstd-1.4.6 API. This code is functionally equivalent. Signed-off-by: Nick Terrell --- fs/btrfs/zstd.c | 48 ++++++++++++++++++++++++++++-------------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/fs/btrfs/zstd.c b/fs/btrfs/zstd.c index a7367ff573d4..6b466e090cd7 100644 --- a/fs/btrfs/zstd.c +++ b/fs/btrfs/zstd.c @@ -16,7 +16,7 @@ #include #include #include -#include +#include #include "misc.h" #include "compression.h" #include "ctree.h" @@ -159,8 +159,8 @@ static void zstd_calc_ws_mem_sizes(void) zstd_get_btrfs_parameters(level, ZSTD_BTRFS_MAX_INPUT); size_t level_size = max_t(size_t, - ZSTD_CStreamWorkspaceBound(params.cParams), - ZSTD_DStreamWorkspaceBound(ZSTD_BTRFS_MAX_INPUT)); + ZSTD_estimateCStreamSize_usingCParams(params.cParams), + ZSTD_estimateDStreamSize(ZSTD_BTRFS_MAX_INPUT)); max_size = max_t(size_t, max_size, level_size); zstd_ws_mem_sizes[level - 1] = max_size; @@ -389,13 +389,23 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, *total_in = 0; /* Initialize the stream */ - stream = ZSTD_initCStream(params, len, workspace->mem, - workspace->size); + stream = ZSTD_initStaticCStream(workspace->mem, workspace->size); if (!stream) { - pr_warn("BTRFS: ZSTD_initCStream failed\n"); + pr_warn("BTRFS: ZSTD_initStaticCStream failed\n"); ret = -EIO; goto out; } + { + size_t ret2; + + ret2 = ZSTD_initCStream_advanced(stream, NULL, 0, params, len); + if (ZSTD_isError(ret2)) { + pr_warn("BTRFS: ZSTD_initCStream_advanced returned %s\n", + ZSTD_getErrorName(ret2)); + ret = -EIO; + goto out; + } + } /* map in the first page of input data */ in_page = find_get_page(mapping, start >> PAGE_SHIFT); @@ -421,8 +431,8 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, ret2 = ZSTD_compressStream(stream, &workspace->out_buf, &workspace->in_buf); if (ZSTD_isError(ret2)) { - pr_debug("BTRFS: ZSTD_compressStream returned %d\n", - ZSTD_getErrorCode(ret2)); + pr_debug("BTRFS: ZSTD_compressStream returned %s\n", + ZSTD_getErrorName(ret2)); ret = -EIO; goto out; } @@ -489,8 +499,8 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, ret2 = ZSTD_endStream(stream, &workspace->out_buf); if (ZSTD_isError(ret2)) { - pr_debug("BTRFS: ZSTD_endStream returned %d\n", - ZSTD_getErrorCode(ret2)); + pr_debug("BTRFS: ZSTD_endStream returned %s\n", + ZSTD_getErrorName(ret2)); ret = -EIO; goto out; } @@ -557,10 +567,9 @@ int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb) unsigned long buf_start; unsigned long total_out = 0; - stream = ZSTD_initDStream( - ZSTD_BTRFS_MAX_INPUT, workspace->mem, workspace->size); + stream = ZSTD_initStaticDStream(workspace->mem, workspace->size); if (!stream) { - pr_debug("BTRFS: ZSTD_initDStream failed\n"); + pr_debug("BTRFS: ZSTD_initStaticDStream failed\n"); ret = -EIO; goto done; } @@ -579,8 +588,8 @@ int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb) ret2 = ZSTD_decompressStream(stream, &workspace->out_buf, &workspace->in_buf); if (ZSTD_isError(ret2)) { - pr_debug("BTRFS: ZSTD_decompressStream returned %d\n", - ZSTD_getErrorCode(ret2)); + pr_debug("BTRFS: ZSTD_decompressStream returned %s\n", + ZSTD_getErrorName(ret2)); ret = -EIO; goto done; } @@ -633,10 +642,9 @@ int zstd_decompress(struct list_head *ws, unsigned char *data_in, unsigned long pg_offset = 0; char *kaddr; - stream = ZSTD_initDStream( - ZSTD_BTRFS_MAX_INPUT, workspace->mem, workspace->size); + stream = ZSTD_initStaticDStream(workspace->mem, workspace->size); if (!stream) { - pr_warn("BTRFS: ZSTD_initDStream failed\n"); + pr_warn("BTRFS: ZSTD_initStaticDStream failed\n"); ret = -EIO; goto finish; } @@ -667,8 +675,8 @@ int zstd_decompress(struct list_head *ws, unsigned char *data_in, ret2 = ZSTD_decompressStream(stream, &workspace->out_buf, &workspace->in_buf); if (ZSTD_isError(ret2)) { - pr_debug("BTRFS: ZSTD_decompressStream returned %d\n", - ZSTD_getErrorCode(ret2)); + pr_debug("BTRFS: ZSTD_decompressStream returned %s\n", + ZSTD_getErrorName(ret2)); ret = -EIO; goto finish; } From patchwork Wed Sep 30 06:53:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 252928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAF66C4741F for ; Wed, 30 Sep 2020 06:50:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AA14B2072E for ; Wed, 30 Sep 2020 06:50:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="u0dVGa1O" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728416AbgI3GuQ (ORCPT ); Wed, 30 Sep 2020 02:50:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725884AbgI3GuL (ORCPT ); Wed, 30 Sep 2020 02:50:11 -0400 Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 570BDC0613D0; Tue, 29 Sep 2020 23:50:10 -0700 (PDT) Received: by mail-pf1-x443.google.com with SMTP id o20so485968pfp.11; Tue, 29 Sep 2020 23:50:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sx8gkWC2/W3tgh9YK60ypbpCcfln3QjdLnGplGzjxCM=; b=u0dVGa1Ov64/HojhkM25i3zA7PAg2wk8/lBA4c5KLvobAefLZYwV4pwdP9Fqsqq+RF dRKUOlD6bV4xkD5aHrppE2BUyaA1Ut6VTe6sB0rsPtpOOdOddQXgOiA5NvJ2fTRcFC7q r/RckVt36f+NafHKMj4kxYW/YtEAOLNFn1cuJNee/4fF2hGP4Ldsn7GgBpM23u7UzY53 QwJj/U88+W5zzDrSIXrx+NfcdJcERrxdJIB54HtxerpjcVT2TKs12Vqg8HnvgAZgFJHQ OqQD3aPVGSSMQhom/jbH+ntyvZcD13852KlgSazyPmp7TrOcrwKaCSAeReCyeQkrBMgD 3nUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sx8gkWC2/W3tgh9YK60ypbpCcfln3QjdLnGplGzjxCM=; b=hlTAavtgVC9WjAXfjl6LD0zflp0UF42IZHXHRsqe+rBOZHgt7lrIMPlblp8dXZz0hz B2LhjSkvEO4MUvPL2a8EBREmUbEiTFa3BkV+Fqc3NX2ttoa1FR4Q+3tgxA5nGMwA9+ms BeZIk9zliwaXIJtS0+8Fi9aIlFJM4vuzcyOL8410PgzljMmerIkg6El5WYcUHvSzNhuX wI4bKTUingqExl9Ab2MCwCZL3L413Ffvwh9rdPjU2pSV/7cupmSeJzWe0j032OtMRJtl Vwmja1/16tHu9mmBh8Vfq3Uz8iN2XqQDqCq7rj3k9aaGkYwgpVU0jnWXqsmoCPpMmwtB gE3A== X-Gm-Message-State: AOAM531ATLlLGtt3kLpRmdNj3MbfdW7jSmGP/nMjcHspbZYvA9LCdShE KcX8ghNrNKg6uB0BoiopjHa0ZH8TiVM= X-Google-Smtp-Source: ABdhPJxSR+P8QylGFykX8Hwvl27fIOnZf/hSQTugRmEmysTLBdmHXm2q0T/7G7gkdOhIMPTKbyyRCw== X-Received: by 2002:aa7:9583:0:b029:142:2501:35cd with SMTP id z3-20020aa795830000b0290142250135cdmr1364984pfj.45.1601448609775; Tue, 29 Sep 2020 23:50:09 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id 190sm1100865pfy.22.2020.09.29.23.50.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Sep 2020 23:50:09 -0700 (PDT) From: Nick Terrell To: Herbert Xu Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, squashfs-devel@lists.sourceforge.net, linux-f2fs-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org, Kernel Team , Nick Terrell , Nick Terrell , Chris Mason , Petr Malat , Johannes Weiner , Niket Agarwal , Yann Collet , Christoph Hellwig Subject: [PATCH v4 6/9] f2fs: zstd: Switch to the zstd-1.4.6 API Date: Tue, 29 Sep 2020 23:53:15 -0700 Message-Id: <20200930065318.3326526-7-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200930065318.3326526-1-nickrterrell@gmail.com> References: <20200930065318.3326526-1-nickrterrell@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Nick Terrell Move away from the compatibility wrapper to the zstd-1.4.6 API. This code is more efficient because it uses the single-pass API instead of the streaming API. The streaming API is not necessary because the whole input and output buffers are available. This saves memory because we don't need to allocate a buffer for the window. It is also more efficient because it saves unnecessary memcpy calls. Compression memory increases from 168 KB to 204 KB because upstream uses slightly more memory. Decompression memory decreases from 1.4 MB to 158 KB. Signed-off-by: Nick Terrell --- fs/f2fs/compress.c | 102 +++++++++++++++++---------------------------- 1 file changed, 38 insertions(+), 64 deletions(-) diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index e056f3a2b404..b79efce81651 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -11,7 +11,8 @@ #include #include #include -#include +#include +#include #include "f2fs.h" #include "node.h" @@ -298,21 +299,21 @@ static const struct f2fs_compress_ops f2fs_lz4_ops = { static int zstd_init_compress_ctx(struct compress_ctx *cc) { ZSTD_parameters params; - ZSTD_CStream *stream; + ZSTD_CCtx *ctx; void *workspace; unsigned int workspace_size; params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, cc->rlen, 0); - workspace_size = ZSTD_CStreamWorkspaceBound(params.cParams); + workspace_size = ZSTD_estimateCCtxSize_usingCParams(params.cParams); workspace = f2fs_kvmalloc(F2FS_I_SB(cc->inode), workspace_size, GFP_NOFS); if (!workspace) return -ENOMEM; - stream = ZSTD_initCStream(params, 0, workspace, workspace_size); - if (!stream) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initCStream failed\n", + ctx = ZSTD_initStaticCCtx(workspace, workspace_size); + if (!ctx) { + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_inittaticCStream failed\n", KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, __func__); kvfree(workspace); @@ -320,7 +321,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) } cc->private = workspace; - cc->private2 = stream; + cc->private2 = ctx; cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE; return 0; @@ -335,65 +336,48 @@ static void zstd_destroy_compress_ctx(struct compress_ctx *cc) static int zstd_compress_pages(struct compress_ctx *cc) { - ZSTD_CStream *stream = cc->private2; - ZSTD_inBuffer inbuf; - ZSTD_outBuffer outbuf; - int src_size = cc->rlen; - int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; - int ret; - - inbuf.pos = 0; - inbuf.src = cc->rbuf; - inbuf.size = src_size; - - outbuf.pos = 0; - outbuf.dst = cc->cbuf->cdata; - outbuf.size = dst_size; - - ret = ZSTD_compressStream(stream, &outbuf, &inbuf); - if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n", - KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); - return -EIO; - } - - ret = ZSTD_endStream(stream, &outbuf); + ZSTD_CCtx *ctx = cc->private2; + const size_t src_size = cc->rlen; + const size_t dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; + ZSTD_parameters params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, src_size, 0); + size_t ret; + + ret = ZSTD_compress_advanced( + ctx, cc->cbuf->cdata, dst_size, cc->rbuf, src_size, NULL, 0, params); if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_endStream returned %d\n", + /* + * there is compressed data remained in intermediate buffer due to + * no more space in cbuf.cdata + */ + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) + return -EAGAIN; + /* other compression errors return -EIO */ + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compress_advanced failed, err: %s\n", KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); + __func__, ZSTD_getErrorName(ret)); return -EIO; } - /* - * there is compressed data remained in intermediate buffer due to - * no more space in cbuf.cdata - */ - if (ret) - return -EAGAIN; - - cc->clen = outbuf.pos; + cc->clen = ret; return 0; } static int zstd_init_decompress_ctx(struct decompress_io_ctx *dic) { - ZSTD_DStream *stream; + ZSTD_DCtx *ctx; void *workspace; unsigned int workspace_size; - workspace_size = ZSTD_DStreamWorkspaceBound(MAX_COMPRESS_WINDOW_SIZE); + workspace_size = ZSTD_estimateDCtxSize(); workspace = f2fs_kvmalloc(F2FS_I_SB(dic->inode), workspace_size, GFP_NOFS); if (!workspace) return -ENOMEM; - stream = ZSTD_initDStream(MAX_COMPRESS_WINDOW_SIZE, - workspace, workspace_size); - if (!stream) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initDStream failed\n", + ctx = ZSTD_initStaticDCtx(workspace, workspace_size); + if (!ctx) { + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initStaticDCtx failed\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, __func__); kvfree(workspace); @@ -401,7 +385,7 @@ static int zstd_init_decompress_ctx(struct decompress_io_ctx *dic) } dic->private = workspace; - dic->private2 = stream; + dic->private2 = ctx; return 0; } @@ -415,28 +399,18 @@ static void zstd_destroy_decompress_ctx(struct decompress_io_ctx *dic) static int zstd_decompress_pages(struct decompress_io_ctx *dic) { - ZSTD_DStream *stream = dic->private2; - ZSTD_inBuffer inbuf; - ZSTD_outBuffer outbuf; - int ret; - - inbuf.pos = 0; - inbuf.src = dic->cbuf->cdata; - inbuf.size = dic->clen; - - outbuf.pos = 0; - outbuf.dst = dic->rbuf; - outbuf.size = dic->rlen; + ZSTD_DCtx *ctx = dic->private2; + size_t ret; - ret = ZSTD_decompressStream(stream, &outbuf, &inbuf); + ret = ZSTD_decompressDCtx(ctx, dic->rbuf, dic->rlen, dic->cbuf->cdata, dic->clen); if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n", + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_decompressDCtx failed, err: %s\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); + __func__, ZSTD_getErrorName(ret)); return -EIO; } - if (dic->rlen != outbuf.pos) { + if (dic->rlen != ret) { printk_ratelimited("%sF2FS-fs (%s): %s ZSTD invalid rlen:%zu, " "expected:%lu\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, From patchwork Wed Sep 30 06:53:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 252927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5D71C4727F for ; Wed, 30 Sep 2020 06:50:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6D4DA2075F for ; Wed, 30 Sep 2020 06:50:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Wu1wW7kG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728501AbgI3Guk (ORCPT ); Wed, 30 Sep 2020 02:50:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725798AbgI3GuS (ORCPT ); Wed, 30 Sep 2020 02:50:18 -0400 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68EF9C061755; Tue, 29 Sep 2020 23:50:17 -0700 (PDT) Received: by mail-pl1-x641.google.com with SMTP id x5so421236plo.6; Tue, 29 Sep 2020 23:50:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=krGpYOVpUat0gERMlMR1P1lhK//ROWK8HUHnVTxyKd4=; b=Wu1wW7kGVUX2hnBN6KUwy3RNOwJdKCSWkRfqt1BdsxEV1wgVF6VdZ8NL/HbGwWmxHw AdAXq8grvCkSPFY+1tDKWNvtsGuzSzgeuPhxb1wFZYdMsiAq9SID+Pql3YPWUloILMJf fcUgmzRPqruLLP93zxemu5YuwJIqfYbUJwWg4cfweSSj6P+zHCTe7pWXONkForHoTIUb RMvm4XvDYv4ejTDvAOt+BCRBzgm2Q97XU1LRQvw2P9NOmrBsJd0sCY7wgBy71PLCEqLB Ek484Owhn5P/s0GF9eKQUFYGK2LO7RhES17Vo4zbqACdtEKX26qKGsx+lV1bKUlbxuEI J6OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=krGpYOVpUat0gERMlMR1P1lhK//ROWK8HUHnVTxyKd4=; b=aKZQSz25XoPNg2Ea3w86hyQoZ8NAo2V78gEULshXPtqxrfcRVdw80EeX1scdbNvxS2 uLy9bOIo+BV2uoKIvWc4LTPas5i4Zc9gxo4sr9T8mlQg61hMuNV/wWMxtpJejDZWtnDI 5wFFun9DMsA1aJyV4K2wQZwuMLyg/wRffG79fM0LVvueSdbHBYBpxyKAd3J1M08qO8Ux Uqm612LoDs21PdDD3W86djwA82NMrjmTj37FYhG8ku1mQ7cGkH3AiRenbRpFGbY4sn65 igee+qQEo4uEsCGLHu3QYpaBRaFlxtxb6wqxyTIKTz3sUHxZMrwSC2igYwIUwY0sAb/y NHOw== X-Gm-Message-State: AOAM530obE50/cgpfeZDMg37iF+B52IDY1eRRAiWVLIVJFyGzvzPETHo 15DtMy+ZS7MIHHW5pTl4bYc= X-Google-Smtp-Source: ABdhPJxc5RwMM9+nVdS8pkuhMEUvsg+HwSRqEGKdOiO7375+3pYB55ceWSEbfhp0KE1MZXbDtDiXOA== X-Received: by 2002:a17:902:ee06:b029:d1:8c50:b1ba with SMTP id z6-20020a170902ee06b02900d18c50b1bamr1160641plb.35.1601448616906; Tue, 29 Sep 2020 23:50:16 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id 190sm1100865pfy.22.2020.09.29.23.50.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Sep 2020 23:50:16 -0700 (PDT) From: Nick Terrell To: Herbert Xu Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, squashfs-devel@lists.sourceforge.net, linux-f2fs-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org, Kernel Team , Nick Terrell , Nick Terrell , Chris Mason , Petr Malat , Johannes Weiner , Niket Agarwal , Yann Collet , Christoph Hellwig Subject: [PATCH v4 8/9] lib: unzstd: Switch to the zstd-1.4.6 API Date: Tue, 29 Sep 2020 23:53:17 -0700 Message-Id: <20200930065318.3326526-9-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200930065318.3326526-1-nickrterrell@gmail.com> References: <20200930065318.3326526-1-nickrterrell@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Nick Terrell Move away from the compatibility wrapper to the zstd-1.4.6 API. This code is functionally equivalent. Signed-off-by: Nick Terrell --- lib/decompress_unzstd.c | 40 ++++++++++++++-------------------------- 1 file changed, 14 insertions(+), 26 deletions(-) diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c index a79f705f236d..d4685df0e120 100644 --- a/lib/decompress_unzstd.c +++ b/lib/decompress_unzstd.c @@ -73,7 +73,8 @@ #include #include -#include +#include +#include /* 128MB is the maximum window size supported by zstd. */ #define ZSTD_WINDOWSIZE_MAX (1 << ZSTD_WINDOWLOG_MAX) @@ -120,9 +121,9 @@ static int INIT decompress_single(const u8 *in_buf, long in_len, u8 *out_buf, long out_len, long *in_pos, void (*error)(char *x)) { - const size_t wksp_size = ZSTD_DCtxWorkspaceBound(); + const size_t wksp_size = ZSTD_estimateDCtxSize(); void *wksp = large_malloc(wksp_size); - ZSTD_DCtx *dctx = ZSTD_initDCtx(wksp, wksp_size); + ZSTD_DCtx *dctx = ZSTD_initStaticDCtx(wksp, wksp_size); int err; size_t ret; @@ -165,7 +166,6 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len, { ZSTD_inBuffer in; ZSTD_outBuffer out; - ZSTD_frameParams params; void *in_allocated = NULL; void *out_allocated = NULL; void *wksp = NULL; @@ -229,36 +229,24 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len, out.size = out_len; /* - * We need to know the window size to allocate the ZSTD_DStream. - * Since we are streaming, we need to allocate a buffer for the sliding - * window. The window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX - * (8 MB), so it is important to use the actual value so as not to - * waste memory when it is smaller. + * Zstd determines the workspace size from the window size written + * into the frame header. This ensures that we use the minimum value + * possible, since the window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX + * (1 GB), so it is very important to use the actual value. */ - ret = ZSTD_getFrameParams(¶ms, in.src, in.size); + wksp_size = ZSTD_estimateDStreamSize_fromFrame(in.src, in.size); err = handle_zstd_error(ret, error); if (err) goto out; - if (ret != 0) { - error("ZSTD-compressed data has an incomplete frame header"); - err = -1; - goto out; - } - if (params.windowSize > ZSTD_WINDOWSIZE_MAX) { - error("ZSTD-compressed data has too large a window size"); + wksp = large_malloc(wksp_size); + if (wksp == NULL) { + error("Out of memory while allocating ZSTD_DStream"); err = -1; goto out; } - - /* - * Allocate the ZSTD_DStream now that we know how much memory is - * required. - */ - wksp_size = ZSTD_DStreamWorkspaceBound(params.windowSize); - wksp = large_malloc(wksp_size); - dstream = ZSTD_initDStream(params.windowSize, wksp, wksp_size); + dstream = ZSTD_initStaticDStream(wksp, wksp_size); if (dstream == NULL) { - error("Out of memory while allocating ZSTD_DStream"); + error("ZSTD_initStaticDStream failed"); err = -1; goto out; }