From patchwork Wed Sep 16 03:42:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 253035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9600FC433E2 for ; Wed, 16 Sep 2020 03:40:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5440B21D94 for ; Wed, 16 Sep 2020 03:40:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ug5EbU2Q" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726252AbgIPDkP (ORCPT ); Tue, 15 Sep 2020 23:40:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726028AbgIPDkO (ORCPT ); Tue, 15 Sep 2020 23:40:14 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58D60C06174A; Tue, 15 Sep 2020 20:40:14 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id kk9so831684pjb.2; Tue, 15 Sep 2020 20:40:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=S+BtKFBofP+2qj1vMeMT8ZFQXkI1C+ove1wGqcVzSh0=; b=Ug5EbU2QiuMXX8B4ONwOPlBHJ6tNwfXqonHrLrKYjezkizaz+GZoC7obTu98823d5V LAU9zo+1JFEW7wtzZlRIC0fhpVj02eWFq58FmEaqAyPjBJebu9Gwy2nmn3NItZC0gB0d 0l58mLR8C16/sXFl/Q4WWpf/5KrNCay2qSnycyRJwx49TfLyWDoDo6z98ycEJQzkI+Qv fp7SHqdiicSPgLknqbaah9VR5W07X0nUF6s52wDkTeSB8R7UVZGEOzXq1hN8HDvHTERs KuLPrn57F7eS/NKafRu3DZqVlCZIxDCeYM1XZExomw0tta4Xgers0pkpslykJI/U8VdC /7/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S+BtKFBofP+2qj1vMeMT8ZFQXkI1C+ove1wGqcVzSh0=; b=sdKIgLv///eBCakuai1MFXAvGPvaohkTlfIBVn/ZBziBagPMy53+8bl3/42oMqU3Le zCiwCSTaowHNf9P3N0UGw+5HtpOUHvLxynqE/hBaQsQioo+eeEDed1TG2P2HLuNj4x7W bw9dBdKBW/pDj9M+VUQGxDC0rlKTmMxCWDtyKEQgz98uqhZLg5pKV8EEPszqxFr4t6/y sizWbj/fbcDtaqE7EKaeo4rBTH3BfJV7t8VhDsz7VS3PdFc9QgzJZ5i0XniE/cRoaYD2 r0QCRxB9oE8oZaLVxiSy/PxqGAf9TuWFycew0go2rjSAj92ucgn04+Hl+wlsX/gmpP85 0X0w== X-Gm-Message-State: AOAM532V6hccfyi8+AUs7Q4IR3Uh3r0YQOe7q3SorLLPhp8W23oUoSRE SuWmXo8+OVkSPU8NqtZ7Mq4= X-Google-Smtp-Source: ABdhPJx/OhIWKQmofslaKsOosAbxPzPLyW41LggGay5fxkAAuxmLW7t8MXJuK7NJYchNtzUR0VC5eg== X-Received: by 2002:a17:902:fe13:b029:d0:89f4:6226 with SMTP id g19-20020a170902fe13b02900d089f46226mr22457797plj.14.1600227613759; Tue, 15 Sep 2020 20:40:13 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id i20sm12856635pgk.77.2020.09.15.20.40.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 20:40:13 -0700 (PDT) From: Nick Terrell To: Herbert Xu Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, squashfs-devel@lists.sourceforge.net, linux-f2fs-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org, Kernel Team , Nick Terrell , Nick Terrell , Chris Mason , Petr Malat , Johannes Weiner , Niket Agarwal , Yann Collet Subject: [PATCH 1/9] lib: zstd: Add zstd compatibility wrapper Date: Tue, 15 Sep 2020 20:42:54 -0700 Message-Id: <20200916034307.2092020-2-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200916034307.2092020-1-nickrterrell@gmail.com> References: <20200916034307.2092020-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Nick Terrell Adds zstd_compat.h which provides the necessary functions from the current zstd.h API. It is only active for zstd versions 1.4.6 and newer. That means it is disabled currently, but will become active when a later patch in this series updates the zstd library in the kernel to 1.4.6. This header allows the zstd upgrade to 1.4.6 without changing any callers, since they all include zstd through the compatibility wrapper. Later patches in this series transition each caller away from the compatibility wrapper. After all the callers have been transitioned away from the compatibility wrapper, the final patch in this series deletes it. Signed-off-by: Nick Terrell --- crypto/zstd.c | 2 +- fs/btrfs/zstd.c | 2 +- fs/f2fs/compress.c | 2 +- fs/squashfs/zstd_wrapper.c | 2 +- include/linux/zstd_compat.h | 112 ++++++++++++++++++++++++++++++++++++ lib/decompress_unzstd.c | 2 +- 6 files changed, 117 insertions(+), 5 deletions(-) create mode 100644 include/linux/zstd_compat.h diff --git a/crypto/zstd.c b/crypto/zstd.c index 1a3309f066f7..dcda3cad3b5c 100644 --- a/crypto/zstd.c +++ b/crypto/zstd.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include diff --git a/fs/btrfs/zstd.c b/fs/btrfs/zstd.c index 9a4871636c6c..a7367ff573d4 100644 --- a/fs/btrfs/zstd.c +++ b/fs/btrfs/zstd.c @@ -16,7 +16,7 @@ #include #include #include -#include +#include #include "misc.h" #include "compression.h" #include "ctree.h" diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index 1dfb126a0cb2..e056f3a2b404 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include "f2fs.h" #include "node.h" diff --git a/fs/squashfs/zstd_wrapper.c b/fs/squashfs/zstd_wrapper.c index b7cb1faa652d..f8c512a6204e 100644 --- a/fs/squashfs/zstd_wrapper.c +++ b/fs/squashfs/zstd_wrapper.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include "squashfs_fs.h" diff --git a/include/linux/zstd_compat.h b/include/linux/zstd_compat.h new file mode 100644 index 000000000000..11acf14d9d70 --- /dev/null +++ b/include/linux/zstd_compat.h @@ -0,0 +1,112 @@ +/* + * Copyright (c) 2016-present, Facebook, Inc. + * All rights reserved. + * + * This source code is licensed under the BSD-style license found in the + * LICENSE file in the root directory of https://github.com/facebook/zstd. + * An additional grant of patent rights can be found in the PATENTS file in the + * same directory. + * + * This program is free software; you can redistribute it and/or modify it under + * the terms of the GNU General Public License version 2 as published by the + * Free Software Foundation. This program is dual-licensed; you may select + * either version 2 of the GNU General Public License ("GPL") or BSD license + * ("BSD"). + */ + +#ifndef ZSTD_COMPAT_H +#define ZSTD_COMPAT_H + +#include + +#if defined(ZSTD_VERSION_NUMBER) && (ZSTD_VERSION_NUMBER >= 10406) +/* + * This header provides backwards compatibility for the zstd-1.4.6 library + * upgrade. This header allows us to upgrade the zstd library version without + * modifying any callers. Then we will migrate callers from the compatibility + * wrapper one at a time until none remain. At which point we will delete this + * header. + * + * It is temporary and will be deleted once the upgrade is complete. + */ + +#include + +static inline size_t ZSTD_CCtxWorkspaceBound(ZSTD_compressionParameters compression_params) +{ + return ZSTD_estimateCCtxSize_usingCParams(compression_params); +} + +static inline size_t ZSTD_CStreamWorkspaceBound(ZSTD_compressionParameters compression_params) +{ + return ZSTD_estimateCStreamSize_usingCParams(compression_params); +} + +static inline size_t ZSTD_DCtxWorkspaceBound(void) +{ + return ZSTD_estimateDCtxSize(); +} + +static inline size_t ZSTD_DStreamWorkspaceBound(unsigned long long window_size) +{ + return ZSTD_estimateDStreamSize(window_size); +} + +static inline ZSTD_CCtx* ZSTD_initCCtx(void* wksp, size_t wksp_size) +{ + if (wksp == NULL) + return NULL; + return ZSTD_initStaticCCtx(wksp, wksp_size); +} + +static inline ZSTD_CStream* ZSTD_initCStream_compat(ZSTD_parameters params, size_t pledged_src_size, void* wksp, size_t wksp_size) +{ + ZSTD_CStream* cstream; + size_t ret; + + if (wksp == NULL) + return NULL; + + cstream = ZSTD_initStaticCStream(wksp, wksp_size); + if (cstream == NULL) + return NULL; + + ret = ZSTD_initCStream_advanced(cstream, NULL, 0, params, pledged_src_size); + if (ZSTD_isError(ret)) + return NULL; + + return cstream; +} +#define ZSTD_initCStream ZSTD_initCStream_compat + +static inline ZSTD_DCtx* ZSTD_initDCtx(void* wksp, size_t wksp_size) +{ + if (wksp == NULL) + return NULL; + return ZSTD_initStaticDCtx(wksp, wksp_size); +} + +static inline ZSTD_DStream* ZSTD_initDStream_compat(unsigned long long window_size, void* wksp, size_t wksp_size) +{ + if (wksp == NULL) + return NULL; + (void)window_size; + return ZSTD_initStaticDStream(wksp, wksp_size); +} +#define ZSTD_initDStream ZSTD_initDStream_compat + +typedef ZSTD_frameHeader ZSTD_frameParams; + +static inline size_t ZSTD_getFrameParams(ZSTD_frameParams* frame_params, const void* src, size_t src_size) +{ + return ZSTD_getFrameHeader(frame_params, src, src_size); +} + +static inline size_t ZSTD_compressCCtx_compat(ZSTD_CCtx* cctx, void* dst, size_t dst_capacity, const void* src, size_t src_size, ZSTD_parameters params) +{ + return ZSTD_compress_advanced(cctx, dst, dst_capacity, src, src_size, NULL, 0, params); +} +#define ZSTD_compressCCtx ZSTD_compressCCtx_compat + +#endif /* ZSTD_VERSION_NUMBER >= 10406 */ +#endif /* ZSTD_COMPAT_H */ diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c index 0ad2c15479ed..dbc290af26b4 100644 --- a/lib/decompress_unzstd.c +++ b/lib/decompress_unzstd.c @@ -77,7 +77,7 @@ #include #include -#include +#include /* 128MB is the maximum window size supported by zstd. */ #define ZSTD_WINDOWSIZE_MAX (1 << ZSTD_WINDOWLOG_MAX) From patchwork Wed Sep 16 03:43:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 253034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3E04C433E2 for ; Wed, 16 Sep 2020 03:44:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A2F3B221E3 for ; Wed, 16 Sep 2020 03:44:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KE/LPv47" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726382AbgIPDnc (ORCPT ); Tue, 15 Sep 2020 23:43:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726362AbgIPDnR (ORCPT ); Tue, 15 Sep 2020 23:43:17 -0400 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1288DC061797; Tue, 15 Sep 2020 20:43:08 -0700 (PDT) Received: by mail-pl1-x643.google.com with SMTP id x18so2443670pll.6; Tue, 15 Sep 2020 20:43:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4G5dTWCVv1NOHS3OxfHw4VPNamJGvm8WlcoiqGKDEiA=; b=KE/LPv47oXVIVzAPme4qL2M2JUEKAM2ruFw4CifP+OrXovLTE3+9OJAdmS1aZffxPs 354JRwOdZ8R7NZY3hm16TAnGV3u68aenDumQ+vfUgmw+/ffLntLbL+A1vrHIO65uUKDo yY/kx2DjivoBs6yeEehtTDpTAqMs3cGz/gtkBy+4wORrONq0GZXesiN1s2UhAltESTJj 2KqpIIcDHPbiC2pzjXsRPbHL8eCdOUTBnfNx/N8f+oHRm25gR21nWp4KeGrtZ41JqiwS MvG3E8nu58j4wEOcSXXFb3ozVyQkC3cEAqCBIhaV+1lOCCZjrAL1Y60weA4iG46KamoQ QS4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4G5dTWCVv1NOHS3OxfHw4VPNamJGvm8WlcoiqGKDEiA=; b=i2gqcOPlYqkjeEUIjlVC7VUycDkdAXD9udBBYMW6KqRxbymVA27Hh4PTSp1j9i1coi 6yCmhu8MaUnaGF2vv0/PdTBWdOLpcns4lDnf2uJfBFaBkIkHkjul39V/qiwRU2/EvHfK 9Mg915vdPLEbapX1K4TVPPDWqoRTXrOs0Dz5o0O3IoUa3F5yCoTXe+QzyuoQGOwkBFkI gqBUNWvGKlSXQyvat51tg5Bf93nWQRA77a6Hi9UB1NZ+t6bNdI2pMZq0uxi3DgVZzDM1 +57fxT8DGv2vX5DTtcCIXD9/c/KnqRY3oFOj36CGE2I2XST0egU9uh+hv6PwcUMtmvLP zvQw== X-Gm-Message-State: AOAM533rOa9OWR3hZCMjt2xT+ZJmIFss2Pktkuya/NaNlQddwUFGkl6C 6fAbN92UBk/cS+lGBFwdKJU= X-Google-Smtp-Source: ABdhPJxxMG1sHnH1sAyJe0cuaB6biIqT0wcpqtHMJ+CXOUKYNB2jPzFXuh7kU4yid2AyFqFG+yCFeQ== X-Received: by 2002:a17:902:8306:b029:d0:cbe1:e7aa with SMTP id bd6-20020a1709028306b02900d0cbe1e7aamr22151055plb.27.1600227787535; Tue, 15 Sep 2020 20:43:07 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id i20sm12856635pgk.77.2020.09.15.20.43.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 20:43:07 -0700 (PDT) From: Nick Terrell To: Herbert Xu Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, squashfs-devel@lists.sourceforge.net, linux-f2fs-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org, Kernel Team , Nick Terrell , Nick Terrell , Chris Mason , Petr Malat , Johannes Weiner , Niket Agarwal , Yann Collet Subject: [PATCH 6/9] f2fs: zstd: Switch to the zstd-1.4.6 API Date: Tue, 15 Sep 2020 20:43:01 -0700 Message-Id: <20200916034307.2092020-9-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200916034307.2092020-1-nickrterrell@gmail.com> References: <20200916034307.2092020-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Nick Terrell Move away from the compatibility wrapper to the zstd-1.4.6 API. This code is more efficient because it uses the single-pass API instead of the streaming API. The streaming API is not necessary because the whole input and output buffers are available. This saves memory because we don't need to allocate a buffer for the window. It is also more efficient because it saves unnecessary memcpy calls. I've had problems testing this code because I see data truncation before and after this patchset. Help testing this patch would be much appreciated. Signed-off-by: Nick Terrell --- fs/f2fs/compress.c | 102 +++++++++++++++++---------------------------- 1 file changed, 38 insertions(+), 64 deletions(-) diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index e056f3a2b404..b79efce81651 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -11,7 +11,8 @@ #include #include #include -#include +#include +#include #include "f2fs.h" #include "node.h" @@ -298,21 +299,21 @@ static const struct f2fs_compress_ops f2fs_lz4_ops = { static int zstd_init_compress_ctx(struct compress_ctx *cc) { ZSTD_parameters params; - ZSTD_CStream *stream; + ZSTD_CCtx *ctx; void *workspace; unsigned int workspace_size; params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, cc->rlen, 0); - workspace_size = ZSTD_CStreamWorkspaceBound(params.cParams); + workspace_size = ZSTD_estimateCCtxSize_usingCParams(params.cParams); workspace = f2fs_kvmalloc(F2FS_I_SB(cc->inode), workspace_size, GFP_NOFS); if (!workspace) return -ENOMEM; - stream = ZSTD_initCStream(params, 0, workspace, workspace_size); - if (!stream) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initCStream failed\n", + ctx = ZSTD_initStaticCCtx(workspace, workspace_size); + if (!ctx) { + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_inittaticCStream failed\n", KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, __func__); kvfree(workspace); @@ -320,7 +321,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) } cc->private = workspace; - cc->private2 = stream; + cc->private2 = ctx; cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE; return 0; @@ -335,65 +336,48 @@ static void zstd_destroy_compress_ctx(struct compress_ctx *cc) static int zstd_compress_pages(struct compress_ctx *cc) { - ZSTD_CStream *stream = cc->private2; - ZSTD_inBuffer inbuf; - ZSTD_outBuffer outbuf; - int src_size = cc->rlen; - int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; - int ret; - - inbuf.pos = 0; - inbuf.src = cc->rbuf; - inbuf.size = src_size; - - outbuf.pos = 0; - outbuf.dst = cc->cbuf->cdata; - outbuf.size = dst_size; - - ret = ZSTD_compressStream(stream, &outbuf, &inbuf); - if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n", - KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); - return -EIO; - } - - ret = ZSTD_endStream(stream, &outbuf); + ZSTD_CCtx *ctx = cc->private2; + const size_t src_size = cc->rlen; + const size_t dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; + ZSTD_parameters params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, src_size, 0); + size_t ret; + + ret = ZSTD_compress_advanced( + ctx, cc->cbuf->cdata, dst_size, cc->rbuf, src_size, NULL, 0, params); if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_endStream returned %d\n", + /* + * there is compressed data remained in intermediate buffer due to + * no more space in cbuf.cdata + */ + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) + return -EAGAIN; + /* other compression errors return -EIO */ + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compress_advanced failed, err: %s\n", KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); + __func__, ZSTD_getErrorName(ret)); return -EIO; } - /* - * there is compressed data remained in intermediate buffer due to - * no more space in cbuf.cdata - */ - if (ret) - return -EAGAIN; - - cc->clen = outbuf.pos; + cc->clen = ret; return 0; } static int zstd_init_decompress_ctx(struct decompress_io_ctx *dic) { - ZSTD_DStream *stream; + ZSTD_DCtx *ctx; void *workspace; unsigned int workspace_size; - workspace_size = ZSTD_DStreamWorkspaceBound(MAX_COMPRESS_WINDOW_SIZE); + workspace_size = ZSTD_estimateDCtxSize(); workspace = f2fs_kvmalloc(F2FS_I_SB(dic->inode), workspace_size, GFP_NOFS); if (!workspace) return -ENOMEM; - stream = ZSTD_initDStream(MAX_COMPRESS_WINDOW_SIZE, - workspace, workspace_size); - if (!stream) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initDStream failed\n", + ctx = ZSTD_initStaticDCtx(workspace, workspace_size); + if (!ctx) { + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initStaticDCtx failed\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, __func__); kvfree(workspace); @@ -401,7 +385,7 @@ static int zstd_init_decompress_ctx(struct decompress_io_ctx *dic) } dic->private = workspace; - dic->private2 = stream; + dic->private2 = ctx; return 0; } @@ -415,28 +399,18 @@ static void zstd_destroy_decompress_ctx(struct decompress_io_ctx *dic) static int zstd_decompress_pages(struct decompress_io_ctx *dic) { - ZSTD_DStream *stream = dic->private2; - ZSTD_inBuffer inbuf; - ZSTD_outBuffer outbuf; - int ret; - - inbuf.pos = 0; - inbuf.src = dic->cbuf->cdata; - inbuf.size = dic->clen; - - outbuf.pos = 0; - outbuf.dst = dic->rbuf; - outbuf.size = dic->rlen; + ZSTD_DCtx *ctx = dic->private2; + size_t ret; - ret = ZSTD_decompressStream(stream, &outbuf, &inbuf); + ret = ZSTD_decompressDCtx(ctx, dic->rbuf, dic->rlen, dic->cbuf->cdata, dic->clen); if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n", + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_decompressDCtx failed, err: %s\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); + __func__, ZSTD_getErrorName(ret)); return -EIO; } - if (dic->rlen != outbuf.pos) { + if (dic->rlen != ret) { printk_ratelimited("%sF2FS-fs (%s): %s ZSTD invalid rlen:%zu, " "expected:%lu\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, From patchwork Wed Sep 16 03:43:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 253033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D076BC433E2 for ; Wed, 16 Sep 2020 03:44:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 91E3C21D94 for ; Wed, 16 Sep 2020 03:44:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KEm0UVHF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726155AbgIPDoS (ORCPT ); Tue, 15 Sep 2020 23:44:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726360AbgIPDnR (ORCPT ); Tue, 15 Sep 2020 23:43:17 -0400 Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C78FCC061351; Tue, 15 Sep 2020 20:43:13 -0700 (PDT) Received: by mail-pg1-x544.google.com with SMTP id d13so3118404pgl.6; Tue, 15 Sep 2020 20:43:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=290Bo0sJTh/t678nA8yWOfHRoSh5E7Ekj1ME0hd0Klk=; b=KEm0UVHF1Z/Vd0ckYj7yXuuG2AXFzwVDsLdcdoVQhNvG6DuSNx/aQJ6PPeHhVcQSuZ u2VxPlPV30RHTwhHAM5T8aC/Mftiyqsy8vGBRLuXefq5lF0QCKk6+2e7BbkPxBsgv74a BCBCLAHFqhCI4dTN3g5qGZlL3bUQSf/I/eHUoF+SImgPfbAC2vfI3b+1Dr9PgEbJzJz8 6Pee+NmrM9WC0Cms5Zua+0mBnMCugV6xej/EplN3TrcJayU/UwLp+pdqpCgHSNMZvWOK Y+lDOzxqJeud3zPIRok5iX50CwArL4vaEJsMzHCme7lPl5RigPxKb/V8y7XCvaDFJri2 mY+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=290Bo0sJTh/t678nA8yWOfHRoSh5E7Ekj1ME0hd0Klk=; b=RAn11rJsHuoLA8u+7kcHHTrt7t7xxRJNLna6OjpmFxh7/IhaMJaI5B8Ik+S/DYyuGF lgWXWkrRLLN0VC0AjQgDHwcpHUg5JhuYSNkSa+tweJmly6r3aBG0mKIxtoGfr/87o8oE yGeUgWE9e/YGzzhikyQdEzi6p4v1zBGt361sZfvavIZonUF1sHkS2zUlVcy09IOoHepN 1CqEkU1yjXr2jrYnB9eVBhSuUvO6K5U7AcSDkkCRZDBWdg3AYvxovhdFvPJi0e/QKrbk rtgMetpeDL+K+k/aclW5OdrHkMs/sJGjpTjfJyFgaHhN90FwsW3Domb6HhU86ynoK0V7 tyXg== X-Gm-Message-State: AOAM530vUfXmFkS17RsxFdXkJmu/yUdtvlu7fqlRPTqfVdtCY+auwKW/ lRgS6dphIu8eeG8YlRFYZjI= X-Google-Smtp-Source: ABdhPJwz0tJz5V2Zp6LFHtcG67dfL9Sa620SacOp3zJK/NqJ2OB/Y4U6PnsTvIhiG018YR6FJ37l/w== X-Received: by 2002:aa7:948d:0:b029:13e:cb8d:60e0 with SMTP id z13-20020aa7948d0000b029013ecb8d60e0mr21545205pfk.9.1600227792869; Tue, 15 Sep 2020 20:43:12 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id i20sm12856635pgk.77.2020.09.15.20.43.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 20:43:12 -0700 (PDT) From: Nick Terrell To: Herbert Xu Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, squashfs-devel@lists.sourceforge.net, linux-f2fs-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org, Kernel Team , Nick Terrell , Nick Terrell , Chris Mason , Petr Malat , Johannes Weiner , Niket Agarwal , Yann Collet Subject: [PATCH 7/9] squashfs: zstd: Switch to the zstd-1.4.6 API Date: Tue, 15 Sep 2020 20:43:03 -0700 Message-Id: <20200916034307.2092020-11-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200916034307.2092020-1-nickrterrell@gmail.com> References: <20200916034307.2092020-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Nick Terrell Move away from the compatibility wrapper to the zstd-1.4.6 API. This code is functionally equivalent. Signed-off-by: Nick Terrell --- fs/squashfs/zstd_wrapper.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/fs/squashfs/zstd_wrapper.c b/fs/squashfs/zstd_wrapper.c index f8c512a6204e..add582409866 100644 --- a/fs/squashfs/zstd_wrapper.c +++ b/fs/squashfs/zstd_wrapper.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include "squashfs_fs.h" @@ -34,7 +34,7 @@ static void *zstd_init(struct squashfs_sb_info *msblk, void *buff) goto failed; wksp->window_size = max_t(size_t, msblk->block_size, SQUASHFS_METADATA_SIZE); - wksp->mem_size = ZSTD_DStreamWorkspaceBound(wksp->window_size); + wksp->mem_size = ZSTD_estimateDStreamSize(wksp->window_size); wksp->mem = vmalloc(wksp->mem_size); if (wksp->mem == NULL) goto failed; @@ -71,7 +71,7 @@ static int zstd_uncompress(struct squashfs_sb_info *msblk, void *strm, struct bvec_iter_all iter_all = {}; struct bio_vec *bvec = bvec_init_iter_all(&iter_all); - stream = ZSTD_initDStream(wksp->window_size, wksp->mem, wksp->mem_size); + stream = ZSTD_initStaticDStream(wksp->mem, wksp->mem_size); if (!stream) { ERROR("Failed to initialize zstd decompressor\n"); @@ -122,8 +122,7 @@ static int zstd_uncompress(struct squashfs_sb_info *msblk, void *strm, break; if (ZSTD_isError(zstd_err)) { - ERROR("zstd decompression error: %d\n", - (int)ZSTD_getErrorCode(zstd_err)); + ERROR("zstd decompression error: %s\n", ZSTD_getErrorName(zstd_err)); error = -EIO; break; } From patchwork Wed Sep 16 03:43:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 253032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0590BC43461 for ; Wed, 16 Sep 2020 03:44:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BAD7421D94 for ; Wed, 16 Sep 2020 03:44:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="awgAfqMF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726412AbgIPDoU (ORCPT ); Tue, 15 Sep 2020 23:44:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726368AbgIPDnU (ORCPT ); Tue, 15 Sep 2020 23:43:20 -0400 Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3B7AC06178B; Tue, 15 Sep 2020 20:43:19 -0700 (PDT) Received: by mail-pg1-x542.google.com with SMTP id 34so3083570pgo.13; Tue, 15 Sep 2020 20:43:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=krGpYOVpUat0gERMlMR1P1lhK//ROWK8HUHnVTxyKd4=; b=awgAfqMFGK8yHA1kC0CVgZdnEU8AnnjW634aveD/YajxjIdjnyBcmfkVK0Knyqe+lX ljz+UCj2B5zOzqBdmoFJ3K5TNbOqC3eu77I5dV6SNF7fU2eObT28yeZA4crn0+H4i4FQ L/D64ccQxJb2Je2cQin3B1utHqJQ1jZnRUhemwrxt8LqTbezkc2BcdP3TmilWrS5BkIk Xix9H4iocrKO/j+YljDq79ibMq8FsvREhUAr+/yL6CPNVYMUIEPSItl/+8m/fc1LWvJR /f0c2ZAAZz1NghDogu2KP9KiUkVZO6HpIDaDQ5CQCvJ4alrO1dO+24DDWw+x0pfmAeHc YqCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=krGpYOVpUat0gERMlMR1P1lhK//ROWK8HUHnVTxyKd4=; b=O9EXHEIrMy4tLY4GINnNGzHrATwuD6+2w+Xm1JVp58Z9tijeKBlbbv5/4YvPwqUNij 6FpFkHik8pl7+7ocHkHaL0JsgNBNvOUEwtC9y6MXBPZtIA6Akx2mz1Cf9B+lSYXVV0Jt 1LgO4oVvK84F24G5fFK4eTpeOrF+z3olZBXR/sSlcA21/U2iJH8wfLO3TRxIOxj3XGR8 +ejRFibJnti71WGuGpaOVFB4J71tVie+0z+iBzIbPpN8U/0jrWLcoJcEdDkFSdGYqYwl N5+apNNFdUvR+q1R+sje32SJRfwdNfIUYuWipxsOM3zDHlG7rg52aB3LY+PoLeRRb6Mp 60tw== X-Gm-Message-State: AOAM531PIE9dZD3UieTDhkvqMlTIu9s8FoKB/5LHfUE+e9gpDw3l2LbJ W11yMMEU5jZVx12/xFwtAZE= X-Google-Smtp-Source: ABdhPJxeMIvZUhBNjhZ4RS0dPE1GL7YPRAlPaogyaHuOLDoBmglPFfzH1O1sjFJF805vobT2idx0zA== X-Received: by 2002:a62:19c4:0:b029:13e:d13d:a0fd with SMTP id 187-20020a6219c40000b029013ed13da0fdmr20703922pfz.25.1600227799254; Tue, 15 Sep 2020 20:43:19 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id i20sm12856635pgk.77.2020.09.15.20.43.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 20:43:18 -0700 (PDT) From: Nick Terrell To: Herbert Xu Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, squashfs-devel@lists.sourceforge.net, linux-f2fs-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org, Kernel Team , Nick Terrell , Nick Terrell , Chris Mason , Petr Malat , Johannes Weiner , Niket Agarwal , Yann Collet Subject: [PATCH 8/9] lib: unzstd: Switch to the zstd-1.4.6 API Date: Tue, 15 Sep 2020 20:43:05 -0700 Message-Id: <20200916034307.2092020-13-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200916034307.2092020-1-nickrterrell@gmail.com> References: <20200916034307.2092020-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Nick Terrell Move away from the compatibility wrapper to the zstd-1.4.6 API. This code is functionally equivalent. Signed-off-by: Nick Terrell --- lib/decompress_unzstd.c | 40 ++++++++++++++-------------------------- 1 file changed, 14 insertions(+), 26 deletions(-) diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c index a79f705f236d..d4685df0e120 100644 --- a/lib/decompress_unzstd.c +++ b/lib/decompress_unzstd.c @@ -73,7 +73,8 @@ #include #include -#include +#include +#include /* 128MB is the maximum window size supported by zstd. */ #define ZSTD_WINDOWSIZE_MAX (1 << ZSTD_WINDOWLOG_MAX) @@ -120,9 +121,9 @@ static int INIT decompress_single(const u8 *in_buf, long in_len, u8 *out_buf, long out_len, long *in_pos, void (*error)(char *x)) { - const size_t wksp_size = ZSTD_DCtxWorkspaceBound(); + const size_t wksp_size = ZSTD_estimateDCtxSize(); void *wksp = large_malloc(wksp_size); - ZSTD_DCtx *dctx = ZSTD_initDCtx(wksp, wksp_size); + ZSTD_DCtx *dctx = ZSTD_initStaticDCtx(wksp, wksp_size); int err; size_t ret; @@ -165,7 +166,6 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len, { ZSTD_inBuffer in; ZSTD_outBuffer out; - ZSTD_frameParams params; void *in_allocated = NULL; void *out_allocated = NULL; void *wksp = NULL; @@ -229,36 +229,24 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len, out.size = out_len; /* - * We need to know the window size to allocate the ZSTD_DStream. - * Since we are streaming, we need to allocate a buffer for the sliding - * window. The window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX - * (8 MB), so it is important to use the actual value so as not to - * waste memory when it is smaller. + * Zstd determines the workspace size from the window size written + * into the frame header. This ensures that we use the minimum value + * possible, since the window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX + * (1 GB), so it is very important to use the actual value. */ - ret = ZSTD_getFrameParams(¶ms, in.src, in.size); + wksp_size = ZSTD_estimateDStreamSize_fromFrame(in.src, in.size); err = handle_zstd_error(ret, error); if (err) goto out; - if (ret != 0) { - error("ZSTD-compressed data has an incomplete frame header"); - err = -1; - goto out; - } - if (params.windowSize > ZSTD_WINDOWSIZE_MAX) { - error("ZSTD-compressed data has too large a window size"); + wksp = large_malloc(wksp_size); + if (wksp == NULL) { + error("Out of memory while allocating ZSTD_DStream"); err = -1; goto out; } - - /* - * Allocate the ZSTD_DStream now that we know how much memory is - * required. - */ - wksp_size = ZSTD_DStreamWorkspaceBound(params.windowSize); - wksp = large_malloc(wksp_size); - dstream = ZSTD_initDStream(params.windowSize, wksp, wksp_size); + dstream = ZSTD_initStaticDStream(wksp, wksp_size); if (dstream == NULL) { - error("Out of memory while allocating ZSTD_DStream"); + error("ZSTD_initStaticDStream failed"); err = -1; goto out; }