From patchwork Tue Jul 7 16:34:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Wolf X-Patchwork-Id: 278289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9CFCC433DF for ; Tue, 7 Jul 2020 16:56:57 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A7AAA2065F for ; Tue, 7 Jul 2020 16:56:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="FVvMEkPm" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A7AAA2065F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:52974 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jsqtt-00060d-0h for qemu-devel@archiver.kernel.org; Tue, 07 Jul 2020 12:56:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39698) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jsqZR-0007ZC-3g for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:35:49 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:23227 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jsqZP-000453-JO for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:35:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1594139746; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wNdWeigsQ2eXYqWmSK4M0/4+t4vUL0zE6PnqvXT8QtI=; b=FVvMEkPm3Ub+weVijkNDee0bqRTE1F+bKGp+pazsmTbkJdf+O2BYQTPqbiMWNHt+oDwFlk MkneeBX90ycuDWshoqu89J93d4ystQpEENj4g6rlWJaYmxvLEFmsp3Q3UPDS5SsBiwP9ac kbNEtRsVTwm319WophiJIN5IyPIf9ko= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-420-BrUp9KKzOnyv7VSbHPwuoA-1; Tue, 07 Jul 2020 12:35:42 -0400 X-MC-Unique: BrUp9KKzOnyv7VSbHPwuoA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 75F27EC1BD; Tue, 7 Jul 2020 16:35:41 +0000 (UTC) Received: from linux.fritz.box.com (ovpn-114-113.ams2.redhat.com [10.36.114.113]) by smtp.corp.redhat.com (Postfix) with ESMTP id A89E471684; Tue, 7 Jul 2020 16:35:10 +0000 (UTC) From: Kevin Wolf To: qemu-block@nongnu.org Subject: [PULL 01/12] qemu-img map: Don't limit block status request size Date: Tue, 7 Jul 2020 18:34:53 +0200 Message-Id: <20200707163504.194740-2-kwolf@redhat.com> In-Reply-To: <20200707163504.194740-1-kwolf@redhat.com> References: <20200707163504.194740-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=kwolf@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/07 00:31:00 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -40 X-Spam_score: -4.1 X-Spam_bar: ---- X-Spam_report: (-4.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, peter.maydell@linaro.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from the beginning, though it only cut the maximum in half then because the interface was a signed 32 bit byte count. These days, bdrv_block_status supports a 64 bit byte count, so the arbitrary limit is even worse. On file-posix, bdrv_block_status() eventually maps to SEEK_HOLE and SEEK_DATA, which don't support a limit, but always do all of the work necessary to find the start of the next hole/data. Much of this work may be repeated if we don't use this information fully, but query with an only slightly larger offset in the next loop iteration. Therefore, if bdrv_block_status() is called in a loop, it should always pass the full number of bytes that the whole loop is interested in. This removes the arbitrary limit and speeds up 'qemu-img map' significantly on heavily fragmented images. Signed-off-by: Kevin Wolf Message-Id: <20200707144629.51235-1-kwolf@redhat.com> Reviewed-by: Eric Blake Signed-off-by: Kevin Wolf --- qemu-img.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index bdb9f6aa46..74946f81ca 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -3217,12 +3217,9 @@ static int img_map(int argc, char **argv) curr.start = start_offset; while (curr.start + curr.length < length) { int64_t offset = curr.start + curr.length; - int64_t n; + int64_t n = length - offset; - /* Probe up to 1 GiB at a time. */ - n = MIN(1 * GiB, length - offset); ret = get_block_status(bs, offset, n, &next); - if (ret < 0) { error_report("Could not read file metadata: %s", strerror(-ret)); goto out; From patchwork Tue Jul 7 16:34:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Wolf X-Patchwork-Id: 278297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D04D0C433E0 for ; Tue, 7 Jul 2020 16:47:20 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 897B220708 for ; Tue, 7 Jul 2020 16:47:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Q4aYMWJ5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 897B220708 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37908 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jsqkZ-0002dI-PB for qemu-devel@archiver.kernel.org; Tue, 07 Jul 2020 12:47:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39722) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jsqZT-0007dH-Cw for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:35:51 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:52861 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jsqZQ-00046p-LU for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:35:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1594139748; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IMcWg280qwj4dY28QYW9fIHWQGMPslj2ZzyqLEbKb/M=; b=Q4aYMWJ5RFDOrkAvRjnX2uboyz/Cgba1dE6RSCncRd1/w4jItJ7Ij/dS3uYJvdhY1jDKVz Iz3s7pzbzC6M10ovAur2dBgmE89StOTw8wKey+rsViq6Z4afsqjo+NqyzBF+9uhcFiR+KM P6z7Y9nE05q6nOgdRoaGwp5GmXYfiLA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-371-3UN0mP0wPeuUeIU-uDYGGg-1; Tue, 07 Jul 2020 12:35:44 -0400 X-MC-Unique: 3UN0mP0wPeuUeIU-uDYGGg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 093AD102C817; Tue, 7 Jul 2020 16:35:43 +0000 (UTC) Received: from linux.fritz.box.com (ovpn-114-113.ams2.redhat.com [10.36.114.113]) by smtp.corp.redhat.com (Postfix) with ESMTP id 950A97849E; Tue, 7 Jul 2020 16:35:41 +0000 (UTC) From: Kevin Wolf To: qemu-block@nongnu.org Subject: [PULL 02/12] file-posix: Mitigate file fragmentation with extent size hints Date: Tue, 7 Jul 2020 18:34:54 +0200 Message-Id: <20200707163504.194740-3-kwolf@redhat.com> In-Reply-To: <20200707163504.194740-1-kwolf@redhat.com> References: <20200707163504.194740-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=kwolf@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=kwolf@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/07 00:46:09 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -40 X-Spam_score: -4.1 X-Spam_bar: ---- X-Spam_report: (-4.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, peter.maydell@linaro.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Especially when O_DIRECT is used with image files so that the page cache indirection can't cause a merge of allocating requests, the file will fragment on the file system layer, with a potentially very small fragment size (this depends on the requests the guest sent). On Linux, fragmentation can be reduced by setting an extent size hint when creating the file (at least on XFS, it can't be set any more after the first extent has been allocated), basically giving raw files a "cluster size" for allocation. This adds a create option to set the extent size hint, and changes the default from not setting a hint to setting it to 1 MB. The main reason why qcow2 defaults to smaller cluster sizes is that COW becomes more expensive, which is not an issue with raw files, so we can choose a larger size. The tradeoff here is only potentially wasted disk space. For qcow2 (or other image formats) over file-posix, the advantage should even be greater because they grow sequentially without leaving holes, so there won't be wasted space. Setting even larger extent size hints for such images may make sense. This can be done with the new option, but let's keep the default conservative for now. The effect is very visible with a test that intentionally creates a badly fragmented file with qemu-img bench (the time difference while creating the file is already remarkable) and then looks at the number of extents and the time a simple "qemu-img map" takes. Without an extent size hint: $ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0 $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0 Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192) Run completed in 25.848 seconds. $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096 Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192) Run completed in 19.616 seconds. $ filefrag ~/tmp/test.raw /home/kwolf/tmp/test.raw: 2000000 extents found $ time ./qemu-img map ~/tmp/test.raw Offset Length Mapped to File 0 0x1e8480000 0 /home/kwolf/tmp/test.raw real 0m1,279s user 0m0,043s sys 0m1,226s With the new default extent size hint of 1 MB: $ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576 $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0 Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192) Run completed in 11.833 seconds. $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096 Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192) Run completed in 10.155 seconds. $ filefrag ~/tmp/test.raw /home/kwolf/tmp/test.raw: 178 extents found $ time ./qemu-img map ~/tmp/test.raw Offset Length Mapped to File 0 0x1e8480000 0 /home/kwolf/tmp/test.raw real 0m0,061s user 0m0,040s sys 0m0,014s Signed-off-by: Kevin Wolf Message-Id: <20200707142329.48303-1-kwolf@redhat.com> Reviewed-by: Eric Blake Signed-off-by: Kevin Wolf --- qapi/block-core.json | 11 ++++++---- include/block/block_int.h | 1 + block/file-posix.c | 42 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/082.out | 16 +++++++++++++++ tests/qemu-iotests/243 | 7 ++++--- tests/qemu-iotests/243.out | 16 +++++++-------- 6 files changed, 78 insertions(+), 15 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 0e1c6a59f2..ddd3737d90 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -4185,14 +4185,17 @@ # falloc (if defined CONFIG_POSIX_FALLOCATE), # full (if defined CONFIG_POSIX)) # @nocow: Turn off copy-on-write (valid only on btrfs; default: off) +# @extent-size-hint: Extent size hint to add to the image file; 0 for not +# adding an extent size hint (default: 1 MB, since 5.1) # # Since: 2.12 ## { 'struct': 'BlockdevCreateOptionsFile', - 'data': { 'filename': 'str', - 'size': 'size', - '*preallocation': 'PreallocMode', - '*nocow': 'bool' } } + 'data': { 'filename': 'str', + 'size': 'size', + '*preallocation': 'PreallocMode', + '*nocow': 'bool', + '*extent-size-hint': 'size'} } ## # @BlockdevCreateOptionsGluster: diff --git a/include/block/block_int.h b/include/block/block_int.h index 791de6a59c..03cd266948 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -53,6 +53,7 @@ #define BLOCK_OPT_ADAPTER_TYPE "adapter_type" #define BLOCK_OPT_REDUNDANCY "redundancy" #define BLOCK_OPT_NOCOW "nocow" +#define BLOCK_OPT_EXTENT_SIZE_HINT "extent_size_hint" #define BLOCK_OPT_OBJECT_SIZE "object_size" #define BLOCK_OPT_REFCOUNT_BITS "refcount_bits" #define BLOCK_OPT_DATA_FILE "data_file" diff --git a/block/file-posix.c b/block/file-posix.c index 3ab8f5a0fa..1a74dd3565 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -30,6 +30,7 @@ #include "block/block_int.h" #include "qemu/module.h" #include "qemu/option.h" +#include "qemu/units.h" #include "trace.h" #include "block/thread-pool.h" #include "qemu/iov.h" @@ -2322,6 +2323,14 @@ raw_co_create(BlockdevCreateOptions *options, Error **errp) if (!file_opts->has_preallocation) { file_opts->preallocation = PREALLOC_MODE_OFF; } + if (!file_opts->has_extent_size_hint) { + file_opts->extent_size_hint = 1 * MiB; + } + if (file_opts->extent_size_hint > UINT32_MAX) { + result = -EINVAL; + error_setg(errp, "Extent size hint is too large"); + goto out; + } /* Create file */ fd = qemu_open(file_opts->filename, O_RDWR | O_CREAT | O_BINARY, 0644); @@ -2379,6 +2388,25 @@ raw_co_create(BlockdevCreateOptions *options, Error **errp) } #endif } +#ifdef FS_IOC_FSSETXATTR + /* + * Try to set the extent size hint. Failure is not fatal, and a warning is + * only printed if the option was explicitly specified. + */ + { + struct fsxattr attr; + result = ioctl(fd, FS_IOC_FSGETXATTR, &attr); + if (result == 0) { + attr.fsx_xflags |= FS_XFLAG_EXTSIZE; + attr.fsx_extsize = file_opts->extent_size_hint; + result = ioctl(fd, FS_IOC_FSSETXATTR, &attr); + } + if (result < 0 && file_opts->has_extent_size_hint) { + warn_report("Failed to set extent size hint: %s", + strerror(errno)); + } + } +#endif /* Resize and potentially preallocate the file to the desired * final size */ @@ -2414,6 +2442,8 @@ static int coroutine_fn raw_co_create_opts(BlockDriver *drv, { BlockdevCreateOptions options; int64_t total_size = 0; + int64_t extent_size_hint = 0; + bool has_extent_size_hint = false; bool nocow = false; PreallocMode prealloc; char *buf = NULL; @@ -2425,6 +2455,11 @@ static int coroutine_fn raw_co_create_opts(BlockDriver *drv, /* Read out options */ total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), BDRV_SECTOR_SIZE); + if (qemu_opt_get(opts, BLOCK_OPT_EXTENT_SIZE_HINT)) { + has_extent_size_hint = true; + extent_size_hint = + qemu_opt_get_size_del(opts, BLOCK_OPT_EXTENT_SIZE_HINT, -1); + } nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false); buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC); prealloc = qapi_enum_parse(&PreallocMode_lookup, buf, @@ -2444,6 +2479,8 @@ static int coroutine_fn raw_co_create_opts(BlockDriver *drv, .preallocation = prealloc, .has_nocow = true, .nocow = nocow, + .has_extent_size_hint = has_extent_size_hint, + .extent_size_hint = extent_size_hint, }, }; return raw_co_create(&options, errp); @@ -2937,6 +2974,11 @@ static QemuOptsList raw_create_opts = { #endif ", full)" }, + { + .name = BLOCK_OPT_EXTENT_SIZE_HINT, + .type = QEMU_OPT_SIZE, + .help = "Extent size hint for the image file, 0 to disable" + }, { /* end of list */ } } }; diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out index 529a1214e1..a65e185179 100644 --- a/tests/qemu-iotests/082.out +++ b/tests/qemu-iotests/082.out @@ -62,6 +62,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -86,6 +87,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -110,6 +112,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -134,6 +137,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -158,6 +162,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -182,6 +187,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -206,6 +212,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -230,6 +237,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -353,6 +361,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -377,6 +386,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -401,6 +411,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -425,6 +436,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -449,6 +461,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -473,6 +486,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -497,6 +511,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) @@ -521,6 +536,7 @@ Supported options: encrypt.ivgen-hash-alg= - Name of IV generator hash algorithm encrypt.key-secret= - ID of secret providing qcow AES key or LUKS passphrase encryption= - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes) + extent_size_hint= - Extent size hint for the image file, 0 to disable lazy_refcounts= - Postpone refcount updates nocow= - Turn off copy-on-write (valid only on btrfs) preallocation= - Preallocation mode (allowed values: off, metadata, falloc, full) diff --git a/tests/qemu-iotests/243 b/tests/qemu-iotests/243 index a61852f6d9..17388a4644 100755 --- a/tests/qemu-iotests/243 +++ b/tests/qemu-iotests/243 @@ -51,7 +51,7 @@ for mode in off metadata falloc full; do echo "=== preallocation=$mode ===" echo - _make_test_img -o "preallocation=$mode" 64M + _make_test_img -o "preallocation=$mode,extent_size_hint=0" 64M printf "File size: " du -b $TEST_IMG | cut -f1 @@ -68,7 +68,8 @@ for mode in off metadata falloc full; do echo "=== External data file: preallocation=$mode ===" echo - _make_test_img -o "data_file=$TEST_IMG.data,preallocation=$mode" 64M + _make_test_img \ + -o "data_file=$TEST_IMG.data,preallocation=$mode,extent_size_hint=0" 64M echo -n "qcow2 file size: " du -b $TEST_IMG | cut -f1 @@ -79,7 +80,7 @@ for mode in off metadata falloc full; do echo -n "qcow2 disk usage: " [ $(du -B1 $TEST_IMG | cut -f1) -lt 1048576 ] && echo "low" || echo "high" echo -n "data disk usage: " - [ $(du -B1 $TEST_IMG.data | cut -f1) -lt 1048576 ] && echo "low" || echo "high" + [ $(du -B1 $TEST_IMG.data | cut -f1) -lt 2097152 ] && echo "low" || echo "high" done diff --git a/tests/qemu-iotests/243.out b/tests/qemu-iotests/243.out index dcb33fac32..8bd3d79d66 100644 --- a/tests/qemu-iotests/243.out +++ b/tests/qemu-iotests/243.out @@ -2,31 +2,31 @@ QA output created by 243 === preallocation=off === -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=off +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=off extent_size_hint=0 File size: 196616 Disk usage: low === preallocation=metadata === -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=metadata +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=metadata extent_size_hint=0 File size: 67436544 Disk usage: low === preallocation=falloc === -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=falloc +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=falloc extent_size_hint=0 File size: 67436544 Disk usage: high === preallocation=full === -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=full +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=full extent_size_hint=0 File size: 67436544 Disk usage: high === External data file: preallocation=off === -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=off +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=off extent_size_hint=0 qcow2 file size: 196616 data file size: 67108864 qcow2 disk usage: low @@ -34,7 +34,7 @@ data disk usage: low === External data file: preallocation=metadata === -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=metadata +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=metadata extent_size_hint=0 qcow2 file size: 327680 data file size: 67108864 qcow2 disk usage: low @@ -42,7 +42,7 @@ data disk usage: low === External data file: preallocation=falloc === -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=falloc +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=falloc extent_size_hint=0 qcow2 file size: 327680 data file size: 67108864 qcow2 disk usage: low @@ -50,7 +50,7 @@ data disk usage: high === External data file: preallocation=full === -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=full +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=full extent_size_hint=0 qcow2 file size: 327680 data file size: 67108864 qcow2 disk usage: low From patchwork Tue Jul 7 16:34:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Wolf X-Patchwork-Id: 278294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38B7BC433E0 for ; Tue, 7 Jul 2020 16:50:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0430020708 for ; Tue, 7 Jul 2020 16:50:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LvB3oZYG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0430020708 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:54920 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jsqo6-0002tL-6k for qemu-devel@archiver.kernel.org; Tue, 07 Jul 2020 12:50:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39996) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jsqZn-0008E1-Cq for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:36:11 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:53582 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jsqZj-0004Fc-56 for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:36:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1594139766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DdaB25yWPzDpqrSx4Dkn+Ay1YVyVUTLKcicTCCvTG14=; b=LvB3oZYGmSHhDXyN1pLnx1pDBIPCfmelCMb1hvavm1P1ixqtoX92rjCeTbDwKiClX+UHDP 49iTgAo5LESgu0wfRL1mAV4xp7FUmbrE8/W3cSxBER3sth2FjwYNFbgRz7jtan/XwvjWDt ZzlfmhRUtcNuaBIKAvFjfaPmcsfZ1Yk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-57-xphObT5pOSKjbxXxTh58nA-1; Tue, 07 Jul 2020 12:35:46 -0400 X-MC-Unique: xphObT5pOSKjbxXxTh58nA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9941D84E21A; Tue, 7 Jul 2020 16:35:44 +0000 (UTC) Received: from linux.fritz.box.com (ovpn-114-113.ams2.redhat.com [10.36.114.113]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2908F71674; Tue, 7 Jul 2020 16:35:43 +0000 (UTC) From: Kevin Wolf To: qemu-block@nongnu.org Subject: [PULL 03/12] qemu-img: Flush stdout before before potential stderr messages Date: Tue, 7 Jul 2020 18:34:55 +0200 Message-Id: <20200707163504.194740-4-kwolf@redhat.com> In-Reply-To: <20200707163504.194740-1-kwolf@redhat.com> References: <20200707163504.194740-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=kwolf@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/07 00:20:54 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, peter.maydell@linaro.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Eric Blake During 'qemu-img create ... 2>&1', if --quiet is not in force, we can end up with buffered I/O in stdout that was produced before failure, but which appears in output after failure. This is confusing; the fix is to flush stdout prior to attempting anything that might produce an error message. Several iotests demonstrate the resulting ordering change now that the merged outputs now reflect chronology. (An even better fix would be to avoid printf from within block.c altogether, but that's much more invasive...) Signed-off-by: Eric Blake Message-Id: <20200706203954.341758-2-eblake@redhat.com> Signed-off-by: Kevin Wolf --- block.c | 1 + tests/qemu-iotests/049.out | 8 ++++---- tests/qemu-iotests/054.out | 2 +- tests/qemu-iotests/079.out | 2 +- tests/qemu-iotests/112.out | 4 ++-- tests/qemu-iotests/259.out | 2 +- 6 files changed, 10 insertions(+), 9 deletions(-) diff --git a/block.c b/block.c index 6dbcb7e083..a568196ba2 100644 --- a/block.c +++ b/block.c @@ -6186,6 +6186,7 @@ void bdrv_img_create(const char *filename, const char *fmt, printf("Formatting '%s', fmt=%s ", filename, fmt); qemu_opts_print(opts, " "); puts(""); + fflush(stdout); } ret = bdrv_create(drv, filename, opts, &local_err); diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out index c54ae21b86..22f395246b 100644 --- a/tests/qemu-iotests/049.out +++ b/tests/qemu-iotests/049.out @@ -167,12 +167,12 @@ qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M -qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42' Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib +qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42' qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M -qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar' Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib +qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar' == Check preallocation option == @@ -183,8 +183,8 @@ qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M -qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234' Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib +qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234' == Check encryption option == @@ -206,7 +206,7 @@ qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M -qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater) Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib +qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater) *** done diff --git a/tests/qemu-iotests/054.out b/tests/qemu-iotests/054.out index e6ec430edd..71f18bb987 100644 --- a/tests/qemu-iotests/054.out +++ b/tests/qemu-iotests/054.out @@ -1,8 +1,8 @@ QA output created by 054 creating too large image (1 EB) -qemu-img: TEST_DIR/t.IMGFMT: The image size is too large for file format 'IMGFMT' (try using a larger cluster size) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1152921504606846976 +qemu-img: TEST_DIR/t.IMGFMT: The image size is too large for file format 'IMGFMT' (try using a larger cluster size) creating too large image (1 EB) using qcow2.py Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 diff --git a/tests/qemu-iotests/079.out b/tests/qemu-iotests/079.out index aab922fb36..f65a9ca84f 100644 --- a/tests/qemu-iotests/079.out +++ b/tests/qemu-iotests/079.out @@ -9,6 +9,6 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation=metadat Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation=metadata Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation=metadata Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation=metadata -qemu-img: TEST_DIR/t.IMGFMT: Cluster size must be a power of two between 512 and 2048k Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4294967296 preallocation=metadata +qemu-img: TEST_DIR/t.IMGFMT: Cluster size must be a power of two between 512 and 2048k *** done diff --git a/tests/qemu-iotests/112.out b/tests/qemu-iotests/112.out index ae0318cabe..5e77e38a7a 100644 --- a/tests/qemu-iotests/112.out +++ b/tests/qemu-iotests/112.out @@ -2,7 +2,6 @@ QA output created by 112 === refcount_bits limits === -qemu-img: TEST_DIR/t.IMGFMT: Refcount width must be a power of two and may not exceed 64 bits Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 qemu-img: TEST_DIR/t.IMGFMT: Refcount width must be a power of two and may not exceed 64 bits Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 refcount_bits=-1 @@ -10,6 +9,7 @@ qemu-img: TEST_DIR/t.IMGFMT: Refcount width must be a power of two and may not e Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 qemu-img: TEST_DIR/t.IMGFMT: Refcount width must be a power of two and may not exceed 64 bits Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +qemu-img: TEST_DIR/t.IMGFMT: Refcount width must be a power of two and may not exceed 64 bits Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 refcount bits: 1 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 @@ -21,10 +21,10 @@ refcount bits: 16 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 refcount bits: 16 -qemu-img: TEST_DIR/t.IMGFMT: Different refcount widths than 16 bits require compatibility level 1.1 or above (use version=v3 or greater) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 qemu-img: TEST_DIR/t.IMGFMT: Different refcount widths than 16 bits require compatibility level 1.1 or above (use version=v3 or greater) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +qemu-img: TEST_DIR/t.IMGFMT: Different refcount widths than 16 bits require compatibility level 1.1 or above (use version=v3 or greater) === Snapshot limit on refcount_bits=1 === diff --git a/tests/qemu-iotests/259.out b/tests/qemu-iotests/259.out index ffed19c2a0..e27b9ff38d 100644 --- a/tests/qemu-iotests/259.out +++ b/tests/qemu-iotests/259.out @@ -9,6 +9,6 @@ virtual size: 64 MiB (67108864 bytes) disk size: unavailable --- Testing creation for which the node would need to grow --- -qemu-img: TEST_DIR/t.IMGFMT: Could not resize image: Image format driver does not support resize Formatting 'TEST_DIR/t.IMGFMT', fmt=qcow2 size=67108864 preallocation=metadata +qemu-img: TEST_DIR/t.IMGFMT: Could not resize image: Image format driver does not support resize *** done From patchwork Tue Jul 7 16:34:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Wolf X-Patchwork-Id: 278288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A125C433E0 for ; Tue, 7 Jul 2020 16:59:01 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1543B2065D for ; Tue, 7 Jul 2020 16:59:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="U8qphw15" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1543B2065D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:33730 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jsqvs-0001fT-9h for qemu-devel@archiver.kernel.org; Tue, 07 Jul 2020 12:59:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39778) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jsqZY-0007jn-9Z for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:35:56 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:31642 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jsqZV-0004AM-Ha for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:35:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1594139752; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xV9xQIvDYBqDZoBJhdSVcyP2Syk+NPN6QiMwXim3Yvk=; b=U8qphw151+OEQSOpHInHAFN2RI+tJTLH/bzw/B+bMPhmAvfu5tAbcL13Zz7tP/nQvMg1/S OiPxQqGIBNyhm9Tmu3wCR+ty8iByt/0uivS1qNIrrOe0Vki/7SE/ksQU/TM1AAmc+ndQnc cdVD7HlUIAIAIgx8W1SezAFqQwvYQTc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-37-ZP4RwPQ6NVSepc_wkOAjZA-1; Tue, 07 Jul 2020 12:35:47 -0400 X-MC-Unique: ZP4RwPQ6NVSepc_wkOAjZA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EEC07100A62B; Tue, 7 Jul 2020 16:35:45 +0000 (UTC) Received: from linux.fritz.box.com (ovpn-114-113.ams2.redhat.com [10.36.114.113]) by smtp.corp.redhat.com (Postfix) with ESMTP id 985447848B; Tue, 7 Jul 2020 16:35:44 +0000 (UTC) From: Kevin Wolf To: qemu-block@nongnu.org Subject: [PULL 04/12] block: Finish deprecation of 'qemu-img convert -n -o' Date: Tue, 7 Jul 2020 18:34:56 +0200 Message-Id: <20200707163504.194740-5-kwolf@redhat.com> In-Reply-To: <20200707163504.194740-1-kwolf@redhat.com> References: <20200707163504.194740-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=kwolf@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=kwolf@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/07 00:46:09 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -40 X-Spam_score: -4.1 X-Spam_bar: ---- X-Spam_report: (-4.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, peter.maydell@linaro.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Eric Blake It's been two releases since we started warning; time to make the combination an error as promised. There was no iotest coverage, so add some. While touching the documentation, tweak another section heading for consistent style. Signed-off-by: Eric Blake Message-Id: <20200706203954.341758-3-eblake@redhat.com> Signed-off-by: Kevin Wolf --- docs/system/deprecated.rst | 18 ++++++++---------- qemu-img.c | 4 ++-- tests/qemu-iotests/122 | 7 +++++++ tests/qemu-iotests/122.out | 4 ++++ 4 files changed, 21 insertions(+), 12 deletions(-) diff --git a/docs/system/deprecated.rst b/docs/system/deprecated.rst index 47f84be8e0..73b9d9f378 100644 --- a/docs/system/deprecated.rst +++ b/docs/system/deprecated.rst @@ -418,14 +418,6 @@ kernel in 2018, and has also been dropped from glibc. Related binaries ---------------- -``qemu-img convert -n -o`` (since 4.2.0) -'''''''''''''''''''''''''''''''''''''''' - -All options specified in ``-o`` are image creation options, so -they have no effect when used with ``-n`` to skip image creation. -Silently ignored options can be confusing, so this combination of -options will be made an error in future versions. - Backwards compatibility ----------------------- @@ -531,8 +523,8 @@ spec you can use the ``-cpu rv64gcsu,priv_spec=v1.10.0`` command line argument. Related binaries ---------------- -``qemu-nbd --partition`` (removed in 5.0.0) -''''''''''''''''''''''''''''''''''''''''''' +``qemu-nbd --partition`` (removed in 5.0) +''''''''''''''''''''''''''''''''''''''''' The ``qemu-nbd --partition $digit`` code (also spelled ``-P``) could only handle MBR partitions, and never correctly handled logical @@ -548,6 +540,12 @@ can be rewritten as:: qemu-nbd -t --image-opts driver=raw,offset=1M,size=100M,file.driver=qcow2,file.file.driver=file,file.file.filename=file.qcow2 +``qemu-img convert -n -o`` (removed in 5.1) +''''''''''''''''''''''''''''''''''''''''''' + +All options specified in ``-o`` are image creation options, so +they are now rejected when used with ``-n`` to skip image creation. + Command line options -------------------- diff --git a/qemu-img.c b/qemu-img.c index 74946f81ca..b366a89ce3 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -2369,8 +2369,8 @@ static int img_convert(int argc, char **argv) } if (skip_create && options) { - warn_report("-o has no effect when skipping image creation"); - warn_report("This will become an error in future QEMU versions."); + error_report("-o has no effect when skipping image creation"); + goto fail_getopt; } if (s.has_zero_init && !skip_create) { diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122 index f7a3ae684a..2dc16b2ca4 100755 --- a/tests/qemu-iotests/122 +++ b/tests/qemu-iotests/122 @@ -290,6 +290,13 @@ TEST_IMG="$TEST_IMG".orig _make_test_img 64M # backing file" $QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base -n "$TEST_IMG" "$TEST_IMG".orig +echo +echo '=== -n incompatible with -o ===' +echo + +$QEMU_IMG convert -O $IMGFMT -o preallocation=metadata -n \ + "$TEST_IMG" "$TEST_IMG".orig && echo "unexpected success" + # success, all done echo '*** done' rm -f $seq.full diff --git a/tests/qemu-iotests/122.out b/tests/qemu-iotests/122.out index 1a35951a80..c2e154a1e5 100644 --- a/tests/qemu-iotests/122.out +++ b/tests/qemu-iotests/122.out @@ -233,4 +233,8 @@ Images are identical. Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864 + +=== -n incompatible with -o === + +qemu-img: -o has no effect when skipping image creation *** done From patchwork Tue Jul 7 16:35:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Wolf X-Patchwork-Id: 278296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B2A1C433E0 for ; Tue, 7 Jul 2020 16:48:15 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1A48420708 for ; Tue, 7 Jul 2020 16:48:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bWwJ3blk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1A48420708 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:42252 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jsqlS-0005BA-Cg for qemu-devel@archiver.kernel.org; Tue, 07 Jul 2020 12:48:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40036) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jsqZp-0008JA-QG for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:36:13 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:27139 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jsqZn-0004Gq-Im for qemu-devel@nongnu.org; Tue, 07 Jul 2020 12:36:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1594139770; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zfZbGWp1js/gdAdDHlBJW3uP8++K0i7xsyQMSlYyUYk=; b=bWwJ3blkyst179Jt09TgFV6YsRHkHgS59z5wL/M/R2B2vw86qfJDfAy6cuniSToqGiP1KO VLRMEAQ1mAnMu2Fs64i4I8K3nMF/gI2YUH7a2F0/oZByili1RMGD1yfqWEtfNsw05d0Ceb wKM8UQ6D7RbUeKyRpaz35qnezAhGSBk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-465-XuWHbNSGOgiRGeoMIt2MXg-1; Tue, 07 Jul 2020 12:36:00 -0400 X-MC-Unique: XuWHbNSGOgiRGeoMIt2MXg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4902F835B75; Tue, 7 Jul 2020 16:35:57 +0000 (UTC) Received: from linux.fritz.box.com (ovpn-114-113.ams2.redhat.com [10.36.114.113]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5732B512FE; Tue, 7 Jul 2020 16:35:56 +0000 (UTC) From: Kevin Wolf To: qemu-block@nongnu.org Subject: [PULL 12/12] qemu-img: Deprecate use of -b without -F Date: Tue, 7 Jul 2020 18:35:04 +0200 Message-Id: <20200707163504.194740-13-kwolf@redhat.com> In-Reply-To: <20200707163504.194740-1-kwolf@redhat.com> References: <20200707163504.194740-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=kwolf@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/07 00:20:54 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, peter.maydell@linaro.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Eric Blake Creating an image that requires format probing of the backing image is potentially unsafe (we've had several CVEs over the years based on probes leaking information to the guest on a subsequent boot, although these days tools like libvirt are aware of the issue enough to prevent the worst effects). For example, if our probing algorithm ever changes, or if other tools like libvirt determine a different probe result than we do, then subsequent use of that backing file under a different format will present corrupted data to the guest. Fortunately, the worst effects occur only when the backing image is originally raw, and we at least prevent commit into a probed raw backing file that would change its probed type. Still, it is worth starting a deprecation clock so that future qemu-img can refuse to create backing chains that would rely on probing, to encourage clients to avoid unsafe practices. Most warnings are intentionally emitted from bdrv_img_create() in the block layer, but qemu-img convert uses bdrv_create() which cannot emit its own warning without causing spurious warnings on other code paths. In the end, all command-line image creation or backing file rewriting now performs a check. Furthermore, if we probe a backing file as non-raw, then it is safe to explicitly record that result (rather than relying on future probes); only where we probe a raw image do we care about further warnings to the user when using such an image (for example, commits into a probed-raw backing file are prevented), to help them improve their tooling. But whether or not we make the probe results explicit, we still warn the user to remind them to upgrade their workflow to supply -F always. iotest 114 specifically wants to create an unsafe image for later amendment rather than defaulting to our new default of recording a probed format, so it needs an update. While touching it, expand it to cover all of the various warnings enabled by this patch. iotest 293 also shows a change to qcow messages. Signed-off-by: Eric Blake Message-Id: <20200706203954.341758-11-eblake@redhat.com> Signed-off-by: Kevin Wolf --- docs/system/deprecated.rst | 20 ++++++++++++++++++++ block.c | 27 ++++++++++++++++++++++++++- qemu-img.c | 9 ++++++++- tests/qemu-iotests/114 | 12 ++++++++++++ tests/qemu-iotests/114.out | 9 +++++++++ tests/qemu-iotests/293.out | 4 +++- 6 files changed, 78 insertions(+), 3 deletions(-) diff --git a/docs/system/deprecated.rst b/docs/system/deprecated.rst index b312ad27aa..ca994e3ef5 100644 --- a/docs/system/deprecated.rst +++ b/docs/system/deprecated.rst @@ -430,6 +430,26 @@ image). Rather, any changes to the backing chain should be performed with ``qemu-img rebase -u`` either before or after the remaining changes being performed by amend, as appropriate. +qemu-img backing file without format (since 5.1) +'''''''''''''''''''''''''''''''''''''''''''''''' + +The use of ``qemu-img create``, ``qemu-img rebase``, or ``qemu-img +convert`` to create or modify an image that depends on a backing file +now recommends that an explicit backing format be provided. This is +for safety: if QEMU probes a different format than what you thought, +the data presented to the guest will be corrupt; similarly, presenting +a raw image to a guest allows a potential security exploit if a future +probe sees a non-raw image based on guest writes. + +To avoid the warning message, or even future refusal to create an +unsafe image, you must pass ``-o backing_fmt=`` (or the shorthand +``-F`` during create) to specify the intended backing format. You may +use ``qemu-img rebase -u`` to retroactively add a backing format to an +existing image. However, be aware that there are already potential +security risks to blindly using ``qemu-img info`` to probe the format +of an untrusted backing image, when deciding what format to add into +an existing image. + Backwards compatibility ----------------------- diff --git a/block.c b/block.c index 5cfd10f6b4..aad3b3635e 100644 --- a/block.c +++ b/block.c @@ -6161,6 +6161,26 @@ void bdrv_img_create(const char *filename, const char *fmt, error_append_hint(&local_err, "Could not open backing image.\n"); goto out; } else { + if (!backing_fmt) { + warn_report("Deprecated use of backing file without explicit " + "backing format (detected format of %s)", + bs->drv->format_name); + if (bs->drv != &bdrv_raw) { + /* + * A probe of raw deserves the most attention: + * leaving the backing format out of the image + * will ensure bs->probed is set (ensuring we + * don't accidentally commit into the backing + * file), and allow more spots to warn the users + * to fix their toolchain when opening this image + * later. For other images, we can safely record + * the format that we probed. + */ + backing_fmt = bs->drv->format_name; + qemu_opt_set(opts, BLOCK_OPT_BACKING_FMT, backing_fmt, + NULL); + } + } if (size == -1) { /* Opened BS, have no size */ size = bdrv_getlength(bs); @@ -6174,7 +6194,12 @@ void bdrv_img_create(const char *filename, const char *fmt, } bdrv_unref(bs); } - } /* (backing_file && !(flags & BDRV_O_NO_BACKING)) */ + /* (backing_file && !(flags & BDRV_O_NO_BACKING)) */ + } else if (backing_file && !backing_fmt) { + warn_report("Deprecated use of unopened backing file without " + "explicit backing format, use of this image requires " + "potentially unsafe format probing"); + } if (size == -1) { error_setg(errp, "Image creation needs a size parameter"); diff --git a/qemu-img.c b/qemu-img.c index 440c6aef67..cd19e9b0a7 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -2523,6 +2523,13 @@ static int img_convert(int argc, char **argv) goto out; } + if (out_baseimg_param) { + if (!qemu_opt_get(opts, BLOCK_OPT_BACKING_FMT)) { + warn_report("Deprecated use of backing file without explicit " + "backing format"); + } + } + /* Check if compression is supported */ if (s.compressed) { bool encryption = @@ -3804,7 +3811,7 @@ static int img_rebase(int argc, char **argv) * doesn't change when we switch the backing file. */ if (out_baseimg && *out_baseimg) { - ret = bdrv_change_backing_file(bs, out_baseimg, out_basefmt, false); + ret = bdrv_change_backing_file(bs, out_baseimg, out_basefmt, true); } else { ret = bdrv_change_backing_file(bs, NULL, NULL, false); } diff --git a/tests/qemu-iotests/114 b/tests/qemu-iotests/114 index 26104fff6c..1436833813 100755 --- a/tests/qemu-iotests/114 +++ b/tests/qemu-iotests/114 @@ -42,9 +42,16 @@ _unsupported_proto vxhs # qcow2.py does not work too well with external data files _unsupported_imgopts data_file +# Intentionally specify backing file without backing format; demonstrate +# the difference in warning messages when backing file could be probed. +# Note that only a non-raw probe result will affect the resulting image. +truncate --size=64M "$TEST_IMG.orig" +_make_test_img -b "$TEST_IMG.orig" 64M TEST_IMG="$TEST_IMG.base" _make_test_img 64M +$QEMU_IMG convert -O qcow2 -B "$TEST_IMG.orig" "$TEST_IMG.orig" "$TEST_IMG" _make_test_img -b "$TEST_IMG.base" 64M +_make_test_img -u -b "$TEST_IMG.base" 64M # Set an invalid backing file format $PYTHON qcow2.py "$TEST_IMG" add-header-ext 0xE2792ACA "foo" @@ -55,6 +62,11 @@ _img_info $QEMU_IO -c "open $TEST_IMG" -c "read 0 4k" 2>&1 | _filter_qemu_io | _filter_testdir $QEMU_IO -c "open -o backing.driver=$IMGFMT $TEST_IMG" -c "read 0 4k" | _filter_qemu_io +# Rebase the image, to show that omitting backing format triggers a warning, +# but probing now lets us use the backing file. +$QEMU_IMG rebase -u -b "$TEST_IMG.base" "$TEST_IMG" +$QEMU_IO -c "open $TEST_IMG" -c "read 0 4k" 2>&1 | _filter_qemu_io | _filter_testdir + # success, all done echo '*** done' rm -f $seq.full diff --git a/tests/qemu-iotests/114.out b/tests/qemu-iotests/114.out index 67adef37a4..0a37d20c82 100644 --- a/tests/qemu-iotests/114.out +++ b/tests/qemu-iotests/114.out @@ -1,5 +1,11 @@ QA output created by 114 +qemu-img: warning: Deprecated use of backing file without explicit backing format (detected format of raw) +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.orig Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 +qemu-img: warning: Deprecated use of backing file without explicit backing format +qemu-img: warning: Deprecated use of backing file without explicit backing format (detected format of IMGFMT) +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT +qemu-img: warning: Deprecated use of unopened backing file without explicit backing format, use of this image requires potentially unsafe format probing Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base image: TEST_DIR/t.IMGFMT file format: IMGFMT @@ -11,4 +17,7 @@ qemu-io: can't open device TEST_DIR/t.qcow2: Could not open backing file: Unknow no file open, try 'help open' read 4096/4096 bytes at offset 0 4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qemu-img: warning: Deprecated use of backing file without explicit backing format, use of this image requires potentially unsafe format probing +read 4096/4096 bytes at offset 0 +4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) *** done diff --git a/tests/qemu-iotests/293.out b/tests/qemu-iotests/293.out index 3c612903f8..8bed89a67f 100644 --- a/tests/qemu-iotests/293.out +++ b/tests/qemu-iotests/293.out @@ -2,7 +2,8 @@ QA output created by 293 == qcow backed by qcow == Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=33554432 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 backing_file=TEST_DIR/t.IMGFMT.base +qemu-img: warning: Deprecated use of backing file without explicit backing format (detected format of IMGFMT) +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT image: TEST_DIR/t.IMGFMT file format: IMGFMT virtual size: 32 MiB (33554432 bytes) @@ -35,6 +36,7 @@ cluster_size: 512 backing file: TEST_DIR/t.IMGFMT.base == qcow backed by raw == +qemu-img: warning: Deprecated use of backing file without explicit backing format (detected format of raw) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 backing_file=TEST_DIR/t.IMGFMT.base image: TEST_DIR/t.IMGFMT file format: IMGFMT