From patchwork Wed Mar 25 10:02:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rantala, Tommi T. \(Nokia - FI/Espoo\)" X-Patchwork-Id: 228757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11C4BC1975A for ; Wed, 25 Mar 2020 10:02:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CB1282078A for ; Wed, 25 Mar 2020 10:02:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=nokia.onmicrosoft.com header.i=@nokia.onmicrosoft.com header.b="ixX4KYye" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727384AbgCYKCr (ORCPT ); Wed, 25 Mar 2020 06:02:47 -0400 Received: from mail-eopbgr30114.outbound.protection.outlook.com ([40.107.3.114]:42230 "EHLO EUR03-AM5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726225AbgCYKCr (ORCPT ); Wed, 25 Mar 2020 06:02:47 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UB79wf/ZMWPGJ3jZWkZDz80/sMZzVbBds8Kr5ArPDTScxlhni2XR20RdPeQX1fkMNO70OMvVgy+hlsBInGEG91GSq2vrdoUVKDrF1Twlp8/Zv8gbuZvu3mQWoB8ntFkYGm5FwYOibqjKA8yigxf4s2m0cFvC07Egg3FUqmen0InFF3/YFv/36v7pIMNm0ePOI0vZ+vYsTGlDfVIHCWMzZS/Zd94m9atSqiAg4ewSdMnLvtQM2rsFQaHYiaSNkX/OdGjtPrG83NY48FjYDSHksm7JnLKs+gaOG7AGMVFs9SN63tnjD6HCeMUV4X0fdJXncdmxBDTtnMdm86bKbLFxKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lis+06dtj/TfA2Pn9gFLzOGtNERcqEjkMGrOwU1+npM=; b=Dc0RybGe5XCX2fzYJOU+PBm07gpYf6vo9lXZltMcdy4rDG9nYIjX49ZbtUWisScOoXE/nrtvp24f3ZNda1xD2mQTlSMtAKe3gwaqOr3W6IWo2dJe/yyPZdw/mlJvY4sNoJhZlaoNmSQrlLkMg7egM/p3lIanMI/h/0cx1xZFC+l8ZxnjUqIIoQPrIih9ZKFFaGVCExFoMDz48kSYbCEGHvzz11T+Afmm6rBW+chubxz8HvTGhopyrVdH0QDac8woW8DQMXuR9pTHgbD7B7LDwIXHk/G08KVUuIpgtvXUpBCSQSmyBlJc/+u+n8/+I6f19bX306z5dVXlEjS5BMm7Cw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nokia.com; dmarc=pass action=none header.from=nokia.com; dkim=pass header.d=nokia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nokia.onmicrosoft.com; s=selector1-nokia-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lis+06dtj/TfA2Pn9gFLzOGtNERcqEjkMGrOwU1+npM=; b=ixX4KYyeVbYdJJqjqe7xUoi7sWQWNaCXJyuGRvXOsf86wkyOyKZh1B+fJ3kDYKfzdOQmBT9/uhO5UKWlNirC5X0CiBXp91ibCfoUxE9oqs5r15rzQ1FNKsYYhn3PGGOwXL0FD7OjtbVt+aDPxoMDoNN/R8zyUgc80ZCV05kF428= Received: from HE1PR0702MB3675.eurprd07.prod.outlook.com (10.167.127.12) by HE1PR0702MB3659.eurprd07.prod.outlook.com (52.133.6.33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2856.18; Wed, 25 Mar 2020 10:02:41 +0000 Received: from HE1PR0702MB3675.eurprd07.prod.outlook.com ([fe80::2806:c34c:d469:8e87]) by HE1PR0702MB3675.eurprd07.prod.outlook.com ([fe80::2806:c34c:d469:8e87%5]) with mapi id 15.20.2856.017; Wed, 25 Mar 2020 10:02:41 +0000 From: "Rantala, Tommi T. (Nokia - FI/Espoo)" To: "axboe@kernel.dk" , "osandov@fb.com" CC: "stable@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" Subject: 4.19 LTS high /proc/diskstats io_ticks Thread-Topic: 4.19 LTS high /proc/diskstats io_ticks Thread-Index: AQHWAoyFep1pzs45BEyxl7dts+ZHIA== Date: Wed, 25 Mar 2020 10:02:41 +0000 Message-ID: <564f7f3718cdc85f841d27a358a43aee4ca239d6.camel@nokia.com> Accept-Language: fi-FI, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Evolution 3.32.5 (3.32.5-1.fc30) authentication-results: spf=none (sender IP is ) smtp.mailfrom=tommi.t.rantala@nokia.com; x-originating-ip: [131.228.2.1] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: e5989092-aae0-4a2d-344b-08d7d0a3a844 x-ms-traffictypediagnostic: HE1PR0702MB3659: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0353563E2B x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(4636009)(39860400002)(396003)(376002)(346002)(136003)(366004)(81156014)(316002)(6512007)(71200400001)(8676002)(86362001)(54906003)(76116006)(478600001)(81166006)(110136005)(66556008)(64756008)(66446008)(66476007)(66946007)(8936002)(36756003)(4326008)(966005)(2906002)(5660300002)(2616005)(6506007)(26005)(6486002)(186003); DIR:OUT; SFP:1102; SCL:1; SRVR:HE1PR0702MB3659; H:HE1PR0702MB3675.eurprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; received-spf: None (protection.outlook.com: nokia.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: eiP7lbwyplkKIIWE+5V78dv7rDrdC+fheYLsPgvb6CRHtkCeTyP5xElrMQQonhQPgMoK0knfDhscbRm+6w8+fyrHosfbOonWfC1xv1YqN8kWac8fZcxKh4RlqqrnN2laRwzfKfm/FuizqRV/CxP/+C1BRocCrercWyVHslIa+QSJhqvW7SqQTD0tAnm8NCDyHg6lFx+TOTvHDofKLKMeNfaWl+4ZnNXPAvkGYOW8HkHTGd5InBUGBSHszJlo2++FCEgT61qLQe4GivwRHxEKRkxWocdxnVm0Pl61H7U/MYpaS78JcPFUhxcg/hX2gtrFrWTbNpBLzN0EGC4XjXhwtALqp5STcuo2zx30evwAw8/p2nVUHZIwQCZQvsQ7FYGOU5XMo2uQ6gUiOXtvZe5agldDoAv1MDr9vH/ysJRbwRmoGDNF2tmy5gZZmV6GYz9xa7trbTug36TMV3abkMgJyZ5LnsXmWDEdpCefZzaoibRaXHwm5c55mmrN+8MIGM/ae8O7Qw8P83+7IPaIx69bTg== x-ms-exchange-antispam-messagedata: sdqDnKl7YL7XPlamH3kXcbBUw0wEFCYYCGGqGYOr/IXkATImswxMBiB1Z2IuWGRasIZIFACwlb1vLhSZA/5BySBQcIbgvdV5gS1k0tfgLnMFu4UZX2J+JZGSbmXGwD9M77KucvyjO2ThWctkKPGgIA== x-ms-exchange-transport-forked: True Content-ID: MIME-Version: 1.0 X-OriginatorOrg: nokia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e5989092-aae0-4a2d-344b-08d7d0a3a844 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Mar 2020 10:02:41.5964 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 5d471751-9675-428d-917b-70f44f9630b0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: J9M7bFBITNfTZzTSjUB3xaxhQp+CVll4lL7wMaA/ys1eDQfxXPAUFDOTCM8kJZnu8kqIgJfapXhj/s7I5Q3q/iJOqlIHvEPfqn1PahSylkg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0702MB3659 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Hi, Tools like sar and iostat are reporting abnormally high %util with 4.19.y running in VM (the disk is almost idle): $ sar -dp Linux 4.19.107-1.x86_64 03/25/20 _x86_64_ (6 CPU) 00:00:00 DEV tps ... %util 00:10:00 vda 0.55 ... 98.07 ... 10:00:00 vda 0.44 ... 99.74 Average: vda 0.48 ... 98.98 The numbers look reasonable for the partition: # iostat -x -p ALL 1 1 Linux 4.19.107-1.x86_64 03/25/20 _x86_64_ (6 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 10.51 0.00 8.58 0.05 0.11 80.75 Device r/s ... %util vda 0.02 ... 98.25 vda1 0.01 ... 0.09 Lots of io_ticks in /proc/diskstats: # cat /proc/uptime 45787.03 229321.29 # grep vda /proc/diskstats 253 0 vda 760 0 38498 731 28165 43212 1462928 157514 0 44690263 44812032 0 0 0 0 253 1 vda1 350 0 19074 293 26169 43212 1462912 154931 0 41560 150998 0 0 0 0 Other people are apparently seeing this too with 4.19: https://kudzia.eu/b/2019/09/iostat-x-1-reporting-100-utilization-of-nearly-idle-nvme-drives/ I also see this only in 4.19.y and bisected to this (based on the Fixes tag, this should have been taken to 4.14 too...): commit 6131837b1de66116459ef4413e26fdbc70d066dc Author: Omar Sandoval Date: Thu Apr 26 00:21:58 2018 -0700 blk-mq: count allocated but not started requests in iostats inflight In the legacy block case, we increment the counter right after we allocate the request, not when the driver handles it. In both the legacy and blk-mq cases, part_inc_in_flight() is called from blk_account_io_start() right after we've allocated the request. blk-mq only considers requests started requests as inflight, but this is inconsistent with the legacy definition and the intention in the code. This removes the started condition and instead counts all allocated requests. Fixes: f299b7c7a9de ("blk-mq: provide internal in-flight variant") Signed-off-by: Omar Sandoval Signed-off-by: Jens Axboe } void blk_mq_in_flight(struct request_queue *q, struct hd_struct *part, If I get it right, when the disk is idle, and some request is allocated, part_round_stats() with this commit will now add all ticks between previous I/O and current time (now - part->stamp) to io_ticks. Before the commit, part_round_stats() would only update part->stamp when called after request allocation. Any thoughts how to best fix this in 4.19? I see the io_ticks accounting has been reworked in 5.0, do we need to backport those to 4.19, or any ill effects if this commit is reverted in 4.19? -Tommi diff --git a/block/blk-mq.c b/block/blk-mq.c index c3621453ad87..5450cbc61f8d 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -95,18 +95,15 @@ static void blk_mq_check_inflight(struct blk_mq_hw_ctx *hctx, { struct mq_inflight *mi = priv; - if (blk_mq_rq_state(rq) == MQ_RQ_IN_FLIGHT) { - /* - * index[0] counts the specific partition that was asked - * for. index[1] counts the ones that are active on the - * whole device, so increment that if mi->part is indeed - * a partition, and not a whole device. - */ - if (rq->part == mi->part) - mi->inflight[0]++; - if (mi->part->partno) - mi->inflight[1]++; - } + /* + * index[0] counts the specific partition that was asked for. index[1] + * counts the ones that are active on the whole device, so increment + * that if mi->part is indeed a partition, and not a whole device. + */ + if (rq->part == mi->part) + mi->inflight[0]++; + if (mi->part->partno) + mi->inflight[1]++;