From patchwork Tue Feb 9 04:47:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 380009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C897C433DB for ; Tue, 9 Feb 2021 04:52:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D9B8864E7A for ; Tue, 9 Feb 2021 04:52:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229821AbhBIEwb (ORCPT ); Mon, 8 Feb 2021 23:52:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229822AbhBIEw2 (ORCPT ); Mon, 8 Feb 2021 23:52:28 -0500 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0C10C061788 for ; Mon, 8 Feb 2021 20:51:46 -0800 (PST) Received: by mail-ed1-x536.google.com with SMTP id y8so21894651ede.6 for ; Mon, 08 Feb 2021 20:51:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kqVUd/LgI7jqzKstgT9xNmyGsiHrS3yv2AxLf12pQ5E=; b=YJnnTNKmJG6qRqWeweDWicQZJMes2fGOaCvk5uFFGkEvC3hhJxmSz0K0hN4SvErtA2 Jvm3mw5Nqi2FxLN2umXEpDs8O8U7duIg8P7q3HyEa6MZ39TdFAcWCID5ItzL0TBJ3Sfo WlStksYUWpx3LhI6Ej2MVifDXjQTWhjsU0ZZXF2yVTt3mz35u+5wNAPgYFcpmYItcs/p t7LfJhNSFzQNOEfo1wD4dzdKHrdDSvKzL3iWqZd8/u14QRThZob/LJcrBw+iQs9DoVAO nZYWVHoF2dTZaYirgOVfqBDhEXC7qEEX1ZJIo3xDQ16XLz6dGhGRtAJ9bkwXBLGMeiWb FuWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kqVUd/LgI7jqzKstgT9xNmyGsiHrS3yv2AxLf12pQ5E=; b=f1vpW18qpJLSgm0SZ+AHpG8r8rKQ7yomaMECDz9uL9HT2SQaJQxG6BBpD+kt2N8HZj 2RNlb90S5bQU3JQLgb5Rlub26rDEW7GgHKev8XjsryxJC3h7iweUG5Gq5G9gVRZIK8gU 9+Zr5P7wx12CxxYJknsnqRxUk6Tg17wsPz1KqRqhcoyfnRkaKYDc6aAlMtPI5mzuUXiF hQOSA0CqfOMrlLmPv/of8diZ6swKE7mAl8juvTtx6ZKn1rP4+8TtFQ5clXhyDgK9RArW Z01OXw/50oG9IX2DcpPti3q+r9UXeKsTUovQUkEjyAkehVKnDZmi2N1is5Zz1F9j/RCQ +6SQ== X-Gm-Message-State: AOAM530Q+5rMaOsraUbxQHWe1U4YoXpbEO+mxOv2/jT2ESuri6nlYO+r QdBRfpMDssWMHWhzxkYuHeik8X0fYx0= X-Google-Smtp-Source: ABdhPJxDGDf9aJRqOwKXflr6ilfsBWx/HodBTlTcfM8DWKhIHNSQP3JK2QlBQ00XL4eWiXLCNUW2Hw== X-Received: by 2002:a05:6402:4d3:: with SMTP id n19mr9775556edw.2.1612846305148; Mon, 08 Feb 2021 20:51:45 -0800 (PST) Received: from localhost.localdomain ([148.252.128.244]) by smtp.gmail.com with ESMTPSA id g9sm9973445ejp.55.2021.02.08.20.51.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Feb 2021 20:51:44 -0800 (PST) From: Pavel Begunkov To: stable@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 01/16] io_uring: simplify io_task_match() Date: Tue, 9 Feb 2021 04:47:35 +0000 Message-Id: <6545156988d3f911e4bdf9f161b600eca277c95d.1612845821.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org [ Upstream commit 06de5f5973c641c7ae033f133ecfaaf64fe633a6 ] If IORING_SETUP_SQPOLL is set all requests belong to the corresponding SQPOLL task, so skip task checking in that case and always match. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- fs/io_uring.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 907ecaffc338..510a860f8bdf 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1472,11 +1472,7 @@ static bool io_task_match(struct io_kiocb *req, struct task_struct *tsk) if (!tsk || req->task == tsk) return true; - if (ctx->flags & IORING_SETUP_SQPOLL) { - if (ctx->sq_data && req->task == ctx->sq_data->thread) - return true; - } - return false; + return (ctx->flags & IORING_SETUP_SQPOLL); } /* From patchwork Tue Feb 9 04:47:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 380008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A91AC433E9 for ; Tue, 9 Feb 2021 04:52:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2575D64E9D for ; Tue, 9 Feb 2021 04:52:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229843AbhBIEwc (ORCPT ); Mon, 8 Feb 2021 23:52:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229671AbhBIEwa (ORCPT ); Mon, 8 Feb 2021 23:52:30 -0500 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6C41C06178B for ; Mon, 8 Feb 2021 20:51:48 -0800 (PST) Received: by mail-ej1-x62e.google.com with SMTP id w2so29213008ejk.13 for ; Mon, 08 Feb 2021 20:51:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Tq00WglnuuSBa0sXsEQ7TSq+dFTnfHw/bnd9JGFw0oc=; b=ksRaUcnnr4Le6HEa7vPDrly30TFtRk551bhvw1AjUNIlQy6l0daqERf+Y6hGZnijRK S2dZohyh43wahXQhSS+lY3dq0Hc/7m1EOPxCHn88PcfKozBm+Ud/8w0wVF4yZFVdskHd 0Pejocbda2ySQoXWMpLMddblVFrrry7ewFimbc9fLhoG3GBfcz7Ippp2mOdYl+5R+xSP IIe1EPpj8mXaCZe5kB1+PV6gCy7jVV7g5zxaffTrOvssQwH390eMod9vtAml8VeSlLqB Ysa0MKt1OA/8wPYdNWZuHKPA6h38Mmv/iQCD0kIBOdfMPLQ18pcFsw3txY9R2kzgIPIp 68CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Tq00WglnuuSBa0sXsEQ7TSq+dFTnfHw/bnd9JGFw0oc=; b=fUo7oBHkWVvc64kVvZqjVjeCztk+VNYTvV+pcq/MrJTrz2dSBP3i26YN24914BlJas kcjFk47lw3mPzzAhzk7SHHJxpozNaoLRXENkTkA2zlQbVfiD9Js0QZ4omgBg4Xj/rIJF cb1Tr0W0bROUF5wjW5wZIdQBxkDFEU73AV42h6HIAVNzHHCnEcK8aNCjMcY3MdfR2t8g UVYBhQswfpksULA9ZQA3//Cb44A/Z0rJ4P3MB1GEwO7A7purXRcIDI7cthlsKx+XGq+D T4d9jC6Dh8yAcLH/5zcM8tdMtXy5sFLqyFBpSN0Ubc4/dfRU2EnIq8sNKllVOXmhDFto gEzg== X-Gm-Message-State: AOAM530mBgFmTk2JGqvd9mQa3xrNO4pGP0Y21u/aI0jp8x2xVurFFMq0 qEWS504ZerWVtjr63wtVk/RhWP+xC8U= X-Google-Smtp-Source: ABdhPJzzueM3EYnpgwhDxoZZDOXtHyOuOkSHwYi0IOJduEkPeX3UTZ1P07k7PLpIBkFvVXGud6NnuA== X-Received: by 2002:a17:906:2353:: with SMTP id m19mr20773416eja.13.1612846307167; Mon, 08 Feb 2021 20:51:47 -0800 (PST) Received: from localhost.localdomain ([148.252.128.244]) by smtp.gmail.com with ESMTPSA id g9sm9973445ejp.55.2021.02.08.20.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Feb 2021 20:51:46 -0800 (PST) From: Pavel Begunkov To: stable@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 03/16] io_uring: don't iterate io_uring_cancel_files() Date: Tue, 9 Feb 2021 04:47:37 +0000 Message-Id: <509b0588dfa044d9f835bcc8a53d847b5e26de3e.1612845821.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org [ Upstream commit b52fda00dd9df8b4a6de5784df94f9617f6133a1 ] io_uring_cancel_files() guarantees to cancel all matching requests, that's not necessary to do that in a loop. Move it up in the callchain into io_uring_cancel_task_requests(). Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- fs/io_uring.c | 34 ++++++++++++---------------------- 1 file changed, 12 insertions(+), 22 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 71bdd288c396..b8c413830722 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -8654,16 +8654,10 @@ static void io_cancel_defer_files(struct io_ring_ctx *ctx, } } -/* - * Returns true if we found and killed one or more files pinning requests - */ -static bool io_uring_cancel_files(struct io_ring_ctx *ctx, +static void io_uring_cancel_files(struct io_ring_ctx *ctx, struct task_struct *task, struct files_struct *files) { - if (list_empty_careful(&ctx->inflight_list)) - return false; - while (!list_empty_careful(&ctx->inflight_list)) { struct io_kiocb *cancel_req = NULL, *req; DEFINE_WAIT(wait); @@ -8698,8 +8692,6 @@ static bool io_uring_cancel_files(struct io_ring_ctx *ctx, schedule(); finish_wait(&ctx->inflight_wait, &wait); } - - return true; } static bool io_cancel_task_cb(struct io_wq_work *work, void *data) @@ -8710,15 +8702,12 @@ static bool io_cancel_task_cb(struct io_wq_work *work, void *data) return io_task_match(req, task); } -static bool __io_uring_cancel_task_requests(struct io_ring_ctx *ctx, - struct task_struct *task, - struct files_struct *files) +static void __io_uring_cancel_task_requests(struct io_ring_ctx *ctx, + struct task_struct *task) { - bool ret; - - ret = io_uring_cancel_files(ctx, task, files); - if (!files) { + while (1) { enum io_wq_cancel cret; + bool ret = false; cret = io_wq_cancel_cb(ctx->io_wq, io_cancel_task_cb, task, true); if (cret != IO_WQ_CANCEL_NOTFOUND) @@ -8734,9 +8723,11 @@ static bool __io_uring_cancel_task_requests(struct io_ring_ctx *ctx, ret |= io_poll_remove_all(ctx, task); ret |= io_kill_timeouts(ctx, task); + if (!ret) + break; + io_run_task_work(); + cond_resched(); } - - return ret; } static void io_disable_sqo_submit(struct io_ring_ctx *ctx) @@ -8771,11 +8762,10 @@ static void io_uring_cancel_task_requests(struct io_ring_ctx *ctx, io_cancel_defer_files(ctx, task, files); io_cqring_overflow_flush(ctx, true, task, files); + io_uring_cancel_files(ctx, task, files); - while (__io_uring_cancel_task_requests(ctx, task, files)) { - io_run_task_work(); - cond_resched(); - } + if (!files) + __io_uring_cancel_task_requests(ctx, task); if ((ctx->flags & IORING_SETUP_SQPOLL) && ctx->sq_data) { atomic_dec(&task->io_uring->in_idle); From patchwork Tue Feb 9 04:47:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 380007 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8961C433E0 for ; Tue, 9 Feb 2021 04:53:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 751D764E9D for ; Tue, 9 Feb 2021 04:53:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230054AbhBIExO (ORCPT ); Mon, 8 Feb 2021 23:53:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229733AbhBIExK (ORCPT ); Mon, 8 Feb 2021 23:53:10 -0500 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7E42C061793 for ; Mon, 8 Feb 2021 20:51:50 -0800 (PST) Received: by mail-ed1-x532.google.com with SMTP id s11so21911797edd.5 for ; Mon, 08 Feb 2021 20:51:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hlyPUBvUbF5QnEV1U8KR/goAzRJMhixprLiEdyWE1ME=; b=NW5oKcDPAn47pWFU92/BPmra2SmPpTPHB/ZeOnCNoA4+iRqmrOZqFvmYqO39Qc0QcZ 9z295a7Ukq0LwVUmuFKD0ghEanqJetX4W0uX68byFXlLGbOc+uuHQwnqMtaeqopNvcmO 44tyQZo46U1Jac2tJTmE7m9+NKdQWMya4bauQlIy1YnERqgoDm74uiJJAQi1wEH6B3yx qnTYbPygBxaoqZGsvg1sylEfDSE2UMREdtHHKulJYm59qw6mVcwOUM7LUDKPwuik0Jem QHAjNBhX+xy2T8oCslFSajp6LaNue5JARix7kxBsQvec1ESPuFe9+rtyYzZzo7iHo3Tz nwSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hlyPUBvUbF5QnEV1U8KR/goAzRJMhixprLiEdyWE1ME=; b=t8tBAEfzIV82UYG9BUwP+DSGHaM06O5rbBjw7z6mNAWr8N9I0BfFW+Lpr+W6QSJHin tBraQikpttgjlIcfzKHCmwgjiyckkPer114WrEAylkse2Pc/FCMd3Ym5Rxj/7OUHoy8g 2HwUNhsR5D7qDz8she+/5DNbWKAbhF/XV+3aTM/pJ42/0EaG1d1PlhSXKEy52Ka1RrTz KfgM0QYR/O41VWhYR8lhmaE464S99sMbA3K5a7QcxfoqDo1MnJzKL5JQxLvSfA/QYMPC L0qE/hlzyaBrDR+fi+xWiL0P8VuwgS9qATJE49uar7ifR6kwPTUYd0pTWENLNL4nF/hh KeSA== X-Gm-Message-State: AOAM533RFZTIBYGkX5fdi2CwkgAvLxgHUAyITes+q50JJaG8wJiQ0QgH 0ilMO3FJ9+QaO1jTPpIB7awRq7lOhTg= X-Google-Smtp-Source: ABdhPJzQrjHyXx8zh5iyxMfarhuLwOkS9ArgWwsmtACHYt8hGgDtko4Y/RJkUuRkKwRXl/1ViRZ0UQ== X-Received: by 2002:a50:cbc7:: with SMTP id l7mr14651168edi.134.1612846309129; Mon, 08 Feb 2021 20:51:49 -0800 (PST) Received: from localhost.localdomain ([148.252.128.244]) by smtp.gmail.com with ESMTPSA id g9sm9973445ejp.55.2021.02.08.20.51.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Feb 2021 20:51:48 -0800 (PST) From: Pavel Begunkov To: stable@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 05/16] io_uring: always batch cancel in *cancel_files() Date: Tue, 9 Feb 2021 04:47:39 +0000 Message-Id: <9f10a4fb3f492aedfaccfbe631ef884fa4874e1a.1612845821.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org [ Upstream commit f6edbabb8359798c541b0776616c5eab3a840d3d ] Instead of iterating over each request and cancelling it individually in io_uring_cancel_files(), try to cancel all matching requests and use ->inflight_list only to check if there anything left. In many cases it should be faster, and we can reuse a lot of code from task cancellation. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- fs/io-wq.c | 10 ---- fs/io-wq.h | 1 - fs/io_uring.c | 139 ++++++++------------------------------------------ 3 files changed, 20 insertions(+), 130 deletions(-) diff --git a/fs/io-wq.c b/fs/io-wq.c index b53c055bea6a..f72d53848dcb 100644 --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -1078,16 +1078,6 @@ enum io_wq_cancel io_wq_cancel_cb(struct io_wq *wq, work_cancel_fn *cancel, return IO_WQ_CANCEL_NOTFOUND; } -static bool io_wq_io_cb_cancel_data(struct io_wq_work *work, void *data) -{ - return work == data; -} - -enum io_wq_cancel io_wq_cancel_work(struct io_wq *wq, struct io_wq_work *cwork) -{ - return io_wq_cancel_cb(wq, io_wq_io_cb_cancel_data, (void *)cwork, false); -} - struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data) { int ret = -ENOMEM, node; diff --git a/fs/io-wq.h b/fs/io-wq.h index aaa363f35891..75113bcd5889 100644 --- a/fs/io-wq.h +++ b/fs/io-wq.h @@ -130,7 +130,6 @@ static inline bool io_wq_is_hashed(struct io_wq_work *work) } void io_wq_cancel_all(struct io_wq *wq); -enum io_wq_cancel io_wq_cancel_work(struct io_wq *wq, struct io_wq_work *cwork); typedef bool (work_cancel_fn)(struct io_wq_work *, void *); diff --git a/fs/io_uring.c b/fs/io_uring.c index 0a9f938ac3a1..44b859456ef6 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1496,15 +1496,6 @@ static void io_kill_timeout(struct io_kiocb *req) } } -static bool io_task_match(struct io_kiocb *req, struct task_struct *tsk) -{ - struct io_ring_ctx *ctx = req->ctx; - - if (!tsk || req->task == tsk) - return true; - return (ctx->flags & IORING_SETUP_SQPOLL); -} - /* * Returns true if we found and killed one or more timeouts */ @@ -8524,112 +8515,31 @@ static int io_uring_release(struct inode *inode, struct file *file) return 0; } -/* - * Returns true if 'preq' is the link parent of 'req' - */ -static bool io_match_link(struct io_kiocb *preq, struct io_kiocb *req) -{ - struct io_kiocb *link; - - if (!(preq->flags & REQ_F_LINK_HEAD)) - return false; - - list_for_each_entry(link, &preq->link_list, link_list) { - if (link == req) - return true; - } - - return false; -} - -/* - * We're looking to cancel 'req' because it's holding on to our files, but - * 'req' could be a link to another request. See if it is, and cancel that - * parent request if so. - */ -static bool io_poll_remove_link(struct io_ring_ctx *ctx, struct io_kiocb *req) -{ - struct hlist_node *tmp; - struct io_kiocb *preq; - bool found = false; - int i; - - spin_lock_irq(&ctx->completion_lock); - for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct hlist_head *list; - - list = &ctx->cancel_hash[i]; - hlist_for_each_entry_safe(preq, tmp, list, hash_node) { - found = io_match_link(preq, req); - if (found) { - io_poll_remove_one(preq); - break; - } - } - } - spin_unlock_irq(&ctx->completion_lock); - return found; -} - -static bool io_timeout_remove_link(struct io_ring_ctx *ctx, - struct io_kiocb *req) -{ - struct io_kiocb *preq; - bool found = false; - - spin_lock_irq(&ctx->completion_lock); - list_for_each_entry(preq, &ctx->timeout_list, timeout.list) { - found = io_match_link(preq, req); - if (found) { - __io_timeout_cancel(preq); - break; - } - } - spin_unlock_irq(&ctx->completion_lock); - return found; -} +struct io_task_cancel { + struct task_struct *task; + struct files_struct *files; +}; -static bool io_cancel_link_cb(struct io_wq_work *work, void *data) +static bool io_cancel_task_cb(struct io_wq_work *work, void *data) { struct io_kiocb *req = container_of(work, struct io_kiocb, work); + struct io_task_cancel *cancel = data; bool ret; - if (req->flags & REQ_F_LINK_TIMEOUT) { + if (cancel->files && (req->flags & REQ_F_LINK_TIMEOUT)) { unsigned long flags; struct io_ring_ctx *ctx = req->ctx; /* protect against races with linked timeouts */ spin_lock_irqsave(&ctx->completion_lock, flags); - ret = io_match_link(req, data); + ret = io_match_task(req, cancel->task, cancel->files); spin_unlock_irqrestore(&ctx->completion_lock, flags); } else { - ret = io_match_link(req, data); + ret = io_match_task(req, cancel->task, cancel->files); } return ret; } -static void io_attempt_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req) -{ - enum io_wq_cancel cret; - - /* cancel this particular work, if it's running */ - cret = io_wq_cancel_work(ctx->io_wq, &req->work); - if (cret != IO_WQ_CANCEL_NOTFOUND) - return; - - /* find links that hold this pending, cancel those */ - cret = io_wq_cancel_cb(ctx->io_wq, io_cancel_link_cb, req, true); - if (cret != IO_WQ_CANCEL_NOTFOUND) - return; - - /* if we have a poll link holding this pending, cancel that */ - if (io_poll_remove_link(ctx, req)) - return; - - /* final option, timeout link is holding this req pending */ - io_timeout_remove_link(ctx, req); -} - static void io_cancel_defer_files(struct io_ring_ctx *ctx, struct task_struct *task, struct files_struct *files) @@ -8661,8 +8571,10 @@ static void io_uring_cancel_files(struct io_ring_ctx *ctx, struct files_struct *files) { while (!list_empty_careful(&ctx->inflight_list)) { - struct io_kiocb *cancel_req = NULL, *req; + struct io_task_cancel cancel = { .task = task, .files = NULL, }; + struct io_kiocb *req; DEFINE_WAIT(wait); + bool found = false; spin_lock_irq(&ctx->inflight_lock); list_for_each_entry(req, &ctx->inflight_list, inflight_entry) { @@ -8670,25 +8582,21 @@ static void io_uring_cancel_files(struct io_ring_ctx *ctx, (req->work.flags & IO_WQ_WORK_FILES) && req->work.identity->files != files) continue; - /* req is being completed, ignore */ - if (!refcount_inc_not_zero(&req->refs)) - continue; - cancel_req = req; + found = true; break; } - if (cancel_req) + if (found) prepare_to_wait(&ctx->inflight_wait, &wait, TASK_UNINTERRUPTIBLE); spin_unlock_irq(&ctx->inflight_lock); /* We need to keep going until we don't find a matching req */ - if (!cancel_req) + if (!found) break; - /* cancel this request, or head link requests */ - io_attempt_cancel(ctx, cancel_req); - io_cqring_overflow_flush(ctx, true, task, files); - io_put_req(cancel_req); + io_wq_cancel_cb(ctx->io_wq, io_cancel_task_cb, &cancel, true); + io_poll_remove_all(ctx, task, files); + io_kill_timeouts(ctx, task, files); /* cancellations _may_ trigger task work */ io_run_task_work(); schedule(); @@ -8696,22 +8604,15 @@ static void io_uring_cancel_files(struct io_ring_ctx *ctx, } } -static bool io_cancel_task_cb(struct io_wq_work *work, void *data) -{ - struct io_kiocb *req = container_of(work, struct io_kiocb, work); - struct task_struct *task = data; - - return io_task_match(req, task); -} - static void __io_uring_cancel_task_requests(struct io_ring_ctx *ctx, struct task_struct *task) { while (1) { + struct io_task_cancel cancel = { .task = task, .files = NULL, }; enum io_wq_cancel cret; bool ret = false; - cret = io_wq_cancel_cb(ctx->io_wq, io_cancel_task_cb, task, true); + cret = io_wq_cancel_cb(ctx->io_wq, io_cancel_task_cb, &cancel, true); if (cret != IO_WQ_CANCEL_NOTFOUND) ret = true; From patchwork Tue Feb 9 04:47:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 380006 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 070ACC433DB for ; Tue, 9 Feb 2021 04:53:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B06ED64E7A for ; Tue, 9 Feb 2021 04:53:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229850AbhBIExR (ORCPT ); Mon, 8 Feb 2021 23:53:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229891AbhBIExK (ORCPT ); Mon, 8 Feb 2021 23:53:10 -0500 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB739C061797 for ; Mon, 8 Feb 2021 20:51:52 -0800 (PST) Received: by mail-ej1-x636.google.com with SMTP id lg21so29292945ejb.3 for ; Mon, 08 Feb 2021 20:51:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pynK/xnd+nC2KvWAKX/Qo0kFMdqW3cOoRzYMswrz2gk=; b=h5wOOFeDtkZdPQHuRmtJB4zPGxqymzhC/Utfao/yf8cqUM9wYY/Km+UrHadNGqNx7k EeU1Tk1elcTLEtn87yQQh4km8MXR6K5NcIfFFmoysgJp0gJlMeSgdtPMxl+sCPSmia0P WZiR1ILrb1eQKM/IWvbOXbO7eQshnwBd48Yo5WcuW2eLz6lidSkghcko7VDww7oYW0ac TuidNpJjTobyYIAA0Q6huUFDuraV+hAzrM7RS/SpJhZb5Ci5vo4MqJyuQwXbOaiAIiqT OVsPNSJlTANM/BLlq0qxG3Fv7S5n40/0VbHSbIu4ZTHo7T+rwgoOfVEFyoCZJ4/N7yug Q4Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pynK/xnd+nC2KvWAKX/Qo0kFMdqW3cOoRzYMswrz2gk=; b=WZzt98eR44J+BW9ZwzWmQRfWDGuDxX+Xmg/xjdawping2sr/bVdCwtWUHRYVOeZBek H8s8SY1qxNwk3Q9P0cIgu6xoV8gY2HUBibqsT+M90SB49aPtUxLvY33iBxCKAUK0XZxV 0gpQnvQVltVlVUXtppkvjR3UfesBeXOIePV9yrIh4cIAmFaIrItoxLlwklZl0N6lx1VH m9Aoj/SJQLDUZCWPNN6ZPab5agUmArZrfr5Qu5V4XAtL7He566jR9qfGlLS8VPPSoWEf 9nPKJNI+EBd2vIRhF+YBA5n1TPFQYQzSseJOZGr9HCWUW6xgpClSUVZ7Wyqrj6ernwjc /7MA== X-Gm-Message-State: AOAM531PPg167vZpgX8ptCUo/oal2CzvRAiqoXt1U5R/QahjjjhcVY11 nLo+amDTf2VJKPwv1D2HHL+7ThBHIgs= X-Google-Smtp-Source: ABdhPJyQbQqZzqAFofm451aVDi+xWTQpJVaNYAV53iM7HPVkQ96RjV335oNtxvPgqaNKjP3WLult2Q== X-Received: by 2002:a17:906:1d51:: with SMTP id o17mr20467646ejh.85.1612846311219; Mon, 08 Feb 2021 20:51:51 -0800 (PST) Received: from localhost.localdomain ([148.252.128.244]) by smtp.gmail.com with ESMTPSA id g9sm9973445ejp.55.2021.02.08.20.51.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Feb 2021 20:51:50 -0800 (PST) From: Pavel Begunkov To: stable@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 07/16] io_uring: account io_uring internal files as REQ_F_INFLIGHT Date: Tue, 9 Feb 2021 04:47:41 +0000 Message-Id: <9c1d1608c20fad5b92196b550ec0b7463046590f.1612845821.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Jens Axboe [ Upstream commit 02a13674fa0e8dd326de8b9f4514b41b03d99003 ] We need to actively cancel anything that introduces a potential circular loop, where io_uring holds a reference to itself. If the file in question is an io_uring file, then add the request to the inflight list. Cc: stable@vger.kernel.org # 5.9+ Signed-off-by: Jens Axboe Signed-off-by: Pavel Begunkov --- fs/io_uring.c | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index a40ee81e6438..9801fa9b00ba 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1000,6 +1000,9 @@ static inline void io_clean_op(struct io_kiocb *req) static inline bool __io_match_files(struct io_kiocb *req, struct files_struct *files) { + if (req->file && req->file->f_op == &io_uring_fops) + return true; + return ((req->flags & REQ_F_WORK_INITIALIZED) && (req->work.flags & IO_WQ_WORK_FILES)) && req->work.identity->files == files; @@ -1398,11 +1401,14 @@ static bool io_grab_identity(struct io_kiocb *req) return false; atomic_inc(&id->files->count); get_nsproxy(id->nsproxy); - req->flags |= REQ_F_INFLIGHT; - spin_lock_irq(&ctx->inflight_lock); - list_add(&req->inflight_entry, &ctx->inflight_list); - spin_unlock_irq(&ctx->inflight_lock); + if (!(req->flags & REQ_F_INFLIGHT)) { + req->flags |= REQ_F_INFLIGHT; + + spin_lock_irq(&ctx->inflight_lock); + list_add(&req->inflight_entry, &ctx->inflight_list); + spin_unlock_irq(&ctx->inflight_lock); + } req->work.flags |= IO_WQ_WORK_FILES; } if (!(req->work.flags & IO_WQ_WORK_MM) && @@ -5886,8 +5892,10 @@ static void io_req_drop_files(struct io_kiocb *req) struct io_ring_ctx *ctx = req->ctx; unsigned long flags; - put_files_struct(req->work.identity->files); - put_nsproxy(req->work.identity->nsproxy); + if (req->work.flags & IO_WQ_WORK_FILES) { + put_files_struct(req->work.identity->files); + put_nsproxy(req->work.identity->nsproxy); + } spin_lock_irqsave(&ctx->inflight_lock, flags); list_del(&req->inflight_entry); spin_unlock_irqrestore(&ctx->inflight_lock, flags); @@ -6159,6 +6167,15 @@ static struct file *io_file_get(struct io_submit_state *state, file = __io_file_get(state, fd); } + if (file && file->f_op == &io_uring_fops) { + io_req_init_async(req); + req->flags |= REQ_F_INFLIGHT; + + spin_lock_irq(&ctx->inflight_lock); + list_add(&req->inflight_entry, &ctx->inflight_list); + spin_unlock_irq(&ctx->inflight_lock); + } + return file; } @@ -8578,8 +8595,7 @@ static void io_uring_cancel_files(struct io_ring_ctx *ctx, spin_lock_irq(&ctx->inflight_lock); list_for_each_entry(req, &ctx->inflight_list, inflight_entry) { - if (req->task != task || - req->work.identity->files != files) + if (!io_match_task(req, task, files)) continue; found = true; break; From patchwork Tue Feb 9 04:47:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 380005 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 178A9C433E0 for ; Tue, 9 Feb 2021 04:53:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA2AA64E7A for ; Tue, 9 Feb 2021 04:53:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229913AbhBIExU (ORCPT ); Mon, 8 Feb 2021 23:53:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230018AbhBIExL (ORCPT ); Mon, 8 Feb 2021 23:53:11 -0500 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4B48C0617A9 for ; Mon, 8 Feb 2021 20:51:55 -0800 (PST) Received: by mail-ej1-x630.google.com with SMTP id p20so29274800ejb.6 for ; Mon, 08 Feb 2021 20:51:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XnTh1kq6yYReSLQmjbQgzSeBr0n/F6+SFtHddS6S2EA=; b=MB3r7wgLjONzAdTgW/85qr1LUwgnGMnfv2W8y3S8kNebKmVKJkiZvqL7MhTkWswsxd rQHaGMrsgpHrQAoe12yQcP3XdjxOXib8IK8ExxHHdfCW1xazNp1A90lvIU9TkRUsqpfa oVFrGPqmOSQT0PfFuDvbMds5fSAW5HhKJTzvlDE1iEBWlHZ0OUedNrdC/PLDTrrEs3eI fT0nx1AybOySW2kwoU0ugSvtLulRh/9ihtX9d93XGDRgIzFpfep8qq4KikaVIHd+8cnd KF8+6MDjwl9hu/7o5mg+Wl7HT/0a+WLHcMPQ7DWR++r0ipCWSvxoDeQsmNkjQq2ccrRv 8Zgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XnTh1kq6yYReSLQmjbQgzSeBr0n/F6+SFtHddS6S2EA=; b=V8ckqXCM5zmtIU5sQZPdX5aSDrAmca1S2HayTAEMouszXUN/krPZ3R40Jtbe2H+n0T GfA9Y2RcQG6nVjtn7m5QJtGdUkZuhCveoqKmRcOYXDmalN66Hc2mAEFLL7Z1yP4Mk3+2 gCRM9IBN9qSMGpCcrdDkDR/VQuFdCQaJDZRl/fIcklP5pivNt6EMU2pOrw333jzBKQRR rEOxOCG5sgCpeWYE1EetMV+uwTCs9+9fh0Ww3LLqJlGg7H0xzo8jio8EmXhfQopvQ9UX XstR01/8RagbduQAu873yILXJyzPsQ7V6S7gPBd+IR25TMXfD+Bh8o6i4fyCJ7aLYxGC nE7g== X-Gm-Message-State: AOAM5315Ixzkh2SlZrhkDhTKEhGfW/tJVh5ivm88W0T+BtwqSSqzMsRF glTONGNVPqclg6kJDBxqY4OOUPFVGVQ= X-Google-Smtp-Source: ABdhPJxdoALmWeTpHLp+y5LPEQt+VAy8BDx/KV5Kfjq5Dv3GHPwZqo7P1B7fWVIj+eQlD3UcJEgGhA== X-Received: by 2002:a17:906:3fc1:: with SMTP id k1mr21176395ejj.58.1612846313073; Mon, 08 Feb 2021 20:51:53 -0800 (PST) Received: from localhost.localdomain ([148.252.128.244]) by smtp.gmail.com with ESMTPSA id g9sm9973445ejp.55.2021.02.08.20.51.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Feb 2021 20:51:52 -0800 (PST) From: Pavel Begunkov To: stable@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 09/16] io_uring: fix __io_uring_files_cancel() with TASK_UNINTERRUPTIBLE Date: Tue, 9 Feb 2021 04:47:43 +0000 Message-Id: <39d4dfb46962c4d3956f768bdf7395601d9fbae2.1612845821.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org [ Upstream commit a1bb3cd58913338e1b627ea6b8c03c2ae82d293f ] If the tctx inflight number haven't changed because of cancellation, __io_uring_task_cancel() will continue leaving the task in TASK_UNINTERRUPTIBLE state, that's not expected by __io_uring_files_cancel(). Ensure we always call finish_wait() before retrying. Cc: stable@vger.kernel.org # 5.9+ Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- fs/io_uring.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 95faa3d913b1..170f980c3243 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -8835,15 +8835,15 @@ void __io_uring_task_cancel(void) prepare_to_wait(&tctx->wait, &wait, TASK_UNINTERRUPTIBLE); /* - * If we've seen completions, retry. This avoids a race where - * a completion comes in before we did prepare_to_wait(). + * If we've seen completions, retry without waiting. This + * avoids a race where a completion comes in before we did + * prepare_to_wait(). */ - if (inflight != tctx_inflight(tctx)) - continue; - schedule(); + if (inflight == tctx_inflight(tctx)) + schedule(); + finish_wait(&tctx->wait, &wait); } while (1); - finish_wait(&tctx->wait, &wait); atomic_dec(&tctx->in_idle); io_uring_remove_task_files(tctx); From patchwork Tue Feb 9 04:47:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 380004 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36023C433E0 for ; Tue, 9 Feb 2021 04:53:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F106264E9D for ; Tue, 9 Feb 2021 04:53:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230020AbhBIExX (ORCPT ); Mon, 8 Feb 2021 23:53:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230029AbhBIExL (ORCPT ); Mon, 8 Feb 2021 23:53:11 -0500 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84151C0617AB for ; Mon, 8 Feb 2021 20:51:57 -0800 (PST) Received: by mail-ed1-x52b.google.com with SMTP id s3so21892121edi.7 for ; Mon, 08 Feb 2021 20:51:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Fz7TC4lv21r3oiKSo78gV7ZqqZF+k/bhoq0zw/cytOQ=; b=SYjNvwc2MpEncBKHA9xEqcjNy9r92oxPO0T7Jqti3T1rFVn9unEmW1bAX+Ju/10d35 tzJ6w3hYMtVchBf0wqJD4JONKQpREXhL/7FfP7Hn2ZMVqbaeqbD0VmLNgscO+81XFLqK zaR3f/kJg4XaVJe54LXbfd1PxMyhSr7MD1td6roGKnJQLKIWeFMat5yQyZFNf0PzIlfo xKq7YufjCVoN2YvnIr7lhAi68RFpooLpffkKe4YBDRrTLwTMl1dvjkmasoJxr/qgrM4r I8pAzdpzdKw//E0IQunKNlAKO6rLU/cq28bw7URBz9yJPl9dmIMU9Lb2Z5h3YjhPNbY8 DaZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Fz7TC4lv21r3oiKSo78gV7ZqqZF+k/bhoq0zw/cytOQ=; b=Zbm9MK6f7t2jpxwPg2AXh9r2X+M905UkvwCqLxQ+fnHDNX6C4h/IjTyFlueYu29kJp F3SUB7fc8fVkJoK5/BZsteB7KHNws/ZdKfWhvI88ifH/Zr1o+UMawnt9zg9Q7XGA9rXq YR+uzykbxnhgdHchbdDCg9jPMFMogZp7kBsCDQGFsndhLrwmH/KXE/VITFIp7BIp2OXU bdrLCndzR22I4UkH9V7+84zgK6Hs1wB+M4jIZQCOF1g4VM8OG+mmDEhqpS1H7sJvuc5i Cl1ENdK83Io3eLtdnmCxGuyChlMKXL+R0gp5dDVQhG1FOttc5CkOsV6atH8wWsDYdD+2 3ouA== X-Gm-Message-State: AOAM530G8oCwTtxilaXum8ZYg2ZIqe1X8J9aktEYIG8aORPGefQ61VCw LsY3wNKPpENcMDmdBfjchoztcmIng8Y= X-Google-Smtp-Source: ABdhPJydKq0WT1Lo6oePAqxpbNYozRpWQnVr8Ygbt3apajoOQZr5oijVyzqvxOgrCVimr3cPBvtJyQ== X-Received: by 2002:a05:6402:702:: with SMTP id w2mr21219038edx.78.1612846316025; Mon, 08 Feb 2021 20:51:56 -0800 (PST) Received: from localhost.localdomain ([148.252.128.244]) by smtp.gmail.com with ESMTPSA id g9sm9973445ejp.55.2021.02.08.20.51.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Feb 2021 20:51:55 -0800 (PST) From: Pavel Begunkov To: stable@vger.kernel.org Cc: Jens Axboe , syzbot+f655445043a26a7cfab8@syzkaller.appspotmail.com Subject: [PATCH 11/16] io_uring: fix cancellation taking mutex while TASK_UNINTERRUPTIBLE Date: Tue, 9 Feb 2021 04:47:45 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org [ Upstream commit ca70f00bed6cb255b7a9b91aa18a2717c9217f70 ] do not call blocking ops when !TASK_RUNNING; state=2 set at [<00000000ced9dbfc>] prepare_to_wait+0x1f4/0x3b0 kernel/sched/wait.c:262 WARNING: CPU: 1 PID: 19888 at kernel/sched/core.c:7853 __might_sleep+0xed/0x100 kernel/sched/core.c:7848 RIP: 0010:__might_sleep+0xed/0x100 kernel/sched/core.c:7848 Call Trace: __mutex_lock_common+0xc4/0x2ef0 kernel/locking/mutex.c:935 __mutex_lock kernel/locking/mutex.c:1103 [inline] mutex_lock_nested+0x1a/0x20 kernel/locking/mutex.c:1118 io_wq_submit_work+0x39a/0x720 fs/io_uring.c:6411 io_run_cancel fs/io-wq.c:856 [inline] io_wqe_cancel_pending_work fs/io-wq.c:990 [inline] io_wq_cancel_cb+0x614/0xcb0 fs/io-wq.c:1027 io_uring_cancel_files fs/io_uring.c:8874 [inline] io_uring_cancel_task_requests fs/io_uring.c:8952 [inline] __io_uring_files_cancel+0x115d/0x19e0 fs/io_uring.c:9038 io_uring_files_cancel include/linux/io_uring.h:51 [inline] do_exit+0x2e6/0x2490 kernel/exit.c:780 do_group_exit+0x168/0x2d0 kernel/exit.c:922 get_signal+0x16b5/0x2030 kernel/signal.c:2770 arch_do_signal_or_restart+0x8e/0x6a0 arch/x86/kernel/signal.c:811 handle_signal_work kernel/entry/common.c:147 [inline] exit_to_user_mode_loop kernel/entry/common.c:171 [inline] exit_to_user_mode_prepare+0xac/0x1e0 kernel/entry/common.c:201 __syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline] syscall_exit_to_user_mode+0x48/0x190 kernel/entry/common.c:302 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Rewrite io_uring_cancel_files() to mimic __io_uring_task_cancel()'s counting scheme, so it does all the heavy work before setting TASK_UNINTERRUPTIBLE. Cc: stable@vger.kernel.org # 5.9+ Reported-by: syzbot+f655445043a26a7cfab8@syzkaller.appspotmail.com Signed-off-by: Pavel Begunkov [axboe: fix inverted task check] Signed-off-by: Jens Axboe --- fs/io_uring.c | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index e3ae0daf97f7..03c1185270d1 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -8586,30 +8586,31 @@ static void io_cancel_defer_files(struct io_ring_ctx *ctx, } } +static int io_uring_count_inflight(struct io_ring_ctx *ctx, + struct task_struct *task, + struct files_struct *files) +{ + struct io_kiocb *req; + int cnt = 0; + + spin_lock_irq(&ctx->inflight_lock); + list_for_each_entry(req, &ctx->inflight_list, inflight_entry) + cnt += io_match_task(req, task, files); + spin_unlock_irq(&ctx->inflight_lock); + return cnt; +} + static void io_uring_cancel_files(struct io_ring_ctx *ctx, struct task_struct *task, struct files_struct *files) { while (!list_empty_careful(&ctx->inflight_list)) { struct io_task_cancel cancel = { .task = task, .files = files }; - struct io_kiocb *req; DEFINE_WAIT(wait); - bool found = false; - - spin_lock_irq(&ctx->inflight_lock); - list_for_each_entry(req, &ctx->inflight_list, inflight_entry) { - if (!io_match_task(req, task, files)) - continue; - found = true; - break; - } - if (found) - prepare_to_wait(&task->io_uring->wait, &wait, - TASK_UNINTERRUPTIBLE); - spin_unlock_irq(&ctx->inflight_lock); + int inflight; - /* We need to keep going until we don't find a matching req */ - if (!found) + inflight = io_uring_count_inflight(ctx, task, files); + if (!inflight) break; io_wq_cancel_cb(ctx->io_wq, io_cancel_task_cb, &cancel, true); @@ -8617,7 +8618,11 @@ static void io_uring_cancel_files(struct io_ring_ctx *ctx, io_kill_timeouts(ctx, task, files); /* cancellations _may_ trigger task work */ io_run_task_work(); - schedule(); + + prepare_to_wait(&task->io_uring->wait, &wait, + TASK_UNINTERRUPTIBLE); + if (inflight == io_uring_count_inflight(ctx, task, files)) + schedule(); finish_wait(&task->io_uring->wait, &wait); } } From patchwork Tue Feb 9 04:47:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 380003 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98921C433E9 for ; Tue, 9 Feb 2021 04:53:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 565AF64EB4 for ; Tue, 9 Feb 2021 04:53:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230034AbhBIExZ (ORCPT ); Mon, 8 Feb 2021 23:53:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230038AbhBIExL (ORCPT ); Mon, 8 Feb 2021 23:53:11 -0500 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82CA9C06121C for ; Mon, 8 Feb 2021 20:51:59 -0800 (PST) Received: by mail-ed1-x52f.google.com with SMTP id s11so21912052edd.5 for ; Mon, 08 Feb 2021 20:51:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3mk+54N0sp9sB5kMDzgz4t7dZbdpPAhaoei/vff2jSo=; b=PIPsbjKJzds0F8+/GXi2O04IbZX8payUiPAgR1wSuS9oCBwgtqigSW0+IXu+y4SGjd NDdIYItBu/j+FHKsuD4FmRhuAXSzeF6YLjij5r+g0EfyLXslLzeMSct4tEB4PI1kGHwi UPFPY7glsT0XlPjd4nvEOmSP/nHjXzDN7clFK2eaA4PIMX8kkuDeibtf0dle0cbvYAwU zHKxLmMBDcGzTXB8lfBwvQilJO36G2yF1eyrSznzGTPRtGQWx1h06cV9NQHG/zKjKET1 O2ivVdB2rWsO65fec+CEBalcRZO8YAEpLyTxrUKFCURi9QmiRroUeMx2Le4xlZFJPYDw AQ/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3mk+54N0sp9sB5kMDzgz4t7dZbdpPAhaoei/vff2jSo=; b=flBWS7CvxqFqNqVsjtZ8+AThVy01pnON5xFuE7lqJ7BRDvjQ8D72g1E4K4VED5SXAe QBPoxEw1PaEHmqxkVYaddnn+5JMXWR1P42HBE7dXgxWsgYZgePPEg35ViPCZcObWQXeH zek04drfDxZmViiqNKofwaEhpUEAuYfC+tAzN9TIWgRkvyx15OXP4L+vegSJLs9Ie/xf Fo1gQGY/aLBxH21rFFwR+i/1J+kXx3IUPyDprHLuhbw/BXPZvVJ6xyjJLCq1IIYCokva gwwSWxGrPS+QzABMA+CebDY20hEuIF2cZcAXCfY7ymdwAzq+8FaQCM19qw5AcJA/SUOw kE8Q== X-Gm-Message-State: AOAM533sqLeiqKqKYHv9KBMn8WbEKViIguJu2kiiPE7noOauq/q0eXA+ ANTuqSI4rkEAH0Rh2Ze1ucIWT+iE7UQ= X-Google-Smtp-Source: ABdhPJxtsmZkxhvjQa0OizH8ktK4ZiG0m5+D7D880NKJL62dbQt2rmR9QsqNHUTklCLlmHcB0kNxVw== X-Received: by 2002:aa7:d5c1:: with SMTP id d1mr20502980eds.359.1612846318072; Mon, 08 Feb 2021 20:51:58 -0800 (PST) Received: from localhost.localdomain ([148.252.128.244]) by smtp.gmail.com with ESMTPSA id g9sm9973445ejp.55.2021.02.08.20.51.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Feb 2021 20:51:57 -0800 (PST) From: Pavel Begunkov To: stable@vger.kernel.org Cc: Jens Axboe , syzbot+6879187cf57845801267@syzkaller.appspotmail.com Subject: [PATCH 13/16] io_uring: fix list corruption for splice file_get Date: Tue, 9 Feb 2021 04:47:47 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org [ Upstream commit f609cbb8911e40e15f9055e8f945f926ac906924 ] kernel BUG at lib/list_debug.c:29! Call Trace: __list_add include/linux/list.h:67 [inline] list_add include/linux/list.h:86 [inline] io_file_get+0x8cc/0xdb0 fs/io_uring.c:6466 __io_splice_prep+0x1bc/0x530 fs/io_uring.c:3866 io_splice_prep fs/io_uring.c:3920 [inline] io_req_prep+0x3546/0x4e80 fs/io_uring.c:6081 io_queue_sqe+0x609/0x10d0 fs/io_uring.c:6628 io_submit_sqe fs/io_uring.c:6705 [inline] io_submit_sqes+0x1495/0x2720 fs/io_uring.c:6953 __do_sys_io_uring_enter+0x107d/0x1f30 fs/io_uring.c:9353 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xa9 io_file_get() may be called from splice, and so REQ_F_INFLIGHT may already be set. Fixes: 02a13674fa0e8 ("io_uring: account io_uring internal files as REQ_F_INFLIGHT") Cc: stable@vger.kernel.org # 5.9+ Reported-by: syzbot+6879187cf57845801267@syzkaller.appspotmail.com Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- fs/io_uring.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index b09a59edf6aa..296b4cb44c7d 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -6170,7 +6170,8 @@ static struct file *io_file_get(struct io_submit_state *state, file = __io_file_get(state, fd); } - if (file && file->f_op == &io_uring_fops) { + if (file && file->f_op == &io_uring_fops && + !(req->flags & REQ_F_INFLIGHT)) { io_req_init_async(req); req->flags |= REQ_F_INFLIGHT; From patchwork Tue Feb 9 04:47:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 380002 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11436C433DB for ; Tue, 9 Feb 2021 04:53:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BDE8164E8A for ; Tue, 9 Feb 2021 04:53:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230048AbhBIExh (ORCPT ); Mon, 8 Feb 2021 23:53:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230038AbhBIEx1 (ORCPT ); Mon, 8 Feb 2021 23:53:27 -0500 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 521D2C06121E for ; Mon, 8 Feb 2021 20:52:01 -0800 (PST) Received: by mail-ej1-x62f.google.com with SMTP id f14so29254006ejc.8 for ; Mon, 08 Feb 2021 20:52:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=s61aGjjqahl4JKgzDvB3FLYKY5kavUzgJ00+m2swdPI=; b=erBR8uuu5iYfYfCKTlby5qzXm/2MmmyaKmMqF+0aqQxtohYmrr1Y9EU0La19ySU3Zk ZrfUXyS66gisq70i/mAP0wdlJB+my4nPFSMczrb+1LiLf9RUbJQIdJZcycC+xuECqAZL WU8Q6trL7lzc5UKuuwvz3wCOaP9L9RVDYK5Gzr9QKWGqX8S7w5mbgjUp2xS0OInNqm/a ouCr7SIyGo7rtB5k6kfPZRZ+QG2c3Jl2yt7XuB4nIdQGxcKdoSEDUpyKqNj/WUmgR1fb ZrjqBx2tJB2iMM+dfG+9cGa7OMrIGY+0/Qw7QwdOxHFAN74QlM4nE5c/DATpZ/zDvbwE UvOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s61aGjjqahl4JKgzDvB3FLYKY5kavUzgJ00+m2swdPI=; b=Rx2S/pgyhfp49OICTQ7W1QVp8PGNm+b9NvtXdRQuLPrO8cwtB6hY09NCzUBfIl9Lg3 qQj452ETKoscAI8gf678kzv22LwdVOnDGWp9UAe6I5Mfs8jpc2dCa1VjZIxPt/ez9Gth LZqshdEhB+xVepu2pSu1Lf/iY/wh6NIBQETv+5M5c1ssh4osSClOXICh95wXSyivf6lz wiYJOyocR8MbfAnCIZcI3TJp4YgSzVaw5z7pvqggsHW96yj/J6M+Pk1pekcagX98+eM4 sYw3Cp6k9JlYBPd/Y3cOGTGhXhzH9eL1xzNIHxRdub8ZfOzebJuxN1Zx9svspBUgQ8xe kKaw== X-Gm-Message-State: AOAM531OxICqUixRzzis0NprVXwQXzBbn58lzthqCaHYL0HibezxwsVB SadBqbtU/yXorasq/bq4SGtfgfLC9FM= X-Google-Smtp-Source: ABdhPJy3/osHMeTHp1sI3qVp5JSXSr9y6BKkkRCM3xZ6di1XfPKhcaYHanmnXkbEYb74lBVULwX8ZA== X-Received: by 2002:a17:906:3603:: with SMTP id q3mr15381538ejb.201.1612846319941; Mon, 08 Feb 2021 20:51:59 -0800 (PST) Received: from localhost.localdomain ([148.252.128.244]) by smtp.gmail.com with ESMTPSA id g9sm9973445ejp.55.2021.02.08.20.51.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Feb 2021 20:51:59 -0800 (PST) From: Pavel Begunkov To: stable@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 15/16] io_uring: reinforce cancel on flush during exit Date: Tue, 9 Feb 2021 04:47:49 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org [ Upstream commit 3a7efd1ad269ccaf9c1423364d97c9661ba6dafa ] What 84965ff8a84f0 ("io_uring: if we see flush on exit, cancel related tasks") really wants is to cancel all relevant REQ_F_INFLIGHT requests reliably. That can be achieved by io_uring_cancel_files(), but we'll miss it calling io_uring_cancel_task_requests(files=NULL) from io_uring_flush(), because it will go through __io_uring_cancel_task_requests(). Just always call io_uring_cancel_files() during cancel, it's good enough for now. Cc: stable@vger.kernel.org # 5.9+ Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- fs/io_uring.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 0bda8cc25845..5eaf3a7badcc 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -8692,10 +8692,9 @@ static void io_uring_cancel_task_requests(struct io_ring_ctx *ctx, io_cancel_defer_files(ctx, task, files); io_cqring_overflow_flush(ctx, true, task, files); + io_uring_cancel_files(ctx, task, files); if (!files) __io_uring_cancel_task_requests(ctx, task); - else - io_uring_cancel_files(ctx, task, files); if ((ctx->flags & IORING_SETUP_SQPOLL) && ctx->sq_data) { atomic_dec(&task->io_uring->in_idle);