From patchwork Tue Mar 8 00:39:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 550024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61039C433F5 for ; Tue, 8 Mar 2022 00:40:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238164AbiCHAlP (ORCPT ); Mon, 7 Mar 2022 19:41:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231687AbiCHAlO (ORCPT ); Mon, 7 Mar 2022 19:41:14 -0500 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A79F3C736 for ; Mon, 7 Mar 2022 16:40:19 -0800 (PST) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 227LJDms009309; Tue, 8 Mar 2022 00:40:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=MGiJam2DnM6U9MaBab+hw6LnBnRa1R54D8W29l/uEDA=; b=XW620OLOnT0sMyYaV1nP9dQofwLc6nnDRN9JPzSk6PmRbGxA6ZXrmxtjcKgy3l0nw3Oo zQHpmADJz8udmtdEnXL8+n/FpMGxybBAQVW6y3O/LWjcJo4PROPEjiaNfqzTgmQ3KN0/ F/Hxi9+jWc5lup3RLV2MiFvuEDQ8iUb2zcpUOHT0W3dHC4GssZXABJV7f5xFfuUuoN1F j6L4lzPuAS5eoM0bCXIvtx2hYRtHBcSZmYb/NeuMSvsuizyQUcwvNVpTvabRVcgJ7TjU GvGBNuYkoN3ZRan09s8pW0uQfdbS/3jj0TBfiwnV09uC5AIPWaNqNLdhDjIQBMz6Dwjd vw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3ekx9cdft0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 08 Mar 2022 00:40:11 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 2280alEB119127; Tue, 8 Mar 2022 00:40:10 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2175.outbound.protection.outlook.com [104.47.59.175]) by userp3020.oracle.com with ESMTP id 3em1ajmyke-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 08 Mar 2022 00:40:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PXl1QrmLgl+9+UbcxmtadWmOaROCEDRYaOxCBKVLJS0/9zoJCNe0M9nHTnRSgXWAMc7f/W1w0gL2JBcChIhLaqs2ua4wCD4Oxmmrl+4kQVfJhLUmdqIkKhrcKbH9gNVVNUhgMMv9iTBD8e8WEVzcVvMRVNlfAx0yw93YxbXupwFXQITtqce15kxg8RHmA9IYpXVm6709sp8cTurInX/+RzatVv1I0xELAFCSBUwBSok3z1V0SGb0UgmQlGuen+HYzVgEqp7qwyw6WYk2RauHKW9toFU5D88Asb0D3oIylVMRRq2TSoDCh9iJISwWw/SQIJNwbeRl87mCF/GdSl3miA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MGiJam2DnM6U9MaBab+hw6LnBnRa1R54D8W29l/uEDA=; b=nnJVicTy/zBiBEAO4dYDZTnWkPngdQWmUDhoUNLInPu9HUfM5Tiv5c48fvvDGkzfMfdWZGLzbM6xx+r4Z/TfG+QyYf1ZeR3OrI740YXTmjVUZ+NgnR0BL+F0w/gUem+orVByfYtNy4lmtKKJW9z0kFv57479BvtX/la9WFR3gn9VmuvrQvYg3UlDNrD+0q2WvpdkQsbdT8xJDq8/jueD5eZLbAQCAOwU9sTRDx/Q1YtTejhlVTk6qJTCLd8FzvuXveDUOkEFfcBjx3iKyCez2dezFn+fX0SVA+B4Do8EWUsQ8qx0SjDHG8q40owPMZFcAWL1qe0//KeYKsVI5kNKOA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MGiJam2DnM6U9MaBab+hw6LnBnRa1R54D8W29l/uEDA=; b=Wg3dDXWB1Nf5nR4E1Wqlva36e9DolQr4aYvDpFfWItrhk+4hMCG1PYC4FiypHvbLc9mxzsYdaKkcL3SSEcGyvkj7butACTXc437YNAu16oXqDukOjTlxC9kWBEo/PtojgrrClCv4dj+AcxXYySWtxXA0o1PrCnaYYMfdVMoS6Do= Received: from DM5PR10MB1466.namprd10.prod.outlook.com (10.168.103.135) by MN2PR10MB3680.namprd10.prod.outlook.com (20.179.98.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Tue, 8 Mar 2022 00:40:05 +0000 Received: from DM5PR10MB1466.namprd10.prod.outlook.com ([fe80::3dd8:6b8:e2e6:c3a2]) by DM5PR10MB1466.namprd10.prod.outlook.com ([fe80::3dd8:6b8:e2e6:c3a2%12]) with mapi id 15.20.5038.027; Tue, 8 Mar 2022 00:40:05 +0000 From: Mike Christie To: bvanassche@acm.org, lduncan@suse.com, cleech@redhat.com, ming.lei@redhat.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, james.bottomley@hansenpartnership.com Cc: Mike Christie Subject: [RFC PATCH 1/4] scsi: Allow drivers to set BLK_MQ_F_BLOCKING Date: Mon, 7 Mar 2022 18:39:54 -0600 Message-Id: <20220308003957.123312-2-michael.christie@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308003957.123312-1-michael.christie@oracle.com> References: <20220308003957.123312-1-michael.christie@oracle.com> X-ClientProxiedBy: DM6PR03CA0020.namprd03.prod.outlook.com (2603:10b6:5:40::33) To DM5PR10MB1466.namprd10.prod.outlook.com (2603:10b6:3:b::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f47aafe5-c2a0-4566-8a5d-08da009c3079 X-MS-TrafficTypeDiagnostic: MN2PR10MB3680:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: s/GuVeVeDraG0jQh8Emfn4ndyR5LGfx1cKdvmtrmlkt9h2oZgh9Pmd2IEkz5h+DGE5yctUXgn9XTXetBnik6mVKcO7DGuBy64AkxZAZ+4KxurbneM4VYu76aHRs+pgh/KR20XL8EsOR1+V1Ic9T2Fh3ZG7W7u8zNriYyVbMPbruL7RqoMwyoOF33YNkokMy7GKhexCdkS7X5N3Eu8ndSxqPPBgJ4i38yrgtA7fhoeuoeSPP1EqmPW8wway5vv88n4QsQTSUcn0HJlJ7uSSSiglZmdKhRtsXjfvyd9VtyH7cQObF7W70ah+9SJPq/slB9vtfThagsgBgGmYieAsIIHhsYJ2ngaY/REfjAQEtlyjwcnPAtd2+TNDRhMgOoFYtBdDI1V+B9k5VSnNNbWwcOMoHev5dbC1eaJ1fTFjH2GtKTZs+pVsjFlhhIVbiow2QmHZ4G0GsaatPHCbqM2HVfU/vzI4mY8ehz7XzIBmQMMkv45f9eWpTYBtwlpArwOI+V8RiMz63p9/6tIzkr8W0NtBnIVXp4rLiyfPll4Nr8BiUrKYq8aSplxH3yPtdzK/WvsY8TQEluAvlD/mNgyRPx+AAzCPpWkUTZzz0fmvb8MvuCMf9cMlRJeMRrMOzQoPkKkeXjWvS0Kq1IM2ennmT30A+VDRqq+EM/IR2RLiG3tY2g0vdJxG0d1pkYBOA7w/egrOk8A1CX0PwiopyEVc5AHw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM5PR10MB1466.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(5660300002)(508600001)(6506007)(52116002)(86362001)(8936002)(66556008)(6666004)(8676002)(66946007)(6486002)(6512007)(107886003)(1076003)(2906002)(26005)(186003)(4326008)(66476007)(38100700002)(38350700002)(316002)(36756003)(83380400001)(2616005); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: rQ/sPnGiZT9AbxxRxMCv+znG2Y5GIBwE2nr5fzY+4LtaqqJOmyH9QoaUIiM5oodyQ+mHKMC7EOEZ4lQ3q4BV74sJtNxN79Pe8vwjAw3Q2jlnK3gAOr6WqLcVtaw+DR/PSiIDUo02TgAROoZjBBKK7F/pFr2XZCE1XtRtqLHwJ8rSQ4vy2xzTcb6F2zUR3jYy9cGIWO2P+/7uFoD3uIqVkA6k89eoWnIqNxRUb43hGBnJVPMI4J65cHJxei4a2mRzPXiLKfOmK8lNwX0aeQTSVqnxr6DFlNVDFjGCr1IlUzZpYh6lLgjRFj8AaGUA2FmnBxQ1KW8Rn2lMaKFHmUaOYeQF1aeeywATVWubGPgL7Eq/J4Mgnhl2xNT2rGgroADgANQAWEi0lxUMHbveOMNVRX376uDi2nTSf2WMrTCVBzU/Br6Y5WvRtDwU3ceWFoKnQ7MzL9sMc2+YpAljijEAES+KCwmSghO/R4GPdPIl1qgQU1de8nezxj7pgQUJuunsj8QADrpi7wxUAkfTcDrhEUwoXU3VYINDCKM2vmKZNK9N3iZrvy6aIjOuRkQqypuVmvmSmwrAcq7abSMxlgbkvV12Xk43jiG3C8UEgn4zwZHnrtL5vwj7UKj2xfNDup6hn5bRmiIc2Uq8ODOVrdPIovc5+fU2ii46qWUFPjFgBDmM1T3DmQ7bapg/avllBcETS39nnnWYOXdlIs2Cv/6u2PXLfTDggK78Ah9OLrj6zkc7j4TvzXFcUmnlxgMBRGplKuFgJCTuxvMcs7ADEaCQ6fsLrV/1jX6W/akTrurSJdh6+PZG3dNGzQ5TzYiq1HN0OyTBtnKDIcFmeU2IuiL47mooX2hehTelsX5SpP/edi4+6nusR6jafzmCZei2Kj9IJz3kYpeIBA8imYCmrybr7WTXwS39lGfSdudYGH++p0r4OVl/vUWNIlIiWuJ5+Zz9jJO6a0qsoKVTciKNrK6IWgK8kogp/7mYbCaYNHzigWobkgg1GGjdcHliOmzr7U36i3K7CsulGfuGt8fPQzl0q4LjSch0T8XYi1ypI8pVveGscVzexzkl53dtsCgOi78C7g7koWlhrJ9LqlFpdw901qyEAJRiIRYuXquellW9NSa33l/cPm6NG9N0s+/Ws/7l8vIxwuQ4s2/uwUb+r31Di+p71+Q5YnxPt9zIlBKUhKKjLmn7jJWu113XqcKTDD+vi4kRm/0t0wZfc++2BkjnTkF3fYFjs344kNrv/nF5uEt9ozot9v358Uw51ReCg5nGth2N3P7U5+nzk61KTGHgLFFF/yETohUsivugPww1iGu+RLEauzeHjBpOeSKuMslfU1fbyO5q8aTSVh1ytfyG92+O2SdzM82ho/vwf6pWWbDPzGYxVtZ7gB1Ab96JoH5TD61X1NnMBqqbkqO2R2ewbeVLtf+89ZGEGvSDpKCF1bKAGlgAT6uoqmaKtTf0LqjYyyxu8ytAFFx+ISljnvCqUXzUkvU0C6BNgl+N5gmBvRz6cN0UAYEIf5LB3kDi61Yxwknbjg1SVpcFZTO/9rKWWOqn5BDWVm6CUrdVVShjAaJDDkEzSKCRs/Y4Ytyv2L900w6JWkTWMytHnkRGwgP3Grw//YD9VfN51xKsdwEoE5sUmllDiGXSYjR1hTWm3g8Wn3AWpj+8CMrrGkS8OmKzow== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f47aafe5-c2a0-4566-8a5d-08da009c3079 X-MS-Exchange-CrossTenant-AuthSource: DM5PR10MB1466.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 00:40:05.6391 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 98EiPT/ZNCTGZYpwtetL7PyRNYbMn7V2eFMkYwAtdn4J3a/mrVaHu8pOSGuCpAEUTYJlLakVNkK8sCxaWzVemZEQfECC/HVbospXPe/khRc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB3680 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10279 signatures=690470 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=942 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203080000 X-Proofpoint-ORIG-GUID: zgEMjOsFanlL6E9TVDMWT44DtCNSq2aR X-Proofpoint-GUID: zgEMjOsFanlL6E9TVDMWT44DtCNSq2aR Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org The software iscsi driver's queuecommand can block and taking the extra hop from kblockd to its workqueue results in a performance hit. Allowing it to set BLK_MQ_F_BLOCKING and transmit from that context directly results in a 20-30% improvement in IOPs for workloads like: fio --filename=/dev/sdb --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --numjobs=1 and for all write workloads. when using the none scheduler and the app and iscsi bound to the same CPUs. Throughput tests show similar gains. This patch adds a new scsi_host_template field so drivers can tell scsi-ml that they can block so scsi-ml can setup the tag set with the BLK_MQ_F_BLOCKING flag. Signed-off-by: Mike Christie --- drivers/scsi/scsi_lib.c | 6 ++++-- include/scsi/scsi_host.h | 4 ++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 0a70aa763a96..a5dbeb9994ae 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -1989,6 +1989,8 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) tag_set->driver_data = shost; if (shost->host_tagset) tag_set->flags |= BLK_MQ_F_TAG_HCTX_SHARED; + if (shost->hostt->queuecommand_blocks) + tag_set->flags |= BLK_MQ_F_BLOCKING; return blk_mq_alloc_tag_set(tag_set); } @@ -2952,8 +2954,8 @@ scsi_host_block(struct Scsi_Host *shost) } /* - * SCSI never enables blk-mq's BLK_MQ_F_BLOCKING flag so - * calling synchronize_rcu() once is enough. + * Drivers that use this helper enable blk-mq's BLK_MQ_F_BLOCKING flag + * so calling synchronize_rcu() once is enough. */ WARN_ON_ONCE(shost->tag_set.flags & BLK_MQ_F_BLOCKING); diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h index 72e1a347baa6..0d106dc9309d 100644 --- a/include/scsi/scsi_host.h +++ b/include/scsi/scsi_host.h @@ -75,6 +75,10 @@ struct scsi_host_template { */ int (* queuecommand)(struct Scsi_Host *, struct scsi_cmnd *); + /* + * Set To true if the queuecommand function can block. + */ + bool queuecommand_blocks; /* * The commit_rqs function is used to trigger a hardware * doorbell after some requests have been queued with From patchwork Tue Mar 8 00:39:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 549624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD32C433FE for ; Tue, 8 Mar 2022 00:40:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240503AbiCHAlP (ORCPT ); Mon, 7 Mar 2022 19:41:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237768AbiCHAlP (ORCPT ); Mon, 7 Mar 2022 19:41:15 -0500 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7DAB93DA6B for ; Mon, 7 Mar 2022 16:40:19 -0800 (PST) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 227LG1OE002100; Tue, 8 Mar 2022 00:40:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=r8OO8RTmpIpgxfgvDMnSgpXuwsXIYksXMlM2q6AW/os=; b=cKd7NfGaHZlSPZEbPBrXxNJEj8DgFLGrbOx+zCCI55TsbEhPLs45Bmen7HrTckco8S00 jrhlR2lQG0n0yHJRQjTIJkHr6ryPtz0EHZGWGzIs7itzUysVLhQcAJMzRxMTsuVHpYS/ K1ABq6xVEnM3esJkk2fPOs+QvlZgLrRJl9KFBAXpuNnyIxYBhb8rjtAI7WDrbi6A8r9q yO2s4fB0hzvKNvaiTrklBRTi/E4NiMcjjfu8QtRsOXAaYS3XMUncWiI5g1pMRRpXOS9X pEnDDmYNLXv/HYpvShXfRO4gPp3cZ60N96NvKljK8RPjUqczCdl0zr+cZFhaHgi7FkGz +w== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3ekyfsdh9u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 08 Mar 2022 00:40:12 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 2280alEC119127; Tue, 8 Mar 2022 00:40:11 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2175.outbound.protection.outlook.com [104.47.59.175]) by userp3020.oracle.com with ESMTP id 3em1ajmyke-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 08 Mar 2022 00:40:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EFlxil2C510tTUXbgRkRO2b087FeRTaKvAyIs/mqGZsL5dFQK6jQedV9e4C+h2aeX1Ji2q2plrSDiiP1GzFklES9/8x4tagKbBhCRmaLgdwd1Yy0w3EVEwiz08JF24qsEoQkNLE8SaFPpmkR709LHSNd+Yo9NQkz3A8qRVAdieTL49hpjHmxCr1weyyUNh32nMNKKz54CCoN9K9e8rPpth/Sf/W1bAt5CfLf+suujdWLu1YFl7xhLL49/hCMdZyV/O+wJYIOLeT1TVvZCGQJltOE7RxiWCDOj7+K1VPi8GWvSvYAXNDhrYC02fih6F5rEDpXMnamynYXJMxT6dzuRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=r8OO8RTmpIpgxfgvDMnSgpXuwsXIYksXMlM2q6AW/os=; b=c+j314NYzMddyaogJsh1dvrXSZhhgMP4fbRlX/iUPEcEQmOkQ9RpgZn3AkJsi9+2lOndLfp9rTyOiXKLnxfUDDXefvgeaG7A3CTKp8AUm7bxYtcrEra59XnE5w9baV/eJuHDWsGUdD58awmfiVG8Le/OooNgSOuvZ9g89a/tF9dGGw2T180AviuuQ4W2ncqrTM8QQdu42Aiy5bS66I8LXf727gFZ+M+cT2V2NNdQgijeANXBqX0+PNCTOPovXuHc51qKHZx01FQ9wnU2uP6QJBZ+hCFMCTIOLeyh+48ateWSflJBIuwu1CUrXz8S4dJuc16BXKQHItAN05zT/zF/Iw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=r8OO8RTmpIpgxfgvDMnSgpXuwsXIYksXMlM2q6AW/os=; b=vwBAQ/Ctg2Ri7yv0Wz8zLEcM8odLEE2y3wto9J9sKECgFylV72WwlgnLriPaefzh0MSbLsTCeonur1TsdrzYdLr7/pqVVpmDq7acZLyzTerfCl8x7QpHdIckWW8vnoGAK4wnJsJRHdP84mVHVa9VhxzOw64EsHhF18ov5A9wI6M= Received: from DM5PR10MB1466.namprd10.prod.outlook.com (10.168.103.135) by MN2PR10MB3680.namprd10.prod.outlook.com (20.179.98.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Tue, 8 Mar 2022 00:40:06 +0000 Received: from DM5PR10MB1466.namprd10.prod.outlook.com ([fe80::3dd8:6b8:e2e6:c3a2]) by DM5PR10MB1466.namprd10.prod.outlook.com ([fe80::3dd8:6b8:e2e6:c3a2%12]) with mapi id 15.20.5038.027; Tue, 8 Mar 2022 00:40:06 +0000 From: Mike Christie To: bvanassche@acm.org, lduncan@suse.com, cleech@redhat.com, ming.lei@redhat.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, james.bottomley@hansenpartnership.com Cc: Mike Christie Subject: [RFC PATCH 2/4] scsi: iscsi: Tell drivers when we must not block Date: Mon, 7 Mar 2022 18:39:55 -0600 Message-Id: <20220308003957.123312-3-michael.christie@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308003957.123312-1-michael.christie@oracle.com> References: <20220308003957.123312-1-michael.christie@oracle.com> X-ClientProxiedBy: DM6PR03CA0020.namprd03.prod.outlook.com (2603:10b6:5:40::33) To DM5PR10MB1466.namprd10.prod.outlook.com (2603:10b6:3:b::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a94204cb-a5e3-460f-5efd-08da009c30cf X-MS-TrafficTypeDiagnostic: MN2PR10MB3680:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NlYcE8u8NCJA15RcbLNiF6/B3KpcxHlO3eFcfsVveXM1RVd8CPvfjeBeLp74a3SDBGk0iDWSnj7wWenRP4pALlHCiRgsm7ggHxKaMRlsomv+UwK+4ZxnYH7oQuy6hR77/ood0mXxLHbfxqg2DO116+skorC9vyqyiT+5b5ttCu7bMLH5YuMcITtog+P/fCkRuXFJSzszBHFE8m0n25ErE/aW66pQ7u8xpEqAqo0vaXm16SflxjSwkXdAzT4H6JDNARsyKFmeJnFZ6x0PJyYK7YFRijEigbDX5iy2NEXMkzTUueb5hXSYKLpzqLBzPhbOdCb+PuZpr2xTLmRXrMLKJejVMwCXX1w9GHCJWObHHcNF/NlFzKGGClIq3ZsekXFDjaYbLZh49ZIuyQ/AH2CJEu8KCvgR2/8j5pUm1mD5TWHQOQJCORlGnTyxIcFcZ5KpDkp8NkzF3hLqIbdY85DRgYcemczBE4GKAgtkaSLLVRVf0UQVTU28nhiBBMR8wL4jUL+UpsfLmGIlzvdp9vZjcdZ7Q0dMk/AhmWHuiTMhXYdyCb0dWAJPch+3jybvh0ALfXLyXPC3GLqCBF9Qqt4HOHfPIXr0J+MxHqYCq6b4mmXWS8HnGUlwYg+BHn9sLLK3Sg7D9z3r3wcJYMxfN9voURH0o3EeVDYfkCzDR473+a+aWUPkKTMaGwC1f8IgJQxcLAGlDhMhYshS5MA+MlKm8g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM5PR10MB1466.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(5660300002)(508600001)(6506007)(52116002)(86362001)(8936002)(66556008)(30864003)(6666004)(8676002)(66946007)(6486002)(6512007)(107886003)(1076003)(2906002)(26005)(186003)(4326008)(66476007)(38100700002)(38350700002)(316002)(36756003)(83380400001)(2616005); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: XoTn4hazVxv+Z7di7Mu0QWg0zre/C8AmRXLbFod4w12j8eWDm9gL8pxj3RKF+oIdxB/Q2xD133KZATcMWtoIwQWF9VPANYtPfDPIWJd9HxMYENLCotu5FkIuMdbyDvg/lS6WSUCCLBXlaEWvyYAQaucwOSniBnvdFi8gpac+arPKML1+jNxOXkckw53pV+be4yBVBX1GnDjMcXbL918W14K8yms5d5bkZUOoG5CTKImoY+/cudMHPikQFefGePj1qHyW9KtU4Oyz4at+7tOxgHMkeZPlAB6VHBATdo5e/6FiRuxcJzfRmYVBdulDM0NFLEEKxDzJkeRWiE1tr3bjPkrWkKbG93DkjTUeCg19TiZV8YQH+VAeBAcCpWMEerOKEJyDa6OheBy68cmWJm1SiER5zfNqB37nYQqli97b6NLy6JC3LLN7hexMhlj/BlfQdJlw9FrTJE7tB794ygQSKGjIzae6qRTmS4evjYLs367/pugJth+NIQTPRGvmAJZCrSBK2ibQwcQYV3GodsMJNwdpPRjOa121dDRbQqPPj1Koyo7jmuc6XvL+bBGVhpd8jiLWULND7yOzg1Wr4aCtuT9CijZi5ECFFNXaaHpBsb2/0aYQjzG/6y99WFlItcWRyC3xNqL2wKLspgOhhynP526ySj5N/1ETgzb9owDqu97EWZXBEJ0NHf2odkY/trzdafVnGq4iFvkjftZxzXcziYrJsPtmwIiXEbrYlC94QdDddjpiGx3GNhCThmlIALnOgmYE6urNt964j9tlu5rmFrBtvj5SSylzwsxvaSclnZH+gO2az3Y7cKli1Cm1lw+WBmFHzSxOOip/uwWA4gXiZIFzjB3KIo4kTtgcL6aqvGx9sOjgp1xZXwNRk30oLjldBS2uOB/pgcLQegFdRxfwsLywXdQPU6j39hGoL4caR52XDwffVq8UzWnlvVT8Kzb0cXDkQ/uipkv/LVULaCmVDDHJOwQPyRcb4QLADojUjNgsYobCGCQkI5PaNaCM1j3Ls1YFfRhnqqc3uhP5IVAmSVFlwyLzqdyr7Ah8v1UVCWbLwI8AhyYm4jzZafOy2/wSBOQfOeuYD2RI5eRh/AOubwKg2CPxfZ7Y103mFjQkM5572oGHXPKETFprKfk1wA5H3Baen9M/ITQVkwVHzNK/KNlae0MoZD/r35Q/Iz1f0D0b7VKcQGoXMeYv+5iEnKlcomzuQ29ZuvfNYBX9GrhL2mmKPVPvXPyEDmoJTEMatscOL+AVrNOlaQA9cjfUK88mO7LsFFDFrAyLEdp0BP+oAsQssTlk/BUjy8Y7RrCo6V7lteSrzsQ/sgCx2i+V6cLJ+FQfPNv07TWv2l2b1BVA6/An+llr8IXDFEBJurlTabRssW2omCdrKVA+SdnvGcxj9bEtPF7EI9cbLaz5NUSmYTszJ0DigYowUupnl3p8RhYqbJIbnVBkBf39QyIEi8nM2EFwXBG53DVn1mk9LafVXFNrETgzt43ygsSUJmmKgUTl6kv60ustZlr/sdvOHF1XtiobtJ5hHCRIJE7lnUkxMrDiOyy/WK6ScMJA4H6EOuFrIPLW8SGsB5uRHUTz7sOh1OAICY/kUkg+LuC8S8xJWxZUh3yetGrpUPHtUjWgrijmEU8bD/kDX/kK+zQeWKp+ynK0qZVticW+UDv2cgZB9w== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a94204cb-a5e3-460f-5efd-08da009c30cf X-MS-Exchange-CrossTenant-AuthSource: DM5PR10MB1466.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 00:40:06.1235 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: BNj/f2it8bDt/s2au/qRiKHVP9QiNKsRr6+XRyfLNT/rYKBZ4pPg3U4V5DeGBIVlH8lX9cLbDkNQk6WXpR1YRPhqwCk7LanH+ZruD3fwiQU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB3680 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10279 signatures=690470 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203080000 X-Proofpoint-GUID: QNBmPZe5A8aSXajFwVcb9iMGWdqnfer2 X-Proofpoint-ORIG-GUID: QNBmPZe5A8aSXajFwVcb9iMGWdqnfer2 Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org When in the next patches we xmit from kblockd we will want to perform non-blocking IO so we don't affect other sessions while waiting for memory. This adds an arg to the task and pdu xmit functions so libiscsi can tell the drivers that support transmitting from both queuecommand and the iscsi_q wq when they can't block. Signed-off-by: Mike Christie --- drivers/infiniband/ulp/iser/iscsi_iser.c | 2 +- drivers/scsi/be2iscsi/be_main.c | 2 +- drivers/scsi/bnx2i/bnx2i_iscsi.c | 3 ++- drivers/scsi/cxgbi/libcxgbi.c | 2 +- drivers/scsi/cxgbi/libcxgbi.h | 2 +- drivers/scsi/iscsi_tcp.c | 16 +++++++++++----- drivers/scsi/libiscsi.c | 16 ++++++++-------- drivers/scsi/libiscsi_tcp.c | 5 +++-- drivers/scsi/qedi/qedi_iscsi.c | 2 +- drivers/scsi/qla4xxx/ql4_os.c | 4 ++-- include/scsi/libiscsi_tcp.h | 2 +- include/scsi/scsi_transport_iscsi.h | 4 ++-- 12 files changed, 34 insertions(+), 26 deletions(-) diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c index 07e47021a71f..244a4540dbf6 100644 --- a/drivers/infiniband/ulp/iser/iscsi_iser.c +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c @@ -317,7 +317,7 @@ static int iscsi_iser_task_xmit_unsol_data(struct iscsi_conn *conn, * * Return: zero on success or escalates $error on failure. */ -static int iscsi_iser_task_xmit(struct iscsi_task *task) +static int iscsi_iser_task_xmit(struct iscsi_task *task, bool dontwait) { struct iscsi_conn *conn = task->conn; struct iscsi_iser_task *iser_task = task->dd_data; diff --git a/drivers/scsi/be2iscsi/be_main.c b/drivers/scsi/be2iscsi/be_main.c index 36fddce2786d..fbd0c6981097 100644 --- a/drivers/scsi/be2iscsi/be_main.c +++ b/drivers/scsi/be2iscsi/be_main.c @@ -4745,7 +4745,7 @@ static int beiscsi_mtask(struct iscsi_task *task) return 0; } -static int beiscsi_task_xmit(struct iscsi_task *task) +static int beiscsi_task_xmit(struct iscsi_task *task, bool dontwait) { struct beiscsi_io_task *io_task = task->dd_data; struct scsi_cmnd *sc = task->sc; diff --git a/drivers/scsi/bnx2i/bnx2i_iscsi.c b/drivers/scsi/bnx2i/bnx2i_iscsi.c index a592ca8602f9..d887157c1c02 100644 --- a/drivers/scsi/bnx2i/bnx2i_iscsi.c +++ b/drivers/scsi/bnx2i/bnx2i_iscsi.c @@ -1213,10 +1213,11 @@ bnx2i_mtask_xmit(struct iscsi_conn *conn, struct iscsi_task *task) /** * bnx2i_task_xmit - transmit iscsi command to chip for further processing * @task: transport layer command structure pointer + * @dontwait: true if the driver should not block * * maps SG buffers and send request to chip/firmware in the form of SQ WQE */ -static int bnx2i_task_xmit(struct iscsi_task *task) +static int bnx2i_task_xmit(struct iscsi_task *task, bool dontwait) { struct iscsi_conn *conn = task->conn; struct iscsi_session *session = conn->session; diff --git a/drivers/scsi/cxgbi/libcxgbi.c b/drivers/scsi/cxgbi/libcxgbi.c index 411b0d386fad..6fc6cf1a0090 100644 --- a/drivers/scsi/cxgbi/libcxgbi.c +++ b/drivers/scsi/cxgbi/libcxgbi.c @@ -2358,7 +2358,7 @@ static int cxgbi_sock_send_skb(struct cxgbi_sock *csk, struct sk_buff *skb) return len; } -int cxgbi_conn_xmit_pdu(struct iscsi_task *task) +int cxgbi_conn_xmit_pdu(struct iscsi_task *task, bool dontwait) { struct iscsi_tcp_conn *tcp_conn = task->conn->dd_data; struct cxgbi_conn *cconn = tcp_conn->dd_data; diff --git a/drivers/scsi/cxgbi/libcxgbi.h b/drivers/scsi/cxgbi/libcxgbi.h index 3687b5c0cf90..a852dc31171a 100644 --- a/drivers/scsi/cxgbi/libcxgbi.h +++ b/drivers/scsi/cxgbi/libcxgbi.h @@ -604,7 +604,7 @@ void cxgbi_conn_tx_open(struct cxgbi_sock *); void cxgbi_conn_pdu_ready(struct cxgbi_sock *); int cxgbi_conn_alloc_pdu(struct iscsi_task *, u8); int cxgbi_conn_init_pdu(struct iscsi_task *, unsigned int , unsigned int); -int cxgbi_conn_xmit_pdu(struct iscsi_task *); +int cxgbi_conn_xmit_pdu(struct iscsi_task *, bool); void cxgbi_cleanup_task(struct iscsi_task *task); diff --git a/drivers/scsi/iscsi_tcp.c b/drivers/scsi/iscsi_tcp.c index 974245eab605..c2627505011d 100644 --- a/drivers/scsi/iscsi_tcp.c +++ b/drivers/scsi/iscsi_tcp.c @@ -279,6 +279,7 @@ iscsi_sw_tcp_conn_restore_callbacks(struct iscsi_conn *conn) * iscsi_sw_tcp_xmit_segment - transmit segment * @tcp_conn: the iSCSI TCP connection * @segment: the buffer to transmnit + * @dontwait: true if we should use MSG_DONTWAIT * * This function transmits as much of the buffer as * the network layer will accept, and returns the number of @@ -289,7 +290,8 @@ iscsi_sw_tcp_conn_restore_callbacks(struct iscsi_conn *conn) * it will retrieve the hash value and send it as well. */ static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn, - struct iscsi_segment *segment) + struct iscsi_segment *segment, + bool dontwait) { struct iscsi_sw_tcp_conn *tcp_sw_conn = tcp_conn->dd_data; struct socket *sk = tcp_sw_conn->sock; @@ -308,6 +310,9 @@ static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn, if (segment->total_copied + segment->size < segment->total_size) flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; + if (dontwait) + flags |= MSG_DONTWAIT; + /* Use sendpage if we can; else fall back to sendmsg */ if (!segment->data) { sg = segment->sg; @@ -350,8 +355,9 @@ static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn, /** * iscsi_sw_tcp_xmit - TCP transmit * @conn: iscsi connection + * @dontwait: true if we should perform nonblocking IO **/ -static int iscsi_sw_tcp_xmit(struct iscsi_conn *conn) +static int iscsi_sw_tcp_xmit(struct iscsi_conn *conn, bool dontwait) { struct iscsi_tcp_conn *tcp_conn = conn->dd_data; struct iscsi_sw_tcp_conn *tcp_sw_conn = tcp_conn->dd_data; @@ -360,7 +366,7 @@ static int iscsi_sw_tcp_xmit(struct iscsi_conn *conn) int rc = 0; while (1) { - rc = iscsi_sw_tcp_xmit_segment(tcp_conn, segment); + rc = iscsi_sw_tcp_xmit_segment(tcp_conn, segment, dontwait); /* * We may not have been able to send data because the conn * is getting stopped. libiscsi will know so propagate err @@ -411,7 +417,7 @@ static inline int iscsi_sw_tcp_xmit_qlen(struct iscsi_conn *conn) return segment->total_copied - segment->total_size; } -static int iscsi_sw_tcp_pdu_xmit(struct iscsi_task *task) +static int iscsi_sw_tcp_pdu_xmit(struct iscsi_task *task, bool dontwait) { struct iscsi_conn *conn = task->conn; unsigned int noreclaim_flag; @@ -428,7 +434,7 @@ static int iscsi_sw_tcp_pdu_xmit(struct iscsi_task *task) noreclaim_flag = memalloc_noreclaim_save(); while (iscsi_sw_tcp_xmit_qlen(conn)) { - rc = iscsi_sw_tcp_xmit(conn); + rc = iscsi_sw_tcp_xmit(conn, dontwait); if (rc == 0) { rc = -EAGAIN; break; diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c index fcf5c30614ba..63e0d97df50f 100644 --- a/drivers/scsi/libiscsi.c +++ b/drivers/scsi/libiscsi.c @@ -812,7 +812,7 @@ static int iscsi_send_mgmt_task(struct iscsi_task *task) if (rc) return rc; - rc = session->tt->xmit_task(task); + rc = session->tt->xmit_task(task, false); if (rc) return rc; } else { @@ -1498,7 +1498,7 @@ static int iscsi_check_cmdsn_window_closed(struct iscsi_conn *conn) } static int iscsi_xmit_task(struct iscsi_conn *conn, struct iscsi_task *task, - bool was_requeue) + bool was_requeue, bool dontwait) { int rc; @@ -1539,7 +1539,7 @@ static int iscsi_xmit_task(struct iscsi_conn *conn, struct iscsi_task *task, } spin_unlock_bh(&conn->session->frwd_lock); - rc = conn->session->tt->xmit_task(task); + rc = conn->session->tt->xmit_task(task, dontwait); spin_lock_bh(&conn->session->frwd_lock); if (!rc) { /* done with this task */ @@ -1608,7 +1608,7 @@ static int iscsi_data_xmit(struct iscsi_conn *conn) } if (conn->task) { - rc = iscsi_xmit_task(conn, conn->task, false); + rc = iscsi_xmit_task(conn, conn->task, false, false); if (rc) goto done; } @@ -1630,7 +1630,7 @@ static int iscsi_data_xmit(struct iscsi_conn *conn) spin_unlock_bh(&conn->session->back_lock); continue; } - rc = iscsi_xmit_task(conn, task, false); + rc = iscsi_xmit_task(conn, task, false, false); if (rc) goto done; } @@ -1652,7 +1652,7 @@ static int iscsi_data_xmit(struct iscsi_conn *conn) fail_scsi_task(task, DID_ABORT); continue; } - rc = iscsi_xmit_task(conn, task, false); + rc = iscsi_xmit_task(conn, task, false, false); if (rc) goto done; /* @@ -1678,7 +1678,7 @@ static int iscsi_data_xmit(struct iscsi_conn *conn) break; list_del_init(&task->running); - rc = iscsi_xmit_task(conn, task, true); + rc = iscsi_xmit_task(conn, task, true, false); if (rc) goto done; if (!list_empty(&conn->mgmtqueue)) @@ -1843,7 +1843,7 @@ int iscsi_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *sc) goto prepd_fault; } } - if (session->tt->xmit_task(task)) { + if (session->tt->xmit_task(task, true)) { session->cmdsn--; reason = FAILURE_SESSION_NOT_READY; goto prepd_reject; diff --git a/drivers/scsi/libiscsi_tcp.c b/drivers/scsi/libiscsi_tcp.c index defe08142b75..2c783862963d 100644 --- a/drivers/scsi/libiscsi_tcp.c +++ b/drivers/scsi/libiscsi_tcp.c @@ -1060,12 +1060,13 @@ static struct iscsi_r2t_info *iscsi_tcp_get_curr_r2t(struct iscsi_task *task) /** * iscsi_tcp_task_xmit - xmit normal PDU task * @task: iscsi command task + * @dontwait: true if the driver should not wait for wmem space * * We're expected to return 0 when everything was transmitted successfully, * -EAGAIN if there's still data in the queue, or != 0 for any other kind * of error. */ -int iscsi_tcp_task_xmit(struct iscsi_task *task) +int iscsi_tcp_task_xmit(struct iscsi_task *task, bool dontwait) { struct iscsi_conn *conn = task->conn; struct iscsi_session *session = conn->session; @@ -1074,7 +1075,7 @@ int iscsi_tcp_task_xmit(struct iscsi_task *task) flush: /* Flush any pending data first. */ - rc = session->tt->xmit_pdu(task); + rc = session->tt->xmit_pdu(task, dontwait); if (rc < 0) return rc; diff --git a/drivers/scsi/qedi/qedi_iscsi.c b/drivers/scsi/qedi/qedi_iscsi.c index 282ecb4e39bb..80a9bd4ef65e 100644 --- a/drivers/scsi/qedi/qedi_iscsi.c +++ b/drivers/scsi/qedi/qedi_iscsi.c @@ -827,7 +827,7 @@ static int qedi_mtask_xmit(struct iscsi_conn *conn, struct iscsi_task *task) return qedi_iscsi_send_generic_request(task); } -static int qedi_task_xmit(struct iscsi_task *task) +static int qedi_task_xmit(struct iscsi_task *task, bool dontwait) { struct iscsi_conn *conn = task->conn; struct qedi_conn *qedi_conn = conn->dd_data; diff --git a/drivers/scsi/qla4xxx/ql4_os.c b/drivers/scsi/qla4xxx/ql4_os.c index 955d8cb675f1..60f8c10c000d 100644 --- a/drivers/scsi/qla4xxx/ql4_os.c +++ b/drivers/scsi/qla4xxx/ql4_os.c @@ -137,7 +137,7 @@ qla4xxx_session_create(struct iscsi_endpoint *ep, uint16_t cmds_max, static void qla4xxx_session_destroy(struct iscsi_cls_session *sess); static void qla4xxx_task_work(struct work_struct *wdata); static int qla4xxx_alloc_pdu(struct iscsi_task *, uint8_t); -static int qla4xxx_task_xmit(struct iscsi_task *); +static int qla4xxx_task_xmit(struct iscsi_task *, bool); static void qla4xxx_task_cleanup(struct iscsi_task *); static void qla4xxx_fail_session(struct iscsi_cls_session *cls_session); static void qla4xxx_conn_get_stats(struct iscsi_cls_conn *cls_conn, @@ -3477,7 +3477,7 @@ static void qla4xxx_task_cleanup(struct iscsi_task *task) return; } -static int qla4xxx_task_xmit(struct iscsi_task *task) +static int qla4xxx_task_xmit(struct iscsi_task *task, bool dontwait) { struct scsi_cmnd *sc = task->sc; struct iscsi_session *sess = task->conn->session; diff --git a/include/scsi/libiscsi_tcp.h b/include/scsi/libiscsi_tcp.h index 7c8ba9d7378b..9fb97cbfa05c 100644 --- a/include/scsi/libiscsi_tcp.h +++ b/include/scsi/libiscsi_tcp.h @@ -88,7 +88,7 @@ extern int iscsi_tcp_recv_skb(struct iscsi_conn *conn, struct sk_buff *skb, unsigned int offset, bool offloaded, int *status); extern void iscsi_tcp_cleanup_task(struct iscsi_task *task); extern int iscsi_tcp_task_init(struct iscsi_task *task); -extern int iscsi_tcp_task_xmit(struct iscsi_task *task); +extern int iscsi_tcp_task_xmit(struct iscsi_task *task, bool dontwait); /* segment helpers */ extern int iscsi_tcp_recv_segment_is_hdr(struct iscsi_tcp_conn *tcp_conn); diff --git a/include/scsi/scsi_transport_iscsi.h b/include/scsi/scsi_transport_iscsi.h index 7a0d24d3b916..d3b37d28e11a 100644 --- a/include/scsi/scsi_transport_iscsi.h +++ b/include/scsi/scsi_transport_iscsi.h @@ -108,11 +108,11 @@ struct iscsi_transport { struct iscsi_stats *stats); int (*init_task) (struct iscsi_task *task); - int (*xmit_task) (struct iscsi_task *task); + int (*xmit_task) (struct iscsi_task *task, bool dontwait); void (*cleanup_task) (struct iscsi_task *task); int (*alloc_pdu) (struct iscsi_task *task, uint8_t opcode); - int (*xmit_pdu) (struct iscsi_task *task); + int (*xmit_pdu) (struct iscsi_task *task, bool dontwait); int (*init_pdu) (struct iscsi_task *task, unsigned int offset, unsigned int count); void (*parse_pdu_itt) (struct iscsi_conn *conn, itt_t itt, From patchwork Tue Mar 8 00:39:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 549623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90CFBC433FE for ; Tue, 8 Mar 2022 00:40:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244216AbiCHAl1 (ORCPT ); Mon, 7 Mar 2022 19:41:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242501AbiCHAlR (ORCPT ); Mon, 7 Mar 2022 19:41:17 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6030C3DA78 for ; Mon, 7 Mar 2022 16:40:22 -0800 (PST) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 227LLAl6031932; Tue, 8 Mar 2022 00:40:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=ehqCyzYLsxgcxEhuQVNRYODXoB2YR79PeZl9pdFm8ls=; b=lJh/+6SvypIX9zc7Oaoe0Cuwzb9TDu0rhAbGPcxZRmryGwuTuxRVkmXHObdsR8abLYc9 1FIpJkPW5YXLPeFfKD6JIgCJIwv0L89AvD4uZIcMUW7I3meSGcYyuEK6fwWH0mSJHqt0 mWDGz967BZQEIWo77CtLHSqsfO7Dz+knNPdrLzuw/V7YMYl8oczZZq/tBaTaGyHj6jhN 1lpJANd/r9xsQD4AizXoNbg1vpZnhathHMkhOjvFj6DpREFsM54TGzbkfd+d09KizXfY 2vnSni75EdzELnfZNsWm2Azo5hGb8BXehFts3GHWH2vfDrqvgeB4UFGV0oeA3mMLyN/g Lw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3ekyrancg5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 08 Mar 2022 00:40:13 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 2280alED119127; Tue, 8 Mar 2022 00:40:11 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2175.outbound.protection.outlook.com [104.47.59.175]) by userp3020.oracle.com with ESMTP id 3em1ajmyke-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 08 Mar 2022 00:40:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gyjrLml+6+fjKGDTljzqFhKPJV3WKwav5RJtP0NB4B5MVlBoBEZhVbBcopx6urNZzBum9icxv3r06/S9M8+IIz3WAkXrkJIEgssb6Y0SDVwCUi3izRpjDbiZlrOi6gSlRvMvaaFfQsUmiEqdweYep051GmCTkcLWVGCdpvs26RUn4VYq1rmSrpOm2w1oqFySirzns7yQnY67e7niYf/SchU7i+0J7vvLZXtBa5QL4nGzLjEUDklSxS/ogVfNs21QdzSYrQEdLvppcAA5QE0hiiWyssKVymtTDfCHZpOSa2EBGMw1gZsArQB5ezSLdWmPjtX9RNF0SYb3W93fPfILtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ehqCyzYLsxgcxEhuQVNRYODXoB2YR79PeZl9pdFm8ls=; b=GS1vtFORje5aCMmrARFcXQzx4lEd13sjLSOZFc1C7fUm+tNc2K/YMV/eWE++elnEkgVXM4wMzvRmZrnzrbE3ypSM60745BZz5HfhU3a62oroMbBBl0hSpMS17ZPeWZW2hdXYo3J8s0s4oMM0lAQ+ZJxQLFKwx0YNsNgZZpWIrnSRfaKJ95sr9ldVke0W30MGP/QhA598SIpwNiu4K8m3AHWUxPaNSgrxeuBiDbqN26ZDyyQgpnUqyDEfsgPfsSrggpfAFoOKJT7e+ReQKL+9tdGK2rS1k97tuXGs+rKhYT51roj9uphgghujySNUmxUUkTR8EXGAXTEPkz1uxVN8nQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ehqCyzYLsxgcxEhuQVNRYODXoB2YR79PeZl9pdFm8ls=; b=CB9X7FGhEj0g6NhR6P7Cfx4vzhtv8SllK0y9YKXL5iDTHIcNvGE4h1ZnxGTyxWcxgEGVW31z1XyJaCvDI3b5plcuXWTG/OgCYflQTqZmoU6zEnCfAP7AERbNMHc8q1XTx+CHTovrBVClF2mZouVYMLRGCPFBUzUX/JdacxrqCNc= Received: from DM5PR10MB1466.namprd10.prod.outlook.com (10.168.103.135) by MN2PR10MB3680.namprd10.prod.outlook.com (20.179.98.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Tue, 8 Mar 2022 00:40:06 +0000 Received: from DM5PR10MB1466.namprd10.prod.outlook.com ([fe80::3dd8:6b8:e2e6:c3a2]) by DM5PR10MB1466.namprd10.prod.outlook.com ([fe80::3dd8:6b8:e2e6:c3a2%12]) with mapi id 15.20.5038.027; Tue, 8 Mar 2022 00:40:06 +0000 From: Mike Christie To: bvanassche@acm.org, lduncan@suse.com, cleech@redhat.com, ming.lei@redhat.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, james.bottomley@hansenpartnership.com Cc: Mike Christie Subject: [RFC PATCH 3/4] scsi: iscsi: Support transmit from queuecommand Date: Mon, 7 Mar 2022 18:39:56 -0600 Message-Id: <20220308003957.123312-4-michael.christie@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308003957.123312-1-michael.christie@oracle.com> References: <20220308003957.123312-1-michael.christie@oracle.com> X-ClientProxiedBy: DM6PR03CA0020.namprd03.prod.outlook.com (2603:10b6:5:40::33) To DM5PR10MB1466.namprd10.prod.outlook.com (2603:10b6:3:b::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e7c566f2-ac91-4f74-8333-08da009c3114 X-MS-TrafficTypeDiagnostic: MN2PR10MB3680:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6DXjeYgttfBV7kQEbKyNIB/S3cwYqZeElb/+hof0uKeFC8INKtRCB2Zgi1M/1z69edsGi0Wa2Qy0Kp557mdC9e9OHVPTd/E0e4evw5gkY0JsktZWVk4qJhJCyximKpsp5igkJwVu6sm8rG4JQbjtr+TjzpucxdlrM8S5Bs2AcpQdATaz6Nr4NZg5HreR9eJ4WzS7u8ryr9ao6YjMPtsJkiCtW5slgeodAZSymoYVYqUT8GxRG9imks99d2zhlkHrgvV8aI9Ckdzon3hIj9an4F4wRMM3RVNt7RSsOFQSzYizid09B1wNO4KlKT7pJtub4jQONgoMZ/vt5ALEVj2K/CDksH+oXDmHugf7NAcJ2qraJSAK5EZ8EbSMXrC5BviPsBpA0HxDchDc/htB3/52HsJQ1CVuMDEhZmc3I+MoWQwcRBYOp3c8SmfsRlKSK5Z5WV8/WDVVOnRogY/p+tOumCGq0XDXQ8nMsrigd2gbYOHRR+7kLJ/7Xw5/n+rdALPP6EZlC3xiSwdXPbfP5Wxd5mybpu1fl/59ZwVAl0rWcTyQ6pyRLsb9zy6WUTH8+QCymOL6OcmeKvZV9RyM/w6kDBosUPWnMB/3t+C7MZ61+pqCWPx5sYRIGzE7E8KQyCH9OH95CDQ9kRAuixfYs5KR5InSvmnWWvgpEReEwn8pF1YxYRXb1gyWYtN077QCXAPVPQUoLtRHAyO8irawPfwC5A== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM5PR10MB1466.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(5660300002)(508600001)(6506007)(52116002)(86362001)(8936002)(66556008)(6666004)(8676002)(66946007)(6486002)(6512007)(107886003)(1076003)(2906002)(26005)(186003)(4326008)(66476007)(38100700002)(38350700002)(316002)(36756003)(83380400001)(2616005); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: NiA7uF3SHO2G/JuyeKxMo4DSeclBD9EfsjjMjvnNpdOyr1BFoQqp2Rl4mtnfdNnRGE17GTYzgsfsY+QY28DZXqT18dQhnEHCBGQr6cfRLqalx2wa4i54r1bsYGU4yH8crwjb7WdxYVY+fxWPfsnFMGf3AywnnZpr0FquELO2/qjQGIFIZSk7vYd9Iovm6Y1PIImc7KeTCIjSFtl/SuDlPMAdfJFCjTRcDQEFBkw4FZxLpwWjIKfpLlBa30D1bRS9K7gGfFqNNAcD2/vSURbl6djSJajpj1v9Z6KwbsO58shxXGSPP46F8M2x73Edmmu+QCXWrnYaI+xtC7om5jCdVxNABLzaScCrA3h77eakYB9gTIjU0ZSoegC2dan0PtDMcYSSgu/nNuyBIrF4noXIctEuwQXhbQ0WloH8HfWEaCoFvvFNU/HEtKrIaTeJN6JimAG2KHLrMame+gwHgJ25SXOSNi0eXBiugwi+/fDVEXhrgCA/Ac18Bi++OKX6sGYzDD81N6jALi69F1BAkB9MrA9lSowBpsLLXrLJpqy5T4+YjJ/KfdCoy1ml/zdFQr/rVTxIOZ+qYpG4na+srqH2288P4vBjEdvGgTKRsm9xTixMSiwvgoV+T3MBObwoKkz4mhCohjBSyqBP/LENSecc/XsADRb8mrTypHXMtVNXmROuksGMfUDPcVD1jFmumRGHqW2wLz16J44OSG9Eaz42sZG2FIiLKXY+gy7KLih1E+K1RUzeok/N70S7V17YHen8GlwoKqDnM5M6Pxx53zeUq29SUzpiuBolXMJhUnoc2LixdME7QuBngFRIHmaPSy4UL3lsXdghmVX/rB+IJbfFrK8XA+pi/MTQvLZQOxCM+y9HM5q/cmfp1P13K/rPq+vf/ddEEIN+Wc3zp6281566Fh8LgU66Cpgn2xchziG51j4ADecVAXDhX9insboShwZlwTQ7VzJkbj5ycxeFX5LwgXeGdZx33G7yiY3+8vd/8PePN39eQYBVDmfg2y7Y+q4JT3ckEx6LYv96dBBumoUFoH/06C+bcqN/qXPVjtpPj0c2f7Y6mdBQreu7lDNfZyt/N/nW/qsMqmzwQxMu1ZCCJ5zDIPPnVct3/74fXXlzs6xDjMSj/VHqwW6mHIDEV+H41ijEA9mj/A0aw3pAAdsiGH/XOl8x6Lc7a10P1M1XkRZpip1KpErxXeic5PR9LaDzl0M7nxb8MP/kvvRGAY/5OwcKX/xGC7cfzlS2VZnX7VHujPqMyfUMvX3c/BM8jEaIIvk948Llcuk39d3ajBiv8XvjwOTXrt6rs2VsvUkKYO1/oN7276WcDqznm44pBumkicQ4RMEtCgPw1w0CD+wktshgCNl+GLz31ALwQ6nbcZsuMKMaw5uRuP4+qpUNlG9jywMyIxAxdZWF66oazrhMujk0b6J2y85FxIGV32w0+OVnJO2kCTgkNFYfdPR+FgEy9FLQBd2qOOi37wGq8IwsAk2ep7Hk7AX0za4rSYFQuHsz0NOwiS7F9AJe6G/cWE82tSMDtWcPQPW1qjEpOLYXXbkkCpUppVtrLyaeCsq7KdX31qukhf78lyGwYjQ53XGuuhZsY282B5yrfbuh67EKGbMBQEwwTii9+cxqkwzV5PoY/p5u2qgNDpRcHicy0cnw8p0wQXZuE4IT6siALzfHiQ== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e7c566f2-ac91-4f74-8333-08da009c3114 X-MS-Exchange-CrossTenant-AuthSource: DM5PR10MB1466.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 00:40:06.5922 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: EzuVTGsSfLbkSVfswK2hjBxLgpxLYC8EpW8NKbjJNUOR6JAF4vqC4xDwylNB/VRfrn4wNTMZNA9hNPt4NuhgF+axVGeClMdlQJYjTZM7p14= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB3680 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10279 signatures=690470 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203080000 X-Proofpoint-GUID: 54nkahjeRx7YgM4ZI1r0kqKnaZAcoAfD X-Proofpoint-ORIG-GUID: 54nkahjeRx7YgM4ZI1r0kqKnaZAcoAfD Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org This patch alllow drivers like iscsi_tcp to transmit the task from the queuecommand. For the case where we were going to run from kblockd and the app and iscsi share CPUs, we can skip the extra workqueue runs and it can result in up to 30% increase in 4K IOPs. Signed-off-by: Mike Christie --- drivers/scsi/libiscsi.c | 52 +++++++++++++++++++++++++++++++++++------ include/scsi/libiscsi.h | 11 +++++++-- 2 files changed, 54 insertions(+), 9 deletions(-) diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c index 63e0d97df50f..5a2953260a94 100644 --- a/drivers/scsi/libiscsi.c +++ b/drivers/scsi/libiscsi.c @@ -1700,9 +1700,13 @@ static void iscsi_xmitworker(struct work_struct *work) /* * serialize Xmit worker on a per-connection basis. */ + mutex_lock(&conn->xmit_mutex); + do { rc = iscsi_data_xmit(conn); } while (rc >= 0 || rc == -EAGAIN); + + mutex_unlock(&conn->xmit_mutex); } static inline struct iscsi_task *iscsi_alloc_task(struct iscsi_conn *conn, @@ -1832,9 +1836,16 @@ int iscsi_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *sc) goto reject; } - if (!ihost->workq) { + session->queued_cmdsn++; + + if (!ihost->workq || + (ihost->xmit_from_qc && !conn->task && + list_empty(&conn->cmdqueue) && mutex_trylock(&conn->xmit_mutex))) { reason = iscsi_prep_scsi_cmd_pdu(task); if (reason) { + if (ihost->workq) + mutex_unlock(&conn->xmit_mutex); + if (reason == -ENOMEM || reason == -EACCES) { reason = FAILURE_OOM; goto prepd_reject; @@ -1843,21 +1854,27 @@ int iscsi_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *sc) goto prepd_fault; } } - if (session->tt->xmit_task(task, true)) { - session->cmdsn--; - reason = FAILURE_SESSION_NOT_READY; - goto prepd_reject; + + if (!ihost->workq) { + if (session->tt->xmit_task(task, true)) { + session->cmdsn--; + reason = FAILURE_SESSION_NOT_READY; + goto prepd_reject; + } + } else { + iscsi_xmit_task(conn, task, false, true); + mutex_unlock(&conn->xmit_mutex); } } else { list_add_tail(&task->running, &conn->cmdqueue); iscsi_conn_queue_xmit(conn); } - session->queued_cmdsn++; spin_unlock_bh(&session->frwd_lock); return 0; prepd_reject: + session->queued_cmdsn--; spin_lock_bh(&session->back_lock); iscsi_complete_task(task, ISCSI_TASK_REQUEUE_SCSIQ); spin_unlock_bh(&session->back_lock); @@ -1868,6 +1885,7 @@ int iscsi_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *sc) return SCSI_MLQUEUE_TARGET_BUSY; prepd_fault: + session->queued_cmdsn--; spin_lock_bh(&session->back_lock); iscsi_complete_task(task, ISCSI_TASK_REQUEUE_SCSIQ); spin_unlock_bh(&session->back_lock); @@ -2558,11 +2576,17 @@ int iscsi_eh_device_reset(struct scsi_cmnd *sc) iscsi_suspend_tx(conn); + /* + * If this is called from a SG ioctl the host may not be fully stopped. + * Take the xmit_mutex in case we are xmitting from iscsi_queuecommand. + */ + mutex_lock(&conn->xmit_mutex); spin_lock_bh(&session->frwd_lock); memset(hdr, 0, sizeof(*hdr)); fail_scsi_tasks(conn, sc->device->lun, DID_ERROR); session->tmf_state = TMF_INITIAL; spin_unlock_bh(&session->frwd_lock); + mutex_unlock(&conn->xmit_mutex); iscsi_start_tx(conn); goto done; @@ -2720,11 +2744,17 @@ static int iscsi_eh_target_reset(struct scsi_cmnd *sc) iscsi_suspend_tx(conn); + /* + * If this is called from a SG ioctl the host may not be fully stopped. + * Take the xmit_mutex in case we are xmitting from iscsi_queuecommand. + */ + mutex_lock(&conn->xmit_mutex); spin_lock_bh(&session->frwd_lock); memset(hdr, 0, sizeof(*hdr)); fail_scsi_tasks(conn, -1, DID_ERROR); session->tmf_state = TMF_INITIAL; spin_unlock_bh(&session->frwd_lock); + mutex_unlock(&conn->xmit_mutex); iscsi_start_tx(conn); goto done; @@ -2909,6 +2939,8 @@ struct Scsi_Host *iscsi_host_alloc(struct scsi_host_template *sht, 2, shost->host_no); if (!ihost->workq) goto free_host; + } else { + ihost->xmit_from_qc = true; } spin_lock_init(&ihost->lock); @@ -3165,6 +3197,7 @@ iscsi_conn_setup(struct iscsi_cls_session *cls_session, int dd_size, INIT_LIST_HEAD(&conn->cmdqueue); INIT_LIST_HEAD(&conn->requeue); INIT_WORK(&conn->xmitwork, iscsi_xmitworker); + mutex_init(&conn->xmit_mutex); /* allocate login_task used for the login/text sequences */ spin_lock_bh(&session->frwd_lock); @@ -3395,13 +3428,18 @@ void iscsi_conn_stop(struct iscsi_cls_conn *cls_conn, int flag) } /* - * flush queues. + * When flushing the queues we don't wait for the block to complete + * above because it can be expensive when there are lots of devices. + * To make sure there is not an xmit from iscsi_queuecommand running + * take the xmit_mutex. */ + mutex_lock(&conn->xmit_mutex); spin_lock_bh(&session->frwd_lock); fail_scsi_tasks(conn, -1, DID_TRANSPORT_DISRUPTED); fail_mgmt_tasks(session, conn); memset(&session->tmhdr, 0, sizeof(session->tmhdr)); spin_unlock_bh(&session->frwd_lock); + mutex_unlock(&conn->xmit_mutex); mutex_unlock(&session->eh_mutex); } EXPORT_SYMBOL_GPL(iscsi_conn_stop); diff --git a/include/scsi/libiscsi.h b/include/scsi/libiscsi.h index 412722f44747..1f2accf2bc1b 100644 --- a/include/scsi/libiscsi.h +++ b/include/scsi/libiscsi.h @@ -198,6 +198,12 @@ struct iscsi_conn { struct list_head cmdqueue; /* data-path cmd queue */ struct list_head requeue; /* tasks needing another run */ struct work_struct xmitwork; /* per-conn. xmit workqueue */ + /* + * This must be held while calling the xmit_task callout if it can + * called from the iscsi_q workqueue. It must be taken after the + * eh_mutex if both mutex's are needed. + */ + struct mutex xmit_mutex; /* recv */ struct work_struct recvwork; unsigned long flags; /* ISCSI_CONN_FLAGs */ @@ -267,8 +273,8 @@ struct iscsi_session { struct iscsi_cls_session *cls_session; /* * Syncs up the scsi eh thread with the iscsi eh thread when sending - * task management functions. This must be taken before the session - * and recv lock. + * task management functions. This must be taken before the conn's + * xmit_mutex. */ struct mutex eh_mutex; /* abort */ @@ -370,6 +376,7 @@ struct iscsi_host { int num_sessions; int state; + bool xmit_from_qc; struct workqueue_struct *workq; }; From patchwork Tue Mar 8 00:39:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Christie X-Patchwork-Id: 550022 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C9EFC433EF for ; Tue, 8 Mar 2022 00:40:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244220AbiCHAl2 (ORCPT ); Mon, 7 Mar 2022 19:41:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244091AbiCHAlS (ORCPT ); Mon, 7 Mar 2022 19:41:18 -0500 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A0213DDD8 for ; Mon, 7 Mar 2022 16:40:23 -0800 (PST) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 227LKdsW009310; Tue, 8 Mar 2022 00:40:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=QJY515DfdAWLYH7NeLKujgL2Merle9nsQNIHzHhMn2I=; b=GohQnRC4cs7qXpUYteXgZp0FmkF120gjZI8UmNpczoZmGyuC94DTUu3aDX+Xgv9qZTDs b/0u0VH5S2PmqpSZ+r+LED6O4jAB2S24IedMLh+p61anZVBOD8W9dn0lRjwAGN6FQc3t YUgGxBrXoFUDtf2zgdXkO1RW6Th163tEFy40AEsjwwXcp8Xdm5ofsJRoBW1gzgPITQVv 28YxnZGDBWjEfkq9XFni7ioeR0o7hSUQ0MlAAI3WEjUaUVUHW/4U8v07QE7+mYzSYHus 4Z3MFhLJt1dwPtlWjDFCVjj0QVWLxiQYytQeaJQ/D+fxccGqnZ3gLXgq2KjesAER6iQP Gw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3ekx9cdft1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 08 Mar 2022 00:40:13 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 2280alEE119127; Tue, 8 Mar 2022 00:40:12 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2175.outbound.protection.outlook.com [104.47.59.175]) by userp3020.oracle.com with ESMTP id 3em1ajmyke-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 08 Mar 2022 00:40:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X1pbeq8GoT8c8Ct5N10o6OlsZwZXGbEkoH1NjpxFAuHBbDFxRch8UOoaAAnmmFwNE+htW2dgK3+gvTQTZ1/X04WqiDNEMUfpgWaJEhYuK7IewUaZffYQLs7TIGIsaqsyGubK+ySBSNf8upKY4S7stBKSM1mvaqKVlGkJCzQYZsG3kuLfCOLIVoWHFYGsviBpFkr4yg4OJw9yLkh5Ix2Mx52lagSId+1XasqDyIMYijqROdSADs25vyWBipKQuSwdMqJGXRCGSbKSRL7aGUvDl0wnasziGy6GfA/zZverU9zuM6utIte82qAEc6de632L2Z8umz8NPLT0j23j2jkK7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QJY515DfdAWLYH7NeLKujgL2Merle9nsQNIHzHhMn2I=; b=TAYkTBFayboFdtcevqQ3IYnmxB8LM8oEgWFhtsF7MZeRG/B9LeTj3o2jIPDVm+T8dWSN0sFA4vB5NWqLtIhax5qPgTITRw+DCIDtUMEdF1uZ6bWrmccGU/7nUcbVUQmXsOppPbS7eMh97Kq1dEbj0Ws+0RGCZ7dXztM3wUWS1dbinN/IsnGY+rV7bCk9sGXPT70+nCh3tK/btr08dZgab2LQwJckbXItD+JZxmFgZXQkbAKy2dA45DjQMnye3duE6D5xGGTHasCAbcfgQdlIZ9aZJTMSs3GlQyirTNUuFVYAfCdlH9mjqV8Dkv/qhzB18/YYUTlq3pTqgCRdBHfKOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QJY515DfdAWLYH7NeLKujgL2Merle9nsQNIHzHhMn2I=; b=EtnqbI98btqXZqKeZZ3NC1hxVBqIidI8wzWlResbe6Kn9GOfIW5SGhqpwZNUKqJjkgomgmaEMLstf+khbEdxkZyGm/v4duYwZs2OcIfzpC4x7DuKdm2ZcIxCqPQJkKAuha4836WENGsq/4bluoLuNpSt1X9dTyfmbAnu7HixcDc= Received: from DM5PR10MB1466.namprd10.prod.outlook.com (10.168.103.135) by MN2PR10MB3680.namprd10.prod.outlook.com (20.179.98.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Tue, 8 Mar 2022 00:40:07 +0000 Received: from DM5PR10MB1466.namprd10.prod.outlook.com ([fe80::3dd8:6b8:e2e6:c3a2]) by DM5PR10MB1466.namprd10.prod.outlook.com ([fe80::3dd8:6b8:e2e6:c3a2%12]) with mapi id 15.20.5038.027; Tue, 8 Mar 2022 00:40:07 +0000 From: Mike Christie To: bvanassche@acm.org, lduncan@suse.com, cleech@redhat.com, ming.lei@redhat.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, james.bottomley@hansenpartnership.com Cc: Mike Christie Subject: [RFC PATCH 4/4] scsi: iscsi_tcp: Allow user to control if transmit from queuecommand Date: Mon, 7 Mar 2022 18:39:57 -0600 Message-Id: <20220308003957.123312-5-michael.christie@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308003957.123312-1-michael.christie@oracle.com> References: <20220308003957.123312-1-michael.christie@oracle.com> X-ClientProxiedBy: DM6PR03CA0020.namprd03.prod.outlook.com (2603:10b6:5:40::33) To DM5PR10MB1466.namprd10.prod.outlook.com (2603:10b6:3:b::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ad5883d6-006e-4455-e2a7-08da009c3154 X-MS-TrafficTypeDiagnostic: MN2PR10MB3680:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1IMkAqCvkxvt7DEW60EU0ld0/THfZ0XXBrp0TLOMeCLWmSL/D2zNW7M/RqRA6NR9ZgEQR4so0HvSbK+4mt9jVc9llowfNVy8+xow7dBa7p5OsRlpvyr+dt/b5m6yRI0/KkcZ8mzQlLNbehAsYcz8txQUYBAhzKbMcCPqFqtDccJv6KeIaWkTcAXYkWFeMbr8H5Uxl97thl0NMBCKp3DfKHlNLeaKu3lnnpEYS6COfL1hYGea5HnCaoYKBhGqyycZAxEt/ANbMRijERFY+VSrmoJMrjVqAFge7dKQvFWR6hrPoVQeTBPvhMXcRK9mJpV8PUX3J7miT5n5JodRFwzT1r1+bsEl8yPxETJOnrqeLoqFhye4yDgZ1vVpbrNoy2YOylFGXhb2531vnupeD87yxZAIswVrvcdo0St3x+RftltnSrvCpoT0vjIXLGkFmSqODyL2GlVc930NynM901Jv7bkyKViOPLkwe4ulLsHIkHu3lYM6o7IpQyAq37HZmA7zUq3YxEl1UJW+a/kBaaHdMvgwZG3YsRDWgCN9trAou2+O9T1AK2fPZDdQweDSrmZMuOsayTevADBg1fyU2lksAhBNz3jxFpIdbay0qQIwY8Mo+OB/lBg9WOTNgqK1MltCUgyF+U27/SB7Pzei8Dxx0g8fDwOdFarFVr6YYgbg9ZE+GonQJYM7Xh9wKW1pcUaQkEJ+tKqoXkBWbW8WwDw2ig== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM5PR10MB1466.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(5660300002)(508600001)(6506007)(52116002)(86362001)(8936002)(66556008)(6666004)(8676002)(66946007)(6486002)(6512007)(107886003)(1076003)(2906002)(26005)(186003)(4326008)(66476007)(38100700002)(38350700002)(316002)(36756003)(83380400001)(2616005); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: OnUktDZYfAKG8B0jZmw2yX4lvO3Nm7l75RGcw/6Fqr999sw4g7Wf5R0Llq7vd4emDg5hXEpiw20JoNUZmkXzPnWdb90VaXkXuVLbIe5/zzYq87MHa/GPjX86lX9Wn6cOIxe2doQ3jRG7A0/92BeaSm9ocXsRkEjVc9IJrpxp8sojzS2BkJzhS5exJ1v/B+tVI0u/sHifoYcbs646n1kuhQTsafwi0bWGjdr9JezWoiI3Ds1FSSxPZQtWj+H6gIT+j4UYb8enK+izIBe5vF1+KO/4RNNqnuSs3SpWSPo1svBzwJwP1js8w5AaS0tuzufS+65cCWsShC4qCpq82VdZypvTXTyyyC9KzDbKwRuZOeF319yUbuCDM+Gc6Z0t+FKct0sF6olnD+O1/HnHr/3DQ/q4MDlGXTZTDwdoa9F123d59I0ULZ79VfGHhuFtDNpLCnatmVzchUXpsVpMJc67m9BQNzAVg1cJJo4EzT9LCoV82FWgYmzu9y6uuDep+wyuqAGHvnHicSTGY4jX7RnI9nkEF3FTpzvr8HxmtJYMTZqDtNPcfxROhT8mY5dlFn+ZPQ8nIDJwtsUGQn+NuZWC2QNZM4joCEEeyR7vHt8T7o/FDArSukFGN4sktoN6jCIBZIjZbuNYE0emYCSJpoIxNN0pZMaKX9XJOcQnZSb9F1W9tfkj/zsAvJuglaMqw4Ebn8lYnNkPdTgwq2r8A3lvOVOH2wqVycU669Rq1G9dCRphL+ByyVT4zWfIJez/6FqMydV53YgF5eFZB+THTr989KLcopxFsFo6/bffU5bxlFz3EB8juu4jYNS2R9xoI5Dadr23Bkd9eozvjQR6/bbihjb5Th4x6sir93wMGJ8Ai8mTOgbUQq7pb92iD6TE4i3qjVYyLcSnLA78N7yLvgT1gVizt3XtqsutQc3KoWBpvXDuLFBmWJ/iSLH+3AzoMI76BQswUUJBJ492FNWskomBKwPkUW+HliWF4QL8fH1WR+yJyS9xvk6jaiPXe3l9hawqS80MZmw8/o0kyjpYR/h8mDnTRG41PyweYWe3zw9A89CdHrEIL1g2H3UGT6C7+c6mr7GqcbkDFHgeNGkwN+sdlSP3BCAgCFTtiUJDRGp723ilGAfAIYwhOURCRpS79Z8IW+zDIhc9pwdwosc6C8PZIrygGUeMdEhclAkMJMNO395cYyancpVdacShBXN3KM6Vv6mr63zHA22mgEKC8FC7LN5gB4q712DSr48UlZ/rPK7El2L63wcgu4TxzN80yQh1z/xmFBSnkhlegfsvszTRU/6aq8AeMKzidyYLxdkRFvbh849F8A/sRo81+o4qtdiCDiLhlIgigTeLAnPXX5nezjlzfM/wYgV/1+3KDhIwTqPzC5PBVoWq/+M+FiZulVL2lwf5WBXse51WMB0MhkeAdgWNOombyZBwKwfRpLU5HYEnHNqgLZKtye9c4AgwTv1acLTeddBIra+ehdmR/yTWBFQufJ/1OS6EdCG20H/9uMPmZRT2kvmg/6YIHcTJHCqPSrOqQsggqi/m0+woy7Y/6toYzzCqFJR/V0BsLQRynTVr/9IC+/81SUnNsN088DEn6SHJxLV91lbbAvcYBBs3d2uof41FjFr7GfGtYl7sgFdNNyuHRPxM0G9JHDxNO21Ay3o3Y8K2aHRvZp6UeriwHA== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ad5883d6-006e-4455-e2a7-08da009c3154 X-MS-Exchange-CrossTenant-AuthSource: DM5PR10MB1466.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 00:40:06.9985 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SV/4i3+Xkfr/Ze14I3ibNChu2PqMiU+Ql3AUbVl5ZguDSUB2BkjZqw9xt81Fwozk5o/QeIh4EPpR0fzmi+pc9r3qEa53cMlDCtF0/ip0biM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB3680 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10279 signatures=690470 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203080000 X-Proofpoint-ORIG-GUID: 1nZ41RwQDLoc5CSl_Qs7f-4JC-Se-S7N X-Proofpoint-GUID: 1nZ41RwQDLoc5CSl_Qs7f-4JC-Se-S7N Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Transmitting from the queuecommand is nice when your app and iscsi threads have to run on the same CPU. But, for the case where you want higher throughput/IOPs it's sometimes better to hog multiple CPUs with the app on one CPU and the xmit/recv paths an another. To allow both configs this adds a modparam. Signed-off-by: Mike Christie --- drivers/scsi/iscsi_tcp.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/scsi/iscsi_tcp.c b/drivers/scsi/iscsi_tcp.c index c2627505011d..c48707b53746 100644 --- a/drivers/scsi/iscsi_tcp.c +++ b/drivers/scsi/iscsi_tcp.c @@ -56,6 +56,10 @@ static bool iscsi_use_recv_wq; module_param_named(use_recv_wq, iscsi_use_recv_wq, bool, 0644); MODULE_PARM_DESC(use_recv_wq, "Set to true to read iSCSI data/headers from the iscsi_q workqueue. The default is false which will perform reads from the network softirq context."); +static bool iscsi_xmit_from_qc; +module_param_named(xmit_from_queuecommand, iscsi_xmit_from_qc, bool, 0644); +MODULE_PARM_DESC(xmit_from_queuecommand, "Set to true to try to xmit the task from the queuecommand callout. The default is false wihch will xmit the task from the iscsi_q workqueue."); + static int iscsi_sw_tcp_dbg; module_param_named(debug_iscsi_tcp, iscsi_sw_tcp_dbg, int, S_IRUGO | S_IWUSR); @@ -909,6 +913,7 @@ iscsi_sw_tcp_session_create(struct iscsi_endpoint *ep, uint16_t cmds_max, struct iscsi_cls_session *cls_session; struct iscsi_session *session; struct iscsi_sw_tcp_host *tcp_sw_host; + struct iscsi_host *ihost; struct Scsi_Host *shost; int rc; @@ -928,6 +933,12 @@ iscsi_sw_tcp_session_create(struct iscsi_endpoint *ep, uint16_t cmds_max, shost->max_channel = 0; shost->max_cmd_len = SCSI_MAX_VARLEN_CDB_SIZE; + if (iscsi_xmit_from_qc) { + shost->hostt->queuecommand_blocks = true; + ihost = shost_priv(shost); + ihost->xmit_from_qc = true; + } + rc = iscsi_host_get_max_scsi_cmds(shost, cmds_max); if (rc < 0) goto free_host;