From patchwork Fri Jul 8 19:50:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sreekanth Reddy X-Patchwork-Id: 588846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32BDBC433EF for ; Fri, 8 Jul 2022 19:38:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239931AbiGHTiX (ORCPT ); Fri, 8 Jul 2022 15:38:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239170AbiGHTiV (ORCPT ); Fri, 8 Jul 2022 15:38:21 -0400 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0592771BC3 for ; Fri, 8 Jul 2022 12:38:20 -0700 (PDT) Received: by mail-pl1-x643.google.com with SMTP id d5so17093006plo.12 for ; Fri, 08 Jul 2022 12:38:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=cU3zz7z+eNjk70WbOm9wJUOb55JtJPLPaHY61KfX8VY=; b=gKjVP82Ypx+48CCeARs3L8+ql+DP30Txa6xvE9PkczXOhNSr8SPvsxvosviQle/cKU hIYWdcXK+6OqTo2xXaNp+MN97sVP+WOjwdLzLDh7yEs6W6SDJWzN6mesyM9DyTGfWQL3 WbMphzqaezuBHN8pEfr00xJhkYwr10aTIDfaM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=cU3zz7z+eNjk70WbOm9wJUOb55JtJPLPaHY61KfX8VY=; b=Wp01khfFGGoJfjKGIIbnT1IJpUndTDU5koVdNKCjZY3DSuZBUg4kf8lo3GcSHe3AX2 /Y7+ffOOK8nsJKN7eWLRbRewM81IyinNxz7KT5Ke4d2eXudYrDBLF0pQBFbJkONRtnd9 7uEZqq7Fc6vaJYea1KaTOAbAeIN6eEuV/ERTLWdpYuQlrN87ux+xSjWRw0moCaiMRL0h SPxx650oAyZk6G0fc2gcpSEHLXwyufis4jcfNWjzG+pojoZankZ6+0H2k6mOLy1qsSNb 6HEq3P7+6u8pBfokccY7J8R7KRzaC+s0FO34mAH/Jk8I2FW9O1dPtaaIq6LaWhJ33sNx 9rVg== X-Gm-Message-State: AJIora/CmLu8OKD0YEghXGNvLvdrdNYsEQ1vKRAPBhL5Mli7+JRK2I1V CV1veNXqSbceu0XdZbf6SfjIpDR5tKYkYbhG3CDZc+TzZcAqSW/EoGorqv+kKSJXXHdIz9h8eos ELKQ9+sebwa7Idpfqh72HPMGM4FEO58jKKEX7IeznRSGAj8JUVumQJPA4Du5Ezzb9WSAgEBeu24 gPQGdmyXh9qyY= X-Google-Smtp-Source: AGRyM1swx96NlPiDV9B1orKZ4/ef2APGhqYCwSKrT6eTiRXqf5DZMVbWyqCrE8RuLQs5zfs5d5RJGw== X-Received: by 2002:a17:902:d50e:b0:16c:1664:81e5 with SMTP id b14-20020a170902d50e00b0016c166481e5mr5329300plg.149.1657309098958; Fri, 08 Jul 2022 12:38:18 -0700 (PDT) Received: from dhcp-10-123-20-36.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id i16-20020a170902c95000b0015e8d4eb20dsm14256484pla.87.2022.07.08.12.38.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Jul 2022 12:38:18 -0700 (PDT) From: Sreekanth Reddy To: linux-scsi@vger.kernel.org Cc: martin.petersen@oracle.com, Sreekanth Reddy Subject: [PATCH 2/2] mpi3mr: Reduce VD queue depth on detecting the throttling Date: Sat, 9 Jul 2022 01:20:20 +0530 Message-Id: <20220708195020.8323-3-sreekanth.reddy@broadcom.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220708195020.8323-1-sreekanth.reddy@broadcom.com> References: <20220708195020.8323-1-sreekanth.reddy@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Reduce the VD queue depth on detecting the throttling condition. Signed-off-by: Sreekanth Reddy --- drivers/scsi/mpi3mr/mpi3mr.h | 10 +++ drivers/scsi/mpi3mr/mpi3mr_fw.c | 4 ++ drivers/scsi/mpi3mr/mpi3mr_os.c | 123 ++++++++++++++++++++++++++++++++ 3 files changed, 137 insertions(+) diff --git a/drivers/scsi/mpi3mr/mpi3mr.h b/drivers/scsi/mpi3mr/mpi3mr.h index 6bb3311..0935b2e 100644 --- a/drivers/scsi/mpi3mr/mpi3mr.h +++ b/drivers/scsi/mpi3mr/mpi3mr.h @@ -436,6 +436,10 @@ struct mpi3mr_intr_info { * struct mpi3mr_throttle_group_info - Throttle group info * * @io_divert: Flag indicates io divert is on or off for the TG + * @need_qd_reduction: Flag to indicate QD reduction is needed + * @qd_reduction: Queue Depth reduction in units of 10% + * @fw_qd: QueueDepth value reported by the firmware + * @modified_qd: Modified QueueDepth value due to throttling * @id: Throttle Group ID. * @high: High limit to turn on throttling in 512 byte blocks * @low: Low limit to turn off throttling in 512 byte blocks @@ -443,6 +447,10 @@ struct mpi3mr_intr_info { */ struct mpi3mr_throttle_group_info { u8 io_divert; + u8 need_qd_reduction; + u8 qd_reduction; + u16 fw_qd; + u16 modified_qd; u16 id; u32 high; u32 low; @@ -486,6 +494,7 @@ struct tgt_dev_pcie { * cached from firmware given data * * @state: State of the VD + * @tg_qd_reduction: Queue Depth reduction in units of 10% * @tg_id: VDs throttle group ID * @high: High limit to turn on throttling in 512 byte blocks * @low: Low limit to turn off throttling in 512 byte blocks @@ -493,6 +502,7 @@ struct tgt_dev_pcie { */ struct tgt_dev_vd { u8 state; + u8 tg_qd_reduction; u16 tg_id; u32 tg_high; u32 tg_low; diff --git a/drivers/scsi/mpi3mr/mpi3mr_fw.c b/drivers/scsi/mpi3mr/mpi3mr_fw.c index ab79374..6e39f79 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_fw.c +++ b/drivers/scsi/mpi3mr/mpi3mr_fw.c @@ -4077,9 +4077,13 @@ void mpi3mr_memset_buffers(struct mpi3mr_ioc *mrioc) tg = mrioc->throttle_groups; for (i = 0; i < mrioc->num_io_throttle_group; i++, tg++) { tg->id = 0; + tg->fw_qd = 0; + tg->modified_qd = 0; tg->io_divert = 0; + tg->need_qd_reduction = 0; tg->high = 0; tg->low = 0; + tg->qd_reduction = 0; atomic_set(&tg->pend_large_data_sz, 0); } } diff --git a/drivers/scsi/mpi3mr/mpi3mr_os.c b/drivers/scsi/mpi3mr/mpi3mr_os.c index e1ccb5f..6e36229 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_os.c +++ b/drivers/scsi/mpi3mr/mpi3mr_os.c @@ -38,6 +38,8 @@ MODULE_PARM_DESC(logging_level, static void mpi3mr_send_event_ack(struct mpi3mr_ioc *mrioc, u8 event, struct mpi3mr_drv_cmd *cmdparam, u32 event_ctx); +#define MPI3MR_DRIVER_EVENT_TG_QD_REDUCTION (0xFFFF) + /** * mpi3mr_host_tag_for_scmd - Get host tag for a scmd * @mrioc: Adapter instance reference @@ -354,6 +356,50 @@ void mpi3mr_cleanup_fwevt_list(struct mpi3mr_ioc *mrioc) } } +/** + * mpi3mr_queue_qd_reduction_event -Queue TG QD reduction event + * @mrioc: Adapter instance reference + * @tg: Throttle group information pointer + * + * Accessor to queue on synthetically generated driver event to + * the event worker thread, the driver event will be used to + * reduce the QD of all VDs in the TG from the worker thread. + * + * Return: None. + */ +static void mpi3mr_queue_qd_reduction_event(struct mpi3mr_ioc *mrioc, + struct mpi3mr_throttle_group_info *tg) +{ + struct mpi3mr_fwevt *fwevt; + u16 sz = sizeof(struct mpi3mr_throttle_group_info *); + + /* + * If the QD reduction event is already queued due to throttle and if + * the QD is not restored through device info change event + * then dont queue further reduction events + */ + if (tg->fw_qd != tg->modified_qd) + return; + + fwevt = mpi3mr_alloc_fwevt(sz); + if (!fwevt) { + ioc_warn(mrioc, "failed to queue TG QD reduction event\n"); + return; + } + *(__le64 *)fwevt->event_data = (__le64)tg; + fwevt->mrioc = mrioc; + fwevt->event_id = MPI3MR_DRIVER_EVENT_TG_QD_REDUCTION; + fwevt->send_ack = 0; + fwevt->process_evt = 1; + fwevt->evt_ctx = 0; + fwevt->event_data_size = sz; + tg->modified_qd = max_t(u16, (tg->fw_qd * tg->qd_reduction) / 10, 8); + + dprint_event_bh(mrioc, "qd reduction event queued for tg_id(%d)\n", + tg->id); + mpi3mr_fwevt_add_to_list(mrioc, fwevt); +} + /** * mpi3mr_invalidate_devhandles -Invalidate device handles * @mrioc: Adapter instance reference @@ -880,6 +926,7 @@ static int mpi3mr_change_queue_depth(struct scsi_device *sdev, else if (!q_depth) q_depth = MPI3MR_DEFAULT_SDEV_QD; retval = scsi_change_queue_depth(sdev, q_depth); + sdev->max_queue_depth = sdev->queue_depth; return retval; } @@ -1100,6 +1147,11 @@ static void mpi3mr_update_tgtdev(struct mpi3mr_ioc *mrioc, tg->id = vdinf_io_throttle_group; tg->high = tgtdev->dev_spec.vd_inf.tg_high; tg->low = tgtdev->dev_spec.vd_inf.tg_low; + tg->qd_reduction = + tgtdev->dev_spec.vd_inf.tg_qd_reduction; + if (is_added == true) + tg->fw_qd = tgtdev->q_depth; + tg->modified_qd = tgtdev->q_depth; } tgtdev->dev_spec.vd_inf.tg = tg; if (scsi_tgt_priv_data) @@ -1493,6 +1545,60 @@ static void mpi3mr_logdata_evt_bh(struct mpi3mr_ioc *mrioc, fwevt->event_data_size); } +/** + * mpi3mr_update_sdev_qd - Update SCSI device queue depath + * @sdev: SCSI device reference + * @data: Queue depth reference + * + * This is an iterator function called for each SCSI device in a + * target to update the QD of each SCSI device. + * + * Return: Nothing. + */ +static void mpi3mr_update_sdev_qd(struct scsi_device *sdev, void *data) +{ + u16 *q_depth = (u16 *)data; + + scsi_change_queue_depth(sdev, (int)*q_depth); + sdev->max_queue_depth = sdev->queue_depth; +} + +/** + * mpi3mr_set_qd_for_all_vd_in_tg -set QD for TG VDs + * @mrioc: Adapter instance reference + * @tg: Throttle group information pointer + * + * Accessor to reduce QD for each device associated with the + * given throttle group. + * + * Return: None. + */ +static void mpi3mr_set_qd_for_all_vd_in_tg(struct mpi3mr_ioc *mrioc, + struct mpi3mr_throttle_group_info *tg) +{ + unsigned long flags; + struct mpi3mr_tgt_dev *tgtdev; + struct mpi3mr_stgt_priv_data *tgt_priv; + + + spin_lock_irqsave(&mrioc->tgtdev_lock, flags); + list_for_each_entry(tgtdev, &mrioc->tgtdev_list, list) { + if (tgtdev->starget && tgtdev->starget->hostdata) { + tgt_priv = tgtdev->starget->hostdata; + if (tgt_priv->throttle_group == tg) { + dprint_event_bh(mrioc, + "updating qd due to throttling for persist_id(%d) original_qd(%d), reduced_qd (%d)\n", + tgt_priv->perst_id, tgtdev->q_depth, + tg->modified_qd); + starget_for_each_device(tgtdev->starget, + (void *)&tg->modified_qd, + mpi3mr_update_sdev_qd); + } + } + } + spin_unlock_irqrestore(&mrioc->tgtdev_lock, flags); +} + /** * mpi3mr_fwevt_bh - Firmware event bottomhalf handler * @mrioc: Adapter instance reference @@ -1550,6 +1656,21 @@ static void mpi3mr_fwevt_bh(struct mpi3mr_ioc *mrioc, mpi3mr_logdata_evt_bh(mrioc, fwevt); break; } + case MPI3MR_DRIVER_EVENT_TG_QD_REDUCTION: + { + struct mpi3mr_throttle_group_info *tg; + + tg = (struct mpi3mr_throttle_group_info *) + (*(__le64 *)fwevt->event_data); + dprint_event_bh(mrioc, + "qd reduction event processed for tg_id(%d) reduction_needed(%d)\n", + tg->id, tg->need_qd_reduction); + if (tg->need_qd_reduction) { + mpi3mr_set_qd_for_all_vd_in_tg(mrioc, tg); + tg->need_qd_reduction = 0; + } + break; + } default: break; } @@ -4234,8 +4355,10 @@ static int mpi3mr_qcmd(struct Scsi_Host *shost, mrioc->io_throttle_high) || (tg_pend_data_len >= tg->high))) { tg->io_divert = 1; + tg->need_qd_reduction = 1; mpi3mr_set_io_divert_for_all_vd_in_tg(mrioc, tg, 1); + mpi3mr_queue_qd_reduction_event(mrioc, tg); } } else { ioc_pend_data_len = atomic_add_return(data_len_blks,