From patchwork Fri Jun 5 02:12:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daejun Park X-Patchwork-Id: 213818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00AB7C433DF for ; Fri, 5 Jun 2020 02:20:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C5CB5207ED for ; Fri, 5 Jun 2020 02:20:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="QM7J95yi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726123AbgFECUI (ORCPT ); Thu, 4 Jun 2020 22:20:08 -0400 Received: from mailout1.samsung.com ([203.254.224.24]:43329 "EHLO mailout1.samsung.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726021AbgFECUH (ORCPT ); Thu, 4 Jun 2020 22:20:07 -0400 Received: from epcas1p2.samsung.com (unknown [182.195.41.46]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20200605022003epoutp01d6dd140ed8270d74cd0f8b4dd43dcff4~VhGL1DzEQ1378313783epoutp012 for ; Fri, 5 Jun 2020 02:20:03 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20200605022003epoutp01d6dd140ed8270d74cd0f8b4dd43dcff4~VhGL1DzEQ1378313783epoutp012 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1591323603; bh=HqJ9XLtjvCGgP5la8izwY+izOX7HRKP7K+U8sm7hxpA=; h=Subject:Reply-To:From:To:CC:In-Reply-To:Date:References:From; b=QM7J95yiUtC/9OK9lOopbNOlPnQDId06Dw8lDRtVOAX15IqF5GurDU5RGpGytPuNW srV04bkw7/8EMcgFaf/tL/tU6rVf6X6zsrj+ocrxymgUk4CsEc1q7wHjR86Dfojo7T FRarcbEV3Ivun8Tta/apWHXQilZHXJOD7ugN21uk= Received: from epcpadp1 (unknown [182.195.40.11]) by epcas1p4.samsung.com (KnoxPortal) with ESMTP id 20200605022003epcas1p4b3dbe280d30a997ebc5a051e4ba70f10~VhGLK5ZTw1786717867epcas1p4W; Fri, 5 Jun 2020 02:20:03 +0000 (GMT) Mime-Version: 1.0 Subject: [RFC PATCH 5/5] scsi: ufs: Prepare HPB read for cached sub-region Reply-To: daejun7.park@samsung.com From: Daejun Park To: Daejun Park , ALIM AKHTAR , "avri.altman@wdc.com" , "jejb@linux.ibm.com" , "martin.petersen@oracle.com" , "asutoshd@codeaurora.org" , "beanhuo@micron.com" , "stanley.chu@mediatek.com" , "cang@codeaurora.org" , "bvanassche@acm.org" , "tomas.winkler@intel.com" CC: "linux-scsi@vger.kernel.org" , "linux-kernel@vger.kernel.org" , Sang-yoon Oh , Sung-Jun Park , yongmyung lee , Jinyoung CHOI , Adel Choi , BoRam Shin X-Priority: 3 X-Content-Kind-Code: NORMAL In-Reply-To: <963815509.21591323002276.JavaMail.epsvc@epcpadp1> X-CPGS-Detection: blocking_info_exchange X-Drm-Type: N,general X-Msg-Generator: Mail X-Msg-Type: PERSONAL X-Reply-Demand: N Message-ID: <336371513.41591323603173.JavaMail.epsvc@epcpadp1> Date: Fri, 05 Jun 2020 11:12:16 +0900 X-CMS-MailID: 20200605021216epcms2p2034fed78fd0e5d15083066ef5e99ce21 X-Sendblock-Type: AUTO_CONFIDENTIAL X-CPGSPASS: Y X-CPGSPASS: Y X-Hop-Count: 3 X-CMS-RootMailID: 20200605011604epcms2p8bec8ef6682583d7248dc7d9dc1bfc882 References: <963815509.21591323002276.JavaMail.epsvc@epcpadp1> <231786897.01591322101492.JavaMail.epsvc@epcpadp1> <336371513.41591320902369.JavaMail.epsvc@epcpadp1> <963815509.21591320301642.JavaMail.epsvc@epcpadp1> <231786897.01591320001492.JavaMail.epsvc@epcpadp1> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org This patch changes the read I/O to the HPB read I/O. If the logical address of the read I/O belongs to active sub-region, the HPB driver modifies the read I/O command to HPB read. It modifies the upiu command of UFS instead of modifying the existing SCSI command. In the HPB version 1.0, the maximum read I/O size that can be converted to HPB read is 4KB. The dirty map of the active sub-region prevents an incorrect HPB read that has stale physical page number which is updated by previous write I/O. Signed-off-by: Daejun Park --- drivers/scsi/ufs/ufshpb.c | 249 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 249 insertions(+) diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c index f1aa8e7b5ce0..b3e488ef8675 100644 --- a/drivers/scsi/ufs/ufshpb.c +++ b/drivers/scsi/ufs/ufshpb.c @@ -46,6 +46,35 @@ static struct ufshpb_driver ufshpb_drv; static int ufshpb_create_sysfs(struct ufs_hba *hba, struct ufshpb_lu *hpb); +static inline int ufshpb_is_valid_srgn(struct ufshpb_region *rgn, + struct ufshpb_subregion *srgn) +{ + return rgn->rgn_state != HPB_RGN_INACTIVE && + srgn->srgn_state == HPB_SRGN_CLEAN; +} + +static inline bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd) +{ + if (cmd->cmnd[0] == READ_10 || cmd->cmnd[0] == READ_16) + return true; + + return false; +} + +static inline bool ufshpb_is_write_discard_cmd(struct scsi_cmnd *cmd) +{ + if (cmd->cmnd[0] == WRITE_10 || cmd->cmnd[0] == WRITE_16 || + cmd->cmnd[0] == UNMAP) + return true; + + return false; +} + +static inline bool ufshpb_is_support_chunk(int transfer_len) +{ + return transfer_len <= HPB_MULTI_CHUNK_HIGH; +} + static inline bool ufshpb_is_general_lun(int lun) { return lun < UFS_UPIU_MAX_UNIT_NUM_ID; @@ -137,6 +166,225 @@ static inline void ufshpb_lu_put(struct ufshpb_lu *hpb) put_device(&hpb->hpb_lu_dev); } +static inline u32 ufshpb_get_lpn(struct scsi_cmnd *cmnd) +{ + return blk_rq_pos(cmnd->request) >> + (ilog2(cmnd->device->sector_size) - 9); +} + +static inline unsigned int ufshpb_get_len(struct scsi_cmnd *cmnd) +{ + return blk_rq_sectors(cmnd->request) >> + (ilog2(cmnd->device->sector_size) - 9); +} + +static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx, + int srgn_idx, int srgn_offset, int cnt) +{ + struct ufshpb_region *rgn; + struct ufshpb_subregion *srgn; + int set_bit_len; + int bitmap_len = hpb->entries_per_srgn; + +next_srgn: + rgn = hpb->rgn_tbl + rgn_idx; + srgn = rgn->srgn_tbl + srgn_idx; + + if ((srgn_offset + cnt) > bitmap_len) + set_bit_len = bitmap_len - srgn_offset; + else + set_bit_len = cnt; + + if (rgn->rgn_state != HPB_RGN_INACTIVE) + bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len); + + srgn_offset = 0; + if (++srgn_idx == hpb->srgns_per_rgn) { + srgn_idx = 0; + rgn_idx++; + } + + cnt -= set_bit_len; + if (cnt > 0) + goto next_srgn; + + WARN_ON(cnt < 0); +} + +static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx, + int srgn_idx, int srgn_offset, int cnt) +{ + struct ufshpb_region *rgn; + struct ufshpb_subregion *srgn; + int bitmap_len = hpb->entries_per_srgn; + int i, bit_len; + +next_srgn: + rgn = hpb->rgn_tbl + rgn_idx; + srgn = rgn->srgn_tbl + srgn_idx; + + if (!ufshpb_is_valid_srgn(rgn, srgn)) + return true; + + /* + * If the region state is active, mctx must be allocated. + * In this case, check whether the region is evicted or + * mctx allcation fail. + */ + WARN_ON(!srgn->mctx); + + if ((srgn_offset + cnt) > bitmap_len) + bit_len = bitmap_len - srgn_offset; + else + bit_len = cnt; + + for (i = 0; i < bit_len; i++) { + if (test_bit(srgn_offset + i, srgn->mctx->ppn_dirty)) + return true; + } + + srgn_offset = 0; + if (++srgn_idx == hpb->srgns_per_rgn) { + srgn_idx = 0; + rgn_idx++; + } + + cnt -= bit_len; + if (cnt > 0) + goto next_srgn; + + return false; +} + +static u64 ufshpb_get_ppn(struct ufshpb_lu *hpb, + struct ufshpb_map_ctx *mctx, int pos, int *error) +{ + u64 *ppn_table; + struct page *page; + int index, offset; + + index = pos / (PAGE_SIZE / HPB_ENTRY_SIZE); + offset = pos % (PAGE_SIZE / HPB_ENTRY_SIZE); + + page = mctx->m_page[index]; + if (unlikely(!page)) { + *error = -ENOMEM; + dev_err(&hpb->hpb_lu_dev, + "error. cannot find page in mctx\n"); + return 0; + } + + ppn_table = page_address(page); + if (unlikely(!ppn_table)) { + *error = -ENOMEM; + dev_err(&hpb->hpb_lu_dev, "error. cannot get ppn_table\n"); + return 0; + } + + return ppn_table[offset]; +} + +static inline void +ufshpb_get_pos_from_lpn(struct ufshpb_lu *hpb, unsigned long lpn, int *rgn_idx, + int *srgn_idx, int *offset) +{ + int rgn_offset; + + *rgn_idx = lpn >> hpb->entries_per_rgn_shift; + rgn_offset = lpn & hpb->entries_per_rgn_mask; + *srgn_idx = rgn_offset >> hpb->entries_per_srgn_shift; + *offset = rgn_offset & hpb->entries_per_srgn_mask; +} + +static void +ufshpb_set_hpb_read_to_upiu(struct ufshpb_lu *hpb, struct ufshcd_lrb *lrbp, + u32 lpn, u64 ppn, unsigned int transfer_len) +{ + unsigned char *cdb = lrbp->ucd_req_ptr->sc.cdb; + + cdb[0] = UFSHPB_READ; + + put_unaligned_be32(lpn, &cdb[2]); + put_unaligned_be64(ppn, &cdb[6]); + cdb[14] = transfer_len; +} + +/* routine : READ10 -> HPB_READ */ +static void ufshpb_prep_fn(struct ufs_hba *hba, struct ufshcd_lrb *lrbp) +{ + struct ufshpb_lu *hpb; + struct ufshpb_region *rgn; + struct ufshpb_subregion *srgn; + struct scsi_cmnd *cmd = lrbp->cmd; + u32 lpn; + u64 ppn; + unsigned long flags; + int transfer_len, rgn_idx, srgn_idx, srgn_offset; + int err = 0; + + hpb = ufshpb_get_hpb_data(cmd); + err = ufshpb_lu_get(hpb); + if (unlikely(err)) + return; + + WARN_ON(hpb->lun != cmd->device->lun); + if (!ufshpb_is_write_discard_cmd(cmd) && + !ufshpb_is_read_cmd(cmd)) + goto put_hpb; + + transfer_len = ufshpb_get_len(cmd); + if (unlikely(!transfer_len)) + goto put_hpb; + + lpn = ufshpb_get_lpn(cmd); + ufshpb_get_pos_from_lpn(hpb, lpn, &rgn_idx, &srgn_idx, &srgn_offset); + rgn = hpb->rgn_tbl + rgn_idx; + srgn = rgn->srgn_tbl + srgn_idx; + + /* If commnad type is WRITE and DISCARD, set bitmap as drity */ + if (ufshpb_is_write_discard_cmd(cmd)) { + spin_lock_irqsave(&hpb->hpb_state_lock, flags); + ufshpb_set_ppn_dirty(hpb, rgn_idx, srgn_idx, srgn_offset, + transfer_len); + spin_unlock_irqrestore(&hpb->hpb_state_lock, flags); + goto put_hpb; + } + + WARN_ON(!ufshpb_is_read_cmd(cmd)); + + if (!ufshpb_is_support_chunk(transfer_len)) + goto put_hpb; + + spin_lock_irqsave(&hpb->hpb_state_lock, flags); + if (ufshpb_test_ppn_dirty(hpb, rgn_idx, srgn_idx, srgn_offset, + transfer_len)) { + atomic_inc(&hpb->stats.miss_cnt); + spin_unlock_irqrestore(&hpb->hpb_state_lock, flags); + goto put_hpb; + } + + ppn = ufshpb_get_ppn(hpb, srgn->mctx, srgn_offset, &err); + spin_unlock_irqrestore(&hpb->hpb_state_lock, flags); + if (unlikely(err)) { + /* + * In this case, the region state is active, + * but the ppn table is not allocated. + * Make sure that ppn tabie must be allocated on + * active state + */ + WARN_ON(true); + dev_err(&hpb->hpb_lu_dev, + "ufshpb_get_ppn failed. err %d\n", err); + goto put_hpb; + } + + ufshpb_set_hpb_read_to_upiu(hpb, lrbp, lpn, ppn, transfer_len); + + atomic_inc(&hpb->stats.hit_cnt); +put_hpb: + ufshpb_lu_put(hpb); +} + static struct ufshpb_req *ufshpb_get_map_req(struct ufshpb_lu *hpb, struct ufshpb_subregion *srgn) { @@ -1688,6 +1936,7 @@ static struct ufshpb_driver ufshpb_drv = { .bus = &ufsf_bus_type, }, .ufshpb_ops = { + .prep_fn = ufshpb_prep_fn, .reset = ufshpb_reset, .reset_host = ufshpb_reset_host, .suspend = ufshpb_suspend,