diff mbox series

wifi: ath11k: enable 36 bit mask for stream DMA

Message ID 20240123015201.28939-1-quic_bqiang@quicinc.com
State New
Headers show
Series wifi: ath11k: enable 36 bit mask for stream DMA | expand

Commit Message

Baochen Qiang Jan. 23, 2024, 1:52 a.m. UTC
Currently 32 bit DMA mask is used, telling kernel to get us an DMA
address under 4GB when mapping a buffer. This results in a very high
CPU overhead in the case where IOMMU is disabled and more than 4GB
system memory is installed. The reason is, with more than 4GB memory
installed, kernel is likely to allocate a buffer whose physical
address is above 4GB. While with IOMMU disabled, kernel has to involve
SWIOTLB to map/unmap that buffer, which consumes lots of CPU cycles.

We did hit an issue caused by the reason mentioned above: in a system
that disables IOMMU and gets 8GB memory installed, a total of 40.5%
CPU usage is observed in throughput test. CPU profiling shows nearly
60% of CPU cycles are consumed by SWIOTLB.

By enabling 36 bit DMA mask, we can bypass SWIOTLB for any buffer
whose physical address is below 64GB. There are two types of DMA mask
within struct device, named dma_mask and coherent_dma_mask. Here we
only enable 36 bit for dma_mask, because firmware crashes if
coherent_dma_mask is also enabled, due to some unknown hardware
limitations. This is acceptable because coherent_dma_mask is used for
mapping a consistent DMA buffer, which generally does not happen in
a hot path.

With this change, the total CPU usage mentioned in above issue drops
to 18.9%.

Tested-on: WCN6855 hw2.1 PCI WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.23
Tested-on: QCN9074 hw1.0 PCI WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1
Tested-on: IPQ8074 hw2.0 AHB WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1

Signed-off-by: Baochen Qiang <quic_bqiang@quicinc.com>
---
 drivers/net/wireless/ath/ath11k/mhi.c |  2 +-
 drivers/net/wireless/ath/ath11k/pci.c | 16 +++++++++++++---
 drivers/net/wireless/ath/ath11k/pci.h |  1 +
 3 files changed, 15 insertions(+), 4 deletions(-)


base-commit: 8ff464a183f92836d7fd99edceef50a89d8ea797

Comments

Jeff Johnson Jan. 24, 2024, 2:35 a.m. UTC | #1
On 1/22/2024 5:52 PM, Baochen Qiang wrote:
> Currently 32 bit DMA mask is used, telling kernel to get us an DMA
> address under 4GB when mapping a buffer. This results in a very high
> CPU overhead in the case where IOMMU is disabled and more than 4GB
> system memory is installed. The reason is, with more than 4GB memory
> installed, kernel is likely to allocate a buffer whose physical
> address is above 4GB. While with IOMMU disabled, kernel has to involve
> SWIOTLB to map/unmap that buffer, which consumes lots of CPU cycles.
> 
> We did hit an issue caused by the reason mentioned above: in a system
> that disables IOMMU and gets 8GB memory installed, a total of 40.5%
> CPU usage is observed in throughput test. CPU profiling shows nearly
> 60% of CPU cycles are consumed by SWIOTLB.
> 
> By enabling 36 bit DMA mask, we can bypass SWIOTLB for any buffer
> whose physical address is below 64GB. There are two types of DMA mask
> within struct device, named dma_mask and coherent_dma_mask. Here we
> only enable 36 bit for dma_mask, because firmware crashes if
> coherent_dma_mask is also enabled, due to some unknown hardware
> limitations. This is acceptable because coherent_dma_mask is used for
> mapping a consistent DMA buffer, which generally does not happen in
> a hot path.
> 
> With this change, the total CPU usage mentioned in above issue drops
> to 18.9%.
> 
> Tested-on: WCN6855 hw2.1 PCI WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.23
> Tested-on: QCN9074 hw1.0 PCI WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1
> Tested-on: IPQ8074 hw2.0 AHB WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1
> 
> Signed-off-by: Baochen Qiang <quic_bqiang@quicinc.com>
> ---
>  drivers/net/wireless/ath/ath11k/mhi.c |  2 +-
>  drivers/net/wireless/ath/ath11k/pci.c | 16 +++++++++++++---
>  drivers/net/wireless/ath/ath11k/pci.h |  1 +
>  3 files changed, 15 insertions(+), 4 deletions(-)
My Qualcomm Innovation Center copyright checker reports:
drivers/net/wireless/ath/ath11k/mhi.c copyright missing 2024
drivers/net/wireless/ath/ath11k/pci.c copyright missing 2024
drivers/net/wireless/ath/ath11k/pci.h copyright missing 2024

Kalle can fix this in the pending tree, actual code LGTM
Acked-by: Jeff Johnson <quic_jjohnson@quicinc.com>
Kalle Valo Jan. 25, 2024, 4:37 p.m. UTC | #2
Jeff Johnson <quic_jjohnson@quicinc.com> writes:

> On 1/22/2024 5:52 PM, Baochen Qiang wrote:
>
>> Currently 32 bit DMA mask is used, telling kernel to get us an DMA
>> address under 4GB when mapping a buffer. This results in a very high
>> CPU overhead in the case where IOMMU is disabled and more than 4GB
>> system memory is installed. The reason is, with more than 4GB memory
>> installed, kernel is likely to allocate a buffer whose physical
>> address is above 4GB. While with IOMMU disabled, kernel has to involve
>> SWIOTLB to map/unmap that buffer, which consumes lots of CPU cycles.
>> 
>> We did hit an issue caused by the reason mentioned above: in a system
>> that disables IOMMU and gets 8GB memory installed, a total of 40.5%
>> CPU usage is observed in throughput test. CPU profiling shows nearly
>> 60% of CPU cycles are consumed by SWIOTLB.
>> 
>> By enabling 36 bit DMA mask, we can bypass SWIOTLB for any buffer
>> whose physical address is below 64GB. There are two types of DMA mask
>> within struct device, named dma_mask and coherent_dma_mask. Here we
>> only enable 36 bit for dma_mask, because firmware crashes if
>> coherent_dma_mask is also enabled, due to some unknown hardware
>> limitations. This is acceptable because coherent_dma_mask is used for
>> mapping a consistent DMA buffer, which generally does not happen in
>> a hot path.
>> 
>> With this change, the total CPU usage mentioned in above issue drops
>> to 18.9%.
>> 
>> Tested-on: WCN6855 hw2.1 PCI
>> WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.23
>> Tested-on: QCN9074 hw1.0 PCI WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1
>> Tested-on: IPQ8074 hw2.0 AHB WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1
>> 
>> Signed-off-by: Baochen Qiang <quic_bqiang@quicinc.com>
>> ---
>>  drivers/net/wireless/ath/ath11k/mhi.c |  2 +-
>>  drivers/net/wireless/ath/ath11k/pci.c | 16 +++++++++++++---
>>  drivers/net/wireless/ath/ath11k/pci.h |  1 +
>>  3 files changed, 15 insertions(+), 4 deletions(-)

> My Qualcomm Innovation Center copyright checker reports:
>
> drivers/net/wireless/ath/ath11k/mhi.c copyright missing 2024
> drivers/net/wireless/ath/ath11k/pci.c copyright missing 2024
> drivers/net/wireless/ath/ath11k/pci.h copyright missing 2024
>
> Kalle can fix this in the pending tree

Yup, added those.
Kalle Valo Jan. 25, 2024, 4:38 p.m. UTC | #3
Baochen Qiang <quic_bqiang@quicinc.com> wrote:

> Currently 32 bit DMA mask is used, telling kernel to get us an DMA
> address under 4GB when mapping a buffer. This results in a very high
> CPU overhead in the case where IOMMU is disabled and more than 4GB
> system memory is installed. The reason is, with more than 4GB memory
> installed, kernel is likely to allocate a buffer whose physical
> address is above 4GB. While with IOMMU disabled, kernel has to involve
> SWIOTLB to map/unmap that buffer, which consumes lots of CPU cycles.
> 
> We did hit an issue caused by the reason mentioned above: in a system
> that disables IOMMU and gets 8GB memory installed, a total of 40.5%
> CPU usage is observed in throughput test. CPU profiling shows nearly
> 60% of CPU cycles are consumed by SWIOTLB.
> 
> By enabling 36 bit DMA mask, we can bypass SWIOTLB for any buffer
> whose physical address is below 64GB. There are two types of DMA mask
> within struct device, named dma_mask and coherent_dma_mask. Here we
> only enable 36 bit for dma_mask, because firmware crashes if
> coherent_dma_mask is also enabled, due to some unknown hardware
> limitations. This is acceptable because coherent_dma_mask is used for
> mapping a consistent DMA buffer, which generally does not happen in
> a hot path.
> 
> With this change, the total CPU usage mentioned in above issue drops
> to 18.9%.
> 
> Tested-on: WCN6855 hw2.1 PCI WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.23
> Tested-on: QCN9074 hw1.0 PCI WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1
> Tested-on: IPQ8074 hw2.0 AHB WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1
> 
> Signed-off-by: Baochen Qiang <quic_bqiang@quicinc.com>
> Acked-by: Jeff Johnson <quic_jjohnson@quicinc.com>
> Signed-off-by: Kalle Valo <quic_kvalo@quicinc.com>

Patch applied to ath-next branch of ath.git, thanks.

dbd73acb22d8 wifi: ath11k: enable 36 bit mask for stream DMA
diff mbox series

Patch

diff --git a/drivers/net/wireless/ath/ath11k/mhi.c b/drivers/net/wireless/ath/ath11k/mhi.c
index 6835c14b82cc..8753dde2fe2b 100644
--- a/drivers/net/wireless/ath/ath11k/mhi.c
+++ b/drivers/net/wireless/ath/ath11k/mhi.c
@@ -423,7 +423,7 @@  int ath11k_mhi_register(struct ath11k_pci *ab_pci)
 			goto free_controller;
 	} else {
 		mhi_ctrl->iova_start = 0;
-		mhi_ctrl->iova_stop = 0xFFFFFFFF;
+		mhi_ctrl->iova_stop = ab_pci->dma_mask;
 	}
 
 	mhi_ctrl->rddm_size = RDDM_DUMP_SIZE;
diff --git a/drivers/net/wireless/ath/ath11k/pci.c b/drivers/net/wireless/ath/ath11k/pci.c
index 09e65c5e55c4..1d0f5d6e1cf2 100644
--- a/drivers/net/wireless/ath/ath11k/pci.c
+++ b/drivers/net/wireless/ath/ath11k/pci.c
@@ -18,7 +18,8 @@ 
 #include "qmi.h"
 
 #define ATH11K_PCI_BAR_NUM		0
-#define ATH11K_PCI_DMA_MASK		32
+#define ATH11K_PCI_DMA_MASK		36
+#define ATH11K_PCI_COHERENT_DMA_MASK	32
 
 #define TCSR_SOC_HW_VERSION		0x0224
 #define TCSR_SOC_HW_VERSION_MAJOR_MASK	GENMASK(11, 8)
@@ -526,13 +527,22 @@  static int ath11k_pci_claim(struct ath11k_pci *ab_pci, struct pci_dev *pdev)
 		goto disable_device;
 	}
 
-	ret = dma_set_mask_and_coherent(&pdev->dev,
-					DMA_BIT_MASK(ATH11K_PCI_DMA_MASK));
+	ret = dma_set_mask(&pdev->dev,
+			   DMA_BIT_MASK(ATH11K_PCI_DMA_MASK));
 	if (ret) {
 		ath11k_err(ab, "failed to set pci dma mask to %d: %d\n",
 			   ATH11K_PCI_DMA_MASK, ret);
 		goto release_region;
 	}
+	ab_pci->dma_mask = DMA_BIT_MASK(ATH11K_PCI_DMA_MASK);
+
+	ret = dma_set_coherent_mask(&pdev->dev,
+				    DMA_BIT_MASK(ATH11K_PCI_COHERENT_DMA_MASK));
+	if (ret) {
+		ath11k_err(ab, "failed to set pci coherent dma mask to %d: %d\n",
+			   ATH11K_PCI_COHERENT_DMA_MASK, ret);
+		goto release_region;
+	}
 
 	pci_set_master(pdev);
 
diff --git a/drivers/net/wireless/ath/ath11k/pci.h b/drivers/net/wireless/ath/ath11k/pci.h
index e9a01f344ec6..21274175f0d2 100644
--- a/drivers/net/wireless/ath/ath11k/pci.h
+++ b/drivers/net/wireless/ath/ath11k/pci.h
@@ -72,6 +72,7 @@  struct ath11k_pci {
 	/* enum ath11k_pci_flags */
 	unsigned long flags;
 	u16 link_ctl;
+	u64 dma_mask;
 };
 
 static inline struct ath11k_pci *ath11k_pci_priv(struct ath11k_base *ab)