| Message ID | 20210202071648.1776-5-thunder.leizhen@huawei.com |
|---|---|
| State | New |
| Headers | show |
| Series | ARM: Add support for Hisilicon Kunpeng L3 cache controller | expand |
On Tue, Feb 2, 2021 at 8:16 AM Zhen Lei <thunder.leizhen@huawei.com> wrote: > + > +/* > + * All read and write operations on L3 cache registers are protected by the > + * spinlock, except for l3cache_init(). Each time the L3 cache operation is > + * performed, all related information is filled into its registers. Therefore, > + * there is no memory order problem when only _relaxed() functions are used. Thank you for including the text. I don't think the explanation with the spin_lock() explains why this can be considered safe though, as spin_lock() only contains serialization against other CPUs (smp_mb()) rather than the stronger DMA barriers implied by readl and writel. As Russell previously explained, these barriers are the L1 cache operations (e.g. v7_dma_inv_range) do include stronger barriers, so it would be better to come up with a justification based on those. > + * This can help us achieve some performance improvement: > + * 1) The readl_relaxed() is about 20ns faster than readl(). > + * 2) The writel_relaxed() is about 123ns faster than writel(). These are not really the performance numbers I asked for, as a low-level benchmark comparing the instructions is rather meaningless. The time spent waiting for the barrier depends on what else is going on around the barrier. Also, most of the time would likely be spent spinning in the loop around readl() while the cache operations are in progress, so the latency of a single readl() is not necessarily significant. To have a more useful performance number, try mentioning the most performance sensitive non-coherent DMA master on one of the chips that has this cache controller, and a high-level performance number such as "1.2% more network packets per second" if that is something you can measure easily. Of course, if all high-speed DMA masters on this chip are cache coherent, there is no need for performance numbers, just mention that we don't care about speed in that case. Arnd
On 2021/2/2 16:44, Arnd Bergmann wrote: > On Tue, Feb 2, 2021 at 8:16 AM Zhen Lei <thunder.leizhen@huawei.com> wrote: >> + >> +/* >> + * All read and write operations on L3 cache registers are protected by the >> + * spinlock, except for l3cache_init(). Each time the L3 cache operation is >> + * performed, all related information is filled into its registers. Therefore, >> + * there is no memory order problem when only _relaxed() functions are used. > > Thank you for including the text. > > I don't think the explanation with the spin_lock() explains why this > can be considered safe though, as spin_lock() only contains serialization > against other CPUs (smp_mb()) rather than the stronger DMA barriers > implied by readl and writel. As Russell previously explained, these > barriers are the L1 cache operations (e.g. v7_dma_inv_range) do > include stronger barriers, so it would be better to come up with a > justification based on those. Okay, I'll correct the description. > >> + * This can help us achieve some performance improvement: >> + * 1) The readl_relaxed() is about 20ns faster than readl(). >> + * 2) The writel_relaxed() is about 123ns faster than writel(). > > These are not really the performance numbers I asked for, as a > low-level benchmark comparing the instructions is rather meaningless. > The time spent waiting for the barrier depends on what else is going > on around the barrier. Also, most of the time would likely be > spent spinning in the loop around readl() while the cache operations > are in progress, so the latency of a single readl() is not necessarily > significant. > > To have a more useful performance number, try mentioning the > most performance sensitive non-coherent DMA master on one > of the chips that has this cache controller, and a high-level > performance number such as "1.2% more network packets per > second" if that is something you can measure easily. It's not easy. My board only have debugging NIC, only the downstream products have high-speed service NIC. Software needs to be packaged layer by layer. > > Of course, if all high-speed DMA masters on this chip are > cache coherent, there is no need for performance numbers, just > mention that we don't care about speed in that case. It's not cache coherent, otherwise, the L3 cache does not need to be operated. > > Arnd > > . >
On Tue, Feb 2, 2021 at 1:18 PM Leizhen (ThunderTown) <thunder.leizhen@huawei.com> wrote: > On 2021/2/2 16:44, Arnd Bergmann wrote: > > > > To have a more useful performance number, try mentioning the > > most performance sensitive non-coherent DMA master on one > > of the chips that has this cache controller, and a high-level > > performance number such as "1.2% more network packets per > > second" if that is something you can measure easily. > > It's not easy. My board only have debugging NIC, only the downstream > products have high-speed service NIC. Software needs to be packaged > layer by layer. > > > > > Of course, if all high-speed DMA masters on this chip are > > cache coherent, there is no need for performance numbers, just > > mention that we don't care about speed in that case. > > It's not cache coherent, otherwise, the L3 cache does not need to be > operated. Ok, I see. In this case, just explain that the high-speed NIC is not cache-coherent, so this is expected to make a difference, even if you can't quantify it exactly. Arnd
diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig index 02692fbe2db5c59..d2082503de053d2 100644 --- a/arch/arm/mm/Kconfig +++ b/arch/arm/mm/Kconfig @@ -1070,6 +1070,16 @@ config CACHE_XSC3L2 help This option enables the L2 cache on XScale3. +config CACHE_KUNPENG_L3 + bool "Enable the Hisilicon Kunpeng L3 cache controller" + depends on ARCH_KUNPENG50X && OF + default y + select OUTER_CACHE + help + This option enables the Kunpeng L3 cache controller on Hisilicon + Kunpeng506 and Kunpeng509 SoCs. It supports a maximum of 36-bit + physical addresses. + config ARM_L1_CACHE_SHIFT_6 bool default y if CPU_V7 diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile index 3510503bc5e688b..ececc5489e353eb 100644 --- a/arch/arm/mm/Makefile +++ b/arch/arm/mm/Makefile @@ -112,6 +112,7 @@ obj-$(CONFIG_CACHE_L2X0_PMU) += cache-l2x0-pmu.o obj-$(CONFIG_CACHE_XSC3L2) += cache-xsc3l2.o obj-$(CONFIG_CACHE_TAUROS2) += cache-tauros2.o obj-$(CONFIG_CACHE_UNIPHIER) += cache-uniphier.o +obj-$(CONFIG_CACHE_KUNPENG_L3) += cache-kunpeng-l3.o KASAN_SANITIZE_kasan_init.o := n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm/mm/cache-kunpeng-l3.c b/arch/arm/mm/cache-kunpeng-l3.c new file mode 100644 index 000000000000000..64f892de9d68058 --- /dev/null +++ b/arch/arm/mm/cache-kunpeng-l3.c @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2021 Hisilicon Limited. + */ + +#include <linux/init.h> +#include <linux/spinlock.h> +#include <linux/io.h> +#include <linux/of_address.h> + +#include <asm/cacheflush.h> + +#define L3_CACHE_LINE_SHITF 6 + +#define L3_CTRL 0x0 +#define L3_CTRL_ENABLE (1U << 0) +#define L3_CTRL_DISABLE (0U << 0) + +#define L3_AUCTRL 0x4 +#define L3_AUCTRL_EVENT_EN BIT(23) +#define L3_AUCTRL_ECC_EN BIT(8) + +#define L3_MAINT_CTRL 0x20 +#define L3_MAINT_RANGE_MASK GENMASK(3, 3) +#define L3_MAINT_RANGE_ALL (0U << 3) +#define L3_MAINT_RANGE_ADDR (1U << 3) +#define L3_MAINT_TYPE_MASK GENMASK(2, 1) +#define L3_MAINT_TYPE_CLEAN (1U << 1) +#define L3_MAINT_TYPE_INV (2U << 1) +#define L3_MAINT_TYPE_FLUSH (3U << 1) +#define L3_MAINT_STATUS_MASK GENMASK(0, 0) +#define L3_MAINT_STATUS_START (1U << 0) +#define L3_MAINT_STATUS_END (0U << 0) + +#define L3_MAINT_START 0x24 +#define L3_MAINT_END 0x28 + +static DEFINE_RAW_SPINLOCK(l3cache_lock); +static void __iomem *l3_ctrl_base; + +/* + * All read and write operations on L3 cache registers are protected by the + * spinlock, except for l3cache_init(). Each time the L3 cache operation is + * performed, all related information is filled into its registers. Therefore, + * there is no memory order problem when only _relaxed() functions are used. + * This can help us achieve some performance improvement: + * 1) The readl_relaxed() is about 20ns faster than readl(). + * 2) The writel_relaxed() is about 123ns faster than writel(). + */ +static void l3cache_maint_common(u32 range, u32 op_type) +{ + u32 reg; + + reg = readl_relaxed(l3_ctrl_base + L3_MAINT_CTRL); + reg &= ~(L3_MAINT_RANGE_MASK | L3_MAINT_TYPE_MASK); + reg |= range | op_type; + reg |= L3_MAINT_STATUS_START; + writel_relaxed(reg, l3_ctrl_base + L3_MAINT_CTRL); + + /* Wait until the hardware maintenance operation is complete. */ + do { + cpu_relax(); + reg = readl_relaxed(l3_ctrl_base + L3_MAINT_CTRL); + } while ((reg & L3_MAINT_STATUS_MASK) != L3_MAINT_STATUS_END); +} + +static void l3cache_maint_range(phys_addr_t start, phys_addr_t end, u32 op_type) +{ + start = start >> L3_CACHE_LINE_SHITF; + end = ((end - 1) >> L3_CACHE_LINE_SHITF) + 1; + + writel_relaxed(start, l3_ctrl_base + L3_MAINT_START); + writel_relaxed(end, l3_ctrl_base + L3_MAINT_END); + + l3cache_maint_common(L3_MAINT_RANGE_ADDR, op_type); +} + +static inline void l3cache_flush_all_nolock(void) +{ + l3cache_maint_common(L3_MAINT_RANGE_ALL, L3_MAINT_TYPE_FLUSH); +} + +static void l3cache_flush_all(void) +{ + unsigned long flags; + + raw_spin_lock_irqsave(&l3cache_lock, flags); + l3cache_flush_all_nolock(); + raw_spin_unlock_irqrestore(&l3cache_lock, flags); +} + +static void l3cache_inv_range(phys_addr_t start, phys_addr_t end) +{ + unsigned long flags; + + raw_spin_lock_irqsave(&l3cache_lock, flags); + l3cache_maint_range(start, end, L3_MAINT_TYPE_INV); + raw_spin_unlock_irqrestore(&l3cache_lock, flags); +} + +static void l3cache_clean_range(phys_addr_t start, phys_addr_t end) +{ + unsigned long flags; + + raw_spin_lock_irqsave(&l3cache_lock, flags); + l3cache_maint_range(start, end, L3_MAINT_TYPE_CLEAN); + raw_spin_unlock_irqrestore(&l3cache_lock, flags); +} + +static void l3cache_flush_range(phys_addr_t start, phys_addr_t end) +{ + unsigned long flags; + + raw_spin_lock_irqsave(&l3cache_lock, flags); + l3cache_maint_range(start, end, L3_MAINT_TYPE_FLUSH); + raw_spin_unlock_irqrestore(&l3cache_lock, flags); +} + +static void l3cache_disable(void) +{ + unsigned long flags; + + raw_spin_lock_irqsave(&l3cache_lock, flags); + l3cache_flush_all_nolock(); + writel_relaxed(L3_CTRL_DISABLE, l3_ctrl_base + L3_CTRL); + raw_spin_unlock_irqrestore(&l3cache_lock, flags); +} + +static const struct of_device_id l3cache_ids[] __initconst = { + {.compatible = "hisilicon,kunpeng-l3cache", .data = NULL}, + {} +}; + +static int __init l3cache_init(void) +{ + u32 reg; + struct device_node *node; + + node = of_find_matching_node(NULL, l3cache_ids); + if (!node) + return -ENODEV; + + l3_ctrl_base = of_iomap(node, 0); + if (!l3_ctrl_base) { + pr_err("failed to map Kunpeng L3 cache controller registers\n"); + return -ENOMEM; + } + + reg = readl_relaxed(l3_ctrl_base + L3_CTRL); + if (!(reg & L3_CTRL_ENABLE)) { + /* + * Ensure that no L3 cache hardware maintenance operations are + * being performed before enabling the L3 cache. Wait for it to + * finish. + */ + do { + cpu_relax(); + reg = readl_relaxed(l3_ctrl_base + L3_MAINT_CTRL); + } while ((reg & L3_MAINT_STATUS_MASK) != L3_MAINT_STATUS_END); + + reg = readl_relaxed(l3_ctrl_base + L3_AUCTRL); + reg |= L3_AUCTRL_EVENT_EN | L3_AUCTRL_ECC_EN; + writel_relaxed(reg, l3_ctrl_base + L3_AUCTRL); + + writel_relaxed(L3_CTRL_ENABLE, l3_ctrl_base + L3_CTRL); + } + + outer_cache.inv_range = l3cache_inv_range; + outer_cache.clean_range = l3cache_clean_range; + outer_cache.flush_range = l3cache_flush_range; + outer_cache.flush_all = l3cache_flush_all; + outer_cache.disable = l3cache_disable; + + pr_info("Hisilicon Kunpeng L3 cache controller enabled\n"); + + return 0; +} +arch_initcall(l3cache_init);