| Message ID | 20250402233407.2452429-7-willmcvicker@google.com |
|---|---|
| State | New |
| Headers | show |
| Series | Add module support for Arm64 Exynos MCT driver | expand |
Hi Will, On Wed, Apr 02, 2025 at 04:33:57PM -0700, Will McVicker wrote: > From: Donghoon Yu <hoony.yu@samsung.com> > > On Arm64 platforms the Exynos MCT driver can be built as a module. On > boot (and even after boot) the arch_timer is used as the clocksource and > tick timer. Once the MCT driver is loaded, it can be used as the wakeup > source for the arch_timer.
On 04/15/2025, Daniel Lezcano wrote: > Hi Will, Hi Daniel, > > On Wed, Apr 02, 2025 at 04:33:57PM -0700, Will McVicker wrote: > > From: Donghoon Yu <hoony.yu@samsung.com> > > > > On Arm64 platforms the Exynos MCT driver can be built as a module. On > > boot (and even after boot) the arch_timer is used as the clocksource and > > tick timer. Once the MCT driver is loaded, it can be used as the wakeup > > source for the arch_timer. > > From a previous thread where there is no answer: > > https://lore.kernel.org/all/c1e8abec-680c-451d-b5df-f687291aa413@linaro.org/ > > I don't feel comfortable with changing the clocksource / clockevent drivers to > a module for the reasons explained in the aforementionned thread. > > Before this could be accepted, I really need a strong acked-by from Thomas Thanks for the response! I'll copy-and-paste your replies from that previous thread and try to address your concerns. > * the GKI approach is to have an update for the 'mainline' kernel and > let the different SoC vendors deal with their drivers. I'm afraid this > will prevent driver fixes to be carry on upstream because they will stay > in the OoT kernels I can't speak for that specific thread or their intent, but I can speak to this thread and our intent. This whole patch series is about upstreaming the downstream changes. So saying this will prevent others from upstreaming changes is punishing the folks who are actually trying to upstream changes. I don't think that's a fair way to handle this. Also, rejecting this series will not prevent people from upstreaming their changes, it'll just make it more unlikely because they now have to deal with upstreaming more changes that were rejected in the past. That's daunting for someone who doesn't do upstreaming often. I'm telling this from experience dealing with SoC vendors and asking them to upstream stuff. With that said, let me try to address some of your technical concerns. > * the core code may not be prepared for that, so loading / unloading > the modules with active timers may result into some issues We had the same concern for irqchip drivers. We can easily disable unloading for these clocksource modules just like we did for irqchip by making them permanent modules. > * it may end up with some interactions with cpuidle at boot time and > the broadcast timer If I'm understanding this correctly, no driver is guaranteed to probe at initialization time regardless of whether it is built-in or a module. Taking a look at the other clocksource drivers, I found that the following drivers are all calling `clocksource_register_hz()` and `clockevents_config_and_register()` at probe time. timer-sun5i.c sh_tmu.c sh_cmt.c timer-tegra186.c timer-stm32-lp.c (only calls clockevents_config_and_register()) So this concern is unrelated to building these drivers are modules. Please let me know if I'm missing something here. > * the timekeeping may do jump in the past [if and] when switching the > clocksource Can you clarify how this relates to modules? IIUC, the clocksource can be changed anytime by writing to: /sys/devices/system/clocksource/clocksource0/current_clocksource If there's a bug related to timekeeping and changing the clocksource, then that should be handled separately from the modularization code. For ARM64 in general, the recommendation is to use the ARM architected timer which is not a module and is used for scheduling and timekeeping. While the Exynos MCT driver can functionally be used as the primary clocksource, it's not recommended due to performance issues. So building the MCT driver as a kernel module really shouldn't be an issue and has been thoroughly testing on several generations of Pixel devices which is why we are trying to upstream our downstream technical debt (so we can directly using the upstream version of the Exynos MCT driver). Thanks, Will [...]
On Tue, Apr 15, 2025 at 9:50 AM Daniel Lezcano <daniel.lezcano@linaro.org> wrote: > On Wed, Apr 02, 2025 at 04:33:57PM -0700, Will McVicker wrote: > > From: Donghoon Yu <hoony.yu@samsung.com> > > > > On Arm64 platforms the Exynos MCT driver can be built as a module. On > > boot (and even after boot) the arch_timer is used as the clocksource and > > tick timer. Once the MCT driver is loaded, it can be used as the wakeup > > source for the arch_timer. > > From a previous thread where there is no answer: > > https://lore.kernel.org/all/c1e8abec-680c-451d-b5df-f687291aa413@linaro.org/ > > I don't feel comfortable with changing the clocksource / clockevent drivers to > a module for the reasons explained in the aforementionned thread. I wasn't CC'ed on that, but to address a few of your points: > I have some concerns about this kind of changes: > > * the core code may not be prepared for that, so loading / unloading > the modules with active timers may result into some issues That's a fair concern, but permanent modules (which are loaded but not unloaded) shouldn't suffer this issue. I recognize having modules be fully unloadable is generally cleaner and preferred, but I also see the benefit of allowing permanent modules to be one-way loaded so a generic/distro kernel shared between lots of different platforms doesn't need to be bloated with drivers that aren't used everywhere. Obviously any single driver doesn't make a huge difference, but all the small drivers together does add up. > * it may end up with some interactions with cpuidle at boot time and > the broadcast timer Do you have more details as to your concerns here? I know there can be cases of issues if the built in clockevent drivers are problematic and the working ones don't load until later, you can have races where if the system goes into idle before the module loads it could stall out (there was a recent issue with an older iMac TSC halting in idle and it not reliably getting disqualified before it got stuck in idle). In those cases I could imagine folks reasonably arguing for including the working clock as a built in, but I'm not sure I'd say forcing everything to be built in is the better approach. > * the timekeeping may do jump in the past [if and] when switching the > clocksource ? It shouldn't. We've had tests in kselftest that switch between clocksources checking for inconsistencies for awhile, so if such a jump occurred it would be considered a bug. > * the GKI approach is to have an update for the 'mainline' kernel and > let the different SoC vendors deal with their drivers. I'm afraid this > will prevent driver fixes to be carry on upstream because they will stay > in the OoT kernels I'm not sure I understand this point? Could you expand on it a bit? While I very much can understand concerns and potential downsides of the GKI approach, I'm not sure how that applies to the submission here, as the benefit would apply to classic distro kernels as much as GKI. I realize in the time since I started this reply, Will has already covered much of the above! So apologies for being redundant. That said, there are some non-modularization changes in this series that should be considered even if the modularization logic is a continued sticking point. -john
On Tue, Apr 15, 2025 at 02:25:43PM -0700, William McVicker wrote: > On 04/15/2025, Daniel Lezcano wrote: > > Hi Will, > > Hi Daniel, > > > > > On Wed, Apr 02, 2025 at 04:33:57PM -0700, Will McVicker wrote: > > > From: Donghoon Yu <hoony.yu@samsung.com> > > > > > > On Arm64 platforms the Exynos MCT driver can be built as a module. On > > > boot (and even after boot) the arch_timer is used as the clocksource and > > > tick timer. Once the MCT driver is loaded, it can be used as the wakeup > > > source for the arch_timer. > > > > From a previous thread where there is no answer: > > > > https://lore.kernel.org/all/c1e8abec-680c-451d-b5df-f687291aa413@linaro.org/ > > > > I don't feel comfortable with changing the clocksource / clockevent drivers to > > a module for the reasons explained in the aforementionned thread. > > > > Before this could be accepted, I really need a strong acked-by from Thomas > > Thanks for the response! I'll copy-and-paste your replies from that previous > thread and try to address your concerns. > > > * the GKI approach is to have an update for the 'mainline' kernel and > > let the different SoC vendors deal with their drivers. I'm afraid this > > will prevent driver fixes to be carry on upstream because they will stay > > in the OoT kernels > > I can't speak for that specific thread or their intent, but I can speak to this > thread and our intent. > > This whole patch series is about upstreaming the downstream changes. So saying > this will prevent others from upstreaming changes is punishing the folks who > are actually trying to upstream changes. I don't think that's a fair way to > handle this. > > Also, rejecting this series will not prevent people from upstreaming their > changes, it'll just make it more unlikely because they now have to deal with > upstreaming more changes that were rejected in the past. That's daunting for > someone who doesn't do upstreaming often. I'm telling this from experience > dealing with SoC vendors and asking them to upstream stuff. > > With that said, let me try to address some of your technical concerns. I won't reject the series based on my opinion. Answering the technical concerns will prevail. Why is it needed to convert the timer into a module ? > > * the core code may not be prepared for that, so loading / unloading > > the modules with active timers may result into some issues > > We had the same concern for irqchip drivers. We can easily disable unloading > for these clocksource modules just like we did for irqchip by making them > permanent modules. In the clockevent / clocksource initialization process, depending on the platform, some are needed very early and other can be loaded later. For example, the usual configuration is the architected timers are initialized very early, then the external timer is loaded a bit later. And when this one is loaded it does not take over the architected timers. It acts as a "broadcast" timer to program the next timer event when the current CPU is going to an idle state where the local timer is stopped. Other cases are the architected timers are not desired and the 'external' timer is used in place when it is loaded with a higher rating. Some configuration can mimic local timers by settting a per CPU timer. Some platforms could be without the architected timers, so the 'external' timer is used. Let's imagine the system started, the timers are running and then we load a module with a timer replacing the current ones. Does it work well ? Are we sure, the timer modularization is compatible with all the timer use cases ? > > * it may end up with some interactions with cpuidle at boot time and > > the broadcast timer > > If I'm understanding this correctly, no driver is guaranteed to probe at > initialization time regardless of whether it is built-in or a module. Taking > a look at the other clocksource drivers, I found that the following drivers are > all calling `clocksource_register_hz()` and `clockevents_config_and_register()` > at probe time. > > timer-sun5i.c > sh_tmu.c > sh_cmt.c > timer-tegra186.c > timer-stm32-lp.c (only calls clockevents_config_and_register()) > > So this concern is unrelated to building these drivers are modules. Please let > me know if I'm missing something here. We would have to check each platform individually to answer this question. The interaction between cpuidle and the timer module is about not having a broadcast timer when cpuidle initializes and then having it later when the module is loaded. Did you check the deep idle states are used after loading the module ? > > * the timekeeping may do jump in the past [if and] when switching the > > clocksource > > Can you clarify how this relates to modules? IIUC, the clocksource can be > changed anytime by writing to: > > /sys/devices/system/clocksource/clocksource0/current_clocksource The clocksource counter is stopped when it is not the current one. So when you switch it, the new clocksource counter could be different and can make you jump back in time. > If there's a bug related to timekeeping and changing the clocksource, then that > should be handled separately from the modularization code. I disagree :) The whole point is the time framework may not be totally immune against the timer modularization. It is about identifying the corner cases where the timer driver modularization can have an impact and set the scene to support it. > For ARM64 in general, the recommendation is to use the ARM architected timer > which is not a module and is used for scheduling and timekeeping. While the > Exynos MCT driver can functionally be used as the primary clocksource, it's not > recommended due to performance issues. So building the MCT driver as a kernel > module really shouldn't be an issue and has been thoroughly testing on several > generations of Pixel devices which is why we are trying to upstream our > downstream technical debt (so we can directly using the upstream version of the > Exynos MCT driver). The discussion is not about only the Exynos MCT but as you are not the first one asking to convert the timer driver to a module, we should check what could be the impact on the time framework and the system in general. Others proposed to convert to module and I asked to investigate the impact. Nobody came back with a clear answer and there is no feedback from Thomas. > [...]
On Tue, Apr 15, 2025 at 05:48:41PM -0700, John Stultz wrote: > On Tue, Apr 15, 2025 at 9:50 AM Daniel Lezcano > <daniel.lezcano@linaro.org> wrote: > > On Wed, Apr 02, 2025 at 04:33:57PM -0700, Will McVicker wrote: > > > From: Donghoon Yu <hoony.yu@samsung.com> > > > > > > On Arm64 platforms the Exynos MCT driver can be built as a module. On > > > boot (and even after boot) the arch_timer is used as the clocksource and > > > tick timer. Once the MCT driver is loaded, it can be used as the wakeup > > > source for the arch_timer. > > > > From a previous thread where there is no answer: > > > > https://lore.kernel.org/all/c1e8abec-680c-451d-b5df-f687291aa413@linaro.org/ > > > > I don't feel comfortable with changing the clocksource / clockevent drivers to > > a module for the reasons explained in the aforementionned thread. > > I wasn't CC'ed on that, but to address a few of your points: > > > I have some concerns about this kind of changes: > > > > * the core code may not be prepared for that, so loading / unloading > > the modules with active timers may result into some issues > > That's a fair concern, but permanent modules (which are loaded but not > unloaded) shouldn't suffer this issue. I recognize having modules be > fully unloadable is generally cleaner and preferred, but I also see > the benefit of allowing permanent modules to be one-way loaded so a > generic/distro kernel shared between lots of different platforms > doesn't need to be bloated with drivers that aren't used everywhere. > Obviously any single driver doesn't make a huge difference, but all > the small drivers together does add up. Yes, I agree. So the whole clockevent / clocksource drivers policy would have to be making impossible to unload a module once it is loaded. Do you have any ideas how to ensure that the converted drivers follow this rule without putting more burden on the maintainer? > > * it may end up with some interactions with cpuidle at boot time and > > the broadcast timer > > Do you have more details as to your concerns here? I know there can be > cases of issues if the built in clockevent drivers are problematic and > the working ones don't load until later, you can have races where if > the system goes into idle before the module loads it could stall out > (there was a recent issue with an older iMac TSC halting in idle and > it not reliably getting disqualified before it got stuck in idle). Yes, that is that kind of issue I suspect. > In > those cases I could imagine folks reasonably arguing for including the > working clock as a built in, but I'm not sure I'd say forcing > everything to be built in is the better approach. When the first driver converted as a module will be accepted, I'm pretty sure there will be a wave of patches to convert more drivers into modules. What tool / use cases / tests can we put in place to ensure it is not breaking the existing platforms for the different configurations? > > * the timekeeping may do jump in the past [if and] when switching the > > clocksource > > ? It shouldn't. We've had tests in kselftest that switch between > clocksources checking for inconsistencies for awhile, so if such a > jump occurred it would be considered a bug. But in the context of modules, the current clocksource counter is running but what about the clocksource's counter related to the module which will be started when the driver is loaded and then switches to the new clocksource. Is it possible in this case there is a time drift between the clocksource which was started at boot time and the one started later when the module is loaded ? > > * the GKI approach is to have an update for the 'mainline' kernel and > > let the different SoC vendors deal with their drivers. I'm afraid this > > will prevent driver fixes to be carry on upstream because they will stay > > in the OoT kernels > > I'm not sure I understand this point? Could you expand on it a bit? > While I very much can understand concerns and potential downsides of > the GKI approach, I'm not sure how that applies to the submission > here, as the benefit would apply to classic distro kernels as much as > GKI. Ok let's consider my comment as out of the technical aspects of the changes. I can clarify it but it does not enter into consideration for the module conversion. It is an overall feeling about the direction of all in modules for GKI policy. I'm a little worried about changes not carried on mainline because it is no longer an obstacle to keep OoT drivers. The core kernel is mainline but the drivers can be vendor provided as module. I understand it is already the case but the time framework is the base brick of the system, so there is the risk a platform is supported with less than the minimum functionality.
On Wed, Apr 16, 2025 at 6:46 AM Daniel Lezcano <daniel.lezcano@linaro.org> wrote: > On Tue, Apr 15, 2025 at 05:48:41PM -0700, John Stultz wrote: > > On Tue, Apr 15, 2025 at 9:50 AM Daniel Lezcano > > <daniel.lezcano@linaro.org> wrote: > > > I have some concerns about this kind of changes: > > > > > > * the core code may not be prepared for that, so loading / unloading > > > the modules with active timers may result into some issues > > > > That's a fair concern, but permanent modules (which are loaded but not > > unloaded) shouldn't suffer this issue. I recognize having modules be > > fully unloadable is generally cleaner and preferred, but I also see > > the benefit of allowing permanent modules to be one-way loaded so a > > generic/distro kernel shared between lots of different platforms > > doesn't need to be bloated with drivers that aren't used everywhere. > > Obviously any single driver doesn't make a huge difference, but all > > the small drivers together does add up. > > Yes, I agree. > > So the whole clockevent / clocksource drivers policy would have to be making > impossible to unload a module once it is loaded. > > Do you have any ideas how to ensure that the converted drivers follow this > rule without putting more burden on the maintainer? Permanent modules just don't have a module_exit() hook, so that is pretty easy to look for. Obviously, I don't want to add more burden to the maintainership.
On Wed, Apr 16, 2025 at 12:48 PM John Stultz <jstultz@google.com> wrote: > On Wed, Apr 16, 2025 at 6:46 AM Daniel Lezcano > <daniel.lezcano@linaro.org> wrote: > > So the whole clockevent / clocksource drivers policy would have to be making > > impossible to unload a module once it is loaded. > > > > Do you have any ideas how to ensure that the converted drivers follow this > > rule without putting more burden on the maintainer? > > Permanent modules just don't have a module_exit() hook, so that is > pretty easy to look for. > Obviously, I don't want to add more burden to the maintainership. > > From a given clockevent driver (or maybe a function pointer), we could > check on the registration by calling __module_address(addr) [thanks to > Sami Tolvanen for pointing that function out to me] on one of the > function pointers provided, and check that there isn't a module->exit > pointer. Saravana also pointed out to me another approach that the irqchip code uses: macros to populate an owner field with THIS_MODULE so that one can easily get to the module struct https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/irqchip.h#n41 thanks -john
diff --git a/drivers/clocksource/Kconfig b/drivers/clocksource/Kconfig index 487c85259967..e89373827c3a 100644 --- a/drivers/clocksource/Kconfig +++ b/drivers/clocksource/Kconfig @@ -443,7 +443,8 @@ config ATMEL_TCB_CLKSRC Support for Timer Counter Blocks on Atmel SoCs. config CLKSRC_EXYNOS_MCT - bool "Exynos multi core timer driver" if COMPILE_TEST + tristate "Exynos multi core timer driver" if ARM64 + default y if ARCH_EXYNOS || COMPILE_TEST depends on ARM || ARM64 depends on ARCH_ARTPEC || ARCH_EXYNOS || COMPILE_TEST help diff --git a/drivers/clocksource/exynos_mct.c b/drivers/clocksource/exynos_mct.c index 62febeb4e1de..8943274378be 100644 --- a/drivers/clocksource/exynos_mct.c +++ b/drivers/clocksource/exynos_mct.c @@ -15,9 +15,11 @@ #include <linux/cpu.h> #include <linux/delay.h> #include <linux/percpu.h> +#include <linux/module.h> #include <linux/of.h> #include <linux/of_irq.h> #include <linux/of_address.h> +#include <linux/platform_device.h> #include <linux/clocksource.h> #include <linux/sched_clock.h> @@ -241,7 +243,7 @@ static cycles_t exynos4_read_current_timer(void) } #endif -static int __init exynos4_clocksource_init(bool frc_shared) +static int exynos4_clocksource_init(bool frc_shared) { /* * When the frc is shared, the main processor should have already @@ -511,7 +513,7 @@ static int exynos4_mct_dying_cpu(unsigned int cpu) return 0; } -static int __init exynos4_timer_resources(struct device_node *np) +static int exynos4_timer_resources(struct device_node *np) { struct clk *mct_clk, *tick_clk; @@ -539,7 +541,7 @@ static int __init exynos4_timer_resources(struct device_node *np) * @local_idx: array mapping CPU numbers to local timer indices * @nr_local: size of @local_idx array */ -static int __init exynos4_timer_interrupts(struct device_node *np, +static int exynos4_timer_interrupts(struct device_node *np, unsigned int int_type, const u32 *local_idx, size_t nr_local) @@ -652,7 +654,7 @@ static int __init exynos4_timer_interrupts(struct device_node *np, return err; } -static int __init mct_init_dt(struct device_node *np, unsigned int int_type) +static int mct_init_dt(struct device_node *np, unsigned int int_type) { bool frc_shared = of_property_read_bool(np, "samsung,frc-shared"); u32 local_idx[MCT_NR_LOCAL] = {0}; @@ -700,15 +702,48 @@ static int __init mct_init_dt(struct device_node *np, unsigned int int_type) return exynos4_clockevent_init(); } - -static int __init mct_init_spi(struct device_node *np) +static int mct_init_spi(struct device_node *np) { return mct_init_dt(np, MCT_INT_SPI); } -static int __init mct_init_ppi(struct device_node *np) +static int mct_init_ppi(struct device_node *np) { return mct_init_dt(np, MCT_INT_PPI); } + +#ifdef MODULE +static int exynos4_mct_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + int (*mct_init)(struct device_node *np); + + mct_init = of_device_get_match_data(dev); + if (!mct_init) + return -EINVAL; + + return mct_init(dev->of_node); +} + +static const struct of_device_id exynos4_mct_match_table[] = { + { .compatible = "samsung,exynos4210-mct", .data = &mct_init_spi, }, + { .compatible = "samsung,exynos4412-mct", .data = &mct_init_ppi, }, + {} +}; +MODULE_DEVICE_TABLE(of, exynos4_mct_match_table); + +static struct platform_driver exynos4_mct_driver = { + .probe = exynos4_mct_probe, + .driver = { + .name = "exynos-mct", + .of_match_table = exynos4_mct_match_table, + }, +}; +module_platform_driver(exynos4_mct_driver); +#else TIMER_OF_DECLARE(exynos4210, "samsung,exynos4210-mct", mct_init_spi); TIMER_OF_DECLARE(exynos4412, "samsung,exynos4412-mct", mct_init_ppi); +#endif + +MODULE_DESCRIPTION("Exynos Multi Core Timer Driver"); +MODULE_LICENSE("GPL");