| Message ID | 20210708082128.2832904-1-primoz.fiser@norik.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v2] watchdog: da9062: da9063: prevent pings ahead of machine reset | expand |
On 08 July 2021 09:21, Primoz Fiser wrote: > Proper machine resets via da9062/da9063 PMICs are very tricky as they > require special i2c atomic transfers when interrupts are not available > anymore. This is also a reason why both PMIC's restart handlers do not > use regmap but instead opt for i2c_smbus_write_byte_data() which does > i2c transfer in atomic manner. Under the hood, this function tries to > obtain i2c bus lock with call to i2c_adapter_trylock_bus() which will > return -EAGAIN (-11) if lock is not available. > > Since commit 982bb70517aef ("watchdog: reset last_hw_keepalive time at > start") occasional restart handler failures with "Failed to shutdown > (err = -11)" error messages were observed, indicating that some > process is holding the i2c bus lock. Investigation into the matter > uncovered that sometimes during reboot sequence watchdog ping is issued > late into poweroff/reboot phase which did not happen before mentioned > commit (usually the watchdog ping happened immediately as commit message > suggests). As of now, when watchdog ping usually happens late into > poweroff/reboot stage when interrupts are not available anymore, i2c bus > lock cannot be released anymore and pending restart handler in turn > fails. > > Thus, to prevent such late watchdog pings from happening ahead of > pending machine restart and consequently locking up the i2c bus, check > for system_state in watchdog ping handler and consequently do not send > pings anymore in case system_state > SYSTEM_RUNNING. > > Signed-off-by: Primoz Fiser <primoz.fiser@norik.com> Reviewed-by: Adam Thomson <Adam.Thomson.Opensource@diasemi.com>
diff --git a/drivers/watchdog/da9062_wdt.c b/drivers/watchdog/da9062_wdt.c index 706fb09c2f24..f02cbd530538 100644 --- a/drivers/watchdog/da9062_wdt.c +++ b/drivers/watchdog/da9062_wdt.c @@ -117,6 +117,13 @@ static int da9062_wdt_ping(struct watchdog_device *wdd) struct da9062_watchdog *wdt = watchdog_get_drvdata(wdd); int ret; + /* + * Prevent pings from occurring late in system poweroff/reboot sequence + * and possibly locking out restart handler from accessing i2c bus. + */ + if (system_state > SYSTEM_RUNNING) + return 0; + ret = da9062_reset_watchdog_timer(wdt); if (ret) dev_err(wdt->hw->dev, "Failed to ping the watchdog (err = %d)\n", diff --git a/drivers/watchdog/da9063_wdt.c b/drivers/watchdog/da9063_wdt.c index 423584252606..d79ce64e26a9 100644 --- a/drivers/watchdog/da9063_wdt.c +++ b/drivers/watchdog/da9063_wdt.c @@ -121,6 +121,13 @@ static int da9063_wdt_ping(struct watchdog_device *wdd) struct da9063 *da9063 = watchdog_get_drvdata(wdd); int ret; + /* + * Prevent pings from occurring late in system poweroff/reboot sequence + * and possibly locking out restart handler from accessing i2c bus. + */ + if (system_state > SYSTEM_RUNNING) + return 0; + ret = regmap_write(da9063->regmap, DA9063_REG_CONTROL_F, DA9063_WATCHDOG); if (ret)
Proper machine resets via da9062/da9063 PMICs are very tricky as they require special i2c atomic transfers when interrupts are not available anymore. This is also a reason why both PMIC's restart handlers do not use regmap but instead opt for i2c_smbus_write_byte_data() which does i2c transfer in atomic manner. Under the hood, this function tries to obtain i2c bus lock with call to i2c_adapter_trylock_bus() which will return -EAGAIN (-11) if lock is not available. Since commit 982bb70517aef ("watchdog: reset last_hw_keepalive time at start") occasional restart handler failures with "Failed to shutdown (err = -11)" error messages were observed, indicating that some process is holding the i2c bus lock. Investigation into the matter uncovered that sometimes during reboot sequence watchdog ping is issued late into poweroff/reboot phase which did not happen before mentioned commit (usually the watchdog ping happened immediately as commit message suggests). As of now, when watchdog ping usually happens late into poweroff/reboot stage when interrupts are not available anymore, i2c bus lock cannot be released anymore and pending restart handler in turn fails. Thus, to prevent such late watchdog pings from happening ahead of pending machine restart and consequently locking up the i2c bus, check for system_state in watchdog ping handler and consequently do not send pings anymore in case system_state > SYSTEM_RUNNING. Signed-off-by: Primoz Fiser <primoz.fiser@norik.com> --- Implemented proposal suggested by Guenter Roeck. Re-spun boards in boot/reboot loop tests and done 5000 cycles on each one with flying colors. Changes in v2: - reduce code complexity by removing reboot notifiers and use system_state variable instead - minor commit message rewording drivers/watchdog/da9062_wdt.c | 7 +++++++ drivers/watchdog/da9063_wdt.c | 7 +++++++ 2 files changed, 14 insertions(+)