From patchwork Thu Jan 23 01:34:56 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 23561 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ie0-f200.google.com (mail-ie0-f200.google.com [209.85.223.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 9E3ED203C6 for ; Thu, 23 Jan 2014 01:35:17 +0000 (UTC) Received: by mail-ie0-f200.google.com with SMTP id tp5sf1233125ieb.7 for ; Wed, 22 Jan 2014 17:35:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:in-reply-to:references:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-unsubscribe; bh=/vZdg5ZZUNwnyPXyhnQn83nD75IG461/fots0bNN3Qo=; b=PuWmeq94JDmEN6peSmSTLZ3zfNzEk3nGL5sof7be9t6El/tR48Lwvj2jC9R6ybXHs/ 7TOewNhDw+curHPqopA6oABcUWVlIWE1BGxcFR0xYGRokXZwuSAwU7R7amFLPpNBUaWa YDCxGoBd7ZG+VDYzZVJKSLcT9RMg6pzRLRUZH1URRXC/V6N6ch29m0ERS5zjZRhFDUFK bq1+R/LEUH+YX+Cs6P0pibLYqfOHfdbLajCRKB4uKDG9Ld5app4p7vtCtbTxdqdD8DTk whQ2WTw1cgboAFflTayL1NhUCXjwOtLPlOQOATydtKnZJ+rHCdyAacHzvO5zasv7a97l EJcw== X-Gm-Message-State: ALoCoQlRdjoSZAzdvQKcqVkMXABXPaN3z5DU4Dqg1t8TfxQGmjzWKnX4AiWj1Db1JGgaNjnF444z X-Received: by 10.42.53.135 with SMTP id n7mr1770760icg.11.1390440916882; Wed, 22 Jan 2014 17:35:16 -0800 (PST) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.96.138 with SMTP id k10ls189845qge.66.gmail; Wed, 22 Jan 2014 17:35:16 -0800 (PST) X-Received: by 10.221.37.1 with SMTP id tc1mr3025292vcb.32.1390440916764; Wed, 22 Jan 2014 17:35:16 -0800 (PST) Received: from mail-vc0-f180.google.com (mail-vc0-f180.google.com [209.85.220.180]) by mx.google.com with ESMTPS id x5si5679081veb.35.2014.01.22.17.35.16 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 22 Jan 2014 17:35:16 -0800 (PST) Received-SPF: neutral (google.com: 209.85.220.180 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.180; Received: by mail-vc0-f180.google.com with SMTP id ks9so691881vcb.39 for ; Wed, 22 Jan 2014 17:35:16 -0800 (PST) X-Received: by 10.220.95.139 with SMTP id d11mr2998685vcn.21.1390440916676; Wed, 22 Jan 2014 17:35:16 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.220.174.196 with SMTP id u4csp224915vcz; Wed, 22 Jan 2014 17:35:16 -0800 (PST) X-Received: by 10.68.195.4 with SMTP id ia4mr5039253pbc.142.1390440915706; Wed, 22 Jan 2014 17:35:15 -0800 (PST) Received: from mail-pb0-f45.google.com (mail-pb0-f45.google.com [209.85.160.45]) by mx.google.com with ESMTPS id pk8si11928659pab.39.2014.01.22.17.35.15 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 22 Jan 2014 17:35:15 -0800 (PST) Received-SPF: neutral (google.com: 209.85.160.45 is neither permitted nor denied by best guess record for domain of john.stultz@linaro.org) client-ip=209.85.160.45; Received: by mail-pb0-f45.google.com with SMTP id un15so1172011pbc.4 for ; Wed, 22 Jan 2014 17:35:15 -0800 (PST) X-Received: by 10.68.218.65 with SMTP id pe1mr4984482pbc.1.1390440914882; Wed, 22 Jan 2014 17:35:14 -0800 (PST) Received: from localhost.localdomain (c-67-170-153-23.hsd1.or.comcast.net. [67.170.153.23]) by mx.google.com with ESMTPSA id j3sm29268483pbh.38.2014.01.22.17.35.12 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 22 Jan 2014 17:35:13 -0800 (PST) From: John Stultz To: stable Cc: John Stultz , Thomas Gleixner , Prarit Bhargava , Richard Cochran , Ingo Molnar , Sasha Levin Subject: [PATCH 3/6] 3.13-stable: timekeeping: Avoid possible deadlock from clock_was_set_delayed Date: Wed, 22 Jan 2014 17:34:56 -0800 Message-Id: <1390440899-10273-4-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1390440899-10273-1-git-send-email-john.stultz@linaro.org> References: <1390440899-10273-1-git-send-email-john.stultz@linaro.org> X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: john.stultz@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.180 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , This is a 3.13-stable backport of 6fdda9a9c5db367130cf32df5d6618d08b89f46a (which includes a small thinko fix that is was already addressed upstream) As part of normal operaions, the hrtimer subsystem frequently calls into the timekeeping code, creating a locking order of hrtimer locks -> timekeeping locks clock_was_set_delayed() was suppoed to allow us to avoid deadlocks between the timekeeping the hrtimer subsystem, so that we could notify the hrtimer subsytem the time had changed while holding the timekeeping locks. This was done by scheduling delayed work that would run later once we were out of the timekeeing code. But unfortunately the lock chains are complex enoguh that in scheduling delayed work, we end up eventually trying to grab an hrtimer lock. Sasha Levin noticed this in testing when the new seqlock lockdep enablement triggered the following (somewhat abrieviated) message: [ 251.100221] ====================================================== [ 251.100221] [ INFO: possible circular locking dependency detected ] [ 251.100221] 3.13.0-rc2-next-20131206-sasha-00005-g8be2375-dirty #4053 Not tainted [ 251.101967] ------------------------------------------------------- [ 251.101967] kworker/10:1/4506 is trying to acquire lock: [ 251.101967] (timekeeper_seq){----..}, at: [] retrigger_next_event+0x56/0x70 [ 251.101967] [ 251.101967] but task is already holding lock: [ 251.101967] (hrtimer_bases.lock#11){-.-...}, at: [] retrigger_next_event+0x3c/0x70 [ 251.101967] [ 251.101967] which lock already depends on the new lock. [ 251.101967] [ 251.101967] [ 251.101967] the existing dependency chain (in reverse order) is: [ 251.101967] -> #5 (hrtimer_bases.lock#11){-.-...}: [snipped] -> #4 (&rt_b->rt_runtime_lock){-.-...}: [snipped] -> #3 (&rq->lock){-.-.-.}: [snipped] -> #2 (&p->pi_lock){-.-.-.}: [snipped] -> #1 (&(&pool->lock)->rlock){-.-...}: [ 251.101967] [] validate_chain+0x6c3/0x7b0 [ 251.101967] [] __lock_acquire+0x4ad/0x580 [ 251.101967] [] lock_acquire+0x182/0x1d0 [ 251.101967] [] _raw_spin_lock+0x40/0x80 [ 251.101967] [] __queue_work+0x1a9/0x3f0 [ 251.101967] [] queue_work_on+0x98/0x120 [ 251.101967] [] clock_was_set_delayed+0x21/0x30 [ 251.101967] [] do_adjtimex+0x111/0x160 [ 251.101967] [] compat_sys_adjtimex+0x41/0x70 [ 251.101967] [] ia32_sysret+0x0/0x5 [ 251.101967] -> #0 (timekeeper_seq){----..}: [snipped] [ 251.101967] other info that might help us debug this: [ 251.101967] [ 251.101967] Chain exists of: timekeeper_seq --> &rt_b->rt_runtime_lock --> hrtimer_bases.lock#11 [ 251.101967] Possible unsafe locking scenario: [ 251.101967] [ 251.101967] CPU0 CPU1 [ 251.101967] ---- ---- [ 251.101967] lock(hrtimer_bases.lock#11); [ 251.101967] lock(&rt_b->rt_runtime_lock); [ 251.101967] lock(hrtimer_bases.lock#11); [ 251.101967] lock(timekeeper_seq); [ 251.101967] [ 251.101967] *** DEADLOCK *** [ 251.101967] [ 251.101967] 3 locks held by kworker/10:1/4506: [ 251.101967] #0: (events){.+.+.+}, at: [] process_one_work+0x200/0x530 [ 251.101967] #1: (hrtimer_work){+.+...}, at: [] process_one_work+0x200/0x530 [ 251.101967] #2: (hrtimer_bases.lock#11){-.-...}, at: [] retrigger_next_event+0x3c/0x70 [ 251.101967] [ 251.101967] stack backtrace: [ 251.101967] CPU: 10 PID: 4506 Comm: kworker/10:1 Not tainted 3.13.0-rc2-next-20131206-sasha-00005-g8be2375-dirty #4053 [ 251.101967] Workqueue: events clock_was_set_work So the best solution is to avoid calling clock_was_set_delayed() while holding the timekeeping lock, and instead using a flag variable to decide if we should call clock_was_set() once we've released the locks. This works for the case here, where the do_adjtimex() was the deadlock trigger point. Unfortuantely, in update_wall_time() we still hold the jiffies lock, which would deadlock with the ipi triggered by clock_was_set(), preventing us from calling it even after we drop the timekeeping lock. So instead call clock_was_set_delayed() at that point. Cc: Thomas Gleixner Cc: Prarit Bhargava Cc: Richard Cochran Cc: Ingo Molnar Cc: Sasha Levin Cc: stable #3.10+ Reported-by: Sasha Levin Tested-by: Sasha Levin Signed-off-by: John Stultz fixup --- kernel/time/timekeeping.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 419faec..89bf18e 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -1278,7 +1278,6 @@ static inline unsigned int accumulate_nsecs_to_secs(struct timekeeper *tk) __timekeeping_set_tai_offset(tk, tk->tai_offset - leap); - clock_was_set_delayed(); clock_set = TK_CLOCK_WAS_SET; } } @@ -1442,6 +1441,19 @@ static void update_wall_time(void) write_seqcount_end(&timekeeper_seq); out: raw_spin_unlock_irqrestore(&timekeeper_lock, flags); + if (clock_set) { + /* + * XXX - I'd rather we just call clock_was_set(), but + * since we're currently holding the jiffies lock, calling + * clock_was_set would trigger an ipi which would then grab + * the jiffies lock and we'd deadlock. :( + * The right solution should probably be droping + * the jiffies lock before calling update_wall_time + * but that requires some rework of the tick sched + * code. + */ + clock_was_set_delayed(); + } } /** @@ -1702,11 +1714,13 @@ int do_adjtimex(struct timex *txc) if (tai != orig_tai) { __timekeeping_set_tai_offset(tk, tai); timekeeping_update(tk, TK_MIRROR | TK_CLOCK_WAS_SET); - clock_was_set_delayed(); } write_seqcount_end(&timekeeper_seq); raw_spin_unlock_irqrestore(&timekeeper_lock, flags); + if (tai != orig_tai) + clock_was_set(); + ntp_notify_cmos_timer(); return ret;