From patchwork Tue Jul 10 22:43:25 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: john stultz X-Patchwork-Id: 9941 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id DBA8623E57 for ; Tue, 10 Jul 2012 22:46:19 +0000 (UTC) Received: from mail-yx0-f180.google.com (mail-yx0-f180.google.com [209.85.213.180]) by fiordland.canonical.com (Postfix) with ESMTP id 95CE9A18A3E for ; Tue, 10 Jul 2012 22:46:19 +0000 (UTC) Received: by yenq6 with SMTP id q6so657868yen.11 for ; Tue, 10 Jul 2012 15:46:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf:from:to:cc :subject:date:message-id:x-mailer:in-reply-to:references :x-content-scanned:x-cbid:x-gm-message-state; bh=H2CSYfo6FswXu/mi/IzBHmSM8cDu6/qn3MW6NQ2w8BU=; b=XB9jdIeOfwLPiPWFzQArsO+qQiEz0DwAqJ7+3kY0ElqxbSGfdOaQP7MWvgJnCQO4FI ZtEzfGBFGbxEDfE+4/wxUhTr/F5Xw5ybfQSX3N/GLArMI6PVH0v1iYOEt0Sp0GuuNZbJ dCRG6WDKi7RLobWl9yD59pW3tbEGTnA7+gz0ArvHysqBeCv3en7wi5Dcou4HZsVWAVZ8 lLyqQZxXtKZPgoN52QdkLjZ7iWeGEZGeK2IkcpQ+O+V44Ik/xyqYZ4oJzINFdjWkHaNv WbDJRkW2f944u+QX14ZR/8R/22El6jhJDL1DFnMofB156nWrh7mtXJmS5sn9EAWplUBM oGvw== Received: by 10.50.57.167 with SMTP id j7mr12569396igq.53.1341960368717; Tue, 10 Jul 2012 15:46:08 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.235.202 with SMTP id kh10csp1509ibb; Tue, 10 Jul 2012 15:46:08 -0700 (PDT) Received: by 10.68.134.201 with SMTP id pm9mr74618561pbb.49.1341960367456; Tue, 10 Jul 2012 15:46:07 -0700 (PDT) Received: from e39.co.us.ibm.com (e39.co.us.ibm.com. [32.97.110.160]) by mx.google.com with ESMTPS id wh5si946750pbc.128.2012.07.10.15.46.07 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 10 Jul 2012 15:46:07 -0700 (PDT) Received-SPF: pass (google.com: domain of johnstul@us.ibm.com designates 32.97.110.160 as permitted sender) client-ip=32.97.110.160; Authentication-Results: mx.google.com; spf=pass (google.com: domain of johnstul@us.ibm.com designates 32.97.110.160 as permitted sender) smtp.mail=johnstul@us.ibm.com Received: from /spool/local by e39.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 10 Jul 2012 16:46:06 -0600 Received: from d03dlp03.boulder.ibm.com (9.17.202.179) by e39.co.us.ibm.com (192.168.1.139) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 10 Jul 2012 16:45:06 -0600 Received: from d03relay04.boulder.ibm.com (d03relay04.boulder.ibm.com [9.17.195.106]) by d03dlp03.boulder.ibm.com (Postfix) with ESMTP id 36AB119D8048 for ; Tue, 10 Jul 2012 22:44:39 +0000 (WET) Received: from d03av06.boulder.ibm.com (d03av06.boulder.ibm.com [9.17.195.245]) by d03relay04.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q6AMi2TD252118 for ; Tue, 10 Jul 2012 16:44:23 -0600 Received: from d03av06.boulder.ibm.com (loopback [127.0.0.1]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q6AMiqWD013714 for ; Tue, 10 Jul 2012 16:44:52 -0600 Received: from kernel.stglabs.ibm.com ([9.114.214.19]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q6AMiji8013399; Tue, 10 Jul 2012 16:44:51 -0600 From: John Stultz To: Linux Kernel Cc: John Stultz , Ingo Molnar , Peter Zijlstra , Prarit Bhargava , Thomas Gleixner , stable@vger.kernel.org Subject: [PATCH 6/6] hrtimer: Update hrtimer base offsets each hrtimer_interrupt Date: Tue, 10 Jul 2012 18:43:25 -0400 Message-Id: <1341960205-56738-8-git-send-email-johnstul@us.ibm.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1341960205-56738-1-git-send-email-johnstul@us.ibm.com> References: <1341960205-56738-1-git-send-email-johnstul@us.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12071022-4242-0000-0000-00000247500F X-Gm-Message-State: ALoCoQlPQVA6ZfnPU9xvGp54aJzGzjHpUCjYqYsJsyQ26s9ikLGcDExLZpkFZSJqmaPeIHuk4CEa The update of the hrtimer base offsets on all cpus cannot be made atomically from the timekeeper.lock held and interrupt disabled region as smp function calls are not allowed there. clock_was_set(), which enforces the update on all cpus, is called either from preemptible process context in case of do_settimeofday() or from the softirq context when the offset modification happened in the timer interrupt itself due to a leap second. In both cases there is a race window for an hrtimer interrupt between dropping timekeeper lock, enabling interrupts and clock_was_set() issuing the updates. Any interrupt which arrives in that window will see the new time but operate on stale offsets. So we need to make sure that an hrtimer interrupt always sees a consistent state of time and offsets. ktime_get_update_offsets() allows us to get the current monotonic time and update the per cpu hrtimer base offsets from hrtimer_interrupt() to capture a consistent state of monotonic time and the offsets. The function replaces the existing ktime_get() calls in hrtimer_interrupt(). The overhead of the new function vs. ktime_get() is minimal as it just adds two store operations. This ensures that any changes to realtime or boottime offsets are noticed and stored into the per-cpu hrtimer base structures, prior to any hrtimer expiration and guarantees that timers are not expired early. CC: Ingo Molnar CC: Peter Zijlstra CC: Prarit Bhargava CC: Thomas Gleixner CC: stable@vger.kernel.org Signed-off-by: John Stultz Signed-off-by: Thomas Gleixner Signed-off-by: John Stultz --- kernel/hrtimer.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/kernel/hrtimer.c b/kernel/hrtimer.c index 14a260a..669522b 100644 --- a/kernel/hrtimer.c +++ b/kernel/hrtimer.c @@ -657,6 +657,14 @@ static inline int hrtimer_enqueue_reprogram(struct hrtimer *timer, return 0; } +static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base) +{ + ktime_t *offs_real = &base->clock_base[HRTIMER_BASE_REALTIME].offset; + ktime_t *offs_boot = &base->clock_base[HRTIMER_BASE_BOOTTIME].offset; + + return ktime_get_update_offsets(offs_real, offs_boot); +} + /* * Retrigger next event is called after clock was set * @@ -665,22 +673,12 @@ static inline int hrtimer_enqueue_reprogram(struct hrtimer *timer, static void retrigger_next_event(void *arg) { struct hrtimer_cpu_base *base = &__get_cpu_var(hrtimer_bases); - struct timespec realtime_offset, xtim, wtm, sleep; if (!hrtimer_hres_active()) return; - /* Optimized out for !HIGH_RES */ - get_xtime_and_monotonic_and_sleep_offset(&xtim, &wtm, &sleep); - set_normalized_timespec(&realtime_offset, -wtm.tv_sec, -wtm.tv_nsec); - - /* Adjust CLOCK_REALTIME offset */ raw_spin_lock(&base->lock); - base->clock_base[HRTIMER_BASE_REALTIME].offset = - timespec_to_ktime(realtime_offset); - base->clock_base[HRTIMER_BASE_BOOTTIME].offset = - timespec_to_ktime(sleep); - + hrtimer_update_base(base); hrtimer_force_reprogram(base, 0); raw_spin_unlock(&base->lock); } @@ -710,7 +708,6 @@ static int hrtimer_switch_to_hres(void) base->clock_base[i].resolution = KTIME_HIGH_RES; tick_setup_sched_timer(); - /* "Retrigger" the interrupt to get things going */ retrigger_next_event(NULL); local_irq_restore(flags); @@ -1264,7 +1261,7 @@ void hrtimer_interrupt(struct clock_event_device *dev) dev->next_event.tv64 = KTIME_MAX; raw_spin_lock(&cpu_base->lock); - entry_time = now = ktime_get(); + entry_time = now = hrtimer_update_base(cpu_base); retry: expires_next.tv64 = KTIME_MAX; /* @@ -1342,9 +1339,12 @@ retry: * We need to prevent that we loop forever in the hrtimer * interrupt routine. We give it 3 attempts to avoid * overreacting on some spurious event. + * + * Acquire base lock for updating the offsets and retrieving + * the current time. */ raw_spin_lock(&cpu_base->lock); - now = ktime_get(); + now = hrtimer_update_base(cpu_base); cpu_base->nr_retries++; if (++retries < 3) goto retry;