From patchwork Thu Dec 13 14:09:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 153648 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp846247ljp; Thu, 13 Dec 2018 06:10:06 -0800 (PST) X-Google-Smtp-Source: AFSGD/UX0x9cH98WTrw0Z7HyF8kT+MLm8+JqRoDF14hchGLAE7tMSYKRWSgeYJKcm1HFzJs1cFdE X-Received: by 2002:a17:902:42e4:: with SMTP id h91mr24221264pld.18.1544710206426; Thu, 13 Dec 2018 06:10:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544710206; cv=none; d=google.com; s=arc-20160816; b=OjIzlMdqjWeGzJol1M38R1FViwA8cG1wVjGajwL8w4jzB5jTV2ekPEZQ/VnzAuMvpb 9mLyHO7TaDHD18JBN0s6JnNHjtY+PbPYIjiQ7qBx39XARfJFb/AcwX2MADZnesiCQCvk JcXNml6Lnf9ocYptKUUg29Mn/I8MDMZqwUUj92IeKiuUGAaEJgN5C9Z0xZPhQwxsAjgd g6zv9uyMWbxODMCXZ46WNK8wwha6jDeFBmxHOOqkc3mjODIpN4Hg7kApkDZBYtr9Q8Lz zJnCp8CHFKFMtHY1HZ+MZrzxRq70EJEHbzCHsR0h/0N+79K+MubqBFx/4nQJymi5kYY9 fkFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :illegal-object:reply-to:references:in-reply-to:message-id:date :subject:cc:to:from; bh=0zIfLmGoCvZwqrUpKDEOts4xD3mlmKbhvmQQIwO2IW0=; b=EpUd2WW+fwqmIgBUwLVnsZT027BjHS0AuY80RTFpC0UldGHByMXiSajaJTMhrfLHSv AKZZGIwbpA910y4rJUJyU5m/SyOt0+pwEePxGcyRlhiEjPkUm9Dgl2sZku1cGID3hZUe 1uicfewtfIdDPJ7B5botstn9aBHtFwBYMpwzaCwBKHAmGgYE9t2AaYkhJBKLMA1GFyRj R7BwGc7qGhR+SKgPUfg5DJ7R+4YKLy9+zC3uSVkeeZM767CmMiqeaRKp1oInqpSeBHJJ /myArmNapehxFTE5wVk8a39pI4Emh5u8NIY8LR+hIdz7MaKVeyA1CcZ8GKFsLWwDGvqk n2yA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t16si1817482pfk.139.2018.12.13.06.10.06; Thu, 13 Dec 2018 06:10:06 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728663AbeLMOJt (ORCPT + 15 others); Thu, 13 Dec 2018 09:09:49 -0500 Received: from Galois.linutronix.de ([146.0.238.70]:46534 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728596AbeLMOJt (ORCPT ); Thu, 13 Dec 2018 09:09:49 -0500 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1gXRgF-0001G3-CD; Thu, 13 Dec 2018 15:09:35 +0100 From: Sebastian Andrzej Siewior To: stable@vger.kernel.org Cc: Peter Zijlstra , Will Deacon , Thomas Gleixner , Daniel Wagner , Waiman Long , Linus Torvalds , boqun.feng@gmail.com, linux-arm-kernel@lists.infradead.org, paulmck@linux.vnet.ibm.com, Ingo Molnar , Sebastian Andrzej Siewior Subject: [PATCH STABLE v4.9 03/10] locking/qspinlock: Bound spinning on pending->locked transition in slowpath Date: Thu, 13 Dec 2018 15:09:18 +0100 Message-Id: <20181213140925.6179-4-bigeasy@linutronix.de> X-Mailer: git-send-email 2.20.0 In-Reply-To: <20181213140925.6179-1-bigeasy@linutronix.de> References: <20181213140925.6179-1-bigeasy@linutronix.de> Reply-To: [PATCH STABLE v4.9 00/10], Backport, for, cache line starvation on x86 Illegal-Object: Syntax error in Reply-To: address found on vger.kernel.org: Reply-To: cache line starvation on x86 ^-extraneous tokens in address MIME-Version: 1.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Will Deacon commit 6512276d97b160d90b53285bd06f7f201459a7e3 upstream. If a locker taking the qspinlock slowpath reads a lock value indicating that only the pending bit is set, then it will spin whilst the concurrent pending->locked transition takes effect. Unfortunately, there is no guarantee that such a transition will ever be observed since concurrent lockers could continuously set pending and hand over the lock amongst themselves, leading to starvation. Whilst this would probably resolve in practice, it means that it is not possible to prove liveness properties about the lock and means that lock acquisition time is unbounded. Rather than removing the pending->locked spinning from the slowpath altogether (which has been shown to heavily penalise a 2-threaded locking stress test on x86), this patch replaces the explicit spinning with a call to atomic_cond_read_relaxed and allows the architecture to provide a bound on the number of spins. For architectures that can respond to changes in cacheline state in their smp_cond_load implementation, it should be sufficient to use the default bound of 1. Suggested-by: Waiman Long Signed-off-by: Will Deacon Acked-by: Peter Zijlstra (Intel) Acked-by: Waiman Long Cc: Linus Torvalds Cc: Thomas Gleixner Cc: boqun.feng@gmail.com Cc: linux-arm-kernel@lists.infradead.org Cc: paulmck@linux.vnet.ibm.com Link: http://lkml.kernel.org/r/1524738868-31318-4-git-send-email-will.deacon@arm.com Signed-off-by: Ingo Molnar Signed-off-by: Sebastian Andrzej Siewior --- kernel/locking/qspinlock.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) -- 2.20.0 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 6fce84401dba1..a8da1fc5222eb 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -75,6 +75,18 @@ #define MAX_NODES 4 #endif +/* + * The pending bit spinning loop count. + * This heuristic is used to limit the number of lockword accesses + * made by atomic_cond_read_relaxed when waiting for the lock to + * transition out of the "== _Q_PENDING_VAL" state. We don't spin + * indefinitely because there's no guarantee that we'll make forward + * progress. + */ +#ifndef _Q_PENDING_LOOPS +#define _Q_PENDING_LOOPS 1 +#endif + /* * Per-CPU queue node structures; we can never have more than 4 nested * contexts: task, softirq, hardirq, nmi. @@ -422,13 +434,15 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) return; /* - * wait for in-progress pending->locked hand-overs + * Wait for in-progress pending->locked hand-overs with a bounded + * number of spins so that we guarantee forward progress. * * 0,1,0 -> 0,0,1 */ if (val == _Q_PENDING_VAL) { - while ((val = atomic_read(&lock->val)) == _Q_PENDING_VAL) - cpu_relax(); + int cnt = _Q_PENDING_LOOPS; + val = smp_cond_load_acquire(&lock->val.counter, + (VAL != _Q_PENDING_VAL) || !cnt--); } /*