From patchwork Wed Apr 11 18:01:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 133158 Delivered-To: patch@linaro.org Received: by 10.46.84.29 with SMTP id i29csp895693ljb; Wed, 11 Apr 2018 11:01:51 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+3zeRsRWL1rXk0fP+hTt3PXNBrXLiuMSuQxMkG2oUhYFvuBwS2AlscIQ8OfVBUN6QR0m9l X-Received: by 10.98.189.14 with SMTP id a14mr4947759pff.30.1523469711131; Wed, 11 Apr 2018 11:01:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523469711; cv=none; d=google.com; s=arc-20160816; b=kxS/LqDz/gmFB4bwiZJ+wJ0JeLNGa2I95me94Kfww6olgCka0X2BPR9A4GtmzBHrGm XIvx7AON1K0+9F+b4M95cgH+SEMO9yojq2aJILn6U3QcrMhyX/bfU1dvwbpD4Uwy5uuO 7YFHFxqrIeLKlUQ8w5U6kXkzanx3sbq+tpN8d1G6y1fg4FKF9w3GxN2dBEzaB3dEtlkx Zs/ktWLyeSLCbVg6UMA1FBNQ8t/V8IFLpGWGNJeq2c7wXIohDiokyxat8zz7X4wGJMDQ Ue2La3nXGOLwagCXwNJ972Lvrj+Ax7RBdsdUNa1UYFWvEFGFfJ8TdP4v6YaPMHASo4LX 24Lg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xCwX6s17JMme4myPU+G2D7lG238iBzymvDdYZhpI09E=; b=RqJZDtspApeFr1pfjl+28LyBasNulqr1tiRjwHd2dLJuHgop1PeR+vxB/YGB7sth0d lnsAYM8HWYEk0G5uAGq5XqhqTLKQwJD79NgWGq1hppubE9hiF/NYgkKVsEXX0/qy5BUU f6A4YGQwpHiA89iUNYazHrx0djN5JTQDKeR7jhzhSrenofCe3lsnGTl97xOi8TU3RfnM 0p7XppTBP1SSYU9E3WrqDSFq5Uz6d0aasr6WdsA5Ob4uWWp/izQyoWKn4wQYUb864ID5 rdqfwhgVAQ/3W5O8rSQLjGpi/8QqhMgigi8AIdTYZSyRLJ2Gvm7jkJCR7hZazoclNdDj zG7A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p20si1258214pfi.345.2018.04.11.11.01.50; Wed, 11 Apr 2018 11:01:51 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753584AbeDKSBP (ORCPT + 29 others); Wed, 11 Apr 2018 14:01:15 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:52128 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752588AbeDKSBH (ORCPT ); Wed, 11 Apr 2018 14:01:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 054B21650; Wed, 11 Apr 2018 11:01:07 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id CB88C3F487; Wed, 11 Apr 2018 11:01:06 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 928FC1AE55BA; Wed, 11 Apr 2018 19:01:21 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, Will Deacon Subject: [PATCH v2 05/13] locking/qspinlock: Kill cmpxchg loop when claiming lock from head of queue Date: Wed, 11 Apr 2018 19:01:12 +0100 Message-Id: <1523469680-17699-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1523469680-17699-1-git-send-email-will.deacon@arm.com> References: <1523469680-17699-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a queued locker reaches the head of the queue, it claims the lock by setting _Q_LOCKED_VAL in the lockword. If there isn't contention, it must also clear the tail as part of this operation so that subsequent lockers can avoid taking the slowpath altogether. Currently this is expressed as a cmpxchg loop that practically only runs up to two iterations. This is confusing to the reader and unhelpful to the compiler. Rewrite the cmpxchg loop without the loop, so that a failed cmpxchg implies that there is contention and we just need to write to _Q_LOCKED_VAL without considering the rest of the lockword. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index a8fc402b3f3a..01b660442d87 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -505,24 +505,21 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * and nobody is pending, clear the tail code and grab the lock. * Otherwise, we only need to grab the lock. */ - for (;;) { - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { - set_locked(lock); - break; - } + + /* In the PV case we might already have _Q_LOCKED_VAL set */ + if ((val & _Q_TAIL_MASK) == tail) { /* * The smp_cond_load_acquire() call above has provided the - * necessary acquire semantics required for locking. At most - * two iterations of this loop may be ran. + * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); if (old == val) - goto release; /* No contention */ - - val = old; + goto release; /* No contention */ } + /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ + set_locked(lock); + /* * contended path; wait for next if not observed yet, release. */