From patchwork Thu Apr 26 10:34:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 134480 Delivered-To: patch@linaro.org Received: by 10.46.151.6 with SMTP id r6csp2083627lji; Thu, 26 Apr 2018 03:38:01 -0700 (PDT) X-Google-Smtp-Source: AIpwx4+Nd2NIb/L0ViMkCk9tTN60IoNzYxQUdZKrg/nWCldTQVm1+r1RZTn+xEAhY1ZTkpS3dUOc X-Received: by 10.98.214.218 with SMTP id a87mr27459733pfl.200.1524739080931; Thu, 26 Apr 2018 03:38:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524739080; cv=none; d=google.com; s=arc-20160816; b=KIBHtNjUchBpfqLfG9Ra/RS9K0n0Uqdsk0c/fz1Ky8YIgTcfk7chGh0eC+6cm/gz0Y jczNPKhb+gPMxc5vjMENdJgPVZk74bAn2ptMzRzf69Kkj0epaXIODs8yq/7tNSolIViV BJ+N2LH0z47JmXIasvgkrzvxCaEuGBjhBXxw10S1Ej2ZrTfH69SWKD6txicCqerIhTX9 42S0+qWhmoEQtkEGHyB/SV8TroxcfUCcjHMxL9XHNKFLm76a7ByawMJFaa9M+FEwYh5E qe6kGD4DOvH3sl0DDygTCDWgYUCVSbrhI+bkffhgUbNqkVnnENbFtX+wb8a+HyyRVo3n 8qJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=kux5n1VJT9ByJWEJc8PHGsikEDL9iGj2a/IhBM5/xMI=; b=Kiop29NISf0k9bWxxJV0mQNBtxXsSDV0hvk6LrWOMALMStMsmB9/BgWe1mnozTjOCz 4lhuRAdjjGGsDuYkTXT+VQes/43wik6kZCw6ze9pno3cjNgbwQaKxTso6Vgbg4mmmMZp JD1bCbuSJ9mdMzpfNgn8/TPMQOgZJKGO4AwwzIPWkOAa0d/G9FDVGLjLV5/yL7XQtuWa fXD7CWDQGdy5HXbWJC8yC5rj8P8jw5qaaxAjmMHMWLdwt4WSmYww03jeMTw+4wPF4qnG UB8wuJmsweL5pbvPcNV5ekcOxghxHIye96I80hb1qzbLvEjfMeXuBj6F0HLBY5vv+cto gx0A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b2si15721607pgc.275.2018.04.26.03.38.00; Thu, 26 Apr 2018 03:38:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754903AbeDZKh6 (ORCPT + 29 others); Thu, 26 Apr 2018 06:37:58 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51054 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755017AbeDZKeK (ORCPT ); Thu, 26 Apr 2018 06:34:10 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4BC6F16BA; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1DE023F5AF; Thu, 26 Apr 2018 03:34:10 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 393801AE5133; Thu, 26 Apr 2018 11:34:29 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, peterz@infradead.org, mingo@kernel.org, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, longman@redhat.com, will.deacon@arm.com Subject: [PATCH v3 06/14] locking/qspinlock: Kill cmpxchg loop when claiming lock from head of queue Date: Thu, 26 Apr 2018 11:34:20 +0100 Message-Id: <1524738868-31318-7-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1524738868-31318-1-git-send-email-will.deacon@arm.com> References: <1524738868-31318-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a queued locker reaches the head of the queue, it claims the lock by setting _Q_LOCKED_VAL in the lockword. If there isn't contention, it must also clear the tail as part of this operation so that subsequent lockers can avoid taking the slowpath altogether. Currently this is expressed as a cmpxchg loop that practically only runs up to two iterations. This is confusing to the reader and unhelpful to the compiler. Rewrite the cmpxchg loop without the loop, so that a failed cmpxchg implies that there is contention and we just need to write to _Q_LOCKED_VAL without considering the rest of the lockword. Cc: Peter Zijlstra Cc: Ingo Molnar Signed-off-by: Will Deacon --- kernel/locking/qspinlock.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) -- 2.1.4 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index ad94b7def005..e42f50d69ed6 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -465,24 +465,21 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) * and nobody is pending, clear the tail code and grab the lock. * Otherwise, we only need to grab the lock. */ - for (;;) { - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { - set_locked(lock); - break; - } + + /* In the PV case we might already have _Q_LOCKED_VAL set */ + if ((val & _Q_TAIL_MASK) == tail) { /* * The smp_cond_load_acquire() call above has provided the - * necessary acquire semantics required for locking. At most - * two iterations of this loop may be ran. + * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); if (old == val) - goto release; /* No contention */ - - val = old; + goto release; /* No contention */ } + /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ + set_locked(lock); + /* * contended path; wait for next if not observed yet, release. */