From patchwork Fri Oct 6 13:34:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 115065 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp1376102edb; Fri, 6 Oct 2017 06:35:06 -0700 (PDT) X-Google-Smtp-Source: AOwi7QA1xcECdOc3LrmlbHyFJqUdpaoSz1BRL647z+iDkTJ5acyitSoYBrR/6z6q2VtjsU/j+mqG X-Received: by 10.99.141.200 with SMTP id z191mr2024157pgd.73.1507296906247; Fri, 06 Oct 2017 06:35:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507296906; cv=none; d=google.com; s=arc-20160816; b=vHBA3cZCyMcHKlHv7SjZQ031y3xXtnerCpq+8aEVZGAu6BzjFIlb32/4w+vNdmbSyC bCvOdnzA/gegLYqZ0BnqldmxfN73k4R62UbXVYea/akGdfYgUUB9nKjfe3WXq2GCaylE gST4Z2yYb1tZjZPLKDuRQcG7tFvr/T/LMNqMUCkEn2/ovzHYFBof4otUOJJEauJJK0br 02JW/wQpaaUD3Hyc8CIjsz3R8U6kbkAGdANS66fXPTNJmZDW2yZx/5Wlh6QtW0wqJjIe cxausHKhRRd4mrs9zNhUQOPvn/4bfvZmGmTu7aqN7QcKuWSB74qW4fEdG4qakSIXo2lt 1zog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=nAGtJbEF/NzH75guqHeIWEpKMq2I+ErlPUctlJpNfyI=; b=UjiwAa7Q8j+MH+gbbvZIrAVDdHTNqxwmr3kULJRs+K5qMMCB5bBS3WoFj96FCXnyiu TyA8YfFtXPB54a7Zeed9H9hG028byY6/UoxR0bjM237gE7ZcMyS+O0CmlLrBt9VVghyP zTzJBezhj92etk5vv6WpiKzg/x5Yq7l6dmAY9A/pLUbJWsPIjO7Kjz3/qzEUAXdTfkMI hWCAzfaulBVdqRd0E6nYwZkOqj0Eo+0z/CIztpJNr0VhC1mqOkUayqjJUiQ9KElNrd9g CwGrIsSVqiH8dZCzggusWe0i3SI4eVrMJfNOj0YlDsy0xaqCyiCGBRB28n6Oj+2rQnf8 Bmfg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y72si1252290pfd.385.2017.10.06.06.35.05; Fri, 06 Oct 2017 06:35:06 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752659AbdJFNev (ORCPT + 26 others); Fri, 6 Oct 2017 09:34:51 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:33642 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752498AbdJFNem (ORCPT ); Fri, 6 Oct 2017 09:34:42 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AF4A91684; Fri, 6 Oct 2017 06:34:42 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8158C3F578; Fri, 6 Oct 2017 06:34:42 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 8F4B91AE2F5E; Fri, 6 Oct 2017 14:34:43 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com, Will Deacon Subject: [PATCH v2 5/5] kernel/locking: Prevent slowpath writers getting held up by fastpath Date: Fri, 6 Oct 2017 14:34:42 +0100 Message-Id: <1507296882-18721-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1507296882-18721-1-git-send-email-will.deacon@arm.com> References: <1507296882-18721-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a prospective writer takes the qrwlock locking slowpath due to the lock being held, it attempts to cmpxchg the wmode field from 0 to _QW_WAITING so that concurrent lockers also take the slowpath and queue on the spinlock accordingly, allowing the lockers to drain. Unfortunately, this isn't fair, because a fastpath writer that comes in after the lock is made available but before the _QW_WAITING flag is set can effectively jump the queue. If there is a steady stream of prospective writers, then the waiter will be held off indefinitely. This patch restores fairness by separating _QW_WAITING and _QW_LOCKED into two distinct fields: _QW_LOCKED continues to occupy the bottom byte of the lockword so that it can be cleared unconditionally when unlocking, but _QW_WAITING now occupies what used to be the bottom bit of the reader count. This then forces the slow-path for concurrent lockers. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Signed-off-by: Will Deacon --- include/asm-generic/qrwlock.h | 10 +++++----- include/asm-generic/qrwlock_types.h | 8 ++++---- kernel/locking/qrwlock.c | 20 +++++--------------- 3 files changed, 14 insertions(+), 24 deletions(-) -- 2.1.4 diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 02c0a768e6b0..63cb7d347b25 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -40,10 +40,10 @@ * | rd | wr | * +----+----+----+----+ */ -#define _QW_WAITING 1 /* A writer is waiting */ -#define _QW_LOCKED 0xff /* A writer holds the lock */ -#define _QW_WMASK 0xff /* Writer mask */ -#define _QR_SHIFT 8 /* Reader count shift */ +#define _QW_WAITING 0x100 /* A writer is waiting */ +#define _QW_LOCKED 0x0ff /* A writer holds the lock */ +#define _QW_WMASK 0x1ff /* Writer mask */ +#define _QR_SHIFT 9 /* Reader count shift */ #define _QR_BIAS (1U << _QR_SHIFT) /* @@ -134,7 +134,7 @@ static inline void queued_read_unlock(struct qrwlock *lock) */ static inline void queued_write_unlock(struct qrwlock *lock) { - smp_store_release(&lock->wmode, 0); + smp_store_release(&lock->wlocked, 0); } /* diff --git a/include/asm-generic/qrwlock_types.h b/include/asm-generic/qrwlock_types.h index 507f2dc51bba..8af752acbdc0 100644 --- a/include/asm-generic/qrwlock_types.h +++ b/include/asm-generic/qrwlock_types.h @@ -13,11 +13,11 @@ typedef struct qrwlock { atomic_t cnts; struct { #ifdef __LITTLE_ENDIAN - u8 wmode; /* Writer mode */ - u8 rcnts[3]; /* Reader counts */ + u8 wlocked; /* Locked for write? */ + u8 __lstate[3]; #else - u8 rcnts[3]; /* Reader counts */ - u8 wmode; /* Writer mode */ + u8 __lstate[3]; + u8 wlocked; /* Locked for write? */ #endif }; }; diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index b7ea4647c74d..e940f2c2b4f2 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -40,8 +40,7 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * so spin with ACQUIRE semantics until the lock is available * without waiting in the queue. */ - atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) - != _QW_LOCKED); + atomic_cond_read_acquire(&lock->cnts, !(VAL & _QW_LOCKED)); return; } atomic_sub(_QR_BIAS, &lock->cnts); @@ -57,7 +56,7 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) * that accesses can't leak upwards out of our subsequent critical * section in the case that the lock is currently held for write. */ - atomic_cond_read_acquire(&lock->cnts, (VAL & _QW_WMASK) != _QW_LOCKED); + atomic_cond_read_acquire(&lock->cnts, !(VAL & _QW_LOCKED)); /* * Signal the next one in queue to become queue head @@ -80,19 +79,10 @@ void queued_write_lock_slowpath(struct qrwlock *lock) (atomic_cmpxchg_acquire(&lock->cnts, 0, _QW_LOCKED) == 0)) goto unlock; - /* - * Set the waiting flag to notify readers that a writer is pending, - * or wait for a previous writer to go away. - */ - for (;;) { - if (!READ_ONCE(lock->wmode) && - (cmpxchg_relaxed(&lock->wmode, 0, _QW_WAITING) == 0)) - break; - - cpu_relax(); - } + /* Set the waiting flag to notify readers that a writer is pending */ + atomic_add(_QW_WAITING, &lock->cnts); - /* When no more readers, set the locked flag */ + /* When no more readers or writers, set the locked flag */ do { atomic_cond_read_acquire(&lock->cnts, VAL == _QW_WAITING); } while (atomic_cmpxchg_relaxed(&lock->cnts, _QW_WAITING,