From patchwork Tue Apr 26 11:00:00 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 66684 Delivered-To: patch@linaro.org Received: by 10.140.93.198 with SMTP id d64csp1538609qge; Tue, 26 Apr 2016 04:02:06 -0700 (PDT) X-Received: by 10.66.177.75 with SMTP id co11mr2583656pac.85.1461668526424; Tue, 26 Apr 2016 04:02:06 -0700 (PDT) Return-Path: Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id w127si2830645pfb.38.2016.04.26.04.02.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Apr 2016 04:02:06 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) client-ip=2001:1868:205::9; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) smtp.mailfrom=linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1av0jB-0007eC-0G; Tue, 26 Apr 2016 11:00:25 +0000 Received: from cam-admin0.cambridge.arm.com ([217.140.96.50]) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1av0j7-0006MV-Li for linux-arm-kernel@lists.infradead.org; Tue, 26 Apr 2016 11:00:22 +0000 Received: from edgewater-inn.cambridge.arm.com (edgewater-inn.cambridge.arm.com [10.1.203.121]) by cam-admin0.cambridge.arm.com (8.12.6/8.12.6) with ESMTP id u3QAxwWr029027; Tue, 26 Apr 2016 11:59:58 +0100 (BST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 305921AE3577; Tue, 26 Apr 2016 12:00:01 +0100 (BST) From: Will Deacon To: marc.zyngier@arm.com Subject: [PATCH] irqchip/gic: ensure ordering between read of INTACK and shared data Date: Tue, 26 Apr 2016 12:00:00 +0100 Message-Id: <1461668400-22564-1-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160426_040022_031097_B34C23FA X-CRM114-Status: GOOD ( 15.81 ) X-Spam-Score: -7.9 (-------) X-Spam-Report: SpamAssassin version 3.4.0 on bombadil.infradead.org summary: Content analysis details: (-7.9 points) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain -0.0 SPF_PASS SPF: sender matches SPF record -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high trust [217.140.96.50 listed in list.dnswl.org] -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Will Deacon , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org When an IPI is generated by a CPU, the pattern looks roughly like: smp_wmb(); On the receiving CPU we rely on the fact that, once we've taken the interrupt, then the freshly written shared data must be visible to us. Put another way, the CPU isn't going to speculate taking an interrupt. Unfortunately, this assumption turns out to be broken. Consider that CPUx wants to send an IPI to CPUy, which will cause CPUy to read some shared_data. Before CPUx has done anything, a random peripheral raises an IRQ to the GIC and the IRQ line on CPUy is raised. CPUy then takes the IRQ and starts executing the entry code, heading towards gic_handle_irq. Furthermore, let's assume that a bunch of the previous interrupts handled by CPUy were SGIs, so the branch predictor kicks in and speculates that irqnr will be <16 and we're likely to head into handle_IPI. The prefetcher then grabs a speculative copy of shared_data which contains a stale value. Meanwhile, CPUx gets round to updating shared_data and asking the GIC to send an SGI to CPUy. Internally, the GIC decides that the SGI is more important than the peripheral interrupt (which hasn't yet been ACKed) but doesn't need to do anything to CPUy, because the IRQ line is already raised. CPUy then reads the ACK register on the GIC, sees the SGI value which confirms the branch prediction and we end up with a stale shared_data value. This patch fixes the problem by adding an smp_rmb() to the IPI entry code in gic_handle_irq. As it turns out, the combination of a control dependency and an ISB instruction from the EOI in the GICv3 driver is enough to provide the ordering we need, so we add a comment there justifying the absence of an explicit smp_rmb(). Signed-off-by: Will Deacon --- drivers/irqchip/irq-gic-v3.c | 7 +++++++ drivers/irqchip/irq-gic.c | 8 ++++++++ 2 files changed, 15 insertions(+) -- 2.1.4 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index 5b7d3c2129d8..9052a0f2d2df 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -364,6 +364,13 @@ static asmlinkage void __exception_irq_entry gic_handle_irq(struct pt_regs *regs if (static_key_true(&supports_deactivate)) gic_write_dir(irqnr); #ifdef CONFIG_SMP + /* + * Unlike GICv2, we don't need an smp_rmb() here. + * The control dependency from gic_read_iar to + * the ISB in gic_write_eoir is enough to ensure + * that any shared data read by handle_IPI will + * be read after the ACK. + */ handle_IPI(irqnr, regs); #else WARN_ONCE(true, "Unexpected SGI received!\n"); diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c index 282344b95ec2..5c4da5808b15 100644 --- a/drivers/irqchip/irq-gic.c +++ b/drivers/irqchip/irq-gic.c @@ -344,6 +344,14 @@ static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs) if (static_key_true(&supports_deactivate)) writel_relaxed(irqstat, cpu_base + GIC_CPU_DEACTIVATE); #ifdef CONFIG_SMP + /* + * Ensure any shared data written by the CPU sending + * the IPI is read after we've read the ACK register + * on the GIC. + * + * Pairs with the write barrier in gic_raise_softirq + */ + smp_rmb(); handle_IPI(irqnr, regs); #endif continue;