From patchwork Tue Jan 28 16:51:18 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Campbell X-Patchwork-Id: 23803 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-qa0-f70.google.com (mail-qa0-f70.google.com [209.85.216.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 7F7B1202B2 for ; Tue, 28 Jan 2014 16:53:28 +0000 (UTC) Received: by mail-qa0-f70.google.com with SMTP id ii20sf1182511qab.9 for ; Tue, 28 Jan 2014 08:53:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id :mime-version:cc:subject:precedence:list-id:list-unsubscribe :list-post:list-help:list-subscribe:sender:errors-to :x-original-sender:x-original-authentication-results:mailing-list :list-archive:content-type:content-transfer-encoding; bh=GHTWqxJGZ0Rdp3VBTXaWnLadh6ygLbAIPsc/IMwOrNg=; b=JImlhW+FSvLwacVKbqeKdIESvYMCiC7ZeFyQ9RgtGmtvp8NSUWxSMcS+s97JG/F2wX PgIvrvBTgYyXoHzzPf4lqotpxbme2ZBkvCb+oT0LgfdB3V5u4IrEpVe0LgwBVqgJarc3 SgTRzL1bHmydX1m5Ri6LHSGcZWsLZ4W2nIFH1Ms0ym5tSetWgIdqa6OnRjzTTt/hwRC0 V9QGQwq8QfDpzUfQgvHaD4DWLwukLbdEEKkBfb60wuUizEYoVIkr6UmcB2aB4h4+7/ho t+XSS9HKec5+wPC8kyyVie+AO/BQeJgpe0MdxoTSxAKgOYZVwYMsuifMu8TaupWlRUwL V6Nw== X-Gm-Message-State: ALoCoQkmGoTTk2VGOZlDquS3KisaWku0YKokGLxhgRm9SDJL92pOTqY//+yuCqlIb42PRtnsOtSP X-Received: by 10.236.69.35 with SMTP id m23mr826919yhd.6.1390928007174; Tue, 28 Jan 2014 08:53:27 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.103.118 with SMTP id x109ls2092606qge.43.gmail; Tue, 28 Jan 2014 08:53:27 -0800 (PST) X-Received: by 10.52.232.168 with SMTP id tp8mr386393vdc.38.1390928007099; Tue, 28 Jan 2014 08:53:27 -0800 (PST) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx.google.com with ESMTPS id eo4si791574vdb.147.2014.01.28.08.53.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 28 Jan 2014 08:53:27 -0800 (PST) Received-SPF: neutral (google.com: 209.85.220.182 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.182; Received: by mail-vc0-f182.google.com with SMTP id id10so420197vcb.13 for ; Tue, 28 Jan 2014 08:53:27 -0800 (PST) X-Received: by 10.220.251.73 with SMTP id mr9mr1818060vcb.2.1390928006965; Tue, 28 Jan 2014 08:53:26 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.174.196 with SMTP id u4csp45473vcz; Tue, 28 Jan 2014 08:53:26 -0800 (PST) X-Received: by 10.229.171.8 with SMTP id f8mr4139693qcz.13.1390928006405; Tue, 28 Jan 2014 08:53:26 -0800 (PST) Received: from lists.xen.org (lists.xen.org. [50.57.142.19]) by mx.google.com with ESMTPS id t7si11706436qar.187.2014.01.28.08.53.26 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 28 Jan 2014 08:53:26 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of xen-devel-bounces@lists.xen.org designates 50.57.142.19 as permitted sender) client-ip=50.57.142.19; Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1W8Bsi-0003O3-W0; Tue, 28 Jan 2014 16:51:24 +0000 Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1W8Bsh-0003Ny-NW for xen-devel@lists.xen.org; Tue, 28 Jan 2014 16:51:24 +0000 Received: from [85.158.143.35:13604] by server-3.bemta-4.messagelabs.com id CB/AA-32360-B00E7E25; Tue, 28 Jan 2014 16:51:23 +0000 X-Env-Sender: Ian.Campbell@citrix.com X-Msg-Ref: server-6.tower-21.messagelabs.com!1390927881!1416787!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 7162 invoked from network); 28 Jan 2014 16:51:22 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-6.tower-21.messagelabs.com with RC4-SHA encrypted SMTP; 28 Jan 2014 16:51:22 -0000 X-IronPort-AV: E=Sophos;i="4.95,736,1384300800"; d="scan'208";a="95329217" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 28 Jan 2014 16:51:20 +0000 Received: from norwich.cam.xci-test.com (10.80.248.129) by smtprelay.citrix.com (10.13.107.80) with Microsoft SMTP Server id 14.2.342.4; Tue, 28 Jan 2014 11:51:19 -0500 Received: from drall.uk.xensource.com ([10.80.16.71] helo=drall.uk.xensource.com.) by norwich.cam.xci-test.com with esmtp (Exim 4.72) (envelope-from ) id 1W8Bsc-0003GL-K1; Tue, 28 Jan 2014 16:51:18 +0000 From: Ian Campbell To: Date: Tue, 28 Jan 2014 16:51:18 +0000 Message-ID: <1390927878-7048-1-git-send-email-ian.campbell@citrix.com> X-Mailer: git-send-email 1.7.10.4 MIME-Version: 1.0 X-DLP: MIA1 Cc: Ian Campbell , stefano.stabellini@eu.citrix.com, julien.grall@linaro.org, tim@xen.org, george.dunlap@citrix.com, Oleksandr Tyshchenko Subject: [Xen-devel] [PATCH] xen: arm: increase priority of SGIs used as IPIs X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Post: , List-Help: , List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ian.campbell@citrix.com X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.182 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Archive: Code such as on_selected_cpus expects/requires that an IPI can preempt a processor which is just handling a normal interrupt. Lacking this property can result in a deadlock between two CPUs trying to IPI each other from interrupt context. For the time being there is only two priorities, IRQ and IPI, although it is also conceivable that in the future some IPIs might be higher priority than others. This could be used to implement a better BUG() than we have now, but I haven't tackled that yet. Tested with a debug patch which sends a local IPI from a keyhandler, which is run in serial interrupt context. This should also fix the issue reported by Oleksandr in "xen/arm: maintenance_interrupt SMP fix" without resorting to trylock. Signed-off-by: Ian Campbell Cc: Oleksandr Tyshchenko Acked-by: Julien Grall --- I think this is probably 4.5 material at this point. Tested with "HACK: dump pcpu state keyhandler" which I'll post for completeness. It gives: (XEN) Xen call trace: (XEN) [<0000000000212048>] dump_pcpus+0x28/0x2c (PC) (XEN) [<000000000021256c>] handle_keypress+0x70/0xb0 (LR) (XEN) [<000000000023ed00>] __serial_rx+0x20/0x6c (XEN) [<000000000023f8ac>] serial_rx+0xb4/0xc4 (XEN) [<00000000002409ec>] serial_rx_interrupt+0xb0/0xd4 (XEN) [<00000000002404b4>] ns16550_interrupt+0x6c/0x90 (XEN) [<0000000000245fc0>] do_IRQ+0x144/0x1b4 (XEN) [<0000000000245a28>] gic_interrupt+0x60/0xf8 (XEN) [<000000000024be64>] do_trap_irq+0x10/0x18 (XEN) [<000000000024e240>] hyp_irq+0x5c/0x60 (XEN) [<0000000000249324>] init_done+0x10/0x18 (XEN) [<0000000000000080>] 0000000000000080 --- xen/arch/arm/gic.c | 19 +++++++++++++------ xen/arch/arm/time.c | 6 +++--- xen/include/asm-arm/gic.h | 22 ++++++++++++++++++++++ 3 files changed, 38 insertions(+), 9 deletions(-) diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c index dcf9cd4..ee37019 100644 --- a/xen/arch/arm/gic.c +++ b/xen/arch/arm/gic.c @@ -319,7 +319,8 @@ static void __init gic_dist_init(void) /* Default priority for global interrupts */ for ( i = 32; i < gic.lines; i += 4 ) - GICD[GICD_IPRIORITYR + i / 4] = 0xa0a0a0a0; + GICD[GICD_IPRIORITYR + i / 4] = + GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ; /* Disable all global interrupts */ for ( i = 32; i < gic.lines; i += 32 ) @@ -341,8 +342,12 @@ static void __cpuinit gic_cpu_init(void) GICD[GICD_ICENABLER] = 0xffff0000; /* Disable all PPI */ GICD[GICD_ISENABLER] = 0x0000ffff; /* Enable all SGI */ /* Set PPI and SGI priorities */ - for (i = 0; i < 32; i += 4) - GICD[GICD_IPRIORITYR + i / 4] = 0xa0a0a0a0; + for (i = 0; i < 16; i += 4) + GICD[GICD_IPRIORITYR + i / 4] = + GIC_PRI_IPI<<24 | GIC_PRI_IPI<<16 | GIC_PRI_IPI<<8 | GIC_PRI_IPI; + for (i = 16; i < 32; i += 4) + GICD[GICD_IPRIORITYR + i / 4] = + GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ; /* Local settings: interface controller */ GICC[GICC_PMR] = 0xff; /* Don't mask by priority */ @@ -538,7 +543,8 @@ void gic_disable_cpu(void) void gic_route_ppis(void) { /* GIC maintenance */ - gic_route_dt_irq(&gic.maintenance, cpumask_of(smp_processor_id()), 0xa0); + gic_route_dt_irq(&gic.maintenance, cpumask_of(smp_processor_id()), + GIC_PRI_IRQ); /* Route timer interrupt */ route_timer_interrupt(); } @@ -553,7 +559,8 @@ void gic_route_spis(void) if ( (irq = serial_dt_irq(seridx)) == NULL ) continue; - gic_route_dt_irq(irq, cpumask_of(smp_processor_id()), 0xa0); + gic_route_dt_irq(irq, cpumask_of(smp_processor_id()), + GIC_PRI_IRQ); } } @@ -777,7 +784,7 @@ int gic_route_irq_to_guest(struct domain *d, const struct dt_irq *irq, level = dt_irq_is_level_triggered(irq); gic_set_irq_properties(irq->irq, level, cpumask_of(smp_processor_id()), - 0xa0); + GIC_PRI_IRQ); retval = __setup_irq(desc, irq->irq, action); if (retval) { diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c index 81e3e28..68b939d 100644 --- a/xen/arch/arm/time.c +++ b/xen/arch/arm/time.c @@ -222,11 +222,11 @@ static void vtimer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs) void __cpuinit route_timer_interrupt(void) { gic_route_dt_irq(&timer_irq[TIMER_PHYS_NONSECURE_PPI], - cpumask_of(smp_processor_id()), 0xa0); + cpumask_of(smp_processor_id()), GIC_PRI_IRQ); gic_route_dt_irq(&timer_irq[TIMER_HYP_PPI], - cpumask_of(smp_processor_id()), 0xa0); + cpumask_of(smp_processor_id()), GIC_PRI_IRQ); gic_route_dt_irq(&timer_irq[TIMER_VIRT_PPI], - cpumask_of(smp_processor_id()), 0xa0); + cpumask_of(smp_processor_id()), GIC_PRI_IRQ); } /* Set up the timer interrupt on this CPU */ diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h index 9c6f9bb..25b2b24 100644 --- a/xen/include/asm-arm/gic.h +++ b/xen/include/asm-arm/gic.h @@ -129,6 +129,28 @@ #define GICH_LR_CPUID_SHIFT 9 #define GICH_VTR_NRLRGS 0x3f +/* + * The minimum GICC_BPR is required to be in the range 0-3. We set + * GICC_BPR to 0 but we must expect that it might be 3. This means we + * can rely on premption between the following ranges: + * 0xf0..0xff + * 0xe0..0xdf + * 0xc0..0xcf + * 0xb0..0xbf + * 0xa0..0xaf + * 0x90..0x9f + * 0x80..0x8f + * + * Priorities within a range will not preempt each other. + * + * A GIC must support a mimimum of 16 priority levels. + */ +#define GIC_PRI_LOWEST 0xf0 +#define GIC_PRI_IRQ 0xa0 +#define GIC_PRI_IPI 0x90 /* IPIs must preempt normal interrupts */ +#define GIC_PRI_HIGHEST 0x80 /* Higher priorities belong to Secure-World */ + + #ifndef __ASSEMBLY__ #include