From patchwork Tue May 27 12:11:51 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefano Stabellini X-Patchwork-Id: 30972 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-vc0-f200.google.com (mail-vc0-f200.google.com [209.85.220.200]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 5FF64203C3 for ; Tue, 27 May 2014 12:14:33 +0000 (UTC) Received: by mail-vc0-f200.google.com with SMTP id lc6sf34123816vcb.11 for ; Tue, 27 May 2014 05:14:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:date:from:to:in-reply-to:message-id :references:user-agent:mime-version:cc:subject:precedence:list-id :list-unsubscribe:list-post:list-help:list-subscribe:sender :errors-to:x-original-sender:x-original-authentication-results :mailing-list:list-archive:content-type:content-id; bh=YyoaPgBjcukwN2x2A5yaZDwNJr7Uz+4qLUUdkDhvqZU=; b=VA3lZcYci1PKHnghPYhcP84bv6zcnJvVjjRlxaTAge/yp+8gJwiufijEfO7Ns2LlGw YnDBNJJGnQ2OQi1YygM/kvtRf4OT+9fM9y6NhvL31A5TkIOz2HinFYM6EJKYQE0g8NXm hoZcbW/3vC0YXs3DaYBV8hapIuWXSqcE4AguqnaI0FS73BpsNEEALx7wrgaQGgogjjDd LJSPDT2S5jvY8zZ618avcpgvQ+wG6fxVwZqA1ICae8hziScL48FdOnnaheXMxN6E2kzb 6mw1ah8uEnXT0jGr+CMhPvg8KDdmemvVWS/Y09APV7tU0fJ4DpMh6Yi3VBZwpJ5hLuNT UHyQ== X-Gm-Message-State: ALoCoQnrkfJRv5y0bFXHi1PoCiUS8/Vn9LxTidAn0hMNV8I68WH5boybN4jwftXmj1Ch9wiSb9x9 X-Received: by 10.224.90.2 with SMTP id g2mr12951074qam.1.1401192873709; Tue, 27 May 2014 05:14:33 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.97.200 with SMTP id m66ls3034503qge.83.gmail; Tue, 27 May 2014 05:14:33 -0700 (PDT) X-Received: by 10.221.66.131 with SMTP id xq3mr462846vcb.43.1401192873573; Tue, 27 May 2014 05:14:33 -0700 (PDT) Received: from mail-vc0-f178.google.com (mail-vc0-f178.google.com [209.85.220.178]) by mx.google.com with ESMTPS id sk5si7976018vec.60.2014.05.27.05.14.33 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 27 May 2014 05:14:33 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.178 as permitted sender) client-ip=209.85.220.178; Received: by mail-vc0-f178.google.com with SMTP id ij19so7017976vcb.23 for ; Tue, 27 May 2014 05:14:33 -0700 (PDT) X-Received: by 10.58.216.163 with SMTP id or3mr83249vec.80.1401192873478; Tue, 27 May 2014 05:14:33 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.221.72 with SMTP id ib8csp114063vcb; Tue, 27 May 2014 05:14:33 -0700 (PDT) X-Received: by 10.52.125.147 with SMTP id mq19mr22275670vdb.2.1401192872947; Tue, 27 May 2014 05:14:32 -0700 (PDT) Received: from lists.xen.org (lists.xen.org. [50.57.142.19]) by mx.google.com with ESMTPS id d8si7947707vdb.85.2014.05.27.05.14.32 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 27 May 2014 05:14:32 -0700 (PDT) Received-SPF: none (google.com: xen-devel-bounces@lists.xen.org does not designate permitted sender hosts) client-ip=50.57.142.19; Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1WpGEh-0005a1-Je; Tue, 27 May 2014 12:12:07 +0000 Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1WpGEc-0005Zl-Pi for xen-devel@lists.xen.org; Tue, 27 May 2014 12:12:03 +0000 Received: from [85.158.143.35:62571] by server-3.bemta-4.messagelabs.com id 85/C1-13602-21184835; Tue, 27 May 2014 12:12:02 +0000 X-Env-Sender: Stefano.Stabellini@citrix.com X-Msg-Ref: server-6.tower-21.messagelabs.com!1401192717!7444616!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n X-StarScan-Received: X-StarScan-Version: 6.11.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 18583 invoked from network); 27 May 2014 12:12:00 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-6.tower-21.messagelabs.com with RC4-SHA encrypted SMTP; 27 May 2014 12:12:00 -0000 X-IronPort-AV: E=Sophos;i="4.98,918,1392163200"; d="scan'208";a="135523262" Received: from accessns.citrite.net (HELO FTLPEX01CL02.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 27 May 2014 12:11:57 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.79) with Microsoft SMTP Server id 14.3.181.6; Tue, 27 May 2014 08:11:55 -0400 Received: from kaball.uk.xensource.com ([10.80.2.59]) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1WpGEU-00034F-PR; Tue, 27 May 2014 13:11:54 +0100 Date: Tue, 27 May 2014 13:11:51 +0100 From: Stefano Stabellini X-X-Sender: sstabellini@kaball.uk.xensource.com To: Oleksandr Tyshchenko In-Reply-To: Message-ID: References: User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 X-DLP: MIA2 Cc: Julien Grall , Ian Campbell , Stefano Stabellini , "xen-devel@lists.xen.org" Subject: Re: [Xen-devel] ARM Generic Timer interrupt X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Post: , List-Help: , List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: stefano.stabellini@eu.citrix.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.178 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Archive: Content-ID: On Mon, 26 May 2014, Oleksandr Tyshchenko wrote: > Hello, all. > > I am trying to run QNX as domU under XEN (4.4.0) on OMAP5 platform (ARM32). > For this purposes I have made some changes to origin TI OMAP5 BSP. > Also I have created QNX IFS loader for Xen domain builder. > > Currently, dom0 (Linux Kernel) loads QNX as domU. The QNX boots without crashes, > and I have console (I left only one hw block in QNX - UART, > I need it for debug output while HVC is not implemented). > > During bringing up I have encountered with next problem. > 1. QNX (our BSP) uses ARM Generic Timer as system timer: > QNX doesn't mask/unmask timer interrupt. Of course the QNX doesn't know that interrupt mask may be set by someone else > and that it should be reset for timer interrupt to occur again. > > 2. XEN handles the firing ARM Generic Timer: > Before injecting irq to the guest the XEN sets interrupt mask for the virtual timer. > > .../xen/arch/arm/time.c > > static void vtimer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs) > { >     current->arch.virt_timer.ctl = READ_SYSREG32(CNTV_CTL_EL0); >     WRITE_SYSREG32(current->arch.virt_timer.ctl | CNTx_CTL_MASK, CNTV_CTL_EL0); >     vgic_vcpu_inject_irq(current, current->arch.virt_timer.irq, 1); > } > > And as result of run under XEN we don't have timer interrupts in QNX. > I have changed interrupt handler in QNX to unmask timer interrupt before return. Hi Oleksandr, thanks for reporting this issue and for the good analysis of the problem, as always. > But, I have question: > Should the Hypervisor masks virtual timer IRQ on his own? > It is a guest's resource and the guest itself should decide what to do. > For example, I see that Linux Kernel (3.8) sets and clears timer interrupt mask by itself. In principle I agree with you that the vtimer is a guest resource. However in practice if we don't mask the irq we can easily get into an interrupt storm situation: if the guest doesn't handle the interrupt immediately we could keep receiving the vtimer irq in the hypervisor and busy loop around it. We really need to mask it somehow. So the real question is: how are we going to mask the vtimer irq? We could: 1) write CNTx_CTL_MASK to the ctl register, masking it at the timer level 2) write to GICD_ICENABLER, deactivating it at the GICD level Given that 1) didn't sound right to me, I tried 2) first but I had issues with the ARM emulator at the time. And as an ulterior confirmation that deactivating it is not how ARM thought that the vtimer should be used, Linux and KVM do 1) too. But I don't like the idea of having to modify the vtimer handler in QNX, so I have hacked together this patch, that would do 2) on top of my maintenance interrupt series. Unfortunately it needs to ask for a maintenance interrupt for the vtimer interrupt, so I can't say I am completely convinced that it is the right thing to do. What do people think about this? diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c index 08ae23b..36e2ec0 100644 --- a/xen/arch/arm/gic.c +++ b/xen/arch/arm/gic.c @@ -559,6 +559,8 @@ static inline void gic_set_lr(int lr, struct pending_irq *p, ((p->irq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT); if ( p->desc != NULL ) lr_val |= GICH_LR_HW | (p->desc->irq << GICH_LR_PHYSICAL_SHIFT); + else if ( p->irq == timer_get_irq(TIMER_VIRT_PPI) ) + lr_val |= GICH_LR_MAINTENANCE_IRQ; GICH[GICH_LR + lr] = lr_val; @@ -636,6 +638,16 @@ void gic_raise_guest_irq(struct vcpu *v, unsigned int virtual_irq, gic_add_to_lr_pending(v, irq_to_pending(v, virtual_irq)); } +void gic_disable_vtimer(void) +{ + GICD[GICD_ICENABLER] = (1 << timer_get_irq(TIMER_VIRT_PPI)); +} + +void gic_enable_vtimer(void) +{ + GICD[GICD_ISENABLER] = (1 << timer_get_irq(TIMER_VIRT_PPI)); +} + static void gic_update_one_lr(struct vcpu *v, int i) { struct pending_irq *p; @@ -676,6 +688,8 @@ static void gic_update_one_lr(struct vcpu *v, int i) clear_bit(GIC_IRQ_GUEST_VISIBLE, &p->status); clear_bit(GIC_IRQ_GUEST_ACTIVE, &p->status); p->lr = GIC_INVALID_LR; + if ( timer_get_irq(TIMER_VIRT_PPI) == irq ) + gic_enable_vtimer(); if ( test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) && test_bit(GIC_IRQ_GUEST_QUEUED, &p->status) ) gic_raise_guest_irq(v, irq, p->priority); diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c index 6e8d1f3..f9a6e7e 100644 --- a/xen/arch/arm/time.c +++ b/xen/arch/arm/time.c @@ -214,7 +214,6 @@ static void timer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs) static void vtimer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs) { current->arch.virt_timer.ctl = READ_SYSREG32(CNTV_CTL_EL0); - WRITE_SYSREG32(current->arch.virt_timer.ctl | CNTx_CTL_MASK, CNTV_CTL_EL0); vgic_vcpu_inject_irq(current, current->arch.virt_timer.irq); } diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c index 0f0ba1d..0624aa9 100644 --- a/xen/arch/arm/vgic.c +++ b/xen/arch/arm/vgic.c @@ -757,6 +757,11 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq) if ( !list_empty(&n->inflight) ) { + if ( timer_get_irq(TIMER_VIRT_PPI) == irq ) + { + gic_disable_vtimer(); + goto out; + } set_bit(GIC_IRQ_GUEST_QUEUED, &n->status); gic_raise_inflight_irq(v, irq); goto out; diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h index bf6fb1e..08f3c08 100644 --- a/xen/include/asm-arm/gic.h +++ b/xen/include/asm-arm/gic.h @@ -227,6 +227,9 @@ int gic_irq_xlate(const u32 *intspec, unsigned int intsize, unsigned int *out_hwirq, unsigned int *out_type); void gic_clear_lrs(struct vcpu *v); +extern void gic_disable_vtimer(void); +extern void gic_enable_vtimer(void); + #endif /* __ASSEMBLY__ */ #endif