diff mbox

[v4,00/11] Fix versatile_pci

Message ID 1373265836.27613.14@driftwood
State New
Headers show

Commit Message

Rob Landley July 8, 2013, 6:43 a.m. UTC
On 06/29/2013 06:03:26 AM, Peter Maydell wrote:
> On 28 June 2013 08:01, Rob Landley <rob@landley.net> wrote:
> > Now that the next kernel's about to come out, I'm trying to get my  
> arm
> > versatile image to work under qemu 1.5.0. The old kernel doesn't  
> work, and
> > the current vanilla kernel doesn't work. This change broke it.
> >
> > I'm testing current linux-git both with and without this patch:
> >
> > --- linux/arch/arm/mach-versatile/pci.c 2013-04-28  
> 19:36:01.000000000 -0500
> > +++ linux.bak/arch/arm/mach-versatile/pci.c     2013-04-29
> > 19:09:44.857097553 -0500
> > @@ -333,7 +333,7 @@
> >          *  26     1     IRQ_SIC_PCI2
> >          *  27     1     IRQ_SIC_PCI3
> >          */
> > -       irq = IRQ_SIC_PCI0 + ((slot - 24 + pin - 1) & 3);
> > +       irq = 59; //IRQ_SIC_PCI0 + ((slot - 24 + pin - 1) & 3);
> >
> >         return irq;
> >  }
> >
> > With the patch, qemu 1.2.0 works, but qemu 1.5 versatile does not  
> work with
> > or without the patch.
> 
> The "with the patch" case is uninteresting, because it's not
> actually a fix for anything, it's just a change that happened
> to work with old QEMU, and it's not a change that is in upstream
> Linux.

I want to create a kernel image that works with the existing QEMU image  
people are likely to have installed on their systems, _and_ with  
current versions of QEMU.

Is that what you consider "uninteresting"?

> > Here's a test image to demonstrate the problem: it works fine under  
> qemu
> > 1.2.0, but doesn't under 1.5.0:
> >
> >   http://landley.net/aboriginal/bin/system-image-armv5l.tar.bz2
> >
> > Extract it and ./run-emulator.sh. You get a shell prompt from  
> 1.2.0, from
> > 1.5.0 it never gets a scsi interrupt, and after a timeout goes into  
> an
> > abort/reset loop.
> 
> Is this an image with your patch or without it? Does it work
> if you use the -global option to force legacy IRQ mapping?

Under qemu 1.2? It goes:

   qemu-system-arm: Property '.broken-irq-mapping' not found

See "backwards compatability", above.

I am trying to produce "an image that runs under qemu". That level of  
genericity worked for several years, at least ever since 1.0. When it  
stopped, I complained. You seem to be expressing incredulity at the  
concept.

> >> >> This version of the patchset avoids breaking legacy Linux guests
> >> >> by automatically detecting those broken kernels and switching  
> back
> >> >> to the old mapping.
> >
> >
> > As testing with the above image confirms: it does not.
> 
> I tested with several separate variants of the Linux kernel.

I tried building a vanilla linux 3.10 kernel, and it doesn't work on  
qemu 1.5. I confirmed their current interrupt mapping does not match  
any known hardware or emulator, and that it changed more than once in  
incompatible ways as they did wild-ass-guess du jour about what it  
should be now.

The fact it was obviously untested (or it wouldn't keep changing, since  
there presumably is just one right answer) says to me that this  
hardware is difficult ot come by. (The fact the IRQ spent over 5 years  
in the old state before anyone noticed would also imply a lack of test  
hardware.) However, it's useful to emulate because you can stick a PCI  
bus in it, meaning you can stick arbitrary devices (hard drive and  
network card) into said bus.

So I did the obvious-to-me thing and patched it back to how it _was_,  
which is the only thing that works with previous versions of QEMU. You  
seem to find this approach shocking and unexpected. Do you have a  
better way to achieve compatability with older versions of qemu _and_  
current versions? (I'm open to suggestions...)

> >> >> This works by looking at the values the kernel
> >> >> writes to the PCI_INTERRUPT_LINE register in the config space,  
> which
> >> >> is effectively the interrupt number the kernel expects the  
> device
> >> >> to be using. If this doesn't match reality we use the broken  
> mapping.
> >> >> (Thanks to Michael S. Tsirkin for this suggestion.)
> >
> > Define "reality".
> 
> "would work on real hardware".

The stuff that's so rare the kernel guys couldn't find any to test on?

Maybe I haven't been clear: I want something that works with current  
QEMU _and_ old QEMU.

Your paragraph above implies that there's smoething _different_ I can  
write to PCI_INTERRUPT_LINE that will trigger the old behavior. So my  
patch to do the old IRQ (which worked fine with qemu 1.2) needs to be  
bigger to humor QEMU's heuristic. I'll try to figure out how to extend  
my patch to avoid triggering your new "does not work" mode.

Let's see, stick a printf into qemu, and:

   =====Write 60=12,59=====

So it's requesting IRQ 59, and QEMU's reponse is to _not_ give it IRQ  
59 unless we trigger the "broken" flag.

So... real hardware doesn't get the IRQ it requests? Using the reqested  
IRQ is "broken"? (Apparently your patch for a real hardware kernel  
doesn't request the IRQ it's going to use? I'm confused...)

Right, easy enough to fix:

+       // for compatability with old versions.
+       irq = 59; //IRQ_SIC_PCI0 + ((slot - 24 + pin - 1) & 3);

         return irq;
  }

That should work with new _and_ old QEMU.

> > The kernel changed what irqs it was expecting _three_times_ last  
> year, and
> > as far as I can tell none of them were what they were _trying_ to  
> do.
> >
> > Here's my blog entry and the source control comments where I  
> diagnosed this
> > stuff to show that the kernel guys have no flipping CLUE how an arm
> > versatile board actually works:
> 
> I agree that the kernel has made a bit of a mess in this area,
> so we don't need to have that argument again.

Can we have the backwards compatability argument?

> > The kernel code in this area is CRAP, has changed multiple times in
> > semi-random ways, has obviously NEVER been tested on real hardware,  
> and if
> > you've implemented what the actual versatile documentation says  
> it's clearly
> > not what the kernel is doing.
> 
> I tested these changes against a patchset which Arnd wrote which
> I can guarantee was tested on real hardware because I did the
> testing myself.

Where is this patch? (Was it submitted upstream to linux-kernel?)

> > Do you have a kernel that runs under current qemu arm versatile  
> emulation? I
> > can poke around and figure out which irqs it expects _now_, but  
> it's not
> > "right" and presumably you're just going to change it again when  
> you realize
> > what qemu is doing and what the unpatched kernel is doing don't  
> match.
> 
> The ones on http://people.debian.org/~aurel32/qemu/armel/
> should work.

I'm not seeing a patch there.

Rob

Comments

Peter Maydell July 8, 2013, 7:46 a.m. UTC | #1
On 8 July 2013 07:43, Rob Landley <rob@landley.net> wrote:
> On 06/29/2013 06:03:26 AM, Peter Maydell wrote:
>> The "with the patch" case is uninteresting, because it's not
>> actually a fix for anything, it's just a change that happened
>> to work with old QEMU, and it's not a change that is in upstream
>> Linux.
>
>
> I want to create a kernel image that works with the existing QEMU image
> people are likely to have installed on their systems, _and_ with current
> versions of QEMU.
>
> Is that what you consider "uninteresting"?

Interesting:
 * kernels which actually work on real hardware
 * mainline kernels which worked on old QEMU, because they're
   widely used
Uninteresting:
 * local hacks which worked by fluke on old QEMU but won't
   work on the hardware
 * making mainline kernels which didn't work on old QEMU work
   by adapting to their bugginess

(current vanilla mainline falls into the last of these categories)

> I am trying to produce "an image that runs under qemu". That level of
> genericity worked for several years, at least ever since 1.0. When it
> stopped, I complained. You seem to be expressing incredulity at the concept.

QEMU has always been free to improve its emulation to be
closer to what the hardware does. If your guest doesn't
work on the hardware it is always going to be liable to
break.

It's really common for new kernel changes to require new QEMU,
because they tickle some bit of hardware emulation we had omitted
or hadn't got right. If you want a kernel image that runs on
older QEMU, stick with an older kernel.

> I tried building a vanilla linux 3.10 kernel, and it doesn't work on qemu
> 1.5. I confirmed their current interrupt mapping does not match any known
> hardware or emulator, and that it changed more than once in incompatible
> ways as they did wild-ass-guess du jour about what it should be now.

Current vanilla works for the first PCI card, not for others.
It behaves like this on both QEMU 1.4 and 1.5, ie these changes
have not broken it further than it is already.

> So I did the obvious-to-me thing and patched it back to how it _was_, which
> is the only thing that works with previous versions of QEMU. You seem to
> find this approach shocking and unexpected.

You haven't patched it back like how it was, or it would work;
old 2.x kernels work fine. Old kernels wrote 27 to PCI_INTERRUPT_LINE,
not 59.

>> "would work on real hardware".
>
> The stuff that's so rare the kernel guys couldn't find any to test on?

They could, if they asked, if they cared. I actually went
around and tested Arnd's patchset back in 2011. Nobody's
ever asked me since. The point is that this patchset is a
line in the sand -- we now have a defensible set of behaviour,
because it's what the hardware does. Any future "hey, the kernel
changed and broke things" can be directed straight at the kernel
guys, because we're not changing any further unless it can be
demonstrated that we don't run a kernel that's been tested on h/w.

> Maybe I haven't been clear: I want something that works with current QEMU
> _and_ old QEMU.
>
> Your paragraph above implies that there's smoething _different_ I can write
> to PCI_INTERRUPT_LINE that will trigger the old behavior. So my patch to do
> the old IRQ (which worked fine with qemu 1.2) needs to be bigger to humor
> QEMU's heuristic. I'll try to figure out how to extend my patch to avoid
> triggering your new "does not work" mode.

The old mode is the broken one...

> Let's see, stick a printf into qemu, and:
>
>   =====Write 60=12,59=====
>
> So it's requesting IRQ 59, and QEMU's reponse is to _not_ give it IRQ 59
> unless we trigger the "broken" flag.
>
> So... real hardware doesn't get the IRQ it requests? Using the reqested IRQ
> is "broken"? (Apparently your patch for a real hardware kernel doesn't
> request the IRQ it's going to use? I'm confused...)

On real hardware, PCI_INTERRUPT_LINE has no effect -- all it
does is store a value for the kernel to read back. It has
no effect on the interrupt routing. (This is true for any
PCI controller.) However, the kernel happens to write a value
which corresponds to what it *thinks* the hardware is wired up to.
So we can use this to identify some of the mainline cases which
expect old broken behaviour.

> Right, easy enough to fix:
>
> --- a/arch/arm/mach-versatile/pci.c
> +++ b/arch/arm/mach-versatile/pci.c
> @@ -333,7 +333,13 @@ static int __init versatile_map_irq(const struct
> pci_dev *dev, u8 slot, u8 pin)
>
>          *  26     1     IRQ_SIC_PCI2
>          *  27     1     IRQ_SIC_PCI3
>          */
> -       irq = IRQ_SIC_PCI0 + ((slot - 24 + pin - 1) & 3);
> +       // Hit QEMU 1.5.0 and later with a brick so it uses the IRQ we say.

This comment is totally wrong. More accurate would be:
    // This code won't work on hardware because we still assume
    // the broken interrupt mapping used by old (pre 1.5.0)
    // versions of QEMU. Ensure that QEMU 1.5 is in its backward
    // compatibility mode by writing a definitely wrong number
    // to PCI_INTERRUPT_LINE.

Also you're probably better off doing this write in the
initialization code, rather than in versatile_map_irq().

> +       dev->bus->ops->write(dev->bus, dev->devfn, PCI_INTERRUPT_LINE, 1,
> 27);
> +
> +       // The kernel has no clue where IRQs are, and its current
> assignments
> +       // match neither the hardware nor historic QEMU. Use historic QEMU
> +       // for compatability with old versions.
>
> +       irq = 59; //IRQ_SIC_PCI0 + ((slot - 24 + pin - 1) & 3);
>
>         return irq;
>  }
>
> That should work with new _and_ old QEMU.

>> I tested these changes against a patchset which Arnd wrote which
>> I can guarantee was tested on real hardware because I did the
>> testing myself.
>
>
> Where is this patch? (Was it submitted upstream to linux-kernel?)

Yes. http://comments.gmane.org/gmane.linux.ports.arm.kernel/93393

>> > Do you have a kernel that runs under current qemu arm versatile
>> > emulation? I
>> > can poke around and figure out which irqs it expects _now_, but it's not
>> > "right" and presumably you're just going to change it again when you
>> > realize
>> > what qemu is doing and what the unpatched kernel is doing don't match.
>>
>> The ones on http://people.debian.org/~aurel32/qemu/armel/
>> should work.
>
>
> I'm not seeing a patch there.

You asked for a kernel, not a patch. They're Debian's standard
2.6.32-5.

-- PMM
diff mbox

Patch

--- a/arch/arm/mach-versatile/pci.c
+++ b/arch/arm/mach-versatile/pci.c
@@ -333,7 +333,13 @@  static int __init versatile_map_irq(const struct  
pci_dev *dev, u8 slot, u8 pin)
          *  26     1     IRQ_SIC_PCI2
          *  27     1     IRQ_SIC_PCI3
          */
-       irq = IRQ_SIC_PCI0 + ((slot - 24 + pin - 1) & 3);
+       // Hit QEMU 1.5.0 and later with a brick so it uses the IRQ we  
say.
+       dev->bus->ops->write(dev->bus, dev->devfn, PCI_INTERRUPT_LINE,  
1, 27);
+
+       // The kernel has no clue where IRQs are, and its current  
assignments
+       // match neither the hardware nor historic QEMU. Use historic  
QEMU