mbox series

[RFC,v2,00/11] Linux RISC-V ACLINT Support

Message ID 20210618123851.1344518-1-anup.patel@wdc.com
Headers show
Series Linux RISC-V ACLINT Support | expand

Message

Anup Patel June 18, 2021, 12:38 p.m. UTC
Most of the existing RISC-V platforms use SiFive CLINT to provide M-level
timer and IPI support whereas S-level uses SBI calls for timer and IPI
support. Also, the SiFive CLINT device is a single device providing both
timer and IPI functionality so RISC-V platforms can't partially implement
SiFive CLINT device and provide alternate mechanism for timer and IPI.

The RISC-V Advacned Core Local Interruptor (ACLINT) tries to address the
limitations of SiFive CLINT by:
1) Taking modular approach and defining timer and IPI functionality as
   separate devices so that RISC-V platforms can include only required
   devices
2) Providing dedicated MMIO device for S-level IPIs so that SBI calls
   can be avoided for IPIs in Linux RISC-V
3) Allowing multiple instances of timer and IPI devices for a
   multi-socket (or multi-die) NUMA systems
4) Being backward compatible to SiFive CLINT so that existing RISC-V
   platforms stay compliant with RISC-V ACLINT specification

Latest RISC-V ACLINT specification (will be frozen in a month) can be
found at:
https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc

This series adds RISC-V ACLINT support and can be found in riscv_aclint_v2
branch at:
https://github.com/avpatel/linux

To test this series, the RISC-V ACLINT support for QEMU and OpenSBI
can be found in the riscv_aclint_v1 branch at:
https://github.com/avpatel/qemu
https://github.com/avpatel/opensbi

Changes since v1:
 - Added a new PATCH3 to treat IPIs as normal Linux IRQs for RISC-V kernel
 - New SBI IPI call based irqchip driver in PATCH3 which is only initialized
   by riscv_ipi_setup() when no Linux IRQ numbers are available for IPIs
 - Moved DT bindings patches before corresponding driver patches
 - Implemented ACLINT SWI driver as a irqchip driver in PATCH7
 - Minor nit fixes pointed by Bin Meng

Anup Patel (11):
  RISC-V: Clear SIP bit only when using SBI IPI operations
  RISC-V: Use common print prefix in smp.c
  RISC-V: Treat IPIs as normal Linux IRQs
  RISC-V: Allow marking IPIs as suitable for remote FENCEs
  RISC-V: Use IPIs for remote TLB flush when possible
  dt-bindings: interrupt-controller: Add ACLINT MSWI and SSWI bindings
  irqchip: Add ACLINT software interrupt driver
  RISC-V: Select ACLINT SWI driver for virt machine
  dt-bindings: timer: Add ACLINT MTIMER bindings
  clocksource: clint: Add support for ACLINT MTIMER device
  MAINTAINERS: Add entry for RISC-V ACLINT drivers

 .../riscv,aclint-swi.yaml                     |  82 ++++++
 .../bindings/timer/riscv,aclint-mtimer.yaml   |  55 ++++
 MAINTAINERS                                   |   9 +
 arch/riscv/Kconfig                            |   1 +
 arch/riscv/Kconfig.socs                       |   1 +
 arch/riscv/include/asm/sbi.h                  |   2 +
 arch/riscv/include/asm/smp.h                  |  48 +++-
 arch/riscv/kernel/Makefile                    |   1 +
 arch/riscv/kernel/cpu-hotplug.c               |   2 +
 arch/riscv/kernel/irq.c                       |   1 +
 arch/riscv/kernel/sbi-ipi.c                   | 223 ++++++++++++++
 arch/riscv/kernel/sbi.c                       |  15 -
 arch/riscv/kernel/smp.c                       | 171 +++++------
 arch/riscv/kernel/smpboot.c                   |   4 +-
 arch/riscv/mm/tlbflush.c                      |  62 +++-
 drivers/clocksource/timer-clint.c             |  58 ++--
 drivers/irqchip/Kconfig                       |  11 +
 drivers/irqchip/Makefile                      |   1 +
 drivers/irqchip/irq-aclint-swi.c              | 271 ++++++++++++++++++
 drivers/irqchip/irq-riscv-intc.c              |  55 ++--
 20 files changed, 879 insertions(+), 194 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,aclint-swi.yaml
 create mode 100644 Documentation/devicetree/bindings/timer/riscv,aclint-mtimer.yaml
 create mode 100644 arch/riscv/kernel/sbi-ipi.c
 create mode 100644 drivers/irqchip/irq-aclint-swi.c

Comments

Anup Patel July 26, 2021, 12:45 p.m. UTC | #1
Hi Marc,

I have taken the approach of IPI domains (like you suggested) in this series.

What do you think ?

Regards,
Anup

On Fri, Jun 18, 2021 at 6:09 PM Anup Patel <anup.patel@wdc.com> wrote:
>

> Most of the existing RISC-V platforms use SiFive CLINT to provide M-level

> timer and IPI support whereas S-level uses SBI calls for timer and IPI

> support. Also, the SiFive CLINT device is a single device providing both

> timer and IPI functionality so RISC-V platforms can't partially implement

> SiFive CLINT device and provide alternate mechanism for timer and IPI.

>

> The RISC-V Advacned Core Local Interruptor (ACLINT) tries to address the

> limitations of SiFive CLINT by:

> 1) Taking modular approach and defining timer and IPI functionality as

>    separate devices so that RISC-V platforms can include only required

>    devices

> 2) Providing dedicated MMIO device for S-level IPIs so that SBI calls

>    can be avoided for IPIs in Linux RISC-V

> 3) Allowing multiple instances of timer and IPI devices for a

>    multi-socket (or multi-die) NUMA systems

> 4) Being backward compatible to SiFive CLINT so that existing RISC-V

>    platforms stay compliant with RISC-V ACLINT specification

>

> Latest RISC-V ACLINT specification (will be frozen in a month) can be

> found at:

> https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc

>

> This series adds RISC-V ACLINT support and can be found in riscv_aclint_v2

> branch at:

> https://github.com/avpatel/linux

>

> To test this series, the RISC-V ACLINT support for QEMU and OpenSBI

> can be found in the riscv_aclint_v1 branch at:

> https://github.com/avpatel/qemu

> https://github.com/avpatel/opensbi

>

> Changes since v1:

>  - Added a new PATCH3 to treat IPIs as normal Linux IRQs for RISC-V kernel

>  - New SBI IPI call based irqchip driver in PATCH3 which is only initialized

>    by riscv_ipi_setup() when no Linux IRQ numbers are available for IPIs

>  - Moved DT bindings patches before corresponding driver patches

>  - Implemented ACLINT SWI driver as a irqchip driver in PATCH7

>  - Minor nit fixes pointed by Bin Meng

>

> Anup Patel (11):

>   RISC-V: Clear SIP bit only when using SBI IPI operations

>   RISC-V: Use common print prefix in smp.c

>   RISC-V: Treat IPIs as normal Linux IRQs

>   RISC-V: Allow marking IPIs as suitable for remote FENCEs

>   RISC-V: Use IPIs for remote TLB flush when possible

>   dt-bindings: interrupt-controller: Add ACLINT MSWI and SSWI bindings

>   irqchip: Add ACLINT software interrupt driver

>   RISC-V: Select ACLINT SWI driver for virt machine

>   dt-bindings: timer: Add ACLINT MTIMER bindings

>   clocksource: clint: Add support for ACLINT MTIMER device

>   MAINTAINERS: Add entry for RISC-V ACLINT drivers

>

>  .../riscv,aclint-swi.yaml                     |  82 ++++++

>  .../bindings/timer/riscv,aclint-mtimer.yaml   |  55 ++++

>  MAINTAINERS                                   |   9 +

>  arch/riscv/Kconfig                            |   1 +

>  arch/riscv/Kconfig.socs                       |   1 +

>  arch/riscv/include/asm/sbi.h                  |   2 +

>  arch/riscv/include/asm/smp.h                  |  48 +++-

>  arch/riscv/kernel/Makefile                    |   1 +

>  arch/riscv/kernel/cpu-hotplug.c               |   2 +

>  arch/riscv/kernel/irq.c                       |   1 +

>  arch/riscv/kernel/sbi-ipi.c                   | 223 ++++++++++++++

>  arch/riscv/kernel/sbi.c                       |  15 -

>  arch/riscv/kernel/smp.c                       | 171 +++++------

>  arch/riscv/kernel/smpboot.c                   |   4 +-

>  arch/riscv/mm/tlbflush.c                      |  62 +++-

>  drivers/clocksource/timer-clint.c             |  58 ++--

>  drivers/irqchip/Kconfig                       |  11 +

>  drivers/irqchip/Makefile                      |   1 +

>  drivers/irqchip/irq-aclint-swi.c              | 271 ++++++++++++++++++

>  drivers/irqchip/irq-riscv-intc.c              |  55 ++--

>  20 files changed, 879 insertions(+), 194 deletions(-)

>  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,aclint-swi.yaml

>  create mode 100644 Documentation/devicetree/bindings/timer/riscv,aclint-mtimer.yaml

>  create mode 100644 arch/riscv/kernel/sbi-ipi.c

>  create mode 100644 drivers/irqchip/irq-aclint-swi.c

>

> --

> 2.25.1

>
Anup Patel July 26, 2021, 1:01 p.m. UTC | #2
Hi Marc,

On Mon, Jul 26, 2021 at 8:02 PM Marc Zyngier <maz@kernel.org> wrote:
>

> On Mon, 26 Jul 2021 13:45:20 +0100,

> Anup Patel <anup@brainfault.org> wrote:

> >

> > Hi Marc,

> >

> > I have taken the approach of IPI domains (like you suggested) in this series.

> >

> > What do you think ?

>

> I have commented on the irqchip driver.

>

> As for the RISC-V specific code, I'll let the architecture maintainers

> look into it. I guess the elephant in the room is that this spec seems

> to be evolving, and that there is no HW implementation (how this

> driver maps on SF's CLINT is anybody's guess).


The SiFive CLINT is a more convoluted device and provides M-level
timer functionality and M-level IPI functionality in one MMIO device.

The RISC-V ACLINT specification is more modular and backward
compatible with the SiFive CLINT. In fact, a SiFive CLINT device
can be viewed as a ACLINT MSWI device + ACLINT MTIMER device.
This means existing RISC-V boards having SiFive CLINT will be
automatically compliant to the RISC-V ACLINT specification.

Here's the RISC-V ACLINT spec:
https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc

The RISC-V ACLINT spec is quite stable and we are not seeing any
further changes hence I sent out RFC PATCHes to get feedback. The
RISC-V ACLINT spec will be frozen before 2021 end (i.e. before next
RISC-V summit).

The Linux NoMMU kernel (M-level) will use an ACLINT MSWI device
for IPI support whereas the regular Linux MMU kernel (S-level) will
use an ACLINT SSWI device for IPI support.

The ACLINT SWI driver is a common IPI driver for both ACLINT
MSWI (Linux NoMMU) and ACLINT SSWI (Linux MMU). In fact,
the ACLINT SWI also works for IPI part (i.e. MSWI) of SiFive CLINT.

Regards,
Anup

>

>         M.

>

> --

> Without deviation from the norm, progress is not possible.
Marc Zyngier July 26, 2021, 2:32 p.m. UTC | #3
On Mon, 26 Jul 2021 13:45:20 +0100,
Anup Patel <anup@brainfault.org> wrote:
> 

> Hi Marc,

> 

> I have taken the approach of IPI domains (like you suggested) in this series.

> 

> What do you think ?


I have commented on the irqchip driver.

As for the RISC-V specific code, I'll let the architecture maintainers
look into it. I guess the elephant in the room is that this spec seems
to be evolving, and that there is no HW implementation (how this
driver maps on SF's CLINT is anybody's guess).

	M.

-- 
Without deviation from the norm, progress is not possible.
Palmer Dabbelt July 29, 2021, 4:30 a.m. UTC | #4
On Mon, 26 Jul 2021 06:01:01 PDT (-0700), anup@brainfault.org wrote:
> Hi Marc,

>

> On Mon, Jul 26, 2021 at 8:02 PM Marc Zyngier <maz@kernel.org> wrote:

>>

>> On Mon, 26 Jul 2021 13:45:20 +0100,

>> Anup Patel <anup@brainfault.org> wrote:

>> >

>> > Hi Marc,

>> >

>> > I have taken the approach of IPI domains (like you suggested) in this series.

>> >

>> > What do you think ?

>>

>> I have commented on the irqchip driver.

>>

>> As for the RISC-V specific code, I'll let the architecture maintainers

>> look into it. I guess the elephant in the room is that this spec seems

>> to be evolving, and that there is no HW implementation (how this

>> driver maps on SF's CLINT is anybody's guess).


There's a long history of interrupt controller efforts from the RISC-V 
foundation, and we've yet to have any of them result in hardware.

> The SiFive CLINT is a more convoluted device and provides M-level

> timer functionality and M-level IPI functionality in one MMIO device.

>

> The RISC-V ACLINT specification is more modular and backward

> compatible with the SiFive CLINT. In fact, a SiFive CLINT device

> can be viewed as a ACLINT MSWI device + ACLINT MTIMER device.

> This means existing RISC-V boards having SiFive CLINT will be

> automatically compliant to the RISC-V ACLINT specification.


So is there any hardware that this new specification enables?  It seems 
to be a more convoluted way to describe the same mess we're already in.  
I'm not really inclined to take a bunch of code that just does the same 
thing via a more complicated specification.

> Here's the RISC-V ACLINT spec:

> https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc

>

> The RISC-V ACLINT spec is quite stable and we are not seeing any

> further changes hence I sent out RFC PATCHes to get feedback. The

> RISC-V ACLINT spec will be frozen before 2021 end (i.e. before next

> RISC-V summit).


Have you talked to the other ISA folks about that?

As far as I can tell this new spec allows for multiple MTIME registers, 
which seems to be in direct contradiction to the single -MTIME register 
as defined in the ISA manual.  It also seems to be vaguely incompatible 
WRT the definition of SSIP, but I'm not sure that one really matters all 
that much as it's not like old software can write the new registers.

I just talked to Krste and Andrew, they say they haven't heard of any of 
this.  I don't know what's going on over there, but it's very hard to 
review anything when I can't even tell where the ISA is defined.

> The Linux NoMMU kernel (M-level) will use an ACLINT MSWI device

> for IPI support whereas the regular Linux MMU kernel (S-level) will

> use an ACLINT SSWI device for IPI support.

>

> The ACLINT SWI driver is a common IPI driver for both ACLINT

> MSWI (Linux NoMMU) and ACLINT SSWI (Linux MMU). In fact,

> the ACLINT SWI also works for IPI part (i.e. MSWI) of SiFive CLINT.

>

> Regards,

> Anup

>

>>

>>         M.

>>

>> --

>> Without deviation from the norm, progress is not possible.
Anup Patel July 29, 2021, 4:56 a.m. UTC | #5
On Thu, Jul 29, 2021 at 10:00 AM Palmer Dabbelt
<palmerdabbelt@google.com> wrote:
>

> On Mon, 26 Jul 2021 06:01:01 PDT (-0700), anup@brainfault.org wrote:

> > Hi Marc,

> >

> > On Mon, Jul 26, 2021 at 8:02 PM Marc Zyngier <maz@kernel.org> wrote:

> >>

> >> On Mon, 26 Jul 2021 13:45:20 +0100,

> >> Anup Patel <anup@brainfault.org> wrote:

> >> >

> >> > Hi Marc,

> >> >

> >> > I have taken the approach of IPI domains (like you suggested) in this series.

> >> >

> >> > What do you think ?

> >>

> >> I have commented on the irqchip driver.

> >>

> >> As for the RISC-V specific code, I'll let the architecture maintainers

> >> look into it. I guess the elephant in the room is that this spec seems

> >> to be evolving, and that there is no HW implementation (how this

> >> driver maps on SF's CLINT is anybody's guess).

>

> There's a long history of interrupt controller efforts from the RISC-V

> foundation, and we've yet to have any of them result in hardware.


The RISC-V AIA group was formed last year. Can you point me to which
interrupt controller efforts you are referring to.

>

> > The SiFive CLINT is a more convoluted device and provides M-level

> > timer functionality and M-level IPI functionality in one MMIO device.

> >

> > The RISC-V ACLINT specification is more modular and backward

> > compatible with the SiFive CLINT. In fact, a SiFive CLINT device

> > can be viewed as a ACLINT MSWI device + ACLINT MTIMER device.

> > This means existing RISC-V boards having SiFive CLINT will be

> > automatically compliant to the RISC-V ACLINT specification.

>

> So is there any hardware that this new specification enables?  It seems

> to be a more convoluted way to describe the same mess we're already in.

> I'm not really inclined to take a bunch of code that just does the same

> thing via a more complicated specification.


Nope, it is much cleaner and modular compared to SiFive CLINT and
it is also backward compatible to SiFive CLINT.

Can you elaborate what part of the code you are not okay with ?

>

> > Here's the RISC-V ACLINT spec:

> > https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc

> >

> > The RISC-V ACLINT spec is quite stable and we are not seeing any

> > further changes hence I sent out RFC PATCHes to get feedback. The

> > RISC-V ACLINT spec will be frozen before 2021 end (i.e. before next

> > RISC-V summit).

>

> Have you talked to the other ISA folks about that?


This spec is being developed by the RISC-V AIA group based on the
feedback from ISA folks and HW architects.

>

> As far as I can tell this new spec allows for multiple MTIME registers,

> which seems to be in direct contradiction to the single -MTIME register

> as defined in the ISA manual.  It also seems to be vaguely incompatible

> WRT the definition of SSIP, but I'm not sure that one really matters all

> that much as it's not like old software can write the new registers.


The ACLINT spec clearly defines that if we have multiple MTIME registers
then these registers must be synchronized to meet the architecture
requirements. The spec also defines a software mechanism for MTIME
synchronization. It is also possible for multiple MTIMER devices to share
same MTIME register. Please refer to the latest ACLINT spec.

>

> I just talked to Krste and Andrew, they say they haven't heard of any of

> this.  I don't know what's going on over there, but it's very hard to

> review anything when I can't even tell where the ISA is defined.


I am surprised by the respone you got because the ACLINT spec is
being developed by a working group of RISC-V International. In fact,
it is hosted on the RISC-V International GitHub. How can we host it on
RISC-V GitHub if it is not an official spec being developed by RVI.

Regards,
Anup

>

> > The Linux NoMMU kernel (M-level) will use an ACLINT MSWI device

> > for IPI support whereas the regular Linux MMU kernel (S-level) will

> > use an ACLINT SSWI device for IPI support.

> >

> > The ACLINT SWI driver is a common IPI driver for both ACLINT

> > MSWI (Linux NoMMU) and ACLINT SSWI (Linux MMU). In fact,

> > the ACLINT SWI also works for IPI part (i.e. MSWI) of SiFive CLINT.

> >

> > Regards,

> > Anup

> >

> >>

> >>         M.

> >>

> >> --

> >> Without deviation from the norm, progress is not possible.
Anup Patel July 29, 2021, 5:36 a.m. UTC | #6
On Thu, Jul 29, 2021 at 10:00 AM Palmer Dabbelt
<palmerdabbelt@google.com> wrote:
>

> On Mon, 26 Jul 2021 06:01:01 PDT (-0700), anup@brainfault.org wrote:

> > Hi Marc,

> >

> > On Mon, Jul 26, 2021 at 8:02 PM Marc Zyngier <maz@kernel.org> wrote:

> >>

> >> On Mon, 26 Jul 2021 13:45:20 +0100,

> >> Anup Patel <anup@brainfault.org> wrote:

> >> >

> >> > Hi Marc,

> >> >

> >> > I have taken the approach of IPI domains (like you suggested) in this series.

> >> >

> >> > What do you think ?

> >>

> >> I have commented on the irqchip driver.

> >>

> >> As for the RISC-V specific code, I'll let the architecture maintainers

> >> look into it. I guess the elephant in the room is that this spec seems

> >> to be evolving, and that there is no HW implementation (how this

> >> driver maps on SF's CLINT is anybody's guess).

>

> There's a long history of interrupt controller efforts from the RISC-V

> foundation, and we've yet to have any of them result in hardware.

>

> > The SiFive CLINT is a more convoluted device and provides M-level

> > timer functionality and M-level IPI functionality in one MMIO device.

> >

> > The RISC-V ACLINT specification is more modular and backward

> > compatible with the SiFive CLINT. In fact, a SiFive CLINT device

> > can be viewed as a ACLINT MSWI device + ACLINT MTIMER device.

> > This means existing RISC-V boards having SiFive CLINT will be

> > automatically compliant to the RISC-V ACLINT specification.

>

> So is there any hardware that this new specification enables?  It seems

> to be a more convoluted way to describe the same mess we're already in.

> I'm not really inclined to take a bunch of code that just does the same

> thing via a more complicated specification.

>

> > Here's the RISC-V ACLINT spec:

> > https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc

> >

> > The RISC-V ACLINT spec is quite stable and we are not seeing any

> > further changes hence I sent out RFC PATCHes to get feedback. The

> > RISC-V ACLINT spec will be frozen before 2021 end (i.e. before next

> > RISC-V summit).

>

> Have you talked to the other ISA folks about that?

>

> As far as I can tell this new spec allows for multiple MTIME registers,

> which seems to be in direct contradiction to the single -MTIME register


The RISC-V ISA spec only mandates a single view of MTIME registers
for all the HARTs

The RISC-V ISA spec does not mandate a single physical MTIME register.

In fact, multi-socket and multi-die systems will end-up having multiple
physical MTIME registers so on such systems these physical MTIME
registers need to be synchronized with hardware assistance. Please
refer to the MTIME synchronization section of ACLINT specification.

> as defined in the ISA manual.  It also seems to be vaguely incompatible

> WRT the definition of SSIP, but I'm not sure that one really matters all

> that much as it's not like old software can write the new registers.


Please loot at the ACLINT SSWI spec again. The SSIP bit definition
is to be modified where mip.SSIP bit is logical OR of software writable
bit and external SSIP signal. This way older software which uses
SBI call based IPI injection will work fine. In fact, this aspect of SSIP
bit is tested on QEMU RISC-V as well.

>

> I just talked to Krste and Andrew, they say they haven't heard of any of

> this.  I don't know what's going on over there, but it's very hard to

> review anything when I can't even tell where the ISA is defined.


Like mentioned in other thread, please first to talk to the actual
spec owners first. There are lot of working groups and activities
going on in RVI.

Regards,
Anup

>

> > The Linux NoMMU kernel (M-level) will use an ACLINT MSWI device

> > for IPI support whereas the regular Linux MMU kernel (S-level) will

> > use an ACLINT SSWI device for IPI support.

> >

> > The ACLINT SWI driver is a common IPI driver for both ACLINT

> > MSWI (Linux NoMMU) and ACLINT SSWI (Linux MMU). In fact,

> > the ACLINT SWI also works for IPI part (i.e. MSWI) of SiFive CLINT.

> >

> > Regards,

> > Anup

> >

> >>

> >>         M.

> >>

> >> --

> >> Without deviation from the norm, progress is not possible.