mbox series

[v2,0/3] Qcom: Add GIC-ITS support to SM8450 PCIe controllers

Message ID 20221222133123.50676-1-manivannan.sadhasivam@linaro.org
Headers show
Series Qcom: Add GIC-ITS support to SM8450 PCIe controllers | expand

Message

Manivannan Sadhasivam Dec. 22, 2022, 1:31 p.m. UTC 
Hello,

This series adds GIC-ITS support to SM8450 PCIe controllers for receiving
the MSIs from endpoint devices.

The GIC-ITS MSI implementation provides an advantage over internal MSI
implementation using Locality-specific Peripheral Interrupts (LPI) that
would allow MSIs to be targeted for each CPU core.

This series has been tested on SM8450 based dev board that works using an
out-of-tree dts where the MSIs from endpoint devices are distributed across
the CPU cores.

Thanks,
Mani

Changes in v2:

* Swapped the Device ID for PCIe0 as it causes same issue as PCIe1
* Removed the definition of msi-map and msi-map-mask from binding
* Added Ack from Krzysztof

Manivannan Sadhasivam (3):
  dt-bindings: PCI: qcom: Update maintainers
  dt-bindings: PCI: qcom: Document msi-map and msi-map-mask properties
  arm64: dts: qcom: sm8450: Use GIC-ITS for PCIe0 and PCIe1

 .../devicetree/bindings/pci/qcom,pcie.yaml    | 14 +++++++++----
 arch/arm64/boot/dts/qcom/sm8450.dtsi          | 20 +++++++++++++------
 2 files changed, 24 insertions(+), 10 deletions(-)

Comments

Dmitry Baryshkov Dec. 23, 2022, 6:15 p.m. UTC | #1 
On 23/12/2022 19:45, Manivannan Sadhasivam wrote:
> On Fri, Dec 23, 2022 at 07:18:32PM +0200, Dmitry Baryshkov wrote:
>> On 22/12/2022 15:31, Manivannan Sadhasivam wrote:
>>> Both PCIe0 and PCIe1 controllers are capable of receiving MSIs from
>>> endpoint devices using GIC-ITS MSI controller. Add support for it.
>>>
>>> Currently, BDF (0:0.0) and BDF (1:0.0) are enabled and with the
>>> msi-map-mask of 0xff00, all the 32 devices under these two busses can
>>> share the same Device ID.
>>>
>>> The GIC-ITS MSI implementation provides an advantage over internal MSI
>>> implementation using Locality-specific Peripheral Interrupts (LPI) that
>>> would allow MSIs to be targeted for each CPU core.
>>>
>>> It should be noted that the MSIs for BDF (1:0.0) only works with Device
>>> ID of 0x5980 and 0x5a00. Hence, the IDs are swapped.
>>>
>>> Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
>>> ---
>>>    arch/arm64/boot/dts/qcom/sm8450.dtsi | 20 ++++++++++++++------
>>>    1 file changed, 14 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/arch/arm64/boot/dts/qcom/sm8450.dtsi b/arch/arm64/boot/dts/qcom/sm8450.dtsi
>>> index 570475040d95..c4dd5838fac6 100644
>>> --- a/arch/arm64/boot/dts/qcom/sm8450.dtsi
>>> +++ b/arch/arm64/boot/dts/qcom/sm8450.dtsi
>>> @@ -1733,9 +1733,13 @@ pcie0: pci@1c00000 {
>>>    			ranges = <0x01000000 0x0 0x60200000 0 0x60200000 0x0 0x100000>,
>>>    				 <0x02000000 0x0 0x60300000 0 0x60300000 0x0 0x3d00000>;
>>> -			interrupts = <GIC_SPI 141 IRQ_TYPE_LEVEL_HIGH>;
>>> -			interrupt-names = "msi";
>>> -			#interrupt-cells = <1>;
>>> +			/*
>>> +			 * MSIs for BDF (1:0.0) only works with Device ID 0x5980.
>>> +			 * Hence, the IDs are swapped.
>>> +			 */
>>> +			msi-map = <0x0 &gic_its 0x5981 0x1>,
>>> +				  <0x100 &gic_its 0x5980 0x1>;
>>
>> This definitely doesn't match what has been used in the downstream.
>>
> 
> Yes, I do not know why the downstream Device ID doesn't work. I tried finding
> the answer within Qcom but didn't get any answer so far :/ So I just went with
> the value that works on multiple boards.

Ugh :-(

> 
>> Also if I understand correctly this change would prevent us from using
>> multiple MSI interrupts for the connected device, as the last value of the
>> 0x100 mapping is 0x1, while the vendor kernel uses <0x100 &its 0x5981 0x20>.
>>
> 
> Not true. The controller can still support multiple MSIs for the endpoint
> devices but the only difference is, it would use the same Device ID for all.

I see, please excuse me then. But don't we have to define multiple MSI 
vectors here too?

> 
> The Qcom GIC-ITS implementation could only support 32 Device IDs. By specifying
> the size of 0x20, a separate Device ID would be used for each devices of bus 1.
> But if a PCIe switch is connected and the bus count becomes > 1, then the MSI
> allocation would fail because Device IDs are exhausted.
> 
> The downstream implementation just assumes that there will be only bus 1 and I
> do not want to follow that assumption.
> 
> That's why I used "msi-map-mask" property of value "0xff00" here, as that will
> allow all the devices under the bus 1 to share the same Device ID. For now I
> only mapped bus 1, but extending that in the future for other busses is simple.
> 
> Thanks,
> Mani
> 
>> Do you know by chance, why do we differ from the vendor dtsi?
>>
>>> +			msi-map-mask = <0xff00>;
>>>    			interrupt-map-mask = <0 0 0 0x7>;
>>>    			interrupt-map = <0 0 0 1 &intc 0 0 0 149 IRQ_TYPE_LEVEL_HIGH>, /* int_a */
>>>    					<0 0 0 2 &intc 0 0 0 150 IRQ_TYPE_LEVEL_HIGH>, /* int_b */
>>> @@ -1842,9 +1846,13 @@ pcie1: pci@1c08000 {
>>>    			ranges = <0x01000000 0x0 0x40200000 0 0x40200000 0x0 0x100000>,
>>>    				 <0x02000000 0x0 0x40300000 0 0x40300000 0x0 0x1fd00000>;
>>> -			interrupts = <GIC_SPI 307 IRQ_TYPE_LEVEL_HIGH>;
>>> -			interrupt-names = "msi";
>>> -			#interrupt-cells = <1>;
>>> +			/*
>>> +			 * MSIs for BDF (1:0.0) only works with Device ID 0x5a00.
>>> +			 * Hence, the IDs are swapped.
>>> +			 */
>>> +			msi-map = <0x0 &gic_its 0x5a01 0x1>,
>>> +				  <0x100 &gic_its 0x5a00 0x1>;
>>> +			msi-map-mask = <0xff00>;
>>>    			interrupt-map-mask = <0 0 0 0x7>;
>>>    			interrupt-map = <0 0 0 1 &intc 0 0 0 434 IRQ_TYPE_LEVEL_HIGH>, /* int_a */
>>>    					<0 0 0 2 &intc 0 0 0 435 IRQ_TYPE_LEVEL_HIGH>, /* int_b */
>>
>> -- 
>> With best wishes
>> Dmitry
>>
>
Manivannan Sadhasivam Dec. 28, 2022, 9:04 a.m. UTC | #2 
On Fri, Dec 23, 2022 at 08:15:32PM +0200, Dmitry Baryshkov wrote:
> On 23/12/2022 19:45, Manivannan Sadhasivam wrote:
> > On Fri, Dec 23, 2022 at 07:18:32PM +0200, Dmitry Baryshkov wrote:
> > > On 22/12/2022 15:31, Manivannan Sadhasivam wrote:
> > > > Both PCIe0 and PCIe1 controllers are capable of receiving MSIs from
> > > > endpoint devices using GIC-ITS MSI controller. Add support for it.
> > > > 
> > > > Currently, BDF (0:0.0) and BDF (1:0.0) are enabled and with the
> > > > msi-map-mask of 0xff00, all the 32 devices under these two busses can
> > > > share the same Device ID.
> > > > 
> > > > The GIC-ITS MSI implementation provides an advantage over internal MSI
> > > > implementation using Locality-specific Peripheral Interrupts (LPI) that
> > > > would allow MSIs to be targeted for each CPU core.
> > > > 
> > > > It should be noted that the MSIs for BDF (1:0.0) only works with Device
> > > > ID of 0x5980 and 0x5a00. Hence, the IDs are swapped.
> > > > 
> > > > Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > > > ---
> > > >    arch/arm64/boot/dts/qcom/sm8450.dtsi | 20 ++++++++++++++------
> > > >    1 file changed, 14 insertions(+), 6 deletions(-)
> > > > 
> > > > diff --git a/arch/arm64/boot/dts/qcom/sm8450.dtsi b/arch/arm64/boot/dts/qcom/sm8450.dtsi
> > > > index 570475040d95..c4dd5838fac6 100644
> > > > --- a/arch/arm64/boot/dts/qcom/sm8450.dtsi
> > > > +++ b/arch/arm64/boot/dts/qcom/sm8450.dtsi
> > > > @@ -1733,9 +1733,13 @@ pcie0: pci@1c00000 {
> > > >    			ranges = <0x01000000 0x0 0x60200000 0 0x60200000 0x0 0x100000>,
> > > >    				 <0x02000000 0x0 0x60300000 0 0x60300000 0x0 0x3d00000>;
> > > > -			interrupts = <GIC_SPI 141 IRQ_TYPE_LEVEL_HIGH>;
> > > > -			interrupt-names = "msi";
> > > > -			#interrupt-cells = <1>;
> > > > +			/*
> > > > +			 * MSIs for BDF (1:0.0) only works with Device ID 0x5980.
> > > > +			 * Hence, the IDs are swapped.
> > > > +			 */
> > > > +			msi-map = <0x0 &gic_its 0x5981 0x1>,
> > > > +				  <0x100 &gic_its 0x5980 0x1>;
> > > 
> > > This definitely doesn't match what has been used in the downstream.
> > > 
> > 
> > Yes, I do not know why the downstream Device ID doesn't work. I tried finding
> > the answer within Qcom but didn't get any answer so far :/ So I just went with
> > the value that works on multiple boards.
> 
> Ugh :-(
> 
> > 
> > > Also if I understand correctly this change would prevent us from using
> > > multiple MSI interrupts for the connected device, as the last value of the
> > > 0x100 mapping is 0x1, while the vendor kernel uses <0x100 &its 0x5981 0x20>.
> > > 
> > 
> > Not true. The controller can still support multiple MSIs for the endpoint
> > devices but the only difference is, it would use the same Device ID for all.
> 
> I see, please excuse me then. But don't we have to define multiple MSI
> vectors here too?
> 

No, it is not required. GIC-ITS driver will handle the MSI mapping internally
and devicetree only needs to specify the Device ID for each PCIe device.

Thanks,
Mani

> > 
> > The Qcom GIC-ITS implementation could only support 32 Device IDs. By specifying
> > the size of 0x20, a separate Device ID would be used for each devices of bus 1.
> > But if a PCIe switch is connected and the bus count becomes > 1, then the MSI
> > allocation would fail because Device IDs are exhausted.
> > 
> > The downstream implementation just assumes that there will be only bus 1 and I
> > do not want to follow that assumption.
> > 
> > That's why I used "msi-map-mask" property of value "0xff00" here, as that will
> > allow all the devices under the bus 1 to share the same Device ID. For now I
> > only mapped bus 1, but extending that in the future for other busses is simple.
> > 
> > Thanks,
> > Mani
> > 
> > > Do you know by chance, why do we differ from the vendor dtsi?
> > > 
> > > > +			msi-map-mask = <0xff00>;
> > > >    			interrupt-map-mask = <0 0 0 0x7>;
> > > >    			interrupt-map = <0 0 0 1 &intc 0 0 0 149 IRQ_TYPE_LEVEL_HIGH>, /* int_a */
> > > >    					<0 0 0 2 &intc 0 0 0 150 IRQ_TYPE_LEVEL_HIGH>, /* int_b */
> > > > @@ -1842,9 +1846,13 @@ pcie1: pci@1c08000 {
> > > >    			ranges = <0x01000000 0x0 0x40200000 0 0x40200000 0x0 0x100000>,
> > > >    				 <0x02000000 0x0 0x40300000 0 0x40300000 0x0 0x1fd00000>;
> > > > -			interrupts = <GIC_SPI 307 IRQ_TYPE_LEVEL_HIGH>;
> > > > -			interrupt-names = "msi";
> > > > -			#interrupt-cells = <1>;
> > > > +			/*
> > > > +			 * MSIs for BDF (1:0.0) only works with Device ID 0x5a00.
> > > > +			 * Hence, the IDs are swapped.
> > > > +			 */
> > > > +			msi-map = <0x0 &gic_its 0x5a01 0x1>,
> > > > +				  <0x100 &gic_its 0x5a00 0x1>;
> > > > +			msi-map-mask = <0xff00>;
> > > >    			interrupt-map-mask = <0 0 0 0x7>;
> > > >    			interrupt-map = <0 0 0 1 &intc 0 0 0 434 IRQ_TYPE_LEVEL_HIGH>, /* int_a */
> > > >    					<0 0 0 2 &intc 0 0 0 435 IRQ_TYPE_LEVEL_HIGH>, /* int_b */
> > > 
> > > -- 
> > > With best wishes
> > > Dmitry
> > > 
> > 
> 
> -- 
> With best wishes
> Dmitry
>