| Message ID | 20190531035348.7194-1-elder@linaro.org |
|---|---|
| Headers | show |
| Series | net: introduce Qualcomm IPA driver | expand |
On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: > This series presents the driver for the Qualcomm IP Accelerator > (IPA). > > This is version 2 of the series. This version has addressed almost > all of the feedback received in the first version: > > https://lore.kernel.org/lkml/20190512012508.10608-1-elder@linaro.org/ > More detail is included in the individual patches, but here is a > high-level summary of what's changed since then: > - Two spinlocks have been removed. > - The code for enabling and disabling endpoint interrupts has > been simplified considerably, and the spinlock is no longer > required > - A spinlock used when updating ring buffer pointers is no > longer needed. Integers indexing the ring are used instead > (and they don't even have to be atomic). > - One spinlock remains to protect list updates, but it is always > acquired using spin_lock_bh() (no more irqsave). > - Information about the queueing and completion of messages is now > supplied to the network stack in batches rather than one at a > time. > - I/O completion handling has been simplified, with the IRQ > handler now consisting mainly of disabling the interrupt and > calling napi_schedule(). > - Some comments have been updated and improved througout. > > What follows is the introduction supplied with v1 of the series. > > ----- > > The IPA is a component present in some Qualcomm SoCs that allows > network functions such as aggregation, filtering, routing, and NAT > to be performed without active involvement of the main application > processor (AP). > > Initially, these advanced features are disabled; the IPA driver > simply provides a network interface that makes the modem's LTE > network available to the AP. In addition, only support for the > IPA found in the Qualcomm SDM845 SoC is provided. My question from the Nov 2018 IPA rmnet driver still stands; how does this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is really just a netdev talking to the IPA itself and unrelated to net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- culting rmnet around just because it happens to be a net driver for a QC SoC. Is the firmware that the driver loads already in linux-firmware or going to be there soon? How does the driver support multiple PDNs (eg PDP or EPS contexts) that are enabled through the control plane via QMI messages? I couldn't quite find that out. Thanks, Dan > This code is derived from a driver developed internally by Qualcomm. > A version of the original source can be seen here: > https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree > in the "drivers/platform/msm/ipa" directory. Many were involved in > developing this, but the following individuals deserve explicit > acknowledgement for their substantial contributions: > > Abhishek Choubey > Ady Abraham > Chaitanya Pratapa > David Arinzon > Ghanim Fodi > Gidon Studinski > Ravi Gummadidala > Shihuan Liu > Skylar Chang > > A version of this code was posted in November 2018 as an RFC. > > https://lore.kernel.org/lkml/20181107003250.5832-1-elder@linaro.org/ > All feedback received was addressed. The code has undergone > considerable further rework since that time, and most of the > "future work" described then has now been completed. > > This code is available in buildable form here, based on kernel > v5.2-rc1: > remote: ssh://git@git.linaro.org/people/alex.elder/linux.git > branch: ipa-v2_kernel-v5.2-rc2 > 75adf2ac1266 arm64: defconfig: enable build of IPA code > > The branch depends on a commit now found in in net-next. It has > been cherry-picked, and (in this branch) has this commit ID: > 13c627b5a078 net: qualcomm: rmnet: Move common struct definitions > to include > by > > -Alex > > Alex Elder (17): > bitfield.h: add FIELD_MAX() and field_max() > dt-bindings: soc: qcom: add IPA bindings > soc: qcom: ipa: main code > soc: qcom: ipa: configuration data > soc: qcom: ipa: clocking, interrupts, and memory > soc: qcom: ipa: GSI headers > soc: qcom: ipa: the generic software interface > soc: qcom: ipa: GSI transactions > soc: qcom: ipa: IPA interface to GSI > soc: qcom: ipa: IPA endpoints > soc: qcom: ipa: immediate commands > soc: qcom: ipa: IPA network device and microcontroller > soc: qcom: ipa: AP/modem communications > soc: qcom: ipa: support build of IPA code > MAINTAINERS: add entry for the Qualcomm IPA driver > arm64: dts: sdm845: add IPA information > arm64: defconfig: enable build of IPA code > > .../devicetree/bindings/net/qcom,ipa.yaml | 180 ++ > MAINTAINERS | 6 + > arch/arm64/boot/dts/qcom/sdm845.dtsi | 51 + > arch/arm64/configs/defconfig | 1 + > drivers/net/Kconfig | 2 + > drivers/net/Makefile | 1 + > drivers/net/ipa/Kconfig | 16 + > drivers/net/ipa/Makefile | 7 + > drivers/net/ipa/gsi.c | 1635 > +++++++++++++++++ > drivers/net/ipa/gsi.h | 246 +++ > drivers/net/ipa/gsi_private.h | 148 ++ > drivers/net/ipa/gsi_reg.h | 376 ++++ > drivers/net/ipa/gsi_trans.c | 624 +++++++ > drivers/net/ipa/gsi_trans.h | 116 ++ > drivers/net/ipa/ipa.h | 131 ++ > drivers/net/ipa/ipa_clock.c | 297 +++ > drivers/net/ipa/ipa_clock.h | 52 + > drivers/net/ipa/ipa_cmd.c | 377 ++++ > drivers/net/ipa/ipa_cmd.h | 116 ++ > drivers/net/ipa/ipa_data-sdm845.c | 245 +++ > drivers/net/ipa/ipa_data.h | 267 +++ > drivers/net/ipa/ipa_endpoint.c | 1283 +++++++++++++ > drivers/net/ipa/ipa_endpoint.h | 97 + > drivers/net/ipa/ipa_gsi.c | 48 + > drivers/net/ipa/ipa_gsi.h | 49 + > drivers/net/ipa/ipa_interrupt.c | 279 +++ > drivers/net/ipa/ipa_interrupt.h | 53 + > drivers/net/ipa/ipa_main.c | 921 ++++++++++ > drivers/net/ipa/ipa_mem.c | 234 +++ > drivers/net/ipa/ipa_mem.h | 83 + > drivers/net/ipa/ipa_netdev.c | 251 +++ > drivers/net/ipa/ipa_netdev.h | 24 + > drivers/net/ipa/ipa_qmi.c | 402 ++++ > drivers/net/ipa/ipa_qmi.h | 35 + > drivers/net/ipa/ipa_qmi_msg.c | 583 ++++++ > drivers/net/ipa/ipa_qmi_msg.h | 238 +++ > drivers/net/ipa/ipa_reg.h | 279 +++ > drivers/net/ipa/ipa_smp2p.c | 304 +++ > drivers/net/ipa/ipa_smp2p.h | 47 + > drivers/net/ipa/ipa_uc.c | 208 +++ > drivers/net/ipa/ipa_uc.h | 32 + > include/linux/bitfield.h | 14 + > 42 files changed, 10358 insertions(+) > create mode 100644 > Documentation/devicetree/bindings/net/qcom,ipa.yaml > create mode 100644 drivers/net/ipa/Kconfig > create mode 100644 drivers/net/ipa/Makefile > create mode 100644 drivers/net/ipa/gsi.c > create mode 100644 drivers/net/ipa/gsi.h > create mode 100644 drivers/net/ipa/gsi_private.h > create mode 100644 drivers/net/ipa/gsi_reg.h > create mode 100644 drivers/net/ipa/gsi_trans.c > create mode 100644 drivers/net/ipa/gsi_trans.h > create mode 100644 drivers/net/ipa/ipa.h > create mode 100644 drivers/net/ipa/ipa_clock.c > create mode 100644 drivers/net/ipa/ipa_clock.h > create mode 100644 drivers/net/ipa/ipa_cmd.c > create mode 100644 drivers/net/ipa/ipa_cmd.h > create mode 100644 drivers/net/ipa/ipa_data-sdm845.c > create mode 100644 drivers/net/ipa/ipa_data.h > create mode 100644 drivers/net/ipa/ipa_endpoint.c > create mode 100644 drivers/net/ipa/ipa_endpoint.h > create mode 100644 drivers/net/ipa/ipa_gsi.c > create mode 100644 drivers/net/ipa/ipa_gsi.h > create mode 100644 drivers/net/ipa/ipa_interrupt.c > create mode 100644 drivers/net/ipa/ipa_interrupt.h > create mode 100644 drivers/net/ipa/ipa_main.c > create mode 100644 drivers/net/ipa/ipa_mem.c > create mode 100644 drivers/net/ipa/ipa_mem.h > create mode 100644 drivers/net/ipa/ipa_netdev.c > create mode 100644 drivers/net/ipa/ipa_netdev.h > create mode 100644 drivers/net/ipa/ipa_qmi.c > create mode 100644 drivers/net/ipa/ipa_qmi.h > create mode 100644 drivers/net/ipa/ipa_qmi_msg.c > create mode 100644 drivers/net/ipa/ipa_qmi_msg.h > create mode 100644 drivers/net/ipa/ipa_reg.h > create mode 100644 drivers/net/ipa/ipa_smp2p.c > create mode 100644 drivers/net/ipa/ipa_smp2p.h > create mode 100644 drivers/net/ipa/ipa_uc.c > create mode 100644 drivers/net/ipa/ipa_uc.h >
From: Alex Elder <elder@linaro.org> Date: Thu, 30 May 2019 22:53:34 -0500 > + void *route_virt; ... > + void *filter_virt; ... If these are arrays of u64's, please declare them as "u64 *" instead of the opaque "void *".
On Fri 31 May 12:19 PDT 2019, Arnd Bergmann wrote: > On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: > > On 5/31/19 9:58 AM, Dan Williams wrote: > > > On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: [..] > > So basically, the purpose of the rmnet driver is to handle QMAP > > protocol connections, and right now that's what the modem provides. > > Do you have any idea why this particular design was picked? > From what I've seen of QMAP it seems like a reasonable design choice to have a software component (rmnet) dealing with this, separate from the transport. And I think IPA is the 4th or 5th mechanism for transporting QMAP packets back and forth to the modem. Downstream rmnet is copyright 2007-, and I know of interest in bringing at least one of the other transports upstream. Regards, Bjorn
On 2019-05-31 17:33, Bjorn Andersson wrote: > On Fri 31 May 13:47 PDT 2019, Alex Elder wrote: > >> On 5/31/19 2:19 PM, Arnd Bergmann wrote: >> > On Fri, May 31, 2019 at 6:36 PM Alex Elder <elder@linaro.org> wrote: >> >> On 5/31/19 9:58 AM, Dan Williams wrote: >> >>> On Thu, 2019-05-30 at 22:53 -0500, Alex Elder wrote: >> >>> >> >>> My question from the Nov 2018 IPA rmnet driver still stands; how does >> >>> this relate to net/ethernet/qualcomm/rmnet/ if at all? And if this is >> >>> really just a netdev talking to the IPA itself and unrelated to >> >>> net/ethernet/qualcomm/rmnet, let's call it "ipa%d" and stop cargo- >> >>> culting rmnet around just because it happens to be a net driver for a >> >>> QC SoC. >> >> >> >> First, the relationship between the IPA driver and the rmnet driver >> >> is that the IPA driver is assumed to sit between the rmnet driver >> >> and the hardware. >> > >> > Does this mean that IPA can only be used to back rmnet, and rmnet >> > can only be used on top of IPA, or can or both of them be combined >> > with another driver to talk to instead? >> >> No it does not mean that. >> >> As I understand it, one reason for the rmnet layer was to abstract >> the back end, which would allow using a modem, or using something >> else (a LAN?), without exposing certain details of the hardware. >> (Perhaps to support multiplexing, etc. without duplicating that >> logic in two "back-end" drivers?) >> >> To be perfectly honest, at first I thought having IPA use rmnet >> was a cargo cult thing like Dan suggested, because I didn't see >> the benefit. I now see why one would use that pass-through layer >> to handle the QMAP features. >> >> But back to your question. The other thing is that I see no >> reason the IPA couldn't present a "normal" (non QMAP) interface >> for a modem. It's something I'd really like to be able to do, >> but I can't do it without having the modem firmware change its >> configuration for these endpoints. My access to the people who >> implement the modem firmware has been very limited (something >> I hope to improve), and unless and until I can get corresponding >> changes on the modem side to implement connections that don't >> use QMAP, I can't implement such a thing. >> > > But any such changes would either be years into the future or for > specific devices and as such not applicable to any/most of devices on > the market now or in the coming years. > > > But as Arnd points out, if the software split between IPA and rmnet is > suboptimal your are encouraged to fix that. > > Regards, > Bjorn The split rmnet design was chosen because we could place rmnet over any transport - IPA, PCIe (https://lkml.org/lkml/2018/4/26/1159) or USB. rmnet registers a rx handler, so the rmnet packet processing itself happens in the same softirq when packets are queued to network stack by IPA. -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
On 6/3/19 9:54 AM, Dan Williams wrote: >> To be perfectly honest, at first I thought having IPA use rmnet >> was a cargo cult thing like Dan suggested, because I didn't see > To be clear I only meant cargo-culting the naming, not any > functionality. Clearly IPA/rmnet/QMAP are pretty intimately connected > at this point. But this goes back to whether IPA needs a netdev itself > or whether you need an rmnet device created on top. If the former then > I'd say no cargo-culting, if the later then it's a moot point because > the device name will be rmnet%d anyway. OK I thought you weren't sure why rmnet was a layer at all. As I said, I didn't have a very good understanding of why it was even needed when I first started working on this. I can't (or won't) comment right now on whether IPA needs its own netdev for rmnet to use. The IPA endpoints used for the modem network interfaces are enabled when the netdev is opened and disabled when closed. Outside of that, TX and RX are pretty much immediately passed through to the layer below or above. IPA currently has no other net device operations. -Alex
On Tue, 2019-06-04 at 10:13 +0200, Arnd Bergmann wrote: > On Mon, Jun 3, 2019 at 3:32 PM Alex Elder <elder@linaro.org> wrote: > > On 6/3/19 5:04 AM, Arnd Bergmann wrote: > > > On Sat, Jun 1, 2019 at 1:59 AM Subash Abhinov Kasiviswanathan > > > > > > - What I'm worried about most here is the flow control handling > > > on the > > > transmit side. The IPA driver now uses the modern BQL method to > > > control how much data gets submitted to the hardware at any > > > time. > > > The rmnet driver also uses flow control using the > > > rmnet_map_command() function, that blocks tx on the higher > > > level device when the remote side asks us to. > > > I fear that doing flow control for a single physical device on > > > two > > > separate netdev instances is counterproductive and confuses > > > both sides. > > > > I understand what you're saying here, and instinctively I think > > you're right. > > > > But BQL manages the *local* interface's ability to get rid of > > packets, whereas the QMAP flow control is initiated by the other > > end of the connection (the modem in this case). > > > > With multiplexing, it's possible that one of several logical > > devices on the modem side has exhausted a resource and must > > ask the source of the data on the host side to suspend the > > flow. Meanwhile the other logical devices sharing the physical > > link might be fine, and should not be delayed by the first one. > > > > It is the multiplexing itself that confuses the BQL algorithm. > > The abstraction obscures the *real* rates at which individual > > logical connections are able to transmit data. > > I would assume that the real rate constantly changes, at least > for wireless interfaces that are also shared with other users > on the same network. BQL is meant to deal with that, at least > when using a modern queuing algorithm. > > > Even if the multiple logical interfaces implemented BQL, they > > would not get the feedback they need directly from the IPA > > driver, because transmitting over the physical interface might > > succeed even if the logical interface on the modem side can't > > handle more data. So I think the flow control commands may be > > necessary, given multiplexing. > > Can you describe what kind of multiplexing is actually going on? > I'm still unclear about what we actually use multiple logical > interfaces for here, and how they relate to one another. Each logical interface represents a different "connection" (PDP/EPS context) to the provider network with a distinct IP address and QoS. VLANs may be a suitable analogy but here they are L3+QoS. In realistic example the main interface (say rmnet0) would be used for web browsing and have best-effort QoS. A second interface (say rmnet1) would be used for VOIP and have certain QoS guarantees from both the modem and the network itself. QMAP can also aggregate frames for a given channel (connection/EPS/PDP context/rmnet interface/etc) to better support LTE speeds. Dan > > The rmnet driver could use BQL, and could return NETDEV_TX_BUSY > > for a logical interface when its TX flow has been stopped by a > > QMAP command. That way the feedback for BQL on the logical > > interfaces would be provided in the right place. > > > > I have no good intuition about the interaction between > > two layered BQL managed queues though. > > Returning NETDEV_TX_BUSY is usually a bad idea as that > leads to unnecessary frame drop. > > I do think that using BQL and the QMAP flow command on > the /same/ device would be best, as that throttles the connection > when either of the two algorithms wants us to slow down. > > The question is mainly which of the two devices that should be. > Doing it in the ipa driver is probably easier to implement here, > but ideally I think we'd only have a single queue visible to the > network stack, if we can come up with a way to do that. > > > > - I was a little confused by the location of the rmnet driver in > > > drivers/net/ethernet/... More conventionally, I think as a > > > protocol > > > handler it should go into net/qmap/, with the ipa driver going > > > into drivers/net/qmap/ipa/, similar to what we have fo > > > ethernet, > > > wireless, ppp, appletalk, etc. > > > > > > - The rx_handler uses gro_cells, which as I understand is meant > > > for generic tunnelling setups and takes another loop through > > > NAPI to aggregate data from multiple queues, but in case of > > > IPA's single-queue receive calling gro directly would be > > > simpler > > > and more efficient. > > > > I have been planning to investigate some of the generic GRO > > stuff for IPA but was going to wait on that until the basic > > code was upstream. > > That's ok, that part can easily be changed after the fact, as it > does not impact the user interface or the general design. > > > > From the overall design and the rmnet Kconfig description, it > > > appears as though the intention as that rmnet could be a > > > generic wrapper on top of any device, but from the > > > implementation it seems that IPA is not actually usable that > > > way and would always go through IPA. > > > > As far as I know *nothing* upstream currently uses rmnet; the > > IPA driver will be the first, but as Bjorn said others seem to > > be on the way. I'm not sure what you mean by "IPA is not > > usable that way." Currently the IPA driver assumes a fixed > > configuration, and that configuration assumes the use of QMAP, > > and therefore assumes the rmnet driver is layered above it. > > That doesn't preclude rmnet from using a different back end. > > Yes, that's what I meant above: IPA can only be used through > rmnet (I wrote "through IPA", sorry for the typo), but cannot be > used by itself. > > Arnd
On Tue, 2019-06-04 at 22:04 +0200, Arnd Bergmann wrote: > On Tue, Jun 4, 2019 at 5:18 PM Dan Williams <dcbw@redhat.com> wrote: > > On Tue, 2019-06-04 at 10:13 +0200, Arnd Bergmann wrote: > > > Can you describe what kind of multiplexing is actually going on? > > > I'm still unclear about what we actually use multiple logical > > > interfaces for here, and how they relate to one another. > > > > Each logical interface represents a different "connection" (PDP/EPS > > context) to the provider network with a distinct IP address and > > QoS. > > VLANs may be a suitable analogy but here they are L3+QoS. > > > > In realistic example the main interface (say rmnet0) would be used > > for > > web browsing and have best-effort QoS. A second interface (say > > rmnet1) > > would be used for VOIP and have certain QoS guarantees from both > > the > > modem and the network itself. > > > > QMAP can also aggregate frames for a given channel > > (connection/EPS/PDP > > context/rmnet interface/etc) to better support LTE speeds. > > Thanks, that's a very helpful explanation! > > Is it correct to say then that the concept of having those separate > connections would be required for any proper LTE modem > implementation, > but the QMAP protocol (and based on that, the rmnet implementation) > is Qualcomm specific and shared only among several generations of > modems from that one vendor? Exactly correct. This is what Johannes is discussing in his "cellular modem APIs - take 2" thread about how this should all be organized at the driver level and I think we should figure that out before we commit to IPA-with-a-useless-netdev that requires rmnets to be created on top. That may end up being the solution but let's have that discussion. > > You mentioned the need to have a common user space interface > for configuration, and if the above is true, I agree that we should > try > to achieve that, either by ensuring rmnet is generic enough to > cover other vendors (and non-QMAP clients), or by creating a > new user level interface that IPA/rmnet can be adapted to. I would not suggest making rmnet generic; it's pretty QMAP specific (but QMAP is spoken by many many modems both SoC, USB stick, and PCIe minicard). Instead, I think what Johannes is discussing is a better approach. A kernel WWAN framework with consistent user API that rmnet/IPA/qmi_wwan/MBIM/QMI/serial/Sierra can all implement. That wouldn't affect the core packet processing of IPA/rmnet but instead: 1) when/how an rmnet device actually gets created on top of the IPA (or qmi_wwan) device AND (one of these two) a) whether IPA creates a netdev on probe OR b) whether there is some "WWAN device" kernel object which userspace interacts with create rmnet channels on top of IPA Dan
On 6/4/19 4:29 PM, Dan Williams wrote: > On Tue, 2019-06-04 at 22:04 +0200, Arnd Bergmann wrote: >> On Tue, Jun 4, 2019 at 5:18 PM Dan Williams <dcbw@redhat.com> wrote: >>> On Tue, 2019-06-04 at 10:13 +0200, Arnd Bergmann wrote: >>>> Can you describe what kind of multiplexing is actually going on? >>>> I'm still unclear about what we actually use multiple logical >>>> interfaces for here, and how they relate to one another. >>> >>> Each logical interface represents a different "connection" (PDP/EPS >>> context) to the provider network with a distinct IP address and >>> QoS. >>> VLANs may be a suitable analogy but here they are L3+QoS. >>> >>> In realistic example the main interface (say rmnet0) would be used >>> for >>> web browsing and have best-effort QoS. A second interface (say >>> rmnet1) >>> would be used for VOIP and have certain QoS guarantees from both >>> the >>> modem and the network itself. >>> >>> QMAP can also aggregate frames for a given channel >>> (connection/EPS/PDP >>> context/rmnet interface/etc) to better support LTE speeds. >> >> Thanks, that's a very helpful explanation! >> >> Is it correct to say then that the concept of having those separate >> connections would be required for any proper LTE modem >> implementation, >> but the QMAP protocol (and based on that, the rmnet implementation) >> is Qualcomm specific and shared only among several generations of >> modems from that one vendor? > > Exactly correct. This is what Johannes is discussing in his "cellular > modem APIs - take 2" thread about how this should all be organized at > the driver level and I think we should figure that out before we commit > to IPA-with-a-useless-netdev that requires rmnets to be created on top. > That may end up being the solution but let's have that discussion. I looked at Johannes' message and the follow-on discussion. As I've made clear before, my work on this has been focused on the IPA transport, and some of this higher-level LTE architecture is new to me. But it seems pretty clear that an abstracted WWAN subsystem is a good plan, because these devices represent a superset of what a "normal" netdev implements. HOWEVER I disagree with your suggestion that the IPA code should not be committed until after that is all sorted out. In part it's for selfish reasons, but I think there are legitimate reasons to commit IPA now *knowing* that it will need to be adapted to fit into the generic model that gets defined and developed. Here are some reasons why. First, the layer of the IPA code that actually interacts with rmnet is very small--less than 3% if you simply do a word count of the source files. Arnd actually suggested eliminating the "ipa_netdev" files and merging their content elsewhere. This suggests two things: - The interface to rmnet is isolated, so the effect of whatever updates are made to support a WWAN (rather than netdev) model will be focused - The vast majority of the driver has nothing to do with that upper layer, and deals almost exclusively with managing the IPA hardware. The idea of a generic framework isn't minor, but it isn't the main focus of the IPA driver either, so I don't think it should hold it up. Second, the IPA code has been out for review recently, and has been the subject of some detailed discussion in the past few weeks. Arnd especially has invested considerable time in review and discussion. Delaying things until after a better generic model is settled on (which I'm guessing might be on the order of months) means the IPA driver will have to be submitted for review again after that, at which point it will no longer be "fresh"; it'll be a bit like starting over. Third, having the code upstream actually means the actual requirements for rmnet-over-IPA are clear and explicit. This might not be a huge deal, but I think it's better to devise a generic WWAN scheme that can refer to actual code than to do so with assumptions about what will work with rmnet (and others). As far as I know, the upstream rmnet has no other upstream back end; IPA will make it "real." I support the idea of developing a generic WWAN framework, and I can assure you I'll be involved enough to perhaps be one of the first to implement a new generic scheme. Optimistically, the IPA code itself hasn't seen much feedback for v2; maybe that means it's in good shape? Anyway, I'd obviously like to get the IPA code accepted sooner rather than later, and I think there are good reasons to do that. -Alex >>> You mentioned the need to have a common user space interface >> for configuration, and if the above is true, I agree that we should >> try >> to achieve that, either by ensuring rmnet is generic enough to >> cover other vendors (and non-QMAP clients), or by creating a >> new user level interface that IPA/rmnet can be adapted to. > > I would not suggest making rmnet generic; it's pretty QMAP specific > (but QMAP is spoken by many many modems both SoC, USB stick, and PCIe > minicard). > > Instead, I think what Johannes is discussing is a better approach. A > kernel WWAN framework with consistent user API that > rmnet/IPA/qmi_wwan/MBIM/QMI/serial/Sierra can all implement. > > That wouldn't affect the core packet processing of IPA/rmnet but > instead: > > 1) when/how an rmnet device actually gets created on top of the IPA (or > qmi_wwan) device > > AND (one of these two) > > a) whether IPA creates a netdev on probe > > OR > > b) whether there is some "WWAN device" kernel object which userspace > interacts with create rmnet channels on top of IPA > > Dan >
On Thu, May 30, 2019 at 9:53 PM Alex Elder <elder@linaro.org> wrote: > > Add the binding definitions for the "qcom,ipa" device tree node. > > Signed-off-by: Alex Elder <elder@linaro.org> > --- > .../devicetree/bindings/net/qcom,ipa.yaml | 180 ++++++++++++++++++ > 1 file changed, 180 insertions(+) > create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml > > diff --git a/Documentation/devicetree/bindings/net/qcom,ipa.yaml b/Documentation/devicetree/bindings/net/qcom,ipa.yaml > new file mode 100644 > index 000000000000..0037fc278a61 > --- /dev/null > +++ b/Documentation/devicetree/bindings/net/qcom,ipa.yaml > @@ -0,0 +1,180 @@ > +# SPDX-License-Identifier: GPL-2.0 New bindings are preferred to be dual GPL-2.0 and BSD-2-Clause. But that's really a decision for the submitter. Reviewed-by: Rob Herring <robh@kernel.org>
On Tue, Jun 11, 2019 at 10:12 AM Johannes Berg <johannes@sipsolutions.net> wrote: > > As I've made clear before, my work on this has been focused on the IPA transport, > > and some of this higher-level LTE architecture is new to me. But it > > seems pretty clear that an abstracted WWAN subsystem is a good plan, > > because these devices represent a superset of what a "normal" netdev > > implements. > > I'm not sure I'd actually call it a superset. By themselves, these > netdevs are actually completely useless to the network stack, AFAICT. > Therefore, the overlap with netdevs you can really use with the network > stack is pretty small? I think Alex meant the concept of having a type of netdev with a generic user space interface for wwan and similar to a wlan device, as I understood you had suggested as well, as opposed to a stacked device as in rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/. > > HOWEVER I disagree with your suggestion that the IPA code should > > not be committed until after that is all sorted out. In part it's > > for selfish reasons, but I think there are legitimate reasons to > > commit IPA now *knowing* that it will need to be adapted to fit > > into the generic model that gets defined and developed. Here > > are some reasons why. > > I can't really argue with those, though I would point out that the > converse also holds - if we commit to this now, then we will have to > actually keep the API offered by IPA/rmnet today, so we cannot actually > remove the netdev again, even if we do migrate it to offer support for a > WWAN framework in the future. Right. The interface to support rmnet might be simple enough to keep next to what becomes the generic interface, but it will always continue to be an annoyance. > > Second, the IPA code has been out for review recently, and has been > > the subject of some detailed discussion in the past few weeks. Arnd > > especially has invested considerable time in review and discussion. > > Delaying things until after a better generic model is settled on > > (which I'm guessing might be on the order of months) > > > I dunno if it really has to be months. I think we can cobble something > together relatively quickly that addresses the needs of IPA more > specifically, and then extend later? > > But OTOH it may make sense to take a more paced approach and think > about the details more carefully than we have over in the other thread so far. I would hope that as soon as we can agree on a general approach, it would also be possible to merge a minimal implementation into the kernel along with IPA. Alex already mentioned that IPA in its current state does not actually support more than one data channel, so the necessary setup for it becomes even simpler. At the moment, the rmnet configuration in include/uapi/linux/if_link.h is almost trivial, with the three pieces of information needed being an IFLA_LINK to point to the real device (not needed if there is only one device per channel, instead of two), the IFLA_RMNET_MUX_ID setting the ID of the muxing channel (not needed if there is only one channel ?), a way to specify software bridging between channels (not useful if there is only one channel) and a few flags that I assume must match the remote end: #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) enum { IFLA_RMNET_UNSPEC, IFLA_RMNET_MUX_ID, IFLA_RMNET_FLAGS, __IFLA_RMNET_MAX, }; #define IFLA_RMNET_MAX (__IFLA_RMNET_MAX - 1) struct ifla_rmnet_flags { __u32 flags; __u32 mask; }; > > Third, having the code upstream actually means the actual requirements > > for rmnet-over-IPA are clear and explicit. This might not be a huge > > deal, but I think it's better to devise a generic WWAN scheme that > > can refer to actual code than to do so with assumptions about what > > will work with rmnet (and others). As far as I know, the upstream > > rmnet has no other upstream back end; IPA will make it "real." > > Is that really true? I had previously been told that rmnet actually does > have use with a few existing drivers. > > > If true though, then I think this would be the killer argument *in > favour* of *not* merging this - because that would mean we *don't* have > to actually keep the rmnet API around for all foreseeable future. I would agree with that. From the code I can see no other driver including the rmnet protocol header (see the discussion about moving the header to include/linux in order to merge ipa), and I don't see any other driver referencing ETH_P_MAP either. My understanding is that any driver used by rmnet would require both, but they are all out-of-tree at the moment. Arnd
On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: > On Tue, Jun 11, 2019 at 10:12 AM Johannes Berg > <johannes@sipsolutions.net> wrote: > > > > As I've made clear before, my work on this has been focused on > > > the IPA transport, > > > and some of this higher-level LTE architecture is new to me. But > > > it > > > seems pretty clear that an abstracted WWAN subsystem is a good > > > plan, > > > because these devices represent a superset of what a "normal" > > > netdev > > > implements. > > > > I'm not sure I'd actually call it a superset. By themselves, these > > netdevs are actually completely useless to the network stack, > > AFAICT. > > Therefore, the overlap with netdevs you can really use with the > > network > > stack is pretty small? > > I think Alex meant the concept of having a type of netdev with a > generic > user space interface for wwan and similar to a wlan device, as I > understood > you had suggested as well, as opposed to a stacked device as in > rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, > ...)/. > > > > HOWEVER I disagree with your suggestion that the IPA code should > > > not be committed until after that is all sorted out. In part > > > it's > > > for selfish reasons, but I think there are legitimate reasons to > > > commit IPA now *knowing* that it will need to be adapted to fit > > > into the generic model that gets defined and developed. Here > > > are some reasons why. > > > > I can't really argue with those, though I would point out that the > > converse also holds - if we commit to this now, then we will have > > to > > actually keep the API offered by IPA/rmnet today, so we cannot > > actually > > remove the netdev again, even if we do migrate it to offer support > > for a > > WWAN framework in the future. > > Right. The interface to support rmnet might be simple enough to keep > next to what becomes the generic interface, but it will always > continue > to be an annoyance. > > > > Second, the IPA code has been out for review recently, and has > > > been > > > the subject of some detailed discussion in the past few > > > weeks. Arnd > > > especially has invested considerable time in review and > > > discussion. > > > Delaying things until after a better generic model is settled on > > > (which I'm guessing might be on the order of months) > > > > I dunno if it really has to be months. I think we can cobble > > something > > together relatively quickly that addresses the needs of IPA more > > specifically, and then extend later? > > > > But OTOH it may make sense to take a more paced approach and think > > about the details more carefully than we have over in the other > > thread so far. > > I would hope that as soon as we can agree on a general approach, it > would also be possible to merge a minimal implementation into the > kernel > along with IPA. Alex already mentioned that IPA in its current state > does > not actually support more than one data channel, so the necessary > setup for it becomes even simpler. > > At the moment, the rmnet configuration in > include/uapi/linux/if_link.h > is almost trivial, with the three pieces of information needed being > an IFLA_LINK to point to the real device (not needed if there is only > one device per channel, instead of two), the IFLA_RMNET_MUX_ID > setting the ID of the muxing channel (not needed if there is only > one channel ?), a way to specify software bridging between channels > (not useful if there is only one channel) and a few flags that I > assume > must match the remote end: > > #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) > #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) > #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) > #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) > enum { > IFLA_RMNET_UNSPEC, > IFLA_RMNET_MUX_ID, > IFLA_RMNET_FLAGS, > __IFLA_RMNET_MAX, > }; > #define IFLA_RMNET_MAX (__IFLA_RMNET_MAX - 1) > struct ifla_rmnet_flags { > __u32 flags; > __u32 mask; > }; > > > > Third, having the code upstream actually means the actual > > > requirements > > > for rmnet-over-IPA are clear and explicit. This might not be a > > > huge > > > deal, but I think it's better to devise a generic WWAN scheme > > > that > > > can refer to actual code than to do so with assumptions about > > > what > > > will work with rmnet (and others). As far as I know, the > > > upstream > > > rmnet has no other upstream back end; IPA will make it "real." > > > > Is that really true? I had previously been told that rmnet actually > > does > > have use with a few existing drivers. > > > > > > If true though, then I think this would be the killer argument *in > > favour* of *not* merging this - because that would mean we *don't* > > have > > to actually keep the rmnet API around for all foreseeable future. > > I would agree with that. From the code I can see no other driver > including the rmnet protocol header (see the discussion about moving > the header to include/linux in order to merge ipa), and I don't see > any other driver referencing ETH_P_MAP either. My understanding > is that any driver used by rmnet would require both, but they are > all out-of-tree at the moment. The general plan (and I believe Daniele Palmas was working on it) was to eventually make qmi_wwan use rmnet rather than its internal sysfs- based implementation. qmi_wwan and ipa are at essentially the same level and both could utilize rmnet on top. *That's* what I'd like to see. I don't want to see two different ways to get QMAP packets to modem firmware from two different drivers that really could use the same code. Dan
On Wed, 2019-06-12 at 10:31 +0200, Arnd Bergmann wrote: > On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <dcbw@redhat.com> wrote: > > On Tue, 2019-06-11 at 10:52 -0600, Subash Abhinov Kasiviswanathan > > wrote: > > > > rmnet should handle muxing the QMAP, QoS, and aggregation and pass > > the > > resulting packet to the lower layer. That lower layer could be IPA > > or > > qmi_wwan, which in turn passes that QMAP packet to USB or GSI or > > whatever. This is typically how Linux handles clean abstractions > > between different protocol layers in drivers. > > > > Similar to some WiFi drivers (drivers/net/wireless/marvell/libertas > > for > > example) where the same firmware interface can be accessed via PCI, > > SDIO, USB, SPI, etc. The bus-specific code is self-contained and > > does > > not creep into the upper more generic parts. > > Yes, I think that is a good model. In case of libertas, we have > multiple > layers inheritence from the basic device (slightly different in the > implementation, > but that is how it should be): To be clear (and I probably wasn't earlier) I wasn't talking as deep about the actual code structures as you are here but this a great discussion. I was trying to make the point that rmnet doesn't need to care about how the QMAP packets get to the device itself; it can be pretty generic so that it can be used by IPA/qmi_wwan/rmnet_smd/etc. Your points below are a great discussion though... > struct if_cs_card { /* pcmcia specific */ > struct lbs_private { /* libertas specific */ > struct wireless_dev { /* 802.11 specific */ > struct net_device { > struct device { > ... > }; > ... > }; > ... > }; > ... > }; > ... > }; > The outer structure gets allocated when probing the hardware specific > driver, and everything below it is implemented as direct function > calls > into the more generic code, or as function pointers into the more > specific > code. > > The current rmnet model is different in that by design the upper > layer > (rmnet) and the lower layer (qmi_wwan, ipa, ...) are kept independent > in > both directions, i.e. ipa has (almost) no knowledge of rmnet, and > just > has pointers to the other net_device: > > ipa_device > net_device > > rmnet_port > net_device > > I understand that the rmnet model was intended to provide a cleaner > abstraction, but it's not how we normally structure subsystems in > Linux, and moving to a model more like how wireless_dev works > would improve both readability and performance, as you describe > it, it would be more like (ignoring for now the need for multiple > connections): > > ipa_dev > rmnet_dev > wwan_dev > net_device Perhaps I'm assuming too much from this diagram but this shows a 1:1 between wwan_dev and "lower" devices. What Johannes is proposing (IIRC) is something a bit looser where a wwan_dev does not necessarily provide netdev itself, but is instead the central point that various channels (control, data, gps, sim card, etc) register with. That way the wwan_dev can provide an overall view of the WWAN device to userspace, and userspace can talk to the wwan_dev to ask the lower drivers (ipa, rmnet, etc) to create new channels (netdev, tty, otherwise) when the control channel has told the modem firmware to expect one. For example, say you have told the firmware to create a new data channel with ID 5 via QMI (which the kernel is unaware of because it does not process higher-level QMI requests). Perhaps (and this is all just brainstorming) then userspace asks the wwan_dev to create a new data channel with ID 5 and a certain QoS. IPA (or rmnet because that's the data channel provider for IPA) has registered callbacks to the wwan_dev, receives this request, and creates a new rmnet_dev/net_device, and then the wwan_dev passes the ifindex back to userspace so we don't have to play the dance of "match up my request with a random netlink ADD event". The point being that since data channels aren't actually useful until the control channel agrees with the firmware that one should exist, if we have a wwan_dev that represents the entire WWAN device then we don't need the initial-but-useless net_device. Just some thoughts; Johannes can feel free to correct me at any time :) > Where each layer is a specialization of the next. Note: this is a > common change when moving from proprietary code to upstream > code. If a driver module is designed to live out of tree, there > is a strong incentive to limit the number of interfaces it uses, > but when it gets merged, it becomes much more flexible, as > an internal interface between wwan_dev and the hardware driver(s) > can be easily changed by modifying all drivers at once. Yep, I've seen this time and time again. Dan
On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: > On Tue, Jun 11, 2019 at 10:12 AM Johannes Berg > <johannes@sipsolutions.net> wrote: > > > > As I've made clear before, my work on this has been focused on the IPA transport, > > > and some of this higher-level LTE architecture is new to me. But it > > > seems pretty clear that an abstracted WWAN subsystem is a good plan, > > > because these devices represent a superset of what a "normal" netdev > > > implements. > > > > I'm not sure I'd actually call it a superset. By themselves, these > > netdevs are actually completely useless to the network stack, AFAICT. > > Therefore, the overlap with netdevs you can really use with the network > > stack is pretty small? > > I think Alex meant the concept of having a type of netdev with a generic > user space interface for wwan and similar to a wlan device, as I understood > you had suggested as well, as opposed to a stacked device as in > rmnet or those drivers it seems to be modeled after (vlan, ip tunnel, ...)/. I guess. It is indeed currently modelled after the stacked devices, but those regular netdevs are inherently useful by themselves, you don't *have* to tunnel or use VLANs after all. With rmnet, the underlying netdev *isn't* useful by itself, because you're always forced to have the stacked rmnet device on top. > > > HOWEVER I disagree with your suggestion that the IPA code should > > > not be committed until after that is all sorted out. In part it's > > > for selfish reasons, but I think there are legitimate reasons to > > > commit IPA now *knowing* that it will need to be adapted to fit > > > into the generic model that gets defined and developed. Here > > > are some reasons why. > > > > I can't really argue with those, though I would point out that the > > converse also holds - if we commit to this now, then we will have to > > actually keep the API offered by IPA/rmnet today, so we cannot actually > > remove the netdev again, even if we do migrate it to offer support for a > > WWAN framework in the future. > > Right. The interface to support rmnet might be simple enough to keep > next to what becomes the generic interface, but it will always continue > to be an annoyance. Not easily, because fundamentally it requires an underlying netdev to have an ifindex, so it wouldn't just be another API to keep around (which I'd classify as an annoyance) but also a whole separate netdev that's exposed by this IPA driver, for basically this purpose only. > > I dunno if it really has to be months. I think we can cobble something > > together relatively quickly that addresses the needs of IPA more > > specifically, and then extend later? > > > > But OTOH it may make sense to take a more paced approach and think > > about the details more carefully than we have over in the other thread so far. > > I would hope that as soon as we can agree on a general approach, it > would also be possible to merge a minimal implementation into the kernel > along with IPA. Alex already mentioned that IPA in its current state does > not actually support more than one data channel, so the necessary > setup for it becomes even simpler. Interesting, I'm not even sure how the driver can stop multiple channels in the rmnet model? > At the moment, the rmnet configuration in include/uapi/linux/if_link.h > is almost trivial, with the three pieces of information needed being > an IFLA_LINK to point to the real device (not needed if there is only > one device per channel, instead of two), the IFLA_RMNET_MUX_ID > setting the ID of the muxing channel (not needed if there is only > one channel ?), a way to specify software bridging between channels > (not useful if there is only one channel) I think the MUX ID is something we *would* want, and we'd probably want a channel type as well, so as to not paint ourselves into a corner where the default ends up being whatever IPA supports right now. The software bridging is very questionable to start with, I'd advocate not supporting that at all but adding tracepoints or similar if needed for debugging instead. > and a few flags that I assume > must match the remote end: > > #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) > #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) > #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) > #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) I don't really know about these. > > If true though, then I think this would be the killer argument *in > > favour* of *not* merging this - because that would mean we *don't* have > > to actually keep the rmnet API around for all foreseeable future. > > I would agree with that. From the code I can see no other driver > including the rmnet protocol header (see the discussion about moving > the header to include/linux in order to merge ipa), and I don't see > any other driver referencing ETH_P_MAP either. My understanding > is that any driver used by rmnet would require both, but they are > all out-of-tree at the moment. I guess that would mean we have more work to do here, but it also means we don't have to support these interfaces forever. I'm not *entirely* convinced though. rmnet in itself doesn't really seem to require anything from the underlying netdev, so if there's a driver that just blindly passes things through to the hardware expecting the right configuration, we wouldn't really see it this way? OTOH, such a driver would probably blow up completely if somebody tried to use it without rmnet on top, and so it would at least have to check for ETH_P_MAP? johannes
On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: [...] Looking at the flags again, > #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) This one I'm not sure I understand - seems weird to have such a fundamental thing as a *configuration* on the channel. > #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) Similar here? If you have flow control you probably want to use it? > #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) This again looks like a hardware specific feature (ipv4 checksum)? Not sure why this is set by userspace. > #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) This could be set with ethtool instead, I suppose. johannes
On 6/17/19 6:42 AM, Johannes Berg wrote: > On Wed, 2019-06-12 at 17:06 +0200, Arnd Bergmann wrote: >> On Wed, Jun 12, 2019 at 4:28 PM Dan Williams <dcbw@redhat.com> wrote: >>> On Wed, 2019-06-12 at 10:31 +0200, Arnd Bergmann wrote: >>>> On Tue, Jun 11, 2019 at 7:23 PM Dan Williams <dcbw@redhat.com> wrote: >>> >>> I was trying to make the point that rmnet doesn't need to care about >>> how the QMAP packets get to the device itself; it can be pretty generic >>> so that it can be used by IPA/qmi_wwan/rmnet_smd/etc. >> >> rmnet at the moment is completely generic in that regard already, >> however it is implemented as a tunnel driver talking to another >> device rather than an abstraction layer below that driver. > > It doesn't really actually *do* much other than muck with the headers a > small amount, but even that isn't really much. > > You can probably implement that far more efficiently on some devices > where you have a semi-decent DMA engine that at least supports S/G. If it had a well-defined way of creating new channels to be multiplexed over the connection to the modem, the IPA driver (rather than the rmnet driver) could present network interfaces for each and perform the multiplexing. As I think Arnd suggested, this could at least partially be done with library code (to be shared with other "back-end" interfaces) rather than using a layered driver. This applies to aggregation, channel flow control, and checksum offload as well. But I'm only familiar with IPA; I don't know whether the above statements make any sense for other "back-end" drivers. >>>> I understand that the rmnet model was intended to provide a cleaner >>>> abstraction, but it's not how we normally structure subsystems in >>>> Linux, and moving to a model more like how wireless_dev works >>>> would improve both readability and performance, as you describe >>>> it, it would be more like (ignoring for now the need for multiple >>>> connections): >>>> >>>> ipa_dev >>>> rmnet_dev >>>> wwan_dev >>>> net_device >>> >>> Perhaps I'm assuming too much from this diagram but this shows a 1:1 >>> between wwan_dev and "lower" devices. > > I guess the fuller picture would be something like > > ipa_dev > rmnet_dev > wwan_dev > net_device* > > (i.e. with multiple net_devices) > >>> What Johannes is proposing (IIRC) is something a bit looser where a >>> wwan_dev does not necessarily provide netdev itself, but is instead the >>> central point that various channels (control, data, gps, sim card, etc) >>> register with. That way the wwan_dev can provide an overall view of the >>> WWAN device to userspace, and userspace can talk to the wwan_dev to ask >>> the lower drivers (ipa, rmnet, etc) to create new channels (netdev, >>> tty, otherwise) when the control channel has told the modem firmware to >>> expect one. > > Yeah, that's more what I had in mind after all our discussions (will > continue this below). This is great. The start of a more concrete discussion of the pieces that are missing... >> Right, as I noted above, I simplified it a bit. We probably want to >> have multiple net_device instances for an ipa_dev, so there has >> to be a 1:n relationship instead of 1:1 at one of the intermediate >> levels, but it's not obvious which level that should be. >> >> In theory we could even have a single net_device instance correspond >> to the ipa_dev, but then have multiple IP addresses bound to it, >> so each IP address corresponds to a channel/queue/napi_struct, >> but the user visible object remains a single device. > > I don't think this latter (multiple IP addresses) works well - you want > a hardware specific header ("ETH_P_MAP") to carry the channel ID, > without looking up the IP address and all that. I agree with this. It's not just multiple IP addresses for an interface, it really is multiplexed--with channel ids. It's another addressing parameter orthogonal to the IP space. > But anyway, as I alluded to above, I had something like this in mind: > > driver_dev > struct device *dev (USB, PCI, ...) > net_device NA > net_device NB > tty TA > ... > > (I'm cutting out the rmnet layer here for now) > > while having a separate that just links all the pieces together: > > wwan_device W > ---> dev > ---> NA > ---> NB > ---> TA > > So the driver is still responsible for creating the netdevs (or can of > course delegate that to an "rmnet" library), but then all it also does > is register the netdevs with the WWAN core like > > wwan_add_netdev(dev, NA) > > and the WWAN core would allocate the wwan_device W for this. That would be nice. I believe you're saying that (in my case) the IPA driver creates and owns the netdevices. But I think the IPA driver would register with the WWAN core as a "provider," and then the WWAN core would subsequently request that it instantiate netdevices to represent channels on demand (rather than registering them). > That way, the drivers can concentrate on providing all the necessary > bits, and - crucially - even *different* drivers can end up linking to > the same wwan_device. For example, if you have a modem that has a multi- > function USB device, then an ethernet driver might create the netdev and > a tty driver might create the control channel, but if they both agree on > using the right "struct device" instance, you can still get the correct > wwan_device out of it all. > > And, in fact, some should then be > > wwan_maybe_add_netdev(dev, N) > > because the ethernet driver may not know if it attached to a modem or > not, but if the control channel also attaches it's a modem for sure, > with that ethernet channel attached to it. > > Additionally, I'm thinking API such as > > wwan_add(dev, &ops, opsdata) > > that doesn't automatically attach any channels, but provides "ops" to > the core to create appropriate channels. I think this latter would be > something for IPA/rmnet to use, perhaps for rmnet to offer the right ops > structure. Yes, that's more like what I meant above. I see you're thinking as you write... -Alex > > johannes >
On Tue, Jun 18, 2019 at 3:16 PM Alex Elder <elder@linaro.org> wrote: > On 6/17/19 6:28 AM, Johannes Berg wrote: > > On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: > > I'm probably missing something, but I think the checksum > offload could be handled by the IPA driver rather than > rmnet. It seems to be an add-on that is completely > independent of the multiplexing and aggregation capabilities > that QMAP provides. My best guess is that it is part of rmnet simply because this can be done in a generic way for any qmap based back-end, and rmnet was intended as the abstraction for qmap. A better implementation of the checksumming might be to split it out into a library that is in turn used by qmap drivers. Since this should be transparent to the user interface, it can be moved there later. > >>> If true though, then I think this would be the killer argument *in > >>> favour* of *not* merging this - because that would mean we *don't* have > >>> to actually keep the rmnet API around for all foreseeable future. > > This is because it's a user space API? If so I now understand > what you mean. Yes, I think agreeing on the general user interface is (as usual) the one thing that has to be done as a prerequisite. I had originally hoped that by removing the ioctl interface portion of the driver, this could be avoided, but that was before I had any idea on the upper layers. Arnd
On 6/17/19 7:14 AM, Johannes Berg wrote: > On Tue, 2019-06-11 at 13:56 +0200, Arnd Bergmann wrote: > > [...] > > Looking at the flags again, I sort of talked about this in my message a little earlier, but I now see I was partially mistaken. I thought these flags were used in messages but they're real device ("port") configuration flags. >> #define RMNET_FLAGS_INGRESS_DEAGGREGATION (1U << 0) > > This one I'm not sure I understand - seems weird to have such a > fundamental thing as a *configuration* on the channel. Let me use the term "connection" to refer to the single pathway that carries data between the AP and modem. And "channel" to refer to one of several multiplexed data streams carried over that connection. (If there's better terminology, please say so; I just want to be clear in what I'm talking about.) Deaggregation is a connection property, not a channel property. And it looks like that's exactly how it's used in the rmnet driver. The hardware is capable of aggregating QMAP packets arriving on a connection into a single buffer, so this provides a way of requesting it do that. >> #define RMNET_FLAGS_INGRESS_MAP_COMMANDS (1U << 1) > > Similar here? If you have flow control you probably want to use it? I agree with that, though perhaps there are cases where it is pointless, or can't be supported, so one might want to simply *not* implement/advertise the feature. I don't know. >> #define RMNET_FLAGS_INGRESS_MAP_CKSUMV4 (1U << 2) > > This again looks like a hardware specific feature (ipv4 checksum)? Not > sure why this is set by userspace. > >> #define RMNET_FLAGS_EGRESS_MAP_CKSUMV4 (1U << 3) > > This could be set with ethtool instead, I suppose. As I said in my earlier message, I think I concur about this. I think the IPA driver could easily hide the checksum offload capability, and if it can truly be controlled as needed using existing methods there's no need to encumber the WWAN framework with it. -Alex > johannes >
Just to add to Dan's response, I think he's captured our discussions and thoughts well. > First, a few terms (correct or improve as you like): Thanks for defining, we don't do that nearly often enough. > - WWAN device is a hardware device (like IPA) that presents a > connection between AP and modem, and presents an interface > that allows the use of that connection to be managed. Yes. But I was actually thinking of a "wwan_dev" to be a separate structure, not *directly* owned by a single driver and used to represent the hardware like a (hypothetical) "struct ipa_dev". > - WWAN netdevice represents a Linux network interface, with its > operations and queues, etc., but implements a standardized > set of WWAN-specific operations. It represents a logical > ' channel whose data is multiplexed over the WWAN device. I'm not sure I'd asy it has much WWAN-specific operations? But yeah, I guess it might. > - WWAN channel is a user space abstraction that corresponds > with a WWAN netdevice (but I'm not clear on all the ways > they differ or interact). As Dan said, this could be a different abstraction than a netdevice, like a TTY, etc. > - The WWAN core is kernel code that presents abstractions > for WWAN devices and netdevices, so they can be managed > in a generic way. It is for configuration and communication > and is not at all involved in the data path. > > You're saying that the WWAN driver space calls wwan_add() > to register itself as a new WWAN device. Assuming it knows that it is in fact a WWAN device, like IPA. > You're also saying that a WWAN device "attaches" a WWAN > netdevice, which is basically notifying the WWAN core > that the new netdev/channel is available for use. > - I trust that a "tentative" attachement is necessary. But > I'm not sure what makes it transition into becoming a > "real" one, or how that event gets communicated. I think Dan explained this one well. This wasn't actually on my radar until he pointed it out. Really this only exists with USB devices that appear as multiple functions (ethernet, tty, ...) but still represent a single WWAN device, with each function not necessarily being aware of that since it's just a function driver. Hopefully at least one of the function drivers will be able to figure it out, and then we can combine all of the functions into the WWAN device abstraction. [snip - Dan's explanations are great] Dan also said: > > I read "attach" here as simply associating an existing netdev with the > > "parent" WWAN device. A purely Linux operation that is only book- > > keeping and may not have any interaction with the modem. Now I'm replying out of thread, but yes, that's what I had in mind. What I meant by attaching (in this case) is just that you actually mark that it is (or might be, if tentatively attached) part of a WWAN device. > - Are there any attributes that are only optionally supported, > and if so, how are the supported ones communicated? As Dan said, good point. I hadn't really considered that for now. I sort of know that we need it, but for the sake of simplicity decided to elide it for now. I'm just not sure what really are needed, and netlink attributes make adding them (and discovering the valid ones) pretty easy in the future, when a need arises. > - Which WWAN channel attributes must be set *before* the > channel is activated, and can't be changed? Are there any > that can be changed dynamically? It's a good question. I threw a "u32 pdn" in there, but I'm not actually sure that's what you *really* need? Maybe the modem and userspace just agree on some arbitrary "session identifier"? Dan mentions "MUX ID" or "MBIM Session ID", maybe there really is no good general term for this and we should just call it a "session identifier" and agree that it depends on the control protocol (MBIM vs. QMI vs. ...)? > And while the whole point of this is to make things generic, > it might be nice to have a way to implement a new feature > before it can be "standardized". Not sure I understand this? FWIW, I actually came to this because we want to upstream a driver for an Intel modem, but ... can't really make up our mind on whether or not to use VLAN tags, something like rmnet (but we obviously cannot use rmnet, so that'd be another vendor specific interface like rmnet), or sysfs, or any of the other methods we have today ... :-) johannes
On Tue, Jun 18, 2019 at 9:14 PM Johannes Berg <johannes@sipsolutions.net> wrote: > On Tue, 2019-06-18 at 08:16 -0500, Alex Elder wrote: > > On 6/17/19 6:28 AM, Johannes Berg wrote: > > So getting back to your question, the IPA in its current form only > > has a single "multiplexed" channel carried over the connection > > between the AP and modem. Previously (and in the future) there > > was a way to add or remove channels. > > What would those channels do? > > I've not really been very clear with the differentiation between a > channel and what's multiplexed inside of the channel. > > Using the terminology you defined in your other mail, are you saying > that IPA (originally) allowed multiple *connections* to the device, or > is there basically just one connection, with multiple (QMAP-muxed) > *channels* on top of it? > > If the latter, why did IPA need ioctls, rather than rmnet? From my understanding, the ioctl interface would create the lower netdev after talking to the firmware, and then user space would use the rmnet interface to create a matching upper-level device for that. This is an artifact of the strong separation of ipa and rmnet in the code. > > > The software bridging is very questionable to start with, I'd advocate > > > not supporting that at all but adding tracepoints or similar if needed > > > for debugging instead. > > > > To be honest I don't understand the connection between software > > bridging and debugging, but that's OK. > > It's a mess. Basically, AFAICT, the only use for the rmnet bridging is > in fact debugging. What it does, again AFAICT, is mirror out all the > rmnet packets to the bridge if you attach it to a bridge, so that then > you can attach another netdev to the bridge and forward all the rmnet > packets to another system for debugging. > > It's a very weird way of doing this, IMHO. My understanding for this was that the idea is to use it for connecting bridging between distinct hardware devices behind ipa: if IPA drives both a USB-ether gadget and the 5G modem, you can use to talk to Linux running rmnet, but you can also use rmnet to provide fast usb tethering to 5g and bypass the rest of the network stack. That again may have been a wrong guess on my part. > > I believe the only QMAP commands are for doing essentially > > XON/XOFF flow control on a single channel. In the course of > > the e-mail discussion in the past few weeks I've come to see > > why that would be necessary. > > It does make sense, because you only have a single hardware (DMA) > channel in these cases, so you implement flow control in software on > top. > > (As I said before, the Intel modem uses different hardware channels for > different sessions, so doesn't need something like this - the hardware > ring just fills up and there's your flow control) ipa definitely has multiple hardware queues, and the Alex' driver does implement the data path on those, just not the configuration to enable them. Guessing once more, I suspect the the XON/XOFF flow control was a workaround for the fact that rmnet and ipa have separate queues. The hardware channel on IPA may fill up, but user space talks to rmnet and still add more frames to it because it doesn't know IPA is busy. Another possible explanation would be that this is actually forwarding state from the base station to tell the driver to stop sending data over the air. Arnd
On Tue, 2019-06-18 at 21:59 +0200, Arnd Bergmann wrote: > > From my understanding, the ioctl interface would create the lower > netdev after talking to the firmware, and then user space would use > the rmnet interface to create a matching upper-level device for that. > This is an artifact of the strong separation of ipa and rmnet in the > code. Huh. But if rmnet has muxing, and IPA supports that, why would you ever need multiple lower netdevs? > > > > The software bridging [...] > > My understanding for this was that the idea is to use it for > connecting bridging between distinct hardware devices behind > ipa: if IPA drives both a USB-ether gadget and the 5G modem, > you can use to talk to Linux running rmnet, but you can also > use rmnet to provide fast usb tethering to 5g and bypass the > rest of the network stack. That again may have been a wrong > guess on my part. Hmm. Interesting. It didn't really look to me like that, but I'm really getting lost in the code. Anyway, it seems weird, because then you'd just bridge the upper netdev with the other ethernet and don't need special logic? And I don't see how the ethernet headers would work with this now. > ipa definitely has multiple hardware queues, and the Alex' > driver does implement the data path on those, just not the > configuration to enable them. OK, but perhaps you don't actually have enough to use one for each session? > Guessing once more, I suspect the the XON/XOFF flow control > was a workaround for the fact that rmnet and ipa have separate > queues. The hardware channel on IPA may fill up, but user space > talks to rmnet and still add more frames to it because it doesn't > know IPA is busy. > > Another possible explanation would be that this is actually > forwarding state from the base station to tell the driver to > stop sending data over the air. Yeah, but if you actually have a hardware queue per upper netdev then you don't really need this - you just stop the netdev queue when the hardware queue is full, and you have flow control automatically. So I really don't see any reason to have these messages going back and forth unless you plan to have multiple sessions muxed on a single hardware queue. And really, if you don't mux multiple sessions onto a single hardware queue, you don't need a mux header either, so it all adds up :-) johannes
On Tue, 2019-06-18 at 22:33 +0200, Arnd Bergmann wrote: > > Yeah, but where do you hang that driver? Maybe the TTY function is > > actually a WWAN specific USB driver, but the ethernet is something > > generic that can also work with pure ethernet USB devices, and it's > > difficult to figure out how to tie those together. The modules could > > load in completely different order, or even the ethernet module could > > load but the TTY one doesn't because it's not configured, or vice versa. > > That was more or less my point: The current drivers exist, but don't > lean themselves to fitting into a new framework, so maybe the best > answer is not to try fitting them. > > To clarify: I'm not suggesting to write new USB drivers for these at all, > but instead keep three parts that are completely unaware of each other > a) a regular netdevice driver > b) a regular tty driver > c) the new wwan subsystem that expects a device to be created > from a hardware driver but knows nothing of a) and b) > > To connect these together, we need one glue driver that implements > the wwan_device and talks to a) and b) as the hardware. There are > many ways to do that. One way would be to add a tty ldisc driver. > A small user space helper opens the chardev, sets the ldisc > and then uses an ldisc specific ioctl command to create a wwan > device by passing an identifier of the netdevice and then exits. > From that point on, you have a wwan device like any other. Well, ok. I don't think it'd really work that way ("passing an identifier of the netdevice") because you could have multiple netdevices, but I see what you mean in principle. It seems to me though that this is far more complex than what I'm proposing? What I'm proposing there doesn't even need any userspace involvement, as long as all the pieces are in the different sub-drivers, they'd fall out automatically. And realistically, the wwan_device falls out anyway at some point, the only question is if we really make one specific driver be the "owner" of it. I'm suggesting that we don't, and just make its lifetime depend on the links to parts it has (unless something like IPA actually wants to be an owner). johannes
On Tue, Jun 18, 2019 at 10:36 PM Johannes Berg <johannes@sipsolutions.net> wrote: > > On Tue, 2019-06-18 at 21:59 +0200, Arnd Bergmann wrote: > > > > From my understanding, the ioctl interface would create the lower > > netdev after talking to the firmware, and then user space would use > > the rmnet interface to create a matching upper-level device for that. > > This is an artifact of the strong separation of ipa and rmnet in the > > code. > > Huh. But if rmnet has muxing, and IPA supports that, why would you ever > need multiple lower netdevs? From my reading of the code, there is always exactly a 1:1 relationship between an rmnet netdev an an ipa netdev. rmnet does the encapsulation/ decapsulation of the qmap data and forwards it to the ipa netdev, which then just passes data through between a hardware queue and its netdevice. [side note: on top of that, rmnet also does "aggregation", which may be a confusing term that only means transferring multiple frames at once] > > ipa definitely has multiple hardware queues, and the Alex' > > driver does implement the data path on those, just not the > > configuration to enable them. > > OK, but perhaps you don't actually have enough to use one for each > session? I'm lacking the terminology here, but what I understood was that the netdev and queue again map to a session. > > Guessing once more, I suspect the the XON/XOFF flow control > > was a workaround for the fact that rmnet and ipa have separate > > queues. The hardware channel on IPA may fill up, but user space > > talks to rmnet and still add more frames to it because it doesn't > > know IPA is busy. > > > > Another possible explanation would be that this is actually > > forwarding state from the base station to tell the driver to > > stop sending data over the air. > > Yeah, but if you actually have a hardware queue per upper netdev then > you don't really need this - you just stop the netdev queue when the > hardware queue is full, and you have flow control automatically. > > So I really don't see any reason to have these messages going back and > forth unless you plan to have multiple sessions muxed on a single > hardware queue. Sure, I definitely understand what you mean, and I agree that would be the right way to do it. All I said is that this is not how it was done in rmnet (this was again my main concern about the rmnet design after I learned it was required for ipa) ;-) Arnd
>> There is a n:1 relationship between rmnet and IPA. >> rmnet does the de-muxing to multiple netdevs based on the mux id >> in the MAP header for RX packets and vice versa. > > Oh, so you mean that even though IPA supports multiple channels > and multiple netdev instances for a physical device, all the > rmnet devices end up being thrown into a single channel in IPA? > > What are the other channels for in IPA? I understand that there > is one channel for commands that is separate, while the others > are for network devices, but that seems to make no sense if > we only use a single channel for rmnet data. > AFAIK, the other channels are for use cases like tethering. There is only a single channel which is used for RX data which is then de-muxed using rmnet. -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
On Mon, 2019-06-24 at 11:30 -0500, Alex Elder wrote: > OK I want to try to organize a little more concisely some of the > discussion on this, because there is a very large amount of volume > to date and I think we need to try to narrow the focus back down > again. > > I'm going to use a few terms here. Some of these I really don't > like, but I want to be unambiguous *and* (at least for now) I want > to avoid the very overloaded term "device". > > I have lots more to say, but let's start with a top-level picture, > to make sure we're all on the same page. > > WWAN Communication > Channel (Physical) > | ------------------------ > ------------ v | :+ Control | \ > > |-----------| :+ Data | | > > AP | | WWAN unit :+ Voice | > Functions > > |===========| :+ GPS | | > ------------ ^ | :+ ... | / > | ------------------------- > Multiplexed WWAN > Communication > Channel (Physical) > > - The *AP* is the main CPU complex that's running Linux on one or > more CPU cores. > - A *WWAN unit* is an entity that shares one or more physical > *WWAN communication channels* with the AP. You could just say "WWAN modem" here. > - A *WWAN communication channel* is a bidirectional means of > carrying data between the AP and WWAN unit. > - A WWAN communication channel carries data using a *WWAN protocol*. > - A WWAN unit implements one or more *WWAN functions*, such as > 5G data, LTE voice, GPS, and so on. Go more generic here. Not just 5G data but any WWAN IP-based data (GPRS, EDGE, CDMA, UMTS, EVDO, LTE, 5G, etc). And not just LTE voice but any voice data; plenty of devices don't support LTE but still have "WWAN logical communication channels" > - A WWAN unit shall implement a *WWAN control function*, used to > manage the use of other WWAN functions, as well as the WWAN unit > itself. > - The AP communicates with a WWAN function using a WWAN protocol. > - A WWAN physical channel can be *multiplexed*, in which case it > carries the data for one or more *WWAN logical channels*. It's unclear to me what "physical" means here. USB Interface or Endpoint or PCI Function or SMD channel? Or kernel TTY device? For example on Qualcomm-based USB dongles a given USB Interface's Endpoint represents a QMAP "IP data" channel which itself could be multiplexed into separate "IP data" channels. Or that USB Endpoint(s) could be exposed as a TTY which itself can be MUX-ed dynamically using GSM 07.10. To me "physical" usually means the bus type (PCI, USB, SMD, whatever). A Linux hardware driver (IPA, qmi_wwan, option, sierra, etc) binds to that physical entity using hardware IDs (USB or PCI VID/PID, devicetree properties) and exposes some "WWAN logical communication channels". Those logical channels might be multiplexed and another driver (rmnet) could handle exposing the de-muxed logical channels that the muxed logical channel carries. > - A multiplexed WWAN communication channel uses a *WWAN wultiplexing > protocol*, which is used to separate independent data streams > carrying other WWAN protocols. > - A WWAN logical channel carries a bidirectional stream of WWAN > protocol data between an entity on the AP and a WWAN function. It *usually* is bidirectional. For example some GPS logical communication channels just start spitting out NMEA when you give the control function a command. The NMEA ports themselves don't accept any input. > Does that adequately represent a very high-level picture of what > we're trying to manage? Yes, pretty well. Thanks for trying to specify it all. > And if I understand it right, the purpose of the generic framework > being discussed is to define a common mechanism for managing (i.e., > discovering, creating, destroying, querying, configuring, enabling, > disabling, etc.) WWAN units and the functions they implement, along > with the communication and logical channels used to communicate with > them. Yes. Dan > Comments? > > -Alex
On 6/24/19 2:54 PM, Dan Williams wrote: > On Mon, 2019-06-24 at 11:30 -0500, Alex Elder wrote: >> OK I want to try to organize a little more concisely some of the >> discussion on this, because there is a very large amount of volume >> to date and I think we need to try to narrow the focus back down >> again. >> >> I'm going to use a few terms here. Some of these I really don't >> like, but I want to be unambiguous *and* (at least for now) I want >> to avoid the very overloaded term "device". >> >> I have lots more to say, but let's start with a top-level picture, >> to make sure we're all on the same page. >> >> WWAN Communication >> Channel (Physical) >> | ------------------------ >> ------------ v | :+ Control | \ >>> |-----------| :+ Data | | >>> AP | | WWAN unit :+ Voice | > Functions >>> |===========| :+ GPS | | >> ------------ ^ | :+ ... | / >> | ------------------------- >> Multiplexed WWAN >> Communication >> Channel (Physical) >> >> - The *AP* is the main CPU complex that's running Linux on one or >> more CPU cores. >> - A *WWAN unit* is an entity that shares one or more physical >> *WWAN communication channels* with the AP. > > You could just say "WWAN modem" here. That sounds great to me. >> - A *WWAN communication channel* is a bidirectional means of >> carrying data between the AP and WWAN unit. >> - A WWAN communication channel carries data using a *WWAN protocol*. >> - A WWAN unit implements one or more *WWAN functions*, such as >> 5G data, LTE voice, GPS, and so on. > > Go more generic here. Not just 5G data but any WWAN IP-based data > (GPRS, EDGE, CDMA, UMTS, EVDO, LTE, 5G, etc). And not just LTE voice > but any voice data; plenty of devices don't support LTE but still have > "WWAN logical communication channels" I really meant *any* sort of function, and was only trying to give a few examples. So yes, my meaning was completely generic, as you suggest. >> - A WWAN unit shall implement a *WWAN control function*, used to >> manage the use of other WWAN functions, as well as the WWAN unit >> itself. >> - The AP communicates with a WWAN function using a WWAN protocol. >> - A WWAN physical channel can be *multiplexed*, in which case it >> carries the data for one or more *WWAN logical channels*. > > It's unclear to me what "physical" means here. USB Interface or > Endpoint or PCI Function or SMD channel? Or kernel TTY device? I'm trying to distinguish between (let's say) "hardware" communication channels (such as what IPA basically provides) and logical ones, whose data is multiplexed over a "hardware"/"physical" channel. Maybe "link" would be a better term for what I referred to as a physical channel, and then have "channels" be multiplexed over a link. If you have a good suggestion, please offer it. But I think yes, USB interface, TTY device, etc. are what I mean. I wanted to capture the fact that there might be more than one of these (for example a user space QMI link for control, and IPA link for data?), and that any one of these might also be multiplexed. Multiplexing is a pretty basic capability of a network link, and I think the only reason I called it out separately is to be explicit that it is needs to be supported. > For example on Qualcomm-based USB dongles a given USB Interface's > Endpoint represents a QMAP "IP data" channel which itself could be > multiplexed into separate "IP data" channels. Or that USB Endpoint(s) > could be exposed as a TTY which itself can be MUX-ed dynamically using > GSM 07.10. Yeah. In this case the hardware connection between the AP and the USB dongle would provide a WWAN link (which I think corresponds to the QMAP "IP data" channel). And if you wanted to multiplex that you would use a multiplexing protocol (like QMAP). And that protocol would carry one or more logical channels, each using its own WWAN protocol. It sounds like GSM 07.10 would be another WWAN multiplexing protocol. > To me "physical" usually means the bus type (PCI, USB, SMD, whatever). > A Linux hardware driver (IPA, qmi_wwan, option, sierra, etc) binds to > that physical entity using hardware IDs (USB or PCI VID/PID, devicetree > properties) and exposes some "WWAN logical communication channels". (Or perhaps exposes the ability to create "WWAN logical channels.") "Physical" is probably not a good term. And perhaps focusing on the transport isn't right--maybe the focus should mainly be on the WWAN modem entity. But one of the things we're trying to address here is that there might be several distinct "physical" paths through which the AP and modem can communicate, so a driver's ability to provide such a path should be a sort of first class abstraction. I'm really trying to get this discussion to converge a little, to have a few anchor points to build on. I hope we're getting there. > Those logical channels might be multiplexed and another driver (rmnet) > could handle exposing the de-muxed logical channels that the muxed > logical channel carries. > >> - A multiplexed WWAN communication channel uses a *WWAN wultiplexing >> protocol*, which is used to separate independent data streams >> carrying other WWAN protocols. >> - A WWAN logical channel carries a bidirectional stream of WWAN >> protocol data between an entity on the AP and a WWAN function. > > It *usually* is bidirectional. For example some GPS logical > communication channels just start spitting out NMEA when you give the > control function a command. The NMEA ports themselves don't accept any > input. That's fine... I don't think there's anything wrong with a particular application not using both directions. >> Does that adequately represent a very high-level picture of what >> we're trying to manage? > > Yes, pretty well. Thanks for trying to specify it all. I think we're making progress. I have some more thoughts but I think I'll wait until tomorrow to share them. Thanks a lot Dan. At some point it might be better to have a conference call to make better progress, but I'm not suggesting that yet. -Alex >> And if I understand it right, the purpose of the generic framework >> being discussed is to define a common mechanism for managing (i.e., >> discovering, creating, destroying, querying, configuring, enabling, >> disabling, etc.) WWAN units and the functions they implement, along >> with the communication and logical channels used to communicate with >> them. > > Yes. > > Dan > >> Comments? >> >> -Alex >
This series presents the driver for the Qualcomm IP Accelerator (IPA). This is version 2 of the series. This version has addressed almost all of the feedback received in the first version: https://lore.kernel.org/lkml/20190512012508.10608-1-elder@linaro.org/ More detail is included in the individual patches, but here is a high-level summary of what's changed since then: - Two spinlocks have been removed. - The code for enabling and disabling endpoint interrupts has been simplified considerably, and the spinlock is no longer required - A spinlock used when updating ring buffer pointers is no longer needed. Integers indexing the ring are used instead (and they don't even have to be atomic). - One spinlock remains to protect list updates, but it is always acquired using spin_lock_bh() (no more irqsave). - Information about the queueing and completion of messages is now supplied to the network stack in batches rather than one at a time. - I/O completion handling has been simplified, with the IRQ handler now consisting mainly of disabling the interrupt and calling napi_schedule(). - Some comments have been updated and improved througout. What follows is the introduction supplied with v1 of the series. ----- The IPA is a component present in some Qualcomm SoCs that allows network functions such as aggregation, filtering, routing, and NAT to be performed without active involvement of the main application processor (AP). Initially, these advanced features are disabled; the IPA driver simply provides a network interface that makes the modem's LTE network available to the AP. In addition, only support for the IPA found in the Qualcomm SDM845 SoC is provided. This code is derived from a driver developed internally by Qualcomm. A version of the original source can be seen here: https://source.codeaurora.org/quic/la/kernel/msm-4.9/tree in the "drivers/platform/msm/ipa" directory. Many were involved in developing this, but the following individuals deserve explicit acknowledgement for their substantial contributions: Abhishek Choubey Ady Abraham Chaitanya Pratapa David Arinzon Ghanim Fodi Gidon Studinski Ravi Gummadidala Shihuan Liu Skylar Chang A version of this code was posted in November 2018 as an RFC. https://lore.kernel.org/lkml/20181107003250.5832-1-elder@linaro.org/ All feedback received was addressed. The code has undergone considerable further rework since that time, and most of the "future work" described then has now been completed. This code is available in buildable form here, based on kernel v5.2-rc1: remote: ssh://git@git.linaro.org/people/alex.elder/linux.git branch: ipa-v2_kernel-v5.2-rc2 75adf2ac1266 arm64: defconfig: enable build of IPA code The branch depends on a commit now found in in net-next. It has been cherry-picked, and (in this branch) has this commit ID: 13c627b5a078 net: qualcomm: rmnet: Move common struct definitions to include by -Alex Alex Elder (17): bitfield.h: add FIELD_MAX() and field_max() dt-bindings: soc: qcom: add IPA bindings soc: qcom: ipa: main code soc: qcom: ipa: configuration data soc: qcom: ipa: clocking, interrupts, and memory soc: qcom: ipa: GSI headers soc: qcom: ipa: the generic software interface soc: qcom: ipa: GSI transactions soc: qcom: ipa: IPA interface to GSI soc: qcom: ipa: IPA endpoints soc: qcom: ipa: immediate commands soc: qcom: ipa: IPA network device and microcontroller soc: qcom: ipa: AP/modem communications soc: qcom: ipa: support build of IPA code MAINTAINERS: add entry for the Qualcomm IPA driver arm64: dts: sdm845: add IPA information arm64: defconfig: enable build of IPA code .../devicetree/bindings/net/qcom,ipa.yaml | 180 ++ MAINTAINERS | 6 + arch/arm64/boot/dts/qcom/sdm845.dtsi | 51 + arch/arm64/configs/defconfig | 1 + drivers/net/Kconfig | 2 + drivers/net/Makefile | 1 + drivers/net/ipa/Kconfig | 16 + drivers/net/ipa/Makefile | 7 + drivers/net/ipa/gsi.c | 1635 +++++++++++++++++ drivers/net/ipa/gsi.h | 246 +++ drivers/net/ipa/gsi_private.h | 148 ++ drivers/net/ipa/gsi_reg.h | 376 ++++ drivers/net/ipa/gsi_trans.c | 624 +++++++ drivers/net/ipa/gsi_trans.h | 116 ++ drivers/net/ipa/ipa.h | 131 ++ drivers/net/ipa/ipa_clock.c | 297 +++ drivers/net/ipa/ipa_clock.h | 52 + drivers/net/ipa/ipa_cmd.c | 377 ++++ drivers/net/ipa/ipa_cmd.h | 116 ++ drivers/net/ipa/ipa_data-sdm845.c | 245 +++ drivers/net/ipa/ipa_data.h | 267 +++ drivers/net/ipa/ipa_endpoint.c | 1283 +++++++++++++ drivers/net/ipa/ipa_endpoint.h | 97 + drivers/net/ipa/ipa_gsi.c | 48 + drivers/net/ipa/ipa_gsi.h | 49 + drivers/net/ipa/ipa_interrupt.c | 279 +++ drivers/net/ipa/ipa_interrupt.h | 53 + drivers/net/ipa/ipa_main.c | 921 ++++++++++ drivers/net/ipa/ipa_mem.c | 234 +++ drivers/net/ipa/ipa_mem.h | 83 + drivers/net/ipa/ipa_netdev.c | 251 +++ drivers/net/ipa/ipa_netdev.h | 24 + drivers/net/ipa/ipa_qmi.c | 402 ++++ drivers/net/ipa/ipa_qmi.h | 35 + drivers/net/ipa/ipa_qmi_msg.c | 583 ++++++ drivers/net/ipa/ipa_qmi_msg.h | 238 +++ drivers/net/ipa/ipa_reg.h | 279 +++ drivers/net/ipa/ipa_smp2p.c | 304 +++ drivers/net/ipa/ipa_smp2p.h | 47 + drivers/net/ipa/ipa_uc.c | 208 +++ drivers/net/ipa/ipa_uc.h | 32 + include/linux/bitfield.h | 14 + 42 files changed, 10358 insertions(+) create mode 100644 Documentation/devicetree/bindings/net/qcom,ipa.yaml create mode 100644 drivers/net/ipa/Kconfig create mode 100644 drivers/net/ipa/Makefile create mode 100644 drivers/net/ipa/gsi.c create mode 100644 drivers/net/ipa/gsi.h create mode 100644 drivers/net/ipa/gsi_private.h create mode 100644 drivers/net/ipa/gsi_reg.h create mode 100644 drivers/net/ipa/gsi_trans.c create mode 100644 drivers/net/ipa/gsi_trans.h create mode 100644 drivers/net/ipa/ipa.h create mode 100644 drivers/net/ipa/ipa_clock.c create mode 100644 drivers/net/ipa/ipa_clock.h create mode 100644 drivers/net/ipa/ipa_cmd.c create mode 100644 drivers/net/ipa/ipa_cmd.h create mode 100644 drivers/net/ipa/ipa_data-sdm845.c create mode 100644 drivers/net/ipa/ipa_data.h create mode 100644 drivers/net/ipa/ipa_endpoint.c create mode 100644 drivers/net/ipa/ipa_endpoint.h create mode 100644 drivers/net/ipa/ipa_gsi.c create mode 100644 drivers/net/ipa/ipa_gsi.h create mode 100644 drivers/net/ipa/ipa_interrupt.c create mode 100644 drivers/net/ipa/ipa_interrupt.h create mode 100644 drivers/net/ipa/ipa_main.c create mode 100644 drivers/net/ipa/ipa_mem.c create mode 100644 drivers/net/ipa/ipa_mem.h create mode 100644 drivers/net/ipa/ipa_netdev.c create mode 100644 drivers/net/ipa/ipa_netdev.h create mode 100644 drivers/net/ipa/ipa_qmi.c create mode 100644 drivers/net/ipa/ipa_qmi.h create mode 100644 drivers/net/ipa/ipa_qmi_msg.c create mode 100644 drivers/net/ipa/ipa_qmi_msg.h create mode 100644 drivers/net/ipa/ipa_reg.h create mode 100644 drivers/net/ipa/ipa_smp2p.c create mode 100644 drivers/net/ipa/ipa_smp2p.h create mode 100644 drivers/net/ipa/ipa_uc.c create mode 100644 drivers/net/ipa/ipa_uc.h -- 2.20.1