| Message ID | 20201207221342.553976-1-parav@nvidia.com |
|---|---|
| State | New |
| Headers | show |
| Series | [net-next,v5] devlink: Add devlink port documentation | expand |
Hi-- On 12/7/20 2:13 PM, Parav Pandit wrote: > Added documentation for devlink port and port function related commands. > > Signed-off-by: Parav Pandit <parav@nvidia.com> > Reviewed-by: Jiri Pirko <jiri@nvidia.com> > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > --- > Changelog: > v4->v5: > - described logically ingress and egress point of devlink port > - removed networking from devlink port description > - rephrased port type description > - introdue PCI controller section and description introduce > - rephrased controller, device, function description > - removed confusing eswitch to system wording > - rephrased port function description > - added example of mac address in port function attribute description > --- > .../networking/devlink/devlink-port.rst | 116 ++++++++++++++++++ > Documentation/networking/devlink/index.rst | 1 + > 2 files changed, 117 insertions(+) > create mode 100644 Documentation/networking/devlink/devlink-port.rst > > diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst > new file mode 100644 > index 000000000000..dce87d2c07ac > --- /dev/null > +++ b/Documentation/networking/devlink/devlink-port.rst > @@ -0,0 +1,116 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +============ > +Devlink Port > +============ > + > +``devlink-port`` is a port that exists on the device. It has a logically > +separate ingress/egress point of the device. A devlink port can be of one port can be any one of many flavours. > +among many flavours. A devlink port flavour along with port attributes > +describe what a port represents. > + > +A device driver that intends to publish a devlink port sets the > +devlink port attributes and registers the devlink port. > + > +Devlink port flavours are described below. > + > +.. list-table:: List of devlink port flavours > + :widths: 33 90 > + > + * - Flavour > + - Description > + * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL`` > + - Any kind of physical networking port. This can be an eswitch physical > + port or any other physical port on the device. > + * - ``DEVLINK_PORT_FLAVOUR_DSA`` > + - This indicates a DSA interconnect port. > + * - ``DEVLINK_PORT_FLAVOUR_CPU`` > + - This indicates a CPU port applicable only to DSA. > + * - ``DEVLINK_PORT_FLAVOUR_PCI_PF`` > + - This indicates an eswitch port representing a networking port of > + PCI physical function (PF). > + * - ``DEVLINK_PORT_FLAVOUR_PCI_VF`` > + - This indicates an eswitch port representing a networking port of > + PCI virtual function (VF). > + * - ``DEVLINK_PORT_FLAVOUR_VIRTUAL`` > + - This indicates a virtual port for the virtual PCI device such as PCI VF. > + > +Devlink port can have a different type based on the link layer described below. > + > +.. list-table:: List of devlink port types > + :widths: 23 90 > + > + * - Type > + - Description > + * - ``DEVLINK_PORT_TYPE_ETH`` > + - Driver should set this port type when a link layer of the port is > + Ethernet. > + * - ``DEVLINK_PORT_TYPE_IB`` > + - Driver should set this port type when a link layer of the port is > + InfiniBand. > + * - ``DEVLINK_PORT_TYPE_AUTO`` > + - This type is indicated by the user when driver should detect the port > + type automatically. > + > +PCI controllers > +--------------- > +In most cases PCI device has only one controller. A controller consists of either PCI devices have or a PCI device has > +potentially multiple physical and virtual functions. Such PCI function consists > +of one or more ports. This port of the function is represented by the devlink > +eswitch port. > + > +A PCI Device connected to multiple CPUs or multiple PCI root complex or complexes > +SmartNIC, however, may have multiple controllers. For a device with multiple > +controllers, each controller is distinguished by a unique controller number. > +An eswitch on the PCI device may suppport ports of multiple controllers. support > + > +An example view of two controller systems:: I think that this is An example view of a two-controller system:: instead of 2 controller systems. ? > + > +In this example, external controller (identified by controller number = 1) > +doesn't have eswitch. Local controller (identified by controller number = 0) > +has the eswitch. Devlink instance on local controller has eswitch devlink > +ports representing ports for both the controllers. > + > + --------------------------------------------------------- > + | | > + | --------- --------- ------- ------- | > + ----------- | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | > + | server | | ------- ----/---- ---/----- ------- ---/--- ---/--- | > + | pci rc |=== | pf0 |______/________/ | pf1 |___/_______/ | > + | connect | | ------- ------- | > + ----------- | | controller_num=1 (no eswitch) | > + ------|-------------------------------------------------- > + (internal wire) > + | > + --------------------------------------------------------- > + | devlink eswitch ports and reps | > + | ----------------------------------------------------- | > + | |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | | > + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | > + | ----------------------------------------------------- | > + | |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | | > + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | > + | ----------------------------------------------------- | > + | | > + | | > + ----------- | --------- --------- ------- ------- | > + | smartNIC| | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | > + | pci rc |==| ------- ----/---- ---/----- ------- ---/--- ---/--- | > + | connect | | | pf0 |______/________/ | pf1 |___/_______/ | > + ----------- | ------- ------- | > + | | > + | local controller_num=0 (eswitch) | > + --------------------------------------------------------- > + > +Port function configuration > +=========================== > + > +A user can configure the port function attribute before enumerating the > +PCI function. Usually it means, user should configure port function attribute > +before a bus specific device for the function is created. However, when > +SRIOV is enabled, virtual function devices are created on the PCI bus. > +Hence, function attribute should be configured before binding virtual > +function device to the driver. > + > +User may set the hardware address of the function represented by the devlink > +port function. For Ethernet port function this means a MAC address. -- ~Randy
Hi Randy, > From: Randy Dunlap <rdunlap@infradead.org> > Sent: Tuesday, December 8, 2020 6:45 AM > > > +An eswitch on the PCI device may suppport ports of multiple controllers. > > support > > > + > > +An example view of two controller systems:: > > I think that this is > > An example view of a two-controller system:: > > instead of 2 controller systems. ? Rephrasing it as "An example view of a system with two controllers" along with other comments. Since Saeed is sending the sf patchset which also needs port documentation, I am sending v6 of this patch along with the change log in the subfunction patchset.
> From: Randy Dunlap <rdunlap@infradead.org> > Sent: Tuesday, December 8, 2020 6:45 AM > > Hi-- > > On 12/7/20 2:13 PM, Parav Pandit wrote: > > Added documentation for devlink port and port function related > commands. > > > > Signed-off-by: Parav Pandit <parav@nvidia.com> > > Reviewed-by: Jiri Pirko <jiri@nvidia.com> > > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > > --- > > Changelog: > > v4->v5: > > - described logically ingress and egress point of devlink port > > - removed networking from devlink port description > > - rephrased port type description > > - introdue PCI controller section and description > > introduce > > > - rephrased controller, device, function description > > - removed confusing eswitch to system wording > > - rephrased port function description > > - added example of mac address in port function attribute description > > > --- > > .../networking/devlink/devlink-port.rst | 116 ++++++++++++++++++ > > Documentation/networking/devlink/index.rst | 1 + > > 2 files changed, 117 insertions(+) > > create mode 100644 Documentation/networking/devlink/devlink-port.rst > > > > diff --git a/Documentation/networking/devlink/devlink-port.rst > > b/Documentation/networking/devlink/devlink-port.rst > > new file mode 100644 > > index 000000000000..dce87d2c07ac > > --- /dev/null > > +++ b/Documentation/networking/devlink/devlink-port.rst > > @@ -0,0 +1,116 @@ > > +.. SPDX-License-Identifier: GPL-2.0 > > + > > +============ > > +Devlink Port > > +============ > > + > > +``devlink-port`` is a port that exists on the device. It has a > > +logically separate ingress/egress point of the device. A devlink port > > +can be of one > > port can be any one > of many flavours. > > > +among many flavours. A devlink port flavour along with port > > +attributes describe what a port represents. > > + > > +A device driver that intends to publish a devlink port sets the > > +devlink port attributes and registers the devlink port. > > + > > +Devlink port flavours are described below. > > + > > +.. list-table:: List of devlink port flavours > > + :widths: 33 90 > > + > > + * - Flavour > > + - Description > > + * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL`` > > + - Any kind of physical networking port. This can be an eswitch physical > > + port or any other physical port on the device. > > + * - ``DEVLINK_PORT_FLAVOUR_DSA`` > > + - This indicates a DSA interconnect port. > > + * - ``DEVLINK_PORT_FLAVOUR_CPU`` > > + - This indicates a CPU port applicable only to DSA. > > + * - ``DEVLINK_PORT_FLAVOUR_PCI_PF`` > > + - This indicates an eswitch port representing a networking port of > > + PCI physical function (PF). I forgot to remove 'networking' from the networking port here. Will include this fix this in v6 in Saeed's subfunction patchset along with addressing Randy's comments.
diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst new file mode 100644 index 000000000000..dce87d2c07ac --- /dev/null +++ b/Documentation/networking/devlink/devlink-port.rst @@ -0,0 +1,116 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============ +Devlink Port +============ + +``devlink-port`` is a port that exists on the device. It has a logically +separate ingress/egress point of the device. A devlink port can be of one +among many flavours. A devlink port flavour along with port attributes +describe what a port represents. + +A device driver that intends to publish a devlink port sets the +devlink port attributes and registers the devlink port. + +Devlink port flavours are described below. + +.. list-table:: List of devlink port flavours + :widths: 33 90 + + * - Flavour + - Description + * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL`` + - Any kind of physical networking port. This can be an eswitch physical + port or any other physical port on the device. + * - ``DEVLINK_PORT_FLAVOUR_DSA`` + - This indicates a DSA interconnect port. + * - ``DEVLINK_PORT_FLAVOUR_CPU`` + - This indicates a CPU port applicable only to DSA. + * - ``DEVLINK_PORT_FLAVOUR_PCI_PF`` + - This indicates an eswitch port representing a networking port of + PCI physical function (PF). + * - ``DEVLINK_PORT_FLAVOUR_PCI_VF`` + - This indicates an eswitch port representing a networking port of + PCI virtual function (VF). + * - ``DEVLINK_PORT_FLAVOUR_VIRTUAL`` + - This indicates a virtual port for the virtual PCI device such as PCI VF. + +Devlink port can have a different type based on the link layer described below. + +.. list-table:: List of devlink port types + :widths: 23 90 + + * - Type + - Description + * - ``DEVLINK_PORT_TYPE_ETH`` + - Driver should set this port type when a link layer of the port is + Ethernet. + * - ``DEVLINK_PORT_TYPE_IB`` + - Driver should set this port type when a link layer of the port is + InfiniBand. + * - ``DEVLINK_PORT_TYPE_AUTO`` + - This type is indicated by the user when driver should detect the port + type automatically. + +PCI controllers +--------------- +In most cases PCI device has only one controller. A controller consists of +potentially multiple physical and virtual functions. Such PCI function consists +of one or more ports. This port of the function is represented by the devlink +eswitch port. + +A PCI Device connected to multiple CPUs or multiple PCI root complex or +SmartNIC, however, may have multiple controllers. For a device with multiple +controllers, each controller is distinguished by a unique controller number. +An eswitch on the PCI device may suppport ports of multiple controllers. + +An example view of two controller systems:: + +In this example, external controller (identified by controller number = 1) +doesn't have eswitch. Local controller (identified by controller number = 0) +has the eswitch. Devlink instance on local controller has eswitch devlink +ports representing ports for both the controllers. + + --------------------------------------------------------- + | | + | --------- --------- ------- ------- | + ----------- | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | + | server | | ------- ----/---- ---/----- ------- ---/--- ---/--- | + | pci rc |=== | pf0 |______/________/ | pf1 |___/_______/ | + | connect | | ------- ------- | + ----------- | | controller_num=1 (no eswitch) | + ------|-------------------------------------------------- + (internal wire) + | + --------------------------------------------------------- + | devlink eswitch ports and reps | + | ----------------------------------------------------- | + | |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | | + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | + | ----------------------------------------------------- | + | |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | | + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | + | ----------------------------------------------------- | + | | + | | + ----------- | --------- --------- ------- ------- | + | smartNIC| | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | + | pci rc |==| ------- ----/---- ---/----- ------- ---/--- ---/--- | + | connect | | | pf0 |______/________/ | pf1 |___/_______/ | + ----------- | ------- ------- | + | | + | local controller_num=0 (eswitch) | + --------------------------------------------------------- + +Port function configuration +=========================== + +A user can configure the port function attribute before enumerating the +PCI function. Usually it means, user should configure port function attribute +before a bus specific device for the function is created. However, when +SRIOV is enabled, virtual function devices are created on the PCI bus. +Hence, function attribute should be configured before binding virtual +function device to the driver. + +User may set the hardware address of the function represented by the devlink +port function. For Ethernet port function this means a MAC address. diff --git a/Documentation/networking/devlink/index.rst b/Documentation/networking/devlink/index.rst index d82874760ae2..aab79667f97b 100644 --- a/Documentation/networking/devlink/index.rst +++ b/Documentation/networking/devlink/index.rst @@ -18,6 +18,7 @@ general. devlink-info devlink-flash devlink-params + devlink-port devlink-region devlink-resource devlink-reload