| Message ID | 1456914944-15985-3-git-send-email-bill.fischofer@linaro.org |
|---|---|
| State | Superseded |
| Headers | show |
On 2016-03-02 04:35, Bill Fischofer wrote: > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> > --- > doc/users-guide/users-guide-pktio.adoc | 605 +++++++++++++++++++++++++++++++++ > doc/users-guide/users-guide.adoc | 2 + > 2 files changed, 607 insertions(+) > create mode 100644 doc/users-guide/users-guide-pktio.adoc > > diff --git a/doc/users-guide/users-guide-pktio.adoc b/doc/users-guide/users-guide-pktio.adoc > new file mode 100644 > index 0000000..29fb6a9 > --- /dev/null > +++ b/doc/users-guide/users-guide-pktio.adoc > @@ -0,0 +1,605 @@ > +== PktIO Processing > +Before packets can be manipulated they typically need to be _received_ and > +after they are manipulated they need to be _transmitted_. The ODP abstraction > +that captures these operations is the *Packet I/O (PktIO)*. > +PktIOs are represented by handles of type *odp_pktio_t* and > +represent a logical I/O interface that is mapped in an implementation-defined > +manner to an underlying integrated I/O adapter or NIC. > + > +PktIO objects are manipulated through various state transitions via > ++odp_pktio_xxx()+ API calls as shown below: > + > +.ODP PktIO Finite State Machine > +image::../images/pktio_fsm.svg[align="center"] > + > +PktIOs begin in the *Unallocated* state. From here a call +odp_pktio_open()+ > +is used to create an *odp_pktio_t* handle that is used in all subsequent calls > +to manipulate the object. This call puts the PktIO into the *Unconfigured* > +state. To become operational, a PktIO must first be > +*configured* for Input, Output, or both Input and Output via the > ++odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ APIs, and then > +*started* via the +odp_pktio_start()+ to make it *Ready*. > + > +Following the completion of I/O processing, the +odp_pktio_stop()+ API returns > +the PktIO to the *Configured* state. From here it may be *Reconfigured* via > +additional +odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ calls, > +or *Closed* via the +odp_pktio_close()+ API to return the PktIO to the > +*Unallocated* state. > + > +=== PktIO Allocation > +PktIO objects begin life by being _opened_ via the call: > +[source,c] > +----- > +/** > + * Open a packet IO interface > + * > + * An ODP program can open a single packet IO interface per device, attempts what is a device here? a (possibly physical) interface? if we have a NIC board with 2 physical inerfaces, how many pktio can on open on it? maybe: "An ODP program can open a single packet IO per interface"...? > + * to open an already open device will fail, returning ODP_PKTIO_INVALID with > + * errno set. Use odp_pktio_lookup() to obtain a handle to an already open "opened" instead of open? > + * device. Packet IO parameters provide interface level configuration options. > + * > + * Packet input queue configuration must be setup with > + * odp_pktin_queue_config() before odp_pktio_start() is called. When packet > + * input mode is ODP_PKTIN_MODE_DISABLED, odp_pktin_queue_config() call is > + * optional and will ignore all parameters. > + * > + * Packet output queue configuration must be setup with > + * odp_pktout_queue_config() before odp_pktio_start() is called. When packet > + * output mode is ODP_PKTOUT_MODE_DISABLED or ODP_PKTOUT_MODE_TM, > + * odp_pktout_queue_config() call is optional and will ignore all parameters. > + * Is is not clear at this stage of reading what are the packet input and output modes. I assume that they are parameters to the pktio_open function (in param) as this is the only thing that is done before the configuration, maybe that should be clarified before. (this assumption gets confirmed later on in the doc) Simply adding "Pktio in/output mode will be discussed in the next section" before this code frgament would make sense. > + * Packet receive and transmit on the interface is enabled with a call to > + * odp_pktio_start(). If not specified otherwise, any interface level > + * configuration must not be changed when the interface is active (between start > + * and stop calls). > + * > + * In summary, a typical pktio interface setup sequence is ... > + * * odp_pktio_open() > + * * odp_pktin_queue_config() > + * * odp_pktout_queue_config() > + * * odp_pktio_start() > + * > + * ... and tear down sequence is: > + * * odp_pktio_stop() > + * * odp_pktio_close() > + * > + * @param name Packet IO device name > + * @param pool Default pool from which to allocate storage for packets > + * received over this interface, must be of type ODP_POOL_PACKET > + * @param param Packet IO parameters > + * > + * @return Packet IO handle > + * @retval ODP_PKTIO_INVALID on failure > + * > + * @note The device name "loop" is a reserved name for a loopback device used > + * for testing purposes. > + * > + * @note Packets arriving via this interface assigned to a CoS by the > + * classifier are received into the pool associated with that CoS. This > + * will occur either because this pktio is assigned a default CoS via > + * the odp_pktio_default_cos_set() routine, or because a matching PMR > + * assigned the packet to a specific CoS. The default pool specified > + * here is applicable only for those packets that are not assigned to a > + * more specific CoS. > + * > + * @see odp_pktio_start(), odp_pktio_stop(), odp_pktio_close() > + */ > +odp_pktio_t odp_pktio_open(const char *name, odp_pool_t pool, > + const odp_pktio_param_t *param); > +----- > ++odp_pktio_open()+ takes three arguments: a *name*, which is an > +implementation-defined string that identifies the logical interface to be > +opened, a *pool* that identifies the ODP pool that storage for received > +packets should be allocated from, and a *param* structure that specifies > +I/O options to be associated with this PktIO instance. > +[source,c] > +----- > +/** > + * Packet IO parameters > + * > + * In minimum, user must select input and output modes. Use 0 for defaults. > + * Initialize entire struct with zero to maintain API compatibility. > + */ > +typedef struct odp_pktio_param_t { > + /** Packet input mode > + * > + * The default value is ODP_PKTIN_MODE_DIRECT. */ > + odp_pktin_mode_t in_mode; > + /** Packet output mode > + * > + * The default value is ODP_PKTOUT_MODE_DIRECT. */ > + odp_pktout_mode_t out_mode; > +} odp_pktio_param_t; > +----- > +ODP defines *loop* as a reserved name to indicate that this PktIO represents I think "loop" should be quoted as it is a string. > +a loopback interface. Loopback interfaces are useful as a means of recycling > +packets back for reclassification after decryption or decapsulation, as well as > +for diagnostic or testing purposes. The *pool* specifies the default pool to "Loopback interfaces are useful as a means of recycling packets back for reclassification after decryption or decapsulation". I don't understand what you mean by that... should I? ;-) > +use for packet allocation if not overridden by the classifier due to a > +specific or default Class-of-Service (CoS) match on the packet. The *param* > +struct, in turn, specifies the input and output *modes* of the PktIO. > + > +=== PktIO Input and Output Modes > +PktIO objects support four different Input and Output modes, that may be > +specified independently at *open* time. > + > +.PktIO Input Modes > +* +ODP_PKTIN_MODE_DIRECT+ > +* +ODP_PKTIN_MODE_QUEUE+ > +* +ODP_OKTIN_MODE_SCHED+ > +* +ODP_PKTIN_MODE_DISABLED+ > + > +.PktIO Output Modes > +* +ODP_PKTOUT_MODE_DIRECT+ > +* +ODP_PKTOUT_MODE_QUEUE+ > +* +ODP_PKTOUT_MODE_TM+ > +* +ODP_PKTOUT_MODE_DISABLED+ > + > +The DISABLED modes indicate that either input or output is prohibited on this > +PktIO. Attempts to receive packets on a PktIO whose +in_mode+ is DISABLED > +return no packets while packets sent to a PktIO whose +out_mode+ is DISABLED > +are discarded. > + > +==== Direct I/O Modes > +DIRECT I/O is the default mode for PktIO objects. It is designed to support > +poll-based packet processing, which is often found in legacy applications > +being ported to ODP, and can also be a preferred mode for some types of > +packet processing. By supporting poll-based I/O processing, ODP provides > +maximum flexibility to the data plane application writer. > + > +===== Direct RX Processing > +The processing of DIRECT input is shown below: > + > +.PktIO DIRECT Mode Receive Processing > +image::../images/pktin_direct_recv.svg[align="center"] > + From this picture, I understand that a each rx pktio queue is "associated" to a RX thread at odp_pktin_queue_config time. however, it does not feel like that at all in the rest of the document: odp_pktin_recv() take a queue handle as parameter, hence allowing any ODP thread to read packets from any queue at any time... (the "operation mode", -not really defined for RX- also implies that such an association does not exist, at least in unsafe mode) > +In DIRECT mode, received packets are stored in one or more special PktIO queues > +of type *odp_pktin_queue_t* and are retrieved by threads calling the > ++odp_pktin_recv()+ API. > + > +Once opened, setting up a DIRECT mode PktIO is performed by the > ++odp_pktin_queue_config()+ API. > +[source,c] > +----- > +/** > + * Configure packet input queues > + * > + * Setup a number of packet input queues and configure those. The maximum number > + * of queues is platform dependent and can be queried with > + * odp_pktio_capability(). Use odp_pktin_queue_param_init() to initialize > + * parameters into their default values. Default values are also used when > + * 'param' pointer is NULL. > + * > + * Queue handles for input queues can be requested with odp_pktin_queue() or > + * odp_pktin_event_queue() after this call. All requested queues are setup on > + * success, no queues are setup on failure. Each call reconfigures input queues > + * and may invalidate all previous queue handles. > + * > + * @param pktio Packet IO handle > + * @param param Packet input queue configuration parameters. Uses defaults > + * when NULL. > + * > + * @retval 0 on success > + * @retval <0 on failure > + * > + * @see odp_pktio_capability(), odp_pktin_queue(), odp_pktin_event_queue() > + */ > +int odp_pktin_queue_config(odp_pktio_t pktio, > + const odp_pktin_queue_param_t *param); > +----- > +The second argument to this call is the *odp_pktin_queue_param_t* > +[source,c] > +----- > +/** > + * Packet input queue parameters > + */ > +typedef struct odp_pktin_queue_param_t { > + /** Operation mode > + * > + * The default value is ODP_PKTIO_OP_MT. Application may enable > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE when > + * applicable. */ > + odp_pktio_op_mode_t op_mode; Has the "operation mode" been defined anywhere? not sure what it is. neither do I know what ODP_PKTIO_OP_MT means at this stage of reading... This is defined only when talking about TX, later on... It should be defined here (and possibly reminded for TX) > + > + /** Enable flow hashing > + * 0: Do not hash flows > + * 1: Hash flows to input queues */ > + odp_bool_t hash_enable; > + > + /** Protocol field selection for hashing. Multiple protocols can be > + * selected. */ > + odp_pktin_hash_proto_t hash_proto; > + > + /** Number of input queues to be created. More than one input queue > + * require input hashing or classifier setup. Hash_proto is ignored > + * when hash_enable is zero or num_queues is one. This value must be > + * between 1 and interface capability. Queue type is defined by the > + * input mode. The default value is 1. */ > + unsigned num_queues; > + > + /** Queue parameters for creating input queues in ODP_PKTIN_MODE_QUEUE > + * or ODP_PKTIN_MODE_SCHED modes. Scheduler parameters are considered > + * only in ODP_PKTIN_MODE_SCHED mode. */ > + odp_queue_param_t queue_param; > + > +} odp_pktin_queue_param_t; > +----- > +Note that the *queue_param* field of this struct is ignored in DIRECT mode. > +The purpose of +odp_pktin_queue_config()+ is to specify the number of PktIn > +queues to be created and to set their attributes. > + > +===== Hash Processing > +Another feature of DIRECT mode input is the provision of a *hash* function used > +to distribute incoming packets among the PktIO's PktIn queues. If the > ++hash_enable+ field of the *odp_pktin_queue_param_t* is 1, > +then the +hash_proto+ field is used to specify which field(s) of incoming > +packets should be used as input to an implementation-defined packet > +distribution hash function. > +[source,c] > +----- > +/** > + * Packet input hash protocols > + * > + * The list of protocol header field combinations, which are included into > + * packet input hash calculation. > + */ > +typedef union odp_pktin_hash_proto_t { > + /** Protocol header fields for hashing */ > + struct { > + /** IPv4 addresses and UDP port numbers */ > + uint32_t ipv4_udp : 1; > + /** IPv4 addresses and TCP port numbers */ > + uint32_t ipv4_tcp : 1; > + /** IPv4 addresses */ > + uint32_t ipv4 : 1; > + /** IPv6 addresses and UDP port numbers */ > + uint32_t ipv6_udp : 1; > + /** IPv6 addresses and TCP port numbers */ > + uint32_t ipv6_tcp : 1; > + /** IPv6 addresses */ > + uint32_t ipv6 : 1; > + } proto; > + > + /** All bits of the bit field structure */ > + uint32_t all_bits; > +} odp_pktin_hash_proto_t; > +----- > +Note that the hash function used in PktIO poll mode operation is intended to > +provide simple packet distribution among multiple PktIn queues associated with > +the PktIO. It does not have the sophistication of the *ODP Classifier*, however > +it also does not incur the setup requirements of pattern matching rules, > +making it a simpler choice for less sophisticated applications. Note that > +ODP does not specify how the hash is to be performed. That is left to each > +implementation. The hash only specifies which input packet fields are of > +interest to the application and should be considered by the hash function in > +deciding how to distribute packets among PktIn queues. The only expectation > +is that packets that have the same hash values should all be mapped to the > +same PktIn queue. > + > +===== PktIn Queues > +A *PktIn Queue* is a special type of queue that is used internally by PktIOs > +operating in DIRECT mode. Applications cannot perform enqueues to these queues, > +however they may obtain references to them via the +odp_pktin_queue()+ API > +[source,c] > +----- > +/** > + * Direct packet input queues > + * > + * Returns the number of input queues configured for the interface in > + * ODP_PKTIN_MODE_DIRECT mode. Outputs up to 'num' queue handles when the > + * 'queues' array pointer is not NULL. If return value is larger than 'num', > + * there are more queues than the function was allowed to output. If return > + * value (N) is less than 'num', only queues[0 ... N-1] have been written. > + * > + * Packets from these queues are received with odp_pktin_recv(). > + * > + * @param pktio Packet IO handle > + * @param[out] queues Points to an array of queue handles for output > + * @param num Maximum number of queue handles to output > + * > + * @return Number of packet input queues > + * @retval <0 on failure > + */ > +int odp_pktin_queue(odp_pktio_t pktio, odp_pktin_queue_t queues[], int num); > +----- > +Once configured, prior to receiving packets the PktIO must be placed into the > +*Ready* state via a call to +odp_pktio_start() > +[source,c] > +----- > +/** > + * Start packet receive and transmit > + * > + * Activate packet receive and transmit on a previously opened or stopped > + * interface. The interface can be stopped with a call to odp_pktio_stop(). > + * > + * @param pktio Packet IO handle > + * > + * @retval 0 on success > + * @retval <0 on failure > + * > + * @see odp_pktio_open(), odp_pktio_stop() > + */ > +int odp_pktio_start(odp_pktio_t pktio); > +----- > +Once started, the PktIn queue handles are used as arguments to > ++odp_pktin_recv()+ to receive packets from the PktIO. > +[source,c] > +----- > +/** > + * Receive packets directly from an interface input queue > + * > + * Receives up to 'num' packets from the pktio interface input queue. When > + * input queue parameter 'op_mode' has been set to ODP_PKTIO_OP_MT_UNSAFE, > + * the operation is optimized for single thread operation per queue and the same > + * queue must not be accessed simultaneously from multiple threads. > + * > + * @param queue Packet input queue handle for receiving packets > + * @param[out] packets[] Packet handle array for output of received packets > + * @param num Maximum number of packets to receive > + * > + * @return Number of packets received > + * @retval <0 on failure > + * > + * @see odp_pktin_queue() > + */ > +int odp_pktin_recv(odp_pktin_queue_t queue, odp_packet_t packets[], int num); This means that any thread can read from any queue at any time, right? (does not match my understanding of figure 13) > +----- > + > +===== Direct TX Processing > +A PktIO operating in DIRECT mode performs TX processing as shown here: > + > +.PktIO DIRECT Mode Transmit Processing > +image::../images/pktout_direct_send.svg[align="center"] > + > +Direct TX processing operates similarly to Direct RX processing. Following > +open, the +odp_pktout_queue_config()+ API is used to create and configure > +one or more *PktOut queues* to be used to support packet transmission by > +this PktIO > +[source,c] > +----- > +/** > + * Configure packet output queues > + * > + * Setup a number of packet output queues and configure those. The maximum > + * number of queues is platform dependent and can be queried with > + * odp_pktio_capability(). Use odp_pktout_queue_param_init() to initialize > + * parameters into their default values. Default values are also used when > + * 'param' pointer is NULL. > + * > + * Queue handles for output queues can be requested with odp_pktout_queue() or > + * odp_pktout_event_queue() after this call. All requested queues are setup on > + * success, no queues are setup on failure. Each call reconfigures output queues > + * and may invalidate all previous queue handles. > + * > + * @param pktio Packet IO handle > + * @param param Packet output queue configuration parameters. Uses defaults > + * when NULL. > + * > + * @retval 0 on success > + * @retval <0 on failure > + * > + * @see odp_pktio_capability(), odp_pktout_queue(), odp_pktout_event_queue() > + */ > +int odp_pktout_queue_config(odp_pktio_t pktio, > + const odp_pktout_queue_param_t *param); > +----- > +As with +odp_pktin_queue_config()+, the configuration of PktOut queues > +involves the use of a parameter struct: > +[source,c] > +----- > +/** > + * Packet output queue parameters > + * > + * These parameters are used in ODP_PKTOUT_MODE_DIRECT and > + * ODP_PKTOUT_MODE_QUEUE modes. > + */ > +typedef struct odp_pktout_queue_param_t { > + /** Operation mode > + * > + * The default value is ODP_PKTIO_OP_MT. Application may enable > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE when > + * applicable. */ > + odp_pktio_op_mode_t op_mode; > + > + /** Number of output queues to be created. The value must be between > + * 1 and interface capability. The default value is 1. */ > + unsigned num_queues; > + > +} odp_pktout_queue_param_t; > +----- > +As with direct input, direct output uses one or more special output queues > +of type *odp_pktout_queue_t* that area created and configured by this call. typo: "area" / "are" > + > +As with PktIn queues, the handles for these created PktOut queues may be > +retrieved by the +odp_pktout_queue()+ API: > +[source,c] > +----- > +/** > + * Direct packet output queues > + * > + * Returns the number of output queues configured for the interface in > + * ODP_PKTOUT_MODE_DIRECT mode. Outputs up to 'num' queue handles when the > + * 'queues' array pointer is not NULL. If return value is larger than 'num', > + * there are more queues than the function was allowed to output. If return > + * value (N) is less than 'num', only queues[0 ... N-1] have been written. > + * > + * Packets are sent to these queues with odp_pktout_send(). > + * > + * @param pktio Packet IO handle > + * @param[out] queues Points to an array of queue handles for output > + * @param num Maximum number of queue handles to output > + * > + * @return Number of packet output queues > + * @retval <0 on failure > + */ > +int odp_pktout_queue(odp_pktio_t pktio, odp_pktout_queue_t queues[], int num); > +----- > +Once the PktIO has been configured for output and started via > ++odp_pktio_start()+, packets may be transmitted to the PktIO by calling > +_odp_pktout_send()+: "+" missing before _odp_pktout_send()+. (The + you see is from the patch!) > +[source,c] > +----- > +/** > + * Send packets directly to an interface output queue > + * > + * Sends out a number of packets to the interface output queue. When > + * output queue parameter 'op_mode' has been set to ODP_PKTIO_OP_MT_UNSAFE, > + * the operation is optimized for single thread operation per queue and the same > + * queue must not be accessed simultaneously from multiple threads. The above should be stated when discussing RX! > + * > + * A successful call returns the actual number of packets sent. If return value > + * is less than 'num', the remaining packets at the end of packets[] array > + * are not consumed, and the caller has to take care of them. > + * > + * @param queue Packet output queue handle for sending packets > + * @param packets[] Array of packets to send > + * @param num Number of packets to send > + * > + * @return Number of packets sent > + * @retval <0 on failure > + */ > +int odp_pktout_send(odp_pktout_queue_t queue, odp_packet_t packets[], int num); > +----- > +Note that the argument to this call specifies the PktOut queue to that the is "to that" english? (was expecting "to which") here. If it is, ignore this comment! > +packet is to be added to rather than the PktIO itself. This permits multiple > +threads (presumably operating on different cores) a more efficient means of > +separating I/O processing destined for the same interface. > + > +==== Queued I/O Modes > +To provide additional flexibility when operating in poll mode, PktIOs may also > +be opened in QUEUE Mode. The difference between DIRECT and QUEUE mode is that > +QUEUE mode uses standard ODP event queues to service packets. > + > +===== Queue RX Processing > +The processing for QUEUE input processing is shown below: > + > +.PktIO QUEUE Mode Receive Processing > +image::../images/pktin_queue_recv.svg[align="center"] same comment: this figure seems to imply that one can create an association between the queues and the threads at odp_pktin_qyeye_config time... > + > +In QUEUE mode, received packets are stored in one or more standard ODP queues. > +The difference is that these queues are not created directly by the > +application. Instead, they are created in response to an > ++odp_pktin_queue_config()+ call. > + > +As with DIRECT mode, the +odp_pktin_queue_param_t+ specified to this call > +indicates whether an input hash should be used and if so which field(s) of > +the packet should be considered as input to the has function. > + > +The main difference between DIRECT and QUEUE RX processing is that because > +the PktIO uses standard ODP event queues, other parts of the application can > +use +odp_queue_enq()+ API calls to enqueue packets to these queues for > +"RX" processing in addition to those originating from the PktIO interface > +itself. To obtain the handles of these input queues, the > ++odp_pktin_event_queue()+ API is used: > +[source,c] > +----- > +/** > + * Event queues for packet input > + * > + * Returns the number of input queues configured for the interface in > + * ODP_PKTIN_MODE_QUEUE and ODP_PKTIN_MODE_SCHED modes. Outputs up to 'num' > + * queue handles when the 'queues' array pointer is not NULL. If return value is > + * larger than 'num', there are more queues than the function was allowed to > + * output. If return value (N) is less than 'num', only queues[0 ... N-1] have > + * been written. > + * > + * Packets (and other events) from these queues are received with > + * odp_queue_deq(), odp_schedule(), etc calls. > + * > + * @param pktio Packet IO handle > + * @param[out] queues Points to an array of queue handles for output > + * @param num Maximum number of queue handles to output > + * > + * @return Number of packet input queues > + * @retval <0 on failure > + */ > +int odp_pktin_event_queue(odp_pktio_t pktio, odp_queue_t queues[], int num); > +----- > +Similarly, threads receive packets from PktIOs operating in QUEUE mode by > +making standard +odp_queue_deq()+ calls to one of the event queues associated > +with the PktIO. > + > +===== Queue TX Processing > +Transmit processing for PktIOs operating in QUEUE mode is shown below: > + > +.PktIO QUEUE Mode Transmit Processing > +image::../images/pktout_queue_send.svg[align="center] > + > +For TX processing QUEUE mode behaves similar to DIRECT mode except that > +output queues are regular ODP event queues that receive packets via > ++odp_queue_enq()+ calls rather than special PktOut queues that use > ++odp_pktout_send()+. Again, these queues are created via a call to > ++odp_pktout_queue_config()+ following +odp_pktio_open()+. > + > +The main reason for selecting QUEUE mode for output is flexibility. If an > +application is designed to use a _pipeline model_ where packets flow through > +a series of processing stages via queues, then having the PktIO in QUEUE > +mode means that the application can always use the same enq APIs to pass packets > +from one stage to the next, including the final transmit output stage. > + > +==== Scheduled I/O Modes > +The final PktIO mode supported integrates RX and TX processing with the ODP > +_event model_. For RX processing this involves the use of the *Scheduler* > +while for TX processing this involves the use of the *Traffic Manager*. > + > +Scheduled RX Processing is further divided based on whether or not the > +Classifier is used. > + > +===== Scheduled RX Processing > +When a PktIO is opened with +ODP_PKTIN_MODE_SCHED+, it indicates that the > +input queues created by a subsequent +odp_pktin_queue_config()+ call are to > +be used as input to the *ODP Scheduler*. > + > +.PktIO SCHED Mode Receive Processing > +image::../images/pktin_sched_recv.svg[align="center'] > + > +For basic use, SCHED mode simply associates the PktIO input event queues > +created by +odp_pktin_queue_config()+ with the scheduler. Hashing may still be > +employed to distribute input packets among multiple input queues. However > +instead of these being plain queues they are scheduled queues and have > +associated scheduling attributes like priority, scheduler group, and > +synchronization mode (parallel, atomic, ordered). SCHED mode thus provides > +both packet distribution (via the optional hash) as well as scalability via > +the ODP event model. > + > +In its fullest form, PktIOs operating in SCHED mode use the *ODP Classifier* > +to permit fine-grained flow separation on *Class of Service (CoS)* boundaries. > + > +.PktIO SCHED Mode Receive Processing with Classification > +image::../images/pktin_sched_cls.svg[align="center"] The diagram and the text confuses me a bit, here: my understanding was that: 1)the classifier associates a cos to each incoming packet using a set of PMRs. 2)Each cos maps to a pair {RXpool, RXqueue}. This Mapping is settled during the classifier setup. Packets with a given cos are stored on odp_packets allocated from the corresponding pool, and placed in the corresponding queue. > + > +In this mode of operation, the hash function of +odp_pktin_queue_config()+ is > +not used. Instead, the event queues created by this call, as well as any > +additional event queues created via separate +odp_queue_create()+ calls are > +associated with classes of service via +odp_cls_cos_create()+ calls. > + > +===== Scheduled TX Processing > +Scheduled transmit processing is performed via the *ODP Traffic Manager* and > +is requested when a PktIO is opened with an +out_mode+ of +ODP_PKTOUT_MODE_TM+. > + > +For TX processing via the Traffic Manager, applications use the +odp_tm_enq()+ > +API: > +[source,c] > +----- > +/** The odp_tm_enq() function is used to add packets to a given TM system. > + * Note that the System Metadata associated with the pkt needed by the TM > + * system is (a) a drop_eligible bit, (b) a two bit "pkt_color", (c) a 16-bit > + * pkt_len, and MAYBE? (d) a signed 8-bit shaper_len_adjust. > + * > + * If there is a non-zero shaper_len_adjust, then it is added to the pkt_len > + * after any non-zero shaper_len_adjust that is part of the shaper profile. > + * > + * The pkt_color bits are a result of some earlier Metering/Marking/Policing > + * processing (typically ingress based), and should not be confused with the > + * shaper_color produced from the TM shaper entities within the tm_inputs and > + * tm_nodes. > + * > + * @param[in] tm_queue Specifies the tm_queue (and indirectly the TM system). > + * @param[in] pkt Handle to a packet. > + * @return Returns 0 upon success, < 0 upon failure. One of the > + * more common failure reasons is WRED dropage. > + */ > +int odp_tm_enq(odp_tm_queue_t tm_queue, odp_packet_t pkt); > +----- > +See the *Traffic Manager* section of this document for full information about > +Traffic Manager configuration and operation. > diff --git a/doc/users-guide/users-guide.adoc b/doc/users-guide/users-guide.adoc > index d476225..ea24eaf 100644 > --- a/doc/users-guide/users-guide.adoc > +++ b/doc/users-guide/users-guide.adoc > @@ -1023,6 +1023,8 @@ a valid UDP packet. Applications are expected to exercise appropriate care > when changing packet metadata to ensure that the resulting metadata changes > reflect the actual changed packet structure that the application has made. > > +include::users-guide-pktio.adoc[] > + > == Cryptographic services > > ODP provides support for cryptographic operations required by various security > -- > 2.5.0 > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp Interresting reading, and nice to see the user doc growing! Thanks! Christophe
On Thu, Mar 3, 2016 at 4:00 AM, Christophe Milard < christophe.milard@linaro.org> wrote: > On 2016-03-02 04:35, Bill Fischofer wrote: > > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> > > --- > > doc/users-guide/users-guide-pktio.adoc | 605 > +++++++++++++++++++++++++++++++++ > > doc/users-guide/users-guide.adoc | 2 + > > 2 files changed, 607 insertions(+) > > create mode 100644 doc/users-guide/users-guide-pktio.adoc > > > > diff --git a/doc/users-guide/users-guide-pktio.adoc > b/doc/users-guide/users-guide-pktio.adoc > > new file mode 100644 > > index 0000000..29fb6a9 > > --- /dev/null > > +++ b/doc/users-guide/users-guide-pktio.adoc > > @@ -0,0 +1,605 @@ > > +== PktIO Processing > > +Before packets can be manipulated they typically need to be _received_ > and > > +after they are manipulated they need to be _transmitted_. The ODP > abstraction > > +that captures these operations is the *Packet I/O (PktIO)*. > > +PktIOs are represented by handles of type *odp_pktio_t* and > > +represent a logical I/O interface that is mapped in an > implementation-defined > > +manner to an underlying integrated I/O adapter or NIC. > > + > > +PktIO objects are manipulated through various state transitions via > > ++odp_pktio_xxx()+ API calls as shown below: > > + > > +.ODP PktIO Finite State Machine > > +image::../images/pktio_fsm.svg[align="center"] > > + > > +PktIOs begin in the *Unallocated* state. From here a call > +odp_pktio_open()+ > > +is used to create an *odp_pktio_t* handle that is used in all > subsequent calls > > +to manipulate the object. This call puts the PktIO into the > *Unconfigured* > > +state. To become operational, a PktIO must first be > > +*configured* for Input, Output, or both Input and Output via the > > ++odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ APIs, and > then > > +*started* via the +odp_pktio_start()+ to make it *Ready*. > > + > > +Following the completion of I/O processing, the +odp_pktio_stop()+ API > returns > > +the PktIO to the *Configured* state. From here it may be *Reconfigured* > via > > +additional +odp_pktin_queue_config()+ and/or > +odp_pktout_queue_config()+ calls, > > +or *Closed* via the +odp_pktio_close()+ API to return the PktIO to the > > +*Unallocated* state. > > + > > +=== PktIO Allocation > > +PktIO objects begin life by being _opened_ via the call: > > +[source,c] > > +----- > > +/** > > + * Open a packet IO interface > > + * > > + * An ODP program can open a single packet IO interface per device, > attempts > > what is a device here? a (possibly physical) interface? if we have a NIC > board with 2 > physical inerfaces, how many pktio can on open on it? > maybe: "An ODP program can open a single packet IO per interface"...? > You're not commenting on this document, but on the include/odp/api/spec/packet_io.h file that I'm just excerpting snippets from here. We can improve that commentary, but that would be a separate patch. I'd prefer to keep these two separate at this point and make any doxygen changes as a separate patch series. > > > + * to open an already open device will fail, returning > ODP_PKTIO_INVALID with > > + * errno set. Use odp_pktio_lookup() to obtain a handle to an already > open > > "opened" instead of open? > Same comment as above. > > > + * device. Packet IO parameters provide interface level configuration > options. > > + * > > + * Packet input queue configuration must be setup with > > + * odp_pktin_queue_config() before odp_pktio_start() is called. When > packet > > + * input mode is ODP_PKTIN_MODE_DISABLED, odp_pktin_queue_config() call > is > > + * optional and will ignore all parameters. > > + * > > + * Packet output queue configuration must be setup with > > + * odp_pktout_queue_config() before odp_pktio_start() is called. When > packet > > + * output mode is ODP_PKTOUT_MODE_DISABLED or ODP_PKTOUT_MODE_TM, > > + * odp_pktout_queue_config() call is optional and will ignore all > parameters. > > + * > > Is is not clear at this stage of reading what are the packet input and > output modes. > I assume that they are parameters to the pktio_open function (in param) as > this > is the only thing that is done before the configuration, maybe that should > be clarified before. (this assumption gets confirmed later on in the doc) > Simply adding "Pktio in/output mode will be discussed in the next section" > before > this code frgament would make sense. > Same comment as above. > > > + * Packet receive and transmit on the interface is enabled with a call > to > > + * odp_pktio_start(). If not specified otherwise, any interface level > > + * configuration must not be changed when the interface is active > (between start > > + * and stop calls). > > + * > > + * In summary, a typical pktio interface setup sequence is ... > > + * * odp_pktio_open() > > + * * odp_pktin_queue_config() > > + * * odp_pktout_queue_config() > > + * * odp_pktio_start() > > + * > > + * ... and tear down sequence is: > > + * * odp_pktio_stop() > > + * * odp_pktio_close() > > + * > > + * @param name Packet IO device name > > + * @param pool Default pool from which to allocate storage for packets > > + * received over this interface, must be of type > ODP_POOL_PACKET > > + * @param param Packet IO parameters > > + * > > + * @return Packet IO handle > > + * @retval ODP_PKTIO_INVALID on failure > > + * > > + * @note The device name "loop" is a reserved name for a loopback > device used > > + * for testing purposes. > > + * > > + * @note Packets arriving via this interface assigned to a CoS by the > > + * classifier are received into the pool associated with that CoS. > This > > + * will occur either because this pktio is assigned a default CoS via > > + * the odp_pktio_default_cos_set() routine, or because a matching PMR > > + * assigned the packet to a specific CoS. The default pool specified > > + * here is applicable only for those packets that are not assigned > to a > > + * more specific CoS. > > + * > > + * @see odp_pktio_start(), odp_pktio_stop(), odp_pktio_close() > > + */ > > +odp_pktio_t odp_pktio_open(const char *name, odp_pool_t pool, > > + const odp_pktio_param_t *param); > > +----- > > ++odp_pktio_open()+ takes three arguments: a *name*, which is an > > +implementation-defined string that identifies the logical interface to > be > > +opened, a *pool* that identifies the ODP pool that storage for received > > +packets should be allocated from, and a *param* structure that specifies > > +I/O options to be associated with this PktIO instance. > > +[source,c] > > +----- > > +/** > > + * Packet IO parameters > > + * > > + * In minimum, user must select input and output modes. Use 0 for > defaults. > > + * Initialize entire struct with zero to maintain API compatibility. > > + */ > > +typedef struct odp_pktio_param_t { > > + /** Packet input mode > > + * > > + * The default value is ODP_PKTIN_MODE_DIRECT. */ > > + odp_pktin_mode_t in_mode; > > + /** Packet output mode > > + * > > + * The default value is ODP_PKTOUT_MODE_DIRECT. */ > > + odp_pktout_mode_t out_mode; > > +} odp_pktio_param_t; > > +----- > > +ODP defines *loop* as a reserved name to indicate that this PktIO > represents > > I think "loop" should be quoted as it is a string. > It's bolded for that purpose, but I have no objection to adding quotes. Of course there will always be those who see and try to type the quotes and then wonder why it didn't work. :) > > > +a loopback interface. Loopback interfaces are useful as a means of > recycling > > +packets back for reclassification after decryption or decapsulation, as > well as > > +for diagnostic or testing purposes. The *pool* specifies the default > pool to > > "Loopback interfaces are useful as a means of recycling packets back for > reclassification after decryption or decapsulation". I don't understand > what you > mean by that... should I? ;-) > These are two common reasons why you'd use a loopback interface in non-testing mode. Until you decrypt an IPSEC packet you really have no idea what's really in it. Similar considerations apply to tunneled packets that need to be decapsulated before the "real" packet is visible. > > > +use for packet allocation if not overridden by the classifier due to a > > +specific or default Class-of-Service (CoS) match on the packet. The > *param* > > +struct, in turn, specifies the input and output *modes* of the PktIO. > > + > > +=== PktIO Input and Output Modes > > +PktIO objects support four different Input and Output modes, that may be > > +specified independently at *open* time. > > + > > +.PktIO Input Modes > > +* +ODP_PKTIN_MODE_DIRECT+ > > +* +ODP_PKTIN_MODE_QUEUE+ > > +* +ODP_OKTIN_MODE_SCHED+ > > +* +ODP_PKTIN_MODE_DISABLED+ > > + > > +.PktIO Output Modes > > +* +ODP_PKTOUT_MODE_DIRECT+ > > +* +ODP_PKTOUT_MODE_QUEUE+ > > +* +ODP_PKTOUT_MODE_TM+ > > +* +ODP_PKTOUT_MODE_DISABLED+ > > + > > +The DISABLED modes indicate that either input or output is prohibited > on this > > +PktIO. Attempts to receive packets on a PktIO whose +in_mode+ is > DISABLED > > +return no packets while packets sent to a PktIO whose +out_mode+ is > DISABLED > > +are discarded. > > + > > +==== Direct I/O Modes > > +DIRECT I/O is the default mode for PktIO objects. It is designed to > support > > +poll-based packet processing, which is often found in legacy > applications > > +being ported to ODP, and can also be a preferred mode for some types of > > +packet processing. By supporting poll-based I/O processing, ODP provides > > +maximum flexibility to the data plane application writer. > > + > > +===== Direct RX Processing > > +The processing of DIRECT input is shown below: > > + > > +.PktIO DIRECT Mode Receive Processing > > +image::../images/pktin_direct_recv.svg[align="center"] > > + > > From this picture, I understand that a each rx pktio queue is "associated" > to a RX thread at odp_pktin_queue_config time. however, it does not feel > like > that at all in the rest of the document: odp_pktin_recv() take a queue > handle > as parameter, hence allowing any ODP thread to read packets from any queue > at any time... (the "operation mode", -not really defined for RX- also > implies > that such an association does not exist, at least in unsafe mode) > That is correct. The intent is to allow applications to have a one-to-one map between threads and RX queues, but ODP does not constrain applications to follow such a design. odp_pktin_queue_config() simply says how many RX queues to create/associated with the PktIO. The discipline of how threads use these queues is up to the application. I can add another paragraph here if that isn't clear in context. > > > +In DIRECT mode, received packets are stored in one or more special > PktIO queues > > +of type *odp_pktin_queue_t* and are retrieved by threads calling the > > ++odp_pktin_recv()+ API. > > + > > +Once opened, setting up a DIRECT mode PktIO is performed by the > > ++odp_pktin_queue_config()+ API. > > +[source,c] > > +----- > > +/** > > + * Configure packet input queues > > + * > > + * Setup a number of packet input queues and configure those. The > maximum number > > + * of queues is platform dependent and can be queried with > > + * odp_pktio_capability(). Use odp_pktin_queue_param_init() to > initialize > > + * parameters into their default values. Default values are also used > when > > + * 'param' pointer is NULL. > > + * > > + * Queue handles for input queues can be requested with > odp_pktin_queue() or > > + * odp_pktin_event_queue() after this call. All requested queues are > setup on > > + * success, no queues are setup on failure. Each call reconfigures > input queues > > + * and may invalidate all previous queue handles. > > + * > > + * @param pktio Packet IO handle > > + * @param param Packet input queue configuration parameters. Uses > defaults > > + * when NULL. > > + * > > + * @retval 0 on success > > + * @retval <0 on failure > > + * > > + * @see odp_pktio_capability(), odp_pktin_queue(), > odp_pktin_event_queue() > > + */ > > +int odp_pktin_queue_config(odp_pktio_t pktio, > > + const odp_pktin_queue_param_t *param); > > +----- > > +The second argument to this call is the *odp_pktin_queue_param_t* > > +[source,c] > > +----- > > +/** > > + * Packet input queue parameters > > + */ > > +typedef struct odp_pktin_queue_param_t { > > + /** Operation mode > > + * > > + * The default value is ODP_PKTIO_OP_MT. Application may enable > > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE > when > > + * applicable. */ > > + odp_pktio_op_mode_t op_mode; > > Has the "operation mode" been defined anywhere? > not sure what it is. neither do I > know what ODP_PKTIO_OP_MT means at this stage of reading... > This is defined only when talking about TX, later on... > It should be defined here (and possibly reminded for TX) > Doxygen comments again--not part of this patch series > > > + > > + /** Enable flow hashing > > + * 0: Do not hash flows > > + * 1: Hash flows to input queues */ > > + odp_bool_t hash_enable; > > + > > + /** Protocol field selection for hashing. Multiple protocols can be > > + * selected. */ > > + odp_pktin_hash_proto_t hash_proto; > > + > > + /** Number of input queues to be created. More than one input queue > > + * require input hashing or classifier setup. Hash_proto is > ignored > > + * when hash_enable is zero or num_queues is one. This value must > be > > + * between 1 and interface capability. Queue type is defined by > the > > + * input mode. The default value is 1. */ > > + unsigned num_queues; > > + > > + /** Queue parameters for creating input queues in > ODP_PKTIN_MODE_QUEUE > > + * or ODP_PKTIN_MODE_SCHED modes. Scheduler parameters are > considered > > + * only in ODP_PKTIN_MODE_SCHED mode. */ > > + odp_queue_param_t queue_param; > > + > > +} odp_pktin_queue_param_t; > > +----- > > +Note that the *queue_param* field of this struct is ignored in DIRECT > mode. > > +The purpose of +odp_pktin_queue_config()+ is to specify the number of > PktIn > > +queues to be created and to set their attributes. > > + > > +===== Hash Processing > > +Another feature of DIRECT mode input is the provision of a *hash* > function used > > +to distribute incoming packets among the PktIO's PktIn queues. If the > > ++hash_enable+ field of the *odp_pktin_queue_param_t* is 1, > > +then the +hash_proto+ field is used to specify which field(s) of > incoming > > +packets should be used as input to an implementation-defined packet > > +distribution hash function. > > +[source,c] > > +----- > > +/** > > + * Packet input hash protocols > > + * > > + * The list of protocol header field combinations, which are included > into > > + * packet input hash calculation. > > + */ > > +typedef union odp_pktin_hash_proto_t { > > + /** Protocol header fields for hashing */ > > + struct { > > + /** IPv4 addresses and UDP port numbers */ > > + uint32_t ipv4_udp : 1; > > + /** IPv4 addresses and TCP port numbers */ > > + uint32_t ipv4_tcp : 1; > > + /** IPv4 addresses */ > > + uint32_t ipv4 : 1; > > + /** IPv6 addresses and UDP port numbers */ > > + uint32_t ipv6_udp : 1; > > + /** IPv6 addresses and TCP port numbers */ > > + uint32_t ipv6_tcp : 1; > > + /** IPv6 addresses */ > > + uint32_t ipv6 : 1; > > + } proto; > > + > > + /** All bits of the bit field structure */ > > + uint32_t all_bits; > > +} odp_pktin_hash_proto_t; > > +----- > > +Note that the hash function used in PktIO poll mode operation is > intended to > > +provide simple packet distribution among multiple PktIn queues > associated with > > +the PktIO. It does not have the sophistication of the *ODP Classifier*, > however > > +it also does not incur the setup requirements of pattern matching rules, > > +making it a simpler choice for less sophisticated applications. Note > that > > +ODP does not specify how the hash is to be performed. That is left to > each > > +implementation. The hash only specifies which input packet fields are of > > +interest to the application and should be considered by the hash > function in > > +deciding how to distribute packets among PktIn queues. The only > expectation > > +is that packets that have the same hash values should all be mapped to > the > > +same PktIn queue. > > + > > +===== PktIn Queues > > +A *PktIn Queue* is a special type of queue that is used internally by > PktIOs > > +operating in DIRECT mode. Applications cannot perform enqueues to these > queues, > > +however they may obtain references to them via the +odp_pktin_queue()+ > API > > +[source,c] > > +----- > > +/** > > + * Direct packet input queues > > + * > > + * Returns the number of input queues configured for the interface in > > + * ODP_PKTIN_MODE_DIRECT mode. Outputs up to 'num' queue handles when > the > > + * 'queues' array pointer is not NULL. If return value is larger than > 'num', > > + * there are more queues than the function was allowed to output. If > return > > + * value (N) is less than 'num', only queues[0 ... N-1] have been > written. > > + * > > + * Packets from these queues are received with odp_pktin_recv(). > > + * > > + * @param pktio Packet IO handle > > + * @param[out] queues Points to an array of queue handles for output > > + * @param num Maximum number of queue handles to output > > + * > > + * @return Number of packet input queues > > + * @retval <0 on failure > > + */ > > +int odp_pktin_queue(odp_pktio_t pktio, odp_pktin_queue_t queues[], int > num); > > +----- > > +Once configured, prior to receiving packets the PktIO must be placed > into the > > +*Ready* state via a call to +odp_pktio_start() > > +[source,c] > > +----- > > +/** > > + * Start packet receive and transmit > > + * > > + * Activate packet receive and transmit on a previously opened or > stopped > > + * interface. The interface can be stopped with a call to > odp_pktio_stop(). > > + * > > + * @param pktio Packet IO handle > > + * > > + * @retval 0 on success > > + * @retval <0 on failure > > + * > > + * @see odp_pktio_open(), odp_pktio_stop() > > + */ > > +int odp_pktio_start(odp_pktio_t pktio); > > +----- > > +Once started, the PktIn queue handles are used as arguments to > > ++odp_pktin_recv()+ to receive packets from the PktIO. > > +[source,c] > > +----- > > +/** > > + * Receive packets directly from an interface input queue > > + * > > + * Receives up to 'num' packets from the pktio interface input queue. > When > > + * input queue parameter 'op_mode' has been set to > ODP_PKTIO_OP_MT_UNSAFE, > > + * the operation is optimized for single thread operation per queue and > the same > > + * queue must not be accessed simultaneously from multiple threads. > > + * > > + * @param queue Packet input queue handle for receiving > packets > > + * @param[out] packets[] Packet handle array for output of received > packets > > + * @param num Maximum number of packets to receive > > + * > > + * @return Number of packets received > > + * @retval <0 on failure > > + * > > + * @see odp_pktin_queue() > > + */ > > +int odp_pktin_recv(odp_pktin_queue_t queue, odp_packet_t packets[], int > num); > > This means that any thread can read from any queue at any time, right? > (does not match my understanding of figure 13) > That is correct. As noted above, ODP just provides the tools. Applications must use these tools intelligently. > > > +----- > > + > > +===== Direct TX Processing > > +A PktIO operating in DIRECT mode performs TX processing as shown here: > > + > > +.PktIO DIRECT Mode Transmit Processing > > +image::../images/pktout_direct_send.svg[align="center"] > > + > > +Direct TX processing operates similarly to Direct RX processing. > Following > > +open, the +odp_pktout_queue_config()+ API is used to create and > configure > > +one or more *PktOut queues* to be used to support packet transmission by > > +this PktIO > > +[source,c] > > +----- > > +/** > > + * Configure packet output queues > > + * > > + * Setup a number of packet output queues and configure those. The > maximum > > + * number of queues is platform dependent and can be queried with > > + * odp_pktio_capability(). Use odp_pktout_queue_param_init() to > initialize > > + * parameters into their default values. Default values are also used > when > > + * 'param' pointer is NULL. > > + * > > + * Queue handles for output queues can be requested with > odp_pktout_queue() or > > + * odp_pktout_event_queue() after this call. All requested queues are > setup on > > + * success, no queues are setup on failure. Each call reconfigures > output queues > > + * and may invalidate all previous queue handles. > > + * > > + * @param pktio Packet IO handle > > + * @param param Packet output queue configuration parameters. Uses > defaults > > + * when NULL. > > + * > > + * @retval 0 on success > > + * @retval <0 on failure > > + * > > + * @see odp_pktio_capability(), odp_pktout_queue(), > odp_pktout_event_queue() > > + */ > > +int odp_pktout_queue_config(odp_pktio_t pktio, > > + const odp_pktout_queue_param_t *param); > > +----- > > +As with +odp_pktin_queue_config()+, the configuration of PktOut queues > > +involves the use of a parameter struct: > > +[source,c] > > +----- > > +/** > > + * Packet output queue parameters > > + * > > + * These parameters are used in ODP_PKTOUT_MODE_DIRECT and > > + * ODP_PKTOUT_MODE_QUEUE modes. > > + */ > > +typedef struct odp_pktout_queue_param_t { > > + /** Operation mode > > + * > > + * The default value is ODP_PKTIO_OP_MT. Application may enable > > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE > when > > + * applicable. */ > > + odp_pktio_op_mode_t op_mode; > > + > > + /** Number of output queues to be created. The value must be > between > > + * 1 and interface capability. The default value is 1. */ > > + unsigned num_queues; > > + > > +} odp_pktout_queue_param_t; > > +----- > > +As with direct input, direct output uses one or more special output > queues > > +of type *odp_pktout_queue_t* that area created and configured by this > call. > > typo: "area" / "are" > > Good catch. Thanks. > > + > > +As with PktIn queues, the handles for these created PktOut queues may be > > +retrieved by the +odp_pktout_queue()+ API: > > +[source,c] > > +----- > > +/** > > + * Direct packet output queues > > + * > > + * Returns the number of output queues configured for the interface in > > + * ODP_PKTOUT_MODE_DIRECT mode. Outputs up to 'num' queue handles when > the > > + * 'queues' array pointer is not NULL. If return value is larger than > 'num', > > + * there are more queues than the function was allowed to output. If > return > > + * value (N) is less than 'num', only queues[0 ... N-1] have been > written. > > + * > > + * Packets are sent to these queues with odp_pktout_send(). > > + * > > + * @param pktio Packet IO handle > > + * @param[out] queues Points to an array of queue handles for output > > + * @param num Maximum number of queue handles to output > > + * > > + * @return Number of packet output queues > > + * @retval <0 on failure > > + */ > > +int odp_pktout_queue(odp_pktio_t pktio, odp_pktout_queue_t queues[], > int num); > > +----- > > +Once the PktIO has been configured for output and started via > > ++odp_pktio_start()+, packets may be transmitted to the PktIO by calling > > +_odp_pktout_send()+: > > "+" missing before _odp_pktout_send()+. (The + you see is from the patch!) > Thank you. > > > +[source,c] > > +----- > > +/** > > + * Send packets directly to an interface output queue > > + * > > + * Sends out a number of packets to the interface output queue. When > > + * output queue parameter 'op_mode' has been set to > ODP_PKTIO_OP_MT_UNSAFE, > > + * the operation is optimized for single thread operation per queue and > the same > > + * queue must not be accessed simultaneously from multiple threads. > > The above should be stated when discussing RX! > Noted. v3 will add additional detail to the narrative. > > > + * > > + * A successful call returns the actual number of packets sent. If > return value > > + * is less than 'num', the remaining packets at the end of packets[] > array > > + * are not consumed, and the caller has to take care of them. > > + * > > + * @param queue Packet output queue handle for sending packets > > + * @param packets[] Array of packets to send > > + * @param num Number of packets to send > > + * > > + * @return Number of packets sent > > + * @retval <0 on failure > > + */ > > +int odp_pktout_send(odp_pktout_queue_t queue, odp_packet_t packets[], > int num); > > +----- > > +Note that the argument to this call specifies the PktOut queue to that > the > > is "to that" english? (was expecting "to which") here. If it is, ignore > this > comment! > "to" is an extraneous word here. Good catch. > > > +packet is to be added to rather than the PktIO itself. This permits > multiple > > +threads (presumably operating on different cores) a more efficient > means of > > +separating I/O processing destined for the same interface. > > + > > +==== Queued I/O Modes > > +To provide additional flexibility when operating in poll mode, PktIOs > may also > > +be opened in QUEUE Mode. The difference between DIRECT and QUEUE mode > is that > > +QUEUE mode uses standard ODP event queues to service packets. > > + > > +===== Queue RX Processing > > +The processing for QUEUE input processing is shown below: > > + > > +.PktIO QUEUE Mode Receive Processing > > +image::../images/pktin_queue_recv.svg[align="center"] > > same comment: this figure seems to imply that one can create an association > between the queues and the threads at odp_pktin_qyeye_config time... > No, it's showing how the created queues can be used most efficiently. I'll expand the comments around this in v3. > > > + > > +In QUEUE mode, received packets are stored in one or more standard ODP > queues. > > +The difference is that these queues are not created directly by the > > +application. Instead, they are created in response to an > > ++odp_pktin_queue_config()+ call. > > + > > +As with DIRECT mode, the +odp_pktin_queue_param_t+ specified to this > call > > +indicates whether an input hash should be used and if so which field(s) > of > > +the packet should be considered as input to the has function. > > + > > +The main difference between DIRECT and QUEUE RX processing is that > because > > +the PktIO uses standard ODP event queues, other parts of the > application can > > +use +odp_queue_enq()+ API calls to enqueue packets to these queues for > > +"RX" processing in addition to those originating from the PktIO > interface > > +itself. To obtain the handles of these input queues, the > > ++odp_pktin_event_queue()+ API is used: > > +[source,c] > > +----- > > +/** > > + * Event queues for packet input > > + * > > + * Returns the number of input queues configured for the interface in > > + * ODP_PKTIN_MODE_QUEUE and ODP_PKTIN_MODE_SCHED modes. Outputs up to > 'num' > > + * queue handles when the 'queues' array pointer is not NULL. If return > value is > > + * larger than 'num', there are more queues than the function was > allowed to > > + * output. If return value (N) is less than 'num', only queues[0 ... > N-1] have > > + * been written. > > + * > > + * Packets (and other events) from these queues are received with > > + * odp_queue_deq(), odp_schedule(), etc calls. > > + * > > + * @param pktio Packet IO handle > > + * @param[out] queues Points to an array of queue handles for output > > + * @param num Maximum number of queue handles to output > > + * > > + * @return Number of packet input queues > > + * @retval <0 on failure > > + */ > > +int odp_pktin_event_queue(odp_pktio_t pktio, odp_queue_t queues[], int > num); > > +----- > > +Similarly, threads receive packets from PktIOs operating in QUEUE mode > by > > +making standard +odp_queue_deq()+ calls to one of the event queues > associated > > +with the PktIO. > > + > > +===== Queue TX Processing > > +Transmit processing for PktIOs operating in QUEUE mode is shown below: > > + > > +.PktIO QUEUE Mode Transmit Processing > > +image::../images/pktout_queue_send.svg[align="center] > > + > > +For TX processing QUEUE mode behaves similar to DIRECT mode except that > > +output queues are regular ODP event queues that receive packets via > > ++odp_queue_enq()+ calls rather than special PktOut queues that use > > ++odp_pktout_send()+. Again, these queues are created via a call to > > ++odp_pktout_queue_config()+ following +odp_pktio_open()+. > > + > > +The main reason for selecting QUEUE mode for output is flexibility. If > an > > +application is designed to use a _pipeline model_ where packets flow > through > > +a series of processing stages via queues, then having the PktIO in QUEUE > > +mode means that the application can always use the same enq APIs to > pass packets > > +from one stage to the next, including the final transmit output stage. > > + > > +==== Scheduled I/O Modes > > +The final PktIO mode supported integrates RX and TX processing with the > ODP > > +_event model_. For RX processing this involves the use of the > *Scheduler* > > +while for TX processing this involves the use of the *Traffic Manager*. > > + > > +Scheduled RX Processing is further divided based on whether or not the > > +Classifier is used. > > + > > +===== Scheduled RX Processing > > +When a PktIO is opened with +ODP_PKTIN_MODE_SCHED+, it indicates that > the > > +input queues created by a subsequent +odp_pktin_queue_config()+ call > are to > > +be used as input to the *ODP Scheduler*. > > + > > +.PktIO SCHED Mode Receive Processing > > +image::../images/pktin_sched_recv.svg[align="center'] > > + > > +For basic use, SCHED mode simply associates the PktIO input event queues > > +created by +odp_pktin_queue_config()+ with the scheduler. Hashing may > still be > > +employed to distribute input packets among multiple input queues. > However > > +instead of these being plain queues they are scheduled queues and have > > +associated scheduling attributes like priority, scheduler group, and > > +synchronization mode (parallel, atomic, ordered). SCHED mode thus > provides > > +both packet distribution (via the optional hash) as well as scalability > via > > +the ODP event model. > > + > > +In its fullest form, PktIOs operating in SCHED mode use the *ODP > Classifier* > > +to permit fine-grained flow separation on *Class of Service (CoS)* > boundaries. > > + > > +.PktIO SCHED Mode Receive Processing with Classification > > +image::../images/pktin_sched_cls.svg[align="center"] > > The diagram and the text confuses me a bit, here: my understanding was > that: > 1)the classifier associates a cos to each incoming packet using a set of > PMRs. > 2)Each cos maps to a pair {RXpool, RXqueue}. This Mapping is settled > during the > classifier setup. Packets with a given cos are stored on odp_packets > allocated > from the corresponding pool, and placed in the corresponding queue. > That is correct. What's unclear about that here? The point is that the queues shown here may either be those that the application creates via it's own odp_queue_create() calls or those created by odp_pktin_queue_config(). If the latter, the application will use odp_pktin_queue() to retrieve their handle to be passed when it creates the CoS it's to be associated with. Remember, multiple CoSes can add packets to the same queue--this is not a 1-to-1 association here. > > > + > > +In this mode of operation, the hash function of > +odp_pktin_queue_config()+ is > > +not used. Instead, the event queues created by this call, as well as any > > +additional event queues created via separate +odp_queue_create()+ calls > are > > +associated with classes of service via +odp_cls_cos_create()+ calls. > > + > > +===== Scheduled TX Processing > > +Scheduled transmit processing is performed via the *ODP Traffic > Manager* and > > +is requested when a PktIO is opened with an +out_mode+ of > +ODP_PKTOUT_MODE_TM+. > > + > > +For TX processing via the Traffic Manager, applications use the > +odp_tm_enq()+ > > +API: > > +[source,c] > > +----- > > +/** The odp_tm_enq() function is used to add packets to a given TM > system. > > + * Note that the System Metadata associated with the pkt needed by the > TM > > + * system is (a) a drop_eligible bit, (b) a two bit "pkt_color", (c) a > 16-bit > > + * pkt_len, and MAYBE? (d) a signed 8-bit shaper_len_adjust. > > + * > > + * If there is a non-zero shaper_len_adjust, then it is added to the > pkt_len > > + * after any non-zero shaper_len_adjust that is part of the shaper > profile. > > + * > > + * The pkt_color bits are a result of some earlier > Metering/Marking/Policing > > + * processing (typically ingress based), and should not be confused > with the > > + * shaper_color produced from the TM shaper entities within the > tm_inputs and > > + * tm_nodes. > > + * > > + * @param[in] tm_queue Specifies the tm_queue (and indirectly the TM > system). > > + * @param[in] pkt Handle to a packet. > > + * @return Returns 0 upon success, < 0 upon failure. One > of the > > + * more common failure reasons is WRED dropage. > > + */ > > +int odp_tm_enq(odp_tm_queue_t tm_queue, odp_packet_t pkt); > > +----- > > +See the *Traffic Manager* section of this document for full information > about > > +Traffic Manager configuration and operation. > > diff --git a/doc/users-guide/users-guide.adoc > b/doc/users-guide/users-guide.adoc > > index d476225..ea24eaf 100644 > > --- a/doc/users-guide/users-guide.adoc > > +++ b/doc/users-guide/users-guide.adoc > > @@ -1023,6 +1023,8 @@ a valid UDP packet. Applications are expected to > exercise appropriate care > > when changing packet metadata to ensure that the resulting metadata > changes > > reflect the actual changed packet structure that the application has > made. > > > > +include::users-guide-pktio.adoc[] > > + > > == Cryptographic services > > > > ODP provides support for cryptographic operations required by various > security > > -- > > 2.5.0 > > > > _______________________________________________ > > lng-odp mailing list > > lng-odp@lists.linaro.org > > https://lists.linaro.org/mailman/listinfo/lng-odp > > Interresting reading, and nice to see the user doc growing! Thanks! > Thanks for the careful review and questions. > > Christophe >
I know the comments are copied from the source, but thy are just copied (not referenced), right? I mean, if it does not get fixed here, there will be 2 places to fix... If you have the courage, maybe the source comment can be fixed in a earlyer patch. These comments are relatively minor details. Fix what you can: This text is much better that the emptiness that was before. On 3 March 2016 at 17:12, Bill Fischofer <bill.fischofer@linaro.org> wrote: > > > On Thu, Mar 3, 2016 at 4:00 AM, Christophe Milard < > christophe.milard@linaro.org> wrote: > >> On 2016-03-02 04:35, Bill Fischofer wrote: >> > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >> > --- >> > doc/users-guide/users-guide-pktio.adoc | 605 >> +++++++++++++++++++++++++++++++++ >> > doc/users-guide/users-guide.adoc | 2 + >> > 2 files changed, 607 insertions(+) >> > create mode 100644 doc/users-guide/users-guide-pktio.adoc >> > >> > diff --git a/doc/users-guide/users-guide-pktio.adoc >> b/doc/users-guide/users-guide-pktio.adoc >> > new file mode 100644 >> > index 0000000..29fb6a9 >> > --- /dev/null >> > +++ b/doc/users-guide/users-guide-pktio.adoc >> > @@ -0,0 +1,605 @@ >> > +== PktIO Processing >> > +Before packets can be manipulated they typically need to be _received_ >> and >> > +after they are manipulated they need to be _transmitted_. The ODP >> abstraction >> > +that captures these operations is the *Packet I/O (PktIO)*. >> > +PktIOs are represented by handles of type *odp_pktio_t* and >> > +represent a logical I/O interface that is mapped in an >> implementation-defined >> > +manner to an underlying integrated I/O adapter or NIC. >> > + >> > +PktIO objects are manipulated through various state transitions via >> > ++odp_pktio_xxx()+ API calls as shown below: >> > + >> > +.ODP PktIO Finite State Machine >> > +image::../images/pktio_fsm.svg[align="center"] >> > + >> > +PktIOs begin in the *Unallocated* state. From here a call >> +odp_pktio_open()+ >> > +is used to create an *odp_pktio_t* handle that is used in all >> subsequent calls >> > +to manipulate the object. This call puts the PktIO into the >> *Unconfigured* >> > +state. To become operational, a PktIO must first be >> > +*configured* for Input, Output, or both Input and Output via the >> > ++odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ APIs, >> and then >> > +*started* via the +odp_pktio_start()+ to make it *Ready*. >> > + >> > +Following the completion of I/O processing, the +odp_pktio_stop()+ API >> returns >> > +the PktIO to the *Configured* state. From here it may be >> *Reconfigured* via >> > +additional +odp_pktin_queue_config()+ and/or >> +odp_pktout_queue_config()+ calls, >> > +or *Closed* via the +odp_pktio_close()+ API to return the PktIO to the >> > +*Unallocated* state. >> > + >> > +=== PktIO Allocation >> > +PktIO objects begin life by being _opened_ via the call: >> > +[source,c] >> > +----- >> > +/** >> > + * Open a packet IO interface >> > + * >> > + * An ODP program can open a single packet IO interface per device, >> attempts >> >> what is a device here? a (possibly physical) interface? if we have a NIC >> board with 2 >> physical inerfaces, how many pktio can on open on it? >> maybe: "An ODP program can open a single packet IO per interface"...? >> > > You're not commenting on this document, but on the > include/odp/api/spec/packet_io.h file that I'm just excerpting snippets > from here. We can improve that commentary, but that would be a separate > patch. I'd prefer to keep these two separate at this point and make any > doxygen changes as a separate patch series. > yes I know. see my comment on the top. > >> > + * to open an already open device will fail, returning >> ODP_PKTIO_INVALID with >> > + * errno set. Use odp_pktio_lookup() to obtain a handle to an already >> open >> >> "opened" instead of open? >> > > Same comment as above. > > >> >> > + * device. Packet IO parameters provide interface level configuration >> options. >> > + * >> > + * Packet input queue configuration must be setup with >> > + * odp_pktin_queue_config() before odp_pktio_start() is called. When >> packet >> > + * input mode is ODP_PKTIN_MODE_DISABLED, odp_pktin_queue_config() >> call is >> > + * optional and will ignore all parameters. >> > + * >> > + * Packet output queue configuration must be setup with >> > + * odp_pktout_queue_config() before odp_pktio_start() is called. When >> packet >> > + * output mode is ODP_PKTOUT_MODE_DISABLED or ODP_PKTOUT_MODE_TM, >> > + * odp_pktout_queue_config() call is optional and will ignore all >> parameters. >> > + * >> >> Is is not clear at this stage of reading what are the packet input and >> output modes. >> I assume that they are parameters to the pktio_open function (in param) >> as this >> is the only thing that is done before the configuration, maybe that should >> be clarified before. (this assumption gets confirmed later on in the doc) >> Simply adding "Pktio in/output mode will be discussed in the next >> section" before >> this code frgament would make sense. >> > > Same comment as above. > > >> >> > + * Packet receive and transmit on the interface is enabled with a call >> to >> > + * odp_pktio_start(). If not specified otherwise, any interface level >> > + * configuration must not be changed when the interface is active >> (between start >> > + * and stop calls). >> > + * >> > + * In summary, a typical pktio interface setup sequence is ... >> > + * * odp_pktio_open() >> > + * * odp_pktin_queue_config() >> > + * * odp_pktout_queue_config() >> > + * * odp_pktio_start() >> > + * >> > + * ... and tear down sequence is: >> > + * * odp_pktio_stop() >> > + * * odp_pktio_close() >> > + * >> > + * @param name Packet IO device name >> > + * @param pool Default pool from which to allocate storage for >> packets >> > + * received over this interface, must be of type >> ODP_POOL_PACKET >> > + * @param param Packet IO parameters >> > + * >> > + * @return Packet IO handle >> > + * @retval ODP_PKTIO_INVALID on failure >> > + * >> > + * @note The device name "loop" is a reserved name for a loopback >> device used >> > + * for testing purposes. >> > + * >> > + * @note Packets arriving via this interface assigned to a CoS by the >> > + * classifier are received into the pool associated with that CoS. >> This >> > + * will occur either because this pktio is assigned a default CoS >> via >> > + * the odp_pktio_default_cos_set() routine, or because a matching >> PMR >> > + * assigned the packet to a specific CoS. The default pool specified >> > + * here is applicable only for those packets that are not assigned >> to a >> > + * more specific CoS. >> > + * >> > + * @see odp_pktio_start(), odp_pktio_stop(), odp_pktio_close() >> > + */ >> > +odp_pktio_t odp_pktio_open(const char *name, odp_pool_t pool, >> > + const odp_pktio_param_t *param); >> > +----- >> > ++odp_pktio_open()+ takes three arguments: a *name*, which is an >> > +implementation-defined string that identifies the logical interface to >> be >> > +opened, a *pool* that identifies the ODP pool that storage for received >> > +packets should be allocated from, and a *param* structure that >> specifies >> > +I/O options to be associated with this PktIO instance. >> > +[source,c] >> > +----- >> > +/** >> > + * Packet IO parameters >> > + * >> > + * In minimum, user must select input and output modes. Use 0 for >> defaults. >> > + * Initialize entire struct with zero to maintain API compatibility. >> > + */ >> > +typedef struct odp_pktio_param_t { >> > + /** Packet input mode >> > + * >> > + * The default value is ODP_PKTIN_MODE_DIRECT. */ >> > + odp_pktin_mode_t in_mode; >> > + /** Packet output mode >> > + * >> > + * The default value is ODP_PKTOUT_MODE_DIRECT. */ >> > + odp_pktout_mode_t out_mode; >> > +} odp_pktio_param_t; >> > +----- >> > +ODP defines *loop* as a reserved name to indicate that this PktIO >> represents >> >> I think "loop" should be quoted as it is a string. >> > > It's bolded for that purpose, but I have no objection to adding quotes. Of > course there will always be those who see and try to type the quotes and > then wonder why it didn't work. :) > >> >> > +a loopback interface. Loopback interfaces are useful as a means of >> recycling >> > +packets back for reclassification after decryption or decapsulation, >> as well as >> > +for diagnostic or testing purposes. The *pool* specifies the default >> pool to >> >> "Loopback interfaces are useful as a means of recycling packets back for >> reclassification after decryption or decapsulation". I don't understand >> what you >> mean by that... should I? ;-) >> > > These are two common reasons why you'd use a loopback interface in > non-testing mode. Until you decrypt an IPSEC packet you really have no > idea what's really in it. Similar considerations apply to tunneled packets > that need to be decapsulated before the "real" packet is visible. > I understand that, but I cannot see how a loopback interface would help here. You can explain at BKK ! > > >> >> > +use for packet allocation if not overridden by the classifier due to a >> > +specific or default Class-of-Service (CoS) match on the packet. The >> *param* >> > +struct, in turn, specifies the input and output *modes* of the PktIO. >> > + >> > +=== PktIO Input and Output Modes >> > +PktIO objects support four different Input and Output modes, that may >> be >> > +specified independently at *open* time. >> > + >> > +.PktIO Input Modes >> > +* +ODP_PKTIN_MODE_DIRECT+ >> > +* +ODP_PKTIN_MODE_QUEUE+ >> > +* +ODP_OKTIN_MODE_SCHED+ >> > +* +ODP_PKTIN_MODE_DISABLED+ >> > + >> > +.PktIO Output Modes >> > +* +ODP_PKTOUT_MODE_DIRECT+ >> > +* +ODP_PKTOUT_MODE_QUEUE+ >> > +* +ODP_PKTOUT_MODE_TM+ >> > +* +ODP_PKTOUT_MODE_DISABLED+ >> > + >> > +The DISABLED modes indicate that either input or output is prohibited >> on this >> > +PktIO. Attempts to receive packets on a PktIO whose +in_mode+ is >> DISABLED >> > +return no packets while packets sent to a PktIO whose +out_mode+ is >> DISABLED >> > +are discarded. >> > + >> > +==== Direct I/O Modes >> > +DIRECT I/O is the default mode for PktIO objects. It is designed to >> support >> > +poll-based packet processing, which is often found in legacy >> applications >> > +being ported to ODP, and can also be a preferred mode for some types of >> > +packet processing. By supporting poll-based I/O processing, ODP >> provides >> > +maximum flexibility to the data plane application writer. >> > + >> > +===== Direct RX Processing >> > +The processing of DIRECT input is shown below: >> > + >> > +.PktIO DIRECT Mode Receive Processing >> > +image::../images/pktin_direct_recv.svg[align="center"] >> > + >> >> From this picture, I understand that a each rx pktio queue is "associated" >> to a RX thread at odp_pktin_queue_config time. however, it does not feel >> like >> that at all in the rest of the document: odp_pktin_recv() take a queue >> handle >> as parameter, hence allowing any ODP thread to read packets from any queue >> at any time... (the "operation mode", -not really defined for RX- also >> implies >> that such an association does not exist, at least in unsafe mode) >> > > That is correct. The intent is to allow applications to have a one-to-one > map between threads and RX queues, but ODP does not constrain applications > to follow such a design. odp_pktin_queue_config() simply says how many RX > queues to create/associated with the PktIO. The discipline of how threads > use these queues is up to the application. I can add another paragraph > here if that isn't clear in context. > My understanding is that ODP does not impose any queue-thread relationship, but the application will probably want to have this queue-thread relationship. This relationship is actually visible at 2 places as I understand it: When reading packet from the queue (the queue ID is passed) and when selecting the safe/unsafe mode (choosing safe meaning that One queue will be read by a single thread). I don't see where the odp_pktin_queue_config() function is involved at all in the relationship (appart for configuring N queues, but no relation is made there right?). >> > +In DIRECT mode, received packets are stored in one or more special >> PktIO queues >> > +of type *odp_pktin_queue_t* and are retrieved by threads calling the >> > ++odp_pktin_recv()+ API. >> > + >> > +Once opened, setting up a DIRECT mode PktIO is performed by the >> > ++odp_pktin_queue_config()+ API. >> > +[source,c] >> > +----- >> > +/** >> > + * Configure packet input queues >> > + * >> > + * Setup a number of packet input queues and configure those. The >> maximum number >> > + * of queues is platform dependent and can be queried with >> > + * odp_pktio_capability(). Use odp_pktin_queue_param_init() to >> initialize >> > + * parameters into their default values. Default values are also used >> when >> > + * 'param' pointer is NULL. >> > + * >> > + * Queue handles for input queues can be requested with >> odp_pktin_queue() or >> > + * odp_pktin_event_queue() after this call. All requested queues are >> setup on >> > + * success, no queues are setup on failure. Each call reconfigures >> input queues >> > + * and may invalidate all previous queue handles. >> > + * >> > + * @param pktio Packet IO handle >> > + * @param param Packet input queue configuration parameters. Uses >> defaults >> > + * when NULL. >> > + * >> > + * @retval 0 on success >> > + * @retval <0 on failure >> > + * >> > + * @see odp_pktio_capability(), odp_pktin_queue(), >> odp_pktin_event_queue() >> > + */ >> > +int odp_pktin_queue_config(odp_pktio_t pktio, >> > + const odp_pktin_queue_param_t *param); >> > +----- >> > +The second argument to this call is the *odp_pktin_queue_param_t* >> > +[source,c] >> > +----- >> > +/** >> > + * Packet input queue parameters >> > + */ >> > +typedef struct odp_pktin_queue_param_t { >> > + /** Operation mode >> > + * >> > + * The default value is ODP_PKTIO_OP_MT. Application may enable >> > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE >> when >> > + * applicable. */ >> > + odp_pktio_op_mode_t op_mode; >> >> Has the "operation mode" been defined anywhere? >> not sure what it is. neither do I >> know what ODP_PKTIO_OP_MT means at this stage of reading... >> This is defined only when talking about TX, later on... >> It should be defined here (and possibly reminded for TX) >> > > Doxygen comments again--not part of this patch series > > >> >> > + >> > + /** Enable flow hashing >> > + * 0: Do not hash flows >> > + * 1: Hash flows to input queues */ >> > + odp_bool_t hash_enable; >> > + >> > + /** Protocol field selection for hashing. Multiple protocols can >> be >> > + * selected. */ >> > + odp_pktin_hash_proto_t hash_proto; >> > + >> > + /** Number of input queues to be created. More than one input >> queue >> > + * require input hashing or classifier setup. Hash_proto is >> ignored >> > + * when hash_enable is zero or num_queues is one. This value >> must be >> > + * between 1 and interface capability. Queue type is defined by >> the >> > + * input mode. The default value is 1. */ >> > + unsigned num_queues; >> > + >> > + /** Queue parameters for creating input queues in >> ODP_PKTIN_MODE_QUEUE >> > + * or ODP_PKTIN_MODE_SCHED modes. Scheduler parameters are >> considered >> > + * only in ODP_PKTIN_MODE_SCHED mode. */ >> > + odp_queue_param_t queue_param; >> > + >> > +} odp_pktin_queue_param_t; >> > +----- >> > +Note that the *queue_param* field of this struct is ignored in DIRECT >> mode. >> > +The purpose of +odp_pktin_queue_config()+ is to specify the number of >> PktIn >> > +queues to be created and to set their attributes. >> > + >> > +===== Hash Processing >> > +Another feature of DIRECT mode input is the provision of a *hash* >> function used >> > +to distribute incoming packets among the PktIO's PktIn queues. If the >> > ++hash_enable+ field of the *odp_pktin_queue_param_t* is 1, >> > +then the +hash_proto+ field is used to specify which field(s) of >> incoming >> > +packets should be used as input to an implementation-defined packet >> > +distribution hash function. >> > +[source,c] >> > +----- >> > +/** >> > + * Packet input hash protocols >> > + * >> > + * The list of protocol header field combinations, which are included >> into >> > + * packet input hash calculation. >> > + */ >> > +typedef union odp_pktin_hash_proto_t { >> > + /** Protocol header fields for hashing */ >> > + struct { >> > + /** IPv4 addresses and UDP port numbers */ >> > + uint32_t ipv4_udp : 1; >> > + /** IPv4 addresses and TCP port numbers */ >> > + uint32_t ipv4_tcp : 1; >> > + /** IPv4 addresses */ >> > + uint32_t ipv4 : 1; >> > + /** IPv6 addresses and UDP port numbers */ >> > + uint32_t ipv6_udp : 1; >> > + /** IPv6 addresses and TCP port numbers */ >> > + uint32_t ipv6_tcp : 1; >> > + /** IPv6 addresses */ >> > + uint32_t ipv6 : 1; >> > + } proto; >> > + >> > + /** All bits of the bit field structure */ >> > + uint32_t all_bits; >> > +} odp_pktin_hash_proto_t; >> > +----- >> > +Note that the hash function used in PktIO poll mode operation is >> intended to >> > +provide simple packet distribution among multiple PktIn queues >> associated with >> > +the PktIO. It does not have the sophistication of the *ODP >> Classifier*, however >> > +it also does not incur the setup requirements of pattern matching >> rules, >> > +making it a simpler choice for less sophisticated applications. Note >> that >> > +ODP does not specify how the hash is to be performed. That is left to >> each >> > +implementation. The hash only specifies which input packet fields are >> of >> > +interest to the application and should be considered by the hash >> function in >> > +deciding how to distribute packets among PktIn queues. The only >> expectation >> > +is that packets that have the same hash values should all be mapped to >> the >> > +same PktIn queue. >> > + >> > +===== PktIn Queues >> > +A *PktIn Queue* is a special type of queue that is used internally by >> PktIOs >> > +operating in DIRECT mode. Applications cannot perform enqueues to >> these queues, >> > +however they may obtain references to them via the +odp_pktin_queue()+ >> API >> > +[source,c] >> > +----- >> > +/** >> > + * Direct packet input queues >> > + * >> > + * Returns the number of input queues configured for the interface in >> > + * ODP_PKTIN_MODE_DIRECT mode. Outputs up to 'num' queue handles when >> the >> > + * 'queues' array pointer is not NULL. If return value is larger than >> 'num', >> > + * there are more queues than the function was allowed to output. If >> return >> > + * value (N) is less than 'num', only queues[0 ... N-1] have been >> written. >> > + * >> > + * Packets from these queues are received with odp_pktin_recv(). >> > + * >> > + * @param pktio Packet IO handle >> > + * @param[out] queues Points to an array of queue handles for output >> > + * @param num Maximum number of queue handles to output >> > + * >> > + * @return Number of packet input queues >> > + * @retval <0 on failure >> > + */ >> > +int odp_pktin_queue(odp_pktio_t pktio, odp_pktin_queue_t queues[], int >> num); >> > +----- >> > +Once configured, prior to receiving packets the PktIO must be placed >> into the >> > +*Ready* state via a call to +odp_pktio_start() >> > +[source,c] >> > +----- >> > +/** >> > + * Start packet receive and transmit >> > + * >> > + * Activate packet receive and transmit on a previously opened or >> stopped >> > + * interface. The interface can be stopped with a call to >> odp_pktio_stop(). >> > + * >> > + * @param pktio Packet IO handle >> > + * >> > + * @retval 0 on success >> > + * @retval <0 on failure >> > + * >> > + * @see odp_pktio_open(), odp_pktio_stop() >> > + */ >> > +int odp_pktio_start(odp_pktio_t pktio); >> > +----- >> > +Once started, the PktIn queue handles are used as arguments to >> > ++odp_pktin_recv()+ to receive packets from the PktIO. >> > +[source,c] >> > +----- >> > +/** >> > + * Receive packets directly from an interface input queue >> > + * >> > + * Receives up to 'num' packets from the pktio interface input queue. >> When >> > + * input queue parameter 'op_mode' has been set to >> ODP_PKTIO_OP_MT_UNSAFE, >> > + * the operation is optimized for single thread operation per queue >> and the same >> > + * queue must not be accessed simultaneously from multiple threads. >> > + * >> > + * @param queue Packet input queue handle for receiving >> packets >> > + * @param[out] packets[] Packet handle array for output of received >> packets >> > + * @param num Maximum number of packets to receive >> > + * >> > + * @return Number of packets received >> > + * @retval <0 on failure >> > + * >> > + * @see odp_pktin_queue() >> > + */ >> > +int odp_pktin_recv(odp_pktin_queue_t queue, odp_packet_t packets[], >> int num); >> >> This means that any thread can read from any queue at any time, right? >> (does not match my understanding of figure 13) >> > > That is correct. As noted above, ODP just provides the tools. > Applications must use these tools intelligently. > > >> >> > +----- >> > + >> > +===== Direct TX Processing >> > +A PktIO operating in DIRECT mode performs TX processing as shown here: >> > + >> > +.PktIO DIRECT Mode Transmit Processing >> > +image::../images/pktout_direct_send.svg[align="center"] >> > + >> > +Direct TX processing operates similarly to Direct RX processing. >> Following >> > +open, the +odp_pktout_queue_config()+ API is used to create and >> configure >> > +one or more *PktOut queues* to be used to support packet transmission >> by >> > +this PktIO >> > +[source,c] >> > +----- >> > +/** >> > + * Configure packet output queues >> > + * >> > + * Setup a number of packet output queues and configure those. The >> maximum >> > + * number of queues is platform dependent and can be queried with >> > + * odp_pktio_capability(). Use odp_pktout_queue_param_init() to >> initialize >> > + * parameters into their default values. Default values are also used >> when >> > + * 'param' pointer is NULL. >> > + * >> > + * Queue handles for output queues can be requested with >> odp_pktout_queue() or >> > + * odp_pktout_event_queue() after this call. All requested queues are >> setup on >> > + * success, no queues are setup on failure. Each call reconfigures >> output queues >> > + * and may invalidate all previous queue handles. >> > + * >> > + * @param pktio Packet IO handle >> > + * @param param Packet output queue configuration parameters. Uses >> defaults >> > + * when NULL. >> > + * >> > + * @retval 0 on success >> > + * @retval <0 on failure >> > + * >> > + * @see odp_pktio_capability(), odp_pktout_queue(), >> odp_pktout_event_queue() >> > + */ >> > +int odp_pktout_queue_config(odp_pktio_t pktio, >> > + const odp_pktout_queue_param_t *param); >> > +----- >> > +As with +odp_pktin_queue_config()+, the configuration of PktOut queues >> > +involves the use of a parameter struct: >> > +[source,c] >> > +----- >> > +/** >> > + * Packet output queue parameters >> > + * >> > + * These parameters are used in ODP_PKTOUT_MODE_DIRECT and >> > + * ODP_PKTOUT_MODE_QUEUE modes. >> > + */ >> > +typedef struct odp_pktout_queue_param_t { >> > + /** Operation mode >> > + * >> > + * The default value is ODP_PKTIO_OP_MT. Application may enable >> > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE >> when >> > + * applicable. */ >> > + odp_pktio_op_mode_t op_mode; >> > + >> > + /** Number of output queues to be created. The value must be >> between >> > + * 1 and interface capability. The default value is 1. */ >> > + unsigned num_queues; >> > + >> > +} odp_pktout_queue_param_t; >> > +----- >> > +As with direct input, direct output uses one or more special output >> queues >> > +of type *odp_pktout_queue_t* that area created and configured by this >> call. >> >> typo: "area" / "are" >> >> > Good catch. Thanks. > > >> > + >> > +As with PktIn queues, the handles for these created PktOut queues may >> be >> > +retrieved by the +odp_pktout_queue()+ API: >> > +[source,c] >> > +----- >> > +/** >> > + * Direct packet output queues >> > + * >> > + * Returns the number of output queues configured for the interface in >> > + * ODP_PKTOUT_MODE_DIRECT mode. Outputs up to 'num' queue handles when >> the >> > + * 'queues' array pointer is not NULL. If return value is larger than >> 'num', >> > + * there are more queues than the function was allowed to output. If >> return >> > + * value (N) is less than 'num', only queues[0 ... N-1] have been >> written. >> > + * >> > + * Packets are sent to these queues with odp_pktout_send(). >> > + * >> > + * @param pktio Packet IO handle >> > + * @param[out] queues Points to an array of queue handles for output >> > + * @param num Maximum number of queue handles to output >> > + * >> > + * @return Number of packet output queues >> > + * @retval <0 on failure >> > + */ >> > +int odp_pktout_queue(odp_pktio_t pktio, odp_pktout_queue_t queues[], >> int num); >> > +----- >> > +Once the PktIO has been configured for output and started via >> > ++odp_pktio_start()+, packets may be transmitted to the PktIO by calling >> > +_odp_pktout_send()+: >> >> "+" missing before _odp_pktout_send()+. (The + you see is from the patch!) >> > > Thank you. > > >> >> > +[source,c] >> > +----- >> > +/** >> > + * Send packets directly to an interface output queue >> > + * >> > + * Sends out a number of packets to the interface output queue. When >> > + * output queue parameter 'op_mode' has been set to >> ODP_PKTIO_OP_MT_UNSAFE, >> > + * the operation is optimized for single thread operation per queue >> and the same >> > + * queue must not be accessed simultaneously from multiple threads. >> >> The above should be stated when discussing RX! >> > > Noted. v3 will add additional detail to the narrative. > > >> >> > + * >> > + * A successful call returns the actual number of packets sent. If >> return value >> > + * is less than 'num', the remaining packets at the end of packets[] >> array >> > + * are not consumed, and the caller has to take care of them. >> > + * >> > + * @param queue Packet output queue handle for sending packets >> > + * @param packets[] Array of packets to send >> > + * @param num Number of packets to send >> > + * >> > + * @return Number of packets sent >> > + * @retval <0 on failure >> > + */ >> > +int odp_pktout_send(odp_pktout_queue_t queue, odp_packet_t packets[], >> int num); >> > +----- >> > +Note that the argument to this call specifies the PktOut queue to that >> the >> >> is "to that" english? (was expecting "to which") here. If it is, ignore >> this >> comment! >> > > "to" is an extraneous word here. Good catch. > > >> >> > +packet is to be added to rather than the PktIO itself. This permits >> multiple >> > +threads (presumably operating on different cores) a more efficient >> means of >> > +separating I/O processing destined for the same interface. >> > + >> > +==== Queued I/O Modes >> > +To provide additional flexibility when operating in poll mode, PktIOs >> may also >> > +be opened in QUEUE Mode. The difference between DIRECT and QUEUE mode >> is that >> > +QUEUE mode uses standard ODP event queues to service packets. >> > + >> > +===== Queue RX Processing >> > +The processing for QUEUE input processing is shown below: >> > + >> > +.PktIO QUEUE Mode Receive Processing >> > +image::../images/pktin_queue_recv.svg[align="center"] >> >> same comment: this figure seems to imply that one can create an >> association >> between the queues and the threads at odp_pktin_qyeye_config time... >> > > No, it's showing how the created queues can be used most efficiently. > I'll expand the comments around this in v3. > > >> >> > + >> > +In QUEUE mode, received packets are stored in one or more standard ODP >> queues. >> > +The difference is that these queues are not created directly by the >> > +application. Instead, they are created in response to an >> > ++odp_pktin_queue_config()+ call. >> > + >> > +As with DIRECT mode, the +odp_pktin_queue_param_t+ specified to this >> call >> > +indicates whether an input hash should be used and if so which >> field(s) of >> > +the packet should be considered as input to the has function. >> > + >> > +The main difference between DIRECT and QUEUE RX processing is that >> because >> > +the PktIO uses standard ODP event queues, other parts of the >> application can >> > +use +odp_queue_enq()+ API calls to enqueue packets to these queues for >> > +"RX" processing in addition to those originating from the PktIO >> interface >> > +itself. To obtain the handles of these input queues, the >> > ++odp_pktin_event_queue()+ API is used: >> > +[source,c] >> > +----- >> > +/** >> > + * Event queues for packet input >> > + * >> > + * Returns the number of input queues configured for the interface in >> > + * ODP_PKTIN_MODE_QUEUE and ODP_PKTIN_MODE_SCHED modes. Outputs up to >> 'num' >> > + * queue handles when the 'queues' array pointer is not NULL. If >> return value is >> > + * larger than 'num', there are more queues than the function was >> allowed to >> > + * output. If return value (N) is less than 'num', only queues[0 ... >> N-1] have >> > + * been written. >> > + * >> > + * Packets (and other events) from these queues are received with >> > + * odp_queue_deq(), odp_schedule(), etc calls. >> > + * >> > + * @param pktio Packet IO handle >> > + * @param[out] queues Points to an array of queue handles for output >> > + * @param num Maximum number of queue handles to output >> > + * >> > + * @return Number of packet input queues >> > + * @retval <0 on failure >> > + */ >> > +int odp_pktin_event_queue(odp_pktio_t pktio, odp_queue_t queues[], int >> num); >> > +----- >> > +Similarly, threads receive packets from PktIOs operating in QUEUE mode >> by >> > +making standard +odp_queue_deq()+ calls to one of the event queues >> associated >> > +with the PktIO. >> > + >> > +===== Queue TX Processing >> > +Transmit processing for PktIOs operating in QUEUE mode is shown below: >> > + >> > +.PktIO QUEUE Mode Transmit Processing >> > +image::../images/pktout_queue_send.svg[align="center] >> > + >> > +For TX processing QUEUE mode behaves similar to DIRECT mode except that >> > +output queues are regular ODP event queues that receive packets via >> > ++odp_queue_enq()+ calls rather than special PktOut queues that use >> > ++odp_pktout_send()+. Again, these queues are created via a call to >> > ++odp_pktout_queue_config()+ following +odp_pktio_open()+. >> > + >> > +The main reason for selecting QUEUE mode for output is flexibility. If >> an >> > +application is designed to use a _pipeline model_ where packets flow >> through >> > +a series of processing stages via queues, then having the PktIO in >> QUEUE >> > +mode means that the application can always use the same enq APIs to >> pass packets >> > +from one stage to the next, including the final transmit output stage. >> > + >> > +==== Scheduled I/O Modes >> > +The final PktIO mode supported integrates RX and TX processing with >> the ODP >> > +_event model_. For RX processing this involves the use of the >> *Scheduler* >> > +while for TX processing this involves the use of the *Traffic Manager*. >> > + >> > +Scheduled RX Processing is further divided based on whether or not the >> > +Classifier is used. >> > + >> > +===== Scheduled RX Processing >> > +When a PktIO is opened with +ODP_PKTIN_MODE_SCHED+, it indicates that >> the >> > +input queues created by a subsequent +odp_pktin_queue_config()+ call >> are to >> > +be used as input to the *ODP Scheduler*. >> > + >> > +.PktIO SCHED Mode Receive Processing >> > +image::../images/pktin_sched_recv.svg[align="center'] >> > + >> > +For basic use, SCHED mode simply associates the PktIO input event >> queues >> > +created by +odp_pktin_queue_config()+ with the scheduler. Hashing may >> still be >> > +employed to distribute input packets among multiple input queues. >> However >> > +instead of these being plain queues they are scheduled queues and have >> > +associated scheduling attributes like priority, scheduler group, and >> > +synchronization mode (parallel, atomic, ordered). SCHED mode thus >> provides >> > +both packet distribution (via the optional hash) as well as >> scalability via >> > +the ODP event model. >> > + >> > +In its fullest form, PktIOs operating in SCHED mode use the *ODP >> Classifier* >> > +to permit fine-grained flow separation on *Class of Service (CoS)* >> boundaries. >> > + >> > +.PktIO SCHED Mode Receive Processing with Classification >> > +image::../images/pktin_sched_cls.svg[align="center"] >> >> The diagram and the text confuses me a bit, here: my understanding was >> that: >> 1)the classifier associates a cos to each incoming packet using a set of >> PMRs. >> 2)Each cos maps to a pair {RXpool, RXqueue}. This Mapping is settled >> during the >> classifier setup. Packets with a given cos are stored on odp_packets >> allocated >> from the corresponding pool, and placed in the corresponding queue. >> > > That is correct. What's unclear about that here? The point is that the > queues shown here may either be those that the application creates via it's > own odp_queue_create() calls or those created by odp_pktin_queue_config(). > If the latter, the application will use odp_pktin_queue() to retrieve their > handle to be passed when it creates the CoS it's to be associated with. > Remember, multiple CoSes can add packets to the same queue--this is not a > 1-to-1 association here. > Maybe the figure wants to show too much. I don't really see what's going on there. But maybe it is part of the classifier. Anyway. see U at BKK! /Christophe. > > >> >> > + >> > +In this mode of operation, the hash function of >> +odp_pktin_queue_config()+ is >> > +not used. Instead, the event queues created by this call, as well as >> any >> > +additional event queues created via separate +odp_queue_create()+ >> calls are >> > +associated with classes of service via +odp_cls_cos_create()+ calls. >> > + >> > +===== Scheduled TX Processing >> > +Scheduled transmit processing is performed via the *ODP Traffic >> Manager* and >> > +is requested when a PktIO is opened with an +out_mode+ of >> +ODP_PKTOUT_MODE_TM+. >> > + >> > +For TX processing via the Traffic Manager, applications use the >> +odp_tm_enq()+ >> > +API: >> > +[source,c] >> > +----- >> > +/** The odp_tm_enq() function is used to add packets to a given TM >> system. >> > + * Note that the System Metadata associated with the pkt needed by the >> TM >> > + * system is (a) a drop_eligible bit, (b) a two bit "pkt_color", (c) a >> 16-bit >> > + * pkt_len, and MAYBE? (d) a signed 8-bit shaper_len_adjust. >> > + * >> > + * If there is a non-zero shaper_len_adjust, then it is added to the >> pkt_len >> > + * after any non-zero shaper_len_adjust that is part of the shaper >> profile. >> > + * >> > + * The pkt_color bits are a result of some earlier >> Metering/Marking/Policing >> > + * processing (typically ingress based), and should not be confused >> with the >> > + * shaper_color produced from the TM shaper entities within the >> tm_inputs and >> > + * tm_nodes. >> > + * >> > + * @param[in] tm_queue Specifies the tm_queue (and indirectly the TM >> system). >> > + * @param[in] pkt Handle to a packet. >> > + * @return Returns 0 upon success, < 0 upon failure. One >> of the >> > + * more common failure reasons is WRED dropage. >> > + */ >> > +int odp_tm_enq(odp_tm_queue_t tm_queue, odp_packet_t pkt); >> > +----- >> > +See the *Traffic Manager* section of this document for full >> information about >> > +Traffic Manager configuration and operation. >> > diff --git a/doc/users-guide/users-guide.adoc >> b/doc/users-guide/users-guide.adoc >> > index d476225..ea24eaf 100644 >> > --- a/doc/users-guide/users-guide.adoc >> > +++ b/doc/users-guide/users-guide.adoc >> > @@ -1023,6 +1023,8 @@ a valid UDP packet. Applications are expected to >> exercise appropriate care >> > when changing packet metadata to ensure that the resulting metadata >> changes >> > reflect the actual changed packet structure that the application has >> made. >> > >> > +include::users-guide-pktio.adoc[] >> > + >> > == Cryptographic services >> > >> > ODP provides support for cryptographic operations required by various >> security >> > -- >> > 2.5.0 >> > >> > _______________________________________________ >> > lng-odp mailing list >> > lng-odp@lists.linaro.org >> > https://lists.linaro.org/mailman/listinfo/lng-odp >> >> Interresting reading, and nice to see the user doc growing! Thanks! >> > > Thanks for the careful review and questions. > > >> >> Christophe >> > >
I've posted a v3 that incorporates your review comments. We can discuss with Petri any proposed changes to the doxygen on the APIs while at BKK. Thanks. On Thu, Mar 3, 2016 at 11:44 AM, Christophe Milard < christophe.milard@linaro.org> wrote: > I know the comments are copied from the source, but thy are just copied > (not referenced), right? > I mean, if it does not get fixed here, there will be 2 places to fix... > If you have the courage, maybe the source comment can be fixed in a earlyer > patch. > These comments are relatively minor details. Fix what you can: This text > is much better that the emptiness that was before. > > On 3 March 2016 at 17:12, Bill Fischofer <bill.fischofer@linaro.org> > wrote: > >> >> >> On Thu, Mar 3, 2016 at 4:00 AM, Christophe Milard < >> christophe.milard@linaro.org> wrote: >> >>> On 2016-03-02 04:35, Bill Fischofer wrote: >>> > Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> >>> > --- >>> > doc/users-guide/users-guide-pktio.adoc | 605 >>> +++++++++++++++++++++++++++++++++ >>> > doc/users-guide/users-guide.adoc | 2 + >>> > 2 files changed, 607 insertions(+) >>> > create mode 100644 doc/users-guide/users-guide-pktio.adoc >>> > >>> > diff --git a/doc/users-guide/users-guide-pktio.adoc >>> b/doc/users-guide/users-guide-pktio.adoc >>> > new file mode 100644 >>> > index 0000000..29fb6a9 >>> > --- /dev/null >>> > +++ b/doc/users-guide/users-guide-pktio.adoc >>> > @@ -0,0 +1,605 @@ >>> > +== PktIO Processing >>> > +Before packets can be manipulated they typically need to be >>> _received_ and >>> > +after they are manipulated they need to be _transmitted_. The ODP >>> abstraction >>> > +that captures these operations is the *Packet I/O (PktIO)*. >>> > +PktIOs are represented by handles of type *odp_pktio_t* and >>> > +represent a logical I/O interface that is mapped in an >>> implementation-defined >>> > +manner to an underlying integrated I/O adapter or NIC. >>> > + >>> > +PktIO objects are manipulated through various state transitions via >>> > ++odp_pktio_xxx()+ API calls as shown below: >>> > + >>> > +.ODP PktIO Finite State Machine >>> > +image::../images/pktio_fsm.svg[align="center"] >>> > + >>> > +PktIOs begin in the *Unallocated* state. From here a call >>> +odp_pktio_open()+ >>> > +is used to create an *odp_pktio_t* handle that is used in all >>> subsequent calls >>> > +to manipulate the object. This call puts the PktIO into the >>> *Unconfigured* >>> > +state. To become operational, a PktIO must first be >>> > +*configured* for Input, Output, or both Input and Output via the >>> > ++odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ APIs, >>> and then >>> > +*started* via the +odp_pktio_start()+ to make it *Ready*. >>> > + >>> > +Following the completion of I/O processing, the +odp_pktio_stop()+ >>> API returns >>> > +the PktIO to the *Configured* state. From here it may be >>> *Reconfigured* via >>> > +additional +odp_pktin_queue_config()+ and/or >>> +odp_pktout_queue_config()+ calls, >>> > +or *Closed* via the +odp_pktio_close()+ API to return the PktIO to the >>> > +*Unallocated* state. >>> > + >>> > +=== PktIO Allocation >>> > +PktIO objects begin life by being _opened_ via the call: >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Open a packet IO interface >>> > + * >>> > + * An ODP program can open a single packet IO interface per device, >>> attempts >>> >>> what is a device here? a (possibly physical) interface? if we have a NIC >>> board with 2 >>> physical inerfaces, how many pktio can on open on it? >>> maybe: "An ODP program can open a single packet IO per interface"...? >>> >> >> You're not commenting on this document, but on the >> include/odp/api/spec/packet_io.h file that I'm just excerpting snippets >> from here. We can improve that commentary, but that would be a separate >> patch. I'd prefer to keep these two separate at this point and make any >> doxygen changes as a separate patch series. >> > > yes I know. see my comment on the top. > > >> >>> > + * to open an already open device will fail, returning >>> ODP_PKTIO_INVALID with >>> > + * errno set. Use odp_pktio_lookup() to obtain a handle to an already >>> open >>> >>> "opened" instead of open? >>> >> >> Same comment as above. >> >> >>> >>> > + * device. Packet IO parameters provide interface level configuration >>> options. >>> > + * >>> > + * Packet input queue configuration must be setup with >>> > + * odp_pktin_queue_config() before odp_pktio_start() is called. When >>> packet >>> > + * input mode is ODP_PKTIN_MODE_DISABLED, odp_pktin_queue_config() >>> call is >>> > + * optional and will ignore all parameters. >>> > + * >>> > + * Packet output queue configuration must be setup with >>> > + * odp_pktout_queue_config() before odp_pktio_start() is called. When >>> packet >>> > + * output mode is ODP_PKTOUT_MODE_DISABLED or ODP_PKTOUT_MODE_TM, >>> > + * odp_pktout_queue_config() call is optional and will ignore all >>> parameters. >>> > + * >>> >>> Is is not clear at this stage of reading what are the packet input and >>> output modes. >>> I assume that they are parameters to the pktio_open function (in param) >>> as this >>> is the only thing that is done before the configuration, maybe that >>> should >>> be clarified before. (this assumption gets confirmed later on in the doc) >>> Simply adding "Pktio in/output mode will be discussed in the next >>> section" before >>> this code frgament would make sense. >>> >> >> Same comment as above. >> >> >>> >>> > + * Packet receive and transmit on the interface is enabled with a >>> call to >>> > + * odp_pktio_start(). If not specified otherwise, any interface level >>> > + * configuration must not be changed when the interface is active >>> (between start >>> > + * and stop calls). >>> > + * >>> > + * In summary, a typical pktio interface setup sequence is ... >>> > + * * odp_pktio_open() >>> > + * * odp_pktin_queue_config() >>> > + * * odp_pktout_queue_config() >>> > + * * odp_pktio_start() >>> > + * >>> > + * ... and tear down sequence is: >>> > + * * odp_pktio_stop() >>> > + * * odp_pktio_close() >>> > + * >>> > + * @param name Packet IO device name >>> > + * @param pool Default pool from which to allocate storage for >>> packets >>> > + * received over this interface, must be of type >>> ODP_POOL_PACKET >>> > + * @param param Packet IO parameters >>> > + * >>> > + * @return Packet IO handle >>> > + * @retval ODP_PKTIO_INVALID on failure >>> > + * >>> > + * @note The device name "loop" is a reserved name for a loopback >>> device used >>> > + * for testing purposes. >>> > + * >>> > + * @note Packets arriving via this interface assigned to a CoS by the >>> > + * classifier are received into the pool associated with that CoS. >>> This >>> > + * will occur either because this pktio is assigned a default CoS >>> via >>> > + * the odp_pktio_default_cos_set() routine, or because a matching >>> PMR >>> > + * assigned the packet to a specific CoS. The default pool >>> specified >>> > + * here is applicable only for those packets that are not assigned >>> to a >>> > + * more specific CoS. >>> > + * >>> > + * @see odp_pktio_start(), odp_pktio_stop(), odp_pktio_close() >>> > + */ >>> > +odp_pktio_t odp_pktio_open(const char *name, odp_pool_t pool, >>> > + const odp_pktio_param_t *param); >>> > +----- >>> > ++odp_pktio_open()+ takes three arguments: a *name*, which is an >>> > +implementation-defined string that identifies the logical interface >>> to be >>> > +opened, a *pool* that identifies the ODP pool that storage for >>> received >>> > +packets should be allocated from, and a *param* structure that >>> specifies >>> > +I/O options to be associated with this PktIO instance. >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Packet IO parameters >>> > + * >>> > + * In minimum, user must select input and output modes. Use 0 for >>> defaults. >>> > + * Initialize entire struct with zero to maintain API compatibility. >>> > + */ >>> > +typedef struct odp_pktio_param_t { >>> > + /** Packet input mode >>> > + * >>> > + * The default value is ODP_PKTIN_MODE_DIRECT. */ >>> > + odp_pktin_mode_t in_mode; >>> > + /** Packet output mode >>> > + * >>> > + * The default value is ODP_PKTOUT_MODE_DIRECT. */ >>> > + odp_pktout_mode_t out_mode; >>> > +} odp_pktio_param_t; >>> > +----- >>> > +ODP defines *loop* as a reserved name to indicate that this PktIO >>> represents >>> >>> I think "loop" should be quoted as it is a string. >>> >> >> It's bolded for that purpose, but I have no objection to adding quotes. >> Of course there will always be those who see and try to type the quotes and >> then wonder why it didn't work. :) >> >>> >>> > +a loopback interface. Loopback interfaces are useful as a means of >>> recycling >>> > +packets back for reclassification after decryption or decapsulation, >>> as well as >>> > +for diagnostic or testing purposes. The *pool* specifies the default >>> pool to >>> >>> "Loopback interfaces are useful as a means of recycling packets back for >>> reclassification after decryption or decapsulation". I don't understand >>> what you >>> mean by that... should I? ;-) >>> >> >> These are two common reasons why you'd use a loopback interface in >> non-testing mode. Until you decrypt an IPSEC packet you really have no >> idea what's really in it. Similar considerations apply to tunneled packets >> that need to be decapsulated before the "real" packet is visible. >> > > I understand that, but I cannot see how a loopback interface would help > here. You can explain at BKK ! > >> >> >>> >>> > +use for packet allocation if not overridden by the classifier due to a >>> > +specific or default Class-of-Service (CoS) match on the packet. The >>> *param* >>> > +struct, in turn, specifies the input and output *modes* of the PktIO. >>> > + >>> > +=== PktIO Input and Output Modes >>> > +PktIO objects support four different Input and Output modes, that may >>> be >>> > +specified independently at *open* time. >>> > + >>> > +.PktIO Input Modes >>> > +* +ODP_PKTIN_MODE_DIRECT+ >>> > +* +ODP_PKTIN_MODE_QUEUE+ >>> > +* +ODP_OKTIN_MODE_SCHED+ >>> > +* +ODP_PKTIN_MODE_DISABLED+ >>> > + >>> > +.PktIO Output Modes >>> > +* +ODP_PKTOUT_MODE_DIRECT+ >>> > +* +ODP_PKTOUT_MODE_QUEUE+ >>> > +* +ODP_PKTOUT_MODE_TM+ >>> > +* +ODP_PKTOUT_MODE_DISABLED+ >>> > + >>> > +The DISABLED modes indicate that either input or output is prohibited >>> on this >>> > +PktIO. Attempts to receive packets on a PktIO whose +in_mode+ is >>> DISABLED >>> > +return no packets while packets sent to a PktIO whose +out_mode+ is >>> DISABLED >>> > +are discarded. >>> > + >>> > +==== Direct I/O Modes >>> > +DIRECT I/O is the default mode for PktIO objects. It is designed to >>> support >>> > +poll-based packet processing, which is often found in legacy >>> applications >>> > +being ported to ODP, and can also be a preferred mode for some types >>> of >>> > +packet processing. By supporting poll-based I/O processing, ODP >>> provides >>> > +maximum flexibility to the data plane application writer. >>> > + >>> > +===== Direct RX Processing >>> > +The processing of DIRECT input is shown below: >>> > + >>> > +.PktIO DIRECT Mode Receive Processing >>> > +image::../images/pktin_direct_recv.svg[align="center"] >>> > + >>> >>> From this picture, I understand that a each rx pktio queue is >>> "associated" >>> to a RX thread at odp_pktin_queue_config time. however, it does not feel >>> like >>> that at all in the rest of the document: odp_pktin_recv() take a queue >>> handle >>> as parameter, hence allowing any ODP thread to read packets from any >>> queue >>> at any time... (the "operation mode", -not really defined for RX- also >>> implies >>> that such an association does not exist, at least in unsafe mode) >>> >> >> That is correct. The intent is to allow applications to have a >> one-to-one map between threads and RX queues, but ODP does not constrain >> applications to follow such a design. odp_pktin_queue_config() simply says >> how many RX queues to create/associated with the PktIO. The discipline of >> how threads use these queues is up to the application. I can add another >> paragraph here if that isn't clear in context. >> > > My understanding is that ODP does not impose any queue-thread > relationship, but the application will probably want to have this > queue-thread relationship. This relationship is actually visible at 2 > places as I understand it: When reading packet from the queue (the queue ID > is passed) and when selecting the safe/unsafe mode (choosing safe meaning > that One queue will be read by a single thread). I don't see where the > odp_pktin_queue_config() function is involved at all in the relationship > (appart for configuring N queues, but no relation is made there right?). > > >>> > +In DIRECT mode, received packets are stored in one or more special >>> PktIO queues >>> > +of type *odp_pktin_queue_t* and are retrieved by threads calling the >>> > ++odp_pktin_recv()+ API. >>> > + >>> > +Once opened, setting up a DIRECT mode PktIO is performed by the >>> > ++odp_pktin_queue_config()+ API. >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Configure packet input queues >>> > + * >>> > + * Setup a number of packet input queues and configure those. The >>> maximum number >>> > + * of queues is platform dependent and can be queried with >>> > + * odp_pktio_capability(). Use odp_pktin_queue_param_init() to >>> initialize >>> > + * parameters into their default values. Default values are also used >>> when >>> > + * 'param' pointer is NULL. >>> > + * >>> > + * Queue handles for input queues can be requested with >>> odp_pktin_queue() or >>> > + * odp_pktin_event_queue() after this call. All requested queues are >>> setup on >>> > + * success, no queues are setup on failure. Each call reconfigures >>> input queues >>> > + * and may invalidate all previous queue handles. >>> > + * >>> > + * @param pktio Packet IO handle >>> > + * @param param Packet input queue configuration parameters. Uses >>> defaults >>> > + * when NULL. >>> > + * >>> > + * @retval 0 on success >>> > + * @retval <0 on failure >>> > + * >>> > + * @see odp_pktio_capability(), odp_pktin_queue(), >>> odp_pktin_event_queue() >>> > + */ >>> > +int odp_pktin_queue_config(odp_pktio_t pktio, >>> > + const odp_pktin_queue_param_t *param); >>> > +----- >>> > +The second argument to this call is the *odp_pktin_queue_param_t* >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Packet input queue parameters >>> > + */ >>> > +typedef struct odp_pktin_queue_param_t { >>> > + /** Operation mode >>> > + * >>> > + * The default value is ODP_PKTIO_OP_MT. Application may enable >>> > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE >>> when >>> > + * applicable. */ >>> > + odp_pktio_op_mode_t op_mode; >>> >>> Has the "operation mode" been defined anywhere? >>> not sure what it is. neither do I >>> know what ODP_PKTIO_OP_MT means at this stage of reading... >>> This is defined only when talking about TX, later on... >>> It should be defined here (and possibly reminded for TX) >>> >> >> Doxygen comments again--not part of this patch series >> >> >>> >>> > + >>> > + /** Enable flow hashing >>> > + * 0: Do not hash flows >>> > + * 1: Hash flows to input queues */ >>> > + odp_bool_t hash_enable; >>> > + >>> > + /** Protocol field selection for hashing. Multiple protocols can >>> be >>> > + * selected. */ >>> > + odp_pktin_hash_proto_t hash_proto; >>> > + >>> > + /** Number of input queues to be created. More than one input >>> queue >>> > + * require input hashing or classifier setup. Hash_proto is >>> ignored >>> > + * when hash_enable is zero or num_queues is one. This value >>> must be >>> > + * between 1 and interface capability. Queue type is defined by >>> the >>> > + * input mode. The default value is 1. */ >>> > + unsigned num_queues; >>> > + >>> > + /** Queue parameters for creating input queues in >>> ODP_PKTIN_MODE_QUEUE >>> > + * or ODP_PKTIN_MODE_SCHED modes. Scheduler parameters are >>> considered >>> > + * only in ODP_PKTIN_MODE_SCHED mode. */ >>> > + odp_queue_param_t queue_param; >>> > + >>> > +} odp_pktin_queue_param_t; >>> > +----- >>> > +Note that the *queue_param* field of this struct is ignored in DIRECT >>> mode. >>> > +The purpose of +odp_pktin_queue_config()+ is to specify the number of >>> PktIn >>> > +queues to be created and to set their attributes. >>> > + >>> > +===== Hash Processing >>> > +Another feature of DIRECT mode input is the provision of a *hash* >>> function used >>> > +to distribute incoming packets among the PktIO's PktIn queues. If the >>> > ++hash_enable+ field of the *odp_pktin_queue_param_t* is 1, >>> > +then the +hash_proto+ field is used to specify which field(s) of >>> incoming >>> > +packets should be used as input to an implementation-defined packet >>> > +distribution hash function. >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Packet input hash protocols >>> > + * >>> > + * The list of protocol header field combinations, which are included >>> into >>> > + * packet input hash calculation. >>> > + */ >>> > +typedef union odp_pktin_hash_proto_t { >>> > + /** Protocol header fields for hashing */ >>> > + struct { >>> > + /** IPv4 addresses and UDP port numbers */ >>> > + uint32_t ipv4_udp : 1; >>> > + /** IPv4 addresses and TCP port numbers */ >>> > + uint32_t ipv4_tcp : 1; >>> > + /** IPv4 addresses */ >>> > + uint32_t ipv4 : 1; >>> > + /** IPv6 addresses and UDP port numbers */ >>> > + uint32_t ipv6_udp : 1; >>> > + /** IPv6 addresses and TCP port numbers */ >>> > + uint32_t ipv6_tcp : 1; >>> > + /** IPv6 addresses */ >>> > + uint32_t ipv6 : 1; >>> > + } proto; >>> > + >>> > + /** All bits of the bit field structure */ >>> > + uint32_t all_bits; >>> > +} odp_pktin_hash_proto_t; >>> > +----- >>> > +Note that the hash function used in PktIO poll mode operation is >>> intended to >>> > +provide simple packet distribution among multiple PktIn queues >>> associated with >>> > +the PktIO. It does not have the sophistication of the *ODP >>> Classifier*, however >>> > +it also does not incur the setup requirements of pattern matching >>> rules, >>> > +making it a simpler choice for less sophisticated applications. Note >>> that >>> > +ODP does not specify how the hash is to be performed. That is left to >>> each >>> > +implementation. The hash only specifies which input packet fields are >>> of >>> > +interest to the application and should be considered by the hash >>> function in >>> > +deciding how to distribute packets among PktIn queues. The only >>> expectation >>> > +is that packets that have the same hash values should all be mapped >>> to the >>> > +same PktIn queue. >>> > + >>> > +===== PktIn Queues >>> > +A *PktIn Queue* is a special type of queue that is used internally by >>> PktIOs >>> > +operating in DIRECT mode. Applications cannot perform enqueues to >>> these queues, >>> > +however they may obtain references to them via the >>> +odp_pktin_queue()+ API >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Direct packet input queues >>> > + * >>> > + * Returns the number of input queues configured for the interface in >>> > + * ODP_PKTIN_MODE_DIRECT mode. Outputs up to 'num' queue handles when >>> the >>> > + * 'queues' array pointer is not NULL. If return value is larger than >>> 'num', >>> > + * there are more queues than the function was allowed to output. If >>> return >>> > + * value (N) is less than 'num', only queues[0 ... N-1] have been >>> written. >>> > + * >>> > + * Packets from these queues are received with odp_pktin_recv(). >>> > + * >>> > + * @param pktio Packet IO handle >>> > + * @param[out] queues Points to an array of queue handles for output >>> > + * @param num Maximum number of queue handles to output >>> > + * >>> > + * @return Number of packet input queues >>> > + * @retval <0 on failure >>> > + */ >>> > +int odp_pktin_queue(odp_pktio_t pktio, odp_pktin_queue_t queues[], >>> int num); >>> > +----- >>> > +Once configured, prior to receiving packets the PktIO must be placed >>> into the >>> > +*Ready* state via a call to +odp_pktio_start() >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Start packet receive and transmit >>> > + * >>> > + * Activate packet receive and transmit on a previously opened or >>> stopped >>> > + * interface. The interface can be stopped with a call to >>> odp_pktio_stop(). >>> > + * >>> > + * @param pktio Packet IO handle >>> > + * >>> > + * @retval 0 on success >>> > + * @retval <0 on failure >>> > + * >>> > + * @see odp_pktio_open(), odp_pktio_stop() >>> > + */ >>> > +int odp_pktio_start(odp_pktio_t pktio); >>> > +----- >>> > +Once started, the PktIn queue handles are used as arguments to >>> > ++odp_pktin_recv()+ to receive packets from the PktIO. >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Receive packets directly from an interface input queue >>> > + * >>> > + * Receives up to 'num' packets from the pktio interface input queue. >>> When >>> > + * input queue parameter 'op_mode' has been set to >>> ODP_PKTIO_OP_MT_UNSAFE, >>> > + * the operation is optimized for single thread operation per queue >>> and the same >>> > + * queue must not be accessed simultaneously from multiple threads. >>> > + * >>> > + * @param queue Packet input queue handle for receiving >>> packets >>> > + * @param[out] packets[] Packet handle array for output of received >>> packets >>> > + * @param num Maximum number of packets to receive >>> > + * >>> > + * @return Number of packets received >>> > + * @retval <0 on failure >>> > + * >>> > + * @see odp_pktin_queue() >>> > + */ >>> > +int odp_pktin_recv(odp_pktin_queue_t queue, odp_packet_t packets[], >>> int num); >>> >>> This means that any thread can read from any queue at any time, right? >>> (does not match my understanding of figure 13) >>> >> >> That is correct. As noted above, ODP just provides the tools. >> Applications must use these tools intelligently. >> >> >>> >>> > +----- >>> > + >>> > +===== Direct TX Processing >>> > +A PktIO operating in DIRECT mode performs TX processing as shown here: >>> > + >>> > +.PktIO DIRECT Mode Transmit Processing >>> > +image::../images/pktout_direct_send.svg[align="center"] >>> > + >>> > +Direct TX processing operates similarly to Direct RX processing. >>> Following >>> > +open, the +odp_pktout_queue_config()+ API is used to create and >>> configure >>> > +one or more *PktOut queues* to be used to support packet transmission >>> by >>> > +this PktIO >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Configure packet output queues >>> > + * >>> > + * Setup a number of packet output queues and configure those. The >>> maximum >>> > + * number of queues is platform dependent and can be queried with >>> > + * odp_pktio_capability(). Use odp_pktout_queue_param_init() to >>> initialize >>> > + * parameters into their default values. Default values are also used >>> when >>> > + * 'param' pointer is NULL. >>> > + * >>> > + * Queue handles for output queues can be requested with >>> odp_pktout_queue() or >>> > + * odp_pktout_event_queue() after this call. All requested queues are >>> setup on >>> > + * success, no queues are setup on failure. Each call reconfigures >>> output queues >>> > + * and may invalidate all previous queue handles. >>> > + * >>> > + * @param pktio Packet IO handle >>> > + * @param param Packet output queue configuration parameters. Uses >>> defaults >>> > + * when NULL. >>> > + * >>> > + * @retval 0 on success >>> > + * @retval <0 on failure >>> > + * >>> > + * @see odp_pktio_capability(), odp_pktout_queue(), >>> odp_pktout_event_queue() >>> > + */ >>> > +int odp_pktout_queue_config(odp_pktio_t pktio, >>> > + const odp_pktout_queue_param_t *param); >>> > +----- >>> > +As with +odp_pktin_queue_config()+, the configuration of PktOut queues >>> > +involves the use of a parameter struct: >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Packet output queue parameters >>> > + * >>> > + * These parameters are used in ODP_PKTOUT_MODE_DIRECT and >>> > + * ODP_PKTOUT_MODE_QUEUE modes. >>> > + */ >>> > +typedef struct odp_pktout_queue_param_t { >>> > + /** Operation mode >>> > + * >>> > + * The default value is ODP_PKTIO_OP_MT. Application may enable >>> > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE >>> when >>> > + * applicable. */ >>> > + odp_pktio_op_mode_t op_mode; >>> > + >>> > + /** Number of output queues to be created. The value must be >>> between >>> > + * 1 and interface capability. The default value is 1. */ >>> > + unsigned num_queues; >>> > + >>> > +} odp_pktout_queue_param_t; >>> > +----- >>> > +As with direct input, direct output uses one or more special output >>> queues >>> > +of type *odp_pktout_queue_t* that area created and configured by this >>> call. >>> >>> typo: "area" / "are" >>> >>> >> Good catch. Thanks. >> >> >>> > + >>> > +As with PktIn queues, the handles for these created PktOut queues may >>> be >>> > +retrieved by the +odp_pktout_queue()+ API: >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Direct packet output queues >>> > + * >>> > + * Returns the number of output queues configured for the interface in >>> > + * ODP_PKTOUT_MODE_DIRECT mode. Outputs up to 'num' queue handles >>> when the >>> > + * 'queues' array pointer is not NULL. If return value is larger than >>> 'num', >>> > + * there are more queues than the function was allowed to output. If >>> return >>> > + * value (N) is less than 'num', only queues[0 ... N-1] have been >>> written. >>> > + * >>> > + * Packets are sent to these queues with odp_pktout_send(). >>> > + * >>> > + * @param pktio Packet IO handle >>> > + * @param[out] queues Points to an array of queue handles for output >>> > + * @param num Maximum number of queue handles to output >>> > + * >>> > + * @return Number of packet output queues >>> > + * @retval <0 on failure >>> > + */ >>> > +int odp_pktout_queue(odp_pktio_t pktio, odp_pktout_queue_t queues[], >>> int num); >>> > +----- >>> > +Once the PktIO has been configured for output and started via >>> > ++odp_pktio_start()+, packets may be transmitted to the PktIO by >>> calling >>> > +_odp_pktout_send()+: >>> >>> "+" missing before _odp_pktout_send()+. (The + you see is from the >>> patch!) >>> >> >> Thank you. >> >> >>> >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Send packets directly to an interface output queue >>> > + * >>> > + * Sends out a number of packets to the interface output queue. When >>> > + * output queue parameter 'op_mode' has been set to >>> ODP_PKTIO_OP_MT_UNSAFE, >>> > + * the operation is optimized for single thread operation per queue >>> and the same >>> > + * queue must not be accessed simultaneously from multiple threads. >>> >>> The above should be stated when discussing RX! >>> >> >> Noted. v3 will add additional detail to the narrative. >> >> >>> >>> > + * >>> > + * A successful call returns the actual number of packets sent. If >>> return value >>> > + * is less than 'num', the remaining packets at the end of packets[] >>> array >>> > + * are not consumed, and the caller has to take care of them. >>> > + * >>> > + * @param queue Packet output queue handle for sending packets >>> > + * @param packets[] Array of packets to send >>> > + * @param num Number of packets to send >>> > + * >>> > + * @return Number of packets sent >>> > + * @retval <0 on failure >>> > + */ >>> > +int odp_pktout_send(odp_pktout_queue_t queue, odp_packet_t packets[], >>> int num); >>> > +----- >>> > +Note that the argument to this call specifies the PktOut queue to >>> that the >>> >>> is "to that" english? (was expecting "to which") here. If it is, ignore >>> this >>> comment! >>> >> >> "to" is an extraneous word here. Good catch. >> >> >>> >>> > +packet is to be added to rather than the PktIO itself. This permits >>> multiple >>> > +threads (presumably operating on different cores) a more efficient >>> means of >>> > +separating I/O processing destined for the same interface. >>> > + >>> > +==== Queued I/O Modes >>> > +To provide additional flexibility when operating in poll mode, PktIOs >>> may also >>> > +be opened in QUEUE Mode. The difference between DIRECT and QUEUE mode >>> is that >>> > +QUEUE mode uses standard ODP event queues to service packets. >>> > + >>> > +===== Queue RX Processing >>> > +The processing for QUEUE input processing is shown below: >>> > + >>> > +.PktIO QUEUE Mode Receive Processing >>> > +image::../images/pktin_queue_recv.svg[align="center"] >>> >>> same comment: this figure seems to imply that one can create an >>> association >>> between the queues and the threads at odp_pktin_qyeye_config time... >>> >> >> No, it's showing how the created queues can be used most efficiently. >> I'll expand the comments around this in v3. >> >> >>> >>> > + >>> > +In QUEUE mode, received packets are stored in one or more standard >>> ODP queues. >>> > +The difference is that these queues are not created directly by the >>> > +application. Instead, they are created in response to an >>> > ++odp_pktin_queue_config()+ call. >>> > + >>> > +As with DIRECT mode, the +odp_pktin_queue_param_t+ specified to this >>> call >>> > +indicates whether an input hash should be used and if so which >>> field(s) of >>> > +the packet should be considered as input to the has function. >>> > + >>> > +The main difference between DIRECT and QUEUE RX processing is that >>> because >>> > +the PktIO uses standard ODP event queues, other parts of the >>> application can >>> > +use +odp_queue_enq()+ API calls to enqueue packets to these queues for >>> > +"RX" processing in addition to those originating from the PktIO >>> interface >>> > +itself. To obtain the handles of these input queues, the >>> > ++odp_pktin_event_queue()+ API is used: >>> > +[source,c] >>> > +----- >>> > +/** >>> > + * Event queues for packet input >>> > + * >>> > + * Returns the number of input queues configured for the interface in >>> > + * ODP_PKTIN_MODE_QUEUE and ODP_PKTIN_MODE_SCHED modes. Outputs up to >>> 'num' >>> > + * queue handles when the 'queues' array pointer is not NULL. If >>> return value is >>> > + * larger than 'num', there are more queues than the function was >>> allowed to >>> > + * output. If return value (N) is less than 'num', only queues[0 ... >>> N-1] have >>> > + * been written. >>> > + * >>> > + * Packets (and other events) from these queues are received with >>> > + * odp_queue_deq(), odp_schedule(), etc calls. >>> > + * >>> > + * @param pktio Packet IO handle >>> > + * @param[out] queues Points to an array of queue handles for output >>> > + * @param num Maximum number of queue handles to output >>> > + * >>> > + * @return Number of packet input queues >>> > + * @retval <0 on failure >>> > + */ >>> > +int odp_pktin_event_queue(odp_pktio_t pktio, odp_queue_t queues[], >>> int num); >>> > +----- >>> > +Similarly, threads receive packets from PktIOs operating in QUEUE >>> mode by >>> > +making standard +odp_queue_deq()+ calls to one of the event queues >>> associated >>> > +with the PktIO. >>> > + >>> > +===== Queue TX Processing >>> > +Transmit processing for PktIOs operating in QUEUE mode is shown below: >>> > + >>> > +.PktIO QUEUE Mode Transmit Processing >>> > +image::../images/pktout_queue_send.svg[align="center] >>> > + >>> > +For TX processing QUEUE mode behaves similar to DIRECT mode except >>> that >>> > +output queues are regular ODP event queues that receive packets via >>> > ++odp_queue_enq()+ calls rather than special PktOut queues that use >>> > ++odp_pktout_send()+. Again, these queues are created via a call to >>> > ++odp_pktout_queue_config()+ following +odp_pktio_open()+. >>> > + >>> > +The main reason for selecting QUEUE mode for output is flexibility. >>> If an >>> > +application is designed to use a _pipeline model_ where packets flow >>> through >>> > +a series of processing stages via queues, then having the PktIO in >>> QUEUE >>> > +mode means that the application can always use the same enq APIs to >>> pass packets >>> > +from one stage to the next, including the final transmit output stage. >>> > + >>> > +==== Scheduled I/O Modes >>> > +The final PktIO mode supported integrates RX and TX processing with >>> the ODP >>> > +_event model_. For RX processing this involves the use of the >>> *Scheduler* >>> > +while for TX processing this involves the use of the *Traffic >>> Manager*. >>> > + >>> > +Scheduled RX Processing is further divided based on whether or not the >>> > +Classifier is used. >>> > + >>> > +===== Scheduled RX Processing >>> > +When a PktIO is opened with +ODP_PKTIN_MODE_SCHED+, it indicates that >>> the >>> > +input queues created by a subsequent +odp_pktin_queue_config()+ call >>> are to >>> > +be used as input to the *ODP Scheduler*. >>> > + >>> > +.PktIO SCHED Mode Receive Processing >>> > +image::../images/pktin_sched_recv.svg[align="center'] >>> > + >>> > +For basic use, SCHED mode simply associates the PktIO input event >>> queues >>> > +created by +odp_pktin_queue_config()+ with the scheduler. Hashing may >>> still be >>> > +employed to distribute input packets among multiple input queues. >>> However >>> > +instead of these being plain queues they are scheduled queues and have >>> > +associated scheduling attributes like priority, scheduler group, and >>> > +synchronization mode (parallel, atomic, ordered). SCHED mode thus >>> provides >>> > +both packet distribution (via the optional hash) as well as >>> scalability via >>> > +the ODP event model. >>> > + >>> > +In its fullest form, PktIOs operating in SCHED mode use the *ODP >>> Classifier* >>> > +to permit fine-grained flow separation on *Class of Service (CoS)* >>> boundaries. >>> > + >>> > +.PktIO SCHED Mode Receive Processing with Classification >>> > +image::../images/pktin_sched_cls.svg[align="center"] >>> >>> The diagram and the text confuses me a bit, here: my understanding was >>> that: >>> 1)the classifier associates a cos to each incoming packet using a set of >>> PMRs. >>> 2)Each cos maps to a pair {RXpool, RXqueue}. This Mapping is settled >>> during the >>> classifier setup. Packets with a given cos are stored on odp_packets >>> allocated >>> from the corresponding pool, and placed in the corresponding queue. >>> >> >> That is correct. What's unclear about that here? The point is that the >> queues shown here may either be those that the application creates via it's >> own odp_queue_create() calls or those created by odp_pktin_queue_config(). >> If the latter, the application will use odp_pktin_queue() to retrieve their >> handle to be passed when it creates the CoS it's to be associated with. >> Remember, multiple CoSes can add packets to the same queue--this is not a >> 1-to-1 association here. >> > > Maybe the figure wants to show too much. I don't really see what's going > on there. But maybe it is part of the classifier. > > Anyway. see U at BKK! > > /Christophe. > >> >> >>> >>> > + >>> > +In this mode of operation, the hash function of >>> +odp_pktin_queue_config()+ is >>> > +not used. Instead, the event queues created by this call, as well as >>> any >>> > +additional event queues created via separate +odp_queue_create()+ >>> calls are >>> > +associated with classes of service via +odp_cls_cos_create()+ calls. >>> > + >>> > +===== Scheduled TX Processing >>> > +Scheduled transmit processing is performed via the *ODP Traffic >>> Manager* and >>> > +is requested when a PktIO is opened with an +out_mode+ of >>> +ODP_PKTOUT_MODE_TM+. >>> > + >>> > +For TX processing via the Traffic Manager, applications use the >>> +odp_tm_enq()+ >>> > +API: >>> > +[source,c] >>> > +----- >>> > +/** The odp_tm_enq() function is used to add packets to a given TM >>> system. >>> > + * Note that the System Metadata associated with the pkt needed by >>> the TM >>> > + * system is (a) a drop_eligible bit, (b) a two bit "pkt_color", (c) >>> a 16-bit >>> > + * pkt_len, and MAYBE? (d) a signed 8-bit shaper_len_adjust. >>> > + * >>> > + * If there is a non-zero shaper_len_adjust, then it is added to the >>> pkt_len >>> > + * after any non-zero shaper_len_adjust that is part of the shaper >>> profile. >>> > + * >>> > + * The pkt_color bits are a result of some earlier >>> Metering/Marking/Policing >>> > + * processing (typically ingress based), and should not be confused >>> with the >>> > + * shaper_color produced from the TM shaper entities within the >>> tm_inputs and >>> > + * tm_nodes. >>> > + * >>> > + * @param[in] tm_queue Specifies the tm_queue (and indirectly the TM >>> system). >>> > + * @param[in] pkt Handle to a packet. >>> > + * @return Returns 0 upon success, < 0 upon failure. One >>> of the >>> > + * more common failure reasons is WRED dropage. >>> > + */ >>> > +int odp_tm_enq(odp_tm_queue_t tm_queue, odp_packet_t pkt); >>> > +----- >>> > +See the *Traffic Manager* section of this document for full >>> information about >>> > +Traffic Manager configuration and operation. >>> > diff --git a/doc/users-guide/users-guide.adoc >>> b/doc/users-guide/users-guide.adoc >>> > index d476225..ea24eaf 100644 >>> > --- a/doc/users-guide/users-guide.adoc >>> > +++ b/doc/users-guide/users-guide.adoc >>> > @@ -1023,6 +1023,8 @@ a valid UDP packet. Applications are expected to >>> exercise appropriate care >>> > when changing packet metadata to ensure that the resulting metadata >>> changes >>> > reflect the actual changed packet structure that the application has >>> made. >>> > >>> > +include::users-guide-pktio.adoc[] >>> > + >>> > == Cryptographic services >>> > >>> > ODP provides support for cryptographic operations required by various >>> security >>> > -- >>> > 2.5.0 >>> > >>> > _______________________________________________ >>> > lng-odp mailing list >>> > lng-odp@lists.linaro.org >>> > https://lists.linaro.org/mailman/listinfo/lng-odp >>> >>> Interresting reading, and nice to see the user doc growing! Thanks! >>> >> >> Thanks for the careful review and questions. >> >> >>> >>> Christophe >>> >> >> >
diff --git a/doc/users-guide/users-guide-pktio.adoc b/doc/users-guide/users-guide-pktio.adoc new file mode 100644 index 0000000..29fb6a9 --- /dev/null +++ b/doc/users-guide/users-guide-pktio.adoc @@ -0,0 +1,605 @@ +== PktIO Processing +Before packets can be manipulated they typically need to be _received_ and +after they are manipulated they need to be _transmitted_. The ODP abstraction +that captures these operations is the *Packet I/O (PktIO)*. +PktIOs are represented by handles of type *odp_pktio_t* and +represent a logical I/O interface that is mapped in an implementation-defined +manner to an underlying integrated I/O adapter or NIC. + +PktIO objects are manipulated through various state transitions via ++odp_pktio_xxx()+ API calls as shown below: + +.ODP PktIO Finite State Machine +image::../images/pktio_fsm.svg[align="center"] + +PktIOs begin in the *Unallocated* state. From here a call +odp_pktio_open()+ +is used to create an *odp_pktio_t* handle that is used in all subsequent calls +to manipulate the object. This call puts the PktIO into the *Unconfigured* +state. To become operational, a PktIO must first be +*configured* for Input, Output, or both Input and Output via the ++odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ APIs, and then +*started* via the +odp_pktio_start()+ to make it *Ready*. + +Following the completion of I/O processing, the +odp_pktio_stop()+ API returns +the PktIO to the *Configured* state. From here it may be *Reconfigured* via +additional +odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ calls, +or *Closed* via the +odp_pktio_close()+ API to return the PktIO to the +*Unallocated* state. + +=== PktIO Allocation +PktIO objects begin life by being _opened_ via the call: +[source,c] +----- +/** + * Open a packet IO interface + * + * An ODP program can open a single packet IO interface per device, attempts + * to open an already open device will fail, returning ODP_PKTIO_INVALID with + * errno set. Use odp_pktio_lookup() to obtain a handle to an already open + * device. Packet IO parameters provide interface level configuration options. + * + * Packet input queue configuration must be setup with + * odp_pktin_queue_config() before odp_pktio_start() is called. When packet + * input mode is ODP_PKTIN_MODE_DISABLED, odp_pktin_queue_config() call is + * optional and will ignore all parameters. + * + * Packet output queue configuration must be setup with + * odp_pktout_queue_config() before odp_pktio_start() is called. When packet + * output mode is ODP_PKTOUT_MODE_DISABLED or ODP_PKTOUT_MODE_TM, + * odp_pktout_queue_config() call is optional and will ignore all parameters. + * + * Packet receive and transmit on the interface is enabled with a call to + * odp_pktio_start(). If not specified otherwise, any interface level + * configuration must not be changed when the interface is active (between start + * and stop calls). + * + * In summary, a typical pktio interface setup sequence is ... + * * odp_pktio_open() + * * odp_pktin_queue_config() + * * odp_pktout_queue_config() + * * odp_pktio_start() + * + * ... and tear down sequence is: + * * odp_pktio_stop() + * * odp_pktio_close() + * + * @param name Packet IO device name + * @param pool Default pool from which to allocate storage for packets + * received over this interface, must be of type ODP_POOL_PACKET + * @param param Packet IO parameters + * + * @return Packet IO handle + * @retval ODP_PKTIO_INVALID on failure + * + * @note The device name "loop" is a reserved name for a loopback device used + * for testing purposes. + * + * @note Packets arriving via this interface assigned to a CoS by the + * classifier are received into the pool associated with that CoS. This + * will occur either because this pktio is assigned a default CoS via + * the odp_pktio_default_cos_set() routine, or because a matching PMR + * assigned the packet to a specific CoS. The default pool specified + * here is applicable only for those packets that are not assigned to a + * more specific CoS. + * + * @see odp_pktio_start(), odp_pktio_stop(), odp_pktio_close() + */ +odp_pktio_t odp_pktio_open(const char *name, odp_pool_t pool, + const odp_pktio_param_t *param); +----- ++odp_pktio_open()+ takes three arguments: a *name*, which is an +implementation-defined string that identifies the logical interface to be +opened, a *pool* that identifies the ODP pool that storage for received +packets should be allocated from, and a *param* structure that specifies +I/O options to be associated with this PktIO instance. +[source,c] +----- +/** + * Packet IO parameters + * + * In minimum, user must select input and output modes. Use 0 for defaults. + * Initialize entire struct with zero to maintain API compatibility. + */ +typedef struct odp_pktio_param_t { + /** Packet input mode + * + * The default value is ODP_PKTIN_MODE_DIRECT. */ + odp_pktin_mode_t in_mode; + /** Packet output mode + * + * The default value is ODP_PKTOUT_MODE_DIRECT. */ + odp_pktout_mode_t out_mode; +} odp_pktio_param_t; +----- +ODP defines *loop* as a reserved name to indicate that this PktIO represents +a loopback interface. Loopback interfaces are useful as a means of recycling +packets back for reclassification after decryption or decapsulation, as well as +for diagnostic or testing purposes. The *pool* specifies the default pool to +use for packet allocation if not overridden by the classifier due to a +specific or default Class-of-Service (CoS) match on the packet. The *param* +struct, in turn, specifies the input and output *modes* of the PktIO. + +=== PktIO Input and Output Modes +PktIO objects support four different Input and Output modes, that may be +specified independently at *open* time. + +.PktIO Input Modes +* +ODP_PKTIN_MODE_DIRECT+ +* +ODP_PKTIN_MODE_QUEUE+ +* +ODP_OKTIN_MODE_SCHED+ +* +ODP_PKTIN_MODE_DISABLED+ + +.PktIO Output Modes +* +ODP_PKTOUT_MODE_DIRECT+ +* +ODP_PKTOUT_MODE_QUEUE+ +* +ODP_PKTOUT_MODE_TM+ +* +ODP_PKTOUT_MODE_DISABLED+ + +The DISABLED modes indicate that either input or output is prohibited on this +PktIO. Attempts to receive packets on a PktIO whose +in_mode+ is DISABLED +return no packets while packets sent to a PktIO whose +out_mode+ is DISABLED +are discarded. + +==== Direct I/O Modes +DIRECT I/O is the default mode for PktIO objects. It is designed to support +poll-based packet processing, which is often found in legacy applications +being ported to ODP, and can also be a preferred mode for some types of +packet processing. By supporting poll-based I/O processing, ODP provides +maximum flexibility to the data plane application writer. + +===== Direct RX Processing +The processing of DIRECT input is shown below: + +.PktIO DIRECT Mode Receive Processing +image::../images/pktin_direct_recv.svg[align="center"] + +In DIRECT mode, received packets are stored in one or more special PktIO queues +of type *odp_pktin_queue_t* and are retrieved by threads calling the ++odp_pktin_recv()+ API. + +Once opened, setting up a DIRECT mode PktIO is performed by the ++odp_pktin_queue_config()+ API. +[source,c] +----- +/** + * Configure packet input queues + * + * Setup a number of packet input queues and configure those. The maximum number + * of queues is platform dependent and can be queried with + * odp_pktio_capability(). Use odp_pktin_queue_param_init() to initialize + * parameters into their default values. Default values are also used when + * 'param' pointer is NULL. + * + * Queue handles for input queues can be requested with odp_pktin_queue() or + * odp_pktin_event_queue() after this call. All requested queues are setup on + * success, no queues are setup on failure. Each call reconfigures input queues + * and may invalidate all previous queue handles. + * + * @param pktio Packet IO handle + * @param param Packet input queue configuration parameters. Uses defaults + * when NULL. + * + * @retval 0 on success + * @retval <0 on failure + * + * @see odp_pktio_capability(), odp_pktin_queue(), odp_pktin_event_queue() + */ +int odp_pktin_queue_config(odp_pktio_t pktio, + const odp_pktin_queue_param_t *param); +----- +The second argument to this call is the *odp_pktin_queue_param_t* +[source,c] +----- +/** + * Packet input queue parameters + */ +typedef struct odp_pktin_queue_param_t { + /** Operation mode + * + * The default value is ODP_PKTIO_OP_MT. Application may enable + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE when + * applicable. */ + odp_pktio_op_mode_t op_mode; + + /** Enable flow hashing + * 0: Do not hash flows + * 1: Hash flows to input queues */ + odp_bool_t hash_enable; + + /** Protocol field selection for hashing. Multiple protocols can be + * selected. */ + odp_pktin_hash_proto_t hash_proto; + + /** Number of input queues to be created. More than one input queue + * require input hashing or classifier setup. Hash_proto is ignored + * when hash_enable is zero or num_queues is one. This value must be + * between 1 and interface capability. Queue type is defined by the + * input mode. The default value is 1. */ + unsigned num_queues; + + /** Queue parameters for creating input queues in ODP_PKTIN_MODE_QUEUE + * or ODP_PKTIN_MODE_SCHED modes. Scheduler parameters are considered + * only in ODP_PKTIN_MODE_SCHED mode. */ + odp_queue_param_t queue_param; + +} odp_pktin_queue_param_t; +----- +Note that the *queue_param* field of this struct is ignored in DIRECT mode. +The purpose of +odp_pktin_queue_config()+ is to specify the number of PktIn +queues to be created and to set their attributes. + +===== Hash Processing +Another feature of DIRECT mode input is the provision of a *hash* function used +to distribute incoming packets among the PktIO's PktIn queues. If the ++hash_enable+ field of the *odp_pktin_queue_param_t* is 1, +then the +hash_proto+ field is used to specify which field(s) of incoming +packets should be used as input to an implementation-defined packet +distribution hash function. +[source,c] +----- +/** + * Packet input hash protocols + * + * The list of protocol header field combinations, which are included into + * packet input hash calculation. + */ +typedef union odp_pktin_hash_proto_t { + /** Protocol header fields for hashing */ + struct { + /** IPv4 addresses and UDP port numbers */ + uint32_t ipv4_udp : 1; + /** IPv4 addresses and TCP port numbers */ + uint32_t ipv4_tcp : 1; + /** IPv4 addresses */ + uint32_t ipv4 : 1; + /** IPv6 addresses and UDP port numbers */ + uint32_t ipv6_udp : 1; + /** IPv6 addresses and TCP port numbers */ + uint32_t ipv6_tcp : 1; + /** IPv6 addresses */ + uint32_t ipv6 : 1; + } proto; + + /** All bits of the bit field structure */ + uint32_t all_bits; +} odp_pktin_hash_proto_t; +----- +Note that the hash function used in PktIO poll mode operation is intended to +provide simple packet distribution among multiple PktIn queues associated with +the PktIO. It does not have the sophistication of the *ODP Classifier*, however +it also does not incur the setup requirements of pattern matching rules, +making it a simpler choice for less sophisticated applications. Note that +ODP does not specify how the hash is to be performed. That is left to each +implementation. The hash only specifies which input packet fields are of +interest to the application and should be considered by the hash function in +deciding how to distribute packets among PktIn queues. The only expectation +is that packets that have the same hash values should all be mapped to the +same PktIn queue. + +===== PktIn Queues +A *PktIn Queue* is a special type of queue that is used internally by PktIOs +operating in DIRECT mode. Applications cannot perform enqueues to these queues, +however they may obtain references to them via the +odp_pktin_queue()+ API +[source,c] +----- +/** + * Direct packet input queues + * + * Returns the number of input queues configured for the interface in + * ODP_PKTIN_MODE_DIRECT mode. Outputs up to 'num' queue handles when the + * 'queues' array pointer is not NULL. If return value is larger than 'num', + * there are more queues than the function was allowed to output. If return + * value (N) is less than 'num', only queues[0 ... N-1] have been written. + * + * Packets from these queues are received with odp_pktin_recv(). + * + * @param pktio Packet IO handle + * @param[out] queues Points to an array of queue handles for output + * @param num Maximum number of queue handles to output + * + * @return Number of packet input queues + * @retval <0 on failure + */ +int odp_pktin_queue(odp_pktio_t pktio, odp_pktin_queue_t queues[], int num); +----- +Once configured, prior to receiving packets the PktIO must be placed into the +*Ready* state via a call to +odp_pktio_start() +[source,c] +----- +/** + * Start packet receive and transmit + * + * Activate packet receive and transmit on a previously opened or stopped + * interface. The interface can be stopped with a call to odp_pktio_stop(). + * + * @param pktio Packet IO handle + * + * @retval 0 on success + * @retval <0 on failure + * + * @see odp_pktio_open(), odp_pktio_stop() + */ +int odp_pktio_start(odp_pktio_t pktio); +----- +Once started, the PktIn queue handles are used as arguments to ++odp_pktin_recv()+ to receive packets from the PktIO. +[source,c] +----- +/** + * Receive packets directly from an interface input queue + * + * Receives up to 'num' packets from the pktio interface input queue. When + * input queue parameter 'op_mode' has been set to ODP_PKTIO_OP_MT_UNSAFE, + * the operation is optimized for single thread operation per queue and the same + * queue must not be accessed simultaneously from multiple threads. + * + * @param queue Packet input queue handle for receiving packets + * @param[out] packets[] Packet handle array for output of received packets + * @param num Maximum number of packets to receive + * + * @return Number of packets received + * @retval <0 on failure + * + * @see odp_pktin_queue() + */ +int odp_pktin_recv(odp_pktin_queue_t queue, odp_packet_t packets[], int num); +----- + +===== Direct TX Processing +A PktIO operating in DIRECT mode performs TX processing as shown here: + +.PktIO DIRECT Mode Transmit Processing +image::../images/pktout_direct_send.svg[align="center"] + +Direct TX processing operates similarly to Direct RX processing. Following +open, the +odp_pktout_queue_config()+ API is used to create and configure +one or more *PktOut queues* to be used to support packet transmission by +this PktIO +[source,c] +----- +/** + * Configure packet output queues + * + * Setup a number of packet output queues and configure those. The maximum + * number of queues is platform dependent and can be queried with + * odp_pktio_capability(). Use odp_pktout_queue_param_init() to initialize + * parameters into their default values. Default values are also used when + * 'param' pointer is NULL. + * + * Queue handles for output queues can be requested with odp_pktout_queue() or + * odp_pktout_event_queue() after this call. All requested queues are setup on + * success, no queues are setup on failure. Each call reconfigures output queues + * and may invalidate all previous queue handles. + * + * @param pktio Packet IO handle + * @param param Packet output queue configuration parameters. Uses defaults + * when NULL. + * + * @retval 0 on success + * @retval <0 on failure + * + * @see odp_pktio_capability(), odp_pktout_queue(), odp_pktout_event_queue() + */ +int odp_pktout_queue_config(odp_pktio_t pktio, + const odp_pktout_queue_param_t *param); +----- +As with +odp_pktin_queue_config()+, the configuration of PktOut queues +involves the use of a parameter struct: +[source,c] +----- +/** + * Packet output queue parameters + * + * These parameters are used in ODP_PKTOUT_MODE_DIRECT and + * ODP_PKTOUT_MODE_QUEUE modes. + */ +typedef struct odp_pktout_queue_param_t { + /** Operation mode + * + * The default value is ODP_PKTIO_OP_MT. Application may enable + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE when + * applicable. */ + odp_pktio_op_mode_t op_mode; + + /** Number of output queues to be created. The value must be between + * 1 and interface capability. The default value is 1. */ + unsigned num_queues; + +} odp_pktout_queue_param_t; +----- +As with direct input, direct output uses one or more special output queues +of type *odp_pktout_queue_t* that area created and configured by this call. + +As with PktIn queues, the handles for these created PktOut queues may be +retrieved by the +odp_pktout_queue()+ API: +[source,c] +----- +/** + * Direct packet output queues + * + * Returns the number of output queues configured for the interface in + * ODP_PKTOUT_MODE_DIRECT mode. Outputs up to 'num' queue handles when the + * 'queues' array pointer is not NULL. If return value is larger than 'num', + * there are more queues than the function was allowed to output. If return + * value (N) is less than 'num', only queues[0 ... N-1] have been written. + * + * Packets are sent to these queues with odp_pktout_send(). + * + * @param pktio Packet IO handle + * @param[out] queues Points to an array of queue handles for output + * @param num Maximum number of queue handles to output + * + * @return Number of packet output queues + * @retval <0 on failure + */ +int odp_pktout_queue(odp_pktio_t pktio, odp_pktout_queue_t queues[], int num); +----- +Once the PktIO has been configured for output and started via ++odp_pktio_start()+, packets may be transmitted to the PktIO by calling +_odp_pktout_send()+: +[source,c] +----- +/** + * Send packets directly to an interface output queue + * + * Sends out a number of packets to the interface output queue. When + * output queue parameter 'op_mode' has been set to ODP_PKTIO_OP_MT_UNSAFE, + * the operation is optimized for single thread operation per queue and the same + * queue must not be accessed simultaneously from multiple threads. + * + * A successful call returns the actual number of packets sent. If return value + * is less than 'num', the remaining packets at the end of packets[] array + * are not consumed, and the caller has to take care of them. + * + * @param queue Packet output queue handle for sending packets + * @param packets[] Array of packets to send + * @param num Number of packets to send + * + * @return Number of packets sent + * @retval <0 on failure + */ +int odp_pktout_send(odp_pktout_queue_t queue, odp_packet_t packets[], int num); +----- +Note that the argument to this call specifies the PktOut queue to that the +packet is to be added to rather than the PktIO itself. This permits multiple +threads (presumably operating on different cores) a more efficient means of +separating I/O processing destined for the same interface. + +==== Queued I/O Modes +To provide additional flexibility when operating in poll mode, PktIOs may also +be opened in QUEUE Mode. The difference between DIRECT and QUEUE mode is that +QUEUE mode uses standard ODP event queues to service packets. + +===== Queue RX Processing +The processing for QUEUE input processing is shown below: + +.PktIO QUEUE Mode Receive Processing +image::../images/pktin_queue_recv.svg[align="center"] + +In QUEUE mode, received packets are stored in one or more standard ODP queues. +The difference is that these queues are not created directly by the +application. Instead, they are created in response to an ++odp_pktin_queue_config()+ call. + +As with DIRECT mode, the +odp_pktin_queue_param_t+ specified to this call +indicates whether an input hash should be used and if so which field(s) of +the packet should be considered as input to the has function. + +The main difference between DIRECT and QUEUE RX processing is that because +the PktIO uses standard ODP event queues, other parts of the application can +use +odp_queue_enq()+ API calls to enqueue packets to these queues for +"RX" processing in addition to those originating from the PktIO interface +itself. To obtain the handles of these input queues, the ++odp_pktin_event_queue()+ API is used: +[source,c] +----- +/** + * Event queues for packet input + * + * Returns the number of input queues configured for the interface in + * ODP_PKTIN_MODE_QUEUE and ODP_PKTIN_MODE_SCHED modes. Outputs up to 'num' + * queue handles when the 'queues' array pointer is not NULL. If return value is + * larger than 'num', there are more queues than the function was allowed to + * output. If return value (N) is less than 'num', only queues[0 ... N-1] have + * been written. + * + * Packets (and other events) from these queues are received with + * odp_queue_deq(), odp_schedule(), etc calls. + * + * @param pktio Packet IO handle + * @param[out] queues Points to an array of queue handles for output + * @param num Maximum number of queue handles to output + * + * @return Number of packet input queues + * @retval <0 on failure + */ +int odp_pktin_event_queue(odp_pktio_t pktio, odp_queue_t queues[], int num); +----- +Similarly, threads receive packets from PktIOs operating in QUEUE mode by +making standard +odp_queue_deq()+ calls to one of the event queues associated +with the PktIO. + +===== Queue TX Processing +Transmit processing for PktIOs operating in QUEUE mode is shown below: + +.PktIO QUEUE Mode Transmit Processing +image::../images/pktout_queue_send.svg[align="center] + +For TX processing QUEUE mode behaves similar to DIRECT mode except that +output queues are regular ODP event queues that receive packets via ++odp_queue_enq()+ calls rather than special PktOut queues that use ++odp_pktout_send()+. Again, these queues are created via a call to ++odp_pktout_queue_config()+ following +odp_pktio_open()+. + +The main reason for selecting QUEUE mode for output is flexibility. If an +application is designed to use a _pipeline model_ where packets flow through +a series of processing stages via queues, then having the PktIO in QUEUE +mode means that the application can always use the same enq APIs to pass packets +from one stage to the next, including the final transmit output stage. + +==== Scheduled I/O Modes +The final PktIO mode supported integrates RX and TX processing with the ODP +_event model_. For RX processing this involves the use of the *Scheduler* +while for TX processing this involves the use of the *Traffic Manager*. + +Scheduled RX Processing is further divided based on whether or not the +Classifier is used. + +===== Scheduled RX Processing +When a PktIO is opened with +ODP_PKTIN_MODE_SCHED+, it indicates that the +input queues created by a subsequent +odp_pktin_queue_config()+ call are to +be used as input to the *ODP Scheduler*. + +.PktIO SCHED Mode Receive Processing +image::../images/pktin_sched_recv.svg[align="center'] + +For basic use, SCHED mode simply associates the PktIO input event queues +created by +odp_pktin_queue_config()+ with the scheduler. Hashing may still be +employed to distribute input packets among multiple input queues. However +instead of these being plain queues they are scheduled queues and have +associated scheduling attributes like priority, scheduler group, and +synchronization mode (parallel, atomic, ordered). SCHED mode thus provides +both packet distribution (via the optional hash) as well as scalability via +the ODP event model. + +In its fullest form, PktIOs operating in SCHED mode use the *ODP Classifier* +to permit fine-grained flow separation on *Class of Service (CoS)* boundaries. + +.PktIO SCHED Mode Receive Processing with Classification +image::../images/pktin_sched_cls.svg[align="center"] + +In this mode of operation, the hash function of +odp_pktin_queue_config()+ is +not used. Instead, the event queues created by this call, as well as any +additional event queues created via separate +odp_queue_create()+ calls are +associated with classes of service via +odp_cls_cos_create()+ calls. + +===== Scheduled TX Processing +Scheduled transmit processing is performed via the *ODP Traffic Manager* and +is requested when a PktIO is opened with an +out_mode+ of +ODP_PKTOUT_MODE_TM+. + +For TX processing via the Traffic Manager, applications use the +odp_tm_enq()+ +API: +[source,c] +----- +/** The odp_tm_enq() function is used to add packets to a given TM system. + * Note that the System Metadata associated with the pkt needed by the TM + * system is (a) a drop_eligible bit, (b) a two bit "pkt_color", (c) a 16-bit + * pkt_len, and MAYBE? (d) a signed 8-bit shaper_len_adjust. + * + * If there is a non-zero shaper_len_adjust, then it is added to the pkt_len + * after any non-zero shaper_len_adjust that is part of the shaper profile. + * + * The pkt_color bits are a result of some earlier Metering/Marking/Policing + * processing (typically ingress based), and should not be confused with the + * shaper_color produced from the TM shaper entities within the tm_inputs and + * tm_nodes. + * + * @param[in] tm_queue Specifies the tm_queue (and indirectly the TM system). + * @param[in] pkt Handle to a packet. + * @return Returns 0 upon success, < 0 upon failure. One of the + * more common failure reasons is WRED dropage. + */ +int odp_tm_enq(odp_tm_queue_t tm_queue, odp_packet_t pkt); +----- +See the *Traffic Manager* section of this document for full information about +Traffic Manager configuration and operation. diff --git a/doc/users-guide/users-guide.adoc b/doc/users-guide/users-guide.adoc index d476225..ea24eaf 100644 --- a/doc/users-guide/users-guide.adoc +++ b/doc/users-guide/users-guide.adoc @@ -1023,6 +1023,8 @@ a valid UDP packet. Applications are expected to exercise appropriate care when changing packet metadata to ensure that the resulting metadata changes reflect the actual changed packet structure that the application has made. +include::users-guide-pktio.adoc[] + == Cryptographic services ODP provides support for cryptographic operations required by various security
Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org> --- doc/users-guide/users-guide-pktio.adoc | 605 +++++++++++++++++++++++++++++++++ doc/users-guide/users-guide.adoc | 2 + 2 files changed, 607 insertions(+) create mode 100644 doc/users-guide/users-guide-pktio.adoc