[API-NEXT,PATCHv3,3/3] doc: userguide: update pktio section for new modes and apis

diff --git a/doc/users-guide/users-guide-pktio.adoc b/doc/users-guide/users-guide-pktio.adoc
index 2dd0b15..b19b958 100644
--- a/doc/users-guide/users-guide-pktio.adoc
+++ b/doc/users-guide/users-guide-pktio.adoc
@@ -10,21 +10,23 @@  PktIO objects are manipulated through various state transitions via
 +odp_pktio_xxx()+ API calls as shown below:
 
 .ODP PktIO Finite State Machine
-image::../images/pktio_fsm.svg[align="center"]
+image::pktio_fsm.svg[align="center"]
 
 PktIOs begin in the *Unallocated* state. From here a call +odp_pktio_open()+
 is used to create an *odp_pktio_t* handle that is used in all subsequent calls
 to manipulate the object. This call puts the PktIO into the *Unconfigured*
 state. To become operational, a PktIO must first be
 *configured* for Input, Output, or both Input and Output via the
-+odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ APIs, and then
-*started* via the +odp_pktio_start()+ to make it *Ready*.
++odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ APIs. PktIOs
+may also have general options configured via the +odp_pktio_config()+ API.
+Once configured, PktIOs are then *started* via the +odp_pktio_start()+ API
+to make it *Ready*.
 
 Following the completion of I/O processing, the +odp_pktio_stop()+ API returns
 the PktIO to the *Configured* state. From here it may be *Reconfigured* via
-additional +odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ calls,
-or *Closed* via the +odp_pktio_close()+ API to return the PktIO to the
-*Unallocated* state.
+additional +odp_pktio_config()+, +odp_pktin_queue_config()+ and/or
++odp_pktout_queue_config()+ calls, or *Closed* via the +odp_pktio_close()+
+API to return the PktIO to the *Unallocated* state.
 
 === PktIO Allocation
 PktIO objects begin life by being _opened_ via the call:
@@ -38,6 +40,9 @@  PktIO objects begin life by being _opened_ via the call:
  * errno set. Use odp_pktio_lookup() to obtain a handle to an already open
  * device. Packet IO parameters provide interface level configuration options.
  *
+ * Use odp_pktio_param_init() to initialize packet IO parameters into their
+ * default values. Default values are also used when 'param' pointer is NULL.
+ *
  * Packet input queue configuration must be setup with
  * odp_pktin_queue_config() before odp_pktio_start() is called. When packet
  * input mode is ODP_PKTIN_MODE_DISABLED, odp_pktin_queue_config() call is
@@ -66,7 +71,7 @@  PktIO objects begin life by being _opened_ via the call:
  * @param name   Packet IO device name
  * @param pool   Default pool from which to allocate storage for packets
  *               received over this interface, must be of type ODP_POOL_PACKET
- * @param param  Packet IO parameters
+ * @param param  Packet IO parameters. Uses defaults when NULL.
  *
  * @return Packet IO handle
  * @retval ODP_PKTIO_INVALID on failure
@@ -85,7 +90,7 @@  PktIO objects begin life by being _opened_ via the call:
  * @see odp_pktio_start(), odp_pktio_stop(), odp_pktio_close()
  */
 odp_pktio_t odp_pktio_open(const char *name, odp_pool_t pool,
-                           const odp_pktio_param_t *param);
+			   const odp_pktio_param_t *param);
 -----
 +odp_pktio_open()+ takes three arguments: a *name*, which is an
 implementation-defined string that identifies the logical interface to be
@@ -97,18 +102,20 @@  I/O options to be associated with this PktIO instance.
 /**
  * Packet IO parameters
  *
- * In minimum, user must select input and output modes. Use 0 for defaults.
- * Initialize entire struct with zero to maintain API compatibility.
+ * Packet IO interface level parameters. Use odp_pktio_param_init() to
+ * initialize the structure with default values.
  */
 typedef struct odp_pktio_param_t {
 	/** Packet input mode
 	  *
 	  * The default value is ODP_PKTIN_MODE_DIRECT. */
 	odp_pktin_mode_t in_mode;
+
 	/** Packet output mode
 	  *
 	  * The default value is ODP_PKTOUT_MODE_DIRECT. */
 	odp_pktout_mode_t out_mode;
+
 } odp_pktio_param_t;
 -----
 ODP defines *"loop"* as a reserved name to indicate that this PktIO represents
@@ -159,7 +166,7 @@  maximum flexibility to the data plane application writer.
 The processing of DIRECT input is shown below:
 
 .PktIO DIRECT Mode Receive Processing
-image::../images/pktin_direct_recv.svg[align="center"]
+image::pktin_direct_recv.svg[align="center"]
 
 In DIRECT mode, received packets are stored in one or more special PktIO queues
 of type *odp_pktin_queue_t* and are retrieved by threads calling the
@@ -209,30 +216,53 @@  typedef struct odp_pktin_queue_param_t {
 	  * applicable. */
 	odp_pktio_op_mode_t op_mode;
 
+	/** Enable classifier
+	  *
+	  * * 0: Classifier is disabled (default)
+	  * * 1: Classifier is enabled. Use classifier to direct incoming
+	  *      packets into pktin event queues. Classifier can be enabled
+	  *      only in ODP_PKTIN_MODE_SCHED and ODP_PKTIN_MODE_QUEUE modes.
+	  *      Both classifier and hashing cannot be enabled simultaneously
+	  *      ('hash_enable' must be 0). */
+	odp_bool_t classifier_enable;
+
 	/** Enable flow hashing
-	  * 0: Do not hash flows
-	  * 1: Hash flows to input queues */
+	  *
+	  * * 0: Do not hash flows (default)
+	  * * 1: Enable flow hashing. Use flow hashing to spread incoming
+	  *      packets into input queues. Hashing can be enabled in all
+	  *      modes. Both classifier and hashing cannot be enabled
+	  *      simultaneously ('classifier_enable' must be 0). */
 	odp_bool_t hash_enable;
 
-	/** Protocol field selection for hashing. Multiple protocols can be
-	  * selected. */
+	/** Protocol field selection for hashing
+	  *
+	  * Multiple protocols can be selected. Ignored when 'hash_enable' is
+	  * zero. The default value is all bits zero. */
 	odp_pktin_hash_proto_t hash_proto;
 
-	/** Number of input queues to be created. More than one input queue
-	  * require input hashing or classifier setup. Hash_proto is ignored
-	  * when hash_enable is zero or num_queues is one. This value must be
-	  * between 1 and interface capability. Queue type is defined by the
+	/** Number of input queues to be created
+	  *
+	  * When classifier is enabled the number of queues may be zero
+	  * (in odp_pktin_queue_config() step), otherwise at least one
+	  * queue is required. More than one input queues require either flow
+	  * hashing or classifier enabled. The maximum value is defined by
+	  * pktio capability 'max_input_queues'. Queue type is defined by the
 	  * input mode. The default value is 1. */
 	unsigned num_queues;
 
-	/** Queue parameters for creating input queues in ODP_PKTIN_MODE_QUEUE
+	/** Queue parameters
+	  *
+	  * These are used for input queue creation in ODP_PKTIN_MODE_QUEUE
 	  * or ODP_PKTIN_MODE_SCHED modes. Scheduler parameters are considered
-	  * only in ODP_PKTIN_MODE_SCHED mode. */
+	  * only in ODP_PKTIN_MODE_SCHED mode. Default values are defined in
+	  * odp_queue_param_t documentation. */
 	odp_queue_param_t queue_param;
 
 } odp_pktin_queue_param_t;
 -----
-Note that the *queue_param* field of this struct is ignored in DIRECT mode.
+Note that the *queue_param* and *classifier_enable* fields of this struct are
+ignored in DIRECT mode.
 The purpose of +odp_pktin_queue_config()+ is to specify the number of PktIn
 queues to be created and to set their attributes.
 
@@ -347,8 +377,10 @@  Once started, the PktIn queue handles are used as arguments to
 /**
  * Receive packets directly from an interface input queue
  *
- * Receives up to 'num' packets from the pktio interface input queue. When
- * input queue parameter 'op_mode' has been set to ODP_PKTIO_OP_MT_UNSAFE,
+ * Receives up to 'num' packets from the pktio interface input queue. Returns
+ * the number of packets received.
+ *
+ * When input queue parameter 'op_mode' has been set to ODP_PKTIO_OP_MT_UNSAFE,
  * the operation is optimized for single thread operation per queue and the same
  * queue must not be accessed simultaneously from multiple threads.
  *
@@ -372,11 +404,27 @@  with a single RX thread (in which case the PktIn queue can be marked
 MT_UNSAFE), however this is up to the application to determine how best
 to structure itself.
 
+===== Direct Receive with Timeout
+PktIOs operating in DIRECT mode can also be received with timeout as shown
+here:
+
+.PktIO Direct Mode Receive with Timeout
+image::pktin_direct_tmo.svg[align="center"]
+
+Two enhanced versions of +odp_pktin_recv()+ are provided for this purpose.
++odp_pktin_recv_tmo()+ operates the same as +odp_pktin_recv()+ except that it
+accepts a _timeout_ parameter that ensures that the call will return if no
+packet is received from the PktIn queue within a specified interval. To permit
+applications to poll multiple queues (possibly from separate PktIOs) in
+parallel, the +odp_pktin_mq_tmo()+ routine is provided. This latter routine
+is particularly useful in allowing a poll-mode application to scan multiple
+input queues (possibly spanning multiple PktIO interfaces) in a single call.
+
 ===== Direct TX Processing
 A PktIO operating in DIRECT mode performs TX processing as shown here:
 
 .PktIO DIRECT Mode Transmit Processing
-image::../images/pktout_direct_send.svg[align="center"]
+image::pktout_direct_send.svg[align="center"]
 
 Direct TX processing operates similarly to Direct RX processing. Following
 open, the +odp_pktout_queue_config()+ API is used to create and configure
@@ -495,28 +543,32 @@  separating I/O processing destined for the same interface.
 ==== Queued I/O Modes
 To provide additional flexibility when operating in poll mode, PktIOs may also
 be opened in QUEUE Mode. The difference between DIRECT and QUEUE mode is that
-QUEUE mode uses standard ODP event queues to service packets.
+QUEUE mode uses standard ODP event queues to service packets and QUEUE mode
+also supports the use of the *ODP Classifier*.
 
 ===== Queue RX Processing
 The processing for QUEUE input processing is shown below:
 
 .PktIO QUEUE Mode Receive Processing
-image::../images/pktin_queue_recv.svg[align="center"]
+image::pktin_queue_recv.svg[align="center"]
 
 In QUEUE mode, received packets are stored in one or more standard ODP queues.
-The difference is that these queues are not created directly by the
-application. Instead, they are created in response to an
-+odp_pktin_queue_config()+ call.
+The number of these queues is specified by the +num_queues+ field of the
++odp_pktin_queue_param_t+ passed to +odp_pktin_queue_config()+.
 
 As with DIRECT mode, the +odp_pktin_queue_param_t+ specified to this call
 indicates whether an input hash should be used and if so which field(s) of
-the packet should be considered as input to the has function.
-
-The main difference between DIRECT and QUEUE RX processing is that because
-the PktIO uses standard ODP event queues, other parts of the application can
-use +odp_queue_enq()+ API calls to enqueue packets to these queues for
-"RX" processing in addition to those originating from the PktIO interface
-itself. To obtain the handles of these input queues, the
+the packet should be considered as input to the has function. For QUEUE mode,
+however, the application may use the classifier rather than the hash
+function. In this case, the application may specify the +num_queues+
+as zero, in which case the application itself will create them via normal
++odp_queue_create()+ calls.
+
+The main operational difference between DIRECT and QUEUE RX processing is that
+because the PktIO uses standard ODP event queues, other parts of the
+application can use +odp_queue_enq()+ API calls to enqueue packets to these
+queues for "RX" processing in addition to those originating from the PktIO
+interface itself. To obtain the handles of these input queues, the
 +odp_pktin_event_queue()+ API is used:
 [source,c]
 -----
@@ -550,7 +602,7 @@  with the PktIO.
 Transmit processing for PktIOs operating in QUEUE mode is shown below:
 
 .PktIO QUEUE Mode Transmit Processing
-image::../images/pktout_queue_send.svg[align="center]
+image::pktout_queue_send.svg[align="center]
 
 For TX processing QUEUE mode behaves similar to DIRECT mode except that
 output queues are regular ODP event queues that receive packets via
@@ -578,30 +630,32 @@  input queues created by a subsequent +odp_pktin_queue_config()+ call are to
 be used as input to the *ODP Scheduler*.
 
 .PktIO SCHED Mode Receive Processing
-image::../images/pktin_sched_recv.svg[align="center']
+image::pktin_sched_recv.svg[align="center']
 
-For basic use, SCHED mode simply associates the PktIO input event queues
-created by +odp_pktin_queue_config()+ with the scheduler. Hashing may still be
-employed to distribute input packets among multiple input queues. However
+PktIO SCHED mode simply associates the PktIO input event queues
+created by +odp_pktin_queue_config()+ with the scheduler. As with QUEUE mode,
+hashing may be employed to distribute input packets among multiple input
+queues, or full classification may be employed. However
 instead of these being plain queues they are scheduled queues and have
 associated scheduling attributes like priority, scheduler group, and
 synchronization mode (parallel, atomic, ordered). SCHED mode thus provides
-both packet distribution (via the optional hash) as well as scalability via
-the ODP event model.
+both packet distribution (via the optional hash or classification) as well as
+scalability via the ODP event model.
 
 In its fullest form, PktIOs operating in SCHED mode use the *ODP Classifier*
 to permit fine-grained flow separation on *Class of Service (CoS)* boundaries.
 
 .PktIO SCHED Mode Receive Processing with Classification
-image::../images/pktin_sched_cls.svg[align="center"]
+image::pktin_sched_cls.svg[align="center"]
 
 In this mode of operation, the hash function of +odp_pktin_queue_config()+ is
-typically not used. Instead, the event queues created by this call,
+not used. Instead, the +classifier_enable+ bit in the
++odp_pktin_queue_config_t+ specifies that input from this PktIO is to be routed
+through the ODP classifier.  Any event queues created by this call,
 as well as any additional event queues created via separate
 +odp_queue_create()+ calls are associated with classes of service via
-+odp_cls_cos_create()+ calls. Classification is enabled for the PktIO as a
-whole by assigning a _default_ CoS via the +odp_pktio_default_cos_set()+
-API.
++odp_cls_cos_create()+ calls. In addition a _default_ CoS for the PktIO can
+be assigned via the +odp_pktio_default_cos_set()+ API.
 
 When operating in SCHED mode, applications do not call PktIn receive functions.
 Instead the PktIn queues are scanned by the scheduler and, if classification
@@ -643,3 +697,170 @@  int odp_tm_enq(odp_tm_queue_t tm_queue, odp_packet_t pkt);
 -----
 See the *Traffic Manager* section of this document for full information about
 Traffic Manager configuration and operation.
+
+=== PktIO Configuration
+In addition to I/O mode configuration, the +odp_pktio_config()+ API is used
+to perform general configuration of PktIO interfaces:
+[source,c]
+-----
+/**
+ * Configure packet IO interface options
+ *
+ * Select interface level configuration options before the interface is
+ * activated (before odp_pktio_start() call). This step is optional in pktio
+ * interface setup sequence. Use odp_pktio_capability() to query configuration
+ * capabilities. Use odp_pktin_queue_param_init() to initialize
+ * configuration options into their default values. Default values are used
+ * when 'config' pointer is NULL.
+ *
+ * @param pktio    Packet IO handle
+ * @param config   Packet IO interface configuration. Uses defaults
+ *                 when NULL.
+ *
+ * @retval 0 on success
+ * @retval <0 on failure
+ */
+int odp_pktio_config(odp_pktio_t pktio, const odp_pktio_config_t *config);
+-----
+This API takes two argument, the PktIO handle, and a pointer to an
++odp_pktio_config_t+ struct that specifies the configuration requested for this
+PktIO:
+[source,c]
+-----
+/**
+ * Packet IO configuration options
+ *
+ * Packet IO interface level configuration options. Use odp_pktio_capability()
+ * to see which options are supported by the implementation.
+ * Use odp_pktio_config_init() to initialize the structure with default values.
+ */
+typedef struct odp_pktio_config_t {
+	/** Packet input configuration options bit field
+	 *
+	 *  Default value for all bits is zero. */
+	odp_pktin_config_opt_t pktin;
+
+	/** Packet output configuration options bit field
+	 *
+	 *  Default value for all bits is zero. */
+	odp_pktout_config_opt_t pktout;
+
+} odp_pktio_config_t;
+-----
+The +odp_pktio_config_t+ contains two sub-structs that identify the PktIn and
+PktOut configuration options requested for this PktIO.
+
+==== PktIn Configuration
+PktIn configurations are specified by the +odp_pktin_config_opt_t+ struct:
+[source,c]
+-----
+/**
+ * Packet input configuration options bit field
+ *
+ * Packet input configuration options listed in a bit field structure. Packet
+ * input timestamping may be enabled for all packets or at least for those that
+ * belong to time synchronization protocol (PTP).
+ *
+ * Packet input checksum checking may be enabled or disabled. When it is
+ * enabled, implementation will verify checksum correctness on incoming packets
+ * and depending on drop configuration either deliver erroneous packets with
+ * appropriate flags set (e.g. odp_packet_has_l3_error()) or drop those.
+ * When packet droping is enabled, application will never receive a packet
+ * with the specified error and may avoid to check the error flag.
+ */
+typedef union odp_pktin_config_opt_t {
+	/** Option flags */
+	struct {
+		/** Timestamp all packets on packet input */
+		uint64_t ts_all        : 1;
+
+		/** Timestamp (at least) IEEE1588 / PTP packets
+		  * on packet input */
+		uint64_t ts_ptp        : 1;
+
+		/** Check IPv4 header checksum on packet input */
+		uint64_t ipv4_chksum   : 1;
+
+		/** Check UDP checksum on packet input */
+		uint64_t udp_chksum    : 1;
+
+		/** Check TCP checksum on packet input */
+		uint64_t tcp_chksum    : 1;
+
+		/** Check SCTP checksum on packet input */
+		uint64_t sctp_chksum   : 1;
+
+		/** Drop packets with an IPv4 error on packet input */
+		uint64_t drop_ipv4_err : 1;
+
+		/** Drop packets with an IPv6 error on packet input */
+		uint64_t drop_ipv6_err : 1;
+
+		/** Drop packets with a UDP error on packet input */
+		uint64_t drop_udp_err  : 1;
+
+		/** Drop packets with a TCP error on packet input */
+		uint64_t drop_tcp_err  : 1;
+
+		/** Drop packets with a SCTP error on packet input */
+		uint64_t drop_sctp_err : 1;
+
+	} bit;
+
+	/** All bits of the bit field structure
+	  *
+	  * This field can be used to set/clear all flags, or bitwise
+	  * operations over the entire structure. */
+	uint64_t all_bits;
+} odp_pktin_config_opt_t;
+-----
+The PktIn options supported include whether input packets should be
+_timestamped_, and if so whether these should be applied to all packets or just
+IEEE 1588 PTP packets. Also specified is whether incoming packets should have
+their Layer 3 and/or Layer 4 checksums verified and whether packets failing
+checksum validation should be dropped.
+
+==== PktOut Configuration
+PktOut configurations are specified by the +odp_pktout_config_opt_t+ struct:
+[source,c]
+-----
+/**
+ * Packet output configuration options bit field
+ *
+ * Packet output configuration options listed in a bit field structure. Packet
+ * output checksum insertion may be enabled or disabled. When it is enabled,
+ * implementation will calculate and insert checksum into every outgoing packet
+ * by default. Application may use a packet metadata flag to disable checksum
+ * insertion per packet bases. For correct operation, packet metadata must
+ * provide valid offsets for the appropriate protocols. For example, UDP
+ * checksum calculation needs both L3 and L4 offsets (to access IP and UDP
+ * headers). When application (e.g. a switch) does not modify L3/L4 data and
+ * thus checksum does not need to be updated, output checksum insertion should
+ * be disabled for optimal performance.
+ */
+typedef union odp_pktout_config_opt_t {
+	/** Option flags */
+	struct {
+		/** Insert IPv4 header checksum on packet output */
+		uint64_t ipv4_chksum  : 1;
+
+		/** Insert UDP checksum on packet output */
+		uint64_t udp_chksum   : 1;
+
+		/** Insert TCP checksum on packet output */
+		uint64_t tcp_chksum   : 1;
+
+		/** Insert SCTP checksum on packet output */
+		uint64_t sctp_chksum  : 1;
+
+	} bit;
+
+	/** All bits of the bit field structure
+	  *
+	  * This field can be used to set/clear all flags, or bitwise
+	  * operations over the entire structure. */
+	uint64_t all_bits;
+} odp_pktout_config_opt_t;
+-----
+The PktOut options supported control whether outgoing packets should have
+Layer 3 and/or Layer 4 checksums added.