diff mbox

[PATCHv2] example: ODP classifier example

Message ID 1428512512-9216-1-git-send-email-bala.manoharan@linaro.org
State New
Headers show

Commit Message

Balasubramanian Manoharan April 8, 2015, 5:01 p.m. UTC
From: Balasubramanian Manoharan <bala.manoharan@linaro.org>

ODP Classifier example

This programs gets pmr rules as command-line parameter and configures the classification engine
in the system.

This initial version supports the following
* ODP_PMR_SIP_ADDR pmr term
* PMR term MATCH and RANGE type
* Multiple PMR rule can be set on a single pktio interface with different queues associated to each PMR rule
* Automatically configures a default queue and provides statistics for the same
* Prints statistics interms of the number of packets dispatched to each queue


Signed-off-by: Balasubramanian Manoharan <bala.manoharan@linaro.org>
---
v2: Incorporates review comments from Maxim
Resending V2 as there were some issue in linaro mail server

 configure.ac                        |   1 +
 example/Makefile.am                 |   2 +-
 example/classifier/Makefile.am      |  10 +
 example/classifier/odp_classifier.c | 789 ++++++++++++++++++++++++++++++++++++
 4 files changed, 801 insertions(+), 1 deletion(-)
 create mode 100644 example/classifier/Makefile.am
 create mode 100644 example/classifier/odp_classifier.c

Comments

Maxim Uvarov April 8, 2015, 5:13 p.m. UTC | #1
On 04/08/15 20:01, bala.manoharan@linaro.org wrote:
> +	/* Reserve memory for args from shared mem */
> +	shm = odp_shm_reserve("cls_shm_args", sizeof(appl_args_t),
> +			      ODP_CACHE_LINE_SIZE, 0);
> +	args = odp_shm_addr(shm);
> +
> +	if (args == NULL) {
> +		EXAMPLE_ERR("Error: shared mem alloc failed.\n");
> +		exit(EXIT_FAILURE);
> +	}
> +
That is not good I think:

void *odp_shm_addr(odp_shm_t shm)
{
     uint32_t i;

     i = from_handle(shm);

     if (i > (ODP_CONFIG_SHM_BLOCKS - 1))
         return NULL;

     return odp_shm_tbl->block[i].addr;
}

There is might be some block which was not freed it's addr.
You need to check shm == ODP_SHM_INVALID.

Thanks,
Maxim.
Mike Holmes April 8, 2015, 6:04 p.m. UTC | #2
when not run as root ./odp_classifier -i eth0

odp_shared_memory.c:265:odp_shm_reserve(): packet_pool:
No huge pages, fall back to normal pages,
check: /proc/sys/vm/nr_hugepages.
odp_packet_socket.c:438:mmap_pkt_socket():socket(SOCK_RAW): Operation not
permitted
odp_packet_socket.c:149:setup_pkt_sock():socket(): Operation not permitted
odp_packet_socket.c:149:setup_pkt_sock():socket(): Operation not permitted
odp_packet_io.c:273:setup_pktio_entry():Unable to init any I/O type.
odp_classifier.c:197:create_pktio(): pktio create failed for eth0
Aborted (core dumped)


Also ./odp_classifier  -h says

...
Mandatory OPTIONS:
  -i, --interface Eth interface
  -p, --policy <odp_pmr_term_e>:<match type>:<value1>:<value2>:<queue name>


Yet I did not pass -p so I assume it is not mandatory ?


On 8 April 2015 at 13:13, Maxim Uvarov <maxim.uvarov@linaro.org> wrote:

> On 04/08/15 20:01, bala.manoharan@linaro.org wrote:
>
>> +       /* Reserve memory for args from shared mem */
>> +       shm = odp_shm_reserve("cls_shm_args", sizeof(appl_args_t),
>> +                             ODP_CACHE_LINE_SIZE, 0);
>> +       args = odp_shm_addr(shm);
>> +
>> +       if (args == NULL) {
>> +               EXAMPLE_ERR("Error: shared mem alloc failed.\n");
>> +               exit(EXIT_FAILURE);
>> +       }
>> +
>>
> That is not good I think:
>
> void *odp_shm_addr(odp_shm_t shm)
> {
>     uint32_t i;
>
>     i = from_handle(shm);
>
>     if (i > (ODP_CONFIG_SHM_BLOCKS - 1))
>         return NULL;
>
>     return odp_shm_tbl->block[i].addr;
> }
>
> There is might be some block which was not freed it's addr.
> You need to check shm == ODP_SHM_INVALID.
>
> Thanks,
> Maxim.
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>
Balasubramanian Manoharan April 9, 2015, 1:12 p.m. UTC | #3
On Wednesday 08 April 2015 11:34 PM, Mike Holmes wrote:
> when not run as root ./odp_classifier -i eth0
>
> odp_shared_memory.c:265:odp_shm_reserve(): packet_pool:
> No huge pages, fall back to normal pages,
> check: /proc/sys/vm/nr_hugepages.
> odp_packet_socket.c:438:mmap_pkt_socket():socket(SOCK_RAW): Operation not
> permitted
> odp_packet_socket.c:149:setup_pkt_sock():socket(): Operation not permitted
> odp_packet_socket.c:149:setup_pkt_sock():socket(): Operation not permitted
> odp_packet_io.c:273:setup_pktio_entry():Unable to init any I/O type.
> odp_classifier.c:197:create_pktio(): pktio create failed for eth0
> Aborted (core dumped)
>
>
> Also ./odp_classifier  -h says
>
> ...
> Mandatory OPTIONS:
>    -i, --interface Eth interface
>    -p, --policy <odp_pmr_term_e>:<match type>:<value1>:<value2>:<queue name>
>
>
> Yet I did not pass -p so I assume it is not mandatory ?

This is an issue and I will correct the same in next version. -p is a 
mandatory option.

Regards,
Bala
>
>
> On 8 April 2015 at 13:13, Maxim Uvarov <maxim.uvarov@linaro.org> wrote:
>
>> On 04/08/15 20:01, bala.manoharan@linaro.org wrote:
>>
>>> +       /* Reserve memory for args from shared mem */
>>> +       shm = odp_shm_reserve("cls_shm_args", sizeof(appl_args_t),
>>> +                             ODP_CACHE_LINE_SIZE, 0);
>>> +       args = odp_shm_addr(shm);
>>> +
>>> +       if (args == NULL) {
>>> +               EXAMPLE_ERR("Error: shared mem alloc failed.\n");
>>> +               exit(EXIT_FAILURE);
>>> +       }
>>> +
>>>
>> That is not good I think:
>>
>> void *odp_shm_addr(odp_shm_t shm)
>> {
>>      uint32_t i;
>>
>>      i = from_handle(shm);
>>
>>      if (i > (ODP_CONFIG_SHM_BLOCKS - 1))
>>          return NULL;
>>
>>      return odp_shm_tbl->block[i].addr;
>> }
>>
>> There is might be some block which was not freed it's addr.
>> You need to check shm == ODP_SHM_INVALID.
>>
>> Thanks,
>> Maxim.
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>
>
>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
Balasubramanian Manoharan April 9, 2015, 1:13 p.m. UTC | #4
On Wednesday 08 April 2015 10:43 PM, Maxim Uvarov wrote:
> On 04/08/15 20:01, bala.manoharan@linaro.org wrote:
>> +    /* Reserve memory for args from shared mem */
>> +    shm = odp_shm_reserve("cls_shm_args", sizeof(appl_args_t),
>> +                  ODP_CACHE_LINE_SIZE, 0);
>> +    args = odp_shm_addr(shm);
>> +
>> +    if (args == NULL) {
>> +        EXAMPLE_ERR("Error: shared mem alloc failed.\n");
>> +        exit(EXIT_FAILURE);
>> +    }
>> +
> That is not good I think:
>
> void *odp_shm_addr(odp_shm_t shm)
> {
>     uint32_t i;
>
>     i = from_handle(shm);
>
>     if (i > (ODP_CONFIG_SHM_BLOCKS - 1))
>         return NULL;
>
>     return odp_shm_tbl->block[i].addr;
> }
>
> There is might be some block which was not freed it's addr.
> You need to check shm == ODP_SHM_INVALID.

Agreed. Will correct in next version

Regards,
Bala
>
> Thanks,
> Maxim.
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
diff mbox

Patch

diff --git a/configure.ac b/configure.ac
index 57054c5..d8d8ca5 100644
--- a/configure.ac
+++ b/configure.ac
@@ -250,6 +250,7 @@  AM_CXXFLAGS="-std=c++11"
 AC_CONFIG_FILES([Makefile
 		 doc/Makefile
 		 example/Makefile
+		 example/classifier/Makefile
 		 example/generator/Makefile
 		 example/ipsec/Makefile
 		 example/l2fwd/Makefile
diff --git a/example/Makefile.am b/example/Makefile.am
index 3021571..33b60c1 100644
--- a/example/Makefile.am
+++ b/example/Makefile.am
@@ -1 +1 @@ 
-SUBDIRS = generator ipsec l2fwd packet timer
+SUBDIRS = classifier generator ipsec l2fwd packet timer
diff --git a/example/classifier/Makefile.am b/example/classifier/Makefile.am
new file mode 100644
index 0000000..938f094
--- /dev/null
+++ b/example/classifier/Makefile.am
@@ -0,0 +1,10 @@ 
+include $(top_srcdir)/example/Makefile.inc
+
+bin_PROGRAMS = odp_classifier
+odp_classifier_LDFLAGS = $(AM_LDFLAGS) -static
+odp_classifier_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example
+
+noinst_HEADERS = \
+		  $(top_srcdir)/example/example_debug.h
+
+dist_odp_classifier_SOURCES = odp_classifier.c
diff --git a/example/classifier/odp_classifier.c b/example/classifier/odp_classifier.c
new file mode 100644
index 0000000..f3dcbbb
--- /dev/null
+++ b/example/classifier/odp_classifier.c
@@ -0,0 +1,789 @@ 
+/* Copyright (c) 2015, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier:     BSD-3-Clause
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <getopt.h>
+#include <unistd.h>
+#include <example_debug.h>
+
+#include <odp.h>
+#include <odp/helper/linux.h>
+#include <odp/helper/eth.h>
+#include <odp/helper/ip.h>
+#include <strings.h>
+#include <stdio.h>
+
+/** @def MAX_WORKERS
+ * @brief Maximum number of worker threads
+ */
+#define MAX_WORKERS            32
+
+/** @def SHM_PKT_POOL_SIZE
+ * @brief Size of the shared memory block
+ */
+#define SHM_PKT_POOL_SIZE      (512*2048)
+
+/** @def SHM_PKT_POOL_BUF_SIZE
+ * @brief Buffer size of the packet pool buffer
+ */
+#define SHM_PKT_POOL_BUF_SIZE  1856
+
+/** @def MAX_PMR_COUNT
+ * @brief Maximum number of Classification Policy
+ */
+#define MAX_PMR_COUNT	8
+
+/** @def DISPLAY_STRING_LEN
+ * @brief Length of string used to display term value
+ */
+#define DISPLAY_STRING_LEN	32
+
+/** Get rid of path in filename - only for unix-type paths using '/' */
+#define NO_PATH(file_name) (strrchr((file_name), '/') ? \
+		strrchr((file_name), '/') + 1 : (file_name))
+
+typedef struct {
+	odp_queue_t queue;	/**< Associated queue handle */
+	odp_cos_t cos;		/**< Associated cos handle */
+	odp_pmr_t pmr;		/**< Associated pmr handle */
+	odp_atomic_u64_t packet_count;	/**< count of received packets */
+	odp_pmr_term_e term;		/**< odp pmr term value */
+	char queue_name[ODP_QUEUE_NAME_LEN];	/**< queue name */
+	odp_pmr_match_type_e match_type;	/**< pmr match type */
+	int val_sz;	/**< size of the pmr term */
+	union {
+		struct {
+			uint32_t val;	/**< pmr term value */
+			uint32_t mask;	/**< pmr term mask */
+		} match;
+		struct  {
+			uint32_t val1;	/**< pmr term start range */
+			uint32_t val2;	/**< pmr term end range */
+		} range;
+	};
+	char value1[DISPLAY_STRING_LEN];	/**< Display string1 */
+	char value2[DISPLAY_STRING_LEN];	/**< Display string2 */
+} global_statistics;
+
+typedef struct {
+	global_statistics stats[MAX_PMR_COUNT];
+	int policy_count;	/**< global policy count */
+	int appl_mode;		/**< application mode */
+	odp_atomic_u64_t total_packets;	/**< total received packets */
+	int cpu_count;		/**< Number of CPUs to use */
+	char *if_name;		/**< pointer to interface names */
+} appl_args_t;
+
+enum packet_mode {
+	APPL_MODE_DROP,		/**< Packet is dropped */
+	APPL_MODE_REPLY		/**< Packet is sent back */
+};
+
+/* helper funcs */
+static int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned len);
+static void swap_pkt_addrs(odp_packet_t pkt_tbl[], unsigned len);
+static void parse_args(int argc, char *argv[], appl_args_t *appl_args);
+static void print_info(char *progname, appl_args_t *appl_args);
+static void usage(char *progname);
+static void configure_cos_queue(odp_pktio_t pktio, appl_args_t *args);
+static void configure_default_queue(odp_pktio_t pktio, appl_args_t *args);
+static int convert_str_to_pmr_enum(char *token, odp_pmr_term_e *term);
+static int parse_pmr_policy(appl_args_t *appl_args, char *argv[], char *optarg);
+
+static inline
+void print_cls_statistics(appl_args_t *args)
+{
+	int i;
+	printf("\n");
+	for (i = 0; i < 40; i++)
+		printf("-");
+	printf("\n");
+	/* print statistics */
+	printf("CLASSIFIER EXAMPLE STATISTICS\n");
+	for (i = 0; i < 40; i++)
+		printf("-");
+	printf("\n");
+	printf("CONFIGURATION\n");
+	printf("\n");
+	printf("QUEUE\tMATCH\tVALUE1\t\tVALUE2\n");
+	for (i = 0; i < 40; i++)
+		printf("-");
+	printf("\n");
+	for (i = 0; i < args->policy_count - 1; i++) {
+		printf("%s\t", args->stats[i].queue_name);
+		if (args->stats[i].match_type == ODP_PMR_MASK)
+			printf("MATCH\t");
+		else
+			printf("RANGE\t");
+		printf("%s\t", args->stats[i].value1);
+		printf("%s\n", args->stats[i].value2);
+	}
+	printf("\n");
+	printf("RECEIVED PACKETS\n");
+	for (i = 0; i < 40; i++)
+		printf("-");
+	printf("\n");
+	for (i = 0; i < args->policy_count; i++)
+		printf("%s\t", args->stats[i].queue_name);
+	printf("Total Packets");
+	printf("\n");
+	do {
+		for (i = 0; i < args->policy_count; i++)
+			printf("%"PRIu64"\t",
+			       odp_atomic_load_u64(&args->stats[i]
+						   .packet_count));
+
+		printf("\t%"PRIu64"\t", odp_atomic_load_u64(&args->
+							    total_packets));
+
+		sleep(1);
+		printf("\r");
+		fflush(stdout);
+	} while (1);
+}
+
+static inline
+int parse_ipv4_addr(const char *ipaddress, uint32_t *addr)
+{
+	int b[4];
+	int converted;
+
+	converted = sscanf(ipaddress, "%d.%d.%d.%d",
+			&b[3], &b[2], &b[1], &b[0]);
+	if (4 != converted)
+		return -1;
+
+	if ((b[0] > 255) || (b[1] > 255) || (b[2] > 255) || (b[3] > 255))
+		return -1;
+
+	*addr = b[0] | b[1] << 8 | b[2] << 16 | b[3] << 24;
+
+	return 0;
+}
+
+static inline
+int parse_ipv4_mask(const char *str, uint32_t *mask)
+{
+	uint32_t b;
+	sscanf(str, "%x", &b);
+	*mask = b;
+	return 0;
+}
+
+/**
+ * Create a pktio handle, optionally associating a default input queue.
+ *
+ * @param dev Device name
+ * @param pool Associated Packet Pool
+ *
+ * @return The handle of the created pktio object.
+ * @retval ODP_PKTIO_INVALID if the create fails.
+ */
+static odp_pktio_t create_pktio(const char *dev, odp_pool_t pool)
+{
+	odp_pktio_t pktio;
+	odp_queue_t inq_def;
+	odp_queue_param_t qparam;
+	char inq_name[ODP_QUEUE_NAME_LEN];
+	int ret;
+
+	/* Open a packet IO instance */
+	pktio = odp_pktio_open(dev, pool);
+	if (pktio == ODP_PKTIO_INVALID)
+		EXAMPLE_ABORT("pktio create failed for %s\n", dev);
+
+	qparam.sched.prio  = ODP_SCHED_PRIO_DEFAULT;
+	qparam.sched.sync  = ODP_SCHED_SYNC_ATOMIC;
+	qparam.sched.group = ODP_SCHED_GROUP_DEFAULT;
+	snprintf(inq_name, sizeof(inq_name), "%" PRIu64 "-pktio_inq_def",
+		 odp_pktio_to_u64(pktio));
+	inq_name[ODP_QUEUE_NAME_LEN - 1] = '\0';
+
+	inq_def = odp_queue_create(inq_name, ODP_QUEUE_TYPE_PKTIN, &qparam);
+	if (inq_def == ODP_QUEUE_INVALID)
+		EXAMPLE_ABORT("pktio inq create failed for %s\n", dev);
+
+	ret = odp_pktio_inq_setdef(pktio, inq_def);
+	if (ret != 0)
+		EXAMPLE_ABORT("default input-Q setup for %s\n", dev);
+
+	printf("  created pktio:%02" PRIu64
+			", dev:%s, queue mode (ATOMIC queues)\n"
+			"  \tdefault pktio%02" PRIu64
+			"-INPUT queue:%" PRIu64 "\n",
+			odp_pktio_to_u64(pktio), dev,
+			odp_pktio_to_u64(pktio), odp_queue_to_u64(inq_def));
+
+	return pktio;
+}
+
+/**
+ * Worker threads to receive the packet
+ *
+ */
+static void *pktio_receive_thread(void *arg)
+{
+	int thr;
+	odp_queue_t outq_def;
+	odp_packet_t pkt;
+	odp_event_t ev;
+	unsigned long err_cnt = 0;
+	odp_queue_t queue;
+	int i;
+	thr = odp_thread_id();
+	appl_args_t *appl = (appl_args_t *)arg;
+	global_statistics *stats;
+
+
+	/* Init this thread */
+	if (odp_init_local())
+		EXAMPLE_ABORT("ODP thread local init failed.\n");
+
+	/* Loop packets */
+	for (;;) {
+		odp_pktio_t pktio_tmp;
+
+		/* Use schedule to get buf from any input queue */
+		ev = odp_schedule(&queue, ODP_SCHED_WAIT);
+
+		/* Loop back to receive packets incase of invalid event */
+		if (odp_unlikely(ev == ODP_EVENT_INVALID))
+			continue;
+
+		pkt = odp_packet_from_event(ev);
+
+		/* Total packets received */
+		odp_atomic_inc_u64(&appl->total_packets);
+
+		/* Drop packets with errors */
+		if (odp_unlikely(drop_err_pkts(&pkt, 1) == 0)) {
+			EXAMPLE_ERR("Drop frame - err_cnt:%lu\n", ++err_cnt);
+			continue;
+		}
+
+		pktio_tmp = odp_packet_input(pkt);
+		outq_def = odp_pktio_outq_getdef(pktio_tmp);
+
+		if (outq_def == ODP_QUEUE_INVALID) {
+			EXAMPLE_ERR("  [%02i] Error: def output-Q query\n",
+				    thr);
+			return NULL;
+		}
+
+		/* Swap Eth MACs and possibly IP-addrs before sending back */
+		swap_pkt_addrs(&pkt, 1);
+
+		for (i = 0; i <  MAX_PMR_COUNT; i++) {
+			stats = &appl->stats[i];
+			if (queue == stats->queue)
+				odp_atomic_inc_u64(&stats->packet_count);
+		}
+
+		if (appl->appl_mode == APPL_MODE_DROP)
+			odp_packet_free(pkt);
+		else
+			odp_queue_enq(outq_def, ev);
+	}
+
+	return NULL;
+}
+
+static void configure_default_queue(odp_pktio_t pktio, appl_args_t *args)
+{
+	odp_queue_param_t qparam;
+	odp_cos_t cos_default;
+	char cos_name[ODP_COS_NAME_LEN];
+	char queue_name[ODP_QUEUE_NAME_LEN];
+	odp_queue_t queue_default;
+	global_statistics *stats = args->stats;
+	sprintf(cos_name, "Default%s", args->if_name);
+	cos_default = odp_cos_create(cos_name);
+
+	qparam.sched.prio = ODP_SCHED_PRIO_DEFAULT;
+	qparam.sched.sync = ODP_SCHED_SYNC_NONE;
+	qparam.sched.group = ODP_SCHED_GROUP_ALL;
+	sprintf(queue_name, "%s", "DefaultQueue");
+	queue_default = odp_queue_create(queue_name,
+			ODP_QUEUE_TYPE_SCHED, &qparam);
+
+	odp_cos_set_queue(cos_default, queue_default);
+	odp_pktio_default_cos_set(pktio, cos_default);
+	stats[args->policy_count].cos = cos_default;
+	/* add default queue to global stats */
+	stats[args->policy_count].queue = queue_default;
+	strcpy(stats[args->policy_count].queue_name, "DefaultQueue");
+	odp_atomic_init_u64(&stats[args->policy_count].packet_count, 0);
+	args->policy_count++;
+}
+
+static void configure_cos_queue(odp_pktio_t pktio, appl_args_t *args)
+{
+	char cos_name[ODP_COS_NAME_LEN];
+	char queue_name[ODP_QUEUE_NAME_LEN];
+	int i;
+	global_statistics *stats;
+	odp_queue_param_t qparam;
+
+	for (i = 0; i < args->policy_count; i++) {
+		stats = &args->stats[i];
+		sprintf(cos_name, "CoS%s", stats->queue_name);
+		stats->cos = odp_cos_create(cos_name);
+
+		if (stats->match_type == ODP_PMR_MASK) {
+			stats->pmr = odp_pmr_create_match(stats->term,
+					&stats->match.val,
+					&stats->match.mask,
+					stats->val_sz);
+		} else {
+			stats->pmr = odp_pmr_create_range(stats->term,
+					&stats->range.val1,
+					&stats->range.val2,
+					stats->val_sz);
+		}
+		qparam.sched.prio = i % odp_schedule_num_prio();
+		qparam.sched.sync = ODP_SCHED_SYNC_NONE;
+		qparam.sched.group = ODP_SCHED_GROUP_ALL;
+
+		sprintf(queue_name, "%s%d", args->stats[i].queue_name, i);
+		stats->queue = odp_queue_create(queue_name,
+						 ODP_QUEUE_TYPE_SCHED,
+						 &qparam);
+		odp_cos_set_queue(stats->cos, stats->queue);
+		odp_pktio_pmr_cos(stats->pmr, pktio, stats->cos);
+
+		odp_atomic_init_u64(&stats->packet_count, 0);
+	}
+}
+
+/**
+ * ODP Classifier example main function
+ */
+int main(int argc, char *argv[])
+{
+	odph_linux_pthread_t thread_tbl[MAX_WORKERS];
+	odp_pool_t pool;
+	int num_workers;
+	int i;
+	int cpu;
+	odp_cpumask_t cpumask;
+	char cpumaskstr[ODP_CPUMASK_STR_SIZE];
+	odp_pool_param_t params;
+	odp_pktio_t pktio;
+	appl_args_t *args;
+	odp_shm_t shm;
+
+	/* Init ODP before calling anything else */
+	if (odp_init_global(NULL, NULL)) {
+		EXAMPLE_ERR("Error: ODP global init failed.\n");
+		exit(EXIT_FAILURE);
+	}
+
+	/* Init this thread */
+	if (odp_init_local()) {
+		EXAMPLE_ERR("Error: ODP local init failed.\n");
+		exit(EXIT_FAILURE);
+	}
+
+	/* Reserve memory for args from shared mem */
+	shm = odp_shm_reserve("cls_shm_args", sizeof(appl_args_t),
+			      ODP_CACHE_LINE_SIZE, 0);
+	args = odp_shm_addr(shm);
+
+	if (args == NULL) {
+		EXAMPLE_ERR("Error: shared mem alloc failed.\n");
+		exit(EXIT_FAILURE);
+	}
+
+	memset(args, 0, sizeof(*args));
+	/* Parse and store the application arguments */
+	parse_args(argc, argv, args);
+
+	/* Print both system and application information */
+	print_info(NO_PATH(argv[0]), args);
+
+	/* Default to system CPU count unless user specified */
+	num_workers = MAX_WORKERS;
+	if (args->cpu_count)
+		num_workers = args->cpu_count;
+
+	/*
+	 * By default CPU #0 runs Linux kernel background tasks.
+	 * Start mapping thread from CPU #1
+	 */
+	num_workers = odph_linux_cpumask_default(&cpumask, num_workers);
+	(void)odp_cpumask_to_str(&cpumask, cpumaskstr, sizeof(cpumaskstr));
+
+	printf("num worker threads: %i\n", num_workers);
+	printf("first CPU:          %i\n", odp_cpumask_first(&cpumask));
+	printf("cpu mask:           %s\n", cpumaskstr);
+
+	/* Create packet pool */
+	memset(&params, 0, sizeof(params));
+	params.pkt.seg_len = SHM_PKT_POOL_BUF_SIZE;
+	params.pkt.len     = SHM_PKT_POOL_BUF_SIZE;
+	params.pkt.num     = SHM_PKT_POOL_SIZE/SHM_PKT_POOL_BUF_SIZE;
+	params.type        = ODP_POOL_PACKET;
+
+	pool = odp_pool_create("packet_pool", ODP_SHM_NULL, &params);
+
+	if (pool == ODP_POOL_INVALID) {
+		EXAMPLE_ERR("Error: packet pool create failed.\n");
+		exit(EXIT_FAILURE);
+	}
+	/* odp_pool_print(pool); */
+	odp_atomic_init_u64(&args->total_packets, 0);
+
+	/* create pktio per interface */
+	pktio = create_pktio(args->if_name, pool);
+
+	configure_cos_queue(pktio, args);
+
+	/* configure default Cos and default queue */
+	configure_default_queue(pktio, args);
+
+	/* Create and init worker threads */
+	memset(thread_tbl, 0, sizeof(thread_tbl));
+
+	cpu = odp_cpumask_first(&cpumask);
+	for (i = 0; i < num_workers; ++i) {
+		odp_cpumask_t thd_mask;
+		/*
+		 * Calls odp_thread_create(cpu) for each thread
+		 */
+		odp_cpumask_zero(&thd_mask);
+		odp_cpumask_set(&thd_mask, cpu);
+		odph_linux_pthread_create(&thread_tbl[i], &thd_mask,
+					  pktio_receive_thread,
+					  args);
+		cpu = odp_cpumask_next(&cpumask, cpu);
+	}
+
+	print_cls_statistics(args);
+
+	/* Master thread wait*/
+	odph_linux_pthread_join(thread_tbl, num_workers);
+
+	for (i = 0; i < args->policy_count; i++) {
+		odp_cos_destroy(args->stats[i].cos);
+		odp_queue_destroy(args->stats[i].queue);
+	}
+
+	free(args->if_name);
+	odp_shm_free(shm);
+	printf("Exit\n\n");
+
+	return 0;
+}
+
+/**
+ * Drop packets which input parsing marked as containing errors.
+ *
+ * Frees packets with error and modifies pkt_tbl[] to only contain packets with
+ * no detected errors.
+ *
+ * @param pkt_tbl  Array of packet
+ * @param len      Length of pkt_tbl[]
+ *
+ * @return Number of packets with no detected error
+ */
+static int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned len)
+{
+	odp_packet_t pkt;
+	unsigned pkt_cnt = len;
+	unsigned i, j;
+
+	for (i = 0, j = 0; i < len; ++i) {
+		pkt = pkt_tbl[i];
+
+		if (odp_unlikely(odp_packet_has_error(pkt))) {
+			odp_packet_free(pkt); /* Drop */
+			pkt_cnt--;
+		} else if (odp_unlikely(i != j++)) {
+			pkt_tbl[j-1] = pkt;
+		}
+	}
+
+	return pkt_cnt;
+}
+
+/**
+ * Swap eth src<->dst and IP src<->dst addresses
+ *
+ * @param pkt_tbl  Array of packets
+ * @param len      Length of pkt_tbl[]
+ */
+static void swap_pkt_addrs(odp_packet_t pkt_tbl[], unsigned len)
+{
+	odp_packet_t pkt;
+	odph_ethhdr_t *eth;
+	odph_ethaddr_t tmp_addr;
+	odph_ipv4hdr_t *ip;
+	uint32be_t ip_tmp_addr; /* tmp ip addr */
+	unsigned i;
+
+	for (i = 0; i < len; ++i) {
+		pkt = pkt_tbl[i];
+		if (odp_packet_has_eth(pkt)) {
+			eth = (odph_ethhdr_t *)odp_packet_l2_ptr(pkt, NULL);
+
+			tmp_addr = eth->dst;
+			eth->dst = eth->src;
+			eth->src = tmp_addr;
+
+			if (odp_packet_has_ipv4(pkt)) {
+				/* IPv4 */
+				ip = (odph_ipv4hdr_t *)
+					odp_packet_l3_ptr(pkt, NULL);
+
+				ip_tmp_addr  = ip->src_addr;
+				ip->src_addr = ip->dst_addr;
+				ip->dst_addr = ip_tmp_addr;
+			}
+		}
+	}
+}
+
+static int convert_str_to_pmr_enum(char *token, odp_pmr_term_e *term)
+{
+	if (0 == strcasecmp(token, "ODP_PMR_SIP_ADDR")) {
+		*term = ODP_PMR_SIP_ADDR;
+		return 0;
+	}
+	return -1;
+}
+
+
+static int parse_pmr_policy(appl_args_t *appl_args, char *argv[], char *optarg)
+{
+	int policy_count;
+	char *token;
+	size_t len;
+	odp_pmr_term_e term;
+	global_statistics *stats;
+	char *pmr_str;
+
+	policy_count = appl_args->policy_count;
+	stats = appl_args->stats;
+
+	/* last array index is needed for default queue */
+	if (policy_count >= MAX_PMR_COUNT - 1) {
+		EXAMPLE_ERR("Maximum allowed PMR reached\n");
+		return -1;
+	}
+
+	len = strlen(optarg);
+	len++;
+	pmr_str = malloc(len);
+	strcpy(pmr_str, optarg);
+
+	/* PMR TERM */
+	token = strtok(pmr_str, ":");
+	if (convert_str_to_pmr_enum(token, &term))
+		EXAMPLE_ABORT("Invalid ODP_PMR_TERM string");
+	stats[policy_count].term = term;
+	/* PMR RANGE vs MATCH */
+	token = strtok(NULL, ":");
+	if (0 == strcasecmp(token, "range")) {
+		stats[policy_count].match_type = ODP_PMR_RANGE;
+	} else if (0 == strcasecmp(token, "match")) {
+		stats[policy_count].match_type = ODP_PMR_MASK;
+	} else {
+		usage(argv[0]);
+		exit(EXIT_FAILURE);
+	}
+
+	/* PMR value */
+	switch (term)	{
+	case ODP_PMR_SIP_ADDR:
+		if (stats[policy_count].match_type == ODP_PMR_MASK) {
+			token = strtok(NULL, ":");
+			strcpy(stats[policy_count].value1, token);
+			parse_ipv4_addr(token, &stats[policy_count].match.val);
+			token = strtok(NULL, ":");
+			strcpy(stats[policy_count].value2, token);
+			parse_ipv4_mask(token, &stats[policy_count].match.mask);
+			stats[policy_count].val_sz = 4;
+		} else {
+			token = strtok(NULL, ":");
+			strcpy(stats[policy_count].value1,
+			       token);
+			parse_ipv4_addr(token, &stats[policy_count].range.val1);
+			token = strtok(NULL, ":");
+			strcpy(stats[policy_count].value2, token);
+			parse_ipv4_addr(token, &stats[policy_count].range.val2);
+			stats[policy_count].val_sz = 4;
+		}
+	break;
+	default:
+		usage(argv[0]);
+		EXAMPLE_ABORT("PMR term not supported");
+	}
+
+	/* Queue Name */
+	token = strtok(NULL, ":");
+	strcpy(stats[policy_count].queue_name, token);
+	appl_args->policy_count++;
+	free(pmr_str);
+	return 0;
+}
+
+/**
+ * Parse and store the command line arguments
+ *
+ * @param argc       argument count
+ * @param argv[]     argument vector
+ * @param appl_args  Store application arguments here
+ */
+static void parse_args(int argc, char *argv[], appl_args_t *appl_args)
+{
+	int opt;
+	int long_index;
+	size_t len;
+	int i;
+
+	static struct option longopts[] = {
+		{"count", required_argument, NULL, 'c'},
+		{"interface", required_argument, NULL, 'i'},	/* return 'i' */
+		{"policy", required_argument, NULL, 'p'},	/* return 'p' */
+		{"mode", required_argument, NULL, 'm'},		/* return 'm' */
+		{"help", no_argument, NULL, 'h'},		/* return 'h' */
+		{NULL, 0, NULL, 0}
+	};
+
+
+	while (1) {
+		opt = getopt_long(argc, argv, "+c:i:p:m:t:h",
+				longopts, &long_index);
+
+		if (opt == -1)
+			break;	/* No more options */
+
+		switch (opt) {
+		case 'c':
+			appl_args->cpu_count = atoi(optarg);
+			break;
+		case 'p':
+			if (0 > parse_pmr_policy(appl_args, argv, optarg))
+				continue;
+			break;
+		case 'i':
+			len = strlen(optarg);
+			if (len == 0) {
+				usage(argv[0]);
+				exit(EXIT_FAILURE);
+			}
+			len += 1;	/* add room for '\0' */
+
+			appl_args->if_name = malloc(len);
+			if (appl_args->if_name == NULL) {
+				usage(argv[0]);
+				exit(EXIT_FAILURE);
+			}
+
+			strcpy(appl_args->if_name, optarg);
+			break;
+
+		case 'h':
+			usage(argv[0]);
+			exit(EXIT_SUCCESS);
+			break;
+		case 'm':
+			i = atoi(optarg);
+			if (i == 0)
+				appl_args->appl_mode = APPL_MODE_DROP;
+			else
+				appl_args->appl_mode = APPL_MODE_REPLY;
+			break;
+
+		default:
+			break;
+		}
+	}
+
+	if (appl_args->if_name == NULL) {
+		usage(argv[0]);
+		exit(EXIT_FAILURE);
+	}
+
+	optind = 1;		/* reset 'extern optind' from the getopt lib */
+}
+
+/**
+ * Print system and application info
+ */
+static void print_info(char *progname, appl_args_t *appl_args)
+{
+	printf("\n"
+			"ODP system info\n"
+			"---------------\n"
+			"ODP API version: %s\n"
+			"CPU model:       %s\n"
+			"CPU freq (hz):   %"PRIu64"\n"
+			"Cache line size: %i\n"
+			"CPU count:       %i\n"
+			"\n",
+			odp_version_api_str(), odp_sys_cpu_model_str(),
+			odp_sys_cpu_hz(), odp_sys_cache_line_size(),
+			odp_cpu_count());
+
+	printf("Running ODP appl: \"%s\"\n"
+			"-----------------\n"
+			"Using IF:%s      ",
+			progname, appl_args->if_name);
+	printf("\n\n");
+	fflush(NULL);
+}
+
+/**
+ * Prinf usage information
+ */
+static void usage(char *progname)
+{
+	printf("\n"
+			"OpenDataPlane Classifier example.\n"
+			"Usage: %s OPTIONS\n"
+			"  E.g. %s -i eth1 -m 0 -p \"ODP_PMR_SIP_ADDR:match:10.10.10.5:FFFFFFFF:queue1\" \\\n"
+			"\t\t\t-p \"ODP_PMR_SIP_ADDR:MATCH:10.10.10.6:FFFFFFFF:queue2\" \\\n"
+			"\t\t\t-p \"ODP_PMR_SIP_ADDR:MATCH:10.10.10.7:000000FF:queue3\" \\\n"
+			"\t\t\t-p \"ODP_PMR_SIP_ADDR:RANGE:10.10.10.10:10.10.10.20:queue3\"\n"
+			"\n"
+			"For the above example configuration the following will be the packet distribution\n"
+			"queue1\t\tPackets with source ip address 10.10.10.5\n"
+			"queue2\t\tPackets with source ip address whose last 8 bits match 7\n"
+			"queue3\t\tPackets with source ip address in the range 10.10.10.10 to 10.10.10.20\n"
+			"\n"
+			"Mandatory OPTIONS:\n"
+			"  -i, --interface Eth interface\n"
+			"  -p, --policy <odp_pmr_term_e>:<match type>:<value1>:<value2>:<queue name>\n"
+			"\n"
+			"<odp_pmr_term_e>	Packet Matching Rule defined with odp_pmr_term_e "
+			"for the policy\n"
+			"\n"
+			"<match type>		PMR Match type.\n"
+			"			MATCH: PMR rule type MATCH\n"
+			"			RANGE: PMR rule type RANCE\n"
+			"\n"
+			"<value1>		PMR value1.\n"
+			"			If match type is MATCH is the the matching value.\n"
+			"			If match type is RANGE it is start range.\n"
+			"\n"
+			"<value2>		PMR value2.\n"
+			"			If match type is \"MATCH\" it is the MASK value\n"
+			"			If match type is \"RANCE\" it is end range.\n"
+			"\n"
+			"Optional OPTIONS\n"
+			"  -c, --count <number> CPU count.\n"
+			"                       default: CPU core count.\n"
+			"\n"
+			"  -m, --mode		0: Packet Drop mode. Received packets will be dropped\n"
+			"			!0: Packet ICMP mode. Received packets will be sent back\n"
+			"                       default: Packet Drop mode\n"
+			"\n"
+			"  -h, --help		Display help and exit.\n"
+			"\n", NO_PATH(progname), NO_PATH(progname)
+	      );
+}