[v2] example: introducing l3fwd

Message ID 1464069523-3668-1-git-send-email-forrest.shi@linaro.org
State New
Headers show

Commit Message

Forrest Shi May 24, 2016, 5:58 a.m.
multi-thread and bi-directional forwarding.
support at most cpu-1 forward threads.
each thread handles traffic of one port,
and each port is handled by only one thread.

Signed-off-by: Xuelin Shi <forrest.shi@linaro.org>
---
change history:
 v2: merge v1 patch set into one patch 

 example/Makefile.am          |   2 +-
 example/l3fwd/Makefile.am    |  11 +
 example/l3fwd/odp_l3fwd.c    | 473 +++++++++++++++++++++++++++++++++++++++++++
 example/l3fwd/odp_l3fwd_db.c | 414 +++++++++++++++++++++++++++++++++++++
 example/l3fwd/odp_l3fwd_db.h | 137 +++++++++++++
 example/m4/configure.m4      |   1 +
 6 files changed, 1037 insertions(+), 1 deletion(-)
 create mode 100644 example/l3fwd/Makefile.am
 create mode 100644 example/l3fwd/odp_l3fwd.c
 create mode 100644 example/l3fwd/odp_l3fwd_db.c
 create mode 100644 example/l3fwd/odp_l3fwd_db.h

Comments

Elo, Matias (Nokia - FI/Espoo) May 30, 2016, 12:42 p.m. | #1
Hi Forrest,

Checkpatch throws one error from the patch:

	ERROR: spaces required around that '==' (ctx:VxW)
	#485: FILE: example/l3fwd/odp_l3fwd.c:427:
	+	if (args->route_str[0]== NULL) {
 
You could add a test script for running the application during make check (for example in the same manner as the l2fwd_simple or switch). This can be done in a separate patch. MQ support would also be a nice addition later on.

Further comments below.

-Matias


> +
> +			if (odp_unlikely(!odp_packet_has_ipv4(pkt))) {
> +				printf("warning: packet has no ipv4 header\n");
> +				return NULL;
> +			}

I would only drop the non-IP packets, not kill the whole thread, as there is no input filtering in odp_pktin_recv().

> +
> +			/*TODO: ipv6 need to be done */

This can be removed.

> +			if (odp_packet_has_udp(pkt) ||
> +			    odp_packet_has_tcp(pkt)) {
> +				/* UDP or TCP*/
> +				void *ptr = odp_packet_l4_ptr(pkt, NULL);
> +
> +				udp = (odph_udphdr_t *)ptr;
> +				key.src_port = odp_be_to_cpu_16(udp-
> >src_port);
> +				key.dst_port = odp_be_to_cpu_16(udp-
> >dst_port);
> +			}

key.src_port and key.dst_port are uninitialized for other than TCP or UDP packets.

> +			ip->ttl--;
> +			ip->chksum = odph_ipv4_csum_update(pkt);
> +			eth = (odph_ethhdr_t *)odp_packet_l2_ptr(pkt, NULL);
> +			memcpy(eth->src.addr, entry->src_mac,
> ODPH_ETHADDR_LEN);
> +			memcpy(eth->dst.addr, entry->dst_mac,
> ODPH_ETHADDR_LEN);
> +			odp_pktout_queue(entry->pktio, &outq, 1);

No need to call this with every packet. E.g. cache values to an array.

> +			odp_pktout_send(outq, &pkt, 1);

Return value not checked. On failure packet should be freed.

Another possible issue here is thread safety. The output queues are configured as ODP_PKTIO_OP_MT_UNSAFE in create_pktion(), but it seems that multiple threads may send to the same port.

> +		/* print mac string, could be used to config pktgen */
> +		sprintf(buf, "%02X:%02X:%02X:%02X:%02X:%02X",
> +			mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);

mac_addr_str() helper could be used.

> +static void print_usage(char *progname)
> +{
> +	printf("\n"
> +	       "ODP L3 forwarding application.\n"
> +	       "\n"
> +	       "Usage: %s OPTIONS\n"
> +	       "  E.g. %s -i eth0,eth1\n"
> +	       " In the above example,\n"
> +	       " eth0 will send pkts to eth1 and vice versa\n"
> +	       "\n"

Mandatory argument '-r' is missing from the example. The lines below could be justified better and cleaned up to make them easier to read (print lines can be over 80 characters long).

> +			} else if ((i & 1) != 0) {
> +				printf("even number of ports expected, "
> +				       "got %u.\n", i);

Why is even number of ports required?

> diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c
> new file mode 100644
> index 0000000..c1e1b5b
> --- /dev/null
> +++ b/example/l3fwd/odp_l3fwd_db.c
> @@ -0,0 +1,414 @@
> +/* Copyright (c) 2014, Linaro Limited

Should be 2016.

> +/**
> + * Parse text string representing an IPv4 address or subnet
> + *
> + * String is of the format "XXX.XXX.XXX.XXX(/W)" where
> + * "XXX" is decimal value and "/W" is optional subnet length
> + *
> + * @param ipaddress  Pointer to IP address/subnet string to convert
> + * @param addr       Pointer to return IPv4 address
> + * @param mask       Pointer (optional) to return IPv4 mask
> + *
> + * @return 0 if successful else -1
> + */
> +static inline
> +int parse_ipv4_string(char *ipaddress, uint32_t *addr, uint32_t *mask)

I would move this function to helper/ip, so it can be reused.

> +/**
> + * Parse text string representing a MAC address into byte araray
> + *
> + * String is of the format "XX.XX.XX.XX.XX.XX" where XX is hexadecimal
> + *
> + * @param macaddress  Pointer to MAC address string to convert
> + * @param mac         Pointer to MAC address byte array to populate
> + *
> + * @return 0 if successful else -1
> + */
> +static inline
> +int parse_mac_string(char *macaddress, uint8_t *mac)

odph_eth_addr_parse() already implements this.

> +/**
> + * Generate text string representing MAC address
> + *
> + * @param b     Pointer to buffer to store string
> + * @param mac   Pointer to MAC address
> + *
> + * @return Pointer to supplied buffer
> + */
> +static inline
> +char *mac_addr_str(char *b, uint8_t *mac)

I would move this to helper/eth.

> +	fwd_lookup_cache.bucket = bucket;
> +	fwd_lookup_cache.count = bucket_count;
> +
> +	/*Inialize Locks*/

Typo.

> diff --git a/example/l3fwd/odp_l3fwd_db.h b/example/l3fwd/odp_l3fwd_db.h
> new file mode 100644
> index 0000000..c632f5b
> --- /dev/null
> +++ b/example/l3fwd/odp_l3fwd_db.h
> @@ -0,0 +1,137 @@
> +/* Copyright (c) 2014, Linaro Limited

Should be 2016.

> +/**
> + * Default number of flows
> + */
> +#define ODP_DEF_FLOW_COUNT		100000
> +
> +/**
> + * Default Hash bucket number
> + */
> +#define ODP_DEF_BUCKET_COUNT	(ODP_DEF_FLOW_COUNT / 8)
> +

Should not use odp_* prefix. Makes hard to differentiate api and application definitions/functions.

> +/**
> + * Hash calculation utility
> + */
> +#define JHASH_GOLDEN_RATIO	0x9e3779b9
> +#define rot(x, k) (((x) << (k)) | ((x) >> (32 - (k))))
> +#define ODP_BJ3_MIX(a, b, c) \
> +{ \
> +	a -= c; a ^= rot(c, 4); c += b; \
> +	b -= a; b ^= rot(a, 6); a += c; \
> +	c -= b; c ^= rot(b, 8); b += a; \
> +	a -= c; a ^= rot(c, 16); c += b; \
> +	b -= a; b ^= rot(a, 19); a += c; \
> +	c -= b; c ^= rot(b, 4); b += a; \
> +}

Copyright missing. Code copied from DPDK.

> +/**
> + * Display one fowarding database entry

Typo.
Elo, Matias (Nokia - FI/Espoo) May 30, 2016, 12:47 p.m. | #2
One more thing, the output binary file should be added to .gitignore.

-Matias

> 

> Checkpatch throws one error from the patch:

> 

> 	ERROR: spaces required around that '==' (ctx:VxW)

> 	#485: FILE: example/l3fwd/odp_l3fwd.c:427:

> 	+	if (args->route_str[0]== NULL) {

> 

> You could add a test script for running the application during make check (for

> example in the same manner as the l2fwd_simple or switch). This can be done in

> a separate patch. MQ support would also be a nice addition later on.

> 

> Further comments below.

> 

> -Matias

> 

> 

> > +

> > +			if (odp_unlikely(!odp_packet_has_ipv4(pkt))) {

> > +				printf("warning: packet has no ipv4 header\n");

> > +				return NULL;

> > +			}

> 

> I would only drop the non-IP packets, not kill the whole thread, as there is no

> input filtering in odp_pktin_recv().

> 

> > +

> > +			/*TODO: ipv6 need to be done */

> 

> This can be removed.

> 

> > +			if (odp_packet_has_udp(pkt) ||

> > +			    odp_packet_has_tcp(pkt)) {

> > +				/* UDP or TCP*/

> > +				void *ptr = odp_packet_l4_ptr(pkt, NULL);

> > +

> > +				udp = (odph_udphdr_t *)ptr;

> > +				key.src_port = odp_be_to_cpu_16(udp-

> > >src_port);

> > +				key.dst_port = odp_be_to_cpu_16(udp-

> > >dst_port);

> > +			}

> 

> key.src_port and key.dst_port are uninitialized for other than TCP or UDP

> packets.

> 

> > +			ip->ttl--;

> > +			ip->chksum = odph_ipv4_csum_update(pkt);

> > +			eth = (odph_ethhdr_t *)odp_packet_l2_ptr(pkt, NULL);

> > +			memcpy(eth->src.addr, entry->src_mac,

> > ODPH_ETHADDR_LEN);

> > +			memcpy(eth->dst.addr, entry->dst_mac,

> > ODPH_ETHADDR_LEN);

> > +			odp_pktout_queue(entry->pktio, &outq, 1);

> 

> No need to call this with every packet. E.g. cache values to an array.

> 

> > +			odp_pktout_send(outq, &pkt, 1);

> 

> Return value not checked. On failure packet should be freed.

> 

> Another possible issue here is thread safety. The output queues are configured

> as ODP_PKTIO_OP_MT_UNSAFE in create_pktion(), but it seems that multiple

> threads may send to the same port.

> 

> > +		/* print mac string, could be used to config pktgen */

> > +		sprintf(buf, "%02X:%02X:%02X:%02X:%02X:%02X",

> > +			mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);

> 

> mac_addr_str() helper could be used.

> 

> > +static void print_usage(char *progname)

> > +{

> > +	printf("\n"

> > +	       "ODP L3 forwarding application.\n"

> > +	       "\n"

> > +	       "Usage: %s OPTIONS\n"

> > +	       "  E.g. %s -i eth0,eth1\n"

> > +	       " In the above example,\n"

> > +	       " eth0 will send pkts to eth1 and vice versa\n"

> > +	       "\n"

> 

> Mandatory argument '-r' is missing from the example. The lines below could be

> justified better and cleaned up to make them easier to read (print lines can be

> over 80 characters long).

> 

> > +			} else if ((i & 1) != 0) {

> > +				printf("even number of ports expected, "

> > +				       "got %u.\n", i);

> 

> Why is even number of ports required?

> 

> > diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c

> > new file mode 100644

> > index 0000000..c1e1b5b

> > --- /dev/null

> > +++ b/example/l3fwd/odp_l3fwd_db.c

> > @@ -0,0 +1,414 @@

> > +/* Copyright (c) 2014, Linaro Limited

> 

> Should be 2016.

> 

> > +/**

> > + * Parse text string representing an IPv4 address or subnet

> > + *

> > + * String is of the format "XXX.XXX.XXX.XXX(/W)" where

> > + * "XXX" is decimal value and "/W" is optional subnet length

> > + *

> > + * @param ipaddress  Pointer to IP address/subnet string to convert

> > + * @param addr       Pointer to return IPv4 address

> > + * @param mask       Pointer (optional) to return IPv4 mask

> > + *

> > + * @return 0 if successful else -1

> > + */

> > +static inline

> > +int parse_ipv4_string(char *ipaddress, uint32_t *addr, uint32_t *mask)

> 

> I would move this function to helper/ip, so it can be reused.

> 

> > +/**

> > + * Parse text string representing a MAC address into byte araray

> > + *

> > + * String is of the format "XX.XX.XX.XX.XX.XX" where XX is hexadecimal

> > + *

> > + * @param macaddress  Pointer to MAC address string to convert

> > + * @param mac         Pointer to MAC address byte array to populate

> > + *

> > + * @return 0 if successful else -1

> > + */

> > +static inline

> > +int parse_mac_string(char *macaddress, uint8_t *mac)

> 

> odph_eth_addr_parse() already implements this.

> 

> > +/**

> > + * Generate text string representing MAC address

> > + *

> > + * @param b     Pointer to buffer to store string

> > + * @param mac   Pointer to MAC address

> > + *

> > + * @return Pointer to supplied buffer

> > + */

> > +static inline

> > +char *mac_addr_str(char *b, uint8_t *mac)

> 

> I would move this to helper/eth.

> 

> > +	fwd_lookup_cache.bucket = bucket;

> > +	fwd_lookup_cache.count = bucket_count;

> > +

> > +	/*Inialize Locks*/

> 

> Typo.

> 

> > diff --git a/example/l3fwd/odp_l3fwd_db.h b/example/l3fwd/odp_l3fwd_db.h

> > new file mode 100644

> > index 0000000..c632f5b

> > --- /dev/null

> > +++ b/example/l3fwd/odp_l3fwd_db.h

> > @@ -0,0 +1,137 @@

> > +/* Copyright (c) 2014, Linaro Limited

> 

> Should be 2016.

> 

> > +/**

> > + * Default number of flows

> > + */

> > +#define ODP_DEF_FLOW_COUNT		100000

> > +

> > +/**

> > + * Default Hash bucket number

> > + */

> > +#define ODP_DEF_BUCKET_COUNT	(ODP_DEF_FLOW_COUNT / 8)

> > +

> 

> Should not use odp_* prefix. Makes hard to differentiate api and application

> definitions/functions.

> 

> > +/**

> > + * Hash calculation utility

> > + */

> > +#define JHASH_GOLDEN_RATIO	0x9e3779b9

> > +#define rot(x, k) (((x) << (k)) | ((x) >> (32 - (k))))

> > +#define ODP_BJ3_MIX(a, b, c) \

> > +{ \

> > +	a -= c; a ^= rot(c, 4); c += b; \

> > +	b -= a; b ^= rot(a, 6); a += c; \

> > +	c -= b; c ^= rot(b, 8); b += a; \

> > +	a -= c; a ^= rot(c, 16); c += b; \

> > +	b -= a; b ^= rot(a, 19); a += c; \

> > +	c -= b; c ^= rot(b, 4); b += a; \

> > +}

> 

> Copyright missing. Code copied from DPDK.

> 

> > +/**

> > + * Display one fowarding database entry

> 

> Typo.

> 

> _______________________________________________

> lng-odp mailing list

> lng-odp@lists.linaro.org

> https://lists.linaro.org/mailman/listinfo/lng-odp

Patch

diff --git a/example/Makefile.am b/example/Makefile.am
index 7f82c4d..67e4389 100644
--- a/example/Makefile.am
+++ b/example/Makefile.am
@@ -1 +1 @@ 
-SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt l2fwd_simple switch
+SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt l2fwd_simple l3fwd switch
diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am
new file mode 100644
index 0000000..0ba4527
--- /dev/null
+++ b/example/l3fwd/Makefile.am
@@ -0,0 +1,11 @@ 
+include $(top_srcdir)/example/Makefile.inc
+
+bin_PROGRAMS = odp_l3fwd$(EXEEXT)
+odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static
+odp_l3fwd_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example
+
+noinst_HEADERS = \
+		  $(top_srcdir)/example/l3fwd/odp_l3fwd_db.h \
+		  $(top_srcdir)/example/example_debug.h
+
+dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c
diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c
new file mode 100644
index 0000000..838f3e5
--- /dev/null
+++ b/example/l3fwd/odp_l3fwd.c
@@ -0,0 +1,473 @@ 
+/* Copyright (c) 2016, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier:     BSD-3-Clause
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+
+#include <example_debug.h>
+#include <odp_api.h>
+#include <odp/helper/linux.h>
+#include <odp/helper/eth.h>
+#include <odp/helper/ip.h>
+#include <odp/helper/udp.h>
+#include <odp/helper/tcp.h>
+
+#include <getopt.h>
+
+#include "odp_l3fwd_db.h"
+
+#define POOL_NUM_PKT 8192
+#define POOL_SEG_LEN 1856
+#define MAX_PKT_BURST 32
+
+#define MAX_NB_WORKER	8
+#define MAX_NB_PKTIO	4
+#define MAX_NB_ROUTE    32
+
+/** Get rid of path in filename - only for unix-type paths using '/' */
+#define NO_PATH(file_name) (strrchr((file_name), '/') ? \
+			    strrchr((file_name), '/') + 1 : (file_name))
+
+typedef struct {
+	char *if_names[MAX_NB_PKTIO];
+	int if_count;
+	char *route_str[MAX_NB_ROUTE];
+	int worker_count;
+} app_args_t;
+
+struct l3fwd_pktio_s {
+	odp_pktio_t pktio;
+	odp_pktin_queue_t ifin;
+	odp_pktout_queue_t ifout;
+};
+
+struct thread_arg_s {
+	uint32_t if_idx;
+};
+
+struct {
+	app_args_t		cmd_args;
+	struct l3fwd_pktio_s	l3fwd_pktios[MAX_NB_PKTIO];
+	odph_linux_pthread_t	l3fwd_workers[MAX_NB_WORKER];
+	struct thread_arg_s	worker_args[MAX_NB_WORKER];
+	uint32_t		nb_pktio;  /* effective pktios */
+	uint32_t		nb_worker; /* effective workers */
+} global;
+
+static void print_usage(char *progname);
+static void print_info(char *progname, app_args_t *args);
+static void parse_cmdline_args(int argc, char *argv[], app_args_t *args);
+
+static odp_pktio_t create_pktio(const char *name, odp_pool_t pool,
+				odp_pktin_queue_t *pktin,
+				odp_pktout_queue_t *pktout)
+{
+	odp_pktio_param_t pktio_param;
+	odp_pktin_queue_param_t in_queue_param;
+	odp_pktout_queue_param_t out_queue_param;
+	odp_pktio_t pktio;
+
+	odp_pktio_param_init(&pktio_param);
+
+	pktio = odp_pktio_open(name, pool, &pktio_param);
+	if (pktio == ODP_PKTIO_INVALID) {
+		printf("Failed to open %s\n", name);
+		exit(1);
+	}
+
+	odp_pktin_queue_param_init(&in_queue_param);
+	odp_pktout_queue_param_init(&out_queue_param);
+
+	in_queue_param.op_mode = ODP_PKTIO_OP_MT_UNSAFE;
+
+	if (odp_pktin_queue_config(pktio, &in_queue_param)) {
+		printf("Failed to config input queue for %s\n", name);
+		exit(1);
+	}
+
+	out_queue_param.op_mode = ODP_PKTIO_OP_MT_UNSAFE;
+
+	if (odp_pktout_queue_config(pktio, &out_queue_param)) {
+		printf("Failed to config output queue for %s\n", name);
+		exit(1);
+	}
+
+	if (odp_pktin_queue(pktio, pktin, 1) != 1) {
+		printf("pktin queue query failed for %s\n", name);
+		exit(1);
+	}
+	if (odp_pktout_queue(pktio, pktout, 1) != 1) {
+		printf("pktout queue query failed for %s\n", name);
+		exit(1);
+	}
+	return pktio;
+}
+
+static void *run_worker(void *arg)
+{
+	odp_packet_t pkt_tbl[MAX_PKT_BURST];
+	odp_packet_t pkt_tbl_drop[MAX_PKT_BURST];
+	uint32_t pkts, i;
+	struct l3fwd_pktio_s *port;
+	char *if_name;
+
+	i = ((struct thread_arg_s *)arg)->if_idx;
+	port = &global.l3fwd_pktios[i];
+	if_name = global.cmd_args.if_names[i];
+	if (odp_pktio_start(port->pktio)) {
+		printf("unable to start pktio: %s\n", if_name);
+		exit(1);
+	}
+
+	printf("start pktio: %s\n", if_name);
+	for (;;) {
+		int need_to_drop = 0;
+
+		pkts = odp_pktin_recv(port->ifin, pkt_tbl, MAX_PKT_BURST);
+		if (odp_unlikely(pkts <= 0))
+			continue;
+		for (i = 0; i < pkts; i++) {
+			odp_packet_t pkt = pkt_tbl[i];
+			odph_ethhdr_t *eth;
+			odph_ipv4hdr_t *ip;
+			odph_udphdr_t  *udp;
+			uint32_t len;
+			fwd_db_entry_t *entry;
+			odp_pktout_queue_t outq;
+			ipv4_tuple5_t key;
+
+			if (odp_unlikely(!odp_packet_has_ipv4(pkt))) {
+				printf("warning: packet has no ipv4 header\n");
+				return NULL;
+			}
+
+			/*TODO: ipv6 need to be done */
+			ip = (odph_ipv4hdr_t *)odp_packet_l3_ptr(pkt, &len);
+			key.dst_ip = odp_be_to_cpu_32(ip->dst_addr);
+			key.src_ip = odp_be_to_cpu_32(ip->src_addr);
+			key.proto = ip->proto;
+			if (odp_packet_has_udp(pkt) ||
+			    odp_packet_has_tcp(pkt)) {
+				/* UDP or TCP*/
+				void *ptr = odp_packet_l4_ptr(pkt, NULL);
+
+				udp = (odph_udphdr_t *)ptr;
+				key.src_port = odp_be_to_cpu_16(udp->src_port);
+				key.dst_port = odp_be_to_cpu_16(udp->dst_port);
+			}
+
+			entry = find_fwd_db_entry(&key);
+			if (!entry) {
+				pkt_tbl_drop[need_to_drop] = pkt;
+				need_to_drop++;
+				continue;
+			}
+
+			if (odp_unlikely(!odp_packet_has_eth(pkt))) {
+				printf("warning: packet has no eth header\n");
+				return NULL;
+			}
+
+			ip->ttl--;
+			ip->chksum = odph_ipv4_csum_update(pkt);
+			eth = (odph_ethhdr_t *)odp_packet_l2_ptr(pkt, NULL);
+			memcpy(eth->src.addr, entry->src_mac, ODPH_ETHADDR_LEN);
+			memcpy(eth->dst.addr, entry->dst_mac, ODPH_ETHADDR_LEN);
+			odp_pktout_queue(entry->pktio, &outq, 1);
+			odp_pktout_send(outq, &pkt, 1);
+		}
+
+		odp_packet_free_multi(pkt_tbl_drop, need_to_drop);
+	}
+	return NULL;
+}
+
+int main(int argc, char **argv)
+{
+	odp_pool_t pool;
+	odp_pool_param_t params;
+	odp_cpumask_t cpumask;
+	odp_instance_t instance;
+	odph_linux_thr_params_t thr_params;
+	uint32_t cpu, i;
+	uint8_t mac[ODPH_ETHADDR_LEN];
+	app_args_t *args;
+
+	if (odp_init_global(&instance, NULL, NULL)) {
+		printf("Error: ODP global init failed.\n");
+		exit(1);
+	}
+
+	if (odp_init_local(instance, ODP_THREAD_CONTROL)) {
+		printf("Error: ODP local init failed.\n");
+		exit(1);
+	}
+
+	/* Clear global argument */
+	memset(&global, 0, sizeof(global));
+
+	/* Parse cmdline arguments */
+	args = &global.cmd_args;
+	parse_cmdline_args(argc, argv, args);
+
+	/* Init l3fwd tale */
+	init_fwd_db();
+
+	/* Add route into table */
+	for (i = 0; i < MAX_NB_ROUTE; i++) {
+		if (args->route_str[i])
+			create_fwd_db_entry(args->route_str[i]);
+	}
+
+	print_info(NO_PATH(argv[0]), args);
+
+	/* Create packet pool */
+	odp_pool_param_init(&params);
+	params.pkt.seg_len = POOL_SEG_LEN;
+	params.pkt.len     = POOL_SEG_LEN;
+	params.pkt.num     = POOL_NUM_PKT;
+	params.type        = ODP_POOL_PACKET;
+
+	pool = odp_pool_create("packet pool", &params);
+
+	if (pool == ODP_POOL_INVALID) {
+		printf("Error: packet pool create failed.\n");
+		exit(1);
+	}
+
+	global.nb_pktio = args->if_count;
+	for (i = 0; i < global.nb_pktio; i++) {
+		struct l3fwd_pktio_s *port;
+		char buf[16];
+
+		port = &global.l3fwd_pktios[i];
+		port->pktio = create_pktio(args->if_names[i], pool, &port->ifin,
+					   &port->ifout);
+		odp_pktio_mac_addr(port->pktio, mac, ODPH_ETHADDR_LEN);
+		resolve_fwd_db(args->if_names[i], port->pktio, mac);
+
+		/* print mac string, could be used to config pktgen */
+		sprintf(buf, "%02X:%02X:%02X:%02X:%02X:%02X",
+			mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
+		printf("create pktio %s, mac %s\n", args->if_names[i], buf);
+	}
+
+	/* at most cpu_count-1 threads, only 1 thread on 1 port.
+	 * TODO: assign more threads for each port
+	 */
+	if (args->worker_count == 0 || args->worker_count > args->if_count)
+		args->worker_count = args->if_count;
+
+	if (args->worker_count >= odp_cpu_count())
+		args->worker_count = odp_cpu_count() - 1;
+
+	global.nb_worker = args->worker_count;
+
+	memset(&thr_params, 0, sizeof(thr_params));
+	thr_params.start    = run_worker;
+	thr_params.thr_type = ODP_THREAD_WORKER;
+	thr_params.instance = instance;
+
+	odp_cpumask_default_worker(&cpumask, global.nb_worker);
+	cpu = odp_cpumask_first(&cpumask);
+	for (i = 0; i < global.nb_worker; i++) {
+		struct thread_arg_s *arg;
+		odp_cpumask_t thr_mask;
+
+		odp_cpumask_zero(&thr_mask);
+		odp_cpumask_set(&thr_mask, cpu);
+		arg = &global.worker_args[i];
+		arg->if_idx = i;
+		thr_params.arg      = arg;
+		odph_linux_pthread_create(&global.l3fwd_workers[i], &thr_mask,
+					  &thr_params);
+		cpu = odp_cpumask_next(&cpumask, cpu);
+	}
+	odph_linux_pthread_join(&global.l3fwd_workers[0], global.nb_worker);
+
+	return 0;
+}
+
+static void print_usage(char *progname)
+{
+	printf("\n"
+	       "ODP L3 forwarding application.\n"
+	       "\n"
+	       "Usage: %s OPTIONS\n"
+	       "  E.g. %s -i eth0,eth1\n"
+	       " In the above example,\n"
+	       " eth0 will send pkts to eth1 and vice versa\n"
+	       "\n"
+	       "Mandatory OPTIONS:\n"
+	       "  -i, --interface eth interfaces (comma-separated, no spaces)\n"
+	       "  -r, --route SubNet:Intf[:NextHopMAC]\n"
+	       "	NextHopMAC can be optional, in this case, zeroed mac\n"
+	       "\n"
+	       "Optional OPTIONS:\n"
+	       "  -t, --thread number of threads to do forwarding\n"
+	       "	optional, default as cpu count\n"
+	       "  -h, --help   Display help and exit.\n\n"
+	       "\n", NO_PATH(progname), NO_PATH(progname)
+	    );
+}
+
+static void parse_cmdline_args(int argc, char *argv[], app_args_t *args)
+{
+	int opt;
+	int long_index;
+	char *token, *local;
+	size_t len, route_index = 0;
+	int i, mem_failure = 0;
+
+	static struct option longopts[] = {
+		{"interface", required_argument, NULL, 'i'},	/* return 'i' */
+		{"route", required_argument, NULL, 'r'},	/* return 'r' */
+		{"thread", optional_argument, NULL, 't'},	/* return 't'*/
+		{"help", no_argument, NULL, 'h'},		/* return 'h' */
+		{NULL, 0, NULL, 0}
+	};
+
+	while (1) {
+		opt = getopt_long(argc, argv, "+t:i:r:h",
+				  longopts, &long_index);
+
+		if (opt == -1)
+			break;	/* No more options */
+
+		switch (opt) {
+		/* parse number of worker threads to be run*/
+		case 't':
+			i = odp_cpu_count();
+			args->worker_count = atoi(optarg);
+			if (args->worker_count > i) {
+				printf("Too many threads,"
+				       "truncate to cpu count: %d\n", i);
+				args->worker_count = i;
+			}
+
+			break;
+		/* parse packet-io interface names */
+		case 'i':
+			len = strlen(optarg);
+			if (len == 0) {
+				print_usage(argv[0]);
+				exit(EXIT_FAILURE);
+			}
+			len += 1;	/* add room for '\0' */
+
+			local = malloc(len);
+			if (!local) {
+				print_usage(argv[0]);
+				exit(EXIT_FAILURE);
+			}
+
+			/* count the number of tokens separated by ',' */
+			strcpy(local, optarg);
+			for (token = strtok(local, ","), i = 0;
+			     token != NULL;
+			     token = strtok(NULL, ","), i++)
+				;
+
+			if (i == 0) {
+				print_usage(argv[0]);
+				exit(EXIT_FAILURE);
+			} else if ((i & 1) != 0) {
+				printf("even number of ports expected, "
+				       "got %u.\n", i);
+				exit(EXIT_FAILURE);
+			} else if (i > MAX_NB_PKTIO) {
+				printf("too many ports specified, "
+				       "truncated to max %d", MAX_NB_PKTIO);
+			}
+			args->if_count = i;
+
+			/* store the if names (reset names string) */
+			strcpy(local, optarg);
+			for (token = strtok(local, ","), i = 0;
+			     token != NULL; token = strtok(NULL, ","), i++) {
+				args->if_names[i] = token;
+			}
+			break;
+
+		/*Configure Route in forwarding database*/
+		case 'r':
+			if (route_index >= MAX_NB_ROUTE) {
+				printf("No more routes can be added\n");
+				break;
+			}
+			local = calloc(1, strlen(optarg) + 1);
+			if (!local) {
+				mem_failure = 1;
+				break;
+			}
+			memcpy(local, optarg, strlen(optarg));
+			local[strlen(optarg)] = '\0';
+			args->route_str[route_index++] = local;
+			break;
+
+		case 'h':
+			print_usage(argv[0]);
+			exit(EXIT_SUCCESS);
+			break;
+
+		default:
+			break;
+		}
+	}
+
+	/* checking arguments */
+	if (args->if_count == 0) {
+		printf("\nNo option -i specified.\n");
+		goto out;
+	}
+
+	if (args->route_str[0]== NULL) {
+		printf("\nNo option -r specified.\n");
+		goto out;
+	}
+
+	if (mem_failure == 1) {
+		printf("\nAllocate memory failure.\n");
+		goto out;
+	}
+	optind = 1;		/* reset 'extern optind' from the getopt lib */
+	return;
+
+out:
+	print_usage(argv[0]);
+	exit(EXIT_FAILURE);
+}
+
+static void print_info(char *progname, app_args_t *args)
+{
+	int i;
+
+	printf("\n"
+	       "ODP system info\n"
+	       "---------------\n"
+	       "ODP API version: %s\n"
+	       "ODP impl name:	 %s\n"
+	       "CPU model:       %s\n"
+	       "CPU freq (hz):   %" PRIu64 "\n"
+	       "Cache line size: %i\n"
+	       "CPU count:       %i\n"
+	       "\n",
+	       odp_version_api_str(), odp_version_impl_name(),
+	       odp_cpu_model_str(), odp_cpu_hz_max(),
+	       odp_sys_cache_line_size(), odp_cpu_count());
+
+	printf("Running ODP appl: \"%s\"\n"
+	       "-----------------\n"
+	       "IF-count:        %i\n"
+	       "Using IFs:      ",
+	       progname, args->if_count);
+
+	for (i = 0; i < args->if_count; ++i)
+		printf(" %s", args->if_names[i]);
+
+	printf("\n\n");
+	fflush(NULL);
+}
diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c
new file mode 100644
index 0000000..c1e1b5b
--- /dev/null
+++ b/example/l3fwd/odp_l3fwd_db.c
@@ -0,0 +1,414 @@ 
+/* Copyright (c) 2014, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier:     BSD-3-Clause
+ */
+
+/* enable strtok */
+#ifndef _GNU_SOURCE
+#define _GNU_SOURCE
+#endif
+
+#include <stdlib.h>
+#include <string.h>
+
+#include <example_debug.h>
+
+#include <odp_api.h>
+
+#include <odp_l3fwd_db.h>
+
+/**
+ * Compute hash value from a flow
+ */
+static inline
+uint64_t odp_l3fwd_calc_hash(ipv4_tuple5_t *key)
+{
+	uint64_t l4_ports = 0;
+	uint32_t dst_ip, src_ip;
+
+	src_ip = key->src_ip;
+	dst_ip = key->dst_ip + JHASH_GOLDEN_RATIO;
+	ODP_BJ3_MIX(src_ip, dst_ip, l4_ports);
+
+	return l4_ports;
+}
+
+/**
+ * Parse text string representing an IPv4 address or subnet
+ *
+ * String is of the format "XXX.XXX.XXX.XXX(/W)" where
+ * "XXX" is decimal value and "/W" is optional subnet length
+ *
+ * @param ipaddress  Pointer to IP address/subnet string to convert
+ * @param addr       Pointer to return IPv4 address
+ * @param mask       Pointer (optional) to return IPv4 mask
+ *
+ * @return 0 if successful else -1
+ */
+static inline
+int parse_ipv4_string(char *ipaddress, uint32_t *addr, uint32_t *mask)
+{
+	int b[4];
+	int qualifier = 32;
+	int converted;
+
+	if (strchr(ipaddress, '/')) {
+		converted = sscanf(ipaddress, "%d.%d.%d.%d/%d",
+				   &b[3], &b[2], &b[1], &b[0],
+				   &qualifier);
+		if (5 != converted)
+			return -1;
+	} else {
+		converted = sscanf(ipaddress, "%d.%d.%d.%d",
+				   &b[3], &b[2], &b[1], &b[0]);
+		if (4 != converted)
+			return -1;
+	}
+
+	if ((b[0] > 255) || (b[1] > 255) || (b[2] > 255) || (b[3] > 255))
+		return -1;
+	if (!qualifier || (qualifier > 32))
+		return -1;
+
+	*addr = b[0] | b[1] << 8 | b[2] << 16 | b[3] << 24;
+	if (mask)
+		*mask = ~(0xFFFFFFFF & ((1ULL << (32 - qualifier)) - 1));
+
+	return 0;
+}
+
+/**
+ * Parse text string representing a MAC address into byte araray
+ *
+ * String is of the format "XX.XX.XX.XX.XX.XX" where XX is hexadecimal
+ *
+ * @param macaddress  Pointer to MAC address string to convert
+ * @param mac         Pointer to MAC address byte array to populate
+ *
+ * @return 0 if successful else -1
+ */
+static inline
+int parse_mac_string(char *macaddress, uint8_t *mac)
+{
+	int macwords[ODPH_ETHADDR_LEN];
+	int converted;
+
+	converted = sscanf(macaddress,
+			   "%x.%x.%x.%x.%x.%x",
+			   &macwords[0], &macwords[1], &macwords[2],
+			   &macwords[3], &macwords[4], &macwords[5]);
+	if (6 != converted)
+		return -1;
+
+	mac[0] = macwords[0];
+	mac[1] = macwords[1];
+	mac[2] = macwords[2];
+	mac[3] = macwords[3];
+	mac[4] = macwords[4];
+	mac[5] = macwords[5];
+
+	return 0;
+}
+
+/**
+ * Generate text string representing IPv4 range/subnet, output
+ * in "XXX.XXX.XXX.XXX/W" format
+ *
+ * @param b     Pointer to buffer to store string
+ * @param range Pointer to IPv4 address range
+ *
+ * @return Pointer to supplied buffer
+ */
+static inline
+char *ipv4_subnet_str(char *b, ip_addr_range_t *range)
+{
+	int idx;
+	int len;
+
+	for (idx = 0; idx < 32; idx++)
+		if (range->mask & (1 << idx))
+			break;
+	len = 32 - idx;
+
+	sprintf(b, "%03d.%03d.%03d.%03d/%d",
+		0xFF & ((range->addr) >> 24),
+		0xFF & ((range->addr) >> 16),
+		0xFF & ((range->addr) >>  8),
+		0xFF & ((range->addr) >>  0),
+		len);
+	return b;
+}
+
+/**
+ * Generate text string representing MAC address
+ *
+ * @param b     Pointer to buffer to store string
+ * @param mac   Pointer to MAC address
+ *
+ * @return Pointer to supplied buffer
+ */
+static inline
+char *mac_addr_str(char *b, uint8_t *mac)
+{
+	sprintf(b, "%02X.%02X.%02X.%02X.%02X.%02X",
+		mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
+	return b;
+}
+
+/**
+ * Flow cache table entry
+ */
+typedef struct flow_entry_s {
+	ipv4_tuple5_t key;		/**< match key */
+	struct flow_entry_s *next;      /**< next entry on the list */
+	fwd_db_entry_t *fwd_entry;	/**< entry info in db */
+} flow_entry_t;
+
+/**
+ * Flow cache table bucket
+ */
+typedef struct flow_bucket_s {
+	odp_spinlock_t		lock;	/**< Bucket lock*/
+	flow_entry_t		*next;	/**< Pointer to first flow entry in bucket*/
+} flow_bucket_t;
+
+/**
+ * Flow hash table, fast lookup cache
+ */
+typedef struct flow_table_s {
+	flow_bucket_t *bucket;
+	uint32_t count;
+} flow_table_t;
+
+static flow_table_t fwd_lookup_cache;
+
+static inline
+void init_fwd_cache(void)
+{
+	odp_shm_t		hash_shm;
+	flow_bucket_t		*bucket;
+	uint32_t		bucket_count;
+	uint32_t		i;
+
+	bucket_count = ODP_DEF_BUCKET_COUNT;
+
+	/*Reserve memory for Routing hash table*/
+	hash_shm = odp_shm_reserve("route_table",
+				   sizeof(flow_bucket_t) * bucket_count,
+				   ODP_CACHE_LINE_SIZE, 0);
+
+	bucket = odp_shm_addr(hash_shm);
+	if (!bucket) {
+		EXAMPLE_ERR("Error: shared mem alloc failed.\n");
+		exit(-1);
+	}
+
+	fwd_lookup_cache.bucket = bucket;
+	fwd_lookup_cache.count = bucket_count;
+
+	/*Inialize Locks*/
+	for (i = 0; i < bucket_count; i++) {
+		bucket = &fwd_lookup_cache.bucket[i];
+		odp_spinlock_init(&bucket->lock);
+		bucket->next = NULL;
+	}
+}
+
+static inline
+int match_key_flow(ipv4_tuple5_t *key, flow_entry_t *flow)
+{
+	if (key->src_ip == flow->key.src_ip &&
+	    key->dst_ip == flow->key.dst_ip &&
+	    key->src_port == flow->key.src_port &&
+	    key->dst_port == flow->key.dst_port &&
+	    key->proto == flow->key.proto)
+		return 1;
+
+	return 0;
+}
+
+static inline
+flow_entry_t *lookup_fwd_cache(ipv4_tuple5_t *key, flow_bucket_t *bucket)
+{
+	flow_entry_t *rst;
+
+	for (rst = bucket->next; rst != NULL; rst = rst->next) {
+		if (match_key_flow(key, rst))
+			break;
+	}
+
+	return rst;
+}
+
+static inline
+flow_entry_t *insert_fwd_cache(ipv4_tuple5_t *key,
+			       flow_bucket_t *bucket,
+			       fwd_db_entry_t *entry)
+{
+	flow_entry_t *flow;
+
+	flow = lookup_fwd_cache(key, bucket);
+	if (flow)
+		return flow;
+
+	flow = malloc(sizeof(flow_entry_t));
+	flow->key = *key;
+	flow->fwd_entry = entry;
+
+	odp_spinlock_lock(&bucket->lock);
+	if (!bucket->next) {
+		bucket->next = flow;
+	} else {
+		flow->next = bucket->next;
+		bucket->next = flow;
+	}
+	odp_spinlock_unlock(&bucket->lock);
+
+	return flow;
+}
+
+/** Global pointer to fwd db */
+fwd_db_t *fwd_db;
+
+void init_fwd_db(void)
+{
+	odp_shm_t shm;
+
+	shm = odp_shm_reserve("shm_fwd_db",
+			      sizeof(fwd_db_t),
+			      ODP_CACHE_LINE_SIZE,
+			      0);
+
+	fwd_db = odp_shm_addr(shm);
+
+	if (fwd_db == NULL) {
+		EXAMPLE_ERR("Error: shared mem alloc failed.\n");
+		exit(EXIT_FAILURE);
+	}
+	memset(fwd_db, 0, sizeof(*fwd_db));
+
+	init_fwd_cache();
+}
+
+int create_fwd_db_entry(char *input)
+{
+	int pos = 0;
+	char *local;
+	char *str;
+	char *save;
+	char *token;
+	fwd_db_entry_t *entry = &fwd_db->array[fwd_db->index];
+
+	/* Verify we haven't run out of space */
+	if (MAX_DB <= fwd_db->index)
+		return -1;
+
+	/* Make a local copy */
+	local = malloc(strlen(input) + 1);
+	if (NULL == local)
+		return -1;
+	strcpy(local, input);
+
+	/* Setup for using "strtok_r" to search input string */
+	str = local;
+	save = NULL;
+
+	/* Parse tokens separated by ':' */
+	while (NULL != (token = strtok_r(str, ":", &save))) {
+		str = NULL;  /* reset str for subsequent strtok_r calls */
+
+		/* Parse token based on its position */
+		switch (pos) {
+		case 0:
+			parse_ipv4_string(token,
+					  &entry->subnet.addr,
+					  &entry->subnet.mask);
+			break;
+		case 1:
+			strncpy(entry->oif, token, OIF_LEN - 1);
+			entry->oif[OIF_LEN - 1] = 0;
+			break;
+		case 2:
+			parse_mac_string(token, entry->dst_mac);
+			break;
+		default:
+			printf("ERROR: extra token \"%s\" at position %d\n",
+			       token, pos);
+			break;
+		}
+
+		/* Advance to next position */
+		pos++;
+	}
+
+	/* Reset pktio to invalid */
+	entry->pktio = ODP_PKTIO_INVALID;
+
+	/* Add route to the list */
+	fwd_db->index++;
+	entry->next = fwd_db->list;
+	fwd_db->list = entry;
+
+	free(local);
+	return 0;
+}
+
+void resolve_fwd_db(char *intf, odp_pktio_t pktio, uint8_t *mac)
+{
+	fwd_db_entry_t *entry;
+
+	/* Walk the list and attempt to set output and MAC */
+	for (entry = fwd_db->list; NULL != entry; entry = entry->next) {
+		if (strcmp(intf, entry->oif))
+			continue;
+
+		entry->pktio = pktio;
+		memcpy(entry->src_mac, mac, ODPH_ETHADDR_LEN);
+	}
+}
+
+void dump_fwd_db_entry(fwd_db_entry_t *entry)
+{
+	char subnet_str[MAX_STRING];
+	char mac_str[MAX_STRING];
+
+	printf(" %s %s %s\n",
+	       ipv4_subnet_str(subnet_str, &entry->subnet),
+	       entry->oif,
+	       mac_addr_str(mac_str, entry->dst_mac));
+}
+
+void dump_fwd_db(void)
+{
+	fwd_db_entry_t *entry;
+
+	printf("\n"
+	       "Routing table\n"
+	       "-------------\n");
+
+	for (entry = fwd_db->list; NULL != entry; entry = entry->next)
+		dump_fwd_db_entry(entry);
+}
+
+fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key)
+{
+	fwd_db_entry_t *entry;
+	flow_entry_t *flow;
+	flow_bucket_t *bucket;
+	uint64_t hash;
+
+	/* first find in cache */
+	hash = odp_l3fwd_calc_hash(key);
+	hash &= fwd_lookup_cache.count - 1;
+	bucket = &fwd_lookup_cache.bucket[hash];
+	flow = lookup_fwd_cache(key, bucket);
+	if (flow)
+		return flow->fwd_entry;
+
+	for (entry = fwd_db->list; NULL != entry; entry = entry->next)
+		if (entry->subnet.addr == (key->dst_ip & entry->subnet.mask))
+			break;
+
+	return entry;
+}
diff --git a/example/l3fwd/odp_l3fwd_db.h b/example/l3fwd/odp_l3fwd_db.h
new file mode 100644
index 0000000..c632f5b
--- /dev/null
+++ b/example/l3fwd/odp_l3fwd_db.h
@@ -0,0 +1,137 @@ 
+/* Copyright (c) 2014, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier:     BSD-3-Clause
+ */
+
+#ifndef ODP_L3FWD_DB_H_
+#define ODP_L3FWD_DB_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <odp_api.h>
+#include <odp/helper/eth.h>
+
+#define OIF_LEN 32
+#define MAX_DB  32
+#define MAX_STRING  32
+
+/**
+ * Default number of flows
+ */
+#define ODP_DEF_FLOW_COUNT		100000
+
+/**
+ * Default Hash bucket number
+ */
+#define ODP_DEF_BUCKET_COUNT	(ODP_DEF_FLOW_COUNT / 8)
+
+/**
+ * Hash calculation utility
+ */
+#define JHASH_GOLDEN_RATIO	0x9e3779b9
+#define rot(x, k) (((x) << (k)) | ((x) >> (32 - (k))))
+#define ODP_BJ3_MIX(a, b, c) \
+{ \
+	a -= c; a ^= rot(c, 4); c += b; \
+	b -= a; b ^= rot(a, 6); a += c; \
+	c -= b; c ^= rot(b, 8); b += a; \
+	a -= c; a ^= rot(c, 16); c += b; \
+	b -= a; b ^= rot(a, 19); a += c; \
+	c -= b; c ^= rot(b, 4); b += a; \
+}
+
+/**
+ * IP address range (subnet)
+ */
+typedef struct ip_addr_range_s {
+	uint32_t  addr;     /**< IP address */
+	uint32_t  mask;     /**< mask, 1 indicates bits are valid */
+} ip_addr_range_t;
+
+/**
+ * TCP/UDP flow
+ */
+typedef struct ipv4_tuple5_s {
+	uint32_t src_ip;
+	uint32_t dst_ip;
+	uint16_t src_port;
+	uint16_t dst_port;
+	uint8_t  proto;
+} ipv4_tuple5_t;
+
+/**
+ * Forwarding data base entry
+ */
+typedef struct fwd_db_entry_s {
+	struct fwd_db_entry_s *next;          /**< Next entry on list */
+	char                    oif[OIF_LEN]; /**< Output interface name */
+	odp_pktio_t		pktio;        /**< Output transmit port */
+	uint8_t   src_mac[ODPH_ETHADDR_LEN];  /**< Output source MAC */
+	uint8_t   dst_mac[ODPH_ETHADDR_LEN];  /**< Output destination MAC */
+	ip_addr_range_t        subnet;        /**< Subnet for this router */
+} fwd_db_entry_t;
+
+/**
+ * Forwarding data base
+ */
+typedef struct fwd_db_s {
+	uint32_t          index;          /**< Next available entry */
+	fwd_db_entry_t   *list;           /**< List of active routes */
+	fwd_db_entry_t    array[MAX_DB];  /**< Entry storage */
+} fwd_db_t;
+
+/** Global pointer to fwd db */
+extern fwd_db_t *fwd_db;
+
+/** Initialize FWD DB */
+void init_fwd_db(void);
+
+/**
+ * Create a forwarding database entry
+ *
+ * String is of the format "SubNet:Intf:NextHopMAC"
+ *
+ * @param input  Pointer to string describing route
+ *
+ * @return 0 if successful else -1
+ */
+int create_fwd_db_entry(char *input);
+
+/**
+ * Scan FWD DB entries and resolve output queue and source MAC address
+ *
+ * @param intf   Interface name string
+ * @param pktio   Output port for packet transmit
+ * @param mac    MAC address of this interface
+ */
+void resolve_fwd_db(char *intf, odp_pktio_t pktio, uint8_t *mac);
+
+/**
+ * Display one fowarding database entry
+ *
+ * @param entry  Pointer to entry to display
+ */
+void dump_fwd_db_entry(fwd_db_entry_t *entry);
+
+/**
+ * Display the forwarding database
+ */
+void dump_fwd_db(void);
+
+/**
+ * Find a matching forwarding database entry
+ *
+ * @param key  ipv4 tuple
+ *
+ * @return pointer to forwarding DB entry else NULL
+ */
+fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/example/m4/configure.m4 b/example/m4/configure.m4
index 9731d81..7868574 100644
--- a/example/m4/configure.m4
+++ b/example/m4/configure.m4
@@ -19,4 +19,5 @@  AC_CONFIG_FILES([example/classifier/Makefile
 		 example/timer/Makefile
 		 example/traffic_mgmt/Makefile
 		 example/l2fwd_simple/Makefile
+		 example/l3fwd/Makefile
 		 example/switch/Makefile])