[1/5] example: introducing l3fwd

Message ID 1463662931-1183-1-git-send-email-forrest.shi@linaro.org
State New
Headers show

Commit Message

Forrest Shi May 19, 2016, 1:02 p.m.
From: Xuelin Shi <forrest.shi@linaro.org>

It is a simple l3fwd, only single thread, single port works.
Would be enhanced in later versions.

Signed-off-by: Xuelin Shi <forrest.shi@linaro.org>
---
 example/Makefile.am          |   2 +-
 example/l3fwd/Makefile.am    |  11 ++
 example/l3fwd/odp_l3fwd.c    | 218 +++++++++++++++++++++++++++++++
 example/l3fwd/odp_l3fwd_db.c | 299 +++++++++++++++++++++++++++++++++++++++++++
 example/l3fwd/odp_l3fwd_db.h | 137 ++++++++++++++++++++
 example/m4/configure.m4      |   1 +
 6 files changed, 667 insertions(+), 1 deletion(-)
 create mode 100644 example/l3fwd/Makefile.am
 create mode 100644 example/l3fwd/odp_l3fwd.c
 create mode 100644 example/l3fwd/odp_l3fwd_db.c
 create mode 100644 example/l3fwd/odp_l3fwd_db.h

Comments

Elo, Matias (Nokia - FI/Espoo) May 19, 2016, 11:58 a.m. | #1
Hi Xuelin,

Some initial comments below. I didn't have yet time to look at the routing part.

-Matias

> +

> +struct {

> +	odp_pktio_t if0, if1;

> +	odp_pktin_queue_t if0in, if1in;

> +	odp_pktout_queue_t if0out, if1out;

> +	odph_ethaddr_t src, dst;

> +} global;


I would prefer the application to support more than two interfaces. This makes the application logic more complex but a router with only two interfaces is a bit too simple in my opinion.

> +	printf("started output interface\n");

> +	printf("started all\n");

> +

> +	for (;;) {

> +		int need_to_drop = 0;

> +

> +		memset(pkt_tbl, 0, sizeof(pkt_tbl_drop));


I see no need to memset() here.

> +			eth = (odph_ethhdr_t *)odp_packet_l2_ptr(pkt, NULL);

> +			memcpy(eth->src.addr, entry->src_mac,

> ODPH_ETHADDR_LEN);

> +			memcpy(eth->dst.addr, entry->dst_mac,

> ODPH_ETHADDR_LEN);

> +			odp_pktout_queue(entry->pktio, &outq, 1);


Could be saved beforehand for better performance. No need to check for every packet.

> +/**

> + * Parse text string representing an IPv4 address or subnet

> + *

> + * String is of the format "XXX.XXX.XXX.XXX(/W)" where

> + * "XXX" is decimal value and "/W" is optional subnet length

> + *

> + * @param ipaddress  Pointer to IP address/subnet string to convert

> + * @param addr       Pointer to return IPv4 address

> + * @param mask       Pointer (optional) to return IPv4 mask

> + *

> + * @return 0 if successful else -1

> + */


There is already a helper function for parsing IP address strings odph_ipv4_addr_parse(). If it doesn't have all the features you need you could improve it.

> +/**

> + * Parse text string representing a MAC address into byte araray

> + *

> + * String is of the format "XX.XX.XX.XX.XX.XX" where XX is hexadecimal

> + *

> + * @param macaddress  Pointer to MAC address string to convert

> + * @param mac         Pointer to MAC address byte array to populate

> + *

> + * @return 0 if successful else -1

> + */

> +static inline

> +int parse_mac_string(char *macaddress, uint8_t *mac)


There is already a helper function for parsing MAC address strings odph_eth_addr_parse.
Elo, Matias (Nokia - FI/Espoo) May 19, 2016, 12:13 p.m. | #2
> 

> > +struct {

> > +	odp_pktio_t if0, if1;

> > +	odp_pktin_queue_t if0in, if1in;

> > +	odp_pktout_queue_t if0out, if1out;

> > +	odph_ethaddr_t src, dst;

> > +} global;

> 

> I would prefer the application to support more than two interfaces. This makes

> the application logic more complex but a router with only two interfaces is a bit

> too simple in my opinion.


Replying to myself. You have already implemented this in a later patch in the same patch set. Rewriting same code segments in multiple patches makes reviewing the patches unnecessarily difficult.

-Matias
Forrest Shi May 20, 2016, 2:22 a.m. | #3
Hi Matias,

Thank you for the review. 

Yes, the feature is added at several phases.

I'm also wondering if I should send the final big patch or a patch set with enhancement history.

Thanks,
Forrest

> -----Original Message-----
> From: Elo, Matias (Nokia - FI/Espoo) [mailto:matias.elo@nokia.com]
> Sent: Thursday, May 19, 2016 20:13
> To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com>;
> forrest.shi@linaro.org; lng-odp@lists.linaro.org
> Subject: RE: [lng-odp] [PATCH 1/5] example: introducing l3fwd
> 
> >
> > > +struct {
> > > +	odp_pktio_t if0, if1;
> > > +	odp_pktin_queue_t if0in, if1in;
> > > +	odp_pktout_queue_t if0out, if1out;
> > > +	odph_ethaddr_t src, dst;
> > > +} global;
> >
> > I would prefer the application to support more than two interfaces.
> > This makes the application logic more complex but a router with only
> > two interfaces is a bit too simple in my opinion.
> 
> Replying to myself. You have already implemented this in a later patch in the
> same patch set. Rewriting same code segments in multiple patches makes
> reviewing the patches unnecessarily difficult.
> 
> -Matias
Elo, Matias (Nokia - FI/Espoo) May 20, 2016, 12:33 p.m. | #4
> I'm also wondering if I should send the final big patch or a patch set with

> enhancement history.


I would prefer you send it as a single patch as it is a whole new application. There is no point in showing the "unfinished" versions in the git history.

-Matias

Patch

diff --git a/example/Makefile.am b/example/Makefile.am
index 7f82c4d..67e4389 100644
--- a/example/Makefile.am
+++ b/example/Makefile.am
@@ -1 +1 @@ 
-SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt l2fwd_simple switch
+SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt l2fwd_simple l3fwd switch
diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am
new file mode 100644
index 0000000..0ba4527
--- /dev/null
+++ b/example/l3fwd/Makefile.am
@@ -0,0 +1,11 @@ 
+include $(top_srcdir)/example/Makefile.inc
+
+bin_PROGRAMS = odp_l3fwd$(EXEEXT)
+odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static
+odp_l3fwd_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example
+
+noinst_HEADERS = \
+		  $(top_srcdir)/example/l3fwd/odp_l3fwd_db.h \
+		  $(top_srcdir)/example/example_debug.h
+
+dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c
diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c
new file mode 100644
index 0000000..704d33e
--- /dev/null
+++ b/example/l3fwd/odp_l3fwd.c
@@ -0,0 +1,218 @@ 
+/* Copyright (c) 2016, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier:     BSD-3-Clause
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+
+#include <odp_api.h>
+#include <odp/helper/linux.h>
+#include <odp/helper/eth.h>
+#include <odp/helper/ip.h>
+
+#include "odp_l3fwd_db.h"
+
+#define POOL_NUM_PKT 8192
+#define POOL_SEG_LEN 1856
+#define MAX_PKT_BURST 32
+
+static const char * const route_str[] = {
+	"1.1.1.0/24:fm1-mac1:00.e0.0c.00.85.00",
+	"2.1.1.0/24:fm1-mac2:00.e0.0c.00.85.01",
+};
+
+struct {
+	odp_pktio_t if0, if1;
+	odp_pktin_queue_t if0in, if1in;
+	odp_pktout_queue_t if0out, if1out;
+	odph_ethaddr_t src, dst;
+} global;
+
+static odp_pktio_t create_pktio(const char *name, odp_pool_t pool,
+				odp_pktin_queue_t *pktin,
+				odp_pktout_queue_t *pktout)
+{
+	odp_pktio_param_t pktio_param;
+	odp_pktin_queue_param_t in_queue_param;
+	odp_pktout_queue_param_t out_queue_param;
+	odp_pktio_t pktio;
+
+	odp_pktio_param_init(&pktio_param);
+
+	pktio = odp_pktio_open(name, pool, &pktio_param);
+	if (pktio == ODP_PKTIO_INVALID) {
+		printf("Failed to open %s\n", name);
+		exit(1);
+	}
+
+	odp_pktin_queue_param_init(&in_queue_param);
+	odp_pktout_queue_param_init(&out_queue_param);
+
+	in_queue_param.op_mode = ODP_PKTIO_OP_MT_UNSAFE;
+
+	if (odp_pktin_queue_config(pktio, &in_queue_param)) {
+		printf("Failed to config input queue for %s\n", name);
+		exit(1);
+	}
+
+	out_queue_param.op_mode = ODP_PKTIO_OP_MT_UNSAFE;
+
+	if (odp_pktout_queue_config(pktio, &out_queue_param)) {
+		printf("Failed to config output queue for %s\n", name);
+		exit(1);
+	}
+
+	if (odp_pktin_queue(pktio, pktin, 1) != 1) {
+		printf("pktin queue query failed for %s\n", name);
+		exit(1);
+	}
+	if (odp_pktout_queue(pktio, pktout, 1) != 1) {
+		printf("pktout queue query failed for %s\n", name);
+		exit(1);
+	}
+	return pktio;
+}
+
+static void *run_worker(void *arg ODP_UNUSED)
+{
+	odp_packet_t pkt_tbl[MAX_PKT_BURST];
+	odp_packet_t pkt_tbl_drop[MAX_PKT_BURST];
+	int pkts, i;
+
+	if (odp_pktio_start(global.if0)) {
+		printf("unable to start input interface\n");
+		exit(1);
+	}
+	printf("started input interface\n");
+	if (odp_pktio_start(global.if1)) {
+		printf("unable to start output interface\n");
+		exit(1);
+	}
+	printf("started output interface\n");
+	printf("started all\n");
+
+	for (;;) {
+		int need_to_drop = 0;
+
+		memset(pkt_tbl, 0, sizeof(pkt_tbl_drop));
+
+		pkts = odp_pktin_recv(global.if0in, pkt_tbl, MAX_PKT_BURST);
+		if (odp_unlikely(pkts <= 0))
+			continue;
+		for (i = 0; i < pkts; i++) {
+			odp_packet_t pkt = pkt_tbl[i];
+			odph_ethhdr_t *eth;
+			odph_ipv4hdr_t *ip;
+			uint32_t len;
+			uint32_t dst_ip;
+			fwd_db_entry_t *entry;
+			odp_pktout_queue_t outq;
+
+			if (odp_unlikely(!odp_packet_has_l3(pkt))) {
+				printf("warning: packet has no ip header\n");
+				return NULL;
+			}
+
+			/*TODO: ipv6 need to be done */
+			ip = (odph_ipv4hdr_t *)odp_packet_l3_ptr(pkt, &len);
+			dst_ip = odp_be_to_cpu_32(ip->dst_addr);
+			entry = find_fwd_db_entry(dst_ip);
+			if (!entry) {
+				pkt_tbl_drop[need_to_drop] = pkt;
+				need_to_drop++;
+				continue;
+			}
+
+			if (odp_unlikely(!odp_packet_has_eth(pkt))) {
+				printf("warning: packet has no eth header\n");
+				return NULL;
+			}
+
+			eth = (odph_ethhdr_t *)odp_packet_l2_ptr(pkt, NULL);
+			memcpy(eth->src.addr, entry->src_mac, ODPH_ETHADDR_LEN);
+			memcpy(eth->dst.addr, entry->dst_mac, ODPH_ETHADDR_LEN);
+			odp_pktout_queue(entry->pktio, &outq, 1);
+			odp_pktout_send(outq, &pkt, 1);
+		}
+
+		odp_packet_free_multi(pkt_tbl_drop, need_to_drop);
+	}
+	return NULL;
+}
+
+int main(int argc, char **argv)
+{
+	odp_pool_t pool;
+	odp_pool_param_t params;
+	odp_cpumask_t cpumask;
+	odph_linux_pthread_t thd;
+	odp_instance_t instance;
+	odph_linux_thr_params_t thr_params;
+	size_t i;
+	uint8_t mac[ODPH_ETHADDR_LEN];
+
+	if (argc != 3) {
+		printf("Usage: odp_l3fwd eth0 eth1\n");
+		printf("Where eth0 and eth1 are the used interfaces"
+		       " (must have 2 of them)\n");
+		exit(1);
+	}
+
+	if (odp_init_global(&instance, NULL, NULL)) {
+		printf("Error: ODP global init failed.\n");
+		exit(1);
+	}
+
+	if (odp_init_local(instance, ODP_THREAD_CONTROL)) {
+		printf("Error: ODP local init failed.\n");
+		exit(1);
+	}
+
+	/* Init l3fwd tale */
+	init_fwd_db();
+
+	/* Add route into table */
+	for (i = 0; i < sizeof(route_str) / sizeof(char *); i++) {
+		char buf[128];
+
+		snprintf(buf, 128, "%s", route_str[i]);
+		create_fwd_db_entry(buf);
+	}
+
+	/* Create packet pool */
+	odp_pool_param_init(&params);
+	params.pkt.seg_len = POOL_SEG_LEN;
+	params.pkt.len     = POOL_SEG_LEN;
+	params.pkt.num     = POOL_NUM_PKT;
+	params.type        = ODP_POOL_PACKET;
+
+	pool = odp_pool_create("packet pool", &params);
+
+	if (pool == ODP_POOL_INVALID) {
+		printf("Error: packet pool create failed.\n");
+		exit(1);
+	}
+
+	global.if0 = create_pktio(argv[1], pool, &global.if0in, &global.if0out);
+	global.if1 = create_pktio(argv[2], pool, &global.if1in, &global.if1out);
+
+	/* resolve route table */
+	odp_pktio_mac_addr(global.if0, mac, ODPH_ETHADDR_LEN);
+	resolve_fwd_db(argv[1], global.if0, mac);
+	odp_pktio_mac_addr(global.if1, mac, ODPH_ETHADDR_LEN);
+	resolve_fwd_db(argv[2], global.if1, mac);
+
+	odp_cpumask_default_worker(&cpumask, 1);
+
+	memset(&thr_params, 0, sizeof(thr_params));
+	thr_params.start    = run_worker;
+	thr_params.arg      = NULL;
+	thr_params.thr_type = ODP_THREAD_WORKER;
+	thr_params.instance = instance;
+
+	odph_linux_pthread_create(&thd, &cpumask, &thr_params);
+	odph_linux_pthread_join(&thd, 1);
+	return 0;
+}
diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c
new file mode 100644
index 0000000..2140cca
--- /dev/null
+++ b/example/l3fwd/odp_l3fwd_db.c
@@ -0,0 +1,299 @@ 
+/* Copyright (c) 2014, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier:     BSD-3-Clause
+ */
+
+/* enable strtok */
+#ifndef _GNU_SOURCE
+#define _GNU_SOURCE
+#endif
+
+#include <stdlib.h>
+#include <string.h>
+
+#include <example_debug.h>
+
+#include <odp_api.h>
+
+#include <odp_l3fwd_db.h>
+
+/**
+ * Compute hash value from a flow
+ */
+static inline
+uint64_t odp_l3fwd_calc_hash(ipv4_tuple5_t *flow)
+{
+	uint64_t l4_ports = 0;
+	ipv4_tuple5_t key;
+
+	key = *flow;
+
+	key.dst_ip += JHASH_GOLDEN_RATIO;
+	ODP_BJ3_MIX(key.src_ip, key.dst_ip, l4_ports);
+
+	return l4_ports;
+}
+
+/**
+ * Parse text string representing an IPv4 address or subnet
+ *
+ * String is of the format "XXX.XXX.XXX.XXX(/W)" where
+ * "XXX" is decimal value and "/W" is optional subnet length
+ *
+ * @param ipaddress  Pointer to IP address/subnet string to convert
+ * @param addr       Pointer to return IPv4 address
+ * @param mask       Pointer (optional) to return IPv4 mask
+ *
+ * @return 0 if successful else -1
+ */
+static inline
+int parse_ipv4_string(char *ipaddress, uint32_t *addr, uint32_t *mask)
+{
+	int b[4];
+	int qualifier = 32;
+	int converted;
+
+	if (strchr(ipaddress, '/')) {
+		converted = sscanf(ipaddress, "%d.%d.%d.%d/%d",
+				   &b[3], &b[2], &b[1], &b[0],
+				   &qualifier);
+		if (5 != converted)
+			return -1;
+	} else {
+		converted = sscanf(ipaddress, "%d.%d.%d.%d",
+				   &b[3], &b[2], &b[1], &b[0]);
+		if (4 != converted)
+			return -1;
+	}
+
+	if ((b[0] > 255) || (b[1] > 255) || (b[2] > 255) || (b[3] > 255))
+		return -1;
+	if (!qualifier || (qualifier > 32))
+		return -1;
+
+	*addr = b[0] | b[1] << 8 | b[2] << 16 | b[3] << 24;
+	if (mask)
+		*mask = ~(0xFFFFFFFF & ((1ULL << (32 - qualifier)) - 1));
+
+	return 0;
+}
+
+/**
+ * Parse text string representing a MAC address into byte araray
+ *
+ * String is of the format "XX.XX.XX.XX.XX.XX" where XX is hexadecimal
+ *
+ * @param macaddress  Pointer to MAC address string to convert
+ * @param mac         Pointer to MAC address byte array to populate
+ *
+ * @return 0 if successful else -1
+ */
+static inline
+int parse_mac_string(char *macaddress, uint8_t *mac)
+{
+	int macwords[ODPH_ETHADDR_LEN];
+	int converted;
+
+	converted = sscanf(macaddress,
+			   "%x.%x.%x.%x.%x.%x",
+			   &macwords[0], &macwords[1], &macwords[2],
+			   &macwords[3], &macwords[4], &macwords[5]);
+	if (6 != converted)
+		return -1;
+
+	mac[0] = macwords[0];
+	mac[1] = macwords[1];
+	mac[2] = macwords[2];
+	mac[3] = macwords[3];
+	mac[4] = macwords[4];
+	mac[5] = macwords[5];
+
+	return 0;
+}
+
+/**
+ * Generate text string representing IPv4 range/subnet, output
+ * in "XXX.XXX.XXX.XXX/W" format
+ *
+ * @param b     Pointer to buffer to store string
+ * @param range Pointer to IPv4 address range
+ *
+ * @return Pointer to supplied buffer
+ */
+static inline
+char *ipv4_subnet_str(char *b, ip_addr_range_t *range)
+{
+	int idx;
+	int len;
+
+	for (idx = 0; idx < 32; idx++)
+		if (range->mask & (1 << idx))
+			break;
+	len = 32 - idx;
+
+	sprintf(b, "%03d.%03d.%03d.%03d/%d",
+		0xFF & ((range->addr) >> 24),
+		0xFF & ((range->addr) >> 16),
+		0xFF & ((range->addr) >>  8),
+		0xFF & ((range->addr) >>  0),
+		len);
+	return b;
+}
+
+/**
+ * Generate text string representing MAC address
+ *
+ * @param b     Pointer to buffer to store string
+ * @param mac   Pointer to MAC address
+ *
+ * @return Pointer to supplied buffer
+ */
+static inline
+char *mac_addr_str(char *b, uint8_t *mac)
+{
+	sprintf(b, "%02X.%02X.%02X.%02X.%02X.%02X",
+		mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
+	return b;
+}
+
+/** Global pointer to fwd db */
+fwd_db_t *fwd_db;
+
+void init_fwd_db(void)
+{
+	odp_shm_t shm;
+
+	shm = odp_shm_reserve("shm_fwd_db",
+			      sizeof(fwd_db_t),
+			      ODP_CACHE_LINE_SIZE,
+			      0);
+
+	fwd_db = odp_shm_addr(shm);
+
+	if (fwd_db == NULL) {
+		EXAMPLE_ERR("Error: shared mem alloc failed.\n");
+		exit(EXIT_FAILURE);
+	}
+	memset(fwd_db, 0, sizeof(*fwd_db));
+}
+
+int create_fwd_db_entry(char *input)
+{
+	int pos = 0;
+	char *local;
+	char *str;
+	char *save;
+	char *token;
+	fwd_db_entry_t *entry = &fwd_db->array[fwd_db->index];
+
+	/* Verify we haven't run out of space */
+	if (MAX_DB <= fwd_db->index)
+		return -1;
+
+	/* Make a local copy */
+	local = malloc(strlen(input) + 1);
+	if (NULL == local)
+		return -1;
+	strcpy(local, input);
+
+	/* Setup for using "strtok_r" to search input string */
+	str = local;
+	save = NULL;
+
+	/* Parse tokens separated by ':' */
+	while (NULL != (token = strtok_r(str, ":", &save))) {
+		str = NULL;  /* reset str for subsequent strtok_r calls */
+
+		/* Parse token based on its position */
+		switch (pos) {
+		case 0:
+			parse_ipv4_string(token,
+					  &entry->subnet.addr,
+					  &entry->subnet.mask);
+			break;
+		case 1:
+			strncpy(entry->oif, token, OIF_LEN - 1);
+			entry->oif[OIF_LEN - 1] = 0;
+			break;
+		case 2:
+			parse_mac_string(token, entry->dst_mac);
+			break;
+		default:
+			printf("ERROR: extra token \"%s\" at position %d\n",
+			       token, pos);
+			break;
+		}
+
+		/* Advance to next position */
+		pos++;
+	}
+
+	/* Verify we parsed exactly the number of tokens we expected */
+	if (3 != pos) {
+		printf("ERROR: \"%s\" contains %d tokens, expected 3\n",
+		       input,
+		       pos);
+		free(local);
+		return -1;
+	}
+
+	/* Reset pktio to invalid */
+	entry->pktio = ODP_PKTIO_INVALID;
+
+	/* Add route to the list */
+	fwd_db->index++;
+	entry->next = fwd_db->list;
+	fwd_db->list = entry;
+
+	free(local);
+	return 0;
+}
+
+void resolve_fwd_db(char *intf, odp_pktio_t pktio, uint8_t *mac)
+{
+	fwd_db_entry_t *entry;
+
+	/* Walk the list and attempt to set output and MAC */
+	for (entry = fwd_db->list; NULL != entry; entry = entry->next) {
+		if (strcmp(intf, entry->oif))
+			continue;
+
+		entry->pktio = pktio;
+		memcpy(entry->src_mac, mac, ODPH_ETHADDR_LEN);
+	}
+}
+
+void dump_fwd_db_entry(fwd_db_entry_t *entry)
+{
+	char subnet_str[MAX_STRING];
+	char mac_str[MAX_STRING];
+
+	printf(" %s %s %s\n",
+	       ipv4_subnet_str(subnet_str, &entry->subnet),
+	       entry->oif,
+	       mac_addr_str(mac_str, entry->dst_mac));
+}
+
+void dump_fwd_db(void)
+{
+	fwd_db_entry_t *entry;
+
+	printf("\n"
+	       "Routing table\n"
+	       "-------------\n");
+
+	for (entry = fwd_db->list; NULL != entry; entry = entry->next)
+		dump_fwd_db_entry(entry);
+}
+
+fwd_db_entry_t *find_fwd_db_entry(uint32_t dst_ip)
+{
+	fwd_db_entry_t *entry;
+
+	for (entry = fwd_db->list; NULL != entry; entry = entry->next)
+		if (entry->subnet.addr == (dst_ip & entry->subnet.mask))
+			break;
+
+	return entry;
+}
diff --git a/example/l3fwd/odp_l3fwd_db.h b/example/l3fwd/odp_l3fwd_db.h
new file mode 100644
index 0000000..bd27b2a
--- /dev/null
+++ b/example/l3fwd/odp_l3fwd_db.h
@@ -0,0 +1,137 @@ 
+/* Copyright (c) 2014, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier:     BSD-3-Clause
+ */
+
+#ifndef ODP_L3FWD_DB_H_
+#define ODP_L3FWD_DB_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <odp_api.h>
+#include <odp/helper/eth.h>
+
+#define OIF_LEN 32
+#define MAX_DB  32
+#define MAX_STRING  32
+
+/**
+ * Default number of flows
+ */
+#define ODP_MAX_FLOW_COUNT		100000
+
+/**
+ * Default Hash bucket number
+ */
+#define ODP_MAX_BUCKET_COUNT	(ODP_MAX_FLOW_COUNT / 8)
+
+/**
+ * Hash calculation utility
+ */
+#define JHASH_GOLDEN_RATIO	0x9e3779b9
+#define rot(x, k) (((x) << (k)) | ((x) >> (32 - (k))))
+#define ODP_BJ3_MIX(a, b, c) \
+{ \
+	a -= c; a ^= rot(c, 4); c += b; \
+	b -= a; b ^= rot(a, 6); a += c; \
+	c -= b; c ^= rot(b, 8); b += a; \
+	a -= c; a ^= rot(c, 16); c += b; \
+	b -= a; b ^= rot(a, 19); a += c; \
+	c -= b; c ^= rot(b, 4); b += a; \
+}
+
+/**
+ * IP address range (subnet)
+ */
+typedef struct ip_addr_range_s {
+	uint32_t  addr;     /**< IP address */
+	uint32_t  mask;     /**< mask, 1 indicates bits are valid */
+} ip_addr_range_t;
+
+/**
+ * TCP/UDP flow
+ */
+typedef struct ipv4_tuple5_s {
+	uint32_t src_ip;
+	uint32_t dst_ip;
+	uint16_t src_port;
+	uint16_t dst_port;
+	uint8_t  proto;
+} ipv4_tuple5_t;
+
+/**
+ * Forwarding data base entry
+ */
+typedef struct fwd_db_entry_s {
+	struct fwd_db_entry_s *next;          /**< Next entry on list */
+	char                    oif[OIF_LEN]; /**< Output interface name */
+	odp_pktio_t		pktio;        /**< Output transmit port */
+	uint8_t   src_mac[ODPH_ETHADDR_LEN];  /**< Output source MAC */
+	uint8_t   dst_mac[ODPH_ETHADDR_LEN];  /**< Output destination MAC */
+	ip_addr_range_t        subnet;        /**< Subnet for this router */
+} fwd_db_entry_t;
+
+/**
+ * Forwarding data base hash structure
+ */
+typedef struct fwd_db_s {
+	uint32_t          index;          /**< Next available entry */
+	fwd_db_entry_t   *list;           /**< List of active routes */
+	fwd_db_entry_t    array[MAX_DB];  /**< Entry storage */
+} fwd_db_t;
+
+/** Global pointer to fwd db */
+extern fwd_db_t *fwd_db;
+
+/** Initialize FWD DB */
+void init_fwd_db(void);
+
+/**
+ * Create a forwarding database entry
+ *
+ * String is of the format "SubNet:Intf:NextHopMAC"
+ *
+ * @param input  Pointer to string describing route
+ *
+ * @return 0 if successful else -1
+ */
+int create_fwd_db_entry(char *input);
+
+/**
+ * Scan FWD DB entries and resolve output queue and source MAC address
+ *
+ * @param intf   Interface name string
+ * @param pktio   Output port for packet transmit
+ * @param mac    MAC address of this interface
+ */
+void resolve_fwd_db(char *intf, odp_pktio_t pktio, uint8_t *mac);
+
+/**
+ * Display one fowarding database entry
+ *
+ * @param entry  Pointer to entry to display
+ */
+void dump_fwd_db_entry(fwd_db_entry_t *entry);
+
+/**
+ * Display the forwarding database
+ */
+void dump_fwd_db(void);
+
+/**
+ * Find a matching forwarding database entry
+ *
+ * @param dst_ip  Destination IPv4 address
+ *
+ * @return pointer to forwarding DB entry else NULL
+ */
+fwd_db_entry_t *find_fwd_db_entry(uint32_t dst_ip);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/example/m4/configure.m4 b/example/m4/configure.m4
index 9731d81..7868574 100644
--- a/example/m4/configure.m4
+++ b/example/m4/configure.m4
@@ -19,4 +19,5 @@  AC_CONFIG_FILES([example/classifier/Makefile
 		 example/timer/Makefile
 		 example/traffic_mgmt/Makefile
 		 example/l2fwd_simple/Makefile
+		 example/l3fwd/Makefile
 		 example/switch/Makefile])