[2/2] linux-generic: netmap: use separate rx and tx descriptors

Message ID 20151021113710.GA17144@localhost
State New
Headers show

Commit Message

Stuart Haslam Oct. 21, 2015, 11:37 a.m.
On Wed, Oct 21, 2015 at 06:26:39AM +0000, Elo, Matias (Nokia - FI/Espoo) wrote:
> Hi, 
> 
> I'm still unable to reproduce the problem. Have you tried reloading the netmap kernel module and driver?
> 

I found that if I apply only this patch and not the link state patch, it
works.

The problem is related to the fact the ports are directly connected with
an external loopback cable, so when one goes down the other does too.

odp_pktio_open(eth5)
[596320.186076] ixgbe 0000:04:00.0 eth1: NIC Link is Down
[596320.490564] ixgbe 0000:04:00.1 eth5: detected SFP+: 65535
[596320.490604] ixgbe 0000:04:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: RX/TX
odp_pktio_open(eth5) returns, test starts sending packets, but the other
end of the link didn't come up until ~700ms later..
[596321.165492] ixgbe 0000:04:00.0 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Looking at that, I'm not entirely sure how it works without the state
check, but it does work reliably. I guess it was hitting a timing window
and managing to squirt out the 1-4 packets used in the test before the
link bounced.

Anyway, just moving the sleep so that it waits at least 1 second after
(this end of) the link comes up resolves it for me.

It's a bit nasty doing this sort of thing in the pktio itself rather
than in the application, but the link bouncing is a particular netmap
quirk and we don't have a link state check API yet, so I think this
makes sense for now.

Stuart.



> I rebased the particular two patches to the HEAD and this is the output from the pktio validation test:
> 
> $ sudo ODP_PKTIO_IF0=em49 ODP_PKTIO_IF1=em50 ./test/validation/pktio/pktio_main
> odp_pool.c:102:odp_pool_init_global():
> Pool init global
> odp_pool.c:103:odp_pool_init_global():  pool_entry_s size     320
> odp_pool.c:104:odp_pool_init_global():  pool_entry_t size     320
> odp_pool.c:105:odp_pool_init_global():  odp_buffer_hdr_t size 160
> odp_pool.c:106:odp_pool_init_global():
> odp_queue.c:109:odp_queue_init_global():Queue init ... odp_queue.c:131:odp_queue_init_global():done
> odp_queue.c:132:odp_queue_init_global():Queue init global
> odp_queue.c:134:odp_queue_init_global():  struct queue_entry_s size 256
> odp_queue.c:136:odp_queue_init_global():  queue_entry_t size        256
> odp_queue.c:137:odp_queue_init_global():
> odp_schedule.c:122:odp_schedule_init_global():Schedule init ... odp_schedule.c:187:odp_schedule_init_global():done
>         ODP API version: 1.3.0
>         ODP implementation version: linux-generic 1.3.0-0 (v1.3.0) Oct 21 2015 09:14:07 'https://git.linaro.org/lng/odp.git' (v1.3.0.0-46-ge4a5493-dirty)
> 
> 
>      CUnit - A unit testing framework for C - Version 2.1-3
>      http://cunit.sourceforge.net/
> 
> Using paired interfaces: em49 em50
> 
> Suite: Packet I/O Unsegmented
>   Test: pktio_test_open ...080.753804 nm_open [839] NIOCREGIF failed: No such device or address nothere
> 080.753826 nm_open [839] NIOCREGIF failed: No such device or address nothere
> pktio/netmap.c:145:netmap_open():nm_open(netmap:nothere) failed
> pktio/socket.c:107:mac_addr_get_fd():ioctl(SIOCGIFHWADDR): No such device: "nothere".
> pktio/socket.c:237:sock_setup_pkt():ioctl(SIOCGIFINDEX): No such device: "nothere".
> odp_packet_io.c:240:setup_pktio_entry():Unable to init any I/O type.
> passed
>   Test: pktio_test_lookup ...passed
>   Test: pktio_test_inq ...passed
>   Test: pktio_test_poll_queue ...passed
>   Test: pktio_test_poll_multi ...passed
>   Test: pktio_test_sched_queue ...passed
>   Test: pktio_test_sched_multi ...passed
>   Test: pktio_test_jumbo ...FAILED
>     1. pktio.c:384  - CU_FAIL("failed to receive transmitted packet")
>     2. pktio.c:450  - i == num_pkts
>   Test: pktio_test_mtu ... 1500 passed
>   Test: pktio_test_promisc ...passed
>   Test: pktio_test_mac ...testing mac for em49
>  5C:B9:1:89:88:8 passed
>   Test: pktio_test_inq_remdef ...passed
>   Test: pktio_test_start_stop ...FAILED
>     1. pktio.c:775  - pkts == allocUsing paired interfaces: em49 em50
> 
> Suite: Packet I/O Segmented
>   Test: pktio_test_poll_queue ...passed
>   Test: pktio_test_poll_multi ...passed
>   Test: pktio_test_sched_queue ...passed
>   Test: pktio_test_sched_multi ...passed
>   Test: pktio_test_jumbo ...FAILED
>     1. pktio.c:384  - CU_FAIL("failed to receive transmitted packet")
>     2. pktio.c:450  - i == num_pkts
> 
> Run Summary:    Type  Total    Ran Passed Failed Inactive
>               suites      2      2    n/a      0        0
>                tests     18     18     15      3        0
>              asserts    296    296    291      5      n/a
> 
> Elapsed time =    8.793 seconds
> 
> -Matias
> 
> > -----Original Message-----
> > From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org]
> > Sent: Tuesday, October 20, 2015 7:37 PM
> > To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com>
> > Cc: lng-odp@lists.linaro.org
> > Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx and tx
> > descriptors
> > 
> > On Tue, Oct 20, 2015 at 02:20:05PM +0000, Elo, Matias (Nokia - FI/Espoo) wrote:
> > > > -----Original Message-----
> > > > From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org]
> > > > Sent: Tuesday, October 20, 2015 3:29 PM
> > > > To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com>
> > > > Cc: lng-odp@lists.linaro.org
> > > > Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx and
> > tx
> > > > descriptors
> > > >
> > > > On Fri, Oct 16, 2015 at 03:45:50PM +0300, Matias Elo wrote:
> > > > > Using separate file descriptors enables using rx and tx
> > > > > functions simultaneously from different threads.
> > > > >
> > > > > Previously netmap tx queues were flushed only after no free
> > > > > slots were left which caused long delays on slow packet
> > > > > rates. Now queues are flushed after each packet
> > > > > burst.
> > > >
> > > > This isn't working for me. Same setup as earlier and I get similar
> > > > results. I've tried adding a NIOCTXSYNC on send and a NIOCRXSYNC on recv
> > > > but it didn't seem to help. I'll look further to figure out if it's the
> > > > rx or tx side that's not working.
> > > >
> > >
> > > If you use e.g. ethtool, is the link between your nic ports detected?
> > 
> > Yes, the link state check in odp_pktio_open() is detecting the link too.
> > 
> > > I'm wondering if the problem is related to directly connecting two netmap ports
> > on the same host.
> > >
> > 
> > Possibly, although the same setup works fine on HEAD just with
> > NETMAP_NO_TX_POLL removed - not suggesting this as a fix, but it
> > seems to indicate my setup is alright.
> > 
> > > -Matias
> > >
> > > > --
> > > > Stuart.
> > > >
> > > > > Signed-off-by: Matias Elo <matias.elo@nokia.com>
> > > > > ---
> > > > >  platform/linux-generic/include/odp_packet_netmap.h |  3 ++-
> > > > >  platform/linux-generic/pktio/netmap.c              | 29 ++++++++++++++--------
> > > > >  2 files changed, 21 insertions(+), 11 deletions(-)
> > > > >
> > > > > diff --git a/platform/linux-generic/include/odp_packet_netmap.h
> > > > b/platform/linux-generic/include/odp_packet_netmap.h
> > > > > index 23aea5b..0577dfe 100644
> > > > > --- a/platform/linux-generic/include/odp_packet_netmap.h
> > > > > +++ b/platform/linux-generic/include/odp_packet_netmap.h
> > > > > @@ -15,7 +15,8 @@
> > > > >  typedef struct {
> > > > >  	odp_pool_t pool;		/**< pool to alloc packets from */
> > > > >  	size_t max_frame_len;		/**< buf_size - sizeof(pkt_hdr) */
> > > > > -	struct nm_desc *desc;		/**< netmap meta-data for the device */
> > > > > +	struct nm_desc *rx_desc;	/**< netmap meta-data for the device */
> > > > > +	struct nm_desc *tx_desc;	/**< netmap meta-data for the device */
> > > > >  	uint32_t if_flags;		/**< interface flags */
> > > > >  	int sockfd;			/**< control socket */
> > > > >  	unsigned char if_mac[ETH_ALEN]; /**< eth mac address */
> > > > > diff --git a/platform/linux-generic/pktio/netmap.c b/platform/linux-
> > > > generic/pktio/netmap.c
> > > > > index 0c9ad35..9ad6a22 100644
> > > > > --- a/platform/linux-generic/pktio/netmap.c
> > > > > +++ b/platform/linux-generic/pktio/netmap.c
> > > > > @@ -89,10 +89,13 @@ static int netmap_close(pktio_entry_t *pktio_entry)
> > > > >  {
> > > > >  	pkt_netmap_t *pkt_nm = &pktio_entry->s.pkt_nm;
> > > > >
> > > > > -	if (pkt_nm->desc != NULL) {
> > > > > -		nm_close(pkt_nm->desc);
> > > > > +	if (pkt_nm->rx_desc != NULL) {
> > > > > +		nm_close(pkt_nm->rx_desc);
> > > > >  		mmap_desc.mem = NULL;
> > > > >  	}
> > > > > +	if (pkt_nm->tx_desc != NULL)
> > > > > +		nm_close(pkt_nm->tx_desc);
> > > > > +
> > > > >  	if (pkt_nm->sockfd != -1 && close(pkt_nm->sockfd) != 0) {
> > > > >  		__odp_errno = errno;
> > > > >  		ODP_ERR("close(sockfd): %s\n", strerror(errno));
> > > > > @@ -131,18 +134,21 @@ static int netmap_open(odp_pktio_t id
> > > > ODP_UNUSED, pktio_entry_t *pktio_entry,
> > > > >  	snprintf(ifname, sizeof(ifname), "netmap:%s", netdev);
> > > > >
> > > > >  	if (mmap_desc.mem == NULL)
> > > > > -		pkt_nm->desc = nm_open(ifname, NULL,
> > > > NETMAP_NO_TX_POLL, NULL);
> > > > > +		pkt_nm->rx_desc = nm_open(ifname, NULL,
> > > > NETMAP_NO_TX_POLL,
> > > > > +					  NULL);
> > > > >  	else
> > > > > -		pkt_nm->desc = nm_open(ifname, NULL, NETMAP_NO_TX_POLL
> > > > |
> > > > > -				       NM_OPEN_NO_MMAP, &mmap_desc);
> > > > > -	if (pkt_nm->desc == NULL) {
> > > > > +		pkt_nm->rx_desc = nm_open(ifname, NULL,
> > > > NETMAP_NO_TX_POLL |
> > > > > +					  NM_OPEN_NO_MMAP, &mmap_desc);
> > > > > +	pkt_nm->tx_desc = nm_open(ifname, NULL, NM_OPEN_NO_MMAP,
> > > > &mmap_desc);
> > > > > +
> > > > > +	if (pkt_nm->rx_desc == NULL || pkt_nm->tx_desc == NULL) {
> > > > >  		ODP_ERR("nm_open(%s) failed\n", ifname);
> > > > >  		goto error;
> > > > >  	}
> > > > >
> > > > >  	if (mmap_desc.mem == NULL) {
> > > > > -		mmap_desc.mem = pkt_nm->desc->mem;
> > > > > -		mmap_desc.memsize = pkt_nm->desc->memsize;
> > > > > +		mmap_desc.mem = pkt_nm->rx_desc->mem;
> > > > > +		mmap_desc.memsize = pkt_nm->rx_desc->memsize;
> > > > >  	}
> > > > >
> > > > >  	sockfd = socket(AF_INET, SOCK_DGRAM, 0);
> > > > > @@ -215,7 +221,7 @@ static int netmap_recv(pktio_entry_t *pktio_entry,
> > > > odp_packet_t pkt_table[],
> > > > >  		       unsigned num)
> > > > >  {
> > > > >  	struct dispatch_args args;
> > > > > -	struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.desc;
> > > > > +	struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.rx_desc;
> > > > >  	struct pollfd polld;
> > > > >
> > > > >  	polld.fd = nm_desc->fd;
> > > > > @@ -236,8 +242,8 @@ static int netmap_recv(pktio_entry_t *pktio_entry,
> > > > odp_packet_t pkt_table[],
> > > > >  static int netmap_send(pktio_entry_t *pktio_entry, odp_packet_t
> > pkt_table[],
> > > > >  		       unsigned num)
> > > > >  {
> > > > > -	struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.desc;
> > > > >  	struct pollfd polld;
> > > > > +	struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.tx_desc;
> > > > >  	unsigned i, nb_tx;
> > > > >  	uint8_t *frame;
> > > > >  	uint32_t frame_len;
> > > > > @@ -259,6 +265,9 @@ static int netmap_send(pktio_entry_t *pktio_entry,
> > > > odp_packet_t pkt_table[],
> > > > >  			break;
> > > > >  		}
> > > > >  	}
> > > > > +	/* Send pending packets */
> > > > > +	poll(&polld, 1, 0);
> > > > > +
> > > > >  	for (i = 0; i < nb_tx; i++)
> > > > >  		odp_packet_free(pkt_table[i]);
> > > > >
> > > > > --
> > > > > 1.9.1

Comments

Elo, Matias (Nokia - FI/Espoo) Oct. 22, 2015, 11:56 a.m. | #1
> -----Original Message-----
> From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org]
> Sent: Wednesday, October 21, 2015 2:37 PM
> To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com>
> Cc: lng-odp@lists.linaro.org
> Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx and tx
> descriptors
> 
> On Wed, Oct 21, 2015 at 06:26:39AM +0000, Elo, Matias (Nokia - FI/Espoo) wrote:
> > Hi,
> >
> > I'm still unable to reproduce the problem. Have you tried reloading the netmap
> kernel module and driver?
> >
> 
> I found that if I apply only this patch and not the link state patch, it
> works.
> 
> The problem is related to the fact the ports are directly connected with
> an external loopback cable, so when one goes down the other does too.
> 
> odp_pktio_open(eth5)
> [596320.186076] ixgbe 0000:04:00.0 eth1: NIC Link is Down
> [596320.490564] ixgbe 0000:04:00.1 eth5: detected SFP+: 65535
> [596320.490604] ixgbe 0000:04:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control:
> RX/TX
> odp_pktio_open(eth5) returns, test starts sending packets, but the other
> end of the link didn't come up until ~700ms later..
> [596321.165492] ixgbe 0000:04:00.0 eth1: NIC Link is Up 10 Gbps, Flow Control:
> RX/TX
> 
> Looking at that, I'm not entirely sure how it works without the state
> check, but it does work reliably. I guess it was hitting a timing window
> and managing to squirt out the 1-4 packets used in the test before the
> link bounced.
> 
> Anyway, just moving the sleep so that it waits at least 1 second after
> (this end of) the link comes up resolves it for me.
> 
> It's a bit nasty doing this sort of thing in the pktio itself rather
> than in the application, but the link bouncing is a particular netmap
> quirk and we don't have a link state check API yet, so I think this
> makes sense for now.
> 
> Stuart.

OK, on my system both ends of the link come up at about the same time. I just submitted a v2 of the patch, with the fix you suggested, if you could review it.

-Matias


> 
> diff --git a/platform/linux-generic/pktio/netmap.c b/platform/linux-
> generic/pktio/netmap.c
> index 9ad6a22..b3a2ca2 100644
> --- a/platform/linux-generic/pktio/netmap.c
> +++ b/platform/linux-generic/pktio/netmap.c
> @@ -171,9 +171,9 @@ static int netmap_open(odp_pktio_t id ODP_UNUSED,
> pktio_entry_t *pktio_entry,
>         /* Wait for the link to come up */
>         for (i = 0; i < NM_OPEN_RETRIES; i++) {
>                 err = netmap_do_ioctl(pktio_entry, SIOCETHTOOL, ETHTOOL_GLINK);
> +               sleep(1);
>                 if (err == 0)
>                         return 0;
> -               sleep(1);
>         }
>         ODP_ERR("%s didn't come up\n", pktio_entry->s.name);
> 
> 
> 
> > I rebased the particular two patches to the HEAD and this is the output from the
> pktio validation test:
> >
> > $ sudo ODP_PKTIO_IF0=em49 ODP_PKTIO_IF1=em50
> ./test/validation/pktio/pktio_main
> > odp_pool.c:102:odp_pool_init_global():
> > Pool init global
> > odp_pool.c:103:odp_pool_init_global():  pool_entry_s size     320
> > odp_pool.c:104:odp_pool_init_global():  pool_entry_t size     320
> > odp_pool.c:105:odp_pool_init_global():  odp_buffer_hdr_t size 160
> > odp_pool.c:106:odp_pool_init_global():
> > odp_queue.c:109:odp_queue_init_global():Queue init ...
> odp_queue.c:131:odp_queue_init_global():done
> > odp_queue.c:132:odp_queue_init_global():Queue init global
> > odp_queue.c:134:odp_queue_init_global():  struct queue_entry_s size 256
> > odp_queue.c:136:odp_queue_init_global():  queue_entry_t size        256
> > odp_queue.c:137:odp_queue_init_global():
> > odp_schedule.c:122:odp_schedule_init_global():Schedule init ...
> odp_schedule.c:187:odp_schedule_init_global():done
> >         ODP API version: 1.3.0
> >         ODP implementation version: linux-generic 1.3.0-0 (v1.3.0) Oct 21 2015
> 09:14:07 'https://git.linaro.org/lng/odp.git' (v1.3.0.0-46-ge4a5493-dirty)
> >
> >
> >      CUnit - A unit testing framework for C - Version 2.1-3
> >      http://cunit.sourceforge.net/
> >
> > Using paired interfaces: em49 em50
> >
> > Suite: Packet I/O Unsegmented
> >   Test: pktio_test_open ...080.753804 nm_open [839] NIOCREGIF failed: No such
> device or address nothere
> > 080.753826 nm_open [839] NIOCREGIF failed: No such device or address
> nothere
> > pktio/netmap.c:145:netmap_open():nm_open(netmap:nothere) failed
> > pktio/socket.c:107:mac_addr_get_fd():ioctl(SIOCGIFHWADDR): No such device:
> "nothere".
> > pktio/socket.c:237:sock_setup_pkt():ioctl(SIOCGIFINDEX): No such device:
> "nothere".
> > odp_packet_io.c:240:setup_pktio_entry():Unable to init any I/O type.
> > passed
> >   Test: pktio_test_lookup ...passed
> >   Test: pktio_test_inq ...passed
> >   Test: pktio_test_poll_queue ...passed
> >   Test: pktio_test_poll_multi ...passed
> >   Test: pktio_test_sched_queue ...passed
> >   Test: pktio_test_sched_multi ...passed
> >   Test: pktio_test_jumbo ...FAILED
> >     1. pktio.c:384  - CU_FAIL("failed to receive transmitted packet")
> >     2. pktio.c:450  - i == num_pkts
> >   Test: pktio_test_mtu ... 1500 passed
> >   Test: pktio_test_promisc ...passed
> >   Test: pktio_test_mac ...testing mac for em49
> >  5C:B9:1:89:88:8 passed
> >   Test: pktio_test_inq_remdef ...passed
> >   Test: pktio_test_start_stop ...FAILED
> >     1. pktio.c:775  - pkts == allocUsing paired interfaces: em49 em50
> >
> > Suite: Packet I/O Segmented
> >   Test: pktio_test_poll_queue ...passed
> >   Test: pktio_test_poll_multi ...passed
> >   Test: pktio_test_sched_queue ...passed
> >   Test: pktio_test_sched_multi ...passed
> >   Test: pktio_test_jumbo ...FAILED
> >     1. pktio.c:384  - CU_FAIL("failed to receive transmitted packet")
> >     2. pktio.c:450  - i == num_pkts
> >
> > Run Summary:    Type  Total    Ran Passed Failed Inactive
> >               suites      2      2    n/a      0        0
> >                tests     18     18     15      3        0
> >              asserts    296    296    291      5      n/a
> >
> > Elapsed time =    8.793 seconds
> >
> > -Matias
> >
> > > -----Original Message-----
> > > From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org]
> > > Sent: Tuesday, October 20, 2015 7:37 PM
> > > To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com>
> > > Cc: lng-odp@lists.linaro.org
> > > Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx and
> tx
> > > descriptors
> > >
> > > On Tue, Oct 20, 2015 at 02:20:05PM +0000, Elo, Matias (Nokia - FI/Espoo)
> wrote:
> > > > > -----Original Message-----
> > > > > From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org]
> > > > > Sent: Tuesday, October 20, 2015 3:29 PM
> > > > > To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com>
> > > > > Cc: lng-odp@lists.linaro.org
> > > > > Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx
> and
> > > tx
> > > > > descriptors
> > > > >
> > > > > On Fri, Oct 16, 2015 at 03:45:50PM +0300, Matias Elo wrote:
> > > > > > Using separate file descriptors enables using rx and tx
> > > > > > functions simultaneously from different threads.
> > > > > >
> > > > > > Previously netmap tx queues were flushed only after no free
> > > > > > slots were left which caused long delays on slow packet
> > > > > > rates. Now queues are flushed after each packet
> > > > > > burst.
> > > > >
> > > > > This isn't working for me. Same setup as earlier and I get similar
> > > > > results. I've tried adding a NIOCTXSYNC on send and a NIOCRXSYNC on
> recv
> > > > > but it didn't seem to help. I'll look further to figure out if it's the
> > > > > rx or tx side that's not working.
> > > > >
> > > >
> > > > If you use e.g. ethtool, is the link between your nic ports detected?
> > >
> > > Yes, the link state check in odp_pktio_open() is detecting the link too.
> > >
> > > > I'm wondering if the problem is related to directly connecting two netmap
> ports
> > > on the same host.
> > > >
> > >
> > > Possibly, although the same setup works fine on HEAD just with
> > > NETMAP_NO_TX_POLL removed - not suggesting this as a fix, but it
> > > seems to indicate my setup is alright.
> > >
> > > > -Matias
> > > >
> > > > > --
> > > > > Stuart.
> > > > >
> > > > > > Signed-off-by: Matias Elo <matias.elo@nokia.com>
> > > > > > ---
> > > > > >  platform/linux-generic/include/odp_packet_netmap.h |  3 ++-
> > > > > >  platform/linux-generic/pktio/netmap.c              | 29 ++++++++++++++---
> -----
> > > > > >  2 files changed, 21 insertions(+), 11 deletions(-)
> > > > > >
> > > > > > diff --git a/platform/linux-generic/include/odp_packet_netmap.h
> > > > > b/platform/linux-generic/include/odp_packet_netmap.h
> > > > > > index 23aea5b..0577dfe 100644
> > > > > > --- a/platform/linux-generic/include/odp_packet_netmap.h
> > > > > > +++ b/platform/linux-generic/include/odp_packet_netmap.h
> > > > > > @@ -15,7 +15,8 @@
> > > > > >  typedef struct {
> > > > > >  	odp_pool_t pool;		/**< pool to alloc packets from */
> > > > > >  	size_t max_frame_len;		/**< buf_size - sizeof(pkt_hdr) */
> > > > > > -	struct nm_desc *desc;		/**< netmap meta-data for the
> device */
> > > > > > +	struct nm_desc *rx_desc;	/**< netmap meta-data for the
> device */
> > > > > > +	struct nm_desc *tx_desc;	/**< netmap meta-data for the
> device */
> > > > > >  	uint32_t if_flags;		/**< interface flags */
> > > > > >  	int sockfd;			/**< control socket */
> > > > > >  	unsigned char if_mac[ETH_ALEN]; /**< eth mac address */
> > > > > > diff --git a/platform/linux-generic/pktio/netmap.c b/platform/linux-
> > > > > generic/pktio/netmap.c
> > > > > > index 0c9ad35..9ad6a22 100644
> > > > > > --- a/platform/linux-generic/pktio/netmap.c
> > > > > > +++ b/platform/linux-generic/pktio/netmap.c
> > > > > > @@ -89,10 +89,13 @@ static int netmap_close(pktio_entry_t
> *pktio_entry)
> > > > > >  {
> > > > > >  	pkt_netmap_t *pkt_nm = &pktio_entry->s.pkt_nm;
> > > > > >
> > > > > > -	if (pkt_nm->desc != NULL) {
> > > > > > -		nm_close(pkt_nm->desc);
> > > > > > +	if (pkt_nm->rx_desc != NULL) {
> > > > > > +		nm_close(pkt_nm->rx_desc);
> > > > > >  		mmap_desc.mem = NULL;
> > > > > >  	}
> > > > > > +	if (pkt_nm->tx_desc != NULL)
> > > > > > +		nm_close(pkt_nm->tx_desc);
> > > > > > +
> > > > > >  	if (pkt_nm->sockfd != -1 && close(pkt_nm->sockfd) != 0) {
> > > > > >  		__odp_errno = errno;
> > > > > >  		ODP_ERR("close(sockfd): %s\n", strerror(errno));
> > > > > > @@ -131,18 +134,21 @@ static int netmap_open(odp_pktio_t id
> > > > > ODP_UNUSED, pktio_entry_t *pktio_entry,
> > > > > >  	snprintf(ifname, sizeof(ifname), "netmap:%s", netdev);
> > > > > >
> > > > > >  	if (mmap_desc.mem == NULL)
> > > > > > -		pkt_nm->desc = nm_open(ifname, NULL,
> > > > > NETMAP_NO_TX_POLL, NULL);
> > > > > > +		pkt_nm->rx_desc = nm_open(ifname, NULL,
> > > > > NETMAP_NO_TX_POLL,
> > > > > > +					  NULL);
> > > > > >  	else
> > > > > > -		pkt_nm->desc = nm_open(ifname, NULL,
> NETMAP_NO_TX_POLL
> > > > > |
> > > > > > -				       NM_OPEN_NO_MMAP,
> &mmap_desc);
> > > > > > -	if (pkt_nm->desc == NULL) {
> > > > > > +		pkt_nm->rx_desc = nm_open(ifname, NULL,
> > > > > NETMAP_NO_TX_POLL |
> > > > > > +					  NM_OPEN_NO_MMAP,
> &mmap_desc);
> > > > > > +	pkt_nm->tx_desc = nm_open(ifname, NULL,
> NM_OPEN_NO_MMAP,
> > > > > &mmap_desc);
> > > > > > +
> > > > > > +	if (pkt_nm->rx_desc == NULL || pkt_nm->tx_desc == NULL) {
> > > > > >  		ODP_ERR("nm_open(%s) failed\n", ifname);
> > > > > >  		goto error;
> > > > > >  	}
> > > > > >
> > > > > >  	if (mmap_desc.mem == NULL) {
> > > > > > -		mmap_desc.mem = pkt_nm->desc->mem;
> > > > > > -		mmap_desc.memsize = pkt_nm->desc->memsize;
> > > > > > +		mmap_desc.mem = pkt_nm->rx_desc->mem;
> > > > > > +		mmap_desc.memsize = pkt_nm->rx_desc->memsize;
> > > > > >  	}
> > > > > >
> > > > > >  	sockfd = socket(AF_INET, SOCK_DGRAM, 0);
> > > > > > @@ -215,7 +221,7 @@ static int netmap_recv(pktio_entry_t
> *pktio_entry,
> > > > > odp_packet_t pkt_table[],
> > > > > >  		       unsigned num)
> > > > > >  {
> > > > > >  	struct dispatch_args args;
> > > > > > -	struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.desc;
> > > > > > +	struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.rx_desc;
> > > > > >  	struct pollfd polld;
> > > > > >
> > > > > >  	polld.fd = nm_desc->fd;
> > > > > > @@ -236,8 +242,8 @@ static int netmap_recv(pktio_entry_t
> *pktio_entry,
> > > > > odp_packet_t pkt_table[],
> > > > > >  static int netmap_send(pktio_entry_t *pktio_entry, odp_packet_t
> > > pkt_table[],
> > > > > >  		       unsigned num)
> > > > > >  {
> > > > > > -	struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.desc;
> > > > > >  	struct pollfd polld;
> > > > > > +	struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.tx_desc;
> > > > > >  	unsigned i, nb_tx;
> > > > > >  	uint8_t *frame;
> > > > > >  	uint32_t frame_len;
> > > > > > @@ -259,6 +265,9 @@ static int netmap_send(pktio_entry_t
> *pktio_entry,
> > > > > odp_packet_t pkt_table[],
> > > > > >  			break;
> > > > > >  		}
> > > > > >  	}
> > > > > > +	/* Send pending packets */
> > > > > > +	poll(&polld, 1, 0);
> > > > > > +
> > > > > >  	for (i = 0; i < nb_tx; i++)
> > > > > >  		odp_packet_free(pkt_table[i]);
> > > > > >
> > > > > > --
> > > > > > 1.9.1

Patch

diff --git a/platform/linux-generic/pktio/netmap.c b/platform/linux-generic/pktio/netmap.c
index 9ad6a22..b3a2ca2 100644
--- a/platform/linux-generic/pktio/netmap.c
+++ b/platform/linux-generic/pktio/netmap.c
@@ -171,9 +171,9 @@  static int netmap_open(odp_pktio_t id ODP_UNUSED, pktio_entry_t *pktio_entry,
        /* Wait for the link to come up */
        for (i = 0; i < NM_OPEN_RETRIES; i++) {
                err = netmap_do_ioctl(pktio_entry, SIOCETHTOOL, ETHTOOL_GLINK);
+               sleep(1);
                if (err == 0)
                        return 0;
-               sleep(1);
        }
        ODP_ERR("%s didn't come up\n", pktio_entry->s.name);