| Message ID | 20151021113710.GA17144@localhost |
|---|---|
| State | New |
| Headers | show |
> -----Original Message----- > From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org] > Sent: Wednesday, October 21, 2015 2:37 PM > To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com> > Cc: lng-odp@lists.linaro.org > Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx and tx > descriptors > > On Wed, Oct 21, 2015 at 06:26:39AM +0000, Elo, Matias (Nokia - FI/Espoo) wrote: > > Hi, > > > > I'm still unable to reproduce the problem. Have you tried reloading the netmap > kernel module and driver? > > > > I found that if I apply only this patch and not the link state patch, it > works. > > The problem is related to the fact the ports are directly connected with > an external loopback cable, so when one goes down the other does too. > > odp_pktio_open(eth5) > [596320.186076] ixgbe 0000:04:00.0 eth1: NIC Link is Down > [596320.490564] ixgbe 0000:04:00.1 eth5: detected SFP+: 65535 > [596320.490604] ixgbe 0000:04:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: > RX/TX > odp_pktio_open(eth5) returns, test starts sending packets, but the other > end of the link didn't come up until ~700ms later.. > [596321.165492] ixgbe 0000:04:00.0 eth1: NIC Link is Up 10 Gbps, Flow Control: > RX/TX > > Looking at that, I'm not entirely sure how it works without the state > check, but it does work reliably. I guess it was hitting a timing window > and managing to squirt out the 1-4 packets used in the test before the > link bounced. > > Anyway, just moving the sleep so that it waits at least 1 second after > (this end of) the link comes up resolves it for me. > > It's a bit nasty doing this sort of thing in the pktio itself rather > than in the application, but the link bouncing is a particular netmap > quirk and we don't have a link state check API yet, so I think this > makes sense for now. > > Stuart. OK, on my system both ends of the link come up at about the same time. I just submitted a v2 of the patch, with the fix you suggested, if you could review it. -Matias > > diff --git a/platform/linux-generic/pktio/netmap.c b/platform/linux- > generic/pktio/netmap.c > index 9ad6a22..b3a2ca2 100644 > --- a/platform/linux-generic/pktio/netmap.c > +++ b/platform/linux-generic/pktio/netmap.c > @@ -171,9 +171,9 @@ static int netmap_open(odp_pktio_t id ODP_UNUSED, > pktio_entry_t *pktio_entry, > /* Wait for the link to come up */ > for (i = 0; i < NM_OPEN_RETRIES; i++) { > err = netmap_do_ioctl(pktio_entry, SIOCETHTOOL, ETHTOOL_GLINK); > + sleep(1); > if (err == 0) > return 0; > - sleep(1); > } > ODP_ERR("%s didn't come up\n", pktio_entry->s.name); > > > > > I rebased the particular two patches to the HEAD and this is the output from the > pktio validation test: > > > > $ sudo ODP_PKTIO_IF0=em49 ODP_PKTIO_IF1=em50 > ./test/validation/pktio/pktio_main > > odp_pool.c:102:odp_pool_init_global(): > > Pool init global > > odp_pool.c:103:odp_pool_init_global(): pool_entry_s size 320 > > odp_pool.c:104:odp_pool_init_global(): pool_entry_t size 320 > > odp_pool.c:105:odp_pool_init_global(): odp_buffer_hdr_t size 160 > > odp_pool.c:106:odp_pool_init_global(): > > odp_queue.c:109:odp_queue_init_global():Queue init ... > odp_queue.c:131:odp_queue_init_global():done > > odp_queue.c:132:odp_queue_init_global():Queue init global > > odp_queue.c:134:odp_queue_init_global(): struct queue_entry_s size 256 > > odp_queue.c:136:odp_queue_init_global(): queue_entry_t size 256 > > odp_queue.c:137:odp_queue_init_global(): > > odp_schedule.c:122:odp_schedule_init_global():Schedule init ... > odp_schedule.c:187:odp_schedule_init_global():done > > ODP API version: 1.3.0 > > ODP implementation version: linux-generic 1.3.0-0 (v1.3.0) Oct 21 2015 > 09:14:07 'https://git.linaro.org/lng/odp.git' (v1.3.0.0-46-ge4a5493-dirty) > > > > > > CUnit - A unit testing framework for C - Version 2.1-3 > > http://cunit.sourceforge.net/ > > > > Using paired interfaces: em49 em50 > > > > Suite: Packet I/O Unsegmented > > Test: pktio_test_open ...080.753804 nm_open [839] NIOCREGIF failed: No such > device or address nothere > > 080.753826 nm_open [839] NIOCREGIF failed: No such device or address > nothere > > pktio/netmap.c:145:netmap_open():nm_open(netmap:nothere) failed > > pktio/socket.c:107:mac_addr_get_fd():ioctl(SIOCGIFHWADDR): No such device: > "nothere". > > pktio/socket.c:237:sock_setup_pkt():ioctl(SIOCGIFINDEX): No such device: > "nothere". > > odp_packet_io.c:240:setup_pktio_entry():Unable to init any I/O type. > > passed > > Test: pktio_test_lookup ...passed > > Test: pktio_test_inq ...passed > > Test: pktio_test_poll_queue ...passed > > Test: pktio_test_poll_multi ...passed > > Test: pktio_test_sched_queue ...passed > > Test: pktio_test_sched_multi ...passed > > Test: pktio_test_jumbo ...FAILED > > 1. pktio.c:384 - CU_FAIL("failed to receive transmitted packet") > > 2. pktio.c:450 - i == num_pkts > > Test: pktio_test_mtu ... 1500 passed > > Test: pktio_test_promisc ...passed > > Test: pktio_test_mac ...testing mac for em49 > > 5C:B9:1:89:88:8 passed > > Test: pktio_test_inq_remdef ...passed > > Test: pktio_test_start_stop ...FAILED > > 1. pktio.c:775 - pkts == allocUsing paired interfaces: em49 em50 > > > > Suite: Packet I/O Segmented > > Test: pktio_test_poll_queue ...passed > > Test: pktio_test_poll_multi ...passed > > Test: pktio_test_sched_queue ...passed > > Test: pktio_test_sched_multi ...passed > > Test: pktio_test_jumbo ...FAILED > > 1. pktio.c:384 - CU_FAIL("failed to receive transmitted packet") > > 2. pktio.c:450 - i == num_pkts > > > > Run Summary: Type Total Ran Passed Failed Inactive > > suites 2 2 n/a 0 0 > > tests 18 18 15 3 0 > > asserts 296 296 291 5 n/a > > > > Elapsed time = 8.793 seconds > > > > -Matias > > > > > -----Original Message----- > > > From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org] > > > Sent: Tuesday, October 20, 2015 7:37 PM > > > To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com> > > > Cc: lng-odp@lists.linaro.org > > > Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx and > tx > > > descriptors > > > > > > On Tue, Oct 20, 2015 at 02:20:05PM +0000, Elo, Matias (Nokia - FI/Espoo) > wrote: > > > > > -----Original Message----- > > > > > From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org] > > > > > Sent: Tuesday, October 20, 2015 3:29 PM > > > > > To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia.com> > > > > > Cc: lng-odp@lists.linaro.org > > > > > Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx > and > > > tx > > > > > descriptors > > > > > > > > > > On Fri, Oct 16, 2015 at 03:45:50PM +0300, Matias Elo wrote: > > > > > > Using separate file descriptors enables using rx and tx > > > > > > functions simultaneously from different threads. > > > > > > > > > > > > Previously netmap tx queues were flushed only after no free > > > > > > slots were left which caused long delays on slow packet > > > > > > rates. Now queues are flushed after each packet > > > > > > burst. > > > > > > > > > > This isn't working for me. Same setup as earlier and I get similar > > > > > results. I've tried adding a NIOCTXSYNC on send and a NIOCRXSYNC on > recv > > > > > but it didn't seem to help. I'll look further to figure out if it's the > > > > > rx or tx side that's not working. > > > > > > > > > > > > > If you use e.g. ethtool, is the link between your nic ports detected? > > > > > > Yes, the link state check in odp_pktio_open() is detecting the link too. > > > > > > > I'm wondering if the problem is related to directly connecting two netmap > ports > > > on the same host. > > > > > > > > > > Possibly, although the same setup works fine on HEAD just with > > > NETMAP_NO_TX_POLL removed - not suggesting this as a fix, but it > > > seems to indicate my setup is alright. > > > > > > > -Matias > > > > > > > > > -- > > > > > Stuart. > > > > > > > > > > > Signed-off-by: Matias Elo <matias.elo@nokia.com> > > > > > > --- > > > > > > platform/linux-generic/include/odp_packet_netmap.h | 3 ++- > > > > > > platform/linux-generic/pktio/netmap.c | 29 ++++++++++++++--- > ----- > > > > > > 2 files changed, 21 insertions(+), 11 deletions(-) > > > > > > > > > > > > diff --git a/platform/linux-generic/include/odp_packet_netmap.h > > > > > b/platform/linux-generic/include/odp_packet_netmap.h > > > > > > index 23aea5b..0577dfe 100644 > > > > > > --- a/platform/linux-generic/include/odp_packet_netmap.h > > > > > > +++ b/platform/linux-generic/include/odp_packet_netmap.h > > > > > > @@ -15,7 +15,8 @@ > > > > > > typedef struct { > > > > > > odp_pool_t pool; /**< pool to alloc packets from */ > > > > > > size_t max_frame_len; /**< buf_size - sizeof(pkt_hdr) */ > > > > > > - struct nm_desc *desc; /**< netmap meta-data for the > device */ > > > > > > + struct nm_desc *rx_desc; /**< netmap meta-data for the > device */ > > > > > > + struct nm_desc *tx_desc; /**< netmap meta-data for the > device */ > > > > > > uint32_t if_flags; /**< interface flags */ > > > > > > int sockfd; /**< control socket */ > > > > > > unsigned char if_mac[ETH_ALEN]; /**< eth mac address */ > > > > > > diff --git a/platform/linux-generic/pktio/netmap.c b/platform/linux- > > > > > generic/pktio/netmap.c > > > > > > index 0c9ad35..9ad6a22 100644 > > > > > > --- a/platform/linux-generic/pktio/netmap.c > > > > > > +++ b/platform/linux-generic/pktio/netmap.c > > > > > > @@ -89,10 +89,13 @@ static int netmap_close(pktio_entry_t > *pktio_entry) > > > > > > { > > > > > > pkt_netmap_t *pkt_nm = &pktio_entry->s.pkt_nm; > > > > > > > > > > > > - if (pkt_nm->desc != NULL) { > > > > > > - nm_close(pkt_nm->desc); > > > > > > + if (pkt_nm->rx_desc != NULL) { > > > > > > + nm_close(pkt_nm->rx_desc); > > > > > > mmap_desc.mem = NULL; > > > > > > } > > > > > > + if (pkt_nm->tx_desc != NULL) > > > > > > + nm_close(pkt_nm->tx_desc); > > > > > > + > > > > > > if (pkt_nm->sockfd != -1 && close(pkt_nm->sockfd) != 0) { > > > > > > __odp_errno = errno; > > > > > > ODP_ERR("close(sockfd): %s\n", strerror(errno)); > > > > > > @@ -131,18 +134,21 @@ static int netmap_open(odp_pktio_t id > > > > > ODP_UNUSED, pktio_entry_t *pktio_entry, > > > > > > snprintf(ifname, sizeof(ifname), "netmap:%s", netdev); > > > > > > > > > > > > if (mmap_desc.mem == NULL) > > > > > > - pkt_nm->desc = nm_open(ifname, NULL, > > > > > NETMAP_NO_TX_POLL, NULL); > > > > > > + pkt_nm->rx_desc = nm_open(ifname, NULL, > > > > > NETMAP_NO_TX_POLL, > > > > > > + NULL); > > > > > > else > > > > > > - pkt_nm->desc = nm_open(ifname, NULL, > NETMAP_NO_TX_POLL > > > > > | > > > > > > - NM_OPEN_NO_MMAP, > &mmap_desc); > > > > > > - if (pkt_nm->desc == NULL) { > > > > > > + pkt_nm->rx_desc = nm_open(ifname, NULL, > > > > > NETMAP_NO_TX_POLL | > > > > > > + NM_OPEN_NO_MMAP, > &mmap_desc); > > > > > > + pkt_nm->tx_desc = nm_open(ifname, NULL, > NM_OPEN_NO_MMAP, > > > > > &mmap_desc); > > > > > > + > > > > > > + if (pkt_nm->rx_desc == NULL || pkt_nm->tx_desc == NULL) { > > > > > > ODP_ERR("nm_open(%s) failed\n", ifname); > > > > > > goto error; > > > > > > } > > > > > > > > > > > > if (mmap_desc.mem == NULL) { > > > > > > - mmap_desc.mem = pkt_nm->desc->mem; > > > > > > - mmap_desc.memsize = pkt_nm->desc->memsize; > > > > > > + mmap_desc.mem = pkt_nm->rx_desc->mem; > > > > > > + mmap_desc.memsize = pkt_nm->rx_desc->memsize; > > > > > > } > > > > > > > > > > > > sockfd = socket(AF_INET, SOCK_DGRAM, 0); > > > > > > @@ -215,7 +221,7 @@ static int netmap_recv(pktio_entry_t > *pktio_entry, > > > > > odp_packet_t pkt_table[], > > > > > > unsigned num) > > > > > > { > > > > > > struct dispatch_args args; > > > > > > - struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.desc; > > > > > > + struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.rx_desc; > > > > > > struct pollfd polld; > > > > > > > > > > > > polld.fd = nm_desc->fd; > > > > > > @@ -236,8 +242,8 @@ static int netmap_recv(pktio_entry_t > *pktio_entry, > > > > > odp_packet_t pkt_table[], > > > > > > static int netmap_send(pktio_entry_t *pktio_entry, odp_packet_t > > > pkt_table[], > > > > > > unsigned num) > > > > > > { > > > > > > - struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.desc; > > > > > > struct pollfd polld; > > > > > > + struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.tx_desc; > > > > > > unsigned i, nb_tx; > > > > > > uint8_t *frame; > > > > > > uint32_t frame_len; > > > > > > @@ -259,6 +265,9 @@ static int netmap_send(pktio_entry_t > *pktio_entry, > > > > > odp_packet_t pkt_table[], > > > > > > break; > > > > > > } > > > > > > } > > > > > > + /* Send pending packets */ > > > > > > + poll(&polld, 1, 0); > > > > > > + > > > > > > for (i = 0; i < nb_tx; i++) > > > > > > odp_packet_free(pkt_table[i]); > > > > > > > > > > > > -- > > > > > > 1.9.1
diff --git a/platform/linux-generic/pktio/netmap.c b/platform/linux-generic/pktio/netmap.c index 9ad6a22..b3a2ca2 100644 --- a/platform/linux-generic/pktio/netmap.c +++ b/platform/linux-generic/pktio/netmap.c @@ -171,9 +171,9 @@ static int netmap_open(odp_pktio_t id ODP_UNUSED, pktio_entry_t *pktio_entry, /* Wait for the link to come up */ for (i = 0; i < NM_OPEN_RETRIES; i++) { err = netmap_do_ioctl(pktio_entry, SIOCETHTOOL, ETHTOOL_GLINK); + sleep(1); if (err == 0) return 0; - sleep(1); } ODP_ERR("%s didn't come up\n", pktio_entry->s.name);