From patchwork Tue Jul 4 10:23:58 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianbo Liu X-Patchwork-Id: 106930 Delivered-To: patch@linaro.org Received: by 10.182.135.102 with SMTP id pr6csp854123obb; Tue, 4 Jul 2017 03:25:13 -0700 (PDT) X-Received: by 10.28.109.210 with SMTP id b79mr26345815wmi.34.1499163913119; Tue, 04 Jul 2017 03:25:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1499163913; cv=none; d=google.com; s=arc-20160816; b=YiiAzMNRSGYSW5BJuC1OYrDi5MZztmjQeB9Dzhiq/KCex1csLa4p+lguNKMMZNpsCb PFQkGpOuL5v7EaqzUMbKoMA9MYeWcWYrDt/r9jrEl2vE45DAHYiyNyWafb47MaNByY0o ndOKyYThP6Lq0d2n8rclZs6l2H8m01y40mMOWu3NACKRJJrRtfesWUP5NZHL7VCHwyEf /613ylKit4hhf47gboAYBr0DmLbPeB19LENcChyotwW53naieiI6lH/iZJSVoHZ0ImEZ PorObfEKMeXCb4duiHPVUPi7Fxyu1liO8+q+wCRVMKImOTFa7B0uVo23XxfPYveqJRrv D4Uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:cc:to:from:arc-authentication-results; bh=dcS6pn6l2Ba09nAMmLi3x2XCreKrxW+MHXLMQduEVvo=; b=A+qbmFwQEQP1fhaA3dCwb96Sx/wWnvdz71tft2p/AyG2hH1A/8JaxGHTdPUFEjMP7Z mWogdz0puD1cctFEhoJGJg1VSC1Koo5jNZ7GuRXKJv74iLy0N1qcbV/AYUoyQipcS9V9 9FR/IzVClBSEzTI+lDlJe1h5ihZppg2Pa4q5AmqmZk4ss40s94AdU8k+aZz11sfYH2FG p3l2AgstHyb0XTmchaIyQRGVDehovC6dl/T9l8gcy+oyZs/86Ht+hwphOG2OwXY68hNd nNgd9fS1wFCg0bVvHmJvzWsP3+2AEO7d8LxKSWKTyepK7fVgrm156/s8gSOOsxdXuG4n tURA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 2001:4b98:dc0:41:216:3eff:fe72:dd13 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from dpdk.org ([2001:4b98:dc0:41:216:3eff:fe72:dd13]) by mx.google.com with ESMTP id g145si12214178wmg.198.2017.07.04.03.25.12; Tue, 04 Jul 2017 03:25:13 -0700 (PDT) Received-SPF: pass (google.com: domain of dev-bounces@dpdk.org designates 2001:4b98:dc0:41:216:3eff:fe72:dd13 as permitted sender) client-ip=2001:4b98:dc0:41:216:3eff:fe72:dd13; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 2001:4b98:dc0:41:216:3eff:fe72:dd13 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 441FF5592; Tue, 4 Jul 2017 12:25:06 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.101.70]) by dpdk.org (Postfix) with ESMTP id 57A043195 for ; Tue, 4 Jul 2017 12:25:03 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C15A21596; Tue, 4 Jul 2017 03:25:02 -0700 (PDT) Received: from localhost.localdomain.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EB57E3F581; Tue, 4 Jul 2017 03:25:01 -0700 (PDT) From: Jianbo Liu To: dev@dpdk.org, tomasz.kantecki@intel.com, jerin.jacob@caviumnetworks.com, ashwin.sekhar@caviumnetworks.com Cc: Jianbo Liu Date: Tue, 4 Jul 2017 18:23:58 +0800 Message-Id: <1499163845-6674-2-git-send-email-jianbo.liu@linaro.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1499163845-6674-1-git-send-email-jianbo.liu@linaro.org> References: <1493709255-8887-1-git-send-email-jianbo.liu@linaro.org> <1499163845-6674-1-git-send-email-jianbo.liu@linaro.org> Subject: [dpdk-dev] [PATCH v5 1/8] examples/l3fwd: extract arch independent code from multi hash lookup X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Extract common code from l3fwd_em_hlm_sse.h, and add to the new file l3fwd_em_hlm.h. Signed-off-by: Jianbo Liu --- examples/l3fwd/l3fwd_em.c | 2 +- examples/l3fwd/l3fwd_em_hlm.h | 302 ++++++++++++++++++++++++++++++++++++++ examples/l3fwd/l3fwd_em_hlm_sse.h | 276 +--------------------------------- 3 files changed, 308 insertions(+), 272 deletions(-) create mode 100644 examples/l3fwd/l3fwd_em_hlm.h -- 1.8.3.1 diff --git a/examples/l3fwd/l3fwd_em.c b/examples/l3fwd/l3fwd_em.c index 9cc4460..939a16d 100644 --- a/examples/l3fwd/l3fwd_em.c +++ b/examples/l3fwd/l3fwd_em.c @@ -332,7 +332,7 @@ struct ipv6_l3fwd_em_route { #if defined(NO_HASH_MULTI_LOOKUP) #include "l3fwd_em_sse.h" #else -#include "l3fwd_em_hlm_sse.h" +#include "l3fwd_em_hlm.h" #endif #else #include "l3fwd_em.h" diff --git a/examples/l3fwd/l3fwd_em_hlm.h b/examples/l3fwd/l3fwd_em_hlm.h new file mode 100644 index 0000000..9fb5ff6 --- /dev/null +++ b/examples/l3fwd/l3fwd_em_hlm.h @@ -0,0 +1,302 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016 Intel Corporation. All rights reserved. + * Copyright(c) 2017, Linaro Limited + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef __L3FWD_EM_HLM_H__ +#define __L3FWD_EM_HLM_H__ + +#include "l3fwd_sse.h" +#include "l3fwd_em_hlm_sse.h" + +static __rte_always_inline void +em_get_dst_port_ipv4x8(struct lcore_conf *qconf, struct rte_mbuf *m[8], + uint8_t portid, uint16_t dst_port[8]) +{ + int32_t ret[8]; + union ipv4_5tuple_host key[8]; + + get_ipv4_5tuple(m[0], mask0.x, &key[0]); + get_ipv4_5tuple(m[1], mask0.x, &key[1]); + get_ipv4_5tuple(m[2], mask0.x, &key[2]); + get_ipv4_5tuple(m[3], mask0.x, &key[3]); + get_ipv4_5tuple(m[4], mask0.x, &key[4]); + get_ipv4_5tuple(m[5], mask0.x, &key[5]); + get_ipv4_5tuple(m[6], mask0.x, &key[6]); + get_ipv4_5tuple(m[7], mask0.x, &key[7]); + + const void *key_array[8] = {&key[0], &key[1], &key[2], &key[3], + &key[4], &key[5], &key[6], &key[7]}; + + rte_hash_lookup_bulk(qconf->ipv4_lookup_struct, &key_array[0], 8, ret); + + dst_port[0] = (uint8_t) ((ret[0] < 0) ? + portid : ipv4_l3fwd_out_if[ret[0]]); + dst_port[1] = (uint8_t) ((ret[1] < 0) ? + portid : ipv4_l3fwd_out_if[ret[1]]); + dst_port[2] = (uint8_t) ((ret[2] < 0) ? + portid : ipv4_l3fwd_out_if[ret[2]]); + dst_port[3] = (uint8_t) ((ret[3] < 0) ? + portid : ipv4_l3fwd_out_if[ret[3]]); + dst_port[4] = (uint8_t) ((ret[4] < 0) ? + portid : ipv4_l3fwd_out_if[ret[4]]); + dst_port[5] = (uint8_t) ((ret[5] < 0) ? + portid : ipv4_l3fwd_out_if[ret[5]]); + dst_port[6] = (uint8_t) ((ret[6] < 0) ? + portid : ipv4_l3fwd_out_if[ret[6]]); + dst_port[7] = (uint8_t) ((ret[7] < 0) ? + portid : ipv4_l3fwd_out_if[ret[7]]); + + if (dst_port[0] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[0]) == 0) + dst_port[0] = portid; + + if (dst_port[1] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[1]) == 0) + dst_port[1] = portid; + + if (dst_port[2] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[2]) == 0) + dst_port[2] = portid; + + if (dst_port[3] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[3]) == 0) + dst_port[3] = portid; + + if (dst_port[4] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[4]) == 0) + dst_port[4] = portid; + + if (dst_port[5] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[5]) == 0) + dst_port[5] = portid; + + if (dst_port[6] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[6]) == 0) + dst_port[6] = portid; + + if (dst_port[7] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[7]) == 0) + dst_port[7] = portid; + +} + +static __rte_always_inline void +em_get_dst_port_ipv6x8(struct lcore_conf *qconf, struct rte_mbuf *m[8], + uint8_t portid, uint16_t dst_port[8]) +{ + int32_t ret[8]; + union ipv6_5tuple_host key[8]; + + get_ipv6_5tuple(m[0], mask1.x, mask2.x, &key[0]); + get_ipv6_5tuple(m[1], mask1.x, mask2.x, &key[1]); + get_ipv6_5tuple(m[2], mask1.x, mask2.x, &key[2]); + get_ipv6_5tuple(m[3], mask1.x, mask2.x, &key[3]); + get_ipv6_5tuple(m[4], mask1.x, mask2.x, &key[4]); + get_ipv6_5tuple(m[5], mask1.x, mask2.x, &key[5]); + get_ipv6_5tuple(m[6], mask1.x, mask2.x, &key[6]); + get_ipv6_5tuple(m[7], mask1.x, mask2.x, &key[7]); + + const void *key_array[8] = {&key[0], &key[1], &key[2], &key[3], + &key[4], &key[5], &key[6], &key[7]}; + + rte_hash_lookup_bulk(qconf->ipv6_lookup_struct, &key_array[0], 8, ret); + + dst_port[0] = (uint8_t) ((ret[0] < 0) ? + portid : ipv6_l3fwd_out_if[ret[0]]); + dst_port[1] = (uint8_t) ((ret[1] < 0) ? + portid : ipv6_l3fwd_out_if[ret[1]]); + dst_port[2] = (uint8_t) ((ret[2] < 0) ? + portid : ipv6_l3fwd_out_if[ret[2]]); + dst_port[3] = (uint8_t) ((ret[3] < 0) ? + portid : ipv6_l3fwd_out_if[ret[3]]); + dst_port[4] = (uint8_t) ((ret[4] < 0) ? + portid : ipv6_l3fwd_out_if[ret[4]]); + dst_port[5] = (uint8_t) ((ret[5] < 0) ? + portid : ipv6_l3fwd_out_if[ret[5]]); + dst_port[6] = (uint8_t) ((ret[6] < 0) ? + portid : ipv6_l3fwd_out_if[ret[6]]); + dst_port[7] = (uint8_t) ((ret[7] < 0) ? + portid : ipv6_l3fwd_out_if[ret[7]]); + + if (dst_port[0] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[0]) == 0) + dst_port[0] = portid; + + if (dst_port[1] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[1]) == 0) + dst_port[1] = portid; + + if (dst_port[2] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[2]) == 0) + dst_port[2] = portid; + + if (dst_port[3] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[3]) == 0) + dst_port[3] = portid; + + if (dst_port[4] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[4]) == 0) + dst_port[4] = portid; + + if (dst_port[5] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[5]) == 0) + dst_port[5] = portid; + + if (dst_port[6] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[6]) == 0) + dst_port[6] = portid; + + if (dst_port[7] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[7]) == 0) + dst_port[7] = portid; + +} + +static __rte_always_inline uint16_t +em_get_dst_port(const struct lcore_conf *qconf, struct rte_mbuf *pkt, + uint8_t portid) +{ + uint8_t next_hop; + struct ipv4_hdr *ipv4_hdr; + struct ipv6_hdr *ipv6_hdr; + uint32_t tcp_or_udp; + uint32_t l3_ptypes; + + tcp_or_udp = pkt->packet_type & (RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP); + l3_ptypes = pkt->packet_type & RTE_PTYPE_L3_MASK; + + if (tcp_or_udp && (l3_ptypes == RTE_PTYPE_L3_IPV4)) { + + /* Handle IPv4 headers.*/ + ipv4_hdr = rte_pktmbuf_mtod_offset(pkt, struct ipv4_hdr *, + sizeof(struct ether_hdr)); + + next_hop = em_get_ipv4_dst_port(ipv4_hdr, portid, + qconf->ipv4_lookup_struct); + + if (next_hop >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << next_hop) == 0) + next_hop = portid; + + return next_hop; + + } else if (tcp_or_udp && (l3_ptypes == RTE_PTYPE_L3_IPV6)) { + + /* Handle IPv6 headers.*/ + ipv6_hdr = rte_pktmbuf_mtod_offset(pkt, struct ipv6_hdr *, + sizeof(struct ether_hdr)); + + next_hop = em_get_ipv6_dst_port(ipv6_hdr, portid, + qconf->ipv6_lookup_struct); + + if (next_hop >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << next_hop) == 0) + next_hop = portid; + + return next_hop; + + } + + return portid; +} + +/* + * Buffer optimized handling of packets, invoked + * from main_loop. + */ +static inline void +l3fwd_em_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, + uint8_t portid, struct lcore_conf *qconf) +{ + int32_t j; + uint16_t dst_port[MAX_PKT_BURST]; + + /* + * Send nb_rx - nb_rx%8 packets + * in groups of 8. + */ + int32_t n = RTE_ALIGN_FLOOR(nb_rx, 8); + + for (j = 0; j < n; j += 8) { + + uint32_t pkt_type = + pkts_burst[j]->packet_type & + pkts_burst[j+1]->packet_type & + pkts_burst[j+2]->packet_type & + pkts_burst[j+3]->packet_type & + pkts_burst[j+4]->packet_type & + pkts_burst[j+5]->packet_type & + pkts_burst[j+6]->packet_type & + pkts_burst[j+7]->packet_type; + + uint32_t l3_type = pkt_type & RTE_PTYPE_L3_MASK; + uint32_t tcp_or_udp = pkt_type & + (RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP); + + if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV4)) { + + em_get_dst_port_ipv4x8(qconf, &pkts_burst[j], portid, + &dst_port[j]); + + } else if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV6)) { + + em_get_dst_port_ipv6x8(qconf, &pkts_burst[j], portid, + &dst_port[j]); + + } else { + dst_port[j] = em_get_dst_port(qconf, pkts_burst[j], + portid); + dst_port[j+1] = em_get_dst_port(qconf, pkts_burst[j+1], + portid); + dst_port[j+2] = em_get_dst_port(qconf, pkts_burst[j+2], + portid); + dst_port[j+3] = em_get_dst_port(qconf, pkts_burst[j+3], + portid); + dst_port[j+4] = em_get_dst_port(qconf, pkts_burst[j+4], + portid); + dst_port[j+5] = em_get_dst_port(qconf, pkts_burst[j+5], + portid); + dst_port[j+6] = em_get_dst_port(qconf, pkts_burst[j+6], + portid); + dst_port[j+7] = em_get_dst_port(qconf, pkts_burst[j+7], + portid); + } + } + + for (; j < nb_rx; j++) + dst_port[j] = em_get_dst_port(qconf, pkts_burst[j], portid); + + send_packets_multi(qconf, pkts_burst, dst_port, nb_rx); + +} +#endif /* __L3FWD_EM_HLM_H__ */ diff --git a/examples/l3fwd/l3fwd_em_hlm_sse.h b/examples/l3fwd/l3fwd_em_hlm_sse.h index d272f11..0dd44df 100644 --- a/examples/l3fwd/l3fwd_em_hlm_sse.h +++ b/examples/l3fwd/l3fwd_em_hlm_sse.h @@ -37,101 +37,15 @@ #include "l3fwd_sse.h" static __rte_always_inline void -em_get_dst_port_ipv4x8(struct lcore_conf *qconf, struct rte_mbuf *m[8], - uint8_t portid, uint16_t dst_port[8]) +get_ipv4_5tuple(struct rte_mbuf *m0, __m128i mask0, + union ipv4_5tuple_host *key) { - int32_t ret[8]; - union ipv4_5tuple_host key[8]; - __m128i data[8]; - - data[0] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[0], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[1] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[1], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[2] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[2], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[3] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[3], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[4] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[4], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[5] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[5], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[6] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[6], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[7] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[7], __m128i *, + __m128i tmpdata0 = _mm_loadu_si128( + rte_pktmbuf_mtod_offset(m0, __m128i *, sizeof(struct ether_hdr) + offsetof(struct ipv4_hdr, time_to_live))); - key[0].xmm = _mm_and_si128(data[0], mask0.x); - key[1].xmm = _mm_and_si128(data[1], mask0.x); - key[2].xmm = _mm_and_si128(data[2], mask0.x); - key[3].xmm = _mm_and_si128(data[3], mask0.x); - key[4].xmm = _mm_and_si128(data[4], mask0.x); - key[5].xmm = _mm_and_si128(data[5], mask0.x); - key[6].xmm = _mm_and_si128(data[6], mask0.x); - key[7].xmm = _mm_and_si128(data[7], mask0.x); - - const void *key_array[8] = {&key[0], &key[1], &key[2], &key[3], - &key[4], &key[5], &key[6], &key[7]}; - - rte_hash_lookup_bulk(qconf->ipv4_lookup_struct, &key_array[0], 8, ret); - - dst_port[0] = (uint8_t) ((ret[0] < 0) ? - portid : ipv4_l3fwd_out_if[ret[0]]); - dst_port[1] = (uint8_t) ((ret[1] < 0) ? - portid : ipv4_l3fwd_out_if[ret[1]]); - dst_port[2] = (uint8_t) ((ret[2] < 0) ? - portid : ipv4_l3fwd_out_if[ret[2]]); - dst_port[3] = (uint8_t) ((ret[3] < 0) ? - portid : ipv4_l3fwd_out_if[ret[3]]); - dst_port[4] = (uint8_t) ((ret[4] < 0) ? - portid : ipv4_l3fwd_out_if[ret[4]]); - dst_port[5] = (uint8_t) ((ret[5] < 0) ? - portid : ipv4_l3fwd_out_if[ret[5]]); - dst_port[6] = (uint8_t) ((ret[6] < 0) ? - portid : ipv4_l3fwd_out_if[ret[6]]); - dst_port[7] = (uint8_t) ((ret[7] < 0) ? - portid : ipv4_l3fwd_out_if[ret[7]]); - - if (dst_port[0] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[0]) == 0) - dst_port[0] = portid; - - if (dst_port[1] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[1]) == 0) - dst_port[1] = portid; - - if (dst_port[2] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[2]) == 0) - dst_port[2] = portid; - - if (dst_port[3] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[3]) == 0) - dst_port[3] = portid; - - if (dst_port[4] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[4]) == 0) - dst_port[4] = portid; - - if (dst_port[5] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[5]) == 0) - dst_port[5] = portid; - - if (dst_port[6] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[6]) == 0) - dst_port[6] = portid; - - if (dst_port[7] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[7]) == 0) - dst_port[7] = portid; - + key->xmm = _mm_and_si128(tmpdata0, mask0); } static inline void @@ -159,184 +73,4 @@ key->xmm[1] = tmpdata1; key->xmm[2] = _mm_and_si128(tmpdata2, mask1); } - -static __rte_always_inline void -em_get_dst_port_ipv6x8(struct lcore_conf *qconf, struct rte_mbuf *m[8], - uint8_t portid, uint16_t dst_port[8]) -{ - int32_t ret[8]; - union ipv6_5tuple_host key[8]; - - get_ipv6_5tuple(m[0], mask1.x, mask2.x, &key[0]); - get_ipv6_5tuple(m[1], mask1.x, mask2.x, &key[1]); - get_ipv6_5tuple(m[2], mask1.x, mask2.x, &key[2]); - get_ipv6_5tuple(m[3], mask1.x, mask2.x, &key[3]); - get_ipv6_5tuple(m[4], mask1.x, mask2.x, &key[4]); - get_ipv6_5tuple(m[5], mask1.x, mask2.x, &key[5]); - get_ipv6_5tuple(m[6], mask1.x, mask2.x, &key[6]); - get_ipv6_5tuple(m[7], mask1.x, mask2.x, &key[7]); - - const void *key_array[8] = {&key[0], &key[1], &key[2], &key[3], - &key[4], &key[5], &key[6], &key[7]}; - - rte_hash_lookup_bulk(qconf->ipv6_lookup_struct, &key_array[0], 8, ret); - - dst_port[0] = (uint8_t) ((ret[0] < 0) ? - portid : ipv6_l3fwd_out_if[ret[0]]); - dst_port[1] = (uint8_t) ((ret[1] < 0) ? - portid : ipv6_l3fwd_out_if[ret[1]]); - dst_port[2] = (uint8_t) ((ret[2] < 0) ? - portid : ipv6_l3fwd_out_if[ret[2]]); - dst_port[3] = (uint8_t) ((ret[3] < 0) ? - portid : ipv6_l3fwd_out_if[ret[3]]); - dst_port[4] = (uint8_t) ((ret[4] < 0) ? - portid : ipv6_l3fwd_out_if[ret[4]]); - dst_port[5] = (uint8_t) ((ret[5] < 0) ? - portid : ipv6_l3fwd_out_if[ret[5]]); - dst_port[6] = (uint8_t) ((ret[6] < 0) ? - portid : ipv6_l3fwd_out_if[ret[6]]); - dst_port[7] = (uint8_t) ((ret[7] < 0) ? - portid : ipv6_l3fwd_out_if[ret[7]]); - - if (dst_port[0] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[0]) == 0) - dst_port[0] = portid; - - if (dst_port[1] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[1]) == 0) - dst_port[1] = portid; - - if (dst_port[2] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[2]) == 0) - dst_port[2] = portid; - - if (dst_port[3] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[3]) == 0) - dst_port[3] = portid; - - if (dst_port[4] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[4]) == 0) - dst_port[4] = portid; - - if (dst_port[5] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[5]) == 0) - dst_port[5] = portid; - - if (dst_port[6] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[6]) == 0) - dst_port[6] = portid; - - if (dst_port[7] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[7]) == 0) - dst_port[7] = portid; - -} - -static __rte_always_inline uint16_t -em_get_dst_port(const struct lcore_conf *qconf, struct rte_mbuf *pkt, - uint8_t portid) -{ - uint8_t next_hop; - struct ipv4_hdr *ipv4_hdr; - struct ipv6_hdr *ipv6_hdr; - uint32_t tcp_or_udp; - uint32_t l3_ptypes; - - tcp_or_udp = pkt->packet_type & (RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP); - l3_ptypes = pkt->packet_type & RTE_PTYPE_L3_MASK; - - if (tcp_or_udp && (l3_ptypes == RTE_PTYPE_L3_IPV4)) { - - /* Handle IPv4 headers.*/ - ipv4_hdr = rte_pktmbuf_mtod_offset(pkt, struct ipv4_hdr *, - sizeof(struct ether_hdr)); - - next_hop = em_get_ipv4_dst_port(ipv4_hdr, portid, - qconf->ipv4_lookup_struct); - - if (next_hop >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << next_hop) == 0) - next_hop = portid; - - return next_hop; - - } else if (tcp_or_udp && (l3_ptypes == RTE_PTYPE_L3_IPV6)) { - - /* Handle IPv6 headers.*/ - ipv6_hdr = rte_pktmbuf_mtod_offset(pkt, struct ipv6_hdr *, - sizeof(struct ether_hdr)); - - next_hop = em_get_ipv6_dst_port(ipv6_hdr, portid, - qconf->ipv6_lookup_struct); - - if (next_hop >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << next_hop) == 0) - next_hop = portid; - - return next_hop; - - } - - return portid; -} - -/* - * Buffer optimized handling of packets, invoked - * from main_loop. - */ -static inline void -l3fwd_em_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, - uint8_t portid, struct lcore_conf *qconf) -{ - int32_t j; - uint16_t dst_port[MAX_PKT_BURST]; - - /* - * Send nb_rx - nb_rx%8 packets - * in groups of 8. - */ - int32_t n = RTE_ALIGN_FLOOR(nb_rx, 8); - - for (j = 0; j < n; j += 8) { - - uint32_t pkt_type = - pkts_burst[j]->packet_type & - pkts_burst[j+1]->packet_type & - pkts_burst[j+2]->packet_type & - pkts_burst[j+3]->packet_type & - pkts_burst[j+4]->packet_type & - pkts_burst[j+5]->packet_type & - pkts_burst[j+6]->packet_type & - pkts_burst[j+7]->packet_type; - - uint32_t l3_type = pkt_type & RTE_PTYPE_L3_MASK; - uint32_t tcp_or_udp = pkt_type & - (RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP); - - if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV4)) { - - em_get_dst_port_ipv4x8(qconf, &pkts_burst[j], portid, &dst_port[j]); - - } else if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV6)) { - - em_get_dst_port_ipv6x8(qconf, &pkts_burst[j], portid, &dst_port[j]); - - } else { - dst_port[j] = em_get_dst_port(qconf, pkts_burst[j], portid); - dst_port[j+1] = em_get_dst_port(qconf, pkts_burst[j+1], portid); - dst_port[j+2] = em_get_dst_port(qconf, pkts_burst[j+2], portid); - dst_port[j+3] = em_get_dst_port(qconf, pkts_burst[j+3], portid); - dst_port[j+4] = em_get_dst_port(qconf, pkts_burst[j+4], portid); - dst_port[j+5] = em_get_dst_port(qconf, pkts_burst[j+5], portid); - dst_port[j+6] = em_get_dst_port(qconf, pkts_burst[j+6], portid); - dst_port[j+7] = em_get_dst_port(qconf, pkts_burst[j+7], portid); - } - } - - for (; j < nb_rx; j++) - dst_port[j] = em_get_dst_port(qconf, pkts_burst[j], portid); - - send_packets_multi(qconf, pkts_burst, dst_port, nb_rx); - -} #endif /* __L3FWD_EM_SSE_HLM_H__ */