From patchwork Mon May 15 03:34:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianbo Liu X-Patchwork-Id: 99761 Delivered-To: patch@linaro.org Received: by 10.182.142.97 with SMTP id rv1csp1240353obb; Sun, 14 May 2017 20:35:27 -0700 (PDT) X-Received: by 10.223.171.26 with SMTP id q26mr2302477wrc.15.1494819327072; Sun, 14 May 2017 20:35:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1494819327; cv=none; d=google.com; s=arc-20160816; b=GN79UpoWaBu0fYP3Niq/WfNcZdxXrcTdGjrzcCD9zzARYO1ehRMLtkIPHLqMAWBlmD 1BSJPatPFMhkyLkqkDJq4BPSlhiiyj5EL6zHRHDyr/w7Ru8OesHhzURBjre/f8mivi91 +cRKWTukO84/FIQ7fr5pK7x8Papmnl5keTp/G+D0+UP5xtJ8tVL1MrQEpyMwucs86dQq kLIZXLynGCsCsaAHQz7D/2iYaNXVlqkszdrM51lCsyMOvfaxh0IKbh5pAV8D/odQkos1 dr40zwE/OTAou0EKDGL6irAppnsJ4QLzCfFBjxQYl4VpP+RyvZZhlNcySTGt2u6bxk8g rWvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:cc:to:from:arc-authentication-results; bh=eB9MeKdgDrpfucOaCIuV1zUfzNflwlkaMA5xmEz5k9w=; b=Hmqq8//wHtr11Aujh7AUbsMsDGHvdaGlVVVdMfluNF+kgYB06v/6X0mfTX5UYMvKp5 K6fB2oLE7eLNFTUSs2Ea+yO5wS1UuuUSBUpE0NzrnoaGZqIszx12unfzqWNktNxkRzIQ E3aQfe/jD4intcxhN+gx8aC9YSGO/IZ6VVQmQWIO2VT5paK47A6DI4Yxf4YjyYBkrpE9 r3h8pEhnI0oucM2ZhXkl+DXW5zrgL6r0Zu13Br/A2t4AFhoWyPcN1HSVv+OIju+QRMD2 hV8p1fyMpbxzp144iSY4K7j9iU8r8HvoRws7aR7dimplC54ncmQSnTpu4zCgS6wAjIfP KANg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from dpdk.org (dpdk.org. [92.243.14.124]) by mx.google.com with ESMTP id 93si10527670wrq.153.2017.05.14.20.35.26; Sun, 14 May 2017 20:35:27 -0700 (PDT) Received-SPF: pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) client-ip=92.243.14.124; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 2C53258FA; Mon, 15 May 2017 05:35:19 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.101.70]) by dpdk.org (Postfix) with ESMTP id CB7B511D4 for ; Mon, 15 May 2017 05:35:14 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3FE5D1435; Sun, 14 May 2017 20:35:14 -0700 (PDT) Received: from localhost.localdomain.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 68E623F23B; Sun, 14 May 2017 20:35:13 -0700 (PDT) From: Jianbo Liu To: dev@dpdk.org, tomasz.kantecki@intel.com, jerin.jacob@caviumnetworks.com, ashwin.sekhar@caviumnetworks.com Cc: Jianbo Liu Date: Mon, 15 May 2017 11:34:49 +0800 Message-Id: <1494819296-23049-2-git-send-email-jianbo.liu@linaro.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1494819296-23049-1-git-send-email-jianbo.liu@linaro.org> References: <1493709255-8887-1-git-send-email-jianbo.liu@linaro.org> <1494819296-23049-1-git-send-email-jianbo.liu@linaro.org> Subject: [dpdk-dev] [PATCH v4 1/8] examples/l3fwd: extract arch independent code from multi hash lookup X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Extract common code from l3fwd_em_hlm_sse.h, and add to the new file l3fwd_em_hlm.h. Signed-off-by: Jianbo Liu --- examples/l3fwd/l3fwd_em.c | 2 +- examples/l3fwd/l3fwd_em_hlm.h | 302 ++++++++++++++++++++++++++++++++++++++ examples/l3fwd/l3fwd_em_hlm_sse.h | 280 +---------------------------------- 3 files changed, 309 insertions(+), 275 deletions(-) create mode 100644 examples/l3fwd/l3fwd_em_hlm.h -- 1.8.3.1 diff --git a/examples/l3fwd/l3fwd_em.c b/examples/l3fwd/l3fwd_em.c index 9cc4460..939a16d 100644 --- a/examples/l3fwd/l3fwd_em.c +++ b/examples/l3fwd/l3fwd_em.c @@ -332,7 +332,7 @@ struct ipv6_l3fwd_em_route { #if defined(NO_HASH_MULTI_LOOKUP) #include "l3fwd_em_sse.h" #else -#include "l3fwd_em_hlm_sse.h" +#include "l3fwd_em_hlm.h" #endif #else #include "l3fwd_em.h" diff --git a/examples/l3fwd/l3fwd_em_hlm.h b/examples/l3fwd/l3fwd_em_hlm.h new file mode 100644 index 0000000..636dea4 --- /dev/null +++ b/examples/l3fwd/l3fwd_em_hlm.h @@ -0,0 +1,302 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016 Intel Corporation. All rights reserved. + * Copyright(c) 2017, Linaro Limited + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef __L3FWD_EM_HLM_H__ +#define __L3FWD_EM_HLM_H__ + +#include "l3fwd_sse.h" +#include "l3fwd_em_hlm_sse.h" + +static inline __attribute__((always_inline)) void +em_get_dst_port_ipv4x8(struct lcore_conf *qconf, struct rte_mbuf *m[8], + uint8_t portid, uint16_t dst_port[8]) +{ + int32_t ret[8]; + union ipv4_5tuple_host key[8]; + + get_ipv4_5tuple(m[0], mask0.x, &key[0]); + get_ipv4_5tuple(m[1], mask0.x, &key[1]); + get_ipv4_5tuple(m[2], mask0.x, &key[2]); + get_ipv4_5tuple(m[3], mask0.x, &key[3]); + get_ipv4_5tuple(m[4], mask0.x, &key[4]); + get_ipv4_5tuple(m[5], mask0.x, &key[5]); + get_ipv4_5tuple(m[6], mask0.x, &key[6]); + get_ipv4_5tuple(m[7], mask0.x, &key[7]); + + const void *key_array[8] = {&key[0], &key[1], &key[2], &key[3], + &key[4], &key[5], &key[6], &key[7]}; + + rte_hash_lookup_bulk(qconf->ipv4_lookup_struct, &key_array[0], 8, ret); + + dst_port[0] = (uint8_t) ((ret[0] < 0) ? + portid : ipv4_l3fwd_out_if[ret[0]]); + dst_port[1] = (uint8_t) ((ret[1] < 0) ? + portid : ipv4_l3fwd_out_if[ret[1]]); + dst_port[2] = (uint8_t) ((ret[2] < 0) ? + portid : ipv4_l3fwd_out_if[ret[2]]); + dst_port[3] = (uint8_t) ((ret[3] < 0) ? + portid : ipv4_l3fwd_out_if[ret[3]]); + dst_port[4] = (uint8_t) ((ret[4] < 0) ? + portid : ipv4_l3fwd_out_if[ret[4]]); + dst_port[5] = (uint8_t) ((ret[5] < 0) ? + portid : ipv4_l3fwd_out_if[ret[5]]); + dst_port[6] = (uint8_t) ((ret[6] < 0) ? + portid : ipv4_l3fwd_out_if[ret[6]]); + dst_port[7] = (uint8_t) ((ret[7] < 0) ? + portid : ipv4_l3fwd_out_if[ret[7]]); + + if (dst_port[0] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[0]) == 0) + dst_port[0] = portid; + + if (dst_port[1] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[1]) == 0) + dst_port[1] = portid; + + if (dst_port[2] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[2]) == 0) + dst_port[2] = portid; + + if (dst_port[3] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[3]) == 0) + dst_port[3] = portid; + + if (dst_port[4] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[4]) == 0) + dst_port[4] = portid; + + if (dst_port[5] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[5]) == 0) + dst_port[5] = portid; + + if (dst_port[6] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[6]) == 0) + dst_port[6] = portid; + + if (dst_port[7] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[7]) == 0) + dst_port[7] = portid; + +} + +static inline __attribute__((always_inline)) void +em_get_dst_port_ipv6x8(struct lcore_conf *qconf, struct rte_mbuf *m[8], + uint8_t portid, uint16_t dst_port[8]) +{ + int32_t ret[8]; + union ipv6_5tuple_host key[8]; + + get_ipv6_5tuple(m[0], mask1.x, mask2.x, &key[0]); + get_ipv6_5tuple(m[1], mask1.x, mask2.x, &key[1]); + get_ipv6_5tuple(m[2], mask1.x, mask2.x, &key[2]); + get_ipv6_5tuple(m[3], mask1.x, mask2.x, &key[3]); + get_ipv6_5tuple(m[4], mask1.x, mask2.x, &key[4]); + get_ipv6_5tuple(m[5], mask1.x, mask2.x, &key[5]); + get_ipv6_5tuple(m[6], mask1.x, mask2.x, &key[6]); + get_ipv6_5tuple(m[7], mask1.x, mask2.x, &key[7]); + + const void *key_array[8] = {&key[0], &key[1], &key[2], &key[3], + &key[4], &key[5], &key[6], &key[7]}; + + rte_hash_lookup_bulk(qconf->ipv6_lookup_struct, &key_array[0], 8, ret); + + dst_port[0] = (uint8_t) ((ret[0] < 0) ? + portid : ipv6_l3fwd_out_if[ret[0]]); + dst_port[1] = (uint8_t) ((ret[1] < 0) ? + portid : ipv6_l3fwd_out_if[ret[1]]); + dst_port[2] = (uint8_t) ((ret[2] < 0) ? + portid : ipv6_l3fwd_out_if[ret[2]]); + dst_port[3] = (uint8_t) ((ret[3] < 0) ? + portid : ipv6_l3fwd_out_if[ret[3]]); + dst_port[4] = (uint8_t) ((ret[4] < 0) ? + portid : ipv6_l3fwd_out_if[ret[4]]); + dst_port[5] = (uint8_t) ((ret[5] < 0) ? + portid : ipv6_l3fwd_out_if[ret[5]]); + dst_port[6] = (uint8_t) ((ret[6] < 0) ? + portid : ipv6_l3fwd_out_if[ret[6]]); + dst_port[7] = (uint8_t) ((ret[7] < 0) ? + portid : ipv6_l3fwd_out_if[ret[7]]); + + if (dst_port[0] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[0]) == 0) + dst_port[0] = portid; + + if (dst_port[1] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[1]) == 0) + dst_port[1] = portid; + + if (dst_port[2] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[2]) == 0) + dst_port[2] = portid; + + if (dst_port[3] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[3]) == 0) + dst_port[3] = portid; + + if (dst_port[4] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[4]) == 0) + dst_port[4] = portid; + + if (dst_port[5] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[5]) == 0) + dst_port[5] = portid; + + if (dst_port[6] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[6]) == 0) + dst_port[6] = portid; + + if (dst_port[7] >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << dst_port[7]) == 0) + dst_port[7] = portid; + +} + +static inline __attribute__((always_inline)) uint16_t +em_get_dst_port(const struct lcore_conf *qconf, struct rte_mbuf *pkt, + uint8_t portid) +{ + uint8_t next_hop; + struct ipv4_hdr *ipv4_hdr; + struct ipv6_hdr *ipv6_hdr; + uint32_t tcp_or_udp; + uint32_t l3_ptypes; + + tcp_or_udp = pkt->packet_type & (RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP); + l3_ptypes = pkt->packet_type & RTE_PTYPE_L3_MASK; + + if (tcp_or_udp && (l3_ptypes == RTE_PTYPE_L3_IPV4)) { + + /* Handle IPv4 headers.*/ + ipv4_hdr = rte_pktmbuf_mtod_offset(pkt, struct ipv4_hdr *, + sizeof(struct ether_hdr)); + + next_hop = em_get_ipv4_dst_port(ipv4_hdr, portid, + qconf->ipv4_lookup_struct); + + if (next_hop >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << next_hop) == 0) + next_hop = portid; + + return next_hop; + + } else if (tcp_or_udp && (l3_ptypes == RTE_PTYPE_L3_IPV6)) { + + /* Handle IPv6 headers.*/ + ipv6_hdr = rte_pktmbuf_mtod_offset(pkt, struct ipv6_hdr *, + sizeof(struct ether_hdr)); + + next_hop = em_get_ipv6_dst_port(ipv6_hdr, portid, + qconf->ipv6_lookup_struct); + + if (next_hop >= RTE_MAX_ETHPORTS || + (enabled_port_mask & 1 << next_hop) == 0) + next_hop = portid; + + return next_hop; + + } + + return portid; +} + +/* + * Buffer optimized handling of packets, invoked + * from main_loop. + */ +static inline void +l3fwd_em_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, + uint8_t portid, struct lcore_conf *qconf) +{ + int32_t j; + uint16_t dst_port[MAX_PKT_BURST]; + + /* + * Send nb_rx - nb_rx%8 packets + * in groups of 8. + */ + int32_t n = RTE_ALIGN_FLOOR(nb_rx, 8); + + for (j = 0; j < n; j += 8) { + + uint32_t pkt_type = + pkts_burst[j]->packet_type & + pkts_burst[j+1]->packet_type & + pkts_burst[j+2]->packet_type & + pkts_burst[j+3]->packet_type & + pkts_burst[j+4]->packet_type & + pkts_burst[j+5]->packet_type & + pkts_burst[j+6]->packet_type & + pkts_burst[j+7]->packet_type; + + uint32_t l3_type = pkt_type & RTE_PTYPE_L3_MASK; + uint32_t tcp_or_udp = pkt_type & + (RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP); + + if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV4)) { + + em_get_dst_port_ipv4x8(qconf, &pkts_burst[j], portid, + &dst_port[j]); + + } else if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV6)) { + + em_get_dst_port_ipv6x8(qconf, &pkts_burst[j], portid, + &dst_port[j]); + + } else { + dst_port[j] = em_get_dst_port(qconf, pkts_burst[j], + portid); + dst_port[j+1] = em_get_dst_port(qconf, pkts_burst[j+1], + portid); + dst_port[j+2] = em_get_dst_port(qconf, pkts_burst[j+2], + portid); + dst_port[j+3] = em_get_dst_port(qconf, pkts_burst[j+3], + portid); + dst_port[j+4] = em_get_dst_port(qconf, pkts_burst[j+4], + portid); + dst_port[j+5] = em_get_dst_port(qconf, pkts_burst[j+5], + portid); + dst_port[j+6] = em_get_dst_port(qconf, pkts_burst[j+6], + portid); + dst_port[j+7] = em_get_dst_port(qconf, pkts_burst[j+7], + portid); + } + } + + for (; j < nb_rx; j++) + dst_port[j] = em_get_dst_port(qconf, pkts_burst[j], portid); + + send_packets_multi(qconf, pkts_burst, dst_port, nb_rx); + +} +#endif /* __L3FWD_EM_HLM_H__ */ diff --git a/examples/l3fwd/l3fwd_em_hlm_sse.h b/examples/l3fwd/l3fwd_em_hlm_sse.h index 7714a20..cb1304f 100644 --- a/examples/l3fwd/l3fwd_em_hlm_sse.h +++ b/examples/l3fwd/l3fwd_em_hlm_sse.h @@ -34,104 +34,16 @@ #ifndef __L3FWD_EM_HLM_SSE_H__ #define __L3FWD_EM_HLM_SSE_H__ -#include "l3fwd_sse.h" - -static inline __attribute__((always_inline)) void -em_get_dst_port_ipv4x8(struct lcore_conf *qconf, struct rte_mbuf *m[8], - uint8_t portid, uint16_t dst_port[8]) +static inline void +get_ipv4_5tuple(struct rte_mbuf *m0, __m128i mask0, + union ipv4_5tuple_host *key) { - int32_t ret[8]; - union ipv4_5tuple_host key[8]; - __m128i data[8]; - - data[0] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[0], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[1] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[1], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[2] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[2], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[3] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[3], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[4] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[4], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[5] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[5], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[6] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[6], __m128i *, - sizeof(struct ether_hdr) + - offsetof(struct ipv4_hdr, time_to_live))); - data[7] = _mm_loadu_si128(rte_pktmbuf_mtod_offset(m[7], __m128i *, + __m128i tmpdata0 = _mm_loadu_si128( + rte_pktmbuf_mtod_offset(m0, __m128i *, sizeof(struct ether_hdr) + offsetof(struct ipv4_hdr, time_to_live))); - key[0].xmm = _mm_and_si128(data[0], mask0.x); - key[1].xmm = _mm_and_si128(data[1], mask0.x); - key[2].xmm = _mm_and_si128(data[2], mask0.x); - key[3].xmm = _mm_and_si128(data[3], mask0.x); - key[4].xmm = _mm_and_si128(data[4], mask0.x); - key[5].xmm = _mm_and_si128(data[5], mask0.x); - key[6].xmm = _mm_and_si128(data[6], mask0.x); - key[7].xmm = _mm_and_si128(data[7], mask0.x); - - const void *key_array[8] = {&key[0], &key[1], &key[2], &key[3], - &key[4], &key[5], &key[6], &key[7]}; - - rte_hash_lookup_bulk(qconf->ipv4_lookup_struct, &key_array[0], 8, ret); - - dst_port[0] = (uint8_t) ((ret[0] < 0) ? - portid : ipv4_l3fwd_out_if[ret[0]]); - dst_port[1] = (uint8_t) ((ret[1] < 0) ? - portid : ipv4_l3fwd_out_if[ret[1]]); - dst_port[2] = (uint8_t) ((ret[2] < 0) ? - portid : ipv4_l3fwd_out_if[ret[2]]); - dst_port[3] = (uint8_t) ((ret[3] < 0) ? - portid : ipv4_l3fwd_out_if[ret[3]]); - dst_port[4] = (uint8_t) ((ret[4] < 0) ? - portid : ipv4_l3fwd_out_if[ret[4]]); - dst_port[5] = (uint8_t) ((ret[5] < 0) ? - portid : ipv4_l3fwd_out_if[ret[5]]); - dst_port[6] = (uint8_t) ((ret[6] < 0) ? - portid : ipv4_l3fwd_out_if[ret[6]]); - dst_port[7] = (uint8_t) ((ret[7] < 0) ? - portid : ipv4_l3fwd_out_if[ret[7]]); - - if (dst_port[0] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[0]) == 0) - dst_port[0] = portid; - - if (dst_port[1] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[1]) == 0) - dst_port[1] = portid; - - if (dst_port[2] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[2]) == 0) - dst_port[2] = portid; - - if (dst_port[3] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[3]) == 0) - dst_port[3] = portid; - - if (dst_port[4] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[4]) == 0) - dst_port[4] = portid; - - if (dst_port[5] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[5]) == 0) - dst_port[5] = portid; - - if (dst_port[6] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[6]) == 0) - dst_port[6] = portid; - - if (dst_port[7] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[7]) == 0) - dst_port[7] = portid; - + key->xmm = _mm_and_si128(tmpdata0, mask0); } static inline void @@ -159,184 +71,4 @@ static inline __attribute__((always_inline)) void key->xmm[1] = tmpdata1; key->xmm[2] = _mm_and_si128(tmpdata2, mask1); } - -static inline __attribute__((always_inline)) void -em_get_dst_port_ipv6x8(struct lcore_conf *qconf, struct rte_mbuf *m[8], - uint8_t portid, uint16_t dst_port[8]) -{ - int32_t ret[8]; - union ipv6_5tuple_host key[8]; - - get_ipv6_5tuple(m[0], mask1.x, mask2.x, &key[0]); - get_ipv6_5tuple(m[1], mask1.x, mask2.x, &key[1]); - get_ipv6_5tuple(m[2], mask1.x, mask2.x, &key[2]); - get_ipv6_5tuple(m[3], mask1.x, mask2.x, &key[3]); - get_ipv6_5tuple(m[4], mask1.x, mask2.x, &key[4]); - get_ipv6_5tuple(m[5], mask1.x, mask2.x, &key[5]); - get_ipv6_5tuple(m[6], mask1.x, mask2.x, &key[6]); - get_ipv6_5tuple(m[7], mask1.x, mask2.x, &key[7]); - - const void *key_array[8] = {&key[0], &key[1], &key[2], &key[3], - &key[4], &key[5], &key[6], &key[7]}; - - rte_hash_lookup_bulk(qconf->ipv6_lookup_struct, &key_array[0], 8, ret); - - dst_port[0] = (uint8_t) ((ret[0] < 0) ? - portid : ipv6_l3fwd_out_if[ret[0]]); - dst_port[1] = (uint8_t) ((ret[1] < 0) ? - portid : ipv6_l3fwd_out_if[ret[1]]); - dst_port[2] = (uint8_t) ((ret[2] < 0) ? - portid : ipv6_l3fwd_out_if[ret[2]]); - dst_port[3] = (uint8_t) ((ret[3] < 0) ? - portid : ipv6_l3fwd_out_if[ret[3]]); - dst_port[4] = (uint8_t) ((ret[4] < 0) ? - portid : ipv6_l3fwd_out_if[ret[4]]); - dst_port[5] = (uint8_t) ((ret[5] < 0) ? - portid : ipv6_l3fwd_out_if[ret[5]]); - dst_port[6] = (uint8_t) ((ret[6] < 0) ? - portid : ipv6_l3fwd_out_if[ret[6]]); - dst_port[7] = (uint8_t) ((ret[7] < 0) ? - portid : ipv6_l3fwd_out_if[ret[7]]); - - if (dst_port[0] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[0]) == 0) - dst_port[0] = portid; - - if (dst_port[1] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[1]) == 0) - dst_port[1] = portid; - - if (dst_port[2] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[2]) == 0) - dst_port[2] = portid; - - if (dst_port[3] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[3]) == 0) - dst_port[3] = portid; - - if (dst_port[4] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[4]) == 0) - dst_port[4] = portid; - - if (dst_port[5] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[5]) == 0) - dst_port[5] = portid; - - if (dst_port[6] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[6]) == 0) - dst_port[6] = portid; - - if (dst_port[7] >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << dst_port[7]) == 0) - dst_port[7] = portid; - -} - -static inline __attribute__((always_inline)) uint16_t -em_get_dst_port(const struct lcore_conf *qconf, struct rte_mbuf *pkt, - uint8_t portid) -{ - uint8_t next_hop; - struct ipv4_hdr *ipv4_hdr; - struct ipv6_hdr *ipv6_hdr; - uint32_t tcp_or_udp; - uint32_t l3_ptypes; - - tcp_or_udp = pkt->packet_type & (RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP); - l3_ptypes = pkt->packet_type & RTE_PTYPE_L3_MASK; - - if (tcp_or_udp && (l3_ptypes == RTE_PTYPE_L3_IPV4)) { - - /* Handle IPv4 headers.*/ - ipv4_hdr = rte_pktmbuf_mtod_offset(pkt, struct ipv4_hdr *, - sizeof(struct ether_hdr)); - - next_hop = em_get_ipv4_dst_port(ipv4_hdr, portid, - qconf->ipv4_lookup_struct); - - if (next_hop >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << next_hop) == 0) - next_hop = portid; - - return next_hop; - - } else if (tcp_or_udp && (l3_ptypes == RTE_PTYPE_L3_IPV6)) { - - /* Handle IPv6 headers.*/ - ipv6_hdr = rte_pktmbuf_mtod_offset(pkt, struct ipv6_hdr *, - sizeof(struct ether_hdr)); - - next_hop = em_get_ipv6_dst_port(ipv6_hdr, portid, - qconf->ipv6_lookup_struct); - - if (next_hop >= RTE_MAX_ETHPORTS || - (enabled_port_mask & 1 << next_hop) == 0) - next_hop = portid; - - return next_hop; - - } - - return portid; -} - -/* - * Buffer optimized handling of packets, invoked - * from main_loop. - */ -static inline void -l3fwd_em_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, - uint8_t portid, struct lcore_conf *qconf) -{ - int32_t j; - uint16_t dst_port[MAX_PKT_BURST]; - - /* - * Send nb_rx - nb_rx%8 packets - * in groups of 8. - */ - int32_t n = RTE_ALIGN_FLOOR(nb_rx, 8); - - for (j = 0; j < n; j += 8) { - - uint32_t pkt_type = - pkts_burst[j]->packet_type & - pkts_burst[j+1]->packet_type & - pkts_burst[j+2]->packet_type & - pkts_burst[j+3]->packet_type & - pkts_burst[j+4]->packet_type & - pkts_burst[j+5]->packet_type & - pkts_burst[j+6]->packet_type & - pkts_burst[j+7]->packet_type; - - uint32_t l3_type = pkt_type & RTE_PTYPE_L3_MASK; - uint32_t tcp_or_udp = pkt_type & - (RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP); - - if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV4)) { - - em_get_dst_port_ipv4x8(qconf, &pkts_burst[j], portid, &dst_port[j]); - - } else if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV6)) { - - em_get_dst_port_ipv6x8(qconf, &pkts_burst[j], portid, &dst_port[j]); - - } else { - dst_port[j] = em_get_dst_port(qconf, pkts_burst[j], portid); - dst_port[j+1] = em_get_dst_port(qconf, pkts_burst[j+1], portid); - dst_port[j+2] = em_get_dst_port(qconf, pkts_burst[j+2], portid); - dst_port[j+3] = em_get_dst_port(qconf, pkts_burst[j+3], portid); - dst_port[j+4] = em_get_dst_port(qconf, pkts_burst[j+4], portid); - dst_port[j+5] = em_get_dst_port(qconf, pkts_burst[j+5], portid); - dst_port[j+6] = em_get_dst_port(qconf, pkts_burst[j+6], portid); - dst_port[j+7] = em_get_dst_port(qconf, pkts_burst[j+7], portid); - } - } - - for (; j < nb_rx; j++) - dst_port[j] = em_get_dst_port(qconf, pkts_burst[j], portid); - - send_packets_multi(qconf, pkts_burst, dst_port, nb_rx); - -} #endif /* __L3FWD_EM_SSE_HLM_H__ */