From patchwork Fri Jul 4 13:04:26 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Uvarov X-Patchwork-Id: 33092 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-pd0-f199.google.com (mail-pd0-f199.google.com [209.85.192.199]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id DF93820C88 for ; Fri, 4 Jul 2014 13:04:54 +0000 (UTC) Received: by mail-pd0-f199.google.com with SMTP id r10sf9321198pdi.2 for ; Fri, 04 Jul 2014 06:04:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id:subject :precedence:list-id:list-unsubscribe:list-archive:list-post :list-help:list-subscribe:mime-version:errors-to:sender :x-original-sender:x-original-authentication-results:mailing-list :content-type:content-transfer-encoding; bh=o2fg03Hx77Ti6IqP2DMV7E3fpKAN0rKTjnuS/uUzVs8=; b=HHIF8cxchVvHEJwWaBOSPePskv85ZMqxLUqw/vIGBz0is658EsHzeep5MenhnKfOhL pJPoz0VAPbxNbKL9HPCMyHrFvWyOhExefHsVuwViFY4sIIzVAvYJwaypXAY0ZuKpzf2P koCXGVr+IhyzgHz43VVlBQAE/T9Aki3YC8qEQOQ4C4p8FRIGC3oCZXKriJD6oDe1C6AW LsD7HpeForFggdzKdFnDYq3G+OXhetqJeEh0WFqrJYeRohTLBqbguN9rG1QDPZ3gvqBO zW4I7Zh5LON8E9ijz7HynPx0a0XjUpDqKazRpRgfzTaUjbNgRpvDlRzMdH4tHqRzPL0u W7zg== X-Gm-Message-State: ALoCoQmB0DXgxV6hSFS6MCohG82HrnYd7Ufqycb3l3bLwwLNzl/0QQOu8AwjeKC4xW8XJhnaVrn7 X-Received: by 10.66.65.202 with SMTP id z10mr5374488pas.45.1404479092942; Fri, 04 Jul 2014 06:04:52 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.89.104 with SMTP id u95ls893199qgd.5.gmail; Fri, 04 Jul 2014 06:04:52 -0700 (PDT) X-Received: by 10.52.121.112 with SMTP id lj16mr8116902vdb.29.1404479092786; Fri, 04 Jul 2014 06:04:52 -0700 (PDT) Received: from mail-ve0-f179.google.com (mail-ve0-f179.google.com [209.85.128.179]) by mx.google.com with ESMTPS id kg19si1378377vdb.63.2014.07.04.06.04.52 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 04 Jul 2014 06:04:52 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.128.179 as permitted sender) client-ip=209.85.128.179; Received: by mail-ve0-f179.google.com with SMTP id sa20so1650609veb.10 for ; Fri, 04 Jul 2014 06:04:52 -0700 (PDT) X-Received: by 10.52.253.131 with SMTP id aa3mr8056853vdd.25.1404479092670; Fri, 04 Jul 2014 06:04:52 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.221.37.5 with SMTP id tc5csp464834vcb; Fri, 4 Jul 2014 06:04:52 -0700 (PDT) X-Received: by 10.229.178.202 with SMTP id bn10mr18129418qcb.6.1404479092143; Fri, 04 Jul 2014 06:04:52 -0700 (PDT) Received: from ip-10-141-164-156.ec2.internal (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTPS id n4si2400071qax.95.2014.07.04.06.04.51 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Fri, 04 Jul 2014 06:04:52 -0700 (PDT) Received-SPF: none (google.com: lng-odp-bounces@lists.linaro.org does not designate permitted sender hosts) client-ip=54.225.227.206; Received: from localhost ([127.0.0.1] helo=ip-10-141-164-156.ec2.internal) by ip-10-141-164-156.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1X338V-0001cN-3t; Fri, 04 Jul 2014 13:02:43 +0000 Received: from mail-lb0-f173.google.com ([209.85.217.173]) by ip-10-141-164-156.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1X338K-0001bk-WE for lng-odp@lists.linaro.org; Fri, 04 Jul 2014 13:02:33 +0000 Received: by mail-lb0-f173.google.com with SMTP id s7so1166079lbd.32 for ; Fri, 04 Jul 2014 06:04:34 -0700 (PDT) X-Received: by 10.112.180.70 with SMTP id dm6mr8112895lbc.32.1404479073603; Fri, 04 Jul 2014 06:04:33 -0700 (PDT) Received: from maxim-lap.localhost.onion ([92.39.133.154]) by mx.google.com with ESMTPSA id tg1sm13191694lbb.11.2014.07.04.06.04.32 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 04 Jul 2014 06:04:32 -0700 (PDT) From: Maxim Uvarov To: lng-odp@lists.linaro.org Date: Fri, 4 Jul 2014 17:04:26 +0400 Message-Id: <1404479068-17690-1-git-send-email-maxim.uvarov@linaro.org> X-Mailer: git-send-email 1.8.5.1.163.gd7aced9 X-Topics: patch Subject: [lng-odp] [APPS/PATCH 1/3] add lock, configure option and fix segfault on recv X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: lng-odp-bounces@lists.linaro.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: maxim.uvarov@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.128.179 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 - fixed typo which generated segfault on recv; - added command line parsing and better incorporation with snort; - odp setups changed to getenv(); - added README file. - tested with 1 and 4 cores with raw socket, mmsg and mmap. Signed-off-by: Maxim Uvarov --- snort/odpsnort/Makefile | 3 +- snort/odpsnort/README | 39 ++++++++ snort/odpsnort/odp_pktio.c | 240 +++++++++++++++------------------------------ 3 files changed, 118 insertions(+), 164 deletions(-) create mode 100644 snort/odpsnort/README diff --git a/snort/odpsnort/Makefile b/snort/odpsnort/Makefile index d348521..81c8e40 100644 --- a/snort/odpsnort/Makefile +++ b/snort/odpsnort/Makefile @@ -23,6 +23,7 @@ INC += "-I$(SNORT)/src/control" INC += "-I$(SNORT)/src/file-process" INC += "-I$(SNORT)/src/file-process/libs" INC += "-I$(SNORT)/src/preprocessors/Stream5" +INC += "-I$(SNORT)/src" INC += "-I$(SNORT)" ODP_DIR = $(PWD)/../../odp.git @@ -38,7 +39,7 @@ ODP_CFLAGS += -I$(DAQ) all: $(CC) -c -O0 -g snort.c ${CFLAGS} ${LDFLAGS} $(INC) -o snort.o - $(CC) -c -O0 -g -pthread $(ODP_CFLAGS) odp_pktio.c -o odp_pktio.o + $(CC) -c -O0 -g -pthread ${INC} $(ODP_CFLAGS) odp_pktio.c -o odp_pktio.o $(CC) snort.o odp_pktio.o ${CFLAGS} ${LDFLAGS} $(INC) ${LIBSNORT} ${DAQ_LIB} ${LIBDNET} -ldl -lz -lpcap -pthread -lpcre -lssl -lcrypto -lm -lrt -o odp-snort clean: rm -rf odp-snort *.o diff --git a/snort/odpsnort/README b/snort/odpsnort/README new file mode 100644 index 0000000..52d4ccc --- /dev/null +++ b/snort/odpsnort/README @@ -0,0 +1,39 @@ +ODP application which use Snort for analyzing packets + + +1. To run execute: +./odp-snort -i eth0,eth1 + +eth0,eth1 any number of interfaces to listen for packets. + +Standard Snort arguments are supported. Run --help for more options. + +2. ODP tuning: + +2a. ODP can work in queue and burst mode. By default queue mode + is selected. To switch to burst mode do: + + export ODP_PTK_BURST + +2b. If you are using linux-generic then you can select one of supported + packets i/o. Export appropriate env: + + export ODP_PKTIO_TYPE_SOCKET=1 // for ODP_PKTIO_TYPE_SOCKET_BASIC + export ODP_PKTIO_TYPE_SOCKET=2 // for ODP_PKTIO_TYPE_SOCKET_MMSG + export ODP_PKTIO_TYPE_SOCKET=3 // for ODP_PKTIO_TYPE_SOCKET_MMAP + +2c. Number of cores for packet processing: + export ODP_CORES=4 + +3. Limitation + + Snort 2.9.x is not thread safe due to: + - global variables; + - calls to thread unsafe functions: like ctime(), strtok(); + - files logs, init process should be different for several threads; + Threads has to be fixed in new SnortSP (which is snort 3.0). + + In current example 'odp_spinlock_t lock' is used to defend calls to snort analyze packet + function. That is leads to limitation to scale on different cpus. Further experiments will + be done with SnortSP to get rid of that limitation. + diff --git a/snort/odpsnort/odp_pktio.c b/snort/odpsnort/odp_pktio.c index 81dc301..cf38e23 100644 --- a/snort/odpsnort/odp_pktio.c +++ b/snort/odpsnort/odp_pktio.c @@ -7,7 +7,7 @@ /** * @file * - * @example odp_example_pktio.c ODP basic packet IO loopback test application + * @example odp_pktio.c ODP packet IO example for Snort application */ #include @@ -23,6 +23,7 @@ /* Snort */ #include +#include "snort.h" #define MAX_WORKERS 32 #define SHM_PKT_POOL_SIZE (512*2048) @@ -75,12 +76,12 @@ typedef struct { static args_t *args; /* helper funcs */ -static int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned len); -static void swap_pkt_addrs(odp_packet_t pkt_tbl[], unsigned len); static void parse_args(int argc, char *argv[], appl_args_t *appl_args); static void print_info(char *progname, appl_args_t *appl_args); static void usage(char *progname); +static odp_spinlock_t lock; + static void analyze_packet_in_snort(odp_packet_t pkt, int thr) { DAQ_PktHdr_t daqhdr; @@ -89,15 +90,13 @@ static void analyze_packet_in_snort(odp_packet_t pkt, int thr) data = odp_packet_l2(pkt); if (!data) { - printf("no l2 offset, packet dropped\n"); return; } gettimeofday(&daqhdr.ts, NULL); daqhdr.caplen = odp_buffer_size(pkt); - printf("%s() odp recieved packet len %d. thread %d\n", __func__, odp_packet_get_len(pkt), thr); daqhdr.pktlen = odp_packet_get_len(pkt); - daqhdr.ingress_index = 0; + daqhdr.ingress_index = thr; daqhdr.egress_index = DAQ_PKTHDR_UNKNOWN; daqhdr.ingress_group = DAQ_PKTHDR_UNKNOWN; daqhdr.egress_group = DAQ_PKTHDR_UNKNOWN; @@ -107,10 +106,27 @@ static void analyze_packet_in_snort(odp_packet_t pkt, int thr) daqhdr.address_space_id = 0; /* Pass packet to Snort */ - verd = PacketCallback( "NULL", daqhdr, data); + odp_spinlock_lock(&lock); + verd = PacketCallback( "NULL", &daqhdr, data); + odp_spinlock_unlock(&lock); return; } +static int snort_analyze_packets_tbl(odp_packet_t pkt_tbl[], unsigned len, int thr) +{ + odp_packet_t pkt; + unsigned pkt_cnt = len; + unsigned i, j; + + for (i = 0; i < len; ++i) { + pkt = pkt_tbl[i]; + analyze_packet_in_snort(pkt, thr); + odp_packet_free(pkt); + } + + return pkt_cnt; +} + /** * Packet IO loopback worker thread using ODP queues * @@ -148,8 +164,8 @@ static void *pktio_queue_thread(void *arg) } /* Open a packet IO instance for this thread */ - sock_params->type = ODP_PKTIO_TYPE_SOCKET_BASIC; - sock_params->fanout = 0; + sock_params->type = thr_args->type; + sock_params->fanout = thr_args->fanout; pktio = odp_pktio_open(thr_args->pktio_dev, pkt_pool, ¶ms); if (pktio == ODP_PKTIO_INVALID) { ODP_ERR(" [%02i] Error: pktio create failed\n", thr); @@ -194,20 +210,16 @@ static void *pktio_queue_thread(void *arg) buf = odp_queue_deq(inq_def); #endif - if (!odp_buffer_is_valid(buf)) + + if (!odp_buffer_is_valid(buf)) { continue; + } pkt = odp_packet_from_buffer(buf); analyze_packet_in_snort(pkt, thr); - -#if 0 - /* Print packet counts every once in a while */ - if (odp_unlikely(pkt_cnt++ % 100000 == 0)) { - printf(" [%02i] pkt_cnt:%lu\n", thr, pkt_cnt); - fflush(NULL); - } -#endif + odp_buffer_free(buf); + pc.total_from_daq++; } /* unreachable */ @@ -240,12 +252,13 @@ static void *pktio_ifburst_thread(void *arg) /* Lookup the packet pool */ pkt_pool = odp_buffer_pool_lookup("packet_pool"); - if (pkt_pool == ODP_BUFFER_POOL_INVALID || pkt_pool != thr_args->pool) { + if (pkt_pool == ODP_BUFFER_POOL_INVALID) { ODP_ERR(" [%02i] Error: pkt_pool not found\n", thr); return NULL; } /* Open a packet IO instance for this thread */ + sock_params->type = thr_args->type; sock_params->fanout = thr_args->fanout; pktio = odp_pktio_open(thr_args->pktio_dev, pkt_pool, ¶ms); @@ -261,27 +274,8 @@ static void *pktio_ifburst_thread(void *arg) for (;;) { pkts = odp_pktio_recv(pktio, pkt_tbl, MAX_PKT_BURST); if (pkts > 0) { - /* Drop packets with errors */ - pkts_ok = drop_err_pkts(pkt_tbl, pkts); - if (pkts_ok > 0) { - /* Swap Eth MACs and IP-addrs */ - swap_pkt_addrs(pkt_tbl, pkts_ok); - odp_pktio_send(pktio, pkt_tbl, pkts_ok); - } - - if (odp_unlikely(pkts_ok != pkts)) - ODP_ERR("Dropped frames:%u - err_cnt:%lu\n", - pkts-pkts_ok, ++err_cnt); - - /* Print packet counts every once in a while */ - tmp += pkts_ok; - if (odp_unlikely((tmp >= 100000) || /* OR first print:*/ - ((pkt_cnt == 0) && ((tmp-1) < MAX_PKT_BURST)))) { - pkt_cnt += tmp; - printf(" [%02i] pkt_cnt:%lu\n", thr, pkt_cnt); - fflush(NULL); - tmp = 0; - } + snort_analyze_packets_tbl(pkt_tbl, pkts, thr); + pc.total_from_daq += pkts; } } @@ -315,15 +309,44 @@ int do_odp_init(int argc, char *argv[]) } memset(args, 0, sizeof(*args)); - /* Parse and store the application arguments */ - //parse_args(argc, argv, &args->appl); - args->appl.core_count = 1; - args->appl.if_count = 1; - args->appl.if_names = calloc(args->appl.if_count, sizeof(char *)); - args->appl.if_names[0] = strdup("eth0"); + /* Parse -i argument, other arguments will be parsed by Snort. */ + parse_args(argc, argv, &args->appl); + + char *tmp = getenv("ODP_CORES"); + if (tmp) + args->appl.core_count = atoi(tmp); + else + args->appl.core_count = 1; - args->appl.fanout = 0; - args->appl.mode = APPL_MODE_PKT_QUEUE; + /* always use fanout. */ + args->appl.fanout = 1; + + /* Use queque mode by default */ + tmp = getenv("ODP_PTK_BURST"); + if (tmp) + args->appl.mode = APPL_MODE_PKT_BURST; + else + args->appl.mode = APPL_MODE_PKT_QUEUE; + + + tmp = getenv("ODP_PKTIO_TYPE_SOCKET"); + if (tmp) + args->appl.type = atoi(tmp); + else + args->appl.type = ODP_PKTIO_TYPE_SOCKET_BASIC; + + switch (args->appl.type) { + case ODP_PKTIO_TYPE_SOCKET_MMSG: + printf("using ODP_PKTIO_TYPE_SOCKET_MMSG\n"); + break; + case ODP_PKTIO_TYPE_SOCKET_MMAP: + printf("using ODP_PKTIO_TYPE_SOCKET_MMAP\n"); + break; + case ODP_PKTIO_TYPE_SOCKET_BASIC: + default: + printf("using ODP_PKTIO_TYPE_SOCKET_BASIC\n"); + break; + } /* Print both system and application information */ print_info(NO_PATH(argv[0]), &args->appl); @@ -384,6 +407,11 @@ void odp_snort_run_threads(void) printf("Num worker threads: %i\n", num_workers); + /* Clear Snort stats */ + memset(&pc, 0, sizeof(PacketCount)); + /* Set snort start time */ + TimeStart(); + /* Create and init worker threads */ memset(thread_tbl, 0, sizeof(thread_tbl)); for (i = 0; i < num_workers; ++i) { @@ -418,75 +446,6 @@ void odp_snort_run_threads(void) printf("Exit\n\n"); } - -/** - * Drop packets which input parsing marked as containing errors. - * - * Frees packets with error and modifies pkt_tbl[] to only contain packets with - * no detected errors. - * - * @param pkt_tbl Array of packet - * @param len Length of pkt_tbl[] - * - * @return Number of packets with no detected error - */ -static int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned len) -{ - odp_packet_t pkt; - unsigned pkt_cnt = len; - unsigned i, j; - - for (i = 0, j = 0; i < len; ++i) { - pkt = pkt_tbl[i]; - - if (odp_unlikely(odp_packet_error(pkt))) { - odp_packet_free(pkt); /* Drop */ - pkt_cnt--; - } else if (odp_unlikely(i != j++)) { - pkt_tbl[j] = pkt; - } - } - - return pkt_cnt; -} - -/** - * Swap eth src<->dst and IP src<->dst addresses - * - * @param pkt_tbl Array of packets - * @param len Length of pkt_tbl[] - */ - -static void swap_pkt_addrs(odp_packet_t pkt_tbl[], unsigned len) -{ - odp_packet_t pkt; - odp_ethhdr_t *eth; - odp_ethaddr_t tmp_addr; - odp_ipv4hdr_t *ip; - uint32be_t ip_tmp_addr; /* tmp ip addr */ - unsigned i; - - for (i = 0; i < len; ++i) { - pkt = pkt_tbl[i]; - if (odp_packet_inflag_eth(pkt)) { - eth = (odp_ethhdr_t *)odp_packet_l2(pkt); - - tmp_addr = eth->dst; - eth->dst = eth->src; - eth->src = tmp_addr; - - if (odp_packet_inflag_ipv4(pkt)) { - /* IPv4 */ - ip = (odp_ipv4hdr_t *)odp_packet_l3(pkt); - - ip_tmp_addr = ip->src_addr; - ip->src_addr = ip->dst_addr; - ip->dst_addr = ip_tmp_addr; - } - } - } -} - /** * Parse and store the command line arguments * @@ -502,33 +461,17 @@ static void parse_args(int argc, char *argv[], appl_args_t *appl_args) size_t len; int i; static struct option longopts[] = { - {"count", required_argument, NULL, 'c'}, {"interface", required_argument, NULL, 'i'}, /* return 'i' */ - {"mode", required_argument, NULL, 'm'}, /* return 'm' */ - {"help", no_argument, NULL, 'h'}, /* return 'h' */ {NULL, 0, NULL, 0} }; - appl_args->mode = -1; /* Invalid, must be changed by parsing */ - appl_args->type = 3; /* 3: ODP_PKTIO_TYPE_SOCKET_MMAP */ - appl_args->fanout = 1; /* turn off fanout by default for mmap */ - - for (i = 0; i < argc; i++) - printf("argv %s\n", argv[i]); - while (1) { - opt = getopt_long(argc, argv, "+c:i:m:t:f:h", + opt = getopt_long(argc, argv, "+i:", longopts, &long_index); - - printf("opt %d\n", opt); if (opt == -1) break; /* No more options */ switch (opt) { - case 'c': - appl_args->core_count = atoi(optarg); - break; - /* parse packet-io interface names */ case 'i': len = strlen(optarg); if (len == 0) { @@ -570,40 +513,11 @@ static void parse_args(int argc, char *argv[], appl_args_t *appl_args) appl_args->if_names[i] = token; } break; - - case 'm': - i = atoi(optarg); - if (i == 0) - appl_args->mode = APPL_MODE_PKT_BURST; - else - appl_args->mode = APPL_MODE_PKT_QUEUE; - break; - - case 't': - appl_args->type = atoi(optarg); - break; - - case 'f': - appl_args->fanout = atoi(optarg); - break; - - case 'h': - usage(argv[0]); - exit(EXIT_SUCCESS); - break; - default: break; } } - if (appl_args->if_count == 0 || appl_args->mode == -1) { - usage(argv[0]); - printf("unable to parse odp args if_count %d mode %d\n", - appl_args->if_count, appl_args->mode); - exit(EXIT_FAILURE); - } - optind = 1; /* reset 'extern optind' from the getopt lib */ }