From patchwork Wed Nov 6 15:37:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikko Rapeli X-Patchwork-Id: 178744 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp880641ilf; Wed, 6 Nov 2019 09:00:31 -0800 (PST) X-Google-Smtp-Source: APXvYqyTsKueM9Vn8XlqEPC3ja6dYJvuZVBAxmpxfwoRROxceRJ/BvOwzSwxhOYLNmM/U3ExIK9t X-Received: by 2002:a65:6149:: with SMTP id o9mr4025399pgv.228.1573059631275; Wed, 06 Nov 2019 09:00:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573059631; cv=none; d=google.com; s=arc-20160816; b=kzRG1WdSnQvmmut1ATYU6Gtlh3lIXM0rznK/7Cdh9paY0M/x4pJdun47gLztqBHo45 rl55xQAapI0RdO80n+f1YmwSy1tLxWQaA27ThkEvjPWrIy4gtAA6Oh2VHKhRuwOOTsZg LmTWSxVlff6nsDZJ3Lr+ImajAVGSt813YPj6XODz/NXahgjJr3NU/Lvo/o4ITyPeJnfI zM9RavQepYkFrKxf22gKgajTEtCFSI8iLQrKeLNNhJBYU6t4DHnoh9Y5k/4nKcwZoWr/ tcNdHqtRtj9HdakcTmBLCLmXUK5wctJXjqvStRkantUitRyooBOrBmC793cGpMiDAmqZ wBKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:references :in-reply-to:message-id:date:to:from:dkim-signature:delivered-to; bh=Ath0qUY0eoT30GzffJHqzHsFjV8d+wmQfQbzjYHt0ns=; b=aDjznVDW+OxYiXN9MKXu4R7HdBLA6JlpSXn9541x7EUy+iNqT4dIaZUxwCwZdB6Bbl MGccr7mwdjcICePLGV95iWzXR3/cHkO8s5KeBJGeJmxamfX7zdIfuJYhA+1JICb4eCOc cZZOhA4kkNDesaKq3Hedfl1xCHcMsHgIl9gX0qf8eJTqglth6eA+qYMzYme0yzqFXFGl PrUfxEx9yIvy/c0qnJ5S1JvtTRH6RECo+6bXyaLA88ZIDRgIb6OwQI0YBaMtk85Z0ipU VEcNBTBCL1ueE8aNO7qvG8ZwUWwUFzM6/+DWv5E8ctj8TdMlTXYpCkucSCYhGct/iiGY fsHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@kapsi.fi header.s=20161220 header.b=hcUYAWi9; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bmw.de Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id y144si1701809pfc.145.2019.11.06.09.00.30; Wed, 06 Nov 2019 09:00:31 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@kapsi.fi header.s=20161220 header.b=hcUYAWi9; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bmw.de Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id DAC647FA35; Wed, 6 Nov 2019 17:00:09 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail.kapsi.fi (mail.kapsi.fi [91.232.154.25]) by mail.openembedded.org (Postfix) with ESMTP id 38AC07F889 for ; Wed, 6 Nov 2019 15:38:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kapsi.fi; s=20161220; h=Subject:References:In-Reply-To:Message-Id:Date:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=k4MQMpMc0QHdn9nlctlEHw4Xrn9vqeDVwI0xIcK/uqk=; b=hcUYAWi9lF2qVK03kQ6FgqVZgJ JOWv6CGKiLN44f0VGzUCwDs+3TWzXg6hCeh1nQVNezOVcipNZnCMSTwiprx2/kHoRTU2RRMDmDNIs eU67qXbPsEMA3zSVAYmHQHeRyAFez9yUHRVey+LcpZPnz/Wb4j0qNBvVwYM+4/u2xo9SmCiE0JDwG uJM22bqeKMxiy1wB/AU4kOgi/ZQOfvc2QBgxV7LoEbwIffGhtjMgRBcGjCNDzQaUah6wlC92qFdl9 Xq0ox1SNejPb6ZjT+77MHjANjzHyqBatKtXpPmmZ1D50IKjr2GXBOin2W5+YO6ElwnLzfqwxXGLwg oRBRvhJg==; Received: from [2001:67c:1be8::12] (helo=lakka.kapsi.fi) by mail.kapsi.fi with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1iSNOa-00009s-7b; Wed, 06 Nov 2019 17:38:56 +0200 Received: from mcfrisk by lakka.kapsi.fi with local (Exim 4.84_2) (envelope-from ) id 1iSNO5-0007XB-OL; Wed, 06 Nov 2019 17:38:25 +0200 From: Mikko Rapeli To: openembedded-core@lists.openembedded.org Date: Wed, 6 Nov 2019 17:37:39 +0200 Message-Id: <45dd5c1feb860bc1153bc0e22b1707b122868eb2.1573047194.git.mikko.rapeli@bmw.de> X-Mailer: git-send-email 2.1.4 In-Reply-To: References: In-Reply-To: References: X-Rspam-Score: 8.3 (++++++++) X-Rspam-Report: Action: add header Symbol: ARC_NA(0.00) Symbol: DMARC_POLICY_SOFTFAIL(0.10) Symbol: FROM_HAS_DN(0.00) Symbol: RCPT_COUNT_THREE(0.00) Symbol: R_SPF_ALLOW(-0.20) Symbol: MULTIPLE_UNIQUE_HEADERS(4.89) Symbol: MIME_GOOD(-0.10) Symbol: TO_MATCH_ENVRCPT_ALL(0.00) Symbol: NEURAL_SPAM(0.00) Symbol: TO_DN_SOME(0.00) Symbol: RCVD_TLS_LAST(0.00) Symbol: MID_CONTAINS_FROM(1.00) Symbol: IP_SCORE(-0.19) Symbol: FORGED_SENDER(0.30) Symbol: R_DKIM_NA(0.00) Symbol: MIME_TRACE(0.00) Symbol: ASN(0.00) Symbol: FROM_NEQ_ENVFROM(0.00) Symbol: HFILTER_HOSTNAME_UNKNOWN(2.50) Symbol: RCVD_COUNT_TWO(0.00) Message: (SPF): spf allow Message-ID: 45dd5c1feb860bc1153bc0e22b1707b122868eb2.1573047194.git.mikko.rapeli@bmw.de X-Rspam-Status: Yes X-Rspam-Bar: ++++++++ X-SA-Exim-Connect-IP: 2001:67c:1be8::12 X-SA-Exim-Mail-From: mcfrisk@kapsi.fi X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail X-Spam-Level: X-Spam-Status: No, score=-0.1 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,RDNS_NONE,SPF_HELO_NONE,SPF_NEUTRAL, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.2 X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on mail.kapsi.fi) Subject: [OE-core] [PATCH RFC CFH][sumo 24/47] cve-update-db-native: use executemany() to optimise CPE insertion X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Instead of calling execute() repeatedly, rewrite the function to be a generator and use executemany() for performance. (From OE-Core rev: b309840b6aa3423b909a43499356e929c8761318) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/recipes-core/meta/cve-update-db-native.bb | 85 ++++++++++---------------- 1 file changed, 32 insertions(+), 53 deletions(-) -- 1.9.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 9470cbe..a5d8e32 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -97,70 +97,49 @@ def initialize_db(c): VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ VERSION_END TEXT, OPERATOR_END TEXT)") -def insert_elt(c, db_values): - query = "insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)" - c.execute(query, db_values) - def parse_node_and_insert(c, node, cveId): # Parse children node if needed - try: - for child in node['children']: - parse_node_and_insert(c, child, cveId) - except: - pass - - # Exit if the cpe_match node does not exists - try: - cpe_match = node['cpe_match'] - except: - return - - for cpe in cpe_match: - if not cpe['vulnerable']: - return - cpe23 = cpe['cpe23Uri'].split(':') - vendor = cpe23[3] - product = cpe23[4] - version = cpe23[5] - - if version != '*': - # Version is defined, this is a '=' match - db_values = [cveId, vendor, product, version, '=', '', ''] - insert_elt(c, db_values) - else: - # Parse start version, end version and operators - op_start = '' - op_end = '' - v_start = '' - v_end = '' - - try: - if cpe['versionStartIncluding']: + for child in node.get('children', ()): + parse_node_and_insert(c, child, cveId) + + def cpe_generator(): + for cpe in node.get('cpe_match', ()): + if not cpe['vulnerable']: + return + cpe23 = cpe['cpe23Uri'].split(':') + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + + if version != '*': + # Version is defined, this is a '=' match + yield [cveId, vendor, product, version, '=', '', ''] + else: + # Parse start version, end version and operators + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + if 'versionStartIncluding' in cpe: op_start = '>=' v_start = cpe['versionStartIncluding'] - except: - pass - try: - if cpe['versionStartExcluding']: + + if 'versionStartExcluding' in cpe: op_start = '>' v_start = cpe['versionStartExcluding'] - except: - pass - try: - if cpe['versionEndIncluding']: + + if 'versionEndIncluding' in cpe: op_end = '<=' v_end = cpe['versionEndIncluding'] - except: - pass - try: - if cpe['versionEndExcluding']: + + if 'versionEndExcluding' in cpe: op_end = '<' v_end = cpe['versionEndExcluding'] - except: - pass - db_values = [cveId, vendor, product, v_start, op_start, v_end, op_end] - insert_elt(c, db_values) + yield [cveId, vendor, product, v_start, op_start, v_end, op_end] + + c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) def update_db(c, json_filename): import json