From patchwork Fri Jul 19 20:33:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 169261 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp4300021ilk; Fri, 19 Jul 2019 13:33:28 -0700 (PDT) X-Google-Smtp-Source: APXvYqz2FfJkzTXwfXig0GWhk7QKym4HK6Ocgv2/mURx4Q/zqdPhLPPoK4bKffs/MVtcKQMKiJy8 X-Received: by 2002:a63:36cc:: with SMTP id d195mr15688036pga.157.1563568408364; Fri, 19 Jul 2019 13:33:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563568408; cv=none; d=google.com; s=arc-20160816; b=FBW4UtcAsxWN0YT1BD3lkaA3Ed0Gji7XGF4+lo+hrcvcvvrw+4lzWoxHqmBVjdTCOg TIZpPrYiQjUH6MDjoiwrVjB2fc9Rr+5HJmHqk5nztxbGh4tpfHjxmBYF+S74W8mykx49 jN/cTz8DGhXFCzPbgqHLQVc3aRW8mwuQktO23E7wsaOQvrv9EB/TmRBPB5cJckGe265/ hwU/EWResoOEwSfy1LhDymO+AOVWVtLc4wVYcpbOnP3hAV9fd5hw+RybQJ3Y5sBIz1i2 aDYG5ssPrpY8Iyyr7nz0UChoe9MI9vZsT7oPReI2FAarwb/YU5nv5292uBqmVzhFK7bj 6Idg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:message-id:date:to:from:dkim-signature:delivered-to; bh=zifQIgBxw8Rflk/JH8+Xmb3NzWzjqrBHanGVs2s9+hY=; b=XNljBmj9DoLBH0OV4G6umHH4c4lOJpmee+rRc9zQzF7GzSt1j7ks2y2icylW+4kuSb vjvEDqFsh8E7bPTnxUS0sXV/Bw90lHSaQuJXEcz8phYWJjyZzULcE6Q0D64xQPBHtdHt 7NzW+ZxBtr4FT2j4iI/qLmd2D5yw/iUfbIR4gChhqNqrRVWIUeI929g04FpmnX1RyXhu sv3n4fbvq6UEDqdavfEqB5KDndntLfLlMaYXGdlwdVTBTdqCdeP4nGE636zC4faIE8Vo UfpsMZ9pcdWrnUoFqJ+JOdTUrsdsPk3j+zhAvRX7Yzi5W3fS3D+uBxFyo3N3BY7jrXNr Y/mg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=iv5j58w2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id t17si2760399pfe.269.2019.07.19.13.33.27; Fri, 19 Jul 2019 13:33:28 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=iv5j58w2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id E0C307F1D4; Fri, 19 Jul 2019 20:33:24 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by mail.openembedded.org (Postfix) with ESMTP id BBDAE7F022 for ; Fri, 19 Jul 2019 20:33:23 +0000 (UTC) Received: by mail-wm1-f47.google.com with SMTP id g67so26012586wme.1 for ; Fri, 19 Jul 2019 13:33:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=jG6IE2c5pJrCXjYWZfnMOlyX0lyiXgAM+gzZDJT1IC0=; b=iv5j58w21Kp6/+2bj+x3QmMdiw7IWYqpPuX7e7P+YKqidN64GgBNY3jyOwxO6zlrMO a/kcUTQoGbUFCHmePDDKy97nb78EHqwgMzN6+1qyNlI2CQUp505PvXPjJyEF+GTDfudT lE0GlR2FiGoST/RkOWCJn9fvThz4qWlKBuENa3UtSdCWMbIyePEIB1YVUqhgwO0BzWXk v5/aSq5Mikj5CdLVblfa8vBmtUyd9gBbMmT6Xr1gyxQaYrSamMYU49tZV02mzjGO8Whz Whn6OryHk+uDS6fbKoB5NC1XUJqA0dXO8NTkxH7gm1JLDm0yDCih36kgLkvspVoTKnYU Wf9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=jG6IE2c5pJrCXjYWZfnMOlyX0lyiXgAM+gzZDJT1IC0=; b=e2xhUFwXwBXkYQOu3zeHwAJKCzqj0Bv6XlN8GqUMVmBzs5wwdW8m+1RkS6ZsPS3jha z40xFQ7+8VTXiuiCmSbg7EBHjdXnqACnAoPIre4d6GJ6BiBzfKWq4e6F4Ca47OvzLc3Z fs0i66k+HEYC99FXp5WZuwsPc9tuacNV4k8Y+RH1T78BHrtM+gYrBsp95nUSKq8bAanh 0Z9SLR2LAdt5l1IPqH0zu6xAY4cTIyhw7r47eOAkZXnuYCh8bRWvwuPb0NmuaW2rS3UF 5LuTIcqNQg6AItUy7CjAPifwX6+2z1i6s2D2ZCt74nNrtxFC4iQJTcs9WAfJcfcwOvJ9 xlcQ== X-Gm-Message-State: APjAAAUgrzmmCtMzvpwqFcKKuMs5Igwk7RZYfVeSYWqvLzSNLfqNiGxo S/N1TqN5QxaUw2Ug6QxUuJkIxzDwRYo= X-Received: by 2002:a05:600c:2189:: with SMTP id e9mr46986179wme.56.1563568404093; Fri, 19 Jul 2019 13:33:24 -0700 (PDT) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id z1sm35298988wrp.51.2019.07.19.13.33.23 for (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Fri, 19 Jul 2019 13:33:23 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Fri, 19 Jul 2019 21:33:17 +0100 Message-Id: <20190719203319.20580-1-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [OE-core] [PATCH 1/3] cve-update-db-native: use executemany() to optimise CPE insertion X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Instead of calling execute() repeatedly, rewrite the function to be a generator and use executemany() for performance. Signed-off-by: Ross Burton --- .../recipes-core/meta/cve-update-db-native.bb | 85 +++++++------------ 1 file changed, 32 insertions(+), 53 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index cabbde5066c..09e19c0aaef 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -102,70 +102,49 @@ def initialize_db(c): VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ VERSION_END TEXT, OPERATOR_END TEXT)") -def insert_elt(c, db_values): - query = "insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)" - c.execute(query, db_values) - def parse_node_and_insert(c, node, cveId): # Parse children node if needed - try: - for child in node['children']: - parse_node_and_insert(c, child, cveId) - except: - pass - - # Exit if the cpe_match node does not exists - try: - cpe_match = node['cpe_match'] - except: - return - - for cpe in cpe_match: - if not cpe['vulnerable']: - return - cpe23 = cpe['cpe23Uri'].split(':') - vendor = cpe23[3] - product = cpe23[4] - version = cpe23[5] - - if version != '*': - # Version is defined, this is a '=' match - db_values = [cveId, vendor, product, version, '=', '', ''] - insert_elt(c, db_values) - else: - # Parse start version, end version and operators - op_start = '' - op_end = '' - v_start = '' - v_end = '' - - try: - if cpe['versionStartIncluding']: + for child in node.get('children', ()): + parse_node_and_insert(c, child, cveId) + + def cpe_generator(): + for cpe in node.get('cpe_match', ()): + if not cpe['vulnerable']: + return + cpe23 = cpe['cpe23Uri'].split(':') + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + + if version != '*': + # Version is defined, this is a '=' match + yield [cveId, vendor, product, version, '=', '', ''] + else: + # Parse start version, end version and operators + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + if 'versionStartIncluding' in cpe: op_start = '>=' v_start = cpe['versionStartIncluding'] - except: - pass - try: - if cpe['versionStartExcluding']: + + if 'versionStartExcluding' in cpe: op_start = '>' v_start = cpe['versionStartExcluding'] - except: - pass - try: - if cpe['versionEndIncluding']: + + if 'versionEndIncluding' in cpe: op_end = '<=' v_end = cpe['versionEndIncluding'] - except: - pass - try: - if cpe['versionEndExcluding']: + + if 'versionEndExcluding' in cpe: op_end = '<' v_end = cpe['versionEndExcluding'] - except: - pass - db_values = [cveId, vendor, product, v_start, op_start, v_end, op_end] - insert_elt(c, db_values) + yield [cveId, vendor, product, v_start, op_start, v_end, op_end] + + c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) def update_db(c, json_filename): import json