From patchwork Fri Jul 19 20:33:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 169261 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp4300021ilk; Fri, 19 Jul 2019 13:33:28 -0700 (PDT) X-Google-Smtp-Source: APXvYqz2FfJkzTXwfXig0GWhk7QKym4HK6Ocgv2/mURx4Q/zqdPhLPPoK4bKffs/MVtcKQMKiJy8 X-Received: by 2002:a63:36cc:: with SMTP id d195mr15688036pga.157.1563568408364; Fri, 19 Jul 2019 13:33:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563568408; cv=none; d=google.com; s=arc-20160816; b=FBW4UtcAsxWN0YT1BD3lkaA3Ed0Gji7XGF4+lo+hrcvcvvrw+4lzWoxHqmBVjdTCOg TIZpPrYiQjUH6MDjoiwrVjB2fc9Rr+5HJmHqk5nztxbGh4tpfHjxmBYF+S74W8mykx49 jN/cTz8DGhXFCzPbgqHLQVc3aRW8mwuQktO23E7wsaOQvrv9EB/TmRBPB5cJckGe265/ hwU/EWResoOEwSfy1LhDymO+AOVWVtLc4wVYcpbOnP3hAV9fd5hw+RybQJ3Y5sBIz1i2 aDYG5ssPrpY8Iyyr7nz0UChoe9MI9vZsT7oPReI2FAarwb/YU5nv5292uBqmVzhFK7bj 6Idg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:message-id:date:to:from:dkim-signature:delivered-to; bh=zifQIgBxw8Rflk/JH8+Xmb3NzWzjqrBHanGVs2s9+hY=; b=XNljBmj9DoLBH0OV4G6umHH4c4lOJpmee+rRc9zQzF7GzSt1j7ks2y2icylW+4kuSb vjvEDqFsh8E7bPTnxUS0sXV/Bw90lHSaQuJXEcz8phYWJjyZzULcE6Q0D64xQPBHtdHt 7NzW+ZxBtr4FT2j4iI/qLmd2D5yw/iUfbIR4gChhqNqrRVWIUeI929g04FpmnX1RyXhu sv3n4fbvq6UEDqdavfEqB5KDndntLfLlMaYXGdlwdVTBTdqCdeP4nGE636zC4faIE8Vo UfpsMZ9pcdWrnUoFqJ+JOdTUrsdsPk3j+zhAvRX7Yzi5W3fS3D+uBxFyo3N3BY7jrXNr Y/mg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=iv5j58w2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id t17si2760399pfe.269.2019.07.19.13.33.27; Fri, 19 Jul 2019 13:33:28 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=iv5j58w2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id E0C307F1D4; Fri, 19 Jul 2019 20:33:24 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by mail.openembedded.org (Postfix) with ESMTP id BBDAE7F022 for ; Fri, 19 Jul 2019 20:33:23 +0000 (UTC) Received: by mail-wm1-f47.google.com with SMTP id g67so26012586wme.1 for ; Fri, 19 Jul 2019 13:33:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=jG6IE2c5pJrCXjYWZfnMOlyX0lyiXgAM+gzZDJT1IC0=; b=iv5j58w21Kp6/+2bj+x3QmMdiw7IWYqpPuX7e7P+YKqidN64GgBNY3jyOwxO6zlrMO a/kcUTQoGbUFCHmePDDKy97nb78EHqwgMzN6+1qyNlI2CQUp505PvXPjJyEF+GTDfudT lE0GlR2FiGoST/RkOWCJn9fvThz4qWlKBuENa3UtSdCWMbIyePEIB1YVUqhgwO0BzWXk v5/aSq5Mikj5CdLVblfa8vBmtUyd9gBbMmT6Xr1gyxQaYrSamMYU49tZV02mzjGO8Whz Whn6OryHk+uDS6fbKoB5NC1XUJqA0dXO8NTkxH7gm1JLDm0yDCih36kgLkvspVoTKnYU Wf9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=jG6IE2c5pJrCXjYWZfnMOlyX0lyiXgAM+gzZDJT1IC0=; b=e2xhUFwXwBXkYQOu3zeHwAJKCzqj0Bv6XlN8GqUMVmBzs5wwdW8m+1RkS6ZsPS3jha z40xFQ7+8VTXiuiCmSbg7EBHjdXnqACnAoPIre4d6GJ6BiBzfKWq4e6F4Ca47OvzLc3Z fs0i66k+HEYC99FXp5WZuwsPc9tuacNV4k8Y+RH1T78BHrtM+gYrBsp95nUSKq8bAanh 0Z9SLR2LAdt5l1IPqH0zu6xAY4cTIyhw7r47eOAkZXnuYCh8bRWvwuPb0NmuaW2rS3UF 5LuTIcqNQg6AItUy7CjAPifwX6+2z1i6s2D2ZCt74nNrtxFC4iQJTcs9WAfJcfcwOvJ9 xlcQ== X-Gm-Message-State: APjAAAUgrzmmCtMzvpwqFcKKuMs5Igwk7RZYfVeSYWqvLzSNLfqNiGxo S/N1TqN5QxaUw2Ug6QxUuJkIxzDwRYo= X-Received: by 2002:a05:600c:2189:: with SMTP id e9mr46986179wme.56.1563568404093; Fri, 19 Jul 2019 13:33:24 -0700 (PDT) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id z1sm35298988wrp.51.2019.07.19.13.33.23 for (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Fri, 19 Jul 2019 13:33:23 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Fri, 19 Jul 2019 21:33:17 +0100 Message-Id: <20190719203319.20580-1-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [OE-core] [PATCH 1/3] cve-update-db-native: use executemany() to optimise CPE insertion X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Instead of calling execute() repeatedly, rewrite the function to be a generator and use executemany() for performance. Signed-off-by: Ross Burton --- .../recipes-core/meta/cve-update-db-native.bb | 85 +++++++------------ 1 file changed, 32 insertions(+), 53 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index cabbde5066c..09e19c0aaef 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -102,70 +102,49 @@ def initialize_db(c): VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ VERSION_END TEXT, OPERATOR_END TEXT)") -def insert_elt(c, db_values): - query = "insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)" - c.execute(query, db_values) - def parse_node_and_insert(c, node, cveId): # Parse children node if needed - try: - for child in node['children']: - parse_node_and_insert(c, child, cveId) - except: - pass - - # Exit if the cpe_match node does not exists - try: - cpe_match = node['cpe_match'] - except: - return - - for cpe in cpe_match: - if not cpe['vulnerable']: - return - cpe23 = cpe['cpe23Uri'].split(':') - vendor = cpe23[3] - product = cpe23[4] - version = cpe23[5] - - if version != '*': - # Version is defined, this is a '=' match - db_values = [cveId, vendor, product, version, '=', '', ''] - insert_elt(c, db_values) - else: - # Parse start version, end version and operators - op_start = '' - op_end = '' - v_start = '' - v_end = '' - - try: - if cpe['versionStartIncluding']: + for child in node.get('children', ()): + parse_node_and_insert(c, child, cveId) + + def cpe_generator(): + for cpe in node.get('cpe_match', ()): + if not cpe['vulnerable']: + return + cpe23 = cpe['cpe23Uri'].split(':') + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + + if version != '*': + # Version is defined, this is a '=' match + yield [cveId, vendor, product, version, '=', '', ''] + else: + # Parse start version, end version and operators + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + if 'versionStartIncluding' in cpe: op_start = '>=' v_start = cpe['versionStartIncluding'] - except: - pass - try: - if cpe['versionStartExcluding']: + + if 'versionStartExcluding' in cpe: op_start = '>' v_start = cpe['versionStartExcluding'] - except: - pass - try: - if cpe['versionEndIncluding']: + + if 'versionEndIncluding' in cpe: op_end = '<=' v_end = cpe['versionEndIncluding'] - except: - pass - try: - if cpe['versionEndExcluding']: + + if 'versionEndExcluding' in cpe: op_end = '<' v_end = cpe['versionEndExcluding'] - except: - pass - db_values = [cveId, vendor, product, v_start, op_start, v_end, op_end] - insert_elt(c, db_values) + yield [cveId, vendor, product, v_start, op_start, v_end, op_end] + + c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) def update_db(c, json_filename): import json From patchwork Fri Jul 19 20:33:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 169262 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp4300117ilk; Fri, 19 Jul 2019 13:33:36 -0700 (PDT) X-Google-Smtp-Source: APXvYqwRsHMVTEl/tBd/PJI5GZm38Z+F4qH6drwogvFX3ktdwWMHI/yVZr4j2ATdPZThUw/0JwY8 X-Received: by 2002:a17:902:9a82:: with SMTP id w2mr58447314plp.291.1563568416551; Fri, 19 Jul 2019 13:33:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563568416; cv=none; d=google.com; s=arc-20160816; b=WelgZOxGFk42GWbN+Jepg7zSQbmZ4D4fu2o8E7LBsxkXpewSh+SEusPg9euh7kN6+k ZeAJa6RNvIiw0K8RcG2ioAXXzpzkhhWYfJdXMeMp/+biYwuzT1nYhPIbFagSUJw74BKf eMgwgTR+L4BMpWz21Htr9Kep9UOcLwNQ6gOktldq/khBzaluoAENKxt2i+mbuZfs434j /IFSHFl75uovUok1/jVfoMBf5voQ3lbcqBBEaptcgdF9x5UrHDsn+Li8CKNR8nbG7+fJ aNRqVW+aRBuwz6MlzvOiih1+30ltO17KbPJBkH8/+fO5UFfi5zWPbhiPB97lPfsvHzN1 Pomw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=iU4wwzVS2ZWZLCwyAA4MU0PVhSYE5IACCKiAU5yrZ6I=; b=AxJyVm2pWkhAX5s3h2k8nI8t1P4iGx2R/YGP44s0FS2mJZpsSVpsVEbUctpILHV8vc YxEcFPLY58vNlvnq4LYW6ci8301Kr/Wl8KAfLxd3pbnwdEJNVIZazLqw8CnexH9biC2W E4Mti3M55lfRCII1oI9locMs0rNRbA40urQiqF0p7bsvFnGbMiNfZAS6x0E7fw58YJsm AvCPQGEkYF52KJri5n8C1O5cvEXn2h5Wg2cl79nSk687zMKzB7QH3CGPdUq4pysuvt9/ vbYL+VH7beDvuY3MqkMgapdnuNNqedvMUsl/1YUDcuSkbzbzxKpBeVTN7fyeNsiAmb3U 8GaA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=utLCdmS3; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id u185si587594pgd.561.2019.07.19.13.33.36; Fri, 19 Jul 2019 13:33:36 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=utLCdmS3; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 688307F1D7; Fri, 19 Jul 2019 20:33:33 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by mail.openembedded.org (Postfix) with ESMTP id D3F5F7F1C9 for ; Fri, 19 Jul 2019 20:33:24 +0000 (UTC) Received: by mail-wm1-f46.google.com with SMTP id 207so29903608wma.1 for ; Fri, 19 Jul 2019 13:33:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=OMjkKOr6498GEdEZisk7t+n/Vc+Lnh2P/4ijmX4iutc=; b=utLCdmS3VZoHGUiueNjO8SBECelBeebCb/ycMYHS8YZXQZekbqXn1aUwlu5A2M08Cq Z2uIeOQcUkam1J4RMJ2nUGUf4/dFBEguSQuy4gaDdWUX5k5p6yN4sb0EdY9FpUdG3PJi FRgLQ2HJiR0aWo8gCgjmkTxGLMM7rbB7tuCQ1A+ozs+D7NgTF7+1E1FQUTr4MmoXeKoB ErhqnsyacvQ/ag4wZq+FfDgOF9ymBbL9anEhlVfuPC6zrv6i16pWnXwi9KtXwof+7wPd b8E/Lqo37ibCLTpXV4AUg0pw50n3hArjVK84wLkxmExNi90V3pOp8RlpV6Cl007jW3Kk icgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OMjkKOr6498GEdEZisk7t+n/Vc+Lnh2P/4ijmX4iutc=; b=ni5hLeuSnRQ6kGsMXxqV/ox0KjWL+D5tM8xgcl7awM+if/oEstvZ29/NRMtzlZVd9w yviDx/O9O/VuMBK11vFjThqrVrPLEU1tyL2A9wq1DYMXc2E8TnUkgXbzLUPUL+2tEYWe p1ylJXrf/hntlHpOQrUAvpv8A9rxhP4pPgpxeMCh4cRjdzlAkhP8540mOpjjkzoXlmcJ U1jtYuvhgNSZrL3QbmqvYz38qq5uBZgBZEwN96qc7WY7Arn+wOpNJOXuxQjT+5mLBfhk sjFNF7VixNSA1wsBVUwMykkXAUjOEQyj2V3ydX0uRxpyNDlVpQlZTrsIgMye/Vkh3ZJZ ZSrA== X-Gm-Message-State: APjAAAUgUZqcsggkzlCBThEEQrHEbtWo232BpnN3Rm23VweLLOtScvOa fmm0JJfHCu3ae3gOYibIOZdN0kfPals= X-Received: by 2002:a1c:1a87:: with SMTP id a129mr48949791wma.21.1563568405327; Fri, 19 Jul 2019 13:33:25 -0700 (PDT) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id z1sm35298988wrp.51.2019.07.19.13.33.24 for (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Fri, 19 Jul 2019 13:33:24 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Fri, 19 Jul 2019 21:33:18 +0100 Message-Id: <20190719203319.20580-2-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190719203319.20580-1-ross.burton@intel.com> References: <20190719203319.20580-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 2/3] cve-update-db-native: improve metadata parsing X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org The metadata parser is fragile: first it coerces a bytes() to a str() (so the string is b'LastModifiedDate:2019...'), assumes the first line is the date, and then uses a regex to parse (which then includes the trailing quote as part of the date). Clean this up by parsing the bytes as UTF-8 (ASCII is probably fine, but this is safer), iterate through the lines and split on colons to find the right key/value pair. Signed-off-by: Ross Burton --- meta/recipes-core/meta/cve-update-db-native.bb | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 09e19c0aaef..41a2aa8f207 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -22,7 +22,7 @@ python do_populate_cve_db() { Update NVD database with json data feed """ - import sqlite3, urllib, shutil, gzip, re + import sqlite3, urllib, shutil, gzip from datetime import date BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" @@ -52,13 +52,15 @@ python do_populate_cve_db() { req = urllib.request.Request(meta_url) if proxy: req.set_proxy(proxy, 'https') - try: - with urllib.request.urlopen(req, timeout=1) as r: - date_line = str(r.read().splitlines()[0]) - last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1) - except: - cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') - break + with urllib.request.urlopen(req) as r: + for l in r.read().decode("utf-8").splitlines(): + key, value = l.split(":", 1) + if key == "lastModifiedDate": + last_modified = value + break + else: + bb.warn("Cannot parse CVE metadata, update failed") + return # Compare with current db last modified date c.execute("select DATE from META where YEAR = ?", (year,)) From patchwork Fri Jul 19 20:33:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 169263 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp4300196ilk; Fri, 19 Jul 2019 13:33:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqzqjwxa1vS9olo1IGfp8YbbSNIHmpkpmZY+OSL1BNeJ2cQ1hiIEZWLuKzR9jhFP6xNIjSoF X-Received: by 2002:a63:bd0a:: with SMTP id a10mr55298499pgf.55.1563568423302; Fri, 19 Jul 2019 13:33:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563568423; cv=none; d=google.com; s=arc-20160816; b=USTADVBl/onp1jrZ+1Yo8sQ/1Bz30R32Z751iUql2iRdATEv7VCPOB3+rxHdoF3eXO 4HwZLkdIUrdKz/2l2Aw4qT8ctLklrh+83d3VPps7MEWo105w1DaETcRXBnwvgfdUbOwY M5vNlJ7iLlpA2iiWnm9/7vwtJttlqc+UnBHk2ax0s7OCJNxVfFW0zHDYijVKnUw2a/oW geYQfN5xw9NP6zDDpJsZKTkBPhEvr1nhhBrJmiRxy6SrVENGxVV5XIxwmTYMjIJTTn8o 7nj8X5ej1DBvkpjD4Kf8GnScQqfxORWdM0DGtIgfnb2K8GX4Z2G8tW/X1oAOmAl6OH1d pxjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=3rEFA/REZfEiksIv9wFUFAGiCs4vrdWXE2m4tbSs81I=; b=Do0k8BrT/PGrbsJmfMcJR89A36QZIB0yml3u/0s+mhm1DyRDzcE9Euv/Y3NTDzyx00 aJp+ORcdGFXvoDpqv6VlAgRnv6QrQsKlAOCrLwSflDoVg3ywy1efH7hSGYmjpWQPN8JG 1t9aw3bJrPO5RG6MS13O/aoWVFCyW6mZ/j4OGN342ibgs0prdz2q1bYhPhhWAMUmjDhg alt+B7IJiiDoYRTDlR/RvMPMc3Lu0QYS38eDgRARcxXhdYaWErWoAT4GTyYq8LLd2Mdl dwUdNiQnbtxnobnan+KdXMFHezXtLDcyCcRlwXmtcOWwiSCLb+bd3DbG1Fqj64eGVbBG pqcA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=DFJJbO8m; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id d31si662615pla.84.2019.07.19.13.33.43; Fri, 19 Jul 2019 13:33:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=DFJJbO8m; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 437C77F1DB; Fri, 19 Jul 2019 20:33:40 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by mail.openembedded.org (Postfix) with ESMTP id 6CFB97F1CC for ; Fri, 19 Jul 2019 20:33:26 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id h19so24447085wme.0 for ; Fri, 19 Jul 2019 13:33:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=gRZ2nKNkQHNFH21Pp6bphmKeLPDT4193/vucv/vuzII=; b=DFJJbO8mQQeI9V9A6WZy/OUUZa5Pg0U4IGdSNBqD2xRQlJQKBE1AJQSCvWJhULKfKi UZ34A3MfW7XGkzJ2+mY7Ae2aSIY6jfbv72pjGwnoMoZpxDEKdkGbvTItHVmXoD3GF2sW TnoGK56oKOikJgM+RUgCg0xd+fwMNgqiOmJrzVGAoKhgpphCui6nhO2nIEfvTz3kXCWF A38t+HwJ0AyqDK1DNldJ9oLjjm2y81bdqLvJgi+rdYIXLtpCn2TBEsaDz9+uanDO484r EgPIt+jtR8u6+wLeymdcVJzz4XppCIsKzwSR1c+XIFmiGSiOEbLZW1+IrIwItGGIFIgk H/pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gRZ2nKNkQHNFH21Pp6bphmKeLPDT4193/vucv/vuzII=; b=DMZS///yOp7oE4Jqtzpk6mTa5vQGcxiom6S+3eY20VI/rdn958gbLB3v8wCxZtoFot JdPm63an2sBWS/8hZsIa3YhYTJ1IdJQcVAu/f18cFrE2UlR4K7V5d22sRHlznwpnVFtZ v7d3OgfFzpZt2gaixFl5D2OfL2UGff0epyOfFr8qsbDvK9KUNTcCL5bLSVUKDxCvr0O8 nmMVOMfkHwdmu7hfQiFcySR/si+vYv0TqVg/YYyM+uAuXkFkws8h2FeurExhfHUfnkjT TlQeS4q6D56JHPrh2EmrCSSUwmwVGyfdG9KlWFFJnfb/FQJ7mYI4HrNEdyU708heWluE lNzQ== X-Gm-Message-State: APjAAAUQXgDu+8k4YxqZLBfoUVuqbrGyzazsLD6R0g1aPtNKDBLJ3s9j xn8JBmWxuK76oVJ7MhptAnyz78QgtRc= X-Received: by 2002:a05:600c:228f:: with SMTP id 15mr46115667wmf.60.1563568406986; Fri, 19 Jul 2019 13:33:26 -0700 (PDT) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id z1sm35298988wrp.51.2019.07.19.13.33.25 for (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Fri, 19 Jul 2019 13:33:26 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Fri, 19 Jul 2019 21:33:19 +0100 Message-Id: <20190719203319.20580-3-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190719203319.20580-1-ross.burton@intel.com> References: <20190719203319.20580-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 3/3] cve-update-db-native: clean up JSON fetching X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Currently the code fetches the compressed JSON, writes it to a temporary file, uncompresses that with gzip and passes the fake file object to update_db(). Instead, uncompress the gzip'd data in memory and pass the JSON directly to update_db(). Signed-off-by: Ross Burton --- .../recipes-core/meta/cve-update-db-native.bb | 29 ++++++++----------- 1 file changed, 12 insertions(+), 17 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 41a2aa8f207..9c083bdc991 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -67,25 +67,20 @@ python do_populate_cve_db() { meta = c.fetchone() if not meta or meta[0] != last_modified: # Clear products table entries corresponding to current year - cve_year = 'CVE-' + str(year) + '%' - c.execute("delete from PRODUCTS where ID like ?", (cve_year,)) + c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,)) # Update db with current year json file - req = urllib.request.Request(json_url) - if proxy: - req.set_proxy(proxy, 'https') try: - with urllib.request.urlopen(req, timeout=1) as r, \ - open(json_tmpfile, 'wb') as tmpfile: - shutil.copyfileobj(r, tmpfile) - except: + req = urllib.request.Request(json_url) + if proxy: + req.set_proxy(proxy, 'https') + with urllib.request.urlopen(req) as r: + update_db(c, gzip.decompress(r.read())) + c.execute("insert or replace into META values (?, ?)", [year, last_modified]) + except urllib.error.URLError as e: cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') - break - - with gzip.open(json_tmpfile, 'rt') as jsonfile: - update_db(c, jsonfile) - c.execute("insert or replace into META values (?, ?)", - [year, last_modified]) + bb.warn("Cannot parse CVE data (%s), update failed" % e.reason) + return # Update success, set the date to cve_check file. if year == date.today().year: @@ -148,9 +143,9 @@ def parse_node_and_insert(c, node, cveId): c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) -def update_db(c, json_filename): +def update_db(c, jsondata): import json - root = json.load(json_filename) + root = json.loads(jsondata) for elt in root['CVE_Items']: if not elt['impact']: