From patchwork Sun Dec 8 18:35:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180962 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3446849ile; Sun, 8 Dec 2019 10:36:55 -0800 (PST) X-Google-Smtp-Source: APXvYqwBDD+362jjpE0nAi9ng/x3l5lNj0oxeOB8Ghx4mejbHmbNC9T25/qH3RG6aEtzjiuFrl7Z X-Received: by 2002:a17:902:d696:: with SMTP id v22mr25359341ply.66.1575830215477; Sun, 08 Dec 2019 10:36:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830215; cv=none; d=google.com; s=arc-20160816; b=TWNE7HOKvE5XDhtvg2vk1egB/8TQXvQYoBLzfYb3HTJBm9L+RtElkntRKhi5jl0hgA ieYIFu4aUaXYbd17QEnuyPtXdgWo95CRD2OG61PybEk2RrE/HhbyF4nxBeAUam+SxRBY hCobLL0/XfcxdtplCNCuJy/RajaOuYHOfLwghprEtgV991Fa1DVw3axDMfAx7ICCbKIu jTTpVvhx2tFnjDmgOLRuP5OVEsLCsy0/K3qt47pOTwMnrIiFMWDVCOGgtJsbszc4sSQb ANLrAOC2DdhFUr/Qmnusac8FpyE29vn0Lgysb+eAYIxnWWNJQ4NQWhF7Y1eKh4tIVV0i f79w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :cc:mime-version:message-id:date:to:from:delivered-to; bh=EzV+WUSI7ficAMGmxxj7RpN5l+gr284j6knA5Ze2xH4=; b=MD6Cc5W0DimABzvdMT22pTPKp4KGjeVDGDiygiPqFt5bOw3IXXdmXdxTail+dROqMX oXvmkljTtYqPdL0L7vhJM/u8vXgMZ3Kx+9aoQuGBvC0aLEVI0BpAFXDwnZrSTXKiizmu bu6QHJwfASm0yFNGfBaE3gNe0ZRR13FLJiYkBe0MzNsc3EA2P0A3Py03gVh+DqqPfJbQ YGxsIhncQ7ibnQQk2dKtS+60jexRSymuXhdHGyAR8Wwtu1srvacceCrcTk42jrf+l8Hg GwZgMLvsj5hdgqYByPsY/ZOwWBJlIua/1ZAlKIXr+4cE9kSCKSSk5GwKkgxuPmEPbTeQ AEqg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id c16si17969108pfn.99.2019.12.08.10.36.54; Sun, 08 Dec 2019 10:36:55 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 7F70C7F959; Sun, 8 Dec 2019 18:36:47 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from tulikuusama2.dnainternet.net (tulikuusama3.dnainternet.net [83.102.40.155]) by mail.openembedded.org (Postfix) with ESMTP id B278D7F959 for ; Sun, 8 Dec 2019 18:36:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id 6924729547; Sun, 8 Dec 2019 20:36:43 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 1.653 X-Spam-Level: * X-Spam-Status: No, score=1.653 tagged_above=-9999 required=6 tests=[HEXHASH_WORD=1, SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from tulikuusama2.dnainternet.net ([83.102.40.155]) by localhost (tulikuusama2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id KElX_tz9yCsw; Sun, 8 Dec 2019 20:36:41 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id D353329542; Sun, 8 Dec 2019 20:36:40 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id AA69D2C63; Sun, 8 Dec 2019 20:36:27 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:47 +0200 Message-Id: <20191208183557.32589-1-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Cc: Sergey Popovich , Jussi Kukkonen Subject: [OE-core] [thud-next][PATCH 01/11] cve-check: backport rewrite from master X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton As detailed at [1] the XML feeds provided by NIST are being discontinued on October 9th 2019. As cve-check-tool uses these feeds, cve-check.bbclass will be inoperable after this date. To ensure that cve-check continues working, backport the following commits from master to move away from the unmaintained cve-check-tool to our own Python code that fetches the JSON: 546d14135c5 cve-update-db: New recipe to update CVE database bc144b028f6 cve-check: Remove dependency to cve-check-tool-native 7f62a20b32a cve-check: Manage CVE_PRODUCT with more than one name 3bf63bc6084 cve-check: Consider CVE that affects versions with less than operator c0eabd30d7b cve-update-db: Use std library instead of urllib3 27eb839ee65 cve-check: be idiomatic 09be21f4d17 cve-update-db: Manage proxy if needed. 975793e3825 cve-update-db: do_populate_cve_db depends on do_fetch 0325dd72714 cve-update-db: Catch request.urlopen errors. 4078da92b49 cve-check: Depends on cve-update-db-native f7676e9a38d cve-update-db: Use NVD CPE data to populate PRODUCTS table bc0195be1b1 cve-check: Update unpatched CVE matching c807c2a6409 cve-update-db-native: Skip recipe when cve-check class is not loaded. 07bb8b25e17 cve-check: remove redundant readline CVE whitelisting 5388ed6d137 cve-check-tool: remove 270ac00cb43 cve-check.bbclass: initialize to_append e6bf9000987 cve-check: allow comparison of Vendor as well as Product 91770338f76 cve-update-db-native: use SQL placeholders instead of format strings 7069302a4cc cve-check: Replace CVE_CHECK_CVE_WHITELIST by CVE_CHECK_WHITELIST 78de2cb39d7 cve-update-db-native: Remove hash column from database. 4b301030cf9 cve-update-db-native: use os.path.join instead of + f0d822fad2a cve-update-db: actually inherit native b309840b6aa cve-update-db-native: use executemany() to optimise CPE insertion bb4e53af33d cve-update-db-native: improve metadata parsing 94227459792 cve-update-db-native: clean up JSON fetching 95438d52b73 cve-update-db-native: fix https proxy issues 1f9a963b9ff glibc: exclude child recipes from CVE scanning [1] https://nvd.nist.gov/General/News/XML-Vulnerability-Feed-Retirement (From OE-Core rev: 8c87e78547c598cada1bce92e7b25d85b994e2eb) Signed-off-by: Ross Burton Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie --- meta/classes/cve-check.bbclass | 142 +++++++----- meta/conf/distro/include/maintainers.inc | 1 + meta/recipes-core/glibc/glibc-locale.inc | 3 + meta/recipes-core/glibc/glibc-mtrace.inc | 3 + meta/recipes-core/glibc/glibc-scripts.inc | 3 + .../recipes-core/meta/cve-update-db-native.bb | 195 ++++++++++++++++ .../cve-check-tool/cve-check-tool_5.6.4.bb | 62 ----- ...x-freeing-memory-allocated-by-sqlite.patch | 50 ---- ...erriding-default-CA-certificate-file.patch | 215 ------------------ ...s-in-percent-when-downloading-CVE-db.patch | 135 ----------- ...omputed-vs-expected-sha256-digit-str.patch | 52 ----- ...heck-for-malloc_trim-before-using-it.patch | 51 ----- 12 files changed, 292 insertions(+), 620 deletions(-) create mode 100644 meta/recipes-core/meta/cve-update-db-native.bb delete mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 743bc08a4f..c00d2910be 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}" CVE_VERSION ??= "${PV}" CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK" -CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db" +CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_1.0.db" CVE_CHECK_LOG ?= "${T}/cve.log" CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check" @@ -37,32 +37,33 @@ CVE_CHECK_COPY_FILES ??= "1" CVE_CHECK_CREATE_MANIFEST ??= "1" # Whitelist for packages (PN) -CVE_CHECK_PN_WHITELIST = "\ - glibc-locale \ -" +CVE_CHECK_PN_WHITELIST ?= "" -# Whitelist for CVE and version of package -CVE_CHECK_CVE_WHITELIST = "{\ - 'CVE-2014-2524': ('6.3','5.2',), \ -}" +# Whitelist for CVE. If a CVE is found, then it is considered patched. +# The value is a string containing space separated CVE values: +# +# CVE_CHECK_WHITELIST = 'CVE-2014-2524 CVE-2018-1234' +# +CVE_CHECK_WHITELIST ?= "" python do_cve_check () { """ Check recipe for patched and unpatched CVEs """ - if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE")): + if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")): patched_cves = get_patches_cves(d) patched, unpatched = check_cves(d, patched_cves) if patched or unpatched: cve_data = get_cve_info(d, patched + unpatched) cve_write_data(d, patched, unpatched, cve_data) else: - bb.note("Failed to update CVE database, skipping CVE check") + bb.note("No CVE database found, skipping CVE check") + } addtask cve_check after do_unpack before do_build -do_cve_check[depends] = "cve-check-tool-native:do_populate_sysroot cve-check-tool-native:do_populate_cve_db" +do_cve_check[depends] = "cve-update-db-native:do_populate_cve_db" do_cve_check[nostamp] = "1" python cve_check_cleanup () { @@ -163,65 +164,94 @@ def get_patches_cves(d): def check_cves(d, patched_cves): """ - Run cve-check-tool looking for patched and unpatched CVEs. + Connect to the NVD database and find unpatched cves. """ - import ast, csv, tempfile, subprocess, io + from distutils.version import LooseVersion - cves_patched = [] cves_unpatched = [] - bpn = d.getVar("CVE_PRODUCT") + # CVE_PRODUCT can contain more than one product (eg. curl/libcurl) + products = d.getVar("CVE_PRODUCT").split() # If this has been unset then we're not scanning for CVEs here (for example, image recipes) - if not bpn: + if not products: return ([], []) pv = d.getVar("CVE_VERSION").split("+git")[0] - cves = " ".join(patched_cves) - cve_db_dir = d.getVar("CVE_CHECK_DB_DIR") - cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST")) - cve_cmd = "cve-check-tool" - cmd = [cve_cmd, "--no-html", "--skip-update", "--csv", "--not-affected", "-t", "faux", "-d", cve_db_dir] # If the recipe has been whitlisted we return empty lists if d.getVar("PN") in d.getVar("CVE_CHECK_PN_WHITELIST").split(): bb.note("Recipe has been whitelisted, skipping check") return ([], []) - try: - # Write the faux CSV file to be used with cve-check-tool - fd, faux = tempfile.mkstemp(prefix="cve-faux-") - with os.fdopen(fd, "w") as f: - for pn in bpn.split(): - f.write("%s,%s,%s,\n" % (pn, pv, cves)) - cmd.append(faux) - - output = subprocess.check_output(cmd).decode("utf-8") - bb.debug(2, "Output of command %s:\n%s" % ("\n".join(cmd), output)) - except subprocess.CalledProcessError as e: - bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output)) - finally: - os.remove(faux) - - for row in csv.reader(io.StringIO(output)): - # Third row has the unpatched CVEs - if row[2]: - for cve in row[2].split(): - # Skip if the CVE has been whitlisted for the current version - if pv in cve_whitelist.get(cve,[]): - bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve)) + old_cve_whitelist = d.getVar("CVE_CHECK_CVE_WHITELIST") + if old_cve_whitelist: + bb.warn("CVE_CHECK_CVE_WHITELIST is deprecated, please use CVE_CHECK_WHITELIST.") + cve_whitelist = d.getVar("CVE_CHECK_WHITELIST").split() + + import sqlite3 + db_file = d.getVar("CVE_CHECK_DB_FILE") + conn = sqlite3.connect(db_file) + + for product in products: + c = conn.cursor() + if ":" in product: + vendor, product = product.split(":", 1) + c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR IS ?", (product, vendor)) + else: + c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,)) + + for row in c: + cve = row[0] + version_start = row[3] + operator_start = row[4] + version_end = row[5] + operator_end = row[6] + + if cve in cve_whitelist: + bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) + elif cve in patched_cves: + bb.note("%s has been patched" % (cve)) + else: + to_append = False + if (operator_start == '=' and pv == version_start): + cves_unpatched.append(cve) else: + if operator_start: + try: + to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) + to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) + except: + bb.note("%s: Failed to compare %s %s %s for %s" % + (product, pv, operator_start, version_start, cve)) + to_append_start = False + else: + to_append_start = False + + if operator_end: + try: + to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) + to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) + except: + bb.note("%s: Failed to compare %s %s %s for %s" % + (product, pv, operator_end, version_end, cve)) + to_append_end = False + else: + to_append_end = False + + if operator_start and operator_end: + to_append = to_append_start and to_append_end + else: + to_append = to_append_start or to_append_end + + if to_append: cves_unpatched.append(cve) - bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve)) - # Fourth row has patched CVEs - if row[3]: - for cve in row[3].split(): - cves_patched.append(cve) - bb.debug(2, "%s-%s is patched for %s" % (bpn, pv, cve)) + bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve)) + conn.close() - return (cves_patched, cves_unpatched) + return (list(patched_cves), cves_unpatched) def get_cve_info(d, cves): """ - Get CVE information from the database used by cve-check-tool. + Get CVE information from the database. Unfortunately the only way to get CVE info is set the output to html (hard to parse) or query directly the database. @@ -241,9 +271,10 @@ def get_cve_info(d, cves): for row in cur.execute(query, tuple(cves)): cve_data[row[0]] = {} cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["score"] = row[2] - cve_data[row[0]]["modified"] = row[3] - cve_data[row[0]]["vector"] = row[4] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] conn.close() return cve_data @@ -270,7 +301,8 @@ def cve_write_data(d, patched, unpatched, cve_data): unpatched_cves.append(cve) write_string += "CVE STATUS: Unpatched\n" write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"] - write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"] + write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["scorev2"] + write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["scorev3"] write_string += "VECTOR: %s\n" % cve_data[cve]["vector"] write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve) diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc index 672f067792..c027901fdf 100644 --- a/meta/conf/distro/include/maintainers.inc +++ b/meta/conf/distro/include/maintainers.inc @@ -116,6 +116,7 @@ RECIPE_MAINTAINER_pn-cryptodev-tests = "Robert Yang " RECIPE_MAINTAINER_pn-cups = "Chen Qi " RECIPE_MAINTAINER_pn-curl = "Armin Kuster " RECIPE_MAINTAINER_pn-cve-check-tool = "Ross Burton " +RECIPE_MAINTAINER_pn-cve-update-db-native = "Ross Burton " RECIPE_MAINTAINER_pn-cwautomacros = "Ross Burton " RECIPE_MAINTAINER_pn-db = "Mark Hatle " RECIPE_MAINTAINER_pn-dbus = "Chen Qi " diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc index 1b676dc26e..97d83cb856 100644 --- a/meta/recipes-core/glibc/glibc-locale.inc +++ b/meta/recipes-core/glibc/glibc-locale.inc @@ -95,3 +95,6 @@ do_install () { inherit libc-package BBCLASSEXTEND = "nativesdk" + +# Don't scan for CVEs as glibc will be scanned +CVE_PRODUCT = "" diff --git a/meta/recipes-core/glibc/glibc-mtrace.inc b/meta/recipes-core/glibc/glibc-mtrace.inc index d703c14bdc..ef9d60ec23 100644 --- a/meta/recipes-core/glibc/glibc-mtrace.inc +++ b/meta/recipes-core/glibc/glibc-mtrace.inc @@ -11,3 +11,6 @@ do_install() { install -d -m 0755 ${D}${bindir} install -m 0755 ${SRC}/mtrace ${D}${bindir}/ } + +# Don't scan for CVEs as glibc will be scanned +CVE_PRODUCT = "" diff --git a/meta/recipes-core/glibc/glibc-scripts.inc b/meta/recipes-core/glibc/glibc-scripts.inc index 2a2b41507e..14a14e4512 100644 --- a/meta/recipes-core/glibc/glibc-scripts.inc +++ b/meta/recipes-core/glibc/glibc-scripts.inc @@ -18,3 +18,6 @@ do_install() { # sotruss script requires sotruss-lib.so (given by libsotruss package), # to produce trace of the library calls. RDEPENDS_${PN} += "libsotruss" + +# Don't scan for CVEs as glibc will be scanned +CVE_PRODUCT = "" diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb new file mode 100644 index 0000000000..2c427a5884 --- /dev/null +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -0,0 +1,195 @@ +SUMMARY = "Updates the NVD CVE database" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +inherit native + +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install +deltask do_populate_sysroot + +python () { + if not d.getVar("CVE_CHECK_DB_FILE"): + raise bb.parse.SkipRecipe("Skip recipe when cve-check class is not loaded.") +} + +python do_populate_cve_db() { + """ + Update NVD database with json data feed + """ + + import sqlite3, urllib, urllib.parse, shutil, gzip + from datetime import date + + BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" + YEAR_START = 2002 + + db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK') + db_file = os.path.join(db_dir, 'nvdcve_1.0.db') + json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') + proxy = d.getVar("https_proxy") + + if proxy: + # instantiate an opener but do not install it as the global + # opener unless if we're really sure it's applicable for all + # urllib requests + proxy_handler = urllib.request.ProxyHandler({'https': proxy}) + proxy_opener = urllib.request.build_opener(proxy_handler) + else: + proxy_opener = None + + cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') + + if not os.path.isdir(db_dir): + os.mkdir(db_dir) + + # Connect to database + conn = sqlite3.connect(db_file) + c = conn.cursor() + + initialize_db(c) + + for year in range(YEAR_START, date.today().year + 1): + year_url = BASE_URL + str(year) + meta_url = year_url + ".meta" + json_url = year_url + ".json.gz" + + # Retrieve meta last modified date + + response = None + + if proxy_opener: + response = proxy_opener.open(meta_url) + else: + req = urllib.request.Request(meta_url) + response = urllib.request.urlopen(req) + + if response: + for l in response.read().decode("utf-8").splitlines(): + key, value = l.split(":", 1) + if key == "lastModifiedDate": + last_modified = value + break + else: + bb.warn("Cannot parse CVE metadata, update failed") + return + + # Compare with current db last modified date + c.execute("select DATE from META where YEAR = ?", (year,)) + meta = c.fetchone() + if not meta or meta[0] != last_modified: + # Clear products table entries corresponding to current year + c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,)) + + # Update db with current year json file + try: + if proxy_opener: + response = proxy_opener.open(json_url) + else: + req = urllib.request.Request(json_url) + response = urllib.request.urlopen(req) + + if response: + update_db(c, gzip.decompress(response.read()).decode('utf-8')) + c.execute("insert or replace into META values (?, ?)", [year, last_modified]) + except urllib.error.URLError as e: + cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') + bb.warn("Cannot parse CVE data (%s), update failed" % e.reason) + return + + # Update success, set the date to cve_check file. + if year == date.today().year: + cve_f.write('CVE database update : %s\n\n' % date.today()) + + cve_f.close() + conn.commit() + conn.close() +} + +def initialize_db(c): + c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ + SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \ + VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ + VERSION_END TEXT, OPERATOR_END TEXT)") + +def parse_node_and_insert(c, node, cveId): + # Parse children node if needed + for child in node.get('children', ()): + parse_node_and_insert(c, child, cveId) + + def cpe_generator(): + for cpe in node.get('cpe_match', ()): + if not cpe['vulnerable']: + return + cpe23 = cpe['cpe23Uri'].split(':') + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + + if version != '*': + # Version is defined, this is a '=' match + yield [cveId, vendor, product, version, '=', '', ''] + else: + # Parse start version, end version and operators + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + if 'versionStartIncluding' in cpe: + op_start = '>=' + v_start = cpe['versionStartIncluding'] + + if 'versionStartExcluding' in cpe: + op_start = '>' + v_start = cpe['versionStartExcluding'] + + if 'versionEndIncluding' in cpe: + op_end = '<=' + v_end = cpe['versionEndIncluding'] + + if 'versionEndExcluding' in cpe: + op_end = '<' + v_end = cpe['versionEndExcluding'] + + yield [cveId, vendor, product, v_start, op_start, v_end, op_end] + + c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) + +def update_db(c, jsondata): + import json + root = json.loads(jsondata) + + for elt in root['CVE_Items']: + if not elt['impact']: + continue + + cveId = elt['cve']['CVE_data_meta']['ID'] + cveDesc = elt['cve']['description']['description_data'][0]['value'] + date = elt['lastModifiedDate'] + accessVector = elt['impact']['baseMetricV2']['cvssV2']['accessVector'] + cvssv2 = elt['impact']['baseMetricV2']['cvssV2']['baseScore'] + + try: + cvssv3 = elt['impact']['baseMetricV3']['cvssV3']['baseScore'] + except: + cvssv3 = 0.0 + + c.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)", + [cveId, cveDesc, cvssv2, cvssv3, date, accessVector]) + + configurations = elt['configurations']['nodes'] + for config in configurations: + parse_node_and_insert(c, config, cveId) + + +addtask do_populate_cve_db before do_fetch +do_populate_cve_db[nostamp] = "1" + +EXCLUDE_FROM_WORLD = "1" diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb deleted file mode 100644 index 1c84fb1cf2..0000000000 --- a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb +++ /dev/null @@ -1,62 +0,0 @@ -SUMMARY = "cve-check-tool" -DESCRIPTION = "cve-check-tool is a tool for checking known (public) CVEs.\ -The tool will identify potentially vunlnerable software packages within Linux distributions through version matching." -HOMEPAGE = "https://github.com/ikeydoherty/cve-check-tool" -SECTION = "Development/Tools" -LICENSE = "GPL-2.0+" -LIC_FILES_CHKSUM = "file://LICENSE;md5=e8c1458438ead3c34974bc0be3a03ed6" - -SRC_URI = "https://github.com/ikeydoherty/${BPN}/releases/download/v${PV}/${BP}.tar.xz \ - file://check-for-malloc_trim-before-using-it.patch \ - file://0001-print-progress-in-percent-when-downloading-CVE-db.patch \ - file://0001-curl-allow-overriding-default-CA-certificate-file.patch \ - file://0001-update-Compare-computed-vs-expected-sha256-digit-str.patch \ - file://0001-Fix-freeing-memory-allocated-by-sqlite.patch \ - " - -SRC_URI[md5sum] = "c5f4247140fc9be3bf41491d31a34155" -SRC_URI[sha256sum] = "b8f283be718af8d31232ac1bfc10a0378fb958aaaa49af39168f8acf501e6a5b" - -UPSTREAM_CHECK_URI = "https://github.com/ikeydoherty/cve-check-tool/releases" - -DEPENDS = "libcheck glib-2.0 json-glib curl libxml2 sqlite3 openssl ca-certificates" - -RDEPENDS_${PN} = "ca-certificates" - -inherit pkgconfig autotools - -EXTRA_OECONF = "--disable-coverage --enable-relative-plugins" -CFLAGS_append = " -Wno-error=pedantic" - -do_populate_cve_db() { - if [ "${BB_NO_NETWORK}" = "1" ] ; then - bbwarn "BB_NO_NETWORK is set; Can't update cve-check-tool database, new CVEs won't be detected" - return - fi - - # In case we don't inherit cve-check class, use default values defined in the class. - cve_dir="${CVE_CHECK_DB_DIR}" - cve_file="${CVE_CHECK_TMP_FILE}" - - [ -z "${cve_dir}" ] && cve_dir="${DL_DIR}/CVE_CHECK" - [ -z "${cve_file}" ] && cve_file="${TMPDIR}/cve_check" - - unused="${@bb.utils.export_proxies(d)}" - bbdebug 2 "Updating cve-check-tool database located in $cve_dir" - # --cacert works around curl-native not finding the CA bundle - if cve-check-update --cacert ${sysconfdir}/ssl/certs/ca-certificates.crt -d "$cve_dir" ; then - printf "CVE database was updated on %s UTC\n\n" "$(LANG=C date --utc +'%F %T')" > "$cve_file" - else - bbwarn "Error in executing cve-check-update" - if [ "${@'1' if bb.data.inherits_class('cve-check', d) else '0'}" -ne 0 ] ; then - bbwarn "Failed to update cve-check-tool database, CVEs won't be checked" - fi - fi -} - -addtask populate_cve_db after do_populate_sysroot -do_populate_cve_db[depends] = "cve-check-tool-native:do_populate_sysroot" -do_populate_cve_db[nostamp] = "1" -do_populate_cve_db[progress] = "percent" - -BBCLASSEXTEND = "native nativesdk" diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch b/meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch deleted file mode 100644 index 4a82cf2dde..0000000000 --- a/meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch +++ /dev/null @@ -1,50 +0,0 @@ -From a3353429652f83bb8b0316500faa88fa2555542d Mon Sep 17 00:00:00 2001 -From: Peter Marko -Date: Thu, 13 Apr 2017 23:09:52 +0200 -Subject: [PATCH] Fix freeing memory allocated by sqlite - -Upstream-Status: Backport -Signed-off-by: Peter Marko ---- - src/core.c | 8 ++++---- - 1 file changed, 4 insertions(+), 4 deletions(-) - -diff --git a/src/core.c b/src/core.c -index 6263031..6788f16 100644 ---- a/src/core.c -+++ b/src/core.c -@@ -82,7 +82,7 @@ static bool ensure_table(CveDB *self) - rc = sqlite3_exec(self->db, query, NULL, NULL, &err); - if (rc != SQLITE_OK) { - fprintf(stderr, "ensure_table(): %s\n", err); -- free(err); -+ sqlite3_free(err); - return false; - } - -@@ -91,7 +91,7 @@ static bool ensure_table(CveDB *self) - rc = sqlite3_exec(self->db, query, NULL, NULL, &err); - if (rc != SQLITE_OK) { - fprintf(stderr, "ensure_table(): %s\n", err); -- free(err); -+ sqlite3_free(err); - return false; - } - -@@ -99,11 +99,11 @@ static bool ensure_table(CveDB *self) - rc = sqlite3_exec(self->db, query, NULL, NULL, &err); - if (rc != SQLITE_OK) { - fprintf(stderr, "ensure_table(): %s\n", err); -- free(err); -+ sqlite3_free(err); - return false; - } - if (err) { -- free(err); -+ sqlite3_free(err); - } - - return true; --- -2.1.4 - diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch b/meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch deleted file mode 100644 index 3d8ebd1bd2..0000000000 --- a/meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch +++ /dev/null @@ -1,215 +0,0 @@ -From 825a9969dea052b02ba868bdf39e676349f10dce Mon Sep 17 00:00:00 2001 -From: Jussi Kukkonen -Date: Thu, 9 Feb 2017 14:51:28 +0200 -Subject: [PATCH] curl: allow overriding default CA certificate file - -Similar to curl, --cacert can now be used in cve-check-tool and -cve-check-update to override the default CA certificate file. Useful -in cases where the system default is unsuitable (for example, -out-dated) or broken (as in OE's current native libcurl, which embeds -a path string from one build host and then uses it on another although -the right path may have become something different). - -Upstream-Status: Submitted [https://github.com/ikeydoherty/cve-check-tool/pull/45] - -Signed-off-by: Patrick Ohly - - -Took Patrick Ohlys original patch from meta-security-isafw, rebased -on top of other patches. - -Signed-off-by: Jussi Kukkonen ---- - src/library/cve-check-tool.h | 1 + - src/library/fetch.c | 10 +++++++++- - src/library/fetch.h | 3 ++- - src/main.c | 5 ++++- - src/update-main.c | 4 +++- - src/update.c | 12 +++++++----- - src/update.h | 2 +- - 7 files changed, 27 insertions(+), 10 deletions(-) - -diff --git a/src/library/cve-check-tool.h b/src/library/cve-check-tool.h -index e4bb5b1..f89eade 100644 ---- a/src/library/cve-check-tool.h -+++ b/src/library/cve-check-tool.h -@@ -43,6 +43,7 @@ typedef struct CveCheckTool { - bool bugs; /**output_file = output_file; -+ self->cacert_file = cacert_file; - - if (!csv_mode && self->output_file) { - quiet = false; -@@ -530,7 +533,7 @@ int main(int argc, char **argv) - if (status) { - fprintf(stderr, "Update of db forced\n"); - cve_db_unlock(); -- if (!update_db(quiet, db_path->str)) { -+ if (!update_db(quiet, db_path->str, self->cacert_file)) { - fprintf(stderr, "DB update failure\n"); - goto cleanup; - } -diff --git a/src/update-main.c b/src/update-main.c -index 2379cfa..c52d9d0 100644 ---- a/src/update-main.c -+++ b/src/update-main.c -@@ -43,11 +43,13 @@ the Free Software Foundation; either version 2 of the License, or\n\ - static gchar *nvds = NULL; - static bool _show_version = false; - static bool _quiet = false; -+static const char *_cacert_file = NULL; - - static GOptionEntry _entries[] = { - { "nvd-dir", 'd', 0, G_OPTION_ARG_STRING, &nvds, "NVD directory in filesystem", NULL }, - { "version", 'v', 0, G_OPTION_ARG_NONE, &_show_version, "Show version", NULL }, - { "quiet", 'q', 0, G_OPTION_ARG_NONE, &_quiet, "Run silently", NULL }, -+ { "cacert", 'C', 0, G_OPTION_ARG_STRING, &_cacert_file, "Path to the combined SSL certificates file (system default is used if not set)", NULL}, - { .short_name = 0 } - }; - -@@ -88,7 +90,7 @@ int main(int argc, char **argv) - goto end; - } - -- if (update_db(_quiet, db_path->str)) { -+ if (update_db(_quiet, db_path->str, _cacert_file)) { - ret = EXIT_SUCCESS; - } else { - fprintf(stderr, "Failed to update database\n"); -diff --git a/src/update.c b/src/update.c -index 070560a..8cb4a39 100644 ---- a/src/update.c -+++ b/src/update.c -@@ -267,7 +267,8 @@ static inline void update_end(int fd, const char *update_fname, bool ok) - - static int do_fetch_update(int year, const char *db_dir, CveDB *cve_db, - bool db_exist, bool verbose, -- unsigned int this_percent, unsigned int next_percent) -+ unsigned int this_percent, unsigned int next_percent, -+ const char *cacert_file) - { - const char nvd_uri[] = URI_PREFIX; - autofree(cve_string) *uri_meta = NULL; -@@ -331,14 +332,14 @@ refetch: - } - - /* Fetch NVD META file */ -- st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose, this_percent, this_percent); -+ st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose, this_percent, this_percent, cacert_file); - if (st == FETCH_STATUS_FAIL) { - fprintf(stderr, "Failed to fetch %s\n", uri_meta->str); - return -1; - } - - /* Fetch NVD XML file */ -- st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose, this_percent, next_percent); -+ st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose, this_percent, next_percent, cacert_file); - switch (st) { - case FETCH_STATUS_FAIL: - fprintf(stderr, "Failed to fetch %s\n", uri_data_gz->str); -@@ -391,7 +392,7 @@ refetch: - return 0; - } - --bool update_db(bool quiet, const char *db_file) -+bool update_db(bool quiet, const char *db_file, const char *cacert_file) - { - autofree(char) *db_dir = NULL; - autofree(CveDB) *cve_db = NULL; -@@ -466,7 +467,8 @@ bool update_db(bool quiet, const char *db_file) - if (!quiet) - fprintf(stderr, "completed: %u%%\r", start_percent); - rc = do_fetch_update(y, db_dir, cve_db, db_exist, !quiet, -- start_percent, end_percent); -+ start_percent, end_percent, -+ cacert_file); - switch (rc) { - case 0: - if (!quiet) -diff --git a/src/update.h b/src/update.h -index b8e9911..ceea0c3 100644 ---- a/src/update.h -+++ b/src/update.h -@@ -15,7 +15,7 @@ cve_string *get_db_path(const char *path); - - int update_required(const char *db_file); - --bool update_db(bool quiet, const char *db_file); -+bool update_db(bool quiet, const char *db_file, const char *cacert_file); - - - /* --- -2.1.4 - diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch b/meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch deleted file mode 100644 index 8ea6f686e3..0000000000 --- a/meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch +++ /dev/null @@ -1,135 +0,0 @@ -From e9ed26cde63f8ca7607a010a518329339f8c02d3 Mon Sep 17 00:00:00 2001 -From: =?UTF-8?q?Andr=C3=A9=20Draszik?= -Date: Mon, 26 Sep 2016 12:12:41 +0100 -Subject: [PATCH] print progress in percent when downloading CVE db -MIME-Version: 1.0 -Content-Type: text/plain; charset=UTF-8 -Content-Transfer-Encoding: 8bit - -Upstream-Status: Pending -Signed-off-by: André Draszik ---- - src/library/fetch.c | 28 +++++++++++++++++++++++++++- - src/library/fetch.h | 3 ++- - src/update.c | 16 ++++++++++++---- - 3 files changed, 41 insertions(+), 6 deletions(-) - -diff --git a/src/library/fetch.c b/src/library/fetch.c -index 06d4b30..0fe6d76 100644 ---- a/src/library/fetch.c -+++ b/src/library/fetch.c -@@ -37,13 +37,37 @@ static size_t write_func(void *ptr, size_t size, size_t nmemb, struct fetch_t *f - return fwrite(ptr, size, nmemb, f->f); - } - --FetchStatus fetch_uri(const char *uri, const char *target, bool verbose) -+struct percent_t { -+ unsigned int start; -+ unsigned int end; -+}; -+ -+static int progress_callback_new(void *ptr, curl_off_t dltotal, curl_off_t dlnow, curl_off_t ultotal, curl_off_t ulnow) -+{ -+ (void) ultotal; -+ (void) ulnow; -+ -+ struct percent_t *percent = (struct percent_t *) ptr; -+ -+ if (dltotal && percent && percent->end >= percent->start) { -+ unsigned int diff = percent->end - percent->start; -+ if (diff) { -+ fprintf(stderr,"completed: %"CURL_FORMAT_CURL_OFF_T"%%\r", percent->start + (diff * dlnow / dltotal)); -+ } -+ } -+ -+ return 0; -+} -+ -+FetchStatus fetch_uri(const char *uri, const char *target, bool verbose, -+ unsigned int start_percent, unsigned int end_percent) - { - FetchStatus ret = FETCH_STATUS_FAIL; - CURLcode res; - struct stat st; - CURL *curl = NULL; - struct fetch_t *f = NULL; -+ struct percent_t percent = { .start = start_percent, .end = end_percent }; - - curl = curl_easy_init(); - if (!curl) { -@@ -67,6 +91,8 @@ FetchStatus fetch_uri(const char *uri, const char *target, bool verbose) - } - if (verbose) { - (void)curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L); -+ (void)curl_easy_setopt(curl, CURLOPT_XFERINFODATA, &percent); -+ (void)curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progress_callback_new); - } - res = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, (curl_write_callback)write_func); - if (res != CURLE_OK) { -diff --git a/src/library/fetch.h b/src/library/fetch.h -index 70c3779..4cce5d1 100644 ---- a/src/library/fetch.h -+++ b/src/library/fetch.h -@@ -28,7 +28,8 @@ typedef enum { - * @param verbose Whether to be verbose - * @return A FetchStatus, indicating the operation taken - */ --FetchStatus fetch_uri(const char *uri, const char *target, bool verbose); -+FetchStatus fetch_uri(const char *uri, const char *target, bool verbose, -+ unsigned int this_percent, unsigned int next_percent); - - /** - * Attempt to extract the given gzipped file -diff --git a/src/update.c b/src/update.c -index 30fbe96..eaeeefd 100644 ---- a/src/update.c -+++ b/src/update.c -@@ -266,7 +266,8 @@ static inline void update_end(int fd, const char *update_fname, bool ok) - } - - static int do_fetch_update(int year, const char *db_dir, CveDB *cve_db, -- bool db_exist, bool verbose) -+ bool db_exist, bool verbose, -+ unsigned int this_percent, unsigned int next_percent) - { - const char nvd_uri[] = URI_PREFIX; - autofree(cve_string) *uri_meta = NULL; -@@ -330,14 +331,14 @@ refetch: - } - - /* Fetch NVD META file */ -- st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose); -+ st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose, this_percent, this_percent); - if (st == FETCH_STATUS_FAIL) { - fprintf(stderr, "Failed to fetch %s\n", uri_meta->str); - return -1; - } - - /* Fetch NVD XML file */ -- st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose); -+ st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose, this_percent, next_percent); - switch (st) { - case FETCH_STATUS_FAIL: - fprintf(stderr, "Failed to fetch %s\n", uri_data_gz->str); -@@ -459,10 +460,17 @@ bool update_db(bool quiet, const char *db_file) - for (int i = YEAR_START; i <= year+1; i++) { - int y = i > year ? -1 : i; - int rc; -+ unsigned int start_percent = ((i+0 - YEAR_START) * 100) / (year+2 - YEAR_START); -+ unsigned int end_percent = ((i+1 - YEAR_START) * 100) / (year+2 - YEAR_START); - -- rc = do_fetch_update(y, db_dir, cve_db, db_exist, !quiet); -+ if (!quiet) -+ fprintf(stderr, "completed: %u%%\r", start_percent); -+ rc = do_fetch_update(y, db_dir, cve_db, db_exist, !quiet, -+ start_percent, end_percent); - switch (rc) { - case 0: -+ if (!quiet) -+ fprintf(stderr,"completed: %u%%\r", end_percent); - continue; - case ENOMEM: - goto oom; --- -2.9.3 - diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch b/meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch deleted file mode 100644 index 458c0cc84e..0000000000 --- a/meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch +++ /dev/null @@ -1,52 +0,0 @@ -From b0426e63c9ac61657e029f689bcb8dd051e752c6 Mon Sep 17 00:00:00 2001 -From: Sergey Popovich -Date: Fri, 21 Apr 2017 07:32:23 -0700 -Subject: [PATCH] update: Compare computed vs expected sha256 digit string - ignoring case - -We produce sha256 digest string using %x snprintf() -qualifier for each byte of digest which uses alphabetic -characters from "a" to "f" in lower case to represent -integer values from 10 to 15. - -Previously all of the NVD META files supply sha256 -digest string for corresponding XML file in lower case. - -However due to some reason this changed recently to -provide digest digits in upper case causing fetched -data consistency checks to fail. This prevents database -from being updated periodically. - -While commit c4f6e94 (update: Do not treat sha256 failure -as fatal if requested) adds useful option to skip -digest validation at all and thus provides workaround for -this situation, it might be unacceptable for some -deployments where we need to ensure that downloaded -data is consistent before start parsing it and update -SQLite database. - -Use strcasecmp() to compare two digest strings case -insensitively and addressing this case. - -Upstream-Status: Backport -Signed-off-by: Sergey Popovich ---- - src/update.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -diff --git a/src/update.c b/src/update.c -index 8588f38..3cc6b67 100644 ---- a/src/update.c -+++ b/src/update.c -@@ -187,7 +187,7 @@ static bool nvdcve_data_ok(const char *meta, const char *data) - snprintf(&csum_data[idx], len, "%02hhx", digest[i]); - } - -- ret = streq(csum_meta, csum_data); -+ ret = !strcasecmp(csum_meta, csum_data); - - err_unmap: - munmap(buffer, length); --- -2.11.0 - diff --git a/meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch b/meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch deleted file mode 100644 index 0774ad946a..0000000000 --- a/meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch +++ /dev/null @@ -1,51 +0,0 @@ -From ce64633b9733e962b8d8482244301f614d8b5845 Mon Sep 17 00:00:00 2001 -From: Khem Raj -Date: Mon, 22 Aug 2016 22:54:24 -0700 -Subject: [PATCH] Check for malloc_trim before using it - -malloc_trim is gnu specific and not all libc -implement it, threfore write a configure check -to poke for it first and use the define to -guard its use. - -Helps in compiling on musl based systems - -Signed-off-by: Khem Raj ---- -Upstream-Status: Submitted [https://github.com/ikeydoherty/cve-check-tool/pull/48] - configure.ac | 2 ++ - src/core.c | 4 ++-- - 2 files changed, 4 insertions(+), 2 deletions(-) - -diff --git a/configure.ac b/configure.ac -index d3b66ce..79c3542 100644 ---- a/configure.ac -+++ b/configure.ac -@@ -19,6 +19,8 @@ m4_define([json_required_version], [0.16.0]) - m4_define([openssl_required_version],[1.0.0]) - # TODO: Set minimum sqlite - -+AC_CHECK_FUNCS_ONCE(malloc_trim) -+ - PKG_CHECK_MODULES(CVE_CHECK_TOOL, - [ - glib-2.0 >= glib_required_version, -diff --git a/src/core.c b/src/core.c -index 6263031..0d5df29 100644 ---- a/src/core.c -+++ b/src/core.c -@@ -498,9 +498,9 @@ bool cve_db_load(CveDB *self, const char *fname) - } - - b = true; -- -+#ifdef HAVE_MALLOC_TRIM - malloc_trim(0); -- -+#endif - xmlFreeTextReader(r); - if (fd) { - close(fd); --- -2.9.3 - From patchwork Sun Dec 8 18:35:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180963 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3447774ile; Sun, 8 Dec 2019 10:38:24 -0800 (PST) X-Google-Smtp-Source: APXvYqxPR91IrgawfJ0HOEyVkxNGPM6lz3OrRUQxkzYYhtc8b/mwy0Gy58XF9WGnIyOqoKeWbPEm X-Received: by 2002:a17:902:b68c:: with SMTP id c12mr25302722pls.126.1575830304037; Sun, 08 Dec 2019 10:38:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830304; cv=none; d=google.com; s=arc-20160816; b=gXk2HwGLt65m7jZnhfZQ+xsiJsqIdbspsfeRGRV5T/8rkjHzTYkEV5DyTla6lnUX3z Ucn3lBtGX9L6IXuMTgpTwlhRplgG3812GXzLyAZpyb698JgXtXH+UynHOzwt5isXLq4z v2hBjmXVWj+d5pKpPLw5BLQ3q98qxl5O48Zz2DsXrguwVOo35SoVng1nx9AlIsrAWV2r OGcu8AHiXuX8+LAPiOv2L52YSXCSRAs41X69t1fUAzOJrz/YegjvGNa7DsWWLZ65EMsV b9fC2Eko/T9T/ehZSYH234WERBH++QRBpOGrv/Lk9GSwo9fwcHJ7V6axXddyr3hf/okH EXbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=qUd8oe/jMeHK0JiI7jNHX4DLPQMHnnqbFAWBXbED52w=; b=crHZG9dSuEiq5wx/1gmX1TsXCSUZiVa3IBZNLs1P7KAWgECcHV2sJdormBbx15Z7em 4XQ/cqyilB/wdIroD/xWH4PBmjBwmMoALxulPSgHXR+4CtBiswKJcoi8vcX8V3juc58j x3OIuVs5Q3t/4SuT5EbcB/oECWJ7ZY2XKsaKLo4PsPTYQCzd5f+KOyIRz8ZODz6QLz61 jd2w6dWk8/WXS+N4UPCBZSO4AvCOkhWnFCua2UVnM4WU6fH45OaXUxsMxEIRAEZJ4Fi6 Ak9eUzhke7YE//FymhJLep03/oOTfW/hNHSgsJIbtRE9h4Asvv/zGh1jmGoO/1gTCqnk 4wgQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id 21si978621pgn.257.2019.12.08.10.38.23; Sun, 08 Dec 2019 10:38:24 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id B7F757FBA6; Sun, 8 Dec 2019 18:38:21 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from sinikuusama2.dnainternet.net (sinikuusama2.dnainternet.net [83.102.40.152]) by mail.openembedded.org (Postfix) with ESMTP id 0A99A7F9B4 for ; Sun, 8 Dec 2019 18:38:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sinikuusama2.dnainternet.net (Postfix) with ESMTP id 2A7E5F9A5; Sun, 8 Dec 2019 20:38:20 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from sinikuusama2.dnainternet.net ([83.102.40.152]) by localhost (sinikuusama2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id mwIzW070HLRb; Sun, 8 Dec 2019 20:38:19 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by sinikuusama2.dnainternet.net (Postfix) with ESMTP id CD0E1F8A1; Sun, 8 Dec 2019 20:38:19 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id 01D802C63; Sun, 8 Dec 2019 20:38:14 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:48 +0200 Message-Id: <20191208183557.32589-2-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 02/11] cve-check: ensure all known CVEs are in the report X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton CVEs that are whitelisted or were not vulnerable when there are version comparisons were not included in the report, so alter the logic to ensure that all relevant CVEs are in the report for completeness. (From OE-Core rev: 98256ff05fcfe9d5ccad360582c36eafb577c264) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/classes/cve-check.bbclass | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index c00d2910be..f87bcc9dc6 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -208,12 +208,14 @@ def check_cves(d, patched_cves): if cve in cve_whitelist: bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) + # TODO: this should be in the report as 'whitelisted' + patched_cves.add(cve) elif cve in patched_cves: bb.note("%s has been patched" % (cve)) else: to_append = False if (operator_start == '=' and pv == version_start): - cves_unpatched.append(cve) + to_append = True else: if operator_start: try: @@ -243,8 +245,11 @@ def check_cves(d, patched_cves): to_append = to_append_start or to_append_end if to_append: + bb.note("%s-%s is vulnerable to %s" % (product, pv, cve)) cves_unpatched.append(cve) - bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve)) + else: + bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) + patched_cves.add(cve) conn.close() return (list(patched_cves), cves_unpatched) From patchwork Sun Dec 8 18:35:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180964 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3448332ile; Sun, 8 Dec 2019 10:39:10 -0800 (PST) X-Google-Smtp-Source: APXvYqyF3v3is/SXC77FNeQug1wwxY97IlxOqGQpzrd5XOZPwOdDdoNP3RxWMC1AM3QvcNXaTwaj X-Received: by 2002:a17:902:8491:: with SMTP id c17mr24177703plo.143.1575830349862; Sun, 08 Dec 2019 10:39:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830349; cv=none; d=google.com; s=arc-20160816; b=Ix3LxfS643NS/B7joQAR00a9lmzpZLhEmc49NmjcuOEQDXCxso+W20FEFBVwhd1AhU Fd48R0oBsRErIEcHG55GHlnEOYCNqkGQwCbe/iy/tVH/UdDD3WzvqO+8cur6TveL+2ax SBn3DNIp4bkcGmiLi1mklkjPoomz9C4hKHsDJRpq/3Ezqn/monW8kwpvxsomQ6FN0xdo O96M8K3xVZc0iZH7IkgfkIajLeJHRcPQosNXuqc4CNeWROWDfqnG6h4n7SpersCzOyUj XkUHqeXskHDqR6cEgGtR0ICHwRLZprLDAjfPT0XMi5C9p7YVoSCwl5L6BhOjbieJErj5 jmkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=gXhcLsHhubAVpMUQ90kv6Hjuc7Bg7S0Fz+MLs9jUjSU=; b=ALrMij8e6T7bz2h+tuJMxpVG39MSqLJmhfD6ZNMjgm5MHXkt8pUkx/3+0ml75CmHLb Y804kJaDV90jFa2WZujRYstqxOI4h48Nn+ZllRXo5pu2ZXUbh236jzE3Re1DFVckqQpu vK30seuc10UE3QcgJRd58O3jT2+tXeX+zPkbkz6alY2oyJ+aIP6CAHD2P1xOHUxs3UZH cdLahHVrtMUdyPexprmC5bPtQ4oRsEMmB+iWt5qsHgrrQSihLFlSMLr+fB6NVjUwH7GT n1Tgn0hQBpvA54sL7hjHs6UvbeGDsepY5Bcj7V/AL58KBV/2F69VOnbHTv+g5dN105n+ HwBQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id n5si15148824plk.143.2019.12.08.10.39.09; Sun, 08 Dec 2019 10:39:09 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id BDC657FBA6; Sun, 8 Dec 2019 18:39:07 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from sypressi2.dnainternet.net (sypressi2.dnainternet.net [83.102.40.154]) by mail.openembedded.org (Postfix) with ESMTP id 82BAF7F9B4 for ; Sun, 8 Dec 2019 18:39:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sypressi2.dnainternet.net (Postfix) with ESMTP id BDD801521C; Sun, 8 Dec 2019 20:39:06 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from sypressi2.dnainternet.net ([83.102.40.154]) by localhost (sypressi2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id RkP3UPUMV_QH; Sun, 8 Dec 2019 20:39:06 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by sypressi2.dnainternet.net (Postfix) with ESMTP id 475A7150DD; Sun, 8 Dec 2019 20:39:06 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id 78A2D2C61; Sun, 8 Dec 2019 20:39:01 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:49 +0200 Message-Id: <20191208183557.32589-3-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 03/11] cve-check: failure to parse versions should be more visible X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Signed-off-by: Richard Purdie --- meta/classes/cve-check.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index f87bcc9dc6..1c8b2223a2 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -222,7 +222,7 @@ def check_cves(d, patched_cves): to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) except: - bb.note("%s: Failed to compare %s %s %s for %s" % + bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_start, version_start, cve)) to_append_start = False else: @@ -233,7 +233,7 @@ def check_cves(d, patched_cves): to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) except: - bb.note("%s: Failed to compare %s %s %s for %s" % + bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_end, version_end, cve)) to_append_end = False else: From patchwork Sun Dec 8 18:35:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180965 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3448641ile; Sun, 8 Dec 2019 10:39:38 -0800 (PST) X-Google-Smtp-Source: APXvYqypFH3BUvQMFhiimOEE4o9CSFN0F2Dn1aHXLNYmKHdnepHfB4rlfAbLGolJtX+f2rqn1ozz X-Received: by 2002:a17:902:760c:: with SMTP id k12mr25413782pll.33.1575830378391; Sun, 08 Dec 2019 10:39:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830378; cv=none; d=google.com; s=arc-20160816; b=YIU1iBLYS+5hgYUMJAGcpE2udnP/4RY9M3NZciHE1Z+5Q4h8SS6ULcJEHLFoik1Y+l CeNuXIKDLycStrlz5RxSpmu+A3DiE5dk6kF9G/CzqdH/MbhyN20q0LfoW3o6S2H/5HgV puT5VILzfiar1HYbZPRzdXKKVNZ7enRZr0m3JGnGcjQ2zB4GaPbcl58LXpabHbZ/VCPq pKItonH3tyyRi0AYdljJP5IG1qStJGArj1MACvzHj/v38dAGsciHIvTLvB8qA6wfDzIr CYwgxwzE84hz9GApPtSAZc3MLDVJgu0DGOLoY2Z4EIfFX5eeS2+X9gOOQQK87vPTFd4g lBdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=A/tElk01rCVi1XhnTUVW+mT51iw1JZCQ8GwHYOKKeR4=; b=NYpXsgiQ7PhL18USCBMb7B11sLDVct3ARZIcecF0+ZXsKvDOWxbwo2bGKaXIsAvLZo WKGEiefZII/3CKUCZDGBWNfBUXcc+Zp4Wo/NclbgloCFrNpOp+qLlaF0R8UQ2H4mr5QT YuBtKHKGi+1qSP2cr4Uv2tmtwoiDR3RpWEMY+Y7qD83FQ8B2tD5q8ev2qNDM19uG8HZS E/xKW3HV4GY8o78hk1jD31sK8AOhYfS4+J25SM5C9YRh71OegA6qfM/63hr+8A5MRT2u sfl4WBuN14nHaHklQgbxldVocmqT6U8Tg5yq7B0vntHYdHqNxBECiEvmM+U/6BSrZRpB Zwaw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id a14si19139922pfr.142.2019.12.08.10.39.38; Sun, 08 Dec 2019 10:39:38 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 3F2D57FBA6; Sun, 8 Dec 2019 18:39:36 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from sypressi2.dnainternet.net (sypressi2.dnainternet.net [83.102.40.154]) by mail.openembedded.org (Postfix) with ESMTP id 613207FBA6 for ; Sun, 8 Dec 2019 18:39:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sypressi2.dnainternet.net (Postfix) with ESMTP id 3352015269; Sun, 8 Dec 2019 20:39:36 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from sypressi2.dnainternet.net ([83.102.40.154]) by localhost (sypressi2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id P_er6RjLi08M; Sun, 8 Dec 2019 20:39:35 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by sypressi2.dnainternet.net (Postfix) with ESMTP id BD0E0150FC; Sun, 8 Dec 2019 20:39:35 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id D95DB79; Sun, 8 Dec 2019 20:39:30 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:50 +0200 Message-Id: <20191208183557.32589-4-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 04/11] cve-check: we don't actually need to unpack to check X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton The patch scanner works with patch files in the layer, not in the workdir, so it doesn't need to unpack. (From OE-Core rev: 2cba6ada970deb5156e1ba0182f4f372851e3c17) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/classes/cve-check.bbclass | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 1c8b2223a2..3326944d79 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -62,7 +62,7 @@ python do_cve_check () { } -addtask cve_check after do_unpack before do_build +addtask cve_check before do_build do_cve_check[depends] = "cve-update-db-native:do_populate_cve_db" do_cve_check[nostamp] = "1" @@ -70,7 +70,6 @@ python cve_check_cleanup () { """ Delete the file used to gather all the CVE information. """ - bb.utils.remove(e.data.getVar("CVE_CHECK_TMP_FILE")) } From patchwork Sun Dec 8 18:35:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180966 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3449050ile; Sun, 8 Dec 2019 10:40:19 -0800 (PST) X-Google-Smtp-Source: APXvYqyLqoi+NjsxJeRGzijeOo5T4L6BhO285ip1W8rrjqNoo/jQfby9Nse37DfOogJ2TvnEGGqi X-Received: by 2002:a17:902:aa92:: with SMTP id d18mr25352373plr.157.1575830418664; Sun, 08 Dec 2019 10:40:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830418; cv=none; d=google.com; s=arc-20160816; b=Tb7BTGF3+QXNSL1UHjmtgZRb3lCtFaQT8QRx8vFTXF1gsDDoT3U1Wz1afUYHvgVKUd FIO8gmtKSntUavxNsJ4bGXah0PKza1aLuxWe4WU2IbnglmDVhVf4As1vWt6NXxWDvi7f NSPwKqjBtFUtahQdreSkegzt5Gu/9LBnzX1zt37OhDl+7LV+tfM1BOCC/yXS4F1bK7Lg V4hRmTfIdoe7YKFWF1ZFn+0lMTBHjsm1Skqe1L5RN10SoQWMYkW3gzbQQi351SL845DB mKk+V/xdOk7ZOtNHrNQ2/FVUhwE/4AmUWqGdDGjlZ9OSJAtgGL1gl9s85jN0kSLGaWkY Rxhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=wdB5DBxhAqI+E8RHKOCEkNv2RklrDglJ/PHEAsSjbUg=; b=q1HraVeAkRrtLc+GYP7bvJgPxP1m7JIKYxTwc5HTA2kjA/SgqOrcIwOS0BMKK0Uu6G ZQzb254Ari6Dne+6Zxng12VI/1DjV7msRbWWWXjoV3qlAheH501OPwCkWX3FXKLhD7lL +zAjipI2xqeIrdIpJtv6mwuzeE+Ts8j2tvtGR0QAq4W9Ww/a1psvhRrU1q0wUkv+FLoQ V0IsMuaMAaRdn5eS2Y64v1YxDlC69cwyaxLO8coS+oLZdHkiplIqH4isGUU94x/rEbF/ AzqEgjGfhtCwXNueVrDGURlXty7/bsDmjLUhf3N/2hpa93OD+vw69FL7T8j/lT3fdggO an3Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id u127si16979705pgu.147.2019.12.08.10.40.17; Sun, 08 Dec 2019 10:40:18 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 1F22E7FBAC; Sun, 8 Dec 2019 18:40:16 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from tulikuusama2.dnainternet.net (tulikuusama3.dnainternet.net [83.102.40.155]) by mail.openembedded.org (Postfix) with ESMTP id C40157F9B4 for ; Sun, 8 Dec 2019 18:40:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id 9C9D7291B3; Sun, 8 Dec 2019 20:40:15 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from tulikuusama2.dnainternet.net ([83.102.40.155]) by localhost (tulikuusama2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id UVgJmpui5RHG; Sun, 8 Dec 2019 20:40:15 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id 24185293FA; Sun, 8 Dec 2019 20:40:15 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id 49C4E74; Sun, 8 Dec 2019 20:40:09 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:51 +0200 Message-Id: <20191208183557.32589-5-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 05/11] cve-update-db-native: don't refresh more than once an hour X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton We already fetch the yearly CVE metadata and check that for updates before downloading the full data, but we can speed up CVE checking further by only checking the CVE metadata once an hour. (From OE-Core rev: 50d898fd360c58fe85460517d965f62b7654771a) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/recipes-core/meta/cve-update-db-native.bb | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 2c427a5884..19875a49b1 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -31,8 +31,16 @@ python do_populate_cve_db() { db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK') db_file = os.path.join(db_dir, 'nvdcve_1.0.db') json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') - proxy = d.getVar("https_proxy") + # Don't refresh the database more than once an hour + try: + import time + if time.time() - os.path.getmtime(db_file) < (60*60): + return + except OSError: + pass + + proxy = d.getVar("https_proxy") if proxy: # instantiate an opener but do not install it as the global # opener unless if we're really sure it's applicable for all From patchwork Sun Dec 8 18:35:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180967 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3449214ile; Sun, 8 Dec 2019 10:40:34 -0800 (PST) X-Google-Smtp-Source: APXvYqyrCtwHsIvUDIPkAP6qMHhld1+rwUemZZHjAO/v70+p7u9PhWoyiPpqL/0btkpxdIGNy72K X-Received: by 2002:a63:774a:: with SMTP id s71mr14941000pgc.57.1575830432420; Sun, 08 Dec 2019 10:40:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830432; cv=none; d=google.com; s=arc-20160816; b=ybDJB8O9Pa6QPri73wcQ+pJk6FYDLRp557USJ+nkVdsKZNPPD1aN7zyvkbEvFEZE/s Ft0YbYEnBIlO5EkVYaSPmoK01JlllMbzvukGIVxgCVNNchDkwhUyi6akUL4wI4QHkXDD oRfky7jstVF2pKBPVoKoW4Gg8AyYRmXwmmmVGEWg7bBKIZRqSjF887MRVGxuPdSuxmFs vrLNaaHHmxjWYPSwu3DZ7bGZtu2LIN/IyD6o4MPRa+TbpvcD7L9qBf13KlovOfBYfuoG +NRjXrmLoEMlDlho6HXVgWMrc9cX/DUZEbS5UYKFOMD6DG/I/Sg5I+Ehjhu7nHrduiGi 1h/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=r4X8fai96My1vBJsaeCn0s1DImJGpD0YxuKSwth7wKs=; b=GquNseXgWfKa1B/koAQxZQ9piKb4RFI1z7Rg/b3zR3OPgewu3FWNcUbpd5uCZx9mEq 4sqSOUdr3FVsEwkW3RJjIAARaCfHNN/kZJv+gR2cHSOm+UabNoHGTv/p0b6HSOR6Stge JplRM4QeN2XNyvmSqAzUn84hZ2i4T2S1i8Z7bvy2eIRntv+e5UucpK+8vo6/ddeQwFhj gnBeX/xWwdciFOZIJCCwJOElST9MSWQ2WrLl8fY5JXXjRt3MtrYtFb8SD/Zx4yjbkXjz 4U2PkSlx9uqsUzCd7AmaHL6bIshyKY8KcNvawL7CnU70vXt+84e1yEZ8efan691XF74l +QzA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id h7si15551809pls.28.2019.12.08.10.40.31; Sun, 08 Dec 2019 10:40:32 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 1C7E47FBBD; Sun, 8 Dec 2019 18:40:30 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from sinikuusama2.dnainternet.net (sinikuusama2.dnainternet.net [83.102.40.152]) by mail.openembedded.org (Postfix) with ESMTP id AB61B7F9DE for ; Sun, 8 Dec 2019 18:40:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sinikuusama2.dnainternet.net (Postfix) with ESMTP id 86C5CF9A5; Sun, 8 Dec 2019 20:40:29 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from sinikuusama2.dnainternet.net ([83.102.40.152]) by localhost (sinikuusama2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id nDg_jOZyMKT5; Sun, 8 Dec 2019 20:40:29 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by sinikuusama2.dnainternet.net (Postfix) with ESMTP id 3346810992; Sun, 8 Dec 2019 20:40:29 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id 5613B74; Sun, 8 Dec 2019 20:40:24 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:52 +0200 Message-Id: <20191208183557.32589-6-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 06/11] cve-update-db-native: don't hardcode the database name X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Don't hardcode the database filename, there's a variable for this in cve-check.bbclass. (From OE-Core rev: 0d188a9dc4ae64c64cd661e9d9c3841e86f226ab) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/recipes-core/meta/cve-update-db-native.bb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 19875a49b1..c15534de08 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -28,8 +28,8 @@ python do_populate_cve_db() { BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 - db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK') - db_file = os.path.join(db_dir, 'nvdcve_1.0.db') + db_file = d.getVar("CVE_CHECK_DB_FILE") + db_dir = os.path.dirname(db_file) json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') # Don't refresh the database more than once an hour From patchwork Sun Dec 8 18:35:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180968 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3449522ile; Sun, 8 Dec 2019 10:41:05 -0800 (PST) X-Google-Smtp-Source: APXvYqzmIvH78ekM2P1SCev9BefwsLhwtb2eXJPFIp9ERIo3MfluSAnWaYoE0XwbwAXyGSLJjMvS X-Received: by 2002:a17:90a:77c5:: with SMTP id e5mr10227789pjs.83.1575830465214; Sun, 08 Dec 2019 10:41:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830465; cv=none; d=google.com; s=arc-20160816; b=AOwfDE43jW2sHpoSsk3CjQNcEOsBiSumILgoZmGV/3/CAnf/YINLOLTBRwY9E6MUbL ONzmQSgiUsNvZxl5dqaCS0IeQDkszOlIsTjWay1b916yGRSXwFraNVhYygOhHUO0AQuN iFi78M3Kjge3DIJhJjFZ5zyVUjP9UyVwcTbKuPXf0Q963bKyRlDXK2mwfJ+RCpdgCxRM T3Ke1XfJed8nMsEJeBxmjCv9eB6tXyWCj6q4p6uyQ2MpfPq3ea8N8Kzzd9I0QC7yQ9l2 vycvmwiVp7Ru/JrYoLQ81LyWi25VA0lpUFmBWLnU4gJ/c22EewdggyYe3Ro35UAzaExe 1Fog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=WlRYfH2Hf5QFw6TWVtPe5W5TC0Zt+tqkeWgD2JQTuBI=; b=r0XcVSu/cp7ebzvm+vQCVwtMSSxeAJgYbKvciTNBt7I74ntIooWjXLo4P57FGT8vwO 4X4HTUd4BEzbtjowkUEeUyrV3oE/I/YOWfPvf2EBFSPpKxuuRVQJoueqVT1/8cdukjcs RDLFvhiVjbWbc1vmw3AWekdT3DddcM9sNR1VTzSw2osmdl2+2SHpS2f+obIp7L68R9oy yBIsg1HXZBT0bymY4F3A7Wq2HgPX3YseH44jSDGDUZw9cmKHPuF4IO2irhsoieZIbiB5 wfgcX+r+xu9zir177VVjSAI5uHQK9FH/zuTI23kx9TfFr+SZm1jtX9pQc4JPMNNNIXgp 0RGA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id m4si12924440pga.92.2019.12.08.10.41.04; Sun, 08 Dec 2019 10:41:05 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 27FD27FACA; Sun, 8 Dec 2019 18:41:02 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from sypressi2.dnainternet.net (sypressi2.dnainternet.net [83.102.40.154]) by mail.openembedded.org (Postfix) with ESMTP id 313E07F7B3 for ; Sun, 8 Dec 2019 18:40:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sypressi2.dnainternet.net (Postfix) with ESMTP id DFD31149A7; Sun, 8 Dec 2019 20:40:43 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from sypressi2.dnainternet.net ([83.102.40.154]) by localhost (sypressi2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id L64765v49E4D; Sun, 8 Dec 2019 20:40:43 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by sypressi2.dnainternet.net (Postfix) with ESMTP id 7D55114E96; Sun, 8 Dec 2019 20:40:43 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id A1D5D74; Sun, 8 Dec 2019 20:40:38 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:53 +0200 Message-Id: <20191208183557.32589-7-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 07/11] cve-update-db-native: add an index on the CVE ID column X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Create an index on the PRODUCTS table which contains a row for each CPE, drastically increasing the performance of lookups for a specific CVE. (From OE-Core rev: b4048b05b3a00d85c40d09961f846eadcebd812e) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/recipes-core/meta/cve-update-db-native.bb | 3 +++ 1 file changed, 3 insertions(+) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index c15534de08..08b18f064f 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -120,11 +120,14 @@ python do_populate_cve_db() { def initialize_db(c): c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \ VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ VERSION_END TEXT, OPERATOR_END TEXT)") + c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_ID_IDX on PRODUCTS(ID);") def parse_node_and_insert(c, node, cveId): # Parse children node if needed From patchwork Sun Dec 8 18:35:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180969 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3449574ile; Sun, 8 Dec 2019 10:41:10 -0800 (PST) X-Google-Smtp-Source: APXvYqwBF5gUpIADfIutWhdS1HFIlLj3FiZRmE38mgL8zh1tnqMpRQtAcLUE64ZxnLDKr6BEaqxP X-Received: by 2002:a65:66c4:: with SMTP id c4mr15063500pgw.429.1575830469151; Sun, 08 Dec 2019 10:41:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830469; cv=none; d=google.com; s=arc-20160816; b=x00xV9IJpTogBMz9ISRTK1BRsOmfhGP1csCnDNysGhXrQgZBJ1b1ndly79puXjO/Cs Bv1928mFqs1vodzP/OipdvihG3y7ew0ZwucJBN13juN4VpZbj2QbTZ+VVyaotLOrXDDx c4Us34/JrBRL/BcXLyDsUwaTQ8MznM1zv2PWLuTZ5LtHhh1j0g3cI61LVA55Zw8OdMmp KLzNtEDmjjlVwxtnQkfZ90sdWFmHJzkWHwKkA1xkUZl7+NPkkVVCIbVLYjasPMCQAIrH ba0mJ9CPOQNIGp6CqF/eKTQ0X2eDvW29wVSvRKnNzEQ2fFDN/eNoKuYjw3EO+xwKxRf5 RpBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=gdFzUZnwnF2tDHt2qJLlHMIzqxHuCCwATvUeR9WiAdc=; b=iaL611ONrM12oO7qHOxSFR+3yaWs0DGgiraSVzA1wDHGcWrdIEKcsGvkH0Da+UNzCV m9Di2K+Jrqac5X6X4AKlqm4Y2MnDRQFObD0wmpa9NxdBCoSS+eM0oHINo2ulRYun6Wr2 tu00mPrHehOzaCXAU82gjJ5CF7jzOYQrPppjEwaUwGovQC9DPvi9ubI0iW8QZDhOsctv l/qx2PYL/Dq0i0VRskS8yxm3htPD7aPLnwCWaFZCPkafXOTdU7tFo1ZEJYkWcGkKd9UR htraGA4OwxKpAiHl+UkB6KqXFesINOGQ4Debh26B1XxvbainC1tDTKKrvdt+1YSakkWA SI1Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id d9si16789059plj.318.2019.12.08.10.41.08; Sun, 08 Dec 2019 10:41:09 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 055187FBE2; Sun, 8 Dec 2019 18:41:04 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from sypressi2.dnainternet.net (sypressi2.dnainternet.net [83.102.40.154]) by mail.openembedded.org (Postfix) with ESMTP id DE9557F849 for ; Sun, 8 Dec 2019 18:40:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sypressi2.dnainternet.net (Postfix) with ESMTP id 9C3EF14E51; Sun, 8 Dec 2019 20:40:56 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from sypressi2.dnainternet.net ([83.102.40.154]) by localhost (sypressi2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id Pu3TWmUkL8zn; Sun, 8 Dec 2019 20:40:56 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by sypressi2.dnainternet.net (Postfix) with ESMTP id 11144149A7; Sun, 8 Dec 2019 20:40:56 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id 1A15F89; Sun, 8 Dec 2019 20:40:50 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:54 +0200 Message-Id: <20191208183557.32589-8-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 08/11] cve-update-db-native: clean up proxy handling X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton urllib handles adding proxy handlers if the proxies are set in the environment, so call bb.utils.export_proxies() to do that and remove the manual setup. (From OE-Core rev: 6b73004668b3b71c9c38814b79fbb58c893ed434) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- .../recipes-core/meta/cve-update-db-native.bb | 31 +++---------------- 1 file changed, 5 insertions(+), 26 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 08b18f064f..db1d69a28e 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -21,10 +21,12 @@ python do_populate_cve_db() { """ Update NVD database with json data feed """ - + import bb.utils import sqlite3, urllib, urllib.parse, shutil, gzip from datetime import date + bb.utils.export_proxies(d) + BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 @@ -40,16 +42,6 @@ python do_populate_cve_db() { except OSError: pass - proxy = d.getVar("https_proxy") - if proxy: - # instantiate an opener but do not install it as the global - # opener unless if we're really sure it's applicable for all - # urllib requests - proxy_handler = urllib.request.ProxyHandler({'https': proxy}) - proxy_opener = urllib.request.build_opener(proxy_handler) - else: - proxy_opener = None - cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') if not os.path.isdir(db_dir): @@ -67,15 +59,7 @@ python do_populate_cve_db() { json_url = year_url + ".json.gz" # Retrieve meta last modified date - - response = None - - if proxy_opener: - response = proxy_opener.open(meta_url) - else: - req = urllib.request.Request(meta_url) - response = urllib.request.urlopen(req) - + response = urllib.request.urlopen(meta_url) if response: for l in response.read().decode("utf-8").splitlines(): key, value = l.split(":", 1) @@ -95,12 +79,7 @@ python do_populate_cve_db() { # Update db with current year json file try: - if proxy_opener: - response = proxy_opener.open(json_url) - else: - req = urllib.request.Request(json_url) - response = urllib.request.urlopen(req) - + response = urllib.request.urlopen(json_url) if response: update_db(c, gzip.decompress(response.read()).decode('utf-8')) c.execute("insert or replace into META values (?, ?)", [year, last_modified]) From patchwork Sun Dec 8 18:35:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180972 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3451042ile; Sun, 8 Dec 2019 10:42:57 -0800 (PST) X-Google-Smtp-Source: APXvYqzRaZFShWu9OiKkukHz8B7ak/uKEbK7TdlyxCftoTDTqAmzys+XjdWEwlcF2XA2lxMrBcqC X-Received: by 2002:a17:902:9686:: with SMTP id n6mr25087286plp.249.1575830475858; Sun, 08 Dec 2019 10:41:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830475; cv=none; d=google.com; s=arc-20160816; b=U2aoRnyqvT3E+F8HtLJESRaMl+vr7dfW858Xq22v/+Ajo3zocUCNZJVXvGyGCUAY4J 8aNOax5ei0zK8BGEt1813DdSO9UZKre28cR7qrLta8v1VaBOqgrAzFwkuunmLm+U9bXk d/JZbwELle+sOGBoBvKUTNKryTwDFXsRNZwhMS0QI+H8X3JfrBd5LSFexlegAfnSLNui Gc4W0g55i8X9VCTrSSK/Vt4GXMRkHVwL0U2NzA9Q2uNih1wAmJpRSY+SMThVF+C4mZGo +zlm0GOXHDdTs5eokjyWU8Xs6FusHovrVvg+oLCGuLYvgui/uLpDCaJKSTe8sGsStgLX teKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=oZYRaZxmp9UfZJlBtOoIlvF9CeDTcZAMAIn7RGqno0E=; b=V2s2KEGACppafO4OGxZhMYKsAJQFajIDT6oWG48Rj5aEZogiuogKMwnFm4mj+7LaYw uTK7NiEbgOwN0MU5xPKmAPwI3uJG9EvmAotCqNr1f98/nBcwMPDVOFxxE8UDX+R1H28N m/yjgee24fkiZ1maqMCTbfb4Ug3RK0OVzcLbEggnMa0rPi1EncXGEIZpSj37akM2eCz6 CR2bTdCL7KNei+V5xBvYzHDLRF/0rPkS7EtSOuQOhyIkRW4jKRBRxAodVgieUs+4FYtu obaPhXuu/ZySserYU+OnQLi5ecWCQWjmFIUqqBDBEL3WhgIp9UR0zNQgO65BIDcDoMxn Q7Jg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id a25si14635238pgw.287.2019.12.08.10.41.15; Sun, 08 Dec 2019 10:41:15 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 485287F9DE; Sun, 8 Dec 2019 18:41:13 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from tulikuusama2.dnainternet.net (tulikuusama3.dnainternet.net [83.102.40.155]) by mail.openembedded.org (Postfix) with ESMTP id E095A7F33D for ; Sun, 8 Dec 2019 18:41:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id B579F293A2; Sun, 8 Dec 2019 20:41:10 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from tulikuusama2.dnainternet.net ([83.102.40.155]) by localhost (tulikuusama2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id Mkzu2FuY0r5G; Sun, 8 Dec 2019 20:41:10 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id 2778829349; Sun, 8 Dec 2019 20:41:10 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id 4878F89; Sun, 8 Dec 2019 20:41:04 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:55 +0200 Message-Id: <20191208183557.32589-9-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 09/11] cve-check: rewrite look to fix false negatives X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton A previous optimisation was premature and resulted in false-negatives in the report. Rewrite the checking algorithm to first get the list of potential CVEs by vendor:product, then iterate through every matching CPE for that CVE to determine if the bounds match or not. By doing this in two stages we can know if we've checked every CPE, instead of accidentally breaking out of the scan too early. (From OE-Core rev: d61aff9e22704ad69df1f7ab0f8784f4e7cc0c69) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/classes/cve-check.bbclass | 63 ++++++++++++++++++---------------- 1 file changed, 34 insertions(+), 29 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 3326944d79..c1cbdbde7b 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -165,7 +165,6 @@ def check_cves(d, patched_cves): """ Connect to the NVD database and find unpatched cves. """ - import ast, csv, tempfile, subprocess, io from distutils.version import LooseVersion cves_unpatched = [] @@ -187,68 +186,74 @@ def check_cves(d, patched_cves): cve_whitelist = d.getVar("CVE_CHECK_WHITELIST").split() import sqlite3 - db_file = d.getVar("CVE_CHECK_DB_FILE") - conn = sqlite3.connect(db_file) + db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro") + conn = sqlite3.connect(db_file, uri=True) + # For each of the known product names (e.g. curl has CPEs using curl and libcurl)... for product in products: - c = conn.cursor() if ":" in product: vendor, product = product.split(":", 1) - c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR IS ?", (product, vendor)) else: - c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,)) + vendor = "%" - for row in c: - cve = row[0] - version_start = row[3] - operator_start = row[4] - version_end = row[5] - operator_end = row[6] + # Find all relevant CVE IDs. + for cverow in conn.execute("SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)): + cve = cverow[0] if cve in cve_whitelist: bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) # TODO: this should be in the report as 'whitelisted' patched_cves.add(cve) + continue elif cve in patched_cves: bb.note("%s has been patched" % (cve)) - else: - to_append = False + continue + + vulnerable = False + for row in conn.execute("SELECT * FROM PRODUCTS WHERE ID IS ? AND PRODUCT IS ? AND VENDOR LIKE ?", (cve, product, vendor)): + (_, _, _, version_start, operator_start, version_end, operator_end) = row + #bb.debug(2, "Evaluating row " + str(row)) + if (operator_start == '=' and pv == version_start): - to_append = True + vulnerable = True else: if operator_start: try: - to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) - to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) + vulnerable_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) + vulnerable_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) except: bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_start, version_start, cve)) - to_append_start = False + vulnerable_start = False else: - to_append_start = False + vulnerable_start = False if operator_end: try: - to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) - to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) + vulnerable_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) + vulnerable_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) except: bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_end, version_end, cve)) - to_append_end = False + vulnerable_end = False else: - to_append_end = False + vulnerable_end = False if operator_start and operator_end: - to_append = to_append_start and to_append_end + vulnerable = vulnerable_start and vulnerable_end else: - to_append = to_append_start or to_append_end + vulnerable = vulnerable_start or vulnerable_end - if to_append: + if vulnerable: bb.note("%s-%s is vulnerable to %s" % (product, pv, cve)) cves_unpatched.append(cve) - else: - bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) - patched_cves.add(cve) + break + + if not vulnerable: + bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) + # TODO: not patched but not vulnerable + patched_cves.add(cve) + conn.close() return (list(patched_cves), cves_unpatched) From patchwork Sun Dec 8 18:35:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180970 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3449797ile; Sun, 8 Dec 2019 10:41:28 -0800 (PST) X-Google-Smtp-Source: APXvYqziFMXaZyINB3ZbIR2BqK1oxrHblWD+6gdDtAakTKqYjh7BMCgxgeDTCilkgX2gFkRE30B6 X-Received: by 2002:a17:902:fe98:: with SMTP id x24mr25812800plm.155.1575830487579; Sun, 08 Dec 2019 10:41:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830487; cv=none; d=google.com; s=arc-20160816; b=AIRCOn0m+aS+Ie61cK6wUDZwfZL3tOR8r0ArhFJYe4ocRvujARXqI5sYcVK+wFeBBI NKkTaVq+WzKbvcy1iNv/6eR+RryXH3I57YXhGcAr7OnWU0rDfRP2Xjl5CIrN4P/TpLeU feWzm/evXOpWAf0U1QgcdDavda3RkJqKHN/wmJjUAfz00Jx15YCILu49fYWBU4pAiwQM B0k5p0KKn7breSXARYL/yDejz2AB/Q0/R9a3UyYYJYwLpMfP6Znzqot8aol6yp6IwwjE B/6J0t0hepQEdowwr6bUHDQGpdg6I7mtQ1UvrW0jGNKeNctroNjy4C3T457jCYJtxfxI 3ONQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=mh5LxRA0J1Qsk9K5qGZWDoVCg/ts9B/s4ZzcFH/Y1vs=; b=Ng1RuMVo0Hri5tFhK5vWRHMyzKiojhiNR407C0Cvd33OOr6rVGNw74f86fI2/pGeG7 heVd0O4zg7RE7gp5O4k4ifrGE/u3/9Y4OqTsG1UEqI6qV6qo6RnvvP3DJQnUw58TDNH5 5xSXohc1L/weOQWjEolKb0Z3xrZ0U+Eipeqyws2hUL+jFwv1cmWe9Ikf3HOk/U7u4Q4u rm/vOyDmjCUFSp/6iEJ314Z9TrXnicR+4a/QPHRpwDALuNZzOag3tKk/Lv+Yh5G7gH3w UwqIj8O3shEkZCvUN9KKNzm75zLxCQGO9HxB3Mr6Npl0KQ10Rkab9YdyZ/M5rTB6OYRk 7lEw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id bc6si14712501plb.367.2019.12.08.10.41.26; Sun, 08 Dec 2019 10:41:27 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id C14817FBC1; Sun, 8 Dec 2019 18:41:24 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from sypressi2.dnainternet.net (sypressi2.dnainternet.net [83.102.40.154]) by mail.openembedded.org (Postfix) with ESMTP id B7E9D7F8DB for ; Sun, 8 Dec 2019 18:41:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sypressi2.dnainternet.net (Postfix) with ESMTP id 6729C150FC; Sun, 8 Dec 2019 20:41:24 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from sypressi2.dnainternet.net ([83.102.40.154]) by localhost (sypressi2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id 3imeECL7qwIB; Sun, 8 Dec 2019 20:41:23 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by sypressi2.dnainternet.net (Postfix) with ESMTP id E0E8B14E51; Sun, 8 Dec 2019 20:41:23 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id 15B8489; Sun, 8 Dec 2019 20:41:18 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:56 +0200 Message-Id: <20191208183557.32589-10-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 10/11] cve-check: neaten get_cve_info X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Remove obsolete Python 2 code, and use convenience methods for neatness. (From OE-Core rev: f19253cc9e70c974a8e21a142086c13d7cde04ff) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/classes/cve-check.bbclass | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index c1cbdbde7b..e95716d9de 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -261,23 +261,15 @@ def check_cves(d, patched_cves): def get_cve_info(d, cves): """ Get CVE information from the database. - - Unfortunately the only way to get CVE info is set the output to - html (hard to parse) or query directly the database. """ - try: - import sqlite3 - except ImportError: - from pysqlite2 import dbapi2 as sqlite3 + import sqlite3 cve_data = {} - db_file = d.getVar("CVE_CHECK_DB_FILE") - placeholder = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder - conn = sqlite3.connect(db_file) - cur = conn.cursor() - for row in cur.execute(query, tuple(cves)): + conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) + placeholders = ",".join("?" * len(cves)) + query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders + for row in conn.execute(query, tuple(cves)): cve_data[row[0]] = {} cve_data[row[0]]["summary"] = row[1] cve_data[row[0]]["scorev2"] = row[2] From patchwork Sun Dec 8 18:35:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180971 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3449967ile; Sun, 8 Dec 2019 10:41:39 -0800 (PST) X-Google-Smtp-Source: APXvYqykqXCxT2y11/jzjDHHT6en+sNbVQylCsjxvyxRy5mCNLP7jvbHfosl12b3QFPJ4eg9sGHk X-Received: by 2002:a17:902:7286:: with SMTP id d6mr12418161pll.59.1575830499361; Sun, 08 Dec 2019 10:41:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830499; cv=none; d=google.com; s=arc-20160816; b=iBGMvfyc67fcKDCdYJTfjELl9l/idkkAwEUQjrTjMW0WBYK12x1l8c3SUvAGLNFyQB o0f5ry/vGqhqXs/JcSgQaS4/EIFwsVLttmsKob3tOWgmbvOvDWcFzWLPTkc6TjXJCceZ 2MjeagSzOT6cHqL0acHavaUBEOHs/kMATxpLPdM/URF7N78hQeaaTsm7oMVXNPdDk6wh 9KAEPbF0QvIssYX8QyNKBj3HtVFJKystNTI7SkSwJt1bM02fCInKFK1ACY/paDQXpkUi X/LnOq0kDMDdjswRINJ51HlDbQQ1MOmflzSc1SjuGzVryaMTSaD6JtaHaReqQphKpZeQ fjWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=5lULkA9YSKBrR86jEj/gX99eVIILsh0IiqjT25bQI5g=; b=fbaBk8zHb4vichVZGmhOSpU7TI2ZpXh+TQM3aBc5z86G8f5ZqoBsQKJ2i59YAjFzQP RG/qzUuHu3KqnjgWW8YOZJPy9Ie7XYWkgJVLZHSJu4laT2fdS3bEwJzwg7KZFFPFEtUn Fnii1O49v0DklcB1IbV18Mlop0k4rdieB1aSgImK6xGRov/5/bkWtt0ipJvq5DL32ecq hJlFQLk0mJLIUDZIUawBxcMe1EkS93pK7lvBpcL5QOcdlhgW3B3BihgCAB47XjldIiZs 1xQiFMKIpT97y/OyUyzISko7MoYxRZc73F+iRi5127g92wZSyJodd3+xWiDqyQd552MP jHaA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id v33si17466368pgk.286.2019.12.08.10.41.39; Sun, 08 Dec 2019 10:41:39 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 4ABC37FBFD; Sun, 8 Dec 2019 18:41:37 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from tulikuusama2.dnainternet.net (tulikuusama3.dnainternet.net [83.102.40.155]) by mail.openembedded.org (Postfix) with ESMTP id A58DF7F33D for ; Sun, 8 Dec 2019 18:41:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id 7E31229573; Sun, 8 Dec 2019 20:41:36 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from tulikuusama2.dnainternet.net ([83.102.40.155]) by localhost (tulikuusama2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id FiXClqNnx9Bi; Sun, 8 Dec 2019 20:41:36 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id 0A84C2940F; Sun, 8 Dec 2019 20:41:36 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id 313F674; Sun, 8 Dec 2019 20:41:30 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:57 +0200 Message-Id: <20191208183557.32589-11-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 11/11] cve-check: fetch CVE data once at a time instead of in a single call X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton This code used to construct a single SQL statement that fetched the NVD data for every CVE requested. For recipes such as the kernel where there are over 2000 CVEs to report this can hit the variable count limit and the query fails with "sqlite3.OperationalError: too many SQL variables". The default limit is 999 variables, but some distributions such as Debian set the default to 250000. As the NVD table has an index on the ID column, whilst requesting the data CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time different is insignificant: 0.05s verses 0.01s on my machine. (From OE-Core rev: 53d0cc1e9b7190fa66d7ff1c59518f91b0128d99) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/classes/cve-check.bbclass | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index e95716d9de..19ed5548b3 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -267,17 +267,17 @@ def get_cve_info(d, cves): cve_data = {} conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) - placeholders = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders - for row in conn.execute(query, tuple(cves)): - cve_data[row[0]] = {} - cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["scorev2"] = row[2] - cve_data[row[0]]["scorev3"] = row[3] - cve_data[row[0]]["modified"] = row[4] - cve_data[row[0]]["vector"] = row[5] - conn.close() + for cve in cves: + for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)): + cve_data[row[0]] = {} + cve_data[row[0]]["summary"] = row[1] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] + + conn.close() return cve_data def cve_write_data(d, patched, unpatched, cve_data):