From patchwork Sun Dec 8 18:35:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niko Mauno X-Patchwork-Id: 180971 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp3449967ile; Sun, 8 Dec 2019 10:41:39 -0800 (PST) X-Google-Smtp-Source: APXvYqykqXCxT2y11/jzjDHHT6en+sNbVQylCsjxvyxRy5mCNLP7jvbHfosl12b3QFPJ4eg9sGHk X-Received: by 2002:a17:902:7286:: with SMTP id d6mr12418161pll.59.1575830499361; Sun, 08 Dec 2019 10:41:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575830499; cv=none; d=google.com; s=arc-20160816; b=iBGMvfyc67fcKDCdYJTfjELl9l/idkkAwEUQjrTjMW0WBYK12x1l8c3SUvAGLNFyQB o0f5ry/vGqhqXs/JcSgQaS4/EIFwsVLttmsKob3tOWgmbvOvDWcFzWLPTkc6TjXJCceZ 2MjeagSzOT6cHqL0acHavaUBEOHs/kMATxpLPdM/URF7N78hQeaaTsm7oMVXNPdDk6wh 9KAEPbF0QvIssYX8QyNKBj3HtVFJKystNTI7SkSwJt1bM02fCInKFK1ACY/paDQXpkUi X/LnOq0kDMDdjswRINJ51HlDbQQ1MOmflzSc1SjuGzVryaMTSaD6JtaHaReqQphKpZeQ fjWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=5lULkA9YSKBrR86jEj/gX99eVIILsh0IiqjT25bQI5g=; b=fbaBk8zHb4vichVZGmhOSpU7TI2ZpXh+TQM3aBc5z86G8f5ZqoBsQKJ2i59YAjFzQP RG/qzUuHu3KqnjgWW8YOZJPy9Ie7XYWkgJVLZHSJu4laT2fdS3bEwJzwg7KZFFPFEtUn Fnii1O49v0DklcB1IbV18Mlop0k4rdieB1aSgImK6xGRov/5/bkWtt0ipJvq5DL32ecq hJlFQLk0mJLIUDZIUawBxcMe1EkS93pK7lvBpcL5QOcdlhgW3B3BihgCAB47XjldIiZs 1xQiFMKIpT97y/OyUyzISko7MoYxRZc73F+iRi5127g92wZSyJodd3+xWiDqyQd552MP jHaA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id v33si17466368pgk.286.2019.12.08.10.41.39; Sun, 08 Dec 2019 10:41:39 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=iki.fi Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 4ABC37FBFD; Sun, 8 Dec 2019 18:41:37 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from tulikuusama2.dnainternet.net (tulikuusama3.dnainternet.net [83.102.40.155]) by mail.openembedded.org (Postfix) with ESMTP id A58DF7F33D for ; Sun, 8 Dec 2019 18:41:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id 7E31229573; Sun, 8 Dec 2019 20:41:36 +0200 (EET) X-Virus-Scanned: DNA Internet at dnainternet.net X-Spam-Flag: NO X-Spam-Score: 0.653 X-Spam-Level: X-Spam-Status: No, score=0.653 tagged_above=-9999 required=6 tests=[SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from tulikuusama2.dnainternet.net ([83.102.40.155]) by localhost (tulikuusama2.dnainternet.net [127.0.0.1]) (DNA Internet, port 10041) with ESMTP id FiXClqNnx9Bi; Sun, 8 Dec 2019 20:41:36 +0200 (EET) Received: from luumupuu2.dnainternet.net (luumupuu2.dnainternet.net [83.102.40.55]) by tulikuusama2.dnainternet.net (Postfix) with ESMTP id 0A84C2940F; Sun, 8 Dec 2019 20:41:36 +0200 (EET) Received: from localhost.localdomain (62-78-161-185.bb.dnainternet.fi [62.78.161.185]) by luumupuu2.dnainternet.net (Postfix) with ESMTP id 313F674; Sun, 8 Dec 2019 20:41:30 +0200 (EET) From: Niko Mauno To: openembedded-core@lists.openembedded.org Date: Sun, 8 Dec 2019 20:35:57 +0200 Message-Id: <20191208183557.32589-11-niko.mauno@iki.fi> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191208183557.32589-1-niko.mauno@iki.fi> References: <20191208183557.32589-1-niko.mauno@iki.fi> MIME-Version: 1.0 Subject: [OE-core] [thud-next][PATCH 11/11] cve-check: fetch CVE data once at a time instead of in a single call X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton This code used to construct a single SQL statement that fetched the NVD data for every CVE requested. For recipes such as the kernel where there are over 2000 CVEs to report this can hit the variable count limit and the query fails with "sqlite3.OperationalError: too many SQL variables". The default limit is 999 variables, but some distributions such as Debian set the default to 250000. As the NVD table has an index on the ID column, whilst requesting the data CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time different is insignificant: 0.05s verses 0.01s on my machine. (From OE-Core rev: 53d0cc1e9b7190fa66d7ff1c59518f91b0128d99) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/classes/cve-check.bbclass | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index e95716d9de..19ed5548b3 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -267,17 +267,17 @@ def get_cve_info(d, cves): cve_data = {} conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) - placeholders = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders - for row in conn.execute(query, tuple(cves)): - cve_data[row[0]] = {} - cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["scorev2"] = row[2] - cve_data[row[0]]["scorev3"] = row[3] - cve_data[row[0]]["modified"] = row[4] - cve_data[row[0]]["vector"] = row[5] - conn.close() + for cve in cves: + for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)): + cve_data[row[0]] = {} + cve_data[row[0]]["summary"] = row[1] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] + + conn.close() return cve_data def cve_write_data(d, patched, unpatched, cve_data):