From patchwork Wed Nov 13 15:35:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179331 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp9765274ilf; Wed, 13 Nov 2019 07:37:40 -0800 (PST) X-Google-Smtp-Source: APXvYqyjQzeKpqkA1ZCHoQ/JE7onMmUjfqwG5PaU05CRxklSK+hsb6WgBxhI7x52JOGPkv6l/7qJ X-Received: by 2002:a17:90a:b109:: with SMTP id z9mr5675468pjq.108.1573659460647; Wed, 13 Nov 2019 07:37:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573659460; cv=none; d=google.com; s=arc-20160816; b=Z8them1lG4//4PjIW7qQTgoF52MOD3XctXBF4uW3OaKsWTcRAaT1jqlD1eFNJftL+6 +ogqTjxs9NRYQ0Md18McBBL6yedLXAPl/S1U5ESuA1AoMwHSyOlZIW9uKX+zIN1RtQNY hqJXl48JXJv6MUBUo/9s3EpQJESG4Q4TnpRGQiYpj3PJ7jKH2eyLsjauXX+/Q4WLOq6G aJiQ5ZmxUQuob1qEtwYscQVExZelY6vqcVqXxQN0Z9/T+Sir67x0xKVCPXXe7QKjJQnT nnZ2N/3CudhLy1eg+OOfBkbylhDyhk/iZAhtKgZN9/2IpDuG2iOdVjFkVks1BWaL2KJO tusg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:message-id:date:to:from:dkim-signature:delivered-to; bh=4VAvH/23KL+9U8ajRA9wcqi2yHN2/tpNRWAN+VGVeYA=; b=lLum/j6ZYXmSYy/XfQ2i/8yRawnfBfuF/DLemi1flUbjd+BV58HotjYpTD4uS2G6an 0jU9Cq+Xx0u0oDUY3vVjMQshO/IDH7BftU+BBAhF01stCDkWYWrZVUBhN834Uz4qIJmO lgkn2Qd+WqGxAS2nikqUE8fPGuEeoGvFjpfDJ7nhjg9O6ywVuhPt+iDeFF9hEB0478MA ov7t9ZpXSJt+3/5F6EjrV9vLbdRRy9qGdySGwTRWljfsYWDDM11is0VUufgtt3hkcRO1 MJ/x2uggPlHF6WcHswneEry3NuwhpDxBIHLT76Pj7e34A48Xy4U2y/PRRQJxOgfoCtIL l6mA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b="obr23LC/"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id v1si3222529pgs.468.2019.11.13.07.37.40; Wed, 13 Nov 2019 07:37:40 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b="obr23LC/"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 92F757F7B5; Wed, 13 Nov 2019 15:35:53 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by mail.openembedded.org (Postfix) with ESMTP id DD7067F639 for ; Wed, 13 Nov 2019 15:35:52 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id z26so2468132wmi.4 for ; Wed, 13 Nov 2019 07:35:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=j1IWFqB7azBIn6DpZdrm8PijAFQZ1TpqjFaEslwaegg=; b=obr23LC/2Xk9MzStUHX0DZymqmy+oLgN4BDHgyK/FiNY37BcmQytjVttr6QS16HJNQ gf8Nhw+CnFYZl/EgTEnJ3kRk8NVvY17zf69hFIDaHQweE1TVycxmppdQB0A1FD+yPVlZ JmDOF8o+3fUlttkh+Glp5rxjBPMY1DmV1p2mzW+tPfYuOBLo57ChymEibZu4IgDNo3ir PXsIYvwED4ZM4Ij6+z7W2j14Sc6Gaj/6bfpWpDxB1hFfIDKNoQCrBUJxLcaxTR81Ynrd wCy75/Kd4dsJLO0DDWYlKbgPM+FRNt2Wfx4vw6bVZm8I7UmFpwBiMHtapsu+EFeTrsZU BeUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=j1IWFqB7azBIn6DpZdrm8PijAFQZ1TpqjFaEslwaegg=; b=UXO0pxXX6l4MDdgxLGCuPCmCjyMYeWsfzUPu2LZogZUw25eFgPAxcPEKnveVgHQKb6 C34NOkOWe7IbCmmIHpCvdHUjjSV2bh2YsiyiT/h/2Jg6BFbnFvRMjhAtuxodRiB2bWbx qJgTGRkjxFYFnSOD4qeANMo3KiW1tfDfxujaJGZL5Nr34ZdpF0+FppwC2q5cNu/SfEPX SrsLcQbJZ3y0oMfewaEndyhvtoclNGVdD0eBnJKxjK/wqiZy2qlQGJ4XV5Fs4u7GiA30 JGusgmKE9PCiflVfysfqS/vO6pfd1IhUXDhreUAwsZasf+JNidgZAMOlBWkxrcz80hjE 9OTQ== X-Gm-Message-State: APjAAAWQHXDURJVDygqXkBmGwxnksCfs7q/VkpT11D4KBzxrlE9+oixA iC56PFr4NXRG3OCNH2yxZNK7OePgCxU= X-Received: by 2002:a05:600c:2202:: with SMTP id z2mr3245714wml.162.1573659353318; Wed, 13 Nov 2019 07:35:53 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id z8sm3116812wrp.49.2019.11.13.07.35.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2019 07:35:51 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Wed, 13 Nov 2019 15:35:48 +0000 Message-Id: <20191113153548.8644-1-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [OE-core] [PATCH] cve-check: fetch CVE data once at a time instead of in a single call X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org This code used to construct a single SQL statement that fetched the NVD data for every CVE requested. For recipes such as the kernel where there are over 2000 CVEs to report this can hit limits in older sqlite releases and the query fails with "sqlite3.OperationalError: too many SQL variables". As the NVD table has an index on the ID column, whilst requesting the data CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time different is insignificant: 0.05s verses 0.01s on my machine. Signed-off-by: Ross Burton --- meta/classes/cve-check.bbclass | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index e95716d9ded..f23f683ae80 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -267,17 +267,17 @@ def get_cve_info(d, cves): cve_data = {} conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) - placeholders = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders - for row in conn.execute(query, tuple(cves)): + + for cve in cves: + row = conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)).fetchone() cve_data[row[0]] = {} cve_data[row[0]]["summary"] = row[1] cve_data[row[0]]["scorev2"] = row[2] cve_data[row[0]]["scorev3"] = row[3] cve_data[row[0]]["modified"] = row[4] cve_data[row[0]]["vector"] = row[5] - conn.close() + conn.close() return cve_data def cve_write_data(d, patched, unpatched, cve_data):