From patchwork Wed Nov 6 15:37:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikko Rapeli X-Patchwork-Id: 178740 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp874385ilf; Wed, 6 Nov 2019 08:55:21 -0800 (PST) X-Google-Smtp-Source: APXvYqxw+xUbeWPpeaKfx2v37f4aQ1LKt9MVb1+DHLdQLPFHPAV7z+/ttqdYXXWsKO12SGHp4QDK X-Received: by 2002:a63:234c:: with SMTP id u12mr3903152pgm.384.1573059321431; Wed, 06 Nov 2019 08:55:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573059321; cv=none; d=google.com; s=arc-20160816; b=KGzio9/swCi5d6kY4zSOucFiUlJMzcW1v95Ac3UxSwgJIG1dyJM9N5EzjYAunrycre yRN7wdP0d2+fp1ti5NaoDryJzVfjX2cXHaVA77ctdSz6+o0YR5GudphCeF5uhzE4ThEe IQ7KM8/Wf9vN1uTxrb2jLKwzCk+Xhx4qmR8QdeTG2g/YXJCLSafZCKMKohEvvfZgZ/Sy BlPpqvHxigu+hCmxOETdu4vWAsHHVFLL+fZqzyLbVpbik7dXgl2PTDXel4KE0DEvuKss Ekl1tFoOgEd3W/5oXPiX6v0CSAvKpJZuUTSlomXul1kCqxDV+BawLPtsWv+0bGLjR+S/ Okog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:references :in-reply-to:message-id:date:to:from:dkim-signature:delivered-to; bh=Kq/IsCpS/yYJPTcWA2s6+Dc4OivuIZ2D7JkHqK93DMk=; b=evPcyVso5zjqXaj3ZVeKo4/QXeOJIdRtYp+gDEpPSASc5bSJmWcvMgL2SfcP0neTrl qEZTwfw1AQ+UHWX3Zv0T4FIeu6rN5QmowIaHSa8P2uzsr6QWLrcoWV6p07ajQ5nJfFXa 1SJ7XVb5s/hmQzZGm1rKxUxtpeQssqk4kadU2YQQqDculCdt3jflZzCQsdHcn0GrwfQP ScuVq2kP/vPpPeAkF0l7PcB4oo6Yh+RBDnWykMI4NxkLTbhTYCpk61TzstGI+UHFzcGD Zcs3Yk6X/tkZs0fm16eUOEfHEqlfvk/OypDt3oGe46YWqVlTAFSdb2Qa+9EQW2x0citI kMxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@kapsi.fi header.s=20161220 header.b="jGU/z9yR"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bmw.de Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id u15si14275893pgc.477.2019.11.06.08.55.21; Wed, 06 Nov 2019 08:55:21 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@kapsi.fi header.s=20161220 header.b="jGU/z9yR"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bmw.de Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 513F97FA20; Wed, 6 Nov 2019 16:55:01 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail.kapsi.fi (mail.kapsi.fi [91.232.154.25]) by mail.openembedded.org (Postfix) with ESMTP id 9C5FC7F899 for ; Wed, 6 Nov 2019 15:38:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kapsi.fi; s=20161220; h=References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=uGziAfjOSxnkNrexjZ/jkqOxvYb38GQuJYlUjrRSyek=; b=jGU/z9yR0QulafT0bJwt4k82UO c+50TUSHDhxc+deOgxsMVyVLNPN2gXVqv5rekTakhzEUKaQL4ZFATrdark77hWzIkMua1fmoi3tmR YWsQo6x58n6vtBJPoa2oEfrmhCSno8LZsI9N+/5pH7P7WA4G75x5G2Z2AyRQ2FL0aC4Y/pIghzenl jHICUY06yTFkeSnFURdoAurT2RLIDE/L2x9ZqY0DL264L8+PB//L6UGoH+oeA6ZHtkLSlnZz9CaX7 d4QGZrkD77Um2vatzKeyW/4a3KtP5KmlbtsNWYvlfdaNSGA5xP4TT4DY/UpSwyrcZvaNVLwkF/r/+ OQES51Bg==; Received: from kapsi.fi ([91.232.154.11] helo=lakka.kapsi.fi) by mail.kapsi.fi with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1iSNOa-0000AC-1n; Wed, 06 Nov 2019 17:38:56 +0200 Received: from mcfrisk by lakka.kapsi.fi with local (Exim 4.84_2) (envelope-from ) id 1iSNO5-0007XP-TV; Wed, 06 Nov 2019 17:38:25 +0200 From: Mikko Rapeli To: openembedded-core@lists.openembedded.org Date: Wed, 6 Nov 2019 17:37:41 +0200 Message-Id: <311cfe9aa30c7d4a9476dec12769d2c4290f5a91.1573047194.git.mikko.rapeli@bmw.de> X-Mailer: git-send-email 2.1.4 In-Reply-To: References: In-Reply-To: References: X-Rspam-Score: 6.0 (++++++) X-Rspam-Report: Action: add header Symbol: ARC_NA(0.00) Symbol: DMARC_POLICY_SOFTFAIL(0.10) Symbol: R_SPF_NEUTRAL(0.00) Symbol: FROM_HAS_DN(0.00) Symbol: TO_DN_SOME(0.00) Symbol: MULTIPLE_UNIQUE_HEADERS(4.89) Symbol: MIME_GOOD(-0.10) Symbol: TO_MATCH_ENVRCPT_ALL(0.00) Symbol: RCPT_COUNT_THREE(0.00) Symbol: RCVD_TLS_LAST(0.00) Symbol: MID_CONTAINS_FROM(1.00) Symbol: NEURAL_HAM(-0.00) Symbol: IP_SCORE(-0.15) Symbol: FORGED_SENDER(0.30) Symbol: R_DKIM_NA(0.00) Symbol: MIME_TRACE(0.00) Symbol: ASN(0.00) Symbol: FROM_NEQ_ENVFROM(0.00) Symbol: RCVD_COUNT_TWO(0.00) Message: (SPF): spf neutral Message-ID: 311cfe9aa30c7d4a9476dec12769d2c4290f5a91.1573047194.git.mikko.rapeli@bmw.de X-Rspam-Status: Yes X-Rspam-Bar: ++++++ X-SA-Exim-Connect-IP: 91.232.154.11 X-SA-Exim-Mail-From: mcfrisk@kapsi.fi X-SA-Exim-Scanned: No (on mail.kapsi.fi); SAEximRunCond expanded to false Subject: [OE-core] [PATCH RFC CFH][sumo 26/47] cve-update-db-native: clean up JSON fetching X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Currently the code fetches the compressed JSON, writes it to a temporary file, uncompresses that with gzip and passes the fake file object to update_db(). Instead, uncompress the gzip'd data in memory and pass the JSON directly to update_db(). (From OE-Core rev: 9422745979256c442f533770203f62ec071c18fb) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/recipes-core/meta/cve-update-db-native.bb | 29 +++++++++++--------------- 1 file changed, 12 insertions(+), 17 deletions(-) -- 1.9.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 6907197..a06b74a 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -62,25 +62,20 @@ python do_populate_cve_db() { meta = c.fetchone() if not meta or meta[0] != last_modified: # Clear products table entries corresponding to current year - cve_year = 'CVE-' + str(year) + '%' - c.execute("delete from PRODUCTS where ID like ?", (cve_year,)) + c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,)) # Update db with current year json file - req = urllib.request.Request(json_url) - if proxy: - req.set_proxy(proxy, 'https') try: - with urllib.request.urlopen(req, timeout=1) as r, \ - open(json_tmpfile, 'wb') as tmpfile: - shutil.copyfileobj(r, tmpfile) - except: + req = urllib.request.Request(json_url) + if proxy: + req.set_proxy(proxy, 'https') + with urllib.request.urlopen(req) as r: + update_db(c, gzip.decompress(r.read())) + c.execute("insert or replace into META values (?, ?)", [year, last_modified]) + except urllib.error.URLError as e: cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') - break - - with gzip.open(json_tmpfile, 'rt') as jsonfile: - update_db(c, jsonfile) - c.execute("insert or replace into META values (?, ?)", - [year, last_modified]) + bb.warn("Cannot parse CVE data (%s), update failed" % e.reason) + return # Update success, set the date to cve_check file. if year == date.today().year: @@ -143,9 +138,9 @@ def parse_node_and_insert(c, node, cveId): c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) -def update_db(c, json_filename): +def update_db(c, jsondata): import json - root = json.load(json_filename) + root = json.loads(jsondata) for elt in root['CVE_Items']: if not elt['impact']: