From patchwork Mon Dec 16 15:59:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181738 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4479637och; Mon, 16 Dec 2019 08:00:40 -0800 (PST) X-Google-Smtp-Source: APXvYqw1USC+DjLWr3XISMdneITzz+Mtb7U8P3o3TOcdHtIzT96KOICzGWx4HxOPQrW6WZbjq5xs X-Received: by 2002:a17:902:bd8b:: with SMTP id q11mr16627352pls.182.1576512039800; Mon, 16 Dec 2019 08:00:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512039; cv=none; d=google.com; s=arc-20160816; b=0AZMlAlVyTVc5KhYRxnoUkbs/0Wz0a+PA8aOHwhpzXg9jYHJarossc4uAemG5jn5gU +vwMMSb0qzF8M+CqSFZoOC4aASP0SswNjmiN+XNNDRhmlKvYaudawH00qrRpzjfGYnnO lgtTwPyZrpU2QouVVUKh5PzB52c8gDijW44/aBt5ly72/yEbbMAeR5jI58Y9idIfRqRQ Ha5S35I8x8rgqO6P0DrXYQxuTxEkDeKCgW0ja0IzsgJ9YVa+VboVTKSWjfM1ojbo0H2D fI7tOMFdk1G1+thEQIyVDvgZtTarhkmeGUd1XmuBfQnIpi+E+6cLhAraLcmUBN1HTX6V G5/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=+ccJQrMq56cIcnf2Z3/NmIPglD3qjw+vEtE7R7YoUVc=; b=XNzGqvUzGsgh1n/8W/n40V5uQkRzfrBtHrQ3Ltum4XR+8CMtg22vw4qjdB0mOY9rFX JFie9wbzbPhCK0oT8cvAWPzdeqBvxwqmZwJFvOZMVYmhuwRrZnsq6/1fOaM+JpH2fz5c akt7Y/TTYq6gN370W+KZLb5YeKK17nDWJc5GhDbsnBArgiqnuNOWKrnR73bfpe7/puui 1wEk5xqQ0jUuZ7qLTTOYmL6jE+6dUoFz09BU13jcj5F1pwTj8s+/XF+lNjdkivBMacK0 HsIIc2Ih5yIx5L8aah1hBPSfiXMmdV9maBQGaCm/LAUiNOohrrCZV9vnjKnmqCeZAXzR 2Nuw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="DCdxnz/+"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id cm5si16504796pjb.95.2019.12.16.08.00.39; Mon, 16 Dec 2019 08:00:39 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="DCdxnz/+"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id D40507FDDF; Mon, 16 Dec 2019 16:00:30 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by mail.openembedded.org (Postfix) with ESMTP id EDC017FDA1 for ; Mon, 16 Dec 2019 16:00:11 +0000 (UTC) Received: by mail-pg1-f175.google.com with SMTP id q127so3963291pga.4 for ; Mon, 16 Dec 2019 08:00:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=2/glOOHy3oqk/KUsu4K7YjaSpxAyMsywLlB6f9m561o=; b=DCdxnz/+119K5GTYfWcInlNdq9kQBPHYBx6J8gx8Doic21YQbtuJCCi8/TMw1Kn00z CRyrA2F5FKpRY2tPCz0aW9NoREUPJDpbvjaQI+1kFWfbnPEwBy4MSv610lthhp3onu2X iJ+KJS1pVVLkNMtACFHChxbyvvUI8j4Y5XMpcYwaoPZBw3FXKVSAmTkPfUr6YStww/q2 dzlD2wptkXaw3u1IBF1A5CBHi2kwnmVjQOmsZ2wRI4EAhVeMCrm4Xqijebqy58kWompP vdC5jWU9Awv3eH5A8oNudMLhwRNdvXph0ZIGECCB1LgnJlo/ic4jwf1vLQ/Uffb91BPu P3Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=2/glOOHy3oqk/KUsu4K7YjaSpxAyMsywLlB6f9m561o=; b=IsTOGMzX1TVvXki3Do0tjzS58kjbSsi99BKIDzWOFVt6q253FVZM12veL3KTuyLvp5 Ev9pgqoaZV6NN5Y2v/u+KHiMEfXLzoeElfVPzjIe08qMsqQQEkmoA0Jwdw09WuHe+HIY +2qsy4IlnVsnlneW05OJJBtrK6nivUATAPmtdwtKNCHTlwzAr4hOFIS8oMLF2ogmeTfn jhXCtwnxkyjZO3ME0SEcA70Ne0mGABULzndHvt4vG6RXKvqjx7yGcVAzm+2m2KG6NEmd dZB5oGtC6QBGeSqU3aUtNCZqDQ6jW5tu1eSWcCXMC3ufXMphHWGgcGSWPyD3fRH6J1kQ Aq1Q== X-Gm-Message-State: APjAAAUw/Wuaw0ruUqK+fhC+uXp9A94VfcDC6iHlMiLGFoOeaCo4LAQs /YO76Jy35bS7OZRzT4pvg+uvN34jLPc= X-Received: by 2002:a63:646:: with SMTP id 67mr19218061pgg.150.1576512012858; Mon, 16 Dec 2019 08:00:12 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:12 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 07:59:52 -0800 Message-Id: <4a17afb3fe42cbc01c52b2d5357f6021bf782c01.1576511913.git.akuster808@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 02/18] sdk: Install nativesdk locales for all TCLIBC variants X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Khem Raj install_locales() here is actually operating on nativesdk and only glibc is the default library for nativesdk, since thats what most of desktop/server distros use, therefore bailing out based on TCLIBC is not needed here, since nativesdk-glibc would be required for all non-glibc targetting SDKs as well. Fixes SDK install time error ERROR: OE-core's config sanity checker detected a potential misconfiguration. Either fix the cause of this error or at your own risk disable the checker (see sanity.conf). Following is the list of potential problems / advisories: Your system needs to support the en_US.UTF-8 locale. ERROR: SDK preparation failed Signed-off-by: Khem Raj Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/lib/oe/sdk.py | 4 ---- 1 file changed, 4 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/lib/oe/sdk.py b/meta/lib/oe/sdk.py index 153b07d..ef81f8c 100644 --- a/meta/lib/oe/sdk.py +++ b/meta/lib/oe/sdk.py @@ -84,10 +84,6 @@ class Sdk(object, metaclass=ABCMeta): bb.warn("cannot remove SDK dir: %s" % path) def install_locales(self, pm): - # This is only relevant for glibc - if self.d.getVar("TCLIBC") != "glibc": - return - linguas = self.d.getVar("SDKIMAGE_LINGUAS") if linguas: import fnmatch From patchwork Mon Dec 16 15:59:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181740 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4480894och; Mon, 16 Dec 2019 08:01:32 -0800 (PST) X-Google-Smtp-Source: APXvYqw2J+QkDpgmjzSA4TLaASCvf2CiAHs2mqKsW4N3jYNcibCXBcmXT8CKibYjXO9NYZRzqbDW X-Received: by 2002:a17:902:fe10:: with SMTP id g16mr16499035plj.66.1576512091650; Mon, 16 Dec 2019 08:01:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512091; cv=none; d=google.com; s=arc-20160816; b=0addY6rHFmp8uDQs3X/lR92GXZ5CK3xaNxN9JZMcG+GPatSZliiZGl0xujJB92qpdr F1txdgoculT4cBxYfphiJIY/92ypv3rzF5NiDVUeNDP2NyRJS1LDgOOxsGyA6U6qpi0y wpHHrYcUHTPXrTW25SE3hTjNsE8+KNDfrDBI4IWtWfIIG5CZI9Wc/GwoVODrMrrXjN6G YCLcEM+pKCyQyoiReRzuJ2q4kzsJwH8EEuXd7xAqcleydh2lEJoMVcfe2bo+lyK8/3UI QBFv7LSUvjgTMrHshGTqUQR5lAifIr6za9y6b2RDbO7dVWKCXAQj/Cjy1+yQYVi6Qqa2 LFUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :references:in-reply-to:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=l8TeJcjgV1AtNYOURXZPsNd1T/vIINOF7fO9oMj++vs=; b=wOXqK95SfCkBuoN0TRwz6JYyfrWvsa2ODv1UMGSZWceQfWH/Eof5UustzvjGHVRzfd mHIU8C/SCV8JiH+lDNILFpIQSflf9o7tifx1NvYtzZNRkecx0fDSXeDqalCo9j3Z91yg 0LGkJt8sL83KUrARNlRPTQIvn+aURSGELGi1uPSZMOwILgghf6BSLOXYMNlWgk+q/giM mv69S7oApBbmcPDcoRVWzVPCv1xD3OVIkg6YXx6sg1lUoFU4m/TOLwK0RSAywru+qaZS Uz8VFQeY1RnlQIz6siNtHiknT0My4BfUU3vu+K2pq4P3YesGnpumC31CSckkyq0DLH39 26iQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=VMXSuqun; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id z64si19666150pgd.9.2019.12.16.08.01.31; Mon, 16 Dec 2019 08:01:31 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=VMXSuqun; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 363C17FDF7; Mon, 16 Dec 2019 16:00:41 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by mail.openembedded.org (Postfix) with ESMTP id 303927FDB1 for ; Mon, 16 Dec 2019 16:00:18 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id o11so3156923pjp.9 for ; Mon, 16 Dec 2019 08:00:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version:in-reply-to:references :content-transfer-encoding; bh=D789KSiBZyq0onz80h8JmanAVg7At77qfV3WY6kt4Og=; b=VMXSuqun02MF42qnz6omTI5CbvQF80Y92Z91YB/OdlukkzPIpUZGfnE0rAY4PNuNBk 2cjwcrwUTgeg3ZIWXBsZJCvJzA4ytL2bsv8q3UctTMj7MZDjqssIAcNfSk7UZpMmopUt A0jAR83bDjaQqapO/1NYjMmLEKIrCXNQt9ebFOS/J7XNsWCA1gwnymnRa+krnseA++YO 0ATHU/x3Dy5Oa2+HHdIXNUmwk5zrQSFTk4mQZvwVCgZJLBZ7mbODLqI8RuUmQPg4p0g3 FjX0p69QQeTjupFJyHV6I+SZy7oa0ZJMGdE0YtU7ZmGwTEKaclmj0w6Nb5E3WIJP2MWW 6kaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :in-reply-to:references:content-transfer-encoding; bh=D789KSiBZyq0onz80h8JmanAVg7At77qfV3WY6kt4Og=; b=QHsBfA0djhGpcHBzm8Mx8JWoCWn5BEJq2Mg9Beb/H8eZ9Yk5cC0M05wFAvGm1v2nQ5 PeGmSedmjcRniuKiuRUrDCj45/EAk/qUgMVbOnL6ZmfEG+Xp9yZV13AlUKZubs79emHw qFmwxL1UQ3I6CnGzCGmBc0AkN2ntnzXVqKRecTjmtTh/ern81eOE2E/WDr4ol+ima0GE sZaccmFtpeic/imzeEmFJ4ffIhR7oTlEvmcpwdN6Z84n75MDvxyI9G4GnNpFuSOEvB+z 7PWZjRTj7jxFArO2f8blYDPIchnO+HG3YsNYCp7/ORe5W2kX38m5Qq4Aa0/RAraLcriv ozzg== X-Gm-Message-State: APjAAAUCzRARcY6zYEORwkjHenghpx5KjJwYy+89T1Oe7iqRn311zb9b qPp1ojxBrHBD3tuVvi549fEIu5WsdFg= X-Received: by 2002:a17:902:b709:: with SMTP id d9mr16854977pls.235.1576512017706; Mon, 16 Dec 2019 08:00:17 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.16 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:16 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 07:59:56 -0800 Message-Id: X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 In-Reply-To: References: Subject: [OE-core] [thud 06/18] cve-check: backport rewrite from master X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton As detailed at [1] the XML feeds provided by NIST are being discontinued on October 9th 2019. As cve-check-tool uses these feeds, cve-check.bbclass will be inoperable after this date. To ensure that cve-check continues working, backport the following commits from master to move away from the unmaintained cve-check-tool to our own Python code that fetches the JSON: 546d14135c5 cve-update-db: New recipe to update CVE database bc144b028f6 cve-check: Remove dependency to cve-check-tool-native 7f62a20b32a cve-check: Manage CVE_PRODUCT with more than one name 3bf63bc6084 cve-check: Consider CVE that affects versions with less than operator c0eabd30d7b cve-update-db: Use std library instead of urllib3 27eb839ee65 cve-check: be idiomatic 09be21f4d17 cve-update-db: Manage proxy if needed. 975793e3825 cve-update-db: do_populate_cve_db depends on do_fetch 0325dd72714 cve-update-db: Catch request.urlopen errors. 4078da92b49 cve-check: Depends on cve-update-db-native f7676e9a38d cve-update-db: Use NVD CPE data to populate PRODUCTS table bc0195be1b1 cve-check: Update unpatched CVE matching c807c2a6409 cve-update-db-native: Skip recipe when cve-check class is not loaded. 07bb8b25e17 cve-check: remove redundant readline CVE whitelisting 5388ed6d137 cve-check-tool: remove 270ac00cb43 cve-check.bbclass: initialize to_append e6bf9000987 cve-check: allow comparison of Vendor as well as Product 91770338f76 cve-update-db-native: use SQL placeholders instead of format strings 7069302a4cc cve-check: Replace CVE_CHECK_CVE_WHITELIST by CVE_CHECK_WHITELIST 78de2cb39d7 cve-update-db-native: Remove hash column from database. 4b301030cf9 cve-update-db-native: use os.path.join instead of + f0d822fad2a cve-update-db: actually inherit native b309840b6aa cve-update-db-native: use executemany() to optimise CPE insertion bb4e53af33d cve-update-db-native: improve metadata parsing 94227459792 cve-update-db-native: clean up JSON fetching 95438d52b73 cve-update-db-native: fix https proxy issues 1f9a963b9ff glibc: exclude child recipes from CVE scanning [1] https://nvd.nist.gov/General/News/XML-Vulnerability-Feed-Retirement (From OE-Core rev: 8c87e78547c598cada1bce92e7b25d85b994e2eb) Signed-off-by: Ross Burton Signed-off-by: Armin Kuster Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/classes/cve-check.bbclass | 142 ++++++++------ meta/conf/distro/include/maintainers.inc | 1 + meta/recipes-core/glibc/glibc-locale.inc | 3 + meta/recipes-core/glibc/glibc-mtrace.inc | 3 + meta/recipes-core/glibc/glibc-scripts.inc | 3 + meta/recipes-core/meta/cve-update-db-native.bb | 195 +++++++++++++++++++ .../cve-check-tool/cve-check-tool_5.6.4.bb | 62 ------ ...01-Fix-freeing-memory-allocated-by-sqlite.patch | 50 ----- ...ow-overriding-default-CA-certificate-file.patch | 215 --------------------- ...ogress-in-percent-when-downloading-CVE-db.patch | 135 ------------- ...are-computed-vs-expected-sha256-digit-str.patch | 52 ----- .../check-for-malloc_trim-before-using-it.patch | 51 ----- 12 files changed, 292 insertions(+), 620 deletions(-) create mode 100644 meta/recipes-core/meta/cve-update-db-native.bb delete mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 743bc08..c00d291 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}" CVE_VERSION ??= "${PV}" CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK" -CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db" +CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_1.0.db" CVE_CHECK_LOG ?= "${T}/cve.log" CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check" @@ -37,32 +37,33 @@ CVE_CHECK_COPY_FILES ??= "1" CVE_CHECK_CREATE_MANIFEST ??= "1" # Whitelist for packages (PN) -CVE_CHECK_PN_WHITELIST = "\ - glibc-locale \ -" +CVE_CHECK_PN_WHITELIST ?= "" -# Whitelist for CVE and version of package -CVE_CHECK_CVE_WHITELIST = "{\ - 'CVE-2014-2524': ('6.3','5.2',), \ -}" +# Whitelist for CVE. If a CVE is found, then it is considered patched. +# The value is a string containing space separated CVE values: +# +# CVE_CHECK_WHITELIST = 'CVE-2014-2524 CVE-2018-1234' +# +CVE_CHECK_WHITELIST ?= "" python do_cve_check () { """ Check recipe for patched and unpatched CVEs """ - if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE")): + if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")): patched_cves = get_patches_cves(d) patched, unpatched = check_cves(d, patched_cves) if patched or unpatched: cve_data = get_cve_info(d, patched + unpatched) cve_write_data(d, patched, unpatched, cve_data) else: - bb.note("Failed to update CVE database, skipping CVE check") + bb.note("No CVE database found, skipping CVE check") + } addtask cve_check after do_unpack before do_build -do_cve_check[depends] = "cve-check-tool-native:do_populate_sysroot cve-check-tool-native:do_populate_cve_db" +do_cve_check[depends] = "cve-update-db-native:do_populate_cve_db" do_cve_check[nostamp] = "1" python cve_check_cleanup () { @@ -163,65 +164,94 @@ def get_patches_cves(d): def check_cves(d, patched_cves): """ - Run cve-check-tool looking for patched and unpatched CVEs. + Connect to the NVD database and find unpatched cves. """ - import ast, csv, tempfile, subprocess, io + from distutils.version import LooseVersion - cves_patched = [] cves_unpatched = [] - bpn = d.getVar("CVE_PRODUCT") + # CVE_PRODUCT can contain more than one product (eg. curl/libcurl) + products = d.getVar("CVE_PRODUCT").split() # If this has been unset then we're not scanning for CVEs here (for example, image recipes) - if not bpn: + if not products: return ([], []) pv = d.getVar("CVE_VERSION").split("+git")[0] - cves = " ".join(patched_cves) - cve_db_dir = d.getVar("CVE_CHECK_DB_DIR") - cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST")) - cve_cmd = "cve-check-tool" - cmd = [cve_cmd, "--no-html", "--skip-update", "--csv", "--not-affected", "-t", "faux", "-d", cve_db_dir] # If the recipe has been whitlisted we return empty lists if d.getVar("PN") in d.getVar("CVE_CHECK_PN_WHITELIST").split(): bb.note("Recipe has been whitelisted, skipping check") return ([], []) - try: - # Write the faux CSV file to be used with cve-check-tool - fd, faux = tempfile.mkstemp(prefix="cve-faux-") - with os.fdopen(fd, "w") as f: - for pn in bpn.split(): - f.write("%s,%s,%s,\n" % (pn, pv, cves)) - cmd.append(faux) - - output = subprocess.check_output(cmd).decode("utf-8") - bb.debug(2, "Output of command %s:\n%s" % ("\n".join(cmd), output)) - except subprocess.CalledProcessError as e: - bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output)) - finally: - os.remove(faux) - - for row in csv.reader(io.StringIO(output)): - # Third row has the unpatched CVEs - if row[2]: - for cve in row[2].split(): - # Skip if the CVE has been whitlisted for the current version - if pv in cve_whitelist.get(cve,[]): - bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve)) + old_cve_whitelist = d.getVar("CVE_CHECK_CVE_WHITELIST") + if old_cve_whitelist: + bb.warn("CVE_CHECK_CVE_WHITELIST is deprecated, please use CVE_CHECK_WHITELIST.") + cve_whitelist = d.getVar("CVE_CHECK_WHITELIST").split() + + import sqlite3 + db_file = d.getVar("CVE_CHECK_DB_FILE") + conn = sqlite3.connect(db_file) + + for product in products: + c = conn.cursor() + if ":" in product: + vendor, product = product.split(":", 1) + c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR IS ?", (product, vendor)) + else: + c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,)) + + for row in c: + cve = row[0] + version_start = row[3] + operator_start = row[4] + version_end = row[5] + operator_end = row[6] + + if cve in cve_whitelist: + bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) + elif cve in patched_cves: + bb.note("%s has been patched" % (cve)) + else: + to_append = False + if (operator_start == '=' and pv == version_start): + cves_unpatched.append(cve) else: + if operator_start: + try: + to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) + to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) + except: + bb.note("%s: Failed to compare %s %s %s for %s" % + (product, pv, operator_start, version_start, cve)) + to_append_start = False + else: + to_append_start = False + + if operator_end: + try: + to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) + to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) + except: + bb.note("%s: Failed to compare %s %s %s for %s" % + (product, pv, operator_end, version_end, cve)) + to_append_end = False + else: + to_append_end = False + + if operator_start and operator_end: + to_append = to_append_start and to_append_end + else: + to_append = to_append_start or to_append_end + + if to_append: cves_unpatched.append(cve) - bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve)) - # Fourth row has patched CVEs - if row[3]: - for cve in row[3].split(): - cves_patched.append(cve) - bb.debug(2, "%s-%s is patched for %s" % (bpn, pv, cve)) + bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve)) + conn.close() - return (cves_patched, cves_unpatched) + return (list(patched_cves), cves_unpatched) def get_cve_info(d, cves): """ - Get CVE information from the database used by cve-check-tool. + Get CVE information from the database. Unfortunately the only way to get CVE info is set the output to html (hard to parse) or query directly the database. @@ -241,9 +271,10 @@ def get_cve_info(d, cves): for row in cur.execute(query, tuple(cves)): cve_data[row[0]] = {} cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["score"] = row[2] - cve_data[row[0]]["modified"] = row[3] - cve_data[row[0]]["vector"] = row[4] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] conn.close() return cve_data @@ -270,7 +301,8 @@ def cve_write_data(d, patched, unpatched, cve_data): unpatched_cves.append(cve) write_string += "CVE STATUS: Unpatched\n" write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"] - write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"] + write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["scorev2"] + write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["scorev3"] write_string += "VECTOR: %s\n" % cve_data[cve]["vector"] write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve) diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc index 672f067..c027901 100644 --- a/meta/conf/distro/include/maintainers.inc +++ b/meta/conf/distro/include/maintainers.inc @@ -116,6 +116,7 @@ RECIPE_MAINTAINER_pn-cryptodev-tests = "Robert Yang " RECIPE_MAINTAINER_pn-cups = "Chen Qi " RECIPE_MAINTAINER_pn-curl = "Armin Kuster " RECIPE_MAINTAINER_pn-cve-check-tool = "Ross Burton " +RECIPE_MAINTAINER_pn-cve-update-db-native = "Ross Burton " RECIPE_MAINTAINER_pn-cwautomacros = "Ross Burton " RECIPE_MAINTAINER_pn-db = "Mark Hatle " RECIPE_MAINTAINER_pn-dbus = "Chen Qi " diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc index 1b676dc..97d83cb 100644 --- a/meta/recipes-core/glibc/glibc-locale.inc +++ b/meta/recipes-core/glibc/glibc-locale.inc @@ -95,3 +95,6 @@ do_install () { inherit libc-package BBCLASSEXTEND = "nativesdk" + +# Don't scan for CVEs as glibc will be scanned +CVE_PRODUCT = "" diff --git a/meta/recipes-core/glibc/glibc-mtrace.inc b/meta/recipes-core/glibc/glibc-mtrace.inc index d703c14..ef9d60e 100644 --- a/meta/recipes-core/glibc/glibc-mtrace.inc +++ b/meta/recipes-core/glibc/glibc-mtrace.inc @@ -11,3 +11,6 @@ do_install() { install -d -m 0755 ${D}${bindir} install -m 0755 ${SRC}/mtrace ${D}${bindir}/ } + +# Don't scan for CVEs as glibc will be scanned +CVE_PRODUCT = "" diff --git a/meta/recipes-core/glibc/glibc-scripts.inc b/meta/recipes-core/glibc/glibc-scripts.inc index 2a2b415..14a14e4 100644 --- a/meta/recipes-core/glibc/glibc-scripts.inc +++ b/meta/recipes-core/glibc/glibc-scripts.inc @@ -18,3 +18,6 @@ do_install() { # sotruss script requires sotruss-lib.so (given by libsotruss package), # to produce trace of the library calls. RDEPENDS_${PN} += "libsotruss" + +# Don't scan for CVEs as glibc will be scanned +CVE_PRODUCT = "" diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb new file mode 100644 index 0000000..2c427a5 --- /dev/null +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -0,0 +1,195 @@ +SUMMARY = "Updates the NVD CVE database" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +inherit native + +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install +deltask do_populate_sysroot + +python () { + if not d.getVar("CVE_CHECK_DB_FILE"): + raise bb.parse.SkipRecipe("Skip recipe when cve-check class is not loaded.") +} + +python do_populate_cve_db() { + """ + Update NVD database with json data feed + """ + + import sqlite3, urllib, urllib.parse, shutil, gzip + from datetime import date + + BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" + YEAR_START = 2002 + + db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK') + db_file = os.path.join(db_dir, 'nvdcve_1.0.db') + json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') + proxy = d.getVar("https_proxy") + + if proxy: + # instantiate an opener but do not install it as the global + # opener unless if we're really sure it's applicable for all + # urllib requests + proxy_handler = urllib.request.ProxyHandler({'https': proxy}) + proxy_opener = urllib.request.build_opener(proxy_handler) + else: + proxy_opener = None + + cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') + + if not os.path.isdir(db_dir): + os.mkdir(db_dir) + + # Connect to database + conn = sqlite3.connect(db_file) + c = conn.cursor() + + initialize_db(c) + + for year in range(YEAR_START, date.today().year + 1): + year_url = BASE_URL + str(year) + meta_url = year_url + ".meta" + json_url = year_url + ".json.gz" + + # Retrieve meta last modified date + + response = None + + if proxy_opener: + response = proxy_opener.open(meta_url) + else: + req = urllib.request.Request(meta_url) + response = urllib.request.urlopen(req) + + if response: + for l in response.read().decode("utf-8").splitlines(): + key, value = l.split(":", 1) + if key == "lastModifiedDate": + last_modified = value + break + else: + bb.warn("Cannot parse CVE metadata, update failed") + return + + # Compare with current db last modified date + c.execute("select DATE from META where YEAR = ?", (year,)) + meta = c.fetchone() + if not meta or meta[0] != last_modified: + # Clear products table entries corresponding to current year + c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,)) + + # Update db with current year json file + try: + if proxy_opener: + response = proxy_opener.open(json_url) + else: + req = urllib.request.Request(json_url) + response = urllib.request.urlopen(req) + + if response: + update_db(c, gzip.decompress(response.read()).decode('utf-8')) + c.execute("insert or replace into META values (?, ?)", [year, last_modified]) + except urllib.error.URLError as e: + cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') + bb.warn("Cannot parse CVE data (%s), update failed" % e.reason) + return + + # Update success, set the date to cve_check file. + if year == date.today().year: + cve_f.write('CVE database update : %s\n\n' % date.today()) + + cve_f.close() + conn.commit() + conn.close() +} + +def initialize_db(c): + c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ + SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \ + VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ + VERSION_END TEXT, OPERATOR_END TEXT)") + +def parse_node_and_insert(c, node, cveId): + # Parse children node if needed + for child in node.get('children', ()): + parse_node_and_insert(c, child, cveId) + + def cpe_generator(): + for cpe in node.get('cpe_match', ()): + if not cpe['vulnerable']: + return + cpe23 = cpe['cpe23Uri'].split(':') + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + + if version != '*': + # Version is defined, this is a '=' match + yield [cveId, vendor, product, version, '=', '', ''] + else: + # Parse start version, end version and operators + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + if 'versionStartIncluding' in cpe: + op_start = '>=' + v_start = cpe['versionStartIncluding'] + + if 'versionStartExcluding' in cpe: + op_start = '>' + v_start = cpe['versionStartExcluding'] + + if 'versionEndIncluding' in cpe: + op_end = '<=' + v_end = cpe['versionEndIncluding'] + + if 'versionEndExcluding' in cpe: + op_end = '<' + v_end = cpe['versionEndExcluding'] + + yield [cveId, vendor, product, v_start, op_start, v_end, op_end] + + c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) + +def update_db(c, jsondata): + import json + root = json.loads(jsondata) + + for elt in root['CVE_Items']: + if not elt['impact']: + continue + + cveId = elt['cve']['CVE_data_meta']['ID'] + cveDesc = elt['cve']['description']['description_data'][0]['value'] + date = elt['lastModifiedDate'] + accessVector = elt['impact']['baseMetricV2']['cvssV2']['accessVector'] + cvssv2 = elt['impact']['baseMetricV2']['cvssV2']['baseScore'] + + try: + cvssv3 = elt['impact']['baseMetricV3']['cvssV3']['baseScore'] + except: + cvssv3 = 0.0 + + c.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)", + [cveId, cveDesc, cvssv2, cvssv3, date, accessVector]) + + configurations = elt['configurations']['nodes'] + for config in configurations: + parse_node_and_insert(c, config, cveId) + + +addtask do_populate_cve_db before do_fetch +do_populate_cve_db[nostamp] = "1" + +EXCLUDE_FROM_WORLD = "1" diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb deleted file mode 100644 index 1c84fb1..0000000 --- a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb +++ /dev/null @@ -1,62 +0,0 @@ -SUMMARY = "cve-check-tool" -DESCRIPTION = "cve-check-tool is a tool for checking known (public) CVEs.\ -The tool will identify potentially vunlnerable software packages within Linux distributions through version matching." -HOMEPAGE = "https://github.com/ikeydoherty/cve-check-tool" -SECTION = "Development/Tools" -LICENSE = "GPL-2.0+" -LIC_FILES_CHKSUM = "file://LICENSE;md5=e8c1458438ead3c34974bc0be3a03ed6" - -SRC_URI = "https://github.com/ikeydoherty/${BPN}/releases/download/v${PV}/${BP}.tar.xz \ - file://check-for-malloc_trim-before-using-it.patch \ - file://0001-print-progress-in-percent-when-downloading-CVE-db.patch \ - file://0001-curl-allow-overriding-default-CA-certificate-file.patch \ - file://0001-update-Compare-computed-vs-expected-sha256-digit-str.patch \ - file://0001-Fix-freeing-memory-allocated-by-sqlite.patch \ - " - -SRC_URI[md5sum] = "c5f4247140fc9be3bf41491d31a34155" -SRC_URI[sha256sum] = "b8f283be718af8d31232ac1bfc10a0378fb958aaaa49af39168f8acf501e6a5b" - -UPSTREAM_CHECK_URI = "https://github.com/ikeydoherty/cve-check-tool/releases" - -DEPENDS = "libcheck glib-2.0 json-glib curl libxml2 sqlite3 openssl ca-certificates" - -RDEPENDS_${PN} = "ca-certificates" - -inherit pkgconfig autotools - -EXTRA_OECONF = "--disable-coverage --enable-relative-plugins" -CFLAGS_append = " -Wno-error=pedantic" - -do_populate_cve_db() { - if [ "${BB_NO_NETWORK}" = "1" ] ; then - bbwarn "BB_NO_NETWORK is set; Can't update cve-check-tool database, new CVEs won't be detected" - return - fi - - # In case we don't inherit cve-check class, use default values defined in the class. - cve_dir="${CVE_CHECK_DB_DIR}" - cve_file="${CVE_CHECK_TMP_FILE}" - - [ -z "${cve_dir}" ] && cve_dir="${DL_DIR}/CVE_CHECK" - [ -z "${cve_file}" ] && cve_file="${TMPDIR}/cve_check" - - unused="${@bb.utils.export_proxies(d)}" - bbdebug 2 "Updating cve-check-tool database located in $cve_dir" - # --cacert works around curl-native not finding the CA bundle - if cve-check-update --cacert ${sysconfdir}/ssl/certs/ca-certificates.crt -d "$cve_dir" ; then - printf "CVE database was updated on %s UTC\n\n" "$(LANG=C date --utc +'%F %T')" > "$cve_file" - else - bbwarn "Error in executing cve-check-update" - if [ "${@'1' if bb.data.inherits_class('cve-check', d) else '0'}" -ne 0 ] ; then - bbwarn "Failed to update cve-check-tool database, CVEs won't be checked" - fi - fi -} - -addtask populate_cve_db after do_populate_sysroot -do_populate_cve_db[depends] = "cve-check-tool-native:do_populate_sysroot" -do_populate_cve_db[nostamp] = "1" -do_populate_cve_db[progress] = "percent" - -BBCLASSEXTEND = "native nativesdk" diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch b/meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch deleted file mode 100644 index 4a82cf2..0000000 --- a/meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch +++ /dev/null @@ -1,50 +0,0 @@ -From a3353429652f83bb8b0316500faa88fa2555542d Mon Sep 17 00:00:00 2001 -From: Peter Marko -Date: Thu, 13 Apr 2017 23:09:52 +0200 -Subject: [PATCH] Fix freeing memory allocated by sqlite - -Upstream-Status: Backport -Signed-off-by: Peter Marko ---- - src/core.c | 8 ++++---- - 1 file changed, 4 insertions(+), 4 deletions(-) - -diff --git a/src/core.c b/src/core.c -index 6263031..6788f16 100644 ---- a/src/core.c -+++ b/src/core.c -@@ -82,7 +82,7 @@ static bool ensure_table(CveDB *self) - rc = sqlite3_exec(self->db, query, NULL, NULL, &err); - if (rc != SQLITE_OK) { - fprintf(stderr, "ensure_table(): %s\n", err); -- free(err); -+ sqlite3_free(err); - return false; - } - -@@ -91,7 +91,7 @@ static bool ensure_table(CveDB *self) - rc = sqlite3_exec(self->db, query, NULL, NULL, &err); - if (rc != SQLITE_OK) { - fprintf(stderr, "ensure_table(): %s\n", err); -- free(err); -+ sqlite3_free(err); - return false; - } - -@@ -99,11 +99,11 @@ static bool ensure_table(CveDB *self) - rc = sqlite3_exec(self->db, query, NULL, NULL, &err); - if (rc != SQLITE_OK) { - fprintf(stderr, "ensure_table(): %s\n", err); -- free(err); -+ sqlite3_free(err); - return false; - } - if (err) { -- free(err); -+ sqlite3_free(err); - } - - return true; --- -2.1.4 - diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch b/meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch deleted file mode 100644 index 3d8ebd1..0000000 --- a/meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch +++ /dev/null @@ -1,215 +0,0 @@ -From 825a9969dea052b02ba868bdf39e676349f10dce Mon Sep 17 00:00:00 2001 -From: Jussi Kukkonen -Date: Thu, 9 Feb 2017 14:51:28 +0200 -Subject: [PATCH] curl: allow overriding default CA certificate file - -Similar to curl, --cacert can now be used in cve-check-tool and -cve-check-update to override the default CA certificate file. Useful -in cases where the system default is unsuitable (for example, -out-dated) or broken (as in OE's current native libcurl, which embeds -a path string from one build host and then uses it on another although -the right path may have become something different). - -Upstream-Status: Submitted [https://github.com/ikeydoherty/cve-check-tool/pull/45] - -Signed-off-by: Patrick Ohly - - -Took Patrick Ohlys original patch from meta-security-isafw, rebased -on top of other patches. - -Signed-off-by: Jussi Kukkonen ---- - src/library/cve-check-tool.h | 1 + - src/library/fetch.c | 10 +++++++++- - src/library/fetch.h | 3 ++- - src/main.c | 5 ++++- - src/update-main.c | 4 +++- - src/update.c | 12 +++++++----- - src/update.h | 2 +- - 7 files changed, 27 insertions(+), 10 deletions(-) - -diff --git a/src/library/cve-check-tool.h b/src/library/cve-check-tool.h -index e4bb5b1..f89eade 100644 ---- a/src/library/cve-check-tool.h -+++ b/src/library/cve-check-tool.h -@@ -43,6 +43,7 @@ typedef struct CveCheckTool { - bool bugs; /**output_file = output_file; -+ self->cacert_file = cacert_file; - - if (!csv_mode && self->output_file) { - quiet = false; -@@ -530,7 +533,7 @@ int main(int argc, char **argv) - if (status) { - fprintf(stderr, "Update of db forced\n"); - cve_db_unlock(); -- if (!update_db(quiet, db_path->str)) { -+ if (!update_db(quiet, db_path->str, self->cacert_file)) { - fprintf(stderr, "DB update failure\n"); - goto cleanup; - } -diff --git a/src/update-main.c b/src/update-main.c -index 2379cfa..c52d9d0 100644 ---- a/src/update-main.c -+++ b/src/update-main.c -@@ -43,11 +43,13 @@ the Free Software Foundation; either version 2 of the License, or\n\ - static gchar *nvds = NULL; - static bool _show_version = false; - static bool _quiet = false; -+static const char *_cacert_file = NULL; - - static GOptionEntry _entries[] = { - { "nvd-dir", 'd', 0, G_OPTION_ARG_STRING, &nvds, "NVD directory in filesystem", NULL }, - { "version", 'v', 0, G_OPTION_ARG_NONE, &_show_version, "Show version", NULL }, - { "quiet", 'q', 0, G_OPTION_ARG_NONE, &_quiet, "Run silently", NULL }, -+ { "cacert", 'C', 0, G_OPTION_ARG_STRING, &_cacert_file, "Path to the combined SSL certificates file (system default is used if not set)", NULL}, - { .short_name = 0 } - }; - -@@ -88,7 +90,7 @@ int main(int argc, char **argv) - goto end; - } - -- if (update_db(_quiet, db_path->str)) { -+ if (update_db(_quiet, db_path->str, _cacert_file)) { - ret = EXIT_SUCCESS; - } else { - fprintf(stderr, "Failed to update database\n"); -diff --git a/src/update.c b/src/update.c -index 070560a..8cb4a39 100644 ---- a/src/update.c -+++ b/src/update.c -@@ -267,7 +267,8 @@ static inline void update_end(int fd, const char *update_fname, bool ok) - - static int do_fetch_update(int year, const char *db_dir, CveDB *cve_db, - bool db_exist, bool verbose, -- unsigned int this_percent, unsigned int next_percent) -+ unsigned int this_percent, unsigned int next_percent, -+ const char *cacert_file) - { - const char nvd_uri[] = URI_PREFIX; - autofree(cve_string) *uri_meta = NULL; -@@ -331,14 +332,14 @@ refetch: - } - - /* Fetch NVD META file */ -- st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose, this_percent, this_percent); -+ st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose, this_percent, this_percent, cacert_file); - if (st == FETCH_STATUS_FAIL) { - fprintf(stderr, "Failed to fetch %s\n", uri_meta->str); - return -1; - } - - /* Fetch NVD XML file */ -- st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose, this_percent, next_percent); -+ st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose, this_percent, next_percent, cacert_file); - switch (st) { - case FETCH_STATUS_FAIL: - fprintf(stderr, "Failed to fetch %s\n", uri_data_gz->str); -@@ -391,7 +392,7 @@ refetch: - return 0; - } - --bool update_db(bool quiet, const char *db_file) -+bool update_db(bool quiet, const char *db_file, const char *cacert_file) - { - autofree(char) *db_dir = NULL; - autofree(CveDB) *cve_db = NULL; -@@ -466,7 +467,8 @@ bool update_db(bool quiet, const char *db_file) - if (!quiet) - fprintf(stderr, "completed: %u%%\r", start_percent); - rc = do_fetch_update(y, db_dir, cve_db, db_exist, !quiet, -- start_percent, end_percent); -+ start_percent, end_percent, -+ cacert_file); - switch (rc) { - case 0: - if (!quiet) -diff --git a/src/update.h b/src/update.h -index b8e9911..ceea0c3 100644 ---- a/src/update.h -+++ b/src/update.h -@@ -15,7 +15,7 @@ cve_string *get_db_path(const char *path); - - int update_required(const char *db_file); - --bool update_db(bool quiet, const char *db_file); -+bool update_db(bool quiet, const char *db_file, const char *cacert_file); - - - /* --- -2.1.4 - diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch b/meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch deleted file mode 100644 index 8ea6f68..0000000 --- a/meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch +++ /dev/null @@ -1,135 +0,0 @@ -From e9ed26cde63f8ca7607a010a518329339f8c02d3 Mon Sep 17 00:00:00 2001 -From: =?UTF-8?q?Andr=C3=A9=20Draszik?= -Date: Mon, 26 Sep 2016 12:12:41 +0100 -Subject: [PATCH] print progress in percent when downloading CVE db -MIME-Version: 1.0 -Content-Type: text/plain; charset=UTF-8 -Content-Transfer-Encoding: 8bit - -Upstream-Status: Pending -Signed-off-by: André Draszik ---- - src/library/fetch.c | 28 +++++++++++++++++++++++++++- - src/library/fetch.h | 3 ++- - src/update.c | 16 ++++++++++++---- - 3 files changed, 41 insertions(+), 6 deletions(-) - -diff --git a/src/library/fetch.c b/src/library/fetch.c -index 06d4b30..0fe6d76 100644 ---- a/src/library/fetch.c -+++ b/src/library/fetch.c -@@ -37,13 +37,37 @@ static size_t write_func(void *ptr, size_t size, size_t nmemb, struct fetch_t *f - return fwrite(ptr, size, nmemb, f->f); - } - --FetchStatus fetch_uri(const char *uri, const char *target, bool verbose) -+struct percent_t { -+ unsigned int start; -+ unsigned int end; -+}; -+ -+static int progress_callback_new(void *ptr, curl_off_t dltotal, curl_off_t dlnow, curl_off_t ultotal, curl_off_t ulnow) -+{ -+ (void) ultotal; -+ (void) ulnow; -+ -+ struct percent_t *percent = (struct percent_t *) ptr; -+ -+ if (dltotal && percent && percent->end >= percent->start) { -+ unsigned int diff = percent->end - percent->start; -+ if (diff) { -+ fprintf(stderr,"completed: %"CURL_FORMAT_CURL_OFF_T"%%\r", percent->start + (diff * dlnow / dltotal)); -+ } -+ } -+ -+ return 0; -+} -+ -+FetchStatus fetch_uri(const char *uri, const char *target, bool verbose, -+ unsigned int start_percent, unsigned int end_percent) - { - FetchStatus ret = FETCH_STATUS_FAIL; - CURLcode res; - struct stat st; - CURL *curl = NULL; - struct fetch_t *f = NULL; -+ struct percent_t percent = { .start = start_percent, .end = end_percent }; - - curl = curl_easy_init(); - if (!curl) { -@@ -67,6 +91,8 @@ FetchStatus fetch_uri(const char *uri, const char *target, bool verbose) - } - if (verbose) { - (void)curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L); -+ (void)curl_easy_setopt(curl, CURLOPT_XFERINFODATA, &percent); -+ (void)curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progress_callback_new); - } - res = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, (curl_write_callback)write_func); - if (res != CURLE_OK) { -diff --git a/src/library/fetch.h b/src/library/fetch.h -index 70c3779..4cce5d1 100644 ---- a/src/library/fetch.h -+++ b/src/library/fetch.h -@@ -28,7 +28,8 @@ typedef enum { - * @param verbose Whether to be verbose - * @return A FetchStatus, indicating the operation taken - */ --FetchStatus fetch_uri(const char *uri, const char *target, bool verbose); -+FetchStatus fetch_uri(const char *uri, const char *target, bool verbose, -+ unsigned int this_percent, unsigned int next_percent); - - /** - * Attempt to extract the given gzipped file -diff --git a/src/update.c b/src/update.c -index 30fbe96..eaeeefd 100644 ---- a/src/update.c -+++ b/src/update.c -@@ -266,7 +266,8 @@ static inline void update_end(int fd, const char *update_fname, bool ok) - } - - static int do_fetch_update(int year, const char *db_dir, CveDB *cve_db, -- bool db_exist, bool verbose) -+ bool db_exist, bool verbose, -+ unsigned int this_percent, unsigned int next_percent) - { - const char nvd_uri[] = URI_PREFIX; - autofree(cve_string) *uri_meta = NULL; -@@ -330,14 +331,14 @@ refetch: - } - - /* Fetch NVD META file */ -- st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose); -+ st = fetch_uri(uri_meta->str, nvdcve_meta->str, verbose, this_percent, this_percent); - if (st == FETCH_STATUS_FAIL) { - fprintf(stderr, "Failed to fetch %s\n", uri_meta->str); - return -1; - } - - /* Fetch NVD XML file */ -- st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose); -+ st = fetch_uri(uri_data_gz->str, nvdcve_data_gz->str, verbose, this_percent, next_percent); - switch (st) { - case FETCH_STATUS_FAIL: - fprintf(stderr, "Failed to fetch %s\n", uri_data_gz->str); -@@ -459,10 +460,17 @@ bool update_db(bool quiet, const char *db_file) - for (int i = YEAR_START; i <= year+1; i++) { - int y = i > year ? -1 : i; - int rc; -+ unsigned int start_percent = ((i+0 - YEAR_START) * 100) / (year+2 - YEAR_START); -+ unsigned int end_percent = ((i+1 - YEAR_START) * 100) / (year+2 - YEAR_START); - -- rc = do_fetch_update(y, db_dir, cve_db, db_exist, !quiet); -+ if (!quiet) -+ fprintf(stderr, "completed: %u%%\r", start_percent); -+ rc = do_fetch_update(y, db_dir, cve_db, db_exist, !quiet, -+ start_percent, end_percent); - switch (rc) { - case 0: -+ if (!quiet) -+ fprintf(stderr,"completed: %u%%\r", end_percent); - continue; - case ENOMEM: - goto oom; --- -2.9.3 - diff --git a/meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch b/meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch deleted file mode 100644 index 458c0cc..0000000 --- a/meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch +++ /dev/null @@ -1,52 +0,0 @@ -From b0426e63c9ac61657e029f689bcb8dd051e752c6 Mon Sep 17 00:00:00 2001 -From: Sergey Popovich -Date: Fri, 21 Apr 2017 07:32:23 -0700 -Subject: [PATCH] update: Compare computed vs expected sha256 digit string - ignoring case - -We produce sha256 digest string using %x snprintf() -qualifier for each byte of digest which uses alphabetic -characters from "a" to "f" in lower case to represent -integer values from 10 to 15. - -Previously all of the NVD META files supply sha256 -digest string for corresponding XML file in lower case. - -However due to some reason this changed recently to -provide digest digits in upper case causing fetched -data consistency checks to fail. This prevents database -from being updated periodically. - -While commit c4f6e94 (update: Do not treat sha256 failure -as fatal if requested) adds useful option to skip -digest validation at all and thus provides workaround for -this situation, it might be unacceptable for some -deployments where we need to ensure that downloaded -data is consistent before start parsing it and update -SQLite database. - -Use strcasecmp() to compare two digest strings case -insensitively and addressing this case. - -Upstream-Status: Backport -Signed-off-by: Sergey Popovich ---- - src/update.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -diff --git a/src/update.c b/src/update.c -index 8588f38..3cc6b67 100644 ---- a/src/update.c -+++ b/src/update.c -@@ -187,7 +187,7 @@ static bool nvdcve_data_ok(const char *meta, const char *data) - snprintf(&csum_data[idx], len, "%02hhx", digest[i]); - } - -- ret = streq(csum_meta, csum_data); -+ ret = !strcasecmp(csum_meta, csum_data); - - err_unmap: - munmap(buffer, length); --- -2.11.0 - diff --git a/meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch b/meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch deleted file mode 100644 index 0774ad9..0000000 --- a/meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch +++ /dev/null @@ -1,51 +0,0 @@ -From ce64633b9733e962b8d8482244301f614d8b5845 Mon Sep 17 00:00:00 2001 -From: Khem Raj -Date: Mon, 22 Aug 2016 22:54:24 -0700 -Subject: [PATCH] Check for malloc_trim before using it - -malloc_trim is gnu specific and not all libc -implement it, threfore write a configure check -to poke for it first and use the define to -guard its use. - -Helps in compiling on musl based systems - -Signed-off-by: Khem Raj ---- -Upstream-Status: Submitted [https://github.com/ikeydoherty/cve-check-tool/pull/48] - configure.ac | 2 ++ - src/core.c | 4 ++-- - 2 files changed, 4 insertions(+), 2 deletions(-) - -diff --git a/configure.ac b/configure.ac -index d3b66ce..79c3542 100644 ---- a/configure.ac -+++ b/configure.ac -@@ -19,6 +19,8 @@ m4_define([json_required_version], [0.16.0]) - m4_define([openssl_required_version],[1.0.0]) - # TODO: Set minimum sqlite - -+AC_CHECK_FUNCS_ONCE(malloc_trim) -+ - PKG_CHECK_MODULES(CVE_CHECK_TOOL, - [ - glib-2.0 >= glib_required_version, -diff --git a/src/core.c b/src/core.c -index 6263031..0d5df29 100644 ---- a/src/core.c -+++ b/src/core.c -@@ -498,9 +498,9 @@ bool cve_db_load(CveDB *self, const char *fname) - } - - b = true; -- -+#ifdef HAVE_MALLOC_TRIM - malloc_trim(0); -- -+#endif - xmlFreeTextReader(r); - if (fd) { - close(fd); --- -2.9.3 - From patchwork Mon Dec 16 15:59:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181739 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4480704och; Mon, 16 Dec 2019 08:01:23 -0800 (PST) X-Google-Smtp-Source: APXvYqwHaicEglzOAwqBX1ROzXZWD5kWraHrewe1MqhTgROedjONVhyGxGVKonYP50+50OudldlX X-Received: by 2002:a17:90a:1f8e:: with SMTP id x14mr19021626pja.29.1576512083247; Mon, 16 Dec 2019 08:01:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512083; cv=none; d=google.com; s=arc-20160816; b=v6xcPNMrUcfTeZb06CsCjZTir9RW7JPIpx1N2NyRc/tnB6xr9x7LvKt91XdA61aiKX INCzL5rZOyomZRI2E9BerJGDvvr/sepoeHytHDY0zSUV06lx3DAWLkbb91y0w4wOlEh+ 50/zK19zkbCz0bBVtyb2l2UMynvr5yo05FHbd+iwUfl5XvFZX805I+akox6ZaSmGHJZG I+Z8hDcPxr6AwzMdHCMuAiIJRWtVLxxMWYieJ7iAYhwRgEqXGe21kJMRMqiW3CwbdIuS cP56gSZlR9DU00tkx59lN8tr5uMifEuqTi2feUSPdGIlNNEunrQ/kPlzkC3Ttc7bVCdR orFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=5L+GrkzClyiHj+bLkG+2efA08/BmRGtmrh7VgNJeJCI=; b=weLvrdeFkFDU0v7mEarMpR6Cq32u7CdqkfQUoJeYpBcf4SuQKvHoR/tvUTJpxzsW9h 6w7N+8kCxf7vfu6D2A9uizwHo5xKi7bR8oKn5SMl6HHKyq9UnJkHweNwWxITN2lZY/Aq gieiAI/HBAU5yLFcde1RKcPo9o2PWPMELyMF6qlGffnbcMoyCH1RgLwaLjzBFSdMtl8G xrckIf8IIzHnGn1qJUV8vps0/UWTfyNde/ie6IP7GE8KvRZgNqb3coEWpR/vBE6/AycR h+ZdHmBG1xiJnY7APDJpknMa4WRRm8q75NRWOcqxMv1OxWm/K+9rN8lyl97YFF2ey9Am zGaw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=uMxwr0zA; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id f11si15933510pjw.63.2019.12.16.08.01.22; Mon, 16 Dec 2019 08:01:23 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=uMxwr0zA; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 0AD557FDF3; Mon, 16 Dec 2019 16:00:40 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by mail.openembedded.org (Postfix) with ESMTP id 8EAC67FDA1 for ; Mon, 16 Dec 2019 16:00:17 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id z3so3124414plk.6 for ; Mon, 16 Dec 2019 08:00:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=oLIjUHhuFis3lCiRnTDn8qETvmM4p4Cs7ZdatrBDJNk=; b=uMxwr0zAntfp5NrbbSQHnYCfDgzhpy50sQMWMnJMasFtbO3NkLTZL7q2qme1WAiU+8 BKmjy2degXTTGlTR+ch1V8RscZkOJJ2Kiwh0yJjYjs6hc+tc80T7x6CVJMsOlW9XH71N ZbVxwP4lBd631VwZ2hNoA2He9t+S2l+JYjZZqtvnuzPx1C9hORje/iHZ7xQkigvvGffW kBpEt1sawyWjeaA4q7J+eljgT0U0NyPPB23fsAg5hXDb4GdK9J/TaqC0H7y5kU9Jyrnd ztugsrmP3I/kgwQFxG+GzSi9YiRG/eO401zdoFQCGFNqRiCByvwzG+ue36Et7Z+31juP SyMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=oLIjUHhuFis3lCiRnTDn8qETvmM4p4Cs7ZdatrBDJNk=; b=GsOPT/Tzv0oTtesXnrGfZFQ9GZGvCvEgOSJJZxMoX+3CK1E+0/BFLdF5zj0XBahU3p 1ukqdGEtiO27sgXzOEcpE2hjseecDm7a/VTyu4oHHrK473D2zZQDCPIrpAYrcrCSWQdR W+Xe++y+i9eHIPP0E9oBP3iYbcvObsgQOfir+JEHNYWYfpWGE8U0z2NYr3H1WnzNxwa9 fIYe50eUB7lSPr/FQzcuzBZP3QJzUxVND4DjldCbcLuc6cRNoDdwBwsoSLtYJdJF3JL/ 6NPhUYDNh2SpvFQIV6ezLY9OsRpsitSvDthAP3eIhB6NkXxOS2BXa9RJjtv96FSjJ9SL QZXA== X-Gm-Message-State: APjAAAXfY9rM4hpPD7+/LC6O5V1ZbWkeSqFmzP72dRz3mTUIRtQw4yEp 5Y0hFLE4o6A6kOfQzJ+zDMkkUW36hX8= X-Received: by 2002:a17:902:724a:: with SMTP id c10mr17090489pll.39.1576512018530; Mon, 16 Dec 2019 08:00:18 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:18 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 07:59:57 -0800 Message-Id: <9d01a64844998d98fcfcebbe8580422094cd2dde.1576511913.git.akuster808@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 07/18] cve-check: ensure all known CVEs are in the report X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton CVEs that are whitelisted or were not vulnerable when there are version comparisons were not included in the report, so alter the logic to ensure that all relevant CVEs are in the report for completeness. (From OE-Core rev: 98256ff05fcfe9d5ccad360582c36eafb577c264) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/classes/cve-check.bbclass | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index c00d291..f87bcc9 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -208,12 +208,14 @@ def check_cves(d, patched_cves): if cve in cve_whitelist: bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) + # TODO: this should be in the report as 'whitelisted' + patched_cves.add(cve) elif cve in patched_cves: bb.note("%s has been patched" % (cve)) else: to_append = False if (operator_start == '=' and pv == version_start): - cves_unpatched.append(cve) + to_append = True else: if operator_start: try: @@ -243,8 +245,11 @@ def check_cves(d, patched_cves): to_append = to_append_start or to_append_end if to_append: + bb.note("%s-%s is vulnerable to %s" % (product, pv, cve)) cves_unpatched.append(cve) - bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve)) + else: + bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) + patched_cves.add(cve) conn.close() return (list(patched_cves), cves_unpatched) From patchwork Mon Dec 16 15:59:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181741 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4481149och; Mon, 16 Dec 2019 08:01:42 -0800 (PST) X-Google-Smtp-Source: APXvYqw96P1d4N2QB61GIcE5O52ch5ptjKqyfUfSIHJfMRFQcFb0cHAb5iBzpG7eDrMWHN0xwwM7 X-Received: by 2002:a63:5657:: with SMTP id g23mr18841466pgm.452.1576512102502; Mon, 16 Dec 2019 08:01:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512102; cv=none; d=google.com; s=arc-20160816; b=uYk/Er9FKJ4PoClardfejLph2nLM60i5Kj3cx9Wgi0CG2P/VJk2r1fFjOLC18SGwik wCMjKikAlzzuykm1RetKcXK9N1esm5aMg8j7kg8JPfJNadlNZ+Gw1JyJ8ka2QA2onEc9 srzyaUeZxBQZZaqYtC9Wyucrgqy5t8aM/Wuz+QgpTZEwR32R9Dho4tM9PKW0SN/Edsi8 zZUnIKnf/G+cMUsI4BF0Wy+usG6A7RWvbT/d8ki38NDBrq41PfC2SWRIF5Zdf4AWSrJU Q8NIdC6voQcuuIFjAo8G6x/QEcGYvEtj53qAvDkwzzuFMnVMHgHTmF8YQwfairQGU/XN d7oQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=+k3vK0/YpEiC5VrM+DT6D0LTL4SoyH6ksyG4X9LJPv4=; b=ihzE1XRfGatlI4fFyZoZtk+dNIdXO+k8gI3apXJWUioHQYinPxrS1BAIyiONejqaW/ 491/Zt/yD1bC/XeUPIV4bueyRv54JW6JUQpzSbGuUQf7gQN/xs7uK/bc3WKKHXovc2Dl hd7LQ8lm6E7vlvXzPkHJltOoBeLz3PXilRfoMiHi85jDU/JvJyWl+ercheDo9qpPrTul nA//jcgvrcIdNGjyFhwRKfDN70i/t+rPnKBXZ7TsDRYkb0TpYe+DJlY4GpOHut3u+moa FGoHQHs/JyPK68AmTc31rnADJOZ3R6uysDbU59DEcciWYTkeNBvo8IYiwmHdnzzzvcWy B7dA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="KjvVH+/8"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id e3si1156631pge.556.2019.12.16.08.01.42; Mon, 16 Dec 2019 08:01:42 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="KjvVH+/8"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 8FFE67FDFC; Mon, 16 Dec 2019 16:00:42 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by mail.openembedded.org (Postfix) with ESMTP id B5ADA7FDB1 for ; Mon, 16 Dec 2019 16:00:18 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id bd4so4646890plb.8 for ; Mon, 16 Dec 2019 08:00:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=pPY31qSR6r6PrioBX4Nlcy8j62eWZMbSoMjOCDh4wlA=; b=KjvVH+/8g5GbgiyekQ9veFK75y6nhqwfiI3xtk8yWqrCS+ssJeJ/oz0u8GOAYDwddZ 4Gek1qProC12O0SlbDGSYTA//X80ElaVdXIV5ibbpGWQ6Ro7qHxmkQnMjf6gUjXrFQnS e3PozakB3Hb/jy4iaH2E+l+LKScWK5y2SCzuvdtYh/sH8z4jneB9idQnn2kkKyUYfNVY liK9FdL+PznaPoblZS6Pf3usAz74hUkBeDM0sqFzJ/2yaXAYjoMlaenHkj1Nhpkmwman 0m/LD2J8WHw+IkHgQKoFWGYGVZn8Mc0RE0I4R6znGFAbZ0T8DQsoyftn9DffPrjJ66P6 LJ7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=pPY31qSR6r6PrioBX4Nlcy8j62eWZMbSoMjOCDh4wlA=; b=jBnjHkYnH4b5X5aQM6MB/T9LIHAB9IZ6Kv3yEov0Um/M0y4iqYuUuQiOyfH8ZEJrsM 5vSyY00eu7kJ2emkXa1X7PCk2s6OVtbHCFupkbYjYP8rekle6c9wiDprPmYjvFmvNVmI Aforx9z8UjN/3XBAXHw+BvjSKa+xRIbvFATYyWqOVspRRRgZbVui8lEt82SprirDUxrM edWF7IYFio0qsbH2nq7nXARjZCCaKvCA5VmVB6sO5jFElCEFXm1Pdkc6AQDuevkgmxNt qE2RmOLm9H2eUI8JUv6tfuQYvXfOLMjf3P1qpqiYCmx6qNMfirCflvIU6g20JLF5tTbu K32A== X-Gm-Message-State: APjAAAVI1oHt9QmK6aAoGGH5zVYYnUd5M3nmSRzHIdgKHsIPGUI8YT90 ra+Jge85EQouv13kWjAZpSC1brxP0kw= X-Received: by 2002:a17:902:9897:: with SMTP id s23mr16813280plp.193.1576512019520; Mon, 16 Dec 2019 08:00:19 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:19 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 07:59:58 -0800 Message-Id: <72f44bef3867295f73f8b91e17294b2876447c89.1576511913.git.akuster808@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 08/18] cve-check: failure to parse versions should be more visible X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/classes/cve-check.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index f87bcc9..1c8b222 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -222,7 +222,7 @@ def check_cves(d, patched_cves): to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) except: - bb.note("%s: Failed to compare %s %s %s for %s" % + bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_start, version_start, cve)) to_append_start = False else: @@ -233,7 +233,7 @@ def check_cves(d, patched_cves): to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) except: - bb.note("%s: Failed to compare %s %s %s for %s" % + bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_end, version_end, cve)) to_append_end = False else: From patchwork Mon Dec 16 15:59:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181742 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4481396och; Mon, 16 Dec 2019 08:01:52 -0800 (PST) X-Google-Smtp-Source: APXvYqxpDlr3tFRXT2Es/4Hc1Ojycdc3BXaVq1nlertIH+8DPTC5kIfUkaNt6LauPN/I+cEOlMT+ X-Received: by 2002:a17:90a:26e1:: with SMTP id m88mr18737028pje.101.1576512111983; Mon, 16 Dec 2019 08:01:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512111; cv=none; d=google.com; s=arc-20160816; b=KtdZZvE+ZLDhLqYzguKjTR1adib5HW1kVJMIYpddWp8SOg+QwUgP/YAfD6P8/QZpj6 BWVbFH6VmscXy5lBJMgkHlRnmhxLhYnUKLwXYOSrXQ0meLEdx0Vn1NHCjT8jTV8HZ5ei 9ZAgfuw0XmYNo5LcM1I5zZxXLH0K7830QBJoRmqxw2rC9x6NcTb3yHxOZv5y2wrG8KcX UkUMmlIycWTIKTF3Qmta3wwiYNhkE1u/5FWwLSJRIa6rh6KeVdsMcGJDk6SH9WivaYxb IpEowVyQ1mZ41+B7zG9qPIqDI26b3kST7ylkLEUL+aTCSxn6sbtns/5XjPP9Mn9mI0qL HoQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=qgzTs53dpxeh0JcPmlCNLvC92jdJZJjPYxztCoNerj4=; b=PWAQwUDjtd3zNK173lXfworG592uy+5tlSAjpZYjamrmjzPAuXx9Oad/IM/CjmA5bW nGuPUqbja1+YF20EJZBjcJfBKA2zy5ff54EmThYPUqihz1j2ZiscONdKgTMMUbcJJoo9 UMJsPEq2r7dPpd4XGCFQER7UKMkQM123h/kic964X8P9iitlREgAocCxCGwOl62n2HMY dIL2oenw5ydVMP7hRxClIVDoF5ZnbGus2ivdT13aGY+NhrPYcEY0+zhTPCDaljcrvdVP V9OdaRrenFw3rB9v6WpNTrhe3DPYN/4zFGBREjgKauKo3Hw120TqotoUthUkJ2YFkA80 6Fgw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=R658k+aB; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id o4si16562611plk.52.2019.12.16.08.01.51; Mon, 16 Dec 2019 08:01:51 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=R658k+aB; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id C8A317FE01; Mon, 16 Dec 2019 16:00:43 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by mail.openembedded.org (Postfix) with ESMTP id 7F88C7FDC3 for ; Mon, 16 Dec 2019 16:00:19 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id az3so4638631plb.11 for ; Mon, 16 Dec 2019 08:00:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=LuP2wiFsof9rpbca9j7vIN1wkcfCRmcayt1itdNjwW4=; b=R658k+aBQaJWNRgjiKw+LcqVvVMNIdxlyrNXeVPRlUU0RIv8/bPit9v0p37+26vY2n c2MsMY50ZHtdBguLshP+hWPm5zCX94WzoX9EIMje5Qo/mcuquPYqKFN0egNUIBtce/IM tjmgJLdxhKNDRqjf9R3GGp05zStbksM94HXTNJ5yhrVLgOe/dLB8lvk7bYp2idWHDrQf EwJNovO5Dmb0hkkV6ivAs/KTvA5CUDZG2RB7PkinjWhZeQsFNpWWnkuaUmmT5064WTvz F0u15CY8BsbNCP5eJbi22AXyCaEIU3bnqeoyYu1kFJkcVvZeE2uakYH73KGtcTau6Zpi MSTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=LuP2wiFsof9rpbca9j7vIN1wkcfCRmcayt1itdNjwW4=; b=N/4SYm8FhMO8xIKAd9ZaC0vPegF64SK1e467ZUYFVD+XAPL1o5uUoYoVidsuYIX6YT illxMGKZUfheUQquvI2My5ThFNfWdQMR2y9xQUUn1PmtQL787rGJgNJs7C6HuSYGJ08m lVcSmMnTHl2xaH+90mBKDSgC6eQ4I9YsWgXGOetPjabH7rhk4kxlktQ0UynIkypPWynd k1l7GpqN76Ga7cKgqDz6BMzf3ou4YKsYji1swJiR3A3SYcn4WvoV0PQDy1k2A35Neg95 2vneiQGkYX7DeNrm0ise3d8ntXTSWaQ/O5QnxEcqU7kSHIKRMjHUQ4TPb91UwfvfTfGj kJBA== X-Gm-Message-State: APjAAAUbPMq/6+6A8vbmQdsArWJCk9cMfeVyDnRNek0DP7rs//pLUG3+ a+IpZ9BR1nNzHkIDJFsii+cR9rkkksI= X-Received: by 2002:a17:902:8649:: with SMTP id y9mr16854539plt.67.1576512020546; Mon, 16 Dec 2019 08:00:20 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:19 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 07:59:59 -0800 Message-Id: X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 09/18] cve-check: we don't actually need to unpack to check X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton The patch scanner works with patch files in the layer, not in the workdir, so it doesn't need to unpack. (From OE-Core rev: 2cba6ada970deb5156e1ba0182f4f372851e3c17) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/classes/cve-check.bbclass | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 1c8b222..3326944 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -62,7 +62,7 @@ python do_cve_check () { } -addtask cve_check after do_unpack before do_build +addtask cve_check before do_build do_cve_check[depends] = "cve-update-db-native:do_populate_cve_db" do_cve_check[nostamp] = "1" @@ -70,7 +70,6 @@ python cve_check_cleanup () { """ Delete the file used to gather all the CVE information. """ - bb.utils.remove(e.data.getVar("CVE_CHECK_TMP_FILE")) } From patchwork Mon Dec 16 16:00:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181743 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4481698och; Mon, 16 Dec 2019 08:02:03 -0800 (PST) X-Google-Smtp-Source: APXvYqw+n7VcUDhZ2+NbHhtYOSsYrKhKObzPCIVyiBjeS9x40oIgWOZAe3CttB0e3ZP0nl5tVN6+ X-Received: by 2002:a63:1945:: with SMTP id 5mr19503028pgz.310.1576512123609; Mon, 16 Dec 2019 08:02:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512123; cv=none; d=google.com; s=arc-20160816; b=1I+pgvYXpq/iPs8OQ0dqUGhyI9DSiNf0hPUanY46HtbZ2DabUN/AE+ryeAS3Iy99Wi 1EBVfItZDT4a38Bt232wGTWLEPfzA7x/Trt7wfVkwTZTyJhrPhNhiNee/p+JEdk0N7Ax 514R4fIACVLA8BKTI5X4dKTYkitjbY6ww5PVKPqle2oHsaaJ02eHpOM3LqBVP3NWL6n6 /BxxpeDvx5kASZRiabFXjoAOoAIelM3o4P9EqBeeeaTl+O05nct2KQAFutHsCLynHObg j5r/gh8tWwPbHnmYZxoDKSZK/ePmf6/qRY/Y34O5W+A2h7juOilCjH1ZRyJPN4N9zEQ+ K3+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=NvTB6eScUNP3c+dDjD9qnrqPzm+bI8CLSaCxd/OSWX4=; b=VUcpfadk8Q/ALHqZSxB3LEfnmU4DtdKXcT63VmW6rZo8XI1yAT5pWtCmJVP0Ydl6B/ 9rOC5wgoi1WYxgsb2vkveHvCLTdnhzhS+cLPgGq/iYQwIEdqat4HeYh3qcQDmXR05DxR UWhr1J5XgTKU3pyVApwb2N7WhDmfbdzvpVr9IZbzUQprGQ3I5ygiavDdU3R7KwvUKyEW j4IdozvWP+zA8rIfOM/5ndzG0tibh6JaGv5LMw5v6GNQ38gIV/c5JfCuFiEI/8UWe0Lv 1FhWMEnzZyTr9uPh5ou1xmrvmq8Ou+MxZyAgWwFlwK5qKbbW4GTpn30TKCaJL2arpPvV M3XQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=bX1CmdYF; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id u19si16652821pfm.100.2019.12.16.08.02.03; Mon, 16 Dec 2019 08:02:03 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=bX1CmdYF; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 33D3C7FE09; Mon, 16 Dec 2019 16:00:45 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by mail.openembedded.org (Postfix) with ESMTP id E3B6F7FDA4 for ; Mon, 16 Dec 2019 16:00:20 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id y206so5828652pfb.0 for ; Mon, 16 Dec 2019 08:00:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=6A2GdUKIpJUnY1JrZ0fhucnm89CqvsL5MZei6drO4Sg=; b=bX1CmdYFB5ZpS/FGUzp5qRgonsDT4ozsq4ps1xSP+NX1CjJpDHMd8narQkHH6MWxd6 CJMldEyWCIGEk5U30c+aTfHzVGOl/FGZS2SZhvtT4ei/v201fiGCPV+h4GPiusbfmE1r ACKJugTxdk3cW6ZCwCbvGtlga43kbWah+xwGqNAcT86siWlG67zxCQBHvzfp6SW4yg48 Ohq8BsQZtmEylhG5UuxM0AlwtoDaL1IZkB1b6ohx+oloB7H+/hbp4VjN9X7UIAYOS/n3 6adVWOlO5HaCcSv193rPNLp6UTrrNXgbsCU7yc64KZGGKfuN52B8KT105ErtfVZ6vG1J nPGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=6A2GdUKIpJUnY1JrZ0fhucnm89CqvsL5MZei6drO4Sg=; b=O8/Y/nfiOWRoVgvPxfINnYu8dj3jUjPeOhVfOJ2LzT1sg15uH44X7xnmflot06wl2/ mmTWWNSrnZXS71XCL6NdM7kT0WuSBsC5TyxIE4j7US/2UOM11q4ycXcd90Ibd6/igoW0 6C5sLqbLGRU/xKGjJsrR/GIwWHhpXzs8S3pp1ptemHY2hlU6ZGk3i5AxnelPlSfkFQEh K/bz/sUVX6rAnh25LcpJFgBlPQGDi/FjSMDbClZnIJoBKjU9RWSzG/LC/knF8AOcvDKW fJ0/ylByJiqQowt1jg6u4O32hFeQ8S1yDqOZRXRXOm04InY7YzokUvw6GopXpvonCpCJ OT2A== X-Gm-Message-State: APjAAAX98YKTsGL+4j4Id7IWpiuzIbyN8a1HBQe2unwLoaeZJD/5Fze+ TQzQ7HC8xgCrQg9wMP0tGnhH7MP2hYU= X-Received: by 2002:a63:d442:: with SMTP id i2mr19646641pgj.349.1576512021851; Mon, 16 Dec 2019 08:00:21 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:21 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 08:00:00 -0800 Message-Id: <091a35cfbd2f3e82a7783ba9c8fd5586433ba59f.1576511913.git.akuster808@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 10/18] cve-update-db-native: don't refresh more than once an hour X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton We already fetch the yearly CVE metadata and check that for updates before downloading the full data, but we can speed up CVE checking further by only checking the CVE metadata once an hour. (From OE-Core rev: 50d898fd360c58fe85460517d965f62b7654771a) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/recipes-core/meta/cve-update-db-native.bb | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 2c427a5..19875a4 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -31,8 +31,16 @@ python do_populate_cve_db() { db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK') db_file = os.path.join(db_dir, 'nvdcve_1.0.db') json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') - proxy = d.getVar("https_proxy") + # Don't refresh the database more than once an hour + try: + import time + if time.time() - os.path.getmtime(db_file) < (60*60): + return + except OSError: + pass + + proxy = d.getVar("https_proxy") if proxy: # instantiate an opener but do not install it as the global # opener unless if we're really sure it's applicable for all From patchwork Mon Dec 16 16:00:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181744 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4481947och; Mon, 16 Dec 2019 08:02:13 -0800 (PST) X-Google-Smtp-Source: APXvYqyHMgVPPMVDruFZOGQoa8jSRssP41vvKn2hLcWCrRPDzYbd796+wo2W1Q9HID409VVhFsQv X-Received: by 2002:aa7:9567:: with SMTP id x7mr16705383pfq.133.1576512133193; Mon, 16 Dec 2019 08:02:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512133; cv=none; d=google.com; s=arc-20160816; b=ZsnRvSKMTcLyCz5EM3Phlkol5sDlx+xyxRZFtfqBfqigtALdoW9p9mfgKsKEAGajdM elEOzllIzffYrqPdhRvSbiSDbg7xg2LgtkOWGb4ICeCeCSRCtOojSCaosswqem4QoaLr QGalThRF/szlwQctqibpWJ3/iO/mF3Xo3QqqdHGnFG0nS+oBiFCZN2nsRU5youOEWNtf DoDB2zLcTKts9h+kEe1QpooMmxU0RuG2nUyuHOni3Ehxbhu/jVg7VQ66F2O89unwH3sb XLGGn24H3Rvk9bR8eJrno6XCzh1eIYy8pGh2hfbDeSc1rLX7sUqwqPsEW6jQT66CnbhX IHHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=zNdg1LtpwJO6MAOm3mXn9wOLbKRuzLKhcACd7suBqCU=; b=VVSmJh6cdjXCXMeWacvqKABk87SAYSy5UhO9gRUv5EjPSPMLGs11E2CnwJX9QlMe9Q aEdKvFfA3FW5jhMaOJID2wrW+liUi4eKfo+GJk9OuAmq8ABQbx1lGY0YFwsRmQFqTLDB Fy0IrrGURFtpE+YO+MEeiZA5LlWLrTUZ1g0sZdVdJgZ+unjnr7YxdbQ3of9Frn6Mjnop eyqnPBd3/boWnPt/x2AACSsnRfnB9+X0cV4djYaYISlKU/FU+4594M4FB14tFVwCelMI Mi16/DOVwQvLd0zx1U3aQem+syTGZl36EDoD9XX3+OjVb2TxMcCyja8/w0/RArJ8eRUg 6srQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=DqBx+MAZ; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id a18si8768202pgi.97.2019.12.16.08.02.12; Mon, 16 Dec 2019 08:02:13 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=DqBx+MAZ; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 63ED17FE0E; Mon, 16 Dec 2019 16:00:46 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by mail.openembedded.org (Postfix) with ESMTP id E84407FDA4 for ; Mon, 16 Dec 2019 16:00:21 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id ca19so3164168pjb.8 for ; Mon, 16 Dec 2019 08:00:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=905yWqdeIGCayTdsC6Nq6OoyUJiefrqmQru1RYtH1Mc=; b=DqBx+MAZzrtEIIg+h1qkYg6Zj/ra2jT/QGrU1yUiViXHXBl/IKSu3D0IhLbE5Gomiy DSKtzHHQ8VVcuIqO5PHu2SPdpBZjJNmI/sPavJE3m44N9V8FOtpCskr1iJSh7G2DrV4U cAb6xW3F6Fcu/fb5UCNPNmyWB6vyxR4QUjN3NAGZTXh/8DOXrVHhOxGFzDX/iJJsNdB+ 9B9Tk+6pcoBIldBUjx7cIBq/VbeVXosRGe8P2oAd9qLuy5HL+XwSEJnuagnELW+oQABx YzX1E2gv2Ef5MpzJtPqomC9oVDxOVMyIIi9m6k4DMP2uy8eNgB131zomjN5w1DLrQm2U CH1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=905yWqdeIGCayTdsC6Nq6OoyUJiefrqmQru1RYtH1Mc=; b=qoUD0B3i/FCbgqPJGpCTwpJh86s3AoPgYrUaOGgNAWVslCOjLeTXvJMAUYSyJJEfjs ZSk4ncJ1cCmMVO6mFsfOQsQgKGySoKVfm+WmwoXRi/HJXbmlIQj3d4X9OB7TJRv1S2fh pNbG+r/KZerMYl2LpArYcD69WlQbIUVnLmjn8MMPe55sZD8be8EDHUxeefF9E64x5SBB flvw+SjEsAumkiO2QFZj3VYUfaHfWeqH8ovHEPYo8ypXytWXY4ylNJBUKGcCH6BEbmwM /eQzJlZYC8V63r6JxSdrpHgvbiVrEpcI81Bz4alTQ+ATRKpRQ/Xkvhm9+Z6qcVBmBAYL S2Ng== X-Gm-Message-State: APjAAAWubtqp7gZDzfQHLSbUYl/KScr/7t/EVfLEpRthO4v+9KKIIhGT huvByVskhEmgDOXd5ETZ/8Nl7UU532c= X-Received: by 2002:a17:902:ff10:: with SMTP id f16mr16806117plj.312.1576512022884; Mon, 16 Dec 2019 08:00:22 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:22 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 08:00:01 -0800 Message-Id: <29cc2b5cd4bcce1c9e93395a1640014877486d7a.1576511913.git.akuster808@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 11/18] cve-update-db-native: don't hardcode the database name X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Don't hardcode the database filename, there's a variable for this in cve-check.bbclass. (From OE-Core rev: 0d188a9dc4ae64c64cd661e9d9c3841e86f226ab) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/recipes-core/meta/cve-update-db-native.bb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 19875a4..c15534d 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -28,8 +28,8 @@ python do_populate_cve_db() { BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 - db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK') - db_file = os.path.join(db_dir, 'nvdcve_1.0.db') + db_file = d.getVar("CVE_CHECK_DB_FILE") + db_dir = os.path.dirname(db_file) json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') # Don't refresh the database more than once an hour From patchwork Mon Dec 16 16:00:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181745 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4482172och; Mon, 16 Dec 2019 08:02:22 -0800 (PST) X-Google-Smtp-Source: APXvYqzjPCs8mUlk4XiwGzpjD/o6qy3Gx5mWeyfUpUssF5wXWOLW+NR1gsdVreLesiOB+Qnddb92 X-Received: by 2002:a17:902:9f88:: with SMTP id g8mr16985088plq.100.1576512142054; Mon, 16 Dec 2019 08:02:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512142; cv=none; d=google.com; s=arc-20160816; b=T7dEGQO/gne5w7niUPQepMePMTNai5rCFDzbDl7KHemPwxBG2LZ5Tpt6PAmeLOBMVs YvzKPmoCjQf5158sNyDLEPWeU+bz22T3cLSUslFPDekEbwj0AnPYVutoAGeXctpAfOOT hVJumyZhih0FEg9fzNBlppP5PYYsKx+X86yTogY1WXyJW52IA5miQKCNphAeSabh8oB5 ACzeB/YntqZEa5btqlrnqsNZuy+TY61zfx6Io5iIwFneXB9YZZKT8FGayTrqJD/bOwyC 5J7tNOHGgTUyVLQ0zl2omtjAcSkNfi+H7kHiKH2ez95Gqf94DD72GegGodBCrIMeDraR TMqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=kbAFI19BVgh819QgX8ZGRrxKkVvWxdlIJmlDXkkJboI=; b=0DLX0qekNMMOJoj/OJw2xaio7wYJZLuRSh2S62QgutncPrDXMl4wvqAdwKAESxOWlj B4CkbdxZemgF8g/200Adn6zUEu70SuwUEWeFkpGXSY2Xr4gEOHMwsa0F/uV/VfWbCmUW veBxnO2P1P7/SjYPSqAoSYnaXQLGng+rl2T5wNLKKgK6uYcN0FEKvQA2JG09DbKM9f4x VRk2z/V6G8Dt41H26a4LG0Exk0z/hjHscpgjfzRfBTb4UjZ/0J9x/xsILcDRWH1GVz6o +isUAccJ99QKzCPUgkUuyyztHvnGF80XKSMvshRMpO0eDu8S/7c05ilaz65os6hxVTjY NFsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=aCdfNsG3; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id x5si17310000pll.276.2019.12.16.08.02.21; Mon, 16 Dec 2019 08:02:22 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=aCdfNsG3; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 900C37FE10; Mon, 16 Dec 2019 16:00:47 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by mail.openembedded.org (Postfix) with ESMTP id DFA877FDCF for ; Mon, 16 Dec 2019 16:00:22 +0000 (UTC) Received: by mail-pj1-f51.google.com with SMTP id ep17so3167568pjb.4 for ; Mon, 16 Dec 2019 08:00:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=8Le6d7C523lNb84zEbuxPQle3X0mR+I5HfoAKRveFIs=; b=aCdfNsG38KCFe8j9XkrxvlnTM/OLCTYXorhcmHF8A+WMYBhIyIjSmd96Bw50vico2D WZiybBCmaCXCwr/MivQXXL5Au2G1++ryskDV90eoARC9xZAUF1sDBWcN+j69C2dYxHRi qtVd2lFvHWuXh0r+ynq8RGcry6JTmup+jh6fdD6iaf5Xav5tOWBNb+cbN3+kGLJ3DjQO X33jzUxITVQw2OO0P6AXd/P9wiPMIXO5ww8abTzyRQMxGReLkbDDgIzaVAawNOOa1TvV t+kaqbHbp2iHKbG7Dh0bf6LgcyvhoZ7eABbQcwUA6jdouYYBJqNLiAUunNaiKSzKMuY3 5PrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=8Le6d7C523lNb84zEbuxPQle3X0mR+I5HfoAKRveFIs=; b=FMrneETUu7YK4PdjAx7lptlzAQYECOROloENTqtERkML0E+qg4c6XTN0uIuc+lWdfp pmOuC2pXGhsO0TNuXD/4ZAqrnluFaRbNf46kcRJdWJcAs0KJJbH44nQ9eN+s6s0ueQWw LEX2GjvDnZNpLck/FfOQk1xA6fOwWhaghNsnCNpfoLhjpLtIdzD3szWWw033/SOk9geZ 92xqk91PjTk1UabmE/cKDu4Vnyi0e8l26o3k2rw0NWZim9SbqD4MszOd6zxnEuwYFXap wo5o0yp3cHYIF0bzHnUgKPgSR5DI0AUncG1Edd9r5dQUqRgGfKK75UMZWZC32xj1JXQf /39A== X-Gm-Message-State: APjAAAXRNWB6Z5HhC4OqugYeEaD4NxtKsu6VRCputVeWWUKaMn5VFhub C9nyoFAIHQU8aNeRYVnAspPrJd+DKy8= X-Received: by 2002:a17:902:b08d:: with SMTP id p13mr1884999plr.109.1576512023861; Mon, 16 Dec 2019 08:00:23 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:23 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 08:00:02 -0800 Message-Id: <27ee95bd1ec2076509cfc2230eadb876fb35d6c2.1576511913.git.akuster808@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 12/18] cve-update-db-native: add an index on the CVE ID column X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Create an index on the PRODUCTS table which contains a row for each CPE, drastically increasing the performance of lookups for a specific CVE. (From OE-Core rev: b4048b05b3a00d85c40d09961f846eadcebd812e) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/recipes-core/meta/cve-update-db-native.bb | 3 +++ 1 file changed, 3 insertions(+) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index c15534d..08b18f0 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -120,11 +120,14 @@ python do_populate_cve_db() { def initialize_db(c): c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \ VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ VERSION_END TEXT, OPERATOR_END TEXT)") + c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_ID_IDX on PRODUCTS(ID);") def parse_node_and_insert(c, node, cveId): # Parse children node if needed From patchwork Mon Dec 16 16:00:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181746 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4482321och; Mon, 16 Dec 2019 08:02:30 -0800 (PST) X-Google-Smtp-Source: APXvYqyg6jTtnY9DmpBsTmgZ2or27WGWcwrenQEsGuDcryKNuLT+Db5MLmie16rH9JJmNBzxv0DY X-Received: by 2002:a65:6815:: with SMTP id l21mr18489405pgt.283.1576512149988; Mon, 16 Dec 2019 08:02:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512149; cv=none; d=google.com; s=arc-20160816; b=c3sUi62Qm10KHdeoILXE345jB2keTnU1cv9Z6dd7mBavSWf+Xru7PPLiH8eW/i/8ZY xUQpP7YWj+eh27OkcdsqiP5ESVzb3vwbSHQfOPdJQpNwPwhG5l3v3nSQaChzBv2vyphL kfrPUgm7NcKf4ptyD1QGIar+sZClsgglgViHssL18LxtxqiaoUdqnxlsFEqM3TSL9z55 RxRzOHT8IpRnxIAB9EL25uKRfa/0S9SyyDhzLs7TNcvqCwn3SZVqkleB59z5fcqHPeB9 gEPjZZZb+k+shAhRZ4WfMN6IXN0MnDE5D2oW4WOH4Wf4yi+xkfdWdtAwIKXr3L55Uxrz eNsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=Rg9OcnIt/pOoGLUjgTDxrwzQqmjFm/CjH3xZgv7EJvA=; b=bHxTEuivxrA3tTX/Jl9XY7Nj1RRJsFa3R4ACFk3e9wWH4mF3MwR15wp/D7BC/XcKdb LGebBgLmHkKVuZEFWGRFeSvkScRpZR84aRoslSoBhyD1pQ8YSdMiXm60siqI/eYC3JjZ qBGsD8k2Hnr9I/iaGZHr1wqEjcyfdCsgZBYY2ZiTuoniLSnJtKkwyJux0HucDzAuGhmN 3+MhbFyra5uiEM8l2qgtvk7+ZvxcMCJJvj7YCqpMepQzEJO/dMBLDf1G7QdAkD5f9jCs oYrolIAoByVE3PYJXYPPNhxlOzSnFbhBE3/KIGpvCRAndEU+VW88iU5LEgTvPuJR8G4/ x8Lw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=sWuilZB8; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id z23si16580394pju.99.2019.12.16.08.02.29; Mon, 16 Dec 2019 08:02:29 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=sWuilZB8; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id BC93F7FE14; Mon, 16 Dec 2019 16:00:48 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by mail.openembedded.org (Postfix) with ESMTP id A8CF37FDC9 for ; Mon, 16 Dec 2019 16:00:23 +0000 (UTC) Received: by mail-pg1-f170.google.com with SMTP id q127so3963649pga.4 for ; Mon, 16 Dec 2019 08:00:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=lVCLDTPb0GFJoEWDw+Gg93KUJz04qQroZhCJYvxuk0E=; b=sWuilZB8F0LS6JkHR2J1gr3jonJfTHDzARzpPkr4nKd5/OU7yLLFAiNSYhpgC1+/gf pcbGm3v/L7LrCXfYblFujPm0EWFlCUuKnZcxSse5Vb2vnvGhnxy+ELKgDhqZGDudT4Ka jc6W1TQD6LenGjy/NM+G0ynEPXfJsPTlVGk/zLTe+DMsMuGcllHoYdzd81QBg2PcN5t7 BuyXvHnbgW87PpGb/bZhXvPI/tGKcrcBFFS7XooNMWdiJaun41LLdmd6RRfMNpHu3Bm+ Qa+IQyHN2JNVtLhlZuVcF83xVu52sj4ZrTUgnVcrvM0dYv1zZRYgm5VPY5b6+76OAB7Q SfqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=lVCLDTPb0GFJoEWDw+Gg93KUJz04qQroZhCJYvxuk0E=; b=h9OmJcLkCes7X/TtucwE7lHOHH+CFImMCpvHtmV2QJVS5yOtXfW5v8xqjwx7k+xXJ3 WfITjuR6qoni3T5kTIOtHfWhn+8/vsFEbbeY1rIAbw4CfW1ZalaiKm/mDaZ51pv1uL7R IR4ox3CCiU3e1iWXekQxRkPlTtkwHAvfTGba0Yaq+xchDfyCwNr5jxyLKTuqljMZf5xf 3V4i9mIDdTWeQJd1e33NOesAcqALoKdph21Z7XB0of2/xvMAjKI/p8wi+dKACrMROOST jvQSDok+ODJewc7HadwemZlRx2QfPNckp5X43fF/cIrbEbHirRe7bJBv4nTdwB5f7/Nz +fhg== X-Gm-Message-State: APjAAAUhvpjs7WpgzfrpdWP1+n7SfZ8tpwbypBwPPUGWV+Wlr808Cnr6 E/6UjzdQCEvDelxboQHAOVSGjM865UA= X-Received: by 2002:a63:f64a:: with SMTP id u10mr18666101pgj.16.1576512024590; Mon, 16 Dec 2019 08:00:24 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:24 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 08:00:03 -0800 Message-Id: X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 13/18] cve-update-db-native: clean up proxy handling X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton urllib handles adding proxy handlers if the proxies are set in the environment, so call bb.utils.export_proxies() to do that and remove the manual setup. (From OE-Core rev: 6b73004668b3b71c9c38814b79fbb58c893ed434) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/recipes-core/meta/cve-update-db-native.bb | 31 +++++--------------------- 1 file changed, 5 insertions(+), 26 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 08b18f0..db1d69a 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -21,10 +21,12 @@ python do_populate_cve_db() { """ Update NVD database with json data feed """ - + import bb.utils import sqlite3, urllib, urllib.parse, shutil, gzip from datetime import date + bb.utils.export_proxies(d) + BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 @@ -40,16 +42,6 @@ python do_populate_cve_db() { except OSError: pass - proxy = d.getVar("https_proxy") - if proxy: - # instantiate an opener but do not install it as the global - # opener unless if we're really sure it's applicable for all - # urllib requests - proxy_handler = urllib.request.ProxyHandler({'https': proxy}) - proxy_opener = urllib.request.build_opener(proxy_handler) - else: - proxy_opener = None - cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') if not os.path.isdir(db_dir): @@ -67,15 +59,7 @@ python do_populate_cve_db() { json_url = year_url + ".json.gz" # Retrieve meta last modified date - - response = None - - if proxy_opener: - response = proxy_opener.open(meta_url) - else: - req = urllib.request.Request(meta_url) - response = urllib.request.urlopen(req) - + response = urllib.request.urlopen(meta_url) if response: for l in response.read().decode("utf-8").splitlines(): key, value = l.split(":", 1) @@ -95,12 +79,7 @@ python do_populate_cve_db() { # Update db with current year json file try: - if proxy_opener: - response = proxy_opener.open(json_url) - else: - req = urllib.request.Request(json_url) - response = urllib.request.urlopen(req) - + response = urllib.request.urlopen(json_url) if response: update_db(c, gzip.decompress(response.read()).decode('utf-8')) c.execute("insert or replace into META values (?, ?)", [year, last_modified]) From patchwork Mon Dec 16 16:00:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181747 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4482463och; Mon, 16 Dec 2019 08:02:38 -0800 (PST) X-Google-Smtp-Source: APXvYqz8xZk1DsN2bcU55HJhfkFHVSfvszS2FPVmKC9sxo0AG/FDqJGvncNuma8usQFVT5WUExow X-Received: by 2002:a63:1101:: with SMTP id g1mr18525077pgl.435.1576512157961; Mon, 16 Dec 2019 08:02:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512157; cv=none; d=google.com; s=arc-20160816; b=GPDn0e/mg5QEkBAUBfbwfj6vPL2Q2Zx64hpQkQmSfbqHyzVC5IWcSDdf/JWwDENlID qvfHieuKGb/L38MehJxyEFVDkrKshMa164OuLEGOtspAiKxls81vh+VPrdKy3e1er9xF Y72wj8ddnm3NRBXi4TYaXBSjJsRmsqDA5OhLALoiq0ocClzvReRB6Y3O0fFLRqNTPyIe JXnc0G8GTWnJ0NZovlq+pQmNsHB5GQHi/m5AUnpaPb2EOSIy5cZTUw+HdZHADAa7tGQx VvnHdWs9NofDQBRW/vpbAjZ2JxQUVkCOoO5sGv2o5EMPZnulV8scgCdzZWJVP+DkWdRU 8zzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=m+m0fixajNsYzRP3nAWnJtAt0YPOzjXpAK6xz8eoJfc=; b=hTwCd8BThvezxy8aT/1Lr5lwK0/qf77QAGNyckVf0H5BQz0IStHvjL4T1cFQ/1S/EY s0GBunF35QvIgOVL/ejq1Li9cfgMwQ0LhvbExaqwJKj25EyB/UgS36gK039JOaE2QErj 64IMauwJKQiBAxolrHhmXaTXVv4SwvsimyPkETLPe0Cb/8loX4PKQGzmkeILaUvzUEsJ bCY21MM6v80UrOzQL80GEvImLeIP6/jgXWyJ7wjn6bXTxLjX3jxxnPnX/moBeuU4RaYj gwLmIiXIYFcKvH5bym1HnwVpw+IbnzuEK8kefMCrPR6HVtNQ5mrjX1L2behWK9y9i3Sb bliw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=PJTuy58o; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id b11si12520765plz.236.2019.12.16.08.02.37; Mon, 16 Dec 2019 08:02:37 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=PJTuy58o; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id E8CCC7FE18; Mon, 16 Dec 2019 16:00:49 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by mail.openembedded.org (Postfix) with ESMTP id 9EF887FDD1 for ; Mon, 16 Dec 2019 16:00:24 +0000 (UTC) Received: by mail-pl1-f177.google.com with SMTP id x17so4665931pln.1 for ; Mon, 16 Dec 2019 08:00:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=ZFs0MnN/bBfSg/6D+ln1OmgzSOl42k/KJkc0HvAoc1g=; b=PJTuy58ow2S2FE7eIICG33DJUTi+Hsew0X8d2edtyyRGHus5x3+XGwcahXka4qX8RC mMuPp6I985ep267lRanC8YkOgRW0Q3TJXs3f0UWDhpiwI7jFhT/zaR68nZv8U4geWMqN Rlnc3H9OVcdAo9cRyzjgchGT/NHPtiXPuGdpqq06DXEaklcTDCUmNEr/pxkegfBJbo8e GkeeRhTTAAdbZwNcLKmxVert2ABb8oT0K+y9yckqae1SrpNg6a1Or+K8LMbwnR+LWa6r eaqKF2j3GGVSNobOnC3mUXGif3WTmx9oNpCz7G+ySW7DHhYLFjkLgZHt+6K9hMBW19HQ uGGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=ZFs0MnN/bBfSg/6D+ln1OmgzSOl42k/KJkc0HvAoc1g=; b=qfEC2UAPPdNN0ebM45onWLKR0zIQAgNsGvXzknC9FXl1naSyxxfhKoqyRivN0H4lFp HnPzV2Y8Sx3A+ro0bsvVHN2zNaUy/HdqvsnsvdDBEfJbDOqIHcz0q8YmnfEhHCEXpbVW JcEoDUh5jLA+7AjQFIbOi3Eeu80aJICIPMBZxp2867fSg6Zw42IDPJfaS0yWGP9HiZXo jdqntJa6LP3bv1RO2AN3B97Rsd/03oAXoCHzGZymKpjIZJ1aJ+HaRpb7BaUaoAPJODEP nM+yGtEqBFz1PT3bhaitl+f9nHQwc0BO6TWp7byauo7P2OkIm2JdoKL/kFgVx4fiif3H rI3Q== X-Gm-Message-State: APjAAAWimrhzC1Ir5susdfI5fE4u5ZHlGqTsG9qWl4vcvgA3MRQ/lZc9 QSgQpDNUZmOMBbjJE+0gGiRphDZ8VeM= X-Received: by 2002:a17:902:9b84:: with SMTP id y4mr16175242plp.13.1576512025495; Mon, 16 Dec 2019 08:00:25 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:24 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 08:00:04 -0800 Message-Id: <541dc24d974d3e22c45a650c34298eebc45121e8.1576511913.git.akuster808@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 14/18] cve-check: rewrite look to fix false negatives X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton A previous optimisation was premature and resulted in false-negatives in the report. Rewrite the checking algorithm to first get the list of potential CVEs by vendor:product, then iterate through every matching CPE for that CVE to determine if the bounds match or not. By doing this in two stages we can know if we've checked every CPE, instead of accidentally breaking out of the scan too early. (From OE-Core rev: d61aff9e22704ad69df1f7ab0f8784f4e7cc0c69) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/classes/cve-check.bbclass | 63 +++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 29 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 3326944..c1cbdbd 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -165,7 +165,6 @@ def check_cves(d, patched_cves): """ Connect to the NVD database and find unpatched cves. """ - import ast, csv, tempfile, subprocess, io from distutils.version import LooseVersion cves_unpatched = [] @@ -187,68 +186,74 @@ def check_cves(d, patched_cves): cve_whitelist = d.getVar("CVE_CHECK_WHITELIST").split() import sqlite3 - db_file = d.getVar("CVE_CHECK_DB_FILE") - conn = sqlite3.connect(db_file) + db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro") + conn = sqlite3.connect(db_file, uri=True) + # For each of the known product names (e.g. curl has CPEs using curl and libcurl)... for product in products: - c = conn.cursor() if ":" in product: vendor, product = product.split(":", 1) - c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR IS ?", (product, vendor)) else: - c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,)) + vendor = "%" - for row in c: - cve = row[0] - version_start = row[3] - operator_start = row[4] - version_end = row[5] - operator_end = row[6] + # Find all relevant CVE IDs. + for cverow in conn.execute("SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)): + cve = cverow[0] if cve in cve_whitelist: bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) # TODO: this should be in the report as 'whitelisted' patched_cves.add(cve) + continue elif cve in patched_cves: bb.note("%s has been patched" % (cve)) - else: - to_append = False + continue + + vulnerable = False + for row in conn.execute("SELECT * FROM PRODUCTS WHERE ID IS ? AND PRODUCT IS ? AND VENDOR LIKE ?", (cve, product, vendor)): + (_, _, _, version_start, operator_start, version_end, operator_end) = row + #bb.debug(2, "Evaluating row " + str(row)) + if (operator_start == '=' and pv == version_start): - to_append = True + vulnerable = True else: if operator_start: try: - to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) - to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) + vulnerable_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) + vulnerable_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) except: bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_start, version_start, cve)) - to_append_start = False + vulnerable_start = False else: - to_append_start = False + vulnerable_start = False if operator_end: try: - to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) - to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) + vulnerable_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) + vulnerable_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) except: bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_end, version_end, cve)) - to_append_end = False + vulnerable_end = False else: - to_append_end = False + vulnerable_end = False if operator_start and operator_end: - to_append = to_append_start and to_append_end + vulnerable = vulnerable_start and vulnerable_end else: - to_append = to_append_start or to_append_end + vulnerable = vulnerable_start or vulnerable_end - if to_append: + if vulnerable: bb.note("%s-%s is vulnerable to %s" % (product, pv, cve)) cves_unpatched.append(cve) - else: - bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) - patched_cves.add(cve) + break + + if not vulnerable: + bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) + # TODO: not patched but not vulnerable + patched_cves.add(cve) + conn.close() return (list(patched_cves), cves_unpatched) From patchwork Mon Dec 16 16:00:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181748 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4482649och; Mon, 16 Dec 2019 08:02:47 -0800 (PST) X-Google-Smtp-Source: APXvYqyf5MgpcfSueQ5BAY97TsLMyd7xuM35l4srcmgHTc+5DMV0n4cCNoT7vxBjTMxPNEVGgMlq X-Received: by 2002:a62:a515:: with SMTP id v21mr16791284pfm.128.1576512167383; Mon, 16 Dec 2019 08:02:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512167; cv=none; d=google.com; s=arc-20160816; b=KwuvYDfF6N7hnfsmYVdtMwa0IXWVPVLAL/YL0OvQT3np4OZK+cQZMkcpFdECrQIsmr v3rlAdZNH+JBlI+LtgR2qcX7e65kXwxP022rPbRHWMESkSkSrCJ0tS9sbwzcCZ9U75ip tl5bsuqtzKGoy5+P9d7wv0LFGYUjRdFqhwlvDFdEkq/PUGkfRO5+s8zyAknbIZaAmHA1 J5CcESmpxVs9Uf0k/a3ufWpildMQi/8P4jK6xziiiZ4hY0OIZwKfi7V8J0J9PDDhvQ3k 6kjEinucYlFKFe3TSnifrnXN2KsBKuu/U8GpCPt1hQVtvZyFKSiw9DuDUFYT5GbZvHKD w+JA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=D9n1YvEhURD3MSU85xI9ylEmYh+jhu3TnuQZNV+Yvmw=; b=uBCTsBx8YDIUC24L8w8ArbSWsVY6Ap1Sl++HXSxYUw4ZYktzIDL86ZNBQy7iY2fMkX U52INud8/8KEFyYBri+UbUs7OlPR0YrCNiYFv1uXkbUBfFS1gmndAKrcSRzu8xpSuMjB zJz3VeqQE9CTM12cET46q/SFP2EsAdnPFiFTklCfL5sKrMIjgMbTtyjb9/F74PZFwjZN cPWCiyKiSprvAbsp29IF1tGtfWk+xrfySvLM5WIAN6j8ZthRzhhn15Khq0kNcTWTXIFM sSbtztodpDtW4Yy3WGOS9ZS3BxrjCQOQ9ZzuerDalP7vIMhPoqr/Y9c9lVkKyAsmQ8zj cHTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=p+BTI0jF; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id x14si19019713pfd.68.2019.12.16.08.02.46; Mon, 16 Dec 2019 08:02:47 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=p+BTI0jF; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 4457C7FE1E; Mon, 16 Dec 2019 16:00:51 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by mail.openembedded.org (Postfix) with ESMTP id D22C07FDA4 for ; Mon, 16 Dec 2019 16:00:25 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id j11so3174267pjs.1 for ; Mon, 16 Dec 2019 08:00:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=Ff/S15i2jB8nQJDaYpdbvIQwA8T9LvpAqbxfrmOPTVQ=; b=p+BTI0jFHOGFfSLnyA2IFU2lbuBAs0pa7/cujAXaGWiTS02iuxTir+plwozcdWWZ1s /l3izxVIgYwj8PEq4rnNRBhxCy7X0wgVIIOnhMYt/uu+h8qOUF6NUhGebxPLKviKdz3O RKkLGNnPQJOekWXeVjRFdHgTCDunlQ0XCYVLzNqGZMVPF4AwElve+PHbTjPTzrdrPNyY nk1uyRIa9pFHusbaocn9ytm8p5y/UcHZYILot69tj2l3yfiJMatxFOE77MmEelUdTtsL SoOQtTLlOXjxrNqRnpriHE2obVUULmCnPx88I25PWIptAGdeqWLHSa1txO1tMgBR/aid pDEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=Ff/S15i2jB8nQJDaYpdbvIQwA8T9LvpAqbxfrmOPTVQ=; b=msdc9p2yD17Mlq8NaF9haR/Rywl/Bj5CJ1SO3SfIZGAmFZLLvfhrKcZY1/pC6DNQfw 2HLD0fdt87JUrVEh8XRJR6FIDohSxUt3w+SyGf+VBeriCWVDUsQJEVBwIlMeq6q6GzzE bjR55mKFSICzKc6+YCctIlcwQwOVxz+iTDTGCDRT/S/6qnLG7IAyCNLaPU+FwZfP809K WK5JDhwc0Olj5fMxXtCRpWgVpa++zddpQy2T8maUBgtJ/y7N/tF1EBW2JrqLc6YLH4mL Cxv5om8KWU7bZLqRHTvEyHSkYqyufe9PwTodp0Tknl7lxXXaR8MyTo2wbIP1cwXHJLrs xh9Q== X-Gm-Message-State: APjAAAWj6H5BJ7ti9p8n+v/iyNxy6/rvXTUIAB+ilOkrs+QXq2aYZLyb 0oMTIIr4zhD5z/b3Hgi/V2EJU0gXV5E= X-Received: by 2002:a17:902:8601:: with SMTP id f1mr2802420plo.308.1576512026494; Mon, 16 Dec 2019 08:00:26 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:25 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 08:00:05 -0800 Message-Id: <1f3863bc31e03207856f55591cbf17543e188587.1576511913.git.akuster808@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 15/18] cve-check: neaten get_cve_info X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Remove obsolete Python 2 code, and use convenience methods for neatness. (From OE-Core rev: f19253cc9e70c974a8e21a142086c13d7cde04ff) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/classes/cve-check.bbclass | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index c1cbdbd..e95716d 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -261,23 +261,15 @@ def check_cves(d, patched_cves): def get_cve_info(d, cves): """ Get CVE information from the database. - - Unfortunately the only way to get CVE info is set the output to - html (hard to parse) or query directly the database. """ - try: - import sqlite3 - except ImportError: - from pysqlite2 import dbapi2 as sqlite3 + import sqlite3 cve_data = {} - db_file = d.getVar("CVE_CHECK_DB_FILE") - placeholder = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder - conn = sqlite3.connect(db_file) - cur = conn.cursor() - for row in cur.execute(query, tuple(cves)): + conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) + placeholders = ",".join("?" * len(cves)) + query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders + for row in conn.execute(query, tuple(cves)): cve_data[row[0]] = {} cve_data[row[0]]["summary"] = row[1] cve_data[row[0]]["scorev2"] = row[2] From patchwork Mon Dec 16 16:00:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181749 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4482859och; Mon, 16 Dec 2019 08:02:57 -0800 (PST) X-Google-Smtp-Source: APXvYqwzOeL1joQZUPIkGETNYkbQLPPNvLS5VE6af8QXvzq6PW01zByNcEPFKw8hnUNSpwg7mtlc X-Received: by 2002:a63:d017:: with SMTP id z23mr19257537pgf.110.1576512177124; Mon, 16 Dec 2019 08:02:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512177; cv=none; d=google.com; s=arc-20160816; b=w3D1Wn12PtN3uiiulW8IZP+VgMYTug8KWyympU6Zo35PsyEltuJGIFPCiJcZwUk0Ia Yd9iG6DKmfTnBCQiUHRlEhfhm5jbFPpDIo6nPxbYz1p+RTibTidAGdHzhUVC/8kB0Nyn LGsCf/Ows8AodQvmB1EnRbQmZn5ljXz/Z0+hUxs4W/+77dbAkVT1a5UCw0n0MmiIvyYh vASAqUwnrDNnwV2NHp9ACp2/NXju4ssm0kP2XHWdHGfudCQDp9YvkuzSkOzNkGdOqKQa QNSzUFtMPqWxsb/GlvFHHdqJpksZ6ucZLQHytFDi8DGtoeYoZ1fLKOM4UAyUcTlswTi2 ZzVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=z3AXyvl94tcfv7b3cmCJ1hEWs1yhsd0SVup6EgfIZ74=; b=rgXCJYiUufng1aW6A5M/0xxMBL4i+quHrVhfFXWiYv+Y4rJaEbA8a18TrjcSGEDZdl AcxBkTNQ5cM+kVpP8C04llJwriMn8Q+UXvz60SHtR18uy0O0JlOyeBjgxK63fNCE4nB5 j37dVOMzUncNQrrF5EhUsO+R86LVgmPiiPdhJCiEyRDlClC+mZ2m+zOpyn9r2wDJw/Dj snogRwNTd0SWeqmKk8YFBTPMtp/GB1wa+4tcR8Pu8LxzidHJIYr2RJ7TVO0J9zA1VV6M mGT8vbJ+0rLDXNNHL8qvsmQHKq9nVnJQTw3ktJadGZ57ZnSnDhEEJUi60DvRzB4KK1nj 1TxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=JKfDqR9d; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id fa3si16810049pjb.64.2019.12.16.08.02.55; Mon, 16 Dec 2019 08:02:57 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=JKfDqR9d; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 6AC2F7FE23; Mon, 16 Dec 2019 16:00:52 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by mail.openembedded.org (Postfix) with ESMTP id 6034E7FDA4 for ; Mon, 16 Dec 2019 16:00:26 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id bd4so4647093plb.8 for ; Mon, 16 Dec 2019 08:00:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=AsGSA6vh4PXyQQ2kwt3gTgoxSfeCfvBSAQkQZ4bRL3w=; b=JKfDqR9dkUddcNbdZL0IoREZwRI663aEXmA2u3sILMyz29yse2ChGtnB6gBJvZUzqn ILvmmERND4ddgewZ+DCqLAXJZnk8B0dNGUTuDyPmYl1KDh+c3EBVYLXAF09umgRo/8UB EpFlU6DSmpofvJ0qPkFX0bPHQ0O/hNUHB2Tv9RBOe15dmTAIbbe/h0bxr3Fkh/yn05Xb S0rMz8ajIGwc2VUtj02363VKR3g0pQXyUBPOyBywZYRX4TSQ4LVnkPn9vIVcmDVMdGcT hO7AWUFWa0jKemd+lJ9Y2XVhWEYzX1qwc6/WzLyKErYGG4QAAH43mbLqp37pXdL2hxjU 3tXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=AsGSA6vh4PXyQQ2kwt3gTgoxSfeCfvBSAQkQZ4bRL3w=; b=O93EIrFf4BYLEt6dbpgLxk5Pm1RqbTlhOv869i08CsuYRrtoDXgITuJXEujBLbSO+E X8NTQn1UUyDsGALed/Ik8kDjyfSwKbS4TTSxdYq3zInGgNzc6BAx679faobgYmGZ8exU S7qwDEnpPJHaUgucBYTjkCnFh2ghE+7ZMQhrTfjwhfe6/JUENPRToFLCQqb2Mp2mqgCe Eddp/e5yvDoxhBRLq9Eo/Vf+IOE01vo3uLKpbzCBeBbIIarkW5CBG3vPu9pWkFtfzssq AjAZ7luI+gi/rVICls4pEPyNy+RBiEPDTRZ9eQwQxhbIHTwQ9RiN/AZbNdeFWiecRbmD 8+KQ== X-Gm-Message-State: APjAAAVawQPWG23fxN6QzCOrOfdq9Q6MLtOTvJGWZZ16osQGHIlEra+5 59IpYnRJA5y5a350LVGj/WSaxM1lKUc= X-Received: by 2002:a17:902:9043:: with SMTP id w3mr16730122plz.8.1576512027366; Mon, 16 Dec 2019 08:00:27 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:26 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 08:00:06 -0800 Message-Id: X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 16/18] cve-check: fetch CVE data once at a time instead of in a single call X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton This code used to construct a single SQL statement that fetched the NVD data for every CVE requested. For recipes such as the kernel where there are over 2000 CVEs to report this can hit the variable count limit and the query fails with "sqlite3.OperationalError: too many SQL variables". The default limit is 999 variables, but some distributions such as Debian set the default to 250000. As the NVD table has an index on the ID column, whilst requesting the data CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time different is insignificant: 0.05s verses 0.01s on my machine. (From OE-Core rev: 53d0cc1e9b7190fa66d7ff1c59518f91b0128d99) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/classes/cve-check.bbclass | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index e95716d..19ed554 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -267,17 +267,17 @@ def get_cve_info(d, cves): cve_data = {} conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) - placeholders = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders - for row in conn.execute(query, tuple(cves)): - cve_data[row[0]] = {} - cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["scorev2"] = row[2] - cve_data[row[0]]["scorev3"] = row[3] - cve_data[row[0]]["modified"] = row[4] - cve_data[row[0]]["vector"] = row[5] - conn.close() + for cve in cves: + for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)): + cve_data[row[0]] = {} + cve_data[row[0]]["summary"] = row[1] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] + + conn.close() return cve_data def cve_write_data(d, patched, unpatched, cve_data): From patchwork Mon Dec 16 16:00:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181750 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4482997och; Mon, 16 Dec 2019 08:03:03 -0800 (PST) X-Google-Smtp-Source: APXvYqwwrvyH9X+I6heihGdcgI0YmurV/k+d8Z/vuXoJvRQlGzgz5ZW7UbC1zhINKDah+SdNtJMH X-Received: by 2002:a62:7b54:: with SMTP id w81mr16680253pfc.127.1576512183430; Mon, 16 Dec 2019 08:03:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512183; cv=none; d=google.com; s=arc-20160816; b=Fk2YtabNbQgdBTH6Z4keLOAs4B7/S8l2M0FvCXCIsySJM2AKdff8KTxAqofZWUF9/g 1XoV+w0UKGPuFRTMSg7OW7Rle1b6JW/0X6g4LF1gf7/gN7oDh/RLZDpNBUFKFToOjRbY F6DZwFq+bxwAtg/nCWTqf2AW7hDq2/vOFsGkGn/X5t/lq7LClKvUb94nDLGd26c5688u 4lTMczkjZQM005ddKmgAMLFMbyU5Z3fvSD2uR81YeEbo2hAKv6Pdq2H9ktr1bkslAo4B OP7m2t0yMhmXrxzVbvvabKNztEdDHa132S9Q8WbWiRkb5VzkzzMSmYRjAJDy8yRCXY2z YJbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=EnotpkiUt45Ckp9+fH2J0kkKlAODPc2OiYEIk1gcHZQ=; b=U91r5tRlBNbM8oyablJc9v3z4/e3difcjfEtKwFqoWsP80NWMCMOnrtKxy+IR7OA// GmjijB4zKAMYKi25uF3WDLF7XfPFGWoVYK6sTQpt5FXz3C4OTtphN1AdhZrGnhc6OJPO uxGVZ7HOdBumwRDXhj4v5pceV7kGLZB5MbP9mGkpOF8/nJojQizZ8jhvT/Afcfz8Zpfd bLw+ZzDJTMCPTtePLHH8S5cDMyFSFVGs8qqEItT46UcH15mb8Em0d+aCQQ6+g9ZcIEra I+ABAQubbYfMNH+o6hl1jHAOywUDYSo9BIUuqlMjkfBAaTUHTyaXJ+qNBo/hAw1sY4nG 9Xzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=raNhWk54; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id 69si17564987plb.333.2019.12.16.08.03.02; Mon, 16 Dec 2019 08:03:03 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=raNhWk54; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id BE82E7FE25; Mon, 16 Dec 2019 16:00:53 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by mail.openembedded.org (Postfix) with ESMTP id C91A47FDA4 for ; Mon, 16 Dec 2019 16:00:28 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id r67so3176412pjb.0 for ; Mon, 16 Dec 2019 08:00:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=YdD3VcEF+a7PvParkWCL2/YffV5xH5fz3ICIWEFXuy8=; b=raNhWk54iifJBMR4prwovBpLC0YoH4iHAJAEngWetTRODs1naZGc5sEQ66XVCJeVGm pR1+7PV9KHO9TEM6gBvINh6mjdExRWtzT0ksfTT6zDjk5iKMDIeXEGVo2lFNM6lDpfVP OvF7tstDxE8xAVkNHh/XNSwt0jI+NuLkoPuKIEnUMJ1fio7GipKP9zMO5J82i7jXn9jS xRZPVfTBw5u9W94wBwtFxWjMZsY8zUk+ySPhPFKaOP5PWikqXuitBh6vR6EjOo0FOxN+ ujJF/CRIsZLf9INM7J+lkMCS1drqeYFwnJekTWJHPKv27uneBI599L6lRMtQDoSfHhxr 8bLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=YdD3VcEF+a7PvParkWCL2/YffV5xH5fz3ICIWEFXuy8=; b=ghvpPw3cDkR6arPCv77eAdO8cilVgTZooy01Zs83l7o5Au5E2pg+Os6uXtKijazCtF n1a38TUwU30ygH5iBsrfHq5Eg7fFb+ibD8LPOvoqZb/O0BHaWkWHkQqCphREoF67dJK2 nMDOrArK9X381LargZuUV2FyetFoLNevDWv/ZCyNILbgY38wYyrULcxoeGGYFpE24Zbx TIbW7R0tsdAN4Zxo/v+5K9LI6SwbavdS+fwGixzVAV+MOe4C2eT12/0P+ii8bHqFYFN2 qzK+QHiSWR5MAOCt/uTTYRXxrNRS9GVjdNVKXjjWZTdVweDruM9fbcf/JVYdkxbXFOGD Pa9w== X-Gm-Message-State: APjAAAXU1tVPOkzJQWDIc9pL/o9oNsCcJMS9ZT/h8IWN+eCG0UE/FYUS plSZoRuQOgiQ43K7KDPl+OR9vTqhJds= X-Received: by 2002:a17:902:fe8d:: with SMTP id x13mr17129200plm.232.1576512028396; Mon, 16 Dec 2019 08:00:28 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:27 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 08:00:07 -0800 Message-Id: X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 17/18] glibc: finish incomplete fix for CVE-2016-10739 X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton Somehow the patch for this CVE only included one of the four required patches. Signed-off-by: Ross Burton Signed-off-by: Armin Kuster --- meta/recipes-core/glibc/glibc/CVE-2016-10739.patch | 910 ++++++++++++++++++++- 1 file changed, 907 insertions(+), 3 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/glibc/glibc/CVE-2016-10739.patch b/meta/recipes-core/glibc/glibc/CVE-2016-10739.patch index 7eb55d6..7dc8428 100644 --- a/meta/recipes-core/glibc/glibc/CVE-2016-10739.patch +++ b/meta/recipes-core/glibc/glibc/CVE-2016-10739.patch @@ -5,12 +5,12 @@ Signed-off-by: Ross Burton From 8e92ca5dd7a7e38a4dddf1ebc4e1e8f0cb27e4aa Mon Sep 17 00:00:00 2001 From: Florian Weimer Date: Mon, 21 Jan 2019 08:59:42 +0100 -Subject: [PATCH] resolv: Reformat inet_addr, inet_aton to GNU style +Subject: [PATCH 1/4] resolv: Reformat inet_addr, inet_aton to GNU style (cherry picked from commit 5e30b8ef0758763effa115634e0ed7d8938e4bc0) --- ChangeLog | 5 ++ - resolv/inet_addr.c | 192 ++++++++++++++++++++++++++++------------------------- + resolv/inet_addr.c | 192 ++++++++++++++++++++++++--------------------- 2 files changed, 106 insertions(+), 91 deletions(-) diff --git a/resolv/inet_addr.c b/resolv/inet_addr.c @@ -229,4 +229,908 @@ index 022f7ea084..32f58b0e13 100644 weak_alias (__inet_aton, inet_aton) libc_hidden_def (__inet_aton) -- -2.11.0 +2.20.1 + + +From 37edf1d3f8ab9adefb61cc466ac52b53114fbd5b Mon Sep 17 00:00:00 2001 +From: Florian Weimer +Date: Mon, 21 Jan 2019 09:26:41 +0100 +Subject: [PATCH 2/4] resolv: Do not send queries for non-host-names in nss_dns + [BZ #24112] + +Before this commit, nss_dns would send a query which did not contain a +host name as the query name (such as invalid\032name.example.com) and +then reject the answer in getanswer_r and gaih_getanswer_slice, using +a check based on res_hnok. With this commit, no query is sent, and a +host-not-found error is returned to NSS without network interaction. + +(cherry picked from commit 6ca53a2453598804a2559a548a08424fca96434a) +--- + ChangeLog | 9 +++++++++ + resolv/nss_dns/dns-host.c | 24 ++++++++++++++++++++++-- + 2 files changed, 31 insertions(+), 2 deletions(-) + +diff --git a/resolv/nss_dns/dns-host.c b/resolv/nss_dns/dns-host.c +index 5dc2829cd1..99c3b61e1c 100644 +--- a/resolv/nss_dns/dns-host.c ++++ b/resolv/nss_dns/dns-host.c +@@ -274,11 +274,26 @@ gethostbyname3_context (struct resolv_context *ctx, + return status; + } + ++/* Verify that the name looks like a host name. There is no point in ++ sending a query which will not produce a usable name in the ++ response. */ ++static enum nss_status ++check_name (const char *name, int *h_errnop) ++{ ++ if (res_hnok (name)) ++ return NSS_STATUS_SUCCESS; ++ *h_errnop = HOST_NOT_FOUND; ++ return NSS_STATUS_NOTFOUND; ++} ++ + enum nss_status + _nss_dns_gethostbyname2_r (const char *name, int af, struct hostent *result, + char *buffer, size_t buflen, int *errnop, + int *h_errnop) + { ++ enum nss_status status = check_name (name, h_errnop); ++ if (status != NSS_STATUS_SUCCESS) ++ return status; + return _nss_dns_gethostbyname3_r (name, af, result, buffer, buflen, errnop, + h_errnop, NULL, NULL); + } +@@ -289,6 +304,9 @@ _nss_dns_gethostbyname_r (const char *name, struct hostent *result, + char *buffer, size_t buflen, int *errnop, + int *h_errnop) + { ++ enum nss_status status = check_name (name, h_errnop); ++ if (status != NSS_STATUS_SUCCESS) ++ return status; + struct resolv_context *ctx = __resolv_context_get (); + if (ctx == NULL) + { +@@ -296,7 +314,7 @@ _nss_dns_gethostbyname_r (const char *name, struct hostent *result, + *h_errnop = NETDB_INTERNAL; + return NSS_STATUS_UNAVAIL; + } +- enum nss_status status = NSS_STATUS_NOTFOUND; ++ status = NSS_STATUS_NOTFOUND; + if (res_use_inet6 ()) + status = gethostbyname3_context (ctx, name, AF_INET6, result, buffer, + buflen, errnop, h_errnop, NULL, NULL); +@@ -313,6 +331,9 @@ _nss_dns_gethostbyname4_r (const char *name, struct gaih_addrtuple **pat, + char *buffer, size_t buflen, int *errnop, + int *herrnop, int32_t *ttlp) + { ++ enum nss_status status = check_name (name, herrnop); ++ if (status != NSS_STATUS_SUCCESS) ++ return status; + struct resolv_context *ctx = __resolv_context_get (); + if (ctx == NULL) + { +@@ -347,7 +368,6 @@ _nss_dns_gethostbyname4_r (const char *name, struct gaih_addrtuple **pat, + int ans2p_malloced = 0; + + int olderr = errno; +- enum nss_status status; + int n = __res_context_search (ctx, name, C_IN, T_QUERY_A_AND_AAAA, + host_buffer.buf->buf, 2048, &host_buffer.ptr, + &ans2p, &nans2p, &resplen2, &ans2p_malloced); +-- +2.20.1 + + +From 2373941bd73cb288c8a42a33e23e7f7bb81151e7 Mon Sep 17 00:00:00 2001 +From: Florian Weimer +Date: Mon, 21 Jan 2019 21:26:03 +0100 +Subject: [PATCH 3/4] CVE-2016-10739: getaddrinfo: Fully parse IPv4 address + strings [BZ #20018] + +The IPv4 address parser in the getaddrinfo function is changed so that +it does not ignore trailing whitespace and all characters after it. +For backwards compatibility, the getaddrinfo function still recognizes +legacy name syntax, such as 192.000.002.010 interpreted as 192.0.2.8 +(octal). + +This commit does not change the behavior of inet_addr and inet_aton. +gethostbyname already had additional sanity checks (but is switched +over to the new __inet_aton_exact function for completeness as well). + +To avoid sending the problematic query names over DNS, commit +6ca53a2453598804a2559a548a08424fca96434a ("resolv: Do not send queries +for non-host-names in nss_dns [BZ #24112]") is needed. + +(cherry picked from commit 108bc4049f8ae82710aec26a92ffdb4b439c83fd) +--- + ChangeLog | 33 ++++++++ + NEWS | 4 + + include/arpa/inet.h | 6 +- + nscd/gai.c | 1 - + nscd/gethstbynm3_r.c | 2 - + nss/digits_dots.c | 3 +- + resolv/Makefile | 7 ++ + resolv/Versions | 1 + + resolv/inet_addr.c | 62 ++++++++++----- + resolv/res_init.c | 17 ++-- + resolv/tst-aton.c | 35 +++++++-- + resolv/tst-inet_aton_exact.c | 47 +++++++++++ + resolv/tst-resolv-nondecimal.c | 139 +++++++++++++++++++++++++++++++++ + resolv/tst-resolv-trailing.c | 136 ++++++++++++++++++++++++++++++++ + sysdeps/posix/getaddrinfo.c | 2 +- + 15 files changed, 455 insertions(+), 40 deletions(-) + create mode 100644 resolv/tst-inet_aton_exact.c + create mode 100644 resolv/tst-resolv-nondecimal.c + create mode 100644 resolv/tst-resolv-trailing.c + +diff --git a/include/arpa/inet.h b/include/arpa/inet.h +index c3f28f2baa..19aec74275 100644 +--- a/include/arpa/inet.h ++++ b/include/arpa/inet.h +@@ -1,10 +1,10 @@ + #include + + #ifndef _ISOMAC +-extern int __inet_aton (const char *__cp, struct in_addr *__inp); +-libc_hidden_proto (__inet_aton) ++/* Variant of inet_aton which rejects trailing garbage. */ ++extern int __inet_aton_exact (const char *__cp, struct in_addr *__inp); ++libc_hidden_proto (__inet_aton_exact) + +-libc_hidden_proto (inet_aton) + libc_hidden_proto (inet_ntop) + libc_hidden_proto (inet_pton) + extern __typeof (inet_pton) __inet_pton; +diff --git a/nscd/gai.c b/nscd/gai.c +index 24bdfee1db..f57f396f57 100644 +--- a/nscd/gai.c ++++ b/nscd/gai.c +@@ -19,7 +19,6 @@ + + /* This file uses the getaddrinfo code but it compiles it without NSCD + support. We just need a few symbol renames. */ +-#define __inet_aton inet_aton + #define __ioctl ioctl + #define __getsockname getsockname + #define __socket socket +diff --git a/nscd/gethstbynm3_r.c b/nscd/gethstbynm3_r.c +index 7beb9dce9f..f792c4fcd0 100644 +--- a/nscd/gethstbynm3_r.c ++++ b/nscd/gethstbynm3_r.c +@@ -38,8 +38,6 @@ + #define HAVE_LOOKUP_BUFFER 1 + #define HAVE_AF 1 + +-#define __inet_aton inet_aton +- + /* We are nscd, so we don't want to be talking to ourselves. */ + #undef USE_NSCD + +diff --git a/nss/digits_dots.c b/nss/digits_dots.c +index 39bff38865..5441bce16e 100644 +--- a/nss/digits_dots.c ++++ b/nss/digits_dots.c +@@ -29,7 +29,6 @@ + #include "nsswitch.h" + + #ifdef USE_NSCD +-# define inet_aton __inet_aton + # include + #endif + +@@ -160,7 +159,7 @@ __nss_hostname_digits_dots_context (struct resolv_context *ctx, + 255.255.255.255? The test below will succeed + spuriously... ??? */ + if (af == AF_INET) +- ok = __inet_aton (name, (struct in_addr *) host_addr); ++ ok = __inet_aton_exact (name, (struct in_addr *) host_addr); + else + { + assert (af == AF_INET6); +diff --git a/resolv/Makefile b/resolv/Makefile +index ea395ac3eb..d36eedd34a 100644 +--- a/resolv/Makefile ++++ b/resolv/Makefile +@@ -34,6 +34,9 @@ routines := herror inet_addr inet_ntop inet_pton nsap_addr res_init \ + tests = tst-aton tst-leaks tst-inet_ntop + xtests = tst-leaks2 + ++tests-internal += tst-inet_aton_exact ++ ++ + generate := mtrace-tst-leaks.out tst-leaks.mtrace tst-leaks2.mtrace + + extra-libs := libresolv libnss_dns +@@ -54,8 +57,10 @@ tests += \ + tst-resolv-binary \ + tst-resolv-edns \ + tst-resolv-network \ ++ tst-resolv-nondecimal \ + tst-resolv-res_init-multi \ + tst-resolv-search \ ++ tst-resolv-trailing \ + + # These tests need libdl. + ifeq (yes,$(build-shared)) +@@ -190,9 +195,11 @@ $(objpfx)tst-resolv-res_init-multi: $(objpfx)libresolv.so \ + $(shared-thread-library) + $(objpfx)tst-resolv-res_init-thread: $(libdl) $(objpfx)libresolv.so \ + $(shared-thread-library) ++$(objpfx)tst-resolv-nondecimal: $(objpfx)libresolv.so $(shared-thread-library) + $(objpfx)tst-resolv-qtypes: $(objpfx)libresolv.so $(shared-thread-library) + $(objpfx)tst-resolv-rotate: $(objpfx)libresolv.so $(shared-thread-library) + $(objpfx)tst-resolv-search: $(objpfx)libresolv.so $(shared-thread-library) ++$(objpfx)tst-resolv-trailing: $(objpfx)libresolv.so $(shared-thread-library) + $(objpfx)tst-resolv-threads: \ + $(libdl) $(objpfx)libresolv.so $(shared-thread-library) + $(objpfx)tst-resolv-canonname: \ +diff --git a/resolv/Versions b/resolv/Versions +index b05778d965..9a82704af7 100644 +--- a/resolv/Versions ++++ b/resolv/Versions +@@ -27,6 +27,7 @@ libc { + __h_errno; __resp; + + __res_iclose; ++ __inet_aton_exact; + __inet_pton_length; + __resolv_context_get; + __resolv_context_get_preinit; +diff --git a/resolv/inet_addr.c b/resolv/inet_addr.c +index 32f58b0e13..41b6166a5b 100644 +--- a/resolv/inet_addr.c ++++ b/resolv/inet_addr.c +@@ -96,26 +96,14 @@ + #include + #include + +-/* ASCII IPv4 Internet address interpretation routine. The value +- returned is in network order. */ +-in_addr_t +-__inet_addr (const char *cp) +-{ +- struct in_addr val; +- +- if (__inet_aton (cp, &val)) +- return val.s_addr; +- return INADDR_NONE; +-} +-weak_alias (__inet_addr, inet_addr) +- + /* Check whether "cp" is a valid ASCII representation of an IPv4 + Internet address and convert it to a binary address. Returns 1 if + the address is valid, 0 if not. This replaces inet_addr, the + return value from which cannot distinguish between failure and a +- local broadcast address. */ +-int +-__inet_aton (const char *cp, struct in_addr *addr) ++ local broadcast address. Write a pointer to the first ++ non-converted character to *endp. */ ++static int ++inet_aton_end (const char *cp, struct in_addr *addr, const char **endp) + { + static const in_addr_t max[4] = { 0xffffffff, 0xffffff, 0xffff, 0xff }; + in_addr_t val; +@@ -180,6 +168,7 @@ __inet_aton (const char *cp, struct in_addr *addr) + + if (addr != NULL) + addr->s_addr = res.word | htonl (val); ++ *endp = cp; + + __set_errno (saved_errno); + return 1; +@@ -188,6 +177,41 @@ __inet_aton (const char *cp, struct in_addr *addr) + __set_errno (saved_errno); + return 0; + } +-weak_alias (__inet_aton, inet_aton) +-libc_hidden_def (__inet_aton) +-libc_hidden_weak (inet_aton) ++ ++int ++__inet_aton_exact (const char *cp, struct in_addr *addr) ++{ ++ struct in_addr val; ++ const char *endp; ++ /* Check that inet_aton_end parsed the entire string. */ ++ if (inet_aton_end (cp, &val, &endp) != 0 && *endp == 0) ++ { ++ *addr = val; ++ return 1; ++ } ++ else ++ return 0; ++} ++libc_hidden_def (__inet_aton_exact) ++ ++/* inet_aton ignores trailing garbage. */ ++int ++__inet_aton_ignore_trailing (const char *cp, struct in_addr *addr) ++{ ++ const char *endp; ++ return inet_aton_end (cp, addr, &endp); ++} ++weak_alias (__inet_aton_ignore_trailing, inet_aton) ++ ++/* ASCII IPv4 Internet address interpretation routine. The value ++ returned is in network order. */ ++in_addr_t ++__inet_addr (const char *cp) ++{ ++ struct in_addr val; ++ const char *endp; ++ if (inet_aton_end (cp, &val, &endp)) ++ return val.s_addr; ++ return INADDR_NONE; ++} ++weak_alias (__inet_addr, inet_addr) +diff --git a/resolv/res_init.c b/resolv/res_init.c +index f5e52cbbb9..94743a252e 100644 +--- a/resolv/res_init.c ++++ b/resolv/res_init.c +@@ -399,8 +399,16 @@ res_vinit_1 (FILE *fp, struct resolv_conf_parser *parser) + cp = parser->buffer + sizeof ("nameserver") - 1; + while (*cp == ' ' || *cp == '\t') + cp++; ++ ++ /* Ignore trailing contents on the name server line. */ ++ { ++ char *el; ++ if ((el = strpbrk (cp, " \t\n")) != NULL) ++ *el = '\0'; ++ } ++ + struct sockaddr *sa; +- if ((*cp != '\0') && (*cp != '\n') && __inet_aton (cp, &a)) ++ if ((*cp != '\0') && (*cp != '\n') && __inet_aton_exact (cp, &a)) + { + sa = allocate_address_v4 (a, NAMESERVER_PORT); + if (sa == NULL) +@@ -410,9 +418,6 @@ res_vinit_1 (FILE *fp, struct resolv_conf_parser *parser) + { + struct in6_addr a6; + char *el; +- +- if ((el = strpbrk (cp, " \t\n")) != NULL) +- *el = '\0'; + if ((el = strchr (cp, SCOPE_DELIMITER)) != NULL) + *el = '\0'; + if ((*cp != '\0') && (__inet_pton (AF_INET6, cp, &a6) > 0)) +@@ -472,7 +477,7 @@ res_vinit_1 (FILE *fp, struct resolv_conf_parser *parser) + char separator = *cp; + *cp = 0; + struct resolv_sortlist_entry e; +- if (__inet_aton (net, &a)) ++ if (__inet_aton_exact (net, &a)) + { + e.addr = a; + if (is_sort_mask (separator)) +@@ -484,7 +489,7 @@ res_vinit_1 (FILE *fp, struct resolv_conf_parser *parser) + cp++; + separator = *cp; + *cp = 0; +- if (__inet_aton (net, &a)) ++ if (__inet_aton_exact (net, &a)) + e.mask = a.s_addr; + else + e.mask = net_mask (e.addr); +diff --git a/resolv/tst-aton.c b/resolv/tst-aton.c +index 08110a007a..eb734d7758 100644 +--- a/resolv/tst-aton.c ++++ b/resolv/tst-aton.c +@@ -1,11 +1,29 @@ ++/* Test legacy IPv4 text-to-address function inet_aton. ++ Copyright (C) 1998-2019 Free Software Foundation, Inc. ++ This file is part of the GNU C Library. ++ ++ The GNU C Library is free software; you can redistribute it and/or ++ modify it under the terms of the GNU Lesser General Public ++ License as published by the Free Software Foundation; either ++ version 2.1 of the License, or (at your option) any later version. ++ ++ The GNU C Library is distributed in the hope that it will be useful, ++ but WITHOUT ANY WARRANTY; without even the implied warranty of ++ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ++ Lesser General Public License for more details. ++ ++ You should have received a copy of the GNU Lesser General Public ++ License along with the GNU C Library; if not, see ++ . */ ++ ++#include + #include + #include + #include + #include + #include + +- +-static struct tests ++static const struct tests + { + const char *input; + int valid; +@@ -16,6 +34,7 @@ static struct tests + { "-1", 0, 0 }, + { "256", 1, 0x00000100 }, + { "256.", 0, 0 }, ++ { "255a", 0, 0 }, + { "256a", 0, 0 }, + { "0x100", 1, 0x00000100 }, + { "0200.0x123456", 1, 0x80123456 }, +@@ -40,7 +59,12 @@ static struct tests + { "1.2.256.4", 0, 0 }, + { "1.2.3.0x100", 0, 0 }, + { "323543357756889", 0, 0 }, +- { "10.1.2.3.4", 0, 0}, ++ { "10.1.2.3.4", 0, 0 }, ++ { "192.0.2.1", 1, 0xc0000201 }, ++ { "192.0.2.2\nX", 1, 0xc0000202 }, ++ { "192.0.2.3 Y", 1, 0xc0000203 }, ++ { "192.0.2.3Z", 0, 0 }, ++ { "192.000.002.010", 1, 0xc0000208 }, + }; + + +@@ -50,7 +74,7 @@ do_test (void) + int result = 0; + size_t cnt; + +- for (cnt = 0; cnt < sizeof (tests) / sizeof (tests[0]); ++cnt) ++ for (cnt = 0; cnt < array_length (tests); ++cnt) + { + struct in_addr addr; + +@@ -73,5 +97,4 @@ do_test (void) + return result; + } + +-#define TEST_FUNCTION do_test () +-#include "../test-skeleton.c" ++#include +diff --git a/resolv/tst-inet_aton_exact.c b/resolv/tst-inet_aton_exact.c +new file mode 100644 +index 0000000000..0fdfa3d6aa +--- /dev/null ++++ b/resolv/tst-inet_aton_exact.c +@@ -0,0 +1,47 @@ ++/* Test internal legacy IPv4 text-to-address function __inet_aton_exact. ++ Copyright (C) 2019 Free Software Foundation, Inc. ++ This file is part of the GNU C Library. ++ ++ The GNU C Library is free software; you can redistribute it and/or ++ modify it under the terms of the GNU Lesser General Public ++ License as published by the Free Software Foundation; either ++ version 2.1 of the License, or (at your option) any later version. ++ ++ The GNU C Library is distributed in the hope that it will be useful, ++ but WITHOUT ANY WARRANTY; without even the implied warranty of ++ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ++ Lesser General Public License for more details. ++ ++ You should have received a copy of the GNU Lesser General Public ++ License along with the GNU C Library; if not, see ++ . */ ++ ++#include ++#include ++ ++static int ++do_test (void) ++{ ++ struct in_addr addr = { }; ++ ++ TEST_COMPARE (__inet_aton_exact ("192.0.2.1", &addr), 1); ++ TEST_COMPARE (ntohl (addr.s_addr), 0xC0000201); ++ ++ TEST_COMPARE (__inet_aton_exact ("192.000.002.010", &addr), 1); ++ TEST_COMPARE (ntohl (addr.s_addr), 0xC0000208); ++ TEST_COMPARE (__inet_aton_exact ("0xC0000234", &addr), 1); ++ TEST_COMPARE (ntohl (addr.s_addr), 0xC0000234); ++ ++ /* Trailing content is not accepted. */ ++ TEST_COMPARE (__inet_aton_exact ("192.0.2.2X", &addr), 0); ++ TEST_COMPARE (__inet_aton_exact ("192.0.2.3 Y", &addr), 0); ++ TEST_COMPARE (__inet_aton_exact ("192.0.2.4\nZ", &addr), 0); ++ TEST_COMPARE (__inet_aton_exact ("192.0.2.5\tT", &addr), 0); ++ TEST_COMPARE (__inet_aton_exact ("192.0.2.6 Y", &addr), 0); ++ TEST_COMPARE (__inet_aton_exact ("192.0.2.7\n", &addr), 0); ++ TEST_COMPARE (__inet_aton_exact ("192.0.2.8\t", &addr), 0); ++ ++ return 0; ++} ++ ++#include +diff --git a/resolv/tst-resolv-nondecimal.c b/resolv/tst-resolv-nondecimal.c +new file mode 100644 +index 0000000000..a0df6f332a +--- /dev/null ++++ b/resolv/tst-resolv-nondecimal.c +@@ -0,0 +1,139 @@ ++/* Test name resolution behavior for octal, hexadecimal IPv4 addresses. ++ Copyright (C) 2019 Free Software Foundation, Inc. ++ This file is part of the GNU C Library. ++ ++ The GNU C Library is free software; you can redistribute it and/or ++ modify it under the terms of the GNU Lesser General Public ++ License as published by the Free Software Foundation; either ++ version 2.1 of the License, or (at your option) any later version. ++ ++ The GNU C Library is distributed in the hope that it will be useful, ++ but WITHOUT ANY WARRANTY; without even the implied warranty of ++ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ++ Lesser General Public License for more details. ++ ++ You should have received a copy of the GNU Lesser General Public ++ License along with the GNU C Library; if not, see ++ . */ ++ ++#include ++#include ++#include ++#include ++#include ++#include ++ ++static void ++response (const struct resolv_response_context *ctx, ++ struct resolv_response_builder *b, ++ const char *qname, uint16_t qclass, uint16_t qtype) ++{ ++ /* The tests are not supposed send any DNS queries. */ ++ FAIL_EXIT1 ("unexpected DNS query for %s/%d/%d", qname, qclass, qtype); ++} ++ ++static void ++run_query_addrinfo (const char *query, const char *address) ++{ ++ char *quoted_query = support_quote_string (query); ++ ++ struct addrinfo *ai; ++ struct addrinfo hints = ++ { ++ .ai_socktype = SOCK_STREAM, ++ .ai_protocol = IPPROTO_TCP, ++ }; ++ ++ char *context = xasprintf ("getaddrinfo \"%s\" AF_INET", quoted_query); ++ char *expected = xasprintf ("address: STREAM/TCP %s 80\n", address); ++ hints.ai_family = AF_INET; ++ int ret = getaddrinfo (query, "80", &hints, &ai); ++ check_addrinfo (context, ai, ret, expected); ++ if (ret == 0) ++ freeaddrinfo (ai); ++ free (context); ++ ++ context = xasprintf ("getaddrinfo \"%s\" AF_UNSPEC", quoted_query); ++ hints.ai_family = AF_UNSPEC; ++ ret = getaddrinfo (query, "80", &hints, &ai); ++ check_addrinfo (context, ai, ret, expected); ++ if (ret == 0) ++ freeaddrinfo (ai); ++ free (expected); ++ free (context); ++ ++ context = xasprintf ("getaddrinfo \"%s\" AF_INET6", quoted_query); ++ expected = xasprintf ("flags: AI_V4MAPPED\n" ++ "address: STREAM/TCP ::ffff:%s 80\n", ++ address); ++ hints.ai_family = AF_INET6; ++ hints.ai_flags = AI_V4MAPPED; ++ ret = getaddrinfo (query, "80", &hints, &ai); ++ check_addrinfo (context, ai, ret, expected); ++ if (ret == 0) ++ freeaddrinfo (ai); ++ free (expected); ++ free (context); ++ ++ free (quoted_query); ++} ++ ++static void ++run_query (const char *query, const char *address) ++{ ++ char *quoted_query = support_quote_string (query); ++ char *context = xasprintf ("gethostbyname (\"%s\")", quoted_query); ++ char *expected = xasprintf ("name: %s\n" ++ "address: %s\n", query, address); ++ check_hostent (context, gethostbyname (query), expected); ++ free (context); ++ ++ context = xasprintf ("gethostbyname_r \"%s\"", quoted_query); ++ struct hostent storage; ++ char buf[4096]; ++ struct hostent *e = NULL; ++ TEST_COMPARE (gethostbyname_r (query, &storage, buf, sizeof (buf), ++ &e, &h_errno), 0); ++ check_hostent (context, e, expected); ++ free (context); ++ ++ context = xasprintf ("gethostbyname2 (\"%s\", AF_INET)", quoted_query); ++ check_hostent (context, gethostbyname2 (query, AF_INET), expected); ++ free (context); ++ ++ context = xasprintf ("gethostbyname2_r \"%s\" AF_INET", quoted_query); ++ e = NULL; ++ TEST_COMPARE (gethostbyname2_r (query, AF_INET, &storage, buf, sizeof (buf), ++ &e, &h_errno), 0); ++ check_hostent (context, e, expected); ++ free (context); ++ free (expected); ++ ++ free (quoted_query); ++ ++ /* The gethostbyname tests are always valid for getaddrinfo, but not ++ vice versa. */ ++ run_query_addrinfo (query, address); ++} ++ ++static int ++do_test (void) ++{ ++ struct resolv_test *aux = resolv_test_start ++ ((struct resolv_redirect_config) ++ { ++ .response_callback = response, ++ }); ++ ++ run_query ("192.000.002.010", "192.0.2.8"); ++ ++ /* Hexadecimal numbers are not accepted by gethostbyname. */ ++ run_query_addrinfo ("0xc0000210", "192.0.2.16"); ++ run_query_addrinfo ("192.0x234", "192.0.2.52"); ++ ++ resolv_test_end (aux); ++ ++ return 0; ++} ++ ++#include +diff --git a/resolv/tst-resolv-trailing.c b/resolv/tst-resolv-trailing.c +new file mode 100644 +index 0000000000..7504bdae57 +--- /dev/null ++++ b/resolv/tst-resolv-trailing.c +@@ -0,0 +1,136 @@ ++/* Test name resolution behavior with trailing characters. ++ Copyright (C) 2019 Free Software Foundation, Inc. ++ This file is part of the GNU C Library. ++ ++ The GNU C Library is free software; you can redistribute it and/or ++ modify it under the terms of the GNU Lesser General Public ++ License as published by the Free Software Foundation; either ++ version 2.1 of the License, or (at your option) any later version. ++ ++ The GNU C Library is distributed in the hope that it will be useful, ++ but WITHOUT ANY WARRANTY; without even the implied warranty of ++ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ++ Lesser General Public License for more details. ++ ++ You should have received a copy of the GNU Lesser General Public ++ License along with the GNU C Library; if not, see ++ . */ ++ ++#include ++#include ++#include ++#include ++#include ++#include ++ ++static void ++response (const struct resolv_response_context *ctx, ++ struct resolv_response_builder *b, ++ const char *qname, uint16_t qclass, uint16_t qtype) ++{ ++ /* The tests are not supposed send any DNS queries. */ ++ FAIL_EXIT1 ("unexpected DNS query for %s/%d/%d", qname, qclass, qtype); ++} ++ ++static int ++do_test (void) ++{ ++ struct resolv_test *aux = resolv_test_start ++ ((struct resolv_redirect_config) ++ { ++ .response_callback = response, ++ }); ++ ++ static const char *const queries[] = ++ { ++ "192.0.2.1 ", ++ "192.0.2.2\t", ++ "192.0.2.3\n", ++ "192.0.2.4 X", ++ "192.0.2.5\tY", ++ "192.0.2.6\nZ", ++ "192.0.2. ", ++ "192.0.2.\t", ++ "192.0.2.\n", ++ "192.0.2. X", ++ "192.0.2.\tY", ++ "192.0.2.\nZ", ++ "2001:db8::1 ", ++ "2001:db8::2\t", ++ "2001:db8::3\n", ++ "2001:db8::4 X", ++ "2001:db8::5\tY", ++ "2001:db8::6\nZ", ++ }; ++ for (size_t query_idx = 0; query_idx < array_length (queries); ++query_idx) ++ { ++ const char *query = queries[query_idx]; ++ struct hostent storage; ++ char buf[4096]; ++ struct hostent *e; ++ ++ h_errno = 0; ++ TEST_VERIFY (gethostbyname (query) == NULL); ++ TEST_COMPARE (h_errno, HOST_NOT_FOUND); ++ ++ h_errno = 0; ++ e = NULL; ++ TEST_COMPARE (gethostbyname_r (query, &storage, buf, sizeof (buf), ++ &e, &h_errno), 0); ++ TEST_VERIFY (e == NULL); ++ TEST_COMPARE (h_errno, HOST_NOT_FOUND); ++ ++ h_errno = 0; ++ TEST_VERIFY (gethostbyname2 (query, AF_INET) == NULL); ++ TEST_COMPARE (h_errno, HOST_NOT_FOUND); ++ ++ h_errno = 0; ++ e = NULL; ++ TEST_COMPARE (gethostbyname2_r (query, AF_INET, ++ &storage, buf, sizeof (buf), ++ &e, &h_errno), 0); ++ TEST_VERIFY (e == NULL); ++ TEST_COMPARE (h_errno, HOST_NOT_FOUND); ++ ++ h_errno = 0; ++ TEST_VERIFY (gethostbyname2 (query, AF_INET6) == NULL); ++ TEST_COMPARE (h_errno, HOST_NOT_FOUND); ++ ++ h_errno = 0; ++ e = NULL; ++ TEST_COMPARE (gethostbyname2_r (query, AF_INET6, ++ &storage, buf, sizeof (buf), ++ &e, &h_errno), 0); ++ TEST_VERIFY (e == NULL); ++ TEST_COMPARE (h_errno, HOST_NOT_FOUND); ++ ++ static const int gai_flags[] = ++ { ++ 0, ++ AI_ADDRCONFIG, ++ AI_NUMERICHOST, ++ AI_IDN, ++ AI_IDN | AI_NUMERICHOST, ++ AI_V4MAPPED, ++ AI_V4MAPPED | AI_NUMERICHOST, ++ }; ++ for (size_t gai_flags_idx; gai_flags_idx < array_length (gai_flags); ++ ++gai_flags_idx) ++ { ++ struct addrinfo hints = { .ai_flags = gai_flags[gai_flags_idx], }; ++ struct addrinfo *ai; ++ hints.ai_family = AF_INET; ++ TEST_COMPARE (getaddrinfo (query, "80", &hints, &ai), EAI_NONAME); ++ hints.ai_family = AF_INET6; ++ TEST_COMPARE (getaddrinfo (query, "80", &hints, &ai), EAI_NONAME); ++ hints.ai_family = AF_UNSPEC; ++ TEST_COMPARE (getaddrinfo (query, "80", &hints, &ai), EAI_NONAME); ++ } ++ }; ++ ++ resolv_test_end (aux); ++ ++ return 0; ++} ++ ++#include +diff --git a/sysdeps/posix/getaddrinfo.c b/sysdeps/posix/getaddrinfo.c +index 553833d1f2..c91b281e31 100644 +--- a/sysdeps/posix/getaddrinfo.c ++++ b/sysdeps/posix/getaddrinfo.c +@@ -488,7 +488,7 @@ gaih_inet (const char *name, const struct gaih_service *service, + malloc_name = true; + } + +- if (__inet_aton (name, (struct in_addr *) at->addr) != 0) ++ if (__inet_aton_exact (name, (struct in_addr *) at->addr) != 0) + { + if (req->ai_family == AF_UNSPEC || req->ai_family == AF_INET) + at->family = AF_INET; +-- +2.20.1 + + +From c533244b8e00ae701583ec50aeb43377d292452d Mon Sep 17 00:00:00 2001 +From: Florian Weimer +Date: Mon, 4 Feb 2019 20:07:18 +0100 +Subject: [PATCH 4/4] nscd: Do not use __inet_aton_exact@GLIBC_PRIVATE [BZ + #20018] + +This commit avoids referencing the __inet_aton_exact@GLIBC_PRIVATE +symbol from nscd. In master, the separately-compiled getaddrinfo +implementation in nscd needs it, however such an internal ABI change +is not desirable on a release branch if it can be avoided. +--- + ChangeLog | 10 ++++++++++ + nscd/Makefile | 2 +- + nscd/gai.c | 6 ++++++ + nscd/nscd-inet_addr.c | 32 ++++++++++++++++++++++++++++++++ + 4 files changed, 49 insertions(+), 1 deletion(-) + create mode 100644 nscd/nscd-inet_addr.c + +diff --git a/nscd/Makefile b/nscd/Makefile +index b713a84c49..eb23c01a39 100644 +--- a/nscd/Makefile ++++ b/nscd/Makefile +@@ -36,7 +36,7 @@ nscd-modules := nscd connections pwdcache getpwnam_r getpwuid_r grpcache \ + getsrvbynm_r getsrvbypt_r servicescache \ + dbg_log nscd_conf nscd_stat cache mem nscd_setup_thread \ + xmalloc xstrdup aicache initgrcache gai res_hconf \ +- netgroupcache ++ netgroupcache nscd-inet_addr + + ifeq ($(build-nscd)$(have-thread-library),yesyes) + +diff --git a/nscd/gai.c b/nscd/gai.c +index f57f396f57..68a4abd30e 100644 +--- a/nscd/gai.c ++++ b/nscd/gai.c +@@ -33,6 +33,12 @@ + #define __getifaddrs getifaddrs + #define __freeifaddrs freeifaddrs + ++/* We do not want to export __inet_aton_exact. Get the prototype and ++ change its visibility to hidden. */ ++#include ++__typeof__ (__inet_aton_exact) __inet_aton_exact ++ __attribute__ ((visibility ("hidden"))); ++ + /* We are nscd, so we don't want to be talking to ourselves. */ + #undef USE_NSCD + +diff --git a/nscd/nscd-inet_addr.c b/nscd/nscd-inet_addr.c +new file mode 100644 +index 0000000000..f366b9567d +--- /dev/null ++++ b/nscd/nscd-inet_addr.c +@@ -0,0 +1,32 @@ ++/* Legacy IPv4 text-to-address functions. Version for nscd. ++ Copyright (C) 2019 Free Software Foundation, Inc. ++ This file is part of the GNU C Library. ++ ++ The GNU C Library is free software; you can redistribute it and/or ++ modify it under the terms of the GNU Lesser General Public ++ License as published by the Free Software Foundation; either ++ version 2.1 of the License, or (at your option) any later version. ++ ++ The GNU C Library is distributed in the hope that it will be useful, ++ but WITHOUT ANY WARRANTY; without even the implied warranty of ++ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ++ Lesser General Public License for more details. ++ ++ You should have received a copy of the GNU Lesser General Public ++ License along with the GNU C Library; if not, see ++ . */ ++ ++#include ++ ++/* We do not want to export __inet_aton_exact. Get the prototype and ++ change the visibility to hidden. */ ++#include ++__typeof__ (__inet_aton_exact) __inet_aton_exact ++ __attribute__ ((visibility ("hidden"))); ++ ++/* Do not provide definitions of the public symbols exported from ++ libc. */ ++#undef weak_alias ++#define weak_alias(from, to) ++ ++#include +-- +2.20.1