From patchwork Tue Apr 4 14:59:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 670034 Delivered-To: patch@linaro.org Received: by 2002:a5d:4d08:0:0:0:0:0 with SMTP id z8csp2492358wrt; Tue, 4 Apr 2023 07:59:55 -0700 (PDT) X-Google-Smtp-Source: AKy350YGEA9mUGX17JT92Ah6brFkHbypZ+mwrOEhT2Kc5IdRFrQdKRw53QObBb0gvPeX6+FS52YG X-Received: by 2002:a05:622a:1d5:b0:3e4:e4aa:b0c4 with SMTP id t21-20020a05622a01d500b003e4e4aab0c4mr3799757qtw.67.1680620395171; Tue, 04 Apr 2023 07:59:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680620395; cv=none; d=google.com; s=arc-20160816; b=zY9VFQ8MLx5sj73paR0Y46kiMOeuiuiB07pz9ak4kGcA2OsKhcyg7PUARfunBpIlAz YKCWWSXGlghM1S7KjyWc+BP/RzlAj+nOGvUG7oYohZluXsKaFwY6SNf/hurAAvOpxODB mQeZxH7SGlGA//bZ38qImsAjVpRVhUpqLQyNLSIqIUSrLrOohD8ySKVjL4AT1SXzUcg6 vtNqYP4u0EO/Zzrm0TZsyrakQV+vTLdR/TZXjAhOKXf0iCl3S8BQqk9Bn//dKgi58Evn fhSxy9142TehVU/duIvnOmLkqC4MTLSSxgjIPHCOJfJj09vjEZizqOQYtXP02yGuu14m v5pg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=0wPinvsk7vEdGY5KkltJh2yfDiVq++tbF2ZWCeEQzY0=; b=DSSeEbcc09dx964hgJpAhyPLfZbemvTYEelM1wIvFDVUngX4RqUgTvSw4NkYvk3Srm Ii3GsmvA7DQV5e3CSCooV5N+StiLM+vkUueysVA6uqQ1mdKG/ytBed6kX7rd4WVb1xTh vGZAteXN3x3e/GFb7ErtK1ILog340KZozfBneHMAFwGOQIUHDh+ZRHacPOxFsi8KUuXy HmtJY9MJHttAgZ90QAkLpXzz6rKXiZspi8NKr0pacu2zQAmgCd6RFTkCxsd9TImfKSjR tvAsZcZ8+QNfmbJgOqQ3LnOBqO7MsvR/Hpdtqp7jquzr9HWTgRjTDuF9q9sHBwweWoGp xTxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="AQiIE/Hq"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id a22-20020a05622a02d600b003e630c10a9fsi8217968qtx.497.2023.04.04.07.59.55 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 04 Apr 2023 07:59:55 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="AQiIE/Hq"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pji8K-0004Gg-IJ; Tue, 04 Apr 2023 10:59:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pji8E-0004DJ-OT for qemu-devel@nongnu.org; Tue, 04 Apr 2023 10:59:35 -0400 Received: from mail-wm1-x332.google.com ([2a00:1450:4864:20::332]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pji87-0005AH-WF for qemu-devel@nongnu.org; Tue, 04 Apr 2023 10:59:30 -0400 Received: by mail-wm1-x332.google.com with SMTP id i5-20020a05600c354500b003edd24054e0so22012778wmq.4 for ; Tue, 04 Apr 2023 07:59:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1680620366; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0wPinvsk7vEdGY5KkltJh2yfDiVq++tbF2ZWCeEQzY0=; b=AQiIE/HqEu999x1122TC93rZcWcSNo9AhLhZXJIdi771f1vO+xVZQ9NgKcIBJKU3JE uXtiD3mCTdx8LjoYuyBOLo8sQUtVJTsfwjv87+JFXrZ4vb5MxYGEvEc4F+O9DEYWZ8ym AIKdxykb/lGe3MI7IubuN+u93JibFbmmsfFc2rYE9VBDlUDRHbFxhK/yrOKejv6yPUrL PUB3imVeQOR2DIN64YTCCOl8NSAjlGaLjIINDADAthMBpUlFi3GigBzbvk/TgVXRHalx DQLCwmxN7bPh2ld2gqSzZO8aOtYy/SVMoNWQQB63tRmNZrZjdwEoW3XGeI1jybfDsQhr VqYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680620366; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0wPinvsk7vEdGY5KkltJh2yfDiVq++tbF2ZWCeEQzY0=; b=edoXmuOsu8SsfRfe/mtYyUuAomCqoQFulT8PSZPcw0KMq6VTHBjhgJXnufN97iHEqn Se0ipwsFw4g1+NqcKkDrniJqCWdtkfqRnaiJCTKK9n6HJtHtd4DoODq5q6wncByJR29f +XBpZDzHzjDAeQSNAZq7SxEBwTP6EpaUHUzVa6QotT71knwviZtKbFsH3A7vdlxrc7LN C9V4fnDNMHPpFlLAax2NKjGjEANhN9NMh2omz7CnssfPajvg/t13muIixiNzuUMPt+LT 019W1abvWXL55ZXNXXWMh+NqNBrWRyyJdbcUQhraNCh6oAQJ0q5+KdVR3jZEcUWpyCu1 UGdg== X-Gm-Message-State: AAQBX9cZ8FuAVqpygnGuGrkLMO6tgx0fjsjXKDcPL1sceFx4EraWsrH/ Q5sQFcBSC1pwIIRHpgqshzFg8p1lHZNB8+YVnT8= X-Received: by 2002:a05:600c:b51:b0:3ed:2a91:3bc9 with SMTP id k17-20020a05600c0b5100b003ed2a913bc9mr2564726wmr.15.1680620365845; Tue, 04 Apr 2023 07:59:25 -0700 (PDT) Received: from zen.linaroharston ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id 24-20020a05600c231800b003ed2276cd0dsm15456264wmo.38.2023.04.04.07.59.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Apr 2023 07:59:25 -0700 (PDT) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id 3C33D1FFB8; Tue, 4 Apr 2023 15:59:25 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Cc: peter.maydell@linaro.org, =?utf-8?q?Alex_Benn=C3=A9e?= , Kautuk Consul , Thomas Huth Subject: [PULL 01/10] scripts/coverage: initial coverage comparison script Date: Tue, 4 Apr 2023 15:59:16 +0100 Message-Id: <20230404145925.2638152-2-alex.bennee@linaro.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230404145925.2638152-1-alex.bennee@linaro.org> References: <20230404145925.2638152-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::332; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x332.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org This is a very rough and ready first pass at comparing gcovr's json output between two different runs. At the moment it will give you a file level diff between two runs but hopefully it wont be too hard to extend to give better insight. After generating the coverage results you run with something like: ./scripts/coverage/compare_gcov_json.py \ -a ./builds/gcov.config1/coverage.json \ -b ./builds/gcov.config2/coverage.json My hope is we can use this to remove some redundancy from testing as well as evaluate if new tests are actually providing additional coverage or just burning our precious CI time. Signed-off-by: Alex Bennée Cc: Kautuk Consul Acked-by: Thomas Huth Message-Id: <20230403134920.2132362-2-alex.bennee@linaro.org> diff --git a/MAINTAINERS b/MAINTAINERS index ef45b5e71e..9e1a60ea24 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3908,3 +3908,8 @@ Performance Tools and Tests M: Ahmed Karaman S: Maintained F: scripts/performance/ + +Code Coverage Tools +M: Alex Bennée +S: Odd Fixes +F: scripts/coverage/ diff --git a/scripts/coverage/compare_gcov_json.py b/scripts/coverage/compare_gcov_json.py new file mode 100755 index 0000000000..1b92dc2c8c --- /dev/null +++ b/scripts/coverage/compare_gcov_json.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +# +# Compare output of two gcovr JSON reports and report differences. To +# generate the required output first: +# - create two build dirs with --enable-gcov +# - run set of tests in each +# - run make coverage-html in each +# - run gcovr --json --exclude-unreachable-branches \ +# --print-summary -o coverage.json --root ../../ . *.p +# +# Author: Alex Bennée +# +# SPDX-License-Identifier: GPL-2.0-or-later +# + +import argparse +import json +import sys +from pathlib import Path + +def create_parser(): + parser = argparse.ArgumentParser( + prog='compare_gcov_json', + description='analyse the differences in coverage between two runs') + + parser.add_argument('-a', type=Path, default=None, + help=('First file to check')) + + parser.add_argument('-b', type=Path, default=None, + help=('Second file to check')) + + parser.add_argument('--verbose', action='store_true', default=False, + help=('A minimal verbosity level that prints the ' + 'overall result of the check/wait')) + return parser + + +# See https://gcovr.com/en/stable/output/json.html#json-format-reference +def load_json(json_file_path: Path, verbose = False) -> dict[str, set[int]]: + + with open(json_file_path) as f: + data = json.load(f) + + root_dir = json_file_path.absolute().parent + covered_lines = dict() + + for filecov in data["files"]: + file_path = Path(filecov["file"]) + + # account for generated files - map into src tree + resolved_path = Path(file_path).absolute() + if resolved_path.is_relative_to(root_dir): + file_path = resolved_path.relative_to(root_dir) + # print(f"remapped {resolved_path} to {file_path}") + + lines = filecov["lines"] + + executed_lines = set( + linecov["line_number"] + for linecov in filecov["lines"] + if linecov["count"] != 0 and not linecov["gcovr/noncode"] + ) + + # if this file has any coverage add it to the system + if len(executed_lines) > 0: + if verbose: + print(f"file {file_path} {len(executed_lines)}/{len(lines)}") + covered_lines[str(file_path)] = executed_lines + + return covered_lines + +def find_missing_files(first, second): + """ + Return a list of files not covered in the second set + """ + missing_files = [] + for f in sorted(first): + file_a = first[f] + try: + file_b = second[f] + except KeyError: + missing_files.append(f) + + return missing_files + +def main(): + """ + Script entry point + """ + parser = create_parser() + args = parser.parse_args() + + if not args.a or not args.b: + print("We need two files to compare") + sys.exit(1) + + first_coverage = load_json(args.a, args.verbose) + second_coverage = load_json(args.b, args.verbose) + + first_missing = find_missing_files(first_coverage, + second_coverage) + + second_missing = find_missing_files(second_coverage, + first_coverage) + + a_name = args.a.parent.name + b_name = args.b.parent.name + + print(f"{b_name} missing coverage in {len(first_missing)} files") + for f in first_missing: + print(f" {f}") + + print(f"{a_name} missing coverage in {len(second_missing)} files") + for f in second_missing: + print(f" {f}") + + +if __name__ == '__main__': + main()