From patchwork Wed Apr 29 20:07:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 185849 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp1178087ilf; Wed, 29 Apr 2020 13:08:43 -0700 (PDT) X-Google-Smtp-Source: APiQypIk0Yz3ay7Ro5cO/bEtVxHHPMI5WE+z48ua7gFPOI9kG6KnCFgFxD66iaO2s2ZcZnMUz4Pi X-Received: by 2002:a37:6c7:: with SMTP id 190mr139952qkg.281.1588190923787; Wed, 29 Apr 2020 13:08:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1588190923; cv=none; d=google.com; s=arc-20160816; b=FXm+NTAQXLRwJBJtKTBmaABeLvNwW+qJQubknv6FXZoOn8JgrkUaPbg8bhxsP1ViFA +cKUrydsVVd0/2RWOSc1EeUIFODELLlNJKh+70oTTLnL9ko7+WEottSbLe16ALFlZJU/ go1ix/9KO3u62yLDnPwDu1DB4wxtvzhAuMBWegmTaUdlnQptjxqUxuSsPpYSSdnk7RKg P55mYm1393wkIRqbrgxz7WMF1BVpYGjX22SYtiy3tMq8hubLnDlp41qGCZu0CyuYVkuY 0rOA8UBToYveNc5GKbIdzeOPhlXWYOSYGhUNNZUm3BRMUZu4FoiizBWvxEOZjZgqJuc3 07tQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:message-id:date:subject:to:from:dkim-signature; bh=MWJD9VHKHIRcnOU6dGwwOL9CEZLQrTExPBPRSPkn0ts=; b=mXZenTHCRl5PtkHO+ZmsuwXoGXBNYF6HPKdGzZEZDlICIwudylJwcfiW+yGTDVEtky oGXQxmrbrN1xhj3FSDdCSRWdIwRkoUXbLKRMCwZ1YZLV8PWtw0n6I1GATDpdgiRsWegz VFwjIu1DlmNGVgeqpzVgG7xdT2X6rvQTkiMdvpygwNftn1azrnpr7mNun0kMcBtsHIrB eXl8TD3K2+i8XxbP2ZR+uE/+zaVe+cCabVBuLDynhDe1VL7H+Nq4PmY2RmmNTTgVkaZA pMM5CwOhW31UyGcNWn9KXzdyUn0k++I8sXrs82Fthvlpw5Xwzl0eqrSqHewkEMFTtzQq YawA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=w1nNbUHk; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:470:142::17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:470:142::17]) by mx.google.com with ESMTPS id y62si170111qka.278.2020.04.29.13.08.43 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 29 Apr 2020 13:08:43 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:470:142::17 as permitted sender) client-ip=2001:470:142::17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=w1nNbUHk; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:470:142::17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:51594 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jTt0d-0006vG-8q for patch@linaro.org; Wed, 29 Apr 2020 16:08:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51014) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jTt01-0006ti-Uq for qemu-devel@nongnu.org; Wed, 29 Apr 2020 16:08:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jTt00-00029j-S2 for qemu-devel@nongnu.org; Wed, 29 Apr 2020 16:08:05 -0400 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]:33897) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jTt00-00029P-Da for qemu-devel@nongnu.org; Wed, 29 Apr 2020 16:08:04 -0400 Received: by mail-wm1-x333.google.com with SMTP id v4so5777906wme.1 for ; Wed, 29 Apr 2020 13:08:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=MWJD9VHKHIRcnOU6dGwwOL9CEZLQrTExPBPRSPkn0ts=; b=w1nNbUHkVU99YI78lv/7S7UN0y4LHIC9tS26xgZ1+593Y8QShmMFfSJ/qwDp3O+9JL nsLoVPAoU6Lnj+HCOOWZQoqVQ1v09QtaZjVVmI+S42Gvg7QBuRg4vNoRv3sejhDHB2NR jUn0GjbWF+iYikK79VgrFQl6uRb1Ohb7fiBXvL4f7bS9y3ZirfCrrA/c1sYNddpfjAku i6E/AvL6RYFfpDBxicH+oKag4HYCtM5oWvlElCz9TAbihfva/28vyygnin/rAMyw2V/q eCyTdhpRopRO3OmC0wGEklwmOe/2bPTT0UusnbqBdXBLZg7/9CeIWWG/xSlRZACctcxx 4ztg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=MWJD9VHKHIRcnOU6dGwwOL9CEZLQrTExPBPRSPkn0ts=; b=RlrojpAwmdkMSBVIWsfNJURQW7si3xBRMGrM9PX0jJageWCkAykHsQjr+FHjhe0tFU AmhNuu1gSYtodp41oou15E6hUkQSV1I8h4W3YQ0gC2Mm2GDRROwesjk+/kUaoyBTHDOj zWCAfjVwS2ToZ5c2bZVMb7a9X3o6mol8iKivSwaj04XvWA6WNl9VJ/PpZO0ZOMQ0J+uX 2z0kGvLz0+XpSSDrEnjuTHiNboSB1w7hn9ppbn19M/GPakD1kMCVkn7jXaAJp4UKf9m7 sTRIIF5YQ4hqW35smSQ+ukeJ9ogW9e5MSU219DoLjcMuRz06Ps9v5ooZpbbJ2N1RkEF5 +uZw== X-Gm-Message-State: AGi0PubeMXlgyUK42OUU2bbfQ52rnWBbCygIpw8l9/JuyyHR5J8Yx+kz EAeAVikm2KsS/L5rQI5fBsuhFQ== X-Received: by 2002:a1c:4b16:: with SMTP id y22mr5091413wma.170.1588190882607; Wed, 29 Apr 2020 13:08:02 -0700 (PDT) Received: from zen.linaroharston ([51.148.130.216]) by smtp.gmail.com with ESMTPSA id g186sm9459737wme.7.2020.04.29.13.08.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Apr 2020 13:08:00 -0700 (PDT) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id AE6381FF7E; Wed, 29 Apr 2020 21:07:59 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v2] plugins: new lockstep plugin for debugging TCG changes Date: Wed, 29 Apr 2020 21:07:54 +0100 Message-Id: <20200429200754.18327-1-alex.bennee@linaro.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::333; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x333.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2a00:1450:4864:20::333 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?utf-8?q?Alex_Benn=C3=A9e?= , Mark Cave-Ayland , richard.henderson@linaro.org, =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" When we make changes to the TCG we sometimes cause regressions that are deep into the execution cycle of the guest. Debugging this often requires comparing large volumes of trace information to figure out where behaviour has diverged. The lockstep plugin utilises a shared socket so two QEMU's running with the plugin will write their current execution position and wait to receive the position of their partner process. When execution diverges the plugins output where they were and the previous few blocks before unloading themselves and letting execution continue. Originally I planned for this to be most useful with -icount but it turns out you can get divergence pretty quickly due to asynchronous qemu_cpu_kick_rr_cpus() events causing one side to eventually run into a short block a few cycles before the other side. For this reason I've added a bit of tracking and I think the divergence reporting could be finessed to report only if we really start to diverge in execution. An example run would be: qemu-system-sparc -monitor none -parallel none -net none \ -M SS-20 -m 256 -kernel day11/zImage.elf \ -plugin ./tests/plugin/liblockstep.so,arg=lockstep-sparc.sock \ -d plugin,nochain with an identical command in another window in the same working directory. Signed-off-by: Alex Bennée Cc: Richard Henderson Cc: Mark Cave-Ayland --- v2 - track BlockInfo in a separate list to avoid double-free - cleaner abstraction between run/exec/log state - exit cleanly when out of sync - more verbiage and explanation in commit message - skip liblockstep in check-tcg --- tests/plugin/lockstep.c | 307 ++++++++++++++++++++++++++++++++++++++ tests/plugin/Makefile | 1 + tests/tcg/Makefile.target | 2 +- 3 files changed, 309 insertions(+), 1 deletion(-) create mode 100644 tests/plugin/lockstep.c -- 2.20.1 diff --git a/tests/plugin/lockstep.c b/tests/plugin/lockstep.c new file mode 100644 index 0000000000..b8a60cdc6b --- /dev/null +++ b/tests/plugin/lockstep.c @@ -0,0 +1,307 @@ +/* + * Lockstep Execution Plugin + * + * Allows you to execute two QEMU instances in lockstep and report + * when their execution diverges. This is mainly useful for developers + * who want to see where a change to TCG code generation has + * introduced a subtle and hard to find bug. + * + * Caveats: + * - single-threaded linux-user apps only with non-deterministic syscalls + * - no MTTCG enabled system emulation (icount may help) + * + * While icount makes things more deterministic it doesn't mean a + * particular run may execute the exact same sequence of blocks. An + * asynchronous event (for example X11 graphics update) may cause a + * block to end early and a new partial block to start. This means + * serial only test cases are a better bet. -d nochain may also help. + * + * This code is not thread safe! + * + * Copyright (c) 2020 Linaro Ltd + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include +#include +#include +#include +#include +#include +#include + +#include + +QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION; + +/* saved so we can uninstall later */ +static qemu_plugin_id_t our_id; + +static unsigned long bb_count; +static unsigned long insn_count; + +/* Information about a translated block */ +typedef struct { + uint64_t pc; + uint64_t insns; +} BlockInfo; + +/* Information about an execution state in the log */ +typedef struct { + BlockInfo *block; + unsigned long insn_count; + unsigned long block_count; +} ExecInfo; + +/* The execution state we compare */ +typedef struct { + uint64_t pc; + unsigned long insn_count; +} ExecState; + +/* list of translated block info */ +static GSList *blocks; + +/* execution log and points of divergence */ +static GSList *log, *divergence_log; + +static int divergence_count; +static int max_divergence = 10; + +static int socket_fd; +static char *path_to_unlink; + + +static void plugin_cleanup(qemu_plugin_id_t id) +{ + /* Free our block data */ + g_slist_free_full(blocks, &g_free); + g_slist_free_full(log, &g_free); + g_slist_free(divergence_log); + + close(socket_fd); + if (path_to_unlink) { + unlink(path_to_unlink); + } +} + +static void plugin_exit(qemu_plugin_id_t id, void *p) +{ + g_autoptr(GString) out = g_string_new("No divergence :-)\n"); + g_string_append_printf(out, "Executed %ld/%d blocks\n", + bb_count, g_slist_length(log)); + g_string_append_printf(out, "Executed ~%ld instructions\n", insn_count); + qemu_plugin_outs(out->str); + + plugin_cleanup(id); +} + +static void report_divergance(ExecState *us, ExecState *them) +{ + int i; + GSList *entry = log; + g_autoptr(GString) out = g_string_new("I feel the "); + + divergence_count++; + + g_string_append_printf(out, + "%d%s divergence in the force\n", divergence_count, + divergence_count > 3 ? "th" : + divergence_count == 3 ? "rd" : + divergence_count == 2 ? "nd" : "st"); + + g_string_append_printf(out, + "We have executed ~%ld instructions over %ld blocks\n", + insn_count, bb_count); + g_string_append_printf(out, "We are @ %#016lx vs %#016lx\n", us->pc, them->pc); + + for (i = 0; i < 3; i++) { + ExecInfo *prev = (ExecInfo *) entry->data; + g_string_append_printf(out, " previously @ %#016lx/%ld (%ld insn, %ld blocks)\n", + prev->block->pc, prev->block->insns, + prev->insn_count, prev->block_count); + entry = g_slist_next(entry); + } + + qemu_plugin_outs(out->str); + + /* Do we continue or uninstall ourselves */ + if (divergence_count > max_divergence) { + qemu_plugin_outs("too much divergence... giving up."); + qemu_plugin_uninstall(our_id, plugin_cleanup); + } +} + +static void vcpu_tb_exec(unsigned int cpu_index, void *udata) +{ + BlockInfo *bi = (BlockInfo *) udata; + ExecState us, them; + ssize_t bytes; + ExecInfo *exec; + + us.pc = bi->pc; + us.insn_count = insn_count; + + /* + * Write our current position to the other end. If we fail the + * other end has probably died and we should shut down gracefully. + */ + bytes = write(socket_fd, &us, sizeof(ExecState)); + if (bytes < sizeof(ExecState)) { + qemu_plugin_outs(bytes < 0 ? + "problem writing to socket" : + "wrote less than expected to socket"); + qemu_plugin_uninstall(our_id, plugin_cleanup); + return; + } + + /* + * Now read where our peer has reached. Again a failure probably + * indicates the other end died and we should close down cleanly. + */ + bytes = read(socket_fd, &them, sizeof(ExecState)); + if (bytes < sizeof(ExecState)) { + qemu_plugin_outs(bytes < 0 ? + "problem reading from socket" : + "read less than expected"); + qemu_plugin_uninstall(our_id, plugin_cleanup); + return; + } + + /* + * Compare and bail out if we have diverged. It may be a temporary + * blip so save the position we were in to the divergence log for + * later. + */ + if (us.pc != them.pc) { + report_divergance(&us, &them); + divergence_log = g_slist_prepend(divergence_log, log); + } + + /* + * Assume this block will execute fully and record it + * in the execution log. + */ + insn_count += bi->insns; + bb_count++; + exec = g_new0(ExecInfo, 1); + exec->block = bi; + exec->insn_count = insn_count; + exec->block_count = bb_count; + log = g_slist_prepend(log, exec); +} + +static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb) +{ + BlockInfo *bi = g_new0(BlockInfo, 1); + bi->pc = qemu_plugin_tb_vaddr(tb); + bi->insns = qemu_plugin_tb_n_insns(tb); + + // save a reference so we can free later + blocks = g_slist_prepend(blocks, bi); + qemu_plugin_register_vcpu_tb_exec_cb(tb, vcpu_tb_exec, + QEMU_PLUGIN_CB_NO_REGS, (void *)bi); +} + + +/* + * Instead of encoding master/slave status into what is essentially + * two peers we shall just take the simple approach of checking for + * the existence of the pipe and assuming if it's not there we are the + * first process. + */ +static bool setup_socket(const char *path) +{ + struct sockaddr_un sockaddr; + int fd; + + fd = socket(AF_UNIX, SOCK_STREAM, 0); + if (fd < 0) { + perror("create socket"); + return false; + } + + sockaddr.sun_family = AF_UNIX; + g_strlcpy(sockaddr.sun_path, path, sizeof(sockaddr.sun_path) - 1); + if (bind(fd, (struct sockaddr *)&sockaddr, sizeof(sockaddr)) < 0) { + perror("bind socket"); + close(fd); + return false; + } + + /* remember to clean-up */ + path_to_unlink = g_strdup(path); + + if (listen(fd, 1) < 0) { + perror("listen socket"); + close(fd); + return false; + } + + socket_fd = accept(fd, NULL, NULL); + if (socket_fd < 0 && errno != EINTR) { + perror("accept socket"); + return false; + } + + qemu_plugin_outs("setup_socket::ready\n"); + + return true; +} + +static bool connect_socket(const char *path) +{ + int fd; + struct sockaddr_un sockaddr; + + fd = socket(AF_UNIX, SOCK_STREAM, 0); + if (fd < 0) { + perror("create socket"); + return false; + } + + sockaddr.sun_family = AF_UNIX; + g_strlcpy(sockaddr.sun_path, path, sizeof(sockaddr.sun_path) - 1); + + if (connect(fd, (struct sockaddr *)&sockaddr, sizeof(sockaddr)) < 0) { + perror("failed to connect"); + return false; + } + + qemu_plugin_outs("connect_socket::ready\n"); + + socket_fd = fd; + return true; +} + +static bool setup_unix_socket(const char *path) +{ + if (g_file_test(path, G_FILE_TEST_EXISTS)) { + return connect_socket(path); + } else { + return setup_socket(path); + } +} + + +QEMU_PLUGIN_EXPORT int qemu_plugin_install(qemu_plugin_id_t id, + const qemu_info_t *info, + int argc, char **argv) +{ + if (!argc || !argv[0]) { + qemu_plugin_outs("Need a socket path to talk to other instance."); + return -1; + } + + if (!setup_unix_socket(argv[0])) { + qemu_plugin_outs("Failed to setup socket for communications."); + return -1; + } + + our_id = id; + + qemu_plugin_register_vcpu_tb_trans_cb(id, vcpu_tb_trans); + qemu_plugin_register_atexit_cb(id, plugin_exit, NULL); + return 0; +} diff --git a/tests/plugin/Makefile b/tests/plugin/Makefile index 75467b6db8..b3250e2504 100644 --- a/tests/plugin/Makefile +++ b/tests/plugin/Makefile @@ -13,6 +13,7 @@ NAMES += mem NAMES += hotblocks NAMES += howvec NAMES += hotpages +NAMES += lockstep SONAMES := $(addsuffix .so,$(addprefix lib,$(NAMES))) diff --git a/tests/tcg/Makefile.target b/tests/tcg/Makefile.target index b3cff3cad1..075daf3d22 100644 --- a/tests/tcg/Makefile.target +++ b/tests/tcg/Makefile.target @@ -128,7 +128,7 @@ RUN_TESTS=$(patsubst %,run-%, $(TESTS)) ifeq ($(CONFIG_PLUGIN),y) PLUGIN_DIR=../../plugin VPATH+=$(PLUGIN_DIR) -PLUGINS=$(notdir $(wildcard $(PLUGIN_DIR)/*.so)) +PLUGINS=$(filter-out liblockstep.so,$(notdir $(wildcard $(PLUGIN_DIR)/*.so))) # We need to ensure expand the run-plugin-TEST-with-PLUGIN # pre-requistes manually here as we can't use stems to handle it. We