From patchwork Fri Jun 15 19:43:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138752 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1244810lji; Fri, 15 Jun 2018 12:44:36 -0700 (PDT) X-Google-Smtp-Source: ADUXVKL92legBRKTT5rQb8nFqRrn2cyQynupYJWP3+skIHr9A2N3DibsKJXfTaJkijX4sA0k3XOs X-Received: by 2002:a37:b603:: with SMTP id g3-v6mr2659468qkf.208.1529091876604; Fri, 15 Jun 2018 12:44:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529091876; cv=none; d=google.com; s=arc-20160816; b=SjAoyhlqONx/rDSS4Qpx7kd2fnYAJH2vvUk4YxZCzY2JedqHPBXcyTeSOc1X4z6Gw+ jh4zT2LwbHdvNHeZQFlbQh6zG1Ija034L6AEhAnYO5D3YftuTujORqSn1PT5rTXqxfrX vCeVAi0JblEiNbDDpDSib/9SaJtbejpQ1oZMQg3IHixBY2MpKC1QZJCIXMA4pK+jk/2o SNHT8yix6xyxGsiOF5aLJ390gAIiRFCs9rwh4N0XNFnvTzUFmLIWca6P9ldaYk70L6rN usqW5wJBisScruWykXbkIz2DzVKQGy8LiVKpbMqvoy02iRRfwPNsZYwAlDwdmnprWylP mCQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=8yfaMikkCfKbwAVcUNBZHbpP9x83/6cBIZm5WMmsiqk=; b=TaykK8F4pZUbahIk0yLMhuQrR7LAheLLkEeNjAfsoMOwxvi0U5j/1NwPuDz3M9dTT6 9ZWqCyhwC2ViSe3JvstJtnIlRrBjveqGdh2AP6Wa4bh9rpZgcL3AJfrufliILcjorV+m GOWQSjyujYoP/2jhrVNzB1lyD2tfIFvIFiJUz/CsnKejw5RaAjKawbsU6tFWR5ysfN+L 4Bb8C3l26EaOI9XovpPAMy5ugOl0zcgO+ZnoETH3iezDKU72JXzW3w5LK52+3aCrYiP+ jo8ODPVH+M7/u4SoiA4HGcZj1rZqmVJFL71iJkVLWEsB+1vN3nBM+Y1i9PD7UMcqVWmJ bPBA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=XcYxOWjT; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id k73-v6si7927262qke.106.2018.06.15.12.44.36 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:44:36 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=XcYxOWjT; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48955 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTueC-0000Gk-3m for patch@linaro.org; Fri, 15 Jun 2018 15:44:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51475) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudi-0000E3-3i for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTude-0002Lo-Nz for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:06 -0400 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:41327) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTude-0002LI-Gy for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:02 -0400 Received: by mail-pg0-x241.google.com with SMTP id l65-v6so4852853pgl.8 for ; Fri, 15 Jun 2018 12:44:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8yfaMikkCfKbwAVcUNBZHbpP9x83/6cBIZm5WMmsiqk=; b=XcYxOWjTiQ6BHxWDeh+Q4JM+LKrGFxfsUkG3jf5dbUjE9bzK99MEFrRBbzNp95T78W jKce6Og5L0c9cVwdzSbcIhonvmJCNgHYWUuUcOrqXwzVibJOFiyB0x+fxtIhzB5f3iFH 5geB9o45KYYh97HvkrwfyShg4Y6nOEfhN4pks= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8yfaMikkCfKbwAVcUNBZHbpP9x83/6cBIZm5WMmsiqk=; b=Hut+xX4ezs5zBLR5fram6QgAo4fi3muM6km+6WGES95hhYtP4I3RupLYBiyZEkLG7p fbDx+CaSJ+Og3wKoby6ciYh/kllJ82lxt84B6vlIOnj9tJ1s346ciITEdkIT6Z6FRqLu dx+Wpsaaoh/+B0CKpDewN9JiY3lwb7YbfpA2chOcQ7kSyQDBMZBFnk/DjrVV8A2jDXwg O52WZnvFvlYNFiNxglAE1t/bz6DtcX9C9dLzZxFxyJhXj+DXKTrEBtXTSvdIjiKbduGd p9rI+OubZd2w7lRhPsegVZUh6q5q4pcZ9rK6kuGuIcd3d7TkhuK9mRrqH5RSUVNH0F1l d8Gg== X-Gm-Message-State: APt69E2y3cbD8kLoWPWFBHQMg5zovZG/sgMWJnzKRQs3/ycuKh0IPD3U Usm/lxjN3MSMyjdxWMyCAF39nqEpR5E= X-Received: by 2002:aa7:854e:: with SMTP id y14-v6mr3302088pfn.165.1529091841277; Fri, 15 Jun 2018 12:44:01 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.43.59 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:00 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:36 -1000 Message-Id: <20180615194354.12489-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PULL v2 01/19] tcg/i386: Use byte form of xgetbv instruction X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, John Arbuckle Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: John Arbuckle The assembler in most versions of Mac OS X is pretty old and does not support the xgetbv instruction. To go around this problem, the raw encoding of the instruction is used instead. Signed-off-by: John Arbuckle Message-Id: <20180604215102.11002-1-programmingkidx@gmail.com> Signed-off-by: Richard Henderson --- tcg/i386/tcg-target.inc.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -- 2.17.1 diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index 5357909fff..0d0ff524b7 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -3501,7 +3501,10 @@ static void tcg_target_init(TCGContext *s) sure of not hitting invalid opcode. */ if (c & bit_OSXSAVE) { unsigned xcrl, xcrh; - asm ("xgetbv" : "=a" (xcrl), "=d" (xcrh) : "c" (0)); + /* The xgetbv instruction is not available to older versions of + * the assembler, so we encode the instruction manually. + */ + asm(".byte 0x0f, 0x01, 0xd0" : "=a" (xcrl), "=d" (xcrh) : "c" (0)); if ((xcrl & 6) == 6) { have_avx1 = (c & bit_AVX) != 0; have_avx2 = (b7 & bit_AVX2) != 0; From patchwork Fri Jun 15 19:43:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138754 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1247350lji; Fri, 15 Jun 2018 12:47:32 -0700 (PDT) X-Google-Smtp-Source: ADUXVKK1vQEYpRGDAjQdiIxZqg6GTiTa9BWTWBxrau7tBTS4SuPPPIxBQdFKAr8uAZhMGzyydhyx X-Received: by 2002:a0c:d19d:: with SMTP id e29-v6mr2706710qvh.129.1529092052857; Fri, 15 Jun 2018 12:47:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092052; cv=none; d=google.com; s=arc-20160816; b=j/6plRI3Sv7FM3CUWkns3BBYuyDMtpnlr1SZsinKNkmUcjU7mogsZH93yU62hyMDde s5DMjCs/ScTQBJ6HJY6BmUNGllclGaHxYk+TnhUj6AYsuRZnJVO8AXbvWPS+xKnj4AmE +CKgRBNv73ug65DCU9wv3Ab7d/6lcetJ2Z5aJw4bmtRpxQLV38utlY0xlUAjpvknInTT KRQUQ3AMf4I8lTRs0sePw3pOLDwNQomNVWeb/+La3j/YKYStJz7Xax4KbRUk/951Oezf TiOvWEnkpd9o6iAsis6loWmVvMKlkTmKLpBcNaaCuTGRp6TQYv1+BGFoOLbnOge1/Gc/ 8scA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=sz2BaebJjs1mTuxuMCcvE8lTXLIpIUPZqbhBVCI8o+M=; b=cLHsDDrFiEf13nUfG5znPhyU8T5nensZUf0w/IWIVrJLaXZkDaR9dVPT28190WURdf qGRBgG3KuXpAMNDm80gsVdWTzmtM1JAHMACuz2JR+FBW+phFlCL10LHkgzzSgnQaWD6o 0lP24URgn0yvXxy/qx9bEsHcab4e33SJJeWKIm/Co28Fiw9oAVZhThgwrXnBL+btw7Tw ZhPHS1UjHf4DmKXL53Qyf3BsAVM3TgxhBOkGN8DfBPjeMSkfwbgcDVtQj2F+onh2qbra k/CAMVjs9SL/XpqbphZ14yhGK+jUGFVcLjH8xaTewYEDl0Po7rhcwRi311P2+Nzwz68W jLBA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=EQb6muPs; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id v51-v6si2215233qta.51.2018.06.15.12.47.32 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:47:32 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=EQb6muPs; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48969 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuh2-0002kZ-9M for patch@linaro.org; Fri, 15 Jun 2018 15:47:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51490) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudi-0000E5-O5 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudg-0002NF-SZ for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:06 -0400 Received: from mail-pg0-x233.google.com ([2607:f8b0:400e:c05::233]:41044) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudg-0002Mj-K2 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:04 -0400 Received: by mail-pg0-x233.google.com with SMTP id l65-v6so4852882pgl.8 for ; Fri, 15 Jun 2018 12:44:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sz2BaebJjs1mTuxuMCcvE8lTXLIpIUPZqbhBVCI8o+M=; b=EQb6muPs3dkJrSmByp4YtjfpN5dT99ceck+u73BidTCixC+wqlJu8neCpD2rgdsOxm 83N8u2nAPz2rA4gsJWdY3rsZcWjY6FcXw5336MvbJHtNzX+ToobD2fVg+hSGqJp+gHnE v4+m9kJzmhDQ0H2HTy7mKJQM0b6Lc4jQ5PE/U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sz2BaebJjs1mTuxuMCcvE8lTXLIpIUPZqbhBVCI8o+M=; b=ACuJcVTczG8fo1L1qyXmCfyxDq5y3fG/DBGsGm6BeA9dNYY0TQfufu+o9FyXjupTnp 6KcVdfWXjMXyixN7qEvSsa8zuusSQg65gFA8NCaXSpew3LveWRMm30WWoTPW3JXPNu9y mRPnhzs30NfUcT+gBEvLjNbs382jjeMaPZSEHD9rUaNA2HdNOJwfLPi9iorfPNTANCeO pdy2rrGWamoASLrfn2jgjC/ApskJ+FkIIyJXC9dnQ+EkaQA2VNhk+WA/4hUngMKe0UGe RUYOJSDDRBt+HluZaTiW+rjWP8kGZQ/+l2ibktfrIznX2yxgWvTKxX6LC1mf/FqZlDdK Faug== X-Gm-Message-State: APt69E3UCLPNcXGfYm4vamPyFVESTDSVPULEr2J/x4Tu8gi9Jhoqzlg9 Dp5Sf8QWe4bCWZDNg9ew5KtfqRWjkcU= X-Received: by 2002:a62:e816:: with SMTP id c22-v6mr3353168pfi.124.1529091843143; Fri, 15 Jun 2018 12:44:03 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.01 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:02 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:37 -1000 Message-Id: <20180615194354.12489-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::233 Subject: [Qemu-devel] [PULL v2 02/19] qht: require a default comparison function X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" qht_lookup now uses the default cmp function. qht_lookup_custom is defined to retain the old behaviour, that is a cmp function is explicitly provided. qht_insert will gain use of the default cmp in the next patch. Note that we move qht_lookup_custom's @func to be the last argument, which makes the new qht_lookup as simple as possible. Instead of this (i.e. keeping @func 2nd): 0000000000010750 : 10750: 89 d1 mov %edx,%ecx 10752: 48 89 f2 mov %rsi,%rdx 10755: 48 8b 77 08 mov 0x8(%rdi),%rsi 10759: e9 22 ff ff ff jmpq 10680 1075e: 66 90 xchg %ax,%ax We get: 0000000000010740 : 10740: 48 8b 4f 08 mov 0x8(%rdi),%rcx 10744: e9 37 ff ff ff jmpq 10680 10749: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) Reviewed-by: Richard Henderson Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- include/qemu/qht.h | 25 ++++++++++++++++++++----- accel/tcg/cpu-exec.c | 4 ++-- accel/tcg/translate-all.c | 16 +++++++++++++++- tests/qht-bench.c | 14 +++++++------- tests/test-qht.c | 15 ++++++++++----- util/qht.c | 14 +++++++++++--- 6 files changed, 65 insertions(+), 23 deletions(-) -- 2.17.1 diff --git a/include/qemu/qht.h b/include/qemu/qht.h index 531aa95325..5f03a0f4cb 100644 --- a/include/qemu/qht.h +++ b/include/qemu/qht.h @@ -11,8 +11,11 @@ #include "qemu/thread.h" #include "qemu/qdist.h" +typedef bool (*qht_cmp_func_t)(const void *a, const void *b); + struct qht { struct qht_map *map; + qht_cmp_func_t cmp; QemuMutex lock; /* serializes setters of ht->map */ unsigned int mode; }; @@ -47,10 +50,12 @@ typedef void (*qht_iter_func_t)(struct qht *ht, void *p, uint32_t h, void *up); /** * qht_init - Initialize a QHT * @ht: QHT to be initialized + * @cmp: default comparison function. Cannot be NULL. * @n_elems: number of entries the hash table should be optimized for. * @mode: bitmask with OR'ed QHT_MODE_* */ -void qht_init(struct qht *ht, size_t n_elems, unsigned int mode); +void qht_init(struct qht *ht, qht_cmp_func_t cmp, size_t n_elems, + unsigned int mode); /** * qht_destroy - destroy a previously initialized QHT @@ -78,11 +83,11 @@ void qht_destroy(struct qht *ht); bool qht_insert(struct qht *ht, void *p, uint32_t hash); /** - * qht_lookup - Look up a pointer in a QHT + * qht_lookup_custom - Look up a pointer using a custom comparison function. * @ht: QHT to be looked up - * @func: function to compare existing pointers against @userp * @userp: pointer to pass to @func * @hash: hash of the pointer to be looked up + * @func: function to compare existing pointers against @userp * * Needs to be called under an RCU read-critical section. * @@ -94,8 +99,18 @@ bool qht_insert(struct qht *ht, void *p, uint32_t hash); * Returns the corresponding pointer when a match is found. * Returns NULL otherwise. */ -void *qht_lookup(struct qht *ht, qht_lookup_func_t func, const void *userp, - uint32_t hash); +void *qht_lookup_custom(struct qht *ht, const void *userp, uint32_t hash, + qht_lookup_func_t func); + +/** + * qht_lookup - Look up a pointer in a QHT + * @ht: QHT to be looked up + * @userp: pointer to pass to the comparison function + * @hash: hash of the pointer to be looked up + * + * Calls qht_lookup_custom() using @ht's default comparison function. + */ +void *qht_lookup(struct qht *ht, const void *userp, uint32_t hash); /** * qht_remove - remove a pointer from the hash table diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c index 4ef95d8dd3..6d6c51b686 100644 --- a/accel/tcg/cpu-exec.c +++ b/accel/tcg/cpu-exec.c @@ -295,7 +295,7 @@ struct tb_desc { uint32_t trace_vcpu_dstate; }; -static bool tb_cmp(const void *p, const void *d) +static bool tb_lookup_cmp(const void *p, const void *d) { const TranslationBlock *tb = p; const struct tb_desc *desc = d; @@ -340,7 +340,7 @@ TranslationBlock *tb_htable_lookup(CPUState *cpu, target_ulong pc, phys_pc = get_page_addr_code(desc.env, pc); desc.phys_page1 = phys_pc & TARGET_PAGE_MASK; h = tb_hash_func(phys_pc, pc, flags, cf_mask, *cpu->trace_dstate); - return qht_lookup(&tb_ctx.htable, tb_cmp, &desc, h); + return qht_lookup_custom(&tb_ctx.htable, &desc, h, tb_lookup_cmp); } void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr) diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index d48b56ca38..f39123bd5a 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -782,11 +782,25 @@ static inline void code_gen_alloc(size_t tb_size) qemu_mutex_init(&tb_ctx.tb_lock); } +static bool tb_cmp(const void *ap, const void *bp) +{ + const TranslationBlock *a = ap; + const TranslationBlock *b = bp; + + return a->pc == b->pc && + a->cs_base == b->cs_base && + a->flags == b->flags && + (tb_cflags(a) & CF_HASH_MASK) == (tb_cflags(b) & CF_HASH_MASK) && + a->trace_vcpu_dstate == b->trace_vcpu_dstate && + a->page_addr[0] == b->page_addr[0] && + a->page_addr[1] == b->page_addr[1]; +} + static void tb_htable_init(void) { unsigned int mode = QHT_MODE_AUTO_RESIZE; - qht_init(&tb_ctx.htable, CODE_GEN_HTABLE_SIZE, mode); + qht_init(&tb_ctx.htable, tb_cmp, CODE_GEN_HTABLE_SIZE, mode); } /* Must be called before using the QEMU cpus. 'tb_size' is the size diff --git a/tests/qht-bench.c b/tests/qht-bench.c index 4cabdfd62a..c94ac25158 100644 --- a/tests/qht-bench.c +++ b/tests/qht-bench.c @@ -93,10 +93,10 @@ static void usage_complete(int argc, char *argv[]) exit(-1); } -static bool is_equal(const void *obj, const void *userp) +static bool is_equal(const void *ap, const void *bp) { - const long *a = obj; - const long *b = userp; + const long *a = ap; + const long *b = bp; return *a == *b; } @@ -150,7 +150,7 @@ static void do_rw(struct thread_info *info) p = &keys[info->r & (lookup_range - 1)]; hash = h(*p); - read = qht_lookup(&ht, is_equal, p, hash); + read = qht_lookup(&ht, p, hash); if (read) { stats->rd++; } else { @@ -162,7 +162,7 @@ static void do_rw(struct thread_info *info) if (info->write_op) { bool written = false; - if (qht_lookup(&ht, is_equal, p, hash) == NULL) { + if (qht_lookup(&ht, p, hash) == NULL) { written = qht_insert(&ht, p, hash); } if (written) { @@ -173,7 +173,7 @@ static void do_rw(struct thread_info *info) } else { bool removed = false; - if (qht_lookup(&ht, is_equal, p, hash)) { + if (qht_lookup(&ht, p, hash)) { removed = qht_remove(&ht, p, hash); } if (removed) { @@ -308,7 +308,7 @@ static void htable_init(void) } /* initialize the hash table */ - qht_init(&ht, qht_n_elems, qht_mode); + qht_init(&ht, is_equal, qht_n_elems, qht_mode); assert(init_size <= init_range); pr_params(); diff --git a/tests/test-qht.c b/tests/test-qht.c index 9b7423abb6..b069881342 100644 --- a/tests/test-qht.c +++ b/tests/test-qht.c @@ -13,10 +13,10 @@ static struct qht ht; static int32_t arr[N * 2]; -static bool is_equal(const void *obj, const void *userp) +static bool is_equal(const void *ap, const void *bp) { - const int32_t *a = obj; - const int32_t *b = userp; + const int32_t *a = ap; + const int32_t *b = bp; return *a == *b; } @@ -60,7 +60,12 @@ static void check(int a, int b, bool expected) val = i; hash = i; - p = qht_lookup(&ht, is_equal, &val, hash); + /* test both lookup variants; results should be the same */ + if (i % 2) { + p = qht_lookup(&ht, &val, hash); + } else { + p = qht_lookup_custom(&ht, &val, hash, is_equal); + } g_assert_true(!!p == expected); } rcu_read_unlock(); @@ -102,7 +107,7 @@ static void qht_do_test(unsigned int mode, size_t init_entries) /* under KVM we might fetch stats from an uninitialized qht */ check_n(0); - qht_init(&ht, 0, mode); + qht_init(&ht, is_equal, 0, mode); check_n(0); insert(0, N); diff --git a/util/qht.c b/util/qht.c index 55b0738cd1..487e9237bc 100644 --- a/util/qht.c +++ b/util/qht.c @@ -351,11 +351,14 @@ static struct qht_map *qht_map_create(size_t n_buckets) return map; } -void qht_init(struct qht *ht, size_t n_elems, unsigned int mode) +void qht_init(struct qht *ht, qht_cmp_func_t cmp, size_t n_elems, + unsigned int mode) { struct qht_map *map; size_t n_buckets = qht_elems_to_buckets(n_elems); + g_assert(cmp); + ht->cmp = cmp; ht->mode = mode; qemu_mutex_init(&ht->lock); map = qht_map_create(n_buckets); @@ -479,8 +482,8 @@ void *qht_lookup__slowpath(struct qht_bucket *b, qht_lookup_func_t func, return ret; } -void *qht_lookup(struct qht *ht, qht_lookup_func_t func, const void *userp, - uint32_t hash) +void *qht_lookup_custom(struct qht *ht, const void *userp, uint32_t hash, + qht_lookup_func_t func) { struct qht_bucket *b; struct qht_map *map; @@ -502,6 +505,11 @@ void *qht_lookup(struct qht *ht, qht_lookup_func_t func, const void *userp, return qht_lookup__slowpath(b, func, userp, hash); } +void *qht_lookup(struct qht *ht, const void *userp, uint32_t hash) +{ + return qht_lookup_custom(ht, userp, hash, ht->cmp); +} + /* call with head->lock held */ static bool qht_insert__locked(struct qht *ht, struct qht_map *map, struct qht_bucket *head, void *p, uint32_t hash, From patchwork Fri Jun 15 19:43:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138757 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1250011lji; Fri, 15 Jun 2018 12:50:46 -0700 (PDT) X-Google-Smtp-Source: ADUXVKIkuzDs9IpSD3IOaMV+J6C4SBRxCA1rFhTmiNAbDpWYx7jTL87GudCoh+FuKlxQ965i7inY X-Received: by 2002:a37:4792:: with SMTP id u140-v6mr2618337qka.100.1529092246260; Fri, 15 Jun 2018 12:50:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092246; cv=none; d=google.com; s=arc-20160816; b=ZLS2+8IlBJ/cAKa/vqj9IzjIfBCKjk6zAu9RKK7RpSZn7NN4zT9x3bnj5MumigPtMZ NdZrp+PiI6vcj5dtB6uBsbKdaTiY5U4BlbAb5Onckcz+pf8yDsJM9XxPf40wa5F+D8lY t9hKCRC/0T7ZbrAWZE9tpd3SHrIEDubaj3o6d29YKj6+fqpOmomINv+/uCqqgPdnxwqE yk8e4lhTytGrLKfkQrbnaIY9roquA58wDpQbY93McBHZ/aIYw+tlFy4CETWu/0giGp1R gDnfu7wZcwWFsluJf6wn0UOQwDUwetGXg/b9yMRae1McmE6RJyOZotpxQXg7JTU0FLzX bbWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=Kf4X6ZLsDNDTu5GJpyFJvunVvn0Jy3QznoC7UWjWK1g=; b=DfZDWa/rfmXB7kv1CaiZq2hrd+rWcFDf7ujMG2/HocCmX4B1uXbaqZGZo2cerNO15Z Vhb0ccAGDGLp18A0ejC0w0iuDRxk9IyIHpENeBOhslYov5jBTPkXJwCE5qMjfHl4FwnP w/b/Smy47wA2bYc+BkrkZsTqnyg7ZFPAFl0ZNUyTOVPVMtf3ojtD7aoRuZkCa2IbA1Nu w1eV45+GkKhji96xnVBGkhq2mLM1gAcwh5Fvlkau+N8yJtvEoT8wchKXi1z610CUGOUh Kd/PmnxHos/SNQiMs48uClnCfAjVG49Bhh+pjkZT4Gv+0NgjCz1wQD9NTZoczeRq6tO4 FteA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=h5FjaImO; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id g7-v6si4620347qvj.177.2018.06.15.12.50.46 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:50:46 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=h5FjaImO; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48986 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuk9-0005JJ-Km for patch@linaro.org; Fri, 15 Jun 2018 15:50:45 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51504) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudk-0000EF-Co for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudi-0002Om-Jw for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:08 -0400 Received: from mail-pg0-x230.google.com ([2607:f8b0:400e:c05::230]:45213) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudi-0002Nu-BB for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:06 -0400 Received: by mail-pg0-x230.google.com with SMTP id z1-v6so4841108pgv.12 for ; Fri, 15 Jun 2018 12:44:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Kf4X6ZLsDNDTu5GJpyFJvunVvn0Jy3QznoC7UWjWK1g=; b=h5FjaImO5j2oAHct82VZWGpoDtI0BZbt/lC7LbfPtVPHekH2MVsuPc9zF8zyGQfxDw ek2CvvYOqLnvioEcrmAxbItiP8iC+ZHyCEPJKZdSCG3glpSIrN/GOwi+JGnngcephk+j Dq3LtB1qfSIDQwNhwAGRn7YabXPJB0lCoGqBU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Kf4X6ZLsDNDTu5GJpyFJvunVvn0Jy3QznoC7UWjWK1g=; b=YSnTH6eHzkv+K+C/EWp88ljGTdu2AzUxMnhloUsTmzJed4lUgsAYuHJMCgTGv36NNy VwxuVuO1hXQ9k+T51Wsqjf1c7DrF9DFPU1XaM3/R7rPvJmX+ehX+dSvXuTL9/eJ74MZ9 pM9j2UVIcQxWBCzTm5o5zngUQq3rJ5pq3iHds3InFTq4l9Gf3MoEKKNlLPyg/gCd0PYz kKnfS80JRPKEX+lnHYqc54KC10YKWx5M+16FzbRnkup0p291TpwOxZf3X+nbV47VSb4U iBAlwf9l37zvjSC24iSXTuBkOxk97hOvWAPCB9fAS7fNd9epg2IOHdeK/su0AwlHlqLa ZRYQ== X-Gm-Message-State: APt69E1oHfmolaGFQi0tp+MRQtgl2bKZ9Isw3fOx4yxubqpUr1EOqXHa 9r4B55r31ak2hbPndjDTQgzwXDdWrcQ= X-Received: by 2002:aa7:8311:: with SMTP id t17-v6mr3382395pfm.45.1529091844942; Fri, 15 Jun 2018 12:44:04 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.03 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:04 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:38 -1000 Message-Id: <20180615194354.12489-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::230 Subject: [Qemu-devel] [PULL v2 03/19] qht: return existing entry when qht_insert fails X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" The meaning of "existing" is now changed to "matches in hash and ht->cmp result". This is saner than just checking the pointer value. Suggested-by: Richard Henderson Reviewed-by: Richard Henderson Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- include/qemu/qht.h | 7 +++++-- accel/tcg/translate-all.c | 2 +- tests/qht-bench.c | 4 ++-- tests/test-qht.c | 8 +++++++- util/qht.c | 27 +++++++++++++++++---------- 5 files changed, 32 insertions(+), 16 deletions(-) -- 2.17.1 diff --git a/include/qemu/qht.h b/include/qemu/qht.h index 5f03a0f4cb..1fb9116fa0 100644 --- a/include/qemu/qht.h +++ b/include/qemu/qht.h @@ -70,6 +70,7 @@ void qht_destroy(struct qht *ht); * @ht: QHT to insert to * @p: pointer to be inserted * @hash: hash corresponding to @p + * @existing: address where the pointer to an existing entry can be copied to * * Attempting to insert a NULL @p is a bug. * Inserting the same pointer @p with different @hash values is a bug. @@ -78,9 +79,11 @@ void qht_destroy(struct qht *ht); * inserted into the hash table. * * Returns true on success. - * Returns false if the @p-@hash pair already exists in the hash table. + * Returns false if there is an existing entry in the table that is equivalent + * (i.e. ht->cmp matches and the hash is the same) to @p-@h. If @existing + * is !NULL, a pointer to this existing entry is copied to it. */ -bool qht_insert(struct qht *ht, void *p, uint32_t hash); +bool qht_insert(struct qht *ht, void *p, uint32_t hash, void **existing); /** * qht_lookup_custom - Look up a pointer using a custom comparison function. diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index f39123bd5a..1695f8c352 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -1242,7 +1242,7 @@ static void tb_link_page(TranslationBlock *tb, tb_page_addr_t phys_pc, /* add in the hash table */ h = tb_hash_func(phys_pc, tb->pc, tb->flags, tb->cflags & CF_HASH_MASK, tb->trace_vcpu_dstate); - qht_insert(&tb_ctx.htable, tb, h); + qht_insert(&tb_ctx.htable, tb, h, NULL); #ifdef CONFIG_USER_ONLY if (DEBUG_TB_CHECK_GATE) { diff --git a/tests/qht-bench.c b/tests/qht-bench.c index c94ac25158..f492b3a20a 100644 --- a/tests/qht-bench.c +++ b/tests/qht-bench.c @@ -163,7 +163,7 @@ static void do_rw(struct thread_info *info) bool written = false; if (qht_lookup(&ht, p, hash) == NULL) { - written = qht_insert(&ht, p, hash); + written = qht_insert(&ht, p, hash, NULL); } if (written) { stats->in++; @@ -322,7 +322,7 @@ static void htable_init(void) r = xorshift64star(r); p = &keys[r & (init_range - 1)]; hash = h(*p); - if (qht_insert(&ht, p, hash)) { + if (qht_insert(&ht, p, hash, NULL)) { break; } retries++; diff --git a/tests/test-qht.c b/tests/test-qht.c index b069881342..dda6a067be 100644 --- a/tests/test-qht.c +++ b/tests/test-qht.c @@ -27,11 +27,17 @@ static void insert(int a, int b) for (i = a; i < b; i++) { uint32_t hash; + void *existing; + bool inserted; arr[i] = i; hash = i; - qht_insert(&ht, &arr[i], hash); + inserted = qht_insert(&ht, &arr[i], hash, NULL); + g_assert_true(inserted); + inserted = qht_insert(&ht, &arr[i], hash, &existing); + g_assert_false(inserted); + g_assert_true(existing == &arr[i]); } } diff --git a/util/qht.c b/util/qht.c index 487e9237bc..c138777a9c 100644 --- a/util/qht.c +++ b/util/qht.c @@ -511,9 +511,9 @@ void *qht_lookup(struct qht *ht, const void *userp, uint32_t hash) } /* call with head->lock held */ -static bool qht_insert__locked(struct qht *ht, struct qht_map *map, - struct qht_bucket *head, void *p, uint32_t hash, - bool *needs_resize) +static void *qht_insert__locked(struct qht *ht, struct qht_map *map, + struct qht_bucket *head, void *p, uint32_t hash, + bool *needs_resize) { struct qht_bucket *b = head; struct qht_bucket *prev = NULL; @@ -523,8 +523,9 @@ static bool qht_insert__locked(struct qht *ht, struct qht_map *map, do { for (i = 0; i < QHT_BUCKET_ENTRIES; i++) { if (b->pointers[i]) { - if (unlikely(b->pointers[i] == p)) { - return false; + if (unlikely(b->hashes[i] == hash && + ht->cmp(b->pointers[i], p))) { + return b->pointers[i]; } } else { goto found; @@ -553,7 +554,7 @@ static bool qht_insert__locked(struct qht *ht, struct qht_map *map, atomic_set(&b->hashes[i], hash); atomic_set(&b->pointers[i], p); seqlock_write_end(&head->sequence); - return true; + return NULL; } static __attribute__((noinline)) void qht_grow_maybe(struct qht *ht) @@ -577,25 +578,31 @@ static __attribute__((noinline)) void qht_grow_maybe(struct qht *ht) qemu_mutex_unlock(&ht->lock); } -bool qht_insert(struct qht *ht, void *p, uint32_t hash) +bool qht_insert(struct qht *ht, void *p, uint32_t hash, void **existing) { struct qht_bucket *b; struct qht_map *map; bool needs_resize = false; - bool ret; + void *prev; /* NULL pointers are not supported */ qht_debug_assert(p); b = qht_bucket_lock__no_stale(ht, hash, &map); - ret = qht_insert__locked(ht, map, b, p, hash, &needs_resize); + prev = qht_insert__locked(ht, map, b, p, hash, &needs_resize); qht_bucket_debug__locked(b); qemu_spin_unlock(&b->lock); if (unlikely(needs_resize) && ht->mode & QHT_MODE_AUTO_RESIZE) { qht_grow_maybe(ht); } - return ret; + if (likely(prev == NULL)) { + return true; + } + if (existing) { + *existing = prev; + } + return false; } static inline bool qht_entry_is_last(struct qht_bucket *b, int pos) From patchwork Fri Jun 15 19:43:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138755 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1247810lji; Fri, 15 Jun 2018 12:48:07 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJlpgEZ/4MrceX/3b6eeJpwSP8+ulwnUbfRRAKieNHq1TRP9lhjOVYIVQj3DBHJMOH8ILej X-Received: by 2002:a0c:cc91:: with SMTP id f17-v6mr2635533qvl.239.1529092087522; Fri, 15 Jun 2018 12:48:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092087; cv=none; d=google.com; s=arc-20160816; b=Qc559S10kMxIHTnotE78hbQSemEytqvluv43fOfOkgWhM8i7GuG4Ni7iSc9ITRc/S5 /nk90ZnNP7PykzxBKDJvMZ2K6TdIkA/JwnWt8gex3St6CQTYD6HdhcmjmHvQK7xmxmFk v7vtpZFAz9HRbk1eleocFz2TLJKuBRuAEIlSPKsnfDb89XKJfx5Wxm2AnH3SXIi6bTEU GSXqyR7stcK48wd21/QbQcVTfMjV8K95npdNp8Ms/veOTQRvFtTDYPQEl06MVtUsVtln Ymjw9rhqkSIXwLp3OXyeHvcLxoau3+FEDdsSnirRpecLqxrx7NCReGvvTXGo9/nwJKS3 ZS2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=2eXp/0v31+I85MHKNIFJ+BSApDB0NAmwq4blTyk6igY=; b=JiwaHhKJQkWQpLxQ290AYdtpjJWSdo8fll/1owfomYbLeEAv6kP7BunOx6DJF9+U6S sz0zepK2Esg8hhakRJqEFVwDJGRsuKkxx9dYJVoqJn8YWykXvPJ0iryiGQhpVZns3DZk gnoB5paHBAvss2C2zzidkW5BxrnaKtKl39gcLoRPVO6aHYrx2jqDKaS2vh2FzB419ovW vzTP28WI+tFP3/r99WmMK/+WIiggs1YtPE2WoQfyKJE6+rXJAioUzMxpy/yMV3f5+RC5 l+g4e2YZfaV/NPf7FogvnnG4l6wEeoYPWVKWfKdcYgz1t3N9AK2VkH1P/cWv/9lzxsbX wt8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kLBt+GaJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id p5-v6si7769024qkc.230.2018.06.15.12.48.07 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:48:07 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kLBt+GaJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48971 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuhb-0002o7-0E for patch@linaro.org; Fri, 15 Jun 2018 15:48:07 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51521) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudn-0000GL-6S for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudk-0002RH-LJ for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:11 -0400 Received: from mail-pg0-x22e.google.com ([2607:f8b0:400e:c05::22e]:35222) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudk-0002QY-AS for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:08 -0400 Received: by mail-pg0-x22e.google.com with SMTP id 15-v6so4858774pge.2 for ; Fri, 15 Jun 2018 12:44:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2eXp/0v31+I85MHKNIFJ+BSApDB0NAmwq4blTyk6igY=; b=kLBt+GaJwQz5NC7TWXZdEPcYeCYnYUzZg/1A/lGg54chdV+uhrbSb9Ntmt95Q/M+Tf aFNTj3eB1qdMs/qj0SvBDt8FueciNHVuNWKregXghtg/weAWGGbP+3+yb3wLvxYWqtrw 35+UHXfuGsPn5poH+SEo9vyHqjEtvPNEd3scc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2eXp/0v31+I85MHKNIFJ+BSApDB0NAmwq4blTyk6igY=; b=FY8XZLaaYMlY7ZxKWyYL7gUlRwKlBolipOCW1ojSDDckFaxpUDH4At2eit+QD6/7fy hHou0kf2O4gB1rIVfUvBmw29hmVX/P+jpAN5YV3ONNIIhEIRLdFXh6ETVbTt1TkHVfYO /ms0usPwUbfmX/alypx2h2TPh8h0NLIaaiFNAt4joV5L2EPsZd7VKs9N8ehkJ5Kj0EcR uM7ffEtYGZW8Zq9UEGoYNryMHY41gWppR6eTP2vOzIVwpFuXmJnr9mOkUAs7smU2oXzs e83BqfvUVchnseSbLk250+H2IZk4wOMi7ZEn0kqbd7sEdHN4v4DRGvqvlzwCsYhG7VXv lsAg== X-Gm-Message-State: APt69E2iiqVzuRL4MPsTT4CebR1zMdUEvctEIbPQVDTW6uO+KsbrelFp EbFyXzL4pqaE+DfcQ0JwUU5buxWg5wg= X-Received: by 2002:a62:3101:: with SMTP id x1-v6mr3331117pfx.246.1529091846867; Fri, 15 Jun 2018 12:44:06 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:05 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:39 -1000 Message-Id: <20180615194354.12489-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::22e Subject: [Qemu-devel] [PULL v2 04/19] tcg: track TBs with per-region BST's X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" This paves the way for enabling scalable parallel generation of TCG code. Instead of tracking TBs with a single binary search tree (BST), use a BST for each TCG region, protecting it with a lock. This is as scalable as it gets, since each TCG thread operates on a separate region. The core of this change is the introduction of struct tcg_region_tree, which contains a pointer to a GTree and an associated lock to serialize accesses to it. We then allocate an array of tcg_region_tree's, adding the appropriate padding to avoid false sharing based on qemu_dcache_linesize. Given a tc_ptr, we first find the corresponding region_tree. This is done by special-casing the first and last regions first, since they might be of size != region.size; otherwise we just divide the offset by region.stride. I was worried about this division (several dozen cycles of latency), but profiling shows that this is not a fast path. Note that region.stride is not required to be a power of two; it is only required to be a multiple of the host's page size. Note that with this design we can also provide consistent snapshots about all region trees at once; for instance, tcg_tb_foreach acquires/releases all region_tree locks before/after iterating over them. For this reason we now drop tb_lock in dump_exec_info(). As an alternative I considered implementing a concurrent BST, but this can be tricky to get right, offers no consistent snapshots of the BST, and performance and scalability-wise I don't think it could ever beat having separate GTrees, given that our workload is insert-mostly (all concurrent BST designs I've seen focus, understandably, on making lookups fast, which comes at the expense of convoluted, non-wait-free insertions/removals). Reviewed-by: Richard Henderson Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- include/exec/exec-all.h | 1 - include/exec/tb-context.h | 1 - tcg/tcg.h | 6 ++ accel/tcg/cpu-exec.c | 2 +- accel/tcg/translate-all.c | 101 +++----------------- tcg/tcg.c | 191 ++++++++++++++++++++++++++++++++++++++ 6 files changed, 213 insertions(+), 89 deletions(-) -- 2.17.1 diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 8bbea787a9..8d4306ac25 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -405,7 +405,6 @@ static inline uint32_t curr_cflags(void) | (use_icount ? CF_USE_ICOUNT : 0); } -void tb_remove(TranslationBlock *tb); void tb_flush(CPUState *cpu); void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr); TranslationBlock *tb_htable_lookup(CPUState *cpu, target_ulong pc, diff --git a/include/exec/tb-context.h b/include/exec/tb-context.h index 1d41202485..d8472c88fb 100644 --- a/include/exec/tb-context.h +++ b/include/exec/tb-context.h @@ -31,7 +31,6 @@ typedef struct TBContext TBContext; struct TBContext { - GTree *tb_tree; struct qht htable; /* any access to the tbs or the page table must use this lock */ QemuMutex tb_lock; diff --git a/tcg/tcg.h b/tcg/tcg.h index 509f4d65d2..1e6df1906f 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -866,6 +866,12 @@ void tcg_region_reset_all(void); size_t tcg_code_size(void); size_t tcg_code_capacity(void); +void tcg_tb_insert(TranslationBlock *tb); +void tcg_tb_remove(TranslationBlock *tb); +TranslationBlock *tcg_tb_lookup(uintptr_t tc_ptr); +void tcg_tb_foreach(GTraverseFunc func, gpointer user_data); +size_t tcg_nb_tbs(void); + /* user-mode: Called with tb_lock held. */ static inline void *tcg_malloc(int size) { diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c index 6d6c51b686..7570c59f09 100644 --- a/accel/tcg/cpu-exec.c +++ b/accel/tcg/cpu-exec.c @@ -224,7 +224,7 @@ static void cpu_exec_nocache(CPUState *cpu, int max_cycles, tb_lock(); tb_phys_invalidate(tb, -1); - tb_remove(tb); + tcg_tb_remove(tb); tb_unlock(); } #endif diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 1695f8c352..ef841c82cc 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -205,8 +205,6 @@ void tb_lock_reset(void) } } -static TranslationBlock *tb_find_pc(uintptr_t tc_ptr); - void cpu_gen_init(void) { tcg_context_init(&tcg_init_ctx); @@ -375,13 +373,13 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t host_pc, bool will_exit) if (check_offset < tcg_init_ctx.code_gen_buffer_size) { tb_lock(); - tb = tb_find_pc(host_pc); + tb = tcg_tb_lookup(host_pc); if (tb) { cpu_restore_state_from_tb(cpu, tb, host_pc, will_exit); if (tb->cflags & CF_NOCACHE) { /* one-shot translation, invalidate it immediately */ tb_phys_invalidate(tb, -1); - tb_remove(tb); + tcg_tb_remove(tb); } r = true; } @@ -728,48 +726,6 @@ static inline void *alloc_code_gen_buffer(void) } #endif /* USE_STATIC_CODE_GEN_BUFFER, WIN32, POSIX */ -/* compare a pointer @ptr and a tb_tc @s */ -static int ptr_cmp_tb_tc(const void *ptr, const struct tb_tc *s) -{ - if (ptr >= s->ptr + s->size) { - return 1; - } else if (ptr < s->ptr) { - return -1; - } - return 0; -} - -static gint tb_tc_cmp(gconstpointer ap, gconstpointer bp) -{ - const struct tb_tc *a = ap; - const struct tb_tc *b = bp; - - /* - * When both sizes are set, we know this isn't a lookup. - * This is the most likely case: every TB must be inserted; lookups - * are a lot less frequent. - */ - if (likely(a->size && b->size)) { - if (a->ptr > b->ptr) { - return 1; - } else if (a->ptr < b->ptr) { - return -1; - } - /* a->ptr == b->ptr should happen only on deletions */ - g_assert(a->size == b->size); - return 0; - } - /* - * All lookups have either .size field set to 0. - * From the glib sources we see that @ap is always the lookup key. However - * the docs provide no guarantee, so we just mark this case as likely. - */ - if (likely(a->size == 0)) { - return ptr_cmp_tb_tc(a->ptr, b); - } - return ptr_cmp_tb_tc(b->ptr, a); -} - static inline void code_gen_alloc(size_t tb_size) { tcg_ctx->code_gen_buffer_size = size_code_gen_buffer(tb_size); @@ -778,7 +734,6 @@ static inline void code_gen_alloc(size_t tb_size) fprintf(stderr, "Could not allocate dynamic translator buffer\n"); exit(1); } - tb_ctx.tb_tree = g_tree_new(tb_tc_cmp); qemu_mutex_init(&tb_ctx.tb_lock); } @@ -839,14 +794,6 @@ static TranslationBlock *tb_alloc(target_ulong pc) return tb; } -/* Called with tb_lock held. */ -void tb_remove(TranslationBlock *tb) -{ - assert_tb_locked(); - - g_tree_remove(tb_ctx.tb_tree, &tb->tc); -} - static inline void invalidate_page_bitmap(PageDesc *p) { #ifdef CONFIG_SOFTMMU @@ -911,10 +858,10 @@ static void do_tb_flush(CPUState *cpu, run_on_cpu_data tb_flush_count) } if (DEBUG_TB_FLUSH_GATE) { - size_t nb_tbs = g_tree_nnodes(tb_ctx.tb_tree); + size_t nb_tbs = tcg_nb_tbs(); size_t host_size = 0; - g_tree_foreach(tb_ctx.tb_tree, tb_host_size_iter, &host_size); + tcg_tb_foreach(tb_host_size_iter, &host_size); printf("qemu: flush code_size=%zu nb_tbs=%zu avg_tb_size=%zu\n", tcg_code_size(), nb_tbs, nb_tbs > 0 ? host_size / nb_tbs : 0); } @@ -923,10 +870,6 @@ static void do_tb_flush(CPUState *cpu, run_on_cpu_data tb_flush_count) cpu_tb_jmp_cache_clear(cpu); } - /* Increment the refcount first so that destroy acts as a reset */ - g_tree_ref(tb_ctx.tb_tree); - g_tree_destroy(tb_ctx.tb_tree); - qht_reset_size(&tb_ctx.htable, CODE_GEN_HTABLE_SIZE); page_flush_tb(); @@ -1406,7 +1349,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu, * through the physical hash table and physical page list. */ tb_link_page(tb, phys_pc, phys_page2); - g_tree_insert(tb_ctx.tb_tree, &tb->tc, tb); + tcg_tb_insert(tb); return tb; } @@ -1510,7 +1453,7 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, current_tb = NULL; if (cpu->mem_io_pc) { /* now we have a real cpu fault */ - current_tb = tb_find_pc(cpu->mem_io_pc); + current_tb = tcg_tb_lookup(cpu->mem_io_pc); } } if (current_tb == tb && @@ -1627,7 +1570,7 @@ static bool tb_invalidate_phys_page(tb_page_addr_t addr, uintptr_t pc) tb = p->first_tb; #ifdef TARGET_HAS_PRECISE_SMC if (tb && pc != 0) { - current_tb = tb_find_pc(pc); + current_tb = tcg_tb_lookup(pc); } if (cpu != NULL) { env = cpu->env_ptr; @@ -1670,18 +1613,6 @@ static bool tb_invalidate_phys_page(tb_page_addr_t addr, uintptr_t pc) } #endif -/* - * Find the TB 'tb' such that - * tb->tc.ptr <= tc_ptr < tb->tc.ptr + tb->tc.size - * Return NULL if not found. - */ -static TranslationBlock *tb_find_pc(uintptr_t tc_ptr) -{ - struct tb_tc s = { .ptr = (void *)tc_ptr }; - - return g_tree_lookup(tb_ctx.tb_tree, &s); -} - #if !defined(CONFIG_USER_ONLY) void tb_invalidate_phys_addr(AddressSpace *as, hwaddr addr, MemTxAttrs attrs) { @@ -1709,7 +1640,7 @@ void tb_check_watchpoint(CPUState *cpu) { TranslationBlock *tb; - tb = tb_find_pc(cpu->mem_io_pc); + tb = tcg_tb_lookup(cpu->mem_io_pc); if (tb) { /* We can use retranslation to find the PC. */ cpu_restore_state_from_tb(cpu, tb, cpu->mem_io_pc, true); @@ -1743,7 +1674,7 @@ void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr) uint32_t n; tb_lock(); - tb = tb_find_pc(retaddr); + tb = tcg_tb_lookup(retaddr); if (!tb) { cpu_abort(cpu, "cpu_io_recompile: could not find TB for pc=%p", (void *)retaddr); @@ -1782,7 +1713,7 @@ void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr) * cpu_exec_nocache() */ tb_phys_invalidate(tb->orig_tb, -1); } - tb_remove(tb); + tcg_tb_remove(tb); } /* TODO: If env->pc != tb->pc (i.e. the faulting instruction was not @@ -1853,6 +1784,7 @@ static void print_qht_statistics(FILE *f, fprintf_function cpu_fprintf, } struct tb_tree_stats { + size_t nb_tbs; size_t host_size; size_t target_size; size_t max_target_size; @@ -1866,6 +1798,7 @@ static gboolean tb_tree_stats_iter(gpointer key, gpointer value, gpointer data) const TranslationBlock *tb = value; struct tb_tree_stats *tst = data; + tst->nb_tbs++; tst->host_size += tb->tc.size; tst->target_size += tb->size; if (tb->size > tst->max_target_size) { @@ -1889,10 +1822,8 @@ void dump_exec_info(FILE *f, fprintf_function cpu_fprintf) struct qht_stats hst; size_t nb_tbs; - tb_lock(); - - nb_tbs = g_tree_nnodes(tb_ctx.tb_tree); - g_tree_foreach(tb_ctx.tb_tree, tb_tree_stats_iter, &tst); + tcg_tb_foreach(tb_tree_stats_iter, &tst); + nb_tbs = tst.nb_tbs; /* XXX: avoid using doubles ? */ cpu_fprintf(f, "Translation buffer state:\n"); /* @@ -1927,8 +1858,6 @@ void dump_exec_info(FILE *f, fprintf_function cpu_fprintf) cpu_fprintf(f, "TB invalidate count %d\n", tb_ctx.tb_phys_invalidate_count); cpu_fprintf(f, "TLB flush count %zu\n", tlb_flush_count()); tcg_dump_info(f, cpu_fprintf); - - tb_unlock(); } void dump_opcount_info(FILE *f, fprintf_function cpu_fprintf) @@ -2196,7 +2125,7 @@ int page_unprotect(target_ulong address, uintptr_t pc) * set the page to PAGE_WRITE and did the TB invalidate for us. */ #ifdef TARGET_HAS_PRECISE_SMC - TranslationBlock *current_tb = tb_find_pc(pc); + TranslationBlock *current_tb = tcg_tb_lookup(pc); if (current_tb) { current_tb_invalidated = tb_cflags(current_tb) & CF_INVALID; } diff --git a/tcg/tcg.c b/tcg/tcg.c index 6eeebe0624..62e3391020 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -135,6 +135,12 @@ static TCGContext **tcg_ctxs; static unsigned int n_tcg_ctxs; TCGv_env cpu_env = 0; +struct tcg_region_tree { + QemuMutex lock; + GTree *tree; + /* padding to avoid false sharing is computed at run-time */ +}; + /* * We divide code_gen_buffer into equally-sized "regions" that TCG threads * dynamically allocate from as demand dictates. Given appropriate region @@ -158,6 +164,13 @@ struct tcg_region_state { }; static struct tcg_region_state region; +/* + * This is an array of struct tcg_region_tree's, with padding. + * We use void * to simplify the computation of region_trees[i]; each + * struct is found every tree_size bytes. + */ +static void *region_trees; +static size_t tree_size; static TCGRegSet tcg_target_available_regs[TCG_TYPE_COUNT]; static TCGRegSet tcg_target_call_clobber_regs; @@ -295,6 +308,180 @@ TCGLabel *gen_new_label(void) #include "tcg-target.inc.c" +/* compare a pointer @ptr and a tb_tc @s */ +static int ptr_cmp_tb_tc(const void *ptr, const struct tb_tc *s) +{ + if (ptr >= s->ptr + s->size) { + return 1; + } else if (ptr < s->ptr) { + return -1; + } + return 0; +} + +static gint tb_tc_cmp(gconstpointer ap, gconstpointer bp) +{ + const struct tb_tc *a = ap; + const struct tb_tc *b = bp; + + /* + * When both sizes are set, we know this isn't a lookup. + * This is the most likely case: every TB must be inserted; lookups + * are a lot less frequent. + */ + if (likely(a->size && b->size)) { + if (a->ptr > b->ptr) { + return 1; + } else if (a->ptr < b->ptr) { + return -1; + } + /* a->ptr == b->ptr should happen only on deletions */ + g_assert(a->size == b->size); + return 0; + } + /* + * All lookups have either .size field set to 0. + * From the glib sources we see that @ap is always the lookup key. However + * the docs provide no guarantee, so we just mark this case as likely. + */ + if (likely(a->size == 0)) { + return ptr_cmp_tb_tc(a->ptr, b); + } + return ptr_cmp_tb_tc(b->ptr, a); +} + +static void tcg_region_trees_init(void) +{ + size_t i; + + tree_size = ROUND_UP(sizeof(struct tcg_region_tree), qemu_dcache_linesize); + region_trees = qemu_memalign(qemu_dcache_linesize, region.n * tree_size); + for (i = 0; i < region.n; i++) { + struct tcg_region_tree *rt = region_trees + i * tree_size; + + qemu_mutex_init(&rt->lock); + rt->tree = g_tree_new(tb_tc_cmp); + } +} + +static struct tcg_region_tree *tc_ptr_to_region_tree(void *p) +{ + size_t region_idx; + + if (p < region.start_aligned) { + region_idx = 0; + } else { + ptrdiff_t offset = p - region.start_aligned; + + if (offset > region.stride * (region.n - 1)) { + region_idx = region.n - 1; + } else { + region_idx = offset / region.stride; + } + } + return region_trees + region_idx * tree_size; +} + +void tcg_tb_insert(TranslationBlock *tb) +{ + struct tcg_region_tree *rt = tc_ptr_to_region_tree(tb->tc.ptr); + + qemu_mutex_lock(&rt->lock); + g_tree_insert(rt->tree, &tb->tc, tb); + qemu_mutex_unlock(&rt->lock); +} + +void tcg_tb_remove(TranslationBlock *tb) +{ + struct tcg_region_tree *rt = tc_ptr_to_region_tree(tb->tc.ptr); + + qemu_mutex_lock(&rt->lock); + g_tree_remove(rt->tree, &tb->tc); + qemu_mutex_unlock(&rt->lock); +} + +/* + * Find the TB 'tb' such that + * tb->tc.ptr <= tc_ptr < tb->tc.ptr + tb->tc.size + * Return NULL if not found. + */ +TranslationBlock *tcg_tb_lookup(uintptr_t tc_ptr) +{ + struct tcg_region_tree *rt = tc_ptr_to_region_tree((void *)tc_ptr); + TranslationBlock *tb; + struct tb_tc s = { .ptr = (void *)tc_ptr }; + + qemu_mutex_lock(&rt->lock); + tb = g_tree_lookup(rt->tree, &s); + qemu_mutex_unlock(&rt->lock); + return tb; +} + +static void tcg_region_tree_lock_all(void) +{ + size_t i; + + for (i = 0; i < region.n; i++) { + struct tcg_region_tree *rt = region_trees + i * tree_size; + + qemu_mutex_lock(&rt->lock); + } +} + +static void tcg_region_tree_unlock_all(void) +{ + size_t i; + + for (i = 0; i < region.n; i++) { + struct tcg_region_tree *rt = region_trees + i * tree_size; + + qemu_mutex_unlock(&rt->lock); + } +} + +void tcg_tb_foreach(GTraverseFunc func, gpointer user_data) +{ + size_t i; + + tcg_region_tree_lock_all(); + for (i = 0; i < region.n; i++) { + struct tcg_region_tree *rt = region_trees + i * tree_size; + + g_tree_foreach(rt->tree, func, user_data); + } + tcg_region_tree_unlock_all(); +} + +size_t tcg_nb_tbs(void) +{ + size_t nb_tbs = 0; + size_t i; + + tcg_region_tree_lock_all(); + for (i = 0; i < region.n; i++) { + struct tcg_region_tree *rt = region_trees + i * tree_size; + + nb_tbs += g_tree_nnodes(rt->tree); + } + tcg_region_tree_unlock_all(); + return nb_tbs; +} + +static void tcg_region_tree_reset_all(void) +{ + size_t i; + + tcg_region_tree_lock_all(); + for (i = 0; i < region.n; i++) { + struct tcg_region_tree *rt = region_trees + i * tree_size; + + /* Increment the refcount first so that destroy acts as a reset */ + g_tree_ref(rt->tree); + g_tree_destroy(rt->tree); + } + tcg_region_tree_unlock_all(); +} + static void tcg_region_bounds(size_t curr_region, void **pstart, void **pend) { void *start, *end; @@ -380,6 +567,8 @@ void tcg_region_reset_all(void) g_assert(!err); } qemu_mutex_unlock(®ion.lock); + + tcg_region_tree_reset_all(); } #ifdef CONFIG_USER_ONLY @@ -496,6 +685,8 @@ void tcg_region_init(void) g_assert(!rc); } + tcg_region_trees_init(); + /* In user-mode we support only one ctx, so do the initial allocation now */ #ifdef CONFIG_USER_ONLY { From patchwork Fri Jun 15 19:43:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138756 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1249976lji; Fri, 15 Jun 2018 12:50:44 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJqDRpYZkCrJiIgjftI9K1Q0IrPYgS+2Irv+jquEViyNLIiX+7DAOHXnT8RieSGZcJ0XWOJ X-Received: by 2002:a37:a705:: with SMTP id q5-v6mr2589447qke.154.1529092244445; Fri, 15 Jun 2018 12:50:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092244; cv=none; d=google.com; s=arc-20160816; b=njqIpnOksMkGppdU1TdEZJ/Xv0t7FeyHBT6W0iMck6pip3dSmQwMIPCqajD4UrTKyv 8rbSeWxWj+BV+Q+Vj5q/IADVuuH8b26IZkKPsNm05XdD4Snex4iliRtv2vSmjxhCEFtN bLC+FLNzkhJVTtgTeey/RE+qvuMygaVLLpmZXpoI82T9c+H+9JaXPfd/Lx/+lAWuLFCz Ff4A/1odAeJKdJAefv/nymYSxZAKTpbPOTUyzmolnHjLzDpaZDXQR5wN+OREA2IXdGnF /9jJc8AHKBiEzEgCuJbfL/QDryTgWti0OE4wW+hh9MLxflkDNMTos73eIW8CSIruOxdP hHbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=jwAP7X29dbyxdynmu1epQ4EzxJvbPBd/BBvpAfZSEjM=; b=lN9EHr8dJm79qzPYxqdzZYMoMvfPbJB4vm+xmmGYqWcyAKewcVM1CYPA0+8jkvmx6q YgxR5mLjP1rrcdThyKlYyMEfe7XNqpobM/tKZxNtRyf7/Tq50RB6ZYx/sET9vxcfHd4O 733PmEvlyuB5hqGUG8CtrWxKzF447oVSO+iL3ZFATLGhBR6nqh6r0poMwEJTQmOVrOQT Vf9SNybPuJnKbmCLzHywflOhyW/4W+GU35aNAlzfIu3dV6i25GTYZsOy7bctYGcOQwu+ eZxCjpyYW6zQNefFbHW0GOcVH6n6USaUod2kwuTNvimt5/d4zKeFwa5f+uzGFexNC7e7 pbrw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=jjtozDVc; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id 47-v6si2039463qvj.156.2018.06.15.12.50.44 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:50:44 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=jjtozDVc; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48985 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuk7-0005Hq-TV for patch@linaro.org; Fri, 15 Jun 2018 15:50:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51523) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudn-0000Gc-Fy for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudm-0002SK-Fl for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:11 -0400 Received: from mail-pg0-x236.google.com ([2607:f8b0:400e:c05::236]:34014) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudm-0002Rx-9G for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:10 -0400 Received: by mail-pg0-x236.google.com with SMTP id q4-v6so4861902pgr.1 for ; Fri, 15 Jun 2018 12:44:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jwAP7X29dbyxdynmu1epQ4EzxJvbPBd/BBvpAfZSEjM=; b=jjtozDVcnQuWI1QWST27zefaVlCW4yA6dlmxEfBck0I8YvGAOlaDznS94tT4p+lkzH +WxNlw5NquuSMfOAkzK3BlSvwoOZ98SHI264qzoF8zqExfcM5p9dQP9+GCOjrU8XElGR pZ8uCehNaIL2Br9A8rZxIdOF/uNwkDlJjzeZQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jwAP7X29dbyxdynmu1epQ4EzxJvbPBd/BBvpAfZSEjM=; b=jT0sZ2b3qvjUK/8UsYDce3gud/BPaXUNZhg1ewzaVOWfb6BQYt26M6QuiepphplEEg n6RX1wg2IzV2K3f7eWnQCxnh1aJq4B+snNFxpl+iV3PomnCimk6RDNXzEvhnuaiNJzeq TrgSKz787M72HVrcV1Pn8QPoQpwhsqnITI/iIEGitOxP5gizo/b90j6cTFCK+cwNEd5c MRwY/KzrGDMnN+HxUfqc7j1e1MKEZTmPEfOEkKYhdGNCP2HTm3c7qOVIp2fHswq6Bmv8 RtaOp343D5ZPgECxVjMY0J3u58NEP5Je9pEpFZ9G9NNMazbz40NrGp4oCPRXeQNYY/bS fokw== X-Gm-Message-State: APt69E1Ssikr3ELbAsMJ+L4HHzaziFvP5w4FgvCP/KQK5O1oS9SAka6u E9TAHtBuegZBiLrfTF4KUrHBjHDjOx8= X-Received: by 2002:a62:7db:: with SMTP id 88-v6mr3371988pfh.178.1529091849009; Fri, 15 Jun 2018 12:44:09 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:08 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:40 -1000 Message-Id: <20180615194354.12489-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::236 Subject: [Qemu-devel] [PULL v2 05/19] tcg: move tb_ctx.tb_phys_invalidate_count to tcg_ctx X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" Thereby making it per-TCGContext. Once we remove tb_lock, this will avoid an atomic increment every time a TB is invalidated. Reviewed-by: Richard Henderson Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- include/exec/tb-context.h | 1 - tcg/tcg.h | 3 +++ accel/tcg/translate-all.c | 5 +++-- tcg/tcg.c | 14 ++++++++++++++ 4 files changed, 20 insertions(+), 3 deletions(-) -- 2.17.1 diff --git a/include/exec/tb-context.h b/include/exec/tb-context.h index d8472c88fb..8c9b49c98e 100644 --- a/include/exec/tb-context.h +++ b/include/exec/tb-context.h @@ -37,7 +37,6 @@ struct TBContext { /* statistics */ unsigned tb_flush_count; - int tb_phys_invalidate_count; }; extern TBContext tb_ctx; diff --git a/tcg/tcg.h b/tcg/tcg.h index 1e6df1906f..e49b289ba1 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -695,6 +695,8 @@ struct TCGContext { /* Threshold to flush the translated code buffer. */ void *code_gen_highwater; + size_t tb_phys_invalidate_count; + /* Track which vCPU triggers events */ CPUState *cpu; /* *_trans */ @@ -868,6 +870,7 @@ size_t tcg_code_capacity(void); void tcg_tb_insert(TranslationBlock *tb); void tcg_tb_remove(TranslationBlock *tb); +size_t tcg_tb_phys_invalidate_count(void); TranslationBlock *tcg_tb_lookup(uintptr_t tc_ptr); void tcg_tb_foreach(GTraverseFunc func, gpointer user_data); size_t tcg_nb_tbs(void); diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index ef841c82cc..a9f2bfb468 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -1069,7 +1069,8 @@ void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr) /* suppress any remaining jumps to this TB */ tb_jmp_unlink(tb); - tb_ctx.tb_phys_invalidate_count++; + atomic_set(&tcg_ctx->tb_phys_invalidate_count, + tcg_ctx->tb_phys_invalidate_count + 1); } #ifdef CONFIG_SOFTMMU @@ -1855,7 +1856,7 @@ void dump_exec_info(FILE *f, fprintf_function cpu_fprintf) cpu_fprintf(f, "\nStatistics:\n"); cpu_fprintf(f, "TB flush count %u\n", atomic_read(&tb_ctx.tb_flush_count)); - cpu_fprintf(f, "TB invalidate count %d\n", tb_ctx.tb_phys_invalidate_count); + cpu_fprintf(f, "TB invalidate count %zu\n", tcg_tb_phys_invalidate_count()); cpu_fprintf(f, "TLB flush count %zu\n", tlb_flush_count()); tcg_dump_info(f, cpu_fprintf); } diff --git a/tcg/tcg.c b/tcg/tcg.c index 62e3391020..1d1dfd7f7c 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -791,6 +791,20 @@ size_t tcg_code_capacity(void) return capacity; } +size_t tcg_tb_phys_invalidate_count(void) +{ + unsigned int n_ctxs = atomic_read(&n_tcg_ctxs); + unsigned int i; + size_t total = 0; + + for (i = 0; i < n_ctxs; i++) { + const TCGContext *s = atomic_read(&tcg_ctxs[i]); + + total += atomic_read(&s->tb_phys_invalidate_count); + } + return total; +} + /* pool based memory allocation */ void *tcg_malloc_internal(TCGContext *s, int size) { From patchwork Fri Jun 15 19:43:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138758 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1250375lji; Fri, 15 Jun 2018 12:51:14 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKL1ySQzkaB1D52pzCzzf/nCn/oe4FRmBxLAjBMnggVmYUPjOhbo5GbON+aaKY2TS8BoKfX X-Received: by 2002:a37:1807:: with SMTP id j7-v6mr2664667qkh.333.1529092274132; Fri, 15 Jun 2018 12:51:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092274; cv=none; d=google.com; s=arc-20160816; b=wFPybmJYcejGFij8E5e2Rco7AorpLYV6jyEO33Ab6tgA9HzlZHhlg3Y91jOizQP92d 2Xf8nZsMHMmZIb9Oua9d/LzOZzjSwGksEaYd0ict4zEkW2i9/LGtsJp+mahQGkpThCrR 0Didp+oFsFdDjz5hzfxfNEo8p+anXAlaIlLeX4lxENwt+wpsHtfyAzeFaHbRrCZ7keez cxvbZSOd0v6NtNwyQn6UtmEjtStLp/R+sbKDq6da+7FuZt+62HwCWLEdJjG0fTP4I5Aa DiuOInT9TzBo88H4oziJSbv5NNukf/wX5/2j9XrXtl5Zz4TPPoNdpYkMqvBYSU70ozPf +Knw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=0FQ+ZwY1R2rA+XDd8sFaaKFP5ITR5yBL2RD0xEoLhgc=; b=VuMZ6KDjyTO1NCcKO13KKQ0iI4DLkOoDHM2tHAyIdNAFdB0JLAH460q8O255I5hBkE LfVHY+8J+ViOg/RGsdOf95XlATkhVLF6u7xiI+0LMzMjyGT/7qr0R25b6RycocbqVVJH 5DkNmWImQPXesz8Y8o1KBLZUa/YkzdTl9lPl4ZN+7PQ/5OJMKqFfBh3rVxq0LaAdTDd3 64qbZR4t4EKHqL8Mvh6pALOW3EF0ynSmZfTojrkZMcwTmRORUMbLkK7llRFPBYOKfENZ ecxkBjxcJ9xNabQoMaZY5iT09NV+nrNa8LlnvnRXiu0u1garjSsiY6FLZ9ezu3DdhigS yA1A== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=UJZRTTkH; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id d36-v6si5259509qvh.276.2018.06.15.12.51.13 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:51:14 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=UJZRTTkH; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48988 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTukb-0005OF-If for patch@linaro.org; Fri, 15 Jun 2018 15:51:13 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51554) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudt-0000Mu-5L for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudo-0002Tv-M6 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:17 -0400 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:41330) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudo-0002TQ-D1 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:12 -0400 Received: by mail-pg0-x244.google.com with SMTP id l65-v6so4852985pgl.8 for ; Fri, 15 Jun 2018 12:44:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0FQ+ZwY1R2rA+XDd8sFaaKFP5ITR5yBL2RD0xEoLhgc=; b=UJZRTTkH0rgauJdubrnPVrx+R24hQNz6XkeWojLwctMyhoQ0fbdakaZ+JNTcvaJ+mw qXrgZvk6QJ3V1xiMGDDXTNgATaWpCK3ndwRTvLyL5vlRIFHA59loJCGy8KcTcbdNfDMy 0/ixLm0l7KIzmiBDq1ZzV4+k5aYnaCWCQfyDg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0FQ+ZwY1R2rA+XDd8sFaaKFP5ITR5yBL2RD0xEoLhgc=; b=ky4yFazfU0dvDPXoT/rsD+GGYfzzMaDAtbHry6emB2wdCxL0wqbFWrMgJ79477M/R3 VAJQO6WzmDeBqw73jfSiMQg45Xs0SAPm7pA8ZvDMDd/zMa9oAwqQ3P3FMWVikjzk496C AuQcVNgVTUpAsQRuWJSMJneVadh5DIgN4Np93toyAFb9Hj+rPiiZN+p+BeDKrsE0IHZ1 C4zvc/LSu0+v6f7ehS+L+4itymD+KEQvAKUHOVDD5DLkgh/OJlWCzhYevHGhkXfUmUKo kG9JiSJfrvsBJuXnrSZy8F6MRBKPIXZ3Gu9E2FfjKmofoCicIMGS/TOEtArep+cTUcW6 4Q9g== X-Gm-Message-State: APt69E397LWoFQ3IhSA0TtwVHdMcUS4C9YzsbYqEffpLyDCNpAkrp3et kVwKwBYjQwZw+CDShcvrzTRkfk5TdHk= X-Received: by 2002:a62:904c:: with SMTP id a73-v6mr3364885pfe.145.1529091851051; Fri, 15 Jun 2018 12:44:11 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.09 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:10 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:41 -1000 Message-Id: <20180615194354.12489-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PULL v2 06/19] translate-all: iterate over TBs in a page with PAGE_FOR_EACH_TB X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" This commit does several things, but to avoid churn I merged them all into the same commit. To wit: - Use uintptr_t instead of TranslationBlock * for the list of TBs in a page. Just like we did in (c37e6d7e "tcg: Use uintptr_t type for jmp_list_{next|first} fields of TB"), the rationale is the same: these are tagged pointers, not pointers. So use a more appropriate type. - Only check the least significant bit of the tagged pointers. Masking with 3/~3 is unnecessary and confusing. - Introduce the TB_FOR_EACH_TAGGED macro, and use it to define PAGE_FOR_EACH_TB, which improves readability. Note that TB_FOR_EACH_TAGGED will gain another user in a subsequent patch. - Update tb_page_remove to use PAGE_FOR_EACH_TB. In case there is a bug and we attempt to remove a TB that is not in the list, instead of segfaulting (since the list is NULL-terminated) we will reach g_assert_not_reached(). Reviewed-by: Richard Henderson Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- include/exec/exec-all.h | 2 +- accel/tcg/translate-all.c | 62 ++++++++++++++++++--------------------- 2 files changed, 30 insertions(+), 34 deletions(-) -- 2.17.1 diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 8d4306ac25..07653d3c92 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -360,7 +360,7 @@ struct TranslationBlock { struct TranslationBlock *orig_tb; /* first and second physical page containing code. The lower bit of the pointer tells the index in page_next[] */ - struct TranslationBlock *page_next[2]; + uintptr_t page_next[2]; tb_page_addr_t page_addr[2]; /* The following data are used to directly call another TB from diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index a9f2bfb468..52e62125ed 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -103,7 +103,7 @@ typedef struct PageDesc { /* list of TBs intersecting this ram page */ - TranslationBlock *first_tb; + uintptr_t first_tb; #ifdef CONFIG_SOFTMMU /* in order to optimize self modifying code, we count the number of lookups we do to a given page to use a bitmap */ @@ -114,6 +114,15 @@ typedef struct PageDesc { #endif } PageDesc; +/* list iterators for lists of tagged pointers in TranslationBlock */ +#define TB_FOR_EACH_TAGGED(head, tb, n, field) \ + for (n = (head) & 1, tb = (TranslationBlock *)((head) & ~1); \ + tb; tb = (TranslationBlock *)tb->field[n], n = (uintptr_t)tb & 1, \ + tb = (TranslationBlock *)((uintptr_t)tb & ~1)) + +#define PAGE_FOR_EACH_TB(pagedesc, tb, n) \ + TB_FOR_EACH_TAGGED((pagedesc)->first_tb, tb, n, page_next) + /* In system mode we want L1_MAP to be based on ram offsets, while in user mode we want it to be based on virtual addresses. */ #if !defined(CONFIG_USER_ONLY) @@ -815,7 +824,7 @@ static void page_flush_tb_1(int level, void **lp) PageDesc *pd = *lp; for (i = 0; i < V_L2_SIZE; ++i) { - pd[i].first_tb = NULL; + pd[i].first_tb = (uintptr_t)NULL; invalidate_page_bitmap(pd + i); } } else { @@ -943,21 +952,21 @@ static void tb_page_check(void) #endif /* CONFIG_USER_ONLY */ -static inline void tb_page_remove(TranslationBlock **ptb, TranslationBlock *tb) +static inline void tb_page_remove(PageDesc *pd, TranslationBlock *tb) { TranslationBlock *tb1; + uintptr_t *pprev; unsigned int n1; - for (;;) { - tb1 = *ptb; - n1 = (uintptr_t)tb1 & 3; - tb1 = (TranslationBlock *)((uintptr_t)tb1 & ~3); + pprev = &pd->first_tb; + PAGE_FOR_EACH_TB(pd, tb1, n1) { if (tb1 == tb) { - *ptb = tb1->page_next[n1]; - break; + *pprev = tb1->page_next[n1]; + return; } - ptb = &tb1->page_next[n1]; + pprev = &tb1->page_next[n1]; } + g_assert_not_reached(); } /* remove the TB from a list of TBs jumping to the n-th jump target of the TB */ @@ -1045,12 +1054,12 @@ void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr) /* remove the TB from the page list */ if (tb->page_addr[0] != page_addr) { p = page_find(tb->page_addr[0] >> TARGET_PAGE_BITS); - tb_page_remove(&p->first_tb, tb); + tb_page_remove(p, tb); invalidate_page_bitmap(p); } if (tb->page_addr[1] != -1 && tb->page_addr[1] != page_addr) { p = page_find(tb->page_addr[1] >> TARGET_PAGE_BITS); - tb_page_remove(&p->first_tb, tb); + tb_page_remove(p, tb); invalidate_page_bitmap(p); } @@ -1081,10 +1090,7 @@ static void build_page_bitmap(PageDesc *p) p->code_bitmap = bitmap_new(TARGET_PAGE_SIZE); - tb = p->first_tb; - while (tb != NULL) { - n = (uintptr_t)tb & 3; - tb = (TranslationBlock *)((uintptr_t)tb & ~3); + PAGE_FOR_EACH_TB(p, tb, n) { /* NOTE: this is subtle as a TB may span two physical pages */ if (n == 0) { /* NOTE: tb_end may be after the end of the page, but @@ -1099,7 +1105,6 @@ static void build_page_bitmap(PageDesc *p) tb_end = ((tb->pc + tb->size) & ~TARGET_PAGE_MASK); } bitmap_set(p->code_bitmap, tb_start, tb_end - tb_start); - tb = tb->page_next[n]; } } #endif @@ -1122,9 +1127,9 @@ static inline void tb_alloc_page(TranslationBlock *tb, p = page_find_alloc(page_addr >> TARGET_PAGE_BITS, 1); tb->page_next[n] = p->first_tb; #ifndef CONFIG_USER_ONLY - page_already_protected = p->first_tb != NULL; + page_already_protected = p->first_tb != (uintptr_t)NULL; #endif - p->first_tb = (TranslationBlock *)((uintptr_t)tb | n); + p->first_tb = (uintptr_t)tb | n; invalidate_page_bitmap(p); #if defined(CONFIG_USER_ONLY) @@ -1401,7 +1406,7 @@ void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, int is_cpu_write_access) { - TranslationBlock *tb, *tb_next; + TranslationBlock *tb; tb_page_addr_t tb_start, tb_end; PageDesc *p; int n; @@ -1432,11 +1437,7 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, /* we remove all the TBs in the range [start, end[ */ /* XXX: see if in some cases it could be faster to invalidate all the code */ - tb = p->first_tb; - while (tb != NULL) { - n = (uintptr_t)tb & 3; - tb = (TranslationBlock *)((uintptr_t)tb & ~3); - tb_next = tb->page_next[n]; + PAGE_FOR_EACH_TB(p, tb, n) { /* NOTE: this is subtle as a TB may span two physical pages */ if (n == 0) { /* NOTE: tb_end may be after the end of the page, but @@ -1474,7 +1475,6 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, #endif /* TARGET_HAS_PRECISE_SMC */ tb_phys_invalidate(tb, -1); } - tb = tb_next; } #if !defined(CONFIG_USER_ONLY) /* if no code remaining, no need to continue to use slow writes */ @@ -1568,18 +1568,15 @@ static bool tb_invalidate_phys_page(tb_page_addr_t addr, uintptr_t pc) } tb_lock(); - tb = p->first_tb; #ifdef TARGET_HAS_PRECISE_SMC - if (tb && pc != 0) { + if (p->first_tb && pc != 0) { current_tb = tcg_tb_lookup(pc); } if (cpu != NULL) { env = cpu->env_ptr; } #endif - while (tb != NULL) { - n = (uintptr_t)tb & 3; - tb = (TranslationBlock *)((uintptr_t)tb & ~3); + PAGE_FOR_EACH_TB(p, tb, n) { #ifdef TARGET_HAS_PRECISE_SMC if (current_tb == tb && (current_tb->cflags & CF_COUNT_MASK) != 1) { @@ -1596,9 +1593,8 @@ static bool tb_invalidate_phys_page(tb_page_addr_t addr, uintptr_t pc) } #endif /* TARGET_HAS_PRECISE_SMC */ tb_phys_invalidate(tb, addr); - tb = tb->page_next[n]; } - p->first_tb = NULL; + p->first_tb = (uintptr_t)NULL; #ifdef TARGET_HAS_PRECISE_SMC if (current_tb_modified) { /* Force execution of one insn next time. */ From patchwork Fri Jun 15 19:43:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138753 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1245949lji; Fri, 15 Jun 2018 12:45:49 -0700 (PDT) X-Google-Smtp-Source: ADUXVKLwt1beHYIw3sJuH7vtdh3PGTHjsMemioxvDVPoeV98HVS6nAUI8AEZQ6DUeFdkKOIF6SKp X-Received: by 2002:a37:660f:: with SMTP id a15-v6mr2596919qkc.440.1529091949725; Fri, 15 Jun 2018 12:45:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529091949; cv=none; d=google.com; s=arc-20160816; b=ppktLa9pwyjHdnjEaH8VF8gcMh4E1IUtErjlLfplSgUtK1aMxjPdOmK9j4YGfEMSO8 uA4l70fwFhk3/amABenhgID5A57ByXMNnoQNLxKIfP51Kn+/HLIyJ+xMEES2wi7IJQdH qgACcYUeMpFdwljNkAHhbkfAlDzXEJ5yXo+r0HwxvYLu802+9RfzVweVGGYBEMpGl32j pMBhHMzFgQjZ4+6HV/CtXzKRS8RqSASgeCVz8Vd6eX0VQ3eAq8TgjS3HS7RrjKaBNtEQ tvGNFeq3aAlFDzBjvGc+bncBdhNuwh65nv2jfxP7o3QD4eNsVonwq+8AVZHa4gfc5ZsQ QMyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=+wmxg/OG714qNbfIRNlSF9jWomNEICTIUlUkDChX+hI=; b=YKoO/hJUWSjbYtPTEGVJS/xasCVrsI/qboS789mNMGQ5gOPN2r6kCG27L5eHfXdvbd 4NOqaANcdUsdP6RhsK8n9i4xxidkn0LudQsWteBl1zbf6ghqJxXnwG89jYXgrS6Z1/02 54GaBT57m1jy7mrslrDWWqLgihIE/+EPazdox8YpdJil1UV6xyQ8r+98fW+pGFHrxBSP wlDwssRQPycbxTJfmQ9Cc/sGrnMCb69OnbFalHX5j8r7NvFA4PtqnqiJbMh67la89pei ppAB2v4aP1uMEHtPmLrJxc7tKXGBwBUlmuoLq53fRngrSWHTzMiJtq1sSNux/0D0FVHF Br0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=GxNYARp6; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id e187-v6si948180qkc.193.2018.06.15.12.45.49 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:45:49 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=GxNYARp6; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48959 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTufN-00019d-4W for patch@linaro.org; Fri, 15 Jun 2018 15:45:49 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51556) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudt-0000NT-KX for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudq-0002Uk-9C for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:17 -0400 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:33039) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudq-0002UR-1m for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:14 -0400 Received: by mail-pg0-x241.google.com with SMTP id e11-v6so4860360pgq.0 for ; Fri, 15 Jun 2018 12:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+wmxg/OG714qNbfIRNlSF9jWomNEICTIUlUkDChX+hI=; b=GxNYARp6istrOzdiCpiy+RtjcQ0v41KsbIXBK/ErHeuDSbE+8uV9TxWksl4gnKq67P 744PSFgWhQE5nXLWfvqMtDvUoKywJZl8eimc4DqZPi77su7dCEri753B/2jMwiRBKDkM JD8y8iMHWn7N9CxCKB39iFQ36qIdpvyvQ9t30= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+wmxg/OG714qNbfIRNlSF9jWomNEICTIUlUkDChX+hI=; b=KF3slaaf7ic5Pw/tHrSWvv2CaP2yShbiwp0oLePSkSPW5/tvB9czTzvGbcgTfmfWRB pqU81g+MtyfG4MWbKCT0bNUm6rgH7u12I/K3QSV+GYXZ3cKDHQsHO8v49VePtBnEk9Id seCfHDNQ4dC6m/zoDqEuse4FBg2slZ8f/HW59I/TjiwC7ajVwuHGKOy0DawzW1nR6Jsz r/+5HblH6Nc0bxGJ70gP+cGJuuMHTkx2HlsBsXPiyzXOYqQK9nFbykWK4vzk7dQufirk YMYibcg6l1c5+QTyxLXR8E9JN78rfY3q0kxAUX06qwPpbItRBZDJRE3LUGbEEFliXW4w zwjg== X-Gm-Message-State: APt69E1N35vOgRB3iu7GmGEDpx/t76AwgYIT/v3O7McqSWaxE51r/O/V 9CSKW/+dZNOFzdZQ7lNwLMPXnWRFGKQ= X-Received: by 2002:a63:85c8:: with SMTP id u191-v6mr2678283pgd.300.1529091852810; Fri, 15 Jun 2018 12:44:12 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.11 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:11 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:42 -1000 Message-Id: <20180615194354.12489-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PULL v2 07/19] translate-all: make l1_map lockless X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" Groundwork for supporting parallel TCG generation. We never remove entries from the radix tree, so we can use cmpxchg to implement lockless insertions. Reviewed-by: Richard Henderson Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/translate-all.c | 24 ++++++++++++++---------- docs/devel/multi-thread-tcg.txt | 4 ++-- 2 files changed, 16 insertions(+), 12 deletions(-) -- 2.17.1 diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 52e62125ed..52273e5846 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -469,20 +469,12 @@ static void page_init(void) #endif } -/* If alloc=1: - * Called with tb_lock held for system emulation. - * Called with mmap_lock held for user-mode emulation. - */ static PageDesc *page_find_alloc(tb_page_addr_t index, int alloc) { PageDesc *pd; void **lp; int i; - if (alloc) { - assert_memory_lock(); - } - /* Level 1. Always allocated. */ lp = l1_map + ((index >> v_l1_shift) & (v_l1_size - 1)); @@ -491,11 +483,17 @@ static PageDesc *page_find_alloc(tb_page_addr_t index, int alloc) void **p = atomic_rcu_read(lp); if (p == NULL) { + void *existing; + if (!alloc) { return NULL; } p = g_new0(void *, V_L2_SIZE); - atomic_rcu_set(lp, p); + existing = atomic_cmpxchg(lp, NULL, p); + if (unlikely(existing)) { + g_free(p); + p = existing; + } } lp = p + ((index >> (i * V_L2_BITS)) & (V_L2_SIZE - 1)); @@ -503,11 +501,17 @@ static PageDesc *page_find_alloc(tb_page_addr_t index, int alloc) pd = atomic_rcu_read(lp); if (pd == NULL) { + void *existing; + if (!alloc) { return NULL; } pd = g_new0(PageDesc, V_L2_SIZE); - atomic_rcu_set(lp, pd); + existing = atomic_cmpxchg(lp, NULL, pd); + if (unlikely(existing)) { + g_free(pd); + pd = existing; + } } return pd + (index & (V_L2_SIZE - 1)); diff --git a/docs/devel/multi-thread-tcg.txt b/docs/devel/multi-thread-tcg.txt index a99b4564c6..faf8918b23 100644 --- a/docs/devel/multi-thread-tcg.txt +++ b/docs/devel/multi-thread-tcg.txt @@ -134,8 +134,8 @@ tb_set_jmp_target() code. Modification to the linked lists that allow searching for linked pages are done under the protect of the tb_lock(). -The global page table is protected by the tb_lock() in system-mode and -mmap_lock() in linux-user mode. +The global page table is a lockless radix tree; cmpxchg is used +to atomically insert new elements. The lookup caches are updated atomically and the lookup hash uses QHT which is designed for concurrent safe lookup. From patchwork Fri Jun 15 19:43:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138766 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1254743lji; Fri, 15 Jun 2018 12:56:25 -0700 (PDT) X-Google-Smtp-Source: ADUXVKIpWkWd+Yh37wKodGTeIpMP6fik+Bx8kgYfF6msac2OwB0aZhfVq+5hJtOPh+Kd6Iqxd2KB X-Received: by 2002:a37:ab04:: with SMTP id u4-v6mr2518131qke.92.1529092585179; Fri, 15 Jun 2018 12:56:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092585; cv=none; d=google.com; s=arc-20160816; b=pQa5hrEVF8OScUh24Bk4HWkO0FYlfzzdrbJoGvlbDEZ1LDePF1SBT3touqWbGfhQAn Ua77+jvvCtJPfJ3szPZPQHPrR0MjK3Gj8pNKkkByaKzYO2L3vFmlhBEORKj8C94oP8Yk I6xlKV7jbXmidgOI99gQPLm1eO39p9MRbR6p1GULKeqiLwEQ2VrcHfL3NiVqlQo3yCDE wuVr0jo+6tlwWnlVs4dDUgomJieq3OUv8Xh+53lKJyCq9xOlOVWoU76BDxsq0E6sd0oX yuGRqZj11Cne0wcF70+lngtLxMFQ5aBkcv6K6zJtGHMD1AEGoRuyFa9IdeXtSHs7HVoy pvQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=PUl84RGj2OqEjV3sjRt0a3dQMiShyLF4sGf17shcl8g=; b=uXuwQ6ELsU/b3AuYwytIESuVA+/ymJ9bB/Uv8Y+gutAg08vqs8afSmdM2WsRymB4kS N5woraHki3KGQKqVggsYFJW4IekGxBlJwl+lKEfDB64pgiM2cxSFr7GbLhuCotv4HMGS rIqqkKrvt082dfygsmrcYTlDSX/vOmbx/VZx4JlbHnP4uo7RBRA+XJSW2f2QQVRNFgxK XHm9pUrEEJQ4P0ti4QGOSNAGa3QFTUuf693W/xSAArvnNpscPTs5KCkyT+69DJrNaydq resJyj/ce5sOlMML7haOiZONWMXZIPc4pM/5qG41bJdxHMzEcUNluzWM6I+I5FSNgf8v J4hw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=crLplm91; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id o82-v6si1086276qki.221.2018.06.15.12.56.25 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:56:25 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=crLplm91; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49018 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTupc-0001bi-Li for patch@linaro.org; Fri, 15 Jun 2018 15:56:24 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51557) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudt-0000NU-KU for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTuds-0002Vu-18 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:17 -0400 Received: from mail-pf0-x244.google.com ([2607:f8b0:400e:c00::244]:43839) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudr-0002VK-RC for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:15 -0400 Received: by mail-pf0-x244.google.com with SMTP id y8-v6so5326250pfm.10 for ; Fri, 15 Jun 2018 12:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PUl84RGj2OqEjV3sjRt0a3dQMiShyLF4sGf17shcl8g=; b=crLplm91IwD0isbt/KQD+71R3MQWPV2dtRrVU4wtv5f2eMVyLbr3TBM7ji2jCOHqqO o8m3w+ew9GS+mSfmzZGEOAZM9jHN6KRzbob71lWZ1VTSSdvqyBTHkPsBENhuWZlquNvd x43rq4wW5sN/mbPswor+r84aQJTwEl4/uhG3k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PUl84RGj2OqEjV3sjRt0a3dQMiShyLF4sGf17shcl8g=; b=iwgs5ror+ePj07bg9ZgLzIuI/oJxAWgSN6DtSUaTBD373Tayeh7fHtroSiXGmk4tDJ hq9Nz7i6qw7ALqOXBg+P+1yCGswYp3CC0T8c+gKPWpVFh5mCfXC1QL5QrIFOhFvi0Qfu twqVKynl9HfkkcMkYZhlpxHaJARQU2JFWnZYLmSzzTcmeJESe0IPhNlfWaTV1VGecW9p 6lG0uOyO6w7ub1FbW4DyV7UFuV7d9uLoeTe73wQ/bnPhW/y7lwmjOf2K5VBWGatwtalz EeaMjZj7R9XVC63u8ZVAK3D8SJDhhwHj0tALX7x9VQXhFZ358W4fnn8r6iUv/Cvz04rD cOhQ== X-Gm-Message-State: APt69E2Qtzc6uaxZBq46dYA1M2S40urX/68v2cpNN0eINNJYDVWDOP5T 1XC7xBvAZ9/70uRBzeBA3WlSxhkR7d4= X-Received: by 2002:a63:5fc1:: with SMTP id t184-v6mr2838071pgb.132.1529091854516; Fri, 15 Jun 2018 12:44:14 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.12 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:13 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:43 -1000 Message-Id: <20180615194354.12489-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::244 Subject: [Qemu-devel] [PULL v2 08/19] translate-all: remove hole in PageDesc X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" Groundwork for supporting parallel TCG generation. Move the hole to the end of the struct, so that a u32 field can be added there without bloating the struct. Reviewed-by: Richard Henderson Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/translate-all.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.17.1 diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 52273e5846..b9c36a3e45 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -107,8 +107,8 @@ typedef struct PageDesc { #ifdef CONFIG_SOFTMMU /* in order to optimize self modifying code, we count the number of lookups we do to a given page to use a bitmap */ - unsigned int code_write_count; unsigned long *code_bitmap; + unsigned int code_write_count; #else unsigned long flags; #endif From patchwork Fri Jun 15 19:43:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138767 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1255120lji; Fri, 15 Jun 2018 12:56:52 -0700 (PDT) X-Google-Smtp-Source: ADUXVKIuLYl9Wj4CApwCj+e0BWZuYSSJv3MYY7zJdCLdwhX0IYwHyrfYm88Hq5MT6/Tvi4bqYq3q X-Received: by 2002:a37:b603:: with SMTP id g3-v6mr2693713qkf.208.1529092612754; Fri, 15 Jun 2018 12:56:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092612; cv=none; d=google.com; s=arc-20160816; b=fy5K0Bc5T1P0GsXcXza2oL7dc9erC4o+rDQ4rvgxW9ev41rLbpgKcP/9zq4Ly1vZf+ oXKS301aIS3fgJ+aTGzLkPwcFuDbyRPAMWPuokaqccO0ZMIENbtyVMNd6RXef4KLl6pG eJn+FDSSJUK2tOMqBov8tau8pkSoVvgWH87iT6UQG+wAjZZphuiEsepno7ardZzX4tIq Gw127NS4EoUm97QfwgCXnQjdVFs87QAOU1ryxK9601LhTiJiIX6m20omWjh88kSTmC3A zHwH+mKV1GIIM2502cQrAyrIMHGEg5r9dJsaxif5fE5F94qkhKkMxR722enkqv85dmu0 Ohpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=Wexsdf6RCRL9p3KT+WfakrG7waRhQx2Db6LZYDP544g=; b=F8Tw0a3FIj7HvSbvuVktH2RoDrBXgt0zyRxvrRU7ec7D5Qv1iqcekZ6mxbvFc7hAxE 7Efzmr2/FXhusNU8uWBtImq+Z+OuclKZzoxubENEEY5p3hVmKWOBZyAljCmffxuOH8Kh BMLpxSqIpjEir8EedE8p77IzeDb6U5HdBvpevTMJgmCN3VGLD3mx6/99gSwfOQeeX0PG AFBMRHXwEtgpulQejtH4d35CJGoUgxDt2X7EfOqR5qpYUl5uHDJvWzmZfzN96xH/PtEN p8IhwEaWSG1HxYoayyqCfx+K6bfjjEOWFIYtIv1vKaBDKeQxM3dkEnIfV/V8YWw702fk a7UA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Iuk9ZE3E; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id i58-v6si1581983qti.134.2018.06.15.12.56.52 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:56:52 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Iuk9ZE3E; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49019 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuq4-0001ft-8u for patch@linaro.org; Fri, 15 Jun 2018 15:56:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51579) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudv-0000Ow-0a for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:20 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudt-0002Wz-UK for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:19 -0400 Received: from mail-pl0-x22a.google.com ([2607:f8b0:400e:c01::22a]:33307) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudt-0002WS-Oa for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:17 -0400 Received: by mail-pl0-x22a.google.com with SMTP id 6-v6so5312755plb.0 for ; Fri, 15 Jun 2018 12:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Wexsdf6RCRL9p3KT+WfakrG7waRhQx2Db6LZYDP544g=; b=Iuk9ZE3EDSVu2etBCW3lyy9zANN38sQmYghkrev9scXgob7smDg3yvrRe6S+OOh4Hf XPOkQKt/mcbiCdhrUAguqxTwzGydKq7zYZg/8bve/squdE4TCvwwHMz6b1fY0GhWirZq 7bbFnMjJn57lQM5wpGT96OfiFTl8DHucrbFbw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Wexsdf6RCRL9p3KT+WfakrG7waRhQx2Db6LZYDP544g=; b=nvHCmxidHhSGrpXDKrnxADxDwJzWdKU8t3QrzZot9/RsRjLLoIPLb9rgsW5q28LylU xImkCSR6OzFsq2/77kzkd+Dx+UPPL0ErU1auE9pcwI/8jBkJ7a9b/UzQZoeMwcq289Fk rU2GflFFLXeAM9pj78crAbiAJF+uwl+2nEo0YjqWAdxwHThWz/WWma07Bg8mtB6d24Ao Ddc3ZF0NL+owztB9R4N5CdGSUFJwbesXQ1I22VRq2oX978PJBqTw+5lFSLHkU6o8YBhF 6By2fzhYAtm8SzQdsRflmk+A/eruiADrd8UajfdqZp1yM/N/1CpC4ZzfMlwMy69LKTox O46A== X-Gm-Message-State: APt69E3pKLLkRQhTnliGID6QgaWzMCZO2zBBROio2Fv4VX87BgMGW48M 556yp0kSE4PLLbOrb8W/eL7er6YWGgM= X-Received: by 2002:a17:902:748b:: with SMTP id h11-v6mr3553130pll.12.1529091856448; Fri, 15 Jun 2018 12:44:16 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:15 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:44 -1000 Message-Id: <20180615194354.12489-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::22a Subject: [Qemu-devel] [PULL v2 09/19] translate-all: work page-by-page in tb_invalidate_phys_range_1 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" So that we pass a same-page range to tb_invalidate_phys_page_range, instead of always passing an end address that could be on a different page. As discussed with Peter Maydell on the list [1], tb_invalidate_phys_page_range doesn't actually do much with 'end', which explains why we have never hit a bug despite going against what the comment on top of tb_invalidate_phys_page_range requires: > * Invalidate all TBs which intersect with the target physical address range > * [start;end[. NOTE: start and end must refer to the *same* physical page. The appended honours the comment, which avoids confusion. While at it, rework the loop into a for loop, which is less error prone (e.g. "continue" won't result in an infinite loop). [1] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg09165.html Reviewed-by: Richard Henderson Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/translate-all.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) -- 2.17.1 diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index b9c36a3e45..f32904d4a3 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -1375,10 +1375,14 @@ TranslationBlock *tb_gen_code(CPUState *cpu, */ static void tb_invalidate_phys_range_1(tb_page_addr_t start, tb_page_addr_t end) { - while (start < end) { - tb_invalidate_phys_page_range(start, end, 0); - start &= TARGET_PAGE_MASK; - start += TARGET_PAGE_SIZE; + tb_page_addr_t next; + + for (next = (start & TARGET_PAGE_MASK) + TARGET_PAGE_SIZE; + start < end; + start = next, next += TARGET_PAGE_SIZE) { + tb_page_addr_t bound = MIN(next, end); + + tb_invalidate_phys_page_range(start, bound, 0); } } From patchwork Fri Jun 15 19:43:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138770 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1257190lji; Fri, 15 Jun 2018 12:59:33 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKmQC50aC2AU6b0HZM/jgmLHCmPaGnS5K+0IY59FurJBBWQctblrEWn2o2yQrdHYHU1WloT X-Received: by 2002:a37:4cd5:: with SMTP id z204-v6mr2697800qka.302.1529092773143; Fri, 15 Jun 2018 12:59:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092773; cv=none; d=google.com; s=arc-20160816; b=STUM/Rx76N7xlLDiaM9YNNQfcN0gCXm/W8w6yAvpMFc8OPITpKldZ9dgVyggt5pr51 5Yojq+1g64DFqk90eBr2bUojhg+XUCGb/tuO/87AUY6b9sA6QK0KFflyHZQiHemeSSp7 iNrDOJTjdfxyXPFrEHaA7aLU5OvnxtpKTNedLHGH6IdhgD/WfsFx/bAwTMQniGdCy4y7 ioBkTEaELhrIvjt4DB9chDOngwaov7t2uKKXGelD2E10RE5tr5kcWfZg6kd/iL64qmQK nRHURSvK9JLF8CmhkDechYdyvCnWDo3MwmmkY2g0Tjo1dVuSN6+1iyu/uWzC3lr4XywH I62w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=ShUlIMM0KTqpBxnqZKpmUtzBtiivBcjvXZG919nAn2g=; b=jsL6OSTewBDU8x4+Ol2gZrMXeHwZctPJebPxh39j94MRSqtWLHgJZ7Omog9X1YwWPc ih575d7+NC2wDfyqpHAtYprvuah6JgWS8wB4DYT8Qpod7zLIndVcu3MbOBWXkzEXgO9y 2r2IYpPoIFiEmJRURP5kf5a2lJ8h+3Kwx1oNYQCWePsdIItPHPnDwOACACxGmlOzzxeZ 3S74TCu7PlYuhOpUJZKEQQ09M1Sg3mK8CMQpPrL0sGpiwNkfTuwqfctslmJ5iY3pRGrj NIYcwVTgDGnoflyI/wYXRgGpg5cod9lLGCsYL1RHpdTST6LFC1g/jaZcqAbgaalus2EQ R6hw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=hzSrR46m; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id 1-v6si7136412qtm.251.2018.06.15.12.59.32 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:59:33 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=hzSrR46m; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49037 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuse-0003xF-Ij for patch@linaro.org; Fri, 15 Jun 2018 15:59:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51595) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTudw-0000Ql-QD for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudv-0002YA-Oe for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:20 -0400 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:45856) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudv-0002Xv-I3 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:19 -0400 Received: by mail-pf0-x243.google.com with SMTP id a22-v6so5321841pfo.12 for ; Fri, 15 Jun 2018 12:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ShUlIMM0KTqpBxnqZKpmUtzBtiivBcjvXZG919nAn2g=; b=hzSrR46mvfqF17i20lCdDa7GnPxu9G9ym7CsFtKfG2r1YK99SdkEwJroBagYNPb4YS PV0A3ucl8lZod8TbHFXLVb9oOfYXy1FLWj8EuQ1NiTy3xUCWh2XHKZd1FOXW7bdxGT2T PAs4zBl2E1n4FT0kxc25LyU4yjHvTW1+MjAu8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ShUlIMM0KTqpBxnqZKpmUtzBtiivBcjvXZG919nAn2g=; b=eDJA/rmXvPhrJ+b2lk0CQcP9onfXrvDt0dRSbwiR/D2kj/UuclB/8Ydpc/zF1i6OG0 75ABLX7j1brJxn1T0D86xwkc3ZGhQFffKJrrK4WxTiB07VlM06m+2m3JyEUL7v2Xk/ay tzeGXVh5b4vvZIIvwAy7VzC9fudk8GYoJwZ47Vwjmwdrb4DwIEFZL1vyEfAlrZKU3pgf DO2fWk6lYh8OWRgyDpwSg2EusQspqUOYhk7PUaZU34wSk3jHtZr+e1dvqiLXm4R4moi4 uQ6hCnDgzUZkj1oVoDZRdEX2h+SazNKOBDGRAAxFrFH3ukIyf631LtaY6wjUUM8Flyt7 BxWQ== X-Gm-Message-State: APt69E1ymx1I87PFUtxB23dVQ9EzbPt+taGBjyc/kRe5VxLOEPxz0lzP GQC9maUs2Uqlk/uPW2rUddnsxqlNv1o= X-Received: by 2002:a62:8a83:: with SMTP id o3-v6mr3424471pfk.12.1529091858272; Fri, 15 Jun 2018 12:44:18 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.16 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:17 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:45 -1000 Message-Id: <20180615194354.12489-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PULL v2 10/19] translate-all: move tb_invalidate_phys_page_range up in the file X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" This greatly simplifies next commit's diff. Reviewed-by: Richard Henderson Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/translate-all.c | 77 ++++++++++++++++++++------------------- 1 file changed, 39 insertions(+), 38 deletions(-) -- 2.17.1 diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index f32904d4a3..83bb40fb20 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -1363,44 +1363,6 @@ TranslationBlock *tb_gen_code(CPUState *cpu, return tb; } -/* - * Invalidate all TBs which intersect with the target physical address range - * [start;end[. NOTE: start and end may refer to *different* physical pages. - * 'is_cpu_write_access' should be true if called from a real cpu write - * access: the virtual CPU will exit the current TB if code is modified inside - * this TB. - * - * Called with mmap_lock held for user-mode emulation, grabs tb_lock - * Called with tb_lock held for system-mode emulation - */ -static void tb_invalidate_phys_range_1(tb_page_addr_t start, tb_page_addr_t end) -{ - tb_page_addr_t next; - - for (next = (start & TARGET_PAGE_MASK) + TARGET_PAGE_SIZE; - start < end; - start = next, next += TARGET_PAGE_SIZE) { - tb_page_addr_t bound = MIN(next, end); - - tb_invalidate_phys_page_range(start, bound, 0); - } -} - -#ifdef CONFIG_SOFTMMU -void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) -{ - assert_tb_locked(); - tb_invalidate_phys_range_1(start, end); -} -#else -void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) -{ - assert_memory_lock(); - tb_lock(); - tb_invalidate_phys_range_1(start, end); - tb_unlock(); -} -#endif /* * Invalidate all TBs which intersect with the target physical address range * [start;end[. NOTE: start and end must refer to the *same* physical page. @@ -1500,6 +1462,45 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, #endif } +/* + * Invalidate all TBs which intersect with the target physical address range + * [start;end[. NOTE: start and end may refer to *different* physical pages. + * 'is_cpu_write_access' should be true if called from a real cpu write + * access: the virtual CPU will exit the current TB if code is modified inside + * this TB. + * + * Called with mmap_lock held for user-mode emulation, grabs tb_lock + * Called with tb_lock held for system-mode emulation + */ +static void tb_invalidate_phys_range_1(tb_page_addr_t start, tb_page_addr_t end) +{ + tb_page_addr_t next; + + for (next = (start & TARGET_PAGE_MASK) + TARGET_PAGE_SIZE; + start < end; + start = next, next += TARGET_PAGE_SIZE) { + tb_page_addr_t bound = MIN(next, end); + + tb_invalidate_phys_page_range(start, bound, 0); + } +} + +#ifdef CONFIG_SOFTMMU +void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) +{ + assert_tb_locked(); + tb_invalidate_phys_range_1(start, end); +} +#else +void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) +{ + assert_memory_lock(); + tb_lock(); + tb_invalidate_phys_range_1(start, end); + tb_unlock(); +} +#endif + #ifdef CONFIG_SOFTMMU /* len must be <= 8 and start must be a multiple of len. * Called via softmmu_template.h when code areas are written to with From patchwork Fri Jun 15 19:43:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138773 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1260838lji; Fri, 15 Jun 2018 13:02:43 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKqeJGe+YLpnmn0YFZXEj51mlFbhY/5gc/CpOEWIwx64+ER6UiCbhB65okWaeSRdePq/CY1 X-Received: by 2002:a0c:92f4:: with SMTP id c49-v6mr2758702qvc.101.1529092963310; Fri, 15 Jun 2018 13:02:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092963; cv=none; d=google.com; s=arc-20160816; b=Sd3IbUWXVNF+bbQTU0fr5TmEPqShvkQUo3IMriaMfPBD99iMf1ZqCNwhWuTdcB6+g/ hcpYfY9m4cnXs1/zRXJbxfHP7eCbQkC2cPgdiZ9GXbfz3Q4he9rciU8XjAsE6r3c4u13 Z2j3a+IN9VQRqIKg2Kc3MGkiQ9XxahxfvRBFakUYZSmO3Q9X2qXPtjQEu7DPRjUnz3pz prVmkfeb8f/d35NaDNjZnGbYS4qxADU16sQnNX2e8/LnZ0v7AdpkbRM5ly3ygHO2iQIA aihJu9vn7zOljLVJNK5bis/q7ASf/2354xhnOs8NW6Y/rDNkVK5sKZMo7irbhid4I6yC fYgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=M42XtL4pwgKM7OvxKJv5hYdYEoKNl+u9A/oll76Ihd4=; b=cBKIo3I6M99luZuY6aENpEQvMCywSM9NgQKtHAQ3rgY6da5J/j4EvvPYr7btJkmCW6 0lBU9bKEiIXPTHHg9dmzc3GE4cwCgd2tOJCp1BrIcN0ttBF2MfEnnNF2FdYtp5K4RFa/ Ui7PXCixGw5ydAIc2ITtKXPibqVvpyyEbuE75XA2bcvghZGQM/mD+BuP3zDh8FppvJzs Zx26k7YBTl1nBQP/PYRPUEBXjwwOxaea7o+vXoSvJ4x5FKxmgktutPXEmeeZJY1FEr3q kCmh7pOGJzx52mWsNwzIZsfjpYDWM83OfJr26siIcjnII3NDI4BVgi79P9NrS2+gVRrS vqSw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=C+7ZfFC9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id y2-v6si3344380qvi.233.2018.06.15.13.02.43 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 13:02:43 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=C+7ZfFC9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49058 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuvi-0006Xa-Nt for patch@linaro.org; Fri, 15 Jun 2018 16:02:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51616) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTue0-0000T1-R1 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudy-0002Zr-BL for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:24 -0400 Received: from mail-pg0-x22b.google.com ([2607:f8b0:400e:c05::22b]:46705) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudx-0002ZR-W5 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:22 -0400 Received: by mail-pg0-x22b.google.com with SMTP id d2-v6so4848020pga.13 for ; Fri, 15 Jun 2018 12:44:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=M42XtL4pwgKM7OvxKJv5hYdYEoKNl+u9A/oll76Ihd4=; b=C+7ZfFC9s4B8o/dj02dACkBDdY69TmRPMBT80LdqRC5b3nDzQmdIe8j0oPFH3ttRVC 9aPfbJhMWY6bYzr91UvDqFdHXuiANeiNtH6O/mIgbMsoyij1tVrzQnsSf6OBBt30Lfod vgACbix40WqkreR4zVEsCg8VQ530D8WBQ8F9s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=M42XtL4pwgKM7OvxKJv5hYdYEoKNl+u9A/oll76Ihd4=; b=kjqb34rsEQjgUbQR3pyr9wz1bTEkF49jIJs0QTwWgDIYHCnTA770I5eTSAWWPaAzQT /kccQlFWZhLPOU9QkpqjlJ3SzmyMJAzYDc9je9eoUTHLNzopENfNHi1RlSfiUOJt9sjd SiOZC7zv4t2Z2Vu0YidtdhS3IC2tujnff+7CgI6LpgEy05yJeJC1Bfhvo+AhjADT7aO8 0vI8JTR5RkYnoNNKAhEzZplHWxBdbiTJfl54tfzeTW3/SM08Ez5+LUA7uKqJAMxrT8En oFAYwFAVZgx4Pa/p0cLyt3BUQsbJHFcO6YCulWUHFMXkT7FSRB9QPx41dk8usqDsqQLY /2QQ== X-Gm-Message-State: APt69E1QKJo6eJLepy7umMvZmNMIwkLwLV7am1sy7fLjSJpux4S8VFsw 7ipF/V0MUzX4WhYaloimKrSgYoCCHu4= X-Received: by 2002:a62:484d:: with SMTP id v74-v6mr3407545pfa.256.1529091860389; Fri, 15 Jun 2018 12:44:20 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.18 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:19 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:46 -1000 Message-Id: <20180615194354.12489-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::22b Subject: [Qemu-devel] [PULL v2 11/19] translate-all: use per-page locking in !user-mode X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" Groundwork for supporting parallel TCG generation. Instead of using a global lock (tb_lock) to protect changes to pages, use fine-grained, per-page locks in !user-mode. User-mode stays with mmap_lock. Sometimes changes need to happen atomically on more than one page (e.g. when a TB that spans across two pages is added/invalidated, or when a range of pages is invalidated). We therefore introduce struct page_collection, which helps us keep track of a set of pages that have been locked in the appropriate locking order (i.e. by ascending page index). This commit first introduces the structs and the function helpers, to then convert the calling code to use per-page locking. Note that tb_lock is not removed yet. While at it, rename tb_alloc_page to tb_page_add, which pairs with tb_page_remove. Reviewed-by: Richard Henderson Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/translate-all.h | 3 + include/exec/exec-all.h | 3 +- accel/tcg/translate-all.c | 444 ++++++++++++++++++++++++++++++++++---- 3 files changed, 409 insertions(+), 41 deletions(-) -- 2.17.1 diff --git a/accel/tcg/translate-all.h b/accel/tcg/translate-all.h index ba8e4d63c4..6d1d2588b5 100644 --- a/accel/tcg/translate-all.h +++ b/accel/tcg/translate-all.h @@ -23,6 +23,9 @@ /* translate-all.c */ +struct page_collection *page_collection_lock(tb_page_addr_t start, + tb_page_addr_t end); +void page_collection_unlock(struct page_collection *set); void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len); void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, int is_cpu_write_access); diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 07653d3c92..8d92e3cea9 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -359,7 +359,8 @@ struct TranslationBlock { /* original tb when cflags has CF_NOCACHE */ struct TranslationBlock *orig_tb; /* first and second physical page containing code. The lower bit - of the pointer tells the index in page_next[] */ + of the pointer tells the index in page_next[]. + The list is protected by the TB's page('s) lock(s) */ uintptr_t page_next[2]; tb_page_addr_t page_addr[2]; diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 83bb40fb20..1cc7aab82c 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -112,8 +112,55 @@ typedef struct PageDesc { #else unsigned long flags; #endif +#ifndef CONFIG_USER_ONLY + QemuSpin lock; +#endif } PageDesc; +/** + * struct page_entry - page descriptor entry + * @pd: pointer to the &struct PageDesc of the page this entry represents + * @index: page index of the page + * @locked: whether the page is locked + * + * This struct helps us keep track of the locked state of a page, without + * bloating &struct PageDesc. + * + * A page lock protects accesses to all fields of &struct PageDesc. + * + * See also: &struct page_collection. + */ +struct page_entry { + PageDesc *pd; + tb_page_addr_t index; + bool locked; +}; + +/** + * struct page_collection - tracks a set of pages (i.e. &struct page_entry's) + * @tree: Binary search tree (BST) of the pages, with key == page index + * @max: Pointer to the page in @tree with the highest page index + * + * To avoid deadlock we lock pages in ascending order of page index. + * When operating on a set of pages, we need to keep track of them so that + * we can lock them in order and also unlock them later. For this we collect + * pages (i.e. &struct page_entry's) in a binary search @tree. Given that the + * @tree implementation we use does not provide an O(1) operation to obtain the + * highest-ranked element, we use @max to keep track of the inserted page + * with the highest index. This is valuable because if a page is not in + * the tree and its index is higher than @max's, then we can lock it + * without breaking the locking order rule. + * + * Note on naming: 'struct page_set' would be shorter, but we already have a few + * page_set_*() helpers, so page_collection is used instead to avoid confusion. + * + * See also: page_collection_lock(). + */ +struct page_collection { + GTree *tree; + struct page_entry *max; +}; + /* list iterators for lists of tagged pointers in TranslationBlock */ #define TB_FOR_EACH_TAGGED(head, tb, n, field) \ for (n = (head) & 1, tb = (TranslationBlock *)((head) & ~1); \ @@ -507,6 +554,15 @@ static PageDesc *page_find_alloc(tb_page_addr_t index, int alloc) return NULL; } pd = g_new0(PageDesc, V_L2_SIZE); +#ifndef CONFIG_USER_ONLY + { + int i; + + for (i = 0; i < V_L2_SIZE; i++) { + qemu_spin_init(&pd[i].lock); + } + } +#endif existing = atomic_cmpxchg(lp, NULL, pd); if (unlikely(existing)) { g_free(pd); @@ -522,6 +578,253 @@ static inline PageDesc *page_find(tb_page_addr_t index) return page_find_alloc(index, 0); } +static void page_lock_pair(PageDesc **ret_p1, tb_page_addr_t phys1, + PageDesc **ret_p2, tb_page_addr_t phys2, int alloc); + +/* In user-mode page locks aren't used; mmap_lock is enough */ +#ifdef CONFIG_USER_ONLY +static inline void page_lock(PageDesc *pd) +{ } + +static inline void page_unlock(PageDesc *pd) +{ } + +static inline void page_lock_tb(const TranslationBlock *tb) +{ } + +static inline void page_unlock_tb(const TranslationBlock *tb) +{ } + +struct page_collection * +page_collection_lock(tb_page_addr_t start, tb_page_addr_t end) +{ + return NULL; +} + +void page_collection_unlock(struct page_collection *set) +{ } +#else /* !CONFIG_USER_ONLY */ + +static inline void page_lock(PageDesc *pd) +{ + qemu_spin_lock(&pd->lock); +} + +static inline void page_unlock(PageDesc *pd) +{ + qemu_spin_unlock(&pd->lock); +} + +/* lock the page(s) of a TB in the correct acquisition order */ +static inline void page_lock_tb(const TranslationBlock *tb) +{ + page_lock_pair(NULL, tb->page_addr[0], NULL, tb->page_addr[1], 0); +} + +static inline void page_unlock_tb(const TranslationBlock *tb) +{ + page_unlock(page_find(tb->page_addr[0] >> TARGET_PAGE_BITS)); + if (unlikely(tb->page_addr[1] != -1)) { + page_unlock(page_find(tb->page_addr[1] >> TARGET_PAGE_BITS)); + } +} + +static inline struct page_entry * +page_entry_new(PageDesc *pd, tb_page_addr_t index) +{ + struct page_entry *pe = g_malloc(sizeof(*pe)); + + pe->index = index; + pe->pd = pd; + pe->locked = false; + return pe; +} + +static void page_entry_destroy(gpointer p) +{ + struct page_entry *pe = p; + + g_assert(pe->locked); + page_unlock(pe->pd); + g_free(pe); +} + +/* returns false on success */ +static bool page_entry_trylock(struct page_entry *pe) +{ + bool busy; + + busy = qemu_spin_trylock(&pe->pd->lock); + if (!busy) { + g_assert(!pe->locked); + pe->locked = true; + } + return busy; +} + +static void do_page_entry_lock(struct page_entry *pe) +{ + page_lock(pe->pd); + g_assert(!pe->locked); + pe->locked = true; +} + +static gboolean page_entry_lock(gpointer key, gpointer value, gpointer data) +{ + struct page_entry *pe = value; + + do_page_entry_lock(pe); + return FALSE; +} + +static gboolean page_entry_unlock(gpointer key, gpointer value, gpointer data) +{ + struct page_entry *pe = value; + + if (pe->locked) { + pe->locked = false; + page_unlock(pe->pd); + } + return FALSE; +} + +/* + * Trylock a page, and if successful, add the page to a collection. + * Returns true ("busy") if the page could not be locked; false otherwise. + */ +static bool page_trylock_add(struct page_collection *set, tb_page_addr_t addr) +{ + tb_page_addr_t index = addr >> TARGET_PAGE_BITS; + struct page_entry *pe; + PageDesc *pd; + + pe = g_tree_lookup(set->tree, &index); + if (pe) { + return false; + } + + pd = page_find(index); + if (pd == NULL) { + return false; + } + + pe = page_entry_new(pd, index); + g_tree_insert(set->tree, &pe->index, pe); + + /* + * If this is either (1) the first insertion or (2) a page whose index + * is higher than any other so far, just lock the page and move on. + */ + if (set->max == NULL || pe->index > set->max->index) { + set->max = pe; + do_page_entry_lock(pe); + return false; + } + /* + * Try to acquire out-of-order lock; if busy, return busy so that we acquire + * locks in order. + */ + return page_entry_trylock(pe); +} + +static gint tb_page_addr_cmp(gconstpointer ap, gconstpointer bp, gpointer udata) +{ + tb_page_addr_t a = *(const tb_page_addr_t *)ap; + tb_page_addr_t b = *(const tb_page_addr_t *)bp; + + if (a == b) { + return 0; + } else if (a < b) { + return -1; + } + return 1; +} + +/* + * Lock a range of pages ([@start,@end[) as well as the pages of all + * intersecting TBs. + * Locking order: acquire locks in ascending order of page index. + */ +struct page_collection * +page_collection_lock(tb_page_addr_t start, tb_page_addr_t end) +{ + struct page_collection *set = g_malloc(sizeof(*set)); + tb_page_addr_t index; + PageDesc *pd; + + start >>= TARGET_PAGE_BITS; + end >>= TARGET_PAGE_BITS; + g_assert(start <= end); + + set->tree = g_tree_new_full(tb_page_addr_cmp, NULL, NULL, + page_entry_destroy); + set->max = NULL; + + retry: + g_tree_foreach(set->tree, page_entry_lock, NULL); + + for (index = start; index <= end; index++) { + TranslationBlock *tb; + int n; + + pd = page_find(index); + if (pd == NULL) { + continue; + } + if (page_trylock_add(set, index << TARGET_PAGE_BITS)) { + g_tree_foreach(set->tree, page_entry_unlock, NULL); + goto retry; + } + PAGE_FOR_EACH_TB(pd, tb, n) { + if (page_trylock_add(set, tb->page_addr[0]) || + (tb->page_addr[1] != -1 && + page_trylock_add(set, tb->page_addr[1]))) { + /* drop all locks, and reacquire in order */ + g_tree_foreach(set->tree, page_entry_unlock, NULL); + goto retry; + } + } + } + return set; +} + +void page_collection_unlock(struct page_collection *set) +{ + /* entries are unlocked and freed via page_entry_destroy */ + g_tree_destroy(set->tree); + g_free(set); +} + +#endif /* !CONFIG_USER_ONLY */ + +static void page_lock_pair(PageDesc **ret_p1, tb_page_addr_t phys1, + PageDesc **ret_p2, tb_page_addr_t phys2, int alloc) +{ + PageDesc *p1, *p2; + + assert_memory_lock(); + g_assert(phys1 != -1 && phys1 != phys2); + p1 = page_find_alloc(phys1 >> TARGET_PAGE_BITS, alloc); + if (ret_p1) { + *ret_p1 = p1; + } + if (likely(phys2 == -1)) { + page_lock(p1); + return; + } + p2 = page_find_alloc(phys2 >> TARGET_PAGE_BITS, alloc); + if (ret_p2) { + *ret_p2 = p2; + } + if (phys1 < phys2) { + page_lock(p1); + page_lock(p2); + } else { + page_lock(p2); + page_lock(p1); + } +} + #if defined(CONFIG_USER_ONLY) /* Currently it is not recommended to allocate big chunks of data in user mode. It will change when a dedicated libc will be used. */ @@ -807,6 +1110,7 @@ static TranslationBlock *tb_alloc(target_ulong pc) return tb; } +/* call with @p->lock held */ static inline void invalidate_page_bitmap(PageDesc *p) { #ifdef CONFIG_SOFTMMU @@ -828,8 +1132,10 @@ static void page_flush_tb_1(int level, void **lp) PageDesc *pd = *lp; for (i = 0; i < V_L2_SIZE; ++i) { + page_lock(&pd[i]); pd[i].first_tb = (uintptr_t)NULL; invalidate_page_bitmap(pd + i); + page_unlock(&pd[i]); } } else { void **pp = *lp; @@ -956,6 +1262,7 @@ static void tb_page_check(void) #endif /* CONFIG_USER_ONLY */ +/* call with @pd->lock held */ static inline void tb_page_remove(PageDesc *pd, TranslationBlock *tb) { TranslationBlock *tb1; @@ -1032,11 +1339,8 @@ static inline void tb_jmp_unlink(TranslationBlock *tb) } } -/* invalidate one TB - * - * Called with tb_lock held. - */ -void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr) +/* If @rm_from_page_list is set, call with the TB's pages' locks held */ +static void do_tb_phys_invalidate(TranslationBlock *tb, bool rm_from_page_list) { CPUState *cpu; PageDesc *p; @@ -1056,15 +1360,15 @@ void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr) } /* remove the TB from the page list */ - if (tb->page_addr[0] != page_addr) { + if (rm_from_page_list) { p = page_find(tb->page_addr[0] >> TARGET_PAGE_BITS); tb_page_remove(p, tb); invalidate_page_bitmap(p); - } - if (tb->page_addr[1] != -1 && tb->page_addr[1] != page_addr) { - p = page_find(tb->page_addr[1] >> TARGET_PAGE_BITS); - tb_page_remove(p, tb); - invalidate_page_bitmap(p); + if (tb->page_addr[1] != -1) { + p = page_find(tb->page_addr[1] >> TARGET_PAGE_BITS); + tb_page_remove(p, tb); + invalidate_page_bitmap(p); + } } /* remove the TB from the hash list */ @@ -1086,7 +1390,28 @@ void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr) tcg_ctx->tb_phys_invalidate_count + 1); } +static void tb_phys_invalidate__locked(TranslationBlock *tb) +{ + do_tb_phys_invalidate(tb, true); +} + +/* invalidate one TB + * + * Called with tb_lock held. + */ +void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr) +{ + if (page_addr == -1) { + page_lock_tb(tb); + do_tb_phys_invalidate(tb, true); + page_unlock_tb(tb); + } else { + do_tb_phys_invalidate(tb, false); + } +} + #ifdef CONFIG_SOFTMMU +/* call with @p->lock held */ static void build_page_bitmap(PageDesc *p) { int n, tb_start, tb_end; @@ -1116,11 +1441,11 @@ static void build_page_bitmap(PageDesc *p) /* add the tb in the target page and protect it if necessary * * Called with mmap_lock held for user-mode emulation. + * Called with @p->lock held. */ -static inline void tb_alloc_page(TranslationBlock *tb, - unsigned int n, tb_page_addr_t page_addr) +static inline void tb_page_add(PageDesc *p, TranslationBlock *tb, + unsigned int n, tb_page_addr_t page_addr) { - PageDesc *p; #ifndef CONFIG_USER_ONLY bool page_already_protected; #endif @@ -1128,7 +1453,6 @@ static inline void tb_alloc_page(TranslationBlock *tb, assert_memory_lock(); tb->page_addr[n] = page_addr; - p = page_find_alloc(page_addr >> TARGET_PAGE_BITS, 1); tb->page_next[n] = p->first_tb; #ifndef CONFIG_USER_ONLY page_already_protected = p->first_tb != (uintptr_t)NULL; @@ -1180,18 +1504,28 @@ static inline void tb_alloc_page(TranslationBlock *tb, static void tb_link_page(TranslationBlock *tb, tb_page_addr_t phys_pc, tb_page_addr_t phys_page2) { + PageDesc *p; + PageDesc *p2 = NULL; uint32_t h; assert_memory_lock(); - /* add in the page list */ - tb_alloc_page(tb, 0, phys_pc & TARGET_PAGE_MASK); - if (phys_page2 != -1) { - tb_alloc_page(tb, 1, phys_page2); + /* + * Add the TB to the page list, acquiring first the pages's locks. + */ + page_lock_pair(&p, phys_pc, &p2, phys_page2, 1); + tb_page_add(p, tb, 0, phys_pc & TARGET_PAGE_MASK); + if (p2) { + tb_page_add(p2, tb, 1, phys_page2); } else { tb->page_addr[1] = -1; } + if (p2) { + page_unlock(p2); + } + page_unlock(p); + /* add in the hash table */ h = tb_hash_func(phys_pc, tb->pc, tb->flags, tb->cflags & CF_HASH_MASK, tb->trace_vcpu_dstate); @@ -1364,21 +1698,17 @@ TranslationBlock *tb_gen_code(CPUState *cpu, } /* - * Invalidate all TBs which intersect with the target physical address range - * [start;end[. NOTE: start and end must refer to the *same* physical page. - * 'is_cpu_write_access' should be true if called from a real cpu write - * access: the virtual CPU will exit the current TB if code is modified inside - * this TB. - * - * Called with tb_lock/mmap_lock held for user-mode emulation - * Called with tb_lock held for system-mode emulation + * Call with all @pages locked. + * @p must be non-NULL. */ -void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, - int is_cpu_write_access) +static void +tb_invalidate_phys_page_range__locked(struct page_collection *pages, + PageDesc *p, tb_page_addr_t start, + tb_page_addr_t end, + int is_cpu_write_access) { TranslationBlock *tb; tb_page_addr_t tb_start, tb_end; - PageDesc *p; int n; #ifdef TARGET_HAS_PRECISE_SMC CPUState *cpu = current_cpu; @@ -1394,10 +1724,6 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, assert_memory_lock(); assert_tb_locked(); - p = page_find(start >> TARGET_PAGE_BITS); - if (!p) { - return; - } #if defined(TARGET_HAS_PRECISE_SMC) if (cpu != NULL) { env = cpu->env_ptr; @@ -1443,7 +1769,7 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, ¤t_flags); } #endif /* TARGET_HAS_PRECISE_SMC */ - tb_phys_invalidate(tb, -1); + tb_phys_invalidate__locked(tb); } } #if !defined(CONFIG_USER_ONLY) @@ -1455,6 +1781,7 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, #endif #ifdef TARGET_HAS_PRECISE_SMC if (current_tb_modified) { + page_collection_unlock(pages); /* Force execution of one insn next time. */ cpu->cflags_next_tb = 1 | curr_cflags(); cpu_loop_exit_noexc(cpu); @@ -1462,6 +1789,35 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, #endif } +/* + * Invalidate all TBs which intersect with the target physical address range + * [start;end[. NOTE: start and end must refer to the *same* physical page. + * 'is_cpu_write_access' should be true if called from a real cpu write + * access: the virtual CPU will exit the current TB if code is modified inside + * this TB. + * + * Called with tb_lock/mmap_lock held for user-mode emulation + * Called with tb_lock held for system-mode emulation + */ +void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, + int is_cpu_write_access) +{ + struct page_collection *pages; + PageDesc *p; + + assert_memory_lock(); + assert_tb_locked(); + + p = page_find(start >> TARGET_PAGE_BITS); + if (p == NULL) { + return; + } + pages = page_collection_lock(start, end); + tb_invalidate_phys_page_range__locked(pages, p, start, end, + is_cpu_write_access); + page_collection_unlock(pages); +} + /* * Invalidate all TBs which intersect with the target physical address range * [start;end[. NOTE: start and end may refer to *different* physical pages. @@ -1474,15 +1830,22 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, */ static void tb_invalidate_phys_range_1(tb_page_addr_t start, tb_page_addr_t end) { + struct page_collection *pages; tb_page_addr_t next; + pages = page_collection_lock(start, end); for (next = (start & TARGET_PAGE_MASK) + TARGET_PAGE_SIZE; start < end; start = next, next += TARGET_PAGE_SIZE) { + PageDesc *pd = page_find(start >> TARGET_PAGE_BITS); tb_page_addr_t bound = MIN(next, end); - tb_invalidate_phys_page_range(start, bound, 0); + if (pd == NULL) { + continue; + } + tb_invalidate_phys_page_range__locked(pages, pd, start, bound, 0); } + page_collection_unlock(pages); } #ifdef CONFIG_SOFTMMU @@ -1508,6 +1871,7 @@ void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) */ void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len) { + struct page_collection *pages; PageDesc *p; #if 0 @@ -1525,11 +1889,10 @@ void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len) if (!p) { return; } + + pages = page_collection_lock(start, start + len); if (!p->code_bitmap && ++p->code_write_count >= SMC_BITMAP_USE_THRESHOLD) { - /* build code bitmap. FIXME: writes should be protected by - * tb_lock, reads by tb_lock or RCU. - */ build_page_bitmap(p); } if (p->code_bitmap) { @@ -1543,8 +1906,9 @@ void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len) } } else { do_invalidate: - tb_invalidate_phys_page_range(start, start + len, 1); + tb_invalidate_phys_page_range__locked(pages, p, start, start + len, 1); } + page_collection_unlock(pages); } #else /* Called with mmap_lock held. If pc is not 0 then it indicates the From patchwork Fri Jun 15 19:43:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138777 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1264401lji; Fri, 15 Jun 2018 13:06:05 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKducXwg209auHsjI6SKlJvnUomI+7tU5R8Y4Q2f0fc8NXOF5vdIwivIFZtdeZtLS2sBWDj X-Received: by 2002:a37:e207:: with SMTP id g7-v6mr2664631qki.180.1529093165115; Fri, 15 Jun 2018 13:06:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529093165; cv=none; d=google.com; s=arc-20160816; b=pvhUI/8h0rU28BLBXLwbXvfqRzqqv7bd/MeHj4ZGfMnOix63REjUdF31ZfRclO7XiY k9UZWXkIDKwNj7TCgUNqdGDdD/xkKcE7nYV8QRobMY78dLW4fDCpzR9FiUAJp4P2ZkcS KnPnkvdmOLED4xmiAO8k5lqp6AS/IZ70lD7HDzVLQ45JtU8nFhLS4y31gawuHcbiHLmw Vi816KwB2JH7HY5IeJDz1qxrYSFw7BqMVoZm3MZsqM/8/eTAYJsAL9jzXrcXcvx9hpZb hxBlBQ4bT8VS741qKKYIGzlvNS3zBsQsXcoPSa0Ge5W6CnYg1ZNusgZyvPcFbrvuyj5c MyHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=B2SA4mmIv/f/uj1OaCwYNsFOq5P2Uw7DvOSNDc7St/o=; b=ILs+gEFANmXaV805GUfDA3Ewofzoq8VT3+Oo+aVVP7RFWuSg7VeABA+/7hpqhiKcvO hXZcDwJ6VYyHYOEp2poivaMoHLbKmYJPTk6IVo5bfW+Wkykdoi92w5d8CoSDp9TKv0t4 vXANGFKrGPf5oIEljqVZXkshYX+AaNxUTVQaWXXeO/CNTt89p4vi26/O+wI1iVWK8K6x QtmqCHqtcnRHAqH+OcWRgEh/W09ExHUafXZEJVGiYhaZKL4R4eIAu1OPiZaCToURb9km 89KqVOZcOiTnuY55xVd1e4WkmfYdnm36+1rX1rwLKcQBMZ+8J5WaDsTv+w2SDhKShtYp gm6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=RwF92+uQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id 13-v6si1548966qts.384.2018.06.15.13.06.04 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 13:06:05 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=RwF92+uQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49076 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuyy-0000pS-HV for patch@linaro.org; Fri, 15 Jun 2018 16:06:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51644) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTue4-0000Vl-7z for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTudz-0002ap-Q5 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:28 -0400 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:37971) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTudz-0002aP-Gk for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:23 -0400 Received: by mail-pf0-x243.google.com with SMTP id b74-v6so5326324pfl.5 for ; Fri, 15 Jun 2018 12:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B2SA4mmIv/f/uj1OaCwYNsFOq5P2Uw7DvOSNDc7St/o=; b=RwF92+uQrtyscbKm/krbAw4wxpS/seATNW4DrRpuQY50ACr9ggKrWxRJBsg4tnppyc QoIxlPKLJ3/QXd5pC2lRJBMuNA7edutTjzJaVGaNff4v+QP0HWIrq9E9bdMZZZiP06my 3WQY3hDevsqYaFpr+0MmWSWhfcg5RgV2ruD80= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B2SA4mmIv/f/uj1OaCwYNsFOq5P2Uw7DvOSNDc7St/o=; b=GAmx2b4gxitNcUVfYzJSDXeBPkXT+AuhwQY9BLC175qge1zGjGmSoEBN822FNAWOAe pn2yZqLPXGv5lfsK3Ystpyj/QpVa6pCUdlA330zxzHMP+M/CAt4Y8mER62MKyaUb0hhD bXK1Uzcgs4m/piFV2UDXGtc39Q9vW6K9NDYmt+AXSM4vVQgDVdWYADeccxi83tg25z0h T4C+QCEqGPBEn5R5sdQn/3MWW0W2HJj02NqGpmMxUJ40Pq4eRwau0uOxBCPxPo4N4r4S 4LjAeR3eiATmbv5RDK6Kyhuc4c0gYRJEdKfOpmPugc/zRNvvmSm0YdDhS9tKnw5Xn6/y Ov8Q== X-Gm-Message-State: APt69E0yCTtxX5annqeRB8g5tIR/D5UJ60ANb2ZEmMmfRg2HvzsobGPr Q1lGe9+WCpwRnFdKh8OmTd1eSIckHhU= X-Received: by 2002:a63:7b07:: with SMTP id w7-v6mr2713145pgc.199.1529091862279; Fri, 15 Jun 2018 12:44:22 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.20 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:21 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:47 -1000 Message-Id: <20180615194354.12489-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PULL v2 12/19] translate-all: add page_locked assertions X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" This is only compiled under CONFIG_DEBUG_TCG to avoid bloating the binary. In user-mode, assert_page_locked is equivalent to assert_mmap_lock. Note: There are some tb_lock assertions left that will be removed by later patches. Reviewed-by: Richard Henderson Suggested-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/translate-all.c | 82 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 79 insertions(+), 3 deletions(-) -- 2.17.1 diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 1cc7aab82c..8b378586f4 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -583,6 +583,9 @@ static void page_lock_pair(PageDesc **ret_p1, tb_page_addr_t phys1, /* In user-mode page locks aren't used; mmap_lock is enough */ #ifdef CONFIG_USER_ONLY + +#define assert_page_locked(pd) tcg_debug_assert(have_mmap_lock()) + static inline void page_lock(PageDesc *pd) { } @@ -605,14 +608,80 @@ void page_collection_unlock(struct page_collection *set) { } #else /* !CONFIG_USER_ONLY */ +#ifdef CONFIG_DEBUG_TCG + +static __thread GHashTable *ht_pages_locked_debug; + +static void ht_pages_locked_debug_init(void) +{ + if (ht_pages_locked_debug) { + return; + } + ht_pages_locked_debug = g_hash_table_new(NULL, NULL); +} + +static bool page_is_locked(const PageDesc *pd) +{ + PageDesc *found; + + ht_pages_locked_debug_init(); + found = g_hash_table_lookup(ht_pages_locked_debug, pd); + return !!found; +} + +static void page_lock__debug(PageDesc *pd) +{ + ht_pages_locked_debug_init(); + g_assert(!page_is_locked(pd)); + g_hash_table_insert(ht_pages_locked_debug, pd, pd); +} + +static void page_unlock__debug(const PageDesc *pd) +{ + bool removed; + + ht_pages_locked_debug_init(); + g_assert(page_is_locked(pd)); + removed = g_hash_table_remove(ht_pages_locked_debug, pd); + g_assert(removed); +} + +static void +do_assert_page_locked(const PageDesc *pd, const char *file, int line) +{ + if (unlikely(!page_is_locked(pd))) { + error_report("assert_page_lock: PageDesc %p not locked @ %s:%d", + pd, file, line); + abort(); + } +} + +#define assert_page_locked(pd) do_assert_page_locked(pd, __FILE__, __LINE__) + +#else /* !CONFIG_DEBUG_TCG */ + +#define assert_page_locked(pd) + +static inline void page_lock__debug(const PageDesc *pd) +{ +} + +static inline void page_unlock__debug(const PageDesc *pd) +{ +} + +#endif /* CONFIG_DEBUG_TCG */ + static inline void page_lock(PageDesc *pd) { + page_lock__debug(pd); qemu_spin_lock(&pd->lock); } static inline void page_unlock(PageDesc *pd) { qemu_spin_unlock(&pd->lock); + page_unlock__debug(pd); } /* lock the page(s) of a TB in the correct acquisition order */ @@ -658,6 +727,7 @@ static bool page_entry_trylock(struct page_entry *pe) if (!busy) { g_assert(!pe->locked); pe->locked = true; + page_lock__debug(pe->pd); } return busy; } @@ -775,6 +845,7 @@ page_collection_lock(tb_page_addr_t start, tb_page_addr_t end) g_tree_foreach(set->tree, page_entry_unlock, NULL); goto retry; } + assert_page_locked(pd); PAGE_FOR_EACH_TB(pd, tb, n) { if (page_trylock_add(set, tb->page_addr[0]) || (tb->page_addr[1] != -1 && @@ -1113,6 +1184,7 @@ static TranslationBlock *tb_alloc(target_ulong pc) /* call with @p->lock held */ static inline void invalidate_page_bitmap(PageDesc *p) { + assert_page_locked(p); #ifdef CONFIG_SOFTMMU g_free(p->code_bitmap); p->code_bitmap = NULL; @@ -1269,6 +1341,7 @@ static inline void tb_page_remove(PageDesc *pd, TranslationBlock *tb) uintptr_t *pprev; unsigned int n1; + assert_page_locked(pd); pprev = &pd->first_tb; PAGE_FOR_EACH_TB(pd, tb1, n1) { if (tb1 == tb) { @@ -1417,6 +1490,7 @@ static void build_page_bitmap(PageDesc *p) int n, tb_start, tb_end; TranslationBlock *tb; + assert_page_locked(p); p->code_bitmap = bitmap_new(TARGET_PAGE_SIZE); PAGE_FOR_EACH_TB(p, tb, n) { @@ -1450,7 +1524,7 @@ static inline void tb_page_add(PageDesc *p, TranslationBlock *tb, bool page_already_protected; #endif - assert_memory_lock(); + assert_page_locked(p); tb->page_addr[n] = page_addr; tb->page_next[n] = p->first_tb; @@ -1721,8 +1795,7 @@ tb_invalidate_phys_page_range__locked(struct page_collection *pages, uint32_t current_flags = 0; #endif /* TARGET_HAS_PRECISE_SMC */ - assert_memory_lock(); - assert_tb_locked(); + assert_page_locked(p); #if defined(TARGET_HAS_PRECISE_SMC) if (cpu != NULL) { @@ -1734,6 +1807,7 @@ tb_invalidate_phys_page_range__locked(struct page_collection *pages, /* XXX: see if in some cases it could be faster to invalidate all the code */ PAGE_FOR_EACH_TB(p, tb, n) { + assert_page_locked(p); /* NOTE: this is subtle as a TB may span two physical pages */ if (n == 0) { /* NOTE: tb_end may be after the end of the page, but @@ -1891,6 +1965,7 @@ void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len) } pages = page_collection_lock(start, start + len); + assert_page_locked(p); if (!p->code_bitmap && ++p->code_write_count >= SMC_BITMAP_USE_THRESHOLD) { build_page_bitmap(p); @@ -1949,6 +2024,7 @@ static bool tb_invalidate_phys_page(tb_page_addr_t addr, uintptr_t pc) env = cpu->env_ptr; } #endif + assert_page_locked(p); PAGE_FOR_EACH_TB(p, tb, n) { #ifdef TARGET_HAS_PRECISE_SMC if (current_tb == tb && From patchwork Fri Jun 15 19:43:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138759 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1250390lji; Fri, 15 Jun 2018 12:51:15 -0700 (PDT) X-Google-Smtp-Source: ADUXVKI8V9u+I/jLESiqbWHaH0+S3TQSci7z+SfwUH2c7XrGIvV9wT0NwMag/i9sUuNwtv5+P241 X-Received: by 2002:a0c:ae63:: with SMTP id z32-v6mr2658979qvc.108.1529092275194; Fri, 15 Jun 2018 12:51:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092275; cv=none; d=google.com; s=arc-20160816; b=KV/oc/agGKjcjNw2zVsWlvU7U8cq5lMGV16SHasXm7uLNAbtC/IvyhlFPxsy+iDKEL FL8WggT6+wyi/heAJZNdRn6B1v3yUlbzxrpg5JKx+AR63BVM4NaPB6+IjdcmdH6EHhrR 7C7p7QGACjgrzdqu8uz05z4obXy6tIOLN6UkAVtVovrRUbB5Gl/WFn4KTW0H/vpf0diE Sc3AznTrlBqXy/UuDcsTAlS+7ux4uR6sScY/BZaM+4ygkf0n66yAWAoAevSHvvXTBDaq 8zMtwTTScUyezAnQmJO5BiGX3sLE+eO+Pn3UFAbPrjA1AH3UCxBf5369+tTd/t08hHj5 tdYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=zOVWT9oyOnUXGeg0GOygk0Sk7eelIiTSGEWZKkbw/co=; b=yT5weMrSUfZf99YK8KRt4x7yM+gHeOhdKsT0chq6w82WycMnz/YOrRzP5FEzCs+H33 FCy6M0u5+ZvnLMqEipsxjp10QOTWtq3nYAVAaTPihZna/rOWLIAcTX2JYX9/n6SK4d6O 0glGRbpg7mQRRaxUUrb2NdVCRSVSd+8It26hAkWhbNRBd5J9tUjKLEI5PfELI1nC4K+j Dry94OdsworgnXYUelmvHh7w94GZlSp6It0viSeVWtiFgTh3oYOPnbvY8Vx/hbZ53l7v 8TGJfGJa4jlrBSkiQisubaiEGZ+iHq9fW32ikKUsLyEkcRy+8YZPoHq+SrI/GrFIDTWn eZLw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=cpyupyEq; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id k30-v6si8440462qtk.393.2018.06.15.12.51.15 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:51:15 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=cpyupyEq; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48989 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTukc-0005Pd-JX for patch@linaro.org; Fri, 15 Jun 2018 15:51:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51649) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTue4-0000WB-NN for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTue1-0002cL-GH for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:28 -0400 Received: from mail-pg0-x22c.google.com ([2607:f8b0:400e:c05::22c]:43852) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTue1-0002bl-Ad for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:25 -0400 Received: by mail-pg0-x22c.google.com with SMTP id a14-v6so4852466pgw.10 for ; Fri, 15 Jun 2018 12:44:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zOVWT9oyOnUXGeg0GOygk0Sk7eelIiTSGEWZKkbw/co=; b=cpyupyEqk1Mqy9ZJW+rxN0ixuiGO/lZ05bgKHYbQs2k0KZORaX1JOy6AIJE5+trDg4 0bFsfW3skcv8MdXzj+SUIkdQ9lkKj0+e+xNx7otrnQAHy1jdCfXTrwx9SA1RFwza4r1U 0AIpFPPvoPdpt1GKMI3FU+Lulp8AHpWcNW1Tw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zOVWT9oyOnUXGeg0GOygk0Sk7eelIiTSGEWZKkbw/co=; b=GigprvWhGzCJtxpBhua7BcnJkQ9074KGfV12XOshhXsx4ziyBT04/doj2daid8AwBQ Idcs96VEXHPM+FW2CDuj19E62Oo+fTzl0z+Te4snPFOz0gIDMB4nsbCnro5zdnN4nxhv ihH1YrzzWRoXnBSYk5gMFj+IFANEXOwDIY/r7AavOzVP6D7aNeBuGpdSQCjxSDLOPegv OXUrRsD9oMWWlw8QeTYKN8l9utxCXr2t+r3nZ048kJucpVcPB0hn3stuYb+t7148zqdI n4OnWfaagV5U0rV/i8POtD1RVG+5REEuq6YD6oKTWIVxs8wpNd49hUF8EdWHdeVVITbr dsxA== X-Gm-Message-State: APt69E1eHxSNfljBK7WCv3Hh68Ktn/CMYIUNMrmv9NJPACq6XC5foeTr QVSfXWKhQ4Hsxq7FNoAlppyvaotRp/I= X-Received: by 2002:a63:6d05:: with SMTP id i5-v6mr2825858pgc.321.1529091864036; Fri, 15 Jun 2018 12:44:24 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.22 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:23 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:48 -1000 Message-Id: <20180615194354.12489-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::22c Subject: [Qemu-devel] [PULL v2 13/19] translate-all: introduce assert_no_pages_locked X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" The appended adds assertions to make sure we do not longjmp with page locks held. Note that user-mode has nothing to check, since page_locks are !user-mode only. Reviewed-by: Richard Henderson Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- include/exec/exec-all.h | 8 ++++++++ accel/tcg/cpu-exec.c | 1 + accel/tcg/translate-all.c | 7 +++++++ 3 files changed, 16 insertions(+) -- 2.17.1 diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 8d92e3cea9..4f07a17052 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -435,6 +435,14 @@ void tb_lock(void); void tb_unlock(void); void tb_lock_reset(void); +#if !defined(CONFIG_USER_ONLY) && defined(CONFIG_DEBUG_TCG) +void assert_no_pages_locked(void); +#else +static inline void assert_no_pages_locked(void) +{ +} +#endif + #if !defined(CONFIG_USER_ONLY) /** diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c index 7570c59f09..d75c35380a 100644 --- a/accel/tcg/cpu-exec.c +++ b/accel/tcg/cpu-exec.c @@ -273,6 +273,7 @@ void cpu_exec_step_atomic(CPUState *cpu) tcg_debug_assert(!have_mmap_lock()); #endif tb_lock_reset(); + assert_no_pages_locked(); } if (in_exclusive_region) { diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 8b378586f4..c75298d08a 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -658,6 +658,12 @@ do_assert_page_locked(const PageDesc *pd, const char *file, int line) #define assert_page_locked(pd) do_assert_page_locked(pd, __FILE__, __LINE__) +void assert_no_pages_locked(void) +{ + ht_pages_locked_debug_init(); + g_assert(g_hash_table_size(ht_pages_locked_debug) == 0); +} + #else /* !CONFIG_DEBUG_TCG */ #define assert_page_locked(pd) @@ -829,6 +835,7 @@ page_collection_lock(tb_page_addr_t start, tb_page_addr_t end) set->tree = g_tree_new_full(tb_page_addr_cmp, NULL, NULL, page_entry_destroy); set->max = NULL; + assert_no_pages_locked(); retry: g_tree_foreach(set->tree, page_entry_lock, NULL); From patchwork Fri Jun 15 19:43:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138760 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1250813lji; Fri, 15 Jun 2018 12:51:45 -0700 (PDT) X-Google-Smtp-Source: ADUXVKK/Bl4OlT/2tPmaDZGDkR419e73vg1TchGxhKSVY1cAUaHoTm9YJSHDjZPbSSLnmzSTNKlD X-Received: by 2002:ac8:2485:: with SMTP id s5-v6mr2793717qts.350.1529092305684; Fri, 15 Jun 2018 12:51:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092305; cv=none; d=google.com; s=arc-20160816; b=BtF+6OA1haIP4+ki5FZJAefZpRb0z7/N/k3dbj7FnwlYt85bCM0UMfcjLBryO7lqoH VFRy7ltJiu0kejhHaOvVuCYuBrz0GLozQRTbipADPzCAUmMtQNMU8oXyeaVw3lV/Nvg5 +oJbv5lt2GZuflIU2KIJh6+rx06Q/XvTP7RlwbERBbBwIDHKHLFCz7jP+SNDbhlUDRaU Elzx/b4YS71VTqHEPu+yeST9T3Y9Z04buzp+OgDQZUweZK79+7uTioco0uNPibp+X9xH jGnfLkcgupL/vuCdkq9zJQ1Lm5KPogcA0y8gLNX0uAn5F+lD+ovya15k+5XgTBqFlaDg iKeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=qODRUs4RMttww7F/RbxWXsb4AB7UFsX6gB2qWNJq0wA=; b=Hs3W0ZHlZSY9JSguG3tCMhgJvh5KIzNntNuD5ezPY1svTe3v4YeykU3tLr4uwQjufw VwvCL3PokALrpdwwN9hhQbjOYTAAAVBiKMTVzRNlVQjpjvvTSn/yx8JHKnlj0TVI8lf+ 6w8QKABtld9NapBfhJAEfRZebMyFqP+I/gLSk7HaTVJt3wTZuRLmcvPga15Dxxm1L7Gh c+Lu3RhAeyp3lchc4yXEf9Qjq0z7HoK5bcJx8UVFGQC4jYuo83SAaB/0erU9cTJFvq6Q 5lbGfBvvZJ6Pui+H73HSPl5qS2WTUpUF3RNRxEXf0McZLjzejPXUPQ1Jdwq93/pGgRu8 yv3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=IoKUZ7Bu; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id y9-v6si8321178qtk.239.2018.06.15.12.51.45 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:51:45 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=IoKUZ7Bu; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48993 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTul7-00063m-3W for patch@linaro.org; Fri, 15 Jun 2018 15:51:45 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51651) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTue4-0000WG-PE for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:30 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTue3-0002dq-HX for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:28 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:39914) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTue3-0002d8-99 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:27 -0400 Received: by mail-pf0-x241.google.com with SMTP id r11-v6so5325969pfl.6 for ; Fri, 15 Jun 2018 12:44:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qODRUs4RMttww7F/RbxWXsb4AB7UFsX6gB2qWNJq0wA=; b=IoKUZ7BuohF5+X6E0vS0OB0kNhLPzkIDeujRysXawcaQQQ1frJM4dPbjQ87RsUORrR pT98r42Qnk+ku18iKpf2ORJjxZjLuubveIA8wSpoOp6w4TR6NySIODsI2sRuNPWMpPD/ 3MQTKKqSIVnne6/n/Oi3pAnNk2TIlX8sc8uNY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qODRUs4RMttww7F/RbxWXsb4AB7UFsX6gB2qWNJq0wA=; b=MmFtf15VRg0/LMhMjR0bNH0jSNc2Nc+1WeigOciYA27yvCyYyU0rmw4yP/XL8Ts8/A qDx0WGuIhsVMm1qUwIolyi01xinvYDL3FJ8Ta57NbBgyy+XZJTMqmPZR9vNqH6pJavt4 mbCS5OK7N8qaOrIJPlsrHyFpEoPO53rn1cl8etAUgnKodvZInEPoKmGxl73AhuQVo5Dq biBZPPAAoyb4AOOpm+a/2Kd5jdhxvRserdzynLJaJBWTtUapcAz2OTJBCXh0LoR2/yDc zbsi/4bVVKjp8Sz3TyDwg9MKP+fvX9StfgmhT6H8HEvz1qKW7778qyIk2FhBO4ddtdtn ZExA== X-Gm-Message-State: APt69E1X9vImLxHnVsnxuC2QF5EXhOlxZVCVdXuW2VtNI/kup98qSCIt wEjop3hcwxrX6c/BH13Qv8Kivfz6zdc= X-Received: by 2002:a63:24c4:: with SMTP id k187-v6mr2802268pgk.434.1529091865931; Fri, 15 Jun 2018 12:44:25 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.24 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:25 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:49 -1000 Message-Id: <20180615194354.12489-15-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PULL v2 14/19] translate-all: discard TB when tb_link_page returns an existing matching TB X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" Use the recently-gained QHT feature of returning the matching TB if it already exists. This allows us to get rid of the lookup we perform right after acquiring tb_lock. Suggested-by: Richard Henderson Reviewed-by: Richard Henderson Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/cpu-exec.c | 14 ++------- accel/tcg/translate-all.c | 50 +++++++++++++++++++++++++++------ docs/devel/multi-thread-tcg.txt | 3 ++ 3 files changed, 46 insertions(+), 21 deletions(-) -- 2.17.1 diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c index d75c35380a..45f6ebc65e 100644 --- a/accel/tcg/cpu-exec.c +++ b/accel/tcg/cpu-exec.c @@ -245,10 +245,7 @@ void cpu_exec_step_atomic(CPUState *cpu) if (tb == NULL) { mmap_lock(); tb_lock(); - tb = tb_htable_lookup(cpu, pc, cs_base, flags, cf_mask); - if (likely(tb == NULL)) { - tb = tb_gen_code(cpu, pc, cs_base, flags, cflags); - } + tb = tb_gen_code(cpu, pc, cs_base, flags, cflags); tb_unlock(); mmap_unlock(); } @@ -398,14 +395,7 @@ static inline TranslationBlock *tb_find(CPUState *cpu, tb_lock(); acquired_tb_lock = true; - /* There's a chance that our desired tb has been translated while - * taking the locks so we check again inside the lock. - */ - tb = tb_htable_lookup(cpu, pc, cs_base, flags, cf_mask); - if (likely(tb == NULL)) { - /* if no translated code available, then translate it now */ - tb = tb_gen_code(cpu, pc, cs_base, flags, cf_mask); - } + tb = tb_gen_code(cpu, pc, cs_base, flags, cf_mask); mmap_unlock(); /* We add the TB in the virtual pc hash table for the fast lookup */ diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index c75298d08a..3f977532bf 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -1581,18 +1581,30 @@ static inline void tb_page_add(PageDesc *p, TranslationBlock *tb, * (-1) to indicate that only one page contains the TB. * * Called with mmap_lock held for user-mode emulation. + * + * Returns a pointer @tb, or a pointer to an existing TB that matches @tb. + * Note that in !user-mode, another thread might have already added a TB + * for the same block of guest code that @tb corresponds to. In that case, + * the caller should discard the original @tb, and use instead the returned TB. */ -static void tb_link_page(TranslationBlock *tb, tb_page_addr_t phys_pc, - tb_page_addr_t phys_page2) +static TranslationBlock * +tb_link_page(TranslationBlock *tb, tb_page_addr_t phys_pc, + tb_page_addr_t phys_page2) { PageDesc *p; PageDesc *p2 = NULL; + void *existing_tb = NULL; uint32_t h; assert_memory_lock(); /* * Add the TB to the page list, acquiring first the pages's locks. + * We keep the locks held until after inserting the TB in the hash table, + * so that if the insertion fails we know for sure that the TBs are still + * in the page descriptors. + * Note that inserting into the hash table first isn't an option, since + * we can only insert TBs that are fully initialized. */ page_lock_pair(&p, phys_pc, &p2, phys_page2, 1); tb_page_add(p, tb, 0, phys_pc & TARGET_PAGE_MASK); @@ -1602,21 +1614,33 @@ static void tb_link_page(TranslationBlock *tb, tb_page_addr_t phys_pc, tb->page_addr[1] = -1; } + /* add in the hash table */ + h = tb_hash_func(phys_pc, tb->pc, tb->flags, tb->cflags & CF_HASH_MASK, + tb->trace_vcpu_dstate); + qht_insert(&tb_ctx.htable, tb, h, &existing_tb); + + /* remove TB from the page(s) if we couldn't insert it */ + if (unlikely(existing_tb)) { + tb_page_remove(p, tb); + invalidate_page_bitmap(p); + if (p2) { + tb_page_remove(p2, tb); + invalidate_page_bitmap(p2); + } + tb = existing_tb; + } + if (p2) { page_unlock(p2); } page_unlock(p); - /* add in the hash table */ - h = tb_hash_func(phys_pc, tb->pc, tb->flags, tb->cflags & CF_HASH_MASK, - tb->trace_vcpu_dstate); - qht_insert(&tb_ctx.htable, tb, h, NULL); - #ifdef CONFIG_USER_ONLY if (DEBUG_TB_CHECK_GATE) { tb_page_check(); } #endif + return tb; } /* Called with mmap_lock held for user mode emulation. */ @@ -1625,7 +1649,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu, uint32_t flags, int cflags) { CPUArchState *env = cpu->env_ptr; - TranslationBlock *tb; + TranslationBlock *tb, *existing_tb; tb_page_addr_t phys_pc, phys_page2; target_ulong virt_page2; tcg_insn_unit *gen_code_buf; @@ -1773,7 +1797,15 @@ TranslationBlock *tb_gen_code(CPUState *cpu, * memory barrier is required before tb_link_page() makes the TB visible * through the physical hash table and physical page list. */ - tb_link_page(tb, phys_pc, phys_page2); + existing_tb = tb_link_page(tb, phys_pc, phys_page2); + /* if the TB already exists, discard what we just translated */ + if (unlikely(existing_tb != tb)) { + uintptr_t orig_aligned = (uintptr_t)gen_code_buf; + + orig_aligned -= ROUND_UP(sizeof(*tb), qemu_icache_linesize); + atomic_set(&tcg_ctx->code_gen_ptr, (void *)orig_aligned); + return existing_tb; + } tcg_tb_insert(tb); return tb; } diff --git a/docs/devel/multi-thread-tcg.txt b/docs/devel/multi-thread-tcg.txt index faf8918b23..faf09c6069 100644 --- a/docs/devel/multi-thread-tcg.txt +++ b/docs/devel/multi-thread-tcg.txt @@ -140,6 +140,9 @@ to atomically insert new elements. The lookup caches are updated atomically and the lookup hash uses QHT which is designed for concurrent safe lookup. +Parallel code generation is supported. QHT is used at insertion time +as the synchronization point across threads, thereby ensuring that we only +keep track of a single TranslationBlock for each guest code block. Memory maps and TLBs -------------------- From patchwork Fri Jun 15 19:43:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138761 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1251226lji; Fri, 15 Jun 2018 12:52:17 -0700 (PDT) X-Google-Smtp-Source: ADUXVKLn8f0qdeRVsKfwuZQPaBKs2Aj73xT0TxMrbj/3gVSg+Y0L2EE+I6z2X73IXIvEw3ztO7Kl X-Received: by 2002:a37:a18d:: with SMTP id k135-v6mr2524396qke.33.1529092337469; Fri, 15 Jun 2018 12:52:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092337; cv=none; d=google.com; s=arc-20160816; b=GibqRFjaFjZxyZXFoVbMgE7ZpO2Vq4Gp0FDMCdno0QHhzeDq4dy8gyAvgBQE/Z99TF smuYXhtO3YfQAf74pEixOE4YNmaadsu0bFtsUTL/IZFE7fiNTsQOF+MK5ZWWLo2YDDSU brS3U3DGsmVC5eQkF05jtDeUWt7HNet802Pnqx6IUo9YfVsgOzLpii4ZPyuxyySrh6BO GJfkCaYywFcZnWkcis0YHj3e5HQEonkwKmqEg+aQ06ngt3cqqdIwsni4TGKcCzNIo6X3 MgT3lVznK1IRMq0ayuJG2xHAEXUVDRPYXWcenDtYkWqG1yUIQuh7pGpYBhJoq9USsKtw KcCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=EqSrgo0lsMrQ4+eaMrxENRtzhrS4M8dalPCYAmoFtpU=; b=E7ro96jOpb0YWvWZMhGvAKNG/w/J2az11XENFYqOn5lz/Y4sOzq2YMZC3YcvgQJV0U lndrOlUDbnHkoMu4zLRbq8+b0c+g3Xz8hLRRgPEZHWqxVjj/HkJkrk46Ro1HvD6pUrzw QJy2Vsdz2Eci9i7v53rQDKajyRzSzzl61zeAcU6U4qFRPXkthvkxwAzHzeODl41SSWuP wCIoMDuuSNn4VJxMpW8B+UCp7Ea7QjHbzKJXAx3pLWilwVgOeKI+YOMv+ear3oDQV9sE Izrb2DxAt/5OdyniMKeyktY3UnUDHu1raE9DCkqtoPKTXr9oCq2I9snitZaogjSqTISv o0aA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=NuT1IMgQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id u11-v6si8881204qvg.40.2018.06.15.12.52.17 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:52:17 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=NuT1IMgQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:48994 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTulc-0006AN-Sx for patch@linaro.org; Fri, 15 Jun 2018 15:52:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51681) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTue7-0000Zi-Un for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTue5-0002fg-HG for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:31 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:46080) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTue5-0002ew-7R for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:29 -0400 Received: by mail-pl0-x241.google.com with SMTP id 30-v6so5845036pld.13 for ; Fri, 15 Jun 2018 12:44:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EqSrgo0lsMrQ4+eaMrxENRtzhrS4M8dalPCYAmoFtpU=; b=NuT1IMgQj2LrStszKyWHwN1ZL6VfL//BkhLlehcA7Zqld2Sxcm7Ty5MBzZWhgQAMJK OH8sAJPICDRMeuScQOE40GcLKyokF83mMlb9wHPsZ/bgdWetsh9HEiU+jy+HB4ypv7Mf hqVwN1kojcJODio8MOTAdEAeMTAbOQ5oS/Gvw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EqSrgo0lsMrQ4+eaMrxENRtzhrS4M8dalPCYAmoFtpU=; b=LkVqKzc+9Rszte9sfrRysNG+IXaa0cWCYYrZko2k9hL9cTLOxUa8EyQh4QwPFrrIUv r4SeEXe7qhT9EUmPxZGCgCLtsgK3VeJ/M9waiTBhLIpjxSL2fajCla/T1ofZsT2jXZF4 RiVgi/AGBqMDJPMMjte6ZtqAxGf62PD5ap10XTjYetWCQ5D7j+8CfttI3qZ8gieHEq8R q60QLV1vHxHgi9OPRyZDY2E+/evH5jVoiH4qURpmhUig/02QBWnfl6mWrUTNlJ/Oyssx SDOj61erqJGactHCFlaBl5+DDOby1PT7CAUQYggmvPvKCUZUzr3PvLJQkHou6PGG7tTP v10w== X-Gm-Message-State: APt69E0m9DlIalnA8LnZdmIOpK3oFRv8sNBQPfcXDF5Fkthy6d3xgpRH SQYJllC6u4FQYcVrVLcI8GGhcRnQhZg= X-Received: by 2002:a17:902:714e:: with SMTP id u14-v6mr3528228plm.289.1529091867851; Fri, 15 Jun 2018 12:44:27 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.26 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:26 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:50 -1000 Message-Id: <20180615194354.12489-16-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PULL v2 15/19] translate-all: protect TB jumps with a per-destination-TB lock X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" This applies to both user-mode and !user-mode emulation. Instead of relying on a global lock, protect the list of incoming jumps with tb->jmp_lock. This lock also protects tb->cflags, so update all tb->cflags readers outside tb->jmp_lock to use atomic reads via tb_cflags(). In order to find the destination TB (and therefore its jmp_lock) from the origin TB, we introduce tb->jmp_dest[]. I considered not using a linked list of jumps, which simplifies code and makes the struct smaller. However, it unnecessarily increases memory usage, which results in a performance decrease. See for instance these numbers booting+shutting down debian-arm: Time (s) Rel. err (%) Abs. err (s) Rel. slowdown (%) ------------------------------------------------------------------------------ before 20.88 0.74 0.154512 0. after 20.81 0.38 0.079078 -0.33524904 GTree 21.02 0.28 0.058856 0.67049808 GHashTable + xxhash 21.63 1.08 0.233604 3.5919540 Using a hash table or a binary tree to keep track of the jumps doesn't really pay off, not only due to the increased memory usage, but also because most TBs have only 0 or 1 jumps to them. The maximum number of jumps when booting debian-arm that I measured is 35, but as we can see in the histogram below a TB with that many incoming jumps is extremely rare; the average TB has 0.80 incoming jumps. n_jumps: 379208; avg jumps/tb: 0.801099 dist: [0.0,1.0)|▄█▁▁▁▁▁▁▁▁▁▁▁ ▁▁▁▁▁▁ ▁▁▁ ▁▁▁ ▁|[34.0,35.0] Reviewed-by: Richard Henderson Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- include/exec/exec-all.h | 35 ++++++---- accel/tcg/cpu-exec.c | 41 +++++++---- accel/tcg/translate-all.c | 120 +++++++++++++++++++------------- docs/devel/multi-thread-tcg.txt | 6 +- 4 files changed, 125 insertions(+), 77 deletions(-) -- 2.17.1 diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 4f07a17052..3c2a0efb55 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -345,7 +345,7 @@ struct TranslationBlock { #define CF_LAST_IO 0x00008000 /* Last insn may be an IO access. */ #define CF_NOCACHE 0x00010000 /* To be freed after execution */ #define CF_USE_ICOUNT 0x00020000 -#define CF_INVALID 0x00040000 /* TB is stale. Setters need tb_lock */ +#define CF_INVALID 0x00040000 /* TB is stale. Set with @jmp_lock held */ #define CF_PARALLEL 0x00080000 /* Generate code for a parallel context */ /* cflags' mask for hashing/comparison */ #define CF_HASH_MASK \ @@ -364,6 +364,9 @@ struct TranslationBlock { uintptr_t page_next[2]; tb_page_addr_t page_addr[2]; + /* jmp_lock placed here to fill a 4-byte hole. Its documentation is below */ + QemuSpin jmp_lock; + /* The following data are used to directly call another TB from * the code of this one. This can be done either by emitting direct or * indirect native jump instructions. These jumps are reset so that the TB @@ -375,20 +378,26 @@ struct TranslationBlock { #define TB_JMP_RESET_OFFSET_INVALID 0xffff /* indicates no jump generated */ uintptr_t jmp_target_arg[2]; /* target address or offset */ - /* Each TB has an associated circular list of TBs jumping to this one. - * jmp_list_first points to the first TB jumping to this one. - * jmp_list_next is used to point to the next TB in a list. - * Since each TB can have two jumps, it can participate in two lists. - * jmp_list_first and jmp_list_next are 4-byte aligned pointers to a - * TranslationBlock structure, but the two least significant bits of - * them are used to encode which data field of the pointed TB should - * be used to traverse the list further from that TB: - * 0 => jmp_list_next[0], 1 => jmp_list_next[1], 2 => jmp_list_first. - * In other words, 0/1 tells which jump is used in the pointed TB, - * and 2 means that this is a pointer back to the target TB of this list. + /* + * Each TB has a NULL-terminated list (jmp_list_head) of incoming jumps. + * Each TB can have two outgoing jumps, and therefore can participate + * in two lists. The list entries are kept in jmp_list_next[2]. The least + * significant bit (LSB) of the pointers in these lists is used to encode + * which of the two list entries is to be used in the pointed TB. + * + * List traversals are protected by jmp_lock. The destination TB of each + * outgoing jump is kept in jmp_dest[] so that the appropriate jmp_lock + * can be acquired from any origin TB. + * + * jmp_dest[] are tagged pointers as well. The LSB is set when the TB is + * being invalidated, so that no further outgoing jumps from it can be set. + * + * jmp_lock also protects the CF_INVALID cflag; a jump must not be chained + * to a destination TB that has CF_INVALID set. */ + uintptr_t jmp_list_head; uintptr_t jmp_list_next[2]; - uintptr_t jmp_list_first; + uintptr_t jmp_dest[2]; }; extern bool parallel_cpus; diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c index 45f6ebc65e..c482008bc7 100644 --- a/accel/tcg/cpu-exec.c +++ b/accel/tcg/cpu-exec.c @@ -352,28 +352,43 @@ void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr) } } -/* Called with tb_lock held. */ static inline void tb_add_jump(TranslationBlock *tb, int n, TranslationBlock *tb_next) { + uintptr_t old; + assert(n < ARRAY_SIZE(tb->jmp_list_next)); - if (tb->jmp_list_next[n]) { - /* Another thread has already done this while we were - * outside of the lock; nothing to do in this case */ - return; + qemu_spin_lock(&tb_next->jmp_lock); + + /* make sure the destination TB is valid */ + if (tb_next->cflags & CF_INVALID) { + goto out_unlock_next; } + /* Atomically claim the jump destination slot only if it was NULL */ + old = atomic_cmpxchg(&tb->jmp_dest[n], (uintptr_t)NULL, (uintptr_t)tb_next); + if (old) { + goto out_unlock_next; + } + + /* patch the native jump address */ + tb_set_jmp_target(tb, n, (uintptr_t)tb_next->tc.ptr); + + /* add in TB jmp list */ + tb->jmp_list_next[n] = tb_next->jmp_list_head; + tb_next->jmp_list_head = (uintptr_t)tb | n; + + qemu_spin_unlock(&tb_next->jmp_lock); + qemu_log_mask_and_addr(CPU_LOG_EXEC, tb->pc, "Linking TBs %p [" TARGET_FMT_lx "] index %d -> %p [" TARGET_FMT_lx "]\n", tb->tc.ptr, tb->pc, n, tb_next->tc.ptr, tb_next->pc); + return; - /* patch the native jump address */ - tb_set_jmp_target(tb, n, (uintptr_t)tb_next->tc.ptr); - - /* add in TB jmp circular list */ - tb->jmp_list_next[n] = tb_next->jmp_list_first; - tb_next->jmp_list_first = (uintptr_t)tb | n; + out_unlock_next: + qemu_spin_unlock(&tb_next->jmp_lock); + return; } static inline TranslationBlock *tb_find(CPUState *cpu, @@ -416,9 +431,7 @@ static inline TranslationBlock *tb_find(CPUState *cpu, tb_lock(); acquired_tb_lock = true; } - if (!(tb->cflags & CF_INVALID)) { - tb_add_jump(last_tb, tb_exit, tb); - } + tb_add_jump(last_tb, tb_exit, tb); } if (acquired_tb_lock) { tb_unlock(); diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 3f977532bf..55c9e1b196 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -170,6 +170,9 @@ struct page_collection { #define PAGE_FOR_EACH_TB(pagedesc, tb, n) \ TB_FOR_EACH_TAGGED((pagedesc)->first_tb, tb, n, page_next) +#define TB_FOR_EACH_JMP(head_tb, tb, n) \ + TB_FOR_EACH_TAGGED((head_tb)->jmp_list_head, tb, n, jmp_list_next) + /* In system mode we want L1_MAP to be based on ram offsets, while in user mode we want it to be based on virtual addresses. */ #if !defined(CONFIG_USER_ONLY) @@ -389,7 +392,7 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb, return -1; found: - if (reset_icount && (tb->cflags & CF_USE_ICOUNT)) { + if (reset_icount && (tb_cflags(tb) & CF_USE_ICOUNT)) { assert(use_icount); /* Reset the cycle counter to the start of the block and shift if to the number of actually executed instructions */ @@ -432,7 +435,7 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t host_pc, bool will_exit) tb = tcg_tb_lookup(host_pc); if (tb) { cpu_restore_state_from_tb(cpu, tb, host_pc, will_exit); - if (tb->cflags & CF_NOCACHE) { + if (tb_cflags(tb) & CF_NOCACHE) { /* one-shot translation, invalidate it immediately */ tb_phys_invalidate(tb, -1); tcg_tb_remove(tb); @@ -1360,34 +1363,53 @@ static inline void tb_page_remove(PageDesc *pd, TranslationBlock *tb) g_assert_not_reached(); } -/* remove the TB from a list of TBs jumping to the n-th jump target of the TB */ -static inline void tb_remove_from_jmp_list(TranslationBlock *tb, int n) +/* remove @orig from its @n_orig-th jump list */ +static inline void tb_remove_from_jmp_list(TranslationBlock *orig, int n_orig) { - TranslationBlock *tb1; - uintptr_t *ptb, ntb; - unsigned int n1; + uintptr_t ptr, ptr_locked; + TranslationBlock *dest; + TranslationBlock *tb; + uintptr_t *pprev; + int n; - ptb = &tb->jmp_list_next[n]; - if (*ptb) { - /* find tb(n) in circular list */ - for (;;) { - ntb = *ptb; - n1 = ntb & 3; - tb1 = (TranslationBlock *)(ntb & ~3); - if (n1 == n && tb1 == tb) { - break; - } - if (n1 == 2) { - ptb = &tb1->jmp_list_first; - } else { - ptb = &tb1->jmp_list_next[n1]; - } - } - /* now we can suppress tb(n) from the list */ - *ptb = tb->jmp_list_next[n]; - - tb->jmp_list_next[n] = (uintptr_t)NULL; + /* mark the LSB of jmp_dest[] so that no further jumps can be inserted */ + ptr = atomic_or_fetch(&orig->jmp_dest[n_orig], 1); + dest = (TranslationBlock *)(ptr & ~1); + if (dest == NULL) { + return; } + + qemu_spin_lock(&dest->jmp_lock); + /* + * While acquiring the lock, the jump might have been removed if the + * destination TB was invalidated; check again. + */ + ptr_locked = atomic_read(&orig->jmp_dest[n_orig]); + if (ptr_locked != ptr) { + qemu_spin_unlock(&dest->jmp_lock); + /* + * The only possibility is that the jump was unlinked via + * tb_jump_unlink(dest). Seeing here another destination would be a bug, + * because we set the LSB above. + */ + g_assert(ptr_locked == 1 && dest->cflags & CF_INVALID); + return; + } + /* + * We first acquired the lock, and since the destination pointer matches, + * we know for sure that @orig is in the jmp list. + */ + pprev = &dest->jmp_list_head; + TB_FOR_EACH_JMP(dest, tb, n) { + if (tb == orig && n == n_orig) { + *pprev = tb->jmp_list_next[n]; + /* no need to set orig->jmp_dest[n]; setting the LSB was enough */ + qemu_spin_unlock(&dest->jmp_lock); + return; + } + pprev = &tb->jmp_list_next[n]; + } + g_assert_not_reached(); } /* reset the jump entry 'n' of a TB so that it is not chained to @@ -1399,24 +1421,21 @@ static inline void tb_reset_jump(TranslationBlock *tb, int n) } /* remove any jumps to the TB */ -static inline void tb_jmp_unlink(TranslationBlock *tb) +static inline void tb_jmp_unlink(TranslationBlock *dest) { - TranslationBlock *tb1; - uintptr_t *ptb, ntb; - unsigned int n1; + TranslationBlock *tb; + int n; - ptb = &tb->jmp_list_first; - for (;;) { - ntb = *ptb; - n1 = ntb & 3; - tb1 = (TranslationBlock *)(ntb & ~3); - if (n1 == 2) { - break; - } - tb_reset_jump(tb1, n1); - *ptb = tb1->jmp_list_next[n1]; - tb1->jmp_list_next[n1] = (uintptr_t)NULL; + qemu_spin_lock(&dest->jmp_lock); + + TB_FOR_EACH_JMP(dest, tb, n) { + tb_reset_jump(tb, n); + atomic_and(&tb->jmp_dest[n], (uintptr_t)NULL | 1); + /* No need to clear the list entry; setting the dest ptr is enough */ } + dest->jmp_list_head = (uintptr_t)NULL; + + qemu_spin_unlock(&dest->jmp_lock); } /* If @rm_from_page_list is set, call with the TB's pages' locks held */ @@ -1429,11 +1448,14 @@ static void do_tb_phys_invalidate(TranslationBlock *tb, bool rm_from_page_list) assert_tb_locked(); + /* make sure no further incoming jumps will be chained to this TB */ + qemu_spin_lock(&tb->jmp_lock); atomic_set(&tb->cflags, tb->cflags | CF_INVALID); + qemu_spin_unlock(&tb->jmp_lock); /* remove the TB from the hash list */ phys_pc = tb->page_addr[0] + (tb->pc & ~TARGET_PAGE_MASK); - h = tb_hash_func(phys_pc, tb->pc, tb->flags, tb->cflags & CF_HASH_MASK, + h = tb_hash_func(phys_pc, tb->pc, tb->flags, tb_cflags(tb) & CF_HASH_MASK, tb->trace_vcpu_dstate); if (!qht_remove(&tb_ctx.htable, tb, h)) { return; @@ -1773,10 +1795,12 @@ TranslationBlock *tb_gen_code(CPUState *cpu, CODE_GEN_ALIGN)); /* init jump list */ - assert(((uintptr_t)tb & 3) == 0); - tb->jmp_list_first = (uintptr_t)tb | 2; + qemu_spin_init(&tb->jmp_lock); + tb->jmp_list_head = (uintptr_t)NULL; tb->jmp_list_next[0] = (uintptr_t)NULL; tb->jmp_list_next[1] = (uintptr_t)NULL; + tb->jmp_dest[0] = (uintptr_t)NULL; + tb->jmp_dest[1] = (uintptr_t)NULL; /* init original jump addresses wich has been set during tcg_gen_code() */ if (tb->jmp_reset_offset[0] != TB_JMP_RESET_OFFSET_INVALID) { @@ -1868,7 +1892,7 @@ tb_invalidate_phys_page_range__locked(struct page_collection *pages, } } if (current_tb == tb && - (current_tb->cflags & CF_COUNT_MASK) != 1) { + (tb_cflags(current_tb) & CF_COUNT_MASK) != 1) { /* If we are modifying the current TB, we must stop its execution. We could be more precise by checking that the modification is after the current PC, but it @@ -2067,7 +2091,7 @@ static bool tb_invalidate_phys_page(tb_page_addr_t addr, uintptr_t pc) PAGE_FOR_EACH_TB(p, tb, n) { #ifdef TARGET_HAS_PRECISE_SMC if (current_tb == tb && - (current_tb->cflags & CF_COUNT_MASK) != 1) { + (tb_cflags(current_tb) & CF_COUNT_MASK) != 1) { /* If we are modifying the current TB, we must stop its execution. We could be more precise by checking that the modification is after the current PC, but it @@ -2192,7 +2216,7 @@ void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr) /* Generate a new TB executing the I/O insn. */ cpu->cflags_next_tb = curr_cflags() | CF_LAST_IO | n; - if (tb->cflags & CF_NOCACHE) { + if (tb_cflags(tb) & CF_NOCACHE) { if (tb->orig_tb) { /* Invalidate original TB if this TB was generated in * cpu_exec_nocache() */ diff --git a/docs/devel/multi-thread-tcg.txt b/docs/devel/multi-thread-tcg.txt index faf09c6069..df83445ccf 100644 --- a/docs/devel/multi-thread-tcg.txt +++ b/docs/devel/multi-thread-tcg.txt @@ -131,8 +131,10 @@ DESIGN REQUIREMENT: Safely handle invalidation of TBs The direct jump themselves are updated atomically by the TCG tb_set_jmp_target() code. Modification to the linked lists that allow -searching for linked pages are done under the protect of the -tb_lock(). +searching for linked pages are done under the protection of tb->jmp_lock, +where tb is the destination block of a jump. Each origin block keeps a +pointer to its destinations so that the appropriate lock can be acquired before +iterating over a jump list. The global page table is a lockless radix tree; cmpxchg is used to atomically insert new elements. From patchwork Fri Jun 15 19:43:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138783 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1269404lji; Fri, 15 Jun 2018 13:11:31 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJQa1lAdS3K4i8iiQEjmEFwliaZphbPoeJorI/62qrUR+z2oYzDquVMRHoXN1Z+K2VL0R3S X-Received: by 2002:a37:16e0:: with SMTP id 93-v6mr2780109qkw.148.1529093491438; Fri, 15 Jun 2018 13:11:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529093491; cv=none; d=google.com; s=arc-20160816; b=1Jz+Ijn36EXGuhXr15yMhwvz+/fFu7//2d5jphjjiwoelYk3iD8Mz8grHBFWmKL1sf Jw1qIOxA8OKoHdekAWiB15Q+ZdimE1WzBse/w5A3QpkFaRXrOTN+vji0dt8prcK3Cfnm pwAZ4nOAG2Fp0l2zg/Z10JGMSS27xDU+OfEEhHxgYD85Th2+hWFYoBnGdFpX+tmNKL3J 08nBdo15WIEE+e42q2DzhUL4ze9006PwoGtVqVwrvTKo878DLCPS2a6a+LRmQq5G1dcb HW3I+0jKJQCXKrqUgFyUMsggGGJCNyYWJnK8SlqVTJ2E+h0DGTmjdL8Owm7O7G0YuNY8 /wRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=KuNXtalN93hxfYtS8iiaj+taOz8s1bCsuF+GaVINmmo=; b=a3eba4o6LZNU2H0GLJMnbu3avyQAZk0/1agA1tER9f0lo4vnrpn3mVNUj/3J8cNsVg PMJ8xwdot28heIgNCv++eJvzqtZlGj3yXEO2JdUUzVpjzhS6Dxawc3GWcmyt0nmQFBFj hAsSBNrfiawrVmnj2zCBhXCjtzRBDehpvo63pByFzICIy3IISAQ48+CzFdl0/XWk/urZ 3A9bkrKOtEQKqIeVRnSQzpeo/5WNXTGRMWJO5BcSzSJLzOBi+zU/6Skst+nEPUyH+pPv RLd9kHmm3+oAec2HJCKCpUSVIR4ASu/7/NZS3SYcC7AMuKml9kZkNHphMIQCfb8bit+A OGlw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=N5PWQ6t4; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id k1-v6si4809006qtk.400.2018.06.15.13.11.31 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 13:11:31 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=N5PWQ6t4; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49103 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTv4E-0005Yx-Sx for patch@linaro.org; Fri, 15 Jun 2018 16:11:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51683) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTue8-0000Zo-1M for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTue7-0002gn-4a for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:32 -0400 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:37929) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTue6-0002gH-UT for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:31 -0400 Received: by mail-pg0-x241.google.com with SMTP id c9-v6so4855085pgf.5 for ; Fri, 15 Jun 2018 12:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KuNXtalN93hxfYtS8iiaj+taOz8s1bCsuF+GaVINmmo=; b=N5PWQ6t4IrIfOrsuqEOmgnmm7U9Wn/eSq5VbBm9NC3vD/0NcezLllubFwMRuFKueGk KKENHTTL1OvdqnEZcDLnVe0GBy3ORR28BfGva+fdSaDRRWjGpasJ/WLFBLaOyzVwEa66 JBtti1hvI6ZEEx4CXHdqwc2YAzHoEbJqRF7Qw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KuNXtalN93hxfYtS8iiaj+taOz8s1bCsuF+GaVINmmo=; b=lEiEJ/S4MsqSX1UrDvweaknz2DqV7Djo1jVI1ObxoB2/nnQQPYaUwXXCxDvovhCU8K NZK+EvWm8YQxvAH8gMlD5FTXO3j+4eaPY0jHRE1QugCa8Dz3NryZzLaTC0tFjVw/PXLY KhsacGZI3qIpAlX/EWKmm5y6szt5Mq1zjzcbGo8iXgzkdP3JPOifj6WQv4zfVmi4Y8CZ RmDF97a12uKb4HiHQucn5/Lfw5jEiKOmSlMpP9OUmNvisiMQdJxXxL9VBuK0GUm9hRTS 8TgKpvO25H1rLPyeSt0XovzSRk7EPaMBFFH86z6NEqQIJ7+PmQa/g7DdwQjDA/fkWnE8 lMYQ== X-Gm-Message-State: APt69E0LM+jdADp4kj8zKPfMRa+ieOn21Z3iBhgnF6Lg2SfTv4Q/ugfi w5aCXLy2kz46qTtc4isxHhAyexVSB4U= X-Received: by 2002:a62:b90f:: with SMTP id z15-v6mr3413246pfe.254.1529091869682; Fri, 15 Jun 2018 12:44:29 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.28 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:28 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:51 -1000 Message-Id: <20180615194354.12489-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PULL v2 16/19] cputlb: remove tb_lock from tlb_flush functions X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" The acquisition of tb_lock was added when the async tlb_flush was introduced in e3b9ca810 ("cputlb: introduce tlb_flush_* async work.") tb_lock was there to allow us to do memset() on the tb_jmp_cache's. However, since f3ced3c5928 ("tcg: consistently access cpu->tb_jmp_cache atomically") all accesses to tb_jmp_cache are atomic, so tb_lock is not needed here. Get rid of it. Reviewed-by: Alex Bennée Reviewed-by: Richard Henderson Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/cputlb.c | 8 -------- 1 file changed, 8 deletions(-) -- 2.17.1 diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 0a721bb9c4..719cca2268 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -125,8 +125,6 @@ static void tlb_flush_nocheck(CPUState *cpu) atomic_set(&env->tlb_flush_count, env->tlb_flush_count + 1); tlb_debug("(count: %zu)\n", tlb_flush_count()); - tb_lock(); - memset(env->tlb_table, -1, sizeof(env->tlb_table)); memset(env->tlb_v_table, -1, sizeof(env->tlb_v_table)); cpu_tb_jmp_cache_clear(cpu); @@ -135,8 +133,6 @@ static void tlb_flush_nocheck(CPUState *cpu) env->tlb_flush_addr = -1; env->tlb_flush_mask = 0; - tb_unlock(); - atomic_mb_set(&cpu->pending_tlb_flush, 0); } @@ -180,8 +176,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data) assert_cpu_is_self(cpu); - tb_lock(); - tlb_debug("start: mmu_idx:0x%04lx\n", mmu_idx_bitmask); for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { @@ -197,8 +191,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data) cpu_tb_jmp_cache_clear(cpu); tlb_debug("done\n"); - - tb_unlock(); } void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) From patchwork Fri Jun 15 19:43:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138762 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1252508lji; Fri, 15 Jun 2018 12:53:48 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKn2Ji7MljrhejTmJ1FjlNO45IGuRtCElIaHt11IejaatETMtes7fZlKI+0FrbyfCZuGmzS X-Received: by 2002:a0c:9621:: with SMTP id 30-v6mr2661029qvx.127.1529092428112; Fri, 15 Jun 2018 12:53:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092428; cv=none; d=google.com; s=arc-20160816; b=H1dv3VhvMcKB7nIJMShLJd3xQvJzBDKOI+lG+VrKFecOE8/8Iai8akmFebMn4z1KIH yt5Fbkuf/DgQeZpWJTgv0vfDlT+vobWCv8KMgQKQMRHZeg5esORgJ0MWmPaNgjC80bCz qOjZmyJ1kbk5q/yzZ8843KS0UfKofl4LavS1n4iBSBWLrHD9tCUt8BrXmyqmQhnWcu5B oEXjC/PxBirU0WLGw31hWNm0NqMt35KSjNE9GcG/h/IW1Qo2iizieHnq2DPgIfY2SrLg 8WBuZX2nBcKqrmOJCSq3cxd5CUOpDY2pL2p4nJoLkHD95L07dNlRnkZC712IHTzLIyjz a8gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=90PvNAHeckXJjwcM439RbSqujLTH8kYi9wIh5hWAJn8=; b=TjpW3QRMBJ7U9Sd0ecIrBm43zg+T737qynNLgriDzuOzVTQLSmGXndLpUnpSRxH+rC qbNurHniCOGlRwupZbg2cgKZv1QORRbjPDbxFBE1KqfMutaM6eLv5our8G2cpOhAw9gm We+SjWei74CmKG713j7lqSGUtl+YN42whiTYxaLVIakdSi5HW+6/slbMMl8i+ofkEpWJ qxQtnuv1BzZGJa0aJjAyDZ6YJDs4X1vYpxXSuBmbDyqReAKCUCsLJUJA6qiL72zrHwxc Zi0PjkIow05aVWoISFWee6mVi0YL1m6JpFt4xnG++trR9WmtmYwxU3p+y14txGyRYkFH iEJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=QmsbIQbj; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id b64-v6si7486888qkb.260.2018.06.15.12.53.47 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:53:48 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=QmsbIQbj; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49002 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTun5-0007ey-Gb for patch@linaro.org; Fri, 15 Jun 2018 15:53:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51695) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTue9-0000bM-K9 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTue8-0002i8-UE for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:33 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:38076) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTue8-0002hZ-Oz for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:32 -0400 Received: by mail-pl0-x242.google.com with SMTP id b14-v6so5849376pls.5 for ; Fri, 15 Jun 2018 12:44:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=90PvNAHeckXJjwcM439RbSqujLTH8kYi9wIh5hWAJn8=; b=QmsbIQbjbk/wqhoNbFlDn16Uzq+BPGSFMAJzLpGk3u/a27Cw8zJOgRskvldVbqFVp2 v+fFxMyHPAQgg2/vpvtXzX64sT6OrIWwtJKp7yxPhcN+bEkRCwhmTY6x99EEmuRWnWIV CwFa1afnlbmH/YOMcDUrmycSVAL+uztn4zX3w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=90PvNAHeckXJjwcM439RbSqujLTH8kYi9wIh5hWAJn8=; b=ICYrUjNGF6XsKvJEXwWr8NtbhjKGoCUyVyxh2p27IsK5e2MwIQc5Dv0rya29wGw1Ej J40Yrp0QoUAqmX94YWGlpSluBTr3qjTy+GpbI4jBzLu3Cv13CW5EjXr4eQDWtJaacBz+ SzckKJSbMShpBFjwYdnTaPcX+/cG+is8wMWwWi95MHG3xhZRAgnXsqQ9UwFohzdaMHBY ApaC3ZKn4TJczRKghq9ImOMmGlXzFX+YkawvWU30qP+WuEqKjX38FVZmFBmTxzFhMy09 VJJog3Kh6qtRBUU49HwiKJhxPXJkMkkVLfT/otepRCdgrJEMl7+jRrIMpsbmn3dMEhIA TEKw== X-Gm-Message-State: APt69E1mG+c3ufhnGYN8IMJOG4jL60tc166e7O178H3ZAscnETqKGb7P gKkYIv6gTgpVDVbwdYj5XSIEOEvEW2E= X-Received: by 2002:a17:902:aa95:: with SMTP id d21-v6mr3507996plr.73.1529091871507; Fri, 15 Jun 2018 12:44:31 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.29 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:30 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:52 -1000 Message-Id: <20180615194354.12489-18-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PULL v2 17/19] translate-all: remove tb_lock mention from cpu_restore_state_from_tb X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" tb_lock was needed when the function did retranslation. However, since fca8a500d519 ("tcg: Save insn data and use it in cpu_restore_state_from_tb") we don't do retranslation. Get rid of the comment. Reviewed-by: Richard Henderson Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/translate-all.c | 1 - 1 file changed, 1 deletion(-) -- 2.17.1 diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 55c9e1b196..b33cd9bc98 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -355,7 +355,6 @@ static int encode_search(TranslationBlock *tb, uint8_t *block) } /* The cpu state corresponding to 'searched_pc' is restored. - * Called with tb_lock held. * When reset_icount is true, current TB will be interrupted and * icount should be recalculated. */ From patchwork Fri Jun 15 19:43:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138765 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1253717lji; Fri, 15 Jun 2018 12:55:13 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJOV9aLDuPE+iKkvARqGk1vjDqDpyqzL0+1FUzA0NNUfdDzXyxchmssMz6PAeDW30DiR/RU X-Received: by 2002:aed:2512:: with SMTP id v18-v6mr2734611qtc.396.1529092513150; Fri, 15 Jun 2018 12:55:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092513; cv=none; d=google.com; s=arc-20160816; b=juyFpr1TOrM/crcrSTuKL48PXw7TPQxs5CQh4EKOMGqb6Cd7EloeBiEDBLebLF8aB7 nhecl0Lqdv4tNtjuExgq4UZ8AxdmqpF2FRb5C410tllEBHko2n3HtyqL1v6taPvScmZP puU/ulqog6pwG2bCLtSFnzg6estXAf/3tyOkH47T2Z6JW6E0NalxCmc7USRYihwXvg3X 3Fp4xI37UTb2ovigQVkD4iBHFlyTtDznC6iO8/a9ma+mAhwDsHvB0LKGDCtOgLnHVCsL fufDI+R+8zXXYkSfyjwwrtT1SAKG1gnem8A9ZIPE8jJBHni8DVRsk7YCTAZMGvDhzZwh rv7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=HLQgwyJfm0Wvm9CGkm1dh++7JQGvFnBJ5/SVt3YVwcg=; b=ZtFmHj7I1LC70bOND77zt8EbXflBzLY63QNh5vxWjzZNx5qOrgSrUip0Psv40d6mZ6 ndO3NpdJPzNZ1Eh1A0Lu7lJq//ipCZa9Ynqs9a7PWztvZTBxW3PwJB5yT5Z+LTYHUAiS /U0P1Xe7ZGJx0BNAYmqTOftHztu+WJcAJuUiSw6pkkhBq9jmBKBe+KCgJjSqNLiIKnlr Tj396wuR1vSYGWh1SsZPlW0Wt5pMGPhWnwziCnDluTTRm5pUzZCfxcxatbR4LG8wPK8B K6Cbhyy4cL/08AWgbUe8EdEpGLxbGBgMwr3Dpr3acIg/ZGcCZoeAvjwp/l6Hoclomrb4 DaAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=CMfHTRsp; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id l35-v6si6116124qve.23.2018.06.15.12.55.12 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:55:13 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=CMfHTRsp; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49008 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuoS-00007l-I0 for patch@linaro.org; Fri, 15 Jun 2018 15:55:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51727) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTueK-0000i9-6R for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTueG-0002mq-20 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:44 -0400 Received: from mail-pl0-x232.google.com ([2607:f8b0:400e:c01::232]:33315) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTueF-0002ly-Lv for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:39 -0400 Received: by mail-pl0-x232.google.com with SMTP id 6-v6so5313135plb.0 for ; Fri, 15 Jun 2018 12:44:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HLQgwyJfm0Wvm9CGkm1dh++7JQGvFnBJ5/SVt3YVwcg=; b=CMfHTRspfWVj0fTLR0MpVRjxdujU0j1YPiOmt+wXWfwPJl9K+P/ATs/9EnialAn1LM VvthjwR0u/LACsgsB0UeSZQFTUiu7xQppLr9x0DOcjWuBZqTRTRJbJBkBk8fhN+SXrAV swy6bl5t/AADj1ET2YEURirRgd0OmLtR/RORI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HLQgwyJfm0Wvm9CGkm1dh++7JQGvFnBJ5/SVt3YVwcg=; b=jgXF4gx73NmwdfRmiIYsali+nMkhzyUatFCkh8yvR2Vy9zqWjPfyWE0nwYN+zB6kQO BNwFX2au6STRN2edbQYlKXJiQrWMAdRUbkdDjtjrSx1MGO+SbSkXY5efLfJ7qfvYEuWq V61OBXv3ba3GacLJF9NmR9BZv2AJ7R41BuR48YoxT6DK2niM8PN4fT0un1OBEbS2FZSn 6djy2Tk2akZOjfBWD2Bxu4aBkwHFlFmsjc4faKy+PJo65yGQwMvpIGeWsQKU3K4h70Ap BWbpH0kq20a0F42RNExhMIYilqVHFynL98ehXyBC3V9itr/vXXzoS9mlWTZjWZWeHf75 4ftA== X-Gm-Message-State: APt69E3SEEwCq0D9Hge8UPEflS6Y43c0xRCHqVumYiry8MgTZdNcGCZp hgfyn7HwwIobzwQrYrFh/aQsajGHzN8= X-Received: by 2002:a17:902:b483:: with SMTP id y3-v6mr3583974plr.66.1529091878137; Fri, 15 Jun 2018 12:44:38 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.36 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:37 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:53 -1000 Message-Id: <20180615194354.12489-19-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::232 Subject: [Qemu-devel] [PULL v2 18/19] tcg: remove tb_lock X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, "Emilio G. Cota" Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: "Emilio G. Cota" Use mmap_lock in user-mode to protect TCG state and the page descriptors. In !user-mode, each vCPU has its own TCG state, so no locks needed. Per-page locks are used to protect the page descriptors. Per-TB locks are used in both modes to protect TB jumps. Some notes: - tb_lock is removed from notdirty_mem_write by passing a locked page_collection to tb_invalidate_phys_page_fast. - tcg_tb_lookup/remove/insert/etc have their own internal lock(s), so there is no need to further serialize access to them. - do_tb_flush is run in a safe async context, meaning no other vCPU threads are running. Therefore acquiring mmap_lock there is just to please tools such as thread sanitizer. - Not visible in the diff, but tb_invalidate_phys_page already has an assert_memory_lock. - cpu_io_recompile is !user-only, so no mmap_lock there. - Added mmap_unlock()'s before all siglongjmp's that could be called in user-mode while mmap_lock is held. + Added an assert for !have_mmap_lock() after returning from the longjmp in cpu_exec, just like we do in cpu_exec_step_atomic. Performance numbers before/after: Host: AMD Opteron(tm) Processor 6376 ubuntu 17.04 ppc64 bootup+shutdown time 700 +-+--+----+------+------------+-----------+------------*--+-+ | + + + + + *B | | before ***B*** ** * | |tb lock removal ###D### *** | 600 +-+ *** +-+ | ** # | | *B* #D | | *** * ## | 500 +-+ *** ### +-+ | * *** ### | | *B* # ## | | ** * #D# | 400 +-+ ** ## +-+ | ** ### | | ** ## | | ** # ## | 300 +-+ * B* #D# +-+ | B *** ### | | * ** #### | | * *** ### | 200 +-+ B *B #D# +-+ | #B* * ## # | | #* ## | | + D##D# + + + + | 100 +-+--+----+------+------------+-----------+------------+--+-+ 1 8 16 Guest CPUs 48 64 png: https://imgur.com/HwmBHXe debian jessie aarch64 bootup+shutdown time 90 +-+--+-----+-----+------------+------------+------------+--+-+ | + + + + + + | | before ***B*** B | 80 +tb lock removal ###D### **D +-+ | **### | | **## | 70 +-+ ** # +-+ | ** ## | | ** # | 60 +-+ *B ## +-+ | ** ## | | *** #D | 50 +-+ *** ## +-+ | * ** ### | | **B* ### | 40 +-+ **** # ## +-+ | **** #D# | | ***B** ### | 30 +-+ B***B** #### +-+ | B * * # ### | | B ###D# | 20 +-+ D ##D## +-+ | D# | | + + + + + + | 10 +-+--+-----+-----+------------+------------+------------+--+-+ 1 8 16 Guest CPUs 48 64 png: https://imgur.com/iGpGFtv The gains are high for 4-8 CPUs. Beyond that point, however, unrelated lock contention significantly hurts scalability. Reviewed-by: Richard Henderson Reviewed-by: Alex Bennée Signed-off-by: Emilio G. Cota Signed-off-by: Richard Henderson --- accel/tcg/translate-all.h | 3 +- include/exec/cpu-common.h | 2 +- include/exec/exec-all.h | 4 - include/exec/memory-internal.h | 6 +- include/exec/tb-context.h | 2 - tcg/tcg.h | 4 +- accel/tcg/cpu-exec.c | 34 ++------ accel/tcg/translate-all.c | 132 ++++++++++---------------------- exec.c | 26 +++---- linux-user/main.c | 3 - docs/devel/multi-thread-tcg.txt | 11 ++- 11 files changed, 75 insertions(+), 152 deletions(-) -- 2.17.1 diff --git a/accel/tcg/translate-all.h b/accel/tcg/translate-all.h index 6d1d2588b5..e6cb963d7e 100644 --- a/accel/tcg/translate-all.h +++ b/accel/tcg/translate-all.h @@ -26,7 +26,8 @@ struct page_collection *page_collection_lock(tb_page_addr_t start, tb_page_addr_t end); void page_collection_unlock(struct page_collection *set); -void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len); +void tb_invalidate_phys_page_fast(struct page_collection *pages, + tb_page_addr_t start, int len); void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, int is_cpu_write_access); void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end); diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h index 0b58e262f3..18b40d6145 100644 --- a/include/exec/cpu-common.h +++ b/include/exec/cpu-common.h @@ -23,7 +23,7 @@ typedef struct CPUListState { FILE *file; } CPUListState; -/* The CPU list lock nests outside tb_lock/tb_unlock. */ +/* The CPU list lock nests outside page_(un)lock or mmap_(un)lock */ void qemu_init_cpu_list(void); void cpu_list_lock(void); void cpu_list_unlock(void); diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 3c2a0efb55..25a6f28ab8 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -440,10 +440,6 @@ extern uintptr_t tci_tb_ptr; smaller than 4 bytes, so we don't worry about special-casing this. */ #define GETPC_ADJ 2 -void tb_lock(void); -void tb_unlock(void); -void tb_lock_reset(void); - #if !defined(CONFIG_USER_ONLY) && defined(CONFIG_DEBUG_TCG) void assert_no_pages_locked(void); #else diff --git a/include/exec/memory-internal.h b/include/exec/memory-internal.h index 56c25c0ef7..bb08fa4d2f 100644 --- a/include/exec/memory-internal.h +++ b/include/exec/memory-internal.h @@ -49,6 +49,8 @@ void mtree_print_dispatch(fprintf_function mon, void *f, struct AddressSpaceDispatch *d, MemoryRegion *root); +struct page_collection; + /* Opaque struct for passing info from memory_notdirty_write_prepare() * to memory_notdirty_write_complete(). Callers should treat all fields * as private, with the exception of @active. @@ -60,10 +62,10 @@ void mtree_print_dispatch(fprintf_function mon, void *f, */ typedef struct { CPUState *cpu; + struct page_collection *pages; ram_addr_t ram_addr; vaddr mem_vaddr; unsigned size; - bool locked; bool active; } NotDirtyInfo; @@ -91,7 +93,7 @@ typedef struct { * * This must only be called if we are using TCG; it will assert otherwise. * - * We may take a lock in the prepare call, so callers must ensure that + * We may take locks in the prepare call, so callers must ensure that * they don't exit (via longjump or otherwise) without calling complete. * * This call must only be made inside an RCU critical section. diff --git a/include/exec/tb-context.h b/include/exec/tb-context.h index 8c9b49c98e..feb585e0a7 100644 --- a/include/exec/tb-context.h +++ b/include/exec/tb-context.h @@ -32,8 +32,6 @@ typedef struct TBContext TBContext; struct TBContext { struct qht htable; - /* any access to the tbs or the page table must use this lock */ - QemuMutex tb_lock; /* statistics */ unsigned tb_flush_count; diff --git a/tcg/tcg.h b/tcg/tcg.h index e49b289ba1..532d2a0710 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -857,7 +857,7 @@ static inline bool tcg_op_buf_full(void) /* pool based memory allocation */ -/* user-mode: tb_lock must be held for tcg_malloc_internal. */ +/* user-mode: mmap_lock must be held for tcg_malloc_internal. */ void *tcg_malloc_internal(TCGContext *s, int size); void tcg_pool_reset(TCGContext *s); TranslationBlock *tcg_tb_alloc(TCGContext *s); @@ -875,7 +875,7 @@ TranslationBlock *tcg_tb_lookup(uintptr_t tc_ptr); void tcg_tb_foreach(GTraverseFunc func, gpointer user_data); size_t tcg_nb_tbs(void); -/* user-mode: Called with tb_lock held. */ +/* user-mode: Called with mmap_lock held. */ static inline void *tcg_malloc(int size) { TCGContext *s = tcg_ctx; diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c index c482008bc7..c738b7f7d6 100644 --- a/accel/tcg/cpu-exec.c +++ b/accel/tcg/cpu-exec.c @@ -212,20 +212,20 @@ static void cpu_exec_nocache(CPUState *cpu, int max_cycles, We only end up here when an existing TB is too long. */ cflags |= MIN(max_cycles, CF_COUNT_MASK); - tb_lock(); + mmap_lock(); tb = tb_gen_code(cpu, orig_tb->pc, orig_tb->cs_base, orig_tb->flags, cflags); tb->orig_tb = orig_tb; - tb_unlock(); + mmap_unlock(); /* execute the generated code */ trace_exec_tb_nocache(tb, tb->pc); cpu_tb_exec(cpu, tb); - tb_lock(); + mmap_lock(); tb_phys_invalidate(tb, -1); + mmap_unlock(); tcg_tb_remove(tb); - tb_unlock(); } #endif @@ -244,9 +244,7 @@ void cpu_exec_step_atomic(CPUState *cpu) tb = tb_lookup__cpu_state(cpu, &pc, &cs_base, &flags, cf_mask); if (tb == NULL) { mmap_lock(); - tb_lock(); tb = tb_gen_code(cpu, pc, cs_base, flags, cflags); - tb_unlock(); mmap_unlock(); } @@ -261,15 +259,13 @@ void cpu_exec_step_atomic(CPUState *cpu) cpu_tb_exec(cpu, tb); cc->cpu_exec_exit(cpu); } else { - /* We may have exited due to another problem here, so we need - * to reset any tb_locks we may have taken but didn't release. + /* * The mmap_lock is dropped by tb_gen_code if it runs out of * memory. */ #ifndef CONFIG_SOFTMMU tcg_debug_assert(!have_mmap_lock()); #endif - tb_lock_reset(); assert_no_pages_locked(); } @@ -398,20 +394,11 @@ static inline TranslationBlock *tb_find(CPUState *cpu, TranslationBlock *tb; target_ulong cs_base, pc; uint32_t flags; - bool acquired_tb_lock = false; tb = tb_lookup__cpu_state(cpu, &pc, &cs_base, &flags, cf_mask); if (tb == NULL) { - /* mmap_lock is needed by tb_gen_code, and mmap_lock must be - * taken outside tb_lock. As system emulation is currently - * single threaded the locks are NOPs. - */ mmap_lock(); - tb_lock(); - acquired_tb_lock = true; - tb = tb_gen_code(cpu, pc, cs_base, flags, cf_mask); - mmap_unlock(); /* We add the TB in the virtual pc hash table for the fast lookup */ atomic_set(&cpu->tb_jmp_cache[tb_jmp_cache_hash_func(pc)], tb); @@ -427,15 +414,8 @@ static inline TranslationBlock *tb_find(CPUState *cpu, #endif /* See if we can patch the calling TB. */ if (last_tb && !qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { - if (!acquired_tb_lock) { - tb_lock(); - acquired_tb_lock = true; - } tb_add_jump(last_tb, tb_exit, tb); } - if (acquired_tb_lock) { - tb_unlock(); - } return tb; } @@ -710,7 +690,9 @@ int cpu_exec(CPUState *cpu) g_assert(cpu == current_cpu); g_assert(cc == CPU_GET_CLASS(cpu)); #endif /* buggy compiler */ - tb_lock_reset(); +#ifndef CONFIG_SOFTMMU + tcg_debug_assert(!have_mmap_lock()); +#endif if (qemu_mutex_iothread_locked()) { qemu_mutex_unlock_iothread(); } diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index b33cd9bc98..f0c3fd4d03 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -88,13 +88,13 @@ #endif /* Access to the various translations structures need to be serialised via locks - * for consistency. This is automatic for SoftMMU based system - * emulation due to its single threaded nature. In user-mode emulation - * access to the memory related structures are protected with the - * mmap_lock. + * for consistency. + * In user-mode emulation access to the memory related structures are protected + * with mmap_lock. + * In !user-mode we use per-page locks. */ #ifdef CONFIG_SOFTMMU -#define assert_memory_lock() tcg_debug_assert(have_tb_lock) +#define assert_memory_lock() #else #define assert_memory_lock() tcg_debug_assert(have_mmap_lock()) #endif @@ -216,9 +216,6 @@ __thread TCGContext *tcg_ctx; TBContext tb_ctx; bool parallel_cpus; -/* translation block context */ -static __thread int have_tb_lock; - static void page_table_config_init(void) { uint32_t v_l1_bits; @@ -239,31 +236,6 @@ static void page_table_config_init(void) assert(v_l2_levels >= 0); } -#define assert_tb_locked() tcg_debug_assert(have_tb_lock) -#define assert_tb_unlocked() tcg_debug_assert(!have_tb_lock) - -void tb_lock(void) -{ - assert_tb_unlocked(); - qemu_mutex_lock(&tb_ctx.tb_lock); - have_tb_lock++; -} - -void tb_unlock(void) -{ - assert_tb_locked(); - have_tb_lock--; - qemu_mutex_unlock(&tb_ctx.tb_lock); -} - -void tb_lock_reset(void) -{ - if (have_tb_lock) { - qemu_mutex_unlock(&tb_ctx.tb_lock); - have_tb_lock = 0; - } -} - void cpu_gen_init(void) { tcg_context_init(&tcg_init_ctx); @@ -420,8 +392,7 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t host_pc, bool will_exit) * - fault during translation (instruction fetch) * - fault from helper (not using GETPC() macro) * - * Either way we need return early to avoid blowing up on a - * recursive tb_lock() as we can't resolve it here. + * Either way we need return early as we can't resolve it here. * * We are using unsigned arithmetic so if host_pc < * tcg_init_ctx.code_gen_buffer check_offset will wrap to way @@ -430,7 +401,6 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t host_pc, bool will_exit) check_offset = host_pc - (uintptr_t) tcg_init_ctx.code_gen_buffer; if (check_offset < tcg_init_ctx.code_gen_buffer_size) { - tb_lock(); tb = tcg_tb_lookup(host_pc); if (tb) { cpu_restore_state_from_tb(cpu, tb, host_pc, will_exit); @@ -441,7 +411,6 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t host_pc, bool will_exit) } r = true; } - tb_unlock(); } return r; @@ -1130,7 +1099,6 @@ static inline void code_gen_alloc(size_t tb_size) fprintf(stderr, "Could not allocate dynamic translator buffer\n"); exit(1); } - qemu_mutex_init(&tb_ctx.tb_lock); } static bool tb_cmp(const void *ap, const void *bp) @@ -1174,14 +1142,12 @@ void tcg_exec_init(unsigned long tb_size) /* * Allocate a new translation block. Flush the translation buffer if * too many translation blocks or too much generated code. - * - * Called with tb_lock held. */ static TranslationBlock *tb_alloc(target_ulong pc) { TranslationBlock *tb; - assert_tb_locked(); + assert_memory_lock(); tb = tcg_tb_alloc(tcg_ctx); if (unlikely(tb == NULL)) { @@ -1248,8 +1214,7 @@ static gboolean tb_host_size_iter(gpointer key, gpointer value, gpointer data) /* flush all the translation blocks */ static void do_tb_flush(CPUState *cpu, run_on_cpu_data tb_flush_count) { - tb_lock(); - + mmap_lock(); /* If it is already been done on request of another CPU, * just retry. */ @@ -1279,7 +1244,7 @@ static void do_tb_flush(CPUState *cpu, run_on_cpu_data tb_flush_count) atomic_mb_set(&tb_ctx.tb_flush_count, tb_ctx.tb_flush_count + 1); done: - tb_unlock(); + mmap_unlock(); } void tb_flush(CPUState *cpu) @@ -1313,7 +1278,7 @@ do_tb_invalidate_check(struct qht *ht, void *p, uint32_t hash, void *userp) /* verify that all the pages have correct rights for code * - * Called with tb_lock held. + * Called with mmap_lock held. */ static void tb_invalidate_check(target_ulong address) { @@ -1343,7 +1308,10 @@ static void tb_page_check(void) #endif /* CONFIG_USER_ONLY */ -/* call with @pd->lock held */ +/* + * user-mode: call with mmap_lock held + * !user-mode: call with @pd->lock held + */ static inline void tb_page_remove(PageDesc *pd, TranslationBlock *tb) { TranslationBlock *tb1; @@ -1437,7 +1405,11 @@ static inline void tb_jmp_unlink(TranslationBlock *dest) qemu_spin_unlock(&dest->jmp_lock); } -/* If @rm_from_page_list is set, call with the TB's pages' locks held */ +/* + * In user-mode, call with mmap_lock held. + * In !user-mode, if @rm_from_page_list is set, call with the TB's pages' + * locks held. + */ static void do_tb_phys_invalidate(TranslationBlock *tb, bool rm_from_page_list) { CPUState *cpu; @@ -1445,7 +1417,7 @@ static void do_tb_phys_invalidate(TranslationBlock *tb, bool rm_from_page_list) uint32_t h; tb_page_addr_t phys_pc; - assert_tb_locked(); + assert_memory_lock(); /* make sure no further incoming jumps will be chained to this TB */ qemu_spin_lock(&tb->jmp_lock); @@ -1498,7 +1470,7 @@ static void tb_phys_invalidate__locked(TranslationBlock *tb) /* invalidate one TB * - * Called with tb_lock held. + * Called with mmap_lock held in user-mode. */ void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr) { @@ -1543,7 +1515,7 @@ static void build_page_bitmap(PageDesc *p) /* add the tb in the target page and protect it if necessary * * Called with mmap_lock held for user-mode emulation. - * Called with @p->lock held. + * Called with @p->lock held in !user-mode. */ static inline void tb_page_add(PageDesc *p, TranslationBlock *tb, unsigned int n, tb_page_addr_t page_addr) @@ -1815,10 +1787,9 @@ TranslationBlock *tb_gen_code(CPUState *cpu, if ((pc & TARGET_PAGE_MASK) != virt_page2) { phys_page2 = get_page_addr_code(env, virt_page2); } - /* As long as consistency of the TB stuff is provided by tb_lock in user - * mode and is implicit in single-threaded softmmu emulation, no explicit - * memory barrier is required before tb_link_page() makes the TB visible - * through the physical hash table and physical page list. + /* + * No explicit memory barrier is required -- tb_link_page() makes the + * TB visible in a consistent state. */ existing_tb = tb_link_page(tb, phys_pc, phys_page2); /* if the TB already exists, discard what we just translated */ @@ -1834,8 +1805,9 @@ TranslationBlock *tb_gen_code(CPUState *cpu, } /* - * Call with all @pages locked. * @p must be non-NULL. + * user-mode: call with mmap_lock held. + * !user-mode: call with all @pages locked. */ static void tb_invalidate_phys_page_range__locked(struct page_collection *pages, @@ -1920,6 +1892,7 @@ tb_invalidate_phys_page_range__locked(struct page_collection *pages, page_collection_unlock(pages); /* Force execution of one insn next time. */ cpu->cflags_next_tb = 1 | curr_cflags(); + mmap_unlock(); cpu_loop_exit_noexc(cpu); } #endif @@ -1932,8 +1905,7 @@ tb_invalidate_phys_page_range__locked(struct page_collection *pages, * access: the virtual CPU will exit the current TB if code is modified inside * this TB. * - * Called with tb_lock/mmap_lock held for user-mode emulation - * Called with tb_lock held for system-mode emulation + * Called with mmap_lock held for user-mode emulation */ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, int is_cpu_write_access) @@ -1942,7 +1914,6 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, PageDesc *p; assert_memory_lock(); - assert_tb_locked(); p = page_find(start >> TARGET_PAGE_BITS); if (p == NULL) { @@ -1961,14 +1932,15 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end, * access: the virtual CPU will exit the current TB if code is modified inside * this TB. * - * Called with mmap_lock held for user-mode emulation, grabs tb_lock - * Called with tb_lock held for system-mode emulation + * Called with mmap_lock held for user-mode emulation. */ -static void tb_invalidate_phys_range_1(tb_page_addr_t start, tb_page_addr_t end) +void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) { struct page_collection *pages; tb_page_addr_t next; + assert_memory_lock(); + pages = page_collection_lock(start, end); for (next = (start & TARGET_PAGE_MASK) + TARGET_PAGE_SIZE; start < end; @@ -1984,30 +1956,16 @@ static void tb_invalidate_phys_range_1(tb_page_addr_t start, tb_page_addr_t end) page_collection_unlock(pages); } -#ifdef CONFIG_SOFTMMU -void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) -{ - assert_tb_locked(); - tb_invalidate_phys_range_1(start, end); -} -#else -void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) -{ - assert_memory_lock(); - tb_lock(); - tb_invalidate_phys_range_1(start, end); - tb_unlock(); -} -#endif - #ifdef CONFIG_SOFTMMU /* len must be <= 8 and start must be a multiple of len. * Called via softmmu_template.h when code areas are written to with * iothread mutex not held. + * + * Call with all @pages in the range [@start, @start + len[ locked. */ -void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len) +void tb_invalidate_phys_page_fast(struct page_collection *pages, + tb_page_addr_t start, int len) { - struct page_collection *pages; PageDesc *p; #if 0 @@ -2026,7 +1984,6 @@ void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len) return; } - pages = page_collection_lock(start, start + len); assert_page_locked(p); if (!p->code_bitmap && ++p->code_write_count >= SMC_BITMAP_USE_THRESHOLD) { @@ -2045,7 +2002,6 @@ void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len) do_invalidate: tb_invalidate_phys_page_range__locked(pages, p, start, start + len, 1); } - page_collection_unlock(pages); } #else /* Called with mmap_lock held. If pc is not 0 then it indicates the @@ -2077,7 +2033,6 @@ static bool tb_invalidate_phys_page(tb_page_addr_t addr, uintptr_t pc) return false; } - tb_lock(); #ifdef TARGET_HAS_PRECISE_SMC if (p->first_tb && pc != 0) { current_tb = tcg_tb_lookup(pc); @@ -2110,12 +2065,9 @@ static bool tb_invalidate_phys_page(tb_page_addr_t addr, uintptr_t pc) if (current_tb_modified) { /* Force execution of one insn next time. */ cpu->cflags_next_tb = 1 | curr_cflags(); - /* tb_lock will be reset after cpu_loop_exit_noexc longjmps - * back into the cpu_exec loop. */ return true; } #endif - tb_unlock(); return false; } @@ -2136,18 +2088,18 @@ void tb_invalidate_phys_addr(AddressSpace *as, hwaddr addr, MemTxAttrs attrs) return; } ram_addr = memory_region_get_ram_addr(mr) + addr; - tb_lock(); tb_invalidate_phys_page_range(ram_addr, ram_addr + 1, 0); - tb_unlock(); rcu_read_unlock(); } #endif /* !defined(CONFIG_USER_ONLY) */ -/* Called with tb_lock held. */ +/* user-mode: call with mmap_lock held */ void tb_check_watchpoint(CPUState *cpu) { TranslationBlock *tb; + assert_memory_lock(); + tb = tcg_tb_lookup(cpu->mem_io_pc); if (tb) { /* We can use retranslation to find the PC. */ @@ -2181,7 +2133,6 @@ void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr) TranslationBlock *tb; uint32_t n; - tb_lock(); tb = tcg_tb_lookup(retaddr); if (!tb) { cpu_abort(cpu, "cpu_io_recompile: could not find TB for pc=%p", @@ -2229,9 +2180,6 @@ void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr) * repeating the fault, which is horribly inefficient. * Better would be to execute just this insn uncached, or generate a * second new TB. - * - * cpu_loop_exit_noexc will longjmp back to cpu_exec where the - * tb_lock gets reset. */ cpu_loop_exit_noexc(cpu); } diff --git a/exec.c b/exec.c index ebadc0e302..28f9bdcbf9 100644 --- a/exec.c +++ b/exec.c @@ -1031,9 +1031,7 @@ const char *parse_cpu_model(const char *cpu_model) static void breakpoint_invalidate(CPUState *cpu, target_ulong pc) { mmap_lock(); - tb_lock(); tb_invalidate_phys_page_range(pc, pc + 1, 0); - tb_unlock(); mmap_unlock(); } #else @@ -2644,21 +2642,21 @@ void memory_notdirty_write_prepare(NotDirtyInfo *ndi, ndi->ram_addr = ram_addr; ndi->mem_vaddr = mem_vaddr; ndi->size = size; - ndi->locked = false; + ndi->pages = NULL; assert(tcg_enabled()); if (!cpu_physical_memory_get_dirty_flag(ram_addr, DIRTY_MEMORY_CODE)) { - ndi->locked = true; - tb_lock(); - tb_invalidate_phys_page_fast(ram_addr, size); + ndi->pages = page_collection_lock(ram_addr, ram_addr + size); + tb_invalidate_phys_page_fast(ndi->pages, ram_addr, size); } } /* Called within RCU critical section. */ void memory_notdirty_write_complete(NotDirtyInfo *ndi) { - if (ndi->locked) { - tb_unlock(); + if (ndi->pages) { + page_collection_unlock(ndi->pages); + ndi->pages = NULL; } /* Set both VGA and migration bits for simplicity and to remove @@ -2745,18 +2743,16 @@ static void check_watchpoint(int offset, int len, MemTxAttrs attrs, int flags) } cpu->watchpoint_hit = wp; - /* Both tb_lock and iothread_mutex will be reset when - * cpu_loop_exit or cpu_loop_exit_noexc longjmp - * back into the cpu_exec main loop. - */ - tb_lock(); + mmap_lock(); tb_check_watchpoint(cpu); if (wp->flags & BP_STOP_BEFORE_ACCESS) { cpu->exception_index = EXCP_DEBUG; + mmap_unlock(); cpu_loop_exit(cpu); } else { /* Force execution of one insn next time. */ cpu->cflags_next_tb = 1 | curr_cflags(); + mmap_unlock(); cpu_loop_exit_noexc(cpu); } } @@ -3147,9 +3143,9 @@ static void invalidate_and_set_dirty(MemoryRegion *mr, hwaddr addr, } if (dirty_log_mask & (1 << DIRTY_MEMORY_CODE)) { assert(tcg_enabled()); - tb_lock(); + mmap_lock(); tb_invalidate_phys_range(addr, addr + length); - tb_unlock(); + mmap_unlock(); dirty_log_mask &= ~(1 << DIRTY_MEMORY_CODE); } cpu_physical_memory_set_dirty_range(addr, length, dirty_log_mask); diff --git a/linux-user/main.c b/linux-user/main.c index 78d6d3e7eb..84e9ec9335 100644 --- a/linux-user/main.c +++ b/linux-user/main.c @@ -120,7 +120,6 @@ void fork_start(void) { start_exclusive(); mmap_fork_start(); - qemu_mutex_lock(&tb_ctx.tb_lock); cpu_list_lock(); } @@ -136,14 +135,12 @@ void fork_end(int child) QTAILQ_REMOVE(&cpus, cpu, node); } } - qemu_mutex_init(&tb_ctx.tb_lock); qemu_init_cpu_list(); gdbserver_fork(thread_cpu); /* qemu_init_cpu_list() takes care of reinitializing the * exclusive state, so we don't need to end_exclusive() here. */ } else { - qemu_mutex_unlock(&tb_ctx.tb_lock); cpu_list_unlock(); end_exclusive(); } diff --git a/docs/devel/multi-thread-tcg.txt b/docs/devel/multi-thread-tcg.txt index df83445ccf..06530be1e9 100644 --- a/docs/devel/multi-thread-tcg.txt +++ b/docs/devel/multi-thread-tcg.txt @@ -61,6 +61,7 @@ have their block-to-block jumps patched. Global TCG State ---------------- +### User-mode emulation We need to protect the entire code generation cycle including any post generation patching of the translated code. This also implies a shared translation buffer which contains code running on all cores. Any @@ -75,9 +76,11 @@ patching. (Current solution) -Mainly as part of the linux-user work all code generation is -serialised with a tb_lock(). For the SoftMMU tb_lock() also takes the -place of mmap_lock() in linux-user. +Code generation is serialised with mmap_lock(). + +### !User-mode emulation +Each vCPU has its own TCG context and associated TCG region, thereby +requiring no locking. Translation Blocks ------------------ @@ -195,7 +198,7 @@ work as "safe work" and exiting the cpu run loop. This ensure by the time execution restarts all flush operations have completed. TLB flag updates are all done atomically and are also protected by the -tb_lock() which is used by the functions that update the TLB in bulk. +corresponding page lock. (Known limitation) From patchwork Fri Jun 15 19:43:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 138764 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1253660lji; Fri, 15 Jun 2018 12:55:08 -0700 (PDT) X-Google-Smtp-Source: ADUXVKIkoYBOsH5t/TZCYuPyBS5wppj0FPGenX6hlRlM8M17fnUMDKHWQ6uMXgEmySe4bm0uBr7f X-Received: by 2002:ac8:2372:: with SMTP id b47-v6mr2847290qtb.358.1529092508868; Fri, 15 Jun 2018 12:55:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529092508; cv=none; d=google.com; s=arc-20160816; b=QhiSjvdZ5aDhlVK+UXfq7pHsjIUROGpyBlUr8Ao22YApB6wF6h4hOuk19XBJr/ztWt RdYxpVLm0SqbdheJsf+6l+LWEnOxSHw7nq2uI6enx/lDGLDkGQRs0kUPgE36QZnSpVlH BBwmDu2DgB4KNt3JWGQZYmpq++ONzHG5/G0IUydovWdx3+4LdIRZU+NCPWeWVo4xUDEW 5z4xVf785SdEp8yVwpHCPcjWff1DxNKA7lwMdtvRjQBXXI65L9XO19kf+vCNRqOUjNPr r8Z7ilXaOWwQPUJi5nVEkjkKI6L0dJ/AkWpI5sqgOjPS5abRURyWCbde5zCz/6MUp/ei Hi2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=uXTRBTDADa3Js0VPWdrswx2J3J4crajm5lnDYFt+lbA=; b=wSvFlVYE9Fd/CjPN3i1Xy0j1TIE9g76vksiKDZiQQAPo5xxvtZQwM/NEBBCYjRT5Yp h+6QPPWdyzeobuVyV0TjbiMgKsxonL224KkBaB4K6qjVNDCkNWWf/XjngNPOeOw9/JMx 9MrDfdlc2nBBAtPKHD4aUc3ydVCBucxWr2+5eG8ZiAU15M0XHSr2hCJN2vJsUew7qjTa uC1EVXt77tq5gWmGL4TOp/VugxjtdmpDZS6YMSpDoCQpakLnUTFvQqUMTmzhSVeCPGLv w5ZEGvRzR2msAZOs3lqz2gI0Nlr7uSuARbtqgQap8np0n49JOYeVIWNzFEXehRxMOPMw o5Mg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=UR+5bQi1; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id n44-v6si7970803qvd.126.2018.06.15.12.55.08 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 12:55:08 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=UR+5bQi1; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49007 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuoO-0008UV-8O for patch@linaro.org; Fri, 15 Jun 2018 15:55:08 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51729) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTueL-0000iA-15 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTueH-0002oU-Ln for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:45 -0400 Received: from mail-pl0-x229.google.com ([2607:f8b0:400e:c01::229]:41005) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTueH-0002ny-Dc for qemu-devel@nongnu.org; Fri, 15 Jun 2018 15:44:41 -0400 Received: by mail-pl0-x229.google.com with SMTP id w8-v6so4343069ply.8 for ; Fri, 15 Jun 2018 12:44:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uXTRBTDADa3Js0VPWdrswx2J3J4crajm5lnDYFt+lbA=; b=UR+5bQi1Mn2zlfDu4LIo4RqcfP/+HDWqOgytIOpOnFhO0UmAhu7uxMHULQUwpgtvfY UV6PVxKCsL9duSLjvkpGFCv5h3GD+3d183w9MyIN6A5iFW7bK+vbvll1p9X5i7nv6Nk2 tdwuTu0DXpDBUX2QItgzxYa5k1namfgjP2KIQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uXTRBTDADa3Js0VPWdrswx2J3J4crajm5lnDYFt+lbA=; b=AVIQtgeouSCI4aeVzj5kELpT+iWvSeJ/JukagE6t5dmqxaQsxSIc0qNoZtWYPUZv1a nkfMX9VxK9HFD35i8V6klZqIMJ10znh/q6KLG0bHjRK5OE/nwncNO/3BPQUa8XU/feU+ +uVkgSkVdjICBEf4FBtl2cOWUEI26dK2cJORIPTBPRWM9EK9ls27Jf5kX3JogxSjCypj 7W7yn6MvjsFg1MF6TukDKMfp9gdYIPRu/SQr2AEkLj5oAK2+5wEfp6Lv9dfk0F6hosfv /WUbmjj8FW+shJA84BcltiY8G1TLhnDp4cmVVcUCDvgta529x+Fo0XHtIlC2qucYgprJ 3sEg== X-Gm-Message-State: APt69E0EqNJjBrWW21dHDKLcYY+YU4V17Ve3nFt3s0h7sdtuf2qho0xx t26YES+8tMNRqAiksX78cc4cLYUY6A4= X-Received: by 2002:a17:902:d20c:: with SMTP id t12-v6mr3561059ply.63.1529091880141; Fri, 15 Jun 2018 12:44:40 -0700 (PDT) Received: from cloudburst.twiddle.net (rrcs-173-198-77-219.west.biz.rr.com. [173.198.77.219]) by smtp.gmail.com with ESMTPSA id 29-v6sm14038360pfj.14.2018.06.15.12.44.38 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 12:44:39 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 15 Jun 2018 09:43:54 -1000 Message-Id: <20180615194354.12489-20-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194354.12489-1-richard.henderson@linaro.org> References: <20180615194354.12489-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::229 Subject: [Qemu-devel] [PULL v2 19/19] tcg: Reduce max TB opcode count X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-stable@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Also, assert that we don't overflow any of two different offsets into the TB. Both unwind and goto_tb both record a uint16_t for later use. This fixes an arm-softmmu test case utilizing NEON in which there is a TB generated that runs to 7800 opcodes, and compiles to 96k on an x86_64 host. This overflows the 16-bit offset in which we record the goto_tb reset offset. Because of that overflow, we install a jump destination that goes to neverland. Boom. With this reduced op count, the same TB compiles to about 48k for aarch64, ppc64le, and x86_64 hosts, and neither assertion fires. Cc: qemu-stable@nongnu.org Reported-by: "Jason A. Donenfeld" Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Richard Henderson --- tcg/tcg.h | 6 ++++-- tcg/aarch64/tcg-target.inc.c | 2 +- tcg/arm/tcg-target.inc.c | 2 +- tcg/i386/tcg-target.inc.c | 2 +- tcg/mips/tcg-target.inc.c | 2 +- tcg/ppc/tcg-target.inc.c | 4 ++-- tcg/s390/tcg-target.inc.c | 2 +- tcg/sparc/tcg-target.inc.c | 4 ++-- tcg/tcg.c | 13 ++++++++++++- tcg/tci/tcg-target.inc.c | 2 +- 10 files changed, 26 insertions(+), 13 deletions(-) -- 2.17.1 diff --git a/tcg/tcg.h b/tcg/tcg.h index 532d2a0710..f9f12378e9 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -850,9 +850,11 @@ static inline bool tcg_op_buf_full(void) /* This is not a hard limit, it merely stops translation when * we have produced "enough" opcodes. We want to limit TB size * such that a RISC host can reasonably use a 16-bit signed - * branch within the TB. + * branch within the TB. We also need to be mindful of the + * 16-bit unsigned offsets, TranslationBlock.jmp_reset_offset[] + * and TCGContext.gen_insn_end_off[]. */ - return tcg_ctx->nb_ops >= 8000; + return tcg_ctx->nb_ops >= 4000; } /* pool based memory allocation */ diff --git a/tcg/aarch64/tcg-target.inc.c b/tcg/aarch64/tcg-target.inc.c index be3192078d..4562d36d1b 100644 --- a/tcg/aarch64/tcg-target.inc.c +++ b/tcg/aarch64/tcg-target.inc.c @@ -1733,7 +1733,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, tcg_out_insn(s, 3305, LDR, offset, TCG_REG_TMP); } tcg_out_insn(s, 3207, BR, TCG_REG_TMP); - s->tb_jmp_reset_offset[a0] = tcg_current_code_size(s); + set_jmp_reset_offset(s, a0); break; case INDEX_op_goto_ptr: diff --git a/tcg/arm/tcg-target.inc.c b/tcg/arm/tcg-target.inc.c index 56a32a470f..e1fbf465cb 100644 --- a/tcg/arm/tcg-target.inc.c +++ b/tcg/arm/tcg-target.inc.c @@ -1822,7 +1822,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc, tcg_out_movi32(s, COND_AL, base, ptr - dil); } tcg_out_ld32_12(s, COND_AL, TCG_REG_PC, base, dil); - s->tb_jmp_reset_offset[args[0]] = tcg_current_code_size(s); + set_jmp_reset_offset(s, args[0]); } break; case INDEX_op_goto_ptr: diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index 0d0ff524b7..e87b0d445e 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -2245,7 +2245,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc, tcg_out_modrm_offset(s, OPC_GRP5, EXT5_JMPN_Ev, -1, (intptr_t)(s->tb_jmp_target_addr + a0)); } - s->tb_jmp_reset_offset[a0] = tcg_current_code_size(s); + set_jmp_reset_offset(s, a0); break; case INDEX_op_goto_ptr: /* jmp to the given host address (could be epilogue) */ diff --git a/tcg/mips/tcg-target.inc.c b/tcg/mips/tcg-target.inc.c index ca5f1d4894..cff525373b 100644 --- a/tcg/mips/tcg-target.inc.c +++ b/tcg/mips/tcg-target.inc.c @@ -1744,7 +1744,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc, tcg_out_opc_reg(s, OPC_JR, 0, TCG_TMP0, 0); } tcg_out_nop(s); - s->tb_jmp_reset_offset[a0] = tcg_current_code_size(s); + set_jmp_reset_offset(s, a0); break; case INDEX_op_goto_ptr: /* jmp to the given host address (could be epilogue) */ diff --git a/tcg/ppc/tcg-target.inc.c b/tcg/ppc/tcg-target.inc.c index 86f7de5f7e..c2f729ee8f 100644 --- a/tcg/ppc/tcg-target.inc.c +++ b/tcg/ppc/tcg-target.inc.c @@ -2025,10 +2025,10 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, const TCGArg *args, } tcg_out32(s, MTSPR | RS(TCG_REG_TB) | CTR); tcg_out32(s, BCCTR | BO_ALWAYS); - s->tb_jmp_reset_offset[args[0]] = c = tcg_current_code_size(s); + set_jmp_reset_offset(s, args[0]); if (USE_REG_TB) { /* For the unlinked case, need to reset TCG_REG_TB. */ - c = -c; + c = -tcg_current_code_size(s); assert(c == (int16_t)c); tcg_out32(s, ADDI | TAI(TCG_REG_TB, TCG_REG_TB, c)); } diff --git a/tcg/s390/tcg-target.inc.c b/tcg/s390/tcg-target.inc.c index 9af6dcef05..17c435ade5 100644 --- a/tcg/s390/tcg-target.inc.c +++ b/tcg/s390/tcg-target.inc.c @@ -1783,7 +1783,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc, /* and go there */ tcg_out_insn(s, RR, BCR, S390_CC_ALWAYS, TCG_REG_TB); } - s->tb_jmp_reset_offset[a0] = tcg_current_code_size(s); + set_jmp_reset_offset(s, a0); /* For the unlinked path of goto_tb, we need to reset TCG_REG_TB to the beginning of this TB. */ diff --git a/tcg/sparc/tcg-target.inc.c b/tcg/sparc/tcg-target.inc.c index bc673bd8c6..04bdc3df5e 100644 --- a/tcg/sparc/tcg-target.inc.c +++ b/tcg/sparc/tcg-target.inc.c @@ -1388,12 +1388,12 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, tcg_out_arithi(s, TCG_REG_G0, TCG_REG_TB, 0, JMPL); tcg_out_nop(s); } - s->tb_jmp_reset_offset[a0] = c = tcg_current_code_size(s); + set_jmp_reset_offset(s, a0); /* For the unlinked path of goto_tb, we need to reset TCG_REG_TB to the beginning of this TB. */ if (USE_REG_TB) { - c = -c; + c = -tcg_current_code_size(s); if (check_fit_i32(c, 13)) { tcg_out_arithi(s, TCG_REG_TB, TCG_REG_TB, c, ARITH_ADD); } else { diff --git a/tcg/tcg.c b/tcg/tcg.c index 1d1dfd7f7c..f27b22bd3c 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -306,6 +306,14 @@ TCGLabel *gen_new_label(void) return l; } +static void set_jmp_reset_offset(TCGContext *s, int which) +{ + size_t off = tcg_current_code_size(s); + s->tb_jmp_reset_offset[which] = off; + /* Make sure that we didn't overflow the stored offset. */ + assert(s->tb_jmp_reset_offset[which] == off); +} + #include "tcg-target.inc.c" /* compare a pointer @ptr and a tb_tc @s */ @@ -3532,7 +3540,10 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb) break; case INDEX_op_insn_start: if (num_insns >= 0) { - s->gen_insn_end_off[num_insns] = tcg_current_code_size(s); + size_t off = tcg_current_code_size(s); + s->gen_insn_end_off[num_insns] = off; + /* Assert that we do not overflow our stored offset. */ + assert(s->gen_insn_end_off[num_insns] == off); } num_insns++; for (i = 0; i < TARGET_INSN_START_WORDS; ++i) { diff --git a/tcg/tci/tcg-target.inc.c b/tcg/tci/tcg-target.inc.c index cc949bea85..62ed097254 100644 --- a/tcg/tci/tcg-target.inc.c +++ b/tcg/tci/tcg-target.inc.c @@ -574,7 +574,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, const TCGArg *args, /* Indirect jump method. */ TODO(); } - s->tb_jmp_reset_offset[args[0]] = tcg_current_code_size(s); + set_jmp_reset_offset(s, args[0]); break; case INDEX_op_br: tci_out_label(s, arg_label(args[0]));