From patchwork Fri Nov 23 14:45:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 151883 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp2213920ljp; Fri, 23 Nov 2018 06:52:26 -0800 (PST) X-Google-Smtp-Source: AFSGD/WQMlTxbklOrVIgsHOUSEPBUtqM3iGNN17zfD33wv8gnFJKbSwpG/RnhHERH++lX0WAh0s9 X-Received: by 2002:a25:4bc4:: with SMTP id y187-v6mr15935367yba.264.1542984746526; Fri, 23 Nov 2018 06:52:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542984746; cv=none; d=google.com; s=arc-20160816; b=Ndh8TpCl63aniILSZYgXpr71Y9fJbyVh91+ZLYpWMQx0RvcjuWgWkJadPw0oc4AD7F qUXovxGKbdR1A/swDuBFducldRWU7K5a7bLSFegt/HqhEEzsfHGOV+AVh5Zy/WS1Ys2I lCFtK9J3IY0ieMn5PKzf49OYd0Oumn3IBZAHsNWCVIe8CbiEHwsVV92/xo8uRwYywXBM E2mffvNZWbSuSPUh23ZxC8kasBc8txa+Wejy/w+nF2tOfv9klSL4EYw6vY7zpyqhJ1M/ iVUWxzRsySNgVIhITgFEsBvlI9UwRXO5u0n3+mHKTgzZS4p1Lbj4sKF4Rcz09diRjN1k 2VpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature; bh=gMtbi1YlmG7TeeMSvQDC1HT4x/YMqKbJWvOlr3IvkwQ=; b=GgFUgBY0SlcvTxzKCJtB/IaHA1kGEltDUU79HHPJeziFt9KMXW+Om/Pw3VwvzgT84/ KVOWk65g8mzwNOSAAdirkOe/cVPR42pN3sT0WPk6boA+76khKPsaM8iguIEGAV0xWpb9 eZBbxAlqR54tTo2iHxJYBSxLEecpVBSgBrhrp3TJ1rYxQMY58ToCrtvXbvCgeqoT7jot xj//zphdXCXUR2MNj+MzIX4uYaoCqAAr3MNgnEl/NAivlqoL7ow3Gru1iT8X70yS1tsg fpQmVEkpqG9ADh01//6lt/t8jSF4MsW1sx/zEnld3kKkA6Pk0IX6T19Fk7phq5eP9Vjj nSag== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=WI97c4vs; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id r187-v6si34901956ywh.460.2018.11.23.06.52.26 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 23 Nov 2018 06:52:26 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=WI97c4vs; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:52785 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gQCoj-0006E0-SE for patch@linaro.org; Fri, 23 Nov 2018 09:52:25 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43769) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gQCif-0007yT-Qu for qemu-devel@nongnu.org; Fri, 23 Nov 2018 09:46:12 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gQCid-00038B-MO for qemu-devel@nongnu.org; Fri, 23 Nov 2018 09:46:09 -0500 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]:50875) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gQCic-00036n-KQ for qemu-devel@nongnu.org; Fri, 23 Nov 2018 09:46:07 -0500 Received: by mail-wm1-x342.google.com with SMTP id 125so12199204wmh.0 for ; Fri, 23 Nov 2018 06:46:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gMtbi1YlmG7TeeMSvQDC1HT4x/YMqKbJWvOlr3IvkwQ=; b=WI97c4vsLQIhI8ZI+ykjd9EbCJUVakHqvRhgolNwE7j6N2/h18B4bktAfpRUgZvfUN rlVLW6I4UDmp7W0ESy/+KvZ3E2OCfDQM/a6gg1spCN9GSzOWpoG2CnYFaXC2046hCsHq s7Pr+uucuMItAX9ILPCFgScPwVOnMv/cy4pLc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gMtbi1YlmG7TeeMSvQDC1HT4x/YMqKbJWvOlr3IvkwQ=; b=gKKid48oo9b45qMAsXqt1060OqGrrlf1siacWrENHKvmYSrihPmQ7sMOTgVtmlVbvo eKdGsbWRhquy06Fy43+znuLlndya9UThAStWaKH2tJ6PxgLRixEbFDY0WUzE23G1nEgE HXH7ZcgonLAz3aMkSGa5XuisGH07OcGvR5Yc+QnCVv7KJFjwX7jVbbJGG3CyskTaF1Qu +ndDIN8+KiGrRy1DceB4gx78x9HRx90tCaqgpl+RRYSWbgRxo3LBqt+CR2jEFiM9Ns1O i1QNFqp82p0Zn5JcNMYvaZulc6M0JiZOv44VC4CvAlz7j5H/lkBDv/Fp5OkWhySapAqR n4mQ== X-Gm-Message-State: AA+aEWYLhSkxo+Li7qXD9BLMbXsZBJ3VB0aFpTlI3CcE//PciWOmcSq8 NdNV8F/mHyWVaz6qxgVgiUCXFwZOyl3b9Q== X-Received: by 2002:a1c:aecb:: with SMTP id x194mr13020337wme.96.1542984364704; Fri, 23 Nov 2018 06:46:04 -0800 (PST) Received: from cloudburst.twiddle.net ([195.77.246.50]) by smtp.gmail.com with ESMTPSA id p74sm10339630wmd.29.2018.11.23.06.46.03 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 23 Nov 2018 06:46:04 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 23 Nov 2018 15:45:25 +0100 Message-Id: <20181123144558.5048-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181123144558.5048-1-richard.henderson@linaro.org> References: <20181123144558.5048-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::342 Subject: [Qemu-devel] [PATCH for-4.0 v2 04/37] tcg: Add TCG_TARGET_NEED_LDST_OOL_LABELS X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alistair.Francis@wdc.com Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This variant of tcg-ldst.inc.c allows the entire thunk to be moved out-of-line, with caching across TBs within a region. Signed-off-by: Richard Henderson --- tcg/tcg.h | 5 +++ accel/tcg/translate-all.c | 15 +++++-- tcg/tcg-ldst-ool.inc.c | 95 +++++++++++++++++++++++++++++++++++++++ tcg/tcg.c | 28 ++++++++++++ 4 files changed, 140 insertions(+), 3 deletions(-) create mode 100644 tcg/tcg-ldst-ool.inc.c -- 2.17.2 diff --git a/tcg/tcg.h b/tcg/tcg.h index f4efbaa680..73737dc671 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -706,6 +706,11 @@ struct TCGContext { #ifdef TCG_TARGET_NEED_LDST_LABELS QSIMPLEQ_HEAD(ldst_labels, TCGLabelQemuLdst) ldst_labels; #endif +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + QSIMPLEQ_HEAD(ldst_labels, TCGLabelQemuLdstOol) ldst_ool_labels; + GHashTable *ldst_ool_thunks; + size_t ldst_ool_generation; +#endif #ifdef TCG_TARGET_NEED_POOL_LABELS struct TCGLabelPoolData *pool_labels; #endif diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 639f0b2728..dd9332b24c 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -1678,6 +1678,9 @@ TranslationBlock *tb_gen_code(CPUState *cpu, target_ulong virt_page2; tcg_insn_unit *gen_code_buf; int gen_code_size, search_size; +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + size_t ldst_ool_generation = tcg_ctx->ldst_ool_generation; +#endif #ifdef CONFIG_PROFILER TCGProfile *prof = &tcg_ctx->prof; int64_t ti; @@ -1831,10 +1834,16 @@ TranslationBlock *tb_gen_code(CPUState *cpu, existing_tb = tb_link_page(tb, phys_pc, phys_page2); /* if the TB already exists, discard what we just translated */ if (unlikely(existing_tb != tb)) { - uintptr_t orig_aligned = (uintptr_t)gen_code_buf; + bool discard = true; - orig_aligned -= ROUND_UP(sizeof(*tb), qemu_icache_linesize); - atomic_set(&tcg_ctx->code_gen_ptr, (void *)orig_aligned); +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + discard = ldst_ool_generation == tcg_ctx->ldst_ool_generation; +#endif + if (discard) { + uintptr_t orig_aligned = (uintptr_t)gen_code_buf; + orig_aligned -= ROUND_UP(sizeof(*tb), qemu_icache_linesize); + atomic_set(&tcg_ctx->code_gen_ptr, (void *)orig_aligned); + } return existing_tb; } tcg_tb_insert(tb); diff --git a/tcg/tcg-ldst-ool.inc.c b/tcg/tcg-ldst-ool.inc.c new file mode 100644 index 0000000000..70b8789797 --- /dev/null +++ b/tcg/tcg-ldst-ool.inc.c @@ -0,0 +1,95 @@ +/* + * TCG Backend Data: load-store optimization only. + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + * THE SOFTWARE. + */ + +typedef struct TCGLabelQemuLdstOol { + QSIMPLEQ_ENTRY(TCGLabelQemuLdstOol) next; + tcg_insn_unit *label; /* label pointer to be updated */ + int reloc; /* relocation type from label_ptr */ + intptr_t addend; /* relocation addend from label_ptr */ + uint32_t key; /* oi : is_64 : is_ld */ +} TCGLabelQemuLdstOol; + + +/* + * Generate TB finalization at the end of block + */ + +static tcg_insn_unit *tcg_out_qemu_ldst_ool(TCGContext *s, bool is_ld, + bool is64, TCGMemOpIdx oi); + +static bool tcg_out_ldst_ool_finalize(TCGContext *s) +{ + TCGLabelQemuLdstOol *lb; + + /* qemu_ld/st slow paths */ + QSIMPLEQ_FOREACH(lb, &s->ldst_ool_labels, next) { + gpointer dest, key = (gpointer)(uintptr_t)lb->key; + TCGMemOpIdx oi; + bool is_ld, is_64, ok; + + /* If we have generated the thunk, and it's still in range, all ok. */ + dest = g_hash_table_lookup(s->ldst_ool_thunks, key); + if (dest && + patch_reloc(lb->label, lb->reloc, (intptr_t)dest, lb->addend)) { + continue; + } + + /* Generate a new thunk. */ + is_ld = extract32(lb->key, 0, 1); + is_64 = extract32(lb->key, 1, 1); + oi = extract32(lb->key, 2, 30); + dest = tcg_out_qemu_ldst_ool(s, is_ld, is_64, oi); + + /* Test for (pending) buffer overflow. The assumption is that any + one thunk beginning below the high water mark cannot overrun + the buffer completely. Thus we can test for overflow after + generating code without having to check during generation. */ + if (unlikely((void *)s->code_ptr > s->code_gen_highwater)) { + return false; + } + + /* Remember the thunk for next time. */ + g_hash_table_replace(s->ldst_ool_thunks, key, dest); + s->ldst_ool_generation++; + + /* The new thunk must be in range. */ + ok = patch_reloc(lb->label, lb->reloc, (intptr_t)dest, lb->addend); + tcg_debug_assert(ok); + } + return true; +} + +/* + * Allocate a new TCGLabelQemuLdstOol entry. + */ + +static void add_ldst_ool_label(TCGContext *s, bool is_ld, bool is_64, + TCGMemOpIdx oi, int reloc, intptr_t addend) +{ + TCGLabelQemuLdstOol *lb = tcg_malloc(sizeof(*lb)); + + QSIMPLEQ_INSERT_TAIL(&s->ldst_ool_labels, lb, next); + lb->label = s->code_ptr; + lb->reloc = reloc; + lb->addend = addend; + lb->key = is_ld | (is_64 << 1) | (oi << 2); +} diff --git a/tcg/tcg.c b/tcg/tcg.c index 54f1272187..17c193791f 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -521,6 +521,13 @@ static void tcg_region_assign(TCGContext *s, size_t curr_region) s->code_gen_ptr = start; s->code_gen_buffer_size = end - start; s->code_gen_highwater = end - TCG_HIGHWATER; + +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + /* No thunks yet generated this region. Even if they were in range, + this is also the most convenient place to clear the table after a + full tb_flush. */ + g_hash_table_remove_all(s->ldst_ool_thunks); +#endif } static bool tcg_region_alloc__locked(TCGContext *s) @@ -756,6 +763,14 @@ void tcg_register_thread(void) err = tcg_region_initial_alloc__locked(tcg_ctx); g_assert(!err); qemu_mutex_unlock(®ion.lock); + +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + /* If n == 0, keep the hash table we allocated in tcg_context_init. */ + if (n) { + /* Both key and value are raw pointers. */ + s->ldst_ool_thunks = g_hash_table_new(NULL, NULL); + } +#endif } #endif /* !CONFIG_USER_ONLY */ @@ -964,6 +979,11 @@ void tcg_context_init(TCGContext *s) tcg_debug_assert(!tcg_regset_test_reg(s->reserved_regs, TCG_AREG0)); ts = tcg_global_reg_new_internal(s, TCG_TYPE_PTR, TCG_AREG0, "env"); cpu_env = temp_tcgv_ptr(ts); + +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + /* Both key and value are raw pointers. */ + s->ldst_ool_thunks = g_hash_table_new(NULL, NULL); +#endif } /* @@ -3540,6 +3560,9 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb) #ifdef TCG_TARGET_NEED_LDST_LABELS QSIMPLEQ_INIT(&s->ldst_labels); #endif +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + QSIMPLEQ_INIT(&s->ldst_ool_labels); +#endif #ifdef TCG_TARGET_NEED_POOL_LABELS s->pool_labels = NULL; #endif @@ -3620,6 +3643,11 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb) return -1; } #endif +#ifdef TCG_TARGET_NEED_LDST_OOL_LABELS + if (!tcg_out_ldst_ool_finalize(s)) { + return -1; + } +#endif #ifdef TCG_TARGET_NEED_POOL_LABELS if (!tcg_out_pool_finalize(s)) { return -1;