From patchwork Thu Oct 27 15:10:22 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 79734 Delivered-To: patch@linaro.org Received: by 10.80.142.83 with SMTP id 19csp718029edx; Thu, 27 Oct 2016 08:47:48 -0700 (PDT) X-Received: by 10.31.65.195 with SMTP id o186mr6354666vka.3.1477583268628; Thu, 27 Oct 2016 08:47:48 -0700 (PDT) Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id h191si3584166vka.148.2016.10.27.08.47.48 for (version=TLS1 cipher=AES128-SHA bits=128/128); Thu, 27 Oct 2016 08:47:48 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:42408 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bzmuB-0000zc-NL for patch@linaro.org; Thu, 27 Oct 2016 11:47:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45446) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bzmTL-0003hf-Cp for qemu-devel@nongnu.org; Thu, 27 Oct 2016 11:20:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bzmTJ-0004Rz-9h for qemu-devel@nongnu.org; Thu, 27 Oct 2016 11:20:03 -0400 Received: from mail-wm0-x22f.google.com ([2a00:1450:400c:c09::22f]:38354) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1bzmTI-0004Qv-Uy for qemu-devel@nongnu.org; Thu, 27 Oct 2016 11:20:01 -0400 Received: by mail-wm0-x22f.google.com with SMTP id n67so51136426wme.1 for ; Thu, 27 Oct 2016 08:20:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=d4fLEurrsmPDu7Ylak8tjyk5NCvHaOXeGE/9ix0939U=; b=UXZ3ZJWwQaDwKBma35LRuAsP81ITU1s+NuduTnjm54FQZjc6GXP+L1fYzu8nj25rpI mIOxqSqkD+TVFbMhv3wg022sNh3Ily6El/IqMYujdTdhkBIXQjGll+oyJM2/+Ox8uyRK DIHE+OueQPYtLFUQGseH5TsuXe7a4nVsDbenA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d4fLEurrsmPDu7Ylak8tjyk5NCvHaOXeGE/9ix0939U=; b=Lf6K5ay9fEECxhNTwH/JjUY8ZgCTBODdzNOgs4N2aEtltETaVI56JyceMiqxLnYRtv BYQsdkrXNIZdUYpS9J//qDKgaqNaqvpdwpFwOuDMD9liaq3di4DJTxitnXkv4Rl+zLvM EyzPkrLhfmKodESYc7oyU8kTP85EPXqu20b8C9PN9QpQ4LWmOBgSyptQPID4V/II9hEf aRfC/kgpTx0qStqHTzDnv9XSZfysonbCNnE3wfWrsJV2XSKZz0bf2ChVJ6EQwnMtqx2Q 8gy1G4XEGxOLaGhQqw8axcCtaoph1TbmAiIdUlB5UBZplueyeYHYWlvSlAkuaBXMl7db mWHA== X-Gm-Message-State: ABUngvd6WWixkhg6/h/AYbNbZpCzgK8Ilj1wPwsKxNf3H5JiXw9Gq2XrLX/vtms7fM6f5bZ9 X-Received: by 10.28.158.209 with SMTP id h200mr14727836wme.54.1477581599701; Thu, 27 Oct 2016 08:19:59 -0700 (PDT) Received: from zen.linaro.local ([81.128.185.34]) by smtp.gmail.com with ESMTPSA id a1sm9015305wjl.28.2016.10.27.08.19.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Oct 2016 08:19:58 -0700 (PDT) Received: from zen.linaroharston (localhost [127.0.0.1]) by zen.linaro.local (Postfix) with ESMTP id 193643E0473; Thu, 27 Oct 2016 16:11:01 +0100 (BST) From: =?UTF-8?q?Alex=20Benn=C3=A9e?= To: pbonzini@redhat.com Date: Thu, 27 Oct 2016 16:10:22 +0100 Message-Id: <20161027151030.20863-26-alex.bennee@linaro.org> X-Mailer: git-send-email 2.10.1 In-Reply-To: <20161027151030.20863-1-alex.bennee@linaro.org> References: <20161027151030.20863-1-alex.bennee@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a00:1450:400c:c09::22f Subject: [Qemu-devel] [PATCH v5 25/33] cputlb: introduce tlb_flush_* async work. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mttcg@listserver.greensocs.com, peter.maydell@linaro.org, claudio.fontana@huawei.com, nikunj@linux.vnet.ibm.com, Peter Crosthwaite , jan.kiszka@siemens.com, mark.burton@greensocs.com, a.rigo@virtualopensystems.com, qemu-devel@nongnu.org, cota@braap.org, serge.fdrv@gmail.com, bobby.prani@gmail.com, rth@twiddle.net, =?UTF-8?q?Alex=20Benn=C3=A9e?= , fred.konrad@greensocs.com Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: KONRAD Frederic Some architectures allow to flush the tlb of other VCPUs. This is not a problem when we have only one thread for all VCPUs but it definitely needs to be an asynchronous work when we are in true multithreaded work. We take the tb_lock() when doing this to avoid racing with other threads which may be invalidating TB's at the same time. The alternative would be to use proper atomic primitives to clear the tlb entries en-mass. This patch doesn't do anything to protect other cputlb function being called in MTTCG mode making cross vCPU changes. Signed-off-by: KONRAD Frederic [AJB: remove need for g_malloc on defer, make check fixes, tb_lock] Signed-off-by: Alex Bennée --- v5 (base patches) - take tb_lock() for memset - ensure tb_flush_page properly asyncs work for other vCPUs - use run_on_cpu_data v4 (base_patches) - brought forward from arm enabling series - restore pending_tlb_flush flag v1 - Remove tlb_flush_all just do the check in tlb_flush. - remove the need to g_malloc - tlb_flush calls direct if !cpu->created --- cputlb.c | 94 +++++++++++++++++++++++++++++++++++++++++-------- include/exec/exec-all.h | 1 + include/qom/cpu.h | 6 ++++ 3 files changed, 87 insertions(+), 14 deletions(-) -- 2.10.1 diff --git a/cputlb.c b/cputlb.c index eec1c39..4190d9c 100644 --- a/cputlb.c +++ b/cputlb.c @@ -65,28 +65,35 @@ } \ } while (0) +/* run_on_cpu_data.target_ptr should always be big enough for a + * target_ulong even on 32 bit builds */ +QEMU_BUILD_BUG_ON(sizeof(target_ulong) > sizeof(run_on_cpu_data)); + +static inline run_on_cpu_data target_ptr(target_ulong tptr) +{ + run_on_cpu_data d = { .target_ptr = (vaddr) tptr }; + return d; +} + +static inline run_on_cpu_data host_int(int hint) +{ + run_on_cpu_data d = { .host_int = hint }; + return d; +} + + /* statistics */ int tlb_flush_count; -/* NOTE: - * If flush_global is true (the usual case), flush all tlb entries. - * If flush_global is false, flush (at least) all tlb entries not - * marked global. - * - * Since QEMU doesn't currently implement a global/not-global flag - * for tlb entries, at the moment tlb_flush() will also flush all - * tlb entries in the flush_global == false case. This is OK because - * CPU architectures generally permit an implementation to drop - * entries from the TLB at any time, so flushing more entries than - * required is only an efficiency issue, not a correctness issue. - */ -void tlb_flush(CPUState *cpu, int flush_global) +static void tlb_flush_nocheck(CPUState *cpu, int flush_global) { CPUArchState *env = cpu->env_ptr; assert_cpu_is_self(cpu); tlb_debug("(%d)\n", flush_global); + tb_lock(); + memset(env->tlb_table, -1, sizeof(env->tlb_table)); memset(env->tlb_v_table, -1, sizeof(env->tlb_v_table)); memset(cpu->tb_jmp_cache, 0, sizeof(cpu->tb_jmp_cache)); @@ -95,6 +102,39 @@ void tlb_flush(CPUState *cpu, int flush_global) env->tlb_flush_addr = -1; env->tlb_flush_mask = 0; tlb_flush_count++; + + tb_unlock(); + + atomic_mb_set(&cpu->pending_tlb_flush, false); +} + +static void tlb_flush_global_async_work(CPUState *cpu, run_on_cpu_data data) +{ + tlb_flush_nocheck(cpu, data.host_int); +} + +/* NOTE: + * If flush_global is true (the usual case), flush all tlb entries. + * If flush_global is false, flush (at least) all tlb entries not + * marked global. + * + * Since QEMU doesn't currently implement a global/not-global flag + * for tlb entries, at the moment tlb_flush() will also flush all + * tlb entries in the flush_global == false case. This is OK because + * CPU architectures generally permit an implementation to drop + * entries from the TLB at any time, so flushing more entries than + * required is only an efficiency issue, not a correctness issue. + */ +void tlb_flush(CPUState *cpu, int flush_global) +{ + if (cpu->created && !qemu_cpu_is_self(cpu)) { + if (atomic_bool_cmpxchg(&cpu->pending_tlb_flush, false, true)) { + async_run_on_cpu(cpu, tlb_flush_global_async_work, + host_int(flush_global)); + } + } else { + tlb_flush_nocheck(cpu, flush_global); + } } static inline void v_tlb_flush_by_mmuidx(CPUState *cpu, va_list argp) @@ -104,6 +144,8 @@ static inline void v_tlb_flush_by_mmuidx(CPUState *cpu, va_list argp) assert_cpu_is_self(cpu); tlb_debug("start\n"); + tb_lock(); + for (;;) { int mmu_idx = va_arg(argp, int); @@ -118,6 +160,8 @@ static inline void v_tlb_flush_by_mmuidx(CPUState *cpu, va_list argp) } memset(cpu->tb_jmp_cache, 0, sizeof(cpu->tb_jmp_cache)); + + tb_unlock(); } void tlb_flush_by_mmuidx(CPUState *cpu, ...) @@ -140,13 +184,15 @@ static inline void tlb_flush_entry(CPUTLBEntry *tlb_entry, target_ulong addr) } } -void tlb_flush_page(CPUState *cpu, target_ulong addr) +static void tlb_flush_page_async_work(CPUState *cpu, run_on_cpu_data data) { CPUArchState *env = cpu->env_ptr; + target_ulong addr = (target_ulong) data.target_ptr; int i; int mmu_idx; assert_cpu_is_self(cpu); + tlb_debug("page :" TARGET_FMT_lx "\n", addr); /* Check if we need to flush due to large pages. */ @@ -176,6 +222,17 @@ void tlb_flush_page(CPUState *cpu, target_ulong addr) tb_flush_jmp_cache(cpu, addr); } +void tlb_flush_page(CPUState *cpu, target_ulong addr) +{ + tlb_debug("page :" TARGET_FMT_lx "\n", addr); + + if (!qemu_cpu_is_self(cpu)) { + async_run_on_cpu(cpu, tlb_flush_page_async_work, target_ptr(addr)); + } else { + tlb_flush_page_async_work(cpu, target_ptr(addr)); + } +} + void tlb_flush_page_by_mmuidx(CPUState *cpu, target_ulong addr, ...) { CPUArchState *env = cpu->env_ptr; @@ -222,6 +279,15 @@ void tlb_flush_page_by_mmuidx(CPUState *cpu, target_ulong addr, ...) tb_flush_jmp_cache(cpu, addr); } +void tlb_flush_page_all(target_ulong addr) +{ + CPUState *cpu; + + CPU_FOREACH(cpu) { + async_run_on_cpu(cpu, tlb_flush_page_async_work, target_ptr(addr)); + } +} + /* update the TLBs so that writes to code in the virtual page 'addr' can be detected */ void tlb_protect_code(ram_addr_t ram_addr) diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index d097515..5483f5f 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -161,6 +161,7 @@ void tlb_set_page(CPUState *cpu, target_ulong vaddr, void tb_invalidate_phys_addr(AddressSpace *as, hwaddr addr); void probe_write(CPUArchState *env, target_ulong addr, int mmu_idx, uintptr_t retaddr); +void tlb_flush_page_all(target_ulong addr); #else static inline void tlb_flush_page(CPUState *cpu, target_ulong addr) { diff --git a/include/qom/cpu.h b/include/qom/cpu.h index d8e6702..1fe5b99 100644 --- a/include/qom/cpu.h +++ b/include/qom/cpu.h @@ -396,6 +396,12 @@ struct CPUState { (absolute value) offset as small as possible. This reduces code size, especially for hosts without large memory offsets. */ uint32_t tcg_exit_req; + + /* The pending_tlb_flush flag is set and cleared atomically to + * avoid potential races. The aim of the flag is to avoid + * unnecessary flushes. + */ + bool pending_tlb_flush; }; QTAILQ_HEAD(CPUTailQ, CPUState);