From patchwork Sat Feb 4 16:32:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 650648 Delivered-To: patch@linaro.org Received: by 2002:a17:522:d8c:b0:4be:c3dc:14d8 with SMTP id d12csp1446568pva; Sat, 4 Feb 2023 08:35:53 -0800 (PST) X-Google-Smtp-Source: AK7set9PTVHL6rFP9d0bu40LFtBsqD3tPRRg0JKaKZP+Fl7lFctE1oKn6Yjxt8gSzDl2ygZ7BcKf X-Received: by 2002:a05:622a:48e:b0:3b8:6215:789b with SMTP id p14-20020a05622a048e00b003b86215789bmr25333214qtx.35.1675528553156; Sat, 04 Feb 2023 08:35:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675528553; cv=none; d=google.com; s=arc-20160816; b=BOYrrTE1Q4/tMylXvOfnGNtF9J8lnqkNWAtbAyWVfGNfAhoRrEZpsVJgODqsg70+kl 2gwNq+cR6V8zponsiEykLU3Us8rC4UCjtMdo2+wESp3SwU8FXGDRGe8ih7PcIw3H/CrN AJL2hFQ7Zq0flCOAu/i9qLI438I8vFbh8Uzk/dl/nSXahoEhYrUeoPq6kHOxGJXp91sm Vfy0Pei1w4KFdKkg3pxW3xniqcqnQhDNxFnwywcDOObFM2EbMHMvzeHfEuEbTehbB0eH QJhjIQJb5iCDJDIWwciQwWtxdVqP1pbdBk835U3rSMOxZOKwlJgCVtRDrnXyDLsrs99l B2WQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=s0umLfjfiPmWFnFtaxL4TolrCcW7qLWhU/FoBD1NbWA=; b=iYyH5EBYK8z6LZ24Go0Xvv44EINHxJ3KLZ/S/TRuhabm2XYc2D/q6y5+hX1SNqT6SX c7rIWR6nK55n0BgZMobh6XhlOgyVscim8dRO8nnOBUuY1mVUYQcOZZ4kcA3I6fykipvW xmBOyMQ6BRE2SkWrXzs1Xd/ZaCzjn5qHBdSZumJzvez0nL+sYXTA9N+ZyUXdWPMcA3e6 LbQ5GE25gc8RXFtyiAXuqsVI/Qp1ODauKB706/dUVwPDU97KIOf/Q4BiCXQrfk0tALev zUdMphD7cvO5Fq1TRPB/IyER3FgfSIJfcOSjdNa3atnJt2yLYlrmo6s3crfXkCFKaWBr y0kQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=C6+25y9m; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id 2-20020ac85702000000b003b9b25060c7si4296415qtw.265.2023.02.04.08.35.53 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sat, 04 Feb 2023 08:35:53 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=C6+25y9m; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pOLU4-00058Y-9j; Sat, 04 Feb 2023 11:33:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOLU2-00057S-Bn for qemu-devel@nongnu.org; Sat, 04 Feb 2023 11:33:46 -0500 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pOLTz-0002U9-7H for qemu-devel@nongnu.org; Sat, 04 Feb 2023 11:33:46 -0500 Received: by mail-pl1-x635.google.com with SMTP id v23so8176816plo.1 for ; Sat, 04 Feb 2023 08:33:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s0umLfjfiPmWFnFtaxL4TolrCcW7qLWhU/FoBD1NbWA=; b=C6+25y9mCCwGcm9yFUf8W00yK8rMIi09dyYBM3Lbx1jNMxDrlciZiTdlhvXPnGcntE CTr3+GlU0PiZLTYKTHMCePFYkBiYi8LlH+HrltKrQ9niuP8B7vFZMwbCxV7zJPDk93Lq TaOTocwRfSQfQgzxoUEBXDk1RIELMCUNSCux9a4YFOb8a5bxhylkxr2X1iN0azAFhy2D QZYTBd6JbpMwlVNQpccssg8412UjLNjuQxoXa0YVzzJWu8M7z+qBRH+mVOSSLlwXFn0n kQ7JteuAyXWesk38s6PCAQwG1tkFpP4ZSba6r0fHwzk4b8Q9Te5Z/P1kJDl/qK6po8VI hmqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s0umLfjfiPmWFnFtaxL4TolrCcW7qLWhU/FoBD1NbWA=; b=hDGgWOJdMkesSaLewUcfm4AKGag0nA12PsH9/XKBoErCUaLLFI45N6dbBZUEI+pv1D tLH9Gitm36nS5nWP0Xec7/LsRmAvCovwNn47B/7/RrwoUFMAkr04JwnIlnV/IgXoKReT b0DKr15A6Tw7gwaqKl8i616uoGp7D1bVhfItSsmVjg2iCT+5Hmn48mzFd4cm1a6RDQBj L0r+UlyzCrMoz6l7E42tLqiG4/Slyl81I2S8nl1cBAmF8aiNXHB32gZ5Q1KO0ZXqCuDB ag3lWpqUaopa/NZuaB7dfgQTSJJffrLcr5r0Zf4oshnUVueG/q4C3eaIAH/wM+LmMGY8 5yfg== X-Gm-Message-State: AO0yUKVRjpSA+yYO5/2TJOeitsXqEK8ui++8HWz+ubO3YAhnPm6q61Wq ICBlHpJ0TBFlFBuPpv9fVdLDi3U4g9nu4J/i X-Received: by 2002:a17:902:d405:b0:198:ecd6:e2d5 with SMTP id b5-20020a170902d40500b00198ecd6e2d5mr2838822ple.9.1675528422294; Sat, 04 Feb 2023 08:33:42 -0800 (PST) Received: from stoup.. (rrcs-173-197-98-118.west.biz.rr.com. [173.197.98.118]) by smtp.gmail.com with ESMTPSA id w19-20020a1709029a9300b00186b69157ecsm3660859plp.202.2023.02.04.08.33.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 04 Feb 2023 08:33:41 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Cc: peter.maydell@linaro.org, =?utf-8?q?Alex_Benn=C3=A9e?= , =?utf-8?q?Philippe_M?= =?utf-8?q?athieu-Daud=C3=A9?= Subject: [PULL 18/40] tcg: Add tcg_gen_{non}atomic_cmpxchg_i128 Date: Sat, 4 Feb 2023 06:32:48 -1000 Message-Id: <20230204163310.815536-19-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230204163310.815536-1-richard.henderson@linaro.org> References: <20230204163310.815536-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::635; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org This will allow targets to avoid rolling their own. Reviewed-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Richard Henderson --- accel/tcg/tcg-runtime.h | 11 +++++ include/tcg/tcg-op.h | 5 +++ tcg/tcg-op.c | 85 +++++++++++++++++++++++++++++++++++ accel/tcg/atomic_common.c.inc | 45 +++++++++++++++++++ 4 files changed, 146 insertions(+) diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h index 37cbd722bf..e141a6ab24 100644 --- a/accel/tcg/tcg-runtime.h +++ b/accel/tcg/tcg-runtime.h @@ -55,6 +55,17 @@ DEF_HELPER_FLAGS_5(atomic_cmpxchgq_be, TCG_CALL_NO_WG, DEF_HELPER_FLAGS_5(atomic_cmpxchgq_le, TCG_CALL_NO_WG, i64, env, tl, i64, i64, i32) #endif +#ifdef CONFIG_CMPXCHG128 +DEF_HELPER_FLAGS_5(atomic_cmpxchgo_be, TCG_CALL_NO_WG, + i128, env, tl, i128, i128, i32) +DEF_HELPER_FLAGS_5(atomic_cmpxchgo_le, TCG_CALL_NO_WG, + i128, env, tl, i128, i128, i32) +#endif + +DEF_HELPER_FLAGS_5(nonatomic_cmpxchgo_be, TCG_CALL_NO_WG, + i128, env, tl, i128, i128, i32) +DEF_HELPER_FLAGS_5(nonatomic_cmpxchgo_le, TCG_CALL_NO_WG, + i128, env, tl, i128, i128, i32) #ifdef CONFIG_ATOMIC64 #define GEN_ATOMIC_HELPERS(NAME) \ diff --git a/include/tcg/tcg-op.h b/include/tcg/tcg-op.h index e5f5b63c37..31bf3d287e 100644 --- a/include/tcg/tcg-op.h +++ b/include/tcg/tcg-op.h @@ -907,6 +907,11 @@ void tcg_gen_atomic_cmpxchg_i32(TCGv_i32, TCGv, TCGv_i32, TCGv_i32, TCGArg, MemOp); void tcg_gen_atomic_cmpxchg_i64(TCGv_i64, TCGv, TCGv_i64, TCGv_i64, TCGArg, MemOp); +void tcg_gen_atomic_cmpxchg_i128(TCGv_i128, TCGv, TCGv_i128, TCGv_i128, + TCGArg, MemOp); + +void tcg_gen_nonatomic_cmpxchg_i128(TCGv_i128, TCGv, TCGv_i128, TCGv_i128, + TCGArg, MemOp); void tcg_gen_atomic_xchg_i32(TCGv_i32, TCGv, TCGv_i32, TCGArg, MemOp); void tcg_gen_atomic_xchg_i64(TCGv_i64, TCGv, TCGv_i64, TCGArg, MemOp); diff --git a/tcg/tcg-op.c b/tcg/tcg-op.c index 33ef325f6e..5811ecd3e7 100644 --- a/tcg/tcg-op.c +++ b/tcg/tcg-op.c @@ -3295,6 +3295,8 @@ typedef void (*gen_atomic_cx_i32)(TCGv_i32, TCGv_env, TCGv, TCGv_i32, TCGv_i32, TCGv_i32); typedef void (*gen_atomic_cx_i64)(TCGv_i64, TCGv_env, TCGv, TCGv_i64, TCGv_i64, TCGv_i32); +typedef void (*gen_atomic_cx_i128)(TCGv_i128, TCGv_env, TCGv, + TCGv_i128, TCGv_i128, TCGv_i32); typedef void (*gen_atomic_op_i32)(TCGv_i32, TCGv_env, TCGv, TCGv_i32, TCGv_i32); typedef void (*gen_atomic_op_i64)(TCGv_i64, TCGv_env, TCGv, @@ -3305,6 +3307,11 @@ typedef void (*gen_atomic_op_i64)(TCGv_i64, TCGv_env, TCGv, #else # define WITH_ATOMIC64(X) #endif +#ifdef CONFIG_CMPXCHG128 +# define WITH_ATOMIC128(X) X, +#else +# define WITH_ATOMIC128(X) +#endif static void * const table_cmpxchg[(MO_SIZE | MO_BSWAP) + 1] = { [MO_8] = gen_helper_atomic_cmpxchgb, @@ -3314,6 +3321,8 @@ static void * const table_cmpxchg[(MO_SIZE | MO_BSWAP) + 1] = { [MO_32 | MO_BE] = gen_helper_atomic_cmpxchgl_be, WITH_ATOMIC64([MO_64 | MO_LE] = gen_helper_atomic_cmpxchgq_le) WITH_ATOMIC64([MO_64 | MO_BE] = gen_helper_atomic_cmpxchgq_be) + WITH_ATOMIC128([MO_128 | MO_LE] = gen_helper_atomic_cmpxchgo_le) + WITH_ATOMIC128([MO_128 | MO_BE] = gen_helper_atomic_cmpxchgo_be) }; void tcg_gen_atomic_cmpxchg_i32(TCGv_i32 retv, TCGv addr, TCGv_i32 cmpv, @@ -3412,6 +3421,82 @@ void tcg_gen_atomic_cmpxchg_i64(TCGv_i64 retv, TCGv addr, TCGv_i64 cmpv, } } +void tcg_gen_nonatomic_cmpxchg_i128(TCGv_i128 retv, TCGv addr, TCGv_i128 cmpv, + TCGv_i128 newv, TCGArg idx, MemOp memop) +{ + if (TCG_TARGET_REG_BITS == 32) { + /* Inline expansion below is simply too large for 32-bit hosts. */ + gen_atomic_cx_i128 gen = ((memop & MO_BSWAP) == MO_LE + ? gen_helper_nonatomic_cmpxchgo_le + : gen_helper_nonatomic_cmpxchgo_be); + MemOpIdx oi = make_memop_idx(memop, idx); + + tcg_debug_assert((memop & MO_SIZE) == MO_128); + tcg_debug_assert((memop & MO_SIGN) == 0); + + gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi)); + } else { + TCGv_i128 oldv = tcg_temp_new_i128(); + TCGv_i128 tmpv = tcg_temp_new_i128(); + TCGv_i64 t0 = tcg_temp_new_i64(); + TCGv_i64 t1 = tcg_temp_new_i64(); + TCGv_i64 z = tcg_constant_i64(0); + + tcg_gen_qemu_ld_i128(oldv, addr, idx, memop); + + /* Compare i128 */ + tcg_gen_xor_i64(t0, TCGV128_LOW(oldv), TCGV128_LOW(cmpv)); + tcg_gen_xor_i64(t1, TCGV128_HIGH(oldv), TCGV128_HIGH(cmpv)); + tcg_gen_or_i64(t0, t0, t1); + + /* tmpv = equal ? newv : oldv */ + tcg_gen_movcond_i64(TCG_COND_EQ, TCGV128_LOW(tmpv), t0, z, + TCGV128_LOW(newv), TCGV128_LOW(oldv)); + tcg_gen_movcond_i64(TCG_COND_EQ, TCGV128_HIGH(tmpv), t0, z, + TCGV128_HIGH(newv), TCGV128_HIGH(oldv)); + + /* Unconditional writeback. */ + tcg_gen_qemu_st_i128(tmpv, addr, idx, memop); + tcg_gen_mov_i128(retv, oldv); + + tcg_temp_free_i64(t0); + tcg_temp_free_i64(t1); + tcg_temp_free_i128(tmpv); + tcg_temp_free_i128(oldv); + } +} + +void tcg_gen_atomic_cmpxchg_i128(TCGv_i128 retv, TCGv addr, TCGv_i128 cmpv, + TCGv_i128 newv, TCGArg idx, MemOp memop) +{ + gen_atomic_cx_i128 gen; + + if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) { + tcg_gen_nonatomic_cmpxchg_i128(retv, addr, cmpv, newv, idx, memop); + return; + } + + tcg_debug_assert((memop & MO_SIZE) == MO_128); + tcg_debug_assert((memop & MO_SIGN) == 0); + gen = table_cmpxchg[memop & (MO_SIZE | MO_BSWAP)]; + + if (gen) { + MemOpIdx oi = make_memop_idx(memop, idx); + gen(retv, cpu_env, addr, cmpv, newv, tcg_constant_i32(oi)); + return; + } + + gen_helper_exit_atomic(cpu_env); + + /* + * Produce a result for a well-formed opcode stream. This satisfies + * liveness for set before used, which happens before this dead code + * is removed. + */ + tcg_gen_movi_i64(TCGV128_LOW(retv), 0); + tcg_gen_movi_i64(TCGV128_HIGH(retv), 0); +} + static void do_nonatomic_op_i32(TCGv_i32 ret, TCGv addr, TCGv_i32 val, TCGArg idx, MemOp memop, bool new_val, void (*gen)(TCGv_i32, TCGv_i32, TCGv_i32)) diff --git a/accel/tcg/atomic_common.c.inc b/accel/tcg/atomic_common.c.inc index 6602d7689f..8f2ce43ee6 100644 --- a/accel/tcg/atomic_common.c.inc +++ b/accel/tcg/atomic_common.c.inc @@ -55,8 +55,53 @@ CMPXCHG_HELPER(cmpxchgq_be, uint64_t) CMPXCHG_HELPER(cmpxchgq_le, uint64_t) #endif +#ifdef CONFIG_CMPXCHG128 +CMPXCHG_HELPER(cmpxchgo_be, Int128) +CMPXCHG_HELPER(cmpxchgo_le, Int128) +#endif + #undef CMPXCHG_HELPER +Int128 HELPER(nonatomic_cmpxchgo_be)(CPUArchState *env, target_ulong addr, + Int128 cmpv, Int128 newv, uint32_t oi) +{ +#if TCG_TARGET_REG_BITS == 32 + uintptr_t ra = GETPC(); + Int128 oldv; + + oldv = cpu_ld16_be_mmu(env, addr, oi, ra); + if (int128_eq(oldv, cmpv)) { + cpu_st16_be_mmu(env, addr, newv, oi, ra); + } else { + /* Even with comparison failure, still need a write cycle. */ + probe_write(env, addr, 16, get_mmuidx(oi), ra); + } + return oldv; +#else + g_assert_not_reached(); +#endif +} + +Int128 HELPER(nonatomic_cmpxchgo_le)(CPUArchState *env, target_ulong addr, + Int128 cmpv, Int128 newv, uint32_t oi) +{ +#if TCG_TARGET_REG_BITS == 32 + uintptr_t ra = GETPC(); + Int128 oldv; + + oldv = cpu_ld16_le_mmu(env, addr, oi, ra); + if (int128_eq(oldv, cmpv)) { + cpu_st16_le_mmu(env, addr, newv, oi, ra); + } else { + /* Even with comparison failure, still need a write cycle. */ + probe_write(env, addr, 16, get_mmuidx(oi), ra); + } + return oldv; +#else + g_assert_not_reached(); +#endif +} + #define ATOMIC_HELPER(OP, TYPE) \ TYPE HELPER(glue(atomic_,OP))(CPUArchState *env, target_ulong addr, \ TYPE val, uint32_t oi) \