[v2,000/101] target/arm: Implement FEAT_SME2p1

v1: 20250206195715.2150758-1-richard.henderson@linaro.org

This implements the Scalar Matrix Extensions, version 2.1.

Version 1 only implemented FEAT_SME2, but it turns out that
all of the example code expects FEAT_SVE2p1 as well.
Since the presence of both SME2 and SVE2p1 requires SME2p1,
I've gone all the way to a point of consistency.

SME2 is the first vector-like extension we've had that has dynamic
indexing of registers: ZArray[(rv + offset) % svl], where RV is a
general register.  So the first thing I do is extend TCG's gvec
support to handle TCGv_ptr base + offset addressing.  I only changed
enough to handle what I needed within SME2; changing it all would be
a big job, and it would (at least for the moment) remain unused.

This has been only lightly tested, with the testcase in the
final patch.  I intend to do more testing next week, but as
there is significant interest, I wanted to get this out earlier.


r~


Richard Henderson (101):
  tcg: Add dbase argument to do_dup_store
  tcg: Add dbase argument to do_dup
  tcg: Add dbase argument to expand_clr
  tcg: Add base arguments to check_overlap_[234]
  tcg: Split out tcg_gen_gvec_2_var
  tcg: Split out tcg_gen_gvec_3_var
  tcg: Split out tcg_gen_gvec_mov_var
  tcg: Split out tcg_gen_gvec_{add,sub}_var
  tcg: Split out tcg_gen_gvec_dup_imm_var
  linux-user/aarch64: Update hwcap bits from 6.14
  target/arm: Remove CPUARMState.vfp.scratch
  target/arm: Introduce FPST_ZA, FPST_ZA_F16
  target/arm: Use FPST_ZA for sme_fmopa_[hsd]
  target/arm: Rename zarray to za_state.za
  target/arm: Add isar feature tests for SME2, SVE2p1
  target/arm: Add ZT0
  target/arm: Add zt0_excp_el to DisasContext
  target/arm: Implement SME2 ZERO ZT0
  target/arm: Implement SME2 LDR/STR ZT0
  target/arm: Implement SME2 MOVT
  target/arm: Split get_tile_rowcol argument tile_index
  target/arm: Rename MOVA for translate
  target/arm: Implement SME2 MOVA to/from tile, multiple registers
  target/arm: Split out get_zarray
  target/arm: Implement SME2 MOVA to/from array, multiple registers
  target/arm: Implement SME2 BMOPA
  target/arm: Implement SME2 SMOPS, UMOPS (2-way)
  target/arm: Introduce gen_gvec_sve2_sqdmulh
  target/arm: Implement SME2 Multiple and Single SVE Destructive
  target/arm: Implement SME2 Multiple Vectors SVE Destructive
  target/arm: Implement SME2 ADD/SUB (array results, multiple and single
    vector)
  target/arm: Implement SME2 ADD/SUB (array results, multiple vectors)
  target/arm: Pass ZA to helper_sve2_fmlal_zz[zx]w_s
  target/arm: Implement SME2 FMLAL, BFMLAL
  target/arm: Implement SME2 FDOT
  target/arm: Implement SME2 BFDOT
  target/arm: Implement SME2 FVDOT, BFVDOT
  target/arm: Rename helper_gvec_*dot_[bh] to *_4[bh]
  target/arm: Remove helper_gvec_sudot_idx_4b
  target/arm: Implemement SME2 SDOT, UDOT, USDOT, SUDOT
  target/arm: Rename SVE SDOT and UDOT patterns
  target/arm: Tighten USDOT (vectors) decode
  target/arm: Implement SDOT, UDOT (2-way) for SME2/SVE2p1
  target/arm: Implement SME2 SVDOT, UVDOT, SUVDOT, USVDOT
  target/arm: Implement SME2 SMLAL, SMLSL, UMLAL, UMLSL
  target/arm: Implement SME2 SMLALL, SMLSLL, UMLALL, UMLSLL
  target/arm: Rename gvec_fml[as]_[hs] with _nf_ infix
  target/arm: Implement SME2 FMLA, FMLS
  target/arm: Implement SME2 BFMLA, BFMLS
  target/arm: Implement SME2 FADD, FSUB, BFADD, BFSUB
  target/arm: Implement SME2 BFCVT, BFCVTN, FCVT, FCVTN
  target/arm: Implement SME2 FCVT (widening), FCVTL
  target/arm: Implement SME2 FCVTZS, FCVTZU
  target/arm: Implement SME2 SCVTF, UCVTF
  target/arm: Implement SME2 FRINTN, FRINTP, FRINTM, FRINTA
  target/arm: Introduce do_[us]sat_[bhs] macros
  target/arm: Use do_[us]sat_[bhs] in sve_helper.c
  target/arm: Implement SME2 SQCVT, UQCVT, SQCVTU
  target/arm: Implement SQCVTN, UQCVTN, SQCVTUN for SME2/SVE2p1
  target/arm: Implement SME2 SUNPK, UUNPK
  target/arm: Implement SME2 ZIP, UZP (four registers)
  target/arm: Move do_urshr, do_srshr to vec_internal.h
  target/arm: Implement SME2 SQRSHR, UQRSHR, SQRSHRN
  target/arm: Implement SME2 ZIP, UZP (two registers)
  target/arm: Implement SME2 FCLAMP, SCLAMP, UCLAMP
  target/arm: Enable SCLAMP, UCLAMP for SVE2p1
  target/arm: Implement FCLAMP for SME2, SVE2p1
  target/arm: Implement SME2 SEL
  target/arm: Implement SME2p1 Multiple Zero
  target/arm: Introduce pred_count_test
  target/arm: Fold predtest_ones into helper_sve_brkns
  target/arm: Split out do_whilel from helper_sve_whilel
  target/arm: Split out do_whileg from helper_sve_whileg
  target/arm: Move scale by esz into helper_sve_while*
  target/arm: Split trans_WHILE to lt and gt
  target/arm: Implement SVE2p1 WHILE (predicate pair)
  target/arm: Implement SVE2p1 WHILE (predicate as counter)
  target/arm: Implement SVE2p1 PTRUE (predicate as counter)
  target/arm: Enable PSEL for SVE2p1
  target/arm: Implement {ADD,SMIN,SMAX,UMIN,UMAX}QV for SVE2p1
  target/arm: Implement SVE2p1 PEXT
  target/arm: Implement ANDQV, ORQV, EORQV for SVE2p1
  target/arm: Implement FADDQV, F{MIN,MAX}{NM}QV for SVE2p1
  target/arm: Implement BFMLSLB{L,T} for SME2/SVE2p1
  target/arm: Implement CNTP (predicate as counter) for SME2/SVE2p1
  target/arm: Implement DUPQ for SME2p1/SVE2p1
  target/arm: Implement EXTQ for SME2p1/SVE2p1
  target/arm: Implement PMOV for SME2p1/SVE2p1
  target/arm: Implement ZIPQ, UZPQ for SME2p1/SVE2p1
  target/arm: Implement TBLQ, TBXQ for SME2p1/SVE2p1
  target/arm: Implement SME2 counted predicate register load/store
  target/arm: Split the ST_zpri and ST_zprr patterns
  target/arm: Implement {LD1,ST1}{W,D} (128-bit element) for SVE2p1
  target/arm: Move ld1qq and st1qq primitives to sve_ldst_internal.h
  target/arm: Implement {LD,ST}[234]Q for SME2p1/SVE2p1
  target/arm: Implement LD1Q, ST1Q for SVE2p1
  target/arm: Implement LUTI2, LUTI4 for SME2/SME2p1
  target/arm: Implement MOVAZ for SME2p1
  linux-user/aarch64: Set hwcap bits for SME2p1/SVE2p1
  target/arm: Enable FEAT_SME2p1 on -cpu max
  tests/tcg/aarch64: Add sme2-matmul test case

 include/tcg/tcg-op-gvec-common.h   |   23 +
 target/arm/cpu-features.h          |   55 +
 target/arm/cpu.h                   |   68 +-
 target/arm/syndrome.h              |    1 +
 target/arm/tcg/helper-sme.h        |  198 ++++
 target/arm/tcg/helper-sve.h        |  212 ++++
 target/arm/tcg/helper.h            |   57 +-
 target/arm/tcg/sve_ldst_internal.h |   89 ++
 target/arm/tcg/translate-a64.h     |    4 +
 target/arm/tcg/translate.h         |   11 +
 target/arm/tcg/vec_internal.h      |   47 +
 linux-user/aarch64/signal.c        |    4 +-
 linux-user/elfload.c               |   83 +-
 target/arm/cpu.c                   |   11 +-
 target/arm/helper.c                |    2 +-
 target/arm/machine.c               |   22 +-
 target/arm/tcg/cpu64.c             |   10 +-
 target/arm/tcg/gengvec.c           |    6 +
 target/arm/tcg/gengvec64.c         |   11 +
 target/arm/tcg/helper-a64.c        |    2 +
 target/arm/tcg/hflags.c            |   34 +-
 target/arm/tcg/mve_helper.c        |   21 -
 target/arm/tcg/sme_helper.c        | 1081 +++++++++++++++++--
 target/arm/tcg/sve_helper.c        | 1568 ++++++++++++++++++++++++----
 target/arm/tcg/translate-a64.c     |   15 +-
 target/arm/tcg/translate-neon.c    |   18 +-
 target/arm/tcg/translate-sme.c     | 1359 +++++++++++++++++++++++-
 target/arm/tcg/translate-sve.c     |  895 ++++++++++++++--
 target/arm/tcg/vec_helper.c        |  218 +++-
 target/arm/tcg/vfp_helper.c        |   10 +
 tcg/tcg-op-gvec.c                  |  371 ++++---
 tests/tcg/aarch64/sme2-matmul-0.c  |  236 +++++
 docs/system/arm/emulation.rst      |    6 +
 target/arm/tcg/sme.decode          |  928 +++++++++++++++-
 target/arm/tcg/sve.decode          |  281 ++++-
 tests/tcg/aarch64/Makefile.target  |   11 +-
 tests/tcg/aarch64/sme2-matmul-1.S  |  321 ++++++
 37 files changed, 7598 insertions(+), 691 deletions(-)
 create mode 100644 tests/tcg/aarch64/sme2-matmul-0.c
 create mode 100644 tests/tcg/aarch64/sme2-matmul-1.S