From patchwork Wed Apr 30 11:07:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886563 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C798423A9BF; Wed, 30 Apr 2025 11:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012410; cv=none; b=HiHtW27T2OGyey6Hx7+yFT/+qF+a3sFZMxof/CXn+J/fmUPZ8PqOosMsKoAqFQQL3R2TOCh7BgnyUa0L5suSB3ctNbqKbDkcetFDP71610Ydlumh7wojTAW8Gqmy+zyfLprl1KlSh6pRbwbNRxJXFZieSFV1k2ACZjHMgzIu3vw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012410; c=relaxed/simple; bh=pcCiP5FTA+Lm/bobq6YiOZWGbXIV/H7h3fpmxDOHmBo=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=ag7+OWwllduhyuEZoPTjB5/aiNp9uyD7SCZ2e3YMolXA+cRr3jFRSP4S9qbmsek8Au2GrpbAkm3Aglajk8LqlyiSasfslF5EHuqzfxJEp/mB1Ywcd8mcwEiH6PZd8Sy+NWiaT4fQHzUXKZDgpFMUohSULhkr02YvXduxytMojAk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=VMuR0wpU; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="VMuR0wpU" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Af6nUOSfZkjof68kUCgJ+wjJzCjJKMFbRfwFCJ9l/cg=; b=VMuR0wpUdRiBcwgvlB5GE+AwLe nwXKGBftj9AYTH9hnm0ya7mDbWLIKByzGMB8HKwCS+Eb/dYwSvX+qpTnuk5ZiVmoBrvYreJKv/f6I gDn9T8j4v/OYU/Hv8GK7pNZ2gegw5fc94au5Rr9SFFVchMD4VcA9vRCTjs+voqDEZ85MmxGFcdqwX /3Wi8HTReX62hvDL70gY/KVQtSscGxZOhESHqrJL1y32ryoWpJa5tIRaoX4sYNttRIGNYh1WxBKGE kBuM2cPdyS4nd7/IaZDpCEVIfl8hT9q/3vJRdqfDcMpF/HynNm+Z9hxA14ED/ej8v9cGxdbBwMsqs 0u5G+diA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.1 #2 (Red Hat Linux)) id 1uA5aG-0000000Dm7n-17Ez; Wed, 30 Apr 2025 11:26:36 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id CAF8B30072F; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112349.089758096@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:35 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 01/13] x86/kvm/emulate: Implement test_cc() in C References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Current test_cc() uses the fastop infrastructure to test flags using SETcc instructions. However, int3_emulate_jcc() already fully implements the flags->CC mapping, use that. Removes a pile of gnarly asm. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/text-patching.h | 20 +++++++++++++------- arch/x86/kvm/emulate.c | 34 ++-------------------------------- 2 files changed, 15 insertions(+), 39 deletions(-) --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -177,9 +177,9 @@ void int3_emulate_ret(struct pt_regs *re } static __always_inline -void int3_emulate_jcc(struct pt_regs *regs, u8 cc, unsigned long ip, unsigned long disp) +bool __emulate_cc(unsigned long flags, u8 cc) { - static const unsigned long jcc_mask[6] = { + static const unsigned long cc_mask[6] = { [0] = X86_EFLAGS_OF, [1] = X86_EFLAGS_CF, [2] = X86_EFLAGS_ZF, @@ -192,15 +192,21 @@ void int3_emulate_jcc(struct pt_regs *re bool match; if (cc < 0xc) { - match = regs->flags & jcc_mask[cc >> 1]; + match = flags & cc_mask[cc >> 1]; } else { - match = ((regs->flags & X86_EFLAGS_SF) >> X86_EFLAGS_SF_BIT) ^ - ((regs->flags & X86_EFLAGS_OF) >> X86_EFLAGS_OF_BIT); + match = ((flags & X86_EFLAGS_SF) >> X86_EFLAGS_SF_BIT) ^ + ((flags & X86_EFLAGS_OF) >> X86_EFLAGS_OF_BIT); if (cc >= 0xe) - match = match || (regs->flags & X86_EFLAGS_ZF); + match = match || (flags & X86_EFLAGS_ZF); } - if ((match && !invert) || (!match && invert)) + return (match && !invert) || (!match && invert); +} + +static __always_inline +void int3_emulate_jcc(struct pt_regs *regs, u8 cc, unsigned long ip, unsigned long disp) +{ + if (__emulate_cc(regs->flags, cc)) ip += disp; int3_emulate_jmp(regs, ip); --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "x86.h" #include "tss.h" @@ -416,31 +417,6 @@ static int fastop(struct x86_emulate_ctx ON64(FOP3E(op##q, rax, rdx, cl)) \ FOP_END -/* Special case for SETcc - 1 instruction per cc */ -#define FOP_SETCC(op) \ - FOP_FUNC(op) \ - #op " %al \n\t" \ - FOP_RET(op) - -FOP_START(setcc) -FOP_SETCC(seto) -FOP_SETCC(setno) -FOP_SETCC(setc) -FOP_SETCC(setnc) -FOP_SETCC(setz) -FOP_SETCC(setnz) -FOP_SETCC(setbe) -FOP_SETCC(setnbe) -FOP_SETCC(sets) -FOP_SETCC(setns) -FOP_SETCC(setp) -FOP_SETCC(setnp) -FOP_SETCC(setl) -FOP_SETCC(setnl) -FOP_SETCC(setle) -FOP_SETCC(setnle) -FOP_END; - FOP_START(salc) FOP_FUNC(salc) "pushf; sbb %al, %al; popf \n\t" @@ -1068,13 +1044,7 @@ static int em_bsr_c(struct x86_emulate_c static __always_inline u8 test_cc(unsigned int condition, unsigned long flags) { - u8 rc; - void (*fop)(void) = (void *)em_setcc + FASTOP_SIZE * (condition & 0xf); - - flags = (flags & EFLAGS_MASK) | X86_EFLAGS_IF; - asm("push %[flags]; popf; " CALL_NOSPEC - : "=a"(rc), ASM_CALL_CONSTRAINT : [thunk_target]"r"(fop), [flags]"r"(flags)); - return rc; + return __emulate_cc(flags, condition & 0xf); } static void fetch_register_operand(struct operand *op) From patchwork Wed Apr 30 11:07:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886185 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1E51231848; Wed, 30 Apr 2025 11:26:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012411; cv=none; b=eFCUM87s0ctuVZhdV8aouHDSceZ9xXkZXZnEc78iBvCPccoe8+gfkJRt0+YoHp1Tjx3mYo/9n8dLDq8UwpVY2rtACp+jkNHUSr3nrYS8NNgVpZeYXnH3i9XK3/LEVlSD3gJJ/NP+0F/I1+LC/CSghqDaKugcoI2hUTiINToG9FE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012411; c=relaxed/simple; bh=eRpt3IJXVe8NDJUYqDudEQFeYF/5Z60nW7F8AopGLSY=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=PgkP67A89SBvBKcUlje85OxtTaru9gzPXrizea/F0VEOadhxtU8d+9dM+CPV4vH2IbiiHIDuVUpATjuSzCO99ID8aE7HGB0LWkFciQUsY6p56Z8VqeyHTZGoOjZ2t7XTL4pw0ZXOsc4xcL3YUWjNekGa5NKzt+soH+8YOocJFWg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=JdAj9E/O; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="JdAj9E/O" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=SY3cT5EHeTQj82upQDn6RhIKWrY4Z9Jf5tOAOyEzNnc=; b=JdAj9E/OoW0Y4j36FFEwKZksY/ zY2Ar4nlrHSwnufeafB8+FcQ/DyXXhrIl1bkxS7HxhnjEEtPFSs90kV+XvGUNIgd/Z6m3YWmEGdFG +7qDomtRfDKvEfCA14Ycjl5SZhw7GQNEH76TNx+UmUHhv6IA5tw3kgNHNR1iDiI/YOAYsdmEBsqWb Gok4J9fLeRvsz2MefzfMafG+z+73nRvZqQa4IV0jKp3GfIjWZo4x5SvTfYKQIbxLS2Yl74p5YVe4D cNm7p2usZBAKkJ4oTbOZDiAKHFN5eBH+mggom01HK26hePqP6lgm6/dvloSagAxBYAhZ63ODEHM5x lMSIaNew==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1uA5aG-0000000C6Zt-12Kf; Wed, 30 Apr 2025 11:26:36 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id CE9F030078C; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112349.208590367@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:36 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 02/13] x86/kvm/emulate: Introduce COP1 References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Replace fastops with C-ops. There are a bunch of problems with the current fastop infrastructure, most all related to their special calling convention, which bypasses the normal C-ABI. There are two immediate problems with this at present: - it relies on RET preserving EFLAGS; whereas C-ABI does not. - it circumvents compiler based control-flow-integrity checking because its all asm magic. The first is a problem for some mitigations where the x86_indirect_return_thunk needs to include non-trivial work that clobbers EFLAGS (eg. the Skylake call depth tracking thing). The second is a problem because it presents a 'naked' indirect call on kCFI builds, making it a prime target for control flow hijacking. Additionally, given that a large chunk of virtual machine performance relies on absolutely avoiding vmexit these days, this emulation stuff just isn't that critical for performance anymore. As such, replace the fastop calls with a normal C function using the 'execute' member. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 69 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 57 insertions(+), 12 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -267,11 +267,56 @@ static void invalidate_registers(struct X86_EFLAGS_PF|X86_EFLAGS_CF) #ifdef CONFIG_X86_64 -#define ON64(x) x +#define ON64(x...) x #else #define ON64(x) #endif +#define COP_START(op) \ +static int em_##op(struct x86_emulate_ctxt *ctxt) \ +{ \ + unsigned long flags = (ctxt->eflags & EFLAGS_MASK) | X86_EFLAGS_IF; \ + int bytes = 1, ok = 1; \ + if (!(ctxt->d & ByteOp)) \ + bytes = ctxt->dst.bytes; \ + switch (bytes) { + +#define COP_ASM(str) \ + asm("push %[flags]; popf \n\t" \ + "10: " str \ + "pushf; pop %[flags] \n\t" \ + "11: \n\t" \ + : "+a" (ctxt->dst.val), \ + "+d" (ctxt->src.val), \ + [flags] "+D" (flags), \ + "+S" (ok) \ + : "c" (ctxt->src2.val)) + +#define COP_ASM1(op, dst) \ + COP_ASM(#op " %%" #dst " \n\t") + +#define COP_ASM1_EX(op, dst) \ + COP_ASM(#op " %%" #dst " \n\t" \ + _ASM_EXTABLE_TYPE_REG(10b, 11f, EX_TYPE_ZERO_REG, %%esi)) + +#define COP_ASM2(op, dst, src) \ + COP_ASM(#op " %" #src ", %" #dst " \n\t") + +#define COP_END \ + } \ + ctxt->eflags = (ctxt->eflags & ~EFLAGS_MASK) | (flags & EFLAGS_MASK); \ + return !ok ? emulate_de(ctxt) : X86EMUL_CONTINUE; \ +} + +/* 1-operand, using "a" (dst) */ +#define COP1(op) \ + COP_START(op) \ + case 1: COP_ASM1(op##b, al); break; \ + case 2: COP_ASM1(op##w, ax); break; \ + case 4: COP_ASM1(op##l, eax); break; \ + ON64(case 8: COP_ASM1(op##q, rax); break;) \ + COP_END + /* * fastop functions have a special calling convention: * @@ -1002,10 +1047,10 @@ FASTOP3WCL(shrd); FASTOP2W(imul); -FASTOP1(not); -FASTOP1(neg); -FASTOP1(inc); -FASTOP1(dec); +COP1(not); +COP1(neg); +COP1(inc); +COP1(dec); FASTOP2CL(rol); FASTOP2CL(ror); @@ -4021,8 +4066,8 @@ static const struct opcode group2[] = { static const struct opcode group3[] = { F(DstMem | SrcImm | NoWrite, em_test), F(DstMem | SrcImm | NoWrite, em_test), - F(DstMem | SrcNone | Lock, em_not), - F(DstMem | SrcNone | Lock, em_neg), + I(DstMem | SrcNone | Lock, em_not), + I(DstMem | SrcNone | Lock, em_neg), F(DstXacc | Src2Mem, em_mul_ex), F(DstXacc | Src2Mem, em_imul_ex), F(DstXacc | Src2Mem, em_div_ex), @@ -4030,14 +4075,14 @@ static const struct opcode group3[] = { }; static const struct opcode group4[] = { - F(ByteOp | DstMem | SrcNone | Lock, em_inc), - F(ByteOp | DstMem | SrcNone | Lock, em_dec), + I(ByteOp | DstMem | SrcNone | Lock, em_inc), + I(ByteOp | DstMem | SrcNone | Lock, em_dec), N, N, N, N, N, N, }; static const struct opcode group5[] = { - F(DstMem | SrcNone | Lock, em_inc), - F(DstMem | SrcNone | Lock, em_dec), + I(DstMem | SrcNone | Lock, em_inc), + I(DstMem | SrcNone | Lock, em_dec), I(SrcMem | NearBranch | IsBranch, em_call_near_abs), I(SrcMemFAddr | ImplicitOps | IsBranch, em_call_far), I(SrcMem | NearBranch | IsBranch, em_jmp_abs), @@ -4237,7 +4282,7 @@ static const struct opcode opcode_table[ /* 0x38 - 0x3F */ F6ALU(NoWrite, em_cmp), N, N, /* 0x40 - 0x4F */ - X8(F(DstReg, em_inc)), X8(F(DstReg, em_dec)), + X8(I(DstReg, em_inc)), X8(I(DstReg, em_dec)), /* 0x50 - 0x57 */ X8(I(SrcReg | Stack, em_push)), /* 0x58 - 0x5F */ From patchwork Wed Apr 30 11:07:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886562 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1F0225333E; Wed, 30 Apr 2025 11:26:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012411; cv=none; b=mfVuhVwXaD6UrwHvfWhLcKFwMNiQ6t0AfTUI18ITX50urWUUsJG8SybyekAu3wXYYwkhEs9cGwe1xnutViml6zsnzvhLy3VSG+8EIRFtRATAcy2Vfh+GNQf7pn/TKpEy5gW7dcaRb8ApfxxUqaGvPrz2nVsrsHJemaBYAaSRyCU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012411; c=relaxed/simple; bh=PWxeJw7JX/cCrAKNbgSe7RGslQYLclCWSz18auxBU0c=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=RD6IObziCEQFMxlrbK+oFN2tjO1mOfTgPX9X3Py4RxrCyHswSugZH3GtIw3UPj3jJFCGd4q/piitGFPi+XBi77jmf8el/xeOcnO1WqtDeqUE3+KtUi6B4ARbHRBifq1nZmvyegFv/bdczDHqCb6+U1h7uRgddLYdCkbpbgXerv4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=mFZuaM5r; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="mFZuaM5r" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=DrCeATcJD9NVn1UQf4+qjGtuQpGFm0cVOCX9YPFGKbs=; b=mFZuaM5r/wYFRqIx2Ge1lfSvg0 e2vDTxyucqxXlGodFKoqkwn7jZlg97tKQasYEDx18rg+CxpiiMCCS+TwQj16rk1bQYrTPO6YNHwRe J+giptLSshsx89+3to12i6aloYCjj5ogHEg9QaJhibtRuWrV7zpldk/HeliExNvLzGCUeZAuix2P4 sgyRKpDn4L3JSpqA70BeBs6F4HpzKphFzDt1KEmOyQRQI9Er/zshh4AeJVF72CB/sng+nEdpeDrTY 4mQn4npZwdsunUhOf7GHtMK3EJGzG0nDH2CFqDF/SCwqWcYIHutPGA0FenGc3EohvXZnYOfusLca2 2M5KaaxA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.1 #2 (Red Hat Linux)) id 1uA5aG-0000000Dm7o-144D; Wed, 30 Apr 2025 11:26:36 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D259F300F1D; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112349.315832833@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:37 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 03/13] x86/kvm/emulate: Introduce COP2 References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Replace the FASTOP2 instructions. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 87 +++++++++++++++++++++++++++---------------------- 1 file changed, 48 insertions(+), 39 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -300,7 +300,7 @@ static int em_##op(struct x86_emulate_ct _ASM_EXTABLE_TYPE_REG(10b, 11f, EX_TYPE_ZERO_REG, %%esi)) #define COP_ASM2(op, dst, src) \ - COP_ASM(#op " %" #src ", %" #dst " \n\t") + COP_ASM(#op " %%" #src ", %%" #dst " \n\t") #define COP_END \ } \ @@ -317,6 +317,15 @@ static int em_##op(struct x86_emulate_ct ON64(case 8: COP_ASM1(op##q, rax); break;) \ COP_END +/* 2-operand, using "a" (dst), "d" (src) */ +#define COP2(op) \ + COP_START(op) \ + case 1: COP_ASM2(op##b, al, dl); break; \ + case 2: COP_ASM2(op##w, ax, dx); break; \ + case 4: COP_ASM2(op##l, eax, edx); break; \ + ON64(case 8: COP_ASM2(op##q, rax, rdx); break;) \ + COP_END + /* * fastop functions have a special calling convention: * @@ -1027,15 +1036,16 @@ static int read_descriptor(struct x86_em return rc; } -FASTOP2(add); -FASTOP2(or); -FASTOP2(adc); -FASTOP2(sbb); -FASTOP2(and); -FASTOP2(sub); -FASTOP2(xor); -FASTOP2(cmp); -FASTOP2(test); +COP2(add); +COP2(or); +COP2(adc); +COP2(sbb); +COP2(and); +COP2(sub); +COP2(xor); +COP2(cmp); +COP2(test); +COP2(xadd); FASTOP1SRC2(mul, mul_ex); FASTOP1SRC2(imul, imul_ex); @@ -1067,7 +1077,6 @@ FASTOP2W(bts); FASTOP2W(btr); FASTOP2W(btc); -FASTOP2(xadd); FASTOP2R(cmp, cmp_r); @@ -2304,7 +2313,7 @@ static int em_cmpxchg(struct x86_emulate ctxt->dst.val = reg_read(ctxt, VCPU_REGS_RAX); ctxt->src.orig_val = ctxt->src.val; ctxt->src.val = ctxt->dst.orig_val; - fastop(ctxt, em_cmp); + em_cmp(ctxt); if (ctxt->eflags & X86_EFLAGS_ZF) { /* Success: write back to memory; no update of EAX */ @@ -3069,7 +3078,7 @@ static int em_das(struct x86_emulate_ctx ctxt->src.type = OP_IMM; ctxt->src.val = 0; ctxt->src.bytes = 1; - fastop(ctxt, em_or); + em_or(ctxt); ctxt->eflags &= ~(X86_EFLAGS_AF | X86_EFLAGS_CF); if (cf) ctxt->eflags |= X86_EFLAGS_CF; @@ -3095,7 +3104,7 @@ static int em_aam(struct x86_emulate_ctx ctxt->src.type = OP_IMM; ctxt->src.val = 0; ctxt->src.bytes = 1; - fastop(ctxt, em_or); + em_or(ctxt); return X86EMUL_CONTINUE; } @@ -3113,7 +3122,7 @@ static int em_aad(struct x86_emulate_ctx ctxt->src.type = OP_IMM; ctxt->src.val = 0; ctxt->src.bytes = 1; - fastop(ctxt, em_or); + em_or(ctxt); return X86EMUL_CONTINUE; } @@ -3998,9 +4007,9 @@ static int check_perm_out(struct x86_emu #define I2bvIP(_f, _e, _i, _p) \ IIP((_f) | ByteOp, _e, _i, _p), IIP(_f, _e, _i, _p) -#define F6ALU(_f, _e) F2bv((_f) | DstMem | SrcReg | ModRM, _e), \ - F2bv(((_f) | DstReg | SrcMem | ModRM) & ~Lock, _e), \ - F2bv(((_f) & ~Lock) | DstAcc | SrcImm, _e) +#define I6ALU(_f, _e) I2bv((_f) | DstMem | SrcReg | ModRM, _e), \ + I2bv(((_f) | DstReg | SrcMem | ModRM) & ~Lock, _e), \ + I2bv(((_f) & ~Lock) | DstAcc | SrcImm, _e) static const struct opcode group7_rm0[] = { N, @@ -4038,14 +4047,14 @@ static const struct opcode group7_rm7[] }; static const struct opcode group1[] = { - F(Lock, em_add), - F(Lock | PageTable, em_or), - F(Lock, em_adc), - F(Lock, em_sbb), - F(Lock | PageTable, em_and), - F(Lock, em_sub), - F(Lock, em_xor), - F(NoWrite, em_cmp), + I(Lock, em_add), + I(Lock | PageTable, em_or), + I(Lock, em_adc), + I(Lock, em_sbb), + I(Lock | PageTable, em_and), + I(Lock, em_sub), + I(Lock, em_xor), + I(NoWrite, em_cmp), }; static const struct opcode group1A[] = { @@ -4064,8 +4073,8 @@ static const struct opcode group2[] = { }; static const struct opcode group3[] = { - F(DstMem | SrcImm | NoWrite, em_test), - F(DstMem | SrcImm | NoWrite, em_test), + I(DstMem | SrcImm | NoWrite, em_test), + I(DstMem | SrcImm | NoWrite, em_test), I(DstMem | SrcNone | Lock, em_not), I(DstMem | SrcNone | Lock, em_neg), F(DstXacc | Src2Mem, em_mul_ex), @@ -4258,29 +4267,29 @@ static const struct instr_dual instr_dua static const struct opcode opcode_table[256] = { /* 0x00 - 0x07 */ - F6ALU(Lock, em_add), + I6ALU(Lock, em_add), I(ImplicitOps | Stack | No64 | Src2ES, em_push_sreg), I(ImplicitOps | Stack | No64 | Src2ES, em_pop_sreg), /* 0x08 - 0x0F */ - F6ALU(Lock | PageTable, em_or), + I6ALU(Lock | PageTable, em_or), I(ImplicitOps | Stack | No64 | Src2CS, em_push_sreg), N, /* 0x10 - 0x17 */ - F6ALU(Lock, em_adc), + I6ALU(Lock, em_adc), I(ImplicitOps | Stack | No64 | Src2SS, em_push_sreg), I(ImplicitOps | Stack | No64 | Src2SS, em_pop_sreg), /* 0x18 - 0x1F */ - F6ALU(Lock, em_sbb), + I6ALU(Lock, em_sbb), I(ImplicitOps | Stack | No64 | Src2DS, em_push_sreg), I(ImplicitOps | Stack | No64 | Src2DS, em_pop_sreg), /* 0x20 - 0x27 */ - F6ALU(Lock | PageTable, em_and), N, N, + I6ALU(Lock | PageTable, em_and), N, N, /* 0x28 - 0x2F */ - F6ALU(Lock, em_sub), N, I(ByteOp | DstAcc | No64, em_das), + I6ALU(Lock, em_sub), N, I(ByteOp | DstAcc | No64, em_das), /* 0x30 - 0x37 */ - F6ALU(Lock, em_xor), N, N, + I6ALU(Lock, em_xor), N, N, /* 0x38 - 0x3F */ - F6ALU(NoWrite, em_cmp), N, N, + I6ALU(NoWrite, em_cmp), N, N, /* 0x40 - 0x4F */ X8(I(DstReg, em_inc)), X8(I(DstReg, em_dec)), /* 0x50 - 0x57 */ @@ -4306,7 +4315,7 @@ static const struct opcode opcode_table[ G(DstMem | SrcImm, group1), G(ByteOp | DstMem | SrcImm | No64, group1), G(DstMem | SrcImmByte, group1), - F2bv(DstMem | SrcReg | ModRM | NoWrite, em_test), + I2bv(DstMem | SrcReg | ModRM | NoWrite, em_test), I2bv(DstMem | SrcReg | ModRM | Lock | PageTable, em_xchg), /* 0x88 - 0x8F */ I2bv(DstMem | SrcReg | ModRM | Mov | PageTable, em_mov), @@ -4329,7 +4338,7 @@ static const struct opcode opcode_table[ I2bv(SrcSI | DstDI | Mov | String | TwoMemOp, em_mov), F2bv(SrcSI | DstDI | String | NoWrite | TwoMemOp, em_cmp_r), /* 0xA8 - 0xAF */ - F2bv(DstAcc | SrcImm | NoWrite, em_test), + I2bv(DstAcc | SrcImm | NoWrite, em_test), I2bv(SrcAcc | DstDI | Mov | String, em_mov), I2bv(SrcSI | DstAcc | Mov | String, em_mov), F2bv(SrcAcc | DstDI | String | NoWrite, em_cmp_r), @@ -4467,7 +4476,7 @@ static const struct opcode twobyte_table I(DstReg | SrcMem | ModRM, em_bsr_c), D(DstReg | SrcMem8 | ModRM | Mov), D(DstReg | SrcMem16 | ModRM | Mov), /* 0xC0 - 0xC7 */ - F2bv(DstMem | SrcReg | ModRM | SrcWrite | Lock, em_xadd), + I2bv(DstMem | SrcReg | ModRM | SrcWrite | Lock, em_xadd), N, ID(0, &instr_dual_0f_c3), N, N, N, GD(0, &group9), /* 0xC8 - 0xCF */ From patchwork Wed Apr 30 11:07:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886189 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5033C1F0984; Wed, 30 Apr 2025 11:26:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012408; cv=none; b=aEg2hhOJvknablymrZPnMNFvxMl3hN1da/MEMDNE7+MlNPl71AIdhrfOrn5WyGaPZZCAaKIWOl3s7qtjZYV8vBr5j3hgemKgzIG1rXfsliWziTP51O9CiuwA/Bz+1TQpgJ488CzRIbNos9xg69YtGzkuUDVHxaBCoFlxvqoSExc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012408; c=relaxed/simple; bh=iBm8Y1E/7H3HGUJT6kN14OnT4mmu+SMHRsgTsBGWXGI=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=iofAHF2fsOiSk/oHHPURrj4i6g0OextXsJgfRMYPLH3iH46ayRp6xevmK00tLQ1R8dBqh5lK5fLJylrRydN/d2c/8/pIgmb/p5qlStpTQJRFD9DaYn2JtzjcgcxTkW5N1HqgyAcmqA8db7HKg0e0c39b2ZpRY+Rjf6QdTRJl2y0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Ziwuczlz; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Ziwuczlz" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=mF7wwCYZMtUCn0kF704+nA6en6Dcv2ccdZkU2OGjtIw=; b=ZiwuczlzmaUS2jCJiINL7wkt+j T8oC8iZuM7E15F3Q7BbYcRsSQOZVN9XCoAHw5I+0Vg7CbmcOyfxBP5718fLqNqqumv2xF+jP+aQXZ pnDMhk8LXWgB2GDGYWLrdlAkkl3JP9SalFfAk8/1dHfm0FeKrMnLlKv96cfVmP3Qvo2lE+22yAGl9 olAUt4oIG9Yfv2ThgRGPljwOh8gn3IF8C94ZSj1y8ku/D3lB/YTVJbUH6g/yBdOtDxSo2lui01dv8 +R4ucnQ+ZCeB3LPw60gTOSwHilwR+qPJOkmFDcPapdTrNHYdTAUliQeQid+hrQM1RF88AjFOKyOn7 lbb2TYIw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1uA5aG-0000000C6Zv-12aF; Wed, 30 Apr 2025 11:26:36 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D5CD7301150; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112349.419161890@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:38 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 04/13] x86/kvm/emulate: Introduce COP2R References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Replace the FASTOP2R instruction. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -326,6 +326,15 @@ static int em_##op(struct x86_emulate_ct ON64(case 8: COP_ASM2(op##q, rax, rdx); break;) \ COP_END +/* 2-operand, reversed */ +#define COP2R(op, name) \ + COP_START(name) \ + case 1: COP_ASM2(op##b, dl, al); break; \ + case 2: COP_ASM2(op##w, dx, ax); break; \ + case 4: COP_ASM2(op##l, edx, eax); break; \ + ON64(case 8: COP_ASM2(op##q, rdx, rax); break;) \ + COP_END + /* * fastop functions have a special calling convention: * @@ -1077,8 +1086,7 @@ FASTOP2W(bts); FASTOP2W(btr); FASTOP2W(btc); - -FASTOP2R(cmp, cmp_r); +COP2R(cmp, cmp_r); static int em_bsf_c(struct x86_emulate_ctxt *ctxt) { @@ -4336,12 +4344,12 @@ static const struct opcode opcode_table[ I2bv(DstAcc | SrcMem | Mov | MemAbs, em_mov), I2bv(DstMem | SrcAcc | Mov | MemAbs | PageTable, em_mov), I2bv(SrcSI | DstDI | Mov | String | TwoMemOp, em_mov), - F2bv(SrcSI | DstDI | String | NoWrite | TwoMemOp, em_cmp_r), + I2bv(SrcSI | DstDI | String | NoWrite | TwoMemOp, em_cmp_r), /* 0xA8 - 0xAF */ I2bv(DstAcc | SrcImm | NoWrite, em_test), I2bv(SrcAcc | DstDI | Mov | String, em_mov), I2bv(SrcSI | DstAcc | Mov | String, em_mov), - F2bv(SrcAcc | DstDI | String | NoWrite, em_cmp_r), + I2bv(SrcAcc | DstDI | String | NoWrite, em_cmp_r), /* 0xB0 - 0xB7 */ X8(I(ByteOp | DstReg | SrcImm | Mov, em_mov)), /* 0xB8 - 0xBF */ From patchwork Wed Apr 30 11:07:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886186 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EF0C24A047; Wed, 30 Apr 2025 11:26:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012410; cv=none; b=W5a2POJOxFksXQgFUVwcWFSK4bAJHag0fH/MDlbTwVqPiZSUJ69lI8Kylg4xSt7YzGY8QEMaDoY+AynyJyg9v5mPi5XSkDuAvVZjvzs3pTPJI6s5103fyOA+UUxcwDbGFB7KV/wuEF5GTpy9d3JKkEwW1RIbSjGeez1T5JPYgFo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012410; c=relaxed/simple; bh=tskvLx3A/90lRbxLNLzK9gXJCjHC1JDHK9WLIR2sVAU=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=SEgps30htVP9wIgPc0K0Q+w3AZAzS4snD7E6+PYePkrBCHNawT5JwIiHYXWEzWyDZpiMNpZM9T+ZLldOPC+SJ+V24GjV9RjhuYpebICQAycrFfQV9e1H8p1RdWQ9UPJT3VAX9iCxsEqElGZzvZ6ZLJc874bJrx6fXK0jKla3Zgk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=UGv+MF41; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="UGv+MF41" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=pyZj6ujzhsAiSHKF9sTlt+P1SIFUzJGpqnjLwaUJEus=; b=UGv+MF41Qujbmmxih7/US5QKm3 VG7vkHZGv645tqxzqe7JYiKFt/5RENUJk2MvUDORStSekA4chgETWKfQkHthTOs0T1eB+4NP+R3qK TOfIIKxI6vHTroJW2xJGBq/0OUXYTlx9YZyRA46TUgDdDX3REEjjqBxdpUHiPw21YOeqe8jayX9J7 RovE71uTeIHoJmfWs8bgtQBHczeQSXR+g7ODesw76HbLxoSYBpCB0m3J/TIrnk0OhIrWJUH5S/Ull sZPt/uPSWTzEGDJjhOQkZKtRxcxmlR/doy2404MliFSrjrmaztNk2Uf3GORUoIcLXEUySYQFAIswh syiF6Leg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.1 #2 (Red Hat Linux)) id 1uA5aH-0000000Dm7w-01wx; Wed, 30 Apr 2025 11:26:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D96703014B0; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112349.531091551@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:39 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 05/13] x86/kvm/emulate: Introduce COP2W References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Replace the FASTOP2W instructions. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -335,6 +335,15 @@ static int em_##op(struct x86_emulate_ct ON64(case 8: COP_ASM2(op##q, rdx, rax); break;) \ COP_END +/* 2-operand, word only (no byte op) */ +#define COP2W(op) \ + COP_START(op) \ + case 1: break; \ + case 2: COP_ASM2(op##w, ax, dx); break; \ + case 4: COP_ASM2(op##l, eax, edx); break; \ + ON64(case 8: COP_ASM2(op##q, rax, rdx); break;) \ + COP_END + /* * fastop functions have a special calling convention: * @@ -1064,7 +1073,7 @@ FASTOP1SRC2EX(idiv, idiv_ex); FASTOP3WCL(shld); FASTOP3WCL(shrd); -FASTOP2W(imul); +COP2W(imul); COP1(not); COP1(neg); @@ -1079,12 +1088,12 @@ FASTOP2CL(shl); FASTOP2CL(shr); FASTOP2CL(sar); -FASTOP2W(bsf); -FASTOP2W(bsr); -FASTOP2W(bt); -FASTOP2W(bts); -FASTOP2W(btr); -FASTOP2W(btc); +COP2W(bsf); +COP2W(bsr); +COP2W(bt); +COP2W(bts); +COP2W(btr); +COP2W(btc); COP2R(cmp, cmp_r); @@ -1093,7 +1102,7 @@ static int em_bsf_c(struct x86_emulate_c /* If src is zero, do not writeback, but update flags */ if (ctxt->src.val == 0) ctxt->dst.type = OP_NONE; - return fastop(ctxt, em_bsf); + return em_bsf(ctxt); } static int em_bsr_c(struct x86_emulate_ctxt *ctxt) @@ -1101,7 +1110,7 @@ static int em_bsr_c(struct x86_emulate_c /* If src is zero, do not writeback, but update flags */ if (ctxt->src.val == 0) ctxt->dst.type = OP_NONE; - return fastop(ctxt, em_bsr); + return em_bsr(ctxt); } static __always_inline u8 test_cc(unsigned int condition, unsigned long flags) @@ -3221,7 +3230,7 @@ static int em_xchg(struct x86_emulate_ct static int em_imul_3op(struct x86_emulate_ctxt *ctxt) { ctxt->dst.val = ctxt->src2.val; - return fastop(ctxt, em_imul); + return em_imul(ctxt); } static int em_cwd(struct x86_emulate_ctxt *ctxt) @@ -4135,10 +4144,10 @@ static const struct group_dual group7 = static const struct opcode group8[] = { N, N, N, N, - F(DstMem | SrcImmByte | NoWrite, em_bt), - F(DstMem | SrcImmByte | Lock | PageTable, em_bts), - F(DstMem | SrcImmByte | Lock, em_btr), - F(DstMem | SrcImmByte | Lock | PageTable, em_btc), + I(DstMem | SrcImmByte | NoWrite, em_bt), + I(DstMem | SrcImmByte | Lock | PageTable, em_bts), + I(DstMem | SrcImmByte | Lock, em_btr), + I(DstMem | SrcImmByte | Lock | PageTable, em_btc), }; /* @@ -4459,27 +4468,27 @@ static const struct opcode twobyte_table /* 0xA0 - 0xA7 */ I(Stack | Src2FS, em_push_sreg), I(Stack | Src2FS, em_pop_sreg), II(ImplicitOps, em_cpuid, cpuid), - F(DstMem | SrcReg | ModRM | BitOp | NoWrite, em_bt), + I(DstMem | SrcReg | ModRM | BitOp | NoWrite, em_bt), F(DstMem | SrcReg | Src2ImmByte | ModRM, em_shld), F(DstMem | SrcReg | Src2CL | ModRM, em_shld), N, N, /* 0xA8 - 0xAF */ I(Stack | Src2GS, em_push_sreg), I(Stack | Src2GS, em_pop_sreg), II(EmulateOnUD | ImplicitOps, em_rsm, rsm), - F(DstMem | SrcReg | ModRM | BitOp | Lock | PageTable, em_bts), + I(DstMem | SrcReg | ModRM | BitOp | Lock | PageTable, em_bts), F(DstMem | SrcReg | Src2ImmByte | ModRM, em_shrd), F(DstMem | SrcReg | Src2CL | ModRM, em_shrd), - GD(0, &group15), F(DstReg | SrcMem | ModRM, em_imul), + GD(0, &group15), I(DstReg | SrcMem | ModRM, em_imul), /* 0xB0 - 0xB7 */ I2bv(DstMem | SrcReg | ModRM | Lock | PageTable | SrcWrite, em_cmpxchg), I(DstReg | SrcMemFAddr | ModRM | Src2SS, em_lseg), - F(DstMem | SrcReg | ModRM | BitOp | Lock, em_btr), + I(DstMem | SrcReg | ModRM | BitOp | Lock, em_btr), I(DstReg | SrcMemFAddr | ModRM | Src2FS, em_lseg), I(DstReg | SrcMemFAddr | ModRM | Src2GS, em_lseg), D(DstReg | SrcMem8 | ModRM | Mov), D(DstReg | SrcMem16 | ModRM | Mov), /* 0xB8 - 0xBF */ N, N, G(BitOp, group8), - F(DstMem | SrcReg | ModRM | BitOp | Lock | PageTable, em_btc), + I(DstMem | SrcReg | ModRM | BitOp | Lock | PageTable, em_btc), I(DstReg | SrcMem | ModRM, em_bsf_c), I(DstReg | SrcMem | ModRM, em_bsr_c), D(DstReg | SrcMem8 | ModRM | Mov), D(DstReg | SrcMem16 | ModRM | Mov), From patchwork Wed Apr 30 11:07:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886565 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7918238171; Wed, 30 Apr 2025 11:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012409; cv=none; b=eKyLV6liWDNcWITnG9Emm29DdhLMC2O27jPajmQemzp4PWRhJvZbFMUB7JdKhnz80xgv76d3zRYqsVK+fl2rjyd/89mz6OdCMFYypp/4k5djGGbBHpXlrQSfUsBBGmKFovI697laKxqlAYjpO1ZoSli6VyQtVpweVxFy+4SzZfQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012409; c=relaxed/simple; bh=iy74pMiqFQ9A70eRBzWcq+Z/PM5B/ScXQmxmFRSa63k=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=RpRSQ8ROt6uRPtf3XFqB9Iq2fTlQFQ/75908viKv46349n+fsxaTAyCRq/xDsAu1CPFaOKkGsNJfs9ZwrDxn2AqRviARigSA5kAdPPOA+g6XrPua9hItRhe24qujdvBg+V31WZlhyDVAQU0EKu6fxK0D0FMgMHalDLdRvKTJ+cQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=PdEKVnFT; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="PdEKVnFT" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ApnWKpIB65Vl54zlJrOsr0YOMRYeS7E3a7J3fKNYWYk=; b=PdEKVnFTK/FNHyx0opbCC1hBaC gvGnfJkFcKFzVfx949iAotfLnfrxwuXKGJdj+/XkHh2y41lIV9pOAMDBvS8X0IRhTkjwKj8QlZtUE rS+2wTewuvQMC4TwhQmcFx60lrnUfmEUvEGQRZNSzY2hSn58Xvpfhh5exQnZjVpPQTa0GYwu2ECSn 5sWTGl8eNDpJQwsBtqSYP/cuUngUnlSvaQlx95vlQ9sPVztQJWIP3Do17z6oOeiukfIsKlJbyCjhk 3u1FKeb7fzZh8zTv8wviU4cyrYypbVJhmnEjVBo/oG7wRJuSuKDZ4qapxO2LcVWoACTjE9WkiCE+V adUcceYA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.1 #2 (Red Hat Linux)) id 1uA5aH-0000000Dm7x-00KN; Wed, 30 Apr 2025 11:26:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id DCD0A301D03; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112349.639363816@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:40 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 06/13] x86/kvm/emulate: Introduce COP2CL References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Replace the FASTOP2CL instructions. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -344,6 +344,15 @@ static int em_##op(struct x86_emulate_ct ON64(case 8: COP_ASM2(op##q, rax, rdx); break;) \ COP_END +/* 2-operand, using "a" (dst) and CL (src2) */ +#define COP2CL(op) \ + COP_START(op) \ + case 1: COP_ASM2(op##b, al, cl); break; \ + case 2: COP_ASM2(op##w, ax, cl); break; \ + case 4: COP_ASM2(op##l, eax, cl); break; \ + ON64(case 8: COP_ASM2(op##q, rax, cl); break;) \ + COP_END + /* * fastop functions have a special calling convention: * @@ -1080,13 +1089,13 @@ COP1(neg); COP1(inc); COP1(dec); -FASTOP2CL(rol); -FASTOP2CL(ror); -FASTOP2CL(rcl); -FASTOP2CL(rcr); -FASTOP2CL(shl); -FASTOP2CL(shr); -FASTOP2CL(sar); +COP2CL(rol); +COP2CL(ror); +COP2CL(rcl); +COP2CL(rcr); +COP2CL(shl); +COP2CL(shr); +COP2CL(sar); COP2W(bsf); COP2W(bsr); @@ -4079,14 +4088,14 @@ static const struct opcode group1A[] = { }; static const struct opcode group2[] = { - F(DstMem | ModRM, em_rol), - F(DstMem | ModRM, em_ror), - F(DstMem | ModRM, em_rcl), - F(DstMem | ModRM, em_rcr), - F(DstMem | ModRM, em_shl), - F(DstMem | ModRM, em_shr), - F(DstMem | ModRM, em_shl), - F(DstMem | ModRM, em_sar), + I(DstMem | ModRM, em_rol), + I(DstMem | ModRM, em_ror), + I(DstMem | ModRM, em_rcl), + I(DstMem | ModRM, em_rcr), + I(DstMem | ModRM, em_shl), + I(DstMem | ModRM, em_shr), + I(DstMem | ModRM, em_shl), + I(DstMem | ModRM, em_sar), }; static const struct opcode group3[] = { From patchwork Wed Apr 30 11:07:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886190 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50398204F9C; Wed, 30 Apr 2025 11:26:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012407; cv=none; b=YZF4AVrajaUfSIeAgcb1u9kFvihdqYTMs1M5pdazXmjO3KIkKQlD4yL+UkVZfdEPIMPHxZPo2G09Ym48VXc+QONFA66k7cMzFq0gYONhzi9iKQvA0/3cYlfoVA1daaCo+kZwiNIeSSO4sn+eDvYUCD+BTmSS2xpji1tC0GzQ3W4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012407; c=relaxed/simple; bh=pQ9Yaxx54zBVXHxbZ2BkYpBzdP6e5jjIv6A+opO62k8=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=S27yRrdjUjQ4ydtm3aMx7dte02O6/mQbGRmegJCvbyfxYB1yBgWkUW0907BLOMkadAegZTBNGNHZo9lhw344DLYB+pAPjI6QfAOlspwjwfj7c3aSOrscBF4STv5rB90vNKbCHGrAByHkZ17FU3EEXYhwB3G9BPq1thcMDqGznkA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=MnYymzMs; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="MnYymzMs" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=2FI0P7JQN/8TK4ts59kC/fO10NTZfLMEix72+dV+X7U=; b=MnYymzMsf8tkCwJFiDMJ/8gnOa DXoYNnPBrg2IMJdeR7FTzqwJuhRsvmXO9rQXn7ozeb6wlzWjSQYKRaYMYvXWFQL0DHRhYbqpD7S5O gws0BM8Eb4LNTlQzIogrPJ98yx16HDhgwh2o6SIMTFonBPQchAx10WKaxbILUiWHHv3lxntQZjvvY OEs/98UXbBfPGF1zkrWkqfMMl7gwBxqoTTUDKIb+0IIyloGLx37zzqH1GjnG2m0wRsg/i/QOZTVhY Pku0geu+XPfgXCc3W61vDb8mIE2Frb2NwEiYR2z2QyAnC3j8vMk2v+V6l3Pe8Hkk0lXcIbq1NaObm FbHC/Sxg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1uA5aH-0000000C6a5-02iR; Wed, 30 Apr 2025 11:26:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E08E63061EE; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112349.751028945@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:41 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 07/13] x86/kvm/emulate: Introduce COP1SRC2 References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Replace the FASTOP1SRC2*() instructions. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -317,6 +317,24 @@ static int em_##op(struct x86_emulate_ct ON64(case 8: COP_ASM1(op##q, rax); break;) \ COP_END +/* 1-operand, using "c" (src2) */ +#define COP1SRC2(op, name) \ + COP_START(name) \ + case 1: COP_ASM1(op##b, cl); break; \ + case 2: COP_ASM1(op##w, cx); break; \ + case 4: COP_ASM1(op##l, ecx); break; \ + ON64(case 8: COP_ASM1(op##q, rcx); break;) \ + COP_END + +/* 1-operand, using "c" (src2) with exception */ +#define COP1SRC2EX(op, name) \ + COP_START(name) \ + case 1: COP_ASM1_EX(op##b, cl); break; \ + case 2: COP_ASM1_EX(op##w, cx); break; \ + case 4: COP_ASM1_EX(op##l, ecx); break; \ + ON64(case 8: COP_ASM1(op##q, rcx); break;) \ + COP_END + /* 2-operand, using "a" (dst), "d" (src) */ #define COP2(op) \ COP_START(op) \ @@ -1074,10 +1092,10 @@ COP2(cmp); COP2(test); COP2(xadd); -FASTOP1SRC2(mul, mul_ex); -FASTOP1SRC2(imul, imul_ex); -FASTOP1SRC2EX(div, div_ex); -FASTOP1SRC2EX(idiv, idiv_ex); +COP1SRC2(mul, mul_ex); +COP1SRC2(imul, imul_ex); +COP1SRC2EX(div, div_ex); +COP1SRC2EX(idiv, idiv_ex); FASTOP3WCL(shld); FASTOP3WCL(shrd); @@ -4103,10 +4121,10 @@ static const struct opcode group3[] = { I(DstMem | SrcImm | NoWrite, em_test), I(DstMem | SrcNone | Lock, em_not), I(DstMem | SrcNone | Lock, em_neg), - F(DstXacc | Src2Mem, em_mul_ex), - F(DstXacc | Src2Mem, em_imul_ex), - F(DstXacc | Src2Mem, em_div_ex), - F(DstXacc | Src2Mem, em_idiv_ex), + I(DstXacc | Src2Mem, em_mul_ex), + I(DstXacc | Src2Mem, em_imul_ex), + I(DstXacc | Src2Mem, em_div_ex), + I(DstXacc | Src2Mem, em_idiv_ex), }; static const struct opcode group4[] = { From patchwork Wed Apr 30 11:07:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886187 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC2E6246795; Wed, 30 Apr 2025 11:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012410; cv=none; b=K4uvLOI+GDICxAS51UrBDFfPxaT+lyXS6tn7/0/923eDgHv50ixiyssCy1RrTjw1CiYGQifIGKZmwrw22PwfUS4lH8Eul1DC6gjRSl1NyyYvUFrWeNCCyVX+2iEfvK7XGm9P0BGhvLgga4XjZHN+KkYVncDpgfotOwwDN27vS1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012410; c=relaxed/simple; bh=mg8jKvSmgfVVHoZ7hw5hkCmvXf95pqqRq6UeCwgAa34=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=YbRVvgFADiLOJ/4v1LcRSPFPzelb947s2+8kLeQiTkDlHbkjYCDdBhfaAfcun3aHFo1rHehJcn3e3Ro+fHapvXqwrwi46/L/imESXhI03bKDAcBBSjMA3/Ph4dr5gGEY9H/MilbnwbLeYLRsqS9LdLG4nREpMjBfGKGTDUgEx1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=NRChl+xy; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="NRChl+xy" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=a3oTVFV42BAnlcjOkQsl89vvtKHMlPa0kSRiVt7oMbc=; b=NRChl+xyv1Fb8q6cvEUds9mj+r XKE1uCDILG+/gdVWShdljUZ6z4/vQHLeKrkQJPptSDFFdHsc+JMtyj2mK+kbFcQfcE5NfSZ5ykNCe 5i1B8N5S3/PhRQS5e3kNfBbTq89pyhz7AQwLZyjUNleAmlCWurTepxvEByCqAfb8p/x7VagcYeuJL 4TOG3Cm5GMF5DewNfQiasHh6aN94LGcEQZwG2cV32yRNfo8i7xI909lgl+k7pLkSTVv+zoDL6ZRCw cxQUF+NbK/MoEQ/bq6zzimTrFkHUYnfP1xvmxv/khd9cd+R9nA5r8QZIM+nBWrarLN6LPoxXQu25r eP4/EiWA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.1 #2 (Red Hat Linux)) id 1uA5aH-0000000Dm7y-06eH; Wed, 30 Apr 2025 11:26:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E478F307FAE; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112349.855046292@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:42 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 08/13] x86/kvm/emulate: Introduce COP3WCL References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Replace the FASTOP3WCL instructions. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -302,6 +302,9 @@ static int em_##op(struct x86_emulate_ct #define COP_ASM2(op, dst, src) \ COP_ASM(#op " %%" #src ", %%" #dst " \n\t") +#define COP_ASM3(op, dst, src, src2) \ + COP_ASM(#op " %%" #src2 ", %%" #src ", %%" #dst " \n\t") + #define COP_END \ } \ ctxt->eflags = (ctxt->eflags & ~EFLAGS_MASK) | (flags & EFLAGS_MASK); \ @@ -371,6 +374,16 @@ static int em_##op(struct x86_emulate_ct ON64(case 8: COP_ASM2(op##q, rax, cl); break;) \ COP_END +/* 3-operand, using "a" (dst), "d" (src) and CL (src2) */ +#define COP3WCL(op) \ + COP_START(op) \ + case 1: break; \ + case 2: COP_ASM3(op##w, ax, dx, cl); break; \ + case 4: COP_ASM3(op##l, eax, edx, cl); break; \ + ON64(case 8: COP_ASM3(op##q, rax, rdx, cl); break;) \ + COP_END + + /* * fastop functions have a special calling convention: * @@ -1097,8 +1110,8 @@ COP1SRC2(imul, imul_ex); COP1SRC2EX(div, div_ex); COP1SRC2EX(idiv, idiv_ex); -FASTOP3WCL(shld); -FASTOP3WCL(shrd); +COP3WCL(shld); +COP3WCL(shrd); COP2W(imul); @@ -4496,14 +4509,14 @@ static const struct opcode twobyte_table I(Stack | Src2FS, em_push_sreg), I(Stack | Src2FS, em_pop_sreg), II(ImplicitOps, em_cpuid, cpuid), I(DstMem | SrcReg | ModRM | BitOp | NoWrite, em_bt), - F(DstMem | SrcReg | Src2ImmByte | ModRM, em_shld), - F(DstMem | SrcReg | Src2CL | ModRM, em_shld), N, N, + I(DstMem | SrcReg | Src2ImmByte | ModRM, em_shld), + I(DstMem | SrcReg | Src2CL | ModRM, em_shld), N, N, /* 0xA8 - 0xAF */ I(Stack | Src2GS, em_push_sreg), I(Stack | Src2GS, em_pop_sreg), II(EmulateOnUD | ImplicitOps, em_rsm, rsm), I(DstMem | SrcReg | ModRM | BitOp | Lock | PageTable, em_bts), - F(DstMem | SrcReg | Src2ImmByte | ModRM, em_shrd), - F(DstMem | SrcReg | Src2CL | ModRM, em_shrd), + I(DstMem | SrcReg | Src2ImmByte | ModRM, em_shrd), + I(DstMem | SrcReg | Src2CL | ModRM, em_shrd), GD(0, &group15), I(DstReg | SrcMem | ModRM, em_imul), /* 0xB0 - 0xB7 */ I2bv(DstMem | SrcReg | ModRM | Lock | PageTable | SrcWrite, em_cmpxchg), From patchwork Wed Apr 30 11:07:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886567 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDC381DE2A4; Wed, 30 Apr 2025 11:26:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012408; cv=none; b=L8Ky7054r56fd1AFxHXZLXjqI+qUEv048yw7ZVrtT+7SuVOu3LhUObmO/YlvaSyxNeFC041ncJ6l5sFqvXVcFDHZW7obkplwiI5BPfllpiNm2xmzZLvJf2oqdBD35SwYCP0IW7iUK/lzgdBkYb5ogdtCJQbqNX0ic9RXOpSdHFI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012408; c=relaxed/simple; bh=vpKuQizapCMKqH9WA11ms5jc5OISD3N0+Dznv7nE1MM=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=ZZxviFznUF7r3qQJzWz8cpsw9EmUmWPnfBf3rvjwem2/J14rTZHLldFw8UzhZ5Bppqrfm1M2BLvRghTcWJoTBxBzcIj8YlWanel5B4QhkEO0PblxSDcZuoY0fBxOgE18Wodn9iYpn5RVSulCTU6s0FLst0GQZSnx6lUtok99AUI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=tr601hlE; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="tr601hlE" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=e4AF8GQacEV/iOhFocsW7bzZKWyP7FsAsEjPpw7Nh70=; b=tr601hlExqdVHsAm1ZyrbgR4gL B6q9+8YRnqXLpJJnvkWNR2kl5t7t4bzp/E8niAF0V9IntOTx1VnA6EQQlfHava3TsTgpa4C96BUaf xmV8SpwsinHszs8VV7U7Z/HI79XcTg5NF6H6t8Jl9+0W/CqMLxjIELVSo6R2JN4agQ+kH+28avdHo e6CV3p7tY+QjN53YCGAr3kb4B609TuY66KcsC9wsfhXbH09CTDMuRHNuD7nqE3tpXVPyUUVCRaaFl oQJLQNZL0pDHHWTHaB/RjDDuCLTmxzeXaBa2/ndwqOeiDV/Hl2fCNZIBVRHeRK1857yIjTGbNqYM1 s4T1GCUQ==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1uA5aH-0000000C6a7-04pg; Wed, 30 Apr 2025 11:26:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E7BB73080C5; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112349.963740147@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:43 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 09/13] x86/kvm/emulate: Convert em_salc() to C References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Implement the SALC (Set AL if Carry) instruction in C. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -529,11 +529,14 @@ static int fastop(struct x86_emulate_ctx ON64(FOP3E(op##q, rax, rdx, cl)) \ FOP_END -FOP_START(salc) -FOP_FUNC(salc) -"pushf; sbb %al, %al; popf \n\t" -FOP_RET(salc) -FOP_END; +static int em_salc(struct x86_emulate_ctxt *ctxt) +{ + /* + * Set AL 0xFF if CF is set, or 0x00 when clear. + */ + ctxt->dst.val = 0xFF * !!(ctxt->eflags & X86_EFLAGS_CF); + return X86EMUL_CONTINUE; +} /* * XXX: inoutclob user must know where the argument is being expanded. @@ -4423,7 +4426,7 @@ static const struct opcode opcode_table[ G(Src2CL | ByteOp, group2), G(Src2CL, group2), I(DstAcc | SrcImmUByte | No64, em_aam), I(DstAcc | SrcImmUByte | No64, em_aad), - F(DstAcc | ByteOp | No64, em_salc), + I(DstAcc | ByteOp | No64, em_salc), I(DstAcc | SrcXLat | ByteOp, em_mov), /* 0xD8 - 0xDF */ N, E(0, &escape_d9), N, E(0, &escape_db), N, E(0, &escape_dd), N, N, From patchwork Wed Apr 30 11:07:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886566 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67784235041; Wed, 30 Apr 2025 11:26:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012408; cv=none; b=d/+7qrC008ylD+BEFQZWF3sraiUQhMaRgI9H05MO6lL8iTjPbDqEEPv4JP8P/2dlcMnGxm8eLyoYJ1o3Hu3Bl2exTgOHnD72riCSbJyHnUc9V1og9VOb0KqOwIJdt8qxX3HfFHPa8X+xCI25CAuQvYBUhzrR1tkRf5uD7csRrnE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012408; c=relaxed/simple; bh=703Q9hLoTjKjywZjUE40ypLwzRZx2aojN9B3PKzWCCQ=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=GtPyWY0xdF+Uw4u16y/CIctx6YcqegAWrMaYNdHH9GJgHPfO3FoKgklvpLka+Jv8538rs2L+JCMtG5HKMd1XP/5Smb6OMAPS2A3YFLY8oD9Ygc/eL5qxe76GR4wt8qLAXePeQcz7co3CjzuCb2CHXutOCdIW7YrHGGhbgXnLJ5w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=KIjHLl0U; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="KIjHLl0U" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=gAIYbiGJcnCsmbEBuv1qv3aO7Zk23KeF3JM5n7xi2ic=; b=KIjHLl0U9fF6pJgWaS0rcGrxki 063kVRfFRbarN15CIVG5NmTtmxQr6zby29EnUNkQ1hrDqUiYXOYSXANwKEOtoxQ1NWefXx8oRtOW9 7/CnvXbVhJF+wnMkINjB2mYSKcTEmajy8Dbzm433MbXigAxgTRtuXfZQO8OuBr00dk8zEmB+W5Ut1 sFW49Z7hL8le0XlCwU8x9XZFUeNcw/Y72l7logo89EHI9ZC3STX3HAfHMw/p9JEQTPvy8E/C+HZ6b wwK/qImdb1AmxyRs4NZ04ZxhyJvxTRcz146l0CbYvTVKcOrD+sHezZxKqKH8UXwJ9umwa45g78Tlq HIkeGVpA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1uA5aH-0000000C6a6-07Ub; Wed, 30 Apr 2025 11:26:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id EB86E308136; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112350.085829848@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:44 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 10/13] x86/kvm/emulate: Remove fastops References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 No more FASTOPs, remove the remains. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kvm/emulate.c | 172 ------------------------------------------------- 1 file changed, 1 insertion(+), 171 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -167,7 +167,6 @@ #define Unaligned ((u64)2 << 41) /* Explicitly unaligned (e.g. MOVDQU) */ #define Avx ((u64)3 << 41) /* Advanced Vector Extensions */ #define Aligned16 ((u64)4 << 41) /* Aligned to 16 byte boundary (e.g. FXSAVE) */ -#define Fastop ((u64)1 << 44) /* Use opcode::u.fastop */ #define NoWrite ((u64)1 << 45) /* No writeback */ #define SrcWrite ((u64)1 << 46) /* Write back src operand */ #define NoMod ((u64)1 << 47) /* Mod field is ignored */ @@ -203,7 +202,6 @@ struct opcode { const struct escape *esc; const struct instr_dual *idual; const struct mode_dual *mdual; - void (*fastop)(struct fastop *fake); } u; int (*check_perm)(struct x86_emulate_ctxt *ctxt); }; @@ -383,152 +381,6 @@ static int em_##op(struct x86_emulate_ct ON64(case 8: COP_ASM3(op##q, rax, rdx, cl); break;) \ COP_END - -/* - * fastop functions have a special calling convention: - * - * dst: rax (in/out) - * src: rdx (in/out) - * src2: rcx (in) - * flags: rflags (in/out) - * ex: rsi (in:fastop pointer, out:zero if exception) - * - * Moreover, they are all exactly FASTOP_SIZE bytes long, so functions for - * different operand sizes can be reached by calculation, rather than a jump - * table (which would be bigger than the code). - * - * The 16 byte alignment, considering 5 bytes for the RET thunk, 3 for ENDBR - * and 1 for the straight line speculation INT3, leaves 7 bytes for the - * body of the function. Currently none is larger than 4. - */ -static int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop); - -#define FASTOP_SIZE 16 - -#define __FOP_FUNC(name) \ - ".align " __stringify(FASTOP_SIZE) " \n\t" \ - ".type " name ", @function \n\t" \ - name ":\n\t" \ - ASM_ENDBR \ - IBT_NOSEAL(name) - -#define FOP_FUNC(name) \ - __FOP_FUNC(#name) - -#define __FOP_RET(name) \ - "11: " ASM_RET \ - ".size " name ", .-" name "\n\t" - -#define FOP_RET(name) \ - __FOP_RET(#name) - -#define __FOP_START(op, align) \ - extern void em_##op(struct fastop *fake); \ - asm(".pushsection .text, \"ax\" \n\t" \ - ".global em_" #op " \n\t" \ - ".align " __stringify(align) " \n\t" \ - "em_" #op ":\n\t" - -#define FOP_START(op) __FOP_START(op, FASTOP_SIZE) - -#define FOP_END \ - ".popsection") - -#define __FOPNOP(name) \ - __FOP_FUNC(name) \ - __FOP_RET(name) - -#define FOPNOP() \ - __FOPNOP(__stringify(__UNIQUE_ID(nop))) - -#define FOP1E(op, dst) \ - __FOP_FUNC(#op "_" #dst) \ - "10: " #op " %" #dst " \n\t" \ - __FOP_RET(#op "_" #dst) - -#define FOP1EEX(op, dst) \ - FOP1E(op, dst) _ASM_EXTABLE_TYPE_REG(10b, 11b, EX_TYPE_ZERO_REG, %%esi) - -#define FASTOP1(op) \ - FOP_START(op) \ - FOP1E(op##b, al) \ - FOP1E(op##w, ax) \ - FOP1E(op##l, eax) \ - ON64(FOP1E(op##q, rax)) \ - FOP_END - -/* 1-operand, using src2 (for MUL/DIV r/m) */ -#define FASTOP1SRC2(op, name) \ - FOP_START(name) \ - FOP1E(op, cl) \ - FOP1E(op, cx) \ - FOP1E(op, ecx) \ - ON64(FOP1E(op, rcx)) \ - FOP_END - -/* 1-operand, using src2 (for MUL/DIV r/m), with exceptions */ -#define FASTOP1SRC2EX(op, name) \ - FOP_START(name) \ - FOP1EEX(op, cl) \ - FOP1EEX(op, cx) \ - FOP1EEX(op, ecx) \ - ON64(FOP1EEX(op, rcx)) \ - FOP_END - -#define FOP2E(op, dst, src) \ - __FOP_FUNC(#op "_" #dst "_" #src) \ - #op " %" #src ", %" #dst " \n\t" \ - __FOP_RET(#op "_" #dst "_" #src) - -#define FASTOP2(op) \ - FOP_START(op) \ - FOP2E(op##b, al, dl) \ - FOP2E(op##w, ax, dx) \ - FOP2E(op##l, eax, edx) \ - ON64(FOP2E(op##q, rax, rdx)) \ - FOP_END - -/* 2 operand, word only */ -#define FASTOP2W(op) \ - FOP_START(op) \ - FOPNOP() \ - FOP2E(op##w, ax, dx) \ - FOP2E(op##l, eax, edx) \ - ON64(FOP2E(op##q, rax, rdx)) \ - FOP_END - -/* 2 operand, src is CL */ -#define FASTOP2CL(op) \ - FOP_START(op) \ - FOP2E(op##b, al, cl) \ - FOP2E(op##w, ax, cl) \ - FOP2E(op##l, eax, cl) \ - ON64(FOP2E(op##q, rax, cl)) \ - FOP_END - -/* 2 operand, src and dest are reversed */ -#define FASTOP2R(op, name) \ - FOP_START(name) \ - FOP2E(op##b, dl, al) \ - FOP2E(op##w, dx, ax) \ - FOP2E(op##l, edx, eax) \ - ON64(FOP2E(op##q, rdx, rax)) \ - FOP_END - -#define FOP3E(op, dst, src, src2) \ - __FOP_FUNC(#op "_" #dst "_" #src "_" #src2) \ - #op " %" #src2 ", %" #src ", %" #dst " \n\t"\ - __FOP_RET(#op "_" #dst "_" #src "_" #src2) - -/* 3-operand, word-only, src2=cl */ -#define FASTOP3WCL(op) \ - FOP_START(op) \ - FOPNOP() \ - FOP3E(op##w, ax, dx, cl) \ - FOP3E(op##l, eax, edx, cl) \ - ON64(FOP3E(op##q, rax, rdx, cl)) \ - FOP_END - static int em_salc(struct x86_emulate_ctxt *ctxt) { /* @@ -4052,7 +3904,6 @@ static int check_perm_out(struct x86_emu #define MD(_f, _m) { .flags = ((_f) | ModeDual), .u.mdual = (_m) } #define E(_f, _e) { .flags = ((_f) | Escape | ModRM), .u.esc = (_e) } #define I(_f, _e) { .flags = (_f), .u.execute = (_e) } -#define F(_f, _e) { .flags = (_f) | Fastop, .u.fastop = (_e) } #define II(_f, _e, _i) \ { .flags = (_f)|Intercept, .u.execute = (_e), .intercept = x86_intercept_##_i } #define IIP(_f, _e, _i, _p) \ @@ -5158,24 +5009,6 @@ static void fetch_possible_mmx_operand(s kvm_read_mmx_reg(op->addr.mm, &op->mm_val); } -static int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop) -{ - ulong flags = (ctxt->eflags & EFLAGS_MASK) | X86_EFLAGS_IF; - - if (!(ctxt->d & ByteOp)) - fop += __ffs(ctxt->dst.bytes) * FASTOP_SIZE; - - asm("push %[flags]; popf; " CALL_NOSPEC " ; pushf; pop %[flags]\n" - : "+a"(ctxt->dst.val), "+d"(ctxt->src.val), [flags]"+D"(flags), - [thunk_target]"+S"(fop), ASM_CALL_CONSTRAINT - : "c"(ctxt->src2.val)); - - ctxt->eflags = (ctxt->eflags & ~EFLAGS_MASK) | (flags & EFLAGS_MASK); - if (!fop) /* exception is returned in fop variable */ - return emulate_de(ctxt); - return X86EMUL_CONTINUE; -} - void init_decode_cache(struct x86_emulate_ctxt *ctxt) { /* Clear fields that are set conditionally but read without a guard. */ @@ -5340,10 +5173,7 @@ int x86_emulate_insn(struct x86_emulate_ ctxt->eflags &= ~X86_EFLAGS_RF; if (ctxt->execute) { - if (ctxt->d & Fastop) - rc = fastop(ctxt, ctxt->fop); - else - rc = ctxt->execute(ctxt); + rc = ctxt->execute(ctxt); if (rc != X86EMUL_CONTINUE) goto done; goto writeback; From patchwork Wed Apr 30 11:07:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886564 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6B9823504D; Wed, 30 Apr 2025 11:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012410; cv=none; b=MUfWdzfkR6/WGmP7yvZ87XIJt/vALm9gPsPKzkrO5KQZJMU5RO/uyQz0ymxVBUcfqo167qa7oY+iCQMl0vGjk5c2NT7/oPaHXdR4TSZ1F6SSQ9xTqYQ6NZNzWjpiTSed9Dv6KtKSKxogGoXwzn9gYfCeVmkzdbJTkkLbxlZ+jCw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012410; c=relaxed/simple; bh=lhKu5HOtgpbAFGXcKElUshSTx+uTc3VFG2oXFOKFPUY=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=XsIcl1ZZca4WEo6BldNUAH5lK2uA5k3oHgzSejJXghsxckxi8ykSkFjaJ5TfpK4xOEM7Lh5dxqX8glX8c9ZRzDa9Ohr0PSUHyRI2t/8ZjsAeFSb3eyNE6KzhOM1wvWK8OAGcOfrSeayo1ZKYZTf7FKQiHG7Miv1vimEDp7stv28= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=A5/hdivw; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="A5/hdivw" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=lUblvaO4TRbJSr6U7JLwwiOE6m5EUlv1vlS2etHnYj8=; b=A5/hdivwlOFACTQ/rzqCIm1RAT Z2lOPf0WBqQP5gqXThHZF/j9J13ICCvaMhMI3L8HbeLT1B+/cDzvSvZ/4+Sbu2bqcfFdWiU2Joet8 8GgCBaOq5sQ9rVCo2jodL0ZKmt6WhARzfhSIW2Y/zNbvcsfsyl+k4X+Z4VpvdCIZqKoCrrptz9Nsv iR3R0IQjbxprPgM7Tt7dOMQlY9J5EB4BKFV/Mmp/euE4EjIsjvQS66OaIfGQB8LX2+Fzu4JQdpmtQ fLDSXGol/scWVq7T9RoFF+vbxKs+TmYUMYbOtYuhQww7xJyILb3ggIP4BtTm/0kUonsuA3iqXtrIV S2C5qOmA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1uA5aH-0000000C6a8-07jm; Wed, 30 Apr 2025 11:26:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id F057530813A; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112350.225006522@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:45 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 11/13] x86,hyperv: Clean up hv_do_hypercall() References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 What used to be a simple few instructions has turned into a giant mess (for x86_64). Not only does it use static_branch wrong, it mixes it with dynamic branches for no apparent reason. Notably it uses static_branch through an out-of-line function call, which completely defeats the purpose, since instead of a simple JMP/NOP site, you get a CALL+RET+TEST+Jcc sequence in return, which is absolutely idiotic. Add to that a dynamic test of hyperv_paravisor_present, something which is set once and never changed. Replace all this idiocy with a single direct function call to the right hypercall variant. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/hyperv/hv_init.c | 20 +++++ arch/x86/hyperv/ivm.c | 15 ++++ arch/x86/include/asm/mshyperv.h | 137 +++++++++++----------------------------- arch/x86/kernel/cpu/mshyperv.c | 19 +++-- 4 files changed, 89 insertions(+), 102 deletions(-) --- a/arch/x86/hyperv/hv_init.c +++ b/arch/x86/hyperv/hv_init.c @@ -35,7 +35,27 @@ #include void *hv_hypercall_pg; + +#ifdef CONFIG_X86_64 +u64 hv_std_hypercall(u64 control, u64 param1, u64 param2) +{ + u64 hv_status; + + if (!hv_hypercall_pg) + return U64_MAX; + + register u64 __r8 asm("r8") = param2; + asm volatile (CALL_NOSPEC + : "=a" (hv_status), ASM_CALL_CONSTRAINT, + "+c" (control), "+d" (param1), "+r" (__r8) + : THUNK_TARGET(hv_hypercall_pg) + : "cc", "memory", "r9", "r10", "r11"); + + return hv_status; +} +#else EXPORT_SYMBOL_GPL(hv_hypercall_pg); +#endif union hv_ghcb * __percpu *hv_ghcb_pg; --- a/arch/x86/hyperv/ivm.c +++ b/arch/x86/hyperv/ivm.c @@ -376,9 +376,23 @@ int hv_snp_boot_ap(u32 cpu, unsigned lon return ret; } +u64 hv_snp_hypercall(u64 control, u64 param1, u64 param2) +{ + u64 hv_status; + + register u64 __r8 asm("r8") = param2; + asm volatile("vmmcall" + : "=a" (hv_status), ASM_CALL_CONSTRAINT, + "+c" (control), "+d" (param1), "+r" (__r8) + : : "cc", "memory", "r9", "r10", "r11"); + + return hv_status; +} + #else static inline void hv_ghcb_msr_write(u64 msr, u64 value) {} static inline void hv_ghcb_msr_read(u64 msr, u64 *value) {} +u64 hv_snp_hypercall(u64 control, u64 param1, u64 param2) { return U64_MAX; } #endif /* CONFIG_AMD_MEM_ENCRYPT */ #ifdef CONFIG_INTEL_TDX_GUEST @@ -428,6 +442,7 @@ u64 hv_tdx_hypercall(u64 control, u64 pa #else static inline void hv_tdx_msr_write(u64 msr, u64 value) {} static inline void hv_tdx_msr_read(u64 msr, u64 *value) {} +u64 hv_tdx_hypercall(u64 control, u64 param1, u64 param2) { return U64_MAX; } #endif /* CONFIG_INTEL_TDX_GUEST */ #if defined(CONFIG_AMD_MEM_ENCRYPT) || defined(CONFIG_INTEL_TDX_GUEST) --- a/arch/x86/include/asm/mshyperv.h +++ b/arch/x86/include/asm/mshyperv.h @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -38,16 +39,21 @@ static inline unsigned char hv_get_nmi_r return 0; } -#if IS_ENABLED(CONFIG_HYPERV) -extern bool hyperv_paravisor_present; +extern u64 hv_tdx_hypercall(u64 control, u64 param1, u64 param2); +extern u64 hv_snp_hypercall(u64 control, u64 param1, u64 param2); +extern u64 hv_std_hypercall(u64 control, u64 param1, u64 param2); +#if IS_ENABLED(CONFIG_HYPERV) extern void *hv_hypercall_pg; extern union hv_ghcb * __percpu *hv_ghcb_pg; bool hv_isolation_type_snp(void); bool hv_isolation_type_tdx(void); -u64 hv_tdx_hypercall(u64 control, u64 param1, u64 param2); + +#ifdef CONFIG_X86_64 +DECLARE_STATIC_CALL(hv_hypercall, hv_std_hypercall); +#endif /* * DEFAULT INIT GPAT and SEGMENT LIMIT value in struct VMSA @@ -64,37 +70,15 @@ static inline u64 hv_do_hypercall(u64 co { u64 input_address = input ? virt_to_phys(input) : 0; u64 output_address = output ? virt_to_phys(output) : 0; - u64 hv_status; #ifdef CONFIG_X86_64 - if (hv_isolation_type_tdx() && !hyperv_paravisor_present) - return hv_tdx_hypercall(control, input_address, output_address); - - if (hv_isolation_type_snp() && !hyperv_paravisor_present) { - __asm__ __volatile__("mov %[output_address], %%r8\n" - "vmmcall" - : "=a" (hv_status), ASM_CALL_CONSTRAINT, - "+c" (control), "+d" (input_address) - : [output_address] "r" (output_address) - : "cc", "memory", "r8", "r9", "r10", "r11"); - return hv_status; - } - - if (!hv_hypercall_pg) - return U64_MAX; - - __asm__ __volatile__("mov %[output_address], %%r8\n" - CALL_NOSPEC - : "=a" (hv_status), ASM_CALL_CONSTRAINT, - "+c" (control), "+d" (input_address) - : [output_address] "r" (output_address), - THUNK_TARGET(hv_hypercall_pg) - : "cc", "memory", "r8", "r9", "r10", "r11"); + return static_call_mod(hv_hypercall)(control, input_address, output_address); #else u32 input_address_hi = upper_32_bits(input_address); u32 input_address_lo = lower_32_bits(input_address); u32 output_address_hi = upper_32_bits(output_address); u32 output_address_lo = lower_32_bits(output_address); + u64 hv_status; if (!hv_hypercall_pg) return U64_MAX; @@ -107,8 +91,8 @@ static inline u64 hv_do_hypercall(u64 co "D"(output_address_hi), "S"(output_address_lo), THUNK_TARGET(hv_hypercall_pg) : "cc", "memory"); -#endif /* !x86_64 */ return hv_status; +#endif /* !x86_64 */ } /* Hypercall to the L0 hypervisor */ @@ -120,41 +104,23 @@ static inline u64 hv_do_nested_hypercall /* Fast hypercall with 8 bytes of input and no output */ static inline u64 _hv_do_fast_hypercall8(u64 control, u64 input1) { - u64 hv_status; - #ifdef CONFIG_X86_64 - if (hv_isolation_type_tdx() && !hyperv_paravisor_present) - return hv_tdx_hypercall(control, input1, 0); - - if (hv_isolation_type_snp() && !hyperv_paravisor_present) { - __asm__ __volatile__( - "vmmcall" - : "=a" (hv_status), ASM_CALL_CONSTRAINT, - "+c" (control), "+d" (input1) - :: "cc", "r8", "r9", "r10", "r11"); - } else { - __asm__ __volatile__(CALL_NOSPEC - : "=a" (hv_status), ASM_CALL_CONSTRAINT, - "+c" (control), "+d" (input1) - : THUNK_TARGET(hv_hypercall_pg) - : "cc", "r8", "r9", "r10", "r11"); - } + return static_call_mod(hv_hypercall)(control, input1, 0); #else - { - u32 input1_hi = upper_32_bits(input1); - u32 input1_lo = lower_32_bits(input1); - - __asm__ __volatile__ (CALL_NOSPEC - : "=A"(hv_status), - "+c"(input1_lo), - ASM_CALL_CONSTRAINT - : "A" (control), - "b" (input1_hi), - THUNK_TARGET(hv_hypercall_pg) - : "cc", "edi", "esi"); - } -#endif + u32 input1_hi = upper_32_bits(input1); + u32 input1_lo = lower_32_bits(input1); + u64 hv_status; + + __asm__ __volatile__ (CALL_NOSPEC + : "=A"(hv_status), + "+c"(input1_lo), + ASM_CALL_CONSTRAINT + : "A" (control), + "b" (input1_hi), + THUNK_TARGET(hv_hypercall_pg) + : "cc", "edi", "esi"); return hv_status; +#endif } static inline u64 hv_do_fast_hypercall8(u16 code, u64 input1) @@ -174,45 +140,24 @@ static inline u64 hv_do_fast_nested_hype /* Fast hypercall with 16 bytes of input */ static inline u64 _hv_do_fast_hypercall16(u64 control, u64 input1, u64 input2) { - u64 hv_status; - #ifdef CONFIG_X86_64 - if (hv_isolation_type_tdx() && !hyperv_paravisor_present) - return hv_tdx_hypercall(control, input1, input2); - - if (hv_isolation_type_snp() && !hyperv_paravisor_present) { - __asm__ __volatile__("mov %[input2], %%r8\n" - "vmmcall" - : "=a" (hv_status), ASM_CALL_CONSTRAINT, - "+c" (control), "+d" (input1) - : [input2] "r" (input2) - : "cc", "r8", "r9", "r10", "r11"); - } else { - __asm__ __volatile__("mov %[input2], %%r8\n" - CALL_NOSPEC - : "=a" (hv_status), ASM_CALL_CONSTRAINT, - "+c" (control), "+d" (input1) - : [input2] "r" (input2), - THUNK_TARGET(hv_hypercall_pg) - : "cc", "r8", "r9", "r10", "r11"); - } + return static_call_mod(hv_hypercall)(control, input1, input2); #else - { - u32 input1_hi = upper_32_bits(input1); - u32 input1_lo = lower_32_bits(input1); - u32 input2_hi = upper_32_bits(input2); - u32 input2_lo = lower_32_bits(input2); - - __asm__ __volatile__ (CALL_NOSPEC - : "=A"(hv_status), - "+c"(input1_lo), ASM_CALL_CONSTRAINT - : "A" (control), "b" (input1_hi), - "D"(input2_hi), "S"(input2_lo), - THUNK_TARGET(hv_hypercall_pg) - : "cc"); - } -#endif + u32 input1_hi = upper_32_bits(input1); + u32 input1_lo = lower_32_bits(input1); + u32 input2_hi = upper_32_bits(input2); + u32 input2_lo = lower_32_bits(input2); + u64 hv_status; + + __asm__ __volatile__ (CALL_NOSPEC + : "=A"(hv_status), + "+c"(input1_lo), ASM_CALL_CONSTRAINT + : "A" (control), "b" (input1_hi), + "D"(input2_hi), "S"(input2_lo), + THUNK_TARGET(hv_hypercall_pg) + : "cc"); return hv_status; +#endif } static inline u64 hv_do_fast_hypercall16(u16 code, u64 input1, u64 input2) --- a/arch/x86/kernel/cpu/mshyperv.c +++ b/arch/x86/kernel/cpu/mshyperv.c @@ -37,10 +37,6 @@ bool hv_nested; struct ms_hyperv_info ms_hyperv; -/* Used in modules via hv_do_hypercall(): see arch/x86/include/asm/mshyperv.h */ -bool hyperv_paravisor_present __ro_after_init; -EXPORT_SYMBOL_GPL(hyperv_paravisor_present); - #if IS_ENABLED(CONFIG_HYPERV) static inline unsigned int hv_get_nested_msr(unsigned int reg) { @@ -287,8 +283,18 @@ static void __init x86_setup_ops_for_tsc old_restore_sched_clock_state = x86_platform.restore_sched_clock_state; x86_platform.restore_sched_clock_state = hv_restore_sched_clock_state; } + +#ifdef CONFIG_X86_64 +DEFINE_STATIC_CALL(hv_hypercall, hv_std_hypercall); +EXPORT_STATIC_CALL_TRAMP_GPL(hv_hypercall); +#define hypercall_update(hc) static_call_update(hv_hypercall, hc) +#endif #endif /* CONFIG_HYPERV */ +#ifndef hypercall_update +#define hypercall_update(hc) (void)hc +#endif + static uint32_t __init ms_hyperv_platform(void) { u32 eax; @@ -483,14 +489,14 @@ static void __init ms_hyperv_init_platfo ms_hyperv.shared_gpa_boundary = BIT_ULL(ms_hyperv.shared_gpa_boundary_bits); - hyperv_paravisor_present = !!ms_hyperv.paravisor_present; - pr_info("Hyper-V: Isolation Config: Group A 0x%x, Group B 0x%x\n", ms_hyperv.isolation_config_a, ms_hyperv.isolation_config_b); if (hv_get_isolation_type() == HV_ISOLATION_TYPE_SNP) { static_branch_enable(&isolation_type_snp); + if (!ms_hyperv.paravisor_present) + hypercall_update(hv_snp_hypercall); } else if (hv_get_isolation_type() == HV_ISOLATION_TYPE_TDX) { static_branch_enable(&isolation_type_tdx); @@ -498,6 +504,7 @@ static void __init ms_hyperv_init_platfo ms_hyperv.hints &= ~HV_X64_APIC_ACCESS_RECOMMENDED; if (!ms_hyperv.paravisor_present) { + hypercall_update(hv_tdx_hypercall); /* * Mark the Hyper-V TSC page feature as disabled * in a TDX VM without paravisor so that the From patchwork Wed Apr 30 11:07:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886561 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6B32214A69; Wed, 30 Apr 2025 11:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012411; cv=none; b=sbVf1cl81ig4Vk3zKxwnj9KiaGR5JlQis49nqznB1cbt8CGKcgKY8LIn3UxtQ0+z3FCExos7jd2G9tT/7QqVm3Am9QE1GuzndrNsmn1hPZiqJmFrn/BsKdpD5ehY82CfPONCr7Nb/wi5Fb4RvZVjkpDh+2vXoFhKyR29lZ8+jC0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012411; c=relaxed/simple; bh=qO+daH7K0+EbHMG1EBSRjZ1DCQklvHH8L11hAYggMz4=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=CgE7+ufsbfUEIjtoFEf8Ub304SVwF6YGKVWFTjJBCGejh+5C/bCW4iSjt1cNlYLV1/dxa+f1JFNuJD+RoSIgnUmMhgJK1gKMNMmNjsoxWlBpCA4731+Nop+963D77mKp/ZI+IYCc7oVIYwRY29hPwBDs+d0TAoilF7dhPOHW7vI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=fBHp9OSz; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="fBHp9OSz" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ONWXz7W+JVAmB/DU8qyxuEQHV8WCN6yeJIfEzdykSxI=; b=fBHp9OSzJFKle+0Fypl5yCvmW7 K9eiM9ogIubzUOmPR3VUIK7Ze555za3Hhi1bB+2+jKW2IBZ0kyCKNgr8nKaqtrNeXmJgjLHp0vNsO SKPUg8Te+otzZ0e3l8S9dLuJnA61WZogLecJN9xrA4JUo8+jL3mBcsWSDSeemFvgVTNhEJ/XTRVT9 o5duKkZvSujZNE89a4WxBO4W6Juhsk6LnVq2lcxnkj5GRluObytGBhoYKOfszEXYk5P3oy2S6u7S+ 3sgCTeQoNm1d6H+nT+48DNM+TzWailmVxDmh9Wnf6Cryp24I09hNBE8d4jaCFNWocRu8Oh2AgURQE J3aNm0mg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1uA5aH-0000000C6a9-0ELt; Wed, 30 Apr 2025 11:26:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id F2A2C3085D1; Wed, 30 Apr 2025 13:26:35 +0200 (CEST) Message-ID: <20250430112350.335273952@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:46 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 12/13] x86_64,hyperv: Use direct call to hypercall-page References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Instead of using an indirect call to the hypercall page, use a direct call instead. This avoids all CFI problems, including the one where the hypercall page doesn't have IBT on. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Michael Kelley --- arch/x86/hyperv/hv_init.c | 60 +++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 30 deletions(-) --- a/arch/x86/hyperv/hv_init.c +++ b/arch/x86/hyperv/hv_init.c @@ -37,23 +37,41 @@ void *hv_hypercall_pg; #ifdef CONFIG_X86_64 +static u64 __hv_hyperfail(u64 control, u64 param1, u64 param2) +{ + return U64_MAX; +} + +DEFINE_STATIC_CALL(__hv_hypercall, __hv_hyperfail); + u64 hv_std_hypercall(u64 control, u64 param1, u64 param2) { u64 hv_status; - if (!hv_hypercall_pg) - return U64_MAX; - register u64 __r8 asm("r8") = param2; - asm volatile (CALL_NOSPEC + asm volatile ("call " STATIC_CALL_TRAMP_STR(__hv_hypercall) : "=a" (hv_status), ASM_CALL_CONSTRAINT, "+c" (control), "+d" (param1), "+r" (__r8) - : THUNK_TARGET(hv_hypercall_pg) - : "cc", "memory", "r9", "r10", "r11"); + : : "cc", "memory", "r9", "r10", "r11"); return hv_status; } + +typedef u64 (*hv_hypercall_f)(u64 control, u64 param1, u64 param2); + +static inline void hv_set_hypercall_pg(void *ptr) +{ + hv_hypercall_pg = ptr; + + if (!ptr) + ptr = &__hv_hyperfail; + static_call_update(__hv_hypercall, (hv_hypercall_f)ptr); +} #else +static inline void hv_set_hypercall_pg(void *ptr) +{ + hv_hypercall_pg = ptr; +} EXPORT_SYMBOL_GPL(hv_hypercall_pg); #endif @@ -348,7 +366,7 @@ static int hv_suspend(void) * pointer is restored on resume. */ hv_hypercall_pg_saved = hv_hypercall_pg; - hv_hypercall_pg = NULL; + hv_set_hypercall_pg(NULL); /* Disable the hypercall page in the hypervisor */ rdmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64); @@ -374,7 +392,7 @@ static void hv_resume(void) vmalloc_to_pfn(hv_hypercall_pg_saved); wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64); - hv_hypercall_pg = hv_hypercall_pg_saved; + hv_set_hypercall_pg(hv_hypercall_pg_saved); hv_hypercall_pg_saved = NULL; /* @@ -528,8 +546,8 @@ void __init hyperv_init(void) if (hv_isolation_type_tdx() && !ms_hyperv.paravisor_present) goto skip_hypercall_pg_init; - hv_hypercall_pg = __vmalloc_node_range(PAGE_SIZE, 1, VMALLOC_START, - VMALLOC_END, GFP_KERNEL, PAGE_KERNEL_ROX, + hv_hypercall_pg = __vmalloc_node_range(PAGE_SIZE, 1, MODULES_VADDR, + MODULES_END, GFP_KERNEL, PAGE_KERNEL_ROX, VM_FLUSH_RESET_PERMS, NUMA_NO_NODE, __builtin_return_address(0)); if (hv_hypercall_pg == NULL) @@ -567,27 +585,9 @@ void __init hyperv_init(void) wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64); } -skip_hypercall_pg_init: - /* - * Some versions of Hyper-V that provide IBT in guest VMs have a bug - * in that there's no ENDBR64 instruction at the entry to the - * hypercall page. Because hypercalls are invoked via an indirect call - * to the hypercall page, all hypercall attempts fail when IBT is - * enabled, and Linux panics. For such buggy versions, disable IBT. - * - * Fixed versions of Hyper-V always provide ENDBR64 on the hypercall - * page, so if future Linux kernel versions enable IBT for 32-bit - * builds, additional hypercall page hackery will be required here - * to provide an ENDBR32. - */ -#ifdef CONFIG_X86_KERNEL_IBT - if (cpu_feature_enabled(X86_FEATURE_IBT) && - *(u32 *)hv_hypercall_pg != gen_endbr()) { - setup_clear_cpu_cap(X86_FEATURE_IBT); - pr_warn("Disabling IBT because of Hyper-V bug\n"); - } -#endif + hv_set_hypercall_pg(hv_hypercall_pg); +skip_hypercall_pg_init: /* * hyperv_init() is called before LAPIC is initialized: see * apic_intr_mode_init() -> x86_platform.apic_post_init() and From patchwork Wed Apr 30 11:07:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 886184 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FE69248F77; Wed, 30 Apr 2025 11:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012411; cv=none; b=iEwnlI/GcDKQ7pY/c5O28epAidwMQXrRgiWamcR1hLwL20BAn9OBeMDxZv0uwmY8A2WIdusLMCOWD47DpIGS/eTsb3XSkvnR2uZz6esuD41LHik4mi7hJbDG7iexA1j/Px7QQIocf94Z6zlhH+uXWUdXc0adC/+zn5d4UAAuc4s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746012411; c=relaxed/simple; bh=9+/BuRDGAA6TLDb0yk7rHBW8mHDlaavPQl2vlVEAVkI=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=khYqu/PxefYnawpXdg4WTkA971cWWv9IcRLPXQ0/o6QzTPy/LfgO6RnDOkNVwWUnIOtWO3TuQkj1UzTfJ66Te5TbB9xTOjfjC6ugl+xIxDu2QahRzBKdFNgoyU1ydVxwjFnX530H6+Q/QPIKebrO7ribaNEw2ggnfwvvZ3f/bas= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=W1jXDB8d; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="W1jXDB8d" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=k9Ri4ZyqEibe0HpDT575gbrV5ZF8Kp/wefhRDjxVdRo=; b=W1jXDB8dLl80ZTVasLrOzMMEzh nvwN1guUG/MrY2bLLuthDILrx8Ibgv6CJWPLpz5RouJhh1LnVsA7FH3OzBrgjDqTRwUj2A5NR+dfH A9gBoUxEFgokwWwbM3xdPmqE+z+MmF6/Kv/YCpNb7bFDbmbrRHrabmPPZZcEPOVPN5CxVhoCymmzG YeTmrD+7eYkUFrcEnXXOgSnHyTuoP2Mu/c7Cgl1K+uB0aBhA67topbnJgXxusnUvWIhICoNrA09QW wTNQxOmcrHWCvC1vEZgE7Yp5KlsVeBmCsfEcWSXh8+mTopS1HAcHUojp5AESrS5oivzW6qA6SnOTZ em/Kue3A==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.1 #2 (Red Hat Linux)) id 1uA5aH-0000000Dm7z-0HsX; Wed, 30 Apr 2025 11:26:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 024933085D7; Wed, 30 Apr 2025 13:26:36 +0200 (CEST) Message-ID: <20250430112350.443414861@infradead.org> User-Agent: quilt/0.66 Date: Wed, 30 Apr 2025 13:07:47 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, ardb@kernel.org, kees@kernel.org, Arnd Bergmann , gregkh@linuxfoundation.org, jpoimboe@kernel.org, peterz@infradead.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, samitolvanen@google.com, ojeda@kernel.org Subject: [PATCH v2 13/13] objtool: Validate kCFI calls References: <20250430110734.392235199@infradead.org> Precedence: bulk X-Mailing-List: linux-efi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Validate that all indirect calls adhere to kCFI rules. Notably doing nocfi indirect call to a cfi function is broken. Apparently some Rust 'core' code violates this and explodes when ran with FineIBT. All the ANNOTATE_NOCFI_SYM sites are prime targets for attackers. - runtime EFI is especially henous because it also needs to disable IBT. Basically calling unknown code without CFI protection at runtime is a massice security issue. - Kexec image handover; if you can exploit this, you get to keep it :-) - KVM, for the interrupt injection calling IDT gates directly. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kernel/machine_kexec_64.c | 4 +++ arch/x86/kvm/vmx/vmenter.S | 5 ++++ arch/x86/platform/efi/efi_stub_64.S | 4 +++ drivers/misc/lkdtm/perms.c | 5 ++++ include/linux/objtool.h | 10 ++++++++ include/linux/objtool_types.h | 1 tools/include/linux/objtool_types.h | 1 tools/objtool/check.c | 41 ++++++++++++++++++++++++++++++++++++ tools/objtool/include/objtool/elf.h | 1 9 files changed, 72 insertions(+) --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -421,6 +421,10 @@ void __nocfi machine_kexec(struct kimage __ftrace_enabled_restore(save_ftrace_enabled); } +/* + * Handover to the next kernel, no CFI concern. + */ +ANNOTATE_NOCFI_SYM(machine_kexec); /* arch-dependent functionality related to kexec file-based syscall */ --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -363,5 +363,10 @@ SYM_FUNC_END(vmread_error_trampoline) .section .text, "ax" SYM_FUNC_START(vmx_do_interrupt_irqoff) + /* + * Calling an IDT gate directly; annotate away the CFI concern for now. + * Should be fixed if possible. + */ + ANNOTATE_NOCFI_SYM VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 SYM_FUNC_END(vmx_do_interrupt_irqoff) --- a/arch/x86/platform/efi/efi_stub_64.S +++ b/arch/x86/platform/efi/efi_stub_64.S @@ -11,6 +11,10 @@ #include SYM_FUNC_START(__efi_call) + /* + * The EFI code doesn't have any CFI, annotate away the CFI violation. + */ + ANNOTATE_NOCFI_SYM pushq %rbp movq %rsp, %rbp and $~0xf, %rsp --- a/drivers/misc/lkdtm/perms.c +++ b/drivers/misc/lkdtm/perms.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -86,6 +87,10 @@ static noinline __nocfi void execute_loc func(); pr_err("FAIL: func returned\n"); } +/* + * Explicitly doing the wrong thing for testing. + */ +ANNOTATE_NOCFI_SYM(execute_location); static void execute_user_location(void *dst) { --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -184,6 +184,15 @@ * WARN using UD2. */ #define ANNOTATE_REACHABLE(label) __ASM_ANNOTATE(label, ANNOTYPE_REACHABLE) +/* + * This should not be used; it annotates away CFI violations. There are a few + * valid use cases like kexec handover to the next kernel image, and there is + * no security concern there. + * + * There are also a few real issues annotated away, like EFI because we can't + * control the EFI code. + */ +#define ANNOTATE_NOCFI_SYM(sym) asm(__ASM_ANNOTATE(sym, ANNOTYPE_NOCFI)) #else #define ANNOTATE_NOENDBR ANNOTATE type=ANNOTYPE_NOENDBR @@ -194,6 +203,7 @@ #define ANNOTATE_INTRA_FUNCTION_CALL ANNOTATE type=ANNOTYPE_INTRA_FUNCTION_CALL #define ANNOTATE_UNRET_BEGIN ANNOTATE type=ANNOTYPE_UNRET_BEGIN #define ANNOTATE_REACHABLE ANNOTATE type=ANNOTYPE_REACHABLE +#define ANNOTATE_NOCFI_SYM ANNOTATE type=ANNOTYPE_NOCFI #endif #if defined(CONFIG_NOINSTR_VALIDATION) && \ --- a/include/linux/objtool_types.h +++ b/include/linux/objtool_types.h @@ -65,5 +65,6 @@ struct unwind_hint { #define ANNOTYPE_IGNORE_ALTS 6 #define ANNOTYPE_INTRA_FUNCTION_CALL 7 #define ANNOTYPE_REACHABLE 8 +#define ANNOTYPE_NOCFI 9 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/include/linux/objtool_types.h +++ b/tools/include/linux/objtool_types.h @@ -65,5 +65,6 @@ struct unwind_hint { #define ANNOTYPE_IGNORE_ALTS 6 #define ANNOTYPE_INTRA_FUNCTION_CALL 7 #define ANNOTYPE_REACHABLE 8 +#define ANNOTYPE_NOCFI 9 #endif /* _LINUX_OBJTOOL_TYPES_H */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2388,6 +2388,8 @@ static int __annotate_ifc(struct objtool static int __annotate_late(struct objtool_file *file, int type, struct instruction *insn) { + struct symbol *sym; + switch (type) { case ANNOTYPE_NOENDBR: /* early */ @@ -2429,6 +2431,15 @@ static int __annotate_late(struct objtoo insn->dead_end = false; break; + case ANNOTYPE_NOCFI: + sym = insn->sym; + if (!sym) { + ERROR_INSN(insn, "dodgy NOCFI annotation"); + break; + } + insn->sym->nocfi = 1; + break; + default: ERROR_INSN(insn, "Unknown annotation type: %d", type); return -1; @@ -3998,6 +4009,36 @@ static int validate_retpoline(struct obj warnings++; } + if (!opts.cfi) + return warnings; + + /* + * kCFI call sites look like: + * + * movl $(-0x12345678), %r10d + * addl -4(%r11), %r10d + * jz 1f + * ud2 + * 1: cs call __x86_indirect_thunk_r11 + * + * Verify all indirect calls are kCFI adorned by checking for the + * UD2. Notably, doing __nocfi calls to regular (cfi) functions is + * broken. + */ + list_for_each_entry(insn, &file->retpoline_call_list, call_node) { + struct symbol *sym = insn->sym; + + if (sym && sym->type == STT_FUNC && !sym->nocfi) { + struct instruction *prev = + prev_insn_same_sym(file, insn); + + if (!prev || prev->type != INSN_BUG) { + WARN_INSN(insn, "no-cfi indirect call!"); + warnings++; + } + } + } + return warnings; } --- a/tools/objtool/include/objtool/elf.h +++ b/tools/objtool/include/objtool/elf.h @@ -70,6 +70,7 @@ struct symbol { u8 local_label : 1; u8 frame_pointer : 1; u8 ignore : 1; + u8 nocfi : 1; struct list_head pv_target; struct reloc *relocs; };