[bpf] bpf: use NOP_ATOMIC5 instead of emit_nops(&prog, 5) for BPF_TRAMP_F_CALL_ORIG

Message ID 20210320000001.915366-1-sdf@google.com
State New
Headers show
Series
  • [bpf] bpf: use NOP_ATOMIC5 instead of emit_nops(&prog, 5) for BPF_TRAMP_F_CALL_ORIG
Related show

Commit Message

Stanislav Fomichev March 20, 2021, midnight
__bpf_arch_text_poke does rewrite only for atomic nop5, emit_nops(xxx, 5)
emits non-atomic one which breaks fentry/fexit with k8 atomics:

P6_NOP5 == P6_NOP5_ATOMIC (0f1f440000 == 0f1f440000)
K8_NOP5 != K8_NOP5_ATOMIC (6666906690 != 6666666690)

Can be reproduced by doing "ideal_nops = k8_nops" in "arch_init_ideal_nops()
and running fexit_bpf2bpf selftest.

Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")
Signed-off-by: Stanislav Fomichev <sdf@google.com>
---
 arch/x86/net/bpf_jit_comp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Comments

Alexei Starovoitov March 20, 2021, 12:14 a.m. | #1
On Fri, Mar 19, 2021 at 5:00 PM Stanislav Fomichev <sdf@google.com> wrote:
>
> __bpf_arch_text_poke does rewrite only for atomic nop5, emit_nops(xxx, 5)
> emits non-atomic one which breaks fentry/fexit with k8 atomics:
>
> P6_NOP5 == P6_NOP5_ATOMIC (0f1f440000 == 0f1f440000)
> K8_NOP5 != K8_NOP5_ATOMIC (6666906690 != 6666666690)
>
> Can be reproduced by doing "ideal_nops = k8_nops" in "arch_init_ideal_nops()
> and running fexit_bpf2bpf selftest.
>
> Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")
> Signed-off-by: Stanislav Fomichev <sdf@google.com>
> ---
>  arch/x86/net/bpf_jit_comp.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index 72b5a57e9e31..b35fc8023884 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -2012,7 +2012,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
>                 /* remember return value in a stack for bpf prog to access */
>                 emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
>                 im->ip_after_call = prog;
> -               emit_nops(&prog, 5);
> +               memcpy(prog, ideal_nops[NOP_ATOMIC5], X86_PATCH_SIZE);
> +               prog += X86_PATCH_SIZE;

I'm well aware, but ideal_nops are pretty much gone already.
The changes are already in the -tip tree.
So I decided to reduce the conflicts for the merge window.

Do you actually see the breakage or it's purely theoretical?
Stanislav Fomichev March 20, 2021, 12:25 a.m. | #2
On Fri, Mar 19, 2021 at 5:14 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>

> On Fri, Mar 19, 2021 at 5:00 PM Stanislav Fomichev <sdf@google.com> wrote:

> >

> > __bpf_arch_text_poke does rewrite only for atomic nop5, emit_nops(xxx, 5)

> > emits non-atomic one which breaks fentry/fexit with k8 atomics:

> >

> > P6_NOP5 == P6_NOP5_ATOMIC (0f1f440000 == 0f1f440000)

> > K8_NOP5 != K8_NOP5_ATOMIC (6666906690 != 6666666690)

> >

> > Can be reproduced by doing "ideal_nops = k8_nops" in "arch_init_ideal_nops()

> > and running fexit_bpf2bpf selftest.

> >

> > Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")

> > Signed-off-by: Stanislav Fomichev <sdf@google.com>

> > ---

> >  arch/x86/net/bpf_jit_comp.c | 3 ++-

> >  1 file changed, 2 insertions(+), 1 deletion(-)

> >

> > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c

> > index 72b5a57e9e31..b35fc8023884 100644

> > --- a/arch/x86/net/bpf_jit_comp.c

> > +++ b/arch/x86/net/bpf_jit_comp.c

> > @@ -2012,7 +2012,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i

> >                 /* remember return value in a stack for bpf prog to access */

> >                 emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);

> >                 im->ip_after_call = prog;

> > -               emit_nops(&prog, 5);

> > +               memcpy(prog, ideal_nops[NOP_ATOMIC5], X86_PATCH_SIZE);

> > +               prog += X86_PATCH_SIZE;

>

> I'm well aware, but ideal_nops are pretty much gone already.

> The changes are already in the -tip tree.

> So I decided to reduce the conflicts for the merge window.

>

> Do you actually see the breakage or it's purely theoretical?

We do see it, but it's on our tree that pulls from bpf.
And it obviously doesn't have that "x86: Remove dynamic NOP selection" yet.
Thanks for the pointer, I guess I can just wait for the real merge then.
Alexei Starovoitov March 20, 2021, 12:33 a.m. | #3
On Fri, Mar 19, 2021 at 5:25 PM Stanislav Fomichev <sdf@google.com> wrote:
>

> On Fri, Mar 19, 2021 at 5:14 PM Alexei Starovoitov

> <alexei.starovoitov@gmail.com> wrote:

> >

> > On Fri, Mar 19, 2021 at 5:00 PM Stanislav Fomichev <sdf@google.com> wrote:

> > >

> > > __bpf_arch_text_poke does rewrite only for atomic nop5, emit_nops(xxx, 5)

> > > emits non-atomic one which breaks fentry/fexit with k8 atomics:

> > >

> > > P6_NOP5 == P6_NOP5_ATOMIC (0f1f440000 == 0f1f440000)

> > > K8_NOP5 != K8_NOP5_ATOMIC (6666906690 != 6666666690)

> > >

> > > Can be reproduced by doing "ideal_nops = k8_nops" in "arch_init_ideal_nops()

> > > and running fexit_bpf2bpf selftest.

> > >

> > > Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")

> > > Signed-off-by: Stanislav Fomichev <sdf@google.com>

> > > ---

> > >  arch/x86/net/bpf_jit_comp.c | 3 ++-

> > >  1 file changed, 2 insertions(+), 1 deletion(-)

> > >

> > > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c

> > > index 72b5a57e9e31..b35fc8023884 100644

> > > --- a/arch/x86/net/bpf_jit_comp.c

> > > +++ b/arch/x86/net/bpf_jit_comp.c

> > > @@ -2012,7 +2012,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i

> > >                 /* remember return value in a stack for bpf prog to access */

> > >                 emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);

> > >                 im->ip_after_call = prog;

> > > -               emit_nops(&prog, 5);

> > > +               memcpy(prog, ideal_nops[NOP_ATOMIC5], X86_PATCH_SIZE);

> > > +               prog += X86_PATCH_SIZE;

> >

> > I'm well aware, but ideal_nops are pretty much gone already.

> > The changes are already in the -tip tree.

> > So I decided to reduce the conflicts for the merge window.

> >

> > Do you actually see the breakage or it's purely theoretical?

> We do see it, but it's on our tree that pulls from bpf.

> And it obviously doesn't have that "x86: Remove dynamic NOP selection" yet.

> Thanks for the pointer, I guess I can just wait for the real merge then.


If it breaks the real users we have to land the fix, but let me ask how
come that you run with k8 cpu? k8 does other nasty things.
Do you run with all of amd errata?
Stanislav Fomichev March 20, 2021, 1:40 a.m. | #4
On Fri, Mar 19, 2021 at 5:33 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>

> On Fri, Mar 19, 2021 at 5:25 PM Stanislav Fomichev <sdf@google.com> wrote:

> >

> > On Fri, Mar 19, 2021 at 5:14 PM Alexei Starovoitov

> > <alexei.starovoitov@gmail.com> wrote:

> > >

> > > On Fri, Mar 19, 2021 at 5:00 PM Stanislav Fomichev <sdf@google.com> wrote:

> > > >

> > > > __bpf_arch_text_poke does rewrite only for atomic nop5, emit_nops(xxx, 5)

> > > > emits non-atomic one which breaks fentry/fexit with k8 atomics:

> > > >

> > > > P6_NOP5 == P6_NOP5_ATOMIC (0f1f440000 == 0f1f440000)

> > > > K8_NOP5 != K8_NOP5_ATOMIC (6666906690 != 6666666690)

> > > >

> > > > Can be reproduced by doing "ideal_nops = k8_nops" in "arch_init_ideal_nops()

> > > > and running fexit_bpf2bpf selftest.

> > > >

> > > > Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")

> > > > Signed-off-by: Stanislav Fomichev <sdf@google.com>

> > > > ---

> > > >  arch/x86/net/bpf_jit_comp.c | 3 ++-

> > > >  1 file changed, 2 insertions(+), 1 deletion(-)

> > > >

> > > > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c

> > > > index 72b5a57e9e31..b35fc8023884 100644

> > > > --- a/arch/x86/net/bpf_jit_comp.c

> > > > +++ b/arch/x86/net/bpf_jit_comp.c

> > > > @@ -2012,7 +2012,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i

> > > >                 /* remember return value in a stack for bpf prog to access */

> > > >                 emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);

> > > >                 im->ip_after_call = prog;

> > > > -               emit_nops(&prog, 5);

> > > > +               memcpy(prog, ideal_nops[NOP_ATOMIC5], X86_PATCH_SIZE);

> > > > +               prog += X86_PATCH_SIZE;

> > >

> > > I'm well aware, but ideal_nops are pretty much gone already.

> > > The changes are already in the -tip tree.

> > > So I decided to reduce the conflicts for the merge window.

> > >

> > > Do you actually see the breakage or it's purely theoretical?

> > We do see it, but it's on our tree that pulls from bpf.

> > And it obviously doesn't have that "x86: Remove dynamic NOP selection" yet.

> > Thanks for the pointer, I guess I can just wait for the real merge then.

>

> If it breaks the real users we have to land the fix, but let me ask how

> come that you run with k8 cpu? k8 does other nasty things.

> Do you run with all of amd errata?

It's not amd, it's intel:

cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2689 0 @ 2.60GHz

I think I'm hitting the following from the arch/x86/kernel/alternative.c:

/*
* Due to a decoder implementation quirk, some
* specific Intel CPUs actually perform better with
* the "k8_nops" than with the SDM-recommended NOPs.
*/
if (boot_cpu_data.x86 == 6 &&
   boot_cpu_data.x86_model >= 0x0f &&
   boot_cpu_data.x86_model != 0x1c &&
   boot_cpu_data.x86_model != 0x26 &&
   boot_cpu_data.x86_model != 0x27 &&
   boot_cpu_data.x86_model < 0x30) {
ideal_nops = k8_nops;
Alexei Starovoitov March 20, 2021, 2:31 a.m. | #5
On Fri, Mar 19, 2021 at 6:40 PM Stanislav Fomichev <sdf@google.com> wrote:
>

> On Fri, Mar 19, 2021 at 5:33 PM Alexei Starovoitov

> <alexei.starovoitov@gmail.com> wrote:

> >

> > On Fri, Mar 19, 2021 at 5:25 PM Stanislav Fomichev <sdf@google.com> wrote:

> > >

> > > On Fri, Mar 19, 2021 at 5:14 PM Alexei Starovoitov

> > > <alexei.starovoitov@gmail.com> wrote:

> > > >

> > > > On Fri, Mar 19, 2021 at 5:00 PM Stanislav Fomichev <sdf@google.com> wrote:

> > > > >

> > > > > __bpf_arch_text_poke does rewrite only for atomic nop5, emit_nops(xxx, 5)

> > > > > emits non-atomic one which breaks fentry/fexit with k8 atomics:

> > > > >

> > > > > P6_NOP5 == P6_NOP5_ATOMIC (0f1f440000 == 0f1f440000)

> > > > > K8_NOP5 != K8_NOP5_ATOMIC (6666906690 != 6666666690)

> > > > >

> > > > > Can be reproduced by doing "ideal_nops = k8_nops" in "arch_init_ideal_nops()

> > > > > and running fexit_bpf2bpf selftest.

> > > > >

> > > > > Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")

> > > > > Signed-off-by: Stanislav Fomichev <sdf@google.com>

> > > > > ---

> > > > >  arch/x86/net/bpf_jit_comp.c | 3 ++-

> > > > >  1 file changed, 2 insertions(+), 1 deletion(-)

> > > > >

> > > > > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c

> > > > > index 72b5a57e9e31..b35fc8023884 100644

> > > > > --- a/arch/x86/net/bpf_jit_comp.c

> > > > > +++ b/arch/x86/net/bpf_jit_comp.c

> > > > > @@ -2012,7 +2012,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i

> > > > >                 /* remember return value in a stack for bpf prog to access */

> > > > >                 emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);

> > > > >                 im->ip_after_call = prog;

> > > > > -               emit_nops(&prog, 5);

> > > > > +               memcpy(prog, ideal_nops[NOP_ATOMIC5], X86_PATCH_SIZE);

> > > > > +               prog += X86_PATCH_SIZE;

> > > >

> > > > I'm well aware, but ideal_nops are pretty much gone already.

> > > > The changes are already in the -tip tree.

> > > > So I decided to reduce the conflicts for the merge window.

> > > >

> > > > Do you actually see the breakage or it's purely theoretical?

> > > We do see it, but it's on our tree that pulls from bpf.

> > > And it obviously doesn't have that "x86: Remove dynamic NOP selection" yet.

> > > Thanks for the pointer, I guess I can just wait for the real merge then.

> >

> > If it breaks the real users we have to land the fix, but let me ask how

> > come that you run with k8 cpu? k8 does other nasty things.

> > Do you run with all of amd errata?

> It's not amd, it's intel:

>

> cpu family      : 6

> model           : 45

> model name      : Intel(R) Xeon(R) CPU E5-2689 0 @ 2.60GHz

>

> I think I'm hitting the following from the arch/x86/kernel/alternative.c:

>

> /*

> * Due to a decoder implementation quirk, some

> * specific Intel CPUs actually perform better with

> * the "k8_nops" than with the SDM-recommended NOPs.

> */

> if (boot_cpu_data.x86 == 6 &&

>    boot_cpu_data.x86_model >= 0x0f &&

>    boot_cpu_data.x86_model != 0x1c &&

>    boot_cpu_data.x86_model != 0x26 &&

>    boot_cpu_data.x86_model != 0x27 &&

>    boot_cpu_data.x86_model < 0x30) {

> ideal_nops = k8_nops;


Ohh. Thanks for explaining. Applied.
patchwork-bot+netdevbpf@kernel.org March 20, 2021, 2:40 a.m. | #6
Hello:

This patch was applied to bpf/bpf.git (refs/heads/master):

On Fri, 19 Mar 2021 17:00:01 -0700 you wrote:
> __bpf_arch_text_poke does rewrite only for atomic nop5, emit_nops(xxx, 5)

> emits non-atomic one which breaks fentry/fexit with k8 atomics:

> 

> P6_NOP5 == P6_NOP5_ATOMIC (0f1f440000 == 0f1f440000)

> K8_NOP5 != K8_NOP5_ATOMIC (6666906690 != 6666666690)

> 

> Can be reproduced by doing "ideal_nops = k8_nops" in "arch_init_ideal_nops()

> and running fexit_bpf2bpf selftest.

> 

> [...]


Here is the summary with links:
  - [bpf] bpf: use NOP_ATOMIC5 instead of emit_nops(&prog, 5) for BPF_TRAMP_F_CALL_ORIG
    https://git.kernel.org/bpf/bpf/c/b90829704780

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
Yauheni Kaliuta March 22, 2021, 6:39 a.m. | #7
On Sat, Mar 20, 2021 at 2:01 AM Stanislav Fomichev <sdf@google.com> wrote:
>

> __bpf_arch_text_poke does rewrite only for atomic nop5, emit_nops(xxx, 5)

> emits non-atomic one which breaks fentry/fexit with k8 atomics:

>

> P6_NOP5 == P6_NOP5_ATOMIC (0f1f440000 == 0f1f440000)

> K8_NOP5 != K8_NOP5_ATOMIC (6666906690 != 6666666690)

>

> Can be reproduced by doing "ideal_nops = k8_nops" in "arch_init_ideal_nops()

> and running fexit_bpf2bpf selftest.

>

> Fixes: e21aa341785c ("bpf: Fix fexit trampoline.")

> Signed-off-by: Stanislav Fomichev <sdf@google.com>


Could you mention x86 in the subject?

> ---

>  arch/x86/net/bpf_jit_comp.c | 3 ++-

>  1 file changed, 2 insertions(+), 1 deletion(-)

>

> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c

> index 72b5a57e9e31..b35fc8023884 100644

> --- a/arch/x86/net/bpf_jit_comp.c

> +++ b/arch/x86/net/bpf_jit_comp.c

> @@ -2012,7 +2012,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i

>                 /* remember return value in a stack for bpf prog to access */

>                 emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);

>                 im->ip_after_call = prog;

> -               emit_nops(&prog, 5);

> +               memcpy(prog, ideal_nops[NOP_ATOMIC5], X86_PATCH_SIZE);

> +               prog += X86_PATCH_SIZE;

>         }

>

>         if (fmod_ret->nr_progs) {

> --

> 2.31.0.rc2.261.g7f71774620-goog

>



-- 
WBR, Yauheni

Patch

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 72b5a57e9e31..b35fc8023884 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2012,7 +2012,8 @@  int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
 		/* remember return value in a stack for bpf prog to access */
 		emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
 		im->ip_after_call = prog;
-		emit_nops(&prog, 5);
+		memcpy(prog, ideal_nops[NOP_ATOMIC5], X86_PATCH_SIZE);
+		prog += X86_PATCH_SIZE;
 	}
 
 	if (fmod_ret->nr_progs) {