From patchwork Tue Sep 15 11:40:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 260815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2136C433E2 for ; Wed, 16 Sep 2020 00:49:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F1B520770 for ; Wed, 16 Sep 2020 00:49:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="c+jhPYvn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726905AbgIPAtT (ORCPT ); Tue, 15 Sep 2020 20:49:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:60774 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726400AbgIOLlh (ORCPT ); Tue, 15 Sep 2020 07:41:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600170064; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1WmH/XIEtFA0UaTFoyCv5XynY2J3oKLWx4ph1AwbyRE=; b=c+jhPYvnPZ/uspyB14atByCuqx5nxcUGtjy+zYuSF9jE5rh1SpvYQOmhr8BrUCz2sg4+8b ssUJ+/pXlLy16Xk7RvfYY7IrPVVdsZd9reo6yGhBhEb8Ff6ojOCYxcni72dCMz5YIBfXa5 ePAxFi8y3cu0OTRE1YauVm0f7s/hMM4= Received: from mail-ej1-f72.google.com (mail-ej1-f72.google.com [209.85.218.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-31-kGXvwUzWNb604t8HHTMhOA-1; Tue, 15 Sep 2020 07:41:02 -0400 X-MC-Unique: kGXvwUzWNb604t8HHTMhOA-1 Received: by mail-ej1-f72.google.com with SMTP id i14so1178058ejc.0 for ; Tue, 15 Sep 2020 04:41:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=1WmH/XIEtFA0UaTFoyCv5XynY2J3oKLWx4ph1AwbyRE=; b=ZdRLlxRMcOKcG87O7UyDIaPc6/+2qJt1Via0BpQC7rHuDZO8urJ6pmi00FzPEDUa45 OXVsMuWbpCJfJCSzI79dbQvpVxwGR1JAmujROobJXc+o2hMKT3ALsKqDXw0pmVNZ/jIv cOYn/imM6rClLkKC+KKgQI+d5dkjpqOirGgsFH8HWOz4T+rMxsjCpwyuJdMvNJFt36LI cwcWOUU4+IKmoUj6o+3AQCcWYSGPoi3boX6puan6YshT5lbwzmzAmJYckabKLPGcT3wZ 6VntxAEsJUPqZ0BhigHmTgZ9t0S3pnBiP/DyYayEmMFKslk/Wxnd7vrVEPMmRgqmIiBf //1w== X-Gm-Message-State: AOAM533zmQMZfukt8jPTdKUiSgGmIj8BGLmRqOVBwr183PXa9wsLjM46 k/WnYjAW4X1m4+k2r48dkICKhMQZmQlvkKzuRN1MPf4fKlEr2kZ7PhZSLkGJRRn4Ku7EGXxFR1j jSUCFg9o0dvcfLhkv X-Received: by 2002:a05:6402:3c8:: with SMTP id t8mr21204316edw.266.1600170060695; Tue, 15 Sep 2020 04:41:00 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwfx3zt98gTP5VtcP/H/rwLngR9Ogb8w/6GmwnIf56cUgWF6FKWEV6vNBIHptbyQmybqs+Ipg== X-Received: by 2002:a05:6402:3c8:: with SMTP id t8mr21204292edw.266.1600170060441; Tue, 15 Sep 2020 04:41:00 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id u9sm10071844eje.119.2020.09.15.04.40.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 04:40:59 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 3A5601829CE; Tue, 15 Sep 2020 13:40:59 +0200 (CEST) Subject: [PATCH bpf-next v5 2/8] bpf: verifier: refactor check_attach_btf_id() From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov Cc: Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , Andrii Nakryiko , John Fastabend , Jiri Olsa , Eelco Chaudron , KP Singh , netdev@vger.kernel.org, bpf@vger.kernel.org Date: Tue, 15 Sep 2020 13:40:59 +0200 Message-ID: <160017005916.98230.1736872862729846213.stgit@toke.dk> In-Reply-To: <160017005691.98230.13648200635390228683.stgit@toke.dk> References: <160017005691.98230.13648200635390228683.stgit@toke.dk> User-Agent: StGit/0.23 MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toke Høiland-Jørgensen The check_attach_btf_id() function really does three things: 1. It performs a bunch of checks on the program to ensure that the attachment is valid. 2. It stores a bunch of state about the attachment being requested in the verifier environment and struct bpf_prog objects. 3. It allocates a trampoline for the attachment. This patch splits out (1.) and (3.) into separate functions in preparation for reusing them when the actual attachment is happening (in the raw_tracepoint_open syscall operation), which will allow tracing programs to have multiple (compatible) attachments. No functional change is intended with this patch. Signed-off-by: Toke Høiland-Jørgensen Acked-by: Andrii Nakryiko --- include/linux/bpf.h | 7 + include/linux/bpf_verifier.h | 9 ++ kernel/bpf/trampoline.c | 20 ++++ kernel/bpf/verifier.c | 197 ++++++++++++++++++++++++------------------ 4 files changed, 149 insertions(+), 84 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 5ad4a935a24e..dcf0c70348a4 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -616,6 +616,8 @@ static __always_inline unsigned int bpf_dispatcher_nop_func( struct bpf_trampoline *bpf_trampoline_lookup(u64 key); int bpf_trampoline_link_prog(struct bpf_prog *prog); int bpf_trampoline_unlink_prog(struct bpf_prog *prog); +struct bpf_trampoline *bpf_trampoline_get(u64 key, void *addr, + struct btf_func_model *fmodel); void bpf_trampoline_put(struct bpf_trampoline *tr); #define BPF_DISPATCHER_INIT(_name) { \ .mutex = __MUTEX_INITIALIZER(_name.mutex), \ @@ -672,6 +674,11 @@ static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog) { return -ENOTSUPP; } +static inline struct bpf_trampoline *bpf_trampoline_get(u64 key, void *addr, + struct btf_func_model *fmodel) +{ + return ERR_PTR(-EOPNOTSUPP); +} static inline void bpf_trampoline_put(struct bpf_trampoline *tr) {} #define DEFINE_BPF_DISPATCHER(name) #define DECLARE_BPF_DISPATCHER(name) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 20009e766805..db3db0b69aad 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -447,4 +447,13 @@ bpf_prog_offload_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt); int check_ctx_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, int regno); +int bpf_check_attach_target(struct bpf_verifier_log *log, + const struct bpf_prog *prog, + const struct bpf_prog *tgt_prog, + u32 btf_id, + struct btf_func_model *fmodel, + long *tgt_addr, + const char **tgt_name, + const struct btf_type **tgt_type); + #endif /* _LINUX_BPF_VERIFIER_H */ diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 7dd523a7e32d..7845913e7e41 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -336,6 +336,26 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog) return err; } +struct bpf_trampoline *bpf_trampoline_get(u64 key, void *addr, + struct btf_func_model *fmodel) +{ + struct bpf_trampoline *tr; + + tr = bpf_trampoline_lookup(key); + if (!tr) + return ERR_PTR(-ENOMEM); + + mutex_lock(&tr->mutex); + if (tr->func.addr) + goto out; + + memcpy(&tr->func.model, fmodel, sizeof(*fmodel)); + tr->func.addr = addr; +out: + mutex_unlock(&tr->mutex); + return tr; +} + void bpf_trampoline_put(struct bpf_trampoline *tr) { if (!tr) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0be7a187fb7f..d38678319ca4 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10997,11 +10997,11 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) } #define SECURITY_PREFIX "security_" -static int check_attach_modify_return(struct bpf_prog *prog, unsigned long addr) +static int check_attach_modify_return(const struct bpf_prog *prog, unsigned long addr, + const char *func_name) { if (within_error_injection_list(addr) || - !strncmp(SECURITY_PREFIX, prog->aux->attach_func_name, - sizeof(SECURITY_PREFIX) - 1)) + !strncmp(SECURITY_PREFIX, func_name, sizeof(SECURITY_PREFIX) - 1)) return 0; return -EINVAL; @@ -11038,43 +11038,29 @@ static int check_non_sleepable_error_inject(u32 btf_id) return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id); } -static int check_attach_btf_id(struct bpf_verifier_env *env) +int bpf_check_attach_target(struct bpf_verifier_log *log, + const struct bpf_prog *prog, + const struct bpf_prog *tgt_prog, + u32 btf_id, + struct btf_func_model *fmodel, + long *tgt_addr, + const char **tgt_name, + const struct btf_type **tgt_type) { - struct bpf_prog *prog = env->prog; bool prog_extension = prog->type == BPF_PROG_TYPE_EXT; - struct bpf_prog *tgt_prog = prog->aux->linked_prog; - struct bpf_verifier_log *log = &env->log; - u32 btf_id = prog->aux->attach_btf_id; const char prefix[] = "btf_trace_"; - struct btf_func_model fmodel; int ret = 0, subprog = -1, i; - struct bpf_trampoline *tr; const struct btf_type *t; bool conservative = true; const char *tname; struct btf *btf; - long addr; - u64 key; - - if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING && - prog->type != BPF_PROG_TYPE_LSM) { - verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n"); - return -EINVAL; - } - - if (prog->type == BPF_PROG_TYPE_STRUCT_OPS) - return check_struct_ops_btf_id(env); - - if (prog->type != BPF_PROG_TYPE_TRACING && - prog->type != BPF_PROG_TYPE_LSM && - !prog_extension) - return 0; + long addr = 0; if (!btf_id) { bpf_log(log, "Tracing programs must provide btf_id\n"); return -EINVAL; } - btf = bpf_prog_get_target_btf(prog); + btf = tgt_prog ? tgt_prog->aux->btf : btf_vmlinux; if (!btf) { bpf_log(log, "FENTRY/FEXIT program can only be attached to another program annotated with BTF\n"); @@ -11114,8 +11100,6 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) "Extension programs should be JITed\n"); return -EINVAL; } - env->ops = bpf_verifier_ops[tgt_prog->type]; - prog->expected_attach_type = tgt_prog->expected_attach_type; } if (!tgt_prog->jited) { bpf_log(log, "Can attach to only JITed progs\n"); @@ -11151,13 +11135,11 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) bpf_log(log, "Cannot extend fentry/fexit\n"); return -EINVAL; } - key = ((u64)aux->id) << 32 | btf_id; } else { if (prog_extension) { bpf_log(log, "Cannot replace kernel functions\n"); return -EINVAL; } - key = btf_id; } switch (prog->expected_attach_type) { @@ -11187,13 +11169,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) /* should never happen in valid vmlinux build */ return -EINVAL; - /* remember two read only pointers that are valid for - * the life time of the kernel - */ - prog->aux->attach_func_name = tname; - prog->aux->attach_func_proto = t; - prog->aux->attach_btf_trace = true; - return 0; + break; case BPF_TRACE_ITER: if (!btf_type_is_func(t)) { bpf_log(log, "attach_btf_id %u is not a function\n", @@ -11203,12 +11179,10 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) t = btf_type_by_id(btf, t->type); if (!btf_type_is_func_proto(t)) return -EINVAL; - prog->aux->attach_func_name = tname; - prog->aux->attach_func_proto = t; - if (!bpf_iter_prog_supported(prog)) - return -EINVAL; - ret = btf_distill_func_proto(log, btf, t, tname, &fmodel); - return ret; + ret = btf_distill_func_proto(log, btf, t, tname, fmodel); + if (ret) + return ret; + break; default: if (!prog_extension) return -EINVAL; @@ -11217,13 +11191,6 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) case BPF_LSM_MAC: case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: - prog->aux->attach_func_name = tname; - if (prog->type == BPF_PROG_TYPE_LSM) { - ret = bpf_lsm_verify_prog(log, prog); - if (ret < 0) - return ret; - } - if (!btf_type_is_func(t)) { bpf_log(log, "attach_btf_id %u is not a function\n", btf_id); @@ -11235,24 +11202,14 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) t = btf_type_by_id(btf, t->type); if (!btf_type_is_func_proto(t)) return -EINVAL; - tr = bpf_trampoline_lookup(key); - if (!tr) - return -ENOMEM; - /* t is either vmlinux type or another program's type */ - prog->aux->attach_func_proto = t; - mutex_lock(&tr->mutex); - if (tr->func.addr) { - prog->aux->trampoline = tr; - goto out; - } - if (tgt_prog && conservative) { - prog->aux->attach_func_proto = NULL; + + if (tgt_prog && conservative) t = NULL; - } - ret = btf_distill_func_proto(log, btf, t, - tname, &tr->func.model); + + ret = btf_distill_func_proto(log, btf, t, tname, fmodel); if (ret < 0) - goto out; + return ret; + if (tgt_prog) { if (subprog == 0) addr = (long) tgt_prog->bpf_func; @@ -11264,8 +11221,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) bpf_log(log, "The address of function %s cannot be found\n", tname); - ret = -ENOENT; - goto out; + return -ENOENT; } } @@ -11290,25 +11246,98 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) default: break; } - if (ret) - bpf_log(log, "%s is not sleepable\n", - prog->aux->attach_func_name); + if (ret) { + bpf_log(log, "%s is not sleepable\n", tname); + return ret; + } } else if (prog->expected_attach_type == BPF_MODIFY_RETURN) { - ret = check_attach_modify_return(prog, addr); - if (ret) - bpf_log(log, "%s() is not modifiable\n", - prog->aux->attach_func_name); + ret = check_attach_modify_return(prog, addr, tname); + if (ret) { + bpf_log(log, "%s() is not modifiable\n", tname); + return ret; + } } - if (ret) - goto out; - tr->func.addr = (void *)addr; - prog->aux->trampoline = tr; -out: - mutex_unlock(&tr->mutex); - if (ret) - bpf_trampoline_put(tr); + + break; + } + *tgt_addr = addr; + if (tgt_name) + *tgt_name = tname; + if (tgt_type) + *tgt_type = t; + return 0; +} + +static int check_attach_btf_id(struct bpf_verifier_env *env) +{ + struct bpf_prog *prog = env->prog; + struct bpf_prog *tgt_prog = prog->aux->linked_prog; + u32 btf_id = prog->aux->attach_btf_id; + struct btf_func_model fmodel; + struct bpf_trampoline *tr; + const struct btf_type *t; + const char *tname; + long addr; + int ret; + u64 key; + + if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING && + prog->type != BPF_PROG_TYPE_LSM) { + verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n"); + return -EINVAL; + } + + if (prog->type == BPF_PROG_TYPE_STRUCT_OPS) + return check_struct_ops_btf_id(env); + + if (prog->type != BPF_PROG_TYPE_TRACING && + prog->type != BPF_PROG_TYPE_LSM && + prog->type != BPF_PROG_TYPE_EXT) + return 0; + + ret = bpf_check_attach_target(&env->log, prog, tgt_prog, btf_id, + &fmodel, &addr, &tname, &t); + if (ret) return ret; + + if (tgt_prog) { + if (prog->type == BPF_PROG_TYPE_EXT) { + env->ops = bpf_verifier_ops[tgt_prog->type]; + prog->expected_attach_type = + tgt_prog->expected_attach_type; + } + key = ((u64)tgt_prog->aux->id) << 32 | btf_id; + } else { + key = btf_id; } + + /* remember two read only pointers that are valid for + * the life time of the kernel + */ + prog->aux->attach_func_proto = t; + prog->aux->attach_func_name = tname; + + if (prog->expected_attach_type == BPF_TRACE_RAW_TP) { + prog->aux->attach_btf_trace = true; + return 0; + } else if (prog->expected_attach_type == BPF_TRACE_ITER) { + if (!bpf_iter_prog_supported(prog)) + return -EINVAL; + return 0; + } + + if (prog->type == BPF_PROG_TYPE_LSM) { + ret = bpf_lsm_verify_prog(&env->log, prog); + if (ret < 0) + return ret; + } + + tr = bpf_trampoline_get(key, (void *)addr, &fmodel); + if (IS_ERR(tr)) + return PTR_ERR(tr); + + prog->aux->trampoline = tr; + return 0; } int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, From patchwork Tue Sep 15 11:41:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 260816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE3EBC433E2 for ; Wed, 16 Sep 2020 00:49:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 925C220936 for ; Wed, 16 Sep 2020 00:49:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="dk7dHh6G" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727239AbgIPAtJ (ORCPT ); Tue, 15 Sep 2020 20:49:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:46399 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726401AbgIOLli (ORCPT ); Tue, 15 Sep 2020 07:41:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600170064; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JgEP1D81FI8N+HdhPcvEFCpdpL+CRIhETRMQsPVnENE=; b=dk7dHh6GCdcrLCFjobaujoHRSljH3iBpgGojbdZg9WL30KnL5tBCbAKGCQZwyPfegTVi9P WtdP1oGZMmD5Ukb5OmYx6hdHRR+UxY7DYp/O4aV5bFRGfipYjM8dxs3khOqfzjsxCZQ35o odrGhT9X2VERCxIYW1IhhQxcq90MKf0= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-125-MBnigCEyPJ2Fy0l7xIophg-1; Tue, 15 Sep 2020 07:41:03 -0400 X-MC-Unique: MBnigCEyPJ2Fy0l7xIophg-1 Received: by mail-ed1-f69.google.com with SMTP id d13so1118278edz.18 for ; Tue, 15 Sep 2020 04:41:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=JgEP1D81FI8N+HdhPcvEFCpdpL+CRIhETRMQsPVnENE=; b=hw+NUBeFPcClL8SFXVFuBCt9s0VcQwluoRNmfht58LgojN+nKapdc0sXzelrDAJPps tbm2HrV0l39QRDr7MszNaCGRDJ9h/Gr76C6pWnCJpG90wZXNH9Ys13G5KvJlg+mOuI7w P/vRw1gCewKJR3X1def//VvOUFJLHXEEXjPkWzc+kBYWkgqekbWqoRrzuZwkk9uK8CzV /MJovGZ37weYBXWgIeHvz0CxmzYwbrz3618QVseiYaJtAlp3aPb6aE5s8eIxDVyo14K2 YFn/LUhMPTOrj+HjSGNEjeRaQIDMtamrKKxzQj7bCvvUPyfZxGIbqyPfnq+vMSJBowAw Hs3A== X-Gm-Message-State: AOAM531ZuARiigog8TqDJOHzxvMrQRtbaTVIC8Gk0ug46XtqZLSmaGmI gcopp/v/TOeph9OXeI4DdMfgka2+KUU3X8qy7o+GzcB22ypUe/AMb22TSnLdcrEBZVPQpFo7utY dBN1uL1cv6sNaTAbL X-Received: by 2002:a50:fa88:: with SMTP id w8mr22066088edr.179.1600170061469; Tue, 15 Sep 2020 04:41:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwDSHYEmi7qmqULKM8vR5D6F3SJvWxvV1ZwKDafsJrKKdz+3rTJrAc3eIQ1hMHxL8eJCxcl7g== X-Received: by 2002:a50:fa88:: with SMTP id w8mr22066069edr.179.1600170061216; Tue, 15 Sep 2020 04:41:01 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id v25sm11603515edr.29.2020.09.15.04.41.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 04:41:00 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 520791829CB; Tue, 15 Sep 2020 13:41:00 +0200 (CEST) Subject: [PATCH bpf-next v5 3/8] bpf: move prog->aux->linked_prog and trampoline into bpf_link on attach From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov Cc: Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , Andrii Nakryiko , John Fastabend , Jiri Olsa , Eelco Chaudron , KP Singh , netdev@vger.kernel.org, bpf@vger.kernel.org Date: Tue, 15 Sep 2020 13:41:00 +0200 Message-ID: <160017006024.98230.18011033601869719353.stgit@toke.dk> In-Reply-To: <160017005691.98230.13648200635390228683.stgit@toke.dk> References: <160017005691.98230.13648200635390228683.stgit@toke.dk> User-Agent: StGit/0.23 MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toke Høiland-Jørgensen In preparation for allowing multiple attachments of freplace programs, move the references to the target program and trampoline into the bpf_tracing_link structure when that is created. To do this atomically, introduce a new mutex in prog->aux to protect writing to the two pointers to target prog and trampoline, and rename the members to make it clear that they are related. With this change, it is no longer possible to attach the same tracing program multiple times (detaching in-between), since the reference from the tracing program to the target disappears on the first attach. However, since the next patch will let the caller supply an attach target, that will also make it possible to attach to the same place multiple times. Signed-off-by: Toke Høiland-Jørgensen Acked-by: Andrii Nakryiko --- include/linux/bpf.h | 15 +++++++++------ kernel/bpf/btf.c | 6 +++--- kernel/bpf/core.c | 9 ++++++--- kernel/bpf/syscall.c | 46 ++++++++++++++++++++++++++++++++++++++++------ kernel/bpf/trampoline.c | 12 ++++-------- kernel/bpf/verifier.c | 9 +++++---- 6 files changed, 67 insertions(+), 30 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index dcf0c70348a4..939b37c78d55 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -614,8 +614,8 @@ static __always_inline unsigned int bpf_dispatcher_nop_func( } #ifdef CONFIG_BPF_JIT struct bpf_trampoline *bpf_trampoline_lookup(u64 key); -int bpf_trampoline_link_prog(struct bpf_prog *prog); -int bpf_trampoline_unlink_prog(struct bpf_prog *prog); +int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); +int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); struct bpf_trampoline *bpf_trampoline_get(u64 key, void *addr, struct btf_func_model *fmodel); void bpf_trampoline_put(struct bpf_trampoline *tr); @@ -666,11 +666,13 @@ static inline struct bpf_trampoline *bpf_trampoline_lookup(u64 key) { return NULL; } -static inline int bpf_trampoline_link_prog(struct bpf_prog *prog) +static inline int bpf_trampoline_link_prog(struct bpf_prog *prog, + struct bpf_trampoline *tr) { return -ENOTSUPP; } -static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog) +static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog, + struct bpf_trampoline *tr) { return -ENOTSUPP; } @@ -738,14 +740,15 @@ struct bpf_prog_aux { u32 max_rdonly_access; u32 max_rdwr_access; const struct bpf_ctx_arg_aux *ctx_arg_info; - struct bpf_prog *linked_prog; + struct mutex tgt_mutex; /* protects writing of tgt_* pointers below */ + struct bpf_prog *tgt_prog; + struct bpf_trampoline *tgt_trampoline; bool verifier_zext; /* Zero extensions has been inserted by verifier. */ bool offload_requested; bool attach_btf_trace; /* true if attaching to BTF-enabled raw tp */ bool func_proto_unreliable; bool sleepable; enum bpf_tramp_prog_type trampoline_prog_type; - struct bpf_trampoline *trampoline; struct hlist_node tramp_hlist; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 2ace56c99c36..9228af9917a8 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3706,7 +3706,7 @@ struct btf *btf_parse_vmlinux(void) struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog) { - struct bpf_prog *tgt_prog = prog->aux->linked_prog; + struct bpf_prog *tgt_prog = prog->aux->tgt_prog; if (tgt_prog) { return tgt_prog->aux->btf; @@ -3733,7 +3733,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, struct bpf_insn_access_aux *info) { const struct btf_type *t = prog->aux->attach_func_proto; - struct bpf_prog *tgt_prog = prog->aux->linked_prog; + struct bpf_prog *tgt_prog = prog->aux->tgt_prog; struct btf *btf = bpf_prog_get_target_btf(prog); const char *tname = prog->aux->attach_func_name; struct bpf_verifier_log *log = info->log; @@ -4572,7 +4572,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, return -EFAULT; } if (prog_type == BPF_PROG_TYPE_EXT) - prog_type = prog->aux->linked_prog->type; + prog_type = prog->aux->tgt_prog->type; t = btf_type_by_id(btf, t->type); if (!t || !btf_type_is_func_proto(t)) { diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ed0b3578867c..1f11484c2ec2 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -98,6 +98,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag fp->jit_requested = ebpf_jit_enabled(); INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode); + mutex_init(&fp->aux->tgt_mutex); return fp; } @@ -253,6 +254,7 @@ struct bpf_prog *bpf_prog_realloc(struct bpf_prog *fp_old, unsigned int size, void __bpf_prog_free(struct bpf_prog *fp) { if (fp->aux) { + mutex_destroy(&fp->aux->tgt_mutex); free_percpu(fp->aux->stats); kfree(fp->aux->poke_tab); kfree(fp->aux); @@ -2130,7 +2132,8 @@ static void bpf_prog_free_deferred(struct work_struct *work) if (aux->prog->has_callchain_buf) put_callchain_buffers(); #endif - bpf_trampoline_put(aux->trampoline); + if (aux->tgt_trampoline) + bpf_trampoline_put(aux->tgt_trampoline); for (i = 0; i < aux->func_cnt; i++) bpf_jit_free(aux->func[i]); if (aux->func_cnt) { @@ -2146,8 +2149,8 @@ void bpf_prog_free(struct bpf_prog *fp) { struct bpf_prog_aux *aux = fp->aux; - if (aux->linked_prog) - bpf_prog_put(aux->linked_prog); + if (aux->tgt_prog) + bpf_prog_put(aux->tgt_prog); INIT_WORK(&aux->work, bpf_prog_free_deferred); schedule_work(&aux->work); } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 4108ef3b828b..fc2bca2d9f05 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2161,7 +2161,9 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) err = PTR_ERR(tgt_prog); goto free_prog_nouncharge; } - prog->aux->linked_prog = tgt_prog; + mutex_lock(&prog->aux->tgt_mutex); + prog->aux->tgt_prog = tgt_prog; + mutex_unlock(&prog->aux->tgt_mutex); } prog->aux->offload_requested = !!attr->prog_ifindex; @@ -2498,11 +2500,22 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd) struct bpf_tracing_link { struct bpf_link link; enum bpf_attach_type attach_type; + struct bpf_trampoline *trampoline; + struct bpf_prog *tgt_prog; }; static void bpf_tracing_link_release(struct bpf_link *link) { - WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog)); + struct bpf_tracing_link *tr_link = + container_of(link, struct bpf_tracing_link, link); + + WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, + tr_link->trampoline)); + + bpf_trampoline_put(tr_link->trampoline); + + if (tr_link->tgt_prog) + bpf_prog_put(tr_link->tgt_prog); } static void bpf_tracing_link_dealloc(struct bpf_link *link) @@ -2545,7 +2558,9 @@ static const struct bpf_link_ops bpf_tracing_link_lops = { static int bpf_tracing_prog_attach(struct bpf_prog *prog) { struct bpf_link_primer link_primer; + struct bpf_prog *tgt_prog = NULL; struct bpf_tracing_link *link; + struct bpf_trampoline *tr; int err; switch (prog->type) { @@ -2583,19 +2598,38 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) &bpf_tracing_link_lops, prog); link->attach_type = prog->expected_attach_type; + mutex_lock(&prog->aux->tgt_mutex); + + if (!prog->aux->tgt_trampoline) { + err = -ENOENT; + goto out_unlock; + } + tr = prog->aux->tgt_trampoline; + tgt_prog = prog->aux->tgt_prog; + err = bpf_link_prime(&link->link, &link_primer); if (err) { - kfree(link); - goto out_put_prog; + goto out_unlock; } - err = bpf_trampoline_link_prog(prog); + err = bpf_trampoline_link_prog(prog, tr); if (err) { bpf_link_cleanup(&link_primer); - goto out_put_prog; + link = NULL; + goto out_unlock; } + link->tgt_prog = tgt_prog; + link->trampoline = tr; + + prog->aux->tgt_prog = NULL; + prog->aux->tgt_trampoline = NULL; + mutex_unlock(&prog->aux->tgt_mutex); + return bpf_link_settle(&link_primer); +out_unlock: + mutex_unlock(&prog->aux->tgt_mutex); + kfree(link); out_put_prog: bpf_prog_put(prog); return err; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 7845913e7e41..e010a0641e99 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -261,14 +261,12 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) } } -int bpf_trampoline_link_prog(struct bpf_prog *prog) +int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) { enum bpf_tramp_prog_type kind; - struct bpf_trampoline *tr; int err = 0; int cnt; - tr = prog->aux->trampoline; kind = bpf_attach_type_to_tramp(prog); mutex_lock(&tr->mutex); if (tr->extension_prog) { @@ -301,7 +299,7 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog) } hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]); tr->progs_cnt[kind]++; - err = bpf_trampoline_update(prog->aux->trampoline); + err = bpf_trampoline_update(tr); if (err) { hlist_del(&prog->aux->tramp_hlist); tr->progs_cnt[kind]--; @@ -312,13 +310,11 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog) } /* bpf_trampoline_unlink_prog() should never fail. */ -int bpf_trampoline_unlink_prog(struct bpf_prog *prog) +int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) { enum bpf_tramp_prog_type kind; - struct bpf_trampoline *tr; int err; - tr = prog->aux->trampoline; kind = bpf_attach_type_to_tramp(prog); mutex_lock(&tr->mutex); if (kind == BPF_TRAMP_REPLACE) { @@ -330,7 +326,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog) } hlist_del(&prog->aux->tramp_hlist); tr->progs_cnt[kind]--; - err = bpf_trampoline_update(prog->aux->trampoline); + err = bpf_trampoline_update(tr); out: mutex_unlock(&tr->mutex); return err; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d38678319ca4..02f704367014 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2628,8 +2628,7 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, static enum bpf_prog_type resolve_prog_type(struct bpf_prog *prog) { - return prog->aux->linked_prog ? prog->aux->linked_prog->type - : prog->type; + return prog->aux->tgt_prog ? prog->aux->tgt_prog->type : prog->type; } static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, @@ -11271,8 +11270,8 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, static int check_attach_btf_id(struct bpf_verifier_env *env) { struct bpf_prog *prog = env->prog; - struct bpf_prog *tgt_prog = prog->aux->linked_prog; u32 btf_id = prog->aux->attach_btf_id; + struct bpf_prog *tgt_prog = prog->aux->tgt_prog; struct btf_func_model fmodel; struct bpf_trampoline *tr; const struct btf_type *t; @@ -11336,7 +11335,9 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) if (IS_ERR(tr)) return PTR_ERR(tr); - prog->aux->trampoline = tr; + mutex_lock(&prog->aux->tgt_mutex); + prog->aux->tgt_trampoline = tr; + mutex_unlock(&prog->aux->tgt_mutex); return 0; } From patchwork Tue Sep 15 11:41:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 260813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D7B8C2D0E3 for ; Wed, 16 Sep 2020 00:50:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D958D20770 for ; Wed, 16 Sep 2020 00:50:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ezngq0M4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727231AbgIPAuA (ORCPT ); Tue, 15 Sep 2020 20:50:00 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:42836 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726411AbgIOLlg (ORCPT ); Tue, 15 Sep 2020 07:41:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600170068; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w9fKJXKN7VEhK7rTxzSRwDVyT2JXTZwoaKL+6VZUoBs=; b=ezngq0M4NEYBloXnPWHLQDK7SBuSo0x0m1GxKU3xU2bxOqS0aO284MIrh3+UBdXEw1gADq aYlwq1STFvDaK1dlKPP7Zv5hKSWevyUGSPgb/JeIU0ZfU3RzLvqVdVjwyYH7y4+mN1aSLC TRpo7DhfwIspl4VvOaKRq7lZpJ1dHRY= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-188-KLktm3z8P26cJ6GLSOlVSQ-1; Tue, 15 Sep 2020 07:41:06 -0400 X-MC-Unique: KLktm3z8P26cJ6GLSOlVSQ-1 Received: by mail-ed1-f72.google.com with SMTP id i23so1136803edr.14 for ; Tue, 15 Sep 2020 04:41:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=w9fKJXKN7VEhK7rTxzSRwDVyT2JXTZwoaKL+6VZUoBs=; b=Sbf1Z470L9HdO9UFiPXtY2bP7sbYUttLUApedPgvwahrSojcpzUywmj8lJEWZ+ZwTc NMCyCQlcS4gRrWONouy7BgYTwNfJaOcZ0awRQiSA6oece8dTHvEJXh3hlh+5W03oVp/U ztGHu7lLwLdANLLLjGt+2PfqlIfm8OgLqeJT36rYulNLz0vAnIWFs0hzovPYmAqFjhQF sy4LEE6C7rsxEb1X9v9lMUFHIEOyBVjd4wQYQ+5F147q4GauhbW2qxHR05lYf9dL8Yof fxwduAItgpH8QW/TW3OF2mNRN1h9VscqoalWYw5CiIRb0s1FFx0z7lyIREnkjZIZ/vUg q6Nw== X-Gm-Message-State: AOAM530MFtaD2/xZCItulzKWVM2mL4ZzFlPiSUWoPK1/FM4DFXGy0wjb uAyMfsv6fKM321BsflvzcfF7z6vZ3/Y2MmUAxgkf8yXtuzyW+cpq33ZDbSQ5lphklmF8MKTwvUc EHYQc/s+PiK33Zuc1 X-Received: by 2002:a17:906:3759:: with SMTP id e25mr19398383ejc.281.1600170065258; Tue, 15 Sep 2020 04:41:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyX7RfRm6iniM5SmaodndIFtQgzIT5Ww5VKOvVQJmL14fylehW5jLhmGjq0VOp4n5vy3qNh3w== X-Received: by 2002:a17:906:3759:: with SMTP id e25mr19398348ejc.281.1600170064896; Tue, 15 Sep 2020 04:41:04 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id y9sm9937836ejw.96.2020.09.15.04.41.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 04:41:03 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 996C01829CB; Tue, 15 Sep 2020 13:41:03 +0200 (CEST) Subject: [PATCH bpf-next v5 6/8] libbpf: add support for freplace attachment in bpf_link_create From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov Cc: Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , Andrii Nakryiko , John Fastabend , Jiri Olsa , Eelco Chaudron , KP Singh , netdev@vger.kernel.org, bpf@vger.kernel.org Date: Tue, 15 Sep 2020 13:41:03 +0200 Message-ID: <160017006352.98230.621859348254499900.stgit@toke.dk> In-Reply-To: <160017005691.98230.13648200635390228683.stgit@toke.dk> References: <160017005691.98230.13648200635390228683.stgit@toke.dk> User-Agent: StGit/0.23 MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toke Høiland-Jørgensen This adds support for supplying a target btf ID for the bpf_link_create() operation, and adds a new bpf_program__attach_freplace() high-level API for attaching freplace functions with a target. Signed-off-by: Toke Høiland-Jørgensen --- tools/lib/bpf/bpf.c | 1 + tools/lib/bpf/bpf.h | 3 ++- tools/lib/bpf/libbpf.c | 24 ++++++++++++++++++------ tools/lib/bpf/libbpf.h | 3 +++ tools/lib/bpf/libbpf.map | 1 + 5 files changed, 25 insertions(+), 7 deletions(-) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 82b983ff6569..e928456c0dd6 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -599,6 +599,7 @@ int bpf_link_create(int prog_fd, int target_fd, attr.link_create.iter_info = ptr_to_u64(OPTS_GET(opts, iter_info, (void *)0)); attr.link_create.iter_info_len = OPTS_GET(opts, iter_info_len, 0); + attr.link_create.target_btf_id = OPTS_GET(opts, target_btf_id, 0); return sys_bpf(BPF_LINK_CREATE, &attr, sizeof(attr)); } diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 015d13f25fcc..f8dbf666b62b 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -174,8 +174,9 @@ struct bpf_link_create_opts { __u32 flags; union bpf_iter_link_info *iter_info; __u32 iter_info_len; + __u32 target_btf_id; }; -#define bpf_link_create_opts__last_field iter_info_len +#define bpf_link_create_opts__last_field target_btf_id LIBBPF_API int bpf_link_create(int prog_fd, int target_fd, enum bpf_attach_type attach_type, diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 550950eb1860..165131c73f40 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -9322,12 +9322,14 @@ static struct bpf_link *attach_iter(const struct bpf_sec_def *sec, static struct bpf_link * bpf_program__attach_fd(struct bpf_program *prog, int target_fd, - const char *target_name) + int target_btf_id, const char *target_name) { enum bpf_attach_type attach_type; char errmsg[STRERR_BUFSIZE]; struct bpf_link *link; int prog_fd, link_fd; + DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts, + .target_btf_id = target_btf_id); prog_fd = bpf_program__fd(prog); if (prog_fd < 0) { @@ -9340,8 +9342,12 @@ bpf_program__attach_fd(struct bpf_program *prog, int target_fd, return ERR_PTR(-ENOMEM); link->detach = &bpf_link__detach_fd; - attach_type = bpf_program__get_expected_attach_type(prog); - link_fd = bpf_link_create(prog_fd, target_fd, attach_type, NULL); + if (bpf_program__get_type(prog) == BPF_PROG_TYPE_EXT) + attach_type = BPF_TRACE_FREPLACE; + else + attach_type = bpf_program__get_expected_attach_type(prog); + + link_fd = bpf_link_create(prog_fd, target_fd, attach_type, &opts); if (link_fd < 0) { link_fd = -errno; free(link); @@ -9357,19 +9363,25 @@ bpf_program__attach_fd(struct bpf_program *prog, int target_fd, struct bpf_link * bpf_program__attach_cgroup(struct bpf_program *prog, int cgroup_fd) { - return bpf_program__attach_fd(prog, cgroup_fd, "cgroup"); + return bpf_program__attach_fd(prog, cgroup_fd, 0, "cgroup"); } struct bpf_link * bpf_program__attach_netns(struct bpf_program *prog, int netns_fd) { - return bpf_program__attach_fd(prog, netns_fd, "netns"); + return bpf_program__attach_fd(prog, netns_fd, 0, "netns"); } struct bpf_link *bpf_program__attach_xdp(struct bpf_program *prog, int ifindex) { /* target_fd/target_ifindex use the same field in LINK_CREATE */ - return bpf_program__attach_fd(prog, ifindex, "xdp"); + return bpf_program__attach_fd(prog, ifindex, 0, "xdp"); +} + +struct bpf_link *bpf_program__attach_freplace(struct bpf_program *prog, + int target_fd, int target_btf_id) +{ + return bpf_program__attach_fd(prog, target_fd, target_btf_id, "freplace"); } struct bpf_link * diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index a750f67a23f6..ce5add9b9203 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -261,6 +261,9 @@ LIBBPF_API struct bpf_link * bpf_program__attach_netns(struct bpf_program *prog, int netns_fd); LIBBPF_API struct bpf_link * bpf_program__attach_xdp(struct bpf_program *prog, int ifindex); +LIBBPF_API struct bpf_link * +bpf_program__attach_freplace(struct bpf_program *prog, + int target_fd, int target_btf_id); struct bpf_map; diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 92ceb48a5ca2..3cc2c42f435d 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -302,6 +302,7 @@ LIBBPF_0.1.0 { LIBBPF_0.2.0 { global: + bpf_program__attach_freplace; bpf_program__section_name; perf_buffer__buffer_cnt; perf_buffer__buffer_fd; From patchwork Tue Sep 15 11:41:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 260814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A31D4C2D0F6 for ; Wed, 16 Sep 2020 00:49:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6DAF02076B for ; Wed, 16 Sep 2020 00:49:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ann15Hem" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726649AbgIPAtr (ORCPT ); Tue, 15 Sep 2020 20:49:47 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:48203 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726414AbgIOLlh (ORCPT ); Tue, 15 Sep 2020 07:41:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600170069; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9f/6eHNeg0s6Jj9GHDvmuH44xs14vn5rt2HXh2zzaBw=; b=Ann15HemyMVY+SocGqlfTlBicAURGZa+BK5rbVp1hg7HLqQWOL6CA/BjrQgLpxJak+noaB O82uLCi82+jxUq9ta2kSdHVskP+Xdf1C8p2yW1173cXYvyRaOuOT9Bo3hE0dHUmiKgduFg j+qO9awzoy7wL6+Uq4HY5+YycpybNYs= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-252-RDmu7ny9PiGiLrv0aF2ffA-1; Tue, 15 Sep 2020 07:41:07 -0400 X-MC-Unique: RDmu7ny9PiGiLrv0aF2ffA-1 Received: by mail-ed1-f71.google.com with SMTP id c25so1128898edx.11 for ; Tue, 15 Sep 2020 04:41:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=9f/6eHNeg0s6Jj9GHDvmuH44xs14vn5rt2HXh2zzaBw=; b=oRI7kExLk+pU4O5ZaDUWlyC6MPXjQ9OMVF/phYL50ScyemfldPE9USb+l3TRz44zQU LoDtjg55d4T/9iuwq6EvQL+7cUMS8LTP2IbYghq8eFgsaAfjSvNcAnkkil3059wE1XiP 2ZzND3ZAg2BdmCDzb/yagHZD8PwKJwl7V+TcpUh+phvyLFq9Uf0ZWE902jZLgBUfUg1m Dgch4ZzrT8xGEYONIEMBDLtP4nAw/Bqz4cnvZ7oR/hGqrl3OsZE79Mk8CKT+NjtbBBji AuJo+qJ5QP2pwPz7ok3TIutCrsWzPvgRunrowKmtuWIe6Zv/Cmh9CBtZ5gxEmG1bxK1s 734A== X-Gm-Message-State: AOAM532GGHzeJ5PFv3FCBLuWJDxvbvpwMOT6HrST9bHPF0sO4o8qGpEJ DlkxhVjCHK2OUE4RMGKt8GMjTffmoUsAz4crmqbdPDP26kjgwyrk3LZA++mCP91H9yaiSR38bbh ssWxvhrsHYJom83R2 X-Received: by 2002:a05:6402:1818:: with SMTP id g24mr21490389edy.332.1600170066024; Tue, 15 Sep 2020 04:41:06 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzLf225edBty+O7aa8SiXMzOyvo/W/zux7fXC+lFDqqW/qjopGbr/zoKWlpy6Y2vJrpUWid+g== X-Received: by 2002:a05:6402:1818:: with SMTP id g24mr21490369edy.332.1600170065615; Tue, 15 Sep 2020 04:41:05 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id o3sm11488272edt.79.2020.09.15.04.41.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 04:41:05 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id B2F2E1829CC; Tue, 15 Sep 2020 13:41:04 +0200 (CEST) Subject: [PATCH bpf-next v5 7/8] selftests: add test for multiple attachments of freplace program From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov Cc: Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , Andrii Nakryiko , John Fastabend , Jiri Olsa , Eelco Chaudron , KP Singh , netdev@vger.kernel.org, bpf@vger.kernel.org Date: Tue, 15 Sep 2020 13:41:04 +0200 Message-ID: <160017006463.98230.8855335735634444835.stgit@toke.dk> In-Reply-To: <160017005691.98230.13648200635390228683.stgit@toke.dk> References: <160017005691.98230.13648200635390228683.stgit@toke.dk> User-Agent: StGit/0.23 MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toke Høiland-Jørgensen This adds a selftest for attaching an freplace program to multiple targets simultaneously. Signed-off-by: Toke Høiland-Jørgensen --- .../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 171 ++++++++++++++++---- .../selftests/bpf/progs/freplace_get_constant.c | 15 ++ 2 files changed, 154 insertions(+), 32 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/freplace_get_constant.c diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c index eda682727787..3621d66e47dd 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c @@ -2,36 +2,79 @@ /* Copyright (c) 2019 Facebook */ #include #include +#include + +typedef int (*test_cb)(struct bpf_object *obj); + +static int check_data_map(struct bpf_object *obj, int prog_cnt, bool reset) +{ + struct bpf_map *data_map = NULL, *map; + __u64 *result = NULL; + const int zero = 0; + __u32 duration = 0; + int ret = -1, i; + + result = malloc((prog_cnt + 32 /* spare */) * sizeof(__u64)); + if (CHECK(!result, "alloc_memory", "failed to alloc memory")) + return -ENOMEM; + + bpf_object__for_each_map(map, obj) + if (bpf_map__is_internal(map)) { + data_map = map; + break; + } + if (CHECK(!data_map, "find_data_map", "data map not found\n")) + goto out; + + ret = bpf_map_lookup_elem(bpf_map__fd(data_map), &zero, result); + if (CHECK(ret, "get_result", + "failed to get output data: %d\n", ret)) + goto out; + + for (i = 0; i < prog_cnt; i++) { + if (CHECK(result[i] != 1, "result", + "fexit_bpf2bpf result[%d] failed err %llu\n", + i, result[i])) + goto out; + result[i] = 0; + } + if (reset) { + ret = bpf_map_update_elem(bpf_map__fd(data_map), &zero, result, 0); + if (CHECK(ret, "reset_result", "failed to reset result\n")) + goto out; + } + + ret = 0; +out: + free(result); + return ret; +} static void test_fexit_bpf2bpf_common(const char *obj_file, const char *target_obj_file, int prog_cnt, const char **prog_name, - bool run_prog) + bool run_prog, + test_cb cb) { - struct bpf_object *obj = NULL, *pkt_obj; - int err, pkt_fd, i; - struct bpf_link **link = NULL; + struct bpf_object *obj = NULL, *tgt_obj; struct bpf_program **prog = NULL; + struct bpf_link **link = NULL; __u32 duration = 0, retval; - struct bpf_map *data_map; - const int zero = 0; - __u64 *result = NULL; + int err, tgt_fd, i; err = bpf_prog_load(target_obj_file, BPF_PROG_TYPE_UNSPEC, - &pkt_obj, &pkt_fd); + &tgt_obj, &tgt_fd); if (CHECK(err, "tgt_prog_load", "file %s err %d errno %d\n", target_obj_file, err, errno)) return; DECLARE_LIBBPF_OPTS(bpf_object_open_opts, opts, - .attach_prog_fd = pkt_fd, + .attach_prog_fd = tgt_fd, ); link = calloc(sizeof(struct bpf_link *), prog_cnt); prog = calloc(sizeof(struct bpf_program *), prog_cnt); - result = malloc((prog_cnt + 32 /* spare */) * sizeof(__u64)); - if (CHECK(!link || !prog || !result, "alloc_memory", - "failed to alloc memory")) + if (CHECK(!link || !prog, "alloc_memory", "failed to alloc memory")) goto close_prog; obj = bpf_object__open_file(obj_file, &opts); @@ -53,39 +96,33 @@ static void test_fexit_bpf2bpf_common(const char *obj_file, goto close_prog; } - if (!run_prog) - goto close_prog; + if (cb) { + err = cb(obj); + if (err) + goto close_prog; + } - data_map = bpf_object__find_map_by_name(obj, "fexit_bp.bss"); - if (CHECK(!data_map, "find_data_map", "data map not found\n")) + if (!run_prog) goto close_prog; - err = bpf_prog_test_run(pkt_fd, 1, &pkt_v6, sizeof(pkt_v6), + err = bpf_prog_test_run(tgt_fd, 1, &pkt_v6, sizeof(pkt_v6), NULL, NULL, &retval, &duration); CHECK(err || retval, "ipv6", "err %d errno %d retval %d duration %d\n", err, errno, retval, duration); - err = bpf_map_lookup_elem(bpf_map__fd(data_map), &zero, result); - if (CHECK(err, "get_result", - "failed to get output data: %d\n", err)) + if (check_data_map(obj, prog_cnt, false)) goto close_prog; - for (i = 0; i < prog_cnt; i++) - if (CHECK(result[i] != 1, "result", "fexit_bpf2bpf failed err %llu\n", - result[i])) - goto close_prog; - close_prog: for (i = 0; i < prog_cnt; i++) if (!IS_ERR_OR_NULL(link[i])) bpf_link__destroy(link[i]); if (!IS_ERR_OR_NULL(obj)) bpf_object__close(obj); - bpf_object__close(pkt_obj); + bpf_object__close(tgt_obj); free(link); free(prog); - free(result); } static void test_target_no_callees(void) @@ -96,7 +133,7 @@ static void test_target_no_callees(void) test_fexit_bpf2bpf_common("./fexit_bpf2bpf_simple.o", "./test_pkt_md_access.o", ARRAY_SIZE(prog_name), - prog_name, true); + prog_name, true, NULL); } static void test_target_yes_callees(void) @@ -110,7 +147,7 @@ static void test_target_yes_callees(void) test_fexit_bpf2bpf_common("./fexit_bpf2bpf.o", "./test_pkt_access.o", ARRAY_SIZE(prog_name), - prog_name, true); + prog_name, true, NULL); } static void test_func_replace(void) @@ -128,7 +165,7 @@ static void test_func_replace(void) test_fexit_bpf2bpf_common("./fexit_bpf2bpf.o", "./test_pkt_access.o", ARRAY_SIZE(prog_name), - prog_name, true); + prog_name, true, NULL); } static void test_func_replace_verify(void) @@ -139,7 +176,75 @@ static void test_func_replace_verify(void) test_fexit_bpf2bpf_common("./freplace_connect4.o", "./connect4_prog.o", ARRAY_SIZE(prog_name), - prog_name, false); + prog_name, false, NULL); +} + +static int test_second_attach(struct bpf_object *obj) +{ + const char *prog_name = "freplace/get_constant"; + const char *tgt_name = prog_name + 9; /* cut off freplace/ */ + const char *tgt_obj_file = "./test_pkt_access.o"; + int err = 0, tgt_fd, tgt_btf_id, link_fd = -1; + struct bpf_program *prog = NULL; + struct bpf_object *tgt_obj; + __u32 duration = 0, retval; + struct btf *btf; + + prog = bpf_object__find_program_by_title(obj, prog_name); + if (CHECK(!prog, "find_prog", "prog %s not found\n", prog_name)) + return -ENOENT; + + err = bpf_prog_load(tgt_obj_file, BPF_PROG_TYPE_UNSPEC, + &tgt_obj, &tgt_fd); + if (CHECK(err, "second_prog_load", "file %s err %d errno %d\n", + tgt_obj_file, err, errno)) + return err; + + btf = bpf_object__btf(tgt_obj); + tgt_btf_id = btf__find_by_name_kind(btf, tgt_name, BTF_KIND_FUNC); + if (CHECK(tgt_btf_id < 0, "find_btf", "no BTF ID found for func %s\n", prog_name)) { + err = -ENOENT; + goto out; + } + + DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts, + .target_btf_id = tgt_btf_id, + ); + link_fd = bpf_link_create(bpf_program__fd(prog), tgt_fd, + BPF_TRACE_FREPLACE, &opts); + if (CHECK(link_fd < 0, "second_link", "err %d errno %d", + link_fd, errno)) { + err = link_fd; + goto out; + } + + err = bpf_prog_test_run(tgt_fd, 1, &pkt_v6, sizeof(pkt_v6), + NULL, NULL, &retval, &duration); + if (CHECK(err || retval, "ipv6", + "err %d errno %d retval %d duration %d\n", + err, errno, retval, duration)) + goto out; + + err = check_data_map(obj, 1, true); + if (err) + goto out; + +out: + if (link_fd >= 0) + close(link_fd); + bpf_object__close(tgt_obj); + return err; +} + +static void test_func_replace_multi(void) +{ + const char *prog_name[] = { + "freplace/get_constant", + }; + test_fexit_bpf2bpf_common("./freplace_get_constant.o", + "./test_pkt_access.o", + ARRAY_SIZE(prog_name), + prog_name, true, test_second_attach); } static void test_func_sockmap_update(void) @@ -150,7 +255,7 @@ static void test_func_sockmap_update(void) test_fexit_bpf2bpf_common("./freplace_cls_redirect.o", "./test_cls_redirect.o", ARRAY_SIZE(prog_name), - prog_name, false); + prog_name, false, NULL); } static void test_obj_load_failure_common(const char *obj_file, @@ -222,4 +327,6 @@ void test_fexit_bpf2bpf(void) test_func_replace_return_code(); if (test__start_subtest("func_map_prog_compatibility")) test_func_map_prog_compatibility(); + if (test__start_subtest("func_replace_multi")) + test_func_replace_multi(); } diff --git a/tools/testing/selftests/bpf/progs/freplace_get_constant.c b/tools/testing/selftests/bpf/progs/freplace_get_constant.c new file mode 100644 index 000000000000..8f0ecf94e533 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/freplace_get_constant.c @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +volatile __u64 test_get_constant = 0; +SEC("freplace/get_constant") +int new_get_constant(long val) +{ + if (val != 123) + return 0; + test_get_constant = 1; + return test_get_constant; /* original get_constant() returns val - 122 */ +} +char _license[] SEC("license") = "GPL";