From patchwork Mon Mar 11 09:35:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 779588 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3649D2C69A for ; Mon, 11 Mar 2024 09:34:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710149685; cv=none; b=sMutQByfqZu0mtqp1PJkTMCa9FQvoNHH6t4CuTrHBPVAW3PvNRKbuz38zDipAgD0NskSWPwTjw6DZ4lkwkaUdWqxrV+DLJnUPOl8AgeUi4X7M+Ye7s4/ug1wogCFJOsaCrRq2EVC0YCPQkFWBO3SHVqdcishrk07P8d72vK2WSY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710149685; c=relaxed/simple; bh=gPSHW2/9unWuAhFncICuut3hiBk/07uto75Q0YZZP+U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BBEqwjTDvZtCc0j/MQrUmBLTbtoSODDhP2MGS1WGxizcaB/ZM55dxk6bWDp0MlZfyIbcMv2YHGi4pOpmvtp9KbDFK6J5z27zK4/rFh3rp6yjPxc6nC0t2lPxwhmGheRlQ7RotjM2vtzUdBE4efnUpEQ8Q58IzbyeV6yupimGO1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=fCndtFff; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="fCndtFff" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6e6082eab17so3801193b3a.1 for ; Mon, 11 Mar 2024 02:34:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149683; x=1710754483; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QjSWV41JlB3dq7HwIpKEF4e0TQa/o2d+dOhHoSuTF7U=; b=fCndtFffu1Ss2NcYN91svt0lbgvg/8XEuiKcZFBYKp7qeeF+XLcmbg+m63YznekLld GRZI7k/jvSHxYlpUhAl51sQNowp1tN0KStmqyTY58z+irKn+wdBqUqbuygyeyIrgBtEU t5F747/SOt7yRqdumXdtJrM2+s7N27vDv/23Tc/kpoR86DnCGK5aM5jRiUfNM2R9d8LX QXoebEhGQoGQzulAGDVWHrS43KwnxkOLJvXdDsl3XESVcCGBiArmh9gtRJBpz5sElNA/ Moey35A9/O63/cbjNgddzv+E44rO0Q1G7dbN91co4eRiYSx6W5P9WJ8/6UKao6Mbsq+6 sEHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149683; x=1710754483; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QjSWV41JlB3dq7HwIpKEF4e0TQa/o2d+dOhHoSuTF7U=; b=Y4uIZKneMt9MR9oP6hliU55cLeyV+IcGrvhoivKHrdOstoOIHvWUiKj9d5KdnpdOTZ vC4OURR4Cp1VP92uAc1NNHENKjYJUchSOYYjPqQTO/GHlRmUM/Kvld5puPTedm6ScGe5 dhnsctjvBmn+gmyism1oNJaLbU7NV4hYrVtpswvvR7LXnEpYybqOXx6VbdlKRLUlcq1V Sq3/c0110PVblMwDa31XLa6qLMJQOtiNa3LJQ839obaaI7rSVe/nhKfGsvRXngxNyL2Z sbLe8QflhS/cCZdZxIvqO0AaGfbDJie+kNsn8WXGDc/8Eu9y1fA2mu87QJSIL/oZQKzL avag== X-Forwarded-Encrypted: i=1; AJvYcCXeSJfpgZdxqJmJAgErTq75L5Dnwv2vrYaw+rsUdMx6Fpw7M50w7tQeA8mmHGmZtps/hDdptHV0y2eMY53drSy7Fph86U66pmrkNyAfwOim X-Gm-Message-State: AOJu0Yz87RAgbSLLVrgzUyUJhSms1gv9prb8zMNlZPGncZPFm8b/XqRw MwiFp+82p5LGCxrBuF5+4rIkwpGHXcOvLjXsNaiiFddjI7Fa7oYbFwiFrLcR2eU= X-Google-Smtp-Source: AGHT+IFbj6YfcHBYvXMM2NWizkuYpniqg3bImjqE4WhHnt6u4m7Hz7unmoAwRceekSZcHEkG6VDAnw== X-Received: by 2002:a05:6a20:438a:b0:1a1:431c:6696 with SMTP id i10-20020a056a20438a00b001a1431c6696mr4574882pzl.49.1710149683554; Mon, 11 Mar 2024 02:34:43 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.34.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:34:43 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 2/9] bpf: refactor the modules_array to ptr_array Date: Mon, 11 Mar 2024 17:35:19 +0800 Message-Id: <20240311093526.1010158-3-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Refactor the struct modules_array to more general struct ptr_array, which is used to store the pointers. Meanwhiles, introduce the bpf_try_add_ptr(), which checks the existing of the ptr before adding it to the array. Seems it should be moved to another files in "lib", and I'm not sure where to add it now, and let's move it to kernel/bpf/syscall.c for now. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 10 +++++++++ kernel/bpf/syscall.c | 37 +++++++++++++++++++++++++++++++ kernel/trace/bpf_trace.c | 48 ++++++---------------------------------- 3 files changed, 54 insertions(+), 41 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 0f677fdcfcc7..997765cdf474 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -304,6 +304,16 @@ struct bpf_map { s64 __percpu *elem_count; }; +struct ptr_array { + void **ptrs; + int cnt; + int cap; +}; + +int bpf_add_ptr(struct ptr_array *arr, void *ptr); +bool bpf_has_ptr(struct ptr_array *arr, struct module *mod); +int bpf_try_add_ptr(struct ptr_array *arr, void *ptr); + static inline const char *btf_field_type_name(enum btf_field_type type) { switch (type) { diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index f63f4da4db5e..4f230fd1f8e4 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -479,6 +479,43 @@ static void bpf_map_release_memcg(struct bpf_map *map) } #endif +int bpf_add_ptr(struct ptr_array *arr, void *ptr) +{ + void **ptrs; + + if (arr->cnt == arr->cap) { + arr->cap = max(16, arr->cap * 3 / 2); + ptrs = krealloc_array(arr->ptrs, arr->cap, sizeof(*ptrs), GFP_KERNEL); + if (!ptrs) + return -ENOMEM; + arr->ptrs = ptrs; + } + + arr->ptrs[arr->cnt] = ptr; + arr->cnt++; + return 0; +} + +bool bpf_has_ptr(struct ptr_array *arr, struct module *mod) +{ + int i; + + for (i = arr->cnt - 1; i >= 0; i--) { + if (arr->ptrs[i] == mod) + return true; + } + return false; +} + +int bpf_try_add_ptr(struct ptr_array *arr, void *ptr) +{ + if (bpf_has_ptr(arr, ptr)) + return -EEXIST; + if (bpf_add_ptr(arr, ptr)) + return -ENOMEM; + return 0; +} + static int btf_field_cmp(const void *a, const void *b) { const struct btf_field *f1 = a, *f2 = b; diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 241ddf5e3895..791e97a3f8e3 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2873,43 +2873,9 @@ static void symbols_swap_r(void *a, void *b, int size, const void *priv) } } -struct modules_array { - struct module **mods; - int mods_cnt; - int mods_cap; -}; - -static int add_module(struct modules_array *arr, struct module *mod) -{ - struct module **mods; - - if (arr->mods_cnt == arr->mods_cap) { - arr->mods_cap = max(16, arr->mods_cap * 3 / 2); - mods = krealloc_array(arr->mods, arr->mods_cap, sizeof(*mods), GFP_KERNEL); - if (!mods) - return -ENOMEM; - arr->mods = mods; - } - - arr->mods[arr->mods_cnt] = mod; - arr->mods_cnt++; - return 0; -} - -static bool has_module(struct modules_array *arr, struct module *mod) -{ - int i; - - for (i = arr->mods_cnt - 1; i >= 0; i--) { - if (arr->mods[i] == mod) - return true; - } - return false; -} - static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u32 addrs_cnt) { - struct modules_array arr = {}; + struct ptr_array arr = {}; u32 i, err = 0; for (i = 0; i < addrs_cnt; i++) { @@ -2918,7 +2884,7 @@ static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u3 preempt_disable(); mod = __module_address(addrs[i]); /* Either no module or we it's already stored */ - if (!mod || has_module(&arr, mod)) { + if (!mod || bpf_has_ptr(&arr, mod)) { preempt_enable(); continue; } @@ -2927,7 +2893,7 @@ static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u3 preempt_enable(); if (err) break; - err = add_module(&arr, mod); + err = bpf_add_ptr(&arr, mod); if (err) { module_put(mod); break; @@ -2936,14 +2902,14 @@ static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u3 /* We return either err < 0 in case of error, ... */ if (err) { - kprobe_multi_put_modules(arr.mods, arr.mods_cnt); - kfree(arr.mods); + kprobe_multi_put_modules((struct module **)arr.ptrs, arr.cnt); + kfree(arr.ptrs); return err; } /* or number of modules found if everything is ok. */ - *mods = arr.mods; - return arr.mods_cnt; + *mods = (struct module **)arr.ptrs; + return arr.cnt; } static int addrs_check_error_injection_list(unsigned long *addrs, u32 cnt) From patchwork Mon Mar 11 09:35:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 779587 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B59038F9A for ; Mon, 11 Mar 2024 09:34:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710149701; cv=none; b=qTCoOVfHOWUdjWj/61mpp4KBHHkd5sKmw/H5ELWcrShWnh6xG7aSNR+p0iwE6X8smhCUAIoxH57fjSP2JJ8zq5JCfzT1mBf4nQdBM8UDbK33UZgAKH9tohY0870rWR+mh0d6Jp2Jz80q47yAfeDkwTCeScZ34lwqIn4iyQy8nj0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710149701; c=relaxed/simple; bh=TE0qpmdPB+wes4LW3PVEl7Os9qDezEyQM77BodsjCrg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iJhm3pn71q/pN1eRj+T9zNWyaQ1Y3F0CxXGKeAhnXiHKhGaD+JNvfWlgRW/4R4/8FF8oNJix+7Xnx7oUtz1T1XBOmbhXF9n6K0w6LAjasSy76wkOOM+alW15zBr9kx9JQ+Q2YSQ3gAFfaoWvosdwdnGjQs9Rm78bV4w5aVCyTW4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=adpl9kIe; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="adpl9kIe" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-53fbf2c42bfso2555542a12.3 for ; Mon, 11 Mar 2024 02:34:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149699; x=1710754499; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mQHWXqakG53bwp/DHGcig31BilBtYFxqqjfUWdtPR6g=; b=adpl9kIeZoKs45VwHW8g6CUq/VGzVREUPqjUfsVt7/FJN2vEnM8n/ShhzcJnC4rNrz 0IehkKKhiHwR+f1CEoQkIMfD8bFCQ9Mc8Qz3YyHqcyt4+jwsBUONCAL4dcRcix4xK9ei 7XqXA1FRABEGSyWGJGB5ZdPu8yvuUNq7kNsEVhFlSKrQjCEMSvzcXg9JOP+XitxZva/Z DXUjXG9ekOBe71+TZng1vW4kVf0wTRluMLo+Yc6D12jY5nXVqXEFAjuC5xo4OT3JSDHN 1Kw8IiwUHDogxYKBFj/1gxH4phK2gyfH4Ho9yALQK88PnPSqT7YVJ1h3/JrAaXL/BY6q incg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149699; x=1710754499; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mQHWXqakG53bwp/DHGcig31BilBtYFxqqjfUWdtPR6g=; b=a0YQVQh+GlJbZnsbVr5FE8Mmq23f2oNSI2EqkauG+wpd1iCDgYDLz3GBCVsouEwVX/ cvuvSPY+m13yDwpTqUH9D+EN0feXjQf6NeGw5Tj3yPqmrzxoLrSUcwnZ5acr3laDXX0T EbKNZOUaCRiqS6RVAOfPKGHWMx5LYFtS/FgCKIC7wfzgoT2mWr48gLw5KHRN7o5RFNCQ 6LizSegRBps2Mt808hd356HRk3xlFIBMUUvCuQjSoRl8AkP7ofmZYxALkaCScQBAhSrd syFkTs7XKaOecpBBqkeniYObjDNdGUhqvZtRXH9Ymwk38dI21QDSMbk2GWQ9uC/ShWrH yWxA== X-Forwarded-Encrypted: i=1; AJvYcCXiTLuXzAWvyyiOrLUjIH3mXafDlFDKw1y+MioOFV2+OFgIKPUbt7gYA5yB57YC5Hg3yL5uQ54m03K+GRvYg5qJZT/a0ChXctDChrxVB/mI X-Gm-Message-State: AOJu0YzuqqVPAJttpU41BgM1tcM8B5pcKDxOUA69TQudxWF94O9/M3SF mwMhvDnb4gbvUbmiwkN+jx4k2Am3Y3vwqlQr17F974ut01FWe7mWICr5eSOAVyk= X-Google-Smtp-Source: AGHT+IHwrlcPzEPWwJhB9wbDVFgwEUaTXK8PIDDimv1XZruHWD6zTTqAqef2QwBY7Qc+7bmyEVvx+w== X-Received: by 2002:a05:6a20:6f8f:b0:1a3:1d7e:342b with SMTP id gv15-20020a056a206f8f00b001a31d7e342bmr631985pzb.48.1710149698990; Mon, 11 Mar 2024 02:34:58 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.34.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:34:58 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 4/9] bpf: trampoline: introduce bpf_tramp_multi_link Date: Mon, 11 Mar 2024 17:35:21 +0800 Message-Id: <20240311093526.1010158-5-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Introduce the struct bpf_tramp_multi_link, which is used to attach a bpf_link to multi trampoline. Meanwhile, introduce corresponding function bpf_trampoline_multi_{link,unlink}_prog. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 14 ++++++++++++ kernel/bpf/trampoline.c | 47 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 2b5cd6100fc4..4e8f17d9f022 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -57,6 +57,7 @@ struct user_namespace; struct super_block; struct inode; struct bpf_tramp_link; +struct bpf_tramp_multi_link; extern struct idr btf_idr; extern spinlock_t btf_idr_lock; @@ -1282,6 +1283,8 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs); +int bpf_trampoline_multi_link_prog(struct bpf_tramp_multi_link *link); +int bpf_trampoline_multi_unlink_prog(struct bpf_tramp_multi_link *link); /* * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn @@ -1614,6 +1617,17 @@ struct bpf_shim_tramp_link { struct bpf_trampoline *trampoline; }; +struct bpf_tramp_multi_link_entry { + struct bpf_trampoline *trampoline; + struct bpf_tramp_link_conn conn; +}; + +struct bpf_tramp_multi_link { + struct bpf_link link; + u32 cnt; + struct bpf_tramp_multi_link_entry *entries; +}; + struct bpf_tracing_link { struct bpf_tramp_link link; enum bpf_attach_type attach_type; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index cf9b84f785f3..2167aa3fe583 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -607,6 +607,53 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampolin return err; } +static int __bpf_trampoline_multi_unlink_prog(struct bpf_tramp_multi_link *link, + u32 cnt) +{ + struct bpf_tramp_multi_link_entry *entry; + struct bpf_trampoline *tr; + int err = 0, i; + + for (i = 0; i < cnt; i++) { + entry = &link->entries[i]; + tr = entry->trampoline; + mutex_lock(&tr->mutex); + err = __bpf_trampoline_unlink_prog(&entry->conn, + entry->trampoline); + mutex_unlock(&tr->mutex); + if (err) + break; + } + return err; +} + +int bpf_trampoline_multi_unlink_prog(struct bpf_tramp_multi_link *link) +{ + return __bpf_trampoline_multi_unlink_prog(link, link->cnt); +} + +int bpf_trampoline_multi_link_prog(struct bpf_tramp_multi_link *link) +{ + struct bpf_tramp_multi_link_entry *entry; + struct bpf_trampoline *tr; + int err = 0, i; + + for (i = 0; i < link->cnt; i++) { + entry = &link->entries[i]; + tr = entry->trampoline; + mutex_lock(&tr->mutex); + err = __bpf_trampoline_link_prog(&entry->conn, tr); + mutex_unlock(&tr->mutex); + if (err) + goto unlink; + } + + return 0; +unlink: + __bpf_trampoline_multi_unlink_prog(link, i); + return err; +} + #if defined(CONFIG_CGROUP_BPF) && defined(CONFIG_BPF_LSM) static void bpf_shim_tramp_link_release(struct bpf_link *link) { From patchwork Mon Mar 11 09:35:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 779586 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECF403C097 for ; Mon, 11 Mar 2024 09:35:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710149717; cv=none; b=VcvxqO5lgtzikJyOdj4myylA1digp9b7M/N2DFLS3PU+jtKhk5Bxv8mAZfpQa9X0qLe3c7bh1cJd8342Bc4S9TLMrVjzAQBuwLND5v6EePs/lgjBhGz3Z40cHJWZqOOknHsGj2q/DbQPUChqXzRoSAo5uHILc9upuA8T1+Awv3c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710149717; c=relaxed/simple; bh=5yrNkXsmiVwG0cDbfE3tyiE7ULVW96PO0jWtPWePKH4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=X1goSKPSi99g79xiohRDLMZG6uMAhAPQh15oNFsanfTZuJ9q42AwrN6qRTkoGE4IcLdEEo1iwNN6+oeMOr7KFi/l+HzM4RGNUINEQGjVjx3KNG95SZfxWN8rIvHnOTQ+iR5SbapAG+//oi/01rx2qooly/KumrbVqcSFwtg4s+o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PJ37o6q+; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PJ37o6q+" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1dc49b00bdbso19974465ad.3 for ; Mon, 11 Mar 2024 02:35:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149714; x=1710754514; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mXY72K13/QWs+XYqq+1yre2I5i9spjZlqZiX5c3uWc8=; b=PJ37o6q+ts5vfjZv9xBt2UU5r8dmqXF0DMmVsVWfvlSMvHivUOeGyMxS3OoCy2OwPe KN5mpZilEOqN6S3akNnE0dVoY6bjke7s/vVU0ieNlMTe8mrxRPjtPAFD9GIQrekqY11O 0eaj6k/hSjxHYx63BiofKluEVKgxIdncBEq6jR68yi/GjOctI4oLxYYw64niqVKCd8DP +3sL5YB9JhYdSq66y+WEJ6gw2LdJuwLFsKWnSjlHQ5XJBW4opAA0stmvzYTHA7uRsQSw 6Kv5w1W4TX+Az18y2e/xUqZsYq9Jkm4YVJzkl950gwJkblv50tdnUgc1tcbCec6mtyE2 eYew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149714; x=1710754514; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mXY72K13/QWs+XYqq+1yre2I5i9spjZlqZiX5c3uWc8=; b=XGg/wzQS0zg4aYMLsEB5ee7fxr+yNI3ApmA9ujd4x95uGLw00MlMGQ2DG11PeTCOPL WrWcoy6X5n6yk9EreUT/24qPErvSyuEfxNNyJCS+fp2QwpYjf2/5XRCx595m2IrH63w1 2AR4Tl33h9cCrHFCwjMJYfYuvhkY4SHeQaRjq8H8PAHAjydXn3LJU/70kfB3k7rQrGVU TFCP5xLWI0z29F2SF6e6h08TjfPeDdSqOZbD3dv3xMLIIq4d7TuCFtZwr9tP88HTTmDE EcAOTFGLvew8Cz6dNipCW7J+jmozw3PDrKlKQHZEf7z7XuyB0uJclypW0V+WJpB+3/Y5 DUxQ== X-Forwarded-Encrypted: i=1; AJvYcCWFqyZsHeJBOFTDwafawhK1fYtHUVfeXqmnxQ6AxZwq83JWLGlzVQkgWfOViyZz8F2QffmFahSVvJctle38b0fKqqSI+MGBKPnZg2bjUAg/ X-Gm-Message-State: AOJu0Yx+v2C5+fiwhqoCwLATmup8JXJcdhREeaEKdoSdiHAREL440tZM UkqPukxOBbWwjq3JWJL7Uqg0PeWxzn78VtfSgMbH4LmyfYvVI5cK0aWRhlapaCk= X-Google-Smtp-Source: AGHT+IG1tHZkGA6ilF0C9eA74u4gWwk/Df5+HU+geMG3CnFVxYVwwj+wjdOVgVKG4WncqAPf+3Dvvw== X-Received: by 2002:a17:902:e806:b0:1dd:88cf:205c with SMTP id u6-20020a170902e80600b001dd88cf205cmr4529480plg.28.1710149714274; Mon, 11 Mar 2024 02:35:14 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.35.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:35:14 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 6/9] bpf: tracing: add multi-link support Date: Mon, 11 Mar 2024 17:35:23 +0800 Message-Id: <20240311093526.1010158-7-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In this commit, we add the support to allow attaching a tracing BPF program to multi hooks, which is similar to BPF_TRACE_KPROBE_MULTI. The use case is obvious. For now, we have to create a BPF program for each kernel function, for which we want to trace, even through all the program have the same (or similar logic). This can consume extra memory, and make the program loading slow if we have plenty of kernel function to trace. The KPROBE_MULTI maybe a alternative, but it can't do what TRACING do. For example, the kretprobe can't obtain the function args, but the FEXIT can. For now, we support to create multi-link for fentry/fexit/modify_return with the following new attach types that we introduce: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI We introduce the struct bpf_tracing_multi_link for this purpose, which can hold all the kernel modules, target bpf program (for attaching to bpf program) or target btf (for attaching to kernel function) that we referenced. Meanwhiles, every trampoline for the function that we attach to is also stored in "struct bpf_tramp_multi_link_entry *entries" in struct bpf_tramp_multi_link. During loading, the first target is used for verification by the verifer. And during attaching, we check the consistency of all the targets with the target that we loaded, which is the first target. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 11 + include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 10 + kernel/bpf/btf.c | 5 + kernel/bpf/syscall.c | 379 +++++++++++++++++++++++++++++++++ kernel/bpf/trampoline.c | 7 +- kernel/bpf/verifier.c | 16 +- net/core/bpf_sk_storage.c | 2 + tools/include/uapi/linux/bpf.h | 10 + 9 files changed, 438 insertions(+), 3 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 4e8f17d9f022..28fac2d0964a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1635,6 +1635,17 @@ struct bpf_tracing_link { struct bpf_prog *tgt_prog; }; +struct bpf_tracing_multi_link { + struct bpf_tramp_multi_link link; + enum bpf_attach_type attach_type; + u32 prog_cnt; + u32 btf_cnt; + struct bpf_prog **tgt_progs; + struct btf **tgt_btfs; + u32 mods_cnt; + struct module **mods; +}; + struct bpf_link_primer { struct bpf_link *link; struct file *file; diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h index 94baced5a1ad..a93195bd825a 100644 --- a/include/linux/bpf_types.h +++ b/include/linux/bpf_types.h @@ -152,3 +152,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf) BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi) BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops) BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi) +BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING_MULTI, tracing_multi) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 85ec7fc799d7..f01c4f463c0d 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1114,6 +1114,9 @@ enum bpf_attach_type { BPF_CGROUP_UNIX_GETSOCKNAME, BPF_NETKIT_PRIMARY, BPF_NETKIT_PEER, + BPF_TRACE_FENTRY_MULTI, + BPF_TRACE_FEXIT_MULTI, + BPF_MODIFY_RETURN_MULTI, __MAX_BPF_ATTACH_TYPE }; @@ -1134,6 +1137,7 @@ enum bpf_link_type { BPF_LINK_TYPE_TCX = 11, BPF_LINK_TYPE_UPROBE_MULTI = 12, BPF_LINK_TYPE_NETKIT = 13, + BPF_LINK_TYPE_TRACING_MULTI = 14, __MAX_BPF_LINK_TYPE, }; @@ -1726,6 +1730,12 @@ union bpf_attr { */ __u64 cookie; } tracing; + struct { + __u32 cnt; + __aligned_u64 tgt_fds; + __aligned_u64 btf_ids; + __aligned_u64 cookies; + } tracing_multi; struct { __u32 pf; __u32 hooknum; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index c2a0299d4358..2d6e9e680091 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -5879,6 +5879,9 @@ static int btf_validate_prog_ctx_type(struct bpf_verifier_log *log, const struct case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: case BPF_MODIFY_RETURN: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: /* allow u64* as ctx */ if (btf_is_int(t) && t->size == 8) return 0; @@ -6238,6 +6241,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, case BPF_LSM_CGROUP: case BPF_LSM_MAC: case BPF_TRACE_FEXIT: + case BPF_TRACE_FEXIT_MULTI: /* When LSM programs are attached to void LSM hooks * they use FEXIT trampolines and when attached to * int LSM hooks, they use MODIFY_RETURN trampolines. @@ -6256,6 +6260,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, t = btf_type_by_id(btf, t->type); break; case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: /* For now the BPF_MODIFY_RETURN can only be attached to * functions that return an int. */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 6128c3131141..3e45584e4898 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3273,6 +3273,34 @@ static const struct bpf_link_ops bpf_tracing_link_lops = { .fill_link_info = bpf_tracing_link_fill_link_info, }; +static int bpf_tracing_check_multi(struct bpf_prog *prog, + struct bpf_prog *tgt_prog, + struct btf *btf2, + const struct btf_type *t2) +{ + const struct btf_type *t1; + struct btf *btf1; + + /* this case is already valided in bpf_check_attach_target() */ + if (prog->type == BPF_PROG_TYPE_EXT) + return 0; + + btf1 = prog->aux->dst_prog ? prog->aux->dst_prog->aux->btf : + prog->aux->attach_btf; + if (!btf1) + return -EOPNOTSUPP; + + btf2 = btf2 ?: tgt_prog->aux->btf; + t1 = prog->aux->attach_func_proto; + + /* the target is the same as the origin one, this is a re-attach */ + if (t1 == t2) + return 0; + + return btf_check_func_part_match(btf1, t1, btf2, t2, + prog->aux->accessed_args); +} + static int bpf_tracing_prog_attach(struct bpf_prog *prog, int tgt_prog_fd, u32 btf_id, @@ -3473,6 +3501,350 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, return err; } +static void __bpf_tracing_multi_link_release(struct bpf_tracing_multi_link *link) +{ + int i; + + if (link->mods_cnt) { + for (i = 0; i < link->mods_cnt; i++) + module_put(link->mods[i]); + kfree(link->mods); + } + + if (link->prog_cnt) { + for (i = 0; i < link->prog_cnt; i++) + bpf_prog_put(link->tgt_progs[i]); + kfree(link->tgt_progs); + } + + if (link->btf_cnt) { + for (i = 0; i < link->btf_cnt; i++) + btf_put(link->tgt_btfs[i]); + kfree(link->tgt_btfs); + } + + if (link->link.cnt) { + for (i = 0; i < link->link.cnt; i++) + bpf_trampoline_put(link->link.entries[i].trampoline); + kfree(link->link.entries); + } +} + +static void bpf_tracing_multi_link_release(struct bpf_link *link) +{ + struct bpf_tracing_multi_link *multi_link = + container_of(link, struct bpf_tracing_multi_link, link.link); + + bpf_trampoline_multi_unlink_prog(&multi_link->link); + __bpf_tracing_multi_link_release(multi_link); +} + +static void bpf_tracing_multi_link_dealloc(struct bpf_link *link) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link.link); + + kfree(tr_link); +} + +static void bpf_tracing_multi_link_show_fdinfo(const struct bpf_link *link, + struct seq_file *seq) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link.link); + u32 target_btf_id, target_obj_id; + int i; + + for (i = 0; i < tr_link->link.cnt; i++) { + bpf_trampoline_unpack_key(tr_link->link.entries[i].trampoline->key, + &target_obj_id, &target_btf_id); + seq_printf(seq, + "attach_type:\t%d\n" + "target_obj_id:\t%u\n" + "target_btf_id:\t%u\n", + tr_link->attach_type, + target_obj_id, + target_btf_id); + } +} + +static const struct bpf_link_ops bpf_tracing_multi_link_lops = { + .release = bpf_tracing_multi_link_release, + .dealloc = bpf_tracing_multi_link_dealloc, + .show_fdinfo = bpf_tracing_multi_link_show_fdinfo, +}; + +#define MAX_TRACING_MULTI_CNT 1024 + +static int bpf_tracing_get_target(u32 fd, struct bpf_prog **tgt_prog, + struct btf **tgt_btf) +{ + struct bpf_prog *prog = NULL; + struct btf *btf = NULL; + int err = 0; + + if (fd) { + prog = bpf_prog_get(fd); + if (!IS_ERR(prog)) + goto found; + + prog = NULL; + /* "fd" is the fd of the kernel module BTF */ + btf = btf_get_by_fd(fd); + if (IS_ERR(btf)) { + err = PTR_ERR(btf); + goto err; + } + if (!btf_is_kernel(btf)) { + btf_put(btf); + err = -EOPNOTSUPP; + goto err; + } + } else { + btf = bpf_get_btf_vmlinux(); + if (IS_ERR(btf)) { + err = PTR_ERR(btf); + goto err; + } + if (!btf) { + err = -EINVAL; + goto err; + } + btf_get(btf); + } +found: + *tgt_prog = prog; + *tgt_btf = btf; + return 0; +err: + *tgt_prog = NULL; + *tgt_btf = NULL; + return err; +} + +static int bpf_tracing_multi_link_check(const union bpf_attr *attr, u32 **btf_ids, + u32 **tgt_fds, u64 **cookies, + u32 cnt) +{ + void __user *ubtf_ids; + void __user *utgt_fds; + void __user *ucookies; + void *tmp; + int i; + + if (!cnt) + return -EINVAL; + + if (cnt > MAX_TRACING_MULTI_CNT) + return -E2BIG; + + ucookies = u64_to_user_ptr(attr->link_create.tracing_multi.cookies); + if (ucookies) { + tmp = kvmalloc_array(cnt, sizeof(**cookies), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + *cookies = tmp; + if (copy_from_user(tmp, ucookies, cnt * sizeof(**cookies))) + return -EFAULT; + } + + utgt_fds = u64_to_user_ptr(attr->link_create.tracing_multi.tgt_fds); + if (utgt_fds) { + tmp = kvmalloc_array(cnt, sizeof(**tgt_fds), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + *tgt_fds = tmp; + if (copy_from_user(tmp, utgt_fds, cnt * sizeof(**tgt_fds))) + return -EFAULT; + } + + ubtf_ids = u64_to_user_ptr(attr->link_create.tracing_multi.btf_ids); + if (!ubtf_ids) + return -EINVAL; + + tmp = kvmalloc_array(cnt, sizeof(**btf_ids), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + *btf_ids = tmp; + if (copy_from_user(tmp, ubtf_ids, cnt * sizeof(**btf_ids))) + return -EFAULT; + + for (i = 0; i < cnt; i++) { + if (!(*btf_ids)[i]) + return -EINVAL; + } + + return 0; +} + +static void bpf_tracing_multi_link_ptr_fill(struct bpf_tracing_multi_link *link, + struct ptr_array *progs, + struct ptr_array *mods, + struct ptr_array *btfs) +{ + link->mods = (struct module **) mods->ptrs; + link->mods_cnt = mods->cnt; + link->tgt_btfs = (struct btf **) btfs->ptrs; + link->btf_cnt = btfs->cnt; + link->tgt_progs = (struct bpf_prog **) progs->ptrs; + link->prog_cnt = progs->cnt; +} + +static int bpf_tracing_prog_attach_multi(const union bpf_attr *attr, + struct bpf_prog *prog) +{ + struct bpf_tracing_multi_link *link = NULL; + u32 cnt, *btf_ids = NULL, *tgt_fds = NULL; + struct bpf_link_primer link_primer; + struct ptr_array prog_array = { }; + struct ptr_array btf_array = { }; + struct ptr_array mod_array = { }; + u64 *cookies = NULL; + int err = 0, i; + + if ((prog->expected_attach_type != BPF_TRACE_FENTRY_MULTI && + prog->expected_attach_type != BPF_TRACE_FEXIT_MULTI && + prog->expected_attach_type != BPF_MODIFY_RETURN_MULTI) || + prog->type != BPF_PROG_TYPE_TRACING) + return -EINVAL; + + cnt = attr->link_create.tracing_multi.cnt; + err = bpf_tracing_multi_link_check(attr, &btf_ids, &tgt_fds, &cookies, + cnt); + if (err) + goto err_out; + + link = kzalloc(sizeof(*link), GFP_USER); + if (!link) { + err = -ENOMEM; + goto err_out; + } + link->link.entries = kzalloc(sizeof(*link->link.entries) * cnt, + GFP_USER); + if (!link->link.entries) { + err = -ENOMEM; + goto err_out; + } + + bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING_MULTI, + &bpf_tracing_multi_link_lops, prog); + link->attach_type = prog->expected_attach_type; + + mutex_lock(&prog->aux->dst_mutex); + + /* program is already attached, re-attach is not supported here yet */ + if (!prog->aux->dst_trampoline) { + err = -EEXIST; + goto err_out_unlock; + } + + for (i = 0; i < cnt; i++) { + struct bpf_attach_target_info tgt_info = {}; + struct bpf_tramp_multi_link_entry *entry; + struct bpf_prog *tgt_prog = NULL; + struct bpf_trampoline *tr = NULL; + u32 tgt_fd, btf_id = btf_ids[i]; + struct btf *tgt_btf = NULL; + struct module *mod = NULL; + u64 key = 0; + + entry = &link->link.entries[i]; + tgt_fd = tgt_fds ? tgt_fds[i] : 0; + err = bpf_tracing_get_target(tgt_fd, &tgt_prog, &tgt_btf); + if (err) + goto err_out_unlock; + + if (tgt_prog) { + err = bpf_try_add_ptr(&prog_array, tgt_prog); + if (err) { + bpf_prog_put(tgt_prog); + if (err != -EEXIST) + goto err_out_unlock; + } + } + + if (tgt_btf) { + err = bpf_try_add_ptr(&btf_array, tgt_btf); + if (err) { + btf_put(tgt_btf); + if (err != -EEXIST) + goto err_out_unlock; + } + } + + prog->aux->attach_tracing_prog = tgt_prog && + tgt_prog->type == BPF_PROG_TYPE_TRACING && + prog->type == BPF_PROG_TYPE_TRACING; + + err = bpf_check_attach_target(NULL, prog, tgt_prog, tgt_btf, + btf_id, &tgt_info); + if (err) + goto err_out_unlock; + + mod = tgt_info.tgt_mod; + if (mod) { + err = bpf_try_add_ptr(&mod_array, mod); + if (err) { + module_put(mod); + if (err != -EEXIST) + goto err_out_unlock; + } + } + + err = bpf_tracing_check_multi(prog, tgt_prog, tgt_btf, + tgt_info.tgt_type); + if (err) + goto err_out_unlock; + + key = bpf_trampoline_compute_key(tgt_prog, tgt_btf, btf_id); + tr = bpf_trampoline_get(key, &tgt_info); + if (!tr) { + err = -ENOMEM; + goto err_out_unlock; + } + + entry->conn.cookie = cookies ? cookies[i] : 0; + entry->conn.link = &link->link.link; + entry->trampoline = tr; + link->link.cnt++; + } + + err = bpf_trampoline_multi_link_prog(&link->link); + if (err) + goto err_out_unlock; + + err = bpf_link_prime(&link->link.link, &link_primer); + if (err) { + bpf_trampoline_multi_unlink_prog(&link->link); + goto err_out_unlock; + } + + bpf_tracing_multi_link_ptr_fill(link, &prog_array, &mod_array, + &btf_array); + bpf_trampoline_put(prog->aux->dst_trampoline); + prog->aux->dst_trampoline = NULL; + mutex_unlock(&prog->aux->dst_mutex); + + kfree(btf_ids); + kfree(tgt_fds); + kfree(cookies); + return bpf_link_settle(&link_primer); +err_out_unlock: + bpf_tracing_multi_link_ptr_fill(link, &prog_array, &mod_array, + &btf_array); + __bpf_tracing_multi_link_release(link); + mutex_unlock(&prog->aux->dst_mutex); +err_out: + kfree(btf_ids); + kfree(tgt_fds); + kfree(cookies); + kfree(link); + return err; +} + struct bpf_raw_tp_link { struct bpf_link link; struct bpf_raw_event_map *btp; @@ -3924,6 +4296,9 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: case BPF_MODIFY_RETURN: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: return BPF_PROG_TYPE_TRACING; case BPF_LSM_MAC: return BPF_PROG_TYPE_LSM; @@ -5201,6 +5576,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) ret = bpf_iter_link_attach(attr, uattr, prog); else if (prog->expected_attach_type == BPF_LSM_CGROUP) ret = cgroup_bpf_link_attach(attr, prog); + else if (prog->expected_attach_type == BPF_TRACE_FENTRY_MULTI || + prog->expected_attach_type == BPF_TRACE_FEXIT_MULTI || + prog->expected_attach_type == BPF_MODIFY_RETURN_MULTI) + ret = bpf_tracing_prog_attach_multi(attr, prog); else ret = bpf_tracing_prog_attach(prog, attr->link_create.target_fd, diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index b00d53af8fcb..6d249303a96e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -111,7 +111,9 @@ bool bpf_prog_has_trampoline(const struct bpf_prog *prog) return (ptype == BPF_PROG_TYPE_TRACING && (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT || - eatype == BPF_MODIFY_RETURN)) || + eatype == BPF_MODIFY_RETURN || + eatype == BPF_TRACE_FENTRY_MULTI || eatype == BPF_TRACE_FEXIT_MULTI || + eatype == BPF_MODIFY_RETURN_MULTI)) || (ptype == BPF_PROG_TYPE_LSM && eatype == BPF_LSM_MAC); } @@ -503,10 +505,13 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) { switch (prog->expected_attach_type) { case BPF_TRACE_FENTRY: + case BPF_TRACE_FENTRY_MULTI: return BPF_TRAMP_FENTRY; case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: return BPF_TRAMP_MODIFY_RETURN; case BPF_TRACE_FEXIT: + case BPF_TRACE_FEXIT_MULTI: return BPF_TRAMP_FEXIT; case BPF_LSM_MAC: if (!prog->aux->attach_func_proto->type) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4493ecc23597..f878edfcf987 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15405,10 +15405,13 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char switch (env->prog->expected_attach_type) { case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: range = retval_range(0, 0); break; case BPF_TRACE_RAW_TP: case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: return 0; case BPF_TRACE_ITER: break; @@ -20709,7 +20712,9 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, if (tgt_prog->type == BPF_PROG_TYPE_TRACING && prog_extension && (tgt_prog->expected_attach_type == BPF_TRACE_FENTRY || - tgt_prog->expected_attach_type == BPF_TRACE_FEXIT)) { + tgt_prog->expected_attach_type == BPF_TRACE_FEXIT || + tgt_prog->expected_attach_type == BPF_TRACE_FENTRY_MULTI || + tgt_prog->expected_attach_type == BPF_TRACE_FEXIT_MULTI)) { /* Program extensions can extend all program types * except fentry/fexit. The reason is the following. * The fentry/fexit programs are used for performance @@ -20784,6 +20789,9 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, case BPF_LSM_CGROUP: case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_MODIFY_RETURN_MULTI: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: if (!btf_type_is_func(t)) { bpf_log(log, "attach_btf_id %u is not a function\n", btf_id); @@ -20869,7 +20877,8 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, bpf_log(log, "%s is not sleepable\n", tname); return ret; } - } else if (prog->expected_attach_type == BPF_MODIFY_RETURN) { + } else if (prog->expected_attach_type == BPF_MODIFY_RETURN || + prog->expected_attach_type == BPF_MODIFY_RETURN_MULTI) { if (tgt_prog) { module_put(mod); bpf_log(log, "can't modify return codes of BPF programs\n"); @@ -20922,6 +20931,9 @@ static bool can_be_sleepable(struct bpf_prog *prog) case BPF_TRACE_FEXIT: case BPF_MODIFY_RETURN: case BPF_TRACE_ITER: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: return true; default: return false; diff --git a/net/core/bpf_sk_storage.c b/net/core/bpf_sk_storage.c index 6c4d90b24d46..712ae31593e5 100644 --- a/net/core/bpf_sk_storage.c +++ b/net/core/bpf_sk_storage.c @@ -371,6 +371,8 @@ static bool bpf_sk_storage_tracing_allowed(const struct bpf_prog *prog) return true; case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: btf_vmlinux = bpf_get_btf_vmlinux(); if (IS_ERR_OR_NULL(btf_vmlinux)) return false; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 85ec7fc799d7..f01c4f463c0d 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1114,6 +1114,9 @@ enum bpf_attach_type { BPF_CGROUP_UNIX_GETSOCKNAME, BPF_NETKIT_PRIMARY, BPF_NETKIT_PEER, + BPF_TRACE_FENTRY_MULTI, + BPF_TRACE_FEXIT_MULTI, + BPF_MODIFY_RETURN_MULTI, __MAX_BPF_ATTACH_TYPE }; @@ -1134,6 +1137,7 @@ enum bpf_link_type { BPF_LINK_TYPE_TCX = 11, BPF_LINK_TYPE_UPROBE_MULTI = 12, BPF_LINK_TYPE_NETKIT = 13, + BPF_LINK_TYPE_TRACING_MULTI = 14, __MAX_BPF_LINK_TYPE, }; @@ -1726,6 +1730,12 @@ union bpf_attr { */ __u64 cookie; } tracing; + struct { + __u32 cnt; + __aligned_u64 tgt_fds; + __aligned_u64 btf_ids; + __aligned_u64 cookies; + } tracing_multi; struct { __u32 pf; __u32 hooknum; From patchwork Mon Mar 11 09:35:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?5qKm6b6Z6JGj?= X-Patchwork-Id: 779585 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01E203DBA1 for ; Mon, 11 Mar 2024 09:35:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710149733; cv=none; b=ESkZSzrHtZZQxeBw6zqwwCkmXn2Gf9yRz2d85wDFOxzGZVehNjDuvuMuAz1Z/lXncCvFEm1UmNdswf6K9Z3jIJYaoNk0Iaz0ohDEiFCkhaz4ZWbyTGb8QEdQjt685pZJ6dV+w2z1CDnu9C3/V/ih3oy3reaVCB0vpj1Wp3M3FEM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710149733; c=relaxed/simple; bh=ibzml+PzSFzKH42zzI5+IahEHBtbwOKK2hMI0h78KBM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pB0CtKt0NEnz1KtDxdeZAUHs5fqU5XYInGB4lsUKoli/McN5iw0YtRP7MvVe3Wq73aTWpDLRGHkMiiJzmeipqXc6eOXjIz84UXiIDOu3TsL380ASxj+UK8qdUg6wD5xvJBTkPVqGWiWYAN9KPyBo9HKWV5aRHPl1jUFyOEKTRGk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=IJw1BOVE; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="IJw1BOVE" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1d944e8f367so31718455ad.0 for ; Mon, 11 Mar 2024 02:35:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710149731; x=1710754531; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JFf08lEs3m84/dbvV4ZlLUOMjMFpJvm5xegIfbyJ9HU=; b=IJw1BOVEKlp2EdC5OdG9+ILZTcu8cTvJvjCj56vkSBEDC3OZrC7HF3TOVTr+vFIiq4 FSIvwTwJlo1In9aEsOEp+lPbCVGRKr78CMgSnTgggi46BE3Lhj5i5GtVpgGjfsADGSnO Kf4dR0pU6Bf8V8TXsn+cyi5IE3GMaJ8C2COzMosFZSuXJPDhkiGJXPznCFJRvIeTa1cL CngqTNIQRxBl/7yQZ3xFFqBH95DBKK0oaklwawrBmD5gyjh6i798cJofSmslddv0URsh fw1mGBkYfMIR4nBibl2QlMk0dAY8QvpgvIYpQObYYRM5fnI4GW2WbaG8IZufYBxTjIeT vf0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710149731; x=1710754531; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JFf08lEs3m84/dbvV4ZlLUOMjMFpJvm5xegIfbyJ9HU=; b=GhXGa+5cS12adqpYGD5LwJvd+8GsTl8396ZgAeMU/fkASFZJNAJI07qOAQZ8Wtnuww sDIOAE3y9Km+sstlzgD86KlTSO2pZ7Q5q2WwQXN020Blvu5yirKDRBvJfR5GkMepMBAb hbEExkOmeHPXJwL/nsUlP/8oCX/K7p8+O8tiOefTjTab9mjhM+8aYoE10RdTf04tDT6d iN80nCKGBilxqhn7wh0eZ4o3FBa1sHjTrMsNZy5hv6jyQziWJ20Dr9htH+k/iKlDKjz7 jj8qDV8yJuCBmYXTl/TOsvOcSdeW3LT3qgLk6ZuaOOPZt2w1c9qosdjFRBlbjTznq/+h BZqw== X-Forwarded-Encrypted: i=1; AJvYcCXxu4M0WqUAMTFdxXt0FMcZAuHYmSa9GN5F85czyC27vupmlWgX7YbVxiYBIr4RMxuzu/gm9atZB/zEs08+z07GNsmiJ8f77H+OCNuboG4+ X-Gm-Message-State: AOJu0YzqZoH3CkAo2K27BQpTVqea2eGKbZ51jBJcWwMRcFt5SYiq9z5X ErxWfWVLc4uV9Y4Cvy/9J9LA8X3HA4wHgpm3JjA3XFmQ12Mg0mhX/di+upDnBW0= X-Google-Smtp-Source: AGHT+IFQIVN/SFlbgP5cEEuZ7moW9KP5l9jB6qy51la06bqNfTxjZ7YG5WDDYrE2U6yDVhQ3L+9Yxw== X-Received: by 2002:a17:902:988c:b0:1dd:772e:d504 with SMTP id s12-20020a170902988c00b001dd772ed504mr4091891plp.66.1710149729346; Mon, 11 Mar 2024 02:35:29 -0700 (PDT) Received: from localhost.localdomain ([43.129.25.208]) by smtp.gmail.com with ESMTPSA id h9-20020a170902f7c900b001dcad9cbf8bsm4253365plw.239.2024.03.11.02.35.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 02:35:29 -0700 (PDT) From: Menglong Dong To: andrii@kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, dsahern@kernel.org, dave.hansen@linux.intel.com, x86@kernel.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, quentin@isovalent.com, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-stm32@st-md-mailman.stormreply.com, Menglong Dong Subject: [PATCH bpf-next v2 8/9] libbpf: add support for the multi-link of tracing Date: Mon, 11 Mar 2024 17:35:25 +0800 Message-Id: <20240311093526.1010158-9-dongmenglong.8@bytedance.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> References: <20240311093526.1010158-1-dongmenglong.8@bytedance.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add support for the attach types of: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI Signed-off-by: Menglong Dong --- tools/bpf/bpftool/common.c | 3 + tools/lib/bpf/bpf.c | 10 +++ tools/lib/bpf/bpf.h | 6 ++ tools/lib/bpf/libbpf.c | 168 ++++++++++++++++++++++++++++++++++++- tools/lib/bpf/libbpf.h | 14 ++++ tools/lib/bpf/libbpf.map | 1 + 6 files changed, 199 insertions(+), 3 deletions(-) diff --git a/tools/bpf/bpftool/common.c b/tools/bpf/bpftool/common.c index cc6e6aae2447..ffc85256671d 100644 --- a/tools/bpf/bpftool/common.c +++ b/tools/bpf/bpftool/common.c @@ -1089,6 +1089,9 @@ const char *bpf_attach_type_input_str(enum bpf_attach_type t) case BPF_TRACE_FENTRY: return "fentry"; case BPF_TRACE_FEXIT: return "fexit"; case BPF_MODIFY_RETURN: return "mod_ret"; + case BPF_TRACE_FENTRY_MULTI: return "fentry_multi"; + case BPF_TRACE_FEXIT_MULTI: return "fexit_multi"; + case BPF_MODIFY_RETURN_MULTI: return "mod_ret_multi"; case BPF_SK_REUSEPORT_SELECT: return "sk_skb_reuseport_select"; case BPF_SK_REUSEPORT_SELECT_OR_MIGRATE: return "sk_skb_reuseport_select_or_migrate"; default: return libbpf_bpf_attach_type_str(t); diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 97ec005c3c47..63d4734dbae4 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -793,6 +793,16 @@ int bpf_link_create(int prog_fd, int target_fd, if (!OPTS_ZEROED(opts, tracing)) return libbpf_err(-EINVAL); break; + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: + attr.link_create.tracing_multi.btf_ids = ptr_to_u64(OPTS_GET(opts, tracing_multi.btf_ids, 0)); + attr.link_create.tracing_multi.tgt_fds = ptr_to_u64(OPTS_GET(opts, tracing_multi.tgt_fds, 0)); + attr.link_create.tracing_multi.cookies = ptr_to_u64(OPTS_GET(opts, tracing_multi.cookies, 0)); + attr.link_create.tracing_multi.cnt = OPTS_GET(opts, tracing_multi.cnt, 0); + if (!OPTS_ZEROED(opts, tracing_multi)) + return libbpf_err(-EINVAL); + break; case BPF_NETFILTER: attr.link_create.netfilter.pf = OPTS_GET(opts, netfilter.pf, 0); attr.link_create.netfilter.hooknum = OPTS_GET(opts, netfilter.hooknum, 0); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index df0db2f0cdb7..e28c88d6cfa4 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -419,6 +419,12 @@ struct bpf_link_create_opts { struct { __u64 cookie; } tracing; + struct { + __u32 cnt; + const __u32 *btf_ids; + const __u32 *tgt_fds; + const __u64 *cookies; + } tracing_multi; struct { __u32 pf; __u32 hooknum; diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index fd5428494a7e..821214774941 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -132,6 +132,9 @@ static const char * const attach_type_name[] = { [BPF_TRACE_UPROBE_MULTI] = "trace_uprobe_multi", [BPF_NETKIT_PRIMARY] = "netkit_primary", [BPF_NETKIT_PEER] = "netkit_peer", + [BPF_TRACE_FENTRY_MULTI] = "trace_fentry_multi", + [BPF_TRACE_FEXIT_MULTI] = "trace_fexit_multi", + [BPF_MODIFY_RETURN_MULTI] = "modify_return_multi", }; static const char * const link_type_name[] = { @@ -381,6 +384,8 @@ enum sec_def_flags { SEC_XDP_FRAGS = 16, /* Setup proper attach type for usdt probes. */ SEC_USDT = 32, + /* attachment target is multi-link */ + SEC_ATTACH_BTF_MULTI = 64, }; struct bpf_sec_def { @@ -7160,9 +7165,9 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog, if ((def & SEC_USDT) && kernel_supports(prog->obj, FEAT_UPROBE_MULTI_LINK)) prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI; - if ((def & SEC_ATTACH_BTF) && !prog->attach_btf_id) { + if ((def & (SEC_ATTACH_BTF | SEC_ATTACH_BTF_MULTI)) && !prog->attach_btf_id) { int btf_obj_fd = 0, btf_type_id = 0, err; - const char *attach_name; + const char *attach_name, *name_end; attach_name = strchr(prog->sec_name, '/'); if (!attach_name) { @@ -7181,7 +7186,27 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog, } attach_name++; /* skip over / */ - err = libbpf_find_attach_btf_id(prog, attach_name, &btf_obj_fd, &btf_type_id); + name_end = strchr(attach_name, ','); + /* for multi-link tracing, use the first target symbol during + * loading. + */ + if ((def & SEC_ATTACH_BTF_MULTI) && name_end) { + int len = name_end - attach_name + 1; + char *first_tgt; + + first_tgt = malloc(len); + if (!first_tgt) + return -ENOMEM; + strncpy(first_tgt, attach_name, len); + first_tgt[len - 1] = '\0'; + err = libbpf_find_attach_btf_id(prog, first_tgt, &btf_obj_fd, + &btf_type_id); + free(first_tgt); + } else { + err = libbpf_find_attach_btf_id(prog, attach_name, &btf_obj_fd, + &btf_type_id); + } + if (err) return err; @@ -9149,6 +9174,7 @@ static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link); static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link); static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link); +static int attach_trace_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link); static const struct bpf_sec_def section_defs[] = { SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE), @@ -9192,6 +9218,13 @@ static const struct bpf_sec_def section_defs[] = { SEC_DEF("fentry.s+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), SEC_DEF("fmod_ret.s+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), + SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_trace), + SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_ATTACH_BTF_MULTI, attach_trace_multi), + SEC_DEF("fmod_ret.multi+", TRACING, BPF_MODIFY_RETURN_MULTI, SEC_ATTACH_BTF_MULTI, attach_trace_multi), + SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_ATTACH_BTF_MULTI, attach_trace_multi), + SEC_DEF("fentry.multi.s+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_ATTACH_BTF_MULTI | SEC_SLEEPABLE, attach_trace_multi), + SEC_DEF("fmod_ret.multi.s+", TRACING, BPF_MODIFY_RETURN_MULTI, SEC_ATTACH_BTF_MULTI | SEC_SLEEPABLE, attach_trace_multi), + SEC_DEF("fexit.multi.s+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_ATTACH_BTF_MULTI | SEC_SLEEPABLE, attach_trace_multi), SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace), SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm), SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm), @@ -12300,6 +12333,135 @@ static int attach_trace(const struct bpf_program *prog, long cookie, struct bpf_ return libbpf_get_error(*link); } +struct bpf_link *bpf_program__attach_trace_multi_opts(const struct bpf_program *prog, + const struct bpf_trace_multi_opts *opts) +{ + LIBBPF_OPTS(bpf_link_create_opts, link_opts); + __u32 *btf_ids = NULL, *tgt_fds = NULL; + struct bpf_link *link = NULL; + char errmsg[STRERR_BUFSIZE]; + int prog_fd, pfd, cnt, err; + + if (!OPTS_VALID(opts, bpf_trace_multi_opts)) + return libbpf_err_ptr(-EINVAL); + + prog_fd = bpf_program__fd(prog); + if (prog_fd < 0) { + pr_warn("prog '%s': can't attach before loaded\n", prog->name); + return libbpf_err_ptr(-EINVAL); + } + + cnt = OPTS_GET(opts, cnt, 0); + if (opts->syms) { + int btf_obj_fd, btf_type_id, i; + + if (opts->btf_ids || opts->tgt_fds) { + pr_warn("can set both opts->syms and opts->btf_ids\n"); + return libbpf_err_ptr(-EINVAL); + } + + btf_ids = malloc(sizeof(*btf_ids) * cnt); + tgt_fds = malloc(sizeof(*tgt_fds) * cnt); + if (!btf_ids || !tgt_fds) { + err = -ENOMEM; + goto err_free; + } + for (i = 0; i < cnt; i++) { + btf_obj_fd = btf_type_id = 0; + + err = find_kernel_btf_id(prog->obj, opts->syms[i], + prog->expected_attach_type, &btf_obj_fd, + &btf_type_id); + if (err) + goto err_free; + btf_ids[i] = btf_type_id; + tgt_fds[i] = btf_obj_fd; + } + link_opts.tracing_multi.btf_ids = btf_ids; + link_opts.tracing_multi.tgt_fds = tgt_fds; + } else { + link_opts.tracing_multi.btf_ids = OPTS_GET(opts, btf_ids, 0); + link_opts.tracing_multi.tgt_fds = OPTS_GET(opts, tgt_fds, 0); + } + + link = calloc(1, sizeof(*link)); + if (!link) { + err = -ENOMEM; + goto err_free; + } + link->detach = &bpf_link__detach_fd; + + link_opts.tracing_multi.cookies = OPTS_GET(opts, cookies, 0); + link_opts.tracing_multi.cnt = cnt; + + pfd = bpf_link_create(prog_fd, 0, bpf_program__expected_attach_type(prog), &link_opts); + if (pfd < 0) { + err = -errno; + pr_warn("prog '%s': failed to attach: %s\n", + prog->name, libbpf_strerror_r(pfd, errmsg, sizeof(errmsg))); + goto err_free; + } + link->fd = pfd; + + free(btf_ids); + free(tgt_fds); + return link; +err_free: + free(btf_ids); + free(tgt_fds); + free(link); + return libbpf_err_ptr(err); +} + +static int attach_trace_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link) +{ + LIBBPF_OPTS(bpf_trace_multi_opts, opts); + int i, err, len, cnt = 1; + char **syms, *buf, *name; + const char *spec; + + spec = strchr(prog->sec_name, '/'); + if (!spec || !*(++spec)) + return -EINVAL; + + len = strlen(spec); + buf = malloc(len + 1); + if (!buf) + return -ENOMEM; + + strcpy(buf, spec); + for (i = 0; i < len; i++) { + if (buf[i] == ',') + cnt++; + } + + syms = malloc(sizeof(*syms) * cnt); + if (!syms) { + err = -ENOMEM; + goto out_free; + } + + opts.syms = (const char **)syms; + opts.cnt = cnt; + name = buf; + err = -EINVAL; + while (name) { + if (*name == '\0') + goto out_free; + *(syms++) = name; + name = strchr(name, ','); + if (name) + *(name++) = '\0'; + } + + *link = bpf_program__attach_trace_multi_opts(prog, &opts); + err = libbpf_get_error(*link); +out_free: + free(buf); + free(opts.syms); + return err; +} + static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link) { *link = bpf_program__attach_lsm(prog); diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index c41a909ea4c1..9bca44d5adfa 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -790,6 +790,20 @@ bpf_program__attach_xdp(const struct bpf_program *prog, int ifindex); LIBBPF_API struct bpf_link * bpf_program__attach_freplace(const struct bpf_program *prog, int target_fd, const char *attach_func_name); +struct bpf_trace_multi_opts { + /* size of this struct, for forward/backward compatibility */ + size_t sz; + const char **syms; + __u32 *btf_ids; + __u32 *tgt_fds; + __u64 *cookies; + size_t cnt; +}; +#define bpf_trace_multi_opts__last_field cnt + +LIBBPF_API struct bpf_link * +bpf_program__attach_trace_multi_opts(const struct bpf_program *prog, + const struct bpf_trace_multi_opts *opts); struct bpf_netfilter_opts { /* size of this struct, for forward/backward compatibility */ diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 57642b78917f..94933898df44 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -414,4 +414,5 @@ LIBBPF_1.4.0 { btf__new_split; btf_ext__raw_data; bpf_object__free_btfs; + bpf_program__attach_trace_multi_opts; } LIBBPF_1.3.0;