From patchwork Fri Jan 8 21:02:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 359517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.6 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D96E4C43381 for ; Fri, 8 Jan 2021 21:03:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A2B0523A7F for ; Fri, 8 Jan 2021 21:03:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729678AbhAHVDS (ORCPT ); Fri, 8 Jan 2021 16:03:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729522AbhAHVDS (ORCPT ); Fri, 8 Jan 2021 16:03:18 -0500 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38A30C061799 for ; Fri, 8 Jan 2021 13:02:30 -0800 (PST) Received: by mail-qk1-x749.google.com with SMTP id 189so10250759qko.1 for ; Fri, 08 Jan 2021 13:02:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=if/p6hl4f7LIK9IcixrLlQ36UeteZrMy3UWZSeRXeX8=; b=LVDyITHDL3kvCEEqrdYGU6RavUeBpcumoP1rxpeP+r6Xnh+CU/KJGqXzr5RLE+Qxxs VYoAxSAS7CFfBaSxNigq1610BBEMejS0lkRUjG+6ZxpQMQx2oG+IHDmVsa4bFAgx6m4w EXNcelLmTnOmyx5tPQjHhim8hU39oYcmMJhNcmkdagFViKlE7Qoh0QkWb9pjIC7moOqn HYBGLOFjxxwrKP/8LNotKb2TkF2IjWmcPXM8WzCQiIaJRg9YZ5m+kFCuC3FLlR71rE1X 0t4C+1Zv6JlG1XwzEkMqTanl9eXDg+zMbYTaeihKjimo2uoAOInOhtM+6K7/d2SA9LSJ cYtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=if/p6hl4f7LIK9IcixrLlQ36UeteZrMy3UWZSeRXeX8=; b=Iy4NUWZyebI1rM0ovyvg8gQrNo8dd0xgo/Kz20TmyL6Bn/GyOsys8yTZ80HrWZNJCF Wxs4bqlbuEOR+r9ksA9VKQzW0aOkuS+kCO00I6NCj1RKFknDV5Z4IUFlVJT4eid2JYQA 9I2x9EA+gKVRT3Tz4epSvsPOgOlVees23egUlZ6E80Xa3i5yED6PMpcI92CJ8j5Prt53 IOlnl46O57kft89madZSHp06mwNgrSXANK1AbuMPddShhanQ4on6EF+LAL8rP9fxBpa5 Sebw7z0U8qEBaL1r1ott2Fhc6C/aSI6eZO0Kkld+IgXsHs3PeYujlspagwzuB0EzPfAE DL0w== X-Gm-Message-State: AOAM5315uC1N7aI8ucjni4/HaMnHlKNM+eqeudY9mjPIAjBs/yVxlY35 nYg4Jw0KM6lK+0tz36a4zc8J5oqjXSH1Yghgu3Il4R6ByFcxqzBDkoueVSPYbJyP24vcNpv4FGc OzHY2gukNKueK7O8hrNxG2sjNNhPzJB9gXPqJ9O48eS6JV76H2xw5lQ== X-Google-Smtp-Source: ABdhPJxC5BXPXoUZI1fQHY/lOJyMkux9Qu8KVsXudD+vCXrqXD3A3t6L3jjPuTc8lri6lnwcFzxmjVM= Sender: "sdf via sendgmr" X-Received: from sdf2.svl.corp.google.com ([2620:15c:2c4:1:7220:84ff:fe09:7732]) (user=sdf job=sendgmr) by 2002:a0c:f888:: with SMTP id u8mr8663035qvn.47.1610139749309; Fri, 08 Jan 2021 13:02:29 -0800 (PST) Date: Fri, 8 Jan 2021 13:02:22 -0800 In-Reply-To: <20210108210223.972802-1-sdf@google.com> Message-Id: <20210108210223.972802-3-sdf@google.com> Mime-Version: 1.0 References: <20210108210223.972802-1-sdf@google.com> X-Mailer: git-send-email 2.30.0.284.gd98b1dd5eaa7-goog Subject: [PATCH bpf-next v6 2/3] bpf: try to avoid kzalloc in cgroup/{s, g}etsockopt From: Stanislav Fomichev To: netdev@vger.kernel.org, bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, Stanislav Fomichev , Martin KaFai Lau , Song Liu Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org When we attach a bpf program to cgroup/getsockopt any other getsockopt() syscall starts incurring kzalloc/kfree cost. Let add a small buffer on the stack and use it for small (majority) {s,g}etsockopt values. The buffer is small enough to fit into the cache line and cover the majority of simple options (most of them are 4 byte ints). It seems natural to do the same for setsockopt, but it's a bit more involved when the BPF program modifies the data (where we have to kmalloc). The assumption is that for the majority of setsockopt calls (which are doing pure BPF options or apply policy) this will bring some benefit as well. Without this patch (we remove about 1% __kmalloc): 3.38% 0.07% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt | --3.30%--__cgroup_bpf_run_filter_getsockopt | --0.81%--__kmalloc Signed-off-by: Stanislav Fomichev Cc: Martin KaFai Lau Cc: Song Liu --- include/linux/filter.h | 5 ++++ kernel/bpf/cgroup.c | 52 ++++++++++++++++++++++++++++++++++++------ 2 files changed, 50 insertions(+), 7 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 29c27656165b..8739f1d4cac4 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1281,6 +1281,11 @@ struct bpf_sysctl_kern { u64 tmp_reg; }; +#define BPF_SOCKOPT_KERN_BUF_SIZE 32 +struct bpf_sockopt_buf { + u8 data[BPF_SOCKOPT_KERN_BUF_SIZE]; +}; + struct bpf_sockopt_kern { struct sock *sk; u8 *optval; diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index c41bb2f34013..a9aad9c419e1 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -1298,7 +1298,8 @@ static bool __cgroup_bpf_prog_array_is_empty(struct cgroup *cgrp, return empty; } -static int sockopt_alloc_buf(struct bpf_sockopt_kern *ctx, int max_optlen) +static int sockopt_alloc_buf(struct bpf_sockopt_kern *ctx, int max_optlen, + struct bpf_sockopt_buf *buf) { if (unlikely(max_optlen < 0)) return -EINVAL; @@ -1310,6 +1311,15 @@ static int sockopt_alloc_buf(struct bpf_sockopt_kern *ctx, int max_optlen) max_optlen = PAGE_SIZE; } + if (max_optlen <= sizeof(buf->data)) { + /* When the optval fits into BPF_SOCKOPT_KERN_BUF_SIZE + * bytes avoid the cost of kzalloc. + */ + ctx->optval = buf->data; + ctx->optval_end = ctx->optval + max_optlen; + return max_optlen; + } + ctx->optval = kzalloc(max_optlen, GFP_USER); if (!ctx->optval) return -ENOMEM; @@ -1319,16 +1329,26 @@ static int sockopt_alloc_buf(struct bpf_sockopt_kern *ctx, int max_optlen) return max_optlen; } -static void sockopt_free_buf(struct bpf_sockopt_kern *ctx) +static void sockopt_free_buf(struct bpf_sockopt_kern *ctx, + struct bpf_sockopt_buf *buf) { + if (ctx->optval == buf->data) + return; kfree(ctx->optval); } +static bool sockopt_buf_allocated(struct bpf_sockopt_kern *ctx, + struct bpf_sockopt_buf *buf) +{ + return ctx->optval != buf->data; +} + int __cgroup_bpf_run_filter_setsockopt(struct sock *sk, int *level, int *optname, char __user *optval, int *optlen, char **kernel_optval) { struct cgroup *cgrp = sock_cgroup_ptr(&sk->sk_cgrp_data); + struct bpf_sockopt_buf buf = {}; struct bpf_sockopt_kern ctx = { .sk = sk, .level = *level, @@ -1350,7 +1370,7 @@ int __cgroup_bpf_run_filter_setsockopt(struct sock *sk, int *level, */ max_optlen = max_t(int, 16, *optlen); - max_optlen = sockopt_alloc_buf(&ctx, max_optlen); + max_optlen = sockopt_alloc_buf(&ctx, max_optlen, &buf); if (max_optlen < 0) return max_optlen; @@ -1390,13 +1410,30 @@ int __cgroup_bpf_run_filter_setsockopt(struct sock *sk, int *level, */ if (ctx.optlen != 0) { *optlen = ctx.optlen; - *kernel_optval = ctx.optval; + /* We've used bpf_sockopt_kern->buf as an intermediary + * storage, but the BPF program indicates that we need + * to pass this data to the kernel setsockopt handler. + * No way to export on-stack buf, have to allocate a + * new buffer. + */ + if (!sockopt_buf_allocated(&ctx, &buf)) { + void *p = kzalloc(ctx.optlen, GFP_USER); + + if (!p) { + ret = -ENOMEM; + goto out; + } + memcpy(p, ctx.optval, ctx.optlen); + *kernel_optval = p; + } else { + *kernel_optval = ctx.optval; + } } } out: if (ret) - sockopt_free_buf(&ctx); + sockopt_free_buf(&ctx, &buf); return ret; } @@ -1406,6 +1443,7 @@ int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, int retval) { struct cgroup *cgrp = sock_cgroup_ptr(&sk->sk_cgrp_data); + struct bpf_sockopt_buf buf = {}; struct bpf_sockopt_kern ctx = { .sk = sk, .level = level, @@ -1424,7 +1462,7 @@ int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, ctx.optlen = max_optlen; - max_optlen = sockopt_alloc_buf(&ctx, max_optlen); + max_optlen = sockopt_alloc_buf(&ctx, max_optlen, &buf); if (max_optlen < 0) return max_optlen; @@ -1482,7 +1520,7 @@ int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level, ret = ctx.retval; out: - sockopt_free_buf(&ctx); + sockopt_free_buf(&ctx, &buf); return ret; }