From patchwork Wed Aug 18 23:52:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 499508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15B5DC4320A for ; Wed, 18 Aug 2021 23:52:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ED04E610D2 for ; Wed, 18 Aug 2021 23:52:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235232AbhHRXw4 (ORCPT ); Wed, 18 Aug 2021 19:52:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235064AbhHRXwy (ORCPT ); Wed, 18 Aug 2021 19:52:54 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9638FC0613CF for ; Wed, 18 Aug 2021 16:52:19 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id p71-20020a25424a0000b029056092741626so4622505yba.19 for ; Wed, 18 Aug 2021 16:52:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=u0j+ACVINI5xe3P0ek73QrWfPnNxV4g8NgRpnGbWpfs=; b=Wy4STULLtVajrdLOnlpq/PE5xQWKsl8nS+pXJEnCYp5N8MjL4tAaBBNhvUgfEv1g3i 2ZiExlTmi1eKUQ2lG9TQa8NzX3J8Lqg+2X0UKSTXoZHmxI4OoO7hcIy+xnkt7YH9IqSw AyIzhtXbIWBfa0yNNYy59nqi3DsSu0gtXHIogGgJrwFkPGU5iWgrHHy5oEZOclH7eMaP xb/+j26DYr21RVm1ldNhrgPZCYsyAdP9/UIZ/y5p2+SqN2YmbNRYhzu8CQU4hAJetiSH r5GHRbr17ClLTWr6fbaF4wzOpUbhYdV+2K4ah15yAkZzhMELlTsBMauyUGkbrZG5Xrxb sCSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=u0j+ACVINI5xe3P0ek73QrWfPnNxV4g8NgRpnGbWpfs=; b=q8Scd3jf5WkI3yLAOU927j1lLH8cul7MbzbBm2IgnuuEDXbKcsMpaCgnuBmkGwdS/v uVV850ZnhUEkqdq8E35tgA8XAxhzwj70fa4IK09NhjR2qIHWtbQEHQGe52EwWjTlOy27 VCDAmWNeDjN3WN3qhpMuWxd538ua5Rz8fG6B00618mPxK33lp1HrRPLk/RJcMnoCIbqc ZtDHewjtqwNILPjR/HNzBLPJuN/NZhyQYhDcM0FE9n6u6hO6wPsxIsd1vzUB4gwPaZiQ 2tKvMol8jnAegTfI9EntEKjsEuNCil8QWfaGdP3gdCGE1qz6czjIjAL0ZbnzLXEN2Qw4 4NIg== X-Gm-Message-State: AOAM5332IEqW+DD/39M7TmoZawBPjh8m8fxgrs4AKMnfLcgPoaWwz6pF fb7MXVrX7oRuE0A4opdbyO8KstClZbeo7eYdEBbPGk0XLKVXpohwsN9zp/24bFx2g4+xIPAc9pY UXJINDkKnyq4x5g0lNn8e20h4qYPnLy5sYAARSwn0/38EmvMnhBKIHQ== X-Google-Smtp-Source: ABdhPJwSq9Q050723AUzv+Aid3+xmwYFXnRVUE9mBI0RtJtjb4VO0e7+elxW9Cw2lN5a6JluTFlXH0Y= X-Received: from sdf2.svl.corp.google.com ([2620:15c:2c4:201:7401:ef23:e119:5cbc]) (user=sdf job=sendgmr) by 2002:a25:a163:: with SMTP id z90mr15484065ybh.378.1629330738819; Wed, 18 Aug 2021 16:52:18 -0700 (PDT) Date: Wed, 18 Aug 2021 16:52:15 -0700 Message-Id: <20210818235216.1159202-1-sdf@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.rc2.250.ged5fa647cd-goog Subject: [PATCH bpf-next v3 1/2] bpf: use kvmalloc for map values in syscall From: Stanislav Fomichev To: netdev@vger.kernel.org, bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, Stanislav Fomichev , Song Liu Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Use kvmalloc/kvfree for temporary value when manipulating a map via syscall. kmalloc might not be sufficient for percpu maps where the value is big (and further multiplied by hundreds of CPUs). Can be reproduced with netcnt test on qemu with "-smp 255". Signed-off-by: Stanislav Fomichev Acked-by: Song Liu --- kernel/bpf/syscall.c | 28 +++++++++++----------------- 1 file changed, 11 insertions(+), 17 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 7420e1334ab2..075f650d297a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1076,7 +1076,7 @@ static int map_lookup_elem(union bpf_attr *attr) value_size = bpf_map_value_size(map); err = -ENOMEM; - value = kmalloc(value_size, GFP_USER | __GFP_NOWARN); + value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN); if (!value) goto free_key; @@ -1091,7 +1091,7 @@ static int map_lookup_elem(union bpf_attr *attr) err = 0; free_value: - kfree(value); + kvfree(value); free_key: kfree(key); err_put: @@ -1137,16 +1137,10 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto err_put; } - if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || - map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || - map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY || - map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE) - value_size = round_up(map->value_size, 8) * num_possible_cpus(); - else - value_size = map->value_size; + value_size = bpf_map_value_size(map); err = -ENOMEM; - value = kmalloc(value_size, GFP_USER | __GFP_NOWARN); + value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN); if (!value) goto free_key; @@ -1157,7 +1151,7 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) err = bpf_map_update_value(map, f, key, value, attr->flags); free_value: - kfree(value); + kvfree(value); free_key: kfree(key); err_put: @@ -1367,7 +1361,7 @@ int generic_map_update_batch(struct bpf_map *map, if (!key) return -ENOMEM; - value = kmalloc(value_size, GFP_USER | __GFP_NOWARN); + value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN); if (!value) { kfree(key); return -ENOMEM; @@ -1390,7 +1384,7 @@ int generic_map_update_batch(struct bpf_map *map, if (copy_to_user(&uattr->batch.count, &cp, sizeof(cp))) err = -EFAULT; - kfree(value); + kvfree(value); kfree(key); return err; } @@ -1429,7 +1423,7 @@ int generic_map_lookup_batch(struct bpf_map *map, if (!buf_prevkey) return -ENOMEM; - buf = kmalloc(map->key_size + value_size, GFP_USER | __GFP_NOWARN); + buf = kvmalloc(map->key_size + value_size, GFP_USER | __GFP_NOWARN); if (!buf) { kfree(buf_prevkey); return -ENOMEM; @@ -1492,7 +1486,7 @@ int generic_map_lookup_batch(struct bpf_map *map, free_buf: kfree(buf_prevkey); - kfree(buf); + kvfree(buf); return err; } @@ -1547,7 +1541,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) value_size = bpf_map_value_size(map); err = -ENOMEM; - value = kmalloc(value_size, GFP_USER | __GFP_NOWARN); + value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN); if (!value) goto free_key; @@ -1579,7 +1573,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) err = 0; free_value: - kfree(value); + kvfree(value); free_key: kfree(key); err_put: From patchwork Wed Aug 18 23:52:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 500242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72CDAC432BE for ; Wed, 18 Aug 2021 23:52:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 53ED261051 for ; Wed, 18 Aug 2021 23:52:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235385AbhHRXw5 (ORCPT ); Wed, 18 Aug 2021 19:52:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235064AbhHRXw5 (ORCPT ); Wed, 18 Aug 2021 19:52:57 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B26F0C0613CF for ; Wed, 18 Aug 2021 16:52:21 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id t101-20020a25aaee0000b0290578c0c455b2so4654250ybi.13 for ; Wed, 18 Aug 2021 16:52:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=dNzVzG00YuL9BPubaFReOEmMNwvCEpVxzU7LeS/yc5w=; b=K4KV3UcU5YxbQ1Qg4BbCZXyoiODQYzv/uqz/faq2JQf1e/KVZI3U0R9m9Fxdx0cff5 CasFwoKrR4WBAQ0c1TZOVIxk2/4CfUyMDZnvs1nX55Sw8SSdweXNid0r5FBSSM25VhGo oOobBVGTZZUqlxkX17PTJ2yB+NafV4zRdCTDf7auL4Whh+/QXXyw3rrvTnLA1EMxeWvK ttSHlFLBsRNeL9vQ3JTMSk/UJX2XS6z+FZPdLl2XJXQj8pDuopwXpSzTWXPG0/ZJFLfK 2RVIHCw3Bs3wbYUuQaMeqy7XnK4ZWpMDSrnl/MJJcG3c/Z2TTh6xh1cTMdUm8EW0jTsC aPQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dNzVzG00YuL9BPubaFReOEmMNwvCEpVxzU7LeS/yc5w=; b=cXxbkxq9bNMt4Jd6qUzcY7m06KVGJKHtvYjl2nrgGLhdvTls8crqt2BksP1fNL5e7g /M2rMYboQajDftaqSrAlw1dGkhMvouSakfqyPQCFeweBrQ4Ey0KAyT0lVQWFJr5o1DH2 IlPxvDi1amHFZL2Xb5F4fbMCqBYhgYF4q2O2N/T9OEN5PGzyEfZIldbwdnnZaCUnVdpw jO961w5eRWK9X4afgJSVgoKLws5nvGZYki0u3By172XvptScWsZ4Zoi0l0InLAbQyCjj w4nnBzaEsZOnaO4C0yyeBjmPwe3YbqpTnS2er1P6xiG/f8VpwjVs62PLCjWG4hOG0RDi ///w== X-Gm-Message-State: AOAM531494Z2mAz0UXGBfVDxQiafpujhpX5kHs2WHmdCH4cEgkwb7Lsc 3+8lHYcGR2jtq8CSU9l4tuSUVmCJmBNSndzk+v8MpKBa6l2zCuLk69nNh6sWcLKA/nInj/1jgOT fwCeSYYtfeGP1op6Lzw3XU5m40jf0UcQ+hi/eMmbaND5waHF55vuDsA== X-Google-Smtp-Source: ABdhPJwq/fLcYMJf7f4rPMWxBqhc4/YvPJqHBBwjUW1EphvmK0ERzdTWW/WB1aCVUWqjUYZoZtij67Q= X-Received: from sdf2.svl.corp.google.com ([2620:15c:2c4:201:7401:ef23:e119:5cbc]) (user=sdf job=sendgmr) by 2002:a25:420b:: with SMTP id p11mr15255535yba.377.1629330740916; Wed, 18 Aug 2021 16:52:20 -0700 (PDT) Date: Wed, 18 Aug 2021 16:52:16 -0700 In-Reply-To: <20210818235216.1159202-1-sdf@google.com> Message-Id: <20210818235216.1159202-2-sdf@google.com> Mime-Version: 1.0 References: <20210818235216.1159202-1-sdf@google.com> X-Mailer: git-send-email 2.33.0.rc2.250.ged5fa647cd-goog Subject: [PATCH bpf-next v3 2/2] bpf: use kvmalloc for map keys in syscalls From: Stanislav Fomichev To: netdev@vger.kernel.org, bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, Stanislav Fomichev , Song Liu Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Same as previous patch but for the keys. memdup_bpfptr is renamed to kvmemdup_bpfptr (and converted to kvmalloc). v3: * kvmemdup_bpfptr and copy_from_bpfptr (Daniel Borkmann) Signed-off-by: Stanislav Fomichev Acked-by: Song Liu --- include/linux/bpfptr.h | 12 ++++++++++-- kernel/bpf/syscall.c | 34 +++++++++++++++++----------------- 2 files changed, 27 insertions(+), 19 deletions(-) diff --git a/include/linux/bpfptr.h b/include/linux/bpfptr.h index 5cdeab497cb3..546e27fc6d46 100644 --- a/include/linux/bpfptr.h +++ b/include/linux/bpfptr.h @@ -62,9 +62,17 @@ static inline int copy_to_bpfptr_offset(bpfptr_t dst, size_t offset, return copy_to_sockptr_offset((sockptr_t) dst, offset, src, size); } -static inline void *memdup_bpfptr(bpfptr_t src, size_t len) +static inline void *kvmemdup_bpfptr(bpfptr_t src, size_t len) { - return memdup_sockptr((sockptr_t) src, len); + void *p = kvmalloc(len, GFP_USER | __GFP_NOWARN); + + if (!p) + return ERR_PTR(-ENOMEM); + if (copy_from_bpfptr(p, src, len)) { + kvfree(p); + return ERR_PTR(-EFAULT); + } + return p; } static inline long strncpy_from_bpfptr(char *dst, bpfptr_t src, size_t count) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 075f650d297a..4e50c0bfdb7d 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1013,7 +1013,7 @@ int __weak bpf_stackmap_copy(struct bpf_map *map, void *key, void *value) static void *__bpf_copy_key(void __user *ukey, u64 key_size) { if (key_size) - return memdup_user(ukey, key_size); + return vmemdup_user(ukey, key_size); if (ukey) return ERR_PTR(-EINVAL); @@ -1024,7 +1024,7 @@ static void *__bpf_copy_key(void __user *ukey, u64 key_size) static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size) { if (key_size) - return memdup_bpfptr(ukey, key_size); + return kvmemdup_bpfptr(ukey, key_size); if (!bpfptr_is_null(ukey)) return ERR_PTR(-EINVAL); @@ -1093,7 +1093,7 @@ static int map_lookup_elem(union bpf_attr *attr) free_value: kvfree(value); free_key: - kfree(key); + kvfree(key); err_put: fdput(f); return err; @@ -1153,7 +1153,7 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) free_value: kvfree(value); free_key: - kfree(key); + kvfree(key); err_put: fdput(f); return err; @@ -1205,7 +1205,7 @@ static int map_delete_elem(union bpf_attr *attr) bpf_enable_instrumentation(); maybe_wait_bpf_programs(map); out: - kfree(key); + kvfree(key); err_put: fdput(f); return err; @@ -1247,7 +1247,7 @@ static int map_get_next_key(union bpf_attr *attr) } err = -ENOMEM; - next_key = kmalloc(map->key_size, GFP_USER); + next_key = kvmalloc(map->key_size, GFP_USER); if (!next_key) goto free_key; @@ -1270,9 +1270,9 @@ static int map_get_next_key(union bpf_attr *attr) err = 0; free_next_key: - kfree(next_key); + kvfree(next_key); free_key: - kfree(key); + kvfree(key); err_put: fdput(f); return err; @@ -1299,7 +1299,7 @@ int generic_map_delete_batch(struct bpf_map *map, if (!max_count) return 0; - key = kmalloc(map->key_size, GFP_USER | __GFP_NOWARN); + key = kvmalloc(map->key_size, GFP_USER | __GFP_NOWARN); if (!key) return -ENOMEM; @@ -1326,7 +1326,7 @@ int generic_map_delete_batch(struct bpf_map *map, if (copy_to_user(&uattr->batch.count, &cp, sizeof(cp))) err = -EFAULT; - kfree(key); + kvfree(key); return err; } @@ -1357,13 +1357,13 @@ int generic_map_update_batch(struct bpf_map *map, if (!max_count) return 0; - key = kmalloc(map->key_size, GFP_USER | __GFP_NOWARN); + key = kvmalloc(map->key_size, GFP_USER | __GFP_NOWARN); if (!key) return -ENOMEM; value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN); if (!value) { - kfree(key); + kvfree(key); return -ENOMEM; } @@ -1385,7 +1385,7 @@ int generic_map_update_batch(struct bpf_map *map, err = -EFAULT; kvfree(value); - kfree(key); + kvfree(key); return err; } @@ -1419,13 +1419,13 @@ int generic_map_lookup_batch(struct bpf_map *map, if (put_user(0, &uattr->batch.count)) return -EFAULT; - buf_prevkey = kmalloc(map->key_size, GFP_USER | __GFP_NOWARN); + buf_prevkey = kvmalloc(map->key_size, GFP_USER | __GFP_NOWARN); if (!buf_prevkey) return -ENOMEM; buf = kvmalloc(map->key_size + value_size, GFP_USER | __GFP_NOWARN); if (!buf) { - kfree(buf_prevkey); + kvfree(buf_prevkey); return -ENOMEM; } @@ -1485,7 +1485,7 @@ int generic_map_lookup_batch(struct bpf_map *map, err = -EFAULT; free_buf: - kfree(buf_prevkey); + kvfree(buf_prevkey); kvfree(buf); return err; } @@ -1575,7 +1575,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) free_value: kvfree(value); free_key: - kfree(key); + kvfree(key); err_put: fdput(f); return err;