From patchwork Wed Jul 21 10:15:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 483738 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 176B4C63798 for ; Wed, 21 Jul 2021 11:27:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 002476108B for ; Wed, 21 Jul 2021 11:27:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240195AbhGUKqO (ORCPT ); Wed, 21 Jul 2021 06:46:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238913AbhGUJgC (ORCPT ); Wed, 21 Jul 2021 05:36:02 -0400 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74419C061766 for ; Wed, 21 Jul 2021 03:15:33 -0700 (PDT) Received: by mail-pj1-x1030.google.com with SMTP id g22-20020a17090a5796b02901763aca3df6so729296pji.5 for ; Wed, 21 Jul 2021 03:15:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=KKo1EcfFUbZjUj2sF2Hy4VlPupKMjn+rZiu2OC+0fkY=; b=sStCXx1Znb9NnU8uCE2GF3/m4CfZB8QRiv0toGK3PYZpog6q2BswdJpRhZW+754LcI 777XOFYMR4Gg20xBoHl4FvSD/a/g2vZS/CkJgZYZyeBP7xwIoEuBip3JNkPsi5jA7dvb KbiVKnGvpgYEuC4KJygZ+Y3IGXKzvadBO4hrQYr+93ycFLdQs1UttpcU/zGwr/hUJV4S xfzt3sVhaQQI4dvOU4ZRp9pKuwJi/Umcc0opvuH//XGa8Qt4o9Km2Zt2dAu7pirVgmtG JMR2WOCReKlzJFWEWjJ7bq5kclmpSqNZtZLiWiUWG9n0u6m69d4+MG6SYXWV3uHJ2NTG xofg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=KKo1EcfFUbZjUj2sF2Hy4VlPupKMjn+rZiu2OC+0fkY=; b=s0mqR/829QFdajjk8NTy5NsebYSG6PAlNHaKKSEp25L3F8mOR54u+lqYe4xVB1HFwF 0jZWRExowVf/hHVZAYU8T6FMH+i6CgwXQ39hyi8uuyPgL7DNKUKp1o/UYjeC+EC1vcV2 35rafXhTKWCwNTIW72/uzA9I+e3Bz/LSlt0ctQwo2UxQdXuXVPpvcBKK/gTLYvg6vJfj rCR1UgyGRjZ8Eu8rd9aQAFTu/NUvj+kSIS4Z5reg5e6pJxC2l3thWMmQNqhI6dnDzj7N DR+21lKmY/C4f8V6Q2YPSkDiRTtmukr+8FozaNCdzK5RJLTpBbPC9ORsyMmdRSwdsrHQ 8G5g== X-Gm-Message-State: AOAM533QtDUqNk8Q9Rax0aY+UstAT4sKNRCMFcqOhTe+J4OvElXinkwd UDX9qKvO/yjBMp6hqnSMSeg= X-Google-Smtp-Source: ABdhPJyzenfi41CLW+RQT2R+CqTcu0bFYrzhsbEuaktWW+jO5e7NfuMufNQN6TCfGB6Tl0uECwaMzg== X-Received: by 2002:a17:902:a9c7:b029:12b:349:b318 with SMTP id b7-20020a170902a9c7b029012b0349b318mr27265309plr.13.1626862533047; Wed, 21 Jul 2021 03:15:33 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:7460:f7c2:e664:b708]) by smtp.gmail.com with ESMTPSA id a4sm22738319pfk.5.2021.07.21.03.15.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Jul 2021 03:15:32 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Yuchung Cheng Subject: [PATCH net-next] tcp: tweak len/truesize ratio for coalesce candidates Date: Wed, 21 Jul 2021 03:15:28 -0700 Message-Id: <20210721101528.329723-1-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Eric Dumazet tcp_grow_window() is using skb->len/skb->truesize to increase tp->rcv_ssthresh which has a direct impact on advertized window sizes. We added TCP coalescing in linux-3.4 & linux-3.5: Instead of storing skbs with one or two MSS in receive queue (or OFO queue), we try to append segments together to reduce memory overhead. High performance network drivers tend to cook skb with 3 parts : 1) sk_buff structure (256 bytes) 2) skb->head contains room to copy headers as needed, and skb_shared_info 3) page fragment(s) containing the ~1514 bytes frame (or more depending on MTU) Once coalesced into a previous skb, 1) and 2) are freed. We can therefore tweak the way we compute len/truesize ratio knowing that skb->truesize is inflated by 1) and 2) soon to be freed. This is done only for in-order skb, or skb coalesced into OFO queue. The result is that low rate flows no longer pay the memory price of having low GRO aggregation factor. Same result for drivers not using GRO. This is critical to allow a big enough receiver window, typically tcp_rmem[2] / 2. We have been using this at Google for about 5 years, it is due time to make it upstream. Signed-off-by: Eric Dumazet Cc: Soheil Hassas Yeganeh Cc: Neal Cardwell Cc: Yuchung Cheng Acked-by: Yuchung Cheng --- net/ipv4/tcp_input.c | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index bef2c8b64d83a0f3d4cca90f9b12912bf3d00807..501d8d4d4ba46f9a5de322ab690c320757e0990c 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -454,11 +454,12 @@ static void tcp_sndbuf_expand(struct sock *sk) */ /* Slow part of check#2. */ -static int __tcp_grow_window(const struct sock *sk, const struct sk_buff *skb) +static int __tcp_grow_window(const struct sock *sk, const struct sk_buff *skb, + unsigned int skbtruesize) { struct tcp_sock *tp = tcp_sk(sk); /* Optimize this! */ - int truesize = tcp_win_from_space(sk, skb->truesize) >> 1; + int truesize = tcp_win_from_space(sk, skbtruesize) >> 1; int window = tcp_win_from_space(sk, sock_net(sk)->ipv4.sysctl_tcp_rmem[2]) >> 1; while (tp->rcv_ssthresh <= window) { @@ -471,7 +472,27 @@ static int __tcp_grow_window(const struct sock *sk, const struct sk_buff *skb) return 0; } -static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb) +/* Even if skb appears to have a bad len/truesize ratio, TCP coalescing + * can play nice with us, as sk_buff and skb->head might be either + * freed or shared with up to MAX_SKB_FRAGS segments. + * Only give a boost to drivers using page frag(s) to hold the frame(s), + * and if no payload was pulled in skb->head before reaching us. + */ +static u32 truesize_adjust(bool adjust, const struct sk_buff *skb) +{ + u32 truesize = skb->truesize; + + if (adjust && !skb_headlen(skb)) { + truesize -= SKB_TRUESIZE(skb_end_offset(skb)); + /* paranoid check, some drivers might be buggy */ + if (unlikely((int)truesize < (int)skb->len)) + truesize = skb->truesize; + } + return truesize; +} + +static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb, + bool adjust) { struct tcp_sock *tp = tcp_sk(sk); int room; @@ -480,15 +501,16 @@ static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb) /* Check #1 */ if (room > 0 && !tcp_under_memory_pressure(sk)) { + unsigned int truesize = truesize_adjust(adjust, skb); int incr; /* Check #2. Increase window, if skb with such overhead * will fit to rcvbuf in future. */ - if (tcp_win_from_space(sk, skb->truesize) <= skb->len) + if (tcp_win_from_space(sk, truesize) <= skb->len) incr = 2 * tp->advmss; else - incr = __tcp_grow_window(sk, skb); + incr = __tcp_grow_window(sk, skb, truesize); if (incr) { incr = max_t(int, incr, 2 * skb->len); @@ -782,7 +804,7 @@ static void tcp_event_data_recv(struct sock *sk, struct sk_buff *skb) tcp_ecn_check_ce(sk, skb); if (skb->len >= 128) - tcp_grow_window(sk, skb); + tcp_grow_window(sk, skb, true); } /* Called to compute a smoothed rtt estimate. The data fed to this @@ -4769,7 +4791,7 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb) * and trigger fast retransmit. */ if (tcp_is_sack(tp)) - tcp_grow_window(sk, skb); + tcp_grow_window(sk, skb, true); kfree_skb_partial(skb, fragstolen); skb = NULL; goto add_sack; @@ -4857,7 +4879,7 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb) * and trigger fast retransmit. */ if (tcp_is_sack(tp)) - tcp_grow_window(sk, skb); + tcp_grow_window(sk, skb, false); skb_condense(skb); skb_set_owner_r(skb, sk); }