From patchwork Fri Dec 23 14:24:12 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mart van Santen X-Patchwork-Id: 88953 Delivered-To: patch@linaro.org Received: by 10.140.20.101 with SMTP id 92csp3348205qgi; Fri, 23 Dec 2016 06:26:34 -0800 (PST) X-Received: by 10.107.3.168 with SMTP id e40mr13166562ioi.89.1482503194785; Fri, 23 Dec 2016 06:26:34 -0800 (PST) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org. [192.237.175.120]) by mx.google.com with ESMTPS id 9si21729490itv.103.2016.12.23.06.26.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Dec 2016 06:26:34 -0800 (PST) Received-SPF: neutral (google.com: 192.237.175.120 is neither permitted nor denied by best guess record for domain of xen-devel-bounces@lists.xen.org) client-ip=192.237.175.120; Authentication-Results: mx.google.com; spf=neutral (google.com: 192.237.175.120 is neither permitted nor denied by best guess record for domain of xen-devel-bounces@lists.xen.org) smtp.mailfrom=xen-devel-bounces@lists.xen.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cKQlj-00050Q-WD; Fri, 23 Dec 2016 14:24:23 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cKQli-00050K-EP for xen-devel@lists.xen.org; Fri, 23 Dec 2016 14:24:22 +0000 Received: from [85.158.139.211] by server-1.bemta-5.messagelabs.com id D0/35-12836-5933D585; Fri, 23 Dec 2016 14:24:21 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrOKsWRWlGSWpSXmKPExsVyeJ9MoG6vcWy EwcTFGhZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8bBjzeYCo6qVrz7OI2xgbFdsYuRi0NIoJdJ YtrC42xdjJwcIgLSEtc+X2YEsdkENCX2f7rHDmILC3hLbN+zmQXE5hXQlvi7cCYziM0ioCqxb dkkJhBbVCBc4sGPa4wQNYISJ2c+YQFZwCzQziixv2EXWJGEgKPEoRc9QMs4gGxjiftn8iDCgh JbzzxkhbCFJBZuWghll0l8XDSHEcI2krj3cjb7BEb+WUhWzEK2AiTBLJAi8az5HvssJD/MQvL bLCQ/zELyG0Svv8TXE9OZIWx1iT/zLkHZ2hLLFr6Gsp0lvn26gEXcR+LD/fNQN9hKrFv3Hmqm nsTb371QtqXE+h2f2BcwCqxi1ChOLSpLLdI1MtdLKspMzyjJTczM0TU0MNXLTS0uTkxPzUlMK tZLzs/dxAiM4HoGBsYdjFe3+B1ilORgUhLlVdscEyHEl5SfUpmRWJwRX1Sak1p8iFGGg0NJgr fYKDZCSLAoNT21Ii0zB5hKYNISHDxKIrwpIGne4oLE3OLMdIjUKUZFKXHe+SAJAZBERmkeXBs sfV1ilJUS5mVkYGAQ4ilILcrNLEGVf8UozsGoJMz72RBoCk9mXgnc9FdAi5mAFts2R4MsLklE SEk1MPoZnmtl1rSPD3jRsmzfFNFHKxVjYmRfKOywUlge//OW4vTAFfUqBwJffl/tfGvLEk3R9 4zzr7eZTdP78VisvWuy5HNzoTVvW4577r/1MKA/2eaGouXhXwEuU+zfvHAvkm8WzvjvONtK+r JneqnDQv5knjt+Swyy5movktgbqLrApeZ15ZvjLEosxRmJhlrMRcWJAPvSZgdaAwAA X-Env-Sender: mart@greenhost.nl X-Msg-Ref: server-11.tower-206.messagelabs.com!1482503053!64858273!1 X-Originating-IP: [195.190.28.81] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG, ML_RADAR_SPEW_LINKS_8,spamassassin: ,async_handler: YXN5bmNfZGVsYXk6IDAgKHRpbWVvdXQp\n X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 49399 invoked from network); 23 Dec 2016 14:24:13 -0000 Received: from smarthost1.greenhost.nl (HELO smarthost1.greenhost.nl) (195.190.28.81) by server-11.tower-206.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 23 Dec 2016 14:24:13 -0000 Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1cKQlZ-0000vR-1H for xen-devel@lists.xen.org; Fri, 23 Dec 2016 15:24:13 +0100 To: xen-devel@lists.xen.org From: Mart van Santen Message-ID: <585D338C.5020905@greenhost.nl> Date: Fri, 23 Dec 2016 15:24:12 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 X-Authenticated-As-Hash: b10b6bfa8788b4f1a1f309b89bda8ff743e7b8b0 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: --- X-Spam-Score: -3.3 X-Spam-Status: No, score=-3.3 required=5.0 tests=ALL_TRUSTED, BAYES_50, RP_MATCHES_RCVD autolearn=disabled version=3.4.0 X-Scan-Signature: 503f1a2b1db20d3cc8283cfb339c155f Subject: [Xen-devel] [BUG]: [PATCH added] Incorrect queue counters in netback interface X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" Hello Xen Devel, We encountered an issue with one of our instances pushing a lot of (network) data, resulting in incorrect bandwidth reporting. We do poll interface traffic every 5 minutes for every interface of our instances (from Dom0) and store those values for reporting/graphing. At some point we discovered reports of machines pushing a lot of data, were incorrect and were going down in chucks of about 4 GB, while you expect this is an always increasing number (or wraps around a 32 or 64 bit counter) After futher analysing the problem and the source code we discovered that the counters of the queues in the code are handled as 32-bit integers, while the total bytes tx/rx for the while interface are counted as 64-bit longs. Attached a patch to solve the problem. Regards, Mart van Santen Below test results: Test existed of 2 DomU's on the same Dom0. Stats were gatherer from Dom0. The 2 DomU's were pushing data to each other using a network performance tool (iperf) Non-patched kernel results: node309 statistics # pwd /sys/devices/vif-2-0/net/vif2.0/statistics node309 statistics # node309 statistics # while [ 1 == 1 ]; do cat rx_bytes; sleep 1; done 873025622 1235062978 1572261463 2146257487 2700756931 3270058797 3843728761 113171325 665259289 1190130825 1555559865 1950490941 2499614765 3062263901 3640042389 4202300253 477729635 1046926733 ^C ---- Patched kernel: node310 statistics # pwd /sys/devices/vif-2-0/net/vif2.0/statistics node310 statistics # while [ 1 == 1 ]; do cat rx_bytes; sleep 1; done 38069388494 38632298582 39187057066 39749628321 40316842297 40875579821 41440446165 42005641261 42539933757 42991421265 43543245209 44104655317 44666040545 45218321305 45780709593 46346749753 46897399881 47393645401 47687655485 48069251037 48463066305 48866467737 49269999633 49695523708 50119597344 50541042376 50985332831 51446717847 51856736763 52234568347 52684797743 53250974835 53828014155 54406444595 54972895475 55548898027 55894007361 56408304870 56865463506 57435079678 57998185298 58577528786 59149894510 59715804286 60292539214 60861187894 61431075562 ^C (etc) (increasing numbers, as expected) -- Mart van Santen Greenhost E: mart@greenhost.nl T: +31 20 4890444 W: https://greenhost.nl A PGP signature can be attached to this e-mail, you need PGP software to verify it. My public key is available in keyserver(s) see: http://tinyurl.com/openpgp-manual PGP Fingerprint: CA85 EB11 2B70 042D AF66 B29A 6437 01A1 10A3 D3A5 --- a/drivers/net/xen-netback/common.h 2016-12-22 15:41:07.785535748 +0000 +++ b/drivers/net/xen-netback/common.h 2016-12-23 13:08:18.123080064 +0000 @@ -119,10 +119,10 @@ * A subset of struct net_device_stats that contains only the * fields that are updated in netback.c for each queue. */ - unsigned int rx_bytes; - unsigned int rx_packets; - unsigned int tx_bytes; - unsigned int tx_packets; + unsigned long rx_bytes; + unsigned long rx_packets; + unsigned long tx_bytes; + unsigned long tx_packets; /* Additional stats used by xenvif */ unsigned long rx_gso_checksum_fixup;