From patchwork Wed Jan 22 10:38:08 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: saeed bishara X-Patchwork-Id: 23502 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-gg0-f199.google.com (mail-gg0-f199.google.com [209.85.161.199]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 5795C218DC for ; Wed, 22 Jan 2014 10:38:38 +0000 (UTC) Received: by mail-gg0-f199.google.com with SMTP id n5sf14481520ggj.10 for ; Wed, 22 Jan 2014 02:38:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:mime-version:in-reply-to:references :from:date:message-id:subject:to:cc:sender:precedence:list-id :x-original-sender:x-original-authentication-results:mailing-list :list-post:list-help:list-archive:list-unsubscribe:content-type; bh=IhObYBq9zeQDlABR/fGzt1gO+TVaohZ5hBL4ZsVg1b8=; b=fjWnhP5SSUMJcw4HIjaVs1KAmE5YBlif75f+q2by8cAufj3q7ATgEXT/ruYwGNzCFg EKlFee5oVnip/5JG2IAlLahSP98WsZfoEJT+ve5nHNQBvdgXldBJ+E23HS7UlBZGYtRo tMTAsZ1m4TL+il8wbSaccCCB9GPEZCPujpsgpEJO2tDLCtZwbhi4eqtpGYFOhb3ccB95 Xvq0FitFjSv/T3cEbtXeDdqp/wXYtvINcEjIfLMLJoy4DDe8XwZp4LKFCGzGZUwnfMzX aPKVYh+9YmrZUggzvrX5FXkGtg04VK2c4lazT0H8O5SJt4JJjhMTn1hht1tCH1XAb5QM rCwA== X-Gm-Message-State: ALoCoQnGCfNlo2uQTpR1Q5Yk0iWnfGCbSuyIaVpOOVRnsOkg0jgGIEY2wbkG0y3+Rduac54b91IH X-Received: by 10.236.82.110 with SMTP id n74mr185495yhe.57.1390387117287; Wed, 22 Jan 2014 02:38:37 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.108.73 with SMTP id i67ls27601qgf.3.gmail; Wed, 22 Jan 2014 02:38:37 -0800 (PST) X-Received: by 10.236.204.74 with SMTP id g50mr301291yho.127.1390387117162; Wed, 22 Jan 2014 02:38:37 -0800 (PST) Received: from mail-vb0-x22b.google.com (mail-vb0-x22b.google.com [2607:f8b0:400c:c02::22b]) by mx.google.com with ESMTPS id s6si10304406yho.14.2014.01.22.02.38.37 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 22 Jan 2014 02:38:37 -0800 (PST) Received-SPF: neutral (google.com: 2607:f8b0:400c:c02::22b is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=2607:f8b0:400c:c02::22b; Received: by mail-vb0-f43.google.com with SMTP id p5so107625vbn.16 for ; Wed, 22 Jan 2014 02:38:36 -0800 (PST) X-Received: by 10.220.92.135 with SMTP id r7mr359499vcm.11.1390387116756; Wed, 22 Jan 2014 02:38:36 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.174.196 with SMTP id u4csp175858vcz; Wed, 22 Jan 2014 02:38:35 -0800 (PST) X-Received: by 10.68.197.133 with SMTP id iu5mr818095pbc.132.1390387114932; Wed, 22 Jan 2014 02:38:34 -0800 (PST) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a6si9296976pao.186.2014.01.22.02.38.34; Wed, 22 Jan 2014 02:38:34 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755075AbaAVKic (ORCPT + 2 others); Wed, 22 Jan 2014 05:38:32 -0500 Received: from mail-lb0-f173.google.com ([209.85.217.173]:39847 "EHLO mail-lb0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751516AbaAVKia (ORCPT ); Wed, 22 Jan 2014 05:38:30 -0500 Received: by mail-lb0-f173.google.com with SMTP id y6so157351lbh.18 for ; Wed, 22 Jan 2014 02:38:28 -0800 (PST) X-Received: by 10.152.22.201 with SMTP id g9mr522235laf.27.1390387108329; Wed, 22 Jan 2014 02:38:28 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.154.7 with HTTP; Wed, 22 Jan 2014 02:38:08 -0800 (PST) In-Reply-To: References: <20140114004509.27138.50345.stgit@viggo.jf.intel.com> <20140114004622.27138.54103.stgit@viggo.jf.intel.com> From: saeed bishara Date: Wed, 22 Jan 2014 12:38:08 +0200 Message-ID: Subject: Re: [PATCH v3 1/4] net_dma: simple removal To: Dan Williams Cc: "dmaengine@vger.kernel.org" , Alexander Duyck , Dave Jiang , Vinod Koul , "netdev@vger.kernel.org" , David Whipple , lkml , "David S. Miller" Sender: netdev-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: netdev@vger.kernel.org X-Original-Sender: saeed.bishara@gmail.com X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 2607:f8b0:400c:c02::22b is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org; dkim=neutral (bad format) header.i=@gmail.com; dmarc=fail (p=NONE dis=NONE) header.from=gmail.com Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , On Tue, Jan 21, 2014 at 11:44 AM, Dan Williams wrote: > On Fri, Jan 17, 2014 at 12:16 PM, saeed bishara wrote: >> Dan, >> >> isn't this issue similar to direct io case? >> can you please look at the following article >> http://lwn.net/Articles/322795/ > > I guess it's similar, but the NET_DMA dma api violation is more > blatant. The same thread that requested DMA is also writing to those > same pages with the cpu. The fix is either guaranteeing that only the > dma engine ever touches the gup'd pages or synchronizing dma before > every cpu fallback. > >> regarding performance improvement using NET_DMA, I don't have concrete >> numbers, but it should be around 15-20%. my system is i/o coherent. > > That sounds too high... is that throughput or cpu utilization? It that's the throughput improvement, my test is iperf server (no special flags, 1500 mtu). the iperf and 10G eth interrupts bound to same cpu which is the bottleneck in my case. I ran the following configurations a. NET_DMA=n b. NET_DMA=y c. NET_DMA=y + your dma_debug patch below, d. same as 3. by with my simple fix path. results in Gbps: a. 5.41 b. 6.17 (+14%) c. 5.93 (+9%) d. 5.92 (+9%) Dan, my simple fix is just to call tcp_service_net_dma(sk, true) whenever the cpu is going to copy the data. proper fix ofcourse can be smarter. do you think this is sufficient? { @@ -1302,6 +1303,7 @@ static int tcp_peek_sndq(struct sock *sk, struct msghdr *msg, int len) int copied = 0, err = 0; /* XXX -- need to support SO_PEEK_OFF */ + tcp_service_net_dma(sk, true); /* Wait for queue to drain */ skb_queue_walk(&sk->sk_write_queue, skb) { err = skb_copy_datagram_iovec(skb, 0, msg->msg_iov, skb->len); @@ -1861,6 +1863,8 @@ do_prequeue: } else #endif { + tcp_service_net_dma(sk, true); /* Wait for queue to drain */ + err = skb_copy_datagram_iovec(skb, offset, msg->msg_iov, used); if (err) { > sounds high because NET_DMA also makes the data cache cold while the > cpu copy warms the data before handing it to the application. for iperf case the test doesn't touch the data. also, for some applications, specially storage, the data can also be moved using dma. so this actually can be big advantage. > > Can you measure relative numbers and share your testing details? You > will need to fix the data corruption and verify that the performance > advantage is still there before proposing NET_DMA be restored. see above. > > I have a new dma_debug capability in Andrew's tree that can you help > you identify holes in the implementation. > > http://ozlabs.org/~akpm/mmots/broken-out/dma-debug-introduce-debug_dma_assert_idle.patch > > -- > Dan > >> >> saeed >> >> On Wed, Jan 15, 2014 at 11:33 PM, Dan Williams wrote: >>> On Wed, Jan 15, 2014 at 1:31 PM, Dan Williams wrote: >>>> On Wed, Jan 15, 2014 at 1:20 PM, saeed bishara wrote: >>>>> Hi Dan, >>>>> >>>>> I'm using net_dma on my system and I achieve meaningful performance >>>>> boost when running Iperf receive. >>>>> >>>>> As far as I know the net_dma is used by many embedded systems out >>>>> there and might effect their performance. >>>>> Can you please elaborate on the exact scenario that cause the memory corruption? >>>>> >>>>> Is the scenario mentioned here caused by "real life" application or >>>>> this is more of theoretical issue found through manual testing, I was >>>>> trying to find the thread describing the failing scenario and couldn't >>>>> find it, any pointer will be appreciated. >>>> >>>> Did you see the referenced commit? >>>> >>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=77873803363c >>>> >>>> This is a real issue in that any app that forks() while receiving data >>>> can cause the dma data to be lost. The problem is that the copy >>>> operation falls back to cpu at many locations. Any one of those >>>> instance could touch a mapped page and trigger a copy-on-write event. >>>> The dma completes to the wrong location. >>>> >>> >>> Btw, do you have benchmark data showing that NET_DMA is beneficial on >>> these platforms? I would have expected worse performance on platforms >>> without i/o coherent caches. --- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1295,6 +1295,7 @@ static int tcp_recv_urg(struct sock *sk, struct msghdr *msg, int len, int flags) */ return -EAGAIN; } +static void tcp_service_net_dma(struct sock *sk, bool wait); static int tcp_peek_sndq(struct sock *sk, struct msghdr *msg, int len)