From patchwork Tue Jun 2 13:05:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Michael S. Tsirkin" X-Patchwork-Id: 218041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6863BC433DF for ; Tue, 2 Jun 2020 13:08:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 39E8620674 for ; Tue, 2 Jun 2020 13:08:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TL/g72e+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727798AbgFBNI1 (ORCPT ); Tue, 2 Jun 2020 09:08:27 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:54827 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727946AbgFBNGH (ORCPT ); Tue, 2 Jun 2020 09:06:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591103165; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rp7LLbgW9GZ/JWYVr15hCRITL9UNlGoYkjFP3Zgo2mI=; b=TL/g72e+jyP1RIpOoZMZ/0AEohIb958W9kCAmfjRvGt/U90KMvGxeJZUP4qC1KvqUdlbMB vL0ZLkwvhq7DxoW/AtncuN3Gjl3XTskaQ5J4xvETC133rj1WiK+32psjwUjq5PoMSXukRS S1/L0r7DyDQN/COl/+5e4Rs3ch7kayo= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-78-KLuXCVUEPFGXrzMU4VHwtg-1; Tue, 02 Jun 2020 09:06:02 -0400 X-MC-Unique: KLuXCVUEPFGXrzMU4VHwtg-1 Received: by mail-wm1-f70.google.com with SMTP id p24so936647wmc.1 for ; Tue, 02 Jun 2020 06:06:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=rp7LLbgW9GZ/JWYVr15hCRITL9UNlGoYkjFP3Zgo2mI=; b=sL09J0AwEqTxGZztdqnXEg3Gd48GGxfUaqkf9xQizo82U4ymCDIiAB9URH9TqbSqdg cte/DiAWCHTPx/SmiqNjkYuyXmzmGmq+idJvmc7r4juUo/DLw75VPONCZw/EFH+DgoJ7 bCAMv65lU1QglKQ9hSxbqDkLF1Kr2iOzB8SkyCSgy1m3Gb8MVAVpXjwVOZP3/TWXCf2Q 0MuB6uifh9Cqw2uT8FSofvtQztgNEh0IN2JA9BkCJDh6fzliHJ4SJcb3FzfVbYI3BW7l B5EYsOjlbJb/bCgZq2WqsAt5q63l4QqzpdQ25C/3m4AUeu30AxMrgUvtjbWTVa70wqbM meXQ== X-Gm-Message-State: AOAM532U3Xdwj+3XmSm5CW3zChrfMbzNBdPwKXzCUQtTEUnO4G85TDgv XpBKN6cD8nZPlIHNgTjWeRwHsRvg422xdXT1R3j1WP2RFE6Ll133ysAepZyHDQIH8vkqkM4jRhL VXGIEcF/3EGirPDDw X-Received: by 2002:a1c:e389:: with SMTP id a131mr4267828wmh.46.1591103161352; Tue, 02 Jun 2020 06:06:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxQAysGHW2SBec5bd1cJygH5e17kJ6EnFsTlUkz5zjzv3vKi01gkCOuXebuiVoD5lN+73WQeQ== X-Received: by 2002:a1c:e389:: with SMTP id a131mr4267807wmh.46.1591103160991; Tue, 02 Jun 2020 06:06:00 -0700 (PDT) Received: from redhat.com (bzq-109-64-41-91.red.bezeqint.net. [109.64.41.91]) by smtp.gmail.com with ESMTPSA id m129sm3760204wmf.2.2020.06.02.06.05.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 06:06:00 -0700 (PDT) Date: Tue, 2 Jun 2020 09:05:59 -0400 From: "Michael S. Tsirkin" To: linux-kernel@vger.kernel.org Cc: Eugenio =?utf-8?b?UMOpcmV6?= , Jason Wang , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Subject: [PATCH RFC 02/13] vhost: use batched version by default Message-ID: <20200602130543.578420-3-mst@redhat.com> References: <20200602130543.578420-1-mst@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200602130543.578420-1-mst@redhat.com> X-Mailer: git-send-email 2.24.1.751.gd10ce2899c X-Mutt-Fcc: =sent Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org As testing shows no performance change, switch to that now. Signed-off-by: Michael S. Tsirkin Signed-off-by: Eugenio Pérez Link: https://lore.kernel.org/r/20200401183118.8334-3-eperezma@redhat.com Signed-off-by: Michael S. Tsirkin --- drivers/vhost/vhost.c | 251 +----------------------------------------- drivers/vhost/vhost.h | 4 - 2 files changed, 2 insertions(+), 253 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 105fc97af2c8..8f9a07282625 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -2038,253 +2038,6 @@ static unsigned next_desc(struct vhost_virtqueue *vq, struct vring_desc *desc) return next; } -static int get_indirect(struct vhost_virtqueue *vq, - struct iovec iov[], unsigned int iov_size, - unsigned int *out_num, unsigned int *in_num, - struct vhost_log *log, unsigned int *log_num, - struct vring_desc *indirect) -{ - struct vring_desc desc; - unsigned int i = 0, count, found = 0; - u32 len = vhost32_to_cpu(vq, indirect->len); - struct iov_iter from; - int ret, access; - - /* Sanity check */ - if (unlikely(len % sizeof desc)) { - vq_err(vq, "Invalid length in indirect descriptor: " - "len 0x%llx not multiple of 0x%zx\n", - (unsigned long long)len, - sizeof desc); - return -EINVAL; - } - - ret = translate_desc(vq, vhost64_to_cpu(vq, indirect->addr), len, vq->indirect, - UIO_MAXIOV, VHOST_ACCESS_RO); - if (unlikely(ret < 0)) { - if (ret != -EAGAIN) - vq_err(vq, "Translation failure %d in indirect.\n", ret); - return ret; - } - iov_iter_init(&from, READ, vq->indirect, ret, len); - - /* We will use the result as an address to read from, so most - * architectures only need a compiler barrier here. */ - read_barrier_depends(); - - count = len / sizeof desc; - /* Buffers are chained via a 16 bit next field, so - * we can have at most 2^16 of these. */ - if (unlikely(count > USHRT_MAX + 1)) { - vq_err(vq, "Indirect buffer length too big: %d\n", - indirect->len); - return -E2BIG; - } - - do { - unsigned iov_count = *in_num + *out_num; - if (unlikely(++found > count)) { - vq_err(vq, "Loop detected: last one at %u " - "indirect size %u\n", - i, count); - return -EINVAL; - } - if (unlikely(!copy_from_iter_full(&desc, sizeof(desc), &from))) { - vq_err(vq, "Failed indirect descriptor: idx %d, %zx\n", - i, (size_t)vhost64_to_cpu(vq, indirect->addr) + i * sizeof desc); - return -EINVAL; - } - if (unlikely(desc.flags & cpu_to_vhost16(vq, VRING_DESC_F_INDIRECT))) { - vq_err(vq, "Nested indirect descriptor: idx %d, %zx\n", - i, (size_t)vhost64_to_cpu(vq, indirect->addr) + i * sizeof desc); - return -EINVAL; - } - - if (desc.flags & cpu_to_vhost16(vq, VRING_DESC_F_WRITE)) - access = VHOST_ACCESS_WO; - else - access = VHOST_ACCESS_RO; - - ret = translate_desc(vq, vhost64_to_cpu(vq, desc.addr), - vhost32_to_cpu(vq, desc.len), iov + iov_count, - iov_size - iov_count, access); - if (unlikely(ret < 0)) { - if (ret != -EAGAIN) - vq_err(vq, "Translation failure %d indirect idx %d\n", - ret, i); - return ret; - } - /* If this is an input descriptor, increment that count. */ - if (access == VHOST_ACCESS_WO) { - *in_num += ret; - if (unlikely(log && ret)) { - log[*log_num].addr = vhost64_to_cpu(vq, desc.addr); - log[*log_num].len = vhost32_to_cpu(vq, desc.len); - ++*log_num; - } - } else { - /* If it's an output descriptor, they're all supposed - * to come before any input descriptors. */ - if (unlikely(*in_num)) { - vq_err(vq, "Indirect descriptor " - "has out after in: idx %d\n", i); - return -EINVAL; - } - *out_num += ret; - } - } while ((i = next_desc(vq, &desc)) != -1); - return 0; -} - -/* This looks in the virtqueue and for the first available buffer, and converts - * it to an iovec for convenient access. Since descriptors consist of some - * number of output then some number of input descriptors, it's actually two - * iovecs, but we pack them into one and note how many of each there were. - * - * This function returns the descriptor number found, or vq->num (which is - * never a valid descriptor number) if none was found. A negative code is - * returned on error. */ -int vhost_get_vq_desc(struct vhost_virtqueue *vq, - struct iovec iov[], unsigned int iov_size, - unsigned int *out_num, unsigned int *in_num, - struct vhost_log *log, unsigned int *log_num) -{ - struct vring_desc desc; - unsigned int i, head, found = 0; - u16 last_avail_idx; - __virtio16 avail_idx; - __virtio16 ring_head; - int ret, access; - - /* Check it isn't doing very strange things with descriptor numbers. */ - last_avail_idx = vq->last_avail_idx; - - if (vq->avail_idx == vq->last_avail_idx) { - if (unlikely(vhost_get_avail_idx(vq, &avail_idx))) { - vq_err(vq, "Failed to access avail idx at %p\n", - &vq->avail->idx); - return -EFAULT; - } - vq->avail_idx = vhost16_to_cpu(vq, avail_idx); - - if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) { - vq_err(vq, "Guest moved used index from %u to %u", - last_avail_idx, vq->avail_idx); - return -EFAULT; - } - - /* If there's nothing new since last we looked, return - * invalid. - */ - if (vq->avail_idx == last_avail_idx) - return vq->num; - - /* Only get avail ring entries after they have been - * exposed by guest. - */ - smp_rmb(); - } - - /* Grab the next descriptor number they're advertising, and increment - * the index we've seen. */ - if (unlikely(vhost_get_avail_head(vq, &ring_head, last_avail_idx))) { - vq_err(vq, "Failed to read head: idx %d address %p\n", - last_avail_idx, - &vq->avail->ring[last_avail_idx % vq->num]); - return -EFAULT; - } - - head = vhost16_to_cpu(vq, ring_head); - - /* If their number is silly, that's an error. */ - if (unlikely(head >= vq->num)) { - vq_err(vq, "Guest says index %u > %u is available", - head, vq->num); - return -EINVAL; - } - - /* When we start there are none of either input nor output. */ - *out_num = *in_num = 0; - if (unlikely(log)) - *log_num = 0; - - i = head; - do { - unsigned iov_count = *in_num + *out_num; - if (unlikely(i >= vq->num)) { - vq_err(vq, "Desc index is %u > %u, head = %u", - i, vq->num, head); - return -EINVAL; - } - if (unlikely(++found > vq->num)) { - vq_err(vq, "Loop detected: last one at %u " - "vq size %u head %u\n", - i, vq->num, head); - return -EINVAL; - } - ret = vhost_get_desc(vq, &desc, i); - if (unlikely(ret)) { - vq_err(vq, "Failed to get descriptor: idx %d addr %p\n", - i, vq->desc + i); - return -EFAULT; - } - if (desc.flags & cpu_to_vhost16(vq, VRING_DESC_F_INDIRECT)) { - ret = get_indirect(vq, iov, iov_size, - out_num, in_num, - log, log_num, &desc); - if (unlikely(ret < 0)) { - if (ret != -EAGAIN) - vq_err(vq, "Failure detected " - "in indirect descriptor at idx %d\n", i); - return ret; - } - continue; - } - - if (desc.flags & cpu_to_vhost16(vq, VRING_DESC_F_WRITE)) - access = VHOST_ACCESS_WO; - else - access = VHOST_ACCESS_RO; - ret = translate_desc(vq, vhost64_to_cpu(vq, desc.addr), - vhost32_to_cpu(vq, desc.len), iov + iov_count, - iov_size - iov_count, access); - if (unlikely(ret < 0)) { - if (ret != -EAGAIN) - vq_err(vq, "Translation failure %d descriptor idx %d\n", - ret, i); - return ret; - } - if (access == VHOST_ACCESS_WO) { - /* If this is an input descriptor, - * increment that count. */ - *in_num += ret; - if (unlikely(log && ret)) { - log[*log_num].addr = vhost64_to_cpu(vq, desc.addr); - log[*log_num].len = vhost32_to_cpu(vq, desc.len); - ++*log_num; - } - } else { - /* If it's an output descriptor, they're all supposed - * to come before any input descriptors. */ - if (unlikely(*in_num)) { - vq_err(vq, "Descriptor has out after in: " - "idx %d\n", i); - return -EINVAL; - } - *out_num += ret; - } - } while ((i = next_desc(vq, &desc)) != -1); - - /* On success, increment avail index. */ - vq->last_avail_idx++; - - /* Assume notifications from guest are disabled at this point, - * if they aren't we would need to update avail_event index. */ - BUG_ON(!(vq->used_flags & VRING_USED_F_NO_NOTIFY)); - return head; -} -EXPORT_SYMBOL_GPL(vhost_get_vq_desc); - static struct vhost_desc *peek_split_desc(struct vhost_virtqueue *vq) { BUG_ON(!vq->ndescs); @@ -2495,7 +2248,7 @@ static int fetch_descs(struct vhost_virtqueue *vq) * This function returns the descriptor number found, or vq->num (which is * never a valid descriptor number) if none was found. A negative code is * returned on error. */ -int vhost_get_vq_desc_batch(struct vhost_virtqueue *vq, +int vhost_get_vq_desc(struct vhost_virtqueue *vq, struct iovec iov[], unsigned int iov_size, unsigned int *out_num, unsigned int *in_num, struct vhost_log *log, unsigned int *log_num) @@ -2570,7 +2323,7 @@ int vhost_get_vq_desc_batch(struct vhost_virtqueue *vq, return ret; } -EXPORT_SYMBOL_GPL(vhost_get_vq_desc_batch); +EXPORT_SYMBOL_GPL(vhost_get_vq_desc); /* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */ void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n) diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 0976a2853935..76356edee8e5 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -187,10 +187,6 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg bool vhost_vq_access_ok(struct vhost_virtqueue *vq); bool vhost_log_access_ok(struct vhost_dev *); -int vhost_get_vq_desc_batch(struct vhost_virtqueue *, - struct iovec iov[], unsigned int iov_count, - unsigned int *out_num, unsigned int *in_num, - struct vhost_log *log, unsigned int *log_num); int vhost_get_vq_desc(struct vhost_virtqueue *, struct iovec iov[], unsigned int iov_count, unsigned int *out_num, unsigned int *in_num, From patchwork Tue Jun 2 13:06:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Michael S. Tsirkin" X-Patchwork-Id: 218042 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C727BC433E1 for ; Tue, 2 Jun 2020 13:08:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9453720674 for ; Tue, 2 Jun 2020 13:08:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bpybDcQg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728310AbgFBNIR (ORCPT ); Tue, 2 Jun 2020 09:08:17 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:48225 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727804AbgFBNGL (ORCPT ); Tue, 2 Jun 2020 09:06:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591103168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3IWgXNVKogl+PigLuUvA2+zzqi/klGlvqGa6EgR6oPQ=; b=bpybDcQgTXk1PT1yMLBOcaD0cidgFEBe/1nWPhtyXOZewVYL7bbSr0bFvx23CGzsnMQvii CrM0cgbfmb03XeIJp7kuW4TaVuAx4z6vwCSM93INLaeVXLKQUW/LnEQxwdjZtDq1RhUTvg KYBpxefFLdgESnriTLS1LCSm+DZa864= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-161-LXQkGixjPbiY7iDBY_qzVA-1; Tue, 02 Jun 2020 09:06:05 -0400 X-MC-Unique: LXQkGixjPbiY7iDBY_qzVA-1 Received: by mail-wm1-f69.google.com with SMTP id u15so821164wmm.5 for ; Tue, 02 Jun 2020 06:06:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=3IWgXNVKogl+PigLuUvA2+zzqi/klGlvqGa6EgR6oPQ=; b=RMXZ0dNPLX/Wghu25G7JtjUSdi21HpAxRX3wDI5uAJIf5ymSOawHeEcDv0lWE+Xzsl kzzfJldxSu4/1rouvADBi9wGWzptuNyN/ggQx2+6KJdqh1SgljHEoCt94zJ+4Hw2TJmu dBi2Ll3cnprPdGhQIp294Rlp98u1HHrtXUmBlksIfqzlZ4HcBpYywbPCoXKjOEZjaKM6 D+R63xuyZcr6VMKC6Tz4f+oHiburbKAm2AXCXijJgi4bNj2o+cpexhn3W+rKgbcpo2+L XZ1khqaJco0d7wTF0NnUYB2tjT8+m6VvSQvbFEGvN4po7W8EZU3RkeCjXfjCf/VkG64O MkMw== X-Gm-Message-State: AOAM531X5rlgFHx8Gf57IH+AYIlDIu9vYtn2DSoBBQl5L1Obl6FhaX6S opEVNZLhaVNb01/eindzlIi8FX6xqjv76e17lkvj8GdiF1C6j8zSfq85hz8v52/TxUtaP/ua/9a rWY17ghkUIboJW+CZ X-Received: by 2002:adf:fc4b:: with SMTP id e11mr26457240wrs.202.1591103163732; Tue, 02 Jun 2020 06:06:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxbDJW9yzQN1NiCWnH3a4CKeN9AHsOoJMM0ESrlqkyLlEv5xh7pLBJWYQkMdgeqlIVO4xG2Fw== X-Received: by 2002:adf:fc4b:: with SMTP id e11mr26457209wrs.202.1591103163391; Tue, 02 Jun 2020 06:06:03 -0700 (PDT) Received: from redhat.com (bzq-109-64-41-91.red.bezeqint.net. [109.64.41.91]) by smtp.gmail.com with ESMTPSA id a3sm3462243wmb.7.2020.06.02.06.06.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 06:06:02 -0700 (PDT) Date: Tue, 2 Jun 2020 09:06:01 -0400 From: "Michael S. Tsirkin" To: linux-kernel@vger.kernel.org Cc: Eugenio =?utf-8?b?UMOpcmV6?= , Jason Wang , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Subject: [PATCH RFC 03/13] vhost: batching fetches Message-ID: <20200602130543.578420-4-mst@redhat.com> References: <20200602130543.578420-1-mst@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200602130543.578420-1-mst@redhat.com> X-Mailer: git-send-email 2.24.1.751.gd10ce2899c X-Mutt-Fcc: =sent Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org With this patch applied, new and old code perform identically. Lots of extra optimizations are now possible, e.g. we can fetch multiple heads with copy_from/to_user now. We can get rid of maintaining the log array. Etc etc. Signed-off-by: Michael S. Tsirkin Signed-off-by: Eugenio Pérez Link: https://lore.kernel.org/r/20200401183118.8334-4-eperezma@redhat.com Signed-off-by: Michael S. Tsirkin --- drivers/vhost/test.c | 2 +- drivers/vhost/vhost.c | 47 ++++++++++++++++++++++++++++++++++++++----- drivers/vhost/vhost.h | 5 ++++- 3 files changed, 47 insertions(+), 7 deletions(-) diff --git a/drivers/vhost/test.c b/drivers/vhost/test.c index 9a3a09005e03..02806d6f84ef 100644 --- a/drivers/vhost/test.c +++ b/drivers/vhost/test.c @@ -119,7 +119,7 @@ static int vhost_test_open(struct inode *inode, struct file *f) dev = &n->dev; vqs[VHOST_TEST_VQ] = &n->vqs[VHOST_TEST_VQ]; n->vqs[VHOST_TEST_VQ].handle_kick = handle_vq_kick; - vhost_dev_init(dev, vqs, VHOST_TEST_VQ_MAX, UIO_MAXIOV, + vhost_dev_init(dev, vqs, VHOST_TEST_VQ_MAX, UIO_MAXIOV + 64, VHOST_TEST_PKT_WEIGHT, VHOST_TEST_WEIGHT, NULL); f->private_data = n; diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 8f9a07282625..aca2a5b0d078 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -299,6 +299,7 @@ static void vhost_vq_reset(struct vhost_dev *dev, { vq->num = 1; vq->ndescs = 0; + vq->first_desc = 0; vq->desc = NULL; vq->avail = NULL; vq->used = NULL; @@ -367,6 +368,11 @@ static int vhost_worker(void *data) return 0; } +static int vhost_vq_num_batch_descs(struct vhost_virtqueue *vq) +{ + return vq->max_descs - UIO_MAXIOV; +} + static void vhost_vq_free_iovecs(struct vhost_virtqueue *vq) { kfree(vq->descs); @@ -389,6 +395,9 @@ static long vhost_dev_alloc_iovecs(struct vhost_dev *dev) for (i = 0; i < dev->nvqs; ++i) { vq = dev->vqs[i]; vq->max_descs = dev->iov_limit; + if (vhost_vq_num_batch_descs(vq) < 0) { + return -EINVAL; + } vq->descs = kmalloc_array(vq->max_descs, sizeof(*vq->descs), GFP_KERNEL); @@ -1570,6 +1579,7 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg vq->last_avail_idx = s.num; /* Forget the cached index value. */ vq->avail_idx = vq->last_avail_idx; + vq->ndescs = vq->first_desc = 0; break; case VHOST_GET_VRING_BASE: s.index = idx; @@ -2136,7 +2146,7 @@ static int fetch_indirect_descs(struct vhost_virtqueue *vq, return 0; } -static int fetch_descs(struct vhost_virtqueue *vq) +static int fetch_buf(struct vhost_virtqueue *vq) { unsigned int i, head, found = 0; struct vhost_desc *last; @@ -2149,7 +2159,11 @@ static int fetch_descs(struct vhost_virtqueue *vq) /* Check it isn't doing very strange things with descriptor numbers. */ last_avail_idx = vq->last_avail_idx; - if (vq->avail_idx == vq->last_avail_idx) { + if (unlikely(vq->avail_idx == vq->last_avail_idx)) { + /* If we already have work to do, don't bother re-checking. */ + if (likely(vq->ndescs)) + return vq->num; + if (unlikely(vhost_get_avail_idx(vq, &avail_idx))) { vq_err(vq, "Failed to access avail idx at %p\n", &vq->avail->idx); @@ -2240,6 +2254,24 @@ static int fetch_descs(struct vhost_virtqueue *vq) return 0; } +static int fetch_descs(struct vhost_virtqueue *vq) +{ + int ret = 0; + + if (unlikely(vq->first_desc >= vq->ndescs)) { + vq->first_desc = 0; + vq->ndescs = 0; + } + + if (vq->ndescs) + return 0; + + while (!ret && vq->ndescs <= vhost_vq_num_batch_descs(vq)) + ret = fetch_buf(vq); + + return vq->ndescs ? 0 : ret; +} + /* This looks in the virtqueue and for the first available buffer, and converts * it to an iovec for convenient access. Since descriptors consist of some * number of output then some number of input descriptors, it's actually two @@ -2265,7 +2297,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, if (unlikely(log)) *log_num = 0; - for (i = 0; i < vq->ndescs; ++i) { + for (i = vq->first_desc; i < vq->ndescs; ++i) { unsigned iov_count = *in_num + *out_num; struct vhost_desc *desc = &vq->descs[i]; int access; @@ -2311,14 +2343,19 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, } ret = desc->id; + + if (!(desc->flags & VRING_DESC_F_NEXT)) + break; } - vq->ndescs = 0; + vq->first_desc = i + 1; return ret; err: - vhost_discard_vq_desc(vq, 1); + for (i = vq->first_desc; i < vq->ndescs; ++i) + if (!(vq->descs[i].flags & VRING_DESC_F_NEXT)) + vhost_discard_vq_desc(vq, 1); vq->ndescs = 0; return ret; diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 76356edee8e5..a67bda9792ec 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -81,6 +81,7 @@ struct vhost_virtqueue { struct vhost_desc *descs; int ndescs; + int first_desc; int max_descs; struct file *kick; @@ -229,7 +230,7 @@ void vhost_iotlb_map_free(struct vhost_iotlb *iotlb, struct vhost_iotlb_map *map); #define vq_err(vq, fmt, ...) do { \ - pr_debug(pr_fmt(fmt), ##__VA_ARGS__); \ + pr_err(pr_fmt(fmt), ##__VA_ARGS__); \ if ((vq)->error_ctx) \ eventfd_signal((vq)->error_ctx, 1);\ } while (0) @@ -255,6 +256,8 @@ static inline void vhost_vq_set_backend(struct vhost_virtqueue *vq, void *private_data) { vq->private_data = private_data; + vq->ndescs = 0; + vq->first_desc = 0; } /** From patchwork Tue Jun 2 13:06:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Michael S. Tsirkin" X-Patchwork-Id: 218043 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AC1EC433E0 for ; Tue, 2 Jun 2020 13:07:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E12E720674 for ; Tue, 2 Jun 2020 13:07:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fSexiHW9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728214AbgFBNGO (ORCPT ); Tue, 2 Jun 2020 09:06:14 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:36712 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728083AbgFBNGK (ORCPT ); Tue, 2 Jun 2020 09:06:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591103169; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=EXfjPrDyqejhZJQKUWL2cxT5rK4Jnqa149Nn7T1gq18=; b=fSexiHW9l09yNuKon60LwwEn3hNOIHwld2GdSwkkeMqZKklg3vWr1LDJtLD3mFCDQF3o8f ZsLUfw9BBx8DygUvOh5pdqsvC0sCUC/6ZWepBzN3pwqaxbiJx88k3dWWBkYAG1/3j/7jP2 RmBZegbTUHIvfAL0yMl/BKc8Il6IpKg= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-224-GjyHK8i2M3SjeQsFJdkB4A-1; Tue, 02 Jun 2020 09:06:07 -0400 X-MC-Unique: GjyHK8i2M3SjeQsFJdkB4A-1 Received: by mail-wr1-f70.google.com with SMTP id d6so1383080wrn.1 for ; Tue, 02 Jun 2020 06:06:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=EXfjPrDyqejhZJQKUWL2cxT5rK4Jnqa149Nn7T1gq18=; b=kwhJA1mGUSnzOz8PKiuthIIYwFEpLOeugsOo2h994mjPMTN0mfG/NOWKOIZvpsCg8Q FiFIQneNfT52EOO34rfIpKbwQFgtPmCdKuZss2ufqJeJeJFaMcPPio4kCPn64T3ebbVj Dloi3JtBgl21YzOX49U/0aWe6pIDxV+6eXgUfrWLL7u6zaOObPn+jvgFGCoX3U/HMlQL 44ZDIhZEppbOzxkGY3y9QvnkIPRAd/7geGY1NEyRyidLS5+u4ZbBqF5BPEl33P5XILnh B5o0etMHhsSBPkqDUBNK1zVOnxDoNZblht7NS6v6Vz7M9jtr4xtxZ148RUxpQXaKKo5D FcGA== X-Gm-Message-State: AOAM530Uo6PDAH3KAQkrgoFq2Y4ig+Y+QpwR9yw4v2kGoKlHrhz77lZq SGVl57aINMEnBsGB3XwnhwoD7p0m1MwPbNGLw9sTSu1/Kl0BlgMe1ZjPN4OYCpV2iaJ//uPrerb WunsX8varI3sgBR1c X-Received: by 2002:a5d:6391:: with SMTP id p17mr28295032wru.118.1591103165844; Tue, 02 Jun 2020 06:06:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJycUSkpBa2fgUPLZAf/ybOapg7EllouqLaj5tY/W/L7i8gJ0j5W3iBZNA/fqWHGkhJHN82J9A== X-Received: by 2002:a5d:6391:: with SMTP id p17mr28295007wru.118.1591103165610; Tue, 02 Jun 2020 06:06:05 -0700 (PDT) Received: from redhat.com (bzq-109-64-41-91.red.bezeqint.net. [109.64.41.91]) by smtp.gmail.com with ESMTPSA id m129sm3760525wmf.2.2020.06.02.06.06.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 06:06:05 -0700 (PDT) Date: Tue, 2 Jun 2020 09:06:03 -0400 From: "Michael S. Tsirkin" To: linux-kernel@vger.kernel.org Cc: Eugenio =?utf-8?b?UMOpcmV6?= , Jason Wang , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Subject: [PATCH RFC 04/13] vhost: cleanup fetch_buf return code handling Message-ID: <20200602130543.578420-5-mst@redhat.com> References: <20200602130543.578420-1-mst@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200602130543.578420-1-mst@redhat.com> X-Mailer: git-send-email 2.24.1.751.gd10ce2899c X-Mutt-Fcc: =sent Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Return code of fetch_buf is confusing, so callers resort to tricks to get to sane values. Let's switch to something standard: 0 empty, >0 non-empty, <0 error. Signed-off-by: Michael S. Tsirkin --- drivers/vhost/vhost.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index aca2a5b0d078..bd52b44b0d23 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -2146,6 +2146,8 @@ static int fetch_indirect_descs(struct vhost_virtqueue *vq, return 0; } +/* This function returns a value > 0 if a descriptor was found, or 0 if none were found. + * A negative code is returned on error. */ static int fetch_buf(struct vhost_virtqueue *vq) { unsigned int i, head, found = 0; @@ -2162,7 +2164,7 @@ static int fetch_buf(struct vhost_virtqueue *vq) if (unlikely(vq->avail_idx == vq->last_avail_idx)) { /* If we already have work to do, don't bother re-checking. */ if (likely(vq->ndescs)) - return vq->num; + return 0; if (unlikely(vhost_get_avail_idx(vq, &avail_idx))) { vq_err(vq, "Failed to access avail idx at %p\n", @@ -2181,7 +2183,7 @@ static int fetch_buf(struct vhost_virtqueue *vq) * invalid. */ if (vq->avail_idx == last_avail_idx) - return vq->num; + return 0; /* Only get avail ring entries after they have been * exposed by guest. @@ -2251,12 +2253,14 @@ static int fetch_buf(struct vhost_virtqueue *vq) /* On success, increment avail index. */ vq->last_avail_idx++; - return 0; + return 1; } +/* This function returns a value > 0 if a descriptor was found, or 0 if none were found. + * A negative code is returned on error. */ static int fetch_descs(struct vhost_virtqueue *vq) { - int ret = 0; + int ret; if (unlikely(vq->first_desc >= vq->ndescs)) { vq->first_desc = 0; @@ -2266,10 +2270,14 @@ static int fetch_descs(struct vhost_virtqueue *vq) if (vq->ndescs) return 0; - while (!ret && vq->ndescs <= vhost_vq_num_batch_descs(vq)) - ret = fetch_buf(vq); + for (ret = 1; + ret > 0 && vq->ndescs <= vhost_vq_num_batch_descs(vq); + ret = fetch_buf(vq)) + ; - return vq->ndescs ? 0 : ret; + /* On success we expect some descs */ + BUG_ON(ret > 0 && !vq->ndescs); + return ret; } /* This looks in the virtqueue and for the first available buffer, and converts @@ -2288,7 +2296,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, int ret = fetch_descs(vq); int i; - if (ret) + if (ret <= 0) return ret; /* Now convert to IOV */ From patchwork Tue Jun 2 13:06:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Michael S. Tsirkin" X-Patchwork-Id: 218047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AC80C433DF for ; Tue, 2 Jun 2020 13:06:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4495020674 for ; Tue, 2 Jun 2020 13:06:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LuXp1wpS" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728330AbgFBNGU (ORCPT ); Tue, 2 Jun 2020 09:06:20 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:21446 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728114AbgFBNGS (ORCPT ); Tue, 2 Jun 2020 09:06:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591103175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+9ArrlnZ76Z7x7tEJ0GmRFFHnk/lfGSrrAX14yhuVlU=; b=LuXp1wpSQyjdD4BbMfO+NICx+gSs404gET+3eNhfpzGgjqbexSTPD1bG2T40crmajQ6dLk sDSGiwV0Q4SrvffmERnsoXOHS/SIwd1Y0vtp1ntul/ge1bND584GKEwFJw4N/zmTPh3Kpj K0I+JEChJGrhzntUCTGYBw9/sY6oF64= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-350-QOXxdYUNNieViH1cQzqK7g-1; Tue, 02 Jun 2020 09:06:14 -0400 X-MC-Unique: QOXxdYUNNieViH1cQzqK7g-1 Received: by mail-wm1-f70.google.com with SMTP id p24so936826wmc.1 for ; Tue, 02 Jun 2020 06:06:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=+9ArrlnZ76Z7x7tEJ0GmRFFHnk/lfGSrrAX14yhuVlU=; b=lGaF/tCLDwPbrl63/8ILXLmhxZ52QZMtl3NuB3VYCN0osjnpJHayqZe/98oZPIKpUX Wh7R6p/ANO12vJFV4PFVCzf5L73eqSM2dG97aLQTA0XUSGgokk3dnXuBwGFl7Pyk7eou dGnl3GSkcAJoMdt0VbU38HHG+xUFJMsf84xokRQoeLycL7HSUXNMdAfycOWtXPYMIkcl JhKlEp/tfzhH02OkpZKirftBBBASPDUixxS2AvMjjtKlw9rVc2C3UbnmxWl4NEXkinzU otb+baUj2OM466HXxRRGg2uyqU5Di4GNnmjNdE4meldsrUfSAnA0kaBZkvWfDQUw7e43 h4og== X-Gm-Message-State: AOAM533blUEvUNBP4yvPRDHREXMXONIFqZKJKoGQ5A5vb0GCiTmzEryy wR5zwF7VcQjcJnIhFHvvJ9tDLoY14DURy4BPP5HkPMdDGhE8d5IFaAyCGbY2QJ3rExcQ/u1Eyxv GVZTPUkqpO69SY/U+ X-Received: by 2002:a7b:c201:: with SMTP id x1mr4041710wmi.58.1591103172911; Tue, 02 Jun 2020 06:06:12 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxAQ1hhO0DwfriHXGOJkXG1RkMZz6fc+8BSr/fDJ8YL0qd5AbUcevXhy22swxOsw0LrNtT9bQ== X-Received: by 2002:a7b:c201:: with SMTP id x1mr4041687wmi.58.1591103172611; Tue, 02 Jun 2020 06:06:12 -0700 (PDT) Received: from redhat.com (bzq-109-64-41-91.red.bezeqint.net. [109.64.41.91]) by smtp.gmail.com with ESMTPSA id s7sm3687653wrr.60.2020.06.02.06.06.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 06:06:12 -0700 (PDT) Date: Tue, 2 Jun 2020 09:06:10 -0400 From: "Michael S. Tsirkin" To: linux-kernel@vger.kernel.org Cc: Eugenio =?utf-8?b?UMOpcmV6?= , Jason Wang , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Subject: [PATCH RFC 07/13] vhost: format-independent API for used buffers Message-ID: <20200602130543.578420-8-mst@redhat.com> References: <20200602130543.578420-1-mst@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200602130543.578420-1-mst@redhat.com> X-Mailer: git-send-email 2.24.1.751.gd10ce2899c X-Mutt-Fcc: =sent Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add a new API that doesn't assume used ring, heads, etc. For now, we keep the old APIs around to make it easier to convert drivers. Signed-off-by: Michael S. Tsirkin --- drivers/vhost/vhost.c | 52 ++++++++++++++++++++++++++++++++++--------- drivers/vhost/vhost.h | 17 +++++++++++++- 2 files changed, 58 insertions(+), 11 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index b4a6e44d56a8..be822f0c9428 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -2292,13 +2292,12 @@ static int fetch_descs(struct vhost_virtqueue *vq) * number of output then some number of input descriptors, it's actually two * iovecs, but we pack them into one and note how many of each there were. * - * This function returns the descriptor number found, or vq->num (which is - * never a valid descriptor number) if none was found. A negative code is - * returned on error. */ -int vhost_get_vq_desc(struct vhost_virtqueue *vq, - struct iovec iov[], unsigned int iov_size, - unsigned int *out_num, unsigned int *in_num, - struct vhost_log *log, unsigned int *log_num) + * This function returns a value > 0 if a descriptor was found, or 0 if none were found. + * A negative code is returned on error. */ +int vhost_get_avail_buf(struct vhost_virtqueue *vq, struct vhost_buf *buf, + struct iovec iov[], unsigned int iov_size, + unsigned int *out_num, unsigned int *in_num, + struct vhost_log *log, unsigned int *log_num) { int ret = fetch_descs(vq); int i; @@ -2311,6 +2310,8 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, *out_num = *in_num = 0; if (unlikely(log)) *log_num = 0; + buf->in_len = buf->out_len = 0; + buf->descs = 0; for (i = vq->first_desc; i < vq->ndescs; ++i) { unsigned iov_count = *in_num + *out_num; @@ -2340,6 +2341,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, /* If this is an input descriptor, * increment that count. */ *in_num += ret; + buf->in_len += desc->len; if (unlikely(log && ret)) { log[*log_num].addr = desc->addr; log[*log_num].len = desc->len; @@ -2355,9 +2357,11 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, goto err; } *out_num += ret; + buf->out_len += desc->len; } - ret = desc->id; + buf->id = desc->id; + ++buf->descs; if (!(desc->flags & VRING_DESC_F_NEXT)) break; @@ -2365,7 +2369,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, vq->first_desc = i + 1; - return ret; + return 1; err: for (i = vq->first_desc; i < vq->ndescs; ++i) @@ -2375,7 +2379,15 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, return ret; } -EXPORT_SYMBOL_GPL(vhost_get_vq_desc); +EXPORT_SYMBOL_GPL(vhost_get_avail_buf); + +/* Reverse the effect of vhost_get_avail_buf. Useful for error handling. */ +void vhost_discard_avail_bufs(struct vhost_virtqueue *vq, + struct vhost_buf *buf, unsigned count) +{ + vhost_discard_vq_desc(vq, count); +} +EXPORT_SYMBOL_GPL(vhost_discard_avail_bufs); static int __vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads, @@ -2459,6 +2471,26 @@ int vhost_add_used(struct vhost_virtqueue *vq, unsigned int head, int len) } EXPORT_SYMBOL_GPL(vhost_add_used); +int vhost_put_used_buf(struct vhost_virtqueue *vq, struct vhost_buf *buf) +{ + return vhost_add_used(vq, buf->id, buf->in_len); +} +EXPORT_SYMBOL_GPL(vhost_put_used_buf); + +int vhost_put_used_n_bufs(struct vhost_virtqueue *vq, + struct vhost_buf *bufs, unsigned count) +{ + unsigned i; + + for (i = 0; i < count; ++i) { + vq->heads[i].id = cpu_to_vhost32(vq, bufs[i].id); + vq->heads[i].len = cpu_to_vhost32(vq, bufs[i].in_len); + } + + return vhost_add_used_n(vq, vq->heads, count); +} +EXPORT_SYMBOL_GPL(vhost_put_used_n_bufs); + static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) { __u16 old, new; diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index a67bda9792ec..6c10e99ff334 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -67,6 +67,13 @@ struct vhost_desc { u16 id; }; +struct vhost_buf { + u32 out_len; + u32 in_len; + u16 descs; + u16 id; +}; + /* The virtqueue structure describes a queue attached to a device. */ struct vhost_virtqueue { struct vhost_dev *dev; @@ -193,7 +200,12 @@ int vhost_get_vq_desc(struct vhost_virtqueue *, unsigned int *out_num, unsigned int *in_num, struct vhost_log *log, unsigned int *log_num); void vhost_discard_vq_desc(struct vhost_virtqueue *, int n); - +int vhost_get_avail_buf(struct vhost_virtqueue *, struct vhost_buf *buf, + struct iovec iov[], unsigned int iov_count, + unsigned int *out_num, unsigned int *in_num, + struct vhost_log *log, unsigned int *log_num); +void vhost_discard_avail_bufs(struct vhost_virtqueue *, + struct vhost_buf *, unsigned count); int vhost_vq_init_access(struct vhost_virtqueue *); int vhost_add_used(struct vhost_virtqueue *, unsigned int head, int len); int vhost_add_used_n(struct vhost_virtqueue *, struct vring_used_elem *heads, @@ -202,6 +214,9 @@ void vhost_add_used_and_signal(struct vhost_dev *, struct vhost_virtqueue *, unsigned int id, int len); void vhost_add_used_and_signal_n(struct vhost_dev *, struct vhost_virtqueue *, struct vring_used_elem *heads, unsigned count); +int vhost_put_used_buf(struct vhost_virtqueue *, struct vhost_buf *buf); +int vhost_put_used_n_bufs(struct vhost_virtqueue *, + struct vhost_buf *bufs, unsigned count); void vhost_signal(struct vhost_dev *, struct vhost_virtqueue *); void vhost_disable_notify(struct vhost_dev *, struct vhost_virtqueue *); bool vhost_vq_avail_empty(struct vhost_dev *, struct vhost_virtqueue *); From patchwork Tue Jun 2 13:06:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Michael S. Tsirkin" X-Patchwork-Id: 218044 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E45E1C433E0 for ; Tue, 2 Jun 2020 13:07:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B103120674 for ; Tue, 2 Jun 2020 13:07:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XLAzDElJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728511AbgFBNHG (ORCPT ); Tue, 2 Jun 2020 09:07:06 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:24797 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727993AbgFBNGV (ORCPT ); Tue, 2 Jun 2020 09:06:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591103178; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=6LjFF09GNTgapUCQye5EFdG/jfLEFmFuYlAzJu5ptC0=; b=XLAzDElJ9Eyj6NG6LX0b7KgyLCc8TZbpOfmQUZMI8BT7Be5kEkrNIDtgGePAESaXpOp7Qb Qc2+IzmHzxJy7HRs10i9q8QKtKLdYMdeKkidG1SRtDSxv810GFIUfIGSmB485BrzRfNUOi /7aDQM9XTwYnAVH3oIAIqCvKF8bgnvw= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-428-qgkzTqUiOOiVJVtg_30qkQ-1; Tue, 02 Jun 2020 09:06:17 -0400 X-MC-Unique: qgkzTqUiOOiVJVtg_30qkQ-1 Received: by mail-wr1-f72.google.com with SMTP id f4so1368684wrp.21 for ; Tue, 02 Jun 2020 06:06:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=6LjFF09GNTgapUCQye5EFdG/jfLEFmFuYlAzJu5ptC0=; b=MQsFXKMdkK6XE5c4vzJKnFKvmEmsjXC/qj989qXZJ6hDga9pcHcxSI/ddHqx/2AKkP hfdeD7eMZUA4l4W7IaCiM6FH4d0WNzC8hwpCLKvDOf3mtvRYP9zqVFAp3dFcT8aNMVB0 nAtgD+96l8yb6tlIbuKfvn+aptJeWQeEEySfu4AdXwmLhF0rYa1ktAUwWqFZUTdRIli0 yCoVBmMJ4VPNpE/nLmHgsVHcRHXbeNv/TjXApWnO7RAbixJw4mHIjGHZleHPXW6mxaCc rWYyHly6DcSBwE5sEeNo2ZAAgQvgWl9mNkYuq2C2rjehpFySy59P4pFDqPXi/aNEzq3s xV8w== X-Gm-Message-State: AOAM530uQiZ0tZN0N5o57gDNKN9gxRvlMY/afxh+bnTxUuXh4qhZsFFF N6stbdbBApZn69IS5B0sQcY37UppHvUlxMF2Fyd6gQKiC6kQIoivWkaGnEJRFbZ/cyLX82lzPjh QmX/RAfbVEgUHFhta X-Received: by 2002:adf:f58b:: with SMTP id f11mr26486562wro.155.1591103175572; Tue, 02 Jun 2020 06:06:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwJk2gPbQL5PIo6tiUhknyXIJdqgEJHs3GNaPHN+6GAAY4cxZnjQts8QfWklgl+QRzL6ncSOg== X-Received: by 2002:adf:f58b:: with SMTP id f11mr26486535wro.155.1591103175036; Tue, 02 Jun 2020 06:06:15 -0700 (PDT) Received: from redhat.com (bzq-109-64-41-91.red.bezeqint.net. [109.64.41.91]) by smtp.gmail.com with ESMTPSA id e5sm3853312wrw.19.2020.06.02.06.06.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 06:06:14 -0700 (PDT) Date: Tue, 2 Jun 2020 09:06:13 -0400 From: "Michael S. Tsirkin" To: linux-kernel@vger.kernel.org Cc: Eugenio =?utf-8?b?UMOpcmV6?= , Jason Wang , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Subject: [PATCH RFC 08/13] vhost/net: convert to new API: heads->bufs Message-ID: <20200602130543.578420-9-mst@redhat.com> References: <20200602130543.578420-1-mst@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200602130543.578420-1-mst@redhat.com> X-Mailer: git-send-email 2.24.1.751.gd10ce2899c X-Mutt-Fcc: =sent Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Convert vhost net to use the new format-agnostic API. In particular, don't poke at vq internals such as the heads array. Signed-off-by: Michael S. Tsirkin --- drivers/vhost/net.c | 153 +++++++++++++++++++++++--------------------- 1 file changed, 81 insertions(+), 72 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 749a9cf51a59..47af3d1ce3dd 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -59,13 +59,13 @@ MODULE_PARM_DESC(experimental_zcopytx, "Enable Zero Copy TX;" * status internally; used for zerocopy tx only. */ /* Lower device DMA failed */ -#define VHOST_DMA_FAILED_LEN ((__force __virtio32)3) +#define VHOST_DMA_FAILED_LEN (3) /* Lower device DMA done */ -#define VHOST_DMA_DONE_LEN ((__force __virtio32)2) +#define VHOST_DMA_DONE_LEN (2) /* Lower device DMA in progress */ -#define VHOST_DMA_IN_PROGRESS ((__force __virtio32)1) +#define VHOST_DMA_IN_PROGRESS (1) /* Buffer unused */ -#define VHOST_DMA_CLEAR_LEN ((__force __virtio32)0) +#define VHOST_DMA_CLEAR_LEN (0) #define VHOST_DMA_IS_DONE(len) ((__force u32)(len) >= (__force u32)VHOST_DMA_DONE_LEN) @@ -112,9 +112,12 @@ struct vhost_net_virtqueue { /* last used idx for outstanding DMA zerocopy buffers */ int upend_idx; /* For TX, first used idx for DMA done zerocopy buffers - * For RX, number of batched heads + * For RX, number of batched bufs */ int done_idx; + /* Outstanding user bufs. UIO_MAXIOV in length. */ + /* TODO: we can make this smaller for sure. */ + struct vhost_buf *bufs; /* Number of XDP frames batched */ int batched_xdp; /* an array of userspace buffers info */ @@ -271,6 +274,8 @@ static void vhost_net_clear_ubuf_info(struct vhost_net *n) int i; for (i = 0; i < VHOST_NET_VQ_MAX; ++i) { + kfree(n->vqs[i].bufs); + n->vqs[i].bufs = NULL; kfree(n->vqs[i].ubuf_info); n->vqs[i].ubuf_info = NULL; } @@ -282,6 +287,12 @@ static int vhost_net_set_ubuf_info(struct vhost_net *n) int i; for (i = 0; i < VHOST_NET_VQ_MAX; ++i) { + n->vqs[i].bufs = kmalloc_array(UIO_MAXIOV, + sizeof(*n->vqs[i].bufs), + GFP_KERNEL); + if (!n->vqs[i].bufs) + goto err; + zcopy = vhost_net_zcopy_mask & (0x1 << i); if (!zcopy) continue; @@ -364,18 +375,18 @@ static void vhost_zerocopy_signal_used(struct vhost_net *net, int j = 0; for (i = nvq->done_idx; i != nvq->upend_idx; i = (i + 1) % UIO_MAXIOV) { - if (vq->heads[i].len == VHOST_DMA_FAILED_LEN) + if (nvq->bufs[i].in_len == VHOST_DMA_FAILED_LEN) vhost_net_tx_err(net); - if (VHOST_DMA_IS_DONE(vq->heads[i].len)) { - vq->heads[i].len = VHOST_DMA_CLEAR_LEN; + if (VHOST_DMA_IS_DONE(nvq->bufs[i].in_len)) { + nvq->bufs[i].in_len = VHOST_DMA_CLEAR_LEN; ++j; } else break; } while (j) { add = min(UIO_MAXIOV - nvq->done_idx, j); - vhost_add_used_and_signal_n(vq->dev, vq, - &vq->heads[nvq->done_idx], add); + vhost_put_used_n_bufs(vq, &nvq->bufs[nvq->done_idx], add); + vhost_signal(vq->dev, vq); nvq->done_idx = (nvq->done_idx + add) % UIO_MAXIOV; j -= add; } @@ -390,7 +401,7 @@ static void vhost_zerocopy_callback(struct ubuf_info *ubuf, bool success) rcu_read_lock_bh(); /* set len to mark this desc buffers done DMA */ - nvq->vq.heads[ubuf->desc].in_len = success ? + nvq->bufs[ubuf->desc].in_len = success ? VHOST_DMA_DONE_LEN : VHOST_DMA_FAILED_LEN; cnt = vhost_net_ubuf_put(ubufs); @@ -452,7 +463,8 @@ static void vhost_net_signal_used(struct vhost_net_virtqueue *nvq) if (!nvq->done_idx) return; - vhost_add_used_and_signal_n(dev, vq, vq->heads, nvq->done_idx); + vhost_put_used_n_bufs(vq, nvq->bufs, nvq->done_idx); + vhost_signal(dev, vq); nvq->done_idx = 0; } @@ -558,6 +570,7 @@ static void vhost_net_busy_poll(struct vhost_net *net, static int vhost_net_tx_get_vq_desc(struct vhost_net *net, struct vhost_net_virtqueue *tnvq, + struct vhost_buf *buf, unsigned int *out_num, unsigned int *in_num, struct msghdr *msghdr, bool *busyloop_intr) { @@ -565,10 +578,10 @@ static int vhost_net_tx_get_vq_desc(struct vhost_net *net, struct vhost_virtqueue *rvq = &rnvq->vq; struct vhost_virtqueue *tvq = &tnvq->vq; - int r = vhost_get_vq_desc(tvq, tvq->iov, ARRAY_SIZE(tvq->iov), - out_num, in_num, NULL, NULL); + int r = vhost_get_avail_buf(tvq, buf, tvq->iov, ARRAY_SIZE(tvq->iov), + out_num, in_num, NULL, NULL); - if (r == tvq->num && tvq->busyloop_timeout) { + if (!r && tvq->busyloop_timeout) { /* Flush batched packets first */ if (!vhost_sock_zcopy(vhost_vq_get_backend(tvq))) vhost_tx_batch(net, tnvq, @@ -577,8 +590,8 @@ static int vhost_net_tx_get_vq_desc(struct vhost_net *net, vhost_net_busy_poll(net, rvq, tvq, busyloop_intr, false); - r = vhost_get_vq_desc(tvq, tvq->iov, ARRAY_SIZE(tvq->iov), - out_num, in_num, NULL, NULL); + r = vhost_get_avail_buf(tvq, buf, tvq->iov, ARRAY_SIZE(tvq->iov), + out_num, in_num, NULL, NULL); } return r; @@ -607,6 +620,7 @@ static size_t init_iov_iter(struct vhost_virtqueue *vq, struct iov_iter *iter, static int get_tx_bufs(struct vhost_net *net, struct vhost_net_virtqueue *nvq, + struct vhost_buf *buf, struct msghdr *msg, unsigned int *out, unsigned int *in, size_t *len, bool *busyloop_intr) @@ -614,9 +628,9 @@ static int get_tx_bufs(struct vhost_net *net, struct vhost_virtqueue *vq = &nvq->vq; int ret; - ret = vhost_net_tx_get_vq_desc(net, nvq, out, in, msg, busyloop_intr); + ret = vhost_net_tx_get_vq_desc(net, nvq, buf, out, in, msg, busyloop_intr); - if (ret < 0 || ret == vq->num) + if (ret <= 0) return ret; if (*in) { @@ -761,7 +775,7 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock) struct vhost_net_virtqueue *nvq = &net->vqs[VHOST_NET_VQ_TX]; struct vhost_virtqueue *vq = &nvq->vq; unsigned out, in; - int head; + int ret; struct msghdr msg = { .msg_name = NULL, .msg_namelen = 0, @@ -773,6 +787,7 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock) int err; int sent_pkts = 0; bool sock_can_batch = (sock->sk->sk_sndbuf == INT_MAX); + struct vhost_buf buf; do { bool busyloop_intr = false; @@ -780,13 +795,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock) if (nvq->done_idx == VHOST_NET_BATCH) vhost_tx_batch(net, nvq, sock, &msg); - head = get_tx_bufs(net, nvq, &msg, &out, &in, &len, - &busyloop_intr); + ret = get_tx_bufs(net, nvq, &buf, &msg, &out, &in, &len, + &busyloop_intr); /* On error, stop handling until the next kick. */ - if (unlikely(head < 0)) + if (unlikely(ret < 0)) break; /* Nothing new? Wait for eventfd to tell us they refilled. */ - if (head == vq->num) { + if (!ret) { if (unlikely(busyloop_intr)) { vhost_poll_queue(&vq->poll); } else if (unlikely(vhost_enable_notify(&net->dev, @@ -808,7 +823,7 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock) goto done; } else if (unlikely(err != -ENOSPC)) { vhost_tx_batch(net, nvq, sock, &msg); - vhost_discard_vq_desc(vq, 1); + vhost_discard_avail_bufs(vq, &buf, 1); vhost_net_enable_vq(net, vq); break; } @@ -829,7 +844,7 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock) /* TODO: Check specific error and bomb out unless ENOBUFS? */ err = sock->ops->sendmsg(sock, &msg, len); if (unlikely(err < 0)) { - vhost_discard_vq_desc(vq, 1); + vhost_discard_avail_bufs(vq, &buf, 1); vhost_net_enable_vq(net, vq); break; } @@ -837,8 +852,7 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock) pr_debug("Truncated TX packet: len %d != %zd\n", err, len); done: - vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head); - vq->heads[nvq->done_idx].len = 0; + nvq->bufs[nvq->done_idx] = buf; ++nvq->done_idx; } while (likely(!vhost_exceeds_weight(vq, ++sent_pkts, total_len))); @@ -850,7 +864,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) struct vhost_net_virtqueue *nvq = &net->vqs[VHOST_NET_VQ_TX]; struct vhost_virtqueue *vq = &nvq->vq; unsigned out, in; - int head; + int ret; struct msghdr msg = { .msg_name = NULL, .msg_namelen = 0, @@ -864,6 +878,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) struct vhost_net_ubuf_ref *uninitialized_var(ubufs); bool zcopy_used; int sent_pkts = 0; + struct vhost_buf buf; do { bool busyloop_intr; @@ -872,13 +887,13 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) vhost_zerocopy_signal_used(net, vq); busyloop_intr = false; - head = get_tx_bufs(net, nvq, &msg, &out, &in, &len, - &busyloop_intr); + ret = get_tx_bufs(net, nvq, &buf, &msg, &out, &in, &len, + &busyloop_intr); /* On error, stop handling until the next kick. */ - if (unlikely(head < 0)) + if (unlikely(ret < 0)) break; /* Nothing new? Wait for eventfd to tell us they refilled. */ - if (head == vq->num) { + if (!ret) { if (unlikely(busyloop_intr)) { vhost_poll_queue(&vq->poll); } else if (unlikely(vhost_enable_notify(&net->dev, vq))) { @@ -897,8 +912,8 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) struct ubuf_info *ubuf; ubuf = nvq->ubuf_info + nvq->upend_idx; - vq->heads[nvq->upend_idx].id = cpu_to_vhost32(vq, head); - vq->heads[nvq->upend_idx].len = VHOST_DMA_IN_PROGRESS; + nvq->bufs[nvq->upend_idx] = buf; + nvq->bufs[nvq->upend_idx].in_len = VHOST_DMA_IN_PROGRESS; ubuf->callback = vhost_zerocopy_callback; ubuf->ctx = nvq->ubufs; ubuf->desc = nvq->upend_idx; @@ -930,17 +945,19 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) nvq->upend_idx = ((unsigned)nvq->upend_idx - 1) % UIO_MAXIOV; } - vhost_discard_vq_desc(vq, 1); + vhost_discard_avail_bufs(vq, &buf, 1); vhost_net_enable_vq(net, vq); break; } if (err != len) pr_debug("Truncated TX packet: " " len %d != %zd\n", err, len); - if (!zcopy_used) - vhost_add_used_and_signal(&net->dev, vq, head, 0); - else + if (!zcopy_used) { + vhost_put_used_buf(vq, &buf); + vhost_signal(&net->dev, vq); + } else { vhost_zerocopy_signal_used(net, vq); + } vhost_net_tx_packet(net); } while (likely(!vhost_exceeds_weight(vq, ++sent_pkts, total_len))); } @@ -1004,7 +1021,7 @@ static int vhost_net_rx_peek_head_len(struct vhost_net *net, struct sock *sk, int len = peek_head_len(rnvq, sk); if (!len && rvq->busyloop_timeout) { - /* Flush batched heads first */ + /* Flush batched bufs first */ vhost_net_signal_used(rnvq); /* Both tx vq and rx socket were polled here */ vhost_net_busy_poll(net, rvq, tvq, busyloop_intr, true); @@ -1022,11 +1039,11 @@ static int vhost_net_rx_peek_head_len(struct vhost_net *net, struct sock *sk, * @iovcount - returned count of io vectors we fill * @log - vhost log * @log_num - log offset - * @quota - headcount quota, 1 for big buffer - * returns number of buffer heads allocated, negative on error + * @quota - bufcount quota, 1 for big buffer + * returns number of buffers allocated, negative on error */ static int get_rx_bufs(struct vhost_virtqueue *vq, - struct vring_used_elem *heads, + struct vhost_buf *bufs, int datalen, unsigned *iovcount, struct vhost_log *log, @@ -1035,30 +1052,24 @@ static int get_rx_bufs(struct vhost_virtqueue *vq, { unsigned int out, in; int seg = 0; - int headcount = 0; - unsigned d; + int bufcount = 0; int r, nlogs = 0; /* len is always initialized before use since we are always called with * datalen > 0. */ u32 uninitialized_var(len); - while (datalen > 0 && headcount < quota) { + while (datalen > 0 && bufcount < quota) { if (unlikely(seg >= UIO_MAXIOV)) { r = -ENOBUFS; goto err; } - r = vhost_get_vq_desc(vq, vq->iov + seg, - ARRAY_SIZE(vq->iov) - seg, &out, - &in, log, log_num); - if (unlikely(r < 0)) + r = vhost_get_avail_buf(vq, bufs + bufcount, vq->iov + seg, + ARRAY_SIZE(vq->iov) - seg, &out, + &in, log, log_num); + if (unlikely(r <= 0)) goto err; - d = r; - if (d == vq->num) { - r = 0; - goto err; - } if (unlikely(out || in <= 0)) { vq_err(vq, "unexpected descriptor format for RX: " "out %d, in %d\n", out, in); @@ -1069,14 +1080,12 @@ static int get_rx_bufs(struct vhost_virtqueue *vq, nlogs += *log_num; log += *log_num; } - heads[headcount].id = cpu_to_vhost32(vq, d); len = iov_length(vq->iov + seg, in); - heads[headcount].len = cpu_to_vhost32(vq, len); datalen -= len; - ++headcount; + ++bufcount; seg += in; } - heads[headcount - 1].len = cpu_to_vhost32(vq, len + datalen); + bufs[bufcount - 1].in_len = len + datalen; *iovcount = seg; if (unlikely(log)) *log_num = nlogs; @@ -1086,9 +1095,9 @@ static int get_rx_bufs(struct vhost_virtqueue *vq, r = UIO_MAXIOV + 1; goto err; } - return headcount; + return bufcount; err: - vhost_discard_vq_desc(vq, headcount); + vhost_discard_avail_bufs(vq, bufs, bufcount); return r; } @@ -1113,7 +1122,7 @@ static void handle_rx(struct vhost_net *net) }; size_t total_len = 0; int err, mergeable; - s16 headcount; + int bufcount; size_t vhost_hlen, sock_hlen; size_t vhost_len, sock_len; bool busyloop_intr = false; @@ -1147,14 +1156,14 @@ static void handle_rx(struct vhost_net *net) break; sock_len += sock_hlen; vhost_len = sock_len + vhost_hlen; - headcount = get_rx_bufs(vq, vq->heads + nvq->done_idx, - vhost_len, &in, vq_log, &log, - likely(mergeable) ? UIO_MAXIOV : 1); + bufcount = get_rx_bufs(vq, nvq->bufs + nvq->done_idx, + vhost_len, &in, vq_log, &log, + likely(mergeable) ? UIO_MAXIOV : 1); /* On error, stop handling until the next kick. */ - if (unlikely(headcount < 0)) + if (unlikely(bufcount < 0)) goto out; /* OK, now we need to know about added descriptors. */ - if (!headcount) { + if (!bufcount) { if (unlikely(busyloop_intr)) { vhost_poll_queue(&vq->poll); } else if (unlikely(vhost_enable_notify(&net->dev, vq))) { @@ -1171,7 +1180,7 @@ static void handle_rx(struct vhost_net *net) if (nvq->rx_ring) msg.msg_control = vhost_net_buf_consume(&nvq->rxq); /* On overrun, truncate and discard */ - if (unlikely(headcount > UIO_MAXIOV)) { + if (unlikely(bufcount > UIO_MAXIOV)) { iov_iter_init(&msg.msg_iter, READ, vq->iov, 1, 1); err = sock->ops->recvmsg(sock, &msg, 1, MSG_DONTWAIT | MSG_TRUNC); @@ -1195,7 +1204,7 @@ static void handle_rx(struct vhost_net *net) if (unlikely(err != sock_len)) { pr_debug("Discarded rx packet: " " len %d, expected %zd\n", err, sock_len); - vhost_discard_vq_desc(vq, headcount); + vhost_discard_avail_bufs(vq, nvq->bufs + nvq->done_idx, bufcount); continue; } /* Supply virtio_net_hdr if VHOST_NET_F_VIRTIO_NET_HDR */ @@ -1214,15 +1223,15 @@ static void handle_rx(struct vhost_net *net) } /* TODO: Should check and handle checksum. */ - num_buffers = cpu_to_vhost16(vq, headcount); + num_buffers = cpu_to_vhost16(vq, bufcount); if (likely(mergeable) && copy_to_iter(&num_buffers, sizeof num_buffers, &fixup) != sizeof num_buffers) { vq_err(vq, "Failed num_buffers write"); - vhost_discard_vq_desc(vq, headcount); + vhost_discard_avail_bufs(vq, nvq->bufs + nvq->done_idx, bufcount); goto out; } - nvq->done_idx += headcount; + nvq->done_idx += bufcount; if (nvq->done_idx > VHOST_NET_BATCH) vhost_net_signal_used(nvq); if (unlikely(vq_log)) From patchwork Tue Jun 2 13:06:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Michael S. Tsirkin" X-Patchwork-Id: 218045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.1 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C38AC433DF for ; Tue, 2 Jun 2020 13:06:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D9CBD20674 for ; Tue, 2 Jun 2020 13:06:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VrgHtPis" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728485AbgFBNG5 (ORCPT ); Tue, 2 Jun 2020 09:06:57 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:35415 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728358AbgFBNG3 (ORCPT ); Tue, 2 Jun 2020 09:06:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591103187; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=nZ+SseV/ZcYFSq4g3IfECGD3NtX/OI1iTKB8+Z0Evh0=; b=VrgHtPis2kvjyQFTxq9acZGG2LxhXUUQRr+PaZUfF3sfjId2ZoczoAZKUGzdMhSeZVC0Ws M5vv7bzG5paBrezj2IJOaTtZMXHVhdczyhZhkb8Sz6OXR8ae6kyB73/kq9c48XSbFJgSd0 tVKvuYDBnxjfTg1Ay6c7I8Q92WTJCuw= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-236-GqRwbR9UOVmh6x-tzZVgxg-1; Tue, 02 Jun 2020 09:06:26 -0400 X-MC-Unique: GqRwbR9UOVmh6x-tzZVgxg-1 Received: by mail-wr1-f71.google.com with SMTP id l1so1374581wrc.8 for ; Tue, 02 Jun 2020 06:06:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=nZ+SseV/ZcYFSq4g3IfECGD3NtX/OI1iTKB8+Z0Evh0=; b=lY0E4+JcVXYMzvoUWCcPEwPruLfUdU7f20C5RzkUJrC2KyE09zd8LDAQ5MiGAav3IW 3ZJG75UDKPfdw832beaOnSVpOAmx0HtDuI5/l1TV5lFgo3YDePJiCQPFj9OSBlPqrxBt S2vYhE96WRwJDapPSMe3P7M8AYES165C35Q00wcH1hXS6sz3iEEgUecmChOvj327ySWE eTQVBzEYJfuBT/MdlAAOIGCBkhOO/Zl+8iTx/Nd+HdoNhXy3MI+IwTZVT2H8VbJ/vYJq mv96OaEpbmilY18syBDfec+Qk2E1sh8LVkL6l+F7OV8OY+hR6aYBdY91QGCi67V3G9nI nSHw== X-Gm-Message-State: AOAM53327FtVvNtdD4az25WJt/fqiacZO+5CZxUtksBxGDea7/agrR5s uqE/f7tFvGHgyeq4SSkX0owPucYZ+yn9QLVlzEyCs3T+aJDWvYiMuRXFQ3XLuo4LcWXJf/uNS1i 0lUQ7UL3fMrt8CT3f X-Received: by 2002:a5d:6289:: with SMTP id k9mr27798491wru.358.1591103184856; Tue, 02 Jun 2020 06:06:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzV89PYQkmoqnUsoXjBfNQaiS/ysgLH5sFD2QE7LqFLiCerhfIUk7XIzd7ub9CmcrxwG1Onyg== X-Received: by 2002:a5d:6289:: with SMTP id k9mr27798483wru.358.1591103184639; Tue, 02 Jun 2020 06:06:24 -0700 (PDT) Received: from redhat.com (bzq-109-64-41-91.red.bezeqint.net. [109.64.41.91]) by smtp.gmail.com with ESMTPSA id o10sm3857545wrj.37.2020.06.02.06.06.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 06:06:24 -0700 (PDT) Date: Tue, 2 Jun 2020 09:06:22 -0400 From: "Michael S. Tsirkin" To: linux-kernel@vger.kernel.org Cc: Eugenio =?utf-8?b?UMOpcmV6?= , Jason Wang , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, Stefan Hajnoczi , Stefano Garzarella Subject: [PATCH RFC 12/13] vhost/vsock: switch to the buf API Message-ID: <20200602130543.578420-13-mst@redhat.com> References: <20200602130543.578420-1-mst@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200602130543.578420-1-mst@redhat.com> X-Mailer: git-send-email 2.24.1.751.gd10ce2899c X-Mutt-Fcc: =sent Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org A straight-forward conversion. Signed-off-by: Michael S. Tsirkin --- drivers/vhost/vsock.c | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c index fb4e944c4d0d..07d1fb340fb4 100644 --- a/drivers/vhost/vsock.c +++ b/drivers/vhost/vsock.c @@ -103,7 +103,8 @@ vhost_transport_do_send_pkt(struct vhost_vsock *vsock, unsigned out, in; size_t nbytes; size_t iov_len, payload_len; - int head; + struct vhost_buf buf; + int ret; spin_lock_bh(&vsock->send_pkt_list_lock); if (list_empty(&vsock->send_pkt_list)) { @@ -117,16 +118,17 @@ vhost_transport_do_send_pkt(struct vhost_vsock *vsock, list_del_init(&pkt->list); spin_unlock_bh(&vsock->send_pkt_list_lock); - head = vhost_get_vq_desc(vq, vq->iov, ARRAY_SIZE(vq->iov), - &out, &in, NULL, NULL); - if (head < 0) { + ret = vhost_get_avail_buf(vq, &buf, + vq->iov, ARRAY_SIZE(vq->iov), + &out, &in, NULL, NULL); + if (ret < 0) { spin_lock_bh(&vsock->send_pkt_list_lock); list_add(&pkt->list, &vsock->send_pkt_list); spin_unlock_bh(&vsock->send_pkt_list_lock); break; } - if (head == vq->num) { + if (!ret) { spin_lock_bh(&vsock->send_pkt_list_lock); list_add(&pkt->list, &vsock->send_pkt_list); spin_unlock_bh(&vsock->send_pkt_list_lock); @@ -186,7 +188,8 @@ vhost_transport_do_send_pkt(struct vhost_vsock *vsock, */ virtio_transport_deliver_tap_pkt(pkt); - vhost_add_used(vq, head, sizeof(pkt->hdr) + payload_len); + buf.in_len = sizeof(pkt->hdr) + payload_len; + vhost_put_used_buf(vq, &buf); added = true; pkt->off += payload_len; @@ -440,7 +443,8 @@ static void vhost_vsock_handle_tx_kick(struct vhost_work *work) struct vhost_vsock *vsock = container_of(vq->dev, struct vhost_vsock, dev); struct virtio_vsock_pkt *pkt; - int head, pkts = 0, total_len = 0; + int ret, pkts = 0, total_len = 0; + struct vhost_buf buf; unsigned int out, in; bool added = false; @@ -461,12 +465,13 @@ static void vhost_vsock_handle_tx_kick(struct vhost_work *work) goto no_more_replies; } - head = vhost_get_vq_desc(vq, vq->iov, ARRAY_SIZE(vq->iov), - &out, &in, NULL, NULL); - if (head < 0) + ret = vhost_get_avail_buf(vq, &buf, + vq->iov, ARRAY_SIZE(vq->iov), + &out, &in, NULL, NULL); + if (ret < 0) break; - if (head == vq->num) { + if (!ret) { if (unlikely(vhost_enable_notify(&vsock->dev, vq))) { vhost_disable_notify(&vsock->dev, vq); continue; @@ -494,7 +499,8 @@ static void vhost_vsock_handle_tx_kick(struct vhost_work *work) virtio_transport_free_pkt(pkt); len += sizeof(pkt->hdr); - vhost_add_used(vq, head, len); + buf.in_len = len; + vhost_put_used_buf(vq, &buf); total_len += len; added = true; } while(likely(!vhost_exceeds_weight(vq, ++pkts, total_len))); From patchwork Tue Jun 2 13:06:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Michael S. Tsirkin" X-Patchwork-Id: 218046 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85CE3C433E1 for ; Tue, 2 Jun 2020 13:06:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5960720674 for ; Tue, 2 Jun 2020 13:06:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IShkhbjn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728474AbgFBNGq (ORCPT ); Tue, 2 Jun 2020 09:06:46 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:55058 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728405AbgFBNGb (ORCPT ); Tue, 2 Jun 2020 09:06:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591103189; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=uCok80LFYJ4Lm/PMCty2sCIaVjds7Ty5xQFab3NSrrY=; b=IShkhbjnQJMa5w0IGp+5HfAqTh8qDrenQxW0dlKOFJCHX6IfAirU0VZLMjgX+zVgTou5F8 H4mzXbFvBIDGJSlJce4rOzWrGE1AWxuuKffviXK6j26XiIcnTVO4gHHbGTL013+7AprYEr f4Ebo2oTodfxB0wUCC9nMVs5cacQQ+o= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-46-tAMdM-byOL-qLqPyTw06jA-1; Tue, 02 Jun 2020 09:06:28 -0400 X-MC-Unique: tAMdM-byOL-qLqPyTw06jA-1 Received: by mail-wm1-f69.google.com with SMTP id h6so812671wmb.7 for ; Tue, 02 Jun 2020 06:06:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=uCok80LFYJ4Lm/PMCty2sCIaVjds7Ty5xQFab3NSrrY=; b=lRvCQyhdgoY0XS5z6WT5zMUJ0E35c//BJYJzfyy6GX2ZhOxqUxyBpaSP93x3XeYw0y pAvUOSd1PdlsFSIWcWFr0eyVqWh+uOmuybHxRsUK6x5XBYwbaNvU4S6ueprZ12zu79Oh mu9qD52jCriFAA2jwCkq45rJRPsul9oRFepCjKgALMFfvbAZ1r2yHk0EG83mX4hUJNSa vysrgAch+HxHBu1wu5TZqjtF/TRR5a0ypw9D0zoroRHbYmRFe9dJReoqmDXgXBavNLGx nTj5kHiKv8ncXecMk9Bn+Lrg4Y9ufTJbpavj0XWf6h5XVvzP97ev3hgys99w/JxqEY2v YG/g== X-Gm-Message-State: AOAM533oBy3ydI2e21I02EKUSMYClWH/CzyI19hE8o5yTpkVv7DHcFGT HZ7KAL4BIcgqd3NPU970mX4C795JO9njwZRUDfMtergnarP0HEnUXDffGBo//KEPTq9qjmsBwF/ 17ugI6N6+RZknNKkE X-Received: by 2002:adf:a350:: with SMTP id d16mr27202900wrb.237.1591103187036; Tue, 02 Jun 2020 06:06:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzSMYsr9uLo9HwO9qp8x9NdNINKCKw+Yq/4jaF7d1uMiQ2jKEDbPOfPn52lZ3rMs5ZJSOurmQ== X-Received: by 2002:adf:a350:: with SMTP id d16mr27202884wrb.237.1591103186812; Tue, 02 Jun 2020 06:06:26 -0700 (PDT) Received: from redhat.com (bzq-109-64-41-91.red.bezeqint.net. [109.64.41.91]) by smtp.gmail.com with ESMTPSA id n23sm3456907wmc.0.2020.06.02.06.06.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 06:06:26 -0700 (PDT) Date: Tue, 2 Jun 2020 09:06:25 -0400 From: "Michael S. Tsirkin" To: linux-kernel@vger.kernel.org Cc: Eugenio =?utf-8?b?UMOpcmV6?= , Jason Wang , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Subject: [PATCH RFC 13/13] vhost: drop head based APIs Message-ID: <20200602130543.578420-14-mst@redhat.com> References: <20200602130543.578420-1-mst@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200602130543.578420-1-mst@redhat.com> X-Mailer: git-send-email 2.24.1.751.gd10ce2899c X-Mutt-Fcc: =sent Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Everyone's using buf APIs, no need for head based ones anymore. Signed-off-by: Michael S. Tsirkin --- drivers/vhost/vhost.c | 36 ++++++++---------------------------- drivers/vhost/vhost.h | 12 ------------ 2 files changed, 8 insertions(+), 40 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index be822f0c9428..412923cc96df 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -2256,12 +2256,12 @@ static int fetch_buf(struct vhost_virtqueue *vq) return 1; } -/* Reverse the effect of vhost_get_vq_desc. Useful for error handling. */ +/* Revert the effect of fetch_buf. Useful for error handling. */ +static void vhost_discard_vq_desc(struct vhost_virtqueue *vq, int n) { vq->last_avail_idx -= n; } -EXPORT_SYMBOL_GPL(vhost_discard_vq_desc); /* This function returns a value > 0 if a descriptor was found, or 0 if none were found. * A negative code is returned on error. */ @@ -2421,8 +2421,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq, return 0; } -/* After we've used one of their buffers, we tell them about it. We'll then - * want to notify the guest, using eventfd. */ +static int vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads, unsigned count) { @@ -2456,10 +2455,8 @@ int vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads, } return r; } -EXPORT_SYMBOL_GPL(vhost_add_used_n); -/* After we've used one of their buffers, we tell them about it. We'll then - * want to notify the guest, using eventfd. */ +static int vhost_add_used(struct vhost_virtqueue *vq, unsigned int head, int len) { struct vring_used_elem heads = { @@ -2469,14 +2466,17 @@ int vhost_add_used(struct vhost_virtqueue *vq, unsigned int head, int len) return vhost_add_used_n(vq, &heads, 1); } -EXPORT_SYMBOL_GPL(vhost_add_used); +/* After we've used one of their buffers, we tell them about it. We'll then + * want to notify the guest, using vhost_signal. */ int vhost_put_used_buf(struct vhost_virtqueue *vq, struct vhost_buf *buf) { return vhost_add_used(vq, buf->id, buf->in_len); } EXPORT_SYMBOL_GPL(vhost_put_used_buf); +/* After we've used one of their buffers, we tell them about it. We'll then + * want to notify the guest, using vhost_signal. */ int vhost_put_used_n_bufs(struct vhost_virtqueue *vq, struct vhost_buf *bufs, unsigned count) { @@ -2537,26 +2537,6 @@ void vhost_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq) } EXPORT_SYMBOL_GPL(vhost_signal); -/* And here's the combo meal deal. Supersize me! */ -void vhost_add_used_and_signal(struct vhost_dev *dev, - struct vhost_virtqueue *vq, - unsigned int head, int len) -{ - vhost_add_used(vq, head, len); - vhost_signal(dev, vq); -} -EXPORT_SYMBOL_GPL(vhost_add_used_and_signal); - -/* multi-buffer version of vhost_add_used_and_signal */ -void vhost_add_used_and_signal_n(struct vhost_dev *dev, - struct vhost_virtqueue *vq, - struct vring_used_elem *heads, unsigned count) -{ - vhost_add_used_n(vq, heads, count); - vhost_signal(dev, vq); -} -EXPORT_SYMBOL_GPL(vhost_add_used_and_signal_n); - /* return true if we're sure that avaiable ring is empty */ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq) { diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 6c10e99ff334..4fcf59153fc7 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -195,11 +195,6 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg bool vhost_vq_access_ok(struct vhost_virtqueue *vq); bool vhost_log_access_ok(struct vhost_dev *); -int vhost_get_vq_desc(struct vhost_virtqueue *, - struct iovec iov[], unsigned int iov_count, - unsigned int *out_num, unsigned int *in_num, - struct vhost_log *log, unsigned int *log_num); -void vhost_discard_vq_desc(struct vhost_virtqueue *, int n); int vhost_get_avail_buf(struct vhost_virtqueue *, struct vhost_buf *buf, struct iovec iov[], unsigned int iov_count, unsigned int *out_num, unsigned int *in_num, @@ -207,13 +202,6 @@ int vhost_get_avail_buf(struct vhost_virtqueue *, struct vhost_buf *buf, void vhost_discard_avail_bufs(struct vhost_virtqueue *, struct vhost_buf *, unsigned count); int vhost_vq_init_access(struct vhost_virtqueue *); -int vhost_add_used(struct vhost_virtqueue *, unsigned int head, int len); -int vhost_add_used_n(struct vhost_virtqueue *, struct vring_used_elem *heads, - unsigned count); -void vhost_add_used_and_signal(struct vhost_dev *, struct vhost_virtqueue *, - unsigned int id, int len); -void vhost_add_used_and_signal_n(struct vhost_dev *, struct vhost_virtqueue *, - struct vring_used_elem *heads, unsigned count); int vhost_put_used_buf(struct vhost_virtqueue *, struct vhost_buf *buf); int vhost_put_used_n_bufs(struct vhost_virtqueue *, struct vhost_buf *bufs, unsigned count);