From patchwork Mon Sep 3 00:51:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kenneth Lee X-Patchwork-Id: 145738 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1927211ljw; Sun, 2 Sep 2018 17:52:32 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYGGkhvmY6BRG9tqMf60FUpSs6KfdtGlLKiQPiTq+l/RFx3H8OVDvw/5u5OvV3v/4E4aTlX X-Received: by 2002:a63:1a61:: with SMTP id a33-v6mr24160942pgm.9.1535935952759; Sun, 02 Sep 2018 17:52:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535935952; cv=none; d=google.com; s=arc-20160816; b=axeQc6IzLCc9OF8QPFzDBoarM3caO1PuTkjmWwp9UP09J7wc+4ct9lw8QRb/z0txQg +o6WrXxD8AM8YzgIAW1L1/v55LXARTaGUdcXbHZSMdDrfGN2V+n6IoDGXFqPjJhye1/a GdA3jLHwJWoYciy+o7CKioauWSeZKnijpviRCkqfaEKEg6g5O8toBgEMwptJBj7w3gp9 nDBWgQsL6qJ8FCS3gDCR2kiGfyjvab7WJtS0b3f9d+71Kgm5Yt86lUKSJ2RlieU1pL5+ qQ2etpMFg0ylKm0wPpzTBJqqgnrfQ4eVYTSWX3yRfSVwEujRCkoBi3gJS+SRczU+WY99 DSLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=sa17iEnf/25lNE7MuUyFrCggJ8HGDk9yuNb+pUO+7vk=; b=XXBQw91BUolnTQ4NeYBNzgGXblx831tnKG9FHjX1F+WJSXBFqHJtjusX2w0zDAcHHf 0xnrtP/CsQvnYnbFJ/gDtl38fq4v1FUbFQ8XZnM9lrRMFls0YTn5WhflHVhW/ELy3pP1 cNzSQ2Ad1Iyl0GWBMbS3pw4GcqQgXrXvuRvNdJbrMkRM1F/YTbOuZirgVLFvDn2Q4RoB a0AAsgFjhrgsmeEJ2P9bZ6T013N5hSTNLT2Slq4WgSeGcMhTZFJK4VhqS/mgkhxFmEn9 lW+fXaKihMREcnHJqqmkGrqT1qaHmiXUBkXR0X8LQ3Ol+ZDkIGTfdLTNsxH52AFWWPvH 5OsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=G2ojUcCl; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w13-v6si16049908pll.449.2018.09.02.17.52.32; Sun, 02 Sep 2018 17:52:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=G2ojUcCl; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726022AbeICFKC (ORCPT + 32 others); Mon, 3 Sep 2018 01:10:02 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:41330 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725810AbeICFKC (ORCPT ); Mon, 3 Sep 2018 01:10:02 -0400 Received: by mail-pl1-f195.google.com with SMTP id b12-v6so7823294plr.8; Sun, 02 Sep 2018 17:52:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sa17iEnf/25lNE7MuUyFrCggJ8HGDk9yuNb+pUO+7vk=; b=G2ojUcCl9aw/F1TuaQOD1xrg0vx7RUkVuTkc8vfoUxHVPSaR8bfuEjeavLWcfIQwI/ O171Qwo32R6YBkogZybmwVQt6N1SQ7eHjocaTgRUnFzNzBFs/SCdrFQLOHEmBJYGtY+X wvfTZqAa6qi02txnUbnsx31qkZb+NzgFtKT9PtBJcPezs4+6Gx97VgFQrsipGPE6hpUZ kSbXG2LAto53pnx20LbnBGAiZL2vn4tcGu0WT+BGeF0IY5qSUDHAfVP9Y428VCrp5axc 0DtZubl6KVnkPZt2U5gakqhJp7pxEJKUuzGFgusqp1p3G+46coH09MiHjZ6aI9arh+R0 Vw/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sa17iEnf/25lNE7MuUyFrCggJ8HGDk9yuNb+pUO+7vk=; b=cnzFu0zdN1zTgmaeSUdz0D3LjfBzN+gUyCiA1fWQwDviQHeA6o44IOXEigV0vrlr0v l3sK50pWKKK6UO5KXMY8r+1o9GJ+3D/RDQN+Rbj5cxHdtvGGPMxo1eU1JYiro4E++VC1 YR0Qwa8pXBWcqyXtxooTiMz3E1Aqj53IjcwXee+sRYHl7K3VFMS06TdRrIJskhvOs580 G4gmN+GYfaDqRKzxlqFM+Z0upf+Piss4B9LZQGyPi5VNKsxymNIihdGEtaVBH9UOpI5g p/cjhESioGJC5gHHzfvCW28L+OkXBhVQiY+whgBZM/FswykNYmoBfdU5pfK4b6kTKMM6 dIMA== X-Gm-Message-State: APzg51CvAtZHqIJVnEN9RddGQU9d+48ckn2gH2aWQvGfbDa6wLLsRMmo RvHg+PwCfIVJyJFQ4cZ24Y0= X-Received: by 2002:a17:902:261:: with SMTP id 88-v6mr25736392plc.331.1535935940956; Sun, 02 Sep 2018 17:52:20 -0700 (PDT) Received: from localhost.localdomain ([203.160.89.93]) by smtp.gmail.com with ESMTPSA id w81-v6sm31487064pfk.92.2018.09.02.17.52.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 17:52:20 -0700 (PDT) From: Kenneth Lee To: Jonathan Corbet , Herbert Xu , "David S . Miller" , Joerg Roedel , Alex Williamson , Kenneth Lee , Hao Fang , Zhou Wang , Zaibo Xu , Philippe Ombredanne , Greg Kroah-Hartman , Thomas Gleixner , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, linux-accelerators@lists.ozlabs.org, Lu Baolu , Sanjay Kumar Cc: linuxarm@huawei.com Subject: [PATCH 1/7] vfio/sdmdev: Add documents for WarpDrive framework Date: Mon, 3 Sep 2018 08:51:58 +0800 Message-Id: <20180903005204.26041-2-nek.in.cn@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903005204.26041-1-nek.in.cn@gmail.com> References: <20180903005204.26041-1-nek.in.cn@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Kenneth Lee WarpDrive is a common user space accelerator framework. Its main component in Kernel is called sdmdev, Share Domain Mediated Device. It exposes the hardware capabilities to the user space via vfio-mdev. So processes in user land can obtain a "queue" by open the device and direct access the hardware MMIO space or do DMA operation via VFIO interface. WarpDrive is intended to be used with Jean Philippe Brucker's SVA patchset to support multi-process. But This is not a must. Without the SVA patches, WarpDrive can still work for one process for every hardware device. This patch add detail documents for the framework. Signed-off-by: Kenneth Lee --- Documentation/00-INDEX | 2 + Documentation/warpdrive/warpdrive.rst | 100 ++++ Documentation/warpdrive/wd-arch.svg | 728 ++++++++++++++++++++++++++ 3 files changed, 830 insertions(+) create mode 100644 Documentation/warpdrive/warpdrive.rst create mode 100644 Documentation/warpdrive/wd-arch.svg -- 2.17.1 diff --git a/Documentation/00-INDEX b/Documentation/00-INDEX index 2754fe83f0d4..9959affab599 100644 --- a/Documentation/00-INDEX +++ b/Documentation/00-INDEX @@ -410,6 +410,8 @@ vm/ - directory with info on the Linux vm code. w1/ - directory with documents regarding the 1-wire (w1) subsystem. +warpdrive/ + - directory with documents about WarpDrive accelerator framework. watchdog/ - how to auto-reboot Linux if it has "fallen and can't get up". ;-) wimax/ diff --git a/Documentation/warpdrive/warpdrive.rst b/Documentation/warpdrive/warpdrive.rst new file mode 100644 index 000000000000..6d2a5d1e08c4 --- /dev/null +++ b/Documentation/warpdrive/warpdrive.rst @@ -0,0 +1,100 @@ +Introduction of WarpDrive +========================= + +*WarpDrive* is a general accelerator framework for user space. It intends to +provide interface for the user process to send request to hardware +accelerator without heavy user-kernel interaction cost. + +The *WarpDrive* user library is supposed to provide a pipe-based API, such as: + :: + int wd_request_queue(struct wd_queue *q); + void wd_release_queue(struct wd_queue *q); + + int wd_send(struct wd_queue *q, void *req); + int wd_recv(struct wd_queue *q, void **req); + int wd_recv_sync(struct wd_queue *q, void **req); + int wd_flush(struct wd_queue *q); + +*wd_request_queue* creates the pipe connection, *queue*, between the +application and the hardware. The application sends request and pulls the +answer back by asynchronized wd_send/wd_recv, which directly interact with the +hardware (by MMIO or share memory) without syscall. + +*WarpDrive* maintains a unified application address space among all involved +accelerators. With the following APIs: :: + + int wd_mem_share(struct wd_queue *q, const void *addr, + size_t size, int flags); + void wd_mem_unshare(struct wd_queue *q, const void *addr, size_t size); + +The referred process space shared by these APIs can be directly referred by the +hardware. The process can also dedicate its whole process space with flags, +*WD_SHARE_ALL* (not in this patch yet). + +The name *WarpDrive* is simply a cool and general name meaning the framework +makes the application faster. As it will be explained in this text later, the +facility in kernel is called *SDMDEV*, namely "Share Domain Mediated Device". + + +How does it work +================ + +*WarpDrive* is built upon *VFIO-MDEV*. The queue is wrapped as *mdev* in VFIO. +So memory sharing can be done via standard VFIO standard DMA interface. + +The architecture is illustrated as follow figure: + +.. image:: wd-arch.svg + :alt: WarpDrive Architecture + +Accelerator driver shares its capability via *SDMDEV* API: :: + + vfio_sdmdev_register(struct vfio_sdmdev *sdmdev); + vfio_sdmdev_unregister(struct vfio_sdmdev *sdmdev); + vfio_sdmdev_wake_up(struct spimdev_queue *q); + +*vfio_sdmdev_register* is a helper function to register the hardware to the +*VFIO_MDEV* framework. The queue creation is done by *mdev* creation interface. + +*WarpDrive* User library mmap the mdev to access its mmio space and shared +memory. Request can be sent to, or receive from, hardware in this mmap-ed +space until the queue is full or empty. + +The user library can wait on the queue by ioctl(VFIO_SDMDEV_CMD_WAIT) the mdev +if the queue is full or empty. If the queue status is changed, the hardware +driver use *vfio_sdmdev_wake_up* to wake up the waiting process. + + +Multiple processes support +========================== + +In the latest mainline kernel (4.18) when this document is written, +multi-process is not supported in VFIO yet. + +Jean Philippe Brucker has a patchset to enable it[1]_. We have tested it +with our hardware (which is known as *D06*). It works well. *WarpDrive* rely +on them to support multiple processes. If it is not enabled, *WarpDrive* can +still work, but it support only one mdev for a process, which will share the +same io map table with kernel. (But it is not going to be a security problem, +since the user application cannot access the kernel address space) + +When multiprocess is support, mdev can be created based on how many +hardware resource (queue) is available. Because the VFIO framework accepts only +one open from one mdev iommu_group. Mdev become the smallest unit for process +to use queue. And the mdev will not be released if the user process exist. So +it will need a resource agent to manage the mdev allocation for the user +process. This is not in this document's range. + + +Legacy Mode Support +=================== +For the hardware on which IOMMU is not support, WarpDrive can run on *NOIOMMU* +mode. That require some update to the mdev driver, which is not included in +this version yet. + + +References +========== +.. [1] https://patchwork.kernel.org/patch/10394851/ + +.. vim: tw=78 diff --git a/Documentation/warpdrive/wd-arch.svg b/Documentation/warpdrive/wd-arch.svg new file mode 100644 index 000000000000..2b0c467ee399 --- /dev/null +++ b/Documentation/warpdrive/wd-arch.svg @@ -0,0 +1,728 @@ + + + + + + + + + + + + + + + + + + + + + + + + generation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + WarpDrive + + + user_driver + + + + sdmdev + + + + + Device Driver + * + 1 + + <<vfio>>resource management + 1 + * + + + other standard framework(crypto/nic/others) + + <<lkm>> + register as mdev with"share domain" attribute + register to other subsystem + <<user_lib>> + <<vfio>>Hardware Accessing + wd user api + + + + Device(Hardware) + + Share Domain mdev + + From patchwork Mon Sep 3 00:51:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kenneth Lee X-Patchwork-Id: 145737 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1927219ljw; Sun, 2 Sep 2018 17:52:33 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaV54mTlAqM+2Rsm+lk4XTWle0W6rVx4P39TvUvSfJkTeuTN6t3dB0I25mun4iPeGnBKMHk X-Received: by 2002:a62:4a88:: with SMTP id c8-v6mr2604651pfj.232.1535935953135; Sun, 02 Sep 2018 17:52:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535935953; cv=none; d=google.com; s=arc-20160816; b=OsdVlSy41Yjq5wWnPzNaI+Hrn/C1crFhqr6J96Ime2PNtJUckPdD5+hCJ3Aj5zZXQd Kxe92xcMbv9PAIOay/waTctjI9rBr1KNAbKN/svRD5jXhsBtx+u0iah9fXMgAAAaYLUU GYO1hvnD/N1cLfMVYjfuE1cXcMDMgcswYq3/81N4y6seTdRr47xWDLiue4BGHLAj9i7a gILis+uHTFvdEZw9YwUQKJn/mxYd+t1V1cCuFMNHAL7bm04rReZKXFsOuY6TaMcEHFTt dCvPHF28M9rrexIIIy3dwK5AhmHc35GluDPZIFY5PyQaWv38dlPWsOHANOCqpBqaJQ6B H5cQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=PqNEZCF3NFy3Jzxsnv6juVWEGf6EoQqZyW/gnVxyV+M=; b=tuj1sdiT0I60l+/YkFjoU+1vAerBlqf8QDrnq8T7g7svouoC/vnmbTYshg4brE58MW ooaCG0DZy4nsV1dNyJDO5yUmNcNy2V21r7rRkmM39WYLD6GKYt4+nEbB/NQY4y0fkak+ gCIh7e6dC8gcMrlaMuU4U15sr+lJhPdcJbjOWWDaP6etTxgRdRaRTrYZRpl0rwvp3d8n h9Fnq4OgA0t++/UcW5CR4BzDgLM3euzEsxfjhZfXpsnHAJJ6qquysV9GVd0bYiMawMsI dNN3zc6HklDW9ts1ZHb82mpfMs2Yd2+gDE/oFx1opYs0lLr8JqpSfDHypIA80eiO7CEg PJDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=nZQ6Fj3Y; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w13-v6si16049908pll.449.2018.09.02.17.52.32; Sun, 02 Sep 2018 17:52:33 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=nZQ6Fj3Y; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726146AbeICFKH (ORCPT + 32 others); Mon, 3 Sep 2018 01:10:07 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:46124 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725762AbeICFKG (ORCPT ); Mon, 3 Sep 2018 01:10:06 -0400 Received: by mail-pl1-f195.google.com with SMTP id t19-v6so110251ply.13; Sun, 02 Sep 2018 17:52:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PqNEZCF3NFy3Jzxsnv6juVWEGf6EoQqZyW/gnVxyV+M=; b=nZQ6Fj3YkpjhrG+K+fDsDVinPrzX48NDfx7hCJNHveoRnUUX2Dl97U9nWbe5pFEO2Y DDjK6MsdoN5exy2lz1qchFT3hiS8HzLO4WycB+I6eNNARhX2aLXJ5x+UtDSk44C4jR6e ja9xRrTWgTfovwLvvd9XH/4+o6dMLteoN3JZPLFyfnZgwR2QvWYZlCPMnukV/ODO0+5V 4vKNCGycErrIwPAs7Ijzpyqa08m17DYKh6o+gHNRo5WRh6JHeriDHUfOoRReeiNahuDh vfs3ZTPLPUcF4mGYCCG8Ni6kGmTXrHtNW6BKyS/wkHvNa7b2aVmwxp53H/crlo2rb/oo jJVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PqNEZCF3NFy3Jzxsnv6juVWEGf6EoQqZyW/gnVxyV+M=; b=JYftu6s2e065P5f4URi6ReGNKMvC7UlGsHzIhh85XWIRAKHwVqlP9mJxzfR24+SEW6 Z9NqKVWYQP+ZyBfxrisqbpY2Rqx7z5n3kL9syIIs2mDwIHhfpTENYUW5EP4nM35c70Ge aNahxJk7fMLjLX9y+RfHXtmykrUDROFZHaHiQEcljERiPh9nUf7p1pMWiVKTUDsi3dur YLuOn6UpYpLlaJ5k1UHFIyhHjJwPwODuXFgP7poGFpA/mqEmk+zX/PHoKtUekmWWsd0R O+4d+gwaqsuCaksSEiwuBff4fgzS6vY370OHD7Atx1MiJtZc8DzfesYx/LdW9tgwF2mm c+QA== X-Gm-Message-State: APzg51A6QK2WxwH0fvnj14vDwmFX7Ci8HbVgKypCdWxuxKxIQKp6ThQ6 INJY2xVlbvFwHvY8+BmHTfk= X-Received: by 2002:a17:902:8a92:: with SMTP id p18-v6mr25383105plo.148.1535935946150; Sun, 02 Sep 2018 17:52:26 -0700 (PDT) Received: from localhost.localdomain ([203.160.89.93]) by smtp.gmail.com with ESMTPSA id w81-v6sm31487064pfk.92.2018.09.02.17.52.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 17:52:25 -0700 (PDT) From: Kenneth Lee To: Jonathan Corbet , Herbert Xu , "David S . Miller" , Joerg Roedel , Alex Williamson , Kenneth Lee , Hao Fang , Zhou Wang , Zaibo Xu , Philippe Ombredanne , Greg Kroah-Hartman , Thomas Gleixner , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, linux-accelerators@lists.ozlabs.org, Lu Baolu , Sanjay Kumar Cc: linuxarm@huawei.com Subject: [PATCH 2/7] iommu: Add share domain interface in iommu for sdmdev Date: Mon, 3 Sep 2018 08:51:59 +0800 Message-Id: <20180903005204.26041-3-nek.in.cn@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903005204.26041-1-nek.in.cn@gmail.com> References: <20180903005204.26041-1-nek.in.cn@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Kenneth Lee This patch add sharing interface for a iommu_group. The new interface: iommu_group_share_domain() iommu_group_unshare_domain() can be used by some virtual iommu_group (such as iommu_group of sdmdev) to share their parent's iommu_group. When the domain of a group is shared, it cannot be changed before being unshared. By this way, all domain users can assume the shared IOMMU have the same configuration. In the future, notification can be added if update is required. Signed-off-by: Kenneth Lee --- drivers/iommu/iommu.c | 29 ++++++++++++++++++++++++++++- include/linux/iommu.h | 15 +++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) -- 2.17.1 diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 8c15c5980299..8e567e1037dd 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -58,6 +58,9 @@ struct iommu_group { int id; struct iommu_domain *default_domain; struct iommu_domain *domain; + atomic_t domain_shared_ref; /* Number of user of current domain. + * The domain cannot be modified if ref > 0 + */ }; struct group_device { @@ -385,6 +388,7 @@ struct iommu_group *iommu_group_alloc(void) return ERR_PTR(ret); } group->id = ret; + atomic_set(&group->domain_shared_ref, 0); ret = kobject_init_and_add(&group->kobj, &iommu_group_ktype, NULL, "%d", group->id); @@ -518,6 +522,26 @@ int iommu_group_set_name(struct iommu_group *group, const char *name) } EXPORT_SYMBOL_GPL(iommu_group_set_name); +struct iommu_domain *iommu_group_share_domain(struct iommu_group *group) +{ + /* the domain can be shared only when the default domain is used */ + /* todo: more shareable check */ + if (group->domain != group->default_domain) + return ERR_PTR(-EINVAL); + + atomic_inc(&group->domain_shared_ref); + return group->domain; +} +EXPORT_SYMBOL_GPL(iommu_group_share_domain); + +struct iommu_domain *iommu_group_unshare_domain(struct iommu_group *group) +{ + atomic_dec(&group->domain_shared_ref); + WARN_ON(atomic_read(&group->domain_shared_ref) < 0); + return group->domain; +} +EXPORT_SYMBOL_GPL(iommu_group_unshare_domain); + static int iommu_group_create_direct_mappings(struct iommu_group *group, struct device *dev) { @@ -1437,7 +1461,8 @@ static int __iommu_attach_group(struct iommu_domain *domain, { int ret; - if (group->default_domain && group->domain != group->default_domain) + if ((group->default_domain && group->domain != group->default_domain) || + atomic_read(&group->domain_shared_ref) > 0) return -EBUSY; ret = __iommu_group_for_each_dev(group, domain, @@ -1474,6 +1499,8 @@ static void __iommu_detach_group(struct iommu_domain *domain, { int ret; + WARN_ON(atomic_read(&group->domain_shared_ref) > 0); + if (!group->default_domain) { __iommu_group_for_each_dev(group, domain, iommu_group_do_detach_device); diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 87994c265bf5..013ac400b643 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -344,6 +344,9 @@ extern int iommu_domain_get_attr(struct iommu_domain *domain, enum iommu_attr, void *data); extern int iommu_domain_set_attr(struct iommu_domain *domain, enum iommu_attr, void *data); +extern struct iommu_domain *iommu_group_share_domain(struct iommu_group *group); +extern struct iommu_domain *iommu_group_unshare_domain( + struct iommu_group *group); /* Window handling function prototypes */ extern int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, @@ -616,6 +619,18 @@ static inline int iommu_domain_set_attr(struct iommu_domain *domain, return -EINVAL; } +static inline struct iommu_domain *iommu_group_share_domain( + struct iommu_group *group) +{ + return NULL; +} + +static inline struct iommu_domain *iommu_group_unshare_domain( + struct iommu_group *group) +{ + return NULL; +} + static inline int iommu_device_register(struct iommu_device *iommu) { return -ENODEV; From patchwork Mon Sep 3 00:52:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kenneth Lee X-Patchwork-Id: 145739 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1927311ljw; Sun, 2 Sep 2018 17:52:40 -0700 (PDT) X-Google-Smtp-Source: ANB0VdarooX5XNiJvn5XKZHAHam0UbAE10tn42uAJYDJX+o0P1BFv5gm+Xx8DU6Bs5qhuutjCG+p X-Received: by 2002:a62:6547:: with SMTP id z68-v6mr26786514pfb.20.1535935960492; Sun, 02 Sep 2018 17:52:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535935960; cv=none; d=google.com; s=arc-20160816; b=lJF2gMQiNsZVZdBuj1238sK5nJMGbZKfLq6wUGJ9yZuAeRzQ4Q1tafZ1yRbQ225S/s dgwfppXSqOxqjNrSnS0/+l58G0FJ3hl+RtYoRVQQWKtT/02FGZv4ZVdM9yeAi+Cf1cfa E7PayN7YPm6zxfCcKINK1ociekWTJpCmqSFOf61vZVLK+jK4yeMuX7T7+ekz4tZa3Ji2 lGOR2J1HaXc89j6kHBgCS6iSUmtoh1VznNqbV/6t3EolxawRMlqdVcFBQyEhZiZcm9Nd /A/HbVmbR3Q0xbcJolpwLgKNM+UeD2SEeNEkJ3FVEAHD7YFa7pIr4AD1ZdFzb5ee2MnN 4Eeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=WC1/Ye6nyqnNK5M/Q9663vlqezVLsdl9y22w3LpCMyM=; b=oNHhMC0+Ro/9Imyw8L63sgTcHPGKvkMHU4brioLhUTW2PQZ8qQ15rl3Yrfi/TTZNRe Pr/QKfSyR0csZR6SvOY1ybctc/l6dhJV+MATRNlldB2qKr0Tgze5rLAVm2VfhMf5kEmJ oRMtaiHoBzoI3dWJRbOxi8mBsiBLzKMAj41q7H/jNtsQwU09o1QsAXGHyFbVvx5aJt9O gdNLjf2Al36LY2KuSUuyy4WB6TgINWfj4qpwGkwc7ZqmGDZL7UXdxX/O2kwfkUrlKEH4 gMpOxsZqb+s9TEhYVrgYsl8mS8AMzUqKN4rq/i4xCAW6qgfIwA75DQk1j9lB6tLsth13 9YbA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=VzFsIRv3; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s1-v6si16115936pgj.405.2018.09.02.17.52.40; Sun, 02 Sep 2018 17:52:40 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=VzFsIRv3; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726328AbeICFKN (ORCPT + 32 others); Mon, 3 Sep 2018 01:10:13 -0400 Received: from mail-pl1-f196.google.com ([209.85.214.196]:45360 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726160AbeICFKM (ORCPT ); Mon, 3 Sep 2018 01:10:12 -0400 Received: by mail-pl1-f196.google.com with SMTP id j8-v6so7806435pll.12; Sun, 02 Sep 2018 17:52:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WC1/Ye6nyqnNK5M/Q9663vlqezVLsdl9y22w3LpCMyM=; b=VzFsIRv3xv9FXt/2O4v7wNInmgAls/clR1HscE2dz1F07b0LQurcURL0kwPU/CStB9 PCpdOm1o0gqxpihcbZQTYTjtpryglIhA0X+ICkQ3z8aIEkAJyr2BUOaezKgE6Fm1Xdhw xHB5xsPXbmxQ09S/cAlkhqg/rUZtK/ymyRwYHfXfnFoq8VQD28IU4MjlJI0UYc25OYJG fakH2Hv44UpTaZtO9caUQbWnf5DQa/Hn+QeAP6KxM7Qq79NrbV2unQVEmrtrNRDrsxHl wzSs7Ts1pso41ldIriVJktoibcexQ+KaP362Q/ropPybeX5QBVs/051BJVBWlPAQ9RlL RCAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=WC1/Ye6nyqnNK5M/Q9663vlqezVLsdl9y22w3LpCMyM=; b=kKkc1pEXvKvKC7XLYpTxPRn4fjimultPrhfMO/ixG+9Tlmze7Ddy+t7ncm5Vg2o8L+ sFqjBNZg+cU9EFx3Fp2yJePTma8M3+Xb2FiGcI0z76lQStqXCnY2I1MP6C5Xsl/d7PqK DgC8xyb5pECROjryCO/ky1cAYJ4Sb+qj4poa1qks5n1YI6KyZrbofN9sXajF5anQa+Kp fKaXfhFFPtOy1u5noqeswFOq6XbmyIIrtjq0T1xg0OedTvlvs9bUy4zp4s5H0o52l9Ws QTGdEqZOrOxAgpEDYh8OMAsQY5edC5XAZRYre+8+SddXUXXqQEKmBNpCysIq1WJ4MBXJ 67VA== X-Gm-Message-State: APzg51A70xuQu+UbmIQcDuvyVEjgH+LwiLjZv1VoXyhTqs/Kc1pBXV4C skJ6QkUEgrs2NAMFnoB9iUk= X-Received: by 2002:a17:902:b089:: with SMTP id p9-v6mr25092603plr.254.1535935951678; Sun, 02 Sep 2018 17:52:31 -0700 (PDT) Received: from localhost.localdomain ([203.160.89.93]) by smtp.gmail.com with ESMTPSA id w81-v6sm31487064pfk.92.2018.09.02.17.52.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 17:52:31 -0700 (PDT) From: Kenneth Lee To: Jonathan Corbet , Herbert Xu , "David S . Miller" , Joerg Roedel , Alex Williamson , Kenneth Lee , Hao Fang , Zhou Wang , Zaibo Xu , Philippe Ombredanne , Greg Kroah-Hartman , Thomas Gleixner , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, linux-accelerators@lists.ozlabs.org, Lu Baolu , Sanjay Kumar Cc: linuxarm@huawei.com Subject: [PATCH 3/7] vfio: add sdmdev support Date: Mon, 3 Sep 2018 08:52:00 +0800 Message-Id: <20180903005204.26041-4-nek.in.cn@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903005204.26041-1-nek.in.cn@gmail.com> References: <20180903005204.26041-1-nek.in.cn@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Kenneth Lee SDMDEV is "Share Domain Mdev". It is a vfio-mdev. But differ from the general vfio-mdev, it shares its parent's IOMMU. If Multi-PASID support is enabled in the IOMMU (not yet in the current kernel HEAD), multiple process can share the IOMMU by different PASID. If it is not support, only one process can share the IOMMU with the kernel driver. Currently only the vfio type-1 driver is updated to make it to be aware of. Signed-off-by: Kenneth Lee Signed-off-by: Zaibo Xu Signed-off-by: Zhou Wang --- drivers/vfio/Kconfig | 1 + drivers/vfio/Makefile | 1 + drivers/vfio/sdmdev/Kconfig | 10 + drivers/vfio/sdmdev/Makefile | 3 + drivers/vfio/sdmdev/vfio_sdmdev.c | 363 ++++++++++++++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 151 ++++++++++++- include/linux/vfio_sdmdev.h | 96 ++++++++ include/uapi/linux/vfio_sdmdev.h | 29 +++ 8 files changed, 648 insertions(+), 6 deletions(-) create mode 100644 drivers/vfio/sdmdev/Kconfig create mode 100644 drivers/vfio/sdmdev/Makefile create mode 100644 drivers/vfio/sdmdev/vfio_sdmdev.c create mode 100644 include/linux/vfio_sdmdev.h create mode 100644 include/uapi/linux/vfio_sdmdev.h -- 2.17.1 diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig index c84333eb5eb5..5af7d1db505e 100644 --- a/drivers/vfio/Kconfig +++ b/drivers/vfio/Kconfig @@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU source "drivers/vfio/pci/Kconfig" source "drivers/vfio/platform/Kconfig" source "drivers/vfio/mdev/Kconfig" +source "drivers/vfio/sdmdev/Kconfig" source "virt/lib/Kconfig" diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile index de67c4725cce..678592360a7a 100644 --- a/drivers/vfio/Makefile +++ b/drivers/vfio/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o obj-$(CONFIG_VFIO_PCI) += pci/ obj-$(CONFIG_VFIO_PLATFORM) += platform/ obj-$(CONFIG_VFIO_MDEV) += mdev/ +obj-$(CONFIG_VFIO_SDMDEV) += sdmdev/ diff --git a/drivers/vfio/sdmdev/Kconfig b/drivers/vfio/sdmdev/Kconfig new file mode 100644 index 000000000000..51474272870d --- /dev/null +++ b/drivers/vfio/sdmdev/Kconfig @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0 +config VFIO_SDMDEV + tristate "Support for Share Domain MDEV" + depends on VFIO_MDEV_DEVICE + help + Support for VFIO Share Domain MDEV, which enables the kernel to + support light weight hardware accelerator framework, WarpDrive. + + To compile this as a module, choose M here: the module will be called + sdmdev. diff --git a/drivers/vfio/sdmdev/Makefile b/drivers/vfio/sdmdev/Makefile new file mode 100644 index 000000000000..ccaaa03f3184 --- /dev/null +++ b/drivers/vfio/sdmdev/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 +sdmdev-y := sdmdev.o +obj-$(CONFIG_VFIO_SDMDEV) += vfio_sdmdev.o diff --git a/drivers/vfio/sdmdev/vfio_sdmdev.c b/drivers/vfio/sdmdev/vfio_sdmdev.c new file mode 100644 index 000000000000..c6eb5d4bdab0 --- /dev/null +++ b/drivers/vfio/sdmdev/vfio_sdmdev.c @@ -0,0 +1,363 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include + +static struct class *sdmdev_class; + +static int vfio_sdmdev_dev_exist(struct device *dev, void *data) +{ + return !strcmp(dev_name(dev), dev_name((struct device *)data)); +} + +#ifdef CONFIG_IOMMU_SVA +static bool vfio_sdmdev_is_valid_pasid(int pasid) +{ + struct mm_struct *mm; + + mm = iommu_sva_find(pasid); + if (mm) { + mmput(mm); + return mm == current->mm; + } + + return false; +} +#endif + +/* Check if the device is a mediated device belongs to vfio_sdmdev */ +int vfio_sdmdev_is_sdmdev(struct device *dev) +{ + struct mdev_device *mdev; + struct device *pdev; + + mdev = mdev_from_dev(dev); + if (!mdev) + return 0; + + pdev = mdev_parent_dev(mdev); + if (!pdev) + return 0; + + return class_for_each_device(sdmdev_class, NULL, pdev, + vfio_sdmdev_dev_exist); +} +EXPORT_SYMBOL_GPL(vfio_sdmdev_is_sdmdev); + +struct vfio_sdmdev *vfio_sdmdev_pdev_sdmdev(struct device *dev) +{ + struct device *class_dev; + + if (!dev) + return ERR_PTR(-EINVAL); + + class_dev = class_find_device(sdmdev_class, NULL, dev, + (int(*)(struct device *, const void *))vfio_sdmdev_dev_exist); + if (!class_dev) + return ERR_PTR(-ENODEV); + + return container_of(class_dev, struct vfio_sdmdev, cls_dev); +} +EXPORT_SYMBOL_GPL(vfio_sdmdev_pdev_sdmdev); + +struct vfio_sdmdev *mdev_sdmdev(struct mdev_device *mdev) +{ + struct device *pdev = mdev_parent_dev(mdev); + + return vfio_sdmdev_pdev_sdmdev(pdev); +} +EXPORT_SYMBOL_GPL(mdev_sdmdev); + +static ssize_t iommu_type_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct vfio_sdmdev *sdmdev = vfio_sdmdev_pdev_sdmdev(dev); + + if (!sdmdev) + return -ENODEV; + + return sprintf(buf, "%d\n", sdmdev->iommu_type); +} + +static DEVICE_ATTR_RO(iommu_type); + +static ssize_t dma_flag_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct vfio_sdmdev *sdmdev = vfio_sdmdev_pdev_sdmdev(dev); + + if (!sdmdev) + return -ENODEV; + + return sprintf(buf, "%d\n", sdmdev->dma_flag); +} + +static DEVICE_ATTR_RO(dma_flag); + +/* mdev->dev_attr_groups */ +static struct attribute *vfio_sdmdev_attrs[] = { + &dev_attr_iommu_type.attr, + &dev_attr_dma_flag.attr, + NULL, +}; +static const struct attribute_group vfio_sdmdev_group = { + .name = VFIO_SDMDEV_PDEV_ATTRS_GRP_NAME, + .attrs = vfio_sdmdev_attrs, +}; +const struct attribute_group *vfio_sdmdev_groups[] = { + &vfio_sdmdev_group, + NULL, +}; + +/* default attributes for mdev->supported_type_groups, used by registerer*/ +#define MDEV_TYPE_ATTR_RO_EXPORT(name) \ + MDEV_TYPE_ATTR_RO(name); \ + EXPORT_SYMBOL_GPL(mdev_type_attr_##name); + +#define DEF_SIMPLE_SDMDEV_ATTR(_name, sdmdev_member, format) \ +static ssize_t _name##_show(struct kobject *kobj, struct device *dev, \ + char *buf) \ +{ \ + struct vfio_sdmdev *sdmdev = vfio_sdmdev_pdev_sdmdev(dev); \ + if (!sdmdev) \ + return -ENODEV; \ + return sprintf(buf, format, sdmdev->sdmdev_member); \ +} \ +MDEV_TYPE_ATTR_RO_EXPORT(_name) + +DEF_SIMPLE_SDMDEV_ATTR(flags, flags, "%d"); +DEF_SIMPLE_SDMDEV_ATTR(name, name, "%s"); /* this should be algorithm name, */ + /* but you would not care if you have only one algorithm */ +DEF_SIMPLE_SDMDEV_ATTR(device_api, api_ver, "%s"); + +static ssize_t +available_instances_show(struct kobject *kobj, struct device *dev, char *buf) +{ + struct vfio_sdmdev *sdmdev = vfio_sdmdev_pdev_sdmdev(dev); + int nr_inst = 0; + + nr_inst = sdmdev->ops->get_available_instances ? + sdmdev->ops->get_available_instances(sdmdev) : 0; + return sprintf(buf, "%d", nr_inst); +} +MDEV_TYPE_ATTR_RO_EXPORT(available_instances); + +static int vfio_sdmdev_mdev_create(struct kobject *kobj, + struct mdev_device *mdev) +{ + struct device *pdev = mdev_parent_dev(mdev); + struct vfio_sdmdev_queue *q; + struct vfio_sdmdev *sdmdev = mdev_sdmdev(mdev); + int ret; + + if (!sdmdev->ops->get_queue) + return -ENODEV; + + ret = sdmdev->ops->get_queue(sdmdev, &q); + if (ret) + return ret; + + q->sdmdev = sdmdev; + q->mdev = mdev; + init_waitqueue_head(&q->wait); + + mdev_set_drvdata(mdev, q); + get_device(pdev); + + return 0; +} + +static int vfio_sdmdev_mdev_remove(struct mdev_device *mdev) +{ + struct vfio_sdmdev_queue *q = + (struct vfio_sdmdev_queue *)mdev_get_drvdata(mdev); + struct vfio_sdmdev *sdmdev = q->sdmdev; + struct device *pdev = mdev_parent_dev(mdev); + + put_device(pdev); + + if (sdmdev->ops->put_queue); + sdmdev->ops->put_queue(q); + + return 0; +} + +/* Wake up the process who is waiting this queue */ +void vfio_sdmdev_wake_up(struct vfio_sdmdev_queue *q) +{ + wake_up_all(&q->wait); +} +EXPORT_SYMBOL_GPL(vfio_sdmdev_wake_up); + +static int vfio_sdmdev_mdev_mmap(struct mdev_device *mdev, + struct vm_area_struct *vma) +{ + struct vfio_sdmdev_queue *q = + (struct vfio_sdmdev_queue *)mdev_get_drvdata(mdev); + struct vfio_sdmdev *sdmdev = q->sdmdev; + + if (sdmdev->ops->mmap) + return sdmdev->ops->mmap(q, vma); + + dev_err(sdmdev->dev, "no driver mmap!\n"); + return -EINVAL; +} + +static inline int vfio_sdmdev_wait(struct vfio_sdmdev_queue *q, + unsigned long timeout) +{ + int ret; + struct vfio_sdmdev *sdmdev = q->sdmdev; + + if (!sdmdev->ops->mask_notify) + return -ENODEV; + + sdmdev->ops->mask_notify(q, VFIO_SDMDEV_EVENT_Q_UPDATE); + + ret = timeout ? wait_event_interruptible_timeout(q->wait, + sdmdev->ops->is_q_updated(q), timeout) : + wait_event_interruptible(q->wait, + sdmdev->ops->is_q_updated(q)); + + sdmdev->ops->mask_notify(q, 0); + + return ret; +} + +static long vfio_sdmdev_mdev_ioctl(struct mdev_device *mdev, unsigned int cmd, + unsigned long arg) +{ + struct vfio_sdmdev_queue *q = + (struct vfio_sdmdev_queue *)mdev_get_drvdata(mdev); + struct vfio_sdmdev *sdmdev = q->sdmdev; + + switch (cmd) { + case VFIO_SDMDEV_CMD_WAIT: + return vfio_sdmdev_wait(q, arg); + +#ifdef CONFIG_IOMMU_SVA + case VFIO_SDMDEV_CMD_BIND_PASID: + int ret; + + if (!vfio_sdmdev_is_valid_pasid(arg)) + return -EINVAL; + + mutex_lock(&q->mutex); + q->pasid = arg; + + if (sdmdev->ops->start_queue) + ret = sdmdev->ops->start_queue(q); + + mutex_unlock(&q->mutex); + + return ret; +#endif + + default: + if (sdmdev->ops->ioctl) + return sdmdev->ops->ioctl(q, cmd, arg); + + dev_err(sdmdev->dev, "ioctl cmd (%d) is not supported!\n", cmd); + return -EINVAL; + } +} + +static void vfio_sdmdev_release(struct device *dev) { } + +static void vfio_sdmdev_mdev_release(struct mdev_device *mdev) +{ + struct vfio_sdmdev_queue *q = + (struct vfio_sdmdev_queue *)mdev_get_drvdata(mdev); + struct vfio_sdmdev *sdmdev = q->sdmdev; + + if (sdmdev->ops->stop_queue) + sdmdev->ops->stop_queue(q); +} + +static int vfio_sdmdev_mdev_open(struct mdev_device *mdev) +{ +#ifndef CONFIG_IOMMU_SVA + struct vfio_sdmdev_queue *q = + (struct vfio_sdmdev_queue *)mdev_get_drvdata(mdev); + struct vfio_sdmdev *sdmdev = q->sdmdev; + + if (sdmdev->ops->start_queue) + sdmdev->ops->start_queue(q); +#endif + + return 0; +} + +/** + * vfio_sdmdev_register - register a sdmdev + * @sdmdev: device structure + */ +int vfio_sdmdev_register(struct vfio_sdmdev *sdmdev) +{ + int ret; + + if (!sdmdev->dev) + return -ENODEV; + + atomic_set(&sdmdev->ref, 0); + sdmdev->cls_dev.parent = sdmdev->dev; + sdmdev->cls_dev.class = sdmdev_class; + sdmdev->cls_dev.release = vfio_sdmdev_release; + dev_set_name(&sdmdev->cls_dev, "%s", dev_name(sdmdev->dev)); + ret = device_register(&sdmdev->cls_dev); + if (ret) + goto err; + + sdmdev->mdev_fops.owner = THIS_MODULE; + sdmdev->mdev_fops.dev_attr_groups = vfio_sdmdev_groups; + WARN_ON(!sdmdev->mdev_fops.supported_type_groups); + sdmdev->mdev_fops.create = vfio_sdmdev_mdev_create; + sdmdev->mdev_fops.remove = vfio_sdmdev_mdev_remove; + sdmdev->mdev_fops.ioctl = vfio_sdmdev_mdev_ioctl; + sdmdev->mdev_fops.open = vfio_sdmdev_mdev_open; + sdmdev->mdev_fops.release = vfio_sdmdev_mdev_release; + sdmdev->mdev_fops.mmap = vfio_sdmdev_mdev_mmap, + + ret = mdev_register_device(sdmdev->dev, &sdmdev->mdev_fops); + if (ret) + goto err_with_cls_dev; + + return 0; + +err_with_cls_dev: + device_unregister(&sdmdev->cls_dev); +err: + return ret; +} +EXPORT_SYMBOL_GPL(vfio_sdmdev_register); + +/** + * vfio_sdmdev_unregister - unregisters a sdmdev + * @sdmdev: device to unregister + * + * Unregister a sdmdev that wat previously successully registered with + * vfio_sdmdev_register(). + */ +void vfio_sdmdev_unregister(struct vfio_sdmdev *sdmdev) +{ + mdev_unregister_device(sdmdev->dev); + device_unregister(&sdmdev->cls_dev); +} +EXPORT_SYMBOL_GPL(vfio_sdmdev_unregister); + +static int __init vfio_sdmdev_init(void) +{ + sdmdev_class = class_create(THIS_MODULE, VFIO_SDMDEV_CLASS_NAME); + return PTR_ERR_OR_ZERO(sdmdev_class); +} + +static __exit void vfio_sdmdev_exit(void) +{ + class_destroy(sdmdev_class); +} + +module_init(vfio_sdmdev_init); +module_exit(vfio_sdmdev_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Hisilicon Tech. Co., Ltd."); +MODULE_DESCRIPTION("VFIO Share Domain Mediated Device"); diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index d9fd3188615d..ba73231d8692 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -41,6 +41,7 @@ #include #include #include +#include #define DRIVER_VERSION "0.2" #define DRIVER_AUTHOR "Alex Williamson " @@ -89,6 +90,8 @@ struct vfio_dma { }; struct vfio_group { + /* iommu_group of mdev's parent device */ + struct iommu_group *parent_group; struct iommu_group *iommu_group; struct list_head next; }; @@ -1327,6 +1330,109 @@ static bool vfio_iommu_has_sw_msi(struct iommu_group *group, phys_addr_t *base) return ret; } +/* return 0 if the device is not sdmdev. + * return 1 if the device is sdmdev, the data will be updated with parent + * device's group. + * return -errno if other error. + */ +static int vfio_sdmdev_type(struct device *dev, void *data) +{ + struct iommu_group **group = data; + struct iommu_group *pgroup; + int (*_is_sdmdev)(struct device *dev); + struct device *pdev; + int ret = 1; + + /* vfio_sdmdev module is not configurated */ + _is_sdmdev = symbol_get(vfio_sdmdev_is_sdmdev); + if (!_is_sdmdev) + return 0; + + /* check if it belongs to vfio_sdmdev device */ + if (!_is_sdmdev(dev)) { + ret = 0; + goto out; + } + + pdev = dev->parent; + pgroup = iommu_group_get(pdev); + if (!pgroup) { + ret = -ENODEV; + goto out; + } + + if (group) { + /* check if all parent devices is the same */ + if (*group && *group != pgroup) + ret = -ENODEV; + else + *group = pgroup; + } + + iommu_group_put(pgroup); + +out: + symbol_put(vfio_sdmdev_is_sdmdev); + + return ret; +} + +/* return 0 or -errno */ +static int vfio_sdmdev_bus(struct device *dev, void *data) +{ + struct bus_type **bus = data; + + if (!dev->bus) + return -ENODEV; + + /* ensure all devices has the same bus_type */ + if (*bus && *bus != dev->bus) + return -EINVAL; + + *bus = dev->bus; + return 0; +} + +/* return 0 means it is not sd group, 1 means it is, or -EXXX for error */ +static int vfio_iommu_type1_attach_sdgroup(struct vfio_domain *domain, + struct vfio_group *group, + struct iommu_group *iommu_group) +{ + int ret; + struct bus_type *pbus = NULL; + struct iommu_group *pgroup = NULL; + + ret = iommu_group_for_each_dev(iommu_group, &pgroup, + vfio_sdmdev_type); + if (ret < 0) + goto out; + else if (ret > 0) { + domain->domain = iommu_group_share_domain(pgroup); + if (IS_ERR(domain->domain)) + goto out; + ret = iommu_group_for_each_dev(pgroup, &pbus, + vfio_sdmdev_bus); + if (ret < 0) + goto err_with_share_domain; + + if (pbus && iommu_capable(pbus, IOMMU_CAP_CACHE_COHERENCY)) + domain->prot |= IOMMU_CACHE; + + group->parent_group = pgroup; + INIT_LIST_HEAD(&domain->group_list); + list_add(&group->next, &domain->group_list); + + return 1; + } + + return 0; + +err_with_share_domain: + iommu_group_unshare_domain(pgroup); +out: + return ret; +} + static int vfio_iommu_type1_attach_group(void *iommu_data, struct iommu_group *iommu_group) { @@ -1335,8 +1441,8 @@ static int vfio_iommu_type1_attach_group(void *iommu_data, struct vfio_domain *domain, *d; struct bus_type *bus = NULL, *mdev_bus; int ret; - bool resv_msi, msi_remap; - phys_addr_t resv_msi_base; + bool resv_msi = false, msi_remap; + phys_addr_t resv_msi_base = 0; mutex_lock(&iommu->lock); @@ -1373,6 +1479,14 @@ static int vfio_iommu_type1_attach_group(void *iommu_data, if (mdev_bus) { if ((bus == mdev_bus) && !iommu_present(bus)) { symbol_put(mdev_bus_type); + + ret = vfio_iommu_type1_attach_sdgroup(domain, group, + iommu_group); + if (ret < 0) + goto out_free; + else if (ret > 0) + goto replay_check; + if (!iommu->external_domain) { INIT_LIST_HEAD(&domain->group_list); iommu->external_domain = domain; @@ -1451,12 +1565,13 @@ static int vfio_iommu_type1_attach_group(void *iommu_data, vfio_test_domain_fgsp(domain); +replay_check: /* replay mappings on new domains */ ret = vfio_iommu_replay(iommu, domain); if (ret) goto out_detach; - if (resv_msi) { + if (!group->parent_group && resv_msi) { ret = iommu_get_msi_cookie(domain->domain, resv_msi_base); if (ret) goto out_detach; @@ -1471,7 +1586,10 @@ static int vfio_iommu_type1_attach_group(void *iommu_data, out_detach: iommu_detach_group(domain->domain, iommu_group); out_domain: - iommu_domain_free(domain->domain); + if (group->parent_group) + iommu_group_unshare_domain(group->parent_group); + else + iommu_domain_free(domain->domain); out_free: kfree(domain); kfree(group); @@ -1527,12 +1645,25 @@ static void vfio_sanity_check_pfn_list(struct vfio_iommu *iommu) WARN_ON(iommu->notifier.head); } +static void vfio_iommu_undo(struct vfio_iommu *iommu, + struct iommu_domain *domain) +{ + struct rb_node *n = rb_first(&iommu->dma_list); + struct vfio_dma *dma; + + for (; n; n = rb_next(n)) { + dma = rb_entry(n, struct vfio_dma, node); + iommu_unmap(domain, dma->iova, dma->size); + } +} + static void vfio_iommu_type1_detach_group(void *iommu_data, struct iommu_group *iommu_group) { struct vfio_iommu *iommu = iommu_data; struct vfio_domain *domain; struct vfio_group *group; + struct iommu_domain *sdomain = NULL; mutex_lock(&iommu->lock); @@ -1560,7 +1691,12 @@ static void vfio_iommu_type1_detach_group(void *iommu_data, if (!group) continue; - iommu_detach_group(domain->domain, iommu_group); + if (group->parent_group) + sdomain = iommu_group_unshare_domain( + group->parent_group); + else + iommu_detach_group(domain->domain, iommu_group); + list_del(&group->next); kfree(group); /* @@ -1577,7 +1713,10 @@ static void vfio_iommu_type1_detach_group(void *iommu_data, else vfio_iommu_unmap_unpin_reaccount(iommu); } - iommu_domain_free(domain->domain); + if (domain->domain != sdomain) + iommu_domain_free(domain->domain); + else + vfio_iommu_undo(iommu, sdomain); list_del(&domain->next); kfree(domain); } diff --git a/include/linux/vfio_sdmdev.h b/include/linux/vfio_sdmdev.h new file mode 100644 index 000000000000..fbc9fb3f4abc --- /dev/null +++ b/include/linux/vfio_sdmdev.h @@ -0,0 +1,96 @@ +/* SPDX-License-Identifier: GPL-2.0+ */ +#ifndef __VFIO_SDMDEV_H +#define __VFIO_SDMDEV_H + +#include +#include +#include +#include +#include + +struct vfio_sdmdev_queue; +struct vfio_sdmdev; + +/* event bit used to mask the hardware irq */ +#define VFIO_SDMDEV_EVENT_Q_UPDATE BIT(0) /* irq if queue is updated */ + +/** + * struct vfio_sdmdev_ops - WD device operations + * @get_queue: get a queue from the device according to algorithm + * @put_queue: free a queue to the device + * @start_queue: put queue into action with current process's pasid. + * @stop_queue: stop queue from running state + * @is_q_updated: check whether the task is finished + * @mask_notify: mask the task irq of queue + * @mmap: mmap addresses of queue to user space + * @reset: reset the WD device + * @reset_queue: reset the queue + * @ioctl: ioctl for user space users of the queue + * @get_available_instances: get numbers of the queue remained + */ +struct vfio_sdmdev_ops { + int (*get_queue)(struct vfio_sdmdev *sdmdev, + struct vfio_sdmdev_queue **q); + void (*put_queue)(struct vfio_sdmdev_queue *q); + int (*start_queue)(struct vfio_sdmdev_queue *q); + void (*stop_queue)(struct vfio_sdmdev_queue *q); + int (*is_q_updated)(struct vfio_sdmdev_queue *q); + void (*mask_notify)(struct vfio_sdmdev_queue *q, int event_mask); + int (*mmap)(struct vfio_sdmdev_queue *q, struct vm_area_struct *vma); + int (*reset)(struct vfio_sdmdev *sdmdev); + int (*reset_queue)(struct vfio_sdmdev_queue *q); + long (*ioctl)(struct vfio_sdmdev_queue *q, unsigned int cmd, + unsigned long arg); + int (*get_available_instances)(struct vfio_sdmdev *sdmdev); +}; + +struct vfio_sdmdev_queue { + struct mutex mutex; + struct vfio_sdmdev *sdmdev; + __u32 flags; + void *priv; + wait_queue_head_t wait; + struct mdev_device *mdev; + int fd; + int container; +#ifdef CONFIG_IOMMU_SVA + int pasid; +#endif +}; + +struct vfio_sdmdev { + const char *name; + int status; + atomic_t ref; + const struct vfio_sdmdev_ops *ops; + struct device *dev; + struct device cls_dev; + bool is_vf; + u32 iommu_type; + u32 dma_flag; + void *priv; + int flags; + const char *api_ver; + struct mdev_parent_ops mdev_fops; +}; + +int vfio_sdmdev_register(struct vfio_sdmdev *sdmdev); +void vfio_sdmdev_unregister(struct vfio_sdmdev *sdmdev); +void vfio_sdmdev_wake_up(struct vfio_sdmdev_queue *q); +int vfio_sdmdev_is_sdmdev(struct device *dev); +struct vfio_sdmdev *vfio_sdmdev_pdev_sdmdev(struct device *dev); +struct vfio_sdmdev *mdev_sdmdev(struct mdev_device *mdev); + +extern struct mdev_type_attribute mdev_type_attr_flags; +extern struct mdev_type_attribute mdev_type_attr_name; +extern struct mdev_type_attribute mdev_type_attr_device_api; +extern struct mdev_type_attribute mdev_type_attr_available_instances; +#define VFIO_SDMDEV_DEFAULT_MDEV_TYPE_ATTRS \ + &mdev_type_attr_name.attr, \ + &mdev_type_attr_device_api.attr, \ + &mdev_type_attr_available_instances.attr, \ + &mdev_type_attr_flags.attr + +#define _VFIO_SDMDEV_REGION(vm_pgoff) (vm_pgoff & 0xf) + +#endif diff --git a/include/uapi/linux/vfio_sdmdev.h b/include/uapi/linux/vfio_sdmdev.h new file mode 100644 index 000000000000..79fa33fbc8c0 --- /dev/null +++ b/include/uapi/linux/vfio_sdmdev.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0+ */ +#ifndef _UAPIVFIO_SDMDEV_H +#define _UAPIVFIO_SDMDEV_H + +#include + +#define VFIO_SDMDEV_CLASS_NAME "sdmdev" + +/* Device ATTRs in parent dev SYSFS DIR */ +#define VFIO_SDMDEV_PDEV_ATTRS_GRP_NAME "params" + +/* Parent device attributes */ +#define SDMDEV_IOMMU_TYPE "iommu_type" +#define SDMDEV_DMA_FLAG "dma_flag" + +/* Maximum length of algorithm name string */ +#define VFIO_SDMDEV_ALG_NAME_SIZE 64 + +/* the bits used in SDMDEV_DMA_FLAG attributes */ +#define VFIO_SDMDEV_DMA_INVALID 0 +#define VFIO_SDMDEV_DMA_SINGLE_PROC_MAP 1 +#define VFIO_SDMDEV_DMA_MULTI_PROC_MAP 2 +#define VFIO_SDMDEV_DMA_SVM 4 +#define VFIO_SDMDEV_DMA_SVM_NO_FAULT 8 +#define VFIO_SDMDEV_DMA_PHY 16 + +#define VFIO_SDMDEV_CMD_WAIT _IO('W', 1) +#define VFIO_SDMDEV_CMD_BIND_PASID _IO('W', 2) +#endif From patchwork Mon Sep 3 00:52:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kenneth Lee X-Patchwork-Id: 145740 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1927327ljw; Sun, 2 Sep 2018 17:52:42 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZ8LE4yQVOJcGeuBViXoHrbFimHAmLbtX1AEtp6aCgnjkL/nx96Rk9k5HctIZcXbAQIeBis X-Received: by 2002:a63:d10c:: with SMTP id k12-v6mr24473268pgg.49.1535935962204; Sun, 02 Sep 2018 17:52:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535935962; cv=none; d=google.com; s=arc-20160816; b=HL5xOoWf7IYK/viI+6WX27mjGxMJzvl7kd3r/dvSsd6HEyl1jqpNOAz+CizoKmVjte 6PYG28yLTI+VgIa0alBFnH0G6aaa/VuE3z96tlUsxQn/zA8HQ+qX1DA7YmwU3P1yGk/X xkC0EqNEyOsihN13JBaFI5y+jvi7zppAM4C3JitbVketyq9UJPQ62DO8yw5k9KMCL51N JWQmmGlADXiK7w5ai0kFLti6sLCd70xFJZZTSJJabR8sUCZ2pWfYWrrlxRwEKqSH6XPz OT4lJlJEkmjFCsep9jEgwCYaH8TeVW6b850DNWWzfcFjctYKeE1SpeySjk7puwOAYHCF r93A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=ifnGz23tcfOCOYXeXWUntse7AHEi6PUi1GBXUgQwneM=; b=VGWDybORQVV+3E/JXHSJPo8tRi5zi+K+yOPb8omKYXfLcxxFry8wrwqHCpKigro8Eq 9fhrkmN3Bjp0+h7k4GM4c7CgNZciVQQY/HS2QjODKWgFf+jIZikKpWnHwDDYl5hHHsOs jpcC8/8cyzuXpHfeZ5C68T9nyGEcoXDIZ+76Qsp0/Z7NhbfuY1beDmsLooKM4Ue4wJ9p lma1xSffXJkWfCsXKHP/8T6EFOxm2V06GJo+hEOWFHUlsPyEzmJJ0E5Muw+Hv0EagOxa CuvzmvzzWKIekfOgXAfVQcWEnD9otX9USOq/pD2eHF8FOPwgMrvr5vV0oc+y8ya3pTMK aAmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=GEfnbq+f; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id bc6-v6si16341959plb.115.2018.09.02.17.52.41; Sun, 02 Sep 2018 17:52:42 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=GEfnbq+f; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727156AbeICFKS (ORCPT + 32 others); Mon, 3 Sep 2018 01:10:18 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:46133 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726157AbeICFKS (ORCPT ); Mon, 3 Sep 2018 01:10:18 -0400 Received: by mail-pl1-f195.google.com with SMTP id t19-v6so110403ply.13; Sun, 02 Sep 2018 17:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ifnGz23tcfOCOYXeXWUntse7AHEi6PUi1GBXUgQwneM=; b=GEfnbq+flSCi0dJZlgCm7LfcPP3fidPz55L9PbUu1tWAplmAFi0TedIRAyCQPUaHmK uPMtf4wUIndPJ7N5rGhUY1g7CjjqSW6mecaycVTSRiKVymudhgwXFAOyBS6mmxuFfi6T RRfj7PLlRDpXHqk1lngfpZVI1m/7XPbTsGKiCViykFGSmSe+a2cVdEz5nms3ny3haqjA Y8fB8nCF2dX4BimY8YsbeWwMrVvdbTRSVK8AKXM9lCgaDOTnHnDqtHy/DL3RasAYotTh jCoPcveOay0rB5AkTvHX6d/4WEPnL1VzmgDyejHVOKaITE2iy2ek9ARVmNZhQRjIafvo xtVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ifnGz23tcfOCOYXeXWUntse7AHEi6PUi1GBXUgQwneM=; b=IvP5KfizdRfTBNjPpvgh55Y394/nZWmZGpaID7y57ochelC/e8a/4m4ixDkhLLkWLV qiEMA5+9Cgvo9ratcB9pE+Pk9UlNUtfkPw3jKPKg4XygLI1b2gXLJvXOOvyQ//D6h4Z+ t0bRElAEqxjuaLs1ZttaU0AKE1r1ASPGZEL2I4D0yRRoEuC7pBCbNup1r81iyeXGTUzY xqVOKB+0kbURXG9vumnI1O992RhquOC4X2XnOfqqlvsVSTCPVEudDNmEDW1LnLkxCQe7 ugD8uiBcCa2NvI83plc9VGQqJoSFHDT3UyMiej4qKlIMLYO5v+/kKUjqaHuRXjNOhiiz wo1g== X-Gm-Message-State: APzg51AlyN5GdkMTMg+f947FyFrrcAx1VnWMJJlT1LzxxdfvVt96xMol /irLF0UgyR7a8kNmVc1gOmw= X-Received: by 2002:a17:902:6845:: with SMTP id f5-v6mr26298903pln.17.1535935957067; Sun, 02 Sep 2018 17:52:37 -0700 (PDT) Received: from localhost.localdomain ([203.160.89.93]) by smtp.gmail.com with ESMTPSA id w81-v6sm31487064pfk.92.2018.09.02.17.52.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 17:52:36 -0700 (PDT) From: Kenneth Lee To: Jonathan Corbet , Herbert Xu , "David S . Miller" , Joerg Roedel , Alex Williamson , Kenneth Lee , Hao Fang , Zhou Wang , Zaibo Xu , Philippe Ombredanne , Greg Kroah-Hartman , Thomas Gleixner , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, linux-accelerators@lists.ozlabs.org, Lu Baolu , Sanjay Kumar Cc: linuxarm@huawei.com Subject: [PATCH 4/7] crypto: add hisilicon Queue Manager driver Date: Mon, 3 Sep 2018 08:52:01 +0800 Message-Id: <20180903005204.26041-5-nek.in.cn@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903005204.26041-1-nek.in.cn@gmail.com> References: <20180903005204.26041-1-nek.in.cn@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Kenneth Lee Hisilicon QM is a general IP used by some Hisilicon accelerators. It provides a general PCIE interface for the CPU and the accelerator to share a group of queues. This commit includes a library used by the accelerator driver to access the QM hardware. Signed-off-by: Kenneth Lee Signed-off-by: Zhou Wang Signed-off-by: Hao Fang --- drivers/crypto/Makefile | 2 +- drivers/crypto/hisilicon/Kconfig | 8 + drivers/crypto/hisilicon/Makefile | 1 + drivers/crypto/hisilicon/qm.c | 820 ++++++++++++++++++++++++++++++ drivers/crypto/hisilicon/qm.h | 110 ++++ 5 files changed, 940 insertions(+), 1 deletion(-) create mode 100644 drivers/crypto/hisilicon/qm.c create mode 100644 drivers/crypto/hisilicon/qm.h -- 2.17.1 diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile index c23396f32c8a..f3a7abe42424 100644 --- a/drivers/crypto/Makefile +++ b/drivers/crypto/Makefile @@ -46,4 +46,4 @@ obj-$(CONFIG_CRYPTO_DEV_VMX) += vmx/ obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) += bcm/ obj-$(CONFIG_CRYPTO_DEV_SAFEXCEL) += inside-secure/ obj-$(CONFIG_CRYPTO_DEV_ARTPEC6) += axis/ -obj-y += hisilicon/ +obj-$(CONFIG_CRYPTO_DEV_HISILICON) += hisilicon/ diff --git a/drivers/crypto/hisilicon/Kconfig b/drivers/crypto/hisilicon/Kconfig index 8ca9c503bcb0..02a6eef84101 100644 --- a/drivers/crypto/hisilicon/Kconfig +++ b/drivers/crypto/hisilicon/Kconfig @@ -1,4 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 +config CRYPTO_DEV_HISILICON + tristate "Support for HISILICON CRYPTO ACCELERATOR" + help + Enable this to use Hisilicon Hardware Accelerators config CRYPTO_DEV_HISI_SEC tristate "Support for Hisilicon SEC crypto block cipher accelerator" @@ -12,3 +16,7 @@ config CRYPTO_DEV_HISI_SEC To compile this as a module, choose M here: the module will be called hisi_sec. + +config CRYPTO_DEV_HISI_QM + tristate + depends on ARM64 && PCI diff --git a/drivers/crypto/hisilicon/Makefile b/drivers/crypto/hisilicon/Makefile index 463f46ace182..05e9052e0f52 100644 --- a/drivers/crypto/hisilicon/Makefile +++ b/drivers/crypto/hisilicon/Makefile @@ -1,2 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CRYPTO_DEV_HISI_SEC) += sec/ +obj-$(CONFIG_CRYPTO_DEV_HISI_QM) += qm.o diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c new file mode 100644 index 000000000000..ea618b4d0929 --- /dev/null +++ b/drivers/crypto/hisilicon/qm.c @@ -0,0 +1,820 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include "qm.h" + +#define QM_DEF_Q_NUM 128 + +/* eq/aeq irq enable */ +#define QM_VF_AEQ_INT_SOURCE 0x0 +#define QM_VF_AEQ_INT_MASK 0x4 +#define QM_VF_EQ_INT_SOURCE 0x8 +#define QM_VF_EQ_INT_MASK 0xc + +/* mailbox */ +#define MAILBOX_CMD_SQC 0x0 +#define MAILBOX_CMD_CQC 0x1 +#define MAILBOX_CMD_EQC 0x2 +#define MAILBOX_CMD_SQC_BT 0x4 +#define MAILBOX_CMD_CQC_BT 0x5 + +#define MAILBOX_CMD_SEND_BASE 0x300 +#define MAILBOX_EVENT_SHIFT 8 +#define MAILBOX_STATUS_SHIFT 9 +#define MAILBOX_BUSY_SHIFT 13 +#define MAILBOX_OP_SHIFT 14 +#define MAILBOX_QUEUE_SHIFT 16 + +/* sqc shift */ +#define SQ_HEAD_SHIFT 0 +#define SQ_TAIL_SHIFI 16 +#define SQ_HOP_NUM_SHIFT 0 +#define SQ_PAGE_SIZE_SHIFT 4 +#define SQ_BUF_SIZE_SHIFT 8 +#define SQ_SQE_SIZE_SHIFT 12 +#define SQ_HEAD_IDX_SIG_SHIFT 0 +#define SQ_TAIL_IDX_SIG_SHIFT 0 +#define SQ_CQN_SHIFT 0 +#define SQ_PRIORITY_SHIFT 0 +#define SQ_ORDERS_SHIFT 4 +#define SQ_TYPE_SHIFT 8 + +#define SQ_TYPE_MASK 0xf + +/* cqc shift */ +#define CQ_HEAD_SHIFT 0 +#define CQ_TAIL_SHIFI 16 +#define CQ_HOP_NUM_SHIFT 0 +#define CQ_PAGE_SIZE_SHIFT 4 +#define CQ_BUF_SIZE_SHIFT 8 +#define CQ_SQE_SIZE_SHIFT 12 +#define CQ_PASID 0 +#define CQ_HEAD_IDX_SIG_SHIFT 0 +#define CQ_TAIL_IDX_SIG_SHIFT 0 +#define CQ_CQN_SHIFT 0 +#define CQ_PRIORITY_SHIFT 16 +#define CQ_ORDERS_SHIFT 0 +#define CQ_TYPE_SHIFT 0 +#define CQ_PHASE_SHIFT 0 +#define CQ_FLAG_SHIFT 1 + +#define CQC_HEAD_INDEX(cqc) ((cqc)->cq_head) +#define CQC_PHASE(cqc) (((cqc)->dw6) & 0x1) +#define CQC_CQ_ADDRESS(cqc) (((u64)((cqc)->cq_base_h) << 32) | \ + ((cqc)->cq_base_l)) +#define CQC_PHASE_BIT 0x1 + +/* eqc shift */ +#define MB_EQC_EQE_SHIFT 12 +#define MB_EQC_PHASE_SHIFT 16 + +#define EQC_HEAD_INDEX(eqc) ((eqc)->eq_head) +#define EQC_TAIL_INDEX(eqc) ((eqc)->eq_tail) +#define EQC_PHASE(eqc) ((((eqc)->dw6) >> 16) & 0x1) + +#define EQC_PHASE_BIT 0x00010000 + +/* cqe shift */ +#define CQE_PHASE(cqe) ((cqe)->w7 & 0x1) +#define CQE_SQ_NUM(cqe) ((cqe)->sq_num) +#define CQE_SQ_HEAD_INDEX(cqe) ((cqe)->sq_head) + +/* eqe shift */ +#define EQE_PHASE(eqe) (((eqe)->dw0 >> 16) & 0x1) +#define EQE_CQN(eqe) (((eqe)->dw0) & 0xffff) + +#define QM_EQE_CQN_MASK 0xffff + +/* doorbell */ +#define DOORBELL_CMD_SQ 0 +#define DOORBELL_CMD_CQ 1 +#define DOORBELL_CMD_EQ 2 +#define DOORBELL_CMD_AEQ 3 + +#define DOORBELL_CMD_SEND_BASE 0x340 + +#define QM_MEM_START_INIT 0x100040 +#define QM_MEM_INIT_DONE 0x100044 +#define QM_VFT_CFG_RDY 0x10006c +#define QM_VFT_CFG_OP_WR 0x100058 +#define QM_VFT_CFG_TYPE 0x10005c +#define QM_SQC_VFT 0x0 +#define QM_CQC_VFT 0x1 +#define QM_VFT_CFG_ADDRESS 0x100060 +#define QM_VFT_CFG_OP_ENABLE 0x100054 + +#define QM_VFT_CFG_DATA_L 0x100064 +#define QM_VFT_CFG_DATA_H 0x100068 +#define QM_SQC_VFT_BUF_SIZE (7ULL << 8) +#define QM_SQC_VFT_SQC_SIZE (5ULL << 12) +#define QM_SQC_VFT_INDEX_NUMBER (1ULL << 16) +#define QM_SQC_VFT_BT_INDEX_SHIFT 22 +#define QM_SQC_VFT_START_SQN_SHIFT 28 +#define QM_SQC_VFT_VALID (1ULL << 44) +#define QM_CQC_VFT_BUF_SIZE (7ULL << 8) +#define QM_CQC_VFT_SQC_SIZE (5ULL << 12) +#define QM_CQC_VFT_INDEX_NUMBER (1ULL << 16) +#define QM_CQC_VFT_BT_INDEX_SHIFT 22 +#define QM_CQC_VFT_VALID (1ULL << 28) + +struct cqe { + __le32 rsvd0; + __le16 cmd_id; + __le16 rsvd1; + __le16 sq_head; + __le16 sq_num; + __le16 rsvd2; + __le16 w7; +}; + +struct eqe { + __le32 dw0; +}; + +struct sqc { + __le16 head; + __le16 tail; + __le32 base_l; + __le32 base_h; + __le32 dw3; + __le16 qes; + __le16 rsvd0; + __le16 pasid; + __le16 w11; + __le16 cq_num; + __le16 w13; + __le32 rsvd1; +}; + +struct cqc { + __le16 head; + __le16 tail; + __le32 base_l; + __le32 base_h; + __le32 dw3; + __le16 qes; + __le16 rsvd0; + __le16 pasid; + __le16 w11; + __le32 dw6; + __le32 rsvd1; +}; + +#define INIT_QC(qc, base) do { \ + (qc)->head = 0; \ + (qc)->tail = 0; \ + (qc)->base_l = lower_32_bits(base); \ + (qc)->base_h = upper_32_bits(base); \ + (qc)->pasid = 0; \ + (qc)->w11 = 0; \ + (qc)->rsvd1 = 0; \ + (qc)->qes = QM_Q_DEPTH - 1; \ +} while (0) + +struct eqc { + __le16 head; + __le16 tail; + __le32 base_l; + __le32 base_h; + __le32 dw3; + __le32 rsvd[2]; + __le32 dw6; +}; + +struct mailbox { + __le16 w0; + __le16 queue_num; + __le32 base_l; + __le32 base_h; + __le32 rsvd; +}; + +struct doorbell { + __le16 queue_num; + __le16 cmd; + __le16 index; + __le16 priority; +}; + +#define QM_DMA_BUF(p, buf) ((struct buf *)(p)->buf.addr) +#define QM_SQC(p) QM_DMA_BUF(p, sqc) +#define QM_CQC(p) QM_DMA_BUF(p, cqc) +#define QM_EQC(p) QM_DMA_BUF(p, eqc) +#define QM_EQE(p) QM_DMA_BUF(p, eqe) + +#define QP_SQE_DMA(qp) ((qp)->scqe.dma) +#define QP_CQE(qp) ((struct cqe *)((qp)->scqe.addr + \ + qp->qm->sqe_size * QM_Q_DEPTH)) +#define QP_CQE_DMA(qp) ((qp)->scqe.dma + qp->qm->sqe_size * QM_Q_DEPTH) + +static inline void qm_writel(struct qm_info *qm, u32 val, u32 offset) +{ + writel(val, qm->io_base + offset); +} + +struct qm_info; + +struct hisi_acc_qm_hw_ops { + int (*vft_config)(struct qm_info *qm, u16 base, u32 number); +}; + +static inline int hacc_qm_mb_is_busy(struct qm_info *qm) +{ + u32 val; + + return readl_relaxed_poll_timeout(QM_ADDR(qm, MAILBOX_CMD_SEND_BASE), + val, !((val >> MAILBOX_BUSY_SHIFT) & 0x1), 10, 1000); +} + +static inline void qm_mb_write(struct qm_info *qm, void *src) +{ + void __iomem *fun_base = QM_ADDR(qm, MAILBOX_CMD_SEND_BASE); + unsigned long tmp0 = 0, tmp1 = 0; + + asm volatile("ldp %0, %1, %3\n" + "stp %0, %1, %2\n" + "dsb sy\n" + : "=&r" (tmp0), + "=&r" (tmp1), + "+Q" (*((char *)fun_base)) + : "Q" (*((char *)src)) + : "memory"); +} + +static int qm_mb(struct qm_info *qm, u8 cmd, dma_addr_t dma_addr, u16 queue, + bool op, bool event) +{ + struct mailbox mailbox; + int i = 0; + int ret = 0; + + memset(&mailbox, 0, sizeof(struct mailbox)); + + mailbox.w0 = cmd | + (event ? 0x1 << MAILBOX_EVENT_SHIFT : 0) | + (op ? 0x1 << MAILBOX_OP_SHIFT : 0) | + (0x1 << MAILBOX_BUSY_SHIFT); + mailbox.queue_num = queue; + mailbox.base_l = lower_32_bits(dma_addr); + mailbox.base_h = upper_32_bits(dma_addr); + mailbox.rsvd = 0; + + mutex_lock(&qm->mailbox_lock); + + while (hacc_qm_mb_is_busy(qm) && i < 10) + i++; + if (i >= 10) { + ret = -EBUSY; + dev_err(&qm->pdev->dev, "QM mail box is busy!"); + goto busy_unlock; + } + qm_mb_write(qm, &mailbox); + i = 0; + while (hacc_qm_mb_is_busy(qm) && i < 10) + i++; + if (i >= 10) { + ret = -EBUSY; + dev_err(&qm->pdev->dev, "QM mail box is still busy!"); + goto busy_unlock; + } + +busy_unlock: + mutex_unlock(&qm->mailbox_lock); + + return ret; +} + +static void qm_db(struct qm_info *qm, u16 qn, u8 cmd, u16 index, u8 priority) +{ + u64 doorbell = 0; + + doorbell = (u64)qn | ((u64)cmd << 16); + doorbell |= ((u64)index | ((u64)priority << 16)) << 32; + + writeq(doorbell, QM_ADDR(qm, DOORBELL_CMD_SEND_BASE)); +} + +/* @return 0 - cq/eq event, 1 - async event, 2 - abnormal error */ +static u32 qm_get_irq_source(struct qm_info *qm) +{ + return readl(QM_ADDR(qm, QM_VF_EQ_INT_SOURCE)); +} + +static inline struct hisi_qp *to_hisi_qp(struct qm_info *qm, struct eqe *eqe) +{ + u16 cqn = eqe->dw0 & QM_EQE_CQN_MASK; + struct hisi_qp *qp; + + read_lock(&qm->qps_lock); + qp = qm->qp_array[cqn]; + read_unlock(&qm->qps_lock); + + return qp; +} + +static inline void qm_cq_head_update(struct hisi_qp *qp) +{ + if (qp->qp_status.cq_head == QM_Q_DEPTH - 1) { + QM_CQC(qp)->dw6 = QM_CQC(qp)->dw6 ^ CQC_PHASE_BIT; + qp->qp_status.cq_head = 0; + } else { + qp->qp_status.cq_head++; + } +} + +static inline void qm_poll_qp(struct hisi_qp *qp, struct qm_info *qm) +{ + struct cqe *cqe; + + cqe = QP_CQE(qp) + qp->qp_status.cq_head; + + if (qp->req_cb) { + while (CQE_PHASE(cqe) == CQC_PHASE(QM_CQC(qp))) { + dma_rmb(); + qp->req_cb(qp, QP_SQE_ADDR(qp) + + qm->sqe_size * + CQE_SQ_HEAD_INDEX(cqe)); + qm_cq_head_update(qp); + cqe = QP_CQE(qp) + qp->qp_status.cq_head; + } + } else if (qp->event_cb) { + qp->event_cb(qp); + qm_cq_head_update(qp); + cqe = QP_CQE(qp) + qp->qp_status.cq_head; + } + + qm_db(qm, qp->queue_id, DOORBELL_CMD_CQ, qp->qp_status.cq_head, 0); + + /* set c_flag */ + qm_db(qm, qp->queue_id, DOORBELL_CMD_CQ, qp->qp_status.cq_head, 1); +} + +static irqreturn_t qm_irq_thread(int irq, void *data) +{ + struct qm_info *qm = data; + struct eqe *eqe = QM_EQE(qm) + qm->eq_head; + struct eqc *eqc = QM_EQC(qm); + struct hisi_qp *qp; + + while (EQE_PHASE(eqe) == EQC_PHASE(eqc)) { + qp = to_hisi_qp(qm, eqe); + if (qp) + qm_poll_qp(qp, qm); + + if (qm->eq_head == QM_Q_DEPTH - 1) { + eqc->dw6 = eqc->dw6 ^ EQC_PHASE_BIT; + eqe = QM_EQE(qm); + qm->eq_head = 0; + } else { + eqe++; + qm->eq_head++; + } + + qm_db(qm, 0, DOORBELL_CMD_EQ, qm->eq_head, 0); + } + + return IRQ_HANDLED; +} + +static void qm_init_qp_status(struct hisi_qp *qp) +{ + struct hisi_acc_qp_status *qp_status = &qp->qp_status; + + qp_status->sq_tail = 0; + qp_status->sq_head = 0; + qp_status->cq_head = 0; + qp_status->cqc_phase = 1; + qp_status->is_sq_full = 0; +} + +/* check if bit in regs is 1 */ +static inline int qm_acc_check(struct qm_info *qm, u32 offset, u32 bit) +{ + int val; + + return readl_relaxed_poll_timeout(QM_ADDR(qm, offset), val, + val & BIT(bit), 10, 1000); +} + +static inline int qm_init_q_buffer(struct device *dev, size_t size, + struct qm_dma_buffer *db) +{ + db->size = size; + db->addr = dma_zalloc_coherent(dev, size, &db->dma, GFP_KERNEL); + if (!db->addr) + return -ENOMEM; + + return 0; +} + +static inline void qm_uninit_q_buffer(struct device *dev, + struct qm_dma_buffer *db) +{ + dma_free_coherent(dev, db->size, db->addr, db->dma); +} + +static inline int qm_init_bt(struct qm_info *qm, struct device *dev, + size_t size, struct qm_dma_buffer *db, int mb_cmd) +{ + int ret; + + ret = qm_init_q_buffer(dev, size, db); + if (ret) + return -ENOMEM; + + ret = qm_mb(qm, mb_cmd, db->dma, 0, 0, 0); + if (ret) { + qm_uninit_q_buffer(dev, db); + return ret; + } + + return 0; +} + +/* the config should be conducted after hisi_acc_init_qm_mem() */ +static int qm_vft_common_config(struct qm_info *qm, u16 base, u32 number) +{ + u64 tmp; + int ret; + + ret = qm_acc_check(qm, QM_VFT_CFG_RDY, 0); + if (ret) + return ret; + qm_writel(qm, 0x0, QM_VFT_CFG_OP_WR); + qm_writel(qm, QM_SQC_VFT, QM_VFT_CFG_TYPE); + qm_writel(qm, qm->pdev->devfn, QM_VFT_CFG_ADDRESS); + + tmp = QM_SQC_VFT_BUF_SIZE | + QM_SQC_VFT_SQC_SIZE | + QM_SQC_VFT_INDEX_NUMBER | + QM_SQC_VFT_VALID | + (u64)base << QM_SQC_VFT_START_SQN_SHIFT; + + qm_writel(qm, tmp & 0xffffffff, QM_VFT_CFG_DATA_L); + qm_writel(qm, tmp >> 32, QM_VFT_CFG_DATA_H); + + qm_writel(qm, 0x0, QM_VFT_CFG_RDY); + qm_writel(qm, 0x1, QM_VFT_CFG_OP_ENABLE); + ret = qm_acc_check(qm, QM_VFT_CFG_RDY, 0); + if (ret) + return ret; + tmp = 0; + + qm_writel(qm, 0x0, QM_VFT_CFG_OP_WR); + qm_writel(qm, QM_CQC_VFT, QM_VFT_CFG_TYPE); + qm_writel(qm, qm->pdev->devfn, QM_VFT_CFG_ADDRESS); + + tmp = QM_CQC_VFT_BUF_SIZE | + QM_CQC_VFT_SQC_SIZE | + QM_CQC_VFT_INDEX_NUMBER | + QM_CQC_VFT_VALID; + + qm_writel(qm, tmp & 0xffffffff, QM_VFT_CFG_DATA_L); + qm_writel(qm, tmp >> 32, QM_VFT_CFG_DATA_H); + + qm_writel(qm, 0x0, QM_VFT_CFG_RDY); + qm_writel(qm, 0x1, QM_VFT_CFG_OP_ENABLE); + ret = qm_acc_check(qm, QM_VFT_CFG_RDY, 0); + if (ret) + return ret; + return 0; +} + +static struct hisi_acc_qm_hw_ops qm_hw_ops_v1 = { + .vft_config = qm_vft_common_config, +}; + +struct hisi_qp *hisi_qm_create_qp(struct qm_info *qm, u8 alg_type) +{ + struct hisi_qp *qp; + int qp_index; + int ret; + + write_lock(&qm->qps_lock); + qp_index = find_first_zero_bit(qm->qp_bitmap, qm->qp_num); + if (qp_index >= qm->qp_num) { + write_unlock(&qm->qps_lock); + return ERR_PTR(-EBUSY); + } + set_bit(qp_index, qm->qp_bitmap); + + qp = kzalloc(sizeof(*qp), GFP_KERNEL); + if (!qp) { + ret = -ENOMEM; + write_unlock(&qm->qps_lock); + goto err_with_bitset; + } + + qp->queue_id = qp_index; + qp->qm = qm; + qp->alg_type = alg_type; + qm_init_qp_status(qp); + + write_unlock(&qm->qps_lock); + return qp; + +err_with_bitset: + clear_bit(qp_index, qm->qp_bitmap); + + return ERR_PTR(ret); +} +EXPORT_SYMBOL_GPL(hisi_qm_create_qp); + +int hisi_qm_start_qp(struct hisi_qp *qp, unsigned long arg) +{ + struct qm_info *qm = qp->qm; + struct device *dev = &qm->pdev->dev; + struct sqc *sqc; + struct cqc *cqc; + int qp_index = qp->queue_id; + int pasid = arg; + int ret; + + /* set sq and cq context */ + qp->sqc.addr = QM_SQC(qm) + qp_index; + qp->sqc.dma = qm->sqc.dma + qp_index * sizeof(struct sqc); + sqc = QM_SQC(qp); + + qp->cqc.addr = QM_CQC(qm) + qp_index; + qp->cqc.dma = qm->cqc.dma + qp_index * sizeof(struct cqc); + cqc = QM_CQC(qp); + + /* allocate sq and cq */ + ret = qm_init_q_buffer(dev, + qm->sqe_size * QM_Q_DEPTH + sizeof(struct cqe) * QM_Q_DEPTH, + &qp->scqe); + if (ret) + return ret; + + INIT_QC(sqc, qp->scqe.dma); + sqc->pasid = pasid; + sqc->dw3 = (0 << SQ_HOP_NUM_SHIFT) | + (0 << SQ_PAGE_SIZE_SHIFT) | + (0 << SQ_BUF_SIZE_SHIFT) | + (ilog2(qm->sqe_size) << SQ_SQE_SIZE_SHIFT); + sqc->cq_num = qp_index; + sqc->w13 = 0 << SQ_PRIORITY_SHIFT | + 1 << SQ_ORDERS_SHIFT | + (qp->alg_type & SQ_TYPE_MASK) << SQ_TYPE_SHIFT; + + ret = qm_mb(qm, MAILBOX_CMD_SQC, qp->sqc.dma, qp_index, 0, 0); + if (ret) + return ret; + + INIT_QC(cqc, qp->scqe.dma + qm->sqe_size * QM_Q_DEPTH); + cqc->dw3 = (0 << CQ_HOP_NUM_SHIFT) | + (0 << CQ_PAGE_SIZE_SHIFT) | + (0 << CQ_BUF_SIZE_SHIFT) | + (4 << CQ_SQE_SIZE_SHIFT); + cqc->dw6 = 1 << CQ_PHASE_SHIFT | 1 << CQ_FLAG_SHIFT; + + ret = qm_mb(qm, MAILBOX_CMD_CQC, qp->cqc.dma, qp_index, 0, 0); + if (ret) + return ret; + + write_lock(&qm->qps_lock); + qm->qp_array[qp_index] = qp; + init_completion(&qp->completion); + write_unlock(&qm->qps_lock); + + return qp_index; +} +EXPORT_SYMBOL_GPL(hisi_qm_start_qp); + +void hisi_qm_release_qp(struct hisi_qp *qp) +{ + struct qm_info *qm = qp->qm; + struct device *dev = &qm->pdev->dev; + + write_lock(&qm->qps_lock); + qm->qp_array[qp->queue_id] = NULL; + bitmap_clear(qm->qp_bitmap, qp->queue_id, 1); + write_unlock(&qm->qps_lock); + + qm_uninit_q_buffer(dev, &qp->scqe); + kfree(qp); +} +EXPORT_SYMBOL_GPL(hisi_qm_release_qp); + +static void *qm_get_avail_sqe(struct hisi_qp *qp) +{ + struct hisi_acc_qp_status *qp_status = &qp->qp_status; + void *sq_base = QP_SQE_ADDR(qp); + u16 sq_tail = qp_status->sq_tail; + + if (qp_status->is_sq_full == 1) + return NULL; + + return sq_base + sq_tail * qp->qm->sqe_size; +} + +int hisi_qp_send(struct hisi_qp *qp, void *msg) +{ + struct hisi_acc_qp_status *qp_status = &qp->qp_status; + u16 sq_tail = qp_status->sq_tail; + u16 sq_tail_next = (sq_tail + 1) % QM_Q_DEPTH; + unsigned long timeout = 100; + void *sqe = qm_get_avail_sqe(qp); + + if (!sqe) + return -ENOSPC; + + memcpy(sqe, msg, qp->qm->sqe_size); + + qm_db(qp->qm, qp->queue_id, DOORBELL_CMD_SQ, sq_tail_next, 0); + + qp_status->sq_tail = sq_tail_next; + + if (qp_status->sq_tail == qp_status->sq_head) + qp_status->is_sq_full = 1; + + /* wait until job finished */ + wait_for_completion_timeout(&qp->completion, timeout); + + return 0; +} +EXPORT_SYMBOL_GPL(hisi_qp_send); + +int hisi_qm_init(const char *dev_name, struct qm_info *qm) +{ + struct pci_dev *pdev = qm->pdev; + int ret; + + ret = pci_enable_device_mem(pdev); + if (ret < 0) { + dev_err(&pdev->dev, "Can't enable device mem!\n"); + return ret; + } + + ret = pci_request_mem_regions(pdev, dev_name); + if (ret < 0) { + dev_err(&pdev->dev, "Can't request mem regions!\n"); + goto err_with_pcidev; + } + + qm->dev_name = dev_name; + qm->phys_base = pci_resource_start(pdev, 2); + qm->size = pci_resource_len(qm->pdev, 2); + qm->io_base = devm_ioremap(&pdev->dev, qm->phys_base, qm->size); + if (!qm->io_base) { + ret = -EIO; + goto err_with_mem_regions; + } + + dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)); + pci_set_master(pdev); + + ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI); + if (ret < 0) { + dev_err(&pdev->dev, "Enable MSI vectors fail!\n"); + goto err_with_mem_regions; + } + + qm->eq_head = 0; + mutex_init(&qm->mailbox_lock); + rwlock_init(&qm->qps_lock); + + if (qm->ver) + qm->ops = &qm_hw_ops_v1; + + return 0; + +err_with_mem_regions: + pci_release_mem_regions(pdev); +err_with_pcidev: + pci_disable_device(pdev); + + return ret; +} +EXPORT_SYMBOL_GPL(hisi_qm_init); + +void hisi_qm_uninit(struct qm_info *qm) +{ + struct pci_dev *pdev = qm->pdev; + + pci_free_irq_vectors(pdev); + pci_release_mem_regions(pdev); + pci_disable_device(pdev); +} +EXPORT_SYMBOL_GPL(hisi_qm_uninit); + +static irqreturn_t qm_irq(int irq, void *data) +{ + struct qm_info *qm = data; + u32 int_source; + + int_source = qm_get_irq_source(qm); + if (int_source) + return IRQ_WAKE_THREAD; + + dev_err(&qm->pdev->dev, "invalid int source %d\n", int_source); + + return IRQ_HANDLED; +} + +int hisi_qm_start(struct qm_info *qm) +{ + struct pci_dev *pdev = qm->pdev; + struct device *dev = &pdev->dev; + int ret; + + if (qm->pdev->is_physfn) + qm->ops->vft_config(qm, qm->qp_base, qm->qp_num); + + ret = qm_init_q_buffer(dev, sizeof(struct eqc), &qm->eqc); + if (ret) + goto err_out; + + ret = qm_init_q_buffer(dev, sizeof(struct eqe) * QM_Q_DEPTH, &qm->eqe); + if (ret) + goto err_with_eqc; + + QM_EQC(qm)->base_l = lower_32_bits(qm->eqe.dma); + QM_EQC(qm)->base_h = upper_32_bits(qm->eqe.dma); + QM_EQC(qm)->dw3 = 2 << MB_EQC_EQE_SHIFT; + QM_EQC(qm)->dw6 = (QM_Q_DEPTH - 1) | (1 << MB_EQC_PHASE_SHIFT); + ret = qm_mb(qm, MAILBOX_CMD_EQC, qm->eqc.dma, 0, 0, 0); + if (ret) + goto err_with_eqe; + + qm->qp_bitmap = kcalloc(BITS_TO_LONGS(qm->qp_num), sizeof(long), + GFP_KERNEL); + if (!qm->qp_bitmap) + goto err_with_eqe; + + qm->qp_array = kcalloc(qm->qp_num, sizeof(struct hisi_qp *), + GFP_KERNEL); + if (!qm->qp_array) + goto err_with_bitmap; + + /* Init sqc_bt */ + ret = qm_init_bt(qm, dev, sizeof(struct sqc) * qm->qp_num, &qm->sqc, + MAILBOX_CMD_SQC_BT); + if (ret) + goto err_with_qp_array; + + /* Init cqc_bt */ + ret = qm_init_bt(qm, dev, sizeof(struct cqc) * qm->qp_num, &qm->cqc, + MAILBOX_CMD_CQC_BT); + if (ret) + goto err_with_sqc; + + ret = request_threaded_irq(pci_irq_vector(pdev, 0), qm_irq, + qm_irq_thread, IRQF_SHARED, qm->dev_name, + qm); + if (ret) + goto err_with_cqc; + + writel(0x0, QM_ADDR(qm, QM_VF_EQ_INT_MASK)); + + return 0; + +err_with_cqc: + qm_uninit_q_buffer(dev, &qm->cqc); +err_with_sqc: + qm_uninit_q_buffer(dev, &qm->sqc); +err_with_qp_array: + kfree(qm->qp_array); +err_with_bitmap: + kfree(qm->qp_bitmap); +err_with_eqe: + qm_uninit_q_buffer(dev, &qm->eqe); +err_with_eqc: + qm_uninit_q_buffer(dev, &qm->eqc); +err_out: + return ret; +} +EXPORT_SYMBOL_GPL(hisi_qm_start); + +void hisi_qm_stop(struct qm_info *qm) +{ + struct pci_dev *pdev = qm->pdev; + struct device *dev = &pdev->dev; + + free_irq(pci_irq_vector(pdev, 0), qm); + qm_uninit_q_buffer(dev, &qm->cqc); + kfree(qm->qp_array); + kfree(qm->qp_bitmap); + qm_uninit_q_buffer(dev, &qm->eqe); + qm_uninit_q_buffer(dev, &qm->eqc); +} +EXPORT_SYMBOL_GPL(hisi_qm_stop); + +/* put qm into init state, so the acce config become available */ +int hisi_qm_mem_start(struct qm_info *qm) +{ + u32 val; + + qm_writel(qm, 0x1, QM_MEM_START_INIT); + return readl_relaxed_poll_timeout(QM_ADDR(qm, QM_MEM_INIT_DONE), val, + val & BIT(0), 10, 1000); +} +EXPORT_SYMBOL_GPL(hisi_qm_mem_start); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Zhou Wang "); +MODULE_DESCRIPTION("HiSilicon Accelerator queue manager driver"); diff --git a/drivers/crypto/hisilicon/qm.h b/drivers/crypto/hisilicon/qm.h new file mode 100644 index 000000000000..0e81182ac6a8 --- /dev/null +++ b/drivers/crypto/hisilicon/qm.h @@ -0,0 +1,110 @@ +/* SPDX-License-Identifier: GPL-2.0+ */ +#ifndef HISI_ACC_QM_H +#define HISI_ACC_QM_H + +#include +#include +#include +#include +#include + +#define QM_CQE_SIZE 16 +/* default queue depth for sq/cq/eq */ +#define QM_Q_DEPTH 1024 + +/* qm user domain */ +#define QM_ARUSER_M_CFG_1 0x100088 +#define QM_ARUSER_M_CFG_ENABLE 0x100090 +#define QM_AWUSER_M_CFG_1 0x100098 +#define QM_AWUSER_M_CFG_ENABLE 0x1000a0 +#define QM_WUSER_M_CFG_ENABLE 0x1000a8 + +/* qm cache */ +#define QM_CACHE_CTL 0x100050 +#define QM_AXI_M_CFG 0x1000ac +#define QM_AXI_M_CFG_ENABLE 0x1000b0 +#define QM_PEH_AXUSER_CFG 0x1000cc +#define QM_PEH_AXUSER_CFG_ENABLE 0x1000d0 + +#define QP_SQE_ADDR(qp) ((qp)->scqe.addr) + +struct qm_dma_buffer { + int size; + void *addr; + dma_addr_t dma; +}; + +struct qm_info { + int ver; + const char *dev_name; + struct pci_dev *pdev; + + resource_size_t phys_base; + resource_size_t size; + void __iomem *io_base; + + u32 sqe_size; + u32 qp_base; + u32 qp_num; + + struct qm_dma_buffer sqc, cqc, eqc, eqe; + + u32 eq_head; + + rwlock_t qps_lock; + unsigned long *qp_bitmap; + struct hisi_qp **qp_array; + + struct mutex mailbox_lock; + + struct hisi_acc_qm_hw_ops *ops; + +}; +#define QM_ADDR(qm, off) ((qm)->io_base + off) + +struct hisi_acc_qp_status { + u16 sq_tail; + u16 sq_head; + u16 cq_head; + bool cqc_phase; + int is_sq_full; +}; + +struct hisi_qp; + +struct hisi_qp_ops { + int (*fill_sqe)(void *sqe, void *q_parm, void *d_parm); +}; + +struct hisi_qp { + /* sq number in this function */ + u32 queue_id; + u8 alg_type; + u8 req_type; + + struct qm_dma_buffer sqc, cqc; + struct qm_dma_buffer scqe; + + struct hisi_acc_qp_status qp_status; + + struct qm_info *qm; + + /* for crypto sync API */ + struct completion completion; + + struct hisi_qp_ops *hw_ops; + void *qp_ctx; + void (*event_cb)(struct hisi_qp *qp); + void (*req_cb)(struct hisi_qp *qp, void *data); +}; + +extern int hisi_qm_init(const char *dev_name, struct qm_info *qm); +extern void hisi_qm_uninit(struct qm_info *qm); +extern int hisi_qm_start(struct qm_info *qm); +extern void hisi_qm_stop(struct qm_info *qm); +extern int hisi_qm_mem_start(struct qm_info *qm); +extern struct hisi_qp *hisi_qm_create_qp(struct qm_info *qm, u8 alg_type); +extern int hisi_qm_start_qp(struct hisi_qp *qp, unsigned long arg); +extern void hisi_qm_release_qp(struct hisi_qp *qp); +extern int hisi_qp_send(struct hisi_qp *qp, void *msg); +#endif From patchwork Mon Sep 3 00:52:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kenneth Lee X-Patchwork-Id: 145741 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1927388ljw; Sun, 2 Sep 2018 17:52:47 -0700 (PDT) X-Google-Smtp-Source: ANB0VdY+oRtup1m2OkXJ/nK1Lp57wnurgYKJA+EOWZEGRfBEN8Texo9fT0Gef12wWLKXwN2B4rr2 X-Received: by 2002:a62:6104:: with SMTP id v4-v6mr27007687pfb.122.1535935967107; Sun, 02 Sep 2018 17:52:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535935967; cv=none; d=google.com; s=arc-20160816; b=rSiV9y6uLyoPmi5eAEKTZgMxs9wtXTk+eE9fXCp0LWXy6dhAicgCSe6u09s63f0Ohj OfFMc/YCZPe4pxhIFHcUYcjepXAb9hbfhlQ5X9FRFP1m7Q0wkBvPW01NnKERCAnSGo/R 5VEaowni4NXu1ra8oCDdCrkWmKJFRVquaZO5wnrXiSTXgOVJ9qgdUMcbZV3eHPG8Uj28 SQKbVzp/sEeKuXWPPkJ1xxxvzE7Ok3zFb9sj3hDvCHxOi3W9vE7NTb9bn7TPOvNGM9t6 IHrteg6vj9mkURBYId88MPsi4ChHqILkweQ8h42IAbz3a36eaEsViYU3n3dLgC4+8j65 GaXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=GNC0Lr1Mdew6/n5ZNUMu2kkk0uptEGRsrfIP6xFki7M=; b=wJ3SKtnsjpwNXeLyOeXSIBIlYdnDetwlFdBFeQ63hR0RMQhdvtkMtF9t/dyBR5lPOM 3x99fk2ecnH7QtKuT+8rv3scc4FOXX9ZhrthK/1xi67PpKkgzTY5WeT6U05Us60yKNT+ VV/zFETSQIg8xSGKikSq5Cn90Eg2axPaUPCkNzooAym2L4FB8v84T/QeNMgKy3H+fNSy Nvx4Zkc05Ze4iW27BEVCPYIV9d8QpPVn0msJqmhhaBWPeOmcbWSN4GD185TIcaUanRbE WXVEf+U2c2Xzwh/T07lYu+KRg4YvgZPm3pbeuoke1ysNEdhJY/sl6ZVnUGVepI3fK1zg C+4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=e6YssQA2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d37-v6si16679718plb.430.2018.09.02.17.52.46; Sun, 02 Sep 2018 17:52:47 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=e6YssQA2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727257AbeICFKZ (ORCPT + 32 others); Mon, 3 Sep 2018 01:10:25 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:46417 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726157AbeICFKY (ORCPT ); Mon, 3 Sep 2018 01:10:24 -0400 Received: by mail-pf1-f193.google.com with SMTP id u24-v6so7883521pfn.13; Sun, 02 Sep 2018 17:52:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GNC0Lr1Mdew6/n5ZNUMu2kkk0uptEGRsrfIP6xFki7M=; b=e6YssQA2ZchpE/+kteq7fefHVS4vEcSf8lyPF03LE5duLM8iE+AcvTdCqxxzg8oSjr amWzDpqtMNScixzol9b8LiaC8D1uBCH2xe8Lgn06Qr7vLA/hHtwOLtWDblc9f2xSZXdi 8wrPfAc2ok3Gk56fjxjpflbam8Bf2RBjuw8V/Zcp4VqRZFYQP5CHRxOsl7CeYM4l1h6S D741u9VtwCxh79swfPxs4a5sdx8+DNWZLnzQ+h94DYk8xxPhZvD/vpJQDNNUm2EznW8U dZuw+C70A1JcIrgJYpyqXUISzuktBLhoinxTaqeJx3f7GnUDlwHxVNRATijSsCxq0hEM S0vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GNC0Lr1Mdew6/n5ZNUMu2kkk0uptEGRsrfIP6xFki7M=; b=LuPXmUKEiS/OcAS2QKg8je98hpxbD/d1rI0IFVkFbLM1wZAqsWg7zzzhkKDT4DLCqy ptjknF4ZICqHoWYSAnQMUKBb2vU6eZJxK3LxUZiYR1uqbT0odsLjyhoSdr9oQzrjNxSz AoQndXoKhQM+KSUfRoDbbY57F97i5QbTfd/L4I+hGUPdvg9ioJhhESuMx6drcd570l5d txH4Q0TQQvOBBloQuDCFlbVMvPlqyZfo5LHR3jBI6ahEw50gumN3Zfi0SHWa8QR4nKkm bMW6fMzJAjHJsuweQcIm5kC9JALM3hhK4pCyAhf/MSpV/3J/F8kRFOdOm8JEOd6JucIH Xd+A== X-Gm-Message-State: APzg51DDDpLRfeTcXTbpVF71kjnTE1v6gCgpL5g+fglc1l/gOFRHyJdZ PurgM++DU1gv9QsLqWP3iX0= X-Received: by 2002:a62:2119:: with SMTP id h25-v6mr27197959pfh.112.1535935962592; Sun, 02 Sep 2018 17:52:42 -0700 (PDT) Received: from localhost.localdomain ([203.160.89.93]) by smtp.gmail.com with ESMTPSA id w81-v6sm31487064pfk.92.2018.09.02.17.52.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 17:52:42 -0700 (PDT) From: Kenneth Lee To: Jonathan Corbet , Herbert Xu , "David S . Miller" , Joerg Roedel , Alex Williamson , Kenneth Lee , Hao Fang , Zhou Wang , Zaibo Xu , Philippe Ombredanne , Greg Kroah-Hartman , Thomas Gleixner , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, linux-accelerators@lists.ozlabs.org, Lu Baolu , Sanjay Kumar Cc: linuxarm@huawei.com Subject: [PATCH 5/7] crypto: Add Hisilicon Zip driver Date: Mon, 3 Sep 2018 08:52:02 +0800 Message-Id: <20180903005204.26041-6-nek.in.cn@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903005204.26041-1-nek.in.cn@gmail.com> References: <20180903005204.26041-1-nek.in.cn@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Kenneth Lee The Hisilicon ZIP accelerator implements zlib and gzip algorithm support for the software. It uses Hisilicon QM as the interface to the CPU, so it is shown up as a PCIE device to the CPU with a group of queues. This commit provides PCIE driver to the accelerator and register it to the crypto subsystem. Signed-off-by: Kenneth Lee Signed-off-by: Zhou Wang Signed-off-by: Hao Fang --- drivers/crypto/hisilicon/Kconfig | 7 + drivers/crypto/hisilicon/Makefile | 1 + drivers/crypto/hisilicon/zip/Makefile | 2 + drivers/crypto/hisilicon/zip/zip.h | 57 ++++ drivers/crypto/hisilicon/zip/zip_crypto.c | 353 ++++++++++++++++++++++ drivers/crypto/hisilicon/zip/zip_crypto.h | 8 + drivers/crypto/hisilicon/zip/zip_main.c | 195 ++++++++++++ 7 files changed, 623 insertions(+) create mode 100644 drivers/crypto/hisilicon/zip/Makefile create mode 100644 drivers/crypto/hisilicon/zip/zip.h create mode 100644 drivers/crypto/hisilicon/zip/zip_crypto.c create mode 100644 drivers/crypto/hisilicon/zip/zip_crypto.h create mode 100644 drivers/crypto/hisilicon/zip/zip_main.c -- 2.17.1 diff --git a/drivers/crypto/hisilicon/Kconfig b/drivers/crypto/hisilicon/Kconfig index 02a6eef84101..1d155708cd69 100644 --- a/drivers/crypto/hisilicon/Kconfig +++ b/drivers/crypto/hisilicon/Kconfig @@ -20,3 +20,10 @@ config CRYPTO_DEV_HISI_SEC config CRYPTO_DEV_HISI_QM tristate depends on ARM64 && PCI + +config CRYPTO_DEV_HISI_ZIP + tristate "Support for HISI ZIP Driver" + depends on ARM64 && CRYPTO_DEV_HISILICON + select CRYPTO_DEV_HISI_QM + help + Support for HiSilicon HIP08 ZIP Driver diff --git a/drivers/crypto/hisilicon/Makefile b/drivers/crypto/hisilicon/Makefile index 05e9052e0f52..c97c5b27c3cb 100644 --- a/drivers/crypto/hisilicon/Makefile +++ b/drivers/crypto/hisilicon/Makefile @@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CRYPTO_DEV_HISI_SEC) += sec/ obj-$(CONFIG_CRYPTO_DEV_HISI_QM) += qm.o +obj-$(CONFIG_CRYPTO_DEV_HISI_ZIP) += zip/ diff --git a/drivers/crypto/hisilicon/zip/Makefile b/drivers/crypto/hisilicon/zip/Makefile new file mode 100644 index 000000000000..a936f099ee22 --- /dev/null +++ b/drivers/crypto/hisilicon/zip/Makefile @@ -0,0 +1,2 @@ +obj-$(CONFIG_CRYPTO_DEV_HISI_ZIP) += hisi_zip.o +hisi_zip-objs = zip_main.o zip_crypto.o diff --git a/drivers/crypto/hisilicon/zip/zip.h b/drivers/crypto/hisilicon/zip/zip.h new file mode 100644 index 000000000000..87515e158b17 --- /dev/null +++ b/drivers/crypto/hisilicon/zip/zip.h @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: GPL-2.0+ */ +#ifndef HISI_ZIP_H +#define HISI_ZIP_H + +#include +#include "../qm.h" + +#define HZIP_SQE_SIZE 128 +#define HZIP_SQ_SIZE (HZIP_SQE_SIZE * QM_Q_DEPTH) +#define QM_CQ_SIZE (QM_CQE_SIZE * QM_Q_DEPTH) +#define HZIP_PF_DEF_Q_NUM 64 +#define HZIP_PF_DEF_Q_BASE 0 + +struct hisi_zip { + struct qm_info qm; + struct list_head list; + +#ifdef CONFIG_CRYPTO_DEV_HISI_SPIMDEV + struct vfio_spimdev *spimdev; +#endif +}; + +struct hisi_zip_sqe { + __u32 consumed; + __u32 produced; + __u32 comp_data_length; + __u32 dw3; + __u32 input_data_length; + __u32 lba_l; + __u32 lba_h; + __u32 dw7; + __u32 dw8; + __u32 dw9; + __u32 dw10; + __u32 priv_info; + __u32 dw12; + __u32 tag; + __u32 dest_avail_out; + __u32 rsvd0; + __u32 comp_head_addr_l; + __u32 comp_head_addr_h; + __u32 source_addr_l; + __u32 source_addr_h; + __u32 dest_addr_l; + __u32 dest_addr_h; + __u32 stream_ctx_addr_l; + __u32 stream_ctx_addr_h; + __u32 cipher_key1_addr_l; + __u32 cipher_key1_addr_h; + __u32 cipher_key2_addr_l; + __u32 cipher_key2_addr_h; + __u32 rsvd1[4]; +}; + +extern struct list_head hisi_zip_list; + +#endif diff --git a/drivers/crypto/hisilicon/zip/zip_crypto.c b/drivers/crypto/hisilicon/zip/zip_crypto.c new file mode 100644 index 000000000000..0a0e3de8e1d6 --- /dev/null +++ b/drivers/crypto/hisilicon/zip/zip_crypto.c @@ -0,0 +1,353 @@ +/* SPDX-License-Identifier: GPL-2.0+ */ +#include +#include +#include +#include +#include "../qm.h" +#include "zip.h" + +#define INPUT_BUFFER_SIZE (64 * 1024) +#define OUTPUT_BUFFER_SIZE (64 * 1024) + +#define COMP_NAME_TO_TYPE(alg_name) \ + (!strcmp((alg_name), "zlib-deflate") ? 0x02 : \ + !strcmp((alg_name), "gzip") ? 0x03 : 0) \ + +struct hisi_zip_buffer { + u8 *input; + dma_addr_t input_dma; + u8 *output; + dma_addr_t output_dma; +}; + +struct hisi_zip_qp_ctx { + struct hisi_zip_buffer buffer; + struct hisi_qp *qp; + struct hisi_zip_sqe zip_sqe; +}; + +struct hisi_zip_ctx { +#define QPC_COMP 0 +#define QPC_DECOMP 1 + struct hisi_zip_qp_ctx qp_ctx[2]; +}; + +static struct hisi_zip *find_zip_device(int node) +{ + struct hisi_zip *hisi_zip, *ret = NULL; + struct device *dev; + int min_distance = 100; + + list_for_each_entry(hisi_zip, &hisi_zip_list, list) { + dev = &hisi_zip->qm.pdev->dev; + if (node_distance(dev->numa_node, node) < min_distance) { + ret = hisi_zip; + min_distance = node_distance(dev->numa_node, node); + } + } + + return ret; +} + +static void hisi_zip_qp_event_notifier(struct hisi_qp *qp) +{ + complete(&qp->completion); +} + +static int hisi_zip_fill_sqe_v1(void *sqe, void *q_parm, u32 len) +{ + struct hisi_zip_sqe *zip_sqe = (struct hisi_zip_sqe *)sqe; + struct hisi_zip_qp_ctx *qp_ctx = (struct hisi_zip_qp_ctx *)q_parm; + struct hisi_zip_buffer *buffer = &qp_ctx->buffer; + + memset(zip_sqe, 0, sizeof(struct hisi_zip_sqe)); + + zip_sqe->input_data_length = len; + zip_sqe->dw9 = qp_ctx->qp->req_type; + zip_sqe->dest_avail_out = OUTPUT_BUFFER_SIZE; + zip_sqe->source_addr_l = lower_32_bits(buffer->input_dma); + zip_sqe->source_addr_h = upper_32_bits(buffer->input_dma); + zip_sqe->dest_addr_l = lower_32_bits(buffer->output_dma); + zip_sqe->dest_addr_h = upper_32_bits(buffer->output_dma); + + return 0; +} + +/* let's allocate one buffer now, may have problem in async case */ +static int hisi_zip_alloc_qp_buffer(struct hisi_zip_qp_ctx *hisi_zip_qp_ctx) +{ + struct hisi_zip_buffer *buffer = &hisi_zip_qp_ctx->buffer; + struct hisi_qp *qp = hisi_zip_qp_ctx->qp; + struct device *dev = &qp->qm->pdev->dev; + int ret; + + buffer->input = dma_alloc_coherent(dev, INPUT_BUFFER_SIZE, + &buffer->input_dma, GFP_KERNEL); + if (!buffer->input) + return -ENOMEM; + + buffer->output = dma_alloc_coherent(dev, OUTPUT_BUFFER_SIZE, + &buffer->output_dma, GFP_KERNEL); + if (!buffer->output) { + ret = -ENOMEM; + goto err_alloc_output_buffer; + } + + return 0; + +err_alloc_output_buffer: + dma_free_coherent(dev, INPUT_BUFFER_SIZE, buffer->input, + buffer->input_dma); + return ret; +} + +static void hisi_zip_free_qp_buffer(struct hisi_zip_qp_ctx *hisi_zip_qp_ctx) +{ + struct hisi_zip_buffer *buffer = &hisi_zip_qp_ctx->buffer; + struct hisi_qp *qp = hisi_zip_qp_ctx->qp; + struct device *dev = &qp->qm->pdev->dev; + + dma_free_coherent(dev, INPUT_BUFFER_SIZE, buffer->input, + buffer->input_dma); + dma_free_coherent(dev, OUTPUT_BUFFER_SIZE, buffer->output, + buffer->output_dma); +} + +static int hisi_zip_create_qp(struct qm_info *qm, struct hisi_zip_qp_ctx *ctx, + int alg_type, int req_type) +{ + struct hisi_qp *qp; + int ret; + + qp = hisi_qm_create_qp(qm, alg_type); + + if (IS_ERR(qp)) + return PTR_ERR(qp); + + qp->event_cb = hisi_zip_qp_event_notifier; + qp->req_type = req_type; + + qp->qp_ctx = ctx; + ctx->qp = qp; + + ret = hisi_zip_alloc_qp_buffer(ctx); + if (ret) + goto err_with_qp; + + ret = hisi_qm_start_qp(qp, 0); + if (ret < 0) + goto err_with_qp_buffer; + + return 0; +err_with_qp_buffer: + hisi_zip_free_qp_buffer(ctx); +err_with_qp: + hisi_qm_release_qp(qp); + return ret; +} + +static void hisi_zip_release_qp(struct hisi_zip_qp_ctx *ctx) +{ + hisi_qm_release_qp(ctx->qp); + hisi_zip_free_qp_buffer(ctx); +} + +static int hisi_zip_alloc_comp_ctx(struct crypto_tfm *tfm) +{ + struct hisi_zip_ctx *hisi_zip_ctx = crypto_tfm_ctx(tfm); + const char *alg_name = crypto_tfm_alg_name(tfm); + struct hisi_zip *hisi_zip; + struct qm_info *qm; + int ret, i, j; + + u8 req_type = COMP_NAME_TO_TYPE(alg_name); + + /* find the proper zip device */ + hisi_zip = find_zip_device(cpu_to_node(smp_processor_id())); + if (!hisi_zip) { + pr_err("Can not find proper ZIP device!\n"); + return -ENODEV; + } + qm = &hisi_zip->qm; + + for (i = 0; i < 2; i++) { + /* it is just happen that 0 is compress, 1 is decompress on alg_type */ + ret = hisi_zip_create_qp(qm, &hisi_zip_ctx->qp_ctx[i], i, + req_type); + if (ret) + goto err; + } + + return 0; +err: + for (j = i-1; j >= 0; j--) + hisi_zip_release_qp(&hisi_zip_ctx->qp_ctx[j]); + + return ret; +} + +static void hisi_zip_free_comp_ctx(struct crypto_tfm *tfm) +{ + struct hisi_zip_ctx *hisi_zip_ctx = crypto_tfm_ctx(tfm); + int i; + + /* release the qp */ + for (i = 1; i >= 0; i--) + hisi_zip_release_qp(&hisi_zip_ctx->qp_ctx[i]); +} + +static int hisi_zip_copy_data_to_buffer(struct hisi_zip_qp_ctx *qp_ctx, + const u8 *src, unsigned int slen) +{ + struct hisi_zip_buffer *buffer = &qp_ctx->buffer; + + if (slen > INPUT_BUFFER_SIZE) + return -EINVAL; + + memcpy(buffer->input, src, slen); + + return 0; +} + +static struct hisi_zip_sqe *hisi_zip_get_writeback_sqe(struct hisi_qp *qp) +{ + struct hisi_acc_qp_status *qp_status = &qp->qp_status; + struct hisi_zip_sqe *sq_base = QP_SQE_ADDR(qp); + u16 sq_head = qp_status->sq_head; + + return sq_base + sq_head; +} + +static int hisi_zip_copy_data_from_buffer(struct hisi_zip_qp_ctx *qp_ctx, + u8 *dst, unsigned int *dlen) +{ + struct hisi_zip_buffer *buffer = &qp_ctx->buffer; + struct hisi_qp *qp = qp_ctx->qp; + struct hisi_zip_sqe *zip_sqe = hisi_zip_get_writeback_sqe(qp); + u32 status = zip_sqe->dw3 & 0xff; + u16 sq_head; + + if (status != 0) { + pr_err("hisi zip: %s fail!\n", (qp->alg_type == 0) ? + "compression" : "decompression"); + return status; + } + + if (zip_sqe->produced > OUTPUT_BUFFER_SIZE) + return -ENOMEM; + + memcpy(dst, buffer->output, zip_sqe->produced); + *dlen = zip_sqe->produced; + + sq_head = qp->qp_status.sq_head; + if (sq_head == QM_Q_DEPTH - 1) + qp->qp_status.sq_head = 0; + else + qp->qp_status.sq_head++; + + return 0; +} + +static int hisi_zip_compress(struct crypto_tfm *tfm, const u8 *src, + unsigned int slen, u8 *dst, unsigned int *dlen) +{ + struct hisi_zip_ctx *hisi_zip_ctx = crypto_tfm_ctx(tfm); + struct hisi_zip_qp_ctx *qp_ctx = &hisi_zip_ctx->qp_ctx[QPC_COMP]; + struct hisi_qp *qp = qp_ctx->qp; + struct hisi_zip_sqe *zip_sqe = &qp_ctx->zip_sqe; + int ret; + + ret = hisi_zip_copy_data_to_buffer(qp_ctx, src, slen); + if (ret < 0) + return ret; + + hisi_zip_fill_sqe_v1(zip_sqe, qp_ctx, slen); + + /* send command to start the compress job */ + hisi_qp_send(qp, zip_sqe); + + return hisi_zip_copy_data_from_buffer(qp_ctx, dst, dlen); +} + +static int hisi_zip_decompress(struct crypto_tfm *tfm, const u8 *src, + unsigned int slen, u8 *dst, unsigned int *dlen) +{ + struct hisi_zip_ctx *hisi_zip_ctx = crypto_tfm_ctx(tfm); + struct hisi_zip_qp_ctx *qp_ctx = &hisi_zip_ctx->qp_ctx[QPC_DECOMP]; + struct hisi_qp *qp = qp_ctx->qp; + struct hisi_zip_sqe *zip_sqe = &qp_ctx->zip_sqe; + int ret; + + ret = hisi_zip_copy_data_to_buffer(qp_ctx, src, slen); + if (ret < 0) + return ret; + + hisi_zip_fill_sqe_v1(zip_sqe, qp_ctx, slen); + + /* send command to start the decompress job */ + hisi_qp_send(qp, zip_sqe); + + return hisi_zip_copy_data_from_buffer(qp_ctx, dst, dlen); +} + +static struct crypto_alg hisi_zip_zlib = { + .cra_name = "zlib-deflate", + .cra_flags = CRYPTO_ALG_TYPE_COMPRESS, + .cra_ctxsize = sizeof(struct hisi_zip_ctx), + .cra_priority = 300, + .cra_module = THIS_MODULE, + .cra_init = hisi_zip_alloc_comp_ctx, + .cra_exit = hisi_zip_free_comp_ctx, + .cra_u = { + .compress = { + .coa_compress = hisi_zip_compress, + .coa_decompress = hisi_zip_decompress + } + } +}; + +static struct crypto_alg hisi_zip_gzip = { + .cra_name = "gzip", + .cra_flags = CRYPTO_ALG_TYPE_COMPRESS, + .cra_ctxsize = sizeof(struct hisi_zip_ctx), + .cra_priority = 300, + .cra_module = THIS_MODULE, + .cra_init = hisi_zip_alloc_comp_ctx, + .cra_exit = hisi_zip_free_comp_ctx, + .cra_u = { + .compress = { + .coa_compress = hisi_zip_compress, + .coa_decompress = hisi_zip_decompress + } + } +}; + +int hisi_zip_register_to_crypto(void) +{ + int ret; + + ret = crypto_register_alg(&hisi_zip_zlib); + if (ret < 0) { + pr_err("Zlib algorithm registration failed\n"); + return ret; + } + + ret = crypto_register_alg(&hisi_zip_gzip); + if (ret < 0) { + pr_err("Gzip algorithm registration failed\n"); + goto err_unregister_zlib; + } + + return 0; + +err_unregister_zlib: + crypto_unregister_alg(&hisi_zip_zlib); + + return ret; +} + +void hisi_zip_unregister_from_crypto(void) +{ + crypto_unregister_alg(&hisi_zip_zlib); + crypto_unregister_alg(&hisi_zip_gzip); +} diff --git a/drivers/crypto/hisilicon/zip/zip_crypto.h b/drivers/crypto/hisilicon/zip/zip_crypto.h new file mode 100644 index 000000000000..84eefd74c9c4 --- /dev/null +++ b/drivers/crypto/hisilicon/zip/zip_crypto.h @@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0+ */ +#ifndef HISI_ZIP_CRYPTO_H +#define HISI_ZIP_CRYPTO_H + +int hisi_zip_register_to_crypto(void); +void hisi_zip_unregister_from_crypto(void); + +#endif diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c new file mode 100644 index 000000000000..cad4c97f4826 --- /dev/null +++ b/drivers/crypto/hisilicon/zip/zip_main.c @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include "zip.h" +#include "zip_crypto.h" + +#define HZIP_VF_NUM 63 +#define HZIP_QUEUE_NUM_V1 4096 +#define HZIP_QUEUE_NUM_V2 1024 + +#define HZIP_FSM_MAX_CNT 0x301008 + +#define HZIP_PORT_ARCA_CHE_0 0x301040 +#define HZIP_PORT_ARCA_CHE_1 0x301044 +#define HZIP_PORT_AWCA_CHE_0 0x301060 +#define HZIP_PORT_AWCA_CHE_1 0x301064 + +#define HZIP_BD_RUSER_32_63 0x301110 +#define HZIP_SGL_RUSER_32_63 0x30111c +#define HZIP_DATA_RUSER_32_63 0x301128 +#define HZIP_DATA_WUSER_32_63 0x301134 +#define HZIP_BD_WUSER_32_63 0x301140 + +LIST_HEAD(hisi_zip_list); +DEFINE_MUTEX(hisi_zip_list_lock); + +static const char hisi_zip_name[] = "hisi_zip"; + +static const struct pci_device_id hisi_zip_dev_ids[] = { + { PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, 0xa250) }, + { 0, } +}; + +static inline void hisi_zip_add_to_list(struct hisi_zip *hisi_zip) +{ + mutex_lock(&hisi_zip_list_lock); + list_add_tail(&hisi_zip->list, &hisi_zip_list); + mutex_unlock(&hisi_zip_list_lock); +} + +static inline void hisi_zip_remove_from_list(struct hisi_zip *hisi_zip) +{ + mutex_lock(&hisi_zip_list_lock); + list_del(&hisi_zip->list); + mutex_unlock(&hisi_zip_list_lock); +} + +static void hisi_zip_set_user_domain_and_cache(struct hisi_zip *hisi_zip) +{ + u32 val; + + /* qm user domain */ + writel(0x40001070, hisi_zip->qm.io_base + QM_ARUSER_M_CFG_1); + writel(0xfffffffe, hisi_zip->qm.io_base + QM_ARUSER_M_CFG_ENABLE); + writel(0x40001070, hisi_zip->qm.io_base + QM_AWUSER_M_CFG_1); + writel(0xfffffffe, hisi_zip->qm.io_base + QM_AWUSER_M_CFG_ENABLE); + writel(0xffffffff, hisi_zip->qm.io_base + QM_WUSER_M_CFG_ENABLE); + writel(0x4893, hisi_zip->qm.io_base + QM_CACHE_CTL); + + val = readl(hisi_zip->qm.io_base + QM_PEH_AXUSER_CFG); + val |= (1 << 11); + writel(val, hisi_zip->qm.io_base + QM_PEH_AXUSER_CFG); + + /* qm cache */ + writel(0xffff, hisi_zip->qm.io_base + QM_AXI_M_CFG); + writel(0xffffffff, hisi_zip->qm.io_base + QM_AXI_M_CFG_ENABLE); + writel(0xffffffff, hisi_zip->qm.io_base + QM_PEH_AXUSER_CFG_ENABLE); + + /* cache */ + writel(0xffffffff, hisi_zip->qm.io_base + HZIP_PORT_ARCA_CHE_0); + writel(0xffffffff, hisi_zip->qm.io_base + HZIP_PORT_ARCA_CHE_1); + writel(0xffffffff, hisi_zip->qm.io_base + HZIP_PORT_AWCA_CHE_0); + writel(0xffffffff, hisi_zip->qm.io_base + HZIP_PORT_AWCA_CHE_1); + /* user domain configurations */ + writel(0x40001070, hisi_zip->qm.io_base + HZIP_BD_RUSER_32_63); + writel(0x40001070, hisi_zip->qm.io_base + HZIP_SGL_RUSER_32_63); +#ifdef CONFIG_IOMMU_SVA + writel(0x40001071, hisi_zip->qm.io_base + HZIP_DATA_RUSER_32_63); + writel(0x40001071, hisi_zip->qm.io_base + HZIP_DATA_WUSER_32_63); +#else + writel(0x40001070, hisi_zip->qm.io_base + HZIP_DATA_RUSER_32_63); + writel(0x40001070, hisi_zip->qm.io_base + HZIP_DATA_WUSER_32_63); +#endif + writel(0x40001070, hisi_zip->qm.io_base + HZIP_BD_WUSER_32_63); + + /* fsm count */ + writel(0xfffffff, hisi_zip->qm.io_base + HZIP_FSM_MAX_CNT); + + /* clock gating, core, decompress verify enable */ + writel(0x10005, hisi_zip->qm.io_base + 0x301004); +} + +static int hisi_zip_probe(struct pci_dev *pdev, const struct pci_device_id *id) +{ + struct hisi_zip *hisi_zip; + struct qm_info *qm; + int ret; + u8 rev_id; + + hisi_zip = devm_kzalloc(&pdev->dev, sizeof(*hisi_zip), GFP_KERNEL); + if (!hisi_zip) + return -ENOMEM; + hisi_zip_add_to_list(hisi_zip); + pci_set_drvdata(pdev, hisi_zip); + + qm = &hisi_zip->qm; + qm->pdev = pdev; + + pci_read_config_byte(pdev, PCI_REVISION_ID, &rev_id); + if (rev_id == 0x20) + qm->ver = 1; + qm->sqe_size = HZIP_SQE_SIZE; + ret = hisi_qm_init(hisi_zip_name, qm); + if (ret) + goto err_with_hisi_zip; + + if (pdev->is_physfn) { + ret = hisi_qm_mem_start(qm); + if (ret) + goto err_with_qm_init; + + hisi_zip_set_user_domain_and_cache(hisi_zip); + + qm->qp_base = HZIP_PF_DEF_Q_BASE; + qm->qp_num = HZIP_PF_DEF_Q_NUM; + } + + ret = hisi_qm_start(qm); + if (ret) + goto err_with_qm_init; + + return 0; + +err_with_qm_init: + hisi_qm_uninit(qm); +err_with_hisi_zip: + kfree(hisi_zip); + return ret; +} + +static void hisi_zip_remove(struct pci_dev *pdev) +{ + struct hisi_zip *hisi_zip = pci_get_drvdata(pdev); + struct qm_info *qm = &hisi_zip->qm; + + hisi_qm_stop(qm); + hisi_qm_uninit(qm); + hisi_zip_remove_from_list(hisi_zip); + kfree(hisi_zip); +} + +static struct pci_driver hisi_zip_pci_driver = { + .name = "hisi_zip", + .id_table = hisi_zip_dev_ids, + .probe = hisi_zip_probe, + .remove = hisi_zip_remove, +}; + +static int __init hisi_zip_init(void) +{ + int ret; + + ret = pci_register_driver(&hisi_zip_pci_driver); + if (ret < 0) { + pr_err("zip: can't register hisi zip driver.\n"); + return ret; + } + + ret = hisi_zip_register_to_crypto(); + if (ret < 0) { + pr_err("zip: can't register hisi zip to crypto.\n"); + pci_unregister_driver(&hisi_zip_pci_driver); + return ret; + } + + return 0; +} + +static void __exit hisi_zip_exit(void) +{ + hisi_zip_unregister_from_crypto(); + pci_unregister_driver(&hisi_zip_pci_driver); +} + +module_init(hisi_zip_init); +module_exit(hisi_zip_exit); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Zhou Wang "); +MODULE_DESCRIPTION("Driver for HiSilicon ZIP accelerator"); +MODULE_DEVICE_TABLE(pci, hisi_zip_dev_ids); From patchwork Mon Sep 3 00:52:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kenneth Lee X-Patchwork-Id: 145742 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1927426ljw; Sun, 2 Sep 2018 17:52:53 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbqBgq757YzQpriFiRY1Vo6iUXzhZND+agGxtxqmj7dXb02maxT+zNhw+kZGNjV93/647lt X-Received: by 2002:a63:4306:: with SMTP id q6-v6mr22799368pga.181.1535935973192; Sun, 02 Sep 2018 17:52:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535935973; cv=none; d=google.com; s=arc-20160816; b=g5/QlOSKBRHbkhR1UCM1QckaxkPFR2+oSSLD/wpioib5SO8MZMEa9+OlPW+qAR4rIl fdckiFMg7d8CISBlZTDkE7IiiQ+gksUR+kMyMuKSDDX2Gyn1JDakDRe39Pd7LhE43HiI Z9ffMSe09VOnRyh3GJ2ObUgXYORol0DHqTBXIj9YFCWdswdgMUcO14Jd/RAq0Z9xUHQq 90LZb09cldaWvYrYp2ralRHve0dq7SKARHGvyH/lUJCbK0xEXARhep7jKT3FLPkib6a3 CdaT1gVK0sqP3EuGuUn5y+dWkVwe1W26qnVOwi8jRB3GI8WmxRMzEiDm+EyysgT0GXtu QRMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=SUwLjjYE2/0VEcbHs3skl0isNdI/P0eG6H27U43Yjdg=; b=rxSm8XyaeqY2APCzm71Spo2WVBKlRDyIPdT8/OHENbXD+qtmB37kgh9ss6NLW9mU/w kQAp/229QLyjNFiMOGtBhqEgVLJTOLVtOqbM7k2k7BsZqMwR1hWISsyZY1f2w2tWIwlY KOLDmOYQWB6cOCFUs/nUzCU4z48b9gFq905RJFWfY7OESdMnL7YjvnDg+S7xVnsYlSMi PNdmq8igmsgIi4nuzTrcdX37A71HDnPAZksMWLsEZvAkay16Xa1tswdCAk7OJsKZBCip Sm48jdQUBa2rJzGBYEGcob2G7Bpd2OxWKCSrIHzPrBugXiCZTRqoxBdO3VRW20SHcFCm GqEw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=rlcmjZt9; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d6-v6si16145729pgi.506.2018.09.02.17.52.52; Sun, 02 Sep 2018 17:52:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=rlcmjZt9; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727310AbeICFK3 (ORCPT + 32 others); Mon, 3 Sep 2018 01:10:29 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:46420 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727267AbeICFK3 (ORCPT ); Mon, 3 Sep 2018 01:10:29 -0400 Received: by mail-pf1-f193.google.com with SMTP id u24-v6so7883585pfn.13; Sun, 02 Sep 2018 17:52:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SUwLjjYE2/0VEcbHs3skl0isNdI/P0eG6H27U43Yjdg=; b=rlcmjZt95hNyVePJLRTNF4DCtvHDtmU47TST5iM0yMps9AFppoXfc1l2uz+XeEDDtw mTJ8VchUmkW+Jb+8MammuEhMMsKE/wn38MsviyN+wXWmPye93Gmw9hHa3TSfF6FvdqnG lQuTv++4wM1+TMMbW7foXN4V9E0MY/7io5xRsnB9+4pKT/Fq6ZvimKzA1onXy7cfwcK5 SsuovENnzi5WNKeMesLBVIbcAo92gIW/EbsgKVZ/osEPqX9b/jxysNwBRCXfxhAl3/EZ 0NLiqgX6RxRzAjXD/e+GYoGxLbpeAynjSFj0sNIdm5l6833a2AMsp469dpa7VJS3jwUK NVpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SUwLjjYE2/0VEcbHs3skl0isNdI/P0eG6H27U43Yjdg=; b=JMSTFq3dxBcukCDbg24Y2WEu1JVWSmcQy5JD4MOOP1pZ317TJjCUwkDWGNbIfgL4BW VBklZBsYH6TPkmBSFYK1A3GK9NLyo4Rmvj2bYWS8RrV53EVvvMWsLqZjD3+wgwvUSTQF WCAJqyDggJNQD3qLLqNyqwsVLhQjIOsuizZkDcGndbVdldj+6VwVeviBTfVNEDQk5A5W InYFxrulCcre3WEfOvldHHbndEamSCqsua2cd1QTUZiuwJn2bbyDND/kgz91yBmnMW44 QNhPe9ujLa5rqb4ZwY0zRqseXHSTAnyx/4bHZsqZUFkVovw1HSGcZ8bqS5lJSp07FPm6 oxOQ== X-Gm-Message-State: APzg51AwUPUfgT6bM/xjTHtxQOdgAVFk7BumO8cOt2uyA94yMIi/ZEC2 zuqbgHT82q51PRUQS8xw8x8= X-Received: by 2002:aa7:83cd:: with SMTP id j13-v6mr26661343pfn.236.1535935968041; Sun, 02 Sep 2018 17:52:48 -0700 (PDT) Received: from localhost.localdomain ([203.160.89.93]) by smtp.gmail.com with ESMTPSA id w81-v6sm31487064pfk.92.2018.09.02.17.52.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 17:52:47 -0700 (PDT) From: Kenneth Lee To: Jonathan Corbet , Herbert Xu , "David S . Miller" , Joerg Roedel , Alex Williamson , Kenneth Lee , Hao Fang , Zhou Wang , Zaibo Xu , Philippe Ombredanne , Greg Kroah-Hartman , Thomas Gleixner , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, linux-accelerators@lists.ozlabs.org, Lu Baolu , Sanjay Kumar Cc: linuxarm@huawei.com Subject: [PATCH 6/7] crypto: add sdmdev support to Hisilicon QM Date: Mon, 3 Sep 2018 08:52:03 +0800 Message-Id: <20180903005204.26041-7-nek.in.cn@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903005204.26041-1-nek.in.cn@gmail.com> References: <20180903005204.26041-1-nek.in.cn@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Kenneth Lee This commit add spimdev support to the Hislicon QM driver, any accelerator that use QM can expose its queues to the user space. Signed-off-by: Kenneth Lee Signed-off-by: Zhou Wang Signed-off-by: Hao Fang Signed-off-by: Zaibo Xu --- drivers/crypto/hisilicon/Kconfig | 10 ++ drivers/crypto/hisilicon/qm.c | 159 +++++++++++++++++++++++++++++++ drivers/crypto/hisilicon/qm.h | 12 +++ 3 files changed, 181 insertions(+) -- 2.17.1 diff --git a/drivers/crypto/hisilicon/Kconfig b/drivers/crypto/hisilicon/Kconfig index 1d155708cd69..b85fab48fdab 100644 --- a/drivers/crypto/hisilicon/Kconfig +++ b/drivers/crypto/hisilicon/Kconfig @@ -17,6 +17,16 @@ config CRYPTO_DEV_HISI_SEC To compile this as a module, choose M here: the module will be called hisi_sec. +config CRYPTO_DEV_HISI_SDMDEV + bool "Enable SDMDEV interface" + depends on CRYPTO_DEV_HISILICON + select VFIO_SDMDEV + help + Enable this enable the SDMDEV, "shared IOMMU Domain Mediated Device" + interface for all Hisilicon accelerators if they can. The SDMDEV + enable the WarpDrive user space accelerator driver to access the + hardware function directly. + config CRYPTO_DEV_HISI_QM tristate depends on ARM64 && PCI diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c index ea618b4d0929..c94e138e9b99 100644 --- a/drivers/crypto/hisilicon/qm.c +++ b/drivers/crypto/hisilicon/qm.c @@ -639,6 +639,155 @@ int hisi_qp_send(struct hisi_qp *qp, void *msg) } EXPORT_SYMBOL_GPL(hisi_qp_send); +#ifdef CONFIG_CRYPTO_DEV_HISI_SDMDEV +/* mdev->supported_type_groups */ +static struct attribute *hisi_qm_type_attrs[] = { + VFIO_SDMDEV_DEFAULT_MDEV_TYPE_ATTRS, + NULL, +}; +static struct attribute_group hisi_qm_type_group = { + .attrs = hisi_qm_type_attrs, +}; +static struct attribute_group *mdev_type_groups[] = { + &hisi_qm_type_group, + NULL, +}; + +static void qm_qp_event_notifier(struct hisi_qp *qp) +{ + vfio_sdmdev_wake_up(qp->sdmdev_q); +} + +static int hisi_qm_get_queue(struct vfio_sdmdev *sdmdev, + struct vfio_sdmdev_queue **q) +{ + struct qm_info *qm = sdmdev->priv; + struct hisi_qp *qp = NULL; + struct vfio_sdmdev_queue *wd_q; + u8 alg_type = 0; /* fix me here */ + int ret; + + qp = hisi_qm_create_qp(qm, alg_type); + if (IS_ERR(qp)) + return PTR_ERR(qp); + + wd_q = kzalloc(sizeof(struct vfio_sdmdev_queue), GFP_KERNEL); + if (!wd_q) { + ret = -ENOMEM; + goto err_with_qp; + } + + wd_q->priv = qp; + wd_q->sdmdev = sdmdev; + *q = wd_q; + qp->sdmdev_q = wd_q; + qp->event_cb = qm_qp_event_notifier; + + return 0; + +err_with_qp: + hisi_qm_release_qp(qp); + return ret; +} + +void hisi_qm_put_queue(struct vfio_sdmdev_queue *q) +{ + struct hisi_qp *qp = q->priv; + + hisi_qm_release_qp(qp); + kfree(q); +} + +/* map sq/cq/doorbell to user space */ +static int hisi_qm_mmap(struct vfio_sdmdev_queue *q, + struct vm_area_struct *vma) +{ + struct hisi_qp *qp = (struct hisi_qp *)q->priv; + struct qm_info *qm = qp->qm; + struct device *dev = &qm->pdev->dev; + size_t sz = vma->vm_end - vma->vm_start; + u8 region; + + vma->vm_flags |= (VM_IO | VM_LOCKED | VM_DONTEXPAND | VM_DONTDUMP); + region = _VFIO_SDMDEV_REGION(vma->vm_pgoff); + + switch (region) { + case 0: + if (sz > PAGE_SIZE) + return -EINVAL; + /* + * Warning: This is not safe as multiple queues use the same + * doorbell, v1 hardware interface problem. v2 will fix it + */ + return remap_pfn_range(vma, vma->vm_start, + qm->phys_base >> PAGE_SHIFT, + sz, pgprot_noncached(vma->vm_page_prot)); + case 1: + vma->vm_pgoff = 0; + if (sz > qp->scqe.size) + return -EINVAL; + + return dma_mmap_coherent(dev, vma, qp->scqe.addr, qp->scqe.dma, + sz); + + default: + return -EINVAL; + } +} + +static int hisi_qm_start_queue(struct vfio_sdmdev_queue *q) +{ + struct hisi_qp *qp = q->priv; + +#ifdef CONFIG_IOMMU_SVA + return hisi_qm_start_qp(qp, q->pasid); +#else + return hisi_qm_start_qp(qp, 0); +#endif +} + +static void hisi_qm_stop_queue(struct vfio_sdmdev_queue *q) +{ + /* need to stop hardware, but can not support in v1 */ +} + +static const struct vfio_sdmdev_ops qm_ops = { + .get_queue = hisi_qm_get_queue, + .put_queue = hisi_qm_put_queue, + .start_queue = hisi_qm_start_queue, + .stop_queue = hisi_qm_stop_queue, + .mmap = hisi_qm_mmap, +}; + +static int qm_register_sdmdev(struct qm_info *qm) +{ + struct pci_dev *pdev = qm->pdev; + struct vfio_sdmdev *sdmdev = &qm->sdmdev; + + sdmdev->iommu_type = VFIO_TYPE1_IOMMU; + +#ifdef CONFIG_IOMMU_SVA + sdmdev->dma_flag = VFIO_SDMDEV_DMA_MULTI_PROC_MAP; +#else + sdmdev->dma_flag = VFIO_SDMDEV_DMA_SINGLE_PROC_MAP; +#endif + + sdmdev->name = qm->dev_name; + sdmdev->dev = &pdev->dev; + sdmdev->is_vf = pdev->is_virtfn; + sdmdev->priv = qm; + sdmdev->api_ver = "hisi_qm_v1"; + sdmdev->flags = 0; + + sdmdev->mdev_fops.mdev_attr_groups = qm->mdev_dev_groups; + hisi_qm_type_group.name = qm->dev_name; + sdmdev->mdev_fops.supported_type_groups = mdev_type_groups; + sdmdev->ops = &qm_ops; + + return vfio_sdmdev_register(sdmdev); +} +#endif + int hisi_qm_init(const char *dev_name, struct qm_info *qm) { struct pci_dev *pdev = qm->pdev; @@ -769,6 +918,12 @@ int hisi_qm_start(struct qm_info *qm) if (ret) goto err_with_cqc; +#ifdef CONFIG_CRYPTO_DEV_HISI_SDMDEV + ret = qm_register_sdmdev(qm); + if (ret) + goto err_with_cqc; +#endif + writel(0x0, QM_ADDR(qm, QM_VF_EQ_INT_MASK)); return 0; @@ -795,6 +950,10 @@ void hisi_qm_stop(struct qm_info *qm) struct pci_dev *pdev = qm->pdev; struct device *dev = &pdev->dev; +#ifdef CONFIG_CRYPTO_DEV_HISI_SDMDEV + vfio_sdmdev_unregister(&qm->sdmdev); +#endif + free_irq(pci_irq_vector(pdev, 0), qm); qm_uninit_q_buffer(dev, &qm->cqc); kfree(qm->qp_array); diff --git a/drivers/crypto/hisilicon/qm.h b/drivers/crypto/hisilicon/qm.h index 0e81182ac6a8..0d24e0bd42e8 100644 --- a/drivers/crypto/hisilicon/qm.h +++ b/drivers/crypto/hisilicon/qm.h @@ -8,6 +8,10 @@ #include #include +#ifdef CONFIG_CRYPTO_DEV_HISI_SDMDEV +#include +#endif + #define QM_CQE_SIZE 16 /* default queue depth for sq/cq/eq */ #define QM_Q_DEPTH 1024 @@ -59,6 +63,10 @@ struct qm_info { struct hisi_acc_qm_hw_ops *ops; +#ifdef CONFIG_CRYPTO_DEV_HISI_SDMDEV + struct vfio_sdmdev sdmdev; + const struct attribute_group **mdev_dev_groups; +#endif }; #define QM_ADDR(qm, off) ((qm)->io_base + off) @@ -89,6 +97,10 @@ struct hisi_qp { struct qm_info *qm; +#ifdef CONFIG_CRYPTO_DEV_HISI_SDMDEV + struct vfio_sdmdev_queue *sdmdev_q; +#endif + /* for crypto sync API */ struct completion completion; From patchwork Mon Sep 3 00:52:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kenneth Lee X-Patchwork-Id: 145743 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp1927467ljw; Sun, 2 Sep 2018 17:52:58 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZDHfQ9926H07aaU/rHIcYYs51v6A7qkSjWBGPyV5xW+Vf5Ub6BiUuBHxPQNrT0iWofpFXx X-Received: by 2002:a63:ed4f:: with SMTP id m15-v6mr23948132pgk.147.1535935978709; Sun, 02 Sep 2018 17:52:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535935978; cv=none; d=google.com; s=arc-20160816; b=EwV1x5Q/+2Ier7THcivl73jzdFD7I/JUAlTl7WUqiDayDdBlkQsX3GHzNc5K2EHnF4 hDY6ri1WC6e0EEfwWRZeQjOQgH5gNkcjxuekV6QyJB9t3h4Jknyp5oXIPEYFR/0xtbTO H10P52QkQNsx8iMQu1qTvyTq94dR45G12w0vHPcWWtUhYKa3fEq66SU02QSP8QYe7bnO m02vvarq0xgadUO0F1Ezz/8vsnbI/BT2hyd36Ys8lf/Vf78FwsaoJ1jKcYgj8zH8iN18 IcE3zrCQjAiWBaU1epFNu2qWTU3ceAbMAhxg/RWvDHAOwU2qSf5+3sSoIoVwBQ+DIUqv 0pJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=CfxNC5HLCvYUhuZr0dgn4PtrshptuTaTbXFu/dmpj1Y=; b=v3438LJXNYvQ10S/Fd+Mh/HP0qiKnJNasrg7pMgBTzoZYnZHjTlMFOJmENe5TRFAqt L0jJpp4FJINoZ1kWwdDgsnGr+KJn0JqWIqT2OqbwuUDQ4e5Cg4Ykn7/JhzftMtr7ckcM 2oLb78Zejm4ngNvWdyZ7RPxXQ1z0Guf3uoRO5Mn8OOCFWyZYllNe5em4pnll/3TyGAx4 KZpy9D6Oe07pkmEypY4EsLxXxv/rv7TL6VhKKVJpWKo0M0A7P1gLFyaoTUhk74VtMQN/ YghOP6nLoNXLbKIxTghjsxGMVu490U88qO1WL+9XihBUku2eYUaeHv5mOIo5JxmgWP7C LeLA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=KI6iBz0+; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 69-v6si17793609plc.388.2018.09.02.17.52.58; Sun, 02 Sep 2018 17:52:58 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=KI6iBz0+; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727361AbeICFKg (ORCPT + 32 others); Mon, 3 Sep 2018 01:10:36 -0400 Received: from mail-pg1-f196.google.com ([209.85.215.196]:44054 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726160AbeICFKf (ORCPT ); Mon, 3 Sep 2018 01:10:35 -0400 Received: by mail-pg1-f196.google.com with SMTP id r1-v6so7789993pgp.11; Sun, 02 Sep 2018 17:52:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CfxNC5HLCvYUhuZr0dgn4PtrshptuTaTbXFu/dmpj1Y=; b=KI6iBz0+o5HuLPHHYUl7PZNEoiiPY4fIrJDa6aRYWEEe1BQp7OlccyjlT3zggyQA7T j4aaDky00qLrOd3MFbq0rlu+9j8+++JQS/gRzRw1lv63PAQOCtGBuyspJxZ4+4NHdOe5 60G/HQmUkZxKRO1s6QRIbQx5ZxG2cPDvTnsFZz8+ZcWbUMSmjWKQ26h2u56p1LEBHhn5 Wz6/GNXREkI8L2y/LX5Aw90tscbN5SNZ8A6jjj5UFrqVaB1OS5e20CWjgNJSyW9oXMcA jJq/vxvwuSUETfln3Ub0UdYTIyscjnlMI/le0nBCrxbXDDJrWK1ppUxvmsI4pjhcWpjd cBNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=CfxNC5HLCvYUhuZr0dgn4PtrshptuTaTbXFu/dmpj1Y=; b=taXajiPiPeofYDXasK/+HIlyf7ShNkZt7vETRhOdalLCPQJ3lijS7V+fl+82TOLLeC QbHnjlCb8TP/AdqCIq5YplyCcDjBOb77puSKWG++BfQOKstsCp/Q+r3Gsh4IjY8oYhjl Mi6KWdLDjGJOoYHqqnZEKcJWC7AulEBRT0Gvk7LU9o2pY4pwwj+medqkFt1NVQILlPA3 ArwYSG6IZF3vuMLFSnA6ZkEK4ADAp03vEMzsAfcilKcnxf8QfcVTHYrvQSPE2QUVrUUw P26Sq3eiRDj1sN4AWySPbccf0DZngNYjdxRkUtL5EMV3SXgyHH7srMCF5UZMf1ci9oAy rBwQ== X-Gm-Message-State: APzg51CkOi+I3ZhkKpPZPTM6bjLP6hWTAUlhMpsdJ+PyWEiW5pzde8O5 Kfs9vYe+2rgRaGMakUN10kg= X-Received: by 2002:aa7:818f:: with SMTP id g15-v6mr26717074pfi.71.1535935973545; Sun, 02 Sep 2018 17:52:53 -0700 (PDT) Received: from localhost.localdomain ([203.160.89.93]) by smtp.gmail.com with ESMTPSA id w81-v6sm31487064pfk.92.2018.09.02.17.52.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 17:52:53 -0700 (PDT) From: Kenneth Lee To: Jonathan Corbet , Herbert Xu , "David S . Miller" , Joerg Roedel , Alex Williamson , Kenneth Lee , Hao Fang , Zhou Wang , Zaibo Xu , Philippe Ombredanne , Greg Kroah-Hartman , Thomas Gleixner , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, linux-accelerators@lists.ozlabs.org, Lu Baolu , Sanjay Kumar Cc: linuxarm@huawei.com Subject: [PATCH 7/7] vfio/sdmdev: add user sample Date: Mon, 3 Sep 2018 08:52:04 +0800 Message-Id: <20180903005204.26041-8-nek.in.cn@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903005204.26041-1-nek.in.cn@gmail.com> References: <20180903005204.26041-1-nek.in.cn@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Kenneth Lee This is the sample code to demostrate how WrapDrive user application should be. It contains: 1. wd.[ch], the common library to provide WrapDrive interface. 2. drv/*, the user driver to access the hardware upon spimdev 3. test/*, the test application to use WrapDrive interface to access the hardware queue(s) of the accelerator. The Hisilicon HIP08 ZIP accelerator is used in this sample. Signed-off-by: Zaibo Xu Signed-off-by: Kenneth Lee Signed-off-by: Hao Fang Signed-off-by: Zhou Wang --- samples/warpdrive/AUTHORS | 2 + samples/warpdrive/ChangeLog | 1 + samples/warpdrive/Makefile.am | 9 + samples/warpdrive/NEWS | 1 + samples/warpdrive/README | 32 +++ samples/warpdrive/autogen.sh | 3 + samples/warpdrive/cleanup.sh | 13 ++ samples/warpdrive/configure.ac | 52 +++++ samples/warpdrive/drv/hisi_qm_udrv.c | 223 ++++++++++++++++++ samples/warpdrive/drv/hisi_qm_udrv.h | 53 +++++ samples/warpdrive/test/Makefile.am | 7 + samples/warpdrive/test/comp_hw.h | 23 ++ samples/warpdrive/test/test_hisi_zip.c | 206 +++++++++++++++++ samples/warpdrive/wd.c | 309 +++++++++++++++++++++++++ samples/warpdrive/wd.h | 154 ++++++++++++ samples/warpdrive/wd_adapter.c | 74 ++++++ samples/warpdrive/wd_adapter.h | 43 ++++ 17 files changed, 1205 insertions(+) create mode 100644 samples/warpdrive/AUTHORS create mode 100644 samples/warpdrive/ChangeLog create mode 100644 samples/warpdrive/Makefile.am create mode 100644 samples/warpdrive/NEWS create mode 100644 samples/warpdrive/README create mode 100755 samples/warpdrive/autogen.sh create mode 100755 samples/warpdrive/cleanup.sh create mode 100644 samples/warpdrive/configure.ac create mode 100644 samples/warpdrive/drv/hisi_qm_udrv.c create mode 100644 samples/warpdrive/drv/hisi_qm_udrv.h create mode 100644 samples/warpdrive/test/Makefile.am create mode 100644 samples/warpdrive/test/comp_hw.h create mode 100644 samples/warpdrive/test/test_hisi_zip.c create mode 100644 samples/warpdrive/wd.c create mode 100644 samples/warpdrive/wd.h create mode 100644 samples/warpdrive/wd_adapter.c create mode 100644 samples/warpdrive/wd_adapter.h -- 2.17.1 diff --git a/samples/warpdrive/AUTHORS b/samples/warpdrive/AUTHORS new file mode 100644 index 000000000000..fe7dc2413b0d --- /dev/null +++ b/samples/warpdrive/AUTHORS @@ -0,0 +1,2 @@ +Kenneth Lee +Zaibo Xu diff --git a/samples/warpdrive/ChangeLog b/samples/warpdrive/ChangeLog new file mode 100644 index 000000000000..b1b716105590 --- /dev/null +++ b/samples/warpdrive/ChangeLog @@ -0,0 +1 @@ +init diff --git a/samples/warpdrive/Makefile.am b/samples/warpdrive/Makefile.am new file mode 100644 index 000000000000..41154a880a97 --- /dev/null +++ b/samples/warpdrive/Makefile.am @@ -0,0 +1,9 @@ +ACLOCAL_AMFLAGS = -I m4 +AUTOMAKE_OPTIONS = foreign subdir-objects +AM_CFLAGS=-Wall -O0 -fno-strict-aliasing + +lib_LTLIBRARIES=libwd.la +libwd_la_SOURCES=wd.c wd_adapter.c wd.h wd_adapter.h \ + drv/hisi_qm_udrv.c drv/hisi_qm_udrv.h + +SUBDIRS=. test diff --git a/samples/warpdrive/NEWS b/samples/warpdrive/NEWS new file mode 100644 index 000000000000..b1b716105590 --- /dev/null +++ b/samples/warpdrive/NEWS @@ -0,0 +1 @@ +init diff --git a/samples/warpdrive/README b/samples/warpdrive/README new file mode 100644 index 000000000000..3adf66b112fc --- /dev/null +++ b/samples/warpdrive/README @@ -0,0 +1,32 @@ +WD User Land Demonstration +========================== + +This directory contains some applications and libraries to demonstrate how a + +WrapDrive application can be constructed. + + +As a demo, we try to make it simple and clear for understanding. It is not + +supposed to be used in business scenario. + + +The directory contains the following elements: + +wd.[ch] + A demonstration WrapDrive fundamental library which wraps the basic + operations to the WrapDrive-ed device. + +wd_adapter.[ch] + User driver adaptor for wd.[ch] + +wd_utils.[ch] + Some utitlities function used by WD and its drivers + +drv/* + User drivers. It helps to fulfill the semantic of wd.[ch] for + particular hardware + +test/* + Test applications to use the wrapdrive library + diff --git a/samples/warpdrive/autogen.sh b/samples/warpdrive/autogen.sh new file mode 100755 index 000000000000..58deaf49de2a --- /dev/null +++ b/samples/warpdrive/autogen.sh @@ -0,0 +1,3 @@ +#!/bin/sh -x + +autoreconf -i -f -v diff --git a/samples/warpdrive/cleanup.sh b/samples/warpdrive/cleanup.sh new file mode 100755 index 000000000000..c5f3d21e5dc1 --- /dev/null +++ b/samples/warpdrive/cleanup.sh @@ -0,0 +1,13 @@ +#!/bin/sh + +if [ -r Makefile ]; then + make distclean +fi + +FILES="aclocal.m4 autom4te.cache compile config.guess config.h.in config.log \ + config.status config.sub configure cscope.out depcomp install-sh \ + libsrc/Makefile libsrc/Makefile.in libtool ltmain.sh Makefile \ + ar-lib m4 \ + Makefile.in missing src/Makefile src/Makefile.in test/Makefile.in" + +rm -vRf $FILES diff --git a/samples/warpdrive/configure.ac b/samples/warpdrive/configure.ac new file mode 100644 index 000000000000..53262f3197c2 --- /dev/null +++ b/samples/warpdrive/configure.ac @@ -0,0 +1,52 @@ +AC_PREREQ([2.69]) +AC_INIT([wrapdrive], [0.1], [liguozhu@hisilicon.com]) +AC_CONFIG_SRCDIR([wd.c]) +AM_INIT_AUTOMAKE([1.10 no-define]) + +AC_CONFIG_MACRO_DIR([m4]) +AC_CONFIG_HEADERS([config.h]) + +# Checks for programs. +AC_PROG_CXX +AC_PROG_AWK +AC_PROG_CC +AC_PROG_CPP +AC_PROG_INSTALL +AC_PROG_LN_S +AC_PROG_MAKE_SET +AC_PROG_RANLIB + +AM_PROG_AR +AC_PROG_LIBTOOL +AM_PROG_LIBTOOL +LT_INIT +AM_PROG_CC_C_O + +AC_DEFINE([HAVE_SVA], [0], [enable SVA support]) +AC_ARG_ENABLE([sva], + [ --enable-sva enable to support sva feature], + AC_DEFINE([HAVE_SVA], [1])) + +# Checks for libraries. + +# Checks for header files. +AC_CHECK_HEADERS([fcntl.h stdint.h stdlib.h string.h sys/ioctl.h sys/time.h unistd.h]) + +# Checks for typedefs, structures, and compiler characteristics. +AC_CHECK_HEADER_STDBOOL +AC_C_INLINE +AC_TYPE_OFF_T +AC_TYPE_SIZE_T +AC_TYPE_UINT16_T +AC_TYPE_UINT32_T +AC_TYPE_UINT64_T +AC_TYPE_UINT8_T + +# Checks for library functions. +AC_FUNC_MALLOC +AC_FUNC_MMAP +AC_CHECK_FUNCS([memset munmap]) + +AC_CONFIG_FILES([Makefile + test/Makefile]) +AC_OUTPUT diff --git a/samples/warpdrive/drv/hisi_qm_udrv.c b/samples/warpdrive/drv/hisi_qm_udrv.c new file mode 100644 index 000000000000..777e2e3cff18 --- /dev/null +++ b/samples/warpdrive/drv/hisi_qm_udrv.c @@ -0,0 +1,223 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "hisi_qm_udrv.h" + +#if __AARCH64EL__ == 1 +#define mb() {asm volatile("dsb sy" : : : "memory"); } +#else +#warning "this file need to be used on AARCH64EL mode" +#define mb() +#endif + +#define QM_SQE_SIZE 128 +#define QM_CQE_SIZE 16 +#define QM_EQ_DEPTH 1024 + +/* cqe shift */ +#define CQE_PHASE(cq) (((*((__u32 *)(cq) + 3)) >> 16) & 0x1) +#define CQE_SQ_NUM(cq) ((*((__u32 *)(cq) + 2)) >> 16) +#define CQE_SQ_HEAD_INDEX(cq) ((*((__u32 *)(cq) + 2)) & 0xffff) + +#define QM_IOMEM_SIZE 4096 + +#define QM_DOORBELL_OFFSET 0x340 + +struct cqe { + __le32 rsvd0; + __le16 cmd_id; + __le16 rsvd1; + __le16 sq_head; + __le16 sq_num; + __le16 rsvd2; + __le16 w7; /* phase, status */ +}; + +struct hisi_qm_queue_info { + void *sq_base; + void *cq_base; + void *doorbell_base; + __u16 sq_tail_index; + __u16 sq_head_index; + __u16 cq_head_index; + __u16 sqn; + bool cqc_phase; + void *req_cache[QM_EQ_DEPTH]; + int is_sq_full; +}; + +int hacc_db(struct hisi_qm_queue_info *q, __u8 cmd, __u16 index, __u8 priority) +{ + void *base = q->doorbell_base; + __u16 sqn = q->sqn; + __u64 doorbell = 0; + + doorbell = (__u64)sqn | ((__u64)cmd << 16); + doorbell |= ((__u64)index | ((__u64)priority << 16)) << 32; + + *((__u64 *)base) = doorbell; + + return 0; +} + +static int hisi_qm_fill_sqe(void *msg, struct hisi_qm_queue_info *info, __u16 i) +{ + struct hisi_qm_msg *sqe = (struct hisi_qm_msg *)info->sq_base + i; + memcpy((void *)sqe, msg, sizeof(struct hisi_qm_msg)); + assert(!info->req_cache[i]); + info->req_cache[i] = msg; + + return 0; +} + +static int hisi_qm_recv_sqe(struct hisi_qm_msg *sqe, struct hisi_qm_queue_info *info, __u16 i) +{ + __u32 status = sqe->dw3 & 0xff; + __u32 type = sqe->dw9 & 0xff; + + if (status != 0 && status != 0x0d) { + fprintf(stderr, "bad status (s=%d, t=%d)\n", status, type); + return -EIO; + } + + assert(info->req_cache[i]); + memcpy((void *)info->req_cache[i], sqe, sizeof(struct hisi_qm_msg)); + return 0; +} + +int hisi_qm_set_queue_dio(struct wd_queue *q) +{ + struct hisi_qm_queue_info *info; + void *vaddr; + int ret; + + alloc_obj(info); + if (!info) + return -1; + + q->priv = info; + + vaddr = mmap(NULL, + QM_SQE_SIZE * QM_EQ_DEPTH + QM_CQE_SIZE * QM_EQ_DEPTH, + PROT_READ | PROT_WRITE, MAP_SHARED, q->mdev, 4096); + if (vaddr <= 0) { + ret = (intptr_t)vaddr; + goto err_with_info; + } + info->sq_base = vaddr; + info->cq_base = vaddr + QM_SQE_SIZE * QM_EQ_DEPTH; + + vaddr = mmap(NULL, QM_IOMEM_SIZE, + PROT_READ | PROT_WRITE, MAP_SHARED, q->mdev, 0); + if (vaddr <= 0) { + ret = (intptr_t)vaddr; + goto err_with_scq; + } + info->doorbell_base = vaddr + QM_DOORBELL_OFFSET; + info->sq_tail_index = 0; + info->sq_head_index = 0; + info->cq_head_index = 0; + info->cqc_phase = 1; + + info->is_sq_full = 0; + + return 0; + +err_with_scq: + munmap(info->sq_base, + QM_SQE_SIZE * QM_EQ_DEPTH + QM_CQE_SIZE * QM_EQ_DEPTH); +err_with_info: + free(info); + return ret; +} + +void hisi_qm_unset_queue_dio(struct wd_queue *q) +{ + struct hisi_qm_queue_info *info = (struct hisi_qm_queue_info *)q->priv; + + munmap(info->doorbell_base - QM_DOORBELL_OFFSET, QM_IOMEM_SIZE); + munmap(info->cq_base, QM_CQE_SIZE * QM_EQ_DEPTH); + munmap(info->sq_base, QM_SQE_SIZE * QM_EQ_DEPTH); + free(info); + q->priv = NULL; +} + +int hisi_qm_add_to_dio_q(struct wd_queue *q, void *req) +{ + struct hisi_qm_queue_info *info = (struct hisi_qm_queue_info *)q->priv; + __u16 i; + + if (info->is_sq_full) + return -EBUSY; + + i = info->sq_tail_index; + + hisi_qm_fill_sqe(req, q->priv, i); + + mb(); + + if (i == (QM_EQ_DEPTH - 1)) + i = 0; + else + i++; + + hacc_db(info, DOORBELL_CMD_SQ, i, 0); + + info->sq_tail_index = i; + + if (i == info->sq_head_index) + info->is_sq_full = 1; + + return 0; +} + +int hisi_qm_get_from_dio_q(struct wd_queue *q, void **resp) +{ + struct hisi_qm_queue_info *info = (struct hisi_qm_queue_info *)q->priv; + __u16 i = info->cq_head_index; + struct cqe *cq_base = info->cq_base; + struct hisi_qm_msg *sq_base = info->sq_base; + struct cqe *cqe = cq_base + i; + struct hisi_qm_msg *sqe; + int ret; + + if (info->cqc_phase == CQE_PHASE(cqe)) { + sqe = sq_base + CQE_SQ_HEAD_INDEX(cqe); + ret = hisi_qm_recv_sqe(sqe, info, i); + if (ret < 0) + return -EIO; + + if (info->is_sq_full) + info->is_sq_full = 0; + } else { + return -EAGAIN; + } + + *resp = info->req_cache[i]; + info->req_cache[i] = NULL; + + if (i == (QM_EQ_DEPTH - 1)) { + info->cqc_phase = !(info->cqc_phase); + i = 0; + } else + i++; + + hacc_db(info, DOORBELL_CMD_CQ, i, 0); + + info->cq_head_index = i; + info->sq_head_index = i; + + + return ret; +} diff --git a/samples/warpdrive/drv/hisi_qm_udrv.h b/samples/warpdrive/drv/hisi_qm_udrv.h new file mode 100644 index 000000000000..6a7a06a089c9 --- /dev/null +++ b/samples/warpdrive/drv/hisi_qm_udrv.h @@ -0,0 +1,53 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef __HZIP_DRV_H__ +#define __HZIP_DRV_H__ + +#include +#include "../wd.h" + +/* this is unnecessary big, the hardware should optimize it */ +struct hisi_qm_msg { + __u32 consumed; + __u32 produced; + __u32 comp_date_length; + __u32 dw3; + __u32 input_date_length; + __u32 lba_l; + __u32 lba_h; + __u32 dw7; /* ... */ + __u32 dw8; /* ... */ + __u32 dw9; /* ... */ + __u32 dw10; /* ... */ + __u32 priv_info; + __u32 dw12; /* ... */ + __u32 tag; + __u32 dest_avail_out; + __u32 rsvd0; + __u32 comp_head_addr_l; + __u32 comp_head_addr_h; + __u32 source_addr_l; + __u32 source_addr_h; + __u32 dest_addr_l; + __u32 dest_addr_h; + __u32 stream_ctx_addr_l; + __u32 stream_ctx_addr_h; + __u32 cipher_key1_addr_l; + __u32 cipher_key1_addr_h; + __u32 cipher_key2_addr_l; + __u32 cipher_key2_addr_h; + __u32 rsvd1[4]; +}; + +struct hisi_acc_qm_sqc { + __u16 sqn; +}; + +#define DOORBELL_CMD_SQ 0 +#define DOORBELL_CMD_CQ 1 + +int hisi_qm_set_queue_dio(struct wd_queue *q); +void hisi_qm_unset_queue_dio(struct wd_queue *q); +int hisi_qm_add_to_dio_q(struct wd_queue *q, void *req); +int hisi_qm_get_from_dio_q(struct wd_queue *q, void **resp); + +#endif diff --git a/samples/warpdrive/test/Makefile.am b/samples/warpdrive/test/Makefile.am new file mode 100644 index 000000000000..ad80e80a47d7 --- /dev/null +++ b/samples/warpdrive/test/Makefile.am @@ -0,0 +1,7 @@ +AM_CFLAGS=-Wall -O0 -fno-strict-aliasing + +bin_PROGRAMS=test_hisi_zip + +test_hisi_zip_SOURCES=test_hisi_zip.c + +test_hisi_zip_LDADD=../.libs/libwd.a diff --git a/samples/warpdrive/test/comp_hw.h b/samples/warpdrive/test/comp_hw.h new file mode 100644 index 000000000000..79328fd0c1a0 --- /dev/null +++ b/samples/warpdrive/test/comp_hw.h @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0 +/** + * This file is shared bewteen user and kernel space Wrapdrive which is + * including algorithm attibutions that both user and driver are caring for + */ + +#ifndef __VFIO_WDEV_COMP_H +#define __VFIO_WDEV_COMP_H + +/* De-compressing algorithms' parameters */ +struct vfio_wdev_comp_param { + __u32 window_size; + __u32 comp_level; + __u32 mode; + __u32 alg; +}; + +/* WD defines all the De-compressing algorithm names here */ +#define VFIO_WDEV_ZLIB "zlib" +#define VFIO_WDEV_GZIP "gzip" +#define VFIO_WDEV_LZ4 "lz4" + +#endif diff --git a/samples/warpdrive/test/test_hisi_zip.c b/samples/warpdrive/test/test_hisi_zip.c new file mode 100644 index 000000000000..5bf90c6d0e81 --- /dev/null +++ b/samples/warpdrive/test/test_hisi_zip.c @@ -0,0 +1,206 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include "../wd.h" +#include "comp_hw.h" +#include "../drv/hisi_qm_udrv.h" + +#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(__CYGWIN__) +# include +# include +# define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY) +#else +# define SET_BINARY_MODE(file) +#endif + +#define PAGE_SHIFT 12 +#define PAGE_SIZE (1 << PAGE_SHIFT) + +#define ASIZE (8*512*4096) /*16MB*/ + +#define SYS_ERR_COND(cond, msg) \ +do { \ + if (cond) { \ + perror(msg); \ + exit(EXIT_FAILURE); \ + } \ +} while (0) + +#define ZLIB 0 +#define GZIP 1 + +#define CHUNK 65535 + + +int hizip_deflate(FILE *source, FILE *dest, int type) +{ + __u64 in, out; + struct wd_queue q; + struct hisi_qm_msg *msg, *recv_msg; + void *a, *b; + char *src, *dst; + int ret, total_len; + int output_num; + int fd, file_msize; + + memset(&q, 0, sizeof(q)); + q.container = -1; + q.mdev_name = "22e09922-7a82-11e8-9cf6-d74cffa9e87b"; + q.vfio_group_path = "/dev/vfio/10"; //fixme to the right path + q.iommu_ext_path = "/sys/class/sdmdev/0000:75:00.0/device/params/iommu_type"; + q.dmaflag_ext_path = "/sys/class/sdmdev/0000:75:00.0/device/params/dma_flag"; + q.device_api_path = "/sys/class/sdmdev/0000:75:00.0/device/mdev_supported_types/hisi_zip-hisi_zip/device_api"; + ret = wd_request_queue(&q); + SYS_ERR_COND(ret, "wd_request_queue"); + + fprintf(stderr, "pasid=%d, dma_flag=%d\n", q.pasid, q.dma_flag); + fd = fileno(source); + struct stat s; + + if (fstat(fd, &s) < 0) { + close(fd); + perror("fd error\n"); + return -1; + } + total_len = s.st_size; + + if (!total_len) { + ret = -EINVAL; + SYS_ERR_COND(ret, "input file length zero"); + } + if (total_len > 16*1024*1024) { + fputs("error, input file size too large(<16MB)!\n", stderr); + goto release_q; + } + file_msize = !(total_len%PAGE_SIZE) ? total_len : + (total_len/PAGE_SIZE+1)*PAGE_SIZE; + /* mmap file and DMA mapping */ + a = mmap((void *)0x0, file_msize, PROT_READ | PROT_WRITE, + MAP_PRIVATE, fd, 0); + if (!a) { + fputs("mmap file fail!\n", stderr); + goto release_q; + } + ret = wd_mem_share(&q, a, file_msize, 0); + if (ret) { + fprintf(stderr, "wd_mem_share dma a buf fail!err=%d\n", -errno); + goto unmap_file; + } + /* Allocate some space and setup a DMA mapping */ + b = mmap((void *)0x0, ASIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, 0, 0); + if (!b) { + fputs("mmap b fail!\n", stderr); + goto unshare_file; + } + memset(b, 0, ASIZE); + ret = wd_mem_share(&q, b, ASIZE, 0); + if (ret) { + fputs("wd_mem_share dma b buf fail!\n", stderr); + goto unmap_mem; + } + src = (char *)a; + dst = (char *)b; + + msg = malloc(sizeof(*msg)); + if (!msg) { + fputs("alloc msg fail!\n", stderr); + goto alloc_msg_fail; + } + memset((void *)msg, 0, sizeof(*msg)); + msg->input_date_length = total_len; + if (type == ZLIB) + msg->dw9 = 2; + else + msg->dw9 = 3; + msg->dest_avail_out = 0x800000; + + in = (__u64)src; + out = (__u64)dst; + + msg->source_addr_l = in & 0xffffffff; + msg->source_addr_h = in >> 32; + msg->dest_addr_l = out & 0xffffffff; + msg->dest_addr_h = out >> 32; + + ret = wd_send(&q, msg); + + if (ret == -EBUSY) { + usleep(1); + goto recv_again; + } + SYS_ERR_COND(ret, "send fail!\n"); +recv_again: + ret = wd_recv(&q, (void **)&recv_msg); + if (ret == -EIO) { + fputs(" wd_recv fail!\n", stderr); + goto alloc_msg_fail; + /* synchronous mode, if get none, then get again */ + } else if (ret == -EAGAIN) + goto recv_again; + + output_num = recv_msg->produced; + /* add zlib compress head and write head + compressed date to a file */ + char zip_head[2] = {0x78, 0x9c}; + + fwrite(zip_head, 1, 2, dest); + fwrite((char *)out, 1, output_num, dest); + fclose(dest); + + free(msg); +alloc_msg_fail: + wd_mem_unshare(&q, b, ASIZE); +unmap_mem: + munmap(b, ASIZE); +unshare_file: + wd_mem_unshare(&q, a, file_msize); +unmap_file: + munmap(a, file_msize); +release_q: + wd_release_queue(&q); + + return ret; +} + +int main(int argc, char *argv[]) +{ + int alg_type = 0; + + /* avoid end-of-line conversions */ + SET_BINARY_MODE(stdin); + SET_BINARY_MODE(stdout); + + if (!argv[1]) { + fputs("<>\n", stderr); + goto EXIT; + } + + if (!strcmp(argv[1], "-z")) + alg_type = ZLIB; + else if (!strcmp(argv[1], "-g")) { + alg_type = GZIP; + } else if (!strcmp(argv[1], "-h")) { + fputs("[version]:1.0.2\n", stderr); + fputs("[usage]: ./test_hisi_zip [type] dest_file\n", + stderr); + fputs(" [type]:\n", stderr); + fputs(" -z = zlib\n", stderr); + fputs(" -g = gzip\n", stderr); + fputs(" -h = usage\n", stderr); + fputs("Example:\n", stderr); + fputs("./test_hisi_zip -z < test.data > out.data\n", stderr); + goto EXIT; + } else { + fputs("Unknow option\n", stderr); + fputs("<>\n", + stderr); + goto EXIT; + } + + hizip_deflate(stdin, stdout, alg_type); +EXIT: + return EXIT_SUCCESS; +} diff --git a/samples/warpdrive/wd.c b/samples/warpdrive/wd.c new file mode 100644 index 000000000000..8df071637341 --- /dev/null +++ b/samples/warpdrive/wd.c @@ -0,0 +1,309 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "config.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "wd.h" +#include "wd_adapter.h" + +#if (defined(HAVE_SVA) & HAVE_SVA) +static int _wd_bind_process(struct wd_queue *q) +{ + struct bind_data { + struct vfio_iommu_type1_bind bind; + struct vfio_iommu_type1_bind_process data; + } wd_bind; + int ret; + __u32 flags = 0; + + if (q->dma_flag & VFIO_SDMDEV_DMA_MULTI_PROC_MAP) + flags = VFIO_IOMMU_BIND_PRIV; + else if (q->dma_flag & VFIO_SDMDEV_DMA_SVM_NO_FAULT) + flags = VFIO_IOMMU_BIND_NOPF; + + wd_bind.bind.flags = VFIO_IOMMU_BIND_PROCESS; + wd_bind.bind.argsz = sizeof(wd_bind); + wd_bind.data.flags = flags; + ret = ioctl(q->container, VFIO_IOMMU_BIND, &wd_bind); + if (ret) + return ret; + q->pasid = wd_bind.data.pasid; + return ret; +} + +static int _wd_unbind_process(struct wd_queue *q) +{ + struct bind_data { + struct vfio_iommu_type1_bind bind; + struct vfio_iommu_type1_bind_process data; + } wd_bind; + __u32 flags = 0; + + if (q->dma_flag & VFIO_SDMDEV_DMA_MULTI_PROC_MAP) + flags = VFIO_IOMMU_BIND_PRIV; + else if (q->dma_flag & VFIO_SDMDEV_DMA_SVM_NO_FAULT) + flags = VFIO_IOMMU_BIND_NOPF; + + wd_bind.bind.flags = VFIO_IOMMU_BIND_PROCESS; + wd_bind.data.pasid = q->pasid; + wd_bind.data.flags = flags; + wd_bind.bind.argsz = sizeof(wd_bind); + + return ioctl(q->container, VFIO_IOMMU_UNBIND, &wd_bind); +} +#endif + +int wd_request_queue(struct wd_queue *q) +{ + struct vfio_group_status group_status = { + .argsz = sizeof(group_status) }; + int iommu_ext; + int ret; + + if (!q->vfio_group_path || + !q->device_api_path || + !q->dmaflag_ext_path || + !q->iommu_ext_path) { + WD_ERR("please set vfio_group_path, dmaflag_ext_path, " + "device_api_path, and iommu_ext_path before call %s", __func__); + return -EINVAL; + } + + q->hw_type_id = 0; /* this can be set according to the device api_version in the future */ + + q->group = open(q->vfio_group_path, O_RDWR); + if (q->group < 0) { + WD_ERR("open vfio group(%s) fail, errno=%d\n", + q->vfio_group_path, errno); + return -errno; + } + + if (q->container <= 0) { + q->container = open("/dev/vfio/vfio", O_RDWR); + if (q->container < 0) { + WD_ERR("Create VFIO container fail!\n"); + ret = -ENODEV; + goto err_with_group; + } + } + + if (ioctl(q->container, VFIO_GET_API_VERSION) != VFIO_API_VERSION) { + WD_ERR("VFIO version check fail!\n"); + ret = -EINVAL; + goto err_with_container; + } + + q->dma_flag = _get_attr_int(q->dmaflag_ext_path); + if (q->dma_flag == INT_MIN) { + ret = -EINVAL; + goto err_with_container; + } + + iommu_ext = _get_attr_int(q->iommu_ext_path); + if (iommu_ext == INT_MIN) { + ret = -EINVAL; + goto err_with_container; + } + + ret = ioctl(q->container, VFIO_CHECK_EXTENSION, iommu_ext); + if (!ret) { + WD_ERR("VFIO iommu check (%d) fail (%d)!\n", iommu_ext, ret); + goto err_with_container; + } + + ret = _get_attr_str(q->device_api_path, q->hw_type); + if (ret) + goto err_with_container; + + ret = ioctl(q->group, VFIO_GROUP_GET_STATUS, &group_status); + if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) { + WD_ERR("VFIO group is not viable\n"); + goto err_with_container; + } + + ret = ioctl(q->group, VFIO_GROUP_SET_CONTAINER, &q->container); + if (ret) { + WD_ERR("VFIO group fail on VFIO_GROUP_SET_CONTAINER\n"); + goto err_with_container; + } + + ret = ioctl(q->container, VFIO_SET_IOMMU, iommu_ext); + if (ret) { + WD_ERR("VFIO fail on VFIO_SET_IOMMU(%d)\n", iommu_ext); + goto err_with_container; + } + + q->mdev = ioctl(q->group, VFIO_GROUP_GET_DEVICE_FD, q->mdev_name); + if (q->mdev < 0) { + WD_ERR("VFIO fail on VFIO_GROUP_GET_DEVICE_FD (%d)\n", q->mdev); + ret = q->mdev; + goto err_with_container; + } + +#if (defined(HAVE_SVA) & HAVE_SVA) + if (!(q->dma_flag & (VFIO_SDMDEV_DMA_PHY | VFIO_SDMDEV_DMA_SINGLE_PROC_MAP))) { + ret = _wd_bind_process(q); + if (ret) { + close(q->mdev); + WD_ERR("VFIO fails to bind process!\n"); + goto err_with_mdev; + + } + } + + ret = ioctl(q->mdev, VFIO_SDMDEV_CMD_BIND_PASID, (unsigned long)q->pasid); + if (ret < 0) { + WD_ERR("fail to bind paisd to device,ret=%d\n", errno); + goto err_with_mdev; + } +#endif + + ret = drv_open(q); + if (ret) + goto err_with_mdev; + + return 0; + +err_with_mdev: + close(q->mdev); +err_with_container: + close(q->container); +err_with_group: + close(q->group); + return ret; +} + +void wd_release_queue(struct wd_queue *q) +{ + drv_close(q); + +#if (defined(HAVE_SVA) & HAVE_SVA) + if (!(q->dma_flag & (VFIO_SDMDEV_DMA_PHY | VFIO_SDMDEV_DMA_SINGLE_PROC_MAP))) { + if (q->pasid <= 0) { + WD_ERR("Wd queue pasid ! pasid=%d\n", q->pasid); + return; + } + if (_wd_unbind_process(q)) { + WD_ERR("VFIO fails to unbind process!\n"); + return; + } + } +#endif + + close(q->mdev); + close(q->container); + close(q->group); +} + +int wd_send(struct wd_queue *q, void *req) +{ + return drv_send(q, req); +} + +int wd_recv(struct wd_queue *q, void **resp) +{ + return drv_recv(q, resp); +} + +static int wd_flush_and_wait(struct wd_queue *q, int ms) +{ + wd_flush(q); + return ioctl(q->mdev, VFIO_SDMDEV_CMD_WAIT, ms); +} + +int wd_recv_sync(struct wd_queue *q, void **resp, __u16 ms) +{ + int ret; + + while (1) { + ret = wd_recv(q, resp); + if (ret == -EBUSY) { + ret = wd_flush_and_wait(q, ms); + if (ret) + return ret; + } else + return ret; + } +} + +void wd_flush(struct wd_queue *q) +{ + drv_flush(q); +} + +static int _wd_mem_share_type1(struct wd_queue *q, const void *addr, + size_t size, int flags) +{ + struct vfio_iommu_type1_dma_map dma_map; + + if (q->dma_flag & VFIO_SDMDEV_DMA_SVM_NO_FAULT) + return mlock(addr, size); + +#if (defined(HAVE_SVA) & HAVE_SVA) + else if ((q->dma_flag & VFIO_SDMDEV_DMA_MULTI_PROC_MAP) && + (q->pasid > 0)) + dma_map.pasid = q->pasid; +#endif + else if ((q->dma_flag & VFIO_SDMDEV_DMA_SINGLE_PROC_MAP)) + ; //todo + else + return -1; + + dma_map.vaddr = (__u64)addr; + dma_map.size = size; + dma_map.iova = (__u64)addr; + dma_map.flags = + VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE | flags; + dma_map.argsz = sizeof(dma_map); + + return ioctl(q->container, VFIO_IOMMU_MAP_DMA, &dma_map); +} + +static void _wd_mem_unshare_type1(struct wd_queue *q, const void *addr, + size_t size) +{ +#if (defined(HAVE_SVA) & HAVE_SVA) + struct vfio_iommu_type1_dma_unmap dma_unmap; +#endif + + if (q->dma_flag & VFIO_SDMDEV_DMA_SVM_NO_FAULT) { + (void)munlock(addr, size); + return; + } + +#if (defined(HAVE_SVA) & HAVE_SVA) + dma_unmap.iova = (__u64)addr; + if ((q->dma_flag & VFIO_SDMDEV_DMA_MULTI_PROC_MAP) && (q->pasid > 0)) + dma_unmap.flags = 0; + dma_unmap.size = size; + dma_unmap.argsz = sizeof(dma_unmap); + ioctl(q->container, VFIO_IOMMU_UNMAP_DMA, &dma_unmap); +#endif +} + +int wd_mem_share(struct wd_queue *q, const void *addr, size_t size, int flags) +{ + if (drv_can_do_mem_share(q)) + return drv_share(q, addr, size, flags); + else + return _wd_mem_share_type1(q, addr, size, flags); +} + +void wd_mem_unshare(struct wd_queue *q, const void *addr, size_t size) +{ + if (drv_can_do_mem_share(q)) + drv_unshare(q, addr, size); + else + _wd_mem_unshare_type1(q, addr, size); +} + diff --git a/samples/warpdrive/wd.h b/samples/warpdrive/wd.h new file mode 100644 index 000000000000..eccf43dc034d --- /dev/null +++ b/samples/warpdrive/wd.h @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef __WD_H +#define __WD_H +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../include/uapi/linux/vfio.h" +#include "../../include/uapi/linux/vfio_sdmdev.h" + +#define SYS_VAL_SIZE 16 +#define PATH_STR_SIZE 256 +#define WD_NAME_SIZE 64 +#define WD_MAX_MEMLIST_SZ 128 + + +#ifndef dma_addr_t +#define dma_addr_t __u64 +#endif + +typedef int bool; + +#ifndef true +#define true 1 +#endif + +#ifndef false +#define false 0 +#endif + +/* the flags used by wd_capa->flags, the high 16bits are for algorithm + * and the low 16bits are for Framework + */ +#define WD_FLAGS_FW_PREFER_LOCAL_ZONE 1 + +#define WD_FLAGS_FW_MASK 0x0000FFFF +#ifndef WD_ERR +#define WD_ERR(format, args...) fprintf(stderr, format, ##args) +#endif + +/* Default page size should be 4k size */ +#define WDQ_MAP_REGION(region_index) ((region_index << 12) & 0xf000) +#define WDQ_MAP_Q(q_index) ((q_index << 16) & 0xffff0000) + +static inline void wd_reg_write(void *reg_addr, uint32_t value) +{ + *((volatile uint32_t *)reg_addr) = value; +} + +static inline uint32_t wd_reg_read(void *reg_addr) +{ + uint32_t temp; + + temp = *((volatile uint32_t *)reg_addr); + + return temp; +} + +static inline int _get_attr_str(const char *path, char value[PATH_STR_SIZE]) +{ + int fd, ret; + + fd = open(path, O_RDONLY); + if (fd < 0) { + WD_ERR("get_attr_str: open %s fail\n", path); + return fd; + } + memset(value, 0, PATH_STR_SIZE); + ret = read(fd, value, PATH_STR_SIZE); + if (ret > 0) { + close(fd); + return 0; + } + close(fd); + + WD_ERR("read nothing from %s\n", path); + return -EINVAL; +} + +static inline int _get_attr_int(const char *path) +{ + char value[PATH_STR_SIZE]; + if (_get_attr_str(path, value)) + return INT_MIN; + else + return atoi(value); +} + +/* Memory in accelerating message can be different */ +enum wd_addr_flags { + WD_AATTR_INVALID = 0, + + /* Common user virtual memory */ + _WD_AATTR_COM_VIRT = 1, + + /* Physical address*/ + _WD_AATTR_PHYS = 2, + + /* I/O virtual address*/ + _WD_AATTR_IOVA = 4, + + /* SGL, user cares for */ + WD_AATTR_SGL = 8, +}; + +#define WD_CAPA_PRIV_DATA_SIZE 64 + +#define alloc_obj(objp) do { \ + objp = malloc(sizeof(*objp)); \ + memset(objp, 0, sizeof(*objp)); \ +}while(0) +#define free_obj(objp) if (objp)free(objp) + +struct wd_queue { + const char *mdev_name; + char hw_type[PATH_STR_SIZE]; + int hw_type_id; + int dma_flag; + void *priv; /* private data used by the drv layer */ + int container; + int group; + int mdev; + int pasid; + int iommu_type; + char *vfio_group_path; + char *iommu_ext_path; + char *dmaflag_ext_path; + char *device_api_path; +}; + +extern int wd_request_queue(struct wd_queue *q); +extern void wd_release_queue(struct wd_queue *q); +extern int wd_send(struct wd_queue *q, void *req); +extern int wd_recv(struct wd_queue *q, void **resp); +extern void wd_flush(struct wd_queue *q); +extern int wd_recv_sync(struct wd_queue *q, void **resp, __u16 ms); +extern int wd_mem_share(struct wd_queue *q, const void *addr, + size_t size, int flags); +extern void wd_mem_unshare(struct wd_queue *q, const void *addr, size_t size); + +/* for debug only */ +extern int wd_dump_all_algos(void); + +/* this is only for drv used */ +extern int wd_set_queue_attr(struct wd_queue *q, const char *name, + char *value); +extern int __iommu_type(struct wd_queue *q); +#endif diff --git a/samples/warpdrive/wd_adapter.c b/samples/warpdrive/wd_adapter.c new file mode 100644 index 000000000000..d8d55b75e99a --- /dev/null +++ b/samples/warpdrive/wd_adapter.c @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + + +#include "wd_adapter.h" +#include "./drv/hisi_qm_udrv.h" + +static struct wd_drv_dio_if hw_dio_tbl[] = { { + .hw_type = "hisi_qm_v1", + .open = hisi_qm_set_queue_dio, + .close = hisi_qm_unset_queue_dio, + .send = hisi_qm_add_to_dio_q, + .recv = hisi_qm_get_from_dio_q, + }, + /* Add other drivers direct IO operations here */ +}; + +/* todo: there should be some stable way to match the device and the driver */ +#define MAX_HW_TYPE (sizeof(hw_dio_tbl) / sizeof(hw_dio_tbl[0])) + +int drv_open(struct wd_queue *q) +{ + int i; + + //todo: try to find another dev if the user driver is not avaliable + for (i = 0; i < MAX_HW_TYPE; i++) { + if (!strcmp(q->hw_type, + hw_dio_tbl[i].hw_type)) { + q->hw_type_id = i; + return hw_dio_tbl[q->hw_type_id].open(q); + } + } + WD_ERR("No matching driver to use!\n"); + errno = ENODEV; + return -ENODEV; +} + +void drv_close(struct wd_queue *q) +{ + hw_dio_tbl[q->hw_type_id].close(q); +} + +int drv_send(struct wd_queue *q, void *req) +{ + return hw_dio_tbl[q->hw_type_id].send(q, req); +} + +int drv_recv(struct wd_queue *q, void **req) +{ + return hw_dio_tbl[q->hw_type_id].recv(q, req); +} + +int drv_share(struct wd_queue *q, const void *addr, size_t size, int flags) +{ + return hw_dio_tbl[q->hw_type_id].share(q, addr, size, flags); +} + +void drv_unshare(struct wd_queue *q, const void *addr, size_t size) +{ + hw_dio_tbl[q->hw_type_id].unshare(q, addr, size); +} + +bool drv_can_do_mem_share(struct wd_queue *q) +{ + return hw_dio_tbl[q->hw_type_id].share != NULL; +} + +void drv_flush(struct wd_queue *q) +{ + if (hw_dio_tbl[q->hw_type_id].flush) + hw_dio_tbl[q->hw_type_id].flush(q); +} diff --git a/samples/warpdrive/wd_adapter.h b/samples/warpdrive/wd_adapter.h new file mode 100644 index 000000000000..bb3ab3ec112a --- /dev/null +++ b/samples/warpdrive/wd_adapter.h @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: GPL-2.0 +/* the common drv header define the unified interface for wd */ +#ifndef __WD_ADAPTER_H__ +#define __WD_ADAPTER_H__ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + + +#include "wd.h" + +struct wd_drv_dio_if { + char *hw_type; + int (*open)(struct wd_queue *q); + void (*close)(struct wd_queue *q); + int (*set_pasid)(struct wd_queue *q); + int (*unset_pasid)(struct wd_queue *q); + int (*send)(struct wd_queue *q, void *req); + int (*recv)(struct wd_queue *q, void **req); + void (*flush)(struct wd_queue *q); + int (*share)(struct wd_queue *q, const void *addr, + size_t size, int flags); + int (*unshare)(struct wd_queue *q, const void *addr, size_t size); +}; + +extern int drv_open(struct wd_queue *q); +extern void drv_close(struct wd_queue *q); +extern int drv_send(struct wd_queue *q, void *req); +extern int drv_recv(struct wd_queue *q, void **req); +extern void drv_flush(struct wd_queue *q); +extern int drv_share(struct wd_queue *q, const void *addr, + size_t size, int flags); +extern void drv_unshare(struct wd_queue *q, const void *addr, size_t size); +extern bool drv_can_do_mem_share(struct wd_queue *q); + +#endif