From patchwork Mon Aug 2 16:05:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lizhi Hou X-Patchwork-Id: 490436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05B72C4320A for ; Mon, 2 Aug 2021 16:05:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E2AD86112E for ; Mon, 2 Aug 2021 16:05:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232108AbhHBQFz (ORCPT ); Mon, 2 Aug 2021 12:05:55 -0400 Received: from mail-mw2nam12on2050.outbound.protection.outlook.com ([40.107.244.50]:25153 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229805AbhHBQFy (ORCPT ); Mon, 2 Aug 2021 12:05:54 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ANfqGduMbvNhegmXB189iqQUUmb8qS3+HXYTQ145WaWCIyCbcQQzOJErydNGHUYmOT9bkuwbVqCFPV0mLKqcKQs8hsXNGDWLgq9IXi7E+iGbEGLKm6tTg6LcFf2DkQ/ynlEVY66YD4u2Elz1Wb+HD02GM5FtdaQcxC65zofoDjZamUdOgRBszGL5E8XovNylCdzI3ChIvThBOIpnVY5sAeKvr9C76OXv3YVWyPPZdTK2s78oFuHHRZMFICcUlKKCqmMedjZ/QcDhc3pL+Fqr7X6qg8UX9B9MTmZUYYHVp8iO3iW8IoQ+R54U7UFrxY+1a8Pn7q0d8WweyAYsTrAT7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qG9Q4DeKG93FH4PxyINdLZBqdnKacv7fIyT4tk68ESY=; b=evP7a1xnjpRqlJ8ySV3myeypDqVHQF7KPLzkbePtQmdTzLt5AwsNt5ICQtNGeZ8a0wCYqq5kouhbFsgcK1/ivAN6qMz1jb5R2uLaCgPNht1q64CWo2WW2xZK45wtbkYpsYU6791cOS960Rtj8tHvLBlrM03ZFXH7TI5HPYPGDKK4tpy67KlcKtsTGg5+NPg+ZNPSE4HGnJIj2zCluWfyAKF7T6r4pZ87fWP1FpoSIv0Pn6b+gpqvgvb2O2inhp81f1l0pviccPLMKZE8ZEojMCWtA9pBDsp5EcL+BnH2B9xOseTs6tMSn8DWvaBnLafHN4b7Ikrh+kXGqXAsVQXdsA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=kernel.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qG9Q4DeKG93FH4PxyINdLZBqdnKacv7fIyT4tk68ESY=; b=pzsdY75X0DRSCgKFpyzeSM9USACaweFyy5Kwx0GTrAx5N+W78amsfEOAyPvyZZRnOZIcKUVoIQ0V0nglJYhI9QdfUeaP2/rHXyGibBDWsU+CRch4+zBl4pOwXcPcqHk4eSm71j891YC9EHHnwupa9vyP/Jd/0XYbe84JXD7iRaI= Received: from DM5PR13CA0013.namprd13.prod.outlook.com (2603:10b6:3:23::23) by BN6PR02MB2322.namprd02.prod.outlook.com (2603:10b6:404:2f::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.18; Mon, 2 Aug 2021 16:05:42 +0000 Received: from DM3NAM02FT059.eop-nam02.prod.protection.outlook.com (2603:10b6:3:23:cafe::b5) by DM5PR13CA0013.outlook.office365.com (2603:10b6:3:23::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4394.8 via Frontend Transport; Mon, 2 Aug 2021 16:05:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; kernel.org; dkim=none (message not signed) header.d=none; kernel.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by DM3NAM02FT059.mail.protection.outlook.com (10.13.4.97) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:05:42 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 2 Aug 2021 09:05:42 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.2 via Frontend Transport; Mon, 2 Aug 2021 09:05:42 -0700 Envelope-to: mdf@kernel.org, robh@kernel.org, trix@redhat.com, devicetree@vger.kernel.org, linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org Received: from [10.23.145.232] (port=49048 helo=xsj-xw9400.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1mAaRi-00055t-0p; Mon, 02 Aug 2021 09:05:42 -0700 Received: by xsj-xw9400.xilinx.com (Postfix, from userid 21952) id 0EC91601B15; Mon, 2 Aug 2021 09:05:29 -0700 (PDT) From: Lizhi Hou To: CC: Lizhi Hou , , , , , , , , , , , Max Zhen Subject: [PATCH V9 XRT Alveo 01/14] Documentation: fpga: Add a document describing XRT Alveo drivers Date: Mon, 2 Aug 2021 09:05:08 -0700 Message-ID: <20210802160521.331031-2-lizhi.hou@xilinx.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210802160521.331031-1-lizhi.hou@xilinx.com> References: <20210802160521.331031-1-lizhi.hou@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3c56ed53-5566-4570-b7e1-08d955cf6143 X-MS-TrafficTypeDiagnostic: BN6PR02MB2322: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 045j3B0qjC+Snm4V9QfUpjrVA2wfNDvpARqkGKd7rjCb/hRy5b+1YEkD9PFZkci2YAQ7dnbEMfI2NJtWPnoFjG7zAbiYdP/jRxR0mqs+rdJzVGFZXwJjhgWNNPo4qaIqseJjtB0c28rjOJra5g4Ojqft8hOhLxGw0eHJGEMLDuqVjgmVEOSlfqCW5BcYC61QykXPEhP0OLKFXcw+GC1mhJTHQjiDLNJRHVrNG3Wn3YxZ4X5/s2N6EGXytq+C0eeF8S6Dack5wBlVLYBppYmZKzQKPBDPMBDoMbSlrvR5z9lBtMlm+SGw6iMh1wei+buGwgGh6KTu3c7mQv6AIncOrV0WXZiYDRLdotSP4M621tL9kmwDaBnqfRK7li+TD3uqhBkGcRnCIMZcV/yRaRhDnKJ0Tf+9R4vyX6x99VJhV8GRxEmmVHHEVdnolOOimQcUJGgCEBUYXyfyhuoCPLOQ9+8WLAs3g959axGuczPWniVrxd1473Qwu3Ogmm3GAZlg/81RwcGWTavH0cvfkTRppV+Y1KBky5JTkhEHy9KnXMUyXifMNyvifNRVfe/87mgS+lpoj3INnZpb76H4Jp2W5q3nGS/ox7ShProtzlILwEheqtdlxseuf7psMOobUU9bjGaIFjbdemO2BUXrNBugCTEivqlqp8g8ZnGZEakMDnsiaAUPcHdqExKLry7BN+QVsZjJIVR2WwtTbbmUqfOoq1rBQ5FSlnTKJW5Q0yOdD0n7myeO8ynpOxXFdqDm3zW+nBo7Oii36RjNhyMnTf3P0Yu3Jqbms+Gy8Y/Yocf7BdKbFA6hl6GkFiCnO1ahqG8Z/k2IrvFHhvuJJkfCNlobKA== X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(4636009)(376002)(346002)(136003)(39860400002)(396003)(46966006)(36840700001)(478600001)(186003)(70206006)(1076003)(44832011)(2616005)(8936002)(6266002)(426003)(30864003)(8676002)(26005)(6916009)(42186006)(5660300002)(6666004)(70586007)(36906005)(107886003)(316002)(4326008)(54906003)(2906002)(7636003)(36756003)(356005)(47076005)(83380400001)(36860700001)(82740400003)(336012)(966005)(82310400003); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2021 16:05:42.7135 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3c56ed53-5566-4570-b7e1-08d955cf6143 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: DM3NAM02FT059.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR02MB2322 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Describe XRT driver architecture and provide basic overview of Xilinx Alveo platform. Signed-off-by: Sonal Santan Signed-off-by: Max Zhen Signed-off-by: Lizhi Hou Reviewed-by: Tom Rix --- Documentation/fpga/index.rst | 1 + Documentation/fpga/xrt.rst | 870 +++++++++++++++++++++++++++++++++++ MAINTAINERS | 11 + 3 files changed, 882 insertions(+) create mode 100644 Documentation/fpga/xrt.rst diff --git a/Documentation/fpga/index.rst b/Documentation/fpga/index.rst index f80f95667ca2..30134357b70d 100644 --- a/Documentation/fpga/index.rst +++ b/Documentation/fpga/index.rst @@ -8,6 +8,7 @@ fpga :maxdepth: 1 dfl + xrt .. only:: subproject and html diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst new file mode 100644 index 000000000000..84eb41be9ac1 --- /dev/null +++ b/Documentation/fpga/xrt.rst @@ -0,0 +1,870 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================== +XRTV2 Linux Kernel Driver Overview +================================== + +Authors: + +* Sonal Santan +* Max Zhen +* Lizhi Hou + +XRTV2 drivers are second generation `XRT `_ +drivers which support `Alveo `_ +PCIe platforms from Xilinx. + +XRTV2 drivers support *subsystem* style data driven platforms where driver's +configuration and behavior are determined by metadata provided by the platform +(in *device tree* format). Primary management physical function (MPF) driver +is called **xrt-mgmt**. Primary user physical function (UPF) driver is called +**xrt-user** and is under development. xrt_driver framework and HW subsystem +drivers are packaged into a library module called **xrt-lib**, which is shared +by **xrt-mgmt** and **xrt-user** (under development). The xrt_driver framework +implements a ``bus_type`` called **xrt_bus_type** which is used to discover HW +subsystems and facilitate inter HW subsystem interaction. + +Driver Modules +============== + +xrt-lib.ko +---------- + +xrt-lib is the repository of all subsystem drivers and pure software modules that +can potentially be shared between xrt-mgmt and xrt-user. All these drivers are +structured as **xrt_driver** and are instantiated by xrt-mgmt (or xrt-user under +development) based on the metadata associated with the hardware. The metadata is +in the form of a device tree as mentioned before. Each xrt_driver statically +defines a subsystem node array by using a node name or a string in its ``.endpoints`` +property. And this array is eventually translated to IOMEM resources in the +instantiated **xrt_device**. + +The xrt-lib infrastructure provides hooks to xrt_drivers for device node +management, user file operations and ioctl callbacks. The core infrastructure also +provides a bus functionality called **xrt_bus_type** for xrt_driver registration, +discovery and inter xrt_driver calls. xrt-lib does not have any dependency on PCIe +subsystem. + +.. note:: + See code in ``include/xleaf.h`` and ``include/xdevice.h`` + + +xrt-mgmt.ko +------------ + +The xrt-mgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo +PCIe device. It consists of one *root* driver, one or more *group* drivers +and one or more *xleaf* drivers. The group and xleaf drivers are instantiations +of the xrt_driver but are called group and xleaf to symbolize the logical operation +performed by them. + +The root driver manages the life cycle of multiple group drivers, which, in turn, +manages multiple xleaf drivers. This flexibility allows xrt-mgmt.ko and xrt-lib.ko +to support various HW subsystems exposed by different Alveo shells. The differences +among these Alveo shells is handled in xleaf drivers. The root and group +drivers are part of the infrastructure which provide common services to xleaf +drivers found on various Alveo shells. See :ref:`alveo_platform_overview`. + +The instantiation of specific group driver or xleaf drivers is completely data +driven based on metadata (mostly in device tree format) found through VSEC +capability and inside the firmware files, such as platform xsabin or user xclbin +file. + + +Driver Object Model +=================== + +The driver object model looks like the following:: + + +-----------+ + | xroot | + +-----+-----+ + | + +-----------+-----------+ + | | + v v + +-----------+ +-----------+ + | group | ... | group | + +-----+-----+ +------+----+ + | | + | | + +-----+----+ +-----+----+ + | | | | + v v v v + +-------+ +-------+ +-------+ +-------+ + | xleaf |..| xleaf | | xleaf |..| xleaf | + +-------+ +-------+ +-------+ +-------+ + +As an example, for Xilinx Alveo U50 before user xclbin download, the tree +looks like the following:: + + +-----------+ + | xrt-mgmt | + +-----+-----+ + | + +-------------------------+--------------------+ + | | | + v v v + +--------+ +--------+ +--------+ + | group0 | | group1 | | group2 | + +----+---+ +----+---+ +---+----+ + | | | + | | | + +-----+-----+ +----+-----+---+ +-----+-----+----+--------+ + | | | | | | | | | + v v | v v | v v | + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | | + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | + | +---------+ | +------+ +-----------+ | + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+ + +---------+ | +------+ +-----------+ + | +-------+ + +->| CALIB | + +-------+ + +After a xclbin is downloaded, group3 will be added and the tree looks like the +following:: + + +-----------+ + | xrt-mgmt | + +-----+-----+ + | + +-------------------------+--------------------+-----------------+ + | | | | + v v v | + +--------+ +--------+ +--------+ | + | group0 | | group1 | | group2 | | + +----+---+ +----+---+ +---+----+ | + | | | | + | | | | + +-----+-----+ +-----+-----+---+ +-----+-----+----+--------+ | + | | | | | | | | | | + v v | v v | v v | | + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | | + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | | | + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | | + | +---------+ | +------+ +-----------+ | | + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+ | + +---------+ | +------+ +-----------+ | + | +-------+ | + +->| CALIB | | + +-------+ | + +---+----+ | + | group3 |<--------------------------------------------+ + +--------+ + | + | + +-------+--------+---+--+--------+------+-------+ + | | | | | | | + v | v | v | v + +--------+ | +--------+ | +--------+ | +-----+ + | CLOCK0 | | | CLOCK1 | | | CLOCK2 | | | UCS | + +--------+ v +--------+ v +--------+ v +-----+ + +-------------+ +-------------+ +-------------+ + | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 | + +-------------+ +-------------+ +-------------+ + + +root +---- + +The root driver is a PCIe device driver attached to MPF. It's part of the +infrastructure of the MPF driver and resides in xrt-mgmt.ko. This driver + +* manages one or more group drivers +* provides access to functionalities that requires pci_dev, such as PCIE config + space access, to other xleaf drivers through root calls +* facilities inter xleaf driver calls for other xleaf drivers +* facilities event callbacks for other xleaf drivers + +When the root driver starts, it will explicitly create an initial group instance, +which contains xleaf drivers that will trigger the creation of other group +instances. The root driver will wait for all group and xleaf drivers to be +created before it returns from its probe routine and claim success of the +initialization of the entire xrt-mgmt driver. If any xleaf fails to initialize +the xrt-mgmt driver will still come online but with limited functionality. + +.. note:: + See code in ``lib/xroot.c`` and ``mgmt/root.c`` + + +group +----- + +The group driver represents a pseudo device whose life cycle is managed by +root and does not have real IO mem or IRQ resources. It's part of the +infrastructure of the MPF driver and resides in xrt-lib.ko. This driver + +* manages one or more xleaf drivers +* provides access to root from xleaf drivers, so that root calls, event + notifications and inter xleaf calls can happen + +In xrt-mgmt, an initial group driver instance will be created by the root. This +instance contains xleaf drivers that will trigger group instances to be created +to manage groups of xleaf drivers found on different partitions of hardware, +such as VSEC, Shell, and User. + +Every *fpga_region* has a group driver associated with it. The group driver is +created when a xclbin image is loaded on the fpga_region. The existing group +is destroyed when a new xclbin image is loaded. The fpga_region persists +across xclbin downloads. + +.. note:: + See code in ``lib/group.c`` + + +xleaf +----- + +The xleaf driver is a xrt_driver whose life cycle is managed by +a group driver and may or may not have real IO mem or IRQ resources. They +manage HW subsystems they are attached to. + +A xleaf driver without real hardware resources manages in-memory states for +xrt-mgmt. These states are shareable by other xleaf drivers. + +Xleaf drivers assigned to specific hardware resources drive a specific subsystem +in the device. To manipulate the subsystem or carry out a task, a xleaf driver +may ask for help from the root via root calls and/or from other leaves via +inter xleaf calls. + +A xleaf can also broadcast events through infrastructure code for other leaves +to process. It can also receive event notification from infrastructure about +certain events, such as post-creation or pre-exit of a particular xleaf. + +.. note:: + See code in ``lib/xleaf/*.c`` + + +xrt_bus_type +------------ + +xrt_bus_type defines a virtual bus which handles xrt_driver probe, remove and match +operations. All xrt_drivers register with xrt_bus_type as part of xrt-lib driver +``module_init`` and un-register as part of xrt-lib driver ``module_exit``. + +.. note:: + See code in ``lib/lib-drv.c`` + +FPGA Manager Interaction +======================== + +fpga_manager +------------ + +An instance of fpga_manager is created by xmgmt_main and is used for xclbin +image download. fpga_manager requires the full xclbin image before it can +start programming the FPGA configuration engine via Internal Configuration +Access Port (ICAP) xrt_driver. + +fpga_region +----------- + +For every interface exposed by the currently loaded xclbin/xsabin in the +*parent* fpga_region a new instance of fpga_region is created like a *child* +fpga_region. The device tree of the *parent* fpga_region defines the +resources for a new instance of fpga_bridge which isolates the parent from +child fpga_region. This new instance of fpga_bridge will be used when a +xclbin image is loaded on the child fpga_region. After the xclbin image is +downloaded to the fpga_region, an instance of a group is created for the +fpga_region using the device tree obtained as part of the xclbin. If this +device tree defines any child interfaces, it can trigger the creation of +fpga_bridge and fpga_region for the next region in the chain. + +fpga_bridge +----------- + +Like the fpga_region, an fpga_bridge is created by walking the device tree +of the parent group. The bridge is used for isolation between a parent and +its child. + +Driver Interfaces +================= + +xrt-mgmt Driver Ioctls +---------------------- + +Ioctls exposed by the xrt-mgmt driver to user space are enumerated in the +following table: + +== ===================== ============================ ========================== +# Functionality ioctl request code data format +== ===================== ============================ ========================== +1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf +== ===================== ============================ ========================== + +A user xclbin can be downloaded by using the xbmgmt tool from the XRT open source +suite. See example usage below:: + + xbmgmt partition --program --path /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verify.xclbin --force + +xrt-mgmt Driver Sysfs +---------------------- + +The xrt-mgmt driver exposes a rich set of sysfs interfaces. Subsystem xrt +drivers export sysfs node for every platform instance. + +Every partition also exports its UUIDs. See below for examples:: + + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids + + +hwmon +----- + +The xrt-mgmt driver exposes standard hwmon interface to report voltage, current, +temperature, power, etc. These can easily be viewed using *sensors* command line +utility. + +.. _alveo_platform_overview: + +Alveo Platform Overview +======================= + +Alveo platforms are architected as two physical FPGA partitions: *Shell* and +*User*. The Shell provides basic infrastructure for the Alveo platform like +PCIe connectivity, board management, Dynamic Function Exchange (DFX), sensors, +clocking, reset, and security. DFX, partial reconfiguration, is responsible for +loading the user compiled FPGA binary. + +For DFX to work properly, physical partitions require strict HW compatibility +with each other. Every physical partition has two interface UUIDs: the *parent* +UUID and the *child* UUID. For simple single stage platforms, Shell → User forms +the parent child relationship. + +.. note:: + Partition compatibility matching is a key design component of the Alveo platforms + and XRT. Partitions have child and parent relationship. A loaded partition + exposes child partition UUID to advertise its compatibility requirement. When + loading a child partition, the xrt-mgmt driver matches the parent + UUID of the child partition against the child UUID exported by the parent. + The parent and child partition UUIDs are stored in the *xclbin* (for the user) + and the *xsabin* (for the shell). Except for the root UUID exported by VSEC, + the hardware itself does not know about the UUIDs. The UUIDs are stored in + xsabin and xclbin. The image format has a special node called Partition UUIDs + which define the compatibility UUIDs. See :ref:`partition_uuids`. + + +The physical partitions and their loading are illustrated below:: + + SHELL USER + +-----------+ +-------------------+ + | | | | + | VSEC UUID | CHILD PARENT | LOGIC UUID | + | o------->|<--------o | + | | UUID UUID | | + +-----+-----+ +--------+----------+ + | | + . . + | | + +---+---+ +------+--------+ + | POR | | USER COMPILED | + | FLASH | | XCLBIN | + +-------+ +---------------+ + + +Loading Sequence +---------------- + +The Shell partition is loaded from flash at system boot time. It establishes the +PCIe link and exposes two physical functions to the BIOS. After the OS boots, +the xrt-mgmt driver attaches to the PCIe physical function 0 exposed by the Shell +and then looks for VSEC in the PCIe extended configuration space. Using VSEC, it +determines the logic UUID of the Shell and uses the UUID to load matching *xsabin* +file from Linux firmware directory. The xsabin file contains the metadata to +discover the peripherals that are part of the Shell and the firmware for any +embedded soft processors in the Shell. The xsabin file also contains Partition +UUIDs as described here :ref:`partition_uuids`. + +The Shell exports a child interface UUID which is used for the compatibility +check when loading the user compiled xclbin over the User partition as part of DFX. +When a user requests loading of a specific xclbin, the xrt-mgmt driver reads +the parent interface UUID specified in the xclbin and matches it with the child +interface UUID exported by the Shell to determine if the xclbin is compatible with +the Shell. If the match fails, loading of xclbin is denied. + +xclbin loading is requested using the ICAP_DOWNLOAD_AXLF ioctl command. When loading +a xclbin, the xrt-mgmt driver performs the following *logical* operations: + +1. Copy xclbin from user to kernel memory +2. Sanity check the xclbin contents +3. Isolate the User partition +4. Download the bitstream using the FPGA config engine (ICAP) +5. De-isolate the User partition +6. Program the clocks (ClockWiz) driving the User partition +7. Wait for the memory controller (MIG) calibration +8. Return the loading status back to the caller + +`Platform Loading Overview `_ +provides more detailed information on platform loading. + + +xsabin +------ + +Each Alveo platform comes packaged with its own xsabin. The xsabin is a trusted +component of the platform. For format details refer to :ref:`xsabin_xclbin_container_format` +below. xsabin contains basic information like UUIDs, platform name and metadata in the +form of device tree. See :ref:`device_tree_usage` below for details and example. + +xclbin +------ + +xclbin is compiled by end user using +`Vitis `_ +tool set from Xilinx. The xclbin contains sections describing user compiled +acceleration engines/kernels, memory subsystems, clocking information etc. It also +contains an FPGA bitstream for the user partition, UUIDs, platform name, etc. + + +.. _xsabin_xclbin_container_format: + +xsabin/xclbin Container Format +------------------------------ + +xclbin/xsabin is ELF-like binary container format. It is structured as series of +sections. There is a file header followed by several section headers which is +followed by sections. A section header points to an actual section. There is an +optional signature at the end. The format is defined by the header file ``xclbin.h``. +The following figure illustrates a typical xclbin:: + + + +---------------------+ + | | + | HEADER | + +---------------------+ + | SECTION HEADER | + | | + +---------------------+ + | ... | + | | + +---------------------+ + | SECTION HEADER | + | | + +---------------------+ + | SECTION | + | | + +---------------------+ + | ... | + | | + +---------------------+ + | SECTION | + | | + +---------------------+ + | SIGNATURE | + | (OPTIONAL) | + +---------------------+ + + +xclbin/xsabin files can be packaged, un-packaged and inspected using an XRT +utility called **xclbinutil**. xclbinutil is part of the XRT open source +software stack. The source code for xclbinutil can be found at +https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil + +For example, to enumerate the contents of a xclbin/xsabin use the *--info* switch +as shown below:: + + + xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin + xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin + + +.. _device_tree_usage: + +Device Tree Usage +----------------- + +The xsabin file stores metadata which advertise HW subsystems present in a +partition. The metadata is stored in device tree format with a well defined +schema. XRT management driver uses this information to bind *xrt_drivers* to +the subsystem instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel +module. + +Logic UUID +^^^^^^^^^^ +A partition is identified uniquely through ``logic_uuid`` property:: + + /dts-v1/; + / { + logic_uuid = "0123456789abcdef0123456789abcdef"; + ... + } + +Schema Version +^^^^^^^^^^^^^^ +Schema version is defined through the ``schema_version`` node. It contains +``major`` and ``minor`` properties as below:: + + /dts-v1/; + / { + schema_version { + major = <0x01>; + minor = <0x00>; + }; + ... + } + +.. _partition_uuids: + +Partition UUIDs +^^^^^^^^^^^^^^^ +Each partition may have parent and child UUIDs. These UUIDs are +defined by ``interfaces`` node and ``interface_uuid`` property:: + + /dts-v1/; + / { + interfaces { + @0 { + interface_uuid = "0123456789abcdef0123456789abcdef"; + }; + @1 { + interface_uuid = "fedcba9876543210fedcba9876543210"; + }; + ... + }; + ... + } + + +Subsystem Instantiations +^^^^^^^^^^^^^^^^^^^^^^^^ +Subsystem instantiations are captured as children of ``addressable_endpoints`` +node:: + + /dts-v1/; + / { + addressable_endpoints { + abc { + ... + }; + def { + ... + }; + ... + } + } + +Subnode 'abc' and 'def' are the name of subsystem nodes + +Subsystem Node +^^^^^^^^^^^^^^ +Each subsystem node and its properties define a hardware instance:: + + + addressable_endpoints { + abc { + reg = <0x00 0x1f05000 0x00 0x1000>> + pcie_physical_function = <0x0>; + pcie_bar_mapping = <0x2>; + compatible = "abc def"; + interrupts = <0x09 0x0c>; + firmware { + firmware_product_name = "abc" + firmware_branch_name = "def" + firmware_version_major = <1> + firmware_version_minor = <2> + }; + } + ... + } + +:reg: + Property defines an address range. `<0x00 0x1f05000 0x00 0x1000>` indicates + *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address length. +:pcie_physical_function: + Property specifies which PCIe physical function the subsystem node resides. + `<0x0>` implies physical function 0. +:pcie_bar_mapping: + Property specifies which PCIe BAR the subsystem node resides. `<0x2>` implies + BAR 2. A value of 0 means the property is not defined. +:compatible: + Property is a list of strings. The first string in the list specifies the exact + subsystem node. The following strings represent other devices that the device + is compatible with. +:interrupts: + Property specifies start and end interrupts for this subsystem node. + `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem. +:firmware: + Subnode defines the firmware required by this subsystem node. + +Alveo U50 Platform Example +^^^^^^^^^^^^^^^^^^^^^^^^^^ +:: + + /dts-v1/; + + /{ + logic_uuid = "f465b0a3ae8c64f619bc150384ace69b"; + + schema_version { + major = <0x01>; + minor = <0x00>; + }; + + interfaces { + + @0 { + interface_uuid = "862c7020a250293e32036f19956669e5"; + }; + }; + + addressable_endpoints { + + ep_blp_rom_00 { + reg = <0x00 0x1f04000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; + }; + + ep_card_flash_program_00 { + reg = <0x00 0x1f06000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi"; + interrupts = <0x03 0x03>; + }; + + ep_cmc_firmware_mem_00 { + reg = <0x00 0x1e20000 0x00 0x20000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; + + firmware { + firmware_product_name = "cmc"; + firmware_branch_name = "u50"; + firmware_version_major = <0x01>; + firmware_version_minor = <0x00>; + }; + }; + + ep_cmc_intc_00 { + reg = <0x00 0x1e03000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc"; + interrupts = <0x04 0x04>; + }; + + ep_cmc_mutex_00 { + reg = <0x00 0x1e02000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; + }; + + ep_cmc_regmap_00 { + reg = <0x00 0x1e08000 0x00 0x2000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; + + firmware { + firmware_product_name = "sc-fw"; + firmware_branch_name = "u50"; + firmware_version_major = <0x05>; + }; + }; + + ep_cmc_reset_00 { + reg = <0x00 0x1e01000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; + }; + + ep_ddr_mem_calib_00 { + reg = <0x00 0x63000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; + }; + + ep_debug_bscan_mgmt_00 { + reg = <0x00 0x1e90000 0x00 0x10000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge"; + }; + + ep_ert_base_address_00 { + reg = <0x00 0x21000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; + }; + + ep_ert_command_queue_mgmt_00 { + reg = <0x00 0x40000 0x00 0x10000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue"; + }; + + ep_ert_command_queue_user_00 { + reg = <0x00 0x40000 0x00 0x10000>; + pcie_physical_function = <0x01>; + compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue"; + }; + + ep_ert_firmware_mem_00 { + reg = <0x00 0x30000 0x00 0x8000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; + + firmware { + firmware_product_name = "ert"; + firmware_branch_name = "v20"; + firmware_version_major = <0x01>; + }; + }; + + ep_ert_intc_00 { + reg = <0x00 0x23000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc"; + interrupts = <0x05 0x05>; + }; + + ep_ert_reset_00 { + reg = <0x00 0x22000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; + }; + + ep_ert_sched_00 { + reg = <0x00 0x50000 0x00 0x1000>; + pcie_physical_function = <0x01>; + compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched"; + interrupts = <0x09 0x0c>; + }; + + ep_fpga_configuration_00 { + reg = <0x00 0x1e88000 0x00 0x8000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap"; + interrupts = <0x02 0x02>; + }; + + ep_icap_reset_00 { + reg = <0x00 0x1f07000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; + }; + + ep_msix_00 { + reg = <0x00 0x00 0x00 0x20000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-msix-1.0\0msix"; + pcie_bar_mapping = <0x02>; + }; + + ep_pcie_link_mon_00 { + reg = <0x00 0x1f05000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; + }; + + ep_pr_isolate_plp_00 { + reg = <0x00 0x1f01000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; + }; + + ep_pr_isolate_ulp_00 { + reg = <0x00 0x1000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; + }; + + ep_uuid_rom_00 { + reg = <0x00 0x64000 0x00 0x1000>; + pcie_physical_function = <0x00>; + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; + }; + + ep_xdma_00 { + reg = <0x00 0x00 0x00 0x10000>; + pcie_physical_function = <0x01>; + compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma"; + pcie_bar_mapping = <0x02>; + }; + }; + + } + + + +Deployment Models +================= + +Baremetal +--------- + +In bare-metal deployments, both MPF and UPF are visible and accessible. The +xrt-mgmt driver binds to MPF. The xrt-mgmt driver operations are privileged and +available to system administrator. The full stack is illustrated below:: + + HOST + + [XRT-MGMT] [XRT-USER] + | | + | | + +-----+ +-----+ + | MPF | | UPF | + | | | | + | PF0 | | PF1 | + +--+--+ +--+--+ + ......... ^................. ^.......... + | | + | PCIe DEVICE | + | | + +--+------------------+--+ + | SHELL | + | | + +------------------------+ + | USER | + | | + | | + | | + | | + +------------------------+ + + + +Virtualized +----------- + +In virtualized deployments, the privileged MPF is assigned to the host but the +unprivileged UPF is assigned to a guest VM via PCIe pass-through. The xrt-mgmt +driver in host binds to MPF. The xrt-mgmt driver operations are privileged and +only accessible to the MPF. The full stack is illustrated below:: + + + .............. + HOST . VM . + . . + [XRT-MGMT] . [XRT-USER] . + | . | . + | . | . + +-----+ . +-----+ . + | MPF | . | UPF | . + | | . | | . + | PF0 | . | PF1 | . + +--+--+ . +--+--+ . + ......... ^................. ^.......... + | | + | PCIe DEVICE | + | | + +--+------------------+--+ + | SHELL | + | | + +------------------------+ + | USER | + | | + | | + | | + | | + +------------------------+ + + + + + +Platform Security Considerations +================================ + +`Security of Alveo Platform `_ +discusses the deployment options and security implications in great detail. diff --git a/MAINTAINERS b/MAINTAINERS index 056966c9aac9..beeaf0257364 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -7274,6 +7274,17 @@ F: Documentation/fpga/ F: drivers/fpga/ F: include/linux/fpga/ +FPGA XRT DRIVERS +M: Lizhi Hou +R: Max Zhen +R: Sonal Santan +L: linux-fpga@vger.kernel.org +S: Supported +W: https://github.com/Xilinx/XRT +F: Documentation/fpga/xrt.rst +F: drivers/fpga/xrt/ +F: include/uapi/linux/xrt/ + FPU EMULATOR M: Bill Metzenthen S: Maintained From patchwork Mon Aug 2 16:05:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lizhi Hou X-Patchwork-Id: 490434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 081DDC432BE for ; Mon, 2 Aug 2021 16:07:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA68F60EE9 for ; Mon, 2 Aug 2021 16:07:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232374AbhHBQHT (ORCPT ); Mon, 2 Aug 2021 12:07:19 -0400 Received: from mail-mw2nam10on2087.outbound.protection.outlook.com ([40.107.94.87]:57775 "EHLO NAM10-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229946AbhHBQHS (ORCPT ); Mon, 2 Aug 2021 12:07:18 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fKa/ts7JPGo2ERNk60675vYudN6ZKjT/3MFF+yC2Sm7RknzfzetWjvkE9D27WJo/dXmsfyOsz93nIaZInM7BF8mBLcUMokOv/EAYOxt4rX3CfUHF1ZwZjMOeO3Qvlua7cQE5DeScHOW9UqtMOtxvGjxovlHLCNkr2yk3wtg0+mk56dZoHj8qo4CeOTAiZ8BqqODkXf4aLFwbCGM29fZAFS/16hR5nbtY0NfThQgF8jxIvfzDfK3syy0y85wRmWZWIa0ZD0Fg5l/lwMKVUIN4iuS2EejIMSOsNKAyQPMEx+oDfdKsBuc7iGGz6yxLxHVudQfAmwUQiK31wM++ovp5nQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PCX0uLyI8DyDALmjDSx/DCq6ZEGpSQm1m8kEhGOkgTs=; b=Tk4Jjqv1vaigyyniwPMeltvf4imy2ggkS/iANxj0uEnwTydUc1wAaOmauR4aEhVmxrRdHg+cObxS0LsoUcnmKC1ltejJiSOLsp3Xf2k1SGdY/WnbtdjrP7oufpK7f0ea82GSRh2RKG4Dke/8x+NE4XH13IkP4btuU3qHxO9+kM96yobfzkIzTcizvJrDuPb8rRU5Lb8hsFLENIhlEJLIhaOkA9GY7de/X+5lmGVVpdxTq+1vfA/Vs+t7m7t/lOuy+3wUvL6E+8vJYSDI/BZICvjggCwx4aHkoL7WTS7erx5yp4BRMvaYlKkcLq2I6RB45sb9f+p8Xts6hXDzb0UBFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=kernel.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PCX0uLyI8DyDALmjDSx/DCq6ZEGpSQm1m8kEhGOkgTs=; b=aTGb31eCvRZ7Q+kl79/M8slysLf86p0GsHY4kxM1HyfAOCrvGqtfGZcGhqSJmMb/wzkR6Lp5bWTIdZiPK3GgqcAQ0+qbEt/IFofeGnZ9DiZZyKFuCWkJv7Th7Wct5lrw4ZK1FK5+rhNErQkGyNQiv00nTNTPfBwZTRD/LTz2DNs= Received: from DM5PR21CA0011.namprd21.prod.outlook.com (2603:10b6:3:ac::21) by SN6PR02MB4910.namprd02.prod.outlook.com (2603:10b6:805:92::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.25; Mon, 2 Aug 2021 16:07:06 +0000 Received: from DM3NAM02FT047.eop-nam02.prod.protection.outlook.com (2603:10b6:3:ac:cafe::68) by DM5PR21CA0011.outlook.office365.com (2603:10b6:3:ac::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.0 via Frontend Transport; Mon, 2 Aug 2021 16:07:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; kernel.org; dkim=none (message not signed) header.d=none; kernel.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch01.xlnx.xilinx.com; Received: from xsj-pvapexch01.xlnx.xilinx.com (149.199.62.198) by DM3NAM02FT047.mail.protection.outlook.com (10.13.4.114) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:07:06 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch01.xlnx.xilinx.com (172.19.86.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 2 Aug 2021 09:06:30 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.2 via Frontend Transport; Mon, 2 Aug 2021 09:06:30 -0700 Envelope-to: mdf@kernel.org, robh@kernel.org, trix@redhat.com, devicetree@vger.kernel.org, linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org Received: from [10.23.145.232] (port=49058 helo=xsj-xw9400.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1mAaSU-0005BQ-6W; Mon, 02 Aug 2021 09:06:30 -0700 Received: by xsj-xw9400.xilinx.com (Postfix, from userid 21952) id 74C46601B26; Mon, 2 Aug 2021 09:05:30 -0700 (PDT) From: Lizhi Hou To: CC: Lizhi Hou , , , , , , , , , , , Max Zhen Subject: [PATCH V9 XRT Alveo 05/14] fpga: xrt: group driver Date: Mon, 2 Aug 2021 09:05:12 -0700 Message-ID: <20210802160521.331031-6-lizhi.hou@xilinx.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210802160521.331031-1-lizhi.hou@xilinx.com> References: <20210802160521.331031-1-lizhi.hou@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 56de68db-84df-4849-3855-08d955cf933a X-MS-TrafficTypeDiagnostic: SN6PR02MB4910: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:126; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: D0q7iYvY01sD4ALT+b5HegjjaBdw2k58BAD3OFEOrhtesl/GhnPAVvL8723Zmlx1RgYnTPqMMMwF/2/C5CdF5mXm9CIHt6imOZVqe31HqyGuVJmEGsuHC8qdFU68143jEDv0rjinzH0xpsKu6jJcdOQjxc5Vm9QEiwygryVGJi1Utbyco8vzD/JA6UVi8lPU0szdFuLyO2eNuitL0iZ8XbVttsV73gcO7z8chkBQfYZG2IkX25+4kN+W+JBaI9x+IZ62pBBQ4i2APavE6p5w1zu1bLA6AXB5d/+7A2GeqQsR2UKaJsqaWsFM7F241iAq2pyoJGv/YD34xpjoOJPCgJPo7OKQ1YhJybPs37yH+LnY0jk6bce5s1IjZIV8dlKE726VbCQpnw7IXlrCSnuaa52LoV/73kN1tfiCGsZl4wGEGgSUvR4S9DEizVEsgBNhPfzS82uH6qqZG1K1M4rLupzn0KeqvOkmJW5Vqh+EPhLbxenaaIXFKX87toh/3ZS9YbfMxuRUh4uggcC79kh9/vLLh8KubiTdkgdoxmNyWRk3PL2Cb6jTAeu3aS5Hqg0f+9hcPMILvur2aM0MnCg8T7AdvbNCyiC0irPyv5V0bHzKXBltVlT8YrzhGOP8syKDq49uIiN6nDejm0RUPECKWulhqXOk5H8nIS1vNS0mSA7U4Rr0HhVZfshx6eoy8vGKP7AulFTmdyRvDUIOo13VDw== X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch01.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(4636009)(376002)(346002)(39860400002)(396003)(136003)(46966006)(36840700001)(2906002)(36860700001)(6666004)(6266002)(186003)(107886003)(8676002)(44832011)(1076003)(336012)(26005)(8936002)(4326008)(6916009)(82310400003)(478600001)(47076005)(316002)(82740400003)(42186006)(70586007)(70206006)(7636003)(54906003)(83380400001)(36906005)(2616005)(5660300002)(356005)(426003)(36756003); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2021 16:07:06.5383 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 56de68db-84df-4849-3855-08d955cf933a X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch01.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: DM3NAM02FT047.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR02MB4910 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org group driver that manages life cycle of a bunch of leaf driver instances and bridges them with root. Signed-off-by: Sonal Santan Signed-off-by: Max Zhen Signed-off-by: Lizhi Hou Reviewed-by: Tom Rix --- drivers/fpga/xrt/include/group.h | 25 +++ drivers/fpga/xrt/lib/group.c | 278 +++++++++++++++++++++++++++++++ 2 files changed, 303 insertions(+) create mode 100644 drivers/fpga/xrt/include/group.h create mode 100644 drivers/fpga/xrt/lib/group.c diff --git a/drivers/fpga/xrt/include/group.h b/drivers/fpga/xrt/include/group.h new file mode 100644 index 000000000000..09e9d03f53fe --- /dev/null +++ b/drivers/fpga/xrt/include/group.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2020-2021 Xilinx, Inc. + * + * Authors: + * Cheng Zhen + */ + +#ifndef _XRT_GROUP_H_ +#define _XRT_GROUP_H_ + +#include "xleaf.h" + +/* + * Group driver leaf calls. + */ +enum xrt_group_leaf_cmd { + XRT_GROUP_GET_LEAF = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */ + XRT_GROUP_PUT_LEAF, + XRT_GROUP_INIT_CHILDREN, + XRT_GROUP_FINI_CHILDREN, + XRT_GROUP_TRIGGER_EVENT, +}; + +#endif /* _XRT_GROUP_H_ */ diff --git a/drivers/fpga/xrt/lib/group.c b/drivers/fpga/xrt/lib/group.c new file mode 100644 index 000000000000..b45f05449e0b --- /dev/null +++ b/drivers/fpga/xrt/lib/group.c @@ -0,0 +1,278 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Xilinx Alveo FPGA Group Driver + * + * Copyright (C) 2020-2021 Xilinx, Inc. + * + * Authors: + * Cheng Zhen + */ + +#include +#include "xleaf.h" +#include "subdev_pool.h" +#include "group.h" +#include "metadata.h" +#include "lib-drv.h" + +#define XRT_GRP "xrt_group" + +struct xrt_group { + struct xrt_device *xdev; + struct xrt_subdev_pool leaves; + bool leaves_created; + struct mutex lock; /* lock for group */ +}; + +static int xrt_grp_root_cb(struct device *dev, void *parg, + enum xrt_root_cmd cmd, void *arg) +{ + int rc; + struct xrt_device *xdev = + container_of(dev, struct xrt_device, dev); + struct xrt_group *xg = (struct xrt_group *)parg; + + switch (cmd) { + case XRT_ROOT_GET_LEAF_HOLDERS: { + struct xrt_root_get_holders *holders = + (struct xrt_root_get_holders *)arg; + rc = xrt_subdev_pool_get_holders(&xg->leaves, + holders->xpigh_xdev, + holders->xpigh_holder_buf, + holders->xpigh_holder_buf_len); + break; + } + default: + /* Forward parent call to root. */ + rc = xrt_subdev_root_request(xdev, cmd, arg); + break; + } + + return rc; +} + +/* + * Cut subdev's dtb from group's dtb based on passed-in endpoint descriptor. + * Return the subdev's dtb through dtbp, if found. + */ +static int xrt_grp_cut_subdev_dtb(struct xrt_group *xg, struct xrt_dev_endpoints *eps, + char *grp_dtb, char **dtbp) +{ + int ret, i, ep_count = 0; + char *dtb = NULL; + + ret = xrt_md_create(DEV(xg->xdev), &dtb); + if (ret) + return ret; + + for (i = 0; eps->xse_names[i].ep_name || eps->xse_names[i].compat; i++) { + const char *ep_name = eps->xse_names[i].ep_name; + const char *compat = eps->xse_names[i].compat; + + if (!ep_name) + xrt_md_get_compatible_endpoint(DEV(xg->xdev), grp_dtb, compat, &ep_name); + if (!ep_name) + continue; + + ret = xrt_md_copy_endpoint(DEV(xg->xdev), dtb, grp_dtb, ep_name, compat, NULL); + if (ret) + continue; + xrt_md_del_endpoint(DEV(xg->xdev), grp_dtb, ep_name, compat); + ep_count++; + } + /* Found enough endpoints, return the subdev's dtb. */ + if (ep_count >= eps->xse_min_ep) { + *dtbp = dtb; + return 0; + } + + /* Cleanup - Restore all endpoints that has been deleted, if any. */ + if (ep_count > 0) { + xrt_md_copy_endpoint(DEV(xg->xdev), grp_dtb, dtb, + XRT_MD_NODE_ENDPOINTS, NULL, NULL); + } + vfree(dtb); + *dtbp = NULL; + return 0; +} + +static int xrt_grp_create_leaves(struct xrt_group *xg) +{ + struct xrt_subdev_platdata *pdata = DEV_PDATA(xg->xdev); + struct xrt_dev_endpoints *eps = NULL; + int ret = 0, failed = 0; + enum xrt_subdev_id did; + char *grp_dtb = NULL; + unsigned long mlen; + + if (!pdata) + return -EINVAL; + + mlen = xrt_md_size(DEV(xg->xdev), pdata->xsp_dtb); + if (mlen == XRT_MD_INVALID_LENGTH) { + xrt_err(xg->xdev, "invalid dtb, len %ld", mlen); + return -EINVAL; + } + + mutex_lock(&xg->lock); + + if (xg->leaves_created) { + /* + * This is expected since caller does not keep track of the state of the group + * and may, in some cases, still try to create leaves after it has already been + * created. This special error code will let the caller know what is going on. + */ + mutex_unlock(&xg->lock); + return -EEXIST; + } + + grp_dtb = vmalloc(mlen); + if (!grp_dtb) { + mutex_unlock(&xg->lock); + return -ENOMEM; + } + + /* Create all leaves based on dtb. */ + xrt_info(xg->xdev, "bringing up leaves..."); + memcpy(grp_dtb, pdata->xsp_dtb, mlen); + for (did = 0; did < XRT_SUBDEV_NUM; did++) { + eps = xrt_drv_get_endpoints(did); + while (eps && eps->xse_names) { + char *dtb = NULL; + + ret = xrt_grp_cut_subdev_dtb(xg, eps, grp_dtb, &dtb); + if (ret) { + failed++; + xrt_err(xg->xdev, "failed to cut subdev dtb for drv %s: %d", + xrt_drv_name(did), ret); + } + if (!dtb) { + /* + * No more dtb to cut or bad things happened for this instance, + * switch to the next one. + */ + eps++; + continue; + } + + /* Found a dtb for this instance, let's add it. */ + ret = xrt_subdev_pool_add(&xg->leaves, did, xrt_grp_root_cb, xg, dtb); + if (ret < 0) { + /* + * It is not a fatal error here. Some functionality is not usable + * due to this missing device, but the error can be handled + * when the functionality is used. + */ + failed++; + xrt_err(xg->xdev, "failed to add %s: %d", xrt_drv_name(did), ret); + } + vfree(dtb); + /* Continue searching for the same instance from grp_dtb. */ + } + } + + xg->leaves_created = true; + vfree(grp_dtb); + mutex_unlock(&xg->lock); + return failed == 0 ? 0 : -ECHILD; +} + +static void xrt_grp_remove_leaves(struct xrt_group *xg) +{ + mutex_lock(&xg->lock); + + if (!xg->leaves_created) { + mutex_unlock(&xg->lock); + return; + } + + xrt_info(xg->xdev, "tearing down leaves..."); + xrt_subdev_pool_fini(&xg->leaves); + xg->leaves_created = false; + + mutex_unlock(&xg->lock); +} + +static int xrt_grp_probe(struct xrt_device *xdev) +{ + struct xrt_group *xg; + + xrt_info(xdev, "probing..."); + + xg = devm_kzalloc(&xdev->dev, sizeof(*xg), GFP_KERNEL); + if (!xg) + return -ENOMEM; + + xg->xdev = xdev; + mutex_init(&xg->lock); + xrt_subdev_pool_init(DEV(xdev), &xg->leaves); + xrt_set_drvdata(xdev, xg); + + return 0; +} + +static void xrt_grp_remove(struct xrt_device *xdev) +{ + struct xrt_group *xg = xrt_get_drvdata(xdev); + + xrt_info(xdev, "leaving..."); + xrt_grp_remove_leaves(xg); +} + +static int xrt_grp_leaf_call(struct xrt_device *xdev, u32 cmd, void *arg) +{ + int rc = 0; + struct xrt_group *xg = xrt_get_drvdata(xdev); + + switch (cmd) { + case XRT_XLEAF_EVENT: + /* Simply forward to every child. */ + xrt_subdev_pool_handle_event(&xg->leaves, + (struct xrt_event *)arg); + break; + case XRT_GROUP_GET_LEAF: { + struct xrt_root_get_leaf *get_leaf = + (struct xrt_root_get_leaf *)arg; + + rc = xrt_subdev_pool_get(&xg->leaves, get_leaf->xpigl_match_cb, + get_leaf->xpigl_match_arg, + DEV(get_leaf->xpigl_caller_xdev), + &get_leaf->xpigl_tgt_xdev); + break; + } + case XRT_GROUP_PUT_LEAF: { + struct xrt_root_put_leaf *put_leaf = + (struct xrt_root_put_leaf *)arg; + + rc = xrt_subdev_pool_put(&xg->leaves, put_leaf->xpipl_tgt_xdev, + DEV(put_leaf->xpipl_caller_xdev)); + break; + } + case XRT_GROUP_INIT_CHILDREN: + rc = xrt_grp_create_leaves(xg); + break; + case XRT_GROUP_FINI_CHILDREN: + xrt_grp_remove_leaves(xg); + break; + case XRT_GROUP_TRIGGER_EVENT: + xrt_subdev_pool_trigger_event(&xg->leaves, (enum xrt_events)(uintptr_t)arg); + break; + default: + xrt_err(xdev, "unknown IOCTL cmd %d", cmd); + rc = -EINVAL; + break; + } + return rc; +} + +static struct xrt_driver xrt_group_driver = { + .driver = { + .name = XRT_GRP, + }, + .subdev_id = XRT_SUBDEV_GRP, + .probe = xrt_grp_probe, + .remove = xrt_grp_remove, + .leaf_call = xrt_grp_leaf_call, +}; + +XRT_LEAF_INIT_FINI_FUNC(group); From patchwork Mon Aug 2 16:05:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lizhi Hou X-Patchwork-Id: 490435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7B84C432BE for ; Mon, 2 Aug 2021 16:06:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CF7BA60E97 for ; Mon, 2 Aug 2021 16:06:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231357AbhHBQGg (ORCPT ); Mon, 2 Aug 2021 12:06:36 -0400 Received: from mail-mw2nam12on2040.outbound.protection.outlook.com ([40.107.244.40]:31968 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229805AbhHBQGg (ORCPT ); Mon, 2 Aug 2021 12:06:36 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KV/jCDwRKplks3UPWTxQEu9gMozUeIHRXP1A9QTsD0d2xgzfUHI1cfrlLcPn73RtXsstOTAN70mRZwoijaDp9VCVdiAmBBOQpJql6yUlO+mYM9MH9lYfIu3qyf4U7z3EFO6LDjIZfZpvAGWi/vEKGVxVgSwkqt/KKEjgIgSKCQayfgCEi2XJfIK0Aal+8zbAs8Usg2VKzPORSjPpILWNOTx/cKvskrEDic6YWMPtVpGNWaEoZGHESj4tI1gterlHSsYt+RTN4VSXJI5HUEUmR8fD3t2OyR0clrwUGgXWTRvRE12xvSLGvKhjn/Tcu9f4VALqsIY6XUSBBQlX++Z+Tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FJGnxQJe6FQRgxaRo8LXtiTIBNZmF318AW7uIhkSTbU=; b=ksv0Q2H/c7smeQG6hdUerUrTsr38CtXp8TFQtYpvr0tfnafSdmJMwPIMNFiMICa944zPAYkN4jvWESKVpkU6TSVj32vDGLfbX43lpwr2Hn/q3k2YqV8zgKqBSopCTl6kI2ikHKTPKdSmtRr3ChxrWK4TM3LjsM+cfLQYq6CvKrbuPFs11gYCCwicVo1xb4oCarK4aolOLIfug2TNOorJiz5nR6kvBXwgXj25Qc2uaPofuLR9ky1CgOXsjSqvm9rpf6dy5aW0tOIQmwYmkhnfb/gDGYeV4jZpqy+C7asoKVr97OM78wf0d5H7aAegHQitrkFe+rmxOKBcWj8bow5z1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=kernel.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FJGnxQJe6FQRgxaRo8LXtiTIBNZmF318AW7uIhkSTbU=; b=VFUb7h37fZP+pWI4sjmT9FnwseJ7obpmtVPCTvzuzf8WG9orA2o0dUnX/JjGkHwJtdU5gcXBhZXHZnJ0H3xa8jU0aeMKy/F0gLKhgklu609S7nJUyoP6V+guVXXBqmNE9qxueqhgg5rOHqS3uwiX0OiAeP8yL4LjpJCn1fsQMWg= Received: from DM6PR05CA0056.namprd05.prod.outlook.com (2603:10b6:5:335::25) by SN6PR02MB4831.namprd02.prod.outlook.com (2603:10b6:805:94::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.26; Mon, 2 Aug 2021 16:06:23 +0000 Received: from DM3NAM02FT040.eop-nam02.prod.protection.outlook.com (2603:10b6:5:335:cafe::35) by DM6PR05CA0056.outlook.office365.com (2603:10b6:5:335::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4394.12 via Frontend Transport; Mon, 2 Aug 2021 16:06:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; kernel.org; dkim=none (message not signed) header.d=none; kernel.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by DM3NAM02FT040.mail.protection.outlook.com (10.13.5.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:06:23 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 2 Aug 2021 09:06:18 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.2 via Frontend Transport; Mon, 2 Aug 2021 09:06:18 -0700 Envelope-to: mdf@kernel.org, robh@kernel.org, trix@redhat.com, devicetree@vger.kernel.org, linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org Received: from [10.23.145.232] (port=49056 helo=xsj-xw9400.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1mAaSI-0005AZ-53; Mon, 02 Aug 2021 09:06:18 -0700 Received: by xsj-xw9400.xilinx.com (Postfix, from userid 21952) id 5D9D9601B23; Mon, 2 Aug 2021 09:05:30 -0700 (PDT) From: Lizhi Hou To: CC: Lizhi Hou , , , , , , , , , , , Max Zhen Subject: [PATCH V9 XRT Alveo 06/14] fpga: xrt: char dev node helper functions Date: Mon, 2 Aug 2021 09:05:13 -0700 Message-ID: <20210802160521.331031-7-lizhi.hou@xilinx.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210802160521.331031-1-lizhi.hou@xilinx.com> References: <20210802160521.331031-1-lizhi.hou@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e7314606-0168-4eba-7f01-08d955cf797d X-MS-TrafficTypeDiagnostic: SN6PR02MB4831: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:67; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: X5Gvclhc1fH37WPIQ8JGWYa/NPeOC2X/Q+6E9p7ShpA4ovyLACGb0SG4iYzA60PQbSQXbAZ0lUn/u+SokMZDFheBC+zXm9vmelvCBzcmhBMC9e4IuW14ZQH05ds64Ckp25fZUyhPjvYkTYKyNf3OzlvHiUawvU+foVtmXrRpZ7860qmK+DBbRaPKgfx4c2M2z252vsnYi/qlzhFVCu6vPNDJvjhv7DnyAHmvNuO/pB+2JO60cKaPRWia7UDAZeb1G6fPt6ZmqLahusX6nxN3Hv49Ir0tkSI7wBa21d0BbE7jHZ6GYaOwun3Uanb233qdsI4AYxsa8xNPT4L+hpRZaWknxloH6RVaiutZBoBiY6fFCRssCoh/UAfnEjEAiP2W7sheDKGxdfQPjNKYED5pVf2u+f4urX6K4tUVZQTAxNAH9G24iZmiNoq75TxQ3V9xJgjmDFZc/DM0ByXLFMEw/1ftXhtboRex7nYLapa3UhvqGrXiOAS9nWG4ZvsDKAPVqL/WwEzSNXODk9x0WrTxCjPE39RCg453jJW3E3vvmMtdzWK0jBmF9Kp52zuiSuRtDWPiMrX3WemVgFpb8nBFZfoXoHEP+NFnA0MiUOu0axEshcc3o4rhaKIhLd1OA+P1buia0BP1DI6uoPI1LNW6wH2yINVqb8XCGbYrLXR3PplsTBxk3c6ovn5/sE7eYJ9SYZMVpXh8JL+t7Yvm2hxaqg== X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(4636009)(39860400002)(376002)(396003)(346002)(136003)(46966006)(36840700001)(8676002)(82740400003)(83380400001)(2906002)(356005)(186003)(6666004)(8936002)(5660300002)(36756003)(336012)(47076005)(26005)(82310400003)(478600001)(4326008)(107886003)(36906005)(316002)(42186006)(6916009)(70586007)(70206006)(54906003)(44832011)(2616005)(7636003)(426003)(1076003)(6266002)(36860700001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2021 16:06:23.3585 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e7314606-0168-4eba-7f01-08d955cf797d X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: DM3NAM02FT040.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR02MB4831 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Helper functions for char device node creation / removal for xrt drivers. This is part of xrt driver infrastructure. Signed-off-by: Sonal Santan Signed-off-by: Max Zhen Signed-off-by: Lizhi Hou Reviewed-by: Tom Rix --- drivers/fpga/xrt/lib/cdev.c | 209 ++++++++++++++++++++++++++++++++++++ 1 file changed, 209 insertions(+) create mode 100644 drivers/fpga/xrt/lib/cdev.c diff --git a/drivers/fpga/xrt/lib/cdev.c b/drivers/fpga/xrt/lib/cdev.c new file mode 100644 index 000000000000..3c20adac8c03 --- /dev/null +++ b/drivers/fpga/xrt/lib/cdev.c @@ -0,0 +1,209 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Xilinx Alveo FPGA device node helper functions. + * + * Copyright (C) 2020-2021 Xilinx, Inc. + * + * Authors: + * Cheng Zhen + */ + +#include "xleaf.h" +#include "lib-drv.h" + +#define XRT_CDEV_DIR "xrt" +#define INODE2PDATA(inode) \ + container_of((inode)->i_cdev, struct xrt_subdev_platdata, xsp_cdev) +#define INODE2PDEV(inode) \ + to_xrt_dev(kobj_to_dev((inode)->i_cdev->kobj.parent)) +#define CDEV_NAME(sysdev) (strchr((sysdev)->kobj.name, '!') + 1) + +/* Allow it to be accessed from cdev. */ +static void xleaf_devnode_allowed(struct xrt_device *xdev) +{ + struct xrt_subdev_platdata *pdata = DEV_PDATA(xdev); + + /* Allow new opens. */ + mutex_lock(&pdata->xsp_devnode_lock); + pdata->xsp_devnode_online = true; + mutex_unlock(&pdata->xsp_devnode_lock); +} + +/* Turn off access from cdev and wait for all existing user to go away. */ +static void xleaf_devnode_disallowed(struct xrt_device *xdev) +{ + struct xrt_subdev_platdata *pdata = DEV_PDATA(xdev); + + mutex_lock(&pdata->xsp_devnode_lock); + + /* Prevent new opens. */ + pdata->xsp_devnode_online = false; + /* Wait for existing user to close. */ + while (pdata->xsp_devnode_ref) { + mutex_unlock(&pdata->xsp_devnode_lock); + wait_for_completion(&pdata->xsp_devnode_comp); + mutex_lock(&pdata->xsp_devnode_lock); + } + + mutex_unlock(&pdata->xsp_devnode_lock); +} + +static struct xrt_device * +__xleaf_devnode_open(struct inode *inode, bool excl) +{ + struct xrt_subdev_platdata *pdata = INODE2PDATA(inode); + struct xrt_device *xdev = INODE2PDEV(inode); + bool opened = false; + + mutex_lock(&pdata->xsp_devnode_lock); + + if (pdata->xsp_devnode_online) { + if (excl && pdata->xsp_devnode_ref) { + xrt_err(xdev, "%s has already been opened exclusively", + CDEV_NAME(pdata->xsp_sysdev)); + } else if (!excl && pdata->xsp_devnode_excl) { + xrt_err(xdev, "%s has been opened exclusively", + CDEV_NAME(pdata->xsp_sysdev)); + } else { + pdata->xsp_devnode_ref++; + pdata->xsp_devnode_excl = excl; + opened = true; + xrt_info(xdev, "opened %s, ref=%d", + CDEV_NAME(pdata->xsp_sysdev), + pdata->xsp_devnode_ref); + } + } else { + xrt_err(xdev, "%s is offline", CDEV_NAME(pdata->xsp_sysdev)); + } + + mutex_unlock(&pdata->xsp_devnode_lock); + + xdev = opened ? xdev : NULL; + return xdev; +} + +struct xrt_device * +xleaf_devnode_open_excl(struct inode *inode) +{ + return __xleaf_devnode_open(inode, true); +} + +struct xrt_device * +xleaf_devnode_open(struct inode *inode) +{ + return __xleaf_devnode_open(inode, false); +} +EXPORT_SYMBOL_GPL(xleaf_devnode_open); + +void xleaf_devnode_close(struct inode *inode) +{ + struct xrt_subdev_platdata *pdata = INODE2PDATA(inode); + struct xrt_device *xdev = INODE2PDEV(inode); + bool notify = false; + + mutex_lock(&pdata->xsp_devnode_lock); + + WARN_ON(pdata->xsp_devnode_ref == 0); + pdata->xsp_devnode_ref--; + if (pdata->xsp_devnode_ref == 0) { + pdata->xsp_devnode_excl = false; + notify = true; + } + if (notify) + xrt_info(xdev, "closed %s", CDEV_NAME(pdata->xsp_sysdev)); + else + xrt_info(xdev, "closed %s, notifying waiter", CDEV_NAME(pdata->xsp_sysdev)); + + mutex_unlock(&pdata->xsp_devnode_lock); + + if (notify) + complete(&pdata->xsp_devnode_comp); +} +EXPORT_SYMBOL_GPL(xleaf_devnode_close); + +static inline enum xrt_dev_file_mode +devnode_mode(struct xrt_device *xdev) +{ + return DEV_FILE_OPS(xdev)->xsf_mode; +} + +int xleaf_devnode_create(struct xrt_device *xdev, const char *file_name, + const char *inst_name) +{ + struct xrt_subdev_platdata *pdata = DEV_PDATA(xdev); + struct xrt_dev_file_ops *fops = DEV_FILE_OPS(xdev); + struct cdev *cdevp; + struct device *sysdev; + int ret = 0; + char fname[256]; + + mutex_init(&pdata->xsp_devnode_lock); + init_completion(&pdata->xsp_devnode_comp); + + cdevp = &DEV_PDATA(xdev)->xsp_cdev; + cdev_init(cdevp, &fops->xsf_ops); + cdevp->owner = fops->xsf_ops.owner; + cdevp->dev = MKDEV(MAJOR(fops->xsf_dev_t), xdev->instance); + + /* + * Set xdev as parent of cdev so that when xdev (and its platform + * data) will not be freed when cdev is not freed. + */ + cdev_set_parent(cdevp, &DEV(xdev)->kobj); + + ret = cdev_add(cdevp, cdevp->dev, 1); + if (ret) { + xrt_err(xdev, "failed to add cdev: %d", ret); + goto failed; + } + if (!file_name) + file_name = xdev->name; + if (!inst_name) { + if (devnode_mode(xdev) == XRT_DEV_FILE_MULTI_INST) { + snprintf(fname, sizeof(fname), "%s/%s/%s.%u", + XRT_CDEV_DIR, DEV_PDATA(xdev)->xsp_root_name, + file_name, xdev->instance); + } else { + snprintf(fname, sizeof(fname), "%s/%s/%s", + XRT_CDEV_DIR, DEV_PDATA(xdev)->xsp_root_name, + file_name); + } + } else { + snprintf(fname, sizeof(fname), "%s/%s/%s.%s", XRT_CDEV_DIR, + DEV_PDATA(xdev)->xsp_root_name, file_name, inst_name); + } + sysdev = device_create(xrt_class, NULL, cdevp->dev, NULL, "%s", fname); + if (IS_ERR(sysdev)) { + ret = PTR_ERR(sysdev); + xrt_err(xdev, "failed to create device node: %d", ret); + goto failed_cdev_add; + } + pdata->xsp_sysdev = sysdev; + + xleaf_devnode_allowed(xdev); + + xrt_info(xdev, "created (%d, %d): /dev/%s", + MAJOR(cdevp->dev), xdev->instance, fname); + return 0; + +failed_cdev_add: + cdev_del(cdevp); +failed: + cdevp->owner = NULL; + return ret; +} + +void xleaf_devnode_destroy(struct xrt_device *xdev) +{ + struct xrt_subdev_platdata *pdata = DEV_PDATA(xdev); + struct cdev *cdevp = &pdata->xsp_cdev; + dev_t dev = cdevp->dev; + + xleaf_devnode_disallowed(xdev); + + xrt_info(xdev, "removed (%d, %d): /dev/%s/%s", MAJOR(dev), MINOR(dev), + XRT_CDEV_DIR, CDEV_NAME(pdata->xsp_sysdev)); + device_destroy(xrt_class, cdevp->dev); + pdata->xsp_sysdev = NULL; + cdev_del(cdevp); +} From patchwork Mon Aug 2 16:05:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lizhi Hou X-Patchwork-Id: 490432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A64FC432BE for ; Mon, 2 Aug 2021 16:07:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 418BA60F58 for ; Mon, 2 Aug 2021 16:07:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232573AbhHBQHm (ORCPT ); Mon, 2 Aug 2021 12:07:42 -0400 Received: from mail-bn7nam10on2043.outbound.protection.outlook.com ([40.107.92.43]:42258 "EHLO NAM10-BN7-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S232594AbhHBQHl (ORCPT ); Mon, 2 Aug 2021 12:07:41 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AeEuG9qLYiCQRoghBlHm8P9+tPgxbpfd6IgJDt3EDkbW/M9sSwaou0A+2q9+kI9uIRSPa/7IQOVCAnGz6/zoH4CaFb3bRwMHuTPSBH2avj4TcNNcbIbKRdiglZwYtALBbCksFf7e+qzEQdEbjQIFW9SsUl/5aG9RpgqOYIKEuSi597KzVijLuYeCJg14ZR0WjevOMudAy2O7M5ODlo+D2kCn0/gA6Sp3PQjDW0wPW5Yt8C/phObY42jpvsp9L7DEmXX86Cci6j2qyAvX70VS7m6uu8jThWv2F9L1QEfa9zE16sL7iin2eaWzzI65I483+PMCgjXFouh6SXLy7oZq6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=t4phI/p2XdBLZgLW4Q+NuAnAlvtjXs4aJPXk/LWbEFQ=; b=hBsa4MmoxdZAuZco8icPi1XlKcBsrVihv+TysWFn9ZIzmKYCtOGipCqg9IPzBtXY+GBpKk5lvkG7bbZwBsgn7E8kSu/qCbjfhHJzJpfOj9H5pel9KH7UCBt2upP0FM6djRuhDER5G1e+gqHSbm8PFvmVegaBNv47vrhqHfBWAhts+gVFFBA6YcMHmbGTGLO7JQeFKtNtB/0mZvwLFBvg/s+nn/wX16Q95kcW2ZQI3zfBjHSv28vUD9uEYNvBZJYCU7lscsGPgROr+sfhMeCP+ztsXvF+PBmhKxp4GGsTS7ITIpcrlWkAZ6F2SL4xZ4S0MtCuVrqYxmeQQQtxT8ewJQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=kernel.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=t4phI/p2XdBLZgLW4Q+NuAnAlvtjXs4aJPXk/LWbEFQ=; b=LVT6VN42IiYO788Z7B5xrEGgNvH+cVi/W0+j6zbtCjcoT7wA3UiiZaICR5JyPn+jy7eMZQfvkVzYAYQu3Godu+/RoC6YNNGNnjCxsWopAxZTFcRlrjV4YnM5G3frkbK8uDJIvkRoRcVrBrR/eawSuqIOWLv3txdQs6a2esFPdnY= Received: from DM6PR08CA0005.namprd08.prod.outlook.com (2603:10b6:5:80::18) by PH0PR02MB8762.namprd02.prod.outlook.com (2603:10b6:510:da::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.21; Mon, 2 Aug 2021 16:07:29 +0000 Received: from DM3NAM02FT062.eop-nam02.prod.protection.outlook.com (2603:10b6:5:80:cafe::b0) by DM6PR08CA0005.outlook.office365.com (2603:10b6:5:80::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:07:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; kernel.org; dkim=none (message not signed) header.d=none; kernel.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by DM3NAM02FT062.mail.protection.outlook.com (10.13.5.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:07:29 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 2 Aug 2021 09:07:06 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.2 via Frontend Transport; Mon, 2 Aug 2021 09:07:06 -0700 Envelope-to: mdf@kernel.org, robh@kernel.org, trix@redhat.com, devicetree@vger.kernel.org, linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org Received: from [10.23.145.232] (port=49064 helo=xsj-xw9400.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1mAaT4-0005Gp-B7; Mon, 02 Aug 2021 09:07:06 -0700 Received: by xsj-xw9400.xilinx.com (Postfix, from userid 21952) id BF791601B33; Mon, 2 Aug 2021 09:05:30 -0700 (PDT) From: Lizhi Hou To: CC: Lizhi Hou , , , , , , , , , , , Max Zhen Subject: [PATCH V9 XRT Alveo 08/14] fpga: xrt: driver infrastructure Date: Mon, 2 Aug 2021 09:05:15 -0700 Message-ID: <20210802160521.331031-9-lizhi.hou@xilinx.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210802160521.331031-1-lizhi.hou@xilinx.com> References: <20210802160521.331031-1-lizhi.hou@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: de025c97-f955-433c-b046-08d955cfa0bc X-MS-TrafficTypeDiagnostic: PH0PR02MB8762: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:525; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: faI2NTJO072WtGVt5S5cy8oy1Ekj2RoJxlgtX5Xdpdrn0fU1rdyfXh+3uVdN9UcR3+CvZhHHSTlXQqM24BcmjE+xxbncQdG0k9H/kTCeUc6LvOy+9Oq85pPDDFqv/8tS0OA7VwOCC4T8XV1/VX/9H40YPyb+6c9Arzqd0GYuBYp3MZg7HAO0yT0tj8PX8B0NNiLKasxVYQgG0VvQhGxHrcDW15PinoNyACA4DRd94HoS9g/0jXfYJKtJCBV0B9RxyC8ezRcwYD+gsjW4tPW5sVEVMMaBqcPjbiwqWae7P3NuszeGY1pPYWWw16fUF7u4QTdeQ3RbGFisOez4AcP51UszVudVpHRUP3CS4Z6Fk7tOB1C46JK39Xl27xV9VAKvHvxjUCo1cPznQ6D87iSOJXe4Uh7VUfbFGtf9zR4FeuFIaKz55BTv/BvcsDBDQGESre7g3sa85wErKI3OPKDmqx2Z+nH22+2GHRXFxxU0mh5sW3fTpcoN57zlX8wHUVxbbzxhEJrYdBZfn/GBOWAGghD0MZylEw7u+nsBzlj2m75fAcArgLK8OYF//C0brUSDNAD/0Rs8tHz1fk424N4f1enZ5y5PKNo9mQKvc7YDoHJtp5Hq3/kon112zzaakiHwwo03+S/mFLRD7IfknYXISpFkgdzI+P0B2q8Htlyv+7DuCmrCZYz+IfJFVVTAaua+5PZLLsiHsx3bHNx47RwcJA== X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(4636009)(39860400002)(376002)(396003)(346002)(136003)(46966006)(36840700001)(1076003)(36756003)(82310400003)(478600001)(44832011)(6666004)(6916009)(336012)(8936002)(4326008)(7636003)(82740400003)(316002)(36860700001)(42186006)(36906005)(54906003)(6266002)(107886003)(426003)(2616005)(26005)(47076005)(2906002)(5660300002)(186003)(30864003)(356005)(83380400001)(70586007)(8676002)(70206006); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2021 16:07:29.2047 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: de025c97-f955-433c-b046-08d955cfa0bc X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: DM3NAM02FT062.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR02MB8762 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Infrastructure code providing APIs for managing leaf driver instance groups, facilitating inter-leaf driver calls and root calls. Signed-off-by: Sonal Santan Signed-off-by: Max Zhen Signed-off-by: Lizhi Hou Reviewed-by: Tom Rix --- drivers/fpga/xrt/lib/subdev.c | 859 ++++++++++++++++++++++++++++++++++ 1 file changed, 859 insertions(+) create mode 100644 drivers/fpga/xrt/lib/subdev.c diff --git a/drivers/fpga/xrt/lib/subdev.c b/drivers/fpga/xrt/lib/subdev.c new file mode 100644 index 000000000000..350f67d5eb1e --- /dev/null +++ b/drivers/fpga/xrt/lib/subdev.c @@ -0,0 +1,859 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2020-2021 Xilinx, Inc. + * + * Authors: + * Cheng Zhen + */ + +#include +#include +#include "xleaf.h" +#include "subdev_pool.h" +#include "lib-drv.h" +#include "metadata.h" + +extern struct bus_type xrt_bus_type; + +#define IS_ROOT_DEV(dev) ((dev)->bus != &xrt_bus_type) +#define XRT_HOLDER_BUF_SZ 1024 + +static inline struct device *find_root(struct xrt_device *xdev) +{ + struct device *d = DEV(xdev); + + while (!IS_ROOT_DEV(d)) + d = d->parent; + return d; +} + +/* + * It represents a holder of a subdev. One holder can repeatedly hold a subdev + * as long as there is a unhold corresponding to a hold. + */ +struct xrt_subdev_holder { + struct list_head xsh_holder_list; + struct device *xsh_holder; + int xsh_count; + struct kref xsh_kref; +}; + +/* + * It represents a specific instance of platform driver for a subdev, which + * provides services to its clients (another subdev driver or root driver). + */ +struct xrt_subdev { + struct list_head xs_dev_list; + struct list_head xs_holder_list; + enum xrt_subdev_id xs_id; /* type of subdev */ + struct xrt_device *xs_xdev; + struct completion xs_holder_comp; +}; + +static struct xrt_subdev *xrt_subdev_alloc(void) +{ + struct xrt_subdev *sdev = kzalloc(sizeof(*sdev), GFP_KERNEL); + + if (!sdev) + return NULL; + + INIT_LIST_HEAD(&sdev->xs_dev_list); + INIT_LIST_HEAD(&sdev->xs_holder_list); + init_completion(&sdev->xs_holder_comp); + return sdev; +} + +int xrt_subdev_root_request(struct xrt_device *self, u32 cmd, void *arg) +{ + struct device *dev = DEV(self); + struct xrt_subdev_platdata *pdata = DEV_PDATA(self); + + if (!pdata->xsp_root_cb) { + dev_err(dev, "invalid root callback"); + return -EINVAL; + } + return (*pdata->xsp_root_cb)(dev->parent, pdata->xsp_root_cb_arg, cmd, arg); +} + +/* + * Subdev common sysfs nodes. + */ +static ssize_t holders_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + ssize_t len; + struct xrt_device *xdev = to_xrt_dev(dev); + struct xrt_root_get_holders holders = { xdev, buf, XRT_HOLDER_BUF_SZ }; + + len = xrt_subdev_root_request(xdev, XRT_ROOT_GET_LEAF_HOLDERS, &holders); + if (len >= holders.xpigh_holder_buf_len) + return len; + buf[len] = '\n'; + return len + 1; +} +static DEVICE_ATTR_RO(holders); + +static struct attribute *xrt_subdev_attrs[] = { + &dev_attr_holders.attr, + NULL, +}; + +static ssize_t metadata_output(struct file *filp, struct kobject *kobj, + struct bin_attribute *attr, char *buf, loff_t off, size_t count) +{ + struct device *dev = kobj_to_dev(kobj); + struct xrt_device *xdev = to_xrt_dev(dev); + struct xrt_subdev_platdata *pdata = DEV_PDATA(xdev); + unsigned char *blob; + unsigned long size; + ssize_t ret = 0; + + blob = pdata->xsp_dtb; + size = xrt_md_size(dev, blob); + if (size == XRT_MD_INVALID_LENGTH) { + ret = -EINVAL; + goto failed; + } + + if (off >= size) { + dev_dbg(dev, "offset (%lld) beyond total size: %ld\n", off, size); + goto failed; + } + + if (off + count > size) { + dev_dbg(dev, "count (%ld) beyond left bytes: %lld\n", + (unsigned long)count, size - off); + count = size - off; + } + memcpy(buf, blob + off, count); + + ret = count; +failed: + return ret; +} + +static struct bin_attribute meta_data_attr = { + .attr = { + .name = "metadata", + .mode = 0400 + }, + .read = metadata_output, + .size = 0 +}; + +static struct bin_attribute *xrt_subdev_bin_attrs[] = { + &meta_data_attr, + NULL, +}; + +static const struct attribute_group xrt_subdev_attrgroup = { + .attrs = xrt_subdev_attrs, + .bin_attrs = xrt_subdev_bin_attrs, +}; + +/* + * Given the device metadata, parse it to get IO ranges and construct + * resource array. + */ +static int +xrt_subdev_getres(struct device *parent, enum xrt_subdev_id id, + char *dtb, struct resource **res, int *res_num) +{ + struct xrt_subdev_platdata *pdata; + struct resource *pci_res = NULL; + const __be64 *bar_range; + const __be32 *bar_idx; + char *ep_name = NULL, *compat = NULL; + uint bar; + int count1 = 0, count2 = 0, ret; + + if (!dtb) + return -EINVAL; + + pdata = DEV_PDATA(to_xrt_dev(parent)); + + /* go through metadata and count endpoints in it */ + xrt_md_get_next_endpoint(parent, dtb, NULL, NULL, &ep_name, &compat); + while (ep_name) { + ret = xrt_md_get_prop(parent, dtb, ep_name, compat, + XRT_MD_PROP_IO_OFFSET, (const void **)&bar_range, NULL); + if (!ret) + count1++; + xrt_md_get_next_endpoint(parent, dtb, ep_name, compat, &ep_name, &compat); + } + if (!count1) + return 0; + + /* allocate resource array for all endpoints been found in metadata */ + *res = vzalloc(sizeof(**res) * count1); + + /* go through all endpoints again and get IO range for each endpoint */ + ep_name = NULL; + xrt_md_get_next_endpoint(parent, dtb, NULL, NULL, &ep_name, &compat); + while (ep_name) { + ret = xrt_md_get_prop(parent, dtb, ep_name, compat, + XRT_MD_PROP_IO_OFFSET, (const void **)&bar_range, NULL); + if (ret) + continue; + xrt_md_get_prop(parent, dtb, ep_name, compat, + XRT_MD_PROP_BAR_IDX, (const void **)&bar_idx, NULL); + bar = bar_idx ? be32_to_cpu(*bar_idx) : 0; + xleaf_get_root_res(to_xrt_dev(parent), bar, &pci_res); + if (!pci_res) { + dev_err(parent, "Invalid bar defined %d", bar); + ret = -EINVAL; + goto failed; + } + (*res)[count2].start = pci_res->start + be64_to_cpu(bar_range[0]); + (*res)[count2].end = pci_res->start + be64_to_cpu(bar_range[0]) + + be64_to_cpu(bar_range[1]) - 1; + (*res)[count2].flags = IORESOURCE_MEM; + /* check if there is conflicted resource */ + ret = request_resource(pci_res, *res + count2); + if (ret) { + dev_err(parent, "Conflict resource %pR\n", *res + count2); + goto failed; + } + release_resource(*res + count2); + + (*res)[count2].parent = pci_res; + + xrt_md_find_endpoint(parent, pdata->xsp_dtb, ep_name, + compat, &(*res)[count2].name); + + count2++; + xrt_md_get_next_endpoint(parent, dtb, ep_name, compat, &ep_name, &compat); + } + + WARN_ON(count1 != count2); + *res_num = count2; + + return 0; + +failed: + vfree(*res); + *res_num = 0; + *res = NULL; + return ret; +} + +static inline enum xrt_dev_file_mode +xleaf_devnode_mode(struct xrt_device *xdev) +{ + return DEV_FILE_OPS(xdev)->xsf_mode; +} + +static bool xrt_subdev_cdev_auto_creation(struct xrt_device *xdev) +{ + enum xrt_dev_file_mode mode = xleaf_devnode_mode(xdev); + + if (!xleaf_devnode_enabled(xdev)) + return false; + + return (mode == XRT_DEV_FILE_DEFAULT || mode == XRT_DEV_FILE_MULTI_INST); +} + +static struct xrt_subdev * +xrt_subdev_create(struct device *parent, enum xrt_subdev_id id, + xrt_subdev_root_cb_t pcb, void *pcb_arg, char *dtb) +{ + struct xrt_subdev_platdata *pdata = NULL; + struct xrt_subdev *sdev = NULL; + struct xrt_device *xdev = NULL; + struct resource *res = NULL; + unsigned long dtb_len = 0; + bool dtb_alloced = false; + int res_num = 0; + size_t pdata_sz; + int ret; + + sdev = xrt_subdev_alloc(); + if (!sdev) { + dev_err(parent, "failed to alloc subdev for ID %d", id); + return NULL; + } + sdev->xs_id = id; + + if (!dtb) { + ret = xrt_md_create(parent, &dtb); + if (ret) { + dev_err(parent, "can't create empty dtb: %d", ret); + goto fail; + } + dtb_alloced = true; + } + xrt_md_pack(parent, dtb); + dtb_len = xrt_md_size(parent, dtb); + if (dtb_len == XRT_MD_INVALID_LENGTH) { + dev_err(parent, "invalid metadata len %ld", dtb_len); + goto fail1; + } + pdata_sz = sizeof(struct xrt_subdev_platdata) + dtb_len; + + /* Prepare platform data passed to subdev. */ + pdata = vzalloc(pdata_sz); + if (!pdata) + goto fail1; + + pdata->xsp_root_cb = pcb; + pdata->xsp_root_cb_arg = pcb_arg; + memcpy(pdata->xsp_dtb, dtb, dtb_len); + if (id == XRT_SUBDEV_GRP) { + /* Group can only be created by root driver. */ + pdata->xsp_root_name = dev_name(parent); + } else { + struct xrt_device *grp = to_xrt_dev(parent); + + /* Leaf can only be created by group driver. */ + WARN_ON(to_xrt_drv(parent->driver)->subdev_id != XRT_SUBDEV_GRP); + pdata->xsp_root_name = DEV_PDATA(grp)->xsp_root_name; + } + + /* Create subdev. */ + if (id != XRT_SUBDEV_GRP) { + int rc = xrt_subdev_getres(parent, id, dtb, &res, &res_num); + + if (rc) { + dev_err(parent, "failed to get resource for %s: %d", + xrt_drv_name(id), rc); + goto fail2; + } + } + xdev = xrt_device_register(parent, id, res, res_num, pdata, pdata_sz); + vfree(res); + if (!xdev) { + dev_err(parent, "failed to create subdev for %s", xrt_drv_name(id)); + goto fail2; + } + sdev->xs_xdev = xdev; + + if (device_attach(DEV(xdev)) != 1) { + xrt_err(xdev, "failed to attach"); + goto fail3; + } + + if (sysfs_create_group(&DEV(xdev)->kobj, &xrt_subdev_attrgroup)) + xrt_err(xdev, "failed to create sysfs group"); + + /* + * Create sysfs sym link under root for leaves + * under random groups for easy access to them. + */ + if (id != XRT_SUBDEV_GRP) { + if (sysfs_create_link(&find_root(xdev)->kobj, + &DEV(xdev)->kobj, dev_name(DEV(xdev)))) { + xrt_err(xdev, "failed to create sysfs link"); + } + } + + /* All done, ready to handle req thru cdev. */ + if (xrt_subdev_cdev_auto_creation(xdev)) + xleaf_devnode_create(xdev, DEV_FILE_OPS(xdev)->xsf_dev_name, NULL); + + vfree(pdata); + return sdev; + +fail3: + xrt_device_unregister(sdev->xs_xdev); +fail2: + vfree(pdata); +fail1: + if (dtb_alloced) + vfree(dtb); +fail: + kfree(sdev); + return NULL; +} + +static void xrt_subdev_destroy(struct xrt_subdev *sdev) +{ + struct xrt_device *xdev = sdev->xs_xdev; + struct device *dev = DEV(xdev); + + /* Take down the device node */ + if (xrt_subdev_cdev_auto_creation(xdev)) + xleaf_devnode_destroy(xdev); + if (sdev->xs_id != XRT_SUBDEV_GRP) + sysfs_remove_link(&find_root(xdev)->kobj, dev_name(dev)); + sysfs_remove_group(&dev->kobj, &xrt_subdev_attrgroup); + xrt_device_unregister(xdev); + kfree(sdev); +} + +struct xrt_device * +xleaf_get_leaf(struct xrt_device *xdev, xrt_subdev_match_t match_cb, void *match_arg) +{ + int rc; + struct xrt_root_get_leaf get_leaf = { + xdev, match_cb, match_arg, }; + + rc = xrt_subdev_root_request(xdev, XRT_ROOT_GET_LEAF, &get_leaf); + if (rc) + return NULL; + return get_leaf.xpigl_tgt_xdev; +} +EXPORT_SYMBOL_GPL(xleaf_get_leaf); + +bool xleaf_has_endpoint(struct xrt_device *xdev, const char *endpoint_name) +{ + struct resource *res; + int i = 0; + + do { + res = xrt_get_resource(xdev, IORESOURCE_MEM, i); + if (res && !strncmp(res->name, endpoint_name, strlen(res->name) + 1)) + return true; + ++i; + } while (res); + + return false; +} +EXPORT_SYMBOL_GPL(xleaf_has_endpoint); + +int xleaf_put_leaf(struct xrt_device *xdev, struct xrt_device *leaf) +{ + struct xrt_root_put_leaf put_leaf = { xdev, leaf }; + + return xrt_subdev_root_request(xdev, XRT_ROOT_PUT_LEAF, &put_leaf); +} +EXPORT_SYMBOL_GPL(xleaf_put_leaf); + +int xleaf_create_group(struct xrt_device *xdev, char *dtb) +{ + return xrt_subdev_root_request(xdev, XRT_ROOT_CREATE_GROUP, dtb); +} +EXPORT_SYMBOL_GPL(xleaf_create_group); + +int xleaf_destroy_group(struct xrt_device *xdev, int instance) +{ + return xrt_subdev_root_request(xdev, XRT_ROOT_REMOVE_GROUP, (void *)(uintptr_t)instance); +} +EXPORT_SYMBOL_GPL(xleaf_destroy_group); + +int xleaf_wait_for_group_bringup(struct xrt_device *xdev) +{ + return xrt_subdev_root_request(xdev, XRT_ROOT_WAIT_GROUP_BRINGUP, NULL); +} +EXPORT_SYMBOL_GPL(xleaf_wait_for_group_bringup); + +static ssize_t +xrt_subdev_get_holders(struct xrt_subdev *sdev, char *buf, size_t len) +{ + const struct list_head *ptr; + struct xrt_subdev_holder *h; + ssize_t n = 0; + + list_for_each(ptr, &sdev->xs_holder_list) { + h = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list); + n += snprintf(buf + n, len - n, "%s:%d ", + dev_name(h->xsh_holder), kref_read(&h->xsh_kref)); + /* Truncation is fine here. Buffer content is only for debugging. */ + if (n >= (len - 1)) + break; + } + return n; +} + +void xrt_subdev_pool_init(struct device *dev, struct xrt_subdev_pool *spool) +{ + INIT_LIST_HEAD(&spool->xsp_dev_list); + spool->xsp_owner = dev; + mutex_init(&spool->xsp_lock); + spool->xsp_closing = false; +} + +static void xrt_subdev_free_holder(struct xrt_subdev_holder *holder) +{ + list_del(&holder->xsh_holder_list); + vfree(holder); +} + +static void xrt_subdev_pool_wait_for_holders(struct xrt_subdev_pool *spool, struct xrt_subdev *sdev) +{ + const struct list_head *ptr, *next; + char holders[128]; + struct xrt_subdev_holder *holder; + struct mutex *lk = &spool->xsp_lock; + + while (!list_empty(&sdev->xs_holder_list)) { + int rc; + + /* It's most likely a bug if we ever enters this loop. */ + xrt_subdev_get_holders(sdev, holders, sizeof(holders)); + xrt_err(sdev->xs_xdev, "awaits holders: %s", holders); + mutex_unlock(lk); + rc = wait_for_completion_killable(&sdev->xs_holder_comp); + mutex_lock(lk); + if (rc == -ERESTARTSYS) { + xrt_err(sdev->xs_xdev, "give up on waiting for holders, clean up now"); + list_for_each_safe(ptr, next, &sdev->xs_holder_list) { + holder = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list); + xrt_subdev_free_holder(holder); + } + } + } +} + +void xrt_subdev_pool_fini(struct xrt_subdev_pool *spool) +{ + struct list_head *dl = &spool->xsp_dev_list; + struct mutex *lk = &spool->xsp_lock; + + mutex_lock(lk); + if (spool->xsp_closing) { + mutex_unlock(lk); + return; + } + spool->xsp_closing = true; + mutex_unlock(lk); + + /* Remove subdev in the reverse order of added. */ + while (!list_empty(dl)) { + struct xrt_subdev *sdev = list_first_entry(dl, struct xrt_subdev, xs_dev_list); + + xrt_subdev_pool_wait_for_holders(spool, sdev); + list_del(&sdev->xs_dev_list); + xrt_subdev_destroy(sdev); + } +} + +static struct xrt_subdev_holder *xrt_subdev_find_holder(struct xrt_subdev *sdev, + struct device *holder_dev) +{ + struct list_head *hl = &sdev->xs_holder_list; + struct xrt_subdev_holder *holder; + const struct list_head *ptr; + + list_for_each(ptr, hl) { + holder = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list); + if (holder->xsh_holder == holder_dev) + return holder; + } + return NULL; +} + +static int xrt_subdev_hold(struct xrt_subdev *sdev, struct device *holder_dev) +{ + struct xrt_subdev_holder *holder = xrt_subdev_find_holder(sdev, holder_dev); + struct list_head *hl = &sdev->xs_holder_list; + + if (!holder) { + holder = vzalloc(sizeof(*holder)); + if (!holder) + return -ENOMEM; + holder->xsh_holder = holder_dev; + kref_init(&holder->xsh_kref); + list_add_tail(&holder->xsh_holder_list, hl); + } else { + kref_get(&holder->xsh_kref); + } + + return 0; +} + +static void xrt_subdev_free_holder_kref(struct kref *kref) +{ + struct xrt_subdev_holder *holder = container_of(kref, struct xrt_subdev_holder, xsh_kref); + + xrt_subdev_free_holder(holder); +} + +static int +xrt_subdev_release(struct xrt_subdev *sdev, struct device *holder_dev) +{ + struct xrt_subdev_holder *holder = xrt_subdev_find_holder(sdev, holder_dev); + struct list_head *hl = &sdev->xs_holder_list; + + if (!holder) { + dev_err(holder_dev, "can't release, %s did not hold %s", + dev_name(holder_dev), dev_name(DEV(sdev->xs_xdev))); + return -EINVAL; + } + kref_put(&holder->xsh_kref, xrt_subdev_free_holder_kref); + + /* kref_put above may remove holder from list. */ + if (list_empty(hl)) + complete(&sdev->xs_holder_comp); + return 0; +} + +int xrt_subdev_pool_add(struct xrt_subdev_pool *spool, enum xrt_subdev_id id, + xrt_subdev_root_cb_t pcb, void *pcb_arg, char *dtb) +{ + struct mutex *lk = &spool->xsp_lock; + struct list_head *dl = &spool->xsp_dev_list; + struct xrt_subdev *sdev; + int ret = 0; + + sdev = xrt_subdev_create(spool->xsp_owner, id, pcb, pcb_arg, dtb); + if (sdev) { + mutex_lock(lk); + if (spool->xsp_closing) { + /* No new subdev when pool is going away. */ + xrt_err(sdev->xs_xdev, "pool is closing"); + ret = -ENODEV; + } else { + list_add(&sdev->xs_dev_list, dl); + } + mutex_unlock(lk); + if (ret) + xrt_subdev_destroy(sdev); + } else { + ret = -EINVAL; + } + + ret = ret ? ret : sdev->xs_xdev->instance; + return ret; +} + +int xrt_subdev_pool_del(struct xrt_subdev_pool *spool, enum xrt_subdev_id id, int instance) +{ + const struct list_head *ptr; + struct mutex *lk = &spool->xsp_lock; + struct list_head *dl = &spool->xsp_dev_list; + struct xrt_subdev *sdev; + int ret = -ENOENT; + + mutex_lock(lk); + if (spool->xsp_closing) { + /* Pool is going away, all subdevs will be gone. */ + mutex_unlock(lk); + return 0; + } + list_for_each(ptr, dl) { + sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list); + if (sdev->xs_id != id || sdev->xs_xdev->instance != instance) + continue; + xrt_subdev_pool_wait_for_holders(spool, sdev); + list_del(&sdev->xs_dev_list); + ret = 0; + break; + } + mutex_unlock(lk); + if (ret) + return ret; + + xrt_subdev_destroy(sdev); + return 0; +} + +static int xrt_subdev_pool_get_impl(struct xrt_subdev_pool *spool, xrt_subdev_match_t match, + void *arg, struct device *holder_dev, struct xrt_subdev **sdevp) +{ + struct xrt_device *xdev = (struct xrt_device *)arg; + struct list_head *dl = &spool->xsp_dev_list; + struct mutex *lk = &spool->xsp_lock; + struct xrt_subdev *sdev = NULL; + const struct list_head *ptr; + struct xrt_subdev *d = NULL; + int ret = -ENOENT; + + mutex_lock(lk); + + if (!xdev) { + if (match == XRT_SUBDEV_MATCH_PREV) { + sdev = list_empty(dl) ? NULL : + list_last_entry(dl, struct xrt_subdev, xs_dev_list); + } else if (match == XRT_SUBDEV_MATCH_NEXT) { + sdev = list_first_entry_or_null(dl, struct xrt_subdev, xs_dev_list); + } + } + + list_for_each(ptr, dl) { + d = list_entry(ptr, struct xrt_subdev, xs_dev_list); + if (match == XRT_SUBDEV_MATCH_PREV || match == XRT_SUBDEV_MATCH_NEXT) { + if (d->xs_xdev != xdev) + continue; + } else { + if (!match(d->xs_id, d->xs_xdev, arg)) + continue; + } + + if (match == XRT_SUBDEV_MATCH_PREV) + sdev = !list_is_first(ptr, dl) ? list_prev_entry(d, xs_dev_list) : NULL; + else if (match == XRT_SUBDEV_MATCH_NEXT) + sdev = !list_is_last(ptr, dl) ? list_next_entry(d, xs_dev_list) : NULL; + else + sdev = d; + } + + if (sdev) + ret = xrt_subdev_hold(sdev, holder_dev); + + mutex_unlock(lk); + + if (!ret) + *sdevp = sdev; + return ret; +} + +int xrt_subdev_pool_get(struct xrt_subdev_pool *spool, xrt_subdev_match_t match, void *arg, + struct device *holder_dev, struct xrt_device **xdevp) +{ + int rc; + struct xrt_subdev *sdev; + + rc = xrt_subdev_pool_get_impl(spool, match, arg, holder_dev, &sdev); + if (rc) { + if (rc != -ENOENT) + dev_err(holder_dev, "failed to hold device: %d", rc); + return rc; + } + + if (!IS_ROOT_DEV(holder_dev)) { + xrt_dbg(to_xrt_dev(holder_dev), "%s <<==== %s", + dev_name(holder_dev), dev_name(DEV(sdev->xs_xdev))); + } + + *xdevp = sdev->xs_xdev; + return 0; +} + +static int xrt_subdev_pool_put_impl(struct xrt_subdev_pool *spool, struct xrt_device *xdev, + struct device *holder_dev) +{ + const struct list_head *ptr; + struct mutex *lk = &spool->xsp_lock; + struct list_head *dl = &spool->xsp_dev_list; + struct xrt_subdev *sdev; + int ret = -ENOENT; + + mutex_lock(lk); + list_for_each(ptr, dl) { + sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list); + if (sdev->xs_xdev != xdev) + continue; + ret = xrt_subdev_release(sdev, holder_dev); + break; + } + mutex_unlock(lk); + + return ret; +} + +int xrt_subdev_pool_put(struct xrt_subdev_pool *spool, struct xrt_device *xdev, + struct device *holder_dev) +{ + int ret = xrt_subdev_pool_put_impl(spool, xdev, holder_dev); + + if (ret) + return ret; + + if (!IS_ROOT_DEV(holder_dev)) { + xrt_dbg(to_xrt_dev(holder_dev), "%s <<==X== %s", + dev_name(holder_dev), dev_name(DEV(xdev))); + } + return 0; +} + +void xrt_subdev_pool_trigger_event(struct xrt_subdev_pool *spool, enum xrt_events e) +{ + struct xrt_device *tgt = NULL; + struct xrt_subdev *sdev = NULL; + struct xrt_event evt; + + while (!xrt_subdev_pool_get_impl(spool, XRT_SUBDEV_MATCH_NEXT, + tgt, spool->xsp_owner, &sdev)) { + tgt = sdev->xs_xdev; + evt.xe_evt = e; + evt.xe_subdev.xevt_subdev_id = sdev->xs_id; + evt.xe_subdev.xevt_subdev_instance = tgt->instance; + xrt_subdev_root_request(tgt, XRT_ROOT_EVENT_SYNC, &evt); + xrt_subdev_pool_put_impl(spool, tgt, spool->xsp_owner); + } +} + +void xrt_subdev_pool_handle_event(struct xrt_subdev_pool *spool, struct xrt_event *evt) +{ + struct xrt_device *tgt = NULL; + struct xrt_subdev *sdev = NULL; + + while (!xrt_subdev_pool_get_impl(spool, XRT_SUBDEV_MATCH_NEXT, + tgt, spool->xsp_owner, &sdev)) { + tgt = sdev->xs_xdev; + xleaf_call(tgt, XRT_XLEAF_EVENT, evt); + xrt_subdev_pool_put_impl(spool, tgt, spool->xsp_owner); + } +} + +ssize_t xrt_subdev_pool_get_holders(struct xrt_subdev_pool *spool, + struct xrt_device *xdev, char *buf, size_t len) +{ + const struct list_head *ptr; + struct mutex *lk = &spool->xsp_lock; + struct list_head *dl = &spool->xsp_dev_list; + struct xrt_subdev *sdev; + ssize_t ret = 0; + + mutex_lock(lk); + list_for_each(ptr, dl) { + sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list); + if (sdev->xs_xdev != xdev) + continue; + ret = xrt_subdev_get_holders(sdev, buf, len); + break; + } + mutex_unlock(lk); + + return ret; +} +EXPORT_SYMBOL_GPL(xrt_subdev_pool_get_holders); + +int xleaf_broadcast_event(struct xrt_device *xdev, enum xrt_events evt, bool async) +{ + struct xrt_event e = { evt, }; + enum xrt_root_cmd cmd = async ? XRT_ROOT_EVENT_ASYNC : XRT_ROOT_EVENT_SYNC; + + WARN_ON(evt == XRT_EVENT_POST_CREATION || evt == XRT_EVENT_PRE_REMOVAL); + return xrt_subdev_root_request(xdev, cmd, &e); +} +EXPORT_SYMBOL_GPL(xleaf_broadcast_event); + +void xleaf_hot_reset(struct xrt_device *xdev) +{ + xrt_subdev_root_request(xdev, XRT_ROOT_HOT_RESET, NULL); +} +EXPORT_SYMBOL_GPL(xleaf_hot_reset); + +void xleaf_get_root_res(struct xrt_device *xdev, u32 region_id, struct resource **res) +{ + struct xrt_root_get_res arg = { 0 }; + + arg.xpigr_region_id = region_id; + xrt_subdev_root_request(xdev, XRT_ROOT_GET_RESOURCE, &arg); + *res = arg.xpigr_res; +} + +void xleaf_get_root_id(struct xrt_device *xdev, unsigned short *vendor, unsigned short *device, + unsigned short *subvendor, unsigned short *subdevice) +{ + struct xrt_root_get_id id = { 0 }; + + WARN_ON(!vendor && !device && !subvendor && !subdevice); + + xrt_subdev_root_request(xdev, XRT_ROOT_GET_ID, (void *)&id); + if (vendor) + *vendor = id.xpigi_vendor_id; + if (device) + *device = id.xpigi_device_id; + if (subvendor) + *subvendor = id.xpigi_sub_vendor_id; + if (subdevice) + *subdevice = id.xpigi_sub_device_id; +} + +struct device *xleaf_register_hwmon(struct xrt_device *xdev, const char *name, void *drvdata, + const struct attribute_group **grps) +{ + struct xrt_root_hwmon hm = { true, name, drvdata, grps, }; + + xrt_subdev_root_request(xdev, XRT_ROOT_HWMON, (void *)&hm); + return hm.xpih_hwmon_dev; +} + +void xleaf_unregister_hwmon(struct xrt_device *xdev, struct device *hwmon) +{ + struct xrt_root_hwmon hm = { false, }; + + hm.xpih_hwmon_dev = hwmon; + xrt_subdev_root_request(xdev, XRT_ROOT_HWMON, (void *)&hm); +} From patchwork Mon Aug 2 16:05:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lizhi Hou X-Patchwork-Id: 490433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 557ABC43214 for ; Mon, 2 Aug 2021 16:07:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 362C660EE9 for ; Mon, 2 Aug 2021 16:07:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232412AbhHBQHY (ORCPT ); Mon, 2 Aug 2021 12:07:24 -0400 Received: from mail-dm3nam07on2051.outbound.protection.outlook.com ([40.107.95.51]:37600 "EHLO NAM02-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S232483AbhHBQHX (ORCPT ); Mon, 2 Aug 2021 12:07:23 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KPwml2TrCvhERcPXxB/biQDX+lqGKztEgXraRkC44+NdS2WeuXCmNwJjUWRRZAnKBDbBnlX5jqrPpIs6GDjgZG/3lqpVrKb3JkOxTzmmdg5TPBzZu2gd6+Aq8SPYycrEl6NJGOJY/fP5WmwDsnEPRZMT0Ewoyde369SzlBcuESknII4XuLd4C0mIHZt/OAeWoX4MiLaLxOduzbSlyVw0kQ6D05t+hOIvorjOtzcLbGIFwZCfGoiSBOqIwAT0GEMR+MSnUAXKGhA9XRNhE5TojXzmOx/aAN/QBZ3XlRxbPgiHx/cDZKREntIvwxiLUYBBKJuQC6Ua7dyxO9oP0xm0OA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YBbFygzrmYFm17xCvB6Nr8ZBkAeKR6IqRZwzouIua1M=; b=AwCAggm9CFFigvJU8/ElxUOr1dkcTpqCLS0usskTJOoWbFE3MXLZ4M/FXcaRsRfMH36RYmfXTxOm8oK475LwnOtdx29etXfTuvYrBVwQ+kSy94ClhlMof2I/iRIACIFFd8gpeeGXI48dI34viC8LSi0eiK/BAQ/yK0eYp7LLzZTcrw6dCDJXE6aSRkkm72DgP7PGBQQAo67B3+EVoWrSy0FqdOATC9P9nezdkdE6t1N1sX9abYpExsgcN5M0uqMYdXHi7o8rvkJ3LoItuJuVHiUoAjyWNPooXQ2PhVGfN6LimA/tv3tbyW95ccpoqAHqHzq8XSJWtXT4MfbrTU9lsA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=kernel.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YBbFygzrmYFm17xCvB6Nr8ZBkAeKR6IqRZwzouIua1M=; b=luSvN658v9vMZPASbpRpop/jRjr4QDgUO+icwjBh/TnnQpZ+JgJgQC01CFkXlWHPHKx7vzhDaKGBL5hl8YokEKhyrSO/umJlgwhUHnxtItmhpZx2Shn4HkaJtw7CKKLU1xaiMfnjeH9z0DIryP35pK+ofuAJUWbqW/TELpPciW0= Received: from DM6PR11CA0022.namprd11.prod.outlook.com (2603:10b6:5:190::35) by BYAPR02MB5863.namprd02.prod.outlook.com (2603:10b6:a03:128::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.17; Mon, 2 Aug 2021 16:07:11 +0000 Received: from DM3NAM02FT041.eop-nam02.prod.protection.outlook.com (2603:10b6:5:190:cafe::49) by DM6PR11CA0022.outlook.office365.com (2603:10b6:5:190::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:07:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; kernel.org; dkim=none (message not signed) header.d=none; kernel.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch01.xlnx.xilinx.com; Received: from xsj-pvapexch01.xlnx.xilinx.com (149.199.62.198) by DM3NAM02FT041.mail.protection.outlook.com (10.13.5.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:07:10 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch01.xlnx.xilinx.com (172.19.86.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 2 Aug 2021 09:06:54 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.2 via Frontend Transport; Mon, 2 Aug 2021 09:06:54 -0700 Envelope-to: mdf@kernel.org, robh@kernel.org, trix@redhat.com, devicetree@vger.kernel.org, linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org Received: from [10.23.145.232] (port=49062 helo=xsj-xw9400.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1mAaSs-0005Ch-9b; Mon, 02 Aug 2021 09:06:54 -0700 Received: by xsj-xw9400.xilinx.com (Postfix, from userid 21952) id A68A2601B25; Mon, 2 Aug 2021 09:05:30 -0700 (PDT) From: Lizhi Hou To: CC: Lizhi Hou , , , , , , , , , , , Max Zhen Subject: [PATCH V9 XRT Alveo 09/14] fpga: xrt: management physical function driver (root) Date: Mon, 2 Aug 2021 09:05:16 -0700 Message-ID: <20210802160521.331031-10-lizhi.hou@xilinx.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210802160521.331031-1-lizhi.hou@xilinx.com> References: <20210802160521.331031-1-lizhi.hou@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a61a9ee9-4d63-4626-4566-08d955cf95de X-MS-TrafficTypeDiagnostic: BYAPR02MB5863: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:326; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mvGMGTf+i4Em17hTRbRw2ELGpHKiPRjpVvuQ7In3Qbm/gVD+jT8uwZCmFl76Bn5VCl5rnzF3OSiuJjR1cq0SqFEAXekzdKjKKh8DCpaohHbcAQtmUvFKejv3VGEzdfVip4wqraW1DD0dHl/y9+csp6Wx0U/Mgm0Z0hqnqa5rja+wbbUb1gHYMG6im4sn1Y9C3OhjAk8BVC8xjlDdvCntPJcB7abzeEo1N8Mf1KT5z2sH66kIwnn02noL+ivP+Musa2ut02IbRKhbpNHTQNZthuUxGiMZ3SdaYfFziUXf6XD+Pz65UEd1aCJnnMGJdToZeQbC+17WAlpUORjfFHuBMff01PBwIkB80xQLIr1kXMX8N3+xO3cQuDlYEv3eksz/feHDD0qetCDnLh1LPkQ7YekebRDTY/R8hYDdpTNOvEgvoG16r8pbRfOxqWv2xPg2gHURuHhQgwat3ifKP2iRnGCExMCbqsQiscpyLZHEDAEe2rqk/RiWOenbl0AaF6jg3eniO9aeaD9ElwUhOLe9J6Wp3HYzO4n+IHgV2wOBvEInaHO/LwDKKSonI+afwNISqgbBfmkBy1MWhJBJt3SiNXQLJ/r0+PkOjyCib2sv1tgpaM29cfYnC5/mH6J9lvPzWFTp2bjZqUCz3aryPl3fF/xf+exT2VBiHckjm5eJMf2v4spqCVeOvy39GjG8G3nqBaD87kqPJ9Hadqu+zmT3hA== X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch01.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(6916009)(42186006)(54906003)(356005)(70206006)(26005)(426003)(44832011)(70586007)(30864003)(1076003)(6666004)(5660300002)(4326008)(508600001)(36860700001)(36756003)(83380400001)(107886003)(6266002)(82310400003)(36906005)(2906002)(8936002)(336012)(316002)(47076005)(186003)(8676002)(2616005)(7636003); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2021 16:07:10.9689 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a61a9ee9-4d63-4626-4566-08d955cf95de X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch01.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: DM3NAM02FT041.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR02MB5863 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org The PCIE device driver which attaches to management function on Alveo devices. It instantiates one or more group drivers which, in turn, instantiate xrt drivers. The instantiation of group and xrt drivers is completely dtb driven. Signed-off-by: Sonal Santan Signed-off-by: Max Zhen Signed-off-by: Lizhi Hou Reviewed-by: Tom Rix --- drivers/fpga/xrt/mgmt/root.c | 418 +++++++++++++++++++++++++++++++++++ 1 file changed, 418 insertions(+) create mode 100644 drivers/fpga/xrt/mgmt/root.c diff --git a/drivers/fpga/xrt/mgmt/root.c b/drivers/fpga/xrt/mgmt/root.c new file mode 100644 index 000000000000..566a6fcfbecd --- /dev/null +++ b/drivers/fpga/xrt/mgmt/root.c @@ -0,0 +1,418 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Xilinx Alveo Management Function Driver + * + * Copyright (C) 2020-2021 Xilinx, Inc. + * + * Authors: + * Cheng Zhen + */ + +#include +#include +#include +#include +#include + +#include "xroot.h" +#include "xmgmt.h" +#include "metadata.h" + +#define XMGMT_MODULE_NAME "xrt-mgmt" + +#define XMGMT_PDEV(xm) ((xm)->pdev) +#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev)) +#define xmgmt_err(xm, fmt, args...) \ + dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args) +#define xmgmt_warn(xm, fmt, args...) \ + dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args) +#define xmgmt_info(xm, fmt, args...) \ + dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args) +#define xmgmt_dbg(xm, fmt, args...) \ + dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args) +#define XMGMT_DEV_ID(_pcidev) \ + ({ typeof(_pcidev) (pcidev) = (_pcidev); \ + ((pci_domain_nr((pcidev)->bus) << 16) | \ + PCI_DEVID((pcidev)->bus->number, 0)); }) +#define XRT_VSEC_ID 0x20 +#define XRT_MAX_READRQ 512 + +static struct class *xmgmt_class; + +/* PCI Device IDs */ +/* + * Golden image is preloaded on the device when it is shipped to customer. + * Then, customer can load other shells (from Xilinx or some other vendor). + * If something goes wrong with the shell, customer can always go back to + * golden and start over again. + */ +#define PCI_DEVICE_ID_U50_GOLDEN 0xD020 +#define PCI_DEVICE_ID_U50 0x5020 +static const struct pci_device_id xmgmt_pci_ids[] = { + { PCI_DEVICE(PCI_VENDOR_ID_XILINX, PCI_DEVICE_ID_U50_GOLDEN), }, /* Alveo U50 (golden) */ + { PCI_DEVICE(PCI_VENDOR_ID_XILINX, PCI_DEVICE_ID_U50), }, /* Alveo U50 */ + { 0, } +}; + +struct xmgmt { + struct pci_dev *pdev; + void *root; + + bool ready; +}; + +static int xmgmt_config_pci(struct xmgmt *xm) +{ + struct pci_dev *pdev = XMGMT_PDEV(xm); + int rc; + + rc = pcim_enable_device(pdev); + if (rc < 0) { + xmgmt_err(xm, "failed to enable device: %d", rc); + return rc; + } + + rc = pci_enable_pcie_error_reporting(pdev); + if (rc) + xmgmt_warn(xm, "failed to enable AER: %d", rc); + + pci_set_master(pdev); + + rc = pcie_get_readrq(pdev); + if (rc > XRT_MAX_READRQ) + pcie_set_readrq(pdev, XRT_MAX_READRQ); + return 0; +} + +static int xmgmt_match_slot_and_save(struct device *dev, void *data) +{ + struct xmgmt *xm = data; + struct pci_dev *pdev = to_pci_dev(dev); + + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) { + pci_cfg_access_lock(pdev); + pci_save_state(pdev); + } + + return 0; +} + +static void xmgmt_pci_save_config_all(struct xmgmt *xm) +{ + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save); +} + +static int xmgmt_match_slot_and_restore(struct device *dev, void *data) +{ + struct xmgmt *xm = data; + struct pci_dev *pdev = to_pci_dev(dev); + + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) { + pci_restore_state(pdev); + pci_cfg_access_unlock(pdev); + } + + return 0; +} + +static void xmgmt_pci_restore_config_all(struct xmgmt *xm) +{ + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_restore); +} + +static void xmgmt_root_hot_reset(struct device *dev) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct pci_bus *bus; + u16 pci_cmd, devctl; + struct xmgmt *xm; + u8 pci_bctl; + int i, ret; + + xm = pci_get_drvdata(pdev); + xmgmt_info(xm, "hot reset start"); + xmgmt_pci_save_config_all(xm); + pci_disable_device(pdev); + bus = pdev->bus; + + /* + * When flipping the SBR bit, device can fall off the bus. This is + * usually no problem at all so long as drivers are working properly + * after SBR. However, some systems complain bitterly when the device + * falls off the bus. + * The quick solution is to temporarily disable the SERR reporting of + * switch port during SBR. + */ + + pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd); + pci_write_config_word(bus->self, PCI_COMMAND, (pci_cmd & ~PCI_COMMAND_SERR)); + pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl); + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, (devctl & ~PCI_EXP_DEVCTL_FERE)); + pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl); + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl | PCI_BRIDGE_CTL_BUS_RESET); + msleep(100); + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl); + ssleep(1); + + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl); + pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd); + + ret = pci_enable_device(pdev); + if (ret) + xmgmt_err(xm, "failed to enable device, ret %d", ret); + + for (i = 0; i < 300; i++) { + pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd); + if (pci_cmd != 0xffff) + break; + msleep(20); + } + if (i == 300) + xmgmt_err(xm, "timed out waiting for device to be online after reset"); + + xmgmt_info(xm, "waiting for %d ms", i * 20); + xmgmt_pci_restore_config_all(xm); + xmgmt_config_pci(xm); +} + +static int xmgmt_add_vsec_node(struct xmgmt *xm, char *dtb) +{ + struct pci_dev *pdev = XMGMT_PDEV(xm); + struct xrt_md_endpoint ep = { 0 }; + struct device *dev = DEV(pdev); + u32 off_low, off_high, header; + int cap = 0, ret = 0; + __be32 vsec_bar; + __be64 vsec_off; + + while ((cap = pci_find_next_ext_capability(pdev, cap, PCI_EXT_CAP_ID_VNDR))) { + pci_read_config_dword(pdev, cap + PCI_VNDR_HEADER, &header); + if (PCI_VNDR_HEADER_ID(header) == XRT_VSEC_ID) + break; + } + if (!cap) { + xmgmt_info(xm, "No Vendor Specific Capability."); + return -ENOENT; + } + + if (pci_read_config_dword(pdev, cap + 8, &off_low) || + pci_read_config_dword(pdev, cap + 12, &off_high)) { + xmgmt_err(xm, "pci_read vendor specific failed."); + return -EINVAL; + } + + ep.ep_name = XRT_MD_NODE_VSEC; + ret = xrt_md_add_endpoint(dev, dtb, &ep); + if (ret) { + xmgmt_err(xm, "add vsec metadata failed, ret %d", ret); + goto failed; + } + + vsec_bar = cpu_to_be32(off_low & 0xf); + ret = xrt_md_set_prop(dev, dtb, XRT_MD_NODE_VSEC, NULL, + XRT_MD_PROP_BAR_IDX, &vsec_bar, sizeof(vsec_bar)); + if (ret) { + xmgmt_err(xm, "add vsec bar idx failed, ret %d", ret); + goto failed; + } + + vsec_off = cpu_to_be64(((u64)off_high << 32) | (off_low & ~0xfU)); + ret = xrt_md_set_prop(dev, dtb, XRT_MD_NODE_VSEC, NULL, + XRT_MD_PROP_OFFSET, &vsec_off, sizeof(vsec_off)); + if (ret) { + xmgmt_err(xm, "add vsec offset failed, ret %d", ret); + goto failed; + } + +failed: + return ret; +} + +static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb) +{ + char *dtb = NULL; + int ret; + + ret = xrt_md_create(XMGMT_DEV(xm), &dtb); + if (ret) { + xmgmt_err(xm, "create metadata failed, ret %d", ret); + goto failed; + } + + ret = xmgmt_add_vsec_node(xm, dtb); + if (ret == -ENOENT) { + /* + * We may be dealing with a MFG board. + * Try vsec-golden which will bring up all hard-coded leaves + * at hard-coded offsets. + */ + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_VSEC_GOLDEN); + } else if (ret == 0) { + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_MGMT_MAIN); + } + if (ret) + goto failed; + + *root_dtb = dtb; + return 0; + +failed: + vfree(dtb); + return ret; +} + +static ssize_t ready_show(struct device *dev, + struct device_attribute *da, + char *buf) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct xmgmt *xm = pci_get_drvdata(pdev); + + return sprintf(buf, "%d\n", xm->ready); +} +static DEVICE_ATTR_RO(ready); + +static struct attribute *xmgmt_root_attrs[] = { + &dev_attr_ready.attr, + NULL +}; + +static struct attribute_group xmgmt_root_attr_group = { + .attrs = xmgmt_root_attrs, +}; + +static void xmgmt_root_get_id(struct device *dev, struct xrt_root_get_id *rid) +{ + struct pci_dev *pdev = to_pci_dev(dev); + + rid->xpigi_vendor_id = pdev->vendor; + rid->xpigi_device_id = pdev->device; + rid->xpigi_sub_vendor_id = pdev->subsystem_vendor; + rid->xpigi_sub_device_id = pdev->subsystem_device; +} + +static int xmgmt_root_get_resource(struct device *dev, struct xrt_root_get_res *res) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct xmgmt *xm; + + xm = pci_get_drvdata(pdev); + if (res->xpigr_region_id > PCI_STD_RESOURCE_END) { + xmgmt_err(xm, "Invalid bar idx %d", res->xpigr_region_id); + return -EINVAL; + } + + res->xpigr_res = &pdev->resource[res->xpigr_region_id]; + return 0; +} + +static struct xroot_physical_function_callback xmgmt_xroot_pf_cb = { + .xpc_get_id = xmgmt_root_get_id, + .xpc_get_resource = xmgmt_root_get_resource, + .xpc_hot_reset = xmgmt_root_hot_reset, +}; + +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id) +{ + int ret; + struct device *dev = &pdev->dev; + struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL); + char *dtb = NULL; + + if (!xm) + return -ENOMEM; + xm->pdev = pdev; + pci_set_drvdata(pdev, xm); + + ret = xmgmt_config_pci(xm); + if (ret) + goto failed; + + ret = xroot_probe(&pdev->dev, &xmgmt_xroot_pf_cb, &xm->root); + if (ret) + goto failed; + + ret = xmgmt_create_root_metadata(xm, &dtb); + if (ret) + goto failed_metadata; + + ret = xroot_create_group(xm->root, dtb); + vfree(dtb); + if (ret) + xmgmt_err(xm, "failed to create root group: %d", ret); + + if (!xroot_wait_for_bringup(xm->root)) + xmgmt_err(xm, "failed to bringup all groups"); + else + xm->ready = true; + + ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group); + if (ret) { + /* Warning instead of failing the probe. */ + xmgmt_warn(xm, "create xmgmt root attrs failed: %d", ret); + } + + xroot_broadcast(xm->root, XRT_EVENT_POST_CREATION); + xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME); + return 0; + +failed_metadata: + xroot_remove(xm->root); +failed: + pci_set_drvdata(pdev, NULL); + return ret; +} + +static void xmgmt_remove(struct pci_dev *pdev) +{ + struct xmgmt *xm = pci_get_drvdata(pdev); + + xroot_broadcast(xm->root, XRT_EVENT_PRE_REMOVAL); + sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group); + xroot_remove(xm->root); + pci_disable_pcie_error_reporting(xm->pdev); + xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME); +} + +static struct pci_driver xmgmt_driver = { + .name = XMGMT_MODULE_NAME, + .id_table = xmgmt_pci_ids, + .probe = xmgmt_probe, + .remove = xmgmt_remove, +}; + +static int __init xmgmt_init(void) +{ + int res = 0; + + res = xmgmt_register_leaf(); + if (res) + return res; + + xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME); + if (IS_ERR(xmgmt_class)) + return PTR_ERR(xmgmt_class); + + res = pci_register_driver(&xmgmt_driver); + if (res) { + class_destroy(xmgmt_class); + return res; + } + + return 0; +} + +static __exit void xmgmt_exit(void) +{ + pci_unregister_driver(&xmgmt_driver); + class_destroy(xmgmt_class); + xmgmt_unregister_leaf(); +} + +module_init(xmgmt_init); +module_exit(xmgmt_exit); + +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids); +MODULE_AUTHOR("XRT Team "); +MODULE_DESCRIPTION("Xilinx Alveo management function driver"); +MODULE_LICENSE("GPL v2"); From patchwork Mon Aug 2 16:05:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lizhi Hou X-Patchwork-Id: 490431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17D24C4320A for ; Mon, 2 Aug 2021 16:07:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DE71360E97 for ; Mon, 2 Aug 2021 16:07:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232077AbhHBQIC (ORCPT ); Mon, 2 Aug 2021 12:08:02 -0400 Received: from mail-dm6nam10on2042.outbound.protection.outlook.com ([40.107.93.42]:7905 "EHLO NAM10-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229979AbhHBQIA (ORCPT ); Mon, 2 Aug 2021 12:08:00 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RrJUSkzaQTIN746Mrdqq6rU6iTte8G9kwDwHWFfLTbLa5DtfzjPo4CC7ik1Y+us1SyK0Cm8AfPWSIdRH1uHAXGehvg3tScWD+20nbKckJOr7RfhqlnUgFO4/L+b7gD0Ad1Yro55eRdXT3dS9NjsTL0HrcUqgCWrzb1QLO+jYYH65n6/ASv/1981SnZwIH3P73XAINvtMsBRvNyorvjihmNTFesIbspKL8MNS/xEHCtYKNVXgq6+YTiNJRgXNvf0DTcllnwgEx94L84OZocj7gS5l1SCZiU3s9QvMjHoXx4y4wOZ6ZPNFbTc6CBSzcjxgIid8BcRYXYt9ZD/Cf0+k9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iUl7AflccZxU4PvOj7Nplt9qWvSh3441SZg9SXDXutA=; b=YvMUfjrxsOd67iGI6sRqHleX7Xyr1PqrxrLWfwSjHJK9sBrSzE35oV/T7xyd4WxOojmAI6ImaPbHP2iaHBTR0p2S6PCenuSfpH9KGdxfHe5FQ8J5U8zCoCHG9x7oUUX293tOY03adOJcXYA4Mtrn9rNbhhjhkzD6EqgYkzSyn2GTgpP/Ihu2cS9pNDypuXWjhX25Lm6csr+Z1dP8FWZLNELC8p79HAKfFsJKVKlT4HeNplFs1oQxyq2h1Mb6tQlYN0bffF4WTjzivDnab1nIDMjFXB9i8fvLPaOG8PSE4HizVKJBzl4wn9ieBTRyb1Vh1X90yijH2qgKwBpKx/HXzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=kernel.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iUl7AflccZxU4PvOj7Nplt9qWvSh3441SZg9SXDXutA=; b=jWk5ao5NVAURkHi0fDkhvPq7Aisb6GXQlqvBOowfg9wMwuDjm78AS7riD0n0qkVjDiXLxHYX/MokIPFilREFse/VdYEDhPEFuwmc6EWkJoH0BqbIpEwkSOOjF10EFkfedEWiAQDCWRrXzIvb4aW4WojhNuATJSuDIXBnJXJJvgI= Received: from DM3PR12CA0123.namprd12.prod.outlook.com (2603:10b6:0:51::19) by BYAPR02MB5768.namprd02.prod.outlook.com (2603:10b6:a03:11b::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.17; Mon, 2 Aug 2021 16:07:48 +0000 Received: from DM3NAM02FT013.eop-nam02.prod.protection.outlook.com (2603:10b6:0:51:cafe::a6) by DM3PR12CA0123.outlook.office365.com (2603:10b6:0:51::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:07:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; kernel.org; dkim=none (message not signed) header.d=none; kernel.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by DM3NAM02FT013.mail.protection.outlook.com (10.13.5.126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:07:47 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 2 Aug 2021 09:07:42 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.2 via Frontend Transport; Mon, 2 Aug 2021 09:07:42 -0700 Envelope-to: mdf@kernel.org, robh@kernel.org, trix@redhat.com, devicetree@vger.kernel.org, linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org Received: from [10.23.145.232] (port=49070 helo=xsj-xw9400.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1mAaTe-0005JA-GC; Mon, 02 Aug 2021 09:07:42 -0700 Received: by xsj-xw9400.xilinx.com (Postfix, from userid 21952) id EFEEF601B35; Mon, 2 Aug 2021 09:05:30 -0700 (PDT) From: Lizhi Hou To: CC: Lizhi Hou , , , , , , , , , , , Max Zhen Subject: [PATCH V9 XRT Alveo 11/14] fpga: xrt: fpga-mgr and region implementation for xclbin download Date: Mon, 2 Aug 2021 09:05:18 -0700 Message-ID: <20210802160521.331031-12-lizhi.hou@xilinx.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210802160521.331031-1-lizhi.hou@xilinx.com> References: <20210802160521.331031-1-lizhi.hou@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a5a2674d-c6c2-4af9-2eb8-08d955cfabec X-MS-TrafficTypeDiagnostic: BYAPR02MB5768: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:608; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cMLmqvl82xEKZ4/pUTJqfRpy0h4w4msV5sfsgGwtl15GS+u9LC049s0eIDGs8wIDsDtZYS/LUfhgLKWCaTL1SqTlkew2NjRliUcSYio+VokpJ7gSJ7iqy8b8XMG68ol27w2dLshjVOGxP/Qi22APw592PfuwVKD7JOckCJY4L1/WjU6w3Bj1M6nANwPzrW4BFMNtDgpvlv0MWHbF7fkqRXwbHRRWE74qLzYC1ifK/A1ySw0ZUYAMFUfRivNEcJusUvt3cFeilBXSZzVNWamYKzvtoYc9FGGMiJFsevD6Ng7xc6ETs8fLHM1h3U84r9qvO3ICBVi1NSHY45bYi/0kISRgXEVmht2tp6djkHq+zQIrYldaSx1k99I4EFJWOponC+uZyahqM3yY9jS8P0D82hzmOPuTcrGVzmN8uq5/v0iLo4DYwlFHa6nf7a3BbOjYOa0ysdwYIb7mXnZ1ZHZz+Vcbmf76llc1jTJNP1PzVJI99R+AWaBLHpBEfrlRo9WNpmEIqXaqMNq8CBKLANXMNO1xPS2gnVJLx1L9+DoW5zHhD1w6OCnjJ1dJnayoHMjp8FLIpQPHnY+UIc+UDmYK6floHXqTU7kooGJFYpsfjyMhvwmq9PEwRk5U5qZjfaEIXdQoH2pZ3nrwkQOBCnMQmZLwZqxA+JJL8G/yNt/wxsGNm3KgzIL5Lc+VidXyLdPDYUF0On6VUdJ9G3wItuz4fw== X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(4636009)(39860400002)(346002)(376002)(136003)(396003)(36840700001)(46966006)(336012)(47076005)(1076003)(36906005)(426003)(6666004)(107886003)(478600001)(44832011)(2906002)(186003)(356005)(36756003)(26005)(30864003)(4326008)(6916009)(7636003)(2616005)(54906003)(6266002)(82310400003)(36860700001)(42186006)(8936002)(83380400001)(316002)(70206006)(8676002)(5660300002)(82740400003)(70586007); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2021 16:07:47.9709 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a5a2674d-c6c2-4af9-2eb8-08d955cfabec X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: DM3NAM02FT013.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR02MB5768 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org fpga-mgr and region implementation for xclbin download which will be called from main xrt driver Signed-off-by: Sonal Santan Signed-off-by: Max Zhen Signed-off-by: Lizhi Hou Reviewed-by: Tom Rix --- drivers/fpga/xrt/mgmt/xmgmt-main-region.c | 483 ++++++++++++++++++++++ drivers/fpga/xrt/mgmt/xrt-mgr.c | 190 +++++++++ drivers/fpga/xrt/mgmt/xrt-mgr.h | 16 + 3 files changed, 689 insertions(+) create mode 100644 drivers/fpga/xrt/mgmt/xmgmt-main-region.c create mode 100644 drivers/fpga/xrt/mgmt/xrt-mgr.c create mode 100644 drivers/fpga/xrt/mgmt/xrt-mgr.h diff --git a/drivers/fpga/xrt/mgmt/xmgmt-main-region.c b/drivers/fpga/xrt/mgmt/xmgmt-main-region.c new file mode 100644 index 000000000000..6e6a16b13258 --- /dev/null +++ b/drivers/fpga/xrt/mgmt/xmgmt-main-region.c @@ -0,0 +1,483 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * FPGA Region Support for Xilinx Alveo + * + * Copyright (C) 2020-2021 Xilinx, Inc. + * + * Authors: Lizhi.Hou@xilinx.com + */ + +#include +#include +#include +#include +#include "metadata.h" +#include "xleaf.h" +#include "xleaf/axigate.h" +#include "xclbin-helper.h" +#include "xmgmt.h" + +struct xmgmt_bridge { + struct xrt_device *xdev; + const char *bridge_name; +}; + +struct xmgmt_region { + struct xrt_device *xdev; + struct fpga_region *region; + struct fpga_compat_id compat_id; + uuid_t interface_uuid; + struct fpga_bridge *bridge; + int group_instance; + uuid_t depend_uuid; + struct list_head list; +}; + +struct xmgmt_region_match_arg { + struct xrt_device *xdev; + uuid_t *uuids; + u32 uuid_num; +}; + +static int xmgmt_br_enable_set(struct fpga_bridge *bridge, bool enable) +{ + struct xmgmt_bridge *br_data = (struct xmgmt_bridge *)bridge->priv; + struct xrt_device *axigate_leaf; + int rc; + + axigate_leaf = xleaf_get_leaf_by_epname(br_data->xdev, br_data->bridge_name); + if (!axigate_leaf) { + xrt_err(br_data->xdev, "failed to get leaf %s", + br_data->bridge_name); + return -ENOENT; + } + + if (enable) + rc = xleaf_call(axigate_leaf, XRT_AXIGATE_OPEN, NULL); + else + rc = xleaf_call(axigate_leaf, XRT_AXIGATE_CLOSE, NULL); + + if (rc) { + xrt_err(br_data->xdev, "failed to %s gate %s, rc %d", + (enable ? "free" : "freeze"), br_data->bridge_name, + rc); + } + + xleaf_put_leaf(br_data->xdev, axigate_leaf); + + return rc; +} + +static const struct fpga_bridge_ops xmgmt_bridge_ops = { + .enable_set = xmgmt_br_enable_set +}; + +static void xmgmt_destroy_bridge(struct fpga_bridge *br) +{ + struct xmgmt_bridge *br_data = br->priv; + + if (!br_data) + return; + + xrt_info(br_data->xdev, "destroy fpga bridge %s", br_data->bridge_name); + fpga_bridge_unregister(br); + + devm_kfree(DEV(br_data->xdev), br_data); + + fpga_bridge_free(br); +} + +static struct fpga_bridge *xmgmt_create_bridge(struct xrt_device *xdev, + char *dtb) +{ + struct fpga_bridge *br = NULL; + struct xmgmt_bridge *br_data; + const char *gate; + int rc; + + br_data = devm_kzalloc(DEV(xdev), sizeof(*br_data), GFP_KERNEL); + if (!br_data) + return NULL; + br_data->xdev = xdev; + + br_data->bridge_name = XRT_MD_NODE_GATE_ULP; + rc = xrt_md_find_endpoint(&xdev->dev, dtb, XRT_MD_NODE_GATE_ULP, + NULL, &gate); + if (rc) { + br_data->bridge_name = XRT_MD_NODE_GATE_PLP; + rc = xrt_md_find_endpoint(&xdev->dev, dtb, XRT_MD_NODE_GATE_PLP, + NULL, &gate); + } + if (rc) { + xrt_err(xdev, "failed to get axigate, rc %d", rc); + goto failed; + } + + br = fpga_bridge_create(DEV(xdev), br_data->bridge_name, + &xmgmt_bridge_ops, br_data); + if (!br) { + xrt_err(xdev, "failed to create bridge"); + goto failed; + } + + rc = fpga_bridge_register(br); + if (rc) { + xrt_err(xdev, "failed to register bridge, rc %d", rc); + goto failed; + } + + xrt_info(xdev, "created fpga bridge %s", br_data->bridge_name); + + return br; + +failed: + if (br) + fpga_bridge_free(br); + if (br_data) + devm_kfree(DEV(xdev), br_data); + + return NULL; +} + +static void xmgmt_destroy_region(struct fpga_region *region) +{ + struct xmgmt_region *r_data = region->priv; + + xrt_info(r_data->xdev, "destroy fpga region %llx.%llx", + region->compat_id->id_h, region->compat_id->id_l); + + fpga_region_unregister(region); + + if (r_data->group_instance > 0) + xleaf_destroy_group(r_data->xdev, r_data->group_instance); + + if (r_data->bridge) + xmgmt_destroy_bridge(r_data->bridge); + + if (r_data->region->info) { + fpga_image_info_free(r_data->region->info); + r_data->region->info = NULL; + } + + fpga_region_free(region); + + devm_kfree(DEV(r_data->xdev), r_data); +} + +static int xmgmt_region_match(struct device *dev, const void *data) +{ + const struct xmgmt_region_match_arg *arg = data; + const struct fpga_region *match_region; + uuid_t compat_uuid; + int i; + + if (dev->parent != &arg->xdev->dev) + return false; + + match_region = to_fpga_region(dev); + /* + * The device tree provides both parent and child uuids for an + * xclbin in one array. Here we try both uuids to see if it matches + * with target region's compat_id. Strictly speaking we should + * only match xclbin's parent uuid with target region's compat_id + * but given the uuids by design are unique comparing with both + * does not hurt. + */ + import_uuid(&compat_uuid, (const char *)match_region->compat_id); + for (i = 0; i < arg->uuid_num; i++) { + if (uuid_equal(&compat_uuid, &arg->uuids[i])) + return true; + } + + return false; +} + +static int xmgmt_region_match_base(struct device *dev, const void *data) +{ + const struct xmgmt_region_match_arg *arg = data; + const struct fpga_region *match_region; + const struct xmgmt_region *r_data; + + if (dev->parent != &arg->xdev->dev) + return false; + + match_region = to_fpga_region(dev); + r_data = match_region->priv; + if (uuid_is_null(&r_data->depend_uuid)) + return true; + + return false; +} + +static int xmgmt_region_match_by_uuid(struct device *dev, const void *data) +{ + const struct xmgmt_region_match_arg *arg = data; + const struct fpga_region *match_region; + const struct xmgmt_region *r_data; + + if (dev->parent != &arg->xdev->dev) + return false; + + if (arg->uuid_num != 1) + return false; + + match_region = to_fpga_region(dev); + r_data = match_region->priv; + if (uuid_equal(&r_data->depend_uuid, arg->uuids)) + return true; + + return false; +} + +static void xmgmt_region_cleanup(struct fpga_region *region) +{ + struct xmgmt_region *r_data = region->priv, *pdata, *temp; + struct xrt_device *xdev = r_data->xdev; + struct xmgmt_region_match_arg arg = { 0 }; + struct fpga_region *match_region = NULL; + struct device *start_dev = NULL; + LIST_HEAD(free_list); + uuid_t compat_uuid; + + list_add_tail(&r_data->list, &free_list); + arg.xdev = xdev; + arg.uuid_num = 1; + arg.uuids = &compat_uuid; + + /* find all regions depending on this region */ + list_for_each_entry_safe(pdata, temp, &free_list, list) { + import_uuid(arg.uuids, (const char *)pdata->region->compat_id); + start_dev = NULL; + while ((match_region = fpga_region_class_find(start_dev, &arg, + xmgmt_region_match_by_uuid))) { + pdata = match_region->priv; + list_add_tail(&pdata->list, &free_list); + start_dev = &match_region->dev; + put_device(&match_region->dev); + } + } + + list_del(&r_data->list); + + list_for_each_entry_safe_reverse(pdata, temp, &free_list, list) + xmgmt_destroy_region(pdata->region); + + if (r_data->group_instance > 0) { + xleaf_destroy_group(xdev, r_data->group_instance); + r_data->group_instance = -1; + } + if (r_data->region->info) { + fpga_image_info_free(r_data->region->info); + r_data->region->info = NULL; + } +} + +void xmgmt_region_cleanup_all(struct xrt_device *xdev) +{ + struct xmgmt_region_match_arg arg = { 0 }; + struct fpga_region *base_region; + + arg.xdev = xdev; + + while ((base_region = fpga_region_class_find(NULL, &arg, xmgmt_region_match_base))) { + put_device(&base_region->dev); + + xmgmt_region_cleanup(base_region); + xmgmt_destroy_region(base_region); + } +} + +/* + * Program a region with a xclbin image. Bring up the subdevs and the + * group object to contain the subdevs. + */ +static int xmgmt_region_program(struct fpga_region *region, const void *xclbin, char *dtb) +{ + const struct axlf *xclbin_obj = xclbin; + struct fpga_image_info *info; + struct xrt_device *xdev; + struct xmgmt_region *r_data; + int rc; + + r_data = region->priv; + xdev = r_data->xdev; + + info = fpga_image_info_alloc(&xdev->dev); + if (!info) + return -ENOMEM; + + info->buf = xclbin; + info->count = xclbin_obj->header.length; + info->flags |= FPGA_MGR_PARTIAL_RECONFIG; + region->info = info; + rc = fpga_region_program_fpga(region); + if (rc) { + xrt_err(xdev, "programming xclbin failed, rc %d", rc); + return rc; + } + + /* free bridges to allow reprogram */ + if (region->get_bridges) + fpga_bridges_put(®ion->bridge_list); + + /* + * Next bringup the subdevs for this region which will be managed by + * its own group object. + */ + r_data->group_instance = xleaf_create_group(xdev, dtb); + if (r_data->group_instance < 0) { + xrt_err(xdev, "failed to create group, rc %d", + r_data->group_instance); + rc = r_data->group_instance; + return rc; + } + + rc = xleaf_wait_for_group_bringup(xdev); + if (rc) + xrt_err(xdev, "group bringup failed, rc %d", rc); + return rc; +} + +static int xmgmt_get_bridges(struct fpga_region *region) +{ + struct xmgmt_region *r_data = region->priv; + struct device *dev = &r_data->xdev->dev; + + return fpga_bridge_get_to_list(dev, region->info, ®ion->bridge_list); +} + +/* + * Program/create FPGA regions based on input xclbin file. + * 1. Identify a matching existing region for this xclbin + * 2. Tear down any previous objects for the found region + * 3. Program this region with input xclbin + * 4. Iterate over this region's interface uuids to determine if it defines any + * child region. Create fpga_region for the child region. + */ +int xmgmt_process_xclbin(struct xrt_device *xdev, + struct fpga_manager *fmgr, + const struct axlf *xclbin, + enum provider_kind kind) +{ + struct fpga_region *region, *compat_region = NULL; + struct xmgmt_region_match_arg arg = { 0 }; + struct xmgmt_region *r_data; + uuid_t compat_uuid; + char *dtb = NULL; + int rc, i; + + rc = xrt_xclbin_get_metadata(DEV(xdev), xclbin, &dtb); + if (rc) { + xrt_err(xdev, "failed to get dtb: %d", rc); + goto failed; + } + + rc = xrt_md_get_interface_uuids(DEV(xdev), dtb, 0, NULL); + if (rc < 0) { + xrt_err(xdev, "failed to get intf uuid"); + rc = -EINVAL; + goto failed; + } + arg.uuid_num = rc; + arg.uuids = kcalloc(arg.uuid_num, sizeof(uuid_t), GFP_KERNEL); + if (!arg.uuids) { + rc = -ENOMEM; + goto failed; + } + arg.xdev = xdev; + + rc = xrt_md_get_interface_uuids(DEV(xdev), dtb, arg.uuid_num, arg.uuids); + if (rc != arg.uuid_num) { + xrt_err(xdev, "only get %d uuids, expect %d", rc, arg.uuid_num); + rc = -EINVAL; + goto failed; + } + + /* if this is not base firmware, search for a compatible region */ + if (kind != XMGMT_BLP) { + compat_region = fpga_region_class_find(NULL, &arg, xmgmt_region_match); + if (!compat_region) { + xrt_err(xdev, "failed to get compatible region"); + rc = -ENOENT; + goto failed; + } + + xmgmt_region_cleanup(compat_region); + + rc = xmgmt_region_program(compat_region, xclbin, dtb); + if (rc) { + xrt_err(xdev, "failed to program region"); + goto failed; + } + } + + if (compat_region) + import_uuid(&compat_uuid, (const char *)compat_region->compat_id); + + /* create all the new regions contained in this xclbin */ + for (i = 0; i < arg.uuid_num; i++) { + if (compat_region && uuid_equal(&compat_uuid, &arg.uuids[i])) { + /* region for this interface already exists */ + continue; + } + + region = fpga_region_create(DEV(xdev), fmgr, xmgmt_get_bridges); + if (!region) { + xrt_err(xdev, "failed to create fpga region"); + rc = -EFAULT; + goto failed; + } + r_data = devm_kzalloc(DEV(xdev), sizeof(*r_data), GFP_KERNEL); + if (!r_data) { + rc = -ENOMEM; + fpga_region_free(region); + goto failed; + } + r_data->xdev = xdev; + r_data->region = region; + r_data->group_instance = -1; + uuid_copy(&r_data->interface_uuid, &arg.uuids[i]); + if (compat_region) + import_uuid(&r_data->depend_uuid, (const char *)compat_region->compat_id); + r_data->bridge = xmgmt_create_bridge(xdev, dtb); + if (!r_data->bridge) { + xrt_err(xdev, "failed to create fpga bridge"); + rc = -EFAULT; + devm_kfree(DEV(xdev), r_data); + fpga_region_free(region); + goto failed; + } + + region->compat_id = &r_data->compat_id; + export_uuid((char *)region->compat_id, &r_data->interface_uuid); + region->priv = r_data; + + rc = fpga_region_register(region); + if (rc) { + xrt_err(xdev, "failed to register fpga region"); + xmgmt_destroy_bridge(r_data->bridge); + fpga_region_free(region); + devm_kfree(DEV(xdev), r_data); + goto failed; + } + + xrt_info(xdev, "created fpga region %llx.%llx", + region->compat_id->id_h, region->compat_id->id_l); + } + + if (compat_region) + put_device(&compat_region->dev); + vfree(dtb); + kfree(arg.uuids); + return 0; + +failed: + if (compat_region) { + put_device(&compat_region->dev); + xmgmt_region_cleanup(compat_region); + } + + vfree(dtb); + kfree(arg.uuids); + return rc; +} diff --git a/drivers/fpga/xrt/mgmt/xrt-mgr.c b/drivers/fpga/xrt/mgmt/xrt-mgr.c new file mode 100644 index 000000000000..ab253b516e8d --- /dev/null +++ b/drivers/fpga/xrt/mgmt/xrt-mgr.c @@ -0,0 +1,190 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * FPGA Manager Support for Xilinx Alveo + * + * Copyright (C) 2020-2021 Xilinx, Inc. + * + * Authors: Sonal.Santan@xilinx.com + */ + +#include +#include +#include +#include +#include + +#include "xclbin-helper.h" +#include "xleaf.h" +#include "xrt-mgr.h" +#include "xleaf/axigate.h" +#include "xleaf/icap.h" +#include "xmgmt.h" + +struct xfpga_class { + struct xrt_device *xdev; + char name[64]; +}; + +/* + * xclbin download plumbing -- find the download subsystem, ICAP and + * pass the xclbin for heavy lifting + */ +static int xmgmt_download_bitstream(struct xrt_device *xdev, + const struct axlf *xclbin) + +{ + struct xclbin_bit_head_info bit_header = { 0 }; + struct xrt_device *icap_leaf = NULL; + struct xrt_icap_wr arg; + char *bitstream = NULL; + u64 bit_len; + int ret; + + ret = xrt_xclbin_get_section(DEV(xdev), xclbin, BITSTREAM, (void **)&bitstream, &bit_len); + if (ret) { + xrt_err(xdev, "bitstream not found"); + return -ENOENT; + } + ret = xrt_xclbin_parse_bitstream_header(DEV(xdev), bitstream, + XCLBIN_HWICAP_BITFILE_BUF_SZ, + &bit_header); + if (ret) { + ret = -EINVAL; + xrt_err(xdev, "invalid bitstream header"); + goto fail; + } + if (bit_header.header_length + bit_header.bitstream_length > bit_len) { + ret = -EINVAL; + xrt_err(xdev, "invalid bitstream length. header %d, bitstream %d, section len %lld", + bit_header.header_length, bit_header.bitstream_length, bit_len); + goto fail; + } + + icap_leaf = xleaf_get_leaf_by_id(xdev, XRT_SUBDEV_ICAP, XRT_INVALID_DEVICE_INST); + if (!icap_leaf) { + ret = -ENODEV; + xrt_err(xdev, "icap does not exist"); + goto fail; + } + arg.xiiw_bit_data = bitstream + bit_header.header_length; + arg.xiiw_data_len = bit_header.bitstream_length; + ret = xleaf_call(icap_leaf, XRT_ICAP_WRITE, &arg); + if (ret) { + xrt_err(xdev, "write bitstream failed, ret = %d", ret); + xleaf_put_leaf(xdev, icap_leaf); + goto fail; + } + + xleaf_put_leaf(xdev, icap_leaf); + vfree(bitstream); + + return 0; + +fail: + vfree(bitstream); + + return ret; +} + +/* + * There is no HW prep work we do here since we need the full + * xclbin for its sanity check. + */ +static int xmgmt_pr_write_init(struct fpga_manager *mgr, + struct fpga_image_info *info, + const char *buf, size_t count) +{ + const struct axlf *bin = (const struct axlf *)buf; + struct xfpga_class *obj = mgr->priv; + + if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) { + xrt_info(obj->xdev, "%s only supports partial reconfiguration\n", obj->name); + return -EINVAL; + } + + if (count < sizeof(struct axlf)) + return -EINVAL; + + if (count > bin->header.length) + return -EINVAL; + + xrt_info(obj->xdev, "Prepare download of xclbin %pUb of length %lld B", + &bin->header.uuid, bin->header.length); + + return 0; +} + +/* + * The implementation requires full xclbin image before we can start + * programming the hardware via ICAP subsystem. The full image is required + * for checking the validity of xclbin and walking the sections to + * discover the bitstream. + */ +static int xmgmt_pr_write(struct fpga_manager *mgr, + const char *buf, size_t count) +{ + const struct axlf *bin = (const struct axlf *)buf; + struct xfpga_class *obj = mgr->priv; + + if (bin->header.length != count) + return -EINVAL; + + return xmgmt_download_bitstream((void *)obj->xdev, bin); +} + +static int xmgmt_pr_write_complete(struct fpga_manager *mgr, + struct fpga_image_info *info) +{ + const struct axlf *bin = (const struct axlf *)info->buf; + struct xfpga_class *obj = mgr->priv; + + xrt_info(obj->xdev, "Finished download of xclbin %pUb", + &bin->header.uuid); + return 0; +} + +static enum fpga_mgr_states xmgmt_pr_state(struct fpga_manager *mgr) +{ + return FPGA_MGR_STATE_UNKNOWN; +} + +static const struct fpga_manager_ops xmgmt_pr_ops = { + .initial_header_size = sizeof(struct axlf), + .write_init = xmgmt_pr_write_init, + .write = xmgmt_pr_write, + .write_complete = xmgmt_pr_write_complete, + .state = xmgmt_pr_state, +}; + +struct fpga_manager *xmgmt_fmgr_probe(struct xrt_device *xdev) +{ + struct xfpga_class *obj = devm_kzalloc(DEV(xdev), sizeof(struct xfpga_class), + GFP_KERNEL); + struct fpga_manager *fmgr = NULL; + int ret = 0; + + if (!obj) + return ERR_PTR(-ENOMEM); + + snprintf(obj->name, sizeof(obj->name), "Xilinx Alveo FPGA Manager"); + obj->xdev = xdev; + fmgr = fpga_mgr_create(&xdev->dev, + obj->name, + &xmgmt_pr_ops, + obj); + if (!fmgr) + return ERR_PTR(-ENOMEM); + + ret = fpga_mgr_register(fmgr); + if (ret) { + fpga_mgr_free(fmgr); + return ERR_PTR(ret); + } + return fmgr; +} + +int xmgmt_fmgr_remove(struct fpga_manager *fmgr) +{ + fpga_mgr_unregister(fmgr); + return 0; +} diff --git a/drivers/fpga/xrt/mgmt/xrt-mgr.h b/drivers/fpga/xrt/mgmt/xrt-mgr.h new file mode 100644 index 000000000000..a3d1ab1c34f0 --- /dev/null +++ b/drivers/fpga/xrt/mgmt/xrt-mgr.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2020-2021 Xilinx, Inc. + * + * Authors: Sonal.Santan@xilinx.com + */ + +#ifndef _XRT_MGR_H_ +#define _XRT_MGR_H_ + +#include + +struct fpga_manager *xmgmt_fmgr_probe(struct xrt_device *xdev); +int xmgmt_fmgr_remove(struct fpga_manager *fmgr); + +#endif /* _XRT_MGR_H_ */ From patchwork Mon Aug 2 16:05:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lizhi Hou X-Patchwork-Id: 490430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DBB2C4320A for ; Mon, 2 Aug 2021 16:08:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8263F60F70 for ; Mon, 2 Aug 2021 16:08:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232664AbhHBQIb (ORCPT ); Mon, 2 Aug 2021 12:08:31 -0400 Received: from mail-co1nam11on2083.outbound.protection.outlook.com ([40.107.220.83]:63553 "EHLO NAM11-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229780AbhHBQIb (ORCPT ); Mon, 2 Aug 2021 12:08:31 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PJjeE7Z0UqcOV3y0GC+pKnCvvt0PoEp9aOtGIKPtgozMyf6T18LgYoJXZUNDyeywuxSxS/zZM2kQu/D5/SC++cX5BDWS/FLi3GUwctNpNR27bWH2ctTWaW5DM8mEzOYwSBupA1OepwAb3RiBGZOkn/OOnpT/QQ28jJY1ysRu0icJw8KkKF2/hWkcS9LBL4yvz9YFfjooIbeITIUTLD/OiFTUGcqdt2oViAXFhr+HOIAaNkjWTL+MJypZJe0VGTmyhQmKPXf5tPS5lYXZxi5B/qDwwGbw1vphCXnK8DV+K/YNruqu2vPfppqxyZ0deRhWQ2n2UkboP8eYhDtFiGUV2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bAdSl5BtPpX3N6B7GVv+60kBFM5C6Yx6nNmTgjtL26k=; b=bQjVjt6M/qAX21SeKjBDQ5bDJ2ste76X+lH5EsthC9knPrUX8acOu2TxJ0GHkay4orRcS97vQ3O972B/6GkYuPDNDopdul3zDGKe5+voShBbj4CU4BvFY02X/aNZ6bmQ621vjyHI9AEOPxllqxhocS/ARnGiBaGV3n0anwkXU4ejW4zWfu9ovZWCXWiIXqbj0WrnQFIB/HDQ2O28/PjxXlAefihdaBlfVILJGNkRMaL7OvSHfs3wLhZLR8opMyhcud7DOvACaNZiSxdUUKB3DcbzDaWECjjVZDAiaGesmv2AOkImt0xzTet/ZZXeLmedgVewXkKlY3pWpnp1Ay+iIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=kernel.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bAdSl5BtPpX3N6B7GVv+60kBFM5C6Yx6nNmTgjtL26k=; b=qHHrGhwiZkqxQU0lSQr/3HrKqXPyjU4x4wAZeg3AcZO7gal7SGJZnAoT88UgRIyC4KNdRVk6n33f83uVTeVK1CXlGW3NnUMnWHGKAL9zFv7RALXgQ3ndm7QBQr2ZdRVCrJYlJLmJaqAC8J2dxT2WNnd0z63eE7Cs+VkV5R7ZCtw= Received: from DM5PR06CA0059.namprd06.prod.outlook.com (2603:10b6:3:37::21) by BN6PR02MB2385.namprd02.prod.outlook.com (2603:10b6:404:36::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.21; Mon, 2 Aug 2021 16:08:17 +0000 Received: from DM3NAM02FT058.eop-nam02.prod.protection.outlook.com (2603:10b6:3:37:cafe::fa) by DM5PR06CA0059.outlook.office365.com (2603:10b6:3:37::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4373.17 via Frontend Transport; Mon, 2 Aug 2021 16:08:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; kernel.org; dkim=none (message not signed) header.d=none; kernel.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch01.xlnx.xilinx.com; Received: from xsj-pvapexch01.xlnx.xilinx.com (149.199.62.198) by DM3NAM02FT058.mail.protection.outlook.com (10.13.5.42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4373.18 via Frontend Transport; Mon, 2 Aug 2021 16:08:17 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch01.xlnx.xilinx.com (172.19.86.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 2 Aug 2021 09:08:06 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.2 via Frontend Transport; Mon, 2 Aug 2021 09:08:06 -0700 Envelope-to: mdf@kernel.org, robh@kernel.org, trix@redhat.com, devicetree@vger.kernel.org, linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org Received: from [10.23.145.232] (port=49074 helo=xsj-xw9400.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1mAaU2-0005L8-K3; Mon, 02 Aug 2021 09:08:06 -0700 Received: by xsj-xw9400.xilinx.com (Postfix, from userid 21952) id 191D3601B37; Mon, 2 Aug 2021 09:05:31 -0700 (PDT) From: Lizhi Hou To: CC: Lizhi Hou , , , , , , , , , , , Max Zhen Subject: [PATCH V9 XRT Alveo 14/14] fpga: xrt: Kconfig and Makefile updates for XRT drivers Date: Mon, 2 Aug 2021 09:05:21 -0700 Message-ID: <20210802160521.331031-15-lizhi.hou@xilinx.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210802160521.331031-1-lizhi.hou@xilinx.com> References: <20210802160521.331031-1-lizhi.hou@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e3aea5fe-840d-4103-357a-08d955cfbd80 X-MS-TrafficTypeDiagnostic: BN6PR02MB2385: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4941; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5DeoG+gPGNQ473TVzxpdmC20dIznfZs/J3m6+20jwrJaFZUUdcfyecjBB+bhDV/UV5c+VddgE3tSyUNj5cq2JEmD5Nc785VfvLBHtkzVp21eaI1s1q21w9x6v1wlao3Pjes/5IRqDetiw3tzCaAE0tgRNkcQ4/STwJ9r78v8P3nm+CYvD3tvcnbaPdJyIHqYaa76XDwVnWU5AyHcGW2JNJC3T0Dtd1vCqd+dEGiSFLEezfhYZneVYcgtvA4X1lqfnl/xk+2SdwFfvuIaf+qnXwbcHmCgxBsllR/YCJyhnELeyYpSNBgv0Nqs8FXNLAK+RwputGWWCCEryzLqAdgepWzhS3/zzTfRTI1xXZa2Xj6NP9b16cS4xQymRDeV7w5LsQ/HO+eFJB8+2el4tkn7woBelOuNMtIftsLL5ERX3gdfCr/tOvcpmPUqc8rNDXwGdzRF13Y6AClN3/VDEjs20QmBOgpzT3pXea7rQcsDVP8JbehF7dTk/smNnEjdj/Bt+ycPwqdHtot1vwP8P03vzy08+L6smEdwcNb/K6U/8wndONWHuDnUunVGqI1aOUgtDoJPSfnVhskjbMwnn9m4d+dtcUewzV/T3Sg0UCsEZHi6QMcEbB/E0Jq28ej16c5OgsRDN3duGpqXD8QXxR5Mkmq+cvnfK5hFcJ+SJ8KxGEk46BDf6FPdlfTCQmcaqA673Rhk0YtCkZH9Ppa64s6iNQ== X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch01.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(4636009)(396003)(376002)(346002)(39860400002)(136003)(46966006)(36840700001)(70206006)(426003)(26005)(44832011)(5660300002)(356005)(2906002)(4326008)(36756003)(7636003)(82740400003)(70586007)(6266002)(107886003)(2616005)(6666004)(6916009)(36906005)(316002)(186003)(8936002)(82310400003)(336012)(15650500001)(36860700001)(42186006)(1076003)(478600001)(83380400001)(8676002)(47076005)(54906003); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2021 16:08:17.4423 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e3aea5fe-840d-4103-357a-08d955cfbd80 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch01.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: DM3NAM02FT058.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR02MB2385 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Update fpga Kconfig/Makefile and add Kconfig/Makefile for new drivers. Signed-off-by: Sonal Santan Signed-off-by: Max Zhen Signed-off-by: Lizhi Hou Reviewed-by: Tom Rix --- drivers/Makefile | 1 + drivers/fpga/Kconfig | 2 ++ drivers/fpga/Makefile | 5 +++++ drivers/fpga/xrt/Kconfig | 8 ++++++++ drivers/fpga/xrt/lib/Kconfig | 17 +++++++++++++++++ drivers/fpga/xrt/lib/Makefile | 24 ++++++++++++++++++++++++ drivers/fpga/xrt/metadata/Kconfig | 12 ++++++++++++ drivers/fpga/xrt/metadata/Makefile | 16 ++++++++++++++++ drivers/fpga/xrt/mgmt/Kconfig | 15 +++++++++++++++ drivers/fpga/xrt/mgmt/Makefile | 19 +++++++++++++++++++ 10 files changed, 119 insertions(+) create mode 100644 drivers/fpga/xrt/Kconfig create mode 100644 drivers/fpga/xrt/lib/Kconfig create mode 100644 drivers/fpga/xrt/lib/Makefile create mode 100644 drivers/fpga/xrt/metadata/Kconfig create mode 100644 drivers/fpga/xrt/metadata/Makefile create mode 100644 drivers/fpga/xrt/mgmt/Kconfig create mode 100644 drivers/fpga/xrt/mgmt/Makefile diff --git a/drivers/Makefile b/drivers/Makefile index 27c018bdf4de..64fba9d3adb9 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -180,6 +180,7 @@ obj-$(CONFIG_STM) += hwtracing/stm/ obj-$(CONFIG_ANDROID) += android/ obj-$(CONFIG_NVMEM) += nvmem/ obj-$(CONFIG_FPGA) += fpga/ +obj-$(CONFIG_FPGA_XRT_METADATA) += fpga/ obj-$(CONFIG_FSI) += fsi/ obj-$(CONFIG_TEE) += tee/ obj-$(CONFIG_MULTIPLEXER) += mux/ diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig index 8cd454ee20c0..526447770cab 100644 --- a/drivers/fpga/Kconfig +++ b/drivers/fpga/Kconfig @@ -234,4 +234,6 @@ config FPGA_MGR_ZYNQMP_FPGA to configure the programmable logic(PL) through PS on ZynqMP SoC. +source "drivers/fpga/xrt/Kconfig" + endif # FPGA diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile index 18dc9885883a..4b887bf95cb3 100644 --- a/drivers/fpga/Makefile +++ b/drivers/fpga/Makefile @@ -48,3 +48,8 @@ obj-$(CONFIG_FPGA_DFL_NIOS_INTEL_PAC_N3000) += dfl-n3000-nios.o # Drivers for FPGAs which implement DFL obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o + +# XRT drivers for Alveo +obj-$(CONFIG_FPGA_XRT_METADATA) += xrt/metadata/ +obj-$(CONFIG_FPGA_XRT_LIB) += xrt/lib/ +obj-$(CONFIG_FPGA_XRT_XMGMT) += xrt/mgmt/ diff --git a/drivers/fpga/xrt/Kconfig b/drivers/fpga/xrt/Kconfig new file mode 100644 index 000000000000..0e2c59589ddd --- /dev/null +++ b/drivers/fpga/xrt/Kconfig @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Xilinx Alveo FPGA device configuration +# + +source "drivers/fpga/xrt/metadata/Kconfig" +source "drivers/fpga/xrt/lib/Kconfig" +source "drivers/fpga/xrt/mgmt/Kconfig" diff --git a/drivers/fpga/xrt/lib/Kconfig b/drivers/fpga/xrt/lib/Kconfig new file mode 100644 index 000000000000..935369fad570 --- /dev/null +++ b/drivers/fpga/xrt/lib/Kconfig @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# XRT Alveo FPGA device configuration +# + +config FPGA_XRT_LIB + tristate "XRT Alveo Driver Library" + depends on HWMON && PCI && HAS_IOMEM + select FPGA_XRT_METADATA + select REGMAP_MMIO + help + Select this option to enable Xilinx XRT Alveo driver library. This + library is core infrastructure of XRT Alveo FPGA drivers which + provides functions for working with device nodes, iteration and + lookup of platform devices, common interfaces for platform devices, + plumbing of function call and ioctls between platform devices and + parent partitions. diff --git a/drivers/fpga/xrt/lib/Makefile b/drivers/fpga/xrt/lib/Makefile new file mode 100644 index 000000000000..55cd6063a324 --- /dev/null +++ b/drivers/fpga/xrt/lib/Makefile @@ -0,0 +1,24 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved. +# +# Authors: Sonal.Santan@xilinx.com +# + +FULL_XRT_PATH=$(srctree)/$(src)/.. +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt + +obj-$(CONFIG_FPGA_XRT_LIB) += xrt-lib.o + +xrt-lib-objs := \ + lib-drv.o \ + xroot.o \ + xclbin.o \ + subdev.o \ + cdev.o \ + group.o \ + xleaf/axigate.o \ + xleaf/icap.o + +ccflags-y := -I$(FULL_XRT_PATH)/include \ + -I$(FULL_DTC_PATH) diff --git a/drivers/fpga/xrt/metadata/Kconfig b/drivers/fpga/xrt/metadata/Kconfig new file mode 100644 index 000000000000..129adda47e94 --- /dev/null +++ b/drivers/fpga/xrt/metadata/Kconfig @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# XRT Alveo FPGA device configuration +# + +config FPGA_XRT_METADATA + bool "XRT Alveo Driver Metadata Parser" + select LIBFDT + help + This option provides helper functions to parse Xilinx Alveo FPGA + firmware metadata. The metadata is in device tree format and the + XRT driver uses it to discover the HW subsystems behind PCIe BAR. diff --git a/drivers/fpga/xrt/metadata/Makefile b/drivers/fpga/xrt/metadata/Makefile new file mode 100644 index 000000000000..14f65ef1595c --- /dev/null +++ b/drivers/fpga/xrt/metadata/Makefile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved. +# +# Authors: Sonal.Santan@xilinx.com +# + +FULL_XRT_PATH=$(srctree)/$(src)/.. +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt + +obj-$(CONFIG_FPGA_XRT_METADATA) += xrt-md.o + +xrt-md-objs := metadata.o + +ccflags-y := -I$(FULL_XRT_PATH)/include \ + -I$(FULL_DTC_PATH) diff --git a/drivers/fpga/xrt/mgmt/Kconfig b/drivers/fpga/xrt/mgmt/Kconfig new file mode 100644 index 000000000000..31e9e19fffb8 --- /dev/null +++ b/drivers/fpga/xrt/mgmt/Kconfig @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Xilinx XRT FPGA device configuration +# + +config FPGA_XRT_XMGMT + tristate "Xilinx Alveo Management Driver" + depends on FPGA_XRT_LIB + select FPGA_XRT_METADATA + select FPGA_BRIDGE + select FPGA_REGION + help + Select this option to enable XRT PCIe driver for Xilinx Alveo FPGA. + This driver provides interfaces for userspace application to access + Alveo FPGA device. diff --git a/drivers/fpga/xrt/mgmt/Makefile b/drivers/fpga/xrt/mgmt/Makefile new file mode 100644 index 000000000000..16644571b673 --- /dev/null +++ b/drivers/fpga/xrt/mgmt/Makefile @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved. +# +# Authors: Sonal.Santan@xilinx.com +# + +FULL_XRT_PATH=$(srctree)/$(src)/.. +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt + +obj-$(CONFIG_FPGA_XRT_XMGMT) += xrt-mgmt.o + +xrt-mgmt-objs := root.o \ + xmgmt-main.o \ + xrt-mgr.o \ + xmgmt-main-region.o + +ccflags-y := -I$(FULL_XRT_PATH)/include \ + -I$(FULL_DTC_PATH)