From patchwork Sat Sep 26 04:24:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313564 Delivered-To: patches@linaro.org Received: by 2002:a92:5ad1:0:0:0:0:0 with SMTP id b78csp1680530ilg; Fri, 25 Sep 2020 21:24:58 -0700 (PDT) X-Received: by 2002:a63:161e:: with SMTP id w30mr1570521pgl.255.1601094298785; Fri, 25 Sep 2020 21:24:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601094298; cv=none; d=google.com; s=arc-20160816; b=m/LjXmXcMM88vN6xG2fLM4sIjJskIgI8w3lN9POgsBwhwWHXGZ1sIJ3YgMXbThlL0O F20yAFtOYobQisMkgLd7Dq7IHtNRlYjtf8C/J4UM5sXGh7+Y8nOSrul69LXEnkz1975V rSokdk2GVwydxm94m3gbxNlioZFqHlEJa+DVabGLBpOK13MDRLndQsqL+4Gx+o30kOAv 0mnv3NJnEPeIqK3rAk4uLhr9M5esRs8NhvYosq/EhcDa6s7OQZP7m5wppZ8Jq8uQNVR8 PDIaPmIcyioBUREkxyTeIsGljmTQN8eh/z8jJ8DpBLz4ZU1qoo4cWENrxkVs2MdwMK40 zfZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=NEH7dSpxqauBk78qE1pmPtBx4lDfCsXkpGiAZ8uIYoU=; b=bTVvtiBmUZQxcjIC/XNGJhAzHSTT/McAPuMUzaRrmCPe8fTd8jkLjA6cnhmjPpl3N2 HAQsoIgkeTVeej+23KBWrLAUIpJwmfmH04Qxn4h6ZgUuf26a6w3EiPKeNNdtp7KsLCFt o5PhkgX5MtLkinMWPJcuiJIDzJgCHcUGCUH4C9KtVuvdyiDSKIXi+K3w6/wVjQfWo8We 2V1r+zGsahW6+d5ST4QcTzTVDYP4c+6yQddXc3/9nHdlQRD+9nNsGrh4Ss8eNvPdTrz2 sFU6FNEPZyrAMKl1JgfIGVU2Lk5JmWVjCcf+2frq4qqxGDr4R2C4z0FhTtyD12JYFkSD SEEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=qYnAL5+5; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id j4sor436244pjd.9.2020.09.25.21.24.58 for (Google Transport Security); Fri, 25 Sep 2020 21:24:58 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=qYnAL5+5; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NEH7dSpxqauBk78qE1pmPtBx4lDfCsXkpGiAZ8uIYoU=; b=qYnAL5+5YevrqN1Msnjai60b/eeSbfx4+aAdG1jq+kWoKGB5iX+IJHR2gapx9udbGB oyJwv7pmiTIzpMSE/mDQoYU9AqpaGzogty93KtTaw++bljZMr3NrVaL90mW4ww5Sd3D0 K01vIfSy/CNgnkUuLHgiKenT1urK4e6V7qwo5IjqEAymLlaqgZ6u7ZFfK0oRiytkTsbI iXSA/LCin6mqBpDfNlr9VY/mKWmschenAgLKR4bcVjgyJmbOAuShUl+B5Unj4uIqHVF1 8anJbl4ObNCpffkq1qEinuche4DSddCl1cPtq3ayZuokY33zv38QrK2d0F1X/BVW20Qf K6zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NEH7dSpxqauBk78qE1pmPtBx4lDfCsXkpGiAZ8uIYoU=; b=ty5kNFVDSt53ceHemKHhJ+gsJC/CyKSC5TXCh+wp+jPDOifJyORkPJNBkcHfCxbFr/ 2ylLhvFil2RqGbfpJy03zNaDDTVFVp47YLOxFxx0gphbR1jyYKaRyuWQMQ01Sc2QfOuF v8VPqw/W6uoc7kjgamKuxbAflfLSDpB5jFrrb5enagk3BLmEodIb3T3zXP0Ehh5qEVNv h4tmbmcyK9H9KwhyP2hOL6XFcMC5J8P8EZjpeu2nmeDZvu17rZD23PKUjJZ0SRXQE7m7 9pr+MZSSPxDSgGJVme6hYdj9hc5iiC3tcYFXFoKYrH+7sOXFMQrfLaMOsjxfmjx1dzng iMAQ== X-Gm-Message-State: AOAM5336OUDP5kHPehEJEjPpnXf8q3AHMXd8oR9tSV3E42GKUi2wSs0r po2FVhv2PW5D23D6aBAM0fEdLdrZ X-Google-Smtp-Source: ABdhPJwetII4Mw9P6pPBXOCqMXrXkdH0dbYKAMabb15GOX4+ef7pGIUAr10E9lRSd+VVywrZzwLRLA== X-Received: by 2002:a17:90a:f407:: with SMTP id ch7mr694079pjb.142.1601094298340; Fri, 25 Sep 2020 21:24:58 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id a5sm3585886pgk.13.2020.09.25.21.24.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Sep 2020 21:24:57 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [RFC][PATCH 1/6] dma-buf: system_heap: Rework system heap to use sgtables instead of pagelists Date: Sat, 26 Sep 2020 04:24:48 +0000 Message-Id: <20200926042453.67517-2-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200926042453.67517-1-john.stultz@linaro.org> References: <20200926042453.67517-1-john.stultz@linaro.org> MIME-Version: 1.0 In preparation for some patches to optmize the system heap code, rework the dmabuf exporter to utilize sgtables rather then pageslists for tracking the associated pages. This will allow for large order page allocations, as well as more efficient page pooling. In doing so, the system heap stops using the heap-helpers logic which sadly is not quite as generic as I was hoping it to be, so this patch adds heap specific implementations of the dma_buf_ops function handlers. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/system_heap.c | 343 ++++++++++++++++++++++++---- 1 file changed, 297 insertions(+), 46 deletions(-) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index 0bf688e3c023..ddfa17dc48a8 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -3,7 +3,11 @@ * DMABUF System heap exporter * * Copyright (C) 2011 Google, Inc. - * Copyright (C) 2019 Linaro Ltd. + * Copyright (C) 2019, 2020 Linaro Ltd. + * + * Portions based off of Andrew Davis' SRAM heap: + * Copyright (C) 2019 Texas Instruments Incorporated - http://www.ti.com/ + * Andrew F. Davis */ #include @@ -15,72 +19,320 @@ #include #include #include -#include -#include - -#include "heap-helpers.h" +#include struct dma_heap *sys_heap; -static void system_heap_free(struct heap_helper_buffer *buffer) +struct system_heap_buffer { + struct dma_heap *heap; + struct list_head attachments; + struct mutex lock; + unsigned long len; + struct sg_table sg_table; + int vmap_cnt; + void *vaddr; +}; + +struct dma_heap_attachment { + struct device *dev; + struct sg_table *table; + struct list_head list; +}; + +static struct sg_table *dup_sg_table(struct sg_table *table) { - pgoff_t pg; + struct sg_table *new_table; + int ret, i; + struct scatterlist *sg, *new_sg; + + new_table = kzalloc(sizeof(*new_table), GFP_KERNEL); + if (!new_table) + return ERR_PTR(-ENOMEM); + + ret = sg_alloc_table(new_table, table->nents, GFP_KERNEL); + if (ret) { + kfree(new_table); + return ERR_PTR(-ENOMEM); + } + + new_sg = new_table->sgl; + for_each_sgtable_sg(table, sg, i) { + sg_set_page(new_sg, sg_page(sg), sg->length, sg->offset); + new_sg = sg_next(new_sg); + } + + return new_table; +} + +static int system_heap_attach(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + struct sg_table *table; + + a = kzalloc(sizeof(*a), GFP_KERNEL); + if (!a) + return -ENOMEM; + + table = dup_sg_table(&buffer->sg_table); + if (IS_ERR(table)) { + kfree(a); + return -ENOMEM; + } + + a->table = table; + a->dev = attachment->dev; + INIT_LIST_HEAD(&a->list); + + attachment->priv = a; + + mutex_lock(&buffer->lock); + list_add(&a->list, &buffer->attachments); + mutex_unlock(&buffer->lock); + + return 0; +} + +static void system_heap_detatch(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a = attachment->priv; + + mutex_lock(&buffer->lock); + list_del(&a->list); + mutex_unlock(&buffer->lock); + + sg_free_table(a->table); + kfree(a->table); + kfree(a); +} + +static struct sg_table *system_heap_map_dma_buf(struct dma_buf_attachment *attachment, + enum dma_data_direction direction) +{ + struct dma_heap_attachment *a = attachment->priv; + struct sg_table *table = a->table; + + if (!dma_map_sg(attachment->dev, table->sgl, table->nents, direction)) + return ERR_PTR(-ENOMEM); + + return table; +} + +static void system_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, + struct sg_table *table, + enum dma_data_direction direction) +{ + dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); +} + +static int system_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + int ret = 0; + + mutex_lock(&buffer->lock); + + if (buffer->vmap_cnt) + invalidate_kernel_vmap_range(buffer->vaddr, buffer->len); + + list_for_each_entry(a, &buffer->attachments, list) { + dma_sync_sg_for_cpu(a->dev, a->table->sgl, a->table->nents, + direction); + } + mutex_unlock(&buffer->lock); + + return ret; +} + +static int system_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + + mutex_lock(&buffer->lock); + + if (buffer->vmap_cnt) + flush_kernel_vmap_range(buffer->vaddr, buffer->len); + + list_for_each_entry(a, &buffer->attachments, list) { + dma_sync_sg_for_device(a->dev, a->table->sgl, a->table->nents, + direction); + } + mutex_unlock(&buffer->lock); + + return 0; +} + +static int system_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct sg_table *table = &buffer->sg_table; + unsigned long addr = vma->vm_start; + struct sg_page_iter piter; + int ret; + + for_each_sgtable_page(table, &piter, vma->vm_pgoff) { + struct page *page = sg_page_iter_page(&piter); + + ret = remap_pfn_range(vma, addr, page_to_pfn(page), PAGE_SIZE, + vma->vm_page_prot); + if (ret) + return ret; + addr += PAGE_SIZE; + if (addr >= vma->vm_end) + return 0; + } + return 0; +} + +static void *system_heap_do_vmap(struct system_heap_buffer *buffer) +{ + struct sg_table *table = &buffer->sg_table; + int npages = PAGE_ALIGN(buffer->len) / PAGE_SIZE; + struct page **pages = vmalloc(sizeof(struct page *) * npages); + struct page **tmp = pages; + struct sg_page_iter piter; + void *vaddr; + + if (!pages) + return ERR_PTR(-ENOMEM); + + for_each_sgtable_page(table, &piter, 0) { + WARN_ON(tmp - pages >= npages); + *tmp++ = sg_page_iter_page(&piter); + } + + vaddr = vmap(pages, npages, VM_MAP, PAGE_KERNEL); + vfree(pages); + + if (!vaddr) + return ERR_PTR(-ENOMEM); + + return vaddr; +} - for (pg = 0; pg < buffer->pagecount; pg++) - __free_page(buffer->pages[pg]); - kfree(buffer->pages); +static void *system_heap_vmap(struct dma_buf *dmabuf) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + void *vaddr; + + mutex_lock(&buffer->lock); + if (buffer->vmap_cnt) { + buffer->vmap_cnt++; + return buffer->vaddr; + } + + vaddr = system_heap_do_vmap(buffer); + if (IS_ERR(vaddr)) + return vaddr; + + buffer->vaddr = vaddr; + buffer->vmap_cnt++; + mutex_unlock(&buffer->lock); + + return vaddr; +} + +static void system_heap_vunmap(struct dma_buf *dmabuf, void *vaddr) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + + mutex_lock(&buffer->lock); + if (!--buffer->vmap_cnt) { + vunmap(buffer->vaddr); + buffer->vaddr = NULL; + } + mutex_unlock(&buffer->lock); +} + +static void system_heap_dma_buf_release(struct dma_buf *dmabuf) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct sg_table *table; + struct scatterlist *sg; + int i; + + table = &buffer->sg_table; + for_each_sgtable_sg(table, sg, i) + __free_page(sg_page(sg)); + sg_free_table(table); kfree(buffer); } +const struct dma_buf_ops system_heap_buf_ops = { + .attach = system_heap_attach, + .detach = system_heap_detatch, + .map_dma_buf = system_heap_map_dma_buf, + .unmap_dma_buf = system_heap_unmap_dma_buf, + .begin_cpu_access = system_heap_dma_buf_begin_cpu_access, + .end_cpu_access = system_heap_dma_buf_end_cpu_access, + .mmap = system_heap_mmap, + .vmap = system_heap_vmap, + .vunmap = system_heap_vunmap, + .release = system_heap_dma_buf_release, +}; + static int system_heap_allocate(struct dma_heap *heap, unsigned long len, unsigned long fd_flags, unsigned long heap_flags) { - struct heap_helper_buffer *helper_buffer; + struct system_heap_buffer *buffer; + DEFINE_DMA_BUF_EXPORT_INFO(exp_info); struct dma_buf *dmabuf; - int ret = -ENOMEM; + struct sg_table *table; + struct scatterlist *sg; + pgoff_t pagecount; pgoff_t pg; + int i, ret = -ENOMEM; - helper_buffer = kzalloc(sizeof(*helper_buffer), GFP_KERNEL); - if (!helper_buffer) + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); + if (!buffer) return -ENOMEM; - init_heap_helper_buffer(helper_buffer, system_heap_free); - helper_buffer->heap = heap; - helper_buffer->size = len; - - helper_buffer->pagecount = len / PAGE_SIZE; - helper_buffer->pages = kmalloc_array(helper_buffer->pagecount, - sizeof(*helper_buffer->pages), - GFP_KERNEL); - if (!helper_buffer->pages) { - ret = -ENOMEM; - goto err0; - } + INIT_LIST_HEAD(&buffer->attachments); + mutex_init(&buffer->lock); + buffer->heap = heap; + buffer->len = len; - for (pg = 0; pg < helper_buffer->pagecount; pg++) { + table = &buffer->sg_table; + pagecount = len / PAGE_SIZE; + if (sg_alloc_table(table, pagecount, GFP_KERNEL)) + goto free_buffer; + + sg = table->sgl; + for (pg = 0; pg < pagecount; pg++) { + struct page *page; /* * Avoid trying to allocate memory if the process - * has been killed by by SIGKILL + * has been killed by SIGKILL */ if (fatal_signal_pending(current)) - goto err1; - - helper_buffer->pages[pg] = alloc_page(GFP_KERNEL | __GFP_ZERO); - if (!helper_buffer->pages[pg]) - goto err1; + goto free_pages; + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + goto free_pages; + sg_set_page(sg, page, page_size(page), 0); + sg = sg_next(sg); } /* create the dmabuf */ - dmabuf = heap_helper_export_dmabuf(helper_buffer, fd_flags); + exp_info.ops = &system_heap_buf_ops; + exp_info.size = buffer->len; + exp_info.flags = fd_flags; + exp_info.priv = buffer; + dmabuf = dma_buf_export(&exp_info); if (IS_ERR(dmabuf)) { ret = PTR_ERR(dmabuf); - goto err1; + goto free_pages; } - helper_buffer->dmabuf = dmabuf; - ret = dma_buf_fd(dmabuf, fd_flags); if (ret < 0) { dma_buf_put(dmabuf); @@ -90,12 +342,12 @@ static int system_heap_allocate(struct dma_heap *heap, return ret; -err1: - while (pg > 0) - __free_page(helper_buffer->pages[--pg]); - kfree(helper_buffer->pages); -err0: - kfree(helper_buffer); +free_pages: + for_each_sgtable_sg(table, sg, i) + __free_page(sg_page(sg)); + sg_free_table(table); +free_buffer: + kfree(buffer); return ret; } @@ -107,7 +359,6 @@ static const struct dma_heap_ops system_heap_ops = { static int system_heap_create(void) { struct dma_heap_export_info exp_info; - int ret = 0; exp_info.name = "system"; exp_info.ops = &system_heap_ops; @@ -115,9 +366,9 @@ static int system_heap_create(void) sys_heap = dma_heap_add(&exp_info); if (IS_ERR(sys_heap)) - ret = PTR_ERR(sys_heap); + return PTR_ERR(sys_heap); - return ret; + return 0; } module_init(system_heap_create); MODULE_LICENSE("GPL v2"); From patchwork Sat Sep 26 04:24:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313565 Delivered-To: patches@linaro.org Received: by 2002:a92:5ad1:0:0:0:0:0 with SMTP id b78csp1680555ilg; Fri, 25 Sep 2020 21:25:01 -0700 (PDT) X-Received: by 2002:a63:f961:: with SMTP id q33mr1530117pgk.95.1601094300946; Fri, 25 Sep 2020 21:25:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601094300; cv=none; d=google.com; s=arc-20160816; b=sUdORWjks7gNvVRbcx70KnXGnt6a+34Doi2u+PeRnKMMlLAQ04ksy9yY6vJOIfEiUg lkkkC0+XUn5/U4d2DwqX6QLvsjMhTE8X+4ZIjh9E5DbmLorsQZhdBbWiMMs3AiFgDegr qZo/IVZlvS3kvuT/TstdN482M1xr/yvX6ijTQsIcXQhTfZjfjKi1htZHmTH2IqMTC5KB sjVd4Vaogg+he8PvQ0aiQ8X1jSWopAWFl0s4JjyaQYVHFLZw6if5Ds7cD4CJyrEImacH 8mMLFp8zqQKwEv5927Dcc0jI9JXWNHTRdp/lMrHUhpKHpJjC/yk7ywBWXbITXG4Py5zd sAIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=q/YHFkJjnrzpougboI5H9Mm3RBmSahSsWVWvUb6SAeY=; b=ToZCAVWGaNoJ3TxPgbUXs808IwGWTxhSjUpK8BjhUNETVB8n4AtIk9h1IiKukpwV5r 3GPHGc5tDUw0sBmz8N2ClYruN5NiDIWiwJgbrXpta8OqqEw5KUqwa4TQgPhl2m6wROze zKGh/G6ZV16NrQc/a606J5zrnL/lRCAK68zTpFfBmW0O5YYYoPxw9bwjeXVRSzWEXrdQ OeWNSpRLm0AKtDvQgRx8aJbGwmXjPUZuiGwp9ufv7POPE+cmm/Ye5E7q5/hbC4lynFzZ s1KELgB4hvsP1Es5Zvms+hjE1595x5pwrVDDiXj+j7NR2lqOc0tgpzS8ICM0V2/0foLl SQcA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pYshy51K; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z5sor415139pju.33.2020.09.25.21.25.00 for (Google Transport Security); Fri, 25 Sep 2020 21:25:00 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pYshy51K; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=q/YHFkJjnrzpougboI5H9Mm3RBmSahSsWVWvUb6SAeY=; b=pYshy51K50oZf9Caenwo7Vuah8DqJtbjU74d8OWrnajJ3wXbRD1+qQHDhBmzF5Qx0T 0/iMUp5K7BYEV91ZGP3/iMCUyZoSahF377/2Nz+Z8xEwcPQdnKHkRmSmyCHxwoLjxgGA izkCHUHyzWUVeYuWKue9lojLJzZPeH7tAcNjuW7jXRpS5CAJ3GoaD7ip5+p8/z/qodl1 QObOiQGpmp9x2dSHZv8QZv90j1qWpfNlKfGfUCyGxBg1XMMtSvCQA9Jz6WXghLzBbvFv 9TUiXBFMnHxg8lklqJLcRLvXYySPlXDS5zNlERERZghAklqgyW148Ax8dBYBsVssF82E LoLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=q/YHFkJjnrzpougboI5H9Mm3RBmSahSsWVWvUb6SAeY=; b=HHo+Q0Dp/tGlrQqLHgxyKJCptk9NfJ77Am55qHfAk20EDo7Y7GbgPXy6gE6Xa4D/yl zqHoBLpW/RN9wyx1sbNMfkaTGiUStblnWJT/wSZx+mNOqPxy25p6+uzF+ZKFDc1erzUL 9XH8ULf9dWAvIlzF/R4QkjnFkDJTONIFf94saZJeqbm4qa+mLDVNZFuDdMBdanDGPxnO TYL5NDOSRxJUrep+tY1yz1vD0/6OjGgv7FrcwZQqNyVEsBxy4um2yLSZQ4v3495erLhQ ORpPGdMXZzb336rF5eevmPilCvhmatH+VW0UbywnVZrfeZad589/zuxpbAlSU2Trpzu4 6QSQ== X-Gm-Message-State: AOAM531ivMLe6WXPea/8jS6QTE1ahHGNkV5fdO8aLIo7tSpNw+cOx8k1 ZCvnzt9N8fnGIdF0gjULLVEIBF1f X-Google-Smtp-Source: ABdhPJyezNoUAo89S8cL5s0K0ZQlVU56nzJuFtG9yIunTjup+PAhqX2pDoDhIGA3V6oo0lGSUEQdyg== X-Received: by 2002:a17:90a:3984:: with SMTP id z4mr659973pjb.131.1601094300525; Fri, 25 Sep 2020 21:25:00 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id a5sm3585886pgk.13.2020.09.25.21.24.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Sep 2020 21:24:59 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [RFC][PATCH 2/6] dma-buf: heaps: Move heap-helper logic into the cma_heap implementation Date: Sat, 26 Sep 2020 04:24:49 +0000 Message-Id: <20200926042453.67517-3-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200926042453.67517-1-john.stultz@linaro.org> References: <20200926042453.67517-1-john.stultz@linaro.org> MIME-Version: 1.0 Since the heap-helpers logic ended up not being as generic as hoped, move the heap-helpers dma_buf_ops implementations into the cma_heap directly. This will allow us to remove the heap_helpers code in a following patch. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/cma_heap.c | 322 ++++++++++++++++++++++++++----- 1 file changed, 270 insertions(+), 52 deletions(-) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index 626cf7fd033a..3adfdbed0829 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -2,76 +2,291 @@ /* * DMABUF CMA heap exporter * - * Copyright (C) 2012, 2019 Linaro Ltd. + * Copyright (C) 2012, 2019, 2020 Linaro Ltd. * Author: for ST-Ericsson. + * + * Also utilizing parts of Andrew Davis' SRAM heap: + * Copyright (C) 2019 Texas Instruments Incorporated - http://www.ti.com/ + * Andrew F. Davis */ - #include -#include #include -#include #include +#include +#include #include -#include #include +#include +#include #include -#include #include -#include +#include -#include "heap-helpers.h" struct cma_heap { struct dma_heap *heap; struct cma *cma; }; -static void cma_heap_free(struct heap_helper_buffer *buffer) +struct cma_heap_buffer { + struct cma_heap *heap; + struct list_head attachments; + struct mutex lock; + unsigned long len; + struct page *cma_pages; + struct page **pages; + pgoff_t pagecount; + int vmap_cnt; + void *vaddr; +}; + +struct dma_heap_attachment { + struct device *dev; + struct sg_table table; + struct list_head list; +}; + +static int cma_heap_attach(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) { - struct cma_heap *cma_heap = dma_heap_get_drvdata(buffer->heap); - unsigned long nr_pages = buffer->pagecount; - struct page *cma_pages = buffer->priv_virt; + struct cma_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + int ret; - /* free page list */ - kfree(buffer->pages); - /* release memory */ - cma_release(cma_heap->cma, cma_pages, nr_pages); + a = kzalloc(sizeof(*a), GFP_KERNEL); + if (!a) + return -ENOMEM; + + ret = sg_alloc_table_from_pages(&a->table, buffer->pages, + buffer->pagecount, 0, + buffer->pagecount << PAGE_SHIFT, + GFP_KERNEL); + if (ret) { + kfree(a); + return ret; + } + + a->dev = attachment->dev; + INIT_LIST_HEAD(&a->list); + + attachment->priv = a; + + mutex_lock(&buffer->lock); + list_add(&a->list, &buffer->attachments); + mutex_unlock(&buffer->lock); + + return 0; +} + +static void cma_heap_detatch(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a = attachment->priv; + + mutex_lock(&buffer->lock); + list_del(&a->list); + mutex_unlock(&buffer->lock); + + sg_free_table(&a->table); + kfree(a); +} + +static struct sg_table *cma_heap_map_dma_buf(struct dma_buf_attachment *attachment, + enum dma_data_direction direction) +{ + struct dma_heap_attachment *a = attachment->priv; + struct sg_table *table = &a->table; + + if (!dma_map_sg(attachment->dev, table->sgl, table->nents, + direction)) + table = ERR_PTR(-ENOMEM); + return table; +} + +static void cma_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, + struct sg_table *table, + enum dma_data_direction direction) +{ + dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); +} + +static int cma_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + int ret = 0; + + if (buffer->vmap_cnt) + invalidate_kernel_vmap_range(buffer->vaddr, buffer->len); + + mutex_lock(&buffer->lock); + list_for_each_entry(a, &buffer->attachments, list) { + dma_sync_sg_for_cpu(a->dev, a->table.sgl, a->table.nents, + direction); + } + mutex_unlock(&buffer->lock); + + return ret; +} + +static int cma_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + + if (buffer->vmap_cnt) + flush_kernel_vmap_range(buffer->vaddr, buffer->len); + + mutex_lock(&buffer->lock); + list_for_each_entry(a, &buffer->attachments, list) { + dma_sync_sg_for_device(a->dev, a->table.sgl, a->table.nents, + direction); + } + mutex_unlock(&buffer->lock); + + return 0; +} + +static vm_fault_t cma_heap_vm_fault(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct cma_heap_buffer *buffer = vma->vm_private_data; + + if (vmf->pgoff > buffer->pagecount) + return VM_FAULT_SIGBUS; + + vmf->page = buffer->pages[vmf->pgoff]; + get_page(vmf->page); + + return 0; +} + +static const struct vm_operations_struct dma_heap_vm_ops = { + .fault = cma_heap_vm_fault, +}; + +static int cma_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + + if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0) + return -EINVAL; + + vma->vm_ops = &dma_heap_vm_ops; + vma->vm_private_data = buffer; + + return 0; +} + +static void *cma_heap_do_vmap(struct cma_heap_buffer *buffer) +{ + void *vaddr; + + vaddr = vmap(buffer->pages, buffer->pagecount, VM_MAP, PAGE_KERNEL); + if (!vaddr) + return ERR_PTR(-ENOMEM); + + return vaddr; +} + +static void *cma_heap_vmap(struct dma_buf *dmabuf) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + void *vaddr; + + mutex_lock(&buffer->lock); + if (buffer->vmap_cnt) { + buffer->vmap_cnt++; + return buffer->vaddr; + } + + vaddr = cma_heap_do_vmap(buffer); + if (IS_ERR(vaddr)) + return vaddr; + + buffer->vaddr = vaddr; + buffer->vmap_cnt++; + mutex_unlock(&buffer->lock); + + return vaddr; +} + +static void cma_heap_vunmap(struct dma_buf *dmabuf, void *vaddr) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + + mutex_lock(&buffer->lock); + if (!--buffer->vmap_cnt) { + vunmap(buffer->vaddr); + buffer->vaddr = NULL; + } + mutex_unlock(&buffer->lock); +} + +static void cma_heap_dma_buf_release(struct dma_buf *dmabuf) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + struct cma_heap *cma_heap = buffer->heap; + + if (buffer->vmap_cnt > 0) { + WARN(1, "%s: buffer still mapped in the kernel\n", __func__); + vunmap(buffer->vaddr); + } + + cma_release(cma_heap->cma, buffer->cma_pages, buffer->pagecount); kfree(buffer); } -/* dmabuf heap CMA operations functions */ +const struct dma_buf_ops cma_heap_buf_ops = { + .attach = cma_heap_attach, + .detach = cma_heap_detatch, + .map_dma_buf = cma_heap_map_dma_buf, + .unmap_dma_buf = cma_heap_unmap_dma_buf, + .begin_cpu_access = cma_heap_dma_buf_begin_cpu_access, + .end_cpu_access = cma_heap_dma_buf_end_cpu_access, + .mmap = cma_heap_mmap, + .vmap = cma_heap_vmap, + .vunmap = cma_heap_vunmap, + .release = cma_heap_dma_buf_release, +}; + static int cma_heap_allocate(struct dma_heap *heap, - unsigned long len, - unsigned long fd_flags, - unsigned long heap_flags) + unsigned long len, + unsigned long fd_flags, + unsigned long heap_flags) { struct cma_heap *cma_heap = dma_heap_get_drvdata(heap); - struct heap_helper_buffer *helper_buffer; - struct page *cma_pages; + struct cma_heap_buffer *buffer; + DEFINE_DMA_BUF_EXPORT_INFO(exp_info); size_t size = PAGE_ALIGN(len); - unsigned long nr_pages = size >> PAGE_SHIFT; + pgoff_t pagecount = size >> PAGE_SHIFT; unsigned long align = get_order(size); + struct page *cma_pages; struct dma_buf *dmabuf; - int ret = -ENOMEM; pgoff_t pg; + int ret; - if (align > CONFIG_CMA_ALIGNMENT) - align = CONFIG_CMA_ALIGNMENT; - - helper_buffer = kzalloc(sizeof(*helper_buffer), GFP_KERNEL); - if (!helper_buffer) + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); + if (!buffer) return -ENOMEM; - init_heap_helper_buffer(helper_buffer, cma_heap_free); - helper_buffer->heap = heap; - helper_buffer->size = len; + INIT_LIST_HEAD(&buffer->attachments); + mutex_init(&buffer->lock); + buffer->len = size; - cma_pages = cma_alloc(cma_heap->cma, nr_pages, align, false); + if (align > CONFIG_CMA_ALIGNMENT) + align = CONFIG_CMA_ALIGNMENT; + + cma_pages = cma_alloc(cma_heap->cma, pagecount, align, false); if (!cma_pages) - goto free_buf; + goto free_buffer; + /* Clear the cma pages */ if (PageHighMem(cma_pages)) { - unsigned long nr_clear_pages = nr_pages; + unsigned long nr_clear_pages = pagecount; struct page *page = cma_pages; while (nr_clear_pages > 0) { @@ -85,7 +300,6 @@ static int cma_heap_allocate(struct dma_heap *heap, */ if (fatal_signal_pending(current)) goto free_cma; - page++; nr_clear_pages--; } @@ -93,28 +307,30 @@ static int cma_heap_allocate(struct dma_heap *heap, memset(page_address(cma_pages), 0, size); } - helper_buffer->pagecount = nr_pages; - helper_buffer->pages = kmalloc_array(helper_buffer->pagecount, - sizeof(*helper_buffer->pages), - GFP_KERNEL); - if (!helper_buffer->pages) { + buffer->pages = kmalloc_array(pagecount, sizeof(*buffer->pages), GFP_KERNEL); + if (!buffer->pages) { ret = -ENOMEM; goto free_cma; } - for (pg = 0; pg < helper_buffer->pagecount; pg++) - helper_buffer->pages[pg] = &cma_pages[pg]; + for (pg = 0; pg < pagecount; pg++) + buffer->pages[pg] = &cma_pages[pg]; + + buffer->cma_pages = cma_pages; + buffer->heap = cma_heap; + buffer->pagecount = pagecount; /* create the dmabuf */ - dmabuf = heap_helper_export_dmabuf(helper_buffer, fd_flags); + exp_info.ops = &cma_heap_buf_ops; + exp_info.size = buffer->len; + exp_info.flags = fd_flags; + exp_info.priv = buffer; + dmabuf = dma_buf_export(&exp_info); if (IS_ERR(dmabuf)) { ret = PTR_ERR(dmabuf); goto free_pages; } - helper_buffer->dmabuf = dmabuf; - helper_buffer->priv_virt = cma_pages; - ret = dma_buf_fd(dmabuf, fd_flags); if (ret < 0) { dma_buf_put(dmabuf); @@ -125,15 +341,16 @@ static int cma_heap_allocate(struct dma_heap *heap, return ret; free_pages: - kfree(helper_buffer->pages); + kfree(buffer->pages); free_cma: - cma_release(cma_heap->cma, cma_pages, nr_pages); -free_buf: - kfree(helper_buffer); + cma_release(cma_heap->cma, cma_pages, pagecount); +free_buffer: + kfree(buffer); + return ret; } -static const struct dma_heap_ops cma_heap_ops = { +static struct dma_heap_ops cma_heap_ops = { .allocate = cma_heap_allocate, }; @@ -175,3 +392,4 @@ static int add_default_cma_heap(void) module_init(add_default_cma_heap); MODULE_DESCRIPTION("DMA-BUF CMA Heap"); MODULE_LICENSE("GPL v2"); + From patchwork Sat Sep 26 04:24:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313566 Delivered-To: patches@linaro.org Received: by 2002:a92:5ad1:0:0:0:0:0 with SMTP id b78csp1680567ilg; Fri, 25 Sep 2020 21:25:02 -0700 (PDT) X-Received: by 2002:a17:90a:4d84:: with SMTP id m4mr666246pjh.59.1601094302354; Fri, 25 Sep 2020 21:25:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601094302; cv=none; d=google.com; s=arc-20160816; b=Uniq9969l3jO+djNCYuHUtcZHSWcJayyLci5FHjVjr54/tuhpOmbmywM7QsEmtsV80 dSgjPyLQSpuxIkEfCbg8bnLNuoeqUM3mXv8rqR3Q7W0g35T45svWNm7j3IC5+NUQ9UcS L7lHVqREdqlbej8NSWMIGQnWAQuJbvFJPAO9T86I/hwlUH46B8/s5Nxhhwtd28embsjX oH07Yseb1IGkFgOl6EvK9ACelvP/OB+Ag8oVzy1Ifl0OjqXlJN+uBR4ri1U5Q2O7U7UK hpVU/GkOAcli8Z+oi/UhJcw662BUpgfmrBZQpuAMTVVYRRUQEFUzvkl1ZfEkFQWd2z40 AAyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=06XEeGHHOGK/ICro7P5tlQH3HvkQIwMvcU78rykFbzY=; b=nXUQHolGqk2TqjJS0nQMv+fR0+RJqrgZ8sYo7Sd91X2T0vM4uTR4fiLOwkGjektAm4 Qp3sfoqThHpne7z3uspGecb1ugis+7qsSWEgpXlAW9D0gfzEPwURmTTLb8IYuYPtmYGE BVHQCUbaYDO7Xtm3tLwbx6Imm7U/5GNUTaYSqaaykd3tDeqSytNIxD0lnxTK0tmSCi96 8uieP8c8H13ibryJnyEjJo5iQmDanz9L9K6iDIUx265d9GE5CdIM9FOeszO926Nr63tc qipfM42oGqhbE9je+/fTaChNSJGFtpeBuUFyv/qQv5UA1+bNZZO2jFEdGdrMvesbEDp1 M25Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=zvgDh4o5; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id bo15sor427176pjb.24.2020.09.25.21.25.02 for (Google Transport Security); Fri, 25 Sep 2020 21:25:02 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=zvgDh4o5; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=06XEeGHHOGK/ICro7P5tlQH3HvkQIwMvcU78rykFbzY=; b=zvgDh4o5Ge79q49paWYi3m66nq39k6wE7QrzuzFKjgrLCt8H6LTyl4Xw3VTyN+n9lY rky20HuPWlBlj+KWAX4O2GSLc8Q5OdlMwGIpopsrMZZVxaeOBONzNdNKIXPGsPKDWlwW 5xWD+b3YsxF3/Ld8pR2QpunLtPqhxmZAbGkAepYz9x2gwNkc2yNwvU6723eJIZ6E8fLA hPV1S9ltYjCPplxqTslDhPUnjJOt02AAgpJvKtwqlYNuwnb8jFmPfHvU7RycoOAt/JWQ dC9V7Mcf9aJ6DwzrM1Hbtajy0ROoW+thxsif1+wjUq+KfX54sUSUBQoh/WCqDfF0bkpE 73aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=06XEeGHHOGK/ICro7P5tlQH3HvkQIwMvcU78rykFbzY=; b=ZYt7aj8of6D+xlA2jqBKqbA7qaOmSjjROV/Llrl92+U8Ex4jQDJ1hZ53DdQETSF7i+ usXCWBMbpxX3rGG2oAbnWpfQSjEw7ENw3Ot3KMleyIoRxXogmYtNpCvcx5xCOhS3VvKF //apJvURewNesMWOivqkZMGqvzveYYftaPMUkg1PQtj7kDsf4ADTy32dGry9Z/AgwAFX 2n2T04W1cmMVpaiHSOf+HqDQL62HXQMYf41942TWXFMVHxeM2lTYmiRy6Cy/lXMwWo3K I1L/lTg4qZqKfyi+g2OrHe7ab7PU0y11wdHnfwntV4a0SEn9h454ONWdyKpspQlBe3F9 gyVg== X-Gm-Message-State: AOAM530ncxebZZU+RTGkT99JRP+rx6Y7diaVTwUxn5uaK/nm4A6iw8ck AQdTS25Fzmgi8dh1etXmWngk3/fu X-Google-Smtp-Source: ABdhPJwGMjuNMK0u01WspM91aYEUZeR21jjDkWdw78qV4eHRAn/Uu873Er9oNPryyHIHXILdP7eGZQ== X-Received: by 2002:a17:90a:aa94:: with SMTP id l20mr668572pjq.95.1601094301975; Fri, 25 Sep 2020 21:25:01 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id a5sm3585886pgk.13.2020.09.25.21.25.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Sep 2020 21:25:01 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [RFC][PATCH 3/6] dma-buf: heaps: Remove heap-helpers code Date: Sat, 26 Sep 2020 04:24:50 +0000 Message-Id: <20200926042453.67517-4-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200926042453.67517-1-john.stultz@linaro.org> References: <20200926042453.67517-1-john.stultz@linaro.org> MIME-Version: 1.0 The heap-helpers code was not as generic as initially hoped and it is now not being used, so remove it from the tree. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/Makefile | 1 - drivers/dma-buf/heaps/heap-helpers.c | 271 --------------------------- drivers/dma-buf/heaps/heap-helpers.h | 53 ------ 3 files changed, 325 deletions(-) delete mode 100644 drivers/dma-buf/heaps/heap-helpers.c delete mode 100644 drivers/dma-buf/heaps/heap-helpers.h -- 2.17.1 diff --git a/drivers/dma-buf/heaps/Makefile b/drivers/dma-buf/heaps/Makefile index 6e54cdec3da0..974467791032 100644 --- a/drivers/dma-buf/heaps/Makefile +++ b/drivers/dma-buf/heaps/Makefile @@ -1,4 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += heap-helpers.o obj-$(CONFIG_DMABUF_HEAPS_SYSTEM) += system_heap.o obj-$(CONFIG_DMABUF_HEAPS_CMA) += cma_heap.o diff --git a/drivers/dma-buf/heaps/heap-helpers.c b/drivers/dma-buf/heaps/heap-helpers.c deleted file mode 100644 index 9f964ca3f59c..000000000000 --- a/drivers/dma-buf/heaps/heap-helpers.c +++ /dev/null @@ -1,271 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "heap-helpers.h" - -void init_heap_helper_buffer(struct heap_helper_buffer *buffer, - void (*free)(struct heap_helper_buffer *)) -{ - buffer->priv_virt = NULL; - mutex_init(&buffer->lock); - buffer->vmap_cnt = 0; - buffer->vaddr = NULL; - buffer->pagecount = 0; - buffer->pages = NULL; - INIT_LIST_HEAD(&buffer->attachments); - buffer->free = free; -} - -struct dma_buf *heap_helper_export_dmabuf(struct heap_helper_buffer *buffer, - int fd_flags) -{ - DEFINE_DMA_BUF_EXPORT_INFO(exp_info); - - exp_info.ops = &heap_helper_ops; - exp_info.size = buffer->size; - exp_info.flags = fd_flags; - exp_info.priv = buffer; - - return dma_buf_export(&exp_info); -} - -static void *dma_heap_map_kernel(struct heap_helper_buffer *buffer) -{ - void *vaddr; - - vaddr = vmap(buffer->pages, buffer->pagecount, VM_MAP, PAGE_KERNEL); - if (!vaddr) - return ERR_PTR(-ENOMEM); - - return vaddr; -} - -static void dma_heap_buffer_destroy(struct heap_helper_buffer *buffer) -{ - if (buffer->vmap_cnt > 0) { - WARN(1, "%s: buffer still mapped in the kernel\n", __func__); - vunmap(buffer->vaddr); - } - - buffer->free(buffer); -} - -static void *dma_heap_buffer_vmap_get(struct heap_helper_buffer *buffer) -{ - void *vaddr; - - if (buffer->vmap_cnt) { - buffer->vmap_cnt++; - return buffer->vaddr; - } - vaddr = dma_heap_map_kernel(buffer); - if (IS_ERR(vaddr)) - return vaddr; - buffer->vaddr = vaddr; - buffer->vmap_cnt++; - return vaddr; -} - -static void dma_heap_buffer_vmap_put(struct heap_helper_buffer *buffer) -{ - if (!--buffer->vmap_cnt) { - vunmap(buffer->vaddr); - buffer->vaddr = NULL; - } -} - -struct dma_heaps_attachment { - struct device *dev; - struct sg_table table; - struct list_head list; -}; - -static int dma_heap_attach(struct dma_buf *dmabuf, - struct dma_buf_attachment *attachment) -{ - struct dma_heaps_attachment *a; - struct heap_helper_buffer *buffer = dmabuf->priv; - int ret; - - a = kzalloc(sizeof(*a), GFP_KERNEL); - if (!a) - return -ENOMEM; - - ret = sg_alloc_table_from_pages(&a->table, buffer->pages, - buffer->pagecount, 0, - buffer->pagecount << PAGE_SHIFT, - GFP_KERNEL); - if (ret) { - kfree(a); - return ret; - } - - a->dev = attachment->dev; - INIT_LIST_HEAD(&a->list); - - attachment->priv = a; - - mutex_lock(&buffer->lock); - list_add(&a->list, &buffer->attachments); - mutex_unlock(&buffer->lock); - - return 0; -} - -static void dma_heap_detach(struct dma_buf *dmabuf, - struct dma_buf_attachment *attachment) -{ - struct dma_heaps_attachment *a = attachment->priv; - struct heap_helper_buffer *buffer = dmabuf->priv; - - mutex_lock(&buffer->lock); - list_del(&a->list); - mutex_unlock(&buffer->lock); - - sg_free_table(&a->table); - kfree(a); -} - -static -struct sg_table *dma_heap_map_dma_buf(struct dma_buf_attachment *attachment, - enum dma_data_direction direction) -{ - struct dma_heaps_attachment *a = attachment->priv; - struct sg_table *table; - - table = &a->table; - - if (!dma_map_sg(attachment->dev, table->sgl, table->nents, - direction)) - table = ERR_PTR(-ENOMEM); - return table; -} - -static void dma_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, - struct sg_table *table, - enum dma_data_direction direction) -{ - dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); -} - -static vm_fault_t dma_heap_vm_fault(struct vm_fault *vmf) -{ - struct vm_area_struct *vma = vmf->vma; - struct heap_helper_buffer *buffer = vma->vm_private_data; - - if (vmf->pgoff > buffer->pagecount) - return VM_FAULT_SIGBUS; - - vmf->page = buffer->pages[vmf->pgoff]; - get_page(vmf->page); - - return 0; -} - -static const struct vm_operations_struct dma_heap_vm_ops = { - .fault = dma_heap_vm_fault, -}; - -static int dma_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - - if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0) - return -EINVAL; - - vma->vm_ops = &dma_heap_vm_ops; - vma->vm_private_data = buffer; - - return 0; -} - -static void dma_heap_dma_buf_release(struct dma_buf *dmabuf) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - - dma_heap_buffer_destroy(buffer); -} - -static int dma_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, - enum dma_data_direction direction) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - struct dma_heaps_attachment *a; - int ret = 0; - - mutex_lock(&buffer->lock); - - if (buffer->vmap_cnt) - invalidate_kernel_vmap_range(buffer->vaddr, buffer->size); - - list_for_each_entry(a, &buffer->attachments, list) { - dma_sync_sg_for_cpu(a->dev, a->table.sgl, a->table.nents, - direction); - } - mutex_unlock(&buffer->lock); - - return ret; -} - -static int dma_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, - enum dma_data_direction direction) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - struct dma_heaps_attachment *a; - - mutex_lock(&buffer->lock); - - if (buffer->vmap_cnt) - flush_kernel_vmap_range(buffer->vaddr, buffer->size); - - list_for_each_entry(a, &buffer->attachments, list) { - dma_sync_sg_for_device(a->dev, a->table.sgl, a->table.nents, - direction); - } - mutex_unlock(&buffer->lock); - - return 0; -} - -static void *dma_heap_dma_buf_vmap(struct dma_buf *dmabuf) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - void *vaddr; - - mutex_lock(&buffer->lock); - vaddr = dma_heap_buffer_vmap_get(buffer); - mutex_unlock(&buffer->lock); - - return vaddr; -} - -static void dma_heap_dma_buf_vunmap(struct dma_buf *dmabuf, void *vaddr) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - - mutex_lock(&buffer->lock); - dma_heap_buffer_vmap_put(buffer); - mutex_unlock(&buffer->lock); -} - -const struct dma_buf_ops heap_helper_ops = { - .map_dma_buf = dma_heap_map_dma_buf, - .unmap_dma_buf = dma_heap_unmap_dma_buf, - .mmap = dma_heap_mmap, - .release = dma_heap_dma_buf_release, - .attach = dma_heap_attach, - .detach = dma_heap_detach, - .begin_cpu_access = dma_heap_dma_buf_begin_cpu_access, - .end_cpu_access = dma_heap_dma_buf_end_cpu_access, - .vmap = dma_heap_dma_buf_vmap, - .vunmap = dma_heap_dma_buf_vunmap, -}; diff --git a/drivers/dma-buf/heaps/heap-helpers.h b/drivers/dma-buf/heaps/heap-helpers.h deleted file mode 100644 index 805d2df88024..000000000000 --- a/drivers/dma-buf/heaps/heap-helpers.h +++ /dev/null @@ -1,53 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * DMABUF Heaps helper code - * - * Copyright (C) 2011 Google, Inc. - * Copyright (C) 2019 Linaro Ltd. - */ - -#ifndef _HEAP_HELPERS_H -#define _HEAP_HELPERS_H - -#include -#include - -/** - * struct heap_helper_buffer - helper buffer metadata - * @heap: back pointer to the heap the buffer came from - * @dmabuf: backing dma-buf for this buffer - * @size: size of the buffer - * @priv_virt pointer to heap specific private value - * @lock mutext to protect the data in this structure - * @vmap_cnt count of vmap references on the buffer - * @vaddr vmap'ed virtual address - * @pagecount number of pages in the buffer - * @pages list of page pointers - * @attachments list of device attachments - * - * @free heap callback to free the buffer - */ -struct heap_helper_buffer { - struct dma_heap *heap; - struct dma_buf *dmabuf; - size_t size; - - void *priv_virt; - struct mutex lock; - int vmap_cnt; - void *vaddr; - pgoff_t pagecount; - struct page **pages; - struct list_head attachments; - - void (*free)(struct heap_helper_buffer *buffer); -}; - -void init_heap_helper_buffer(struct heap_helper_buffer *buffer, - void (*free)(struct heap_helper_buffer *)); - -struct dma_buf *heap_helper_export_dmabuf(struct heap_helper_buffer *buffer, - int fd_flags); - -extern const struct dma_buf_ops heap_helper_ops; -#endif /* _HEAP_HELPERS_H */ From patchwork Sat Sep 26 04:24:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313567 Delivered-To: patches@linaro.org Received: by 2002:a92:5ad1:0:0:0:0:0 with SMTP id b78csp1680579ilg; Fri, 25 Sep 2020 21:25:03 -0700 (PDT) X-Received: by 2002:a62:6044:0:b029:151:1a04:895 with SMTP id u65-20020a6260440000b02901511a040895mr2225860pfb.34.1601094303779; Fri, 25 Sep 2020 21:25:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601094303; cv=none; d=google.com; s=arc-20160816; b=NLShCTM6AZ1O9Y9Ov3AWPnM8BtrUGA8qc0+vd818xGTX72KWlz8LXqzahLbL7oHALm tigI7jYA6JsNqgWC4ExqCQVQvzopAKYDwKUr19ZNvj2hut9WmDTCnf7j2qc11Tfo/kzB PyukXTDa2YePdCmcMiRDHyhVZBvbTtmYLe96EJXE9Di6hpcJE2Us57ygIUXNzsmQ6maB NqvCNJ5EEB2V8FBaIXlp4my52cn2CbXSgTiE+W0tx18VUKN0k9cRuw2nq8Smt4XDqoLD j3ZSv4ybANq/jOpIEAn4w6NJoVsPS+kdj5us7w0CFt92qNAm3/9Ic38ax3fknqt8Fihb E8QQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=2H1Yr1ZzRbTr0UE566f697qNBNIq6GZE63QK31mOArc=; b=ppvIEAO0nCqpFw4GfiIYCiZsCtQprYP8OkkpflB68a+OULoGrBJDin2Zb30Y//XpkI LMHh/14hTpJNoOM5e+MUlhAe2ok+Z0D3NXNhFcytETfEs+LbkWheRmOWPo6EcOL8lNmT ExMS3BXF1pKUpAZIAbiI2jUYau3qvpjxo1EsDM5YV3EiwI3tkmtxqQRZ02P2dLj8i664 qhBVMFQk6Bh35L7TVKjlU1HG5hHJs5zwyoI4Fhzy3v9XN48wS/Y9iNvFCJYxvNcCPeJF zZxONWo4EJTfGdPQvQ9cS3SJZX64qSrliTKwN2X3mEG3PWWa+B8jU1rO7RD0PE0lUU5M zWqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QR3D70X+; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 135sor2090896pfc.9.2020.09.25.21.25.03 for (Google Transport Security); Fri, 25 Sep 2020 21:25:03 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QR3D70X+; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2H1Yr1ZzRbTr0UE566f697qNBNIq6GZE63QK31mOArc=; b=QR3D70X+iURkNzPXl26udZXdU8x9honZnaMWw4JV2DKKAIvIFtkWhDmU8HykuxoRUj kQOi8jHCzczouvlfYRVDinDjm85nLQlg31by6GieWnYcJNKmVdwEjC7pR1DQH3aw0W6f r0QLTxE4f+JBnu+5C2o5u7ggaWg0rg3bF64Zc79ECONonPRowJwtIm96g3MFqkLw0SpX iG1F+HCHirx0wUURVjmDyERxSdZkEp9LjMNm8XM/f2fylGuUAUxdI/nnySnxYcwg4P/V eysFTGffdcMyz8llW4fqtlxdEldrvDb17KOJ37R3KCOs1Vz3NyLCW4T0XWt1rJKKtsuC kYxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2H1Yr1ZzRbTr0UE566f697qNBNIq6GZE63QK31mOArc=; b=UGhp1mEm4B6cKw1IEzUJyHQ+j1fU8cto5MRkIOfIBI/hyzfx8CWrnO5JvH2Am+tmLR y27l1VaE8fqrVtoqxQc6FS2gNEedcec5n25q4ECiAisK751N4BZ6Pv3PVY9cXh4atrnc 8zEHy9etD3eT3WkYl/mKHGhV61aWhxwHvV6mPO+3J0UunFL3trdtj6+FrcCJVhXcQ8bM ov7CwXfMoTseyfLVdKIY+p5MKjytA8navhWU067nNS88jOL66OLgLYe00GJRXv8X7Btp YWTYF5evHR18dIz+jyfXl4Phy3pnQ/m8dMmqVzWVMk9eHELoaku0jlBTK1PNi2Bxk/Ok 0Wng== X-Gm-Message-State: AOAM532woucF8fDgIgDYXYNDAmAxgE4RAqIXYK3UW/fnMCVtAPsgFksI o4yFk2V245StnYp/tkN8auo346Z1 X-Google-Smtp-Source: ABdhPJxVSgNpa8Val9ZcacJd85QmQO2OmijAvP06AHqBP8I1x0fgkNGF6yDx7vorPACjyRFjjzINHA== X-Received: by 2002:a62:fb1a:0:b029:142:2501:39f9 with SMTP id x26-20020a62fb1a0000b0290142250139f9mr1542805pfm.72.1601094303392; Fri, 25 Sep 2020 21:25:03 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id a5sm3585886pgk.13.2020.09.25.21.25.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Sep 2020 21:25:02 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [RFC][PATCH 4/6] dma-buf: system_heap: Allocate higher order pages if available Date: Sat, 26 Sep 2020 04:24:51 +0000 Message-Id: <20200926042453.67517-5-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200926042453.67517-1-john.stultz@linaro.org> References: <20200926042453.67517-1-john.stultz@linaro.org> MIME-Version: 1.0 While the system heap can return non-contiguous pages, try to allocate larger order pages if possible. This will allow slight performance gains and make implementing page pooling easier. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/system_heap.c | 85 ++++++++++++++++++++++------- 1 file changed, 66 insertions(+), 19 deletions(-) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index ddfa17dc48a8..882a632e9bb7 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -39,6 +39,14 @@ struct dma_heap_attachment { struct list_head list; }; +#define HIGH_ORDER_GFP (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \ + | __GFP_NORETRY) & ~__GFP_RECLAIM) \ + | __GFP_COMP) +#define LOW_ORDER_GFP (GFP_HIGHUSER | __GFP_ZERO | __GFP_COMP) +static gfp_t order_flags[] = {HIGH_ORDER_GFP, LOW_ORDER_GFP, LOW_ORDER_GFP}; +static const unsigned int orders[] = {8, 4, 0}; +#define NUM_ORDERS ARRAY_SIZE(orders) + static struct sg_table *dup_sg_table(struct sg_table *table) { struct sg_table *new_table; @@ -259,8 +267,11 @@ static void system_heap_dma_buf_release(struct dma_buf *dmabuf) int i; table = &buffer->sg_table; - for_each_sgtable_sg(table, sg, i) - __free_page(sg_page(sg)); + for_each_sg(table->sgl, sg, table->nents, i) { + struct page *page = sg_page(sg); + + __free_pages(page, compound_order(page)); + } sg_free_table(table); kfree(buffer); } @@ -278,6 +289,26 @@ const struct dma_buf_ops system_heap_buf_ops = { .release = system_heap_dma_buf_release, }; +static struct page *alloc_largest_available(unsigned long size, + unsigned int max_order) +{ + struct page *page; + int i; + + for (i = 0; i < NUM_ORDERS; i++) { + if (size < (PAGE_SIZE << orders[i])) + continue; + if (max_order < orders[i]) + continue; + + page = alloc_pages(order_flags[i], orders[i]); + if (!page) + continue; + return page; + } + return NULL; +} + static int system_heap_allocate(struct dma_heap *heap, unsigned long len, unsigned long fd_flags, @@ -285,11 +316,13 @@ static int system_heap_allocate(struct dma_heap *heap, { struct system_heap_buffer *buffer; DEFINE_DMA_BUF_EXPORT_INFO(exp_info); + unsigned long size_remaining = len; + unsigned int max_order = orders[0]; struct dma_buf *dmabuf; struct sg_table *table; struct scatterlist *sg; - pgoff_t pagecount; - pgoff_t pg; + struct list_head pages; + struct page *page, *tmp_page; int i, ret = -ENOMEM; buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); @@ -301,25 +334,35 @@ static int system_heap_allocate(struct dma_heap *heap, buffer->heap = heap; buffer->len = len; - table = &buffer->sg_table; - pagecount = len / PAGE_SIZE; - if (sg_alloc_table(table, pagecount, GFP_KERNEL)) - goto free_buffer; - - sg = table->sgl; - for (pg = 0; pg < pagecount; pg++) { - struct page *page; + INIT_LIST_HEAD(&pages); + i = 0; + while (size_remaining > 0) { /* * Avoid trying to allocate memory if the process * has been killed by SIGKILL */ if (fatal_signal_pending(current)) - goto free_pages; - page = alloc_page(GFP_KERNEL | __GFP_ZERO); + goto free_buffer; + + page = alloc_largest_available(size_remaining, max_order); if (!page) - goto free_pages; - sg_set_page(sg, page, page_size(page), 0); + goto free_buffer; + + list_add_tail(&page->lru, &pages); + size_remaining -= PAGE_SIZE << compound_order(page); + max_order = compound_order(page); + i++; + } + + table = &buffer->sg_table; + if (sg_alloc_table(table, i, GFP_KERNEL)) + goto free_buffer; + + sg = table->sgl; + list_for_each_entry_safe(page, tmp_page, &pages, lru) { + sg_set_page(sg, page, PAGE_SIZE << compound_order(page), 0); sg = sg_next(sg); + list_del(&page->lru); } /* create the dmabuf */ @@ -339,14 +382,18 @@ static int system_heap_allocate(struct dma_heap *heap, /* just return, as put will call release and that will free */ return ret; } - return ret; free_pages: - for_each_sgtable_sg(table, sg, i) - __free_page(sg_page(sg)); + for_each_sgtable_sg(table, sg, i) { + struct page *p = sg_page(sg); + + __free_pages(p, compound_order(p)); + } sg_free_table(table); free_buffer: + list_for_each_entry_safe(page, tmp_page, &pages, lru) + __free_pages(page, compound_order(page)); kfree(buffer); return ret; From patchwork Sat Sep 26 04:24:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313568 Delivered-To: patches@linaro.org Received: by 2002:a92:5ad1:0:0:0:0:0 with SMTP id b78csp1680592ilg; Fri, 25 Sep 2020 21:25:05 -0700 (PDT) X-Received: by 2002:a17:90b:793:: with SMTP id l19mr704786pjz.154.1601094305085; Fri, 25 Sep 2020 21:25:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601094305; cv=none; d=google.com; s=arc-20160816; b=K2DDk9izp+UCqNKf/Z8Wzrg/DdFmXVp63rjf0vB3BnjwRD6K0zORRStNrKYYhn78pZ FOta0gCWW/yOHD334roispPWB010eNpUVcT1MiomfPcfPyKiiNXXLuwXmTq1lP5wWrs3 4oBKU6dD5iKmPv0hzeIYKL7ceIHqkrr4g2gZTFDXkIcJOfFfQp/ZkX17uuPJZ+J7N4nz ehhD93mVpP9Des58O7eLBpgltgN/CM00e0WSLOjE9FLauT41wavgDy8PSiqSdQdORaTk osi342LJrKLsnFu3ohHKWTyl7C4BHGVUXE6K8i0pm52/r/VLqn0R6ZEHs9Kj8eU1FHs0 GfXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=/z1sF9p3q5wnfzY3ZTbxrN10Skwo2Y6Rq8ZU166mru4=; b=I85RunYXs3d2uvRq+ra7QAHw8Zf21wVQdmMY3nXoMzg5jJMHhJnX7zRYXJgQ3D6C0o oQOfY4AYMepzpVllhcebBi3YgPunAd7/iMMIWxPSoxH6vIZ3Rn+cIO0RY1T7YTJAaKHC RuawSx07a+9sQuZdXgoArC0ASOy447cmGDQpo4XNzoBw45ZWROx2hK9LbSfaT+Ynmsw9 3zuez7azw15yMO1gYODWCBZUgcyM7nHg44Auk+FHJFwiXGSxlmEa8p1bM3aNLCIPJvlY tExbfyi+qaTU66MrVUjjo86bqAqZ33cUrjgP855a4aJ9OU7TfxL4vR0trWXpr9ufiamW GB+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WSXp6wb5; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c4sor2010252pgm.40.2020.09.25.21.25.04 for (Google Transport Security); Fri, 25 Sep 2020 21:25:05 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WSXp6wb5; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/z1sF9p3q5wnfzY3ZTbxrN10Skwo2Y6Rq8ZU166mru4=; b=WSXp6wb5oEQXway9MAE25SNgkcoGB3A0OZEvTUWHbhaQcCbSV5Ny+E7VBwg8KmOOcq yf+GkI6TgmXX+FInkLEgtXDSfKYP0BgE4pNpZqhGEVrBISYNSgmypbgffcboNFj9ieaA FIYzHMmYEOWJ8Biy3uom7ZyaqCxD04BMyJpW+4kle1dvosbg3dumOY966Amx0qBSu95P A40DSecXHKYkGNqCZ7pz9lmoijccn18mYFX85cE+7mNzeUxoRhgHySufYs1zncvsY/nA wWlPAlUNiGL1Rg5aiqJ4bQPEzuSXFeIMxIrcJEacxBI2b7/pyGyToijcWcIl7O9/Jl8K VrpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/z1sF9p3q5wnfzY3ZTbxrN10Skwo2Y6Rq8ZU166mru4=; b=r7M4d1YD+mZDKO3s8nWbm92ZML9rd5s1XUKqwA58urxr1V8lYVCDNR+NloC0oeO+gu wVsQbPRrwjXwD4RzABDT5yRT9uL1HdNYHNPEh7cFKnVXPbnuPG0DaWGkmmlhbsW+LzYA M3GyqvZr9kMWLOh18jisR8mOWHXhWxSXi1HIgBF8L4ctrN1IkqfaLxXlA8FofsdQYpTT z9PDp5WGlGBmnAJCxITFRNNO57DzyymMHmpDMT9h7bQ17BFnZORMYV6pWeKD2uC4N1K6 l/qVZKuHHgK3TnlLqc8311Aqxe7KwyJ4ae3AtAXzl0XxBVuyiRKSbjFXWZoIcQMje2Y+ lpKw== X-Gm-Message-State: AOAM531Xc24z0hk6A8QSGZVdss8t7KunY1tdf9pYGiVQMAMV9mLQneVM 1CkiaNGHXTIgneX62JG20zVi1sk11yN4v8KT X-Google-Smtp-Source: ABdhPJxJdPim2jtqq6MvkC+z6P836ZUm0PKEAEj8ymE6++fwkRtRu6msPB2Sd2NBFczvSf2ry5Vv2g== X-Received: by 2002:a63:5401:: with SMTP id i1mr1540855pgb.398.1601094304699; Fri, 25 Sep 2020 21:25:04 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id a5sm3585886pgk.13.2020.09.25.21.25.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Sep 2020 21:25:04 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [RFC][PATCH 5/6] dma-buf: system_heap: Add pagepool support to system heap Date: Sat, 26 Sep 2020 04:24:52 +0000 Message-Id: <20200926042453.67517-6-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200926042453.67517-1-john.stultz@linaro.org> References: <20200926042453.67517-1-john.stultz@linaro.org> MIME-Version: 1.0 Reuse/abuse the pagepool code from the network code to speed up allocation performance. This is similar to the ION pagepool usage, but tries to utilize generic code instead of a custom implementation. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/Kconfig | 1 + drivers/dma-buf/heaps/system_heap.c | 32 +++++++++++++++++++++++++---- 2 files changed, 29 insertions(+), 4 deletions(-) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/Kconfig b/drivers/dma-buf/heaps/Kconfig index a5eef06c4226..f13cde4321b1 100644 --- a/drivers/dma-buf/heaps/Kconfig +++ b/drivers/dma-buf/heaps/Kconfig @@ -1,6 +1,7 @@ config DMABUF_HEAPS_SYSTEM bool "DMA-BUF System Heap" depends on DMABUF_HEAPS + select PAGE_POOL help Choose this option to enable the system dmabuf heap. The system heap is backed by pages from the buddy allocator. If in doubt, say Y. diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index 882a632e9bb7..9f57b4c8ae69 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -20,6 +20,7 @@ #include #include #include +#include struct dma_heap *sys_heap; @@ -46,6 +47,7 @@ struct dma_heap_attachment { static gfp_t order_flags[] = {HIGH_ORDER_GFP, LOW_ORDER_GFP, LOW_ORDER_GFP}; static const unsigned int orders[] = {8, 4, 0}; #define NUM_ORDERS ARRAY_SIZE(orders) +struct page_pool *pools[NUM_ORDERS]; static struct sg_table *dup_sg_table(struct sg_table *table) { @@ -264,13 +266,17 @@ static void system_heap_dma_buf_release(struct dma_buf *dmabuf) struct system_heap_buffer *buffer = dmabuf->priv; struct sg_table *table; struct scatterlist *sg; - int i; + int i, j; table = &buffer->sg_table; for_each_sg(table->sgl, sg, table->nents, i) { struct page *page = sg_page(sg); - __free_pages(page, compound_order(page)); + for (j = 0; j < NUM_ORDERS; j++) { + if (compound_order(page) == orders[j]) + break; + } + page_pool_put_full_page(pools[j], page, false); } sg_free_table(table); kfree(buffer); @@ -300,8 +306,7 @@ static struct page *alloc_largest_available(unsigned long size, continue; if (max_order < orders[i]) continue; - - page = alloc_pages(order_flags[i], orders[i]); + page = page_pool_alloc_pages(pools[i], order_flags[i]); if (!page) continue; return page; @@ -406,6 +411,25 @@ static const struct dma_heap_ops system_heap_ops = { static int system_heap_create(void) { struct dma_heap_export_info exp_info; + int i; + + for (i = 0; i < NUM_ORDERS; i++) { + struct page_pool_params pp; + + memset(&pp, 0, sizeof(pp)); + pp.order = orders[i]; + pp.dma_dir = DMA_BIDIRECTIONAL; + pools[i] = page_pool_create(&pp); + + if (IS_ERR(pools[i])) { + int j; + + pr_err("%s: page pool creation failed!\n", __func__); + for (j = 0; j < i; j++) + page_pool_destroy(pools[j]); + return PTR_ERR(pools[i]); + } + } exp_info.name = "system"; exp_info.ops = &system_heap_ops; From patchwork Sat Sep 26 04:24:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313569 Delivered-To: patches@linaro.org Received: by 2002:a92:5ad1:0:0:0:0:0 with SMTP id b78csp1680595ilg; Fri, 25 Sep 2020 21:25:06 -0700 (PDT) X-Received: by 2002:a62:154b:0:b029:150:c091:7146 with SMTP id 72-20020a62154b0000b0290150c0917146mr2171969pfv.80.1601094306618; Fri, 25 Sep 2020 21:25:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601094306; cv=none; d=google.com; s=arc-20160816; b=x0BKaLlADoQZ5ssHjXNS0gIwNYEmPtRFzt9eQNYs4oC7yfxd/STATvUcz5OtTFeEg8 fBrjTmTkbkHGffEWSfCao0Ve7JyFhkHLw1PABEMkqMh12h2dVfLFu0Hcpz5Jo4tVZLqH SnjPZrYg7Wso2QCkkUuXRol6XnfDpx5Q+3tDlHu3z/3qYPHOHEAxwrZaErDIdEJgr4Eo PvCVjY/mIbRpyW/mz1mv1E2FbepwseXgjA+oue0kYAo27f/w1PfEoz/aqVBpx5S76/If n8cshdBCOMvRp/wpQ85YJJ8RlXByO9qYRFw9UwKOGJKLio3GHyMinVPhi4U4T2y11tBl Y3Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=vw5JB3RRsbohaer3Y56R1IPsXZfXmp2s3xn/82rJ0CY=; b=gXqsiwRs0LRdfNo6PkCYYw8WiiVJ40AcTGPa1dKv4UXLMUrkoeQiUE9S64908jXBtx 6R78wE62MoG+ZgcsqZ/JORbrHS4OVRZpu9nhmW6U3utsEqROQ3JUcAgwhvXPFGLJJil+ vOvaC7aWztNmmPBUxXrbOuTeat8Ryw5sarfGgFh/Am8DOW+VkKmqSAujawpTMWXv7ofN muRRCSeJ4XD8NBuqT75jI/xUz9DCCOrRGpleU0nD08AchweDg45UmYnlr2Pit/Ppb8kz 1jqMOwSzV9XxS/IaZzdfiPj35JPdSk0LHw7O1TBN973LYNi9uBB+tm5h4D+Pa32ZO7/n RyEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D1xoiKND; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id s192sor2168599pfs.100.2020.09.25.21.25.06 for (Google Transport Security); Fri, 25 Sep 2020 21:25:06 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D1xoiKND; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vw5JB3RRsbohaer3Y56R1IPsXZfXmp2s3xn/82rJ0CY=; b=D1xoiKNDyt3hh17/hK6uqP4SAJ0L1P3/8IXEZSgrQKf9rBZXqkaliK02LneXiMkHGc hxYhfBZgNzpCVtoUOBPYp/QXx5X4/02K3L9IlZyEnsTLPVYvwP/uqLK4KsdUC+CbktyT 8DNFlax8CYkd9Trrqr9xoxsbV9QIMOilaXcPWav2ATUes6LDMvL85b+AkLVf6tvVHYom +Id89kS5caa8WGXDDtF5fDnl4RarlgFQyvitdpowCBq68FauqBrCZSuZArik8Bp9lmvK Fh0WjZM6QEhINjtqmgHLGVqCcIl0sx1uE4RQxjHaCKgOy70fCPRfjDyzqBwVCDnyFq3g I5WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vw5JB3RRsbohaer3Y56R1IPsXZfXmp2s3xn/82rJ0CY=; b=c95+V5fHqjO6chVPLlxQAEzNM/i9YiDjjWDC43S7hba+PSAx5cSUztct1CIEzRQkCB 2ktLgDORN/woOUyi6LkN/pEkt2a48yCURDDqrEIolWqtfG59pKZ84kgh6lp6JnSwZ7D8 xNqND5bpl8/LSe8rxhc68cjQ8i1FrVTH4ncM9miHmSeis+3WtHUXlb0EkSjrrjWpaSYc 8xyGA7kRP6M7X4u1M46Ku5OjQxFjfHFTiYlZC/3+FhWzckW2LhEr+DmaS9jjwk8LLy9z lbVRBr2tHSrdBs3IOcMlErfTw8/3ckZowq9gxFv9RgYKVmPCyGPViAYRpzTujGS1Vgmu 6M6Q== X-Gm-Message-State: AOAM531lVYgqK9mE28maXs378ppZqjhbQDrMSSw1Qu3cwJusoIG32MxF KAChbG2v9vvQILsExHO43rKSly/X X-Google-Smtp-Source: ABdhPJyvQ50xfxJX8TpnL1MvlzaJmF3SWzncH/tqsov+stEHycC5jsIv3tQvdZov8P+AXKPfVqQiWA== X-Received: by 2002:a62:cd49:0:b029:150:7742:c6c8 with SMTP id o70-20020a62cd490000b02901507742c6c8mr2126731pfg.61.1601094306220; Fri, 25 Sep 2020 21:25:06 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id a5sm3585886pgk.13.2020.09.25.21.25.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Sep 2020 21:25:05 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [RFC][PATCH 6/6] dma-buf: heaps: Skip sync if not mapped Date: Sat, 26 Sep 2020 04:24:53 +0000 Message-Id: <20200926042453.67517-7-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200926042453.67517-1-john.stultz@linaro.org> References: <20200926042453.67517-1-john.stultz@linaro.org> MIME-Version: 1.0 This patch is basically a port of Ørjan Eide's similar patch for ION https://lore.kernel.org/lkml/20200414134629.54567-1-orjan.eide@arm.com/ Only sync the sg-list of dma-buf heap attachment when the attachment is actually mapped on the device. dma-bufs may be synced at any time. It can be reached from user space via DMA_BUF_IOCTL_SYNC, so there are no guarantees from callers on when syncs may be attempted, and dma_buf_end_cpu_access() and dma_buf_begin_cpu_access() may not be paired. Since the sg_list's dma_address isn't set up until the buffer is used on the device, and dma_map_sg() is called on it, the dma_address will be NULL if sync is attempted on the dma-buf before it's mapped on a device. Before v5.0 (commit 55897af63091 ("dma-direct: merge swiotlb_dma_ops into the dma_direct code")) this was a problem as the dma-api (at least the swiotlb_dma_ops on arm64) would use the potentially invalid dma_address. How that failed depended on how the device handled physical address 0. If 0 was a valid address to physical ram, that page would get flushed a lot, while the actual pages in the buffer would not get synced correctly. While if 0 is an invalid physical address it may cause a fault and trigger a crash. In v5.0 this was incidentally fixed by commit 55897af63091 ("dma-direct: merge swiotlb_dma_ops into the dma_direct code"), as this moved the dma-api to use the page pointer in the sg_list, and (for Ion buffers at least) this will always be valid if the sg_list exists at all. But, this issue is re-introduced in v5.3 with commit 449fa54d6815 ("dma-direct: correct the physical addr in dma_direct_sync_sg_for_cpu/device") moves the dma-api back to the old behaviour and picks the dma_address that may be invalid. dma-buf core doesn't ensure that the buffer is mapped on the device, and thus have a valid sg_list, before calling the exporter's begin_cpu_access. Logic and commit message originally by: Ørjan Eide Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/cma_heap.c | 10 ++++++++++ drivers/dma-buf/heaps/system_heap.c | 10 ++++++++++ 2 files changed, 20 insertions(+) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index 3adfdbed0829..b6ab0392ad9a 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -44,6 +44,7 @@ struct dma_heap_attachment { struct device *dev; struct sg_table table; struct list_head list; + bool mapped; }; static int cma_heap_attach(struct dma_buf *dmabuf, @@ -68,6 +69,7 @@ static int cma_heap_attach(struct dma_buf *dmabuf, a->dev = attachment->dev; INIT_LIST_HEAD(&a->list); + a->mapped = false; attachment->priv = a; @@ -101,6 +103,7 @@ static struct sg_table *cma_heap_map_dma_buf(struct dma_buf_attachment *attachme if (!dma_map_sg(attachment->dev, table->sgl, table->nents, direction)) table = ERR_PTR(-ENOMEM); + a->mapped = true; return table; } @@ -108,6 +111,9 @@ static void cma_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, struct sg_table *table, enum dma_data_direction direction) { + struct dma_heap_attachment *a = attachment->priv; + + a->mapped = false; dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); } @@ -123,6 +129,8 @@ static int cma_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, mutex_lock(&buffer->lock); list_for_each_entry(a, &buffer->attachments, list) { + if (!a->mapped) + continue; dma_sync_sg_for_cpu(a->dev, a->table.sgl, a->table.nents, direction); } @@ -142,6 +150,8 @@ static int cma_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, mutex_lock(&buffer->lock); list_for_each_entry(a, &buffer->attachments, list) { + if (!a->mapped) + continue; dma_sync_sg_for_device(a->dev, a->table.sgl, a->table.nents, direction); } diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index 9f57b4c8ae69..8a523b6fd51a 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -38,6 +38,7 @@ struct dma_heap_attachment { struct device *dev; struct sg_table *table; struct list_head list; + bool mapped; }; #define HIGH_ORDER_GFP (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \ @@ -94,6 +95,7 @@ static int system_heap_attach(struct dma_buf *dmabuf, a->table = table; a->dev = attachment->dev; INIT_LIST_HEAD(&a->list); + a->mapped = false; attachment->priv = a; @@ -128,6 +130,7 @@ static struct sg_table *system_heap_map_dma_buf(struct dma_buf_attachment *attac if (!dma_map_sg(attachment->dev, table->sgl, table->nents, direction)) return ERR_PTR(-ENOMEM); + a->mapped = true; return table; } @@ -135,6 +138,9 @@ static void system_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, struct sg_table *table, enum dma_data_direction direction) { + struct dma_heap_attachment *a = attachment->priv; + + a->mapped = false; dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); } @@ -151,6 +157,8 @@ static int system_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, invalidate_kernel_vmap_range(buffer->vaddr, buffer->len); list_for_each_entry(a, &buffer->attachments, list) { + if (!a->mapped) + continue; dma_sync_sg_for_cpu(a->dev, a->table->sgl, a->table->nents, direction); } @@ -171,6 +179,8 @@ static int system_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, flush_kernel_vmap_range(buffer->vaddr, buffer->len); list_for_each_entry(a, &buffer->attachments, list) { + if (!a->mapped) + continue; dma_sync_sg_for_device(a->dev, a->table->sgl, a->table->nents, direction); }