From patchwork Thu Oct 1 01:21:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313914 Delivered-To: patches@linaro.org Received: by 2002:a05:6e02:1081:0:0:0:0 with SMTP id r1csp109003ilj; Wed, 30 Sep 2020 18:21:57 -0700 (PDT) X-Received: by 2002:a17:902:c151:b029:d2:42a6:2fb with SMTP id 17-20020a170902c151b02900d242a602fbmr4952323plj.10.1601515317160; Wed, 30 Sep 2020 18:21:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601515317; cv=none; d=google.com; s=arc-20160816; b=RXMEn17b44ms4Pz2PqxNhyvL0ABojbIifTijVSlXb8w1A58UoKrAUE0i32nqe7Qv42 wICAGLMiX4giapxPH4/YBqjNKEJ8IOK6mcOnceBkSKlqH2+9F8f+hFTdYX2hjgPtkNMu QRT/peYjWh6MOE8ni6csZg1SmNqDPJjYmiC1k4skJaUAcXhqD/k34cJwIcV66sNlfrlg ERVvbHUUDE/8d+y4Y3fXUmmioGniMURvaPKPRCKg2RutulkPCdLM7tlkHrgOgAuMdrHK WCdoREhzMJOksmniPdzRkwjUeO4sxL9ZP62QCpZn6OywI6CME0DUdU2q4pkjyQfka7gv 0oGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=h0WDFYsPW+b6j1jfcUSQQeUriSMq11o0ZvabqoN39UA=; b=yc2za296mIWigSrcjScNA0EN/E0VUMxRj/u0gvKNnmNFSfPc/QE75ZlZ+DK1qw+Dl6 7LGuNAJqdMqPe1CHZmBY0CWFJUv2N46TqoOqfACA/Fc+HBWgnQLZZ923QW//VCK6wxDe e3Fca0MuzzOAyg/VX97yAEkWNtrwM+OTpLPJwwvZSTmUqxyZmGTKVkevj9mGHoucapTJ N8UFnu1IlXYqxBz1HTP5ZMo6HV8YQghaRtR6Cq4cRg+HudxbijHUDTUn0pTLIXR9dXb0 N5d+eN1IZKjM3DQ+RgWLWRhAAdrsEJLBfujjUjlSeR0Iqfc3e6xe/v6jPcVcON5VsHcC 3+cg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=onlQacxe; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id y4sor1630252pgk.63.2020.09.30.18.21.56 for (Google Transport Security); Wed, 30 Sep 2020 18:21:57 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=onlQacxe; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h0WDFYsPW+b6j1jfcUSQQeUriSMq11o0ZvabqoN39UA=; b=onlQacxekbcJDz2YjcVG2DrnNESvH5tbpYAdSHAMQgppHD31QJcbkcQ5mB7WKffcxU e/hZ3yNtxRSIfhUhVpi4ZQvZK2f3RZ5+l7afIPcgKaZiKjGXs73FMFhhi3f9m5HYWrjo uoOcvNbRCp0gYHW/xSM4Okul2NcrJnm/7YW3LSCQCY46b5Sg//iSN8PS7t7T5UIZZwP6 xs9x5DBNfRQ7WIO9IRIScF59LngoXOLPRp/zTawGyxUliCRHz8HR3KdVmI8jKDqsgHPY +GmU1rtZ+Bpr3MGf6SNjM88lWkf61MEWIs7jWQ2Z2L13byhSHs+PeynO2zgWmC/CQjUN HOYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h0WDFYsPW+b6j1jfcUSQQeUriSMq11o0ZvabqoN39UA=; b=kKFW0jHQouN4ZCGbRYeWhIVsd8JvE6xCR5GTXSocoycplqNbrewsPUEyE2Lt/0B8UB KUPCswjZrOaUWWhK7lwk0dmsG6RixVv5AAD1YRUrz2rLRwhhtWqOI+4aW3r1Wwdm3sY+ QbXrJxDBWilXOdLhSx0mIxd3OyypLSdNjCipJs6GOCSXFtxiOMzgo5tLove76FSyDjIV AxPGQ9EukmxylyijS+QZ6P6M0XHDReqUJHEXKOOOZSY4Tht9DnhglhiElCDVzd8E1b3m u27K9O9O9gQp8IYS71US+07mOUfJfctSJQFdMmYtBUxGXbkrmKpIfBDuyZP/Gi8636AI q0Zw== X-Gm-Message-State: AOAM532sNarmY4UmXhZPywbLGhOBbKMplGvkRCvZXvwE3wdwuxDPPlRb TLstHzzSGJEu1XdH9RI2aLIpcmiU X-Google-Smtp-Source: ABdhPJzWNWSslL0iUFeC6X2mHpQYhzxFyfd0Vf6GpXH3E/IgEzy/YetGQYsr8SopJIAsY87hUw3vjg== X-Received: by 2002:a63:df02:: with SMTP id u2mr4108997pgg.270.1601515316600; Wed, 30 Sep 2020 18:21:56 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id s187sm4229372pfc.134.2020.09.30.18.21.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Sep 2020 18:21:55 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH v2 1/6] dma-buf: system_heap: Rework system heap to use sgtables instead of pagelists Date: Thu, 1 Oct 2020 01:21:46 +0000 Message-Id: <20201001012151.21149-2-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201001012151.21149-1-john.stultz@linaro.org> References: <20201001012151.21149-1-john.stultz@linaro.org> MIME-Version: 1.0 In preparation for some patches to optmize the system heap code, rework the dmabuf exporter to utilize sgtables rather then pageslists for tracking the associated pages. This will allow for large order page allocations, as well as more efficient page pooling. In doing so, the system heap stops using the heap-helpers logic which sadly is not quite as generic as I was hoping it to be, so this patch adds heap specific implementations of the dma_buf_ops function handlers. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- v2: * Fix locking issue and an unused return value Reported-by: kernel test robot Julia Lawall * Make system_heap_buf_ops static Reported-by: kernel test robot --- drivers/dma-buf/heaps/system_heap.c | 344 ++++++++++++++++++++++++---- 1 file changed, 298 insertions(+), 46 deletions(-) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index 0bf688e3c023..7ec58f4e2bd3 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -3,7 +3,11 @@ * DMABUF System heap exporter * * Copyright (C) 2011 Google, Inc. - * Copyright (C) 2019 Linaro Ltd. + * Copyright (C) 2019, 2020 Linaro Ltd. + * + * Portions based off of Andrew Davis' SRAM heap: + * Copyright (C) 2019 Texas Instruments Incorporated - http://www.ti.com/ + * Andrew F. Davis */ #include @@ -15,72 +19,321 @@ #include #include #include -#include -#include - -#include "heap-helpers.h" +#include struct dma_heap *sys_heap; -static void system_heap_free(struct heap_helper_buffer *buffer) +struct system_heap_buffer { + struct dma_heap *heap; + struct list_head attachments; + struct mutex lock; + unsigned long len; + struct sg_table sg_table; + int vmap_cnt; + void *vaddr; +}; + +struct dma_heap_attachment { + struct device *dev; + struct sg_table *table; + struct list_head list; +}; + +static struct sg_table *dup_sg_table(struct sg_table *table) { - pgoff_t pg; + struct sg_table *new_table; + int ret, i; + struct scatterlist *sg, *new_sg; + + new_table = kzalloc(sizeof(*new_table), GFP_KERNEL); + if (!new_table) + return ERR_PTR(-ENOMEM); + + ret = sg_alloc_table(new_table, table->nents, GFP_KERNEL); + if (ret) { + kfree(new_table); + return ERR_PTR(-ENOMEM); + } + + new_sg = new_table->sgl; + for_each_sgtable_sg(table, sg, i) { + sg_set_page(new_sg, sg_page(sg), sg->length, sg->offset); + new_sg = sg_next(new_sg); + } + + return new_table; +} + +static int system_heap_attach(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + struct sg_table *table; + + a = kzalloc(sizeof(*a), GFP_KERNEL); + if (!a) + return -ENOMEM; + + table = dup_sg_table(&buffer->sg_table); + if (IS_ERR(table)) { + kfree(a); + return -ENOMEM; + } + + a->table = table; + a->dev = attachment->dev; + INIT_LIST_HEAD(&a->list); + + attachment->priv = a; + + mutex_lock(&buffer->lock); + list_add(&a->list, &buffer->attachments); + mutex_unlock(&buffer->lock); + + return 0; +} + +static void system_heap_detatch(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a = attachment->priv; + + mutex_lock(&buffer->lock); + list_del(&a->list); + mutex_unlock(&buffer->lock); + + sg_free_table(a->table); + kfree(a->table); + kfree(a); +} + +static struct sg_table *system_heap_map_dma_buf(struct dma_buf_attachment *attachment, + enum dma_data_direction direction) +{ + struct dma_heap_attachment *a = attachment->priv; + struct sg_table *table = a->table; + + if (!dma_map_sg(attachment->dev, table->sgl, table->nents, direction)) + return ERR_PTR(-ENOMEM); + + return table; +} + +static void system_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, + struct sg_table *table, + enum dma_data_direction direction) +{ + dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); +} + +static int system_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + + mutex_lock(&buffer->lock); + + if (buffer->vmap_cnt) + invalidate_kernel_vmap_range(buffer->vaddr, buffer->len); + + list_for_each_entry(a, &buffer->attachments, list) { + dma_sync_sg_for_cpu(a->dev, a->table->sgl, a->table->nents, + direction); + } + mutex_unlock(&buffer->lock); + + return 0; +} - for (pg = 0; pg < buffer->pagecount; pg++) - __free_page(buffer->pages[pg]); - kfree(buffer->pages); +static int system_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + + mutex_lock(&buffer->lock); + + if (buffer->vmap_cnt) + flush_kernel_vmap_range(buffer->vaddr, buffer->len); + + list_for_each_entry(a, &buffer->attachments, list) { + dma_sync_sg_for_device(a->dev, a->table->sgl, a->table->nents, + direction); + } + mutex_unlock(&buffer->lock); + + return 0; +} + +static int system_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct sg_table *table = &buffer->sg_table; + unsigned long addr = vma->vm_start; + struct sg_page_iter piter; + int ret; + + for_each_sgtable_page(table, &piter, vma->vm_pgoff) { + struct page *page = sg_page_iter_page(&piter); + + ret = remap_pfn_range(vma, addr, page_to_pfn(page), PAGE_SIZE, + vma->vm_page_prot); + if (ret) + return ret; + addr += PAGE_SIZE; + if (addr >= vma->vm_end) + return 0; + } + return 0; +} + +static void *system_heap_do_vmap(struct system_heap_buffer *buffer) +{ + struct sg_table *table = &buffer->sg_table; + int npages = PAGE_ALIGN(buffer->len) / PAGE_SIZE; + struct page **pages = vmalloc(sizeof(struct page *) * npages); + struct page **tmp = pages; + struct sg_page_iter piter; + void *vaddr; + + if (!pages) + return ERR_PTR(-ENOMEM); + + for_each_sgtable_page(table, &piter, 0) { + WARN_ON(tmp - pages >= npages); + *tmp++ = sg_page_iter_page(&piter); + } + + vaddr = vmap(pages, npages, VM_MAP, PAGE_KERNEL); + vfree(pages); + + if (!vaddr) + return ERR_PTR(-ENOMEM); + + return vaddr; +} + +static void *system_heap_vmap(struct dma_buf *dmabuf) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + void *vaddr; + + mutex_lock(&buffer->lock); + if (buffer->vmap_cnt) { + buffer->vmap_cnt++; + vaddr = buffer->vaddr; + goto out; + } + + vaddr = system_heap_do_vmap(buffer); + if (IS_ERR(vaddr)) + goto out; + + buffer->vaddr = vaddr; + buffer->vmap_cnt++; +out: + mutex_unlock(&buffer->lock); + + return vaddr; +} + +static void system_heap_vunmap(struct dma_buf *dmabuf, void *vaddr) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + + mutex_lock(&buffer->lock); + if (!--buffer->vmap_cnt) { + vunmap(buffer->vaddr); + buffer->vaddr = NULL; + } + mutex_unlock(&buffer->lock); +} + +static void system_heap_dma_buf_release(struct dma_buf *dmabuf) +{ + struct system_heap_buffer *buffer = dmabuf->priv; + struct sg_table *table; + struct scatterlist *sg; + int i; + + table = &buffer->sg_table; + for_each_sgtable_sg(table, sg, i) + __free_page(sg_page(sg)); + sg_free_table(table); kfree(buffer); } +static const struct dma_buf_ops system_heap_buf_ops = { + .attach = system_heap_attach, + .detach = system_heap_detatch, + .map_dma_buf = system_heap_map_dma_buf, + .unmap_dma_buf = system_heap_unmap_dma_buf, + .begin_cpu_access = system_heap_dma_buf_begin_cpu_access, + .end_cpu_access = system_heap_dma_buf_end_cpu_access, + .mmap = system_heap_mmap, + .vmap = system_heap_vmap, + .vunmap = system_heap_vunmap, + .release = system_heap_dma_buf_release, +}; + static int system_heap_allocate(struct dma_heap *heap, unsigned long len, unsigned long fd_flags, unsigned long heap_flags) { - struct heap_helper_buffer *helper_buffer; + struct system_heap_buffer *buffer; + DEFINE_DMA_BUF_EXPORT_INFO(exp_info); struct dma_buf *dmabuf; - int ret = -ENOMEM; + struct sg_table *table; + struct scatterlist *sg; + pgoff_t pagecount; pgoff_t pg; + int i, ret = -ENOMEM; - helper_buffer = kzalloc(sizeof(*helper_buffer), GFP_KERNEL); - if (!helper_buffer) + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); + if (!buffer) return -ENOMEM; - init_heap_helper_buffer(helper_buffer, system_heap_free); - helper_buffer->heap = heap; - helper_buffer->size = len; - - helper_buffer->pagecount = len / PAGE_SIZE; - helper_buffer->pages = kmalloc_array(helper_buffer->pagecount, - sizeof(*helper_buffer->pages), - GFP_KERNEL); - if (!helper_buffer->pages) { - ret = -ENOMEM; - goto err0; - } + INIT_LIST_HEAD(&buffer->attachments); + mutex_init(&buffer->lock); + buffer->heap = heap; + buffer->len = len; - for (pg = 0; pg < helper_buffer->pagecount; pg++) { + table = &buffer->sg_table; + pagecount = len / PAGE_SIZE; + if (sg_alloc_table(table, pagecount, GFP_KERNEL)) + goto free_buffer; + + sg = table->sgl; + for (pg = 0; pg < pagecount; pg++) { + struct page *page; /* * Avoid trying to allocate memory if the process - * has been killed by by SIGKILL + * has been killed by SIGKILL */ if (fatal_signal_pending(current)) - goto err1; - - helper_buffer->pages[pg] = alloc_page(GFP_KERNEL | __GFP_ZERO); - if (!helper_buffer->pages[pg]) - goto err1; + goto free_pages; + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + goto free_pages; + sg_set_page(sg, page, page_size(page), 0); + sg = sg_next(sg); } /* create the dmabuf */ - dmabuf = heap_helper_export_dmabuf(helper_buffer, fd_flags); + exp_info.ops = &system_heap_buf_ops; + exp_info.size = buffer->len; + exp_info.flags = fd_flags; + exp_info.priv = buffer; + dmabuf = dma_buf_export(&exp_info); if (IS_ERR(dmabuf)) { ret = PTR_ERR(dmabuf); - goto err1; + goto free_pages; } - helper_buffer->dmabuf = dmabuf; - ret = dma_buf_fd(dmabuf, fd_flags); if (ret < 0) { dma_buf_put(dmabuf); @@ -90,12 +343,12 @@ static int system_heap_allocate(struct dma_heap *heap, return ret; -err1: - while (pg > 0) - __free_page(helper_buffer->pages[--pg]); - kfree(helper_buffer->pages); -err0: - kfree(helper_buffer); +free_pages: + for_each_sgtable_sg(table, sg, i) + __free_page(sg_page(sg)); + sg_free_table(table); +free_buffer: + kfree(buffer); return ret; } @@ -107,7 +360,6 @@ static const struct dma_heap_ops system_heap_ops = { static int system_heap_create(void) { struct dma_heap_export_info exp_info; - int ret = 0; exp_info.name = "system"; exp_info.ops = &system_heap_ops; @@ -115,9 +367,9 @@ static int system_heap_create(void) sys_heap = dma_heap_add(&exp_info); if (IS_ERR(sys_heap)) - ret = PTR_ERR(sys_heap); + return PTR_ERR(sys_heap); - return ret; + return 0; } module_init(system_heap_create); MODULE_LICENSE("GPL v2"); From patchwork Thu Oct 1 01:21:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313915 Delivered-To: patches@linaro.org Received: by 2002:a05:6e02:1081:0:0:0:0 with SMTP id r1csp109012ilj; Wed, 30 Sep 2020 18:21:58 -0700 (PDT) X-Received: by 2002:a17:902:9a8d:b029:d2:4ef:a1fb with SMTP id w13-20020a1709029a8db02900d204efa1fbmr5060713plp.4.1601515318535; Wed, 30 Sep 2020 18:21:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601515318; cv=none; d=google.com; s=arc-20160816; b=fZastEZo2SjvDl/ky+xTwCF7rzSDiS9IOON4thzMP7CFv/UkT+gDpqz3qyBLiyay8U JbhPcTnuQ21ycBkFOKZroBTaze+hgMd4ZNg6Ow7gMlAcdg0QPMMHuryY4GKNlnSWOgrW 2Nhg8SnL3xxMQSpb+ufpGITtuHJHY4VMwQUokh4Vn1AEZYs46fdpFckuFs+KShmVZX02 J5Il+/ahAioRlDsV0Yodl9+nYCOGez6AkRe2n9b4Ao73sQA9uZVVmmyahbAPO9iz4urk E5fo/1dHGhwtqYBzd9R2mXaiW/jxARusWMJvKL2mBi9YgYYjh3myr9u6vZurdV2yo3l3 9BeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=0krrV9QQqBZ5X418lRts3FHtSVkcKvaoC2dPWsjbk08=; b=MzMu5LCe7i3iz52yq3n92WyQH9mL/BC/BfIkxhlrk1ifhkyIS8Iv+WoS0UvdDIgjZq E+eAMFxysRmIV/O5z1/C2fqRINSE5u7dtyApumjrw6aEXNSnVcV2hlziyv5CU0eSCyX8 N01/9xlytZ2XVZ+ZVfbUOTZWqNJljaPcB8NUMtLzzIuXk148beKeoYL69ZQd2cBCKwiS z/QUuEUBNEPM9d1CuOTniPhbgQecUeBr0N7q9z6y9q93zwcy45e1u8qoOTPE4F8lV31P NxZ9x7b2BIF5L/VlvC6mdY5Rbzbu4QtTLpRyBCYaZey+8R7kb8hRXUKf4dC3TXQT6Moz AiGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=mGde2I67; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id a14sor1593003pgk.79.2020.09.30.18.21.58 for (Google Transport Security); Wed, 30 Sep 2020 18:21:58 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=mGde2I67; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0krrV9QQqBZ5X418lRts3FHtSVkcKvaoC2dPWsjbk08=; b=mGde2I67ZK4J+d0ti4ccvfkiji9Yb8JMb16pA/qCf05mTcU0ajxM8+5sJKMWx8Z+hz GuaBRpc14uxQ8tmbjtgkM4cKAVmN+ZH3SPAG9tDhOiU6GJT/LOXJh0udyjluTvLA5IIE 7hX/CZC8fslNR/x+Rv0Dw01n6nbd3Ff/AP6ejJoFKO8e2vFw2thiYJVZM9K1wA8vIb+F o36emcd/ZPJKRAq6WyhvqpajqkdiAOUXrVHjbhoMzzDJN8n1I4Z8zSmdF4sczOfBRv1H hnFoJTzYxqKVVyKrzR2EmssaA/nJwwQCzAV5+jj6tiTnLgg4e8LehE0Ns+RNL4Fj3hNp GnXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0krrV9QQqBZ5X418lRts3FHtSVkcKvaoC2dPWsjbk08=; b=LjxGy3MP+HbZ1t6i6d6GXqCxk9e9TOuGFMDBfYoy83YkfRp+Y0SfoW0hm/xsdPQOdm q+iJvmvcN+o73CXoZwEvX+lg0BEXnbMJvHVDgQlqfnLDp1pEB75vbaccKO6xA9mppHPx RyPQEKMKOxfFS7klc8Is8IKY4gDwKcov5Tnl9tMG8QEeoRbtY3lLvGu4EZSXzEFpnrmf QZUSIlAPa+5YW3p45vaN6K2+vuPYCiT2oisOmmtxmDUbAVIgo8koNavi2oRA2MnATEJC IJlma8IizqydrEe2niekGCz+ovuIYRb5WwPcj81XVc7yBbKiOq1B85SmhBnxnWY57sT3 jTXA== X-Gm-Message-State: AOAM532TG+QZG5l/ob8SNq166wKzAX7kxID9KClT6J7bZ+11wN/LOUxD 9fhgzsb9FLSu0xdmfZJxSepeAf1v X-Google-Smtp-Source: ABdhPJzKeu30havIBE8S7I6m6TIGRTvYP9rkSyWVYVcaCf1cayBJ3RuNEB454qubWvpAgORUA6DUfA== X-Received: by 2002:a63:1a50:: with SMTP id a16mr4091990pgm.331.1601515318149; Wed, 30 Sep 2020 18:21:58 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id s187sm4229372pfc.134.2020.09.30.18.21.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Sep 2020 18:21:57 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH v2 2/6] dma-buf: heaps: Move heap-helper logic into the cma_heap implementation Date: Thu, 1 Oct 2020 01:21:47 +0000 Message-Id: <20201001012151.21149-3-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201001012151.21149-1-john.stultz@linaro.org> References: <20201001012151.21149-1-john.stultz@linaro.org> MIME-Version: 1.0 Since the heap-helpers logic ended up not being as generic as hoped, move the heap-helpers dma_buf_ops implementations into the cma_heap directly. This will allow us to remove the heap_helpers code in a following patch. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- v2: * Fix unused return value and locking issue Reported-by: kernel test robot Julia Lawall * Make cma_heap_buf_ops static suggested by kernel test robot * Fix uninitialized return in cma Reported-by: kernel test robot * Minor cleanups --- drivers/dma-buf/heaps/cma_heap.c | 318 ++++++++++++++++++++++++++----- 1 file changed, 268 insertions(+), 50 deletions(-) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index 626cf7fd033a..4f20f07872e5 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -2,76 +2,292 @@ /* * DMABUF CMA heap exporter * - * Copyright (C) 2012, 2019 Linaro Ltd. + * Copyright (C) 2012, 2019, 2020 Linaro Ltd. * Author: for ST-Ericsson. + * + * Also utilizing parts of Andrew Davis' SRAM heap: + * Copyright (C) 2019 Texas Instruments Incorporated - http://www.ti.com/ + * Andrew F. Davis */ - #include -#include #include -#include #include +#include +#include #include -#include #include +#include +#include #include -#include #include -#include +#include -#include "heap-helpers.h" struct cma_heap { struct dma_heap *heap; struct cma *cma; }; -static void cma_heap_free(struct heap_helper_buffer *buffer) +struct cma_heap_buffer { + struct cma_heap *heap; + struct list_head attachments; + struct mutex lock; + unsigned long len; + struct page *cma_pages; + struct page **pages; + pgoff_t pagecount; + int vmap_cnt; + void *vaddr; +}; + +struct dma_heap_attachment { + struct device *dev; + struct sg_table table; + struct list_head list; +}; + +static int cma_heap_attach(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) { - struct cma_heap *cma_heap = dma_heap_get_drvdata(buffer->heap); - unsigned long nr_pages = buffer->pagecount; - struct page *cma_pages = buffer->priv_virt; + struct cma_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + int ret; - /* free page list */ - kfree(buffer->pages); - /* release memory */ - cma_release(cma_heap->cma, cma_pages, nr_pages); + a = kzalloc(sizeof(*a), GFP_KERNEL); + if (!a) + return -ENOMEM; + + ret = sg_alloc_table_from_pages(&a->table, buffer->pages, + buffer->pagecount, 0, + buffer->pagecount << PAGE_SHIFT, + GFP_KERNEL); + if (ret) { + kfree(a); + return ret; + } + + a->dev = attachment->dev; + INIT_LIST_HEAD(&a->list); + + attachment->priv = a; + + mutex_lock(&buffer->lock); + list_add(&a->list, &buffer->attachments); + mutex_unlock(&buffer->lock); + + return 0; +} + +static void cma_heap_detatch(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a = attachment->priv; + + mutex_lock(&buffer->lock); + list_del(&a->list); + mutex_unlock(&buffer->lock); + + sg_free_table(&a->table); + kfree(a); +} + +static struct sg_table *cma_heap_map_dma_buf(struct dma_buf_attachment *attachment, + enum dma_data_direction direction) +{ + struct dma_heap_attachment *a = attachment->priv; + struct sg_table *table = &a->table; + + if (!dma_map_sg(attachment->dev, table->sgl, table->nents, + direction)) + table = ERR_PTR(-ENOMEM); + return table; +} + +static void cma_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, + struct sg_table *table, + enum dma_data_direction direction) +{ + dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); +} + +static int cma_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + + if (buffer->vmap_cnt) + invalidate_kernel_vmap_range(buffer->vaddr, buffer->len); + + mutex_lock(&buffer->lock); + list_for_each_entry(a, &buffer->attachments, list) { + dma_sync_sg_for_cpu(a->dev, a->table.sgl, a->table.nents, + direction); + } + mutex_unlock(&buffer->lock); + + return 0; +} + +static int cma_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + struct dma_heap_attachment *a; + + if (buffer->vmap_cnt) + flush_kernel_vmap_range(buffer->vaddr, buffer->len); + + mutex_lock(&buffer->lock); + list_for_each_entry(a, &buffer->attachments, list) { + dma_sync_sg_for_device(a->dev, a->table.sgl, a->table.nents, + direction); + } + mutex_unlock(&buffer->lock); + + return 0; +} + +static vm_fault_t cma_heap_vm_fault(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct cma_heap_buffer *buffer = vma->vm_private_data; + + if (vmf->pgoff > buffer->pagecount) + return VM_FAULT_SIGBUS; + + vmf->page = buffer->pages[vmf->pgoff]; + get_page(vmf->page); + + return 0; +} + +static const struct vm_operations_struct dma_heap_vm_ops = { + .fault = cma_heap_vm_fault, +}; + +static int cma_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + + if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0) + return -EINVAL; + + vma->vm_ops = &dma_heap_vm_ops; + vma->vm_private_data = buffer; + + return 0; +} + +static void *cma_heap_do_vmap(struct cma_heap_buffer *buffer) +{ + void *vaddr; + + vaddr = vmap(buffer->pages, buffer->pagecount, VM_MAP, PAGE_KERNEL); + if (!vaddr) + return ERR_PTR(-ENOMEM); + + return vaddr; +} + +static void *cma_heap_vmap(struct dma_buf *dmabuf) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + void *vaddr; + + mutex_lock(&buffer->lock); + if (buffer->vmap_cnt) { + buffer->vmap_cnt++; + vaddr = buffer->vaddr; + goto out; + } + + vaddr = cma_heap_do_vmap(buffer); + if (IS_ERR(vaddr)) + goto out; + + buffer->vaddr = vaddr; + buffer->vmap_cnt++; +out: + mutex_unlock(&buffer->lock); + + return vaddr; +} + +static void cma_heap_vunmap(struct dma_buf *dmabuf, void *vaddr) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + + mutex_lock(&buffer->lock); + if (!--buffer->vmap_cnt) { + vunmap(buffer->vaddr); + buffer->vaddr = NULL; + } + mutex_unlock(&buffer->lock); +} + +static void cma_heap_dma_buf_release(struct dma_buf *dmabuf) +{ + struct cma_heap_buffer *buffer = dmabuf->priv; + struct cma_heap *cma_heap = buffer->heap; + + if (buffer->vmap_cnt > 0) { + WARN(1, "%s: buffer still mapped in the kernel\n", __func__); + vunmap(buffer->vaddr); + } + + cma_release(cma_heap->cma, buffer->cma_pages, buffer->pagecount); kfree(buffer); } -/* dmabuf heap CMA operations functions */ +static const struct dma_buf_ops cma_heap_buf_ops = { + .attach = cma_heap_attach, + .detach = cma_heap_detatch, + .map_dma_buf = cma_heap_map_dma_buf, + .unmap_dma_buf = cma_heap_unmap_dma_buf, + .begin_cpu_access = cma_heap_dma_buf_begin_cpu_access, + .end_cpu_access = cma_heap_dma_buf_end_cpu_access, + .mmap = cma_heap_mmap, + .vmap = cma_heap_vmap, + .vunmap = cma_heap_vunmap, + .release = cma_heap_dma_buf_release, +}; + static int cma_heap_allocate(struct dma_heap *heap, - unsigned long len, - unsigned long fd_flags, - unsigned long heap_flags) + unsigned long len, + unsigned long fd_flags, + unsigned long heap_flags) { struct cma_heap *cma_heap = dma_heap_get_drvdata(heap); - struct heap_helper_buffer *helper_buffer; - struct page *cma_pages; + struct cma_heap_buffer *buffer; + DEFINE_DMA_BUF_EXPORT_INFO(exp_info); size_t size = PAGE_ALIGN(len); - unsigned long nr_pages = size >> PAGE_SHIFT; + pgoff_t pagecount = size >> PAGE_SHIFT; unsigned long align = get_order(size); + struct page *cma_pages; struct dma_buf *dmabuf; int ret = -ENOMEM; pgoff_t pg; - if (align > CONFIG_CMA_ALIGNMENT) - align = CONFIG_CMA_ALIGNMENT; - - helper_buffer = kzalloc(sizeof(*helper_buffer), GFP_KERNEL); - if (!helper_buffer) + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); + if (!buffer) return -ENOMEM; - init_heap_helper_buffer(helper_buffer, cma_heap_free); - helper_buffer->heap = heap; - helper_buffer->size = len; + INIT_LIST_HEAD(&buffer->attachments); + mutex_init(&buffer->lock); + buffer->len = size; + + if (align > CONFIG_CMA_ALIGNMENT) + align = CONFIG_CMA_ALIGNMENT; - cma_pages = cma_alloc(cma_heap->cma, nr_pages, align, false); + cma_pages = cma_alloc(cma_heap->cma, pagecount, align, false); if (!cma_pages) - goto free_buf; + goto free_buffer; + /* Clear the cma pages */ if (PageHighMem(cma_pages)) { - unsigned long nr_clear_pages = nr_pages; + unsigned long nr_clear_pages = pagecount; struct page *page = cma_pages; while (nr_clear_pages > 0) { @@ -85,7 +301,6 @@ static int cma_heap_allocate(struct dma_heap *heap, */ if (fatal_signal_pending(current)) goto free_cma; - page++; nr_clear_pages--; } @@ -93,28 +308,30 @@ static int cma_heap_allocate(struct dma_heap *heap, memset(page_address(cma_pages), 0, size); } - helper_buffer->pagecount = nr_pages; - helper_buffer->pages = kmalloc_array(helper_buffer->pagecount, - sizeof(*helper_buffer->pages), - GFP_KERNEL); - if (!helper_buffer->pages) { + buffer->pages = kmalloc_array(pagecount, sizeof(*buffer->pages), GFP_KERNEL); + if (!buffer->pages) { ret = -ENOMEM; goto free_cma; } - for (pg = 0; pg < helper_buffer->pagecount; pg++) - helper_buffer->pages[pg] = &cma_pages[pg]; + for (pg = 0; pg < pagecount; pg++) + buffer->pages[pg] = &cma_pages[pg]; + + buffer->cma_pages = cma_pages; + buffer->heap = cma_heap; + buffer->pagecount = pagecount; /* create the dmabuf */ - dmabuf = heap_helper_export_dmabuf(helper_buffer, fd_flags); + exp_info.ops = &cma_heap_buf_ops; + exp_info.size = buffer->len; + exp_info.flags = fd_flags; + exp_info.priv = buffer; + dmabuf = dma_buf_export(&exp_info); if (IS_ERR(dmabuf)) { ret = PTR_ERR(dmabuf); goto free_pages; } - helper_buffer->dmabuf = dmabuf; - helper_buffer->priv_virt = cma_pages; - ret = dma_buf_fd(dmabuf, fd_flags); if (ret < 0) { dma_buf_put(dmabuf); @@ -125,11 +342,12 @@ static int cma_heap_allocate(struct dma_heap *heap, return ret; free_pages: - kfree(helper_buffer->pages); + kfree(buffer->pages); free_cma: - cma_release(cma_heap->cma, cma_pages, nr_pages); -free_buf: - kfree(helper_buffer); + cma_release(cma_heap->cma, cma_pages, pagecount); +free_buffer: + kfree(buffer); + return ret; } From patchwork Thu Oct 1 01:21:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313916 Delivered-To: patches@linaro.org Received: by 2002:a05:6e02:1081:0:0:0:0 with SMTP id r1csp109037ilj; Wed, 30 Sep 2020 18:22:00 -0700 (PDT) X-Received: by 2002:a05:6a00:23cc:b029:142:2501:34e5 with SMTP id g12-20020a056a0023ccb0290142250134e5mr4832546pfc.62.1601515320269; Wed, 30 Sep 2020 18:22:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601515320; cv=none; d=google.com; s=arc-20160816; b=rFXsxN4ItO9SeSO9DAvv8EdXX4JNSjxWINZJpUCy6coSlvZQP+m7ysEWBTCx6gPJED 1uNIXJQQEOYMyCiUifAArtie8xlY5oOGkFscIPdDRY7wVb776ON9C3yk5gSAnRztgz2R fWpC1DhtNhBYjmkbDbbtJCVS7Mt4WvP325e/WB7kwrwWe4WnR4WoXtaK9HcoQJFFMFr8 LSfgkTfr1vIvLvGHqJyzHdNdhOrYwK0Q8Uqb5zB093dWrr9KgHAdDSP/ASp9lvRCwG7j B7q4dOl0zV2vLJChPcg/c189pFtulqYxZl7oV1hNy8VnsmDaCXrrl/Y6jFAfrQKrIahE d/7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=06XEeGHHOGK/ICro7P5tlQH3HvkQIwMvcU78rykFbzY=; b=nruSvxowYdu82qaNGVVSXqC++G10fS6mja+fagQDtDCg5YRFPRY3hwzAuBV+Fg0L0G gj2XjAsGiRzZZgtaap2Vdr9TfvrWEKOUh5x4kxF2F0dWdCErStimobgQAvgpyRc0aG4r hgXQ0FnwJADQ1ip0eVH1g9DQFtISoNKh8ZAdGCNDiE8Y+7jTNay/ArQcPT1huk2Mnxtd gHrMK4uhMwzS8GQoOc/PBjwaYXsCGrFOBtwWygGv8KNMI3Q0ACDg6fONngqmokuPdX1C 6DjR9ky0uF8J5rHhijvOcr6ZF1DN1PxurMcqPatZtalqjWwtp6VNJ9lr2yYyD2sP7WcR Av6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=srlTzlXj; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id i7sor1829223pfa.26.2020.09.30.18.22.00 for (Google Transport Security); Wed, 30 Sep 2020 18:22:00 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=srlTzlXj; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=06XEeGHHOGK/ICro7P5tlQH3HvkQIwMvcU78rykFbzY=; b=srlTzlXjDuniWVbs9GfCKAnJgOS3Cu4jsrEKbpqIl12R55RBEddXKfd0IaROC+qun+ THaAhbiQMd8wrPNJVs9rVooXKWXC2NcgK+jKVpCPll/qzKYu8npNR8j1bVvZswBMfeiA OZAXYxGPqtIyc5oarpoY0QOWTppVJkxoDIF2JCSnFltg6WJjJKyO7UpG83q45aLJwPTf YNw3QPVdXwuDhLKswzRwwvnTKUWpzgjI41RC7cnZVMu2t5RALsNWxRs4k/KpyGK4Jgi5 JfLDDF85c7FfsTWcat+MCdlKR4ODvsVJULeGsCXpshl24OZ/wuJPf4unQyFYbNmQHzZ1 aAAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=06XEeGHHOGK/ICro7P5tlQH3HvkQIwMvcU78rykFbzY=; b=jv9SMoDOHLKjRcsCrYjiBTWpMrzpxeIo7+6JGiCeKVem72Oj24Vm6CNZnjlb5AHqOk zOkL8oUp/XJxCnC4cmtDW/RangDx8SHQyNEx+OrnzIge5nPiEfWuNGjq2HlGl0m7VYNI EV3G2c/WO6dxn+QAvfOHDw7v7Ps401ND+4tZ6ot/1ltLC+O7/bXeES/+quTqOaWuqISO wVAYr0nfL9V40xE4VfmJulP0ibGvbiubzIzPPRhoGtKj3+y3gT8urFI7VCVGeDSplM6F HQ0er2oDEV0lXRuwmGbxU9afj7mgzW5UGrcIQwBx/56ixns1FpRoGDa3utxu3r4o4tAr N/nw== X-Gm-Message-State: AOAM531iRGSlGcqEGzZjn7b8PVzh4KuXi4evoxxfIs6pnmxe4oR5CyPe hN8QVZirRNpTUJLiaaT8Kw+U1ct6 X-Google-Smtp-Source: ABdhPJyz376O8FVlK38Ko3YQBAjOaI+Vnyny0RBl8KY7rhrsoJ1rzFRBMB9sYVPc04+cSPYxQAzq9Q== X-Received: by 2002:a63:f107:: with SMTP id f7mr4043749pgi.155.1601515319879; Wed, 30 Sep 2020 18:21:59 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id s187sm4229372pfc.134.2020.09.30.18.21.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Sep 2020 18:21:58 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH v2 3/6] dma-buf: heaps: Remove heap-helpers code Date: Thu, 1 Oct 2020 01:21:48 +0000 Message-Id: <20201001012151.21149-4-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201001012151.21149-1-john.stultz@linaro.org> References: <20201001012151.21149-1-john.stultz@linaro.org> MIME-Version: 1.0 The heap-helpers code was not as generic as initially hoped and it is now not being used, so remove it from the tree. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/Makefile | 1 - drivers/dma-buf/heaps/heap-helpers.c | 271 --------------------------- drivers/dma-buf/heaps/heap-helpers.h | 53 ------ 3 files changed, 325 deletions(-) delete mode 100644 drivers/dma-buf/heaps/heap-helpers.c delete mode 100644 drivers/dma-buf/heaps/heap-helpers.h -- 2.17.1 diff --git a/drivers/dma-buf/heaps/Makefile b/drivers/dma-buf/heaps/Makefile index 6e54cdec3da0..974467791032 100644 --- a/drivers/dma-buf/heaps/Makefile +++ b/drivers/dma-buf/heaps/Makefile @@ -1,4 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += heap-helpers.o obj-$(CONFIG_DMABUF_HEAPS_SYSTEM) += system_heap.o obj-$(CONFIG_DMABUF_HEAPS_CMA) += cma_heap.o diff --git a/drivers/dma-buf/heaps/heap-helpers.c b/drivers/dma-buf/heaps/heap-helpers.c deleted file mode 100644 index 9f964ca3f59c..000000000000 --- a/drivers/dma-buf/heaps/heap-helpers.c +++ /dev/null @@ -1,271 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "heap-helpers.h" - -void init_heap_helper_buffer(struct heap_helper_buffer *buffer, - void (*free)(struct heap_helper_buffer *)) -{ - buffer->priv_virt = NULL; - mutex_init(&buffer->lock); - buffer->vmap_cnt = 0; - buffer->vaddr = NULL; - buffer->pagecount = 0; - buffer->pages = NULL; - INIT_LIST_HEAD(&buffer->attachments); - buffer->free = free; -} - -struct dma_buf *heap_helper_export_dmabuf(struct heap_helper_buffer *buffer, - int fd_flags) -{ - DEFINE_DMA_BUF_EXPORT_INFO(exp_info); - - exp_info.ops = &heap_helper_ops; - exp_info.size = buffer->size; - exp_info.flags = fd_flags; - exp_info.priv = buffer; - - return dma_buf_export(&exp_info); -} - -static void *dma_heap_map_kernel(struct heap_helper_buffer *buffer) -{ - void *vaddr; - - vaddr = vmap(buffer->pages, buffer->pagecount, VM_MAP, PAGE_KERNEL); - if (!vaddr) - return ERR_PTR(-ENOMEM); - - return vaddr; -} - -static void dma_heap_buffer_destroy(struct heap_helper_buffer *buffer) -{ - if (buffer->vmap_cnt > 0) { - WARN(1, "%s: buffer still mapped in the kernel\n", __func__); - vunmap(buffer->vaddr); - } - - buffer->free(buffer); -} - -static void *dma_heap_buffer_vmap_get(struct heap_helper_buffer *buffer) -{ - void *vaddr; - - if (buffer->vmap_cnt) { - buffer->vmap_cnt++; - return buffer->vaddr; - } - vaddr = dma_heap_map_kernel(buffer); - if (IS_ERR(vaddr)) - return vaddr; - buffer->vaddr = vaddr; - buffer->vmap_cnt++; - return vaddr; -} - -static void dma_heap_buffer_vmap_put(struct heap_helper_buffer *buffer) -{ - if (!--buffer->vmap_cnt) { - vunmap(buffer->vaddr); - buffer->vaddr = NULL; - } -} - -struct dma_heaps_attachment { - struct device *dev; - struct sg_table table; - struct list_head list; -}; - -static int dma_heap_attach(struct dma_buf *dmabuf, - struct dma_buf_attachment *attachment) -{ - struct dma_heaps_attachment *a; - struct heap_helper_buffer *buffer = dmabuf->priv; - int ret; - - a = kzalloc(sizeof(*a), GFP_KERNEL); - if (!a) - return -ENOMEM; - - ret = sg_alloc_table_from_pages(&a->table, buffer->pages, - buffer->pagecount, 0, - buffer->pagecount << PAGE_SHIFT, - GFP_KERNEL); - if (ret) { - kfree(a); - return ret; - } - - a->dev = attachment->dev; - INIT_LIST_HEAD(&a->list); - - attachment->priv = a; - - mutex_lock(&buffer->lock); - list_add(&a->list, &buffer->attachments); - mutex_unlock(&buffer->lock); - - return 0; -} - -static void dma_heap_detach(struct dma_buf *dmabuf, - struct dma_buf_attachment *attachment) -{ - struct dma_heaps_attachment *a = attachment->priv; - struct heap_helper_buffer *buffer = dmabuf->priv; - - mutex_lock(&buffer->lock); - list_del(&a->list); - mutex_unlock(&buffer->lock); - - sg_free_table(&a->table); - kfree(a); -} - -static -struct sg_table *dma_heap_map_dma_buf(struct dma_buf_attachment *attachment, - enum dma_data_direction direction) -{ - struct dma_heaps_attachment *a = attachment->priv; - struct sg_table *table; - - table = &a->table; - - if (!dma_map_sg(attachment->dev, table->sgl, table->nents, - direction)) - table = ERR_PTR(-ENOMEM); - return table; -} - -static void dma_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, - struct sg_table *table, - enum dma_data_direction direction) -{ - dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); -} - -static vm_fault_t dma_heap_vm_fault(struct vm_fault *vmf) -{ - struct vm_area_struct *vma = vmf->vma; - struct heap_helper_buffer *buffer = vma->vm_private_data; - - if (vmf->pgoff > buffer->pagecount) - return VM_FAULT_SIGBUS; - - vmf->page = buffer->pages[vmf->pgoff]; - get_page(vmf->page); - - return 0; -} - -static const struct vm_operations_struct dma_heap_vm_ops = { - .fault = dma_heap_vm_fault, -}; - -static int dma_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - - if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0) - return -EINVAL; - - vma->vm_ops = &dma_heap_vm_ops; - vma->vm_private_data = buffer; - - return 0; -} - -static void dma_heap_dma_buf_release(struct dma_buf *dmabuf) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - - dma_heap_buffer_destroy(buffer); -} - -static int dma_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, - enum dma_data_direction direction) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - struct dma_heaps_attachment *a; - int ret = 0; - - mutex_lock(&buffer->lock); - - if (buffer->vmap_cnt) - invalidate_kernel_vmap_range(buffer->vaddr, buffer->size); - - list_for_each_entry(a, &buffer->attachments, list) { - dma_sync_sg_for_cpu(a->dev, a->table.sgl, a->table.nents, - direction); - } - mutex_unlock(&buffer->lock); - - return ret; -} - -static int dma_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, - enum dma_data_direction direction) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - struct dma_heaps_attachment *a; - - mutex_lock(&buffer->lock); - - if (buffer->vmap_cnt) - flush_kernel_vmap_range(buffer->vaddr, buffer->size); - - list_for_each_entry(a, &buffer->attachments, list) { - dma_sync_sg_for_device(a->dev, a->table.sgl, a->table.nents, - direction); - } - mutex_unlock(&buffer->lock); - - return 0; -} - -static void *dma_heap_dma_buf_vmap(struct dma_buf *dmabuf) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - void *vaddr; - - mutex_lock(&buffer->lock); - vaddr = dma_heap_buffer_vmap_get(buffer); - mutex_unlock(&buffer->lock); - - return vaddr; -} - -static void dma_heap_dma_buf_vunmap(struct dma_buf *dmabuf, void *vaddr) -{ - struct heap_helper_buffer *buffer = dmabuf->priv; - - mutex_lock(&buffer->lock); - dma_heap_buffer_vmap_put(buffer); - mutex_unlock(&buffer->lock); -} - -const struct dma_buf_ops heap_helper_ops = { - .map_dma_buf = dma_heap_map_dma_buf, - .unmap_dma_buf = dma_heap_unmap_dma_buf, - .mmap = dma_heap_mmap, - .release = dma_heap_dma_buf_release, - .attach = dma_heap_attach, - .detach = dma_heap_detach, - .begin_cpu_access = dma_heap_dma_buf_begin_cpu_access, - .end_cpu_access = dma_heap_dma_buf_end_cpu_access, - .vmap = dma_heap_dma_buf_vmap, - .vunmap = dma_heap_dma_buf_vunmap, -}; diff --git a/drivers/dma-buf/heaps/heap-helpers.h b/drivers/dma-buf/heaps/heap-helpers.h deleted file mode 100644 index 805d2df88024..000000000000 --- a/drivers/dma-buf/heaps/heap-helpers.h +++ /dev/null @@ -1,53 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * DMABUF Heaps helper code - * - * Copyright (C) 2011 Google, Inc. - * Copyright (C) 2019 Linaro Ltd. - */ - -#ifndef _HEAP_HELPERS_H -#define _HEAP_HELPERS_H - -#include -#include - -/** - * struct heap_helper_buffer - helper buffer metadata - * @heap: back pointer to the heap the buffer came from - * @dmabuf: backing dma-buf for this buffer - * @size: size of the buffer - * @priv_virt pointer to heap specific private value - * @lock mutext to protect the data in this structure - * @vmap_cnt count of vmap references on the buffer - * @vaddr vmap'ed virtual address - * @pagecount number of pages in the buffer - * @pages list of page pointers - * @attachments list of device attachments - * - * @free heap callback to free the buffer - */ -struct heap_helper_buffer { - struct dma_heap *heap; - struct dma_buf *dmabuf; - size_t size; - - void *priv_virt; - struct mutex lock; - int vmap_cnt; - void *vaddr; - pgoff_t pagecount; - struct page **pages; - struct list_head attachments; - - void (*free)(struct heap_helper_buffer *buffer); -}; - -void init_heap_helper_buffer(struct heap_helper_buffer *buffer, - void (*free)(struct heap_helper_buffer *)); - -struct dma_buf *heap_helper_export_dmabuf(struct heap_helper_buffer *buffer, - int fd_flags); - -extern const struct dma_buf_ops heap_helper_ops; -#endif /* _HEAP_HELPERS_H */ From patchwork Thu Oct 1 01:21:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313917 Delivered-To: patches@linaro.org Received: by 2002:a05:6e02:1081:0:0:0:0 with SMTP id r1csp109050ilj; Wed, 30 Sep 2020 18:22:01 -0700 (PDT) X-Received: by 2002:a17:902:9a06:b029:d0:cb2d:f276 with SMTP id v6-20020a1709029a06b02900d0cb2df276mr4793989plp.15.1601515321848; Wed, 30 Sep 2020 18:22:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601515321; cv=none; d=google.com; s=arc-20160816; b=auLk++FWMr9B63KW1nA4EO66jZS2IDiLlRxWziTiGUTc3XWiH+vzZvv9MWYK69r7+K vgzaCFy8gK6KDua/xFW14fBWfo+qW9s7w1ARwumxPaJvToaRH0209jVQwYorgar0xSvw 4onZoZ1P78wAZFPdDABo8O7BkfVpXDxplHSyB5VXoawJL51bZDjYF5uyG6+Z9OOMgwj8 dVAg23iC9/YMmbcm8nlNK1EMTaupXjEeuvMoUfrFArsMvGxmayo2wHzQ7yvZPRfwxVH5 49REAzTllQOkeXw7Uhzb7uNLU29pmVZ0HLM15o3i6r5HKqQbEmOTJx+YPPhJDdcr1pQy MTDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=1vBTIiPWzM/IlhEEYSWgJfaQ6H5IRqB1EzAsJMQfGDQ=; b=tPbNTfGB/i2O2rYThLDlteS5WerqUHKbxzZuY3rW5WB0g2yQtQgxC/OrjbkQLEW6du g4fsC5oXXuqbwQHwhok/DMzbIhtgeXG8ww4FC0UhApnurjg4osnwO1Zj/FjDlJ72TeAd s3i99Ma0xTVvw0crTMK9JzmrjrQmC1tWHj/A6McX9K4Y9lz4FGOHur6Hfncfetvau2Gd zAQtv+PMa+g2H9BiJwr2edk99C1xuS6cWAGfqiYsjoL3bwb3YOFkYrepUAUvAr79qLHe CSy3GfPQ16a+qOZ+ckspqvLR4Oi7zCaeB01Z3qh9s4t+VSRA4vZiJCRW7UkWuu2ZJCyM /lMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QIHtowFH; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id y18sor1577668pgf.26.2020.09.30.18.22.01 for (Google Transport Security); Wed, 30 Sep 2020 18:22:01 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QIHtowFH; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1vBTIiPWzM/IlhEEYSWgJfaQ6H5IRqB1EzAsJMQfGDQ=; b=QIHtowFHywcb2OA2JmyEG738gATOnYSXSQWXrzJBOgk78EntjJM/HEjEuj6obwBc2W qqlDCHm6CMMkd6JoL+/QVLamZnFiJnYRQqFQIS5cHm9v6n7lv+tQI7YDizxp79dCkiU7 H5PBDkbuGBWM4qF78HWRZITwwodwN6kL0l0sZjrXvtQhvm+e6fzeAuQ/KKAbWH0VTz+M lhwWcq+rtXVA0YTXKvUeVIoUs4xBYTMulAP6BMB7EU4NFoF+4br5eIqRwWiw9XOvk6Ax nktcGv9OsUyY9xZoA7nq6a6uLRItdRQ7NSwUSmBvYezxvmZz6QnL9L64LOw8U8zBhouW 8nJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1vBTIiPWzM/IlhEEYSWgJfaQ6H5IRqB1EzAsJMQfGDQ=; b=KfsYxmaISk0YYa2+27C02ViNtS3aloufhyI44KKHq4iagzY1QYfoApQLCDep+6mreh TCh07MOq8AgHh58jKng0lkgUOPdsKFoDL55ua4d9sbI9OarzTKz9Bj/qGoE/HuAcLMuf H5gJw1PAJDobyNezPhkahb7K4aYmAfGb3tHRxs1ZLy5ulSTYwot+Lr15n7TMglHBgfte e/eh90vmAuQuj62+igIwowXSTi88lzG1hziiN5iUzW5QFr6dUv/3XNmOf7nWNcdb5dmN 6k1nQtyXv4v2N6+x0XOJ+aro8Azz+EDdoNjg9xretP9QXC8TPTA5e0o+l7HoAdl4ybww Q9+g== X-Gm-Message-State: AOAM532SKof66j5l8nqU08wNm7IpneLx24p0/rNJ/7JP40mjvr3GHDwd HNqYCHqcx+KyBxvvgdOv3i/nZb2O X-Google-Smtp-Source: ABdhPJzBHe24b6jfAfo/83c6MDDYvzPWmon/XBZ5L2YBcnRaT9JAl84oGmaEWx3EDUJgl6GJdhmsmg== X-Received: by 2002:a63:4959:: with SMTP id y25mr4275782pgk.206.1601515321478; Wed, 30 Sep 2020 18:22:01 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id s187sm4229372pfc.134.2020.09.30.18.22.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Sep 2020 18:22:00 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH v2 4/6] dma-buf: system_heap: Allocate higher order pages if available Date: Thu, 1 Oct 2020 01:21:49 +0000 Message-Id: <20201001012151.21149-5-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201001012151.21149-1-john.stultz@linaro.org> References: <20201001012151.21149-1-john.stultz@linaro.org> MIME-Version: 1.0 While the system heap can return non-contiguous pages, try to allocate larger order pages if possible. This will allow slight performance gains and make implementing page pooling easier. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/system_heap.c | 85 ++++++++++++++++++++++------- 1 file changed, 66 insertions(+), 19 deletions(-) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index 7ec58f4e2bd3..d18937fa5b18 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -39,6 +39,14 @@ struct dma_heap_attachment { struct list_head list; }; +#define HIGH_ORDER_GFP (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \ + | __GFP_NORETRY) & ~__GFP_RECLAIM) \ + | __GFP_COMP) +#define LOW_ORDER_GFP (GFP_HIGHUSER | __GFP_ZERO | __GFP_COMP) +static gfp_t order_flags[] = {HIGH_ORDER_GFP, LOW_ORDER_GFP, LOW_ORDER_GFP}; +static const unsigned int orders[] = {8, 4, 0}; +#define NUM_ORDERS ARRAY_SIZE(orders) + static struct sg_table *dup_sg_table(struct sg_table *table) { struct sg_table *new_table; @@ -260,8 +268,11 @@ static void system_heap_dma_buf_release(struct dma_buf *dmabuf) int i; table = &buffer->sg_table; - for_each_sgtable_sg(table, sg, i) - __free_page(sg_page(sg)); + for_each_sg(table->sgl, sg, table->nents, i) { + struct page *page = sg_page(sg); + + __free_pages(page, compound_order(page)); + } sg_free_table(table); kfree(buffer); } @@ -279,6 +290,26 @@ static const struct dma_buf_ops system_heap_buf_ops = { .release = system_heap_dma_buf_release, }; +static struct page *alloc_largest_available(unsigned long size, + unsigned int max_order) +{ + struct page *page; + int i; + + for (i = 0; i < NUM_ORDERS; i++) { + if (size < (PAGE_SIZE << orders[i])) + continue; + if (max_order < orders[i]) + continue; + + page = alloc_pages(order_flags[i], orders[i]); + if (!page) + continue; + return page; + } + return NULL; +} + static int system_heap_allocate(struct dma_heap *heap, unsigned long len, unsigned long fd_flags, @@ -286,11 +317,13 @@ static int system_heap_allocate(struct dma_heap *heap, { struct system_heap_buffer *buffer; DEFINE_DMA_BUF_EXPORT_INFO(exp_info); + unsigned long size_remaining = len; + unsigned int max_order = orders[0]; struct dma_buf *dmabuf; struct sg_table *table; struct scatterlist *sg; - pgoff_t pagecount; - pgoff_t pg; + struct list_head pages; + struct page *page, *tmp_page; int i, ret = -ENOMEM; buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); @@ -302,25 +335,35 @@ static int system_heap_allocate(struct dma_heap *heap, buffer->heap = heap; buffer->len = len; - table = &buffer->sg_table; - pagecount = len / PAGE_SIZE; - if (sg_alloc_table(table, pagecount, GFP_KERNEL)) - goto free_buffer; - - sg = table->sgl; - for (pg = 0; pg < pagecount; pg++) { - struct page *page; + INIT_LIST_HEAD(&pages); + i = 0; + while (size_remaining > 0) { /* * Avoid trying to allocate memory if the process * has been killed by SIGKILL */ if (fatal_signal_pending(current)) - goto free_pages; - page = alloc_page(GFP_KERNEL | __GFP_ZERO); + goto free_buffer; + + page = alloc_largest_available(size_remaining, max_order); if (!page) - goto free_pages; - sg_set_page(sg, page, page_size(page), 0); + goto free_buffer; + + list_add_tail(&page->lru, &pages); + size_remaining -= PAGE_SIZE << compound_order(page); + max_order = compound_order(page); + i++; + } + + table = &buffer->sg_table; + if (sg_alloc_table(table, i, GFP_KERNEL)) + goto free_buffer; + + sg = table->sgl; + list_for_each_entry_safe(page, tmp_page, &pages, lru) { + sg_set_page(sg, page, PAGE_SIZE << compound_order(page), 0); sg = sg_next(sg); + list_del(&page->lru); } /* create the dmabuf */ @@ -340,14 +383,18 @@ static int system_heap_allocate(struct dma_heap *heap, /* just return, as put will call release and that will free */ return ret; } - return ret; free_pages: - for_each_sgtable_sg(table, sg, i) - __free_page(sg_page(sg)); + for_each_sgtable_sg(table, sg, i) { + struct page *p = sg_page(sg); + + __free_pages(p, compound_order(p)); + } sg_free_table(table); free_buffer: + list_for_each_entry_safe(page, tmp_page, &pages, lru) + __free_pages(page, compound_order(page)); kfree(buffer); return ret; From patchwork Thu Oct 1 01:21:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313918 Delivered-To: patches@linaro.org Received: by 2002:a05:6e02:1081:0:0:0:0 with SMTP id r1csp109064ilj; Wed, 30 Sep 2020 18:22:03 -0700 (PDT) X-Received: by 2002:aa7:9592:0:b029:13e:d13d:a054 with SMTP id z18-20020aa795920000b029013ed13da054mr4768454pfj.26.1601515323187; Wed, 30 Sep 2020 18:22:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601515323; cv=none; d=google.com; s=arc-20160816; b=QLyObsZtkxuRgrScIWWPV5DrqB8IDNSsYe9JuD6guHXNRgAA4UegfSFCqZTqU26guL hsuiKxI3wf0eyKYrY5rNSP9Sh3ZxLlsumzYYIFU7a9U0Rols2J5AJdDM9YJl+ztBJLG1 hVjIl84xYLnKhEX+jnNaGMIrvl9OQTZkz9jWM4vkq5L/CbaOqgABFZixEJ3CAGhdBfDE 06HFiu+NDoDAJj/k9zU4uiRz0oBlYntcptuiFxNvIJTcWvVPxfU9uPwaGeakRbWGLD8U EeBjyTDyZEPZBxdJxlMgul02fmw3de9s81ahvbbCYymAYWqipplsm1FLqpElkdIBanW7 +brQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=qUE/bFNhODNtHNRsHvBXEA45DqCyOEgbacD10q/Sq0U=; b=RsZL6Nq8mZnpIDLwiGoG7p4RFHtmo9pSKHUuYEwJcCXjZoGQ/cCCJDpyvZEjhqyfxa Ggiq44rqx3oqpBDmsg8NLjbXrgU6gw0TLsj2OkLetBysfCC23su8iqOMp+cqtVq40t9E zcFGjSrFalbs1nF1WGopsvP6Aid9IxFUQyIuUVik4WIzqOKVy9E2xThwudwBSxPKU7Rk kjn13MV79DUql8Ji535edoUUOptVRdcSV3deThsaWmMqATZamysFY1BFwK0YsG0s6jjc fvnEIPVienoqT42wb6Kqfc6IGX7Jr9fvjSqw8xGFCb2kPQYT4SPkra0eJZg0KX8AY5jd RHxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="Q/wAFM39"; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p24sor1827119pjo.1.2020.09.30.18.22.03 for (Google Transport Security); Wed, 30 Sep 2020 18:22:03 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="Q/wAFM39"; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qUE/bFNhODNtHNRsHvBXEA45DqCyOEgbacD10q/Sq0U=; b=Q/wAFM39Rms4Cg6R01+pxD55x29L4ewoWoUZ+S7Z7mxiKyj0RhW8lBRNyebd0Ajsym ZbWsnFfYqThP83i3+/O/No7rhMIGxfFX6rUVBre1ij8ck79HtDUre+cgDGwlT/LebKd7 Z3N6vqdd6ZmYucEAMJrXw+K3tDHLxHrDTyNJVfWV1DLgF+3nMTuE4dzlFKe0dKyXgZ/L +YAOOxe1PoUmw4iwulvRAjtdjroKFzeUaT4zyNguSy9k1VnVUyRr4qZJuGyX9ZzlMCk8 SxRTLFymBz/46QTBHNI0oHkoPII+NSPNBePPKABgP1ToD/AnhtbNl6ONJ8Y5597Es9Hc vLgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qUE/bFNhODNtHNRsHvBXEA45DqCyOEgbacD10q/Sq0U=; b=Dfm7wIdRGAJ+EDhmDT3nYmQpNn7qSAxWj5oO23cjDhtTGf3LLArkzkA+IBUN1vDou6 OfMiLLoRHeWKg8oJ4R9SbFLTEPKYcdtdJNf0vO1R8SuynK7ErkGnKZMSFH9cnOqVcSRM F12GzV5Dk8l0yDZvl4/6x1N/8sxIEDUPso2ghPqtTJnX/vjYcQwDU3hHWxA0G2vlUC0B 5nRyWSH4fYIwI4kPr/3bs0yoK4fu2wlSNh1zHnZa5Pel+a5n+FryhP/IYOS1wk8wZtNx 7GD2FyzzBG3yCZL04Z0k5hVM8qFN2Ge6FjCdLH/Vz9Np7ztRtwIO9c7RaC0Ckmq2tBSc FRkA== X-Gm-Message-State: AOAM533EnqGMUGuud3Gj8vRtLxKFLlGD8KitttlyQOo0vx6aZiwwGpQ7 Ben0GUZn76btr/F6HzwVNSANtFxl X-Google-Smtp-Source: ABdhPJyJUcXPh+NYEvxLYkahDBmDEif30rbaBRcZGNLXe+O6HphWKSvRYIFKc6clx1LMc0IZE+eKKg== X-Received: by 2002:a17:90b:950:: with SMTP id dw16mr3201939pjb.200.1601515322827; Wed, 30 Sep 2020 18:22:02 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id s187sm4229372pfc.134.2020.09.30.18.22.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Sep 2020 18:22:02 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH v2 5/6] dma-buf: system_heap: Add pagepool support to system heap Date: Thu, 1 Oct 2020 01:21:50 +0000 Message-Id: <20201001012151.21149-6-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201001012151.21149-1-john.stultz@linaro.org> References: <20201001012151.21149-1-john.stultz@linaro.org> MIME-Version: 1.0 Reuse/abuse the pagepool code from the network code to speed up allocation performance. This is similar to the ION pagepool usage, but tries to utilize generic code instead of a custom implementation. Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/Kconfig | 1 + drivers/dma-buf/heaps/system_heap.c | 32 +++++++++++++++++++++++++---- 2 files changed, 29 insertions(+), 4 deletions(-) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/Kconfig b/drivers/dma-buf/heaps/Kconfig index a5eef06c4226..f13cde4321b1 100644 --- a/drivers/dma-buf/heaps/Kconfig +++ b/drivers/dma-buf/heaps/Kconfig @@ -1,6 +1,7 @@ config DMABUF_HEAPS_SYSTEM bool "DMA-BUF System Heap" depends on DMABUF_HEAPS + select PAGE_POOL help Choose this option to enable the system dmabuf heap. The system heap is backed by pages from the buddy allocator. If in doubt, say Y. diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index d18937fa5b18..f30904345cce 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -20,6 +20,7 @@ #include #include #include +#include struct dma_heap *sys_heap; @@ -46,6 +47,7 @@ struct dma_heap_attachment { static gfp_t order_flags[] = {HIGH_ORDER_GFP, LOW_ORDER_GFP, LOW_ORDER_GFP}; static const unsigned int orders[] = {8, 4, 0}; #define NUM_ORDERS ARRAY_SIZE(orders) +struct page_pool *pools[NUM_ORDERS]; static struct sg_table *dup_sg_table(struct sg_table *table) { @@ -265,13 +267,17 @@ static void system_heap_dma_buf_release(struct dma_buf *dmabuf) struct system_heap_buffer *buffer = dmabuf->priv; struct sg_table *table; struct scatterlist *sg; - int i; + int i, j; table = &buffer->sg_table; for_each_sg(table->sgl, sg, table->nents, i) { struct page *page = sg_page(sg); - __free_pages(page, compound_order(page)); + for (j = 0; j < NUM_ORDERS; j++) { + if (compound_order(page) == orders[j]) + break; + } + page_pool_put_full_page(pools[j], page, false); } sg_free_table(table); kfree(buffer); @@ -301,8 +307,7 @@ static struct page *alloc_largest_available(unsigned long size, continue; if (max_order < orders[i]) continue; - - page = alloc_pages(order_flags[i], orders[i]); + page = page_pool_alloc_pages(pools[i], order_flags[i]); if (!page) continue; return page; @@ -407,6 +412,25 @@ static const struct dma_heap_ops system_heap_ops = { static int system_heap_create(void) { struct dma_heap_export_info exp_info; + int i; + + for (i = 0; i < NUM_ORDERS; i++) { + struct page_pool_params pp; + + memset(&pp, 0, sizeof(pp)); + pp.order = orders[i]; + pp.dma_dir = DMA_BIDIRECTIONAL; + pools[i] = page_pool_create(&pp); + + if (IS_ERR(pools[i])) { + int j; + + pr_err("%s: page pool creation failed!\n", __func__); + for (j = 0; j < i; j++) + page_pool_destroy(pools[j]); + return PTR_ERR(pools[i]); + } + } exp_info.name = "system"; exp_info.ops = &system_heap_ops; From patchwork Thu Oct 1 01:21:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 313919 Delivered-To: patches@linaro.org Received: by 2002:a05:6e02:1081:0:0:0:0 with SMTP id r1csp109080ilj; Wed, 30 Sep 2020 18:22:04 -0700 (PDT) X-Received: by 2002:a17:90a:a88:: with SMTP id 8mr4846862pjw.105.1601515324643; Wed, 30 Sep 2020 18:22:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601515324; cv=none; d=google.com; s=arc-20160816; b=gvQfmW0OhOZRZmikZxW70neNfUvieIYJYJXuaUertQ2WxZqrD8KX8EV0Se88lt0NBj LRutFNl1xEkf58yYE6iNsQBgkHPlt+O+5sMFbgK6sHxoLxRwlvn2KlMLbZa/riMK5jJq l3wpdUvrViApv+0k5zXpiMLRL4K32CBb5qKvlSKS1LvRUQHJLwbwzZXvyWPlDEjN0LrC IVejGRAt/EefqRjfWhfNORbcd2IB6JA4wyfEuvUvd7GP+bNCrIk2RSFXYUSFLGRM4g1/ 4ANqkAs95frgTyupjh/L70pa7RcJA38ZiS10ogjJkS1cW+vNrXslN0UCiNcpiUZx45sx FKOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=iykkjgQ2zEilsAYb25Oiu9fvwjjl039POyRBKs3+JVs=; b=TsPdTJEVw5hfz4fZ6grHSvKv+ii8ZlmO5PSY2HaslXmBWq1Xp3QQIj0g9ZWUEbgs5q WRC3ovIfcZHxIVURJbsAozufF5oDlr7ck+Ak7RzKOuOJ4iW9+qqn7kIGrfaZN4UJ09aF cuVgGFherBekKWp0avoCWDJ0hrGEMcH8OByarXcm6t/fbd8WS1Rx3YxJL8neYlW5d9c5 vkIfc6gzkcblcEIEb3FSN/FqrL/Gjcbi7Lhu+//HRViiqrshU96xsBtMaQjzgCUDNok+ +xqq4IMlE8jHDaw97Z2KYVtGH5hPQg6Op8yf/UCfxK4EGocrLzld7alrwcPdIPr9lL3t TiiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=sRCpZJu0; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id g23sor1804523pju.30.2020.09.30.18.22.04 for (Google Transport Security); Wed, 30 Sep 2020 18:22:04 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=sRCpZJu0; spf=pass (google.com: domain of john.stultz@linaro.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iykkjgQ2zEilsAYb25Oiu9fvwjjl039POyRBKs3+JVs=; b=sRCpZJu0ngw1dkSkZBTu+JNMJFBCS9quxA7f49WCZeKRdthWNBrWQmcK44uYaoVDUm MBJAkE1sLjU3wZ7MO21Lvong9b3wiwGiTRAjCM7Wmzbv0gwNXyLptvZ7BvggioYZFQg9 TwfcPqtMy1+7JvSS0193XtJDUcC3XiaAeua9QBJ5iXWa552vP+YfB7uLOAcgMeeebYwI 3YDNQrpdn7UgIwWl8IrZKNFukU//OULdNQvK2hceoBohxRKSganWLfoE0CMyltWcDNt5 M80ThdTByb5CX8gNLPFNA0G6v9rl0HcFGRI2fcg6/J1Lg1p9iMsV1Uq9wZNML39DXUyk 9amw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iykkjgQ2zEilsAYb25Oiu9fvwjjl039POyRBKs3+JVs=; b=CEb0kUqlnshN6F2HdG0q0yqdrlYEkf96PPDqjLUpQXgDj9mGcZJPRo2FeaQdB73i0T 8tYt/Gos4gTKvUHLvtWL+VSKKGjqfGXcvQ+5et3jYmBwxXhqa3I83qL0huJOI4u5JeLV s64jkCP0bZslv4vqlsDx2jnRREFrMD24/B95Sdlhdssjq6Yadcm+gKvdfJSALBPkuJmE tPHs6r5H3fDs9ZBYnUVgdfMp9cTvsGX/lD+e9nYDUTFRhIuxONIQ0ormS9rTzW7yOKiQ b7phBXziDuaBqOzoftyqVJH3oppyfUKRQyo8TWqWnkUbX6o009V0hDyxBG/QSJx4QQqy wrzw== X-Gm-Message-State: AOAM533vEji+cWj47eKD+JPxz3QqHpej5pRzhZzpBlUiKNmGxEbbLYoX bmkfCfShr0vDBBBVb0ClQaX8nYbl X-Google-Smtp-Source: ABdhPJzQHrrOZJj4XNEhnUWUqQA8nROXHxb9gvlg9wZaBc4JCEFl7Z24Na9GkT1QihA9kpnNMDS9Kw== X-Received: by 2002:a17:90b:1050:: with SMTP id gq16mr5152246pjb.234.1601515324285; Wed, 30 Sep 2020 18:22:04 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:680:1319:692:26ff:feda:3a81]) by smtp.gmail.com with ESMTPSA id s187sm4229372pfc.134.2020.09.30.18.22.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Sep 2020 18:22:03 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Sumit Semwal , Liam Mark , Laura Abbott , Brian Starkey , Hridya Valsaraju , Suren Baghdasaryan , Sandeep Patil , =?utf-8?q?=C3=98rjan_Eide?= , Robin Murphy , Ezequiel Garcia , Simon Ser , James Jones , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: [PATCH v2 6/6] dma-buf: heaps: Skip sync if not mapped Date: Thu, 1 Oct 2020 01:21:51 +0000 Message-Id: <20201001012151.21149-7-john.stultz@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201001012151.21149-1-john.stultz@linaro.org> References: <20201001012151.21149-1-john.stultz@linaro.org> MIME-Version: 1.0 This patch is basically a port of Ørjan Eide's similar patch for ION https://lore.kernel.org/lkml/20200414134629.54567-1-orjan.eide@arm.com/ Only sync the sg-list of dma-buf heap attachment when the attachment is actually mapped on the device. dma-bufs may be synced at any time. It can be reached from user space via DMA_BUF_IOCTL_SYNC, so there are no guarantees from callers on when syncs may be attempted, and dma_buf_end_cpu_access() and dma_buf_begin_cpu_access() may not be paired. Since the sg_list's dma_address isn't set up until the buffer is used on the device, and dma_map_sg() is called on it, the dma_address will be NULL if sync is attempted on the dma-buf before it's mapped on a device. Before v5.0 (commit 55897af63091 ("dma-direct: merge swiotlb_dma_ops into the dma_direct code")) this was a problem as the dma-api (at least the swiotlb_dma_ops on arm64) would use the potentially invalid dma_address. How that failed depended on how the device handled physical address 0. If 0 was a valid address to physical ram, that page would get flushed a lot, while the actual pages in the buffer would not get synced correctly. While if 0 is an invalid physical address it may cause a fault and trigger a crash. In v5.0 this was incidentally fixed by commit 55897af63091 ("dma-direct: merge swiotlb_dma_ops into the dma_direct code"), as this moved the dma-api to use the page pointer in the sg_list, and (for Ion buffers at least) this will always be valid if the sg_list exists at all. But, this issue is re-introduced in v5.3 with commit 449fa54d6815 ("dma-direct: correct the physical addr in dma_direct_sync_sg_for_cpu/device") moves the dma-api back to the old behaviour and picks the dma_address that may be invalid. dma-buf core doesn't ensure that the buffer is mapped on the device, and thus have a valid sg_list, before calling the exporter's begin_cpu_access. Logic and commit message originally by: Ørjan Eide Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz --- drivers/dma-buf/heaps/cma_heap.c | 10 ++++++++++ drivers/dma-buf/heaps/system_heap.c | 10 ++++++++++ 2 files changed, 20 insertions(+) -- 2.17.1 diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index 4f20f07872e5..e19320f52063 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -44,6 +44,7 @@ struct dma_heap_attachment { struct device *dev; struct sg_table table; struct list_head list; + bool mapped; }; static int cma_heap_attach(struct dma_buf *dmabuf, @@ -68,6 +69,7 @@ static int cma_heap_attach(struct dma_buf *dmabuf, a->dev = attachment->dev; INIT_LIST_HEAD(&a->list); + a->mapped = false; attachment->priv = a; @@ -101,6 +103,7 @@ static struct sg_table *cma_heap_map_dma_buf(struct dma_buf_attachment *attachme if (!dma_map_sg(attachment->dev, table->sgl, table->nents, direction)) table = ERR_PTR(-ENOMEM); + a->mapped = true; return table; } @@ -108,6 +111,9 @@ static void cma_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, struct sg_table *table, enum dma_data_direction direction) { + struct dma_heap_attachment *a = attachment->priv; + + a->mapped = false; dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); } @@ -122,6 +128,8 @@ static int cma_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, mutex_lock(&buffer->lock); list_for_each_entry(a, &buffer->attachments, list) { + if (!a->mapped) + continue; dma_sync_sg_for_cpu(a->dev, a->table.sgl, a->table.nents, direction); } @@ -141,6 +149,8 @@ static int cma_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, mutex_lock(&buffer->lock); list_for_each_entry(a, &buffer->attachments, list) { + if (!a->mapped) + continue; dma_sync_sg_for_device(a->dev, a->table.sgl, a->table.nents, direction); } diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index f30904345cce..c0d051203300 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -38,6 +38,7 @@ struct dma_heap_attachment { struct device *dev; struct sg_table *table; struct list_head list; + bool mapped; }; #define HIGH_ORDER_GFP (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \ @@ -94,6 +95,7 @@ static int system_heap_attach(struct dma_buf *dmabuf, a->table = table; a->dev = attachment->dev; INIT_LIST_HEAD(&a->list); + a->mapped = false; attachment->priv = a; @@ -128,6 +130,7 @@ static struct sg_table *system_heap_map_dma_buf(struct dma_buf_attachment *attac if (!dma_map_sg(attachment->dev, table->sgl, table->nents, direction)) return ERR_PTR(-ENOMEM); + a->mapped = true; return table; } @@ -135,6 +138,9 @@ static void system_heap_unmap_dma_buf(struct dma_buf_attachment *attachment, struct sg_table *table, enum dma_data_direction direction) { + struct dma_heap_attachment *a = attachment->priv; + + a->mapped = false; dma_unmap_sg(attachment->dev, table->sgl, table->nents, direction); } @@ -150,6 +156,8 @@ static int system_heap_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, invalidate_kernel_vmap_range(buffer->vaddr, buffer->len); list_for_each_entry(a, &buffer->attachments, list) { + if (!a->mapped) + continue; dma_sync_sg_for_cpu(a->dev, a->table->sgl, a->table->nents, direction); } @@ -170,6 +178,8 @@ static int system_heap_dma_buf_end_cpu_access(struct dma_buf *dmabuf, flush_kernel_vmap_range(buffer->vaddr, buffer->len); list_for_each_entry(a, &buffer->attachments, list) { + if (!a->mapped) + continue; dma_sync_sg_for_device(a->dev, a->table->sgl, a->table->nents, direction); }