From patchwork Mon May 19 14:04:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Scally X-Patchwork-Id: 891228 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FA8A27CCEB for ; Mon, 19 May 2025 14:04:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=213.167.242.64 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747663492; cv=none; b=jq540jHugy1kyptCW6Cb6pXlT6vYDtXjov02ihdr2hzGddedw64z/cOeMACmrIg5LOQMZzgtfLfQnemmNnyXBXnMR/lyMelYolaE4lWc+2J/TpPiAFBebfjB3LcSy28XIo7zOxSa4lGDgUyqAnqJuyUJEJu1lInrhgou/40EcSE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747663492; c=relaxed/simple; bh=+h/OrU8IBqpwQZE8iMiuzQOz2whERkeYemDAAK2fsEI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t2lI79tE0Q4ySjoB/KSxMP67geKDonGMKt1RckFovq0HGmCcrE7SemHsL6sMiRDsb9y0OOxGKXmyltFsB/FzNo/HxCIhlTZuWdyFF2zCdEkh/yL5RSxjULCxTMnJ8q9pH9Fbpk9w3KRmExAUZzUMvrTtU++bbTFAwU18dX2mKEk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ideasonboard.com; spf=pass smtp.mailfrom=ideasonboard.com; dkim=pass (1024-bit key) header.d=ideasonboard.com header.i=@ideasonboard.com header.b=tM2p4O5z; arc=none smtp.client-ip=213.167.242.64 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ideasonboard.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ideasonboard.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="tM2p4O5z" Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B6754D21; Mon, 19 May 2025 16:04:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1747663467; bh=+h/OrU8IBqpwQZE8iMiuzQOz2whERkeYemDAAK2fsEI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tM2p4O5z02ebRn2so7/5maEFQ2XrOEJlT1phntTg0dazyhWdYR2EGK7tfKy6xGSPB sf4MCDAXWti1ZtJ/cCVzs5Sbo4uT/zbXKYbqb2iCytV/9g+P2KWU+TKsX7RDWmwSFn n4SlyGVg2xx1nk4lECZKtQtwgae1QPFt5GYS4Er4= From: Daniel Scally To: linux-media@vger.kernel.org Cc: sakari.ailus@linux.intel.com, laurent.pinchart@ideasonboard.com, mchehab@kernel.org, Daniel Scally Subject: [PATCH 1/3] media: mc: entity: Add pipeline_started/stopped ops Date: Mon, 19 May 2025 15:04:01 +0100 Message-Id: <20250519140403.443915-2-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250519140403.443915-1-dan.scally@ideasonboard.com> References: <20250519140403.443915-1-dan.scally@ideasonboard.com> Precedence: bulk X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add two new members to struct media_entity_operations, along with new functions in media-entity.c to traverse a media pipeline and call the new operations. The new functions are intended to be used to signal to a media pipeline that it has fully started, with the entity ops allowing drivers to define some action to be taken when those conditions are met. The combination of the new functions and operations allows drivers which are part of a multi-driver pipeline to delay actually starting streaming until all of the conditions for streaming succcessfully are met across all drivers. Signed-off-by: Daniel Scally --- drivers/media/mc/mc-entity.c | 45 ++++++++++++++++++++++++++++++++++++ include/media/media-entity.h | 24 +++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c index 045590905582..e36b1710669d 100644 --- a/drivers/media/mc/mc-entity.c +++ b/drivers/media/mc/mc-entity.c @@ -1053,6 +1053,51 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe, } EXPORT_SYMBOL_GPL(__media_pipeline_entity_iter_next); +int media_pipeline_started(struct media_pipeline *pipe) +{ + struct media_pipeline_entity_iter iter; + struct media_entity *entity; + int ret; + + ret = media_pipeline_entity_iter_init(pipe, &iter); + if (ret) + return ret; + + media_pipeline_for_each_entity(pipe, &iter, entity) { + ret = media_entity_call(entity, pipeline_started); + if (ret && ret != -ENOIOCTLCMD) + goto err_notify_stopped; + } + + media_pipeline_entity_iter_cleanup(&iter); + + return ret == -ENOIOCTLCMD ? 0 : ret; + +err_notify_stopped: + media_pipeline_stopped(pipe); + return ret; +} +EXPORT_SYMBOL_GPL(media_pipeline_started); + +int media_pipeline_stopped(struct media_pipeline *pipe) +{ + struct media_pipeline_entity_iter iter; + struct media_entity *entity; + int ret; + + ret = media_pipeline_entity_iter_init(pipe, &iter); + if (ret) + return ret; + + media_pipeline_for_each_entity(pipe, &iter, entity) + media_entity_call(entity, pipeline_stopped); + + media_pipeline_entity_iter_cleanup(&iter); + + return 0; +} +EXPORT_SYMBOL_GPL(media_pipeline_stopped); + /* ----------------------------------------------------------------------------- * Links management */ diff --git a/include/media/media-entity.h b/include/media/media-entity.h index 64cf590b1134..e858326b95cb 100644 --- a/include/media/media-entity.h +++ b/include/media/media-entity.h @@ -269,6 +269,10 @@ struct media_pad { * media_entity_has_pad_interdep(). * Optional: If the operation isn't implemented all pads * will be considered as interdependent. + * @pipeline_started: Notify this entity that the pipeline it is a part of has + * been started + * @pipeline_stopped: Notify this entity that the pipeline it is a part of has + * been stopped * * .. note:: * @@ -284,6 +288,8 @@ struct media_entity_operations { int (*link_validate)(struct media_link *link); bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0, unsigned int pad1); + int (*pipeline_started)(struct media_entity *entity); + void (*pipeline_stopped)(struct media_entity *entity); }; /** @@ -1261,6 +1267,24 @@ __media_pipeline_entity_iter_next(struct media_pipeline *pipe, entity != NULL; \ entity = __media_pipeline_entity_iter_next((pipe), iter, entity)) +/** + * media_pipeline_started - Inform entities in a pipeline that it has started + * @pipe: The pipeline + * + * Iterate on all entities in a media pipeline and call their pipeline_started + * member of media_entity_operations. + */ +int media_pipeline_started(struct media_pipeline *pipe); + +/** + * media_pipeline_stopped - Inform entities in a pipeline that it has stopped + * @pipe: The pipeline + * + * Iterate on all entities in a media pipeline and call their pipeline_stopped + * member of media_entity_operations. + */ +int media_pipeline_stopped(struct media_pipeline *pipe); + /** * media_pipeline_alloc_start - Mark a pipeline as streaming * @pad: Starting pad From patchwork Mon May 19 14:04:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Scally X-Patchwork-Id: 891227 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E37E227CCEB for ; Mon, 19 May 2025 14:04:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=213.167.242.64 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747663495; cv=none; b=Hjq0imhaq9wqpwc0+uCqn/H2vvvme2lTHw/qCB0Zhl1i6etOwBIgo+S1bZwx01akyMBQaINB4dvoNFrXXJFdAjxaDDTBHYBPyjK4YYlLWg5s0LWuwXH81hlxwC5lSy1vtoU1P0HfwKb2WjG4QTDLqr1F+X2W+XR1mvR6njpk+vs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747663495; c=relaxed/simple; bh=cI3qZu4Y+nInTOuUVLPy6HSJBiDDKcYEIqj7ekFwWj8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bV0D3ucmOoIzfCovBkH0EsvV5GOlrCvFAgmhaeM9MOo428nCd+m9ylPBopUoe8D0hrViAUhMyu/zUlmzagWDv2UUJgEj/RCnIjBmTyu9HGaQCAyHLqKD2Gvq43RI6UK6v+HazHO/MiCsln+89aLxN8OE1Cay/Xs3g1asr0zyTfw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ideasonboard.com; spf=pass smtp.mailfrom=ideasonboard.com; dkim=pass (1024-bit key) header.d=ideasonboard.com header.i=@ideasonboard.com header.b=u/JUJ9UF; arc=none smtp.client-ip=213.167.242.64 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ideasonboard.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ideasonboard.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="u/JUJ9UF" Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 86EDF1116; Mon, 19 May 2025 16:04:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1747663468; bh=cI3qZu4Y+nInTOuUVLPy6HSJBiDDKcYEIqj7ekFwWj8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u/JUJ9UFWvwWk2oY219eSFXL+/tKU7UsYsAqhkiKhjbLGMELiJxCVRDVptjUTctdC iUIaJvAdmZNGzoupSvOJ/ue3JfKvRi/3lFY6zjVhSbi5Zk1b9dYA1j7dPGEpMhnh/P 0daYY96s6QJfz9Zlb9PXMlgab3G/KTLr/lo144z4= From: Daniel Scally To: linux-media@vger.kernel.org Cc: sakari.ailus@linux.intel.com, laurent.pinchart@ideasonboard.com, mchehab@kernel.org, Daniel Scally Subject: [PATCH 3/3] media: Documentation: Add documentation for media jobs Date: Mon, 19 May 2025 15:04:03 +0100 Message-Id: <20250519140403.443915-4-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250519140403.443915-1-dan.scally@ideasonboard.com> References: <20250519140403.443915-1-dan.scally@ideasonboard.com> Precedence: bulk X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a segment to mc-core.rst that explains the purpose behind the media jobs framework and how to use it. Signed-off-by: Daniel Scally --- Documentation/driver-api/media/mc-core.rst | 154 +++++++++++++++++++++ 1 file changed, 154 insertions(+) diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst index 1d010bd7ec49..53f13f857c1f 100644 --- a/Documentation/driver-api/media/mc-core.rst +++ b/Documentation/driver-api/media/mc-core.rst @@ -327,6 +327,158 @@ Call :c:func:`media_device_register()`, if media devnode isn't registered Call :c:func:`media_device_delete()` to free the media_device. Freeing is handled by the kref put handler. +Media Jobs Framework +^^^^^^^^^^^^^^^^^^^^ + +The media jobs framework exists to facilitate situations in which multiple +drivers must work together to properly operate a media pipeline in a driver +agnostic way. The archetypical example is of a memory to memory ISP that does +not include its own DMA input engine, and which must interact with the driver +for one that has been integrated. Because the DMA engine and its driver may be +different between each implementation, hardcoding calls of functions exported by +the DMA engine driver would not be appropriate. The media jobs framework allows +the drivers to define the steps that each must execute to correctly push data +through the pipeline and then schedule the sequence of steps to run in a work +queue. + +To start with each driver must acquire a reference to a +:c:type:`media_jobs_scheduler` by calling :c:func:`media_jobs_get_scheduler()`, +passing the pointer to their :c:type:`media_device`. This ensures that all of +the drivers are working with the same scheduler. Drivers must then call +:c:func:`media_jobs_add_job_setup_func()` to register a function that populates +each job with the dependencies that must be cleared to allow it to operate, and +the steps that must be carried out to execute it. For example: + +.. code-block:: c + + static void isp_driver_run_step(void *data) + { + struct isp *isp = data; + + /* + * Logic here to actually execute the necessary steps, for example we + * might configure some hardware registers. + */ + ...; + } + + static struct media_job_dep_ops ops = { + ..., + }; + + static int isp_driver_add_job_setup_func(struct media_job *job, void *data) + { + int ret; + + ret = media_jobs_add_job_dep(job, &ops, data); + if (ret) + return ret; + + ret = media_jobs_add_job_step(job, isp_driver_run_step, data, + MEDIA_JOBS_FL_STEP_ANYWHERE, 0); + if (ret) + return ret; + + return 0; + } + +The flags parameter of `media_jobs_add_job_step()` must be one of +:c:macro:`MEDIA_JOBS_FL_STEP_ANYWHERE`, :c:macro:`MEDIA_JOBS_FL_STEP_FROM_FRONT` +or :c:macro:`MEDIA_JOBS_FL_STEP_FROM_BACK`. The flag and pos parameters together +define the order of the step within the job. Steps added with +`MEDIA_JOBS_FL_STEP_ANYWHERE` will go after all steps that are added with +`MEDIA_JOBS_FL_STEP_FROM_FRONT` and all steps with `MEDIA_JOBS_FL_STEP_ANYWHERE` +that either have a lower `pos` or were previously added. They will go before all +those added with `MEDIA_JOBS_FL_STEP_FROM_BACK` and all steps with +`MEDIA_JOBS_FL_STEP_ANYWHERE` that have a higher `pos`. Steps added with +`MEDIA_JOBS_FL_STEP_FROM_FRONT` will go `pos` places from the front of the list, +and steps added with `flags` set to `MEDIA_JOBS_FL_STEP_FROM_BACK`` will go +`pos` places from the end of the list. This allows multiple drivers to quite +precisely define which steps need to be executed and what order they should be +executed in. + +Adding a step with the same `flags` and `pos` as a previously added step will +result in an error. + +The functions held in :c:type:`media_job_dep_ops` define how the media jobs +framework handles job dependencies. It is expected that there will be some hard +dependencies before a job can be executed; for example pushing a buffer of image +data through an ISP pipeline necessarily requires that an input buffer be ready +and an output buffer be ready to accept the processed data. The operations ask +the driver if the dependencies are met, tell the driver that a job has been +queued and reset the dependencies in the event the job is cancelled: + +.. code-block:: c + + struct isp { + + ...; + + struct { + struct list_head pending; + struct list_head processing; + } buffers; + } + + static bool isp_driver_check_dep(void *data) + { + struct isp *isp = data; + + /* + * Do we have a buffer queued ready to accept the ISP's output data? + */ + if (list_empty(isp->buffers.pending)) + return false; + + return true; + } + + static void isp_driver_clear_dep(void *data) + { + struct isp *isp = data; + struct buf *buf; + + /* + * We need to "consume" the buffer so that it's not also considered as + * meeting this dependency for the next attempt to queue a job + */ + buf = list_first_entry(&isp->buffers.pending, struct buf, list); + list_move_tail(&buf->list, isp->buffers.processing); + } + + static void isp_driver_reset_dep(void *data) + { + struct isp *isp = data; + struct buf *buf; + + /* + * If a queued job is cancelled then we need to return the dependency to + * its original state, which in this example means returning it to the + * pending queue. + */ + buf = list_first_entry(&isp->buffers.pending, struct buf, list); + list_move_tail(&buf->list, isp->buffers.pending); + } + + static struct media_job_dep_ops ops = { + .check_dep = isp_driver_check_dep, + .clear_dep = isp_driver_clear_dep, + .reset_dep = isp_driver_reset_dep, + }; + +The actual creation and queueing of the jobs should be done by the drivers by +calling :c:func:`media_jobs_try_queue_job()` at any time a dependency of the +job is met - for example (following the earlier example) when a buffer is queued +to either the ISP or DMA engine's driver. When all of the dependencies that are +necessary for a job to be queued are met, this function will push a job to the +scheduler's queue. + +The scheduler has a workqueue that runs the jobs. This is triggered by calls to +the :c:func:`media_jobs_run_jobs()` function, which must be called periodically +as the pipeline is running. When the streaming is finished the drivers should +shut down the workqueue and cancel the queued jobs by calling +:c:func:`media_jobs_cancel_jobs()`. + API Definitions ^^^^^^^^^^^^^^^ @@ -336,6 +488,8 @@ API Definitions .. kernel-doc:: include/media/media-entity.h +.. kernel-doc:: include/media/media-jobs.h + .. kernel-doc:: include/media/media-request.h .. kernel-doc:: include/media/media-dev-allocator.h