From patchwork Tue Feb 18 11:12:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philipp Stanner X-Patchwork-Id: 866639 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66E0623F279; Tue, 18 Feb 2025 11:13:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739877190; cv=none; b=M4PIREVQ04YONEiBp/N5C7OHq+780Z1KYmG9VO0vjvNPvFSQ5GRjMfpRkFMTuoV6pXINz7t0DQdoCOEnSygSlGhDERG2GARKR4ZCMElj4BsLGmwcMKXaeALKKiLJuk3SO3jze86gOSc7i/X1g8rUckpfDecHWG/Q1VXyM/VoGIg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739877190; c=relaxed/simple; bh=wHCQOUz8s89T92fB5TvJiqEYx/Hx0bOuLLu49+hLCDo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fSZjq5z2pkquUj46tHuFsQ0+DXXElCZcH8DKhJf/4atQq6Bx6NlViGj4RLHquofoKgjAaAJATCSk9Wc8MtELleCIctIjmnW+Eezr8aISnHPrrQqeOcgjzE50MKCipcInrhK0ScEqyNe6Nl7E4iLKBkWLyRpP9EyLaha565bsknI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Av5xQcp0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Av5xQcp0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 699B0C4CEE7; Tue, 18 Feb 2025 11:13:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739877189; bh=wHCQOUz8s89T92fB5TvJiqEYx/Hx0bOuLLu49+hLCDo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Av5xQcp0nEEhUsNRY+nRZq/jS84W4K0Fvh/ZaCu8DtgUgRklBoLlQQTCtNHG0Su3D a3HRxXRA21fmmJn3664wdapcQESXljQn/Bmjxo58r9zyV7FADWQIsqHuCoxgJIXy65 RHfbLl+8bjfkGNdjOe797PVEjxd/Decs7xaModCepVbuJkfO/3sVpiUxwXjvG6ccU/ QKXCVu3AnZFWiC0PrzjItCV8dz4yR85B8sF+p0MFO/DWBJlWQYA83FbepqgUTNPD3y bmyeoFjE2dbRa7ssWu/OzK2By3xXcXo7wgdyHQYPOk2nqfdbcIwNN3oFWUHlX+WdZx cGf74VN0N0yfg== From: Philipp Stanner To: Matthew Brost , Danilo Krummrich , Philipp Stanner , =?utf-8?q?Christian_K=C3=B6nig?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Sumit Semwal Cc: dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org, Philipp Stanner Subject: [PATCH v4 1/3] drm/sched: Document run_job() refcount hazard Date: Tue, 18 Feb 2025 12:12:45 +0100 Message-ID: <20250218111246.108266-3-phasta@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250218111246.108266-2-phasta@kernel.org> References: <20250218111246.108266-2-phasta@kernel.org> Precedence: bulk X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Philipp Stanner drm_sched_backend_ops.run_job() returns a dma_fence for the scheduler. That fence is signalled by the driver once the hardware completed the associated job. The scheduler does not increment the reference count on that fence, but implicitly expects to inherit this fence from run_job(). This is relatively subtle and prone to misunderstandings. This implies that, to keep a reference for itself, a driver needs to call dma_fence_get() in addition to dma_fence_init() in that callback. It's further complicated by the fact that the scheduler even decrements the refcount in drm_sched_run_job_work() since it created a new reference in drm_sched_fence_scheduled(). It does, however, still use its pointer to the fence after calling dma_fence_put() - which is safe because of the aforementioned new reference, but actually still violates the refcounting rules. Move the call to dma_fence_put() to the position behind the last usage of the fence. Document the necessity to increment the reference count in drm_sched_backend_ops.run_job(). Suggested-by: Danilo Krummrich Signed-off-by: Philipp Stanner Reviewed-by: Danilo Krummrich --- drivers/gpu/drm/scheduler/sched_main.c | 5 ++--- include/drm/gpu_scheduler.h | 19 +++++++++++++++---- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 8c36a59afb72..02af3f89099d 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -1222,15 +1222,14 @@ static void drm_sched_run_job_work(struct work_struct *w) drm_sched_fence_scheduled(s_fence, fence); if (!IS_ERR_OR_NULL(fence)) { - /* Drop for original kref_init of the fence */ - dma_fence_put(fence); - r = dma_fence_add_callback(fence, &sched_job->cb, drm_sched_job_done_cb); if (r == -ENOENT) drm_sched_job_done(sched_job, fence->error); else if (r) DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", r); + + dma_fence_put(fence); } else { drm_sched_job_done(sched_job, IS_ERR(fence) ? PTR_ERR(fence) : 0); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 6bf458dbce84..916279b5aa00 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -420,10 +420,21 @@ struct drm_sched_backend_ops { struct drm_sched_entity *s_entity); /** - * @run_job: Called to execute the job once all of the dependencies - * have been resolved. This may be called multiple times, if - * timedout_job() has happened and drm_sched_job_recovery() - * decides to try it again. + * @run_job: Called to execute the job once all of the dependencies + * have been resolved. This may be called multiple times, if + * timedout_job() has happened and drm_sched_job_recovery() decides to + * try it again. + * + * @sched_job: the job to run + * + * Returns: dma_fence the driver must signal once the hardware has + * completed the job ("hardware fence"). + * + * Note that the scheduler expects to 'inherit' its own reference to + * this fence from the callback. It does not invoke an extra + * dma_fence_get() on it. Consequently, this callback must take a + * reference for the scheduler, and additional ones for the driver's + * respective needs. */ struct dma_fence *(*run_job)(struct drm_sched_job *sched_job); From patchwork Tue Feb 18 11:12:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philipp Stanner X-Patchwork-Id: 866322 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1AA623F276; Tue, 18 Feb 2025 11:13:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739877194; cv=none; b=FmwLd+i7Sou6n/nnBIq4zQP91VIwM2epxpweM1qfEB2wAzadQwJtWvUCFIl9nDveEBsNBst36yBiF35HDCwT5Ys46QdIA6OnSDDITXh+NIGNAJQA1Wu+davuFkWu5EZNurPtQ5KRe9VVbQCxLRGCVFP2Hg7IhB0oju78xt8d2ew= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739877194; c=relaxed/simple; bh=ryBWSSH+vxnj25RO828aq+7FfD30J+Zn25JvG2s5aZM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QssIaYle7fMiMWMeyIDEDPMShNxK1ZJttBBP/uUi82flK4tbN6gSgULaplu4NcatApAya6ZxHQcAom32HsDmlQea2nozxwfFTKOSDAp5pFkar6D73fwXYWTnZifOwIcvJ4dEZNtAk/ftVwCqVo30mkC6c6miOMLuIVGTzF20n20= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DGl1DWoD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DGl1DWoD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 67E7BC4CEED; Tue, 18 Feb 2025 11:13:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739877193; bh=ryBWSSH+vxnj25RO828aq+7FfD30J+Zn25JvG2s5aZM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DGl1DWoD4yPr9flUvAElnD7BmbAhxGOWSg2uq3MVvp5CfT2MjsIjogEjO9sV+eJrC d7YGG5zAfnsfMh3lB2crsOmzQWlJHnq7iBH6hzXGfKfcEiW/9GuqnNz7SUweysVnW7 3FHDgLhzdvjkfkg0pkZN/AFrowBUTPv+CVVFWjUssENzGeJ4qPxlQ3ZN6ExyMFzAfs s+A95S81MDXoYn2ejkcKkpMLE8FZeNFU2u2j6f6/tNF6XuD3LvZuQC9bf3vu8ayIql abkX2haLB74Z2sTQIrMt8kspozJHxxN79yAOiL68J/os0GW0+rifzQbJmC9vL58YeG G8Hh6vpTWHsCw== From: Philipp Stanner To: Matthew Brost , Danilo Krummrich , Philipp Stanner , =?utf-8?q?Christian_K=C3=B6nig?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Sumit Semwal Cc: dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org Subject: [PATCH v4 2/3] drm/sched: Adjust outdated docu for run_job() Date: Tue, 18 Feb 2025 12:12:46 +0100 Message-ID: <20250218111246.108266-4-phasta@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250218111246.108266-2-phasta@kernel.org> References: <20250218111246.108266-2-phasta@kernel.org> Precedence: bulk X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The documentation for drm_sched_backend_ops.run_job() mentions a certain function called drm_sched_job_recovery(). This function does not exist. What's actually meant is drm_sched_resubmit_jobs(), which is by now also deprecated. Remove the mention of the removed function. Discourage the behavior of drm_sched_backend_ops.run_job() being called multiple times for the same job. Signed-off-by: Philipp Stanner --- include/drm/gpu_scheduler.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 916279b5aa00..3fc43c6dc405 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -421,20 +421,24 @@ struct drm_sched_backend_ops { /** * @run_job: Called to execute the job once all of the dependencies - * have been resolved. This may be called multiple times, if - * timedout_job() has happened and drm_sched_job_recovery() decides to - * try it again. + * have been resolved. + * + * The deprecated drm_sched_resubmit_jobs() (called from + * drm_sched_backend_ops.timedout_job()) can invoke this again with the + * same parameters. * * @sched_job: the job to run * - * Returns: dma_fence the driver must signal once the hardware has - * completed the job ("hardware fence"). - * * Note that the scheduler expects to 'inherit' its own reference to * this fence from the callback. It does not invoke an extra * dma_fence_get() on it. Consequently, this callback must take a * reference for the scheduler, and additional ones for the driver's * respective needs. + * + * Return: + * * On success: dma_fence the driver must signal once the hardware has + * completed the job ("hardware fence"). + * * On failure: NULL or an ERR_PTR. */ struct dma_fence *(*run_job)(struct drm_sched_job *sched_job); From patchwork Tue Feb 18 11:12:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philipp Stanner X-Patchwork-Id: 866638 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29D4723C8AF; Tue, 18 Feb 2025 11:13:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739877197; cv=none; b=RtCdkUwusHKZ2PnMUA5EoCznTZhnGqDpaWyMiov8S4p5kCifjMPQgEvvALH7Y1+c2z+QZOuvTQHUn9rFAsGMR//Cc4+5DE/vzFWvyXrVuKI7GB34lrkdGRIHzHUJGxMvkrvDqaIs7rSszsjke5im3WRfKvP3Lpt3djVgeY5gPBk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739877197; c=relaxed/simple; bh=wC5iF82xLf3tZHQoPf3DtzkHTBCPK/AWniZoE3OEjqU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GKVwrt7vFI8UQ5a062bVZ8GcJj677F2/nYh62sczZzgJX9q7nXYPIpya/b7KFxgOXMhAqRhJ/Xq8zi0LyoWwoh2nOgABfmVU5mp2bRq83YZkQkVkf+wyhakOMsNqetgi+j3dJF6mbCQId0w/tFsfkON0QKTreUMwlnnTRmcc6FY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=J92IXOQQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="J92IXOQQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CD190C4CEE2; Tue, 18 Feb 2025 11:13:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739877196; bh=wC5iF82xLf3tZHQoPf3DtzkHTBCPK/AWniZoE3OEjqU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J92IXOQQ3cH1bgD9InGpYw8e7QxUgOQ2DUm3oB2zAlgRJj3aZktI4E7zqzlOaGFrv 5tXGNeijIQuQN7QTSESCjQ5WfehwkCxWIljaHnSav2bJEYgpEvIAS8DpanautaOiUf yzcJ3pRZmDV+An52IKtglA3SCZR1IWCibgQZ+g3CldrtC+iG8x3sIP99XikzAoM7ow dLiw1pWTimc0+fjifcRXJNnX5PrGMDDtED7b22QlrXaqMjMUDE1O73tdlf32gW8h7m nf1/d5VkT+FS/DilHBR33FzUiNlznyOatDGf63/2mJLnWN2o8OS6swQToSi8Jn+Fmb K7h1v0l/5xzGQ== From: Philipp Stanner To: Matthew Brost , Danilo Krummrich , Philipp Stanner , =?utf-8?q?Christian_K=C3=B6nig?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Sumit Semwal Cc: dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org Subject: [PATCH v4 3/3] drm/sched: Update timedout_job()'s documentation Date: Tue, 18 Feb 2025 12:12:47 +0100 Message-ID: <20250218111246.108266-5-phasta@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250218111246.108266-2-phasta@kernel.org> References: <20250218111246.108266-2-phasta@kernel.org> Precedence: bulk X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 drm_sched_backend_ops.timedout_job()'s documentation is outdated. It mentions the deprecated function drm_sched_resubmit_job(). Furthermore, it does not point out the important distinction between hardware and firmware schedulers. Since firmware schedulers tyipically only use one entity per scheduler, timeout handling is significantly more simple because the entity the faulted job came from can just be killed without affecting innocent processes. Update the documentation with that distinction and other details. Reformat the docstring to work to a unified style with the other handles. Signed-off-by: Philipp Stanner --- include/drm/gpu_scheduler.h | 83 +++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 31 deletions(-) diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 3fc43c6dc405..0ee8c5a948bf 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -393,8 +393,15 @@ static inline bool drm_sched_invalidate_job(struct drm_sched_job *s_job, return s_job && atomic_inc_return(&s_job->karma) > threshold; } +/** + * enum drm_gpu_sched_stat - the scheduler's status + * + * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. + * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. + * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available anymore. + */ enum drm_gpu_sched_stat { - DRM_GPU_SCHED_STAT_NONE, /* Reserve 0 */ + DRM_GPU_SCHED_STAT_NONE, DRM_GPU_SCHED_STAT_NOMINAL, DRM_GPU_SCHED_STAT_ENODEV, }; @@ -427,6 +434,11 @@ struct drm_sched_backend_ops { * drm_sched_backend_ops.timedout_job()) can invoke this again with the * same parameters. * + * This method is called in a workqueue context - either from the + * submit_wq the driver passed through &drm_sched_init(), or, if the + * driver passed NULL, a separate, ordered workqueue the scheduler + * allocated. + * * @sched_job: the job to run * * Note that the scheduler expects to 'inherit' its own reference to @@ -446,43 +458,52 @@ struct drm_sched_backend_ops { * @timedout_job: Called when a job has taken too long to execute, * to trigger GPU recovery. * - * This method is called in a workqueue context. + * @sched_job: The job that has timed out * - * Drivers typically issue a reset to recover from GPU hangs, and this - * procedure usually follows the following workflow: + * Drivers typically issue a reset to recover from GPU hangs. + * This procedure looks very different depending on whether a firmware + * or a hardware scheduler is being used. * - * 1. Stop the scheduler using drm_sched_stop(). This will park the - * scheduler thread and cancel the timeout work, guaranteeing that - * nothing is queued while we reset the hardware queue - * 2. Try to gracefully stop non-faulty jobs (optional) - * 3. Issue a GPU reset (driver-specific) - * 4. Re-submit jobs using drm_sched_resubmit_jobs() - * 5. Restart the scheduler using drm_sched_start(). At that point, new - * jobs can be queued, and the scheduler thread is unblocked + * For a FIRMWARE SCHEDULER, each ring has one scheduler, and each + * scheduler has one entity. Hence, the steps taken typically look as + * follows: + * + * 1. Stop the scheduler using drm_sched_stop(). This will pause the + * scheduler workqueues and cancel the timeout work, guaranteeing + * that nothing is queued while the ring is being removed. + * 2. Remove the ring. The firmware will make sure that the + * corresponding parts of the hardware are resetted, and that other + * rings are not impacted. + * 3. Kill the entity and the associated scheduler. + * + * + * For a HARDWARE SCHEDULER, a scheduler instance schedules jobs from + * one or more entities to one ring. This implies that all entities + * associated with the affected scheduler cannot be torn down, because + * this would effectively also affect innocent userspace processes which + * did not submit faulty jobs (for example). + * + * Consequently, the procedure to recover with a hardware scheduler + * should look like this: + * + * 1. Stop all schedulers impacted by the reset using drm_sched_stop(). + * 3. Kill the entity the faulty job stems from. + * 4. Issue a GPU reset on all faulty rings (driver-specific). + * 5. Re-submit jobs on all schedulers impacted by re-submitting them to + * the entities which are still alive. + * 6. Restart all schedulers that were stopped in step #1 using + * drm_sched_start(). * * Note that some GPUs have distinct hardware queues but need to reset * the GPU globally, which requires extra synchronization between the - * timeout handler of the different &drm_gpu_scheduler. One way to - * achieve this synchronization is to create an ordered workqueue - * (using alloc_ordered_workqueue()) at the driver level, and pass this - * queue to drm_sched_init(), to guarantee that timeout handlers are - * executed sequentially. The above workflow needs to be slightly - * adjusted in that case: + * timeout handlers of different schedulers. One way to achieve this + * synchronization is to create an ordered workqueue (using + * alloc_ordered_workqueue()) at the driver level, and pass this queue + * as drm_sched_init()'s @timeout_wq parameter. This will guarantee + * that timeout handlers are executed sequentially. * - * 1. Stop all schedulers impacted by the reset using drm_sched_stop() - * 2. Try to gracefully stop non-faulty jobs on all queues impacted by - * the reset (optional) - * 3. Issue a GPU reset on all faulty queues (driver-specific) - * 4. Re-submit jobs on all schedulers impacted by the reset using - * drm_sched_resubmit_jobs() - * 5. Restart all schedulers that were stopped in step #1 using - * drm_sched_start() + * Return: The scheduler's status, defined by &drm_gpu_sched_stat * - * Return DRM_GPU_SCHED_STAT_NOMINAL, when all is normal, - * and the underlying driver has started or completed recovery. - * - * Return DRM_GPU_SCHED_STAT_ENODEV, if the device is no longer - * available, i.e. has been unplugged. */ enum drm_gpu_sched_stat (*timedout_job)(struct drm_sched_job *sched_job);