Background Task Processing

Introduction

In order to build responsive Toolkit apps, work needs to be scheduled to run in background threads. The BackgroundTaskManager handles generic background processing and is used by many of the Toolkit frameworks.

When using a ShotgunModel, for example, the model will internally create a ShotgunDataRetriever to handle background communication with Shotgun. The ShotgunDataRetriever in turn uses threads to handle its background processing and uses this BackgroundTaskManager in order to schedule the background processing.

A centralized thread pool

By default, each Shotgun Model will have its own background task manager that runs Shotgun queries in the background. If your app uses a lot of different models or data retrievers, this becomes hard to maintain and may lead to unpredictable states due to the fact that the Shotgun API isn’t thread safe.

Note

We recommend that apps using several Shotgun Models or Data retrievers use a single BackgroundTaskManager for handling of all its background processing.

In these situations, you can maintain a single BackgroundTaskManager for your app and supply it to your ShotgunModel and ShotgunDataRetriever instances when creating them. This allows for a setup where all threaded work is handled by a single thread pool and allows for efficient control and prioritization of the work that needs to be carried out.

Here is an example of how the Toolkit apps typically set this up:

# import the task manager
task_manager = sgtk.platform.import_framework("tk-framework-shotgunutils", "task_manager")


class AppDialog(QtGui.QWidget):
    """
    App main dialog
    """
    def __init__(self, parent):
        # in your main dialog init, create a background task manager
        self._task_manager = task_manager.BackgroundTaskManager(parent=self,
                                                                start_processing=True,
                                                                max_threads=2)

        # create models and request that they use the task manager
        self._model_a = shotgun_model.SimpleShotgunModel(parent=self,
                                                         bg_task_manager=self._task_manager)
        self._model_b = shotgun_model.SimpleShotgunModel(parent=self,
                                                         bg_task_manager=self._task_manager)


    def closeEvent(self, event):
        # gracefully close down threadpool
        self._task_manager.shut_down()

        # okay to close dialog
        event.accept()

Using the Task Manager directly

The task manager isn’t just a controllable building block used by the internal Toolkit libraries; you can also use it directly to control background work that your app is doing.

Simply register a task that you want it to perform and it will queue it up and execute it once a worker becomes available. You can control how many working threads you want the task manager to run in parallel and tasks can easily be prioritized, grouped and organized in a hierarchical fashion.

Class BackgroundTaskManager

Note

Import the module into your Toolkit App using the following statement:

task_manager = sgtk.platform.import_framework("tk-framework-shotgunutils", "task_manager")
class task_manager.BackgroundTaskManager(parent, start_processing=False, max_threads=8)[source]

Main task manager class. Manages a queue of tasks running them asynchronously through a pool of worker threads.

The BackgroundTaskManager class itself is reentrant but not thread-safe so its methods should only be called from the thread it is created in. Typically this would be the main thread of the application.

Signal task_completed(uid, group, result):
 

Emitted when a task has been completed. The uid parameter holds the unique id associated with the task, the group is the group that the task is associated with and the result is the data returned by the task.

Signal task_failed(uid, group, message, traceback_str):
 

Emitted when a task fails for some reason. The uid parameter holds the unique id associated with the task, the group is the group that the task is associated with, the message is a short error message and the traceback_str holds a full traceback.

Signal task_group_finished(group):
 

Emitted when all tasks in a group have finished. The group is the group that has completed.

Parameters:
  • parent (QWidget) – The parent QObject for this instance
  • start_processing – If True then processing of tasks will start immediately
  • max_threads – The maximum number of threads the task manager will use at any time.
next_group_id()[source]

Return the next available group id

Returns:A unique group id to be used for tasks that belong to the same group.
start_processing()[source]

Start processing of tasks

pause_processing()[source]

Pause processing of tasks - any currently running tasks will complete as normal.

shut_down()[source]

Shut down the task manager. This clears the task queue and gracefully stops all running threads. Completion/failure of any currently running tasks will be ignored.

add_task(cbl, priority=None, group=None, upstream_task_ids=None, task_args=None, task_kwargs=None)[source]

Add a new task to the queue. A task is a callable method/class together with any arguments that should be passed to the callable when it is called.

Parameters:
  • cbl – The callable function/class to call when executing the task
  • priority – The priority this task should be run with. Tasks with higher priority are run first.
  • group – The group this task belongs to. Task groups can be used to simplify task management (e.g. stop a whole group, be notified when a group is complete)
  • upstream_task_ids – A list of any upstream tasks that should be completed before this task is run. The results from any upstream tasks are appended to the kwargs for this task.
  • task_args – A list of unnamed parameters to be passed to the callable when running the task
  • task_kwargs – A dictionary of named parameters to be passed to the callable when running the task
Returns:

A unique id representing the task.

add_pass_through_task(priority=None, group=None, upstream_task_ids=None, task_kwargs=None)[source]

Add a pass-through task to the queue. A pass-through task doesn’t perform any work but can be useful when synchronising other tasks (e.g. pulling the results from multiple upstream tasks into a single task)

Parameters:
  • priority – The priority this task should be run with. Tasks with higher priority are run first.
  • group – The group this task belongs to. Task groups can be used to simplify task management (e.g. stop a whole group, be notified when a group is complete). A group is expressed as a string, for example ‘thumbnails’, ‘IO’ or ‘shotgun’.
  • upstream_task_ids – A list of any upstream tasks that should be completed before this task is run. The results from any upstream tasks are appended to the kwargs for this task.
  • task_kwargs – A dictionary of named parameters that will be appended to the result of the pass-through task.
Returns:

A unique id representing the task.

stop_task(task_id, stop_upstream=True, stop_downstream=True)[source]

Stop the specified task from running. If the task is already running then it will complete but the completion/failure signal will be ignored.

Parameters:
  • task_id – The id of the task to stop
  • stop_upstream – If true then all upstream tasks will also be stopped
  • stop_downstream – If true then all downstream tasks will also be stopped
stop_task_group(group, stop_upstream=True, stop_downstream=True)[source]

Stop all tasks in the specified group from running. If any tasks are already running then they will complete but their completion/failure signals will be ignored.

Parameters:
  • group – The task group to stop
  • stop_upstream – If true then all upstream tasks will also be stopped
  • stop_downstream – If true then all downstream tasks will also be stopped
stop_all_tasks()[source]

Stop all currently queued or running tasks. If any tasks are already running then they will complete but their completion/failure signals will be ignored.