mongodb/docs/evergreen-testing/task_generation.md

Evergreen Task Generation

Generating tasks is a way to dynamically create tasks in Evergreen builds. This is done via the 'generate.tasks' evergreen command.

Task generation allow us to do things like dynamically split a task into sub-tasks that can be run in parallel, or generate sub-tasks to run against different mongodb versions.

Task generation is typically done with the mongo-task-generator tool. Refer to its documentation for details on how it works.

Configuring a task to be generated

In order to generate a task, we typically create a placeholder task. By convention the name of these tasks should end in "_gen". Most of the time, generated tasks should inherit the gen_task_template which configures the required dependencies.

The placeholder tasks needs to have the "generate resmoke tasks" function as one of its commands. This is how the mongo-task-generator knows that the task needs to be generated. You can also add vars to the function call to configure how the task will generated. You can refer to the mongo-task-generator documentation for details on what options are available.

Once a placeholder task in defined, you can reference it just like a normal task.

The task generation process

Task generation is performed as a 2-step process.

1. The first step is to generator the configuration for the generated tasks and send that to evergreen to actually create the tasks. This is done by the version_gen task using the mongo-task-generator tool. This only needs to be done once for the entire version and rerunning this task will result in a no-op.

   The tasks will be generated in an "inactive" state. This allows us to generate all available tasks, regardless of whether they are meant to be run or not. This way if we choose to run additional tasks in the future, they will exist to be run.

   This step will also hide all the placeholder tasks into a display task called generator_tasks in each build variant. Once task generation is completed, the user should perform actions on the generated tasks instead of the placeholder tasks, we encourage this by hiding the placeholder tasks from view.

2. After the tasks have been generated, the placeholder tasks are free to run. The placeholder tasks simply find the task generated for them and mark it activated. Since generated tasks are created in the "inactive" state, this will activate any generated tasks whose placeholder task runs. This enables users to select tasks to run on the initial task selection page even though the tasks have not yet been generated.

Note: While this 2-step process allows a similar user experience to working with normal tasks, it does create a few UI quirks. For example, evergreen will hide "inactive" tasks in the UI, as a result, a task might disappear for some time after being created, until the placeholder task runs and activates it.