Parameterize jobs

This article provides an overview of using parameters with jobs and tasks.

The following are foundational concepts for understanding parameters for jobs:

• Job parameter: A key-value pair defined at the job level and pushed down to tasks. See Configure job parameters.

• Task parameter: A key-value pair or JSON array defined at the task level. See Configure task parameters.

• Dynamic value references: A syntax for references job conditions, metadata, and parameters. See What is a dynamic value reference?.

• Task values: A syntax for capturing and referencing values generated during task runs. See Use task values to pass information between tasks.

What can you do with parameters?

Add parameters to jobs and tasks for advanced use cases including the following:

• Add extensible logic to code assets.

• Conditionalize runs.

• Reference common parameters across multiple tasks.

• Use information generated in one task in another task.

• Reference metadata and state information in the job run.

What is the difference between job and task parameters?

Job parameters are key-value pairs defined at the job level. You can override the default settings for job parameters when you Run now with different parameters or trigger a job run using the REST API. Job parameters are pushed down to tasks that use key-value parameters. Other tasks can reference job parameters using dynamic value references.

Task parameters are key-value pairs of JSON arrays defined at the task level. Each task type passes task values to the configured code assets differently. For example, notebook tasks use the dbutils.widgets submodule, while Python scripts pass values as arguments to the script as if it were being called from the command line. Downstream tasks can reference task parameters from upstream tasks using dynamic value references.

Note

Some tasks do not have a dedicated Parameters field, but allow references to task values or dynamic value references within other fields. See Examples of parameterized dbt commands and Add branching logic to a job with the If/else task.

Build workflows using dynamic values

Task parameters set with static values can only be overridden by updating the task definition. Setting a static value for a job parameter is just configuring a default value, which you can override when you Run now with different parameters or trigger a job run using the REST API.

Use dynamic value references when defining task parameters to implement patterns such as the following:

• Use a job parameter as the output_table for one task and the input_table for another.

• Capture the output of a notebook query as a list and loop over it in a For each task.

• Creating forking logic based on the number of records processed using an If/else condition task.

• Refer to parameters from other tasks.

See What is a dynamic value reference?.