Databricks Jobs queueing and concurrency settings

This article contains instructions and recommendations for configuring advanced settings in Databricks Jobs.

Enable queueing of job runs

Note

Queueing is enabled by default for jobs created through the UI after April 15, 2024.

To prevent runs of a job from being skipped because of concurrency limits, you can enable queueing for the job. When queueing is enabled, the run is queued for up to 48 hours if resources are unavailable for a job run. When capacity is available, the job run is dequeued and run. Queued runs are displayed in the runs list for the job and the recent job runs list.

A run is queued when one of the following limits is reached:

• The maximum concurrent active runs in the workspace.

• The maximum concurrent Run Job task runs in the workspace.

• The maximum concurrent runs of the job.

Queueing is a job-level property that queues runs only for that job.

To enable or disable queueing, click Advanced settings and click the Queue toggle button in the Job details side panel.

Configure maximum concurrent runs

By default, the maximum concurrent runs for all new jobs is 1.

Click Edit concurrent runs under Advanced settings to set this job’s maximum number of parallel runs.

Databricks skips the run if the job has already reached its maximum number of active runs when attempting to start a new run.

Set this value higher than 1 to allow multiple concurrent runs of the same job. This is useful, for example, if you trigger your job on a frequent schedule and want to enable consecutive runs to overlap or trigger multiple runs that differ by their input parameters.