Runs CLI

Note

The CLI feature is unavailable on Databricks on Google Cloud as of this release.

Note

If you receive a 500-level error when making job runs CLI requests, Databricks recommends retrying requests for up to 10 minutes (with a minimum 30-second interval between retries).

You run Databricks job runs CLI subcommands by appending them to databricks runs and Databricks jobs CLI subcommands by appending them to databricks jobs. For Databricks jobs CLI subcommands, see the Jobs CLI.

Important

The Databricks job runs CLI supports calls to two versions of the Databricks Jobs REST API: versions 2.1 and 2.0. (Job runs functionality is part of the Jobs REST API.) Version 2.1 adds support for orchestration of jobs with multiple tasks; see Jobs with multiple tasks and Jobs API updates. Databricks recommends that you call version 2.1, unless you have legacy scripts that rely on version 2.0 and cannot be migrated.

Unless otherwise specified, the programmatic behaviors that are described in this article apply equally to versions 2.1 and 2.0.

Requirements to call the Jobs REST API 2.1

To set up the Databricks job runs CLI (and jobs CLI) to call the Jobs REST API 2.1, do the following:

  1. Update the CLI to version 0.16.0 or above.

  2. Do one of the following:

    • Run the command databricks jobs configure --version=2.1. This adds the setting jobs-api-version = 2.1 to the file ~/.databrickscfg on Unix, Linux, or macOS, or %USERPROFILE%\.databrickscfg on Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.1 by default.
    • Manually add the setting jobs-api-version = 2.1 to the file ~/.databrickscfg on Unix, Linux, or macOS, or %USERPROFILE%\.databrickscfg on Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.1 by default.
    • Append the option --version=2.1 (for example, databricks runs list --version=2.1) to instruct the job runs CLI to call the Jobs REST API 2.1 for that call only.

    If you take none of the preceding actions, the job runs CLI (and jobs CLI) will call the Jobs REST API 2.0 by default.

Requirements to call the Jobs REST API 2.0

To set up the Databricks job runs CLI (and jobs CLI) to call the Jobs REST API 2.0, do one of the following:

  • Use a version of the Databricks CLI below 0.16.0, or
  • Update the CLI to version 0.16.0 or above, and then do one of the following:
    • Run the command databricks jobs configure --version=2.0. This adds the setting jobs-api-version = 2.0 to the file ~/.databrickscfg on Unix, Linux, or macOS, or %USERPROFILE%\.databrickscfg on Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.0 by default.
    • Manually add the setting jobs-api-version = 2.0 to the file ~/.databrickscfg on Unix, Linux, or macOS, or %USERPROFILE%\.databrickscfg on Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.0 by default.
    • Append the option --version=2.1 (for example, databricks runs list --version=2.0) to instruct the job runs CLI to call the Jobs REST API 2.0 for that call only.

If you take none of the preceding actions, the job runs CLI (and jobs CLI) will call the Jobs REST API 2.0 by default.

Subcommands and general usage

databricks runs --help
Usage: databricks runs [OPTIONS] COMMAND [ARGS]...

  Utility to interact with jobs runs.

Options:
  -v, --version   [VERSION]
  --debug         Debug mode. Shows full stack trace on error.
  --profile TEXT  CLI connection profile to use. The default profile is
                  "DEFAULT".

  -h, --help      Show this message and exit.

Commands:
  cancel      Cancels the specified run.
  get         Gets the metadata about a run in JSON form.
  get-output  Gets the output of a run.
  list        Lists job runs.
  submit      Submits a one-time run.

Cancel a run

To display usage documentation, run databricks runs cancel --help.

databricks runs cancel --run-id 119
{}

Get information about a run

To display usage documentation, run databricks runs get --help.

Get the output of a run

To display usage documentation, run databricks runs get-output --help.

Note

When a notebook_task returns a value from a call to dbutils.notebook.exit(), Databricks limits the returned value to the first 5 MB of data. To return a larger result, you can store job results in a cloud storage service.

Get information about all runs

To display usage documentation, run databricks runs list --help.

General usage

databricks runs list --output JSON

Jobs CLI 2.1 usage notes and response example

See Runs list in Jobs API updates.

Jobs CLI 2.0 response example

If has_more returns true, information about additional runs is available. Use the --offset option to return information about runs relative to the most recent run. For example, to return information starting with the tenth most recent run, specify --offset 10.

Use the --limit option to return information about a fixed number of runs. For example, to return information for up to the next 5 runs, specify --limit 5. You can specify up to 1000 runs. If not specified, the default is 20.

Submit a one-time run

To display usage documentation, run databricks runs submit --help.

General usage

databricks runs submit --json-file submit-run.json

Jobs REST API 2.1 usage notes and request example

See Runs submit in Jobs API updates.

Jobs REST API 2.0 request and response example

submit-run.json:

{
  "run_id": 123
}