Add email and system notifications for job events

You can monitor the runs of a job and the tasks that are part of that job by configuring notifications when a run starts, completes successfully, fails, or its duration exceeds a configured threshold. Notifications can be sent to one or more email addresses or system destinations such as Slack, Microsoft Teams, PagerDuty, or any webhook-based service.

Configure system notifications

Note

For each job or task, you can configure a maximum of three system destinations for each notification event type.
An administrator must configure system destinations. System destinations are configured by selecting Create new destination in the Edit system notifications dialog or the admin settings page.
Notifications you set at the job level are not sent when failed tasks are retried. To receive a failure notification after every failed task (including every failed retry), use task notifications instead. To add system notifications for task runs, click + Add next to Notifications in the task panel when you add or edit a job task.
A job that has completed in a Succeeded with failures state is considered to be in a successful state. To receive alerts for jobs that complete in this state, you must select Success when you configure notifications.

System notifications integrate with popular notification tools, including:

Slack
PagerDuty
Microsoft Teams
HTTP webhooks

To add one or more system notifications when runs of this job have a notifiable event such as a job start, completion, or failure:

In the Job details panel for your job, click Edit notifications.
Click Add Notification and select a system destination in Destination.
In Select a system destination, select a destination, and click the checkbox for each notification type to send to that destination.
To add another destination, click Add notification again.
Click Confirm.

Important

The content of Slack and Microsoft Teams messages might change in future releases. You should not implement clients or processing that depend on the specific content or formatting of these messages. If you require a specific schema or formatting for notifications, Databricks recommends configuring a user-defined webhook.

Configure email notifications

Note

Notifications you set at the job level are not sent when failed tasks are retried. To receive a failure notification after every failed task (including every failed retry), use task notifications instead. To add email notifications for task runs, click + Add next to Notifications in the task panel when you add or edit a job task.
A job that has completed in a Succeeded with failures state is considered to be in a successful state. To receive alerts for jobs that complete in this state, you must select Success when you configure notifications.

To add one or more email addresses to notify when runs of this job begin, complete, or fail:

In the Job details panel for your job, click Edit notifications.
Click Add Notification and select Email address in Destination.
Enter an email address and click the checkbox for each notification type to send to that address.
To enter another email address for notification, click Add notification again.
Click Confirm.

You can use email notifications to integrate with tools such as Amazon SES and SNS.

Configure notifications for slow running or late jobs

If you have configured an expected duration for a job, you can add an email or system notification if the job exceeds the configured duration. To receive a notification for jobs that exceed the duration threshold, click the checkbox for Duration Warning when you add or edit a notification.

Filter out notifications for skipped or canceled runs

You can reduce the number of notifications sent by filtering out notifications when a run is skipped or canceled. To filter notifications, check Mute notifications for skipped runs or Mute notifications for canceled runs when you add or modify email notifications or system notifications.

Note

Selecting Mute notifications for skipped runs or Mute notifications for canceled runs for a job does not filter out notifications configured for job tasks. To filter all notifications for skipped or canceled runs, you must also filter out any task-level notifications you have configured.

HTTP webhook payloads

If you are using an HTTP webhook to send notifications, the following are example payloads sent by Databricks to your configured endpoint.

Notification for a job run start event:

{
  "event_type": "jobs.on_start",
  "workspace_id": "your_workspace_id",
  "run": {
    "run_id": "run_id"
  },
  "job": {
    "job_id": "job_id",
    "name": "job_name"
  }
}

Notification for a task run start event:

{
  "event_type": "jobs.on_start",
  "workspace_id": "your_workspace_id",
  "task": {
    "task_key": "task_name"
  },
  "run": {
    "run_id": "run_id_of_task"
    "parent_run_id": "run_id_of_parent_job_run"
  },
  "job": {
    "job_id": "job_id",
    "name": "job_name"
  }
}

When configuring destinations, webhooks can be configured for the following event types:

Event code	When is it sent?
`jobs.on_start`	A run starts.
`jobs.on_success`	A run stops and completes in a successful or succeeded with failures state.
`jobs.on_failure`	A run stops in an unsuccessful state.
`jobs.on_duration_warning_threshold_exceeded`	A run has been running for more than the configured expected duration.