Manage service principals

This article explains how to create and manage service principals for your Databricks account and workspaces.

For an overview of the Databricks identity model, see Databricks identities.

To manage access for service principals, see Authentication and access control.

What is a service principal?

A service principal is an identity that you create in Databricks for use with automated tools, jobs, and applications. Service principals give automated tools and scripts API-only access to Databricks resources, providing greater security than using users or groups.

You can grant and restrict a service principal’s access to resources in the same way as you can a Databricks user. For example, you can do the following:

• Give a service principal account admin and workspace admin roles.

• Give a service principal access to data, either at the account level using Unity Catalog, or at the workspace level.

• Add a service principal to a group at both the account and workspace level, including the workspace admins group.

You can also grant Databricks users, service principals, and groups permissions to use a service principal. This allows users to run jobs as the service principal, instead of as their identity. This prevents jobs from failing if a user leaves your organization or a group is modified.

Unlike a Databricks user, a service principal is an API-only identity; it cannot be used to access the Databricks UI.

Note

Service Principals are managed within Databricks. They do not integrate with Google Cloud Service Accounts.

Databricks recommends that you enable your workspaces for identity federation. Identity federation enables you to configure service principals in the account console, and then assign them access to specific workspaces. This simplifies Databricks administration and data governance.

Important

Databricks began to enable new workspaces for identity federation and Unity Catalog automatically on March 6, 2024, with a rollout proceeding gradually across accounts. If your workspace is enabled for identity federation by default, it cannot be disabled. For more information, see Automatic enablement of Unity Catalog.

Who can manage and use service principals?

To manage service principals in Databricks, you must have one of the following: the account admin role, the workspace admin role, or the manager or user role on a service principal.

• Account admins can add service principals to the account and assign them admin roles. They can also assign service principals to workspaces, as long as those workspaces use identity federation.

• Workspace admins can add service principals to a Databricks workspace, assign them the workspace admin role, and manage access to objects and functionality in the workspace, such as the ability to create clusters or access specified persona-based environments.

• Service Principal Managers can manage roles on a service principal. The creator of a service principal becomes the service principal manager. Account admins are service principal managers on all service principals in an account.

Note

If a service principal was created before June 13, 2023, then the creator of the service principal does not have the service principal manager role by default. Ask an account admin to grant you the service principal manager role.

• Service Principal Users can run jobs as the service principal. The job runs using the identity of the service principal, instead of the identity of the job owner. For more information, see Manage identities, permissions, and privileges for Databricks Jobs.

  Service principal Users that are workspace admins can also create tokens on behalf of the service principal.

  Note

  When the RestrictWorkspaceAdmins setting on a workspace is set to ALLOW ALL, workspace admins can create a personal access token on behalf of any service principal in their workspace. See Restrict workspace admins.

For information on how to grant the service principal manager and user roles, see Roles for managing service principals.

Manage service principals in your account

Account admins can add service principals to your Databricks account using the account console.

Add service principals to your account using the account console

1. As an account admin, log in to the account console.

2. In the sidebar, click User management.

3. On the Service principals tab, click Add service principal.

4. Enter a name for the service principal.

5. Click Add.

Assign account admin roles to a service principal

1. As an account admin, log in to the account console.

2. In the sidebar, click User management.

3. On the Service principals tab, find and click the username.

4. On the Roles tab, turn on Account admin or Marketplace admin.

Assign a service principal to a workspace using the account console

To add users to a workspace using the account console, the workspace must be enabled for identity federation. Workspace admins can also assign service principals to workspaces using the workspace admin settings page. For details, see Add a service principal to a workspace using the workspace admin settings.

1. As an account admin, log in to the account console.

2. In the sidebar, click Workspaces.

3. Click your workspace name.

4. On the Permissions tab, click Add permissions.

5. Search for and select the service principal, assign the permission level (workspace User or Admin), and click Save.

Remove a service principal from a workspace using the account console

To remove service principals from a workspace using the account console, the workspace must be enabled for identity federation. When a service principal is removed from a workspace, the service principal can no longer access the workspace, however permissions are maintained on the service principal. If the service principal is later added back to the workspace, it regains its previous permissions.

1. As an account admin, log in to the account console

2. In the sidebar, click Workspaces.

3. Click your workspace name.

4. On the Permissions tab, find the service principal.

5. Click the kebab menu at the far right of the service principal row and select Remove.

6. On the confirmation dialog, click Remove.

Deactivate a service principal in your Databricks account

Account admins can deactivate service principals across a Databricks account. A deactivated service principal cannot authenticate to the Databricks account or workspaces. However, all of the service principal’s permissions and workspace objects remain unchanged. When a service principal is deactivated the following is true:

• The service principal cannot authenticate to the account or any of their workspaces from any method.

• Applications or scripts that use the tokens generated by the service principal are no longer able to access the Databricks API. The tokens remain but cannot be used to authenticate while a service principal is deactivated.

• Clusters owned by the service principal remain running.

• Scheduled jobs created by the service principal fail unless they are assigned to a new owner.

When a service principal is reactivated, it can login to Databricks with the same permissions. Databricks recommends deactivating service principals from the account instead of removing them because removing a service principal is a destructive action. A deactivated service principal’s status is labeled Inactive in the account console. You can also deactivate a service principal from a specific workspace. See Deactivate a service principal in your Databricks workspace.

You cannot deactivate a service principal using the account console. Instead, use the Account Service Principals API.

For example:

curl --netrc -X PATCH \
https://${DATABRICKS_HOST}/api/2.1/accounts/{account_id}/scim/v2/ServicePrincipals/{id} \
--header 'Content-type: application/scim+json' \
--data @update-sp.json \
| jq .

update-sp.json:

{
  "schemas": [ "urn:ietf:params:scim:api:messages:2.0:PatchOp" ],
  "Operations": [
    {
      "op": "replace",
      "path": "active",
      "value": [
        {
          "value": "false"
        }
      ]
    }
  ]
}

Remove service principals from your Databricks account

Account admins can delete service principals from a Databricks account. Workspace admins cannot. When you delete a service principal from the account, that principal is also removed from their workspaces.

Important

When you remove a service principal from the account, that service principal is also removed from their workspaces, regardless of whether or not identity federation has been enabled. We recommend that you refrain from deleting account-level service principals unless you want them to lose access to all workspaces in the account. Be aware of the following consequences of deleting service principals:

• Applications or scripts that use the tokens generated by the service principal can no longer access Databricks APIs

• Jobs owned by the service principal fail

• Clusters owned by the service principal stop

• Queries or dashboards created by the service principal and shared using the Run as Owner credential have to be assigned to a new owner to prevent sharing from failing

To remove a service principal using the account console, do the following:

1. As an account admin, log in to the account console.

2. In the sidebar, click User management.

3. On the Service principals tab, find and click the username.

4. On the Principal Information tab, click the kebab menu in the upper-right corner and select Delete.

5. In the confirmation dialog box, click Confirm delete.

Manage service principals in your workspace

Workspace admins can manage service principals in their workspaces using the workspace admin settings page.

Add a service principal to a workspace using the workspace admin settings

1. As a workspace admin, log in to the Databricks workspace.

2. Click your username in the top bar of the Databricks workspace and select Settings.

3. Click on the Identity and access tab.

4. Next to Service principals, click Manage.

5. Click Add service principal.

6. Select an existing service principal to assign to the workspace or click Add new to create a new one.

7. Click Add.

Note

If your workspace is not enabled for identity federation, you cannot assign existing account service principals to your workspace.

Assign the workspace admin role to a service principal using the workspace admin settings page

1. As a workspace admin, log in to the Databricks workspace.

2. Click your username in the top bar of the Databricks workspace and select Settings.

3. Click on the Identity and access tab.

4. Next to Groups, click Manage.

5. Select the admins system group.

6. Click Add members.

7. Select the service principal and click Confirm.

To remove the workspace admin role from a service principal, remove the service principal from the admin group.

Deactivate a service principal in your Databricks workspace

Workspace admins can deactivate service principals in a Databricks workspace. A deactivated service principal cannot access the workspace from Databricks APIs, however all of the service principal’s permissions and workspace objects remain unchanged. When a service principal is deactivated:

• The service principal cannot authenticate to the workspaces from any method.

• The service principal’s status shows as Inactive in the workspace admin setting page.

• Applications or scripts that use the tokens generated by the service principal can no longer access the Databricks API. The tokens remain but cannot be used to authenticate while a service principal is deactivated.

• Clusters owned by the service principal remain running.

• Scheduled jobs created by the service principal have to be assigned to a new owner to prevent them from failing.

When a service principal is reactivated, it can authenticate to the workspace with the same permissions. Databricks recommends deactivating service principals instead of removing them because removing a service principal is a destructive action.

1. As a workspace admin, log in to the Databricks workspace.

2. Click your username in the top bar of the Databricks workspace and select Settings.

3. Click on the Identity and access tab.

4. Next to Service principals, click Manage.

5. Select the service principal you want to deactivate.

6. Under Status, uncheck Active.

To set a service principal to active, perform the same steps, but check the checkbox instead.

Remove a service principal from a workspace using the workspace admin settings page

Removing a service principal from a workspace does not remove the service principal from the account. To remove a service principal from your account, see Remove service principals from your Databricks account.

When a service principal is removed from a workspace, the service principal can no longer access the workspace, however permissions are maintained on the service principal. If the service principal is later added back to a workspace, it regains its previous permissions.

1. As a workspace admin, log in to the Databricks workspace.

2. Click your username in the top bar of the Databricks workspace and select Settings.

3. Click on the Identity and access tab.

4. Next to Service principals, click Manage.

5. Select the service principal.

6. In the upper-right corner, click Delete.

7. Click Delete to confirm.

Manage service principals using the API

Account admins and workspace admins can manage service principals in the Databricks account and workspaces using Databricks APIs. To manage roles on a service principal using the API, see Manage service principal roles using the Databricks CLI.

Manage service principals in the account using the API

Admins can add and manage service principals in the Databricks account using the Account Service Principals API. Account admins and workspace admins invoke the API using a different endpoint URL:

• Account admins use {account-domain}/api/2.1/accounts/{account_id}/scim/v2/.

• Workspace admins use {workspace-domain}/api/2.0/account/scim/v2/.

For details, see the Account Service Principals API.

Manage service principals in the workspace using the API

Account and workspace admins can use the Workspace Assignment API to assign service principals to workspaces enabled for identity federation. The Workspace Assignment API is supported through the Databricks account and workspaces.

• Account admins use {account-domain}/api/2.0/accounts/{account_id}/workspaces/{workspace_id}/permissionassignments.

• Workspace admins use {workspace-domain}/api/2.0/preview/permissionassignments/principals/{principal_id}.

See Workspace Assignment API.

If your workspace is not enabled for identity federation, a workspace admin can use the workspace-level APIs to assign service principals to their workspaces. See Workspace Service Principals API.

Manage tokens for a service principal

Service principals can authenticate to APIs on Databricks by using Databricks OAuth tokens or Databricks personal access tokens, as follows:

• Databricks OAuth tokens can be used to authenticate to Databricks account-level and workspace-level APIs.

  • Databricks OAuth tokens that are created at the Databricks account level can be used to authenticate to Databricks account-level and workspace-level APIs.

  • Databricks OAuth tokens that are created at the Databricks workspace level can be used to authenticate only to Databricks workspace-level APIs.

• Databricks personal access tokens can be used to authenticate only to Databricks workspace-level APIs.

Manage Databricks OAuth authentication for a service principal

To authenticate to account-level and workspace-level Databricks REST APIs, account admins can use Databricks OAuth tokens for service principals. You can request an OAuth token using the client ID and a client secret for the service principal. For more information, see Authenticate access to Databricks with a service principal using OAuth (OAuth M2M).

Manage personal access tokens for a service principal

To authenticate a service principal to workspace-level APIs on Databricks only, a workspace admin that is a Service Principal User can create a Databricks personal access tokens on behalf of the service principal, as follows:

Note

When the RestrictWorkspaceAdmins setting on a workspace is set to ALLOW ALL, workspace admins can create a personal access token on behalf of any service principal in their workspace. See Restrict workspace admins.

Note

You cannot use the Databricks user interface to generate Databricks personal access tokens for Databricks service principals. This process uses Databricks CLI version 0.205 or above to generate an access token for a Databricks service principal. If you do not already have the Databricks CLI installed, see Install or update the Databricks CLI.

1. Set up authentication for the Databricks CLI, if you have not done so already. One way to set this up is by using Databricks personal access token authentication for your Databricks workspace user first. See Databricks personal access token authentication.

2. Get the application ID for the Databricks service principal, if you do not already have it available:

   1. If the admin console for your workspace is not already open, click your username in the workspace’s top bar and click Settings.

   2. Under Workspace admin, click Identity and access.

   3. Next to Service principals, click Manage.

   4. Click the name of the Databricks service principal to open its settings page. If the name is not visible, use Filter service principals to find it.

   5. On the Configurations tab, note the Application Id value.

3. Use the Databricks CLI to run the following command, which generates the access token for the Databricks service principal.

   Run the following command:

   databricks token-management create-obo-token <application-id> --lifetime-seconds <lifetime-seconds> --comment <comment> -p <profile-name>
   

   • Replace <application-id> with the application ID of the Databricks service principal.

   • --lifetime-seconds: Replace <lifetime-seconds> with the number of seconds that the access token is valid for. For example, 1 day is 86400 seconds. If the --lifetime-seconds option is not specified, the access token is set to never expire (not recommended).

   • --comment: Replace <comment> with a meaningful comment about the access token’s purpose. If the --comment option is not specified, then no comment is generated.

   • --profile-name: Replace <profile-name> with the name of a Databricks configuration profile that contains authentication information for the Databricks service principal and the target workspace. If the -p option is not specified, the Databricks CLI will attempt to find and use a configuration profile named DEFAULT.

4. In the response, copy the value of token_value, which is the access token for your Databricks service principal.

   Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token.

   If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following:

   • Enable or disable personal access token authentication for the workspace

   • Personal access token permissions

A service principal can then use its own Databricks personal access token to create additional Databricks personal access tokens for itself, as follows:

Note

You cannot use the Databricks user interface to generate Databricks personal access tokens for Databricks service principals. This process uses Databricks CLI version 0.205 or above to generate an access token for a Databricks service principal. If you do not already have the Databricks CLI installed, see Install or update the Databricks CLI.

This procedure assumes that you have already generated the first Databricks personal access token for the Databricks service principal. You use this access token to set up the Databricks CLI to authenticate the Databricks service principal so that it can then generate additional access tokens for itself. See Databricks personal access token authentication.

1. Use the Databricks CLI to run the following command, which generates another access token for the Databricks service principal.

   Run the following command:

   databricks tokens create --comment <comment> --lifetime-seconds <lifetime-seconds> -p <profile-name>
   

   • --comment: Replace <comment> with a meaningful comment about the access token’s purpose. If the --comment option is not specified, then no comment is generated.

   • --lifetime-seconds: Replace <lifetime-seconds> with the number of seconds that the access token is valid for. For example, 1 day is 86400 seconds. If the --lifetime-seconds option is not specified, the access token is set to never expire (not recommended).

   • --profile-name: Replace <profile-name> with the name of a Databricks configuration profile that contains authentication information for the Databricks service principal and the target workspace. If the -p option is not specified, the Databricks CLI will attempt to find and use a configuration profile named DEFAULT.

2. In the response, copy the value of token_value, which is the access token for the Databricks service principal.

   Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token.

   If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following:

   • Enable or disable personal access token authentication for the workspace

   • Personal access token permissions