Set up private Git connectivity for Databricks Git folders (Repos)

Learn about and configure Git server proxy for Git folders, which enables you to proxy Git commands from Databricks Git folders to your on-premises repos served by GitHub Enterprise Server, Bitbucket Server, and GitLab self-managed.

Note

Users with a Databricks Git server proxy configured during preview should upgrade cluster permissions for best performance. See Remove global CAN_ATTACH_TO permissions.

What is Git server proxy for Databricks Git folders?

Databricks Git server proxy for Git folders is a feature that allows you to proxy Git commands from your Databricks workspace to an on-premises Git server.

Databricks Git folders (formerly Repos) represents your connected Git repos as folders. The contents of these folders are version-controlled by syncing them to the connected Git repository. By default, Git folders can synchronize only with public Git providers (like public GitHub, GitLab, Azure DevOps, and others). However, if you host your own on-premises Git server (such as GitHub Enterprise Server, Bitbucket Server , or GitLab self-managed), you must use Git server proxy with Git folders to provide Databricks access to your Git server. Your Git server must be accessible from your Databricks data plane (driver node).

Note

Currently, Databricks Git folders can contain only Databricks notebooks and sub-folders, along with a specific set of other asset types. For a current list of supported asset types, see Limits & FAQ for Git integration with Databricks Git folders.

How does Git Server Proxy for Databricks Git folders work?

Git Server Proxy for Databricks Git folders proxies Git commands from the Databricks control plane to a “proxy cluster” running in your Databricks workspace’s compute plane. In this context, the proxy cluster is a cluster configured to run a proxy service for Git commands from Databricks Git folders to your self-hosted Git repo. This proxy service receives Git commands from the Databricks control plane and forwards them to your Git server instance.

The diagram below illustrates the overall system architecture:

Important

Databricks provides an enablement notebook you can run to configure your Git server instance to proxy commands for Databricks Git folders. Get the enablement notebook on GitHub.

Currently, a Git server proxy no longer requires CAN_ATTACH_TO permission for all users. Admins with an existing proxy clusters can now modify the cluster ACL permission to enable this feature. To enable it:

1. Select Compute from the sidebar, and then click the kebab menu next to the Compute entry for the Git Server Proxy you’re running:

   

2. From the dialog, remove the Can Attach To entry for All Users:

   

How do I set up Git Server Proxy for Databricks Git folders?

This section describes how to prepare your Git server instance for Git Server Proxy for Databricks Git folders, create the proxy, and validate your configuration.

Before you begin

Before enabling the proxy, consider the following prerequisites and planning tasks:

• Your workspace has the Databricks Git folders feature enabled.

• Your Git server instance is accessible from your Databricks workspace’s compute plane VPC, and has both HTTPS and personal access tokens (PATs) enabled.

Note

Git server proxy for Databricks works in all regions supported by your VPC.

Step 1: Prepare your Git server instance

To configure your Git server instance:

1. Give the proxy cluster’s driver node access your Git server.

   Your enterprise Git server can have an allowlist of IP addresses from which access is permitted.

   1. To start, associate a static outbound IP address for traffic that originates from your proxy cluster. You can do this by configuring a NAT gateway through the the subnetwork.

   2. Add the IP address from the previous step to your Git server’s allowlist.

2. Set your Git server instance to allow HTTPS transport.

   • For GitHub Enterprise, see Which remote URL should I use in the GitHub Enterprise help.

   • For Bitbucket, go to the Bitbucket server administration page and select server settings. In the HTTP(S) SCM hosting section, enable the HTTP(S) enabled checkbox.

Step 2: Run the enablement notebook

To enable the proxy:

1. Log into your Databricks workspace as a workspace admin with access rights to create a cluster.

2. Import this notebook:

   Enable Git server proxy for Databricks Git folders for private Git server connectivity in Git folders.

3. Select “Run All” to perform the following tasks:

   • Create a single node cluster named “Databricks Git Proxy”, which does not auto-terminate. This is the “proxy cluster” that will process and forward Git commands from your Databricks workspace to your on-premises Git server.

   • Enable a feature flag that controls whether Git requests in Databricks Git folders are proxied via the cluster.

Important

You must be an admin on the workspace with access rights to create a cluster.

Note

You should be aware of the following:

• Running an additional long-running cluster to host the proxy software incurs extra DBUs. To minimize costs, the notebook configures the proxy to use a single node cluster with an inexpensive node type. However, you might want to modify the cluster options to suit your needs.

Step 3: Validate your Git server configuration

To validate your Git server configuration, try to clone a repo hosted on your private Git server via the proxy cluster. A successful clone means that you have successfully enabled Git server proxy for your workspace.

Step 4: Create proxy-enabled repos

After users configure their Git credentials, no further steps are required to create or synchronize your repos. To configure credentials and create a repo in Databricks Git folders, see Configure Git credentials & connect a remote repo to Databricks.

Remove global CAN_ATTACH_TO permissions

Admins with an existing proxy clusters can now modify the cluster ACL permission to leverage generally available Git server proxy behavior.

If you previously configured Databricks Git server proxy with CAN_ATTACH_TO privileges, use the following steps to remove these permissions:

1. Select Compute from the sidebar, and then click the kebab menu next to the Compute entry for the Git server proxy you’re running:

   

2. From the dialog, remove the Can Attach To entry for All Users:

   

Troubleshooting

Did you encounter an error while configuring Git server proxy for Databricks Git folders? Here are some common issues and ways to diagnose them more effectively.

Checklist for common problems

Before you start diagnosing an error, confirm that you’ve completed the following steps:

• Confirm that your proxy cluster is running.

• Confirm that your Databricks Git folders users have “attach to” permissions on the proxy cluster.

• Run the enablement notebook again and capture the results, if you haven’t already. If you are unable to debug the issue, Databricks Support can review the results. You can export and send the enablement notebook as a DBC archive.

Inspect logs on the proxy cluster

The file at /databricks/git-proxy/git-proxy.log on the proxy cluster contains logs that are useful for debugging purposes.

The log file should start with the line Data-plane proxy server binding to ('', 8000)… If it does not, this means that the proxy server did not start properly. Try restarting the cluster, or delete the cluster you created and run the enablement notebook again.

If the log file does start with this line, review the log statements that follow it for each Git request initiated by a Git operation in Databricks Git folders.

For example:

  do_GET: https://server-address/path/to/repo/info/refs?service=git-upload-pack 10.139.0.25 - - [09/Jun/2021 06:53:02] /
  "GET /server-address/path/to/repo/info/refs?service=git-upload-pack HTTP/1.1" 200`

Error logs written to this file can be useful to help you or Databricks Support debug issues.

Common error messages and their resolution

• Secure connection could not be established because of SSL problems

  You might see the following error:

    https://git.consult-prodigy.com/Prodigy/databricks_test: Secure connection to https://git.consult-prodigy.com/Prodigy/databricks_test could not be established because of SLL problems
  

  Often this means that you are using a repository that requires special SSL certificates. Check the content of the /databricks/git-proxy/git-proxy.log file on the proxy cluster. If it says that certificate validation failed, then you must add the certificate of authority to the system certificate chain. First, extract the root certificate (using the browser or other option) and upload it to DBFS. Then, edit the Git folders Git Proxy cluster to use the GIT_PROXY_CA_CERT_PATH environment variable to point to the root certificate file. For more information about editing cluster environment variables, see Environment variables.

  After you have completed that step, restart the cluster.

  

• Failure to clone repository with error “Missing/Invalid Git credentials”

  First, check that you have configured your Git credentials in User Settings.

  You might encounter this error:

    Error: Invalid Git credentials. Go to User Settings -> Git Integration and check that your personal access token or app password has the correct repo access.
  

  If your organization is using SAML SSO, make sure the token has been authorized (this can be done from your Git server’s Personal Access Token (PAT) management page).

Frequently asked questions

What are the security implications of the Git server proxy?

The most important things to know are:

• Proxying does not affect the security architecture of your Databricks control plane.

• You can only have one Git proxy server cluster per workspace.

Is all Databricks Git folders-related Git traffic routed through the proxy cluster, even for public Git repos?

Yes. In the current release, your Databricks workspace does not differentiate between proxied and non-proxied repos.

Does the Git proxy feature work with other Git enterprise server providers?

Databricks Git folders supports GitHub Enterprise, Bitbucket Server, Azure DevOps Server, and GitLab self-managed. Other enterprise Git server providers should work as well if they conform to common Git specifications.

Do Databricks Git folders support GPG signing of commits?

No.

Do Databricks Git folders support SSH transport for Git operations?

No. Only HTTPS is supported.

Is the use of a non-default HTTPS port on the Git server supported?

Currently, the enablement notebook assumes that your Git server uses the default HTTPS port 443. You can set the environment variable GIT_PROXY_CUSTOM_HTTP_PORT to overwrite the port value with a preferred one.

Can you share one proxy for multiple workspaces or do you need one proxy cluster per workspace?

You need one proxy cluster per Databricks workspace.

Does the proxy work with legacy single-notebook versioning?

No, the proxy does not work with legacy single-notebook versioning. Users must migrate to Databricks Git folders versioning.

Can Databricks hide Git server URLs that are proxied? Could users enter the original Git server URLs rather than proxied URLs?

Yes to both questions. Users do not need to adjust their behavior for the proxy. With the current proxy implementation, all Git traffic for Databricks Git folders is routed through the proxy. Users enter the normal Git repo URL such as https://git.company.com/org/repo-name.git.

How often will users work with the Git URLs?

Typically a user would just add the Git URL when they create a new repo or check out an existing repo that they have not already checked out.

Does the feature transparently proxy authentication data to the Git server?

Yes, the proxy uses the user account’s Git server token to authenticate to the Git server.

Is there Databricks access to Git server code?

The Databricks proxy service accesses the Git repository on the Git server using the user-provided credential and synchronizes any code files in the repository with the repo. Access is restricted by the permissions specified in the user-provided personal access token (PAT).