Develop code in Databricks notebooks

This page describes how to develop code in Databricks notebooks, including autocomplete, automatic formatting for Python and SQL, combining Python and SQL in a notebook, and tracking the notebook revision history.

Access notebook for editing

To open a notebook, use the workspace Search function or use the workspace browser to navigate to the notebook and click on the notebook’s name or icon.

Keyboard shortcuts

To display keyboard shortcuts, select Help > Keyboard shortcuts. The keyboard shortcuts available depend on whether the cursor is in a code cell (edit mode) or not (command mode).

Find and replace text

To find and replace text within a notebook, select Edit > Find and Replace. The current match is highlighted in orange and all other matches are highlighted in yellow.

To replace the current match, click Replace. To replace all matches in the notebook, click Replace All.

To move between matches, click the Prev and Next buttons. You can also press shift+enter and enter to go to the previous and next matches, respectively.

To close the find and replace tool, click or press esc.

Autocomplete

You can use Databricks autocomplete to automatically complete code segments as you type them. Databricks supports two types of autocomplete: local and server.

Local autocomplete completes words that are defined in the notebook. Server autocomplete accesses the cluster for defined types, classes, and objects, as well as SQL database and table names. To activate server autocomplete, attach your notebook to a cluster and run all cells that define completable objects.

Important

Server autocomplete in R notebooks is blocked during command execution.

To trigger autocomplete, press Tab after entering a completable object. For example, after you define and run the cells containing the definitions of MyClass and instance, the methods of instance are completable, and a list of valid completions displays when you press Tab.

SQL database and table name completion, type completion, syntax highlighting and SQL autocomplete are available in SQL cells and when you use SQL inside a Python command, such as in a spark.sql command.

Type Completion — — SQL Completion

In Databricks Runtime 7.4 and above, you can display Python docstring hints by pressing Shift+Tab after entering a completable Python object. The docstrings contain the same information as the help() function for an object.

Run selected text

You can highlight code or SQL statements in a notebook cell and run only that selection. This is useful when you want to quickly iterate on code and queries.

Highlight the lines you want to run.
Select Run > Run selected text or use the keyboard shortcut Ctrl+Shift+Enter. If no text is highlighted, Run Selected Text executes the current line.

If you are using mixed languages in a cell, you must include the %<language> line in the selection.

Run selected text also executes collapsed code, if there is any in the highlighted selection.

Special cell commands such as %run, %pip, and %sh are supported.

Run selected text limitations

You cannot use Run selected text on cells that have multiple output tabs (that is, cells where you have defined a data profile or visualization).

If you are not using the new notebook editor, Run selected text works only in edit mode (that is, when the cursor is in a code cell). If the cursor is outside the cell with the selected text, Run selected text does not work. To avoid this limitation, enable the new notebook editor.

Format code cells

Databricks provides tools that allow you to format Python and SQL code in notebook cells quickly and easily. These tools reduce the effort to keep your code formatted and help to enforce the same coding standards across your notebooks.

Format Python cells

Preview

This feature is in Public Preview.

Databricks supports Python code formatting using Black within the notebook. The notebook must be attached to a cluster with black and tokenize-rt Python packages installed, and the Black formatter executes on the cluster that the notebook is attached to.

On Databricks Runtime 11.2 and above, Databricks preinstalls black and tokenize-rt. You can use the formatter directly without needing to install these libraries.

On Databricks Runtime 11.1 and below, you must install black==22.3.0 and tokenize-rt==4.2.1 from PyPI on your notebook or cluster to use the Python formatter. You can run the following command in your notebook:

%pip install black==22.3.0 tokenize-rt==4.2.1

or install the library on your cluster.

For more details about installing libraries, see Python environment management.

How to format Python and SQL cells

You must have Can Edit permission on the notebook to format code.

You can trigger the formatter in the following ways:

Format a single cell
- Keyboard shortcut: Press Cmd+Shift+F.
- Command context menu:
  - Format SQL cell: Select Format SQL in the command context dropdown menu of a SQL cell. This menu item is visible only in SQL notebook cells or those with a %sql language magic.
  - Format Python cell: Select Format Python in the command context dropdown menu of a Python cell. This menu item is visible only in Python notebook cells or those with a %python language magic.
- Notebook Edit menu: Select a Python or SQL cell, and then select Edit > Format Cell(s).
Format multiple cells

Select multiple cells and then select Edit > Format Cell(s). If you select cells of more than one language, only SQL and Python cells are formatted. This includes those that use %sql and %python.
Format all Python and SQL cells in the notebook

Select Edit > Format Notebook. If your notebook contains more than one language, only SQL and Python cells are formatted. This includes those that use %sql and %python.

Limitations of code formatting

Black enforces PEP 8 standards for 4-space indentation. Indentation is not configurable.
Formatting embedded Python strings inside a SQL UDF is not supported. Similarly, formatting SQL strings inside a Python UDF is not supported.

Version history

Databricks notebooks maintain a history of notebook versions, allowing you to view and restore previous snapshots of the notebook. You can perform the following actions on versions: add comments, restore and delete versions, and clear version history.

You can also sync your work in Databricks with a remote Git repository.

To access notebook versions, click revision history icon in the right sidebar. The notebook revision history appears. You can also select File > Version history.

Add a comment

To add a comment to the latest version:

Click the version.
Click Save now.
In the Save Notebook Revision dialog, enter a comment.
Click Save. The notebook version is saved with the entered comment.

Restore a version

To restore a version:

Click the version.
Click Restore this revision.
Click Confirm. The selected version becomes the latest version of the notebook.

Delete a version

To delete a version entry:

Click the version.
Click the trash icon .
Click Yes, erase. The selected version is deleted from the history.

Clear version history

The version history cannot be recovered after it has been cleared.

To clear the version history for a notebook:

Select File > Clear version history.
Click Yes, clear. The notebook version history is cleared.

Code languages in notebooks

Set default language

The default language for the notebook appears next to the notebook name.

To change the default language, click the language button and select the new language from the dropdown menu. To ensure that existing commands continue to work, commands of the previous default language are automatically prefixed with a language magic command.

Mix languages

By default, cells use the default language of the notebook. You can override the default language in a cell by clicking the language button and selecting a language from the dropdown menu.

Alternately, you can use the language magic command %<language> at the beginning of a cell. The supported magic commands are: %python, %r, %scala, and %sql.

Note

When you invoke a language magic command, the command is dispatched to the REPL in the execution context for the notebook. Variables defined in one language (and hence in the REPL for that language) are not available in the REPL of another language. REPLs can share state only through external resources such as files in DBFS or objects in object storage.

Notebooks also support a few auxiliary magic commands:

%sh: Allows you to run shell code in your notebook. To fail the cell if the shell command has a non-zero exit status, add the -e option. This command runs only on the Apache Spark driver, and not the workers. To run a shell command on all nodes, use an init script.
%fs: Allows you to use dbutils filesystem commands. For example, to run the dbutils.fs.ls command to list files, you can specify %fs ls instead. For more information, see How to work with files on Databricks.
%md: Allows you to include various types of documentation, including text, images, and mathematical formulas and equations. See the next section.

SQL syntax highlighting and autocomplete in Python commands

Syntax highlighting and SQL autocomplete are available when you use SQL inside a Python command, such as in a spark.sql command.

Explore SQL cell results in Python notebooks natively using Python

You might want to load data using SQL and explore it using Python. In a Databricks Python notebook, table results from a SQL language cell are automatically made available as a Python DataFrame. The name of the Python DataFrame is _sqldf.

Note

In Python notebooks, the DataFrame _sqldf is not saved automatically and is replaced with the results of the most recent SQL cell run. To save the DataFrame, run this code in a Python cell:
```
new_dataframe_name = _sqldf
```
If the query uses a widget for parameterization, the results are not available as a Python DataFrame.
If the query uses the keywords CACHE TABLE or UNCACHE TABLE, the results are not available as a Python DataFrame.

The screenshot shows an example:

Display images

To display images stored in the FileStore, use the syntax:

%md
![test](files/image.png)

For example, suppose you have the Databricks logo image file in FileStore:

dbfs ls dbfs:/FileStore/

databricks-logo-mobile.png

When you include the following code in a Markdown cell:

the image is rendered in the cell:

Display mathematical equations

Notebooks support KaTeX for displaying mathematical formulas and equations. For example,

%md
\\(c = \\pm\\sqrt{a^2 + b^2} \\)

\\(A{_i}{_j}=B{_i}{_j}\\)

$$c = \\pm\\sqrt{a^2 + b^2}$$

\\[A{_i}{_j}=B{_i}{_j}\\]

renders as:

and

%md
\\( f(\beta)= -Y_t^T X_t \beta + \sum log( 1+{e}^{X_t\bullet\beta}) + \frac{1}{2}\delta^t S_t^{-1}\delta\\)

where \\(\delta=(\beta - \mu_{t-1})\\)

renders as:

Include HTML

You can include HTML in a notebook by using the function displayHTML. See HTML, D3, and SVG in notebooks for an example of how to do this.

Note

The displayHTML iframe is served from the domain databricksusercontent.com and the iframe sandbox includes the allow-same-origin attribute. databricksusercontent.com must be accessible from your browser. If it is currently blocked by your corporate network, it must added to an allow list.

Link to other notebooks

You can link to other notebooks or folders in Markdown cells using relative paths. Specify the href attribute of an anchor tag as the relative path, starting with a $ and then follow the same pattern as in Unix file systems:

%md
<a href="$./myNotebook">Link to notebook in same folder as current notebook</a>
<a href="$../myFolder">Link to folder in parent folder of current notebook</a>
<a href="$./myFolder2/myNotebook2">Link to nested notebook</a>