Delta Live Tables introduction

Preview

This feature is in Public Preview.

Delta Live Tables is a framework for building reliable, maintainable, and testable data processing pipelines. You define the transformations to perform on your data, and Delta Live Tables manages task orchestration, cluster management, monitoring, data quality, and error handling.

Instead of defining your data pipelines using a series of separate Apache Spark tasks, Delta Live Tables manages how your data is transformed based on a target schema you define for each processing step. You can also enforce data quality with Delta Live Tables expectations. Expectations allow you to define expected data quality and specify how to handle records that fail those expectations.

To get started with Delta Live Tables:

• Develop your first Delta Live Tables pipeline with the quickstart.

• Learn about fundamental Delta Live Tables concepts.

• Learn how to create, run, and manage pipelines with the Delta Live Tables user interface.

• Learn how to develop Delta Live Tables pipelines with Python or SQL.

• Learn how to manage data quality in your Delta Live Tables pipelines with expectations.

Learn more about Delta Live Tables:

• Use external data sources in your Delta Live Tables pipelines: Data sources

• Use the data produced by your Delta Live Tables pipelines: Publish data

• Efficiently process continually arriving data in your Delta Live Tables pipelines: Streaming data processing

• Use change data capture (CDC) processing in your Delta Live Tables pipelines: Change data capture with Delta Live Tables

• Use the Delta Live Tables API: API guide

• Use the Delta Live Tables command line interface: CLI

• Configure your Delta Live Tables pipelines: Pipeline settings

• Analyze and report on your Delta Live Tables pipelines: Querying the event log

• Run your Delta Live Tables pipelines with popular workflow orchestration tools: Workflow tool integration

• Learn how to use access control lists (ACLs) to configure permissions on your Delta Live Tables pipelines: Access control

Find answers and solutions for Delta Live Tables:

• Implement common tasks in your Delta Live Tables pipelines: Cookbook

• Review frequently asked questions and issues: FAQ

• Learn best practices to develop, manage, and run your Delta Live Tables pipelines: Best practices

• Learn how the Delta Live Tables upgrade process works and how to test your pipelines with the next system version: Upgrades