Step-by-step: AI and Machine Learning on Databricks

This article guides you through articles that help you learn how to build AI and LLM solutions natively on Databricks. Topics include key steps of the end-to-end AI lifecycle, from data preparation and model building to deployment, monitoring and MLOps.

Prepare your data for model training

Learn how to load and process your data for AI workloads, including data preparation for fine-tuning LLMs. How to prepare your data for model training

Feature engineering

With feature engineering available in Unity Catalog, learn how to create feature tables, track the lineage of features and discover features that others have already built.

Feature engineering in Unity Catalog

Train and register models

Learn how to use AutoML for efficient training and tuning of your ML models, and MLflow for experiment tracking.

• Train models with AutoML

• Register models to Unity Catalog

Production real-time or batch serving

Learn how to use model serving for real-time workloads or deploy MLflow models for offline inference.

• Model serving with Databricks

• Deploy models for batch inference

Bundle assets for programmatic deployment

Learn how to use Databricks Asset Bundles for efficient packaging and deployment of all data and AI assets.

Databricks Asset Bundles

End-to-end MLOps

See how you can use Databricks to combine DataOps, ModelOps and DevOps for end-to-end ML and LLM operations for your AI application.

MLOps on Databricks

Additional resources

If the outlined steps above don’t cater to your needs, a wealth of information is available in the Machine Learning documentation.