Generative AI and large language models (LLMs) on Databricks

This article provides an overview of generative AI on Databricks and includes links to example notebooks and demos.

What is generative AI?

Generative AI is a type of artificial intelligence focused on the ability of computers to use models to create content like images, text, code, and synthetic data.

Generative AI applications are built on top of large language models (LLMs) and foundation models.

LLMs are deep learning models that consume and train on massive datasets to excel in language processing tasks. They create new combinations of text that mimic natural language based on its training data.
Foundation models are large ML models pre-trained with the intention that they are to be fine-tuned for more specific language understanding and generation tasks. These models are utilized to discern patterns within the input data.

After these models have completed their learning processes, together they generate statistically probable outputs when prompted and they can be employed to accomplish various tasks, including:

Image generation based on existing ones or utilizing the style of one image to modify or create a new one.
Speech tasks such as transcription, translation, question/answer generation, and interpretation of the intent or meaning of text.

Important

While many LLMs or other generative AI models have safeguards, they can still generate harmful or inaccurate information.

Generative AI has the following design patterns:

Prompt Engineering: Crafting specialized prompts to guide LLM behavior
Retrieval Augmented Generation (RAG): Combining an LLM with external knowledge retrieval
Fine-tuning: Adapting a pre-trained LLM to specific data sets of domains
Pre-training: Training an LLM from scratch

Develop generative AI and LLMs on Databricks

Databricks unifies the AI lifecycle from data collection and preparation, to model development and LLMOps, to serving and monitoring. The following features are specifically optimized to facilitate the development of generative AI applications:

Unity Catalog for governance, discovery, versioning, and access control for data, features, models, and functions.
Feature engineering and serving.
MLflow for model development tracking and LLM evaluation.

Additional resources

See Retrieval Augmented Generation (RAG) on Databricks.
- See Build a Q&A chatbot with LLama2 and Databricks.
For information about using Hugging Face models on Databricks, see Hugging Face Transformers.
The databricks-ml-examples repo in Github contains example implementations of state-of-the-art (SOTA) LLMs.