Google
Cloud Platform
Amazon Web Services
Microsoft AzureSupported foundation models on Mosaic AI Model Serving
This article describes the foundation models you can serve using Mosaic AI Model Serving.
Foundation models are large, pre-trained neural networks that are trained on both large and broad ranges of data. These models are designed to learn general patterns in language, images, or other data types, and can be fine-tuned for specific tasks with additional training.
Model Serving offers flexible options for hosting and querying foundation models based on your needs:
External models: This option enables access to foundation models hosted outside of Databricks, such as those provided by OpenAI or Anthropic. These models can be centrally managed within Databricks for streamlined governance.
Provisioned throughput: Recommended for production use cases requiring performance guarantees. This option enables the deployment of fine-tuned foundation models with optimized serving endpoints.
Foundation models hosted on Databricks
Databricks hosts state-of-the-art open foundation models, like GTE-Large. These models are made available using Foundation Model APIs provisioned throughput.
Provisioned throughput
Foundation Model APIs provisioned throughput is recommended for production cases. You can create an endpoint that uses provisioned throughput to deploy fine-tuned foundation model architectures. When you use provisioned throughput the serving endpoint is optimized for foundation model workloads that require performance guarantees.
The following table summarizes the supported model architectures for provisioned throughput. Databricks recommends using pretrained foundation models in Unity Catalog for provisioned throughput workloads.
Model architecture |
Task types |
Notes |
|---|---|---|
GTE v1.5 (English) |
Embedding |
Does not generate normalized embeddings. |
BGE v1.5 (English) |
Embedding |
Access foundation models hosted outside of Databricks
Foundation models created by LLM providers, such as OpenAI and Anthropic, are also accessible on Databricks using External models. These models are hosted outside of Databricks and you can create an endpoint to query them. These endpoints can be centrally governed from Databricks, which streamlines the use and management of various LLM providers within your organization.
The following table presents a non-exhaustive list of supported models and corresponding endpoint types. You can use the listed model associations to help you configure your an endpoint for any newly released model types as they become available with a given provider. Customers are responsible for ensuring compliance with applicable model licenses.
Note
With the rapid development of LLMs, there is no guarantee that this list is up to date at all times. New model versions from the same provider are typically supported even if they are not on the list.
Model provider |
llm/v1/completions |
llm/v1/chat |
llm/v1/embeddings |
|---|---|---|---|
OpenAI** |
|
|
|
Azure OpenAI** |
|
|
|
Anthropic |
|
|
|
Cohere** |
|
|
|
Mosaic AI Model Serving |
Databricks serving endpoint |
Databricks serving endpoint |
Databricks serving endpoint |
Amazon Bedrock |
Anthropic:
Cohere:
AI21 Labs:
|
Anthropic:
Cohere:
|
Amazon:
Cohere:
|
AI21 Labs† |
|
||
Google Cloud Vertex AI |
text-bison |
|
|
** Model provider supports fine-tuned completion and chat models. To query a fine-tuned model, populate the name field of the external model configuration with the name of your fine-tuned model.
† Model provider supports custom completion models.
Create foundation model serving endpoints
To query and use foundation models in your AI applications, you must first create a model serving endpoint. Model Serving uses a unified API and UI for creating and updating foundation model serving endpoints.
To create an endpoint that serves fine-tuned variants of foundation models made available using Foundation Model APIs provisioned throughput, see Create your provisioned throughput endpoint using the REST API.
For creating serving endpoints that access foundation models made available using the External models offering, see Create an external model serving endpoint.
Query foundation model serving endpoints
After you create your serving endpoint you are able to query your foundation model. Model Serving uses a unified OpenAI-compatible API and SDK for querying foundation models. This unified experience simplifies how you experiment with and customize foundation models for production across supported clouds and providers.