LLM connections¶
In order to start using the LLM Mesh, an administrator first needs to define connections to LLM models.
There are two kinds of connections to LLM models:
Hosted LLM APIs
Locally-running LLM models, using HuggingFace models running on GPUs.
Hosted LLM APIs¶
The LLM Mesh provides support for a vast number of LLM API providers in order to maximize your options for choosing your preferred LLM provider.
Anthropic¶
The Anthropic connection provides connection to Anthropic text models. You will need a Anthropic API key.
The Claude Opus 5.5, Claude Sonnet 5.5, Claude Fable 5.1, and Claude Haiku 4.5 models are supported.
AWS Bedrock¶
The Bedrock connection provides access to models through Amazon Bedrock. You will need:
An AWS account with Bedrock access enabled
An existing S3 connection with credentials properly setup.
The Bedrock connection provides access to the following Bedrock model families:
The Anthropic Claude 4.5 and Claude 5 models
The OpenAI GPT 5 and GPT 6 models
The Amazon Nova models, Titan v2 text embeddings, and Nova 2 multimodal embeddings
The Cohere Embed and Rerank models
The Google Gemma 3 and 4 models
The Mistral models family
The Meta Llama 3 models
The DeepSeek and xAI Grok models
The Stability AI Stable Image Core and Stable Image Ultra models
Text completion, chat completion, image generation, text/image embedding, and reranking models are supported.
Inference profile¶
A cross-region inference profile allows you to distribute traffic across different AWS regions, therefore increasing throughput and resilience. When provided, an inference profile applies to all enabled models, including custom models. However, the inference profile is not applied to any custom models with an application inference profile ARN as the id.
Note that some models/regions require an inference profile to work on AWS’ side, and that Bedrock’s support for them varies by model/region. See this list of supported inference profiles by region and model. See the AWS documentation for more information.
Bedrock Guardrail¶
You may use an AWS Bedrock Guardrail created on the AWS console.
To do so, enable “Bedrock Guardrail”, specify the Identifier and Version information (version is either DRAFT or a number), and check that the region on the connection page matches the region where the Guardrail was created.
The Bedrock Guardrail is then applied to all models of the connection, including custom models.
Only one Bedrock Guardrail can be set on a given Bedrock connection. For complex filtering, you can combine different types of filters and checks in your Guardrail on the AWS Bedrock console.
AWS SageMaker LLM¶
The SageMaker LLM connection allows connecting to some completion and summarization models deployed as SageMaker endpoints. You will need an existing SageMaker connection.
The following model families have built-in handling modes:
The Anthropic Claude models family
The Amazon Nova and Titan models families
The Cohere Command models family
The Meta Llama 3 models family
Hugging Face models
Other models and endpoints can be configured with a custom query template.
Microsoft Foundry¶
The Microsoft Foundry connection provides connectivity to a wide catalog of models deployed through the Microsoft Foundry resource. You will need:
An Azure account with an Microsoft Foundry resource
A model deployed from the Microsoft Foundry Model Catalog
The Endpoint for your Microsoft Foundry resource
An API Key for your resource
You will need to declare each model by its Deployment Name, which you specify when deploying a model from the catalog. Text completion, chat completion, image generation, text embedding, and reranking models are supported.
Azure OpenAI¶
The Azure OpenAI connection provides connection to Azure OpenAI text and image models. You will need:
An Azure account with Azure OpenAI enabled
A deployed Azure OpenAI service
One or several Azure OpenAI model deployments
An Azure OpenAI API key
You will need to declare each Azure OpenAI model deployment, as well as the underlying model that is being deployed (for the purpose of cost computation).
Text completion, chat completion, image generation and text embedding models are supported.
As of October 2023, Azure OpenAI Terms and Conditions indicate that Azure will not retain your data for enhancing its models.
For Azure OpenAI connections made through an APIM gateway, there are two available options:
Custom headers can be provided in the Azure OpenAI connection, e.g. to set an APIM subscription key
For more complex APIM setups, the Azure OpenAI APIM Custom LLM plugin is available on demand through the Dataiku plugin store, e.g. for setting custom OAuth scopes, and dynamic correlation IDs
Azure LLM¶
The Azure LLM connection provides connectivity for models deployed with Azure Machine Learning (also known as Azure ML or Azure AI | Machine Learning Studio) or Azure AI Studio.
You will need:
An Azure account with Azure ML or Azure AI Studio enabled
If using Azure AI Studio, a deployment on a serverless endpoint
If using Azure ML, a serverless endpoint
In both cases, the Key provided by Azure
You will need to declare each Azure ML Endpoint or Azure AI Studio Deployment, with the Target URI as provided by Azure.
The serverless endpoints should conform to the Open AI v1 API Reference. Chat and Embbeddings endpoints are supported.
Cohere¶
The Cohere connection provides connection to Cohere text models. You will need a Cohere API key.
The Command R and Command R+ models are supported.
Databricks Mosaic AI (previously MosaicML)¶
The Databricks Mosaic AI connection provides connection to Databricks Foundation Model APIs. You will need an existing Databricks Model Deployment connection.
The supported models are:
GPT-5.5 and GPT-5.4 mini and nano
Claude Opus 5, Claude Sonnet 5, Claude Fable 5.1, and Claude Haiku 4.5
Gemini 3.5 Flash
Llama 4 Maverick
GTE Large En (text embedding)
Google Vertex Generative AI¶
The Google Vertex LLM connection provides connection to Vertex AI models. You will need:
a service account key, or OAuth credentials
at least the Vertex AI Service Agent role
The Gemini 3.8 Flash, Gemini 3.5 Flash-Lite, Gemini 3.1 Pro, and Claude 5 models are supported. Gemini Embedding 2, Text Embedding 005, and multilingual and multimodal embedding models are also supported.
The Gemini 2.5 Flash Image and Gemini 3 image generation models are also supported.
Mistral AI¶
The Mistral AI connection provides connection to Mistral AI text models. You will need a Mistral AI API key.
The supported models are:
Mistral Small 4
Mistral Medium 3.5
Mistral Large 3
Codestral
Ministral 3
Mistral Embed
NVIDIA NIM¶
The NVIDIA NIM connection allows you to leverage NVIDIA NIM microservices for optimized inference. You can connect to models hosted in the NVIDIA API catalog, self-hosted on-premises, or in the cloud.
Depending on your deployment, you will need:
An NVIDIA AI Enterprise subscription or NVIDIA Developer Program membership.
A NIM API Key (for hosted models).
A running NIM service (for self-hosted deployments).
The connection supports:
Text and Multimodal Chat Completion models.
Text Embedding models.
Streaming and tool calling for compatible models.
Because NIMs can be self-hosted via a dedicated Kubernetes operator, the setup involves specific plugin configurations and deployment steps. See NVIDIA NIM for the full setup guide and deployment instructions.
OpenAI¶
The OpenAI connection provides connection to OpenAI text models (GPT-6.1 Sol, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Terra, GPT-4.1, o1, o3, and o4-mini) and the image generation model GPT Image 2. You will need an OpenAI API key (not to be confused with a ChatGPT account). You can select which OpenAI models are allowed.
The OpenAI connection supports text completion, image generation, and text embedding. The text-embedding-3 models are supported.
Snowflake Cortex¶
The Snowflake Cortex connection provides connection to some Snowflake Cortex text models. You will need an existing Snowflake connection.
The following chat models are supported:
Claude Opus 5, Claude Sonnet 5, and Claude Haiku 4.5
GPT-5.4, GPT-5.4 mini, and GPT-5.4 nano
The Snowflake Arctic Embed M, E5 Base v2, and NV Embed QA 4 text embedding models are also supported.
Stability AI¶
The Stability AI connection provides connection to image generation models. You will need a Stability AI API key.
The following image generation models are supported:
Stable Image Core
Stable Image Ultra
Stable Diffusion 3.5 Medium
Stable Diffusion 3.5 Large
Stable Diffusion 3.5 Large Turbo