LLM connections

In order to start using the LLM Mesh, an administrator first needs to define connections to LLM.

There are two kinds of connections to LLMs:

  • Hosted LLM APIs

  • Locally-running LLMs, using HuggingFace models running on GPUs.

Hosted LLM APIs

The LLM Mesh provides support for a vast number of LLM API providers in order to maximize your options for choosing your preferred LLM provider.

Anthropic

The Anthropic connection provides connection to Anthropic text models. You will need a Anthropic API key.

The Claude, Claude-instant and Claude 2 models are supported.

AWS Bedrock

The Bedrock connection provides connection to Bedrock text models. You will need:

  • An AWS account with Bedrock access enabled

  • An existing S3 connection with credentials properly setup.

The Bedrock connection provides access to the following Bedrock models:

  • The Anthropic Claude models family

  • The AI21 Labs Jurassic models family

  • The AWS Titan model

Only text completion models are supported.

Azure OpenAI

The Azure OpenAI connection provides connection to Azure OpenAI text models. You will need:

  • An Azure account with Azure OpenAI enabled

  • A deployed Azure OpenAI service

  • One or several Azure OpenAI model deployments

  • An Azure OpenAI API key

You will need to declare each Azure OpenAI model deployment, as well as the underlying model that is being deployed (for the purpose of cost computation).

Only text completion models are supported.

As of October 2023, Azure OpenAI Terms and Conditions indicate that Azure will not retain your data for enhancing its models.

Cohere

The Cohere connection provides connection to Cohere text models. You will need a Cohere API key.

The command and command-light models are supported.

Google Vertex PaLM

The Google Vertex LLM connection provides connection to Vertex PaLM text models. You will need either a service account key, or OAuth credentials.

Only the “chat-bison” model is supported.

MosaicML

The MosaicML connection provides connection to standard MosaicML completion models: mpt-7b-instruct, mpt-30b-instruct and llama2-70b-chat. You will need a MosaicML API key.

Only text completion is supported, on the standard models only.

OpenAI

The OpenAI connection provides connection to OpenAI text models (GPT 3.5 Turbo, GPT 4). You will need an OpenAI API key (not to be confused with a ChatGPT account). You will be able to select which OpenAI models are allowed

The OpenAI connection supports both text completion and embedding. Only “standard” OpenAI models are supported.

As of October 2023, OpenAI Terms and Conditions indicate that OpenAI will not retain your data for enhancing its models.

Locally-running HuggingFace models

See Running HuggingFace models