Working with Vector stores

Vector store types

Managed Knowledge Banks

A managed Knowledge Bank uses a vector store index/table that DSS creates and updates through the embedding flow. These KBs are typically created and written by an Embed Documents or Embed Dataset recipe.

Out of the box, Knowledge Banks are created with a Vector Store called Chroma. This does not require any setup, and provides good performance even for quite a large corpus.

As an alternative, other no-setup Vector Stores are available: Milvus (local), Qdrant, and FAISS.

For more advanced use cases, you may wish to use a dedicated Vector Store. Dataiku supports several third-party vector stores that require you to set up a dedicated connection beforehand:

  • Azure AI Search

  • Elasticsearch

  • OpenSearch, including AWS OpenSearch services (both managed cluster & serverless)

  • Milvus (remote)

  • Pinecone

  • pgvector (based on a PostgreSQL Connection)

  • Vertex Vector Search (based on a Google Cloud Storage Connection)

When creating the Embedding recipe, select the desired vector store type, then select your connection. You can also change the vector store type later, by editing the settings of the Knowledge Bank.

DSS provides a default name for the index (may be known as “table”, “collection” or “service” depending on the type of vector store) that you can update if needed.

If you want DSS to connect to an existing pre-populated external index, see Unmanaged Knowledge Banks.

Databricks AI Search from a dataset

Databricks AI Search offers Delta Sync indexes, which compute embeddings and keep a searchable index synchronized with a source Delta table.

You can use an unmanaged Databricks Knowledge Bank by creating a KB, selecting a Databricks connection, and specifying an existing AI Search index.

You can alternatively have DSS create the Delta Sync index and the unmanaged Knowledge Bank from a selected Databricks dataset. The index is created in the source table’s catalog and schema, using the dataset’s Databricks connection. In this case the link between the dataset and the Knowledge Bank is shown in the Flow.

To use this:

  • the source must be a Databricks dataset backed by a physical Delta table, rather than a SQL query,

  • its Databricks connection must have Allow knowledge banks enabled and the required permissions to create and use the index,

  • a Vector Search endpoint must already exist in Databricks,

  • the source table must have a column that uniquely identifies each row and a string column containing the text to index,

  • Change Data Feed (CDF) must be enabled on the source Delta table when using a standard Vector Search endpoint.

Choose a built-in Databricks embedding model or a user-defined embedding endpoint, and select the columns to expose as Knowledge Bank metadata.

Continuous sync automatically indexes changes to the Delta table. Triggered sync requests an index update after successful DSS builds of the source dataset. Storage-optimized endpoints support only Triggered sync.

When the source dataset is rebuilt, DSS can recreate the index if the underlying table or the schema of indexed columns changes. Index updates are asynchronous, so new data may not be immediately available for retrieval.

If you later change the connection, catalog, schema, or index name in the Knowledge Bank settings, the Knowledge Bank is detached from its source dataset and then behaves like a standard unmanaged Knowledge Bank.

Connections

When setting up an Elasticsearch, OpenSearch, Google Cloud Storage, PostgreSQL, Snowflake, or Databricks connection for Knowledge Bank usage, you must enable the Allow knowledge banks setting in the connection panel.

Limitations

  • You need an Elasticsearch version >=7.14.0 to store a Knowledge Bank.

  • Elasticsearch >=8.0.0 and <8.8.0 supports only embeddings of size smaller than 1024. Embedding models generating larger embedding vectors will not work.

  • Milvus (local) does not support empty values in metadata. Dataiku fills empty values with defaults depending on the type (NaN for numbers, False for booleans, and empty string for other types).

  • Milvus (local and remote) does not support adding new metadata columns to already-built Knowledge Banks created from an Embed Documents recipe. New metadata columns are ignored until the Knowledge Bank is cleared and rebuilt.

  • Milvus (local and remote) supports switching update method from Smart sync to Append only after clearing the Knowledge Bank.

  • Only Private key authentication is supported for Google Cloud Storage connections used for Knowledge bank usage.

  • Smart update methods (Smart sync and Upsert) are not supported on the following vector store types: Pinecone, AWS OpenSearch serverless.

  • Note that, after running the embedding recipe, remote vector stores might take some time to update their indexing data in their respective user interfaces.

  • pgvector supports filtering on metadata columns with names containing only alphanumeric characters and underscore _.

  • In pgvector, metadata columns with names containing only lowercase letters and underscore _ are stored in individual columns, while other columns (with names containing uppercase letters or other symbols) are nested into a json column called metadata.

  • Databricks AI Search does not support DECIMAL or TIMESTAMP_NTZ columns in indexes. Convert them to DOUBLE and TIMESTAMP, respectively, before indexing.