DSS’s Model cache

The model cache

DSS has its own (optional) managed cache to store models from HuggingFace.

If enabled at the connection level, the cache is automatically populated when using the LLM Mesh with pre-trained models from HuggingFace. Models are downloaded from HuggingFace hub.

View the model cache content in: Settings > Misc. You can also delete, export or import models.

Import and export models

If your DSS instance does not have access to HuggingFace (huggingface.co), you can manually import a model archive, typically one exported from the model cache of another DSS design or automation node with network access.

Build your own model archive to import

Note

It is simpler (and recommended) to import a model that was previously exported by DSS when that is possible, instead of creating an archive manually.

If you want to manually create an archive it should contain the following structure:

  • a root folder
    • a folder named “model” containing the HuggingFace model repo content

    • a file named “model_metadata.json”

To retrieve the model folder content from HuggingFace hub:

git lfs install
git clone https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

Example of a model archive content:

sentence-transformers_2fall-MiniLM-L6-v2 ← folder at the root (its name is not important)
├── model ← a folder named "model" containing the HuggingFace model repo content
│   ├── 1_Pooling
│   │   └── config.json
│   ├── README.md
│   ├── config.json
│   ├── config_sentence_transformers.json
│   ├── data_config.json
│   ├── modules.json
│   ├── pytorch_model.bin
│   ├── sentence_bert_config.json
│   ├── special_tokens_map.json
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   ├── train_script.py
│   └── vocab.txt
└── model_metadata.json ← a file named "model_metadata.json"

The model_metadata.json file should have the following schema:

{
    "commitHash": "7dbbc90392e2f80f3d3c277d6e90027e55de9125",
    "downloadDate": 1698300636139,
    "downloadedBy": "admin",
    "lastDssUsage": 1699570884724,
    "lastModified": "2022-11-07T08:44:33.000Z",
    "lastUsedBy": "admin",
    "libraryName": "sentence-transformers",
    "modelDefinition": {
        "folderName": "sentence-transformers_2fall-MiniLM-L6-v2",
        "key": "hf@sentence-transformers/all-MiniLM-L6-v2",
        "modelId": "sentence-transformers/all-MiniLM-L6-v2",
        "revision": "main"
    },
    "pipelineName": "sentence-similarity",
    "sizeInBytes": 91652688,
    "taggedLanguages": [
        "en"
    ],
    "tags": [
        "sentence-transformers",
        "pytorch",
        "tf",
        "rust",
        "bert",
        "feature-extraction",
        "sentence-similarity",
        "en",
        "dataset:s2orc",
        "dataset:flax-sentence-embeddings/stackexchange_xml",
        ...
        "arxiv:2104.08727",
        "arxiv:1704.05179",
        "arxiv:1810.09305",
        "license:apache-2.0",
        "endpoints_compatible",
        "has_space",
        "region:us"
    ],
    "url": "https://huggingface.co/sentence-transformers%2Fall-MiniLM-L6-v2/tree/main",
    "version": 0
}

Most of these fields can be retrieved from the HuggingFace model repository.

The important ones are:

  • modelDefinition
    • revision: the targeted git revision used to retrieve this model (main by default), it can be a branch or a commit hash

    • modelId: the model name

    • key: consists of hf@<modelId> or hf@<modelId>@<revision> if a specific revision was used

    • folderName: consists of modelId or modelId@revision the modelId and revision need to be escaped. Only alphanumerical values and dashes are kept. _xx represent each UTF-8 escaped byte.

  • version: as of now should be 0

  • url: the url used to fetch the model

API Node

A DSS API node also has its own model cache, populated automatically when needed.

To import a model to an API node with no network access to HuggingFace, run the following command:

./bin/apinode-admin __import-model-archive-in-cache model_archive.zip

To clear the model cache on the API node, run the following command:

./bin/apinode-admin __clear-model-cache