Monitoring model instances

DSS provides a monitoring modal for local Hugging Face models. It helps inspect the current state of deployed instances, recent activity, and live serving performance across supported tasks.

Open the modal from the Local Hugging Face connection model list, using the model instance status action for the model to inspect.

At the top of the modal, DSS displays aggregated metrics for the selected model. These metrics come from the model kernel pool and are available across tasks. Depending on the model task and serving engine, this can include:

• Number of model instances

• Request rate

• Input and output throughput

• Number of running requests

• Number of queued requests

These aggregated metrics are refreshed every 5 seconds. Live metrics are computed over the last minute.

Note

You can also monitor local models programmatically with the Python API.

Logs and metrics

The main section of the modal provides access to logs and, when available, metrics.

When the model is served with vLLM, DSS displays live charts. The available charts depend on the model task. For text generation models, DSS can display:

• Inter-token latency, also known as time per output token

• Time to first token

• Input throughput, that is, the number of input tokens processed per second

• Output throughput, that is, the number of generated tokens per second

• KV cache usage

The logs view displays the log tail for the selected instance, including running instances. This is useful to diagnose model startup issues, runtime failures, or serving slowdowns on a specific instance.

For stopped instances, DSS keeps the available log tail, but live charts are no longer available.