Semantic Feedback Management

A common need in Agentic use cases is for the Agent to improve over time, as it gets more feedback about its action.

Feedback can take two main forms:

• Explicit feedback: the user clicked a “thumbs-up” / “thumbs-down” control, and optionally gave explicit indication of what was right or wrong. Explicit feedback is then typically stored in a dataset.

• Implicit feedback: the user stopped interacting, told the Agent it was wrong, or lost their calm

Structured Visual Agents provide two blocks for managing feedback:

The first block typically runs at the end of a conversation turn and detects implicit feedback, which is then stored by a dataset.

In a typical case, there would then be a manual curation phase, where both detected implicit feedback, and user explicit feedback are reviewed, curated, merged, in order to create a unified feedback dataset, with entries such as:

• Kind of feedback: positive, correction, constraint, warning of a typical failure pattern

• Content of the feedback

The second block then typically runs at the start of a conversation turn. It then performs a semantic search within the unified feedback dataset, detects which feedback items are relevant to the current conversation, and inject them in the context, so that the LLM underlying the agent can read it and help steer its behavior

Setup

These blocks are provided by the “Agent Semantic Feedback” plugin, which you need to install.

These blocks require that the Agent runs in a specific Code Env, which you can configure in the “Settings” tab of the agent. The Code Env needs to contain the “pysqlite3” and “sqliteai-vector” packages

Usage

The Detect Feedback block writes into a dataset. The Use Feedback block reads from a dataset. While they can be the same, these two datasets are typically separate, due to the mentioned curation, deduplication and unification process, which is handled as part of a regular Dataiku Flow.