KPIs

KPIs measure whether an agent produces the intended organizational outcome. They complement operational telemetry: a reliable, fast agent can still make poor decisions, and a technically successful interaction can still lead to an undesirable business result.

A KPI is a time series stored in a Dataiku dataset. A Dataiku Flow can join agent decisions with business systems, reference data, delayed outcomes, or human feedback before calculating that time series.

Designing a KPI Flow

Start from a business question that cannot be answered by agent telemetry alone. Identify the agent decision, the authoritative outcome, and the time at which the outcome becomes known.

For a fraud-detection agent, a false-positive-rate Flow can:

  1. Read the agent’s fraud decisions from an interaction or decision dataset.

  2. Read confirmed fraud outcomes from the transaction or investigation system.

  3. Join decisions and outcomes by transaction identifier.

  4. Select transactions that were not fraudulent.

  5. Calculate the percentage that the agent incorrectly flagged as fraud.

  6. Aggregate the result into the required time buckets.

  7. Write the KPI dataset in the Agent Management schema.

For a support agent, a similar Flow can join agent interactions with ticket records and calculate the percentage or volume of tickets resolved entirely by agents.

Use the standard Flow, preparation, join, group, and code recipe capabilities to implement the calculation. Use automation scenarios when the KPI dataset must be refreshed on a schedule.

Required dataset schema

The KPI dataset contains one row per KPI value and time bucket.

KPI dataset columns

Column

Requirement

Description

timestamp

Required

Date, datetime, or ISO 8601 timestamp for the KPI value.

value

Required

Numeric value, or a value that can be parsed as the configured KPI type.

kpi_id

Required when the dataset contains multiple KPIs

Identifier that matches the KPI ID configured in Agent Management.

infra_name

Optional

Platform identifier. When present, it must match the agent’s platform.

agent_id

Optional

Agent identifier. Use it when one dataset contains values for multiple agents.

The example KPI dataset can therefore contain columns such as:

timestamp   value   kpi_id   infra_name            agent_id
2026-09-21  28.19   13       my-design-platform    my-design-platform.PROJECT.AGENT

Keep identifier values stable. If the Flow writes several agents or KPIs into one dataset, every row must identify the intended agent and KPI unambiguously.

Configuring an agent KPI

Open the agent Settings tab and select KPIs. Add a KPI and configure:

  • Mode: define an inline KPI or reference a global KPI.

  • Dataset: select the prepared KPI dataset from its Dataiku project.

  • Label: the name displayed on monitoring pages.

  • ID: the value matched against kpi_id when that column is used.

  • Type: the display and interpretation type, such as amount or percentage.

  • Preferred direction: whether higher or lower values indicate better performance.

  • Description: the business definition and interpretation of the KPI.

Inline and global KPIs

KPIs are currently defined at the agent level. An agent can create one in three ways: conversationally, by describing it in a prompt, provided the data exists; manually, from a dataset; or by re-using an existing KPI definition.

Global KPIs are defined once in the global Agent Management settings, either manually or conversationally. Agents reference a global KPI and inherit its label, type, preferred direction, and description. Use an inline KPI when the definition is specific to one agent. Configure alert rules separately for the required targets.

Referencing a global KPI does not create or calculate its data. Each agent still needs a suitable dataset, and the dataset’s kpi_id must match the global KPI identifier.

Interpreting KPI values

Preferred direction describes how to interpret KPI values; it does not define an alert boundary. For example:

  • For success rate, higher values indicate better performance.

  • For false-positive rate, lower values indicate better performance.

To detect problems, create a separate alert configuration for the KPI. Choose the detection method, boundaries, and targets in that configuration. An agent-specific rule is independent of universal, platform, or scope rules and does not require replacing a referenced global KPI with an inline KPI.

Document the numerator, denominator, exclusions, source systems, refresh cadence, and expected delay in the KPI description or associated project documentation. These details are necessary to compare the KPI consistently over time.