Evaluation Recipe

Similar to single-target regression models, you can use the Evaluation Recipe with multi-target regression models to compute performance metrics on another dataset. You can also store the resulting evaluation in a Model Evaluation Store.

Input dataset

The input dataset for the evaluation recipe must contain the target columns used by the model.

If sample weights were enabled during training and performance computation is enabled in the evaluation recipe, the input dataset must also contain the sample weight column.

Skip scoring

When the evaluation recipe scores the input dataset, DSS computes one prediction column per target.

If you enable skip scoring, the input dataset must already contain one prediction column per target. The prediction column names must use the following format:

prediction_<target>

Skip scoring is supported when using a classic evaluation dataset. Evaluation datasets detected as API node logs, Cloud API node logs, or SageMaker logs are not supported for multi-target regression evaluation recipes.

Output datasets

Output dataset

The evaluation recipe output dataset contains the input rows and one prediction column per target.

Depending on the selected outputs, DSS can also add per-target error columns, such as error_<target>, error_decile_<target>, abs_error_decile_<target>, and relative_error_<target>.

Metrics dataset

The metrics dataset contains the selected regression metrics.

By default, DSS writes one row with global metrics aggregated across targets. If per-target metrics are enabled, DSS writes one row per target and adds a target column to identify the target for each row.

Custom metrics defined in the evaluation recipe are not supported. Custom metrics configured in the model analysis are supported.

Model Evaluation Stores

Multi-target regression evaluations can be stored in Model Evaluation Stores.

This includes evaluations that use skip scoring with a classic evaluation dataset.

Text and image drift are not supported for multi-target regression model evaluations.

Limitations

Multi-target regression has the following evaluation limitations:

  • External or partitioned models are not supported.

  • Evaluation datasets detected as API node logs, Cloud API node logs, or SageMaker logs are not supported.

  • Evaluation recipe assertions are not supported.

  • Custom metrics defined in the evaluation recipe are not supported. Custom metrics configured in the model analysis are supported.

  • Text and image drift are not supported.