Models evaluations

Evaluating a machine learning model consists of computing its performance and behavior on a set of data called the Evaluation set. Model evaluations are the cornerstone of MLOps capabilities. They permit Drift analysis, Model Comparisons and automating retraining of models

Evaluation of LLMs and Agents uses similar concepts, adapted to the specificities of these use cases.

• Concepts
  • When training
  • Subsequent evaluations
• Evaluating Dataiku Prediction models
  • Configuration of the Evaluation Recipe
    • Labels
    • Sampling
    • Custom Evaluation Metrics
  • Limitations
• Evaluating Dataiku Time Series Forecasting models
  • Input dataset
  • Outputs
    • Output dataset
    • Metrics dataset
    • Model Evaluation Store
  • Refitting for statistical models
• Evaluating other models
  • Configuration of the standalone evaluation recipe
    • Labels
    • Sampling
• Analyzing evaluation results
  • The evaluations comparison
  • Model Evaluation details
  • Using evaluation labels
• Automating model evaluations and drift analysis
  • Metrics and Checks
  • Scenarios and feedback loop
  • Feedback loop