Metrics, checks and Data Quality

Metrics allow you to automate computation of various measurements on flow items (datasets, managed folders, saved models and model evaluation stores). You can use Checks to assert whether metric values meet certain conditions.

Data Quality rules are an improvement over the check mechanism for datasets. They allow you to define expectations on a dataset’s contents in a single step and also provide different views to monitor and analyze data quality issues across datasets, projects, and the full Dataiku instance.

• Metrics
  • Probes and metrics
    • Dataset probes
      • Basic info
      • Records
      • Partitioning
      • Basic column statistics
      • Advanced column statistics
      • Column data validity statistics
      • SQL
      • Python
  • Metrics display UI
    • Datasets and managed folders
  • Probe execution engines
  • Metrics on partitioned datasets
• Checks
  • Checks
    • Numeric range check
    • Value in set check
    • Python check
  • Checks display UI
• Data Quality Rules
  • Data Quality rule types
    • Column min in range
    • Column avg in range
    • Column max in range
    • Column sum in range
    • Column median in range
    • Column std dev in range
    • Column values are not empty
    • Column values are empty
    • Column values are unique
    • Column top N values in set
    • Column most frequent value in set
    • Column values are valid according to meaning
    • Metric value in range
    • Metric value in set
    • File size in range
    • Record count in range
    • Column count in range
    • Python code
    • Plugin rules
  • Rule configuration
    • Testing the rule
    • Hard and soft minimums and maximums
    • Enable / disable
    • Auto-run after build
  • Data Quality monitoring views
    • Dataset level view
    • Project level view
    • Instance level view
    • Timezone limitations
  • Other data quality views
    • Dataset right panel
    • Data Quality Flow view
  • Data Quality on partitioned datasets
    • Computation scope
    • Auto-run after build
    • Data Quality monitoring views
  • Retro-compatibility with Checks
• Custom probes and checks
  • Custom probe
    • Return values
    • Partitioned datasets
  • Custom check / Custom Data Quality rule