Metrics, checks and Data Quality

Metrics allow you to automate computation of various measurements on flow items (datasets, managed folders, saved models and model evaluation stores). You can use Checks to assert whether metric values meet certain conditions.

Data Quality rules are an improvement over the check mechanism for datasets. They allow you to define expectations on a dataset’s contents in a single step and also provide different views to monitor and analyze data quality issues across datasets, projects, and the full Dataiku instance.

• Metrics
  • Probes and metrics
    • Dataset probes
      • Basic info
      • Records
      • Partitioning
      • Basic column statistics
      • Advanced column statistics
      • Column data validity statistics
      • SQL
      • Python
  • Metrics display UI
    • Datasets and managed folders
  • Probe execution engines
  • Metrics on partitioned datasets
• Checks
  • Checks
    • Numeric range check
    • Value in set check
    • Python check
  • Checks display UI
• Data Quality Rules
  • Data Quality rule types
    • Column min in range
    • Column min is within its typical range
    • Column avg in range
    • Column avg is within its typical range
    • Column max in range
    • Column max is within its typical range
    • Column sum in range
    • Column sum is within its typical range
    • Column median in range
    • Column median is within its typical range
    • Column std dev in range
    • Column std dev is within its typical range
    • Column values are not empty
    • Column values are empty
    • Column empty value is within its typical range
    • Column values are unique
    • Column unique value is within its typical range
    • Column values in set
    • Column top N values in set
    • Column most frequent value in set
    • Column values are valid according to meaning
    • Metric value in range
    • Metric value in set
    • Metric value is within its typical range
    • File size in range
    • Record count in range
    • Record count is within its typical range
    • Column count in range
    • Column count is within its typical range
    • Python code
    • Compare values of two metrics
    • Plugin rules
  • Rule configuration
    • Testing the rule
    • Hard and soft minimums and maximums
    • Change detection rules and settings
    • Enable / disable
    • Auto-run after build
    • Changing the rule type
  • Data Quality monitoring views
    • Dataset level view
    • Project level view
    • Instance level view
    • Timezone limitations
  • Other data quality views
    • Dataset right panel
    • Data Quality Flow view
  • Data Quality on partitioned datasets
    • Computation scope
    • Auto-run after build
    • Data Quality monitoring views
  • Retro-compatibility with Checks
• Custom probes and checks
  • Custom probe
    • Return values
    • Partitioned datasets
  • Custom check / Custom Data Quality rule
• Data Quality Templates
  • Creating a template
  • Reusing a template
  • Managing templates
  • Security