Automation scenarios, metrics, and checks

Building a dataset or training a model can be done in various ways at the request of the user, for example by selecting the dataset in the flow view and using the Build action, but also in an automated fashion using scenarios.

• Definitions
  • Types of scenarios
  • Triggers
  • Reporters
• Scenario steps
  • How to Copy Scenario Steps
    • Copying Steps
  • How to Duplicate Scenario
    • Steps for Duplicating a Scenario
  • Build / Train
  • Clear
  • Run checks
  • Compute metrics
  • Synchronize Hive table
  • Create notebook export
  • Execute SQL
  • Execute Python code
  • Define variables, Set project variables, Set global variables
  • Run global variables update
  • Send message
  • Run another scenario
  • Package API service
  • Create dashboard export
• Launching a scenario
  • Types of triggers
    • Time-based triggers
    • Dataset modification triggers
    • SQL triggers
    • Python triggers
  • Concurrent triggers
  • Trigger parameters
  • Manual triggers
  • Evaluation
  • Examples
    • Launch a scenario every day at 10:00PM
    • Launch a scenario whenever a HDFS dataset changes
    • Launch a scenario whenever a SQL dataset changes
• Reporting on scenario runs
  • Executing scenario reporters
  • Building the message contents
    • Mail reporter
    • Slack, Microsoft Teams, Webhook and Twilio reporters
      • Slack integration setup
      • Microsoft Teams integration setup
    • Shell reporter
    • Scenario run URL
  • Common variables
• Custom scenarios
  • What can a custom scenario do
  • Examples
    • Basic usage
    • Send custom reports
• Variables in scenarios
  • (Re)defining new variables
  • Usage in partition identifiers
  • Examples
    • Using the current date to refresh a partition
    • Using the date from a time-based trigger
    • Using the results of a previous SQL step
    • Retrieving the message of a check
    • Retrieving the value of a metric
• Step-based execution control
  • On failure, proceed with scenario
  • Run step conditionally
    • Never
    • Always
    • If no prior step failed
    • If some prior step failed
    • If current outcome is
    • If condition satisfied
• Metrics
  • Probes and metrics
    • Dataset probes
      • Basic info
      • Records
      • Partitioning
      • Basic column statistics
      • Advanced column statistics
      • Column data validity statistics
      • SQL
      • Python
  • Metrics display UI
    • Datasets and managed folders
  • Probe execution engines
  • Metrics on partitioned datasets
• Checks
  • Checks
    • Numeric range check
    • Value in set check
    • Python check
  • Checks display UI
• Custom probes and checks
  • Custom probe
    • Return values
    • Partitioned datasets
  • Custom check