Simulation

Risk analysis workflows often require generating realistic scenarios, stress-testing assumptions, and understanding feature dependencies before decisions are made.

The Risk Analysis capability provides three complementary recipes for simulation-focused analysis in Dataiku:

• Generate Simulated Data: Simulate tabular data using configurable univariate distributions, with optional copula-based dependency modeling across columns.

• Generate Category-Conditioned Simulated Data: Simulate columns conditionally by category values, with explicit per-slice configuration and fallback behavior for non-configured slices.

• Output Correlation Matrix: Compute and materialize a correlation matrix from selected features using Pearson, Spearman, or Kendall coefficients.

These recipes are built on standard scientific Python tooling (NumPy, SciPy, scikit-learn, statsmodels) and are designed for practical Monte Carlo-style data generation inside Dataiku Flows.

These simulation capabilities are provided through the Risk Analysis plugin, instructions to install it can be found by following Installing plugins.

Generate Simulated Data

This recipe simulates new rows from an input dataset using one or more simulation configurations.

Core Capabilities

• Per-column univariate simulation with configurable distributions.

• Optional joint simulation over multiple columns using Gaussian or Student-T copula.

• Optional lower and upper simulation bounds.

• Handling of non-configured columns via:

  • value replication

  • Gaussian KDE simulation

• Optional retention of original rows in the output.

Key Parameters

• Number of Iterations Per Simulation: Number of synthetic rows to generate per simulation block.

• Retain Existing Data?: Keep original rows and append simulated rows.

• Simulate Joint Distribution Over Columns Via Copula:

  • If enabled, choose copula type (Gaussian or Student-T).

  • Choose covariance source:

    • use input correlation matrix dataset

    • fit from sample data

• Marginal Distribution Configuration:

  • Select column and distribution.

  • Optionally fit parameters via MLE.

  • Or provide distribution parameters directly.

• Simulation Bounds:

  • optional lower bound

  • optional upper bound

• Method for Non-Configured Columns:

  • replicate values

  • simulate with KDE

Output

• Output dataset with simulated rows.

• Optional Marginal Distribution Fit Report dataset.

• If retaining existing data, output includes a boolean marker column named is_simulated.

Generate Category-Conditioned Simulated Data

This recipe simulates data conditioned on a chosen categorical feature. It is useful when distributional behavior differs by segment (for example, region, product line, or customer tier).

Core Capabilities

• Select one categorical conditioner column.

• Configure distribution behavior per (column_to_simulate, conditioning_value) pair.

• Apply fallback behavior to non-configured segments and columns.

• Optionally pool remaining non-configured categories for KDE fitting.

• Optional retention of original rows with simulation marker.

Key Parameters

• Categorical Variable to Condition On

• Simulation Configurations:

  • column to simulate

  • conditioning categorical value

  • distribution selection and parameters

  • optional bounds

• Method for Non-Configured Columns:

  • replicate values

  • simulate with KDE

• When Using KDE, Handle Remaining Non-Configured Categories:

  • per_category: fit and simulate each remaining category separately

  • pool_remaining_categories_for_kde: fit one pooled KDE across all remaining categories

• Retain Existing Data?

Output Behavior

• Produces simulated rows for each category value of the conditioner.

• Row-count behavior:

  • Without retention: approximately (#categories) x (iterations per simulation).

  • With retention: original rows plus simulated rows per category.

• Adds is_simulated marker column when retention is enabled.

Notes

• Null values must be removed before running simulation.

• KDE can fail on degenerate samples (for example, near-constant values that create singular covariance).

Output Correlation Matrix

This recipe computes a correlation matrix over selected features and writes it back to the Flow. This correlation matrix can be used as input to the copula-based simulation.

Key Parameters

• Choose features to include in correlation matrix

• Correlation type:

  • Pearson

  • Spearman

  • Kendall

• Push resulting correlation matrix to an editable dataset?

Output

• Correlation matrix dataset with an index column and one column per selected feature.

• Optional creation of an additional editable dataset named <output_name>_editable when enabled.

Practical Guidance

• Prefer parametric distributions when domain knowledge is strong and stable.

• Use KDE for flexible non-parametric simulation, but verify sample quality and variance.

• Use copulas when preserving dependency structure is more important than independent feature realism.

• In conditioned simulation, configure critical slices explicitly and use fallback logic for long-tail slices.