Video processing

Note

This capability is provided by the “video-processing” plugin, which you need to install. Please see Installing plugins.

This plugin is Not supported

Features

  • Watch Video Tool: An agent tool that extracts frames from videos and analyzes them using vision-capable LLMs.

  • Intelligent frame sampling: Up to 10 evenly-spaced frames.

  • Automatic image optimization: Efficient LLM processing.

  • Seamless integration: Works within the Dataiku agent framework.

Configuration

Watch Video Tool

Configure the tool with the following parameters:

Parameter

Type

Description

input_folder

Folder

Dataiku managed folder containing your video files

llm_id

LLM

A multimodal LLM connection (must support vision)

Usage

The Watch Video tool is designed to be used by Dataiku agents. When invoked, it accepts:

  • video_name: The filename of the video to analyze (e.g., my_video.mp4)

  • question: A specific question about the video content

Example

Input:

{
  "video_name": "product_demo.mp4",
  "question": "What are the main features shown in this demo?"
}

Output:

{
  "output": "Visual Analysis of 'product_demo.mp4': The demo shows...",
  "sources": []
}

How It Works

  1. Video Validation: Checks that the requested video exists in the configured folder.

  2. Frame Extraction: Downloads the video and extracts up to 10 evenly-spaced frames using OpenCV.

  3. Image Optimization: Resizes frames to 512px width to reduce token consumption.

  4. LLM Analysis: Sends frames with the user’s question to a multimodal LLM.

  5. Response: Returns the LLM’s visual analysis.