---
name: "video_input_analysis"
description: "Turn a video file or URL into an agent-readable comprehension pack: probe, scene-aware frame extraction, native captions, OCR, speech transcription, and a timestamped index you can ask follow-up questions against. Use when the user shares a video and asks what is in it, wants a summary, a transcript, or answers about specific moments."
---

# Video Input Analysis

## Purpose

Most agent setups hand a coding agent a video as an attachment and get guessing in return. This skill converts any video into a structured pack the agent can actually reason over: a probe report, scene-aware key frames, any native captions, an OCR pass over on-screen text, a word-timestamped speech transcript, and a single `index.json` that ties every signal to timestamps. Follow-up questions get answered from the index, not from memory.

## Workflow

1. **Probe first.** Run `bin/probe.py <video>` and read the JSON: duration, resolution, fps, codec, audio streams, file size. Pick your extraction strategy from the duration:
   - under 1 min: scene detection + 1 frame per scene, no cap
   - 1 to 10 min: scene detection at threshold 0.3, cap frames at ~30
   - over 10 min: scene detection, one frame per scene; if scenes are sparse, add a 1-frame-per-60s floor; run the long steps in the background
2. **Get the cheapest text first.** For a URL, try native captions before anything else: `yt-dlp --write-subs --write-auto-subs --sub-langs "en.*" --skip-download <url>`. If captions exist, they become the base transcript; they are never a reason to skip the transcription pass on a disputed passage.
3. **Extract frames.** `bin/extract_frames.py <video> --out frames/ --scene-threshold 0.3 --max-frames 30`. It finds scene boundaries with ffprobe, pulls one representative frame per scene, burns the timestamp into each frame, and writes `frames/manifest.json` (frame file, pts, scene index). If the video is one unchanging shot, it falls back to evenly spaced frames and says so in the manifest.
4. **Transcribe the speech.** `bin/transcribe.py <video> --out transcript.json --model small --language en`. It extracts 16 kHz mono audio and runs faster-whisper with VAD, word timestamps, and conditioning on previous text disabled (this combination prevents the repetition loops long files trigger). Output: `transcript.json` with segments and words, each carrying start/end seconds.
5. **Read the on-screen text.** `bin/ocr.py frames/ --out ocr.json` runs tesseract over the extracted frames (OCR-friendly preprocessing is built in). If tesseract is not installed, the script says so plainly instead of failing; the fallback is to hand the frames to a vision-capable model and note that OCR was not machine-verified.
6. **Build the index.** `bin/build_index.py frames/manifest.json transcript.json ocr.json --out index.json`. It merges everything into one timeline: per scene, start/end time, frame file, transcript segments overlapping that scene, and OCR text visible in that scene. This file is the single source of truth for every question that follows.
7. **Answer from the index.** See `references/qa-workflow.md` for the question-answering pattern: locate the relevant scene or segment by timestamp or keyword, verify against the raw frame or transcript slice, then answer with the timestamp cited. Never answer from a vibe about the video.

## Output Contract

- `probe.json`: duration seconds, resolution, fps, video codec, audio presence and codec, file size bytes.
- `frames/`: timestamp-burned JPGs plus `manifest.json` with one entry per frame: file, pts seconds, scene index, and the strategy used.
- `transcript.json`: `{"segments": [{"start": s, "end": s, "text": "..."}], "words": [{"start", "end", "word"}]}` plus the model used.
- `ocr.json`: per-frame list of recognized text lines with confidence; or an explicit `{"ocr": "unavailable", "reason": "tesseract not installed"}`.
- `index.json`: scenes in time order, each with start, end, frame file, transcript text in range, and OCR text in range.
- A summary written for the user's actual question, with timestamps cited, and a note on what the pack cannot prove (frames are samples; nothing between samples is confirmed).

## Operating Rules

- Probe before extracting. Never assume duration, fps, or that audio exists.
- Scene detection threshold 0.3 is the default. Lower it only when the video is a slow crossfade-heavy edit; raise it when shaky handheld footage triggers false cuts.
- Long files transcribe in the background. Do not block a session on a multi-minute transcription.
- Never cap frames or skip the transcription pass on your own judgment about length. Completeness scales with the file; only the user can ask for a lighter pass.
- State the audio model used in the output. Different runs can disagree on wording; the model name is the provenance.
- If the pack is for a video about a person or private event, handle the index like the video itself: it stays local unless the user says otherwise.
- Clean up the audio WAV and any temp directories after the index is built; keep the frames, transcript, OCR, and index.
- faster-whisper downloads its model on first run via httpx, which can choke on bracketed IPv6 entries in the NO_PROXY env var (proxy parsing error). If you see an httpx `InvalidURL` / `Invalid port` error, sanitize the proxy env first (`unset NO_PROXY no_proxy` in the shell) and rerun.