video_input_analysis v1.0.0

name: "video_input_analysis" description: "Turn a video file or URL into an agent-readable comprehension pack: probe, scene-aware frame extraction, native captions, OCR, speech transcription, and a timestamped index you can ask follow-up questions against. Use when the user shares a video and asks what is in it, wants a summary, a transcript, or answers about specific moments."

Video Input Analysis

Purpose

Most agent setups hand a coding agent a video as an attachment and get guessing in return. This skill converts any video into a structured pack the agent can actually reason over: a probe report, scene-aware key frames, any native captions, an OCR pass over on-screen text, a word-timestamped speech transcript, and a single index.json that ties every signal to timestamps. Follow-up questions get answered from the index, not from memory.

Workflow

1. Probe first. Run bin/probe.py <video> and read the JSON: duration, resolution, fps, codec, audio streams, file size. Pick your extraction strategy from the duration:

• under 1 min: scene detection + 1 frame per scene, no cap
• 1 to 10 min: scene detection at threshold 0.3, cap frames at ~30
• over 10 min: scene detection, one frame per scene; if scenes are sparse, add a 1-frame-per-60s floor; run the long steps in the background

1. Get the cheapest text first. For a URL, try native captions before anything else: yt-dlp --write-subs --write-auto-subs --sub-langs "en.*" --skip-download <url>. If captions exist, they become the base transcript; they are never a reason to skip the transcription pass on a disputed passage.
2. Extract frames. bin/extract_frames.py <video> --out frames/ --scene-threshold 0.3 --max-frames 30. It finds scene boundaries with ffprobe, pulls one representative frame per scene, burns the timestamp into each frame, and writes frames/manifest.json (frame file, pts, scene index). If the video is one unchanging shot, it falls back to evenly spaced frames and says so in the manifest.
3. Transcribe the speech. bin/transcribe.py <video> --out transcript.json --model small --language en. It extracts 16 kHz mono audio and runs faster-whisper with VAD, word timestamps, and conditioning on previous text disabled (this combination prevents the repetition loops long files trigger). Output: transcript.json with segments and words, each carrying start/end seconds.
4. Read the on-screen text. bin/ocr.py frames/ --out ocr.json runs tesseract over the extracted frames (OCR-friendly preprocessing is built in). If tesseract is not installed, the script says so plainly instead of failing; the fallback is to hand the frames to a vision-capable model and note that OCR was not machine-verified.
5. Build the index. bin/build_index.py frames/manifest.json transcript.json ocr.json --out index.json. It merges everything into one timeline: per scene, start/end time, frame file, transcript segments overlapping that scene, and OCR text visible in that scene. This file is the single source of truth for every question that follows.
6. Answer from the index. See references/qa-workflow.md for the question-answering pattern: locate the relevant scene or segment by timestamp or keyword, verify against the raw frame or transcript slice, then answer with the timestamp cited. Never answer from a vibe about the video.

Output Contract

• probe.json: duration seconds, resolution, fps, video codec, audio presence and codec, file size bytes.
• frames/: timestamp-burned JPGs plus manifest.json with one entry per frame: file, pts seconds, scene index, and the strategy used.
• transcript.json: {"segments": [{"start": s, "end": s, "text": "..."}], "words": [{"start", "end", "word"}]} plus the model used.
• ocr.json: per-frame list of recognized text lines with confidence; or an explicit {"ocr": "unavailable", "reason": "tesseract not installed"}.
• index.json: scenes in time order, each with start, end, frame file, transcript text in range, and OCR text in range.
• A summary written for the user's actual question, with timestamps cited, and a note on what the pack cannot prove (frames are samples; nothing between samples is confirmed).

Operating Rules

• Probe before extracting. Never assume duration, fps, or that audio exists.
• Scene detection threshold 0.3 is the default. Lower it only when the video is a slow crossfade-heavy edit; raise it when shaky handheld footage triggers false cuts.
• Long files transcribe in the background. Do not block a session on a multi-minute transcription.
• Never cap frames or skip the transcription pass on your own judgment about length. Completeness scales with the file; only the user can ask for a lighter pass.
• State the audio model used in the output. Different runs can disagree on wording; the model name is the provenance.
• If the pack is for a video about a person or private event, handle the index like the video itself: it stays local unless the user says otherwise.
• Clean up the audio WAV and any temp directories after the index is built; keep the frames, transcript, OCR, and index.
• faster-whisper downloads its model on first run via httpx, which can choke on bracketed IPv6 entries in the NO_PROXY env var (proxy parsing error). If you see an httpx InvalidURL / Invalid port error, sanitize the proxy env first (unset NO_PROXY no_proxy in the shell) and rerun.