ai_video_api_workflows v1.0.0

name: "ai_video_api_workflows" description: "Run AI video generation APIs end to end as an agent: pick the right model, write shot briefs that generate well, keep characters consistent across scenes, batch-render with submit-poll loops, and control spend. Triggers on 'generate video via API', 'text to video', 'image to video', Runway API, Veo API, Kling API, Luma API, 'video generation prompt', 'keep the same character across scenes', 'batch video rendering', or 'how much does AI video generation cost'."

AI Video API Workflows

Purpose

Turn video generation APIs (Runway, Google Veo 3.1, Kling, Luma Ray, or aggregators like fal.ai and Replicate) into a reliable production pipeline an agent can run: plan shots, generate with consistency, poll jobs, assemble clips, and stay inside a budget. This skill covers the workflow. It is not a model review and not a credentials guide.

Workflow

1. Scope the job before spending anything

• Decide: number of shots, seconds per shot, resolution (720p vs 1080p), audio needed or not, aspect ratio (16:9 or 9:16).
• Estimate first with bin/video_job.py estimate. Iterate drafts on the cheapest tier that still exercises your brief; render finals on the quality tier.
• Rule: cost the attempts, not just the accepted clips. Expected cost = attempts x per-attempt price / expected accepted rate. Budget two attempts per accepted clip unless you have measured your own acceptance rate.

2. Write one shot brief per shot

Use references/shot-brief-templates.md. A working brief has exactly these fields, in this order: subject, action, environment, camera move, lighting, visual style, duration, aspect ratio, audio note. Each field is one concrete phrase, never adjectives alone. Example:

Subject: A woman in a red raincoat, mid-30s. Action: She turns and walks toward camera. Environment: Wet neon street at night. Camera: Slow push-in, eye level. Lighting: Cyan rim light, wet reflections. Style: Cinematic, shallow depth of field. Duration: 8s. Aspect: 9:16. Audio: Rain ambience, footsteps.

• Text-to-video for new compositions. Image-to-video when the first frame already exists or a character's look must start locked.
• Generate 2-4 candidates per shot, pick the strongest, move on. Rewriting one bad prompt endlessly costs more than re-rolling.

3. Lock consistency before generating scene two

• Keep a cast/ folder: one locked reference per character (front face, side profile) plus wardrobe and setting notes. Pass the same references to every shot, on every model, every day. Consistency lives in the pipeline, not the engine.
• Between shots, chain frames: feed the last frame of shot N as the first frame of shot N+1, or use the provider's extend/continue feature. references/consistency-playbook.md has the full checklist.
• Where the provider offers multi-reference generation (Kling Elements, Veo Ingredients to Video), use it for cross-scene identity.

4. Submit, poll, store, assemble

• Every provider uses the same shape: POST a job, get a job ID, poll until completed, download immediately. Use bin/video_job.py submit and bin/video_job.py poll, or the --endpoint config for providers not in the catalog.
• Download every clip to your own storage the moment it completes. Do not treat the API as a host; some providers delete outputs within hours.
• Stitch accepted clips with ffmpeg (concat demuxer, re-encode once at the end). Add music, voiceover, or captions after assembly, not per clip.
• Track every attempt in a job ledger: timestamp, model, prompt hash, cost, verdict. The ledger is how you learn your real acceptance rate and where retries burn money.

5. Pick the provider with a reason

references/provider-catalog.md holds the researched table (pricing, durations, audio, consistency features, docs links; verified 2026-10-07). General guidance:

• Drafts and volume: cheapest per-second tier (Veo 3.1 Lite/Fast, Kling Standard silent, Luma 720p).
• Finals with native dialogue: Veo 3.1 Standard or Kling with audio.
• Multi-shot story work: Kling storyboard mode or Seedance-style multi-shot before stitching separate clips.
• One-API-many-models: fal.ai or Replicate, at a small markup, when you want a single integration surface. Verify each model's card; markup varies per model.
• Never build on a consumer subscription for automation. App credits and API access are separate products on every major provider; a subscription grants zero API access.
• Keep two providers configured. Providers change terms and retire models; failover is a config flag, not a rewrite.

Output Contract

A completed run delivers:

1. shots/shot-NN-brief.md for every shot, each with the nine fields filled.
2. clips/ with accepted clips, named shot-NN-take-K.mp4.
3. job-ledger.csv: timestamp, model, provider, seconds, resolution, cost, verdict, notes.
4. final/assembly.mp4 (stitched) and final/cost-report.txt (per-shot cost, total, acceptance rate).
5. A short note on what to change next run (which prompts failed, which tier misbehaved).

Operating Rules

1. Every factual claim about a provider (price, endpoint, limits) is checked against that provider's current docs before a real spend. Pricing moves; the catalog is a starting point, not a contract.
2. No key in the repo. API keys live in environment variables or the credential store; the CLI reads them from env at runtime.
3. Dry-run first: bin/video_job.py --mock exercises the full submit-poll-download loop locally before any real key is touched.
4. Frame anything unverifiable as opinion. If a claim cannot be verified (a price, a new model, an endpoint), say so and cut or reframe it.
5. Disclose synthetic media: generated clips get appropriate AI labels in any customer-facing output, and SynthID-style provenance marks stay intact.
6. Cost caps on unattended runs: set a max spend per run and a max attempts-per-shot before starting; a loop that re-submits blindly is a billing hole.