generated_media_qa v1.0.0

name: "generated_media_qa" description: "Quality checking generated media before it ships: artifact spotting for hands, faces, text, and temporal flicker, prompt adherence scoring, continuity breaks across clips, and a pass, fix, or regenerate verdict per asset. Triggers when AI images or video clips need a pre-ship QA pass, when reviewing generated assets for defects, or when deciding whether to edit or regenerate an asset." metadata: { "includeInPrompt": true }

Generated Media QA

Purpose

Generated images and clips often look right at thumbnail size and fall apart on close inspection. Extra fingers, garbled in-image text, a face that drifts between clips, and background flicker all survive a quick glance. This skill is the pre-ship check that catches them. It covers content quality only: whether the asset shows what the prompt asked for, without visible artifacts, and whether it matches the assets around it. It does not cover technical delivery checks such as codec, resolution, loudness, or file naming. Run those separately.

The taxonomy and scoring methods come from published evaluation work. The five artifact categories are from Kamali et al., CHI 2025. Prompt adherence uses the question-answering method from TIFA. Video dimensions follow VBench. Full citations are in references/artifact-taxonomy.md.

Workflow

1. Lock the reference set before looking at the asset.

• Collect the exact prompt, any negative prompt, the seed or job ID if available, reference images, and the character or product bible for the set.
• Write down what must be present, what must not appear, and any exact text the asset must render word for word.
• If there is no prompt or reference to check against, say so and limit the verdict to artifact spotting only. Do not invent requirements.

2. Break the prompt into an atomic checklist.

• Split the prompt into single claims: one subject, one action, one attribute, one object, one count, one spatial relation at a time. For example, "a red mug on a wooden table, two hands holding it" becomes mug present, mug is red, table present, table is wooden, hands present, hand count is two, hands contact the mug.
• Turn each claim into a yes or no question you can answer from the asset alone. This is the TIFA method: faithfulness is the share of questions the asset answers correctly.
• Mark each question pass, fail, or not visible. A claim that should be visible and is not counts as fail, not as not visible.
• Score adherence as passes divided by total questions. Record which questions failed, because the failed list is what a fix or a new prompt has to address.

3. Inspect still images at full size, then at 100 percent.

• View the whole image first for composition, wrong subjects, and extra people or objects the prompt never asked for.
• Zoom to 100 percent and work a fixed order so nothing gets skipped: hands, faces, text, object edges and contacts, background, reflections and shadows.
• Hands: count fingers per hand, check finger length and joints, check that held objects pass through the grip in a plausible way, check nails and merged fingers.
• Faces: check eye size and gaze direction match, pupils, teeth overlap, ear shape and placement, hairline, skin texture. Waxy or plastic skin and over-smooth hair are stylistic artifacts, not passes.
• Text: transcribe every visible word character by character and compare it to the required text. Do not skim. Garbled glyphs, swapped letters, and plausible looking fake words all count as text failures. If an OCR tool is available, run it and diff its output against the required text, then still read the image yourself because OCR can normalize a wrong word into a right one.
• Edges and contacts: look for merged straps, buckles, handles, and limbs, objects floating without support, and resolution or sharpness that changes between subjects in the same image.
• Physics: check shadow directions agree, reflections match the subject in front of the mirror or window, and perspective lines do not contradict each other.
• Log every defect with its taxonomy category, location, and severity from references/artifact-taxonomy.md. One clear photo or crop per defect if the tooling allows it.

4. Inspect video clips in three passes.

• Pass one, full speed with sound off: watch for subject drift, background warping, and anything that pulls the eye for the wrong reason.
• Pass two, sampled frames: extract or pause on the first frame, the last frame, one frame per second, and every frame around a cut or a fast motion. Run the still image checks from step 3 on the sampled frames.
• Pass three, slow scrub: step through transitions and high motion sections frame by frame. Look for temporal flicker, which is rapid brightness or texture change in areas that should be stable, faces or clothing changing between frames, and objects popping in or out.
• Check motion separately from consistency. A clip can be perfectly consistent because almost nothing moves. Note whether the motion the prompt asked for actually happens, and whether it is smooth rather than jittery or sliding. VBench treats dynamic degree as a separate dimension for exactly this reason: consistency scores reward static clips.
• Check the first and last frames against the clips before and after it in the sequence. Loop points and edit points are where breaks hide.

5. Check continuity across a set, not just inside one asset.

• Build a contact sheet: one representative frame or image per asset, side by side, in story order.
• Compare, in order: face and identity, hair, wardrobe and wardrobe details, props and prop colors, location and background layout, lighting direction and time of day, screen direction and eyeline, aspect and scale of the subject in frame.
• Any change that the story does not explain is a continuity break. Log which assets disagree, and name the earlier approved asset as the source of truth unless the brief says otherwise.
• For a known person or a locked character, compare biometric details across assets: face shape, eye spacing, nose, ears, and distinctive marks. Drift here is a blocker even when each single image looks clean on its own.

6. Score and give one verdict per asset.

• Run scripts/qa_score.py with the checklist and defect log for the asset, or score by hand with references/scoring-sheet.md. The two use the same rules.
• Severity scale: S0 blocker, S1 major, S2 minor, S3 cosmetic. Definitions and examples are in references/artifact-taxonomy.md.
• Verdict rules, applied in this order:
• Regenerate: any S0 defect, or prompt adherence below 0.70, or a continuity break on a locked character, product, or required text.
• Fix: no S0, adherence at or above 0.70, and every S1 or S2 defect is local and repairable without changing identity, composition, or required text. Typical fixes are inpainting one hand, replacing text with a real text layer, cropping out a background defect, or stabilizing or trimming a flickering section.
• Pass: no S0 or S1 defects, adherence at or above 0.90, and no continuity breaks. S3 defects may remain if they are invisible at delivery size, and they still get logged.
• Never average a blocker away. A single S0 outweighs a high adherence score.
• Automated scores support the verdict, they do not replace it. CLIPScore measures text and image alignment in embedding space and does not measure realism or artifact absence. FVD compares a set of videos to a reference distribution and does not grade one clip on its own. If you report a numeric score from a tool, name the tool, the model, and what the score does and does not cover.

7. Write the fix or regeneration instruction from the defect log.

• For a fix: name the region, the defect, and what correct looks like. Keep the instruction local. Tell the editor or the editing model what must not change.
• For a regenerate: change something material, do not rerun the same prompt and hope. Use the failed checklist questions to rewrite the prompt, add or strengthen a reference image, split a crowded prompt into two assets, or move exact text out of the generator and into a composited text layer.
• Re-run the full QA on the new version. A fix for one hand can create a new defect next to it.

Output Contract

Every QA job produces one report per batch, with one row or section per asset:

• Asset identifier, prompt or brief it was checked against, and reviewer date.
• Prompt adherence score with the full atomic checklist and each question marked pass, fail, or not visible.
• Defect log: category, severity, location, and a plain description per defect.
• For video: temporal notes covering flicker, subject consistency, background consistency, and motion, plus first and last frame findings.
• For sets: continuity findings naming the assets that disagree and the source of truth.
• One verdict per asset: pass, fix, or regenerate, with the rule that produced it.
• For fix and regenerate verdicts: the specific instruction for the next attempt.
• A template is in references/scoring-sheet.md. Worked examples are in references/worked-verdicts.md.

Operating Rules

1. Check against the prompt and the references, not against taste. A beautiful asset that shows the wrong thing fails adherence.
2. Exact text is binary. One wrong letter in a required word, logo, or name is a failure, not a minor flaw.
3. Inspect at delivery size and at 100 percent. Defects that vanish at delivery size can be logged as S3. Defects visible at delivery size cannot be downgraded because they are small in the source file.
4. No universal numeric cutoff exists for CLIPScore, FVD, or VBench dimensions across models and prompts. Set thresholds per project from a known good baseline asset, and record the baseline used.
5. Count, do not estimate. Fingers, people, objects, and repeated elements get counted one by one.
6. Keep the defect log with the asset. The next reviewer should not have to rediscover why an asset was rejected.
7. If an asset cannot be viewed at full resolution or a clip cannot be scrubbed frame by frame, state that limit in the report and cap the verdict at fix pending full inspection. Do not pass what you could not actually see.