vibe_code_ship_check v1.0.0

---

name: "vibe_code_ship_check" description: "Turn an AI-built app into a reviewer-ready ship packet: freeze the intent, map the architecture and permissions, run intent-vs-code gap analysis plus security and performance audits, build a test coverage map, and end with a GO, GO WITH FIXES, or NO-GO verdict. Use when an app was built mostly by an AI coding tool (vibe coding), when asked 'is this ready to ship', before handing an AI-built codebase to a reviewer, client, acquirer, or production, or when nobody on the team can explain what the generated code actually does."

---

Vibe Code Ship Check

Purpose

AI coding tools produce apps that run on the first try and read like senior work. That polish hides the gaps: routes with no authorization check, dependencies that do not exist, business rules the model guessed at, tests that cover the happy path only.

Research context, stated as the sources report it:

• Veracode's 2025 GenAI Code Security Report tested over 100 models on 80 coding tasks and found security flaws in 45 percent of cases.
• Tenzai's December 2025 test of five vibe coding tools across 15 apps found 69 vulnerabilities, with the most serious in API authorization and business logic.
• Spracklen et al. (USENIX Security 2025) generated 576,000 samples from 16 models and found 205,474 unique package names that did not exist. Attackers can register those names first. The attack is called slopsquatting.

This skill closes the gap between "it runs" and "a reviewer can sign it." The output is a ship packet: eight short documents a reviewer can work through in one sitting, ending in a clear verdict. The workflow is harness-agnostic. It works the same whether the code came from a chat tool, an IDE assistant, an agent, or a mix.

Workflow

1. Freeze the intent

Before reading code, write down what the app was supposed to do. Sources, in order of trust:

1. The original prompts or task descriptions, if saved.
2. The README, product notes, and any user-facing copy in the app.
3. A short interview with the person who directed the build: who uses this, what can each kind of user do, what must never happen.

Write 01-intent.md: users and roles, the five to ten core jobs the app does, explicit non-goals, and the "must never" list (for example: a signed-out visitor must never read another user's data). If nobody can state the intent, that is the first finding. Stop and get it stated. Every later pass compares code against this document, not against vibes.

2. Run the scanner

bash
python3 bin/ship_check.py <repo> --out ship-packet/00-scanner-draft.md
python3 bin/ship_check.py <repo> --format json --out ship-packet/scanner.json

The scanner reports pattern findings with file and line evidence: likely secrets, debug flags, permissive CORS, string-built SQL, outbound HTTP calls with no timeout, routes in a file with no authorization hint, manifests with no lockfile, env vars with no example file, plus a test map. Treat every finding as a review prompt, not a confirmed flaw. Confirm, fix, or dismiss each one with a written reason.

Caution: the scanner prints the evidence line, which can include the literal secret it found. Treat the draft as sensitive, share it only with the review team, and rotate any real secret it surfaces. Rotation is not optional: a secret that reached a repo is burned even if the repo is private.

3. Architecture doc

Write 02-architecture.md following the standard design doc shape: Context and Scope, Goals and Non-Goals, Design Overview, Alternatives Considered, Cross-cutting Concerns. Keep it factual and code-derived:

• Components and what each one owns, with the entry points (routes, jobs, CLI commands).
• Data flow for the two or three most important jobs, step by step.
• Data stores, external services, and every outbound call, with timeouts and failure behavior.
• Trust boundaries: where untrusted input enters, where it gets validated, where it does not.
• Decisions the model made silently: framework choices, auth library, session handling, file storage. List each one and whether a person would have chosen the same.

If a component cannot be explained in plain sentences, flag it. Unexplainable code is a ship blocker until someone can explain it or it gets replaced.

4. Permission map

Write 03-permission-map.md as a table. One row per route, endpoint, job, or admin action:

| Surface | Signed out | Signed in, own data | Signed in, other user data | Admin | Check location | |---|---|---|---|---|---| | GET /notes/:id | deny | allow | deny | allow | src/notes.py:41 |

Fill the "should be" values from the intent doc first, then read the code and record what the code actually does. Mark every mismatch. Pay special attention to:

• Object-level checks: does the handler verify the record belongs to the caller, or only that the caller is signed in? Missing object checks are the classic broken access control flaw (OWASP A01).
• Actions with side effects reachable by GET, or with no CSRF protection.
• Admin and debug surfaces left routable in production builds.
• API keys and service roles: which key can do what, and is a publishable key ever used where a secret key is required, or the reverse.

5. Intent-vs-code gap analysis

Write 04-gap-analysis.md. Walk the intent doc job by job and record one of four states per job: matches intent, partial, missing, or extra behavior nobody asked for. Extra behavior deserves the same scrutiny as missing behavior: models add plausible features (export endpoints, password reset flows, public profiles) that were never requested and never reviewed. For business rules, test the boundaries a model tends to guess wrong: discounts, refunds, quotas, time windows, negative quantities, zero, repeated submission, and concurrent edits. Tenzai's testers found business logic and authorization were where vibe coded apps failed worst, because those rules live in the prompt's context, not in generic code patterns.

6. Security audit

Write 05-security-audit.md, organized by the OWASP Top 10 (2021): A01 Broken Access Control, A02 Cryptographic Failures, A03 Injection, A04 Insecure Design, A05 Security Misconfiguration, A06 Vulnerable and Outdated Components, A07 Identification and Authentication Failures, A08 Software and Data Integrity Failures, A09 Security Logging and Monitoring Failures, A10 Server-Side Request Forgery. One section per category, even if the section says "checked, nothing found, method: X."

Add the AI-specific pass, which a normal audit misses:

1. Dependency reality check. List every dependency the build added. Verify each one exists on its registry, is the package the author meant (right owner, real download history, a real source repo), and is pinned with a lockfile committed. This is the slopsquatting check from step 2 research: about one in five packages suggested in the USENIX study did not exist, and hallucinated names repeat across runs, so attackers can register them ahead of time.
2. Hallucinated APIs. Spot-check every library call in security-sensitive code (auth, crypto, payments, file handling) against the library's real documentation. A method that reads plausibly and does not exist, or exists with different semantics, is a finding.
3. Secrets sweep. Grep the full history, not just the working tree, for keys and tokens. Check the client bundle for values that were only meant to live on the server.
4. Generated tests lying. Run the test suite with one assertion deliberately broken in a scratch copy. If the suite still passes, the tests are decorative and the coverage map in step 8 must say so.

Map the gate to NIST SP 800-218 (SSDF) terms if the reviewer asks: this audit is PW.7 (review and analyze code) and the next step's test map is PW.8 evidence (test executable code).

7. Performance and reliability audit

Write 06-performance.md. This is a read-and-reason pass, not a load test. Look for the patterns generated code repeats:

• Queries inside loops, list endpoints with no pagination, selects that load whole tables.
• Outbound calls with no timeout, no retry limit, or retries with no backoff.
• Synchronous work in request handlers that belongs in a job: email, image processing, report generation.
• Uploads with no size limit, parsing with no size limit, regexes on untrusted input.
• Missing indexes on columns used in filters and joins, judged from the queries, stated as a hypothesis to verify with the database's explain tool.
• Caching that caches per-user data under a shared key.

Each item gets: location, the load shape that makes it hurt, and the smallest fix. Do not invent timings. If a number matters, measure it with a real run and record the setup, or mark it unmeasured.

8. Test coverage map

Write 07-test-map.md. Coverage percentage alone hides the point. Map tests to risk:

| Area | Tests exist | Tests assert behavior | Gap | |---|---|---|---| | Auth and permissions | yes | partial | no test for cross-user access | | Payments | yes | no, mocks assert nothing | full gap | | Core job 1 | yes | yes | edge cases only |

Use the scanner's per-directory test map as the starting grid, then read the tests. A test that calls the code and asserts nothing, or mocks the thing under test, counts as "exists, does not assert." List the three highest-risk untested areas and write the missing tests, or record them as accepted risk with a name next to the decision.

9. Assemble the packet and give the verdict

Collect the packet:

text
ship-packet/
  00-scanner-draft.md
  01-intent.md
  02-architecture.md
  03-permission-map.md
  04-gap-analysis.md
  05-security-audit.md
  06-performance.md
  07-test-map.md
  08-verdict.md

08-verdict.md states one verdict:

• GO: no open blockers, permission map matches code, dependency check clean, highest-risk areas tested or explicitly accepted.
• GO WITH FIXES: blockers are listed, each with an owner and a fix, and the packet states which fixes must land before real users or real data.
• NO-GO: intent cannot be stated, a permission mismatch exposes user data, a dependency cannot be verified, or a core area is both untested and unexplained. Name the specific blocker and what evidence would clear it.

A worked mini-packet is in references/worked-example.md. A blank packet scaffold is in assets/ship-packet-template.md; copy it and fill it in.

Output Contract

Every run produces the nine files above in a ship-packet/ directory in or next to the repo, plus a short summary in the reply: verdict, blocker count by severity, the top three risks in plain language, and the list of dependencies that failed verification (or "none"). Findings always carry file and line evidence. Anything the reviewer could not verify is listed as unverified, never smoothed over.

Operating Rules

1. Intent first, code second. If the intent cannot be stated, the verdict is NO-GO until it can.
2. Never trust comments, docstrings, or a README written by the same model that wrote the code. They describe what the model meant, not what the code does. Verify against behavior.
3. Verify every dependency exists and is the intended package before calling the audit done. Registry check by hand or with the registry's own CLI; never from memory.
4. Findings need evidence: file, line, and the exact behavior. "Looks insecure" is not a finding.
5. Do not invent numbers: no timings, coverage figures, or vulnerability counts that were not measured or counted in this run.
6. Scanner output is a starting grid, not the audit. The permission map, gap analysis, and business rule checks are manual by design, because those are the failures scanners miss in generated code.
7. A secret found anywhere is rotated, not just deleted. Deleting the line does not clear the history.
8. Keep the packet short enough to read in one sitting. Depth goes in evidence links and line references, not in longer prose.
9. The verdict belongs to the reviewer. The packet's job is to make that verdict easy and defensible.