chief_ai_officer_advisory v1.0.0

name: "chief_ai_officer_advisory" description: "Advise founders and teams on AI strategy: API vs RAG vs fine-tune vs in-house build vs train-from-scratch, cost economics with a worked TCO calculator, AI risk classification (NIST AI RMF, EU AI Act tiers), and the AI hiring sequence. Use when someone asks how to adopt AI, which model approach to pick, what AI will cost, how to classify AI risk, or who to hire first."

Chief AI Officer Advisory

Purpose

Give founders and teams a clear, evidence-backed AI strategy: which build path to take, what it will cost, how to classify the risk, and who to hire. Every claim traces to a primary source listed in references/. This is advisory strategy, not legal advice and not a vendor recommendation.

Workflow

Work through the four decisions in order. Each one feeds the next.

1. Pick the build path

Apply the capability ladder. Never start above the rung that solves the problem.

1. Prompt a frontier API. Minutes to set up. If good prompts, few-shot examples, and output schemas produce acceptable results, stop here.
2. Add retrieval (RAG). Days to weeks. Use when the problem is missing or outdated knowledge. Instant to update, cheap to run.
3. Fine-tune a small model (PEFT/LoRA). Weeks. Use only when prompting and RAG both fail reliably: the model retrieves the right facts but gets the format, tone, or domain logic wrong. Requirements: 500+ quality input/output examples, a 100+ example eval set, a clear success metric. Training cost for 7B-scale runs $5-$200 in GPU time; full fine-tunes run $500-$50K.
4. Self-host an open-weight model. Use for steady high volume, strict latency budgets, or data that cannot leave the box (HIPAA, code, unreleased product info). Needs a production inference server (vLLM, TensorRT-LLM, or SGLang with continuous batching) to be worth it. Naive serving is 5-20x worse on the same hardware.
5. Train from scratch. $1M-$100M+. Only for novel architectures or domain specificity no open model can approximate. Picking this for ego when fine-tuning would ship six months earlier is the classic failure.

Pre-tune checklist: tried 5+ prompt variants, few-shot, RAG; have 500+ labeled examples; have an eval set; can measure improvement; compute cost justified. A "no" to any of the first three means go back and try prompting or RAG first.

Production pattern that works: prototype on a frontier API, then fine-tune a smaller model once the task stabilizes. Published benchmarks show this cutting per-token cost 10-50x while improving accuracy on narrow tasks.

2. Price it

Run bin/advisor.py tco to compare API vs self-hosted cost for the workload's actual numbers. The core formula:

CPM = (cluster $/hr) / (tokens/sec x 3600 / 1,000,000)

Rules of the math:

• tokens/sec must come from a measured benchmark of your workload (batch 1 vs batch 256 can be a 100x difference), never a vendor peak spec.
• Utilization is the real variable. A GPU at 40% utilization costs 2.5x per token of the same GPU saturated.
• Below roughly 50-100M tokens/month, API usually wins on cost alone. Volume argues for self-hosting only when it is short-context and batchable; long-context agentic workloads often never cross over.
• The GPU bill is compute only. Always present the full number: engineering time to run the serving stack, redundancy, autoscaling headroom, and the quality gap between the affordable open model and the frontier API.
• API prices move fast (the September 2026 price war pushed the floor to about $0.10/$0.50 per 1M in/out). Re-run the math quarterly.

See references/cost_economics.md for worked examples with cited numbers.

3. Classify the risk

Map the system through the two frameworks that matter. See references/risk_frameworks.md for the full walkthrough.

• NIST AI RMF 1.0 (voluntary, US): four functions — Govern (policies, roles, accountability), Map (context, stakeholders, failure modes), Measure (test the mapped risks), Manage (mitigations, incident response). Seven trustworthiness targets: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; fair with harmful bias managed. The Generative AI Profile (AI 600-1) adds confabulation, information integrity, and prompt injection to the risk list.
• EU AI Act (Regulation (EU) 2024/1689) (law): four tiers by use, not technology. Prohibited (Article 5: social scoring, emotion inference at work or school, manipulative practices; fines to EUR 35M / 7% turnover). High-risk (Article 6 + Annex III: hiring, credit, education access, essential services, law enforcement, biometrics, critical infrastructure; conformity assessment and documentation). Limited-risk (Article 50: chatbots and synthetic media need disclosure). Minimal-risk: no mandatory obligations. Run bin/advisor.py classify for a first-pass tier screen.

4. Sequence the hiring

• Pre-product-market fit: founder plus one ML-curious engineer on prompts. No AI team.
• Early (Series A): one applied AI engineer who owns prompts, evals, and deployment. A second for evals and quality.
• Growth (Series B): an AI/ML platform engineer for inference, evals, and observability; then production reliability.
• Scale (Series C+): an ML research scientist only if the model itself is the product. A safety or red team if AI faces customers.
• Role rule: most teams need the AI engineer (builds on LLMs) or the ML engineer (ships established methods to production), not the research scientist. Hiring a researcher to solve an engineering problem buys an expensive prototype and an unhappy employee. Fix data readiness before hiring anyone: enough labeled history, unified location, legally usable data.

Output Contract

A strategy engagement ends with:

1. Build-path recommendation with the rung chosen and why the rung above was rejected.
2. TCO sheet: API monthly cost vs self-host monthly cost at the client's measured token volume, with the assumptions (tokens/sec, utilization, GPU $/hr, API $/1M) stated in one table.
3. Risk classification: NIST functions applied, EU AI Act tier with the article cited, and the obligations that follow.
4. Hiring plan: next hire, the hire after, and what data or product milestone each unblocks.
5. Next actions: the smallest shippable step for this week.

Operating Rules

1. Every factual claim needs a verified source from web search or the references. No invented numbers, dates, or quotes. Unverifiable claims get cut or framed as opinion.
2. Use primary sources: standards bodies (NIST, EU regulation text), official docs, published benchmarks. Link them in references, not the main flow.
3. Quote prices with dates. API and GPU prices move fast; any price older than one quarter gets re-verified before it goes in the TCO.
4. Do not recommend train-from-scratch without a written justification that rules out the four cheaper rungs.
5. State what you do not know. If the token volume is unknown, say the crossover math is waiting on that number and give the range it would have to fall in.
6. Never invent team performance data, vendor outcomes, or regulatory actions. Cite the real thing or cut it.