---
name: "data_validation"
description: "Validate data before trusting it: schema contracts for JSON rows, null/empty handling policy, range and enum checks, referential consistency between datasets, runnable assertions that fail loudly, and quarantine-vs-drop policy for bad rows. Triggers on: validate data, check data quality, schema for API response, bad rows, validate LLM output, data contract, assert data shape."
metadata: { "includeInPrompt": true }
---

## Purpose

Put a contract between you and any data you didn't produce (API responses, files, LLM output) and check every row against it before downstream code touches it. Unvalidated data is the classic source of silent corruption: a missing field, a negative price, or a dangling foreign key that poisons results hours later. Validate once at the boundary, fail loudly, and quarantine bad rows instead of silently dropping them.

## Workflow

1. **Write the contract first, in code.** For JSON-shaped data, define a `jsonschema` schema before you look at the payload: types, required fields, `minimum`/`maximum` or `exclusiveMinimum` for ranges, `enum` for allowed values, `additionalProperties: False` to catch renamed fields. A schema is the contract; guessing what fields "should" be there is not a contract.

2. **Decide the null/empty policy explicitly.** `null` missing and `""` empty are three different states. Say in the schema which fields are `required`, whether empty strings pass `minLength: 1`, and whether an absent optional field is fine. If you haven't decided, you haven't validated.

3. **Check references across datasets.** A row's `seller_id` means nothing if the sellers table doesn't contain it. Collect key sets from each dataset and assert membership — referential consistency catches joins that would silently lose rows.

4. **Fail loudly, never silently coerce.** Run checks as assertions (`assert` or explicit `raise`) so a violation stops the run with a clear message. Converting `"N/A"` to `0` or dropping a row without logging hides bugs; coercion is a decision, and silent coercion is a bug.

5. **Quarantine bad rows, don't drop them.** Invalid rows go to a quarantine list with the reason attached, never deleted. Downstream code only ever sees rows that passed; quarantine is inspectable, countable, and reportable — a drop is invisible.

Example suite (run: `python3 -c "..."` after saving; every assertion here was executed):

```python
from jsonschema import Draft202012Validator
SCHEMA = {
    "type": "object",
    "properties": {
        "id": {"type": "string", "minLength": 1},
        "price": {"type": "number", "exclusiveMinimum": 0},
        "currency": {"type": "string", "enum": ["USD", "EUR"]},
    },
    "required": ["id", "price", "currency"],
    "additionalProperties": False,
}
v = Draft202012Validator(SCHEMA).validate
rows = [{"id": "p1", "price": 12.5, "currency": "USD"},
        {"id": "p2", "price": -4.0, "currency": "USD", "seller": "s9"}]
known_sellers = {"s1"}
good, quarantine = [], []
for r in rows:
    try:
        v(r)
        assert r.get("seller", "s1") in known_sellers, "dangling seller ref"
        good.append(r)
    except Exception as e:
        quarantine.append((r, str(e)))  # quarantined with reason
assert len(good) == 1 and len(quarantine) == 1  # loud gate
```

## Output Contract

After applying this skill, an agent should produce:

- A schema (or column contract) written as runnable code before any data was inspected, with the null/empty policy stated explicitly.
- Range and enum checks attached to the fields that have business meaning (prices, statuses, categories), not just type checks.
- A referential-consistency check for any key shared between two datasets.
- A quarantine list: every rejected row, with the reason attached — no silent drops.
- Assertion gates that stop the pipeline on violation, demonstrated passing on good data and failing loudly on bad data.

## Operating Rules

- **Contract before data.** Write the schema first; inspecting the payload and "adapting" the schema to it is testing against yourself.
- **Assert, don't coerce.** If a value is wrong-shaped, raise or quarantine it — converting it silently is a bug you planted.
- **Quarantine, never silently drop.** Every rejected row is kept with its reason; downstream sees only validated rows.
- **Prefer stdlib + jsonschema.** It pip-installs cleanly and was executed here. For heavier tools (pandera, great-expectations), confirm the exact API version in your environment before writing workflows around them — do not quote APIs you haven't run.
- **Soften absolutes.** "Validate at the boundary" is the default; a throwaway debug print of raw payloads during development is fine — just never let unvalidated rows flow into stored results.