---
name: "jq_yq_data_wrangling"
description: "Practical jq/yq/csvkit patterns for agents extracting, filtering, and reshaping JSON/YAML/CSV in shell pipelines. Triggers on: parse this JSON, extract from API output, convert CSV to JSON, reshape YAML, wrangle shell data."
metadata: { "includeInPrompt": true }
---

## Purpose

Extract, filter, and reshape structured data (JSON, YAML, CSV) inside shell pipelines without round-tripping through Python or a browser. Covers jq for JSON, mikefarah yq for YAML, and csvkit for CSV. This skill is about structured-data extraction from APIs, files, and command output — not text scraping (that's web-research / log-forensics territory).

## Workflow

**Default tooling choice:** jq for JSON, mikefarah yq (`eval` subcommand, v4.x style) for YAML, csvkit for CSV. Set `export PATH=$HOME/.local/bin:$PATH` first so the user-local pip installs resolve.

**jq essentials** (wrap the filter in single quotes):

- Field extraction: `jq '.a.b'`
- Raw string output (no quotes): `jq -r '.name'`
- Compact JSON on one line: `jq -c '.'`
- Iterate arrays, build objects: `jq '[.[] | {val: .v}]'` (each `.[]` is one element)
- Filter objects: `jq '[.[] | select(.v > 2)]'`
- Filter strings with regex: `jq '[.[] | select(test("^a"))]'`
- String interpolation: `jq -r '.[] | "\(.n)=\(.v)"'` → `a=1` per line
- Missing/null safety: `jq '.a?.b'` (safe navigation), `jq '.a // "default"'` (fallback)
- Slurp multiple JSON objects into an array: `jq -s '.'`
- Shape inspection: `jq 'length'`, `jq 'keys'`, `jq 'sort_by(.v)'`
- Pass shell variables with `--arg`, not string interpolation: `jq --arg n "$NAME" -r '"hello \($n)"'`
- Emit CSV-ready rows from JSON: `jq -r '[.login, .id] | @csv'`

**yq — know which one you have.** Two flavors exist: mikefarah's Go yq (v4.x, the default here — it has an `eval` subcommand and handles YAML/JSON/TOML) and kislyuk's python yq (a jq wrapper; needs `-y` to emit YAML). If `yq eval '.a.b' file.yaml` errors, you're on python yq — add `-y`, or prefer the mikefarah binary.

- `yq eval '.a.b' file.yaml` — pull a nested value
- `yq eval '.list[]' file.yaml` — stream array elements, one per line
- `yq eval -o=json '.' file.yaml` — convert YAML to JSON (then pipe to jq)
- `yq eval '.a.b = "new"' -i file.yaml` — edit a file in place

**csvkit** (column names are case-sensitive; CRLF input is handled automatically):

- Select/reorder columns: `csvcut -c city,name data.csv`
- Filter rows: `csvgrep -c city -m nyc data.csv`
- Sort: `csvsort -c age data.csv` (add `-r` for descending)
- Eyeball a file: `csvlook data.csv` (pretty aligned table)
- CSV → JSON: `csvjson -i 4 data.csv` (indent 4; `-i` sets indent)
- Quick profile: `csvstat --csv data.csv` (types, min/max, unique counts)
- Join two files on a shared column: `csvjoin -c name a.csv b.csv`
- Rewriting line endings is rarely needed; if it is, `csvformat` with the right terminator flag handles it.

**Common pipeline pattern:** API response → jq → downstream CSV work:

```bash
curl -s "https://api.github.com/users/octocat" \
  | jq -r '[.login, .id, .public_repos] | @csv' \
  | { echo "login,id,repos"; cat; } > out.csv
csvlook out.csv
```

## Output Contract

- Show every command as a copy-paste-ready one-liner or short pipeline; include the exact flags used.
- When a command's output feeds another step, show the intermediate shape (field names, row count) so the caller can verify, not just the final blob.
- If two tools could do the job, name your pick and the one-line reason (e.g. "jq over python for a 3-field CSV extraction: zero new code").
- Never emit a flag you haven't confirmed with `--help` or a local run.

## Operating Rules

- **Single-quote jq filters.** Double quotes let bash eat `$` and `!`; `jq ".a"` will break. The only correct exception is deliberate single-quote wrapping around `--arg`-injected variables.
- **Use `--arg` / `--argjson` for variables.** Never interpolate shell variables into a jq filter string — it's brittle and an injection risk. `jq --arg n "$NAME" '.items[] | select(.name == $n)'`.
- **`-r` when the output is text.** Forgetting it returns `"quoted"` strings that break downstream string comparisons and file writes.
- **csvcut column names are case-sensitive.** `csvcut -c Name` fails when the header is `name`. When unsure, run `csvcut -n data.csv` first to list headers.
- **Prefer csvkit's auto-detection over manual preprocessing.** It handles CRLF and quoting; reach for `csvformat` only after the plain command visibly mis-parses.
- **Soften absolutes.** "Prefer single quotes around filters" beats "always"; "check `cmd --help` before scripting a new flag" beats "never guess flags". State the reason, not a commandment.
- **Verify before claiming.** Every command in a plan gets one local dry run on representative data before being presented as the solution.