jq_yq_data_wrangling v1.0.0

name: "jq_yq_data_wrangling" description: "Practical jq/yq/csvkit patterns for agents extracting, filtering, and reshaping JSON/YAML/CSV in shell pipelines. Triggers on: parse this JSON, extract from API output, convert CSV to JSON, reshape YAML, wrangle shell data." metadata: { "includeInPrompt": true }

Purpose

Extract, filter, and reshape structured data (JSON, YAML, CSV) inside shell pipelines without round-tripping through Python or a browser. Covers jq for JSON, mikefarah yq for YAML, and csvkit for CSV. This skill is about structured-data extraction from APIs, files, and command output — not text scraping (that's web-research / log-forensics territory).

Workflow

Default tooling choice: jq for JSON, mikefarah yq (eval subcommand, v4.x style) for YAML, csvkit for CSV. Set export PATH=$HOME/.local/bin:$PATH first so the user-local pip installs resolve.

jq essentials (wrap the filter in single quotes):

• Field extraction: jq '.a.b'
• Raw string output (no quotes): jq -r '.name'
• Compact JSON on one line: jq -c '.'
• Iterate arrays, build objects: jq '[.[] | {val: .v}]' (each .[] is one element)
• Filter objects: jq '[.[] | select(.v > 2)]'
• Filter strings with regex: jq '[.[] | select(test("^a"))]'
• String interpolation: jq -r '.[] | "\(.n)=\(.v)"' → a=1 per line
• Missing/null safety: jq '.a?.b' (safe navigation), jq '.a // "default"' (fallback)
• Slurp multiple JSON objects into an array: jq -s '.'
• Shape inspection: jq 'length', jq 'keys', jq 'sort_by(.v)'
• Pass shell variables with --arg, not string interpolation: jq --arg n "$NAME" -r '"hello \($n)"'
• Emit CSV-ready rows from JSON: jq -r '[.login, .id] | @csv'

yq — know which one you have. Two flavors exist: mikefarah's Go yq (v4.x, the default here — it has an eval subcommand and handles YAML/JSON/TOML) and kislyuk's python yq (a jq wrapper; needs -y to emit YAML). If yq eval '.a.b' file.yaml errors, you're on python yq — add -y, or prefer the mikefarah binary.

• yq eval '.a.b' file.yaml — pull a nested value
• yq eval '.list[]' file.yaml — stream array elements, one per line
• yq eval -o=json '.' file.yaml — convert YAML to JSON (then pipe to jq)
• yq eval '.a.b = "new"' -i file.yaml — edit a file in place

csvkit (column names are case-sensitive; CRLF input is handled automatically):

• Select/reorder columns: csvcut -c city,name data.csv
• Filter rows: csvgrep -c city -m nyc data.csv
• Sort: csvsort -c age data.csv (add -r for descending)
• Eyeball a file: csvlook data.csv (pretty aligned table)
• CSV → JSON: csvjson -i 4 data.csv (indent 4; -i sets indent)
• Quick profile: csvstat --csv data.csv (types, min/max, unique counts)
• Join two files on a shared column: csvjoin -c name a.csv b.csv
• Rewriting line endings is rarely needed; if it is, csvformat with the right terminator flag handles it.

Common pipeline pattern: API response → jq → downstream CSV work:

bash
curl -s "https://api.github.com/users/octocat" \
  | jq -r '[.login, .id, .public_repos] | @csv' \
  | { echo "login,id,repos"; cat; } > out.csv
csvlook out.csv

Output Contract

• Show every command as a copy-paste-ready one-liner or short pipeline; include the exact flags used.
• When a command's output feeds another step, show the intermediate shape (field names, row count) so the caller can verify, not just the final blob.
• If two tools could do the job, name your pick and the one-line reason (e.g. "jq over python for a 3-field CSV extraction: zero new code").
• Never emit a flag you haven't confirmed with --help or a local run.

Operating Rules

• Single-quote jq filters. Double quotes let bash eat $ and !; jq ".a" will break. The only correct exception is deliberate single-quote wrapping around --arg-injected variables.
• Use --arg / --argjson for variables. Never interpolate shell variables into a jq filter string — it's brittle and an injection risk. jq --arg n "$NAME" '.items[] | select(.name == $n)'.
• -r when the output is text. Forgetting it returns "quoted" strings that break downstream string comparisons and file writes.
• csvcut column names are case-sensitive. csvcut -c Name fails when the header is name. When unsure, run csvcut -n data.csv first to list headers.
• Prefer csvkit's auto-detection over manual preprocessing. It handles CRLF and quoting; reach for csvformat only after the plain command visibly mis-parses.
• Soften absolutes. "Prefer single quotes around filters" beats "always"; "check cmd --help before scripting a new flag" beats "never guess flags". State the reason, not a commandment.
• Verify before claiming. Every command in a plan gets one local dry run on representative data before being presented as the solution.