skill_collision_audit v1.0.0

name: skill_collision_audit description: Audit an installed skill stack for trigger collisions, where two or more skills claim the same request or a skill fires on the wrong job. Use when skills fight each other, the wrong skill loads, several skills load at once, a new skill starts stealing requests from old ones, or a stack owner needs to test and fix skill descriptions as a set. Do not use for writing one new skill from scratch or for tuning a single description with no neighbour in conflict.

Skill Collision Audit

Purpose

A skill is chosen from its name and description alone. The body loads only after the choice, so collisions live in the descriptions: vague or broad descriptions claim jobs that belong to a neighbour, two descriptions with the same verbs and objects give the model no line to follow, and a new skill can start stealing requests from skills that worked fine before it arrived. This skill audits the stack as a set. It inventories every installed skill, ranks suspect pairs by description overlap, proves or clears each pair with a shared test set of real requests, rewrites descriptions as narrow triggers with named boundaries, sets a specificity order and a per-task cap, and re-runs the test set until each request loads the right skill or none.

Workflow

1. Inventory the stack. Run python3 scripts/audit_collisions.py --skills-root <dir> from this skill folder, or point it at each skills directory in turn. Record per skill: name, description, character count, whether it is over the 1024 character specification limit, and whether it carries an exclusion clause. A stack with no inventory has no audit, only opinions about individual skills.
2. Rank suspect pairs. Read the overlap section of the script output. It scores pairs by Jaccard similarity on description terms after stopword removal. Work the list from the top. Treat overlap as a lead, never as proof: two skills can share the word code and own clearly different jobs.
3. Build the collision test set. Write one shared test set for the whole stack, not one file per skill, in the format in references/test-set-format.md. Each query names one expected skill or null. Load it with near misses for every suspect pair, in both directions, plus a none bucket for requests no skill should claim. Split about 60 percent train and 40 percent validation and keep the split fixed.
4. Run the test set through the real agent. Run each query 3 times with the full stack installed, and record which skills actually loaded, from client logs or tool history, into a results file. Score it with python3 scripts/audit_collisions.py --skills-root <dir> --test-set <file> --results <file>. Read the pass rate by split, the co-fired queries, and the confusion list. The pair at the top of the confusion list is the first rewrite target.
5. Rewrite one pair at a time. Follow references/trigger-rewrite-guide.md. One move per round: cut sales language, swap shared verbs for owned verbs, add a mirrored boundary sentence in both skills, or tighten exclusions from the failing train cases. Generalise from failures, do not paste failed query wording into a description. Leave every body untouched, no body sentence can fix a routing fight.
6. Set the specificity order and the cap. Write down which skill wins when a request honestly matches two: exact artefact beats general activity, a skill whose exclusion covers the request loses, an end-to-end workflow beats a background reference skill, and a skill the user named beats every inferred match. Set the per-task cap at 2 loaded skills by default, one owner plus at most one support skill, and log any skill dropped at the cap. Put the order where the agent sees it before choosing, in project instructions or the stack inventory.
7. Re-run and close out. Re-run the full test set after each rewrite round, same split, same run count. Stop when the train set passes and the validation pass rate stops improving, and keep the best round by validation, which may not be the last round. If a pair still co-fires after five rounds, merge, split, or retire one of the pair and say so in the report. Worked numbers for every step are in references/worked-audit.md.

Output Contract

A finished audit produces:

• An inventory table: every installed skill with its description length, limit status, and exclusion status.
• An overlap list: suspect pairs ranked by score, each marked cleared, convicted, or untested after the test run.
• The test set file in the format in references/test-set-format.md, with its train and validation split, plus the results file recorded from real runs.
• A score report per round: pass rate by split, co-fired queries, and the confusion list, so a later round can be compared against an earlier one.
• The rewritten descriptions, one pair at a time, each with the move that was used and the round it landed in.
• A written specificity order and per-task cap, placed where the agent reads it before skill selection.

Operating Rules

1. Descriptions route, bodies do not. If a fix lives in a skill body, it is not a collision fix. Move the when-to-use and boundary language into the description.
2. Never convict a pair on overlap alone. A collision claim needs a test query that actually loaded the wrong skill or loaded both, recorded from a run.
3. One rewrite move per round, one pair per round. Bundled changes make the re-run score impossible to read.
4. Keep descriptions under the 1024 character limit after every rewrite. Boundary and exclusion sentences compete for the same budget as positive triggers, so cut sales language first to make room.
5. Protect the validation set. Guide rewrites from train failures only. A rewrite tuned against validation numbers is overfit and will fail on fresh requests.
6. Add a real failure to the test set the day it happens, with the exact query and what fired. The set gets sharper every time the stack gets caught out.
7. Re-audit after every install. A new skill changes the routing of the whole stack, not just its own corner. Run the full test set, not only the new skill cases.
8. Report rates, not single runs. Triggering varies between runs, so every pass or fail in the report rests on a trigger rate across at least 3 runs, with 0.5 as the default threshold.