Skills help an agent because they fix the procedure, not because they add facts. In matched tests, a SKILL.md written from five earlier runs beat the same five runs given as cleaned traces by 6 points, and cut environment-setup failures from 5.3% to 0.2%. Skills also add a new failure: in 10% of runs the agent misapplies or ignores the guidance. And when the skill library grows from 5 to 100 entries, only 3% of the skills the agent opens are the correct one, yet task success does not fall.
A coding agent solves a task, then forgets how. The next run finds the same setup commands, the same tool order, the same output format, and the same traps again. Teams now store this experience as skills: a short file, usually named SKILL.md, that says what to do, what to check, and what to avoid. Claude Code, Codex, and Gemini CLI all load skill files of this kind.
Until now, teams measured skills in one way: does the task success rate go up? That number does not say why skills matter, or what happens when the library holds one hundred skills instead of five. This paper asks: when do skills help, why do they work, and where do they fail.
The authors keep the experience fixed and change only its form. For each task they collect five earlier runs. Some passed and some failed. From the five runs they build three conditions. Raw: the agent gets nothing. Workflow Memory: the agent gets the five runs as cleaned, step-by-step traces. Skill: the agent gets a SKILL.md that a model wrote from the same five traces. The file sits in the container. The agent can read it, but nobody pushes it into the first prompt.
They test two agent stacks, Codex with GPT-5.3-Codex and Gemini CLI with Gemini-3.1-Pro, on three terminal benchmarks: Terminal-Bench 2, Terminal-Bench-Pro, and SkillsBench. Every task runs in a fixed Docker image, and a test suite decides if the run passed.
The new part is what comes after the success rate. For each task that ran in all three conditions, 528 of them, an LLM judge gives each run two labels. The first label is the failure mode, or the success mode. The second label is the mechanism: what the injected file did for the agent. There are five: it gave a procedure to follow, it gave a fact the agent did not have, it warned about a known failure, the agent did not use it, or it made the run worse. The 12 modes come from an LLM pass over 240 sample runs, merged into one taxonomy. A human then checked every label against three runs and mapped the labels to the 12 modes on their own. The human and the LLM agreed 95.8% of the time.
Skills win, but by a small margin. Skill runs pass 61.9% of the time, Raw 59.1%, Workflow Memory 55.9%. The clear result is Skill against Workflow Memory: +6.06 points. A resampled confidence range stays above zero, from +0.76 to +11.36. Both conditions came from the same five traces, so the gain comes from the form of the file, not from more experience.
The mechanism labels say what a skill does. In 65.7% of Skill runs, the skill works as a procedural anchor: it gives the agent an order of steps, a tool sequence, or a plan to verify the result. In only 4.5% of runs does the skill give the agent a fact it did not have. Skills rarely teach the model new facts. They stop the agent from changing its plan in the middle of a run.
The failure counts agree. Environment and infrastructure failures fall from 5.3% of Raw runs to 0.2% of Skill runs. Output-format mismatches fall from 7.4% to 3.2%. Failures with a background service fall from 2.7% to 0.8%. These are the wrong port, the missing dependency, the file in the wrong schema. The fix is short and the same each time, so it fits well in a skill.
Two checks make this stronger. On 26 Terminal-Bench 2 tasks, a short plan written from the task text alone reaches 47.7%, a generic test-first template 59.2%, and Skill 79.2%. A short hint of any kind does not give the gain. Outcome labels also matter: when the model that writes the skill does not know which of the five runs failed, one Gemini setup falls from 74.6% to 40.0%.
Skills do not fix reasoning. Logic errors in the algorithm stay at 7.4% of Skill runs against 8.3% of Raw runs. Runs where the agent reads its code instead of running it stay near 12% in every condition. A skill can tell the agent to run the tests. It cannot make a wrong algorithm right.
Skills also add a new kind of failure. The mode skill guidance misapplied or ignored appears in 10.0% of Skill runs, against 0.8% of Raw runs. The skill often holds correct guidance. The agent follows it step by step, misses a condition, or keeps an assumption from the source task that is no longer true. The authors call this the applicability problem: a skill does not execute itself. The agent must still decide if it applies and how.
Workflow Memory fails in a different way. Timeouts hit 10.6% of Workflow Memory runs, against 1.7% of Raw runs and 4.4% of Skill runs. The traces hold failed branches and long exploration, and the agent follows them. On 83 matched tasks, Skill costs about 95K more tokens per task than Workflow Memory, but 34K fewer than Raw, with 5.5 more points than Raw.
In production an agent picks a skill from a library. For each SkillsBench task the authors build a pool that holds the correct skill plus 4 to 99 skills for other tasks. They measure selection in three ways. First, a text-similarity search ranks the skill descriptions against the task. Second, an agent picks skills from the list without running the task. Third, an agent runs the task with the whole pool on disk.
The first two measurements get worse slowly. The similarity search puts the correct skill first 88.3% of the time at 5 skills and 76.9% at 100. The agent that picks without running falls from 70.0% to 63.7%. What hurts is not the size of the pool but similar entries: with skills that sound like the target, the search is correct only 53.4% of the time at 100 skills; with random skills, 84.1%.
The third measurement looks very different. Among the skills the agent opens, the share that is the correct one falls from 29.6% at 5 skills to 3.3% at 100. Yet task success moves only from 36.4% to 39.3%. The correct skill is neither sufficient nor necessary: related skills give partial procedural support, and the correct skill does not remove a timeout or a missing dependency. Retrieval precision does not measure what you care about.
Write skills as procedures, not fact sheets. The useful content is the setup sequence, the tool order, the intermediate checks, and the known traps.
Keep outcome labels when you write a skill from traces. Without that signal, a failed run becomes a procedure.
State when the skill applies. The 10% misapplication rate is the cost of abstraction. A line that says when the skill does not apply is as valuable as the steps.
Do not expect skills to fix wrong algorithms or skipped tests. Put runtime verification in the harness, not in the skill text.
Keep libraries small and distinct. Near-duplicate skills with similar names hurt selection most. Measure task success under the full library, not retrieval precision.
The caveats: two agent stacks, terminal benchmarks only, and an LLM judge labeled most of the 528 tasks. The taxonomy sample covers about 3% of all records. The library study used GPT-5.4 in place of GPT-5.3-Codex, so its numbers do not compare directly with the earlier tables.