The authors scanned 3,171 public GitHub repositories that ship Claude Code, Cursor, Copilot, or Codex configurations. 16% of the assembled setups carry a confirmed security defect: an MCP server with no pinned version, a permission grant that looks narrow but allows any command, or a skill that pre-approves the shell. The raw scanner flagged 25.5% of setups with some finding; 18.4% remained after the authors re-derived every finding at a pinned commit.
A coding agent is not only a model. It is the model plus a set of files that developers write and share: instruction files such as CLAUDE.md or AGENTS.md, skills that load on demand, slash commands, hooks that run at fixed points in a session, declarations of MCP servers (external tool processes the agent connects to), and subagent definitions. The authors call this set the harness.
Each component is installed from marketplaces and public repositories, runs with the developer's privileges, and persists across sessions. A normal package is reviewed as code. A skill is a document that tells a privileged agent to read files, run commands, or call the network. It is reviewed as prose, if at all.
The harness has none of a package manager's safety checks. There is no lockfile: a server declared as npx -y @scope/server runs whatever version the npm registry serves on the day the agent starts. There is no install-time check: Claude Code loads a skill that has no name and no description, and improvises both. The permission vocabulary misleads: Bash(python:*) reads as a narrow grant, but it pre-approves any command, because python -c runs any code.
The authors built an open-source static scanner, harness-eval. It discovers harness components across eight agent layouts and runs 31 deterministic rules. No rule calls a model. Each rule runs in milliseconds, so it can block a build.
They measured only rules whose condition is decidable from file bytes and whose consequence is one of three things: a security exposure, a configuration that cannot work as written, or a departure from the Agent Skills specification, the published format for skill files. Rules that judge prose get no headline number.
Then they validated every finding, not a sample, in four passes. After the scanner reports, an independent implementation of each rule, written from the rule's documentation and not from the scanner's code, clones each flagged repository at its pinned commit and re-derives the finding; this covered 8,547 findings. Where the two implementations disagree, or the consequence is unclear, a language model reads the evidence and returns defect, not a defect, or uncertain; only defect counts. A separate model session then re-reads every counted repository-and-rule pair against the repository and the current platform documentation.
A rule earns a headline figure only if the two implementations agree on at least 97% of at least fifty findings, and at least 80% of its repository pairs end as defects. Six of the 31 rules pass this bar.
The corpus splits into two groups that the paper never pools. A setup assembles two or more component types. A collection publishes five or more skills with nothing that composes them. There are 2,660 setups and 511 collections. Collections are what developers install. Setups are what developers run.
Unpinned MCP servers, 9.8% of setups. The typical form is npx -y @scope/server-name with no version. uvx, the Python equivalent, and docker with no tag or digest appear in the same role. The server is fetched from a public registry on every session start and runs with the agent's privileges. Collections carry 0%: they ship skills, not server declarations.
Scoped-looking grants, 3.1% of setups. A permissions.allow entry such as Bash(awk:*), Bash(python:*), Bash(find:*), or Bash(sed:*) pre-approves arbitrary execution. awk runs commands through system(), find spawns processes with -exec, and GNU sed has an e flag that executes text as a command. Grants on curl, make, docker, or ssh are reported apart as advisory. Of 65 setups with a counted grant, 22 also ship a component that invokes the granted tool, so the grant is live and many authors wanted it.
Skills that pre-approve the shell, 3.8% of setups and 3.7% of collections. A skill file starts with a metadata block, the frontmatter. It can declare allowed-tools: the tools the client runs without a prompt while the skill is active. When that list includes an unrestricted Bash, installing the skill installs a shell pre-approval. This is the only security class that appears at publication time, so it is the only one a marketplace scan can catch.
Together, 16.0% of setups carry a confirmed security defect. A further 0.8% define a subagent with no description, which Claude Code skips and logs only to a debug file. 2.4% ship a skill outside the Agent Skills specification. In Claude Code every frontmatter field is optional, so such a skill loads on an improvised description while the specification's own validator rejects it.
Every rule that compares two files failed the consequence bar. The scanner can prove that two assistants' context files differ. It cannot prove that the difference is a mistake, and the adjudicator read most as intentional.
The defect class that motivated the whole instrument, a component that reads credentials and delegates to a network-capable component, fired on six repositories. The authors read all six; in four, an API key went to the vendor that issued it, and none was an exfiltration path. They report this class as absent.
The highest-volume rule, a path referenced in a skill body that does not resolve, fires on 33% of setups and 64% of collections. Roughly 45% of those are dead paths, 20% real files named by the wrong path, and 40% files the skill will create later in the consuming project. The skill format cannot mark a path as input or output, so no static check can tell these apart.
Every headline figure is a lower bound. The validation measures precision (how many reported findings are real), not recall (how many real defects the rules catch), so the authors do not know what the six gating rules miss. No human scored the model verdicts. The reviewing model agreed with the pipeline's verdict on 93.3% of 743 pairs; the authors re-examined each disagreement. The corpus over-represents public repositories that advertise their agent tooling. Consequences are dated to September 2026: during the study, Claude Code v2.1.257 stopped accepting the bypassPermissions setting from project settings, so that count became a version note instead of a defect.
Pin your MCP servers. Put a version in every npx and uvx declaration and a digest on every docker image, as you would in a lockfile. When you review a settings file, read Bash(python:*), Bash(awk:*), Bash(sed:*), and Bash(find:*) as Bash(*). Before you install a skill, open its frontmatter and check allowed-tools. Do not commit .claude/settings.local.json. Give every subagent a description, or it will never run.
If you maintain a team's agent configuration, run harness-eval on the pull request that introduces a component; it emits SARIF, the standard code-scanning report format, so it attaches like any dependency check. If you build scanners or publish prevalence numbers from one, re-derive the findings from the repositories first. The gap between 25.5% raw and 18.4% confirmed measures the detector, not the ecosystem.