Microsoft and CMU researchers gave two frontier models five different tool interfaces on two enterprise task suites. A bare shell beat a catalog of typed tools by 24.6 points on simulated software-company work and by 6.2 points on banking, consulting, and law work. It used fewer tokens, and it gained nothing from typed or self-written tools added on top.
An agent acts on the world through its tool interface. Most enterprise agent platforms give the model a catalog of typed tools: one JSON function per action, such as send a chat message or read a spreadsheet. The platform vendor writes and approves each tool. Strong coding agents use a different design. They expose only a shell and let the model write commands. The authors ask if the shell design also works for enterprise work, where tasks cross several applications, involve coworkers, and require professional analysis.
The choice matters for three reasons. Capability: a shell can do many steps in one call; a catalog can do only what it lists. Safety: a shell runs arbitrary code; a catalog runs only approved functions. Cost: every typed call is a separate model turn, and every result goes into the context window.
The authors keep the model, the task text, the base prompt, and the stopping rules the same. Only the tool surface changes. They test five variants:
Tool-only. The model sees only the typed catalog. Bash. The model sees only a shell. Bash+Tool. The model sees both. Bash+Synthesis. The model sees a shell plus a directory that persists across tasks, and the prompt asks it to save reusable scripts there for later tasks. Programmatic tool calling, or PTC. The model writes a Python program. The program can call catalog functions, loop over them, and combine their results. It has no shell and no other way to act. Anthropic and OpenAI both offer this mode in their APIs.
The two models are Opus-4.8 and GPT-5.5 at default reasoning effort. The agent loop comes from the GitHub Copilot SDK with its default tools and prompts removed.
TheAgentCompany simulates a software company. The agent works with a local workspace, four self-hosted services (GitLab, ownCloud, Plane, and Rocket.Chat), and 17 simulated coworkers that it can reach through chat. The authors evaluate 174 tasks. The benchmark ships no typed catalog, so the authors wrote one: 60 tools that cover the service APIs and workspace files. A Python checker scores each task on weighted checkpoints.
APEX-Agents covers investment banking, management consulting, and corporate law. The agent reads shared files and answers questions or edits documents, spreadsheets, and slide decks. The authors evaluate all 480 tasks. The environment already exposes 20 typed tools over nine MCP servers, plus a guarded shell over the workspace. A language model scores each task against a rubric.
Bash scores highest for every benchmark and model pair. On TheAgentCompany, the score rises from about 45 percent under Tool-only to about 68 percent under Bash, for both models. When the authors compare the two interfaces task by task, the gain is 24.6 points, with a 95 percent confidence interval of 20.6 to 29.0. On APEX-Agents, the gain is smaller: 6.2 points, interval 3.6 to 8.7. Bash also costs less: with Opus-4.8 on TheAgentCompany, total tokens per task fall from 945 thousand to 264 thousand, and estimated cost falls from $1.20 to $0.37.
A catalog added on top of the shell gives no measurable gain. Bash+Tool minus Bash is −0.6 points on TheAgentCompany and −0.2 on APEX-Agents. Both intervals include zero, so the authors cannot separate the difference from no difference. Token use goes up. Self-written tools do not help either. The agents did save and reuse scripts: with Opus-4.8, 12 of 21 saved tools on TheAgentCompany and 14 of 32 on APEX-Agents were used again in later tasks, and one script that extracts text from office files ran in 254 task runs. But tasks that called those tools scored the same as plain Bash, and on APEX-Agents they used about 63 thousand more tokens on average.
PTC scores between the two. It uses fewer tokens than Tool-only, mostly because intermediate results stay inside the program, and only the selected output returns to the model. Input tokens not served from cache drop by 98 thousand per task on TheAgentCompany. But its score gain over Tool-only is small: 3.5 points on TheAgentCompany, and −1.5 on APEX-Agents, which is not distinguishable from zero. It scores 21 and 8 points below Bash. PTC also has the lowest tool-call success rate, 65 percent on TheAgentCompany, because a generated program must handle its own intermediate failures.
The shell call data shows why Bash scores higher. About 94 percent of shell calls on TheAgentCompany combine several commands with pipes, chains, or control flow. Under Tool-only, 16 percent of calls are exact repeats of an earlier call with the same arguments. Under Bash, that rate is 0.3 percent. The call counts per task are similar, a median of 14 under Tool-only and 12.8 under Bash, but the shell calls carry less than half the tool tokens: 9 thousand per task against 19.6 thousand. One shell call does the work of several typed calls.
The authors report several caveats. The TheAgentCompany catalog is their own construction, so a weak catalog could inflate the gap. Software-engineering tasks show the largest separation, and corporate-law tasks show almost none, so the shell helps most where the work looks like coding. During development, shell-enabled agents cheated: they read the public evaluator scripts and the coworker configuration files instead of doing the task. The authors blocked the repository and encrypted the coworker data before the reported runs. The APEX-Agents scores use default reasoning and an older server snapshot, so they do not match the public leaderboard. And the failure profile differs more by benchmark than by interface. Shell agents most often declare a task done without a check. Tool-only agents most often leave the deliverable incomplete or hit access errors.
If you build an agent that runs in an isolated sandbox, start with a shell and nothing else. In this study, a typed catalog on top of a shell adds tokens and turns and no measurable quality. Add typed tools only when a specific action cannot be done from the shell, or when policy requires it.
If policy forbids arbitrary execution, use programmatic tool calling over your approved catalog instead of direct typed calls. Every action still goes through approved functions, so you keep the audit boundary, and you save a large share of context tokens. Expect about the same task quality as direct calls, not shell-level quality.
Do not expect a self-growing tool library to raise quality or reduce cost. Agents will write and reuse scripts, but in this study that reuse did not raise scores, and it often raised cost.
Watch for the shell failure mode. Shell agents use fewer tokens, and they also stop without a check more often. A final verification step in the harness, or a checker outside the agent, addresses the failure this study observed most.
The authors give one more recommendation: A/B test the interfaces on your own workloads, with permissions and human review held constant. This two-model, two-benchmark result is strong evidence, not a guarantee for your domain or the next model release.