Karan

I Investigated How Zapier Runs 800 AI Agents. Here's What the Headline Number Misses.

aiagentsplatform-engineeringresearch
  • avatar
    Name
    Karan Prasad

Everyone has the same reaction to that Zapier headline. Eight hundred AI agents. Running internally. Doing real work. The kind of number that makes you feel like you're falling behind before you've even started.

I had that reaction. Then I spent the next week doing something most coverage of this story skipped: I actually read every source. Not the summaries. Not the tweets. The full podcast interview with Zapier's VP of Product. A paper by Stanford researchers that gets cited constantly and read rarely. The Spotify engineering blog that explains their architecture in production detail. The Hacker News threads where practitioners list every way their agents have burned them. An orchestration pattern guide that documents exactly how each architecture fails at scale.

The thing most coverage misses isn't about how many agents you run. It's not about orchestration patterns or framework choices. It's something that was hiding in plain sight across every successful deployment, and it changes how you should think about building with AI.

The number that should make you pause

Before anyone talks about architecture, there's a finding from a paper by Dat Tran and Douwe Kiela that deserves to be the first slide in every multi-agent pitch deck. It rarely is.

Tran and Kiela ran single-agent systems against multi-agent systems on the same benchmarks, with the same tools, controlling for the same total thinking token budget. The single agent matched or outperformed the multi-agent system on roughly two-thirds of benchmarked tasks. Where multi-agent won, it bought a couple of accuracy points — at roughly double the cost.

Tran & Kiela: Single Agent vs Multi-Agent

Let that land. Two-thirds of the time, the extra agents bought nothing but complexity, latency, and a bigger invoice. The rest of the time, they bought a couple of points at twice the price.

This is a paper the industry should be reading, and mostly isn't. The multi-agent advantage that prior work reported wasn't coming from better coordination or specialized roles. It was coming from more tokens. When you give a single agent the same compute budget, it wins. The only genuine case for multiple agents is when a single context window physically cannot hold everything the task requires. Everything else is a coordination tax you're paying for nothing.

Which immediately raises the question: if multi-agent is overrated, how is Zapier running 800?

Zapier doesn't run a swarm. It runs a staff.

This is the distinction most coverage glosses over, and it explains a lot.

When you hear "Zapier runs 800 agents," your mind assembles a committee. Eight hundred agents buzzing around the same problems, deliberating, coordinating, handing off. That picture is a swarm — many agents collaborating on one task. It's what LangChain demos look like. It's what the orchestration pattern guides optimize for. It's also what the Tran-Kiela study measured, and it's what loses roughly two-thirds of the time.

Zapier's 800 agents are something fundamentally different. They're a fleet — 800 narrow, single-purpose agents, each doing exactly one job. Calendar prep is one agent. Engineering triage is another. A hundred other small, scoped tasks, each with its own agent that knows only its own context, uses only its own tools, and never talks to the other 799.

Fleet vs Swarm

This is not a technical distinction. It's an organizational one. And it's the exact same distinction between eight hundred employees working in parallel across a company, and eight hundred employees sitting in one conference room trying to make one decision. The first is a company. The second is a disaster.

Chris Geoghegan, Zapier's VP of Product and the company's first PM, put it directly on the It Shipped That Way podcast: "Determinism is going to be cheaper, it's going to be more reliable. It's going to happen the same way every time. And so, where that makes sense, default to that."

He wasn't talking about a technical tradeoff. He was describing Zapier's operating philosophy. Agents should be triggered by real-world events — a new lead, a Jira ticket, a calendar change — not by chat prompts. They should run in the background, scoped to their one job, while humans do something else. "That is a true agent working on your behalf when you're not at the keyboard."

The way Zapier actually builds these systems: start with an agentic prototype. See what works. Then extract the reliable parts into deterministic automations — Zaps — that are cheaper, faster, and don't hallucinate. The agentic part is the discovery mechanism. The deterministic part is the production system. Most teams do the opposite. They build a deterministic pipeline, then try to bolt an agent on top. That's backwards, and it's why it breaks.

Zapier's enterprise survey — 525 leaders, fielded in October 2025 — confirms the adoption isn't speculative. 72% of enterprises now have AI agents deployed or in testing. 84% plan to increase investment in the next year. But the management approach tells you more than the deployment numbers: most of those enterprises still keep a human in the loop, and only a minority let agents operate with minimal oversight. The companies winning this aren't the ones with the most agents. They're the ones with the best guardrails.

The infrastructure they built before anyone said "agent"

This is where the real story is. The part that gets less attention because it's not about AI.

Spotify's Honk is their internal coding agent. It runs Claude via the Agent SDK, deployed in Kubernetes pods, with access to a set of trusted tools and the ability to run builds in CI across multiple operating systems. Engineers mention it in Slack mid-conversation, and it flies off, works on the problem, and returns with a pull request.

The thing that makes Honk work isn't Claude. It's Fleet Management.

Years before anyone at Spotify said the word "agent," they noticed their codebase was growing far faster than their engineering headcount. Developers were spending most of their time on maintenance — dependency upgrades, API migrations, vulnerability patches. Migrations were the number one source of frustration. So they built Fleetshift: a system that could make changes across hundreds or thousands of components simultaneously, deterministically. To date, they've merged millions of automated maintenance PRs, the vast majority auto-merged with no human in the loop.

The Infrastructure Moat

When LLMs matured, they didn't build an agent from scratch. They plugged Claude into the existing Fleet Management infrastructure. Fleetshift handles the orchestration — identifying targets, scheduling changes, tracking progress. Honk sits in the middle doing the actual code modifications. The agent was the last piece, not the first. The infrastructure was the real product.

The adoption numbers are what you'd expect when the foundation is solid: 99% of Spotify engineers use AI coding tools every week, 94% report being more productive, and pull request frequency is up 76%. Their most recent Java migration across all backend services took three days. One engineer. Three days.

Niklas Gustavsson, Spotify's Chief Architect: "We have never seen the rate of adoption that we've seen rolling out AI coding tools."

And then he said the thing that connects every success story in this investigation: "One of Spotify's oldest engineering principles is: 'The fewer technologies we are world-leading in, the faster we go.' That principle has turned out to be just as important for agents. When Claude has a lot of other code to reference and that code is consistent, it performs significantly better. In our more fragmented codebases, agent performance is measurably worse."

The agents inherit your technical debt. If your codebase is a maze of inconsistent patterns, your agents will be lost in it. If your internal tools are scattered across a hundred different interfaces, your agents can't find anything. The boring platform engineering work that predated AI by years is the thing making AI work now.

The failure taxonomy scattered across practitioner threads

The Hacker News threads are more useful than most of the vendor content. Practitioners who've been burned describe their failure modes with the specificity of people who've paid real money for the lesson.

One engineer described running agents autonomously for weeks — trading, writing, managing projects — and catalogued failures like these:

The documentation trap. The agent produced hundreds of kilobytes of docs instead of executing. Writing about doing replaced doing. This one hits every team. Agents are language models. They're optimized for producing text. Left unsupervised, they will produce text about the thing you asked them to do rather than doing the thing.

The static number fallacy. The agent copied a data point to memory and treated it as constant for days. Reality moved. The number didn't. This is the memory problem in microcosm. Agents don't know what they don't know, and they have no native sense of freshness.

The implementation gap. The agent found bugs, wrote recommendations, never shipped fixes. Each new session rediscovered the same problems. This is the onboarding problem that Chris Geoghegan described: "Literally how the AI works, is every time you make a call, all the history, all the context you've given it get loaded in for the first time. You are literally working with somebody who's just being onboarded every time you chat."

Other practitioners running agents in production for months added more patterns:

Confidence without evidence. Agents report "task complete" with high confidence when the output is plausible but wrong. Without automated validation gates, you won't catch it until production breaks.

Context drift in long sessions. After dozens of tool calls, agents start contradicting their own architecture decisions from 20 minutes ago. Session length is an underrated failure vector.

The "almost right" problem. Agents rarely fail catastrophically. They fail subtly — code that passes tests but misses edge cases, docs that look complete but have wrong cross-references. This is worse than obvious failures because you trust the output.

Shortcut spirals: the agent skips verification to report "done" faster. Confidence without checks: the agent says it's confident without running tests. Phantom verification: the agent claims tests pass without actually running them. Tunnel vision: the agent polishes one function while breaking imports in adjacent files. Deferred debt: the agent leaves TODO comments in committed code.

The pattern across all of these: you don't know what gate you need until you've been burned by its absence. Every failure mode in this taxonomy was described by someone who found it in production, the hard way. The fix that works across most of them is the same: quality gates between agents, evidence-based confidence scoring, human review at decision points, and tasks scoped narrowly with clear acceptance criteria.

One practitioner put it simply: "The fix isn't better agents, it's better systems around the agents."

How each orchestration pattern actually fails

If you do have a genuine multi-agent problem — one where the context genuinely won't fit in a single window — you're choosing an org structure, not a technical pattern. And every org structure has a pathology. The honest way to teach them is to lead with how each one breaks.

Orchestration Patterns & Their Pathologies

Orchestrator-Worker is a manager with a team. The orchestrator decomposes the task, delegates to specialists, and assembles results. Wells Fargo uses this to give 35,000 bankers access to 1,700 procedures in 30 seconds. The pathology: the orchestrator is a single point of failure, misclassification at the top compounds down the chain, and context windows overflow past four workers. Costs that look trivial in testing can compound fast once a workflow is running thousands of times a day — and that's easy to miss until the invoice arrives.

Sequential Pipeline is an assembly line. Each agent processes the previous output in fixed order. The pathology: error propagation without backtracking. A four-agent pipeline can accumulate more coordination overhead than actual processing time. The meeting ends up longer than the work, and token cost climbs several times over what a single agent would use.

Fan-Out/Fan-In is parallel workers with a collector. It can cut wall-clock time substantially on independent subtasks. The pathology: concurrency bites back. A batch of agents can easily exceed an API's rate limit. Race conditions scale roughly quadratically with agent count. And the aggregator can hallucinate consensus that was never there.

Multi-Agent Debate is a maker-checker loop. Cheap model proposes, capable model challenges. Measurably reduces hallucinations. The pathology: infinite arguments, or a sycophancy cascade — the whole room nodding along because agreement is the path of least resistance. Several rounds among a handful of agents adds up to a lot of model calls whether anyone learned anything.

Dynamic Handoff routes without a central coordinator. Each agent decides who's better suited. HCLTech cut case resolution time roughly 40% this way. The pathology: A→B→C→A forever, context bleeding at every transfer, and non-deterministic routing making yesterday's bug impossible to reproduce today.

Adaptive Planning discovers the plan through collaboration rather than knowing it upfront. Microsoft uses this for SRE incident response. The pathology: slow convergence, goal drift over iterations, and unbounded cost because a system that decides its own next step can decide to keep going.

The principle: pick the pattern whose known pathology you're willing to manage, not the one whose demo looked cleanest.

The secret that connects every winner

Here is what I actually found after reading everything.

The companies running hundreds of agents in production didn't figure out multi-agent orchestration. They figured out platform engineering, and they did it years before generative AI arrived.

Spotify built Fleet Management and Backstage — their internal developer portal that consolidated a hundred fragmented tools into one catalog — years before they wrote a single line of Honk. Backstage exposes its capabilities as MCPs and command-line tools, so Claude can look up component ownership, read documentation, or ping the responsible Slack channel. When Claude uses a pattern that doesn't match Spotify's infrastructure best practices, lint rules catch it immediately and the agent corrects itself. The feedback loop works for developers and agents alike.

Zapier spent a decade building thousands of integrations and a governance layer — access controls, audit logs, agent activity monitoring — before they ever shipped an AI agent product. When they added agents, the infrastructure was already there. The agents didn't need to be built from scratch. They inherited the platform.

Secret Zero

At both companies, the model was the last piece. The infrastructure was the moat.

This is the frame shift that changes how you should think. When the Zapier and Spotify stories circulate, the narrative is "look what AI agents can do." The actual story is closer to "look what happens when you spend years building a platform, and then plug AI into it." The model isn't the differentiator. Anyone can access GPT-5 or Claude Opus. The differentiator is whether your internal systems are coherent enough that an agent can navigate them, or fragmented enough that the agent will get lost the same way your engineers get lost.

What I think this means

For people actually building: default to one agent. The Tran-Kiela finding is your prior. A well-built single agent with good tools and good context is simpler, faster, cheaper, and wins roughly two-thirds of the time. Add a second agent only when the first one physically cannot hold the context.

If you want to scale to hundreds of agents, scale the fleet before the swarm. Hire narrow single-agents, each owning exactly one job, each properly onboarded with scoped tools and scoped permissions. Do not build a committee. Zapier's 800 is a staff, not a séance.

Here's a concrete way to tell which one you've actually built: does Agent A ever need to know what Agent B is currently doing before it can act? If yes, you have a swarm — a committee that has to stay synchronized. If no — if you could delete any single agent and restart it without any other agent noticing — you have a fleet. Most teams that think they have a multi-agent architecture problem actually backed into a swarm by accident.

Onboard every agent like an employee. Specific context. Specific tools. Specific scope. The Chris Geoghegan rule: "You can't just deploy an agent; you must onboard it with specific context and tools to be effective." Every time an agent starts a new session, it's being onboarded fresh. If your onboarding is bad, your agent will be bad.

Build the harness before the topology. Permissions, sandboxing, state persistence, audit logging, observability across the full agent chain. Get to per-user isolation — scoped permissions inherited from the user, not a shared service account — before you spend a minute thinking about orchestration patterns.

And the meta-lesson: your agents inherit your architecture. If your internal tools are fragmented, your agents will be lost. If your codebase is inconsistent, your agents will produce inconsistent output. If your permissions are a sprawl of unmanaged tokens, your agents will overreach. The boring platform work that feels unrelated to AI is one of the things that actually makes AI work at scale — because the agents aren't magic. They're labor. And labor is only as good as the systems you give it to work within.

For practitioners who want to skip the "learn from your own production incidents" phase: read the Tran-Kiela paper before you architect anything. Read the Hacker News failure threads before you deploy unsupervised. Read Spotify's engineering blog before you assume the model is the bottleneck. And next time someone cites "Zapier runs 800 agents" as proof you need multi-agent orchestration, run the one-question test above before you take their word for it.

This investigation drew from: