In the Arena #4: ask for the twentieth run

AI agent reliability is not whether a task works once. It is whether the same task works repeatedly, leaves the system in the right state, and fails safely.

A note from Alex

Alex here. I spend most of my week talking with operators about where AI earns its keep, and the four reads this week all point to the same question: what happens after the demo works once?

A research team spanning Microsoft and three universities ran 507 business tasks twenty times each. The strongest model in the benchmark passed 65 percent on a single attempt, but it passed all twenty runs on only 25 percent of the tasks. That gap is the work. The same theme shows up in model buying and AI-written code: companies are mixing models by task, while the cost that matters moves from generating a change to verifying, reviewing, and governing it.

The last read is the useful counterweight. If you want a team to think bigger, give your most curious people room to build something ambitious. Just give that ambition a place to land: a real workflow, a clear owner, and a number you can check.

If any of this is sitting on your desk this quarter, tell me how you are thinking about it — email aschreiner@comavenai.com or grab thirty minutes with me. No deck, no pitch, just a working conversation.

- Alex

The short version

• One successful run shows capability. Repeated clean runs show reliability.
• Choose the least expensive model that clears the quality and risk bar for the task, then check the mix again next quarter.
• Measure the total cost of accepting safe work, not just the cost of generating it.
• Give AI-curious people ambitious work, plus an owner, an outcome, and a path into production.

Worth your time

How do you measure AI agent reliability? Ask for the twentieth run

One successful run proves that an AI agent can complete a task. It does not prove that the agent is reliable.

ThinkingBox-Bench contains 507 stateful business tasks across retail, hospitality, auto insurance, neobank IT, and consulting support. Each model ran every task twenty times. The benchmark checks the backend state and side effects after the work, not just what the agent says.

The strongest model, GPT-5.4, scored:

• 65.36% pass@1: one-attempt success
• 91.12% pass@20: at least one successful attempt in twenty
• 25.25% pass^20: all twenty attempts succeeded

Those are three different products. The first can make a good demo. The second says retries may find a winning path. The third begins to describe repeatability.

The exact leaderboard will age quickly, and these are realistic synthetic tasks rather than a representative sample of every enterprise workflow. The method is what travels. Before an agent touches a system of record, run your own high-consequence tasks repeatedly, inspect the final state and side effects, and define safe escalation.

Take this into the vendor meeting: what is the repeated-run success rate on our workflow, how is the final state verified, and what happens when the agent cannot finish safely?

Microsoft's ThinkingBox explanation · The August 20 paper · Open-source code

Which AI models are businesses actually buying?

Companies are buying model portfolios, not one winner. They sort work by what it requires, then choose on price inside that tier.

Ramp's lead economist, Ara Kharazian, described Anthropic's Fable 5 as the market's most performant model and its most expensive. In Ramp's technology-skewed token-spend sample, it accounted for 6 percent of Anthropic tokens and 11.4 percent of model-attributed Anthropic spend in its first month.

On Vercel AI Gateway, Anthropic handled 29.8 percent of July tokens but captured 65.1 percent of estimated spend at published list prices. Open-weight models handled 36 percent of tokens for 8.6 percent of spend. Among teams running more than ten million tokens in both June and July, three in four changed at least 10 percent of their model mix.

The operating model is straightforward: set the quality and risk bar for each task, use the least expensive model that consistently clears it, reserve premium models for work where a bad result is expensive, and re-run the evaluation at least quarterly.

Tier the work, not the vendor.

Ramp AI Index, August 2026 · Vercel AI Gateway Production Index, August 2026

What should teams measure when code gets cheap?

Measure cost per accepted change: generation, environment, context, verification, review, remediation, governance, and everything else required to put a useful change into production safely.

GitLab CEO Bill Staples makes the case with company-reported production numbers. Stripe merges more than 1,000 agent-written pull requests each week. Spotify's background coding agent has more than 1,500 merged pull requests. Amplitude cut pull-request cycle time from 5.2 hours to 44 minutes, tripled pull-request volume in six months, and saw reported monthly bugs fall from 715 to 319.

That does not prove AI produces better code. Amplitude earned the result by rebuilding the environment around the model. Its frontend continuous integration (CI) pipeline fell from about thirty minutes to three or four. As Staples puts it, “a thirty-minute CI pipeline defeats every model you point at it.”

The same math is coming to finance, operations, and support. Model generation will look cheap. Verification, exceptions, permissions, review, and accountability will not.

Ask your team: where do time, money, and human attention accumulate between generated work and work you can safely accept?

Bill Staples, “When code is abundant” · Stripe's Minions · Amplitude's six-month overhaul · Spotify's Honk

How do you get a team to think bigger with AI?

Give the most AI-curious people permission to build ambitious projects, then give those projects a route into the real operation.

That was Every's answer to the first of 33 questions submitted by 400 executives. Natalia Quintero, who leads Every's consulting practice, and Mike Taylor, its head of evals, argue that “it's easier to raise the ceiling than the floor.” One fluent person building ten useful skills can create more leverage than asking ten reluctant people to each build one.

It is a point of view, not a benchmark. We would add one requirement: give the ceiling-raiser somewhere to put the work. The project needs a real workflow, an accountable owner, the people who know the process, a measurable outcome, and a path to production. Otherwise it stays a side project.

Read all 33 executive questions in Every. Every asks readers to sign in for free to continue.

From the team

Justin wrote down what forward-deployed engineering actually means. Being in a client's Slack and Jira is table stakes. The real requirements live with the dispatcher, controller, and operations lead doing the work. The job is to build with those people instead of around them. Read what forward-deployed engineering really means

Missed last week? Issue 3 covered the build that beat the buy, Mistral's five verbs, Linear's data on who is shipping with AI, and the agent that found the answer key.

See you in the arena.

Have a workflow that deserves better? Grab thirty minutes with us.