For engineering leaders choosing AI coding agents
A/B test your coding agents on your own code.
Your engineers' real tasks run on two agent setups at once, and they pick the better result without knowing which made it. Yardstick turns those picks into a clear answer: which setup to standardize on, and what each one costs.
Request a pilot See how it works
Kimi K3 is no worse than Sonnet on your work, at a fifth of the cost.
| OpenCode · Kimi K3 | Claude Code · Sonnet | |
|---|---|---|
| CI passed | 86% | 88% |
| Cost per task | $0.28 | $1.40 |
Illustrative: one campaign, 370 tasks judged blind by 22 engineers. The bar is the 95% interval for how often engineers preferred Kimi K3. It stays clear of the line, so Kimi K3 is no worse on this work — how we get the answer.
Most teams are choosing blind.
New models and agents keep arriving, at very different prices. Most teams still pick between them on public benchmarks, vendor marketing, or the loudest opinion in the room.
benchmarksThey aren't your work
Public benchmarks use standard tasks, not your codebase, your conventions or your tests.
impressionsOpinions follow brands
People favour the tool they already know or the brand they like. Hide the label and the preference has to come from the code.
costPrice is the missing half
A setup that is slightly better at five times the price may not be worth it, and nobody has the numbers to say.
A blind taste test for coding agents.
To find out which of two coffees is better, you don't ask people which brand they like. You pour both, hide the labels, and let them taste. Yardstick does the same for coding agents, on your own work.
Your engineers work as usual
They hand Yardstick a real task, the same way they would ask any coding agent.
Two setups take it on
Say Claude Code with Sonnet against OpenCode with Kimi K3. Same code, same instructions, same time limit.
The engineer picks blind
Two results, labelled only A and B. They pick the better one and keep it.
You get a verdict
Which setup to use on your work, with what each one costs alongside.
Your engineers stay at the helm.
Nobody is handed a weaker agent for the sake of a test. Every task produces two options and the engineer keeps the better one, so the experiment never costs your team its best work. Engineers are the judges. The agents are what is being measured.
Your code stays where it should.
It runs in your own GitHub Actions with your own API keys. The keys stay on your runners and never reach Yardstick.
Your code never goes to China-based agents or model hosts, even when you test a Chinese-built open model like Kimi K3.
Built by people who measured experiments at scale.
Yardstick comes out of years of building experimentation and ads marketplace systems, where a badly measured test costs real money and the statistics have to survive scrutiny. We are bringing that discipline to the newest, least measured line item in engineering.
Standardize on evidence.
We are starting pilots with a small number of engineering teams that already use AI coding agents. Bring the decision you're stuck on, and let your engineers settle it on real work.