Verifying AI toolchain updates is hard

I asked the mo development team what was stopping me from vibe coding an AI model and toolchain test system, like mo eval, in a couple of hours. They suggested I needed to revisit statistics and also decide if I wanted the company's tooling and model decisions based on a couple comparison runs of the toolchain, locked to a point in time. Then we got into the details.

Here’s the scenario that we talked through, someone on your team says their prompt resulted in faulty code that broke two other tests in the suite. You pull up the transcript. Something went wrong and you have a hypothesis. You make a change, try the test case again, and it looks better.

Did it really get better? Are there regressions someplace else in the system? If you’re working from a system you built over the weekend, you’re not going to be able to confidently answer these types of questions. Determining if a model or tooling chain fixed an issue and didn’t break something else is always a harder question than it looks.

Every change made to a harness, coding agent, model, prompt, tool policy, etc. is a bet. Does the quality improve, does it cut costs, does it do both, or are things getting worse? A quick spot check plays into that bet and can easily be misleading. When it comes to the core infrastructure we don’t want to be making bets, we want to have solid decisions with firm reasoning behind them.

The types of questions that mo eval has been created to answer are not ones that my weekend vibe coding project could answer:

• Was the task actually solved when judged by an authoritative score, or did the result just read fine on the way through?
• What did that attempt cost, in dollars, tokens, calls, turns, and wall time?
• Is the difference stable across tasks and repeated runs?
• Why did the run succeed, fail, time out, or turn out expensive?
• Can the results, and the decision's justification, be reconstructed three months later?

I’ve come to understand that there are two separate reasons this kind of eval for AI tooling is hard to build. First is that even with perfect tooling, making the decision of what counts as a success and isolating cause from coincidence is a unique discipline leaning heavily on statistics. Second is the operational problem. Running the tests required to create statistically significant results requires the scale and reliability of a purpose built infrastructure, separate from the experiment's design.

Let’s dig into these two ideas of statistically meaningful results and the infrastructure challenges required to get those results.