Comparing AI toolchains is easy, but can you trust the results?

Most home-built evaluation harnesses stop being trustworthy due to the lack of scientific rigor backing the results. A single passing test, under a specific circumstance, does not guarantee long-term results across a broad range of use cases. Creation of an evaluation harness actually needs to start by considering the possible outcomes and working backwards. If you vibe code a webpage where you input a code snippet and select three models to compare, you’ll have a working prototype before the afternoon is over, but you’ll be missing the most important parts of the evaluation. The gap between a quick comparison and something you can actually trust is why mo eval was created.

“Correctness is not as clean as it sounds.”

Going from "the AI toolchain eval process finished" to a verdict of correct, incorrect, errored, or any other number of outcomes isn’t straightforward. Someone, or something (with human oversight), has to create the detailed and nuanced definitions for every possible outcome for each task under test, and those definitions are more important than they get credit for.

An eval task is a stand-in for a real request, and real requests are almost never as clean as the test cases. When you ask a person to complete a task, you don’t hand them a checklist to grade themselves against. An eval system needs to be scoreable, so ambiguity is turned into a fixed set of criteria. Those criteria can’t capture every valid way to solve a task, so some correct solutions get marked wrong, and some technically passing ones don’t actually solve the problem.

When judging the results of AI tool evaluations, clean pass/fail criteria are not possible outside of the most simplistic tests. For any real tests, a decision tree is needed to determine where the results land. Acceptable, partial, degraded but recoverable, correct but slow, etc. Building that tree is part of the evaluation system as much as determining what the tests actually are and which toolsets should be tested. Even with a well-built decision tree, there needs to be a trustworthy judge to apply it.

Ultimately, to turn the process of running evals into a production line, you need to be able to rely on AI tools to correctly report on the results of the tests. In practice though, the same models that are being evaluated repeatedly misread log files and report results that aren’t factual. A model that is trained to judge the results has to be independent of the system being evaluated. So, beyond just testing the results, the judge of the results should be continuously monitored and the judgments verified to be within the bounds of acceptable results.

“The result you got once might not be the result you'd get again.”

The model outputs come in the form of a distribution of possible outcomes. When given a complex task, the same AI toolchain will produce a slightly different result every single time. Determining which of those variants are considered passing results means running the same tests repeatedly. But, what should actually count as a reliable result? Getting an answer that the judge decides is correct five times out of five? Three times out of five? Is one time out of five meaningful or is it the same as zero out of five? Every task is different, and there is no default answer to this question. Costs can increase dramatically as teams try to answer these questions, and a few expensive outliers can dominate the end results. The results from multiple runs can raise issues that a single run wouldn’t expose.

Two distinct concepts fall into the category of “the results aren’t reliable.” One is that the system is stochastic, more similar to rolling a die repeatedly to find the shape of the distribution versus expecting one exact answer. The other is the experimental design of the tasks themselves. That means asking if the tasks are representative of those the business actually cares about, whether variables can be isolated, and whether you get the same results when the tests are repeated.

Being pragmatic about what is under test avoids scenarios where three parameters (for example, harness, model, version) are changing and the results aren’t attributable to any of them individually. An improvement in one may mask a regression in another. Single-variable comparisons, run multiple times, separate simultaneous effects. Understanding what changes are actually impacting your tests is a lot more work, which leads to this often being ignored by basic testing systems.

"Does this work" and "have I confirmed this works" are different questions.

The science behind benchmarking and evals is much closer to applied statistics than software development. Both baseline and confirmatory experiments, with repeatable results, are required to really understand the full AI toolchain and to justify any decisions made. They need to hold up if they are rerun three months later. Trustworthy results come from managing evaluation as a statistical discipline with questions known before tests are run, repeatable results, and analysis that’s independently verified.

mo eval exists to fold that statistical discipline into AI toolset evaluations by default. Explicit questions and expected results should be known before a single token is spent. Recursive testing with known target results and a judging system that has been vetted and trained are the final keys to robust, repeatable test results.

That’s the difference between vibe coding an eval tool and something you’d make a budgeting decision on. Understanding what to measure and building the systems to do it are only part of the challenge. The next post will take a look at how you keep the infrastructure running reliably.