mo eval

Every toolchain change is a bet.
mo eval is how you know if you won.

Every model, harness, or skill change is a bet that things got better, faster, cheaper, or more reliable. A quick spot check on a couple of runs, or trusting that public benchmarks actually reflect your team's workflow, can make the bet look like a good decision. mo eval replaces the guess with evidence, basing tooling decisions on quality-per-dollar, catching regressions before they ship, and keeping a record of why each decision was made that can be reproduced months later.

Rigorous evaluation is what turns “it seems better” into decisions you can defend.

Model and harness selection.

Choosing the best quality-per-dollar option for a specific workload, not just the one that felt fastest in a demo.

Regression detection.

Catching quality, cost, latency, and behavioral regressions before they ship.

Improvement diagnosis.

Understanding why a task succeeded, failed, stalled, or spent more than expected.

Defensible decisions.

Avoiding gut decisions by using explicit questions, reviewable evidence, and claims based on the supporting evidence.

Reusable evidence.

Comparing multiple eval runs without spending new tokens.

Scope

Tested as a system, not just a model

Most evaluation stops at the model. mo eval tests the entire toolchain your developers actually run, the model, the harness executing it, the MCP tools it calls out to, and the skills and context it's given. A model that scores well in isolation can still fail in production if the harness mishandles a tool call or the wrong context gets passed along, testing the pieces separately doesn't tell you whether the whole chain gets the job done together.

Evaluation tasks come from your own repositories, code changes, bug fixes, generated tests, refactors, and internal tooling, instead of simple problems in a public repository designed to be easy to grade. Your team determines task success criteria, including evals staying within your quality, cost, and latency targets.

Your company tasks

Built on your processes and repositories.

  • Code changesNew features, refactors, or bug patches across services.
  • Bug fixesFix regressions in code, processing, or pipelines.
  • Generate testsAdd unit, end-to-end, and discovery test automation.
  • RefactorsImprove or re-architect services and workflows.
  • ToolingBuild internal CLIs, scripts, and utilities.

Acceptable results

Your success is verifiable.

  • Passes testsAll tests are clean.
  • Meets quality barCode quality, style, and standards.
  • No regressionsExisting functionality behaves as expected.
  • Within cost and latency targetsStays within budgeted cost and latency.

Judging

A judge that isn't grading its own work

Every run is scored across multiple dimensions, such as correctness, quality, and completeness, by a judge trained specifically for evaluations. Separating the judging process from the models under test eliminates the hallucinations that have been known to happen when models report their own test results. Human oversight and spot checks make sure the grading system is held to your team's quality bar.

AI toolchain under test (model, harness, tools, and context) produces multiple runs of the same task. An independent judge applies accuracy, quality, completeness, style and clarity, safety, latency, efficiency, and cost criteria with human oversight and spot checks, producing per-run decisions that roll up into a final verdict across runs.

Repeated runs

One passing run doesn't tell the whole story

Complex prompts, run against the same toolchain, won't produce identical results every time. A model can pass a task twice and quietly regress on the third attempt, on cost, latency, or behavior, not just correctness. Making a tool change on a single run risks mistaking noise for signal. mo eval runs each task multiple times and uses the distribution of outcomes to be confident that the change is an actual improvement.

Weekend script

Useful for a point-in-time check.

  • One-off runsPoint-in-time results that can’t be compared reliably.
  • Small samplesToo few tasks and runs to be statistically meaningful.
  • Limited visibilityBasic logs with little context or explanation.
  • Hard to reproduceEnvironment drift and missing details break reproducibility.
  • Gut decisionsBased on hunches, not evidence.

mo eval

Built for evidence you can defend.

  • Continuous, repeatable runsScheduled evaluations you can compare over time.
  • Large, diverse coverageMany tasks and repeated runs for statistical confidence.
  • Deep observabilityCost, quality, latency, traces, and explanations.
  • Reproducible by designImmutable inputs, environments, and versioned results.
  • Defensible decisionsEvidence you can review, share, and trust.

Proof

What this looks like in practice

As of July 2026, mo eval tested GLM 5.2 and Claude Opus 4.8. GLM 5.2 came in at 0.4 times the cost of Opus, while scoring within a tenth of a point on a 0 to 5 quality scale (4.2 vs. 4.3). Both models passed every automated test in the suite. mo eval enables your team to take actionable and defensible toolchain decisions.

Infrastructure

Meaningful results in the smallest number of test runs

Running enough tests for a statistically meaningful answer adds up fast. 20 tasks, run 5 times each to converge on an answer, across 2 harness-and-agent combinations, is 200 runs for a single model change. Simple test systems rerun all combinations under test every time. 4 model changes with 200 test runs is 800 runs. mo eval only re-tests what's actually new and reuses prior baseline results instead of rerunning them, so the same scenario would take only 260 runs.

That efficiency compounds because results are saved as a permanent, queryable record. Every run's environment, inputs, and outcome are versioned and preserved, so a comparison from three months ago can be reconstructed and re-checked. Simple test systems may not count runs that get stuck or time out as failures. Updating the script also gets harder once the person who wrote it has moved on to another project.

Naive approach, re-run everything: 20 tasks x 5 runs x 2 harness/agent combos, 200 runs per model change, 800 runs total after 4 model changes — costly, slow, wasteful. Smarter approach, only run what's new: 200 runs for the first model change, then 20 runs for each subsequent change by re-testing only new tasks, 260 runs total after 4 model changes — smarter, faster, lower cost.

Get started

Put it to the test

Send a few of your team’s real repositories and see the benchmark data, cost comparison, and gateway configuration before you commit to anything.

Request a workload audit →

Further reading

The full argument

For additional details, the statistical reasoning behind repeated runs and the infrastructure challenges most teams don't see coming until they've already built around it, dive deeper.