← Writing

You can't unit-test a system that answers differently every time

Evaluating agents when the same input yields different, equally correct outputs.

Traditional testing assumes determinism: given input X, expect output Y. Autonomous agents break that at every level. The same request run twice can produce different tool calls, different reasoning, and a different final answer, all of which can be equally correct.

DAE is the operational layer for that reality. It models evaluation as a pipeline: capture the full interaction trace, score it, compare against a baseline, gate on business risk, and learn.

Scoring runs on two tracks. Deterministic schema checks always run, in under a millisecond. An LLM judge runs sampled by risk tier, so I am not paying a model to grade every single call.

Gating is tiered by how much the decision costs the business: auto-execute, human-on-the-loop, or human-in-the-loop. Scoring is dimensional, not a binary pass or fail, because a mostly-right answer is real information.

The point is to catch regressions in a system that is non-deterministic by nature, without either trusting it blindly or grading everything it does by hand.

Distilled from my DAE RFC. The full RFCs are on github.com/alinaqi.