There are two common ways to decide automatically whether an AI agent behaved as it should. One asks another model: show it the run and let it grade. The other measures: compare the run with the agent's history using fixed statistics and rules. Both are in production use, and they are good at different things.
This page compares them on the properties that matter for monitoring agents in production: repeatability, cost, privacy, attack surface and what each can see at all.
How an LLM judge works
An LLM judge is a model prompted to evaluate output, often against a rubric: was the answer correct, helpful, safe, on task. It is flexible. It can assess open-ended quality that no simple metric captures, and it needs no history to start.
Its weaknesses are documented. Zheng and colleagues, in the 2023 paper that introduced MT-Bench, found that LLM judges favor the answer shown first, favor longer answers and tend to favor output from their own model family. Each of these can be reduced with careful prompting, and none disappears entirely.
Where the judge struggles in monitoring
- Repeatability. Model output is sampled. The same run can receive different grades on different days, and even at temperature zero providers do not guarantee identical output. A score you cannot reproduce is hard to defend in a review or an incident report.
- Cost and latency. Every judged run is at least one more model call. On a busy agent, the judge can cost a meaningful share of what the agent itself costs.
- Content. To grade a run, the judge has to read it: prompts, retrieved documents, responses. That content goes to another model, and often to another provider.
- Attack surface. A judge reads the same material the agent read. Text crafted to manipulate the agent can be crafted to manipulate the judge as well.
How deterministic detectors work
A deterministic detector applies a fixed rule or statistic to what the agent did. Did it call a tool it has never used for this task? Did the mix of tools shift abruptly compared with its baseline? Has the distribution drifted over the last runs? Did it repeat the identical call several times in a row?
The same input always produces the same result. Anyone with the data and the method can rerun the calculation and get the same number. The detectors work on metadata (which tool, in which order, in which session) and do not need to read content. They are cheap enough to run on every call.
Where deterministic detectors struggle
They only see what is in the data. A detector on tool calls cannot tell whether a well-formed answer is factually wrong, or whether a summary missed the point. It needs a baseline of normal runs before statistical detectors mean anything, although simple rules, like a loop or a never-seen tool, work from the first run. And its thresholds have to be chosen and tuned, which is engineering work, not magic.
Which to use
They answer different questions, so many teams will want both. A judge is the better tool for grading open-ended quality, offline, on samples: evaluation before release, spot checks, research. Deterministic detection is the better tool for continuous monitoring in production, where every run is checked, the result must be reproducible and content should stay where it is.
The cases where we would not rely on a judge alone are those with consequences: alerts that page someone, scores that gate what an agent may do, evidence that ends up in an incident report. There, a number that comes out the same every time is worth more than a nuanced opinion that might not.
Kyde's approach
Kyde Score uses deterministic detectors only: CUSUM for abrupt shifts, windowed Jensen-Shannon divergence for gradual drift, a per-task contract for never-seen tools, and a rule for repeated identical calls. There is no model in the loop and no prompt content is read. The drift detectors are open source, so anyone can rerun them. The task contract is not public.
KYDE Score
Kyde Score scores every run from 0 to 100 with deterministic detectors: CUSUM, Jensen-Shannon divergence, a task contract and a loop rule. Same input, same score. No LLM in the loop and no prompt content read.
Score an agent →