(drift detection)

AI agent drift detection

If you run agents with access to tools, the costly failures rarely come with an error message. An agent gets steered by content it read, gets stuck in a loop, or starts behaving differently after an update, and the first sign is often a bill or a leak. Drift detection compares every run of an agent with how that same agent normally behaves, and tells you when the difference becomes real.

In production

[3 patterns]

What agent drift looks like in production

The research on behavioral drift describes at least seven related forms, which we cover in Behavioral Drift in AI Agents. For teams running agents today, three patterns account for most of the practical risk.

Hijacked

The agent read something and changed course

A coding or support agent reads a prepared web page, email, issue or file and then does something it was not asked to do. The instruction is hidden in the content, so nothing looks wrong in the prompt. What changes is the behavior: a tool the agent does not normally use for this task, a new target, an unusual sequence of calls.

Detecting indirect prompt injection in AI agents →

Looping

The agent repeats itself and burns money

An agent retries the same call, re-plans the same step or calls itself through another agent. Each step looks valid on its own. Taken together, the run makes the identical call again and again, and the cost follows.

Detecting AI agents stuck in a loop →

Silent change

The agent behaves differently after an update

A model version changes, a system prompt is edited or a tool description is updated. Nothing fails, but the agent's mix of tools and the order in which it uses them shift over days. This is the kind of drift that rarely produces an error and often surfaces first as a complaint or an invoice.

Method

[deterministic]

How drift detection works in Kyde Score

Kyde Score reads the OpenTelemetry your agent already emits: which tool was called, when, in which step and with what result. It does not store prompts or payloads. Each agent learns from its first 30 runs over at least 3 days, and that baseline is then frozen and fingerprinted, so later runs are always measured against the same reference.

Every run after that passes three detectors. Their results combine into a score from 0 to 100: 70 to 100 is normal, 40 to 69 is worth watching, and below 40 the agent is deviating from its own baseline. When the score changes band, Kyde sends a webhook signed with HMAC-SHA256 to the URL you choose.

Abrupt shift

CUSUM

Accumulates small deviations from the baseline and fires when they add up to a real break. Catches the run where an agent suddenly starts behaving differently.

Gradual drift

Windowed Jensen-Shannon divergence

Compares the distribution of recent behavior with the baseline distribution. Catches a clear slow shift after a model or prompt change that no single run would reveal. Small changes it does not reliably see.

Task contract

Strict rule per task

Records which tools an agent used for each kind of task during the baseline. A tool that never appeared for that task is flagged on first use. With the optional customer-side collector profile, new targets such as merchants or recipients are covered too, as salted hashes.

The detection math is open source in kydehq/behavioral-drift-detection.

Why statistics

[no LLM judge]

Why the verdict comes from statistics, not from another model

A common way to monitor agents is to ask a second language model whether the first one behaved well. That can be useful for reviewing quality, but it has two weaknesses for drift. The judge is itself a model that may change between versions, and it reads the same content that may have misled the agent in the first place.

Kyde Score keeps language models out of the verdict. The same log always produces the same score, and every score names the baseline and method version it was computed against. In practice, that means an alert can be checked by someone who does not trust Kyde: replay the log and the number comes out the same.

Limits

[what it is not]

What drift detection does not do

Kyde Score does not read prompts, so it cannot tell you that a page contained an injection. It tells you that the agent's behavior changed afterwards, which is often the more useful signal because it holds regardless of how the injection was phrased. The drift score also needs a baseline: during its first 30 runs and at least 3 days, an agent is learning and has no score. Loop detection is the exception. Three identical calls in a row raise an alarm from the first run, provided the agent sends a parameter digest with each tool call.

The hosted service alerts on every plan. From the Pro plan on, the Kyde hook in Claude Code holds a call outside the agent's confirmed task contract until a human decides. If you need an agent stopped or quarantined inside your own network, that is Kyde Control, described in the documentation.

FAQ

[5 questions]

What is AI agent drift?

AI agent drift is a change in how an agent behaves compared with how it behaved before, without anyone deciding that it should change. It can come from a model update, a changed prompt or tool, content the agent read during a task, or a loop. In production it shows up in the agent's actions: which tools it calls, in what order, how often and against which targets.

How do you detect drift in an AI agent?

Record the agent's actions, learn what normal looks like for that specific agent, and compare every new run against it. Kyde Score learns from the first 30 runs over at least 3 days, then runs three deterministic detectors on every later run: one for abrupt shifts, one for gradual drift, and a task contract that flags tools the agent has never used for that task.

Can drift detection catch prompt injection?

Not the injected text itself. Kyde Score never sees prompts or page content. What it can catch is the change in behavior that a successful injection usually causes, such as an agent calling a tool it has never used for that task or suddenly making far more calls than usual.

Do I need an SDK?

No. Kyde Score reads OpenTelemetry. Harnesses with built-in telemetry, such as Claude Code, Codex and OpenCode, only need a few environment variables. For your own agent, any OpenTelemetry exporter works.

Why not use an LLM to judge whether an agent drifted?

An LLM judge is itself a model that can change between versions, and it reads the same content that may have misled the agent. We think the verdict should come from statistics that give the same result for the same log every time, so that anyone can recompute it.

Try it on one agent

Kyde Score is free to start. Create a key, set three environment variables and let the agent run. After 30 runs over at least 3 days it has a baseline, and from then on you get a score per run and a webhook when it changes band.

Start free at score.kyde.com →