(Kyde Research)

Measuring how AI agents behave.

Empirical research on agent behavior. We study how agents change, using execution records from public agent benchmarks and real Claude Code sessions.

Model changes · Run length · Reward hacking · Behavioral stability

Studies
4
Runs analysed
12 764
Public corpora
4
Last measured
2026-10-27

Studies

Draft. Numbers from the open detector repo, not yet confirmed for publication.

T1Model changes and behavioral drift

Does changing the underlying model change how an agent behaves?

FindingA new model can change how the same agent works. Or barely at all.

Tool-use distribution of the same agent framework on SWE-bench, before and after the model behind it changed. 0 means identical, 1 means nothing in common.

FrameworkBeforeAfterRuns afterTool-use shift (JSD)
SWE-agent Claude 3 Opus, 2024-04 GPT-4, 2024-04 496 0.085
SWE-agent GPT-4, 2024-04 Claude 3.5 Sonnet, 2024-06 500 0.116
SWE-agent Claude 3.5 Sonnet, 2024-06 GPT-4o, 2024-07 465 0.091
SWE-agent GPT-4o, 2024-07 SWE-agent-LM-32B, 2025-05 500 0.813
OpenHands Claude 4 Sonnet, 2025-05 Kimi K2, 2025-07 500 0.002
OpenHands Kimi K2, 2025-07 GPT-5, 2025-08 500 0.105
Emergent 2024-10 2024-12, same setup 500 0.000

LimitationsSeparate SWE-bench submissions, not one deployment over time. The last row is the control: no change, no shift.

Corpus: SWE-bench submission logs · 3 461 runs · analysed 2026-10-27 · results and script →

T2Errors over the course of a run

Does the probability of failure increase with run length?

FindingLong runs fail more: errors rise twelvefold over the first 40 steps.

Share of calls with an error status, by how far into the run they happen. One GAIA agent, 117 runs.

Steps into the runCallsError status
0 to 9 1 170 2.1 %
10 to 19 663 7.1 %
20 to 29 409 16.4 %
30 to 39 321 25.5 %
40 to 49 237 21.9 %
50 to 59 184 15.2 %
60 to 69 143 11.2 %

LimitationsLater steps are left out: below 140 calls per bin the rate is noise.

Corpus: TRAIL, GAIA, o3-mini · 117 runs · analysed 2026-10-27 · results and script →

T3Reward hacking

When an agent games its grader, what gives it away?

FindingAgents that game the test say so in their reasoning.

Terminal Wrench runs where the agent exploited the grader instead of solving the task. Detection from the agent's own reasoning text.

Agent modelHacked runsCaught from reasoningFalse alarms
Claude Opus 4.6 590 98.6 % 0.2 %
Gemini 3.1 Pro 1 181 97.5 % 1.2 %
GPT-5.4 783 99.7 % 0.5 %

LimitationsThese agents narrate their exploits openly. A ceiling, not a guarantee.

Corpus: Terminal Wrench · 4 415 runs · analysed 2026-10-27 · results and script →

T4Behavioral stability across tool updates

Do tool updates measurably change agent behavior?

FindingAcross Claude Code updates, tool use shifted no more than where nothing was updated.

Real Claude Code sessions of 171 users. Tool-use shift across 19 version updates, against the same measurement at points with no update.

ComparisonCasesMedian shift (JSD)
Across a version update 19 0.132
Control, no update 65 0.127
Same user, adjacent periods 58 0.149
Same user, first vs. last period 34 0.211
Different users 1 653 0.358

LimitationsA result we did not expect. Behavior within a user varies as much as across an update, while each user's agent keeps its own fingerprint: different users are more than twice as far apart. Baselines therefore have to be per agent, and they age within weeks.

Corpus: SWE-chat, Claude Code sessions of 171 users · 4 771 runs · analysed 2026-10-27 · results and script →

PPapers

The studies rest on two papers.

Both by Jürgen Eckel and Joerg Radehaus, both open under CC BY 4.0, with the code that produced every number.

Survey · v0.3

Behavioral Drift in Autonomous LLM-driven Systems: A Survey of Detection Approaches and the Case for Deterministic Detection

  • Separates seven drift phenomena and maps them onto the EU AI Act (Art. 3(23), Art. 12, Art. 72). Prompt injection is placed as a cause, not a phenomenon.
  • Surveys eight recent detection approaches and shows their reported numbers are not comparable.
  • A detectability matrix: which phenomenon call records alone can establish.
  • The case for determinism: an LLM judge is itself a drifting system, and a disputed verdict must be recomputable from the record.
  • Goal drift
  • Context decay
  • Reward hacking
  • Deception
  • Multi-agent drift
  • Persistent drift
  • Version drift

Measured follow-up · working draft

What Does Content Access Buy? Pricing the Observability Ladder for Behavioral Drift Detection

  • Measures what each step up in observation buys: call records, call parameters, tool outputs, reasoning text. Same corpora, same splits, deterministic detectors throughout.
  • On real Claude Code sessions: every user's agent has its own fingerprint, and a frozen baseline ages within weeks.
MMethods and data

Recorded runs, open detectors, stated limits.

All results come from recorded agent executions. Every study names its corpus, its sample size, its method and its limitations. The detector implementations and analysis scripts are open source.

(Detectors)

Frozen baseline
Learned from an agent's first runs, then fingerprinted. A drifting agent never becomes its own reference.
Gradual drift
Jensen-Shannon divergence over a window, smoothed with an exponential moving average.
Abrupt shifts
CUSUM on the surprisal of each event against the baseline.
Deterministic
Plain statistics, Python standard library only. No language model in the verdict: same record, same result.

(What a record contains)

L0 Call records
Tool name, hashed parameters, status, timing. Never content. What Kyde Score stores.
L1 Parameters
The plain parameters of each call, for example command lines.
L2 Tool outputs
What the tools returned to the agent. Where injected instructions arrive.
L3 Reasoning
The agent's own reasoning text.

(Public corpora)

  • SWE-bench submission logs of agent frameworks on different models (T1)
  • TRAIL GAIA agent runs with error status per step (T2)
  • Terminal Wrench runs where agents exploited the grader (T3)
  • SWE-chat real Claude Code working sessions with real timelines (T4)
  • Real records from public corpora. No sample numbers.
  • Every study names its runs, its corpus and the file it comes from.
  • Method version: commit 990bfef of the open detector repo.
  • No one can pay for a row.

(Measure your agent)

Public benchmarks describe agents in general. Not yours.

They cannot tell you how your own agent behaves in production. Kyde Score learns your agent's normal behavior from its own runs and scores every run against it, with the same detectors.