Does changing the underlying model change how an agent behaves?
FindingA new model can change how the same agent works. Or barely at all.
Tool-use distribution of the same agent framework on SWE-bench, before and after the model behind it changed. 0 means identical, 1 means nothing in common.
| Framework | Before | After | Runs after | Tool-use shift (JSD) |
|---|---|---|---|---|
| SWE-agent | Claude 3 Opus, 2024-04 | GPT-4, 2024-04 | 496 | 0.085 |
| SWE-agent | GPT-4, 2024-04 | Claude 3.5 Sonnet, 2024-06 | 500 | 0.116 |
| SWE-agent | Claude 3.5 Sonnet, 2024-06 | GPT-4o, 2024-07 | 465 | 0.091 |
| SWE-agent | GPT-4o, 2024-07 | SWE-agent-LM-32B, 2025-05 | 500 | 0.813 |
| OpenHands | Claude 4 Sonnet, 2025-05 | Kimi K2, 2025-07 | 500 | 0.002 |
| OpenHands | Kimi K2, 2025-07 | GPT-5, 2025-08 | 500 | 0.105 |
| Emergent | 2024-10 | 2024-12, same setup | 500 | 0.000 |
LimitationsSeparate SWE-bench submissions, not one deployment over time. The last row is the control: no change, no shift.