People run their whole day through one long AI session: dozens of unrelated topics in a single thread. We tested whether that wandering context makes a model easier to jailbreak. It does not. What does is how much of the session stays close to the harm.
TL;DR
Does topic switching erode refusals, at fixed length and content? No. Across six models adding intent drift caused no change in attack success rates.
The hypothesis was credible, not idle. Three documented, entirely benign mechanisms could each erode a guardrail over a long session: attention diluting across long contexts, a cooperative history building pressure to keep saying yes, and on-topic content reading as permission. That is why drift was worth measuring rather than assuming it safe or dangerous.
The risk is dwell, not motion: lingering too long near harmful topics. Topically relevant benign content is still a working attack vector in 2026. One frontier model’s guardrail failed up to 22% of the time once ~300k tokens of harm-adjacent content had accumulated, and the failure rate was still climbing.
The proven threats are steered, not benign wandering. An attacker walking a benign-looking conversation to the goal, crafted malicious context, or cross-trust contamination. An innocent wander onto a sensitive topic can look identical on the wire, so intent is not the thing to watch.
For defenders: watch sensitive-topic concentration, not drift. The signal that tracks real risk is how much a session's content accumulates on a sensitive capability, not whether it drifts, switches, or runs long. Drift alerts are noise; relevance is the axis to watch.
Manifold's Intent Drift Noise made a specific case: intent drift, used as a naive within-session detection rule, is counterproductive at scale. A conservative rule fired 0.40 times per endpoint per day, ~4,000 alerts a day across 10,000 endpoints, and not one lined up with a real security finding. Long, multi-objective sessions are simply how people work. That analysis was careful to keep the underlying concern intact: "intent drift carries real signal at its core, and the threat model behind it is worth taking seriously." Much of it is better framed as a governance question than a detection one.
That leaves a question a detection analysis cannot answer. Set the alarm aside: is the threat model true? Could a long, wandering conversation actually weaken a model's safety training, so a harmful request late in a rambling session succeeds where the same request would be refused on its own? If so, the exposure is wide, because it sits inside how millions of people already use these tools. This study tests that causal claim directly.
“Wandering” bundles three things a session can do, and we unbundle them. Does switching topics erode refusals, with length and content held fixed? No, confirmed equivalent to zero where we could validate the instrument. Does increased benign content on its own erode refusals? No effect on three models out to 200k tokens. Does benign relevance, content that stays near the sensitive topic? Yes, on susceptible models, and unlike everything else we varied it kept growing with scale. Drift alerts are noise; relevance is the signal. The rest of this post is the evidence for those three answers.
What is this "intent drift" anyway?
The term has a longer history than the AI-agent debate: "intent drift" has been formally defined in intent-based networking since 2022 (RFC 9315), where it means a system gradually ceasing to meet a declared intent. That field also reached a similar conclusion to ours (Haikal et al.). Detecting drift is not the same as measuring risk, and RFC 9315 requires grading the severity of drift, not just noting its presence. In the AI-agent conversation, two senses matter, and they carry different risks. We studied the first one.
User intent drift (what we studied): The user's conversation wandering across objectives within one session: a database schema, then a backend refactor, then a Kubernetes config, then pulling tickets. Our telemetry shows this is how people actually work; they keep sessions alive because accumulated context is the leverage. We measure its size as switching: how often the subject changes turn to turn.
Agent goal drift (a separate problem): The agent's own objective sliding away from what it was told to do, with or without an adversary. A real research area, measured at inference time by long-context goal-adherence evals (Arike et al.) and constraint-violation studies (asymmetric goal drift), and distinct again from training-time goal misgeneralization. It is not what this study manipulates.
Our lever is the user side. The outcome we watch is whether the agent's safety guardrail, its willingness to refuse a harmful request, weakens as that user-side context piles up and wanders. Those are two different properties of the context, so we vary them separately.
The Theory of Harm
Three documented effects give reason to expect long, drifting context to soften refusals. None of them require anything malicious in the conversation, which is what makes the concern credible.
Attention dilutes. Safety training bites hardest on short, direct requests. Stretch the context to tens of thousands of tokens and the model's attention spreads thin, the "lost in the middle" degradation documented across long-context models. A refusal reflex tuned on short prompts may not fire as reliably deep in a long one.
Consistency pressure builds. Models are trained to be agreeable and to stay consistent with what they have already said (sycophancy, FlipFlop). A long, cooperative history sets a groove of "yes, here's help." That is the dynamic multi-turn jailbreaks like Crescendo use to walk a model past its own line one step at a time.
Long benign context is read as permission. A wall of innocuous, on-topic text before a request can make that request read as legitimate and in-context. Recent work (NINJA) shows purely benign padding can pull a model out of its refusal-trained distribution.
The first of these, attention dilution, has a name and a literature: context rot. Models measurably lose accuracy as context grows (Lost in the Middle; RULER), and the root cause is a positional attention bias, not the content's importance (Found in the Middle). If safety is just another capability, it should rot too; and if topic diversity dilutes attention at shorter context than raw length would need, intent drift is a shortcut to the same erosion. The other two mechanisms are different animals: consistency pressure is a trained agreeableness (what Crescendo rides), and the permission effect is an out-of-distribution shift, neither of them context rot. But all three predict the same direction, a long benign session softening refusals, which is what we set out to test.
The literature also contained a hint the dilution story was too simple. When Arike et al. asked why agents drift from their goals over long contexts, they ran the direct ablation: replace the accumulated history with length-matched filler, and the drift largely vanished. Sheer distance from the instructions was not the driver; pattern-matching on what the context contains was. If distance alone does not erode goal adherence, there is less reason to expect it to erode refusals. Our design lets the two stories compete: switching and length move distance-style dilution; relevance moves what the context contains.
Why it is worth measuring. The riskiest user is the naive one. A safety-conscious power user opens a fresh chat per task. Most people do not. They keep one thread open for everything, for days, and context accumulates and wanders by accident. If that ordinary habit lowers the odds a harmful request gets refused, the exposure is broad and nobody opted into it. That is exactly the kind of claim a security team should measure before assuming it is either safe or dangerous.
Want to see what your agents are actually doing at runtime, and the detection and response to secure them? That is what Manifold is built for. Talk to Manifold.
Prior evidence: Shaping a model's context does raise attack success
The concern is not idle. Several ways of shaping context reliably increase how often a jailbreak lands (Figure 1). That track record is what made benign drift worth testing as a possible fourth lever.
The four panels of Figure 1:
Benign topic-switching (this study). gemini-3.5-flash-lite · ~300k · same attack, only the wandering prefix varies. 24% → 23% (no change).
Many-shot jailbreaking (Anil et al. 2024). Claude 2.0 · harmful examples stuffed into context. Climbs with dose from a few shots to ~70%.
Crescendo, multi-turn (Russinovich et al., 2024). GPT-4, Gemini Ultra, Claude-3, Llama-3 70B · gradual multi-turn buildup. Single-turn refused → succeeds across all major models with multi-turn buildup.
Benign long context, NINJA (Shah et al., 2025). Llama-3.1-8B · on-topic padding, goal first. 23.7% → 58.8% (+35 pp). Most-susceptible model shown; this is the one lever that also moved in our replication, see below.
Hold the attack fixed, vary only the wandering
A causal answer means changing one thing. We took a published multi-turn jailbreak (Crescendo), held it identical across every condition, and varied only the benign history in front of it. The core comparison set a drifted history that roamed across four topics against a focused one that stayed on a single topic, matched on length, so any difference traces to the wandering itself. We ran the drifted side two ways, the four topics interleaved turn to turn and the same topics gathered into blocks, so we could separate switching between subjects from simply covering more of them. If wandering weakens guardrails, the drifted history should let more attacks through.
Wandering didn't weaken the guardrails, and we built the test so a flat result would mean something. We fixed +10 points in advance as the smallest increase worth taking seriously, then powered the comparison across three model families at a long working-session length (~200k tokens), enough to rule out a rise of that size rather than merely miss it. On gemini-3.5-flash-lite we pushed further, to ~300k tokens, about 200 back-to-back Q&A exchanges. The pattern held everywhere: the drifted-minus-focused gap landed on both sides of zero, never significant, with every point estimate well short of the +10-point line (Figure 2). And the flat result wasn't a dead instrument: on that same model, at that same length, our readout did light up once the benign context was topically relevant instead of off-topic (the next section). Off-topic wandering just wasn't the lever.
The lever that does move: relevance erodes guardrails, and it did not level off
The third question, benign relevance, came back positive, and it is the finding we most want defenders to take away. The NINJA framework had shown that benign but on-topic content raises attack success on small open-weight models. We reproduced it on two open-weight models of the kind NINJA studied (Qwen3-32B, Mistral-7B-v0.2): refusal eroded on both, and real harm rose on Qwen. Then we tested it on more models and longer contexts. Those same two models also fall to many-shot harmful examples, and with far fewer harmful examples than it usually takes. Run benign topic-wandering on them and nothing moves. The dividing line is not how strong the model is, it is whether the added context is harm-relevant.
At short contexts, current frontier models resisted: actual harm at or below 7%, and deepseek-v4-flash stayed flat throughout. But one borderline frontier model, gemini-3.5-flash-lite, kept climbing as on-topic context grew, reaching ~22% actionable-harm success at ~300k tokens with no sign of leveling off (Figure 3). Everything else we varied (switching, off-topic length, drifted or blocked ordering) stayed flat on the same model at the same lengths. One lever moved, and it was the one that tracks what the session concentrates on.
What this does and does not say
A null result is only as good as its stated limits. This is a causal finding about specific manipulations, not a blanket claim that long sessions are safe.
Length, on its own, is not a demonstrated lever. The classic context-rot findings (Lost in the Middle, RULER) are about capability, not safety, and the one study to sweep pure benign length for safety, When Refusals Fail, found the effect runs in opposite directions across models, up for some and down for others. Our own null is now powered across three model families at ~200k tokens and holds out to ~300k on one, which sits inside that picture.
The tightest result is one model. On gemini-3.5-flash-lite at the longest length the drift null is a statistical equivalence to zero; on the others it is the weaker "no detected effect." Frontier models beyond ~300k are unmeasured.
We measured the refusal guardrail in chat, not agentic tool use. Our part-one telemetry is coding agents on endpoints; this study holds a chat-style jailbreak fixed and varies the session in front of it. Whether the same null holds when the "session" is hundreds of tool calls and the guardrail is an action policy rather than a refusal is the obvious next surface, and we have not tested it.
What is actually established
Several threat models carry more evidence than benign drift, and none of them is a wandering conversation:
Adversarial trajectories. An attacker escalates step by step until the model complies (Crescendo). Proven, and it is the attack we hold fixed in this study.
Steered benign-looking escalation. An attacker picks each innocuous-seeming turn to walk the conversation toward the harm, using the model's own knowledge to close the gap (ActorBreaker). Any single turn looks benign; the trajectory is the attack.
Crafted malicious context. Many harmful examples packed into the prompt (many-shot jailbreaking). A proven mechanism, largely mitigated on current frontier models but still firing at low dose on weaker open-weight ones.
Cross-trust contamination. A session ingests poisoned external content and carries it into a privileged action (indirect prompt injection, memory poisoning, the lethal trifecta). A well-demonstrated attack class, and the real substance behind the governance concern.
Bottom line
Three questions, three answers, one picture. Switching topics at held relevance changed nothing: confirmed equivalent to zero where we could validate the instrument. Benign length alone changed nothing: a powered test across three model families at ~200k tokens ruled out any increase worth worrying about. Topically-relevant content is the lever that moves: it eroded weaker models, kept eroding as it grew, and tipped one borderline frontier model into real harm. Put that next to part one and the practical conclusion is the same from both directions: benign drift is a poor detection signal and not a causal guardrail-eroder on the models and lengths we measured. So the shift for defenders is simple: monitor how much a session's content concentrates on a sensitive capability, not whether it drifts, switches, or runs long. Drift alerts are noise; relevance is the signal. And the legitimate governance concerns, like commingling data across contexts in one session, are real; they belong in prevention, not drift alerts.
Want to see what your agents are actually doing at runtime, and the detection and response to secure them? That is what Manifold is built for. Talk to Manifold.
Sources
Arike et al. Evaluating Goal Drift in Language Model Agents. arXiv:2505.02709 (AIES 2025).
Clemm et al. Intent-Based Networking; Concepts and Definitions. RFC 9315, IRTF, 2022.
Liu et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172 (TACL 2024).
Ren et al. ActorBreaker (LLMs know their vulnerabilities). arXiv:2410.10700 (ACL 2025).
Russinovich et al. Crescendo: The Multi-Turn LLM Jailbreak. arXiv:2404.01833.
Shah et al. Jailbreaking in the Haystack (NINJA). arXiv:2511.04707.
Sharma et al. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548 (ICLR 2024).
Latest articles
