Reading the Residual Stream: How Activation Trajectories Reveal Multi-Turn Jailbreaks

The arms race between jailbreak attacks and safety filters has largely been fought at the surface level. Defenders scan for suspicious tokens, known adversarial phrases, or anomalous perplexity scores; attackers respond with Base64 encoding, translation tricks, or roleplay framing that preserves malicious intent while disguising surface features. This dynamic favors attackers, who can iterate on text faster than defenders can patch classifiers. A new approach, detailed in Latent Adversarial Detection: Adaptive Probing of LLM Activations for Multi-Turn Attack Detection by Prashant Kulkarni, proposes shifting the battlefield from text to the model’s internal geometry. By probing the residual stream across conversation turns, this method detects what the model understands rather than what the attacker wrote.

The Limits of Surface Inspection

Current text-based defenses operate on a fundamental asymmetry. Multi-turn prompt injection attacks typically follow a three-phase trajectory: trust-building, pivoting, and escalation. In isolation, each turn appears legitimate. A user might discuss gardening techniques, then ask about fertilizer chemistry, then request synthesis instructions. Text-level filters see benign tokens in each message; they miss the adversarial intent encoded in the conversational arc. Kulkarni notes that off the shelf detection tools achieve 20 to 95% detection rates but suffer 16 to 76% false positive rates, rendering them impractical for production systems where security discussions and technical queries trigger the same surface features as actual attacks.

The core insight of this work is that while surface text varies, the attack path leaves a consistent signature in the model’s internal activations. Each phase shift in the adversarial strategy produces a corresponding shift in the residual stream, creating what the author terms adversarial restlessness. This manifests as elevated cumulative path length in activation space; the model’s internal representation moves further across turns during an attack than during benign conversation.

Trajectory Features and Detection Performance

Rather than analyzing static activation patterns, Kulkarni’s approach tracks dynamics. The method extracts five scalar trajectory features from the residual stream that capture how activations evolve across conversation turns. These features quantify the geometric properties of the activation path, measuring total displacement and directional changes that characterize adversarial steering.

On synthetic held-out data, these trajectory features lift conversation-level detection accuracy from 76.2% to 93.8%. This represents a substantial improvement over both text-level baselines and simpler activation probing methods. The signal proves robust across four model families ranging from 24B to 70B parameters, though critically, the probes are model-specific and do not transfer across architectures. A probe trained on one model family cannot detect attacks in another, suggesting that the geometric signatures of adversarial restlessness are tied to each model’s particular representational structure.

The methodology employs an adaptive probe architecture designed for operational deployment. Once activations are cached, incorporating new labeled data and retraining requires no GPU resources. This enables a practical deployment model where the system improves as production data accumulates. The paper demonstrates that incorporating real-world data with class rebalancing reduces false positives from 97 to 99% down to 2 to 4%, addressing a critical barrier to adoption in high-stakes environments.

The Critical Role of Phase Labels

One of the most technically significant contributions concerns labeling granularity. Existing benchmarks typically provide binary conversation-level labels, marking an entire dialogue as either benign or adversarial. Kulkarni introduces a synthetic dataset with three-phase turn-level labels: benign, pivoting, and adversarial. This granularity enables detection during the steering phase, before overt harmful requests appear.

The experimental results here are stark. Binary conversation-level labels produce 50 to 59% false positives, rendering them unusable for practical deployment. In contrast, the three-phase labeling scheme enables the detector to identify when the conversation has shifted from legitimate interaction to adversarial pivoting. This early detection capability is essential for autonomous systems where intervention must occur before harmful tool use or information disclosure.

The paper further establishes that trajectory dynamics are orthogonal to Sparse Autoencoder (SAE) content features. While SAEs identify specific interpretable features within activations, the trajectory probes measure how the model’s internal state evolves. This suggests a complementary detection paradigm; SAEs might identify what the model is representing at a given moment, while trajectory probes detect the suspicious dynamics of how the model’s understanding is being manipulated across turns.

Generalization Challenges and Deployment Realities

Despite strong performance on synthetic data, the work confronts hard limits regarding generalization. Detection performance is source-dependent; the distribution of attacks in the training data determines effectiveness on real-world inputs. Leave-one-source-out evaluation reveals that synthetic data, LMSYS-Chat-1M, and SafeDialBench each capture distinct attack distributions. When the LMSYS distribution is represented in training, detection on real-world LMSYS data reaches 47 to 71%. Combined three-source training achieves 89.4% detection at 2.4% false positive rate on a held-out mixed set.

This source dependency has important implications for deployment. Organizations cannot simply train on synthetic attacks and expect robust protection against novel real-world jailbreaks. The adaptive probe architecture becomes essential here; continuous retraining on production traffic is not merely an optimization but a requirement for maintaining detection efficacy as attack distributions evolve.

The cross-model validation also reveals practical constraints. Since probes do not transfer across architectures, organizations must maintain separate detection systems for each model family in their fleet. This multiplies operational overhead compared to universal text-based filters, though the improved accuracy may justify the cost for high-security applications.

Implications for Autonomous Systems

This research arrives at a critical moment in AI deployment. Autonomous cyber agents capable of multi-stage campaigns are emerging, and frontier models demonstrate increasing capability in completing complex attack chains. The UK AISI benchmarks show models completing 22 of 32 steps in corporate network attacks, with performance dropping precisely at the reconnaissance-to-exploitation transition. Kulkarni’s trajectory probes detect exactly these phase transitions in the residual stream.

For defenders, this suggests a shift from perimeter-based security to behavioral monitoring of internal states. Just as host-based intrusion detection systems monitor process behavior rather than network packet contents, activation probes monitor model cognition rather than input text. This interior perspective may prove essential as attackers increasingly leverage the model’s own conversational capabilities against it, using multi-turn jailbreaks that exploit the model’s reasoning and tool-use capacities.

Open Questions and Future Work

Several questions remain unanswered. The paper demonstrates robustness to SAE ablation, but the mechanistic interpretation of the five trajectory features remains incomplete. Understanding precisely which computational pathways produce adversarial restlessness could enable more targeted interventions, perhaps allowing defenders to steer models away from harmful compliance during the pivoting phase rather than merely detecting the attack.

The source-dependent generalization also raises fundamental questions about the nature of adversarial distributions. Are the differences between synthetic and real-world attacks merely stylistic variations, or do they represent genuinely distinct steering strategies in activation space? If the latter, then detection may always require exposure to the specific attack distribution, creating a lag time between novel attack emergence and detection capability.

Finally, the adaptive probe architecture suggests a continuous learning scenario that itself may be vulnerable to adversarial manipulation. If attackers know the system retrains on production data, they might attempt data poisoning attacks designed to gradually shift the probe’s decision boundary. The security properties of this adaptive system against sophisticated adversaries who understand the detection mechanism deserve further analysis.

Kulkarni’s work establishes that the residual stream contains reliable signatures of adversarial intent invisible at the text level. As multi-turn attacks become the default mode of sophisticated adversaries, monitoring the geometry of model cognition may become as standard as monitoring network traffic is today. The transition from surface inspection to internal state analysis represents a necessary evolution in AI safety engineering, one that aligns defensive capabilities with the representational richness of modern language models.