Why Decision Transformers Cannot Stitch, and How Hierarchy Fixes It

Decision Transformer (DT) reframed offline reinforcement learning as a sequence modeling problem, and in doing so it imported a great deal of machinery from the language modeling world: causal transformers, autoregressive prediction, and the seductive promise of in-context learning. The pitch is elegant. Train a transformer on trajectories annotated with return-to-go, then at test time prompt the model with a high desired return and let it produce the actions that achieve it. No value functions, no bootstrapping, no temporal difference instability. Just supervised learning on sequences.

The trouble is that this elegance hides a structural weakness. The paper Rethinking Decision Transformer via Hierarchical Reinforcement Learning by Ma, Xiao, Liang, and Hao (2023) makes that weakness precise and, more importantly, offers a principled remedy. The core argument is that DT is a special, frozen case of a more general hierarchical framework, and that unfreezing the right component restores the capability DT lacks: stitching.

The Stitching Problem

Stitching is the ability to recombine pieces of suboptimal trajectories into a better one. Suppose your dataset contains a trajectory that drives from A to B efficiently and another that drives from B to C efficiently, but no single trajectory that goes A to B to C. A competent offline RL algorithm should be able to learn the optimal A to C route by composing the two segments. Classical value-based methods do this naturally, because the Bellman backup propagates value information across state transitions regardless of which trajectory a transition came from.

DT does not have this property in any reliable way. Because it is trained to predict the action that was actually taken given a state and a return-to-go, it is essentially learning a return-conditioned behavior cloning model. When you condition on a high return at test time, you are asking the model to recall what actions tended to precede high returns in the data. If the data never contains a high-return trajectory through a given state, conditioning on a high return there is an out-of-distribution query, and the model has no mechanism to synthesize the missing behavior. The paper makes the standard observation, supported by a body of prior work (Emmons et al., Kostrikov et al., Yamagata et al., Badrinath et al.), that on benchmarks requiring stitching DT often fails to beat plain behavior cloning.

This is the point I emphasized in the short version of this analysis: in-context conditioning is not value learning. The return-to-go prompt looks like it is doing something value-like, but it is just an index into the data distribution. Push it outside that distribution and the prompt magic evaporates.

DT as Frozen Hierarchical RL

The conceptual contribution of the paper is to reinterpret return-conditioned sequence modeling through the lens of hierarchical reinforcement learning. The framework splits decision making into two policies:

• A high-level policy that, given the current state, proposes a prompt. In DT this prompt is the return-to-go.
• A low-level policy that, given the state and the prompt, produces an action. In DT this is the causal transformer.

Under this decomposition, vanilla DT corresponds to a particular and rather rigid choice. The high-level policy is essentially fixed: the prompt is either supplied by the user as a target return or computed deterministically from the dataset during training. The low-level policy is trained to imitate actions conditioned on that prompt. The high-level component never learns to propose prompts that would actually lead to optimal behavior; it is frozen at whatever the data or the user dictates.

Once you see DT this way, the failure mode becomes obvious. All of the burden of finding good behavior is placed on the prompt, but nothing in the training procedure optimizes the prompt toward optimality. The model is asked to generalize from "what return did this trajectory achieve" to "what is the best achievable return from here and what actions realize it," which is precisely the value estimation problem DT claimed to avoid.

The remedy the authors propose is to jointly optimize the high-level and low-level policies. Instead of treating the prompt as a fixed conditioning signal, the high-level policy learns to generate prompts that correspond to high-value behavior, and the low-level policy learns to act on those prompts. The resulting algorithm, which they call Autotuned Decision Transformer (ADT), uses value information to guide what prompts the high-level policy should propose. In one variant the prompt is a learned value or return estimate that reflects the optimal achievable return rather than the empirical return in the trajectory; in another it is a higher-level latent direction. The key architectural detail, shown in their Figure 1, is that the prompt is concatenated with the state rather than treated as a separate token, and predictions are made autoregressively through a causal transformer.

This is where value learning quietly re-enters. To make the high-level policy propose prompts consistent with optimal behavior, you need some estimate of value, and that estimate has to come from bootstrapping or a value-based objective over the dataset. The hierarchical reframing does not eliminate the need for dynamic programming; it isolates it in the high-level policy while letting the transformer continue to do what it is good at, namely flexible conditional sequence generation.

Why This Matters and Where It Breaks

The empirical results in the paper show consistent improvements over DT on locomotion control, navigation, and robotics benchmarks from the offline RL literature, with the largest gains precisely on the navigation tasks that demand stitching. That pattern is the tell. If your improvement concentrates on stitching-heavy tasks, you have addressed the stitching problem rather than simply tuning hyperparameters.

My own read is that this paper is less about a single algorithm and more about a clarifying reframe. The hierarchical decomposition gives a vocabulary for understanding a whole family of DT variants. Return-conditioned methods, prompt-tuning methods, and value-augmented sequence models can all be placed on a spectrum according to how much intelligence they put into the high-level prompt generator versus the low-level actor. Q-learning DT (Yamagata et al.) and the constraint-based and value-guided variants can be seen as different choices of high-level policy within the same skeleton.

There are honest limitations to acknowledge. First, once you introduce value estimation into the high-level policy, you inherit the difficulties that DT was supposed to escape: distribution shift in the value targets, overestimation, and the instability of bootstrapping under function approximation. The transformer does not insulate you from these. Second, the hierarchical structure adds complexity and new hyperparameters governing the interaction between levels, and joint optimization of two coupled policies can be brittle. Third, the abstraction of the prompt as a scalar return-to-go is limiting; the framework allows richer prompts, but learning what a good prompt should be is itself an open representation learning problem.

Closing Thoughts

The deeper lesson generalizes beyond Decision Transformer. The early excitement around sequence models for control rested on the hope that scale and in-context learning could substitute for the explicit credit assignment that classical RL performs through the Bellman equation. This paper is a careful demonstration that the substitution is incomplete. You can move the value computation around, hide it inside a high-level policy, or express it as a learned prompt, but for any problem that requires recombining suboptimal experience, the credit assignment has to happen somewhere.

The open question I find most interesting is whether scale eventually changes this calculus. With enough diverse data and a large enough model, could the high-level prompt generator learn an implicit value function purely from sequence prediction, the way large language models learned arithmetic and reasoning as emergent behaviors? The honest answer today is that we do not know, and the navigation benchmarks suggest that current models are far from that regime. Until then, the most productive path is exactly what this paper advocates: keep the transformer for what it does well, and put a proper value-aware controller on top.