Chain-of-Thought Predictive Control: Reconsidering Unsupervised Subskills in Contact-Rich Domains

Imitation learning for low level control remains stubbornly difficult when demonstrations are suboptimal. In contact rich manipulation, human teleoperation or kinesthetic teaching produces trajectories that are inherently noisy, non Markovian, and dynamically discontinuous. Standard behavior cloning fails here because compounding errors accumulate rapidly when the policy encounters states even slightly outside the training distribution. The recent paper Chain-of-Thought Predictive Control by Jia et al., presented at ICML 2024, proposes a hierarchical solution that discovers subskills from demonstrations without human annotation, then uses these discovered abstractions as a Chain-of-Thought to guide a Transformer based policy. While the architectural contribution is technically sophisticated, the reliance on unsupervised segmentation raises important questions about the semantic validity of the discovered waypoints and the robustness of the resulting hierarchy.

The Algorithmic Architecture of CoTPC

The core innovation of CoTPC lies in its two stage approach to hierarchical imitation learning. First, the authors address the problem of subskill discovery. Rather than requiring human experts to manually segment demonstrations into semantically meaningful primitives, they propose an observation space agnostic method that groups temporally close and functionally similar actions into contiguous segments. The boundaries between these segments constitute the Chain-of-Thought, a sequence of key observations that represent the high level plan embedded within the noisy demonstration.

This discovery process relies on the intuition that subskills should correspond to regions of the action space where the demonstrator applies similar control signals over short time horizons. By clustering actions based on functional similarity and temporal proximity, the algorithm identifies change points where the control strategy shifts significantly. These change points become the supervision targets for the high level planner.

Second, the authors introduce a Transformer based architecture that jointly predicts both the low level actions and the high level CoT subskills. The mechanism couples these predictions through learnable prompt tokens and a hybrid masking strategy. Specifically, the model attends to historical context through causal masking for action generation while allowing bidirectional attention for the CoT prediction tokens. This design enables dynamic updating of the subskill guidance at test time; as the agent executes actions, it continuously replans its high level strategy, which in turn influences the low level motor primitives.

The evaluation spans three challenging benchmarks: Moving Maze, a navigation task requiring long term planning; Franka Kitchen, a multi task manipulation domain; and ManiSkill2, which emphasizes contact rich object interactions. Across these environments, CoTPC reportedly outperforms strong baselines including Behavior Transformer, Diffusion Policy, and hierarchical behavior cloning variants, particularly when trained on suboptimal demonstrations.

The Semantics of Discovered Boundaries

Despite the empirical improvements, the unsupervised discovery mechanism warrants careful scrutiny. The fundamental assumption is that functional similarity in action space correlates with semantically meaningful task decomposition. In contact rich manipulation, however, this assumption becomes problematic.

Consider a peg insertion task where the demonstrator applies seemingly similar forces while sliding along the surface searching for the hole. The clustering algorithm might segment these exploratory motions as a single subskill, whereas a human expert would recognize this as a corrective feedback loop within the insertion primitive. Conversely, abrupt changes in action space might reflect contact discontinuities or noise in the demonstration, rather than intentional transitions between subtasks. The method risks discovering spurious boundaries that capture dynamical artifacts instead of causal task structure.

This epistemological gap matters because the CoT serves as explicit supervision for the high level policy. If the discovered waypoints are not causally meaningful, the model faces a dual risk. First, the Transformer might learn to ignore the CoT guidance, effectively collapsing back to flat behavior cloning and losing the benefits of temporal abstraction. Second, and more concerning, the model might propagate errors from the high level plan into the low level actions. If the planner predicts a subskill boundary prematurely, the low level controller might execute an inappropriate action sequence for the current physical context, exacerbating compounding errors in precisely the contact rich regimes where robustness is most needed.

The paper acknowledges that demonstrations exhibit non Markovian and discontinuous properties, yet the clustering approach treats action sequences as if they possess inherent segmentation structure that can be recovered through geometric similarity. This overlooks the possibility that suboptimal demonstrations often contain redundant or corrective motions that lack hierarchical structure entirely.

Your Take: Hierarchical Credit Assignment in the Suboptimal Regime

The limitations of CoTPC highlight a broader tension in hierarchical imitation learning between unsupervised option discovery and task relevant abstraction. Methods such as CoTPC implicitly assume that the demonstration distribution contains latent subskills waiting to be uncovered through statistical analysis. Yet in practice, suboptimal human demonstrations often represent ad hoc solutions to poorly understood dynamics, where the only consistent structure is the final goal state, not the intermediate waypoints.

From a credit assignment perspective, CoTPC attempts to solve two coupled problems simultaneously: discovering what the subskills are, and learning to predict them. This contrasts with approaches that separate representation learning from policy optimization, or those that incorporate weak supervision through human subgoal annotations. While the end to end approach is elegant, it conflates the discovery of statistical regularities with the identification of causal interventions that actually enable task completion.

My analysis suggests that the robustness of CoTPC likely depends heavily on the quality of the action clustering, a relationship the paper does not fully characterize. In environments like ManiSkill2 where contact modes change abruptly, the Euclidean distance between action vectors may not reflect the semantic similarity of manipulation primitives. A more conservative approach might involve validating discovered subskills through their causal influence on task success, for instance by measuring whether reaching a putative subgoal state actually increases the probability of eventual task completion.

Furthermore, the hybrid masking strategy, while architecturally novel, creates an information asymmetry that could amplify the impact of erroneous CoT predictions. Because the CoT tokens influence action generation through the prompt mechanism, a single mispredicted subskill boundary could corrupt the conditioning context for subsequent action predictions. The paper demonstrates aggregate performance improvements, but the per trajectory failure modes, particularly in highly contact rich episodes, merit deeper investigation.

Conclusion

Chain-of-Thought Predictive Control represents a technically sophisticated attempt to bridge hierarchical reinforcement learning and imitation learning for low level control. By discovering subskills through action clustering and coupling these with a Transformer architecture, Jia et al. provide a scalable alternative to manually annotated hierarchies. However, the reliance on unsupervised segmentation introduces a fragility that the robotics community must address. As we move toward foundational models for manipulation, we must ask whether statistically discovered abstractions provide sufficient causal grounding for robust control, or whether we require hybrid approaches that combine unsupervised discovery with physical reasoning or minimal human supervision. Future work should investigate methods that validate discovered subskills through causal intervention analysis, ensuring that the chains we teach our models to think in are chains that actually bind the physical world.