Thinking in Modality: How Archon Reframes Chain-of-Thought for Digital Humans

The phrase that stuck with me after reading Archon: A Unified Multimodal Model for Holistic Digital Human Generation was not "unified" or "holistic," both of which appear in nearly every multimodal paper these days. It was "Thinking in Modality." That single design choice, more than the tokenizer engineering or the diffusion decoder, points to a generalizable principle about how generative models should handle underdetermined transformations. This post unpacks what Archon actually does and why its central idea matters beyond avatar synthesis.

The problem Archon is built to solve

Digital humans require coordination across an unusually wide spread of signal types: text descriptions, spoken transcripts, audio waveforms, facial and body animation parameters, semantic video, and final pixel video. Historically, each of these has been the province of a specialized expert model. Speech-driven video generation is one research community; image-conditioned text-to-speech is another. Stitching these experts together produces a brittle pipeline because each was trained on a different distribution, with no shared representation and no guarantee that the output of one stage is a valid input to the next.

Archon's authors argue for a single autoregressive model that learns the joint distribution over all these modalities. Concretely, the system:

• Unifies seven modalities through modality-specific tokenizers that map heterogeneous signals into discrete integer tokens merged into a shared vocabulary.
• Pretrains a native autoregressive model on synchronized modality pairs across 72 diverse tasks, so the model learns cross-modal correspondences rather than memorizing single task mappings.
• Supports conditional generation over arbitrary subsets of modalities, which is what enables "any-to-any" generation and editing of any single modality while holding others fixed.

This is the familiar unified-model ambition, but applied to a domain that prior unified models largely sidestepped. Most multimodal language models output only text. Most unified generative models handle text, image, and video but ignore speech audio, or generate only non-speech audio like music. Parsing human speech, animation through a 3D Morphable Model, and preserving identity over time are genuinely underexplored in this setting.

Solving the token explosion with semantic video

Before getting to the reasoning idea, it is worth noting a piece of engineering that makes the whole thing feasible. Talking-head video is dominated by high-frequency dynamics: lip motion, micro-expressions, head pose. Capturing these faithfully demands high frame rates, around 30 fps. The authors point out that a 5-second clip at 30 fps and 256 by 256 resolution amounts to roughly 9K tokens, which already exceeds the typical 8K context window of an MLLM. Worse, forcing a discrete autoregressive model to reason over raw RGB tokens disrupts the continuous structure of video and degrades quality.

Archon's answer is a memory-efficient semantic video reparameterization. Instead of representing the video as RGB tokens, the model represents it as semantic videos: discrete semantic labels that preserve essential structure and dynamics while discarding redundant appearance detail. This yields a 4x reduction in token count. The final pixels are then produced by a semantic-driven video diffusion decoder conditioned on the semantic video, reference images, and text descriptions.

This split is conceptually clean. The autoregressive backbone reasons about what is happening (the semantic and temporal structure), while the diffusion decoder handles how it looks (high-frequency appearance). It is a sensible division of labor: discrete autoregressive models are good at structured sequential reasoning, diffusion models are good at high-fidelity continuous synthesis. Archon does not try to make one architecture do both.

Thinking in Modality: chain-of-thought across representation spaces

Now the central idea. Some cross-modal tasks have large domain gaps and severe ambiguity. Generating video directly from speech, for example, requires extracting explicit attributes (gender, prosody, timing) and inventing missing ones (appearance, expression, background). A single learned mapping from speech to pixels is badly underdetermined, and the usual scaling-to-quality intuition of expert models breaks down because there is no single correct answer to scale toward.

Thinking in Modality decomposes such a task into a stepwise chain that passes through intermediate modalities. Rather than jumping from text to video, the model proceeds text to script to animation to semantic video to video. Each intermediate representation constrains the next step, progressively reducing ambiguity and improving controllability and fidelity.

As I noted in my earlier summary, this is structurally identical to chain-of-thought reasoning, with one substitution: the latent reasoning steps are entire modalities rather than tokens of natural language. In standard chain-of-thought, a language model writes intermediate text that conditions its final answer. In Archon, the model emits an intermediate animation or semantic video that conditions the final pixel output. The reasoning trace happens to be a motion sequence rather than a sentence.

The general heuristic worth extracting is this: when a transformation is underdetermined, insert representations that progressively constrain it. Each inserted modality narrows the space of valid completions. Script fixes content and timing. Animation fixes pose and expression. Semantic video fixes spatial structure and dynamics. By the time the diffusion decoder runs, the only remaining degrees of freedom are appearance, which is exactly what diffusion handles well.

My take: reasoning is not language-specific

The deeper claim implicit in Archon is that reasoning is not a property of language. We have come to associate intermediate reasoning with text because language models are the systems where chain-of-thought first paid off dramatically. But the mechanism is more general. What chain-of-thought really provides is a sequence of partial commitments that decompose a hard joint inference problem into easier conditional steps. There is no reason those partial commitments must be words.

Archon is, in effect, evidence that you can do chain-of-thought in the space of human motion and semantic structure. This has a few consequences I find worth dwelling on:

• Intermediate modalities are also interfaces. Because each step produces a real, inspectable representation (a script, an animation, a semantic video), the chain is editable. This is why Archon can support any-modality editing: you can intervene at the animation stage and propagate downstream. A purely end-to-end text-to-video model gives you no such handle.
• The choice of intermediate modality is a design variable, not a given. Archon picks a chain that maps cleanly onto the digital-human production pipeline. For other domains, the right intermediate representations might be depth maps, scene graphs, or physical state. The principle generalizes; the specific chain does not.
• Decomposition trades compute for controllability. Each modality hop costs an additional generation pass. The payoff is reduced ambiguity and the ability to inject supervision at intermediate stages. This is the same tradeoff that test-time reasoning makes in language models.

Limitations and open questions

A few caveats deserve honesty. First, the 72-task pretraining curriculum and the synchronized multimodal dataset are doing enormous work here. The "any-to-any" flexibility is only as good as the coverage of those synchronized pairs, and the paper's claims of superior or comparable performance need scrutiny against the strongest specialized experts on each individual task, where a unified model often pays a tax.

Second, error propagation in the modality chain is a real risk. If the animation step produces a flawed motion, the semantic video and final pixels inherit that flaw. Chain-of-thought in text suffers the same fragility, and it is unclear whether intermediate-modality reasoning has any self-correction mechanism analogous to revisiting earlier tokens.

Third, the semantic video reparameterization, while elegant, deliberately discards appearance information that the diffusion decoder must then hallucinate. For identity-critical applications, the boundary between what semantic tokens preserve and what the decoder invents matters a great deal, and the paper's 4x compression is a lever that trades fidelity for tractability in ways that need careful auditing.

The forward-looking question I keep returning to is whether the modality chain should be fixed or learned. Archon hand-designs the text to script to animation to semantic video to video path. A more ambitious system would let the model choose which intermediate modalities to route through based on the difficulty of the specific instance, much as adaptive reasoning lets a language model decide how much to think. If that becomes tractable, "Thinking in Modality" stops being a pipeline and becomes a genuinely learned reasoning policy over representation spaces. That, to me, is the more interesting destination this work points toward.