Why Wearable Motion Models Don't Transfer, and What Geometry Has to Do With It

Introduction

If you have ever worn a fitness tracker and noticed it count phantom steps while you were typing, or fail to register a workout because you switched the watch to your other wrist, you have encountered one of the quieter but more stubborn problems in mobile sensing. The inertial measurement unit (IMU) inside the device, a combination of accelerometer and gyroscope, does not measure activity. It measures the motion of a specific point on your body, filtered through where the device sits, how it is oriented, and what hardware produced the reading. A model trained on wrist data does not transfer to a phone in your pocket, and a model trained on one dataset frequently collapses on another.

A recent paper, AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild by Chen and colleagues at UNSW and collaborators, attacks this problem directly. Their thesis is compact and, I think, correct: setup variation is structured, not arbitrary, and that structure can be turned into an inductive bias rather than treated as noise to be averaged away. This post unpacks how they do it and where I think the approach succeeds and falls short.

The Core Problem: Setup Dependence as a Modeling Trap

The fundamental issue is that the same physical activity produces wildly different inertial signatures depending on the sensing setup. The paper enumerates the relevant axes clearly:

• Body location: a wrist-worn watch emphasizes arm dynamics, glasses capture head motion, a pocketed phone couples to torso and leg movement.
• Mounting position and orientation: small changes within a single body part alter measured acceleration and angular velocity.
• Device hardware and sampling protocol: different sensors and sampling rates introduce additional distribution shifts.

The conventional responses to this each fail along a different axis. Classifier-based human activity recognition (HAR) ties the model to a fixed sensor setup and a closed label set. Contrastive methods like IMU2CLIP align sensor data with text but do no explicit setup modeling, so transfer remains weak. Naive LLM-based approaches that feed raw numerical IMU streams into a language model suffer from token cost that scales with sensors, channels, and time, plus poor signal grounding. Synthetic generation pipelines that simulate IMU at joint centers remain tied to sparse placements and specific activities, leaving a synthetic-to-real gap.

What I find compelling here is the reframing. The authors note that an IMU is attached to a body surface, and its signal is the deterministic output of body motion interacting with surface geometry, sensor orientation, and device response. That is a physical generative process, and if you can model it, you can simulate the space of possible setups rather than hoping your training data happened to cover them.

The AnyMo Pipeline: Simulate, Encode, Tokenize, Align

AnyMo is a four-stage framework, and the design of each stage follows from the geometry-first premise.

Geometry-aware IMU simulation. Rather than generating synthetic IMU at a handful of joint centers, AnyMo runs physics-grounded simulation over dense body-surface placements. The output is a broad, physically plausible distribution of where a sensor could sit and what it would read. This is the part that addresses the realism limitation of prior synthetic pipelines. By covering the body surface densely, you generate a continuum of placement views rather than a few canonical positions.

Geometry-aware setup-agnostic pre-training. A graph encoder is pre-trained from paired synthetic placement views combined with masked partial observations. The graph structure matters: body-surface placements have a natural geometric topology, and a graph encoder can represent relationships between positions rather than treating each sensor as an isolated channel. Pairing different placement views of the same underlying motion forces the encoder to learn what is invariant (the body motion) versus what is setup-specific (the placement and orientation effects). Masking partial observations pushes the model toward robustness when only a subset of body locations is observed, which is exactly the real-world condition where you might have only a watch, or only glasses.

Full-body tokenization. Multi-position IMU is tokenized into compact full-body motion tokens. This is the design choice I want to emphasize, because the authors make an explicit architectural argument: language should connect to wearable sensing through a compact motion representation, not raw streams. Raw numerical tokens scale poorly and ground weakly; location-specific tokenizers re-introduce setup dependence. Compact full-body tokens sidestep both by providing a stable, setup-invariant interface.

Motion-language alignment. Finally, these tokens are aligned with an LLM for motion-language understanding, enabling open-vocabulary tasks rather than closed-set classification.

The Results

AnyMo is evaluated on three complementary tasks, and the numbers are substantial:

• Zero-shot activity recognition across 14 unseen downstream datasets: average Accuracy/F1/R@2 improved by 11.7% / 11.6% / 22.6%.
• Cross-modal retrieval: zero-shot IMU-to-text MRR up 15.9%, text-to-IMU MRR up 28.6%.
• Wearable IMU motion captioning: zero-shot BERT-F1 up 18.8%.

The 14-dataset zero-shot evaluation is the headline for me. Most HAR work reports within-dataset or carefully held-out splits, where the setup distribution at test time resembles training. Reporting gains across 14 datasets the model never saw is a much harder bar, and it is the right one if the claim is genuine generalization rather than memorization of a fixed setup.

My Take

The intellectual move that makes AnyMo interesting is recognizing that the nuisance variable (setup) has known physics. In most of machine learning, when a variable hurts transfer, we either augment to cover it, adversarially remove it, or marginalize over it statistically. AnyMo instead generates the nuisance distribution from first principles. Body-surface simulation is essentially a forward physical model, and learning from paired views of that model is a form of supervised invariance learning where the invariances are physically guaranteed rather than empirically hoped for. This is a cleaner formulation than domain randomization, because the randomization is grounded in real geometry rather than arbitrary noise.

There is also a subtle architectural insight worth highlighting. The choice to tokenize into full-body motion tokens rather than per-sensor tokens decouples the LLM interface from the sensing hardware entirely. This is analogous to how vision-language models settled on a fixed patch-token interface regardless of camera. A stable interface is what allows the language prior to do useful work, and I suspect the retrieval gains, especially the 28.6% text-to-IMU jump, come substantially from this design rather than from the simulation alone.

That said, several limitations deserve honest mention:

• Simulation fidelity remains the bottleneck. Dense body-surface simulation is only as good as its body model and the assumed sensor response. Soft tissue artifacts, clothing-mediated coupling, and loose mounting all violate the assumption that the sensor rigidly tracks the body surface. The synthetic-to-real gap is reduced, not closed.
• Hardware diversity is the hardest axis. Body geometry can be simulated; the idiosyncratic noise, bias, and filtering of arbitrary consumer IMUs are harder to model and may be where remaining transfer failures concentrate.
• Captioning evaluation is fragile. BERT-F1 rewards semantic overlap but tolerates vague descriptions. An 18.8% gain is meaningful, but I would want to see whether the captions capture fine-grained motion distinctions or just coarse activity categories.

Conclusion

AnyMo is a careful piece of engineering built on a genuinely good insight: that wearable setup variation is structured by physics and geometry, and therefore learnable rather than merely tolerable. By simulating IMU over dense body surfaces, learning invariances from paired placement views, and committing to a compact full-body token interface for language alignment, the authors demonstrate transfer that holds across 14 unseen datasets and three task families.

The open questions I find most pressing are about the limits of the forward model. How far can simulation carry you before real hardware idiosyncrasies dominate? Could a small amount of real per-device calibration close the residual gap cheaply? And could the same geometry-aware tokenization extend to other body-attached modalities such as EMG or pressure sensing? If the answer to that last question is yes, then the more durable contribution of this work may not be the HAR numbers at all, but the demonstration that physically grounded simulation plus a stable token interface is a general recipe for setup-agnostic sensing.