Semantic Invariance in Agentic AI: When Scale Fails and Small Models Outperform

Introduction

As Large Language Models increasingly migrate from chat interfaces to autonomous agentic systems, the evaluation criteria governing their deployment must evolve beyond simple accuracy metrics. We are no longer asking whether a model can solve a physics problem correctly under ideal conditions; we are asking whether it will solve that problem consistently when the wording shifts, when context expands, or when facts appear in different orders. This property, termed semantic invariance, forms the bedrock of trustworthy agentic AI.

The paper Semantic Invariance in Agentic AI by de Zarzà et al. presents a sobering assessment of current foundation models through the lens of metamorphic testing. Their findings challenge the prevailing assumption that model scale correlates with reliability. In evaluating seven foundation models across four architectural families, the authors demonstrate that smaller models can exhibit substantially greater reasoning stability than their larger counterparts, exposing a critical gap in how we select models for high stakes deployment.

The Metamorphic Testing Methodology

Traditional benchmarks such as MMLU, GSM8K, and MATH evaluate models on fixed, canonical problem formulations. These assessments implicitly assume that performance generalizes across semantically equivalent inputs; an assumption that de Zarzà et al. systematically dismantle. To probe robustness, the authors employ metamorphic testing, a validation framework originally developed for software systems lacking definitive test oracles.

Rather than requiring ground truth labels for every input variation, metamorphic testing specifies metamorphic relations; expected relationships between outputs given known relationships between inputs. For semantic invariance testing, the relation is strict equivalence: if input A and input B convey identical semantic content through different surface forms, the model should produce logically equivalent reasoning and identical conclusions.

The authors operationalize eight distinct semantic preserving transformations across three categories:

Structural transformations include identity (verbatim repetition), paraphrase (lexical and syntactic variation), and fact reordering (permuting premise order without logical consequence).

Verbosity transformations comprise expansion (adding clarifying context) and contraction (reducing to essential elements).

Contextual transformations shift framing between academic and business registers, plus a particularly challenging contrastive formulation that presents the target problem alongside explicit distractors or alternative scenarios.

Applied to 19 multi step reasoning problems spanning eight scientific domains (Physics, Mathematics, Chemistry, Economics, Statistics, Biology, Calculus, and Optimization) at three difficulty levels, this framework generates a robustness profile invisible to standard accuracy metrics. The evaluation encompasses seven models: Hermes 70B and 405B, Qwen3 30B A3B and 235B A22B, DeepSeek R1 0528, and gpt oss 20B and 120B.

The Scale Robustness Inversion and Model Signatures

The central finding of the study defies conventional wisdom about model selection. Model scale does not predict semantic robustness. The smaller Qwen3 30B A3B achieves the highest stability with 79.6% invariant responses and mean semantic similarity of 0.91, outperforming models with four times its parameter count. This scale robustness inversion suggests that raw capacity and reliability are orthogonal properties, at least within current training paradigms.

The analysis reveals distinct model family signatures that hint at architectural predispositions:

Hermes models exhibit consistent fragility under contrastive transformations, suggesting attention mechanisms that cannot effectively filter explicit distractors when alternative scenarios are presented simultaneously.

Qwen3 models demonstrate exceptional robustness across all transformation types, with the smaller variant actually outperforming its larger sibling (235B A22B), indicating potential overparameterization effects or quantization artifacts that degrade stability in the larger configuration.

gpt oss models (both 20B and 120B) display reasoning coherence breakdown under multiple transformations, particularly verbosity shifts, suggesting that OpenAI's architectural choices prioritize fluency over logical consistency across input variations.

DeepSeek R1 shows pronounced structural sensitivity to fact reordering and contraction, implying that its reasoning chains may depend heavily on surface level textual cues rather than abstract semantic representations.

Most striking is the universal contrastive fragility observed across all model families. Contrastive transformations, which present problems alongside explicit contrasts with alternative scenarios, universally destabilize reasoning, with score degradations reaching -0.45. This suggests a fundamental limitation in current attention based architectures when handling negative space reasoning or distractor suppression.

Architectural Black Boxes and the Deployment Dilemma

These findings carry profound implications for the field. We currently lack theoretical frameworks to predict which architectural configurations will yield semantic invariance. The Qwen3 family's robustness appears almost accidental from the perspective of current interpretability tools; we cannot point to specific attention patterns, mixture of experts configurations, or training data compositions that explain its stability advantage.

This creates a deployment paradox. Organizations seeking to build agentic systems for medical diagnosis, financial analysis, or safety critical operations currently face an empirical lottery. Without metamorphic testing, they might select a 405B parameter Hermes model based on benchmark leaderboards, inadvertently deploying a system that fails when clinicians rephrase symptoms or when market reports shift between formal and casual registers. The smaller, more robust Qwen3 30B would be dismissed as "insufficient" based on scale heuristics, despite superior real world reliability.

The contrastive fragility phenomenon deserves particular attention. The consistent degradation when models encounter alternative scenarios suggests that current training regimes, which optimize for next token prediction on corpora where conflicting information appears sequentially rather than simultaneously, fail to build robust discriminative capabilities. Agents deployed in multi agent coordination frameworks or decision support systems will inevitably encounter contrastive formulations; competing hypotheses presented side by side. That all current architectures struggle with this format indicates a training objective mismatch rather than a solvable engineering problem.

Limitations and Open Questions

The study, while comprehensive, leaves several questions unanswered. The evaluation covers only scientific reasoning domains; whether semantic invariance profiles hold for legal reasoning, creative writing, or social inference remains unexamined. Additionally, the semantic similarity metric (likely embedding based cosine similarity or equivalent) may mask subtle logical divergences that human reviewers would flag as meaningful errors.

The temporal validity of these results also warrants caution. Foundation models update frequently, and the specific checkpoint vulnerabilities documented here may shift with subsequent fine tuning. The architectural families sampled, while diverse, do not encompass the full ecosystem of available models, particularly missing recent efficient architectures such as Mamba based state space models or novel recurrent approaches.

Conclusion: Toward Invariance Aware Development

The work of de Zarzà et al. establishes semantic invariance as a necessary evaluation dimension for agentic AI, not merely a desirable property. Their demonstration that smaller models can outperform larger ones on stability metrics suggests we must decouple the pursuit of capability from the pursuit of reliability. The field needs invariance aware training objectives that explicitly penalize sensitivity to semantic preserving transformations, particularly contrastive formulations.

Moving forward, we require mechanistic interpretability research that identifies the architectural substrates of robustness. Why does Qwen3's mixture of experts configuration confer stability advantages? What attention modification would eliminate DeepSeek's sensitivity to fact reordering? Until we can answer these questions, every deployment of autonomous agents in consequential settings remains, as the evidence suggests, a leap of faith. The metrics we have been using to judge progress were measuring the wrong thing. It is time to start measuring what actually matters for systems we intend to trust.