When the Right Answer Is There but Loses: Arbitration Failures in Audio-Language Models

Introduction

Consider a deceptively simple scenario. An audio-language model (ALM) is given a recording in which an alarm is clearly audible. Alongside that audio, the model receives a written context stating that no alarm was triggered. When asked whether the alarm went off, the model answers "no alarm." It followed the text and ignored what it heard.

The obvious diagnosis is a perception failure: the model simply did not encode the alarm. But this conclusion may be wrong, and getting it wrong matters a great deal. If the model never heard the alarm, the fix lies in better acoustic encoding. If the model heard the alarm but let the conflicting text override it, the fix lies in better conflict resolution at the decision point. These are entirely different engineering problems with entirely different solutions.

The paper "Beyond Text Following: Repairable Arbitration Reversals in Audio-Language Models" by Gao and colleagues addresses precisely this distinction. Their central finding is that in a large fraction of conflict cases, the audio-supported answer is in fact encoded and available; it simply loses during the model's internal arbitration between modalities. They call this pattern a repairable arbitration reversal, and they exploit it to build a training-free decoding correction that generalizes surprisingly well.

The Same-Audio Counterfactual

The methodological core of the paper is a clean experimental design that earlier work lacked. Benchmarks such as MCR-Bench and ALME established that ALMs frequently follow text under audio-text conflict, but they could not tell us why. Was the audio answer unavailable, or merely overridden?

Most prior decoding-time correction methods (AAD and ACD in audio, VCD, OPERA, and ICD in vision-language) construct reference distributions by weakening or removing modality evidence. That design answers a different question. It measures how much a model relies on degraded evidence, which is useful for diagnosing hallucination from a damaged input. But it cannot isolate arbitration, because it changes the audio itself.

The authors instead use what they call a same-audio counterfactual. The procedure is straightforward:

• Hold the audio fixed.
• Remove only the conflicting text.
• Measure how the model's answer preference shifts.

This isolates the contribution of the text to the arbitration outcome while keeping the perceptual input constant. If the model still prefers the audio-supported answer once the text is gone, then the audio answer was clearly represented all along.

The headline result is that across five ALMs and four conflict tasks, 64.1% of conflict samples show a sign flip. In the same-audio branch, the model prefers the audio-supported answer; in the joint audio-plus-text branch, it prefers the text-supported answer. The audio answer is available, but the addition of conflicting text reverses the decision. This is the operational meaning of "repairable": these samples form a candidate set where the correct answer can in principle be recovered without touching the encoder.

Localizing the Reversal

A sign-flip statistic on its own would be suggestive but mechanistically thin. The paper strengthens the case with activation patching, a causal interpretability technique that copies activations from one forward pass into another to test which computations are responsible for a behavioral difference.

The authors localize the reversal to the answer-position residual stream. In other words, the decisive computation that flips the model's preference happens at the position where the answer is generated, not diffusely across the sequence. This is consistent with a picture in which the audio evidence is encoded earlier and upstream, but the final read-out at the answer position is where text wins the arbitration.

The more useful finding for practical purposes is the alignment between the patch-induced direction and the output-level score difference. The patch effect tracks the difference between the same-audio score and the joint score, $s_A - s_J$, at the rank level with a Spearman correlation of 0.93. This high correlation matters because it means the repair direction is visible from the model outputs alone. You do not need to run expensive internal patching at inference time to know which way to push; the logit gap between the two branches already points in the right direction.

From Diagnosis to Decoding Rule

This is where the diagnostic becomes a method. The authors propose Gated Audio Counterfactual Logit Correction (GACL), a training-free decoding rule. The structure is intuitive given the analysis above:

• Run both branches: the joint branch (audio + text + question) and the same-audio reference branch (audio + question).
• Interpolate from the joint logits toward the reference logits.
• Gate the interpolation by two signals: branch disagreement (how much the two branches diverge) and reference reliability (how confident the audio-only branch is).

The gating is the conceptually important part. A naive interpolation would risk corrupting cases where there is no genuine conflict, hurting faithfulness on clean inputs. By conditioning the correction on disagreement and reliability, GACL applies the push only where the evidence for a repairable reversal is strong.

The evaluation is conducted under a strict 5 percentage point faithfulness-drop budget, meaning the method is not allowed to degrade ordinary faithful behavior by more than that margin. Under this constraint, GACL improves conflict audio-following by 17.8 nAUC points over the best contrastive decoding baseline. The use of a budgeted metric is a sign of methodological maturity; rescue methods that improve conflict accuracy by sacrificing clean-input behavior are not actually useful, and the budget enforces honesty here.

The most striking result is transfer. Applied without retuning to vision-text arbitration on MC2, the same rule raises adversarial accuracy by up to 40.5 percentage points. This suggests the underlying phenomenon, evidence that is encoded but loses in cross-modal arbitration, is not specific to audio.

My Take

The framing here is what I find most valuable, beyond the specific numbers. The field has a tendency to attribute multimodal failures to perception because perception is the modality we most distrust. Audio and vision encoders are imperfect, so it is natural to blame them. But this paper provides a clean instrument for separating perception from arbitration, and the result reorders our priors: in roughly two thirds of conflict cases, the encoder did its job.

The generalization to vision-text is the inference I would emphasize most. The same-audio counterfactual is really a same-modality counterfactual: hold the non-text evidence fixed, remove the conflicting text, and watch the preference move. That the resulting correction transfers to vision suggests we are looking at a shared property of how instruction-tuned multimodal models fuse evidence. My hypothesis is that instruction tuning systematically up-weights the text channel, because text carries the task instruction and the model is trained to be obedient to it. Arbitration reversals may be a side effect of sycophancy-like instruction following bleeding into evidence evaluation, a connection the authors themselves gesture toward when they cite the knowledge-conflict and sycophancy literature.

The 0.93 correlation between internal patching and output score gaps also deserves attention as a methodological pattern. It is an instance where an interpretability finding pays for itself by yielding a cheap, output-only proxy. That is the kind of interpretability that scales, because it does not require running mechanistic probes at deployment.

Limitations and Open Questions

A few caveats are worth stating plainly. The notion of "repairable" is defined operationally on samples where the same-audio branch favors the audio answer and the joint branch favors the text answer. This is a specific, identifiable subset, and the 64.1% figure should be read as a fraction of conflict samples meeting that condition, not a claim that 64% of all errors are recoverable. The remaining cases, where the audio answer genuinely is not represented, still require encoder improvements.

There is also a question of when text should win. In real agentic settings, written context is sometimes more authoritative than ambient audio. A meeting agenda might correctly state that an item was cancelled even if background noise resembles it. A pure audio-following bias is not always correct arbitration, and GACL's gating must be tuned against this. The faithfulness budget partially addresses this, but the deeper question of normatively correct arbitration remains open.

Looking forward, the most interesting direction is whether arbitration can be improved at training time rather than patched at decoding time. If the reversal localizes to answer-position computation, can we add a fusion objective that calibrates how much each modality contributes when they conflict? The diagnostic in this paper gives us the measurement tool to answer that. The next step is to use it as a training signal, not just a decoding correction.