Why Medical Exam Scores Hide the Hardest Part of Clinical Reasoning

Large language models have spent the last two years racking up headlines for passing medical licensing examinations. The standard narrative treats these scores as a proxy for clinical readiness: if a model can answer USMLE questions at expert level, surely it understands medicine. The paper ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents by Lu and colleagues argues, convincingly, that this framing confuses a narrow skill with a broad competence. Answering a well-posed multiple-choice question and managing a hospitalization are different cognitive tasks, and the latter is where current systems quietly fail.

This article unpacks what ClinEnv actually measures, why its results matter, and what they suggest about the trajectory of medical AI evaluation.

The Structural Gap in Existing Benchmarks

The authors organize prior work along five design dimensions: whether a benchmark uses real EHR data, whether it is multi-stage, whether it requires active information acquisition, whether it scores deterministically, and whether it tracks process metrics. Viewed through this lens, every existing benchmark family makes a structural compromise.

• Knowledge benchmarks (MedQA, MedMCQA, PubMedQA, MultiMedQA) equate competence with factual recall. There is no information to gather and no irreversible commitment under uncertainty. The answer is one of the enumerated options.
• Data-interaction benchmarks (EHRSQL, MIMIC-SQL, FHIR-AgentBench, MedAgentBench) treat the EHR as a queryable database. They measure whether a model can execute well-specified operations, not whether it knows which data matters or what action the data warrants.
• Diagnostic-conversation benchmarks (AgentClinic, MedDialBench, MAQuE, AMIE) introduce interactivity, but their ground truth is bounded by what an LLM-simulated patient was told, typically a short exam vignette, and intermediate behavior is scored by an LLM-as-judge reading free-text dialogue.
• Coding-style executable benchmarks (MedAgentGym, MedCalc-Env) reduce clinical reasoning to code or formula generation.

The unfilled cell in this table is the realistic one: an agent acting as an attending physician over a real, multi-stage admission, forced to gather information actively, with each decision verified deterministically against the recorded clinical course. ClinEnv occupies exactly that cell, and it is the first benchmark to satisfy all five dimensions simultaneously.

How ClinEnv Works: Longitudinal Inpatient Simulation

ClinEnv has three components that together define what the authors call Longitudinal Inpatient Simulation.

First, an automated case construction pipeline converts raw inpatient admissions into ordered sequences of decision stages. Ground-truth decisions, medications, procedures, and diagnoses, are extracted directly from the EHR timeline and discharge documentation. Crucially, this requires no manual annotation, which means the benchmark can scale with available records rather than with annotator hours.

Second, an interactive multi-agent environment withholds information until the model asks. At each stage, the model must query four specialized agents before committing: a patient agent, a nurse agent, a laboratory agent, and a history agent. This mirrors the real structure of inpatient information flow, where a physician must decide what to ask, whom to ask, and when the cost of an additional test outweighs its expected value.

Third, a dual evaluation framework scores both the decision and the process. Decisions are matched deterministically through ontology grounding: ATC codes for medications, and hierarchical ICD F1 for diagnoses and procedures, using Hungarian matching with ontology-based partial credit. Process is scored through coverage metrics and laboratory and medication cost efficiency. This dual scoring is the conceptual heart of the paper, because it separates what the model decides from how it gathers information.

The Results: A Process Gap Made Visible

Across seven LLMs, the strongest model reaches only 0.31 decision F1. That number alone reframes the licensing-exam narrative. A system that scores near the ceiling on MedQA lands at less than a third of perfect performance when asked to actually manage a hospitalization.

Two patterns inside that aggregate number are more revealing than the number itself.

Difficulty concentrates in management, not diagnosis. Models recover discharge diagnoses at 0.51 F1 but management actions, the medications and procedures, at only 0.17 F1. This is intuitive once you think about it. Diagnosis is, structurally, closer to the classification task that licensing exams reward: synthesize findings, name the condition. Management is sequential and combinatorial. There are many acceptable drugs, doses, and orderings; the action space is larger; and the consequences compound over stages. The 0.51 versus 0.17 split is, in my reading, the quantitative signature of the gap between recognizing a disease and treating it over time.

Outcome quality is decoupled from process quality. Models continue to issue redundant queries as cases progress, indicating they do not build an efficient internal model of what they already know. A physician learns the patient over the course of an admission and asks fewer, more targeted questions. The models do not. This is exactly the dimension that outcome-only evaluation renders invisible, and it is the dimension ClinEnv was built to expose.

My Take: Information Seeking Is the Underrated Skill

The tweet version of this paper captures the punchline: the gap is in knowing what to ask, not what to answer. I want to push on why that matters beyond medicine.

Most LLM training and evaluation optimizes for producing a good final answer given a complete prompt. The implicit assumption is that the relevant information is already in context. Real decision-making violates this assumption constantly. Information is costly, partial, and acquired through deliberate action. The redundant-query behavior ClinEnv documents is, I suspect, a general failure mode of models trained primarily on answer-generation objectives. They have weak value-of-information reasoning: they cannot estimate how much a given query would reduce their uncertainty, so they over-query and under-prioritize.

This connects ClinEnv to the broader move in agentic AI, the same arc that took coding evaluation from HumanEval to SWE-bench. Executable, interactive environments do not just measure capability; they generate training signal. A reward that includes cost efficiency and coverage, not just terminal accuracy, could in principle teach a model to ask better questions. ClinEnv's process metrics are the scaffolding for exactly that kind of training objective. I would expect the most interesting follow-up work to use ClinEnv not as a static leaderboard but as an environment for reinforcement learning on information acquisition.

Limitations Worth Naming

ClinEnv is not without compromises of its own. Deriving ground truth from the recorded clinical course assumes the treating physicians made correct decisions, which is not always true; the benchmark partly measures agreement with historical practice rather than optimal practice. Ontology-based matching, while deterministic and reproducible, can penalize clinically equivalent alternatives that map to different codes, or reward superficial code matches. The multi-agent simulators for patient, nurse, lab, and history are still abstractions of messy real-world communication. And like all EHR-derived work, the cases inherit the demographic and institutional biases of their source data.

None of these undermines the central contribution. They are the price of building something realistic enough to be hard.

Closing Thoughts

ClinEnv's most important result is not a number but a reframing. Clinical competence is not a single scalar that licensing exams approximate; it decomposes into knowledge, diagnosis, management, and the meta-skill of acquiring information efficiently under uncertainty. By making the information-acquisition gap directly measurable, the benchmark gives the field a target it could previously only gesture at.

The open questions are sharp. Can value-of-information reasoning be trained directly, and does it transfer across clinical contexts? Will models that learn to ask better questions in ClinEnv generalize to genuinely novel admissions, or merely to the distribution of MIMIC-style records? And more broadly, how many other domains where we celebrate benchmark saturation are quietly hiding a process gap of their own? I suspect the answer is most of them.