Tool-Augmented Reasoning in Radiology: How Reinforcement Learning Enables Transparent CT Interpretation

Introduction

Medical imaging has entered an era where vision-language models (VLMs) generate radiological reports with increasing fluency. Yet this capability masks a fundamental tension in clinical deployment. While models like CT-Chat demonstrate strong performance on benchmark datasets, they operate as opaque inference engines, delivering final conclusions without revealing the path taken to reach them. For high-stakes decisions in radiology, this opacity creates a barrier to clinical adoption, relegating clinicians to passive observers of black box outputs.

In "RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography," Roschewitz et al. propose a fundamental architectural shift. Rather than training an end to end model to map volumetric CT scans directly to text reports, they frame radiological interpretation as an explicit reasoning process mediated by specialized tools. The result is an agent that not only generates reports but produces a complete, inspectable trace of its diagnostic reasoning, allowing clinicians to examine how each finding was derived.

Technical Architecture: Training Agents for Clinical Tool Use

The technical contribution of RadAgent lies in its departure from the prevailing paradigm of training free medical agents. Previous systems such as CT-Agent and CTPA-Agent rely on frozen large language models orchestrating predefined tool sequences through careful prompt engineering. While these systems achieve some degree of modularity, they assume that the underlying LLM possesses sufficient medical knowledge to design appropriate diagnostic plans, and that static tool workflows suffice for complex 3D volumetric analysis.

RadAgent instead employs reinforcement learning from verifiable rewards to train its orchestration policy. This approach treats tool selection and sequencing as a decision making problem where the agent learns through environmental interaction. The agent navigates a space of specialized capabilities including segmentation models for anatomical structures, 2D slice visual question answering for localized inspection, and 3D VQA for holistic volume assessment.

This training methodology enables the discovery of non obvious tool use strategies. Rather than following a rigid checklist, RadAgent learns to iteratively refine hypotheses. For instance, the paper illustrates a trace where preliminary findings of pleural effusion trigger a specific verification sequence: segmentation confirmation followed by targeted slice extraction and VQA validation before incorporation into the final report. This mirrors the cognitive process of radiologists who systematically verify initial impressions against specific imaging evidence.

Performance Metrics and the Faithfulness Breakthrough

The quantitative results demonstrate that this architectural shift yields concrete benefits across multiple dimensions of clinical utility. On internal and external test sets, RadAgent achieves a 6.0 point absolute improvement in macro-F1 over its 3D VLM counterpart CT-Chat, representing a 36.4% relative gain. Micro-F1 improves by 5.4 points (19.6% relative), indicating better performance across both common and rare findings.

More significantly, RadAgent demonstrates markedly superior robustness under adversarial conditions, improving by 24.7 points (41.9% relative). In clinical AI, adversarial robustness often distinguishes models that memorize statistical patterns from those that leverage genuine anatomical understanding. This suggests that the explicit tool grounding forces the model to rely on actual visual evidence rather than spurious correlations.

Perhaps the most intriguing metric is faithfulness, measured using the methodology proposed by Chen et al. Here RadAgent achieves 37.0% compared to 0.0% for CT-Chat. This metric evaluates whether reported findings are actually grounded in the visual evidence presented. The binary nature of this comparison highlights a qualitative difference in model behavior. CT-Chat, operating as a black box, cannot demonstrate that its conclusions derive from specific image regions. RadAgent, through its explicit tool invocations, creates an auditable chain of evidence linking each finding to specific analytical operations.

Critical Assessment: The 37% Problem and Beyond

While the results are compelling, they also illuminate the distance remaining before such systems achieve clinical reliability. The emergence of faithfulness as a measurable capability raises important questions about the current state of medical AI. While 37% represents a significant advance over zero, it also indicates that nearly two thirds of findings still lack clear evidential grounding. This metric likely represents an upper bound constrained by the current tool set and reward structure,