← The Health AI Brief
Loading summary
Transcript1 lines
- [00:01]
A
A diagnostic artificial intelligence system correctly identifies a rare case of dermatomyositis from a clinical vignette. However, a closer inspection reveals that this digital assistant reached the diagnosis without ever opening or analyzing the patient's clinical photograph. This phenomenon represents an important vulnerability in current machine learning architectures, as highlighted by a peer reviewed study published in Nature Medicine. The research, led by Hugo Formally of Microsoft Research and now of ByteDance, the parent company of TikTok, alongside prominent co authors such as Eric Toppol and a team at Microsoft, systematically evaluate the operational readiness of frontier multimodal models in healthcare applications. We've previously discussed what seems to have been an earlier preprint of the same work, but now there's this more fully fleshed out research paper and there's a lot of interesting insight to take away from it. The core of this evaluation lies in moving past standard benchmark accuracy to inspect the structural robustness of state of the art systems. They evaluated models including GPT5 Gemini 2.5 Pro, OpenAI 03, OpenAI 04 Mini, Claude 3.5 Sonnet, and GPT4O, so models that are a few months off the state of the art frontier but still with important learning. While these systems demonstrate high baseline capabilities on standard medical licensing and diagnostic data sets, their clinical utility remains constrained by hidden fragilities. This analysis details how the models performed when subjected to adversarial pressure, where their reasoning pathways break down, and how clinical frameworks need to adapt to measure true capability. As the deployment of multimodal generative AI transitions from a theoretical pursuit into clinical decision support workflows, establishing concrete metrics for safety and reliability is paramount, standard benchmark leaderboards present an incomplete picture of model performance. To address this gap, the research team designed a framework consisting of six sequential adversarial stress tests. These tests were structured to escalating complexity, starting with basic input omissions and formatting modifications, progressing through to visual substitutions and culminating in audits of the model's underlying reasoning steps. The first two stress tests the authors evaluated modality sensitivity and necessity. So when analysing multimodal medical questions such as those from the New England Journal of Medicine's Image Challenge or JAMA Network's Clinical Challenge, the models are provided with a clinical vignette alongside a diagnostic image such as a radiograph, dermatological photograph, or pathology slide in the baseline setting, models achieve remarkably high accuracy. However, when the diagnostic image is completely removed, leaving only the text of the clinical vignette, the performance of the models remains surprisingly high. On the New England Journal of Medicine dataset. Removing the image led to a modest performance decline across most models. The paired accuracy difference ranged from 4.65 to 13.93 percentage points. For example, GPT5 scored 81.3% for the full image and text input but maintained an accuracy of 67.4% when the image was completely omitted. Similarly, Gemini 2.5 Pro dropped from 81% with full inputs to 67% without the image. On the JAMMA dataset the declines were even smaller. GPT5 accuracy decreased by only 4.4 percentage points, while GPT4O decreased by 4.0 percentage points. This very high performance in the absence of visual evidence suggests a significant reliance on text based shortcuts. The models are frequently capable of guessing the correct diagnosis based on clinical context, disease prevalence and memorised associations within their training data rather than the actual diagnostic clinical image. To investigate whether this behaviour persists when visual information is strictly necessary, the researchers curated a 197 item subset of the New England Journal benchmark designated as the New England Journal of Medicine Visual required subset or NEJMVs. This subset comprise clinical cases selected by board certified clinicians specifically because the textual vignette contains minimal diagnostic clues, making visual interpretation of the image mandatory for a correct diagnosis. When evaluated on the NEJM VS with full image and text inputs, the models demonstrated strong diagnostic capabilities. GPT5 achieved 70%, Gemini 2.5 Pro reached 67 and OpenAI 03 achieved 65%. These scores confirm that the cases are solvable when the visual modality is present. However, when the images were removed and the models were forced to evaluate the text alone, accuracy remained far above the 20% random baseline. GPT5 achieved 41% accuracy, Gemini 2.5 Pro 40% and OpenAI03 achieved 39%. This finding points to a persistent reliance on non visual cues and learned text priors. In a clinical environment, a model should recognize when key diagnostic information is missing and abstain from making a clinical prediction. Instead, most frontier models continue to generate diagnostic claims. There was, however, an instructive exception in this behaviour of GPT4O. When the image was removed from this NEGMVS dataset, GPT4O's overall accuracy fell to 16%. This low score doesn't reflect poorer diagnostic capability but rather a high rate of refusal to answer. GPT4O refused to generate a diagnosis in approximately 50% of the image absent cases. When these abstentions are excluded from the calculation of GPT4O's accuracy on the remaining cases, it rises to approximately 33%, aligning with the shortcut performance of the other frontier models. This indicates that GPT4O possessed superior uncertainty calibration and refusal handling, which are essential safety characteristics for anything looking for clinical deployment. Beyond the issue of missing data, clinical AI needs to display stability under superficial formatting changes. The third stress test evaluated format sensitivity by randomising the order of multiple choice answer options without altering any text or visual features under text only conditions. Shuffling the options caused modest declines in accuracy for most models, revealing a reliance on positional cues and answer formatting. Interestingly, when the visual input was restored, performance remained stable or improved slightly. This suggests that the presence of visual evidence can compensate for minor textual perturbations. The fourth stress test focused on distractor manipulation. The researchers iteratively replaced between one and four incorrect answer choices with unrelated options under text only conditions. Model accuracy declined steadily toward the 20% chance baseline as more distractions were replaced. This behaviour is consistent with the progressive removal of shortcuts. Conversely, in the image and text condition, performance improved by up to 40 percentage points. The inclusion of unrelated distractors made the correct option visually and contextually more salient. A key vulnerability was exposed when a single distractor was replaced with the option unknown. Ideally, models would select unknown as a fallback when key information is missing. However, the models treated unknown as an easily eliminated incorrect option and their accuracy increased. This confirmed that frontier models treat unknown as a semantic distractor rather than a functional pathway for expressing uncertainty. These formatting and distractor vulnerabilities represent superficial dependencies. A model with robust clinical understanding should maintain consistent diagnostic performance regardless of the order that answers appear in or variations of distractor text. This brings us to the fifth and perhaps most revealing diagnostic challenge, visual substitution. To isolate the model's visual grounding capabilities, the researchers selected 40 New England Journal cases where the diagnosis was critically dependent on image interpretation. They preserved the text of the vignette and answer choices, but replaced the original clinical image with a highly plausible alternative corresponding to one with the incorrect distractor options. All substituted images were verified by independent clinicians to ensure that they accurately represented the new target diagnosis. A reliable clinical AI would need to dynamically update its diagnostic conclusion when the visual evidence changes. However, most models failed this test. GPT5 accuracy collapsed from 84% in the baseline setting to 35% under visual substitution, representing a drop of 31 percentage points. Gemini 2.5 Pro fell from 76% to 52%. That's a 24% decline. OpenAI04 Mini dropped by 23 percentage points and OpenAI OpenAI03 fell by 30 percentage points. GPT4O was again the sole model to show an increase in performance rising from 26 to 36%, so where there were substantial performance drops. This indicates that standard clinical benchmarks overstate multimodal reasoning capacity. The residual accuracy observed in the perturbed setting such as GPT5 maintaining 53% on certain substitutions points to a reliance on image answer associations or memorization of the training set rather than active visual interpretation. The models struggle to re evaluate the clinical context when confronted with conflicting visual or textual evidence. To understand why these failures occur, we need to audit the internal reasoning steps of the models which was the focus of the sixth stress test. The research team evaluated reasoning signal integrity using two methods first. First, they applied chain of thought prompting to 120 items of the New England Journal dataset and 100 items of the VQA RAD dataset to determine if explicit step by step reasoning improved accuracy. Second, they conducted manual clinical audits of the model generated explanations for the answers that they're given. On the New England Journal dataset, chain of thought prompting failed to yield accuracy gains across all models on VQA RAD it produced only marginal improvements of 0.8 percentage points at most. They then did manual clinical audits of these reasoning chains and found three recurring failures. The first failures characterised by correct final answers accompanied by entirely fabricated visual findings. The model successfully guessed a correct multiple choice option but justified their choice by describing image features that were not present. In the second failure type, an initial visual misperception propagated through the reasoning chain, amplifying the error and leading to an incorrect final conclusion. The third type of failure involved highly structured syntactically coherent statements that were clinically irrelevant or uninformative, so these findings demonstrate that fluency in explanation doesn't equal clinical validity. Frontier models can produce highly convincing narratives that mask underlying reasoning errors or hallucinated features. In clinical practice, an incorrect explanation supporting a correct answer can be highly hazardous as it can mislead clinicians during the diagnostic process. To contextualise these model behaviours, the researchers developed a structured clinician guided Rubric to profile nine widely used medical benchmarks. Three board certified clinicians independently rated 30 to 50 representative cases from each benchmark across 10 dimensions covering complexity of reasoning to visual dependency. The resulting landscape reveals that medical benchmarks vary widely in what they actually measure. The New England Journal Image challenge ranks exceptionally high in both reasoning and visual complexity. In contrast, the JAMMER clinical challenge requires substantial clinical reasoning that's largely solvable using text alone. Benchmarks like VQA rad, PMC vqa, and the Mimic Chest X ray dataset are highly dependent on visual features but require low inference complexity. Meanwhile, something like OmniMedvqa clusters low in both reasoning and visual demand. This variation explains why models may perform well on certain benchmarks while failing under different stress. A model trained on datasets that require simple image localization, such as VQA rad, will struggle on complex diagnostic tasks like the New England Journal case series. They require integration of clinical history, pathophysiological knowledge, and detailed visual interpretation. So they argue, treating all medical benchmarks as interchangeable indicators of medical readiness is a flawed approach. I think it's a slight strawman argument, and I don't think that people do this that much in real practice. So translating these findings into clinical practice needs a shift in how medical AI is evaluated and deployed. The research team proposes three key recommendations to bridge the gap between benchmark performance and clinical safety. First, they argue, all medical benchmarks should be accompanied by detailed metadata describing their reason and visual complexity, as they've done here. They argue this will allow clinical administrators and technical teams to understand what a specific dataset actually measures and prevent the overestimation of model capabilities. However, this first recommendation presents distinct practical challenges. Case difficulty is very subjective and dynamic, meaning fixed complexity ratings offer limited utility. Standardizing these metrics risks burdening independent safety testers, potentially slowing down the creation of new evaluations. Regulators and external evaluators need to maintain complete autonomy free from testing guidelines proposed by commercial developers themselves. The primary clinical requirement is ensuring a system is objectively correct and robust against shortcuts. Abstract complexity scores provide little practical value in that regard. Their second suggestion is that evaluation protocols need to transition towards more active stress based assessments. AI developers and healthcare systems should routinely incorporate adversarial perturbations such as input omission, format shifting and distractor manipulation, and visual substitution in their validation pipelines. These robustness metrics need to be reported alongside standard accuracy scores. I fully agree this is something important for model developers to test and publish openly. Third, models need to be evaluated within the specific institutional, regulatory and operational context of their intended deployment. A closed sourced API based model may achieve very high scores on public leaderboards but face significant integration challenges regarding data privacy, regulatory compliance and things like operational cost. Open source or on premises models, while sometimes scoring lower on general benchmarks, offer healthcare institutions greater control over things like data governance, system auditing and safety monitoring. Ultimately, these findings show that the current multimodal AI models possess impressive but brittle capabilities. They demonstrate a strong capacity for clinical text synthesis and can often make use of non visual shortcuts to achieve high diagnostic scores. However, their visual grounding remains very fragile and their self generated explanations are frequently unaligned with the actual decision making process. This doesn't diminish the clinical promise of multimodal systems. Rather, it provides a rigorous, clinically grounded diagnostic framework to identify where these systems need to improve. By implementing systematic stress testing, clinical metadata profiling, and robust uncertainty handling, the medical AI community can move beyond leaderboard optimization and build systems that are genuinely ready for safe research. Real world clinical deployment It's a thorough piece of research, and again, as ever, I'd recommend anyone read the primary source if they're interested to see more.