Evaluating Generative AI for Medical Image Captioning: A Benchmark Study on Radiology Images

The generation of clinically accurate and contextually rich captions for radiology images is a critical task in medical Artificial Intelligence (AI), with applications in education, documentation, and decision support. In this study, we benchmark the performance of leading generative AI models—including OpenAI’s GPT 4o, Google’s Gemini 2.5 pro, Anthropic’s Claude 4.5 Sonnet and Meta’s LLaMA 4—on the recently released ROCOv2 (Radiology Objects in COntext Version 2) dataset. ROCOv2 offers a large-scale, multimodal resource of radiology images paired with expert-generated captions, enabling robust evaluation of vision-language models in the medical domain. We assess models under zero-shot and few-shot prompting conditions, and evaluate outputs using automated metrics (BLEU, BERTScore, ROUGE and METEOR). Our analysis highlights the capabilities and limitations of current generative models in understanding and describing complex radiological content, discussing the potential for integrating these models into clinical workflows. This work provides a comprehensive evaluation of generative AI for medical image captioning and offers insights into future directions for improving reliability and clinical relevance in multimodal medical AI systems.