Research Repository

NLM at ImageCLEF 2017 caption task

Abacha, AB and Seco De Herrera, AG and Gayen, S and Demner-Fushman, D and Antani, S (2017) NLM at ImageCLEF 2017 caption task. In: CLEF Conference and Labs of the Evaluation Forum, CLEF 2017, 2017-09-11 - 2017-09-14, Dublin; Ireland.

NLM_ImageCLEFcaption2017.pdf - Accepted Version

Download (357kB) | Preview


This paper describes the participation of the U.S. National Library of Medicine (NLM) in the ImageCLEF 2017 caption task. We proposed different machine learning methods using training subsets that we selected from the provided data as well as retrieval methods using external data. For the concept detection subtask, we used Convolutional Neural Networks (CNNs) and Binary Relevance using decision trees for multi-label classification. We also proposed a retrieval-based approach using Open-i image search engine and MetaMapLite to recognize relevant terms and associated Concept Unique Identifiers (CUIs). For the caption prediction subtask, we used the recognized CUIs and the UMLS to generate the captions. We also applied Open-i to retrieve similar images and their captions. We submitted ten runs for the concept detection subtask and six runs for the caption prediction subtask. CNNs provided good results with regards to the size of the selected subsets and the limited number of CUIs used for training. Using the CUIs recognized by the CNNs, our UMLS-based method for caption prediction obtained good results with 0.2247 mean BLUE score. In both subtasks, the best results were achieved using retrieval-based approaches outperforming all submitted runs by all the participants with 0.1718 mean F1 score in the concept detection subtask and 0.5634 mean BLUE score in the caption prediction subtask.

Item Type: Conference or Workshop Item (Paper)
Additional Information: Published proceedings: Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum (CLEF 2017), Dublin, Ireland, September 11-14, 2017.
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions: Faculty of Science and Health
Faculty of Science and Health > Computer Science and Electronic Engineering, School of
SWORD Depositor: Elements
Depositing User: Elements
Date Deposited: 21 Jan 2020 14:28
Last Modified: 23 Sep 2022 19:21

Actions (login required)

View Item View Item