NLM at ImageCLEF 2017 caption task

This paper describes the participation of the U.S. National Library of Medicine (NLM) in the ImageCLEF 2017 caption task. We proposed different machine learning methods using training subsets that we selected from the provided data as well as retrieval methods using external data. For the concept detection subtask, we used Convolutional Neural Networks (CNNs) and Binary Relevance using decision trees for multi-label classification. We also proposed a retrieval-based approach using Open-i image search engine and MetaMapLite to recognize relevant terms and associated Concept Unique Identifiers (CUIs). For the caption prediction subtask, we used the recognized CUIs and the UMLS to generate the captions. We also applied Open-i to retrieve similar images and their captions. We submitted ten runs for the concept detection subtask and six runs for the caption prediction subtask. CNNs provided good results with regards to the size of the selected subsets and the limited number of CUIs used for training. Using the CUIs recognized by the CNNs, our UMLS-based method for caption prediction obtained good results with 0.2247 mean BLUE score. In both subtasks, the best results were achieved using retrieval-based approaches outperforming all submitted runs by all the participants with 0.1718 mean F1 score in the concept detection subtask and 0.5634 mean BLUE score in the caption prediction subtask.