Talebpour, Mozhgan and Garcia Seco De Herrera, Alba and Jameel, Shoaib (2023) Topics in Contextualised Attention Embeddings. In: European Conference on Information Retrieval (ECIR), 2023-04-10 - 2023-04-14, Stavanger, Norway.
Talebpour, Mozhgan and Garcia Seco De Herrera, Alba and Jameel, Shoaib (2023) Topics in Contextualised Attention Embeddings. In: European Conference on Information Retrieval (ECIR), 2023-04-10 - 2023-04-14, Stavanger, Norway.
Talebpour, Mozhgan and Garcia Seco De Herrera, Alba and Jameel, Shoaib (2023) Topics in Contextualised Attention Embeddings. In: European Conference on Information Retrieval (ECIR), 2023-04-10 - 2023-04-14, Stavanger, Norway.
Abstract
Contextualised word vectors obtained via pre-trained language models encode a variety of knowledge that has already been exploited in applications. Complementary to these language models are probabilistic topic models that learn thematic patterns from the text. Recent work has demonstrated that conducting clustering on the wordlevel contextual representations from a language model emulates word clusters that are discovered in latent topics of words from Latent Dirichlet Allocation. The important question is how such topical word clusters are automatically formed, through clustering, in the language model when it has not been explicitly designed to model latent topics. To address this question, we design different probe experiments. Using BERT and DistilBERT, we find that the attention framework plays a key role in modelling such word topic clusters. We strongly believe that our work paves way for further research into the relationships between probabilistic topic models and pre-trained language models.
Item Type: | Conference or Workshop Item (Paper) |
---|---|
Additional Information: | Published proceedings: _not provided_ |
Divisions: | Faculty of Science and Health Faculty of Science and Health > Computer Science and Electronic Engineering, School of |
SWORD Depositor: | Unnamed user with email elements@essex.ac.uk |
Depositing User: | Unnamed user with email elements@essex.ac.uk |
Date Deposited: | 17 Oct 2024 16:56 |
Last Modified: | 30 Oct 2024 21:01 |
URI: | http://repository.essex.ac.uk/id/eprint/35009 |
Available files
Filename: BERT_TopicMining (2).pdf