Research Repository

Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality

Salle, Alex and Villavicencio, Aline (2018) Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality. In: The 56th Annual Meeting of the Association for Computational Linguistics, 2018-07-15 - 2018-07-20, Melbourne.

[img]
Preview
Text
document.pdf - Published Version

Download (239kB) | Preview

Abstract

Increasing the capacity of recurrent neural networks (RNN) usually involves augmenting the size of the hidden layer, with significant increase of computational cost. Recurrent neural tensor networks (RNTN)increase capacity using distinct hidden layer weights for each word, but with greater costs in memory usage. In this paper, we introduce restricted recurrent neural tensor networks (r-RNTN) which re-serve distinct hidden layer weights for frequent vocabulary words while sharing a single set of weights for infrequent words. Perplexity evaluations show that for fixed hidden layer sizes, r-RNTNs improve language model performance over RNNs using only a small fraction of the parameters of unrestricted RNTNs. These results hold for r-RNTNs using Gated Recurrent Units and Long Short-Term Memory.

Item Type: Conference or Workshop Item (Paper)
Additional Information: Published proceedings: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers)
Subjects: P Language and Literature > P Philology. Linguistics
Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions: Faculty of Science and Health > Computer Science and Electronic Engineering, School of
Depositing User: Elements
Date Deposited: 13 Feb 2019 16:30
Last Modified: 13 Feb 2019 17:15
URI: http://repository.essex.ac.uk/id/eprint/22998

Actions (login required)

View Item View Item