Assessing the Memory Ability of Recurrent Neural Networks
Published in 2020 European Conference on Artificial Intelligence (ECAI 2020), 2019
Recommended citation: Cheng Zhang, Qiuchi Li, Lingyu Hua and Dawei Song. (2020). "Assessing the Memory Ability of Recurrent Neural Networks." 2020 European Conference on Artificial Intelligence (ECAI 2020). https://qiuchili.github.io/files/ecai20-long.pdf
It is known that Recurrent Neural Networks (RNNs) can remember, in their hidden layers, part of the semantic information expressed by a sequence (e.g., a sentence) that is being processed. Different types of recurrent units have been designed to enable RNNs to remember information over longer time spans. However, the memory abilities of different recurrent units are still theoretically and empirically unclear, thus limiting the development of more effective and explainable RNNs. To tackle the problem, in this paper, we identify and analyze the internal and external factors that affect the memory ability of RNNs, and propose a Semantic Euclidean Space to represent the semantics expressed by a sequence. Based on the Semantic Euclidean Space, a series of evaluation indicators are defined to measure the memory abilities of different recurrent units and analyze their limitations. These evaluation indicators also provide a useful guidance to select suitable sequence lengths for different RNNs during training.