Work

Test Metrics for Recurrent Neural Networks.

Year: 2019
Type: preprint
Abstract: Recurrent neural networks (RNNs) have been applied to a broad range of applications such as natural language processing, drug discovery, and video recognition. This paper develops a coverage-guided testing approach for a major class of RNNs -- long short-term memory networks (LSTMs). We start from defining a family of three test metrics that are designed to quantify not only the values but also the temporal relations (including both step-wise and bounded-length) learned through LSTM's internal structures. While testing, random mutation enhanced with the coverage knowledge, i.e., targeted mutation, is designed to generate test cases. Based on these, we develop the coverage-guided testing tool testRNN. To our knowledge, this is the first time structural coverage metrics are used to test LSTMs. We extensively evaluate testRNN with a variety of LSTM benchmarks. Experiments confirm that there is a positive correlation between adversary rate and coverage rate, evidence showing that the test metrics are valid indicators of robustness evaluation. Also, we show that testRNN effectively captures erroneous behaviours in RNNs. Furthermore, meaningful information can be collected from testRNN for users to understand what the testing results represent. This is in contrast to most neural network testing works, and we believe testRNN is an important step towards interpretable neural network testing.
Language: English
Subfield: Software
Open Access status: green
OpenAlex ID: W2983209617
Cites: 29
Cited by: 6
Related to: 10