"What is relevant in a text document?": An interpretable machine learning approach

Abstract

Text documents can be described by a number of abstract concepts such as semantic category, writing style, or sentiment. Machine learning (ML) models have been trained to automatically map documents to these abstract concepts, allowing to annotate very large text collections, more than could be processed by a human in a lifetime. Besides predicting the text's category very accurately, it is also highly desirable to understand how and why the categorization process takes place. In this paper, we demonstrate that such understanding can be achieved by tracing the classification decision back to individual words using layer-wise relevance propagation (LRP), a recently developed technique for explaining predictions of complex non-linear classifiers. We train two word-based ML models, a convolutional neural network (CNN) and a bag-of-words SVM classifier, on a topic categorization task and adapt the LRP method to decompose the predictions of these models onto words. Resulting scores indicate how much individual words contribute to the overall classification decision. This enables one to distill relevant information from text documents without an explicit semantic information extraction step. We further use the word-wise relevance scores for generating novel vector-based document representations which capture semantic information. Based on these document vectors, we introduce a measure of model explanatory power and show that, although the SVM and CNN models perform similarly in terms of classification accuracy, the latter exhibits a higher level of explainability which makes it more comprehensible for humans and potentially more useful for other applications.

Fig 1. Diagram of a CNN-based interpretable machine learning system.

It consists of a forward processing that computes for each input document a high-level concept (e.g. semantic category or sentiment), and a redistribution procedure that explains the prediction in terms of words.

Fig 2. LRP heatmaps of the document sci.space 61393 for the CNN2 and SVM model.

Positive relevance is mapped to red, negative to blue. The color opacity is normalized to the maximum absolute relevance per document. The LRP target class and corresponding classification prediction score is indicated on the left.

Fig 3. The 20 most relevant words per class for the CNN2 model.

The words are listed in decreasing order of their LRP(first row)/SA(second row) relevance (value indicated in parentheses). Underlined words do not occur in the training data.

Fig 4. The 20 most relevant words per class for the BoW/SVM model.

The words are listed in decreasing order of their LRP(first row)/SA(second row) relevance (value indicated in parentheses). Underlined words do not occur in the training data.

Fig 5. PCA projection of the summary vectors of the 20Newsgroups test documents.

The LRP/SA based weightings were computed using the ML model’s predicted class, the colors denote the true labels.

Fig 6. Word deletion experiments for the CNN1, CNN2 and CNN3 model.

The LRP/SA target class is either the true document class, and words are deleted in decreasing (first row, lower curve is better) resp. increasing (second row, higher curve is better) order of their LRP/SA relevance, or else the target class is the predicted class (third row, higher curve is better) in which case words are deleted in decreasing order of their relevance. Random (biased) deletion is reported as average over 10 runs.

Fig 7. KNN accuracy when classifying the document summary vectors.

The accuracy is computed on one half of the 20Newsgroups test documents (other half is used as neighbors). Results are averaged over 10 random data splits.