Automated detection of mouse scratching behaviour using convolutional recurrent neural network

Abstract

Scratching is one of the most important behaviours in experimental animals because it can reflect itching and/or psychological stress. Here, we aimed to establish a novel method to detect scratching using deep neural network. Scratching was elicited by injecting a chemical pruritogen lysophosphatidic acid to the back of a mouse, and behaviour was recorded using a standard handy camera. Images showing differences between two consecutive frames in each video were generated, and each frame was manually labelled as showing scratching behaviour or not. Next, a convolutional recurrent neural network (CRNN), composed of sequential convolution, recurrent, and fully connected blocks, was constructed. The CRNN was trained using the manually labelled images and then evaluated for accuracy using a first-look dataset. Sensitivity and positive predictive rates reached 81.6% and 87.9%, respectively. The predicted number and durations of scratching events correlated with those of the human observation. The trained CRNN could also successfully detect scratching in the hapten-induced atopic dermatitis mouse model (sensitivity, 94.8%; positive predictive rate, 82.1%). In conclusion, we established a novel scratching detection method using CRNN and showed that it can be used to study disease models.

Figure 1

Image pre-processing and integration. (a) Video file was divided into the images of each frame. The absolute difference of each pixel between two continuous frames was calculated and cropped in a square shape (300 × 300 pixel) around the geometric centre of mouse. Images were then grey-scaled and binarized. (b) For segment at time t, the pre-processed images from t − 10 to t + 10 was collected and labelled with the value of frame label at time t. The figure shows segment at t − 1 (including images from t − 11 to t + 9, labelled as “0”), segment at t (including images from t − 10 to t + 10, labelled as “1”), and segment at t + 1 (including images from t − 9 to t + 11, labelled as “1”).

Figure 2

CRNN architecture and training. (a) The architecture of CRNN. The images in one segment was separately input into CNN block. The output was flattened and integrated in RNN block and in full connected block. The detailed shapes of output tensor from each layer were shown in Supplementary Table S4. CV convolution, MP maxpooling, LSTM long short-term memory, FC fully connected. (b) The change of loss value during training.

Figure 3

The result of CRNN training. (a) The example of CRNN output interpretation. A segment whose CRNN output was more than 0.5 was classified as scratching segment. (b) The number of true positive/negative segments and false positive/negative segments in the training dataset. (c) The number of true positive/negative segments and false positive/negative segments in the test dataset. (d) The comparison of scratching counts in each video file between prediction and observation in the training dataset. (e) The comparison of duration time of each scratching event between prediction and observation in the training dataset. (f) The comparison of scratching counts in each video file between prediction and observation in the test dataset. (g) The comparison of duration time of each scratching event between prediction and observation in the test dataset. Significantly deviated event was indicated as arrow (see Fig. 4d). The dotted lines indicate the line when prediction was equal to observation.

Figure 4

The detailed investigation of error segments. (a) Three error types. Black boxes indicate scratching segment in prediction or observation. Grey boxes indicate error segment. (b) The number of three errors in the training dataset. (c) The number of three errors in the test dataset. (d) Detailed investigation of the significantly deviated scratching event (indicated by arrow in Fig. 3g). (e) The representative data of grooming and scratching. Black boxes indicate grooming or scratching segment.

Figure 5

Application to DNFB-induced dermatitis model. (a) The number of true positive/negative segments and false positive/negative segments of DNFB-treated mouse video files. (b) The number of three errors. (c) The comparison of scratching counts in each video file between prediction and observation. (d) The comparison of duration time of each scratching event between prediction and observation. The dotted lines indicate the line when prediction was equal to observation.