Automatic real-time detection of endoscopic procedures using temporal features

Abstract

Endoscopy is used for inspection of the inner surface of organs such as the colon. During endoscopic inspection of the colon or colonoscopy, a tiny video camera generates a video signal, which is displayed on a monitor for interpretation in real-time by physicians. In practice, these images are not typically captured, which may be attributed by lack of fully automated tools for capturing, analysis of important contents, and quick and easy retrieval of these contents. This paper presents the description and evaluation results of our novel software that uses new metrics based on image color and motion over time to automatically record all images of an individual endoscopic procedure into a single digitized video file. The software automatically discards out-patient video frames between different endoscopic procedures. We validated our software system on 2464 h of live video (over 265 million frames) from endoscopy units where colonoscopy and upper endoscopy were performed. Our previous classification method achieved a frame-based sensitivity of 100.00%, but only a specificity of 89.22%. Our new method achieved a frame-based sensitivity and specificity of 99.90% and 99.97%, a significant improvement. Our system is robust for day-to-day use in medical practice.

Fig. 1

Examples of (a) inside-the-patient (left) and (b) outside-the-patient (middle) frames. An example of (c) an outside-the-patient frame (right) that resembles the color and brightness of an inside-the-patient frame due to an external light.

Fig. 2

The circular FIFO video frame buffer contains several internal pointers based on how much each frame in the FIFO buffer has been processed. A indicates the head of the FIFO buffer where newly captured frames are written. B points to the oldest unanalyzed frame. C points to the oldest potential procedure image whose classification of inside or outside is still unknown. D points to the tail of the FIFO buffer where images are either written to disk or discarded. Each tick mark represents a single frame.

Fig. 3

Examples of features graphed over time of a sample procedure entrance (a) and exit (b). In the top half of each image, the red line corresponds to the mean-red feature, the magenta line to mean-normalized-red, the yellow line to the histogram difference; and the green and red areas correspond to the mean-normalized-red rise and fall differences, respectively. On the bottom half of the figure, the red line corresponds to the mean-red variance of differences, the magenta line to mean-normalized-red variance of differences, the tan line to mean-red energy histogram, the pink line to the mean-normalized-red energy histogram, and the light yellow line to the hybrid mean-red/mean-normalized-red energy histogram. In the middle, the red, green, and blue bars correspond to frames being detected as frames inside a procedure by our algorithm. The white bar is the ground truth for frames being part of a procedure, set by a real person observing the actual video and marking the precise entrance and exit frames. A sharp rise in mean-normalized-red, one of the features shown on the top half of each graph, is indicative of the precise entrance frame.

Fig. 4

Energy histograms generated from two different sets of data. For the left histogram, a very high peak of one value causes the other bins to be scaled to a smaller value. The high peak is likely due to a flat signal, indicating very little change, and usually only occurs outside a procedure. The “area” computed is the space filled (the histogram bins) within the overall histogram graph (the yellow box). When a high peak causes the rest of the values to be scaled very low, the overall area will also be low. For the right histogram, we have more regular variation in the mean-red values. As a result, more of the histogram bins are allowed to retain a higher value when scaled. The overall area occupied by the bins in the histogram will be significantly larger in this case. This usually occurs during a procedure.

Fig. 5

Double-normalized energy histograms generated from two different sets of data. For the right histogram, a typical signal of a procedure has its “area” (the space taken by the actual bin data) computed within a rectangle whose left and right bounds are the first nonzero bin and the last nonzero bin, respectively. This is shown as the yellow rectangle. We normalized the area computed inside the rectangle, as if the area inside the yellow rectangle were 1.0. This is different than the previous feature in that the width was always 256 bins, whereas in this feature, the width adjusts to the data. For the left histogram, a somewhat flat signal outside-the-patient has resulted in a large peak, similar to the histogram in Figure 4. Although the rectangle that can fully encompass the nonzero bins can be much smaller than the yellow box indicates, we restrict the minimum width of the histogram to 128 bins; otherwise, this peak could cover a considerable area in the resized boundaries, generating a high value.

Fig. 6

Algorithm to detect the entrance frame

Fig. 7

Algorithm to detect the exit frame