The Application of a Deep Learning Algorithm for the Segmentation of Retinal Nerve Fiber Layer Across Different Optic Neuropathies

Transl Vis Sci Technol. 2026 Apr 6;15(4):7. doi: 10.1167/tvst.15.4.7.

Abstract

Purpose: To determine the ability of a deep learning (DL) algorithm to segment retinal nerve fiber layer (RNFL) from optical coherence tomography (OCT) scans in glaucomatous optic neuropathies and anterior optic neuropathies with optic disc edema.

Methods: RNFL-Net was developed after preprocessing and automatically removing blood vessels from peripapillary OCT B-scans. It was trained and validated on 1065 RNFL OCT B-scans, and its performance was assessed using 265 test scans. Two different datasets were used for external testing.

Results: The study involved 106 eyes from healthy controls, 118 eyes with optic disc edema, and 60 eyes with glaucoma for training and validation. The segmentation method achieved a Dice coefficient of 0.95 for the validation dataset and 0.92 for test images when compared with manual segmentation. In measuring RNFL thickness in glaucoma-affected eyes, RNFL-Net showed a mean absolute error (MAE) of 6.21 µm and a mean absolute percentage error (MAPE) of 11.24%. The standard OCT device had MAE of 11.05 µm and MAPE of 16.8%. For optic disc edema, the RNFL-Net MAE was 13.04 µm and MAPE 5.71%, whereas the OCT device reported MAE of 22.94 µm and MAPE of 11.2%. For the external validation data, MAE values for glaucoma (n = 157) and disc edema (n = 32) cases were 7.19 ± 0.14 and 15.41 ± 0.32, respectively.

Conclusions: RNFL-Net can accurately segment RNFL, whereas standard OCT devices produce lower measurements, especially in disc edema.

Translational relevance: RNFL thickness measurements from RNFL-Net matched the ground truth in glaucoma and optic disc edema cases.

MeSH terms

  • Adult
  • Aged
  • Algorithms
  • Deep Learning*
  • Female
  • Glaucoma* / diagnostic imaging
  • Humans
  • Male
  • Middle Aged
  • Nerve Fibers* / pathology
  • Optic Disk / pathology
  • Optic Nerve Diseases* / diagnosis
  • Optic Nerve Diseases* / diagnostic imaging
  • Optic Nerve Diseases* / pathology
  • Papilledema / diagnostic imaging
  • Retinal Ganglion Cells* / pathology
  • Tomography, Optical Coherence / methods