Multimodal fusion with hyperspectral images and RGB information for all-optical informed virtual staining

Opt Express. 2025 May 5;33(9):19820-19836. doi: 10.1364/OE.557848.

Abstract

Histological analysis plays an irreplaceable role in tumor identification and disease development prediction. However, traditional histochemical approaches are inefficient due to their fussy, labor-intensive, and time-consuming workflows, potentially missing optimal treatment timing. In this paper, a deeply learned all-optical staining framework is proposed to achieve real-time, accurate, and plug-and-play pathology scans by introducing optical priors into the digital staining pipeline without any need for reagent staining. An end-to-end intermediate supervised optical-driven staining network (OSNet) is designed for automatic transformation from plain spectral transmittances to stained RGB images. Multi-scale convolution residual block (MultiResBlock) is specifically engineered to extract rich information from input spectral cubes. A pilot optical staining system is constructed to validate this approach using pairs of morphologically identical stained and unstained mice hepatoma slides. Experimental results demonstrate that the reconstructed H&E-stained images are visually equivalent to histochemically stained images, while the average scan time is just 3 minutes, over 13 times faster than conventional workflows. All-optical staining bypasses cumbersome histochemical procedures and offers results with high fidelity, addressing medical concerns related to interpretability and stability, which provides a new computationally generated panel for plain-to-stain processing with enormous potential in time-sensitive diagnosis, accurate treatment, and advanced molecular analysis.

MeSH terms

  • Animals
  • Carcinoma, Hepatocellular* / diagnostic imaging
  • Carcinoma, Hepatocellular* / pathology
  • Hyperspectral Imaging* / methods
  • Image Processing, Computer-Assisted* / methods
  • Liver Neoplasms* / diagnostic imaging
  • Liver Neoplasms* / pathology
  • Mice
  • Staining and Labeling* / methods