Application of augmented reality using automatic markerless registration for facial plastic and reconstructive surgery

Abstract

This study aimed to present a novel markerless augmented reality (AR) system using automatic registration based on machine-learning algorithms that visualize the facial region and provide an intraoperative guide for facial plastic and reconstructive surgeries. This study prospectively enrolled 20 patients scheduled for facial plastic and reconstructive surgeries. The AR system visualizes computed tomographic data in three-dimensional (3D) space by aligning with the point clouds captured by a 3D camera. Point cloud registration consists of two stages: the preliminary registration gives an initial estimate of the transformation using landmark detection, followed by the precise registration using Iterative Closest Point algorithms. Computed Tomography (CT) data can be visualized as two-dimensional slice images or 3D images by the AR system. The AR registration error was defined as the cloud-to-cloud distance between the surface data obtained from the CT and 3D camera. The error was calculated in each facial territory, including the upper, middle, and lower face, while patients were awake and orally intubated, respectively. The mean registration errors were 1.490 ± 0.384 mm and 1.948 ± 0.638 mm while patients were awake and orally intubated, respectively. There was a significant difference in the errors in the lower face between patients while they were awake (1.502 ± 0.480 mm) and orally intubated (2.325 ± 0.971 mm) when stratified by facial territories (p = 0.006). The markerless AR can accurately visualize the facial region with a mean overall registration error of 1-2 mm, with a slight increase in the lower face due to errors arising from tube intubation.