Nuclei Segmentation of In Vivo Optical Coherence Tomography by Deep Learning

S.W. Chang, Y.H. Wu, P.S. Yeh, S.L. Huang
National Taiwan University,

Keywords: Optical coherence tomography, AI, deep learning algorithm


Optical coherence tomography (OCT) enables non-invasive imaging for pathologists to access cellular level images of in vivo human tissues, which could help unveil functions of living organisms and facilitate clinical disease/cancer diagnosis in the early stage. To discriminate between normal and cancer cells, the cell size, orientation, morphology, and distribution are essential indicators. In this study, deep learning algorithms are employed to detect the crucial tissue features, including nuclei and the dermal-epidermal junction of human skins. The segmentation outcome is quantitatively assessed, and the performance can be explained by visualizing the feature activations of the convolutional neural network in response to the cell-like structure of human tissues. The results show that cellular-resolution imaging can be used to identify regions that suggest abnormalities and should be biopsied for histopathological examination. Both morphological recognition, as well as parametric analysis using the backscattering from the subcellular structures, are evaluated. In addition, a pseudo coloring model is proposed to real-time convert OCT images to stained images. The algorithm for pseudo coloring of OCT images is based on CycleGAN architecture using segmentation information from the pathologist. The algorithm uses unpaired OCT and stained images with various loss functions to improve the image conversion accuracy.