Machine learning based detection and deep learning based image inpainting of preparation artefacts in micrographs

A. Jansche, A.K. Choudhary, T. Bernthaler, G. Schneider
Aalen University,

Keywords: machine learning, deep learning, image segmentation, object classification, image inpainting, convolutional neural networks, partial convolutions, quantitative microstructure analysis, non-metallic inclusions


Automated quantitative microstructure analysis (QMA) as well as qualitative inspection of samples by experts are essential tools for materials science as well as industrial applications of materials microscopy like quality assurance. In order to get reliable results based on the microstructure of materials the metallographic preparation of the samples has to be of high standard or at least good enough to not interfere with the parts of the microstructure one intends to analyze. However, achieving preparation free of artefacts over large areas is often not possible due to the challenging nature of the materials or time constraints in industrial settings. We present an extended workflow based on previous work for the analysis of non-metallic inclusions that includes a) the segmentation of micrographs by pixel-wise classification using a multi-layer perceptron and RGB features, b) object classification to distinguish oxides, sulphides and preparation artefacts using a random forest classifier and Haralick texture features and c) an approach based on convolutional neural networks and partial convolutions for image inpainting to remove preparation artefacts as well as other segmented objects from the micrograph. The first two steps of the workflow provide a binary image for each class which can be used to mask the relevant areas for the image inpainting step. Given the accurate segmentation of the first step, the masks cover the bare minimum needed for the inpainting and thereby minimize the manipulation of the image leaving most of the microstructure untouched. The inpainted images can then be presented to experts without distracting artefacts or only showing certain objects/phases. It is also investigated if the inpainted images can be used as artefact-free images for standardized QMA methods (such as EN10247, ASTME45 or ISO4967 in the case of non-metallic inclusions) and how it affects the results. While the latter would enable more reliable QMA results the former would support the viewer during a qualitative assessment of the sample as only the relevant parts of the micrograph are shown. This allows an unobstructed view on certain objects or phases and their size, morphological features or distribution within the sample. Even though the approach is presented for non-metallic inclusions we believe it is transferable to other materials and microstructures.