Automated analysis of high-resolution transmission electron microscopy images of nanoparticles by machine learning
Year: 2024
Type: dissertation
Abstract: This work was focused on building an automated routine to analyse HRTEM images of nanoparticles for the nanoparticle size, shape, and structure. The program ANTEMA (Automated Nanoparticle Transmission Electron Micrograph Analysis) was built to detect the nanoparticles and evaluate them for shape and size related parameters. The main component of the program is the neural network which was trained to separate the particles from the background. Additionally, a postprocessing routine was built to filter out segmentation artefacts and to separate overlapping particles. For the analysis of the particle structure, different approaches were tested to train a neural network to classify particles based on their crystallinity using synthetic images. The approaches were compared to find the best approach that could be used to build a robust classification network. To build the program ANTEMA, the first step was to train a neural network for semantic segmentation to separate the particles from the background. For this, 128 HRTEM images of nanoparticles had to be manually labelled. Two different network architectures as well as two different pre-processing options for the HRTEM images were tested and the best-performing network was chosen to be implemented into the program. The final network has a DeepLabV3+ architecture with a ResNet-18 backbone and reached an accuracy of 96.3 % on the test dataset and an IoU value of 83 %. For further tests of the network performance and the usefulness of this approach, the segmentation using the network was compared to other segmentation methods. The comparison showed that standard techniques implemented for example in ImageJ are not suitable for many images and cannot be used in a highly automated fashion. In contrast, the neural network was able to segment most images as long as they were similar enough to the training data. To build a fully functioning program around the segmentation, a postprocessing routine was established. The postprocessing routine that was implemented cleans up the segmentation by removing small, mislabelled areas (equivalent diameter < 0.5 nm) and particles that cross the image boundary. Furthermore, two options (UECS and watershed) were implemented to separate overlapping particles. At the end of the postprocessing routine, multiple parameters are extracted. These include the circularity, equivalent diameter, minimum and maximum Feret diameter, perimeter, particle area, and centroid position. To make the program useable for users without programming knowledge, a graphical user interface was built. The full program was analysed by comparing it to manual evaluations. No systematic under- or overestimation of the particle size compared to the manual evaluations could be observed. By comparing multiple manual evaluations of the same images, the individual bias in manual evaluations could be shown. While the evaluations performed by the program cannot be said to be more accurate than the manual evaluations, they produce more comparable results as they can be deemed to always have the same bias. Furthermore, the program will always output the same result when analysing the same image. A major advantage of the program is the needed evaluation time which is up to 180 times faster than a manual evaluation while outputting more size and shape parameters. Additionally, the program was built to be able to analyse full image folders at once which not only makes the evaluation of large datasets easier and faster but consequently gives better statistical accuracy compared to manual evaluations. While most manual evaluations are only performed for one image with up to 100 particles the automated evaluation can analyse thousands of particles in a fraction of the time. To showcase the capabilities of the program, it is applied to different image datasets which include ultrasmall gold nanoparticles, larger gold nanoparticles with an equivalent diameter of around 10 nm, ultrasmall platinum metal particles, ultrasmall silver nanoparticles as well as larger platinum nanoparticles with different morphologies. It could be shown that the program is capable of analysing a wide variety of nanoparticle images. A limitation exists for particles of non-convex shapes as overlapping particles cannot be separated by the implemented separation routines which leads to an overestimation of the particle size. Therefore, the program can only be used on non-convex particles if no particles overlap in the image. Another limitation is that the network is only applicable to images that are similar enough to the training data. Therefore, a visual inspection of the generated maps is always necessary for new datasets. One type of dataset that the network is unable to segment properly are images taken in gaseous atmospheres as is the case for many in-situ datasets. As in-situ datasets often consist of hundreds or thousands of images, this automated routine would be especially useful. To make the network applicable to other datasets, transfer learning with few images of the datasets was used. The resulting network could then be used in the program to analyse the new dataset. This was done for two in-situ datasets that were acquired in gaseous atmospheres. After transfer learning shape changes in the particles over the time of the experiment could be quantitively tracked. Additionally, transfer learning was successfully applied to make the program applicable to TEM images of rod-shaped nanoparticles that were acquired at lower resolution. Apart from the shape and size parameters, the HRTEM images also show information on the particle’s crystallinity. To automate the analysis of the particle crystallinity different approaches to train a neural network for a classification of particles were tested. The main problem with training such a classification network is the generation of large amounts of correctly labelled images. As this is not feasible to do manually, two separate approaches for the generation of synthetic training data were tested. The first approach was a simple and computationally inexpensive pattern-based approach. The second approach was computationally more expensive image simulation using a program specifically designed for the simulation of HRTEM. The classification training was done for a two-class system distinguishing between amorphous and crystalline particles and a three-class system distinguishing between amorphous, monocrystalline and polycrystalline particles. To check the performance of the trained networks and their applicability to real data, they were not only tested on the test dataset but also on a small dataset of manually labelled real images. Both approaches were tested with small alterations to the data generation procedure in pursuit of optimizing the training result. It could be shown that the simple pattern-based approach was unfit to train a network that is applicable to real data. The more sophisticated simulation approach showed more promising results with an accuracy of 75.3 % for the first tested dataset. By adding images with a higher amorphous background signal to the dataset, the network’s performance could be further improved to an accuracy of 89.3 %. More accurate results by the same network can be achieved by labelling all particles with high uncertainty in the network’s classification as a separate unknown class. However, this can also lead to systematically removing similar particles from the evaluation which would then need further refinement in an actual application. The training was then expanded to a three-class system. Here, the distinction between monocrystalline and polycrystalline particles produces the most errors. This can be attributed to the limited samples of what a polycrystalline particle could look like. These tests showed that image simulations can be used to train a network but that the variability in real images needs to properly be represented in the simulated data. Given larger amounts of crystallographic data for different kinds of particles, it should be possible to further improve the network’s performance and it could also be possible to expand the classification to more classes. Such a network could then be added to the ANTEMA program to enable a fully automated, complete analysis of the size, shape and structure of nanoparticles from HRTEM images.
Author: Gumbiowski, Nina
Language: English
Subfield: Structural Biology
Domain: Life Sciences
Open Access status: green
OpenAlex ID: W6944134741
Cites:
Cited by:
Related to: