Development of a method for analyzing the positional correlation of local structures in scanning probe microscopy images using template-matching image-processing method

The functionalities of solid materials are determined by the type and arrangement of their constituent atoms. However, perturbations induced by factors such as defects and interactions between adsorbates can often cause significant alterations of their functionality. Such perturbations have been well documented in various fields such as catalysis, superconductivity, and carrier transport, ^1–6) demonstrating their considerable impact on the behavior of solids. Understanding these subtle perturbations is crucial because they often hold the key to unlocking new possibilities in materials science. ^7–16) However, detecting them is challenging because of their inherent weaknesses.

In the quest to understand the mechanism of perturbations, the analysis of local structures using microscopic images of materials has emerged as a potent tool. ^17–25) Among the microscopic techniques available, scanning probe microscopy (SPM) stands out as a premier choice due to its exceptional resolution, including the ability to achieve atomic resolution. ^26,27) Moreover, SPM can reveal minute physical properties, encompassing phenomena such as molecule–molecule interactions, electronic interactions, lattice–electron interactions, and spin–orbit interactions. ^26,27) Despite the fact that SPM images provide us with the treasure trove of information, the detection and interpretation of these intricate, small-scale interactions often pose significant challenges.

To comprehend SPM images better, researchers have traditionally used techniques such as Fourier transform and autocorrelation analysis. ^26,27) These methods have proven to be effective in revealing the features of periodic and simple structures. ^28–30) However, their applicability diminishes when used in non-periodic and complex structural systems, rendering them inadequate for capturing the full scope of local structural intricacies. In particular, the analysis of positional correlations between local structures is challenging. These positional correlations inherently encapsulate the interactions between distinct species, and thus, we aspire to gain insight into their intricate behavior. Nonetheless, in cases where periodicity is absent within the image or where local structures are entwined with complex counterparts, the application of the analysis proves to be demanding.

In response to these challenges, we have developed an image-processing technique that enable us to recognize local structures and analyze the positional correlations among them. In this study, we introduce a method for the automated recognition of structures within SPM images coupled with calculations of positional correlation trends among the identified structures. To verify the validity of our approach, we conducted experiments using SPM images obtained under actual conditions. The results verify the ability of the method to autonomously identify structures and accurately compute their positional correlations. This cutting-edge advancement in image processing not only promises to clarify local structure–function relationships in materials science but also opens new doors for the investigation of intricate perturbations in the sciences.

The purpose of this study is to recognize the local structure of SPM images and analyze their positional correlation from complex and non-periodic systems. The proposed method is divided into two steps: (1) recognition of the structure, and (2) calculation of positional correlations in recognized structure groups. In this paper, we describe a method for automatically recognizing the structures in an image. The characteristics of the SPM images with complex structures are as follows:

• −  
  The structure to be recognized depends on the sample and imaging parameters, each of which exhibits unique characteristics.
• −  
  Considering the actual experimental situation, we assume that the number of images is less than 1000. ³¹⁾

Typical methods for automatic recognition include methods that use deep-learning techniques such as convolutional neural networks. ^32–34) Methods based on the deep-learning technology can automatically extract features from images and achieve high recognition accuracy. ³³⁾ However, deep learning is computationally intensive. In addition, the structures to be recognized can be very different depending on various factors, which indicates we may have to learn models for each structure independently.

Considering the above-mentioned issue, in this study, we employed a method different from the deep-learning technique to automatically recognize structures in SPM images. Figure 1 depicts a summary of our approach to recognizing local structures and calculating positional correlations. From a source image to be analyzed, we manually take a template image which describes the local structure [Fig. 1(a)]. Next, the similarity between the template and the analyzed images is calculated using the zero-mean normalized cross-correlation (ZNCC) of the pixels, as expressed by Eq. (1):

$$\begin{eqnarray}&&\begin{array}{l}{ZNCC}\left(x,y\right)\\ =\,\frac{{\sum }_{i=0}^{w-1}{\sum }_{j=0}^{h-1}\left(I\left(x^{\prime} ,y^{\prime} )-{\mu }_{I(x,y)}\right)\times (T\left(i,j\right)-{\mu }_{T}\right)}{\sqrt{{\sum }_{i=0}^{w-1}{\sum }_{j=0}^{h-1}{\left(I\left(x^{\prime} ,y^{\prime} \right)-{\mu }_{I(x,y)}\right)}^{2}}\times \sqrt{{\sum }_{i=0}^{w-1}{\sum }_{j=0}^{h-1}{\left(T\left(i,j\right)-{\mu }_{T}\right)}^{2}}}\end{array}\end{eqnarray} \tag{ 1 }$$

where

$$\begin{eqnarray*}&&{\mu }_{I(x,y)}=\frac{1}{{wh}}\sum _{i=0}^{{w}_{I}-1}\sum _{j=0}^{{h}_{I}-1}I(x+i,y+j)\end{eqnarray*}$$

$$\begin{eqnarray*}&&{\mu }_{T}=\frac{1}{{wh}}\sum _{i=0}^{{w}_{I}-1}\sum _{j=0}^{{h}_{I}-1}T(i,j)\end{eqnarray*}$$

ZNCC: zero-mean normalized cross-correlation

Zoom In Zoom Out Reset image size

(x, y): Position of point to be examined

I(x', y'): Intensity value of the pixel at (x', y') of the source image

x' = x + i

y' = y + j

${\mu }_{I(x,{y})}:$ Average of intensity values of the pixel in the calculated area of the source image

T(i, j): Intensity value of the pixel at (i, j) of the template image

${\mu }_{T}:$ Average of intensity values of the pixel of the template image

w: Width of the template image

h: Height of the template image

When the ZNCC value at (x, y) is higher than a threshold value that is to be manually adjusted for each image (0.60–0.88 was used in this work), the location is recognized as one having the local structure [Fig. 1(b)]. Note that we may obtain local structures overlapping each other if we just pick up all of the locations according to the above condition only). To address this problem, for the recognition process, we scan a source image in the raster order and detect locations one by one sequentially. We then simply reject locations which are covered by a local structure that is already detected while scanning. This recognition was used in image processing and is called "template matching." ^35–38) After the recognition, the position correlation is calculated [Fig. 1(c)] and the correlations are plotted on a 2D map. In Fig. 1(d), the structure at the center indicates the reference point whose correlations are to be calculated for the other structures. The plots in Fig. 1(e) show the correlations obtained. Note that the self-correlation, to be plotted at the origin, is not counted there. This calculation is performed for each local structure and plots are summed up on the same 2D map. In the map, the number of plots is recorded for each grid point, which is indicated on a color scale in Fig. 1(e).

In this study, we applied the process using the template-matching method for SPM images. This template-matching method can be used to recognize structures in SPM images, even when only one image is used. Therefore, the template-matching method can be applied to individual SPM images with different structures. In addition, we do not have to prepare a large number of images for each structure. Furthermore, since we require only a single scan over image for computing ZNCC values in the recognition task, our method is computationally efficient compared to deep-learning methods.

The positional correlations are collected for each of the recognized structures. In order to obtain the positional correlations for one structure, the distances and angles between the structures are employed. In this study, we employ Euclidean distance. The values of distance and angle obtained from each structure are summarized appropriately (i.e., crystal structure, etc.) and are then collected into a 2D grid map. For each grid point in the map, the number of plots is recorded. Using this method, we analyze the position correlation of the local structure in the SPM image.

To demonstrate the validity of the proposed methods using actual SPM images, we performed an experiment using an SPM image taken for a model sample containing a simple structure on p-Si(100) wafers. A p-Si(100) wafer (Yamanaka Hutech Co.) was washed with acetone, methanol, and ultrapure water under super-sonication for 5 min. After washing, the wafer was treated with O₂ plasma at 150 ml min⁻¹ and 100 W for 1 min in a gas plasma reactor (PR200, Yamato Science Co., Ltd.). An electron beam resist (ZEP520A, ZEON Co.) was spin-coated onto the wafer at 500 rpm for 10 sec and 5000 rpm for 70 sec. The wafer was heat-treated at 453 K for 3 min in a heater. After the treatments, an electron beam was placed on the wafer using an ELS-G100 (ELIONIX Inc.) to write the required structure on the surface.

The proposed analysis technique was applied to the experimentally obtained SPM images. The SPM images were obtained using Dimension XR Icon (Bruker) with a silicon cantilever (ScanAsystAir, Bruker). The measurements were performed in the peakforce tapping mode (2 kHz) in air and at room temperature.

Figure 2(a) shows an SPM image obtained using a prepared model structure. The structure had circular [indicated by the blue arrow in Fig. 2(a)] and shuriken-like [triangular ninja star-like, indicated by the green arrow in Fig. 2(a)] structures. A template of the circular structure [Fig. 2(b)] was extracted from the image and template matching was applied to recognize the structures similar to the template structure. Figure 2(c) shows the results of the recognition. The positions of the recognized circular structures in Fig. 2(c) are indicated by red squares. Although the SPM image is influenced by noise and errors, we can see that all of the circle structures were successfully recognized without any overlap. The results indicate that the template-matching method can be used to recognize local structures in experimentally obtained SPM images.

Zoom In Zoom Out Reset image size

Another experiment was done to examine whether this method could be used to identify the chiral structures. Because chiral isomers cannot be distinguished using conventional vibrational and absorption spectroscopy, it is very useful to distinguish them from SPM images. ^39,40) We prepared a local structure with triangle shuriken-like structure on a Si wafer with a chiral structure [Fig. 3(a)]. This surface contained two types of chiral structures [Figs. 3(b) and 3(c)]. Template matching was applied to recognize the structure [Fig. 3(c)]. The recognition results are presented in Fig. 3(d) and the recognized structures are marked by red squares. Only the structures shown in Fig. 3(c) were recognized and the letter "IMS" could be observed. These results demonstrate that the chiral structures in SPM images can be distinguished using the template-matching method.

Zoom In Zoom Out Reset image size

In our experiments, the positional correlations among the structures have been calculated using the recognition result shown in Fig. 2(c). As shown in Fig. 2(c), the recognized structures (shown as red squares) were used as target points for calculating the positional correlation. First, we have computed positional correlations for a single reference point at the center of the SPM image [Fig. 4(a)]. Figure 4(b) shows the results of plotting the distances and angles between the reference points and target points.The positions of plots are summarized to a regular grid. In the case of Fig. 4(a), 5 × 5 grid has been used. Figure 4(c) shows the result after summarization (the plots are changed to balls). The number given to each plot indicates the number of plots fallen in there. In the case of Fig. 4(c), since we use a single reference point and there is no overlap among structures in the SPM image, no plots are mapped to the same grid point and all the numbers are 1. The results accurately reflect the positional correlation and relative distribution trends between the reference and the others. Next, positional correlations are computed for the nine circles around the center shown in Fig. 4(d). Figure 4(e) shows the results of plotting the distances and angles between the reference points and target points. Taking a closer look at the figure, we see that the plots (violet circles) are not perfectly aligned, but their positions are fluctuated to a certain degree. We see this kind of fluctuation due to several reasons including intrinsic physical properties and external influence caused by noises and errors. In order to address this problem, the positions of plots are summarized to a regular grid (7 × 7). Figure 4(f) shows the results of the summarization of the plots (the plots are changed to balls), which correctly shows the positonal correaltion between the reference point and target points. These results confirm that our proposed methods based on template matching are valid and can be used to analyze positional correlations in SPM images.

Zoom In Zoom Out Reset image size

Finally, we present an example of the analysis using the proposed methods for the data obtained by a more complex, nano-scale real-material system. We used an open data of the SPM image of Li@C₆₀/C₆₀ on Au(111) reported by Chandler et al. ⁴¹⁾ Figure 5(a) shows the SPM image (in the scanning tunneling microscopy mode with a voltage of sample (V_sample) of +2.5 V and tunneling current (I_tunnel) of 0.1 nA) of Li@C₆₀/C₆₀ on Au(111). On this surface with V_sample = +2.5 V, four types of structure were observed depending on the adsorption structure with or without Li. ⁴¹⁾ Among the four types, the circular structure caused by Li@C₆₀ was used as the template, and recognition via template matching was applied [Fig. 5(b)]. Next, the position correlations of Li@C₆₀ were calculated [Figs. 5(c) and 5(d)]. Figure 5(d) shows the zoomed image of the positional correlation analysis. The results were calculated from only one image shown in Fig. 5(a), and thus it is unclear whether the results reflected a general tendency. However, based on the results, high frequencies of Li@C₆₀ at neighboring sites of the reference point (represented by green and red rectangles) were observed. In addition, low frequencies of Li@C₆₀ were observed at sites diagonal to the reference point (represented by violet and blue rectangles). These results indicate that the Li@C₆₀ has a tendency to align with neighboring sites, and they do not align with diagonal sites. Thus, the analytical method can clarify the tendencies of the arrangements of molecules from SPM images. This result indicates that the proposed method is useful for investigating the tendency of arrangements from SPM image of the complex, nano-scale, and real-material system.

Zoom In Zoom Out Reset image size

In this study, we proposed a method for automatically calculating the positional correlation of local structures in SPM images, which is difficult to perform via manual analysis. First, structures were recognized by template matching. While scanning an image for template matching, detecting structures overlapping each other is avoided by rejecting locations that are covered by another structure which is already taken. As a result, the structures in the image will not be wrongly matched to more than one location. The relative positional correlation of the local structures was then calculated and presented by plotting the relative coordinates from the reference points to the other points and summarizing them on a regular grid. The proposed method was applied to actual SPM images, and the positional correlation of the local structures was accurately extracted.

We have shown an example of analysis for a case in which the orientation and scale of the template structure and those of the targeted structure in the source image are the same. The template-matching method can also be applied to another source image which reveals similar targeted structures that are aligned in an orientation and have a constant scale, provided that we adjust the template structure in respect of orientation and scale, according to the targeted structures. Because the proposed method can be used for feature analysis of not only SPM images but also other images, spectra, and experimental data. Additionally, the analysis time presented in this study is short, within the range of 1 to 2 s by a laptop computer (MacBookPro 2021; CPU: Apple M1Pro; RAM: 16 GB). It would be possible to utilize our method in a broad range of scientific research. Such an analysis of experimental data, together with computational science analysis such as first-principles calculations, will provide new insights into condensed matter science. We are currently working on these analyses and plan to publish our results in the near future.

Acknowledgments