2.1. Motion correction

To each MRC/TIFF fraction or EER file (Nakane et al., 2020) generated by Tomography 5, a motion-correction algorithm, based on MotionCor2 (Zheng et al., 2017), is applied to correct for sample movement during the exposure. Global drift correction using 3 × 3 patches and no dose weighting is performed. For EER data, an upsampling of 1 is used. Automatic grouping of frames into fractions and averaging of fractions are performed to reach 1 e⁻ per pixel per fraction.

2.2. Stack cleaner

The Tomo Live stack cleaner processing step is employed to remove tilt images that are obviously not usable for reconstruction, for example, when grid bars are coming into the field of view (Fig. 3d). The determination is based on the computation of a correlation matrix using a histogram of gray value correlation. Example correlation matrices are displayed in Figs. 3(b) and 3(c). The mean and standard deviation correlation values are calculated from the matrix. For each tilt image, a check is performed to determine whether the histogram is similar to other tilt images. A tilt image is considered to be usable if the correlation value is greater than a given threshold of mean + 1σ for more than four tilt images. If this criterion is not met, the tilt image is excluded from the tilt series. The algorithm has been tested with both lamellae and SPA samples with and without fiducials. The presence of high-contrast features such as fiducials does not influence the algorithm. Indeed, as long as fiducials are present on all tilt images, these images will not be removed since the histogram will be almost the same. In case of mistracking in some tilt images where a totally different area is imaged, the algorithm will not remove these images if the histogram is almost the same. However, this image will be removed later during the pair-alignment algorithm.

2.3. Pair alignment

Pair alignment aims to correct for large shifts occurring along the X and Y axes. It involves computation of the cross-correlation between tilt images `i' and `i + 1', with the subsequent identification of the cross-correlation peak (Fig. 3e). The stack is first reordered by tilt and rotated to correct for the tilt-axis angle. An automated filtering algorithm is applied to optimize the shape of the cross-correlation peak for each pair of tilt images. The shift is determined by measuring the distance between the cross-correlation peak and the center of the image. The pairing starts from the 0° tilt image, where it is always assumed that the zero-sample tilt image is of good enough quality to perform the cross-correlation analysis. Consequently, when no robust cross-correlation peak is found between two successive tilt images, the tilt image with a higher tilt is deemed to be non-usable and the correlation is repeated between tilt images `i' and 'i + 2'.

2.4. Tomogram positioning

The tomogram-positioning step is dedicated to the precise determination of the X and Y tilt corrections of the tomogram. Subsequently, the Y tilt correction is applied to the tilt angles, while the X tilt correction is used during the reconstruction step, and the resulting 3D reconstruction is horizontal in the XZ and YZ cross sections (Fig. 3g). The algorithm does not require a 3D reconstruction to be performed to determine those angles; only feature tracking is needed. In this case, the fast tracking of features is performed using a dense optical flow algorithm (not cross-correlation as in Section 2.5), where the movement of each pixel is determined. At the end of the tracking, pixel movement within a given patch is averaged. The 3D positions of individual patches are then determined as described in Section 2.6. These positions are then used to determine a plane equation, where the X and Y angles of the sample are extracted. The plane-fitting algorithm can determine outliers to improve the accuracy of the results (see Fig. 3f, in which outliers are colored red).

2.5. Patch tracking

To determine the alignment parameters that are needed to align each tilt image, fiducial-free tracking utilizing cross-correlation is employed. The initial step involves subdividing the 0° tilt image into 16 patches, each sized at 30% of the input image dimensions. We assume that the 0° tilt image is the horizontal plane of the sample since the tilt angle has been corrected for the sample orientation in the previous step. Subsequently, the patches corresponding to theoretical locations across the tilt series are tracked to generate trajectories. This is performed by cross-correlating neighboring tilt patches, i.e. i × i + 1, i + 1 × i + 2 etc. The Tomo Live patch-tracking process operates autonomously. This autonomy is achieved through an algorithm that relies on three parameters, which are described below.

(i) Automatic patch-movement analysis and replacement. Given that the sample orientation is established following the tomogram-positioning step and the correction of tilt angles, it becomes feasible to calculate the theoretical shifts of patches between two tilt images,

where x is the x coordinate of a patch, w is the image width and α_i is the tilt-angle value for tilt image i.

Once the shift between two patches has been determined (called the measured value), this value is considered abnormal if the difference between the measured and theoretical shift value is greater than 2% of the image size. In such instances, the affected patch is replaced to a new position on the image with contrasting features and a new trajectory is created.

To determine the new position, the tilt image is divided into 200 small subpatches. The contrast is then calculated for each of the subpatches, and they are sorted from most to least contrasted. Each patch occupies multiple subpatches, and when a patch is moved to a new position, that position is chosen from the sorted list, starting from the top. The sub­patches occupying the central part of the patch surface, equivalent to 2/3 of the whole patch surface, are labeled as `used'. If another patch needs to be moved, the next available unused subpatch is chosen.

(ii) Automatic determination of filtering parameters for each patch. Analogous to the pair-alignment step, an automatic bandpass filtering is used to optimize the shape of the cross-correlation peak for each patch. If no cross-correlation peak is found, the patch is repositioned to a new location on the image and a new trajectory is created.

(iii) Automatic removal of non-usable tilt images. During tracking, if a specified percentage of patches within a given tilt image necessitate relocation, that particular tilt image is deemed to be `non-usable' and removed. In Tomo Live, this threshold is set to 75%.

2.6. Determination of the alignment parameters

The trajectories obtained during the patch-tracking step are used to fit the equation that describes the projection images (Frank, 2006). The variables used in the minimization of the projection model are the rotation, overall scaling factor and image shifts, with the tilt angles of individual tilt images kept constant. The minimization procedure comprises three steps.

(i) Determination of the tilt-axis angle. Using the Brent method (Brent, 1973) to iterate on different values of the tilt-axis angle, minimization of the projection equation is performed with a small number of iterations to determine the initial value of the tilt-axis angle.

(ii) Determination of the initial alignment matrix without outliers. For each tilt image, the variables are iteratively determined until convergence is achieved. Convergence is performed on the average residual error between measured and theoretical fiducials. The rotation of each tilt image is initially set using the tilt-axis angle and is subsequently allowed to evolve independently.

(iii) Removing outlier markers. Once the first solution of the alignment matrix has been found, a second iteration of the equation is performed, initializing the alignment matrix with the outcome of step (ii). After each iteration, the mean error and standard deviation are calculated, and trajectories are removed if the residual error is greater than three standard deviations. This iterative process is repeated until convergence is reached.

Once the alignment parameters have been determined, the pair-alignment parameters (tilt-axis rotation and shifts) are included and the alignment is performed on the raw stack to minimize interpolation error.

2.7. CTF determination

Once all alignment steps have been performed and all non-usable tilt images have been identified, Tomo Live determines the CTF parameters from the raw stack where all non-usable tilt images, as identified in the previous steps, have been removed. This avoids having to determine the CTF parameters for tilt images that are not relevant.

In cryo-electron tomography, determining the CTF can be challenging for two main reasons. Firstly, during acquisition the sample is tilted to take images at different tilt angles. For angles that differ from zero, a defocus gradient is present on the sample image in the direction perpendicular to the tilt axis. Moreover, the total dose deposited on the sample is relatively low (up to a maximum of about 140–180 e⁻ Å⁻² for the series and 2–4 e⁻ Å⁻² for individual tilt images), resulting in low contrast on the image, making the Fourier transform noisy.

Tomo Live can automatically determine, without any user input, the CTF parameters for all tilt images of the tilt series. Firstly, since the sample orientation is determined during the tomogram-positioning step, the algorithm determines the CTF parameters (defocus and astigmatism) on the tilt image where the sample is horizontal to avoid any defocus gradient (X tilt is then neglected). For FIB-milled cases, the lamella pre-tilt angle is taken into consideration and the horizontal tilt image is equal to the tilt image with zero sample tilt. The classical 2D CTF fitting described in Rohou & Grigorieff (2015) is applied to determine the defocus and the astigmatism. For the other tilt images of the tilt series, where a defocus gradient is present, the astigmatism is considered to be constant along the tilt series. The defocus determination for these tilt images is based on the work of Xiong et al. (2009). The principle is to divide the images into rows of patches parallel to the tilt axis (Fig. 3h). For the same row of patches, the defocus value is assumed to be constant. These patches are averaged to determine the average power spectrum of a row. Then, for each row parallel to the tilt axis, the equiphase average using the astigmatism of the central tilt image is computed. Each computed 1D CTF curve is then rescaled along the frequency domain to have the CTF zeroes at the same frequencies as the CTF curve computed from the row of patches on the tilt axis. The background is modeled with a spline that passes through the zero crossings of the CTF curve and the high-frequency points (Fig. 3i).

For some data sets where the pixel size is small (<0.5 Å), the low-frequency sampling is low and there are not enough points to describe the CTF oscillations accurately. The image needs to be resampled to improve the signal. To avoid this manual intervention, a simple yet effective automatic algorithm has been developed. This algorithm consists of the following.

(i) Determining the number of points N_exp needed to describe the first oscillation of the CTF curve.

(ii) This value is then compared with a reference value N_ref taken for a pixel size of 1.5 nm.

(iii) The resampling value R (where R is rounded to the nearest half-integer) is determined as

2.8. 3D reconstruction

The process of 3D reconstruction relies on the Astra library (van Aarle et al., 2015), which offers a selection of reconstruction algorithms. As part of its intrinsic design philosophy, Tomo Live defaults to utilization of the simultaneous iterative reconstruction technique (SIRT) algorithm (Gilbert, 1972) due to its notable capacity for noise reduction. Use of SIRT typically results in reconstructions with sufficient contrast for quick quality assessment, which is the overarching objective of Tomo Live.

The tomogram thickness is determined automatically based on contrast. Firstly, the aligned stack is binned down to a 256 × 256 pixel image and reconstruction is performed. It is followed by the computation of contrast for each individual XY plane. The contrast function is then fitted to determine the tomogram thickness. This thickness value is then provided for the final reconstruction, which is performed on the 4× binned aligned stack.

2.9. Integration

The Tomo Live pipeline is integrated into the Data Management Platform (DMP) of the microscope (the offload server for images acquired by the detector). Acquisition files, such as MRC/TIF fractions or EER files, are registered in the platform by Tomography 5. This registration triggers the Tomo Live reconstruction pipeline. At the end of each tilt series, output results, including alignment parameters, aligned tilt and reconstruction stacks, are accessible from the web portal. A dedicated view of the portal utilizing multiple charts allows one to quickly judge the quality of the tomograms. Filtering and export functions will be available to select only relevant data sets for offline processing.

2.10. Evaluating data quality and export of Tomo Live data

The processing workflow implemented in Tomo Live has been optimized for rapid data handling, typically requiring less than 5 min following the acquisition of the last tilt image in a single tilt series. For example, for a typical use case consisting of 41 tilt images recorded at 4k × 4k pixel resolution, SIRT reconstruction (with GPU implementation) takes ∼3 min. Motion correction is performed on the fly as the tilt series is being acquired.

Data files and their associated metadata are then accessible through a web-based interface, facilitating data curation by the user. Metadata plots allow filtering of the data sets based on quality metrics, including Alignment Accuracy, Number of Deleted tilt images (called `slices' in the user interface), Tilt Axis Correction, Defocus, CTF Confidence Range and Astigmatism (Supplementary Fig. S1).

The web-browser interface of Tomo Live also allows inspection of the tilt series (motion-corrected and aligned stacks) and the reconstructed tomograms (Supplementary Fig. S2). Following tomogram selection, users can export the raw data as well as data that were generated by Tomo Live for further analysis. It is also possible to generate additional files and metadata needed to use Tomo Live results in IMOD or RELION 4.0 (Zivanov et al., 2022). The additional files generated are as follows.

(i) Alignment parameters in IMOD format: *.rawtlt, *.tlt, *.xf, *.tltxf, newst.com, tilt.com.

(ii) The initial CTF parameters in Ctfplotter format: *.defocus.

(iii) RELION-specific files: the order list per position (*.csv) and the *.star file.

Files in IMOD and Ctfplotter (Xiong et al., 2009) formats are directly compatible with RELION 4. In cases where the Tomo Live data being exported encompass tilt images identified as unusable (Deleted Slices), Tomo Live creates a new stack (*clean.mrc) during export, excluding the unusable tilt images. This new stack, as well as other generated files, are then used in IMOD or RELION. Importing the Tomo Live aligned stack to IMOD allows the first six steps (from Pre-processing to Tomogram Positioning) to be skipped as the Final Aligned Stack step can be directly used to generate a new aligned stack with a different binning and proceed with a new reconstruction.

3.1. Sample navigation

Search Maps are sets of tiles acquired using medium magnification, which allow the mapping of entire grid squares or lamellae. Multiple Search Maps, for example of all lamellae on the grid, can be queued and acquired automatically, similar to Automated Atlas Acquisition. Once acquired, Batch Positions can be added directly onto Search Maps without the need to acquire any additional Search Images. Sample features are accurately centered during automated acquisition using Search Images extracted from the Search Maps.

While magnification calibration is performed close to focus, imaging further from focus may lead to image rotation and shift. Automated calibration is available to calibrate for settings further from focus and to improve the alignment of individual tiles in Atlases and Search Maps (Fig. 4). This leads to more accurate navigation, which is especially crucial when acquiring multiple exposure areas during a single Batch Position.

Figure 4 Results of the automated calibration to improve the tile stitching. (a) Atlas without the calibration applied, with individual tiles not aligning well with respect to one another. (b) Atlas acquired with the calibration applied, showing a significantly improved tile alignment.

The automated calibration is fast and unattended and measures the difference between the requested stage and/or beam/image shifts and the actual shifts in four directions (±X, ±Y). The resulting four vectors are combined into a single transformation matrix that is used to correct the navigation on the tile sets. Due to this simple approach, the calibration is fast but has the risk of inaccuracies further away from the center of the tile set.

3.2. Multiple exposure areas

Utilizing optical image shift allows the collection of multiple tilt series in parallel, thereby saving time and gaining access to sample areas that would otherwise be used up for tracking (Eisenstein et al., 2023). This is possible in Tomography 5 by adding multiple exposure areas for a single Batch Position, where beam/image shift will be used to reach exposure areas away from the central exposure area (Fig. 5a). A template can be defined to use the same configuration of exposure areas in each Batch Position, as is already possible in the Thermo Scientific EPU software. Each exposure area is saved as an individual tilt series and will be processed automatically during the acquisition.

Figure 5 Multiple exposure positioning and definition of lamella orientation. (a) Yeast lamella illustrating multiple exposure positioning. 1.1 (central exposure area) and 1.2 denote exposure areas on the stage tilt axis (blue line), whereas 1.3 and 1.4 denote exposure areas positioned away from the tilt axis. (b) Defining the lamella orientation (green line) with respect to the stage tilt axis (blue line) to improve tracking.

Distances between the individual tilt images of the single exposure area change significantly during the tilt series, and the risk of missing the feature during tracking increases when the distance between the optical axis and the exposure area becomes longer. Therefore, the initial soft limits to position additional (off-axis) exposure areas were set to 6 µm on the tilt axis and 3 µm off the tilt axis from the stage position. It is still possible to add positions further from the stage position; however, above the soft limits the users are warned of the higher possibility of tracking errors, with the color of the exposure area changing to pink.

To demonstrate that good-quality tomograms can be collected off-axis, we acquired tilt series on a keyhole limpet hemocyanin (KLH) sample using 0, 3 and 6 µm image shift. Cryo-electron tomograms from this experiment have been deposited in the EMDB with accession codes EMD-19672 (stage position, 0 µm image shift), EMD-19673 (3 µm image shift) and EMD-19674 (6 µm image shift). A summary of the experimental setup and data-acquisition parameters can be found in the supporting information.

To accurately keep track of multiple exposure areas during the tilt series, the expected movement of each exposure area needs to be calculated using a theoretical movement path of the sample. This becomes more challenging with milled samples, as lamella orientation, pre-tilt and sample deformations play a role in the sample movement during the tilt series (Eisenstein et al., 2023). To more accurately model the shifts of each individual exposure area throughout the tilt series, the tilt angle and (if necessary) the lamella pre-tilt angle and the orientation of the lamella with respect to the tilt axis need to be specified before the tilt series (Fig. 5b). All three axes are used to model the shifts in 3D to predict the movement of each feature in three dimensions more accurately. To simplify the calculation, it is assumed that the sample is (relatively) flat and rigid.

In reality, most samples, and especially in situ cellular lamellae, show local variations in Z-height within the same lamella (Eisenstein et al., 2023). This can result in large errors in tracking, leading to unprocessable data. To overcome this, the Z-height is estimated during the tilt series, the model is updated continuously for each exposure area, and X and Y shifts per exposure area per tilt are calculated using the more accurate local Z-height. By using this approach, additional tracking errors coming from less rigid lamellae are also corrected.

4.1. Materials and methods

4.2. Tomo Live reconstructions

Six samples in total were chosen as representative examples of Tomo Live reconstructions: two SPA samples (apoferritin and KLH), two pleomorphic virus particles (TGEV and influenza), whole magnetotactic bacteria cells and milled yeast cells. Fig. 6 shows images from the reconstructed tomograms. Cryo-electron tomograms have been deposited in the EMDB with accession codes EMD-19666 (apoferritin), EMD-19667 (KLH), EMD-19668 (TGEV), EMD-19669 (influenza), EMD-19670 (magnetotactic bacterium) and EMD-19671 (yeast). Raw tilt-series data for KLH and yeast have also been deposited in EMPIAR with accession codes EMPIAR-11908 and EMPIAR-11907, respectively.

Figure 6 Examples of Tomo Live SIRT-reconstructed tomograms. Images from the tomograms of (a) apoferritin, (b) keyhole limpet hemocyanin, (c) TGEV virions, (d) influenza virions, (e) a magnetotactic bacterium and (f) yeast. Scale bars 100 nm.

4.3. CTF parameter comparison between Tomo Live and CTFFIND4

We compared the robustness of defocus estimation in Tomo Live versus CTFFIND4 (Rohou & Grigorieff, 2015; Fig. 7), which was originally developed for processing untilted SPA images but is nevertheless often used in the pre-processing of tomograms (Bharat & Scheres, 2016; Scaramuzza & Castaño-Díez, 2021). For those tilts where CTFFIND4 could estimate the defocus reliably (i.e. with an estimated resolution accuracy of better than 30 Å), similar defocus estimates were found by both algorithms, especially at lower tilt angles (≤30°), with median absolute differences in estimated defocus of ∼15 and ∼40 nm for tomograms of KLH and yeast lamellae, respectively (Figs. 7a–7d). At higher tilt angles (>30°), slightly larger differences were observed between the two algorithms (median absolute differences in estimated defocus of ∼30 and ∼75 nm for tomograms of KLH and yeast lamellae, respectively), and some large outliers (>1 µm difference) for tomograms of yeast lamellae (Figs. 7e–7h). While for Tomo Live the estimated defocus shows little deviation from the zero-tilt image for the higher tilt images in the same tilt series, which is what one would expect given that the applied defocus should have been constant throughout the recording of the whole tomogram, larger deviation outliers were observed for CTFFIND4 for tomograms of yeast lamellae. This suggests that Tomo Live estimates defocus for tilted images more accurately in these cases, while both methods perform similarly at low tilt angles. One possible explanation is that unlike Tomo Live, CTFFIND4 does not account for the difference in defocus across images from tilted specimens, which causes blurring of the Thon rings.

Figure 7 CTF estimation by Tomo Live and CTFFIND4. (a) Estimated defocus by Tomo Live minus estimated defocus by CTFFIND4 for purified KLH and yeast lamellae (left) for all data points and (right) zoomed-in. Each point denotes one defocus estimate in one tilt image. The color of the points depicts the tilt angle at which the image was recorded. Defocus estimates with estimated resolution accuracy >30 Å for CTFFIND4 were excluded. The same results as shown in (a) shown per tilt angle for (b) KLH and (c, d) for yeast lamellae: (c) all data points and (d) zoomed-in. (e, f) The difference in defocus estimates for higher tilt angles compared with the estimate for the zero-tilt image of the same tilt series for (e) Tomo Live and (f) CTFFIND4 for KLH. (g, h) The difference in defocus estimates for higher tilt angles compared with the estimate for the zero-tilt image of the same tilt series for (g) Tomo Live and (h) CTFFIND4 for yeast lamella. Each point denotes one defocus estimate in one tilt image. In all panels, the black line shows the median difference between the two estimates and the gray-shaded area shows the interquartile range.

For SPA applications on tilted data or (high-resolution) subtomogram averaging, such errors in CTF estimation would be corrected in subsequent CTF refinement. However, reliable CTF estimates are still important when corrected tomograms are immediately used in other applications, such as morphological studies.

The astigmatism estimated by Tomo Live was similar to that estimated by CTFFIND4, with a median absolute difference of 30 nm (Fig. 8). For tomograms purposefully collected with strong astigmatism (as shown by the power spectra), both algorithms correctly identified the high astigmatism.

Figure 8 (a) Estimated astigmatism by Tomo Live plotted against estimated astigmatism by CTFFIND4. Each data point corresponds to one tilt series. Some tomograms were purposefully collected with strong astigmatism, as shown in (b), (c) and (d), which show power spectra displayed in the Tomo Live web interface.