Sensitivity Testing of Stereophotoclinometry for the OSIRIS-REx Mission. I. The Accuracy and Errors of Digital Terrain Modeling

The starting point for all tests was creating a large set of simulated data. This was a substantial, iterative task, as the test data underwent improvements to capture lessons learned and investigate specific aspects of the effectiveness of SPC. By the end of the effort, the simulated data were mature, realistic, and stable, which allowed for a systematic and practical test of the mission's operational profile (Weirich 2022). All the tests started with a truth DTM and a truth flight trajectory that provided the position and pointing from which the synthetic images were rendered. A second test data set was created comprising synthetic images and a degraded flight trajectory that included errors in the spacecraft position and pointing knowledge. Using these two sets of data inputs, a testing DTM could be constructed and evaluated.

2.1.1. Truth DTMs

For these tests, truth DTMs provided an absolute reference for simulation and comparison. Three truth models that differed in resolution and complexity were created as the evaluation techniques improved, allowing higher-fidelity assessments.

For the global DTM tests, the models were based upon the radar-based DTM of Bennu shown in Figure 1 (Nolan et al. 2013a, 2013b). The radar model had a 1σ uncertainty of 20 m for small-scale topographic features (M. Nolan 2014, personal communication). A utility called CreatorP, created to support the SPC software, was used to change the surface's height, roughness, topographic features, and albedo from those of the radar DTM. This tool underwent several revisions throughout the testing process as different surface characteristics were desired.

Zoom In Zoom Out Reset image size

Shape 1 was an early truth model used for visualizations only. Shape 2 (Figure 1) was the first truth DTM used in these tests. To create shape 2, the radar DTM was deformed, and craters and boulders were added. The resulting DTM had a GSD of 75 cm and did not include albedo information. This DTM lacked the required resolution to generate images for the last part of the Approach mission phase and none of the Preliminary Survey mission phase (Lauretta et al. 2021). It was used for software unit-level testing and as a stepping stone for more expansive testing. This early testing provided lessons on generating synthetic DTMs and the requirements for their testing and evaluation.

Using the experience from shape 2, shape 3 was created with albedo features and rough terrain (Figure 1). The GSD of shape 3 was 5 cm globally, with three regions with finer-resolution synthetic terrain as locations for simulating the OSIRIS-REx Touch-And-Go (TAG) sample collection maneuver (Lauretta et al. 2017, 2022). The GSD in these regions was increased to 1 cm for a 100 × 100 m area at each of the three simulated TAG sites. TAG site 1 had a 25 × 25 m region with a 0.25 cm GSD.

Shape 3 was used extensively for much of the initial testing but had limitations that led us to generate a further improved model for additional testing beyond that required for software certification. First and most importantly, the topography at all scales was too rough to meet the requirements we were testing. There was no location (including the three simulated TAG sites) where the OSIRIS-REx spacecraft could safely perform TAG as defined by the mission requirements (at the time, with a 25 m safety radius). Second, shape 3 was created with versions at multiple resolutions; however, the data generation process was imperfect, such that there were minor differences in features between them. Thus, it was impossible to use data generated at multiple model resolutions in a single coherent test.

The final synthetic asteroid created to support testing for the OSIRIS-REx mission was shape 4 (Figure 1). This model was not used for the tests in this paper but was used in the follow-on tests described in Weirich (2022).

The truth images were created using the flight profile defined in the OSIRIS-REx Design Reference Mission (DRM) Revision C (Lauretta et al. 2017). The DRM defined several phases of mission operations; the Approach, Preliminary Survey, Detailed Survey (Baseball Diamond flybys), and Orbit B phases, in order of increasing detail and finer image resolution, were used in the testing.

The truth DTMs and computed trajectories based on the mission design were used as input for rendering the images (Table 2). For some tests, we altered the navigational error, types of images, number of images, latitude, photometric function, and other parameters. The key sensitivity tests of the F1–F3 testing suites are detailed in Table 3.

The Flight Dynamics group at NASA Goddard Space Flight Center (GSFC) used the software Freespace to render the images from the DTMs using the original McEwen photometric function (McEwen 1991). These images were generated using the GEOMOD geometry modeling tool and the Phillum physically based stochastic ray tracer, both developed at GSFC. Because the point spread for the OSIRIS-REx camera was less than 1, no point-spread function was added to the images. Additionally, no electronic noise or cosmic rays were added. This is a reasonable assumption because during actual flight operations, cosmic rays in images used in generation of the DTM were virtually absent. As such, this approach produced photorealistic synthetic images for which we knew the exact spacecraft pose, again based on the mission design.

As described in Palmer et al. (2022), SPC generates DTMs in a progressive manner, starting with low-resolution images and decreasing the DTM's GSD as the quality of the model and resolution of the images improve during mission progression. We created a starting DTM by using simulated images of the mission phases Approach and Preliminary Survey. The processing was stopped early, resulting in a modest but stable DTM with a GSD of 75 cm over the entire surface. This early model was used as the starting data for most of our tests.

We added the Detailed Survey images with a pixel size of approximately 5 cm. Then, we created a DTM with a 35 cm GSD and used the images to calculate the x-slopes, y-slopes, and albedo at every pixel. We did this over a 50 m square area in a simulated TAG site. We performed 5–50 iterations of the Align, Extract, and Solve (AES) procedure (Palmer et al. 2022), in which image data is extracted and converted into topography. Further high-quality maplets (small 99 × 99 pixel DTMs) were created as needed depending on the testing goals and results.

Because the Preliminary and Detailed Survey mission phase images had significantly higher resolution than the model GSD (15 and 5 cm, respectively), there was sufficient data that allowed the tests to improve the model.

Two types of evaluation of the SPC-derived models were undertaken: internal consistency checks and comparisons with truth models. We performed these evaluations at the global level and regionally. When we were testing the performance at the TAG site, which was 50 × 50 m, we evaluated the center 20 × 20 m as a representation of the entire region in order to expedite testing.

Internal consistency is a measure of how consistent the DTM is with the data used to create it—specifically, how closely every control point on the model and every image agree with one another. In SPC, control points are the centers of the maplets used to construct a DTM. Each control point can be described in two different ways: observed and calculated.

The observed position of a control point is defined as a pixel/line in an image. Every image will have the pixel/line position of each assigned maplet. This pixel/line position is assigned manually or automatically using normalized cross-correlation routines, and it is solely based on the image data and pattern matching.

The position of a control point is calculated by projecting its 3D position onto each image. Within SPC, each maplet has a single point that defines its center position in body-fixed space. Using the position and pointing of the spacecraft's camera, one can calculate at which pixel/line position the control point would intersect the image.

The internal residual error can be calculated as the delta between a maplet's calculated pixel/line position and its observed pixel/line for every image.

Using these residual values, as well as the agreement of maplets with one another, SPC provides statistics defining how well the DTM agrees with each piece of the assimilated source data. Our key metric is called the formal uncertainty, which is the rms elevation difference of the error between the calculated and observed position of all maplets, limb points, and neighboring maplets for every image, as described in Gaskell et al. (2023). The formal uncertainty used for this evaluation is calculated slightly differently than the metric "FormU" developed by Weirich (2022), which uses only the position of all maplets and limb points for every image.

Comparison with other DTMs is the evaluation type used most heavily in this study, but it is not used exclusively. There are three methods of comparison: rms, profiles, and normalized cross-correlation (which does not involve comparison with other DTMs).

One of the most frequently used comparative evaluation techniques for measuring the quality of DTMs is rms. This technique evaluates the difference between the two models. The distance for every vertex is measured between one DTM and the closest surface of another DTM. For our testing, we evaluated each vertex of the SPC-generated DTM and identified the smallest distance to any portion of any of the triangular plates on the synthetic truth DTM. In this study, rms comparisons follow a two-step process. First, a model DTM is translated and rotated to minimize the rms differences between it and the truth DTM. The DTM registration was done using the iterative closest-point method (Besl & McKay 1992), which involves finding the best rigid transformation (translation and rotation) that minimizes the mean-square distance between the data sets. Second, the sum of the squares of all of the differences is rooted to make the rms, which indicates the mean displacement between the two models. This metric is convenient because it provides a distance in physical units and is a standard statistical technique. By measuring the distance from a vertex to the nearest point on the triangular plate, the rms technique avoids issues with GSD size and sampling locations.

The second comparative evaluation technique compares vertical profiles between the truth DTM and the SPC-generated DTM. The height of the surface is plotted between two points that are equivalent on both DTMs. The truth DTM will have more detail in the profile, whereas the generated DTM will typically have a smoother profile. Additional tests were conducted focusing on the impact of the images and their observation geometries; these are described in a companion paper (Palmer et al. 2024a).

The third comparative evaluation technique is the normalized cross-correlation of the real and simulated images (Lewis 1995). While a visual comparison between truth and rendered images provides a strong indication of quality, normalized cross-correlation provides an objective standard of evaluation. This technique computes the correlation between truth images and images rendered from the DTM being tested. Normalized cross-correlation is used heavily in optical navigation for various missions (Mastrodemos et al. 2015; Olds et al. 2022; Adam et al. 2023). One of its most significant benefits is that actual images of the asteroid's surface acquired in flight are used as truth during operations. The more accurate the DTM, the higher the cross-correlation score will be, indicating a higher fidelity in the surface representation. As the normalized cross-correlation score approaches 1.0, the closer the model is to the truth, whereas a score of 0 indicates no correlation at all.

In contrast to the alignment step of the SPC algorithm, where a segment of an image is correlated with a single maplet, DTM evaluation is performed by cross-correlating an entire image frame with its equivalent computed from the multimaplet DTM. Correlation scores will thus be influenced partly by errors in the maplet positions but dominantly by the realism with which photoclinometry is able to reproduce small features within the maplets. The correlation scores are not sensitive to errors in the vertical scale of the maplets that may arise from the choice of photometric function, because such errors cancel out between the photoclinometry and image-rendering steps.

While using a single image for analysis is useful, this technique is most powerful for identifying high-quality DTMs when there is a wide variation in illumination geometries, as described in Palmer et al. (2024a). Multiple illumination geometries allow vertical or albedo errors to be more clearly identified as a lower correlation score.

During an observation campaign, discrepancies between the estimated navigation solution and actual spacecraft positions will influence DTM generation. The first three tests in Table 3 assessed the impact of these errors.

To best describe the different concepts and parameters used in the tests, a few definitions are necessary.

• Planned flight path. What the mission requested the spacecraft to fly. The imaging plan is based on this planned flight path.
• Actual flight path. What the spacecraft actually flew, which can never be fully known. Due to real-world unknowns (e.g., the exact amount of fuel that was used during an orbital maneuver), there will be an offset between the planned flight path and the actual flight path.
• Flight error. The difference between the planned flight path and the actual flight path. This does not impact DTM generation and is ignored in this discussion.
• Reconstructed flight path. The navigation team's postflight estimate of the flight path is based on images, Doppler, and telemetry.
• Knowledge error. The difference between the navigation team's reconstructed flight path and the actual flight path. This error cannot be known precisely because the actual fight path is unknown, but the navigation team can give statistical estimates.
• Navigation uncertainty. The statistical estimate of the knowledge error based on the quality of the data inputs when building the reconstructed flight path. SPC uses this estimate to provide a weighting value for the reconstructed flight path.

The actual flight path, the flight error, and the knowledge error cannot be known during a real mission. However, with a synthetic data set, these parameters are explicitly created, so they are known. This makes it possible to test the quality of DTM generation based on parameters that are important but unavailable during flight.

The following tests place boundaries on how knowledge errors and navigational uncertainties influence an SPC-derived product. We evaluated the quality of DTMs by exploring cases spanning extremes in spacecraft position knowledge—from no error to a large (1 km) error—when approaching and orbiting a small body like Bennu.

3.1.1. No Error in Spacecraft Knowledge (Test F1)

This first test assumed a perfect correspondence between the planned, actual, and reconstructed flight paths. These no-error conditions provided a performance baseline to which other tests could be compared. This test also provided the best performance metric to assess how much better SPC performed over mission requirements.

One of the authors (R.W.G.) performed this test blind, using only the simulated imagery with no access to the truth DTM during the test. Another author (E.E.P.) separately performed this test, not in the blind, and analyzed both models during the testing. In this way, we could assess performance in actual operations under the test assumptions and explore methods of improving future performance. The test lasted for 6 weeks.

The F1 test used images generated from the shape 3 truth model and the planned position of the spacecraft throughout proximity operations. The first test the DTM generated was a 75 cm GSD model based on the simulated data from the Approach and Preliminary Survey mission phases (Figure 2). The performance of the DTM met the accuracy required by the mission (Table 4) and indicated that the imaging plan for the Approach and Preliminary Survey mission phases was sufficient. The model had an absolute accuracy (rms) of 56.5 cm, which met the requirement by a large margin.

Zoom In Zoom Out Reset image size

Table 4. Mission Requirements and Results for the No-error Test

Model	DTM GSD (cm)	Required Accuracy (cm)	Test Accuracy (cm)	Formal Uncertainty a (cm)
Preliminary Survey	75 b	100	56.5	49
Detailed Survey	35 b	75	49.2	45
TAG operations	5 b	14	4.5	5

Notes.

^a FormU was approximated by the average residual error of all data calculated by SPC's internal consistency tool RESIDUALS. ^b These requirements were defined in the OSIRIS-REx Mission Requirement Document that provided the details of each mission requirement and the justification.

Download table as:  ASCIITypeset image

The second DTM generated in this test was a 35 cm GSD DTM based on data collected through the simulated Detailed Survey mission phase. This DTM also met the accuracy requirements (75 cm in this case), which was needed to support navigation (Table 4). However, the test demonstrated that the existing plan for the Detailed Survey had a minimal number of images and observing stations. Although the Baseball Diamond flybys of the Detailed Survey produced images with a pixel scale of 5 cm, the DTM's accuracy was around 49 cm. Previous work with SPC suggested that accuracy should be near the pixel size of the imagery. The 49 cm accuracy first indicated that the image data set was inadequate. An evaluation of the image coverage indicated that the model was predominantly a result of Preliminary Survey images rather than Detailed Survey images. More imaging at the Detailed Survey distance was required. The two observing stations used in the DRM for the Detailed Survey provided only two unique observing geometries for the equator and only one observing geometry for the higher latitudes. Further, even the two equatorial images were too limited in stereo angle to constrain the shape as effectively as desired.

Due to the inadequate performance of the existing observing plan, it was necessary to define a better imaging data set, which drove much of the testing and analysis reported in Palmer et al. (2024a). This effort elucidated the requirements for an effective imaging data set for SPC, eventually leading to additional imaging requirements and an updated mission plan (Lauretta et al. 2017).

The third and final DTM generated for this test was a region of 50 × 50 m around a simulated TAG site. The SPC-generated DTM indicated that the imaging plan for Orbit B was insufficient. Observing the TAG site from the near-terminator orbit resulted in a phase angle that was almost always greater than 90°, which meant that either the incidence or emission angle (or both) had to be significantly compromised with respect to the ideal. These compromises in image geometries produced only a marginal-quality DTM at a 5 cm GSD.

The test met the operational mission requirements regarding accuracy (Table 4). However, because accuracy cannot be measured in flight, FormU and other residual error measurements were computed and compared with accuracy to evaluate their utility as proxies for it that can be used in flight. One pretest expectation was for FormU to be half of the desired accuracy; however, with this data set, that was not accomplished. Further work has indicated that FormU can be an acceptable proxy for accuracy, but only to about a factor of 2 (Weirich 2022), consistent with these results. Additionally, experience suggests that the ruggedness of the terrain may have an impact on FormU, but this hypothesis has not been systematically tested.

3.1.2. Nominal Flight Error in Spacecraft Knowledge (F3A)

3.1.3. Knowledge Error and Navigation Uncertainty (F3G)

Here we evaluate whether different processing of the same data impacts the accuracy of a DTM. SPC has a variety of options that can be turned on or off to improve processing. Some sets of data have difficult or unique situations where the default parameters are not as effective as alternate ones. Thus, we evaluate some of the significant options to identify which ones work best for the expected mission profile.

3.2.1. Limited Evaluation of Photometric Function Selection (F3F)

This test was conducted to evaluate the impact of photometric functions on the DTM. Over the years, various photometric functions have been tried, with the best-performing model being the modified lunar photometric model that combines Lambertian scattering with a Lommel–Seeliger model for SPC. There are two specifications of this model, McEwen (1991) and McEwen (1996), with the former specification having a factor of 2 in one of its terms, which is the favored formula (Equations (1) and (2)). The McEwen (1996) function has been used the most during applications of SPC prior to 2016. However, for the mission, the McEwen (1991) model that included the factor of 2 was used, so it was necessary to systematically evaluate the performance of each function. u_o is the cosine of the incidence angle, u is the cosine of the emission angle, and α is the phase angle:

$$\begin{eqnarray}&&I(u,{u}_{o},\alpha )=(1-L(\alpha )){u}_{o}+\displaystyle \frac{2L(\alpha ){u}_{o}}{u+{u}_{o}},\end{eqnarray} \tag{ 1 }$$

$$\begin{eqnarray}&&L(\alpha )={e}^{\left(\tfrac{-\alpha }{60}\right)}.\end{eqnarray} \tag{ 2 }$$

Two 5 cm GSD DTMs were produced using shape 3 using McEwen (1991) and (1996). The imaging conditions followed the OSIRIS-REx prelaunch plan of two stations for each hemisphere with an approximately 45° emission angle.

In each case, a high-quality DTM was produced, and two metrics were used to measure their quality. The first measured the rms of the deviation between the truth DTM and the SPC-generated testing DTM. The unfavored version, which did not have the factor of 2, had an rms of 5.45 cm, while the favored version resulted in an rms of 5.72 cm. While there is a variation of 0.27 cm, this is well below the scale of the imagery (the image pixel size is a factor of 20 larger), and the deviation should be considered noise.

The other metric used normalized cross-correlation to assess how well the model matched nine images that covered a wide range of emission, incidence, and azimuth angles. The unfavored version had a score of 0.7872, and the favored version had a score of 0.7884, both showing strong agreement with the images. These results imply that the scalar factor correction does not represent a significant improvement in the DTM in the SPC context. So, although the two photometric functions generated a similar result, the key takeaway is the speed at which each formula converged. The favored version converged faster than the version missing the scalar factor of 2.

An obvious issue with this test is that the photometric function used by Freespace to generate the truth images was that of McEwen (1991). As such, the images were rendered with the same function that SPC removed. We believe that this does not invalidate any of the results of the SPC testing because these results match previous experience in generating DTMs on other objects. More specifically, as part of the overall testing to support NFT (Olds 2014; Olds et al. 2022), a simulated physical asteroid surface (3 × 3 m) was constructed, imaged under specific observational parameters, and measured using lidar (Craft et al. 2020). An SPC model of the surface generated using the images was compared to the lidar-derived surface. The results showed that even for a surface without regolith, imaged through an atmosphere, with an unknown photometric function, SPC's standard photometric function (McEwen 1991) performed exceptionally well in a real-world test.

3.2.2. Slope-to-height Parameters (F2-Test 6)

Another seminal test evaluated the quality of the terrain as a function of latitude (or maximum subsolar point). For this test, images were generated so that the terrain was illuminated as if it were at different latitudes, every 10° from 0° down to −70°. To do this, we adjusted the position of the DTM so it was at its respective latitude and set its normal vector to match the truth DTM at that latitude. This test used images derived from shape 3 of simulated TAG site 1, whose original latitude was at −8°. These images were based upon the prelaunch imaging plan for the Detailed Survey Baseball Diamond flybys with four stations, two over each hemisphere.

Eight different scenarios were tested where the Sun vector was varied, and all images were regenerated based on that vector. The same number of images and the same relative spacecraft observing conditions were used for each test; only the effective latitude was changed, so that for effective latitudes closer to the poles, the images had longer shadows, and the terrain was more prominent than the albedo (Figure 3). The input imagery came from the expanded DRM, as depicted in Figure 2.

Zoom In Zoom Out Reset image size

The resulting DTM quality was evaluated using both rms and cross-correlation. The rms between DTMs generated at these different illuminated latitudes were similar, averaging 6.2 cm, and demonstrated no trends with latitude (Table 7). Taken at face value, this result indicates that the latitude of the topography has little impact on the rms of the DTM. The variations in the rms are on the order of 1 cm, while the source imagery pixel scale is 5 cm, so those variations are not significant (i.e., in the noise). This finding would lead one to consider solar geometries of minor importance as long as reasonable stereo angles have been met.

But a visual inspection of the DTMs shows that the terrain was less well represented at higher effective latitudes (Figure 4). The south-facing portions of sloping terrain lacked details, crater bottoms were shallower and without features, large areas were without features (smooth), and the vertical relief of many structures was lower. This is a direct consequence of the increase in the number and extent of shadows at higher latitudes that reduced the amount of information available for terrain building.

Zoom In Zoom Out Reset image size

As part of this test, we used visual inspections of the DTMs that were generated at each latitude. As the latitudes increased, the quality visibly decreased. Using the normalized cross-correlation technique, one can see that as the latitude increases, the correlation between images and the terrain decreases (Table 7). The cross-correlation score for the equator to midlatitudes was high at 0.80 but decreased to 0.76 at −50° and became poor at −70° latitude, consistent with the aforementioned visual inspection. These results support a correlation score of 0.6 as a useful criterion to define a good versus poor rendering.

While rms is an effective tool for describing the agreement between two DTMs, as shown by this test, it struggles when the surface roughness is low compared to the pixel size. For products created by SPC, rms is capable of providing accurate results, typically down to the level of the image's pixel size, in this case approximately 5 cm. This is because a maplet's height error is a function of stereo's ability to identify the correct location in 3D space, which reflects how well images can be coregistered. Coregistering of images can typically be done to a pixel level, or a subpixel level if the terrain is smooth. The pixel size of the image becomes a limit of accuracy for rms; i.e., for SPC products, the rms will seldom be better than the image pixel size.

In this test, the surface roughness of the truth DTM was relatively smooth. The pixel size of the imagery was 5 cm, and the surface roughness was measured at length scales between 15 and 300 cm and was consistently 0.6 cm. As such, much of the topographic variation was smaller than the regional error. Small-scale features are lost to rms because those variations are dwarfed by the stereo error.

In the high-latitude tests, a significant amount of terrain is in shadow, providing no topographic information, and thus is assumed to be flat. In this scenario, the lack of terrain expression in the images does not have an impact on rms, making rms a less effective tool.

We find, therefore, that although rms can be useful in many instances, it should not always be applied alone. The use of additional assessments like visual inspection and the correlation score provide a measure of how closely an SPC DTM (which has both terrain and albedo) matches the original images.

One of the most important findings from these tests was establishing the typical accuracy (i.e., average error) of SPC. We determined that the average error of an SPC model is close in magnitude to the smallest pixel size of the source images for a reasonable data set. The suite of tests allowed numerous comparisons between truth DTMs and SPC-generated testing DTMs constructed with drastically different source images (Table 8). The table describes which mission phase it simulated and thus the GSD of the DTM, how extensive the coverage of the images was, and a subjective estimation of how robust the observation stations were. It also then denotes the accuracy (rms) of the global DTM and compares it to the pixel size (resolution) of the source imagery. If the ratio is above 1, then the DTM's accuracy was worse than the source imagery, whereas a ratio of 1 or lower supports the conclusion that, with a reasonable image set, the accuracy is close to the pixel size.

During the first several tests conducted to validate the SPC software, we discovered that the mission's imaging plan would not provide a sufficient variety of observation geometries. This resulted in the testing DTMs having a lower fidelity relative to the image pixel size than would be expected based on previous missions, such as Dawn's encounter with Vesta (Gaskell 2012) and Rosetta's encounter with 67P Churyumov–Gerasimenko (Jorda et al. 2016). Palmer et al. (2024a) provides more details about which observing stations resulted in the highest DTM accuracy. As the tests progressed, better constraints were developed for the observing conditions, which allowed testing with better data sets. In turn, this allowed measurement of the quality of SPC under both ideal and nonideal conditions.

Beyond the sensitivity testing, mission simulation testing determined the performance of SPC as it relates to imaging conditions (Table 8). If there is limited imagery for a region of terrain, then the rms is 3–6 times the source images' pixel size.

If there is a large suite of observing conditions (a condition typically met for most small-body missions that orbit the object), the rms error is 1–2 times the source images' pixel size. This was indicated during the testing; if there were sufficient images available (three perfect observing conditions or five to six more moderate observing conditions), then the SPC DTM would have an average rms on the order of the size of the image pixels for that terrain (Table 8). This quality is sufficient for many scientific tasks within planetary science and is on par with the results of stereophotogrammetry.

While most planetary science missions have no ability to determine ground truth, this testing with truth models showed that SPC's performance is very good, and the testing results can be extrapolated to provide the expected DTM performance on a mission.

While rms is a common technique for evaluating DTMs, our results show that it only provides part of the story. For most surfaces on a global scale, rms is an effective measurement of the quality of a DTM, but it is limited when it comes to high-resolution DTMs. As seen in the latitude dependence test (Section 3.3), the actual quality of an SPC model is not always reflected in a simple subtraction between the testing DTM and truth DTM even after using rms to translate and rotate the model to minimize the difference between the testing and model DTM. When the sampling of the model approaches the pixel size of the source images, the rms following this minimization approach does not show the improvement that should occur in any of the following scenarios: adding more images, adding higher-quality images, or performing more processing (Palmer et al. 2024a).

Indeed, we demonstrate that cross-correlation between truth images and rendered DTM images provides a clear and consistent evaluation of the quality of the model. Because it is not possible to have a truth model for an actual flight mission, images of the actual surface can be considered ground truth. Using cross-correlation between images and rendered DTM images allows a direct comparison during a mission. Additionally, although factors such as uncorrected lens distortion will impact the model, cross-correlation using multiple images will uncover these errors as a disagreement between the DTM and the images. This dramatically improves our ability to identify uncorrected issues in the source data and is discussed at length in the companion paper (Palmer et al. 2024a).

Our work suggests that most sets of images can be processed to reach a score above 0.6, but that a score of 0.7 is a much better goal. A score of 0.7 reflects a reasonably accurate model that can render images that are clearly representative of the source images. A score of 0.8 indicates an exceptional model representing the features very accurately. Scores above 0.9 are rare.

Finally, optical navigation uses cross-correlation to detect navigational features. Optical navigation requires a DTM to be generated beforehand so they can be matched in navigational imagery. This matching technique is completely dependent on cross-correlation. Therefore, while rms provides an exact measurement of accuracy, correlation scores provide an essential evaluation of the quality of the DTM, particularly in regard to optical navigation.

Though this testing demonstrates what SPC can do well, it also has made clear one of SPC's largest limitations, namely, that it struggles with very steep slopes, such as the sides of boulders. Steep-sided boulders are frequently underrepresented by about half to one-third of their correct height. This is due to the inability to fully orthorectify every image, specifically removing the topography, without a correct DTM. As such, features on the boulder's slopes are not projected into the same x/y position of every orthorectified image.

This error can be seen using vertical profiles (Figure 5). Using a truth model, actual height deviation can be measured in profile view. The profile shows that the boulder only reaches a portion of its correct height in the SPC-generated DTM, but the surrounding terrain is well represented. If the slope is lower, then the boulder can be fully represented. Underrepresentation of boulder heights occurs when the boulder is steep. For imagery with a 5 cm pixel scale, one would typically build 5 cm GSD topography, and boulders smaller than the entire width of the maplet, 4.95 m, would not be well represented. Larger boulders will be better represented because there are more maplets contributing to their shape; thus, more stereo control points are calculated in 3D space rather than just the slope from photoclinometry.

Zoom In Zoom Out Reset image size

Although a variety of techniques have been attempted to fix this error, none of them can remove it, because none of them are able to get a draft DTM that allows the images to be fully orthorectified. Typically, the height of the SPC-generated boulder is about 50% of the truth's height using the vertical profile technique. However, focused effort can bring the height discrepancy down to only 30%. This is done by placing one or more maplets specifically on the top of the boulder. This allows for the 3D position of the boulder's peak to be generated by an independent stereo solution. The maplet data for the top of the boulder are resampled along with the rest of the maplets, yielding a combined solution. More testing is needed to identify the critical slope at which SPC struggles, but that is beyond the scope of our work.

For flight missions, problems with an SPC-generated DTM can be detected in four ways. First, for large features, boulders can be visible on the limb of the asteroid (Figure 6). These composite views (red is the source image, and cyan is rendered from the SPC model) are frequently useful as a tool for quickly assessing the fidelity of the model when working on a global model, which by necessity requires a low-resolution image to capture the limb of the target. The second approach is to evaluate how well the DTM can render images of the surface that match the source images; this is the method quantified by cross-correlation. The third way is to measure shadows; underrepresented boulder heights will create shadows shorter than they should be. The final way is to compare the SPC-generated DTM with DTMs created using other data or techniques, such as lidar.

Zoom In Zoom Out Reset image size

One of the benefits of SPC over traditional stereo topography generation is a consistent and scientifically usable albedo channel. Traditional stereo will generate a height map, but there will be no information about the surface's albedo. On the other hand, SPC will solve for an albedo solution over the entire surface. Because it uses multiple images and a photometric function to correct for phase, emission, and incidence angle at every pixel, the albedo channel of the DTM provides a highly accurate representation of the relative albedo of the surface. This albedo product has been used both for navigation (Olds et al. 2022; Adam et al. 2023) and as input for scientific analysis, e.g., for quantitative photometry (Domingue et al. 2018) and thermal modeling (Palmer & Sykes 2014).

Because cross-correlation was the key criterion for the OSIRIS-REx mission, detailed analysis of the albedo was not conducted. The companion paper, Palmer et al. (2024a), provides more analysis of cross-correlation, which measures the quality of a DTM that contains both topography and albedo. However, for quantitative and specific albedo evaluation, new tools and methodologies are required and are outside of scope of this study.

Another finding is that SPC is scalable over the range of planetary science data. These tests demonstrate that SPC can utilize images at any resolution to make a corresponding topography resolution. Terrain from 6 m GSD down to 0.35 cm GSD was generated in our testing, a difference of 3 orders of magnitude. Figure 7 shows a sample of the results for the highest-resolution topography down to 0.35 cm.

Zoom In Zoom Out Reset image size

SPC has been used for large objects such as Vesta (Gaskell 2012) and Ceres (Park et al. 2019). Considering that global topography was modeled for Vesta and Ceres at kilometer scales, SPC has demonstrated efficacy over an additional 3 orders of magnitude. In total, 6 orders of magnitude in topographic scale have been demonstrated by SPC, from 1 km down to the 0.5 cm terrain used for actual navigation at Bennu (Olds et al. 2022).

From this suite of testing, we can provide some insight as to how SPC handles error. The fusion of stereo and photoclinometry makes it difficult to separate how error propagates because the final result is a single DTM that reflects the benefits of each technique. However, there are a few conclusions that can be drawn.

FormU offers a good indication of the effectiveness of the stereo component. To summarize, FormU averages the error between the maplet's center vector in 3D space and its pixel/line location of the image projected into 3D space (Weirich 2022). FormU provides an estimate of how well the spacecraft position/pointing has been corrected, indicating that the overall structure of the model's inputs are stable. Even though SPC updates the initial navigation solution, not all errors are removed. These errors can be seen in FormU. The better the data (as defined by range of observing conditions), the better SPC can reconstruct the flight trajectory and thus compute a stereo solution for the center vectors of the maplets.

Cross-correlation shows errors in the DTM that can be from either albedo or slopes. The albedo channel is the one most directly related to photoclinometry. We have noted that low-incidence-angle images frequently have some of the lowest cross-correlation scores because the digital number of these images is predominately a function of albedo. However, once the incidence angle is larger than 20°, the topographic signature from the slopes is stronger than the effect of albedo. As such, it is possible that errors in the photometric function could be detectable by the cross-correlation score of low-incidence-angle images.

On the other hand, the slopes, and thus the topography, come from a combination of the pixel-by-pixel calculation of photoclinometry and conditioned with stereo. During the nominal processing of SPC topography, a random 1% of a maplet's pixels will be constrained by stereo calculations. Overall, this processing works exceptionally well without significant error. With a good data set, we can attain cross-correlation scores of over 0.9, which is a very good representation of the surface.

However, SPC's ability to solve for the albedo and slope at each pixel breaks down with steep slopes because the largest source of error within SPC is due to orthorectification of the images to remove the topography. Orthorectification requires a DTM in order to remove the projection effects within an image; however, during DTM generation, the available DTM will be immature. As such, not all topographic features can be removed from the images. Thus, images taken from different viewing geometries have features rendered at different maplet x/y positions. Then, when photoclinometry is conducted, there is not a consistent solution for those x/y positions because they correspond to different features. SPC's difficulty in generating boulders is the clearest example of this. While additional processing of the DTM helps remove some of these effects, there are regions in which the images never have the topography fully removed, resulting in error shown by lower cross-correlation scores.