Skip to main content
SpringerLink
Log in

Graph-based structural joint pose estimation in non-line-of-sight conditions

  • Special Section: Computer Vision Empowering Earthquake Engineering and Engineering Vibration
  • Published:
Earthquake Engineering and Engineering Vibration Aims and scope Submit manuscript

An Erratum to this article was published on 02 August 2023

This article has been updated

Abstract

In post-earthquake surveys, it is difficult (and often infeasible) to observe and quantify displacements beyond line-of-sight (LOS), given seismic force-resisting and gravity systems exist completely or partially within a building’s enclosure. To overcome this limitation, we develop a novel framework that generalizes graph-based state estimation towards structural joint localization via engineered landmarks. These landmarks provide an indirect means to estimate residual displacements where direct LOS is unavailable. Within our framework, engineered landmarks define topologies of uniquely identifiable landmarks that are either visible or non-visible to a robot performing simultaneous localization and mapping (SLAM). Within the SLAM approach, factors encoding robot odometry and robot-to-visible landmark measurements are formulated for the cases of wireless sensing and fiducial object detection and tracking. Visible landmarks are rigidly attached to non-visible landmark subsets for each engineered landmark, where the complete set of non-visible landmarks form globally rigid and localizable connectivity graphs via range-based factors. Complimentary subsets of non-visible landmarks are embedded within the base structure and uniquely define joint pose via geometric factors. All factors are unified within a common graph to solve for the maximum a posteriori estimate of robot, landmark, and joint states via nonlinear least squares optimization. To demonstrate the applicability of our approach, we apply the Monte Carlo method over a parameterization of system noise to calculate residual joint pose error distributions, maximum average inter-story drift ratios, and related summary statistics for a 19-story nonlinear structural model. By performing nonlinear time history analyses over sets of service-level and maximum considered earthquakes, our parametric study gives insight into our method’s application towards post-earthquake building evaluation in non-LOS conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Change history

References

  • Aiello GR and Rogerson GD (2003), “Ultra-Wideband Wireless Systems,” IEEE Microwave Magazine, 4(2): 36–47.

    Article  Google Scholar 

  • Applied Technology Council (2005), Field Manual: Postearthquake Safety Evaluation of Buildings, 2nd ed., Applied Technology Council, Redwood City, CA, USA.

    Google Scholar 

  • ASCE/SEI 43–05 (2005), Seismic Design Criteria for Structures, Systems, and Components in Nuclear Facilities, American Society of Civil Engineers, Reston, VA, USA.

    Google Scholar 

  • ASCE/SEI 7–16 (2016), Minimum Design Loads and Associated Criteria for Buildings and Other Structures, American Society of Civil Engineers, Reston, VA, USA.

    Google Scholar 

  • Cao Y and Beltrame G (2021), “VIR-SLAM: Visual, Inertial, and Ranging Slam for Single and Multi-Robot Systems,” Autonomous Robots, 45: 905–917.

    Article  Google Scholar 

  • Çelebi M (2005), “Structural Monitoring Arrays—Past, Present and Future,” in P Gűlkan and JG Anderson, Editors, Directions in Strong Motion Instrumentation, 157–179.

  • Chepuri SP, Leus G and van der Veen A-J (2014), “Rigid Body Localization Using Sensor Networks,” IEEE Transactions on Signal Processing, 62(18): 4911–4924.

    Article  Google Scholar 

  • Dellaert F and Kaess M (2017), “Factor Graphs for Robot Perception,” Foundations and Trends® in Robotics, 6(1–2): 1–139.

    Google Scholar 

  • Eurocode 8 (2004), Design of Structures for Earthquake Resistance: Part 1: General Rules, Seismic Actions and Rules for Buildings, European Committee for Standardization.

  • Forster C, Carlone L, Dellaert F and Scaramuzza D (2016), “On-Manifold Preintegration for Real-Time Visual-Inertial Odometry,” IEEE Transactions on Robotics, 33(1): 1–21.

    Article  Google Scholar 

  • Guan X, Burton H and Shokrabadi M (2021), “A Database of Seismic Designs, Nonlinear Models, and Seismic Responses for Steel Moment-Resisting Frame Buildings,” Earthquake Spectra, 37(2): 1199–1222.

    Article  Google Scholar 

  • Helmberger M, Morin K, Berner B, Kumar N, Cioffi G and Scaramuzza D (2022), “The Hilti Slam Challenge Dataset,” IEEE Robotics and Automation Letters, 7(3): 7518–7525.

    Article  Google Scholar 

  • Hsu TY and Kuo XJ (2020), “A Stand-Alone Smart Camera System for Online Post-Earthquake Building Safety Assessment,” Sensors, 20(12): 3374.

    Article  Google Scholar 

  • Huang Q, Papalia A and Leonard JJ (2022a), “Nested Sampling for Non-Gaussian Inference in SLAM Factor Graphs,” IEEE Robotics and Automation Letters, 7(4): 9232–9239.

    Article  Google Scholar 

  • Huang Q, Pu C, Khosoussi K, Rosen DM, Fourie D, How JP and Leonard JJ (2022b), “Incremental Non-Gaussian Inference for SLAM Using Normalizing Flows,” IEEE Transactions on Robotics.

  • Kalaitzakis M, Cain B, Carroll S, Ambrosi A, Whitehead C and Vitzilaios N (2021), “Fiducial Markers for Pose Estimation,” Journal of Intelligent & Robotic Systems, 101(4): 1–26.

    Article  Google Scholar 

  • Kallwies J, Forkel B and Wuensche H-J (2020), “Determining and Improving the Localization Accuracy of AprilTag Detection,” In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 8288–8294.

  • Khelifi F, Bradai A, Benslimane A, Rawat P and Atri M (2019), “A Survey of Localization Systems in Internet of Things,” Mobile Networks and Applications, 24: 761–785.

    Article  Google Scholar 

  • Levine NM and Spencer Jr BF (2022), “Post-Earthquake Building Evaluation Using UAVs: A BIM-Based Digital Twin Framework,” Sensors, 22(3): 873.

    Article  Google Scholar 

  • Lutz P, Schuster MJ and Steidle F (2019), “Visual-Inertial SLAM Aided Estimation of Anchor Poses and Sensor Error Model Parameters of UWB Radio Modules,” In: 2019 19th International Conference on Advanced Robotics (ICAR), pp. 739–746.

  • Mao Z, Yan Y, Wu J, Hajjar JF and Padlr T (2018), “Towards Automated Post-Disaster Damage Assessment of Critical Infrastructure with Small Unmanned Aircraft Systems,” In: 2018 IEEE International Symposium on Technologies for Homeland Security (HST), pp. 1–6.

  • Mazzoni S, McKenna F, Scott MH, Fenves GL, et al. (2006), “OpenSees Command Language Manual,” Pacific Earthquake Engineering Research Center (PEER), 264(1): 137–158.

    Google Scholar 

  • McCallen D, Petrone F, Coates J and Repanich N (2017), “A Laser-Based Optical Sensor for Broad-Band Measurements of Building Earthquake Drift,” Earthquake Spectra, 33(4): 1573–1598.

    Article  Google Scholar 

  • Munoz-Salinas R and Medina-Carnicer R (2020), “UcoSLAM: Simultaneous Localization and Mapping by Fusion of Keypoints and Squared Planar Markers,” Pattern Recognition, 101: 107193.

    Article  Google Scholar 

  • Nguyen TH, Nguyen TM and Xie L (2020a), “Tightly-Coupled Single-Anchor Ultra-Wideband-Aided Monocular Visual Odometry System,” In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 665–671.

  • Nguyen TH, Nguyen TM and Xie L (2020b), “Tightly-Coupled Ultra-Wideband-Aided Monocular Visual SLAM with Degenerate Anchor Configurations,” Autonomous Robots, 44(8): 1519–1534.

    Article  Google Scholar 

  • Nguyen TH, Nguyen TM and Xie L (2021a), “Range-Focused Fusion of Camera-IMU-UWB for Accurate and Drift-Reduced Localization,” IEEE Robotics and Automation Letters, 6(2): 1678–1685.

    Article  Google Scholar 

  • Nguyen TM, Cao M, Yuan S, Lyu Y, Nguyen TH and Xie L (2021b), “LIRO: Tightly Coupled Lidar-Inertia-Ranging Odometry,” In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 14484–14490.

  • Nguyen TM, Cao M, Yuan S, Lyu Y, Nguyen TH and Xie L (2021c), “Viral-Fusion: A Visual-Inertial-Ranging-Lidar Sensor Fusion Approach,” IEEE Transactions on Robotics, 38(2): 958–977.

    Article  Google Scholar 

  • Nguyen TM, Yuan S, Cao M, Lyu Y, Nguyen TH and Xie L (2021d), “NTU VIRAL: A Visual-Inertial-Ranging-Lidar Dataset, from an Aerial Vehicle Viewpoint,” The International Journal of Robotics Research, 41(3): 270–280.

    Article  Google Scholar 

  • NZS 1170.5 (2004), Structural Design Actions Part 5: Earthquake Actions - New Zealand, New Zealand.

  • Olson E (2011), “AprilTag: A Robust and Flexible Visual Fiducial System,” In: 2011 IEEE International Conference on Robotics and Automation, pp. 3400–3407.

  • Oppermann I, Hämäläinen M and Iinatti J (2004), UWB: Theory and Applications, John Wiley & Sons, USA.

    Book  Google Scholar 

  • Pacific Earthquake Engineering Research Centre (PEER) (2017), Tall Buildings Initiative (TBI) - Guidelines for Performance-Based Seismic Design of Tall Buildings, Berkeley: University of California, Berkeley, CA, USA.

    Google Scholar 

  • Park SW, Park HS, Kim JH and Adeli H (2015), “3D Displacement Measurement Model for Health Monitoring of Structures Using a Motion Capture System,” Measurement, 59: 352–362.

    Article  Google Scholar 

  • Petrone F, McCallen D, Buckle I and Wu S (2018), “Direct Measurement of Building Transient and Residual Drift Using an Optical Sensor System,” Engineering Structures, 176: 115–126.

    Article  Google Scholar 

  • Priyantha NB, Balakrishnan H, Demaine ED and Teller S (2005), “Mobile-Assisted Localization in Wireless Sensor Networks,” In: Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies, pp. 172–183.

  • Ramezani M, Khosoussi K, Catt G, Moghadam P, Williams J, Borges P, Pauling F and Kottege N (2022), “Wildcat: Online Continuous-Time 3D Lidar-Inertial SLAM,” arXiv, Preprint arXiv:220512595.

  • Recchiuto CT and Sgorbissa A (2018), “Post-Disaster Assessment with Unmanned Aerial Vehicles: A Survey on Practical Implementations and Research Approaches,” Journal of Field Robotics, 35(4): 459–490.

    Article  Google Scholar 

  • Romero-Ramirez FJ, Muñoz-Salinas R and Medina-Carnicer R (2018), “Speeded up Detection of Squared Fiducial Markers,” Image and Vision Computing, 76: 38–47.

    Article  Google Scholar 

  • Shi J, Yang H and Carlone L (2021), “ROBIN: a Graph-Theoretic Approach to Reject Outliers in Robust Estimation Using Invariants,” In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13820–13827.

  • Skolnik DA and Wallace JW (2010), “Critical Assessment of Interstory Drift Measurements,” Journal of Structural Engineering, 136(12): 1574–1584.

    Article  Google Scholar 

  • Stachniss C, Leonard JJ and Thrun S (2016), “Simultaneous Localization and Mapping,” Springer Handbook of Robotics, pp. 1153–1176.

  • Torok M, Fard M and Kochersberger K (2012), “Post-Disaster Robotic Building Assessment: Automated 3D Crack Detection from Image-Based Reconstructions,” In: Computing in Civil Engineering, pp. 397–404.

  • Torok MM, Golparvar-Fard M and Kochersberger KB (2014), “Image-Based Automated 3D Crack Detection for Post-Disaster Building Assessment,” Journal of Computing in Civil Engineering, 28(5): A4014004.

    Article  Google Scholar 

  • Wang C, Zhang H, Nguyen TM and Xie L (2017), “Ultra-Wideband Aided Fast Localization and Mapping System,” In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1602–1609.

  • Wang S, Rodgers C, Zhai G, Matiki TN, Welsh B, Najafi A, Wang J, Narazaki Y, Hoskere V and Spencer Jr BF (2022), “A Graphics-Based Digital Twin Framework for Computer Vision-Based Post-Earthquake Structural Inspection and Evaluation Using Unmanned Aerial Vehicles,” Journal of Infrastructure Intelligence and Resilience, 1(1): 100003.

    Article  Google Scholar 

  • Watanabe K (2018), “ARIADNE with Ambiguity Resolution: Visual Marker Based Rapid Initialization of PPP-AR,” In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7362–7368.

  • Wisth D, Camurri M and Fallon M (2023), “VILENS: Visual, Inertial, Lidar, and Leg Odometry for All-Terrain Legged Robots”, IEEE Transactions on Robotics, 39(1): 309–326.

    Article  Google Scholar 

  • Yang H, Antonante P, Tzoumas V and Carlone L (2020a), “Graduated Non-Convexity for Robust Spatial Perception: From Non-Minimal Solvers to Global Outlier Rejection,” IEEE Robotics and Automation Letters, 5(2): 1127–1134.

    Article  Google Scholar 

  • Yang H and Carlone L (2023), “Certifiably Optimal Outlier-Robust Geometric Perception: Semidefinite Relaxations and Scalable Global Optimization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3): 2816–2834.

    Google Scholar 

  • Yang YS, Xue Q, Chen PY, Weng JH, Li CH, Liu CC, Chen JS and Chen CT (2020b), “Image Analysis Applications for Building Inter-Story Drift Monitoring,” Applied Sciences, 10(20): 7304.

    Article  Google Scholar 

  • Yazgan U and Dazio A (2012), “Post-Earthquake Damage Assessment Using Residual Displacements,” Earthquake Engineering & Structural Dynamics, 41(8): 1257–1276.

    Article  Google Scholar 

  • Yick J, Mukherjee B and Ghosal D (2008), “Wireless Sensor Network Survey,” Computer Networks, 52(12): 2292–2330.

    Article  Google Scholar 

  • Zhang Y and Burton HV (2019), “Pattern Recognition Approach to Assess the Residual Structural Capacity of Damaged Tall Buildings,” Structural Safety, 78: 12–22.

    Article  Google Scholar 

  • Zhang Y, Burton HV, Shokrabadi M and Wallace JW (2019), “Seismic Risk Assessment of a 42-Story Reinforced Concrete Dual-System Building Considering Mainshock and Aftershock Hazard,” Journal of Structural Engineering, 145(11): 04019135.

    Article  Google Scholar 

  • Zhang Y, Burton HV, Sun H and Shokrabadi M (2018), “A Machine Learning Framework for Assessing Post-Earthquake Structural Safety,” Structural Safety, 72: 1–16.

    Article  Google Scholar 

Download references

Acknowledgment

This work was supported by the Natural Sciences and Engineering Research Council of Canada through their Research Tools and Instruments and Fellowship programs, and the Graduate Fellowship program by the University of California, Los Angeles.

Author information

Authors and Affiliations

  1. Department of Mechanical and Aerospace Engineering, University of California - Los Angeles, 580 Portola Plaza, 5731 Boelter Hall, Los Angeles, CA, 90095-159, USA

    Alexander Thoms

  2. Department of Civil and Environmental Engineering, University of Waterloo, Waterloo, Canada

    Zaid Al-Sabbag

  3. Department of Civil and Environmental Engineering, University of California - Los Angeles, 580 Portola Plaza, 5731 Boelter Hall, Los Angeles, CA, 90095-159, USA

    Sriram Narasimhan

Authors
  1. Alexander Thoms
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zaid Al-Sabbag
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sriram Narasimhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sriram Narasimhan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Thoms, A., Al-Sabbag, Z. & Narasimhan, S. Graph-based structural joint pose estimation in non-line-of-sight conditions. Earthq. Eng. Eng. Vib. 22, 371–386 (2023). https://doi.org/10.1007/s11803-023-2175-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11803-023-2175-y

Keywords

Search

Navigation