Skip to main content
SpringerLink
Log in

Obs-tackle: an obstacle detection system to assist navigation of visually impaired using smartphones

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

As the prevalence of vision impairment continues to rise worldwide, there is an increasing need for affordable and accessible solutions that improve the daily experiences of individuals with vision impairment. The Visually Impaired (VI) are often prone to falls and injuries due to their inability to recognize dangers on the path while navigating. It is therefore crucial that they are aware of potential hazards in both known and unknown environments. Obstacle detection plays a key role in navigation assistance solutions for VI users. There has been a surge in experimentation on obstacle detection since the introduction of autonomous navigation in automobiles, robots, and drones. Previously, auditory, laser, and depth sensors dominated obstacle detection; however, advances in computer vision and deep learning have enabled it using simpler tools like smartphone cameras. While previous approaches to obstacle detection using estimated depth data have been effective, they suffer from limitations such as compromised accuracy when adapted for edge devices and the incapability to identify objects in the scene. To address these limitations, we propose an indoor and outdoor obstacle detection and identification technique that combines semantic segmentation with depth estimation data. We hypothesize that this combination of techniques will enhance obstacle detection and identification compared to using depth data alone. To evaluate the effectiveness of our proposed Obstacle detection method, we validated it against ground truth Obstacle data derived from the DIODE and NYU Depth v2 dataset. Our experimental results demonstrate that the proposed method achieves near 85% accuracy in detecting nearby obstacles with lower false positive and false negative rates. The demonstration of the proposed system deployed as an Android app-‘Obs-tackle’ is available at https://youtu.be/PSn-FEc5EQg?si=qPGB13tkYkD1kSOf.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Availability of data and materials

The datasets used for evaluation can be found in the NYUv2 repository (https://cs.nyu.edu/silberman/datasets/nyudepthv2.html) and the DIODE repository (https://diode-dataset.org/). On reasonable request, the corresponding author will provide relevant code. Authors’ contributions - Vijetha U performed code development, experimentation, and analysis, and authored the initial draft of the paper. Dr. Geetha V supervised the research process, provided guidance, and contributed to paper revisions. All authors have read and approved the final manuscript.

Notes

  1. Detailed instruction for this conversion process can be found in the TensorFlow Lite documentation, available at https://www.tensorflow.org/lite/models/convert, which provides comprehensive guidance and best practices for converting TensorFlow models to tflite format.

References

  1. Lundälv, J., Thodelius, C.: Risk of injury events in patients with visual impairments: a Swedish survey study among hospital social workers. J. Vis. Impair. Blind. 115(5), 426–435 (2021)

    Article  Google Scholar 

  2. Vision Atlas. https://www.iapb.org/learn/vision-atlas

  3. Kuriakose, B., Shrestha, R., Sandnes, F.E.: Tools and technologies for blind and visually impaired navigation support: a review. IETE Tech. Rev. 39(1), 3–18 (2022)

  4. NoorCam MyEye https://www.noorcam.com/en-ae/noorcam-myeye

  5. Iris Vision. https://irisvision.com/

  6. NuEyes. https://www.nueyes.com/

  7. https://www.esighteyewear.com/

  8. Chen, H.-C.: Monocular vision-based obstacle detection and avoidance for a multicopter. IEEE Access 7, 167869–167883 (2019)

    Article  Google Scholar 

  9. Vorapatratorn, S.: AI-based obstacle detection and navigation for the blind using convolutional neural network. In: 2021 25th International Computer Science and Engineering Conference (ICSEC), pp. 17–22. IEEE (2021)

  10. Hua, M., Nan, Y., Lian, S.: Small obstacle avoidance based on RGB-D semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2019)

  11. Karunasekera, H., Zhang, H., Xi, T., Wang, H.: Stereo vision based negative obstacle detection. In: 2017 13th IEEE International Conference on Control and Automation (ICCA), pp. 834–838 (2017). https://doi.org/10.1109/ICCA.2017.8003168

  12. Martinez, M., Yang, K., Constantinescu, A., Stiefelhagen, R.: Helping the blind to get through COVID-19: social distancing assistant using real-time semantic segmentation on RGB-D video. Sensors 20(18), 5202 (2020)

    Article  Google Scholar 

  13. Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. Pattern Anal. Machi. Intell. (2020)

  14. Godard, C., Mac Aodha, O., Firman, M., Brostow, G.J.: Digging into self-supervised monocular depth estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3828–3838 (2019)

  15. Yadav, S., Joshi, R.C., Dutta, M.K., Kiac, M., Sikora, P.: Fusion of object recognition and obstacle detection approach for assisting visually challenged person. In: 2020 43rd International Conference on Telecommunications and Signal Processing (TSP), pp. 537–540 (2020)

  16. Suman, S., Mishra, S., Sahoo, K.S., Nayyar, A.: Vision navigator: a smart and intelligent obstacle recognition model for visually impaired users. Mobile Inf. Syst. 2022 (2022)

  17. Masud, U., Saeed, T., Malaikah, H.M., Islam, F.U., Abbas, G.: Smart assistive system for visually impaired people obstruction avoidance through object detection and classification. IEEE Access 10, 13428–13441 (2022)

  18. Atitallah, A.B., Said, Y., Atitallah, M.A.B., Albekairi, M., Kaaniche, K., Alanazi, T.M., Boubaker, S., Atri, M.: Embedded implementation of an obstacle detection system for blind and visually impaired persons’ assistance navigation. Comput. Electr. Eng. 108, 108714 (2023)

  19. Tian, S., Zheng, M., Zou, W., Li, X., Zhang, L.: Dynamic crosswalk scene understanding for the visually impaired. IEEE Trans. Neural Syst. Rehabil. Eng. 29, 1478–1486 (2021)

    Article  Google Scholar 

  20. Shimakawa, M., Matsushita, K., Taguchi, I., Okuma, C., Kiyota, K.: Smartphone apps of obstacle detection for visually impaired and its evaluation. In: Proceedings of the 7th ACIS International Conference on Applied Computing and Information Technology, pp. 1–6 (2019)

  21. Hussain, S.S., Durrani, D., Khan, A.A., Atta, R., Ahmed, L.: In-door obstacle detection and avoidance system for visually impaired people. In: 2020 IEEE Global Humanitarian Technology Conference (GHTC), pp. 1–7. IEEE (2020)

  22. Martínez-Cruz, S., Morales-Hernández, L.A., Pérez-Soto, G.I., Benitez-Rangel, J.P., Camarillo-Gómez, K.A.: An outdoor navigation assistance system for visually impaired people in public transportation. IEEE Access 9, 130767–130777 (2021)

    Article  Google Scholar 

  23. Martinez-Sala, A.S., Losilla, F., Sánchez-Aarnoutse, J.C., García-Haro, J.: Design, implementation and evaluation of an indoor navigation system for visually impaired people. Sensors 15(12), 32168–32187 (2015)

  24. Ahmetovic, D., Gleason, C., Ruan, C., Kitani, K., Takagi, H., Asakawa, C.: NavCog: a navigational cognitive assistant for the blind. In: Proceedings of the 18th International Conference on Human–Computer Interaction with Mobile Devices and Services, pp. 90–99 (2016)

  25. Elmannai, W.M., Elleithy, K.M.: A highly accurate and reliable data fusion framework for guiding the visually impaired. IEEE Access 6, 33029–33054 (2018)

    Article  Google Scholar 

  26. See, A.R., Sasing, B.G., Advincula, W.D.: A smartphone-based mobility assistant using depth imaging for visually impaired and blind. Appl. Sci. 12(6), 2802 (2022)

  27. Jindal, A., Aggarwal, N., Gupta, S.: An obstacle detection method for visually impaired persons by ground plane removal using speeded-up robust features and gray level co-occurrence matrix. Pattern Recognit. Image Anal. 28(2), 288–300 (2018)

    Article  Google Scholar 

  28. Kuriakose, B., Shrestha, R., Sandnes, F.E.: DeepNAVI: a deep learning based smartphone navigation assistant for people with visual impairments. Expert Syst. Appl. 212, 118720 (2023)

  29. Cordeiro, N.H., Pedrino, E.C.: Collision risk prediction for visually impaired people using high level information fusion. Eng. Appl. Artif. Intell. 81, 180–192 (2019)

  30. Khusro, S., Shah, B., Khan, I., Rahman, S.: Haptic feedback to assist blind people in indoor environment using vibration patterns. Sensors 22(1), 361 (2022)

    Article  Google Scholar 

  31. Hoang, V.-N., Nguyen, T.-H., Le, T.-L., Tran, T.-H., Vuong, T.-P., Vuillerme, N.: Obstacle detection and warning system for visually impaired people based on electrode matrix and mobile Kinect. Vietnam J. Comput. Sci. 4(2), 71–83 (2017)

    Article  Google Scholar 

  32. Kuriakose, B., Ness, I.M., Skov Tengstedt, M.Å., Svendsen, J.M., Bjørseth, T., Pradhan, B.L., Shrestha, R.: Turn left turn right-delving type and modality of instructions in navigation assistant systems for people with visual impairments. Int. J. Hum.–Comput. Stud. 103098 (2023)

  33. Singh, A., Kamireddypalli, A., Gandhi, V., Krishna, K.M.: Lidar guided small obstacle segmentation. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8513–8520, IEEE (2020)

  34. Tang, W., Liu, D., Zhao, X., Chen, Z., Zhao, C.: A dataset for the recognition of obstacles on blind sidewalk. Universal Access Inf. Soc. 22(1), 69–82 (2023)

    Article  Google Scholar 

  35. Cordts, M., Omran, M., Ramos, S., Scharwächter, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset. In: CVPR Workshop on the Future of Datasets in Vision, vol. 2. sn (2015)

  36. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: European Conference on Computer Vision, pp. 746–760. Springer, Berlin (2012)

  37. Song, S., Lichtenberg, S.P., Xiao, J.: Sun rgb-d: a rgb-d scene understanding benchmark suite. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 567–576 (2015)

  38. Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through the ade20k dataset. Int. J. Comput. Vis. 127, 302–321 (2019)

    Article  Google Scholar 

  39. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  40. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890 (2017)

  41. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder–decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 801–818 (2018)

  42. Zhao, H., Qi, X., Shen, X., Shi, J., Jia, J.: Icnet for real-time semantic segmentation on high-resolution images. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 405–420 (2018)

  43. Xu, J., Xiong, Z., Bhattacharyya, S.P.: Pidnet: a real-time semantic segmentation network inspired from pid controller. arXiv preprint arXiv:2206.02066 (2022)

  44. Zhang, W., Huang, Z., Luo, G., Chen, T., Wang, X., Liu, W., Yu, G., Shen, C.: TopFormer: token pyramid transformer for mobile semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12083–12093 (2022)

  45. Vasiljevic, I., Kolkin, N., Zhang, S., Luo, R., Wang, H., Dai, F.Z., Daniele, A.F., et al.: Diode: a dense indoor and outdoor depth dataset. arXiv preprint arXiv:1908.00463 (2019)

  46. Gharani, P., Karimi, H.A.: Context-aware obstacle detection for navigation by visually impaired. Image Vis. Comput. 64, 103–115 (2017)

    Article  Google Scholar 

  47. Duh, P.-J., Sung,Y.-C., Chiang, L.-Y.F., Chang, Y.-J., Chen, K.-W.: V-eye: a vision-based navigation system for the visually impaired. IEEE Trans. Multimedia 23, 1567–1580 (2020)

  48. Chen, H., Zhang, Y., Yang, K., Martinez, M., Müller, K., Stiefelhagen, R.: Can we unify perception and localization in assisted navigation? An indoor semantic visual positioning system for visually impaired people. In: International Conference on Computers Helping People with Special Needs, pp. 97–104. Springer, Cham (2020)

  49. Dimas, G., Diamantis, D.E., Kalozoumis, P., Iakovidis, D.K.: Uncertainty-aware visual perception system for outdoor navigation of the visually challenged. Sensors 20(8), 2385 (2020)

  50. Kang, M.-C., Chae, S.-H., Sun, J.-Y., Yoo, J.-W., Ko, S.-J.: A novel obstacle detection method based on deformable grid for the visually impaired. IEEE Trans. Consumer Electron. 61(3), 376–383 (2015)

    Article  Google Scholar 

Download references

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. National Institute of Technology Karnataka, Surathkal, Karnataka, 575025, India

    U. Vijetha & V. Geetha

Authors
  1. U. Vijetha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. V. Geetha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to U. Vijetha.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vijetha, U., Geetha, V. Obs-tackle: an obstacle detection system to assist navigation of visually impaired using smartphones. Machine Vision and Applications 35, 20 (2024). https://doi.org/10.1007/s00138-023-01499-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-023-01499-8

Keywords

Search

Navigation