Skip to main content
SpringerLink
Log in

Autonomous UAV-based surveillance system for multi-target detection using reinforcement learning

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Recent advances in unmanned aerial vehicle (UAV) technology have revolutionized various industries, finding applications in embedded systems, autonomy, control, security, and communication. Autonomous UAVs are distinguished by their ability to make informed decisions, anticipate potential scenarios, and learn from past experiences with the help of AI algorithms. This paper examines a practical monitoring system with an autonomous UAV, a charging station, and multiple targets that move randomly within a defined mission area. The mission area is divided into zones, and the UAV navigates through these zones efficiently. The primary objective is to maximize the probability of detecting targets, considering constraints such as limited battery life and charging station location. This challenge is initially framed as a search benefit maximization problem and subsequently reformulated as a Markov Decision Process (MDP) problem. To address the MDP formulation, we introduce a reinforcement learning (RL)-based approach that enables the UAV to comprehend unpredictable multi-target movements autonomously. The placement of the charging station in the proposed system is determined using the optimal median approach. The simulation results demonstrate that the proposed RL-based detection system significantly outperforms the reference systems in terms of detection rate and convergence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Ahamad, R., Mishra, K.N.: Hybrid approach for suspicious object surveillance using video clips and uav images in cloud-iot-based computing environment. Clust. Comput. 27(1), 761–785 (2024)

    Article  Google Scholar 

  2. Ramachandran, A., Sangaiah, A.K.: A review on object detection in unmanned aerial vehicle surveillance. Int. J. Cogn. Comput. Eng. 2, 215–228 (2021)

    Google Scholar 

  3. Bany Salameh, H., Masadeh, A., Refae, G.. E.: Intelligent drone-base-station placement for improved revenue in b5g/6g systems under uncertain fluctuated demands. IEEE Access 10, 106 740-106 749 (2022)

    Article  Google Scholar 

  4. Singh, P., Salameh, H.B., Bohara, V.A., Srivastava, A., Ayyash, M.: Optimizing connectivity in oirs-assisted uav indoor optical networks: Efficient admission control and mirror-element assignment. In: IEEE Transactions on Network Science and Engineering, pp. 1–11 (2024)

  5. Shakhatreh, H., Khreishah, A., Chakareski, J., Salameh, H.B., Khalil, I.: On the continuous coverage problem for a swarm of UAVs. In: 2016 IEEE 37th Sarnoff Symposium, pp. 130–135 (2016)

  6. Zhai, H., Zhang Y., et al.: Target detection of low-altitude uav based on improved yolov3 network. J. Robot. 2022 (2022)

  7. Hentati, A.I., Fourati, L.C., Rezgui, J.: Cooperative UAVs framework for mobile target search and tracking. Comput. Electr. Eng. 101, 107992 (2022)

    Article  Google Scholar 

  8. Saetchnikov, I., Skakun, V., Tcherniavskaia, E.: Efficient objects tracking from an unmanned aerial vehicle. In: 2021 IEEE 8th International Workshop on Metrology for AeroSpace (MetroAeroSpace), pp. 221–225 (2021)

  9. Bouguettaya, A., Zarzour, H., Kechida, A., Taberkit, A.M.: A survey on deep learning-based identification of plant and crop diseases from uav-based aerial images. Clust. Comput. 26(2), 1297–1317 (2023)

    Article  Google Scholar 

  10. Mustafa, E., Shuja, J., Bilal, K., Mustafa, S., Maqsood, T., Rehman, F., Khan, A.. u. R.: Reinforcement learning for intelligent online computation offloading in wireless powered edge networks. Clust. Comput. 26(2), 1053–1062 (2023)

    Article  Google Scholar 

  11. Redding, J.D. , McLain, T.W., Beard, R.W. , Taylor, C.N.: Vision-based target localization from a fixed-wing miniature air vehicle. In: American Control Conference, pp. 6–12 (2006)

  12. Quigley, M., Goodrich, M.A., Griffiths, S., Eldredge, A., Beard, R.W.: Target acquisition, localization, and surveillance using a fixed-wing mini-uav and gimbaled camera. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, pp. 2600–2605 (2005)

  13. Pham, H.X., La, H.M., Feil-Seifer, D., Van Nguyen, L.: Reinforcement learning for autonomous uav navigation using function approximation. In: 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pp. 1–6 (2018)

  14. Wei, X.L., Huang, X.L., Lu, T., Song, G.G.: An improved method based on deep reinforcement learning for target searching. In: 2019 4th international Conference on Robotics and Automation Engineering (ICRAE), pp. 130–134 (2019)

  15. Guerra, A., Guidi, F., Dardari, D., Djurić, P..M.: Reinforcement learning for uav autonomous navigation, mapping and target detection. In: IEEE/ION Position. Location and Navigation Symposium (PLANS) 2020, 1004–1013 (2020)

  16. Bany Salameh, H., Alhafnawi, M., Masadeh, A., Jararweh, Y.: Federated reinforcement learning approach for detecting uncertain deceptive target using autonomous dual uav system. Inf. Process. Manag. 60(2), 103149 (2023)

    Article  Google Scholar 

  17. Masadeh, A., Alhafnawi, M., Salameh, H.A.B., Musa, A., Jararweh, Y.: Reinforcement learning-based security/safety uav system for intrusion detection under dynamic and uncertain target movement. In: IEEE Transactions on Engineering Management, pp. 1–11 (2022)

  18. Elhussein, A., Miah, M.S.: A novel model-free actor-critic reinforcement learning approach for dynamic target tracking. In: 2020 IEEE Midwest Industry Conference (MIC) 1, 1–6 (2020)

  19. Moon, J., Papaioannou, S., Laoudias, C., Kolios, P., Kim, S.: Deep reinforcement learning multi-uav trajectory control for target tracking. IEEE Internet Things J. 8(20), 15441–15445 (2021)

    Article  Google Scholar 

  20. Liu, J., Jia, R., Li, W., Ma, F., Abdullah, H.M., Ma, H., Mohamed, M.A.: High precision detection algorithm based on improved retinanet for defect recognition of transmission lines. Energy Rep. 6, 2430–2440 (2020)

    Article  Google Scholar 

  21. Hong, T., Liang, H., Yang, Q., Fang, L., Kadoch, M., Cheriet, M.: A real-time tracking algorithm for multi-target uav based on deep learning. Remote Sens. 15(1), 2 (2022)

    Article  Google Scholar 

  22. Yang, F., Ma, B., Wang, J., Gao, H., Liu, Z.: Target detection of uav aerial image based on rotational invariant depth denoising automatic encoder. Xibei Gongye Daxue Xuebao/Journal of Northwestern Polytechnical University 38(6), 1345–1351 (2020)

    Article  Google Scholar 

  23. Alhafnawi, M., Bany Salameh, H.. A., Masadeh, A., Al-Obiedollah, H., Ayyash, M., El-Khazali, R., Elgala, H.: A survey of indoor and outdoor uav-based target tracking systems: Current status, challenges, technologies, and future directions. IEEE Access 11, 68 324-68 339 (2023)

    Article  Google Scholar 

  24. Sohn, S., Lee, B., Kim, J., Kee, C.: Vision-based real-time target localization for single-antenna gps-guided uav. IEEE Trans. Aerospace Electron. Syst. 44(4), 1391–1401 (2008)

    Article  Google Scholar 

  25. Thillainayagi, R., Senthil Kumar, K.: Combination of wavelet transform and singular value decomposition-based contrast enhancement technique for target detection in uav reconnaissance thermal images. J. Mod. Opt. 66(6), 606–617 (2019)

    Article  Google Scholar 

  26. Abdulridha, J., Ampatzidis, Y., Kakarla, S.C., Roberts, P.: Detection of target spot and bacterial spot diseases in tomato using uav-based and benchtop-based hyperspectral imaging techniques. Precis. Agric. 21, 955–978 (2020)

    Article  Google Scholar 

  27. Jiang, M.-X., Deng, C., Pan, Z.-G., Wang, L.-F., Sun, X.: Multiobject tracking in videos based on lstm and deep reinforcement learning. Complexity 2018, 1–12 (2018)

    Google Scholar 

  28. Wang, X., Zhao, H., Han, T., Zhou, H., Li, C.: A grey wolf optimizer using gaussian estimation of distribution and its application in the multi-uav multi-target urban tracking problem. Appl. Soft Comput. 78, 240–260 (2019)

    Article  Google Scholar 

  29. Liu, J.: Multi-target detection method based on yolov4 convolutional neural network. In: Journal of Physics: Conference Series, Vol. no. 1. IOP Publishing 2021, 1–5 (1883)

  30. Xiao, Z., Liu, D., Fei, B., Men, T., Zhou, Z., Zhang, X., Zhou, Y.: Moving target tracking for single uav in open outdoor environment. In: 2020 6th International Conference on Big Data and Information Analytics (BigDIA), pp. 317–324 (2020)

  31. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, Vol. 28 (2015)

  32. Li, J., Ye, D.H., Chung, T., Kolsch, M., Wachs, J., Bouman, C.: Multi-target detection and tracking from a single camera in unmanned aerial vehicles (uavs). In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. 2016, pp. 4992–4997 (2016)

  33. Micheal, A.A., Vani, K., Sanjeevi, S., Lin, C.-H.: Object detection and tracking with uav data using deep learning. J. Indian Soc. Remote Sens. 49, 463–469 (2021)

    Article  Google Scholar 

  34. Yang, B., Cao, X., Yuen, C., Qian, L.: Offloading optimization in edge computing for deep-learning-enabled target tracking by internet of uavs. IEEE Internet Things J. 8(12), 9878–9893 (2020)

    Article  Google Scholar 

  35. Spyridis, Y., Lagkas, T., Sarigiannidis, P., Argyriou, V., Sarigiannidis, A., Eleftherakis, G., Zhang, J.: Towards 6g iot: tracing mobile sensor nodes with deep learning clustering in uav networks. Sensors 21(11), 3936 (2021)

    Article  Google Scholar 

  36. Zhou, L., Sharma, V.D., Li, Q., Prorok, A., Ribeiro, A., Tokekar, P., Kumar, V.: Graph neural networks for decentralized multi-robot target tracking. In: 2022 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pp. 195–202 (2022)

  37. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)

    Google Scholar 

  38. Ladosz, P., Weng, L., Kim, M., Oh, H.: Exploration in deep reinforcement learning: a survey. Inf. Fusion 85, 1–22 (2022)

    Article  Google Scholar 

  39. Masadeh, A., Wang, Z., Kamal, A.E.: Reinforcement learning exploration algorithms for energy harvesting communications systems. In: IEEE International Conference on Communications (ICC), Vol. 2018, pp. 1–6 (2018)

  40. Jamshidi, M.: Median location problem. In: Facility location: Concepts, models, algorithms and case studies, pp. 177–191 (2009)

Download references

Funding

This work was funded by the ASPIRE Award for Research Excellence (AARE-2020), Abu Dhabi, United Arab Emirates (Grant Number: AARE20-161).

Author information

Authors and Affiliations

  1. Telecommunications Engineering Department, Yarmouk University, Irbid, Jordan

    Haythem Bany Salameh, Ayyoub Hussienat & Ahmad Al-Ajlouni

  2. College of Engineering, Al Ain University, Al Ain, 1, United Arab Emirates

    Haythem Bany Salameh

  3. Electrical Engineering Department, Al-Balqa Applied University, Salt, Jordan

    Ayyoub Hussienat & Ahmad Al-Ajlouni

  4. Electrical Engineering Department, Al-Huson University College, Al-Balqa Applied University, Salt, Jordan

    Mohannad Alhafnawi

Authors
  1. Haythem Bany Salameh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayyoub Hussienat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohannad Alhafnawi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ahmad Al-Ajlouni
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: [Haythem Bany Salameh, Mohannad Alhafnawi], Methodology: [Haythem A. Bany Salameh, Ayyoub Hassanat, Ahmad Alajlouni, Mohannad Alhafnawi]; Formal analysis and investigation: [Haythem Bany Salameh, Ayyoub Hassanat]; Writing-original draft preparation: [Haythem Bany Salameh, Ayyoub Hassanat]; Writing-review and editing: [Haythem Bany Salameh, Mohannad Alhafnawi, Ahmad Alajlouni]

Corresponding author

Correspondence to Haythem Bany Salameh.

Ethics declarations

Data Availability Statement:

No data-sets were generated or analyzed during this study.

Ethical Statement:

We state that the paper is original and was not published elsewhere.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bany Salameh, H., Hussienat, A., Alhafnawi, M. et al. Autonomous UAV-based surveillance system for multi-target detection using reinforcement learning. Cluster Comput (2024). https://doi.org/10.1007/s10586-024-04452-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10586-024-04452-0

Keywords

Search

Navigation