Skip to main content

Advertisement

SpringerLink
Log in

Experiments with cooperative robots that can detect object’s shape, color and size to perform tasks in industrial workplaces

  • Regular Paper
  • Published:
International Journal of Intelligent Robotics and Applications Aims and scope Submit manuscript

Abstract

Automation and human-robot collaboration are increasing in modern workplaces such as industrial manufacturing. Nowadays, humans rely heavily on advanced robotic devices to perform tasks quickly and accurately. Modern robots with computer vision and artificial intelligence are gaining attention and popularity rapidly. This paper demonstrates how a robot can automatically detect an object’s shape, color, and size using computer vision techniques and act based on information feedback. In this work, a powerful computational model for a robot has been developed that distinguishes an object’s shape, size, and color in real time with high accuracy. Then it can integrate a robotic arm to pick a specific object. A dataset of 6558 images of various monochromatic objects has been developed, containing three colors against a white background and five shapes for the research. The designed system for detection has achieved 99.8% success in an object’s shape detection. Also, the system demonstrated 100% success in the object’s color and size detection with the OpenCV image processing framework. On the other hand, the prototype robotic system based on Raspberry Pi-4B has achieved 80.7% accuracy for geometrical shape detection and 81.07%, and 59.77% accuracy for color recognition and distance measurement, respectively. Moreover, the system guided a robotic arm to pick up the object based on its color and shape with a mean response time of 19 seconds. The idea is to simulate a workplace environment where a worker will ask the robotic systems to perform a task on a specific object. Our robotic system can accurately identify the object’s attributes (e.g., 100%) and is able to perform the task reliably (81%). However, reliability can be improved by using a more powerful computing system, such as the robotic prototype. The article’s contribution is to use a cutting-edge computer vision technique to detect and categorize objects with the help of a small private dataset to shorten the training duration and enable the suggested system to adapt to components that may be needed for creating a new industrial product in a shorter period. The source code and images of the collected dataset can be found at: https://github.com/TituShahoriar/cse499B_Hardware_Proposed_System.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Algorithm 1
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Abdi, A., Ranjbar, M.H., Park, J.H.: Computer vision-based path planning for robot arms in three-dimensional workspaces using Q-Learning and neural networks. Sensors 22(5), 1697 (2022)

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  • Ahmed, I., Din, S., Jeon, G., Piccialli, F., Fortino, G.: Towards collaborative robotics in top view surveillance: a framework for multiple object tracking by detection using deep learning. IEEE/CAA J. Autom. Sin. 8, 1253–1270 (2021)

    Article  Google Scholar 

  • Arunachalam, M., Sanghavi, V., Kaira, S., Ahuja, N.A.: End-to-end industrial IoT: software optimization and acceleration. IEEE Internet Things Mag. 5, 48–53 (2022)

    Article  Google Scholar 

  • Azad, A., Misbahuddin, M.: Web-based object tracking using collaborated camera network. Adv. Internet Things 8, 13–25 (2018)

    Article  Google Scholar 

  • Bai, Y.: Data Selection Query with Visual Basic.NET, pp. 215–325. Wiley-IEEE Press (2020)

  • Banerjee, D., Yu, K., Aggarwal, G.: Object tracking test automation using a robotic arm. IEEE Access 6, 56378–56394 (2018)

    Article  Google Scholar 

  • Chen, L., Sun, H., Zhao, W., Yu, T.: Robotic arm control system based on AI wearable acceleration sensor. Math. Probl. Eng. 2021 (2021)

  • Chen, C.-S., Hu, N.-T.: Eye-in-hand robotic arm gripping system based on machine learning and state delay optimization. Sensors 23, 1076 (2023)

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  • Cong, V.D., Phuong, L., Chi, H., City, M., Kiji, S.: Design and development of a delta robot system to classify objects using image processing. Int. J. Electr. Comput. Eng. 13, 2669–2676 (2023)

    Google Scholar 

  • Divya, Y., Kumar, C.P.: Machine vision based color recognition by robotic arm using LabVIEW. CVR J. Sci. Technol. 18, 100–104 (2020)

    Article  Google Scholar 

  • Fernandes, L., Shivakumar, B.: Identification and sorting of objects based on shape and colour using robotic arm. In: International Conference on Inventive Systems and Control, pp. 866–871 (2020)

  • Gode, C.S., Khobragade, A.S.: Object detection using color clue and shape feature. In: International Conference on Wireless Communications, Signal Processing and Networking, pp. 464–468 (2016)

  • Grudin, J., Carroll, J.M.: From Tool to Partner: The Evolution of Human-Computer Interaction. Morgan & Claypool, San Rafael (2017)

    Book  Google Scholar 

  • Gupta, S., Singh, Y.J., Kumar, M.: Object detection using multiple shape-based features. In: International Conference on Parallel, Distributed and Grid Computing, pp. 433–437 (2016)

  • Hamzah, M., Thamrin, N., Tajjudin, M.: Robotic arm position control using Mamdani fuzzy logic on Arduino microcontroller. J. Mech. Eng. 19, 235–255 (2022)

    Article  Google Scholar 

  • Imran, M.A., Hussain, S., Abbasi, Q.H.: Ultra Reliable Low Latency Communications as an Enabler for Industry Automation, pp. 89–107. Wiley (2020)

  • Intisar, M., Khan, M., Islam, M., Masud, M.: Computer vision based robotic arm controlled using interactive GUI. Intell. Autom. Soft Comput. 27, 533–550 (2021)

    Article  Google Scholar 

  • Kafadar, O.: RaspMI: Raspberry Pi assisted embedded system for monitoring and recording of seismic ambient noise. IEEE Sens. J. 21, 6306–6313 (2021)

    Article  ADS  Google Scholar 

  • Kondratenko, Y.P., Kuntsevich, V.M., Chikrii, A.A., Gubarev, V.F.: Part I: Advances in Theoretical Research on Automatic Control, pp. 1–1. River Publishers, Aalborg (2021)

    Google Scholar 

  • Koodtalang, W., Sangsuwan,T.: The chicken’s legs size classification using image processing and deep neural network. In: International Symposium on Instrumentation, Control, Artificial Intelligence, and Robotics, pp. 183–186 (2019)

  • Krishnadas, P., Sampathila, N.: Automated detection of malaria implemented by deep learning in PyTorch. In: International Conference on Electronics, Computing and Communication Technologies, pp. 01–05 (2021)

  • Liu, X., Jin, S.: Multi-target visual recognition and positioning methods for sorting robots. In: IOP Conference Series: Materials Science and Engineering, vol. 892 (2020)

  • Maasoumy, M., Sangiovanni-Vincentelli, A.: Smart connected buildings design automation: Foundations and trends. In: Foundations and Trends in Electronic Design Automation, pp. 1–143 (2016)

  • Magadum, R.A., Bombale, U.L.: Intelligent color based object sorting using robotics. Int. J. Robot. Res. Dev. 9, 43–48 (2019)

    Article  Google Scholar 

  • Najmurrokhman, A., Kusnandar, K., Maulana, F., Wibowo, B., Nurlina, E.: Design of a prototype of manipulator arm for implementing pick-and-place task in industrial robot system using TCS3200 color sensor and ATmega2560 microcontroller. J. Phys. Conf. Ser. 1375, 1–7 (2019)

    Article  Google Scholar 

  • Noman, A., Eva, A., Yeahyea, T., Khan, R.: Computer vision-based robotic arm for object color, shape, and size detection. J. Robot. Control 3, 180–186 (2022)

    Article  Google Scholar 

  • Othman, N.A., Salur, M.U., Karaose, M., Aydin, I.: An embedded real-time object detection and measurement of its size. In: International Conference on Artificial Intelligence and Data Processing, pp. 1–4 (2018)

  • Parveen, S., Shah, J.: A motion detection system in Python and OpenCV. In: International Conference on Intelligent Communication Technologies and Virtual Mobile Networks, pp. 1378–1382 (2021)

  • Phelps, R., Krasnicki, M., Rutenbar, R., Carley, L., Hellums, J.: Anaconda: simulation-based synthesis of analog circuits via stochastic pattern search. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 19, 703–717 (2000)

    Article  Google Scholar 

  • Qayum, M.A., Nahar, N., Siddique, N.A., Saifullah, Z.: Interactive intelligent agents with creative minds: experiments with mobile robots in cooperating tasks by using machine learning. In: International Conference on Imaging, Vision & Pattern Recognition, pp. 1–6. IEEE (2017)

  • Rafique, D., Velasco, L.: Machine learning for network automation: overview, architecture, and applications. J. Opt. Commun. Netw. 10, 126–143 (2018)

    Article  Google Scholar 

  • Ranjbarzadeh, R., Caputo, A., Tirkolaee, E.B., Jafarzadeh Ghoushchi, S., Bendechache, M.: Brain tumor segmentation of MRI images: a comprehensive review on the application of artificial intelligence tools. Comput. Biol. Med. 152 (2023)

  • Ranjbarzadeh, R., Ghoushchi, S., Anari, S., Safavi, S., Sarshar, N., Tirkolaee, E. Babaee, Bendechache, M.: A deep learning approach for robust, multi-oriented, and curved text detection. Cogn. Comput. 1–13 (2022)

  • Roy, S., Hazera, C., Das, D., Pir, R., Ahmed, A.S.: A computer vision and artificial intelligence based cost-effective object sensing robot. Int. J. Intell. Robot. Appl. 3, 457–470 (2019)

    Article  Google Scholar 

  • Sanjaya, W.S.M., et al.: Colored object sorting using 5 DoF robot arm based artificial neural network (ANN) method. J. Phys. Conf. Ser. 1090, 1–10 (2018)

    Google Scholar 

  • Sawant, N., Tyagi, A., Sawant, S., Tade, S.L.: Implementation of faster RCNN algorithm for smart robotic arm based on computer vision. In: International Conference on Computing, Communication, Control and Automation, pp. 1–6 (2022)

  • Sekkat, H., Tigani, S., Rachid, S., Chehri, A.: Vision-based robotic arm control algorithm using deep reinforcement learning for autonomous objects grasping. Appl. Sci. 11, 7917 (2021)

    Article  CAS  Google Scholar 

  • Shahid, M.A., Iftikhar, M.A., Gondal, Z.A., Adnan, M., Rathore, S.: Object size measurement through images: an application to measuring human foot size. In: International Conference on Frontiers of Information Technology, pp. 298–302 (2018)

  • Shaikat, A.S., Akter, S., Salma, U.: Computer vision based industrial robotic arm for sorting objects by color and height. J. Eng. Adv. 1, 116–122 (2020)

    Article  Google Scholar 

  • Sivapriyan, R., Ajay, K.V., Ashwath, K.N.: Arduino-Nano based low cost power converter learning kit. In: International Conference on Inventive Systems and Control, pp. 133–137 (2020)

  • Sudhoff, S.: Power magnetic devices: a multi-objective design approach. In: Power Magnetic Devices: A Multi-Objective Design Approach (2014)

  • Wang, Z., Fan, Y., Zhang, H.: Lane-line detection algorithm for complex road based on OpenCV. In: Advanced Information Management, Communicates, Electronic and Automation Control Conference, pp. 1404–1407 (2019)

  • Wang, J., Long, L.: Raspberry Pi 4B-based design of home intelligent access control system. In: Asia-Pacific Conference on Image Processing, Electronics and Computers, pp. 959–962 (2022)

  • Wettinger, J., Andrikopoulos, V., Leymann, F., Strauch, S.: Middleware-oriented deployment automation for cloud applications. IEEE Trans. Cloud Comput. 6, 1054–1066 (2018)

    Article  Google Scholar 

  • Zhang, W., Zhang, C., Li, C., Zhang, H.: Object color recognition and sorting robot based on OpenCV and machine vision. In: International Conference on Mechanical and Intelligent Manufacturing Technologies, pp. 125–129 (2020)

Download references

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering, North South University, Dhaka, Bangladesh

    Md Fahim Shahoriar Titu, S. M. Rezwanul Haque, Rifad Islam, Akram Hossain, Mohammad Abdul Qayum & Riasat Khan

Authors
  1. Md Fahim Shahoriar Titu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. M. Rezwanul Haque
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rifad Islam
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Akram Hossain
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mohammad Abdul Qayum
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Riasat Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Riasat Khan.

Ethics declarations

Conflict of interest

We certify that there is no actual or potential conflict of interest in relation to this article.

Supplementary material

The source code and images of the collected dataset can be found at: https://github.com/TituShahoriar/cse499B_Hardware_Proposed_System.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Titu, M.F.S., Haque, S.M.R., Islam, R. et al. Experiments with cooperative robots that can detect object’s shape, color and size to perform tasks in industrial workplaces. Int J Intell Robot Appl 8, 179–192 (2024). https://doi.org/10.1007/s41315-023-00305-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41315-023-00305-y

Keywords

Search

Navigation