Skip to main content
SpringerLink
Log in

Jubileo: An Immersive Simulation Framework for Social Robot Design

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

This paper introduces Jubileo, an open-source simulated humanoid robot as a framework for the development of human-robot interaction applications. By leveraging the power of the Robot Operating System (ROS) and Unity in a virtual reality environment, this simulation establishes a strong connection to real robotics, faithfully replicating the robot’s physical components down to its motors and enabling communication with servo-actuators to control both the animatronic face and the joints of a real humanoid robot. To validate the capabilities of the framework, we propose English teaching games that integrate Virtual Reality (VR), game-based Human-Robot Interaction (HRI), and advanced large language models such as Generative Pre-trained Transformer (GPT). These games aim to foster linguistic competence within dynamic and interactive virtual environments. The incorporation of large language models bolsters the robot’s capability to generate human-like responses, thus facilitating a more realistic conversational experience. Moreover, the simulation framework reduces real-world testing risks and offers a cost-effective, efficient, and scalable platform for developing new HRI applications. The paper underscores the transformative potential of converging VR, large language models, and HRI, particularly in educational applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Available of data and material

Available in GitHub https://github.com/jajaguto/jubileo.

References

  1. Goodrich, M., Schultz, A.: Human-robot interaction: a survey. Foundations and Trends in Human-Computer Interaction. 1, 203–275 (2007). https://doi.org/10.1561/1100000005

    Article  Google Scholar 

  2. Terzioğlu, Y., Mutlu, B., Şhin, E.: Designing social cues for collaborative robots: the role of gaze and breathing in human-robot collaboration. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-robot Interaction, pp. 343–357 (2020)

  3. Cao, H.-L., Scholz, C., De Winter, J., Makrini, I.E., Vanderborght, B.: Investigating the role of multi-modal social cues in human-robot collaboration in industrial settings. International Journal of Social Robotics, 1–11 (2023)

  4. Pennisi, P., Tonacci, A., Tartarisco, G., Billeci, L., Ruta, L., Gangemi, S., Pioggia, G.: Autism and social robotics: a systematic review. Autism Res. 9(2), 165–183 (2016)

    Article  Google Scholar 

  5. Breazeal, C., Dautenhahn, K., Kanda, T.: Social robotics. Springer handbook of robotics. 1935–1972 (2016)

  6. Ishiguro, H.: Android science. Cognitive neuroscience robotics: a synthetic approaches to human understanding. 193–234 (2016)

  7. Asada, M., Hosoda, K., Kuniyoshi, Y., Ishiguro, H., Inui, T., Yoshikawa, Y., Ogino, M., Yoshida, C.: Cognitive developmental robotics: A survey. IEEE Trans. Auton. Ment. Dev. 1(1), 12–34 (2009)

    Article  Google Scholar 

  8. Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al.: Solving rubik’s cube with a robot hand (2019). arXiv:1910.07113

  9. Monteiro, F.F., Vieira, A.L.B., Teixeira, J.M.X.N., Teichrieb, V., et al.: Simulating real robots in virtual environments using nvidia’s isaac sdk. In: Anais Estendidos do XXI Simpósio de Realidade Virtual e Aumentada. pp. 47–48 (2019). SBC

  10. Tan, J.T.C., Inamura, T.: Sigverse-a cloud computing architecture simulation platform for social human-robot interaction. In: IEEE International Conference on Robotics and Automation. pp. 1310–1315 (2012). IEEE

  11. Liu, O., Rakita, D., Mutlu, B., Gleicher, M.: Understanding human-robot interaction in virtual reality. In: 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). pp. 751–757 (2017). IEEE

  12. Bottega, J.A., Kich, V.A., Kolling, A.H., Dyonisio, J.D., Corçaque, P.L., Guerra, R.d.S., Gamarra, D.F.: Jubileo: An open-source robot and framework for research in human robot social interaction. In: 2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids). pp. 149–154 (2022). IEEE

  13. Bottega, J.A., Steinmetz, R., Kolling, A.H., Kich, V.A., De Jesus, J.C., Grando, R.B., Gamarra, D.F.T.: Virtual reality platform to develop and test applications on human robot social interaction. In: 2022 Latin American Robotics Symposium (LARS), 2022 Brazilian Symposium on Robotics (SBR), and 2022 Workshop on Robotics in Education (WRE), pp. 1–6 (2022). IEEE

  14. Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models. (2023). arXiv:2303.18223

  15. Wang, B., Li, G., Li, Y.: Enabling conversational interaction with mobile ui using large language models. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. pp. 1–17 (2023)

  16. Ross, S.I., Martinez, F., Houde, S., Muller, M., Weisz, J.D.: The programmer’s assistant: Conversational interaction with a large language model for software development. In: Proceedings of the 28th International Conference on Intelligent User Interfaces. pp. 491–514 (2023)

  17. Godwin-Jones, R.: Games in language learning: opportunities and challenges (2014)

  18. Klimova, B., Kacet, J.: Efficacy of computer games on language learning. Turkish Online Journal of Educational Technology-TOJET. 16(4), 19–26 (2017)

    Google Scholar 

  19. al, M.: The uncanny valley [from the field]. IEEE Robotics & Automation Magazine. 19(2), 98–100 (2012)

  20. Sherwani, F., Asad, M.M., Ibrahim, B.S.K.K.: Collaborative robots and industrial revolution 4.0 (ir 4.0). In: 2020 International Conference on Emerging Trends in Smart Technologies (ICETST). pp. 1–5 (2020). IEEE

  21. Goodrich, M.A., Schultz, A.C., et al.: Human–robot interaction: a survey. Foundations and Trends® in Human–Computer Interaction. 1(3), 203–275 (2008)

  22. Sheridan, T.B.: Human–robot interaction: status and challenges. Hum. Factors 58(4), 525–532 (2016)

  23. Yanco, H.A., Drury, J.L.: A taxonomy for human-robot interaction. In: Proceedings of the AAAI Fall Symposium on Human-robot Interaction. pp. 111–119 (2002)

  24. Faraj, Z., Selamet, M., Morales, C., Torres, P., Hossain, M., Chen, B., Lipson, H.: Facially expressive humanoid robotic face. HardwareX. 9, 00117 (2021)

    Google Scholar 

  25. Walters, M.L., Syrdal, D.S., Dautenhahn, K., Te Boekhorst, R., Koay, K.L.: Avoiding the uncanny valley: robot appearance, personality and consistency of behavior in an attention-seeking home scenario for a robot companion. Auton. Robot. 24, 159–178 (2008)

  26. Złotowski, J.A., Sumioka, H., Nishio, S., Glas, D.F., Bartneck, C., Ishiguro, H.: Persistence of the uncanny valley: the influence of repeated interactions and a robot’s attitude on its perception. Front. Psychol. 6, 883 (2015)

  27. Žlajpah, L.: Simulation in robotics. Math. Comput. Simul. 79(4), 879–897 (2008)

    Article  MathSciNet  Google Scholar 

  28. Datteri, E., Schiaffonati, V.: Robotic simulations, simulations of robots. Mind. Mach. 29, 109–125 (2019)

    Article  Google Scholar 

  29. Nascimento, L.M.d., Neri, D.S., Ferreira, T.d.N., Pereira, F.d.A., Albuquerque, E.A.Y., Gonçalves, L.M.G., Sá, S.T.d.L.: Sbotics-gamified framework for educational robotics. Journal of Intelligent & Robotic Systems. 102(1), 17 (2021)

  30. Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., Tanaka, F.: Social robots for education: a review. Sci. Rob. 3(21) (2018)

  31. Toh, L.P.E., Causo, A., Tzuo, P.-W., Chen, I.-M., Yeo, S.H.: A review on the use of robots in education and young children. Journal of Educational Technology & Society. 19(2), 148–163 (2016)

    Google Scholar 

  32. Kim, J.-O., Kim, J.: Development and application of art based steam education program using educational robot. In: Robotic Systems: Concepts, Methodologies, Tools, and Applications. pp. 1675–1687. IGI Global, ??? (2020)

  33. Zheng, J., Chan, K., Gibson, I.: Virtual reality. Ieee. Potentials 17(2), 20–23 (1998)

    Article  Google Scholar 

  34. Chen, Y.-L., Hsu, C.-C.: Self-regulated mobile game-based english learning in a virtual reality environment. Computers & Education. 154, 103910 (2020)

    Article  Google Scholar 

  35. Shim, K.-C., Park, J.-S., Kim, H.-S., Kim, J.-H., Park, Y.-C., Ryu, H.-I.: Application of virtual reality technology in biology education. J. Biol. Educ. 37(2), 71–74 (2003)

    Article  Google Scholar 

  36. Bogusevschi, D., Muntean, C., Muntean, G.-M.: Teaching and learning physics using 3d virtual learning environment: a case study of combined virtual reality and virtual laboratory in secondary school. J. Comput. Math. Sci. Teach. 39(1), 5–18 (2020)

    Google Scholar 

  37. Shahab, M., Taheri, A., Mokhtari, M., Shariati, A., Heidari, R., Meghdari, A., Alemi, M.: Utilizing social virtual reality robot (v2r) for music education to children with high-functioning autism. Education and Information Technologies, 1–25 (2022)

  38. Ye, Y., You, H., Du, J.: Improved trust in human-robot collaboration with chatgpt (2023). arXiv:2304.12529

  39. Christmann, G.H.G.: Um Jogo de Interação Humano-Robô para o Ensino de Língua Inglesa. Bachelor Thesis, UFSM. Advisor: Prof. Dr. Rodrigo S. Guerra (2019). http://rodrigoguerra.com/wp-content/uploads/2019/02/tcc revisado.pdf

  40. Quigley, M., Faust, J., Foote, T., Leibs, J., et al.: Ros: an open-source robot operating system (2009)

  41. Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., Lange, D.: Unity: A General Platform for Intelligent Agents (2020)

  42. Jocher, G., Stoken, A., Borovec, J., NanoCode012, ChristopherSTAN, Changyu, L., Laughing, tkianai, Hogan, A., lorenzomammana, yxNONG, AlexWang1900, Diaconu, L., Marc, wanghaoyang0106, ml5ah, Doug, Ingham, F., Frederik, Guilhen, Hatovix, Poznanski, J., Fang, J., Yu, L., changyu98, Wang, M., Gupta, N., Akhtar, O., PetrDvoracek, Rai, P.: ultralytics/yolov5: v3.1 - Bug Fixes and Performance Improvements. Zenodo (2020). https://doi.org/10.5281/zenodo.4154370

  43. OpenAI: GPT-4 Technical Report (2023)

  44. Google: Google Cloud Speech-to-Text API. https://cloud.google.com/speech-to-text (2023)

  45. Google: Google Cloud Text-to-Speech API. https://cloud.google.com/text-to-speech (2023)

Download references

Acknowledgements

The authors would like to thank the VersusAI team. This work was partly supported by the University of Tsukuba.

Funding

Not applicable.

Author information

Authors and Affiliations

  1. Intelligent Robot Laboratory, University of Tsukuba, Tsukuba, Ibaraki, Japan

    Jair Augusto Bottega & Victor Augusto Kich

  2. Korea Advanced Institute of Science & Technology, Daejeon, South Korea

    Junior Costa de Jesus

  3. Federal University of Santa Maria, Santa Maria, RS, Brazil

    Raul Steinmetz & Daniel Fernando Tello Gamarra

  4. Federal University of Rio Grande, Rio Grande, RS, Brazil

    Alisson Henrique Kolling, Ricardo Bedin Grando & Rodrigo da Silva Guerra

  5. Technological University of Uruguay, Rivera, Uruguay

    Ricardo Bedin Grando

Authors
  1. Jair Augusto Bottega
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Victor Augusto Kich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junior Costa de Jesus
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raul Steinmetz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alisson Henrique Kolling
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ricardo Bedin Grando
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rodrigo da Silva Guerra
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Daniel Fernando Tello Gamarra
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Jair Augusto Bottega conceived the research, writing of the article, designed and program the experiments, collected and processed the test data. Victor Augusto Kich write the article, program the experiments, collected and processed the test data. Junior Costa de Jesus write the article, program the experiments, collected and processed the test data. Raul Steinmetz write the article, program the experiments, collected and processed the test data. Alisson Henrique Kolling write the article, program the experiments, collected and processed the test data. Ricardo Bedin Grando write the article, collected and processed the test data. Rodrigo da Silva Guerra discussion and conception of the main ideas of the article, provided valuable comments. Daniel Fernando Tello Gamarra conceived the research, writing of the article and discussion of the main ideas of the article.

Corresponding author

Correspondence to Jair Augusto Bottega.

Ethics declarations

Ethical Approval

The article has the approval of all the authors.

Consent to Participate

All the authors gave their consent to participate in this article

Consent to Publish

The authors gave their authorization for the publishing of this article.

Competing interest

There are not conflict of interest or competing interest

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bottega, J.A., Kich, V.A., Jesus, J.C.d. et al. Jubileo: An Immersive Simulation Framework for Social Robot Design. J Intell Robot Syst 109, 91 (2023). https://doi.org/10.1007/s10846-023-01991-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-023-01991-3

Keywords

Search

Navigation