Skip to main content
SpringerLink
Log in

What can we learn from quality assurance badges in open-source software?

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

In the development of open-source software (OSS), many developers use badges to give an overview of the software and share some key features/metrics conveniently. Among various badges, quality assurance (QA) badges make up a large proportion and are the most prevalent because QA is of vital importance in software development, and ineffective QA may lead to anomalies or defects. In this paper, we focus on QA badges in open-source projects, which present quality assurance information directly and instantly, and aim to produce some interesting findings and provide practical implications. We collect and analyze 100000 projects written in popular programming languages from GitHub and conduct a comprehensive empirical study both inside and outside QA badges. Inside QA badges, we build a category classification for all QA badges based on the properties they focus on, which shows the types of QA badges developers use. Then, we analyze the frequency of the properties that QA badges focus on, and property combinations, too, which present their use status. We find that QA badges focus on various properties while developers give different preferences to different properties. The use status also differs between different programming languages. For example, projects written in C focus on Security to a great extent. Our findings also provide implications for developers and badge providers. Outside QA badges, we conduct a correlation analysis between QA badges and some software metrics that have potential relationships with code quality, contribution quality, and popularity. We find that QA badges have statistically significant correlations with various software metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Zhao L, Elbaum S. Quality assurance under the open source development model. J Syst Softw, 2003, 66: 65–75

    Article  Google Scholar 

  2. Ma L, Xu F J F, Xue M, et al. Secure deep learning engineering: a software quality assurance perspective. 2018. ArXiv:1810.04538

  3. Bahamdain S S. Open source software (OSS) quality assurance: a survey paper. Procedia Comput Sci, 2015, 56: 459–464

    Article  Google Scholar 

  4. Trockman A, Zhou S, Kästner C, et al. Adding sparkle to social coding: an empirical study of repository badges in the npm ecosystem. In: Proceedings of the 40th International Conference on Software Engineering, 2018. 511–522

  5. Walkinshaw N. Software Quality Assurance. Berlin: Springer, 2017

    Book  Google Scholar 

  6. Laporte C Y, April A. Software Quality Assurance. Hoboken: John Wiley & Sons, 2018

    Google Scholar 

  7. Perera P, Silva R, Perera I. Improve software quality through practicing devops. In: Proceedings of the 17th International Conference on Advances in ICT for Emerging Regions (ICTer), 2017. 1–6

  8. Basu A. Software Quality Assurance, Testing and Metrics. New Delhi: PHI Learning Pvt. Ltd., 2015

    Google Scholar 

  9. Wong W E. Special section on software quality assurance: research and practice. IEEE Trans Rel, 2016, 65: 3

    Article  Google Scholar 

  10. Legay D, Decan A, Mens T. On the usage of badges in open source packages on GitHub. In: Proceedings of the 18th Belgium-Netherlands Software Evolution Workshop, Brussels, 2019

  11. Parsons V L. Stratified Sampling. Hoboken: John Wiley & Sons, 2014

    Book  Google Scholar 

  12. Avelino G, Constantinou E, Valente M T, et al. On the abandonment and survival of open source projects: an empirical investigation. In: Proceedings of ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2019. 1–12

  13. Steinmacher I, Pinto G, Wiese I S, et al. Almost there: a study on quasi-contributors in open-source software projects. In: Proceedings of the 40th International Conference on Software Engineering (ICSE), 2018. 256–266

  14. Seaman C B. Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng, 1999, 25: 557–572

    Article  Google Scholar 

  15. Landis J R, Koch G G. The measurement of observer agreement for categorical data. Biometrics, 1977, 33: 159

    Article  Google Scholar 

  16. Zhang J M, Li F, Hao D, et al. A study of bug resolution characteristics in popular programming languages. IEEE Trans Softw Eng, 2021, 47: 2684–2697

    Article  Google Scholar 

  17. Berger E D, Hollenbeck C, Maj P, et al. On the impact of programming languages on code quality. ACM Trans Program Lang Syst, 2019, 41: 1–24

    Article  Google Scholar 

  18. Borges H, Valente M T. What’s in a GitHub star? Understanding repository starring practices in a social coding platform. J Syst Softw, 2018, 146: 112–129

    Article  Google Scholar 

  19. Ortu M, Marchesi M, Tonelli R. Empirical analysis of affect of merged issues on GitHub. In: Proceedings of the 4th International Workshop on Emotion Awareness in Software Engineering (SEmotion), 2019. 46–48

  20. Ray B, Posnett D, Filkov V, et al. A large scale study of programming languages and code quality in GitHub. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014. 155–165

  21. ISO Central Secretary. Systems and software engineering — systems and software quality requirements and evaluation (square) — system and software quality models. Standard ISO/IEC 25010:2011. https://www.iso.org/standard/35733.html

  22. Fischer-Nielsen A, Fu Z, Su T, et al. The forgotten case of the dependency bugs: on the example of the robot operating system. In: Proceedings of the 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2020. 21–30

  23. Asadollah S A, Sundmark D, Eldh S, et al. Concurrency bugs in open source software: a case study. J Int Serv Appl, 2017, 8: 4

    Article  Google Scholar 

  24. Jin G, Song L, Shi X, et al. Understanding and detecting real-world performance bugs. SIGPLAN Not, 2012, 47: 77–88

    Article  Google Scholar 

  25. Chilenski J J, Miller S P. Applicability of modified condition/decision coverage to software testing. Softw Eng J UK, 1994, 9: 193–200

    Article  Google Scholar 

  26. Zhao Y, Feng Y, Wang Y, et al. Quality assessment of crowdsourced test cases. Sci China Inf Sci, 2020, 63: 190102

    Article  Google Scholar 

  27. Stasinopoulos D M, Rigby R A. Generalized additive models for location scale and shape (GAMLSS) in R. J Stat Soft, 2007, 23: 1–46

    Article  Google Scholar 

  28. Hastie T J, Tibshirani R J. Generalized additive models. Routledge, 2017. https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781118445112.stat03141

  29. Mansfield E R, Helms B P. Detecting multicollinearity. Am Statist, 1982, 36: 158–160

    Google Scholar 

  30. Liu W, Chen B, Peng X, et al. Identifying change patterns of API misuses from code changes. Sci China Inf Sci, 2021, 64: 132101

    Article  Google Scholar 

  31. Rai A, Song H, Troutt M. Software quality assurance: an analytical survey and research prioritization. J Syst Softw, 1998, 40: 67–83

    Article  Google Scholar 

  32. Holck J, Jørgensen N. Continuous integration and quality assurance: a case study of two open source projects. Australas J Inform Syst, 2003, 11: 1

    Google Scholar 

  33. Khanjani A, Sulaiman R. The process of quality assurance under open source software development. In: Proceedings of IEEE Symposium on Computers and Informatics, 2011. 548–552

  34. Axelsson J, Skoglund M. Quality assurance in software ecosystems: a systematic literature mapping and research agenda. J Syst Softw, 2016, 114: 69–81

    Article  Google Scholar 

  35. Hassan M U, Mubashir M, Shabir M A, et al. Software quality assurance techniques: a review. Int J Inform Bus Manag, 2018, 10: 214–221

    Google Scholar 

  36. Felderer M, Ramler R. Quality assurance for AI-based systems: overview and challenges (introduction to interactive session). In: Proceedings of International Conference on Software Quality, 2021. 33–42

  37. Bird C, Gourley A, Devanbu P, et al. Mining email social networks. In: Proceedings of International Workshop on Mining Software Repositories, 2006. 137–143

  38. Guzzi A, Bacchelli A, Lanza M, et al. Communication in open source software development mailing lists. In: Proceedings of the 10th Working Conference on Mining Software Repositories (MSR), 2013. 277–286

  39. Joblin M, Apel S, Hunsen C, et al. Classifying developers into core and peripheral: an empirical study on count and network metrics. In: Proceedings of the 39th International Conference on Software Engineering (ICSE), 2017. 164–174

  40. Singer L, Filho F F, Storey M A. Software engineering at the speed of light: how developers stay current using twitter. In: Proceedings of the 36th International Conference on Software Engineering, 2014. 211–221

  41. Bogart C, Kästner C, Herbsleb J, et al. How to break an API: cost negotiation and community values in three software ecosystems. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2016. 109–120

  42. Decan A, Mens T, Claes M. An empirical comparison of dependency issues in OSS packaging ecosystems. In: Proceedings of the 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2017. 2–12

  43. Raemaekers S, van Deursen A, Visser J. Semantic versioning versus breaking changes: a study of the maven repository. In: Proceedings of the 14th International Working Conference on Source Code Analysis and Manipulation, 2014. 215–224

  44. Bavota G, Canfora G, Di Penta M, et al. How the Apache community upgrades dependencies: an evolutionary study. Empir Softw Eng, 2015, 20: 1275–1317

    Article  Google Scholar 

  45. Cox J, Bouwers E, van Eekelen M, et al. Measuring dependency freshness in software systems. In: Proceedings of the 37th IEEE International Conference on Software Engineering, 2015. 109–118

  46. Kula R G, German D M, Ouni A, et al. Do developers update their library dependencies? Empir Softw Eng, 2018, 23: 384–417

    Article  Google Scholar 

  47. Mirhosseini S, Parnin C. Can automated pull requests encourage software developers to upgrade out-of-date dependencies? In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), 2017. 84–94

  48. Zampetti F, Scalabrino S, Oliveto R, et al. How open source projects use static code analysis tools in continuous integration pipelines. In: Proceedings of the 14th International Conference on Mining Software Repositories (MSR), 2017. 334–344

  49. Hilton M, Tunnell T, Huang K, et al. Usage, costs, and benefits of continuous integration in open-source projects. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), 2016. 426–437

  50. Vasilescu B, Yu Y, Wang H, et al. Quality and productivity outcomes relating to continuous integration in GitHub. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering, 2015. 805–816

  51. Zhao Y, Serebrenik A, Zhou Y, et al. The impact of continuous integration on other software development practices: a large-scale empirical study. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), 2017. 60–71

Download references

Acknowledgements

This work was partly supported by National Natural Science Foundation of China (Grant No. 61872008).

Author information

Authors and Affiliations

  1. MoE Key Lab of HCST, School of Computer Science, Peking University, Beijing, 100871, China

    Feng Li, Jinhao Dong, Yang Li, Xuanzhi Wang, Dan Hao & Lu Zhang

  2. Department of Computer Science, Purdue University, West Lafayette, IN, 47907, USA

    Yiling Lou

  3. School of Computer Science & Engineering, Beihang University (BUAA), Beijing, 100191, China

    Xin Tan

  4. Department of Computer Science, University College London (UCL), London, WC1E 6BT, UK

    Zhenpeng Chen

Authors
  1. Feng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yiling Lou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhenpeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jinhao Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xuanzhi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Dan Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Lu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dan Hao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, F., Lou, Y., Tan, X. et al. What can we learn from quality assurance badges in open-source software?. Sci. China Inf. Sci. 67, 142103 (2024). https://doi.org/10.1007/s11432-022-3611-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-022-3611-3

Keywords

Search

Navigation