Abstract
While code review is a critical component of modern software quality assurance, defects can still slip through the review process undetected. Previous research suggests that the main reason for this is a lack of reviewer awareness about the likelihood of defects in proposed changes; even experienced developers may struggle to evaluate the potential risks. If a change’s riskiness is underestimated, it may not receive adequate attention during review, potentially leading to defects being introduced into the codebase. In this paper, we investigate how risk assessment analytics can influence the level of awareness among developers regarding the potential risks associated with code changes; we also study how effective and efficient reviewers are at detecting defects during code review with the use of such analytics. We conduct a controlled experiment using Gherald, a risk assessment prototype tool that analyzes the riskiness of change sets based on historical data. Following a between-subjects experimental design, we assign participants to the treatment (i.e., with access to Gherald) or control group. All participants are asked to perform risk assessment and code review tasks. Through our experiment with 48 participants, we find that the use of Gherald is associated with statistically significant improvements (one-tailed, unpaired Mann-Whitney U test, \(\alpha \) = 0.05) in developer awareness of riskiness of code changes and code review effectiveness. Moreover, participants in the treatment group tend to identify the known defects more quickly than those in the control group; however, the difference between the two groups is not statistically significant. Our results lead us to conclude that the adoption of a risk assessment tool has a positive impact on code review practices, which provides valuable insights for practitioners seeking to enhance their code review process and highlights the importance for further research to explore more effective and practical risk assessment approaches.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
To facilitate reproduction and foment further research on the field, we make a replication package publicly available.(https://doi.org/10.5281/zenodo.7838135) We also publish Gherald as a Python Package on pip. The source code is available online on our public GitHub repository. (https://github.com/filipe-cogo/gherald)
Notes
This experiment was reviewed by and received ethics clearance from the University of Waterloo Research Ethics Committee (ORE #44022).
References
Ackerman AF, Fowler PJ, Ebenau RG (1984) In: Proc. of a symposium on software validation: inspection-testing-verification-alternatives (Elsevier North-Holland, Inc., USA), p. 13–40
Ackerman A, Buchwald L, Lewski F (1989) IEEE Software 6(3):31. https://doi.org/10.1109/52.28121
Alami A, Leavitt Cohn M, Wasowski A (2019) 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . https://doi.org/10.1109/icse.2019.00111
Ayewah N, Pugh W (2010) In: Proceedings of the 19th international symposium on software testing and analysis (Association for Computing Machinery, New York, USA, 2010), ISSTA ’10, pp 241–252. https://doi.org/10.1145/1831708.1831738
Bacchelli A, Bird C (2013) 35th International Conference on Software Engineering (ICSE) (2013). https://doi.org/10.1109/icse.2013.6606617
Ball T, Bounimova E, Cook B, Levin V, Lichtenberg J, McGarvey C, Ondrusek B, Rajamani SK, Ustuner A In: Proceedings of the 1st ACM SIGOPS/EuroSys European conference on computer systems 2006 (Association for Computing Machinery, New York, USA, 2006), EuroSys ’06, pp 73-85. https://doi.org/10.1145/1217935.1217943
Barnett M, Bird C, Brunet J, Lahiri SK (2015) In: 2015 IEEE/ACM 37th IEEE International conference on software engineering, vol 1, pp 134–144. https://doi.org/10.1109/ICSE.2015.35
Baum T, Schneider K, Bacchelli A (2019) Empirical Software Engineering 24(4):1762–1798. https://doi.org/10.1007/s10664-018-9676-8
Baum T, Liskin O, Niklas K, Schneider K (2016) In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering (Association for Computing Machinery, New York, USA, 2016), FSE 2016, pp 85–96
Baum T, Schneider K, Bacchelli A (2017) In: 2017 IEEE international conference on software maintenance and evolution (ICSME), pp 329–340. https://doi.org/10.1109/ICSME.2017.28
Baysal O, Kononenko O, Holmes R, Godfrey MW (2016) Empir Softw Eng 21(3):932. https://doi.org/10.1007/s10664-015-9366-8
Beller M, Bholanath R, McIntosh S, Zaidman A (2016) In: Proc. of the international conference on software analysis, evolution, and reengineering (SANER), pp 470–481
Biffl S (2000) In: Proceedings seventh Asia-Pacific software engeering conference. APSEC 2000, pp 136–145. https://doi.org/10.1109/APSEC.2000.896692
Bosu A, Greiler M, Bird C (2015) In: Proceedings of the 12th working conference on mining software repositories (IEEE Press), MSR ’15, pp 146–156
Cohen J (2013) Current directions in psychological science 1(3):98
Fagan ME (1976) IBM Syst J 15(3):182. https://doi.org/10.1147/sj.153.0182
Feitelson DG, Frachtenberg E, Beck KL (2013) IEEE Internet Comput 17(4):8. https://doi.org/10.1109/MIC.2013.25
Fregnan E (2023) Assessing review outcomes and cognitive factors to improve code review. Ph.D. thesis
Fregnan E, Braz L, D’Ambros M, Çalıklı G, Bacchelli A (2022) In: Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering (Association for Computing Machinery, New York, USA), ESEC/FSE 2022, pp 483-494. https://doi.org/10.1145/3540250.3549177
Fukushima T, Kamei Y, Mcintosh S, Yamashita K, Ubayashi N (2014) Proceedings of the 11th working conference on mining software repositories - MSR 2014 . https://doi.org/10.1145/2597073.2597075
Gasparini L, Fregnan E, Braz L, Baum T, Bacchelli A (2021) In: 2021 Working conference on software visualization (VISSOFT), pp 115–119. https://doi.org/10.1109/VISSOFT52517.2021.00022
Gonçalves PW, Fregnan E, Baum T, Schneider K, Bacchelli A (2022) Empirical Softw. Engg. 27(4). https://doi.org/10.1007/s10664-022-10123-8
Gousios G, Pinzger M, Deursen Av (2014) In: Proceedings of the 36th international conference on software engineering (Association for Computing Machinery, New York, USA), ICSE 2014, pp 345-355. https://doi.org/10.1145/2568225.2568260
Graves T, Karr A, Marron J, Siy H (2000) IEEE Transactions on Software Engineering 26(7):653. https://doi.org/10.1109/32.859533
Hampton J (2018) In: Laboratory Psychology (Psychology Press), pp 15–37
Heckman S, Williams L (2008) In: Proceedings of the second ACM-IEEE international symposium on empirical software engineering and measurement (Association for Computing Machinery, New York, USA), ESEM ’08, pp 41–50. https://doi.org/10.1145/1414004.1414013
Heckman S, Williams L (2009) In: 2009 International conference on software testing verification and validation, pp 161–170. https://doi.org/10.1109/ICST.2009.45
Herzig K, Zeller A (2013) In: 2013 10th working conference on mining software repositories (MSR), pp 121–130. https://doi.org/10.1109/MSR.2013.6624018
Hoang T, Dam HK, Kamei Y, Lo D, Ubayashi N (2019) 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). https://doi.org/10.1109/msr.2019.00016
Hoang T, Kang HJ, Lo D, Lawall J (2020) Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . https://doi.org/10.1145/3377811.3380361
Hovemeyer D, Pugh W (2004) SIGPLAN Not 39(12):92–106. https://doi.org/10.1145/1052883.1052895
Janes A, Mairegger M, Russo B (2018) In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering (Association for Computing Machinery, New York, USA, 2018), ASE 2018, pp 876-879. https://doi.org/10.1145/3238147.3240488
Johnson B, Song Y, Murphy-Hill E, Bowdidge R (2013) In: 2013 35th international conference on software engineering (ICSE), pp 672–681. https://doi.org/10.1109/ICSE.2013.6606613
Kamei Y, Shihab E, Adams B, Hassan AE, Mockus A, Sinha A, Ubayashi N (2013) IEEE Trans Softw Eng 39(6):757–773. https://doi.org/10.1109/tse.2012.70
Kamei Y, Matsumoto S, Monden A, Matsumoto Ki, Adams B, Hassan AE (2010) In: 2010 IEEE International conference on software maintenance, pp 1–10. https://doi.org/10.1109/ICSM.2010.5609530
Kendall MG (1948) Rank correlation methods (Griffin)
Khanan C, Luewichana W, Pruktharathikoon K, Jiarpakdee J, Tantithamthavorn C, Choetkiertikul M, Ragkhitwetsagul C, Sunetnanta T (2020) In: 2020 35th IEEE/ACM International conference on automated software engineering (ASE), pp 1336–1339
Kim S, Whitehead EJ, Zhang Y (2008) IEEE Transactions on Software Engineering 34(2):181–196. https://doi.org/10.1109/tse.2007.70773
Kim S, Ernst MD (2007) In: Proceedings of the the 6th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (Association for Computing Machinery, New York, USA), ESEC-FSE ’07, pp 45-54. https://doi.org/10.1145/1287624.1287633
Kim S, Zimmermann T, Pan K, Jr. Whitehead EJ (2006) In: 21st IEEE/ACM international conference on automated software engineering (ASE’06), pp 81–90. https://doi.org/10.1109/ASE.2006.23
Kollanus S, Koskinen J (2009) Open Softw Eng J 3(1):15–34. https://doi.org/10.2174/1874107x00903010015
Kononenko O, Baysal O, Godfrey MW (2016) In: Proceedings of the 38th international conference on software engineering (Association for Computing Machinery, New York, USA), ICSE ’16, pp 1028-1038. https://doi.org/10.1145/2884781.2884840
Kononenko O, Baysal O, Guerrouj L, Cao Y, Godfrey MW (2015) In: 2015 IEEE international conference on software maintenance and evolution (ICSME), pp 111–120. https://doi.org/10.1109/ICSM.2015.7332457
Lewis C, Lin Z, Sadowski C, Zhu X, Ou R, Whitehead EJ (2013) In: 2013 35th international conference on software engineering (ICSE), pp 372–381. https://doi.org/10.1109/ICSE.2013.6606583
Li PL, Herbsleb J, Shaw M, Robinson B (2006) In: Proceedings of the 28th international conference on software engineering (Association for Computing Machinery, New York, USA), ICSE ’06, pp 413-422. https://doi.org/10.1145/1134285.1134343
Matsumoto S, Kamei Y, Monden A, Matsumoto Ki, Nakamura M (2010) In: Proceedings of the 6th international conference on predictive models in software engineering (Association for Computing Machinery, New York, USA), PROMISE ’10. https://doi.org/10.1145/1868328.1868356
Mcintosh S, Kamei Y (2018) IEEE Transactions on Software Engineering 44(5):412–428. https://doi.org/10.1109/tse.2017.2693980
McIntosh S, Kamei Y, Adams B, Hassan AE (2014) In: Proc. of the working conference on mining software repositories (MSR), pp 192–201
Mockus A, Weiss DM (2000) Bell Labs Technical Journal 5(2):169. https://doi.org/10.1002/bltj.2229
Oosterwaal S, Deursen Av, Coelho R, Sawant AA, Bacchelli A (2016) In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering (Association for Computing Machinery, New York, USA), FSE 2016, pp 1038-1041. https://doi.org/10.1145/2950290.2983929
Paas F, Tuovinen J, Tabbers H, Van Gerven P (2003) Educational Psychologist - EDUC PSYCHOL 38:63. https://doi.org/10.1207/S15326985EP3801_8
Paixao M, Krinke J, Han D, Ragkhitwetsagul C, Harman M (2017) In: 2017 32nd IEEE/ACM international conference on automated software engineering (ASE) , pp 95–105. https://doi.org/10.1109/ASE.2017.8115622
Pascarella L, Spadini D, Palomba F, Bruntink M, Bacchelli A (2018) Proc ACM Hum-Comput Interact 2(CSCW). https://doi.org/10.1145/3274404
Pornprasit C, Tantithamthavorn CK (2021) 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). https://doi.org/10.1109/msr52588.2021.00049
Rezk C, Kamei Y, McIntosh S (2022) IEEE Transactions on Software Engineering 48(9):3297–3309
Romano J, Kromrey JD, Coraggio J, Skowronek J, Devine L (2006) In: Annual meeting of the Southern association for institutional research, pp 1–51
Rutar N, Almazan C, Foster J (2004) In: 15th international symposium on software reliability engineering, pp 245–256. https://doi.org/10.1109/ISSRE.2004.1
Sadowski C, Aftandilian E, Eagle A, Miller-Cushon L, Jaspan C (2018) Commun ACM 61(4):58–66. https://doi.org/10.1145/3188720
Sadowski C, Söderberg E, Church L, Sipko M, Bacchelli A (2018) In: 2018 IEEE/ACM 40th international conference on software engineering: software engineering in practice track (ICSE-SEIP), pp 181–190
Sadowski C, Van Gogh J, Jaspan C, Soderberg E, Winter C (2015) In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 1, pp 598–608. https://doi.org/10.1109/ICSE.2015.76
Shihab E, Hassan AE, Adams B, Jiang ZM (2012) In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering (Association for Computing Machinery, New York, USA), FSE ’12. https://doi.org/10.1145/2393596.2393670
Shull F, Seaman C (2008) IEEE Softw 25(1):88. https://doi.org/10.1109/MS.2008.7
Spadini D, Bacchelli A (2020) In: 2020 IEEE/ACM 17th international conference on mining software repositories (MSR), pp 528–532. https://doi.org/10.1145/3379597.3387455
Tang H, Lan T, Hao D, Zhang L (2015) In: Proceedings of the 7th Asia-Pacific symposium on internetware (Association for Computing Machinery, New York, USA), Internetware ’15, pp 43-51. https://doi.org/10.1145/2875913.2875922
Tan M, Tan L, Dara S, Mayeux C (2015) In: 2015 IEEE/ACM 37th IEEE international conference on software engineering vol 2, pp 99–108. https://doi.org/10.1109/ICSE.2015.139
Tao Y, Dang Y, Xie T, Zhang D, Kim S (2012) In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering (Association for Computing Machinery, New York, USA), FSE ’12. https://doi.org/10.1145/2393596.2393656
Tao Y, Kim S (2015) In: 2015 IEEE/ACM 12th working conference on mining software repositories, pp 180–190. https://doi.org/10.1109/MSR.2015.24
Thongtanunam P, McIntosh S, Hassan AE, Iida H (2015) In: 2015 IEEE/ACM 12th working conference on mining software repositories pp 168–179. https://doi.org/10.1109/MSR.2015.23
Tomassi DA, Rubio-González C (2021) In: 2021 36th IEEE/ACM international conference on automated software engineering (ASE), pp 292–303. https://doi.org/10.1109/ASE51524.2021.9678535
Tymchuk Y, Mocci A, Lanza M (2015) In: 2015 IEEE 22nd International conference on software analysis, evolution, and reengineering (SANER) , pp 151–160. https://doi.org/10.1109/SANER.2015.7081825
Undefinedliwerski J, Zimmermann T, Zeller A (2005) SIGSOFT Softw. Eng. Notes 30(4):1–5. https://doi.org/10.1145/1082983.1083147
Undefinedliwerski J, Zimmermann T, Zeller A (2005) (Association for Computing Machinery, New York, USA), MSR ’05, p 1-5. https://doi.org/10.1145/1083142.1083147
Uquillas Gómez V, Ducasse S, D’Hondt T (2015) Sci Comput Program 98(P3):376–393. https://doi.org/10.1016/j.scico.2013.08.002
Wan Z, Xia X, Hassan AE, Lo D, Yin J, Yang X (2020) IEEE Transactions on Software Engineering 46(11):1241. https://doi.org/10.1109/TSE.2018.2877678
Zhang T, Song M, Pinedo J, Kim M (2015) In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 1, pp 111–122. https://doi.org/10.1109/ICSE.2015.33
Zimmermann T, Premraj R, Zeller A (2007) In: Third International workshop on predictor models in software engineering (PROMISE’07: ICSE Workshops 2007) , p 9 https://doi.org/10.1109/PROMISE.2007.10
Acknowledgements
The findings and opinions expressed in this paper are those of the authors and do not necessarily represent or reflect those of Huawei and/or its subsidiaries and affiliates. Moreover, our results do not in any way reflect the quality of Huawei’s products.
Funding
This study was funded by Waterloo-Huawei Joint Innovation Lab.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declared that they have no conflict of interest.
Ethics Approval
This study was reviewed by and received ethics clearance from the University of Waterloo Research Ethics Committee (ORE #44022).
Additional information
Communicated by: Fabio Palomba.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yu, X., Cogo, F.R., McIntosh, S. et al. Studying the impact of risk assessment analytics on risk awareness and code review performance. Empir Software Eng 29, 46 (2024). https://doi.org/10.1007/s10664-024-10443-x
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-024-10443-x