Skip to main content
SpringerLink
Log in

Effective Software Mutation-Test Using Program Instructions Classification

  • Published:
Journal of Electronic Testing Aims and scope Submit manuscript

Abstract

The quantity of bugs that a software test-data finds determines its effectiveness. A useful technique for assessing the efficacy of a test set is mutation testing. The primary issues with the mutation test are cost and time requirements. Close to 40% of the injected bugs in the mutation test are effect-less (equivalent). Reducing the number of generated total mutants by decreasing equivalent mutants and reducing the execution time of the mutation test are the main objectives of this study. An error-propagation aware mutation test approach has been suggested in this research. Three steps make up the process. To find a collection of instruction-level characteristics effective on the error propagation rate, the data and instructions of the input program were evaluated in the first step. Utilizing supervised machine learning techniques, an instruction classifier was developed using the prepared dataset in the second step. After classifying the program instructions automatically by the created classifier, the mutation test is performed only on the identified error-propagating instructions; the identified non-error-propagating instructions are avoided to mutate in the proposed mutation testing. The conducted experiments on the set of standard benchmark programs indicate that the proposed method causes about 19% reduction in the number of generated mutants. Furthermore, the proposed method causes a 32.24% reduction in the live mutants. It should be noted that the proposed method eliminated only the affectless mutants. The key technical benefit of the suggested solution is that mutation of the instructions that don't propagate errors is avoided. These findings can lead to a performance improvement in the existing mutation-test methods and tools.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data Availability

The data related to the current study is available in the google. Drive and can be freely accessed by the following link: https://drive.google.com/drive/folders/1QqFFXYpgintZeSgQi7uzytBDPdkNpuBD?usp=sharing

References

  1. Acree A, Budd T, DeMillo R, Lipton R, Sayward F (1980) Mutation Analysis, School of Information and Computer Science. Georgia Inst Technol

  2. Arasteh B (2018) Software Fault-Prediction using Combination of Neural Network and Naive Bayes Algorithm. J Netw Technol 9(3):94–101. https://doi.org/10.6025/jnt/2018/9/3/94-101

    Article  Google Scholar 

  3. Arasteh B (2019) ReDup: A software-based method for detecting soft-error using data analysis. In Comput Electr Eng 78(9):89–107

  4. Arasteh B, Fatolahzadeh A, Kiani F (2022) Savalan: Multi objective and homogeneous method for software modules clustering. J Softw Evol 34(1):2022. https://doi.org/10.1002/smr.2408

    Article  Google Scholar 

  5. Arasteh B, Miremadi SG, Rahmani AM (2014) Developing Inherently Resilient Software Against Soft-Errors Based on Algorithm Level Inherent Features. J Electron Test 30(9):193–212. https://doi.org/10.1007/s10836-014-5438-8

    Article  Google Scholar 

  6. Arasteh B, Pirahesh S, Zakeri A, Arasteh B (2014) Highly Available and Dependable E-learning Services Using Grid System. Procedia Soc Behav Sci 143(2014):471–476. https://doi.org/10.1016/j.sbspro.2014.07.519

    Article  Google Scholar 

  7. Arasteh B, Razieh S, Keyvan A (2020) ARAZ: A software modules clustering method using the combination of particle swarm optimization and genetic algorithms. Intell Decis Technol 14(4):449–462. https://doi.org/10.3233/idt-200070

    Article  Google Scholar 

  8. Barbosa EF, Maldonado JC, Vincenzi AMR (2001) Toward the determination of sufficient mutant operators for C. Softw Test Verif Reliab 11(2):113–136

    Article  Google Scholar 

  9. Binu Rajan MR, Vinod Chandra SS (2017) ABC Metaheuristic Based Optimized Adaptation Planning Logic for Decision Making Intelligent Agents in Self Adaptive Software System. Lect Notes Comput Sci 10387:496–504

    Article  Google Scholar 

  10. Bishop CM (1995) Neural networks for pattern recognition. Clarendon Press, Oxford, England. Oxford University Press, Inc. New York, NY, USA ©1995 ISBN: 0198538642. Available at: http://cs.du.edu/~mitchell/mario_books/Neural_Networks_for_Pattern_Recognition_-_Christopher_Bishop.pdf

  11. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  Google Scholar 

  12. Budd TA (1980) Yale University, Mutation Analysis of Program Test Data

  13. Bouyer A, Arasteh B, Movaghar A (2007) A new hybrid model using case-based reasoning and decision tree methods for improving speedup and accuracy. IADIS International conference of applied computing

    Google Scholar 

  14. Chandra SV, Sankar SS, Anand HS (2022) Smell detection agent optimization approach to path generation in automated software testing. J Electron Test 38(6):623–636. https://doi.org/10.1007/s10836-022-06033-8

    Article  Google Scholar 

  15. Dang X, Gong D, Yao X, Tian T, Liu H (2022) Enhancement of Mutation Testing via Fuzzy Clustering and Multi-Population Genetic Algorithm. IEEE Trans Softw Eng 48(6):2141–2156

  16. Delgado-Pérez P, Medina-Bulo I (2018) Search-based mutant selection for efficient test suite improvement: Evaluation and results. Inf Softw Technol 104(2018):130–143

    Article  Google Scholar 

  17. DeMillo RA, Spafford EH (1986) The Mothra software testing environment, presented at The 11th NASA Softw Eng Lab Workshop Goddard Space Center

  18. Deng L, Offutt J, Ammann P, Mirzaei N (2017) Mutation operators for testing Android apps. Inf Softw Technol 81:154–168

    Article  Google Scholar 

  19. Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Ann Stat 29(5):1189–1232

    Article  MathSciNet  Google Scholar 

  20. Friedman JH (2002) Stochastic gradient boosting. Comput Stat Data Anal 38(4):367–378

    Article  MathSciNet  Google Scholar 

  21. Ghaemi A, Arasteh B (2020) SFLA-based heuristic method to generate software structural test data. J Softw Evol Proc 32:e2228. https://doi.org/10.1002/smr.2228

  22. Gheyi R, Ribeiro M, Souza B, Guimarães M, Fernandes L, d’Amorim M, Alves V, Teixeira L, Fonseca B (2021) Identifying method-level mutation subsumption relations using Z3. Inf Softw Technol 132:106496

  23. Good IJ (1951) Probability and the Weighing of Evidence, Philosophy Volume 26, Issue 97, 1951. Published by Charles Griffin and Company, London 1950. Copyright © The Royal Institute of Philosophy 1951, pp. 163–164. https://doi.org/10.1017/S0031819100026863. Available at Royal Institute of Philosophy website: https://www.cambridge.org/core/journals/philosophy/article/probability-and-the-weighing-of-evidence-by-goodi-j-london-charles-griffin-and-company-1950-pp-viii-119-price-16s/7D911224F3713FDCFD1451BBB2982442

  24. Hosseini S, Arasteh B, Isazadeh A, Mohsenzadeh M, Mirzarezaee M (2021) An error-propagation aware method to reduce the software mutation cost using genetic algorithm. Data Technologies and Applications 55(1):118–148. https://doi.org/10.1108/DTA-03-2020-0073

    Article  Google Scholar 

  25. Howden WE (1982) “Weak mutation testing and completeness of test sets.” IEEE Trans Softw Eng 8(4):371–379

    Article  Google Scholar 

  26. Irvine SA, Pavlinic T, Trigg L, Cleary JG, Inglis S, Utting M (2007) Jumble Java byte code to measure the effectiveness of unit tests. Proceedings of the Test: Acad Ind Proc Pract Res Tech - MUTAT (TAICPART-MUTATION ’07). IEEE Computer Society, USA, pp 169–175. https://doi.org/10.1109/taic.part.2007.38

    Chapter  Google Scholar 

  27. King KN, Offutt AJ (1991) A Fortran language system for mutation-based software testing. Softw: Pract Exper 21(7):685–718

  28. Kintis M, Papadakis M, Malevris N (2010) Evaluating mutation testing alternatives: a collateral experiment. Proc 17th Asia-Pacific Softw Eng Proc (APSEC)

  29. Kurtz B, Ammann P, Delamaro M, Offutt J, Deng L (2014) Mutant subsumption graphs. 2014 IEEE Seventh Int Proc Softw Test Verif Valid Workshops (ICSTW)

  30. Kurtz B, Ammann P, Offutt J (2015) Static analysis of mutant subsumption. IEEE Eighth Int Proc Softw Test Verif Valid Workshops (ICSTW)

  31. Ma YS, Offutt J, Kwon YR (2006) MuJava: A Mutation System for Java. In 28th Int Proc Softw Eng (ICSE ’06)

  32. Malevris N, Yates D (2006) The collateral coverage of data flow criteria when branch testing. Inf Softw Technol 48(8):676–686

    Article  Google Scholar 

  33. McCulloch WS, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 5(4):115–133

    Article  MathSciNet  Google Scholar 

  34. Moore I (2001) Jester - a JUnit test tester

  35. Mresa ES, Bottaci L (1999) Efficiency of mutation operators and selective mutation strategies: an empirical study. Softw Test Verif Reliabil 9(4):205–232

    Article  Google Scholar 

  36. Nilsson NJ (1965) Learning machines. New York: McGraw-Hill.Published in: J IEEE Trans Inf Theory 12(3):407, 1966. Inf Theory 12(3), 1966. https://doi.org/10.1109/TIT.1966.1053912. Available at ACM digital library website: http://dl.acm.org/citation.cfm?id=2267404

  37. Offutt AJ, Lee SD (1994) An empirical evaluation of weak mutation. IEEE Trans Softw Eng 20(5):337–344

    Article  Google Scholar 

  38. Offutt AJ, Rothermel G, Zapf C (1993) An experimental evaluation of selective mutation. Proceedings of the 15th Int Proc Softw Eng, ICSE ’93, IEEE Computer Society Press, Los Alamitos, CA

  39. Offutt J, Lee A, Rothermel G, Untch RH, Zapf C (1996) An Experimental Determination of Sufficient Mutant Operators. ACM Trans Softw Eng Methodol 5:99–118

    Article  Google Scholar 

  40. Offutt J, Ma Y-S, Kwon YR (2006) MuJava: an automated class mutation system. Softw Test Verif Reliab 15:97–133

    Google Scholar 

  41. Osisanwo FY, Akinsola JET, Awodele O, Hinmikaiye JO, Olakanmi O, Akinjobi J (2017) Supervised machine learning algorithms: classification and comparison. Int J Comput Trends Technol (IJCTT) 48(3):128–138

    Article  Google Scholar 

  42. Papadakis M, Malevris N (2010) An empirical evaluation of the first and second order mutation testing strategies. Third Int Proc Softw Test Verif Valid Workshops (ICSTW)

  43. Quinlan JR (1986) Induction of decision trees. Mach Learn 1(1):81–106

    Article  Google Scholar 

  44. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back propagating errors. Nature 323(6088):533

    Article  Google Scholar 

  45. Sharma B, Girdhar I, Taneja M, Basia P, Vadla S, Srivastava PR (2011) Software coverage: A testing approach through ant colony optimization. Lecture notes in computer science, vol 7076. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-27172-4_73

    Chapter  Google Scholar 

  46. Sommerville I (2018) Software engineering, 10th edn. Pearson India (ISBN: 9332582696)

    Google Scholar 

  47. Sridharan M, Siami-Namin A (2010) Prioritizing mutation operators based on importance sampling. In: Proceedings of the IEEE 21th Int Sympo- sium Softw Reliab Eng (ISSRE). IEEE, San Jose, CA, USA, pp. 378–387

  48. Taiwo OA (2010) Types of Machine Learning Algorithms, New Advances in Machine Learning, Yagang Zhang (Ed.), ISBN: 978-953-307-034-6, InTech, University of Portsmouth United Kingdom. Pp 3 – 31. Available at InTech open website: http://www.intechopen.com/books/new-advances-inmachine-learning/types-of-machine-learning-algorithms

  49. Uddin S, Khan A, Hossain ME, Moni MA (2019) Comparing different supervised machine learning algorithms for disease prediction. BMC Med Inform Decis Mak 19(1):1–16. https://doi.org/10.1186/s12911-019-1004-8

    Article  Google Scholar 

  50. Varga A (2001) Discrete event simulation system. In Proc Eur Simul Multiconference (ESM’2001) (pp. 1–7)

  51. Wei C, Yao X, Gong D, Liu H (2021) Spectral clustering based mutant reduction for mutation testing. Inf Softw Technol 132:106502

  52. Wong WE (1993) On mutation and data flow. Ph.D. dissertation, Purdue University

  53. Woodward M, Halewood K (1998) From weak to strong, dead or alive? An analysis of some mutation testing issues. Proc Second Workshop Softw Test Verif Anal

  54. Yao X, Zhang G, Pan F, Gong D, Wei C (2022) Orderly Generation of Test Data via Sorting Mutant Branches Based on Their Dominance Degrees for Weak Mutation Testing. In IEEE Trans Softw Eng 48(4):1169–1184

  55. Zadahmad M, Arasteh B, YousefzadehFard P (2011) A pattern-oriented and web-based architecture to support mobile learning software development. Procedia Soc Behav Sci 28(2011):194–199. https://doi.org/10.1016/j.sbspro.2011.11.037

    Article  Google Scholar 

  56. Zhang L, Gligoric M, Marinov D, Khurshid S (2013) Operator-based and random mutant selection: better together. Autom Softw Eng (ASE),. IEEE/ACM 28th International Proc

  57. Zhang L, Hou S-S, Hu J-J, Xie T, Mei H (2010) Is operator-based mutant selection superior to random mutant selection? Proceedings of the 32nd ACM/IEEE Int Proc Softw Eng, 2010

Download references

Funding

The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran

    Zeinab Asghari & Abbas Koochari

  2. Department of Computer Engineering, Tabriz Branch, Islamic Azad University, Tabriz, Iran

    Bahman Arasteh

  3. Department of Software Engineering, Faculty of Engineering and Natural Science, Istinye University, Istanbul, Turkey

    Bahman Arasteh

Authors
  1. Zeinab Asghari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bahman Arasteh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abbas Koochari
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to this study.

Corresponding author

Correspondence to Bahman Arasteh.

Ethics declarations

Conflict of Interest

The authors have no relevant financial or non-financial conflict interests.

Additional information

Responsible Editor: Y. K. Malaiya.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Asghari, Z., Arasteh, B. & Koochari, A. Effective Software Mutation-Test Using Program Instructions Classification. J Electron Test 39, 631–657 (2023). https://doi.org/10.1007/s10836-023-06089-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10836-023-06089-0

Keywords

Search

Navigation