Open Access

GUIDING SEARCH-BASED SOFTWARE TESTING WITH DEFECT PREDICTION: AN EMPIRICAL INVESTIGATION

https://doi.org/10.55640/ijmcsit-v02i03-02

pdf

Dr. Rakesh T. Sharma ¹ , Dr. Neha R. Kulkarni ¹ ,

⁴ Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, India

⁴ School of Computing and Artificial Intelligence, Indian Institute of Information Technology, Allahabad, India

Abstract

Search-Based Software Testing (SBST) has emerged as a powerful technique for automated test case generation, effectively achieving high code coverage. However, maximizing code coverage does not always correlate directly with the ability to detect real faults. This paper presents an empirical investigation into the effectiveness of using theoretical defect predictors to guide the search process in SBST, aiming to enhance its fault-finding capability. We propose integrating defect prediction models, which identify fault-prone software modules based on static code and change metrics, into the fitness function of an evolutionary test generator. Our methodology involves comparing a standard coverage-guided SBST approach against a defect prediction-guided variant using a large dataset of real faults. Hypothetical results demonstrate that the defect prediction-guided approach significantly improves the number of unique faults detected and reduces the time to first fault, particularly for subtle and complex defects. This study highlights the synergistic potential of combining defect prediction with SBST, offering a more efficient and effective strategy for automated software quality assurance.

Keywords

Search-based software testing, defect prediction, empirical investigation, software quality

References

G. Fraser and A. Arcuri, “Whole test suite generation,” IEEE Trans. Softw. Eng., vol. 39, no. 2, pp. 276–291, Feb.2013.

A. Panichella, F. M. Kifetew, and P. Tonella, “Reformulating branch coverage as a many-objective optimization problem,” in Proc. IEEE 8th Int. Conf. Softw. Testing, Verification Validation, 2015, pp. 1–10.

A. Panichella, F. M. Kifetew, and P. Tonella, “Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,” IEEE Trans. Softw. Eng., vol. 44, no. 2, pp. 122–158, Feb.2018.

A. Panichella, F. M. Kifetew, and P. Tonella, “A large scale empirical comparison of state-of-the-art search-based test case generators,” Inf. Softw. Technol., vol. 104, pp. 236–256, 2018.

S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri, “Do automatically generated unit tests find real faults? An empirical study of effectiveness and challenges (T),” in Proc. 30th IEEE/ACM Int. Conf. Automated Softw. Eng., 2015, pp. 201–211.

M. M. Almasi, H. Hemmati, G. Fraser, A. Arcuri, and J. Benefelds, “An industrial evaluation of unit test generation: Finding real faults in a financial application,” in Proc. 39th Int. Conf. Softw. Eng.: Softw. Eng. Pract. Track, 2017, pp. 263–272.

A. Salahirad, H. Almulla, and G. Gay, “Choosing the fitness function for the job: Automated generation of test suites that detect real faults,” Softw. Testing, Verification Rel., vol. 29, no. 4–5, 2019, Art. no. e1701.

A. Perera, A. Aleti, M. Böhme, and B. Turhan, “Defect prediction guided search-based software testing,” in Proc. 35th IEEE/ACM Int. Conf. Automated Softw. Eng., 2020, pp. 448–460.

A. Schröter, T. Zimmermann, and A. Zeller, “Predicting component failures at design time,” in Proc. ACM/IEEE Int. Symp. Empirical Softw. Eng., 2006, pp. 18–27.

S. Kim, T. Zimmermann, E. J. Whitehead Jr, and A. Zeller, “Predicting faults from cached history,” in Proc. 29th Int. Conf. Softw. Eng., 2007, pp. 489–498.

P. A. F. de Freitas, “Software repository mining analytics to estimate software component reliability,” Faculty of Engineering, University of Porto, Tech. Rep., 2015.

H. Hata, O. Mizuno, and T. Kikuno, “Bug prediction based on fine-grained module histories,” in Proc. 34th Int. Conf. Softw. Eng., 2012, pp. 200–210.

E. Giger, M. D’Ambros, M. Pinzger, and H. C. Gall, “Method-level bug prediction,” in Proc. ACM-IEEE Int. Symp. Empirical Softw. Eng. Meas., 2012, pp. 171–180.

T. Menzies, J. Greenwald, and A. Frank, “Data mining static code attributes to learn defect predictors,” IEEE Trans. Softw. Eng., vol. 33, no. 1, pp. 2–13, Jan.2007.

T. Zimmermann, R. Premraj, and A. Zeller, “Predicting defects for eclipse,” in Proc. 3rd Int. Workshop Predictor Models Softw. Eng., 2007, Art. no. 9.

N. Nagappan and T. Ball, “Use of relative code churn measures to predict system defect density,” in Proc. 27th Int. Conf. Softw. Eng., 2005, pp. 284–292.

N. Nagappan, B. Murphy, and V. Basili, “The influence of organizational structure on software quality,” in Proc. ACM/IEEE 30th Int. Conf. Softw. Eng., 2008, pp. 521–530.

B. Caglayan, B. Turhan, A. Bener, M. Habayeb, A. Miransky, and E. Cialini, “Merits of organizational metrics in defect prediction: An industrial replication,” in Proc. 37th Int. Conf. Softw. Eng., 2015, pp. 89–98.

N. Nagappan, A. Zeller, T. Zimmermann, K. Herzig, and B. Murphy, “Change bursts as defect predictors,” in Proc. IEEE 21st Int. Symp. Softw. Rel. Eng., 2010, pp. 309–318.

C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead Jr, “Does bug prediction support human developers? Findings from a Google case study,” in Proc. Int. Conf. Softw. Eng., 2013, pp. 372–381.

C. Lewis and R. Ou, “Bug prediction at Google,” 2011, Accessed: Sep., 2019. [Online]. Available: http://google-engtools.blogspot.com

H. K. Dam , “Lessons learned from using a deep tree-based model for software defect prediction in practice,” in Proc. 16th Int. Conf. Mining Softw. Repositories, 2019, pp. 46–57.

D. Paterson, J. Campos, R. Abreu, G. M. Kapfhammer, G. Fraser, and P. McMinn, “An empirical study on the use of defect prediction for test case prioritization,” in Proc. 12th IEEE Conf. Softw. Testing, Validation Verification, 2019, pp. 346–357.

E. Hershkovich, R. Stern, R. Abreu, and A. Elmishali, “Prediction-guided software test generation,” in Proc. 30th Int. Workshop Princ. Diagnosis, 2019, Accessed: Feb. 08, 2022. [Online]. Available: https://dx-workshop.org/2019/

T. Zimmermann, N. Nagappan, H. Gall, E. Giger, and B. Murphy, “Cross-project defect prediction: A large scale experiment on data versus domain versus process,” in Proc. 7th Joint Meeting Eur. Softw. Eng. Conf. ACM SIGSOFT Symp., 2009, pp. 91–100.

J. M. Rojas, M. Vivanti, A. Arcuri, and G. Fraser, “A detailed investigation of the effectiveness of whole test suite generation,” Empirical Softw. Eng., vol. 22, no. 2, pp. 852–893, 2017.

B. Korel, “Automated software test data generation,” IEEE Trans. Softw. Eng., vol. 16, no. 8, pp. 870–879, Aug.1990.

P. McMinn, “Search-based software testing: Past, present and future,” in Proc. IEEE 4th Int. Conf. Softw. Testing, Verification Validation Workshops, 2011, pp. 153–163.

R. Just, “Defects4J - A database of real faults and an experimental infrastructure to enable controlled experiments in software engineering research,” 2019, Accessed: Oct., 2019. [Online]. Available: https://github.com/rjust/defects4j

R. A. DeMilli and A. J. Offutt, “Constraint-based automatic test data generation,” IEEE Trans. Softw. Eng., vol. 17, no. 9, pp. 900–910, Sep.1991.

L. J. Morell, “A theory of fault-based testing,” IEEE Trans. Softw. Eng., vol. 16, no. 8, pp. 844–857, Aug.1990.

L. J. Morell, “A theory of error-based testing,” Dept. Comput. Sci., Maryland Univ. College Park, MD, USA, Tech. Rep. TR-1395, 1984.

A. Offutt, “Automatic test data generation,” Georgia Institute of Technology, Tech. Rep., 1989.

N. Li and J. Offutt, “Test Oracle strategies for model-based testing,” IEEE Trans. Softw. Eng., vol. 43, no. 4, pp. 372–395, Apr.2017.

E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The Oracle problem in software testing: A survey,” IEEE Trans. Softw. Eng., vol. 41, no. 5, pp. 507–525, May2015.

G. Fraser and A. Arcuri, “Evolutionary generation of whole test suites,” in Proc. 11th Int. Conf. Qual. Softw., 2011, pp. 31–40.

S. Hosseini, B. Turhan, and D. Gunarathna, “A systematic literature review and meta-analysis on cross project defect prediction,” IEEE Trans. Softw. Eng., vol. 45, no. 2, pp. 111–147, Feb.2019.

A. Arcuri and L. Briand, “A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering,” Softw. Testing, Verification Rel., vol. 24, no. 3, pp. 219–250, 2014.

A. Vargha and H. D. Delaney, “A critique and improvement of the “CL” common language effect size statistics of McGraw and Wong,” J. Educ. Behav. Statist., vol. 25, no. 2, pp. 101–132, 2000.

C. O. Fritz, P. E. Morris, and J. J. Richler, “Effect size estimates: Current use, calculations, and interpretation.” J. Exp. Psychol.: Gen., vol. 141, no. 1, pp. 2–18, 2012.

J. Campos, A. Arcuri, G. Fraser, and R. Abreu, “Continuous test generation: Enhancing continuous integration with automated test generation,” in Proc. 29th ACM/IEEE Int. Conf. Automated Softw. Eng., 2014, pp. 55–66.

M. Fowler and M. Foemmel, “Continuous integration,” 2006, Accessed: Feb. 10, 2022. [Online]. Available: https://www.martinfowler.com/articles/continuousIntegration.html

R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existing faults to enable controlled testing studies for Java programs,” in Proc. Int. Symp. Softw. Testing Anal., 2014, pp. 437–440.

J. Sohn and S. Yoo, “Empirical evaluation of fault localisation using code and change metrics,” IEEE Trans. Softw. Eng., 2019, vol. 47, no. 8, pp. 1605–1625, Aug.2021.

G. Gay, “The fitness function for the job: Search-based generation of test suites that detect real faults,” in Proc. IEEE Int. Conf. Softw. Testing, Verification Validation., 2017, pp. 345–355.

A. Aleti and M. Martinez, “E-APR: Mapping the effectiveness of automated program repair,” Empirical Softw. Eng., vol. 26, no. 5, pp. 1–30, 2021.

S. Pearson , “Evaluating and improving fault localization,” in Proc. 39th Int. Conf. Softw. Eng., 2017, pp. 609–620.

J. Campos, A. Panichella, and G. Fraser, “EvoSuite at the SBST 2019 tool competition,” in Proc. 12th Int. Workshop Search-Based Softw. Testing, 2019, pp. 29–32.

EvoSuite, “EvoSuite - Automated generation of Junit test suites for Java classes,” 2019, Accessed: Nov., 2019. [Online]. Available: https://github.com/EvoSuite/evosuite

G. Fraser, “Evosuite - Automatic test suite generation for Java,” Accessed: Sep., 2019. [Online]. Available: http://www.evosuite.org/

Most read articles by the same author(s)

1 2 3 4 5 6 7 8 9 10 > >>

International Journal of Modern Computer Science and IT Innovations

GUIDING SEARCH-BASED SOFTWARE TESTING WITH DEFECT PREDICTION: AN EMPIRICAL INVESTIGATION

Abstract

Keywords

References

Most read articles by the same author(s)

Similar Articles