Open Access
GUIDING SEARCH-BASED SOFTWARE TESTING WITH DEFECT PREDICTION: AN EMPIRICAL INVESTIGATION
4
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, India
4
School of Computing and Artificial Intelligence, Indian Institute of Information Technology, Allahabad, India
Abstract
Search-Based Software Testing (SBST) has emerged as a powerful technique for automated test case generation, effectively achieving high code coverage. However, maximizing code coverage does not always correlate directly with the ability to detect real faults. This paper presents an empirical investigation into the effectiveness of using theoretical defect predictors to guide the search process in SBST, aiming to enhance its fault-finding capability. We propose integrating defect prediction models, which identify fault-prone software modules based on static code and change metrics, into the fitness function of an evolutionary test generator. Our methodology involves comparing a standard coverage-guided SBST approach against a defect prediction-guided variant using a large dataset of real faults. Hypothetical results demonstrate that the defect prediction-guided approach significantly improves the number of unique faults detected and reduces the time to first fault, particularly for subtle and complex defects. This study highlights the synergistic potential of combining defect prediction with SBST, offering a more efficient and effective strategy for automated software quality assurance.
Keywords
Search-based software testing,
defect prediction,
empirical investigation,
software quality
References
G. Fraser and A. Arcuri, “Whole test suite generation,” IEEE Trans. Softw. Eng., vol. 39, no. 2, pp. 276–291, Feb.2013.
A. Panichella, F. M. Kifetew, and P. Tonella, “Reformulating branch coverage as a many-objective optimization problem,” in Proc. IEEE 8th Int. Conf. Softw. Testing, Verification Validation, 2015, pp. 1–10.
A. Panichella, F. M. Kifetew, and P. Tonella, “Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,” IEEE Trans. Softw. Eng., vol. 44, no. 2, pp. 122–158, Feb.2018.
A. Panichella, F. M. Kifetew, and P. Tonella, “A large scale empirical comparison of state-of-the-art search-based test case generators,” Inf. Softw. Technol., vol. 104, pp. 236–256, 2018.
S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri, “Do automatically generated unit tests find real faults? An empirical study of effectiveness and challenges (T),” in Proc. 30th IEEE/ACM Int. Conf. Automated Softw. Eng., 2015, pp. 201–211.
M. M. Almasi, H. Hemmati, G. Fraser, A. Arcuri, and J. Benefelds, “An industrial evaluation of unit test generation: Finding real faults in a financial application,” in Proc. 39th Int. Conf. Softw. Eng.: Softw. Eng. Pract. Track, 2017, pp. 263–272.
A. Salahirad, H. Almulla, and G. Gay, “Choosing the fitness function for the job: Automated generation of test suites that detect real faults,” Softw. Testing, Verification Rel., vol. 29, no. 4–5, 2019, Art. no. e1701.
A. Perera, A. Aleti, M. Böhme, and B. Turhan, “Defect prediction guided search-based software testing,” in Proc. 35th IEEE/ACM Int. Conf. Automated Softw. Eng., 2020, pp. 448–460.
A. Schröter, T. Zimmermann, and A. Zeller, “Predicting component failures at design time,” in Proc. ACM/IEEE Int. Symp. Empirical Softw. Eng., 2006, pp. 18–27.
S. Kim, T. Zimmermann, E. J. Whitehead Jr, and A. Zeller, “Predicting faults from cached history,” in Proc. 29th Int. Conf. Softw. Eng., 2007, pp. 489–498.
P. A. F. de Freitas, “Software repository mining analytics to estimate software component reliability,” Faculty of Engineering, University of Porto, Tech. Rep., 2015.
H. Hata, O. Mizuno, and T. Kikuno, “Bug prediction based on fine-grained module histories,” in Proc. 34th Int. Conf. Softw. Eng., 2012, pp. 200–210.
E. Giger, M. D’Ambros, M. Pinzger, and H. C. Gall, “Method-level bug prediction,” in Proc. ACM-IEEE Int. Symp. Empirical Softw. Eng. Meas., 2012, pp. 171–180.
T. Menzies, J. Greenwald, and A. Frank, “Data mining static code attributes to learn defect predictors,” IEEE Trans. Softw. Eng., vol. 33, no. 1, pp. 2–13, Jan.2007.
T. Zimmermann, R. Premraj, and A. Zeller, “Predicting defects for eclipse,” in Proc. 3rd Int. Workshop Predictor Models Softw. Eng., 2007, Art. no. 9.
N. Nagappan and T. Ball, “Use of relative code churn measures to predict system defect density,” in Proc. 27th Int. Conf. Softw. Eng., 2005, pp. 284–292.
N. Nagappan, B. Murphy, and V. Basili, “The influence of organizational structure on software quality,” in Proc. ACM/IEEE 30th Int. Conf. Softw. Eng., 2008, pp. 521–530.
B. Caglayan, B. Turhan, A. Bener, M. Habayeb, A. Miransky, and E. Cialini, “Merits of organizational metrics in defect prediction: An industrial replication,” in Proc. 37th Int. Conf. Softw. Eng., 2015, pp. 89–98.
N. Nagappan, A. Zeller, T. Zimmermann, K. Herzig, and B. Murphy, “Change bursts as defect predictors,” in Proc. IEEE 21st Int. Symp. Softw. Rel. Eng., 2010, pp. 309–318.
C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead Jr, “Does bug prediction support human developers? Findings from a Google case study,” in Proc. Int. Conf. Softw. Eng., 2013, pp. 372–381.
C. Lewis and R. Ou, “Bug prediction at Google,” 2011, Accessed: Sep., 2019. [Online]. Available: http://google-engtools.blogspot.com
H. K. Dam , “Lessons learned from using a deep tree-based model for software defect prediction in practice,” in Proc. 16th Int. Conf. Mining Softw. Repositories, 2019, pp. 46–57.
D. Paterson, J. Campos, R. Abreu, G. M. Kapfhammer, G. Fraser, and P. McMinn, “An empirical study on the use of defect prediction for test case prioritization,” in Proc. 12th IEEE Conf. Softw. Testing, Validation Verification, 2019, pp. 346–357.
E. Hershkovich, R. Stern, R. Abreu, and A. Elmishali, “Prediction-guided software test generation,” in Proc. 30th Int. Workshop Princ. Diagnosis, 2019, Accessed: Feb. 08, 2022. [Online]. Available: https://dx-workshop.org/2019/
T. Zimmermann, N. Nagappan, H. Gall, E. Giger, and B. Murphy, “Cross-project defect prediction: A large scale experiment on data versus domain versus process,” in Proc. 7th Joint Meeting Eur. Softw. Eng. Conf. ACM SIGSOFT Symp., 2009, pp. 91–100.
J. M. Rojas, M. Vivanti, A. Arcuri, and G. Fraser, “A detailed investigation of the effectiveness of whole test suite generation,” Empirical Softw. Eng., vol. 22, no. 2, pp. 852–893, 2017.
B. Korel, “Automated software test data generation,” IEEE Trans. Softw. Eng., vol. 16, no. 8, pp. 870–879, Aug.1990.
P. McMinn, “Search-based software testing: Past, present and future,” in Proc. IEEE 4th Int. Conf. Softw. Testing, Verification Validation Workshops, 2011, pp. 153–163.
R. Just, “Defects4J - A database of real faults and an experimental infrastructure to enable controlled experiments in software engineering research,” 2019, Accessed: Oct., 2019. [Online]. Available: https://github.com/rjust/defects4j
R. A. DeMilli and A. J. Offutt, “Constraint-based automatic test data generation,” IEEE Trans. Softw. Eng., vol. 17, no. 9, pp. 900–910, Sep.1991.
L. J. Morell, “A theory of fault-based testing,” IEEE Trans. Softw. Eng., vol. 16, no. 8, pp. 844–857, Aug.1990.
L. J. Morell, “A theory of error-based testing,” Dept. Comput. Sci., Maryland Univ. College Park, MD, USA, Tech. Rep. TR-1395, 1984.
A. Offutt, “Automatic test data generation,” Georgia Institute of Technology, Tech. Rep., 1989.
N. Li and J. Offutt, “Test Oracle strategies for model-based testing,” IEEE Trans. Softw. Eng., vol. 43, no. 4, pp. 372–395, Apr.2017.
E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The Oracle problem in software testing: A survey,” IEEE Trans. Softw. Eng., vol. 41, no. 5, pp. 507–525, May2015.
G. Fraser and A. Arcuri, “Evolutionary generation of whole test suites,” in Proc. 11th Int. Conf. Qual. Softw., 2011, pp. 31–40.
S. Hosseini, B. Turhan, and D. Gunarathna, “A systematic literature review and meta-analysis on cross project defect prediction,” IEEE Trans. Softw. Eng., vol. 45, no. 2, pp. 111–147, Feb.2019.
A. Arcuri and L. Briand, “A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering,” Softw. Testing, Verification Rel., vol. 24, no. 3, pp. 219–250, 2014.
A. Vargha and H. D. Delaney, “A critique and improvement of the “CL” common language effect size statistics of McGraw and Wong,” J. Educ. Behav. Statist., vol. 25, no. 2, pp. 101–132, 2000.
C. O. Fritz, P. E. Morris, and J. J. Richler, “Effect size estimates: Current use, calculations, and interpretation.” J. Exp. Psychol.: Gen., vol. 141, no. 1, pp. 2–18, 2012.
J. Campos, A. Arcuri, G. Fraser, and R. Abreu, “Continuous test generation: Enhancing continuous integration with automated test generation,” in Proc. 29th ACM/IEEE Int. Conf. Automated Softw. Eng., 2014, pp. 55–66.
M. Fowler and M. Foemmel, “Continuous integration,” 2006, Accessed: Feb. 10, 2022. [Online]. Available: https://www.martinfowler.com/articles/continuousIntegration.html
R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existing faults to enable controlled testing studies for Java programs,” in Proc. Int. Symp. Softw. Testing Anal., 2014, pp. 437–440.
J. Sohn and S. Yoo, “Empirical evaluation of fault localisation using code and change metrics,” IEEE Trans. Softw. Eng., 2019, vol. 47, no. 8, pp. 1605–1625, Aug.2021.
G. Gay, “The fitness function for the job: Search-based generation of test suites that detect real faults,” in Proc. IEEE Int. Conf. Softw. Testing, Verification Validation., 2017, pp. 345–355.
A. Aleti and M. Martinez, “E-APR: Mapping the effectiveness of automated program repair,” Empirical Softw. Eng., vol. 26, no. 5, pp. 1–30, 2021.
S. Pearson , “Evaluating and improving fault localization,” in Proc. 39th Int. Conf. Softw. Eng., 2017, pp. 609–620.
J. Campos, A. Panichella, and G. Fraser, “EvoSuite at the SBST 2019 tool competition,” in Proc. 12th Int. Workshop Search-Based Softw. Testing, 2019, pp. 29–32.
EvoSuite, “EvoSuite - Automated generation of Junit test suites for Java classes,” 2019, Accessed: Nov., 2019. [Online]. Available: https://github.com/EvoSuite/evosuite
G. Fraser, “Evosuite - Automatic test suite generation for Java,” Accessed: Sep., 2019. [Online]. Available: http://www.evosuite.org/
Most read articles by the same author(s)
- Dr. Aarav Sharma, Ms. Priya Nair, A Hybrid Deep Learning Framework for Automated Liver Tumor Segmentation and Malignancy Prediction from CT Imaging Data , International Journal of Modern Computer Science and IT Innovations: Vol. 3 No. 07 (2026): Volume 03 Issue 07
- Nguyen Minh Khoa, Tran Ngoc Anh, An Integrated Security Analysis Model for Identifying Software and Hardware System Vulnerabilities , International Journal of Modern Computer Science and IT Innovations: Vol. 3 No. 10 (2026): Volume 03 Issue 10
- Alistair J. Finch, Sustainable Development and Mechanical Performance of Natural Fiber–Reinforced Polymer Composites: Comprehensive Analysis, Methodologies, and Future Directions , International Journal of Modern Computer Science and IT Innovations: Vol. 2 No. 05 (2025): Volume 02 Issue 05
Similar Articles
- Evgenii Lvov, Quality Assurance and Scalability: The Role of High-Test Coverage in Continuous Integration and Deployment Pipelines , International Journal of Modern Computer Science and IT Innovations: Vol. 3 No. 07 (2026): Volume 03 Issue 07
- Alistair J. Finch, Integrating Jira, Jenkins, and Azure DevOps to Optimize Software Release Pipelines , International Journal of Modern Computer Science and IT Innovations: Vol. 2 No. 10 (2025): Volume 02 Issue 10
- Puspita Sari, Nathanael Sianipar, A DESIGN SCIENCE APPROACH TO MITIGATING INTER-SERVICE INTEGRATION FAILURES IN MICROSERVICE ARCHITECTURES: THE CONSUMER-DRIVEN CONTRACT TESTING FRAMEWORK AND PILOT IMPLEMENTATION , International Journal of Modern Computer Science and IT Innovations: Vol. 2 No. 10 (2025): Volume 02 Issue 10
- John M. Langley, Augmenting Data Quality and Model Reliability in Large-Scale Language and Code Models: A Hybrid Framework for Evaluation, Pretraining, and Retrieval-Augmented Techniques , International Journal of Modern Computer Science and IT Innovations: Vol. 2 No. 09 (2025): Volume 02 Issue 09
- Victor E. Halden, Integrating AI-Driven Automation into Modern DevOps: Advancements, Challenges, and Strategic Implications in Software Engineering , International Journal of Modern Computer Science and IT Innovations: Vol. 3 No. 02 (2026): Volume 03 Issue 02
- Dr. Alistair Sterling, Architectural Evolution and Decomposition Strategies: A Comprehensive Analysis of Microservice Migration, Performance Optimization, And Machine Learning-Assisted Service Boundary Detection , International Journal of Modern Computer Science and IT Innovations: Vol. 2 No. 12 (2025): Volume 02 Issue 12
- Dr. Isabella D. Ricci, Dr. Farah A. Rahman, OPTIMIZING WEB DEVELOPMENT THROUGH STRATEGIC WEB FRAMEWORK ADOPTION , International Journal of Modern Computer Science and IT Innovations: Vol. 2 No. 05 (2025): Volume 02 Issue 05
- Kseniia Pereshliuga, ALISMIA AI as a Tool for Digital Empowerment: Redesigning Client Interaction in Beauty Businesses , International Journal of Modern Computer Science and IT Innovations: Vol. 3 No. 05 (2026): Volume 03 Issue 05
- Ngozi Okafor, A Consumer-Driven Contract-Based Approach to Verifying User Interface Integration in Microservices Architectures , International Journal of Modern Computer Science and IT Innovations: Vol. 2 No. 10 (2025): Volume 02 Issue 10
- Dr. Andika Prasetyo, Siti Rahmawati, M.Sc., Rizky Maulana, Structured Teaching Framework Focused on Beginner-Level Software Development Skills , International Journal of Modern Computer Science and IT Innovations: Vol. 3 No. 04 (2026): Volume 03 Issue 04
1-10 of 66
Next
You may also start an advanced similarity search for this article.