Open Access

Methods of Intelligent Automation of Software System Analysis Using AI Agents in the Development and Operation of Web Applications

4 Senior Full Stack Engineer Warsaw, Poland

Abstract

 

AI agents built on large language models now take part in the analysis of software systems, yet evidence on their use across the life cycle of web applications remains scattered among benchmark reports and separate field studies. This article aims to systematize methods of agent-based analysis in the development and operation of web applications and to substantiate a model for assigning agent autonomy. The study combines a structured review of 25 sources published from 2020 to 2026, comparative analysis of benchmark results, content analysis of field studies, and a qualitative case study of two web products from the author's practice. The synthesis suggests that the strength of the verification oracle may be a more stable basis for assigning autonomy than benchmark capability scores; a direct comparative test of this hypothesis remains necessary. The article proposes an oracle-gated model with four layers, a matrix of four autonomy modes, and five monitoring indicators. The conclusions support staged adoption that begins with tasks backed by executable checks. The findings will interest web developers, game developers, engineering managers, and researchers in software engineering.

Keywords

References

Sajadieh, S., Fattorini, L., Perrault, R., Gil, Y., Parli, V., Santarlasci, L., Pava, J., Maslej, N., Altman, R., Brynjolfsson, E., Brodley, C., Clark, J., Dignum, V., Kumar, V., Landay, J., Lyons, T., Manyika, J., Niebles, J. C., Shoham, Y., . . . Weld, D. (2026). Artificial Intelligence Index Report 2026. Stanford Institute for Human-Centered Artificial Intelligence. https://hai.stanford.edu/ai-index/2026-ai-index-report
Stack Overflow. (2025). 2025 Developer Survey. https://survey.stackoverflow.co/2025/
Tinkoff, D., Van der Veken, L., Chui, M., & Balakrishnan, T. (2026, August 25). The state of AI in 2026: On the road to ROI. McKinsey & Company. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
Cui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T. (2026). The effects of generative AI on high-skilled work: Evidence from three field experiments with software developers. Management Science. Advance online publication. https://doi.org/10.1287/mnsc.2025.00535
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024). Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8), 1-79. https://doi.org/10.1145/3695988
He, J., Treude, C., & Lo, D. (2025). LLM-based multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology, 34(5), 1-30. https://doi.org/10.1145/3712003
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024). https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024). SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37 (pp. 50528-50652). https://proceedings.neurips.cc/paper_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A. (2024). AutoCodeRover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 1592-1604). ACM. https://doi.org/10.1145/3650212.3680384
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2025). Demystifying LLM-based software engineering agents. Proceedings of the ACM on Software Engineering, 2(FSE), 801-824. https://doi.org/10.1145/3715754
Yang, J., Jimenez, C. E., Zhang, A. L., Lieret, K., Yang, J., Wu, X., Press, O., Muennighoff, N., Synnaeve, G., Narasimhan, K. R., Yang, D., Wang, S. I., & Press, O. (2025). SWE-bench Multimodal: Do AI systems generalize to visual software domains? In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR 2025). https://proceedings.iclr.cc/paper_files/paper/2025/hash/07d6332ae36730707fddddba736d7b6c-Abstract-Conference.html
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., & Neubig, G. (2024). WebArena: A realistic web environment for building autonomous agents. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024). https://proceedings.iclr.cc/paper_files/paper/2024/hash/4410c0711e9154a7a2d26f9b3816d1ef-Abstract-Conference.html
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024). Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 50(4), 911-936. https://doi.org/10.1109/TSE.2024.3368208
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2024). An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1), 85-105. https://doi.org/10.1109/TSE.2023.3334955
Alshahwan, N., Chheda, J., Finogenova, A., Gokkaya, B., Harman, M., Harper, I., Marginean, A., Sengupta, S., & Wang, E. (2024). Automated unit test improvement using large language models at Meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (pp. 185-196). ACM. https://doi.org/10.1145/3663529.3663839
Cihan, U., Haratian, V., İçöz, A., Gül, M. K., Devran, Ö., Bayendur, E. F., Uçar, B. M., & Tüzün, E. (2025). Automated code review in practice. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (pp. 425-436). IEEE. https://doi.org/10.1109/ICSE-SEIP66354.2025.00043
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024). Measuring GitHub Copilot's impact on productivity. Communications of the ACM, 67(3), 54-63. https://doi.org/10.1145/3633453
Liang, J. T., Yang, C., & Myers, B. A. (2024). A large-scale survey on the usability of AI programming assistants: Successes and challenges. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (pp. 1-13). ACM. https://doi.org/10.1145/3597503.3608128
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2025). Asleep at the keyboard? Assessing the security of GitHub Copilot's code contributions. Communications of the ACM, 68(2), 96-105. https://doi.org/10.1145/3610721
Chen, Y., Xie, H., Ma, M., Kang, Y., Gao, X., Shi, L., Cao, Y., Gao, X., Fan, H., Wen, M., Zeng, J., Ghosh, S., Zhang, X., Zhang, C., Lin, Q., Rajmohan, S., Zhang, D., & Xu, T. (2024). Automatic root cause analysis via large language models for cloud incidents. In Proceedings of the Nineteenth European Conference on Computer Systems (pp. 674-688). ACM. https://doi.org/10.1145/3627703.3629553
Roy, D., Zhang, X., Bhave, R., Bansal, C., Las-Casas, P., Fonseca, R., & Rajmohan, S. (2024). Exploring LLM-based agents for root cause analysis. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (pp. 208-219). ACM. https://doi.org/10.1145/3663529.3663841
Zhang, L., Jia, T., Jia, M., Wu, Y., Liu, A., Yang, Y., Wu, Z., Hu, X., Yu, P. S., & Li, Y. (2026). A survey of AIOps in the era of large language models. ACM Computing Surveys, 58(2), 1-35. https://doi.org/10.1145/3746635
Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., Togelius, J., & Yannakakis, G. N. (2024). Large language models and games: A survey and roadmap. IEEE Transactions on Games. Advance online publication. https://doi.org/10.1109/TG.2024.3461510
Taesiri, M. R., Feng, T., Bezemer, C.-P., & Nguyen, A. (2024). GlitchBench: Can large multimodal models detect video game glitches? In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 22444-22455). IEEE. https://doi.org/10.1109/CVPR52733.2024.02118
Shneiderman, B. (2020). Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Computer Interaction, 36(6), 495-504. https://doi.org/10.1080/10447318.2020.1741118

Most read articles by the same author(s)

1 2 3 4 5 6 7 8 9 10 > >> 

Similar Articles

11-20 of 112

You may also start an advanced similarity search for this article.