The use of the AI-based programming assistant, GitHub Copilot, in Software Quality: A systematic literature review
DOI:
https://doi.org/10.33448/rsd-v15i4.50966Keywords:
Copilot, Software Quality, Software Development, Development Automation.Abstract
The purpose of this paper is to investigate how the scientific literature has reported the use of GitHub Copilot and its impact on code quality. The research was conducted through a systematic literature review, following steps such as defining inclusion and exclusion criteria, searching in databases available through the CAPES Portal of Journals, selecting relevant studies, and performing a qualitative analysis of the selected works. Fifteen studies were identified and analyzed, addressing different aspects of GitHub Copilot, including security issues, the influence of prompt structure, impacts on productivity, performance comparisons with human developers, and the challenges faced by users. The findings highlight both the benefits and limitations of using the tool. The tool proves to be a useful support for developers, serving as a good starting point and guide for coding tasks. However, due to its tendency to generate insecure code and its limited capacity to handle complex tasks, it is not recommended for use in isolation. Moreover, outcomes also depend on how developers interact with GitHub Copilot, as excessive reliance and poorly structured prompts may lead to rework — especially among novice programmers, who represent the tool’s primary user base.
References
Barke, S., James, M. B., & Polikarpova, N. (2023). Grounded Copilot: How Programmers Interact with Code-Generating Models. Proceedings of the ACM on Programming Languages, 7(OOPSLA1), 85–111. https://doi.org/10.1145/3586030
Baskhad Idrisov, & Schlippe, T. (2024). Program Code Generation with Generative AIs. Algorithms, 17(2), 62–62. https://doi.org/10.3390/a17020062
ChatGPT, a inteligência artificial como você nunca viu, é a próxima revolução | Brasil. (2023, February 24). McKinsey & Company. https://www.mckinsey.com/br/our-insights/all-insights/chatgpt-e-a-revolucao-da-inteligencia-artificial?form=MG0AV3
Ensslin, L., Ensslin, S. R., & Pinto, H. de M. (2013). Processo de investigação e análise bibliométrica: avaliação da qualidade dos serviços bancários. Revista de Administração Contemporânea, 17(3), 325–349. https://doi.org/10.1590/s1415-65552013000300005
Fagadau, I. D., Mariani, L., Micucci, D., & Riganelli, O. (2024, February 13). Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilot. ArXiv.org. https://doi.org/10.1145/3643916.3644409
Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., & Chen, J. (2025). Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3716848
GitHub. (2025). GitHub Copilot · Your AI pair programmer. GitHub. https://GitHub.com/features/copilot
GitHub. (2025). What is GitHub Copilot? GitHub Docs. https://docs.GitHub.com/en/copilot/about-GitHub-copilot/what-is-GitHub-copilot
Hussein Mozannar, Bansal, G., Fourney, A., & Horvitz, E. (2024). Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. https://doi.org/10.1145/3613904.3641936
IDC - About - Home. (2019). IDC: The Premier Global Market Intelligence Company. https://www.idc.com/about
Imai, S. (2022, May 1). Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study. IEEE Xplore. https://doi.org/10.1145/3510454.3522684
Introducing ChatGPT. (2022, November 30). OpenAI. https://openai.com/index/chatgpt/
Jyoti, R., & Schubmehl, D. (2024). Business opportunity of AI: Generative AI adoption and business impact. International Data Corporation (IDC). Recuperado de https://info.microsoft.com/ww-landing-business-opportunity-of-ai.html
Krasner, H. (2022). The cost of poor quality software in the US: A 2022 report. Consortium for Information & Software Quality. Recuperado de https://www.it-cisq.org/the-cost-of-poor-quality-software-in-the-us-a-2022-report/
Lopes, A. (2023). Introdução aos LLMs e à IA generativa. BRAINS. https://brains.dev/2023/introducao-aos-llms-e-a-ia-generativa/
Nosek, B. A., & Errington, T. M. (2020). What Is Replication? PLOS Biology, 18(3). https://doi.org/10.1371/journal.pbio.3000691
OBrien, D., Biswas, S., Sayem Mohammad Imtiaz, Rabe Abdalkareem, Emad Shihab, & Rajan, H. (2024). Are Prompt Engineering and TODO Comments Friends or Foes? An Evaluation on GitHub Copilot. https://doi.org/10.1145/3597503.3639176
Paula, J. de. (2024, April 5). Dívida Técnica: como reconhecer, entender e superar. Objective. https://www.objective.com.br/insights/divida-tecnica/
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022, May 1). Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. IEEE Xplore. https://doi.org/10.1109/SP46214.2022.9833571
Peslak, A., & Kovalchick, L. (2024). AI for coders: An analysis of the usage of ChatGPT and GitHub Copilot. Issues in Information Systems. https://iacis.org/iis/2024/4_iis_2024_252-260.pdf
Pressman, R. S., & Maxim, B. R. (2021). Engenharia de software: uma abordagem profissional (9ª ed.). AMGH.
Priberam Informática, S.A. (2024). Dicionário Priberam da Língua Portuguesa. Dicionário Priberam Da Língua Portuguesa. https://dicionario.priberam.org/revolucion%C3%A1rio
RocketCode. (2023, September 23). Entendendo a dívida técnica no desenvolvimento de software. https://rocketcode.com.br/blog/entendendo-a-divida-tecnica-no-desenvolvimento-de-software/
Sauvola, J., Tarkoma, S., Klemettinen, M., Riekki, J., & Doermann, D. (2024). Future of software development with generative AI. Automated Software Engineering, 31(1). https://doi.org/10.1007/s10515-024-00426-z
Sena, J., Barreto, A., Barbosa, J., & Alves, K. (2024). POTENCIALIDADES E DESAFIOS DO GitHub COPILOT COMO FERRAMENTA DA INTELIGÊNCIA ARTIFICIAL. P2P E INOVAÇÃO, 10(2). https://doi.org/10.21728/p2p.2024v10n2e-7031
Shi, Y., Nazmus Sakib, Hossain Shahriar, Lo, D., Chi, H., & Qian, K. (2023). AI-Assisted Security: A Step towards Reimagining Software Development for a Safer Future. https://doi.org/10.1109/compsac57700.2023.00142
Song, F., Agarwal, A., & Wen, W. (2024). The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot. ArXiv.org. https://arxiv.org/abs/2410.02091
Stack Overflow. (2024). 2024 developer survey: AI. https://survey.stackoverflow.co/2024/ai/
Usman, M., Bin Ali, N., & Wohlin, C. (2023). A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering. E-Informatica Software Engineering Journal, 17(1), 230105. https://doi.org/10.37190/e-inf230105
Vahid Majdinasab, Bishop, M. J., Rasheed, S., Arghavan Moradidakhel, Tahir, A., & Foutse Khomh. (2024). Assessing the Security of GitHub Copilot’s Generated Code - A Targeted Replication Study. https://doi.org/10.1109/saner60148.2024.00051
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022). Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. CHI Conference on Human Factors in Computing Systems Extended Abstracts. https://doi.org/10.1145/3491101.3519665
Yetistiren, B., Ozsoy, I., & Tuzun, E. (2022). Assessing the quality of GitHub copilot’s code generation. Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering. https://doi.org/10.1145/3558489.3559072
Zhang, B., Liang, P., Zhou, X., Ahmad, A., & Waseem, M. (2023). Demystifying Practices, Challenges and Expected Features of Using GitHub Copilot. International Journal of Software Engineering and Knowledge Engineering, 1–20. https://doi.org/10.1142/s0218194023410048
Ziegler, A., Eirini Kalliamvakou, X. Alice Li, Rice, A., Rifkin, D., Simister, S., Ganesh Sittampalam, & Aftandilian, E. (2024). Measuring GitHub Copilot’s Impact on Productivity. Communications of the ACM, 67(3), 54–63. https://doi.org/10.1145/3633453
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Lais Aparecida Ferreira de Oliveira, Cristina Corrêa de Oliveira
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
1) Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
2) Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
3) Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
