The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Coding Tasks
When graduates of computing degree programs enter the software industry, they will most likely join teams working on legacy code bases developed by people other than themselves. In these so-called brownfield software development settings, generative artificial intelligence (GenAI) coding assistants like GitHub Copilot are rapidly transforming software development practices, yet the impact of GenAI on student programmers performing brownfield development tasks remains underexplored. This paper investigates how GitHub Copilot influences undergraduate students’ programming performance, behaviors, and understanding when completing brownfield programming tasks, in which they add new code to an unfamiliar code base. We conducted a controlled experiment in which 10 undergraduate computer science students completed isomorphic brownfield development tasks with and without Copilot in a legacy web application. Using a mixed-methods approach combining performance analysis, behavioral analysis, and exit interviews, we found that students completed tasks 34.9% faster (p < 0.05) and made 50% more solution progress (p < 0.05) when using Copilot. Moreover, our analysis revealed that, when using Copilot, students spent 10.63% less time manually writing code (p < 0.05), and 11.6% less time conducting web searches (p < 0.05), providing evidence of a fundamental shift in how they engaged in programming. When using Copilot, higher-performing students tended to be more selective in their use of AI-generated code, preferring granular inline suggestions over adoption of code blocks wholesale. However, in exit interviews, students reported concerns about not understanding how or why Copilot suggestions work, highlighting a crucial tension: GenAI may promote brownfield programming efficiency at the cost of diminished learning. This research suggests the need for computing educators to develop new pedagogical approaches that leverage GenAI assistants’ benefits while fostering reflection on how and why GenAI suggestions address brownfield programming tasks.
Wed 6 AugDisplayed time zone: Eastern Time (US & Canada) change
15:20 - 16:10 | L: Student Behaviours with GenAIResearch Papers at Grove Ballroom I+II Chair(s): Stephen Edwards Virginia Tech | ||
15:20 25mTalk | The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Coding Tasks Research Papers Md Istiak Hossain Shihab Oregon State University, USA, Chris Hundhausen Oregon State University, USA, Ahsun Tariq Oregon State University, Summit Haque Oregon State University, USA, Yunhan Qiao Oregon State University, Brian Mulanda Wise Oregon State University, Christopher Sanchez Oregon State University | ||
15:45 25mTalk | How Do Novice Programmers Solve Code-Tracing Problems When ChatGPT Is Available? A Qualitative Analysis. Research Papers |