Evaluating Programming Assignments and Compile-and-Run Prompts in CS1 by Observing Student Programming Behaviors
Background and Context: Programming assignments and compile-and-run prompts are designed to authenticity assess programming ability by providing an integrated development environment for students to write code to solve a problem. Recent literature supports the use of compile-and-run prompts, where programming ability is assessed in a timed and secure exam environment, but a lack of investigation into student programming behaviors during exams raises questions about the validity and alignment of assignments and compile-and-run prompts. Objectives: Our study evaluates the authenticity, validity, and alignment of programming assignments and compile-and-run prompts in a CS1 course. Specifically, we investigate differences in programming behaviors, how these behaviors relate to performance, and whether assignments effectively prepare students for compile-and-run prompts. Methods: We collect performance and programming process data from students during assignments and exams. By calculating programming behavior metrics derived from keystrokes and compilations, we observe various behaviors, such as students struggling with compiler errors and pasting code from external resources. Findings: We find significant behavioral differences between assignments and exams, with engagement behaviors during assignments correlating with cumulative course performance and behaviors associated with programming ability during exams correlating with performance. By analyzing pasting behaviors, we find that student over-reliance on external resources during assignments is linked to a lack of student preparedness for compile-and-run prompts. Implications: Based on our findings, we recommend compile-and-run prompts on exams as an authentic and secure assessment of individual programming ability and also recommend the tracking of programming process data to monitor student experience and engagement. We discuss the implications of over-reliance on external resources for programming assessments and propose strategies for incorporating compile-and-run prompts into CS1 to improve student learning and mitigate this over-reliance.