This program is tentative and subject to change.

Tue 5 Aug 2025 12:10 - 12:35 at Grove Ballroom I+II - F: Assessment part 1

Background and Context: Programming assignments and compile-and-run prompts are designed to authenticity assess programming ability by providing an integrated development environment for students to write code to solve a problem. Recent literature supports the use of compile-and-run prompts, where programming ability is assessed in a timed and secure exam environment, but a lack of investigation into student programming behaviors during exams raises questions about the validity and alignment of assignments and compile-and-run prompts. Objectives: Our study evaluates the authenticity, validity, and alignment of programming assignments and compile-and-run prompts in a CS1 course. Specifically, we investigate differences in programming behaviors, how these behaviors relate to performance, and whether assignments effectively prepare students for compile-and-run prompts. Methods: We collect performance and programming process data from students during assignments and exams. By calculating programming behavior metrics derived from keystrokes and compilations, we observe various behaviors, such as students struggling with compiler errors and pasting code from external resources. Findings: We find significant behavioral differences between assignments and exams, with engagement behaviors during assignments correlating with cumulative course performance and behaviors associated with programming ability during exams correlating with performance. By analyzing pasting behaviors, we find that student over-reliance on external resources during assignments is linked to a lack of student preparedness for compile-and-run prompts. Implications: Based on our findings, we recommend compile-and-run prompts on exams as an authentic and secure assessment of individual programming ability and also recommend the tracking of programming process data to monitor student experience and engagement. We discuss the implications of over-reliance on external resources for programming assessments and propose strategies for incorporating compile-and-run prompts into CS1 to improve student learning and mitigate this over-reliance.

This program is tentative and subject to change.

Tue 5 Aug

Displayed time zone: Eastern Time (US & Canada) change

11:20 - 12:35
F: Assessment part 1Research Papers at Grove Ballroom I+II
11:20
25m
Talk
Rubric Is All You Need: Improving LLM-based Code Evaluation With Question-Specific Rubrics
Research Papers
Aditya Pathak BITS Pilani, Rachit Gandhi BITS Pilani, Vaibhav Uttam BITS Pilani, Arnav Ramamoorthy BITS Pilani, Pratyush Ghosh BITS Pilani, Shreyash Verma BITS Pilani, Aditya Mittal BITS Pilani, Aashna Ased BITS Pilani, Chirag Khatri BITS Pilani, Yashwanth Nakka BITS Pilani, Aaryan Raj Jindal BITS Pilani, Devansh BITS Pilani, Jagat Sesh Challa BITS Pilani, Dhruv Kumar BITS Pilani
11:45
25m
Talk
Difficulty Generating Factors for Context-free Language Construction Assignments
Research Papers
Florian Schmalstieg Ruhr University Bochum, Marko Schmellenkamp Ruhr University Bochum, Jakob Schwerter TU Dortmund University, Thomas Zeume Ruhr University Bochum
12:10
25m
Talk
Evaluating Programming Assignments and Compile-and-Run Prompts in CS1 by Observing Student Programming Behaviors
Research Papers
Marcus Gubanyi University of Nebraska, Leen-Kiat Soh University of Nebraska-Lincoln