Tue 5 Aug 2025 11:20 - 11:45 at Grove Ballroom I+II - F: Assessment part 1 Chair(s): Andrew Begel

Since the disruption in LLM technology brought about by the release of GPT-3 and ChatGPT, LLMs have shown remarkable promise in programming-related tasks. While code generation remains a popular field of research, code evaluation using LLMs remains a problem with no conclusive solution. In this paper, we focus on LLM-based code evaluation and attempt to fill in the existing gaps. We propose multi-agentic novel approaches using \emph{question-specific rubrics} tailored to the problem statement, arguing that these perform better for logical assessment than the existing approaches that use \emph{question-agnostic rubrics}. To address the lack of suitable evaluation datasets, we introduce two datasets: a Data Structures and Algorithms dataset containing 150 student submissions from a popular Data Structures and Algorithms practice website, and an Object Oriented Programming dataset comprising 80 student submissions from undergraduate computer science courses. In addition to using standard metrics (Spearman Correlation, Cohen’s Kappa), we additionally propose a new metric called as Leniency, which quantifies evaluation strictness relative to expert assessment. Our comprehensive analysis demonstrates that \emph{question-specific rubrics} significantly enhance logical assessment of code in educational settings, providing better feedback aligned with instructional goals beyond mere syntactic correctness.

Tue 5 Aug

Displayed time zone: Eastern Time (US & Canada) change

11:20 - 12:35
F: Assessment part 1Research Papers at Grove Ballroom I+II
Chair(s): Andrew Begel Carnegie Mellon University
11:20
25m
Talk
Rubric Is All You Need: Improving LLM-based Code Evaluation With Question-Specific Rubrics
Research Papers
Aditya Pathak BITS Pilani, Rachit Gandhi BITS Pilani, Vaibhav Uttam BITS Pilani, Arnav Ramamoorthy BITS Pilani, Pratyush Ghosh BITS Pilani, Shreyash Verma BITS Pilani, Aditya Mittal BITS Pilani, Aashna Ased BITS Pilani, Chirag Khatri BITS Pilani, Yashwanth Nakka BITS Pilani, Aaryan Raj Jindal BITS Pilani, Devansh BITS Pilani, Jagat Sesh Challa BITS Pilani, Dhruv Kumar BITS Pilani
11:45
25m
Talk
Difficulty Generating Factors for Context-free Language Construction Assignments
Research Papers
Florian Schmalstieg Ruhr University Bochum, Marko Schmellenkamp Ruhr University Bochum, Jakob Schwerter TU Dortmund University, Thomas Zeume Ruhr University Bochum
12:10
25m
Talk
Evaluating Programming Assignments and Compile-and-Run Prompts in CS1 by Observing Student Programming Behaviors
Research Papers
Marcus Gubanyi University of Nebraska, Leen-Kiat Soh University of Nebraska-Lincoln