Despite the growing number of LLM-centered educational tools being developed, there is a surprisingly limited number of studies on how unfacilitated LLM usage affects learning behaviors. We address this gap by comparing student-generated artifacts, such as code and admissions essays, from before and after the release of ChatGPT to detect any large-scale shifts in behavior. We are currently working on, and plan to incorporate, more metrics that focus on how students’ solution development across all attempts at a single coding assignment differ before pre- and post-ChatGPT.