Toward Fair and Scalable Assessment of Socially Shared Regulation of Learning with Large Language Models

Effective collaborative learning depends on learners’ ability to collectively regulate their learning processes through socially shared regulation of learning (SSRL). Despite its central role in collaboration quality and learning outcomes, SSRL remains challenging to assess due to its multidimensional, dynamic, and interactional nature. Recent advances in large language models (LLMs) offer promising opportunities to automate discourse-based assessment, yet questions remain regarding their validity, robustness, and fairness across student populations. This study investigates the feasibility of using LLMs, specifically GPT-4o and GPT-5, to automatically code SSRL behaviors in collaborative discourse from a scenario-based science task, comprising 5830 student chat turns. Using a theory-driven coding framework, we compare LLM-based coding across prompt designs and model types, evaluate the consistency of LLM performance across underserved and non-underserved school contexts, and examine how SSRL behaviors differ between these student populations. Results show that LLM-based coding achieves strong agreement with human annotation, approaching human–human reliability, with context-enriched prompting yielding improved performance than general prompting. Importantly, LLM performance is stable across school contexts. Analyses further reveal meaningful differences in SSRL behaviors, with teams from an underserved school context demonstrating more cognitive information sharing behaviors, while teams from a non-underserved context engage more frequently in metacognitive planning and affective regulation. These findings highlight the potential of LLMs for scalable and equitable assessment of collaborative regulation and inform the design of responsible, adaptive AI systems that support collaborative learning for diverse learners.