User-centric Evaluation of GenAI Alignment and Recommendations based on Predictive Learning Analytics

Ahmed H., Kayaduman H., López-Pernas S., Tukiainen M., Saqr M.

2nd International Workshop on Generative AI for Learning Analytics, GenAI-LA 2025, Dublin, İrlanda, 03 Mart 2025, cilt.3994, ss.1-10, (Tam Metin Bildiri) identifier

  • Yayın Türü: Bildiri / Tam Metin Bildiri
  • Cilt numarası: 3994
  • Basıldığı Şehir: Dublin
  • Basıldığı Ülke: İrlanda
  • Sayfa Sayıları: ss.1-10
  • Anahtar Kelimeler: Generative AI, Large Language Models, Predictive Learning Analytics, User-centric Evaluation
  • İnönü Üniversitesi Adresli: Evet

Özet

Predictive models are one of the hallmarks of learning analytics research, relying on learner data to predict academic achievement and dropouts, enabling targeted interventions. Using a user-centric evaluation framework, we assessed the recommendations generated by ChatGPT based on the results of 136 studies that used student data for predictive modeling. The evaluation considered general attributes (accuracy, coherence, justification) as well as education-specific criteria (alignment with learning theories, ethics, learner-centeredness). The results indicate that, while LLM-generated recommendations are generally accurate, coherent and useful, they often lack alignment with diverse learning theories and fail to address inclusivity and higher-order cognitive skills effectively. Therefore, to operationalize LLMs to provide automated feedback to students, these aspects should be explicitly considered in the prompt design.