New Perspectives in Science Education

Artificial intelligence (AI) and learning analytics offer promising opportunities to enhance personalized education by predicting student performance and identifying learning needs. Understanding how students' historical interactions, learning behaviors, and knowledge mastery influence future performance can support the development of adaptive learning environments. This study investigates the effectiveness of large language models (LLMs), traditional machine learning algorithms, and educational feature engineering in predicting student learning outcomes. Two prediction tasks are examined: forecasting students' average correctness over their next ten questions and predicting the correctness of their next response.

A Time-LLM-inspired model built on a pretrained GPT-2 architecture is developed to analyze historical student interactions. Its performance is compared with six traditional machine learning approaches: Random Forest, Logistic/Ridge Regression, XGBoost, LightGBM, Histogram Gradient Boosting, and CatBoost. Additional experiments investigate GPT-2 embeddings independently and in combination with engineered educational features. Furthermore, 66 features are organized into seven categories covering historical performance, multi-window learning trends, topic mastery, temporal behavior, learning dynamics, lecture and explanation interactions, and composite learning indicators. Incremental and feature-removal ablation experiments evaluate their contributions.

Results demonstrate that traditional machine learning approaches achieve performance comparable to Time-LLM. Using sequences of 200 interactions, Time-LLM achieved a classification accuracy of 71.61%, an F1-score of 0.8029, a mean absolute error (MAE) of 0.1314, and a root mean squared error (RMSE) of 0.1651. CatBoost achieved 72.19% classification accuracy and an F1-score of 0.8101 in the initial comparison. Subsequent feature engineering experiments achieved a maximum classification accuracy of 73.15% and a maximum F1-score of 0.8185 across different feature configurations. The complete feature set achieved an MAE of 0.1321 and an RMSE of 0.1651. Ablation results highlighted the importance of features like topic mastery, historical learning patterns, and learning dynamics in performance prediction.

These findings suggest that educationally informed AI models can support student performance monitoring, personalized learning, and data-driven instructional decisions.

Keywords:

Artificial Intelligence in Education; Learning Analytics; Student Performance Prediction; Personalized Learning; Time-LLM; Machine Learning; Educational Feature Engineering.

References:

[1] G. Abdelrahman, Q. Wang, and B. Nunes, “Knowledge tracing: A survey,” ACM Computing Surveys, vol. 55, no. 11, Art. no. 224, pp. 1–37, 2023. https://doi.org/10.1145/3569576

[2] S. Shen, Q. Liu, Z. Huang, Y. Zheng, M. Yin, M. Wang, and E. Chen, “A survey of knowledge tracing: Models, variants, and applications,” IEEE Transactions on Learning Technologies, vol. 17, pp. 1858 - 1879, 2024. https://doi.org/10.1109/TLT.2024.3383325   

[3] Y. Cho, R. E. AlMamlook, and T. Gharaibeh, “A systematic review of knowledge tracing and large language models in education: Opportunities, issues, and future research,” arXiv preprint arXiv:2412.09248, 2024. https://doi.org/10.48550/arXiv.2412.09248

[4] Z. Li, J. Yang, J. Wang, L. Shi, J. Feng, and S. Stein, “LBKT: A LSTM BERT-based knowledge tracing model for long-sequence data,” in Generative Intelligence and Intelligent Tutoring Systems, Lecture Notes in Computer Science, vol. 14799, Springer, 2024, pp. 174–184. https://doi.org/10.1007/978-3-031-63031-6_15

[5] Z. Li, M. Jacobsen, L. Shi, Y. Zhou, and J. Wang, “Broader and deeper: A multi-features with latent relations BERT knowledge tracing model,” in Responsive and Sustainable Educational Futures, Lecture Notes in Computer Science, vol. 14200, Springer, 2023, pp. 183–197. https://doi.org/10.1007/978-3-031-42682-7_13