New Perspectives in Science Education

The emergence of Large Language Models such as ChatGPT and Gemini is redefining how students engage with programming education in higher education contexts. These models can generate explanations, examples, and personalized feedback, potentially transforming learning dynamics in computer science courses. However, their pedagogical integration remains underexplored, particularly regarding the reliability of automated feedback, students' critical engagement, and the risks of cognitive passivity or illusion of learning. This study investigates the pedagogical effectiveness of LLM-generated feedback by conducting empirical experiments using open datasets such as Codeforces and HumanEval, to simulate authentic student–AI interactions in coding tasks. Through a mixed-method design, the research evaluates feedback quality based on linguistic accuracy, conceptual relevance, and instructional value, employing both computational metrics (BLEU, CodeBLEU) and human assessments. Results reveal that carefully engineered prompts substantially enhance the coherence and instructional alignment of AI-generated feedback, promoting deeper conceptual understanding and learner autonomy. Yet, challenges persist in overreliance and the limited capacity of current models to foster metacognitive reflection. The study contributes to the growing discourse on the responsible and evidence-based integration of generative AI in higher education by offering design principles for effective feedback systems and proposing a pedagogical framework to balance automation with human guidance. These findings invite educators and institutions to rethink the boundaries between instruction, assistance, and assessment in the age of generative artificial intelligence.