New Perspectives in Science Education

Generative AI technology based on LLM (Large-scale Language Models) has been applied to various fields, and one of them is an attempt to apply the technology of automatic program generation to programming education. By using OpenAI's API, we developed and evaluated a prototype VTA (Virtual Teaching Assistant) system that takes C language programs with errors and error messages as input to LLM and outputs appropriate advice for beginner-level learners. The results showed that the VTA produced appropriate advice for syntactic and semantic programming errors, but for logical errors, the VTA gave only general explanations in some cases. We focused on floating-point errors among logic errors and found that the main causes were “denominator of division is zero,” “divisor of modulus calculation is zero,” and “result of integer multiplication overflows”. Therefore, for each candidate error factor, appropriate auxiliary information, including line numbers, could be extracted from the program and added to the prompts to generate more appropriate advice and then we confirmed that the effectiveness of the VTA system was improved. Therefore, the advice including the program location is successfully generated by using the auxiliary information extracted from the program, and the information is sufficient for the user's program debugging. However, if the number of error occurrence candidates increases, the advice will include many points of irrelevant locations, which may cause confusion to the user. However, for beginners, the program length is short, and programs of less than 100 lines have at most four potential error locations.