Increasing Effectiveness and Reducing Costs of Generative AI Feedback

• Sammy Alashoush, Computer Science, University of Delaware

• John Aromando, Computer & Information Sciences (College of Engineering), University of Delaware

Generative AI serves as a powerful educational tool by possessing the capacity to automatically generate feedback for introductory course assignments. However, feedback produced by large language models (LLMs) such as OpenAI’s ChatGPT lacks the inclusion of contextual information and personalization. Moreover, the high monetary and energy costs associated with LLM processing potentially limit this technology to educational institutions which have a substantial enough budget to fund it, leaving more resource-limited institutions such as small public school districts excluded from the technology entirely. In order to improve AI-generated feedback on students’ assignments, we fine-tuned OpenAI’s GPT-3.5 Turbo model using sample conversations between patients and mental health experts, hoping to orient the feedback to adopt a more mentoring role. Separately, in order to address the issue of the cost barrier, we have explored the implementation of a low cost programming error message classification algorithm, an important first step in identifying a problem in code and giving some general feedback. Initial tests of the fine-tuned model do not indicate any significant improvement in the feedback’s mentorship capabilities compared to the original model. Further adjustments, perhaps with a newer model of ChatGPT or Meta’s Llama, are necessary to achieve the desired outcome.