Evaluating the Effectiveness of Open-Source LLMs for Automatic Short Answer Scoring

Automated Short Answer Grading (ASAG) has become a critical challenge in educational technology, requiring models capable of understanding the semantics and context of student responses. This study investigates the effectiveness of three Large Language Models (LLMs) GPT-3.5 Turbo, LLaMA3-70B-8192, and Deepseek for ASAG tasks by evaluating their accuracy, interpretability, feedback quality, and efficiency. Using the Mohler dataset, which contains 2,273 answers scored by human raters, the models were assessed based on Quadratic Weighted Kappa (QWK) and Mean Absolute Error (MAE). Results show that LLaMA3-70B-8192 achieved the highest QWK (0.9484), while Deepseek obtained the lowest MAE (0.3), indicating superior precision. GPT-3.5 Turbo, though fastest, had lower agreement with human scores. The study also highlights differences in feedback style and cost-efficiency, with open-source models offering scalable and customizable alternatives. This research contributes a comprehensive benchmark and introduces a novel comparative framework for selecting LLMs tailored to ASAG needs, balancing accuracy, cost, and pedagogical effectiveness. © 2025 IEEE.