Accuracy of LLMs in medical education: evidence from a concordance test with medical teacher

This study evaluates Large Language Models (LLMs) compared to experienced medical teachers. The analysis examines the performance of three prominent LLMs—ChatGPT, Gemini, and Copilot. The study employs Fleiss’ Kappa Test to statistically analyze the concordance between LLMs and human responses. In discordance, Cohen’s Kappa test was used to find agreement between three Gen AI tools and a Medical Teacher. Results reveal a significant difference in the performance between LLMs and medical teachers, highlighting potential limitations in using AI alone for medical education.