Research Article

AI-augmented judgement in teacher performance assessment: Evidence from a human-AI moderation workflow

Zara Ersozlu 1 * , Susan Ledger 1 , Mark Babic 1 , Robert Parkes 1
More Detail
1 School of Education, University of Newcastle, Callaghan, NSW, AUSTRALIA* Corresponding Author
Contemporary Educational Technology, 18(3), July 2026, ep674, https://doi.org/10.30935/cedtech/18970
Published: 15 July 2026
OPEN ACCESS   802 Views   57 Downloads
Download Full Text (PDF)

ABSTRACT

Artificial intelligence (AI) is increasingly being explored in educational assessment, but its use in high-stakes performance contexts requires careful design to support quality, consistency, and human accountability. This study designed and evaluated an AI-supported workflow to assist assessors in marking teaching performance assessment (TPA) portfolios. TPA assessors first described how they usually evaluate portfolios, and this process was used to develop a meta-level rubric logic, prompts, and workflow for the AI agent. The workflow was tested by comparing human-only marking with AI supported marking in relation to time, accuracy, cognitive load, feedback quality, usability, bias, and fairness. The findings showed that the AI-supported workflow reduced marking time and perceived workload while supporting more structured, evidence-based feedback. It also contributed to greater consistency and fairness in the moderation process. The AI agent is positioned as a “third eye” that supports human judgement. Human assessors remain responsible for making final decisions and may accept, adapt or override AI-generated suggestions.
Keywords: GenAI, human-in-the-loop, cognitive load, time efficiency, judgement support, moderation, teaching performance assessment

CITATION (APA)

Ersozlu, Z., Ledger, S., Babic, M., & Parkes, R. (2026). AI-augmented judgement in teacher performance assessment: Evidence from a human-AI moderation workflow. Contemporary Educational Technology, 18(3), ep674. https://doi.org/10.30935/cedtech/18970

REFERENCES

  1. Australian Institute for Teaching and School Leadership. (2011). Australian professional standards for teachers. AITSL. https://www.aitsl.edu.au/teach/standards
  2. Babic, M. J., White, R., Ersozlu, Z., Costigan, S., Kennedy, S., Guntoro, J., & Ledger, S. (2026). A systematic review on teaching performance assessments impact on stakeholders [Manuscript in preparation]. School of Education, The University of Newcastle.
  3. Bansal, G., Nushi, B., Kamar, E., Lasecki, W., Weld, D., & Horvitz, E. (2021). Beyond accuracy: The role of mental models in human-AI team performance. In Proceedings of the AAAI Conference on Human Factors in Computing Systems.
  4. Bloxham, S., & Boyd, P. (2012). Accountability in grading student work: Securing academic standards in a twenty-first century quality assurance context. British Educational Research Journal, 38(4), 615-634.
  5. Bloxham, S., Hughes, C., & Adie, L. (2016). What’s the point of moderation? A discussion of the purposes achieved through contemporary moderation practices. Assessment & Evaluation in Higher Education, 41(4), 638-653. https://doi.org/10.1080/02602938.2015.1039932
  6. Boud, D., & Molloy, E. (2013). Rethinking models of feedback for learning: The challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698-712. https://doi.org/10.1080/02602938.2012.691462
  7. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
  8. Brooke, J. (1995). SUS: A quick and dirty usability scale. In P. W. Jordan, B. Thoms, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 189-194). Taylor & Francis.
  9. Corbin, T., Bearman, M., Boud, D., & Dawson, P. (2025). The wicked problem of AI and assessment. Assessment & Evaluation in Higher Education, 51(4), 736-752. https://doi.org/10.1080/02602938.2025.2553340
  10. Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20, Article 22. https://doi.org/10.1186/s41239-023-00392-8
  11. Cummings, M. L. (2004). Automation bias in intelligent time critical decision support systems. In Proceedings of the AIAA 1st Intelligent Systems Technical Conference. https://doi.org/10.2514/6.2004-6313
  12. Dai, W., Tsai, Y-S., Lin, J., Aldino, A., Jin, H., Li, T., Gasevic, D., & Chen, G. (2024). Assessing the proficiency of large language models in automatic feedback generation: An evaluation study. Computers and Education: Artificial Intelligence, 7, Article 100299. https://doi.org/10.1016/j.caeai.2024.100299
  13. Fadel, C., Holmes, W., & Bialik, M. (2019). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign.
  14. Ferrara, E. (2024). Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. Science, 6(1), Article 3. https://doi.org/10.3390/sci6010003
  15. Flodén, J. (2025). Grading exams using large language models: A comparison between human and AI grading of exams in higher education using ChatGPT. British Educational Research Journal, 51, 201-224. https://doi.org/10.1002/berj.4069
  16. Green, B., & Chen, Y. (2019). The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM Human Computer Interaction, 3(CSCW), Article 50. https://doi.org/10.1145/3359152
  17. Hancock, P. A., Billings, D. R., Schaefer, K. E., Chen, J. Y., De Visser, E. J., & Parasuraman, R. (2011). A meta-analysis of factors affecting trust in human-robot interaction. Human Factors, 53(5), 517-527. https://doi.org/10.1177/0018720811417254
  18. Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research. Advances in Psychology, 52, 139-183. https://doi.org/10.1016/S0166-4115(08)62386-9
  19. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. https://doi.org/10.3102/003465430298487
  20. Henderson, M., Bearman, M., Chung, J., Fawns, T., Buckingham Shum, S., Matthews, K., & Heredia, J. (2025). Comparing generative AI and teacher feedback: Student perceptions of usefulness and trustworthiness. Assessment & Evaluation in Higher Education. https://doi.org/10.1080/02602938.2025.2502582
  21. Holmes, W., Bialik, M., & Fadel, C. (2022a). Artificial intelligence in education: Promise and implications for teaching and learning. Center for Curriculum Redesign. https://doi.org/10.58863/20.500.12424/4276068
  22. Holmes, W., Persson, J., Chounta, I.-A., Wasson, B., & Dimitrova, V. (2022b). Artificial intelligence and education: A critical view through the lens of human rights, democracy and the rule of law. Council of Europe.
  23. Ilieva, G., Yankova, T., Ruseva, M., & Kabaivanov, S. (2025). A framework for generative AI-driven assessment in higher education. Information, 16(6), Article 472. https://doi.org/10.3390/info16060472
  24. Ioannidis J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 19(8), Article e1004085. https://doi.org/10.1371/journal.pmed.0020124
  25. Joughin, G. (2008). Assessment, learning and judgement in higher education: A critical review. In G. Joughin (Ed.), Assessment, learning and judgement in higher education (pp. 1-15). Springer. https://doi.org/10.1007/978-1-4020-8905-3_2
  26. Kellogg, R. T. (2008). Training writing skills: A cognitive developmental perspective. Journal of Writing Research, 1(1), 1-26. https://doi.org/10.17239/jowr-2008.01.01.1
  27. Klenowski, V., & Adie, L. (2009). Moderation as judgement practice. Assessment & Evaluation in Higher Education, 34(1), 1-11.
  28. Lee, J. D., & See, K. A. (2004). Trust in automation. Human Factors, 46(1), 50-80. https://doi.org/10.1518/hfes.46.1.50.30392
  29. Li, J., Jangamreddy, N. K., Hisamoto, R., Bhansali, R., Dyda, A., Zaphir, L., & Glencross, M. (2024). AI-assisted marking: Functionality and limitations of ChatGPT in written assessment evaluation. Australasian Journal of Educational Technology, 40(4), 56-72. https://doi.org/10.14742/ajet.9463
  30. Liao, V., & Varshney, K. (2021). Human-centered explainable AI (XAI): From algorithms to user experiences. arXiv. https://doi.org/10.48550/arXiv.2110.10790
  31. Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence unleashed: An argument for AI in education. Pearson.
  32. Mascadri, J., Spina, N., Spooner-Lane, R., & Alonzo, D. (2023). Exploring consistency and variation in assessor judgements in teacher performance assessment. Assessment & Evaluation in Higher Education, 48(7), 1085-1100.
  33. Miller, T. (2017). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1-38. https://doi.org/10.1016/j.artint.2018.07.007
  34. Moorhouse, B., Yeo, M., & Wan, Y. (2023). Generative AI tools and assessment: Guidelines of the world’s top-ranking universities. Computers and Education Open, 5, Article 100151. https://doi.org/10.1016/j.caeo.2023.100151
  35. Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998). Automation bias. International Journal of Human-Computer Studies, 52(4), 701-717.
  36. Oberlader, V., Quinten, L., Schmidt, A. F., & Banse, R. (2025). How can I reduce bias in my work? Discussing debiasing strategies for forensic psychological assessments. Professional Psychology: Research and Practice, 56(3), 211-221. https://doi.org/10.1037/pro0000615
  37. Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1-4. https://doi.org/10.1207/S15326985EP3801_1
  38. Parasuraman, R., & Riley, V. (1997). Humans and automation. Human Factors, 39(2), 230-253. https://doi.org/10.1518/001872097778543886
  39. Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics – Part A, 30(3), 286-297. https://doi.org/10.1109/3468.844354
  40. Raji, I., Smart, A., White, R., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020). Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 33-44). ACM. https://doi.org/10.1145/3351095.3372873
  41. Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676-688. https://doi.org/10.1016/j.tics.2016.07.002
  42. Sadler, D. R. (2013). Assuring academic achievement standards. Assessment in Education, 20(1), 5-16. https://doi.org/10.1080/0969594X.2012.714742
  43. Sinha, S., Goel, S., Kumaraguru, P., Geiping, J., Bethge, M., & Prabhu, A. (2025). Can language models falsify? Evaluating algorithmic reasoning with counterexample creation. arXiv. https://doi.org/10.48550/arXiv.2502.19414
  44. Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991-1006. https://doi.org/10.1006/ijhc.1999.0252
  45. Stacey, M., Talbot, D., & Buchanan, J. (2020). Teacher performance assessments and the making of professional standards. Teaching and Teacher Education, 95, Article 103145.
  46. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257-285. https://doi.org/10.1207/s15516709cog1202_4
  47. Sweller, J. (2011). Cognitive load theory. In J. P. Mestre, & B. H. Ross (Eds.), The psychology of learning and motivation: Cognition in education (pp. 37-76). Elsevier Academic Press. https://doi.org/10.1016/B978-0-12-387691-1.00002-8
  48. Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldanado, R., Lodge, J., Milligan, S., Selwyn, N., & Gasevic, D. (2022). Assessment in the age of artificial intelligence. Computers and Education: Artificial Intelligence, 3, Article 100075. https://doi.org/10.1016/j.caeai.2022.100075
  49. Usher, M. (2025). Generative AI vs. instructor vs. peer assessments: A comparison of grading and feedback in higher education. Assessment & Evaluation in Higher Education, 50(6), 912-927. https://doi.org/10.1080/02602938.2025.2487495
  50. Weston, H., Mooney, A., Thomas, M. K. E., & Iannucci, C. (2026). Graduate teachers’ and the teacher performance assessment: The case for architectures of judgement. Assessment & Evaluation in Higher Education. https://doi.org/10.1080/02602938.2026.2658631
  51. Williamson, B., & Eynon, R. (2020). Historical threads, missing links, and future directions in AI in education. Learning, Media and Technology, 45(3), 223-235. https://doi.org/10.1080/17439884.2020.1798995
  52. Wyatt-Smith, C., Adie, L., & Harris, L. (2024). Supporting teacher judgement and decision-making: Using focused analysis to help teachers see students, learning, and quality in assessment data. British Educational Research Journal, 50(3), 1420-1448. https://doi.org/10.1002/berj.3984
  53. Zacharis, G., & Papadakis, S. (2025). Can AI grade like a human? Validity, reliability, and fairness in university coursework assessment. Educational Process: International Journal, 19, Article e2025591. https://doi.org/10.22521/edupij.2025.19.591
  54. Zawacki-Richter, O., Marin, V. I., Bond, M & Gouverneur, F. (2019). Systematic review of AI applications in higher education. International Journal of Educational Technology in Higher Education, 16, Article 39. https://doi.org/10.1186/s41239-019-0171-0