Research Article

Explore the effectiveness of a multi-level assessment protocol for online learning

Ya Mo 1 * , Eulho Jung 2 , Meehyun Yoon 3
More Detail
1 Department of Educational Foundations, Leadership, and Technology, Auburn University, Auburn, AL, USA2 Department of Health Professions Education, School of Medicine, Uniformed Services University of the Health Sciences, Bethesda, MD, USA3 Department of Education, Chungbuk National University, Cheongju, SOUTH KOREA* Corresponding Author
Contemporary Educational Technology, 17(4), October 2025, ep615, https://doi.org/10.30935/cedtech/17547
Published: 15 December 2025
OPEN ACCESS   495 Views   677 Downloads
Download Full Text (PDF)

ABSTRACT

This study examines the effectiveness of a multi-level assessment protocol for online learning, particularly the use of reflection and self-assessment as learning and assessment tools, by capturing and interpreting five graduate students’ experiences in an online assessment course. Data were collected through surveys, reflection questions, self-assessments, pre-assessments, and module assessments. Likert-scale survey questions and selected-response reflection questions or assessment items were analyzed by counting responses or tallying correct answers. Open-ended survey questions, reflection questions, and assessment items were analyzed qualitatively using thematic analysis. Results reveal that combining traditional tests with reflection enhances assessments as learning tools, emphasizing the necessity of carefully formulated reflection questions. For self-assessment with constructed-response tests, providing students with illustrative responses and detailed rubrics that clearly state the number and scope of key points expected in responses can help standardize students’ subjective judgments. Furthermore, the unanimous preference for hands-on projects among students and the frustration expressed by some regarding asynchronous discussions underscore the importance of offering diverse, flexible, and engaging activities across various modalities to support learning outcomes. Overall, the findings affirm the enhancement of assessments as learning tools through the amalgamation of traditional tests with reflection and self-assessment, while also pinpointing the need for thoughtful implementation.
Keywords: online learning, assessment, reflection, self-assessment

CITATION (APA)

Mo, Y., Jung, E., & Yoon, M. (2025). Explore the effectiveness of a multi-level assessment protocol for online learning. Contemporary Educational Technology, 17(4), ep615. https://doi.org/10.30935/cedtech/17547

REFERENCES

  1. Akhtar, H., & Kovacs, K. (2024). Measurement precision and user experience with adaptive versus non-adaptive psychometric tests. Personality and Individual Differences, 225, Article 112675. https://doi.org/10.1016/j.paid.2024.112675
  2. Alleyne Bayne, G., & Inan, F. A. (2022). Development of the online course overload indicator and the student mental fatigue survey. The International Review of Research in Open and Distributed Learning, 23(4), 75-92. https://doi.org/10.19173/irrodl.v23i4.6223
  3. Arabyat, R. M., Qawasmeh, B. R., Al-Azzam, S. I., Nusair, M. B., & Alzoubi, K. H. (2022). Faculty members’ perceptions and attitudes towards anti-plagiarism detection tools: Applying the theory of planned behavior. Journal of Empirical Research on Human Research Ethics, 17(3), 275-283. https://doi.org/10.1177/15562646221078655
  4. Arend, B. (2007). Course assessment practices and student learning strategies in online courses. Journal of Asynchronous Learning Networks, 11(4), 3-13.
  5. Ash, S. L., & Clayton, P. H. (2009). Generating, deepening, and documenting learning: The power of critical reflection for applied learning. Journal of Applied Learning in Higher Education, 1(1), 25-48. https://doi.org/10.57186/jalhe_2009_v1a2p25-48
  6. Beebe, R., Vonderwell, S., & Boboc, M. (2010). Emerging patterns in transferring assessment practices from F2F to online environments. Electronic Journal of e-Learning, 8(1), 1-12.
  7. Benabbes, S., Algazo, M. A., & Alghazo, S. M. (2025). Exploring the impact of digital scaffolding on collaborative writing practices. Humanities and Social Sciences Communications, 12, Article 1606. https://doi.org/10.1057/s41599-025-05606-0
  8. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policies, and Practice, 5(1), 7-74. https://doi.org/10.1080/0969595980050102
  9. Bloom, B. S. (1956). Taxonomy of educational objectives. Vol. 1: Cognitive domain. McKay.
  10. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
  11. Chen, N., Wei, C., Wu, K., & Uden, L. (2009). Effects of high-level prompts and peer assessment on online learners’ reflection levels. Computers & Education, 52, 283-291. https://doi.org/10.1016/j.compedu.2008.08.007
  12. Creswell, J. (1998). Qualitative inquiry and research design: Choosing among five traditions. SAGE.
  13. Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2020). Implications for educational practice of the science of learning and development. Applied Developmental Science, 24(2), 97-140. https://doi.org/10.1080/10888691.2018.1537791
  14. Davies, J., & Graff, M. (2005). Performance in e-learning: Online participation and student grades. British Journal of Educational Technology, 36, 657-663. https://doi.org/10.1111/j.1467-8535.2005.00542.x
  15. Diarsini, M. S., Artini, L. P., Padmadewi, N. N., Ratminingsih, N. M., Utami, I. G. A. L. P., & Marsakawati, N. P. E. (2022). Challenges and opportunities of online assessment implementation during COVID-19 pandemic in Indonesia based on recent studies. European Journal of Education and Pedagogy, 3(6), 82-88. https://doi.org/10.24018/ejedu.2022.3.6.421
  16. Gomez, M. J., Ruipérez-Valiente, J. A., & Clemente, F. J. G. (2022). A systematic literature review of game-based assessment studies: Trends and challenges. IEEE Transactions on Learning Technologies, 16(4), 500-515. https://doi.org/10.1109/TLT.2022.3226661
  17. Guerrero-Roldán, A. E., & Noguera, I. (2018). A model for aligning assessment with competences and learning activities in online courses. The Internet and Higher Education, 38, 36-46. https://doi.org/10.1016/j.iheduc.2018.04.005
  18. Hafeez, M., Tahira, F., & Leghari, M. F. A. (2021). Challenges faced by the teachers and students in online learning during COVID-19. Pakistan Journal of Educational Research and Evaluation, 9(1), 83-102. https://doi.org/10.21831/cp.v41i1.35411
  19. Harsy, A., & Hoofnagle, A. (2020). Comparing mastery-based testing with traditional testing in Calculus II. International Journal for the Scholarship of Teaching and Learning, 14(2), Article 10. https://doi.org/10.20429/ijsotl.2020.140210
  20. Heil, J., & Ifenthaler, D. (2023). Online assessment in higher education: A systematic review. Online Learning, 27(1), 187-218. https://doi.org/10.24059/olj.v27i1.3398
  21. Hew, K. F., & Cheung, W. S. (2014). Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45-58. https://doi.org/10.1016/j.edurev.2014.05.001
  22. Holden, O. L., Norris, M. E., & Kuhlmeier, V. A. (2021). Academic integrity in online assessment: A research review. Frontiers in Education, 6. https://doi.org/10.3389/feduc.2021.639814
  23. Kayler, M., & Weller, K. (2007). Pedagogy, self-assessment, and online discussion groups. Journal of Educational Technology & Society, 10(1), 136-147.
  24. Kearns, L. (2012). Student assessment in online learning: Challenges and effective practices. Journal of Online Learning and Teaching, 8(3), 198-208.
  25. Keppell,M., Au, E., Ma, A., & Chan, C. (2006). Peer learning and learning-oriented assessment in technology-enhanced environments. Assessment & Evaluation in Higher Education, 31(4), 453-464. https://doi.org/10.1080/02602930600679159
  26. Kim, K. J., & Frick, T. W. (2011). Changes in student motivation during online learning. Journal of Educational Computing Research, 44(1), 1-23. https://doi.org/10.2190/EC.44.1.a
  27. Kirkwood, A., & Price, L. (2016). Technology-enabled learning implementation handbook (version 1). Commonwealth of Learning.
  28. Langenfeld, T. (2020). Internet-based proctored assessment: Security and fairness issues. Educational Measurement: Issues and Practice, 39(3), 24-27. https://doi.org/10.1111/emip.12359
  29. Li, C., Guo, J., Zhang, G., Wang, Y., Sun, Y., & Bie, R. (2019). A blockchain system for e-learning assessment and certification. In Proceedings of 2019 IEEE International Conference on Smart Internet of Things (pp. 212-219). IEEE. https://doi.org/10.1109/SmartIoT.2019.00040
  30. Miles, M. B., Huberman, A. M., & Saldana, J. (2014). Qualitative Data Analysis: A Methods Sourcebook (3rd ed.). Sage.
  31. Nicol, D. J., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102-122. https://doi.org/10.1080/02602938.2013.795518
  32. Noorbehbahani, F., Mohammadi, A., & Aminazadeh, M. (2022). A systematic review of research on cheating in online exams from 2010 to 2021. Education and Information Technologies, 27(6), 8413-8460. https://doi.org/10.1007/s10639-022-10927-7
  33. Oncu, S., & Cakir, H. (2011). Research in online learning environments: Priorities and methodologies. Computers & Education, 57(1), 1098-1108. https://doi.org/10.1016/j.compedu.2010.12.009
  34. Ortega-Ruipérez, B., & Correa-Gorospe, J. M. (2024). Peer assessment to promote self-regulated learning with technology in higher education: Systematic review for improving course design. Frontiers in Education, 9. https://doi.org/10.3389/feduc.2024.1376505
  35. Öztürk, M. (2024). Rethinking online assessment quality from pre-service teachers perspectives. Open Praxis, 16(4), 696-711. https://doi.org/10.55982/openpraxis.16.4.689
  36. Ramaswami, G., Susnjak, T., & Mathrani, A. (2023). Effectiveness of a learning analytics dashboard for increasing student engagement levels. Journal of Learning Analytics, 10(3), 115-134. https://doi.org/10.18608/jla.2023.7935
  37. Reeves, T. C. (2000). Alternative assessment approaches for online learning environments in higher education. Journal of Educational Computing Research, 23(1), 101-111. https://doi.org/10.2190/GYMQ-78FA-WMTX-J06C
  38. Rhode, J., Richter, S., & Miller, T. (2017). Designing personalized online teaching professional development through self-assessment. TechTrends, 61(5), 444-451. https://doi.org/10.1007/s11528-017-0211-3
  39. Roberts, T. S. (Ed.). (2006). Self, peer and group assessment in e-learning. IGI Global. https://doi.org/10.4018/978-1-59140-965-6
  40. Salas-Bustos, D. A., Coral-Padilla, S. J., Bustos-Lozano, H. L., & Belén, M. (2025). The role of formative assessment in higher education: Strategies to improve learning and knowledge retention. Pakistan Journal of Life and Social Sciences, 23, 5441-5455. https://doi.org/10.57239/PJLSS-2025-23.1.00425
  41. Swan, K. (2001). Virtual interaction: Design factors affecting student satisfaction and perceived learning in asynchronous online courses. Distance Education, 22(2), 306-331. https://doi.org/10.1080/0158791010220208
  42. The University of Wisconsin-Madison. (n.d). Best practices and sample questions for course evaluation surveys. The University of Wisconsin-Madison. https://assessment.wisc.edu/best-practices-and-sample-questions-for-course-evaluation-surveys/
  43. Tsai, F.-H., Tsai, C.-C., & Lin, K.-Y. (2015). The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment. Computers & Education, 81, 259-269. https://doi.org/10.1016/j.compedu.2014.10.013
  44. Tseng, S.-C., & Tsai, C.-C. (2010). Taiwan college students’ self-efficacy and motivation of learning in online peer assessment environments. The Internet and Higher Education, 13(3), 164-169. https://doi.org/10.1016/j.iheduc.2010.01.001
  45. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197-221. https://doi.org/10.1080/00461520.2011.611369
  46. Vlachopoulos, D., & Makri, A. (2024). A systematic literature review on authentic assessment in higher education: Best practices for the development of 21st century skills, and policy considerations. Studies in Educational Evaluation, 83, Article 101425. https://doi.org/10.1016/j.stueduc.2024.101425
  47. Vonderwell, S. Liang, X., & Alderman, K. (2007) Asynchronous discussions and assessment in online learning, Journal of Research on Technology in Education, 39(3), 309-328. https://doi.org/10.1080/15391523.2007.10782485
  48. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
  49. Wang, S., Wang, F., Zhu, Z., Wang, J., Tran, T., & Du, Z. (2024). Artificial intelligence in education: A systematic literature review. Expert Systems with Applications, 252, Article 124167. https://doi.org/10.1016/j.eswa.2024.124167
  50. Webb, N. L. (2002). Alignment study in language arts, mathematics, science, and social studies of state standards and assessment for four states. Council of Chief State School Officers.
  51. Yang, H., & Wong, R. (2024). An in-depth literature review of e-portfolio implementation in higher education: Processes, barriers, and strategies. Issues and Trends in Learning Technologies, 12(1). https://doi.org/10.2458/itlt.5809
  52. Yin, R. K. (1989). Case study research: Design and methods. SAGE.
  53. Zhang, J., Desrochers, M. N., & Fensken, M. (2025). Evaluation of teacher, self-assessment, versus combined feedback to increase students’ behavioral observation skills. International Journal of Teaching and Learning in Higher Education, 35(1), Article 4.
  54. Zhang, P., & Tur, G. (2024). A systematic review of e-portfolio use during the pandemic: Inspiration for post-COVID-19 practices. Open Praxis, 16(3), 429-444. https://doi.org/10.55982/openpraxis.16.3.656
  55. Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13-39). Academic Press. https://doi.org/10.1016/B978-012109890-2/50031-7