Research Article

Assumption Validation Process for the Assessment of Technology-Enhanced Learning

Denise Pauline-Graf 1 2 * , Susan E. Mandel 3 , Heather W. Allen 4 , Lynne E. Devnew 5
More Detail
1 Novomatic AG, Austria2 University of Phoenix, Phoenix, AZ, USA3 Lake Health, OH, USA4 SADI Solutions, LLC, MO, USA5 College of Doctoral Studies, University of Phoenix, AZ, USA* Corresponding Author
Contemporary Educational Technology, 13(4), October 2021, ep316, https://doi.org/10.30935/cedtech/11071
OPEN ACCESS   2576 Views   1251 Downloads
Download Full Text (PDF)

ABSTRACT

Technology-enhanced learning (TEL) is a broad term to denote the incorporation of digital technology to mediate activities that support education. Educational researchers need to validate critical assumptions about any new system that involves TEL early and throughout its development course to make quick, informed, de-risked decisions about the progress of TEL. We introduce an assumption validation process for educational researchers to consider utilizing when evolving ideas or prototypes of TEL. We present a preliminary study conducted in Russia of a knowledge sharing (KS) training program using digital games to illustrate the proposed four-stage process. The first stage is listing the assumptions that apply to initiatives of TEL. The second is identifying the methods most useful for testing those assumptions. The third is executing tests on each of those assumptions. The last stage is determining assumption validity. In the illustrative study, a single pilot trial was considered the appropriate approach to validate the assumptions selected in the first stage. We found that determining assumption validity requires testing many of the assumptions individually and in aggregate. Educational researchers can use this assumption validation process to assess the potential of TEL in a variety of settings before investing resources into its further development.
Keywords: instructional digital games, assumption validation, knowledge sharing, pilot trial, self-efficacy, technology-enhanced learning

CITATION (APA)

Pauline-Graf, D., Mandel, S. E., Allen, H. W., & Devnew, L. E. (2021). Assumption Validation Process for the Assessment of Technology-Enhanced Learning. Contemporary Educational Technology, 13(4), ep316. https://doi.org/10.30935/cedtech/11071

REFERENCES

  1. Al-Azawi, R., Al-Faliti, F., & Al-Blushi, M. (2016). Educational gamification vs. game based learning: Comparative study. International Journal of Innovation, Management and Technology, 7(4), 132-136. https://doi.org/10.18178/ijimt.2016.7.4.659
  2. Apiola, M., Lokkila, E., & Laakso, M.-J. (2019). Digital learning approaches in an intermediate-level computer science course. International Journal of Information & Learning Technology, 36(5), 467-484. https://doi.org/10.1108/IJILT-06-2018-0079
  3. Arnold, D., Burns, K., Adhikari, N., Kho, M., Meade, M., & Cooke, D. (2009). The design and interpretation of pilot trials in clinical research in critical care. Critical Care Medicine, 37(1 Suppl), 69-74. https://doi.org/10.1097/CCM.0b013e3181920e33
  4. Baack, D., Dow, D., Parente, R., & Bacon, D. (2015). Confirmation bias in individual-level perceptions of distance: An experimental investigation. Journal of International Business Studies, 46(8), 938-959. https://doi.org/10.1057/jibs.2015.19
  5. Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191-215. https://doi.org/10.1037/0033-295X.84.2.191
  6. Bartolomé, A., Castañeda, L., & Adell, J. (2018). Personalisation in educational technology: The absence of underlying pedagogies. International Journal of Educational Technology in Higher Education, 15(14), 1-17. https://doi.org/10.1186/s41239-018-0095-0
  7. Bayne, S. (2015). What’s the matter with ‘technology enhanced learning’? Learning, Media and Technology, 40(1), 5-20. https://doi.org/10.1080/17439884.2014.915851
  8. Beckstead, J. (2013). On measurements and their quality: Paper 2: Random measurement error and the power of statistical tests. International Journal of Nursing Studies, 50(10), 1416-1422. https://doi.org/10.1016/j.ijnurstu.2013.04.010
  9. Bennett, S., & Oliver, M. (2011). Talking back to theory: The missed opportunities in learning technology research. Research in Learning Technology, 19(3), 179-189. https://doi.org/10.3402/rlt.v19i3.17108
  10. Black, J., Kim, K., Rhee, S., Wang, K., & Sakchutchawan, S. (2019). Self-efficacy and emotional intelligence: Influencing team cohesion to enhance team performance. Team Performance Management: An International Journal, 25(1/2), 100-119. https://doi.org/10.1108/TPM-01-2018-0005
  11. Chen, M.-L., & Lin, C.-P. (2013). Assessing the effects of cultural intelligence on team knowledge sharing from a socio-cognitive perspective. Human Resource Management, 52(5), 675-695. https://doi.org/10.1002/hrm.21558
  12. Cook, D., & Ellaway, R. (2015). Evaluating technology-enhanced learning: a comprehensive framework. Medical Teacher, 37(10), 961-970. https://doi.org/10.3109/0142159X.2015.1009024
  13. Courtright, S., Colbert, A., & Choi, D. (2014). Fired up or burned out? How developmental challenge differentially impacts leader behavior. Journal of Applied Psychology, 99(4), 681-696. https://doi.org/10.1037/a0035790
  14. Cózar-Gutiérrez, R., & Sáez-López, J. (2016). Game-based learning and gamification in initial teaching training in social sciences: an experiment with MinecraftEdu. International journal of Educational technology in Higher Education, 13(1), 2. https://doi.org/10.1186/s41239-016-0003-4
  15. Crook, C., & Gu, X. (2019). How new technology is addressed by researchers in educational studies: Approaches from high-performing universities in China and the UK. British Journal of Educational Technology, 50(3), 1173-1188. https://doi.org/10.1111/bjet.12750
  16. Donaldson, S., & Grant-Vallone, E. (2002). Understanding self-report bias in organizational behavior research. Journal of Business and Psychology, 17(2), 245-260. https://doi.org/10.1023/A:1019637632584
  17. Ellaway, R. (2014). Virtual patients as activities: Exploring the research implications of an activity theoretical stance. Perspectives on Medical Education 3(4), 266-277. https://doi.org/10.1007/s40037-014-0134-z
  18. Ellaway, R., Pusic, M., Gailbraith, R., & Cameron, T. (2014a). Developing the role of big data and analytics in health professional education. Medical Teaching, 36(3), 216-222. https://doi.org/10.3109/0142159X.2014.874553
  19. Ellaway, R., Pusic, M., Yavner, S., & Kalet, A. (2014b, March 9). Context matters. Emergent variability in an effectiveness trial of online teaching modules. Medical Education, 48(4), 386-396. https://doi.org/10.1111/medu.12389
  20. Filippova, T. (2015). Priority fields of e-learning development in Russia. Procedia – Social and Behavioral Sciences, 206, 348-353. https://doi.org/10.1016/j.sbspro.2015.10.063
  21. Frye, W., & Hemmer, P. (2012). Program evaluation models and related theories: AME guide no. 67. Medical Teacher, 34(5) e288-e299. https://doi.org/10.3109/0142159X.2012.668637
  22. Giannakas, F., Kambourakis, G., Papaslouros, A., & Gritzalis, S. (2018). A critical review of 13 years of mobile game-based learning. Educational Technology Research and Development, 66(2), 341-384.
  23. Goodyear, P., & Retalis, S. (2010). Learning, technology and design. In P. Goodyear & S. Retalis (Eds.), Technology-enhanced learning: Design patterns and pattern languages (pp. 1-28). Sense Publishers.
  24. Gresalfi, M., & Barnes, J. (2016). Designing feedback in an immersive videogame: supporting mathematical engagement. Educational Technology Research and Development, 64(1), 65-86. https://doi.org/10.1007/s11423-015-9411-8
  25. Hagemeier, N., Hess, R., Hagen, K., & Sorah, E. (2014). Impact of an interprofessional communication course on nursing, medical, and pharmacy students’ communication skill self-efficacy beliefs. American Journal of Pharmaceutical Education, 78(10), 186-196. https://doi.org/10.5688/ajpe7810186
  26. Harvey, J. (1998). Evaluation cookbook. Edinburgh: Learning Technology Dissemination Initiative. Heriot-Watt University, Edinburgh.
  27. Hussein, M., Ow, S., Cheong, L., Thing, M., & Ebrahim, N. (2019). Effects of digital game-based learning on elementary science learning: A systematic review. IEEE Access, 7, 62465-62478. https://doi.org/10.1109/ACCESS.2019.2916324
  28. Hwa, S. (2018). Pedagogical change in mathematics learning: Harnessing the power of digital game-based learning. Journal of Educational Technology & Society, 21(4), 259-276.
  29. Khan, A., Ahmad, F., & Malik, M. (2017). Use of digital game based learning and gamification in secondary school science: The effect on student engagement, learning and gender difference. Education and Information Technologies, 22(6), 2767-2804. https://doi.org/10.1007/s10639-017-9622-1
  30. Kiili, K. (2005). Digital game-based learning: Towards an experiential gaming model. Internet and Higher Education, 8(1), 13-24. https://doi.org/10.1016/j.iheduc.2004.12.001
  31. Kirkwood, A., & Price, L. (2013). Examining some assumptions and limitations of research on the effects of emerging technologies for teaching and learning in higher education. British Journal of Educational Technology, 44(4), 536-543. https://doi.org/10.111/bjet.12049
  32. Kirkwood, A., & Price, L. (2014). Technology-enhanced learning and teaching in higher education: what is ‘enhanced’ and how do we know? A critical literature review. Learning, Media and Technology, 39(1), 6-36. https://doi.org/10.1080/17439884.2013.770404
  33. Koh, C., Tan, H., Tan, K., Fang, L., & Fong, F., Kan, D., Lye, S., & Wee, M. (2010). Investigating the effect of 3D simulation-based learning on the motivation and performance of engineering students. Journal of Engineering Education, 99(3), 237-251. https://doi.org/10.1002/j.2168-9830.2010.tb01059.x
  34. Kurvinen, E., Kaila, E., Laakso, M.-J., & Salakoski, T. (2020). Long term effects on technology enhanced learning: The use of weekly digital lessons in Mathematics. Informatics in Education, 19(1), 51-75. https://doi.org/10.15388/infedu.2020.04
  35. Larionova, V., Brown, K., Bystrova, T., & Sinitsyn, E. (2018). Russian perspectives of online learning technologies in higher education: An empirical study of a MOOC. Research in Comparative & International Education, 13(1), 70-91. https://doi.org/10.1177/1745499918763420
  36. Lee, Y-H. (2015). Does digital game interactivity always promote self-efficacy? Cyberpsychology, Behavior, and Social Networking, 18(11), 669-673. https://doi.org/10.1089/cyber.2015.0165
  37. Li, Z., Cheng, Y., & Liu, C. (2013). A constructionism framework for designing game-like learning systems: Its effect on different learners. British Journal of Educational Technology, 44(2), 208-224. https://doi.org/10.1111/j.1467-8535.2012.01305.x
  38. Liu, Z.-Y., Shaikh, Z., & Gazizova, F. (2020). Using the concept of game-based learning in education. International Journal of Emerging Technologies in Learning, 15(14), 53-64. https://doi.org/10.39991/ijet.v15i14.14675
  39. Niederhauser, D., Howard, S., Voogt, J., Agyei, D., Laferriere, T., Tondeaur, J., & Cox, M. (2018). Sustainability and scalability in educational technology initiatives: Research-informed practice. Technology, Knowledge and Learning, 23(3), 507-523. https://doi.org/10.1007/s10758-018-9382-z
  40. Ojedokun, O., & Idemudia, E. (2014). Examining the roles of gender and personal dispositions in attitudes toward knowledge sharing of senior administrators. Gender & Behaviour, 12(3), 5857-5867.
  41. Oliver, E. (2018). Digital game-based learning and technology-enhanced learning for theological education. Verbum et Ecclesia, 39, 1-8. https://doi.org/10.4102/ve.v39i1.1900
  42. Oliver, M. (2011). Technological determinism in educational technology research: some alternative ways of thinking about the relationship between learning and technology. Journal of Computer Assisted Learning, 27(5), 373-384. https://doi.org/10.111/j1365-2729.2011.00406.x
  43. Olowodunoye, S. (2015). Knowledge sharing behavior: The role of self-efficacy, organizational justice and organizational tenure. European Scientific Journal, 11(17), 254-264.
  44. Osman, S., Sauid, M., & Azizan, N. (2015). Knowledge sharing patterns among undergraduate students in Universiti Teknologi MARA (UiTM) Johor, Malaysia. International Journal of Management, Accounting and Economics, 2(3), 167-178.
  45. Passey, D. (2019). Technology-enhanced learning: Rethinking the term, the concept, and its theoretical background. British Journal of Educational Technology, 50(3), 972-986. https://doi.org/10.1111/bjet.12783
  46. Pauline-Graf, D., & Mandel, S. E. (2019). Defining preliminary research for digital game-based learning evaluation: Best practices. International Journal of Educational Methodology, 5(4), 623-635. https://doi.org/10.12973/ijem.5.4.623
  47. Persico, D., Pozzi, F., & Goodyear, P. (2018). Teachers as designers of TEL programs. British Journal of Educational Technology, 49(6). https://doi.org/10.1111/bjet.12706
  48. Pickering, J., Lazarus, M., & Hallam, J. (2019). A practitioner’s guide to performing a holistic evaluation of technology-enhanced learning in medical education. Medical Science Educator, 29(4), 1095-1102. https://doi.org/10.1007/s40670-019-00781-7
  49. Prosser, M., Trigwell, K., & Taylor, P. (1994). A phenomenographic study of academics’ conceptions of science learning and teaching. Learning and Instruction, 4(3), 217-232. https://doi.org/10.1016/0959-4752(94)90024-8
  50. Qian, M., & Clark, K. (2016). Game-based learning and 21st century skills: A review of recent research. Computers in Human Behavior, 63, 50-58. https://doi.org/10.1016/j.chb.2016.05.023
  51. Samuelowicz, K., & Bain, J. (1992). Conceptions of teaching held by academic teachers. Higher Education, 24(1), 93-111. https://doi.org/10.1007/BF00138620
  52. Sanders, J. (2012). Technology-enhanced learning. Education for Primary Care, 23(3), 228-229. https://doi.org/10.1080/14739879.2012.11494107
  53. Shaari, R., Rahman, S., & Rajab, A. (2014). Self-efficacy as a determined factor for knowledge sharing awareness. International Journal of Trade Economics and Finance, 5(1), 39-42. https://doi.org/10.7763/IJTEF.2014.V5.337
  54. Shahriarpour, N., & Kafi, Z. (2014). On the effect of playing digital games on Iranian intermediate EFL learners’ motivation toward learning English vocabularies. Procedia – Social and Behavioral Sciences, 98, 1738-1734. https://doi.org/10.1016/j.sbspro.2014.03.601
  55. Shao, Z., Wang, T., & Feng, Y. (2015). Impact of organizational culture and computer self-efficacy on knowledge sharing. Industrial Management & Data Systems, 115(4), 590-611. https://doi.org/10.1108/IMDS-12-2014-0377
  56. Swinkels, M., Koopman, M., & Beijaard, D. (2013). Student teachers’ development of learning focused conceptions. Teaching and Teacher Education, 34, 26-37. https://doi.org/10.1016/j.tate.2013.03.003
  57. Tang, S., Hanneghan, M., & Carter, C. (2013). A platform independent game technology model for model driven serious game development. The Electronic Journal of e-Learning, 11(1), 61-79.
  58. Tasa, K., Taggar, S., & Seijts, G. (2007). The development of collective self-efficacy in teams: A multilevel and longitudinal perspective. Journal of Applied Psychology, 92(1), 17-27. https://doi.org/10.1037/0021-9010.92.1.17
  59. Van Driel, J., & Berry, A. (2012). Teacher professional development focusing on pedagogical content knowledge. Educational Researcher, 41(1), 26-28. https://doi.org/10.3102/0013189X11431010
  60. Vas de Carvalho, C., Lopes, M., & Ramos, A. (2014). Lean games approaches — Simulation games and digital serious games. International Journal of Advanced Corporate Learning, 7(1), 11-16. https://doi.org/10.3991/ijac.v7il.3433
  61. Van Genderen, E. (2014). Strategic knowledge sharing: Culture acting as an inhibitor. Middle East Journal of Business, 9(4), 3-8.
  62. Vetushinskiy, A., & Zhukov, P. (2019). Gamification of school education in Russia: Case study. In J. Arnedo-Moreno, C. S. González, & A. Mora (Eds.), Proceedings of the 3rd International Symposium on Gamification and Games for Learning (GamiLearn’19), Barcelona, Spain. http://ceur-ws.org
  63. Walter, O., Shenaar-Golan, V., & Greenberg, Z. (2015). Effect of short-term program on academic self-efficacy in higher education. Psychology, 6(10), 1199-1215. https://doi.org/10.4236/psych.2015.610118
  64. Zawadzki, M., Danube, C., & Shields, S. (2012). How to talk about gender inequity in the workplace: Using WAGES as an experiential learning tool to reduce reactance and promote self-efficacy. Sex Roles, 67(11-12), 605-616. https://doi.org/10.1007/s11199-012-0181-z