Research Article

ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude

Carmen Köhler 1 * , Johannes Hartig 1
More Detail
1 DIPF | Leibniz Institute for Research and Information in Education, Frankfurt, GERMANY* Corresponding Author
Contemporary Educational Technology, 16(4), October 2024, ep528, https://doi.org/10.30935/cedtech/15144
Published Online: 10 September 2024, Published: 01 October 2024
OPEN ACCESS   1421 Views   1792 Downloads
Download Full Text (PDF)

ABSTRACT

Since ChatGPT-3.5 has been available to the public, the potentials and challenges regarding chatbot usage in education have been widely discussed. However, little evidence exists whether and for which purposes students even apply generative AI tools. The first main purpose of the present study was to develop and test scales that assess students’ (1) knowledge about ChatGPT, (2) actual ChatGPT usage and perceived value of use, and (3) attitude towards ChatGPT. Our second aim was to examine the intercorrelations between these scales, and to investigate differences (a) across five academic fields (i.e., human sciences, social sciences, teaching profession, health sciences, and law and economics) and (b) between stages of education (i.e., number of semesters). N = 693 students from various German universities participated in our online survey. Quality checks (Cronbach’s alpha, MacDonald’s omega, and confirmatory factor analyses) show satisfactory results for all scales. The scales all positively relate to each other, except for the knowledge and attitude scales. This means that more knowledge about ChatGPT is connected to a less favorable attitude regarding the generative AI tool. Lastly, MANOVA and subsequent Bonferroni corrected ANOVA tests show that ChatGPT is mostly used by law and economics students, and most frequently by students in the third year of higher education.
Keywords: ChatGPT in higher education, student knowledge, student use, student attitude, scale development, assessment, large language models (LLMs)

CITATION (APA)

Köhler, C., & Hartig, J. (2024). ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude. Contemporary Educational Technology, 16(4), ep528. https://doi.org/10.30935/cedtech/15144

REFERENCES

  1. Abdaljaleel, M., Barakat, M., Alsanafi, M., Salim, N. A., Abazid, H., Malaeb, D., Mohammed, A. H., Hassan, B. A. R., Wayyes, A. M., Farhan, S. S., Khatib, S. E., Rahal, M., Sahban, A., Abdelaziz, D. H., Mansour, N. O., AlZayer, R., Khalil, R., Fekih-Romdhane, F., Hallit, R., …, & Sallam, M. (2024). A multinational study on the factors influencing university students’ attitudes and usage of ChatGPT. Scientific Reports, 14(1983), 1–14. https://doi.org/10.1038/s41598-024-52549-8
  2. Alkaissi, H., & Mcfarlane, S. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15. https://doi.org/10.7759/cureus.35179
  3. Banihashem, S. K., Kerman, N. T., Noroozi, O., Moon, J., & Drachsler, H. (2024). Feedback sources in essay writing: Peer-generated or AI-generated feedback? International Journal of Educational Technology in Higher Education, 21(1), Article 23. https://doi.org/10.1186/s41239-024-00455-4
  4. Bernaards, C. A., & Jennrich, R. I. (2005). Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educational and Psychological Measurement, 65(5), 676–696. https://doi.org/10.1177/0013164404272507
  5. Bonsu, E. M., & Baffour-Koduah, D. (2023). From the consumers’ side: Determining students’ perception and intention to use ChatGPT in Ghanaian higher education. Journal of Education, Society & Multiculturalism, 4(1), 1–29. https://doi.org/10.2478/jesm-2023-0001
  6. Chan, C. K. Y., & Hu, W. (2023). Students’ voices on generative AI: Perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00411-8
  7. Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148
  8. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
  9. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
  10. Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2023). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460–474. https://doi.org/10.1080/14703297.2023.2195846
  11. Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. https://doi.org/10.3758/BRM.41.4.1149
  12. Fauzi, F., Tuhuteru, L., Sampe, F., Ausat, A. M. A., & Hatta, H. R. (2023). Analysing the role of ChatGPT in improving student productivity in higher education. Journal on Education, 5(4), 14886–14891. https://doi.org/10.31004/joe.v5i4.2563
  13. Firat, M. (2023). What ChatGPT means for universities: Perceptions of scholars and students. Journal of Applied Learning and Teaching, 6(1), 57–63. https://doi.org/10.37074/jalt.2023.6.1.22
  14. Gasser, U., Cortesi, S., Malik, M. M., & Lee, A. (2012). Youth and digital media: From credibility to information quality. SSRN. https://doi.org/10.2139/ssrn.2005272
  15. Grájeda, A., Burgos, J., Córdova, P., & Sanjinés, A. (2023). Assessing student-perceived impact of using artificial intelligence tools: Construction of a synthetic index of application in higher education. Cogent Education, 11(1). https://doi.org/10.1080/2331186X.2023.2287917
  16. Hessisches Kultusministerium. (2023). Handreichung “Künstliche Intelligenz (KI) in Schule und Unterricht” [Handout “artificial intelligence (AI) in schools and teaching”]. Digitale Schule Hessen. https://digitale-schule.hessen.de/unterricht-und-paedagogik/handreichung-kuenstliche-intelligenz-ki-in-schule-und-unterricht
  17. Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453. https://doi.org/10.1037/1082-989X.3.4.424
  18. Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., …, & Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103. https://doi.org/10.1016/j.lindif.2023.102274
  19. Koltay, T. (2011). The media and the literacies: Media literacy, information literacy, digital literacy. Media, Culture & Society, 33(2), 211–221. https://doi.org/10.1177/0163443710393382
  20. Kooli, C. (2023). Chatbots in education and research: A critical examination of ethical implications and solutions. Sustainability, 15(7), Article 5614. https://doi.org/10.3390/su15075614
  21. Küchemann, S., Steinert, S., Kuhn, J., Avila, K., & Ruzika, S. (2024). Large language models—Valuable tools that require a sensitive integration into teaching and learning physics. The Physics Teacher, 62(5), 400–402. https://doi.org/10.1119/5.0212374
  22. Laupichler, M. C., Aster, A., Haverkamp, N., & Raupach, T. (2023). Development of the “scale for the assessment of non-experts’ AI literacy” – An exploratory factor analysis. Computers in Human Behavior Reports, 12, Article 100338. https://doi.org/10.1016/j.chbr.2023.100338
  23. Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education, 21(2). https://doi.org/10.1016/j.ijme.2023.100790
  24. Liu, S.-H., Liao, H.-L., & Pratt, J.-A. (2008). Impact of media richness and flow on e-learning technology acceptance. Computers and Education, 52(3), 599–607. https://doi.org/10.1016/j.compedu.2008.11.002
  25. Maurer, M., Schemer, C., Zlatkin-Troitschanskaia, O., & Jitomirski, J. (2019). Positive and negative media effects on university students’ learning: Preliminary findings and a research program. In O. Zlatkin-Troitschanskaia (Ed.), Frontiers and advances in positive learning in the age of information (PLATO) (pp. 109–119). Springer. https://doi.org/10.1007/978-3-030-26578-6_8
  26. Mills, A., Bali, M., & Eaton, L. (2023). How do we respond to generative AI in education? Open educational practices give us a framework for an ongoing process. Journal of Applied Learning and Teaching, 6(1), 16–30. https://doi.org/10.37074/jalt.2023.6.1.34
  27. Mogali, S. R. (2023). Initial impressions of ChatGPT for anatomy education. Anatomical Sciences Education, 17(2), 444–447. https://doi.org/10.1002/ase.2261
  28. Molerov, D., Zlatkin-Troitschanskaia, O., Nagel, M.-T., Brückner, S., Schmidt, S., & Shavelson, R. J. (2020). Assessing university students’ critical online reasoning ability: A conceptual and assessment framework with preliminary evidence. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.577843
  29. Murray, M. C., & Pérez, J. (n.d.). Unraveling the digital literacy paradox: How higher education fails at the fourth literacy. Issues in Informing Science and Information Technology, 11, 85–100. https://doi.org/10.28945/1982
  30. Nagel, M.-T., Schäfer, S., Zlatkin-Troitschanskaia, O., Schemer, C., Maurer, M., Molerov, D., Schmidt, S., & Brückner, S. (2020). How do university students’ web search behavior, website characteristics, and the interaction of both influence students’ critical online reasoning? Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.565062
  31. National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. The National Academies Press. https://doi.org/10.17226/13398
  32. Neumann, M., Rauschenberger, M., & Schön, E.-M. (2023). “We need to talk about ChatGPT”: The future of AI and higher education. In Proceedings of the IEEE/ACM 5th International Workshop on Software Engineering Education for the Next Generation (pp. 29–32). IEEE. https://doi.org/10.1109/SEENG59157.2023.00010
  33. Noroozi, O., Soleimani, S., Farrokhnia, M., & Banihashem, S. K. (2024). Generative AI in education: Pedagogical, theoretical, and methodological perspectives. International Journal of Technology in Education, 7(3), 373–385. https://doi.org/10.46328/ijte.845
  34. O’Dea, X., & O’Dea, M. (2023). Is artificial intelligence really the next big thing in learning and teaching in higher education? A conceptual paper. Journal of University Teaching and Learning Practice, 20(5). https://doi.org/10.53761/1.20.5.05
  35. OpenAI. (2023). ChatGPT. https://chat.openai.com
  36. PISA. (2023). Learning in the digital world–PISA. https://www.oecd.org/pisa/innovation/learning-digital-world/
  37. R Core Team. (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/
  38. Rasul, T., Nair, S., Kalendra, D., Robin, M., Santini, F., Ladeira, W., Sun, M., Day, I., Rather, A., & Heathcote, L. (2023). The role of ChatGPT in higher education: Benefits, challenges, and future research directions. Journal of Applied learning & Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.29
  39. Revelle, W. (2023). psych: Procedures for psychological, psychometric, and personality research. Northwestern University, Evanston, Illinois. https://CRAN.R-project.org/package=psych
  40. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
  41. Rudolph, J., Tan, S., & Aspland, T. (2023). Fully automated luxury communism or Turing trap? Graduate employability in the generative AI age. Journal of Applied Learning and Teaching, 6(1), 7–15. https://doi.org/10.37074/jalt.2023.6.1.35
  42. Rudolph, J., Tan, S., & Tan, S. (2023a). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 342–363. https://doi.org/10.37074/jalt.2023.6.1.9
  43. Rudolph, J., Tan, S., & Tan, S. (2023b). War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. Journal of Applied Learning and Teaching, 6(1), 364–389. https://doi.org/10.37074/jalt.2023.6.1.23
  44. Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11(6). https://doi.org/10.3390/healthcare11060887
  45. Schepman, A., & Rodway, P. (2020). Initial validation of the general attitudes towards artificial intelligence Scale. Computers in Human Behavior Reports, 1, Article 100014. https://doi.org/10.1016/j.chbr.2020.100014
  46. Schön, E.-M., Neumann, M., Hofmann-Stölting, C., Baeza-Yates, R., & Rauschenberger, M. (2023). How are AI assistants changing higher education? Frontiers in Computer Science, 5. https://doi.org/10.3389/fcomp.2023.1208550
  47. Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023). ChatGPT and other large language models are double-edged swords. Radiology, 307(2). https://doi.org/10.1148/radiol.230163
  48. Sindermann, C., Sha, P., Zhou, M., Wernicke, J., Schmitt, H. S., Li, M., Sariyska, R., Stavrou, M., Becker, B., & Montag, C. (2021). Assessing the attitude towards artificial intelligence: Introduction of a short measure in German, Chinese, and English language. KI – Künstliche Intelligenz, 35(1), 109–118. https://doi.org/10.1007/s13218-020-00689-0
  49. Sok, S., & Heng, K. (2023). ChatGPT for education and research: A review of benefits and risks. SSRN. https://doi.org/10.2139/ssrn.4378735
  50. Sparks, J. R., Katz, I. R., & Beile, P. M. (2016). Assessing digital information literacy in higher education: A review of existing frameworks and assessments with recommendations for next-generation assessment. ETS Research Report Series, 2, 1–33. https://doi.org/10.1002/ets2.12118
  51. Strzelecki, A. (2023). To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interactive Learning Environments. https://doi.org/10.1080/10494820.2023.2209881
  52. Sullivan, M., Kelly, A., & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning. Journal of Applied Learning and Teaching, 6(1), 31–40. https://doi.org/10.37074/jalt.2023.6.1.17
  53. U.S. Department of Education, Office of Educational Technology. (2023). Artificial intelligence and the future of teaching and learning. https://www2.ed.gov/documents/ai-report/ai-report.pdf
  54. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540
  55. Venkatesh, V., Thong, J. Y. L., & Xu, X. (2012). Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Quarterly, 36(1), 157–178. https://doi.org/10.2307/41410412
  56. Wang, B., Rau, P.-L. P., & Yuan, T. (2022). Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology, 42(9), 1324–1337. https://doi.org/10.1080/0144929X.2022.2072768
  57. Xames, M. D., & Shefa, J. (2023). ChatGPT for research and publication: Opportunities and challenges. Journal of Applied Learning & Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.20
  58. Zlatkin-Troitschanskaia, O., Hartig, J., Goldhammer, F., & Krstev, J. (2021). Students’ online information use and learning progress in higher education – A critical literature review. Studies in Higher Education, 46(10), 1996–2021. https://doi.org/10.1080/03075079.2021.1953336