Research Article

Development of an AI literacy assessment for non-technical individuals: What do teachers know?

Lu Ding 1 * , Sohee Kim 1 , R. Allan Allday 2
More Detail
1 Department of Counselling and Instructional Sciences, University of South Alabama, Mobile, AL, USA2 Department of Leadership and Teacher Education, University of South Alabama, Mobile, AL, USA* Corresponding Author
Contemporary Educational Technology, 16(3), July 2024, ep512, https://doi.org/10.30935/cedtech/14619
Published Online: 14 May 2024, Published: 01 July 2024
OPEN ACCESS   1944 Views   1942 Downloads
Download Full Text (PDF)

ABSTRACT

With the exponential development and vast interest in artificial intelligence (AI), the global economic impact of AI is expected to reach $15.7 trillion by 2030. While AI has infiltrated everyday life, a lack of knowledge of what AI is and how AI works is ubiquitous across all ages and professions. Teaching AI literacy to non-technical individuals has become imperative and requires immediate attention, however, assessing AI literacy has heavily relied on subjective measurements such as qualitative assessment and self-reported surveys, which may lead to biased results. This study contributes to the field by developing and validating an assessment created based on a well-established AI literacy framework. A total of 196 responses were collected from pre-and in-service teachers in the United States, and 186 responses were included in the analysis to validate the assessment. The final assessment comprises 25 objective-based items reduced from an originally 31-item assessment. Both experts’ insights were sought, and statistical methodology was employed to ensure the validity of the assessment. The results indicate that pre-and in-service teachers have a moderate level of AI literacy and in-service teachers performed slightly better than pre-service teachers on our assessment. Inconsistent answers across various AI concepts indicate that teachers may possess an even more ambiguous understanding of certain AI concepts.
Keywords: AI literacy, assessment, non-technical, teachers, objective-based

CITATION (APA)

Ding, L., Kim, S., & Allday, R. A. (2024). Development of an AI literacy assessment for non-technical individuals: What do teachers know?. Contemporary Educational Technology, 16(3), ep512. https://doi.org/10.30935/cedtech/14619

REFERENCES

  1. Antonenko, P., & Abramowitz, B. (2023). In-service teachers’ (mis)conceptions of artificial intelligence in K-12 science education. Journal of Research on Technology in Education, 55(1), 64-78. https://doi.org/10.1080/15391523.2022.2119450
  2. Baum, S. D., Goertzel, B., & Goertzel, T. G. (2011). How long until human-level AI? Results from an expert assessment. Technological Forecasting and Social Change, 78(1), 185-195. https://doi.org/10.1016/j.techfore.2010.09.006
  3. Brooks, R. A. (1991). Intelligence without representation. Artificial Intelligence, 47. https://doi.org/10.1016/0004-3702(91)90053-M
  4. Casal-Otero, L., Catala, A., Fernández-Morante, C., Taboada, M., Cebreiro, B., & Barro, S. (2023). AI literacy in K-12: A systematic literature review. International Journal of STEM Education, 10(1), 29. https://doi.org/10.1186/s40594-023-00418-7
  5. Cattell, R. B. (1966). The scree plot test for the number of factors. Multivariate Behavioral Research, 1, 140-161. https://doi.org/10.1207/s15327906mbr0102_10
  6. Cave, S., Coughlan, K., & Dihal, K. (2019). “Scary robots” examining public responses to AI. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (pp. 331-337). ACM. https://doi.org/10.1145/3306618.3314232
  7. CBSE Question Bank. (2023). Introduction to AI: Foundational concepts class 10 artificial intelligence. Department of Skill Education.
  8. Cetindamar, D., Kitto, K., Wu, M., Zhang, Y., Abedin, B., & Knight, S. (2022). Explicating AI literacy of employees at digital workplaces. IEEE Transactions on Engineering Management, 71, 810-823. https://doi.org/10.1109/TEM.2021.3138503
  9. Chubb, J., Reed, D., & Cowling, P. (2022). Expert views about missing AI narratives: Is there an AI story crisis? AI and Society. https://doi.org/10.1007/s00146-022-01548-2
  10. Davis, R., Shrobe, H., & Szolovits, P. (1993). What is a knowledge representation? American Association for Artificial Intelligence. https://courses.csail.mit.edu/6.803/pdf/davis.pdf
  11. De Ayala, R. J. (2013). The theory and practice of item response theory. Guilford Publications. https://doi.org/10.12691/rpbs-7-1-4
  12. Ding, L., Li, T., Jiang, S., & Gapud, A. (2023). Students’ perceptions of using ChatGPT in a physics class as a virtual tutor. International Journal of Educational Technology in Higher Education, 20(1), 63. https://doi.org/10.1186/s41239-023-00434-1
  13. Ding, L., Li, T., & Turkson, A. (In review). (Mis)conceptions and perceptions of artificial intelligence: A scoping review. Manuscript Submitted for Publication.
  14. Goel, A. K., & Davies, J. (2011). Artificial intelligence. Cambridge University Press. https://doi.org/10.1017/CBO9780511977244.024
  15. Gorsuch, R. L. (1973). Using Bartlett’s significance test to determine the number of factors to extract. Educational and Psychological Measurement, 33(2), 361-364. https://doi.org/10.1177/001316447303300216
  16. Hautea, S., Dasgupta, S., & Hill, B. M. (2017). Youth perspectives on critical data literacies. In proceedings of the Conference on Human Factors in Computing Systems (pp. 919-930). https://doi.org/10.1145/3025453.3025823
  17. Hornberger, M., Bewersdorff, A., & Nerdel, C. (2023). What do university students know about Artificial Intelligence? Development and validation of an AI literacy test. Computers and Education: Artificial Intelligence, 5. https://doi.org/10.1016/j.caeai.2023.100165
  18. IBM Corp. (2021). IBM SPSS Statistics for Windows. IBM Corp.
  19. Jones-Jang, S. M., & Park, Y. J. (2023). How do people react to AI failure? Automation bias, algorithmic aversion, and perceived controllability. Journal of Computer-Mediated Communication, 28(1), 1-8. https://doi.org/10.1093/jcmc/zmac029
  20. Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39, 31-36. https://doi.org/10.1007/BF02291575
  21. Kline, P. (2015). A handbook of test construction (psychology revivals): Introduction to psychometric design. Routledge. https://doi.org/10.4324/9781315695990
  22. Kong, S.-C., Man-Yin Cheung, W., & Zhang, G. (2023). Evaluating an artificial intelligence literacy program for developing university students’ conceptual understanding, literacy, empowerment, and ethical awareness. Educational Technology & Society, 26(1), 16-30. https://doi.org/10.30191/ETS.202301_26(1).0002
  23. Laupichler, M. C., Aster, A., Schirch, J., & Raupach, T. (2022). Artificial intelligence literacy in higher and adult education: A scoping literature review. In Computers and education: Artificial intelligence. Elsevier. https://doi.org/10.1016/j.caeai.2022.100101
  24. Lee, I., Ali, S., Zhang, H., Dipaola, D., & Breazeal, C. (2021). Developing middle school students’ AI literacy. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (pp. 191-197). ACM. https://doi.org/10.1145/3408877.3432513
  25. Lindner, A., & Berges, M. (2020). Can you explain AI to me? Teachers’ pre-concepts about artificial intelligence. In Proceedings of the 2020 IEEE Frontiers in Education Conference (pp. 1-9). IEEE. https://doi.org/10.1109/FIE44824.2020.9274136
  26. Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. In Proceedings of the Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376727
  27. Long, D., Teachey, A., & Magerko, B. (2022). Family learning talk in AI literacy learning activities. In Proceedings of the Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3491102.3502091
  28. Maitz, K., Fessl, A., Pammer-Schindler, V., Kaiser, R., & Lindstaedt, S. (2022). What do construction workers know about artificial intelligence? An exploratory case study in an Austrian SME. In Proceedings of the ACM International Conference (pp. 389-393). https://doi.org/10.1145/3543758.3547545
  29. McBride, C. (2015). Children’s literacy development. Routledge. https://doi.org/10.4324/9781315849409
  30. McCarthy, J. (2007). From here to human-level AI. Artificial Intelligence, 171(18), 1174-1182. https://doi.org/10.1016/j.artint.2007.10.009
  31. Mertala, P., Fagerlund, J., & Calderon, O. (2022). Finnish 5th and 6th grade students’ pre-instructional conceptions of artificial intelligence (AI) and their implications for AI literacy education. Computers and Education: Artificial Intelligence, 3. https://doi.org/10.1016/j.caeai.2022.100095
  32. Mondal, B. (2020). Artificial intelligence: State of the art. In V. E. Balas, R. Kumar, & R. Srivastava (Eds.), Recent trends and advances in artificial intelligence and internet of things (pp. 389-425). https://doi.org/10.1007/978-3-030-32644-9_32
  33. Nader, K., Toprac, P., Scott, S., & Baker, S. (2022). Public understanding of artificial intelligence through entertainment media. AI and Society, 39, 713-726. https://doi.org/10.1007/s00146-022-01427-w
  34. Ng, A. (2023). AI for everyone. DeepLearning.
  35. Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021a). AI literacy: Definition, teaching, evaluation and ethical issues. Proceedings of the Association for Information Science and Technology, 58(1), 504-509. https://doi.org/10.1002/pra2.487
  36. Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021b). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2. https://doi.org/10.1016/j.caeai.2021.100041
  37. Olari, V. (2023). Introducing artificial intelligence literacy in schools: A review of competence areas, pedagogical approaches, contexts and formats. In T. Keane, C. Lewin, T. Brinda, & R. Bottino (Eds.), Towards a collaborative society through creative learning. Springer. https://doi.org/10.1007/978-3-031-43393-1_21
  38. Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. SAGE. https://doi.org/10.4135/9781412984898
  39. Prado, J. C., & Marzal, M. Á. (2013). Incorporating data literacy into information literacy programs: Core competencies and contents. Libri, 63(2), 123-134. https://doi.org/10.1515/libri-2013-0010
  40. PwC Global. (2023). Global artificial intelligence study: Sizing the prize. https://www.pwc.com/gx/en/issues/data-and-analytics/publications/artificial-intelligence-study.html
  41. Rizopoulos, D. (2006). ITM: An R package for latent variable modeling and item response theory analyses. Journal of Statistical Software, 17(5), 1-25. https://doi.org/10.18637/jss.v017.i05
  42. Rosenman, R., Tennekoon, V., & Hill, L. G. (2011). Measuring bias in self-reported data. International Journal of Behavioral and Healthcare Research, 2(4), 320. https://doi.org/10.1504/IJBHR.2011.043414
  43. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
  44. Rovinelli, R. J., & Hambleton, R. K. (1977). On the use of content specialists in the assessment of criterion-referenced test item validity. Dutch Journal for Educational Research, 2(2), 49-60.
  45. RStudio Team. (2020). RStudio: Integrated development for R. http://www.rstudio.com/
  46. Saxton, E., Burns, R., Holveck, S., Kelley, S., Prince, D., Rigelman, N., & Skinner, E.A. (2014). A common measurement system for K-12 STEM education: Adopting an educational evaluation methodology that elevates theoretical foundations and systems thinking. Studies in Educational Evaluation, 40, 18-35. https://doi.org/10.1016/j.stueduc.2013.11.005
  47. Steiger, J. H. (1980). Statistically based tests for the number of common factors. In Proceedings of the Annual Meeting of the Psychometric Society.
  48. Su, J., Ng, D. T. K., & Chu, S. K. W. (2023). Artificial intelligence (AI) literacy in early childhood education: The challenges and opportunities. Computers and Education: Artificial Intelligence, 4. https://doi.org/10.1016/j.caeai.2023.100124
  49. Touretzky, D., Gardner-Mccune, C., Martin, F., & Seehorn, D. (2019). Envisioning AI for K-12: What should every child know about AI? In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (pp. 9795-9799). https://doi.org/10.1609/aaai.v33i01.33019795
  50. Turner, R. C., & Carlson, L. (2003). Indexes of item-objective congruence for multidimensional items. International Journal of Testing, 3(2), 163-171. https://doi.org/10.1207/S15327574IJT0302_5
  51. Williams, B. (1978). A sampler on sampling. John Wiley & Sons.
  52. Wong, G. K. W., & Huan, J. (2020). Broadening artificial intelligence education in K-12: Where to start? ACM Inroads, 11(1), 20-29. https://doi.org/https://doi.org/10.1145/3381884
  53. Yue, M., Jong, M. S. Y., & Dai, Y. (2022). Pedagogical design of K-12 artificial intelligence education: A systematic review. Sustainability, 14(23), 15620. https://doi.org/10.3390/su142315620