Research Article
Applying Kano’s two-factor theory to prioritize learning analytics dashboard features for learning technology designers
More Detail
1 DTU Compute, Lyngby, DENMARK* Corresponding Author
Contemporary Educational Technology, 16(2), April 2024, ep496, https://doi.org/10.30935/cedtech/14286
Published Online: 26 February 2024, Published: 01 April 2024
OPEN ACCESS 1472 Views 1043 Downloads
ABSTRACT
Existing methods for software requirements elicitation, five-point Likert scales and voting methods for requirements prioritization, and usability and user experience evaluation methods do not enable prioritizing the learning analytics dashboard requirements. Inspired by management and product design field, this research applies Kano’s two-factor theory to prioritize the features of learning analytics dashboards (LADs) of adaptive learning platform (ALP) called RhapsodeTM learner, based on students’ perceived usefulness to support designers’ decision-making. Comparing usability and user experience methods for evaluating LAD features, this paper contributes with the protocol and a case applying Kano method for evaluating the perceived importance of the dashboards in ALP. The paper applies Kano’s two-factor questionnaire on the 13 LADs features of RhapsodeTM learner. Responses from 17 students are collected using a questionnaire, which is used to showcase the strength of the two-factor theory through five tabular and graphical techniques. Through these five tabular and graphical techniques, we demonstrate the application and usefulness of the method as designers and management are often carried away by the possibilities of insights instead of actual usefulness. The results revealed a variation in the categorization of LADs depending on the technique employed. As the complexity of the techniques increases, additional factors that indicate data uncertainty are gradually incorporated, clearly highlighting the growing requirement for data. In the case of RhapsodeTM learner platform, results based on the students responses show that 11 of 13 LADs being excluded due to low significance level in categorization (technique 1) and low response rate.
CITATION (APA)
Tretow-Fish, T. A. B., & Khalid, M. S. (2024). Applying Kano’s two-factor theory to prioritize learning analytics dashboard features for learning technology designers. Contemporary Educational Technology, 16(2), ep496. https://doi.org/10.30935/cedtech/14286
REFERENCES
- Ahn, J., Campos, F., Hays, M., & Digiacomo, D. (2019). Designing in context: Reaching beyond usability in learning analytics dashboard design. Journal of Learning Analytics, 6(2), 70-85. https://doi.org/10.18608/jla.2019.62.5
- Area9 (2024, January 1). "RhapsodeTM Learner": Area9 RhapsodeTM. https://area9lyceum.com/the-platform/
- Bauk, S., Šćepanović, S., & Kopp, M. (2014). Estimating students’ satisfaction with web based learning system in blended learning environment. Education Research International, 2014, 731720. https://doi.org/10.1155/2014/731720
- Baxter, K., Courage, C., & Caine, K. (2015). Choosing a user experience research activity in understanding your users: A practical guide to user research methods. Morgan Kaufmann. https://doi.org/10.1016/B978-0-12-800232-2.00005-5
- Benyon, D. (2019). Designing user experience. Pearson.
- Berger, C., Blauth, R., Boger, D., Bolster, C., Burchill, G., DuMouchel, W., Pouliot, F., Richter, R., Rubinoff, A., Shen, D., Timko, M., & Walden, D. (1993). A special issue on Kano’s methods for understanding customer defined quality. Center for Quality of Management Journal, 2(4), 3-28.
- Bodily, R., Ikahihifo, T. K., Mackley, B., & Graham, C. R. (2018). The design, development, and implementation of student-facing learning analytics dashboards. Journal of Computing in Higher Education, 30, 572-598. https://doi.org/10.1007/s12528-018-9186-0
- Chen, L.-S., & Hsu, J.-Y. (2019). Discover users’ needs in e-learning by Kano analysis and decision trees. In Proceedings of the 2019 IEEE 6th International Conference on Industrial Engineering and Applications (pp. 572-576). https://doi.org/10.1109/IEA.2019.8714873
- Chen, W.-K., Chang, J.-R., Chen, L.-S., & Hsu, R.-Y. (2022). Using refined kano model and decision trees to discover learners’ needs for teaching videos. Multimedia Tools and Applications, 81(6), 8317-8347. https://doi.org/10.1007/s11042-021-11744-9
- de Kock, E., van Biljon, J., & Pretorius, M. (2009). Usability evaluation methods: Mind the gaps. In Proceedings of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists (pp. 122-131). https://doi.org/10.1145/1632149.1632166
- Earnshaw, Y., Tawfik, A. A., & Schmidt, M. (2018). User experience design. In R. E. West (Ed.), Foundations of learning and instructional design technology. EdTech Books.
- Few, S. (2006). Information dashboard design, the effective visual communication of data. O’Reilly.
- Fong, D. (1996). Using the self-stated importance questionnaire to interpret Kano questionnaire results. Center for Quality Management Journal, 5, 21-24.
- Fontaine, G., Cossette, S., Maheu-Cadotte, M.-A., Mailhot, T., Deschênes, M.-F., Mathieu-Dupuis, G., Côté, J., Gagnon, M.-P., & Dubé, V. (2019). Efficacy of adaptive e-learning for health professionals and students: A systematic review and meta-analysis. BMJ Open, 9(8), e025252. https://doi.org/10.1136/bmjopen-2018-025252
- Hofstede, G. (2024). Country comparison tool. The Culture Factor Group. https://www.hofstede-insights.com/country-comparison-tool
- Hornbæk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79-102. https://doi.org/10.1016/j.ijhcs.2005.06.002
- Iordache, D. D., & Pribeanu, C. (2009). A comparison of quantitative and qualitative data from a formative usability evaluation of an augmented reality learning scenario. Informatica Enconomica [Economic Informatics], 13(3), 67-74.
- Kano, N., Seraku, N., Takahashi, F., & Tsuji, S. (1984). Attractive quality and must-be quality. Journal of the Japanese Society for Quality Control, 41, 39-48.
- Khalid, M. S., Mustafa, A., & Haque, I. (2008). Application of Kano’s model for evaluating information quality of university websites. In H. R. Arabnia & A. Marsh (Eds.), Proceedings of the 2008 International Conference on Semantic Web & Web Services (pp. 277-280). CSREA Press.
- Kim, J., Jo, I. H., & Park, Y. (2016). Effects of learning analytics dashboard: Analyzing the relations among dashboard utilization, satisfaction, and learning achievement. Asia Pacific Education Review, 17, 13-24. https://doi.org/10.1007/s12564-015-9403-8
- Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels. Berrett-Koehler.
- Lesemann, E., Woletz, N., & Koerber, S. (2007). Combining methods to evaluate mobile usability. In Proceedings of the 9th International Conference on Human Computer Interaction with Mobile Devices and Services (pp. 444-447). ACM. https://doi.org/10.1145/1377999.1378051
- Lewis, J. R. (2018). The system usability scale: Past, present, and future. International Journal of Human-Computer Interaction, 34(7), 577-590. https://doi.org/10.1080/10447318.2018.1455307
- Löfgren, M. (2005). Kano’s theory of attractive quality and packaging. Quality Management Journal, 2(3), 7-20. https://doi.org/10.1080/10686967.2005.11919257
- Löfgren, M., & Witell, L. (2008). Two decades of using Kano’s theory of attractive quality: A literature review. Quality Management Journal, 15(1), 59-75. https://doi.org/10.1080/10686967.2008.11918056
- Madzík, P., Budaj, P., Mikuláš, D., & Zimon D. (2009). Application of the Kano model for a better understanding of customer requirements in higher education–A pilot study. Administrative Sciences, 9(1), 11. https://doi.org/10.3390/admsci9010011
- Matzler, K., Hinterhuber, H. H., Bailom, F., & Sauerwein, E. (1996). How to delight your customer. Journal of Product & Brand Management, 5(2), 6-18. https://doi.org/10.1108/10610429610119469
- Nakamura, W. T., de Oliveira, E. H. T., & Conte, T. (2017). Usability and user experience evaluation of learning management systems. In Proceedings of the 19th International Conference on Enterprise Information Systems (pp. 97-108). SciTePress. https://doi.org/10.5220/0006363100970108
- Park, Y., & Jo, I. (2015). Development of the learning analytics dashboard to support students’ learning performance. Journal of Universal Computer Science, 21(1), 110-133.
- Roberts, L. D., Howell, J. A., & Seaman, K. (2017). Give me a customizable dashboard: Personalized learning analytics dashboards in higher education. Technology, Knowledge and Learning, 22, 317-333. https://doi.org/10.1007/s10758-017-9316-1
- Salas, J., Chang, A., Montalvo, L., Nunez, A., Vilcapoma, M., Moquillaza, A., Murillo, B., & Paz, F. (2019). Guidelines to evaluate the usability and user experience of learning support platforms: A systematic review. In P. Ruiz, & V. Agredo-Delgado (Eds.), Human-computer interaction (pp. 238-254). Springer. https://doi.org/10.1007/978-3-030-37386-3_18
- Sauro, J., & Lewis J. R. (2016). Standardized usability questionnaires in quantifying the user experience: Practical statistics for user research. Morgan Kaufmann. https://doi.org/10.1016/B978-0-12-802308-2.00008-4
- Schwendimann, B. A., Rodríguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., Gillet, D., & Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30-41. https://doi.org/10.1109/TLT.2016.2599522
- Sedrakyan, G., Mannens, E., & Verbert, K. (2019). Guiding the choice of learning dashboard visualizations: Linking dashboard design and data visualization concepts. Journal of Computer Languages, 50, 19-38. https://doi.org/10.1016/j.jvlc.2018.11.002
- Tretow-Fish, T. A. B., & Khalid, M. S. (2023). Methods for evaluating Learning Analytics and Learning Analytics Dashboards in Adaptive Learning Platforms: A systematic review. Electronic Journal of E-Learning, 21(5), 430-449. https://doi.org/10.34190/ejel.21.5.3088
- Verbert, K., Ochoa, X., De Croon, R., Dourado, R. A., & De Laet, T. (2020). Learning analytics dashboards: The past, the present and the future. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (pp. 35-40). ACM. https://doi.org/10.1145/3375462.3375504
- Witell, L., Löfgren, M., & Dahlgaard, J. J. (2013). Theory of attractive quality and the Kano methodology–The past, the present, and the future. Total Quality Management and Business Excellence, 24(11-12), 1241-1252. https://doi.org/10.1080/14783363.2013.791117
- Xie, H., Chu, H.-C., Hwang, G.-J., & Wang, C.-C. (2019). Trends and development in technology-enhanced adaptive/personalized learning: A systematic review of journal publications from 2007 to 2017. Computers & Education, 140, 103599. https://doi.org/10.1016/j.compedu.2019.103599
- Yoo, M., & Jin, S. (2020). Development and evaluation of learning analytics dashboards to support online discussion activities. Educational Technology & Society, 23(2), 1-18.