Institution: Trinity College Dublin

Discipline: Teaching and Learning

Author:  Ana Elena Schalk

GenAI tool(s) used: POE Chatbot

 

Situation / Context

The use of chatbots in higher education has significantly increased over the last couple of years. More and more innovative practices have emerged, from facilitating the admission process (Aloqayli, A., Abdelhafez, H., 2023) to supporting the student learning process in many different ways (Gupta & Chen, 2022; Liu et al., 2022; Ait Baha et al., 2023). These approaches frequently shared the view that a chatbot, as a Generative Artificial Intelligence (GenAI) technology, can offer a personalised and instantaneous response to students’ needs. In this regard, Dr Palomares’ research and exploration of the use of a chatbot, a GenAI tool, to support teaching fits in with this innovative trend. Her presentation at the  Annual Teaching and Learning Conference (2024), provided the initial inspiration for this example and the impetus to learn, practice, and embed a chatbot into my teaching practice in the second semester of the 2023-2024 Academic Year.

Specifically, this example explores the use of a chatbot in the Graduate Teaching Assistants online module offered to PhD students from across all Faculties in Trinity College Dublin. The Graduate Teaching Assistants module (commonly known as the GTA module) allows participants to gain insights into key areas of practice relevant to their current and future roles as educators in higher education. The GTA, a 5-credit module delivered fully online over a 10-week period, includes a range of learning activities, such as self-paced study, weekly peer learning meetings, and three synchronous tutorials. Due to high demand, this module is offered twice in each academic year. The overall completion number for both cohorts in the academic year 2023 –? was 83 and 49 students for the first and second cohorts, respectively. Taking into account that the number of students in both cohorts is over 45, they can be considered large groups (Mofrad, 2013).

During the delivery of the GTA module to the first cohort in the 2023-2024 Academic Year, I noticed that there was a high volume of students’ emails whose content was related to seeking confirmation of information that was readily available on the virtual learning environment (VLE). Questions such as, “When is the next tutorial? What are the learning activities for next week? Where can I find the rubric for the formative and summative assessments?” were repeated across many of the students’ emails. On close examination of this issue, I felt that in order to address an unnecessary increase in my weekly workload and to respond to students’ emails in a more timely fashion, the use of a Chatbot function could provide an ideal tool that potentially benefits both students and myself in a practical manner. Also, the use of the chatbot tool may have an additional positive impact and increase students’ engagement with the module. Additionally, it may also increase students’ engagement with the module. Thus, I examined more closely available tools to ease the speed of communication and the delivery of essential information.

Task / Goal

​​The purpose of offering the chatbot option in the GTA module was to explore if it could reduce the time being used to deal with a large number of students’ emails requesting the same information/details and to have more time to interact actively with students through the discussion boards and tutorials.

Moreover, the chatbot could perhaps increase the effectiveness of automatic and timely responses when students need them (Abedi, 2023). As PhD students, some of them studied during the weekend (due to their academic and research commitments during the week), and if they asked any questions, they had to wait until Monday for their response. Timely replies could improve their engagement with the module.

Actions / Implementation

The actions implemented to achieve the goals were:

First of all, after attending Dr Palomares’ presentation, I researched how the use of a chatbot could impact teaching and learning experiences through personalised, repetitive, and automated responses (Liu et al., 2022; Ait Baha et al., 2023).

The chatbot tool used is “Poe” (the same as Dr Palomares’ demonstration), an open GenAI tool that lets you ask questions about the information uploaded to the tool and get instant answers. I researched the publicly available information and selected content that was clear and reliable so that the chatbot tool could provide clear and precise responses during the GTA module. I concluded that this tool offered a good alternative to respond to the need for automatising replies to frequent questions asked in the module,  and was suitable for my initial use of GenAI in my teaching practice.

The next step was the “training” process of the Poe Chatbot, which entailed the following stages:

• uploading only public-domain information (e.g., the calendar available on the website);
• testing some frequently asked questions from the first cohort;
• evaluating the accuracy of the chatbot’s responses obtained as replies. If they did not were clear,  I searched for more public information in relation to that question and uploaded it
• ensuring that the chatbot replies were correct and accurate using the information provided.

In compliance with Trinity College Dublin policies regarding Data Protection Legislation, students were informed from the beginning of the module about how their data would be protected (see https://www.tcd.ie/academicpractice/privacy.php) and reassured that their information would not be used without their explicit consent.

Finally, the chatbot was linked (not embedded) into the institutional VLE, following Trinity’s IT Services policies.   Students were informed and guided on how to use it (if they so wished). Students received clear guidance, which included explicit instructions not to use institutional information such as their email, personal data, or any VLE content.

As the use of the chatbot was optional for students, I put in place another channel of communication: an asynchronous discussion board to discuss administrative and practical aspects of the module. Queries posted in the discussion board were answered by myself and/or students. In the first tutorial, the chatbot was demonstrated, and all questions about using it optionally were clarified.

Outcomes

Providing the second cohort of students in the GDA module with the option to use the chatbot tool had a clear impact on the volume of emails relating to practical everyday aspects of the delivery of the module. The use of the chatbot tool brought about a significant reduction in the overall volume of email. Specifically, I only received 14 emails throughout the 10-week period in clear contrast with more than 100 emails received from students from the first cohort. Furthermore, no emails were received during the weekend period. An example of some of the questions that the chatbot could answer is shown in Fig. 1:

Fig. 1: Example of students’ frequent questions answered by Poe.

 

Regarding some statistics, the free version of this chatbot supplies limited usage data. The available statistics indicated that 150 interactions were posted within the chatbot, but the usage data did not indicate what kinds of questions were asked (Fig. 2), which is a clear limitation of the tools as the data cannot be mined as to what type of questions were more prevalent.

 

Fig. 2: Statistics available in the Poe tool (free version).

 

Considering these conditions, the outcome is a significant reduction in emails throughout the module. In addition, no students were waiting for responses until Monday as there were no emails received throughout the weekends. These two results responded to the goals of this exploratory practice.

This example is an initial attempt, and future explorations of this practice could be considered part of the module. The uses could be expanded not only for the practical aspects of the module but also as a tool for the learning process and to develop AI Literacy in the context of their discipline.

To learn more about this explorative practice, I wanted to categorise those interactions to better understand the impact of this tool in supporting teaching. As mentioned before, it was impossible to get any details about those questions, which is a limitation of this exploratory practice. Nevertheless, the first two outcomes accomplished the goals of this exploratory practice, saving facilitator’s time and decreasing the workload of answering repetitive emails. Students received timely replies, including at weekends. This example can be interpreted as an initial practice to continue exploring in future iterations of this module, or any other, the potential of using chatbots not only to support students’ learning but also to contribute to developing students’ AI literacy in the context of their discipline.

Reflections

This experience showed me that using a chatbot had a positive impact and decreased the overall number of emails I received. Furthermore, the chatbot dealt with students’ questions related to module information appropriately and automatically. Thus, students received the information they sought in a timely fashion instead of waiting for me to respond during working hours.

Despite explicitly asking students to contact me if they were unsure how to use the chatbot tool, I did not receive any queries, which can be interpreted that the instructions provided at the first tutorial were clear and also that the chatbot tool is accessible and user-friendly.  Both elements facilitated students’ use of the chatbot tool without a significant effort.

The discussion board set up for this purpose contained very few student questions. Considering the low number of emails received, it is possible to infer that the chatbot played an important role in supporting students regarding the different information related to the GTA module.

Some further implications for other teaching practices are to consider extending this practice using open-source content of a discipline and contrast these results with other available experiences (Gupta & Chen, 2022; Liu et al., 2022; Ait Baha et al., 2023). This means that students, in the future, could use the chatbot not only for course information but also to support their learning process to better achieve their learning outcomes.

Since this work was completed, more institutional GenAI tools have become available (e.g. Copilot) and more may become available inside the VLE, such as the AI Conversation, which is similar to a chatbot. I will definitely use them instead of a public tool. This change would keep students and staff within the same and protected technological space.

I would like to know more about students’ experiences using this tool and hear their voices and ideas to create more relevant learning opportunities using chatbots in ethical, critical, and responsible teaching practices. Moreover, I would like to explore in-depth what students need to develop their AI Literacy and how we can better respond to them.

Further Reading

Abedi, M., Alshybani, I., Shahadat, M. R. B., & Murillo, M. (2023). Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education. Qeios.

Ait Baha, T., El Hajji, M., Es-Saady, Y., & Fadili, H. (2023). The impact of educational chatbot on student learning experience. Education and Information Technologies, 29 (8), 10153-10176.

Aloqayli, A., & Abdelhafez, H. (2023). Intelligent Chatbot for Admission in Higher Education. International Journal of Information and Education Technology, 13 (9), 1348-1357.

Gupta, S., & Chen, Y. (2022). Supporting inclusive learning using chatbots? A chatbot-led interview study. Journal of Information Systems Education, 33 (1), 98-108.

Liu, L., Subbareddy, R., & Raghavendra, C. G. (2022). AI intelligence Chatbot to improve students learning in the higher education platform. Journal of Interconnection Networks, 22 (Supp02), 2143032.

Mofrad, M. (2013). Using a blended learning model for large group teaching in medical education. Biomedical Pharmacological Journal, 6 (2), 8 – 15.

Digital Resources

www.poe.com

Author Biography

Dr Ana Elena Schalk is an Academic Developer. Digital Learning Lead. Trinity College. She has been a lecturer, researcher, academic developer, and academic manager for over 30 years. She worked at TU Dublin’s Learning, Teaching and Assessment Team, where she led several strategic learning initiatives and taught on the MSc Education and Postgraduate Certificate in University Learning and Teaching. Prior to joining TU Dublin, she led a number of institutional, national, and international digital education initiatives in Latin America, working with the Ministries of Education, UNESCO, and the Iberoamerican States Organisation (OEI).

In Trinity, she leads the Digital Learning and GenAI in Teaching Learning and Assessment plan at the Centre for Academic Practice. This plan includes resources to support academic staff, collaboration with other institutional departments and units, and research development. She teaches as part of the Special Purpose Certificate and other professional development initiatives.

She led the creation and implementation of the open course “GenAI in T&L: How to do it right?” (badge), funded by the National Forum to develop AI Literacy skills for academic staff in Higher Education. This initiative is in conjunction with the University of Limerick and UCD.