Institution:  Dublin City University

Discipline:  Political Sciences

Authors:  Paola Rivetti, David O’Mullane, Rituparna Banerjee

GenAI tool(s) used: ChatGPT

 

Situation / Context 

The project targeted first-year students of International Relations enrolled in the module, Introduction to International Relations and Security, which comprised 203 students in 2023/2024. The students volunteered to participate in weekly laboratories (eight weeks, six sessions per week). An average of 10 students attended each session, resulting in roughly 70 students participating each week, excluding the final week of the semester when attendance lowered.

A team of three educators tutored students: the module coordinator, Paola Rivetti, and two assistants, David O’Mullane and Rituparna Banerjee. During the laboratory sessions, we conducted practical exercises using ChatGPT in a computer-equipped room.

Students could work individually on a computer or collaborate in groups of two to three. They autonomously chose whether to work alone or in groups and selected their group members independently. We used the free version of ChatGPT.

Pedagogically, the exercises were designed to foster a shared understanding between students and educators of how to use ChatGPT and its potential applications for learning about International Relations, building on the module’s contents discussed in class. Each exercise was followed by the students’ feedback. We conducted an end-of-the-semester survey, receiving responses from 72 students. Ethics approval was granted by DCU Research Ethics Committee. For each session, the team had specific intended learning outcomes, including how to write a good prompt, interact with the machine, and cross-verify information. In the lab, exercises were run with the students under the close supervision of the team members, who could interact with individual students or small groups of two.

From the educators’ perspective, the workload involved preparing laboratories, which included identifying in-class exercises relevant to the intended learning outcomes. We also supervised students during the laboratories, led the sessions and interacted with them during the exercises. Additionally, we assessed the students’ performance and the feedback they offered after each laboratory.

Task / Goal 

While significant attention has been devoted to online/blended teaching and related tools, open-access GenAI chatbots and writing programmes have received less attention as instruments that impact our teaching and assessment methods.

The pedagogies of Political Science and International Relations somehow lag behind in understanding and addressing these tools. For those teaching these subjects, it is crucial to come to terms with GenAI and its impact on how we should assess students.

This project aimed to revise the alignment of our understanding of the learning objectives, our teaching, and the most suitable assessment methods in an environment where GenAI is present and utilised regularly. We considered whether to ban it or instead teach students how to use it responsibly and ethically, as well as develop a shared understanding of its capabilities and limitations (if anything at all).

Actions / Implementation 

Each session consisted of three parts: practical exercises with ChatGPT, class discussion about the results, and student feedback.

In Exercise 1, as a starter, we asked students to identify the objectives of the laboratory and what they hoped to gain from it. Responses included learning how to use ChatGPT for IR-related content, which allowed us to both discuss how to discern good scholarship and reflect on the activities that we should prioritize to achieve our goals. After the brainstorming, as Exercise 2, we discussed how to write a prompt, building on the objectives we identified. The authors had prepared a list of characteristics that good prompts should have, and had the students employ them to write their prompt.

During the feedback session, students said that ChatGPT offered high-quality information. To enhance their critical approach, the educators decided to conduct additional exercises focusing on the quality of the results.

In Exercise 3, the students had to draft their own prompt autonomously. As ChatGPT started to generate text, the goal was to interact with and push the machine to generate better information by challenging it with requests for more details, fact-checking the information given, and requesting more examples. The students’ observations became increasingly critical: they noticed that the machine could confabulate, deviate from the command given, and fabricate facts and references.

As the semester progressed, the team was able to assign increasingly elaborate exercises to students.

In Exercise 4, the students worked with ChatGPT to identify a good research question for a series of given topics. This exercise offered the opportunity to discuss what constitutes a good research question and reflect on the applicability and practicality of using ChatGPT by examining the quality of the sources and case studies it supplied to answer them.

In Exercise 5, students had to work with ChatGPT to structure an essay. This presented the opportunity to discuss essay writing and an essay’s structure with the students. The students identified that ChatGPT tended towards confabulation, fake facts, inaccurate references and reading suggestions, but it provided good general information on how to structure an essay and construct an argument. Both exercises contributed to enhancing the students’ critical approach to ChatGPT.

Outcomes

At the end of each laboratory, students were asked to provide feedback on the utility of the exercises and what they learnt about ChatGPT during the session. At the end of the semester, students were also asked to participate in a short survey reflecting on the strengths and weaknesses of ChatGPT for studying International Relations, and the likelihood of them using it in future as a study aid. Seventy-two students completed the survey.

Generally, we have found that GenAI presents significant limitations, and students should learn how to use it effectively from the beginning of their university careers. Assessment methods should be calibrated not to prevent students from using ChatGPT (although this is a possibility), but to allow them to use ChatGPT only for what it is good for and to teach them how to use it appropriately.

As such, involving students in these pedagogical approaches helps them foster a critical outlook towards AI tools rather than rejecting or embracing them unquestioningly. In the end-of-semester survey, respondents reported scepticism about the tools, which evolved into discovering their potential uses in structuring work and gaining broad insights, primarily in providing non-reflective assistance in research which did not need discernment and critical decision-making.

Overall, while a shift occurred towards acknowledging its utility when employed effectively for academic purposes, many participants reported inaccuracy of information, unreliability, and limited usefulness for targeted and specific tasks; the more specific the task was, the less the GenAI chatbots’ reliability.

Reflections

Beyond the core benefits of student learning, the laboratory offered several secondary advantages. It partially addressed challenges associated with a shortened semester and a large class size, and effectively bridged a technology gap. Finally, the team learned that they, and educators in general, are indispensable because ChatGPT’s suggestions are based on probabilistic codes and words, and the machine lacks the nuance, contextual knowledge, and pedagogical consideration that a human educator brings.

Concerns also remain about the risk of language uniformity and standardisation. However, as many survey answers suggest, students will likely continue using ChatGPT, even when aware of its limitations.

We need to adapt our education system to incorporate these technologies. Furthermore, this should encourage institutions to invest money and resources into educating educators and students about GenAI chatbots – something that, unfortunately, is not happening at an appropriate level, raising concerns about the quality of education being provided.

Further Reading 

Rivetti, P., Banerjee, R., & O’Mullane, D. (forthcoming). Testing ChatGPT in IR classrooms: Potentialities, limitations, and what’s next. PS: Political Science & Politics.

Author Biographies: 

Dr Paola Rivetti is an Associate Professor in the School of Law and Government at Dublin City University.

David O’Mullane is a PhD candidate in the School of Applied Language and Intercultural Studies (SALIS) at Dublin City University.

Rituparna Banerjee is a PhD candidate in the School of Communications at Dublin City University.