Institution:  University College Dublin

Discipline:   Energy Systems

Authors:  Paul Cuffe, James O’Donnell, Jiajing Li

GenAI tool(s) used: ChatGPT, Midjourney and DALL.E

 

Situation / Context

In March of 2024, we redesigned one essay assignment (worth 10%) for the UCD module Energy Systems in Buildings I (MEEN30130). Most students in this module are in their 3rd-year of studying Structural Engineering with Architecture. This module had traditionally contained an at-home essay assignment, where students would, over the course of several weeks, research and write a technical essay on a given topic within building energy use. Topics might include building energy flexibility, prefabrication, Building Information Modelling, Internet-of-things, and so on.

Like many colleagues across the academy, we were both impressed and worried to witness the rapid escalation in the ability of tools like ChatGPT to conjure credible-seeming scholarly prose ex-nihilo. Rather than leaving the assessment strategy in MEEN3010 exposed to AI plagiarism, we decided to shift the focus towards a more authentic and interactive learning activity.

We tasked 36 undergraduates with the creation of A1-size posters on several predetermined topics in building energy management that they could select from. These posters were to be presented to a panel of experts during an in-person poster session, at which students would have to field technical questions on their particular topic. This presentation and Q&A session together account for 70% of the component grade, further ensuring that students have genuinely acquired the knowledge, rather than unreflectively using GenAI tools.

However, while students were prohibited from using conversational AI chatbots like ChatGPT to create text content or technical diagrams for their posters, they were encouraged to use AI tools like Midjourney and DALL.E to create graphics to enhance their posters. We felt that this retains an open-mindedness about the role and value of GenAI without violating the integrity or authenticity of the assignment.

Task / Goal

In short, we engaged with generative AI for two main reasons:

First and foremost, we aimed to prevent students from completing their work unreflectively or dishonestly by using ChatGPT or its ilk; we sought to uphold academic integrity in the first instance. This assignment used to be a take-home essay in the past, accounting for 10% of the total grade. Our prior research (Cuffe et al., 2023) indicated that such tasks are among the most susceptible to dishonest use of GenAI, with students more likely to resort to chatbot tools, thereby failing to meet their learning objectives. In this case, we adopted assessment styles like presentations and oral questioning that are less vulnerable to AI plagiarism, allowing us to more directly assess students’ achievement of the learning objectives.

Secondly, we intended to nurture students’ understanding and critical thinking regarding GenAI tools. This move was designed to help students identify the advantages and limitations of AI, which will prepare students for a future where GenAI is an integral part of professional and creative processes: encouraging the use of generative text-to-image covered this aspect.

Actions / Implementation

In this assignment, we asked students to design posters to display their research findings, participate in a presentation, and then answer a rigorous series of probing questions posed by academics with subject-area expertise. The assessment rubric for this assignment is divided into four parts: poster design accounts for 10% of the component grade, content, and scholarly references for 20%, and presentation along with response to the questions (i.e. oral examination) for 70%. In designing this assignment, we adopted a moderately open attitude to GenAI. We integrated it by allowing students to use text-to-image tools like Midjourney and DALL.E for certain graphic designs, while strictly prohibiting the use of GenAI chatbots to generate technical diagrams or textual content. Moreover, we informed students about the severe errors and hallucinations that ChatGPT can make in citations, encouraging them to genuinely learn rather than simply relying on GenAI tools.

Out of 36 students, 29 submitted their posters on time. We printed all the posters and asked the students to stick them to the classroom walls during a timetabled presentation session. Subsequently, six academics were responsible for orally examining the students by inspecting their posters in situ and asking relevant questions. The atmosphere during the presentations was surprisingly active and energetic, with the students being remarkably familiar with the content and diagrams on their posters. The face-to-face questioning eliminated the risk of cheating with GenAI and provided the students with additional insights and inspiration. Each student was orally examined for a duration of 10 to 15 minutes. The intensity of the questioning undoubtedly compelled students to deepen their understanding of the relevant topics.

Fig 1. Screenshots of assignment requirement slides

Outcomes

This intervention brought numerous benefits. First, we successfully prevented students from thoughtlessly or dishonestly using GenAI by implementing an oral presentation session, thus ensuring the authenticity of their learning. Secondly, many students found designing posters to be fun and creative, which encouraged them to explore the energy system in Ireland. By visualising their research findings, students were able to reinforce their understanding of related topics.

Additionally, during the presentations, students had the valuable opportunity to learn from their peers. By showcasing their posters to one another, they could understand the research outcomes from others and get more inspiration. These are transferable skills with increasing professional relevance.

The Q&A session provided the assessors with an opportunity to verify the students’ understanding directly, which also enhanced communication between teachers and students. Furthermore, by encouraging the use of GenAI image generation tools, students developed critical thinking about AI after interacting with it in a low-stakes way. They realised the limitations of GenAI and generally preferred to display accurate images with reliable sources.

Reflections

In the final submissions, only one student used DALL.E to generate graphs for her poster (as shown below), surprisingly revealing that students might not be as keen on using GenAI tools as we had anticipated. When asked why they didn’t use AI tools, many students mentioned they didn’t know how to use them properly, as there was no specific guidance in the course on how to design with AI. Meanwhile, most of them were dissatisfied with their attempts and chose to use templates or images found on Google instead. Therefore, if there is another opportunity, we might consider providing samples of AI designs to assist students better, and to show how to prompt image generation tools. On the other hand, many students reported that generating images with AI is costly because this is not a feature of free AI tools.

It is easy to fear that all students are AI wizards, deploying all kinds of tools against any non-invigilated assessment. Our experience here shows that such paranoia is not realistic; if you want students to use Midjourney, those software skills need to be taught and scaffolding, just like any other computing exercise in an engineering module.

Fig 3. An example of a student using text-to-image AI tools to design decorative graphics for their posters.

Further Reading

Cuffe, P., Healy, J., Doheny, E., Nola, K., Hickey, A., & Ó Faoláin, C. (2023, September). Academic dishonesty facilitated by AI chatbots: Gauging the exposure of modules within UCD College of Engineering & Architecture. UCD Internal Report.

Digital Resources

 

https://drive.google.com/drive/folders/1xK9bvr7u_h7GxFN7HPpDZ8q0dFHTDdWA?usp=sharing

Author Biographies

Dr Paul Cuffe is currently an assistant professor at the UCD School of Electrical & Electronic Engineering, from which he received B.E. and Ph.D. degrees in 2009 and 2013, respectively. He delivers 50% of the content of MEEN30130, focussing on the role of electricity in buildings.

His core research interests are in the optimization and analysis of electrical energy systems, with an emerging interest in the potential for blockchain smart contract instruments to disrupt energy marketplaces. He is a member of the UCD Energy Institute.

Dr James O’Donnell is an Associate Professor and module coordinator for MEEN30130. He joined University College Dublin (UCD) in June 2013. Prior to this appointment, he worked as a senior Scientific Engineering Associate in the Building Technology and Urban Systems Department of the Lawrence Berkeley National Laboratory (LBNL). His current work focuses on the development and deployment of interoperable and BIM-based solutions to support multi-scale energy modelling, from individual buildings to national building stocks. It is in this capacity that he contributes to building SMART’s Building Energy Modelling IDM Expert Panel. In addition to technical research and consulting, James teaches modules focusing on building physics and building energy modelling.

Jiajing Li is a Master’s student in Computer Science at University College Dublin (UCD), where she also serves as a Research Assistant in the School of Electrical and Electronics Engineering. With a keen interest in machine learning and its applications in education, Jiajing is exploring how advanced technologies can revolutionise learning and teaching methodologies.