Institution:  University College Dublin

Discipline:   Robotics

Authors:      Paul Cuffe, Anshu Mukherjee, Jiajing Li

GenAI tool used: ChatGPT

 

Situation / Context

During the Spring trimester of 2024, in the Robotics Design Project (EEEN10020) module with 54 first-year undergraduate engineering students, we deliberately revised the assessment strategy.

We evolved a take-home assignment into a pair of supervised in-class exercises. Students were required to complete these new TinkerCAD designs and submit code and reports directly following a supervised three-hour lab session.

The module has always been very hands-on; students attend every week for a three-hour practical session, in which they build a robot out of Lego parts and programme it, using an Arduino microcontroller (see Fig 1).

Fig 1. Examples of the types of small autonomous Lego robots that student teams build and programmed in EEEN10020 over the case of the trimester. These robots then compete in a final competition.

At the end of the trimester, teams then battle for prizes in a final competition. Each year, students would request some ability to prototype code at home; this was hard to arrange, as microcontroller programming requires access to the physical Arduino units, which need to stay in the lab space. However, in Spring 2023 we rolled out a new assignment (and associated instruction) built around the “TinkerCAD” environment. This is a browser-based prototyping tool in which students can build simple electronic circuits and execute code on a simulated Arduino microcontroller.

While the roll-out of the “TinkerCAD” material in Spring 2023 was welcomed by the students, we noted two shortcomings with the at-home 25% coding exercise that assessed their skills using it:

• The majority of the students submitted very close to the deadline (as in Fig 2), even though they had several weeks to complete the assigned work. This last-minute surge implied that students endured several stressful and guilty weeks of avoiding the assignment before doing it all in a panicky sprint at the eleventh hour. We felt this mode of engagement was not conducive to quality learning or student wellbeing.
• With the emergence of tools like ChatGPT, we felt that the possibility of students dishonestly completing the assignment with excessive help from AI chatbots was problematic. The “TinkerCAD” environment requires some interactivity, with students wiring up virtual components, etc, and so is perhaps not as fully exposed to ChatGPT-style cheating as purely textual assignments; however, we still perceive some level of risk here.

Against this backdrop, we decided to replace the single at-home assignment worth 25% with a pair of in-lab “TinkerCAD” problem sets, each worth 10%. These were rolled out in Spring 2024. These problem sets were to be completed within a designated 3-hour lab session. Teaching assistants were on hand both to gently help and encourage, as well as to invigilate against the use of chatbot tools or other forms of plagiarism.

Task / Goal

The main purpose of restructuring the assignment into a lab setting was not to apply GenAI, but rather to encourage and guide the students to achieve the module learning outcomes. Therefore, this evolution of the module assessment actually emerged, in part, as a response to the emergence of GenAI and the opportunities it offers for students to dishonestly complete non-invigilated assessments.

In our previous research on the exposure of modules within UCD to AI plagiarism (Cuffe et al., 2023), it was opined that the integrity of in-person assignments would be least threatened by GenAI, whereas take-home assignments face significant risks. In this case, by deploying supervision in the lab setting, we want to forestall any risk of students resorting to GenAI for cheating.

Additionally, instead of having students struggle alone at home, we hope to build a conducive learning environment with a moderate amount of accountability and peer motivation. This encourages them to put in real effort, achieve the learning outcomes, and go home with a sense of a good day’s work done.

We aim to prevent procrastination and liberate students from the perils of long deadlines and open timelines, thereby alleviating their after-class stress. By doing so, we can also prevent the situation where all students rush to submit at the last minute (see the graph on cumulative submissions vs days before the deadline below).

Fig 2. The relationship between cumulative submissions and the available time before the deadline for the Spring 2023 at-home exercise. Such a graph would be typical for engineering assignments. It makes clear that students do not have super-human levels of conscientiousness and executive function.

We strive to reach a balance between the unrestricted take-home assignments and the rigidity of formal exams to effectively motivate, encourage and authentically assess student learning while upholding academic integrity in the era of Gen AI.

Actions / Implementation

To provide more granular detail, the first assignment challenges students to create an Arduino system on “TinkerCAD”, where they must design a circuit that allows a potentiometer to rotate from its minimum to maximum position, controlling a servo motor to likewise rotate from 0 to 180 degrees. The angle of the servo motor’s rotation should also be displayed on an LCD screen.

Fig 3. Example virtual prototyping canvas for the TinkerCAD assignment1

The second assignment involves constructing a speed control system for a DC motor, utilizing both a potentiometer and an encoder, with an H-bridge for directional control.

Fig 4. Example virtual prototyping canvas for the TinkerCAD assignment 2

Students were required to complete these tasks during two three-hour in-class “TinkerCAD” sessions, after which they submitted their code and a short report, including screenshots of their designs from “TinkerCAD”. We moved these two assignments into the lab this year to prevent students from lazily completing assignments with AI assistance. This change also provides a practical, cost-effective way for students to engage hands-on with the circuit components and codes, ensuring they successfully achieve the learning objectives.

Outcomes

Regarding the outcomes, 82.4% of students achieved a grade of D or higher in the first assignment, with this figure rising to 92.3% for the second assignment. This improvement reflects the students’ enhanced proficiency in constructing circuits in “TinkerCAD” and executing specific functions through C++ coding.

As expected, another advantage of this approach is its natural deterrent against the misuse of GenAI. From the grade distribution chart below, we can see that students’ grades for the two assignments in 2024 trend towards a normal distribution, as opposed to 2023 where most grades were concentrated at an A. This suggests a higher level of credibility and reflects the new assignments’ robustness against AI.

In fact, there is little need for GenAI in the lab, as peers and the attending TAs can offer quicker and more effective & restrained encouragement. By manually connecting virtual circuit components, students are less likely to become dependent on AI. Instead, they will better achieve the learning objectives.

In line with our goals, this approach also lightened the students’ after-class workload. The students expressed their satisfaction with having more time to review the material or delve into related topics, appreciating the extra freedom this provided for their academic exploration.

Conscientiousness and executive function are in short supply; we all struggle to defer gratification. Many a gym membership goes unused. Good curriculum design works with rather than against normal human psychology. We are social creatures; it is energising and uplifting to work individually on challenging a task alongside peers in a designated block of time within a controlled lab setting.

Fig 5. Grade distribution for the 2023 take-home assignment, which accounts for 25% of the total score.

Fig 6. Grade distribution for 2024 in-class “TinkerCAD” assignments, which account for 10% of the total score each.

Reflections

Even before the emergence of Gen AI tools, there was ample reason to structure engineering coursework assignments in a more timetabled and actively delivered fashion. Lightly supported homework assignments with long timelines loom over a student’s trimester, more often a source of distraction and guilt than an opportunity for deep work and quality learning. University teachers play a powerful social role and have the power to set timetables and impose expectations. Let’s use that for good. It has always been a good idea to proactively and deliberately create assessments that get the best out of students. The emergence of powerful Gen AI tools is just another motivator: now at-home assessments can easily be plagiarized in an undetectable way, so it is very timely to reconsider how we support students in their continual assessment.

Admittedly, the example assignments for EEEN10020 discussed here have their flaws. For one, the assignments may be a bit overly challenging for first-level students. If given another chance, we might add more scaffolding to the assignments to reduce their struggle during lab sessions. Overall, though, we feel that the assessment structure represents an improvement on the at-home approach, even if some of the details could still be refined.

Further Reading

Cuffe, P., Healy, J., Doheny, E., Nola, K., Hickey, A., & Ó Faoláin, C. (September 2023). Academic Dishonesty Facilitated by AI Chatbots: Gauging The Exposure of Modules Within UCD College of Engineering & Architecture. UCD Internal Report.

An alternative version of the above report is forthcoming as a conference publication:

Cuffe, P., Healy, J., Doheny, E., Nola, K., Hickey, A., & Ó Faoláin, C. (2024). A threat assessment framework for screening the integrity of university assessments in the era of large language models. Presented at the IEEE Research and Technologies for Society and Industry Forum, Lecco, Italy.

Digital Resources

https://drive.google.com/drive/folders/1Zr1eQ5ud5Exkte_mQFBKz1W0VgVWn9a8?usp=sharing

Author Biographies

Dr Paul Cuffe is currently an assistant professor at the UCD School of Electrical & Electronic Engineering, from which he received B.E. and Ph.D. degrees in 2009 and 2013, respectively. His core research interests are in the optimization and analysis of electrical energy systems, with an emerging interest in the potential for blockchain smart contract instruments to disrupt energy marketplaces. He is a member of the UCD Energy Institute.

Dr A. Mukherjee is a Lecturer and Assistant Professor at University College Dublin (UCD) in Electrical and Electronics Engineering. He embarked on his academic journey at SRM Institute of Science and Technology, India, where he completed his B.Tech in 2018 and was honoured with a Merit Scholarship. Venturing further into academia, Dr Mukherjee pursued research at the esteemed Indian Institute of Technology (IIT) Patna, diving deep into waveform design during a six-month internship. Continuing his pursuit of knowledge, he earned his Ph.D. from UCD in 2023, with his seminal thesis titled “Optimization Methods for Secure and Energy Efficient MIMO System.” His research domains encompass wireless security, physical layer security, energy efficiency, and the application of optimization and machine learning in wireless communication, with a keen interest in quantum communication.

Jiajing Li is a Master’s student in Computer Science at University College Dublin (UCD), where she also serves as a Research Assistant in the School of Electrical and Electronics Engineering. With a keen interest in machine learning and its applications in education, Jiajing is exploring how advanced technologies can revolutionise learning and teaching methodologies.