Institution: Maynooth University

Discipline: Business

Authors: Shane McLoughlin

GenAI tool(s) used: Microsoft Bing AI

 

Situation / Context

Artificial Intelligence (AI) has been a captivating topic of enquiry for many years within Management Information Systems (MIS), a discipline anchored within European business schools with a focus on developing, applying, and communicating knowledge at the intersection of digital technologies, people, and organisations.

Across industries, AI has been applied some time now, from robotic process automation to decision support systems. However, AI’s role and potential in business and society is becoming increasingly overt and pervasive, due to advancements in a “Generative” branch of AI technology.

GenAI creates “new” content by distilling patterns from training data, guided by user instructions. Advancements have been fast and compelling, with students and employees rushing to exploit its capabilities. For the “Introduction to MIS” module at Maynooth University’s Business School, over 700 students learn how such digital technologies support organisations and enhance their competitiveness, as well as critically assessing opportunities and challenges that these emerging technologies pose.

Task / Goal

This year’s assignment goal was to support the school’s “living lab” approach to bridging education with practice, ensure academic integrity, while meeting key learning outcomes which include critically assessing how technologies impact organisations, along with ethical and social considerations.

To achieve this, 736 students each conducted case studies exploring the impact of new digital technologies on Irish companies. To tackle head on the huge issue of AI use in academic work, students were required to complement their research with a GenAI tool and then critically appraise that tool’s effectiveness. They then worked in groups to discuss their experiences and derive shared learnings, submitted in both written form and orally. To ensure AI complemented, rather than replaced original research, students had to document and make transparent their research process as part fulfilment of their assignment. This involved cataloguing their “search and selection journey,” by highlighting and capturing screenshots of information and sources encountered, submitted as the case study appendices.

Actions / Implementation

To implement the assignment for such a large module, several considerations and steps were taken. The assignment had to align with the university’s policies regarding the privacy of students’ personal data. In the “current” absence of university-tailored and controlled GenAI tools, students were encouraged to use Microsoft Bing AI (Copilot) as it can be used without the student having to log in.

Detailed instructions comprising four parts were designed to communicate the assignment’s objectives and requirements, accompanied by templates to fill in answers to submit for each part. Each student was randomly assigned an Irish company to case study, and 148 groups were assembled for the group component.

Six tutorials were delivered to support the assignment objectives. These included exploring the concept of “Responsible GenAI” in terms of ethical and sustainable use of the technology, alongside such aspects as fairness, inclusivity, transparency, and accountability. Tutorial work also involved facilitating good practice (through exercises and examples) in crafting prompts and appraising responses, as well as facilitating group work and presentations.

In terms of the assignment’s requirement for appraising GenAI’s effectiveness, instructions defined 12 facets for appraising responses: completeness, relevance, comprehension, plausibility, accuracy, assumptions, bias, convenience, creativity, decision-making, learning, and anything else. For the group component of the assignment, students additionally appraised the tool’s overall ease of use and usefulness in terms of its features and functionality.

Outcomes

Students generally reported a positive experience with using GenAI tools as part of their assignment, particularly evident in their appraisals of the tools’ ease of use and overall convenience. However, many emphasised the added efforts needed to verify responses, particularly depending on the tool used and how well prompts were crafted.

What struck many students was the speed and volume of relevant information received even to simple prompts, leading to periods of “information overload,” sometimes resulting in confusion and a sense of being overwhelmed. This could suggests a needed focus on understanding temporal factors in learning and assessment with GenAI, particularly in the context of scaffolded or incremental approaches to learning and its importance.

Overall, there was a good effort made by students to document and make transparent their research process, thereby showing that a focus on documenting the process in continuous assessments can be useful, alongside the focus on outcomes.

Reflections

Overall, the intervention highlighted Zipf’s principle of least effort (1949), where convenience, in many cases trumps effort, with greater attention needed on accounting for and managing the path of least resistance in assessment design. It also suggested a keen focus on the relative importance of temporal factors in learning with GenAI, as well as the benefit of capturing “process” in their use. Here, both the development, tailoring and evidence gathering of education-specific GenAI tools would be welcomed.

Finally, deep questions emerged about the potentially changing nature of learners’ self-efficacy, i.e., beliefs in one’s abilities to successfully perform or accomplish specific tasks or outcomes. Importantly, self-efficacy has been linked to motivation, resilience, and learning strategies that, in turn, produce better outcomes.

The intervention found some students perceived that GenAI augmented and extended their own abilities in producing outcomes of sufficiently high quality, leading to increased self-efficacy. For others, learning to think critically and research rigorously in the context of GenAI’s “seeming” superiority in producing “relevant” content, may lead to decreased self-efficacy where one feels there is little point in even trying to match or compete with a machine’s capabilities. Understanding the nature of how GenAI use factors in self-beliefs and empowerment versus learner disenchantment in one’s abilities and efforts at problem-solving, critical thinking, and creativity; could help inform their future design and use.

Further Reading

Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: W. H. Freeman.

Dwivedi, Y. K., Hughes, D. L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., & Galanos, V. (2021). Artificial intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice, and policy. International Journal of Information Management, 57, Article 101994.

Komarraju, M., & Nadler, D. (2013). Self-efficacy and academic achievement: Why do implicit beliefs, goals, and effort regulation matter? Learning and Individual Differences, 25, 67–72.

Schunk, D. H., & Pajares, F. (2009). Self-efficacy theory. In K. R. Wentzel & A. Wigfield (Eds.), Handbook of motivation at school (pp. 35–53). New York, NY: Routledge.

Sein, M. K., Henfridsson, O., Purao, S., Rossi, M., & Lindgren, R. (2011). Action design research. MIS Quarterly, 35(1), 37–56.

Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. New York, NY: Springer.

Zipf, G. K. (1949). Human behavior and the principle of least effort: An introduction to human ecology. Cambridge, MA: Addison-Wesley Press.

Author Biography

Dr Shane McLoughlin is an Assistant Professor of Management Information Systems at the School of Business at Maynooth University and currently serves as co-director for an undergraduate work placement at the Business School. Shane is a Teaching & Learning fellow for 2024/2025 exploring the responsible use of GenAI for effective learning outcomes. Shane’s research interest is in socio-technical considerations and approaches in the design and diffusion of new digital technologies.