A personal reflection from Dr Elise Pegg, Senior Lecturer in the School of Engineering, on her experience of introducing digital exams into a module in the MSc in Biomedical Engineering using Inspera, the University’s digital exam platform.
Introduction
For the bulk of my career I have used traditional written exams to assess my students. I teach mechanical engineering and we often ask the students to show their mathematical working to evidence their understanding, which lends itself to handwritten answers. During the pandemic I was required to run digital exams, but I was not satisfied the format worked for the kind of questions I wanted to ask and so returned to handwritten exams as soon as I was able. Last year I moved institutions and started here at Newcastle University, and with the move I took on new teaching with different requirements. For the first time I felt like a digital exam could be useful.
Why change to Digital Exams?
I was to teach a module which is part of the MSc in Biomedical Engineering, and although it does include some mathematical working, much of the content is focused on design, failure analysis and clinical factors. The cohort size is small and the programme is taught with block-teaching, so each semester has 3 modules and they are taught sequentially. Previously the module had been assessed with 100% coursework, but I was keen to move to an exam assessment for both practical and pedagogical reasons.
Inspera’s logo
The block teaching made it challenging to fit a coursework assignment into such a short period of time; I wanted students to have time to properly digest the taught content before assessing them. I was also concerned by the increasing use of artificial intelligence software and that I would not be able to ascertain the level of learning from submitted coursework.
The main reason I decided to run it as a digital rather than a handwritten exam was to improve the student experience. The other modules in the programme with exam assessments used Inspera and I wanted the students to be comfortable in the exam rather than preoccupied with worries around an unfamiliar assessment format. Also, Inspera is much more accessible; font sizes can be adjusted, contrast can be increased for readability, it is compatible with screen readers, has text-to-speech functions, and it has a dictionary and a spell checker. All these features are incredibly useful for ensuring that we test the students on solely the module learning outcomes without introducing bias.
I still had concerns. I was worried the students might get confused or distracted by the technology, that I would be limited by the different digital question types and not really be able to test the students how I wanted. I was also anxious I would click a wrong button and accidentally release the paper to the students before the exam date – I was thankful to discover that as an academic you are not given that level of access!
Getting started
The first thing I did was to explore all the different question types available. I was glad to see there are plenty of resources on the Newcastle University Learning and Teaching website pages and also on the Inspera website that explain exactly how each one works. I spent quite a lot of time thinking about the exam design and how I would assess the learning outcomes, while also keeping it practical from a marking standpoint.
This assessment was for an MSc cohort which is FHEQ level 7 and so I needed to be sure the exam could test their problem-solving skills and understanding. I found myself tending towards having a two-part exam, where a section of it would be automatically marked (partly to reduce marking workload, but also for consistency with other exams in the programme), and the second part would be manually marked so I could test the student understanding and skills in greater depth.
I was inspired by how medical schools use multiple response questions and spent quite a long time reading around this, I was keen to use similar techniques to test a deep level of understanding and not just superficial facts. There is a wealth of literature out there, but I found the review article by Coughlin et al. a particularly practical guide (Coughlin, P. A., and C. R. Featherstone. “How to write a high quality multiple choice question (MCQ): a guide for clinicians.” European Journal of Vascular and Endovascular Surgery 54.5 (2017): 654-658). I was also interested in the use of negative marking to ensure the students did not pass the exam through chance rather than ability.
I decided that I would use multiple response questions with automatic marking for 30% of the exam, with the remaining 70% made up of two essay questions with multiple parts. By weighting the exam more heavily by the manually marked essay questions, I could mitigate the risk that I would not assess deep understanding with the multiple response questions. Also, I felt more confident writing the essay-based questions because they are the style I had used previously for the hand-written exams.
The writing
I chose to write the paper offline, so I could focus on writing good exam questions without the distraction of a new software interface. I found a quiet day when I was feeling creative and put the questions together. At this point I asked my colleague Dr Oana Bretcanu, who is the programme director and experienced with Inspera, to look through my draft. I found this incredibly helpful. Oana highlighted issues with how the questions would look in Inspera and suggested adjustments I could make so they would be more understandable. Oana also strongly advised me to not use negative marking as it can negatively impact student experience due to increased anxiety. Instead, she suggested a partial credit with penalty approach, which although it includes a penalty if a student makes a wrong selection, the final mark for each question cannot go below zero. Thus, the stakes are lower if the student makes a mistake, but there are still consequences and they will not benefit from guessing. So I took this approach for the multiple response questions.
Overall, I found the writing process much the same as for a standard handwritten exam. All of the same considerations apply; clarity of writing is very important, marks should be allocated proportionately to effort, questions need to assess the learning outcomes and line up correctly with the taught content, marking schemes should be detailed and specific, and the paper should be completable in the timeframe given. The only real difference is that the question format is slightly more constrained because it needs to fit into one of the Inspera question types.
Checking and moderation
In my experience the checking process was slightly more challenging in Inspera compared to a traditional handwritten exam, mainly because of the Inspera user interface. Because the look of it is unfamiliar to me, I found it harder to notice the errors. When I sent it out to the internal moderator, she found errors in two of the multiple response answers where I had selected the negative rather than the positive answers and there were also some grammatical mistakes. She recommended making full use of the preview functions within Inspera and to test out each multiple choice option and check the marks, so I’ll do that in future. I was lucky to have an internal moderator who was very detail-oriented and did a great job.
The exam had to be printed to a PDF for the external examiner to review which was straight forward, but I learned the default print does not mark the correct answers so this needed to be adjusted in the settings. It was not possible to add the mark scheme for the essay-based questions though, so these had to be sent to the external manually. Overall, the checking experience was quite good, but I have learned that I spot errors much more easily on a physical copy so I will be printing off my draft papers in future.
I view these as teething issues; now that I have been through the process once I feel much more confident in the process. I also know who I can go to if I need help in the future. I know which colleagues have the expertise to advise on digital exams, and for those cases where they cannot help, the Digital Exams Team are very quick to respond to queries. So there is plenty of support available to navigate through any issues which may arise.
Class preparation
This was the first time this module had been assessed by an exam, so I was aware that I would need to provide the students with some extra support so they would know what to expect. I decided to spend a whole lecture on exam preparation, and I also set up a section of the Canvas page with useful links to demo resources and revision advice. In the lecture I signposted again to the Inspera demos – I tried them out myself and they are a really good way to get a feel for what it is like to sit an exam in Inspera. I then ran a mock exam with the whole class as a group. I was aware that the students in the class may feel uncomfortable giving their answers publicly, so I used the Vevox software.
Vevox and Canvas Logos
With Vevox I could put up a question on the screen, and then have the class anonymously vote for their choices, and then as a class we had a discussion about what the correct answer was and why. It worked really well and the students found it helpful, so much so that after the lecture they asked for additional practice questions in the same format they could use for revision. As this was the first time running this exam there were no past papers but luckily I had already created some self-directed Quizzes in Canvas with multiple response questions, so I signposted the students to those. I also wrote an additional essay question they could use for revision. On reflection I should have anticipated the student need for revision questions and prepared more in advance.
Marking process
In theory the automatic marking should make the marking faster, but as this was my first time I was using Inspera I was concerned that I may have made a mistake in the initial set up which would lead to marking errors. So I did a lot more checks than I probably needed to during the marking stage. Happily, the careful moderation we did during the writing stage worked and there were no issues with the automatically marked questions. The moderator and I marked the essays according to the mark scheme, and where our marks varied we needed to agree a mark. We did this offline and then input the final mark into Inspera. This seemed to work well, but it may be possible to do this within Inspera so that is something I will investigate in future.
Reflection
The experience of creating an exam in Inspera I found surprisingly fun! Learning about the software and exploring the possibilities has opened my mind up to different ways I can assess my students. The exam seemed to go well and the students did not report any issues with the experience. My key takeaways are as follows:
- To create a good Inspera exam takes time. Plan in plenty of time to think through exactly how to make the exam work for your module and allow time to play with the software.
- Colleagues are a valuable resource. I learned so much from speaking to more experienced colleagues; they can give you practical advice which is specific to your discipline and save you a lot of time in the long run.
- The Inspera help pages and guides are good! There is a wealth of resources available on the Newcastle University Learning and Teaching website and the Inspera website. There are also courses available for training and the digital exams team are happy to help.
- Inspera shouldn’t dictate the format of your exam. First decide on how you want your exam to be and then make Inspera work for you as you need. It may be that Inspera cannot do what you need and is not the right format for your exam.
- It’s critical to always keep the student experience in mind. We all have our biases and interpret things differently, the more viewpoints you can gather the better you can serve your students. The preview functions within Inspera are also really useful.
Going forwards I plan to continue using Inspera for my module. In addition to the Vevox class practice session, next year I plan to create a mock-exam for the students that they can complete within Inspera during revision to ensure they are completely prepared for the exam. I also plan to continue to develop my multiple-choice question writing skills and look forward to playing with some of the other question types.
Authors
 |
Dr Elise Pegg Senior Lecturer in Mech Engineering School of Engineering |