Equitable Assessment Framework for today’s Digital Learners

Cagri Yildirim

Learning Enhancement and Technology Projects Adviser for an Institute of Coding Project

What did you do?

Introduce and support instructor’s adoption of the Transparency in Learning and Teaching (TILT) framework to help structure and communicate their assessments for adult learners returning to Higher Education. This framework ensures that learners have a fairer and more inclusive starting point, by embedding meta-cognitive thinking strategies (thinking about their thinking process).

Who is involved?

Cagri Yildirim, the Learning Enhancement and Technology Projects Advisor for the Institute of Coding project collaborated with Mr. Gareth James and Mr. Thomas Webb, practitioners and part-time instructors at Newcastle University’s Business School to ensure their modules aligned with the MSc and Apprenticeship standards for assessment and content, Module Outline Form requirements for teaching and learning activities, and adopted Canvas VLE according to the standards of Quality Matters, TILT framework, and branding of the programmes.

However none of this would have been possible without the timely and valuable input of different stakeholders overseeing the programmes, schools and the day to day experiences of the learners. Thank you to my colleagues:

Institute of Coding

Project Manager, Melanie Dunnett
Apprenticeship Tutors, Dr Tejal Shah and Dr Tong Xin
Principal Project Investigator, Professor Aad von Moorsel
Business Partners Manager, Xander Brouwer

School of Computing

Degree Programme Director (DPD), Senior Lecturer, Dr Ahmed Kharrufa
Learning and Teaching Team, Alex Barfield and Maddie Kinnair

Newcastle University Business School (NUBS)

Degree Programme Director, Senior Lecturer, Dr Joanne James
NBS8416 Instructor, Thomas Webb
NBS8417 Instructor, Gareth James
Learning and Teaching Team, David Ormston-Simpson

2020 MSc Degree Apprentices in the Cyber Security, Data Analytics, and Software Engineering programmes (45 in total)

How did you do it?

(1) Collaborative Perspectives

We had the luxury of designing these modules and their assessments from scratch, but we also had to ensure that the assessment formats would support both the MSc and Apprenticeship requirements.

Conversations with different stakeholders involved in the programme design helped to understand the perspective of the instructor, student needs, schools, and business partners.

(2) Understanding Learner Needs

As a group of staff and academics involved in the apprentice’s programme of studies we were able to confirm that our MSc Apprentices came with an inherit access to technology (hardware and software,) and wifi courtesy of their workplaces. But as adult learners returning to Higher Education some felt nervous about the Business modules that focused on scholarly reading, writing and reflection. They had chosen professions that played to their technical strengths. Therefore our challenge would be to provide clear assessment instructions and expectations, embed academic skills (that our full time students might have developed naturally), and increase their self-efficacy to succeed.

(3) Our Challenge

Our instructors could appreciate adult learners’ struggles, but they were not hired to teach academic reading and writing skills explicitly, or dilute (scaffold) their materials to make them more accessible.

Arguably as MSc Apprentices they have access to the University’s Library Services that provide both individual and group study skills support. But unlike regular MSc students, they are also apprentices who work full time. Their workplaces give them a single day off to take classes which are carefully stacked back to back on that day. Therefore another solution was needed. We couldn’t assume that they could all budget time for this support.

(4) The Solution

As the learning designer, my role is to listen, observe, and make selective recommendations that can help solve multiple stakeholder’s needs. With the instructor and learner in mind, I recommended that we adopt the Transparency in Learning and Teaching (TILT) framework a research informed framework that embeds meta-cognitive thinking skills into the assessment instructions to help learners understand why an instructor has set the assessment, how they can breakdown the assessment into smaller more manageable tasks, and clear criteria for what looks good. The combination of these three parts allows the learner to self–regulate their learning. They know what they are aiming for, and they are given a fair and inclusive (equitable) opportunity to approach the assessment.

The instructors and I collaborated for three weeks, discussing and building out the different sections of the assessment. The toughest and most rewarding aspect being the grading criteria. But we were glad to complete all three parts, because instructors found that apprentices asked less logistical questions and more specifics about the content criteria.

Challenge Yourself

Find a summative assessment you or a colleague has written, and see if you can identify the following.

Assessment instructions must include

1. Explanation of the purpose of the assignment.
2. Description of the task in some detail.
3. Explanation of the criteria for grading or success.

Questions to ask yourself

Given the current instructions, do you think you would be able to produce a response that meets the grading requirements?
What aspects of the instruction could be improved so that the learner isn’t left guessing?
What would you assume learners would need to know or do to approach this assessment (e.g. task analysis, project management, collaboration)? See our example NBS8416 assessment for inspiration.
How do you consider the learner’s time and ability to complete the task? Based on your own experience trying out the assessment, do you double or triple that time?

Challenge your Students

Alternatively you could provide your students with Meta-cognitive skills, by asking them questions about the assessment to help them think about how they would approach the assessment. TILT provides sample questions to ask.

Limitations of TILT

TILT is just a framework. You can adapt as needed or opt out.

Formative Assessments that seek to generate open ended discussion the criteria could be merged with the task details as overarching expectations. E.g. in 250 words express your question, and try to respond to at least two other peers and continue the conversation by answering or asking additional questions.
Pedagogical strategies like Problem Based Learning, can adopt this framework and co-generate the descriptions for purpose, tasks, and criteria in collaboration with learners.
Not all learners are equal learners, those who are advanced in meta-cognitive study skills may find that this type of instructional detail oversimplifies, but arguably it gives them a clear baseline for what is expected so that they can measurably excel.

For those who would like a more robust framework, TILT has produced a free checklist.

Who benefits from TILT?

Instructor Wins!

Assessment documents ensure that if the instructor falls sick a DPD or instructor can help oversee the implementation and grading
Easy to scale and moderate grading across multiple instructors
Easy to adjust or adapt for future iterations

External Examiner Wins!

Transparent expectations indicating fair grading criteria
Expedites reviewing and grading

Learner Wins!

Transparent instructions and criteria ensuring fairness in grading
Helps learners to monitor their own progress towards the criteria
Allows learners to judge for themselves how much more or less time they need to complete activities based on the baseline recommendation from the instructor
Helps learners to express what they do/don’t know and when/where they need help
Builds trust between instructor and learners because the details show that the instructor has thought through the assessment steps from the learner perspective and ensures that the work is achievable in the timeframe
They feel optimistic or less intimidated to try and succeed

Why did you do it?

As the learning designer, my role is to listen, observe, and make selective recommendations that can help solve multiple stakeholder’s needs. Given that our apprentices came with equal access to technology, software, and hardware through their places of work, our focus was to ensure that the instruction and particularly assessment instructions were fair, inclusive (equitable), and empowering for adult learners from diverse backgrounds. The TILT framework allows learners to self-regulate their learning, a skill that might not be inherit or taught previously, but helps decode a post graduate higher education assessment without removing the thinking.

Graduate Framework

This approach develops the following attributes:

Critical Thinkers
Confident
Digitally Capable
Engaged

Find out more about the Graduate Framework.

Interested in finding out more?

Take a look at these resources: