Measuring 'Learning Gain' must recognise context

Being able to both measure and provide evidence about what students gain from studying for a higher education qualification, apart from the qualification itself, has long been a challenge for higher education sectors across the world.

And it is not surprising that it has become an increasingly relevant topic of discussion. Andreas Schleicher, Director for Education and Skills at the OECD, highlighted it in his recent speech at the Higher Education Policy Institute (HEPI). The Government’s Green Paper on higher education refers to it directly, and it also came up for discussion at recent hearings of the parliamentary select committee for Business, Innovation and Skills.

What to measure and why

In an increasingly diversified and evolving higher education sector, policymakers are now called on to provide evidence to inform questions such as: How do students’ knowledge, skills and work-readiness change and improve through their experience of higher education? What added value do universities and colleges bring to their students, and how can this be measured? What is the true impact of learning and teaching initiatives on the quality of students’ experience, their attainment and their employment? And how should this influence strategies in higher education and Government to promote student-focused learning and teaching practices?

Traditional measures for evaluating student performance – grades and degree classification – remain the essential outcome from higher education. But they do not provide all of the evidence needed to address these questions.  And this evidence is more important than ever, given the contribution students and their families increasingly make to the cost of study, and the pressures worldwide on public expenditure.

A granular approach

This is why HEFCE has invested £4 million to pilot and evaluate measures of Leaning Gain in the English context.

Our approach is holistic. It builds on initial evidence from a scoping study by RAND Europe, and works closely with students, academic experts and institutional leaders from across the sector in England.

It incorporates a series of interlinked activities, including:

  • 13 institutional pilot projects, analysing different mixed methodological approaches to measuring Learning Gain, and involving over 70 universities and further education colleges.
  • a pilot of an internationally developed and trialled mixed-methodology approaches to measuring Learning Gain
  • assessment of current data on students held by HEFCE to analyse what it can show.

We are also working closely with the pilot institutions. This will help us to disseminate findings widely across the sector, promote knowledge and understanding, and facilitate dialogue to foster future action through regular networking and capacity-building events.

Each of these initiatives takes a granular and in-depth approach. They involve, and are led by, those most involved in shaping and delivering higher education in England – universities and colleges – and they include and cross-reference a full mix of different approaches and metrics. A key concern for the projects is how best to engage students, and to secure the benefits for them in terms of understanding their own progress and refining their learning.

  • The Careers Group (University of London) – is working with 12 universities and colleges to evaluate how they can use Careers Registration data to assess different employability initiatives, measure work-readiness and predict employment outcomes.
  • The Open University – is working in partnership with two other institutions to measure students’ attitudes and motivation, their behaviour, in terms of what they do and how they do it, and improvements in their cognition. The project is based on analysis of Affective-Behaviour-Cognition (ABC) data.
  • Warwick University – is working with 17 other Russell Group universities and using a range of techniques to explore all aspects of learning: study motivation, cognitive learning approaches, meta-cognitive regulation of learning, and views on good learning and teaching. In each case, the outcomes will be compared with established outcomes data.
  • Birmingham City University – is working with three other universities to run a project that will use the CLA+ standardised assessment tool and the UK Engagement Survey (UKES). The CLA+ tool assesses students’ critical thinking, analytical reasoning, problem solving and written communication skills.  UKES evaluates how students are engaging with their learning.
  • University of East Anglia – is running a project to evaluate three approaches to measuring learning gain across five disciplines at two institutions. A ‘concept inventory’ test will examine how far a student has an applicable understanding of a particular set of concepts. The Grade Point Average (GPA) compares student grades at different points in time, and the academic self-efficacy (ASE) self-assessment tool will evaluate students’ confidence over specific tasks or goals.
  • The Manchester College – is running a project with 15 other further education colleges to develop measures that show the value of HE in FE. It will look at ways to understand how students perceive the impact of their experience in higher education. It will assess how programme leaders perceive their cohort’s journey through the year, and analyse student grade information and attendance data.

The English context

Andreas Schleicher’s speech at the HEPI annual lecture sounded a call for an international system to measure learning outcomes. To this end, he drew attention to the OECD’s work on the Assessment of Higher Education Learning Outcomes (AHELO) project.

Many have challenged AHELO on the grounds of the difficulty of evaluating learning outcomes across different countries, cultures and educational systems, and due to the focus on evaluating outcomes rather than distance travelled.

HEFCE’s Learning Gain programmes seek to evaluate and measure within the context of English higher education, which will help us in this country to understand and contribute to international debates, but also make progress in our own sphere.

What is more, and as the saying goes, the journey is as important as the destination, and we aim to eventually gain a deeper understanding of the progression of students’ knowledge and skills through their study, as well as their eventual learning outcomes.

In a sector as diverse as England’s, analysing how best to measure the Learning Gain of students is not an easy challenge. Just as the range of student inputs differs, so will the range of outputs, and encapsulating student’s development in a definitive way will have many obstacles to overcome.

Still, providing clarity on any measures for Learning Gain can have many potential benefits. We do not know, at this stage, what methods of measuring are the most useful and practical. But we do know that shaping future policy and initiatives must solicit evidence from the full breadth of the English higher education sector and its students.

This is the most suitable approach to understand and analyse the measure of Learning Gain.