Smart tools for credible, valid, and reliable assessment in Higher Education

Smart tools for valid and reliable assessments in Higher Education

It is the ultimate goal of assessment development that every paper and result be as credible, reliable, and valid as possible.

This is more important than ever in light of the May 2022 revisions to the quality and standards conditions of registration from the Office for Students.

Condition B4: Assessment and awards:

“The provider must ensure that:

  • students are assessed effectively;
  • each assessment is valid and reliable;
  • academic regulations are designed to ensure that relevant awards are credible;”

The Framework offers the following definitions and guidance to clarify these important terms:

Credible

According to the OfS Framework, “credible” means that:

“…awards reflect students’ knowledge and skills… for this purpose the OfS may take into account factors which include:

  • the number of relevant awards granted, and the classifications attached to them, and the way in which this number and/or the classifications change over time and compare with other providers;
  • the provider’s explanation and evidence in support of the reasons for any changes in the classifications over time or differences with other providers.”
They want to know if results are justified and consistent

The OfS is increasingly concerned about grade inflation, and the preservation of the value of degrees. Assessments will need more and more evidence as to the consistency of challenge and academic consideration which went into their development.

Valid

According to the OfS Framework, “valid” means that:

“…an assessment in fact takes place in a way that results in students demonstrating knowledge and skills in the way intended by design of the assessment.”

Assessments should be designed so that they measure what they are intended to

For a process which produces valid tests, the skill being tested needs to be clearly signalled to everyone involved in reviewing the test. Departments need the flexibility to utilise the most appropriate methods and the opportunity to examine questions for language or images which may complicate what is really being tested.

Reliable

According to the OfS Framework, “reliable” means that:

“…an assessment, in practice, requires students to demonstrate knowledge and skills in a manner which is consistent as between the students registered on a higher education course and over time”

Results must be able to be replicated

The value of a qualification relies on standards being upheld year on year. The knowledge, understanding, and abilities of graduates with the same outcome need to be kept at a comparable level. The assessments they sit and the marks they receive therefore need to be highly accurate.

To ensure that each test meets these high standards, the GradeMaker Pro authoring platform contains a range of easy to use assessment development tools. This system will help you to run a rigorous quality assurance process throughout your university, so you can:

Test the syllabus and meet your academic regulations

With GradeMaker Pro you can ensure all questions are written with a sharp focus on what they are testing to ensure their validity. Questions can be developed directly alongside relevant documents such as marking criteria, the course syllabus, and national frameworks. As a result, you can be confident that the design of the assessment is clearly linked to the skills and knowledge being tested.

Throughout the authoring process you can check test coverage data and take action to balance the distribution of marks. 

You can also ensure there is proper coverage of the course content by tagging items to different parts of your specification and tracking development of items across the syllabus.

Writing valid and reliable exams with a close eye on the mark scheme

Enforce rigorous quality assurance through structured workflows

Keeping a thorough exam authoring process on track is easier in GradeMaker Pro. Whether you are coordinating exam development with internal staff, external reviewers, across a department, or across a whole institution, our tools support meaningful and productive collaboration. 

It allows you to:

  • Create customised workflows and bespoke templates for exams so your tests are written to a specified format.
  • Assign tasks and deadlines to users for each workflow step, so that work automatically moves from one stage to the next.
  • Notify users when tasks have been assigned to them and when they are approaching due date.
  • Ensure no corrections or comments are missed by managing all feedback on one online version.
  • Target review comments to specific contributors and flag issues that must be actioned at the next step.
  • Assign customised QA forms to reviewers as an optional or mandatory task.
Test development versioning: Ensure total comment and correction implementation

Demonstrate the quality of your processes with data rich development

Robust paper development practices can help to ensure that grade improvements are supported by evidence of improved performance. It can, however, be difficult to add greater scrutiny and consideration to exam question creation without increasing staff workload.

GradeMaker Pro makes it easier to bake good practice into test development and automatically captures key auditable information throughout this process, allowing you to:

  • Analyse the credibility and reliability of whole tests by comparing the performance of anchor items over time.
  • Upload key results data such as facility and discrimination, alongside your own custom metrics, to evaluate the validity of questions.
  • Manage an item bank of approved questions for use if questions need to be swapped last minute.
Justify grade inflation with anchor items to ensure questions are credible and reliable

Protecting your papers from leaks is also key to maintaining the integrity of assessments. Find out about the security risks we help you to address in this article by our Chief Technology Officer: The 3 common security flaws putting exam content at risk – and how to fix them.

Introduce greater efficiency and flexibility to your processes

There are a huge range of benefits to adopting a system focussed entirely upon perfecting the authoring process. GradeMaker Pro contains some unique features which make this a workable, long-term option for even the most complex of institutions.

We do this by supporting you in building a modular digital infrastructure, where you can swap in and out the very best systems for each part of your process as your needs change over time. To this end, GradeMaker Pro holds your assessments in a fully flexible format, ready for dual output.

Add efficiency to exams publishing with instant typesetting for printed papers

From one central system you can instantly generate print ready papers in your style, without any need for the extra time and cost of typesetting. The same content can also be delivered online with a digital QTI export, a popular international standard for communicating test content.

If you want to see all these features in action, get in touch to arrange a time for a personalised online demonstration.