Developing valid and reliable tests is complicated, especially in the digital age. The complexity of test production means that you need an e-authoring system which ticks ALL the boxes… but what are those boxes?
If you are part of an awarding body and want to make sure you cover all your bases when identifying and prioritising high-stakes test authoring software requirements, this guide is for you.
Five considerations for test developers when choosing your e-authoring system
1. Can you manage complex workflows efficiently?
Developing high stakes tests is probably the hardest of all publishing tasks, with multiple steps involving large teams collaborating on highly confidential materials. Technology has the potential to improve the efficiency of this process, but only if it can address the following core questions:
- Can you create controlled, customised workflows, so content can travel through the process smoothly?
- Can you manage users easily? You will want to set up different permissions based on users’ role and task, whilst also being responsive to staff changes mid-project.
- Is it easy to track what all users are doing and how your content is progressing? Project managers will want a single flightdeck where they can see the progress and status of tasks by item, paper, workflow, subject or user.
- Does the system fully support the typesetting process? An authoring solution that can output print-ready papers following your organisation’s precise styling rules will significantly reduce overall development time and cut costs.
2. Can you control quality at every stage of the development process?
Quality is at the heart of fair assessment, so the software you choose should allow for quality checking throughout the authoring process without creating bottlenecks. Fundamental to this is being able to send materials through approved workflows, and to deliver customised quality checklists at each stage so users can validate they have carried out the right checks.
Additionally, authors should be able to tag items to highly defined syllabus information and other metadata. This allows for paper coverage to be reviewed in quality control meetings, ensuring the final paper assesses a broad and balanced sample of the specification.
Reviewers need tools that allow them not only to comment on draft questions and papers, but also direct comments to specific users and request comments to be actioned before the workflow can proceed. This will ensure nothing slips through the net that would render an item inaccurate or invalid.
3. Is it flexible enough for your question writers?
A critical requirement for your e-authoring system is that it offers flexible tools for writing appropriate questions in every subject. A rich selection of item types is key, as is the ability to create papers requiring multi-part questions, stimulus sections, optionality, images, tables or complex maths formulae.
As you start to scale up use of your chosen e-authoring system, it’s likely your requirements will become more and more nuanced. Reviewers may need to evaluate complex Levels of Response mark schemes side-by-side with the questions. Authors may need to write audio transcripts alongside questions for Listening papers. Asset managers may need to collate audio files so they download in the same sequence they appear in the papers. To do all this, your e-authoring system needs to offer rich functionality that genuinely covers your full range of needs.
The software should also support different authoring methodologies to future-proof your operation. For instance, you may currently adopt a whole test authoring approach, but have plans to move towards item banking in future. Or maybe you currently adopt item banking and are looking to do more pre-testing and anchoring of items in future. You may be transitioning from print to online delivery, in which case the solution should allow for dual outputs, potentially with the same test in both formats.
4. Will it allow you to adapt and innovate over time?
Just because something works well today, doesn’t mean it will work for you long-term.
It can be tempting to think a single, integrated solution from one supplier is the way forward. Choosing a single technology source may simplify the procurement process but bear in mind it can create serious pitfalls. If you are producing a variety of high-stakes exams at scale, it’s likely there will be areas where one supplier’s capabilities don’t align precisely with the requirements of your operation. Where compromises need to be made, pain points may develop over time that may be hard to remedy without frustrating workarounds or serious upheavals to your system.
If you decide you want to swap out a component of the solution – for example, to use new marking software or a different delivery system, then it is important to know that this is possible.
To get a system which lets you innovate, ask if it meets these criteria:
- Is it modular, so one component can easily be swapped with another if you need to change?
- Do the modules work nicely together, for example letting you preview items for onscreen tests in the candidate player during authoring, or immediately seeing the final print layout during the review process?
- Is the content created in the system held in genuinely open, standards-based formats such as QTI 2.2 or other widely used print layout formats? This is really critical, so your content doesn’t become dependent on supplier standards making switching modules impossible.
Finally, consider how you might want to use your content in the future. If you choose to re-use high stakes items in practice tests, does the solution let you do so? Is this possible even if the first usage was for a paper-based test, and the second time round the test will be online? Content is the crown jewel of an assessment provider and needs to be held in a way that lets you get the most out of it.
5. Can you keep the authoring process tightly secured?
One of the benefits of moving to online authoring is that it reduces security risks associated with handling printouts. But phasing in new technology requires that you address new cyber security risks, with key questions being:
- Is the software hosted securely? You need reassurance that the supplier conforms to international cyber security standards with regular penetration testing and secure data backups to protect your content.
- How tightly can you control system access? Insist on two-factor authentication (where users enter a code sent separately after login) and the option to restrict access based on IP address and country.
- How tightly can you control user permissions? If you don’t want contributors to know what goes into the final paper you’ll need an e-authoring system that has options for managing user access based on roles and tasks.
- Can you track user access? Your project managers will need to see what items users have viewed or edited so that if a breach occurs swift action can be taken.
If the system you choose is flexible enough to deliver in all these important areas, then your test development process will be made much easier for your whole team.