If you’re looking to build a bank of assessment items, your first job will be to find a system to manage them in. One which is secure, easy to use and will help you to significantly improve your organisation’s assessment processes.
You may be considering these two options:
- To self-build, or contract an external developer to build, a custom item-banking system from scratch which will be hosted and run by you.
- To subscribe to an existing item banking platform (a licence model), in which you own your content but don’t have to do any development or maintenance work on the system.
Which of these is the best option for you? We have broken down some of the pro’s, con’s and common misconceptions when it comes to deciding on your next step.
Why a custom-built model is attractive
Assessment organisations have traditionally sought to have permanent control over their systems so they can decide exactly how they work.
This may be fuelled by a desire to build only the functionality they need, rather than pay for functionality they may not use.
It may also be borne from the common perception that licenses cost more over the long-term. Why pay for an annual license if you can build your own system and own the code? Plenty of companies can write bespoke software applications, so designing and developing a specialist assessment platform is a feasible option.
A custom build is often perceived as a one-off project, with a one-off cost, which makes it an especially attractive option for organisations with access to one-off grant-based funding.
Common pitfalls of the custom-build model
However, it’s very rare that development will end after the initial build. Applications need continued development and maintenance to remain secure and functional.
You will need to get your requirements 100% right first time to stay within budget. Very few organisations are able to do this unless they have prior experience in developing similar platforms.
Many find they need to make iterative improvements long after go-live as people identify functionality gaps while performing their day-to-day tasks in the new system.
Although there are a lot of developers offering custom builds, very few vendors have the required subject knowledge to ensure your software will fully support complex exam processes. Without this knowledge, your vendor may not understand your more nuanced requirements.
What to consider when scoping a custom-build project
If you decide to build your own system, there are a number of issues you need to evaluate before you start to scope the project and select a vendor:
- Do you have people skilled in analysing your operational processes and translating them into technical requirements?
- Does the vendor have the necessary assessment domain knowledge to translate your specialist requirements into good software?
- Is it possible your requirements will change in future, requiring ongoing software enhancements?
- Will you be beholden to your vendor for making modifications to the software and if so, can you be sure they will continue to be responsive and price consistently over the long term?
- If you are responsible for the ongoing management of the codebase after the build:
A) Does the vendor have any incentive to ensure the code is maintainable long-term?
B) Do you have the technical expertise to update the code? For example, in response to penetration testing will your team have the knowledge to manage any necessary patching? - Are you certain the vendor will deliver the project in the required timeframe?
- Do you have a team of people to provide functional and technical support?
- At the end of your system’s shelf life, will you be able to lift and shift your content to a new system?
The real costs of building, maintaining and self-hosting a bespoke system over a standard shelf life are illustrated here:
| Custom build and self-hosting costs over 7 years (USD) | |
| Initial build | $200,000 |
| Bug fixing | $50,000 |
| Code maintenance | $100,000 |
| Hosting and services | $50,000 |
| Pen-testing and security governance | $10,000 |
| Ongoing system training and consultancy | $50,000 |
| Technical support provision | $50,000 |
| TOTAL | $500,000 |
These costs should be considered as indicative and dependent on context. However, it is not an exaggeration to speculate the real cost of the project is likely to be over double the cost of the initial software build.
It should be noted that your product is likely to have a defined shelf life, after which the organisation will need to invest in a new system.
This is important because unless your organisation has a team of dedicated software developers responsible for continually upgrading the system, it’s likely to become outdated after a few years.
This is especially true if new applications of technology like generative AI are adopted by specialist providers and start to offer real value in the exam development process.
Why assessment organisations choose a license model
The licensed ‘Software as a Service’ (SaaS) model has become standard in most industries over the last few years because it offers increased security, allows users to benefit from continued functionality updates, and offers much more flexibility.
Tiered pricing based on usage and modular design mean organisations only pay for what they use. At the same time, the system can flex as their needs change.
New functionality that comes onstream also has the potential to improve operational processes.
The shared tenancy model allows organisations to join a community of similar users to inform future upgrades – and these developments are shared with the organisation at no extra cost.
SaaS systems commonly follow universal standards, making content generated in the system more portable. This means integrations with other systems are easier, which is particularly important if moving to new modes of assessment delivery.
As a result, some of the reservations that once drove organisations to choose in-house solutions no longer apply.
Most awarding bodies choose SaaS systems because they take away the burden and risk of managing complex technical infrastructure, allowing the organisation to focus on its primary activities instead.
Crucially, licensing a SaaS product can be more cost effective than custom building and on-premises hosting, or at least cost neutral over a standard 7-10 year shelf life. That’s because expensive ongoing costs are included in the licence, such as:
- Bug fixes and code maintenance
- Hosting costs
- Penetration testing and security updates
- Ongoing training and technical support for users
Hosting and security considerations
One of the major issues for a new assessment storage system is its ability to keep your content fully secure. Your choice of hosting is therefore of primary importance.
Many IT practitioners hold the view that if you can see your server in the corner of the room, and control how it’s protected, then your content will be more secure than using a shared hosting infrastructure.
However, advances in cloud hosting over the last few years mean this is now an outdated perception. Government agencies increasingly trust cloud providers with their sensitive data, just as most of us trust our own money to be managed in the cloud.
Cloud hosting companies are becoming increasingly global. For example, the AWS cloud spans 102 availability zones within 32 geographic regions around the world. It means an awarding body in Africa can have its data hosted in Africa while still benefiting from a cloud infrastructure.
Some of the security benefits of cloud hosting over self-hosting are outlined in this comparison below:
| Cloud hosting | Self hosting | |
| Physical security | Huge, with multiple layers (Perimeter, infrastructure, data, environmental) | Greater risk of physical security breaches, fire, extreme weather and power outage. |
| Patching and maintenance | Cloud providers are the creator/suppliers of security patches and best suited to deploy quickly | Requires expertise and strict governance to ensure the application is patched and maintained properly and in a timely manner |
| Responsibility | Cloud providers manage security for all areas they are responsible for | Everything has to be managed on-premises. Requires considerable expertise and expense. Personnel salary has to be paid all year round. |
| Encryption | Easily enable encryption of data in transit and at rest | Encryption of data has to be configured manually |
| Security provision | Next generation security layers supported by AI, configured to work together by default | Complex and costly to implement state-of-the-art security provisions locally. |
| Data centre resilience | Multiple layers of resilience built into infrastructure (data networks, power, cooling, etc.) | Available premises may not offer an ideal environment for your hosting infrastructure (e.g., could be subject to power outages) |
| Demand | Elastic demand infrastructure enables you to meet peak usage and is economical | Data centre must accommodate for peak usage all year round |
Front end security
It could be argued that frontend security is the most important consideration when planning for an item bank.
Like most government-procured applications, your item bank must offer the best possible protection from cyber-attack. Additionally, it must also be designed to reduce risk of malpractice among its users, given the confidential nature of high-stakes exams.
When you have a wide range of contributors and reviewers needing access, it’s important to make sure that the right people have the right access, at the right times. You will want to ensure that any system you use for assessment management is not at risk of being accessed or altered by users who are unauthorised to see the most confidential content.
Some important elements which should be included in your system include:
- Multi-Factor Authentication
- Lock out on unsuccessful login
- Timeouts following inactivity
- User and role specific permissions controls (so for example, reviewers can only access assigned content on a read-only basis)
- IP address whitelisting / blacklisting
- User activity auditing (allowing administrators to see who accessed what, when and from where)
- Independent penetration testing
- Data backups (daily, weekly and monthly)
Whatever system you choose, whether off-the-shelf or custom-built, make sure it is designed to the ISO 27001 security standard and that accreditation to this standard is maintained throughout the life of the product.
Summary
Your teams are performing one of the hardest jobs in publishing. The content they publish will affect the life chances of young people across the country, so items and tests must go through the most rigorous quality assurance processes. At the same time, they are working to the most challenging, non-negotiable deadlines, and must undertake this work under the tightest security.
Developing a bespoke system that supports these complex requirements is a high-risk strategy. Finalising a system that can be rolled out and maintained across the organisation will be a time-consuming and costly project.
For all these reasons, most national-scale awarding organisations choose to adopt a pre-developed, specialist item banking system.
With a subscription service you can avoid common issues:
- You don’t have to wait before you see the benefits of the system.
- You don’t carry all the design, build and security risk.
- You don’t carry the full burden of training, technical support, bug fixes or long-term maintenance.
- All aspects of security testing and maintenance are included in the license cost.
Instead, you transfer all these risks to the provider, who manages these risks for other significant assessment organizations.
Besides all this, you also gain significant benefits too:
- You have more control over how much you use and spend, with a system which is more cost effective than custom building and on-premises hosting.
- You will join a community of similar users who help to inform product upgrades which will be shared with you.
- You can be confident that you are working with experts in assessment, whose reputation depends on delivering high-quality, efficient and secure services.
If you want to see how a ready-made item banking system could work for you, book a personalised GradeMaker Pro demonstration today.
Our Customers
