By Ethan Phillips-Richardson, Business Development Consultant at GradeMaker
From reviewers, to senior approvers, from asset managers to typesetters, it takes a multitude of people – all with different roles – to get a paper ready for the exam hall.
Navigating this situation can pose logistical challenges for everyone involved.
Many test developers use standard office suite tools to create their tests and pass content between the different stages in development. However, without a centralized tool to help track comments and changes made from multiple users, it creates a lot of manual work to keep a single version up to date.
Using Microsoft Words ‘Track Changes’ feature could be a potential solution to this however, you cannot easily audit comment history using the standard Track Changes function, which can lead to content being edited or deleted without leaving any trace. There is also a risk of edits being missed or ignored. Changes can be made without a record of who made the change and why.
By using a centralized authoring tool like GradeMaker for test development, awarding organizations can be sure that all comments and changes made to the content are recorded and saved. As the test moves through the fully customizable workflow, so does all the feedback data.
In this article you can follow 3 example items from inception, through the GradeMaker workflow and into a test ready for print. Along the way, I will demonstrate how different users can make comments and leave feedback for others to action later in the workflow.
There are 6 different types of comments/notes that be used to help improve the authoring process:
- Inline notes – During review
- Recommendation – When submitting a review
- Notes and Actions – Any workflow stage
- Recommendation – When submitting an approval
- Print Notes – PrintPrep
- Auditable Change Logs
It’s also important to know that all users can have their permissions edited, so you can restrict what individual users can see and edit.
Initial Review – Inline Notes
Once a question has been created, the author sends it on to the initial review step.
In this example we have 2 initial reviewers, Reza and Tom. Taking the first item, which is a mathematic algebra question, Reza wants to suggests a change to a specific part of the question, so he leaves an ‘Inline Comment’ on the relevant part of the item. By clicking the ‘Display Notes’ button he can then highlight a portion of the text and then add a comment to that part. As part of this you can either have:
- A General Note, which all subsequent contributors can see.
- A Targeted Note, which allows you to target the note to a specific person or step later in the item workflow.
Another useful feature within this comment tool is the ‘Note must be actioned box’. This becomes really useful if you discover a major error and need to ensure it gets fixed later on. Here we can see Reza’s comment on the item. Once the comment has been made it shows up highlighted on the item itself.
Reviewer: Actions and Recommendation
Another way to pass feedback along the workflow is by using the ‘Recommendation’ feature both for reviewers and approvers.
Once the user has finished their review, they submit it to the next step in the workflow and have the opportunity to recommend approve, change or reject.
If they choose either change or reject, then the comments box becomes mandatory and they must give a reason as to why they picked that option.
These colour coded options then show up for the next person, so before they open up the item/paper to complete their step they can quickly see their colleagues recommendations.
Notes and Actions Tab
After the first reviews have been completed, the item moves to the next step in the workflow. In this example the next step is ‘First Approval’.
On opening up an item the assigned approver can instantly see any feedback given by the reviewers.
By expanding each note, they can see what the comment is and where in the item it was made.
Where a comment has been ticked as ‘needs to be actioned’. Here they can make changes to the question and can mark each comment when they have been addressed.
Approver: Actions and Recommendation
If the item or paper arrives at its first approval step and doesn’t feel the question is ready, the approver can also choose to add a new step in the workflow or to request changes from the original author.
These are useful when an approver feels that something needs to be checked by someone not currently in the workflow.
In our example let’s say that the item involving the angles of a triangle might be better if we were to include an image asset of a triangle with the angle sizes attached to a point.
By choosing to request changes this will enable the item to go back to the original author, who can then add in an asset or asset brief.
The author can easily see the comment left by the approver, edit the item and make the necessary changes before resubmitting it back into the workflow.
Print Notes Attached to a PrintPrep Version of a Paper
Once the items have been approved and sent into the item bank, you can then use them to create a paper.
GradeMaker supports a dynamic, dual output approach. The same paper can be published digitally to a test player and also typeset for print.
With a print paper you can use the PrintPrep tool to help speed up typesetting. As part of this process you can add notes to specific parts of a paper to help aid the print design process. Once the paper has reached approval you can instantly generate a print preview of the paper.
Once the preview has been generated, you can leave comments and annotate the paper specifically for the typesetter.
The approver has 3 options for highlighting the area in question, a point, line or box. In this example question 3 has been spread over 2 pages, which can make it hard for the candidate to answer. By leaving a comment here this will alert the typesetter that they need to address this.
The approver can then leave as many notes as required before submitting the paper onwards.
Auditable Change Logs
Once a user leaves a note and ticks the ‘note must be actioned’ box, the item cannot be passed into the item bank until someone clicks that it has been actioned.
Once it’s been actioned there is a name, date and time stamp recorded next to it to show who actioned it and when. This can help with any audit procedures carried out in the event of an issue with the exam.
At any stage of assessment development it’s useful to be able to see the details of changes which have been made along the way. That’s why we developed the new Version Compare tool. To see how it works in practice we can take our example question about angles of a triangle, which had a couple of tweaks made during development as seen previously.
When comparing the original version to the final version, we can see the original spelling mistake and its correction, as well as the image that was added at the end. All the new changes are highlighted in green and anything that was changed or removed can be seen in red, making it quick and easy to see what has changed. In Version Compare, we can see who made each change and when, improving the efficiency of the audit process.
User Permissions
Within GradeMaker there are several different roles that users can be assigned including, setter, reviewer, approver, project manager and user manager.
A user manager has permission to assign roles to users within the manage tab. A user can either have full rights or partial rights for any of the roles.
Reza has partial rights to review both items and papers, which means that he can only ever review items or papers, not make any changes to the content itself, but just leave review comments. Currently Reza can only do this for certain qualifications.
Tom, who has some partial Approver permissions can make edits and approve items and papers but again only for certain subjects. This allows awarding organizations to have full control over who can do what and where.
Summary
GradeMaker Pro significantly enhances your ability to audit the entire process, increasing accountability and reducing the likelihood of errors.
This centralized approach significantly enhances your ability to audit the entire process, providing a much richer insight into the data on when changes were made and by who. By utilizing digital workflow tools you can more efficiently maintain quality control throughout even the most complex development cycle.
See how we can support you to do all this with quick demo from one of our friendly team members.
Our Customers
