An independent review has found that last year’s NCEA digital exam assessment glitches, which disrupted and stressed out students, were the result of a number of safeguards failing at the New Zealand Qualifications Authority.
Reviewer Debbie Francis says her investigation revealed the problems were caused by issues within NZQA and with its UK-based technology vendor RM.
“When these events took place, much of the response and recovery effort, while urgent and committed, was based on interpersonal interaction and ad hoc fixes, as opposed to clear frameworks and processes which had been designed in advance to reflect overall strategy,” she said.
In response, NZQA issued an apology this afternoon “for the experience schools and students had”, accepting the findings.
NZQA website glitch ‘a nightmare’ for stressed teachers, students
The problems started when some students couldn’t save their work during the NCEA digital maths co-requisite assessment on October 30.
Then, on November 2, there were lag issues across several digital exams, which saw NZQA stop students from logging on for at least half an hour. Lastly, on November 10, there were more log-on issues, the platform was locked, and there was lagging in the NCEA Level 1 English exam, Level 3 Media Studies, and Level 2 Le Faka Tonga exams.
The review said students were put under considerable stress during these incidents and there were repeated occasions of students walking out of their exams.
The review doesn’t give a number for how many students were affected but points out that 8850 applications for derived grades were received for the NCEA English Level 1 exam. She states that 16,884 students successfully completed the exam.
One of the problems was a defect when logging on to exams on the NZQA website, which caused significantly more attempts to be made between NZQA’s security application Keycloak and the vendor’s assessment platform Assessment Master.
This put the system under pressure and was not detected in testing by the vendor.
Testing by NZQA and the vendor gave different answers on how many people could be sitting digital exams at once, indicating assumptions and test processes weren’t aligned.
Despite being asked, the vendor didn’t provide test results to NZQA in advance of student exams.
Internal issues included a lack of accountability on staff for delivering the assessments successfully, optimism bias and a “defence” communications style with schools.
Chances of repeat low – Francis
Francis says the chances of a repeat are low given the agency’s commitment to improvement, but some schools fear there will be more problems this year.
The review states NZQA received a number of complaints from schools, families and students.
She concluded that she has concerns the agency will only partially address long standing cultural issues within the organisation.
Francis says she’s also concerned about NZQA’s ability to achieve the required change with a modest financial baseline.
“It will be important, as part of the reset of the operating model, that NZQA’s leadership considers how organisational efficiencies can best be secured to allow them to reinvest resources in reliable and failsafe mission-critical services such as digital assessment,” she says.
Francis has made a raft of recommendations to improve the provision of digital assessment. NZQA’s created a schedule for when the recommendations will be put in place.
‘NZQA sincerely apologises’
The authority accepted the findings and apologised. It said NZQA and RM are both committed to delivering “stable, reliable” digital assessment — and Jann Marshall, NZQA deputy chief executive of assessment, told 1News: “NZQA invests considerable effort to meet the needs of students and reflect the digital world in which we learn and live.
“The failure of the digital assessment platform to support the anticipated number of students for digital assessments on November 10 caused disruption to students and NZQA sincerely apologises for the experience schools and students had.”
She said the review was both “helpful and comprehensive”, noting it identifies the value of digital assessment as well as areas of improvement. She said NZQA accepts the recommendations and is working through them.
“We have developed an Action Plan to address the review’s recommendations within the report’s indicative timeframes.
“We are already making progress on the recommendations, including aligning testing protocols, refreshing the digital assessment strategy and reviewing NZQA’s own operating model for delivering digital assessment.”