A report into the big error that was discovered in Statistics Canada’s July jobs report makes several recommendations to prevent a recurrence, including improved oversight, and more formal testing protocols.
StatsCan released the results of the internal review it carried out to investigate the mistake, which caused the agency to revise the jobs numbers from no change to a 40,000 gain in employment for July. The review was initiated by the chief statistician a bid to determine what happened, why the mistake wasn’t caught earlier, and what can be done to mitigate the risk of these sorts of errors in the future.
The report finds that “a number of factors contributed to the publication of incorrect data”, but that the primary factor “was the lack of understanding among team members of the impact of changes made to the dwelling identification number on the system as a whole.” This change resulted in a higher number of individuals who reported being employed in June, and who did not respond in July, being imputed as either out of the labour force or unemployed in July.
The factors that resulted in this change turning into a published error in the jobs report include that there was an “incomplete understanding” of the jobs survey processing system by the team implementing and testing a change; the change was perceived as systems maintenance and the oversight and governance were not adequate; the systems documentation was out of date, inaccurate and erroneously supported assumptions about the system; the testing was not sufficiently comprehensive to catch this type of error; roles and responsibilities within the team were not as clearly defined as they should have been; and, communications among the team, labour analysts and senior management around this particular issue were inadequate.
As a result, the report concludes that the process would benefit from increased oversight and governance, a heightened awareness of risk, a more formal testing protocol, and enhanced communication. To that end, it makes a number of recommendations in the report designed to address the shortcomings that led to the data error.
In terms of governance, the report says that “given the importance of the [jobs report], and the complexity and age of the systems, all changes should be undertaken with a heightened sense of the risks involved and should not be considered to be regular maintenance.” It recommends that “proper governance and oversight be put in place, regardless of the size or perceived simplicity of the changes being made.” It says that the roles of the production manager and project manager, as well as developer, tester and acceptor, should be clearly separated; and, their responsibilities should be formally documented.
In this case, the changes were seen as minor and they were subject to limited testing. “Had a protocol to systematically test all the components rather than a subset been in place, the error might have been avoided,” the report finds. “We recommend that a formal testing protocol be developed and systematically implemented at Statistics Canada. It should take into consideration the importance of the program, as well as the age and complexity of the systems, and should clearly articulate the different roles and responsibilities of those involved in testing, as well as the scope of testing to be conducted.”
It also recommends that additional diagnostics and reports should be added to the production process for the jobs report to ensure that systems are running as expected; that a centralized set of documentation should be systematically updated and reviewed for accuracy when system changes are made; and, that briefings to executive management at the agency should be enhanced.