The 2016 Australian census was, by any account, a complete flop. Technical issues, claims of attacks by ‘hackers’ and privacy concerns marred its transition to an online format. This online transition seems an almost inevitable one. But what exactly was all the trouble?

To begin with, the combination of a new online format, along with news that names would be attached to results and held for four years, kicked up privacy concerns for many Australians.

In December 2015, The Australian Bureau of Statistics (ABS) quietly announced that they would be retaining names for four years, citing the ability to “enable a richer and dynamic statistical picture of Australia”. The mere 260-word media release was vague at best and provided no specific details as to why the change was necessary. The ABS quoted a ‘Privacy Impact Assessment’ which found “retaining names and addresses… has very low risk to privacy, confidentially and security”. Perhaps even less convincing was the revelation that the ‘Privacy Impact Assessment’ was conducted by; you guessed it, the ABS.

In the proceeding months, the inevitable backlash rolled in. The outspoken independent senator Nick Xenophon joined Greens senators Sarah Hanson-Young, Scott Ludlam, Lee Rhiannon and Larissa Waters, as well as the controversial Jacqui Lambie, to publicly boycott attaching their names to the census. Former Deputy Privacy Commissioner of Australia Nigel Waters was employed by the ABS 11 years ago to assess the benefits of identifying data. He lashed out against the ABS, telling the ABC’s 7.30 Report, “What they are now doing is breaking the contract or the deal that they have had with the Australian people for the last 100 years … their part of the deal was to make sure that the information would be de-identified as soon as possible.”

Others like The Australian’s Judith Sloan questioned the need for a census all together. Sloan criticised the $300 to $400 million price tag, writing in August, “There is virtually no information that the census provides that is not available from other sources more accurately and in a more timely fashion.”

Unbeknownst to many, the idea of anonymity in the census is largely an illusion to start with. According to the Sydney Morning Herald, the ABS has always held onto names for up to 18 months. The move to 4 years realistically only formalises and extends an already existing practice. The ABS also, without consent, tracks around 5 per cent of the population through ‘linkage keys’ attached to names that enable longitudinal data observation over 5 year periods.

August saw the roll out of advertisements reminding us why the census was important. Despite public outcry, privacy concerns were largely left unaddressed. Minister for Defence Industry Christopher Pyne dismissed the concerns as ‘tin foil politics’. Prime Minster Malcolm Turnbull took a typically more diplomatic approach, assuring Australians they could trust the integrity of the ABS’s systems.

When census night arrived, #CensusFail exploded and the supposedly robust system flopped. Millions were met with an error message and no explanation. Head of the ABS David Kalisch blamed ‘malicious attacks’ from overseas forcing the ABS to shut down the site for “safety precautions”. Only hours later, the Minister responsible for the census Michael McCormack wound back the rhetoric, telling Sky News there was no ‘attack’ or ‘hack’ rather, an “attempt to frustrate the collection of the data”.

To add to the embarrassment, popular technology commentator Trevor Long reported that two students from the Queensland University of Technology, Austin Wilshire and Bernd Hartzer, have built a system using a expandable cloud server capable of coping with four times the amount of traffic the ABS faced. The students claimed to have built the system in 54 hours during a ‘Hack-a-thon’ for a grand total of $500. That’s $9.6 million less than the ABS paid IBM to build the failed system.

After much confusion, the public was told the failure was due to four distributed denial-of-service (DDoS) attacks. A regular denial-of-service (DoS) is an attack where a website is inundated with requests, overloading the system and blocking out any would be legitimate users. A distributed denial-of-service is the same as a DoS except it is attacked from several systems at once, making it much more difficult to defend.

Although it might be playing a game of semantics, the Minister was right to say that a DDoS isn’t a ‘hack’ as such. No data was compromised in the process, nor was it the aim of the attack.

Not everyone was convinced by the DDoS explanation. The ABC’s The Conversation pointed out that aside from claims from the ABS and Minister McCormack, there was no evidence to suggest a DDoS was to blame. ‘Digital Attack map’, a website that monitors DDoS attacks around the world, reported no activity that would suggest a DDoS attack in that time period. Perhaps it is more plausible that the ABS’s system was simply overloaded or brought to its knees by a very unsophisticated DDoS attack as a result of overloading.

There is a legitimate reason as to why we have still have a census. Despite much of the information already being available, it gathers important data on population composition, geographic distribution, homelessness, welfare of Indigenous Australians and other minority groups. There is also legitimate reasoning in terms of efficiency behind retaining names. Canada, the UK, New Zealand and many other countries across the world retain names and addresses to improve data quality.

Regardless of the merits of retaining personal information, there is an underlying issue of communication. Changes that so closely infringe on individuals’ privacy require careful consultation with the public, not a barely-noticed media release and a couple of tweets. Ministers should certainly not be belittling legitimate privacy concerns in an increasingly data driven world. Had the public been provided with an informed rationale explaining why the step was necessary, and the assurance their data would be safe, perhaps the whole ordeal could have been avoided.