#Censusfail Australia: Not an attack, data safe, no heads to roll “This was not an attack, nor was it a hack”: that's the official government position on the collapse of last night's Australian online Census systems, attributed to a denial-of-service attack. The chief statistician David Kalisch, the small business minister Michael McCormack and the government's infosec spokesperson Alexander …
the Australian Bureau of Statistics (ABS) decided to block all international traffic as its DDoS mitigation strategy.
If the ABS think that I will be going on-line to use their form without using services like TOR to hide my IP address (and probably look like I come from overseas) they are out of their mind. I do most of my web access via TOR.
Hey Oengus,
Think about it, you where posted an access code that you need to log into the portal. So they already have your postal addess and the access code. I am sure they can work out who you are even when using a proxy or any other method to hide your internet presents.
Having just listened to the statements, this is what I heard:
Our DoS protections were sufficient to protect against small-scale attacks during quiet times but failed to protect against a sizable attack during expected usage conditions.
The message we are supposed to take away is that the outage was due to a freak confluence of several (4) factors.
Unfortunately for the minister and the ABS, one of the factors was: "there was normal traffic which was significantly below the purported system capacity". I.e: that was fine. The other three factors all amount to: our DoS mitigation/protection was inadequate.
None of which is acceptable when that kind of DDoS protection is available as a service from any number of providers and can auto-scale on demand. Just the damned website should have been able to. And bloody first-year DevOps numpty rolled out of university should be able to bring THAT up on AWS or Azure today.
Exactly.
I did enjoy the attempt to imply that a router failing at the same time as the DoS hack attack "frustration" is somehow an additional factor that proves how exceptional the circumstances were and thus why the ABS should not be blamed.
Overloading routers is one of the ways you effect a DoS attack so the fact that the router failed emphatically proves that the DoS mitigation/protection supposedly in place was either not working or woefully inadequate.
To pretend this is anything but an unmitigated failure is to attempt to smooth-over incompetency and lies with a nice layer of dishonesty.
"MacGibbon said the vast bulk of DoS attacks are thwarted; ..."
"... Malcolm Turnbull has commented ... , labelling the incident a DDOS attack ..."
It wasn't an attack!
"(and Vulture South could not ask, because the press conference took place in a city where our operatives do not live)"
So, insufficient geographic redundancy, no failover agreements with other news providers, no emergency comms links. What happens if you or Simon are subject to a DDoS (Definite Display of Sickness) attack?
I work as Director of Technology for a "Customer Experience Management" firm (we used to be "Market Research" but that's not cool anymore) - so we basically do what the online census does for a living, setup pretty websites to ask questions and store answers.
Now, I know there's like 24m Australians, so there sample set is a little larger than my companies clients, and with a fine for not partaking, there's more of an incentive to take part, so uptake will be greater.
But, we run these things 24x7, no loss of data, international respondent base.
And we don't spend anywhere near $10m, I'm struggling just to comprehend how you break that all down. I mean, even if you bought hardware (Instead of leasing for the month you're actually online) and pay ridiculous software fees, I'd be hard pushed to spend anywhere near half that, and certainly be able to process more than 260 respondents per second, with failover, and DDoS mitigation..
..but then, secure by design is our motto, I'm not paranoid, you're dealing with confidential information on the internet for big businesses, anything can happen..
Easy, they even highlighted the fact in the story - IBM... Pay lots AND fail.
You get gold plated costs for a service that used to be good (unless you happen to get a few of the decent staff who are still capable and haven't been resource action-ed or left for less broken pastures) and most of the services will be delivered by a long chain of specialist staff that never read the documentation for how they are supposed to provide their services or know how to interact with the different teams within IBM. To address the cross-team/documentation issues, IBM bring in project managers who don't know anybody involved in the project or read the documentation. Finally, if you manage to get these bits working properly, there will be third-parties to work with that even fewer people know how to engage.
Yes, indeed. That is <0.94 Million/hour. Before I retired, I would use SQLite, a simple serverless database engine, to prototype systems - According to their website sqlite.org: "Situations Where SQLite Works Well" "SQLite works great as the database engine for most low to medium traffic websites (which is to say, most websites). The amount of web traffic that SQLite can handle depends on how heavily the website uses its database. Generally speaking, any site that gets fewer than 100K hits/day should work fine with SQLite. The 100K hits/day figure is a conservative estimate, not a hard upper bound. SQLite has been demonstrated to work with 10 times that amount of traffic... The SQLite website (https://www.sqlite.org/) uses SQLite itself, of course, and as of this writing (2015) it handles about 400K to 500K HTTP requests per day, about 15-20% of which are dynamic pages touching the database. Each dynamic page does roughly 200 SQL statements. This setup runs on a single VM that shares a physical server with 23 others and yet still keeps the load average below 0.1 most of the time."
Perhaps the Census could have run two bare metal powerful servers to get a similar result, and saved >$8 Million?
and as of this writing (2015) it handles about 400K to 500K HTTP requests per day
500K requests per day is only about 6 per second.
Perhaps the Census could have run two bare metal powerful servers to get a similar result, and saved >$8 Million?
I think they'd need a bit more than that. But I still don't know how to spend that much money on the job without flagrant gouging...
The other thing to remember is that 260 submissions per second doesn't mean 260 DB transactions per second; far too many shitey websites bury everything in the database, so a single page render can mean >50 transactions. That sort of coding hammers the DB.
Vic.
Sounds like they've just not accounted for the actual load you'll see on a thing like this.
260 forms/second means 22.4m submissions in a 24 hour period, assuming that there are no lulls or peaks (which is preposterous - nobody is going to submit their census at 4am. That means there must be a corresponding increase at some other hour).
Note that Australia has a population of 24m. So this was not specced for the entire population to submit a census online on the day.
If I was building it, I'd assume that 20m people would all submit their census in the 5 hours between 5-10pm, which gives me a figure of ~1100 forms/sec. I'd call that the "bare minimum" figure. If I thought it was important (like, for example, if it was a census), I'd estimate 25m forms in 4 hours, or about 1700 forms/sec, and I'd probably round that up to 2000 for a safety margin. 260/second seems ludicrously low to me.
Yep,
At 7 o'clock many are watching the news, they all get prompted to go fill out the form online, only a rerun of Dr Who, not missing anything.
There's at least 10 to 50 million attempts in minutes, everyone tries at least 3 to 10 times before giving up, (remembering there are stiff fines for not completing the census).
The minutes continue for hours. I retried three times through the evening, with repeated attempts each time.
The system looses it's manners and it's still a basket case. How long will it take to sort out the fuckup ?
I'm checking the site every morning, so much for everyone on Tuesday evening.
"Note that Australia has a population of 24m."
We think the population is around 24mil but we don't know. Maybe there's been a population boom and all 2.4 billion Australians trying to connect to the ABS server at once is what caused it to fall over.
In any case, it should be a fairly easy problem to solve. We can count how many Australians there are by taking a cens.... oh.
- Introduce online collection as an option, in parallel with traditional "humans-at-the-door" paper forms and phase to full online collection when confidence has been earned
- Speak to the public in terms of risk - likelihood and consequence - instead of using indefensible (and therefore easily demolished) absolutes.
We are fortunate that our Government is not evil and the human beings that work for it can be, and often are, among the best at what they do. Nevertheless, this exercise was engineered to fail - expectations were cranked up to maximum and the only two possible outcomes on the night were glorious success and humiliating failure. Glorious success never, ever, ever, happens.
Now, in the aftermath, the Government should be looking at those two things - taking the new system out of the critical path and avoiding the inflated expectations of absolutisms - but instead it sounds like we are going full steam ahead for absolutely secure absolute success.
Excelsior!
> - Introduce online collection as an option, in parallel with traditional "humans-at-the-door" paper forms and phase to full online collection when confidence has been earned
You mean like they did last time? Five years ago, I had a paper form, but filled out the online version, which ended with a verification code to be written on the first page of the otherwise blank paper form. And, if I recall, the extension of the web pages was ".nsf" ... scary!
just after 10 am - morning tea or got the kids off to school and just got home "ish"
just before 12pm - lunch or I'd better do this before I get the kids "ish"
after 7:30 - got home, had dinner, littlies bathed and off to bed or otherwise winding down "ish"
I went to a local club last night to help celebrate a neighbour's birthday. 12 households - 11/12 tried to log on sometime between 7 and 8 (the 12th couple us decided to use a form knowing their would be a titsup).
One Sydney dead tree rag has even quoted from a survey conducted last year that 16% of households use a VPN , and many are set to "US" source.
Massive, it should work in theory, lets forget about real human behaviour, fail.
Yup. That was my first thought when I saw the ABC's timeline: morning tea, lunch and after the reminder on the evening news.
I didn't, however, think of the VPN thing. I only know a few people who use commercial services so the figure of 16% surprises me. But it would explain the "foreign" traffic without resorting to conspiracy theories.
I suspect they mistook traffic spikes for attacks and panicked.
I believe it is the same company who supplied The Queensland Health dept., with a fool proof system as well.
Civil servant mandarins will award contracts on the basis of the best perks available to said employee.
Unfortunately it is the politicians of whatever ilk who take the blame never the civil service.
Now this is just Industry Gossip (tm) and third hand at that, so make of it what you will.
Disclaimer out of the way: rumour has it that the census website did not pass the 1 million users per hour testing but they went ahead full steam anyway because the deadline was looming.
This nugget comes via a colleague who was told by a client (a heavy ABS data user) who had it from one of his contacts at the ABS.
I'm now trying to find any official statement that says it did pass. All I can find is statements to the effect that it was tested...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
