back to article Australian national census fails in the IBM cloud

Australia is today conducting a contentious national census, and things aren't going smoothly. The nation's Bureau of Statistics (ABS) has hired IBM to run the online data collection portal and at the time of writing The Register's antipodean outpost, and plenty of others, are finding it impossible to reach the site. Your …

  1. mr. deadlift

    IBM, i shoulda known.

    there's a reason they're prohibited in QLD, you know.

    oh well guess el reg saw it coming and had this ready to go.

    1. Adam 1

      Re: IBM, i shoulda known.

      If El Reg couldn't see this coming then I would be changing news outlets. Blind Freddy could see that provisioning for a million people per hour isn't enough when most families will get home from work, eat dinner, kids in bed then log in. The saddest part is that because they make names compulsory, the results will be less than honest, negatively impacting public policy decisions for the next 5 years.

  2. Anonymous Coward
    Anonymous Coward

    The actual deadline for completing the census is 23rd September, so anyone who fails to do it today will not get fined and can complete it in the next 4 weeks or so.

    Unfortunately the messaging from the AUS Gov seems to have forgotten to point out that actual deadline day.

    So this is essentially a self inflicted Denial of Service attack.

    What are the penalties in Australian law for this, and who in the Australian government census "marketing" department is going to get prosecuted for this?

    1. BebopWeBop

      And common experience (see the UK historical census data being put online for public consumption - all over 100 years old of course) as a classic example of under provisioning and failing to anticipate demand). The UK tax return is similar - massive spikes in the few days before deadline(and failed service) - but if the deadline is there, then surely it should be honoured - none of the documentation mentions that people who fail to log their returns will be excused if though demand the service can not be provided.

      Now I agree that anyone with an ounce of sense will have filed well before the deadline - little is likely to have changed in the last couple of months, but a little sensible social engineering combined with proper provisioning (hey - isn't that what the cloud is meant to provide?) right alleviate things.

      1. NotArghGeeCee

        Optional

        I realise you are not directly addressing the census but, for full clarity, you cannot complete the census in advance - it is intended as a snapshot of the nation on a particular day. As such the demand for the site should have been anticipated well in advance. Or, if it is a DDoS - as they now claim - then that possibility should've been anticipated as well and had suitable defences organised.

        1. Anonymous Coward
          Anonymous Coward

          Re: Optional

          Uhm, no. I did it last weekend, for the heck of it. And also because I'd had enough wine intake to come up with some "inspired" responses.

  3. Mark Simon

    More from down under …

    As with many others, I cannot connect. There is a phone number you can call.

    The recorded message says (a) there’s too much demand, try again after the 10th; (b) you won’t be fined for competing the census after Census night.

    Tried again later. Now the recorded message says that business hours finish at 10:00 pm. Try again tomorrow or visit the web site.

    I’ll bet nobody saw that coming. Oh wait. Everybody saw that coming except the ABS.

    Twerps.

    1. Anonymous Coward
      Anonymous Coward

      Re: More from down under …

      They obviously didn't expect the servers to crash and burn so quickly. I suppose it should have been expected when it was organised by MBAs who generally don't have a clue.

    2. Mark Simon

      Re: More from down under …

      The ABS says that the site was attacked 4 times by overseas “hackers”, according to the ABC. Whether that really means it was hacked, or simply DOS is not clear.

      What is clear is that they can’t possibly claim that your personal data is safe from attack.

  4. Anonymous Coward
    Anonymous Coward

    Go to bed

    Wow #CensusFail trending on twitter, who would have thought so many Aussies cared about getting their Census submitted.... #GetALife

    1. Poe

      Re: Go to bed

      As opposed to the amazing life of an elReg AC commentard?

    2. dan1980

      Re: Go to bed

      @AC

      You misunderstand - perhaps deliberately.

      The problem that is being highlighted via use of this hashtag - and the numerous other ways the failures have been reported and commented on - is that the ABS have taken the arrogant stance of: "trust us - we are unimpeachable and your refusal to trust us will be punished" and then showing themselves to be utterly unworthy of that trust.

      They defended their unprecedented decision to retain personally-identifiable information by telling us that they had never had a breach in the past and then, when their entire submission system goes offline, they cry: "it's not our fault - we were hacked!".

      The claim that the ABS's defense of previous hacking attempts should somehow be taken as a guarantee of their ability to resist future hacks deliberately avoids acknowledging one very important point: that the data they are collecting this year is - by their own admission - far more 'valuable'.

      However you slice it, they have proven that they really cannot be trusted.

      If the systems were subject to a genuine, malicious, DDOS attack (and there is no independent proof of that, yet) then they have shown that they either underestimated the size of the target (and were therefore under-prepared and under-protected) or they did not do the due diligence to ensure the system being built to capture our private information was fit for purpose.

      If the systems weren't 'hacked' then it went down either due to the traffic exceeding the ABS's expectations, which again means they do not understand the data they are dealing with or, again, they failed to do due diligence to ensure the 3rd-party solution our tax dollars paid for was up to the task.

      With the retention of names and addresses, this census will create one of - if not the - largest single collections of sensitive, personal data in Australian history. This raises questions and concerns and the answer from the ABS has been a revolving door of silence, arrogance, condescension and bravado.

      If, as they demand, we view them as competent due to their past competence then I find it logical, fair and prudent to view them as incompetent when faced with such overwhelming evidence to their current incompetence. After all, if your proof of trustworthiness is based on being infallible then why should we trust them again after such a colossal f$#k-up?

      Some may argue that it's unfair to judge them by one mistake but when you're dealing with the sensitive information of the entire country - information of inestimable value to eager spies and malicious actors - you don't get second chances. You can't wave it away and say: "we'll do better next time". These eggs cannot be unscrambled.

      The collection, handling and storage of this data must be perfect at every stage and for the entire life of the data. No mistakes, no oversights, no cut-corners, no assumptions, no miscalculations, no loop-holes, no bugs, no cracks, no absent-minded omissions.

      If it fails anywhere, even once, then the damage to could be immense. You don't get a second-chance at a perfect record.

  5. iLurker

    Paper forms don't have bandwidth limits, nor limits on the #connections that can be handled concurrently.

    Likewise unable to log in for hours. Going to bed.

  6. Sampler

    Big fat red x

    "Thank you for participating in the Census. The system is very busy at the moment. Please wait for 15 minutes before trying again. Your patience and cooperation are appreciated. [code 9]"

    Not that I want to fill it in online anyway given the other articles, but also, filling in the paper version sounds like it'll be added to the same insecure dataset anyway (and all manner of things done to it too).

  7. Version 1.0 Silver badge
    Joke

    I called .AU tech support ...

    They said that they'll get it sorted out but right now they are busy with a data center in Atlanta that's down.

  8. Anonymous Coward
    Anonymous Coward

    idiots ran a census on Patch Tuesday!

    I wonder how much data big blue can lose for the Aust federal govt?

    Probably safer for the punters anyway. Thanks IBM!

    FFS!

    1. Anonymous Coward
      Anonymous Coward

      Re: idiots ran a census on Patch Tuesday!

      "Patch Tuesday" for you people west of GMT is actually "Patch Wednesday" for us.

  9. Anonymous Coward
    Anonymous Coward

    Site still down

    Just out of curiosity I have been trying to load the site for the last hour but nothing. Not relevant to me as I decided to extend my trip overseas by two days so will miss it.

    1. Anonymous Coward
      Anonymous Coward

      Re: Site still down

      Checked a moment ago, and the main ABS site is down.

      We did ours Monday afternoon, as we predicted the site would crap itself.

      1. Anonymous Coward
        Anonymous Coward

        Re: Site still down

        The site appears to have returned.

  10. Anonymous Coward
    FAIL

    IBM really does stand for Its Better Manually!

    Maybe you should send the Aussie government saying that you couldn't fill out your census, and tell them to bill your penalty to IBM.

  11. The Nazz

    All things considered, not that bad an effort.

    Contrast that with the UK's 2011 Census ( pop'n about 2.5 x that of Oz) :

    Approx cost of some £495m

    Thousands (tens of?) of returned and completed forms lost before census day.

    Thousands (hundreds of?) of useless forms, (duplicate or non valid addresses) issued in advance.

    Return rates in a lot of London Boroughs down in the 60%'s.

    Successful prosecutions for non-return? Very few if any, i guess.

    Yep, the Aussies have some way to go to match that "success" rate.

  12. BasicChimpTheory

    "This site can’t be reached

    stream10.census.abs.gov.au’s server DNS address could not be found.

    DNS_PROBE_FINISHED_NXDOMAIN"

  13. Anonymous Coward
    Anonymous Coward

    Stop offshoring capability IBM

    This is a great example of how IBM's strategy to move all the people who actually deliver their solution to India is a recipe for disaster. The Indians would have done the testing but with no knowledge (or indeed interest in getting that knowledge) of the local requirements. Ask any existing Australian customer of IBM and you will get a story of declining services and the removal of key people at the whim of the US based accountants.

  14. BlackKnight(markb)

    Words in at the moment that the ABS suffered four seperate attacks (non specific details) on tuesday and around 730 disabled the system them selves to protect the integrity of the data. quote from chief statistician.

    expected to be back 9am today.

    1. NoHandleYet

      "expected to be back 9am today." Tried at 11:30am still down.

  15. Phil Kingston

    Their Contact Us page seemed to fair better.

    I used it to tell them that I tried to comply, they failed and they can stick any resultant attempt at issuing a fine into IBM's hole.

  16. Oengus

    Great Ad

    IBM's role in the exercise is also likely to be questioned - it was paid at least AU$9.6m to design and implement the eCensus solution.

    <sarcasm>This has to be a great ad for IBM's cloud services... Not able to scale quickly. Not able to handle DDOS. </sarcasm>

    I thought these were some of the primary reasons cited by the Cloud sales people as the benefit of Cloud services over in-house.

  17. Anonymous Coward
    Anonymous Coward

    Only a government agency...

    would choose SoftLayer.

  18. cbommm

    There was no DDOS

    They budgeted for 1 million 'form posts' per hour - was this users or HTTP POST requests?

    It would not be unexpected to have 1 million people completing the census at the exact same time, 6 million per hour is a more realistic estimate.

    As for the claims of a 'foreign hackers' - geo-blocking at network gateways is pretty common. By definition only people at home in Australia on the night should have completed the census, so no one outside Australia needed access to the site. And if you don't know your friendly Network Engineer, even CDN's normally offer geo-blocking.

  19. Anonymous Coward
    Anonymous Coward

    So obvious

    Called it as soon as I found out that they were planning on everyone doing it the same day - as someone said above - Self inflicted DoS

  20. Richard Freeman

    Statistical fail!

    Apparently they could only handle 1,000,000 form submissions per hour, and it doesn't take a statistical genius to work out that more like, 10,000,000 people would log on after dinner, which would be probably between say 6:30pm (EST) and, well I guess most folk had given up by 7:30 pm (I am guessing word was out by then and SA, NT and WA didn't bother).

    - I mean some people have real jobs and can't log in during the day like our PM evidently did.....

    "Asked about the ability of the online census database to cope with such high traffic numbers, an ABS spokesman said online could handle "1,000,000 form submissions every hour. That's twice the capacity we expect to need.""

    http://www.smh.com.au/business/consumer-affairs/census-2016-chaos-for-australians-ahead-of-august-9-20160802-gqizw5.html

  21. mc nobby
    Alert

    a Mystery DDoS attack you say

    So the ABS is claiming some mystery DDOS attack, yet there seems to be no evidence of that at all

    If you look www.digitalattackmap.com for last night there was no recorded DDOS attacks in this country.

    Also if you are paying $10M for a cloud service

    a) I would hope the SLA the ABS should have with IBM would mean they can get a fair amount of compo

    b) That someone would have thought about a DDOS and designed the system to cope with is.

    Update: I see in the press conference that they had, they are now blaming an overloaded switch.

    https://www.theguardian.com/australia-news/2016/aug/10/census-2016-website-was-not-hacked-or-attacked-government-claims

    so they obviously have never heard of redundancy, or distributed infrastructure

  22. Anonymous Coward
    Anonymous Coward

    eVoting?

    And to think that there are still some people here in Oz who believe we can have electronic voting in the next quarter century.

    Make that the next half century.

    1. kain preacher

      Re: eVoting?

      eVoting will work like a charm. It will cost much less. See since the votes will be counted way before the actual elections happens. This is to make sure that the right people vote, this the web site does not actually need to record any thing. It just needs to be a bland two page web site.

  23. Anonymous Coward
    Anonymous Coward

    23rd Sep not a statistically useful date

    @AC said "The actual deadline for completing the census is 23rd September, so anyone who fails to do it today will not get fined and can complete it in the next 4 weeks or so."

    Which is true. But not useful. The whole point of a census is that it provides an accurate baseline for other statistics. That means the "hours worked last week" figure really needs to be completed on or shortly after 9 Aug, when the hours are clear in the respondent's mind. Not on 23 Sep when the respondent's recall will be inaccurate.

    The original intent of six weeks was to allow for postal and other delays for a small group of respondents, not for a significant proportion of the population.

  24. hypernovasoftware

    ObamaCare web site redux

    Back in October of 2015, the ObamaCare web site was down pretty much most of the time because they didn't do an incremental roll-out, similar to your census problem.

    Even after getting logged into the OC web site, the usability sucked.

    You'd think a web site that cost $2 billion would be damn near perfect. Ha. You'd be wrong.

  25. Doctor Huh?

    Not a TITSUP, though

    Seems to be totally supporting usual performance.

    What new term can describe this?

    Standard Horrible Information Transfer?

    Continued Reduced Apache Performance?

    Failing Accumulated Cloud Engines Performing As Little Machine?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like