back to article Official probe into HPE’s Oz 3Par crashes would create 'further negative publicity' if revealed

The final report into the two major failures of HPE 3Par storage area networks at the Australian Taxation Office (ATO) would likely lead to “further negative publicity” for the vendor – which is one reason the ATO has decided not to release the document. HPE's 3Par kit fell over in December 2016, and then again in February …

  1. amanfromMars 1 Silver badge

    Rotten Apples in the Pork Barrel

    Of course there is something to hide ..... and it be the simple naked truth. Corrupt and perverted systems and systems administrations cannot handle and debunk it.

    And whenever your lives revolve around the maintenance and propagation of ignorance is your world view a contrived virtual reality program with future memories already pre-ordained to result in conflicting resolutions ........ and that is no less than madness confirmed and running riot and rampant throughout hosting media channels.

    1. A Non e-mouse Silver badge

      Re: Rotten Apples in the Pork Barrel

      That almost makes sense. Has amanfromMars’ account been hacked?

      1. John G Imrie
        Terminator

        Re: Rotten Apples in the Pork Barrel

        amanfromMars has been getting more and more coherent recently, I think we are almost at the singularity.

    2. Anonymous Coward
      Anonymous Coward

      Re: Rotten Apples in the Pork Barrel

      "... future memories already pre-ordained..." Rather a lot of that and the trend line is steepening. I can't, sadly, yet see the shape of the pre-ordained outcome.

    3. Anonymous Coward
      Anonymous Coward

      Re: Rotten Apples in the Pork Barrel

      Truly a High Time of the Style'n'Linguistics Shakesperience.

      Funny how many of us dull dusty typing pad bangers are trying to dig out where is the money matter? (-;

      Rhetorical af

  2. naive

    The 3Par of a customer crashed in 2017 as well

    We had the same with the 3PAR operated by the city council of a mid sized city. unavailable for days as a result of an unplanned power outage.

    HPE support is dismal, requiring sticking a NATO alphabet under the screen for explaining things to the, mostly Indian, support engineers. Next time we help customers with tenders exceeding several hundred of thousand Euro's, we will recommend to put in a requirement that support is provided by European or US based staff.

    Annual support costs for the SAN run into the high 5 digits, for that HPE can not even provide support which is pleasant to use.

    1. Anonymous Coward
      Anonymous Coward

      Re: The 3Par of a customer crashed in 2017 as well

      There is no US support anymore. They've laid off their Field Techs and closed the support center in Roseville CA. There is some support based in the EU, but HPE could not move jobs to India fast enough, even though the quality of support from there has always been horrible.

  3. onefang

    I can see a Streisand Effect coming soon.

  4. Pascal Monett Silver badge

    Don't you just love it when so-called democratic governments do a public cover-up ?

    I just love it when any government spouts nonsense to cover up some embarassing failure, but when a "democratic" government does it "for the good of the People", masking why a project paid for by The People's taxes failed abysmally, it has that special "you just continue paying your taxes and watching football, we know what we're doing" tang.

    If it's paid for with public money, the Public that paid for it has the right to know the why and how. The only thing you're really saying is that someone should be losing his pension over a monumental cock-up.

    And we can't have that, now can we ?

    1. FrankAlphaXII

      Re: Don't you just love it when so-called democratic governments do a public cover-up ?

      And what's even better about this situation is that it's not just the people's tax dollars, it's the agency that collects said tax dollars. Can't imagine why they'd want to hide fraud or waste, not at all.

      1. Adam 1

        Re: Don't you just love it when so-called democratic governments do a public cover-up ?

        Fraud!? At the ATO? Shirley not. I mean maybe one or two bad apples at the lower rungs. No-one important though. Nothing like (former) Deputy Commissioner level, that's for sure.

  5. Anonymous Coward
    Anonymous Coward

    Reading between the lines...

    "the prospect of further negative publicity it could generate for HPE"

    It's very interesting that the ATO only mentions HPE in its letter of denial when DXC were also involved "...tech services giant DXC, which installed the storage boxes..." in the implementation.

    Are the ATO trying to protect HPE or are they implying blame?

    1. Stoneshop
      Holmes

      Re: Reading between the lines...

      It's very interesting that the ATO only mentions HPE in its letter of denial when DXC were also involved "...tech services giant DXC, which installed the storage boxes..." in the implementation.

      You are aware that DXC is the services branch of HPE, with just another name and half the knowledge lopped off?

      1. Anonymous Coward
        Anonymous Coward

        Re: Reading between the lines...

        @Stoneshop: No, I wasn't, but I don't think it makes much difference with regard to the question of whether the ATO are trying to protect HPE or point the finger of blame at them. It was the ambiguity in the ATO statement, and whether that ambiguity might have been intentional, or not, that I found interesting.

  6. adam payne

    It later emerged that the ATO had disabled error reporting

    Stupid

    The ATO also did not have a failover rig in place in case of an breakdown because it couldn’t afford one, and was also unprepared for outages.

    Even more stupid

    the report describes proprietary methodologies that HPE and DXC use to design and deploy 3Par SANs, and that if that information were to be available that could hurt HPE

    I can see how that could hurt them if everybody else when decided to use the same kind of methodology and then try the undercut on pricing.

    Unless of course the methodology is so stupid it makes them look like a bunch of amateurs. Now that would be funny.

    1. Agamemnon

      Thank You. was wondering when we were going to get to the IT bit.

      I've trained on 3Par before HP acquired them (also do EMC and HDS, NetApp if I'm slumming it, NexSAN if I'm having fun), it's good kit. Aaaaand Kit Falls Over sometimes. It's called entropy and we plan for it. (Six "Ps")

      But, priceless.

      1. Turned off monitoring. *BAM* The kit isn't to blame. It could have been calling for help all along but we'll never know. Everything Else is finger-pointing masturbation. Seriously, everyone's fired. That is blatant incompetence. Monitoring is there for a reason. Now I want to see their monitoring and alert system. Bets on it's (non)existence?

      2. They didn't get Mid-Tier hot standby for their array. Couldn't Afford was almost certainly Wouldn't Afford. I won't deploy without a mid-tier. Hey, save money now and we just populate trays with just what data-size you have today. Expand as required. No biggie. But, you Will get a second chassis our I Will walk out of the project.

      Now support, come on mates, we all know all support is outsourced, we all know to whom, so to feign surprise and whine is just silly. If it had been architected correctly, that point would have never been reached.

      I don't see any problem here with the kit itself other than, that's what lots of us here get paid for. I see:

      * Manglement.

      * Bad finance decisions.

      * Bad Archicture.

      * Bullshit SysAdmin skills. (THAT team needs LARTed, and blackballed from the industry, no it isn't harsh.)

      * I'd like to see that report that describes HPE's deployment methodology (and here they are https://www.google.com/search?q=hpe+3par+deployment+guide&oq=hpe+3par+deployment ) because I bet it looks more like I describe above (you bet your ass EMCs does, they want to sell you ALL the chassis, and monitoring, and dedupe, and some cold storage, and some of-site, and a pony...so I'm guessing HPE might be ever so slightly less aggressive selling kit). I bet their report is blatantly full of shit.

      1. Anonymous Coward
        Anonymous Coward

        I recently dealt with a company

        That had purchased a mini-3PAR i.e. an 8200. The company had a budget that they were determined to stick to and fought mercilessly with the previous implementers. What they got was an implementation that HPE and the distributor made the client sign off on that it was totally not supported and anything that goes wrong is your own fault.

        They had performance issues from day 1 despite being 100% SSD and took less than 12 months until they had capacity issues. They had a very demanding and large SQL workload that was killing the array with very large block sizes. I had access to the internal HPE sizing tool and even with dedupe and compaction turned off they would never have the required IOPs with what they had.

        The client refused to accept culpability and all suggestions to remediate fell on deaf ears because they didn't like the price of the additional SSDs. They had an exchange server log blowout for failed backup reasons and they didn't have anywhere to go, space wise. It was quite pleasant seeing them go down in flames. Even then, they still didn't want to fix the problem, because money. They messed about for so long that HPE eventually canned the 480GB SSDs they used and the next step up in availability, was 2TB SSDs and they needed multiple to create the new RAID. Suddenly, remediation was at least twice as expensive because it was IOPs more than storage that they needed. HPE were really embarrassed with the bad PR this could bring them that they offered the drives at cost with near $0 implementation cost and they still wouldn't buy them, it was maddening. They had this hair-brained idea that because they were planning to move to Office 365, the Exchange system would go and they would save space/IOPs, but in the 12 months we dealt with them they hadn't even started. We eventually annulled the managed services agreement with respect to SAN management and performance because of their bad behaviour.

        I do think the previous implementers should have walked away from the client and HPE should have refused to sell them the kit, but obviously commission distracted from the disaster waiting to happen.

  7. Stoneshop
    Facepalm

    Crucially, the denial letter we were sent also mentioned that disclosing the final report raised “the prospect of further negative publicity it could generate for HPE”

    And holding back this information won't have customers, current and prospective, figuring there's something dicey to be kept hidden.

  8. Anonymous Coward
    Anonymous Coward

    Please see the TSB meltdown

    ATO management don't want anything published.

    At the moment, everyone just suspects that IT management at ATO all come from the same non-branching family tree in a small Tasmanian town.

    Publishing the papers would confirm it and possibly result in useless idiots being shuffled between cushy management jobs....

  9. JohnFen

    Very clever

    "disclosing the final report raised “the prospect of further negative publicity it could generate for HPE” "

    This is very clever. This statement is an invitation to speculate about what is in the report that would be so unfavorable to HPE, and when people speculate without evidence, the speculation is usually even worse than the truth.

    Could this be a bit of jiu jitsu? Everyone will think the HPE is hiding something truly awful, so they'll push to release the report in order to reduce the impact of that?

  10. J. Cook Silver badge
    WTF?

    My first thought was that "HPE doesn't want the report out, because it implies something we've already known for a couple years now about their service and support."

    But having had some glimpses into the local governments up here and their overall competencies, I can also understand the ATO wanting to keep their skeletons at theirs desks as well. (because even the cheapest POS storage array about there has the ability to send email alerts, and that the monitoring and alerting facilities are near the top of the list to configure on a brand new appliance.)

  11. Anonymous Coward
    Anonymous Coward

    1) We cannot disclose the magnitude of our stupidity as it would tilt the world off its axis and cause harm to everyone.

    2) Our actions were so uniquely stupid that we chose to call them 'proprietary methodology' instead.

    3) We managed to put 'transparent approach' and 'proprietary methodology' into one sentence.

    4) We know we f@cked up, but we want to keep our jobs - so go f@ck yourselves and stop asking questions.

    5) We can get away with 4) because of 1), 2) & 3)

    6) If you continue asking questions we will blame HPE and they will sue us (the government). Then everybody (the taxpayer) has to pay HPE ...

  12. Anonymous Coward
    Anonymous Coward

    rumours

    one controller of a pair failed and support turned off the error reporting. Why is unknown. Secondary controller developed problems, no-one noticed. That ATO got nice new big storage units at a very very good price is just hearsay

    1. Ken Hagan Gold badge

      Re: rumours

      "support turned off the error reporting"

      If true, then I have to ask "Why is it possible to turn off the error reporting?". Ignore it, by all means, but when the shit finally filters through those fan blades, the error reporting is what distinguish a problem caused by the end-user from a problem with the kit. So why would the vendor *let* the end-user remove such evidence from any future investigation?

  13. Anonymous Coward
    Anonymous Coward

    Upgrades are available

    Just over a year ago, HPE acquired Nimble Storage. I see that as a very viable upgrade path for 3PAR users who desire US, UK and APAC (from Japan) based support.

  14. Anonymous Coward
    Anonymous Coward

    Is this the report in question?

    https://www.ato.gov.au/uploadedFiles/Content/CR/downloads/js39322_ATO-systems-report_w.pdf

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like