back to article When Google's robots give your business the death sentence – who you gonna call?

A sysadmin given just three days to respond to the threatened deletion of a mission-critical system has prompted a vigorous debate about the quality of cloud support. Writing on Medium, the user describes how the Google Cloud-hosted asset monitoring system for his firm's wind and solar energy plants was suspended on the …

  1. }{amis}{
    Unhappy

    Sadly Not Really New

    Google is a long way from the first firm to wreak people/companies based on an algorithm (Anyone applied for a mortgage recently?) personally I find it even more infuriating when you have got to a human to talk to but all they can repeat is "Computer Says No".

    1. Ian Michael Gumby
      Boffin

      @Hamish Re: Sadly Not Really New

      Did you pay attention to the article?

      "This guy is running important multimillion dollar production on a consumer plan. This is on him, not Google," observed one Reddit member.

      While I am not a fan of Google (Do know evil to do no evil) , this is on the guy's company and not Google because they were most likely running on the wrong terms (SLAs) because it was cheaper.

      1. Anonymous Coward
        Devil

        Re: @Hamish Sadly Not Really New

        Surely it would be "Do no evil" to "Do know evil", amirite??

        1. Nolveys

          Re: @Hamish Sadly Not Really New

          Surely it would be "Do no evil" to "Do know evil", amirite??

          See, hear and speak no evil.

      2. 9Rune5
        Pirate

        Re: @Hamish Sadly Not Really New

        wrong terms (SLAs)

        ...google has a cheap SLA that says "we will routinely nuke all your data, but at least we won't charge you an arm and a leg for providing this service"?

        Sounds a bit like those cryptovirus providers.

      3. R 11

        Re: @Hamish Sadly Not Really New

        While I am not a fan of Google (Do know evil to do no evil) , this is on the guy's company and not Google because they were most likely running on the wrong terms (SLAs) because it was cheaper.

        It goes beyond that. Saving money by using a cheaper SLA might be fine if you can cope with the downtime. This guy apparently opted for an SLA that didn't meet their needs and didn't keep backups.

        Why anyone would throw all their eggs in one virtual cloud is beyond me. With decent backups and a thought through process, they should be able to move vendor at the drop of a hat, allowing them to take advantage of better pricing or service from a competitor cloud.

      4. Anonymous Coward
        Anonymous Coward

        Re: @Hamish Sadly Not Really New

        >this is on the guy's company and not Google because they were most likely running on the wrong terms (SLAs) because it was cheaper

        Ah so we have a SLA fetishist here. So please tell me, why is it then acceptable for Google to sink his project because a faulty algorithm got itchy?

        1. Rob D.
          Thumb Down

          Re: @Hamish Sadly Not Really New

          By SLA fetishist, this means someone who understands the impact of legal contracts?

          "I would like to pay the minimum for third party, fire and theft car insurance for a full year, but if during the year I have an at-fault accident, I'd like it to be handled just as if I'd paid full price for the fully comprehensive insurance."

          In the actual case, the affected company might not have done their due diligence or might not have understood the difference between 'third party, fire and theft' and 'fully comprehensive' (or even that the different options existed), but ignorance is no defence. The resolution (systems back on without lasting damage to the business) embodies a harsh, proportionate and relevant lesson in why shifting to cloud provision really isn't a case of abdicating all responsibility for business continuity.

          The way the commercial world handles this is through contracts (good and bad ones) with measurable SLAs to define or challenge compliance. No fetish required.

  2. The Man Who Fell To Earth Silver badge

    If its mission critical

    Don't entrust it to someone's cloud.

    And always have a complete local backup in your company's physical possession.

    1. Rich 11

      Re: If its mission critical

      If it's anything but throwaway, don't entrust it to someone's cloud.

      Naturally the beancounters will disagree. It's the cloud, they will gush: what could possibly go wrong?

      1. Mark 85

        Re: If its mission critical

        Naturally the beancounters will disagree. It's the cloud, they will gush: what could possibly go wrong?

        Forward articles like this one to the beancounters is what I'd do first. The more articles, the merrier. Add to it the problems the server farms (Google and AWS notably) have with security, etc. Maybe at some point they'll realize that keeping mission critical data on other peoples computers is never a good idea. But, given that they're beancounters, they put fingers in their ears, yell "La-La-La. I can't hear you" and when things go belly up, guess who takes the blame.... not them.

        1. Anonymous Coward
          Anonymous Coward

          Re: If its mission critical

          Add to it the problems the server farms (Google and AWS notably) have with security, etc.

          Really?

        2. vagabondo

          Re: If its mission critical

          @Mark 85: “Forward articles like this one to the beancounters”

          Beancounters are preprogrammed to ignore articles like this and all “technical” advice. They only take notice of other beancounters opinions. That's why, as a profession, they rush into bad decisions like proverbial lemmings.

          1. Anonymous Coward
            Anonymous Coward

            Re: If its mission critical

            "Beancounters are preprogrammed to ignore articles like this and all “technical” advice. They only take notice of other beancounters opinions."

            All too true.

            At one point I worked for a major bank (1,600+ branches) doing hardware evaluation and selection. I would look at the alternatives, evaluate the technical requirements and solutions, vet the contracts, negotiate the prices, and report my recommendations to the manager of the hardware section in the Systems Division (about 3,000 employees). He would hand it over to the Director of Technical Services (responsible for the hardware, data centres, operating systems, and system software) who would take the report to the system division financial director (an accountant with no strong technical understanding) who would then take it to the Senior Vice President (Systems) who was also an accountant and would only listen to another accountant.

            It was a big mainframe shop, with more than a dozen IBM engineers working full time in each of our data centres maintaining and installing the hardware... with the decisions based on second hand information filtered through an accountant's understanding of the technical issues, conveyed to another accountant.

          2. Anonymous Coward
            Anonymous Coward

            Re: If its mission critical

            >Beancounters are preprogrammed to ignore articles like this and all “technical” advice. They only take notice of other beancounters opinions. That's why, as a profession, they rush into bad decisions like proverbial lemmings.

            Tragically this is 100 percent correct. And the text book example here (and I do mean textbooks in organisational theory) is the Challenger disaster. The techs said do not launch. The beancounters said launch.

            And 7 astronauts were killed. Billions in government properties were obliterated. And the US space programme was lying belly up for a decade. So what happened? Well ... not much. Importantly, none of the guilty were punished and will never see the inside of a cell, far less a court room.

            And that is how the world works.

      2. Someone Else Silver badge

        Re: If its mission critical

        It's the cloud, they will gush: what could possibly go wrong...<click>go wrong...<click>go wrong...<click>go wrong......?

        Ob. Westworld reference.

        The original one, not the current made-for-cable-TV thing

    2. steviebuk Silver badge

      Re: If its mission critical

      True unless you have a director who keeps spouting the bullshit that is "Infrastructure Free".

      A phase that doesn't even fucking exist yet he believes in.

    3. Claptrap314 Silver badge

      Re: If its mission critical

      But why do you trust disk drives that someone else has built?

      If it's mission critical, then you must do a thorough rundown of what can go wrong, and ensure that each of them are properly handled. (Generally in pairs.)

      Let's start with an electrical outage. If your in-house servers are all in the same city, then you do NOT have resiliency against electrical outages, and probably not against cable cuts.

      How many ISPs are you connecting through? If the answer is one, again, you do NOT have resiliency.

      I'm not really in the mood to repeat (from last week) the list of things that can take you down that I learned while an SRE at Google.

      For smaller operations, cloud can get you levels of reliability at far, far lower prices than DIY. It does require that you know what you are doing. And yes, if G is being stupid about the way that they are handling billing problems, then stay away.

      1. croc

        Re: If its mission critical

        A group of scientists built a box with four ways out, differing complexity, to test the intelligence of gorillas. The first gorilla promptly bashed his way through the box, using none of the designed ways.

        As to the whole 'cloud' bit... If you don't own the infrastructure that you make money from, then you deserve what you get. Just ask Mr. Dotcom's clients.

        1. Anonymous Coward
          Anonymous Coward

          Re: If its mission critical

          The number of comments on here that completely ignore they scale and resiliency of cloud services beggars belief. All of you commenting that you can do better in house must work for companies with data centres on at least two continents or you don't understand what you're talking about.

          I used to work for a ten man company. There was no way we could get infrastructure in place to match what was available in other people's clouds. To suggest in house would have worked for that global mobile business is ridiculous. Yet it is possible for ten people to run a reliable global service if they use the cloud.

          It seems most of the animosity towards cloud services should more accurately be targeted at the evil bean counters ignoring your blinkered advice.

          1. Anonymous Coward
            Anonymous Coward

            Re: If its mission critical

            I work for a company with data centres on four continents and we can't do better than public cloud in terms of providing cost effective, modern and resilient infrastructure or making it easily available to development and operations. We keep them for regulatory and historic reasons (cheaper to maintain than to replace with cloud native). For every £1M we can spend, AWS can spend £1B. None of the public cloud providers care about our business, that's our job and none will tailor their SLAs or offerings to suit our business, again that's on us. There is a role for in-house IT but it's not convincing management that we're better than the public cloud providers.

          2. DropBear

            Re: If its mission critical

            "Yet it is possible for ten people to run a reliable global service if they use the cloud."

            Actually, my experience - such as it is - with satellite phone providers so far is frightfully reminiscent of the Google "experience" described in the article.

          3. Anonymous Coward
            Anonymous Coward

            Re: If its mission critical

            All of you commenting that you can do better in house must work for companies with data centres on at least two continents or you don't understand what you're talking about.

            Some people are "working" for state or EU outfits and they have freshly printed money oozing out of every pore all day like the meat sweats -- from 10:00 to 16:00 and the 2 hour lunchtime.

        2. onefang

          Re: If its mission critical

          "If you don't own the infrastructure that you make money from, then you deserve what you get."

          Or as an old boss of mine used to put it - always own the means of your own production.

      2. onefang

        Re: If its mission critical

        "But why do you trust disk drives that someone else has built?"

        You don't, which is why things like RAID and backups exist. One thing you can trust is that if a hard drive fails, the other hard drives from the same manufacturing batch are likely to fail soon to.

        One thing I can trust is that any type of flash storage will fail sooner rather than later in my home, but that's coz the use cases I have for them are rather abusive of their limitations. Meh, that ones on me.

        1. Claptrap314 Silver badge

          Re: If its mission critical

          That is rather my point. You don't have to build the hard drive yourself. You don't have to understand the REALLY crazy stuff the bios is doing. What you need to understand is the MTBF & failure recovery mode for the service provided by this foreign piece of equipment that you have brought into your data center and trustingly plugged into your network. (And RAIDs themselves do fail. That's why the big boys don't use them, by the way.)

          Same thing with cloud services. If you know how to use them, they provide facilities for resilience that would bankrupt an SMB to match. If you don't, well...

      3. Anonymous Coward
        Anonymous Coward

        Re: If its mission critical

        >But why do you trust disk drives that someone else has built?

        Who says we are?? It is standard practice to pack the RAID with disks from different batches, preferably different manufacturers.

        >If it's mission critical, then you must do a thorough rundown of what can go wrong, and ensure that each of them are properly handled. (Generally in pairs.)

        Being nuked by Google is not part of that list, well, at least not up to now, seeing as an ex-Google emplyee defends this practice. I will keep this in mind.

        >Let's start with an electrical outage. If your in-house servers are all in the same city, then you do NOT have resiliency against electrical outages, and probably not against cable cuts.

        Assumptions, assumptions. Anyways, we had large UPS backed by diesel powered generators with real big fuel tanks.

        >How many ISPs are you connecting through? If the answer is one, again, you do NOT have resiliency.

        The norm is at least two.

        >I'm not really in the mood to repeat (from last week) the list of things that can take you down that I learned while an SRE at Google.

        You are instead quite happy to use your status as former Google employee to take a leak on this guy. Nice.

        >For smaller operations, cloud can get you levels of reliability at far, far lower prices than DIY. It does require that you know what you are doing.

        Cloud or no cloud, that is ALWAYS the case.

        >And yes, if G is being stupid about the way that they are handling billing problems, then stay away.

        And THAT is the takeaway from this.

    4. macjules

      Re: If its mission critical

      Totally agree. If something is that mission critical then you should being a Platform as a Service provider cloud and not just one single 'supplier', such as Google or AWS. A good PaaS provider is spread over all platforms, such as Azure, AWS, Google or even the mobile providers.

    5. Ken 16 Silver badge
      Paris Hilton

      Let me rephrase

      If it's mission critical, don't entrust it to just one cloud.

      And have full backups under your companies control.

      I haven't seen a physical backup since the last century and I do this for a living.

  3. Sorry that handle is already taken. Silver badge

    "This guy is running important multimillion dollar production on a consumer plan. This is on him, not Google," observed one Reddit member.

    Well, sure, but Google arbitrarily threatening to delete all his things isn't...

    1. Warm Braw

      Google arbitrarily...

      That seems to be the real problem with Google - not just in this instance but in the way they withdraw or change incompatiblly products and services that people are actively using.

      It's a significant disincentive to use any technology that's Google-exclusive because it's difficult to have confidence it will be there tomorrow,

      1. Pascal Monett Silver badge

        Personally, I find it difficult to believe that it is a good idea to put company data in a beta product, and Google is always in beta.

        I subscribe to the people who think that company data needs to be placed in the control of the company, not on someone else's server.

      2. Someone Else Silver badge

        @Warm Braw -- Re: Google arbitrarily...

        It's a significant disincentive to use any technology that's Google-exclusive because it's difficult to have confidence it will be there tomorrow,

        Funny, I remember a time when the exact same thing was said of another tech monopolist-wanabee....

    2. bazza Silver badge

      Well, sure, but Google arbitrarily threatening to delete all his things isn't...

      Indeed. And furthermore, why should a private consumer face the deletion of their entire stuff simply because they fail to respond within three days. Holidays? Hospital stays?

      If Google can't be arsed keep a private individual's stuff up and running for a reasonable period of time after their crappo-bots crap out, why should an enterprise user paying enterprise money trust them at all? The hypothetical conversation goes,

      E: "Where's my enterprise stuff?"

      Google: "Oh sorry, one of our bots has autodeleted the lot"

      E: "Well gee, thanks a lot"

      1. shedied

        You're back?

        E: And you were, of course, going to inform us? When?

        G: Our automated system sent you the notification. It will be in your Inbox. Er, how long have you been away? Three days?

        E: I was just heading back to work.

        G: Begging your pardon, but my phone is ringing; you should see the email in your Inbox then. Have a nice day.

        1. eldakka

          Re: You're back?

          > G: Begging your pardon, but my phone is ringing; you should see the email in your Inbox then. Have a nice day.

          E. But my email server is hosted on the same Cloud platform that you are emailing me about...

    3. JLV

      presumably he’s also spending a fair bit of money there, so the lack of any dialog - not to ask Google how to use their APIs, but to not get deleted which is a different thing entirely, is worrying

      basically it’s “don’t use our base plans for anything serious”. so who does that leave that version of the Google cloud for? consumer backups? hobby projects? that’s a pretty limiting use case and they should be more upfront about it.

      1. Dagriffi58@gmail.com

        I would not be happy about my hobby project being deleted. It may be of no value to anybody else, but it mine and I love it.

  4. Anonymous Coward
    Anonymous Coward

    "they can engineer products that don't need humans behind them"

    Unless you want to spend money on their advertising platform!

    At my previous place, we had a multi-million pound credit account and the Adwords team on speed dial... but for anything else, it was a case of talk

    - Start a topic on Google Groups (And pray somebody from Google eventually replies)

    - Delete it and start again

    - Move elsewhere

  5. big_D Silver badge

    Don't talk to us...

    Google's attitude is, don't talk to us.

    My previous employer was on the receiving end of a DOS attack last year. We did a quick check of the IP address initiating the attack (it was pushing >100mbps down a 10mbps line). The IP address belonged to Google.

    Call Google... After 10 minutes of bouncing back and forth in their automated telephone system, the answer is: look at the relevant page on our website; only I couldn't find a page about Google hosting DOS attacks.

    I then tried email, abuse@google.com and admin and webmaster... All returned a form-email saying that they get so many emails, they just don't bother reading them and I should refer to the relevant part of the website for support... Again, no part of the website covers being DOSed by Google.

    I then tried Twitter, but no response form Google there, either.

    In the meantime, I had contacted our ISP and they managed to put in a perimeter block on the IP address causing the attack (this is how I know that they were pumping over 100mbps at out 10mbps line). But that only ran for 7 days, after that, we had to pay for continued use of their IP blocking service.

    Luckily, we were in the middle of switching to a new 100mbps line with another ISP, so we just accelerated the tests and switched over, leaving the Google server to continue to wallop a disconnected line.

    It was probably a badly configured server in the Google farm or one of their cloud clients with a badly configured instance... Whatever it was, contacting Google was a nightmare.

  6. tiggity Silver badge

    More info?

    The SLAs link was not v. informative - did not explain differences on consumer v enterprise & did not mention chance of all your content destroyed.

    Medium link gave nothing useful (I'm not on medium & was not going yto join - and it was pulling in js from lots of different sites, possibly some blocked by my blaclists)

    Having no clue about the different Google plans (use different cloud provider), but for many companies, with (relatively compared to huge usage of some sites) little usage, bog standard cloud plans do the job in terms of bandwidth, data processing / computational / storage units (in whatever arbitrary units / definitions the provider uses).

    Normally only use "enterprise" levels if you have heavy usage (I only use a single "service" at enterprise level as it gets a lot of use, exceeding the low level account usage limits), as (with non Google cloud I have used) none of the levels have a "your data will be destroyed at short notice at our whim" clauses!

    1. Frank Bitterlich

      Re: More info?

      "Normally only use "enterprise" levels if you have heavy usage..."

      ... or if your million-dollar business critically depends on the service. Or if you don't think it's necessary to have local backups of said million-dollar business' data and - even more weird - code.

      OK, killing your service and deleting your stuff with just three days of warning is rude. But I don't get it how you can bet your house on a free service that is clearly not meant for such use.

      People just don't get business continuity any more.

      1. Doctor Syntax Silver badge

        Re: More info?

        "But I don't get it how you can bet your house on a free service that is clearly not meant for such use."

        Given that there was a demand for a credit card it doesn't seem likely that a free service was used. In fact, IIRC, the quote in the article about AWS handling billing issues better came from the complainant and not a commenter, again pointing to the fact that it was a paid-for service.

        1. MonkeyCee

          Re: More info?

          " it was a paid-for service."

          But a paid for service doesn't automatically mean that it's suitable for a business.

          A home internet connection isn't suitable for hosting a commercial site. A network connection that is might cost ten times as much for similar speeds, albeit with better uptime, minimum speeds and service support.

          I have several pieces of software that I pay a nominal academic fee (less than 1% of the commercial licencing fee) that are very clear that if used in anger the owners would like their 12 grand a year, thank you very much.

          If an ongoing expense is critical to the running of a business, it shouldn't be being paid for by a credit card. A formal contract, billing and bank payments should be set up, and internally documented so people know why paying a certain bill is Very Important.

          For a dev environment where you don't know what sort of resources you'll want to spin up, sure. But prod should be stable and resilient in both systems and business processes. That means paying the power and computer bills on time :)

          Since I'm being long winded anyway, you should always have a local copy of your cloudy stuff anyway. How else will you replicate it when someone else's computer breaks :D

          1. Anonymous Coward
            Anonymous Coward

            Re: More info?

            "If an ongoing expense is critical to the running of a business, it shouldn't be being paid for by a credit card."

            We have a network with about 5,000 workstations, the bulk of them using a wide variety of both internal and external services. We've got hundreds of servers providing various externally facing services and more providing internal IT functions. Sixty or seventy domain names, dozens of external vendors we use for any number of specialized tasks. Scores of user departments with projects, etc. Hundreds of thousands of end users on our workstations or their laptops, locally, and more coming from outside.

            A lot of what we do is big stuff, done by formal RFQ or RFP and routed via the purchasing department for process administration... but that takes time. There are maybe a hundred people in IT across all functions, and much of that is user facing web stuff, so the OS, DB, security and network functions are handled by a tenth that number. There are a surprising number of relatively tiny things needed to deal with Department Y's urgent project that they didn't mention until three weeks before the public unveiling, and we simply cannot afford the overhead in elapsed time and staff time to route things via purchasing, nor do the external vendors handle invoices as fast as a credit card read over the phone... which is why all managers in the IT department have corporate credit cards. It's the only method that works for some things.

      2. Anonymous Coward
        Anonymous Coward

        Re: More info?

        Business Continuity is about ensuring your essential functions continue even when one of your providers stops providing, whether that's Google deleting your systems or the National Grid cutting your power supply > You have alternatives in place and you have confidence they'll take up the slack, it's not just paying them for a higher service level. I did work somewhere who charged more for a higher SLA and used the extra money to pay service credits for failing to meet it - they still came out ahead.

  7. Blockchain commentard

    3 days? What if Google decided at 6 pm on a Friday before a bank holiday weekend? Come into work on Tuesday and find Google have deleted everything. Nice.

  8. Craigie

    Google 'support'

    From a consumer point of view the absolutely worst thing about Google is the lack of support for their products. That this issue extends to their cloud offering does not surprise me.

    1. DougMac

      Re: Google 'support'

      No MSP will trust Google Apps with their customer's email setup. Support and response to problems is just too messed up. One reason Office365 took off so much.

    2. TReko

      Re: Google 'support'

      We have found the same, even with a higher level service agreement. Any queries get automated replies, or a human cutting and pasting a reply based on keywords in our query. It is very hard to find an educated human to talk to at Google, which I think is by design. Humans cost money.

      Still Google's Office suite is what we must use, but we make a continuous local backup using Syncdocs which has saved our bacon a few times when we had problems.

    3. deadlockvictim

      Google's Product

      Google's product, that is, what they sell and make serious money on, is information about you and me. They take this very seriously and make sure that the advertisers get the best info money can buy.

      All other things built by them are merely means to acquire more information about you and me.

  9. This post has been deleted by its author

  10. Flywheel
    Thumb Down

    El Reg asked Google to comment

    Meanwhile, please consult the appropriate web page.

    1. Gordan

      Re: El Reg asked Google to comment

      I was just about to say something similar. Good luck getting through to a human.

  11. Christian Berger

    I'd like to point out that that application....

    ... doesn't sound like it would generate a _lot_ of data. I mean it's "hundreds of wind turbines and scores of solar plants scattered across 8 countries". Let's assume it's 10000 "devices" they monitor, each one giving a vector of 10 readings every minute. That's just 6 Million values per hour or roughly about a Gigabyte of data per day.

    That's something even your "run of the mill" SQL-database should be to store and process... on a single system. Using specialiced databases (this is not a relational problem after all) you can probably even handle a lot more on a simple modest system.

    1. sabroni Silver badge

      Re: I'd like to point out that that application....

      Looks to me like you have no idea what the requirements of this system are but assume it's been done wrong.

      Do you spend a lot of time on stack overflow answering queries with "why do you want to do that?"

  12. Headley_Grange Silver badge

    But...?

    OK - cloud service are not my area of expertise, so downvote at your leisure but....

    If a company is buying goods or services which are critical to the operation then surely it reads the contract and makes sure its fit for purpose at all levels; price, spec, maintenance, support, edge cases, etc. I bet Google didn't do anything that wasn't in the contract.

    As an example - we once had to get dual, diverse data lines from BT. They were safety related and it was important that they didn't come together anywhere other than at each end and therefore be as immune as reasonably possible to the man-with-a-digger single point failure.

    We gave BT a spec. and we got a proposal and contract from BT. Engineering read it, the lawyers read it, the safety consultant read and we all agreed that it was OK. Then, to make sure, we had a meeting with BT to be absolutely sure they understood what this meant. They hadn't. They did after we explained it and they upped the price a bit - but we were happy.

    I'm guessing that one of the problems with some cloud services is that the contract is pretty one-sided and not for negotiation, but in that case - if it doesn't meet *all* the requirements - you walk away and present the boss with a cost/benefit comparison of cloud vs self hosting, don't you?

    1. Doctor Syntax Silver badge

      Re: But...?

      " you walk away and present the boss with a cost/benefit comparison of cloud vs self hosting, don't you?"

      And probably get ignored by the beancounters. Although in this particular case if it had been a beancounter's decision getting the CFO out of bed to deal with it on his credit card would have been well justified.

    2. Jellied Eel Silver badge

      Re: But...?

      As an example - we once had to get dual, diverse data lines from BT. They were safety related and it was important that they didn't come together anywhere other than at each end and therefore be as immune as reasonably possible to the man-with-a-digger single point failure.

      Ah, that can be a case of understanding the product. Which for BT can be confusing. So you wanted Assured Seperacy. If you'd asked your account mangler for that and they went 'huh'? then get them to find one that knows it. Cos then it's meant to be a minimum 3m(?) seperation along the route for each circuit. Which usually works, but won't necessarily stop a digger cutting a 4m+ trench before noticing. Trick is to get to speak to BT's planners/surveyors and check options like entry opposite side of the building... Which will then lead to 'Excess(ive) Construction Charges', but should get dun right.

      Then as you say, it's down to the contract. And if you've paid for seperacy, you've got some comeback, although naturally compensation won't include consequential losses. Otherwise, like in this example, it's very much caveat emptor. OK, so Google was rather customer unfriendly, but IT types should know that.

    3. Anonymous Coward
      Anonymous Coward

      well done

      I've had meetings with BT's network sales team where they hadn't understood what we wanted or read what we'd specified and ones where they upped the price but none where they eventually got it after it had been patiently explained to them.

      1. SImon Hobson Bronze badge

        Re: well done

        I've had meetings with BT's ...

        I've had conversations with BT people adamant that the building we wanted lines installed into did not have any means of installing them. In the end I had to drive to site (again), climb up high enough to read the label on the JB on the back wall of the building, and then call back the local engineering manager to tell him that the building is served by DP nnn on the back wall of the building in xxx street ! Just telling him over the phone exactly where the DP was did not work - apparently it was too complicated for them to either look in their records, or have someone drop by (less than 1/2 mile from the exchange) and take a look.

        Oh yes, and that's after BT foooked up the installs by sending all the paperwork (including the notification of the install dates) to empty shops and then had the OpenRetch engineers turn up, find no-one there, and just cancel the job without telling anyone - after being given explicit instructions what address to use for paperwork. Oh that was fun, we had site hooked up with 3G modems at one point - great for a new business (our customer) on the high street (not) !

  13. iron Silver badge

    "destruction of the businesses code and data"

    "We would have lost everything – years of work"

    So the only copy of the code was on the live servers in Google's cloud? I find that hard to believe but if it isn't just hyperbole then they deserved to have their code & data deleted to teach them about backups.

  14. A Non e-mouse Silver badge

    My significant other works in a small retail shop. They use a web/cloud based till system. If there's a problem, you have to log a support ticket and wait three days for a response - even if your business can't function. No way to flag an issue as urgent, no telephone option, no way to pay extra for better support. You just have to wait.

    1. Anonymous Coward
      Anonymous Coward

      If there's a problem, you have to log a support ticket and wait three days for a response

      I work in a 7-day-a-week tourist / academic business, several sites spread over a large geographic area. We have our own IT department who only work 5 days a week and will not carry pagers or a department mobile for "complicated, historic resons". So if the shop tills go down on Saturday morning, or there's a WAN problem meaning no-one can log on and do work, it could be Monday before they are sorted. Doesn't happen -very- often, but when it does, it's annoying.

      1. Throatwarbler Mangrove Silver badge
        Holmes

        "complicated, historic resons" (sic)

        As in, the company won't compensate the IT staff for being on call.

        1. A Non e-mouse Silver badge

          @ Throatwarbler Mangrove

          Exactly. At all the places I've worked, I've offered to be on call IF they pay me for being on call and there's a sane rota (I have no desire to be on call 24 hours a day 365 days a year) But none of them wanted to pay, so when things break at the weekend, they stayed broken. The people on the front line of the business complained like hell, but they understood the situation.

        2. Anonymous Coward
          Anonymous Coward

          As in, the company won't compensate the IT staff for being on call

          IOW it's a business decision. Cost of implementing against business cost of downtime. FWIW (original ac here) there is a range of pay scales. Up recently only the 3 lower scales had out-of-hours, overtime, weekend supplement, bank holiday supplement. IT staff are all (AFAIIA) on scale 4+ so have *never* been eligible. At previous jobs I always had a "this is your pay, you are on call [to schedule] and there's no call-out pay, just take TOIL". It would have been easy to put a similar clause in IT contracts a decade or more ago, especially since callouts would be (I believe) rare. (note I am not in the IT dept)

          Now of course no-one gets any extras, except some get bank hol payments. Overtime is "additional hours" and not for scale 4+. Even better, pay scale is still based on 37.5hours but all new jobs are 35 hours. Adverts give the 37.5 amount but say "pro rata".

    2. John Savard

      I suggest they replace it with good old-fashioned cash registers. Or at least have those as a backup.

      1. Jamie Jones Silver badge

        John! Next you'll be saying we don't need our toasters and light bulbs connected to the internet!

        Get with the times!

  15. Iggle Piggle

    I was working for a company using Azure. We were developing on a trial account that required a credit card. When we ran out of credit the accounts would be locked out but there was no question of data being deleted (at least not in the short term) once the credit card holder was located he would enter the details and we'd be off again. It sounds really dodgy if they threaten to delete everything after 3 days, I use Google for my storage, I might need to read some fine print.

  16. Doctor Syntax Silver badge

    We have to keep saying, cloud's not really anything special, it's just somebody else's computer you can't control. But if it gets sold to CxO* types as a panacea there's little chance of getting this through to them except by experience.

    * In absence of evidence to the contrary assume O stands for Orifice and you won't go too far wrong.

    1. sabroni Silver badge

      re: it's just somebody else's computer you can't control

      So to your mind my pc is the cloud? That's really all there is to it? The distribution across contintents isn't part of it? The ability to rapidly scale up and down isn't a thing?

      I get your point, when I use the cloud it's not my computer and I don't control it, but don't pretend the cloud doesn't have any benefits to go along with the bad bits.

      1. Martin an gof Silver badge

        Re: re: it's just somebody else's computer you can't control

        don't pretend the cloud doesn't have any benefits to go along with the bad bits.

        I think the point being made is that there really, really isn't a one-size-fits-all offering. Every use case in every organisation is different.

        We know very little about this particular case, but the valid point is probably that while Google's offering was undoubtedly useful while getting the thing set up (little capital outlay, ability to scale up - or down - as required), by the time it became "mission critical" there should have been a more robust system in place, even if that merely involved stepping up a tier (or two) on the support ladder.

        The problem is that sales droids are extremely good at convincing PHBs that "the cloud" is the answer to everything, without explaining that moving to a cloud-based solution, particularly where an existing on-site solution has been in place for many years, involves looking at every aspect of the operation from the roots upwards and re-evaluating which bits are "worth" what to the company.

        All a PHB sees is "it's working" and "it's cheaper". Once those two things are established, best of luck.

        On a much smaller scale, at home I have a NAS(*) which stores all our digital media. It's based on an E450 AMD processor and six 1TB 2.5" HDDs in Z2 (ZFS equivalent of RAID6) for approximately 4TB online.

        Occasionally a disc dies and is replaced - no problem.

        However, I now need to expand the array and I can't do the trick of swapping out the 1TB discs for (say) 1.5TB or 2TB discs because NAS-capable 2.5" discs simply don't come in sizes greater than 1TB(**). What's that all about? I blame SSDs. I wanted to build a small, low-power, quiet(ish) NAS and I gambled that HDD capacities would continue to rise. Well 3.5" capacities have, but 2.5" HDDs have been stuck at 1TB (except for WD Blue etc.) for about five years now.

        So I need to rebuild the whole caboodle, either to support more than the 6 x 2.5" discs I currently have (i.e. a new motherboard with more SATA and/or more PCIe) or a new (bigger) case which can take 6 or more 3.5" discs. To be honest, a new m/b is a good option anyway as the existing one is well over 5 years old and I recently lost a similar E350.

        In my case, the PHB is my wife, and the constraint is the bank balance. "It's working well, so why do you want to spend all that money building a new one?"

        Now then, what are the current offers on HP microservers?

        M.

        (*)Actually there are two, physically separated, in theory replicating each other. For various reasons this isn't quite working the way I need it to at the moment...

        (**)For home I'd be happy using WD Black or Barracuda Pro rather than a WD Red or Ironwolf, but even they don't come larger than 1TB.

    2. 0laf
      Facepalm

      Bingo. Have yourself a pint.

      Cloud is sold as a single product panacea that is bigger, faster, easier, cheaper and shinier.

      Only it's not.

    3. onefang

      "* In absence of evidence to the contrary assume O stands for Orifice and you won't go too far wrong."

      You can probably assume the C stands for another word for a certain type of orifice.

  17. Anonymous Coward
    Anonymous Coward

    Did they not check support available before embarking on their eggs in one basket cloud escapade?

  18. ibmalone

    So, of course, keep backups that you have control over, and if you can justify it and afford it have redundant architecture that you can switch to. But what exactly is a "consumer plan" in this context? There's no mention of it on the GCP pricing page, https://cloud.google.com/pricing/list there are support options, but those seem to be for technical issues, while billing support is listed as free.

    The advantages of this kind of platform are meant to be scalability and that you are paying for enterprise level availability, so at what level of utilisation do you count as an enterprise user?

    1. Mark 85

      Let's add, "don't keep the backups in the file drawer at Bob's desk". I've heard that bad things happen if you piss off Bob.

  19. Steve Davies 3 Silver badge
    Big Brother

    Computer Says No

    And

    All your business now belongs to us.

    The shape of things to come

    Big Brother is Watching you. (even inside your cloud instance!)

    I hope this is a warning to those who have gone all in with the cloud. All it takes is for a Credit Card to expire and your business is toast.

  20. register_ftw

    Well... naturally

    "El Reg asked Google to comment but had not received a response at publication time."

    Well....duh :-) That's the whole point

  21. Anonymous Coward
    Anonymous Coward

    Wait until the choice is Amzn Cloud, Goog Cloud, MS Cloud or other Cloud and things become political

    Expect POTUS whatever side they're on, to issue take-down orders.

  22. Peter Mylward

    Against the tide

    While I’m no fanboy, experience of Alphabet brethren YouTube involved me speaking to a Mountain View based humanoid within 5 mins of hitting support site (via free callback to UK) - I was very impressed to say the least.

    1. sabroni Silver badge

      Re: experience of Alphabet brethren YouTube involved me speaking to a Mountain View based humanoid

      And they understood you?

  23. Anonymous Coward
    Anonymous Coward

    Capricious Gods

    Many of these new(ish) tech companies act like capricious gods. You can be struck down at random and never find a human or even a robot to sort things out.

    On a smaller scale, there are plenty of forums with discussions where longtime users of Blogger, eBay sellers, Amazon affiliates/marketplace sellers, Google Adsense etc have been barred without notice, any recourse and no-one to take their case up with.

    The lesson is - do not rely on any of services as a source of income.

    1. bazza Silver badge

      Re: Capricious Gods

      I'm sure the reason they act this way is to prevent themselves turning into clones of the old IT kings like, for example, IBM*. If they're to provide a decent level of service with the necessary human cover to adequately and efficiently sort out cock-ups on the rare occasions when they ruin your day, then that's going to cost a whole heap of money.

      I strongly suspect that Google's business model is focused on get as many customers as possible, and if the gullible ones get stung over the poor customer service who gives a shit?

      * I've no experience of IBM themselves, I just imagine that if they're charging as much as I suspect they do, I'd damn well want to be able to speak to someone almost anytime I want and get keyboards rattling sharpish.

      1. Anonymous Coward
        Coffee/keyboard

        Re: Capricious Gods

        I've no experience of IBM themselves, I just imagine that if they're charging as much as I suspect they do, I'd damn well want to be able to speak to someone almost anytime I want and get keyboards rattling sharpish.>

        Bwahhahhahahahahahahaha! Bwahhaaahhaahahaha!

        Somebody get that poor chap a glass of reality, please?

  24. martinusher Silver badge

    Yet another example of social entropy.

    We live in an age of technological marvels -- everything is super sophisticated, ultra-capable, feature rich &tc, &tc,

    Too bad that in our rush to get better, faster, cheaper and more of everything we seem to have lost sight of 'works'. Like the death of a thousand cuts you can't really cite one single instance of things falling apart but if you look at the big picture -- VoIP phones that are appallingly unreliable compared to the old-fashioned wired phones, cellphones that don't work everywhere, applications that hog inordinate amounts of memory and processor resources and are often reliant on low latency local storage or network responses to work effectively -- the list is endless. This cloud debacle is just another example; it replaces local storage -- a single point of failure -- with a system with numerous points of failure....and there's nobody around to fix it or even monitor it for problems.

    It wouldn't hurt to pull back a bit, tell Marketing to take a holiday (or hike) and just focus on getting things to work 100%. Users don't need the latest and greatest, a raft load of new features which are guaranteed to improve productivity (and will probably never use), first and foremost they need the stuff to work. Predictably, reliably, through both natural and man made disasters.

    1. Boris the Cockroach Silver badge

      Re: Yet another example of social entropy.

      But how do I get the suckers.. err customers to buy the latest shiney shiney crap?

      But we should thank the late DNA for this quote

      "Customers who overcame the superficial designs flaws were so happy with the product that they never noticed the fundamental design flaws with the product"

  25. Anonymous Coward
    Anonymous Coward

    El Reg asked Google to comment but had not received a response at publication time.

    Did you get referred to a bot?

    1. onefang
      Coat

      Re: El Reg asked Google to comment but had not received a response at publication time.

      "Did you get referred to a bot?"

      Yes, and the computer said no.

      I started reading these comments thinking I should make that joke, but a couple of others did already. However, I couldn't resist this time.

      I'll get my coat, it's the one with the pocket super computer screaming "NOOOOOooo...!" in the new Australian female accent Google recently released, that sounds more yankie than Aussie.

  26. Anonymous Coward
    Anonymous Coward

    El Reg asked Google to comment

    this is puzzling, I thought their robo-spokes-AI would have fired a prompt and entirely useless respose in no time?

  27. Someone Else Silver badge

    A third commentator suggested it was a cultural issue at Google: "Customer support isn't and never has been in their DNA. It's often rage-inducing how hard it is to contact a human at Google. They seem to think they can engineer products that don't need humans behind them."

    El Reg asked Google to comment but had not received a response at publication time.

    Irony can be so...ironic, can't it?

  28. Astara

    and when someone working at google starts harassing you?

    Who do you call?... I got a DMCA take down for my own artwork on my google-plus account. I complained -- and the response I got was that they changed it to being taken down for violating google's TOS. At that point, I asked what the violate was...but was told that someone had reported my anime-based avatar as offensive and that they weren't allowed to give any more information -- i.e. I could fight the 1st as I had the production history -- but the 2nd you can't fight. I finally found # for google's legal dept (this was at least 3 years back) and someone there looked into it -- someone inside google had a grudge against me and had banned it both times from within. I was fortunate in that I had been able to contact someone who could look into who did it, and do something about it. They told me they found the problem and that it wouldn't happen again. So far, it hasn't. But their first-line user-support answers were mostly worthless.

  29. Winkypop Silver badge
    Meh

    "Google support lacks human judgement"

    And when you do talk to a human*, they lack the approval and process to use human judgement.

    * Admittedly this hasn't happen to me for a few years.

  30. Mr Dogshit

    If you lie down with dogs

    you get puppies.

  31. Paul 87

    Scenarios like this are why when you plan a business case using the cloud, you have to ask what's your backup plan, and your exit stratergy.

    1. anonymous boring coward Silver badge

      "what's your backup plan, and your exit stratergy"

      Sounds like using the cloud is a bit like a SEAL team extracting a hostage.

  32. Neon Teepee
    Coat

    Not Google related

    The grown ups know what and where their infrastructure is, they know how it works and they have it it on an appropriate support package, that goes double if its a multimillion dollar company. They also know how and where its all backed up and where the electronic version and paper printouts of the BCP plan are. We are going to see more and more of this as 'the cloud' gets more 'efficient' (i.e. cheaper but with crappier service). At some point you will discover you are paying more to protect yourself from your cheap cloud provider contract than you are saving by not doing it properly. Of course your beancounters won't see it that way until it all falls in a heap.

    Just keep the emails somewhere safe that's my advice :-)

    Right, I'll just fetch me coat

  33. anonymous boring coward Silver badge

    Relying on The Cloud.

    The Internet of Hackable Things.

    Artificial Stupidity making critical decisions.

    What could possibly go wrong?

  34. Anonymous Coward
    Anonymous Coward

    Awful user experience

    My own experiences with Google Cloud are limited to trying to set up an account. We've been recommended to use GSuite to manage the account. Setting up GSuite requires that you verify ownership of a domain, and all email contact to manage that account is sent to emails on that domain. Google seem to think you will be quite happy to shift all your email addresses to their management.

    In fact, they do provide ways to avoid this, but they are non obvious. We actually have someone from google helping us walk through the process, and after an elapsed time of several weeks, and about two hours of my time online, we still haven't actually set up an account the way we would like it to work (on our domain, and keeping our current arrangements for email).

    Compared with AWS or Azure, Google Cloud is a massive pain so far, and we haven't even tried to use it for anything yet. Google Cloud will be staying a distant third in the public vendor cloud space for the foreseeable future unless they pull themselves together.

    Posting as anonymous coward, because, well...

  35. MatsSvensson

    Lesson

    Never build anything on someone else infrastructure, if you don't have a watertight contract with them and contact-persons there that can be reached and has to answer to you.

    And as for building on top of anything from Google;

    Always check out the "support"-forums for it first.

    If there are page after page of panicked users whos stuff has broken after Google changed something without telling anyone. And zero helpful response from google. And then those panicked voices just fade out after days and weeks and months of no reply or fixes.

    R U N !

  36. steelpillow Silver badge
    Joke

    "El Reg asked Google to comment but had not received a response at publication time."

    When they tried again later, Google replied, "That's interesting, Fromtheregister. Why do you say that?"

    (Icon for Google QOS)

    1. Anonymous Coward
      Anonymous Coward

      Can you elaborate on that?

  37. DropBear

    "The other is that under certain circumstances Google support lacks human judgement"

    There is no such thing as Google "support". Google is the cloud, and it offers exactly as much "support" as one. Your jetpack is out of fuel? Awww, tough luck: down you go and I hope you like craters. The Wall in the North from GoT is dwarfed by the edifice Google puts between its employees and its "customers".

  38. Aodhhan

    Laws

    Right now there isn't a lot, if any regulation on cloud service providers; therefore, they have you by the short and curlies.

    I really hate an increase in government regulations, but CSPs who are currently not responsible for data as a whole (even with PaaS), really need to be reigned in. There is too much consumer information being placed on these systems for CSPs not to be more responsible.

    It's ridiculous when a CSP has you locked in and then they change the rules on you--and you're stuck.

    1. Headley_Grange Silver badge

      Re: Laws

      If a company signs a contract that allows the cloud provider to change the rules/terms without agreement on both sides and suitable consideration then they are bloody fools.

  39. This post has been deleted by its author

  40. Anonymous Coward
    Anonymous Coward

    I think this turned out to be fake news-

    Googles comments linked

    https://www.reddit.com/r/google/comments/8l231x/google_banned_an_entire_company_gsuite_accounts/

    1. Anonymous Coward
      Anonymous Coward

      I think this turned out to be fake...

      That is off topic. That event happened 1 month ago. This is 3 days ago. Also it's hard to verify it as fake when that op said "I'm also posting on a throwaway account for obvious reasons." to avoid getting fired.

      https://www.reddit.com/r/tifu/comments/8kvias/tifu_by_getting_google_to_ban_our_entire_company/

      Considering he shifted the blame to himself, you could still considered something went horribly wrong but the source is for sure unknown.

  41. boatsman

    In addition, we're keen to hear of any insurance policies

    have 2 clouds, not from the same outfit.

    double cost, double trouble. double work. just like it Always used to be.

    only the running a datacenter is what you loose, apart from a lot of cash... :-)

    cloud@home would do nicely, but that depends on who you are and what you need.

  42. Rob D.

    Own the responsibility

    Although few do, the real issue here is a bit of 'what-iffery' required before committing, and doing the due diligence (more formally it's failure mode analysis). Ideally revisit it regularly as well.

    What are the possible failures, how well do you understand them, what is the impact, and what is the mitigation? For example, do you know that all your systems are beholden to the good standing of a single credit card? Or, do you know whether you can actually call a support engineer competent to resolve critical problems?

    Or have you been told that is the case, have a legal document to CYA, and don't really care whether in reality, a year and a half from now, the contract turns out to not be worth the paper it was written on?

    Eventually contracts are needed to define what has been agreed, but buyer beware (or at least be well-informed and thorough).

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like