back to article IBM reminds staff not to break customers in pre-Xmas fix-this-now rush

Bosses at IBM's Australian outpost have been forced to remind staff to do their best work during the pre-Christmas rush – that time of year when outsourced clients want a lot of stuff done in a hurry before much of the world shuts down to overeat. Internal communications seen by The Register this week state that human error by …

  1. Anonymous Coward
    Anonymous Coward

    Someone not following best practices

    In my days most telcos had a 15 of December to 7th prohibition on any deployment or any config change. Any changes from around 15th of November onwards were harder and harder to do as they had to have a "fix before 15th" project plan attached to them to be even looked at.

    WTF is this "rush to fix?"

    1. Jim Mitchell

      Re: Someone not following best practices

      You said it yourself, the rush to fix is to get the fix in before the freeze date.

    2. Mark 85

      Re: Someone not following best practices

      End of calendar year probably. Seems everyone wants everything fixed for the new year's business. The holidays are seen as downtime and prime time to get this work done. The places I worked always had max overtime going for IT and building services so there wouldn't be an impact (allegedly) on the business.

      Plus, a lot of companies want to get things off the books, money wise. And lets not forget the 1st of the year layoff plans many places have now (not just IBM) So hump time for the troops while manglement parties on.

    3. Lord Elpuss Silver badge

      Re: Someone not following best practices

      "WTF is this "rush to fix?""

      To be fair, it doesn't matter when the deadline is - whether it's the Christmas hols, or an arbitrary '15th'. There will always be a panicked rush to get stuff fixed beforehand. If IBM adopted the 15-7th approach (which sounds reasonable to me) there would have been the same mail from the higher-ups saying don't rush it, just this time sent December 1st instead of 14th.

      1. Peter Gathercole Silver badge

        Re: Someone not following best practices

        The idea of starting the period of restricted change half way through December is to allow time for any cockups that do get in before the freeze to be fixed before the real Christmas shutdown.

        In most cases, it is not really a full 'freeze', because changes to fix operational problems may still have to be made, but it is really to hold back on any non-essential service affecting changes that may inadvertently cause a problem. Many organizations still allow changes in their non-customer facing systems.

        1. Anonymous Coward
          Anonymous Coward

          Re: Someone not following best practices

          I work for a major retailer so our change freeze is from mid november to mid january. I think the first week of november and third week of january are probably our most unreliable weeks as projects rush all their changes in just before and just after!

    4. This post has been deleted by its author

  2. Anonymous Coward
    Anonymous Coward

    IBM

    Inundated By Morons

    Inept Beyond Measure

    Incompetent Bloody Management

    Indian By Model

    1. Anonymous Coward
      Anonymous Coward

      Re: IBM

      "Indian By Model" - So, it's some pejorative term !

      Guess, you were kicked in butt by a smarter, nimbler Indian, or some other genital deficiency ?

      1. Anonymous Coward
        Anonymous Coward

        Re: IBM

        smarter, nimbler Indian

        Yea, right you owe me a new keyboard. I've been there and attempted to herd these specific set of cats. Gave up as nothing ever got delivered on time and the excuses were even crap.

        1. Anonymous Coward
          Anonymous Coward

          Re: IBM

          > Gave up as nothing ever got delivered on time and the excuses were even crap.

          And what was delivered was always dreadful quality, because they don't understand the business or learned C on the Internet but don't understand pointers.

        2. Anonymous Coward
          Anonymous Coward

          Re: IBM

          There are plenty of Indians smarter than I, but that does not alter the fact that India does not have a single university in the global top 200. The reason holders of Indian degrees are a laughing stock is not because of where they were born, it's because of where they graduated.

          1. Anonymous Coward
            Anonymous Coward

            Re: IBM

            "India does not have a single university in the global top 20"

            It's like saying one can not be a food connoisseur because his town doesn't have Michelin star restaurants.

            Take a hard look at your biases and prejudices. It might save COST of pills later on.

            1. Cynic_999

              Re: IBM

              I was in India last year, and where I was the average Indian 12 year old schoolchild was a lot better educated than most U.K. 16 year olds. That included formal English grammar. Not every child goes to school, but the majority of those who do tend to take it very seriously.

            2. John H Woods Silver badge

              Re: IBM

              India certainly does not have any Universities in (or, indeed, anywhere near) the global top 20, but it does have three in the top 200.

              According to this it actually has three: IIT Delhi (172); IIT Bombay (179); and IIS Bangalore (190).

              I think it's fair to say that given its size and GDP, India is under-represented in this ranking. And I very much doubt it is because of "biases and prejudices" --- it's more because India severely under performs in this sector.

              That does not mean that any given Indian graduate is going to be poor but it does suggest to me that, on average, Indian degrees are poor.

        3. Anonymous Coward
          Anonymous Coward

          Re: IBM

          Yeah - That's the reason many of successful Valley start-ups were founded by Indians. Many large Tech giants are led by folks, you may not like to break bread with. Yeah, go ahead and dump your keyboard.

          1. Anonymous Coward
            Anonymous Coward

            Re: IBM

            Never heard of an "successful" startup as specified. But when I worked for IBM it was the quality (actually the lack of) of the outsourced workers that was the problem. Why employ one good ozzie when you can get 10 outsourced overseas workers at a cheaper price.

            Now pass me the thousand or so typewriters.....

            1. Anonymous Coward
              Anonymous Coward

              Re: IBM

              Well, living under the rock blinds you to reality. Doesn't it. Look around, and you'll find many startups and enterprises, founded/led by Indians, Nutanix, Rubrik, NetApp, Adobe, Google, Microsoft, Atlantis, Tegile, Cohesity so on and so on. Can't find many "Ozzie", though.

              Regarding outsourcing, a lesson or two in Fundamental of Economics will be helpful. Probably, a nap, when Professor was teaching marginal utility-marginal pricing concept, made it difficult to understand these business decisions.

    2. PassingStrange

      Re: IBM

      "Inundated By Morons"?

      Wildly unfair. Quite the opposite. Onshore, at least, still full at the professional level of very bright, dedicated individuals, doing their best in "trying" circumstances. And - whilst they were hell to work with - it's hardly the fault of the offshore guys that they've never been exposed to the proper ethos and levels of appropriate training, either - bright people with good qualifications, dragged in to "body shop".

      "Instructed By Morons", or "Imbecilic Bloody Management" now - those are VERY different.

    3. Chemical Bob

      Re: IBM

      "Impacted" By Management

      I Bought Magicbeans

      Irritable Bowel Movement

      1. Anonymous Coward
        Anonymous Coward

        Re: IBM

        Insulted By Management ?

        Intubated By Management ?

        Irritated By Management ?

  3. Nate Amsden

    i'm sure

    that individual's high standards for not breaking things during this critical time will weigh heavily into the decision of whether or not IBM lays them off in the near future.

    1. MrBanana

      Re: i'm sure

      Ha, ha. I see what you did there, you thought that IBM might look any further than the country that the employee was working in and the benefit of moving that job offshore. They have committed to moving 80% of jobs, individual standards or not.

    2. Anonymous Coward
      Anonymous Coward

      Re: i'm sure

      You've discovered my escape plan.

      Leave the coat, I have myriad changes to force through.

  4. IGnatius T Foobar

    Important message from IBM upper management!

    "Merry Christmas. We're sacking you next week. In the meantime, please don't break any customer stuff, kthxbye"

    1. Anonymous Coward
      Anonymous Coward

      Re: Important message from IBM upper management!

      P.s if you break anything while carrying out the high risk changes before you are laid off, we will fire you.

    2. DNTP

      Re: Important message from IBM upper management!

      "Beatings will continue until morale improves. Support staff layoffs will continue until customer support improves. Employee shredding will continue until people stop posting the '...to shreds, you say?' meme."

  5. Bob Vistakin
    Facepalm

    These "staff" you speak of

    There are none left. The only employees are managers now. Managing other managers. Using big, long, complicated manager words.

    1. Anonymous Coward
      Anonymous Coward

      Re: These "staff" you speak of

      Onshore it seems there are more management than staff if you count Service Delivery Managers, Delivery Program Executives, Service Integration Leaders etc. I even recently learned our account had it's own Marketing team and manager and perhaps the funniest reaction was that of the CIO of the customer who was surprised there was a marketing team as he'd never heard from them...

      We do have 30 or so 'offshore' staff who have been flown in onshore for the last 12 months - all for skillsets locally available but they still answer to offshore managers. Customer had complained there weren't enough locals to work with them so IBM flew part of the offshore team in and now bill the customer for 'locals' Even with renting apartments, per diem etc. an SDM told me it's still cheaper because they don't pay super etc.

      Offshore the ratio is roughly 1 Level 1 manager per 16 staff so where my counterparts have 90 or so offshore resources there are 6 managers. This makes things extremely difficult when trying to escalate as you have to work out which team member answers to which manager. They don't group the team members of one set of skills/roles under one manager, another set of roles under another manager - you'll have a mix of different roles under each manager but that manager may be responsible for a completely different activity (e.g Manager is people manager of a mix of Windows, VMware, Linux staff but is responsible for Windows compliance activities.)

  6. Anonymous Coward
    Anonymous Coward

    Offshore mistake = onshore pain

    The end result of an offshore person making a massive mistake ends up with IBM wearing a huge SLA penalty and additional costs (customer includes all the hours of reporting back to regulator in their damages costs billed back to IBM.)

    This will result in additional heads rolling in onshore technical teams to make up the lost revenue. We saw this time and time again where every time our counterparts offshore caused a penalty > $250k that we'd lose a local team member shortly thereafter. So the first quarter cuts for 2018 will likely be more savage than those that had been planned originally.

    Failed changes has a wide scope - backed out because customer hadn't adequately tested in a pre-prod environment and prod broke is classed as failed, organising a hardware replacement and the part arriving DOA is marked as failed. An engineer not being available during a change window due to delays on another customer resulting in the change being cancelled was marked as failed. A change running over time due to issues (e.g hardware failure during patching) was marked as failed.Traditionally, IBM had a high standard for what constituted a failed change and what was considered a failed change in IBM was not a fail on some customers standards.

    From personal experience, I know a lot of offshore teams will apply a customer's less rigorous standards if they can get away with it to avoid having to fill in Failed Change Review documentation. I used to have to escalate the offshore team and argue with their managers to get them to fill in FCR's for changes that I knew had failed while they would try to argue the change hadn't failed, it just wasn't successful...

    Because everything is stats driven and all managers want to have 'green' reports and don't want to look bad to upper management there is a lot of 'fudging' to avoid coming to notice on Singapore's radar. I saw outstanding items that were appearing in red for being overdue suddenly disappear overnight - they worked out they could 'close' the item as 'merged with item y' then create a new item z with the old x and new item y as a merged record - problem solved until the new merged item ran overdue because the old x item still hadn't been resolved

    1. Anonymous Coward
      Anonymous Coward

      Re: Offshore mistake = onshore pain

      This will result in additional heads rolling in onshore technical teams to make up the lost revenue. We saw this time and time again where every time our counterparts offshore caused a penalty > $250k that we'd lose a local team member shortly thereafter. So the first quarter cuts for 2018 will likely be more savage than those that had been planned originally.

      Are you suggesting this is part of the offshore teams cunning plan to get more work sent to them.

      One day we will be so reliant on them they won't care about the onshore management.

      1. Anonymous Coward
        Anonymous Coward

        Re: Offshore mistake = onshore pain

        "Because everything is stats driven and all managers want to have 'green' reports and don't want to look bad to upper management there is a lot of 'fudging' to avoid coming to notice"

        Also known as a Watermelon Report: from the outside it looks green, but if you look at the details it's really red on the inside.

      2. Anonymous Coward
        Anonymous Coward

        Re: Offshore mistake = onshore pain

        It was no grand plan, just jaw dropping mistakes, attempts to cover up and obfuscation and evasion by both the staff and management when trying to determine root cause.

        They don't care about onshore management as they don't answer to them - completely different chain of command. All that can be done is escalate their management to get things sorted if the issue isn't high visibility/impact and in those cases it usually takes the customer escalating to get things done. I've lost count of the time that the handful of members of my onshore team have had a local manager push us to do work at short notice that had been with offshore for weeks or months with no traction and the demand for us to complete it because the customer has escalated to highest levels locally.

        What is particularly galling is the same management who will push us hard to fix a project that has been stalled for months due to offshore pushbacks and roadblocks won't push the offshore management about the failure to deliver. The common refrain being "It's India, what can we do?" Last time I responded to what they can do I ended up with a warning and a note in my permanent record - no I wasn't suggesting anything offensive, I had merely suggested that he get some intestinal fortitude and push the offshore management and team as hard as they were pushing us.

        Offshore teams know that IBM is entirely reliant on them now, it's one of the reasons why some things just don't get done. If they don't want to do it, they push back with excuses and roadblocks until the local manager gives up and goes away. Accountability and commitment is a problem for some members of offshore teams - we've been in the middle of high severity incident and the entire team has dropped off a call because they have 'gone for breakfast.'

        There are some very good and dedicated offshore team members but they don't often stay long - IBM is seen as a company for a 'fresher' to get a few months experience working in an IT company, familiarity with processes etc. before they go to another outsourcer who pays more. Some of our more recent hires had been exceptional (onshore team did the final interviews) but they are on a band 3 times higher than the other team members which caused others to leave. These higher band hires have been well worthwhile but overall costs for offshore have gone up as a result as their team mates push to be made a higher band. So account management are looking to achieve cost savings from onshore staff...

  7. Anonymous Coward
    Anonymous Coward

    Customers usually set the change freeze periods, then projects run by the customers go into flat-panic-overdrive to meet their schedules and put changes in. Often it's this customer-created paradox that leads to changes seeking exemption from lead times.

    Having said that, I've seen some really impressive balls-ups in change execution while at IBM so without specifics all we can do is throw popcorn.

  8. Anonymous Coward
    Anonymous Coward

    Changes before change freeze

    I have one of these going through at the moment.

    Getting a change complete by Friday the 15th, got to the point today where the configuration is complete and managed to start testing at 4pm. What I was hoping to achieve isn't quite there but good enough to resolve a software bug causing outages in a version that will never get fixed by replacing it with something else.

    Change is going in on Friday from 9am.

    Fortunately for me, this is on a UAT system and it's only been one year that I've been trying to get this done, prod will get done in early 2018.

    Anon 'coz

  9. Anonymous Coward
    Anonymous Coward

    Won't be a problem for IBM Australia, they have almost no staff left after making them all redundant.

    1. Nolveys
      Childcatcher

      Dear Doug,

      As the last person employed below upper management at IBM Australia we implore you to give an extra strong effort in the days leading up to Christmas. Please also continue your extended efforts during Christmas, between Christmas and New Years, during New Years and during the entire month of January. As per your contract there will be no overtime or holiday pay. Also, as per the specifics of your contract, we have taken possession of your car and sold it.

      We believe you will be very pleased with your Christmas bonus, continued employment through at least the month of January.

      Have a nice day and Merry Christmas,

      - Management

      P.S. Will you have time in January to, on top of your regular duties, to train your replacement?

      1. Anonymous Coward
        Anonymous Coward

        Re: Dear Doug,

        It's Australia - it's Dazza not Doug...

  10. Anonymous Coward
    Anonymous Coward

    how is it...

    that any customers are still using IBM services? I mean, I can understand support for OS390 (ok, maybe understand), but is there any IT manager still living enough under a rock to know that buying ppl from IBM is just dumbassery at it's finest?

    1. Anonymous Coward
      Anonymous Coward

      Re: how is it...

      "...is there any IT manager still living enough under a rock to know that buying ppl from IBM is just dumbassery at it's finest?"

      It's not tho, is it. There are still great people at IBM working extremely hard to deliver great work, despite being hampered by poor management, impossible deadlines and a world laughing at their company.

  11. GruntyMcPugh Silver badge

    'Failed' changes,....

    ... during my tenure at IBM, the criteria for a 'successful' change was quite strict, I did Security and Compliance work, so deployed a lot of patches. Say I raised a change to deploy the monthly round of patches to an environment, if one patch failed to apply on one server, but twenty other servers successfully installed half a dozen or more, I still had to mark the change as 'failed', so a change could be 99%+ successful, but still get marked as a 'failure' even if said missing patch wasn't critical, and there were other opportunities to apply it well within the deadline set by CIRATS (Compliance, Issue, Risk, APAR Tracking System (APAR is an IBM acronym 'Authorized Program Analysis Report' basically coding defect reports, a bit old hat now, as was the entire CIRATS db)). Then once a change was reported as 'Failed' we had the '5 Whys' to go through. The 'CIRATS co-ordinators were an off shore resource, and had been told to make sure they got 5 whys, so telling them a patch failed because there was a data transfer limit in EndPoint Manager which was had since been increased wasn't enough, they'd push for another four reasons. It got quite tedious.

  12. FordPrefect

    Will senior management accept that there is a finite amount of resource so if the amount of change exhausts the amount of resource then no more work can be done? No it will be business as usual pushing for more and more work to be done quickly so it can be billed. Those same senior management will then blame the overworked stressed workers that make a mistake due to having to much work and too little time.

  13. Anonymous Coward
    Anonymous Coward

    Looks at Changes I have scheduled for next 2 days....

    ...17.

    Next week is change freeze, or as I like to call it "Change freeze except for Christmas eve when a dozen departments forget that Christmas Day is on the 25th and need a load of changes made for their customers right now!"

  14. Antron Argaiv Silver badge
    Happy

    Australia border security?

    "Sorry, we're closed. Kangaroo out front should have told ya.:"

  15. Anonymous Coward
    Anonymous Coward

    I knew someone who used to work for IBM and when I commented on them working between Christmas and New Year they replied that it was great .... all theit customers would be off on a Christmas shut-down so it gave them a few days to get some work done without being interrutped!

  16. Anonymous Coward
    Anonymous Coward

    We've just had a customer...

    ...submit a request yesterday afternoon for a major data migration to take place between Christmas and New Year. Involving *at least* 3 other companies.

    What do you think the odds are of this going off smoothly?

    And they will scream blue bloody murder if everything is not up and running without a hiccup.

    See, the problem is that we pulled a magical rabbit out of our arses when they had an "emergency" job, and that level of service has become the new baseline. Sociopathic fuckers.

    It's been escalated to our board vs their board, and we're patiently trying to get everyone to understand the concept of a change freeze. I'm not holding up much hope though.

    1. Antron Argaiv Silver badge
      Happy

      Re: We've just had a customer...

      Easy peasy: take your standard rates, multiply by...oh, let's say, a BEELION...and add a no liability clause to your standard contract. Take it or leave it.

      They want it, let 'em pay through the nose with no guarantees for it...they'd do it to you if the tables were turned.

      // make business FUN again

      1. Anonymous Coward
        Anonymous Coward

        Re: We've just had a customer...

        That's about the situation. Still won't stop them screaming on the day, but it does leave them yelling into the void.

  17. Anonymous Coward
    Anonymous Coward

    This is one of the reasons outsourcing can be dangerous

    If you are a bank then a failed change can potentially kill you (see for instance the 2012 RBS troubles which didn't kill RBS but could have. So you tend to have a culture built around making sure that those things don't happen. If you're an outsourced supplier then a failed change can lose you the contract and possibly result in you needing to pay a big chunk of money to your client, but generally it can't kill you. And of course you need to be seen to be more efficient than the client's own staff were, so there is pressure on you to cut corners, even if it sometimes kills the client.

    (Disclaimer: this is not always true, but it is true often enough that I'd worry about using a bank which had outsourced its core IT.)

    1. Anonymous Coward
      Anonymous Coward

      Re: This is one of the reasons outsourcing can be dangerous

      Which banks have not outsourced the majority of their IT?

  18. Anonymous Coward
    Anonymous Coward

    Haters going to hate

    IT services is a tough game / do more with less, less ....

  19. Anonymous Coward
    Anonymous Coward

    Tough game

    IT Services is a tough game. Do more with less and try and keep clients as happy as possible.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like