back to article Never trust a developer who says 'I can fix this in a few minutes'

In this week's instalment of On-call, our weekend thing in which we share readers' tales of odd things that happen at odd hours, we bring you the adventures of “Foredeck” who once “worked for a small UK ISV ... as a Support Manager, which meant I had a lot of different hats to wear, including being on-call one week in three …

  1. Vanir

    Developers?

    In this situation don't you mean the coders? The people who wrote, quickly, the mess that is the codebase of the product in the first place?

    They probably originally said that they could write the code, deploy and install the product in a 'few minutes', to planned- or should that be questimated? - time, planned - or should that be given? - budget and to specified - eh? - quality.

    But they will go far! They have a 'can do' attitude!

  2. Anonymous Coward
    Anonymous Coward

    Project manager?

    you mean the guy who writes the plan for how the software will work, who will write it and sets a thousand milestones, but thinks a "systems requirement" means which version of windows you use?

    1. Vanir
      Go

      Re: Project manager?

      :-) Could be. But you meant 'should work' didn't you? Not 'will work'. Oh, you're right, it's that can do attitude again. It will work! Failure and bugs are not an option, they're not in the plan, that cunning plan.

      And the coders all shout in their best Scrum voices 'What's a systems requirement?'. And the project manager replies, 'Doesn't matter, it's working code that matters! Agile says so.'

      And all the coders think: 'Nothing's changed then'.

      1. Doctor Syntax Silver badge

        Re: Project manager?

        I take it that you've inside knowledge of this particular situation as you seem to know that the codebase is a mess, that it was written in a hurry, that they're following scrum etc.

        I can think of several alternative ways in which this could have gone wrong. For instance a salesman having sold the client a product that didn't fit with assurances that it could be adapted (I've quit as a developer over having that dumped on me). Or, for instance, the development team, or a good chunk of it, having been pulled off to work on something else, leaving them insufficient time to complete what was, originally, a well estimated project.

        1. Eric Olson

          Re: Project manager?

          Which estimate are you talking about? The one which the senior-level leaders who haven't actually coded in years came up with? Or the one that the mid-level architects came up with when pressured to confirm that high-level estimate, despite knowing that the requirements had already changed enough to render it useless? Or perhaps the actual poor saps tasked with delivering the pig's breakfast. I find that it's the estimate that the last group comes up with is the most accurate, though the most ignored one of the lot.

    2. SVV

      Re: Project manager?

      This is a classic project manager fail.

      They clearly never planned in for appropriate developer testing, user acceptance testing (on an exact copy of the production environment), or rollback in case of an unforseen failure (and of course those rollback procedures HAD actually been tested in the past, hadn't they?)

      So, the reason this happened (and has been for decades and will still happen for far too long into the future) are either

      1) The project manager was not competent enough for the role because they didn't understand the necessity of the above tasks

      2) The company saw them as a "cost with no benefit", did a cheap rough-and-ready rollout and got badly burnt by it. Possibly because "we've never had this problem before", in which case you learned the hard way.

      1. TeeCee Gold badge
        Facepalm

        Re: Project manager?

        Or maybe they did plan it correctly, but:

        Having done so, there is the small matter of the budget allocation. You have in your hand a proposal for a project which will take X months and cost Y, so first on the agenda is off to TPTB to get the budget approved.

        1) Twiddle thumbs until next approval board.

        2) Get sent away with flea in ear to cut costs / improve payback / add cat pics / whatever.

        3) Tweak proposal and twiddle thumbs for another month.

        4) Repeat 2 and 3 a couple of times.

        Presto. A project that takes X and costs Y with X - n months and Y - ${large_random_number} to do it with.

        And TPTB then wonder why every bloody thing is either over time and budget or piggin' unreliable 'cos the testing was skimped.

  3. Disko
    Pint

    of course...

    ...we can only ever upgrade a system that is live, critical and has no fallback, spare or failover option, and when we do, we upgrade every single instance of whatever it is we're upgrading, just to make sure there will be no incompatibilities arising from running different systems, and when soon enough the trolls take control of the bridge, we don't remember rollback protocol but instead steadfastly rely on a random someone yelling random somethings somewhere away from the rather more aggressive streams of consciousness that project from management...

  4. Anonymous Coward
    Anonymous Coward

    Trust

    Never trust a developer who says 'I can fix this in a few minutes'

    And never trust a manager who allows a developer to get into this situation and believes the answer too.

    1. Blank Reg

      Re: Trust

      It might well be that he can fix it in a few minutes, but can he run a full test cycle to make sure that he didn't bugger up something else?

      1. Steve Davies 3 Silver badge

        Re: Trust

        I had a problem described to me a few weeks ago.

        After a few moments of thought I answered, "I know what the problem is and how to fix it. It will only take me a few minutes BUT, they way they are using the application is not covered in the signed off use cases and there is no specific test for this set of operations. That will take 2-3 weeks to draft, get approved and fully regression test the fix and get the customer to sign off on it".

        The PM answered, "But I told the customer thay could have the fix tomorrow. It didn't sound that difficult to me."

        The PM is a failed developer which says a lot really.

        The moral is that changing one line of code may only take 10 mins but everything else could make the time to fix extend out for weeks if not months.

        1. Anonymous Coward
          IT Angle

          Re: Trust

          @Steve Davies 3 - I'd have instantly convened a 5 day brainstorming workshop - naming it something like 'synergising the clear way ahead synergistically at 110%'. There would be hefty PowerPoint presentations ('directionalisting the fix space'), hours of patronising role playing ('climbing the trust tree') and an exceptionally irritating and highly paid external facilitator with a strong accent (they always have a strong accent), followed by forced team bonding with cheap wine over a nasty dinner in an uninspired restaurant. And that'd just be day 1.

          After 5 days of that you wouldn't be mentioning bloody signed off use cases in my presence, matey, you'd be begging me to allow you to fix the problem as fast as possible.

      2. JeffUK

        Re: Trust

        As I responded to a complaint of "You only changed one line of code, why did it take so long?"

        "Changing one line of code: 3 minutes"

        "Changing the right line of code: 3 weeks"

  5. Little Mouse
    Unhappy

    "and never rely on a Project Manager to handle a customer fire-storm"

    In the public sector and NHS at least, the inability to manage projects is no impediment to holding the position of Project Manager

    That role is reserved for the Unwanted who for one reason or another can't be sacked. There's simply no where else to put them. At least the amount of damage they can inflict is limited to a single project

    1. Doctor Syntax Silver badge

      "At least the amount of damage they can inflict is limited to a single project"

      Don't be too sure. Ever heard of multi-tasking?

  6. ecofeco Silver badge

    Just another day in the life

    Just another day in the life of IT world.

    You should see the project I'm working on now. Same old same old. The left hand doesn't know what the right hand is doing and the client is up to the same old trick of trying to sabotage the SLA.

    Many days I wish I'd become a forest ranger.

    1. Anonymous Coward
      Anonymous Coward

      Re: Just another day in the life

      "Many days I wish I'd become a forest ranger."

      I dream of getting work as a human cannonball or a lion tamer armed only with a chair and whip!

      1. Steve I

        Re: Just another day in the life

        "I dream of getting work as a ... lion tamer"

        You need your own hat. One that lights up saying 'lion tamer' in great big neon letters, so that you can tame them after dark when they're less stroppy.

        1. AbelSoul
          Trollface

          Re: after dark when they're less stroppy.

          Stroppy lions? That's tigers you're thinking of.

          Lions are uppity.

    2. KA1AXY

      Re: Just another day in the life

      Many days I wish I'd become a forest ranger.

      Or...a lumberjack!

    3. mathew42
      Coat

      Re: Just another day in the life

      Many days I wish I'd become a forest ranger.

      i sometimes gaze wistfully at the guy driving the ride on lawn mower in circles around the oval, except today when it is raining.

    4. Brewster's Angle Grinder Silver badge
      Paris Hilton

      Re: Just another day in the life

      "The left hand doesn't know what the right hand is doing..."

      Same here. Unfortunately I am both the left hand and the right hand. Ugh. :(

      Paris, because she's hands free.

  7. Roland6 Silver badge

    There was a contingency plan!

    " why the PM had not just gone for a back-out when it went wrong? That would have restored service, albeit on the old version, well within the change window"

    I've been on projects (and in the middle of one now) where the main supplier has no concept of contingency planning, believing the best way forward is to simply burn the bridges and fight the fires as they arise. Current project is just such as case, the system was supposed to go live 10 days ago, well parts of it did - sort of, however,the client's Finance Director is starting to get shouty, as still she is without a functioning accounts system, something the main supplier was supposed to have ensured would not happen...

    1. Naselus

      Re: There was a contingency plan!

      "I've been on projects (and in the middle of one now) where the main supplier has no concept of contingency planning, believing the best way forward is to simply burn the bridges and fight the fires as they arise."

      Wait - you work for the DWP?

  8. Anonymous Coward
    Anonymous Coward

    Being On-call was the worst job I ever had. 1 in 3 and like the scenario here really it is 3 in 3 as if the shirt really hits the fan you would have been expected to go in, what really frustrates me though was the bosses that didn't seem to realise you need a life too and it is a lot of pressure being on-call, even if you get no calls.

    The best calls were from those clients which refused to pay for night and weekend cover but would ring up anyway because "the problem is so critical it can not wait till Monday"! Ok, first let me explain to your the concept of not paying for cover, we don't cover you. Usually I'd have a quick look at whatever their issue was but about 50% of the time end up telling them to jog on as the team which deals with their account does not provide a 24/7 service and I have no idea about their system. Also advising them if they paid for 24/7 cover then their account would be moved to a 24/7 team rather than the much cheaper office hours service. It actually costs to have us around the clock you know.

  9. Rob Crawford

    What a surprise

    Just confirms my opinion of Project Manglers and the majority of developers I have had the misfortune to meet.

    1. TkH11

      Re: What a surprise

      My experience is that most developers are actually pretty good, unless they come from certain countries. Project managers generally I have found are quite poor.

      One PM said to me "Being technical is a disadvantage" and I've heard others say that too. I say BS. The problem with a number of PM's I've come across is that they don't understand what the heck it is they are working on, not being technical enough is the cause of much of the problem. One PM with no less than three university degrees didn't understand what system integration is!

      Sure, it doesn't mean to say that everyone that is strong in a technical capacity can become a good PM, but having good technical skills has got to be good to have to be a PM.

      I've seen over the years a small number of people who were in the technical role, and who were not very good at it, move into a PM role.

  10. Anonymous Coward
    Anonymous Coward

    Planning in real life

    I've had the misfortune, over the years, to be at the sharp end of implementing various IT projects with my staff. All coming from some bright idea of the people above.

    No one had ever asked us what we needed.

    No one had ever consulted with us in the planning stage.

    No one had ever tested usability with us.

    No one had ever sat with us and worked out when would be a good time to make the change over.

    No one had planned a test programme on live data.

    No one had ever planned for the contingency when the thing falls over.

    And every single one then took months of work to make the thing useful ( if it ever did become useful).

    And yet over the same period I was able to work with the engineers to implement a number of smaller projects that we had requested that they support. Some of which were then rolled out to other teams in a similar line. Making sure that those teams were similarly consulted.

    And without causing disruption or pain.

  11. BongoJoe

    Sales

    Project Managers were, in my experience, people who were dropped onto you only to give you grief, provide you with charges that made no sense and never spoke to you but only told you things.

    You'd get a PM for your 'team', aka yourself, after Sales had sold an application or project to a client which hadn't been designed or written yet. Sales droid would have been in the pub for a lengthy drawn out expenses paid-for marathon with client and have decided on the train home, pissed-up, what was required and when it was to be delivered. If there was an element of consultation then it would have only have been between Sales droid and PM over another lengthy pub session.

    Then and only then would I hear about it. Of course a pretty chart from PM being the only form of project requirement and he having a near fatal hangover he wouldn't remember what was required to any useful level. The Sales droid would by now have oozed over to another client and all I was left with was a chart full of milestones and little clue as to what I was supposed to do.

    I now work for myself.

  12. Lallabalalla
    Mushroom

    Those who can - code

    Those who can't - always find a way to bugger everything up.

  13. TkH11

    Sales people

    It puzzles me as to why in this day and age sales people are such a nightmare at promising the impossible to customers, I've seen it repeatedly throughout my entire 20+ year career to-date.

    It p*sses off the customer, (and thereby reduces the chances of repeat business), it p*sses of the development team.

    I suspect sales people are more interested in hitting their targets and getting their nice fat bonus.

    1. Anonymous Coward
      Anonymous Coward

      Re: Sales people

      This is so true. Setting expectations that are unrealistic is the sign of a sales person who is not suited to the technical world. They are more suited to selling snake oil. You need to set expectations that are amazing but will take time, consultation, testing and collaboration. Otherwise like you said you p off the client, you p off your team and nobody ever wins. Except maybe the sales guy on some %. If companies reward sales staff like this - this is a problem on the company side that needs to be addressed so as to correct the problem. Only then, once the sales guy has a kind of SLA will you see real results. Give the customer what they want but be so clear of how long it takes because it is something special that both parties want to work to get absolutely right. I worked in a company that didn't see this for 15 years. I'm happy to say in the last couple they are getting it right.

  14. TkH11

    Trust

    "Never trust a developer who says he can fix it in a few minutes". Developers probably can fix it in a few minutess, but the developer is giving a time estimate of their work only and not thinking about the rest of the activities that need to be completed in order to make a release.

    I'd still trust the developer, but it's my job as the project manager to give a realistic timescale and know all the other activities that need to be undertaken.

  15. TeeCee Gold badge

    Simple install.

    One of favourite "fix it on the fly" gags was a software upgrade that had gone through testing and proved to be a pile of poo in live. Many weeks of tweaking later all was sorted and the go-ahead for further rollout was given.

    The developers had provided a nice, self-extracting, install, migrate and configure package that tidied up after itself. Trouble was, its only real run in anger had been the original "pile of poo" installation and much tweaking of its contents had occurred since.

    The result was something where you pressed a button, it shat all over the environment and then removed all traces of itself (leaving only the thoroughly stuffed remains of the system) so you couldn't see what had gone wrong. Oh how we laughed......

    1. Roland6 Silver badge

      Re: Simple install.

      Years back one of my client's had a developer "fix it on the fly" - no one could get the production system to work apart from the developers... After much investigation (we finally did a clean room build, because we couldn't gain access to a production system) it was discovered that the developers were simply replacing the production system with a copy of their development system (complete with dev tools) - which worked, but the developers had no clue as to why it worked (probably due to excessive use and abuse of root privileges and undocumented hacks)...

      1. Anonymous Coward
        Anonymous Coward

        Re: Simple install.

        Seen debug kernels miraculously fix race conditions in the past (extra logging etc). Hardware vendor deemed it 'good enough' so we went into production with it!

  16. Anonymous Coward
    Anonymous Coward

    Don't understimate managements ability to fuck up

    I worked for a Japanese company some time ago.

    There staff had gotten together (outside office hours) to write a hardware neutral systems requirement, including orders of magnitude for the various elements of the system. All they cared about was it got the job done, not what it ran on.

    And then the the CEO played golf with CEO of a software house.

    Game over. No requirements needed.

    I've also cleaned up the mess of one of those "can do" coders, who can indeed "fix" any software problem in 5 mins.

    Provided the PM schedules another week to fix the fix afterward.

  17. Ynox

    Back out first. Then fix.

    Spent a while in a similar industry.

    General approach I used to always take used to be to spend 5 minutes or so doing some analysis, then if I didn't know what the hell was going on, revert to a previous version. Allows you time to do a proper fix and apply regression tests etc to ensure you're not even more in the shit than you were at the outage.

    That said, I've also had enough angry PMs yelling at me at 11pm. Easier said than done when the shit's hitting the fan!

  18. Anonymous Coward
    Anonymous Coward

    PM

    Ironically I would love to have a decent PM at the moment, even one who bombards you with spreadsheets, project plans and conf calls to ask why you haven't done something. Some examples why:

    We don't need those servers in that colo any more, decom them. 3 weeks later, we need severs in that colo, put them back.

    We urgently need two servers in this colo, ASAP, this is really important etc. The servers are rushed in and then sit there unused for two months. And then it is decided one of the servers is urgently needed in yet another colo.

    November, please implment a middleware setup to do X. It is implemented. Flash forward to April where they expect X to work instantly in production at a day's notice, despite the fact no-one ever tested it. They claim they did test it, but it was using boxes in the same subnet. So it didn't do any inter-colo stuff or pass through any firewalls etc which is where a problem surfaced.

    To cut costs it is decided that in a colo we will decrease the amount of cabinets we're using. So someone goes ahead and decides on a date when the lease will end. All this is done without consulting anyone else on the how/when/who of moving kit out of those particular cabinets.

    Being told that there are 40+ servers that can be decommed straight away as they're not being used. So we ask for a list... and 18 months later we're still waiting.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like