back to article Engineer crashed mega-corp's electricity billing portal, was promoted

Welcome again to “Who, me?”, The Register’s confessional column in which techies unburden their souls by revealing that they have broken stuff. This week meet “Matt” who told us he works for a company that makes “email tracking software for corporate communications.” Said software uses a web server “to detect message opens and …

  1. tip pc Silver badge

    Promoted for failure?

    Maybe that’s where I’ve been going wrong all these years, no one wants a goody 2 shoes, they all want a bad boy!!

    1. My-Handle

      Promoted for failure...

      One previous company I worked for had a rather bad habit of promoting people into positions where they couldn't do much damage. One such person, promoted to the position of project manager, was renowned for having -never- brought a single project to a successful conclusion.

      They would never have dreamed of promoting someone who was -good- at their job. After all, who else would do that job?

      1. Oengus

        Re: Promoted for failure...

        They would never have dreamed of promoting someone who was -good- at their job. After all, who else would do that job?

        My philosophy on this was "Never make yourself indispensable. If you are indispensable you can't be promoted."

      2. Anonymous Coward
        Coffee/keyboard

        @My-Handle

        You must've been a colleague of mine when Andy got promoted.

        Dilbert doesn't do it justice!

      3. Anonymous Coward
        Anonymous Coward

        Re: Promoted for failure...

        ...we call it (I live in a whole other Country) "Falling Upwards". The American military has another name for that...

        A certain former US Senator was once a Marine Pilot... involved in some accidental missile firing on top of a Carrier deck. That missile hit another airplane and it went as well as you'd expect.

        What's that expression? "Polishing some leather chair in the Pentagon with your *ss?" or "Pushing pencils in the Pentagon?" Anyway, since the guy was somebody's son, and the circumstances were not clear, (I wouldn't recall the story) they put him in a position where he couldn't detonate any amount of explosive ordnance, by the distance of a single button press.

        On the other hand, the man on the article was working with insufficient information.

        1. This post has been deleted by its author

        2. Mark 85

          Re: Promoted for failure...

          A certain former US Senator was once a Marine Pilot... involved in some accidental missile firing on top of a Carrier deck. That missile hit another airplane and it went as well as you'd expect.

          A little fact checking here.. The errant missile wasn't launched from his plane. It was from one on the other side of the carrier and due to static as the techs connected the missile to the fire control system. And the senator is still an active senator. Oh, and he was Navy not Marine.

      4. Tweetiepooh

        Re: Promoted for failure...

        Methinks the problem is promoting people who are good at their jobs to positions where they have to spend their time managing people who aren't.

    2. Dan 55 Silver badge

      The trick is to get noticed for fixing problems because you'll never get noticed for doing your job so well that problems don't happen in the first place.

      Never managed to do that.

      That's not a humblebrag because I'm the first to admit I don't do my job that well in the first place.

      1. Prst. V.Jeltz Silver badge

        Tried it both ways

        The trick is to get noticed for fixing problems because you'll never get noticed for doing your job

        Yeah did that , still never got anywhere...Although they do leave me alone and let me get on with stuff now... a bit like Leonard deQuirm , with similar salary.

      2. Alan W. Rateliff, II

        I have heard tell of several agencies which give awards to their employees. Some of the awards are given to people who clear up back-logs of projects, tickets, paperwork, etc. Some of the award recipients are responsible for the back-logs in the first place.

        Strange world we live in.

      3. Stevie

        The trick is to get noticed for fixing problems

        Years ago I was recommended to a high-ranking manager as a replacement for a staff member working on a (needlessly) complex system.

        I swept in and began suggesting immediate fixes and planning to eliminate various systemic problems, including a number of landmines buried in the recompilation process I thought that a more jaundiced eye than mine might view as a job security perimeter.

        Some days later I found that I had been replaced by the original staff member, though no-one told me about it and I found out when our different administration practices collided at full speed and their respective boilers burst spectacularly.

        I never understood why I wasn't popular as my work habits had always stood me in good stead in my previous position, the one from which I had received such a glowing write-up. And yes, I considered the "give him a glowing reference or we'll have to keep him" scenario, but had evidence to suggest this was not the case and the recommendation was genuine.

        Some years later I had moved to a different department and gained some distance and perspective. I suddenly realized that the manager in charge of Project Limpsalong had made his career from leaping into forest fires and noisily directing crack teams of firefighters to Put Things Right.

        In suggesting that we could remove the ignition sources I was threatening his visibility and his promotion prospects.

        Unfortunately, even had I realized this in time I would have been doomed. I can't have the power to fix a broken process and just sit on my hands, belting it with a Brummy Screwdriver every time it stalls. It's a pride in work thing.

        Oh well.

    3. David Nash Silver badge

      He wasn't promoted for failure

      He made a mistake on this occasion.

      He was promoted.

      Who said there was a causal relationship between these two events?

      Should we expect that nobody with "senior" in their job title has ever made a mistake?

      1. Terry 6 Silver badge

        Re: He wasn't promoted for failure

        Firing or disciplining someone for an error is plain stupid. We can all make mistakes. Making staff so error averse that they spend more time watching their backs than getting their jobs done can seriously damage the employer.

    4. veti Silver badge

      Thomas Watson quote seems apposite here

      "Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?"

      Someone who makes a screwup of that magnitude is someone who has a reasonable chance of remembering it. It's a learning experience. Why waste it?

      1. Alan Brown Silver badge

        Re: Thomas Watson quote seems apposite here

        "Someone who makes a screwup of that magnitude is someone who has a reasonable chance of remembering it."

        You assume they even realised they'd done it.

        Salestwits can do even more damage than that to a company in a heartbeat (and usually "they're our most sucessful guy", leaving a trail of peopel swearing never to do business with XYZ again due to overselling and general pushyness(*)), as can incompetent management allowing tech staff to refuse to fix their "rather unique" interpretations of standards (eg: some vendor implementations of SNMP) which don't play nice with anything else and have a similar long term effect on repeat sales (along with being struck off candidate lists for other outfits who've been related the horror stories and rely on ABC standard being implemented the same way as everyone else does it)

        (*) I was related the story of one salesman who would oversell product capabilities to management then let the customers get so angry and goad them with silence until the tech staff would swear at him in meetings - at which point he'd pick up his stuff and walk out, using the outburst as justification that the customer was being unreasonable and he couldn't work with them. It took the employre more than 20 years to realise the tens (maybe hundreds) of millions of dollars in damage he'd done to the company's reputation and long-term sales figures.

    5. John Brown (no body) Silver badge

      "Promoted for failure?"

      Not really accurate. The failure was not giving all the required data for the install to proceed. In fact, not simply "not giving" but actively hiding it. Sadly, that's an all too common occurrence. Job is specced up and when you get on site, it's never as simple as the spec. appears, often made more difficult by not being able to speak to the relevant people for clarification. I deal with "project managers" often enough to know that many are little more than jumped up marketing/sales types who've been trained to tick boxes and have little idea of what it's like on the ground. The good ones always seem to work for other companies.

      1. Alan Brown Silver badge

        "I deal with "project managers" often enough to know that many are little more than jumped up marketing/sales types"

        They're the best sort if you're a vendor. You can make much more money out of them as long as you document all the ways they've screwed up and the customer will have to pay to get things done the way you wanted to do it.

  2. Korev Silver badge
    Go

    Grown up way of dealing with things

    The client even learned something – Matt said “additional protocols were put into place by the client to prevent this from occurring again.”

    Matt’s company learned something too. “We only install our software on dedicated systems for production environments,” Matt told us. “Oh, and I still have a job - as the senior tech engineer.”

    This sounds like quite a grown up company. They didn't hang the techie out to dry for a mistake (which was probably worsened by the Company's procedures) and they change their way of working to lower the chance of it happening again. That sounds good to me.

    1. Anonymous Coward
      Anonymous Coward

      Re: Grown up way of dealing with things

      Well, it it was working remotely it was up to the local techie to know what IP:port could be bound without issues - still, you should be careful when you don't have a clear picture of someone else's system. It was a shared mistake, with little consequences. Yet, it's good they learned that deploying into a complex network requires some planning and proper knowledge of the overall system.

      That said, I never like too much to bind to 0.0.0.0 (or its IPV6 equivalent), because if a new network interface is added for whatever reason (now quite easy with VMs), applications may become bounded where they shouldn't.

      At least this application allowed to control binding - there are some that bound to everything without even asking you.

      1. Anonymous Coward
        Anonymous Coward

        Re: Grown up way of dealing with things

        No, not grown up... just political...

        Hauling a techie over the coals for this would have involved an investigation that was likely to identify and knock over other cans-o-worms that could do with being left alone.

        Just lucky it was a big enough problem to have got away with it, and of course knowing how to reverse the change.

      2. Anonymous Coward
        Anonymous Coward

        Re: Grown up way of dealing with things

        I agree. However, theres a lot of apps that only allow access from the subnet you bound the service to. Either that or its a free for all on 0.0.0.0.

        Its a piss poor security mechanism put in place to save time on an actual security mechanism. Either that or the dev building the services hasnt a clue about networking.

        A lot of hacked together trashy services written in nodejs using the built in webserver are guilty of this.

    2. tip pc Silver badge

      Re: Grown up way of dealing with things

      the customers change control process should have stopped this from happening.

      The project should have had to have done change control on that shared server and explained what was to be done and how.

      The OS's management people should have then reviewed the install process and then either performed the install or provided appropriate access to perform the install.

      Any danger of the existing webservice stopping should have been spotted and appropriate action advised BEFORE the change started.

      Me personally, i would not have let a third party install anything on my important webserver. For what they needed i would have insisted a new server be spun up or vm. In fact i'd have a pre determined process for adding new web servers. In addition i'd have reverse proxies and load balancers and ensure new URL's are either on our existing webservers or if it needed its own software stack installing it'd be on a new dedicated os, isolating something new and different from the existing.

      looks like the customer is at fault here for not having rigorous enough processes to ensure the integrity of their service.

      1. Anonymous Coward
        Anonymous Coward

        Re: Grown up way of dealing with things

        Sadly, a "just get it done" attitude from higher ups increasingly takes precdence over engineering best practices. As lomg as you strictly adhere to ITIL forms, there's no patience for attention to substance. I assume most enterprise shops are now hopelessly borked because of this. Many vendors, Microsoft leading the pack, insist on their pound of flesh for dev and qa systems, resulting in the home team bean counters refusal to approve them. So things are usually brokem before a vendor shows up. Add to this the prevailing "Training? Yeah, we'll get to that. Just ask the consultants. How many tickets did we close today?" attitude of many managers, and you've got a prescription for vendor-inspired mayhem like happened here..

      2. Alan W. Rateliff, II

        Re: Grown up way of dealing with things

        I did a recent stint where change meetings were the boogeyman, directly causing my part of the world to go T.I.T.S.U.P. a couple of times.

        And, really, you never know what kind of vendor you will get. I have worked with more than one vendor which told me they would not support the product if we did not do certain things their way, and this usually happened on the day of installation even with several calls and emails beforehand supposedly detailing the process and our requirements.

        From turning off all workstation firewalls*, to blank SQL sa passswords, to, yes, full take-over of IIS installations in bindings -- as happened here -- or putting an application in the default website rather than its own. As well, the customer had no means to stand up another server just for the application so we would have to go with it, at least for a short time.

        * still forced by a major medical software vendor for one of its Borged products which I will not name, but it does rhyme with Henry Schein.

        1. Anonymous Coward
          Anonymous Coward

          Re: Grown up way of dealing with things

          Working on a site that takes perimeter control seriously we have a very simple form for vendors to complete before installing anything which may need a route through the firewall. In the past 5 years I cannot remember a single vendor who completed it in advance despite frequent reminders. We were not in the position where we could refuse permission for the install to take place, there were always somebody else's critical project with ridiculous timescales. I would normally end up sat with a firewall engineer looking to see what ports and protocols were being bounced from the new server IP address then negotiating with the vendor about what was allowed through.

          The vendor default position is normally cant we jut open all ports to that IP address and they are seriously shocked when this is not allowed

  3. Anonymous Coward
    Anonymous Coward

    Those ominous words on the end of a phone line - "...should it have done that?"

    1. Oengus

      or "is that supposed to happen?"

      1. Peter Ford

        or 'oops'

        The one word you don't want to hear from someone sat at the console of a live production server...

    2. Jon 37

      Or "<IT guy>, did the main company file share get backed up last night?"

    3. Anonymous Coward
      Anonymous Coward

      Or "The iLO*, oh we never bother patching those in"

      *other brands are available.

    4. Antron Argaiv Silver badge

      "I've never seen it do *that* before..."

      also:

      "Where are all the files?"

      1. imanidiot Silver badge

        "Er.... Hang on a minute." Followed by the clattering of a phone on the desk and a rapidly diminishing sound of heavy footsteps.

  4. Dabooka
    Headmaster

    I can forgive 'Matt' for all of his sins...

    Except 'I was quite literally working in the dark'

    No Matt, no you weren't.

    1. Anonymous Coward
      Anonymous Coward

      Re: I can forgive 'Matt' for all of his sins...

      You say that, but many a time I have literally been working in the dark.

      One day they will have the brains to put some lighting in the hot isles....sigh.

      1. Alan Brown Silver badge

        Re: I can forgive 'Matt' for all of his sins...

        "One day they will have the brains to put some lighting in the hot isles....sigh."

        Funny you should say that. That's exactly what I'm working on at the moment.

      2. Annihilator
        Coat

        Re: I can forgive 'Matt' for all of his sins...

        “One day they will have the brains to put some lighting in the hot isles....sigh.“

        Like in the Balearics or the Canaries?

    2. Korev Silver badge
      Joke

      Re: I can forgive 'Matt' for all of his sins...

      Except 'I was quite literally working in the dark'

      No Matt, no you weren't.

      You sound like you'd literally kill him

      1. John H Woods Silver badge

        Re: I can forgive 'Matt' for all of his sins...

        Sorry, but literally doesn't literally mean literally any more.

        1. mrjohn

          Re: I can forgive 'Matt' for all of his sins...

          "Sorry, but literally doesn't literally mean literally any more."

          figuratively speaking it does

    3. Mark 85

      Re: I can forgive 'Matt' for all of his sins...

      Except 'I was quite literally working in the dark'

      No Matt, no you weren't.

      So you've never been a mushroom in IT...? Kept in the dark and fed a low grade of BS? Remarkable.

  5. P. Lee

    Multi-tenant

    That seems like a bit of a euphemism - tenants are usually separated from each other by walls.

    1. Anonymous Coward
      Anonymous Coward

      Re: Multi-tenant

      You have obviously not watched the TV programme about rogue Landlords then...

    2. Anonymous Coward
      Anonymous Coward

      Re: Multi-tenant

      "[...] tenants are usually separated from each other by walls."

      In multi-occupancy the tenants share communal access areas. The local authority licensing regulations are very strict about safety in such areas.

  6. SiFly

    Deserve what they got

    That's really poor :-

    1. Testing installation on live network without validating on test network.

    2. No deployment plan

    3. No approval for required changes to the network.

    4. For something attached to the network no approval by network security.

  7. Lee D Silver badge

    "Matt’s company learned something too. “We only install our software on dedicated systems for production environments,” Matt told us."

    About ten years too late, by the sounds of it, but welcome to common sense.

    Literally, what was their main web portal doing running the email tracking too? That's just stupid.

    Kind of forgiveable in a mom-n-pop kind of place, but the second you're into a "real server" then you should be virtualising out to individual VM's with stated purposes.

    Case in point: When I arrived, my workplace had four physical servers. One of those ran finance and, for some unfathomable reason, file shares and print server (no finance integration, including talk-home software), and a myriad other things.

    Replace with the same number of physical servers, virtualised everything, and now run 30+ virtual machines on pretty much the same hardware. Ironically, not only is everything faster, it's more energy efficient, much more resilient, everything is replicated to more places, and you can safely assume that one server does one job (including the hypervisors which do nothing but... VM hypervising).

    There's no way that you should be doing ANYTHING else on a public-facing web server machine. Hell, it shouldn't even be in the same VLAN / network.

    And if you've not pushed your public-facing stuff through an IDS/IPS reverse proxy (also a separate machine), then you're just opening yourself to attack.

    1. Loyal Commenter Silver badge

      Exactly this.

      Apart from anything else, why would you run a public-facing server on the same network as internal systems? If stuff inside your corporate network needs to be accessible from the public internet, it should be done via tightly controlled ports through a DMZ, tied down as much as humanly possible.

      Do you want to get hacked? Because that's how you get hacked...

  8. Zwuramunga

    It is Classified.

    What is? I can't tell you, it is classified.

  9. amanfromMars 1 Silver badge

    Just DeClassified and for Public Record

    Client was silent on critical network details and little things like multi-tenancy

    :-) Their Pleasures to Ensure and Endure to Glorious Release with Immaculate Relief in the Powerful Energy of Ecstasy. .....with Heavenly Travelling Servers of the COSMIC Trail Trialling Futures for Media Presentation of an Unfolding Augmented Virtual Reality ProgramMING ...... which Commands RePlacement of Presents with Search and Research into the Tale.

    And when the SourceRootRoute is Hellishly Heavenly, is the posit here IT be Almighty with Endless Almighty Forces just doing their Sensational Protection Thing in Lead of Attending Sources.

    Be You They? In The Singularity is Such Easily Possible.

    One small quantum step for Mankind, one giant quantum leap for All Kinds into what Treasured Temptations have to Offer and Realise/BeTrue.

    That's noteworthy news, El Reg. Even an Exclusive.

  10. OzBob

    An option sadly missing from a lot of designs

    is the ability to segment off parts from other parts so external technicians can work on it. Too many systems I encounter house multiple functions from several systems on the same box, which make it a nightmare to pass over to someone else without risking compromising the entire lot. Architects who used to be Developers rarely thinks of this, those that were Sysadmins notice a little better.

  11. Aging Hippy
    Alien

    Promotion Logic

    You do a job quietly and it works first time: The job must have been easy.

    You get into a mess, even of your own making, then get it working: This person is a good problem solver.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like