back to article A ghoulish tale of pigs, devs and docs revived from the dead

"My pages have come alive!" accused one from my pod of guinea pigs, unfeasibly. This (as it turned out) not-so-singular anecdote from my murky professional experience working on large-scale content management projects came to mind this week while reading The Reg. It was triggered by that recent Who, me? story about the little …

  1. Anonymous Coward
    Joke

    So...

    The whole experience was 'Just like you imagined' it would be ?

  2. Bob Wheeler

    "every smooth operation is a cupboardful of IT bods frantically conducting a real-world and real-time disaster limitation exercise"

    Shss, don't tell everyone, it our little secret.

    1. Chairman of the Bored

      Secret is out

      @Bob, one of your mates must have spilled the beans. I asked an IT manager how things are going...

      IT: "Like a duck"

      ME: "What the fsck?"

      IT: "You see us gliding gracefully across the lake. Underneath the feet are paddling like hell"

      1. Anonymous Custard
        Headmaster

        Re: Secret is out

        That's swans.

        Ducks just bob along peacefully and then suddenly disappear completely by diving underwater for no apparent reason before popping up randomly a few feet away.

        Although I've known projects where the latter would actually be more of an accurate description...

        1. CrazyOldCatMan Silver badge

          Re: Secret is out

          Ducks just bob along peacefully and then suddenly disappear completely by diving underwater for no apparent reason before popping up randomly a few feet away.

          Err.. not mallards. They don't dive at all (what they do is called "dabbling" - which means their head is underwater but their tails are stuck up out of the water, pointing at the sky). Some ducks (crested ducks for example) do dive but not for very long.

          1. Spacedinvader
            Happy

            Re: Secret is out

            Nobody told this one - https://www.youtube.com/watch?v=M-KqMsUpekU

  3. Chairman of the Bored

    Love it!

    You've confirmed some of my deepest, darkest suspicions about what really happens when I grovel and beg for old data to come back.

    Sometimes, though, it comes back on its own. Like my night of the living dead:

    Supposedly in this one govt agency emails - once purged off the exchange server- are gone forever. Of course that's utter BS because if management wants to pop you for misconduct they can suddenly get not only what you've written, but even stuff you only dreamed of after a few stiff drinks. Mere mortals do not get their lost emails restored.

    Imagine my surprise when I opened up outlook after a prolonged server outage and magically found my mail spool transported ten years into the past. Instead of my working set of spam, I was seeing one specific day in 2005 all over again.

    Here's the part that keeps me awake: email on the top was in all caps, title screaming: "YOU ARE LATE FOR A MEETING!" Problem is the guy who sent it - one nasty piece of work - had died shortly after sending me that.

    So where the hell is he, and am I late for a meeting upstairs or down?? Is the weather uncomfortably warm??? Speak, ghost of the server, speak!

    1. Bob Wheeler

      Re: Love it!

      Dear Chairman of the Bored

      I have to inform you that not only the 'pages are alive', that 'emails get resurrected' but also there are ghosts in the machines - how else do you explain these damn systems working one moment, then failing then working again without anyone even touching them. I swear the software is taunting us.

      In addition to that, it has been known for many years in the highest level of BOFH'dom that the servers also have gods that not only require, but demand blood sacrifice from any mortal working on them in the machine halls.

      yours faithfully

      IT bod in chief

      1. Anonymous Coward
        Anonymous Coward

        Re: Love it!

        "[...] but also there are ghosts in the machines[...]"

        A large interactive system in the 1970s. The main administration console suddenly types out "I am the ghost of the machine - I have changed your password". Unfortunately the prankster had made a programming error and the message repeated every few seconds - preventing the admin team from logging in.

        He was fired the next day.

      2. Chairman of the Bored

        Re: Love it!

        That's dark and scary. Think we need to start drinking immediately.

      3. Helldesk Dogsbody

        Re: blood sacrifice

        We have found a certain degree of success in utilising the blood of users. You may just require more of it thereby creating a win/win situation. The appropriate oath also appears to still work, you know the one: "Why won't you WORK you &*%!ing piece of £$(^!!!" at around 120 dB.

        1. CrazyOldCatMan Silver badge

          Re: blood sacrifice

          We have found a certain degree of success in utilising the blood of users

          Sadly it seems to require the blood of virgins and they are a diminishing resource round here..

          1. Bob Wheeler
            Mushroom

            Re: blood sacrifice

            @CrazyOldCatguy

            "require the blood of virgins and they are a diminishing resource round here.."

            Nah, all of our PFY's are virgins, ... oh wait, you meant the female kind....

      4. handleoclast
        Coat

        Re: Love it!

        @Bob Wheeler

        there are ghosts in the machines

        Nah, it's the Phantom of the Operating System.

      5. CrazyOldCatMan Silver badge

        Re: Love it!

        ghosts in the machines - how else do you explain these damn systems working one moment

        Mostly, for me, it's the other way round. Stuff won't be working at all until I go and stand next to the computer.

        At which point, it'll work perfectly.

        Maybe it's my sparkling personality (bought on Ebay) or maybe They Know that in my pocket is an etherkiller and I'm not afraid to use it..

      6. Doctor Syntax Silver badge

        Re: Love it!

        "demand blood sacrifice from any mortal working on them in the machine halls."#

        That's why the metal-work has sharp edges.

    2. paulf
      Terminator

      Re: Love it!

      @ Chairman of the Bored "Problem is the guy who sent it - one nasty piece of work - had died shortly after sending me that."

      Was the death, cough, an accident? Did you go all BOFH on him by the top floor windows that have the broken catch as retribution for him typing all in caps?

      1. Chairman of the Bored

        Re: Love it!

        Trust me, the thought crossed my mind. Actually I was more focused on how to make the body disappear. Question - if you shoot a guy for ranting in all caps ... Do you need to use a silencer?

    3. David Nash Silver badge
      Facepalm

      YOU ARE LATE FOR A MEETING

      Because email is famous for getting someone's attention in real-time

      1. Doctor Syntax Silver badge

        Re: YOU ARE LATE FOR A MEETING

        The best reply is "Yes I know. I'm busy working through all your emails about meetings".

    4. Mark 85

      Re: Love it!

      Working in the trenches in IT is like living in real time two movies at once: "Groundhog Day" and "Ghost Busters".

  4. chivo243 Silver badge

    baked bean wrestling

    Watching the Ann Margret scene out of Tommy before bed again, Dabsy?

  5. Anonymous Coward
    Anonymous Coward

    Scarily true to life...

    I read this as I am trying desparately to build a USB drive with the latest UEFI version for my three Lenovo System X VMWare hosts to unfuck the fuckup that was Lenovo's first iteration of Meltdown/Spectre back in January. Just trying to get the latest UEFI from their site to a bootable device is an effort of ridiculous proportions - the default option gives you an older version of it rather than the current one.

    Meanwhile, all my VMs are running on two hosts, and whilst not quite hitting resource limits, I've not got the comfortable feeling I'd like to have...

    1. Anonymous South African Coward Bronze badge

      Re: Scarily true to life...

      And that is the problem with virtualization - you can wipe out more than one server in just one go... and data recovery is well nigh impossible.

      Backup, backup and backup.

      Just not to Tier 4...

      https://www.theregister.co.uk/2014/11/01/bofh_2014_episode_10/

    2. CrazyOldCatMan Silver badge

      Re: Scarily true to life...

      desparately to build a USB drive with the latest UEFI version

      I spent some considerable time recently trying to install Windows 7 on an old 17" MAcbook Pro I had lying around (my wife wanted a laptop to do Excel-type stuff on but simultaneously didn't want to spend actual money on one..).

      Even wiping *everything* on both internal drives and booting from a clean USB or CD install didn't do it. Which is bizarre because (on that self-same Macbook Pro) I used to dual-boot into Windows 7 quite happily.

      So, either there's been a EFI update applied to the MBP that borks the boot process or the latest incarnation of Windows 7 won't work at all on a 2011 MBP.

      Or, knowing my luck, both. Need to find my *old* W7 boot CD. Trouble is, it was done on a cheap CD-R so probably won't be readable either.

  6. Geoff Campbell Silver badge
    Boffin

    Lesson

    Surely there is a lesson to be learned, and it is the remarkably simple one of "Test everything thoroughly before you go live, and don't go live until you know it works"?

    Call me naive if you will...

    GJC

    1. Fading
      Mushroom

      Re: Lesson

      Back in the real world there is time for testing or time for fixing the bugs discovered during testing or neither. Minimal viable product tends to be what is delivered irrespective of development time. This product will lack pretty much all the features that differentiated it from the previous version during the procurement process and if you are very lucky will at least do 80% of what you need it to do.

      Then it breaks and the budget put aside for enhancements rapidly becomes subsumed into fixes and maintenance.

      Then the cycle begins again - lessons may be learnt but as only 2 people were around from the last time and are still suffering PTSD it is unlikely to break the software cycle.....

      1. Dux Leader

        Re: Lesson

        Why did the fist lines in Your comment made me think of C-beams?..

      2. Doctor Syntax Silver badge

        Re: Lesson

        "Back in the real world there is time for testing or time for fixing the bugs discovered during testing or neither. Minimal viable product tends to be what is delivered irrespective of development time."

        This is a lesson that needs to be learned by manglement. It never happens because when you know you're always right, why would you have to learn anything.

      3. Mark 85

        Re: Lesson

        Back in the real world there is time for testing or time for fixing the bugs discovered during testing or neither.

        Usually what happens is "neither". Seems manglement only see the "go live day" as the goal. They'll hammer with "improvements" and "features" to the point that the "testing" schedule gets tossed out the window as no time is left. In an ideal world, if the testing start date slips, so should the "go live date".

      4. J. Cook Silver badge

        Re: Lesson

        Which is why we are stuck with windows 10 instead of a proper rev of windows 7.

        *hides in the anti-nuc bunker*

    2. Anonymous Coward
      Anonymous Coward

      Re: Lesson

      "Test everything thoroughly before you go live, and don't go live until you know it works"?

      As Donald Rumsfeld said - there are always "unknown unknowns". In other words - expect the unexpected.

      The live environment is by definition different from any test environment - no matter how accurately you believe it to have been staged in a dress rehearsal.

      What has often happened in the past is that on the "live" day users start reporting problems as requested. What they fail to tell you is that the problems existed previously but no one had fixed them.

      I remember two major switch-overs.

      1) A large online system was being moved to the next generation hardware - and running a different O/S. This involved virtual machines and all sorts of emulations so that no customer programs needed recompiling. Several of the user sites had resisted the advanced testing of their connections - but had eventually agreed. On the "live" day the reported problems were all previously known to the users on the old system.

      The plan had been to keep the old system as a warm stand-by for a couple of weeks. After three days the customer asked for it to be switched off.

      2) a customer had three mainframes - each with their own comms lines. The project was to use one front end comms processor to allow any terminal to switch between mainframes. The weekend before switchover each comms cable and its port were marked with numbered sticky labels. Different shapes and colours differentiated the three mainframes. The cables were then plugged into the new ports.

      On the day we were ready to move all the cables back if necessary. Only one real fault was reported - and that was for a new comms port that the end user site had refused to allow to be pre-tested. The senior engineer said "I would rap the knuckles of anyone doing this" - as he took the port board out of the live rack and put in a spare. It worked. The removed board had a repair label saying "NFF" (No Fault Found).

      Over the next few days quite a few problems were reported - and they were all pre-existing conditions that the users had previously endured on the old system.

      1. Doctor Syntax Silver badge

        Re: Lesson

        "they were all pre-existing conditions that the users had previously endured on the old system."

        They probably realised they'd a chance of getting them fixed if they reported them afresh.

    3. John 110
      Mushroom

      Re: Lesson

      Unfortunately out here somewhere, we sometimes find ourselves in the situation where the old license runs out before the replacement software is quite ready for primetime - but we have to go live anyway...

    4. juice

      Re: Lesson

      Ha. Haha. Hahahahahhaaa...

      I was talking to a friend recently, who's working for a company which is about to switch some Very Important financial infrastructure into the cloud.

      However, the people doing this work have decided to not tell anyone about this, to prevent the usual "the sky is falling" claims which tend to abound when any changes are made. After all, they've cheerfully proclaimed, everything should Just Work after the changeover.

      Oddly though, the list of people who haven't been engaged in the handover process includes the internal QA team. Nor is there any evidence of a contingency or rollback plans.

      Overall, it does seem like an appropriate time to order some popcorn and get comfy on the sofa...

    5. CrazyOldCatMan Silver badge

      Re: Lesson

      Call me naive if you will...

      Hi naive.

      I presume that you've never worked in a large IT department. That sort of thing is depressingly familiar.

      A combination of restricted resources, senior management unrealistic[1] deadlines and general "if anything can go wrong it will"ness will give you it every time.

      [1] Surely an oxymoron..

      1. JulieM Silver badge
        Headmaster

        Re: Lesson

        "Senior management unrealistic deadlines" isn't an oxymoron, it's a tautology.

      2. Geoff Campbell Silver badge
        Boffin

        Re: Lesson

        For the record, I've been working in and with large IT projects for over three decades now.

        I understand that testing is frequently done supremely badly or not at all. Which is why the lesson I pointed out should be learned by anyone who have been involved in such a clusterfuck.

        Yeah, I'm an idealist. So sue me.

        GJC

    6. Michael Wojcik Silver badge

      Re: Lesson

      it is the remarkably simple one of "Test everything thoroughly before you go live, and don't go live until you know it works"?

      "Always mount a scratch monkey" is simpler yet.

      (N.B. Anyone interested in the corrected version of the scratch monkey story should refer to the first link in the Jargon File entry linked above. Or that link in the previous sentence.)

  7. Elmer Phud

    sympathies

    "i.e. near the chilly windows that didn't shut properly."

    Which is also the same one that the sun mercilessly beams through in warmer days.

    Ah, to live with my friend 'the big shredder' and all the other 'corner' occupants again . . .

  8. juice

    Back in the 90s...

    There was a demo for some of this newfangled internet technology.

    However, as the technology was still a work in progress, the demo involved a certain amount of smoke and mirrors. And the person doing the demo had been given clear instructions about where the smoke and mirrors were placed.

    Alas, said person was nervous and strayed from the script. Cue some furious hackery in the background as the tekkies supporting the demo had to try and fake up the S&M elements in realtime.

    The main tekkie in question is not a small chap; he'd have made a fine rugby forward or viking berserker. As such, when the demo-presenter cheerfully came over afterwards and proclaimed "well, that all went well, didn't it!", he very nearly got throttled...

    1. Anonymous Coward
      Anonymous Coward

      Re: Back in the 90s...

      Same thing in the early 1970s. A roll of pre-punched papertape in a teletype in the mainframe data centre. Ready to be switched back-to-back with the one in the remote press/bigwig hotel venue. A very experienced operator at the demo end - who knew not to deviate from her script.

      For the first time ever the system ran without fault - until a few minutes after the demo ended. It was several weeks before such reliability became standard.

    2. David Nash Silver badge
      WTF?

      Re: Back in the 90s...

      "S&M elements"

    3. CrazyOldCatMan Silver badge

      Re: Back in the 90s...

      fake up the S&M elements in realtime

      I hear there's quite a market for that in some parts of Soho..

  9. Voidstorm
    Facepalm

    This is what happens when...

    ..."Management" publicly promises a live date in six months, before any problem analysis has been done. Or even asking the IT department how feasible it is.

    Having worked for a major UK Telco in their development department, I got really sick of that shit; it precludes anyone doing a decent job, and leaves you with "the only thing we have time for is the shittyest quick fix possible".

    1. Anonymous Coward
      Anonymous Coward

      Re: This is what happens when...

      I have a bit of code here at work with a comment before it that says (and I quote!), "A seriously egregious hack, but we have to do it..." Of course, I'm the one who wrote it... and it really had to be an egregious hack in order to function properly.

      (AC 'cos actual comment in actual code from $employer)

      Of course, my favorite is "TEMPORARY, I HOPE HOPE HOPE" in the final version of the Apollo 11 sourcecode. Apparently it worked.

      1. GrumpenKraut
        Happy

        Re: This is what happens when...

        Here it is: LUNAR_LANDING_GUIDANCE_EQUATIONS.agc (link is to github.com). See lines 179 and 180.

  10. Potemkine! Silver badge

    Plan F

    Plan F is the one in which you screengrab thumbnails of lost documents from the CMS browser window

    Nice. Our plan F is to kill all witnesses, throw the bodies to pigs (after removing teeth) and pretend nothing really happened in our space-time continuum..

    1. Bob Wheeler
      Angel

      Re: Plan F

      @Potemkine

      Some plans need constant attention and tweeking.

      Your Plan F is faultless and is the ultimate expression of BOFH'dom

      1. imanidiot Silver badge

        Re: Plan F

        Unless he also has a solid plan to dispose of the teeth (and the blood pulling teeth, even from a corpse, no doubt involves) it's not all that solid.

  11. Daedalus

    Don't panic

    If you'd looked in the room cupboard next door you would have seen an infinite number of monkeys working on a script for Hamlet.

  12. Anonymous Coward
    Anonymous Coward

    I've been one of the chaps in the metaphorical cupboard. To say it isn't fun would be an understatement.

    1. allthecoolshortnamesweretaken

      Is this where the "S&M elements" mentioned above enter into it?

  13. GBE

    Allways Educational!

    I have to admit that I always end up learning something from Dabbs's columns. Often it's a slang phrase or pop-culture reference unfamiliar to us USAins. This time it was "baked bean wrestling". Somehow, I've spent more than a half-century on this planet without knowing that baked bean wrestling was "a thing".

    1. GrumpenKraut
      Pint

      Re: Allways Educational!

      > ...learning something from Dabbs's columns.

      Yep. And, for me, also from the comments.

      Beer because (1) of that and (2) it's beer o'clock. Prrrost! -------------->

    2. J. Cook Silver badge
      Pint

      Re: Allways Educational!

      ... googling that was a *bad* idea.

      I require vodka to get rid of that image.

      1. onefang

        Re: Allways Educational!

        Doesn't seem to matter how hard I poke at them with the fork, my baked beans refuse to wrestle. Think I'll give up and just eat them.

  14. Mage Silver badge
    Coffee/keyboard

    dedicated TL;DR shortcut on your keyboard

    Hmm... shouldn't be hard on Linux? :D

    something clever than Ctrl-W

  15. Mage Silver badge
    Angel

    Don't bite the hand tossing you those pearls.

    Even if they look suspiciously like rabbit droppings.

  16. LoopyWif

    Toner

    Someone should invent toner-resistant hair gel for these unique types of catastrophe.

  17. Anonymous Coward
    Anonymous Coward

    I’m spurting ink all over the sheets, but she remains, completely unexplained.

  18. Anonymous Coward
    Anonymous Coward

    Hair Raising

    Someone should invent toner-resistant hair gel for these unique types of catastrophe.

  19. StuntMisanthrope

    The Answer - Chaps, it's invisible, so doesn't count.

    I reckon Dabbsy's been to his first class. Computering Anon.. Step 1. Print Confession. Step. 2 Go To.

    That's it I'm afraid. #socialbutterfly d:-)

  20. Dr_N

    Customer Facing

    "Thinking it rather unfair for a customer-facing support minion to be given the brush-off by fellow support minions sitting safely out of harm's way ..."

    Unfair? Ha, ha, ha. Are you not from around here, Mr Dabbs?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like