back to article Techie's test lab lands him in hot water with top tech news site

Monday morning arrives once more for those of you holding the fort while colleagues are on holiday.So why not enjoy this extra special instalment of Who, Me?, El Reg’s weekly confessional column. Because this week, Vulture Towers plays a leading role in the tale that our brave reader "Victor" recounts. "Perhaps 13 years ago, …

  1. Aladdin Sane

    I wonder if Victor will own up in the comments?

    1. Korev Silver badge
      Alien

      And leave Victor to the err Vultures?

    2. vforvictory

      ...

      You mean owning up on the front page is not good enough?

    3. Anonymous Coward
      Anonymous Coward

      I wonder if Victor will own up in the comments?

      Or will it like the end of that movie Sparticus? Or like "Life of Brian"?

  2. Roger Greenwood

    I would like to know what happend subsequently. What's your vector victor?

    1. Admiral Grace Hopper
      Mushroom

      "They bought their tickets. They knew what they were getting into. I say, let 'em crash!"

    2. Zippy´s Sausage Factory
      Joke

      What's your vector victor?

      Is that... is that a Despicable Me reference?

      Where's the minion icon when you need it?

      1. A Non e-mouse Silver badge

        @Zippy

        I'm guessing it's a reference to the film Airplane.

        1. Sgt_Oddball

          Re: @Zippy

          Roger, Roger...

          1. The Nazz

            Re: @Zippy - enlightenment ahead of it's time

            Rodger Roger

          2. Andrew Moore

            Re: @Zippy

            Okay for clearance, Clarence...

        2. Zippy´s Sausage Factory

          Re: @Zippy

          I'm guessing it's a reference to the film Airplane.

          Oh, of course it is. Obvious lack of coffee error there!

      2. Robert Carnegie Silver badge

        Silly! Minions don't speak... English.

        ...I'm not going to count the word "Banana", used in the Minion epic adventure, "Banana".

        1. my farts clear the room
          Coat

          Banana isn't English

          It's Spanish ....

          according to my Spanish word of the day app.

        2. Zippy´s Sausage Factory

          @Robert Carnegie

          Silly! Minions don't speak... English.

          But Vector does. And he was the villain in the first movie (he stole the shrink ray.) And also, his real name was Victor.

          I didn't spot it was an Airplane! reference (although I should have), probably because of coffee underload.

      3. Crisp

        Re: What's your vector victor?

        You need clearance Clarence.

        1. Andre Carneiro

          Re: What's your vector victor?

          You have clearance, over

          It’s “Clarence Oveur”.

    3. Peter Gathercole Silver badge

      "Surely you don't mean that!"

      "I do, and don't call me Shirley"

  3. Lee D Silver badge

    So The Reg didn't have a redundant backup elsewhere? Just two servers in the same place?

    One day you really need to write an article about the tech behind the site, because sometimes it just sounds quite worrying.

    It's like having a bunch of sports journalists who comment on every match but have never had any involvement with the sports themselves.

    1. Steve Davies 3 Silver badge

      Re: commenting

      It's like having a bunch of sports journalists who comment on every match but have never had any involvement with the sports themselves.

      Isn't that sort of mandatory for most [insert sport of choice] journo's?

    2. IglooDude

      Geographic redundancy, thirteen-fourteen years ago? I'm vaguely surprised they had servers in an actual colo at that point, and not sitting underneath the junior journalist's desk.

      1. Lee D Silver badge

        You mean in 2005?

        Er... yes...?

        "Geographic redundancy" = "buy one each from two different companies, rather than two from the same".

        I'm not talking hot-swap, fast-failover, high-availability, but I'm also talking "website doesn't fall over just because someone pressed a switch".

        Go hit Google for "site:theregister.co.uk" and change the date range from to 2005-2006. They were just as big, just as sarky about downtime, doing articles about grid-computing, etc. etc.

        We're talking the .xxx TLD era, not the dark ages.

        1. DJ Smiley

          That works until you find that the two companies you brought from are both buying out the same colo space.... (i.e. the cheapest one)

        2. vforvictory

          Nah.

          Back then it was not as trivial as it is today - it was somewhat expensive to do properly with some pricey F5 GTM devices.

          Two well setup servers behind a load balancer would give you 99.99% without breaking a sweat so it's understandable why they only had two servers back then.

          At this particular hosting company we had staff operating 24x7 who would run over with a crash cart the second anything played up and we had a room full of spares.

          So really unless you are doing something requiring a lot of reliability, why bother with geo-redundancy, it's more ballache than it's worth in many cases.

          1. FIA Silver badge

            Re: Nah.

            So really unless you are doing something requiring a lot of reliability, why bother with geo-redundancy, it's more ballache than it's worth in many cases.

            This. Sometimes being 'good enough' is really 'good enough'

            Also, best practices have to be married with some level of competence too. I remember a previous company I worked at had their primary B2B site running on a solaris box in the server room with an apache front end hosted by a 3rd party. No real redundancy (other than the text box that could be repurposed if needed) however it pretty much just 'worked' for most of the time as the servers were setup correctly and well maintained. (Sure, we were at risk of serious outage if a fire or flood occurred, but still).

            Years passed and we got acquired by another company, they came and gave us the fancy presentation on their shiny new data center. The site was moved to this wondrous place, hosted across 6 servers with load balancing and all the 'best practice' boxes well and truly ticked.

            Unfortunately it ticked boxes, but perhaps wasn't set up quite right, that site had more downtime in the next 3 months than it did over the previous 5 years. (Mainly due to randomly crashing servers, sticky sessions and load balancers configured to blithely ignore if the servers were actually up and responding or not...) Whilst the site was now more 'disaster proof' our customers were much less happy.

            Sometimes working is good enough.

            (Ironically, we did have full multi site geographically separate DR for our core systems, it's just this new fangled web thing had appeared in the meantime...)

    3. pavel.petrman

      Re: ...top tech news site

      I can't help remembering the classical "And that is a baaad miss" in David Mitchell's voice.

    4. The Nazz

      re sports journalists

      IMO, that is no worse, probably better, than say the BBC's practice of "employing" a multitude, let's call it a magnitude, of ex-pro sports people who endlessly babble on about mundane matters or the blatantly obvious, or their relentless agenda. At exhorbitant rates too.

      The less sport they show, the more they chat about it.

    5. Anonymous Coward
      Anonymous Coward

      So The Reg didn't have a redundant backup elsewhere? Just two servers in the same place?

      Even El Reg started up at some point, and the Net wasn't such a resource intensive place it is now (IMHO mainly because of the large volume of sh*ts trying to abuse your resources for something else).

  4. Christopher Rogers

    Has elReg bought a 3rd server yet?

    1. Anonymous South African Coward Bronze badge
      Coat

      Has el Reg bought a 3rd server yet?

      Nevemind that, has they upgraded the hardware, or are they still running on the original servers?

      Pocketses got lots of nifty new nanoservers.

      1. Montreal Sean

        Upgraded the servers

        Of course they've upgraded!

        Don't you see the shiny new boxen?

        Of course the internals are the same, the new internals having been pilferred by the BOFH for home use...

    2. Nick Kew
      Joke

      Has elReg bought a 3rd server yet?

      No, but they've upgraded from the original 640K RAM.

  5. A Non e-mouse Silver badge

    Top Boss

    [The boss] said, ‘I’m not worried that you broke something, that's expected, the most important thing is how you deal with it’,

    And that is how a boss should react. Shouting at people for making mistakes doesn't mean fewer mistakes are made. It just means more are covered up.

    Admit the mistake and learn from it. I know it sounds like management bull****, but mistakes are fantastic learning experiences. Just don't learn by mistakes too often...

    1. blcollier

      Re: Top Boss

      This. Punishing mistakes doesn't avoid mistakes in the future, it makes people better at covering them up.

    2. macjules

      Re: Top Boss

      I would never shout at my staff for bringing down a client's server, failing to spot a bad code merge request or similar.I just make them buy me a beer for every failed code review that gets notified to me by SonarQube

      Hic

    3. Anonymous Coward
      Anonymous Coward

      Re: Top Boss

      And that is how a boss should react. Shouting at people for making mistakes doesn't mean fewer mistakes are made. It just means more are covered up.

      Not just bosses. One of the reasons I've always had the keys to the server room & the root password is that the admin not only has reasonable trust that I won't screw up, he knows that if I do I will call him, own up, and help to fix whatever I did.

    4. I ain't Spartacus Gold badge

      Re: Top Boss

      I heard an interview with an old actor who did some TV in the early 60s. Back when all TV plays were broadcast live, as recording was fearfully expensive.

      So the director is doing his final pep talk. Firslty the helpful, "Remember if you fluff you lines just keep going, there's no time to stop." Then the less helpful, "and anyone who dries up will never work for me again!"

      Because I'm sure that helped with nerves... What an arse.

      1. A Non e-mouse Silver badge
        Facepalm

        @I ain't Spartacus Re: Top Boss

        On a similar vain, I heard the head of a sports body said to an athlete as the athlete went out for their Olympic qualifying trails: "The future funding of our sport depends on you qualifying for the Olympics."

        Yep, the athlete crashed & burned within seconds.

        A textbook example of how not to give a motivational speech.

  6. Chris King

    We don't kill our own...

    ...we just point and laugh.

  7. Victor Ludorum

    To quote Shaggy

    It wasn't me!

  8. Alien8n

    The timing is perfect

    Literally just had to bring up a server that was refusing to come back up.

    Long story short, new servers being built. To facilitate the new servers we power down a QNAP NAS temporarily, plug it back into the correct UPS and power it back up. At this point I'll explain that the job of doing this is actually that of the IT contractor who has come in to build the server.

    So the QNAP is powered down, plugged back into the correct UPS and all is happy. Until about 10 minutes later when the file server starts becoming non-responsive. Decision made to reboot the file server, but it won't boot back up (it's a VM). Turns out it's complaining about a missing virtual drive.

    We disable the virtual drive, boot up the server and check to see which drive appears to have failed. At which point I notice that the drive that's actually stored on the QNAP is the missing one. And the IT contractor suddenly remembers he forgot to turn it back on. QNAP turned back on, missing drive added back to the VM and all back up and running.

    In case anyone thinks it looks like things are now back to normal, we then find out that the new server can't be built because a vendor who's name rhymes with Hell has installed the wrong HD controller card and the nice shiny hard drives can't be configured in RAID. So project delayed by another couple of days.

  9. Anonymous Coward
    Anonymous Coward

    Yes, I *know*, shut up anyway!

    Fortunately, Victor learned his lesson, so El Reg's servers will never again face the risk of being inadvertently shut down at an inopportune mo

    Broadcast message from victor@highlyimportantserver.theregister.co.uk

    (/dev/pts/0) at 16:50 ...

    The system is going down for halt NOW!

    1. Alan Brown Silver badge

      Re: Yes, I *know*, shut up anyway!

      "Fortunately, Victor learned his lesson"

      Only if he installed Mollyguard

  10. A. Coatsworth Silver badge
    Joke

    So... were there any other sites hosted in the same server?

    The title mentions a "top tech news site" that went down, but the article text only mentions El Reg, so I think something is missing.

  11. Boris the Cockroach Silver badge

    I wonder

    if the downtime was any longer that el-reg would have set the BOFH on 'victor'

    Lift shaft accident or fell out of the window...... the world must know

    1. Mark 85

      Re: I wonder

      Well, he lived to tell the tale..... this time. If he had to drive down the road a bit to get his coffee, we have had a different ending involving lifts, windows.

  12. Andrew Barr

    I once included theregister.co.uk in one of our proxy blacklists and we also took it down for about 15 minutes. Once we realised what we did we removed it and it came back up, phew!

    I hope no one else noticed.

  13. The Central Scrutinizer

    Shirley you can't be serious?

  14. swm

    And then there is Major Major Major.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like