‘I crashed a rack full of servers with my butt’ Welcome for the third time to Who, me? The Register’s new column in which readers ‘fess up to messes of their own making. This week, meet “Digby” and his powerful rear. Digby once worked for a business that started life as a division of another company. He arrived just as the new outfit was getting to his feet and found “the …
I was doing some development on a test server and had not saved my work. An associated program would only run if "Turbo Mode" was turned off (this dropped the CPU clock rate down to match an original 8088 chip). Some tower PCs had the turbo button next to their push-button power switch. I pressed the turbo button to start the other program, but hit the power button by mistake, immediately experiencing the well known "How stupid am I?" feeling. I managed to keep the button in, so the power stayed on. Unfortunately, I had used my dominant right hand when I had bent down to touch the switch. Unable to see they keyboard, I managed to find the keys I needed to save my work, and then type in the shutdown command. Then I drank coffee and taped a cardboard flap over the power switch to stop the stupid person doing it again.
Edit: That is one reason why I really liked the large red power switch at the back of original IBM PCs, it was almost impossible to turn it off by mistake.
Hah, I played around with Smoothwall back then.
The tower case (which was the Smoothwall) was conveniently next to my work desk.
Daughter was crawling around all over the place, she saw the tall, white box with blinkenlights, and she did a gefingerpoken at the reset button.
Daddy was NOT amused :)
I disconnected the actual reset and power button cables from the motherboard, then set the motherboard's BIOS settings to turn itself on when power was restored.
Then it was simple - shut it down, it'll power off, and switch off at the wall. When I want to use it again, power on at the wall and the PC'll start up again.
Little fingers still poked the buttons afterwards, but nothing happened. :)
I've got you beat on the little fingers department. Way back when, my son was around 2 years old and the wife's computer was in the basement next to my lair of untidyness. Which she did not appreciate at all! Anyway, I had an old PC (gateway? HP?) where the case slid forward to add/remove stuff inside it. I had canibalized it for some parts and left it on the floor *mostly* closed up. Wife is working with the nipper playing under the desk. Then all hell breaks loose because he pushed in the power switch and the flap of spring steel caught his finger and wouldn't let go. Normally he could have pushed it all day until the cows (or I) came home. But the button popped out and he got trapped.
I got a hysterical call from her and since I was 30+ minutes away and she's not really the techie type, esp under pressure like this when the first born is wailing his head off... I told her to call the local police for help. They came, got the finger out without loss and I got a royal reaming when I showed up just after they had left.
Consequently, her computer was moved upstairs, and I had to do a major cleanup of my mess.
Now the kid is as tall as I am... memories.
and taped a cardboard flap over the power switch
This was a standard part of the installation procedure for some types of cases through most of the 90s. There were some very poor designs. I think the worst I saw was a Tiny midi-tower which, when kept underneath the desk, could be accidentally switched off with a knee.
"I think the worst I saw was a Tiny midi-tower which, when kept underneath the desk, could be accidentally switched off with a knee."
There was a horizontal desktop PC case, monitor on top, that had the power switch on the front - low down and projecting. If you pushed the bulky keyboard too far back on the desk - it hit the switch.
"I think the worst I saw was a Tiny midi-tower which, when kept underneath the desk, could be accidentally switched off with a knee."
@Rich11
Much more recently that there is the Compaq 500B MT about 2010 protruding power switch on the side knee height. At least you can change the function of a soft power button to something other than shutdown.
Anon because they are we still our main PCs
a cardboard flap over the power switch
A "molly-guard". Don't they have the Jargon File where y'all work?
This was a standard part of the installation procedure for some types of cases through most of the 90s. There were some very poor designs.
Dell, a consistent producer of systems with crap design, had a number of these. I had a Dell tower system with a hardware reset button positioned right below the CD-ROM eject button.
"I managed to keep the button in, so the power stayed on"
I did this a few times, back when computers had Turbo buttons and switches were Real Switches instead of these noncy soft-switches they are today. The trick was to very quickly let go of the button, then instantly slam it back in again. Usually, enough residual power resided in the capacitors of the PSU to carry it through, and the computer would stay on.
My main PC is one of those with a power button on the top of the case, which is fine until the cat jumps up and lands her heel on the power switch before sitting down and turning things off with me in mid-work! A strategically placed book seems to have fixed the "problem" now.
I've done the "inadvertently press the power button, realise I'm an idiot" thing as well - I was supposed to press the CD-ROM eject. I didn't immediately release the button, and was able to reach a piece of paper that I folded one handed then wedge into side of the power button until I could cleanly shut the machine down.
Our work PCs have one of those nice setups with a bunch of USB ports, headphone and microphone jacks on the top edge of the case. Very convenient. Except the power button is directly below the headphone jacks so when you're poking around under the desk to get the cable in there's a decent chance you're going to have windows asking if you want to start saving things before shutting down.
Yep. Sounds all in order. Elementary commonsense: you don't leave an important switch where someone might accidentally hit it, regardless of their girth, clumsiness. Or indeed if they're wearing big loose clothes that swish. And in a server room, you *also* give good care to your cables. Hence some of those neat little inventions like recessed switches. Didn't you learn the principle when you were little and your parents told you not to put your glass right at the edge of the table?
Once again, the protagonist seems to be innocent. When are we going to get someone owning up to a proper f***up?
Or taking ownership of a new server room, finding that the door release button was right nest to the emergency power down button.
Despite pointing this was a bad idea to the customer, it was only after the second time the room was unexpectedly powered down did they agree to spend about £150 to move one of the buttons.
When I worked in the server support team for my local health board about 25 years ago, an engineer was called out to fix a faulty open reel tape drive for a development server that sat on a table in the data centre. It was quite a heavy old bit of kit and the table was next to a wall, so space was fairly limited. The engineer picked up the drive and, apparently without thinking through the manoeuvre he was about to make, swivelled round and bumped the emergency power off button on the wall with the back of one of his hands. The data centre was then a hive of busy people, trying to bring up the ICL mainframe, comms kit and multitude of midrange servers that lived there. Once he realised the chaos that was unfolding, said engineer started to sweat profusely as he continued his task of repairing the rarely-used tape drive. Some time later, once services were restored and things reasonably back to normal, a wooden box was placed around the power off button, with a hole cut in it so that it could be pressed when actually required.
Similar situation happened to me at the Home Counties-based headquarters site of a well-known British aerospace company in the late 80s - we had to have the electricians in to do some work on one of the PDUs and, as they went to leave, one of them says "Is this the exit button?" and before we could yell "NOOOOO!!!!!!!!!" he pressed the big red emergency power off button...
We weren't sure whether we should have been relieved that we didn't need to spend the next few hours rebooting various mainframes (the IBM and VAXs wouldn't have been much of a problem, the biggest headache would have been the ICL 2966s...) or extremely p***ed-off that the Emergency Power Off - intended to save our lives if anything went wrong like someone connecting themselves to 415v AC, remember - did absolutely nothing.
Nada.
Not a gorram thing.
We went with p***-ed off, and the Data Centre Manager made sure scalps were collected from the muppets at fault...
"it was only after the second time the room was unexpectedly powered down did they agree to spend about £150 to move one of the buttons."
Vs about £6 for a flip cover?
http://www.boshisafety.com/a/products/4/2015/1015/275.html
Lots of other examples around.
Schnieder even make a transparent one: https://www.schneider-electric.ae/en/home/house-electrical-products/full-time-weatherproof-protection/weatherproof-socket-cover-transparent.jsp
How many times would I told an user to "keep on saving your work, F2 (Turbo Pascal's shortcut to save) will do it"... only to have said user ignore my advice blithely, carry on typing a beautiful and perfectly working program, only to have the PC crash.... (It was my experience with programs and PC's to keep on saving frequently, especially when doing Turbo Assembler stuff. Back then there was no virtual machines).
v2 of the same program was buggy and full of errors. Shame. NOT.
I have lost count of the number of people to whom I have given PCs - and usb back up drives which they never used. Muggins was then expected to recover their data when they had a problem.
For most people it takes a while before paranoia becomes second nature. My friends laugh at me for having at least an idea of the potential "Plan B" (C,D,....) when I do things. They are learning the hard way....
From Schlock Mercenary:
Commander Kevyn: So we're supposed to evacuate the rest of them?
Commodore Tagon: Sure, that would be a great solution. Let's call that "Plan A."
Captain Murtaugh: Also known as "Ablative armor that probably won't protect Plan B."
Captain Tagon: Plan B always takes the bullet for Plan C. That's why the alphabet has more than three letters.
"They do have a plan B. You."
That's usually their Plan A.
It's like Cassandra and the usual Greek gods' attitudes. Gifted by the gods to warn everyone about a foretold disaster - fated to have to suffer the consequences with them. The Ancient Greeks knew human nature - or at least their story tellers and playwrights did.
I have lost count of the number of people to whom I have given PCs - and usb back up drives which they never used. Muggins was then expected to recover their data when they had a problem.
I can be smug here - I did buy the kids USB disks - and then made a royal pain in the arse of myself b buffing them to use them! Paid dividends in one (of three) who has evangelised ever since...
This post has been deleted by its author
"v2 of the same program was buggy and full of errors."
VBA in Excel 2010 has an interesting quirk. If you Ctrl-Z a few times to undo an edit - it regularly corrupts whole swathes of the module. It deletes unrelated chunks of statements in a totally different part of the module - which you only discover when you try to compile.
After any significant change I make copies of the project directory with a yyymmdd_hhmm suffix.
Using the same technique on VB.Net hit an unpredicted problem. Norton security*** regularly quarantines one of the project files that is a .dll type - because it is "new" and has no "reputation". You then have to do the arcane ritual to reinstate it and make it "safe" for the future. According to Norton support that only protects for that specific version's directory name.
This doesn't happen all the time. It was horrifying the first time to see it purging the e-sata back up disk of all instances of that file going back years.
***Norton works for me - and often cleans up people's disks who have used a different AV. YMMV
Me: Can I please have git installed on the Project Millstone server?
My Boss: What's git?
Me: It's a version control and software repository tool. Our vendors use it. I could use it to comply with your order to "stop using .old files to save stuff that gets changed, dammit!"
My Boss: No. Everyone must use the Microsoft version control tool we have.
Me: Okay. What's it called?
My Boss: Dunno. Microsoft something-or-other.
Me: Where's the documentation for it?
My Boss: Dunno. On a share somewhere. Maybe somewhere on sharepoint.
Me: Who owns it? Who's the SA?
My Boss: Dunno. One of the windows guys.
Me: Is there training for it? Can I have some please?
My Boss: No. The budget is all used up.
Time Passes In Month Chunks
Me: I've written my own version control. I run a script that just appends ".old" to everything of the same name before you edit.
My Boss: That's the opposite of what you were told!
Me: I know. I was faced with insurmountable difficulties in implementing your methodology.
My Boss: How in hell do you tell one so-called version from the rest? what happens if you need to reinstate a previous version?
Me: You figure out how many versions you need to go back and count the ".old"s.
My Boss: Whut?
Me: It works very well.
My Boss: It's stupid! And you've soft-linked this "script" to all the servers and filesystems, I'll bet.
Me: Of course not!
My Boss: Well that's something at least ...
Me: I simply copied the script into every directory on every unix computer we own.
My Boss: WHAT!
Me: Very quick and cheap, if a tad cheerful. Works though.
My Boss: Get rid of it!
Me: With pleasure. Can I have git please?
My Boss: No. Everyone has decided to use Toad's built-in version control. I don't know what it's called before you ask.
Me: I do. It's something called "git".
stretching their legs out under the desk was a frequent cause of problems as toes pulled out power leads, keyboard and mouse cables variously.I had told them that proper cantilever desks with cabling trays were required rather than reusing the old 4-leg plain top wooden tables that had been there since 1953.
Good diagnosis, bad solution. Yes, cabling should be kept clear of the meatware's legroom. Real-life computer desks get that horribly wrong by restricting the legroom: sadly I hadn't heard of "constructive dismissal" when I was forced out of office-based working by desks that forced me into postures that were incompatible with my back. The right solution is to keep the legroom but provide alternative safe spaces for cabling!
On a few days backfill work for a insurance co (When they were still called Norwich Union), a lady wrapped her leg around a power bar\cable & brought down a entire row of computers.
Having rectified the issue, she then went & did it again.
Thus prompting a better solution in cable management.
When I was at Uni, we had diskless Sun SLCs as our workstations. There were two SLCs which had massive 1GB HDDs to serve the other SLCs in the room. Unfortunately, it was very easy (by accident) to knock out the ethernet cable from the SLC with the HDD and bring the room to a standstill. (And I'm ignoring the joys of 10Base-2!)
Well if you're running your racks down a wall, then I suspect getting to the back of them may prove to be a little tricky as a starter for 10! (But in principle, I agree that L shaped configurations of racks in rooms is entirely plausible, feasible and indeed, practised in some areas.
As the rather clumsy protagonist responsible for this submission, I will call your BS claim as the story is indeed true and happened to me roughly 5 years ago. Perhaps a bit more detail would help you visualise things as there is the odd minor inaccuracy in the story as it happens:
The server room in question was a small, square room. As you enter, the main server rack is directly facing you (with a rackmount-capable UPS sitting next to it at the left of the rack). Turn 90 degrees to the left from the door and you have two more racks. The rack in the corner of the L shape was the comms rack and was facing directly towards the UPS and the side of the main server rack. Hence when patching phones in the comms rack, the UPS and main server rack were actually directly behind me (not next to me). Crappy diagram time:
< /\
<
Hence, crouch down in-front of the comms rack, arse protrudes out and neatly catches one of the trip switches at the top-rear of the UPS behind me. The cabling issue then came when I poked my head around the back of the rack to check the power cables running to our Oracle server and the movement of the cables dislodged the poorly-seated feed into one of the distribution units in the rack.
Not my cleverest moment, even if sensible cabling would have prevented an outage...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
