Sysadmins: Your best tale of woe wins a PRIZE I'd like to share with you the plight of a good friend of mine. He's a systems administrator for a mid-sized American accounting firm. His story isn't particularly remarkable, but all the more important because of it. Bob has a bachelor's in computer science, and a lot more experience than I'd normally expect someone clutching …
This did make me laugh, cheers!
I occasionally have a similar problem when having to bounce through too many machines to get at the one I want (all Windows, combination of RDP and Dameware), working out which one has caps lock or sticky keys switched on can be a pain!
"I occasionally have a similar problem when having to bounce through too many machines to get at the one I want (all Windows, combination of RDP and Dameware)"...
I've got a mixed language combination of RDP bounces which won't let me get at things like ~, [, ], \.
And folks wonder why my command prompt has these weird characters in it :-)
@moonface
There are times that I hate those who mandate "special characters" in passwords.
I'll hazard not one of them has had to use the wide variety of foreign language keyboards that I do.
And don't even mention "dead key" keyboards to me...
And that's assuming that the login prompt is configured for the same keyboard layout as the user session.
Mine's the bag with the US keyboard in it.
UK Apple keyboards don't seem to have a key for #, which makes logging in difficult if you have one in your password.
And speaking of non English keyboards, Turkish keyboards have a selection of letters which look like, but aren't, English characters, which also makes typing in passwords difficult, should one be trying to check ones gmail from a hotel.
You can get it through a key-combination (right-alt 3, I think? I can't remember without actually being sat at the keyboard), but this is rather non-obvious. No user would guess that, I had to look it up. It also doesn't work when I'm running windows and RDPed into another windows server - whenever I want to join a channel on IRC, I have to copy-paste a hash from somewhere.
Sounds very familiar to the time I had remoted onto a virtual session hosted in India. Had the user on the phone and was desperatly trying to find the local keyboard setting for ". I was unable to and then spent a good 10 minutes trying to convince the user that all I wanted was for them to type " into a command prompt. In the end I email the " to them and copy/pasted it in.
We once purchased a server with Windows Server pre-installed. By the time it was delivered plans had changed and it was reformatted to run Linux. Three months later it broke down. Looked like a simple failed PSU to me, but it was still on warranty, so we called for an engineer.
Some hours later he told us it was OK again and left at a trot. We were surprised that he hadn't left it powered up, and dispatched someone to the machine room to boot it. But it was booted ...
and once again running Windows! The muppet thought GRUB was a hardware error, so he reformatted the disk array and reinstalled Windows. Thank heaven for backups, and that it wasn't desperately mission-critical.
You don't have to outsource to India to get muppets.
I was called in to fix a workstation in an old Victorian basement bit of the site. Quad 140W Opteron thingy. Expected to have a certain amount of fun sourcing a beefy enough power supply. But the lights were on and fans were whirring.
When I took the cover off, a very strange burned organic smell assulted my nostrils. A few seconds later I found a dead mouse with its head wedged between the fan blades and one of the heatsinks. I hope the poor wee thing's neck was broken in an instant. I fear otherwise.
After the mouse was removed and the CPU allowed to cool down, it rebooted without a hitch. The mystyery remains, how did a mouse get into the case? There was no hole anywhere near big enough anywhere in the metalwork that I could see.
What a great story. I work for SolarWinds and would love you to enter our contest, but unfortunately, we are not yet able to open our contests up internationally. We're trying to work that out with legal and hopefully will be able to offer our contests to all soon. For those of you who do live in the U.S. and would like to enter, here is that link: http://thwack.solarwinds.com/community/solarwinds-community/sysadmin-day-contest.
Dunno who downthumbed you. I did pop by and submit a story, and got some rather nice friendly PM's regarding it - seems like a nice community over there. Not being able to win the competition is kind of a bonus anyway - the price is something Apple made called an "iPad" or something. Pretty sure I wouldn't know how to use it.
This post has been deleted by its author
I'm allegedly a developer... (admittedly one with a CCNA and an interest in system administration, Linux and other general geeky stuff)
Except i am charged with Running exchange, the cloud platform, the virtulisation stack, all of the servers and desktops, the helpdesks (yes 2, as a so say manager was beguiled by a nice mug and stickers, didn't even hold out for a pen drive!) , documentation, router kicker (crappy dlink wireless router from PC world powers the office, and dont get me started on the ADSL line!!!!), 1st, 2nd and 3rd line support, explaining basic concepts to the "technical" director (yesterday talking to a potential client his response was "we send some HTTP", when asked a question about SSL) oh and taking the flack for when dev deadlines are missed for all of the above, well here are some WTF's from this job:
1) We upgraded to SBS 2011 becasue the MD bought a mac book and couldn't connect to exchange, said mac now lives on a shelf gathering dust, Migration from SBS 2003 disnt work and we spent 3 weeks with a partial service.
2) Exchange is hosted locally on an ADSL line which has a top down speed of 5Mbs, on a router which crashes 10 to 20 times a day, average mailbox size is 12Gb and the suggestion of archiving went down like a lead balloon and was forced to revery back to all you can eat inboxes.
3) one of the helpdesk servers was a windows XP box bought in 2002 which died last week, that was
fun to fix (now on a proper server)!
4) we have a mixture of 1 to 10 Gig ethernet to every desk yet the MD insists on people using wireless to said crappy router because wires make the place look untidy...
5) It took 6 months of nagging to take the UPS out of the empty comms rack and install it in the server rack
6) The working helpdesk has been replaced by a cgi based web app which crashes when more than 5 people try and access it it also uses a VBScript based file upload and a scheduled task inplace of a service and requires administrative permissions to save to the file server, suffice to say new clients dont have the option to add attachments to support calls (we publicised the "upgrade" to our clients most are not impressed!)
7) A mezzanine floor was removed, but a comms cupboard was left 20' up a wall, i needed to replace a switch in there, new switch wouldn't slide in shoved it harder heard a crunch then a loud buzz, there was a bloody wasps nest in the cupboard!!!! queue rapid decent down the ladder and no network to half the building until the exterminators had been!
I could go on but the phone is ringing and someone needs their hand holding...
I Love deadlines its the whoosing sound they make when they fly past....
Are you me from the future?
our sbs2003 -2011 migration hasn't happened yet, but the cupboard and wasps nest thing sounds familiar.
Just got back from holiday and found:
one of the HD in our RAID array has failed and the guy called in just took the faulty disk out without replacing it.
The cooling to our server room was switched off because it was too noisy
the "internet has been slow" so someone has been hard rebooting the servers every day by unplugging them.
My work area has been "tidied" by putting all of my components and tools onto the warehouse racking at ceiling height.
You tried to migrate straight from SBS2003 to SBS2011? You poor, poor bastard. I spent two weeks once on a 2003-2008 migration -- eight of those ten days with MS Product Support on the phone eight hours a day, for which service I ended up paying not one thin dime, and if you know how MS PS support works then that tells you all you need to know. (If you don't: Unless you have a support contract, which costs $stupid, there's a per-call incident fee of merely $silly, which is waived if the cause of the problem turns out to be Microsoft's fault. They waived it.) And I've got a 2008-2011 migration coming up this weekend which I am sure will also be a nightmare. Straight from 2003 to 2011? I'd rather have a root canal.
I could go on too, but there's a server to bring back up and a couple things to set live before the day's work proper starts off -- back to the coal face...
It wasn't fun i can tell you that, and i have had a root canal before, was definitely more enjoyable, lots of painkillers straight on the nerve :D..., 2008 to 2011 is meant to be less painful but still a royal pain...
Best bit is the one unfixable problem is that the one person in the company who needs access to every ones calendar is the one person who's account wont allow the permissions to be set oh the joy, tried explaining to "management" that exchange nevermind system administration is a full time job in its own right oh well u can but try...
A PC wasn't working, looked like PSU fault. Took it back to my office to check and repair. Plugged in. Pressed power button. Very loud bang. Small amount of smoke. Lights went out. Oops.
Outside office, also no lights. OOPS.
100 yards down corridor, still no lights, lots of people peering out of their offices asking WTF. OOOOOPS.
Somehow, the PSU fault hadn't taken out the 13A fuse in the plug (yes, it should have been a 5A fuse, but it did have the mandatory safety-tested passed-by sticker). Not the 30A breaker on the circuit either. Nor the 100A breaker above that. No, when the electricians finally located the fault, it was a 180A fuse in a box high up on a wall that had probably last been looked at when it was installed in the 1920s.
It took a lot of phoning to source such a monster fuse and they paid for a motorcycle messenger to bring it down to London from Leeds. As far as I know it's still there. Too much effort to schedule replacing it with something modern before the next time (in the 2060s? )
The PC was fine after a PSU transplant.
You don't need complicated things to make a FUBAR. The simple things are also out to get you.
I could also tell you about the exploding substation and the need do do a tap-dance to avoid getting burned by globules of molten copper pouring out under the door, but that's got no IT angle at all.
"it was a 180A fuse in a box high up on a wall"
That's a 6 inch nail isn't it? Or would be in the BOFH stories...
"It took a lot of phoning to source such a monster fuse and they paid for a motorcycle messenger to bring it down to London from Leeds"
Take a note of your nearest B&Q for the next such outage :-)
When I was a callow youth I worked as junior shit-shoveler at a medium sized engineering firm. One afternoon there was an almighty bang and the whole site went dark except for the emergency lighting. It seems both the main HT connector fuses had blown when one of the company sparkies grounded something he shouldn't have (and was very lucky to be alive). Anyway, need new HT link. Head sparky starts making calls but none to be found this side of the country. Nearest replacement is 1000km and two days away. No mobiles back then, so all this had to be done in full public view in the front office, with a growing and restless audience. Management by now is soiling it's collective self about downtime and missed deadlines.
Cue junior sparky and a "hang on lads, I've got an idea" moment: one piece of one inch copper pipe, two blows with a hammer to flatten the ends, punch two holes to fit over the terminals and voila! We are back in business...
10 months down the track we are still trying to give us money to replace our primary (read only) flile server. If it powers down it will not come back until the following routine is followed:
Unplug cable, pop lid, remove BIOS battery, bridge reset, insert power cord, remove cord, unbridge, replace battery, close lid, power up, reconfigure the BIOS again.
Combined with the fact that we are still trying to also get money to replace the faulty tape drive. It goes for a couple of days then fails with unknown HW error. Only solution is to power down drive and server, power up drive, wait, power up server. Sometimes it works first time, sometimes takes a few tries. Our last successful backup was 9 days ago.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
