Nic says he and his boss shared what sound like surprisingly uninhibited hugs at this point
Hahaha - love it. Nice one.
The On-Call inbox is full to overflowing, so we felt like letting another reader's tale of late-night weirdness into the wild. In this tale, reader “Nic” told us that “Back in the 1990s when being a web designer was a job and ISDN was considered hot stuff I worked as a sysadmin at an outfit strategically located above a guitar …
Quite probable, once it's started acting up. The point of freezing/hitting it is to get it going long enough to back it up. If it seems to be behaving after that, maybe leave it running until a more civilised time to do a rebuild.
I suspect the story has been shortened, but reinstalling the OS on a dodgy disk seems like a waste of time, especially if you don't know at that point that you have a data backup...
Unless you're on triple-time for working Christmas day, of course ;-)
If it was the office of an ISP in Denmark Street, I remember a job interview there back in 2000 or so, and I think they were Sun based at that point. One of the selling points was the cool location and listening to people playing guitars. Turned the job down as a chunk of the renumeration was in shares... Still, they still seem to exist, so it mightn't have been a bad thing to accept, but the tide was certainly turning at that point!
This post has been deleted by its author
Indeed we discover he was providing much longer-lasting pleasure to himself, upon which the kibosh was put, like virtually all pleasures, by a ringing phone.
I'll start:
Dashing through the snow
the drunk two wheels conveyed
through the streets he roamed
wailing all the way.
There's even a Christmas miracle in this story, because Nic says that come Dec 27 his budget was increased by the cost of “a real system with proper redundancy and backup processes.”
Why - they managed to recover from the situation without it - so we clearly don't need to spend all that money on what you call 'proper' systems...
The decorations will be up in the shops by the end of next week.
...clashing with the Hallowe'en decorations and similar 'consumer opportunities'.
Confuses the hell out of me. I can never remember which undead monster we're supposed to be worshipping.
I can never remember which undead monster we're supposed to be worshipping.
Christmas is the one where we celebrate how the Easter bunny brought Frankenstein's monster back to life in order to put him on top of the bonfire on November 5th.
I think.
My local Sainsburys had Christmas puddings and Stollen on the shelves in AUGUST, and one hotel was advertising for Christmas party bookings back in JUNE.
At least they didn't play "Fairy Tale of New York" on my birthday (mid-Sept) as they have done in previous years.
Oh hush, the co-op round the corner from me has mini-eggs in already and a small selection of creme eggs (the tortured look on the staffs face when I enquired as to how this has beaten both the Christmas crap and Halloween guff said it all)
Assuming it's the same situation as when I was involved some years ago. If an organisation is big enough, and has storage to cope, to buy at wholesale prices from a Chritmas product supplier - decorations etc, then the supplier will deliver in, say, April and there is no requirement to pay for the goods till December / January following. The manufacturer is happy as they can clear their warehouse and it puts less pressure on them to send out all the orders out in one go, as would happen if they insisted on end of month following delivery for payment. Naturally enough, the retailer(s), whose own warehouse is stuffed to the gunnels with "free" stock is happy to recoup as many sales as early as possible before payment is due and floods the shelves with stock well before the actual "season" starts.
We had a long spell where end user PC hard disks would fail and this was before the time where everyone stored data on servers.
We realised that the disks were failing after a few minutes. This obviously seemed to be a heat related issue, possibly bad silicon on the controllers.
We surmised and were right that if we froze them to chill them down for a while, they'd work for longer & just long enough to copy the data off. We saved a few people back then, who quickly saw the merits of server storage :-)
In my case I discovered that memory chips eg SD cards often respond to freezing.
I got the data off and backed it up *properly" this time, the problem is that every now and then someone claims that this works for HDDs. Sure, in like 1992 but these days it just makes things worse. You are better off sacrificing poultry at it than freezing/heating/whacking/etc the drive, it (usually) won't work but at least the professionals can get it back nearly all of the time.
It is a yuge gamble, but a stuck spindle might be unstuck if struck at just the right moment with just the right impact, usually just before or immediately after you hear the spindle motor start trying to spin. I do not know if the heads move at all before the platters are spinning -- ISTR they need the spin to keep them "floating" above the media.
In any case, I would much rather send the drive off to someone like DriveSavers but this is rarely (close to never) the first resort for customers. I found the pricing model to be the problem: DriveSavers and its ilk post recovery costs up front, whereas the time you will take lighting candles, sacrificing poultry (the Popeye's lunch suffices, IMHO,) formulating new incantations (mostly of curse words, old and new,) and such is an intangible until the invoice is received.
I tested and confirmed this the last time I had to do data recovery in which there was no backup (not a regular client, thankfully.) I worked up an estimate based upon a worse-case scenario of the above-mentioned voodoo with no promise of results, then presented a specialized recovery quote which would not charge if no data was recovered. Off it went in next-day shipping.
This. Very Much.
I worked at one place that had several regional offices, all connected to the Home office which was in a separate city some 200 miles (and a 3 hour drive) away.
The bulk of the users had mostly notebooks; Most were very good about keeping files backed up, either putting them on a CD-R or the remote file server when they were in the office.
The boss of one of the regional offices kept his notebook *in* his desk, on a docking station, and came in one morning to find that it had crashed. I got called in, heard the Clunk Of Death, and asked if he had backed up his files to the file server sitting at the Home office. (this remote office had a full T-1 line there, which was pretty decent for 2004)
He told me no, and by the end of the day I had extracted the drive out of the laptop, packed it in as much foam and bubble wrap as I could manage, and was driving it down to the local air freight depot to express ship the door thing off to OnTrack for them to work their $3,000 magic.
And magic it was; there was a single file they could not recover, and it was an OS file. (I had already gone to the local computer store and bought a replacement drive and had rebuilt the machine whilst management was deciding if they wanted to spend the money to recover the drive.)
To drive the point home of 'we know it sucks, but please back up your files to the remote file server', the recovered files were put on said server, and I pulled them across the line to re-populate the machine.
(It was a small-ish company that didn't bother putting a branch file server at each office, although that would have made sense. *shrugs*)
Many mumble years ago I came home from 'Uni one fine winter's morning to discover that my 500MB laptop drive was deader than Elvis and not spinning after less-than-careful handling courtesy of the luggage monkeys.
Checked to see why and somehow the head had become hung up on the side of the platter.
A few seconds work with a plastic tool precision made for the job and the head wasn't stuck any more.
Got the important files off, put the old 120MB back in and copied them over.
It worked for years after that!
It is a yuge gamble, but a stuck spindle might be unstuck if struck at just the right moment with just the right impact, usually just before or immediately after you hear the spindle motor start trying to spin. I do not know if the heads move at all before the platters are spinning -- ISTR they need the spin to keep them "floating" above the media.
I ran into that with Convergent MiniFrame which was fine when the box was running. Problem was if the machine needed to be moved as disks (Micropolis IIRC) had a habit of not spinning up after they'd cooled down. Not without help from good whack on the side anyway. Rubber mallet was useful tool back in those days.
Lubricants have moved on since those days and these days if disks don't spin up it tends to be the motor packing up rather than sticky bearings. Likewise drives' power consumption is bit lower and they don't generate quite as much heat as the old ones.
I had advised the site director about backups. He assigned someone to look after the job, but despite this, no tapes were swapped.
As predicted the server failed; and the drives were the cause. No tapes swapped, despite my forceful admonitions, so no restore possible. I arranged for the tapes to go to a specialist recovery place; meanwhile, built a new server ready for the data. Got about 85 - 90% back at a cost of about £1400.
You'd think that they had learned their lesson, but oh no. Still no tapes being swapped. So I ended up running a remote backup process, D-2-D-2-T. That seemed to be the only way to prevent a recurrence. Over the next 5 years, didn't lose a single file.
I left the company and I'm told that the overseas parent company demanded they take control of everything. That included backups and they insisted each site manage their own process. About 2 years later, they had a server disk fail and no-one noticed. When a second drive failed, they lost all of the data, because no-one had swapped tapes!
I had a customer with an 8" HDD (it was a long time ago) who couldn't get her DEC RSX11M system to boot and there was no backup. After listening to the disk noises, she got the idea that the disk just wasn't coming up to speed so she powered it down, took the cover off the disk and tried rebooting. Sure enough the disk motor wasn't getting the disk up to speed so she gave it a little help with her finger...
The system booted up and she got all of the data backed up - kudos to her!
I can vouch (sort of) for that kind of hackery... I had a slot loading DVD-ROM drive, back before most non-nerdy people knew DVD was a thing*, which eventually required a "bump start" to get it going; A quick push with a thin, but not sharp, object was enough to get it spinning and it survived like that for years.
* I think even I only had 1 DVD movie for years though as they were still ridiculously expensive.
> disk remedy that involved freezing a disk and then whacking it hard.
Sparc Storage arrays - I once had the (dis)pleasure of having to move a bunch of them to a new site. They had been up and spinning for years by that point..
At the new site, about 40% of the drives refused to spin up - totally nuking the array. In somewhat of a panic (we had backups but they were in packing crates at the old office and the extra array that we'd got in case of hardware failure was doing the same thing) I phoned Sun (this was obviously in the days when they actually had people who knew what they were doing). The advice was along the lines of 'take the drives out, bang them on the desk, stick them back in'
Which worked. I still lost a small proportion of drives but I had sufficient spares that I could bring up the array and let it rebuild itself.
We got rid of those arrays as quickly as possible after that.
I've suggested that particular solution to many customers at one time - the particular gear had a steel washer fixed to an aluminium chassis and occasionally the oxide would go non-conductive rendering the unit non-functional.
The customers would say, "But bang it? won't that break it?"
I told them, "It's already broken - it can't get any worse"
It fixed the issue almost 100% of the time.
In the early 90's, we had a Christmas power down due to electrical work. On returning in January none of our Sun workstations would boot up. Got them going in the end by taking out the hard drives and putting them on the radiators to heat up. Seems like the hard disk manufacturer had used bearing oil that was too thick and wouldn't allow the drives to spin up whilst cold.
Many many moons and cpu generations ago...
Had a batch of "recovered" drives that refused to consistently spin up and had intermittent data errors but were earmarked for a general use public file storage and message system. (BBS)
Suggestion from one drive manufacturer tech was, since the "sleeve bearings" were obviously worn to set them on their sides, reformat and "never" move them again. Tinker toy to the rescue. The array lasted a few more years without issue.
BTDT - once had a script wipe out the mail spool folder (a combination of a script to remove mail folder for users no longer in NIS, /var/yp/securenets and a new subnet being put in which everything got multihomed on, including the NIS master/mail server). That was a bad time to realise we'd never reinstated the mail backups which were causing problems a few months back...
Luckily, it was about a week after I'd taken a full backup of pretty much everything as an ad-hoc yearly backup thing and it was just after New Year (so minimal mails coming through). Managed to restore that and advise users how many mails they'd missed if they asked (by trawling the mail logs). That was a scary morning...
After that, the script merely mailed us the rm commands to remove the mailboxes so we could sanity check it...
“Sometimes it takes the CEO looking destruction in the face before arcane requests for budget have meaning.”
Not just budgets, but in our case (hence the AC) even making basic decisions. Even when we've had good advance notice of something (such as connectivity being terminated at a site) he'll procrastinate and procrastinate, and then when he realises that "yes, this really is going to happen and the fertiliser really will be flying off the air circulator soon" does he go into panic mode and expect everyone to work miracles to deal with it. Usually it's left to the last minute - eg if we've been told the lead time for a new connection is 12 weeks, he'll leave it till 11 weeks to "brown flying stuff day" before ordering and then expect the supplier to pull out the stops.