back to article Rookie almost wipes customer's entire inventory – unbeknownst to sysadmin

Welcome once again to El Reg’s weekly instalment of Who, Me?, where readers get monumental cock-ups and heart-stopping near-misses off their chests. This week, Reginald tells us a personal horror story from the ‘80s, when he worked for what was then a top five minicomputer biz. At the time, he was on site at a customer – a …

  1. Aladdin Sane

    If rm -rf /* doesn't delete anything valuable

    Did it even happen?

    1. Zippy´s Sausage Factory

      Re: If rm -rf /* doesn't delete anything valuable

      No, no it didn't.

      At least, that's my story, and I'm sticking to it...

    2. Anonymous Coward
      Anonymous Coward

      Re: If rm -rf /* doesn't delete anything valuable

      If a domain server falls over in a forest, does it make a sound?

      1. bpfh

        Re: If rm -rf /* doesn't delete anything valuable

        It makes the sound of a thousand wailing users marching to the Helldesk office armed with torches and pitchforks...

    3. Jim 59

      Re: If rm -rf /* doesn't delete anything valuable

      Eh? As I read it, Reginald kicked off the rm -rf /*, then hit the power switch before it deleted too much. The tape rescue revealed that "everything down to /dev" had been deleted, ie. everything in / beginnind a,b,c and some d. On a modern system that might include /boot and /bin, but evidently was not a total disaster on Reg's server.

      1. Aladdin Sane

        Re: If rm -rf /* doesn't delete anything valuable

        That whooshing noise is the joke flying past you.

        1. Prst. V.Jeltz Silver badge

          Re: If rm -rf /* doesn't delete anything valuable

          @Aladdin (see what i did there?)

          Past who?

  2. Anonymous Coward
    Anonymous Coward

    I remember discovering the hard way that when you delete an email account in Thunderbird and it asks if you want to delete all the files associated with it it actually means do you want to delete the entire directory tree below where the account is stored .... so, as I discovered, saying "yes" when the reason you are deleting the account is because you'd just created it in the wrong place in the the directory tree is not a good idea - instead of just deleting the new account I nuked all the data associated with all our family email accounts!

    1. bpfh
      Mushroom

      Been there, done that :(

      And wished that the design team who signed off on this would always be served stale beer for ever and ever. Amen.

    2. jcitron

      Ah yes. I discovered that myself the hard way.

  3. defiler

    And then billed 3 extra hours?

    Can't be any more dishonest than the IBM guys who would lock themselves in the computer room, eat their sandwiches, read the paper for an hours, and then flick a switch to "upgrade" the hard disc capacity... Apparently...

    1. Anonymous Coward
      Anonymous Coward

      Re: And then billed 3 extra hours?

      Ahh...the good old days where you paid IBM and got something useful at the end of it.

      Sure they still ripped you off, but that's nothing new.

    2. big_D Silver badge

      Re: And then billed 3 extra hours?

      Was it IBM or ICL that "upgraded" to a faster machine by removing a resistor board?

      1. smudge

        Re: And then billed 3 extra hours?

        CTL (later ITL) doubled your memory size by connecting up a wire...

        1. Martin an gof Silver badge

          Re: And then billed 3 extra hours?

          doubled your memory size by connecting up a wire...

          Seems to be fairly common in all sorts of areas. Three car-related ones. My mum had a Renault 5 back when we were changing from Leaded to unleaded petrol. The car as delivered used four-star and the upgrade to unleaded (which was lower octane) essentially involved connecting up a jumper wire under the bonnet somewhere.

          Likewise, a friend of mine had a Passat. He discovered that replacing the (?)indicator stalk for one which included the control buttons for cruise control, then flipping a bit somewhere (via the OBD socket connected to his laptop) enabled cruise control; i.e. all the other prerequisites (presumably sensors and things) were already installed and working.

          My own car had a problem with the "leather" on the steering wheel - it started peeling off like bad sunburn. It was only a few months old, so this was fixed by the dealer, but when I got it back I discovered that cruise control had been disabled. Again, turned out to be a bit-flip in the ECU rather than something simple like the garage forgetting to connect the buttons up after swapping the steering wheel.

          Given a few minutes I could probably think of dozens of other examples. I'm not sure if I think it's a good thing or not...

          M.

          1. Antron Argaiv Silver badge
            Unhappy

            @Martin an gof

            ...car-related ones...

            Funny...I'm going through this exact thing right now with the wife's Jeep Wrangler.

            Took it to the dealer (I know...read on) several months back, for [Takata] airbag replacement recall, and just a few weeks ago, discovered that the fog lights no longer turn on.

            I have been all through the wiring and such, nothing's damaged, and there's even voltage *at* the lights, but not enough current to turn them on. Seems they're controlled by the computer, and there's an option bit in the memory which is set if you have paid for fog lights, and cleared if you have not. So, even if you add the control stalk with the fog light switch, and add the wiring, the damn things won't work unless the computer has that bit set.

            So...they *used* to work, car came with them, but ever since the dealer changed the airbag, they...don't. I'm replacing the control stalk on the off chance the switch is bad ($90) but my money's on the dealer reprogramming with the wrong image or something when they changed the airbags.

            If I have to go back to the dealer, I will be going back with a complaint. Unfortunately, I have not a leg to stand on, and no way to prove they did this, but if they magically make them work, and try to charge me, there will be some discussion. I will be asking them exactly what they did to make the lights work again, and if it involves the "fog light disable" bit, I'm going to raise a stink.

            1. Anonymous Coward
              Anonymous Coward

              Re: @Martin an gof

              Took it to the dealer (I know...read on) several months back, for [Takata] airbag replacement recall, and just a few weeks ago, discovered that the fog lights no longer turn on.

              Two options here - either is likely as the other...

              Firstly, the car was reloaded with the CCF stored at build for that VIN, these can sometimes get corrupted or the VIN can be misread by the OCR / barcode reader - what should happen is that the CCF is downladed from the vehicle prior to any work involving the potential for the CCF to be reloaded - it's actually stored in two places, usually the instrument pack and the central electronic module.

              Secondly, it's possible that the CCF is correct from build but someone has upgraded prior to sale (again, the CCF should have been downloaded from the vehicle and stored locally), this usually happens when the dealer buys a load of 'base' spec vehicles then upgrades them prior to sale - charging the customer for 'extras' and getting a kickback from the manufacturer for hitting 'accessories' sales targets.

              A prime example of this is Motab cars having PDC added as accessory fit - because safety!!!, even though the cost of factory fit is negligible (to the dealer) and is more integrated with the other vehicle systems.

              1. Martin an gof Silver badge

                Re: @Martin an gof

                he car was reloaded with the CCF stored at build for that VIN

                In terms of my car, what I didn't mention was that the thing had been back to the dealer several times from new due to sudden unexplained and temporary loss of power. You know, the sort of thing where you'd put your foot down, the thing would start going, then hesitate for a few seconds, then carry on as if nothing had happened.

                Mostly what the dealer did - apparently in consultation "with France" - was re-map the ECU and clock up quite a lot of driving miles while doing so.

                This was until the car pretty much failed on my wife an hour from home and we had to pay to have the car recovered. Taking it to a different dealer, they spotted that the vacuum switch on the turbo was loose - had probably never been fitted properly when manufactured - and had rattled around and eventually cracked (it's part of some bigger component apparently). Fixing this solved everything, but I never worked out what they did with the ECU mappings, and I suspect that the original garage, which was the one which later also replaced the steering wheel, re-loaded something incorrectly, hence defeating cruise control.

                On the bright side, the car has now done some 155,000 miles, still regularly achieves well over 60mpg (62 - 65 being my "normal" range, but then I do a lot of motorway driving) and hasn't had any other "odd" problems, just the usual wear-and-tear. And a completely rusted-through bracket under the radiator. I did mention it was French :-)

                M.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: @Martin an gof

                  That would generally cause a DTC of 'commanded position unreachable' or something along those lines...

                  is your car PSA French or 'Diamond / Nissan' French - you have my sympathy in both cases, however you're more likely to get things resolved to your satisfaction if it's a PSA car combined with an appropriately motivated tech...

                  1. Martin an gof Silver badge

                    Re: @Martin an gof

                    more likely to get things resolved to your satisfaction if it's a PSA car combined with an appropriately motivated tech

                    Diamond/Nissan.

                    I don't usually take my cars to the dealer - I have a "bloke in a backstreet" who is absolutely fantastic that I've been using for 20+ years now. I only used the dealers for those issues because the car was still relatively new and hence under warranty. On more than one occasion my bloke has had to repair something that a dealer had previously "fixed", the classic being a clutch on a second-hand car bought from a main dealer.

                    The original failed within a month of driving away from the forecourt so the dealer "fixed" it, and within six months it had failed again. It wasn't a problem with the clutch per-se, it was (as my "bloke" discovered) that when they screwed the thing back together they had used seventeen almost completely random screws, only a half a dozen of which were the proper length and taper (who knew?) and many of which had therefore failed. Even when I plonked the broken clutch and collection of screws on the desk of the dealer's service manager, it took a good hour of wrangling before they would agree to refund most of the repair cost.

                    M.

                    1. big_D Silver badge

                      Re: @Martin an gof

                      Main Dealers can be a pain... Or downright dangerous.

                      I had a Toyota and the dealer did the first service... Forgot to put the oil cap back on afterwards, by the time I got half way home, the engine bay was covered in oil and it was bellowing smoke.

                      Then, the next time, they bled the brakes and forgot to reseal the lines! That was a scary ride home!

                      The third time, they left a pair of pliers under the hood.

          2. Anonymous Coward
            Anonymous Coward

            Re: And then billed 3 extra hours?

            "The car as delivered used four-star and the upgrade to unleaded (which was lower octane) essentially involved connecting up a jumper wire under the bonnet somewhere."

            That's a perfectly legit example of when it should be done; it's a configuration change to meet different conditions of use, and unleaded wasn't an upgrade; if anything mpg goes down.

            Better than having to spend time in a garage while either an ECU was remapped or the ignition advance and retard was replaced.

            My boiler has something similar to switch from methane to propane, though there is a jet has to changed as well.

          3. Martin-73 Silver badge

            Re: And then billed 3 extra hours?

            indeed, and the same is true of base model GM cars (Vauxhall/Opel/Holden)... or was up until the early naughties. Most of the kit for the higher spec was there, but either not connected, or a link removed, but ONLY on SOME cars (it seemed random)

            Turns out it was a manufacturing thing, they'd estimate demand for each trim level, and if they had any uncertainty, equip more vehicles at the higher spec, and then disable stuff to downgrade if needed.

            I got UNlucky with my Corsa, 2 wire harness to the doors instead of the full 14 wire one, so electric mirrors/windows would've been hard to retrofit :(

          4. Glenturret Single Malt

            Re: And then billed 3 extra hours?

            I know for a fact that I am not the first to put the wrong fuel in the tank soon after changing cars. On this occasion, I realised almost immediately that I was putting petrol in a diesel tank (not as bad as the other way round, thankfully). Since the tank was pretty low, I was able to fill it with about 50 litres of diesel and the subsequent receipt showed I had put in about 450 ml of petrol which turned out to to have no discernable effect on the car's performance which subsequently gave me 150,000 miles of sterling service.

      2. Chris Miller

        @big_D

        Was it IBM or ICL that "upgraded" to a faster machine by removing a resistor board?

        Pretty much all the mainframe manufacturers (IBM plus the BUNCH - Burroughs, Univac, NCR, CDC and Honeywell - ICL didn't appear because this was US jargon!) did this. It actually made economic sense for both manufacturer and customer, mostly because there was just a single type of system to be built rather than half-a-dozen, giving economies of scale (and making upgrades really, really simple).

        Back in the day (c.1980) we ran a Honeywell mainframe, middle of a range of 5. Part of the maintenance contract (excruciatingly expensive, this was actually where most of the money was made - you could get a good deal on hardware, but there was rarely any negotiating on maintenance) was a visit every other week by an engineer to run diagnostic tests. To save himself time, he would reach inside the machine to the 'secret' microswitch that turned it into a top-of-the-range model. Of course, the operators soon sussed this out, which meant that work scheduled for an entire weekend could be accelerated to complete in under a day, leaving extra pub time ...

        1. Aladdin Sane

          Re: @big_D

          Tesla still do this.

          1. Anonymous Custard
            Boffin

            Re: @big_D

            Not to mention Intel and fuses...

        2. Antron Argaiv Silver badge
          Thumb Up

          Re: @big_D

          This was true even with IBM's relay powered plugboard wired unit record equipment (with which I worked in high school -- I'm a sucker for last chances on dinosaurs).

          The guy who ran the facility says "don't do this when IBM's around, but...", then opens up the back, pulls out a relay and the machine magically starts running twice as fast. He said the only drawback was that the motor used couldn't handle the higher speed for long periods, so you had to be judicious about when you did it.

          IBM 402 Accounting Machine, for those who share my love of obsolete machinery. Big as a VW beetle and probably weighed twice as much. When "big iron" really was.

          1. cortland

            re "big iron"...

            When I retired from the Amy I walked into a job in electromagnetic compatibility at Wang Labs, in Massachusetts. It took three of us to wheel one of the TEMPEST machines up the ramp into the test chamber; HDD the size of a Fiat, outside, with cables that were like wrestling world-record Boa Constrictors.

            1. big_D Silver badge

              @cortland, I remember working at a place, where we had just gone from VAX to MicroVAX and hat a box the size of a desk drawer unit, which was a VAX with 4GB storage, a huge amount in those days.

              We then got a contract to run an IBM installation for another customer and their hardware started to arrive, there was a huge (6' high, 2' - 3' wide and the same deep). I asked what that was, the IBM op said "a DASD", I was impressed, if my desktop had a 120MB drive and the MicroVAX had 4GB, then this must have had a HUGE capacity. When I pointed at the MicroVAX and said, well that has 4GB, what capacity does this thing have, being at least 12 times the size of MicroVAX and just being the drive? He turned a little red and mumbled 500MB.

          2. jcitron

            Re: @big_D

            One of these:

            http://www.righto.com/2017/04/1950s-tax-preparation-plugboard.html

            They are quite amazing beasts with all the switches, relays and mechanical parts.

            I saw one of these in the hallway of a company I worked at as it was being carted off to scrap. I didn't know what it was at the time, sadly. You're right it was about the size of a VW Beetle!

        3. Antron Argaiv Silver badge

          Re: @big_D

          ...and not only the mainframe manufacturers.

          DG processor boards had all kinds of little option jumpers. On the newer designs, it was microcode PROMs, so they started covering them with epoxy when the customers figured out that they could change instruction sets (we had commercial and scientific, IIRC, and one cost more than the other) by changing microcode.

          I worked in communications and networking, but even then, there were boards with fewer UARTs which could be upgraded simply by adding the missing parts (they filled the empty holes with solder to make it harder).

          1. ChrisC Silver badge

            Re: @big_D

            "they filled the empty holes with solder to make it harder"

            Whilst that may have been the outcome, I doubt it was the intention. For through-hole PCBs which are being wave soldered, the only reliable way to prevent empty holes from being bridged by solder is to temporarily cover them with kapton tape or similar. As this is an additional manufacturing operation which costs both time and materials, and thus ultimately money, it's the sort of thing which tends to be done *only* if those empty holes need to still be empty after the wave soldering has been completed - e.g. to allow components to be fitted to the underside of the PCB.

          2. jcitron

            Re: @big_D

            This is very familiar. I worked on Visual and Ontel terminals with motherboard and option boards like that. The difference was sometimes only an EPROM change or a jumper clip or add to change the model from one to another.

            At the last manufacturing company I worked at, they made digital proofing systems that either had 2400 dpi imaging and ran at a normal speed, or some that had 2540 dpi with a turbo speed. The difference in cost was quite substantial, but the change was only a switch flip on the motherboard and a different EPROM.

            There are many other examples of this even in recent years. There are some video cards by NVidia that could go from GTX gamer video cards to their Quadro professional series cards by adding a jumper or resistor and cutting an etch on the back of the board.

          3. John Brown (no body) Silver badge

            Re: @big_D

            "they filled the empty holes with solder to make it harder"

            They would be otherwise unused but still be plated through holes so when the board goes aver the solder bath, that's what happens. It wasn't done to sypie you, it was just, from their point of view, a pleasing side effect.

            On a similar note, back in the day I added am extra scart socket to my TV by adding just the socket and a few resistors, found a slow motion playback button hidden behind the fascia of a video disk player and upgraded an analogue satellite receiver with a switch and a crystal to get the newer birds and all the newer channels.

        4. Stoneshop

          Re: @big_D

          Pretty much all the mainframe manufacturers (IBM plus the BUNCH - Burroughs, Univac, NCR, CDC and Honeywell - ICL didn't appear because this was US jargon!) did this.

          The upgrade kit to turn a DEC VAX 82x0 into an 83x0 consisted of the digit 3 to put on the front panel and a set of microcode EPROMS containing fewer NOPs. One of the other VAXes required only a backplane jumper to be (re?)moved.

        5. Anonymous Coward
          Anonymous Coward

          Re: @big_D

          ICL machines from 2900 onwards were throttled by microcode. It was necessary to install the correctly licensed version tied to the CPU serial number to change the machine range and throttle / open up the power. This was a hardware engineers job and the installed Operating system was not affected.

      3. aks

        Re: And then billed 3 extra hours?

        as opposed to changing a bit to enable a feature which wasn't previously being paid for?

      4. swm

        Re: And then billed 3 extra hours?

        The original Dartmouth time sharing system ran on a GE-225 with a 18 usec cycle time. A GE customer bought this system but found the current release wouldn't run because Dartmouth had upgraded to a GE-235 and the students (of course) used all of the new nifty features of the new machine. So GE sold the customer a GE-235 but put a wait loop in the exec to slow it down to GE-225 speeds. This code was clearly commented. The customer didn't need the extra speed and was quite pleased with the resulting system.

        Later I noticed that an IBM 407 accounting machine we use for listing cards would miss every third cycle. I discovered that by pulling out the S2 relay the machine would run full speed. I pointed this out to the customer engineer and he said we were only paying for a 100 line/minute machine and not a 150 line/minute machine so we should leave the relay in. So I did.

      5. trolleybus

        Re: And then billed 3 extra hours?

        Nothing unusual about that. Some Burroughs upgrades were simply moving a jumper. No different than changing software settings to change the way a car works - you pay for better performance, you get better performance.

      6. Steve Kellett

        Re: And then billed 3 extra hours?

        ICL 2905 to 2955 upgrade was to remove a jumper wire that slugged the system performance by making the CPU service interrupts continually.

        Later machine families did this in the firmware.

    3. Anonymous Coward
      Anonymous Coward

      Re: And then billed 3 extra hours?

      One of my bosses had installed security cameras in the server room to protect their investments from going walkies.

      When the IBM team arrived to do contracted scheduled maintenence on the machines, boss watched via the cameras.

      Boss was *VERY* unhappy as he watched the foursome sit on their asses, drink beer, eat their lunches (in a room *nobody* was allowed to eat nor drink in), & play cards for hours.

      Ten minutes before the scheduled time was to end the four got to work: using cans of compressed air to blow out the kit, rewiring various things at random with fresh bits of cable, & rebooting everything.

      That was it, the whole of their "maintenence". For which the company I worked for was paying prices that would make King Midas blanche over.

      Boss said nothing, just made copies of the security tape & submitted it with his revised bill for services rendered.

      "I'm not paying you to sit on your arse, suck down lunch, & play cards for hours while you're supposed to be working. I'm not paying $Amount for a set of swapped cables & rebooting. Don't like it, tough shite. Check the tape for why."

      Strangely enough IBM didn't say a damned thing about the company not paying that bill nor quite a few more.

      Moral of the story: make sure those security cameras are filming the scene of the crime so you can use the evidence to pimp slap the folks trying to fuck with you.

    4. GlenP Silver badge

      Re: And then billed 3 extra hours?

      I seem to recall the clock speed on the Commodore PET could be increased by the simple use of a POKE command to a register. This was only safe for certain CPUs though, if you had the "wrong" one the chip would fry itself.

      Biggest con I reckon with PCs was the so called "turbo" button which was nothing of the sort. It was actually a slow-down button for app compatibility.

      1. cortland

        Re: And then billed 3 extra hours?

        Some of the older Samsung monitors sold with Tandy/Radio Shack computers in the 1990's could be made to run at higher resolutions, which resulted (or so I heard) in a monitor catching fire.

      2. Jeffrey Nonken

        Re: And then billed 3 extra hours?

        "Biggest con I reckon with PCs was the so called "turbo" button which was nothing of the sort. It was actually a slow-down button for app compatibility."

        ...Yes, but everybody knew what it was and what it was for. Calling it "Turbo" was marketing, not deception. They had to add it for backwards compatibility with programs written for the original PC designed with delay loops that depended on the CPU's clock speed.

        The computers would have been cheaper to build without the 2-speed design. Not much, but it was a temporary fix for a real problem that the motherboard guys had no control over.

        Calling it a confidence trick seems a bit odd.

    5. usbac Silver badge

      Re: And then billed 3 extra hours?

      A few years back I bought a fancy new digital oscilloscope. The scope was offered in both a 50MHz and 100Mhz version with about $200 difference in price..

      The manufacturer actually went to the trouble of adding parts to make the 50MHz version. They added an RC filter to the input to limit the bandwidth. Someone discovered that you can just lift the capacitor, and you have the 100MHz version (and thus voiding your warranty).

      Then, someone else figured out that it's not even that hard. All you have to do is load the firmware for the 100MHz version, and the filter is disabled by software.

      It still baffles me that they would spend money to make the scope slower? Why not just offer the 100MHz version only?

      1. SImon Hobson Bronze badge

        Re: And then billed 3 extra hours?

        Why not just offer the 100MHz version only?

        That's marketing for you. If they ony sold the 100MHz version then they'd probably have to sell it for about the cost of the 50MHz version. But doing it like that, they might have to sell the 50MHz version for alittle less, they can price the 100MHz version significantly higher - thus increasing profit margins.

      2. fajensen

        Re: And then billed 3 extra hours?

        It still baffles me that they would spend money to make the scope slower? Why not just offer the 100MHz version only?

        Price Bracketing - Most people will prefer to pay more for the higher performance version over the basic cheap(est) version. The existence of a cheap model "justifies" the higher priced ones. For this strategy to work, the cheap version of course has to exist so they will make one with minimum effort invested. Sports bicycle suppliers does this all the time.

        Sometimes suppliers also have several so-so product variants which merely exists to funnel sales towards "The Original", "The One That Just Works", like Absolute Vodka (or Garmin) are doing.

  4. big_D Silver badge

    .cobol

    I had been working on a major change to some code on a COBOL system. For safety, I copied the original files to from *.COB to *.COBOL. I then spent 3 days working out the changes needed, and testing them.

    I got it all working and error free, so I went to delete the original files, but I only got as far as "del *.COB" befiore hitting return. WAH! I copied the .COBOL back to .COB and started over again. As I knew what I wanted to do this time, it only took about a day to re-do what I had deleted.

    Another time, I had been working late on a presentation for my boss and I went to rename it, right click, select Rename from the menu, hand slip, click, file gone! Delete is right above Rename in the bloody menu! Luckily, I had sent a copy to someone to start checking over about 20 minutes earlier, so I had a recent backup to work with! But that is what happens, when the boss comes in at 5:30 in the evening and says he has a presentation at 10am at CeBIT the next day and needs 40 slides...

    1. defiler

      Re: .cobol

      when the boss comes in at 5:30 in the evening and says he has a presentation at 10am at CeBIT the next day and needs 40 slides...

      That pretty-much means your boss is a bellend. Just saying.

      1. big_D Silver badge

        Re: .cobol

        Yep, one of the many reasons I don't work there any more.

      2. Just An Engineer

        Re: .cobol

        "when the boss comes in at 5:30 in the evening and says he has a presentation at 10am at CeBIT the next day and needs 40 slides..."

        There are 4 little words that come to mind when this happens to me. "sucks to be you".

        Or conversely "it's really going to be a log night for you, boss."

        Usually when this happens the "Boss" has known about it for 3 to 4 weeks, then the Oh Shite moment happens and remembers the night before. Usually when his boss asks if he is ready for tomorrow...

        1. big_D Silver badge

          Re: .cobol

          He knew about it 6 months in advance. I'd told him 4 months in advance not to leave it to the last minute... He said he had everything under control.

          1. Rich 11

            Re: .cobol

            He said he had everything under control.

            That's the warning sign, right there. Along with "Trust me, I know what I'm doing."

    2. DropBear

      Re: .cobol

      "Delete is right above Rename in the bloody menu"

      Really. And you're not even going to try undeleting what is essentially a perfect candidate for recovery as long as you realise what you've done immediately...?

      1. big_D Silver badge

        Re: .cobol

        On a Samba share? No. No trashcan, no VSS.

        1. Blacklight

          Re: .cobol

          My SMB box has a .recycle

          Granted I have to enable it, but it's there - and on every family share "just in case" :)

    3. bpfh

      Re: .cobol

      "Delete is right above Rename in the bloody menu"

      Probably designed by the same person who designed the crontab app then, with the command line options -e to edit and -r to remove immediately without confirmation. Misstype at your peril...

      I found this out - to my peril - about 3 seconds before I realised that it was a good idea for a server's crontab to include a daily executed crontab -l > /foo/bar/crontab-backup.txt ...

      1. schubb

        Re: .cobol

        I remember writing a wrapper to ensure I didn't do this very thing...

      2. Down not across

        Re: .cobol

        Probably designed by the same person who designed the crontab app then, with the command line options -e to edit and -r to remove immediately without confirmation. Misstype at your peril...

        Using crontab -e is asking for trouble even without mistypes. I've see too many corrupted or truncated crontabs after someone has edited them with crontab -e. crontab -l > crontab.txt;vi crontab.txt;crontab crontab.txt is much better way.

        You mean not everyone has crontab entry that backs up crontab at least daily?

    4. Jason Bloomberg Silver badge

      Re: .cobol

      I went to delete the original files, but I only got as far as "del *.COB" befiore hitting return.

      I managed a similar thing but more deliberately; belatedly finding "DEL FOOBAR.???" included files with no extensions when it didn't on a previous version (Win3.1?).

      That wasn't the disaster it could have been but I've had my share of all-nighters making it look like I hadn't accidentally scrubbed a system clean.

    5. MrBanana

      Re: .cobol

      "WAH! I copied the .COBOL back to .COB and started over again. As I knew what I wanted to do this time, it only took about a day to re-do what I had deleted."

      When this has happened to me, I end up with better code than I had before. Re-doing the work gives you a better perspective. Even if functionally no different it will be cleaner, well commented, and laid out more consistently. I sometimes now do it deliberately (although just saving the first new version, not deleting it) to clean up the code.

      1. big_D Silver badge

        Re: .cobol

        I totally agree, the resultant code was better than what I had previously written, because some of the mistakes and assumptions I'd made the first time round and worked around didn't make it into the new code.

  5. Woza

    Reminds me of the classic

    https://www.ee.ryerson.ca/~elf/hack/recovery.html

    1. Anonymous South African Coward Bronze badge

      Re: Reminds me of the classic

      https://www.ee.ryerson.ca/~elf/hack/recovery.html

      Was about to post the same. It is a legendary classic by now.

  6. Anonymous Coward
    Facepalm

    Try uninstalling a video game...

    Only to find out it wiped the entire drive !

    The game put it's saved games in the root of the drive...

    And after anwsering "yes" to the question "do you also want to delete all saved games" it was literally 'game over'...

    1. J.G.Harston Silver badge

      Re: Try uninstalling a video game...

      Not quite, but TEMP=C:\DOS

      Then install something that tidies up the TEMP directory on exit.....

  7. Chairman of the Bored

    One simple trick...

    ...depending on your shell and its configuration a zero size file in each directory you care about called '-i' will force the rampaging recursive rm, mv, or whatever back into interactive mode. By and large it won't defend you against mistakes in a script, but its definitely saved me from myself when running an interactive shell.

    It's proven useful enough to earn its own cronjob that runs once a week and features a 'find -type d' and touch '-i' combo on systems I like.

    Glad the OP's mad dive for the power switch saved him, I wasn't so speedy once. Total bustification. Hence this one simple trick...

    Now if I could ever fdisk the right f$cking disk, I'd be set!

    1. Chris Evans

      Re: One simple trick...

      Can't you enter a command to abort the wipe?

      I'm no Unix expert as you will probable guess.

      1. PickledAardvark

        Re: One simple trick...

        "Can't you enter a command to abort the wipe?"

        Maybe. But you still have to work out what got deleted.

        On the first Unix system I used, an admin configured the rm command with a system alias so that rm required a confirmation. Annoying after a while but handy when learning.

        When you are reconfiguring a system, delete/rm is not the only option. Move/mv protects you from your errors. If the OS has no move/mv, then copy, verify before delete.

        1. Doctor Syntax Silver badge

          Re: One simple trick...

          "Move/mv protects you from your errors."

          Not entirely. I had a similar experience with mv. I was left with a running shell so could cd through the remains of the file system end list files with echo * but not repair it..

          Although we had the CDs (SCO) to reboot the system required a specific driver which wasn't included on the CDs and hadn't been provided by the vendor. It took most of a day before they emailed the correct driver to put on a floppy before I could reboot. After that it only took a few minutes to put everything back in place.

      2. Chairman of the Bored

        Re: One simple trick...

        @Chris Evans,

        Yes there are a number of things you can do. Just like Windows a quick ctrl-C will abort a rm operation taking place in an interactive shell. Destroying the window in which the interactive shell running rm is running will work, too (alt-f4 in most window managers or 'x' out of the window)

        If you know the process id of the rm process you can 'kill $pid' or do a 'killall -KILL rm'

        Couple of problems:

        (1) law of maximum perversity says that the most important bits will be destroyed first in any accident sequence

        (2) by the time you realize the mistake there is no time to kill rm before law 1 is satisfied

        The OP's mad dive for the power button is probably the very best move... provided you are right there at the console. And provided the big red switch is actually connected to anything

  8. Julian 8 Silver badge

    Reminds me of a contractor

    We had in a company a few years ago. He wrote a routine to clear down a specific folder on a cluster. all fairly simple. I did not find out for 3 weeks, but over the last three weekends one of our team had been in fixing an issue with a cluster node that had failed.

    Eventually I was given the script and asked to look over and it all looked OK, nothing obvious, a few commands and a delete command. I then asked why they suspected this script and was told the node's boot drive had been wiped.

    The following weekend and again a node went down. I looked at the script again and ran it line by line instead of just looking and when it got to the delete line instead of wanting to delete the intended folder it wanted to delete the C:\ instead.

    Looked closer at the line and there was a trailing space after the folder name, so in those days Windows decided to erase the folder where it was (which due to how the task had been created it ran in c:\)

    1. Version 1.0 Silver badge
      Thumb Up

      Re: Reminds me of a contractor

      Windows was good at that - I had an employee rename win.ini for some reason back in the old days. When win.ini doesn't exists Windows reported that every single file on the system was called win.ini ... the funny (and best part of this story) was that she requested all the original system and application install disks because "she wanted to verify and understand how the system (our standard 'customers' installation) was setup and configured" ... I gave her all the disks and watched her rebuild the system over a couple of days.

      She learned a lot and because a damn fine customer support and installation engineer! There's always a silver lining to a cloud.

  9. JimC

    One advantage of getting old

    Is that a dim mist has descended over most of my major cockups. I can remember a few of other people's though!

    1. The Man Who Fell To Earth Silver badge
      Unhappy

      Re: One advantage of getting old

      "I can remember a few of other people's though!"

      My (Irish) boss attributes this to "Irish Alzheimer's", which he says is "Long after the memory has gone, the grudge remains."

  10. Alien8n

    New contractor

    We're currently breaking in a new contractor as the previous one has been forceably removed due to the complete donkey's arse of a server upgrade done here at work.

    Basically:

    Built new server.

    Transferred file from old to new.

    Kicked off backup routines.

    Copy remaining files over.

    Kill server.

    Except he made 3 rookie mistakes:

    1. He hadn't ensured all the shared folders had been copied across, as well as all the "secure" folders. So anyone with a locked down set of folders found all their files were missing.

    2. When doing the second transfer he used Robocopy. Not a problem except he was using copy/paste scripts and managed to include the /mir flag. Result being deleting the last 2 days of updates on the new file server. Not a problem, except for error 3.

    3. He kicked of the final Robocopy without first checking to make sure that the new backups had actually worked, meaning we had no backup of the last 2 day's work.

    In good news we'd made the decision prior to the server upgrade to leave the old SQL server in place until after we'd tested the new application server, so all that was lost was a handful of Excel and Word documents. In bad news (for him) he'd managed to cock up at several clients resulting in a nice shiny new P45.

    1. Anonymous South African Coward Bronze badge

      Re: New contractor

      ....resulting in a nice shiny new P45.

      I have to assume that a P45 is a civil court case?

      1. WonkoTheSane
        Headmaster

        Re: New contractor

        "P45" is the UK government standard form used when one is invited to seek alternate employment.

        1. Anonymous South African Coward Bronze badge

          Re: New contractor

          "P45" is the UK government standard form used when one is invited to seek alternate employment.

          Ahh, a "pink slip" then.

          Thanks for letting me (and others) know :)

          1. Fatman
            FAIL

            Re: New contractor

            <quote>Ahh, a "pink slip" then.</quote>

            Also known as walking papers.

            1. FishCounter

              Re: New contractor

              Sounded like an RGE to me; Resume Generating Event. You folks would term that a CVGE, I suppose.

          2. Alien8n

            Re: New contractor

            Yup, basically he'd made so many mistakes at so many clients that he was fired for gross misconduct.

            The bit that beggared belief was that he was still expected to work out a notice period, which meant his last couple of weeks also involved watching him like a hawk. If he'd been working for me directly it would have been a straight forward "go home and don't come back until your disciplinary hearing" and then all his access removed there and then. Some people really don't understand how damaging a disgruntled employee can be. Reminds me of another conversation when engineering.

            "We're going to be making half the company redundant."

            "What are we doing to prevent sabotage of the equipment once we tell everyone?"

            "Nothing, no one would do that."

            Followed by half the factory going down over the next week as the operators started sabotaging the equipment.

            1. Nunyabiznes

              Re: New contractor

              ""We're going to be making half the company redundant."

              "What are we doing to prevent sabotage of the equipment once we tell everyone?"

              "Nothing, no one would do that."

              Followed by half the factory going down over the next week as the operators started sabotaging the equipment."

              Yep, been there, fixed that. Working at a brand new factory that replaced one in another state. All of the equipment we received from the old factory had to be refurbished because of sabotage. We figured it out pretty quick after the first *BANG*.

        2. Christoph

          Re: New contractor

          Which make it interesting when USA folks refer to the current (45th) president as 'P45' when they don't want to use his name.

          1. Antron Argaiv Silver badge
            Thumb Up

            Re: New contractor

            45*

            * indicating that he's "technically" the president, but there are extenuating circumstances...

          2. Bill Gray

            Re: New contractor

            Hadn't heard of that person being referred to as P45. I occasionally refer to him as the Orange Lord, or as He-Who-Must-Not-Be-Named-While-Eating.

            1. Anonymous Coward
              Anonymous Coward

              45

              It has been common practice for at least the past few presidents to sometimes refer to them by their number. Not necessarily due to dislike - at least not STRONG dislike. People who truly dislike a president come up with far more colorful names.

  11. Mephistro
    Happy

    In my first week in my first paid job...

    ... I was told by one of the execs to familiarize myself with a PC running THEOS/OASIS that was to be installed the next day at a customer's. I proceeded to test the software by inputting some fake data, generating queries and reports and all that carp.

    When I finished doing this, I decided to wipe all the data I created by using an "Initialize" option in the menu. To my indescribable horror, it wiped out all the data, the software and most of the OS.

    After twenty minutes of panic, cold sweats and frantic reading of the manuals, I found there was an option to undo the last initialization.

    Lesson learned: RTFM before playing with a system you aren't familiar with! 8^)

  12. Colin Bull 1
    Mushroom

    cp can also be dangerous

    After several years working in a DOS environment I got a job as project Manager / Sys admin on a Unix based customer site for a six month stint. On my second day I wanted to use a test system to learn the software more, so decided to copy the live order files to the test system.

    Unfortunately I forgot the trailing full stop as it was not needed in DOS - so the live order index file over wrote the live data file. And the company only took orders for next day delivery so it wiped all current orders.

    Luckily it printed a sales acknowledgement every time an order was placed so I escaped death and learned never miss the second parameter with cp command.

  13. Anonymous Coward
    Anonymous Coward

    not all heroes wear capes.

    or are heroes.

  14. Anonymous Coward
    Anonymous Coward

    it was like that when i logged.

  15. Anonymous Coward
    Anonymous Coward

    i'd written a script to deploy the latest changes to the live environment. worked great. except one day i'd entered a typo and it was now deploying the same files to the remote directory, over and again.

    it did that for 2 whole years with around 7 code releases. not a single person realised the production system was running the same code after each release with no change in functionality. all the customer cared about was 'was the site up?'

    not a single person realised. not the developers. not the support staff. not me. not the testers. not the customer. just made you think... wtf had we been doing for 2 years???

    1. Yet Another Anonymous coward Silver badge

      Look on the bright side, any bugs your team had introduced in those 2 years had been blocked by your intrinsically secure script

    2. Prst. V.Jeltz Silver badge
      Happy

      not a single person realised. not the developers. not the support staff. not me. not the testers. not the customer. just made you think... wtf had we been doing for 2 years???

      That is Classic! not surprised about the AC!

      Bet some of the beancounters were less than impressed , probly on customer side :)

    3. ma1010

      Sounds like a great idea!

      I'd written a script to deploy the latest changes to the live environment. worked great. except one day i'd entered a typo and it was now deploying the same files to the remote directory, over and again

      Could you PLEASE get a job sending out Windows 10 updates? Or almost any other updates, except for security ones?

  16. Anonymous Coward
    WTF?

    Time machine

    Can I have a go on your time machine? With the shiny modern macbook in a 1980s story!

  17. Robert Helpmann??
    Childcatcher

    Manual Recovery

    The man who one day would become my boss, let's call him "Dick"*, was called in with his fellow floor manager to discuss what needed to be done while their boss went on vacation. The way it was phrased was "When I get back, I want to look at our inventory and not see any of this stuff," motioning to the warehouse floor full of staged shipments. As soon as the boss was gone, Dick went into the inventory system and deleted everything. No inventory in the system. No record of where the stuff was supposed to be shipped. Gone! His counterpart on the next shift came in the next day and had to recreate everything from paper records. Unfortunately, rather than letting folks like that go from management positions, the company transferred them around so the pain never ends.

    * It actually was his name and I called him that every day.

  18. Antron Argaiv Silver badge

    DG/UX: I designed a low cost Token Ring card for those AViiON systems.

    Sadly, they came to a premature end when Motorola decided that there wasn't enough of a market for their 88000 processor, around which the systems were designed.

  19. The Mighty Spang

    way back on a Dec Vax

    operator was deleting a users account. now forgive me ive not touched vax syntax in a long time. IIRC he was trying to do

    del [...]*.*.*

    which should have deleted all the files in this folder and all folders below but did this

    del [*...]*.*.*

    which started at the root of the drive deleting everything under it... oops. back to the mag tapes.

  20. Ima Ballsy
    Facepalm

    Arggggg ....

    Been there, done that, got the T-Shirt .... Ain't no fun.

    Any *nix admin worth their salt has done this ONCE at least in their lifetime ...

  21. Anonymous Coward
    Anonymous Coward

    DEC PDP11 upgrade

    I had a customer upgrade their PDP 11/23 to a 11/23+ ... the DEC engineer came in to do it and in the process re-imaged the boot disk deleting all of our applications ... so I got a phone call about 4pm.

    The customer (a hospital lab) setup a 2400 baud modem, I dialed in, logged in, and spent the next six hours logged in and talking to the customer on another line getting him to load the various tapes and RL02 disks to rebuild everything and restore their data.

    Of course I didn't get paid overtime because that was my job, but the DEC engineer stayed at the lab to see the installation completed and got his overtime.

  22. Anonymous Coward
    Anonymous Coward

    ...then there's backup stories...

    ....like the AS/400 upgrade where we were upgrading the memory and all the disk storage and the OS version...all in a weekend. Near the end of the process, we needed to get some stuff off the last tape backup.

    *

    So then we found out that NONE of the tape backups was readable!!! No one had ever tested a restore!!!

    *

    It all worked out all right in the end....but a lesson there....somewhere!

    1. PickledAardvark

      Re: ...then there's backup stories...

      "So then we found out that NONE of the tape backups was readable!"

      I've been there too on NetWare/ArcServe.

      Write-only backups are secure. You can toss the tape in a bin and nobody can steal your data...

      1. Anonymous Coward
        Anonymous Coward

        Re: ...then there's backup stories...

        ARCserve did have the verify options.. if you had the time :) We used to get lots of calls into support on that topic though, that, and tape rotation scheduling :)

    2. Anonymous Coward
      Anonymous Coward

      Re: ...then there's backup stories...

      Many years ago (pre internet times) a client phones at 5:30 Friday afternoon. It was the IT guy wanting to run through the steps involved in recovering from a backup. Their US headquarters had a hard disk fail on their accounting system. He was talking the Financial Controller through a recovery and while he knew his stuff he just wanted to double check everything.

      8pm the same night the phone rang again - how soon could I fly to the states? Only one of the backup tapes was good. The financial controller had put the sole remaining good backup tape in the drive, then popped out to get a bite to eat at 7pm because it was going to be a late night. At 7:30pm the scheduled backup process copied the corrupted database over the only remaining backup.

      Saturday was spent on the phone trying to talk them through everything I could think of.

      Sunday afternoon I was sitting in a private jet winging it's way to their US HQ. Three days of very hard work later we'd managed to recreate the accounting database from pieces of corrupted databases and log files. Another private jet ride home - this time the pilot was kind enough to tell me there was a cooler full of beer behind my seat.

    3. Anonymous Coward
      Anonymous Coward

      Re: ...then there's backup stories...

      Once was supporting a MAGIC based system, a Pick-like OS that ran on X86 PCs to give a database environment with network terminals. The server has a "massive" 40 MB disk and a (from memory) a 20 MB tape system for backup. Nice clear instructions were provided for each site on how to run the backups with a proper 1 week 2 Friday tape cycle (6 tapes in total with the Fridays being stored off-site and returned on schedule).

      So one of the machines had a disk crash and died. No problem, pulled out the spare server, set it up for that site, loaded the programs and went to the tapes to reload last nights data. Put in the relevant tape, pressed the restore, and up popped the message, "Load Tape 1". When queried, the operator said that this message had started to pop up a short while back - "Load Tape 2", and as there was not allocated "Tape 2" he just pushed the original tape back in.

      So we quickly tested each of the available tapes, all 4 week tapes were tape 2, and so was the Friday tapes. Disaster, no backup at all. After some panic calls to a data recovery specialist who opined that there was a small chance that we could possibly recover the data from the tapes, emphasizing that the chance was small but the fees weren't, we agreed to send a couple of tapes up to try this option via urgent courier. While hunting around for a courier envelope, in a drawer of the operators desk we found a solitary tape marked Monday.

      We asked the operator about it and he expressed surprise, he wondered where it had got to as he'd mislaid it and had used a new tape to replace it. He wasn't sure if it was before the system started asking for a second tape. So we shoved it in the tape unit, hit restore, and it restored. The data was a week old but at least we had somewhere to start !

      Other story is going way, way back. The first PC's that HP made with a Hard Disk inside had a little oddity that they would only boot from drive A:, so the usual setup was that the hard disk was set to be Drive A:. While I was using the machine and had setup a significant amount of data on it for a test system, I wanted to take a break and someone else came along and decided to use the temporarily unattended machine to format a floppy disk. So the placed a floppy in the drive and typed format A: {enter}. When I arrived back at the machine they were still there wondering why it was taking so long to format a floppy disk...

    4. Anonymous Coward
      Anonymous Coward

      Re: ...then there's backup stories...

      ah.. write only back-ups. Classic

  23. IJD

    PDP 11/44 (running RSX/11 IIRC) at uni shared between a dozen postgrads (including me), who needed superuser privilege to use the video hardware. One who had several accounts wanted to remove all his old files so lazily typed rm [*.*]*.*;* instead of removing one UID at a time. Everything including everyone else's files and the OS (and all system utilities) disappeared, and of course the last backup was months old. Of course the files were still physically there but no longer accessible.

    Luckily I'd been running the system debugger and it was still resident in memory (all 64k of it...) as was the printer driver, so I managed to print out the block allocation of the (20MB) hard drive on the line printer. Said user then had to use the debugger to manually reallocate every block on the disk by hand one at a time to the correct UID (including the OS) and filename. Took him all weekend...

    [no it wasn't me, but I was the defacto sysadmin when people suddenly found they hadn't got any files any more]

  24. Mike Moyle

    Welcome to the club!

    Lesson learned: NEVER decide to "clean up some old files" at 4:30 on a Friday afternoon. You WILL look for shortcuts and it WILL bite you on the ass.

    This was back in the mid-'80s, while I was using/adminning Unix boxen with a "Beginner's Guide to Unix", or some such, in hand because -- while I was the Art Department's tech-geekiest member -- I was not a programmer. Fortunately we were a beta site for the machines so;

    A -- Having called the vendor Friday to announce my stupidity, our rep was in on Monday morning before I even got there, re-installing the system and recovering the work files, and;

    B -- A short time later, the company updated the software to include a GUI for admin tasks, taking the command line out of the hands of fumble-fingered amateurs. (Gee, I wonder what inspired THAT decision...?)

    1. Olivier2553

      Re: Welcome to the club!

      "Lesson learned: NEVER decide to "clean up some old files" at 4:30 on a Friday afternoon. You WILL look for shortcuts and it WILL bite you on the ass."

      Do not do anything of some significance on Friday. At all. Any major change, big operation, etc. must be made by Thursday at the latest, so in case of cock-up, you have the Friday (plus days week-end) to repair it.

  25. John R. Macdonald

    @big_D

    IIRC a mainframe manufacturer, one of the seven dwarves at that time, did something similar. You could 'upgrade' to a faster, and more expensive, machine simply by connecting a wire.

  26. JQW

    I once wiped a large portion of a hard drive after using find with exec rm -rf {} - due to not taking into account the fact that some directories on the system had spaces in them.

  27. FuzzyWuzzys
    Happy

    Got you beat, my story begins with "My Wife's...."

    My Wife's system needed rebuilding, so I took a backup of my wife's photo library, something that has 120 years of family history and 110,000 images....

    Long story short I had to pay £10 for a partition recovery software which recovered 98% of the drive and my wife will not let me touch anything on her machine unless she has taken all her own backups first!

    My heart didn't stop my wife threaten to make it stop, permanently if those photos didn't come back and as any married man will tell you, "Hell hath no fury like a wife who's let he husband touch and break something she treasures."

    1. Chairman of the Bored

      Re: Got you beat, my story begins with "My Wife's...."

      @FuzzyWuzzys, dang.... 10 quid only. Plus flowers, honey-do's and probably a lot of subservient behavior. You got off easy!! Question becomes ... how long was the stay on the couch :(

  28. Will Godfrey Silver badge
    Angel

    Defensive typing

    I've long been in the habit of entering dangerous commands partially in reverse, so in the case of theO/Ps one I've have done:

    '-rf /*.old*'

    then gone back top the start of the line and entered the 'rm' bit.

    1. Criggie

      Re: Defensive typing

      I tend to put "echo " in front of any dodgy command.

      rm -rf $dir/

      becomes

      echo rm -rf $dir/

      and you hopefully notice that $dir is empty. Thanks to NVidia for that one.

  29. sisk

    A couple months ago on my home computer (which has several Linux distros installed and which all share a common /home because I apparently like to make life difficult for myself - and yes, that's as close to a logical reason I have for having multiple distros installed on one machine) I was going to get rid of one of the extraneous Linux installs and use the space to expand the root partition for one of the other distros. I realized I'd typed /dev/sdc2 instead of /dev/sdc3 at the same time that I verified that, yes, I wanted to delete the partition. And sdc2 is where the above mentioned shared /home lives. Doh.

    Fortunately I have a good file server and a cron job running rsync every night, so I didn't actually lose any data, but I think my heart stopped for a few seconds before I realized that.

    1. onefang

      "several Linux distros installed and which all share a common /home because I apparently like to make life difficult for myself"

      It's actually a recommended way of doing things by some distros, to the point they'll do that automatically if you don't tell it otherwise. Separate /home and /boot. The reason for /home is so you can share it between distros. The reason for /boot is that in the past you had to have that near the beginning of the drive, something that hasn't been true for a long time.

      "that's as close to a logical reason I have for having multiple distros installed on one machine"

      I have one machine with over two dozen different operating systems on it, mostly Linux. I have a micro SD card that I call "Magic Pixie Dust", with all those distros on it, so I have a variety of choice if I have to boot into something to fix some one else's computer, demonstrate something, or just offer them a choice of distro to install. I keep the master for that on the second hard drive of my test box, with the boxes usual WIndows / Linux on the first drive.

      I don't share my /home, coz different distros, and different versions of distros, have different versions of applications, some of which change the way their config files work. In particular tmux which seems to change the names of options I use with each release.

      I usually prefer the "one big partition" scheme. Otherwise you invariably run out of space on one, while having plenty of space on the others, then you need to juggle files and symlinks, or resize partitions to get things to fit once more.

  30. Kevin Fairhurst

    Came in to work one Monday to find that the Unix system was borked... on investigation it appeared that a large number of files & folders had gone missing, probably by someone doing an incorrect rm.

    Our systems were shared with our US office who supported the UK outside of our core hours (we were in from 7am to ensure trading was ready for 8am, they were available to field staff until 10pm UK time) so we suspected it was one of our US counterparts who had done it, but had no way to prove it.

    Rather than try and fix anything, they'd gone through and deleted all logs and history entries so we could never find the evidence we needed!

    Restoring the system from a recent backup brought everything back online again, as one would expect!

  31. Anonymous Coward
    Anonymous Coward

    ummm, hasn't anyone ever thought to put in a subroutine that, when the "rm" command has been typed in and enter hit, pops up with a "are you sure you want to delete everything on the system?"

    1. DavidRa

      Sure they did, but the universe invented better idiots

      Of course. However, the incompletely-experienced often choose to force bypass that configuration. For example, a lot of systems aliased rm to "rm -i" by default, which would force interactive confirmations. People would then say "UGH, I hate having to do this" and add their own customisations to their shells/profiles etc:

      unalias rm

      alias rm=rm -f

      Lo and behold, now no silly confirmations, regardless of stupidity/typos/etc.

      1. Stuart Castle Silver badge

        Re: Sure they did, but the universe invented better idiots

        Personally, I don't do stuff like that, and won't use any options on the command line to bypass checks unless I'm writing a script that needs to run unattended.

  32. J.G.Harston Silver badge

    My ISP was going out of buisness and turned off my service. Argh! I screamed - all my (recent) email is on there! Ok, we'll turn it back on for a couple of hours. Logged in and fetched all my email locally. Phew.

    Set up new account on another ISP. Configured my email client. Connected to the new ISP and the email client promptly sync'd my local files with the empty account on the new ISP. ARGGGGGGHHHH!!!!!

    On my PC I still have a big directory on the current drive containing files made up from every allocation cluster on the old drive after I imaged it and extracted all the contents, which I go through from time to time. I still have a gap of about a week in my email archive that I haven't found yet.

  33. Stuart Castle Silver badge

    I have my own story where I've done something I've regretted, but one from a friend first.

    Back in 2011, we needed an inventory system. Because it needed to integrate with various existing systems (some of which were custom designed), myself and a couple of colleagues built it from scratch, using SQL server, Java, PHP and other web technologies. We had V1 up and running, and I was adding equipment to it, using the site we'd set up for this purpose. All of a sudden, I started getting errors saying it couldn't find the database. I checked the server was up. It was. So, I logged on to SQL Server Management Studio, and couldn't find the database. I got our DBA to look at it, as he has access to the backups, which I don't. As far as we could determine, one of my colleagues had renamed the database to a full stop, which was apparently preventing the GUI showing it. The DBA got it back, and immediately locked down the database so no one apart from him and another colleague of mine (not the one who renamed the database) could make structural changes.

    My one was when I first started. We were using Windows NT 4, and Microsoft had just released SP 2. I had been testing it for weeks on my machine, and, in my defence, it had passed all the tests with flying colours. So, I confidently started installing it on staff machines. Roughly half of them failed, and had to be re-installed.

  34. Anonymous Coward
    Anonymous Coward

    When we had a Xerox Desktop Publishing system installed at the engineering company where I worked, the installer came with a huge box of 5¼ floppies to install and configure the software. We were given a list of what functions were available, and the cost of having each one installed, and we picked what we thought would be needed and were billed accordingly.

    Some time later, we decided that we needed a couple of extra functions, so a maintenance engineer came along and logged in, ticked the relevant boxes on the installation list, and logged out. I asked how this was possible, to be told that all the necessary software was actually already installed, it just needed a tick in the right box to activate it.

    About a year later, I needed another function to be switched on, so I tried to log in using the original password, but it had expired. After quite a bit of cogitation, I realised that, if I were to disconnect my workstation from the server and reset the date to the previous year, the old password would let me in, I could switch on whatever I liked, and then reset the date to the correct year and reconnect to the server.

    Over the next couple of years I switched on nearly all of the functions on my particular workstation, and no-one was any the wiser.

    Anon because what I did was a bit naughty, and I wouldn't like to be charged retrospectively for the functions I "misappropriated".

  35. steviebuk Silver badge

    My story, again...

    ....being the security minded person that I am but also no expert I was working at a place with Follow Me printing. On Ricoh devices. Looking through the server one day spotted the option "Purge print jobs on logout" Ooo, that would be a good idea to switch on. Purge them from the MFD, good security and all.

    So I set it and forgot about it. Then calls started to appear in our main 2nd line queue. "My print job has only half come out then disappeared". "I sent my print job a few times but it only partly prints". Oh shit. I realised what had happened, grabbed all the calls and quickly closed them, then fixed the issue (turned the option to purge back off).

    I hadn't put in a change request, although don't think they were enforced at the time. It turned out people would go to the MFD, swipe their card, start printing then swipe to logout before the print job had finished. Or their job was so long that the MFD would timeout their session and automatically log them off. And of course, their print jobs would then be purged.

    Oops.

    No one noticed the calls come in so I kept quiet & quickly closed them. Made up some excuse for the users :) although all those extra closed calls helped my stats for the week.

  36. OzBob

    One I heard from an "old sweat" at my govt employer

    they got a copy of DOS something (2.0 I think) from the vendor (not allowed to build it yourself in those days) and typed the"format" command after putting a floppy in the drive. However with no drive specified, "format" wiped C:. So, send PC back to vendor for re-installation.

  37. Anonymous Coward
    Anonymous Coward

    HP 4 and 8Gb DAT drives could be upgraded to 8 and 16Gb if you happened to have the right tape to put in it. The weirdest one I ever saw was when a vendor had taken components off a motherboard for a customer project. The national rail company had a bunch of servers that should have had two adaptec scsi chips on them but had been modified to only have one chip (essentially de-soldered the IC). Every time they hit a certain back-up speed Netware fell over with a kernel panic. After going on site and ending up physically opening one of the servers up I clocked the missing chip. Next day with a scsi bus analyzer I could see what was happening. Echo caused by the now modified bus. Only kicked off when we had our software running flat out. Fortunately our debug menu allowed you to limit the top speed the system would back up at... problem solved. Another good one was re-conditioned tape drives that vendors send back up with lower CRC checks. Looked identical. Returned same firmware and revisions. Was liable to making write only back ups.... grrr

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like