back to article Rookie almost wipes customer's entire inventory – unbeknownst to sysadmin

Welcome once again to El Reg’s weekly instalment of Who, Me?, where readers get monumental cock-ups and heart-stopping near-misses off their chests. This week, Reginald tells us a personal horror story from the ‘80s, when he worked for what was then a top five minicomputer biz. At the time, he was on site at a customer – a …

Page:

  1. Aladdin Sane

    If rm -rf /* doesn't delete anything valuable

    Did it even happen?

    1. Zippy´s Sausage Factory

      Re: If rm -rf /* doesn't delete anything valuable

      No, no it didn't.

      At least, that's my story, and I'm sticking to it...

    2. Anonymous Coward
      Anonymous Coward

      Re: If rm -rf /* doesn't delete anything valuable

      If a domain server falls over in a forest, does it make a sound?

      1. bpfh

        Re: If rm -rf /* doesn't delete anything valuable

        It makes the sound of a thousand wailing users marching to the Helldesk office armed with torches and pitchforks...

    3. Jim 59

      Re: If rm -rf /* doesn't delete anything valuable

      Eh? As I read it, Reginald kicked off the rm -rf /*, then hit the power switch before it deleted too much. The tape rescue revealed that "everything down to /dev" had been deleted, ie. everything in / beginnind a,b,c and some d. On a modern system that might include /boot and /bin, but evidently was not a total disaster on Reg's server.

      1. Aladdin Sane

        Re: If rm -rf /* doesn't delete anything valuable

        That whooshing noise is the joke flying past you.

        1. Prst. V.Jeltz Silver badge

          Re: If rm -rf /* doesn't delete anything valuable

          @Aladdin (see what i did there?)

          Past who?

  2. Anonymous Coward
    Anonymous Coward

    I remember discovering the hard way that when you delete an email account in Thunderbird and it asks if you want to delete all the files associated with it it actually means do you want to delete the entire directory tree below where the account is stored .... so, as I discovered, saying "yes" when the reason you are deleting the account is because you'd just created it in the wrong place in the the directory tree is not a good idea - instead of just deleting the new account I nuked all the data associated with all our family email accounts!

    1. bpfh
      Mushroom

      Been there, done that :(

      And wished that the design team who signed off on this would always be served stale beer for ever and ever. Amen.

    2. jcitron

      Ah yes. I discovered that myself the hard way.

  3. defiler

    And then billed 3 extra hours?

    Can't be any more dishonest than the IBM guys who would lock themselves in the computer room, eat their sandwiches, read the paper for an hours, and then flick a switch to "upgrade" the hard disc capacity... Apparently...

    1. Anonymous Coward
      Anonymous Coward

      Re: And then billed 3 extra hours?

      Ahh...the good old days where you paid IBM and got something useful at the end of it.

      Sure they still ripped you off, but that's nothing new.

    2. big_D Silver badge

      Re: And then billed 3 extra hours?

      Was it IBM or ICL that "upgraded" to a faster machine by removing a resistor board?

      1. smudge

        Re: And then billed 3 extra hours?

        CTL (later ITL) doubled your memory size by connecting up a wire...

        1. Martin an gof Silver badge

          Re: And then billed 3 extra hours?

          doubled your memory size by connecting up a wire...

          Seems to be fairly common in all sorts of areas. Three car-related ones. My mum had a Renault 5 back when we were changing from Leaded to unleaded petrol. The car as delivered used four-star and the upgrade to unleaded (which was lower octane) essentially involved connecting up a jumper wire under the bonnet somewhere.

          Likewise, a friend of mine had a Passat. He discovered that replacing the (?)indicator stalk for one which included the control buttons for cruise control, then flipping a bit somewhere (via the OBD socket connected to his laptop) enabled cruise control; i.e. all the other prerequisites (presumably sensors and things) were already installed and working.

          My own car had a problem with the "leather" on the steering wheel - it started peeling off like bad sunburn. It was only a few months old, so this was fixed by the dealer, but when I got it back I discovered that cruise control had been disabled. Again, turned out to be a bit-flip in the ECU rather than something simple like the garage forgetting to connect the buttons up after swapping the steering wheel.

          Given a few minutes I could probably think of dozens of other examples. I'm not sure if I think it's a good thing or not...

          M.

          1. Antron Argaiv Silver badge
            Unhappy

            @Martin an gof

            ...car-related ones...

            Funny...I'm going through this exact thing right now with the wife's Jeep Wrangler.

            Took it to the dealer (I know...read on) several months back, for [Takata] airbag replacement recall, and just a few weeks ago, discovered that the fog lights no longer turn on.

            I have been all through the wiring and such, nothing's damaged, and there's even voltage *at* the lights, but not enough current to turn them on. Seems they're controlled by the computer, and there's an option bit in the memory which is set if you have paid for fog lights, and cleared if you have not. So, even if you add the control stalk with the fog light switch, and add the wiring, the damn things won't work unless the computer has that bit set.

            So...they *used* to work, car came with them, but ever since the dealer changed the airbag, they...don't. I'm replacing the control stalk on the off chance the switch is bad ($90) but my money's on the dealer reprogramming with the wrong image or something when they changed the airbags.

            If I have to go back to the dealer, I will be going back with a complaint. Unfortunately, I have not a leg to stand on, and no way to prove they did this, but if they magically make them work, and try to charge me, there will be some discussion. I will be asking them exactly what they did to make the lights work again, and if it involves the "fog light disable" bit, I'm going to raise a stink.

            1. Anonymous Coward
              Anonymous Coward

              Re: @Martin an gof

              Took it to the dealer (I know...read on) several months back, for [Takata] airbag replacement recall, and just a few weeks ago, discovered that the fog lights no longer turn on.

              Two options here - either is likely as the other...

              Firstly, the car was reloaded with the CCF stored at build for that VIN, these can sometimes get corrupted or the VIN can be misread by the OCR / barcode reader - what should happen is that the CCF is downladed from the vehicle prior to any work involving the potential for the CCF to be reloaded - it's actually stored in two places, usually the instrument pack and the central electronic module.

              Secondly, it's possible that the CCF is correct from build but someone has upgraded prior to sale (again, the CCF should have been downloaded from the vehicle and stored locally), this usually happens when the dealer buys a load of 'base' spec vehicles then upgrades them prior to sale - charging the customer for 'extras' and getting a kickback from the manufacturer for hitting 'accessories' sales targets.

              A prime example of this is Motab cars having PDC added as accessory fit - because safety!!!, even though the cost of factory fit is negligible (to the dealer) and is more integrated with the other vehicle systems.

              1. Martin an gof Silver badge

                Re: @Martin an gof

                he car was reloaded with the CCF stored at build for that VIN

                In terms of my car, what I didn't mention was that the thing had been back to the dealer several times from new due to sudden unexplained and temporary loss of power. You know, the sort of thing where you'd put your foot down, the thing would start going, then hesitate for a few seconds, then carry on as if nothing had happened.

                Mostly what the dealer did - apparently in consultation "with France" - was re-map the ECU and clock up quite a lot of driving miles while doing so.

                This was until the car pretty much failed on my wife an hour from home and we had to pay to have the car recovered. Taking it to a different dealer, they spotted that the vacuum switch on the turbo was loose - had probably never been fitted properly when manufactured - and had rattled around and eventually cracked (it's part of some bigger component apparently). Fixing this solved everything, but I never worked out what they did with the ECU mappings, and I suspect that the original garage, which was the one which later also replaced the steering wheel, re-loaded something incorrectly, hence defeating cruise control.

                On the bright side, the car has now done some 155,000 miles, still regularly achieves well over 60mpg (62 - 65 being my "normal" range, but then I do a lot of motorway driving) and hasn't had any other "odd" problems, just the usual wear-and-tear. And a completely rusted-through bracket under the radiator. I did mention it was French :-)

                M.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: @Martin an gof

                  That would generally cause a DTC of 'commanded position unreachable' or something along those lines...

                  is your car PSA French or 'Diamond / Nissan' French - you have my sympathy in both cases, however you're more likely to get things resolved to your satisfaction if it's a PSA car combined with an appropriately motivated tech...

                  1. Martin an gof Silver badge

                    Re: @Martin an gof

                    more likely to get things resolved to your satisfaction if it's a PSA car combined with an appropriately motivated tech

                    Diamond/Nissan.

                    I don't usually take my cars to the dealer - I have a "bloke in a backstreet" who is absolutely fantastic that I've been using for 20+ years now. I only used the dealers for those issues because the car was still relatively new and hence under warranty. On more than one occasion my bloke has had to repair something that a dealer had previously "fixed", the classic being a clutch on a second-hand car bought from a main dealer.

                    The original failed within a month of driving away from the forecourt so the dealer "fixed" it, and within six months it had failed again. It wasn't a problem with the clutch per-se, it was (as my "bloke" discovered) that when they screwed the thing back together they had used seventeen almost completely random screws, only a half a dozen of which were the proper length and taper (who knew?) and many of which had therefore failed. Even when I plonked the broken clutch and collection of screws on the desk of the dealer's service manager, it took a good hour of wrangling before they would agree to refund most of the repair cost.

                    M.

                    1. big_D Silver badge

                      Re: @Martin an gof

                      Main Dealers can be a pain... Or downright dangerous.

                      I had a Toyota and the dealer did the first service... Forgot to put the oil cap back on afterwards, by the time I got half way home, the engine bay was covered in oil and it was bellowing smoke.

                      Then, the next time, they bled the brakes and forgot to reseal the lines! That was a scary ride home!

                      The third time, they left a pair of pliers under the hood.

          2. Anonymous Coward
            Anonymous Coward

            Re: And then billed 3 extra hours?

            "The car as delivered used four-star and the upgrade to unleaded (which was lower octane) essentially involved connecting up a jumper wire under the bonnet somewhere."

            That's a perfectly legit example of when it should be done; it's a configuration change to meet different conditions of use, and unleaded wasn't an upgrade; if anything mpg goes down.

            Better than having to spend time in a garage while either an ECU was remapped or the ignition advance and retard was replaced.

            My boiler has something similar to switch from methane to propane, though there is a jet has to changed as well.

          3. Martin-73 Silver badge

            Re: And then billed 3 extra hours?

            indeed, and the same is true of base model GM cars (Vauxhall/Opel/Holden)... or was up until the early naughties. Most of the kit for the higher spec was there, but either not connected, or a link removed, but ONLY on SOME cars (it seemed random)

            Turns out it was a manufacturing thing, they'd estimate demand for each trim level, and if they had any uncertainty, equip more vehicles at the higher spec, and then disable stuff to downgrade if needed.

            I got UNlucky with my Corsa, 2 wire harness to the doors instead of the full 14 wire one, so electric mirrors/windows would've been hard to retrofit :(

          4. Glenturret Single Malt

            Re: And then billed 3 extra hours?

            I know for a fact that I am not the first to put the wrong fuel in the tank soon after changing cars. On this occasion, I realised almost immediately that I was putting petrol in a diesel tank (not as bad as the other way round, thankfully). Since the tank was pretty low, I was able to fill it with about 50 litres of diesel and the subsequent receipt showed I had put in about 450 ml of petrol which turned out to to have no discernable effect on the car's performance which subsequently gave me 150,000 miles of sterling service.

      2. Chris Miller

        @big_D

        Was it IBM or ICL that "upgraded" to a faster machine by removing a resistor board?

        Pretty much all the mainframe manufacturers (IBM plus the BUNCH - Burroughs, Univac, NCR, CDC and Honeywell - ICL didn't appear because this was US jargon!) did this. It actually made economic sense for both manufacturer and customer, mostly because there was just a single type of system to be built rather than half-a-dozen, giving economies of scale (and making upgrades really, really simple).

        Back in the day (c.1980) we ran a Honeywell mainframe, middle of a range of 5. Part of the maintenance contract (excruciatingly expensive, this was actually where most of the money was made - you could get a good deal on hardware, but there was rarely any negotiating on maintenance) was a visit every other week by an engineer to run diagnostic tests. To save himself time, he would reach inside the machine to the 'secret' microswitch that turned it into a top-of-the-range model. Of course, the operators soon sussed this out, which meant that work scheduled for an entire weekend could be accelerated to complete in under a day, leaving extra pub time ...

        1. Aladdin Sane

          Re: @big_D

          Tesla still do this.

          1. Anonymous Custard
            Boffin

            Re: @big_D

            Not to mention Intel and fuses...

        2. Antron Argaiv Silver badge
          Thumb Up

          Re: @big_D

          This was true even with IBM's relay powered plugboard wired unit record equipment (with which I worked in high school -- I'm a sucker for last chances on dinosaurs).

          The guy who ran the facility says "don't do this when IBM's around, but...", then opens up the back, pulls out a relay and the machine magically starts running twice as fast. He said the only drawback was that the motor used couldn't handle the higher speed for long periods, so you had to be judicious about when you did it.

          IBM 402 Accounting Machine, for those who share my love of obsolete machinery. Big as a VW beetle and probably weighed twice as much. When "big iron" really was.

          1. cortland

            re "big iron"...

            When I retired from the Amy I walked into a job in electromagnetic compatibility at Wang Labs, in Massachusetts. It took three of us to wheel one of the TEMPEST machines up the ramp into the test chamber; HDD the size of a Fiat, outside, with cables that were like wrestling world-record Boa Constrictors.

            1. big_D Silver badge

              @cortland, I remember working at a place, where we had just gone from VAX to MicroVAX and hat a box the size of a desk drawer unit, which was a VAX with 4GB storage, a huge amount in those days.

              We then got a contract to run an IBM installation for another customer and their hardware started to arrive, there was a huge (6' high, 2' - 3' wide and the same deep). I asked what that was, the IBM op said "a DASD", I was impressed, if my desktop had a 120MB drive and the MicroVAX had 4GB, then this must have had a HUGE capacity. When I pointed at the MicroVAX and said, well that has 4GB, what capacity does this thing have, being at least 12 times the size of MicroVAX and just being the drive? He turned a little red and mumbled 500MB.

          2. jcitron

            Re: @big_D

            One of these:

            http://www.righto.com/2017/04/1950s-tax-preparation-plugboard.html

            They are quite amazing beasts with all the switches, relays and mechanical parts.

            I saw one of these in the hallway of a company I worked at as it was being carted off to scrap. I didn't know what it was at the time, sadly. You're right it was about the size of a VW Beetle!

        3. Antron Argaiv Silver badge

          Re: @big_D

          ...and not only the mainframe manufacturers.

          DG processor boards had all kinds of little option jumpers. On the newer designs, it was microcode PROMs, so they started covering them with epoxy when the customers figured out that they could change instruction sets (we had commercial and scientific, IIRC, and one cost more than the other) by changing microcode.

          I worked in communications and networking, but even then, there were boards with fewer UARTs which could be upgraded simply by adding the missing parts (they filled the empty holes with solder to make it harder).

          1. ChrisC Silver badge

            Re: @big_D

            "they filled the empty holes with solder to make it harder"

            Whilst that may have been the outcome, I doubt it was the intention. For through-hole PCBs which are being wave soldered, the only reliable way to prevent empty holes from being bridged by solder is to temporarily cover them with kapton tape or similar. As this is an additional manufacturing operation which costs both time and materials, and thus ultimately money, it's the sort of thing which tends to be done *only* if those empty holes need to still be empty after the wave soldering has been completed - e.g. to allow components to be fitted to the underside of the PCB.

          2. jcitron

            Re: @big_D

            This is very familiar. I worked on Visual and Ontel terminals with motherboard and option boards like that. The difference was sometimes only an EPROM change or a jumper clip or add to change the model from one to another.

            At the last manufacturing company I worked at, they made digital proofing systems that either had 2400 dpi imaging and ran at a normal speed, or some that had 2540 dpi with a turbo speed. The difference in cost was quite substantial, but the change was only a switch flip on the motherboard and a different EPROM.

            There are many other examples of this even in recent years. There are some video cards by NVidia that could go from GTX gamer video cards to their Quadro professional series cards by adding a jumper or resistor and cutting an etch on the back of the board.

          3. John Brown (no body) Silver badge

            Re: @big_D

            "they filled the empty holes with solder to make it harder"

            They would be otherwise unused but still be plated through holes so when the board goes aver the solder bath, that's what happens. It wasn't done to sypie you, it was just, from their point of view, a pleasing side effect.

            On a similar note, back in the day I added am extra scart socket to my TV by adding just the socket and a few resistors, found a slow motion playback button hidden behind the fascia of a video disk player and upgraded an analogue satellite receiver with a switch and a crystal to get the newer birds and all the newer channels.

        4. Stoneshop

          Re: @big_D

          Pretty much all the mainframe manufacturers (IBM plus the BUNCH - Burroughs, Univac, NCR, CDC and Honeywell - ICL didn't appear because this was US jargon!) did this.

          The upgrade kit to turn a DEC VAX 82x0 into an 83x0 consisted of the digit 3 to put on the front panel and a set of microcode EPROMS containing fewer NOPs. One of the other VAXes required only a backplane jumper to be (re?)moved.

        5. Anonymous Coward
          Anonymous Coward

          Re: @big_D

          ICL machines from 2900 onwards were throttled by microcode. It was necessary to install the correctly licensed version tied to the CPU serial number to change the machine range and throttle / open up the power. This was a hardware engineers job and the installed Operating system was not affected.

      3. aks

        Re: And then billed 3 extra hours?

        as opposed to changing a bit to enable a feature which wasn't previously being paid for?

      4. swm

        Re: And then billed 3 extra hours?

        The original Dartmouth time sharing system ran on a GE-225 with a 18 usec cycle time. A GE customer bought this system but found the current release wouldn't run because Dartmouth had upgraded to a GE-235 and the students (of course) used all of the new nifty features of the new machine. So GE sold the customer a GE-235 but put a wait loop in the exec to slow it down to GE-225 speeds. This code was clearly commented. The customer didn't need the extra speed and was quite pleased with the resulting system.

        Later I noticed that an IBM 407 accounting machine we use for listing cards would miss every third cycle. I discovered that by pulling out the S2 relay the machine would run full speed. I pointed this out to the customer engineer and he said we were only paying for a 100 line/minute machine and not a 150 line/minute machine so we should leave the relay in. So I did.

      5. trolleybus

        Re: And then billed 3 extra hours?

        Nothing unusual about that. Some Burroughs upgrades were simply moving a jumper. No different than changing software settings to change the way a car works - you pay for better performance, you get better performance.

      6. Steve Kellett

        Re: And then billed 3 extra hours?

        ICL 2905 to 2955 upgrade was to remove a jumper wire that slugged the system performance by making the CPU service interrupts continually.

        Later machine families did this in the firmware.

    3. Anonymous Coward
      Anonymous Coward

      Re: And then billed 3 extra hours?

      One of my bosses had installed security cameras in the server room to protect their investments from going walkies.

      When the IBM team arrived to do contracted scheduled maintenence on the machines, boss watched via the cameras.

      Boss was *VERY* unhappy as he watched the foursome sit on their asses, drink beer, eat their lunches (in a room *nobody* was allowed to eat nor drink in), & play cards for hours.

      Ten minutes before the scheduled time was to end the four got to work: using cans of compressed air to blow out the kit, rewiring various things at random with fresh bits of cable, & rebooting everything.

      That was it, the whole of their "maintenence". For which the company I worked for was paying prices that would make King Midas blanche over.

      Boss said nothing, just made copies of the security tape & submitted it with his revised bill for services rendered.

      "I'm not paying you to sit on your arse, suck down lunch, & play cards for hours while you're supposed to be working. I'm not paying $Amount for a set of swapped cables & rebooting. Don't like it, tough shite. Check the tape for why."

      Strangely enough IBM didn't say a damned thing about the company not paying that bill nor quite a few more.

      Moral of the story: make sure those security cameras are filming the scene of the crime so you can use the evidence to pimp slap the folks trying to fuck with you.

    4. GlenP Silver badge

      Re: And then billed 3 extra hours?

      I seem to recall the clock speed on the Commodore PET could be increased by the simple use of a POKE command to a register. This was only safe for certain CPUs though, if you had the "wrong" one the chip would fry itself.

      Biggest con I reckon with PCs was the so called "turbo" button which was nothing of the sort. It was actually a slow-down button for app compatibility.

      1. cortland

        Re: And then billed 3 extra hours?

        Some of the older Samsung monitors sold with Tandy/Radio Shack computers in the 1990's could be made to run at higher resolutions, which resulted (or so I heard) in a monitor catching fire.

      2. Jeffrey Nonken

        Re: And then billed 3 extra hours?

        "Biggest con I reckon with PCs was the so called "turbo" button which was nothing of the sort. It was actually a slow-down button for app compatibility."

        ...Yes, but everybody knew what it was and what it was for. Calling it "Turbo" was marketing, not deception. They had to add it for backwards compatibility with programs written for the original PC designed with delay loops that depended on the CPU's clock speed.

        The computers would have been cheaper to build without the 2-speed design. Not much, but it was a temporary fix for a real problem that the motherboard guys had no control over.

        Calling it a confidence trick seems a bit odd.

    5. usbac Silver badge

      Re: And then billed 3 extra hours?

      A few years back I bought a fancy new digital oscilloscope. The scope was offered in both a 50MHz and 100Mhz version with about $200 difference in price..

      The manufacturer actually went to the trouble of adding parts to make the 50MHz version. They added an RC filter to the input to limit the bandwidth. Someone discovered that you can just lift the capacitor, and you have the 100MHz version (and thus voiding your warranty).

      Then, someone else figured out that it's not even that hard. All you have to do is load the firmware for the 100MHz version, and the filter is disabled by software.

      It still baffles me that they would spend money to make the scope slower? Why not just offer the 100MHz version only?

      1. SImon Hobson Bronze badge

        Re: And then billed 3 extra hours?

        Why not just offer the 100MHz version only?

        That's marketing for you. If they ony sold the 100MHz version then they'd probably have to sell it for about the cost of the 50MHz version. But doing it like that, they might have to sell the 50MHz version for alittle less, they can price the 100MHz version significantly higher - thus increasing profit margins.

      2. fajensen

        Re: And then billed 3 extra hours?

        It still baffles me that they would spend money to make the scope slower? Why not just offer the 100MHz version only?

        Price Bracketing - Most people will prefer to pay more for the higher performance version over the basic cheap(est) version. The existence of a cheap model "justifies" the higher priced ones. For this strategy to work, the cheap version of course has to exist so they will make one with minimum effort invested. Sports bicycle suppliers does this all the time.

        Sometimes suppliers also have several so-so product variants which merely exists to funnel sales towards "The Original", "The One That Just Works", like Absolute Vodka (or Garmin) are doing.

  4. big_D Silver badge

    .cobol

    I had been working on a major change to some code on a COBOL system. For safety, I copied the original files to from *.COB to *.COBOL. I then spent 3 days working out the changes needed, and testing them.

    I got it all working and error free, so I went to delete the original files, but I only got as far as "del *.COB" befiore hitting return. WAH! I copied the .COBOL back to .COB and started over again. As I knew what I wanted to do this time, it only took about a day to re-do what I had deleted.

    Another time, I had been working late on a presentation for my boss and I went to rename it, right click, select Rename from the menu, hand slip, click, file gone! Delete is right above Rename in the bloody menu! Luckily, I had sent a copy to someone to start checking over about 20 minutes earlier, so I had a recent backup to work with! But that is what happens, when the boss comes in at 5:30 in the evening and says he has a presentation at 10am at CeBIT the next day and needs 40 slides...

    1. defiler

      Re: .cobol

      when the boss comes in at 5:30 in the evening and says he has a presentation at 10am at CeBIT the next day and needs 40 slides...

      That pretty-much means your boss is a bellend. Just saying.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like