Developer recovered deleted data with his face – his Poker face Welcome to the fifth edition of Who, Me? It's a new Register weekly column in which readers confess to times their skills fell just a little – or a lot – short of what was required to stop things going pear-shaped. This week, meet "Dom", who back in the 1990s learned the "Clarion" programming language. Which wasn't much use …
More dangerous than delete and * is the the mirror option (as seen in robocopy and equivalents)
If you dont get your source and destination paths dead right , which can be done by as little as not putting a final "/" on the end , robocopy ends up chewing through your colleagues system32 folder like pacman!
Back in the day, I remember, as SysAdmin, the company using VCS (or something like CVS but not as good). One of the developers had linked the repository from his server, rather than a check out. He did some changes and promptly removed what he though was a local copy.
He used rm -rf which then traversed the link and started deleting there. Thankfully, we had nightly backups.
He used rm -rf which then traversed the link
rm doesn't follow symlinks. The only time it does anything that could be thought of as following a symlink is:
ln -s foo /data/foo && rm -rf foo/
This also is not "following the symlink", the user is explicitly referencing the directory pointed to by the symlink foo by including the trailing slash, so rm is being passed the physical directory it points to, not the symlink.
> rm doesn't follow symlinks
That's dangerously misleading. (It's true, but it misses a key piece of information).
rm -rf recurses into mounted directories, at least on Linux. So if someone mounts the company's main fileshare deeply hidden inside a temporary directory, then I delete the temporary directory with "rm -rf temp_dir" then it deletes everything it can on the company's main file share.
I know this from personal experience! Fortunately there were backups from the previous night.
Tip: Never type "rm -rf", do "rm -rf --one-file-system" to prevent this.
"rm -rf recurses into mounted directories"
You must also be aware that if the matching pattern/regex includes '..' it will go UP a level and then down from there!
At least the time I almost suffered from that, it was a recursive chmod command on '.*' to change hidden files/directories permissions from my home location. And I did it as myself, so when it tried to go up and down in to all other's home directories the standard permission bits stopped it.
But for the grace of $DIETY go I...
You must also be aware that if the matching pattern/regex includes '..' it will go UP a level and then down from there!
No, it is specifically prohibited to delete . or .. and has been since at least the first version of POSIX.
Although I did once hose a SCO OpenServer 5 system where a bug meant it didn't guard .. . I wasn't very impressed by this, particularly when it emerged that SCO were aware of the bug, had fixed it in the latest version, but never thought fit to release a patch or even a warning for the earlier versions.
And yes this was the "old" SCO, not the new SCO with the dubious legal claims.
"No, it is specifically prohibited to delete . or .. and has been since at least the first version of POSIX."
My comment is not about deleting the current/parent directory, but that it will follow them. For example, try this in your home directory as a non-destructive example:
ls -R .* | grep '\.\.' | grep -v $USER
You would expect it to show you just your own hidden files? But you might be surprised by what is also there when you filter out the obvious (1st grep looks for the double-dot parent style of name, 2nd grep should ignore your own files, but of course it is not just your hidden file it is matching either!).
You would expect it to show you just your own hidden files? But you might be surprised by what is also there when you filter out the obvious (1st grep looks for the double-dot parent style of name, 2nd grep should ignore your own files, but of course it is not just your hidden file it is matching either!).
I wouldn't be surprised. That is how wildcards are supposed to work. There is a specific safeguard on deleting . and .., i.e. within rm rather than the globbing mechanism.
Many moons ago, an ex-DEC colleague told me about an incident where someone loosed an unguarded recursive delete with root privileges on an inadequately-secured network. Much hilarity ensued, although pretty much the only people laughing were my colleague's group, since *their* servers were configured not to trust root accesses from off-machine.
The UNIX equivalent command waiting to do this is 'dd', also known as destroy data for an obvious reason. dd can be used to copy whole physical HDD as well as to wipe them be reading /dev/zero (or /dev/random for the more paranoid) as the source of data to write to the HDD.
Just be very, VERY, sure you get the if= and of= options the right way round.
Very slightly less risky but equally odd/comparable to robocopy is the rsync command. It won't by default delete files, but the behaviour in whether it copies/creates the main directory is depended on the trailing '/' in source and destination paths in a way I always seem to forget. So the --dry-run option is always my initial addition until I see roughly what it will attempt to do.
working on a pre-sales team where the sales consulting architect clearly had no idea what the changes were in the latest version of the product.
The nonsense passing his lips was spectacular, however I realised that arguing in front of the customers of millons of dollars of contract was probably not in my best interest.
Poker face...had...to....be...maintained....no...matter...what...
Had to write a lot of "clarifications" later on, and spin them as "free improvements"
Poker faces can be essential when working with infrastructure. A couple of years ago we had a network engineer (a good one actually) that managed to down our entire building's network by connecting a mis-configured switch. Cue an emergency "oh shit", a emergency cable removal from the borked switch, then an emergency "excuse calendar" session for when the executives upstairs start inevitably asking why the entire network went down.
we tend to find that when a lot of machines start to exhibit strange behaviour , and its obviously the backroom boys with their rollouts and updates and policies , they dont need an excuse or a poker face because they fix it* and then deny it was them at all **.
* bugrudgingly after the evidence is overwhelming
** unless they need the help of the grunts to visit a large number of machines personally.
Reminds me of the time we had a major network outage in one of our UK DCs, which completely crippled our largest UK client's, customer data feeds. This was something like 7,000 corporate customers, all sending time critical data in from their sites, over various routes into the DC, Internet FTPS, dial-up, VAN networks etc. (this was around 2003, and the client was very out of date with regards to comms).
I wasn't on call that night, but once I arrived in the morning, I was dragged off to a war room that had been set up (this was a small open plan room, with 4 bays of 6 desks each). I joined my team in one bay (we managed the integration services).
The network guys were in the next bay over, and had already denied any changes at their end, and insisted it was our integration platform at fault, and that they were only there to support us in identifying the issue. One of the other bays had various managers, change management, account managers etc.
Considering the network guys were just there 'to help', they seemed very intent on looking at things like firewall settings etc. So I sat in the desk closest to them, so I'd be in ear shot.
I was running various tests, to try and isolated the issue, some routes and protocols worked, others didn't. I narrowed it down to anything using FTP/FTPS via one network route (which covered about 90% of the customers data traffic). This info was then shared with everyone, to help track down the problem. We knew by this point it wasn't out platform.
The network guys were whispering amongst each other, but I overheard one of them saying 'That doesn't look right', while pointing to a screen and another agreeing. The guy at the keyboard rattled a few buttons off, and a few mouse clicks. But they didn't say anything to anyone outside of their team.
I noticed the failed routes suddenly started to work, and I re-sent some of the failed jobs, and all was working again. So it was obvious to me, whatever they'd just done, had fixed the issue. But they were still not saying anything.
So I chirped up, "Thanks guys, whatever you just changed, seems to have fixed the problem.".
They stumbled a bit, and looked rather sheepish, at which point the management came over and asked what had they done to fix the issue, and of course why it was wrong in the first place.
Turned out one of their guys earlier in the week had changed some settings related to FTP traffic on one of the external firewalls, and had saved, but not applied the changes (this wasn't an authorised change either). Then the previous evening, one of the other guys had restarted the firewall to resolve a different issue, which picked up the FTP change and basically broke the platform!
I will happily (ish) hold my hand up if I've done something wrong, but one thing I cannot stand is one person, or worse a team, blaming someone else for a mistake they made, and I'll happily drop them in the s#!t from a great height if I find anyone doing it!
PS: Rant over, sorry for the long post!
PPS: The network management was outsourced to BT around 2001, it's now done inhouse again!
Any time your client is being a humongous idiot a poker face will resolve things that much quicker.
E.g. "If we set the red colour channel above 255 maybe it will get redder"
"Let's see." clickety-click "I'm not seeing it."
Obviously somewhere along the line, the functions we were passing over 255 to contained a line like min(255,r) so going over 255 did nothing, but it was all a magic box to her. Letting her see for herself that it didn't work was far more effective than the twenty seven swearwords and sixteen synonyms for idiot that the request truly warranted.
, but I still have the dent in my tongue from biting it when a colleague assured that the new laser printers would be a direct replacement for their impact printers and that they could still use their three-part stationery with no changes.
I shuffled myself sideways onto a different project sharpish as soon as I got out of the room.
One afternoon I had a sales weasel tell me with a straight face that the laser printers he was selling would have no difficulty coping with 5-part NCR[0] forms. It was the first question I asked, with the intention of getting rid of the guy quickly. I wasn't expecting that kind of bullshit ... So I decided to string the prickguy along for a bit (supper for the Wife & I at The House of Prime Rib in San Francisco ... recommended!), until I got bored mid-morning the next day. Then I asked for a money back guarantee if they didn't work as advertised, including all parts, labor, downtime, and shipping. In his own handwriting. He swore at me quite nicely before leaving, never to be seen darkening our doorway again ...
[0] That's "No Carbon Required", not "National Cash Register" ...
... baffle them with bullshit.
Back in the day, when I was a 9-5 wage slave working both the voice and the data sides of the isle, I knew I could clear any meeting by asking if there was "any progress in the new algorithm for dynamic bandwidth allocation over adaptive differential pulse code modulation" ...
Why not simplify things and statically adapt with a couple of extra lines?
I sometimes enjoy joining in and escalating the BS to a spectacular height, particularly if management are listening. They start to believe almost anything if enough people play along.
+1 for making me laugh.
So how did he actually get out of that one? Need more details.
He cable-tied himself to the underside of one of the trucks heading south, then once safely over the border he changed his name to Jesus and set up a laptop repair business above a knocking-shop in Tijuana. In exchange for undertaking the weekly clap inspections the madam lets him live there rent-free, and occasionally rewards him with the house's fabled 'special extras' whenever he releases an updated version of the in-house booking app, 'Hola, Shagr'.
I hope that's not too much detail for you.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
