Dev's telnet tinkering lands him on out-of-hour conference call with CEO, CTO, MD Welcome all, to the merry world of Who, Me?, our weekly trip down memory lane for techies who want to get something off their chest. This week, we hear from "Gavin", who was working at an ISP that had a few thousand point-to-multipoint radio links and had been asked to write a script to backup their configs. "They already had …
... I filed a bug report on a batch of bad EEPROMs that were throwing spurious errors. In the bug report, on a lark (and to see if anyone actually read the bugr), I suggested that it was probably Alpha particles off the heavy metals concentrated from sea water evaporation in the salt pile in Redwood City, which was just off our shipping & receiving dock.
PhD Engineers scurried about for about a week, until I confessed to the joke. I nearly got fired. It's amazing how little highly trained people know about stuff outside their field. Me, I generalize ... seems to keep me saner than most.
Note that back then there WERE some EEPROMS that were contaminated by Alpha particles, but that was caused by a manufacturing error before they were sealed up. If you know anything about such things, you'd know why my hoax was obviously bullshit.
Why bring this up here? In the thirty years since then, I've heard the story of the salt pile in Redwood City ruining electronics "due to Alpha Particles" half a dozen times, at half a dozen companies, in three states, Canada, the UK and Australia. Usually in relation to spurious errors in electronic gear. I suspect the hoax will out-live me by many decades. If you run across it in your meanderings and it causes you any trouble, I apologize ... have a homebrew on me :-)
The PhD's didn't know alphas are stopped by a sheet of paper, let alone x number of walls and plastic chip cases?
It's not that unusual: I once had a spirited argument with a Chemistry and Civil engineering PhD students who apparently didn't know what the law of universal gravitation was. It's only after they looked up what G was they believed me...
Have an up vote!
In the poor chemists defense, in chemistry G is usually something entirely different (Gibbs free energy). To much exposure to one field of science (particularly one involving lots of organic solvents preferably halogenated ones) tend to erode any residual knowledge in other fields.
Nowadays for Advanced IntelAIgent Reality, one is expected to follow an Unfolding and Almighty EMPowering Future Narrative.
Or has that always been they way that it has been and is for Superb and Sublime Future Builders?
Share a Good and Grand Tale to Realise a Great View with Improved Vision.
And there is Stellar Comfort in recognising the Fact that if newly discovered, are Elements of Mankind Experimenting in Virgin Ground which they Command and Control, and if one has simply stumbled into long ago uncovered heavenly practices, is Status Quo ACTion and Internet Working Reaction Mandatory for a Semblance of Continuity with Default Engagement for Leading Prime Operating Systems.
"I do remember reading in a "how to repair PC" type book about RAM errors caused by cosmic particles."
I remember seeing that several times back in the 90's as well. If memory serves (heh), they all said the useful lifetime of most RAM was 10,000 years, by which time cosmic rays would have caused too much damage to the crystalline structure of the memory for it to still be reliable. I guess any deep space probes would be subject to this, mostly because they'll be the only things to survive for 10,000 years.
Errors caused by energetic particles were a really serious worry at some points. People didn't understand that you can't just make chip packages out of stuff, you have to clean the radioactive contaminants out of it first. A lot of early dynamic RAM and some early microprocessors suffered badly from this (this is one reason why a lot of 70s/80s machines had ECC memory even if they were not particularly high-end). That problem is solved, but the cosmic ray problem still happens.
Well, strictly cosmic rays are nuclei and electrons, which penetrate quite a bit better than alphas. But although I seem to remember some Google report saying they identified them as a cause of issues over a fleet of millions of computers, I suspect us normals have little to worry about :-)
EMP E1's on the other hand...
Don't forget that PT Barnum, to keep the crowd moving and not overcrowd the side-show tent, famously posted a sign on the exit of that side-show tent that read: "This way to see the egress." It worked, because many folks wanted to see the another strange creature - the "egress."
> down in Brazil, the gauchos hunted the gazebos to the verge of extinction
Sad memories of South American yesteryear. O how I miss the haunting call of the wild gazebos trumpeting in the gathering gloom of the Amazon dusk, as they thundered through the waving fields of cow-stripping piranhas on their way to the shops to grab a packet of biscuits before closing time.
Now, they just sit on the verge and look mournfully at me as I drive past, silently mouthing pleas for last-minute tinction. Alas! too late.
Some time between 1988 and 1992, we replaced our Honeywell DPS-8/70 with a Honeywell-Bull Dual DPS-90.
So, the system has been installed (that was fun, too), and its running. Honeywell-Bull flies a group of sales and engineering folks to our location to try to sell us ARES, their relational database product that's just been release. The computer room's on the first floor, and the dog-and-pony show is going on in the 3rd-floor conference room. And....the system crashes. Of course, this interrupts the activities on the 3rd floor, so when the sytem administrator, who was my boss, comes in to the computer room, he's followed by one of the OS developers. My boss picks up the phone and calls support. When he explains what has happened, and what the error message says, he's met with disbelief. Now, on the DPS-90, the operator console is actually semi-intelligent, so he hits return enough times to advance the console log far enough out of the console printer to read the last line, and he then reads "......YOU CAN'T GET HERE" to support. After they again dismiss his description, he calls the very conveniently on-site OS developer and hands him the phone and says 'you tell them what it says.' The OS dev tells the support person who he is, that the error does, in fact, read 'you can't get here', and to please transfer him to so-and-so, who we find out later was the author of one of the main subsystems. He then explained the situation: hardware - Dual '-90, 6 FEPs, etc; the software: CP-6 (don't remember which version, probably E00), and the error message; and can he please find it and call us back with a recommendation,and he hangs up. We got a call about 20 minutes later with a suggestion to check some physical connection in a specific cabinet (I don't recall whether it was a connection to the tape controller or disk controller), and lo-and-behold, there was a connector that was only secured on one end and had begun to pull apart at the other end.
After reconnecting, and securing, that connection, we were able to boot the sytem normally.
Long ago, my colleagues and I were following up a mainframe bug report and saw a message on the terminal stating something very much like "if you reached this point you're fucked". The original coder was no longer working with us and, unfortunately, we had to agree with his statement...
Or in at the Start of a New Beginning is a Firm Favourite Resource Registered Here. ?:-) Being Virtually Mentored and Monitored and Freely Available from/for Otherly Sources into Immaculate Services.
Care to Show Future AI Lead with Simple Almighty Instructions/COSMIC AIdDirections for Similar Source Supply?
Where has that got you to now ‽
Familiar Territory or Alien Space?
In Unfamiliar Alien Space Territories, Live Operational Virtual Environments are Engaged and EMPowered to Await and Attend to Every Need with Immaculate Seed and Perfect Feed.
As you must surely imagine to be true is to realise the future is Most Probably a Divine Masterpiece, however, a few words of caution. Future MIsUse and Abuse is a Deadly Serious Sin with Terminal Consequences in Just Reward.
Don't Forget to Always Remember That Simple True Fact for Practically Almighty Fictions Exercising Remote Virtual System Command with Advanced IntelAIgent Controls/Quite Almighty Levers.
And here's a Wonderful Ponder for the Truly Romantic ...
"One should not aim at being possible to understand, but impossible to misunderstand". - Marcus Fabius Quintilian“The most dangerous man to any government is the man who is able to think things out for himself, without regard to the prevailing superstitions and taboos. Almost inevitably he comes to the conclusion that the government he lives under is dishonest, insane and intolerable, and so, if he is romantic, he tries to change it. And even if he is not romantic personally he is very apt to spread discontent among those who are.” …… H.L. Mencken
"adaptability to shifty infrastructure and business knowledge was what kept us going as a business."
Never a truer word said - this is critical if you work in infrastructure at any level. Never forget the CEO's pet customer or the dodgy fibre switch that runs one of the database back ends which everybody keeps forgetting to get budget to change or the natting which had to be done on an old server rather than the router . . . all these little gotchas are part of the job and often make it fun :)
"or the dodgy fibre switch that runs one of the database back ends"
Or in our case the switch which has ended up becoming a core switch just because important things were plugged into it willynilly. Which now has degraded to the point where the only way to connect to it is by manually putting it's MAC into your ARP cache.
Of course the things plugged into it don't have redundant connections and can't possibly be unplugged for any reason, or so I'm told at least.
(I might 'accidentally' depower it at some point just to force the issue)
@El Reg said: "Because the firmware on a lot of the devices wasn't updated for years, their telnet client had a number of quirks, such as it wouldn't ask for a password or would only ask for a password," he said.
This is exactly the sort of situation that "expect" was created to deal with. You send this string, receive that response, send this reply, wait for time outs, etc. "Expect" comes back with whatever exit code you define for each situation which you can analyse from the script you called expect from. The "expect" scripts can be stand alone, or you can embed them directly in bash scripts. The main script logic is done in bash (or something similar) while "expect" handles the interactive parts.
In this case I would have defined an "expect" script which looked for the anticipated response, and if the remote system responded differently I would have logged an error to a file and carried on to the next station. I could then have analysed the error log later to see what happened, and then added another "expect" case to the original script to deal with that situation.
I have used "expect" to deal with a somewhat different problem and found it works quite well. I believe that the most common use for it is automating log-ins on systems that can't use ssh keys or the equivalent for whatever reason.
I'm not saying that I wouldn't have made the same error as the original author, but if I had anticipated the problem then there's a tool which exists specifically to deal with this problem.
There is a possibility that the system our protagonist was using didn't have Tcl installed, and thus he didn't have access to expect. However, your basic point that one should always check the sanity of any replies when querying equipment with scripting is valid ... and an issue that I'm sure many of us reading can commiserate with.
"It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so." --Samuel Langhorne Clemens
If it were done in Cobol you could define the subroutine as "EXPECTS", the expected value(s) could be defined in either an 05 level or an array named "A-SPANISH-INQUISITION" and tag the exception case as a 77 level named "NOBODY" and get to display an absolutely genuine diagnostic of epic legendarity. I got into a lot of trouble once for doing this sort of thing. Totally worth it.
Somewhere amongst my files I have a version of expect I wrote in bash, because I needed the tool but wasn't able to install the stuff needed for the "real" expect.
Its actually a pretty simple thing to write, and I even had to expand it beyond the "real" tool's functionality. That wasn't the hard part, the hard part was the challenge/response logic when you are doing something a lot more complicated than a telnet login - and the challenges & responses change as firmware does.
In this case I was doing automated upgrade/configuration of fiber channel switches, some standalone some embedded in a blade chassis. I was questioned a few times about the level of effort I put in to writing this, but given that doing it manually was taking over an hour per switch and the results lacked consistency (causing problems that required hours of troubleshooting later) and there were well over a hundred new ones on the way that would need doing in the next few months the need was clear to me.
I wonder how long they kept using it after I left. Eventually firmware changes would break the challenge/response and someone would need to fix it. I documented it very well, and encouraged them to look it over and ask me any questions before I left, but that's no guarantee someone made the effort to maintain it.
Our protagonist was already familiar with perl, which has made this kind of thing fairly easy pretty much since the year dot. Not all that secure, perhaps, but easy. (I don't think security was a buzzword the ISP in question was aware of.)
You can assume that someone is taking credit for your work right up until the point it stops working... then they will deny all knowledge of ever seeing it.
Ahem. I have vehemently denied ever seeing some code and having no idea about it ...until I've spotted the header revealing me as the original author. Has happened few times, and I honestly have not recognized any of the code that I had apparently written many years ago.
I honestly have not recognized any of the code that I had apparently written many years ago.
In my travels I've come across a saying along the lines of "Code you wrote 6 months ago may as well have been written by someone else".
I've revisited routines that were very familiar to me many years ago, and now for the life of me cannot figure out how or why it works.
Perl does have Expect but it looks fiddly, and Net::Telnet can also be used like expect, instead maybe the basic login method and a unhealthy dose of good faith in non-standard firmware was used.
> There is a possibility that the system our protagonist was using didn't have Tcl installed, and thus he didn't have access to expect.
Tech.Note:
Expect is wholly distinct from Tcl.
I have used it to solve nasty inter-VM problems with no Tcl anywhere in sight, let alone installed on our stripped-to-the-bone systems.
True. It annoys me when the engineer who's been given a task and tested then gets the blame for shit that went down through no fault of his/her own. You would assume they'd have all been patched for security. He'd done enough testing and they still gave the go ahead.
Reminds me of the time I made sure, at the NHS, some tablet devices were put on the network properly and configured properly. By the ex-engineer who'd defected from the IT department to the specific Trust that hated the IT department, so would try to bypass IT with kit. He knew what he was doing, heck he'd written one of the old remote tools we use to use (using other peoples code mind you). I knew he knew what the passwords still were (they'd never been changed, despite me pointing this out a year or two back. But what do I know, I'm only a "contractor"). So I made sure they were connected properly. I informed the stakeholder and heard nothing.
3 months later an e-mail went out asking about them. I piped up and said I'd bought this up 3 months ago. And then all hell broke lose. They attempted to end my contract, despite me having the evidence I'd e-mailed them, warning them 3 months before. And that I'd warned them months before that, that the likes of the ex-engineer had access to everything still and still knew passwords. Luckily I had two managers that defended me so I stayed but still got a bollocking.
Bullshit. If I hadn't needed the money I really should of walked. Essentially the higher ups fucked up included the stakeholder and because I was a contractor, they wanted to make me the scapegoat.
I'm not sorry to say, I hated the director before that and even more so after. So much so I recently heard he'd had cancer for a year and I'm not sorry to have thought "Shame he never died of it".
There are ways to deal with situations in work. Trying to belittle people isn't one of them.
I've worked both with embedded systems and general software applications that provided wonderfully rich functionality via telnet. In a good number of cases though, the handler for the telnet access appeared to in a single threaded main loop, blocking the functioning of the reset of the system. There were some quite awkward "pauses" in operation :-(
I once worked with an unnamed yet completely vital system which outputted its log over a network connection similar to syslog but not.
We often ran the log viewer app on our workstations over a VPN as it was very useful for development.
After a while we started getting very strange errors on this critical system that we couldn't diagnose - random and long, noticeable delays in execution of the time critical code.
We reported to the manufacturer who spent quite a while looking into this problem.
It turned out that the network part of the log output code ran in the main thread, so our office's dodgy Internet were causing the whole system to hang.
... That I happily decommissioned a god awful Windows 95 machine using a large mallet. It was very satisfying, and it freaked out the secretaries.
I heard it was being told as an example of a non health and safety approved way of ensuring obsolete data was, erm, wiped...
My reply to whiners like that is always: My hours are more expensive. If I ever get trapped into fixing a system with this chip/part again it's more expensive.
Oh and when dealing with stuff that you've been messing with for ages that got a little mangled in the frustration last time you had to build it back out: "hey that's just a little bent, can you fix it?". *SNAP* "Oops, Nope". (Sorry, not sorry)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
