Sysadmin sank IBM mainframe by going one VM too deep Welcome to another glimpse inside the dark-curtained (in)box that is "Who, me?" – El Reg's confessional column in which readers seek penance for sins of the past. This week, meet "George", who many years ago was one of two sysadmins in the office products division of a three-letter computer company. And was bored. Bored, bored …
Being Swedish people call # a whole lot of things, though the only one that I can think of that any swede would universally understand is "brädgård" that is the Swedish word for "lumber yard". As Dave126 pointed out above as being the Swedish cartographic symbol for a lumberyard which I had no idea of, now I do feel bad about learning things during my vacation so I better stop reading comments!!!
This has to be one of the best Monday mea culpas to date: a protagonist who is absolutely to blame and in circumstances I (and I expect many of us) can identify with. Did Simon leave you this story lined up, or are you just better at it?
I'm sure the only reason I've never done similar is that I've never been in charge of a mainframe. Though it brings to mind a few 1980s-vintage pranks, from when Unix machines trusted each other and would happily share screens (xhost + was a default setting). Or the one on VMS that had the company go into panic mode for a major security breach 'cos I altered my logout to display a logout screen for "SYSTEM".
Here's a thought. These days when bored like that, one can turn to the 'net and browse something - like the day's crop of Reg stories. I wonder if that's substantially reduced the rate of "bored" pranks, and accidents like this?
Back in the early 80s I was sys admin on a Honeywell DPS-6 used by about thirty developers.
I needed to reboot it one evening but I couldn't hang around to perform the boot myself. I had a tiny assembler program that would perform a "level-2 interrupt" . This would skip any shutdown malarkey and perform an immediate reboot.
I placed this command into the batch queue scheduled for 6pm. The other developers assured me they'd be gone by then. So I expected to arrive the following morning to find a freshly rebooted minicomputer.
Unfortunately, since the level-2 interrupt rebooted with so little bureaucracy, there was no chance for the batch processor to remove the job from the queue. The result being that as soon as the machine started up and looked in the queue it found the reboot command there ready to execute again.
This led to an infinitely looping reboot and a bunch of angry early-bird developers when I arrived the next day.
Going off on a tangent to the tangent (what people call the '#' symbol), the '@' symbol goes by rather a lot of names in different countries. Wikipedia has a list. Quite a few countries refer to it as either a monkey or a snail.
Reminds me of many years ago when a young engineer I worked with realised he could fiddle around with the file system on VAX/VMS and created a recursive definition which he thoughtfully called "pandora/box", under which was a folder "pandora", which pointed to - you guessed it, the parent folder. He gleefully demonstrated the recursion and showed pandora/box/pandora/box/pandora/box/pandora/box... etc. However, he was red-faced the next morning when the backup had failed as it had gone into a black hole down the recursion. Not only red-faced but severely reprimanded. However, he ended up with a job in the IT department, probably so they could keep him on a short leash!
Going back 40 years ago, one of my jobs was "build" on CMS on VM/370 where I had to print off the assembly listings and put them in alphabetical order in the racks. I got to know the ops very well, and they were a great help to me.
One of them, showing off, said this was one command you never, ever, use ... and typed "#cp purge prt * all". The phone rang, and when he finished the call he pressed the enter key. Whoops! This purged all of the spool files for every one!
As the command was audited it was clear who had typed the command. He owned up, and management treated this like a disaster recovery test.
What sysadmin, when presented with a VM system, hasn't tried to boot up a second level system (and then a third level system, and a fourth level system, etc.). And, what sysadmin hasn't gotten confused at which VM level they're on and issue the CP SHUTDOWN command? I know I certainly have. Luckily, it was a "Personal" mainframe I was on, so I only shot myself in the foot. Does anyone remember those IBM P/370 machines? (Ok, so it's been a couple of decades ago. Still, they were pretty darned neat.). I lusted after one of the P/390s, but never could acquire one.
Dave
Back in the '80s (when I was working at a University as a research engineer) I came in one Saturday to get some work done and couldn't get the big IBM 43xx box to respond. Figuring I'd missed an annoucement about some scheduled system patching I went home. I later found out that what had happened was that some kid had figured out he could launch another VM inside his VM. And then proceeded to run a third VM inside /that/ VM. Apparently, while creating these nested VMs, he consumed all of the available temporary disk space (what I noticed during my aborted attempt to get some work done: no temp space available). It was the third VM that did him in. The system operators--likely after /they'd/ gotten notified about the temp space being unavailable--finally figured out was was going on and kicked him off the system. His stunt got him kicked out of the honors program he had been in. No word on whether IBM offered him a job or not.
Setting up a second level VM system (in its entirety) is a non-trivial task.
However, it used to be the case that you could create "cloned" VMs by the simple expedient of diving into the VMBLOK/whatever replaced them and changing the name then logging back in to the original again.
Could be handy if you didn't have access to MAINT/DIRMAINT and wanted to muck around. And great fun if you wanted to give auditing a body swerve.
Although if they gave "kids" class A VMs, they deserved all they got.
In IBM's VM/370, CP is the Control Program, which we now call the Hypervisor. It was responsible for creating and running the various guest operating systems. Since every guest operating system up to that point was definitively not personal in any way (think monster OS/VS1 and bigger monster MVS), the nice folks in IBM Cambridge also developed CMS, the Conversational Monitor System. (It was first called the Cambridge Monitor System but IBM marketing said no sorry). This was an IPL-able (aka bootable) OS that provided a facsimile of a "personal mainframe", which you could use to create and edit files, run some utilities, and interact in a limited way with other CMS users. CMS would not run on a bare metal machine - it required a VM/370 CP environment.
The # character (that would be an EBCDIC # thank you) was the default escape character used to signal that the command should be intercepted by the CP rather than the guest OS. when that became necessary, eg to re-IPL. Another way to escape was to hit the 3270 "PA1" key, which would then normally display a status of "CP READ" allowing direct input of CP commands. Just to muddy things a little more, the CP could receive commands directly from CMS as a courtesy default, meaning the #CP prefix wasn't necessarily always strictly required under CMS.
It was common as dirt to run VM/370 as a guest OS under VM/370. There are likely few VM system programmers from that era who haven't done it just for fun but it also served a real purpose: testing new versions of VM/370, testing new important guest operating systems under new versions of VM/370 and testing local modifications to VM/370 were all useful tools. When running VM under VM (under VM etc), the escape character would be different (it was settable as well). When juggling CPs, one was well advised to keep track of one's place on the perilously teetering stack of operating systems and issue commands to the proper CP using the proper escape character (EBCDIC please). Someone not used to this juggling might reflexively type, eg #CP IPL because that's what motor memory has instilled, but such haste can easily cause the teetering stack to vanish in the click of an ENTER key. Oopsie. Don't do that.
The NYC/northern NJ computing landscape of the late 1970's and early 1980's was littered with IBM mainframe shops running VM/370, MVS, VS1 and DOS/VS. There was more mainframe iron there than anyplace else on the planet, it seemed, and everyone who was anyone in the VM world met up at the MVMUA conferences, usually held at MetLife in Manhattan.
-Grayhaired VM/370 systems programmer
It's been many years, but IIRC, the only *safe* way to shut down a second or third level, etc. VM system was to log in as OPERATOR and type SHUTDOWN there.
And you're right, anything not recognised by CMS got passed to CP by default. Once upon a time, I worked for a well-known software company where it seemed to be a point of honour for any software we wrote could detect that it was running in a VM and issue commands to the base operating system via DIAG X'08'. This was a bit of a faff, 'cause you had to grab a chunk of page-aligned storage, get the real address (via LRA) then stick the command in there and fire it off.
Having had a few incidents where the dreaded "Warm Checkpoint area saved" message appeared, we made a point of
a) Never using SHUTDOWN as a command to stop a product running.
b) Intercepted SHUTDOWN as a precaution.
These days you can customise the command classes quite well so it's hard to get killed inadvertently, but having class ABCDEFG in yer VM directory entry is usually a warning that there is a gun pointed at your foot.
That said, once managed to knock over a VM system at IBM Portsmouth three times in one afternoon from a class G machine writing an APPC thing, and some years earlier, at a major chemist's wholesaler twatting about with DIAG X'7C' (apparently they were a bit behind with applying service).
Haven't worked with it for over 20 years, but VM remains my first love as an operating system.
The service bureau I worked for was a MVS and VM shop. All new releases of both VM and MVS were tested under VM.
My VM sysprog friends joked that running MVS under VM was the only proper way to run MVS. Which is now how I run MVS (rel 3.8J), as a guest system on VM/370R6 , using the Hercules emulator on my PCs.
(for those so inclined, there is even APL on MVT available for the Hercules emulator).
Throw in THE (The Hessling Editor) and/or SPFLite and you're back in the heydays of IBM mainframe computing.
re: My VM sysprog friends joked that running MVS under VM was the only proper way to run MVS.
What do you mean that your friends joked? Nowadays it is the only way.
Bare metal z mainframes don't run operating systems anymore.
For many years now, the only way on a z mainframe to run an operating system, such as z/OS, (a successor to MVS,) is to run it in a "logical partition" under the PR/SM hypervisor, which is a successor to a stripped down version of VM.
Thanks! I was going to post some of this stuff (I did a stint at IBM in Cambridge and knew some of the CMS folks), but you did a much better job.
(I was also going to complain about the comma in "IBM's mainframe OS, CMS" in the article. Even if CMS were an OS in its own right, which it isn't, it's far from IBM's only mainframe OS. So "IBM's mainframe OS" should be an adjectival noun phrase, indicating that CMS is simply an IBM mainframe OS; but the comma makes it an appositive, implying CMS is the IBM mainframe OS. And that is obviously wrong. But it would be pedantic of me to mention that, so I won't. Anyway, most writers seem to have trouble with the distinction between adjectival and appositional noun phrases. Apparently they're hard.)
Jeez. This happened in like 1975. And is the result of someone being an idiot – you don’t do testing in a VM with privileges, for reasons exactly like this.
Plus modern z/VM systems let you (optionally but trivially) specify in the configuration that you have to tell it “shutdown systemname” for exactly this reason. The only reason you would NOT have that option set is if (a) you’re an idiot or (b) because you have some automation that doesn’t specify the system name and you’re too lazy to fix it (which is sort of (a) redux).
Typical Reg article: a germ of truth, wildly distorted. I am disappoint. But not surprised.
The article's headline got it wrong.
Don't say the "Sysadmin sank IBM mainframe by going one VM too deep"
The problem isn't that he went one VM too deep. It's that he went one VM too shallow. He killed the first, top level VM.
If he killed the second level VM, it would have taken its children along with it, but left the top level alone.
The headline could read, "Sysadmin sank IBM Mainframe by not going one VM deep enough."
This isn't exceptional for this environment; it's a obvious consequence of running with too many privileges. Normal users should never have more than class G, which does not include the ability to halt the real machine. Running something like this in a privileged machine is proof of nothing more than you are an idiot.
Wrt to VM-inside-VM virtualization, it is possible with VM to run arbitrary numbers of layers. I personally have exeeded 45 layers deep. After the 3rd layer you lose most of the hardware assists, so it gets dog slow, but if you have enough RAM and disk, you can go as far as you can stand.
Not bad for something released in 1964.
Nerds.
Back to the topic at hand. I confess that in the early 80's I was known to make use of the debugging tool within CICS (name and author escapes me at the moment) to write over the shared control blocks with garbage, bringing down the entire control region.
This was necessary because I was ready to test my program, and I didn't want to wait until the Sysadmins cycled CICS overnight, which at the time was the only way to refresh the binaries in the load libraries.
I was never caught (by the sysadmins - my fellow developers were in cahoots with me since they needed their stuff reloaded too).
I believe I heard the story here in the US in the 1980's. I don't follow VM so it might have been earlier. If memory serves me the limiting factor is real storage (I could be wrong). I also heard that someone at IBM got several hundred, again I don't follow VM.
Back in the '70s, as a VM system programmer, I once had three levels of VM/HPO set up just for fun, and occasionally installed or upgraded a new VM system under the old production VM. i.e. two levels.
And I did know someone that apparently had tested at least seven levels of VM, but he worked for IBM, and lots of toys to play with.
I worked on the same system (VM/CMS) in the 90s. We were doing the same thing, running a "guest" VM system for diagnostic use. This time it wasn't developers, but the whole Military engineering company we were working at that got shut down. My colleague "Fred" typed #CP Shutdown. All over a huge, open plan office, heads popped up, a murmur of concern rippled around and we suddenly realized what had happened. We had shut down the mainframe system upon which the whole company worked.
Solution? Just in case this (excellent) 90s system makes a comeback, ensure that the top level host login does not have "MAINT" privileges. Typing #CP Shutdown then has no effect, other than an error message. You're welcome.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
