Data Center relocate
The company was always looking for cheaper rents, so our NY office after 40 years got elected to be moved. I was brought in to make sure everything (software/hardware) was working. This process took me about an hour, and I signaled everything was OK, The boss said to me are you sure? I said well I could get to every IO device and the software is working as it should be, He said how about testing out some production jobs? I said well if you want to, my tests although did not execute user code but standard OS interfaces. We got production control to try jobs out and of course no issue. I asked if I could go home and the boss said not until tomorrow as that is when the tickets were good for (typical cheap company). Somebody said lets order pizza. I said no thanks I wouldn't say I like NY pizza. I got razzed on by the group. I asked for a menu and chose a salad. Everybody was gorging themselves on PIZZA. I suggested the network people made sure all the network connections worked; they looked at me like I was asking them to work. The network people came back in 3 hours and assured us that everything was OK. I said fine let's start the lines up between Chicago (HQ) and New York, and we stressed them, and everything was OK. I said well there is nothing left for me to do, I am taking the next flight out, anyone, want to join me? I got a silent stare from the boss. So I took the next plane out and charged the fare difference to the company.
Fast forward two days. There were some strange issues with the lines between NY and Chicago. The hardware said it was a software issue. I started to run traces trying to figure out what was going on. I got IBM involved as we were doing some network stuff that IBM said should not cause any issues. My boss said to up the severity to 1, I looked at him and said I don't think so, he insisted, so I did. When you up a problem to severity 1 with IBM, things happen like people are dispatched from where ever they work to your place. I have seen 10 IBMers show up for severity one issues. IBM took its time, but at least I had a live IBMer on the phone for a solid 8 hours. IBM could not figure it out, but they were leaning towards it being a hardware problem. The biggest issue is that it seemed to happen at random times. One of the NY network people asked if it could have anything to do with the elevator?
That is what is called a bingo moment. We set up a test to see if that was the case (elevator door opening). Sure enough, NY AT&T did not use shielded wiring (although they were supposed to). Once they put shielded cabling in the elevator door did not cause the line to go berserk. I was happy they found it, but it was an exhausting 16 hours for me, and I needed a drink. I went down to the bar in the building and had two drinks and came back up to get my winter coat to go home. I looked in on my boss and told him I would be late coming in the next day. He said sure, and I went back. The following day at 8 AM I got a call from my boss saying I had to get into work as there was going to be a finger pointing meeting and I had to be there to keep any fingers pointed at the software group. I was pretty tired but went in any way. The meeting was held in the boardroom, and people were fidgeting as they didn't want to be blamed. The meeting started, and everybody froze, so the CEO of the company launched the finger-pointing. Some of the people that were in attendance were on the phone. After 3 hours I was finally in the spotlight, and they started in on me. I asked if everybody had their say about the software. They said yes, so I launched into what I did and the results I had gotten, all were positive and no error. Then someone asked why we haven't seen this error before but if we had why wasn't IBM able to fix it months ago? I looked t them and said watch the problem was NOT on my side it was a hardware/network issue that occurred because of a wire not being shielded properly, and that was not a software issue, but a wiring issue and I suggested they blame AT&T. That sort of shut down the meeting and I asked to be excused and left and went home for some much-needed sleep.