Tired sysadmin plugged cable into wrong port, unleashed a 'virus' Welcome once more to “Who, me?”, in which Reg readers ‘fess up to messes they made in the pursuit of IT excellence. This week meet “Iqbal” who told us about a Week from Hell he experienced in the year 2010, when working for a law firm. First came “a virus outbreak that infected Windows desktops” that required him to travel …
Once upon a time, I crashed a Cisco 6509 core switch by connecting a new, yet-to-be-configured Netscreen firewall into it-- trust and untrust ports both. I hadn't realized that model firewall shipped in "transparent" mode, so it formed a loop on the switch.
I plugged the second interface in and, maybe two seconds later, every port indicator on the 6509 went dark and I heard some relays tick-over. Portfast was enabled on the switch ports.
After disconnecting the untrust port and configuring the firewall for NAT/routed mode, I was able to reconnect the untrust port without the switch falling over, so it wasn't electrical.
It probably didn't crash from the storm, either, but from a bug in the firmware-- I can't imagine a multi-gigabit, enterprise switch crashing from a measly 100 Mbps loop, but that's what happened.
Our 2950 / 2960 switches are very old, so they only do "old fashioned" Spanning Tree - the protocol which protects against loopback. Old fashioned STP takes about 30 seconds in "learning" (blocking) mode before it transitions to "passing". We have had occasional problems with this, particularly kit which wants a network as soon as it's booting. To this end it is possible to put switch ports into "fast" mode, which allegedly bypasses the learning phase but still allows STP to work. I can see how this might allow a loopback storm to begin... I think.
M.
Bet the little darlings never own up to it though!
I had to deal with a 'network' related issue at a police force, they decided to unplug a span cable to a voice recorder so no calls recorded. They never admitted to it, however it worked before a certain key card accessed the room and stopped shortly afterwards..
"they decided to unplug a span cable to a voice recorder so no calls recorded"
I can think of half a dozen ways of ensuring that pulling that kind of shit would result in a number of instant alarms along with some methods of physically securing the ports (tamper-evident) to ensure casual interference is not an option.
That said, that some rozzers would try perverting the course of justice doesn't really surprise me, nor does the matter that others would cover for them ("good cops" who cover for "bad cops", aren't)
We just used to add computers we were using to different rooms thus sending the network admin on wild goose chases trying to figure out who was saturating the internet connection (a whole 10meg back in 2000 when that was enough for a whole college). Happy times...(for us at least)
"how to screw a network in 30 seconds".
A schoolfriend of mine apparently took out the network (which was only one large classroom / computer lab due to it being the eighties) by buying a 3.5mm-jack-to-BNC adaptor from Tandy and plugging his Sony Walkman into the network.
This post has been deleted by its author
I almost took some internal tech down last year. My Crime? Opening a page that had recently been released to the live interface that was hidden away in the settings and then going for lunch.
When I got back, I found out that my PC has been given a hard shut down as the devs had figured out I was the only person who had access to the page and could have it open.
It was sending so many requests to the DB that everything else had come to a halt.
"That's a normal one by little shits in schools according to my bro who was an IT manager at one until recently."
loopback-detect enable
loopback-detect action block
If you have a network where people pull this kind of shit, then having switches which can cope is not optional. - if not on a per-port level then at least on a per-segment one.
We had someone attempt to get their own smartphones on the network by plugging in a D-link SoHo router which also started generously handing out IP addresses. The corporate DNS servers (QIP?) then stopped doling out IP addresses for that subnet meaning a number of the laboratories went down.
We never found out who did it, but you'd expect scientists to know better -->
"We never found out who did it, but you'd expect scientists to know better"
Scientists are like anyone else - they're good at what they do, but don't expect them to know anything about other stuff, like electrical thingies(*) or networking, which is why any shit like that gets an instant lockout from our network.
(*) Yes, we've even caught the occasional one using nails to replace fuses that kept blowing.
I found a home router in our network a couple years ago being used as a hub.
Still happily offering to hand out IP addresses and collecting followers
We have a projector at work which hands out DHCP even though its address is configured manually and the DHCP function shows "off" in all the relevant menus. Panasonic just deny it's possible.
We also found some "unknown" devices on the network recently. Turns out that the one Mac on the network would happily "share" its network connection with any iDevice nearby. This one was possible to turn off, but unfortunately some users of that machine need Admin (on the machine) rights, so there's nothing to stop them turning it back on again. We suspect it was simply the easiest way to download files from an iPad...
Our "pool" addresses were always banned from external access, but we've turned them off altogether. If DHCP doesn't have your MAC, you don't get an IP. It's only a minor pain when setting up new kit.
M.
"Make the switch port learn the mac and then lock up when it changes"
Not just that, but "only one mac allowed on the port", which discourages people pulling games like plugging in dumb hubs and "only authorised macs allowed", which stops them plugging any old shit in and "802.1x", which stops them mac spoofing to get around the previous rule.
Ports stay disabled until unplugged and authorised equipment is plugged back in.
On the other hand anything acting as a DHCP server gets the port permanently knocked into a "naughty step" vlan until we have a discussion with the user. Users who wander around the building plugging into random ports to get around this policy find themselves unpopular with their peers for some reason.
We have a projector at work which hands out DHCP even though its address is configured manually and the DHCP function shows "off" in all the relevant menus. Panasonic just deny it's possible.
Had the same problem with an extra wireless access point at home. DHCP off but still handing out addresses. No, can't happen according to vendor.
At the call centre offices of a previous employer, there was an area of meeting rooms (for impressing prospective customers) with unsecured public wifi. I suspect there were a lot of things wrong with the setup overall, but it all went very wrong when an unidentified helldesk grunt decided to resolve a complaint of slow performance by plugging a corporate network cable into the router and exposed the entire estate to any passing pedestrian with a wifi enabled device. Given that the corporate philosophy for that particular office was to remove authentication from any system that didn't absolutely need it, that also meant open access to vast swathes of corporate data and systems.
The immediate fix when I reported it to the InfoSec head (because he'd bailed me out of a hole with a customer recently and we were quite good pals) was to remove the cable. The local head of IT was very disgruntled when I also asked what was being done to stop it happening again. "We've told people not to do it". Every single person, including visitors who might be in one of those rooms? SIlence.....
The immediate fix when I reported it to the InfoSec head (because he'd bailed me out of a hole with a customer recently and we were quite good pals) was to remove the cable. The local head of IT was very disgruntled when I also asked what was being done to stop it happening again. "We've told people not to do it". Every single person, including visitors who might be in one of those rooms? SIlence.....
That's one reason why I have a pair of GreenLee 727 cutters in my network toolbag that's capable of cutting through 6 gauge stranded copper cable (the fat stuff that's used for DC power)- It's big and mean looking, and very dramatic when it's used to neat *snicker-snack* the offending cable in front of the dolt that decided it was a good idea to do that. (along with mentioning that it'll go through fingers just as easily to drive the point home...)
On a side tangent, we had a vendor decide to plug their crappy D-link whatsit into our corporate network, and wondered why the port got shut down and twenty minutes later one of the network admins came strolling in to collect the patch cable...
Those 'pass through' ports are, in my experience, often 10 Mbit/s. So when you've specifically allowed for two gigabit ports per desk, to ensure the thin clients' remote desktop connections glide along smoothly without being snarled up by the ancient phones, it's rather frustrating when, every couple of weeks, the office staff do a tidy up and some smartarse decides to start daisy chaining devices, and you immediately get support tickets for 'help! URGENT!! computers are slow again'. And no one will admit to even realising the cables were there, much less having dared to touch them.
I'm so glad I don't work there anymore.
I used to work in an office that had a huge MS access database shared by lots of people, that would invariable fall over at least once a day resulting in ten minutes of recovery by the IT peeps whilst the rest of us had a break.
When they took smoke breaks off us (as it wasn't fair to the none smokers) I found a handy correlation between database crashes and the network socket my PC was plugged into. All I had to do was 'accidentally' unplug the network lead whilst a query was running and could then have as many smoke breaks as I wanted!
"...When they took smoke breaks off us (as it wasn't fair to the none smokers) I found a handy correlation between database crashes and the network socket my PC was plugged into. All I had to do was 'accidentally' unplug the network lead whilst a query was running and could then have as many smoke breaks as I wanted!.."
I've always argued that smokers - as addicts - are selfish.
Had many an argument with my father-in-law when the law came into force banning smoking in pubs etc. His argument was that "non smokers don't have to go in there" and "it should be optional".
And yeah - regular smoking breaks outside of normal breaks are hardly fair to non-smokers.
I've always argued that smokers - as addicts - are selfish.
Reminds me of a conversation I once had with the CFO of a small firm, he jokingly lectured me on all the extra work I'd get done by not taking smoking breaks, rather missing two small details:
I only ever took smoke breaks by subtracting the time from my lunch hour, so the company didn't lose anything.
Taking those breaks was incredibly helpful for coding, when stuck on a difficult problem being able to clear my head (then fill it with smoke) and re-approach the problem got me through an awful lot of tricky problems.
I did give it up in the end but always felt my productivity took a hit in the process, can't say I really miss the lectures from people who think they know better especially so while they're getting less work done than you.
I once tracked down a loop in our network when we allowed these devices on the network. I really lucked out, I saw an office was moving desks and printers around. And half an hour later we had a broadcast storm. Talk about serendipity! We could have been searching for sometime...
Now we have a strict policy about con\prosumer crap being attached to our network.
As tech support manager I was immensely worried when our team suddenly started getting performance issues, lost connections etc, after all we were connected via a local switch to the core network in the data center next door, what was happening elsewhere on the network.
A couple of phone calls to managers in other departments revealed that the end users and the dev teams were not affected, just tech support (including the network team).
They were scrabbling around like ants when I suggested we treat this like any other LAN performance issue and the sniffer was dragged out. We were getting lots of dropped packets collisions etc and then we performed a TDR tdr (yup it was back in thin coax days) test. the network segment in a 20*40 ft room was over 200 meters long when it only had to connect 13 desks.
We had just had the room refurbished and the desks had lovely deep cable management channels in them.
Sure enough on opening the lids it turned out that just about every team member was using a pair of over long cables to connect to the network. every techie had cabled up their own pc between the over use of long fly leads and the real requirement to use them on the network management station, where we had to be able to pull switches easily for testing, they had managed to cripple our network segment. In a couple of cases releasing the lid of the cabling channel had resulted in an explosion of cables as team members had used a pair of 15 metre cables where 2, 3 metre cables would have done. In another couple the ridiculously long cables had been neatly cable tied together in compact bundles breaking all rules about turning radii All in all it was a demonstration of how not to cable up a set of desks, they would never, ever had done this in a customer office.
A quick phone call to our cable supplier later and we had replaced everything with premium cost 1-3 metre cables and normality was restored. I did have to call in a favour to get the cables at short notice then bury the invoice in another job and the incident was never mentioned outside the room.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
