back to article Confessions of a sysadmin

I would like to say that it has been a few days since my last malware infected computer. I have been dealing with a string of these lately, and I’ve had quite enough of them for now, thank you. I would also like to say my network was the epitome of configuration perfection, with every system fully patched, and a team of …

COMMENTS

This topic is closed for new posts.

Page:

  1. Anonymous Coward
    Thumb Up

    Full marks for the hands-up, but...

    Don't "management" ever turn their machines off? Default once a windows update is downloaded is "Install Updates And Shut Down". If they're switching their machines off at night, then they're actively changing that in order to not install the updates.

    B'sides, there's settings in WSUS to not nag for a restart after installing - that should sort them quite happily with auto-updates. Yes, the odd update might cause a little odd behaviour - but the good ol' helldesk "Have you tried restarting your computer?" will rectify in short order.

    1. Trevor_Pott Gold badge

      @AC

      Nope; they never turn thier machines off. We live in a VDI environment, and they *all* have external RDP access. Look for my upcoming articles on VDI; one of the major pains in the neck is that your users can simply "disconnect" instead of rebooting. (Though this can be solved with GPOs...)

      Anyways, I don't want to give too much away, because then I'll be out of material for the VDI article set!

  2. Obvious to me...

    Heard of Solidcore?

    Solidcore have a good solution for this type of 'defenceless' system - At least it used to before it was bought by McAfee. You can still get to the Solidcore website at Solidcore.com for info or look at the McAfee 'Application Control' product.

    Should tick a few boxes - Low system overhead, AV, no signature downloads.

    Might be worth a look.

    I hasten to add that I have no connection (and never have had) to McAfee or Solidcore - I just think Solidcore came up with a good solution.

    1. Trevor_Pott Gold badge

      @Obvious to me...

      Never heard of it, but based on your recommendation I will take the time to check it out. :D

      Honest and true; when you filter the noise out of the signal, the real value of El Reg is in it's readers. <3 commenttards.

  3. Anonymous Coward
    Anonymous Coward

    Ho hum.

    You can do quite a lot to isolate things beyond subnetting. Modern switch hardware has all sorts of separation and even packet blocking features.

    I don't agree it would be a nice idea to move to static IPs for a few machines. That's like saying (and I used to know someone who actually claimed) that DHCP was fine for small networks, but not usable on big ones. Uh, sure mon. DHCP was invented to distribute configuration across the network. It's up to the admin to properly configure the tools, TYVM.

    Then there's the question why large and dangerous machinery requires extremely brittle control software that imposes domain-external requirements on the underlying OS that makes it more brittle. Apparently comes with the choice of using an essentially unsuitable product for the OS. That assessment is justifyable solely on observed "should not happen" problems, qv. obscure patch interactions not relevant to the function of the control device. I'd say the same if I'd observe that on, say, a solaris box.

    1. Anonymous Coward
      Stop

      "Why large and dangerous machinery requires..."

      Quite simple. Because it's bloody expensive kit, and - when it was new - that was just the way the world works. W2k. That sets a rough age to it...

      Just because IT has short asset life, doesn't mean that other kit does too...

      1. Trevor_Pott Gold badge

        @AC 12:52

        Yar. In this case something on the order of 15-20 years...

    2. Anonymous Coward
      FAIL

      Static=Small

      Wow, sounds like it was opposite day for him.

      If anything Static is for small networks.

      DHCP with Mac filtering is the way to go, IMO.

    3. Trevor_Pott Gold badge

      @AC 11:22

      "You can do quite a lot to isolate things beyond subnetting. Modern switch hardware has all sorts of separation and even packet blocking features."

      Yes it does. In fact, even my old-ass semi-managed switches offer VLANing. There are many features that high-end switches (such as Cisco or Procurve) offer that could solve this problem. It's just COMPLETE OVERKILL for this situation.

      Subnetting will put my system “out of reach” for any system under my control, because the OS will honour the subnet and refuse to allow randoms to connect. It will also prevent anyone plugging into my network from simply getting a DHCP address and finding the system. Virus and attackers can configure the network card to NOT respect subnets, and so the subnet really should be combined with a VLAN.

      Being sensible, I will subnet the system *and* VLAN it. Anything beyond that is both totally unnecessary...and not possible with my current switching gear. While I could easily toss together a bit-flinging box to be a router and do more than vlans, I just don’t see what it gets me that simple subnetting and a VLAN don’t.

      My switching gear is okay for an SME; but cisco it is not. Replacing my network infrastructure with Cisco would eat my IT budget for the next two solid years, and I frankly fail to see any benefit. Cisco, (or their trained minions) have yet to show me a single thing that would actually care about doing on my network that their gear can do better than a well configured Linux or BSD box. And for the cost of one of their routers, I could build a multi-system RAIS* 5 (with hot spare) Linux or BSD routing cluster. Since I only have to route gigabit, a bloody Atom can fling all the bits I need.

      *RAIS: Redundant array of inexpensive systems. Fancy clusters on steroids.

    4. informavorette
      Badgers

      why large and dangerous machinery requires extremely brittle control software

      It's kinda company culture problem.

      I recently had to do with the software creation process of one of the biggest engineering manufacturers in the world. They will sell you anything from a single PLC to a nuclear plant. They are full of engineers - you know, real engineers, of the kind who smirk when they hear the programmer wimps call themselves "software engineers". Of course, they cannot get around the fact that their equipment has to be controlled by some kind of software. And of course real engineers who can design a whole power plant can easily oversee the creation of some kind of software for this. The actual coding is done in India anyway.

      Of course there is some kind of QA for the software. It definitely works on the developer's machine, manually tested by hordes of cheap offshore workers. And our manufacturer, unlike Microsoft, cannot afford to sell products with software bugs in them, unless he'd like his nuclear plant to replace Ariane-5 in the software engineering textbooks. So he just goes and tells his customer that he guarantees flawless work - as long as the configuration specifications are met. Which sometimes happen to include a Pentium II or whatever the developer had at the point of writing the software.

      Now imagine you are their customer. You want to purchase a new fully automated production plant for manufacturing your product, worth more than Zimbabwe's annual GDP. Believe me, you won't even glance at the IT specifications for the controller hard- and software. If it requires you to run a Lisp.NET application on a Condor-managed grid of 139 iPads, the cost of finding a specialist who understands this configuration is still only a zillionth of your budget. And besides, there are less than 10 companies in the whole world who can build this for you, and probably none of them offers a better software. So you just buy based on other criteria, and your admin has to make sure your purchase is usable, no matter how. And once you have it working, even if your production plant builder miraculously offered you a new, upgraded version of the controlling software which runs on a newer system, you have no reason at all to change. It will cost a lot, and you are paying an admin to make sure everything runs smoothly.

      Or do you, personally, find the idea of your local nuclear plant updating the controlling software from a working Win2000 version to a new shiny Vista version (bleeding edge, as yet not used outside simulations) appealing?

      1. Anonymous Coward
        Black Helicopters

        Tried and Trusted

        Isn't the UK Navy's Windows for Warships based on W2000?

      2. Anonymous Coward
        Anonymous Coward

        Re: why large and dangerous machinery requires extremely brittle control software

        Thank you for the nice explanation, even if I already knew most of that, and I assume most of us do. I perhaps shouldn't have left the comment stand rhetorically as I did. So, to correct the oversight: I think we, and that's a far more general ``we'' than just the IT janitors guild, but including smirky engineers, senior management, and so on and so forth, must recognize that brittle software itself is inherently dangerous as well as needlessly expensive. And then we need to figure out what to do with it.

        Even if the smirky kind of engineer certainly does have a point about those who cobble together software to the point that outsourcing to india is still an option in the minds of those who commission it. I've been on the receiving end of that, and especially outsourcing the system design (and obstinate project manglers and highly irritating and utterly clueless ``business consultants'') was a key factor to making the project into a spectacular if classic project failure.

        We really need to do better. Cue any comment on software quality by the late E.W. Dijkstra. And it's up to the IT janitors on the receiving end to start communicating this back up the chain and eventually back to the smirky engineers. Read up on management (_The Essential Peter Drucker_ is a nice and condensed overview of the whole thing if a tad abstract and not entirely free of faults) to see how that yanking the chain works, and why management had better listen.

        1. Trevor_Pott Gold badge

          @AC

          Systems Administration is a blue collar gig. The brass never listen to blue collar folks, they just tell 'em to "get it fixed." They don't care about the details of how it gets fixed, or why it happened in the first place; that's why they are hiring you. When was the last time anyone actually paid any attention to what the plumber or the electrician or Janitor said? Only if something bites them in the ass multiple (expensive) times do they begin to ask “why did this happen” as opposed to just barking out the orders “make it better.”

          Sysadmins are digital janitors, and if they agitate too much about things like crappily designed software they risk being replaced; after all, thanks to things like outsourced, offshoring and cloud computing…there’s a massive oversupply of them.

          So every sysadmin has to walk to terrifying line between making enough noise to cover their ass when things go boom, and not speaking out so loudly that when cutbacks roll around theirs is the first face on people’s minds to get rid of.

          1. Anonymous Coward
            Anonymous Coward

            Yes, no, pay attention now.

            Well, that's the point. You have to talk management talk. It's not so much a class issue as a communication issue. Which amounts to the same thing, except the former has a mindset problem tacked onto it, but bear with me.

            It's "IT"'s job to tell "brass" that things work /this/ way, and not /that/ way. Management gets a simple choice, put to them in simple terms. Do you want $very_bad_thing_for_business to happen? Then do that. No? Then do this. In business terms.

            Arguably this interfacing stuff is the line manager's job, but if you are your own line manager, well, sucks to be you. I was, in fact I was the only one left standing and even VP I reported to got booted, leaving me in a void for over a year, and I burned out. Then I read up on management. I'm not there yet, I'm still learning. But getting management right certainly does have hack value.

            1. Trevor_Pott Gold badge
              Unhappy

              @AC

              Agreed. For everything there must be a business case, and it must be presented properly. The issue that blue collar types in general, and Systems Administrators in particular have is that even with the most rational presentation using all the right buzz words and a solid business case...

              ...the brass reserve the right to be batshit crazy. At some point you just have to walk away from the situation and understand that there are some battle you flat out can't win. No matter how rational the argument or how strong the evidence, when you run up against the preconceptions of some people, it's like bouncing off a force field.

              Go here: http://arstechnica.com/science/news/2010/05/when-science-clashes-with-belief-make-science-impotent.ars

              Read that. Then apply the principles and ideas therein to working with management, and you see the difficulties. Dealing with the brass is a large part of my job. Both at my day job, and the various additional networks I administer on the side.

              Blue collar folks, (which includes most Systems Administrators) are generally task or problem oriented: they look at the issue in front of them, and deal with that. The better ones are task oriented, but can visiualise how their particular task affects the larger picture. That comes with experience, and it’s how you tell a good worker from a great one.

              Management on the other hand plays politics. Feelings and egos come into play; presentation and perception, “buy in,” how any given project, admission of guilt, mismangment or requirement for change will reflect on them, their bosses, customers, the company and a host of other factors.

              To an unfortunate amount of upper management, solving a problem is never about the problem itself, but rather about who gets the benefit from solving a particular problem, and who might in some way be offended.

              A Sysadmin will want to solve a problem because a problem exists to be solved. An upper manager might order you to leave the problem in place because it does more harm to their rivals than it does to them. I have fortunately been largely spared during my career, but I have many friends and colleagues who have not been so lucky...

              1. Anonymous Coward
                Anonymous Coward

                UI: The higher science of incompetent management

                Yes, you're quite right that management are not as technically or even as rationally minded as sysadmins. I'm not sure I'd like to call that blue collar work, even though the IT janitor comparison is sadly apt in practice. Blue collars, to my mind, with long experience excel at practicality, but not necessairily rationality.

                Is, oh, stone masonry blue collar work? Definitely. Does it bring the far-reaching impacts IT ops so often has? Well, a cathedral with walls of clay will come down with the first rainfall. Still, it's the architect that has the vision, the mason'll just do his thing.

                The sheer complexity forces sysadmins into either chicken waving or if that doesn't work (more chicken waving, ad nauseam, and then) taking the thing apart, understanding how it ought to tick, fixing it, and on to the next problem. --Tangentially, that's very close to the original definition of engineer back when it was a military term. But I digress.-- In that, we're conditioned to use an extremely classical view. Contrast this with the romantic view that many of the rest of us have. These terms are from Pirsig's _Zen and the art of motorcycle maintenance_ where he explains them quite well, and they're good to understand here.

                In that light I don't think scientific incompetence is much at work here. Rather, it's a symptom of information overload, perhaps somewhat akin to why people turned to religion in the first place: For security. Science is the very epitome of insecurity, so it's easy to villify. As long as scientists insist on mumbling jargon, it stays an easy target. The assumption that human beings are rational is largely false for the simple reason that thinking is hard. Capable of rational thought, arguably, but rational beings? A very different thing. Don't believe me? Look at the energy costs. It's much easier to re-use a canned good enough solution. Even if it isn't. Is what? Good enough? Solution? Easier to use? True? Making sense? Yes.

                You're right that (bad) management plays politics, something that gets easier if you don't look too deeply into the matter at hand, qv romantic view. I think that as head of IT even of a small department there's more management involved than what a mason'd stomach. I liked the following two blog posts for their not to be taken entirely seriously analysis of televised fiction:

                http://www.ribbonfarm.com/2009/10/07/the-gervais-principle-or-the-office-according-to-the-office/

                http://www.ribbonfarm.com/2009/11/11/the-gervais-principle-ii-posturetalk-powertalk-babytalk-and-gametalk/

                I for me am willing, regardless of being able, to deeply cynically (ab)use whatever makes the brass tick. A good sysadmin is obsessive with making systems work, after all, and if that includes the management system... well, alright, there are limits. But I'd try.

                But I'm not afraid for my job. I've survived before and I can survive again on no money at all. I've gotten out, I've taken the pay cut, I'm happier than I was. Absolutely worth it. Of course, I'll keep looking for an even better spot, a fatter paycheck, both.

                It's mindset that does you in, and it's mindset that gets you out. I *know* I'm good enough to run a shop, regardless of what my CV says. And if the management is finicky and stupid about it, well, let them. Please fire me, I'll find a better place. Or perhaps it'll find me. It has happened.

                If the economically incompetent insist on running themselves into the ground, that's good news for everyone else. Except the investors. But that's their risk, so tough cookies.

                If your problem is romantic management that refuses or is entirely incompetent to take a classical view, well, you may have to learn to play their game. Learn to negotiate, learn to communicate, even if that means focusing on superficialities to get the noses pointing in the direction dictated by underlying causes only you understand. You'll likely have to keep abreast of at least what's happening and where the company is going. Having to play politics, however, is a sure sign of rotten management (even for management), so it's time to get out. (Drucker again.)

                Your problem is management and believing they ought to behave rationally. They're the boss, they're under no such constraint at all. But, on the other hand, you're free to use science on them.

        2. informavorette
          Unhappy

          there's more to it.

          I must say that we have quite an interesting general discussion here. I'd like to see the admins succeed in convincing the management of some sound principles, but I'm not very optimistic, based on my own observations of admins and (even IT-friendly) managers.

          Then we have even more complex situations, like the one discussed here. Namely, you can be the admin of the company which buys a large machine together with its embedded controller. And let's say that you're even able to convince your own manager that solid software would be a good thing. Trouble is, your manager has absolutely no leverage with the development team at the machine building company. He is on the demand side of a market with a supply oligiopoly - if not a virtual monopoly, because if he's once bought some automation from, say, GE, he cannot just add some Siemens machines to his setup later on. If he goes to GE and tells them "we want to get better controller software from you next time", they'll laugh in his face.

          Don't get me wrong, I'd love to live in a world where everyone from a Fortune500 CEO to the flower seller in the market realizes that when software is ubiquitous, it had better be good software or else we're in for some bad surprises. I'd even settle for a world where average people know the difference between good and bad software - as of now, most of the CS students I've met don't know it. Only the big question is: how do we get such a world?

          1. Anonymous Coward
            Anonymous Coward

            Re: there's more to it.

            I don't have a simple solution, but the key is communication. This for the appalingly simple fact that deals are made by communicating. So even if you have no leverage you have to make clear that you expect better software. How to get leverage, make the communication lines work, et al. that's management, which is why I mentioned Drucker earlier on. He's dead now but he spent a life long thinking and writing about it, if not exactly from a techie's viewpoint, and he certainly didn't get everything right, as he'll be the first to admit. Management as formal sport is quite the new kid on the block. Which is understandable seeing how large corporations and therefore a pressing need for multi-layer management that works smoothly is relatively new, as such things go.

            As to CS students, well, we've commoditised higher learning and dumbed down the courses so as to make no difference between what graduates are ``produced'' (yuck) here and in areas with several generations backlog of technological development. Deciding whether that was a good idea is left as an excercise for the reader.

          2. Trevor_Pott Gold badge
            Megaphone

            @informavorette

            Well, I believe we are now entering into the realm of philosophy. I can be fairly clear on things that can be observed and catalogued; behaviours that can be qualified as well as quantified. I am good with IT because “action A provokes response B” within a reasonable margin of error. If action A provokes a response other than B, or fails to provoke a response at all then it’s troubleshooting time.

            Computer systems are supposed to work within norms. It’s impossible to say “they should work the same way every time,” if for no other reason than that our hardware is imperfect. Still, they should fail in predictable ways; if our software is good, in graceful and predictable ways.

            Your comment on the other hand veers off of this track. “I'd even settle for a world where average people know the difference between good and bad software - as of now, most of the CS students I've met don't know it. Only the big question is: how do we get such a world?”

            Change the world? If you find the answer let me know. At the end of the day it’s all greed. Software is shoddily designed either due to expediency or lack of knowledge. Lack of knowledge is usually due to the individual being too greedy with their leisure time to devote the proper skills to their craft. There’s a margin of error there for folks who genuinely aren’t greedy, but make me critical mistake of overextending themselves thus resulting in an inability to learn the craft, or finish the project.

            In large part though, shoddy software seems to stem from the very human desire to cut every corner possible in the attempt to do it faster, cheaper or what-have-you. We are all of us guilty; each and every person on the planet. The man who tells you he never cuts corners is not only lying as he says it, he’s so utterly terrified of appearing to be a failure that he will completely overcompensate for it. (A liability in technical circles, a potential ally in political ones.)

            The little things in life are usually where it shows the most. Take cooking for example; when you are alone, and cooking for yourself…how many corners do you cut? If you are cooking for friends or family, you generally try you very best to make a satisfying and delicious meal, but when alone you reach for the box of bachelor chow and the microwave.

            This approach doesn’t end there; it’s extended to every facet of life. Software developers will cut any corners they don’t feel is important, and managers won’t put pressure on developers to not cut corners unless something makes doing that important to them. The company investing in software development needs an incentive to not cut corners and on it goes. People are greedy not only with their money, but with their time and effort as well.

            The only advice or philosophical truth I can offer is that it is because of the recognition of these very facts about our human nature that the field of ENGINEERING was born. There is a difference between a builder and an engineer. There is weight and value to that Iron Ring. An engineer belongs to a fraternity of people sworn NEVER to let such concerns cause the failure of their projects. You don’t have Adam the cheap labour builder create your train bridge across the river valley; you rely on an engineer because it is important and it has to be done right, with zero cut corners. If that engineer ever cut a corner; just a single one and was found out, he would lose his livelihood. The rules of his profession are strict, and they are final.

            Just as being disbarred is a career death sentence for a lawyer, or a Doctor can lose their license to practice medicine, and engineer can have that iron ring taken away. It is what makes these more than trades, but honoured and venerable PROFESSIONS.

            Too many people misuse the term engineer, for that matter, people misuse the term “profession.” If you want a world where software is designed right, the first time, with zero cut corners than you need to make development of software an actual profession. You need to make “software engineers” real, actual engineers, and they need to be bound by that iron ring.

            If you want a world where all software is developed along these lines, then development truly must be a profession; common people are forbidden to practice it unless they belong to an accredited organisation. Just as I cannot claim to be an MD and practice medicine, proclaim myself counsellor and practice law, I can not claim to be an engineer and practice engineering. It is illegal, and were I do attempt to do so and were caught, I would go to jail for it. (It is not a crime you merely get a fine for.)

            This is the world that would have to exist for software to be “done right, the first time.” Software would be enormously expensive, but it would last for decades. It would be slow to evolve and change, but it would cope with a variety of issues.

            If you want a world without the cut corners, then all of us IT folk, be they developers or systems administrators need to accept who wand what we are. Whether you have an MCP, a one year certificate, two year diploma or even a bachelor’s degree, unless there is an iron ring on your hand, you aren’t an engineer. You aren’t even close; don’t claim to be one, don’t pretend you are one, and don’t think that you have the slightest idea what the difference between what you do and what a real engineer does is.

            The IT industry’s collective need to pad our egos is probably responsible for more terrible design issues, implementation failures, buggy code and downright asshattery than anything else in all of human history. For decades we have told ourselves that we are “new’ and “disruptive” and that we “weren’t recognised as being legitmate.” We told ourselves anything possible to convince ourselves we were ‘as good as” doctors or lawyers or engineers.

            Yet we have *never* held ourselves to the same standards. We have never put on the IT equivalent of the iron ring. We have never sworn an oath, and we have never collectively walked away from jobs because what is being asked can’t be done without cutting corners.

            We are tradesmen; food at what we do, and capable of making computers and software function in ways they were simply never designed to. We are problem solvers and tinkerers all, but we are most emphatically not engineers.

            So while I know there are many people who will disagree with this post; in large part because we all want to see ourselves in the best possible light, I’ll hit the submit button anyways, and let the flames fall where they may.

            My name is Trevor Pott, I’m a Systems and Network administrator, and I stand before you to say this is a trade, not a profession. I wish I was an engineer, and if I could do it all again I would have become one, but I am not. I am a digital janitor; a plumber of the tubes.

            Who are you?

  4. Anonymous Coward
    Anonymous Coward

    Good stuff!

    Excellent article!

    What about private vlans, assuming you are using cisco gear for your LAN switches? These would enable you to keep the offending w2k workstations on the same subnet but allow you to lock down where they can broadcast to.

    Also thanks for malwaredomains.com, I had no idea such a site exisited for malware!

  5. Matthew 3
    Thumb Up

    Useful article.

    Many thanks for the honest article. I think that we all learn a lot more from when things go wrong and the typical temptation to gloss over failures makes any lessons less useful. By describing your 'warts and all' situation the article is far more useful and relevant than a bland best-practice guide.

  6. Anonymous Coward
    Thumb Up

    Cracking article...

    ....thanks.

    R.

  7. Anonymous Coward
    FAIL

    Defence in Depth

    Excuse me - what about IDS / IPS updates...?

    These are now available from all the main security vendors as UTM or xTM appliances. IE cheaply.

    The Conficker "vulnerability" - not the "exploit" - was patched by some of those vendors months before MS released the patch or the worm was released into the wild. This means the worm would have been blocked at the perimeter.

    Also, put a bridge mode firewall between the office LAN and the equipment. PVLAN is *not* enough.

    1. Trevor_Pott Gold badge

      @AC

      Wait for the next article on the IDS stuff...

      As to "bridge mode firewall between the office LAN and the equipment," I would like to be enlightened as to why you feel this will provide more security than a separate subnet + VLAN on the switch. (Especially if the equipment will be getting its own firewall.)

      What real protection does your approach provide that mine doesn’t?

  8. Anonymous Coward
    Flame

    Pointy Haired Bosses

    ...will probably remember this and be a little bit more willing to support good security measures.

    Why they think they don't need automatic updates, I don't get. Probably current MBA courses also include one year of Windows Admin training ??

    Seriously, Powerpoint and Excel are well-tested against Windows patches by MS themselves, so these "exec" PCs should have auto-update, too.

    1. Anonymous Coward
      Anonymous Coward

      Forgot about that, whoops.

      ``Seriously, Powerpoint and Excel are well-tested against Windows patches by MS themselves, so these "exec" PCs should have auto-update, too.''

      Oh yes, that's what I originally wanted to note well before I went off on DHCP and other tangents.

      To management, much like the rest of the warm body brigade, their boxes are supposed to *work*, not bother them with upgrades. So those would actually be the first I'd patch if I was into running a windows ``network'' (which I very much am not). Whether to run them automatically or do it manually during their extended lunch break is something else entirely. And how you communicate that (``down for updates, guv, better extend that break some more''), well, that's site specific, innit? Communicating is key though, but do find a message that intrudes on their nice and quiet routine the least.

      On another tangent, it never ceases to amaze me how the ``but it has to just woohooork'' crowd puts up with a company notorious for crappy software that excels in finding new annoying messages to pop up and have its users click on to ignore. You can give me all the perfectly sensible reasons in the world, I can name a hundred myself, and I'd still not get it. Why are we putting up with this, exactly?

  9. Anonymous Coward
    Megaphone

    Another Thing - Compartmentalization

    All network Administrators should think about useful partitioning strategies for the network they manage. Most of the time the marketing people don't need to contact the PCs and the servers of the finance or the HR people. If they need access to something, make that available selectively.

    R&D systems contain a lot of valuable data, so why does HR need access to that ? Different divisions often have not much to communicate to each other, so this would again be a good opportunity to set up isolated subnets.

    This kind of compartmentalization is very similar to modern ships, which can tolearate quite a few leaks because they consist out of waterproof compartments.

    Some companies configure their server in the intranet the same way they configure them for the internet - lock down all unused ports and allow only the IP addresses/ranges that really need access. Also a pretty good idea.

    1. Trevor_Pott Gold badge

      @jlocke

      I couldn't agree more! In larger enterprises this is an honestly top-notch approach. Sadly, in smaller companies every staff member is "an exception." Everyone is wearing two or three hats. Just setting up the windows security permissions is a nightmare; it's probably as complex (or more) for an SME of 150 people than it is for a corporation of 1000.

      When your staff wear so many hats that most of them can't even be reasonably given an actual job title...network security via compartmentalisation becomes a pipe dream.

      I can isolate some of the back-end equipment, but isolating desktops, file servers and similar equipment just ain't gonna happen.

      Still, wherever you can compartmentalise...do so!

  10. Anonymous Coward
    WTF?

    WSUS Optional?

    Madness...

    I have it running and enforced for everyone, myself included - patches are done once approved at 7am every morning, and then nag very 15 mins for a restart.

    I sold this to the office on the basis of Administrator rights - they can have those if they are willing to put up with AV being locked down and strict and the WSUS being aggressive.

  11. A J Stiles
    Stop

    More radical solution required

    As long as anyone continues to buy software which is shipped without the Source Code, this sort of thing is going to keep happening and happening and keep happening again.

    If something is closed so only one person can fix it, then it's *not* secure.

    If something is open so anyone can fix it, then you are damn well going to build it properly in the first place rather than risk being laughed at.

    It's time for drastic measures. The Nuclear Option: Legislation to enforce user access to Source Code. Sure, it would be great if The Market worked and it was enough just to demand the Source Code or threaten to take your business elsewhere; but that hasn't happened in practice. Keeping Source Code hidden has done nothing to prevent rampant piracy, while inconveniencing everyone along the way.

    Just how much more collateral damage is it going to take before anyone realises there's a problem?

    1. Filippo Silver badge

      Re: More radical solution required

      I fail to see how providing source code would help. The typical factory business that uses heavy machinery does not employ anyone who's even remotely qualified to modify control software, and given the extremely specialized nature of said software (it's usually custom-made) you can't hope somebody on some Internet forum will make a patch. You could hire a consultant, but he will just pretend to look at it, tell you the program can't be fixed, and would you like our own solution instead. The source is useless.

      As for why businesses use brittle and insecure control software that only runs on Win2K, it's probably because said software was custom-made for DOS in 1985, it would have to be rebuilt from scratch, and this would cost money.

      The outrage isn't the software, it's that somebody was allowed to use the machine to check his email. It's OK for a box to be unpatched because of old software that understandably wasn't designed to run on OSes 20 years in the future, but at least keep it off the internet.

      1. Trevor_Pott Gold badge

        @FIllipo

        The e-mail checking did not occur on the brittle Win2K box. It occurred on the user's virtual machine. (The one he is assigned to use.) That VM happens to be on a subnet capable of reaching the brittle equipment, and then all things went *poof*.

        The real hell of it is that I can’t take “Local admin” away from the guy because he’s a manger. So if you have local admin on your VM, decide to open an attachment that you shouldn’t….there’s really **** all I can do about it. That system will get pwned in about 0.2 seconds flat. And it ain’t just windows this happens on. (Seriously, I am getting SO BLOODY SICK of cleaning up after Macs that keep getting pwned by these damned “download this file and execute it” Safari exploits.

        So yeah, PHB opens mail, pwns his local system…that reaches out to our network and pwns anything not fully patched. As my article points out: “mea culpa.” That box should have been on it’s own subnet, one that regular desktops or personal VMs can’t route to.

        1. Anonymous Coward
          Anonymous Coward

          Email Virus Checking?

          Did the infected email come through your corporate email system or a private/webmail system? If you have a centralised corporate email system I would have thought it was possible to scan incoming emails for viruses at the server, so the viruses don't make it to the user's inboxes. This would have prevented this infection, wouldn't it?

          HTH

          1. Trevor_Pott Gold badge

            @Stephen Roberts 2

            Check my replies later in this thread...I do explain it all more in detail...

      2. A J Stiles

        How it would help

        Providing Source Code would help by forcing the people who wrote it to write it properly in the first place, lest other people point and laugh and say things like "Aha! What a delightfully stupid schoolboy error!" (Early Mozilla and OpenOffice.org, being formerly-closed code that was opened up, were full of these. OOo 1.x wouldn't even build on 64-bit systems, because someone had assumed that (1) an int and a pointer were always the same size and (2) nobody would ever see the code and find out about this horrendous bodge.)

        Also, if it went wrong, or even if it just didn't quite suit your existing workflow, you could get someone besides the manufacturers to fix it.

        And finally, if the underlying OS changed, it would be possible to get the code to run on the new OS. Maybe just as simple as re-compiling, but at any rate a lot less bother than you're used to.

        1. Trevor_Pott Gold badge

          @AJ Stiles

          "Providing Source Code would help by forcing the people who wrote it to write it properly in the first place, lest other people point and laugh."

          Have you ever used any open source anything? Better yet, have you ever actually looked at the source of open source anything?

          The /vast/ majority of it is as badly coded as anything proprietary, and a truly unfortunate amount of it is utter crap. It is largely coded to solve a specific problem, stupendously inefficient and with zero consideration for extensibility, dealing with errors or unexpected input.

          So essentially identical quality to proprietary code then.

          The only difference is other coders can tear it up /if they choose/.

          They very rarely choose to do so, because frankly it would be less bother to code it from scratch anyways. I think you have some overly idealistic views of open source, sir…

          1. A J Stiles

            Yes

            Yes, I run as near as possible 100% Open Source and have been doing so for the best part of 8 years now. I have even occasionally done some tinkering with the code. Our company now runs almost exclusively on Open Source and in-house written applications accessed through a web browser.

            I really would not have it any other way.

    2. Brian Miller
      Alert

      Yeah, sure, local sysadmins rewrite all softwware

      I've had the unenviable job of working with software that had ten years of cruft on it. Yes, 1/3 assembly and 2/3 C running on MS-DOS x286. No sysadmin has the skills and time to rewrite and port crufty code to another platform.

      As for solutions to problems like Conflicker, I would place little NAT firewalls in front of all of these old machines. Then the problem is solved because the worm can't access the computer. Of course using a NAT box isn't feasible if the machine has an exposed file system. The Conflicker worm uses NETBIOS vulnerabilities to propagate, so you'd need a different solution.

    3. Trevor_Pott Gold badge

      @AJ Stiles

      Your solution, while very passionate and open-sourceily noble would leave us without any of the equipment that actually makes our company money. When there isn't an open source version available, you buy what you can.

      IT exists to server the business, the business does not exist to server IT's ideology. This equipment is what it is, and it is what pays for the wages of everyone I work with. It is my job to make sure it runs, regardless of ideology.

      I am pretty certain this is true everywhere else in the entire world, with the possible exception of certain areas of California.

    4. Nagy, Balázs András
      FAIL

      Re: More radical solution required

      Ever heard of this?

      http://en.wikipedia.org/wiki/Thompson_hack#Reflections_on_Trusting_Trust

      No, using open sourced software is not inherently safe from built-in "extra" features. Plus reviewing, rewrinting or even understanding all code you use is completely and utterly impossible. Especially in heavy industry, where a bug or miscalculation will cause millions in damage and possibly the loss of lives.

  12. jake Silver badge

    This is why friends don't let friends ...

    ... put computers/OSes with a single user, single-tasking, "this desktop only" mindset onto a world-accessible network. It's only asking for trouble.

    Yes, I know, it is done world-wide. Doesn't make it secure. Or the right way to do it.

  13. Ian Sawyer 1

    Dufus

    Your an amateur

    1. foo_bar_baz
      Grenade

      What an insightful and constructive post

      "Your" one eloquent SOB, I wish we all could reach your high standards.

  14. plrndl
    Linux

    Windows for WSUSies

    Anyone who has a WUSS server on their network is asking for it.

  15. Robert Carnegie Silver badge

    Efficient management practice

    If you log off at the end of the working day, you just have to start all your applications again the next day. The smart manager can save time and just switch the monitor off. The "mon-i-tor" is the television thing on top of the box thing.

    1. Keith Williams
      Thumb Down

      Efficient management practice

      You forgot to mention that they have 20 to 100 emails open that "they are working on"

    2. Trevor_Pott Gold badge

      @Robert Carnagie

      No desktops. VDI. That measn you can just disconnect from your session. THAT MAKES IT FUNNER.

      *sob*

  16. Anonymous Coward
    Happy

    Critical Systems, Network Access

    If the aforementioned is a system that controls something via an ancient SCSI cable, and you can no longer patch it anymore, what's the need for network access?

    Just have it run that one application, nothing else. If you are so worried about performance, then that's the only function it should be performing, thus decreasing the number of applications installed/running at once!

    No network = no virus

    Disable drive access too, unless admin.

    This seems VERY preventable via common sense, not even patches! :)

    1. Trevor_Pott Gold badge

      @yehasher

      This system exists to receive files from a master command and control system. It then takes those files and Does Stuff (tm). Without the ability to use the files it is getting over the network it is a quarter million dollar paperweight.

      1. M Gale

        Quarter million dollar paperweight

        Not a vinyl router is it? Only I remember being called up by a friend in a signage firm asking if I can help fix the thing. I thought, and said "A.. err... what?"

        Turns out that what I was expected to fix was a snooker table-sized thing that looks for all the world like a giant air hockey table with an equally oversized plotter bolted onto the top of it. The C&C machine in this case was a PC from the year dot running (or rather, failing to run) DOS v5.00.

        Took a new motherboard/CPU/RAM combo in this case. Cheapest thing in the shop did fine. As far as I know, it still runs DOS...

Page:

This topic is closed for new posts.

Other stories you might like