back to article Water treatment plant hacked, chemical mix changed for tap supplies

Hackers infiltrated a water utility’s control system and changed the levels of chemicals being used to treat tap water, we're told. The cyber-attack is documented in this month’s IT security breach report (available here, registration required) from Verizon Security Solutions. The utility in question is referred to using a …

Page:

  1. Bronek Kozicki
    Unhappy

    no prizes for good guess

    which will happen first:

    1) government wanting even more surveillance on everybody

    2) stiff penalties for companies leaving their systems insecure

    1. Craig 2

      Re: no prizes for good guess

      "stiff penalties for companies leaving their systems insecure"

      Even connecting critical infrastructure to a publicly accessible network should be a criminal offense in my book. The question of whether it's secure or insecure is easy to answer: it's not.

      1. Peter2 Silver badge

        Re: no prizes for good guess

        Frankly, I think it would be nice to have a grown up debate as to what should, and should not be able to be accessed remotely at all.

        My view is that the answer to that is something similar to Asimov's first law. "A <system> may not injure a human being or, through inaction, allow a human being to come to harm."

        The ability to remotely access a car's control systems via a sodding radio's bluetooth/wifi and disable control inputs from the driver (like steering or brakes) should be burned with fire along with the people who allowed the basic system design. Industrial processes and in general anything that can cause harm should be air gapped in the same way the control systems in nuclear power plans are.

        Yes, it's going to raise costs. But doing otherwise is critically dangerous with things like flouride going in drinking water:-

        http://www.nejm.org/doi/full/10.1056/NEJM199401133300203

        From that it seems quite clear that if a hacker had of dumped the entire flouride store into the water supply then nobody would have noticed until either they had to refill it or people started turning up in hospital. Utterly ludicrious.

        1. MyffyW Silver badge

          Re: no prizes for good guess

          The wild libertarian in me answers:

          "doing otherwise is critically dangerous with things like flouride going in drinking water"

          with

          "Stop adding flouride to tap water, I've got an inalienable right to rotten teeth"

          (and in case you think I'm a dirty cow, I am a bit OCD when it comes to brushing, so my gnashers are a pearly white.)

          1. Anonymous Coward
            Anonymous Coward

            Re: no prizes for good guess

            "Stop adding flouride to tap water, I've got an inalienable right to rotten teeth"

            Then don't drink the tap water, buy your own drinking water, problem solved.

            Or , stated using the same line of thtinking: "Get your nanny-state coddling out of my tap water, it's my God-given right to drink fluoride-laden water if I so choose".

            1. Pompous Git Silver badge

              Re: no prizes for good guess

              Then don't drink the tap water, buy your own drinking water, problem solved.

              How do you do that when the municipality you happen to be in has banned sales of bottled water. Apparently the do-gooderesses don't mind Coke, Fanta, Leed etc, but water is a definite no-no.

              1. Anonymous Coward
                Anonymous Coward

                Re: no prizes for good guess

                "How do you do that when the municipality you happen to be in has banned sales of bottled water. "

                Citation needed.

                It seems improbable that any government agency can ban the sale of water.

        2. Captain Badmouth
          Holmes

          Re: no prizes for good guess

          "From that it seems quite clear that if a hacker had dumped the entire flouride store into the water supply then nobody would have noticed until either they had to refill it or people started turning up in hospital. Utterly ludicrious."

          See my Camelford link later in this thread.

        3. I. Aproveofitspendingonspecificprojects

          Your View?

          > My view is that the answer to that is something similar to Asimov's first law. "A <system> may not injure a human being or, through inaction, allow a human being to come to harm."

          You are obviously an idiot or too young to voice an opinion.

          A public utility still using internet access after Stuxnet is liable for manslaughter charges and in any case the management need removing urgently, especially their security bods. If this had happened without such mitigation it would be an act of war. It probably still is.

          I hope that Trump is as bomb proof as his predecessor because he really sounds like the sort of arse that America's enemies (or Israel's friends) want in.

          1. Anonymous Coward
            Anonymous Coward

            Re: Your View?

            "I hope that Trump is as bomb proof as his predecessor because he really sounds like the sort of arse that America's enemies (or Israel's friends) want in."

            For the life of me, I cannot see why this statement is in any way relevant to the discussion.

            A down vote for bad cut & paste, or stupidity, or both ...

            ps: Same applies if you substitute Clinton for Trump

        4. Anonymous Coward
          Anonymous Coward

          It's "fluoride", not "flouride"

          see title

        5. Anonymous Coward
          Anonymous Coward

          Re: no prizes for good guess

          Yes, and then Asimov made out a good living by writing about what the three laws of robotics happened to work in some extreme corner cases (and requiring Susan Calvin to understand what really happened). Moreover it postulated the very way the positronic brain was built had them truly "hardwired" and thereby could not bypassed - without damaging the brain irreparably and inoperative. Unluckily software can be modified, and some systems can't really become wholly inoperative, unless some safety mechanism detect it and put the system is a safe state.

          Truly airgapped system would require all the air to be removed, so no humans could touch those system and plug in their USB drive to watch some porn while monitoring the systems...

        6. Anonymous Coward
          Anonymous Coward

          Re: no prizes for good guess

          Well-designed, critical systems usually have hard limits built into them so that such a thing can't happen - not without someone going out there manually (with appropriate tools) and taking the situation in hand, anyway. For fluorine/chlorine and such, I would generally expect such a system to either just reject a "dump everything" command, or to merely increase things to a higher but still relatively safe level - whatever the hard limit restricts it to.

        7. Rick Manner

          Re: no prizes for good guess

          Regarding the concern about dumping an entire storage tank of fluoride into the water system, I have two bits of information that may make you more comfortable.

          First, fluoride has a bitter taste. So if there is a severe overdose, people will not drink the water. Trying to cover up this bitterness is a large part of why toothpaste has a strong flavor added, as well as the fluoride treatments at your dentist's office.

          Second, most regulators require that for chemicals added to the water that the system run off of what is termed a "day tank". The day tank only stores a limited amount (usually about one day's worth), exactly to prevent the type of overdose that you are referring to. . There are other benefits. Because it is a smaller tank, minor changes in feed rates are noticed sooner.

          By the way, this second idea was started long before hackers were born. It is a practical solution that prevents excessive dosing for whatever reason.

      2. PNGuinn
        Megaphone

        Re: no prizes for good guess @ Craig 2

        It probably IS illegal in just about any jurisdiction you'd like to think of, with probably very large penalties.

        Problem 1 - the kind of scum who do this sort of thing tend to be criminals with every intent of causing mayhem - either for blackmail or political reasons. They know full well what they’re trying to do, know the penalties and know the risks.

        Problem 2 - the authorities in many of those jurisdictions will either (a) not understand their own laws and prosecute on a minor technicality, (b) seek to minimise the crime to cover either their own ineptitude or that of those who run the vulnerable systems or (c) don't want to upset the nice terrists in case they get really mad - hearts and minds and all that carp.

        Solutions 1 - Hit the perps hard - a lot of this stuff endangers life and health apart from being costly. Be aware that this will likely lead to war in some cases. Be aware that it's pointless going to war unless you're prepared to win - and clear up afterwards.

        Solution 2 - Make it very clear in law that there's a clear audit trail of criminal responsibility for all those responsible for critical systems and their security, including their design and maintenance INCLUDING THOSE IN GOVERNMENT. With appropriate penalties. Not chosen by lazy incompetent greedy fat ....

        One can dream.

    2. VinceH
      Facepalm

      Re: no prizes for good guess

      "which will happen first:"

      Well, you could at least have made it a little difficult by not listing the most likely thing as number 1!

    3. asdf

      Re: no prizes for good guess

      Well considering the SCOTUS pitched a fit and overturned the one time the government actually convicted a large corporation (Arthur Anderson) of outright fraud #2 is a pipe dream. At least they can still go after executives for bad behavior you know like they did after the mortgage meltdown. Funny how that works when your whole culture is based around corporatism.

    4. Captain DaFt

      Re: no prizes for good guess

      "which will happen first:

      1) government wanting even more surveillance on everybody

      2) stiff penalties for companies leaving their systems insecure"

      #) Nothing. It's not like They urinated in a reservoir or anything serious like that.

      1. Mark 85

        Re: no prizes for good guess

        #) Nothing. It's not like They urinated in a reservoir or anything serious like that.

        I guess no one pointed out to them that fish, birds, and animals all pee and poop in the reservoir and it doesn't get drained and scrubbed.

        1. allthecoolshortnamesweretaken

          Re: no prizes for good guess

          “I don't drink water. Fish fuck in it.”

          ― W.C. Fields

        2. Goopy

          Re: no prizes for good guess

          Daddy, what are stated towers for?

    5. Gigabob

      Re: no prizes for good guess

      It will be a furious race - but I predict a tie at the finish line.

  2. Jon Massey
    FAIL

    The.. just.. I don't even

    " login credentials for the AS/400 were stored on the front-end web server."

    pardon?!

    1. P. Lee

      Re: The.. just.. I don't even

      ... and what's with the pejorative "ageing as/400" smack-talk?

      If you store credentials on the frontend web server, no amount of "modern" systems or updates are going to save you.

      1. Voland's right hand Silver badge

        Re: The.. just.. I don't even

        If you store credentials

        Question is what credentials. Some credentials - such as what you need to access CRM have to be stored.

        Now the fact that the credentials were such that they allowed to manipulate the actual live industrial control systems is the "criminal negligence" bit. As these control chlorine, cloramine and access to drinking water supply there are quite a few criminal charges applicable for the execs of the water company in question in most legislation. Criminal negligence is just the start. I would slap onto them "being accessory to terrorism" without having a second thought.

      2. Michael Wojcik Silver badge

        Re: The.. just.. I don't even

        ... and what's with the pejorative "ageing as/400" smack-talk?

        Yes. A swing and a miss there for Leyden. I'd much rather have the back end be an AS/400 running, oh, some release of OS/400 V3 than, say, an almost-certainly-misconfigured Win2K system, or never-patched Linux of similar vintage.

    2. Roger Varley

      Re: The.. just.. I don't even

      Any takers for a bet on whether they were for QSECOFR or not?

    3. Anonymous Coward
      Anonymous Coward

      Re: The.. just.. I don't even

      More to the point what the hell is a web server doing connected to the control systems and being accessible from the internet? Taking it further why were the control systems even anywhere near being connected to the internet?

      1. I ain't Spartacus Gold badge

        Re: The.. just.. I don't even

        I would imagine the billing system is probably polling information from the control system. And presumably the treatment controls are on the same system as the network/metering ones. Obviously this should be via a locked down account with no permissions - but I guess it isn't. Well, even more obviously, it shouldn't even be connected - that info should be going to an offline database first.

        I can understand wanting to have central control of the system. Rather than having to control things individually at each pumping station and works. But that should be via a private network, not the internet. And there certainly shouldn't be a bloody web server.

        Admittedly they do regular testing of the water. But although some of that will be manual, so not vulnerable to computer intrusion, I'd expect that this will also be moving towards automation though.

        You can do an amazing amount of damage though. If you control valves, pumps, or worse pumps and valves - then you can easily cause pipes to burst. With chemical dosing you can either overdose or underdose the water and cause problems. Sewage plants are also delicately balanced, in that they have beds which use bacteria to break down some of the waste products - and if too much of certain chemicals gets in there, it kills off the colonies, and stops the treatment plant working.

    4. PNGuinn
      Flame

      "login credentials were stored on the front-end web server."

      Simple solution - someone's b***ocks need to be stored equally publicly on a barbed wire fence. Probably several peoples' .... No need to detach them first,

  3. Alister

    A couple of weeks ago South Derbyshire and North Leicestershire residents were warned not to use their tap water for any reason because the chlorine concentration was at dangerous levels.

    Curious coincidence.

    1. HollyHopDrive

      Well, I was one of those customers and given just how little Severn Trent seemed to know about the incident and how it happened it made me wonder too.

      After 8 hours there was still much confusion. I saw them doing what looked to me as pumping out a water tower into a long like of waiting tankers the next day.

      When I was down getting my 4 litres of free water (generous or what!) We asked the ST woman there why we couldn't shower in it and she said it's chlorine and it's way stronger in concentration that you'd get at the swimming baths. (She really couldn't stress just how much we really shouldn't use it to even wash hands). So if something looked like a computer error or hack this is a likely candidate.

      Then again, could just be coincidence. Guess we will never know!

      1. Roger Greenwood

        Many water companies who abstract ground water (like ST) use superchlorination - they add a lot of chlorine to guarantee to kill any bugs then reduce the chlorine levels before it hits supply, without needing an intermediate tank/reservoir - it goes straight down the pipe. A mechanical/electrical failure at any point in the dosing system could allow high chlorine levels to get through to supply without the system getting hacked.

        1. wardster

          Yep - super chlorination, or shock dosing.

          Anything above 0.5 ppm HClO will kill most bacteria, and your average swimming pool will be 1 to 3 ppm to ensure all those scutty people who don't shower before going for a swim doesn't bring in any nasties, and also to make sure if little Johnny curls off a floater, then it won't need the pool to be evacuated and drained!

          Obviously you don't want to be drinking the contents of your local pool, but it won't kill you.

          Hot spas and things like that can be maintained between 3 and 6 ppm, but as you aren't in for too long, it won't cause any problems.

          Anything above 6ppm however is really not advised, as at this concentration, you will start to get bleaching, and sensitive skin can start getting rashes and irritation.

          If you hit anywhere above 10 - 12 ppm, and you really really do have a problem. I can only assume that the STW recent problem had HClO levels way above 3 - 6 ppm.

          (I recently did the STA water treatment course.....)

          Anyone remember Milton Sterilising Tablets? Maybe someone bunged a few of these into the reservoir.......

      2. John Lilburne

        Hey shit happens. Back in the 1990s, when I worked in a chemical factory, we had a water treatment guy in to dose the cooling tower water with biocide (legionnaires). Unfortunately they didn't tell anyone that they'd done it. So some maintenance fellows comes on shift and opens up a valve to let water into the local canal. A few hours later the surface of the canal was covered in dead and dying fish.

      3. Anonymous Coward
        Anonymous Coward

        AS400?

        Could this be STW? A couple of decades ago (and before Sir Tim invented WWW) I worked on a SCADA system for Severn Trent that could, in theory, be used to control a water treatment plant. Being pre-WWW it didn't have a front end server, and it ran on hardware that was somewhat more mature than the AS/400, (not that I'm prepared to say what it ran on). I did hear from a reliable source that the old software had been ported to new hardware (AS400?) and it is entirely possible that a ropey old web front end was bolted on to the port. I also wonder if this is a coincidence.

  4. Anonymous Coward
    Facepalm

    Bullshit bingo

    Monzy Merza, Splunk’s director of cyber research and chief security evangelist, commented: “Dedicated and opportunistic attackers will continue to exploit low-hanging fruit present in outdated or unpatched systems. We continue to see infrastructure systems being targeted because they are generally under-resourced or believed to be out of band or not connected to the internet.”

    “Beyond the clear need to invest in intrusion detection, prevention, patch management and analytics-driven security measures, this breach underscores the importance of actionable intelligence. Reports like Verizon’s are important sources of insight. Organisations must leverage this information to collectively raise the bar in security to better detect, prevent and respond to advanced attacks. Working collectively is our best route to getting ahead of attackers,” he added.

    Every card a winner!

    Seriously, who writes this stuff?

    1. JoeF

      Re: Bullshit bingo

      And Verizon Enterprise, the guys who do write these intrusion reports, got hacked themselves, according to krebsonsecurity...

      http://krebsonsecurity.com/2016/03/crooks-steal-sell-verizon-enterprise-customer-data/

      1. Michael Wojcik Silver badge

        Re: Bullshit bingo

        And Verizon Enterprise, the guys who do write these intrusion reports, got hacked themselves

        Well, sure. The question is, how good was their report about it?

  5. Ralph B

    Meanwhile ...

    I heard that another US-based hacktivist group had got away with doing similar tricks for some years before they were stopped.

  6. Dan Wilkie

    I don't understand why the control system is linked to the customer payment portal, and why the payment portal would need credentials for the control system.

    Or did both systems just happen to run on the same AS/400? (REALLY?)

    1. Anonymous Coward
      Anonymous Coward

      Because there was a time when "consolidation" was the buzzword like cloud is today (cloud is still a form of consolidation...). The mantra was to run everything on fewer, more powerful systems to save money. Done in the right way it could be OK, done in the wrong way by clueless people "hey, we have this AS/400 let's run both the water control system and accounting from it! See how much we saved?" leads to these situations. Of course IBM told (and sold) you you could run different workloads on it, so why not? The AS supported hardware partitioning - but if used by clueless syadmin, little changes...

      1. PNGuinn
        Joke

        "consolidation" was the buzzword like cloud is today ...

        Ah - there's the solution stairing everyone in the face.

        Clouds bring rain. End of water shortages.

        Someone with good ideas just needs a legup.

    2. I ain't Spartacus Gold badge

      It could be there's some bigger commercial/industrial customers whose meters are reported directly on the network's controls systems. So the billing system uses that info to charge them. Not sensible, but doesn't mean someone hasn't done it.

    3. Adam 52 Silver badge

      I agree on why they had to be linked and why the credentials were so wide, but I don't see anything wrong with running it all on the same box, especially not an AS/400 which supports LPARS. No real difference to running on Xen today.

      1. Anonymous Coward
        Anonymous Coward

        Did they use partitions or not? The fact the AS/400 supports LPARS doesn't mean it was in use.

        Also, even today running software at different security level on the same hypervisor *can* be a security issue. There are bugs in hypervisors (and even in CPUs...) that let an attacker compromise other VMs. Thus, even if it costs more, may be sensible to run software on truly separated hardware.

        But everything becomes useless if there are easy channels between systems and powerful credentials are stored everywhere.

    4. Anonymous Coward
      Anonymous Coward

      Because organizations which run generally safe, sane, and relatively secure systems like the AS/400 (and its successors) don't usually see the need to carve things up unnecessarily, although some separation of duties may have been a wise decision in this particular case. But I have worked with/for several companies now who have gone down the path of "modernizing" their systems, by moving things over to some number of different (mostly) dedicated servers, only to often quickly run into the problem of not knowing why/when/where things are going wrong, nor of course how to fix it.

      I'm dealing with that very issue right now, in fact, where instead of things staying on the AS/400 where they really belonged, they've been spread out across several different servers of various types. But critical things are occasionally failing now where they didn't fail before, and the situation is getting progressively worse, and nobody really understands enough about the whole set-up (nor do they generally have the time or the patience) to really be able to go in and find the problem and fix it. Which is where I come in, because I've had to run such rabbits down in the past, at other organizations.

  7. Anonymous Coward
    Anonymous Coward

    The law needs changing, and soon

    All critical national infrastructure (water, power, etc) should be air gapped from the internet immediately, and anyone who attempts to implement internet connectivity as a cost cutting measure should be imprisoned. Cost cutting will bite us all on the ass eventually.........

    1. Mark 85

      Re: The law needs changing, and soon

      Eventually???? I daresay it already has bit us and bit us hard. The problem is, it hasn't changed the C-suite types thinking since all they focus on is profit.

  8. 2460 Something

    Demarcation?!?

    Why the hell is a control system on a publicly accessible network in the first place? Something like that should be on a self-contained network to prevent anything like this being possible. It beggars belief that all these utility companies don't have better network designs.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like