nav search
Data Center Software Security Transformation DevOps Business Personal Tech Science Emergent Tech Bootnotes BOFH

back to article
Water treatment plant hacked, chemical mix changed for tap supplies

Silver badge
Unhappy

no prizes for good guess

which will happen first:

1) government wanting even more surveillance on everybody

2) stiff penalties for companies leaving their systems insecure

37
0

Re: no prizes for good guess

"stiff penalties for companies leaving their systems insecure"

Even connecting critical infrastructure to a publicly accessible network should be a criminal offense in my book. The question of whether it's secure or insecure is easy to answer: it's not.

50
0
Silver badge

Re: no prizes for good guess

Frankly, I think it would be nice to have a grown up debate as to what should, and should not be able to be accessed remotely at all.

My view is that the answer to that is something similar to Asimov's first law. "A <system> may not injure a human being or, through inaction, allow a human being to come to harm."

The ability to remotely access a car's control systems via a sodding radio's bluetooth/wifi and disable control inputs from the driver (like steering or brakes) should be burned with fire along with the people who allowed the basic system design. Industrial processes and in general anything that can cause harm should be air gapped in the same way the control systems in nuclear power plans are.

Yes, it's going to raise costs. But doing otherwise is critically dangerous with things like flouride going in drinking water:-

http://www.nejm.org/doi/full/10.1056/NEJM199401133300203

From that it seems quite clear that if a hacker had of dumped the entire flouride store into the water supply then nobody would have noticed until either they had to refill it or people started turning up in hospital. Utterly ludicrious.

19
0
Silver badge
Facepalm

Re: no prizes for good guess

"which will happen first:"

Well, you could at least have made it a little difficult by not listing the most likely thing as number 1!

7
0
Silver badge

Re: no prizes for good guess

Well considering the SCOTUS pitched a fit and overturned the one time the government actually convicted a large corporation (Arthur Anderson) of outright fraud #2 is a pipe dream. At least they can still go after executives for bad behavior you know like they did after the mortgage meltdown. Funny how that works when your whole culture is based around corporatism.

5
0
Silver badge

Re: no prizes for good guess

The wild libertarian in me answers:

"doing otherwise is critically dangerous with things like flouride going in drinking water"

with

"Stop adding flouride to tap water, I've got an inalienable right to rotten teeth"

(and in case you think I'm a dirty cow, I am a bit OCD when it comes to brushing, so my gnashers are a pearly white.)

3
0
Silver badge
Megaphone

Re: no prizes for good guess @ Craig 2

It probably IS illegal in just about any jurisdiction you'd like to think of, with probably very large penalties.

Problem 1 - the kind of scum who do this sort of thing tend to be criminals with every intent of causing mayhem - either for blackmail or political reasons. They know full well what they’re trying to do, know the penalties and know the risks.

Problem 2 - the authorities in many of those jurisdictions will either (a) not understand their own laws and prosecute on a minor technicality, (b) seek to minimise the crime to cover either their own ineptitude or that of those who run the vulnerable systems or (c) don't want to upset the nice terrists in case they get really mad - hearts and minds and all that carp.

Solutions 1 - Hit the perps hard - a lot of this stuff endangers life and health apart from being costly. Be aware that this will likely lead to war in some cases. Be aware that it's pointless going to war unless you're prepared to win - and clear up afterwards.

Solution 2 - Make it very clear in law that there's a clear audit trail of criminal responsibility for all those responsible for critical systems and their security, including their design and maintenance INCLUDING THOSE IN GOVERNMENT. With appropriate penalties. Not chosen by lazy incompetent greedy fat ....

One can dream.

3
0
Bronze badge
Holmes

Re: no prizes for good guess

"From that it seems quite clear that if a hacker had dumped the entire flouride store into the water supply then nobody would have noticed until either they had to refill it or people started turning up in hospital. Utterly ludicrious."

See my Camelford link later in this thread.

2
1
Silver badge

Re: no prizes for good guess

"which will happen first:

1) government wanting even more surveillance on everybody

2) stiff penalties for companies leaving their systems insecure"

#) Nothing. It's not like They urinated in a reservoir or anything serious like that.

5
0

Your View?

> My view is that the answer to that is something similar to Asimov's first law. "A <system> may not injure a human being or, through inaction, allow a human being to come to harm."

You are obviously an idiot or too young to voice an opinion.

A public utility still using internet access after Stuxnet is liable for manslaughter charges and in any case the management need removing urgently, especially their security bods. If this had happened without such mitigation it would be an act of war. It probably still is.

I hope that Trump is as bomb proof as his predecessor because he really sounds like the sort of arse that America's enemies (or Israel's friends) want in.

0
13
Anonymous Coward

It's "fluoride", not "flouride"

see title

7
0
LDS
Silver badge

Re: no prizes for good guess

Yes, and then Asimov made out a good living by writing about what the three laws of robotics happened to work in some extreme corner cases (and requiring Susan Calvin to understand what really happened). Moreover it postulated the very way the positronic brain was built had them truly "hardwired" and thereby could not bypassed - without damaging the brain irreparably and inoperative. Unluckily software can be modified, and some systems can't really become wholly inoperative, unless some safety mechanism detect it and put the system is a safe state.

Truly airgapped system would require all the air to be removed, so no humans could touch those system and plug in their USB drive to watch some porn while monitoring the systems...

5
0
Anonymous Coward

Re: no prizes for good guess

"Stop adding flouride to tap water, I've got an inalienable right to rotten teeth"

Then don't drink the tap water, buy your own drinking water, problem solved.

Or , stated using the same line of thtinking: "Get your nanny-state coddling out of my tap water, it's my God-given right to drink fluoride-laden water if I so choose".

2
0
Silver badge

Re: no prizes for good guess

Then don't drink the tap water, buy your own drinking water, problem solved.

How do you do that when the municipality you happen to be in has banned sales of bottled water. Apparently the do-gooderesses don't mind Coke, Fanta, Leed etc, but water is a definite no-no.

1
1
Silver badge

Re: no prizes for good guess

#) Nothing. It's not like They urinated in a reservoir or anything serious like that.

I guess no one pointed out to them that fish, birds, and animals all pee and poop in the reservoir and it doesn't get drained and scrubbed.

1
0
Silver badge

Re: no prizes for good guess

“I don't drink water. Fish fuck in it.”

― W.C. Fields

3
0
Anonymous Coward

Re: Your View?

"I hope that Trump is as bomb proof as his predecessor because he really sounds like the sort of arse that America's enemies (or Israel's friends) want in."

For the life of me, I cannot see why this statement is in any way relevant to the discussion.

A down vote for bad cut & paste, or stupidity, or both ...

ps: Same applies if you substitute Clinton for Trump

1
0
Anonymous Coward

Re: no prizes for good guess

"How do you do that when the municipality you happen to be in has banned sales of bottled water. "

Citation needed.

It seems improbable that any government agency can ban the sale of water.

0
0
Anonymous Coward

Re: no prizes for good guess

Well-designed, critical systems usually have hard limits built into them so that such a thing can't happen - not without someone going out there manually (with appropriate tools) and taking the situation in hand, anyway. For fluorine/chlorine and such, I would generally expect such a system to either just reject a "dump everything" command, or to merely increase things to a higher but still relatively safe level - whatever the hard limit restricts it to.

0
0

Re: no prizes for good guess

Daddy, what are stated towers for?

0
0

Re: no prizes for good guess

It will be a furious race - but I predict a tie at the finish line.

0
0

Re: no prizes for good guess

Regarding the concern about dumping an entire storage tank of fluoride into the water system, I have two bits of information that may make you more comfortable.

First, fluoride has a bitter taste. So if there is a severe overdose, people will not drink the water. Trying to cover up this bitterness is a large part of why toothpaste has a strong flavor added, as well as the fluoride treatments at your dentist's office.

Second, most regulators require that for chemicals added to the water that the system run off of what is termed a "day tank". The day tank only stores a limited amount (usually about one day's worth), exactly to prevent the type of overdose that you are referring to. . There are other benefits. Because it is a smaller tank, minor changes in feed rates are noticed sooner.

By the way, this second idea was started long before hackers were born. It is a practical solution that prevents excessive dosing for whatever reason.

2
0
FAIL

The.. just.. I don't even

" login credentials for the AS/400 were stored on the front-end web server."

pardon?!

36
0
Silver badge

Re: The.. just.. I don't even

... and what's with the pejorative "ageing as/400" smack-talk?

If you store credentials on the frontend web server, no amount of "modern" systems or updates are going to save you.

39
0
Silver badge

Re: The.. just.. I don't even

If you store credentials

Question is what credentials. Some credentials - such as what you need to access CRM have to be stored.

Now the fact that the credentials were such that they allowed to manipulate the actual live industrial control systems is the "criminal negligence" bit. As these control chlorine, cloramine and access to drinking water supply there are quite a few criminal charges applicable for the execs of the water company in question in most legislation. Criminal negligence is just the start. I would slap onto them "being accessory to terrorism" without having a second thought.

12
0

Re: The.. just.. I don't even

Any takers for a bet on whether they were for QSECOFR or not?

2
0
Silver badge

Re: The.. just.. I don't even

More to the point what the hell is a web server doing connected to the control systems and being accessible from the internet? Taking it further why were the control systems even anywhere near being connected to the internet?

13
0
Gold badge

Re: The.. just.. I don't even

I would imagine the billing system is probably polling information from the control system. And presumably the treatment controls are on the same system as the network/metering ones. Obviously this should be via a locked down account with no permissions - but I guess it isn't. Well, even more obviously, it shouldn't even be connected - that info should be going to an offline database first.

I can understand wanting to have central control of the system. Rather than having to control things individually at each pumping station and works. But that should be via a private network, not the internet. And there certainly shouldn't be a bloody web server.

Admittedly they do regular testing of the water. But although some of that will be manual, so not vulnerable to computer intrusion, I'd expect that this will also be moving towards automation though.

You can do an amazing amount of damage though. If you control valves, pumps, or worse pumps and valves - then you can easily cause pipes to burst. With chemical dosing you can either overdose or underdose the water and cause problems. Sewage plants are also delicately balanced, in that they have beds which use bacteria to break down some of the waste products - and if too much of certain chemicals gets in there, it kills off the colonies, and stops the treatment plant working.

5
0
Silver badge
Flame

"login credentials were stored on the front-end web server."

Simple solution - someone's b***ocks need to be stored equally publicly on a barbed wire fence. Probably several peoples' .... No need to detach them first,

9
0

Re: The.. just.. I don't even

... and what's with the pejorative "ageing as/400" smack-talk?

Yes. A swing and a miss there for Leyden. I'd much rather have the back end be an AS/400 running, oh, some release of OS/400 V3 than, say, an almost-certainly-misconfigured Win2K system, or never-patched Linux of similar vintage.

4
0
Silver badge

A couple of weeks ago South Derbyshire and North Leicestershire residents were warned not to use their tap water for any reason because the chlorine concentration was at dangerous levels.

Curious coincidence.

28
0

Well, I was one of those customers and given just how little Severn Trent seemed to know about the incident and how it happened it made me wonder too.

After 8 hours there was still much confusion. I saw them doing what looked to me as pumping out a water tower into a long like of waiting tankers the next day.

When I was down getting my 4 litres of free water (generous or what!) We asked the ST woman there why we couldn't shower in it and she said it's chlorine and it's way stronger in concentration that you'd get at the swimming baths. (She really couldn't stress just how much we really shouldn't use it to even wash hands). So if something looked like a computer error or hack this is a likely candidate.

Then again, could just be coincidence. Guess we will never know!

16
0

Many water companies who abstract ground water (like ST) use superchlorination - they add a lot of chlorine to guarantee to kill any bugs then reduce the chlorine levels before it hits supply, without needing an intermediate tank/reservoir - it goes straight down the pipe. A mechanical/electrical failure at any point in the dosing system could allow high chlorine levels to get through to supply without the system getting hacked.

6
1
Silver badge

Hey shit happens. Back in the 1990s, when I worked in a chemical factory, we had a water treatment guy in to dose the cooling tower water with biocide (legionnaires). Unfortunately they didn't tell anyone that they'd done it. So some maintenance fellows comes on shift and opens up a valve to let water into the local canal. A few hours later the surface of the canal was covered in dead and dying fish.

10
0
Anonymous Coward

AS400?

Could this be STW? A couple of decades ago (and before Sir Tim invented WWW) I worked on a SCADA system for Severn Trent that could, in theory, be used to control a water treatment plant. Being pre-WWW it didn't have a front end server, and it ran on hardware that was somewhat more mature than the AS/400, (not that I'm prepared to say what it ran on). I did hear from a reliable source that the old software had been ported to new hardware (AS400?) and it is entirely possible that a ropey old web front end was bolted on to the port. I also wonder if this is a coincidence.

1
0

Yep - super chlorination, or shock dosing.

Anything above 0.5 ppm HClO will kill most bacteria, and your average swimming pool will be 1 to 3 ppm to ensure all those scutty people who don't shower before going for a swim doesn't bring in any nasties, and also to make sure if little Johnny curls off a floater, then it won't need the pool to be evacuated and drained!

Obviously you don't want to be drinking the contents of your local pool, but it won't kill you.

Hot spas and things like that can be maintained between 3 and 6 ppm, but as you aren't in for too long, it won't cause any problems.

Anything above 6ppm however is really not advised, as at this concentration, you will start to get bleaching, and sensitive skin can start getting rashes and irritation.

If you hit anywhere above 10 - 12 ppm, and you really really do have a problem. I can only assume that the STW recent problem had HClO levels way above 3 - 6 ppm.

(I recently did the STA water treatment course.....)

Anyone remember Milton Sterilising Tablets? Maybe someone bunged a few of these into the reservoir.......

11
0
Silver badge
Facepalm

Bullshit bingo

Monzy Merza, Splunk’s director of cyber research and chief security evangelist, commented: “Dedicated and opportunistic attackers will continue to exploit low-hanging fruit present in outdated or unpatched systems. We continue to see infrastructure systems being targeted because they are generally under-resourced or believed to be out of band or not connected to the internet.”

“Beyond the clear need to invest in intrusion detection, prevention, patch management and analytics-driven security measures, this breach underscores the importance of actionable intelligence. Reports like Verizon’s are important sources of insight. Organisations must leverage this information to collectively raise the bar in security to better detect, prevent and respond to advanced attacks. Working collectively is our best route to getting ahead of attackers,” he added.

Every card a winner!

Seriously, who writes this stuff?

22
0

Re: Bullshit bingo

And Verizon Enterprise, the guys who do write these intrusion reports, got hacked themselves, according to krebsonsecurity...

http://krebsonsecurity.com/2016/03/crooks-steal-sell-verizon-enterprise-customer-data/

6
0

Re: Bullshit bingo

And Verizon Enterprise, the guys who do write these intrusion reports, got hacked themselves

Well, sure. The question is, how good was their report about it?

3
0

Meanwhile ...

I heard that another US-based hacktivist group had got away with doing similar tricks for some years before they were stopped.

7
0

I don't understand why the control system is linked to the customer payment portal, and why the payment portal would need credentials for the control system.

Or did both systems just happen to run on the same AS/400? (REALLY?)

10
0
LDS
Silver badge

Because there was a time when "consolidation" was the buzzword like cloud is today (cloud is still a form of consolidation...). The mantra was to run everything on fewer, more powerful systems to save money. Done in the right way it could be OK, done in the wrong way by clueless people "hey, we have this AS/400 let's run both the water control system and accounting from it! See how much we saved?" leads to these situations. Of course IBM told (and sold) you you could run different workloads on it, so why not? The AS supported hardware partitioning - but if used by clueless syadmin, little changes...

11
0
Gold badge

It could be there's some bigger commercial/industrial customers whose meters are reported directly on the network's controls systems. So the billing system uses that info to charge them. Not sensible, but doesn't mean someone hasn't done it.

1
0
Silver badge
Joke

"consolidation" was the buzzword like cloud is today ...

Ah - there's the solution stairing everyone in the face.

Clouds bring rain. End of water shortages.

Someone with good ideas just needs a legup.

3
0
Silver badge

I agree on why they had to be linked and why the credentials were so wide, but I don't see anything wrong with running it all on the same box, especially not an AS/400 which supports LPARS. No real difference to running on Xen today.

2
1
LDS
Silver badge

Did they use partitions or not? The fact the AS/400 supports LPARS doesn't mean it was in use.

Also, even today running software at different security level on the same hypervisor *can* be a security issue. There are bugs in hypervisors (and even in CPUs...) that let an attacker compromise other VMs. Thus, even if it costs more, may be sensible to run software on truly separated hardware.

But everything becomes useless if there are easy channels between systems and powerful credentials are stored everywhere.

2
0
Anonymous Coward

Because organizations which run generally safe, sane, and relatively secure systems like the AS/400 (and its successors) don't usually see the need to carve things up unnecessarily, although some separation of duties may have been a wise decision in this particular case. But I have worked with/for several companies now who have gone down the path of "modernizing" their systems, by moving things over to some number of different (mostly) dedicated servers, only to often quickly run into the problem of not knowing why/when/where things are going wrong, nor of course how to fix it.

I'm dealing with that very issue right now, in fact, where instead of things staying on the AS/400 where they really belonged, they've been spread out across several different servers of various types. But critical things are occasionally failing now where they didn't fail before, and the situation is getting progressively worse, and nobody really understands enough about the whole set-up (nor do they generally have the time or the patience) to really be able to go in and find the problem and fix it. Which is where I come in, because I've had to run such rabbits down in the past, at other organizations.

0
0
Anonymous Coward

The law needs changing, and soon

All critical national infrastructure (water, power, etc) should be air gapped from the internet immediately, and anyone who attempts to implement internet connectivity as a cost cutting measure should be imprisoned. Cost cutting will bite us all on the ass eventually.........

13
0
Silver badge

Re: The law needs changing, and soon

Eventually???? I daresay it already has bit us and bit us hard. The problem is, it hasn't changed the C-suite types thinking since all they focus on is profit.

6
0
Silver badge

Demarcation?!?

Why the hell is a control system on a publicly accessible network in the first place? Something like that should be on a self-contained network to prevent anything like this being possible. It beggars belief that all these utility companies don't have better network designs.

14
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

The Register - Independent news and views for the tech community. Part of Situation Publishing