nav search
Data Center Software Security Transformation DevOps Business Personal Tech Science Emergent Tech Bootnotes BOFH

back to article
Penguins force-fed root: Cruel security flaw found in systemd v228

Systemd... just FOAD!! **PLEASE** !!!

70
7

Surprise

Once again, the deliberate hole in the *nix security model - SUID/setuid - is at the core.

4
33
Silver badge

Re: Surprise

Are you absolutely sure it's not people violating the basic security tenets of pulling more power to themselves than they need, whilst also declining to relinquish that control and power when they don't need it?

How many of systemd's logic, state tracking, and internal storage systems required God power? I'm guessing none. If they won't even use basic security practise, what makes you think they'll use finer grained ACL systems?

Which OS would you like to evangelise that has the power to restrain the ego of its boot-program's author?

30
1
Silver badge

Re: Surprise

No.

Why the hell is a "touch" command creating a file with any permissions other than that of the user in question, whatsoever, and why the F***ING hell is it creating setuid bits with explicitly being asked to? That's just insane! Literally insane.

It's like a ping command that turns off your firewall when you run it on localhost. So incredibly stupid, it just shouldn't happen.

Systemd is a good "principle" (i.e. let's get rid of old shell scripts and make things predictable and reliable), coupled with a bad design (one massive executable that runs as the init user), tied with god-damn atrocious execution (like this, and a handful of other doozies that it's incorporated).

And, most importantly, isn't even brave enough to say "Whoops, we really messed up back in January! Patch immediately".

15
0

Re: Surprise

> Why the hell is a "touch" command...

Which "touch" command?

Got confused the command and a function called touch() inside systemd?

1
0
Silver badge
Unhappy

Re: Surprise

_NOT_ suid per se [it's actually needed for SOME things] but apparently how it's being utilized by systemd and it's minions

2
0
Joke

Re: Re: Surprise

Systemd is a good "principle" ,,,, coupled with a bad design ,,,,, tied with god-damn atrocious execution

So it's like the European Union?

0
0

How is Debian affected? I see version 232 in unstable and testing, and version 230 in backports and 215 in stable. No version appears to be running 228 and hasn't been for at least 6 months. Version 229 was released almost a year ago, so at this point I doubt there are really any vulnerable systems out there.

6
4
Anonymous Coward

Version 229 was released almost a year ago, so at this point I doubt there are really any vulnerable systems out there

Unless, you know, you run a production system, which you do not update-and-reboot the second a new version of a package rolls off the press.

Monolithic, all-absorbing tools like systemd are an abomination for a production server: it is pretty much guaranteed that -something- inside it is broken, but you can't just replace the broken bit with an updated version. To add an insult to injury, since it has a tentacle in every tiniest crack of the system it can find, it is almost impossible to update it fully without rebooting.

Unfortunately, it is by now impossible to avoid this abomination if you have to stick with a major distribution. I wish its creator have never been born, though - or at least gets to one of the more unpleasant corners of hell once he is no longer with us.

56
5

right ..

I'll take your "abomination" Thanks to SystemD I finally have shared filesystem clusters booting correctly the first time without a ton of hackery. (dovecot depends OCFS2, OCFS2 depends on iSCSI, iSCSI depends on networking) something that was almost impossible to do under the old init system.

As a server admin, SystemD has solved more problems than it has created.

9
31
Silver badge

Re: right ..

How about you start networking first, then iSUSI, then OCF32, then Dovecot? Just throwing that out there.

39
0
Anonymous Coward

Re: right ..

Unless I miss something rather major, all what had to be done under the system V init system was to name your scripts in /etc/init.d/rc5.d in the desired startup order, and it simply worked.

If you wanted to be fancy (or just a believer in LSB), you could also supply the appropriate "Required-Start:" comment lines, and let your GUI tools sort out the startup order.

Either way, hardly a "ton of hackery" or "impossible to do".

46
0
Anonymous Coward

Re: right ..

@g. Mack: I find systemd to be a nightmare to debug, but if it helps, more power to you. But, have you thought about writing your own tool? I'm confused on how systemd made this easier for you. Of course I wonder how easy it will be when... ooops.

9
2
Anonymous Coward

Re: right ..

Init doesn't have dependency-based starting, only delay-based starting, and due to Murphy, sometimes things don't start up in a timely manner, leading to things going up in the wrong order. I recall a network pro mentioning the time the two network interfaces in a firewall unit going up in reverse order and thus getting enumerated wrong. Do one thing sounds nice until you found out it did that one thing WRONG.

7
17
Silver badge

@Gerhard

"Thanks to SystemD I finally have shared filesystem clusters booting correctly the first time without a ton of hackery. (dovecot depends OCFS2, OCFS2 depends on iSCSI, iSCSI depends on networking) "

You almost make it sound as if this wasn't possible without systemd. I guess it's a miracle then that the process of setting something like this up on FreeBSD has always been relatively easy. And FreeBSD knows nothing of systemd.

Instead of thanking systemd I can't help wonder if you shouldn't have been scorning some package maintainers for creating a dependency hell instead. Also: merely removing said dependency hell did not remove nor change the underlying mindset. Within that reasoning I think its safe to conclude that systemd didn't fix anything, it only postponed the inevitable.

29
2

Unless, you know, you run a production system,

You run production systems on testing or unstable?

Did you miss this bit: "215 in stable".

6
0

> I wish its creator have never been born, though - or at least gets to one of the more unpleasant corners of hell once he is no longer with us.

Linus, is that you?

> Unfortunately, it is by now impossible to avoid this abomination if you have to stick with a major distribution.

Write up a costs/benefits analysis for your employer. If you can prove that the business is losing money in sysadmin labour-hours dealing with systemd then maybe you can switch.

3
11
Anonymous Coward

After reading some of the comments here, I think the problem might be that if you administor Unix, you kinda, sorta, have to be a C programmer. Checking for process status, writing modules or just the general debug mind set might not be expected of todays Unix admins.

0
16
Anonymous Coward

Re: right ..

"As a server admin, SystemD has solved more problems than it has created"

If that is true from your perspective, then it seems you've never learned to properly sysadmin Unix using its basic principles ... all of what you described can be easily configured using the level-based init scripts

22
6
Anonymous Coward

"After reading some of the comments here, I think the problem might be that if you administor Unix, you kinda, sorta, have to be a C programmer"

No. You only have to have an grasp of the basic patterns in the Unix methodology. Mostly they're straight forward and make sense, and are designed so that most things are human readable and can be set up in small easily manageable units.

When managed sensibly, a Unix system very rarely needs to be rebooted.

SystemD seems to be the complete opposite of the Unix design approach, and will only make systems more complicated to manage and more vulnerable to failure.

35
1
Orv
Silver badge

Re: right ..

I'm glad it works for you. I missed being able to reboot my servers without resorting to a hardware reset. (RH7+systemd+automounts+NFS causes a hang on shutdown every time.)

6
0
Silver badge

Re: @Gerhard

Yes it was possible to start things up in a defined order with SystemV startups but it was sometimes a little iffy when it came to dependencies. Checking for say a file lock or existence didn't always work in a clustered setup.

Since we moved to SystemD these sort of things have not happened.

Yes some of SystemD is a real PITA and yes debugging it is even worse but after getting through the pain, it was worth it.

For 90% of systems it is a total overkill. SystemV would do just fine but where you have multiple dependencies then I've found SystemD a better bet.

4
8

@Dan 55 Re: right ..

"How about you start networking first, then iSUSI, then OCF32, then Dovecot? Just throwing that out there."

You can mess with the boot order, but the pain comes when you actually want to mount the filesystem. /etc/fstab has an option to wait for the network but none to wait for the iSCSI/OCFS2

and in most distros a failed mount would simply then start loading the rest of the services. This means adding checks to each daemon's startup script to make sure all file systems are mounted.

Under systemd, I can just add a config file for just the mount point that causes me trouble, and it does not even attempt to mount the FS until all needed services are up and then mark the postfix/dovecot as a dependancy of the filesystem. Much less fragile, and my time to setup new services has been reduced and as an added bonus, I don't have daemons saving things to the local drive instead of the network drive which is kind of a pain when it's supposed to be a shared FS.

3
5

Re: @Dan 55 right ..

Tell fstab to not auto mount it and then mount it inside the init script once everything is up?

9
0

No debian release has ever had the problematic version. Your production system would hence not be affected.

2
0
Anonymous Coward

Re: right ..

surely each interface has a mac address and if that is configured in the netconfig files then that should be impossible?

1
0
Silver badge

Re: @Gerhard

"For 90% of systems it is a total overkill. SystemV would do just fine but where you have multiple dependencies then I've found SystemD a better bet."

I've had the opposite experience. On a proper SysV system, it will start daemons in the order they are listed in rc.conf.local. Clustering was never a problem for me since I've always used properly coded applications where their start-up script would check for a file lock and would start it up in a passive/stand-by mode if the file was locked.

Of course, it helped that I made the afternoon-worth of effort to learn basic Perl / shell scripting that the tiny startup scripts were easy for me to follow, unlike the spaghetti that is systemd.exe

8
2
Silver badge
Unhappy

Re: right ..

"As a server admin, SystemD has solved more problems than it has created."

I hear the same kind of logic applied to things like Virus Outbreak (aka MS Outlook), and the use of ".Not" in programming.

My guess is that the hackery is a better choice. Then post what you did in an appropriate place, so the rest of us can benefit from it too.

1
5
Silver badge
Devil

Re: @Gerhard

"And FreeBSD knows nothing of systemd."

A _DEFINITE_ plus!

Additionally, the rc system on FBSD is a bit easier to config than Debian's old sysv startup support, last I checked. though much of the 'hackery' required was simply re-naming, creating, or removing a few symlinks from one of the rc#.d directories

2
0
Anonymous Coward

Re: right ..

Surely you mean rc3.d? Right?

0
0

Re: @Dan 55 right ..

@Gerhard Mack

Fun dependencies like this exist with systemd; for example if you have xinetd and a service binary on NFS... xinetd starts before NFS does and so the service fails. You have to force xinetd to start after NFS if you want this to work properly.

Similarly I have an rc.local script that sends an email to my phone each time my server reboots; postfix is running but can't do DNS resolution and so the message sits there waiting for retry. INN also suffers a DNS failure and so won't allow any incoming connections. I've not fully debugged those ones, yet!

None of those issues hit with sysvinit on CentOS.

4
0
Silver badge

Re: @Gerhard

Waiting until something (eg, some initialisation) is finished is trivial:

1. Make a named pipe (done once)

2. Thing waiting on the dependency reads from the pipe and hence blocks

3. Thing providing the dependency writes "success" to the pipe when it's finished

4. init script wrapper sends "fail" to the pipe after a programmable timeout

5. Thing waiting on the pipe looks for "success" (in which case it stops the timeout program and continues as normal) or "fail[ure]" and does error logging

You could do this dependency stuff in any number of ways, but this is doable in a script that you put into /etc/init.d and requires no more than mkfifo, read, echo, sleep and kill commands.

10
0
Bronze badge
Coffee/keyboard

Re: @Stephen W Harris

If you have a service binary on NFS, you are already asking for trouble no matter what startup system you are using.

7
1
Silver badge

Re: right ..

"all of what you described can be easily configured using the level-based init scripts"

Scripts that can be subtly MISWRITTEN. I speak from experience. Like I said, do one thing sounds nice until it does the one thing WRONG. Think of the term command chain. And chains tend to have weak links; problem is, it's usually tough to find which link is the weak one until it's too late.

0
0
Silver badge

Re: @Dan 55 right ..

Even if you have xinetd on an NFS share - I am curious, but I won't ask why because the standard reply to weird behavior is always "Corporate Policy" - you still have a shell script to start xinetd, right ? how about having a loop in there, checking for "-x /nfs/path/to/xinted"; sleep 3 ?

If you have xinetd on an NFS share, might as well have a loop in your daemon script that starts/stops it.

0
0
Silver badge
Happy

Re: right ..

>Scripts that can be subtly MISWRITTEN.

That is what testing is for, mate. at least, with shell scripts, updates installed won't suddenly expose your init system ... I am still waiting for the network-based 0wnage to happen via systemd, it is only a matter of time. Seriously, look, they could not even get touch() right ...

What do you get from a bunch of young devs ? A monolithic system with an atrocious ui!

If that's what you get from that bunch of young devs, I'll never let them code again!

1
1
Silver badge

Re: right ..

"What do you get from a bunch of young devs ? A monolithic system with an atrocious ui!"

And what do you get from a bunch of devs young and old that don't talk to each other? Programs that talk past each other and result in esoteric gestalt exploits that are a right PITA to debug. Pick your poison.

0
0
Silver badge

Re: right ..

The thing about rc.init routines is that you have to select the start order (and some packages get it wrong)

Additionally, things tend to be singlethreaded, which can make for long startup times.

systemd uses dependencies and runs as much in parallel as possible.

Which is the nice idea part. It does tend to die horribly under some conditions.

Contrary to claims, it's not monolithic. There's a core which fires up startup scripts.

The biggest problem with it is that the developer is an asshat who won't listen to anyone else.

2
0
Silver badge

Re: @Gerhard

And FreeBSD knows nothing of systemd.

Which is exactly why I'm currently going thorugh the pain of migrating my VMs over to FreeBSD[1], despite the learning curve. And reprogramming my fingers to not do ps -ef but ps -ax instead.. Just have to remember my Solaris days.

[1] My friends comment: "You do like making things difficult for yourself!"

0
0

Re: @Dan 55 right ..

@Hans 1 - you misread; xinetd is the standard RedHat one; the _service_ entry points to an NFS mounted executable. eg /etc/xinetd.d/myservice might have "server = /nfs/mounted/path/foobar"

Now xinetd starts before NFS is properly ready (standard RedHat systemd startup), it can't find "foobar" and so disables the service.

This only broke in RedHat 7; it worked fine for decades (and even via inetd on SunOS 4).

0
0
Silver badge
Facepalm

systemd v228

Yeah. That explains everything wrong with systemd in two words.

10
4
Linux

Re: systemd v228

I suppose less must be more than twice as wrong I suppose -- because it uses the same versioning scheme (just do +1) and currently is at v487 (oh, the horror!).

0
2
Silver badge

Re: systemd v228

less shows you a bit of a file, and has been around since the 80's

SystemD runs your computer including almost all the security, devices and kernel interaction, and has been around 6 years.

There's a bit of a difference.

For a start, 'less' has twice as many versions but has been around nearly 5 times as long. It also does SIGNIFICANTLY less (sorry!) than systemd, as in it does basically nothing critical beyond an open() call on a file, and has no security implications inherent in it.

But systemd would have seen MORE updates than less if it had been around as long, but not significantly more which - given that it does almost everything nowadays - is quite damning. I'd expect it to be in the tens of thousands of revisions, with a handful (6?) major versions only. Not 200-odd randomly chosen updates.

And less can be updated in a trice without affecting anything. Systemd, you certainly shouldn't be just throwing out a version every few days and upgrading every machine to it.

1
0

Re: UNIX pagers - less is more - more or less :p

A bit off topic - my apologies - but I had to smile as I thought back to the 'pg' days, which was followed by a program to let you read a page 'more' and then later you could read a page by page (pg) any way you wanted - well 'more' or 'less'

0
0
Silver badge

Re: UNIX pagers - less is more - more or less :p

Although depreciated, `pg` is part of util-linux. If your distro of choice omits pg, you can download it (and the rest ... what other "necessary" files does your distro of choice omit?) from here:

ftp://ftp.kernel.org/pub/linux/utils/util-linux/

pg still comes in handy here, at least once in a while :-)

0
0
Pint

use sysvinit instead

"Unfortunately, it is by now impossible to avoid this abomination if you have to stick with a major distribution".

I hate the philosophy of systemd too, but it's still fairly straightforward to run the current Debian release using sysvinit instead.

I switched all my servers back to sysvinit when I discovered that during a standard reboot systemd was shutting down logging to syslog BEFORE all applications had been cleanly shutdown, thus important messages were lost. For instance, If you just went by syslog it would appear as though Mysql had crashed and not been shut down cleanly.

Anyway a guide to switching back to sysvinit here, it's very simple:

http://without-systemd.org/wiki/index.php/How_to_remove_systemd_from_a_Debian_jessie/sid_installation

24
1
Silver badge
Devil

Re: use sysvinit instead

http://without-systemd.org/wiki/index.php/How_to_remove_systemd_from_a_Debian_jessie/sid_installation

That's what I've been doing too, makes things much nicer. I hope things are as easy when Stretch comes out, or Devuan is ready by then.

Then there's the RedHat/Centos stuff, where systemd is actually beneficial for me. I only touch RedHat/Centos via subcontracting to someone who takes and gives zero feks and who bills by the hour, so broken shitty horror shows pretty much turn directly into money.

14
1

Re: use sysvinit instead

Would that removal of systemd work for the latest Ubuntu/Mint too..? Has anyone tried it..?

0
0
Anonymous Coward

Re: use sysvinit instead

who takes and gives zero feks and who bills by the hour, so broken shitty horror shows pretty much turn directly into money.

Hehehe - Consider adding MySQL replication to you repertoire. One year I almost doubled my salary just by this pile-o-crap losing sync, shitting itself and then manual re-sync - which invariably takes forever on a loaded server. In Asia, which is Night here for Peak Load.

2
0
Ogi
Bronze badge

Re: use sysvinit instead

> I only touch RedHat/Centos via subcontracting to someone who takes and gives zero feks and who bills by the hour, so broken shitty horror shows pretty much turn directly into money.

Maybe I have been round the block one too many times, but when systemD first came out and started being pushed through every oriface by Red Hat and their proponents, I couldn't help but think the reason was so that they could make a mint off it.

Think about it, the billable hours to nurse systems back to health, the many more hours to debug that which is the mess of systemd, the constant patching and security monitoring of this massive blob that has its fingers across the system. The frustration of debugging/managing it, leaving it to magic incantations that only Red Hat Certified(tm) systemD consultants could do, and the extra income to get that certification.

It is like someone at RH took a look at Linux, and said "It's too simple, reliable and robust. We are not making enough money on support contracts. We need to make more like Windows, so we can make more money fixing problems we created in the first place. Perhaps some analogue of the Windows registry and svchost.exe? That causes untold billable support tickets for MCSEs!".

Et voila! systemD was born. Like Windows, it is easy to get started "out of the box", things "just work" as long as you do things as the distro is configured for. That is fine for most novice admins and home desktop users. They may even welcome it as they don't realise how powerful Unix actually is, and treat it more like Windows.

Once you start customising to your needs, or something breaks, all hell breaks loose and the whole thing goes titsup. Now its faster to reinstall Linux than work out why systemD threw its toys out of the pram this time.

(as for me, switched to FreeBSD and Devuan a while ago for my personal and business servers, so far so good. Still running Devuan through its paces before I recommend to clients).

2
1

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

The Register - Independent news and views for the tech community. Part of Situation Publishing