"The Tragedy of systemd", a rebuttal
Posted 2018-11-17 14:46 PST | Tags: software systemd
Benno Rice gave an interesting talk at the 2018 BSDCAN, titled "The Tragedy of systemd" -- https://www.youtube.com/watch?v=6AeWu1fZ7bY
He presents a very sympathetic narrative about systemd from the BSD perspective, and he does it fairly well, but he also makes some assertions of dubious validity.
Early in the talk he proposes there is a "confusion" between system configuration and service bootstrap in traditional UNIXy models. He talks about mounting filesystems and bringing up the network, and how these are "slightly disparate things" which should be treated differently and managed with different tools.
From my perspective, one of the strengths of the UNIX approach is that it abstracts a great many "slightly disparate things" as though they were the same thing. Making (almost) everything a file is a good example of this. When I was new to UNIX, it seemed strange to me, but over time I have grown appreciative of how powerful this abstraction can be.
When things are sufficiently different that they need to be treated as something other than a file, we have system calls like ioctl() specifically for that, and in my experience that has worked well.
Later he made a curious assertion without any accompanying reasoning to support it, that "other things manage services well [..] Windows always had a strong notion of services". He proceeds from there on the assumption that his assertion is true to discuss Windows' contract-oriented (declarative) approach, which he says "has been kind of neat".
I've known people to be used to Windows service management, but I've never known anyone who was familiar with both Windows and UNIX who preferred Windows service management. Mostly I've heard them complain that Windows was inflexible and opaque. I've not spent any time configuring Windows services myself, so perhaps some of you with such experience could weigh in on this. Is there anything to be said about Windows' declarative service management approach which warrants admiration?
Benno also praises the ideas behind launchd -- "if you need to have services running all the time, and you can't call the system booted until services have started, that's such a pain in the ass" and "We know we'll need this at some point but we won't start it until we actually need it." This is an approach systemd borrowed from launchd, not starting services until something requests them.
The problem I have with this approach (which is also applicable to inetd) is that when the system comes up, I want to know that its services will work. The litmus test for this is bringing the service up so that it processes its configuration and finds any errors or missing dependencies. This can be improved upon by monitoring services with things like Nagios, which submits requests and validates the results.
If the service doesn't come up until it's needed, then we don't know if it will work until the moment something actually needs it, which seems like a really bad idea. Also, this approach presupposes a lack of monitoring. Production systems absolutely should be monitored. If Nagios hits up a service immediately or soon after system start and starts the service, then why did we bother deferring the service start in the first place?
He also talks about how modern systems need to be more reactive to changes around them, which is absolutely true, but IMO the traditional UNIXy toolbox has adapted fairly well in this regard. It's not done yet -- things like changing wireless networks without interrupting TCP connections still needs some work -- but pitching the toolbox entirely is unwarranted.
He is dismissive of criticism of systemd as buggy, saying sardonically "it's software" and "we've all had bugs in our code". He says that if we hold init to a higher standard of quality, that implies we can never write another pid 1.
That is simplistic to the point of dishonesty. There are certainly ways we can reduce or mitigate bugs in our software. We can make software simpler, so that there are fewer things to go wrong. Less code means fewer bugs. We can also limit critical dependencies so that a failure in one component doesn't cause the system to more broadly fail (as is the case with systemd when dbus fails). We can also hold on to known-good, already debugged software until such a time that new software has absorbed enough debug/release cycles on ancillary systems that we can trust it on mission-critical systems.
Systemd as a project does none of these things. It is by no means simple. It contains a great deal of code, and thus a great number of bugs, which its developers have shown themselves reluctant to fix. It is tightly integrated with a variety of far-flung components and is vulnerable to any of them failing. It has been forced into adoption on mission-critical systems before it is ready (as RHEL6/CentOS6 fall out of support and there are no sufficiently good alternatives to RHEL-derived distributions in the enterprise).
Benno also says "UNIX as a concept is dead" in that we no longer have a diversity of UNIX systems across which software must be ported, and we don't have to be beholden to POSIX when it is inconvenient. This seemed curious, coming from a BSD developer in a Linux-dominated world. Projects do have reason to be portable. Projects can get away with being Linux-specific because Linux is dominant, but that's no substitute to being portable. How many projects of the past targeted the dominant platform of their time only to be left behind when the dominant platform changed? There is a vast ocean of excellent software which was written for MS-DOS or classic MacOS which were never ported forward to those platforms' successors.
He then talks about how "change can be scary" and "change threatens what we find familiar". These are true things, but he makes it sound like anyone who opposes systemd is a narrow-minded neanderthal. He also does not recognize that some change can be genuinely bad, or that there might be other ways to change which are less bad. He assumes that since people are reacting badly to the changes represented by systemd, then those people will always react badly to changes, and challenges people to overcome their "kneejerk reactions" and accept systemd rather than be opposed to all change in general.
As someone who dislikes systemd on the basis of its design and implementation, that felt a little disingenuous.
He said a lot more which I'm still mulling over, but these points stood out to me as rather uncompelling, and they make me view all of his points with a sharply critical eye.
Blog comments are not working at the moment, but please feel free to reply via these alternative channels:
* My LQ blog -- https://www.linuxquestions.org/questions/blog/ttk-652585/
* IRC channel ##slackware-help on freenode
* Facebook -- https://www.facebook.com/ttkciar
* Twitter -- https://twitter.com/ttk_ciar
* Email (please let me know if I may share your message via this blog) -- ttk (at) ciar (dot) org
Posted 2018-11-17 14:46 PST | Tags: software systemd
Benno Rice gave an interesting talk at the 2018 BSDCAN, titled "The Tragedy of systemd" -- https://www.youtube.com/watch?v=6AeWu1fZ7bY
He presents a very sympathetic narrative about systemd from the BSD perspective, and he does it fairly well, but he also makes some assertions of dubious validity.
Early in the talk he proposes there is a "confusion" between system configuration and service bootstrap in traditional UNIXy models. He talks about mounting filesystems and bringing up the network, and how these are "slightly disparate things" which should be treated differently and managed with different tools.
From my perspective, one of the strengths of the UNIX approach is that it abstracts a great many "slightly disparate things" as though they were the same thing. Making (almost) everything a file is a good example of this. When I was new to UNIX, it seemed strange to me, but over time I have grown appreciative of how powerful this abstraction can be.
When things are sufficiently different that they need to be treated as something other than a file, we have system calls like ioctl() specifically for that, and in my experience that has worked well.
Later he made a curious assertion without any accompanying reasoning to support it, that "other things manage services well [..] Windows always had a strong notion of services". He proceeds from there on the assumption that his assertion is true to discuss Windows' contract-oriented (declarative) approach, which he says "has been kind of neat".
I've known people to be used to Windows service management, but I've never known anyone who was familiar with both Windows and UNIX who preferred Windows service management. Mostly I've heard them complain that Windows was inflexible and opaque. I've not spent any time configuring Windows services myself, so perhaps some of you with such experience could weigh in on this. Is there anything to be said about Windows' declarative service management approach which warrants admiration?
Benno also praises the ideas behind launchd -- "if you need to have services running all the time, and you can't call the system booted until services have started, that's such a pain in the ass" and "We know we'll need this at some point but we won't start it until we actually need it." This is an approach systemd borrowed from launchd, not starting services until something requests them.
The problem I have with this approach (which is also applicable to inetd) is that when the system comes up, I want to know that its services will work. The litmus test for this is bringing the service up so that it processes its configuration and finds any errors or missing dependencies. This can be improved upon by monitoring services with things like Nagios, which submits requests and validates the results.
If the service doesn't come up until it's needed, then we don't know if it will work until the moment something actually needs it, which seems like a really bad idea. Also, this approach presupposes a lack of monitoring. Production systems absolutely should be monitored. If Nagios hits up a service immediately or soon after system start and starts the service, then why did we bother deferring the service start in the first place?
He also talks about how modern systems need to be more reactive to changes around them, which is absolutely true, but IMO the traditional UNIXy toolbox has adapted fairly well in this regard. It's not done yet -- things like changing wireless networks without interrupting TCP connections still needs some work -- but pitching the toolbox entirely is unwarranted.
He is dismissive of criticism of systemd as buggy, saying sardonically "it's software" and "we've all had bugs in our code". He says that if we hold init to a higher standard of quality, that implies we can never write another pid 1.
That is simplistic to the point of dishonesty. There are certainly ways we can reduce or mitigate bugs in our software. We can make software simpler, so that there are fewer things to go wrong. Less code means fewer bugs. We can also limit critical dependencies so that a failure in one component doesn't cause the system to more broadly fail (as is the case with systemd when dbus fails). We can also hold on to known-good, already debugged software until such a time that new software has absorbed enough debug/release cycles on ancillary systems that we can trust it on mission-critical systems.
Systemd as a project does none of these things. It is by no means simple. It contains a great deal of code, and thus a great number of bugs, which its developers have shown themselves reluctant to fix. It is tightly integrated with a variety of far-flung components and is vulnerable to any of them failing. It has been forced into adoption on mission-critical systems before it is ready (as RHEL6/CentOS6 fall out of support and there are no sufficiently good alternatives to RHEL-derived distributions in the enterprise).
Benno also says "UNIX as a concept is dead" in that we no longer have a diversity of UNIX systems across which software must be ported, and we don't have to be beholden to POSIX when it is inconvenient. This seemed curious, coming from a BSD developer in a Linux-dominated world. Projects do have reason to be portable. Projects can get away with being Linux-specific because Linux is dominant, but that's no substitute to being portable. How many projects of the past targeted the dominant platform of their time only to be left behind when the dominant platform changed? There is a vast ocean of excellent software which was written for MS-DOS or classic MacOS which were never ported forward to those platforms' successors.
He then talks about how "change can be scary" and "change threatens what we find familiar". These are true things, but he makes it sound like anyone who opposes systemd is a narrow-minded neanderthal. He also does not recognize that some change can be genuinely bad, or that there might be other ways to change which are less bad. He assumes that since people are reacting badly to the changes represented by systemd, then those people will always react badly to changes, and challenges people to overcome their "kneejerk reactions" and accept systemd rather than be opposed to all change in general.
As someone who dislikes systemd on the basis of its design and implementation, that felt a little disingenuous.
He said a lot more which I'm still mulling over, but these points stood out to me as rather uncompelling, and they make me view all of his points with a sharply critical eye.
Blog comments are not working at the moment, but please feel free to reply via these alternative channels:
* My LQ blog -- https://www.linuxquestions.org/questions/blog/ttk-652585/
* IRC channel ##slackware-help on freenode
* Facebook -- https://www.facebook.com/ttkciar
* Twitter -- https://twitter.com/ttk_ciar
* Email (please let me know if I may share your message via this blog) -- ttk (at) ciar (dot) org