Systemd for web applications: Hrmmm

This debate is such a disaster area I hesitate to add anything. Rest assured no religious strife follows, only a careful look at my own use case for Linux, which I suspect is a common one.

Exposition

First, for those just joining us: systemd is a replacement for traditional Unix init, the first process to be run after the kernel is loaded, and the process responsible for booting the rest of the system. (It has some other interesting properties too, especially that it becomes the parent process of all processes whose parents exit.) There are two historical models for this: Berkeley init, which is (said to be; see below) underengineered, and the System V model, which is (definitely) overengineered. Many of the popular Linux distributions (RedHat, Debian, their progeny) have historically used SysV init, and irritations with it begat upstart, and eventually systemd, as replacements.

Depending on who you ask, you’ll get different answers about what the primary irritants are. But common complaints about SysV init are (a) the arcane interface, which is sort of a finite-state machine controlled by shell scripts and symbolic links; (b) the design centered on “runlevels,” because very few people need to boot the same machine into, say, headless server mode sometimes, and full graphical workstation mode other times; (c) slow boot times, possibly due to having to lanuch everything sequentially rather than in parallel, and so on.

There is every reason to want to be rid of SysV init. (Though that says nothing about what a good replacement would look like.) Systemd answers with a maximalist strategy: It not only boots the system, but monitors active processes, provides a socket-based communication bus, subsumes logging functions, device generation and so on. That is, SysV init isn’t the only problem systemd sets out to solve. A few others:

…and probably a whole bunch of other things. The primary design goal seems to be “make the Linux work.”

The adoption of systemd in major Linux distributions (particularly Debian) was met with a certain level of rancor, for a number of reasons. This conversation has devolved mostly into unintelligble wailing and gnashing of teeth, unfortunately. There are valid points for and against, but (all rancor aside) I think the main problem is the idea that one solution is required, or even desirable, for all Linux use cases.

One-size-fits-all solutions

A few years ago when Mark Shuttleworth was touting the benefits of a single operating system for your phone, desktop and servers, I have to admit I liked the idea. It seems so simple! Upon further reflection I had to give up the fantasy though, mainly because (at the risk of stating the obvious) phones, desktops and servers represent three utterly different use cases. There’s really no reason to expect an entire operating system could be the best solution for all three simultaneously. (And that doesn’t even touch on embedded systems.)

My primary issue with systemd is that it seems to have been conceived under this conceit. (I admit I have other ones, e.g., as a somewhat old school Unix head, binary logging makes me break out in hives, but let’s bracket all of that.) Lennart Poettering himself confirms this in on his systemd myth-dispeller page:

“With systemd we try to cover pretty much the same range [of use cases] as Linux itself does.”

“Of course, in many server setups boot-up is indeed irrelevant, but systemd is supposed to cover the whole range.”

(Emphasis mine.) Let’s note here and now that this idea—that systemd should cover every use case of Linux—is not a technical idea, but a philosophical one.

What web applications need

Take the following use case: A server, likely virtualized and cloud-provisioned, devoted to one primary service, such as http, databases, etc. Because virtual servers are cheap, we follow the best practice of one and only one service per host. (This is actually a compliance issue if you handle credit card transactions.) And, because they are cheap, we have at least two instances running at all times for failover.

In this case:

By my lights, systemd doesn’t really offer a whole lot to people in this use case. That wouldn’t matter if it were highly idiosyncratic, but this is pretty much the profile of a bog-standard web application. And given the “one service per host” principle, many of the issues systemd seeks to address are probably evidence of a screwy production environment: startup dependencies suggest multiple services on the same host, and boot times becoming that critical suggest insufficient horizontal scaling. (Again, I’m sure people can come up with exceptions to this, but I’d be surprised if this wasn’t true for the majority of web/CRUD/etc. applications that follow this pattern of deployment.)

There are as many other use cases for Linux as you can imagine: embedded systems, desktop computing, tablets, phones, and even complicated data center applications where you really do need to run many interdependent services, and booting fast matters. Systemd may be great for those cases:

But I don’t need to boot a car, and I don’t need to boot a Gnome desktop. I need to mount the filesystems, start networking and logging, and call the web server. And the difference between 20 seconds and 45 seconds of boot time is negligible. I do want service monitoring, but there are a lot of ways to do that which don’t incur a redesign of one of the central components of the operating system. The cost in terms of increased complexity, attack surface and expertise gap just doesn’t justify it.

Universal solutions are the problem

A car is not a desktop, which is not a web server, which is not a phone, and so on. I have to ask: Why do Lennart Poettering, et al., even want to cover all Linux use cases in the first place? So far as I know, most of systemd’s PR has been devoted to showing it can cover all Linux use cases, not that doing so is a good idea in the first place. And without an explanation of that, either (a) the belief that a single solution is good looks like mystical, a priori dogma, or (b) we are knowingly pushing suboptimal solutions on one use case or another. (It would be equally invalid of me to deny the benefits of systemd just because I don’t need them.) The entire problem vanishes if you simply deny the premise that we should have a single init (or operating system) for all of our various devices.

I suspect one reason I feel this way is because I don’t operate or maintain a Linux distribution. Ultimately, it matters very little to me whether the Linux on my desktop is the same distribution as the one on my servers, as long as the test suite runs in both places. Organizations like RedHat, Debian or Ubuntu however have a vested interest in preventing schism. But those benefits accrue to them, as an organization, and not to my application. I’m not going to lose support contracts if RedHat can’t simultaneously run an office computer and a car.

Options?

There are absolutely Linux distributions that do not use systemd. Slackware has always used a Berkeley-style init. Devuan was the product of the afore-linked Debian civil war; it is essentially Debian minus systemd. However, this means breaking ranks with a large portion of the Linux community, whose size and scope is taken to be one of the factors recommending Linux in the first place.

Another option: FreeBSD. My first Unix job was on FreeBSD, (back in ‘97!) so I’ve always had a place in my heart for it, or at least don’t look at it sideways for not being Linux. But past that: I have yet to hear exactly what the problem is with Berkeley init, especially for simple cases like one-service-per-server web hosts. (simple != weak) Second, it has a lot to offer for web applications: pf for firewalling, real ZFS in the kernel, kqueue for events, and other strengths. And unlike Linux:

If your goal is a simple, predictable Unix system to run web applications, it at least merits consideration.

Finally: Put up with it, which is what I’ve done. While I’m not thrilled, there are frankly more important things in the world. If I were starting another major project from scratch today, I’d seriously look at FreeBSD for such servers, but given how much stability we’ve had overall with Debian, I’m not blocking out a week to move just on general principle.