You might not need supervisord

Our devices ran two process supervisors, and the second one's recovery logic worked by calling the first. systemd started supervisord. supervisord started our four Python programs. When one of them died for good, supervisord's failure hook restarted the compositor by running systemctl.

Nobody chose that arrangement in one go. It accumulated. supervisord worked, and there was never a day when removing it was the most urgent thing. This post is about how we noticed it had become a cost, the inventory that settled the question in an afternoon, and what moving four programs onto systemd units taught us.

#The question that started it

The trigger was small. We wanted the main device service to start at a higher CPU priority than the local API. supervisord has no setting for that, so the fix was to start each program through nice. Then the build broke: the image's BusyBox is built without nice, so the fix also needed coreutils. coreutils was already in the image, but only by accident, through a package group, so trimming that group later would have broken boot without a warning.

That's a lot of machinery for something systemd does with one line, Nice=-5. Soon after, the question was blunt: we have no reason to use supervisor, so what would it take to move away from it?

#The inventory

The test for whether a second supervisor earns its place isn't a benchmark. It's an inventory. List every supervisord feature your configuration actually uses, and find what systemd offers for each. Then look at the column that matters most: what systemd was already doing one layer down.

Every row had an answer except the last one, and that one had a workable substitute: a start-limit window, described below. Two rows decided it. The recovery hook already depended on systemd, and the device service could already restart units over D-Bus. The second supervisor wasn't adding a capability. It was adding a layer that the real supervisor couldn't see into.

#What the extra layer was costing

Some of the cost had been hiding in plain sight.

A program could go quiet for good. supervisord gives up on a program that fails a few times in quick succession (three, by default) and marks it FATAL. That doesn't crash supervisord, so systemd, watching supervisord, sees nothing wrong. In one incident the data partition filled up, and the main device service failed to start because it couldn't write its settings. Once space came back, it very likely stayed down because of this rule. Under systemd it keeps retrying at a backed-off pace, and in a reproduction it recovered on its own.

One ordering for four programs. systemd can only order supervisor.service as a whole. It had to wait for the boot splash, because the renderer and the display draw on the screen. So the main device service, which doesn't draw anything, waited for the splash too.

As separate units, the device service now starts about 15 seconds earlier in the boot: around 10 seconds after the kernel instead of around 25. The renderer and display still wait for the splash, because they should.

Smaller things, adding up. Stopping supervisor.service killed only supervisord, so restarting the renderer could leave an orphaned compositor behind. Log rotation had to signal supervisord through its pid file. Our code carried two vocabularies for process state, systemd's and supervisord's, including an XML-RPC client and a health check written just to translate between them.

#The design

The four programs became instances of one systemd template, grouped under a target:

[Unit]
Description=app %i
After=network.target bootstrap.service telemetry-collector.service
PartOf=app.target
# a program that keeps failing is retried for good, at a backed-off pace
StartLimitIntervalSec=0

[Service]
ExecStart=/usr/bin/app-%i
Restart=always
RestartSec=1
RestartSteps=5
RestartMaxDelaySec=30
TimeoutStopSec=10
StandardOutput=append:/var/log/app-%i.out.log
StandardError=append:/var/log/app-%i.err.log

[Install]
WantedBy=app.target

Each program adds a small drop-in with only what makes it different: Nice= for the device service and the API, splash ordering for the renderer and the display, and environment variables. The list of programs lives in one place in the image recipe, and the unit names and log paths live in one place in the code.

The renderer gets one more thing: a recovery unit. If it fails four times in two minutes, systemd runs a oneshot that restarts the compositor and then the renderer. That's what supervisord's failure hook did, now expressed directly.

#What the migration taught us

Four things surprised us. Each one is a property of systemd worth knowing before you write your first OnFailure=.

#On systemd 255, every automatic restart passes through "failed"

The first test killed the renderer once. Recovery ran, the compositor restarted, and the display went down with it. One crash shouldn't do that.

With the default RestartMode=normal, systemd 255 marks the unit failed before each automatic restart, and that is enough to fire OnFailure=. The fix is RestartMode=direct, which restarts without passing through the failed state. After that, one crash left the compositor alone. Four crashes hit the start limit, recovery ran once, and everything was back after about 16 seconds.

#A oneshot stops at its first failing command

If the compositor itself failed to restart, the recovery unit stopped there and never started the renderer again, so nothing would trigger recovery a second time. Prefixing the compositor line with - tells systemd to carry on regardless:

[Service]
Type=oneshot
ExecStart=systemctl reset-failed app@render.service
# render is started even if the compositor fails to come back, so a later
# failure reaches the start limit and runs this recovery again
ExecStart=-systemctl restart compositor.service
ExecStart=systemctl --no-block start app@render.service

#Unit patterns in journalctl only match units the journal has seen

The telemetry collector reads the programs' journals. Replacing supervisor.service with the pattern app@* looked natural and matched nothing at boot. journalctl expands a pattern against units already present in the journal. Our journal isn't persistent, and the collector starts before the programs, so at that moment there's nothing to match. Listing each unit by name fixed it.

#append: means nobody reopens the log file

systemd opens append: log files once and hands the descriptor to the program. There's no signal to tell a program to reopen its log after rotation, so rotation has to copy and truncate in place. That opens a small window where lines written between the collector's last read and the truncate could be lost. The collector now reads the rotated .1 file as well. A test that wrote lines across a rotation received every one exactly once.

#How to tell if you need a second supervisor

supervisord is a good tool. It's the right one in a container without an init system, on a platform without systemd, or on a development machine where you want one config file that runs everywhere. It becomes overkill when it runs under systemd on a device you build yourself. Some questions that tell you which situation you're in:

1. Is PID 1 already systemd? If not, stop here. You need something.
2. Map every supervisord feature you use to a systemd directive. The rows with no answer are your real requirements. For us that was one row, and a start limit covered it.
3. Does your supervisor's recovery call systemd anyway? If it shells out to systemctl, you already depend on systemd, just with an extra hop.
4. Can systemd see when a program gives up? If a child can die for good without the parent failing, PID 1 is supervising the wrong process.
5. Are you working around missing features? Wrapping commands in nice, signalling through pid files, and one ordering for every program are signs that the layer underneath already does this better.
6. How much of your code exists to talk to the supervisor? A client and a state-translation layer that only serve the middle layer are part of its cost.

For us the answers came out the same way every time, and the migration took a day.

"All problems in computer science can be solved by another level of indirection, except for the problem of too many levels of indirection."

Attributed to David Wheeler