Writing systemd units that stop gracefully before shutdown (opens in new tab)

I tend to push the metaphor of software being meant to be read and only incidentally to be run by a computer as far as I can. We are telling a story to future us or our successors. Stories have rules and it's jarring when you violate them.

There's an idea in software that's a bit like the corollary of Chekhov's gun. Chekhov's gun is about not presaging jarring story elements that will never come to pass. But it's nearly as jarring to leave important story arcs as complete surprises until the end. Producing the gun moments before the curtain goes down would be quite a WTF. We didn't know that was a possibility. That's a niche that some people occupy, but it is a niche.

Introducing things early fights with Locality of Reference, but when we're talking about things of deep, dramatic importance (like an attempted murder, or a reaper process) it's important to introduce that "character" early in the story so that people know that it exists. Failing to do so is a form of deus ex machina and we only appreciate that in very small doses.

So framed that way, I don't see a problem with having to start a killswitch while you're spinning everything up. It's there, people can see it, and know to ask questions about it.

I have the following unit file saved for that purpose:

    [Unit]
    # https://stackoverflow.com/questions/36729207/trigger-event-on-aws-ec2-instance-stop-terminate
    Description=unlink agent from remote server
    Before=shutdown.target
    [Service]
    Type=oneshot
    EnvironmentFile=-/etc/environment
    KillMode=none
    ExecStart=/bin/true
    ExecStop=/opt/service-name/shutdown-unlink
    RemainAfterExit=yes
    User=root
    [Install]
    WantedBy=multi-user.target

If I recall correctly, the KillMode=none is important as it causes the shutdown-unlink binary to escape systemd process supervision. Without it, you may deal with systemd immediately halting your shutdown unit (and killing the process) when it hits the shutdown target.

That's what this post builds to solving at the end and in the next post - having a unit deletes the instance from a cluster before shutdown

Sadly, I think this would be child's play with SysV init.

This is a weird take; most systems in data centers don’t have people walking from rack to rack yanking power cords, and most consumer systems don’t even have a power cord to yank.

If I'm turning on a computer, it's because I want to use it right now. If I'm turning off a computer, it's because I don't need to be using it right now. I guess you do need to make sure shutdown is fast enough that a laptop won't start cooking itself if someone tells it to shut down and then immediately sticks it in a bag, but it seems great more useful in general to optimize for startup time. Providing the happy path of a clean shutdown is useful for that, even if you do still occasionally need to handle power failure recovery.

Hope for the best, plan for the worst, right?

Here's a contrived analogy: modern airplanes are designed to stay in the air even if an engine burns out, but we would still rather fly with both engines at full power whenever possible.

Just because you need to be able to handle an employee being hit by a bus doesn't mean employees should ghost their companies, or that companies shouldn't have systems for when someone gives their two weeks notice.

The article gives the examples that "A load balancer might stop accepting new connections and disable its readiness endpoint. A database might flush to disk. An agent might inform a cluster it’s leaving the group." All of these seem like they're worth doing, and improve expected case shutdown behavior, though you should also write and test the abrupt shutdown case.

100%

https://en.wikipedia.org/wiki/Crash-only_software

The twenty years of laptops I've had wouldn't even flinch at a power cord disconnect.

I'm with you.

It's easier said than done, of course, but crash-only software is a worthwhile goal IMO.

That's fair, that example would be better with a modest timeout (e.g. if podman had a bug that caused it to hang), without taking away from the main points. Updated.

Someone asked about the opposite on Twitter: https://twitter.com/DannoHung/status/1585350836074446869

> It shouldn't be this hard to stop a service gracefully.

It's not.

> you just need to drop a script into /etc/init.d symlink it from the appropriate rc directories.

You just need to drop a unit file into /etc/systemd/system/ and symlink it from the appropriate /etc/systemd/system/${target}.wants/ directories.

Don't tell me that "shutdown.target.wants" and "reboot.target.wants" are harder than "rc0.d" and "rc6.d".

A lot of the article is about ordering of dependencies (don't stop a dependency until after the dependent has stopped). Don't tell me that adding `Before=` and `After=` lines in the unit file is harder than having to remember all of the dependencies and manually figure out the correct "NN" for it all to work correctly.

A lot of the article is about either having your daemon handle SIGTERM, or coming up with the appropriate `ExecStop=` command. The same command you'd be writing in your rc script (the "handle SIGTERM" stuff being for if your rc script simply says `kill $PID`).

That is: The complex parts of the article are things that were complex with sysvinit too.

Kids of today, etc.

AT power supplies didn't have any mechanism for the system to tell the power supply it wasn't needed any more. So when you shut down the computer, it would wind up at a screen with a message approximating "it is now safe to switch off your computer", at which point the system would halt.

ATX power supplies added the ability for the OS to trigger an actual power off. But that's a different end-state to halting, and if you halt the system then it stays on. You may wonder why anyone would want to halt when power off is an option, and to be honest I'm not entirely sure -- possibly because you have a hardware watchdog which will trigger a reboot of a halted machine but not of a powered off machine?

A "halt -fp" just unmounts file systems and immediately shuts down.

I find that CentOS systems that I've used for a while seem to hang on shutdowns; halt -fp is a way to get them down quickly. It is important to terminate any sensitive processes beforehand.

Hmm, sorry you're not seeing it. Its just a CNAME to Cloudflare Pages, nothing fancy

  dig www.psdn.io @1.1.1.1
  
  ; <<>> DiG 9.16.33-RH <<>> www.psdn.io @1.1.1.1
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37362
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0,   ADDITIONAL: 1
  ;; QUESTION SECTION:
  ;www.psdn.io.                   IN      A
  
  ;; ANSWER SECTION:
  www.psdn.io.            126     IN      CNAME   poseidon-www.pages.dev.
  poseidon-www.pages.dev. 126     IN      A       172.66.45.44
  poseidon-www.pages.dev. 126     IN      A       172.66.46.212