A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it (opens in new tab)

That's a really good point that had never occurred to me.

Edit: I think that the use of ZFS for your /tmp would solve this. You get Error Corrected memory writing to an check-summed file system.

If you have checksum errors reading data from disk, you have much worse issues than ram corruption. Any program you launch will probably be corrupted.

Although if you do swap on a server (and you should), the swap needs to be on a raid, otherwise your server will crash on a disk error.

Swap on a server is not meant for handling low memory issues, instead there's tons of data on a server that's almost never used, so instead swap that out and make more room for cache.

I can see it now: pro ecc sata and m.2 ssds

First, having no swap means anonymous pages cannot be evicted, named pages must be evicted instead.

Second, the binaries of your processes are mapped in as named pages (because they come from the ELF file).

Named pages are generell not understood as "used" memory because they can be evicted and reclaimed, but if you have a service with a 150MB binary running, those 150MB of seemingly "free" memory are absolutely crucial for performance.

Running out of this 150MB of disk cache will result in the machine using up all I/O capacities to re-fetch the ELF from disk and likely become unresponsive. Having swap does significantly delay this lock-up by allowing anonymous pages to be evicted, so the same memory pressure will cause less stalls.

So until the OOM management on Linux gets fixed, you need swap.

That's not how swap is meant to be used on servers.

The purpose of ECC has nothing to do with being "end-to-end". A typical CPU path to/from DRAM will not be end-to-end either, since caches will use different encodings. This is generally considered fine since each I/O segment has error detection in one form or another, both in the CPU-to-memory case and the memory-to-disk case. ECC in general is not like cryptographic authentication where it protects against any possible alteration; it's probabilistic in nature against the most common failure modes.

what swap partition?

I meant this sort of jokingly. I think have a few linux systems that were never configured with swap partitions or swapfiles.

Fedora did this long before debian. I remember doing wget of an .iso file on /tmp and my entire wayland session being killed by the OOM killer.

I still think it's a terrible idea.

Which is a great reason to have a big swap file now.

> I never saw the case of tmp being filled just by forgotten garbage.

It might have more to do with the type of developers I've worked with, but it happens all the time. Monitoring complains and you go into check, and there it is gigabytes of junk dumped there by shitty software or scripts that can't cleanup after themselves.

The issue is that you don't always knows what's safe to delete, if you're the operations person, and not the developer. Periodically auto-cleaning /tmp is going to do break stuff, and it will be easier to demand that the operations team disable auto-cleanup than getting the issue fixed in the developers next sprint.

I tried out variations on this on my daily driver setups. The design choices here were likely threefold:

Store tmpfs in memory: volatile but limited to free ram or swap, and that writes to disk

Store tmpfs on dedicated volume: Since we're going to write onto disk anyway, make it a lightweight special purpose file system that's commited to disk

On disk tmpfs but cleaned up periodically: additional settings to clean up - how often, what should stay, tie file lifetime to machine reboot? The answers to these questions vary more between applications than between filesystems, therefore it's more flexible to leave clean up to userspace.

In the end my main concern turned out to be that I lost files that I didn't want to lose, either to reboot cleanup, on timer cleanup, etc. I opted to clean up my temp files manually as needed.

If you’ve got swap set up then stale files will get written back to disk so at least you’re not RAM indefinitely just because they’re stored on tmpfs.

It’s still not an ideal solution though.

I agree about the auto cleanup, I discovered it a few days after using /tmp as a ramdisk for Yocto build. Lost a few patches but nothing significant.

Computers like a Raspberry Pi, where the OS is on a sdcard, will hugely benefit.

Technically it'll have some impact on the number of write cycles your disk goes through, and marginally reduce the level of wear.

Most disks have a lot of write cycles available that you'll be fine anyway, but it's a tiny benefit.

Why is reading the data back from swap be slower at all -- much less "DRAMATICALLY" so -- than saving the data to disk and reading it back?

This doesn't really make sense. If /tmp was an on-disk directory the same memory pressure that caused swapping would just evict the file from the page cache instead, again leading to a cache miss and a dramatically slower read.

Most systems probably aren't having problems with insufficient RAM nowaday though, do they? And this will reduce wear on your SSD.

Also, you can easily disable it: https://www.debian.org/releases/trixie/release-notes/issues....

Dm-cache! https://www.kernel.org/doc/Documentation/device-mapper/cache...

Do you mean that you can mark files for which still the underlying filesystem is used? As far as I remember there were experiments with that about 20 years ago, but it was decided that the added complexity wasn't worth it. The implementation that replaced all of that has been very stable (unlike the ones before) and i'm using it heavily, so i think they had a point. Some write-through behavior can be scripted on top of that.

EDIT: So, wikipedia lists overlayfs and aufs as active projects and unionfs predates both. Maybe unionfs v2 is what replaced all that? Maybe I'm hallucinating...

Use `/var/tmp` of you want a disk backed tmp. Not sure why the article omits that.

Who runs around with a 100gb+ /tmp partition?

Our default server images come with a 4.4GB /tmp partition...

The "so you wont need to read files from disk" argument is bullshit because tmpfs data can be evicted to swap. If memory pressure is high you will still be reading from disk.

And high memory pressure is also what makes disk-backed /tmp slow. No improvement at all.

You'd think that, but in ext4 the first write to a new file will hit the disk (the code mentions it is a workaround for something). Btrfs does it correctly.

Same budget

It's a dumb idea that came from the systemd people. They've never explained properly why it's a good idea, but it's the systemd default and for some reason distros defer to that.