Linux file write patterns: So you want to write to a file fast (opens in new tab)

It's 2014; why was the author using ext3 instead of ext4? Ext4 does have fallocate support.

Also, if you use fallocate(2) instead of posix_fallocate(3), you don't have to worry about glibc trying to emulate fallocate() for those file systems which don't use it.

Finally, it's a little surprising the author didn't try using O_DIRECT writes.

The code in the second example is wrong. If a write partially succeeds, instead of writing the remaining part it writes again from the beginning of the buffer. The resulting file will be incorrect. That doesn't normally happen on disk writes, but it does when writing to a pipe.

It's a little bit reassuring that there weren't any clear winners and losers. In a perfect world, the OS and hardware would figure out what your intent is and carry it out the fastest way possible, right? Ideally, you'd write the code the most convenient way and it would run the most performant way. Maybe the future is now.

A bit surprising (considering he started off talking about coredumps) that he doesn't mention sparse files. Core dumps can be very sparse, and you might save time and definitely will save space by not writing out the all-zeroes parts.

The first hit for "SSD write speeds" says ~500Mb/s (I hope I got the right b). I didn't bother clicking the links that just the blurb under it. He's dumping 128Mb in ~200 ms. I'm not sure there is much room for improvement.

Am I correct in noting that none of his benchmarks actually timed how long it took for the data to be durably committed to disk, to the limit that the OS can report that?

I would never do XFS benchmarks because in my experience if XFS is writing during powerdown, it trashses the FS (maybe this was fixed in the past 6 years, but after it happened 3 times I haven't touched the OS again).

actually since none of its tests ask to sync the data onto the disk, he might just be measuring each method efficiency in creating dirty pages.

of course that depends on the amount of RAM the system has, and how the kernel VM parameters are tuned (sysctl vm.dirty_*)

just add a fdatasync() call and you will take into account the time it takes to flush all dirty pages into the disk.

Assuming all I/O failures are EINTR is really, really odd, as if to say disks never fill up or fail and sockets never disconnect.

He does say:

> in a real program you’d have to do real error handling instead of assertions, of course

But somebody somewhere is reading this and thinking this is a "semantically correct pattern" (as it is introduced) and may just copy-paste it into their program. Especially when contrasting it with a "wrong way" I think it wouldn't hurt to include real error handling. And that means something that doesn't fall into an infinite loop when the disk fills up.

The author has failed to account for command latency. If you write some bytes, there are a bunch of hardware buffering delays in getting bytes to disk including seek and rotational latency.

Async I/O avoids this. You can tell the I/O subsystem what you want to read next even while doing a write. The I/O is posted to the disk in modern systems, and the disk will begin seeking to the read site in parallel with informing the OS that the write has completed. Posting I/O even helps for SSDs to avoid the idle time on the SSD media between write done and read start.

"For simplicity I’ll try things on my laptop computer with Ext3+dmcrypt and an SSD. This is “read a 128MB file and write it out”, repeated for different block sizes, timing each" The whole thing is completely invalid for measuring actual I/O hierarchy efficiencies because of (a) write sizes too small, would be in buffer cache of unknown hotness, (b) dmcrypt introduces a whole layer of indirection and timing variability and (c) on an SSD, almost anything could be happening regarding cache and syncs. Also, mount options, % disk used, small sample sizes, unknown contention effects, etc. This is a good example of how to convince yourself of something and yet be less accurate than a divining rod.

I would assume the encryption is such a big overhead that most optimizations in the upper levels will be useless.

The second code snippet looks wrong,

write(out, buf, (r - w)) should be write(out, buf + w, r - w)

actually despite authors claims the fs stuff is ram cached a lot, hence the differences in the tests. (specially for a single file write)