Could you share some details of fuzz tests that you've found useful? I tend to work with backend systems and am trying to figure out whether they will still be useful in addition to unit and integration tests.
Doing fuzz testing at the end leads to a lot of low priority but high cost bugs being filed (and many low-cost bugs as well).
The utility of it is quite clear for security bugs. It requires low effort to find lots of crashes or errors that might be exploitable. For development in general, it tends to identify small errors or faulty architectural or synchronization decisions very early, while they are still easy to repair.
I wrote my first fuzz test with a go1.18 prerelease just to get a feel for the new framework. It found a crash instantly: https://github.com/jrockway/json-logs/commit/77c87854914d756.... This commit contains the fuzz test and the fix. (3 cases could be nil; I only ever thought about 2 of them. The fuzzer found the 3rd almost immediately.)
For my project at work, we have an (g)RPC server with a ton of methods. The protos have evolved since 2015 while trying to be backwards compatible with older clients. As a result, many request parameters are optional but not expressed as oneof, so it's hard to figure out what a valid message actually is, and different parts of the code do it differently. (As an aside, oneof is totally backwards compatible... but the protos evolved without knowing that.) I wrote a fuzz test that generates random protos and calls every single RPC with them: https://github.com/pachyderm/pachyderm/blob/master/src/serve...
This test is imperfect in many ways; it knows nothing about protos, and just generates random bytes and sends it to every method. Most of the time, our protoc-gen-validate validators reject the message as being nonsense, but that's totally fine. The test found about 50 crashes pretty immediately (almost null messages, basically a non-null message where the first tag is null), and one really interesting one after about 16 hours of runtime on my 32 core Threadripper workstation. Writing this and fixing the code took about a day of work. I just randomly did it one day after writing the call-GRPC-methods-with-introspection code for a generic CLI interface.
I did this concurrent with a professional security review as part of our company's acquisition and... they didn't find any of these crashes. So, I think I added some value.
It's worth mentioning that most people wrap gRPC calls with a defer func() { ... recover() } block so that bad RPCs don't crash the server. We didn't at the time, but I implemented the fuzz tests with such a wrapper. (So the test doesn't crash when it finds a panic, I just return a special response code when there is a panic, and the test is marked as failing. Then the next test can proceed immediately in the presence of --keep-fuzzing or whatever.) Either way, I prefer an error message like "field foo in message BarRequest is required" to "Aborted: invalid memory reference or nil pointer dereference." and now we have that for all the cases the fuzzer noticed. The user is sad either way, but at least they can maybe self-help in the first case.
All in all, I haven't spent a ton of time writing fuzz tests, but each one has caught some bugs, so I'm happy. A good tool to have in your toolkit.
