I wouldn't try to insist anything I sketched out is the ~right approach. I was just trying to imagine one path through the possibility space, and then reason a bit about what kinds of information it might be able to convey.

There are almost certainly better paths through, and I suspect the idea isn't broadly usable without different kinds of tests that have different kinds of rules. It's probably not helpful to have to increment your major version just because you use snapshot testing and some dependency update causes a trivial shift in the output.

I also fiddled around a little with an idea I called "earmarks", which are basically just version-bound tests. You could use these to express the idea that, say, test_x shouldn't pass or fail until the version is >= a.b.c.

This would make it easy to deprecate an API today and go ahead and ship a test that requires the API to be present and functioning until the next major release but not after. Or, for example, to make a commitment device that asserts the project will hit some doc/lint/typing goals by some clear point.

Since it's an open-ended mechanism, I imagine something like it is the lower-friction way for a real project to explore applying these concepts without full toolchain support.