Indeed it does, but in order to verify them I see only two options.

One is that you have to choose test cases where the answer is trivially determined. However, with this strategy, it seems you must ultimately rely on the refactoring step to magically convert your implementation to support the general case, so the hard part isn’t really test-driven at all.

The other is that you verify non-trivial results. However, to do so you must inevitably reimplement the relevant mathematics one way or another to support your tests. Of course, if you could do that reliably, then you wouldn’t need the tests in the first place.

This isn’t to say that writing multiple independent implementations in parallel is a bad thing for testing purposes. If you do that and then run a statistical test comparing their outputs for a large sample of inputs, a perfect match will provide some confidence that you did at least implement the same thing each time, so it is likely that all versions correctly implement the spec. (Whether the spec itself is correct is another question, but then we’re getting into verification vs. validation, a different issue.) However, again for a simple example like rendering a fractal, you could do that by reimplementing the entire algorithm as a whole, without any of the overheads that TDD might impose.

I don't have any problem imagining myself developing a Mandelbrot set program using TDD.

I’m genuinely curious about how you’d see that going.