Edit: I'll elaborate here. I went all sciencey and wrote a script to test all combinations of a range of encoder settings that included ranges of comp, max delay, frame size, bitrate etc. Multiplied out, that was a lot of different combinations. I ran the script (which took ages because I had to record the startup latency for each one and capture audio on the remote end, logging ffmpeg output etc etc). Sometimes a particular setting was so bad across 5 tests (with different values for other settings) that I'd scrap that whole batch to save time. The ranges of configurations that were promising were used "in real life" to play music in our living room. I narrowed them down to a few that were "usable", but I asked myself "if I bought this thing as a product, would I send it back?". The answer was "yes".

FWIW, the sender was a Macbook Pro 2015 and the receiver was a Raspberry pi 4B

TL;DR At or below 5ms it had reliability issues (drop-outs) that would occur randomly but frequently enough to ruin music listening experience (like, even a 1s drop per song is completely unacceptable). But it also seemed to stretch audio because the delay would increase over time. Even when I got a high quality (zero drop-out steam) with acceptable latency, the latency would slowly increase over time. So about 1 hour into listening, of I paused the source to answer a call, the decoder end would continue playing for 30+ seconds.

I realised that over a LAN, streaming raw audio technically works, with almost zero latency, so I knew that it was the encode/decode that introduced the latency. That's when I switched over to flac, and noticed an immediate improvement in initial latency, but also stopped having the increasing delay over time.