I do remember when PulseAudio seemed to be the source of many problems, but this was over a decade ago.

One of the big problems is that the Linux audio stack is built out of components that rely on assumptions about the next lower level of the stack that are often untrue. You might assume that your audio hardware has a volume control, assume it can perform mixing, assume it doesn't change configuration, assume there's only one user, assume (or want to assume) it has a specific sample rate, etc. PulseAudio papers over this stuff and deals with permissions, so your applications can actually work.

If you have sufficiently boring PC hardware and sufficiently boring use cases, the audio will just kind of work anyway, without PulseAudio.

Linux audio has more than its fair share of problems in the stack, but in my experience, PulseAudio is more of a solver of problems and less of a source of problems.