Isn't the high-precision timer required to detect a cache hit or miss-- as in, the side channel being exploited here is in the timing of a cache hit or miss; there's no data leaked directly into Javascript?
That's not to say that removing SharedArrayBuffer (and high-precision performance timers, which were removed a couple years back to mitigate some other timing-related vulnerabilities) is enough to completely eliminate Spectre; there might be other methods that can time accurately enough to reveal information.
(I might be completely wrong here, but this is my current understanding of the situation, at least.)
