the interesting aspect that spectre attacks bring to light, is that you can cause memory accesses which are explicitly forbidden because the code performs e.g. bounds checks.

speculative execution "bypasses" the checks, because that's the whole point of speculative execution: speed up the execution under the assumption that one (or the other) case is taken. If the CPU guesses that it's more likely that the bounds check will be successful, it will perform better when the input is well formed, but it let potentially malicious inputs to cause side effects detectable with side-channels leak sensitive information that the bounds check design to avoid in the first place!

As you can see, there is no easy way out here. This is a fundamental feature that makes modern CPU fast. Making sure that code can continue executing only when the results of the check is known, will affect performance of the happy path because the CPU will sit idle waiting instead of doing useful work.