Compilers are better at eliding array index checks than about eliding numeric overflow checks; an ordinary function that operates on arrays will often be able to reasonably determine both the length of the array and the scope of the indexing variable (Rust's iterators are very good at this), whereas an ordinary math function often has almost no information that usefully limits which values it might be getting called with.

I don't disagree that making more obvious errors into panics would be nice, but the performance implications are often quite unforgiving.