The idea is good, MS did same in C# since the first version in 2002. Allows to do pointer arithmetic when you really need to, for performance or native interop.
What I don’t like about Rust is they use unsafe to workaround language limitations.
In C# I have only used unsafe code couple times over the course of 20 years. The standard library doesn’t use unsafe code either, at least in modern .NET all these collections are 100% safe code. The main, safe subset of the language is good enough for the majority of practical applications.
In Rust there’s tons of unsafe code in standard library, substantial amount of unsafe code in third-party libraries, and if your software is complex enough, your own code will likely contain usafe as well. This causes safety issues: https://medium.com/@shnatsel/how-rusts-standard-library-was-...