You can't use smart pointers exclusively, because there is no "smart observational pointer", and there cannot be one, because that would be a reference type checked by a borrow checker, which is exactly what rust uses.
You can emulate it with shared pointers, but then you're left with basically a reference counted language. Why not using swift at this point? Also this creates a semantic impedance mismatch that tends to manifest with cycles.
Source: walked a codebase that would (ab)use shared pointers