> I have debug symbols turned on, which activated dev dependencies

Nope that doesn't happen. It's not compiled into your binary if it's a dev or build dependency. Cargo may have downloaded the crate source according to the lockfile and that's it, it shouldn't build anything unneeded.