Trying to get any sort of resolution out of Amazon is impossible, as they insulate their logistics department from any customer service complaints. All you can do is point out that it happens nearly every delivery.
Plus it's just plain weird, that some guy in joggers turns up and hands you a package. There's no way to associate the courier as a courier, and as such my first thought is usually "Why does this man have my stuff? Did I just let a random guy into my complex?"