I run containers in Amazon. Not using their service, because their service is silly, but on Mesos.

I am not locked into a 1:1 tenancy between applications and instances (though I could have it if I wanted). Multitenancy is trivial. I have the ability to spin up new instances of my applications to combat spike loads or instance failures in single-digit seconds rather than in minutes. My developers can run every container within a single boot2docker VM instead of incurring the overhead of running six virtual machines. It's easier to integration-test because my test harness doesn't have to fight with Amazon, but can rather use a portable, cross-service system in Mesos. In addition, I don't have to autoscale with the crude primitive of an instance in production. Multitenancy means that I can scale individual applications within the cluster to its headroom, and only when the entire cluster nears capacity must I autoscale. I can better leverage economies of scale, while allowing me to leverage more vCPU power to applications that need it (running two dozen applications on a c3.8xlarge is very unlikely to bring to bear at any given time less computational performance to a given application than running each application on its own m3.medium).

I could do this without containers and with only Mesos. It would be worse, but I could do it. I could not do this at all with baked AMIs and instances without spending more money, doing more work, and being frustrated by my environment. I know this I've built the same system you describe (I preferred vagrant-aws because when something broke it was easier to debug, but we moved to Packer before I left) and I would never go back to it. It was more fragile and harder to componentize than a containerized architecture with a smart resource allocation layer. The running context of a container should be "anywhere", and it should be "I don't know", and you caring about that is a defect in your mental model.

The one thing I don't believe you, is that you don't have a performance penalty. If you compare a bare metal machine running VMs with a bare metal machine running containers, there will always be a performance penalty for the VMs - they are more heavyweight by definition by running an additional kernel for each VM. Even assuming CPU doesn't get a hit at all, it still incurs memory and diskspace penalties. As an effect, from an IaaS POV, a container can be made available cheaper than a VM, and you can think much less about using them from a performance point of view - does it make sense logically or from a security standpoint? -> use it.

Well.. in Docker's case they are? It's based on lxc, a Linux kernel technology. Of course BSD had it before that, although with a different name.