I can speak from the OpenShift perspective (which is just Kube as far as storage is concerned):
OpenShift Online and Dedicated (both hosted Kube/OpenShift) use AWS and EBS persistent volumes for elasticsearch and Cassandra storage, which is moderately high IOPs although not "all things tuned for performance". Most small non-cloud OpenShift deployments I know of are using NFS for medium / large shared storage - file or data sharing workloads. There are several medium sized deployments on OpenStack using Ceph under Cinder, and their experience is roughly comparable with AWS EBS and GCE disks.
Basically, if you need to fine tune many details of the storage medium, are carefully planning for IOPs and latency, Kube makes it slightly harder to plan that because it's abstracting the mounting / placement decisions. It's definitely possible, but if you're not dealing with tens of apps or more it might be overkill.
OpenShift Online Dev Preview (the free 30-day trial env) is Kube 1.2+ and uses the upcoming dynamic provisioning feature (which creates PV on demand) and is used for many thousands of small ~1GB volumes. Remember though the more volumes you mount to any node the less network bandwidth you have available to the EBS backplane, so Kube doesn't prevent you from having to understand your storage infra in detail.
Also, be very careful using NFS with replication controllers - the guarantee on RCs is there is at least N replicas, not at most N replicas, so you can and will have two+ pods running talking to NFS if you have an RC of scale 1.
Edit: typos
