Yeah, that was it! It was so confusing initially trying to figure out what should go in the EC2 instances' local disk storage, what should go into EBS, and what should go into S3. It all seems so obvious now of course...
It was also confusing to the users. I remember seeing a post every other week on HN about a service going down or losing data because they accidentally used the ephemeral devices for their databases. Haven't seen one of those in ages.