Reproducibility is of course a valid concern, but I never had issues with that thanks to Rocker and MRAN.
I do sympathize with long from-source build times; as a Linux user I don't think I had binaries available until I stopped using R, so I've spent days, perhaps weeks, waiting for dplyr to install over the course of my R usage.
> different image for different sets of R packages because they might have different build dependencies
Is this not true of all software in all languages?