(I feel like i'm all over this thread...)
I actually tried this for a bit. I found that it really increased the complexity of the build system for not that much of a gain.
Instead of having a "dev" build and a "prod" build, we needed a "chromeDev", a "otherBrowsersDev", and a "prod" build.
So now you have 3 separate environments for your babel configs which you need to keep in-sync, and you run the risk of the semantics being slightly different natively vs the transpiled version (Which IMO isn't that bad when you go from transpiled->native as the transpiled is normally a subset of the "native" functionality, so you are less likely to hit issues)