The problems are 1) side channels. Caching those would mean that a malicious script could use timing to find out if the user accessed specific other websites. Also it enables a channel for cross-domain communication. 2) malicious user tracking would not work as well anymore, which would be good but Google probably will not support that. Currently they get all those nice http log entries from sites that only include fonts etc.
For sensitive files with known names, you should be able to set that the cache is only valid for requests from the same host website (as the server might decide to send another response when the host is different). That would fix the sidechannel.
