Do you have any more evidence as to why these categorically don't work?
They don't. Loud voices parroting George, with nothing to back it up.
Here are another couple good links:
https://www.evp.cloud/post/diving-deeper-insights-from-our-l...
https://www.databricks.com/blog/training-llms-scale-amd-mi25...
Not even from cloud providers which should be telling enough.
Rocm is sensitive to matching kernel version to driver version to userspace version. Staying very much on the kernel version from a official release and using the corresponding driver is drastically more robust than optimistically mixing different components. In particular, rocm is released and tested as one large blob, and running that large blob on a slightly different kernel version can go very badly. Mixing things from GitHub with things from your package manager is also optimistic.
Imagine it as huge ball of code where cross version compatibility of pieces is totally untested.
Even Nvidia GPU's are tricky to sandbox, and it sounds like the AMD cards are really easy for the tenant to break (or at least force a restart of the underlying host).
AWS does have a Gaudi instance which is interesting, but overall I don't see why Azure, AWS & Google would deploy AMD or Intel GPU's at scale vs their own chips.
They need some competitor to Nvidia to help negotiate, but if its going to be a painful software support story suited to only a few enterprise customers, why not do it with your own chip?
