LLMs also cost what they cost because NVIDIA won't sell you a 5090 upgraded with 80GB RAM for $3000 instead of $2000 (which is overkill in the first place). Yo have to buy a H200 for $40000
If there is demand, someone will sell that eventually - while NVIDIA has a headstart, they "just" fab stuff on TSMC anyway. AMD and to a degree Intel are already starting to sell cards with more VRAM.
