You could build a giant space pyramid out of anything in principle, according to Isaac Arthur on YouTube[1]. The limiting factor quickly becomes the size of "giant" needed for most materials and the limited size of Earth to have a wide enough base. Maybe being Hydrogen-infused would mean less pressure on the lower levels and able to build it higher.

Really though, doesn't AeroGel work by being mostly nothing? The more gas and less substance it has, at some point you may as well use a cloth balloon and only have the surface area size, no?

[1] In this video on Space Elevators - https://www.youtube.com/watch?v=dc8_AuzeYKE