It's not the pointers themselves so much as what they're typically used for. How would you do dynamic sizing? Imagine sending just a struct of integer arrays this way, you'd have to either know their sizes ahead of time or just be ok with sending a lot of empty bits up to some max size. And recursive structures would be impossible.

You could get around this with a ton of effort around serdes, but it'd amount to reinventing ASN1 or Protobuf.