I haven't had a problem. In my experience, I can actually get content up more quickly than a traditional CMS-type system.
Gatsby does some intelligent bundling so each page is pretty much self-bundled.
* Root HTML file
* Core, common JS (cached after first load)
* Page specific JS (also cached on subsequent loads)
For me, this is 257 kB uncompressed - with most of that being the Root HTML file (I realized I don't have minification or Gzip on - which takes that down substantially). Not a tiny payload, but the content is all static so the response comes back extremely quickly.
Even at slow 3G speeds, I barely get a flash. The actual page loads just about as quickly as HN.
With modern internet, the load time is nearly instant AND is literally instant on subsequent page loads (since most of the payload is locally cached).