Why would you hand out data that gets tunneled back in?

There are lots of use cases for this exact pattern. An acceleration structure to speed up searching complex geometry. The internal state of a streaming parser. A lazy cache of an expensive property that has a convenient accessor. An unsafe pointer that the struct provides consistent, threadsafe access patterns for. I've used this pattern for all these things, and there are many more uses for encapsulation. It's not just an OO concern.

I think the bigger issue with "public" and "private" is that is insufficiently granular, being essentially all or nothing. The use of those APIs in various parts of the code base is not self-documenting. Hyrum's Law is undefeated.

C++ has the PassKey idiom that allows you to whitelist what objects are allowed to access each part of the public API at compile-time. This is a significant improvement but a pain to manage for complex whitelists because the language wasn't designed with this in mind. C++26 has added language features specifically to make this idiom scale more naturally.

I'd love to see more explicit ACLs on APIs as a general programming language feature.

Concur. Or, the in-between: Set the structs to be private if you need. I make heavy use of private structs and modules, but rarely private fields.