I think OP is maybe confusing two different things. The 4096 is, IIRC, not "pixels" but rather "units of precision". They are saying that within a vector tile, the far left is 0, the far right is 4095, with 4096 discrete distance values between.
The 512 is about the portion of the map that is occupied by a vector tile, in "pixels". So if you have a vector tile with 4096 units of precision crammed into 512 pixels, that means you actually get a bit of sub-pixel precision, which is really helpful when you overzoom - you can stretch that same vector tile out over to a bigger and bigger area as you zoom in.
This matters because generally you generate vector tiles to some finite depth (eg, zoom 13) but then you want to be able to display them through to say zoom 17, which means the tile is going to be stretched 4 more zoom depths, so 2^4 (16) times wider/higher.
FYI you might enjoy this talk I gave a few years back: https://www.youtube.com/watch?v=Zuo80hZYA1k
Or if your data is more dynamic, you can also use the ST_AsMVT extension to query PostGIS, which will generate tiles on-the-fly: https://postgis.net/docs/ST_AsMVT.html
(I've been basically a full time freelance Mapbox consultant for the last 7 years. When I'm generating tiles, it's either statically (tippecanoe), through PostGIS, or occasionally using a JavaScript library. Never had to get my hands quite as dirty as you here...)
It is a real shame that Mapbox GL JS doesn't support GeoJSON vector tiles, which makes a lot of this stuff easier. Yes, they're not as compact as PBF, but for ease of development, debuggability etc, they're a great step along the way and flatten the learning curve. (From memory, OpenLayers does support them...)
Also curious whether you looked at https://docs.protomaps.com/pmtiles/
Any idea why Mapbox GL JS doesn't support GeoJSON vector tiles?
Point being: If you're doing GIS stuff on a website, it's worth making sure you have folks who actually can understand these underlying technologies.
[2] https://github.com/geoserver/geoserver/tree/main/src/extensi...
Geoserver does not really do much directly: it's just a kind of web glue for various kinds of backends. So for tile generation, you're better off using Martin (https://maplibre.org/martin/martin-cp.html) or similar.