Furthermore, "constrained by physics" would imply the speed of light, which is so much faster than nerve conduction...

My flippant guess would be that time perception changing with age would be related to the NMDA receptor. NMDA receptor function decreases with age[0]. My personal anecdotal experience with NMDAR antagonists is that it feels like the `Turbo` mode on old IBM compatibles[1] -- i.e. downclocking. And the slower you process, the faster the world seems to move around you.

[0]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3181613/

[1]: https://en.wikipedia.org/wiki/Turbo_button

From what I remember from development 101, there are also more possible synaptic connections in early childhood, and these get pruned down over time. This would also reduce the amount signals travel.