I have to imagine its because so much of its training data is readily available programming docs, tutorials, and general Q&A that there is an amazing abundance of online. How many times have you just pasted an error into google and hoped someone else has asked the exact same question on stack overflow?

As Chomsky points out, the AIs understand syntax not semantics. Code is all (or almost all) syntax whereas human language has both syntax and meaning. Meaning is not something ChatGPT understands.

Is it because programming is a more limited and specific language than the ones people speak? There's less room for double-meanings, slang, meaning, or even sentence structure.

It seems the improvement in the models is the size of the training set. Hence the "Large" in large language models. I am assuming that it is better at programming because there a is a ton of training data available on Github, etc. There isn't a similar set of data on solving a variety of physics problems.