That's a bit too simple, not all functions work well as fields (differentiability is quite desirable) and you have no way to interact with the fields that way.

I think principal bundles come closest to what physicists call fields. Though I'm holding open the option that really the things in most equations are more like elements of the corresponding Lie-algebra.