The fruit has the color orange when it's ripe. It's probably one of the most orange things you'll see on nature.
But most people don't even eat it ripe (throwing it away before that point), and the association between the fruit and the color just flies over a lot of people's heads. And yeah, the internals of most of them are yellow.