The reason so many people with aphantasia don't realize until adulthood is that we do everything you do. There's no real difference in capability, many things are identical or very similar, but the mental experience is different.

My understanding is: Each sense has an phantastic analogue for phantastics. The hyperphantastic can override their senses with their phantastic analogues. Most people have more-or-less full control(?) of their visual and auditory phantastic analogues. The aphantastic have no/very stunted analogues with limited control, or only a partial selection, but people with a visual analogue and without other senses would probably never realize, and so aphantistics can be assumed to missing at a minimum the visual analogue but very commonly have none.

I didn’t find out it was a thing until I was 38 or 39. And yes, I daydream. But it’s not like watching a movie. I don’t know how to describe it besides my mind wanders and there’s a narrative.

I think it is the case where you just assume that people are embellishing or being metaphorical about those things and you just refuse to consider for a moment that they are actually seeing something. But it does give this thought that why do people like to embellish or be metaphorical so much? I guess the answer is that they are not!