It does have non-flat spectrum, meaning some values are more probable than others, but that only means you need to whiten it. (A rough analogy might be a 6-sided die labeled with 1,1,1,2,3,4 - yes, number 1 is much more likely to come out. No, this does not make it "not really random", and some trivial math can produce ideal random stream out of it)
The only problem with audio input is that you may end up with non-random value - like all-zero output. But properly implemented whitener should detect this and stop outputting any value at all.