When talking about tiny 8-bit floats, it does waste a lot: if your exponent is only 3 bits, you've "wasted" 1/8 of all 256 possible values, which is a lot. With normal-sized floats, it's much less of an issue: 1/256 of the billions of possible 32-bit values, and 1/2048 of all possible 64-bit values.

(Also, the real "waste" is only on the multiple NaN values, since the zeros always "waste" only a single value for the "negative zero", and the infinities always "waste" only two values; AFAIK, both negative zero and the infinities are necessary for stability of some calculations.)