In most cases the cost of fabric itself is a pretty minor part of the garment price- you're paying for someone to design the clothes, assemble them, ship them, and operate a store that sells them, and those costs are pretty much the same for small and large sizes. Adjusting the price based on the amount of fabric used would probably end up being a dollar or so for the things most people wear on a daily basis.
Unusually large or small sizes can end up more expensive (and/or only manufactured in limited quantities) because they're not commonly bought and they take up space on the shop floor and in inventory which could be used for things with higher turnover. (Edit: Also at the extreme ends of sizing simply enlarging or shrinking the pattern won't work well, you have to redraft it so it sits correctly on a petite or plus-sized frame).
