I used to think so, too, but solved this the following:
Hanging EVERYTHING. Not only shirts and jackets, but also t-shirts, pants, shorts.
This leaves one drawer each for socks and underwear, which both is not horrible from a laundry management perspetive.