Say you walk into a store and are followed or stopped quite often for a receipt check. If you're black in certain parts of the world, this is quite common, but much more rare if you're not black.
The privilege is a result of a system. You cannot say x people have privilege, and likewise people get offended when people yell at them for having privilege saying, "It's not something I choose." Is one person from one ethnic group responsible for all the ills cast upon others? Are we responsible for the debts of our fathers?
Privilege is a result of a system, and it is a system that grants privilege and socialite (and often subconscious) level, and that system of beliefs is what needs to be changed. So fighting against privilege involves changing the narrative of the system we live in. It takes time and it takes diligence and it takes careful and critical thought. It's not as easy as throwing a word or blame around.