Depends how much confidence you have that it wasn't soaked in something nasty that doesn't easily wash out.

I ordered replacement earpads from Amazon for one of my headphones. Ended up developing a rash around my ears. So it's not just about clothing, and, yes, you should be concerned.

Dangerous material being in contact of your skin all day and night long does seem risky