Yup I'm the same. I'm horrified at how my skin feels covered in chalk dust.
Chalk feels like it made sense in a time back when people didn't bathe daily and didn't know about deodorant. When the words "skin care" were decades away from being uttered for the first time. The fact that chalk is white-colored doesn't make it any less dirty or filthy.
It doesn't feel like it belongs in the twenty-first century, not in civilized places like lecture halls or meeting rooms.
(Chalking up before sweaty rock climbing, of course, is a totally different story.)