I'm sure I've seen "just doing my job" invoked in a pejorative sense, in close parallel to "just following orders"? When one's job is to harm society, society might step in, even if one "didn't know".