Suppose postdoc X submits a terrible review to a conference. Conference committee member Y requested the review from X, and thus learns that X is a bad reviewer. Later on, Y's colleague Z is considering hiring X for a faculty position and asks Y about X. Y tells the story about the terrible review. X doesn't get the job.
Yes for the paper authors, but what about the committee member who allocates reviewers to papers? Are they converted into "Reviewer 1", etc.? I wouldn't know, I have never been on a conference/journal committee.