Blame is maybe a too-strong word in some cases, but if someone gets hired onto my team and is a bit of a dud for whatever reason, and I know someone influential on another team referred them, it starts to feel more like nepotism than simply "using a referral network".
Especially at a larger company where the hiring process can be a little obscured from the actual team. Where not every, or even most engineers on a team get to meet a new engineer before they're hired.