Also victims are nearly always emotionally involved, and emotional-based decisions aren't generally good. Punishments would be much more severe if it were up to the victims.
If victims determined the sentences, I expect people would spend a lot more time in prison, way more than a non-emotionally involved and wronged person would think fair.
IMHO letting victims set the sentence would be the worst way to do it.