That's a rather broad blanket statement bordering on collective punishment (which BTW is classified as a war crime by the UN). The status of taxpayer makes you a victim of the state, not necessarily someone with influence over its behavior and certainly not an ardent supporter of all its policies. The vast majority of the taxpayers had nothing to do with the situation and might well be opposed to keeping these people in prison if the facts were explained to them.

Perhaps the matter should be put to a general vote—those actually in favor of keeping the inmates in prison can split the cost of any wrongful-imprisonment suit in the event the state loses.