But they can be undone with pencil and paper too. The footgun of automation here can only be undone with either a really good patch (git commit -m 'finally finally works for real this time') or a lot of pencil and paper work that's slower than the processes that caused the problem.

I'll note that this isn't the first time that people have said "well its the algorithm" when they were responsible. The example that springs to mind is bail risk assessments. You're very correct in that there are people making real decisions that are very cruel here. The machines give them something to hide behind.