Well, that's a very evasive argument in my opinion. It's absolutely true that objects are neutral, but you can't make a blanket argument with topics like these. That argument has been made lots of times by many (including me).
But it eventually breaks down. You can't give a child a gun and when the kid shoots someone say it was the kid's fault. Whose fault is it? I'm guessing you're going to say the adult responsible for putting the gun within the reach of a child. You're still taking away an object from the kid. In this case it was a kid who didn't know better.
Now this isn't a narrow argument. This becomes interesting when you get to powerful things, like say nuclear weapons. They aren't inherently evil too. But if you look around, UN is trying to ban them[1]. Shouldn't UN ban them?
What I'm trying to get at is that you can't always but the blame on people. Just like you can't blame a child for not knowing better, you can't blame a person for knowing better (although people do). Sometimes you just have to take the gun (or nuclear weapons or encryption) away.