Well, unless I'm silly enough to try and bruteforce that PIN as well, after 3 failures I'd be left with an expensive piece of plastic... Fortunately I'm rarely that drunk.
EDIT: Actually as the sibling comment points out you can still reset the token even if you mess up the admin PIN. So at least you won't "brick" your token completely.