My bank has a separate passphrase that I have to use on the phone and I call them rarely enough that remembering it is always a challenge. Asking for my mother's maiden name can hardly be considered secret anymore, and remembering the answers to other security questions is a pain: what did I claim was my favourite movie a year ago?
If I've called them I don't really have a problem reading my password to them. If I don't trust the call center staff I can always change it afterwards.