I ALWAYS have my phone though. Quite frankly, I look forward to the day when my phone replaces my access badge/key-card at work.
Of course there is the arguement "but someone could take your phone and access the MDF!" -- well they can always steal my access badge as well.
I have forgotten my badge countless times in my career though I have forgotten my phone all of once.