I've been reading up on localisation SEO lately - as far as I understand, Google only uses "server IP is located in X country" as an indicator of where a site might be localised for, if it can't get any better information.
For example, if you're using a ccTLD for the domain, or if it's a generic-TLD and you declare a country in Google Webmaster Tools, that will be a much stronger weighting.
Of course, if that's wrong I'd love to know!