The user's browser downloads the font off of Google's server, which gives Google their IP address, and the browser also tells Google's server what webpage it's currently visiting, via the HTTP referer.
And given that almost every webpage ships something from Google, Google has an almost complete browsing history for every public IP. There's generally multiple devices behind one public IP, especially for corporate networks or VPNs, so they still have to demultiplex that with further tracking, e.g. Google Analytics, Chrome Sync, Android, but that's rarely a problem either, as even if you're carefully avoiding these, everyone else under your public IP using them would be enough to single you out.