I was able to log in to GitHub using a Yubikey on my Pixel without a special app.
Check whether your Yubikey supports resident keys (aka discoverable credentials) and whether the FIDO key for your account was created with residentKey: true, otherwise it’s a completely different (older) flow under the hood, where the private key actually gets sent to the server, and it wouldn’t surprise me if that’s the underlying cause of what’s happening to you.
