Some options:
You can keep track of which email addresses have been invited to a specific org (or match by domain if specific domains have been allow-listed) and post-validate that the email address you get from social login is allowed. Maybe your invite flow is just the login flow.
Alternatively ou can bind some state in your backend to the state parameter in the oauth flow. The state value gets passed to your oauth callback so you can look up the details for the invite and do validation at that point. This lets you be quite specific about how a particular invite link gets processed.
Both of these are a ltitle bit sucky because they throw errors right at the end of the flow and you have to ask the user to start over again. This hurts because users often aren't clear what account(s) they are logged into in their social provider, how to log out of one, or how to switch between them.
