Copilot doesn't retrain on data generated by you in the moment; so I don't see why this is an issue unless you push the files - with the keys - to github.

credentials should never be committed. By the time you're ready to commit code, you should be reading from the environment or a config outside of the codebase, or at least .gitignore'd

Once that key is in your git history, it's in the history. You might be able to edit it, but it's going to be a nightmare to do it.