> Is this where the translation from GR -> QM breaks down?
Sort of, to get a graviton you introduce perturbations on a background metric. (Basically small wiggles of spacetime around an 'average.') You don't do anything like that when you solve the Einstein equations. Consequently, the background spacetime ( that is, the black hole) is not really made of gravitons. (At least in some sense.)