I think you're missing the point of why they are there. To continue the above sports reference – you could say the referee has nothing on the line as they can't win or lose the game, and they just get to sit there and enjoy the show, and enjoy the afterparty.

This is obviously incorrect. If something goes wrong, the FAA rep will be there to ensure the correct procedures are followed, to assess whether the SpaceX team met their licence obligations in how they handle mishaps, and to provide oversight in general. On the happy path, sure, they might not be doing a lot more than a "vibe check", but on the unhappy path they may be the difference between SpaceX getting a launch licence for Starship again in months or years.