Actually AEs, and especially SAEs, are more often than not handled by the clinical investigators or trial sponsor, for a variety of reasons.

Insurance policies often don't cover you while you're enrolled in a trial ("you wouldn't have broken your leg if you hadn't been in that cholesterol drug trial so we don't cover that").

The agency really cares a lot, especially about SAEs, and if someone is hit by a trolley while in a cholesterol drug study they will want the company to promptly determine that the drug didn't cause them to stumble, or lose vision, or be confused about where they were. We used to joke at one company, "but being hit by a meteor is not an SAE."