They definitely aren't vacuous memorizers. GP's description of them as "bullshit generators" is correct. They generate plausible-sounding text that (when it includes facts) is often counterfactual.