That is, if you ask GPT a question directly, it will give the most glib, unfiltered answer possible like an average Reddit user. But if you tell it to role-play a persona it will apply elements of that persona, causing it to filter itself. Or if you tell it "describe your reasoning step by step" it will proceed down a path with greater causality. Or if you tell it "here are the rules of a language, give a response to the prompt in this language" it will attempt to produce a response that looks like those rules, using the weighting associated with logical words like "must," "cannot," "if," etc.

Measuring calibration is the problem now: we know some prompts do much better than others, but not how to optimize that in a general sense, to make an LLM always adopt the persona needed for the job.

This is true in trivial ways: ask it to reply only in json.

This is also true in slightly less trivial ways: ask for a “description of X”, a “short description of X”, and a “one-sentence description of X”.

This continues to be true in increasingly more complex ways: Prompt it with a quote, a rating of that quote as 5/10 on complexity, and request it to give you two new quotes with ratings of 2/10 and 8/10 on complexity. Follow it up with a request for two more quotes at -5/10 and 20/10 ratings. Then try the whole process again but with metrics other than “complexity”: information content, eloquence, humour.

In this way, if there is a single correct output, there is a corresponding single best input prompt that most tightly constrains the output space to the smallest space that still contains the correct output.