undefined | Better HN

0 pointsHarHarVeryFunny1y ago0 comments

I think you're confusing training and inference. During training there are things like initialization, data shuffling and dropout that depend on random numbers. At inference time these don't apply.

0 comments

jampekka1y ago

Decoding (sampling) uses (pseudo) random numbers. Otherwise same prompt would always give the same response.

Computing entropy generally does not.

See e.g. https://huggingface.co/blog/how-to-generate

HarHarVeryFunnyOP1y ago

Sure - but that's not the output of the model itself, that's the process of (typically) randomly sampling from the output of the model.

throwaway3141551y ago

Right, sampling from a model, also known as *inference* (for LLM's).

The inference here is perhaps less pure than what you refer to but you're talking to human beings; there's no need for heavy pedantry.

j / k navigate · click thread line to collapse