undefined | Better HN

0 pointscatlifeonmars2mo ago0 comments

Doesn’t that apply to the OP as well?

0 comments

Yes, I'm not going to fill my precious context with documentation for a programming language

This seems like a research dead end to me, the fundamentals are not there

It seems kind of silly that you can’t teach an LLM new tricks though, doesn’t it? This doesn’t sound like an intrinsic limitation and more an artifact of how we produce model weights today.

verdverm2mo ago

getting tricks embedded into the weights is expensive, it doesn't happen in a single pass

they's why we teach them new tricks on the fly (in-context learning) with instruction files

1 more reply

j / k navigate · click thread line to collapse

0 comments

verdverm2mo ago

Yes, I'm not going to fill my precious context with documentation for a programming language

This seems like a research dead end to me, the fundamentals are not there

catlifeonmarsOP2mo ago

It seems kind of silly that you can’t teach an LLM new tricks though, doesn’t it? This doesn’t sound like an intrinsic limitation and more an artifact of how we produce model weights today.

verdverm2mo ago

getting tricks embedded into the weights is expensive, it doesn't happen in a single pass

they's why we teach them new tricks on the fly (in-context learning) with instruction files

1 more reply

j / k navigate · click thread line to collapse