undefined | Better HN

0 pointsalew12y ago0 comments

But the model ultimately still has to process the comma, the newline, the "job". Is the main time savings that this can be done in parallel (on a GPU), whereas in typical generation it would be sequential?

0 comments

sebzim45002y ago

Yes. If you look at the biggest models on OpenAI and Anthropic apis, the prompt tokens are significantly cheaper than the response tokens.

j / k navigate · click thread line to collapse

0 pointsalew12y ago0 comments

But the model ultimately still has to process the comma, the newline, the "job". Is the main time savings that this can be done in parallel (on a GPU), whereas in typical generation it would be sequential?

0 comments

sebzim45002y ago

Yes. If you look at the biggest models on OpenAI and Anthropic apis, the prompt tokens are significantly cheaper than the response tokens.

j / k navigate · click thread line to collapse