undefined | Better HN

0 pointsspott1y ago0 comments

They aren’t using the prompt caching on the query side, only on the embedding side… so you cache the document in the context window when ingesting it, but not during retrieval.

0 comments

KTibow1y ago

It seems a little odd to make multiple requests instead of using one request to create all the context for all the chunks.

j / k navigate · click thread line to collapse

0 pointsspott1y ago0 comments

They aren’t using the prompt caching on the query side, only on the embedding side… so you cache the document in the context window when ingesting it, but not during retrieval.

0 comments

KTibow1y ago

It seems a little odd to make multiple requests instead of using one request to create all the context for all the chunks.

j / k navigate · click thread line to collapse