undefined | Better HN

0 pointsrolisz2y ago0 comments

That's not quite tfidf though. I agree you can get better results than that with Ada embeddings, but I would argue you can get even better results with embeddings from smaller chunks.

0 comments

simonw2y ago

I guess technically it's bm25, since it's using the rank mechanism in SQLite FTS5: https://www.sqlite.org/fts5.html#sorting_by_auxiliary_functi...

j / k navigate · click thread line to collapse

0 pointsrolisz2y ago0 comments

That's not quite tfidf though. I agree you can get better results than that with Ada embeddings, but I would argue you can get even better results with embeddings from smaller chunks.

0 comments

simonw2y ago

I guess technically it's bm25, since it's using the rank mechanism in SQLite FTS5: https://www.sqlite.org/fts5.html#sorting_by_auxiliary_functi...

j / k navigate · click thread line to collapse