undefined | Better HN

0 pointsMuffinFlavored3y ago0 comments

Everything is only fast because of an index, right?

How it catches all file writes to update that index near instantaneously, I have no idea.

0 comments

Everything uses NTFS journaling as far as I know.

I actually solved that by using the native macOS fsevents API which can watch the whole filesystem for changes, and I can instantly update the SQLite db I use for the index through that.

The problem is actually enumerating and searching the whole database, that's the slow part. Right now I have over 5 million rows in it and passing the SELECT results to fzf is the slowest part, taking a few seconds at least on an M1 Max.

I'm not sure how Everything presented the results so fast, with metadata and everything.

cube22223y ago

The fsevents part is interesting, didn't know that!

To your point regarding the slowness - in that case it's not really acting as an index though, is it, if you're scanning the whole list and passing it to fzf?

A proper fuzzy text search index (I think elastic and similar engines have these built-in) should yield orders of magnitude better performance.

alin233y ago

Yes, you're right. I did try Meilisearch for this but I wanted close-to-instant Smith-Waterman fuzzy searching which is why I settled on FZF.

In the end I might have to reimplement that in Swift with an always cached index and native Spotlight like search bar to really make this as easy to use as I want.

It's not practical to reach for the browser or Terminal to do this search.

j / k navigate · click thread line to collapse

0 comments

alin233y ago

Everything uses NTFS journaling as far as I know.

I actually solved that by using the native macOS fsevents API which can watch the whole filesystem for changes, and I can instantly update the SQLite db I use for the index through that.

I'm not sure how Everything presented the results so fast, with metadata and everything.

cube22223y ago

The fsevents part is interesting, didn't know that!

To your point regarding the slowness - in that case it's not really acting as an index though, is it, if you're scanning the whole list and passing it to fzf?

A proper fuzzy text search index (I think elastic and similar engines have these built-in) should yield orders of magnitude better performance.

alin233y ago

Yes, you're right. I did try Meilisearch for this but I wanted close-to-instant Smith-Waterman fuzzy searching which is why I settled on FZF.

In the end I might have to reimplement that in Swift with an always cached index and native Spotlight like search bar to really make this as easy to use as I want.

It's not practical to reach for the browser or Terminal to do this search.

j / k navigate · click thread line to collapse