> A network agnostic DHT crawler and monitor. The crawler connects to DHT bootstrappers and then recursively follows all entries in their k-buckets until all peers have been visited.
Following the Wikipedia link for "DHT" yielded some clues. (Ah. Distributed hash table.) But I've still been looking at this for several minutes now and am basically just puzzled. But the graphs are pretty! Reading the word "amino" a little further down threw me off the scent for a bit. But I gather that is actually a proper noun, and we aren't really talking about proteins here.
Maybe an initial sentence that makes fewer assumptions about the reader's familiarity with the jargon would be helpful.
One issue is that peers go offline and online all the time, so the network is ever changing, if you turn off your client for a week and then come back, your only hope is that at least one of the peers you know is still online, if that's the case then that's fine, if that's not the case, or you're starting the client for the first time, then there's no way for you to connect to the network and query for keys. In bittorrent this is not an issue as most torrents include trackers, the original centralized way of finding peers on the network, but it seems that each project listed on this page has it's own separate DHT network that doesn't connect to the main network (the one used by bittorrent), so for you to connect to these networks for the first time you need to use a bootstrap peer, this is just a normal peer on the network that is known to be always online, usually hosted by owner of the project, and it'll give you a starting point to find other peers in the network.
What this project does in essence, is connect to a bootstrap peer, then use the properties of the routing algorithm to efficiently find out all the peers that are currently online.
I'm talking about of course very late 90s, 00's P2P file sharing, Kademlia, torrents but also later "eventually consistent" databases ( remember those? )
The crypto 20 somethings hype beasts came way later.
The part you quote even provides links to DHT and k-buckets if you want more information. That first paragraph is important real-estate and shouldn't be wasted on someone who most likely has no use of the project.
Every domain has these words that sound alien to newcomers and you can't explain them to everyone all the time.
The detractors of the ecosystem (myself included, to be honest) will be quick to point out that obfuscating the tech as magic as much as possible, as well as creating an inside group lingo, is key to onboarding and retaining people into it. But it's fascinating how that percolates throughout the dev community behind it as well.
I get it, nebulae are cool.
This is just a crawler for DHTs that use IPFS's implementation, or at least smell very similar to it.
a) Many other tools exist for that.
b) Bittorrent DHT modes are simple and interchangeable. They can give you a list of peer addresses associated with certain (torrent) hash — and only if you know the exact hash. Even client versions can't generally be collected (apart from some protocol extensions). The only thing you learn about DHT member is that it exists. On the contrary, this project is for heterogeneous networks in which peers announce various services.
c) Number of Bittorrent DHT nodes is… bigger.
d) To collect interesting data from Bittorrent DHT, one needs to observe as much third party torrent hash requests as possible. To do that, multiple nodes are needed. Moreover, they need to run for a long time, not just because it takes time to make a lot of requests to a lot of nodes, but also because of external preference for long-running nodes. Not sure how important it is, but, anecdotally, a fresh DHT node sees twice as much requests after a week than after a day.
It only supports IPFS and derivitaves thereof.
When I think of a crawler, I think of a non-homogonous network (if that is the right term).
But with the blockchain, isn't it the case that each node has an entire copy of the blockchain, so you don't need to "crawl" it, it works more like a database.
What am I not understanding about this?
