Show HN: robots.txt as a service, check web crawl rules through an API (opens in new tab)

The project have multiple subprojects, and one of these is the parser. Any developer can compile or extend it without more effort and create a library. Just know to code in Kotlin / Java.

The aim of this project is only check if a given web resource can be crawled by a user-agent, but using a API

I implemented this service thinking in make a network request for each new URL that needs to be crawled. Internally the service caches all requests by the base domain and user agent. The responses are very fast if these domain was previously checked.

For example if you want to check the url https://example.com/test/user/1 with a user agent MyUserAgentBot, the first request can be slow (~730ms) but subsequent requests with different paths but same base url, port and protocol, will use the cached version (just ~190ms). Note that this version is in alpha and many things can be optimized. The balance between managing these files in different projects or the time between network requests must be sought.

Anyway, any person can compile the parser module and create a library to check robots.txt rules by itself ;-)

PS: thanks for the feedback

Yes, in the practice people sometimes don't want to be polite with webmasters, and choose not obey robots.txt rules. Thanks for the suggestion!