I have been noticing AI websites adding support for llms.txt standard, which inspired me to read more about it. llms.txt is similar to robots.txt but for LLMs so they can better understand a website with less tokens. I have seen a few directories, but submission is typically through a pull request to a Github repo so I went ahead and created one with automated submission and a rough draft llms.txt generator.
I plan to keep improving it as more websites get added.
Take a look, and let me know what you think!
