It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically.
I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...
Blog post that explains the rationale behind the library: https://philippeoger.com/pages/can-we-rag-the-whole-web
Just submit your XML sitemap into a python class, and it will do the crawling, chunking, vectorizing and storage in an SQLite file for you. It's using SQLiteVSS integration with Langchain, but thinking of moving away from it, and do an integration with the new sqlite-vec instead.
A relational crawler on a particular subject with nuanced, opaque, seemingly-temporally-unrelated connections that show a particular MIC conduction of acts::
"Follow all the congress members who have been a part of a particular committee, track their signatory/support for particular ACTs that have been passed, and look at their investment history from open data, quiver, etc - and show language in any public speaking talking about conflicts and arms deals occurring whereby their support of the funding for said conflicts are traceable to their ACTs, committee seat, speaking engagements, investment profit and reporting as compared to their stated net worth over each year as compared to the stated gains stated by their filings for investment. Apply this pattern to all congress, and their public-profile orbit of folks, without violating their otherwise private-related actions."
And give it a series of URLs with known content for which these nuances may be gleaned.
Or have a trainer bot that will constantly only consume this context from the open internet over time such that you can just have a graph over time for the data...
PYTHON: Run it all through txtai / your library ? nodes and ask questions of the data in real time?
(And it reminds me of the work of this fine person/it::
I have not used sqlite-vec much because it was only alpha-released for now, but it finally came out a few days ago. I'm looking into integrating it and use it to make sqlite more my go-to RAG database.
1) The returned output from a query seems pretty limited in length and breadth.
2) No apparent way to adjust my prompts to improve/adjust the output e.g. not really 'conversational' (not sure if that is your intent)
Otherwise keep developing and be sure to push update notifications to your new mailing list! ;-)
Brilliant idea, btw, I like it :-)
Soon websites/apps whatever you want to call them will have their own built-in handling for AI.
It's inefficient and rude to be scraping pages for content. Especially for profit.
It is like every website having search engine vs google.
I would guess the hardest thing by far in developing the advertised product would be user management, authentication, payments and wrapping the subscription model's business logic around the core loop. And probably scaling, as running embeddings over hundreds of scraped pages adds up quickly when free tier users start hammering you.
My question when deciding to sell something I've built is, if building the service model is harder than building the actual service, where is the value add?
My take on the natural evolution is that collating and caching documents, websites etc for search (with source attribution ideally) is a problem that will I think ultimately be solved by OS vendors. Why sign up for SaaS and expose all your content to untrustworthy 3rd parties, when it's built right in and handled by your "trusty" OS.
In the meantime, I reckon someone more dedicated than me will (or probably already has) open source something like I built but better, probably as a CLI tool, which will eventually reach maturity and be stolen cough I mean adopted by the top end of town.
Ethically I think nothing's changed for centuries in regards to plagiarism and attribution. It gets easier to copy work and thinking, but it also ultimately gets easier to acknowledge sources. Good folk will do the right thing as they always have done.
Regarding efficiency, I think tools like this have a place in making access to relevant and summarised knowledge during general research more efficient, when doing the broad strokes to find areas of interest to zoom in on, when more traditional approaches take over.
Interesting times anyway. I have to give credit to people that try, but I'm taking a back seat in thinking of ideas to productise in this space, as by the time I've thought it through, something new comes along that instantly makes it obsolete.
This isn’t “niche”, it’s a pretty cool thing OP has built.
How about instead of commenting and trivialising what people have done, you say something positive
Lmfao. God bless HN for keeping this meme going for decades by now.
Nobody cares how you would build it because you haven’t. At least not in any form that we can see.
A key question that the docs should answer (and perhaps the "How it works" page too): chunking. You generate an embedding for the entire page? Or do you generate embeddings for sections? And what's the size limit per page? Some of our docs pages have thousands of words per page. I'm doubtful you can ingest all that, let alone whether the embedding would be that useful in practice.
But: I feel the more of these services come to being, the more likely it is that every website starts putting up gates to keep the bots away.
Sort of like a weird GenAI take on Cixin Liu's Dark Forest hypothesis (https://en.wikipedia.org/wiki/Dark_forest_hypothesis).
(Edited to add a reference.)
"We've been sitting in our tree chirping like foolish birds for over a century now, wondering why no other birds answered. The galactic skies are full of hawks, that's why." (The Forge of God, Legend edition, 1989, pg 315).
Yeah, same concept and even the same imagery.
Source: https://warwick.ac.uk/fac/sci/physics/research/astro/people/...
That's why we need microtransactions, because I'd rather be able to have both nice AI services and useful data repositories that they pull from, than have to choose just one. (and that one would be AI services, because you can't stop all the scrapers, so data sources will just keep tightening their restrictions)
https://github.com/langchain-ai/langchain/blob/master/cookbo...
FWIW, the pricing model of jumping from free to "contact us" is slightly ominous.
> Turn any website into a knowledge base for LLMs
https://github.com/harvard-lil/warc-gpt
https://lil.law.harvard.edu/blog/2024/02/12/warc-gpt-an-open...
https://github.com/MittaAI/SlothAI/blob/main/SlothAI/lib/pro...
https://github.com/MittaAI/mitta-community/tree/main/service...
There's code in there that just reads PDF meta data as well, but you can't always guarantee it's there in a PDF.
In addition, this scraper doesn't even identify itself (I checked). It pretends to be a normal browser, without saying it's a scraper.
You can of course decouple from the big discussion and isolate your content with access restrictions, but the real interesting activity will be outside. Look for example the llama.cpp and other open source AI tools we have gotten recently. So much energy and enthusiasm, so much collaboration. Closed stuff doesn't get that level of energy.
I think IP laws are in for a reckoning, protecting creativity by restricting it is not the best idea in the world. There are better models. Copyright is anachronic, it was invented in the era of the printing press when copying was made easy, LLMs remix they don't simply copy, even the name is unfitting for the new reality. We need to rename it remixright.
The LLM era doesn't give credit or attribution to its sources. It erases exposure. So there's a disincentive to collaborate with it, because it only takes.
> I think IP laws are in for a reckoning, protecting creativity by restricting it is not the best idea in the world.
We've been having this discussion for over 20 years since the Napster era, or even the era of elaborate anti piracy measures for computer games distributed on tapes 40 years ago.
I've reached the conclusion that the stable equilibrium is "small shadow world": enough IP leakage for piracy and preservation, but on a noncommercial scale. We sit with our Plex boxes and our adblockers, knowing that 90% of the world isn't doing that and is paying for it. Too much control is an IP monopoly stranglehold where it costs multiple dollars to set a song as your phone ringtone or briefly heard background music gets your video vaporised off social media. Too _little_ control and eventually there is actually a real economic loss from piracy, and original content does not get made.
AI presents a third threat: unlimited pseudo-creative "slop", which is cheap and adequate to fill people's scrolling time but does not pay humans for its creation and atrophys the creative ecosystem.
This is not an easy problem to solve. In my naive take, authors get to decide how their work is used, not scrapers.
There's still room for an ethical development of such crawlers and technologies, but it needs to be consent-first, with strong ethical and legal standards. The crazy development of such tools has been a massive issue for a number of small online organisations that struggle with poorly implemented or maintained bots (as discussed for OpenStreetMap or Read The Docs).
Because if you save the pages you browse on some site, they're yours (authors don't own your cache).
Perhaps you're arguing that if you wrote a lightweight script/browser (which is just your user agent) to save some website for offline use, that'd be unethical and GDPR violating? Again, I don't think so but maybe I'm missing something. But perhaps this turns on what defines a "user agent".
Perhaps this becomes a "depth of pre-fetch" question. If your browser prefetches linked pages, that's "automated" downloading, akin to the script approach above. Downloading. To your cache. Which you own. (Where I struggle to see an ethical violation)
Genuinely curious where the line is, or what exactly here is triggering ethics, GDPR and practical standards?
In this case, if this tool is used to scrape a website, there are too direct issues: 1/ no immediate way for the website owner to exclude this particular scraper (what is the useragent?) 2/ no way for data subjects (whose data is present on the website) to search whether the scraper learned their personal data in the embeddings. Data being available publicly doesn't mean it can be widely used [at least outside the US, where we have much stricter rules on scraping].
Twitter and Reddit locked down their APIs. Soon enough, you’ll need an account to even access any content
#2. If you are interested in knowledge bases, see #1
If there is no certifications or compliance information then I don't think there is anything to discuss about any enterprise plan.
All of that code is Open Source, and works well for most sites. Some sites block Google IPs, but the Playwright container can run locally, so should be able to work around it with some minimal effort.
You can also do this on AWS now fairly easily. https://medium.com/data-reply-it-datatech/how-to-build-a-cus...
The lablab.ai Discord community is a pretty good place to learn how this product category is evolving.
Also I've checked their docs to see if there is any mention about the user agents or IP ranges they use for scraping, with no luck.