Ask HN: How to build an AI to find info I care about online?

2 pointshikerell1y ago8 comments

I want to build an AI agent that lets me define what information interests me, such as "the hottest design trend discussions on Reddit" or "who's looking for a remote designer collaboration." This AI agent would automatically browse various forums and news sites daily, dig up information, and only push the most relevant stuff to me. I'd like to ask everyone:

1. Besides web scraping and large language models, do I need any other technologies?

2. Besides writing a dedicated scraper for each website, is there a simpler technology available?

3. I plan to have every piece of information analyzed by a large language model to see if it meets my criteria, but it feels like it consumes a lot of tokens. Is there a more cost-effective way?

4. Is my technical approach completely wrong? Is there an easier way?

8 comments

boivr1y ago

Its a complex problem cause -

1. Info is exploding while the time you have available per day to consume info is finite. This time can be divided between Info that has short term value, long term value, no value (ie entertainment).

2. Since info is exploding, a gap grows every day between supply and demand. If you run an automated process every day it can easily overwhelm, what your actual need in the 3 categories cuz it will easily generate more content than you have time.

3. Complicating everything further is People's Needs constantly change with time. In all 3 categories.

hikerellOP1y ago

You're a philosopher!

Perhaps it would be simpler to automate the filtering of information in just one specific direction, as I truly can't figure out how to efficiently handle all the streams of information I'm interested in.

Dahmonium11y ago

a complex problem yes, unsolvable no.

Exactly the fact that info is exploding justifies a search for a solution. A solution as suggested by TS. And maybe this could be solved with the help of ai.

Dahmonium11y ago

1. automation and presentation 2. How about RSS ? 3. Local llm? 4. You say criteria, but as in keywords? Like "topics: design, designer, design trends" etc? I would suggest it would be profile based like "I'm a 25 year old designer from NY interested in..." etc.

hikerellOP1y ago

Indeed, I am using RSS, but there's too much new information every day. I feel that I should first automate the filtering of my RSS feeds.

cranberryturkey1y ago

just use perplexity

hikerellOP1y ago

Perplexity is great for searching recent information, but it doesn't allow me to limit the results to just the information from yesterday to today. I want an AI to automatically search for, filter, and recommend information of interest to me on a daily basis, and it seems that Perplexity cannot be integrated into my daily automated workflow.

hikerellOP1y ago

I thought that Perplexity couldn't meet my needs, but I just had the idea to write a script to use Perplexity to search for me daily and then send me the results. I need to try it out and see how effective it is.

j / k navigate · click thread line to collapse

Ask HN: How to build an AI to find info I care about online?

2 pointshikerell1y ago8 comments

1. Besides web scraping and large language models, do I need any other technologies?

2. Besides writing a dedicated scraper for each website, is there a simpler technology available?

3. I plan to have every piece of information analyzed by a large language model to see if it meets my criteria, but it feels like it consumes a lot of tokens. Is there a more cost-effective way?

4. Is my technical approach completely wrong? Is there an easier way?

8 comments

boivr1y ago

Its a complex problem cause -

3. Complicating everything further is People's Needs constantly change with time. In all 3 categories.

hikerellOP1y ago

You're a philosopher!

Dahmonium11y ago

a complex problem yes, unsolvable no.

Exactly the fact that info is exploding justifies a search for a solution. A solution as suggested by TS. And maybe this could be solved with the help of ai.

Dahmonium11y ago

hikerellOP1y ago

Indeed, I am using RSS, but there's too much new information every day. I feel that I should first automate the filtering of my RSS feeds.

cranberryturkey1y ago

just use perplexity

hikerellOP1y ago

j / k navigate · click thread line to collapse