Ask HN: How the AI companies collect data to train models?

1 pointspiotrke2y ago1 comments

From the Internet, obviously, but how? Are they crawling through every website out there based on the IPs or domain names? Or do they piggyback on Google. Or is there all-internet-data store to just download the latest 'Internet data' dump?

1 comments

richardjam732y ago

They use datasets like common crawl.

j / k navigate · click thread line to collapse

Ask HN: How the AI companies collect data to train models?

1 pointspiotrke2y ago1 comments

From the Internet, obviously, but how? Are they crawling through every website out there based on the IPs or domain names? Or do they piggyback on Google. Or is there all-internet-data store to just download the latest 'Internet data' dump?

1 comments

richardjam732y ago

They use datasets like common crawl.

j / k navigate · click thread line to collapse