Ask YC: How to web scrape 100s of sites/forms?

1 pointsbdouglas17y ago5 comments

hi...

a potential research app requires the scraping of a few hundred websites/forms and diving into the child links to obtain the linked/parent structure ie. company->dept->title->name.

in this case, this would involve going 4 levels deep, and getting the required information.

so, does anyone know of a method/app/company that can be used to accomplish this. orm am i going to have to figure out how to get a number of cheap guys to write a bunch of python scripts!!

thanks

5 comments

nreece17y ago

...cheap guys to write a bunch of python scripts

You know what will be 'cheap'. Writing it yourself.

qhoxie17y ago

Libraries like mechanize and hpricot are shrinking the curve for scraping tasks. That's not to say it is easy, but it should not take a bunch of people working on it. One good developer with proper experience would be ample in my opinion.

olefoo17y ago

Or get one expensive guy to write you a script that

writes the scripts to scrape the sites by scraping the sites to read the structure to write the scripts to scrape the sites.

Anon8417y ago

Check out the search.wikia.org project. They make their crawler (and crawl data) available. Maybe you can get away with using theirs. That would really be cheap!

gaius17y ago

I am unable to think of an application for this technology other than spamming. Care to provide more details before we shoot ourselves in the face by helping you?

j / k navigate · click thread line to collapse