Ask HN: How do you handle repetitive research tasks in your workflow?

One problem of mine that seems to fall in line with this: There is a set of data that some other daily process requires. This data is only available on a well structured and difficult to quickly navigate web page and any of it can change at any time.

I wrote a small Python program using Pandas and Beautiful Soup to parse the web page and insert all the data into a table. Cron schedules it to run hourly such that you get Cron->wget->Python->database. I considered using this approach on a variety of other things such that instead of browsing the raw data I could use SQL to query a database and detect changes, trends, etc.

Of course this opens the door of easy programatic access to that data. Honestly the need for me was never great enough to go any deeper.