In case it's helpful: we're going to make a new HN API that will be much easier to use. The idea is that adding something like "json/" to any HN URL will return a JSON version of that page.
Curiously enough, we should also be able to eliminate the pagination of comments by then as well. Both changes are waiting on some software work that we expect to improve performance significantly. I don't know when this will all be done, but I hope it's this year.