We don't start from the applications though. We hear from news stories when startups are doing well, and then we go back and look at their application.
Given the funnel of startup failure, there are probably more startups that didn't succeed because you passed than startups that succeed despite your passing.
My understanding of the essays I've read is that there wouldn't be much point in it. Ideas evolve or change entirely, and more importantly people improve. If the gut feeling was no then, all you know is that something changed.
Or you missed something that was already there - IMO more likely, considering how little you can learn about someone from the application, and that the founders at least had the potential to become what they are today.
Q: Have there been any startups you've later regretted rejecting from YCombinator? A: Sure, several. But I can't name names because it's not for me to disclose that they applied.
Additionally they also mention about contacting them on rejected applications[2]:
"If you do, we'd appreciate it if you'd send us an email telling us about it; we want to learn from our mistakes"
Here are some that are known to have been rejected by YC and have gone onto raise funding (which is not a perfect metric by any means since funding ≠ success) or have been acquired:
- SendGrid - http://sendgrid.com/ - Raised $27.4M[3]
- CouchOne - http://www.couchbase.com/ - Raised $56M[4]
- AfterTheDeadline - http://afterthedeadline.com/ - Acquired By Automattic
- LightSail Energy - http://lightsailenergy.com/ - Raised $42.8M[5]
- SignPost - http://www.signpost.com/ - Raised $15M[6]
- Storenvy - http://www.storenvy.com - although they were kicked out of YC[7] Raised $6.5M [8]
[1] https://web.archive.org/web/20120814111130/http://askolo.com...
[2] http://ycuniverse.com/yc-applying-interviewees
[3] http://www.crunchbase.com/company/sendgrid
[4] http://www.crunchbase.com/company/couchbase
[5] http://www.crunchbase.com/company/lightsail-energy
[6] http://www.crunchbase.com/company/signpost
[7] http://joncrawford.com/post/20378314843/how-i-got-kicked-out...
[8] http://techcrunch.com/2013/02/12/storenvy-goes-from-getting-...
https://www.quora.com/Y-Combinator/Whats-it-like-to-get-reje...
Edit: with older data included> http://tctechcrunch2011.files.wordpress.com/2013/11/unicorn-...
Presumably, someone on the YC staff monitors the success of companies both in the program and out of the program and uses that data to improve future acceptance decisions.
PG says that's not how it works. I guess you think the way YC works is stupid.
Only a small number of those will have made it big since then, though (by whatever definition). So they'd only have to look at a small amount to see if there were any indicators that they might have overlooked.
A script could probably do the most time consuming bits.