Ask HN: Is there a standard web framework performance benchmark test?

I would probably do:

1. Return a few hundred bytes of json with no I/O.

2. Return tens of thousands of bytes of json with no I/O.

3. Do the same 2 tests as above but read the data from a DB.

4. Do the same 3 tests as above but render templates instead of sending json.

5. Do everything again with it cached.

6. Repeat all steps with 1, 2, 5, 10, 50, 100, 200, 500, and 1000 concurrent connections while paying close attention to the response times at the 99% level in addition to the reqs/second.

Also don't forget to look at the bytes transferred. There's going to be differences in the http headers between frameworks. If you're going to compare them directly to yours try to adjust the headers so it matches.

I would would be keen to run the test against Contentigo, but not very keen to do this for all the other frameworks! I would hope that if a good set of standard tests could be found, then the various vendors/project could run the tests themselves.