undefined | Better HN

0 pointsbilbo0s12y ago0 comments

"..The multiple query test has a client-provided count of queries, so in most Java cases, we create a fixed-size array to hold the results fetched from the database. I wrote the Servlet and Gemini tests, so I can confirm that behavior in those tests..."

I agree, that approach would be best. I just was unsure why you didn't do it in Vert.x.

"...it is our understanding that idiomatic Vert.x usage encourages the use of asynchronous queries..."

Someone can correct me if I am wrong, but my understanding of Vert.x is that any query you send to the event bus is already asynchronous. There is no need for a developer to worry about threads at all when writing a vert.x handler. That handler will only ever be called from a single thread. So using a simple array is fine. Using the JsonArray is even better, because then it matches the Onion test idiomatically speaking. Which, I agree, is what you should be going for.

"...The Onion test was contributed by a reader and admittedly its compliance with the specification we've created is perhaps a bit dubious. We want a JSON serializer to process an in-memory object into JSON..."

Please don't misunderstand, the Onion test does what you want it to do. As well, it does it in the correct idiomatic fashion. That's exactly how I would write the Onion test. I was just wondering why the other tests went out of their way to decode Json and the reencode Json for each result. Onion only ever encodes to Json once, other tests are encoding and decoding multiple times. I only pointed out Vert.x because it was the most egregious. I mean in that case the answer from the persistor is already in Json. It is put in a non Json data structure... and then that data structure is encoded to Json??? Just seemed weird.

----

EDIT: Just verified that there is no need for thread safe code in a Vert.x handler. (Gotta say... that is pretty slick)

On a connected note... man ... these tests are a VERY good way to learn more about these different frameworks!

----

0 comments

bhauer12y ago

You're right, this is fascinating stuff. Like I said, we had not yet received a pull request for the Vert.x test, but presumably we will get one before Round 5? :)

We had not previously understood that there was no need for thread-safe behavior within a Vert.x handler. Removing that (apparently fictional) requirement allows us to use just a simple array. Out of curiosity, can you point me to where you found confirmation that handlers do not require thread safety?

Thanks again for your feedback!

Edit: spot checking Vert.x with a simple array does not appear to affect performance to a measurable degree.

bilbo0sOP12y ago

Sorry (for the tardy response)... minor emergency came up.

Anyway, the relevant part of the manual is:

http://vertx.io/manual.html#concurrency

Also discovered you are running the equivalent of a single threaded environment anyway. (8 workers on an 8 core machine)

Any reason for that???

Maybe more confusion about Vert.x concurrency??? If so, here is the relevant documentation on worker verticles:

http://vertx.io/manual.html#worker-verticles

Long story short, you dedicated the entire machine (8 cores) to the equivalent of database connection management, (the persistor). Very little of the machine, (whatever is context switched in effectively), is dedicated to request handling. Try something a bit more fair... like 4 workers and 4 web request handlers.

Also, I was going through the Node.js tests and I had a question... do you guys do any clustering for the Node tests at all? Or are these results from the tests run on a single Node?

Sorry for all the questions, just want to make all of the tests do the same thing across all of the frameworks so that when you guys run it again, we can use that data here in a more meaningful fashion. For us, it is useful.

bhauer12y ago

We used 8 Vert.x verticles on i7 because there are 8 HT cores and our understanding is that the "best practice" for Vert.x is to create a number of verticles equal to the server's core count [1]. Obviously we would be happy to hear from Vert.x experts about an more idealized configuration. Admittedly, we have not spent a great deal of time attempting to tune Vert.x and, like I said earlier, we have not received a pull request [2].

In all tests, the database test is allocating a greater amount of effort to database connection management (in totality: the handling of connections, statements, queries, and result sets) versus request handling. This is not unique to Vert.x. The reason some frameworks' database tests achieve nearly 50% as many requests per second on i7 versus the pure JSON test is simply that at the ~210k rps range for the JSON tests, we are running into a Gigabit Ethernet wall (which I have commented about elsewhere). If we had 10 GBE, the JSON test results on i7 would be even higher. (Also see comments elsewhere about our intent to normalize, to a degree, the response header requirements since the variation observed is attributable to response-headers [3].)

Yes, the node.js tests are running with the cluster module [4].

Thanks for the comments. We have received great feedback in the previous rounds and this round received even more attention so there have been some more good questions. Unfortunately, there has also been some rehashing, which indicates we're not doing a great job of explaining to people how each environment is configured (linking to the repository only goes so far). That said, we also continue to receive some fantastic pull requests. Thanks to everyone who has helped out!

[1] http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-...

[2] https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...

[3] https://github.com/TechEmpower/FrameworkBenchmarks/issues/11...

[4] https://news.ycombinator.com/item?id=5645598

j / k navigate · click thread line to collapse

0 pointsbilbo0s12y ago0 comments

I agree, that approach would be best. I just was unsure why you didn't do it in Vert.x.

"...it is our understanding that idiomatic Vert.x usage encourages the use of asynchronous queries..."

----

EDIT: Just verified that there is no need for thread safe code in a Vert.x handler. (Gotta say... that is pretty slick)

On a connected note... man ... these tests are a VERY good way to learn more about these different frameworks!

----

0 comments

bhauer12y ago

You're right, this is fascinating stuff. Like I said, we had not yet received a pull request for the Vert.x test, but presumably we will get one before Round 5? :)

Thanks again for your feedback!

Edit: spot checking Vert.x with a simple array does not appear to affect performance to a measurable degree.

bilbo0sOP12y ago

Sorry (for the tardy response)... minor emergency came up.

Anyway, the relevant part of the manual is:

http://vertx.io/manual.html#concurrency

Also discovered you are running the equivalent of a single threaded environment anyway. (8 workers on an 8 core machine)

Any reason for that???

Maybe more confusion about Vert.x concurrency??? If so, here is the relevant documentation on worker verticles:

http://vertx.io/manual.html#worker-verticles

Also, I was going through the Node.js tests and I had a question... do you guys do any clustering for the Node tests at all? Or are these results from the tests run on a single Node?

bhauer12y ago

Yes, the node.js tests are running with the cluster module [4].

[1] http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-...

[2] https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...

[3] https://github.com/TechEmpower/FrameworkBenchmarks/issues/11...

[4] https://news.ycombinator.com/item?id=5645598

j / k navigate · click thread line to collapse