This has a big resemblance IMO to most formal study methods including University and natural languages. Once you know the basics you can optimize for learning or for grades, depending on your goals. They are correlated (but not strongly correlated), but the time investment is not negligible so you'll have to choose either of them.
We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.
That said, FWIW I think Edge/Chakra gets the crown on Windows!
(I work on Chakra)
One example could be focusing on throughput instead of startup latency which could yield distorted outcomes for real world apps, or mostly app code that is mostly early-bound style OO code/monomorphic functional code vs code that is hostile to class shape analysis/PIC.
It seems in the last few years, especially with the idea of Web competing with native with stuff like asm.js for games, there was a focus on maximize throughput, but it's rare your typical interactive web app is running code that benefits from that.
This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests".
I'm inclined to agree that becnhmarks do become obsolete as workloads change and development targets benchmark pointscoring. The solution is to make a new standardised benchmark, or indeed many of them. If they are clear about what they are measuring then the relative perfomance of different JavaScript engines can be understood by developers to give a nuanced view of what performs well.
We always wondered why Android was so crazy ridiculously slow at our real world JS vs. iOS starting in late 2013 onward.. getting worse and worse over 2014 and 2015.. and "gee, we optimized for the wrong thing aka Octane" was a big part of that.
http://benediktmeurer.de/2016/12/16/the-truth-about-traditio...
You can read more about the history at https://meta.discourse.org/t/the-state-of-javascript-on-andr...
This shows that browser vendors are not really good at writing JS benchmarks representative of real world, which is not surprising considering that they are not doing real web development.
Ideally benchmarks would come from developers of popular websites like Facebook, Twitter, etc.
Basically, Javascript performance is "done". It's not going to get much faster at this point, and higher performance will come from paradigm shifts (SIMD, WebAssembly, pick your fad).
