New Computer Language Benchmarks Game metric: time + source code size (opens in new tab)

Geometric mean of (time + gzipped source code size in bytes) seems statistically wrong.

What if you shifted time to nanoseconds ? Or source code size in terms of Megabytes. The rankings could change. The culprit is the '+'

I would think Geometric mean of (time x gzipped source code size) is the correct way to compare languages together. It would not matter what the units of time or size are in that case.

[Here the geometric mean is the geometric mean of (time x gzipped size) of all benchmark programs of a particular language.]

I really wish they aggregated the metric of build time (+ whatever).

That is a huge metric I care about.

You can figure out it somewhat by clicking on each language benchmark but it is not aggregated.

BTW as biased guy in the Java world I can tell you this is one area Java is actually mostly the winner even beating out many scripting languages apparently.

I'd be interested to see "C compiled with Clang" added as another language to the benchmark games. In part, digging into Clang vs gcc benchmarks is always interesting, and in part, as Rust & Clang share the same LLVM backend, it would shed light on how much of the C vs Rust difference is from frontend language stuff vs backend code gen stuff.

With a totally arbitrary conversion of 1 second = 1 gzipped byte.

This is basically meaningless. I don't see why you'd even need to do this. You can easily show code size and performance on the same graph.

This presentation is pretty bad, there should be more context, some kind of color scheme or labels instead of text in the background, spacing between the languages represented, other benchmarks than the geometric mean, etc.

For comparing multiple implementations of a single benchmark in a single language, this sort of data would be interesting as a 2D plot, to see how many lines it takes to improve performance by how much. But for cross-language benchmarking this seems somewhat confounding, as the richness of standard libraries varies between languages (and counting the lines of external dependencies sounds extremely annoying, not only because you have to decide whether to include standard libraries (including libc), you also need to find a way not to penalize those for having many lines devoted to tests).

The thing they should change is to forbid the nonsense like:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Actually if you look at all the top net core submissions the only one fast are the one using low level intrinsics etc ...

Just curious, why does this benchmark not include D language? I remember seeing it a few years ago. Was it removed recently?

Predictably, "The Computer Language Benchmarks Game" once again proves the worthlessness of "The Computer Language Benchmarks Game".

This thing has been a long running joke in the software industry, exceeded only by the level of their defensiveness.

SMH.

I think APL had already shown, that brevity in itself is not desirable.

these pseudo benchmarks should be banned

I dont buy these results at all. Julia at second place looks like plain lie and complete nonsense, to the point I'm gonna look into this and run it myself.

After trying hard to use julia for about a year and I came to conclusion it's one of the slowest things around. Maybe the stuff changed? Maybe, but julia code still remains incorrect.

I hope they fix both things, speed (including start up speed, it counts A LOT) and correctness.