A 40-line fix eliminated a 400x performance gap (opens in new tab)

Yes you need some lazy setup in thread-local state to use this. And short-lived threads should be avoided anyway :)

> Presumably you mean you just double check the page value after the rdtsc to make sure it hasn't changed and retry if it has?

Yes, that's exactly what a seqlock (reader) is.

clock_gettime() goes through the vDSO shim, but whether it avoids a syscall depends on the clock ID and (in some cases) the clock source. For thread-specific CPU user time, the vDSO shim cannot resolve the request in user space and must transit into the kernel. In this specific case, there is absolutely a syscall.

So someone else linked the original flamegraph site [0] and it describes icicle graphs as "inverting the y axis" but that's not only what's happening, right? You bucket top-down the stack opposed to bottom-up, correct?

[0] https://www.brendangregg.com/flamegraphs.html

Yes, they are made with: http://www.brendangregg.com/flamegraphs.html and

https://github.com/brendangregg/FlameGraph

Useful site if you are on to perf/eBPF/performance things with many examples and descriptions even for other uses as e.g. memory usage, disk usage (prefer heatmaps here but they are nice if you want to send someone a interactive view of their directory tree ...).

In Java it can be a bad toString() implementation hiding behind a + used for string assembly.

Or another great one: new instances of ObjectMapper created inside a method for a single call and then thrown away.

> but the 2nd is also not a pattern I really see apart from in JS codebases.

If you're referring to "one-line logger calls that trigger expensive serialization", it's also common in java.

Flame graphs have an official web site, maintained by Brendan Gregg, who invented them: https://www.brendangregg.com/flamegraphs.html. It's a useful starting point.

I would also try hotspot, it is a interactive viewer for perf graphs.

I use them all the time on Perl code.

https://metacpan.org/pod/Devel::NYTProf

Stability and accuracy, when applied to clocks, are generally about dynamic range, i.e. how good is the scale with which you are measuring time. So if you're talking about nanoseconds across a long time period, seconds or longer, then yeah, you probably should care about your clock. But when you're measuring nanoseconds out of a millisecond or microsecond, it really doesn't matter that much and you're going to be OK with the average crystal oscillator in a PC. (and if you're measuring a 10% difference like in the article, you're going to be fine with a mechanical clock as your reference if you can do the operation a billion times in a row).

Fair point. These were run on a standard dev workstation under load, which may account for the noise. I haven't done a deep dive into the outliers yet, but the distribution definitely warrants a more isolated look.

I was unsure whether to post it or not so I am glad you found it useful!

Thanks for the kind words and the link :).

You are spot on that the original author had a valid reason: at the time, it was literally the only way to do it.

The method in question (Java 1.5) was released in September 2004. While the POSIX standard existed, it only provided a way to get total CPU time, not the specific user time that Java needed. You can read about it more in the history section here: https://norlinder.nu/posts/User-CPU-Time-JVM/#a-walk-through....

But it's worth noting that while this specific case can be "fixed" with a function call, parsing /proc is still the standard way to get data in Linux.

Even today, a vast amount of kernel telemetry is only exposed via the filesystem. If you look at the source code for tools like htop, they are still busy parsing text files from /proc to get memory stats (/proc/meminfo), network I/O, or per-process limits. See here https://github.com/hishamhm/htop/blob/master/linux/LinuxProc....

Note that pprof produces much fancier interactive flame graphs. I'm not sure they're a single SVG though.

Also `samply` and the Firefox profiler are pretty fancy too.

There's really no reason to use the original flamegraph scripts.

Oh for some time after its introduction, CLOCK_MONOTONIC_RAW wasn't vDSO'd and it took some time and syscall profiling ('huh, why do I see these as syscalls in perf record -e syscalls' ...) to understand what was going on.

Thanks, I really should've looked deeper than that.

Exactly this.

It depends. If you’re doing continuous profiling, it’d make a call to get the current time at every method entry and exit, each of which could then add a context switch. In an absolute sense it appears to be small, but it could really add up.

This is what flame graphs are super helpful for, to see whether it’s really a problem or not.

Also, remember that every extra moment running instructions is a lost opportunity to put the CPU to sleep, so this has energy efficiency impact as well.

28us is still solid amount of time