undefined | Better HN

0 pointsspatley3y ago0 comments

Agreed that switching to lower level languages give the potential of many orders of magnitude. But the thing that was most enlightening was that removing pandas made a 9900% increase in speed without even a change to language. 20 minutes down to 12 seconds is a very big deal, and I still don't have to remember how to manage pointers.

0 comments

kzrdude3y ago

In some cases you can add pandas to a solution and speed it up by 10 seconds. It's about using the tools right

twobitshifter3y ago

I think that should be emphasized. The rest of the optimizations are entirely unneeded and added complexity to the code base. The next guy to work on this needs to be a cpp dev, but the requirements were only asking for 500ms which was more than met by the first fix. What the payoff of this added performance with and at what cost?

throwaway143563y ago

3 sec on 1 to 200 ms on 32 cores on the other hand smells like things could be better.

Aeolun3y ago

That feels like fairly expected overhead for managing a bunch of threads?

I mean, you could potentially push another 2x speedup out of it, but you’re already at 10000x, is that really necessary?

kaba03y ago

I don’t believe orders of magnitude is achievable in general. Even python, which is perhaps the slowest mainstream language clocks in at around 10x that of C.

Sure, there will be some specialized program where keeping the cache manually small you can achieve big improvements, but most mainstream managed languages have very great performance. The slowdown is caused by the frameworks and whatnot, not the language itself.

igouy3y ago

Guess it depends what you mean by "achievable in general".

Guess we don't have a way to measure "in general" so we are left with tiny tiny benchmarks programs.

    simple

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

    cpu secs Python 3 versus C gcc

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

tcfhgj3y ago

I tried counting a large amount of numbers based on a condition in a loop. 70 times difference between python3 and cargo run --release

kaba03y ago

That’s a tiny as hell microbenchmark though, where rust likely was able to vectorize even. The difference won’t be as drastic for larger (more meaningful) applications.

xigoi3y ago

According to these benchmarks, pure Python is about 100 times slower than C and other fast languages: https://github.com/kostya/benchmarks

pclmulqdq3y ago

Python programs often need to use a lot of optimized C and C++ libraries to get anywhere near reasonable performance. I would be shocked if a webserver written in Python was only 10x slower than one written in C (or Go or Rust for that matter).

j / k navigate · click thread line to collapse

0 comments

kzrdude3y ago

In some cases you can add pandas to a solution and speed it up by 10 seconds. It's about using the tools right

twobitshifter3y ago

throwaway143563y ago

3 sec on 1 to 200 ms on 32 cores on the other hand smells like things could be better.

Aeolun3y ago

That feels like fairly expected overhead for managing a bunch of threads?

I mean, you could potentially push another 2x speedup out of it, but you’re already at 10000x, is that really necessary?

kaba03y ago

I don’t believe orders of magnitude is achievable in general. Even python, which is perhaps the slowest mainstream language clocks in at around 10x that of C.

igouy3y ago

Guess it depends what you mean by "achievable in general".

Guess we don't have a way to measure "in general" so we are left with tiny tiny benchmarks programs.

    simple

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

    cpu secs Python 3 versus C gcc

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

tcfhgj3y ago

I tried counting a large amount of numbers based on a condition in a loop. 70 times difference between python3 and cargo run --release

kaba03y ago

That’s a tiny as hell microbenchmark though, where rust likely was able to vectorize even. The difference won’t be as drastic for larger (more meaningful) applications.

xigoi3y ago

According to these benchmarks, pure Python is about 100 times slower than C and other fast languages: https://github.com/kostya/benchmarks

pclmulqdq3y ago

j / k navigate · click thread line to collapse