As it's beyond my ken I would like to know from someone capable of answering if this is going to be a constant constraint, stemming from the design of the project, or if memory usage is likely to go down at some point.
FWIW, my own tests of pypy 1.7 have shown the memory overhead to be about 3x: http://groups.google.com/group/python-tornado/browse_thread/...
If you search for my username in that thread I asked for someone to run the code from the story on a similar dataset with PyPy. Twice as fast but x10 memory usage. But as I said elsewhere a bit of searching suggests it won't be a showstopper.
(No incremental build, it takes roughly three hours on a high-end nehalem workstation, and if it fails for any reason, you get to start all over again!)
A bit of searching and it seems the memory problems I raised in my other comment aren't so drastic after all. Theoretically it can use less memory in many operations and the current blowouts are not as high as I thought (I was going from one benchmark a HNer, brianh, was kind enough to run for me[1]).
Not to belittle the project, but I tried a vanilla order entry implementation on my test box in nasdaq. Essentially it runs an epoll loop using a C extension to take advantage of the myricom DBL calls. RT latency was roughly 5 usec faster using cpython. As for other parts of the system (eg feed handler), the performances were comparable.
For other applications (eg compliance reporting) pypy is 2x cpython speed, but I could care less about that timing (even if it ran 1000x slower than cpython, it wouldn't matter)
TL;DR: it needs to be useful. And I just don't see the usefulness for my Python applications.
Especially after Google invested engineers on Unladen Swallow, which fizzled out. I've read that Google likes to max out their servers to the point where OOM is not unlikely, so PyPy memory usage might not be worth the runtime performance gains.
Perhaps there is a business opportunity here? Python as a service, sort of like how Python runs on Google App Engine, but using PyPy. PyPy's sandboxing makes this easier actually, and the main advantage of course would be performance.
It actually works right out of the box. You can download it and get real world examples to work in five minutes. It works with C++. It works with numpy.
http://wiki.cython.org/WrappingCPlusPlus
http://wiki.cython.org/tutorials/numpy
It's the real deal: ridiculously easy to use as a drop-in replacement for slow functions. It works on problems that are not toys. It allows you to use high performance C and C++ libraries, it's used in a large and well maintained project (sage), and development was at least partially funded by Google.
With all due respect to Maciej for his work, PyPy has been in the works for many years now, and projects like that don't tend to ship -- or to meet expectations if they actually do ship.
Also, it's true that pypy took many years to build, but it actually does work. If you claim that pypy never shipped, it very likely means none of the "new python interpreter" projects would ever ship for you, but hey, there are people out there for whom it works.
Cheers, fijal
- The JIT optimization doesn't survive between different runs of the script. Since I run a different script for every collage, I haven't see a noticeable speedup.
- The script also download the 300-500 images from facebook with curl, this module is not supported yet by pypy
- Since the most of the time is spent on images resize (PIL.resize) which is in C, the JIT can't do many optimizations.
I think the reason it works for clang is because they're really trying hard to be drop-in compatible with gcc. I remember reporting some minor thing like: command line flag -- in gcc generated a warning, but it didn't return an error code on exit and it did in clang, so they've changed it, because it caused crashes some project builds.
Not sure how pypy handles the compatibility now, because I couldn't compile it yet, tried like 4 times already and it always crashes randomly or it gets out of memory or something.
PyPy needs to use their great implementation stack to develop their own killer feature. Like for instance a small and fast python for embedded scripting or compiling normal python program to a small and optimized native executable.
