undefined | Better HN

0 pointsquietbritishjim7y ago0 comments

Let's say I write this:

    def add_and_mult(a, b, c):
        return a + b @ c

If a, b and c are numpy arrays then this function releases the GIL and so will run in multiple threads with no further work and with little overhead (if a, b and c are large). I would describe this as a function "written in Python", even though numpy uses C under the hood. It seems you describe this snippet as being "written in C instead of Python"; I find that odd, but OK.

But, if I understand you right, you are also suggesting that the other commenters here that talk about the GIL would also describe this as "written in C". They realise that this releases the GIL and will run on multiple threads, but the point of their comments is that proper pure Python function wouldn't. I disagree. I think that most others would describe this function as "written in Python", and when they say that functions written in Python can't be parallelised they do so because they don't realise that functions like this can be.

0 comments

chrisseaton7y ago

This only gives you parallelism in one specific situation though - where operations like '+' and '@' take a long time. If they were fine-grained operations, then this doesn't help you.

If instead of operating on a numerical matrix, you were instead operating on something like a graph of Python objects, something like a graph traversal would be hard to parallelise as you could not stay out of the GIL long enough to get anything done.

quietbritishjimOP7y ago

I agree that not quite all situations will be covered by coarse locks like this. But many will, and my original comment was meant to draw attention to those situations. Your previous comment seemed to be saying that everyone already knew about those, but I still believe that some people commenting or reading here weren't aware you could release the GIL with just a numpy call.

I did also concede that if you do have to write your algorithm completely from scratch, with no scope for using existing C extensions (be they general purpose like numpy or more specialist that implements the whole algorithm) then yes you'll be caught be the GIL, so I agree with you on that. But I also made the point that you'll be caught even more (orders of magintude more!) by the slowness of Python, so any discussion about parallelism or the GIL is a red herring. It's like worrying that you car's windscreen will start to melt if you travel at 500mph; even if that's technically true, it's not the problem you should be focusing on.

It's interesting you mention graphs because the most popular liberally licensed graph library is NetworkX, which is indeed pure Python and so presumably isn't particularly fast. There are graph libraries written as C extension modules but I believe they are less popular and less librally licensed (GPL-style rather than BSD-style). So I definitely agree that this is a big weakness of the Python ecosystem.

galangalalgol7y ago

This is a good point. Fine grained parallelism works well with c extensions, but coarse grained does not. Which is why data science and MATLAB like tasks are so often done in python without worrying too much about the python penalty. But if you have a lot of small dense matrix operations, even VBA in excel will be faster by more than 10x because you keep popping back into python.

quietbritishjimOP7y ago

I agree with the idea, but you have fine grained and course grained switched.

bogomipz7y ago

Could you explain the return statement in your example? I only know the '@' as a decorator in Python. This looks like invalid syntax to me what am I missing?

augusto21127y ago

It's matrix multiplication.

Has been in Python since version 3.5

https://www.python.org/dev/peps/pep-0465/

bogomipz7y ago

Thanks, I'm not quite that up to date on my 3.x Python. Cheers.

j / k navigate · click thread line to collapse

0 pointsquietbritishjim7y ago0 comments

Let's say I write this:

    def add_and_mult(a, b, c):
        return a + b @ c

0 comments

chrisseaton7y ago

This only gives you parallelism in one specific situation though - where operations like '+' and '@' take a long time. If they were fine-grained operations, then this doesn't help you.

quietbritishjimOP7y ago

galangalalgol7y ago

quietbritishjimOP7y ago

I agree with the idea, but you have fine grained and course grained switched.

bogomipz7y ago

Could you explain the return statement in your example? I only know the '@' as a decorator in Python. This looks like invalid syntax to me what am I missing?

augusto21127y ago

It's matrix multiplication.

Has been in Python since version 3.5

https://www.python.org/dev/peps/pep-0465/

bogomipz7y ago

Thanks, I'm not quite that up to date on my 3.x Python. Cheers.

j / k navigate · click thread line to collapse