nogil

Extra links for the no gil work for anyone else curious about this [0], [1]. [0] Multithreaded Python without the GIL https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD... [1] Github repo https://github.com/colesbury/nogil

This might be a dumb question, but why would removing the GIL break FFI? Is it just that existing no-GIL implementations/proposals have discarded/ignored it, or is there a fundamental requirement, e.g. C programs unavoidably interact directly with the GIL? (In which case, couldn't a "legacy FFI" wrapper be created?) I know that the C-API is only stable between minor releases [0] compiled in the same manner [1], so it's not like the ecosystem is dependent upon it never changing. I cannot seem to find much discussion about this. I have found a no-GIL interpreter that works with numpy, scikit, etc. [2][3] so it doesn't seem to be a hard limit. (That said, it was not stated if that particular no-GIL implementation requires specially built versions of C-API libs or if it's a drop-in replacement.) [0]: https://docs.python.org/3/c-api/stable.htm

https://github.com/colesbury/nogil does manage to get rid of the GIL, but it's not certain to make it into Python core. The main problem is the amount of existing libraries that depend on the existence of the GIL without realizing it - breaking those would be extremely disruptive.

Important to note that this is different from the nogil project ( https://github.com/colesbury/nogil ) - this is about the feature that's landing soon in Python 3.12 which will allow multiple Python sub-interpreters each with their own GIL - hence enabling Python code to run multi-threaded in the same process, albeit with a fair amount of extra work to co-ordinate between those different interpreters. The tutorial itself is fantastic - it's not an easy feature to try out right now but I got there by following the step-by-step instructions. Also interesting is this project, linked to from the end of the tutorial: https://github.com/jsbueno/extrainterpreters Posted my own set of short notes on my blog: https://simonwillison.net/2023/May/15/per-interpreter-gils/

"PEP 703 – Making the Global Interpreter Lock Optional in CPython" (2023) https://github.com/python/peps/blob/main/pep-0703.rst https://peps.python.org/pep-0703/ colesbury/nogil https://github.com/colesbury/nogil : docker run -it nogil/python docker run -it nogil/python-cuda

I'm worried about the speedup My understanding is that it's based on the most recent attempt to remove the GIL by Sam Gross https://github.com/colesbury/nogil In addition to some ways to try to not have nogil have as much overhead he added a lot of unrelated speed improvements so that python without the gil would still be faster not slower in single thread mode. They seem to have merged those performance patches first that means if they add his Gil removal patches in say python 3.12 it will still be substantially slower then 3.11 although faster then 3.10. I hope that doesn't stop them from removing the gil (at least by default)

Looks like it's still active to me: https://github.com/colesbury/nogil/

I think you mean the work by Sam Gross: https://github.com/colesbury/nogil/ Interesting article about it here: https://lukasz.langa.pl/5d044f91-49c1-4170-aed1-62b6763e6ad0...

> Now let's get a sane concurrency story This is in very active development[1]! And seems like the Core Team is not totally against the idea[2]. [1] https://github.com/colesbury/nogil [2] https://pyfound.blogspot.com/2022/05/the-2022-python-languag...

>Because it's not a huge game changer. That is just not true. 1 - This helps parallelize regular python code too, like the kind that goes into writing web services. 2 - While you'd still write native extensions to get optimum performance for single threaded code, having the thread dispatch layer in python makes parallelizing execution convenient and robust. For example, see the comment from people who maintain scikit-learn below. I'd love to see python get composable task parallelism like Julia where you can freely spawn threads within threads without worrying about their lifetimes. link: https://github.com/colesbury/nogil/issues/36#issuecomment-10...

Is it actually a corporate initiative? The project is hosted on Sam's personal GitHub[1]. [1]: https://github.com/colesbury/nogil