zproc

It would be really cool to have the sort of Erlang-style processes in python -- where each process is a green process, so don't pay the cost of OS handling stuff, but at the same time you run multiple interpreters so that you still get the benefit of multiple cores [1]. Implementation-wise, this could be made simpler than removing the GIL, if we just fully give up on the concept of shared memory. (This is above my pay-grade, anyone with CPython internals knowledge, please share your thoughts!) I've been trying to build a shared state system over message passing that doesn't use shared memory at all [2], and I have experienced that OS level processes are a high cost to pay, when you try to do things the Erlang way, launching thousands of processes. [1] https://hamidreza-s.github.io/erlang/scheduling/real-time/pr... [2] https://github.com/p

Working on my multiprocessing library, ZProc https://github.com/pycampers/zproc It's cool because it tries to bring some creativity into the rather standstill multitasking world :)

Updated readme to be a little better https://github.com/pycampers/zproc I use it in a couple of my own stuff which I have open sourced yet. I do plan to release them, and I hope I can prove the usefulness of the library using those... professional (corporate) stuff would be a far fetch for me, obviously.

Update - I hopefully made the language a little better? https://github.com/pycampers/zproc/blob/master/README.md

> "high performance" I never claimed it to be performant! "Above all, ZProc is written for safety and the ease of use." (Read here - https://github.com/pycampers/zproc?files=1#faq) > It's not a revolution I totally agree. It's just a better way of doing things zmq already perfected. Like, tell me if you've ever seen a python object that has a `dict` API, but does message passing in the background. > central (pubsub?) server. Central server, yes. It uses PUB-SUB for state watching and REQ-REP for everything else. > you've just "discovered" message-passing Guess you're right? 2 years is a peanut on the time scale... P.S. Thanks for all the feedback, I've been dying to hear something for a while now.

I believe that since the Advent of zeromq, parallelism is possible in almost any language, including python. My library lets you do parallelism in a unique way, where you do message passing parallelism without being explicit about it. https://github.com/pycampers/zproc/

> deal with serialization and communication I thought a lot about this problem, for over 2 years, and came up with zproc https://github.com/pycampers/zproc Basically, > It lets you do message passing parallelism without the effort of tedious wiring. You'll be doing message passing without ever dealing with sockets! Also, Shared memory parallelism is hard to get right irregardless of which language you use. I would recommend strongly against it, unless you're writing some really really really niche thing where message passing is a bottleneck (it isn't most of the time)

zproc looks like am interesting approach to concurrency in python. For anyone else curious about it, here is the github project: https://github.com/pycampers/zproc

I created a library for multiprocessing, which is loosely based on actor model. The interesting bit is that you'll never have to do the wiring yourself, zproc does it all in the background. https://github.com/pycampers/zproc

Threads are not hard. In fact, threads are extremely easy to implement. However, real Threading code is just incredibly difficult to reason, just by looking at it. This makes it easy for you to introduce race conditions without even knowing that there is one! There is also the fact that locks don't lock anything! They are just a flag, that a any code may choose to ignore. They are a not an enforcing tool, just a cooperative one. (More here: https://www.youtube.com/watch?v=9zinZmE3Ogk) P.S. I created a library, that makes it easier to write safer multiprocessing code https://github.com/pycampers/zproc