partisan

I was curious about this claim, so I tracked it down: https://blog.stenmans.org/theBeamBook/#_distribution_in_a_la... > Even though Erlang’s asynchronous message-passing model allows it to handle network latency effectively, a process does not need to wait for a response after sending a message, allowing it to continue executing other tasks. It is still discouraged to use Erlang distribution in a geographically distributed system. The Erlang distribution was designed for communication within a data center or preferably within the same rack in a data center. For geographically distributed systems other asynchronous communication patterns are suggested. Not clear why they make this claim, but I think it refers to how Erlang/OTP handles distribution out of the box. Tools like Partisan seem to provide better defaults: https://github.com/lasp-

> I do wonder if maybe streaming large data chunks over Erlang distribution might be a problem and a secondary data channel (e.g. over udp or sctp) might be worth playing with. You may want to take a look at the partisan[0] library written in Erlang. It is basically that, a reimagination of distributed Erlang, except that it can be multiplexed over multiple connections. [0] - https://github.com/lasp-lang/partisan/

You might be interested in Partisan's fault injector. I don't know if @cmeiklejohn is still working on it, but it's really cool nonetheless. https://www.youtube.com/watch?v=KrwhOkiifQ8 https://github.com/lasp-lang/partisan

I can also mirror this general guideline. I've run 250+ node erlang clusters just fine in the past. There were some caveats to how some built-in OTP libraries behaved but they were easy to replace or workaround in the past. That was many years ago as well. The distributed erlang story has improved with more recent releases (better performance on remote monitors for example) which might push the number a little higher than 300 if you are careful. Keep in mind the default style of clustering is fully connected so there is some danger in managing that many network connections (quadratically scaling for each node added) during network partitions which can be a problem if you're not tuning things like TCP's TIME_WAIT for local networking conditions. Even better, these days there are great libraries like partisan (https://github.com/lasp-lang/p

In large Erlang applications, the "send to multiple peers" broadcast tendencies of its message passing can indeed cause the network to become a bottleneck; so can the default chattiness that a lot of Erlang application developers assume for their app behavior. Fortunately, there are lots of simple-to-apply patterns and solutions when you get to that point, for example the excellent Partisan library: https://github.com/lasp-lang/partisan

Lasp use Partisan for its distribution, not the default erlang distribution. It was built exactly to deal with a high level of churn, specifically for edge computing with spotty network. So yes it does. https://github.com/lasp-lang/partisan

Direct link to the github : https://github.com/lasp-lang/partisan

I, too, have a bit of experience with backpressure implementations in both Erlang and Pony. Erlang's "penalize the sender" doesn't work in many cases, so I'm not surprised that it's being removed. Erlang's remote distribution implementation & messaging semantics are Mostly Great but is definitely not Perfect. 1. Head-of-line blocking caused by congestion on the single TCP connection used for message transmission between any two Erlang nodes. This can cause major problems for apps control vs. data plane design, such as Riak and Lasp. Work on Partisan (https://github.com/lasp-lang/partisan) appears to be a substantial improvement. 2. If the single remote distribution TCP connection between two nodes is broken, then the first Erlang process to send a message to the remote node is, hrrm, well, borrowed/co-opted by the BEAM VM to connect th

Instead of rolling your own connection subsystem, there is also this https://github.com/lasp-lang/partisan Although they call it still experimental, it seems to be pretty actively developed.