async-http-client

usually be performed synchronously so the returned future would already be complete and the application would continue. However, sometimes the write wouldn’t fully complete and the future could not be completed until the next time around the selector loop. But the I/O thread was blocked, so the loop couldn’t proceed, so the future would never complete. * Since the I/O thread was stuck, it couldn’t respond to any further messages from any of its managed sockets. Eventually the client would give up and send a FIN segment, and the kernel’s TCP/IP stack would put the socket in the CLOSE_WAIT state. Usually the I/O thread handles this in its worker loop by calling close on the socket, but it was stuck so this code never ran and the socket would never close. * The boss thread was still running, so the system continued to accept new connections and assign some of them to the stuck I/O thread. To add insult to injury, Netty actually has a deadlock detector to prevent this scenario from occurring. Unfortunately, one of the libraries that we were using at the time is also based on Netty, and they explicitly disabled the deadlock checker, globally! https://github.com/AsyncHttpClient/async-http-client/commit/... So there you have it. Don't block in your Netty worker threads. This should have been obvious at the time - I had worked with Netty before - but we were too busy investigating to sit back and just think. If I had stood in front of a whiteboard for an hour, I suspect I could have worked it out, but instead we spent anxious days tearing our hair out attacking the problem by stepping through multiple threads of code in a debugger (racing timeouts is always fun) and taking endless stack traces and thread dumps. The short-term mitigation ended up being trivial: we could still block the worker thread; we just had to check to see if the channel was already closed first! This is not a perfect fix, but it let us preserve the illusion that the application code had a real bona-fide blocking java.io.OutputStream. The long-term solution is to switch the API handlers over to using real async I/O. To this day, my fellow engineer investigating this issue refers to this saga as the Christmas Miracle bug. :)