Poirot.jl

It currently has problems with some classes of models on the GPU, but this is just due to memory management and is going to fixed soon. Julia is natively compiled, so doesn't use a tape, tracing, or for memory management simple ref counting (in this case not as good). This is slated to be fixed in the short term with an abstract tracing framework which eliminates memory allocations. Given Julia's type information and ability to manipulate IR from third party programs, this is more general and powerful than pytorch's tracing system. Works on a larger set of code (the whole language), doesn't require actually running the code (abstract tracing), and allows for other program transforms/ analysis like compilation to XLA, shape inference, compile time errors, source to source prob programming : https://github.com/MikeInnes/Poirot.jl, and ot

Even in swift it isn't "turtles all the way down". The AD stuff is hardcored into the C++ guts of the compiler, whereas Julia's source to source autodiff accesses a compiler pass from a fully Julia user package. Aside from making it easier to hack and improve the AD system as just a Julia user, this capability enables other package program transforms like that in https://github.com/MikeInnes/Poirot.jl for prob programming. So Julia is already further ahead in that regard and it's more hackable.