AutoGrad.jl

We have so many AD packages than that in julia. Julia suffers the lisp problem for AD. http://winestockwebdesign.com/Essays/Lisp_Curse.html It is a real problem. One of my on going projects is ChainRules (http://www.juliadiff.org/ChainRulesCore.jl/dev/) which will unite them under one set of custom senstitivities and more generally make it easier to mix and match them I need to update the JuliaDiff website, I want to list all of them in a table with some some key points. Forward mode: - [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff.jl) - [ForwardDiff2](https://github.com/YingboMa//ForwardDiff2.jl) Reverse Mode: - [Nabla](https://github.com/invenia/Nabla.jl/) - [Tracker](https://github.com/FluxML/Tracker.jl) - [Yota](https://github.com/dfdx/Yota.jl) - [Zygote](https://github.com/FluxML/Zygote.jl) - [ReverseDiff](https://github.com

Those are based on overloading functions to add the new behaviors (computing the forward diff or building the graph within a special variable for the reverse mode) which in Julia is implemented for example in [1], [2], [3] and [4], while the one in OP is based on source analysis and transformation. And the similarities with Lisp are more than just the parser being written in it, Julia's programming paradigm is based on the CLOS, everything is an expression, hygienic macros and reader macros and code is data. It mostly just lacks sexpr. [1] https://github.com/JuliaDiff/ForwardDiff.jl [2] https://github.com/dfdx/Yota.jl [3] https://github.com/denizyuret/AutoGrad.jl (from the Knet framework) [4] https://fluxml.ai/Flux.jl/stable/internals/tracker/ (from the Flux framework)

I think you're assuming that "automatic differentiation" (AD) is the same thing as "using dual numbers", but AD is a family of methods including dual numbers. The blog post you link to says they're using https://github.com/denizyuret/AutoGrad.jl which uses reverse-mode differentiation not Dual numbers. That means they're performing the same operations (or nearly the same) as standard backpropagation. Dual numbers are just the wrong approach for neural nets, which have many parameters. The amazing thing about backprop / reverse-mode differentiation is that you get the derivatives wrt all the weights in one reverse sweep through the network.