Zygote.jl

ies (https://github.com/JuliaGPU/CUDA.jl), the same automatic differentiation library (https://github.com/FluxML/Zygote.jl), and of course the same JIT compiler (Julia itself). These two libraries are far enough apart that people say "Flux is to PyTorch as Lux is to JAX/flax", but while in the Python world those share almost 0 code or implementation, in the Julia world they share >90% of the core internals but have different higher levels APIs. If one hasn't participated in this space it's a bit hard to fathom how much code reuse goes on and how that is influenced by the design of multiple dispatch. This is one of the reasons there is so much cohesion in the community since it doesn't matter if one person is an ecologist and the other is a financial engineer, you may both be contributing to the same library like Distances.jl just adding a distance function which is then used in thousands of places. With the Python ecosystem you tend to have a lot more "megapackages", PyTorch, SciPy, etc. where the barrier to entry is generally a lot higher (and sometimes requires handling the build systems, fun times). But in the Julia ecosystem you have a lot of core development happening in "small" but central libraries, like Distances.jl or Distributions.jl, which are simple enough for an undergrad to get productive in a week but is then used everywhere (Distributions.jl for example is used in every statistics package, and definitions of prior distributions for Turing.jl's probabilistic programming language, etc.). I had never seen anything like that before in the R or Python space, by comparison Python almost feels like it's all solo projects.

e. And again, this is not limited to the lexical extent of a macro, and it does not disable compilation. It is instead a programmable step in compilation. A classic example of this would be Zygote.jl [1] which takes a program and recursively generates code to that would give the derivative of that program. Doing this requires you to modify literally every function call in the callstack, and you can do this all in pure julia without ever knowing a lick of LLVM. Granted, there is something ideologically appealing about the idea of a self-hosted langauge. I'm just arguing that on a practical level, the compiler tools allow total control over code execution without changing languages, just as you'd get in a self-hosted language. [1] https://github.com/FluxML/Zygote.jl

clude debugger interupts. This stuff also has a bunch of use for probabalistic programming languages etc. The thing is Cassette and IRTools implement this at the lowered code level. It runs on the SSA-form IR. This IR was never really designed for the user to write it. Though IRTools does a noble attempt to make it nice. Arborist instead runs at the Abstract Syntax Tree (AST) level. Just like macros. This is designed to be written and manipulated by users. Users do that all the time by writing macros. I haven't touched it in a while (apparently 2 years). Largely because AFAICT it's not useful outside of pedagogical purposes. AST is not actually that nice for any program tranforms i want to do, since there are so many ways to write the same thing. E.g. different kinds of loops and branches, where as in SSA form IR they all look the same. Probably could be made to work, but i just don't really have a usecase that I can't do easier on IR. (probably people who haven't been deep into IR on the other hand would think of some) There are a bunch of edge cases where it crashes still, for things that are not allowed to appear in generated code. It wouldn't be too hard to make it work though, not really. Particularly since we now have tools like https://github.com/SciML/RuntimeGeneratedFunctions.jl https://github.com/JuliaStaging/GeneralizedGenerated.jl

formance optimizations and deploy capabilities of a tensorflow graph and with an easy way to plug any new operation and it's gradient by writing in the same host language (so no need to learn or restrict yourself the tensorflow/pytorch defined methods/DSL). You don't even need to change the compiler or define a new language for it. Julia's Zygote [1] is just a 100% Julia library you can import, for which you can add at any point any custom gradient even if the library creators never added them, and them run either on CPU or GPU (for which you can also fully extend using just pure Julia [2]). And of course, you can also use a higher level framework like Flux [3] which is also high level Julia code. I think the heart of differentiable programming is just another step in the evolution, from early (lua) torch-like libraries that gave you the high level blocks to compose, to autodiff libraries that gave you easy access to low level math operators to build the blocks to the point where you can easily create your own operators to create the high level blocks. [1] https://github.com/FluxML/Zygote.jl [2] https://github.com/JuliaGPU/CUDAnative.jl [3] https://github.com/FluxML/Flux.jl

/JuliaDiff/ReverseDiff.jl) - [AutoGrad.jl](https://github.com/denizyuret/AutoGrad.jl) - [NiLang](https://github.com/GiggleLiu/NiLang.jl) (arguably not reverse mode) Symbolic: - [ModelingToolKit](https://github.com/JuliaDiffEq/ModelingToolkit.jl) - [XGrad.jl](https://github.com/dfdx/XGrad.jl) Finite Differencing: - [Calculus](https://github.com/JuliaMath/Calculus.jl) (please stop) - [FiniteDifferences](https://github.com/JuliaDiff/FiniteDifferences.jl) - [FiniteDiff](https://github.com/JuliaDiff/FiniteDiff.jl)

code within the Julia ecosystem. Specifically capabilities such as Zygote.jl (https://github.com/FluxML/Zygote.jl) for differentiable programming have the potential to dramatically make it easy to apply ML techniques to scientific codebases. Compiler contributors are hard to come along, so all contributions to compiler technology are incredibly valuable. The DiffEqFlux.jl ecosystem in Julia is a good example of combining mechanistic models with ML (https://github.com/JuliaDiffEq/DiffEqFlux.jl). Hop on to the Julia slack channel or discourse to dig in deeper. Another thing Bret Victor speaks about in his blog post is working with agencies such as ARPA-E on advanced projects. Julia Computing is participating in an ARPA-E project to bring these capabilities to many energy related simulation and development technologies. This press release gives a broad idea: ARPA-E press release: https://www.energy.gov/articles/department-energy-announces-... Funded projects: https://arpa-e.energy.gov/sites/default/files/documents/file... Julia Computing press release: https://juliacomputing.com/communication/2019/12/09/arpa-e.h...

w by themselves. 3) It doesn't need a language extension (Julia doesn't, but that's because you can access Julia IR using the language itself, not many language support this level of metaprogramming). The OOP strategy (like Pytorch) overload methods so every time you call the overloaded method it builds the graph. It depends on customized types and methods and does not support code that isn't using those custom types that can hold the graph (native/custom types), native control flow (only implicitly by changing what graph is being created), reduces the possible optimizations (you only have one subderivative, not the complete graph to optimize) and end up creating a DSL with it's own error messages and quirks that is less natural than just using the host language. I can't say how it would look using monads though. [1] https://github.com/FluxML/Zygote.jl