CUDAnative.jl

formance optimizations and deploy capabilities of a tensorflow graph and with an easy way to plug any new operation and it's gradient by writing in the same host language (so no need to learn or restrict yourself the tensorflow/pytorch defined methods/DSL). You don't even need to change the compiler or define a new language for it. Julia's Zygote [1] is just a 100% Julia library you can import, for which you can add at any point any custom gradient even if the library creators never added them, and them run either on CPU or GPU (for which you can also fully extend using just pure Julia [2]). And of course, you can also use a higher level framework like Flux [3] which is also high level Julia code. I think the heart of differentiable programming is just another step in the evolution, from early (lua) torch-like libraries that gave you the high level blocks to compose, to autodiff libraries that gave you easy access to low level math operators to build the blocks to the point where you can easily create your own operators to create the high level blocks. [1] https://github.com/FluxML/Zygote.jl [2] https://github.com/JuliaGPU/CUDAnative.jl [3] https://github.com/FluxML/Flux.jl

run on GPUs and TPUs. The major restriction on GPUs is that you cannot do dynamic allocation, so you need to write non-allocating Julia code, but that's common requirement for high-performance code anyway. [1] https://github.com/JuliaGPU/CUDAnative.jl

detriments e.g. warp divergence. It turns out newer GPU architectures give you more leeway in that regard than any of us on the team expected. > How does caching come into play? Depends on which kind of "caching" you're referring to. If you mean tape-level partial derivative caching/memory usage: Broadcasting a forward-mode derivative operator, as presented in this paper, can save on memory when it enables better fusion than reverse-mode on complicated kernels (resulting in fewer temporaries). However, there is also a question of when this technique should actually be employed: during the forward pass, or during the reverse pass? If employed in the forward pass, then the primal and partial derivative calculations can be fused, reducing compute cost. However, doing so means that the memory required to store the partial derivatives is held captive until those derivatives can be backpropagated in the reverse pass. Conversely, employing the technique in the reverse pass allows you to free the partial derivative storage quickly, but features some redundant computation. Section 2.2.3 of the paper discusses this a bit. If you mean instruction-level caching, i.e. efficient pipelining of memory into registers: On the CPU, it's quite easy to thrash cache for high-arity dual number calculations (i.e. calculations where dual number instances carry around a long stack-allocated array of partial derivatives). Our experiment in Section 3.4.1 tries to characterize the analogous GPU behavior by measuring how occupancy scales with target calculation arity. Also, there was definitely a bit of implementation work to ensure that loads from our GPU-backed "dual number" arrays coalesced properly, that indexing calculations were compiled away when possible, etc. The cool part is that the dual numbers themselves were just the implementation provided by the ForwardDiff package (https://github.com/JuliaDiff/ForwardDiff.jl), which contains no GPU-specific specialization, and they're automagically JIT-compiled for the GPU by CUDAnative (https://github.com/JuliaGPU/CUDAnative.jl). > What about intrinsic condensing functions? Hmm...I'm not positive I know what "intrinsic condensing functions" are. Apologies!

ost advanced tools at that time. We start using Julia now.

ds-checked arrays are not supported yet, due to a bug [1] in the NVIDIA PTX compiler. [0] We ran into what appears to be the same issue [2] about a year and a half ago. nvidia is well aware of the issue, but I don't expect a fix except by upgrading to Volta hardware. [0] https://julialang.org/blog/2017/03/cudanative [1] https://github.com/JuliaGPU/CUDAnative.jl/issues/4 [2] https://bugs.llvm.org/show_bug.cgi?id=27738