torchdiffeq

In 2016 yes. But this has been very much addressed. At this point, not only is Julia well-tested, but it's so well-tested that it has to carry around its own patched LLVM and its own math library in order for its full tests to pass, because for example it requires things like sin to be correct to 1ulp which isn't actually true in a lot of implementations! Then when you go to packages, there's a huge culture of testing. Compare OrdinaryDiffEq's (just the ODE part of DifferentialEquations.jl) test suite: https://travis-ci.org/JuliaDiffEq/OrdinaryDiffEq.jl which does convergence tests on every algorithm, along with regression tests on every interpolation and analytical value unit tests on each feature, etc. Meanwhile, scipy integrate has some of its own algorithms, but only a few regression tests, and most tests just check that things run: h

There is a PyTorch implementation by the authors https://github.com/rtqichen/torchdiffeq and there is a Julia implementation due: https://news.ycombinator.com/item?id=18676986#18691054. As for TensorFlow, we will have to wait and see if it can be done. The adjoint method is matrix multiplication, but the tricky part will also be integrating with numerical ODE solvers.

My understanding is that the advantage of something like a SNN is mainly in custom-hardware ultra-lower-power applications. But it adds a lot of complexity and makes training a lot more difficult, compared to something like ODE nets or standard discrete neural nets. Looking at the links you provided, they don't appear to train the network dynamics, only the feedout weights. In principle you could differentiate through the ODE solvers used in the software package linked to train the dynamics, but as far as I know we were the first people to release an open-source set of differentiable solvers using reverse-mode adjoint sensitivities (the most scalable variant of autodiff for training scalar losses). [https://github.com/rtqichen/torchdiffeq] In the paper we talk a bit about how you can now more easily train Possion process likelihoods, w

Senior author here, I'm happy to answer any questions. We just released source code: https://github.com/rtqichen/torchdiffeq . This includes PyTorch implementations of adaptive ODE solvers that can be differentiated through automatically. So you can mix and match these ODE solvers with any other differentiable model component. There's already been a bit of follow-up work, turning Continuous Normalizing Flows into a practical generative density model: https://arxiv.org/abs/1810.01367 And now we're mainly working on 1) Regularizing ODE nets to be faster to solve and 2) getting the time-series model to scale up and extend it to stochastic differential equations.

The paper the article is based on is here: https://arxiv.org/abs/1806.07366 And there was mention on HN about a month ago of a PyTorch implementation for ODEs (the name for this new NN design) here: https://github.com/rtqichen/torchdiffeq/blob/master/README.m...