diffrax

Did you use https://github.com/patrick-kidger/diffrax ? JAX Talk: Diffrax https://www.youtube.com/watch?v=Jy5Jw8hNiAQ

I wrote a JAX-based neural network library (Equinox [1]) and numerical differential equation solving library (Diffrax [2]). At the time I was just exploring some new research ideas in numerics -- and frankly, procrastinating from writing up my PhD thesis! But then one of the teams at Google starting using them, so they offered me a job to keep developing them for their needs. Plus I'd get to work in biotech, which was a big interest of mine. This was a clear dream job offer, so I accepted. Since then both have grown steadily in popularity (~2.6k GitHub stars) and now see pretty widespread use! I've since started writing several other JAX libraries and we've turned this into a bit of a foundation for a JAX sciML ecosystem. [1] https://github.com/patrick-kidger/equinox [2] https://github.com/patrick-kidger/diffrax

Hi everyone! I wanted to advertise my new JAX optimisation library Optimistix! Optimistix has high-level APIs for minimisation, least-squares, root-finding, and fixed-point iteration and was written to take care of these kinds of subroutines in Diffrax (https://github.com/patrick-kidger/diffrax). Here is the GitHub: https://github.com/patrick-kidger/optimistix The elevator pitch is Optimistix is really fast, especially to compile. It plays nicely with Optax for first-order gradient-based methods, and takes a lot of design inspiration from Equinox (https://github.com/patrick-kidger/equinox), representing the state of all the solvers as standard JAX PyTrees. For those familiar with classical nonlinear unconstrained optimisation, Optimistix does some pretty nifty new things. It introduces new abstractions for modular optimisers, allowing us

At least prior to this announcement: JAX was much faster than PyTorch for differentiable physics. (Better JIT compiler; reduced Python-level overhead.) E.g for numerical ODE simulation, I've found that Diffrax (https://github.com/patrick-kidger/diffrax) is ~100 times faster than torchdiffeq on the forward pass. The backward pass is much closer, and for this Diffrax is about 1.5 times faster. It remains to be seen how PyTorch 2.0 will compare, of course! Right now my job is actually building out the scientific computing ecosystem in JAX, so feel free to ping me with any other questions.

+1 for JAX. Basically designed to be the successor to TensorFlow, and much nicer to work with. Strangely I've not seen it discussed around HN much but it's what I do 100% of my work in these days. Whilst I'm here: shameless self-promotion for Equinox and Diffrax: https://github.com/patrick-kidger/equinox https://github.com/patrick-kidger/diffrax Which are neural network and differential equation libraries for JAX. [Obligatory I-am-a-googler-my-opinions-do-not-represent-your-employer...]

Autodifferentiable programming! Neural networks are the famous example of this, of course -- but this can be extended to all of scientific computing. ODE/SDE solvers, root-finding algorithms, LQP, molecular dynamics, ... These days I'm doing all my work in JAX. (E.g. see Equinox or Diffrax: https://github.com/patrick-kidger/equinox, https://github.com/patrick-kidger/diffrax). A lot of modern work is now based around hybridising such techniques with neural networks. I'd really encourage anyone interested to learn how JAX works under-the-hood as well. (Look up "autodidax") Lots of clever/novel ideas in its design.

You may already know of it, but if you want differential-equations-in-JAX then allow me to quickly advertise Diffrax: https://github.com/patrick-kidger/diffrax (of which I am the author, disclaimer).

Taking a union of differential equations is a pretty bad idea for exactly the reasons you describe. But it's absolutely possible to parallelise multiple diffeq solves without needing to use the same time steps for each solve. That is, the time steps can be vectorised, rather than broadcast, over the batch axis. So the only thing you actually need are the same number of steps for each solve, which can be easily accomplished by just padding out the solves that finish with slightly fewer steps. In practice this ends up introducing negligible overhead whilst solving the above issue very neatly. For example this is precisely what Diffrax (https://github.com/patrick-kidger/diffrax) does under `jax.vmap`. I've not dug into what Julia does here; is this not already done when broadcasting `DifferentialEquations.solve`?