SmallPebble

Incorrect Pytorch gradients with Apple MPS backend... Yep this kind of thing can happen. I found and reported incorrect gradients for Apple's Metal-backed tensorflow conv2d in 2021 [1]. (Pretty sure I've seen incorrect gradients with another Pytorch backend, but that was a few years ago and I don't seem to have raised an issue to refer to... ) One might think this class of errors would be caught by a test suite. Autodiff can be tested quite comprehensively against numerical differentiation [2]. (Although this example is from a much simpler lib than Pytorch, so I could be missing something.) [1] https://github.com/apple/tensorflow_macos/issues/230 [2] https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92...

So, they are included in the normal backpropagation of the model. But there is no one-hot encoding, because, although you are correct that it is equivalent, it would be very inefficient to do it that way. You can make indexing differentiable, i.e. gradient descent flows back to the vectors that were selected, which is more efficient than a one-hot matmul. (If you're curious about the details, there's an example of making indexing differentiable in my minimal deep learning library here: https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92...)

You can implement autograd as a library. Just take a look at this https://github.com/sradc/SmallPebble The first line of the description is: > SmallPebble is a minimal automatic differentiation and deep learning library written from scratch in Python, using NumPy/CuPy. The author of this library wrote a blog post, which is in my opinion the best introduction to reverse mode automatic differentiation that exists: https://sidsite.com/posts/autodiff/

Thanks for this. My approach to speeding up an autodiff system like this was to write it in terms of nd-arrays rather than scalars, using numpy/cupy [1]. But it's still slower than deep learning frameworks that compile / fuse operations. Wondering how it compares to the approach in this post. (I might try to benchmark.) [1] https://github.com/sradc/SmallPebble

Here's what you do: you watch this video of Andrew Karpathy [1] called "Becoming a backprop ninja". Then you pick up a function that you like and implement this backprop (which is a different way of saying reverse mode automatic differentiation) using just numpy. If you use some numpy broadcasting, an np.sum, some for-loops, you'll start getting a good feel for what's going on. Then you can go and read this fabulous blog post [2], and if you like what you see, you go to the framework built by its author, called Small Pebble [3]. Despite the name, it's not all that small. If you peruse the code you'll get some appreciation of what it takes to build a solid autodiff library, and if push comes to shove, you'll be able to build one yourself. [1] https://www.youtube.com/watch?v=q8SA3rM6ckI [2] https://sidsite.com/posts/autodiff/ [3] https://gi