ml-by-hand

i created something similar a while back. Inspired by micrograd for showing the connection between math/calculus and code, then building along the way to a full NumPy deep learning library that I pretrained GPT-2 124M model with it. One way to learn is to trace through the PRs merged to the repo in chronological order. The philosophy is the same: "What I cannot create, I do not understand." https://github.com/workofart/ml-by-hand

Interesting project. Do you think manual memory management help understand computational graph lifecycle better, or does it distract from backprop itself? btw, I went down the micrograd path with numpy-primitives all the way to building a PyTorch clone that can pre-train and post-train LLMs (https://github.com/workofart/ml-by-hand). My learning focus was on the math/calculus <-> high-level APIs, instead of efficiency. I'm glad to see more people tackling this problem from different angles.

I'm continuing to expand my own deep learning library [1] built with numpy-primitives to support LLM post-training techniques like supervised fine-tuning (SFT) and reinforcement learning with GRPO. It's a good learning experience to work without all the high-level abstractions to "build a wheel" and "use that wheel to build a car". I'm also looking into coding harness self-improvement [2]. An inner LLM (raw LLM request) + harness solves coding tasks, an outer agent like Claude or Codex that proposes harness changes. I experimented with many things in the past few months that made me realize this self-improvement thing that everyone is talking about is just an experiment design problem. I wrote about it here [3]. I'm continuing to improve the infra around the self-improvement loop, to increase signal-to-noise ratio per experiment. I'm also

I'm continuing to expand my own deep learning library [1] (PyTorch-clone built with Python and Numpy) to support LLM post-training techniques like supervised fine-tuning (SFT) [2] and reinforcement learning with GRPO [3] . It's a good learning experience to work without all the high-level abstractions to "build a wheel" and "use that wheel to build a car". Post-training results are still cooking, since training on my MacBookPro is quite slow with "unoptimized PyTorch" :) 1. https://github.com/workofart/ml-by-hand 2. https://github.com/workofart/ml-by-hand/blob/main/examples/s... 3. https://github.com/workofart/ml-by-hand/blob/main/examples/g...

Good introduction! Building pytorch-lite using python and numpy is the way to go. Free book: https://zekcrates.quarto.pub/deep-learning-library/ Ml by hand : https://github.com/workofart/ml-by-hand Micrograd: https://github.com/karpathy/micrograd

Thanks for sharing! It's inspiring to see more people "reinventing for insight" in the age of AI. This reminds me of my similar previous project a year ago when I built an entire PyTorch-style machine learning library [1] from scratch, using nothing but Python and NumPy. I started with a tiny autograd engine, then gradually created layer modules, optimizers, data loaders etc... I simply wanted to learn machine learning from first principles. Along the way I attempted to reproduce classical convnets [2] all the way to a toy GPT-2 [3] using the library I built. It definitely helped me understand how machine learning worked underneath the hood without all the fancy abstractions that PyTorch/TensorFlow provides. I eventually wrote a blog post [4] of this journey. [1] https://github.com/workofart/ml-by-hand [2] https://github.com/workofart/ml-

Love the educational value of this "nano-sized" project. This reminded me of the from-scratch project I created to learn about deep learning libraries, neural networks all the way to LLMs like GPT-2 using just Numpy and Python [1]. Learning is done by "re-inventing the wheel" yourself, one step at a time :) [1] https://github.com/workofart/ml-by-hand

Thanks for this inspiring essay, I couldn’t agree more that “reinventing for insight” is one of the best ways to learn. I had a similar experience couple months ago when I built an entire PyTorch-style machine learning library [1] from scratch, using nothing but Python and NumPy. I started with a tiny autograd engine, then gradually created layer modules, optimizers, data loaders etc... I simply wanted to learn machine learning from first principles. Along the way I attempted to reproduce classical convnets [2] all the way to a toy GPT-2 [3] using the library I built. It definitely helped me understand how machine learning worked underneath the hood without all the fancy abstractions that PyTorch/TensorFlow provides. Kinda like reinventing the car using the wheel I reinvented :) [1] https://github.com/workofart/ml-by-hand [2] https://gith

I built a machine learning library [1] (similar to PyTorch's API) entirely from scratch using only Python and NumPy. It was inspired by Andrej Karpathy's Micrograd project [2]. I slowly added more functionality and evolved it into a fully functional ML library that can build and train classical CNNs [3] to even a toy GPT-2 [4]. I wanted to understand how models learn, like literally bridging the gap between mathematical formulas and high-level API calls. I feel like, as a beginner in machine learning, it's important to strip away the abstractions and understand how these libraries work from the ground up before leveraging these "high-level" libraries such as PyTorch and Tensorflow. Oh I also wrote a blog post [5] on the journey. [1] https://github.com/workofart/ml-by-hand [2] https://github.com/karpathy/micrograd [3] https://github.com/wo

Hi HN, I built a machine learning library entirely from scratch using only Python and NumPy. I then used it to train a range of models—from classical CNNs, ResNets, RNNs, and LSTMs to modern Transformers and even a toy GPT-2. The motivation came from my curiosity about how to build deep learning models from scratch, like literally from mathematical formulas. I built this project not to replace production-ready libraries like PyTorch or TensorFlow, but to strip away the abstractions and reveal the underlying mathematics of machine learning. Key points: - Everything is derived in code — no opaque black boxes. - API mirrors PyTorch so you can pick it up quickly. - You can train CNNs, RNNs, Transformers, and even GPT models. - Designed more for learning/debugging than raw performance. What’s different here? While there are many powerful ML li

Hi HN, I built a machine learning library entirely from scratch using only Python and NumPy. I then used it to train a range of models—from classical CNNs, ResNets, RNNs, and LSTMs to modern Transformers and even a toy GPT-2. The motivation came from my curiosity about how to build deep learning models from scratch, like literally from mathematical formulas. I built this project not to replace production-ready libraries like PyTorch or TensorFlow, but to strip away the abstractions and reveal the underlying mathematics of machine learning. Key points: - Everything is derived in code — no opaque black boxes. - API mirrors PyTorch so you can pick it up quickly. - You can train CNNs, RNNs, Transformers, and even GPT models. - Designed more for learning/debugging than raw performance. What’s different here? While there are many powerful ML li