hlb-CIFAR10

ral benchmarks, nearly tripling the speed of a longstanding CIFAR10 world record (~17.1s -> 6.3s, https://github.com/tysam-code/hlb-CIFAR10, recognized by Karpathy: https://x.com/karpathy/status/1620103412686942208), setting a modded-nanogpt speed record and helping with several others (5.03m -> 4.66m, https://github.com/KellerJordan/modded-nanogpt/blob/master/r... ), improving the relative loss of DiLoCo speedrunning on modded-nanogpt by over 40% (https://x.com/hi_tysam/status/1928561266533990587), and miscellaneous other misadventures throughout the years. I’m looking to be challenged in a strong technical environment with other people who I can learn from, and I’m open to work on most interesting problems that I think are neutral or bring something positive to the world. I’m very hard-problem oriented, and am quite skilled at distilling complex, slow, brittle training procedures down to simple, performant, and hackable ones. The kinds of problems that engage me the most are ones where I can always be growing and learning, and that’s the kind of environment that I’ll thrive in. Learning new subdomains (molecular, etc) is something that keeps me driven, I love the process of expanding out my toolkit. I also very much enjoy working on problems well within my domain of expertise. My focus is on data/architecture/training dynamics and debugging for training efficiency more than raw low-level kernel creation, though the two go hand-in-hand. I enjoy mentoring people quite a bit! I am looking to learn working with larger-scale distributed runs, scale is something that I’d like to work with a bit more. If you think this may align with what you’re doing – say hi! I’d love to chat.

status/1620103412686942208), setting a modded-nanogpt speed record (5.03m -> 4.66m, https://github.com/KellerJordan/modded-nanogpt/blob/master/r... ), improving the relative loss of DiLoCo speedrunning on modded-nanogpt by over 40% (https://x.com/hi_tysam/status/1928561266533990587), and miscellaneous other misadventures throughout the years. I’m looking to be challenged in a strong technical environment with other people who I can learn from, and I’m open to work on most interesting problems as long as they’re roughly net neutral or positive. I’m very hard-problem oriented, and am quite skilled at distilling complex, slow, brittle training procedures down to simple, performant, and hackable ones. The kinds of problems that engage me the most are ones where I can always be growing and learning, and that’s the kind of environment that I’ll thrive in. I’m certainly open to learning new subdomains (molecular, etc). I tend to focus more on data/architecture/training dynamics than raw low-level kernel stuff, though the two tend to go hand-in-hand. I have a tendency to enjoy mentoring people, casual mentorship is something that I greatly enjoy. I am looking to learn working with larger-scale distributed runs, scale is something that I’d like to work with a bit more. I’m keeping an eye open for a good opportunity that’s a good mutual fit for both sides, so I will approach most interview processes thoughtfully. If you think you have something that would be a good fit for us both – say hi!

-compact-transformers-fro... CPU compute: https://twitter.com/WaltonStevenj/status/1382045610283397120 Crazy optimizations (no affiliation): 94% on CIFAR-10 in <6.3 seconds on a single A100 : https://github.com/tysam-code/hlb-CIFAR10 I also want to give maybe some better information about ViTs in general. Lucas Beyer is a good source and has some lectures as well as Hila Chefer and Sayak Paul's tutorials. Also, just follow Ross Wightman, the man is a beast Lucas Beyer: https://twitter.com/giffmana/status/1570152923233144832 Chefer & Paul's All Things ViT: https://all-things-vits.github.io/atv/ Ross Wightman : https://twitter.com/wightmanr His very famous timm package https://github.com/huggingface/pytorch-image-models

resource-constrained) research. To do that, you have to maximize information efficiency at every stage of the pipeline, including temporally (!!!!). It's not perfect, but I've used info theory as a strong guiding light for that repo. There's more to say here, but it's a long conversation about the expected utility of doing research a few different kinds of ways.

ined by the governance of the various countries in the world over different economic and/or informational policies. Other than that, the information asymmetry is going to make it a power-bloodbath as we go through our informational-industrial revolution. While I'm here, while I think Hotz does contribute a lot of good to the field, though I do have a bit of an actual minor personal beef with him. He said he was going to reimplement https://github.com/tysam-code/hlb-CIFAR10 in tinygrad, bashed a few parts of the code (my feelings!) for a while on stream, and then gave up a few hours later because of the empirical speed/occupancy numbers. >:( I want my fast reimplementation, George. Just do it.

r more issues. Both of the main repos that I've published in terms of speed benchmarks train directly in pure fp16 and bf16 respectively without any fp32 frippery, if you want to see an example of both paradigms successfully feel free to take a look (I'll note that bf16 is simpler on the whole for a few reasons, generally seamless): https://github.com/tysam-code/hlb-CIFAR10 [for fp16] and https://github.com/tysam-code/hlb-gpt [for bf16] Personally from my experience, I think fp16/bf16 is honestly a bit too expressive for what we need, fp8 seems to do just fine and I think will be quite alright with some accommodations, just as with pure fp16. The what and the how of that is a story for a different day (and at this point, the max pooling operation is basically one of the slowest now). You'll have to excuse my frustration a bit, it just is a bit jarring to see a streetsign from way in the past fly forward in the wind to hit you in the face before tumbling on its merry way. And additionally in the comment section the general discussion doesn't seem to talk about what seems to be a pretty clearly-established consensus in certain research circles. It's not really too much of a debate anymore, it works and we're off to bigger and better problems that I think we should talk about. I guess in one sense it does justify the paper's utility, but also a bit frustrating because it normalizes the conversation as a few notches back from where I personally feel that it actually is at the moment. We've got to move out of the past, this fp32 business to me personally is like writing a Relu-activated VGG network in Keras on Tensorflow. Phew. And while we're at it, if I shall throw my frumpy-grumpy hat right back into the ring, this is an information-theoretic problem! Not enough discussion of Shannon and co. Let's please fix that too. See my other rants for x-references to that, should you be so-inclined to punish yourself in that manner.

g network that infers at 94% in under 1 MB. I certainly believe it's possible, but with pareto laws the way they are, it would take a whole lot longer to train and might not be as fast on a GPU during inference as the main net (despite having fewer parameters). That might not be true, however. There's a few NP-hard problems that actually exist in this space that not a lot of people talk about but I feel will be considered a core part of the theory of training neural networks at some point in the future. The size of the network is a very interesting tradeoff that opens up certain mathematically interesting properties on either end of the spectrum. Bigger is not always better, though it is simpler and simple oftentimes survives, especially (especially especially especially) when skilled technical workers in a very particular niche live in the shallow long tail of [insert correct distribution of skills rarity here]. One of the common threads (might be a "common", I'm not sure to be honest as I live in my own personal bubble of research interests and community and etc) is the dimensionality of the problem at hand. That plays into the scale of the network used to solve a problem. I remember some discussion being sparked a while back from some Uber research about the inherent dimensionality of a particular problem for some given particular neural network (though of course it's naturally linked to your inductive bias so please take that as you will). As you noted, some networks do quite well with very few neurons, 15 is a record however from what I've heard (and I'd love to see that -- I have a guess as to which particular method, or, at least, method family, it is... ;P I'm...casually interested in that arena of research). In any case, as you can see I am quite interested and passionate about this topic and am happy to discuss it at length further.

head. However, both are meant to be simple (and extremely hackable if you want to poke around and take apart some pieces/add different watchpoints to see how different pieces evolve, etc. I am partially inspired by, among many things, one of those see-through engine kits that I saw in a magazine growing up as a child that I thought was a very cool, dynamic, and hands-on way to just watch how the pieces moved in a difficult topic. Sometimes that is the best way that our brains can learn, though we are all different and learn best differently through different mediums in my experience). Feel free to let me know if you have any specific questions and I'll endeavor to do my best to help you here. Welcome to an interest in the field! I guess to briefly touch on one topic -- some people focus on the technical only first, like backprop, and though math is required heavily for more advanced research, I don't learn concepts very well through details only. Knowing that backprop is "Calculate the slope for the error in this high-dimensional space for how a neural network was wrong at a certain point, then take a tiny step towards minimizing the error. After N steps, we converge to a representation that is like a zip file of our input data within a mathematical function" is probably enough for 90-95% of the usecases you will do as a ML practitioner, if you do so. The math is cool but there are more important things to sweat over IMO, and I think messaging to the contrary raises the barrier to entry to the field and distracts from the important things, which we do not need as much. It's good to learn after you have space in your brain for it after you understand how the whole thing works together, though that is just my personal opinion after all. Much love and care and all that and again feel free to let me know if you have any questions please. :) <3

e that is a bit simpler (at least in layout) and faster than before. We also eliminate a hyperparameter that seems to be no longer useful. One downside of these changes is that we do overfit slightly more on longer runs, but that can be mitigated enough with cutout, and we hope to fix this in future releases, as this is not a terrible problem to have when trying to set speed records. We test our code on CIFAR100 without any modifications (other than to the dataloaders to load the correct number of data and the correct number of classes) and show that performance for those two different network sizes is comparable. To do this, we show (at least in rough initial explorations), that both of the small networks matched the performance of SOTA networks in around the same year, and that increasing the base depth of the network by a factor of two improved the performance of both networks by about a year to match respective SOTAs from the same time period. This indicates that this code (hopefully) has some good generalization capabilities beyond just this dataset, though we have not experimented with different image sizes yet (it's rather expensive and the information might get stale very quickly!) There's a lot more in here, but as in previous posts, the mantra of sorts is 'doing the basics and doing them very well'. This goes a whole lot further than it might seem otherwise, having 'the new shiny' when developing neural networks is oftentimes more of a toy and a distraction than sticking with the basics and doing them well. Which is understandably a very difficult thing. That said, as we run out of runway for the 'easier' changes, we will likely need to get more and more creative. But until then, the goal is to stay as simple as possible! If you'd like more info, please do read the release notes, they are very informative although longer. Future releases could have more to do with speed improvements or other things. Additionally, this still should be an excellent researcher's toolbench for prototyping and experimenting with ideas. Many ideas I've been able to implement in 5 minutes or less, most of which are actually running in the code by that point. For some, like sometimes architecture changes, for example, I'm able to get a complete initial go/no go filter answer oftentimes within 1-2 minutes of the idea. I just quickly tweak it, let it run through a few runs, and either have to let it run more runs to see if it is just a noisy answer or if it definitely doesn't work. This is indispensable and part of why I built this tool. It's also partially responsible for the rapid progress in developing this tool -- I'm able to apply the very-rapidly-gained insights from this tool to itself. I'll be hanging around here for questions/comments/etc. I can't answer all of them, but I'll do the best that I can! :D

is not too terribly difficult, I believe. Effectively, the loss function compresses the task we've described with our labels from our training dataset into our neural network. This includes (ideally, at least), 'all' the information the neural network needs to perform that task well, according to the data we have, at least. If you'd like to know more about the specifics of this, I'd refer you to the original Shannon-Weaver paper on information theory -- Weaver's introduction to the topic is in plain English and accessible to (I believe) nearly anyone off of the street with enough time and energy to think through and parse some of the concepts. Very good stuff! An initial read-through should take no more than half an hour to an hour or so, and should change the way you think about the world if you've not been introduced to the topic before. You can read a scan of the book at a university hosted link here: https://raley.english.ucsb.edu/wp-content/Engl800/Shannon-We... Using some of the concepts of Shannon's theory, we can see that anything that minimizes an information-theoretic loss function should indeed learn as well those prerequisites to the task at hand (features that identify xyz, features that move information about xyz from place A to B in the neural network, etc). In this case, even though it appears we do not have labels -- we certainly do! We are training on predicting the _next words_ in a sequence, and so thus by consequence humans have already created a very, _very_ richly labeled dataset for free! In this way, getting the data is much easier and the bar to entry for high performance for a neural network is very low -- especially if we want to pivot and 'fine-tune' to other tasks. This is because...to learn the task of predicting the next word, we have to learn tons of other sub-tasks inside of the neural network which overlap with the tasks that we want to perform. And because of the nature of spoken/written language -- to truly perform incredibly well, sometimes we have to learn all of these alternative tasks well enough that little-to-no-finetuning on human-labeled data for this 'secondary' task (for example, question answering) is required! Very cool stuff. This is a very rough introduction, I have not condensed it as much as it could be and certainly, some of the words are more than they should be. But it's an internet comment so this is probably the most I should put into it for now. I hope this helps set you forward a bit on your journey of neural network explanation! :D :D <3 <3 :)))))))))) :fireworks: For reference, I'm interested very much in what I refer to as Kolmogorov-minimal explanations (Wikipedia 'Kolmogorov complexity' once you chew through some of that paper if you're interested! I am still very much a student of it, but it is a fun explanation). In fact (though this repo performs several functions), I made https://github.com/tysam-code/hlb-CIFAR10 as beginner-friendly as possible. One does have to make some decisions to keep verbosity down, and I assume a very basic understand of what's happening in neural networks here too. I have yet to find a good go-to explanation of neural networks as a conceptual intro (I started with Hinton -- love the man but extremely mathematically technical for foundation! D:). Karpathy might have a really good one, I think I saw a zero-to-hero course from him a little while back that seemed really good. Andrej (practically) got me into deep learning via some of his earlier work, and I really love basically everything that I've seen the man put out. I skimmed the first video of his from this series and it seems pretty darn good, I trust his content. You should take a look! (Github and first video: https://github.com/karpathy/nn-zero-to-hero, https://youtu.be/VMj-3S1tku0) For reference, he is the person that's made a lot of cool things recently, including his own minimal GPT (https://github.com/karpathy/minGPT), and the much smaller version of it (https://github.com/karpathy/nanoGPT). But of course, since we are in this blog post I would refer you to this 60 line numpy GPT first (A. to keep us on track, B. because I skimmed it and it seemed very helpful! I'd recommend taking a look at outside sources if you're feeling particularly voracious in expanding your knowledge here.) I hope this helps give you a solid introduction to the basics of this concept, and/or for anyone else reading this, feel free to let me know if you have any technically (or-otherwise) appropriate questions here, many thanks and much love! <3 <3 <3 <3 :DDDDDDDD :)))))))) :)))) :))))