llm.c

results of the paper only accounted for 5 of those. That's 53 of the 1876 training runs. Your $1k doesn't get you nearly as far as you'd think. If we simplify things and say everything in that 5 V100 years cost $1k then that means they spent $85k before that. They spent $18k before they even went ahead with that project. If you want realistic numbers, multiply that by 5 because that's roughly what a V100 will run you (discounted for scale). ~$110k ain't too bad, but that is outside the budget of most small labs (including most of academia). And remember, that's just the cost of the GPUs, that doesn't pay for any of the people running that stuff. I don't expect you to know any of this stuff if you're not a researcher. Why would you? It's hard enough to keep up with the general AI trends, let alone niche topics lol. It's not an intelligence problem, it's a logistics problem, right? A researcher's day job is being in those weeds. You just get a lot more hours in the space. I mean I'm pretty out of touch of plenty of domains just because time constraints. > I don't think this is feasible for large investments. I may be wrong, but i also don't think other avenues aren't being funded. So I'm trying to say, I think your bar has been met. And I think if we are actually looking at the numbers, yeah, I do not think these avenues are being funded. But don't take it from me, take it from FeiFei Li[3] | Not a single university today can train a ChatGPT model I'm not sure if you're a researcher or not, you haven't answered that question. But I think if you were you'd be aware of this issue because you'd be living with it. If you were a PhD student you would see the massive imbalance of GPU resources given to those working closely with big tech vs those trying to do things on their own. If you were a researcher you'd also know that even inside those companies that there aren't much resources given to people to do these things. You get them on occasion like the StarFlow and TarFlow I pointed out before, but these tend to be pretty sporadic. Even a big reason we talk about Mamba is because of how much they spent on it. But if you aren't a researcher I'd ask why you have such confidence that these things are being funded and that these things cannot be scaled or improved[4]. History is riddled with examples of inferior tech winning mostly due to marketing. I know we get hyped around new tech, hell, that's why I'm a researcher. But isn't that hype a reason we should try to address this fundamental problem? Because the hype is about the advance of technology, right? I really don't think it is about the advancement of a specific team, so if we have the opportunity for greater and faster advancement, isn't that something we should encourage? Because I don't understand why you're arguing against that. An exciting thing of working at the bleeding edge is seeing all the possibilities. But a disheartening thing about working at the bleeding edge is seeing many promising avenues be passed by for things like funding and publicity. Do we want meritocracy to win out or the dollar? I guess you'll have to ask yourself: what's driving your excitement? [0] I mean the first Mamba paper, not the first SSM paper btw: https://arxiv.org/abs/2312.00752 [1] https://github.com/karpathy/llm.c/discussions/677 [2] https://arxiv.org/abs/2106.12423 [3] https://www.ft.com/content/d5f91c27-3be8-454a-bea5-bb8ff2a85... [4] I'm not saying any of this stuff is straight de fact better. But there definitely is an attention imbalance and you have to compare like to like. If you get to x in 1000 man hours and someone else gets there in 100, it may be worth taking a look deeper. That's all.

c and made a podcast about that: https://notebooklm.google.com/notebook/2585c187-b059-475a-b4... I don't know if I'd exactly agree with the framing of the conversation and the emphasis or the descriptions of layernorm and matmul etc but there's hints of greatness here and in any case it's highly entertaining. Imo LLM capability (IQ, but also memory (context length), multimodal, etc.) is getting way ahead of the UIUX of packaging it into products. Think Code Interpreter, Claude Artifacts, Cursor/Replit, NotebookLM, etc. I expect (and look forward to) a lot more and different paradigms of interaction than just chat. That's what I think is ultimately so compelling about the 2-person podcast format as a UIUX exploration. It lifts two major "barriers to enjoyment" of LLMs. 1 Chat is hard. You don't know what to say or ask. In the 2-person podcast format, the question asking is also delegated to an AI so you get a lot more chill experience instead of being a synchronous constraint in the generating process. 2 Reading is hard and it's much easier to just lean back and listen.

e for anyone who wants to: a. Learn the principles behind quadrotor autonomy. Code robotics in Rust. b. Explore the challenges of using Rust before incorporating it into their projects. Throughout the development of this project, I have strived to work entirely within the Rust ecosystem. I am now convinced that Rust is ready for serious robotics development. Here are some tools I've been using: 1. [zed.dev](https://yangrobotics.com) for code editing. 2. [Rerun](rerun.io) for visualization. 3. [tokei](https://github.com/XAMPPRocky/tokei) for code line statistics. 4. [nalgebra](https://nalgebra.org) for matrix operations. There are still a few things I'm searching for, such as a robust optimization library. Currently, I'm experimenting with [OpEn](https://alphaville.github.io/optimization-engine) for nonlinear model predictive control, but I hope to find something akin to [acados](https://github.com/acados) in Rust. Additionally, while nalgebra is solid, I believe it still doesn't outperform [Eigen](https://eigen.tuxfamily.org) in terms of performance, though Rust's language features might limit the possibility of implementing something like Eigen. I'm open to suggestions and feedback! For instance, I’m still debating whether to modularize the pipeline and add async features, or whether to keep things simpler to avoid overwhelming beginner learners. Current code statistics: Rust: 1263 Markdown: 305 Comments: 19 Blanks: 44 PS: When naming the project "Peng," I had no idea that "peng" also means attractive! Well, I hope you find the project attractive too.