LLMsPracticalGuide

Not a complete answer, but here are the most helpful resources for understanding transformer basics in particular: Original transformer paper: https://arxiv.org/abs/1706.03762 Illustrated transformer: http://jalammar.github.io/illustrated-transformer/ Transformer visualization: https://bbycroft.net/llm minGPT (Karpathy): https://github.com/karpathy/minGPT --- Next, some foundational textbooks for general ML and deep learning: Elements of Statistical Learning (aka the bible): https://hastie.su.domains/ElemStatLearn/ Probabilistic ML: https://probml.github.io/pml-book/book2.html Deep Learning Book (Goodfellow/Bengio): https://www.deeplearningbook.org/ Understanding Deep Learning: https://udlbook.github.io/udlbook/ --- Finally, assorted tutorials/resources/intro courses: Beyond the Illustrated Transformer: https://news.ycombinator.com/item?

Here are some high level answers: "7B" refers to the number of parameters or weights for a model. For a specific model, the versions with more parameters take more compute power to train and perform better. A foundational model is the part of a ML model that is "pretrained" on a massive data set (and usually is the bulk of the compute cost). This is usually considered the "raw" model after which it is fine-tuned for specific tasks (turned into a chatbot). "8K length" refers to the Context Window length (in tokens). This is basically an LLM's short term memory - you can think of it as its attention span and what it can generate reasonable output for. "1.5T tokens" refers to the size of the corpus of the training set. In general Wikipedia (or I suppose ChatGPT 4/Bing Chat with Web Browsing) is a decent enough place to start reading/asking b