curated-transformers

Nice! For what it is worth, a colleague and I made a library a while ago that factors out most shared model code, with which many models can be implemented in about 100 lines (excluding Python import ceremony and comments). E.g.: BERT: https://github.com/explosion/curated-transformers/blob/main/... Llama 1/2: https://github.com/explosion/curated-transformers/blob/main/... MPT: https://github.com/explosion/curated-transformers/blob/main/... With various stuff enabled, including support for TorchScript JIT, PyTorch flash attention, etc.

I am so happy about them releasing this. A few years ago I wrote a multi-task syntax annotator in Rust using Laurent Mazare's excellent Torch binding (tch-rs, it looks like he is also working on Candle): https://github.com/tensordot/syntaxdot However, the deployment story was always quite difficult. The PyTorch C++ API is not stable, so a particular version of tch-rs will only work with a particular PyTorch version. So, anyone wanting to use SyntaxDot always had to get exactly the right version of libtorch (and set some environment variables) to build the project. The idea of making an abstraction over Torch and Rust ndarray (similar to Burn) crossed my mind several times, but there is only so much that I could do as a solo developer. So Candle would be a god-given if I was still working on this project. Seeing Candle wants to make me por

Yeah. In curated transformers [1] we are seeing completely deterministic output across multiple popular transformer architectures on a single GPU (there can be variance between GPUs due to different kernels). Of course, it completely depends on what ops and implementations you are using. But most transformers do not use ops that are typically non-deterministic to be fast (like scatter-add). One non-determinism we see with a temperature of 0 is that once you have quantized weights, many predicted pieces will have the same probability, including multiple pieces with the highest probability. And then the sampler (if you are not using a greedy decoder) will sample from those pieces. So, generation is non-deterministic with a temperature of 0. In other words, a temperature of 0 is a poor man’s greedy decoding. (It is totally possible that Open

This is a Python library that Madeesh Kannan and I have developed. It started out as a small transformer library to use in spaCy pipelines. The last two months we have extended it to an independent PyTorch library and added support for modern LLMs such as Falcon, LLaMA, and Dolly V2. Our focus has been on making a library of reusable components, so that new transformer architectures can be added with as little code as possible. We will release a semver-stable 1.0 release over the coming weeks.

There are no big differences compared to other LLM architecturally. The largest differences compared to NeoX are: no biases in linear layers, shared heads for the key and value representations (but not query). Of course, it has 40B parameters, but there is also a 7B parameter version. The primary issue is that the current upstream version (on Huggingface) hasn't implemented key-value caching correctly. KV caching is needed to bring the complexity down from O(n^3) to O(n^2). The issues are: (1) their implementation uses Torch' scaled dot-product attention, which uses incorrect causal masks when the query/key sizes are not the same (which it the case when generating with a cache). (2) They don't index the rotary embeddings correctly when using key-value cache, so the rotary embedding of the first token is used for all generated tokens. Toge