t5x

underdeveloped (for instance, there's no way to specify non-default weight initialization strategies without manually doing model surgery to set weights). - MaxText was just very difficult to work with. We felt like we were fighting against it every time we needed to change something because we would be digging through numerous needless layers of abstraction. My favorite was after one long day of debugging, I found a function who's only purpose was to pass its arguments to another function untouched; this function's only purpose was to pass its arguments untouched to a new, third function, that then slightly changed them and passed them to a fourth function that did the work. - EasyLM is, as the name says, easy. But on a deeper dive, the sharding functionality seems to be underdeveloped. What they call "FSDP" is not necessarily true FSDP, it's literally just a certain axis that the JAX mesh is being sharded around that happens to shard some data axes and some model weight axes. I'm still searching for a "perfect" JAX LLM codebase - any pointers?

etter let them take the call. A lot of fluff, a lot of chatter online, so everyone is figuring out stuff.