DeepSpeed

DeepSpeed [1] is amazing tool to enable the different kind of parallelisms and optimizations on your model. I would definitely not recommend reimplementing everything yourself. Probably FairScale [2] too, but never tried it myself. [1]: https://github.com/microsoft/DeepSpeed [2]: https://github.com/facebookresearch/fairscale

This is a very interesting claim. I find it credible because it stands to reason that projects like DeepSpeed[1] and TorchScript[2] wouldn't need to exist if inference performance of research PyTorch models was satisfactory for production, but often case it isn't. It appears as though Flashlight is built on ArrayFire. I haven't seen how gradients are managed in arrayfire-ml, but perhaps it is the case that the autograd implementation in PyTorch was a bottleneck and this is a ground up approach. Editing as I didn't address your second point. I can neither confirm or deny the motivations for creating Torch being related to FB's desire to depend on an Alphabet-managed codebase. I know there are lots of reasons why programmers prefer the UX of the PyTorch Python API (I do as well), and there are probably other reasons I can't recall off the t

Hi! I’m the one who wrote this code. My ZeRO-3 implementation is currently not working, but I’ve spoken with DeepSpeed devs and they’ve explained to me what I’ve been doing wrong. I haven’t had time to implement the fix but I don’t see any reason to assume it won’t work. https://github.com/microsoft/DeepSpeed/issues/846 Also, the specific problem described in that Issue was due to a bug I found in DeepSpeed that has since been corrected.