pytorch-image-models

Those are some pointless tests. E.g. test_activation_stats_functions [1] that just checks that the returned value is a float, and that it can take random numbers as input. test_get_state_dict_custom_unwrap [2] is probably supposed to check that custom_unwrap is invoked, but since it doesn't either record being called, or transform its input, the assertions can't actually check that it was called. [1] https://github.com/huggingface/pytorch-image-models/pull/233... [2] https://github.com/huggingface/pytorch-image-models/pull/233...

Logged into my personal account for this one! I'm a lead author on a paper that explored exactly. It does enable faster training and smaller model sizes. For reference, you can get 80% accuracy on CIFAR-10 in ~30 minutes of CPU (not using crazy optimizations). There are open questions about scaling but at the time we did not have access to big compute (really still don't) and our goals were focused on addressing the original ViT's claims of data constraints and necessities of pretraining for smaller datasets (spoiler, augmentation + overlapping patches plays a huge role). Basically we wanted to make a network that allowed people to train transformers from scratch for their data projects because pretrained models aren't always the best solutions or practical. Paper: https://arxiv.org/abs/2104.05704 Blog: https://medium.com/pytorch/training

This is still being pursued. Ross Wightmann's timm[0,1] package (now on Hugging Face) has done a lot of this. There's also a V2 of ConvNext[2]. Ross does write about this a lot on Twitter fwiw. I should also mention that there are still many transformer based networks that still beat convs. So there probably won't be a resurgence in convs until someone can show that there's a really strong reason for them. They have some advantages but they also might not be flexible enough for the long range tasks in segmentation and detection. But maybe they are. FAIR definitely did great work with ConvNext, and I do hope to see more. There always needs to be people pushing unpopular paradigms. [0] https://github.com/huggingface/pytorch-image-models [1] https://arxiv.org/abs/2110.00476 [2] https://arxiv.org/abs/2301.00808