datasets

I tried Librispeech, a very common dataset for speech recognition, in both HF and TFDS. TFDS performed extremely bad. First it failed because the official hosting server only allows 5 simultaneous connections, and TFDS totally ignored that and makes up to 50 simultaneous downloads and that breaks. I wonder if anyone actually tested this? Then you need to have some computer with 30GB to do the preparation, which might fail on your computer. This is where I stopped. https://github.com/tensorflow/datasets/issues/3887. It might be fixed now but it took them 8 months to respond to my issue. On HF, it just worked. There was a smaller issue in how the dataset was split up but that is fixed now, and their response was very fast and great.

I was looking for a Python package to manage science datasets, mainly description, download, extraction, on-disk handling, etc... ML frameworks tend to roll their own: - https://github.com/nilearn/nilearn/blob/master/nilearn/datasets/utils.py - https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/downlo… - https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/base.py - https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py I was hoping that somebody had made a generic package for this purpose so that I don't roll yet another one.