deeplake

er version, that was much slower than what we have now, see this: https://pasteboard.co/la3DmCUR2iFb.png). HF under the hood uses Git-LFS (to the best of my knowledge) and is not opinionated on formats, so LAION just dumps Parquet files on their storage. While your setup would work for a few TBs, scaling to PB would be tricky including maintaining your own infrastructure. And yep, as you said NAS/NFS would neither be able to handle the scale (especially writes with 1k workers). I am also slightly curious about your use of mmap files with image/video compressed data (as zero-copy won’t happen) unless you decompress inside the GPU ;), but would love to learn more from you! Re: pricing thanks for the feedback, storage is one component and customly priced for PB-scale workloads.

nd (c) deep learning frameworks, letting you do AI magic on your data while fully utilizing your GPUs. Here’s a 10-minute demo: https://www.youtube.com/watch?v=SxsofpSIw3k&t. Back in 2016, I started my Ph.D. research in Deep Learning and witnessed the transition from GBs to TBs, then petabyte datasets. To run our models at scale, we needed to rethink how we handled data. One of the ways we optimized our workflows included streaming the data, while asynchronously running the computation on GPUs. This served as an inspiration for creating Activeloop. When you want to use unstructured data for deep learning purposes, you’ll encounter the following options: - Storing metadata (pointers to the unstructured data) in a regular database, and images in object storage. It is inefficient to query the metadata table and then fetch images from object storages for high-throughput workloads. - Store images inside a database. This typically explodes the memory cost and will cost you money. For example, storing images in MongoDB and using them to train a model would cost 20x more than a Deep Lake setup [2]. - Extend Parquet or Arrow to store images. On the plus side, you can now use existing analytical tools such as Spark, Kafka, and even DuckDB. But even major self-driving car companies failed on this path. - Build custom infrastructure aligned with your data in-house. Assuming you have the money and access to 10 solid data engineers with PhD-level knowledge, this still takes time (~2.5+ years), is difficult to extend beyond the initial vertical, will be hard to maintain, and will defocus your data scientists. Whatever the case, you'll get slow iteration cycles, under-utilized GPUs, and lots of ML engineer busywork (thus high costs). Your unstructured data already sits in a data lake such as S3 or a distributed file system (e.g., Lustre) and you probably don’t want to change this. Deep Lake keeps everything that a regular data lake makes great. It helps you version-control, run SQL queries, ingest billion-row data efficiently, and visualize terabyte-scale datasets in your browser or notebook. But there is one key difference from traditional data lakes: we store complex data, such as images, audio, videos, annotations/labels, and tabular data, in a tensorial form that is optimized for deep learning and GPU utilization. Some stats/benchmarks since our launch: - In a third-party benchmark by Yale University [3], Deep Lake provided the fastest data loader for PyTorch, especially when it comes to networked loading; -Deep Lake handles scale and long distance: we trained a 1B parameter CLIP model on 16xA100 GPUs on the same machine on LAION-400M dataset, streaming the data from US-EAST (AWS) to US-CENTRAL (GCP) [4] [5]; - You can access datasets as large as 200M samples of image-text pairs) in seconds (as compared to the 100+ hours it takes via traditional methods) with one line of code. [6] What's free and what's not: the data format, the Python dataloader, version control, and data lineage (a log of how the data came to its current state) with the Python API are open-source [7]. The query language, fast streaming, and visualization engines are built in C++ and are closed-source for the time being, but are accessible via a Python interface. Users can store up to 300GB of their data with us for free. Our growth plan is $995/month and includes an optimized query engine, the fast data loader, and features like analytics. If you're an academic, you can get this plan for free. Finally, we have an enterprise plan including role-based access control, security, integrations, and more than 10 TB of managed data [8]. Teams at Intel, Google, & MILA use Deep Lake. If you want to read more, we have an enterprise-y whitepaper at https://www.deeplake.ai/whitepaper, an academic paper at https://arxiv.org/abs/2209.10785, and a launch blog post with deep dive into features at https://www.activeloop.ai/resources/introducing-deep-lake-th.... I would love to hear your thoughts on this, especially anything about how you manage your deep learning data and what issues you run in with your infra. I look forward to all your comments. Thanks a lot! [1] https://www.activeloop.ai/resources/introducing-deep-lake-th... [2] https://imgur.com/a/AZtWSkA [3] https://arxiv.org/abs/2209.13705 [4] https://imgur.com/a/POtHklM [5] https://github.com/activeloopai/deeplake-laion-clip [6] https://datasets.activeloop.ai/docs/ml/datasets/coco-dataset... [7] https://github.com/activeloopai/deeplake [8] https://app.activeloop.ai/pricing

tore them on my machine (my datasets can be up to 10GB+ big, but it works just as well with 100GB+ datasets like ImageNet (https://docs.activeloop.ai/datasets/imagenet-dataset), for instance). Hub allows us to store images, audio, video data in a way that can be accessed at lightning speed. The data can be stored on GCS/S3 buckets, local storage, or on Activeloop cloud. The data can directly be used in the training TensorFlow/ PyTorch models so that you don't need to set up data pipelines. The package also comes with data version control, dataset search queries, and distributed workloads. For me, personally the simplicity of the API stands out, for instance: Loading datasets in seconds import hub ds = hub.load("hub://activeloop/cifar10-train") Lets you access and train a model on CIFAR 10 dataset in seconds (look for all the datasets available here https://docs.activeloop.ai/datasets). Dataset filtering Hub has its own Pythonic query language (see below), or you can define your own functions to filter through data (since it's a multi-dimensional array, it's easy to do so). ds_view = ds.filter("labels == 'automobile' or labels == 'automobile'", scheduler = 'threaded', num_workers = 0) Dataset Version Control Dataset Version Control All things you may already know from git, e.g commit, log, branch, diff, checkout also work for Hub using similar commands. (https://docs.activeloop.ai/getting-started/dataset-version-control) ----------------------------------------------------------------------------------------------------------------------------------- My experience with Hub was amazing: I was able to create and push data to the cloud within a couple of minutes. I also put together a tutorial on how to get started with dataset management with Activeloop Hub so you can finish your ML projects faster You will learn: - How to initializing a dataset on the Activeloop Cloud - Processing the images - Pushing the data to the cloud - Data version control - Data visualization Read more here: https://www.kdnuggets.com/2022/03/new-way-managing-deep-learning-datasets.html Activeloop Hub docs: https://docs.activeloop.ai/ Let me know if you have any questions! I just started with Hub quite recently, but might be able to help out. :) @1abidaliawan