cudf

mPy/SciPy-compatible array library for GPU-accelerated computing with Python. CuPy acts as a drop-in replacement to run existing NumPy/SciPy code on NVIDIA CUDA or AMD ROCm platforms. https://cupy.dev/ : > CuPy is an open-source array library for GPU-accelerated computing with Python. CuPy utilizes CUDA Toolkit libraries including cuBLAS, cuRAND, cuSOLVER, cuSPARSE, cuFFT, cuDNN and NCCL to make full use of the GPU architecture. > The figure shows CuPy speedup over NumPy. Most operations perform well on a GPU using CuPy out of the box. CuPy speeds up some operations more than 100X. Read the original benchmark article Single-GPU CuPy Speedups on the RAPIDS AI Medium blog CuDF: https://github.com/rapidsai/cudf CuML: https://github.com/rapidsai/cuml : > cuML is a suite of libraries that implement machine learning algorithms and mathematical primitives functions that share compatible APIs with other RAPIDS projects.* > cuML enables data scientists, researchers, and software engineers to run traditional tabular ML tasks on GPUs without going into the details of CUDA programming. In most cases, cuML's Python API matches the API from scikit-learn. > For large datasets, these GPU-based implementations can complete 10-50x faster than their CPU equivalents. For details on performance, see the cuML Benchmarks Notebook. FWICS there's now a ROCm version of CuPy, so it says CUDA (NVIDIA only) but also compiles for AMD. IDK whether there are plans to support Intel OneAPI, too. What of the non-Arrow parts of other pandas-compatible and not pandas-compatible DataFrame libraries can be ported back to Pandas (and R)?

gregating, filtering, and otherwise manipulating data. > cuDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of CUDA programming. Dask-ML makes scalable scikit-learn, XGBoost, TensorFlow really easy. https://dask-ml.readthedocs.io/en/latest/ ... re: the OT: While it's possible to write C++ code that's really fast, it's generally inflexible, expensive to develop, and dangerous for devs with experience in their respective domains of experience to write. Much saner to put a Python API on top and optimize that during compilation. There are a few C++ frameworks in the top quartile of the TechEmpower framework benchmarks. https://www.techempower.com/benchmarks/ Hardware/hosting is relatively cheap. Developers and memory vulnerabilities aren't.

eral standard from DMLC that any similar ML library can adopt. Also, if you're doing nvJpeg decompression on GPU, also check out Dali, which implements super-fast data loading and augmentation steps all on GPU. For this one, there wasn't an existing standard, so we had to build our own approach, but it integrates with PyTorch, TF, and MXNet pipelines nicely: https://docs.nvidia.com/deeplearning/sdk/dali-developer-guid... Hopefully it becomes like the cuDNN of data prep - integrated into all the frameworks well enough that you don't even have to think about it.

chine learning jobs with zero copy ipc and get the results back the same way. We are all about interoperability with the rapidsai eco system.