server

one of the many tricks it uses for very performant inference - even across multiple chassis over another network layer. [0] - https://github.com/triton-inference-server/server

or the GPU processing part of your model, and then automatically batch inputs into this process, etc. etc. (and people do) but all this is just to work around the lack of proper threading support in Python. By the way, if anyone is struggling with these challenges today, I recommend taking a peek at nvidia's Triton inference server (https://github.com/triton-inference-server/server), which handles a lot of these details for you. It supports things like zero-copy sharing of tensors between parts of your model running in different processes/threads and does auto-batching between requests as well. Especially auto-batching gave us big throughput increase with a minor latency penalty!

back whatever the model output is. So now everything else in your architecture other than Triton doesn't even know what a GPU or ML is. It also makes deploying new models, versions, whatever much simpler - you POST them to Triton (or it loads them from S3, local disk, whatever) and they're instantly available everywhere. Not a sales pitch for Triton but it (like some others) can also do things like dynamic batching with QoS parameters, automated model profiling and performance optimization[3], really granular control over resources, response caching, python middleware for application/biz logic, accelerated media processing with Nvidia DALI, memory management and control, all kinds of stuff. [0] - https://github.com/kserve/kserve [1] - https://github.com/triton-inference-server/server [2] - https://github.com/triton-inference-server/client [3] - https://github.com/triton-inference-server/model_analyzer

of this mention is that they utilize a shared memory architecture where the ARM CPU and CUDA cores share memory. Not only does this cut down on hardware costs and power usage, it increases performance. - Nvidia has acquired Mellanox for high performance network I/O across various technologies (Ethernet and Infiniband). Nvidia is also actively working to be able to remove the host CPU from as many GPGPU tasks as possible (network I/O and data storage): https://developer.nvidia.com/gpudirect - Nvidia already has publicly available software in place to effectively make their CUDA compute available over the network using various APIs: https://github.com/triton-inference-server/server Going on just the name Triton is currently only available for inference but it provides the ability to not only directly serve GPGPU resources via network API at scale but ALSO accelerate various models with TensorRT optimization: https://docs.nvidia.com/deeplearning/triton-inference-server... Given these points I think this is an obvious move for Nvidia. TDP and performance is increasingly important across all of their target markets. They already have something in place for edge inference tasks powered by ARM with Jetson but looking at ARM core CPU benchmarks it's sub-optimal. Why continue to pay ARM licensing fees when you can buy the company, collect licensing fees, get talent, and (presumably) drastically improve performance and TDP for your edge GPGPU hardware? In the cloud/datacenter, why continue to give up watts in terms of TDP and performance to sub-optimal Intel/AMP/x86_64 CPUs and their required baggage (motherboard bridges, buses, system RAM, etc) when all you really want to do is shuffle data between your GPUs, network, and storage as quickly and efficiently as possible? Of course many applications will still require a somewhat general purpose CPU for various tasks, customer code, etc. AWS already has their own optimized ARM cores in place. aarch64 is more and more becoming a first class citizen across the entire open source ecosystem. As platform and software as a service continues to eat the world cloud providers likely have already started migrating the underlying hardware powering these various services to ARM cores for improved performance and TDP (same product, more margin). Various ARM cores are already showing to be quite capable for most CPU tasks but given the other architectural components in place here even the lowliest of modern ARM cores is likely to be asleep most of the time for the applications Nvidia currently cares about. Giving up licensing, die space, power, tighter integration, etc to x86_64 just seems to be foolish at this point. Meanwhile (of course) if you still need x86_64 (or any other arch) for whatever reason you can hit a network API powered by hardware using Nvidia/Mellanox I/O, GPU, and ARM. Potentially (eventually) completely transparently using standard CUDA libraries and existing frameworks (see work like Apex): https://github.com/NVIDIA/apex I, for one, am excited to see what comes from this.