k8s-device-plugin

I skip some minor issues I encountered in the articles. If it happens to be the ones I faced, I can share how I solved them.

P4s are relatively cheap and, if you use them for just 10s of hours a month, you can get away with a bill of 10s of dollars [1]. Either option works fine (IMHO). Another option I am unsure about because I never tried it is to use something like [2] to multiplex your GPU(s) so that you can pretend you have more GPUs to mess around with. However, if your goal is to learn how to manage/write software for multi-gpus/multi-machine clusters this is somewhat limiting because 1) it doesn't teach you much about data locality/compact placement since transfers between virtual GPUs will be extremely fast (i.e. they are sharing the same VRAM pool after all) and 2) you will still have a single machine (even if you are using multiple virtual K8s nodes). [1] 3 instances with 4 GPUs each used for 24h in a month cost you 46$ according to: https://cloud.google.com/products/calculator/?hl=en&dl=CiRhY... [2] https://github.com/NVIDIA/k8s-device-plugin

s from other teams as long as they are not using them. https://github.com/nebuly-ai/nos What is your thoughts on this project? :) Nos addresses some key challenges of Kubernetes tied to the fact that Kubernetes was not designed to support GPU and AI / machine learning workloads. In Kubernetes, GPUs are managed with [3] Nvidia k8s Device Plugin, which has a few major downsides. First, it requires the allocation of an integer number of GPUs per workload, not allowing workloads to request only fractions of GPU. Second, when enabling GPU shared access either with time-slicing or MIG, the device plugin advertises to Kubernetes a fixed set of GPU resources that do not dynamically adapt to the requests of the Pods at each time. This often leads to both underutilized GPUs and pending Pods, and/or the cluster admin having to spend a lot of time looking for workarounds to make the best use of GPUs. For example, consider a company with a k8s cluster with 20 GPUs, where 3 of these GPUs have been reserved for the data science team using Resource Quota objects. In most cases, the workloads of data scientists (notebooks, scripts, etc.) require much less memory/compute resources than those of an entire GPU, yet Kubernetes will force each container to consume an entire GPU. Also, if the team once needs to run a heavy workload, it may want to use as many resources as possible. However, the Resource Quota over their namespace would constrain the team to use at most the 3 GPUs reserved for them, even if the company cluster may be full of unused GPUs! Instead, with nos the data science team would use nos Dynamic GPU Partitioning to request GPU slices so that many workloads can share the same GPU. Also, Elastic Resource Quotas would allow the team to consume more than the 3 reserved GPUs, borrowing quotas from other teams that are not using them. To recap, the team would be able to launch more Pods and the company would likely need fewer nodes. All this with minimal effort required by the cluster admin, who only has to set up nos. Let me know what you think of nos, feedback would be very helpful! :) And please leave a star on GitHub if you like this opensource https://github.com/nebuly-ai/nos Here are some other links that may be useful - Tutorial on how to use Dynamic GPU Partitioning with Nvidia MIG https://towardsdatascience.com/dynamic-mig-partitioning-in-k... - Pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS https://docs.nebuly.com/nos/dynamic-gpu-partitioning/partiti... - Nos documentation https://docs.nebuly.com/nos/overview/ References [1] Multi-Process Service (MPS) https://docs.nvidia.com/deploy/mps/index.html [2] Multi-Instance GPU (MIG) https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ [3] NVIDIA k8s Device Plugin https://github.com/NVIDIA/k8s-device-plugin

s from other teams as long as they are not using them. Let me know your thoughts on the project in the comments :) Nos addresses some key challenges of Kubernetes tied to the fact that Kubernetes was not designed to support GPU and AI / machine learning workloads. In Kubernetes, GPUs are managed with [3] Nvidia k8s Device Plugin, which has a few major downsides. First, it requires the allocation of an integer number of GPUs per workload, not allowing workloads to request only fractions of GPU. Second, when enabling GPU shared access either with time-slicing or MIG, the device plugin advertises to Kubernetes a fixed set of GPU resources that do not dynamically adapt to the requests of the Pods at each time. This often leads to both underutilized GPUs and pending Pods, and/or the cluster admin having to spend a lot of time looking for workarounds to make the best use of GPUs. For example, consider a company with a k8s cluster with 20 GPUs, where 3 of these GPUs has been reserved to the data science team using Resource Quota objects. In most cases, the workloads of data scientists (notebooks, scripts, etc.) require much less memory/compute resources than those of an entire GPU, yet Kubernetes will force each container to consume an entire GPU. Also, if the team once needs to run a heavy workload, it may want to use as many resources as possible. However, the Resource Quota over their namespace would constrain the team to use at most the 3 GPUs reserved for them, even if the company cluster may be full of unused GPUs! Instead, with nos the data science team would use nos Dynamic GPU Partitioning to request GPU slices so that many workloads can share the same GPU. Also, Elastic Resource Quotas would allow the team to consume more than the 3 reserved GPUs, borrowing quotas from other teams that are not using them. To recap, the team would be able to launch more Pods and the company would likely need fewer nodes. All this with minimal effort required by the cluster admin, who only has to set up nos. Let me know what you think of nos, feedback would be very helpful! :) And please leave a star on GitHub if you like this opensource https://github.com/nebuly-ai/nos Here are some other links that may be useful - Tutorial on how to use Dynamic GPU Partitioning with Nvidia MIG https://towardsdatascience.com/dynamic-mig-partitioning-in-k... - Pros and cons of different technologies for sharing resources among workloads in Kubernetes: time-slicing, MIG, and MPS https://docs.nebuly.com/nos/dynamic-gpu-partitioning/partiti... - Nos documentation https://docs.nebuly.com/nos/overview/ References [1] Multi-Process Service (MPS) https://docs.nvidia.com/deploy/mps/index.html [2] Multi-Instance GPU (MIG) https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ [3] NVIDIA k8s Device Plugin https://github.com/NVIDIA/k8s-device-plugin