pcm

Last time I checked, Intel's MSRs (https://en.wikipedia.org/wiki/Model-specific_register) allow Intel PCM (https://github.com/intel/pcm) to work, are indeed used to profile, or "measure performance" (sorry if my vocabulary is not the most accurate). Last time I checked the code of Intel PCM, it still relies on hardcoded values for each CPU which are as close as possible to reality but are still an estimation. It doesn't mean that you get wrong measurements, it means there's a level of inaccuracy that has to be accepted. BTW, I am aware that Intel PCM is not a profiler, and more of a measurement tool, however you CAN you use it to 'profile' your program and see how it behaves in terms of computing and memory utilization (with deep analysis of cache behavior (cache hit, cache miss, etc.))

Intel has a Performance Monitoring Unit on its core that has significant overlap. Forgetting this tool-space, but at least some of these tools can make use of that hardware: https://github.com/intel/pcm https://github.com/andikleen/pmu-tools

Long running hosts at the moment, but we can drain most workloads off a specific host/rack if required and reschedule it pretty fast. We have the advantage of having a custom scheduler/orchestrator we've been working on for years, so we have a lot of control on that layer than with Kube or Nomad. Re: Live Migration We're working on adding Live Migration support to our orchestrator atm. We aim to have it running this quarter. That'll makes things super seamless. Re: kernels We've already seen some perf improvements somewhere between 6.0 and 6.5 (I forget the exact reason/version) - but it was some fix specific to the Sapphire Rapids cpus we had. But I wish we had more time to science on it, it's really fun playing with all the knobs and benchmarking stuff. Some of the telemetry on the new CPUs is also crazy - there's stuff like Intel PCM t

It depends. On AWS you can get "metal" instances where afaik you get pretty much unrestricted access. In addition on certain instance types/sizes you get access to virtualized counters (vPMU). See Q11 here https://github.com/intel/pcm/blob/master/doc/FAQ.md#q11 or tables here https://www.intel.com/content/www/us/en/developer/articles/t... dunno about others

Since kernel 3.3 or so, RAPL is also exposed through `/sys/devices/virtual/powercap/intel-rapl/*/energy_uj` in micro-joules (if not, `modprobe intel_rapl`). So if you want to do a quick power measurement, it can be done using just POSIX sh (root required): # in milli-watt (1000 = 1W) because shell arithmetic doesn't do floating point while true; do LAST_MJ=$MJ MJ=$(cat /sys/devices/virtual/powercap/intel-rapl/intel-rapl:0/energy_uj) echo $(((MJ - LAST_MJ) / 1000)) sleep 1 done Despite the powercap name being intel-rapl, the powercap interface is also available on AMD machines. For a more detailed reading on several more metrics about the CPU, I think pcm[1] may be a better tool (it's a successor to the Intel Power Gadget the project was forked from). Though, it only works on Intel CPU. [1]: ht

I don't know how often it's a problem, but I work for a company doing software video encoding, and we always fill up all the dimm slots on servers to have as much bandwidth as possible, even if we have only really use maybe 1/4 of the RAM. I'm not sure any of the standard Linux tools can show you memory bandwidth usage easily (maybe perf), I know we use Intel PCM (https://github.com/intel/pcm) and AMDuProfPCM (https://www.amd.com/en/developer/uprof.html)

The Intel PCM suite does what you seem to be describing w/o a kernel module, and it uses nicely abstracted MSR accesses so that its portable to any OS. https://github.com/intel/pcm

If you're a low level hacker a reading knowledge of these events is useful but really we should be using VTune as our PME tool. Still, it's possible that a particular event may shed light on a particular piece of code and using an API like PCM. https://github.com/opcm/pcm

It would be interesting to see exactly how much data is transferred across the bus during training. Of course it would be great if you could fit your whole dataset on the other side but typically a GPU during training will max out at batches of anywhere from 8 to 64 depending on image size and number of channels. So you'll be moving quite a bit of data. If found a tool to monitor this unfortunately it only works on Xeons and not on what's in my desktop. https://github.com/opcm/pcm

PCM is similar and is userspace. However on OSX, you need a driver and then various other hoops must be jumped in order to enable it. On Linux, I think it just runs out of the box. But in any case, PCM itself is userspace. https://github.com/opcm/pcm