traceml

Hi, I am building TraceML (open-source) focused on one practical outcome: Automatically identify why a PyTorch training run is slow and suggest what to change. Instead of raw traces or yet another dashboard, the goal is: - highlight the top bottleneck(s) (data stalls, GPU waiting, comm/sync, memory/alloc issues) - keep it lightweight enough to run regularly, not only during deep debugging I would love honest feedback from people who run real training workloads: Would you use something like this, or does it inevitably become another dashboard to maintain? What would make it clearly useful vs noise? Repo: https://github.com/traceopt-ai/traceml Happy to clarify anything.

Hi, I keep running into a frustrating issue with PyTorch DDP on a single node (2–8 GPUs): adding GPUs sometimes makes training slower (not scale proportionally), and it is hard to tell what's actually gating the step. In practice I see: one rank silently becomes the “worst rank” and gates every step step time spikes where GPUs look idle, but it’s unclear if the culprit is dataloader stalls, CPU contention, batch/sequence-length imbalance, or NCCL sync Questions for folks who run multi-GPU training: What do you suspect first when scaling regresses on a single node? What signals do you look at to distinguish data vs compute vs comms/sync? Any repeatable workflow / checklist that gets you to root cause fast? Context: I am building a small OSS tool that shows live per-rank step timing + stall attribution (always-on; not a replacement for PyTo

I built a small tool that shows live, step-by-step: step time (CUDA events, no forced GPU sync) dataloader time GPU memory (incl. reserved vs allocated) The goal is to make it easy to correlate stalls ↔ memory pressure ↔ compute time while the run is happening (instead of only post-hoc profiling). Usage is intentionally minimal: 3 hooks (a context manager + a function + a decorator). Output can be viewed as a local dashboard / CLI / notebook. Repo: https://github.com/traceopt-ai/traceml/ Feedback I would love: What’s the one signal you always wish you had when debugging slow training? What should this integrate with (or avoid) to stay low-overhead?