INingve·Dec 21, 2022Submission
Guide to Go profiling, tracing and observability
175 points · Open discussionDataDog
go-profiler-notes
Jupyter Notebook2021–2022
DataDog / go-profiler-notes
go-profiler-notes
felixge's notes on the various go profiling methods that are available.
5Distinct people
6Mentions
0Explicit recommenders
3.7KGitHub stars
What to know before exploring
What it is
felixge's notes on the various go profiling methods that are available.
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by month6 dated mentions
Observed mentions: 2021-05-11 16:49:44 UTC to 2022-12-21 19:34:20 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| May 2021 | 2021-05-01 00:00:00 UTC | 2021-06-01 00:00:00 UTC | 2 | 1 | 1 |
| Jun 2021 | 2021-06-01 00:00:00 UTC | 2021-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2021 | 2021-07-01 00:00:00 UTC | 2021-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2021 | 2021-08-01 00:00:00 UTC | 2021-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2021 | 2021-09-01 00:00:00 UTC | 2021-10-01 00:00:00 UTC | 2 | 2 | 0 |
| Oct 2021 | 2021-10-01 00:00:00 UTC | 2021-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2021 | 2021-11-01 00:00:00 UTC | 2021-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2021 | 2021-12-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 2022 | 2022-01-01 00:00:00 UTC | 2022-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2022 | 2022-02-01 00:00:00 UTC | 2022-03-01 00:00:00 UTC | 0 | 0 | 0 |
| Mar 2022 | 2022-03-01 00:00:00 UTC | 2022-04-01 00:00:00 UTC | 0 | 0 | 0 |
| Apr 2022 | 2022-04-01 00:00:00 UTC | 2022-05-01 00:00:00 UTC | 0 | 0 | 0 |
| May 2022 | 2022-05-01 00:00:00 UTC | 2022-06-01 00:00:00 UTC | 0 | 0 | 0 |
| Jun 2022 | 2022-06-01 00:00:00 UTC | 2022-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2022 | 2022-07-01 00:00:00 UTC | 2022-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2022 | 2022-08-01 00:00:00 UTC | 2022-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2022 | 2022-09-01 00:00:00 UTC | 2022-10-01 00:00:00 UTC | 0 | 0 | 0 |
| Oct 2022 | 2022-10-01 00:00:00 UTC | 2022-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2022 | 2022-11-01 00:00:00 UTC | 2022-12-01 00:00:00 UTC | 1 | 0 | 1 |
| Dec 2022 | 2022-12-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
6 sourcesThanks!
As @brancz mentioned, Delve uses DWARF unwind information to produce backtraces (they are stored in the .debug_frame section for Go).
You are right, Go enabled frame pointers for all architectures as of 1.17 [0]. This is enabled to allow profilers to work well, without having to use to techniques such as the one we describe in our post.
When it gets funny is that there's `gopclntab`, a 3rd option in Go to unwind stacks, used by `panic` and I believe other parts of the runtime. If you are interested in more details, Felix Geisendörfer's repo contains way more details [1]
[0]: https://go.dev/doc/go1.17
[1]: https://github.com/DataDog/go-profiler-notes/blob/main/stack...
In DWARF-Based Stack Walking Using eBPFThe Busy Developers's Guide to Go Profiling, Tracing and Observability
23 points · Open discussionThe Busy Developers's Guide to Go Profiling, Tracing and Observability
1 points · Open discussionI think this is great work. Hopefully this can motivate upstream to reconsider the proposal for it. In the long run I don't think anybody wants to run with a forked version of Go for better profiling.
> The ability to monitor go programs with a very high sampling frequency — up to 10s of microseconds
I'm curious about the overhead of this and will probably try to measure. This work certainly overcomes the limitations of setitimer(2) for sampling rate, but faster sampling is going to increase the performance overhead significantly.
In particular Go's built-in stack unwinding APIs use gopclntab (which is Go's Plan9 inspired version of eh_frame). This is rather slow compared to frame pointer unwinding (up to 55x in my testing). I think on average you can expect to spend at least 1 microsecond on unwinding, and then some additional time hashi
Read the rest
ng the stack and updating the in-memory profile (perhaps another microsecond? I haven't measured this part yet). I've started writing some of this up here [1] but it's still WIP.
Anyway, based on the above, sampling every 10usec will likely slow the target program by ~20%. This is probably a very optimistic estimate because it doesn't take into account all the CPU cache thrashing that will happen.
As I said, I'll try to take a closer look later, but if anybody has some insights/comments here I'd be very interested.
[1] https://github.com/DataDog/go-profiler-notes/blob/main/stack...
Disclaimer: I work at Datadog on Continuous Go Profiling, but I'm here speaking for myself.