pyroscope

I wonder how this compares to grafana pyroscope, which is really good for this sort of thing and already quite mature: https://grafana.com/oss/pyroscope/ https://github.com/grafana/pyroscope

Great question! Perforator indeed looks similar to Pyroscope. However, we think that the closest existing solutions are https://parca.dev, closed-source Google Wide Profiling, and, speaking of the agent, the beautiful OpenTelemetry eBPF profiler. The main technical differences with Pyroscope we see are: - Pyroscope's Java support is superior as of now because Pyroscope offloads it to the amazing async-profiler. - Pyroscope expects native binaries to be compiled with frame pointers: https://grafana.com/docs/pyroscope/latest/configure-client/g.... This is often not the case, and that's the problem we've tried to solve with Perforator. Perforator uses .eh_frame, which is nearly universal and does not impose additional requirements on compiled binaries. - Pyroscope symbolizes using symtab: https://grafana.com/docs/pyroscope/latest/configure-c

I'm curious about the differences from Pyroscope. https://github.com/grafana/pyroscope

Since it hasn't been mentioned in this entire thread, frame pointers are required to get good high resolution flame graphs. https://www.brendangregg.com/flamegraphs.html With systems like Phlare/Pyroscope SRE can monitor application performance in a very granular way in realtime. https://grafana.com/blog/2023/03/15/pyroscope-grafana-phlare... https://github.com/grafana/pyroscope

Pyroscope's license has already been converted from Apache to AGPL v3. Two commits on LICENSE, second was today a647634d6c05db86cd6b066d31323456528f9bf0: https://github.com/grafana/pyroscope/blob/a647634d6c05db86cd... Previous: https://github.com/grafana/pyroscope/commit/a539a5ff69a4a390...

Tries (or prefix trees). We use them a lot at Pyroscope for compressing strings that have common prefixes. They are also used in databases (e.g indexes in Mongo) or file formats (e.g debug symbols in macOS/iOS Mach-O format are compressed using tries). We have an article with some animations that illustrate the concept in case anyone's interested [0]. [0] https://github.com/pyroscope-io/pyroscope/blob/main/docs/sto...

Semantic pull requests [0] + conventional-changelog [1] + squashing PRs and you get almost the same results (and more) with great automation and self-documentation for the whole process: * you'll never forget to add something to changelog * you get links to PRs in your changelog * it's much faster to make edits to PR messages compared to editing files * outside contributors get familiarized with the practice much faster (imagine getting new contributors to update changelogs) We recently adopted this practice at Pyroscope and it's been working out pretty well for us [2], I can certainly recommend it. [0] - https://github.com/apps/semantic-pull-requests [1] - https://github.com/conventional-changelog/conventional-chang... [2] - https://github.com/pyroscope-io/pyroscope/blob/main/CHANGELO...

Tries are also used to efficiently store names of functions in macOS / iOS binaries (Mach-O format) and work great for that purpose because a lot of functions have the same prefixes [0]. Indexes in mongodb use a similar concept (they call that prefix compression [1]), and that allows them to store indexes more efficiently. We use them at Pyroscope and we wrote a blog post about our storage design [2] There are even some animations :) [0] https://www.m4b.io/reverse/engineering/mach/binaries/2015/03... [1] https://docs.mongodb.com/manual/reference/glossary/#std-term... [2] https://github.com/pyroscope-io/pyroscope/blob/main/docs/sto...

Pyroscope (YC W21) | Senior Software Engineer | REMOTE | Full-time / Contract Pyroscope is open source performance monitoring software, currently with a focus on Continuous Profiling. Pyroscope was founded at the beginning of 2021 and we've already seen a lot of rapid growth/community adoption as well as feedback for what features developers are excited for us to build. We're looking for Senior Software Engineers for both frontend and backend. Our frontend uses React and Redux and our custom storage engine is written in Golang (with agents for many other languages as well) Nice-to-haves: * Interest in / experience with Open Source * Performance analysis experience * SRE experience * Experience with low-level Linux matters * Solid CS knowledge, particularly understanding of data structures (we use trees and tries extensively see https://gi

Pyroscope (YC W21) | Senior Software Engineer | REMOTE | Full-time / Contract Pyroscope is open source performance monitoring software, currently with a focus on Continuous Profiling. Pyroscope was founded at the beginning of 2021 and we've already seen a lot of rapid growth/community adoption as well as feedback for what features developers are excited for us to build. We're looking for Senior Software Engineers for both frontend and backend. Our frontend uses React and Redux and our custom storage engine is written in Golang (with agents for many other languages as well) Nice-to-haves: * Interest in / experience with Open Source * Performance analysis experience * SRE experience * Solid CS knowledge, particularly understanding of data structures (we use trees and tries extensively see https://github.com/pyroscope-io/pyroscope/blob/main/

Pyroscope (YC W21) | Senior Software Engineer | REMOTE | Fulltime / Contract Pyroscope is open source performance monitoring software, currently with a focus on Continuous Profiling. Pyroscope was founded at the beginning of 2021 and we've already seen a lot of rapid growth/community adoption as well as feedback for what features developers are excited for us to build. We're looking for Senior Software Engineers for both frontend and backend. Our frontend uses React and Redux and our custom storage engine is written in Golang (with agents for many other languages as well) Nice-to-haves: * Interest in / experience with Open Source * Performance analysis experience * SRE experience * Solid CS knowledge, particularly understanding of data structures (we use trees and tries extensively see https://github.com/pyroscope-io/pyroscope/blob/main/d

Pyroscope (YC W21) | Senior Software Engineer | REMOTE | Full-time/contractors | Pyroscope is open source performance monitoring software, currently with a focus on Continuous Profiling. Pyroscope was founded at the beginning of 2021 and we've already seen a lot of rapid growth/community adoption as well as feedback for what features developers are excited for us to build. We're hiring Senior Software Engineers with Golang experience. Bonus points if you have: * performance analysis experience * SRE experience * database architecture / maintenance experience * solid CS knowledge, particularly understanding of data structures (we use trees and tries extensively see https://github.com/pyroscope-io/pyroscope/blob/main/docs/sto...) Senior Software Engineer: https://apply.workable.com/pyroscope/j/28E1725C98/ Questions?: hn@pyroscope.io

This is really cool stuff, we're at Pyroscope [1] would love to add support for it. Too bad it's a fork of Go runtime. As @akshayshah mentioned, it seems like Go team is hesitant about straight up merging this. I wonder if a better approach would be to open up Go runtime apis so that this can be a module instead of a whole fork. I think opening up these APIs would also enable more innovation in profiling space. [1]: https://github.com/pyroscope-io/pyroscope

The costs of observability/monitoring are obviously primarily the cost of the labor to add/upkeep these tools and the standard costs of running them. While the costs are more clearly defined, the benefits of these tools is much less clear. I'm currently building an open source continuous profiling library (https://github.com/pyroscope-io/pyroscope) and trying to understand more about how people justify adding another observability/monitoring tool to their workflow. For some it seems like there is some sort of quantitative cost-to-benefit analysis where ultimately a tool that produces net benefit should be added to the system. For others, it seems the justification is less quantifiable in that some companies just err on the side of anything that produces a "better" understanding of underlying infrastructure should be added to the system. H