opentelemetry-specification

very cool - made me consider the contrasts with token-bucket rate limiting for log collection and stumbled across a interesting discussion https://github.com/open-telemetry/opentelemetry-specificatio...

Well actually. They (python SDK maintainers) argue their implementation is the correct one according to the spec. See this issue thread for example. https://github.com/open-telemetry/opentelemetry-specificatio... There are more. This is a symptom of a how hard it is to dive into Otel due to its surface area being so big.

At the risk of hijacking the comments, I've been trying to use OTel recently to debug performance of a complex webpage with lots of async sibling spans, and finding it very very difficult to identify the critical path / bottlenecks. There's no causal relationships between sibling spans. I think in theory "span links" solves this, but afaict this is not a widely used feature in SDKs are UI viewers. (I wrote about this here https://github.com/open-telemetry/opentelemetry-specificatio...)

Sure, happy to provide more specifics! Our main issue was the lack of a synchronous gauge. The officially supported asynchronous API of registering a callback function to report a gauge metric is very different from how we were doing things before, and would have required lots of refactoring of our code. Instead, we wrote a wrapper that exposes a synchronous-like API: https://gist.github.com/yolken-airplane/027867b753840f7d15d6.... It seems like this is a common feature request across many of the SDKs, and it's in the process of being fixed in some of them (https://github.com/open-telemetry/opentelemetry-specificatio...)? I'm not sure what the plans are for the golang SDK specifically. Another, more minor issue, is the lack of support for "constant" attributes that are applied to all observations of a metric. We use these to identify the

Two problems with OpenTelemetry: 1. It doesn't know what the hell it is. Is it a semantic standard? Is a protocol? It is a facade? It is a library? What layer of abstraction does it provide? Answer: All of the above! All the things! All the layers! 2. No one from OpenTelemetry has actually tried instrumenting a library. And if they have, they haven't the first suggestion on how instrumenters should actually use metrics, traces, and logs. Do you write to all three? To one? I asked this question two years ago, zero answers :( [1] [1] https://github.com/open-telemetry/opentelemetry-specificatio...

That's why you have otel logging: https://github.com/open-telemetry/opentelemetry-specificatio...

There are a large amount of subtle tradeoffs around the bucketing scheme (log, vs. log-linear, base) and memory layout (sparse, dense, chunked) the amount of configurability in the histogram space (circllhist, DDSketch, HDRHistogram, ...). A good overview is this discussion here: https://github.com/open-telemetry/opentelemetry-specificatio... As for the circllhist: There are no knobs to turn. It uses base 10 and two decimal digits of precision. In the last 8 years I have not seen a single use-case in the operational domain where this was not appropriate.

A good place to look at is the milestones on GitHub: https://github.com/open-telemetry/opentelemetry-specificatio... Logging is still experimental in the spec. Metrics API is feature freeze and the protocol is stable, so it's more on language SDKs to stabilize their implementations. This is a focus for several of them right now.

I think you are right and data model should have been event based rather than span based. https://medium.com/opentracing/open-for-event-based-tracing-... digs into that topic and basically says that distributed tracing is about causality & partial ordering which bring us back to the 70's with Lamport's logical clocks and all following work. However, this design "flaw" is well known and seems assumed. I'm not able to find relevant GitHub issues right now, but I remember this topic being discussed on OpenTracing or OpenTelemetry bug tracker and the outcome was something like "Spans might not be the best data model, but people are now used to it and we have to ship the spec within a reasonable time, so let's stick to it". Edit: https://github.com/open-telemetry/opentelemetry-specificatio... might be relevant to your concerns.

OpenMetrics relates to the metrics part of OpenTelemetry, where the latter is currently in alpha/experimental phase. OpenTelemetry has a statement on being interoperable with OpenMetrics for both accepting, forwarding and generating in the OpenMetrics format [0]. So they are two separate CNCF projects where OpenMetrics has overlap with a part of OpenTelemetry. [0] https://github.com/open-telemetry/opentelemetry-specificatio...

For the long running scenario, why not use Links? The specification describes[1] a similar scenario and how links can be leveraged to causally relate spans even across trace boundaries. As for the crashing scenario, this seems like an application-level concern. Ideally it is not waiting until the crash to send traces. Depending on the environment, the application could handle the crash and deliver the telemetry leading up to it before shutting down. [1]: https://github.com/open-telemetry/opentelemetry-specificatio...

Indeed. For those unfamiliar, take a quick read of https://github.com/open-telemetry/opentelemetry-specificatio... There are a few concepts to understand, but instrumenting it right from the start is valuable as it grows into a bigger system. Being a cross-cutting concern, the more code that standardizes on this the more useful it will be.

You can check the first link that states the overview - https://github.com/open-telemetry/opentelemetry-specificatio... It's a distributed request tracing specification & set of libraries implementing the said spec

Here is the draft plan for logs: https://github.com/open-telemetry/opentelemetry-specificatio... Logs are not going to be part of OpenTelemetry 1.0 release (only traces and metrics will). Logs are coming later (no specific timeline yet). Disclaimer: I work on OpenTelemetry spec and wrote most of the linked doc. Comments/issues/PRs welcome in the repo.