FRfragmede·Aug 18, 2024
for data collection, veneur is pretty nice, and is open source and vendor agnostic. by stripe.
https://github.com/stripe/veneur
In Ask HN: What do you monitor on your servers?stripe
veneur
Go2017–2024
stripe / veneur · Archived
veneur
A distributed, fault-tolerant pipeline for observability data
8Distinct people
10Mentions
0Explicit recommenders
1.7KGitHub stars
What to know before exploring
What it is
A distributed, fault-tolerant pipeline for observability data
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year10 dated mentions
Observed mentions: 2017-06-26 15:54:56 UTC to 2024-08-18 13:34:12 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 3 | 2 | 1 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 2 | 0 | 2 |
What Hacker News says
10 sourcesIn terms of Datadog - the per host pricing on infrastructure in a k8/microservices world is perhaps the most egregious of pricing models across all datadog services. Triply true if you use spot instances for short lived workloads.
For folks running k8 at any sort of scale, I generally recommend aggregating metrics BEFORE sending them to datadog, either on a per deployment or per cluster level. Individual host metrics tend to also matter less once you have a large fleet.
You can use opensource tools like veneur (https://github.com/stripe/veneur) to do this. And if you don't want to set this up yourself, third party services like Nimbus (https://nimbus.dev/) can do this for you automatically (note that this is currently a preview feature). Disclaimer also that I'm the founder of Nimbus (we help companies cut datadog costs by over 60%) and h
Read the rest
ave a dog in this fight.
This was the idea behind Stripe's Veneur project - spans, logs, and metrics all in the same format, "automatically" rolling up cardinality as needed - which I thought was cool but also that it would be very hard to get non-SRE developers on board with when I saw a talk about it a few years ago.
https://github.com/stripe/veneur
In OpenTelemetry in 2023One pain point with Prometheus is that is has relatively weak support for quantiles, histograms, and sets[1]:
- Histograms require manually specifying the distribution of your data, which is time-consuming, lossy, and can introduce significant error bands around your quantile estimates.
- Quantiles calculated via the Prometheus "summary" feature are specific to a given host, and not aggregatable, which is almost never what you want (you normally want to see e.g. the 95th percentile value of request latency for all servers of a given type, or all servers within a region). Quantiles can be calculated from histograms instead, but that requires a well-specified histogram and can be expensive at query time.
- As far as I know, Prometheus doesn't have any explicit support for unique sets. You can compute this at query time, but persisting and t
Read the rest
hen querying high-cardinality data in this way is expensive.
Understanding the distribution of your data (rather than just averages) is arguably the most important feature you want from a monitoring dashboard, so the weak support for quantiles is very limiting.
Veneur[2] addresses these use-cases for applications that use DogStatsD[3] by using clever data structures for approximate histograms[4] and approximate sets[5], but I believe its integration with Prometheus is limited and currently only one-way - there is a CLI app to poll Prometheus metrics and push them into Veneur[6], but there's no output sink for Veneur to write to Prometheus (or expose metrics for a Prometheus instance to poll), and you aren't able to use the approximate histogram or approximate set datatypes if you go that route, because they can't be expressed as Prometheus metrics.
It would be extremely useful to have something similar for Prometheus, either by integrating with Veneur or implementing those data structures as an extension to Prometheus.
[1] https://prometheus.io/docs/practices/histograms/
[2] https://github.com/stripe/veneur
[3] https://docs.datadoghq.com/developers/dogstatsd/
[4] https://github.com/stripe/veneur#approximate-histograms
[5] https://github.com/stripe/veneur#approximate-sets
[6] https://github.com/stripe/veneur/tree/master/cmd/veneur-prom...
Veneur[1] can compute approximate global percentiles (among other things) during metric collection, and the percentiles can be stored and queried in even in a datastore that doesn't know anything about distributions.
[1] https://github.com/stripe/veneur
In MetricsDB: TimeSeries Database for storing metrics at TwitterSeems similar to Veneur (like many other projects mentioned in comments here; didn't realize this space was so crowded!) - down to the first two letters of the name: https://github.com/stripe/veneur
Veneur is more metrics-focused, but might offer inspiration as you work on metrics support in Vector - in particular the SSF source, internal aggregation, and Datadog and SignalFX sinks.
In Show HN: Vector – A High-Performance Log and Metric Router Written in Rust> It's from 2016
2016 was the first public release, but the project has grown a lot in that time. You can take a look at the changelog to see what's new, ever since we switched to a six-week release cycle last year: https://github.com/stripe/veneur/blob/master/CHANGELOG.md
Source: I work on the Observability team at Stripe and I am the PM for Veneur.
In Stripe’s Veneur: A distributed, fault-tolerant pipeline for observability dataStripe’s Veneur: A distributed, fault-tolerant pipeline for observability data
109 points · Open discussionThis is awesome! We make heavy use of HyperLogLogs in our monitoring systems[0], which are also written in Go and currently use Clark Duvall's library, which this library is based on.
I'm excited to try this out on our systems and see what results we get.
[0] https://github.com/stripe/veneur
In Show HN: An improved version of HyperLogLog