gigapipe

Thank you for that. I absolutely love that this uses tantivy. I was previously leaning torwards VictoriaMetrics and VictoriaTraces (I will need both) but I think that OpenObserve is even simpler. Later I found Gigapipe/qryn https://github.com/metrico/gigapipe Does OpenObserve ships something to view traces and metrics? (it appears that Gigapipe does). Or am I supposed to just use Grafana? I want to cut down on moving pieces.

I'm not the person you asked -- and I also want to be transparent that I only PoC-ed it and due to external circumstances didn't get it all the way out to production -- but I really like how https://github.com/metrico/qryn (AGPLv3) thinks about the world. It is, like SigNoz, unified (logs, metrics, traces) but it actually implements several of the common endpoint schemes allowing it to pretend to be "your favorite tool" which plausibly helps any integration story <https://github.com/metrico/qryn#%EF%B8%8F-query> and <https://github.com/metrico/qryn#-vendors-compatibility>. Had I gotten it up and running, I was going to contribute a Splunk ingest adapter since that was what we were trying to replace I was going to take advantage of Clickhouse using S3 as warm-to-cold storage since my mental model is that most logs, metrics, and traces are

Would be interesting to hear if qryn (née cloki - clickhouse Loki) with vector might be a valid alternative to both Loki and traditional syslog. AFAIK logshipping with journald is bespoke - but vector should be able to handle both syslog and journald - for relatively painless setup ... Not sure if tailing is better with qryn though. https://github.com/metrico/qryn https://vector.dev/docs/reference/configuration/sources/sysl... https://vector.dev/docs/reference/configuration/sources/jour... https://www.digitalocean.com/community/tutorials/how-to-cent...

Isn't qryn (nee cLoki) just as good as rolling your own? https://github.com/metrico/qryn Ed: given: "... for two years now" - I think clog/qryn was a bit rougher around the edges at the time - so my question is more for today - would qryn do 80% to 200% of your home grown solution today, with 10%-50% of the effort?

How does it compare with loki/qryn("née cloki/clickhouse loki)? https://grafana.com/oss/loki/ https://github.com/metrico/qryn Aside - with so much to choose from wrt logs and metrics - I'd assume "prometheus for logs" implied "pulling" log data (rather than push). While maybe "Victoria metrics for logs" might imply "easier to operate, efficient, fewer moving parts".

cloki can be used to read metrics out of any CH table so it should work fine. we also just introduced experimental support for ingesting OTLP/ZIPKIN spans and a tempo-compatible API in cloki, looking for testers to validate this feature: https://github.com/lmangani/cLoki/wiki/Tempo-Tracing#clickho... Internally trace spans are stored as tagged JSON logs, meaning they are available from both Loki and Tempo APIs and can be used from pretty much any visualization, too!

There's also cLoki. It's a new project that puts a Loki gateway over a ClickHouse backend store. We're looking at it and plan a presentation from the author(s) at the next ClickHouse SF Bay Area Meetup. https://github.com/lmangani/cLoki

I just there was a foss loki-like solution built on ch - that was stable and used in production. I know there's a few projects (see below) - but I'm not aware of anything mature.. https://github.com/QXIP/cloki-go https://github.com/lmangani/cloki

While logs and metrics have common parts, they are quite different beasts. It is quite expensive producing and storing logs for frequently changing metrics such as requests rate or response latency on highly loaded system with millions of rps. I'd recommend: - Storing logs in high-performance systems such as cLoki [1], which give enough flexibility to generate arbitrary metrics from high log volumes in real time. - Storing metrics in high-performance time series database optimized for high volumes of data points (trillions) and high number of time series (aka high cardinality). An example of such a TSDB is VictoriaMetrics [2]. Implementing dashboards with time-based correlation between metrics and logs based on shared attributes (labels) such as datacenter, cluster, instance, job, subsystem, etc [3]. [1] https://github.com/lmangani/cLoki