cortex

the AGPLv3 license, and hold themselves to the same obligations that others are held to with regard to providing corresponding source code of derivative works when it is made available to users over a network. In my personal opinion, too often companies use a contributor agreement that excuses them from those obligations, and also allows them to sell the software to others under licenses that do not carry copyleft obligations. See [2] for a blog post that goes into some detail about this. If you look at the Coretex project MAINTAINERS file [3], you will see that there are two folks listed that currently work at AWS, but no other company other than Grafana Labs today. I would love to see more diversity in maintainers for a project like this, as I think too many maintainers from any one company isn't the best for long term project sustainability. I think if you look at the Cortex Community Meeting minutes [4], you can see that AWS folks are regularly "showing up" in healthy numbers, and working collaboratively with anyone who accepts the open invitation to participate. There have been some pretty big improvements to Coretex that have merged lately, like some of the work on parallel compaction [5, 6]. TL;DR, I think it is easy to jump to some conclusions about how things are going in a FOSS project that don't hold water if you do some cursory exploration. I think best way to know what's going on in a project is to get involved! -- [1] the rights needed to: run the program for any purpose; to study how the program works, and modify it; to redistribute copies; to distribute copies of modified versions to others [2] https://meshedinsights.com/2021/06/14/legally-ignoring-the-l... [3] https://github.com/cortexproject/cortex/blob/master/MAINTAIN... [4] https://docs.google.com/document/d/1shtXSAqp3t7fiC-9uZcKkq3m... [5] https://aws.amazon.com/blogs/opensource/scaling-cortex-with-... [6] https://github.com/cortexproject/cortex/pull/4624

along: using aggregates (such as mean, standard deviation, standard error, quantiles, ...) implies dropping information. Going from the full distribution to a simplified representation naturally implies that what we talk about is a lossy transformation of data. Of course, one wants to be smart about _which_ information to drop. It should be intuitive that one can only be smart about this choice when having knowledge about the underlying distribution. Often, data is not normally distributed, not Poisson-distributed, but instead somewhat uniquely distributed based on the use case -- in a way that deserves brief characterization (a quick look is often enough!); which then allows for making informed decisions about which aggregate parameters to look at -- and which pieces of information are fine to drop. > Histograms require manually specifying the distribution of your data, which is time-consuming, lossy, and can introduce significant error bands around your quantile estimates. Yes! Great point. Honestly, I was a little bit shocked when I saw how this works in the Prometheus ecosystem. I happen to have an example for this I think: we (Opstrace) have contributed a tiny patch to Cortex where we changed the parameterization of a specific histogram metric, because the upper band was super broad, leading to a blind spot (a lack of resolution) in the range of values that was most interesting to us -- see https://github.com/cortexproject/cortex/issues/2530 and https://github.com/cortexproject/cortex/pull/2540. I feel like in our industry when people do the readout and perform some basic dashboarding they're often OK with using aggregates that have not been sanity-checked. That might sometimes be a fair approach. If a readout is not useful, after all one will learn about that very fact through incidents :). > Veneur[2] addresses these use-cases for applications that use DogStatsD[3] by using clever data structures for approximate histograms[4] and approximate sets[5] This is extremely interesting, thanks for sharing. We'll have a look! Having performed many different kinds of data analyses in my science career, I'm rather convinced of the idea that there should be an easy way to perform advanced (yet standard) data analyses for irregular distributions in multi-dimensional data, such as clustering (hierarchical, k-means, DBSCAN, you name it), Principal Component Analysis (PCA), and the likes. I have ideas in this regime -- let's see what makes sense and how far we get with Opstrace! Again, thanks for the links and for sharing your perspective.

d eventually transition the company to Kubernetes. In 2019, I was at RedHat and Mat was still at Mesosphere. A few months after IBM announced purchasing RedHat, Mat and I started brainstorming problems that we could solve in the infrastructure space. We started interviewing a lot of companies, always asking them the same questions: "How do you build and test your code? How do you deploy? What technologies do you use? How do you monitor your system? Logs? Outages?" A clear set of common problems emerged. Companies that used external vendors—such as CloudWatch, Datadog, SignalFX—grew to a certain size where cost became unpredictable and wildly excessive. As a result (one of many downsides we would come to uncover) they monitored less (i.e. just error logs, no real metrics/logs in staging/dev and turning metrics off in prod to reduce cost). Companies going the opposite route—choosing to build in-house with open source software—had different problems. Building their stack took time away from their product development, and resulted in poorly maintained, complicated messes. Those companies are usually tempted to go to SaaS but at their scale, the cost is often prohibitive. It seemed crazy to us that we are still stuck in this world where we have to choose between these two paths. As infrastructure engineers, we take pride in building good software for other engineers. So we started Opstrace to fix it. Opstrace started with a few core principles: (1) The customer should always own their data; Opstrace runs entirely in your cloud account and your data never leaves your network. (2) We don’t want to be a storage vendor—that is, we won’t bill customers by data volume because this creates the wrong incentives for us. (AWS and GCP are already pretty good at storage.) (3) Transparency and predictability of costs—you pay your cloud provider for the storage/network/compute for running Opstrace and can take advantage of any credits/discounts you negotiate with them. We are incentivized to help you understand exactly where you are spending money because you pay us for the value you get from our product with per-user pricing. (For more about costs, see our recent blog post here: https://opstrace.com/blog/pulling-cost-curtain-back). (4) It should be REAL Open Source with the Apache License, Version 2.0. To get started, you install Opstrace into your AWS or GCP account with one command: `opstrace create`. This installs Opstrace in your account, creates a domain name and sets up authentication for you for free. Once logged in you can create tenants that each contain APIs for Prometheus, Fluentd/Loki and more. Each tenant has a Grafana instance you can use. A tenant can be used to logically separate domains, for example, things like prod, test, staging or teams. Whatever you prefer. At the heart of Opstrace runs a Cortex (https://github.com/cortexproject/cortex) cluster to provide the above-mentioned scalable Prometheus API, and a Loki (https://github.com/grafana/loki) cluster for the logs. We front those with authenticated endpoints (all public in our repo). All the data ends up stored only in S3 thanks to the amazing work of the developers on those projects. An "open source Datadog" requires more than just metrics and logs. We are actively working on a new UI for managing, querying and visualizing your data and many more features, like automatic ingestion of logs/metrics from cloud services (CloudWatch/Stackdriver), Datadog compatible API endpoints to ease migrations and side by side comparisons and synthetics (e.g. Pingdom). You can follow along on our public roadmap: https://opstrace.com/docs/references/roadmap. We will always be open source, and we make money by charging a per-user subscription for our commercial version which will contain fine-grained authz, bring-your-own OIDC and custom domains. Check out our repo (https://github.com/opstrace/opstrace) and give it a spin (https://opstrace.com/docs/quickstart). We’d love to hear what your perspective is. What are your experiences related to the problems discussed here? Are you all happy with the tools you’re using today?

within an existing ecosystem is a non-trivial task. I'm sure it will come together eventually, but the commitment to ship it "early next year" seems unlikely to me.

m/cortexproject/cortex [5] https://github.com/VictoriaMetrics/VictoriaMetrics/ [6] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...