pushgateway

hello, i'm not quite sure what you mean by "monitoring" ... but i do 2 things * enable logs to some log-collection facility a la (remote)syslog/elk/grafana loki/... * enable metrics for example with prometheus-pushgateway for more information * https://github.com/prometheus/pushgateway from its description: "The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus." just my 0.02€

While I somehow understand Prometheus idea that pull is easier to scale than push I've had a bad luck with it. First of all Prometheus doesn't even consider monitoring of long-running jobs other than pull way (which didn't make sense for me). There is push gateway [0] but clients libraries seem to consider it only for short-lived jobs where you can send the metrics at the end [1]. It seems I couldn't "push" from long living jobs trivially Second when using it for example with django you have to be careful with how you handle multiprocessing that UWSGI/gunicorn does, see [2] - it has bitten me at leas once. Comparing to push model where I can just push metrics to [3] statsd_exporter directly and be done with it, but support for statsd is lacking both in terms of frameworks (everyone seems to be migrating to native clients...) and functiona

There's a push gateway: https://github.com/prometheus/pushgateway

I guess the https://github.com/prometheus/pushgateway could help with that? As for the query performance there's a lot of things you can do with recording rules, that might help a lot with speeding up dashboards or queries.

It is interesting to see this because (AFAIK, etc.) the only option for Prometheus has been on-premises. I had hoped to batch analytics and import into Prometheus out of production (aka off-premises), but my very preliminary investigation appeared to show that it requires "live" data, since it assigns the timestamp when events are ingested. I'm still confused whether or not bulk import will ever be a supported scenario. https://github.com/prometheus/prometheus/issues/398 Improve staleness handling (2014-2017) https://github.com/prometheus/pushgateway/pull/119 Reject pushes with timestamps (2017)

How is this different from https://github.com/prometheus/pushgateway ?

The https://github.com/prometheus/pushgateway serves those cases, primarily service-level batch jobs.

The default instrumentation that comes with node_exporter covers far more than what systat provides. Retrieving data from /proc at every scrape also gives you a more accurate timestamp, which helps reduce graph artifacts. As an aside, if you have machine-level cronjobs you want to expose metrics from you can use the textfile[1] module of the node_exporter, which reads in data from *.prom files in the same format as accepted by the Pushgateway. [1]https://github.com/prometheus/node_exporter/blob/master/coll... [2]https://github.com/prometheus/pushgateway

(initial Prometheus author here) I wouldn't totally rule out the pushgateway for this use case. If we decided to implement a metrics timeout in the pushgateway (https://github.com/prometheus/pushgateway/issues/19), this would also take care of stale metrics from targets that are down or decommissioned. The pushing clients would probably even want to set a client-side timestamp in that case, as they are expected to be pushing regularly enough for the timestamp to not become stale (currently Prometheus considers time series stale that are >5 minutes old by default, see also the "Improved staleness handling" item in http://prometheus.io/docs/introduction/roadmap/.