TANK

ons. They capture the full context of a request(complete with annotations, hierarchies, etc). Those are also persisted to TANK topics So, effectively, everything’s available on TANK. This has all sorts of benefits. Some of them include: - We can, and, have all sort of consumers who process those generates events, looking for interesting/unexpected events and reacting to them (e.g notifying whoever needs to know about them, etc) - It’s trivial to query those TANK topics using `tank-cli`, like so: tank-cli -b <tank-endpoint> -t apache/0 get -T T-2h+15m -f "GET /search" | grep -a Googlebot This will fetch all events starting 2 hours ago, for up to 15 minutes later, that include “GET /search” All told, we are very happy with our setup, and if we were to start over, we’d do it the same way again.

(but it’s pretty slow), a tool our ops folks built (“otinanai”: https://github.com/phaistos-networks/otinanai) and we have a few smaller systems that generate graphs and reports. - For alerts and tickets, our ops folk built another tool that monitors all those events, filters them and executes domain-specific logic that deals with outliers, notifications routing, and more. This solves most of our needs, but we plan to improve this setup further, by monitoring even more resources and introducing more tools(Tank consumers) to get more out of our data.

d they can do so at whatever pace makes sense(i.e depending on how fast they can process messages/events). You are probably familiar with the the log abstraction anyway. See https://engineering.linkedin.com/distributed-systems/log-wha... and http://www.confluent.io/blog/stream-data-platform-1/ Using Kafka as a core infrastructure technology, and for persisting any incoming and generated messages/events to it should be the default strategy. Also, it’s extremely unlikely you will hit service capacity limits, because publishing to Kafka, or a Kafka like service; publishing is mostly about appending data to a file and consuming is mostly about streaming (sequential scan) from files — all very fast and low overhead operations. Shameless plug: if you want a Kafka like service, with, currently, fewer features but with better performance and far less requirements and a standalone operation mode(no need for ZooKepeper), you may want to check https://github.com/phaistos-networks/TANK Anyway, I can’t recommend Kafka and investing on logs enough. Also, Kafka Streams is very elegant; likely based on Google’s DataFlow design and semantics. If you are using Kafka or plan to use it, you may want to evaluate it and adopt it over other heavier footprint and more complicated systems (e.g Storm or Spark).