hobbes

This is a good introduction, and a very useful abstraction. At Morgan Stanley we have built a PL/compiler and tools around a method of logging like this -- logging algebraic data types and live querying out of them with Haskell-like comprehensions/pattern-matching/etc: https://github.com/Morgan-Stanley/hobbes

The one I linked to in my previous post also produces code that's fast enough to use in the critical path in low-latency trading systems. As you may know, q's "scalar performance" is not great; similar to Scheme or python due to boxing overhead (as you can see in the linked k.h file below). Also, the fact that q is untyped has a severe impact on its safe use in large and complex projects. (ETA: https://github.com/Morgan-Stanley/hobbes)

This approach to processing real-time and historical data works very well. We use a similar approach at Morgan Stanley and have a PL/compiler for real-time index construction and query processing on top (https://github.com/Morgan-Stanley/hobbes). In addition to the general model of log writing + reactive processing, I've also found that we spend a lot of time focused on getting the _type structure_ of our data right (which allows for flexible/generic downstream processes that can also be efficient and type-checked against the structure of the data ahead of time). I wonder if/how this aspect of the problem has been considered by the Facebook team.