loman

I used to build quant investment notebooks that had to be deployed in production. Lots of problems with that. Mine were: Notebook cells run out of order, so you often have something that works in a session, but not in a fresh run. Developing against limited datasets, so you fail against things you didn’t know to test for. Small adaptions that have to be made every time the notebook is translated into a code file. We streamlined it by making a graph-structured Computation a first class object that tracked staleness as code or data was updated. Then that class could be directly published, and when failures happened in production, the graph could be serialized with the inputs and intermediate calculation data that caused failure, for investigation in a notebook. We open sourced the implementation https://github.com/janushendersonassetallocat

fn_graph: https://github.com/BusinessOptics/fn_graph (My favorite one ^) pyungo: https://github.com/cedricleroy/pyungo Loman: https://github.com/janushendersonassetallocation/loman Tributary: https://github.com/timkpaine/tributary

The culture shock you mention was very real. I joined in 2010 (when Athena was 3 years old) and left 8 years later in 2018. I remember my first few months unlearning normal Python and figuring out how to build the 'pixie graph', a lazily-evaluated Python DAG suited for calculating financial instruments. It took a while to get your head around this, but when you did it was a very powerful and productive way to building trading and risk management applications. To get some sense in how this worked, here are two public projects on github with good introductions: * https://github.com/timkpaine/tributary * https://github.com/janushendersonassetallocation/loman

I'm a big fan of Graphviz. My old team created a library called Loman, which we open-sourced, which uses DAGs to represent calculations. Each node represents a part of the calculation and contains a value, similar to a cell in an spreadsheet, and Loman tracks what is stale as you update inputs. Loman includes built in support for creating diagrams using Graphviz. In our quant research we have found that invaluable when revisiting old code, as it allows you to quickly see the structure and meaning of graphs with hundreds of nodes, containing thousands of lines of code. We've found it quite useful for quant research, and in production it works nicely because you can serialize entire computation graph which gives an easy way to diagnose what failed and why in hundreds of interdependent computations. It's also useful for real-time displays, w

My team has a similar library called Loman, which we open-sourced. Instead of nodes representing tasks, they represent data, and the library keeps track of which nodes are up-to-date or stale as you provide new inputs or change how nodes are computed. Each node is either an input node with a provided value, or a computed node with a function to calculate its value. Think of it as a grown-up Excel calculation tree. We've found it quite useful for quant research, and in production it works nicely because you can serialize entire computation graph which gives an easy way to diagnose what failed and why in hundreds of interdependent computations. It's also useful for real-time displays, where you can bind market and UI inputs to nodes and calculated nodes back to the UI - some things you want to recalculate frequently, whereas some are slow a

The ease of the calculation tree in Excel versus having to keep track of what cells in a notebook you have updated was a large part of why we built and open-sourced Loman [1]. It's a computation graph that keeps track of state as you update data or computation functions for nodes. It also ends up being useful for real-time interfaces, where you can just drop what you need at the top of a computation graph and recalculate what needs updating, and also for batch processes where you can serialize the entire graph for easy debugging of failures (there are always eventually failures). We also put together some examples relevant to finance [2] [1] https://loman.readthedocs.io/en/latest/user/quickstart.html [2] https://github.com/janushendersonassetallocation/loman/tree/...

I had the same trouble with order dependence as notebooks got to a certain size, so my team and I created and open-sourced a library, Loman, to help with that. It allows you to interactively create a graph, where nodes represent inputs or functions, and then keeps track of state as you change or add inputs, intermediate functions and request recalculations. Our experience has been broadly positive with this way of working. As graphs get larger, it's easy to lift them into code files in libraries, while continuing to modify or extend them in notebooks. The graph structure and visualization make it easy to return to loman graphs with up to low hundreds of nodes, which would make for a fearsome notebook otherwise. It also makes it easy to bolt Qt or Bokeh UIs onto them for interactive dashboards - just bind UI widgets and events to the input