fold

I'm not sure why you think neural networks can't handle variable numbers of inputs. Recurrent networks that ingest whole sequences have been around for a long time, and other structures have their own network topologies. Support for things other than classic RNNs is more limited, but e.g. TensorFlow Fold (https://github.com/tensorflow/fold) was specifically designed for that.

> - Neural network programs are usually not large and therefore do not need the type safety that Rust offers TensorFlow Fold [1] uses static type checking at model compilation time, presumably because it is meant to consume structured data, and needs to know what the opaque tensors it passes around are supposed to represent. I don't fully comprehend that type system, and debugging type errors was annoying (huge stacktrace full of passing type info around), but after I managed to imitate it well enough to hack up my own blocks, I didn't investigate further. A language-level type system like Rust's would probably have made things easier for me. EDIT: I just remembered that there's also Leaf [2], which claims to be among the faster neural network libraries (although their comparison is a bit outdated). Evidently there are people seriously pu

Thanks, insightful questions. You're absolutely right about combinators and long-range dependencies. We have a special block type in the high-level API (https://github.com/tensorflow/fold/blob/master/tensorflow_fo...) for accumulating results here without having to explicitly shuttle them around; not perfect but very handy in many cases. Regarding your second question, the equivalent of padding in dynamic batching is "pass through" do-nothing ops, which are introduced transparently but worth being aware of if you want to understand the machinery. The worst case scenario here is chains of wildly varying lengths, where we need to add pass-throughs to the shorter chains to match the length of the longest chain.

Github link: https://github.com/tensorflow/fold Paper link: https://openreview.net/pdf?id=ryrGawqex