nupic-legacy

the RNNs illustrated are also iterating over an internal state (flowing from left to right though the same network for each new word) - and this internal state is also an embedding of some kind. But it's going to be very difficult to decipher what each dimension here represents, as it's being built purely as a function of the input word vectors, its own previous state and a NN with initially random weights. Now, although actual 'brain experiments' have shown that individual neuron (or local clusters) apparently light up when particular thoughts are had (alternatively, cause thoughts to be had), each cluster seems likely to be just one aspect of (say) 'dogginess'. So, one area will correspond to the smell of dogs, others to wet noses, others to being outdoors (i.e. all aspects of the overall 'dogginess' concept) - but these things will all overlap in multiple ways with other concept 'vectors'. Which is how huge spaces of ideas are searched in parallel, rather than sequentially (using, say, an is_doggy_quality symbol). There are also parallels here with the Numenta Sparse Distributed Representations [2]. Overall, this presentation seems to be probing at the frontier of what works, and how to leverage that up into something that's more about 'general thinking' rather than pattern matching. It also appears to be a thought-piece, rather than a conference presentation (though, of course, Hinton deserves to be heard on just about anything in NNs, IMHO). [1] http://colah.github.io/posts/2014-07-NLP-RNNs-Representation... [2] https://github.com/numenta/nupic/wiki/Sparse-Distributed-Rep...