keras-vis

Hi! I'm from the AI subteam. We actually didn't have time to implement this part of the solution. Our idea was to use this library: https://github.com/raghakot/keras-vis

I'll agree with you that it's much harder than it should be (thankfully, finding the implementations is the hard part, not using them), but yes, these methods do exist. DeepLIFT (the method I linked in my original comment: https://github.com/kundajelab/deeplift), takes a Keras model (with Theano or TensorFlow backend) as input and provides feature importance scores for any desired layer of the network (raw data inputs, inputs to dense layers following convolution, etc.). Keras-Vis (https://github.com/raghakot/keras-vis) is another nice package that allows for easy visualization of saliency maps and convolutional filters. Perturbing inputs and looking at the effect on the output of the network is another technique people use pretty often. I think there's a lot of room for this space to become easier to use, especially for newer deep learni