“Deep” Independent Component Analysis in Tensorflow [ Manual Back Prop in TF ]
We can capture local changes using Independent components analysis, however, the image data we encounter in real life lives in a very high dimensional space. I wanted to see if we can perform ICA with a combination of deep learning.
Please note that this post is just for fun as well as self-learning.
ICA on Face Data or MRI Data
As seen above, when we perform ICA on image of faces or Brain MRI, we can notice that we are able to capture (clearly) the local changes. However, would this work on real life images? What I mean by this is would we able to capture the general structure of an object from set of image that contains many variances.
As seen above, there can be a lot of difference of how an image look like for a given object. (with scales or variance). But if we think about it, image of faces as well as brain MRI, already are centered and does not contain many variations. How about a subset of bird images from STL data set?
As seen above, all of the them contains images of birds, but some are flying and others are sitting. There are variances in scale as well as rotation, angle etc….
Direct Approach of PCA, KPCA, and ICA
Lets use the high level API approach to perform some dimensionality reduction techniques on 500 images of birds from STL data set. (I am going to use Sklearn.) Also, I will only get the first 40 principle component as well as independent components.
We can see that in our first principle image we have a white blob in the center of our image. (That actually does look like a bird). And moving on the axis we can observe that few of the principle component does resemble a bird like figure. However, after the third row, it is unclear what is exactly in the image.
Again, lets use the high level api approach to perform KPCA, similar results with PCA. Again, the first few principle images does resembles a bird like figure.
Finally, the high level ICA approach. We can see that in general it have captured the local changes in the image. If I look closely enough I actually can see some bird like figures in the image.
“Deep” Learning Approach
Yellow Box → Convolutional Layer
Blue Box → Principle Component Analysis Layer
Red Box → Independent component Analysis Layer
Now from the above graph we can already know what I wanted to try out, just have couple of convolution layer to extract higher level of features from image and then perform PCA to reduce dimension, and use ICA to statistically make those principle components independent from one another.
The reason why I am using PCA before ICA, is from my past experiments, using ICA as a dimensionality reduction did not converge so well. Also please note that there are multiple method of ICA we can choose, such as Gradient ascent. I am just going to use FastICA.
Before moving on lets first implement PCA layer, as seen above, also one another thing to note here is the fact that I am going to use Adam to optimize the weighs for Fast ICA rather than direct assignment.
Finally, rather than having on ICA to optimize for all, I wanted to split into four different streams.
Shape of the Functions we can use for Fast ICA
I won’t go into the details, but for Fast ICA there are couple of activation function (not the proper name but I’ll call it that.), that we can choose to use. And I wanted to visualize them.
ICA as a loss function
Top Left → original image
Top Right → Image after first layer
Bottom Left → Image after second layer
Bottom Right → Image after third layer
So as image proceed to move up the layer some of the part in the images gets darken while other parts gets more whiten. Now lets see the 40 combined principle images.
Some blackish figures does look like birds, but not that crispy, also when we view 40 independent components.
We can see that some of the parts does look more like birds, however not as crisp, I guess there is a long way to go. The convergence process can be seen below.
Code for this post
For Google Colab, you would need a google account to view the codes, also you can’t run read only scripts in Google Colab so make a copy on your play ground. Finally, I will never ask for permission to access your files on Google Drive, just FYI. Happy Coding!
To access the codes for sklearn High level click here.
To access the codes for Tensorflow click here.
Final Words
If any errors are found, please email me at jae.duk.seo@gmail.com, if you wish to see the list of all of my writing please view my website here.
Meanwhile follow me on my twitter here, and visit my website, or my Youtube channel for more content. I also implemented Wide Residual Networks, please click here to view the blog post.
Reference
- PIL?, H. (2018). How can I save an image with PIL?. Stack Overflow. Retrieved 15 September 2018, from https://stackoverflow.com/questions/14452824/how-can-i-save-an-image-with-pil
- PIL, S. (2018). Saving Image with PIL. Stack Overflow. Retrieved 15 September 2018, from https://stackoverflow.com/questions/19651055/saving-image-with-pil
- slow, S. (2018). Saving graphs using Matplotlib is very slow. Raspberry Pi Stack Exchange. Retrieved 16 September 2018, from https://raspberrypi.stackexchange.com/questions/72542/saving-graphs-using-matplotlib-is-very-slow
- CS231n Convolutional Neural Networks for Visual Recognition. (2018). Cs231n.github.io. Retrieved 16 September 2018, from http://cs231n.github.io/classification/
