[ Experimental Model ] Implementing Tree Style Deep Neural Network for CIFAR 10 Classification [ Manual Back Prop with TF ]

Gif from here

So today, I wanted to make an experimental model, and this network architecture came to my thought yesterday evening. An architecture that resembles the shape of a tree, but if you know of any well-known academic paper that supports this innovation or network architecture, please comment below so that I may provide a proper reference.

** UPDATE ** Peter Baylies provided me with a paper that has a similar network architecture as the one posted in this post, please click here to view the paper.

Please note that this is an experimental model, hence performance is not my goal.

Again, as always lets compare .how a manual back propagation method compares with a traditional automatic differentiation model. If you are not aware of automatic differentiation, please click here.

Network Architecture ( Basic Building Blocks / Full Architecture )

Black Rectangle → Input Image
Green / Red Rectangle → Convolution Layer (Kernel size 3) and Batch Normalization
Yellow Rectangle → Bottle Neck Convolution Layer (Kernel size 1) and Batch Normalization
Blue Circle → Addition Operation (Residual Connection)

So above residual block is our basic block of our full, network. Now, if you see the full architecture, you’ll understand why I call this network a Tree Network. (It is most because the network is shaped like a tree.)

Circles → Each Convolution Residual Block that we covered above
Light Brown Rectangle → Fully connected Network
Orange Rectangle → Soft Max Layer

So if we rotate our model 90 degrees, it looks like a tree! (Well kinda…)

Also, please take note on the purple arrows, there we are going to do something different as well.

Layer3_a → Only adding up results from Red Boxed Circle, and Blue Boxed Circle.
Layer3_b → Only adding up results from Blue Boxed Circle and Yellow Boxed Circle.
Layer3_c → Only adding up results from Red Boxed Circle and Yellow Boxed Circle.

So purple arrow is different from pinkish arrows since, we do not add up all of the variables.

Results (Auto Differentiation)

On the 100 epoch, we can see that the model is clearly over fitting, this may be because of many reasons. But it is well known that in deep learning models, Adam optimzers does not generalize so well. If you are not aware with Adam optimizer please click here.

Results (Broken Dilated Back Propagation)

For this case, manual back propagation was horrible. The model was not even able to achieve 40 percent accuracy.

Please note, we performed batch normalization during feed forward operation however, I did not back propagated throughout the process, hence the broken dilated back propagation.

Transparency

In my hopes to make this experiment more transparent, I uploaded all of the command log from my window prompt to my Github. To view the output generated by auto differentiation please click here, for manual back propagation please click here.

Interactive Code

For Google Colab, you would need a google account to view the codes, also you can’t run read only scripts in Google Colab so make a copy on your play ground. Finally, I will never ask for permission to access your files on Google Drive, just FYI. Happy Coding!

To access the auto differentiation code please click here.
To access the manual back propagation model, please click here.

Final Words

I really love doing experimental models, they may not guarantee performance, but they fuel my creativity.

If any errors are found, please email me at jae.duk.seo@gmail.com, if you wish to see the list of all of my writing please view my website here.

Meanwhile follow me on my twitter here, and visit my website, or my Youtube channel for more content. I also did comparison of Decoupled Neural Network here if you are interested.

Reference

1. file?, W. (2018). What command do I need to unzip/extract a .tar.gz file?. Askubuntu.com. Retrieved 30 March 2018, from https://askubuntu.com/questions/25347/what-command-do-i-need-to-unzip-extract-a-tar-gz-file
2. CIFAR-10 and CIFAR-100 datasets. (2018). Cs.toronto.edu. Retrieved 30 March 2018, from https://www.cs.toronto.edu/~kriz/cifar.html
3. Brownlee, J. (2017). Gentle Introduction to the Adam Optimization Algorithm for Deep Learning — Machine Learning Mastery. Machine Learning Mastery. Retrieved 1 April 2018, from https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/
4. JaeDukSeo/Only_Numpy_Basic. (2018). GitHub. Retrieved 1 April 2018, from https://github.com/JaeDukSeo/Only_Numpy_Basic/blob/master/treenet/man1.txt
5. Automatic differentiation. (2018). En.wikipedia.org. Retrieved 1 April 2018, from https://en.wikipedia.org/wiki/Automatic_differentiation