Newbie’s guide to Deep Learning
Taking baby steps when starting DL
I have been asked by quite a few people on how to start Machine Learning and Deep Learning. Here, I have curated a list of resources which I used and the path I took when I first learnt Machine Learning. I will keep on updating this article as I find more helpful resources.
Free Courses
Start out with Andrew Ng’s Machine Learning course on Coursera. This will teach you the ropes of Machine Learning and will brush up your Linear Algebra skill a little bit. Make sure you do all the assignments and after you have completed the course, you will get a hold of Machine Learning concepts such as; Linear Regression, Logistics Regression, SVM, Neural Networks and K-means clustering. You will also have a chance to build a toy recommender system while doing the course. This will open your eyes and make you feel more confident to venture further into the land of ML and Deep Learning. But be aware of the fact that there are so many other crucial concepts that this course will not be able to cover. So, be prepared to broaden your knowledge after this course. If you are super stoked after completing Dr. Ng’s course, you should check out his other courses, offered as parts of Deep Learning Specialization on Coursera.
Fast.ai offers a free online course on Deep Learning and they offer two parts in their course:
- Deep Learning Part 1: Practical Deep Learning for Coders
- Deep Learning Part 2: Cutting Edge Deep Learning for Coders
After those courses, you may be ready to tackle Hinton’s Neural Networks for Machine Learning. Hinton’s course is relatively harder compared to previously mentioned courses since the lectures are quite dry and they contain more Math concepts. If you feel like you cannot tackle the course yet, don’t get discouraged! Leave it for a while and do the Math part (described in next section) and then come back. You will be able to definitely tackle the course this time! Remember, determination, determination and yes, more determination.
Math
Deep Learning definitely requires you to have a strong command of Linear Algebra, Differential Calculus and Vector Calculus, just to name a few. If you want to quickly brush up some elementary Linear Algebra and start coding, Andrej Karpathy’s Hacker’s guide to Neural Networks is highly recommended. I find hadrienj’s notes on Deep Learning Book extremely useful to actually see how underlying Math concepts work using Python (Numpy). If you like to learn from videos, 3blue1brown has one of the most intuitive videos for concepts in Linear Algebra, Calculus, Neural Networks and other interesting Math topics. Implementing your own CPU-based backpropagation algorithm on a non-convolution based problem is also a good place to start to truly understand how backpropagation works.
Getting Serious
If you want to take a notch up your Machine Learning knowledge and ready to get serious (I mean graduate-level serious), dive into Learning From Data by Caltech Professor Yaser Abu-Mostafa. Be prepared to do all the Math. It can be a bit challenging but it will definitely be rewarding once you have gone through it and did your work. I believe it would be hard for textbooks to capture the current state of Deep Learning since the field is moving at a very fast pace. But the go-to textbook would be Deep Learning Book by Goodfellow, Bengio, and Courville. It is freely available online so you might as well download chapter by chapter and tackle the textbook one chapter at a time.
Papers, papers, papers, oh dang it I can’t catch ’em up anymore
Yeah, the knowledge of deep learning comes primarily from papers and the rate at they are being published is extreme these days. A good starting point is Reddit. Subscribe to /r/machinelearning and /r/deeplearning. I find machinelearning subreddit more useful though. ArxivSanity is a good place to check out papers which are related to the ones you are looking for. One important thing to do when reading papers in Deep Learning is doing a good literature review. Doing a good literature review gives you a good sense of how things evolve. A way to tackle doing literature review is to install Google Scholar Chrome Extension and search for the paper you want to look up. You can follow the “Related articles” and “Cited by” to follow the prior work as well as newer work based on that paper. A good habit to form when reading a paper is to draw a mind map of the concepts in the paper.
The advantage of a mind map is that it is a good way to keep track of the relationships of concepts presented in the paper. I find mind maps very useful to keep track of related literature and how they relate to the paper I am reading. Mind maps give me a clear picture of a paper and also serves as a good summary of the paper after I have read it.
I find Twitter very useful to follow Machine Learning and Deep Learning research. You can start by following well-known individuals in the ML/DL field and branch out from there. As I usually retweet researches on Adversarial Machine Learning and on self-driving cars, you can also follow me on twitter and treat as your feed. What you can do is check out the people whom I have retweeted, check out their tweets and follow other researchers in their circles. Twitter will also recommend good people to follow once you have provided enough data, i.e. followed enough ML/DL researchers (ML FTW!).
Kaggle
I cannot stress how useful Kaggle is. I highly recommend trying out Kaggle competitions even though you have a slim chance of even getting to top-100. The value of Kaggle competitions is the community. Read the kernels and take good practices from them. Read comments and engage in discussions. That’s where you will learn tremendously. You will learn how people do Exploratory Data Analysis and how they handle various cases of missing data, skewed data, etc. You will also learn how people make decisions on why they chose certain models over the other. There is so much knowledge in Kaggle competitions.
Inspirations
Visual Introduction to Machine Learning is a good way to visually grasp how statistical learning techniques are used to identify patterns in data.
Google’s Seedbank is a great resource to get inspired! Take a look at the examples and follow the literature.
Distill.pub is a good place to learn some of the DL concepts interactively. I wish Distill has more articles than it has now.
Tip of the Iceberg
Nothing matters unless you implement what you have learnt yourself. ML and DL sound like magic until you implement the whole pipeline yourself. The whole pipeline includes Data Sourcing, Data Collection, Data Quality Assessment, Data Cleaning, Data Annotation, Data Preprocessing, Building Workflow, Building Models, Tuning Models, Assessing Models, Deploying Models and Reiterating Models. Those steps are just some of the steps in the whole ML/DL pipeline. Those who have done full-scale DL work knows how important it is to keep the whole development operations as streamlined as possible. Not only the whole data sourcing, collection, annotation, cleaning and assessing steps take at least 60% of the whole project, but also they can be one of the most expensive parts of the project (aside from your power-hungry GPUs!).
All in all, ML/DL field is a growing field and you have to keep your ears, eyes, and mind wide open. Don’t just jump onto a shiny new technique just because a paper/blog/tutorial/person/YouTube video says it performs very well on a particular dataset. I have seen a lot of shiny new techniques come and go quite quickly. Always be aware of the fact that it is important to distinguish signal from noise!
This article is a living and breathing matter. Your feedback matters to me. Please leave comments on what resources I should add and what you found the most helpful when you were learning ML/DL. Thanks a lot for reading!
[1] Triantafillou, E., Zemel, R., & Urtasun, R. (2017). Few-shot learning through an information retrieval lens. In Advances in Neural Information Processing Systems (pp. 2255–2265).
