Review of Deeplearning.ai Courses
Recently I’ve finished the last course of Andrew Ng’s deeplearning.ai specialization on Coursera, so I want to share my thoughts and experiences in taking this set of courses. I’ve found the review on the first three courses by Arvind N very useful in taking the decision to enroll in the first course, so I hope, maybe this can also be useful for someone else.
tl;dr
Taking the five courses is very instructive. The content is well structured and good to follow for everyone with at least a bit of an understanding on matrix algebra. Some experience in writing Python code is a requirement. The programming assignments are well designed in general. Apart of their instructive character, it’s mostly enjoyable to work on them, too. Finally, in my opinion, doing this specialization is a fantastic way to get you started on the various topics in Deep Learning. After taking the courses, you should know in which field of Deep Learning you wanna specialize further on.
Background
I have to admit, that I was a sceptic about Neural Networks (NN) before taking these courses. Coming from traditional Machine Learning (ML), I couldn’t think that a black-box approach like switching together some functions (neurons), which I’m not able to train and evaluate on separately, may outperform a fine-tuned, well-evaluated model. And the fact, that Deep Learning (DL) and Artificial Intelligence (AI) became such buzzwords, made me even more sceptical.
Nontheless, every now and then I heard about DL from people I’m taking seriously. Especially a talk by Shoaib Burq, he gave at an Apache Spark meetup in Zurich was a mind-changer. So I decided last year to have a look, what’s really behind all the buzz. First, I started off with watching some videos, reading blogposts and doing some tutorials. Unfortunately, this fostered my assumption that the math behind it, might be a bit too advanced for me. I deeply enjoy practical aspects of math, but when it comes to derivation for the sake of derivation or abstract theories, I’m definitely out.
When I’ve heard about the deeplearning.ai specialization for the first time, I got really excited. In previous courses I experienced Coursera as a platform that fits my way of learning very well. And from videos of his first Massive Open Online Course (MOOC), I knew that Andrew Ng is a great lecturer in the field of ML. But I’ve never done the assignments in that course, because of Octave. Reading that the assignments of the actual courses are now in Python (my primary programming language), finally convinced me, that this series of courses might be a good opportunity to get into the field of DL in a structured manner. But first, I haven’t had enough time for doing the course work.
That changed, when I was suffering from a (not severe, but anyhow troublesome) health issue in the middle of last year. When I felt a bit better, I took the decision to finally enroll in the first course. Normally, I enroll only in a specific course on a topic I wanna learn, binge watch the content and complete the assignments as fast as possible. But this time, I decided to do it thoroughly and step-by-step, repectively course-by-course. I was hoping, the work on a cognitive challenging topic might help me in the process of getting well soonish. And doing the programming assignments have been a welcome opportunity to get back into coding and regular working on a computer again. In fact, during the first few weeks, I was only able to sit in front of a monitor for a very short and limited time span. So I had to print out the assignments, solved it on a piece of paper and typed-in the missing code later, before submitting it to the grader. Furthermore a positive, rather unexpected sideeffect happened during the beginning. In the first three courses there are optional videos, where Andrew interviews heroes of DL (Hinton, Bengio, Karpathy, etc). These videos were not only informative, but also very motivational, at least for me— especially the one with Ian Goodfellow.
What to Expect
On a professional level, when you are rather new to the topic, you can learn a lot of doing the deeplearning.ai specialization. First and foremost, you learn the basic concepts of NN. How does a forward pass in simple sequential models look like, what’s a backpropagation, and so on. I read and heard about this basic building blocks of NN once in a while before. But never it was so clear and structured presented like by Andrew Ng. So I experienced this set of courses as a very time-effective way to learn the basics and worth more than all the tutorials, blog posts and talks, which I went through beforehand.
As a sidenote, the first lectures quickly proved the assumption wrong, that the math is probably too advanced for me. In fact, with most of the concepts I’m familiar since school or my studies — and I don’t have a master in Tech, so don’t let you scare off from some fancy looking greek letters in formulas. With a superficial knowledge on how to do matrix algebra, taking derivatives to calculate gradients and a basic understanding on linear regression and the gradient-descent algorithm, you’re good to go — Andrew will teach you the rest.
Though otherwise stated in lots of marketing stuff around the technology, you learn also in the first introductory courses, that NN don’t have a counterpart in biological models. Signal processing in neurons is quite different from the functions (linear ones, with an applied non-linearity) a NN consists of.
What’s very useful for newbies is to learn about different approaches for DL projects. Wether to use pre-trained models to do transfer learning or take an end-to-end learning approach. Also the concept of data augmentation is addressed, at least on the methodological level.
In the more advanced courses, you learn about the topics of image recognition (course 4) and sequence models (course 5). There the most common variants of Convolutional Neural Networks (CNN), respectively Recurrent Neural Networks (RNN) are taught. I think it’s a major strength of this specialization, that you get a wide range of state-of-the-art models and approaches. It probably will not make you a specialist in DL, but you’ll get a sense in which part of the field you can specialize further.
What I’ve found very useful to deepen the understanding is to complement the course work with the book “Deep Learning with Python” by François Chollet. The methodological base of the technology, which is not in scope of the book, is well addressed in the course lectures. And on the other hand, the practical aspects of DL projects, which are somehow addressed in the course, but not extensivly practised in the assignments, are well covered in the book. Especially the data preprocessing part is definitely missing in the programming assignments of the courses. You do get tutorials on using DL frameworks (tensorflow and Keras) in the second, respectively fourth MOOC, but it’s obvious that a book by the inital creator of Keras will teach you how to implement a DL model more profoundly.
What you can specifically expect from the five courses, and some personal experiences in doing the course work, is listed in the following part. After that, I’ll conclude with some final thoughts.
Neural Networks and Deep Learning
This course teaches you the basic building blocks of NN. You’ll learn about Logistic Regression, cost functions, activations and how (sochastic- & mini-batch-) gradient descent works. Also you get a quick introduction on matrix algebra with numpy in Python.
The course is a straight forward introduction. Andrew Ng is a great lecturer and even persons with a less stronger background in mathematics should be able to follow the content well.
With the assignments, you start off with a single perceptron for binary classification, graduate to a multi-layer perceptron for the same task and end up in coding a deep NN with numpy. Especially the two image classification assignments were instructive and rewarding in a sense, that you’ll get out of it a working cat classifier. As you can see on the picture, it determines if a cat is on the image or not — purr ;)
Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
As its title suggests, in this course you learn how to fine-tune your deep NN. The most frequent problems, like overfitting or vanishing/exploding gradients are addressed in these lectures. You learn how to find the right weight initialization, use dropouts, regularization and normalization. And of course, how different variants of optimization algorithms work and which one is the right to choose for your problem.
The most useful insight of this course was for me to use random values for hyperparameter tuning instead of a more structured approach. It turns out, that picking random values in a defined space and on the right scale, is more efficient than using a grid search, with which you should be familiar from traditional ML.
The assignments in this course are a bit dry, I guess because of the content they have to deal with. But, every single one is very instructive — especially the one about optimization methods. As a reward, you’ll get at the end of the course a tutorial about how to use tensorflow, which is quite useful for upcoming assignments in the following courses.
Structuring Machine Learning Projects
This is definitely a black swan. As its content is for two weeks of study only, I expected a quick filler between the first two introductory courses and the advanced ones afterwards, about CNN and RNN. Also, I thought that I’m pretty used to, how to structure ML projects. Nonetheless, it turns out, that this became the most valuable course for me.
In this course you learn good practices in developing DL models. Splitting your data into a train-, dev- and test-set should sound familiar to most of ML practitioners. Before starting a project, decide thoroughly what metrices you want to optimize on. And you should quantify Bayes-Optimal-Error (BOE) of the domain in which your model performs, respectively what the Human-Level-Error (HLE) is. This is an important step, which I wasn’t that aware of beforehand (normally, I’m comparing performance to baseline models — which is nonetheless important, too). When you have to evaluate the performance of the model, you then compare the dev error to this BOE (resp. HLE) and training error, of course. With that you can compare the avoidable bias (BOE to training error) to the variance (training to dev error) of your model. And on which of these two are larger depends, what tactics you should use to increase the performance furthermore. For example, if there’s a problem in variance, you could try get more data, add regularization or try a completely different approach (e.g. alternative architecture or different hyperparameter search).
You also learn about different strategies to set up a project and what the specifics are on transfer, respectively end-to-end learning.
Convolutional Neural Networks
As I was not very interested in computer vision, at least before taking this course, my expectation on its content wasn’t that high. But it turns out, that this became the most instructive one in the whole series of courses for me.
In this course you learn mostly about CNN and how they can be applied to computer vision tasks. From the lecture videos you get a glance on the building blocks of CNN and how they are able to transform the tensors. Some videos are also dedicated to Residual Network (ResNet) and Inception architecture.
I personally found the videos, respectively the assignment, about the YOLO algorithm fascinating. The basic functionality is so well visualized in the lectures and I haven’t thought before, that object detection can be such an enjoyable task. It was also enlightening that it’s sometimes not enough to build an outstanding, but complex model. In the context of YOLO, and especially its successors, it is quite clear that speed of prediction is also an important metric to consider.
The most instructive assignment over all five courses became one, where you implement a CNN architecture on a low-level of abstraction. The optional part of coding the backpropagation deepened my understanding how the reverse learning step really works enormously.
There are two assignments on face verification, respectively on face recognition. It’s a nice move that, during the lectures and assignments on these topics, you’re getting to know the deeplearning.ai team members — at least from their pictures, because these are used as example images to verify.
An artistic assignment is the one about neural style transfer. Basically, you have to implement the architecture of the Gatys et al., 2015 paper in tensorflow. I also played along with this model apart of the course with some splendid, but also some rather spooky results. As you go through the intermediate logged results, you can see how your model learns and applies the style to the input picture over the epochs. Once I felt a bit like Frankenstein for a moment, because my model learned from its source image the eye area of a person and applied it to the face of the person on the input photo. So it became a DeepFake by accident.
Sequence Models
The last one, I think is the hardest. You learn the concepts of RNN, Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM), including their bidirectional implementations. Although it was for me the ultimate goal in taking this specialization to understand and use these kinds of models, I’ve found the content hard to follow. That might be because of the complexity of concepts like backpropation through time, word embeddings or beam search. And I think also, the amount of these non-trivial topics would be better split up in four, instead of the actual three weeks.
On the other hand, quizzes and programming assignments of this course appeard to be straight forward. You learn how to develop RNN that learn from sequences of characters to come up with new, similar content. For example, you’ve to code a model that comes up with names for dinosaurs. LSTMs pop-up in various assignments. You build one that writes a poem in the (learned) style of Shakespeare, given a Sequence to start with. In another assignment you can become artistic again. You’ve to build a LSTM, which learns musical patterns in a corpus of Jazz music. Afterwards you then use this model to generate a new piece of Jazz improvisation. Mine sounds like this — nothing to come up with in Montreux, but at least, it sounds like Jazz indeed. And it’s again a LSTM, combined with an embedding layer beforehand, which detects the sentiment of an input sequence and adds the most appropriate emoji at the end of the sentence. And yes, it emojifies all the things!
It’s fantastic that you learn in the second week not only about Word Embeddings, but about its problem with social biases contained in the embeddings also. And most import, you learn how to tackle this problem in a three step approach: identify — neutralize — equalize. And finally, a very instructive one is the last programming assignment. You build a Trigger Word Detector like the one you find in Amazon Echo or Google Home devices to wake them up. I solemnly pledge, my model understands me better than the Google Assistant — and it even has a more pleasant wake up word ;)
Final Thoughts
Finally, I would say, you can benefit most from taking this specialization, if you are relatively new to the topic. If you’re already familiar with the basics of NN, skip the first two courses. And if you are also very familiar with image recognition and sequence models, I would suggest to take the course on “Structuring Machine Learning Projects” only.
On the other hand, be aware of which learning type you are. If you are a strict hands-on one, this specialization is probably not for you and there are most likely courses, which fits your needs better. People say, fast.ai delivers more of such an experience. Also, if you’re only interested in theoretical stuff without practical implementation, you probably won’t get happy with these courses — maybe take some courses at your local university. Perhaps you are only interested in a specific field of DL, than there are also probably more suitable courses for you. Say, if you want to learn about autonomous driving only, it might be more efficient to enroll in the “Self-driving Car” nanodegree on Udacity. What you learn on this topic in the third course of deeplearning.ai, might be too superficial and it lacks the practical implementation. But, if you value a thorough introduction to the methodology and want to combine this with some hands-on experiences in various fields of DL — I can definitely recommend to do the deeplearning.ai specialization.
Doing this specialization is probably more than the first step into DL. I would say, each course is a single step in the right direction, so you end up with five steps in total. Nonetheless, I’m quite aware that this is definitely not enough to pursue a further career in AI. I think it builds a fundamental understanding of the field. But going further, you have to practice a lot and eventually it might be useful also to read more about the methodological background of DL variants (e.g. in the more advanced papers that are mentioned in the lectures). But doing the course work gets you started in a structured manner — which is worth a lot, especially in a field with so much buzz around it.
If you want to have more informations on the deeplearning.ai specialization and hear another (but rather similar) point of view on it: I can recommend to watch Christoph Bonitz’s talk about his experience in taking this series of MOOCs, he gave at Vienna Deep Learning Meetup. You can watch the recordings here.
Above all, I cannot regret spending my time in doing this specialization on Coursera. Most of my hopes have been fulfilled and I learned a lot on a professional level. It had been a good decision also, to do all the courses thoroughly, including the optional parts. I highly appreciate that Andrew Ng encourages you to read papers for digging deeper into the specific topics. Thereby you get a curated reading list from the lectures of the MOOC, which I’ve found quite useful.
And finally, my key take-away from this spezialization: Now I’m absolutely convinced of the DL approach and its power. Its major strength is in the scalability with lots of data and the ability of a model to generalize to similar tasks, which you probably won’t get from tradtional ML models. So, I want to thank Andrew Ng, the whole deeplearning.ai team and Coursera for providing such a valuable content on DL. And I definitely hope, there might be a sixth course in this specialization in the near future — on the topic of Deep Reinforcement Learning!
FYI, I’m not affiliated to deeplearning.ai, Coursera or another provider of MOOCs. I wrote about my personal experience in taking these courses, in the time period of 2017–11 to 2018–02. Also, this story doesn’t have the claim to be an universal source of contents of the courses (as they might chance over time). But I can definitely recommend to enroll and form your own opinion about this specialization.
