My experience with Udacity Deep Learning Foundations Nanodegree

I recently graduated from Udacity Deep Learning Foundations Nanodegree and want to share my experience I had with the program.

Motivation

After 20 years of pure software development in different areas, from image processing to web applications and at companies of different sizes, from start-ups (one of them I co-founded) to traditional German enterprises, I looked for something new. In last years we see huge progress in machine learning and, particular, artificial intelligence and its practical application. There are lot of fascinating products and researches where those techniques are applied. Since I had no idea, how it worked, but wanted to know, I decided to start a journey into this wonderland.

Way to Udacity

I’m a practical person and it is difficult for me to study something without applying it to the real problems. I took Data Science course at Coursera, but it was too academic for me.

Next I took famous Machine Learning course, taught by Andrew Ng. It is a brilliant entry point into the world of machine learning and I highly recommend it to everyone who want to study this field. Andrew Ng gave me another important idea— I started thinking about machine learning not only as a research field but as a toolkit to solve real-world problems.

I heard that Udacity offered practice-oriented courses and decided to gave it a try. Basically I wanted to enroll Udacity Artificial Intelligence course, but I felt myself not prepared enough for it. Luckily Udacity opened Deep Learning Foundations and advertised it as a preparation step for the Artificial Intelligence course, so I applied in January 2017 (first cohort).

Tooling

As a student you’re mainly working with Python, NumPy and TensorFlow. I strongly recommend to install Anaconda — it makes the life much easier. And do not worry — Udacity gives you a pretty good introduction to the tooling and development environment.

For me, as a professional developer, it was quite easy to catch up the libraries, I didn’t worked before.

Hardware

I worked mainly on my 6-years old laptop, Lenovo Thinkpad T510, with installed Ubuntu. It has Intel i5 processor, 8 GB RAM and no training-capable GPU. It was sufficient to develop models and fix bugs, but for the training it was really slow. Udacity provides 100$ credit for AWS and instructions how to set it up. I used GPU-powered EC2 instances and train my models there. Newly, Udacity is working close with FloydHub and you can choose to use it instead of AWS.

Course content

Deep Learning Foundations is all about neural networks, their background and concepts. As a student you implement simple neural networks from scratch using only NumPy and then going forward to (more) complex architectures and use more intelligent libraries such as TensorFlow. The course consists basically of 4 main parts:

• Introduction to neural networks, where you are taught, what a neural network is (inputs, outputs, hidden layers, weights, etc.), basic concepts such as forward and backward propagation and how to build a neural network.
• Convolutional Neural Networks (CNNs). Here you get an idea, what makes CNNs so special and what kind of problems (image classification, image recognition) they are able to solve.
• Recurrent Neural Networks (RNNs) and Long-Short Term Memory (LSTM) specifically. Here you get an impression about different architectures of RNNs and LSTMs and which problems you can solve with them (text generation, language translation, autoencoder).
• Generative Adversarial Networks (GANs). GANs are pretty new but they produces very impressive results, generating something reasonable from, literally, a noise.

Each part earns its own program, so do not expect very deep, rather a broad overview of capabilities. Then determine a most interesting topic (for me it was CNNs) and start studying more details about it.

How the students are taught

In contrast to most Udacity courses, Deep Learning Foundations is not self-paced. You have deadlines to make your work done. The course materials are portioned and released each week — 1 or 2 lessons per week plus short video from Siraj Raval, plus you can participate in his weekly live session. The content from Siraj is available for free — you can check it at YouTube.

This way of teaching was just right for me — I’m a bit lazy and need some push ;-) The content from Siraj I didn’t like at all and, at the end, gave up to watch his videos. I know, there are a LOT of people who love his lessons, but personally I couldn’t manage it.

The lessons structured is a way that you get an explanation of a certain topic and then should solve a problem yourself. If you’re stuck, an instructor provides his own solution and explains how it works. The sources with exercises and solutions are available at GitHub.

The lessons were of different quality. The best lessons were provided by Andrew Trask, Vincent Vanhoucke and Ian Goodfellow. The main instructor, Mat Leonard, did and is doing his best and I really appreciated that.

As a student of the first cohort I had a bad luck, since the course materials were not mature and Udacity made great improvement since then. I suppose as a student of later cohorts you will enjoy better quality :-)

Additional materials

During each lesson Udacity provides tons of links to additional materials to read and watch if you want dig deeper. I found those links very helpful and read and watched a lot, since I like to know how the things are working under the hood.

Frankly I expected more detailed explanations. But after thinking about it — an instructor cannot, for instance, explain CNN better than Andrej Karpathy. So why not just give a rough idea and let people read and watch additional materials from the experts if they want.

Projects

In total I implemented 5 projects:

• Simple neural network to predict daily bike rental ridership
• Image classification on CIFAR-10 data set using CNN
• Language translation using LSTM
• TV script generation using LSTM
• Face generation using GAN

The projects were demanding. As a student you get a prepared Python Notebook and necessary helper functions and should implement core parts of a neural network. So you do not implement everything from scratch and can use a lot from exercises, you worked on during the lessons. Upon submission the projects were reviewed. Usually after 1–2 hours I got quite valuable and personal feedback from a reviewer.

The only disappointment was a second project — in syllabus Udacity advertised “Image recognition” but then changed it to more simple “Image classification” :-(

You can check my solution at GitHub.

Community

Additionally to the forum, Udacity setup a slack channel where you could ask for a help, share you success, give feedback, report a problem and so on. I really appreciated and valued this way-of-doing. The community is really great and you never get stuck with problems.

Udacity staff are actively participating and they really hear to your feedback. That is one of the reason, why the quality of materials improved significantly.

Required time

It took me 4 months with 8–12 hours per week to finish and graduate. I could dump it to 5-6 hours per week skipping additional materials. I highly recommend invest time, reading more, going deeper and trying to create models yourself — it is great feeling to see, how your own work produce reasonable results!

Costs

The course costs 400$, it is not cheap. Does it provide value for the money? For me the answer is definitely “yes”. I enjoyed the course and learned a LOT last 4 months. Besides if I would calculate 100$ credits for AWS and 100$ credits for submission to Artificial Intelligence, Self-Driving Car or Robotics Nanodegree programs, then it costs less than 400$ :-) So the money were good investment.

Next steps

I consider Deep Learning Foundations as a preparation step for more demanding Artificial Intelligence Engineer Nanodegree program. I achieved my targets. Now I take a break and, most probably, will apply to that course. At the end I want to switch from pure software development to be a machine learning/artificial intelligence engineer and hope that Udacity Nanodegree programs will help me.

Conclusion

It was very interesting and stressful last 4 months (I have a full-time job and a family), I enjoyed a lot the learning materials, exercises, projects and community and would definitely do that again.

I warmly recommend this Foundations program to beginners and people with small experience who want to understand what problems you can solve with neural networks and how practically create solutions. From the other hand, experienced people could find the program boring or not-demanding enough.

Before applying you must be ready not only to read/watch the course materials and make the exercises. Instead you should plan time for additional reading and try-the-things-yourself if you want to take the most from the learning.

I want to thank Udacity team for a great experience that I had with the Deep Learning Foundations program.