Mastering Deep Reinforcement Learning with OpenAI’s new ‘Spinning Up in Deep RL’ package
Reinforcement Learning is a machine learning approach for teaching agents how to solve tasks by trial and error. Deep Reinforcement Learning refers to the combination of Reinforcement Learning with deep learning.
OpenAI released their educational package for Deep Reinforcement Learning, Spinning Up in Deep RL on Nov 8 ‘ 2018. Their release statement seemed quite appealing to me, which stated:
“At OpenAI, we believe that deep learning generally — and deep reinforcement learning specifically — will play central roles in the development of powerful AI technology. While there are numerous resources available to let people quickly ramp up in deep learning, deep reinforcement learning is more challenging to break into. We’ve designed Spinning Up to help people learn to use these technologies and to develop intuitions about them. We’ve also seen that being competent in RL can help people participate in interdisciplinary research areas like AI safety, which involve a mix of reinforcement learning and other skills. We’ve had so many people ask for guidance in learning RL from scratch, that we’ve decided to formalize the informal advice we’ve been giving.”
So, I decided to quickly glance through the whole package and here’s a short tour and a bit of advice for those who wish to go through the complete package.
What exactly is ‘Spinning Up in Deep RL’ ?
“We observed that it’s possible for people with little-to-no experience in machine learning to rapidly ramp up as practitioners, if the right guidance and resources are available to them. Spinning Up in Deep RL was built with this need in mind.”
The Spinning Up in Deep RL package as whole consists of :
Introduction to RL:
A brief introduction to reinforcement learning with short videos and moderately detailed short notes. The topics covered are Key Concepts in RL, RL Algorithms and Intro to Policy Optimization.
I found the overall learning material a little difficult to comprehend for a beginner. For any doubts regarding Reinforcement Learning basics, you can always refer to David Silver’s RL course and lecture series.
Deep-RL Research Papers:
Some of the top research papers in this field are arranged topic wise, further divided into sub-topics. The overall arrangements of papers is in a very suitable order for any beginner to comfortably go through them one-by-one.
Algorithms:
Some of the top algorithms like Vanilla Policy Gradient and Deep Deterministic Policy Gradient are already implemented in this package and are ready to use. They are all implemented with MLP (non-recurrent) actor-critics, making them suitable for fully-observed, non-image-based RL environments. The complete documentation for these algorithms can be found here.
Experiments and Environments:
Various experiments and OpenAI environments are incorporated within the package. An ExperimentGrid is also provided for Hyperparameter tuning. The whole interaction to experiments and environments can be mostly controlled via the Command Line (Shell).
Exercises:
Once, you are done with the learning oriented content in the package, there’s a couple of Problem Sets at the end to test your skills. Notably, there’s also an option for contributing to OpenAI’s standing requests for research.
Other Utilities:
Utils like Logger and Plotter are provided with the package to monitor and study the outputs and results from the experiments in a better way.
Suggestions:
The package shows high quality learning content which is far more than enough for you to be an active member in the Deep RL community, mostly application wise. But many sections of its learning content (like the research papers and algorithms’ implementation) can be a little bit hard to comprehend for beginners in the fields of Reinforcement Learning and Deep Learning. It would be better to at least hold a little bit of knowledge about some of the basic concepts in these fields before starting with this Spinning Up in Deep RL package by OpenAI.
I hope that you’ll kickstart your Deep Reinforcement Learning journey with this amazing resource from OpenAI. If you wish to suggest any other equally good or better learning resources for Deep RL, feel free to mention them in the comments below.
