From Tensorflow 1.0 to PyTorch & back to Tensorflow 2.0
It is a common situation where we have to move back and forth between libraries like Tensorflow and Pytorch for developing Deep Learning projects.
I started my journey in Machine Learning around 2015 when I was in my late teens. Without any clear vision about the field, I read many articles and watched a ton of YouTube videos. I did not have any clue what the field was or how it works. That was the time when Google’s popular Machine Learning library, Tensorflow was released.
Stepping into Tensorflow
Tensorflow was released in November 2015 as an ‘Open Source Software Library for Machine Intelligence’. As soon as it was released, people started jumping into it. The number of forks in GitHub just went up exponentially. But there was a fundamental flaw in it: ‘Static Graph’.
Under the hood, Tensorflow used static graph implementation to manage the flow of data and operations. This meant that from a programmer perspective, you had to first create a complete architecture of the model, store it into a graph, then launch the graph using a ‘session’. While this was a plus for creating production models, it lacked the pythonic nature.
The community started to complain about this problem and the Tensorflow team created ‘Eager Execution’ to fix it. But still, it wasn’t added as the default mode. Tensorflow also had a lot (a bit too much) of APIs that it became confusing. There was also a lot of redundancy for the same function definition.
Moving onto PyTorch
During this time (around 2017), I had to do a project that required simultaneous training of two neural networks with data flow between them. I started my work using Tensorflow as it was a mess. Multiple graphs, switching between graphs, manual selection of parameters for updating… a pure mess. Finally, I gave up on Tensorflow and started using PyTorch.
It was much easier using PyTorch… concrete definitions, stable designs, a common design structure. I loved it a lot. Still, I didn’t drop Tensorflow completely, I took workshops and lectures using Keras with a bit of Tensorflow and Tensorboard. Later, I completed my project, did a few more in PyTorch but I felt that the community wasn’t strong enough.
Back to Tensorflow
During this time Tensorflow was getting stronger, with tight integrations with packages, easy training with GPUs, and great support for production as Tensorflow Serving. It felt like TensorFlow was moving in a new direction.
On 1st October 2019, the first stable version of TF2.0 was released. The new features include tight integration with Keras, Eager Execution as default (finally 😅), functions and not sessions, Multi-GPU support and much more. Another beautiful feature is tf.function decorator, that converts a block of code into optimized graphs thus providing faster execution.
Also, APIs like tf.probability, tensorflow_model_optimization, tf.agents pushes the capabilities of Deep Learning into other domains. As a suggestion, I would love to see different packages for Reinforcement Learning like Tensorflow Agents, TFRL(truffle), Dopamine combined into a single API.
Thank You!!
