DEEP REINFORCEMENT LEARNING EXPLAINED – 05

Last year, Facebook announced that version 1.1 of PyTorch offers support for TensorBoard (TensorFlow’s visualization toolkit). TensorBoard provides the visualization and tooling needed for Deep Learning experimentation. Undoubtedly TensorBoard is a very useful tool to understand the behavior of neural networks and help us with hyperparameters during training.
TensorBoard is a visualization tool in the TensorFlow ecosystem, which can be used to plot various quantitative metrics and the results of several intermediate calculations. In this post we will emphasise the main features that we will need to start to use it. In future posts we will introduce new features as we need them. A detailed tutorial from PyTorch documentation that illustrates its functionality can be found here.
Next we are going to do a general description and summarise the steps to setup Tensorboard inside Colab, which will certainly be very useful to us in this series of posts.
Colab Environment
In this post we assume that you want to run TensorBoard into the Google research project Colaboratory (Colab). As we described in the previous post, it basically consists of a Jupyter notebook environment that requires no configuration and runs completely in the Cloud allowing the use different Deep Learning libraries as PyTorch and TensorFlow. Detailed information about the service can be found on the faq page.
Following this link you can execute the code of this post as a colab google notebook.
Setup TensorBoard
In order to setup TensorBoard you only need to follow the following steps:
1- import tensorboard
from torch.utils
defining aSummaryWriter
, our key object for writing information to TensorBoard:
from torch.utils.tensorboard import SummaryWriter
2- Default log folder where TensorBoard will be looking into for records to consume is runs
. We could be more specific:
writer = SummaryWriter('runs/working_directory')
Note that this line alone creates a runs/working_directory
folder.
To indicate that we have finished registering we can invoque the the close
method of the object:
writer.close()
3- Load the TensorBoard notebook extension:
%load_ext tensorboard
4- Launch the TensorBoard:
tensorboard --logdir=runs
Example of use
To show the use of TensorBoard we suggest to use the same code from previous post:
import torch
import torchvision
import numpy as np
EPOCHS = 10
BATCH_SIZE= 64
xy_trainPT = torchvision.datasets.MNIST(root='./data', train=True, download=True,transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor()]))
xy_trainPT_loader = torch.utils.data.DataLoader(xy_trainPT, batch_size=BATCH_SIZE)
def model(hidden):
model= torch.nn.Sequential(
torch.nn.Linear(784,hidden),
torch.nn.Sigmoid(),
torch.nn.Linear(hidden,10),
torch.nn.LogSoftmax(dim=1)
)
return model
modelPT = model(10)
criterion = torch.nn.NLLLoss()
optimizer = torch.optim.SGD(modelPT.parameters(), lr=0.01)
After running the next two lines of code:
from torch.utils.tensorboard import SummaryWriter
%load_ext tensorboard
We can already make the first training as in the previous post, but now we will also view a Loss chart. The following code highlights the added lines:
writer = SummaryWriter()
for e in range(EPOCHS):
running_loss = 0
for images, labels in xy_trainPT_loader:
images = images.view(images.shape[0], -1)
output = modelPT(images)
loss = criterion(output, labels)
loss.backward()
optimizer.step()
optimizer.zero_grad()
running_loss += loss.item()
print("Epoch {} - Training loss: {}".format(e,
running_loss/len(xy_trainPT_loader)))
writer.add_scalar("loss x epoch",
running_loss/len(xy_trainPT_loader), e)
writer.close()

We can see the result by invoking TensorBoard:
tensorboard --logdir=runs

Remember that the file with the data is stored in the runs
folder. We can see this by selecting the folder icon at the left side menu:

But if what we need is to know how the Loss goes down for each iteration (over 9000 iterations) we can’t do it with a simple print, we need a more graphical representation:
modelPT = model(10)
criterion = torch.nn.NLLLoss()
optimizer = torch.optim.SGD(modelPT.parameters(), lr=0.01)
def training_loop():
writer = SummaryWriter()
iter_no=0
for e in range(EPOCHS):
running_loss = 0
for images, labels in xy_trainPT_loader:
images = images.view(images.shape[0], -1)
output = modelPT(images)
loss = criterion(output, labels)
loss.backward()
optimizer.step()
optimizer.zero_grad()
running_loss += loss.item()
writer.add_scalar("loss x iter", loss.item(), iter_no)
iter_no += 1
writer.close()
training_loop()
The information for this run has been saved in a new directory within runs
folder. We can see the result by running the following command again:
tensorboar --logdir=runs

But perhaps what interest us is to compare, for example, the behavior of a neural network with more neurons. Let’s try this example of a neural network with 32 neurons:
modelPT = model(32)
criterion = torch.nn.NLLLoss()
optimizer = torch.optim.SGD(modelPT.parameters(), lr=0.01)
training_loop()
We can plot on the same graph by using the same name of the graph (in this example "loss x iter") **** when recording the data. The result for this case will be:

We can see the comparison of the loss behavior in both networks. Or for example what about another optimizer, instead of SGD the Adam?
modelPT = model(32)
criterion = torch.nn.NLLLoss()
optimizer = torch.optim.Adam(modelPT.parameters(), lr=0.01)
training_loop()

In this case we can see that it converges much better!
This has been just a small sample of the opportunities that TensorBoard offers. I invite the readers to explore for themselves this powerful tool, which also serves for other middlewares, especially TensorFlow, the framework in which TensorBoard was created.
See you in the next post!
Deep Reinforcement Learning Explained Series
by UPC Barcelona Tech and Barcelona Supercomputing Center
A relaxed introductory series that gradually and with a practical approach introduces the reader to this exciting technology that is the real enabler of the latest disruptive advances in the field of Artificial Intelligence.
About this series
I started to write this series in May, during the period of lockdown in Barcelona. Honestly, writing these posts in my spare time helped me to #StayAtHome because of the lockdown. Thank you for reading this publication in those days; it justifies the effort I made.
Disclaimers – These posts were written during this period of lockdown in Barcelona as a personal distraction and dissemination of scientific knowledge, in case it could be of help to someone, but without the purpose of being an academic reference document in the DRL area. If the reader needs a more rigorous document, the last post in the series offers an extensive list of academic resources and books that the reader can consult. The author is aware that this series of posts may contain some errors and suffers from a revision of the English text to improve it if the purpose were an academic document. But although the author would like to improve the content in quantity and quality, his professional commitments do not leave him free time to do so. However, the author agrees to refine all those errors that readers can report as soon as he can.