KERAS : Under the Hood

INTRODUCTION

Getting started with Deep Learning has become very simple and convenient, all thanks to wonderful duo of keras and tensorflow. You just need to do some imports, define some layers and bingo, you have your deep learning architecture ready to be trained and eventually give some amazing results. Keras has made such an amazing abstraction that even a total stranger to the topic as well can start training their own deep learning models. However if you are calling yourself a Data Scientist/Machine Learning Engineer then having some basic understanding of what’s happening under the hood is a must, I am not saying you need to exactly know the hundreds of lines of code behind it but at least have a some understanding what those lines of code are doing.

GETTING STARTED

Without wasting anytime lets dive into some of the most commonly used Keras components and try to understand them piece by piece, this will involve a basic understanding of Object Oriented Programming but don’t worry I will try to keep it as simple as possible. I will take a very simple deep learning architecture and and then try to explain the working line by line, in order to explain things better I have created create a custom dense layer that I would be using in my architecture.

class MyDenseLayer(tf.keras.layers.Layer):

    def __init__(self, nodes):
        super(MyDenseLayer, self).__init__()
        self.num_outputs = nodes

    def build(self, input_shape):
        print("="*20)
        print('we are here in build')
        self.kernel = self.add_weight("kernel",shape=      [int(input_shape[-1]),self.num_outputs],dtype=tf.float32)        

    def call(self, input):
        print('='*20)
        print('we are here in call')
        return tf.matmul(input, self.kernel)

You might be wondering what are these build/call methods inside, I will dig deeper into their details in a while and below is a very simple model architecture using keras Functional API.

input_layer = Input(shape = (5,))
dense_layer  = MyDenseLayer(nodes=10)
dense_layer_op = dense_layer_one(input_layer)
model = Model(inputs = input_layer,outputs = dense_layer_op)

Now let us look at the architecture and try to understand it line by line. First line is pretty straightforward, we are creating an object of a class called Input which basically creates a symbolic tensor/placeholder (it means we don’t need actual input data here, instead a block of memory is reserved for data to flow in the network later) this tensor tells the network what will be the size of the input that would be given to the model and this is generally the entry point into the network, in the shape argument we specify the dimensions of input vector. ( (5,) means input is 1×5 vector and the batch size is not decided yet i.e model can take any batch size that would be defined later during model fitting)

Second and third lines are where the actual thing begins, here we define the very first layer of our neural network architecture. Second line creates an object of our custom layer class, this line basically defines the number of nodes in our custom layer.

Third line is where our actual layer is built, weights of the layer are created and initialized here, you might be having some questions like why the weights are not initialized during layer object creation itself?, moreover how is this layer object being used as some python method?, if so then what is this method doing?. Let me answer them one by one, at the time of object creation we might not know the size of the input beforehand (if we were to create weights during object creation then we would need to pass the size explicitly every time as an argument to the class) and without the size of the input we cannot create this weight matrix. Hence we want to create these weights lazily only when the size of inputs is known, which is what happens in our third line, here once we get the information of the input shape we build our layer fully, to prove my point I have fetched the layer weights after creating the layer object and after passing the input shape to this layer object respectively, you can clearly see weights are initialized only when the input shape to the layer is known, elegant right?.

now comes the second question how is this object being treated as a python method and if it is a method then what is it doing?, before that let us talk first a bit about these methods that we defined in our custom class.

init () : This is one is pretty straightforward, for those of you who have worked with other programming languages like JAVA this init is equivalent to a constructor this is used to initialize a class object.

build () : As the name suggests, this method is used to actually build the layer i.e it creates the layer weights, this is called once we get the shape of the input that would be fed to the model and it creates the weights accordingly

call () : This is where logic of the layer lives, i.e it takes the input that is being fed to the layer processes it and spits out the output of that layer, which might be used by other layers

We discussed about what these methods do, but we didn’t talk about one very important thing, how and when are these methods invoked, init is pretty straightforward it gets invoked at the time of object creation, as for these build and call methods these are invoked at the time when we are using our layer object as a method, it might get a bit confusing but please be patient, lets break it down, as of now we know the what part i.e on being treated as a method what is this object actually doing, now comes the how part, how is this object being treated as a method, how are the call and build methods called without any explicit function calls.

well this is a special kind of object – the callable object, this object can be treated as an function with a bit of magic. That is exactly what is required to make this magic happen, a magic method, python has these special types of methods whose invocation does not require any explicit function call. They are called internally ( called by some operator overloading or built-in functions), we may not realize it but we have been using some of these magic methods quite extensively already , here is a quick example, in python we know everything is an object when we do something like a = 5 then actually a is an object of a class str, when we do print(a) how is the print method able to generate this output here is where another magic function comes into play called str, this method returns the output of an object in string form and it is invoked automatically when we put that object inside print method, again here is a small code piece to make you more comfortable around the above statement.

class PrintTest():
    def __init__(self,obj):
        self.obj = obj
    def __str__(self):
        return "__str__ method is invoked when print object is put inside print/str method"
str_obj = PrintTest('Hello')

And here are the outputs when we put an object of the PrintTest class inside built in methods like print.

( Note : In python there is class called Object which is by default a parent class to every class created in python this str is already defined there, here in our example we had simply overridden the str method. )

Now coming back to our original discussion, there is another magic method called call that helps your class object work as regular method, it is invoked automatically once we put brackets round brackets () after of the object name, in simple words obj() is equivalent to obj.call() .

So this why when we create an object of our original layer, and then do a function call using the same object it works, keen observers would have noticed one thing, that in our custom layer class we did not explicitly define any call method, we do have a call method but it is a regular method, it ain’t no magic method, well to answer this I would like to touch upon a concept of OOPs called Inheritance, you must have noticed our custom layer class has inherited a base class called Layer from Keras , you might be aware of basic concept of inheritance, once a class inherits another class (parent class) it gets access to all the functionality (methods) of that parent class so this magic method call is written in the Layer class and when we inherit it our custom class gets access to this call magic method. These build and call methods are also not some random methods, they also have a significance in layer building. These methods are actually already defined in the Base Layer Class, and they have these special roles to play that we discussed above already.

Both of these methods are called via call method, so to summarize the whole flow once we pass a tensor/placeholder through layer callable object then call method is invoked which in turn calls first the build method to instantiate the layer and later call method to run the logic of the layer. To give the proof of my statement I had added print statements inside the call and build methods, let’s see the output once the input tensor/placeholder is passed to the callable layer object.

One thing you would have noticed by now that while defining architectures we don’t need the actual data, we just need the shape of tensors that would be needed by the layer, and a placeholder is created for that tensor, the actual data is passed during model.fit() and the data forward propagates through the architecture and later error/loss is calculated which then back propagates hence updating the weights in the process and training the model.

CONCLUSION

Though these things might look small, but knowing them helps specially when we move into more complex deep learning architectures like LSTM’s, encoder-decoders, etc and having a bit of understanding of these basic concepts helps us later to get a better hang of these complex architectures and eventually modify and create our own.