3 steps to update parameters of Faster R-CNN/SSD models in TensorFlow Object Detection API

I wrote an article on configuring TensorFlow Object Detection API. Kindly, refer to that story here to configure the API. That story is a prerequisite for this article.

In this story, I will discuss how to change the configuration of pre-trained model. The aim of this article is that you can configure TensorFlow/models based on your application and API will no longer be a black box!

The article’s overview:

• Understanding of protocol buffers and proto files.
• Using knowledge of proto files, how can we understand the config files of the model
• 3 steps to follow to update the parameters of the model
• Miscellaneous examples:

1. Changing the weight initializer
2. Changing the weight optimizer
3. Evaluating the pre-trained model.

Protocol Buffers

To modify the model, we need to understand it’s inner mechanisms. TensorFlow Object Detection API uses Protocol Buffers, which is language-independent, platform-independent, and extensible mechanism for serializing structured data. It’s like XML at a smaller scale, but faster and simpler. API uses the proto2 version of the protocol buffers language. I will try to explain this language which is required to update the pre-configured model. For more details on protocol buffers language, refer to this documentation and Python tutorial.

Working of protocol buffers can be explained in 3 steps as below:

• Define a message format in .proto file. This file acts like a blueprint of all messages, which shows what all parameters are accepted by the message, what should be the data type of the parameter, whether the parameter is required or optional, what is the tag number of the parameter, what is the default value of the parameter etc. API’s protos files can be found here. For the purpose of understanding, I am using grid_anchor_generator.proto file.

It is clear from the lines 30–33 that the parameters scales and aspect_ratios are mandatory for message GridAnchorGenerator, while rest of the parameters are optional if not passed, it will take default values.

• After defining a message format, we need to compile the protocol buffer. This compiler will generate the classes file from .proto file. During installation of API we had run the below command which will compile the protocol buffers:

# From tensorflow/models/research/
protoc object_detection/protos/*.proto --python_out=.

• After defining and compiling the protocol buffer, we need to use Python protocol buffer API to write and read messages. In our case, we can consider the config file as protocol buffer API, which can write and read messages, ignoring the inner mechanisms of TensorFlow API. In other words, we can update the parameters of the pre-trained model by appropriately changing the config file.

Understanding the config file

It is clear that config files can help us to change the parameters of the model according to our needs. Next question that pops up is how can we change the parameters of the model? This section and next section answers this question where knowledge of proto files will be handy. For demonstration purpose I am using faster_rcnn_resnet50_pets.config file.

Lines 7–10 implies that num_classes is one of the parameters of faster_rcnn message, which in turn is the parameter of the message model. Similarly, optimizer is the child message of parent train_config message, while batch_size is another parameter of train_config message. We can verify this by checking out the corresponding protos file.

Snippet of faster_rcnn.proto

It is clear from lines 20 and 26 that num_classes is one of the optional parameter of the message faster_rcnn. I hope discussion till now helps to understand the organization of config files. Now, it is the time to correctly update one of the parameters of the model.

Step 1: Decide the parameter to be updated

Let’s say we need to update image_resizer parameter mentioned in line 10 of faster_rcnn_resnet50_pets.config file.

Step 2: Search for the given parameter in the repository

The goal is to locate the proto file of the parameter. For that we need to search in the repository.

Search in the repository

We need to search the following code:

parameter_name path:research/object_detection/protos
#in our case parameter_name="image_resizer" thus,
image_resizer path:research/object_detection/protos

Here path:research/object_detection/protos confines the search domain. More information on how to search on GitHub can be found here. The output on searching image_resizer path:research/object_detection/protos is shown below:

Search result of image_resizer path:research/object_detection/proros

It is clear from the output that to update the image_resizer parameter we need to analyse the image_resizer.proto file.

Step 3: Analyze the proto file

image_resizer.proto file.

It is clear from lines 8–10 that we can resize the image using keep_aspect_ratio_resizer orfixed_shape_resizer. On analyzing lines 23–44, we can observe that message keep_aspect_ratio_resizer have parameters: min_dimension, max_dimension, resize_method, pad_to_max_dimension, convert_to_grayscale,and per_channel_pad_value. Moreover, fixed_shape_resizer have parameters: height, width, resize_method,and convert_to_grayscale. All the parameters have their datatypes mentioned in the proto file. So, to change the image_resizer type we can change the following lines in the config file.

#before
image_resizer {
keep_aspect_ratio_resizer {
min_dimension: 600 
max_dimension: 1024
    }
}
#after
image_resizer {
fixed_shape_resizer {
height: 600
width: 500
resize_method: AREA
  }
}

The above code will resize the image to 500 * 600 using AREA resize method. The various resize methods available in TensorFlow can be found here.

Miscellaneous Examples

We can update/add any parameter using steps discussed in the above section. I will demonstrate some of the examples here which are frequently used but the steps, discussed above can be helpful to update/add any parameters of the model.

Changing the weight intializer

• Decided to change the parameter initializer on line 35 of the faster_rcnn_resnet50_pets.config file.
• Search for the initializer path:research/object_detection/protos in the repository. From the search result, it is clear that we need to analyze the hyperparams.proto file.

Search result of initializer path:research/object_detection/proros

• Line 68–74 in the hyperparams.proto file, explains initializer configuration.

message Initializer {
  oneof initializer_oneof {
    TruncatedNormalInitializer truncated_normal_initializer = 1;
    VarianceScalingInitializer variance_scaling_initializer = 2;
    RandomNormalInitializer random_normal_initializer = 3;
  }
}

We can use random_normal_intializer instead of truncated_normal_initializer, for that we need to analyze line 99–102 in the hyperparams.proto file.

message RandomNormalInitializer {
  optional float mean = 1 [default = 0.0];
  optional float stddev = 2 [default = 1.0];
}

It is clear that random_normal_intializer have 2 parameters mean and stddev. We can change the following lines in the config file to use random_normal_intializer.

#before
initializer {
    truncated_normal_initializer {
        stddev: 0.01
       }
}
#after
initializer {
    random_normal_intializer{
       mean: 1 
       stddev: 0.5
       }
}

Changing the weight optimizer

• Decided to change the parameter momentum_optimizer of parent message optimizer on line 87 of the faster_rcnn_resnet50_pets.config file.
• Search for the optimizer path:research/object_detection/protos in the repository. From the search result, it is clear that we need to analyze the optimizer.proto file.

Search result of optimizer path:research/object_detection/proros

• Lines 9–14 in optimizer.proto file, expain optimizer configuration.

message Optimizer {
  oneof optimizer {
    RMSPropOptimizer rms_prop_optimizer = 1;
    MomentumOptimizer momentum_optimizer = 2;
    AdamOptimizer adam_optimizer = 3;
  }

It is clear that instead of momentum_optimizer we can use adam_optimizer which has been proved to be good optimizer. To do that we need following changes in the faster_rcnn_resnet50_pets.config file.

#before
optimizer {  
  momentum_optimizer: {
      learning_rate: {
           manual_step_learning_rate {
          initial_learning_rate: 0.0003
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
#after
optimizer {
  adam_optimizer: {
      learning_rate: {
       manual_step_learning_rate {
          initial_learning_rate: 0.0003
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
    }

Evaluating pre-trained model

Eval waits for 300 seconds to check whether training model is updated or not! If your GPU is good, you can train and eval both simultaneously! Generally, resources will be exhausted. To overcome this, we can train the model first, save it in the directory, and evaluate the model later. To evaluate later, we need following changes in the config file:

#Before
eval_config: {
  num_examples: 2000
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 10
}
#after
eval_config: {
num_examples: 10
num_visualizations: 10
eval_interval_secs: 0
}

num_visualizations should be equal to number of to be evaluated! Higher the number of visualizations, more time it will take to evaluate. If your GPU is capable enough to train and eval simultaneously, you can keep eval_interval_secs: 300. This parameter is to decide how often to run evaluation. I followed 3 steps discussed above to come to this conclusion.

In a nutshell, the knowledge of protocol buffers helped us to understand that parameters to the model are passed in the form of messages and to update the parameter we can refer .proto file. 3 simple steps were discussed to find the correct .proto file for updating the parameter. Recently, I came across this article on neptune.ai which can be useful for further reference.

Kindly mention any parameter you want to update/add in the config file in the comments.

I hope this article helps.

-SodhA