Pushing the Limits of Deep Image Inpainting Using Partial Convolutions
Image Inpainting for Irregular Holes Using Partial Convolutions
Hi. Today, I would like to talk about a good deep image inpainting paper which has broken some limitations on previous inpainting work. In short, most of the previous papers assume that the missing region(s) is/are regular (i.e. a center missing rectangular hole or multiple small rectangular holes) and this paper proposes a Partial Convolutional (PConv) layer to deal with irregular holes. Figure 1 shows some inpainting results using the proposed PConv. Are they good? Let’s grasp the main idea of PConv together!
Motivation
First of all, previous deep image inpainting approaches treat missing pixels and valid pixels the same in the sense that they fill a fixed pixel value (255 or 1 before/after normalization) to all the missing pixels in an image and apply standard convolutions to the input image for the task of inpainting. There are two problems here. i) Is it appropriate to fix the pixel values of the missing pixels to a pre-defined value? ii) Is it suitable for convolving the input image regardless of the validness of the pixels? So, it may be a good option to perform operations only on valid pixels.
Secondly, existing approaches assume that the missing region(s) is/are regular/rectangular. Some of them employ local discriminator to distinguish generated content from real content. For the case of a fixed size center missing hole, they input the filled center hole to the local discriminator for enhancing the local texture details. But how about irregular missing areas? Is it possible to have fine local details without the employment of a discriminator? If you have tried using the existing approaches for handling irregular masked images, you can find that the inpainting results are not satisfactory as shown in Figure 2. For practical inpainting methods, they should be able to handle irregular masked images.
Introduction
Similar to my previous posts, I assume that readers have a basic understanding of deep image inpainting such as network architecture, loss function (i.e. L1 and Adversarial losses), and related terminology (i.e. valid pixels, missing pixels, etc.). If you need or want, please have a quick recall of my previous posts. In this section, I will briefly go through the things that I think are less important, hence we can have more time to dive into the most important idea of this paper, Partial Convolution, in the following sections.
The authors of this paper employ a U-Net like network with skip connections in which all standard convolutional layers are replaced by the proposed partial convolutional layers. If you are interested in their network architecture, you may refer to the paper and they provide detailed tables of their model.
Interestingly, no discriminator is used in this work. Apart from standard L1 loss and total variation loss (TV loss), the authors adopt two high-level feature losses to complete the masked images with fine textures. I will introduce these two losses in detail later on.
Solution (spotlight)
As mentioned in Motivation, the key idea is to separate the missing pixels from the valid pixels during convolutions such that the results of convolutions only depend on the valid pixels. This is the reason why the proposed convolution is named partial convolution. The convolution is partially performed on the input based on a binary mask image that can be updated automatically.
Approach
Partial Convolutional Layer
Let us define W and b be the weights and bias of the convolution filter. X represents the pixel values (or feature activation values) being convolved and M is the corresponding binary mask which indicates the validness of each pixel/feature value (0 for missing pixels and 1 for valid pixels). The proposed partial convolution is computed,
where ⦿ means element-wise multiplication and 1 is a matrix of ones that has the same shape as M. From this equation, you can see that the results of the partial convolution only depend on the valid input values (as X ⦿ M). sum(1)/sum(M) is a scaling factor to adjust the results as the number of valid input values for each convolution is varying.
Update the binary mask after each partial convolutional layer. The proposed rule to update the binary mask is quite easy. If the result of the current convolution is conditioned on at least one valid input value, the corresponding location will be regarded as valid for the next partial convolutional layer.
as you can see above, the updating rule is simple to understand.
Figure 3 shows a simple example to illustrate the proposed partial convolution. We consider a simple 5×5 input with its corresponding 5×5 binary mask image (1 for valid pixels and 0 for hole pixels) and a 3×3 W with fixed weights. Assume that we want to keep the same output size as the input size 5×5, hence we perform zero paddings before doing the convolution. Let’s consider the top-left corner (orange-bounded) first. X and M for this convolution are clearly shown in the figure and the number of valid input values is 3. Hence, the output of this location is -9+b. Also, the value of the corresponding location in the updated binary mask is 1 as there are 3 valid input values.
Considering the middle (purple-bounded) box, this time, as you can see, there is no valid input value for this convolution, so the result is 0+b and the updated mask value is also 0. The bottom-right (blue-bounded) box is another convolution example for showing the role of the scaling factor. By the scaling factor, the network can distinguish -3 computed by 3 valid input values from -3 computed by 5 valid input values.
For readers’ information, the updated binary mask after this partial convolutional layer is shown in the top-right corner of Figure 3. You can see that there are fewer zeros in the updated binary mask. When we perform more and more partial convolutions, the binary mask will be eventually updated to have all ones. This means that we can control the information to be passed inside the network regardless of the size and the shape of the missing regions.
Loss Functions
In total, there are 4 loss terms in their final loss function, namely L1 loss, Perceptual loss, Style loss, and TV loss.
L1 loss (per-pixel loss)
This loss is for ensuring the pixel-wise reconstruction accuracy.
where I_out and I_gt are the output of the network and the ground truth respectively. M is the binary mask, 0 for holes, and 1 for valid pixels. N_I_gt is the total number of pixel values in an image, which equals C×H×W, C is the channel size (3 for RGB image), H and W are the height and width of the image I_gt. You can see that L_hole and L_valid are the L1 loss of the hole pixels and valid pixels respectively.
Perceptual loss (VGG loss)
The perceptual loss is proposed by Gatys et al. [2] and we have introduced this loss previously. Simply speaking, we want the filled image and the ground truth image to have similar feature representations computed by a pre-trained network like VGG-16. Specifically, we feed the ground truth image and the filled image to a pre-trained VGG-16 to extract features. Then, we calculate the L1 distance between their feature values at all or several layers.
For the above equation, I_comp is the same as I_out except the valid pixels are directly replaced by the ground truth pixels. Ψ^I_p is the feature maps of the p-th layer computed by a pre-trained VGG-16 given the input I. N_Ψ^I_p is the number of elements in Ψ^I_p. According to Gatys et al. [2], this perceptual is small when the completed image is semantically close to its ground truth image. Perhaps, it is because deeper layers (higher level) provide more semantic information of an image and similar high-level feature representations represent better semantic correctness of the completion. For readers’ information, VGG-16 pool1, pool2, and pool3 layers are used to compute the perceptual loss.
Style loss
Apart from the perceptual loss, the authors also adopt the style loss as shown in the above. You can see that the style loss is also computed using the feature maps given by a pre-trained VGG-16. This time, we first calculate the auto-correlation of each feature map and it is called Gram matrix in [2]. According to [2], the Gram matrix contains the style information of an image such as textures and colours. This is also the reason why this loss is named Style loss. Thus, we compute the L1 distance between the Gram matrices of the completed image and the ground truth image. Note that Ψ^I_p is with size of (H_p×W_p)×C_p and its Gram matrix is with shape of C_p×C_p. K_p is a normalizing factor which depends on the spatial size of the feature maps at p-th layer.
Total Variation loss
The last loss term in their final loss function is the TV loss. We have talked about this loss in my previous posts. Simply speaking, this loss is adopted to ensure the smoothness of the completed images. This is also a common loss in many image processing tasks.
where N_I_comp is the total number of pixel values in I_comp.
Final loss
This is the final loss function to train the proposed model. The hyper-parameters used to control the importance of each loss term are set based on experiments on 100 validation images.
Ablation study
The authors did experiments to show the effects of different loss terms. The results are shown in Figure 4 in the above. First of all, Figure 4(b) shows the inpainting result without using style loss. They found that the use of the style loss is necessary in their model to generate fine local textures. However, the hyper-parameter for the style loss has to be carefully selected. As you can see in Figure 4(f), small weight for the style loss would cause some obvious artifacts as compared to the result using the full loss (Figure 4(g)). Apart from the style loss, the perceptual loss is also important. They also found that the employment of the perceptual loss can reduce the grid-shaped artifacts. Please see Figure 4(j) and (k) for the effect of the use of the perceptual loss.
In fact, the use of the high-level feature loss has not been fully studied. We cannot 100% say that the perceptual loss or the style loss must be useful for image inpainting. So, we have to do our own experiments to check the effectiveness of different loss terms for our desired applications.
Experiments
In their experiments, all the mask, training, and testing images are with the size of 512×512. The authors divided the testing images into two groups, i) mask with holes close to border. ii) mask without holes close to border. Images with all the holes with distance of at least 50 pixels from the border are classified into the second group. Figure 5 shows some examples of these two groups of masks. Furthermore, the authors generate 6 types of masks according to the hole-to-image area ratios: (0.01, 0.1], (0.1, 0.2], (0.2, 0.3], (0.3, 0.4], (0.4, 0.5], and (0.5, 0.6]. This means that the largest mask would mask out 60% of the original image content.
Training data. Similar to previous work, the authors evaluated their model on 3 publicly available datasets, namely, ImageNet, Places2 and CelebA-HQ datasets.
Figure 6 and 7 show the visual comparisons of different approaches on ImageNet and Places2 respectively. PatchMatch is the state-of-the-art conventional approach. GLCIC and Contextual Attention are two state-of-the-art deep learning approaches we have introduced before. As you can see, GLCIC (c) and Contextual Attention (d) cannot offer inpainting results with good visual quality. It may due to the fact that these two previous deep learning approaches are trained for regular masked images instead of irregular masked images. If you are interested, please zoom in for a better view of the inpainting results.
Figure 8 shows the inpainting results on CelebA-HQ dataset. You may zoom in for a better view of the results.
Table 1 lists out several objective evaluation metric numbers for readers’ information. Clearly, the proposed PConv offers the best numbers in nearly all the cases. Note that IScore is the inception score which is used as an estimation of the visual quality, and the lower the better estimated visual quality.
Apart from the qualitative and quantitative comparisons, the authors also conducted a human subjective study to evaluate the visual quality of different approaches. Interested readers may refer to the paper for the study.
Limitation
At the end of this paper, the authors also mention some limitations of the current deep image inpainting approaches. First, it is difficult to complete an image with a large missing area as shown in the right of Figure 9. Second, when the image contains complex structure, it is also difficult to complete the image with good visual quality as shown in Figure 10. There is still no a comprehensive method to handle extreme large masked and complicated images. So, you may try to propose a good solution to this extreme image inpainting problem. :)
Conclusion
Obviously, Partial Convolution is the main idea of this paper. I hope that my simple example can explain clearly to you how the partial convolution is performed and how a binary mask is updated after each partial convolution layer. By using Partial Convolution, the results of convolution would only depend on valid pixels, hence we can have the control of the information pass inside the network and this may be useful for the task of image inpainting (at least the authors provide evidence that partial convolution is useful in their case). Apart from image inpainting, the authors have also tried to extend the partial convolution to the task of super-resolution as it shares similarity with image inpainting. Interested readers are highly recommended to refer to their paper.
Takeaways
Without a doubt, I hope you can understand what is partial convolution. Starting from this paper, the later deep image inpainting methods can deal with both regular and irregular masks. Together with my previous posts related to image inpainting, I also hope that you can have better understanding of the field of image inpainting. You should know some common techniques and remaining challenges in image inpainting. For example, dilation convolution, contextual attention layer, etc. It is also difficult to fill in an image when the hole(s) in an image is/are too large and the image is with complex structure.
What’s Next?
Next time, we will look at another paper which makes use of additional information to help filling in the masked images. Hope you enjoy! Let’s learn together! :)
References
[1] Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro, “Image Inpainting for Irregular Holes Using Partial Convolution,” Proc. European Conference on Computer Vision (ECCV), 2018.
[2] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge, “A Neural Algorithm of Artistic Style,” arXiv preprint arXiv:1508.06576, 2015.
Thanks for reading my post! If you have any questions, please feel free to send my an email or leave comments here. Any suggestions are welcome. Thank you very much again and hope to see you next time! :)
