Hands-on Tutorials

Introduction
It is well known among deep-learning manias that bilinear upsampling layers in TensorFlow have pixel-offset issues. This has been partly fixed by adding an ‘align_corner’ attribute to them in TensorFlow 2.x. But the problem remains to cause inconsistent computation flow when exporting a trained model in TensorFlow into another DL framework through various versions.
In my case, a neural network model with bilinear upsampling layers showed weird behavior when converting the trained model from TensorFlow 2.5 to Apple Core ML by using coremltools 3.4. After uncountable coding, trials, and delete-delete-delete, I nearly gave up the consistent results of the upsampling layer between TensorFlow and Core ML.
I wanted to use Keras in the latest TensorFlow 2.5 for training in Windows PC, and I wanted to use the previous coremltools 3.4 for converting the trained model to Core ML for my macOS laptop. This is because the version 2.5 has stable Automatic Mixed Precision computing, and because I could not use coremltools 4.x for TF 2.5 due to dependency errors in anaconda and pip in macOS.
What is the pixel-offset issue?
Some respectful programmers provide good explanations of this (troublesome) specification defined in TensorFlow. They are helpful for me, and maybe for you too:
- Oleksandr Savsunenko – How Tensorflow’s tf.image.resize stole 60 days of my life
- Matthijs Hollemans – Upsampling in Core ML
- Bart Wronski – Bilinear down/upsampling, aligning pixel grids, and that infamous GPU half pixel offset
Problem to be Solved
"coremltools 3.4 cannot convert a bilinear upsampling layer in TensorFlow 2.5 properly."
This is not a surprising thing, just a version mismatch in which TensorFlow 2.5 can align the image corners in that layer, but coremltools 3.4 uses the original method that tends to shift the images as TensorFlow 1.x has done.
However, anyway, I had to get consistent results in my macOS from my trained Keras model with TensorFlow, regardless of their versions…
Implementing Bilinear Upsampling Layer
I just upload my efforts (=code) here. They are custom layers of TF-Keras and Core ML for bilinear upsampling.
In Keras
It is a simple custom layer without any trainable parameters. You must implement call() to calculate the tensors. I use tf.compat.v1.image.resize_bilinear() but tf.compat.v1.image.resize() will be equivalent. Note that you must use align_corners=True. You may be able to use other upsampling or image scaling methods in TensorFlow.
In Core ML (objective C)
NOTE: I updated the Core ML implementation in July 2nd, 2021. Please refer to the latest contents at the bottom too.
In your custom layer class based on MLCustomLayer, you must implement encodeToCommandBuffer method to activate GPU. Apple’s Metal Performance Shaders provide MPSCNNUpsamplingBilinear which works well, and the result seems consistent with that of tf…image.resize_bilinear used in Keras above. Thanks to this MPSCNNUpsamplingBilinear, I did not have to write METAL code by myself.
Honestly, I used MPSImageBilinearScale, not MPSCNN…, at first. The result was obviously unstable and unexpected. In my understanding, they should be the same ‘bilinear’ and ‘upsampling (=rescaling)’ with aligned corners, but they are not. I guess that MPSCNN… is a pure bilinear upsampling because the result is also consistent with the bare code for CPU. While MPSImage… may be have some customizations to keep good looking in rescaled images.
The additional work is evaluateOnCPUWithInputs to use CPU instead. You must implement the method for the case a computer cannot use or does not have GPU in it. I implemented bilinear upsampling code based on Matthijs‘s website (his guidance is very helpful).
In Converter (coremltools 3.4)
It is impossible to convert a trained model of TF-Keras 2.5 into a Core ML model directly using the previous coremltools 3.4. coremltools…convert() or keras…load_model() methods do not work. But it is possible by using them indirectly as follows:
- I newly build a bare model from my neural network code that had been used for the training in the latest TF-Keras 2.5. Note that I use pure Keras 2.2.x for this empty model, instead of TF-Keras.
- Because the versions of pure Keras 2.2.x and coremltools 3.4 match, the bare model can be converted into a Core ML model by coremltools 3.4. But the model does not have trained parameters.
-
I use keras_model.load_weights() to load the trained parameters of TF-Keras 2.5 into the newly built model of pure Keras 2.2.x. This works well, and coremltools 3.4 can create a trained Core ML model from it.
Finally,
I was relieved that I could implement a bilinear upsampling layer keeping consistent computation results through TensorFlow 2.5 and Core ML using coremltools 3.4.
I hope this will be some help for someone trying this issue.
P.S. I’m saving money for coming MacBook Pro 2021… but when it comes?
Update [July 2, 2021]
I found some cases in which the above MPSCNNUpsamplingBilinear showed inconsistent results, even though it seemed to be consistent in other cases at first. Thus, I tried another Metal Performance Shader function, MPSNNResizeBilinear to implement the Keras counterpart in Core ML, ** and found it had the better result**.
The following pictures show the segmentation results of a trained neural network model with bilinear upsampling layers using the above Keras code.
The red area is the result of the code for CPU, the red line is that of MPSNNResizeBilinear and the yellow is MPSCNNUpsamplingBilinear.

And, I updated the objective C code for Core ML as follows: