How to make your Operation Team Proud

Several weeks ago one of our business unit members told me that "life can be more convenient" if they will be able to run our DL engines on their servers. The notion "servers" means running on Cygwin engines where python is not always installed and if it is, the version is unknown and obviously no packages such as Keras or Pytorch can be installed (due to inner constrain). Ideally speaking from his side was having a shell tool that performs DL tasks. At that time I knew well how Torch or Keras work when I use them on my PC, but taking a models and "Cygwin" them? I knew nothing about. Probably some of the readers are claiming now, "well it is trivial" they are right. However, when I began to work on it I found that there are many websites that discuss many of the actions but nothing is an end to end description. This is the motivation of this post.
Early Steps
When I began to to work, I was aspiring to develop the entire solution on Cygwin (dreams always push you forward). Therefore the early steps I took were throughout Cygwin installer and only by using it. The first "station" on my way was the select packages screen.

In order to add packages, one has to set this screen to Full rather pending. I ** installed Openssh git and curl** (Installation is performed as seen in the image below)


After picking these packages and moving forward with the installation process, one can open the Cygwin screen and see the following:

We can verify that our installations worked well

It can be see that curl and git are located within the Cygwin. Python? the Cygwin identifies my PC’s Python (Indeed, I assume that if you read this post you are likely to have a Python on your PC). Since this project is developed for environments that don’t necessarily have an installed Python I wanted the entire work to be done from Cygwin. I returned therefore to the installation process and added Python in the Select Packages screen (Clearly, Cygwin offers many versions.. I have no favorite one)

After the installation "which python" outputs this answer

We can even verify that Python works within Cygwin as the two screens below present:


As I mentioned in the beginning, the objective is having shell files running from a Cygwin screen. Hence, it is time to check if Cygwin allows this (very trivial but tests are needed). I wrote a tiny toy file (uu1.py , I am outstanding with names) and saved it in home/username under the Cygwin.
if __name__=="__main__":
a=2
b=a+3
print("gggg ", a, b)
Before running, write in the beginning of the file :
#!/usr/bin/python3
The folder is the one that obtained from which python3
In the Cygwin screen you write
chmod a+x file_name (in my case chmod a+x uu1.py)
Then simply ./uu1.py. We got:

OK, Python works. Now we need to bring some extra libraries (e.g. numpy, matplotlib etc.)
Obviously we need to return to the Cygwin installer. Select Packages allows installing both numpy and matplotlib. This installation is pretty slow since there are several packages that it adds. . After the installation was completed, I added some code to uu1.py:
if __name__=="__main__":
a=2
b=a+3
xx=[1,1,8]
plt.plot(xx, 'r', label="trial Cyg")
plt.savefig("mygraph2.png")
print(xx)
x = np.array([1, 2, 5.])
print(x)
print("gggg ", a, b)
print(np.random.randn(3, 4))
he output was :

Undoubtedly numpy works. What about matplotlib?
Checking the folder (home/natank/) I got the required file with the proper graph. For me it was sufficient. Those who wish to get graphs on the screen from plt.show() can read here
What about SciPy?
As you probably noticed, until now, we didn’t use pip at all.
For Scipy it is needed. Before moving forward , verify that you use the right pip , namely the one that is located within Cygwin

In the screen above, we need to use the lower pip (pip3). The upper is related to another Python
If you don’t have pip in the Cygwin, you can go and install it from the Selected Packages screen.
Installing Scipy is trivial: simply using pip3 install Scipy. However, this part can be a huge a time consumer. Although some of the required packages were already installed (probably during the early installation of numpy and matplotlib) I still needed to return to Sleeted Packages and install additional ones. Reading [here](https://github.com/python-pillow/Pillow/issues/2860), here and here, was extremely beneficial: . I needed to add the entire list:
liblapack-devel
libopenblas
wget
python37-devel
python37-paramiko
python37-crypto
gcc-fortran
python-gtk2
- tcl-tk
libpng
- pkg-config
- libffi – devel
- zlib-devel
In some cases you may need to add python37-wheel
Now I managed to install Scipy. I even asked uu1.py’s approval
#!/usr/bin/python3
import numpy as np
import matplotlib.pyplot as plt
import scipy as sc
import os
from scipy.special import cbrt
if __name__=="__main__":
a=2
b=a+3
xx=[1,2,8]
plt.plot(xx, 'r', label="trial Cyg")
plt.savefig("mygraph2.png")
print (xx )
x = np.array([1, 2, 5.])
print (x)
cb = cbrt([27, 64])
print("scipy ",cb)
print ("gggg ",a,b)
print (np.random.randn(3,4))

So it all works. I was now confident that running Keras using Cygwin is two minutes away. But, as you may know a minute’s length depends on the observer and two minutes may become two days spent mostly on fighting with Tensorflow rejection errors
I had to find another framework. One option was offered here. I decided to try creating exe files using my own Python and verify that Cygwin supports it
Creating Exe in Anaconda
PyInstaller
According to the web, there are several methods to create .exe files in Python. The package that I decided to test was Pyinstaller which is installed using a regular pip.
How Does it actually work ?
I will use the following toy example test_dl.py:
import numpy as np
from keras.layers import Input,Dense,Dropout,concatenate
from keras.models import Model
import sys
if __name__ =='__main__':
print (sys.argv)
inp_dim =int(sys.argv[1])
batch_size=int(sys.argv[2])
r0 = np.random.randn(10,2*inp_dim)
tt = Input(shape=(inp_dim,))
tt1= Input(shape=(inp_dim,))
xx=Dense(256,activation="relu")(tt)
xx1 = Dense(256)(tt1)
xx= concatenate([xx,xx1],axis=1)
xx = Dropout(0.1)(xx)
xx = Dense(256,activation="relu")(xx)
xx = Dropout(0.1)(xx)
xx= Dense(100,activation="relu")(xx)
xx = Dropout(0.1)(xx)
xx =Dense(1,activation="sigmoid")(xx)
model = Model(inputs=[tt,tt1], outputs=xx)
model.compile(loss='binary_crossentropy', optimizer='adam')
print (model.summary())
zz = model.predict([r0[:,:inp_dim],r0[:,inp_dim:]])
print (zz)
print (sys.argv[3])
We set our CMD screen (or anaconda prompt) to the folder that contains our source and type the following "Pyinstaller – onefile file_name.py"

When this process ends, you will see in the project viewer two new folders build and dist and a file with the name file_name.spec

Inside dist you will find the .exe file

There are two ways to run it:
Go to the dist folder using cd dist and run (with the arguments for sys.argv)

Or copy to the Cygwin folder (home/user/) and run as follow:

Sys.argv
I assume that most of you are familiar with sys.argv. For those of you who are not, sys.argv is a list that contains the exterior arguments for the exe file if there are any. The first cell is always the name of the file.
Finally Shell file
The objective was: creating a shell. Here is the one for our toy example:

We run it as follow:

I hope that this post will be beneficial for some readers (assuming the existence of readers)
Acknowledgment
I wish to acknowledge Yuval Shachaf for spending non-neglected time both in doing the project and with some proof reading (The mistakes that remained are since I didn’t listen)
Sources
https://yourpcfriend.com/wp-content/uploads/2019/05/Launching-and-using-Cygwin.png (image)
https://upload.wikimedia.org/wikipedia/commons/thumb/2/29/Cygwin_logo.svg/1200px-Cygwin_logo.svg.png