Optimization of Neural Networks with Linear Solvers
How to optimize nonlinear neural networks in more than one dimension using linear solvers.
Note: If you don’t have a medium subscription, you can read the article for free here (if you have a subscription, please continue reading here, thanks! 🥰)
Recently, I stumbled over a problem that required me to create a model which takes more than one input feature and predicts a continuous output.
Then, I needed to get the best possible output from that model, which in my case was the lowest possible numerical value. So, in other words, I needed to solve an optimization problem.
The issue was (and I only realized it at that stage) that the environment I was working in did not allow me to use nonlinear things or sophisticated frameworks— so no neural networks, no nonlinear solvers, nothing...
But, the model I created worked well (considering the low number of data points I trained it on), and I did not want to delete all my codes and start from scratch with a linear model.
So, after a cup of coffee ☕, I decided to use this nonlinear model I already trained to generate a number of small linear ones. Then I could use a linear solver to solve the optimization problem.