Simulate Any Functions with a Neural Network

We know that the Linear Regression model can help generate a line to simulate "Linear Data". A function in the form of f(x) = ax + b.

So, what about other functions, like non-linear ones? Even an irregular shape that we know nothing about. Can we still simulate a model to fit it? like the one in the header picture and even a spiral shape?

Recently, I was inspired by this video: Why Neural Networks can learn almost anything. I realized that with real data in hand, we can…

Simulate any functions with the Neural Network model!

The video by Emergent Garden is really an underrated one, highly recommend you to watch it too, and the links attached in the video are also of very high quality.

In this article, I am going to implement an any-function-simulator using PyTorch. For two purposes:

1. Witness the power of the neural network model. It is really fun to see the fitting process.
2. The trained model can be also used to detect anomaly data. regular data is usually in a stable pattern, say, relative high during the weekday and dropping during the weekend. A trained NN model can be used to detect anomaly data.

Generate training data

The following code will initialize x numbers and use the following function to generate y numbers for simulation.

import numpy as np
import matplotlib.pyplot as plt
X = np.array([*range(-20,20)],dtype=np.float32)
X = X*0.1
y = [x**3+ x**2 -3*x -1 for x in X]
plt.plot(X,y,'ro')

Now we have both input x and output y, next let’s forget the above function because I am going to build a Neural Network to simulate the "unknown" function.

Transform data for PyTorch Model

Before feeding the data to NN Model, I need to transform the list and NumPy array data to PyTorch Tensors.

import torch
import torch.nn as nn
import torch.nn.functional as F

X_t = torch.tensor(X,dtype=torch.float32)
y_t = torch.tensor(y,dtype=torch.float32)
X_t = X_t.view(X_t.shape[0],1)
y_t = y_t.view(y_t.shape[0],1)

Now, both X_t and y_t are 2d tensor arrays.

Define the NN model with one hidden layer

Here I am going to define a simple NN model with only one hidden layer, the hidden layer neural numbers are set to 16 for easier visualization. In the code, I am going to increase the number to 128.

Here is the code:

class func_simulator(nn.Module):
    def __init__(self):
        super(func_simulator,self).__init__()
        self.l1 = nn.Linear(1,128)
        self.l2 = nn.Linear(128,1)

    def forward(self,x):
        out = F.relu(self.l1(x))
        out = self.l2(out)
        return out

The ReLU activation is the key here, don’t change to sigmoid.

Train and verify the result

Now, let’s set learn rate, epoch num, loss, and gradient functions to start training.

learning_rate,num_epochs    = 0.0001,100000
model                       = func_simulator()
loss                        = nn.MSELoss()
gradient = torch.optim.SGD(model.parameters(),lr=learning_rate)

# start training
for epoch in range(num_epochs):
    y_pred = model.forward(X_t)
    l = loss(y_pred,y_t)
    l.backward()
    gradient.step()
    gradient.zero_grad()
    if(epoch+1)%1000==0:
        print(f'epoch:{epoch+1},loss={l.item():.3f}')

Check out the result:

predicted = model(X_t).detach().numpy()
plt.plot(X_t,y_t,'ro')
plt.plot(X_t,predicted,'b')
plt.show()

Red dots are the training points, the blue line is the one from the trained model. The simulation isn’t that good in some regions, especially the start and end parts, I added another hidden layer to see if the 2 layers can bring a better simulation.

Fine-tune the model with 2 hidden layer

class func_simulator(nn.Module):
    def __init__(self):
        super(func_simulator,self).__init__()
        self.l1 = nn.Linear(1,128)
        self.l2 = nn.Linear(128,10)
        self.l3 = nn.Linear(10,1)

    def forward(self,x):
        out = F.relu(self.l1(x))
        out = F.relu(self.l2(out))
        out = self.l3(out)
        return out

I highlighted the code that is different from the one-layer version, the 2 layer model generates a better simulation.

Wrap up

Why can neural network model simulate any functions? This powerful capability is inherited from the nature of the Neural Networks Model. In essence, NN is a multi-layer parameter system. The more train data you feed the model, the more parameters are needed (and also the more layers). each neural in a layer capture a tiny feature, several combined neural determine a major feature, etc.

From the above sample, we can see the two layers NN generated a better fitting line compared with the model with one hidden layer. The more training data you have, the more neurons and layers are needed to train the model. But in the end, if a limitless computer exists, with enough data, we can simulate almost any functions.

---

When I realized that computers and Neural Networks can simulate any functions with enough training data. My understanding of image object detection is elevated and leads to an even more shocking conjecture:

if a super-duper powerful computer exists, can it simulate our real world, even the whole universe?

Maybe we are all living in a simulated world.

References

1. Can neural networks solve any problem?
2. A visual proof that neural nets can compute any function
3. Why Neural Networks can learn almost anything
4. Deep Feedforward Networks

Appendix – code

Here is the complete code used in this article, you can copy and run it on your machine with PyTorch installed, and no GPU is needed.