Introduction
In one of my previous articles, I discussed the difference between prediction and inference in the context of Statistical Learning. Despite their main difference with respect to the end goal, in both approaches we need to estimate an unknown function f.
In other words, we need to learn a function that maps the input (i.e. the set of independent variables X) to the output (i.e. the target variable Y) as shown below.
Y = f(X) + ε
In order to estimate the unknown function, we need to fit a model over the data (the training data to be more precise).The form of the function we are trying to estimate, is usually unknown and thus we may have to apply different models in order to get there or make some assumptions with respect to the form of function f. In general, this process can be parametric or non-parametric.
In today’s article, we will discuss about both parametric and non-parametric methods in the context of Machine Learning. Additionally, we will explore their main differences as well as their main advantages and disadvantages.
Parametric Methods
In parametric methods, we typically make an assumption with regards to the form of the function f. For example, you could make an assumption that the unknown function f is linear. In other words, we assume that the function is of the form
f(X) = β₀ + β₁ X₁ + … + βₚ Xₚ
where f(X) is the unknown function to be estimated, β are the coefficients to be learned, p is the number of independent variables and X are the corresponding inputs.
Now that we have made an assumption about the form of the function to be estimated and selected a model that aligns with this assumption, we need a learning process that will eventually help us to train the model and estimated the coefficients.
To summarise, parametric methods in Machine Learning usually take a model-based approach where we make an assumption with respect to form of the function to be estimated and then we select a suitable model based on this assumption in order to estimate the set of parameters.
The biggest disadvantage of parametric methods is that the assumptions we make may not always be true. For instance, you may assume that the form of the function is linear, whilst it is not. Therefore, these methods involve less flexible algorithms and are usually used for less complex problems.
However, parametric methods tend to be quite fast and they also require significantly less data compared to non-parametric methods (more on this in the following section). Additionally, since parametric methods tend to be less flexible and suitable for less complex problems, they are more interpretable.
Some examples of parametric methods in Machine Learning include Linear Discriminant Analysis, Naive Bayes and Perceptron.
Non-Parametric Methods
On the other hand, non-parametric methods refer to a set of algorithms that do not make any underlying assumptions with respect to the form of the function to be estimated. And since no assumption is being made, such methods are capable of estimating the unknown function f that could be of any form.
Non-parametric methods tend to be more accurate as they seek to best fit the data points. However, this comes at the expense of requiring a very large number of observations that is needed in order to estimate the unknown function f accurately. Additionally, these methods tend to be less efficient when it comes to training the models. Furthermore, non-parametric methods may sometimes introduce overfitting. Since these algorithms tend to be more flexible, they may sometimes learn the errors and noise in a way that they cannot generalise well to new, unseen data points.
On the flip side, non-parametric methods are quite flexible and can lead to better model performance since no assumptions are being made about the underlying function.
Some examples of non-parametric methods in Machine Learning include Support Vector Machines and K-Nearest Neighbours.
Final Thoughts
In today’s article we discussed about parametric and non-parametric methods in the context of Machine Learning. Additionally, we explored their advantages and disadvantages.
Parametric methods refer to a set of algorithms that tend to be less flexible and accurate but more interpretable whilst non-parametric methods tend to be more flexible (and thus suitable for more complex problems) and accurate but less interpretable.
Even though parametric methods are less flexible and sometimes less accurate they are still useful in numerous use cases since the use of a quite flexible non-parametric method in a simpler problem may result in overfitting.
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read.
You may also like
