Machine Learning Algorithms you should learn to become an expert Data Scientist

In this post, we will discuss about the various Machine Learning Algorithm. The main objective of this blog is to give you a basic level of understanding on what are the different types of Machine Learning Algorithm present.

There are many algorithms and it might seem a bit overwhelming to see a bunch of them, but trust me, this will be a good one and fun to learn as well!!

Machine Learning Algorithm are usually used in varied number of applications such as fraud detection, computer vision, autonomous vehicles, where it is not computationally feasible to develop conventional algorithms to fulfil the required

To better understand what these different Machine Learning Algorithm are, let’s just take a step back and see the broader umbrellas which is otherwise called the learning style [1] under which these algorithms are categorized.

The Learning Style is important as it defines how a Machine Learning Algorithm can model a problem.

It is always important to first understand the Learning style that an algorithm can have. This will help us to model the data in a more efficient manner and select the most appropriate algorithm which will provide us the best results.

Machine Learning Algorithm based on Learning Styles

Now let’s look on the different Learning Styles generally used in Machine Learning algorithms. We will go into greater detail on each of the Learning styles in our upcoming post.

Supervised Learning

Supervised Learning : Image by Author

As the name suggests, the model will be supervised to train in order to produce results. What that means is, the input data that we will have will be the training data and will have a output label.

The model will be trained on this training data and is required to make predictions. Based on the accuracy of the predictions, the model is further trained until it achieves a desired level of accuracy on the training data.

Some of supervised learning problems include classification and Regression problems.

Example: Linear Regression, Logistic Regression.

Unsupervised Learning

Unsupervised Learning: Image by Author

In case of unsupervised learning, the input data does not have any kind of label. Based on the structures and patterns in the input data, the model is created.

The created model can be prepared either through identifying patterns within the input data or through extracting general rules or through some mathematical derivations or simply organizing data based on similarity.

Some of the examples for Unsupervised learning is clustering, dimensionality reduction and association rule mining.

Example of Unsupervised algorithm is K-Means clustering, the Apriori algorithm.

Semi-supervised Learning

Semi-supervised Learning: Image by Author

Semi-supervised learning involves few components.

1. Model (Classification or Regression model) — The above figure shows a classification model. The model is first trained with labelled training data.
2. After the classifier is trained on the labelled data, the model is provided with several unlabeled data to classify them. Once the model classifies the unlabeled data, it is further combined with the originally available labelled data to further retrain the classifier model to improve the accuracy of the model.

Example of Semi-supervised algorithm includes the classification and regression problems.

Machine Learning Algorithm based on Functions

The above categorization was the grouping of Algorithm based on the Learning Styles. Now let’s drill down a bit more and group them based on their specific functions.

Grouping by function involves consolidating the different Machine Learning Algorithms based on how they work.

Example: There are many types of Regression Algorithms. Here we will group them under one umbrella of Regression Algorithms.

Regression Machine Learning Algorithm

Regression: Image by Author

Regression is the process of estimating the relationship between a dependent variable(label) and one or more independent variable (features). The most used regression algorithm is Linear Regression.

Some of the most commonly used Regression Algorithms are:

1. Linear Regression
2. Logistic Regression
3. Ordinary Least Squares Regression (OLS)
4. Stepwise Regression
5. Multivariate Adaptive Regression Splines (MARS)
6. Locally Estimated Scatter Plot Smoothing (LOESS)
7. Polynomial Regression

Instance based Machine Learning Algorithm

Instance based Machine Learning Algorithm: Image by Author

The ability of a ML model to perform well on new unseen instances of data is known as generalization.

Usually the main aim of a Machine Learning model is to make accurate predictions. Having a good performance/accuracy on trained data is good, but the true goal is to perform well on new unseen instances.

There are 2 main approaches to Generalization: Instance based Learning, Model based Learning.

Instance Based learning also known as memory-based learning is a type of machine learning approach where instead of performing generalization, the algorithm compares new instances of data with the instances seen/learnt during training which are stored on the memory.

Some of the commonly used Instance based Learning Algorithms are [2]:

1. k-Nearest Neighbor(kNN)
2. Decision Tree
3. Support Vector Machine (SVM)
4. Self-Organizing Map (SOM)
5. Locally Weighted Learning (LWL)
6. Learning Vector Quantization (LVQ)

Regularization based Machine Learning Algorithm

Regularization is a form of regression that penalizes the coefficient estimates towards zero. This technique mostly prevents the model learning more complex and flexible model thereby avoiding the overfitting situation.

Some commonly used regularization algorithms are:

1. Ridge Regression
2. Lasso(Least Absolute Shrinkage and selection operator) Regression
3. ElasticNet Regression
4. Least-Angle Regression (LARS)

Decision Tree based Machine Learning Algorithm

Decision tree based Machine Learning Algorithm: Image by Author

In case of Decision tree, a model of decision is constructed based on the attribute values of the data. The decisions keep branching out until a prediction decision is made for a given record.

Decision trees are usually pretty fast and have good accuracy and are often trained for classification and Regression problems.

Some of the commonly used decision tree algorithms are:

1. C4.5 and C5.0
2. M5
3. Classification and Regression Tree
4. Decision Stump
5. Chi-squared Automatic Interaction Detection (CHAID)
6. Conditional Decision Tree
7. Iterative Dichotomiser 3 (ID3)

Clustering based Machine Learning Algorithms

Clustering : Image by Author

Clustering is a machine learning technique through which we could group data points. Given a dataset containing a variety of data points, we can use a clustering algorithm to classify each data point into a specific group. Based on the theoretical understanding, data points which belong to the same group have similar properties. Clustering falls under the umbrella of unsupervised learning algorithms.

Some commonly used Clustering Algorithms are:

1. K-means clustering
2. Mean-shift clustering
3. Agglomerative Hierarchical clustering
4. K-medians
5. Density -Based Spatial clustering of Applications with Noisec(DBSCAN)
6. Expectation- Maximization (EM) clustering using Gaussian Mixture models (GMM)

Association Rule based Machine Learning Algorithms

Association Rule : Image by Author

It is a rule-based machine learning method which is useful in discovering relation between different features in a large dataset.

As shown in the above figure, given the sale of items, every time a customer buys bread, he also buys milk. Same happens with Soda, where he buys chips along with it.

It basically finds patterns in data which might include:

1. Co-occurring features
2. Correlated features

Some commonly used Association rule learning algorithms are

1. Eclat algorithm
2. Apriori algorithm

Bayesian Algorithms

Bayesian Algorithms : Image by Author

Bayesian Algorithms are the one which follow the principle of Bayes theorem for solving problems of Classification and Regression.

The Bayes theorem describes the probability of an even based on prior understanding of events and occurrences that might be related to the event.

Let’s consider that the chances of developing some kind of health conditions increases with age.

Based on this prior knowledge of health condition relation with age, Bayes theorem accesses an individual more accurately by conditioning it on their age.

Some of the commonly used Bayesian Algorithms are:

1. Naïve Bayes
2. Gaussian Naïve Bayes
3. Bayesian Network
4. Bayesian Belief Network
5. Multinomial Naïve Bayes
6. Average One-Dependence Estimators (AODE)

Artificial Neural Network

Artificial Neural Network : Image by Author

Artificial Neural Network is a branch or AI that tries to replicate the functioning of the human brain.

An Artificial Neural Network is a collection of connected units or nodes which are considered as artificial neurons. This structure if loosely modeled depicts the connected neurons in a biological brain.

The most important part of ANN are the neurons. As the human brain has the cells, similarly an ANN consists of a collection of Neurons that are interconnected.

Components of an ANN

1. As discussed above, an ANN has 3 categories of Neurons — Input Neuron, Hidden Neuron, Output neuron.
2. The other component of ANN is the activation function which is used to generate the output from the hidden neuron to the output neuron. This output generated through the activation function can be passed on to the subsequent neuron which will later become the input to those neurons.

Neurons: Image by Author

Some commonly used Artificial Neural Network Algorithms are:

1. Feed-Forward Neural Network
2. Radial Basis Function Network (RBFN)
3. Kohonen self-organizing neural network
4. Perceptron
5. Multi-Layer Perceptron
6. Back-Propagation
7. Stochastic Gradient Descent
8. Modular Neural Networks (MNN)
9. Hopfield Network

Deep Learning Algorithms

Deep Learning Algorithm: Image by Author

Deep Learning is a branch of Artificial Neural Network which ate computationally more powerful and are capable of solving real complex problems which involves building larger and complex neural networks.

Some of the commonly used Deep Learning Algorithms are:

1. Convolutional Neural Networks (CNN)
2. Recurrent Neural Networks (RNN)
3. Long Short-Term Memory Network (LSTM)
4. Generative Adversarial Networks (GANs)
5. Deep Belief Networks (DBNs)
6. Autoencoders
7. Restricted Boltzmann Machines (RBMs)

Dimensionality Reduction Algorithms

Dimensionality Reduction Algorithms : Image by Author

Dimensionality reduction is a technique of transforming the data from a high-dimensional space to a low dimension space in such a way that the low dimensional data still retains the meaningful properties of the original data.

Dimensionality Reduction algorithm basically exploit the structure of the data in an unsupervised manner or order to summarize or describe the data using lesser data or features.

Some of the commonly used Dimensionality Reduction Algorithms are:

1. Principal component analysis (PCA)
2. Non-negative matrix factorization (NMF)
3. Kernel PCA
4. Linear Discriminant Analysis (LDA)
5. Generalized Discriminant Analysis (GDA)
6. Autoencoder
7. t-SNE (T-distributed Stochastic Neighbor Embedding)
8. UMAP (Uniform Manifold Approximation and Projection)
9. Principal component Regression (PCR)
10. Partial Least Squares Regression (PLSR)
11. Sammon Mapping
12. Multidimensional Scaling (MDS)
13. Projection Pursuit
14. Mixture Discriminant Analysis (MDA)
15. Quadratic Discriminant Analysis (QDA)
16. Flexible Discriminant Analysis (FDA)

Ensemble Algorithms

Ensemble Algorithms : Image by Author

Ensemble technique are those which involve use of multiple models to obtain better accuracy on predictive performance which could not be obtained by using either of the algorithms alone.

Some of the commonly used algorithms are:

1. Boosting
2. Bootstrapped Aggregation (Bagging)
3. AdaBoost
4. Weighted Average (Blending)
5. Stacked Generalization (Stacking)
6. Gradient Boosting Machines (GBM)
7. Gradient Boosted Regression Trees (GBRT)
8. Random Forest

Reference :

1. https://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/
2. https://www.geeksforgeeks.org/instance-based-learning/
3. https://en.wikipedia.org/wiki/Regularization_(mathematics)
4. https://www.digitaltrends.com/cool-tech/what-is-an-artificial-neural-network/
5. https://www.researchgate.net/figure/Principle-of-semi-supervised-learning-1-a-model-eg-CSP-LDA-classifier-is-first_fig4_277605013
6. https://www.kdnuggets.com/2019/01/ensemble-learning-5-main-approaches.html
7. https://en.wikipedia.org/wiki/Dimensionality_reduction
8. https://towardsdatascience.com/an-illustrated-guide-to-artificial-neural-networks-f149a549ba74
9. https://en.wikipedia.org/wiki/Bayes%27_theorem
10. https://towardsdatascience.com/association-rule-mining-be4122fc1793
11. https://medium.com/@sanidhyaagrawal08/what-is-instance-based-and-model-based-learning-s1e10-8e68364ae084
12. https://www.analyticsvidhya.com/blog/2015/08/comprehensive-guide-regression/

Originally published at https://ai-ml-analytics.com on December 22, 2020.