The world’s leading publication for data science, AI, and ML professionals.

How Do You Implement AdaBoost with Python?

Boosting algorithms in machine learning – Part 2

Boosting Techniques

Photo by John Price on Unsplash
Photo by John Price on Unsplash

We’re continuing from [Part 1](https://towardsdatascience.com/introduction-to-boosted-trees-2692b6653b53) of Boosting algorithms in machine learning article series. Have you read Part 1? It gives you a short introduction to boosting and definitions of some key technical terms that are all important to understand today’s content.

Today, we’re going to learn one of the most popular boosting algorithms: Adaboost (Adaptive Boosting). More emphasis will be given to the implementation part of the algorithm under the following topics.

  • Scikit-learn classes for AdaBoost
  • Train and evaluate an AdaBoost classification model on Wine data
  • Compare the AdaBoost model with a decision tree stump
  • Important hyperparameters in AdaBoost
  • Measure the effect of hyperparameter _n_estimators_
  • Measure the effect of hyperparameter _learning_rate_
  • Find the optimal hyperparameter values using Grid Search

All code samples will be included as GitHub gists so that you can easily work with them! At the end of the article, you will be able to implement the AdaBoost algorithm on a given dataset with optimal hyperparameter values.

Let’s get started!

Scikit-learn classes for AdaBoost

The Python implementation of AdaBoost is fulfilled by two Scikit-learn classes: AdaBoostClassifier() for classification (both binary and multi-class) and AdaBoostRegressor() for regression. The import conventions are:

from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import AdaBoostRegressor

Create a decision tree stump model

By early stopping the tree growth with max_depth=1, we’ll build a decision stump on Wine data. This is to compare the decision stump with the AdaBoost model. We also use this stump model as the base learner for AdaBoost.

The following Python code creates a decision tree stump on Wine data and evaluates its performance.

(Image by author)
(Image by author)

The model’s performance is poor. This is not a surprise as we strictly regulated the tree growth by setting max_depth=1.

Create an AdaBoost classification model

Now, we’re going to build an AdaBoost classification model on Wine data.

(Image by author)
(Image by author)

Ohh that score! The AdaBoost model is performing really well compared to the decision tree stump.

Important hyperparameters in AdaBoost

The following are the most important hyperparameters in both AdaBoostClassifier() and AdaBoostRegressor().

  • base_estimator: This is the base learner used in AdaBoost Algorithms. The default and most common learner is a decision tree stump (a decision tree with max_depth=1) as we discussed earlier.
  • n_estimators: The maximum number of estimators (models) to train sequentially. The default is 50. We’ll measure the effect of this hyperparameter soon.
  • learning_rate: This determines the weight applied to each estimator in the Boosting process. The default is 1. Smaller values such as 0.05, 0.1 force the algorithm to train slower but with high-performance scores. We’ll measure the effect of this hyperparameter soon.

Measure the effect of _n_estimators_

We’ll manually measure the effect of the hyperparameter n_estimators by varying its values from 2 to 100 and plot the test scores given by the AdaBoost model.

(Image by author)
(Image by author)

After about 30 estimators, the accuracy score is constant at 0.972 that we obtained earlier. You can use any integer above 30. But keep in mind that if you increase the value, the algorithm will take much time for training. So, in this case, it is better to use the default value, 50.

Measure the effect of _learning_rate_

We’ll manually measure the effect of the hyperparameter learning_rate by varying its values from 0.1 to 1 and plot the test scores given by the AdaBoost model.

(Image by author)
(Image by author)

The highest accuracy is given at 0.4, 0.5, 0.7 and 0.8. We’ve used 0.5 in the above model.

Both measurements were done manually. There is an easier way to find the optimal values for n_estimators and learning_rate.

Find the optimal hyperparameter values using Grid Search

Here, we automatically tune (find optimal values) both hyperparameters at the same time by using Grid Search. In contrast, in the previous steps, we tuned one hyperparameter at a time.

(Image by author)
(Image by author)

We can use these values to get the highest accuracy of the AdaBoost model. Some other possible combinations that give the same accuracy are:

  • learning_rate=0.5, n_estimators=50
  • learning_rate=0.8, n_estimators=50
  • learning_rate=0.8, n_estimators=100
  • learning_rate=0.4, n_estimators=50

Summary

Building AdaBoost models in Python is straightforward. All you have to do is to determine the type of problem (regression/classification) you want to solve and select the suitable AdaBoost class provided in Scikit-learn. However, the hyperparameter tuning procedure is a real challenge. You need to make some visualizations, do parallel computations for hyperparameter tuning. Actually, you’ll create thousands of models. Luckily, we can automate the tuning process using advanced methods as we discussed in this post.

The model’s outputs can be largely affected by random data splitting. You may get different accuracy scores depending on the integer specified in the random_state. It is always recommended to specify an integer to get static results at different executions.

Compared to other tree-based methods such as decision trees and random forests, boosting algorithms like AdaBoost perform really well. This is because tress in boosting are sequentially trained considering the errors created by initial tress in previous rounds.

In Part 3, we’ll discuss Gradient Boosting, another popular boosting algorithm. See you in the next story. Happy learning to everyone!


My readers can sign up for a membership through the following link to get full access to every story I write and I will receive a portion of your membership fee.

Join Medium with my referral link – Rukshan Pramoditha

Thank you so much for your continuous support!

Special credit goes to John Price on Unsplash, **** who provides me with a nice cover image for this post.

Rukshan Pramoditha 2021–10–22


Related Articles