Hi there! This is the 4th milestone of our XGBoost journey. Up to now, we have basically discussed performing regression and Classification tasks with XGBoost. Today, more emphasis is given to XGBoost model evaluation through cross-validation. Did you remember that in the "Classification with XGBoost" (Milestone 2) article, if you set a different integer value for the random_state parameter in the train_test_split() function, you will get slightly different accuracy scores? Today, we will address this issue through cross-validation.
Basically, we will perform cross-validation with both Scikit-learn compatible and non-Scikit-learn compatible APIs. But, one last thing! Before proceeding, just make sure to read the following articles which are prerequisites for today’s content. The concepts discussed in those articles are very important and I highly recommend you reading them.
Prerequisites
- A Journey through XGBoost: Milestone 1 (Setting up the background)
- A Journey through XGBoost: Milestone 2 (Classification with XGBoost)
- k-fold cross-validation explained in plain English
Let’s get started!
Evaluating XGBoost model with Scikit-learn compatible API
Here, we use the same "heart disease" dataset (download here) and the classification model built in the "A Journey through XGBoost: Milestone 2 (Classification with XGBoost)" article.
"Scikit-learn compatible" means that here we can use the Scikit-learn cross_val_score() function to perform cross-validation with XGBoost.
Let’s write the Python code.
The output is:

The average accuracy of 0.818 is more robust and it is a good performance estimate for our model. Note that here we performed 10-fold cross-validation and got 10 different accuracy values for our model.
Here, we just input X and y. We don’t need to split the dataset as X_train, y_train, X_test, y_test. In cross-validation, the splitting is done internally based on the number of folds we specified in cv (here 10). Using cross-validation guarantees that the accuracy score of our Xgboost model isn’t much affected by the random data splitting process. If we just use the train_test_split() function _withou_t cross-validation, the accuracy score will vary significantly based on the random_state we provide inside the train_test_split() function. Here in cross-validation, the accuracy is calculated using the average of 10 (cv=10) such iterations!
In k-fold cross-validation, we make an assumption that all observations in the dataset are nicely distributed in a way that the data are not biased. That is why we first shuffle the dataset using the shuffle function.
Next, we perform the same cross-validation procedure with the XGBoost non-Scikit-learn compatible API.
Evaluating XGBoost model with XGBoost non-Scikit-learn compatible API
Another way to perform cross-validation with XGBoost is to use XGBoost’s own non-Scikit-learn compatible API. "Non-Scikit-learn compatible" means that here we do not use the Scikit-learn cross_val_score() function, instead we use XGBoost’s cv() function with explicitly created DMatrices. A DMatrix is an internal data structure used by XGBoost.
Let’s write the Python code.
The output is:

Here, you will get a nicely formatted output with a lot of details. The XGBoost’s cv() function returns train and test errors for each fold. You can get the accuracy using 1-error. Here the average accuracy score is much closer to the previous one. There are small differences because randomness involves when shuffling the data.
Summary
Today, we have discussed the cross-validation procedure that can be applied to evaluate our XGBoost in a much better way. We have performed cross-validation with two different APIs of XGBoost. We have obtained the same accuracy score in both APIs, but much more details with the XGBoost’s own non-Scikit-learn compatible API.
In the next article, we will discuss how to tune XGBoost’s hyperparameters. Cross-validation involves there also. So, if you’re still not much familiar with cross-validation, read my "k-fold cross-validation explained in plain English" article. See you next time!
Thanks for reading!
This tutorial was designed and created by Rukshan Pramoditha, the Author of Data Science 365 Blog.
Read my other articles at https://rukshanpramoditha.medium.com
2021–03–19