Why I Dislike XGBoost and Why I Like XGBoost

First, the title of this blog post is moderately click-bait. I dislike many charateristics of XGBoost but I like some of them too.

XGBoost (“extreme gradient boost”) is a huge library of many functions, with hundreds of parameters and possible argument values. XGBoost can create machine learning models for multi-class classification, binary classification, and regression. XGBoost works by creating many decision trees; each tree learns to improve on the previous tree using an error gradient. The results are aggregated to final predictions.

I rarely use XGBoost but a lot of my colleagues who are relatively new to machine learning use XGBoost so I take XGBoost out for a spin every now and then.

The pros of XGBoost: 1.) It usually works very well for relatively simple tabular data style problems (multi-class and binary classification and regression). 2.) You don’t need to know much theory to get a prediction model up and running.

The cons: 1.) Because XGBoost is essentially a black box, you have to learn an enormous number of parameters to use it to its potential. 2.) Because XGBoost is a black box, you don’t gain new skills that transfer to other scenarios. For example, a knowledge of deep neural networks ala Keras or PyTorch allows you to create all kinds of interesting and complex systems. (But the tradeoff is you have to learn a lot of ML theory.)

I installed the xgboost library (v1.4.1) using pip with the command “pip install xgboost”. Then I coded up a demo that classifies the Iris dataset. I cheated a bit (to save time) and used scikit to load the 150-item, 3-class Iris data and split it into 135 training items (90%) and 15 test items (10%). In a non-demo scenario, data preparation takes a long time.

The Iris dataset is not challenging so it’s not a surprise that the trained model predicted all 15 test items correctly.

OK, that’s my dose of XGBoost for a while. Not great fun, but good-enough fun.

Artist Linda Wooten created this beautiful dollhouse-sized model of a pirate tavern. It reminds me of my days working on the Pirates of the Caribbean ride at Disneyland when I was a student in college at UC Irvine. Creating a detailed model with custom parts is like creating a machine learning model using a low-level library like PyTorch. Creating a pirate tavern model using Lego would be like creating a machine learning model using a high-level tool like XGBoost — quicker but not as beautiful.

# xgb_iris.py

# xgboost 1.4.1
# Python 1.8.0-CPU  Windows 10

import numpy as np
import xgboost as xgb
from sklearn import datasets
from sklearn.model_selection import train_test_split

def main():
  print("\nBegin Iris classification using XGBoost ")
  np.set_printoptions(precision=4, suppress=True)

  iris = datasets.load_iris()
  X = iris.data
  y = iris.target
  X_train, X_test, y_train, y_test = \
    train_test_split(X, y, test_size=0.10, random_state=1)

  d_train = xgb.DMatrix(X_train, label=y_train)
  d_test = xgb.DMatrix(X_test, label=y_test)

  params = {
    'max_depth': 3,  # tree depth 
    'eta': 0.3,      # training step
    'objective': 'multi:softprob',
    'eval_metric' : 'merror',
    'num_class': 3

  num_rounds = 5
  model = xgb.train(params, d_train, num_boost_round=num_rounds)

  preds = model.predict(d_test)
  print("\nprediction p-values: ")

  print("\npredicted classes: ")
  pc = np.argmax(preds, axis=1)

  print("\nactual classes: ")

  print("\nEnd demo ")

if __name__ == "__main__":
This entry was posted in Machine Learning. Bookmark the permalink.

2 Responses to Why I Dislike XGBoost and Why I Like XGBoost

  1. J. Michael Menke says:

    I take it this is python?

  2. Yes. You install the library with the command pip install xgboost.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s