XGBoost Algorithm

XGBoost, short for extreme gradient boosting, is an algorithm and accompanying open source software library for building an ensemble of decision trees, one tree at a time, where each new tree is trained to correct the errors left behind by the trees already built. It extends ordinary gradient boosting by applying a second order Taylor approximation to the loss function, similar in spirit to Newton's method, and by adding explicit penalties that discourage overly complex trees, along with engineering techniques such as weighted quantile sketching that let it train efficiently on very large datasets. Tianqi Chen began the project at the University of Washington and released it publicly on March 27, 2014, and the library gained wide recognition after helping win the Higgs Boson Machine Learning Challenge that same year. XGBoost went on to become one of the most consistently successful tools in machine learning competitions through the mid-2010s, particularly popular within the Kaggle competitive data science community.