Gradient Boosted Models (GBMs) are ensemble methods that build models in a forward stepwise manner, using decision trees as base learners. The algorithm can be computationally intensive, making it a somewhat challenging learning model. However, through its improved accuracy and speed, along with advanced regularization techniques, it's still a popular option.
Extreme Gradient Boosting (XGBoost) is a variant of this algorithm optimized for both speed and performance, offering several unique features.
Key Concepts and Methodology
Sequential Training: XGBoost uses an approach called boosting, where each new model targets errors from the prior ones, creating an additive sequence of predictors.
Gradient Optimization: The algorithm minimizes a predefined loss function by following the steepest descent in the model's parameter space.
Principle Components: Boosting iteratively fits small, simple models to the residuals of the preceding model, refining these fits over time and, in turn, improving the overall prediction.
Shrinkage (Learning Rate): Learning rate, typically small (e.g., 0.1), scales the contribution of each tree. Lower learning rates yield better accuracy at the cost of slower convergence.
Tree Pruning: The trees can be pruned of their most irrelevant subtrees, which both boosts efficiency and limits overfitting.
Regularization: Both L1 and L2 regularization are used, reducing the risk of overfitting.
Feature Importance: The model calculates the importance of each feature, aiding in the selection process.
Support for Missing Data: The models natively handle missing data.
Cross-Validation: A built-in function for cross-validation helps choose the optimal number of trees.
XGBoost Enhancements
XGBoost has a set of features that make it faster and more effective than traditional GBMs:
Algorithmic Enhancements: Several techniques, like approximate tree learning, enhance efficiency without compromising accuracy.
Hardware Optimization: Multi-threading and selective GPU support improves speed and performance.
Built-in Cross-Validation: Its algorithms include an efficient method for cross-validation, significantly simplifying the validation process.
Integrated Regularization: The model automatically applies L1 and L2 regularization.
Monotonicity Constraints: XGBoost lets you specify whether the model should have a positive or negative relationship with each feature, implementing business logic into the model.
