XGBoost is a highly optimized implementation of gradient boosted trees designed for both efficiency and better performance. Let's explore the key differences between XGBoost and GBM.
Tree Construction
XGBoost: Trees are built level-wise, which can sometimes sacrifice detailed local adjustments but often speeds up the building process and brings better overall performance.
GBM: Trees are commonly built using a leaf-wise strategy, potentially leading to more refined models for regions with many observations.
Regularization Techniques
XGBoost: It employs both L1 (LASSO) and L2 (Ridge Regression) regularizations, enhancing generalization and combating overfitting.
GBM: While conventional implementations permit only L2 regularization, newer versions may also support L1.
Parallelism
XGBoost: Offers parallelism for tree construction, node splitting, and column selection.
GBM: Some libraries do support multi-threading, but for the most part, tree building is sequential.
Missing Data Handling
XGBoost: It automatically determines the best path for missing values during training.
GBM: Missing data handling usually requires explicit handling and definition at the user's end.
Algorithm Complexity
XGBoost: Optimization techniques like column block structures, sparse-aware data representation, and out-of-core computing make it computationally efficient.
GBM: Although conceptually simpler, its lack of specialized techniques might make it less efficient for certain datasets.
Split Finding Techniques
XGBoost: Employs the exact or approximate greedy algorithm for split discovery.
GBM: Conventionally utilizes the greedy algorithm.
Custom Loss and Evaluation Metrics
XGBoost: Users can define custom objective and evaluation metrics for specific tasks.
GBM: Supports only the predefined objective and evaluation metrics.
Code Example: Training a Simple Model Using XGBoost
Here is the code:
import xgboost as xgb
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
# Load the Boston housing dataset
boston = load_boston()
X, y = boston.data, boston.target
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Convert the dataset to DMatrix
data_dmatrix = xgb.DMatrix(data=X, label=y)
# Instantiate and train the XGBoost model
xg_reg = xgb.XGBRegressor(objective ='reg:squarederror', colsample_bytree = 0.3, learning_rate = 0.1,
max_depth = 5, alpha = 10, n_estimators = 100)
xg_reg.fit(X_train, y_train)
# Make predictions and evaluate the model
preds = xg_reg.predict(X_test)
rmse = mean_squared_error(y_test, preds, squared=False)
print("RMSE: %.2f" % (rmse))
