XGBoost offers various performance measure approaches, known as loss functions, to optimize model training for regression and classification tasks.
Regression Loss Functions
Squared Loss (L2 Loss): Primarily used for mean target estimation in regression tasks. It measures the difference between predicted and actual values, summing up the squared differences across all instances. L2 regularization adds a penalty term based on the sum of squared model weights.
equation
Absolute Loss (L1 Loss): This loss function uses the absolute difference between the predicted and actual values instead of squares.
equation
Huber Loss: A combination of L1 and L2 Loss that is less sensitive to outliers. It switches to L1 Loss if the absolute difference is above a
threshold.
def huber_loss(y_true, y_predicted, delta):
error = y_true - y_predicted
return np.where(np.abs(error) < delta, 0.5 * error**2, delta * (np.abs(error) - 0.5 * delta))
Classification Loss Functions
Logistic Loss: Commonly employed in binary classification problems. It calculates the likelihood of the predicted class, converting it to a probability with a sigmoid function.
Softmax Loss: Generally used for multi-class classification tasks. It calculates a probability distribution for each class and maximizes the likelihood across all classes.
Adaptive Log Loss (ALogLoss): Introduced in XGBoost, this loss function provides a balance between speed and accuracy. It's derived by approximating the Poisson likelihood.
