Bias Term Inclusion in L2 Loss Regularization - Chapter 14 Page 469

[liereynaldo]

This loss function with L2 penalty includes bias term (added together with weight term), which is not the standard convention because usually only weight is included in regularization term

[rasbt]

Yes, you are correct. The traditional convention is to only apply it to the weights (e.g., like I explained on pg 74)

I suppose here it was done to mimic the default in PyTorch, which achieves something similar via the weight_decay setting in the optimizer. There, by default, it's applied to all parameters. However, one could manually specify it like so to only target the weights:

optimizer = torch.optim.Adam([
    {'params': model.linear.weight, 'weight_decay': 1e-4},
    {'params': model.linear.bias, 'weight_decay': 0.0}
], lr=1e-3)