Hi Ygor, excellent question! This is something I glossed over when researching this topic.

The most popular loss function for GAMs is Restricted Maximum Likelihood Estimation (REML). It seems in R's mgcv package, this is optimized using "Penalized Iteratively Re-weighted Least Squares" . I'm not sure what PyGAM does but I'd imagine its similar.

You can read the mgcv documentation here: https://www.rdocumentation.org/packages/mgcv/versions/1.8-35/topics/gam