It is known that the least squares estimator
in the classical linear model coincides with the maximum-likelihood
estimator under the imposed normal distribution.
By using appropriate distributional assumptions for
in
GLM, one may stay in the framework of maximum-likelihood in this case.
For maximum-likelihood estimation, one assumes that the
distribution of belongs to
an exponential family. Exponential families cover a broad
range of distributions, for example discrete
as the Binomial and Poisson distribution or continuous as
the Gaussian (normal) and Gamma distribution.
A distribution is said to belong to an exponential family
if its probability function (if discrete) or its density
function (if
continuous) has the structure
Generally speaking, we are interested in estimating
, the canonical parameter.
is a nuisance parameter
(as the variance
in linear regression for example).
Apart from the distribution of
, the link function is another
essential part of the generalized linear model. Recall the notations
All models in the glm library are estimated by maximum-likelihood. The default numerical algorithm is the Newton-Raphson iteration (except for ordinary regression where no iteration is necessary). Optionally, a Fisher Scoring can be chosen, which uses the expectation of the Hessian matrix instead of the Hessian itself. In the case of a canonical link function, the Newton-Raphson algorithm and the Fisher scoring algorithm coincide.