Maximum likelihood estimation

Maximum likelihood estimation (MLE) is one of the methods to find the coefficients of a model that minimizes the RSS in linear regression. MLE does this by maximizing the likelihood of observing the training data given a model.

Background

Consider objective function

$y=w_{0}x_{0}+w_{1}x_{1}+\ldots +w_{m}x_{m}+\epsilon =g(x)+\epsilon$

where $y=g(x)$ is the true relationship and $\epsilon$ is the residual error/noise

We assume that $x_{0}=1$, and $\epsilon \sim N(0,\sigma ^{2})$

Likelihood function

The likelihood function determines the likelihood of observing the data given the parameters of the model. A high likelihood indicates a good model.

$L(w_{i},\sigma ^{2}|x,y)=\prod {\frac {1}{\sqrt {2\pi \sigma ^{2}}}}exp\left(-{\frac {(y_{i}-g(x_{i}))^{2}}{2\sigma ^{2}}}\right)$

For every data point, the likelihood is computed. The product of all likelihoods are taken.

The weights are then changed to fit it better, and the process repeats.