Maximum likelihood estimation - Revision history

Rice: /* Assumptions */

2024-05-08T03:00:23Z

Assumptions

← Older revision		Revision as of 03:00, 8 May 2024
Line 21:		Line 21:
	== Assumptions ==		== Assumptions ==

	~~We assume that~~ the error is ~~normally distributed.~~		MLE has the following assumptions
			* $x_0 = 1$
			* $X$ is independent variable, and $y$ is dependent variable
			* Residual error $\epsilon ~ N(0,\sigma^2)$ for a constant $\sigma^2$
			* $y$ is independent across observations

	== Optimizations with log ==		== Optimizations with log ==

Rice: /* Likelihood function */

2024-05-08T02:54:47Z

Likelihood function

← Older revision		Revision as of 02:54, 8 May 2024
Line 18:		Line 18:

	The weights are then changed to fit it better, and the process repeats.		The weights are then changed to fit it better, and the process repeats.

			== Assumptions ==

			We assume that the error is normally distributed.

	== Optimizations with log ==		== Optimizations with log ==

Rice: /* Loss function */

2024-04-26T00:51:16Z

Loss function

← Older revision		Revision as of 00:51, 26 April 2024
Line 25:		Line 25:

	== Loss function ==		== Loss function ==
	~~In line with gradient descent~~, we can ~~come up with the loss function of MLE by taking~~ the negative of the likelihood function.		Instead of maximizing the likelihood function, we can minimize the negative of the likelihood function. This way, we can just use OLS.
	[[Category:Machine Learning]]		[[Category:Machine Learning]]

Rice: /* Likelihood function */

2024-04-26T00:47:07Z

Likelihood function

← Older revision		Revision as of 00:47, 26 April 2024
Line 19:		Line 19:
	The weights are then changed to fit it better, and the process repeats.		The weights are then changed to fit it better, and the process repeats.

	~~The computation can be simplified~~ to ~~the following~~		== Optimizations with log ==
			Multiplication of many large numbers is computationally expensive. To optimize, the ''log'' of the likelihood function is computed. Since log of multiplied values is the sum of log of each value, we simplify the above down to

	<math		<math>\log(L(\theta\|x,y) ) = \sum \log \left( \frac{1}{ \sqrt{ 2 \pi \sigma^2}} exp \left( - \frac{(y_i - g(x_i))^2}{2 \sigma^2 } \right) \right)</math>

			== Loss function ==
			In line with gradient descent, we can come up with the loss function of MLE by taking the negative of the likelihood function.
	[[Category:Machine Learning]]		[[Category:Machine Learning]]

Rice: /* Background */

2024-04-25T04:16:41Z

Background

← Older revision		Revision as of 04:16, 25 April 2024
Line 8:		Line 8:
	where <math>y = g(x)</math> is the true relationship and <math>\epsilon</math> is the residual error/noise		where <math>y = g(x)</math> is the true relationship and <math>\epsilon</math> is the residual error/noise

	We assume that <math>x_0 = 1</math>, and <math>\epsilon \sim N(0, \sigma^2)</math>		We assume that <math>x_0 = 1</math>, y values are independent of each other, and <math>\epsilon \sim N(0, \sigma^2)</math>

	= Likelihood function =		= Likelihood function =

Rice: /* Likelihood function */

2024-04-25T04:09:42Z

Likelihood function

← Older revision		Revision as of 04:09, 25 April 2024
Line 15:		Line 15:
	<math>L(w_i, \sigma^2\|x,y) = \prod \frac{1}{ \sqrt{ 2 \pi \sigma^2}} exp \left( - \frac{(y_i - g(x_i))^2}{2 \sigma^2 } \right)</math>		<math>L(w_i, \sigma^2\|x,y) = \prod \frac{1}{ \sqrt{ 2 \pi \sigma^2}} exp \left( - \frac{(y_i - g(x_i))^2}{2 \sigma^2 } \right)</math>

	~~For every~~ data point, the ~~likelihood is computed. The product~~ of ~~all likelihoods are taken~~.		The likelihood of observing the data is the product of observing each data point, given by the probability density function of standard distribution.

	The weights are then changed to fit it better, and the process repeats.		The weights are then changed to fit it better, and the process repeats.

Rice: /* Likelihood function */

2024-04-17T19:59:13Z

Likelihood function

← Older revision		Revision as of 19:59, 17 April 2024
Line 18:		Line 18:

	The weights are then changed to fit it better, and the process repeats.		The weights are then changed to fit it better, and the process repeats.

			The computation can be simplified to the following

			<math
	[[Category:Machine Learning]]		[[Category:Machine Learning]]

Rice: Created page with "'''Maximum likelihood estimation (MLE)''' is one of the methods to find the coefficients of a model that minimizes the RSS in linear regression. MLE does this by maximizing the likelihood of observing the training data given a model. = Background = Consider objective function $y = w_0 x_0 + w_1 x_1 + \ldots + w_m x_m + \epsilon = g(x) + \epsilon$ where $y = g(x)$ is the true relationship and $\epsilon$ is the res..."

2024-04-17T18:21:05Z

Created page with "'''Maximum likelihood estimation (MLE)''' is one of the methods to find the coefficients of a model that minimizes the RSS in linear regression. MLE does this by maximizing the likelihood of observing the training data given a model. = Background = Consider objective function <math>y = w_0 x_0 + w_1 x_1 + \ldots + w_m x_m + \epsilon = g(x) + \epsilon</math> where <math>y = g(x)</math> is the true relationship and <math>\epsilon</math> is the res..."

New page

'''Maximum likelihood estimation (MLE)''' is one of the methods to find the coefficients of a model that minimizes the RSS in [[Linear Regression|linear regression]]. MLE does this by maximizing the likelihood of observing the training data given a model.

= Background =
Consider objective function

<math>y = w_0 x_0 + w_1 x_1 + \ldots + w_m x_m + \epsilon = g(x) + \epsilon</math>

where <math>y = g(x)</math> is the true relationship and <math>\epsilon</math> is the residual error/noise

We assume that <math>x_0 = 1</math>, and <math>\epsilon \sim N(0, \sigma^2)</math>

= Likelihood function =
The '''likelihood function''' determines the likelihood of observing the data given the parameters of the model. A ''high'' likelihood indicates a good model.

<math>L(w_i, \sigma^2|x,y) = \prod \frac{1}{ \sqrt{ 2 \pi \sigma^2}} exp \left( - \frac{(y_i - g(x_i))^2}{2 \sigma^2 } \right)</math>

For every data point, the likelihood is computed. The product of all likelihoods are taken.

The weights are then changed to fit it better, and the process repeats.
[[Category:Machine Learning]]