Linear regression

Linear regression is one of the simplest used techniques for predictive modeling. It estimates a linear relationship between dependent continuous variable $y$ and attributes (aka. independent variables) $X$.

$y=f(X)$

There are different types

• Simple linear regression: one attribute
• Multiple linear regression: multiple attributes

Let the following function model the true relationship between $y$ and $X$

${\begin{aligned}y&=\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\ldots +\epsilon \\y&=w_{0}+w_{1}x_{1}+w_{2}x_{2}+\ldots +\epsilon \\y&=\sum w_{j}x_{j}+\epsilon \\y&={\hat {y}}\\\end{aligned}}$

where $\beta _{i}$ is the weight coefficient of the attribute $x_i$ to be learned, and $\epsilon$ is residual error.

To train a linear regression model is to learn weight coefficients that minimize error. Error is numerically assigned a value with cost functions, usually RSS.

Minimizing RSS

There are several ways to minimize the RSS

1. Ordinary least squares
2. Maximum likelihood estimation
3. Gradient Descent
4. Newton's Method