Linear regression
Linear regression is one of the simplest used techniques for predictive modeling. It estimates a linear relationship between dependent continuous variable $y$ and attributes (aka. independent variables) $X$.
There are different types
- Simple linear regression: one attribute
- Multiple linear regression: multiple attributes
Let the following function model the true relationship between $y$ and $X$
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \begin{aligned} y &= \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \ldots + \epsilon \\ y &= w_0 + w_1 x_1 + w_2 x_2 + \ldots + \epsilon \\ y &= \sum w_j x_j + \epsilon \\ y &= \hat{y} \\ \end{aligned}}
where Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \beta_i} is the weight coefficient of the attribute $x_i$ to be learned, and $\epsilon$ is residual error.
To train a linear regression model is to learn weight coefficients that minimize error. Error is numerically assigned a value with cost functions, usually RSS.
Minimizing RSS
There are several ways to minimize the RSS
