Back propagation

Back propagation is a error calculation technique. It consists of passing a loss function backwards through a neural network layer-by-layer to update its weights.

Procedure

Consider the following RSS loss function (halved for easier derivative).

$E={\frac {1}{2}}\sum (y_{i}-{\bf {{w}{\bf {{x})^{2}}}}}$

After each feed-forward pass where one data point is passed through the neural network, the gradient of the loss function is computed.

We compute gradient separately at each layer due to difference in activation function by computing the loss function for each neuron of that layer and add up the result.

$E=\sum \left[{\frac {1}{2}}\sum (y_{i}-{\bf {{w}{\bf {{x})^{2}}}}}\right]$

Then, we compute the gradient of the loss function on that layer w.r.t. the weights that created that layer.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \frac{\partial E}{\partial w} }