Artificial neural network

An artificial neural network (ANN) is a machine learning algorithm that uses three layers of neurons to perform classification/regression.

Structure

All ANN's consist of three layers of neurons: the input layer, the hidden layer, and the output layer. The number of neurons in each layer determines the complexity of the model, with a larger layer indicating a more complicated model.

Each neuron takes in information from neurons of the previous layer (and for the case of the input layer, the input). An ANN is considered dense if each neuron takes input from every neuron of the previous layer.

Loss function

Figure 1. Convex vs. non-convex loss function. Non-convex has local minimums that needs to be filtered out.

The loss function in ANN is usually non-convex, making minimization more difficult than setting the gradient to 0.

Neuron

The neuron is the data-processing unit that composes an ANN. It combines information from multiple neurons (or input) and become activated or deactivated. The output is determined by the following steps:

1. A neuron generates a linear combination of input data, $z$
2. $z$ is fed into an activation function
3. The activation function determines whether the neuron is activated (1) or not (0)