Neural network

A neural network is a type of machine learning model that utilizes layers of connected neurons to perform classification and regression.

Structure

Neuron

The neuron is the data-processing unit that composes a neural network. It combines information from multiple neurons (or input) and become activated or deactivated. The output is determined by the following steps:

1. A neuron generates a linear combination of input data, z. This is done with a set of weights.
2. z is fed into an activation function
3. The activation function determines whether the neuron is activated (1) or not (0)

Layers

Many neurons form a layer, and many layers form a neural network. Each neuron takes input from neurons of the previous layer, and send outputs to the next layer.

The first layer of an NN is the input layer. In contrast to other layers, each neuron in the input layer takes one attribute as their input. As such, the size of the input layer is the same as the number of attributes used to train the model.

The last layer of an NN is the output layer. Pretty self explanatory.

Any layer in between are called hidden layers.

Loss function

Figure 1. Convex vs. non-convex loss function. Non-convex has local minimums that needs to be filtered out.

The loss function in ANN is usually non-convex, making minimization more difficult than setting the gradient to 0.

Classification

The number of layers and the size of each layer determines the complexity of the model.

• Artificial neural network has 3 layers
• Deep neural network has a lot of layers.