Naive Bayes

Naive Bayes is an approach to Bayesian networks that simplify the computation of joint probability of an outcome based on high dimensional features.

Motivation

Consider binary classification output C is dependent on binary features X1~X3. By Bayes theorem, we can compute C's probability based on the features with Bayes' theorem:

$P(C|X_{1},X_{2},X_{3})={\frac {P(X_{1},X_{2},X_{3}|C)P(C)}{P(X_{1},X_{2},X_{3})}}$

This, in turn, mean that we need to estimate the probability of every combination of features (0 0 0, 0 0 1...). This is computationally expensive.

How it works

By assuming that the features are independent, Naive Bayes simplifies the computation to

$P(C|X_{1},X_{2},X_{3})\propto P(X_{1}|C)P(X_{2}|C)P(X_{3}|C)$

We can divide P(C|X) over P(notC|X) to avoid calculating P(X1,X2,X3). We can then apply a log to avoid zero denominators.