Rectifier (neural networks)

In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function^[1][2] is an activation function defined as the non-negative part of its argument:

${\displaystyle f(x)=x^{+}=\max(0,x)={\frac {x+|x|}{2}}={\begin{cases}x&{\text{if }}x>0,\\0&{\text{otherwise}},\end{cases}}}$

where ${\displaystyle x}$ is the input to a neuron. This is also known as a ramp function and is analogous to half-wave rectification in electrical engineering.

ReLU is one of the most popular activation function for artificial neural networks,^[3] and finds application in computer vision^[4] and speech recognition^[5][6] using deep neural nets and computational neuroscience.^[7][8][9]

It was first used by Alston Householder in 1941 as a mathematical abstraction of biological neural networks.^[10] It was introduced by Kunihiko Fukushima in 1969 in the context of visual feature extraction in hierarchical neural networks.^[11][12] It was later argued that it has strong biological motivations and mathematical justifications.^[13][14] In 2011,^[4] ReLU activation enabled training deep supervised neural networks without unsupervised pre-training, compared to the widely used activation functions prior to 2011, e.g., the logistic sigmoid (which is inspired by probability theory; see logistic regression) and its more practical^[15] counterpart, the hyperbolic tangent.