Delta rule

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages) This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Delta rule" – news · newspapers · books · scholar · JSTOR (November 2012) (Learn how and when to remove this message) This article may be confusing or unclear to readers. Please help clarify the article. There might be a discussion about this on the talk page. (September 2012) (Learn how and when to remove this message) (Learn how and when to remove this message)

In machine learning, the delta rule is a gradient descent learning rule for updating the weights of the inputs to artificial neurons in a single-layer neural network.^[1] It can be derived as the backpropagation algorithm for a single-layer neural network with mean-square error loss function.

For a neuron ${\displaystyle j}$ with activation function ${\displaystyle g(x)}$, the delta rule for neuron ${\displaystyle j}$'s ${\displaystyle i}$-th weight ${\displaystyle w_{ji}}$ is given by

$${\displaystyle \Delta w_{ji}=\alpha (t_{j}-y_{j})g'(h_{j})x_{i},}$$

where

• ${\displaystyle \alpha }$ is a small constant called learning rate
• ${\displaystyle g(x)}$ is the neuron's activation function
• ${\displaystyle g'}$ is the derivative of ${\displaystyle g}$
• ${\displaystyle t_{j}}$ is the target output
• ${\displaystyle h_{j}}$ is the weighted sum of the neuron's inputs
• ${\displaystyle y_{j}}$ is the actual output
• ${\displaystyle x_{i}}$ is the ${\displaystyle i}$-th input.

It holds that ${\textstyle h_{j}=\sum _{i}x_{i}w_{ji}}$ and ${\displaystyle y_{j}=g(h_{j})}$.

The delta rule is commonly stated in simplified form for a neuron with a linear activation function as $${\displaystyle \Delta w_{ji}=\alpha \left(t_{j}-y_{j}\right)x_{i}}$$

While the delta rule is similar to the perceptron's update rule, the derivation is different. The perceptron uses the Heaviside step function as the activation function ${\displaystyle g(h)}$, and that means that ${\displaystyle g'(h)}$ does not exist at zero, and is equal to zero elsewhere, which makes the direct application of the delta rule impossible.