Back propagation is a learning technique that adjusts weights in the neural…

Concept

A multi-layer feed-forward network is trained by gradient descent, which needs the partial derivative of the error with respect to every single weight. Backpropagation produces those derivatives with the chain rule, one layer at a time: the error measured where the network’s prediction emerges is first converted into a local error term for that layer, and the error term of any earlier layer is then written using the terms already obtained for the layer closer to the prediction.

An earlier layer’s error term therefore cannot be formed until the terms of the layer after it exist. The information that drives the weight changes must consequently travel in the direction opposite to the one along which the activations were computed. In the graph view of the network used by this question’s wording, the input nodes are the sources and the output node is the sink.

Application

1. Forward pass: each input node accepts its value, every hidden node combines the values of the layer before it through its incoming weights and applies its activation function, and the output node finally emits the prediction. Activations thus move input side → hidden layers → output side.

2. Error measurement: the prediction is compared with the target value, which yields the error at the output side of the network.

3. Output-layer error term: δk = (tk − ok) f′(netk) is computed for output node k. It is the only error term that can be formed from directly measured quantities, and it exists at the output side alone.

4. Hidden-layer error term: for a hidden node j, δj = f′(netj) Σk δk wkj. The summation runs over the nodes of the layer nearer the prediction, so this term consumes the terms found in the previous step.

5. Weight update: Δwji = η δj xi is applied to every weight, the weights leaving the input nodes included, which is possible only once the error terms have been carried right back to the layer that touches those input nodes.

Steps 3 to 5 visit the layers in the order output side → hidden layers → input side, which is exactly the reverse of the order followed in step 1.

Cross-check

Two different flows travel through the same network and must not be conflated:

• Activations flow from the input side to the output side. That is the forward pass, and it carries values, not the weight changes the stem speaks of.

• Error terms, and the weight changes computed from them, flow from the output side back to the input side. That is the backward pass.

• The backward pass cannot halt at the hidden layer: were it to stop there, the weights joining the input nodes to the first hidden layer would never receive a gradient and would remain at their initial random values, so the network could not learn.

Backpropagation therefore adjusts the weights by propagating the weight changes backward from sink to source.