back!
First of all, thank you so much for the response to the first two parts of this series.
It feels good that many of you have found them helpful.
As always, if you have any thoughts, questions, or suggestions while reading, I’d love to hear your perspective.
Now, let’s pick up where we left off in part 2.
Why Recompute the Same Gradients?
We calculated the gradient for using the chain rule.
[
frac{partial L}{partial w_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial w_1}
]
We got the same equation that we had previously derived using classical differentiation in Part 1.
[
frac{partial L}{partial w_1}
=
-2(y-hat{y})
cdot
w_3
cdot
mathrm{ReLU}'(w_1x+b_1)
cdot
x
]
We then came to understand just how important the chain rule is.
Now, what about the gradients for the other parameters?
[
b_1,; w_2,; b_2,; w_3,; w_4,; b_3
]
We’ve already seen the entire process for . So, we generally think of repeating the same steps which is using the chain rule for the remaining parameters.
If we have a look at all the chain rule equations for all parameters, we can observe that many of the partial derivatives appear more than once.
[
frac{partial L}{partial w_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial w_1}
]
[
frac{partial L}{partial b_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial b_1}
]
[
frac{partial L}{partial w_2}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_2}
cdot
frac{partial a_2}{partial z_2}
cdot
frac{partial z_2}{partial w_2}
]
[
frac{partial L}{partial b_2}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_2}
cdot
frac{partial a_2}{partial z_2}
cdot
frac{partial z_2}{partial b_2}
]
[
frac{partial L}{partial w_3}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial w_3}
]
[
frac{partial L}{partial w_4}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial w_4}
]
[
frac{partial L}{partial b_3}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial b_3}
]
Let’s consider
[b_1]
The chain rule equation for this parameter is
[
frac{partial L}{partial b_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial b_1}
]
Now, let’s compare this with chain rule equation of .
[
frac{partial L}{partial w_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial w_1}
]
We can observe that most of the chain is actually identical.
Both equations contain
[
frac{partial L}{partial hat{y}}
]
They also contain
[
frac{partial hat{y}}{partial a_1}
]
and
[
frac{partial a_1}{partial z_1}
]
The only difference is the final partial derivative.
For , the final term is
[
frac{partial z_1}{partial w_1}
]
whereas for b1, the final term is
[
frac{partial z_1}{partial b_1}
]
Now, if we calculate the gradient for b1, then first three partial derivatives are computed twice, even though their values are exactly the same.
The same pattern appears when we compute the gradients for
[
w_2,; b_2,; w_3,; w_4,; b_3
]
In each case, many of the intermediate partial derivatives appear more than once.
At this point, we generally have a question.
Why are we recomputing the same intermediate partial derivatives which we’ve already computed once?
If we continue this way, then we end up doing many unnecessary calculations.
For our small neural network, this may not look like a big issue, but in real world we have neural networks with millions of parameters.
If we proceed in the same way for huge neural networks, then it results in increasing the time and computational resources required to train the model, making the training process much more expensive.
What can we do here?
The one idea that comes to our mind is, instead of recomputing the same values, why don’t we reuse those values whenever we need them.
This is exactly the idea behind backpropagation.
It doesn’t use a different way for computing gradients but relies entirely on the chain rule.
Now we know the idea behind backpropagation.
Then we might think, when we compute the gradient for ,
[
frac{partial L}{partial w_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial w_1}.
]
we store the intermediate partial derivatives
[
frac{partial L}{partial hat{y}},
qquad
frac{partial L}{partial a_1},
qquad
frac{partial L}{partial z_1}
]
Next, while computing the gradient of
[frac{partial L}{partial b_1}]
we can simply reuse the previously computed values of
[
frac{partial L}{partial hat{y}},
qquad
frac{partial L}{partial a_1},
qquad
frac{partial L}{partial z_1}
]
Similarly, we continue computing the gradients of the remaining parameters, storing intermediate partial derivatives whenever possible and reusing them whenever they are needed.
What do you think about this approach?
From an Idea to an Algorithm
If this is the approach, then why is it called backpropagation?
Here, we are working with a small neural network to understand the idea behind backpropagation.
For our small neural network, this approach can be used or we can say it is manageable.
But what if we have a neural network with thousands or even millions of parameters?
It becomes increasingly difficult to decide which intermediate values should be stored, when they should be reused, and in what order the gradients should be computed.
But why are we even thinking about this?
After all, in practice, who computes these gradients manually?
Don’t we simply write code?
Yes, we do.
If we implement the approach we just discussed for this neural network, it works.
Now let’s say we build a different neural network with a different architecture.
Should we redesign our gradient computation procedure again?
As neural networks become more complex, continually changing our implementation becomes difficult.
What can we observe from this?
We don’t want to redesign the logic for gradient computation procedure every time the architecture changes.
Instead, we need a general algorithm that computes gradients efficiently for any differentiable neural network, regardless of its architecture.
This is exactly what backpropagation provides us.
Backpropagation is not just about storing intermediate values and reusing them whenever needed. It is much more than that.
It provides us an algorithm which means a systematic way of computing gradients efficiently for neural networks of different sizes and architectures.
Let’s See Backpropagation Step by Step
Before learning how to implement backpropagation in code, let’s first understand the complete mathematics behind it and see how it works step by step.
Before proceeding, let’s try to remember what happened during the forward pass.
A Quick Recap
We already know that in forward pass, information travels from the input layer to output layer.
At every layer, the neural network does a small computation.
Let’s see those computations one by one.
We began with the input feature [x]
The first hidden neuron computed
[
z_1=w_1x+b_1
]
The second hidden neuron computed
[
z_2=w_2x+b_2
]
At this point, we have computed the linear combinations for both hidden neurons.
Then the outputs from the hidden neurons are then passed through the ReLU activation function.
For the first hidden neuron, we have
[
a_1=mathrm{ReLU}(z_1)
]
Similarly, for the second hidden neuron
[
a_2=mathrm{ReLU}(z_2)
]
These activation values are now the inputs to the output layer.
Using the outputs from both hidden neurons, the output neuron computed the final prediction.
[
hat{y}=w_3a_1+w_4a_2+b_3
]
At this point, we got our predicted value.
Finally, we compared the predicted value with the actual value using the Mean Squared Error (MSE) loss function.
[
L=frac{1}{n}sum_{i=1}^{n}(y_i-hat{y}_i)^2
]
This gave us the loss, from which we understood how far our prediction is from the actual value.
Now, here’s something new which we should focus on.
Once the forward pass is done, we have the values of
[
z_1,quad
a_1,quad
z_2,quad
a_2,quad
hat{y},
quadtext{and}quad
L
]
for every training example in the dataset.
We’ll reuse all of these values during the backward pass.
Now, the forward pass is complete, and we have the prediction and the corresponding loss.
The next step is to determine how each weight and bias contributed to this loss.
Once we know how the loss changes with respect to every parameter, we can update them in a direction that reduces the loss.
This is exactly where the backward pass begins.
Note
Backward Pass: the execution phase during training.
Backpropagation: the algorithm used to compute gradients.
It’s time for backpropagation.
We have already seen that in the previous parts of this series, whenever we wanted to compute the gradient of a parameter, such as , we started from that parameter and applied the chain rule until we eventually reached the loss function.
Backpropagation approaches the same problem differently.
Instead of starting from a parameter, it starts from the loss function and systematically moves backward through the network.
As we have seen that the loss directly depends on the prediction, the first quantity computed during the backward pass is
[
frac{partial L}{partial hat{y}}
]
Before computing the gradients of any weight or bias, we must first know how the loss changes with respect to the prediction.
You might think that we’ve already computed it in previous parts, while finding the gradient of .
The difference is that we’re now solving it using the backpropagation algorithm, which always begins here at the loss.
We know our loss function
[
L=frac{1}{n}sum_{i=1}^{n}(y_i-hat{y}_i)^2
]
and in previous parts we already calculated this
[
frac{partial L}{partialhat{y}_i}
=
-frac{2}{n}(y_i-hat{y}_i)
]
We have calculated the first gradient during the backward pass.
Now, in backpropagation algorithm, once a gradient has been calculated, we don’t calculate it again.
Instead, we keep it and reuse it wherever it is required while moving backward through the network.
This is followed throughout the network.
Now that we have computed the first gradient, we know how the loss changes with respect to the predicted value.
Our next goal is to determine how the loss changes with respect to each weight and bias so that we can update these parameters to reduce the loss.
Moving Back to Output Layer
Now let’s move one step backward through our network. The next layer we arrive at is the output layer.
Here, we need to find how the loss changes with respect to , and .
We have the output equation
[
hat{y}=w_3a_1+w_4a_2+b_3
]
Let’s start with .
Using the chain rule,
[
frac{partial L}{partial w_3}
=
frac{partial L}{partialhat{y}}
cdot
frac{partialhat{y}}{partial w_3}
]
As we already calculated this in detail in previous parts, let’s just consider the final solution, as our main aim is to understand the process.
[
frac{partialhat{y}}{partial w_3}
=
a_1
]
From the previous step, we already stored
[
frac{partial L}{partialhat{y}}
=
-frac{2}{n}(y-hat{y})
]
Substituting both the results, we get
[
frac{partial L}{partial w_3}
=
-frac{2}{n}(y-hat{y})a_1
]
Similarly, for and
[
frac{partial L}{partial w_4}
=
-frac{2}{n}(y-hat{y})a_2
]
and
[
frac{partial L}{partial b_3}
=
-frac{2}{n}(y-hat{y})
]
We have now computed the gradients for the parameters in the output layer.
Moving Back to Hidden Layer
Now we move one step backward to the hidden layer.
It’s time to compute the gradients of the parameters in the hidden layer.
[
w_1,qquad b_1,qquad w_2,qquad text{and}qquad b_2
]
We have two hidden neurons in our hidden layers
Let’s first focus on the first hidden neuron.
During the backward pass, the gradient propagates through the following sequence:
[
L
rightarrow
hat{y}
rightarrow
a_1
rightarrow
z_1
rightarrow
w_1,; b_1
]
We can observe that before reaching the parameters [w_1] and [b_1] the gradient must first pass through the activation function.
We already know from the forward pass that
[
a_1=mathrm{ReLU}(z_1)
]
Therefore, the next step is to determine how the loss changes with respect to the hidden layer activation,
[
a_1
]
Once we have calculate this gradient, the next step is to move backward through the ReLU activation and see how the loss changes with respect to
[
z_1
]
After that, we can compute the gradients of [w_1] and [b_1]
Similarly, for the second hidden neuron, the gradient follows
[
L
rightarrow
hat{y}
rightarrow
a_2
rightarrow
z_2
rightarrow
w_2,; b_2
]
As both hidden neurons follow exactly the same sequence of steps, we’ll derive the gradients for the first hidden neuron.
The same procedure can then be applied to the second hidden neuron.
Let’s start by calculating the gradient of the hidden layer activation
[
a_1
]
Using the chain rule,
[
frac{partial L}{partial a_1}
=
frac{partial L}{partialhat{y}}
cdot
frac{partialhat{y}}{partial a_1}
]
From the previous steps, we already know
[
frac{partial L}{partialhat{y}}
=
-frac{2}{n}(y-hat{y})
]
and from the output layer equation,
[
frac{partialhat{y}}{partial a_1}
=
w_3
]
Substituting these into the chain rule, we get
[
frac{partial L}{partial a_1}
=
-frac{2}{n}(y-hat{y})w_3
]
It tells us how sensitive the loss is to changes in the hidden layer activation,
[
a_1
]
Now that we know how the loss changes with respect to [a_1] we can continue moving one step backward through the ReLU activation.
So far, we have determined how the loss changes with respect to the hidden layer activation
[
a_1
]
But our goal is to calculate the gradients of [w_1] and [b_1]
For that, we first need to move one step backward through the ReLU activation.
From the forward pass, we know that
[
a_1=mathrm{ReLU}(z_1)
]
So, our next goal is to determine how the loss changes with respect to
[
z_1
]
Using the chain rule,
[
frac{partial L}{partial z_1}
=
frac{partial L}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
]
We already have
[
frac{partial L}{partial a_1}
]
Now we only need to compute
[
frac{partial a_1}{partial z_1}
]
We know what ReLU does.
If the input to ReLU is negative, its derivative is
[
frac{partial a_1}{partial z_1}=0
]
If the input to ReLU is positive, its derivative is
[
frac{partial a_1}{partial z_1}=1
]
Substituting this into the chain rule, we get
[
frac{partial L}{partial z_1}
=
-frac{2}{n}(y-hat{y})w_3frac{partial a_1}{partial z_1}
]
Here, we simply write this derivative as
[
mathrm{ReLU}'(z_1)
]
Substituting this into the chain rule, we obtain
[
frac{partial L}{partial z_1}
=
-frac{2}{n}(y-hat{y})w_3mathrm{ReLU}'(z_1)
]
What does this equation tell us?
It tells us how sensitive the loss is to changes in
[
z_1
]
Now that we know how the loss changes with respect to [z_1] we can finally compute the gradients of [w_1] and [b_1]
From the forward pass we have
[
z_1=w_1x+b_1
]
Let’s start with [w_1]
Using the chain rule,
[
frac{partial L}{partial w_1}
=
frac{partial L}{partial z_1}
cdot
frac{partial z_1}{partial w_1}
]
we already computed
[
frac{partial L}{partial z_1}
]
So, we only need to compute
[
frac{partial z_1}{partial w_1}
]
as
[
z_1=w_1x+b_1
]
we get
[
frac{partial z_1}{partial w_1}=x
]
Substituting these into the chain rule, we get
[
frac{partial L}{partial w_1}
=
-frac{2}{n}(y-hat{y})w_3mathrm{ReLU}'(z_1)x
]
We have seen this already in part 2, which tells us how the loss changes with respect to
[
w_1
]
Next, we need to compute the gradient of [b_1]
Using the chain rule,
[
frac{partial L}{partial b_1}
=
frac{partial L}{partial z_1}
cdot
frac{partial z_1}{partial b_1}
]
Once again, we reuse the previously computed gradient
[
frac{partial L}{partial z_1}
]
Since
[
z_1=w_1x+b_1
]
we have
[
frac{partial z_1}{partial b_1}=1
]
Therefore,
[
frac{partial L}{partial b_1}
=
-frac{2}{n}(y-hat{y})w_3mathrm{ReLU}'(z_1)
]
Now, following the same procedure for the second hidden neuron, we get
[
frac{partial L}{partial w_2}
=
-frac{2}{n}(y-hat{y})w_4mathrm{ReLU}'(z_2)x
]
and
[
frac{partial L}{partial b_2}
=
-frac{2}{n}(y-hat{y})w_4mathrm{ReLU}'(z_2)
]
Finally, we have computed the gradients of every parameter in our neural network using the backpropagation algorithm.
Putting It All Together
If we look at the previous parts of the series, we can observe that the mathematics didn’t change.
The chain rule is still the foundation of backpropagation.
But the way we applied the chain rule has changed.
Instead of deriving the gradient of every parameter from scratch, backpropagation starts from the loss, moves backward through the network, and reuses previously computed gradients whenever they are needed.
This avoids repeating the same intermediate calculations and makes the process much more systematic.
This same strategy can be applied to much larger neural networks.
Whatever the number of layers or parameters be, backpropagation follows the same process to compute the gradients of all parameters.
The same process of forward pass and backward pass is repeated for every observation in the dataset.
For every training example, we get different intermediate values and gradients.
After computing the gradients, we update the weights and biases based on the optimization method being used, such as Batch Gradient Descent, Stochastic Gradient Descent (SGD) or Mini-batch Gradient Descent.
I hope you found this series helpful, gained something to build on, and enjoyed learning along the way.
If you’re new to this series and would like to read the previous articles, you can find them here.
I’d love to hear your thoughts. If you have any questions or feedback, feel free to leave a comment on LinkedIn.
If there’s a topic you’d like me to cover in a future article, I’d be happy to hear.
Complex ideas become simple when we understand them one step at a time.
Thanks for reading!