Welcome back!
We recently discussed backpropagation, and I hope you now have an idea of what backpropagation is and how it actually works.
Let’s continue the deep learning journey.
Even though we apply the backpropagation algorithm to a neural network, we still have some problems, and vanishing gradients is one of them.
While I was learning about vanishing gradients, I came across the sigmoid function.
We all know that it is used in logistic regression, where we apply the sigmoid function to a value to obtain an output between 0 and 1.
Now, here in neural networks, it can be used as an activation function.
What I know about sigmoid is the equation we have and its usage in logistic regression and neural networks.
I was curious about how we get this equation and the story behind it.
In this blog, let’s see how we get to the sigmoid equation.
By the way, if you haven’t read Part 3 of the backpropagation series, you can read it here.
···
How Do We Actually Use Sigmoid?
We already know the equation of the sigmoid function.
Before we proceed, let’s see how we use it in logistic regression.
For example, we want to predict whether a student will pass or fail based on the number of hours they studied.
We are using the logistic regression model here.
First, it calculates a score
Let’s say the score for a student is:
This score is not a probability.
It is just the linear combination of parameters.
Now we pass it through the sigmoid function:
we get,
The sigmoid function always produces an output between 0 and 1.
Here the output is approximately 0.88 or 88%.
In logistic regression, this can be interpreted as an 88% probability of the student passing the exam.
We can then use a threshold, such as 0.5, to make the final classification.
In short, the flow can be like
That’s how we commonly use the sigmoid function in logistic regression.
But What Is This “e”?
Now, let’s once again look at the sigmoid equation.
The first thing we notice is the e.
We know that it is a mathematical constant and its value is
But what exactly is ‘e’?
Why is this number present in the sigmoid equation?
Let’s take a step back and understand where this number comes from.
One thing is that here we are not trying to discover ‘e’, but the goal is to understand the significance of ‘e’ and see where it naturally appears.
Now let’s go to the bank and see what we can observe.
Let’s Start with a Simple Bank Example
Imagine we deposited Rs.100 into a bank account.
Let’s say the bank is giving us a 100% annual interest rate.
If the bank adds the entire year’s interest at the end of the year, we earn Rs.100 in interest.
So after one year, we have
We can also write it as
Rs.100 became Rs.200 after one year.
But now let’s change one thing.
What if the bank doesn’t wait until the end of the year to add the interest?
What if it adds the interest twice a year?
The annual interest rate is still 100%.
But now the year is divided into two periods.
So for each six-month period we get half of the annual interest rate:
During the first six months, we get
After six months, we have Rs.150.
During the next six months, the interest is calculated on this new amount
Then we have
Why did we get Rs.225 instead of Rs.200?
Because the interest earned during the first six months also earned interest during the second six months.
In simple terms we can say
‘interest earns interest’
This is the basic idea behind compound interest.
What Happens When We Compound More Frequently?
Now let’s make the compounding more frequent.
If we compound four times a year:
If we compound 12 times a year:
If we compound every day:
Observe the pattern.
As we increase the number of compounding periods, the final amount keeps increasing.
The reason is that growth is being applied repeatedly to an amount that has already increased.
Where Does e Come From?
The Rs.100 is not the important part here.
Let’s remove it and look at the growth factor:
Here, ‘n’ represents the number of times we compound during the year.
For example:
As we make the compounding more and more frequent, the value gets closer and closer to
This number is called ‘e’
Mathematically, we can express this idea using a limit
The notation may look complex, but the idea is simple.
Here, we are asking:
“What value does this expression approach as ‘n’ becomes larger and larger?”
As ‘n’ increases:
gets closer and closer to:
That limiting value is ‘e’.
So, What Does the Bank Have to Do with Sigmoid?
But why are we talking about this and what does this bank account have to do with sigmoid.
This example isn’t to explain compound interest, but it gives us an intuition for where ‘e’ naturally appears.
The important idea here is repeated growth.
When growth is repeatedly applied to an amount that has already grown, we get a compounding process.
And when that process happens again and again more frequently, the number ‘e’ naturally appears.
So instead of simply memorizing that
we now have some intuition behind it.
The Special Property of e
From the bank example, we observed that ‘e’ naturally appears when we look at repeated growth and continuous compounding.
But ‘e’ is more than just a number that appears in compound interest.
It has a very special property when we look at it through calculus.
Let’s consider the exponential function
If we differentiate this function, we get
This formula we already know.
But what does the derivative tell us?
We already know that it tells us the rate of change of a function.
For example, if we have
its derivative is
This means that the rate at which x2 changes depends on the value of x.
At x=1:
At x=3:
So, for x2, the function and its rate of change are different.
Now let’s look at ex.
For
we have
This means that the rate of change of ex is equal to its current value.
Let’s look at some values.
When x=0
and
When x=1
and
When x=2
and
So, here we can say that
Rate of change = Current value
This is one of the most important properties of the exponential function with base e.
Why Is the Derivative of ex Equal to ex?
We now have an idea of an important property of ‘e’ in calculus.
We just discussed what it is but let’s see why does this happen?
If you already know why
then use this section for quick revision as we connect it back to the sigmoid function.
Starting with a General Exponential
First let’s consider a general exponential function.
Here, z is the base and x is the exponent.
are all examples of this form.
Now let’s see what happens when we differentiate zx
We have,
Using the exponent rule we get
Therefore
Now notice that zx appears in both terms in the numerator.
We can factor it out
Here zx does not depend on h, so we can take it outside the limit
And this is where things get interesting.
Our result is
Look at the two parts separately.
The first part is
That is our original exponential function.
The second part is
We can see that there is no ‘x’ in this expression.
It depends on the base ‘z’, but not on ‘x’.
This means, for any value of ‘z’, this entire limit is just a constant.
Let’s call this constant ‘C’.
Therefore we can write it as,
This tells us something important.
When we differentiate an exponential function, we get the original exponential function, multiplied by a constant.
In other way,
The Constant Depends on the Base
Now let’s take an example of exponential function:
From our result, we have
For z=3, the constant is
Now we need to find the value of this limit.
Let’s understand this in intuitive way.
For the base 3, the value of the constant is approximately
Therefore,
Let’s see what this tells us by using at different ‘x’ values.
When
we have
the rate of change here is approximately
When
we get
The rate of change is
And when
we have
The rate of change is approximately
We can see that the derivative is not exactly equal to 3x.
Instead, we got
The function and its rate of change have the same exponential shape, but the rate of change is scaled by a constant.
Finding the Special Base
Now, we know that
The value of ‘C’ depended on the base.
For 3x,
Ok but what if we could find a base for which C is exactly 1?
Do we have any number?
If yes, then we get
Our derivative would become
In other words, we can say that the function would be exactly equal to its own derivative.
So, now we are looking for a base z that satisfies
There is one particular positive number that satisfies this condition and you all know what’s that number is.
We call this number
and its numerical value is
For this particular base, the constant becomes
Therefore,
which gives us
···
So What Did We Actually Discover?
We started with a general exponential function
Using the definition of a derivative, we found
We then observed that the limit is simply a constant that depends on the base.
Then we have written it as
Then we asked:
Is there a base for which C=1?
The answer is yes.
That special base is e.
Therefore,
Now we have an idea of how we got the derivative.
In the earlier bank example, ‘e’ appeared through repeated growth and continuous compounding.
Now, through calculus, we have seen another special property of the same number
In simple words, we can say that ex grows at a rate equal to its current value.
Now, Let’s Return to Sigmoid
Let’s once again look at the sigmoid equation.
Now we have some idea of what ‘e’ actually is.
Now we focus on the whole equation.
The question here is why does the sigmoid function is in this particular form?
To understand this we should go back to logistic regression.
We started with a raw score
‘z’ can be any real number.
But for classification, we wanted to interpret the model’s output as a probability.
A probability must lie between 0 and 1
0<p<1
So we want to transform any value of ‘z’ into a value between 0 and 1.
In other words, we want something that can receive
and produce:
Building a Function That Outputs Between 0 and 1
Now, the task is to construct such transformation.
But how can we do that?
Let’s start with a very simple observation.
Suppose we have a number greater than 1.
For example
If we take its reciprocal, we get
which is between 0 and 1.
The same idea works for any numbers greater than 1
Here we can notice that
If
then
This gives us a simple idea.
If we can have a quantity that is always greater than 1, then taking its reciprocal will automatically give us a value between 0 and 1.
And that is exactly the range we want for a probability.
However, there is one more thing we need.
We do not want to use a fixed number such as 5 in the denominator.
because that always give us the same output.
Our output should change when the input ‘x’ changes.
For example, we want a positive input to produce a larger probability, while a negative input should produce a smaller probability.
So, we need a quantity that changes with x.
Now e Enters the Picture
You are right. It’s time for ‘e’ to enter.
This is where the exponential function we just learned about becomes useful.
Exponential functions are always positive, which means
for every real value of x.
For example:
Whether the x is negative, zero, or positive, ex never becomes negative or zero.
But the sigmoid equation contains e-x.
Till here we only discussed about ex.
So let’s first see what a negative exponent means.
We already know what a positive exponent means.
For example:
and:
A negative exponent represents the reciprocal of the corresponding positive exponent.
For example:
Similarly
and
In general, we can write as
So, e-x is not a completely different function.
It is simply the reciprocal of ex.
Now we can use what we already know about ex.
Since:
its reciprocal is also positive
and because
we get
for every real value of x.
This is important because it gives us exactly the kind of quantity we need.
If e-x is always positive, then adding 1 gives us a quantity that is always greater than 1
And now we can use our reciprocal idea.
If a number is greater than 1, its reciprocal lies between 0 and 1
Now we have a function whose output is always between 0 and 1.
The expression we just got is
and this is exactly the sigmoid function we started with
So instead of looking at the sigmoid equation as a formula, now we can understand the intuition behind its structure.
We wanted the output to lie between 0 and 1.
We observed that the reciprocal of a number greater than 1 lies between 0 and 1.
As e-x is always positive, we used it to construct a quantity greater than 1
Taking its reciprocal gave us
This gave us the range we wanted.
But does this equation actually behave the way we expected it to do?
Here, our goal is to understand the intuition behind the structure of the sigmoid function.
There are other functions that can map values to the range 0 to 1, and why logistic regression uses sigmoid is related to odds and log-odds, a topic which we will explore in future blogs.
Does the Sigmoid Behave the Way We Expected?
Let’s test on few values.
First, let’s consider
Substituting into the sigmoid function:
as
we get
When the input is 0, the sigmoid gives us exactly 0.5.
Now let’s take a positive number
then
We already seen
which gives
The sigmoid converted the input 2 into approximately 0.881 or 88.1%.
Now let’s see what happens when the input is a negative number.
Consider
Then
We know
Finally we get
So the sigmoid converted the input -2 into approximately 0.119 or 11.9%.
Now we can see how the sigmoid behaves.
For a negative input:
For zero:
For a positive input:
So as x increases, the sigmoid output moves from values close to 0, passes through 0.5 and moves toward 1.
In the extreme cases:
and
This is exactly the behavior we wanted from a function that transforms any real number into something between 0 and 1.
Now we have an idea of how we got the equation of the sigmoid function.
If you remember, in my recent blogs, when we discussed backpropagation and neural networks in general, we talked about activation functions and why they are important.
We used the ReLU activation function to understand those concepts.
Now, we can also use sigmoid as an activation function.
But if we use sigmoid as an activation function, there is one more thing we need to know.
During the backward pass, we already know that the network calculates gradients using derivatives.
So, if sigmoid is part of the network, we need to differentiate it as well.
Now let’s focus only on deriving the derivative of the sigmoid function step by step.
Instead of carrying the exponential term throughout calculations, we can simply use the sigmoid output itself.
This is the derivative we use whenever sigmoid appears in the gradient calculations of a neural network.
···
Summary
In the upcoming blogs, we are going to discuss topics like vanishing gradients and exploding gradients.
As we explore these topics, we will come across the sigmoid function, and we will also need its derivative.
If we derive the sigmoid function and its derivative in those blogs, the discussion could become long, and we may lose focus on the actual concept that we are trying to understand.
It would also be better to have an idea of where the sigmoid function and its derivative come from before using them in further concepts.
We first started with the bank example to see how e appears. We then learned about its important property in calculus and, using these ideas, gradually built the sigmoid equation.
We saw how this equation is used in logistic regression and neural networks, and we also derived its derivative.
Now, when we move on to the upcoming topics, we already have this foundation which will be useful for us.
I hope you found this blog helpful in understanding a concept that we frequently use.
If you have any questions or suggestions for improvement, feel free to share them in the comments on LinkedIn.
And if you haven’t read my latest blog series on backpropagation yet, you can read it here.
Sometimes, moving forward means going back and understanding the basics.
Thanks for reading!
···