Today I Finally Understood Why We Plot the Derivative of Activation Functions

When I started learning Deep Learning, I thought activation functions were just mathematical equations we had to memorize.

Today, while implementing Sigmoid, Tanh, ReLU, Leaky ReLU, and Softmax from scratch in Python, I realized something much more interesting.

The activation function itself is only half of the story.

The derivative is what actually teaches the neural network how to learn.

My Experiment

I generated 100 equally spaced values between -10 and 10:

z = np.linspace(-10, 10, 100)

Enter fullscreen mode Exit fullscreen mode

Then I implemented:

  • Sigmoid
  • Tanh
  • ReLU
  • Leaky ReLU
  • Softmax

For the first four, I plotted both the activation function and its derivative.

Initially, I wondered:

Why am I plotting the derivative? Isn't the activation function enough?

After learning about backpropagation, the answer became clear.

What the Activation Function Tells Us

The activation function answers:

"What output should this neuron produce?"

For example, Sigmoid compresses any input into a value between 0 and 1.

ReLU outputs 0 for negative inputs and passes positive inputs unchanged.

This is what happens during the forward pass.

Input
   ↓
Weighted Sum
   ↓
Activation Function
   ↓
Prediction

Enter fullscreen mode Exit fullscreen mode

What the Derivative Tells Us

The derivative answers a completely different question:

"How much should this neuron change to reduce the error?"

During the backward pass, the neural network computes gradients.

Those gradients come directly from the derivatives of the activation functions.

Without derivatives, the model wouldn't know how to update its weights.

Prediction
     ↓
Calculate Error
     ↓
Compute Derivatives
     ↓
Update Weights

Enter fullscreen mode Exit fullscreen mode

That was the moment when the plots finally made sense to me.

My Biggest Observation

Sigmoid

Beautiful S-shaped curve.

But its derivative almost becomes zero for very large positive or negative inputs.

That means learning slows down because the gradients almost disappear.

This is called the vanishing gradient problem.

Tanh

At first glance, it looked almost identical to Sigmoid.

But I noticed something important:

Its output is centered around zero.

That small difference helps optimization because the gradients are more balanced around the origin.

ReLU

The simplest graph was also the most surprising.

f(x)=max(0,x)

Enter fullscreen mode Exit fullscreen mode

Its derivative is

0   (negative inputs)

1   (positive inputs)

Enter fullscreen mode Exit fullscreen mode

This explains why ReLU trains deep neural networks much faster than Sigmoid.

Leaky ReLU

ReLU has one weakness.

Negative neurons completely stop learning because the derivative becomes zero.

Leaky ReLU fixes that with a tiny negative slope.

Instead of

Gradient = 0

Enter fullscreen mode Exit fullscreen mode

it becomes

Gradient = 0.01

Enter fullscreen mode Exit fullscreen mode

A small change mathematically—but an important one during training.

Softmax

Softmax was different from the others.

It doesn't transform a single value.

It transforms an entire vector into probabilities.

Example:

Input

[2,4,1]

↓

Output

[0.114
 0.844
 0.042]

Enter fullscreen mode Exit fullscreen mode

Every probability depends on every other input.

That explains why we usually don't visualize Softmax as a simple curve.

Today I stopped thinking of activation functions as just formulas.

Now I see them as two complementary ideas:

  • The activation function decides what a neuron outputs.
  • Its derivative decides how that neuron learns.

The graph explains the prediction.

The derivative explains the learning.

My next goal is to implement:

  • Forward propagation from scratch
  • Backpropagation from scratch
  • Gradient Descent
  • A simple neural network without TensorFlow or PyTorch

I want to understand the mathematics before relying on deep learning frameworks.

When you first learned neural networks, what concept changed your understanding the most—activation functions, backpropagation, or gradient descent?

DeepLearning #MachineLearning #Python #ArtificialIntelligence #NeuralNetworks #LearningInPublic #Bioinformatics #100DaysOfCode #DevCommunity