Understanding softmax activation function

Foundation

For me, the best way to understand a mathematical function is by observing its behavior.

By doing this, we can see how a function behaves when we input different types of parameters and observe the outputs it produces. This is actually a general approach we can use to analyze mathematical functions.

I’ll start by looking at the behavior of the Exponential function. Pay attention to the graph below. Ignore the Python code, because that is not the focus here.

# Creating a range of x values from -5 to 5
x = np.linspace(-5, 5, 1000)
 
# Exponential function
y = np.exp(x)
 
# Creating the plot
plt.figure(figsize=(10, 6))
plt.plot(x, y, label='y = e^x')
 
# Adding details
plt.title('Plot of the Exponential Function y = e^x')
plt.xticks(range(-5, 5 + 1, 1))
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.grid(True)
 
# Displaying the plot
plt.show()

From the graph above, we can identify several properties of the exponential function:

1. The curve is always increasing (monotonically increasing) from left to right.
2. It never crosses the x-axis because:

3. It crosses the y-axis at $x = 0$, where $y = 1$.

These three properties are basic conclusions that we can derive directly from the behavior of the curve.

We can also observe that when $x$ is positive, the growth of $y$ becomes extremely fast. On the other hand, when $x$ is negative, $y$ decreases slowly toward $0$, but never actually reaches $0$.

In other words, $y = 0$ acts as a horizontal asymptote.

This function shows us an important characteristic: the larger the input value, the faster the output grows.

The growth is not linear. Instead, it is self-accelerating: as the input increases, the output grows at an increasingly faster rate.

From this perspective, $x = 0$ can be viewed as an important transition point where the output starts moving from values below $1$ to values above $1$, and the exponential growth becomes increasingly dominant as $x$ increases.

Softmax Correlation

Softmax is a function that intentionally takes advantage of this property of the exponential function.

The exponential function does something very important for Softmax:

It amplifies differences between values.

A relatively small difference in the input can become a much larger difference after applying the exponential function.

For example:

Notice what happens here.

The difference between $3$ and $4$ is only:

But after applying the exponential function, the values become:

and

The gap has become much larger:

So, exponential amplification makes larger input values increasingly dominant.

This behavior is exactly what Softmax takes advantage of when it converts a set of arbitrary scores into a probability distribution.