Activation functions

Summary

An activation inserts a nonlinear map between affine layers. Without it, repeated affine layers collapse into one affine map.

Take $h = 2 x + 1$ , then compare $y = 3 h - 4$ with $y = 3 ReLU (h) - 4$ . What changes at $x = - 1$ and $x = 1$ ? The graphs show activation output $g (h)$ , not the complete network output $y$ .

Optimization intuition

Flat activation regions weaken the gradient travelling backward

Derivation

Sigmoid: two saturating tails

The slope approaches zero at both output limits.

ReLU vs exact GELU

A half-line of zero slope differs from one smooth shoulder.

Read it this way: sigmoid loses slope in both tails. ReLU keeps unit slope for positive inputs and zero slope for negative inputs. Exact GELU has a smooth negative shoulder, but its derivative can still equal zero.

Check: Where would an incoming gradient shrink or become zero? How is that different for GELU?

Read it this way: ReLU has zero derivative on its negative side. Sigmoid slopes become small in both tails; their forward outputs are not gradients.
Where does the same small input move produce almost no output change?

Trace the two inputs

Without activation, substitution gives $y = 3 (2 x + 1) - 4 = 6 x - 1$ everywhere. ReLU instead clips negative $h$ to zero.

Input $x$	$h = 2 x + 1$	Affine output	ReLU output	ReLU path derivative $d y / d x$
$- 1$	$- 1$	$- 7$	$- 4$	$0$
$1$	$3$	$5$	$5$	$6$

The activated network is constant at $- 4$ when $x < - 1/2$ , and equals $6 x - 1$ when $x > - 1/2$ . Its breakpoint is $x = - 1/2$ . Both branches meet there, but the derivative is not unique. Frameworks choose a derivative convention at the kink.

For positive $h$ , ReLU’s local slope is one. For negative $h$ , it is zero, removing this path’s gradient on that input. Sigmoid has derivative $σ (h) (1 - σ (h))$ , which approaches zero in both tails. Full gradient flow also depends on weights and other paths.

Change the requirement

1. Keep a negative-region gradient

New task: this path repeatedly receives $h = - 2$ and needs a negative-region update.

Picture and change: replace ReLU by leaky ReLU with negative slope $0.1$ . This changes both values and gradients.

Check: activation output becomes $- 0.2$ with local slope $0.1$ , instead of zero output and slope. The complete output is $- 4.6$ , and $d y / d x = 3 (0.1) (2) = 0.6$ .

Use when: persistent inactivity is observed, with the changed function acceptable for the task.

2. Change label exclusivity

New task: logits are $(0, lo g 4)$ . Choose outputs for one exclusive class versus two labels that can coexist.

Picture and change: softmax normalizes jointly; separate sigmoids produce individual Bernoulli probabilities.

Check: softmax gives $(0.2, 0.8)$ ; sigmoids give $(0.5, 0.8)$ . The latter need not sum to one.

Use when: target structure changes, rather than merely changing a hidden activation.

Check the loss input contract

Do not apply softmax before a cross-entropy API that expects logits. It would transform already transformed values. Check whether the API accepts logits, probabilities or log probabilities.

Try diagnosing a negative ReLU input

Does one negative input mean the unit is permanently dead?

Check your answer

No. Other inputs or upstream parameter changes can reactivate it. Persistent inactivity across relevant inputs is the concern.

The standard family

Here $α > 0$ for ELU and $0 < α < 1$ for leaky ReLU.

Activation	Formula	Range	Constraint or role
Sigmoid	$σ (h) = 1/ (1 + e^{- h})$	$(0, 1)$	Bernoulli outputs and gates; saturating tails
Tanh	$tanh h = 2 σ (2 h) - 1$	$(- 1, 1)$	Odd and centered at the origin, not necessarily zero-mean over data
ReLU	$max (0, h)$	$[0, \infty)$	Unit positive slope, zero negative slope
Leaky ReLU	$max (α h, h)$	$R$	Nonzero negative slope
ELU	$h$ if $h > 0$ ; $α (e^{h} - 1)$ otherwise	$(- α, \infty)$	Negative outputs; smooth at zero only when $α = 1$
Exact GELU	$h Φ (h)$ , normal CDF $Φ$	Bounded below, unbounded above	Small negative shoulder; not every real output is attainable
SiLU / swish with unit scale	$hσ (h)$	Bounded below, unbounded above	Smooth, nonmonotone negative shoulder
Softmax	$e^{z_{i}} / \sum_{j} e^{z_{j}}$	Probability simplex interior for finite logits	Couples coordinates; not an elementwise activation

Smoothness does not guarantee gradient flow

Exact GELU has derivative $Φ (h) + h ϕ (h)$ , where $ϕ$ is the normal density. It can vanish at isolated points. Exact and approximate GELU differ; match framework settings for reproducibility. Activation choice and initialization must be considered together. No activation guarantees better task accuracy from its curve alone.

References: ReLU, GELU, swish, and feedforward networks.

SwiGLU and gated activations

For row-vector batches $x \in R^{n \times d}$ , use $W_{g}, W_{u} \in R^{d \times m}$ and $W_{d} \in R^{m \times d}$ :

SwiGLU (x) = (SiLU (x W_{g}) ⊙ x W_{u}) W_{d} .

This has $3 d m$ projection parameters, excluding biases, versus $2 d m$ for a two-projection block at equal hidden width. Equal parameter budgets require adjusting width; performance comparisons must state that budget.

When to use which output activation

Task	Output activation
Binary classification	Sigmoid
Multi-class classification	Softmax
Multi-label classification	Separate sigmoid heads; labels need not be mutually exclusive
Regression (unbounded)	Identity (no activation)
Regression (bounded $[0, 1]$ )	Sigmoid
Probability over a discrete distribution	Softmax
Embedding output	Identity; normalize if the similarity objective requires it

Common pitfalls

Output constraints must match the target: softplus is positive, while ReLU can produce exact zeros and inactive gradients. Bounded sigmoid outputs do not by themselves establish calibrated probabilities.

Weight initialization. Initialization is activation-dependent.
Exploding and vanishing gradients. What motivates ReLU and gated activations.