LSTM and GRU: gating as Hadamard products

Summary

LSTM and GRU are recurrent units with learned gates that decide, per timestep, what to keep and what to forget. Each gate is a sigmoid output, applied via Hadamard (elementwise) product to state or candidate vectors. The LSTM’s direct cell-state path is scaled coordinate by coordinate instead of transformed by another dense matrix, so gradients have a controlled route through time.

A vanilla RNN updates its hidden state as $h_t=\tanh (W{h}_{t-1}+U{x}_t)$. Backpropagating through $T$ timesteps multiplies the gradient by $W$ a total of $T$ times. Eigenvalues of $W$ less than 1 vanish; eigenvalues greater than 1 explode. Both kill learning (Pascanu et al., 2013).

LSTMs (Hochreiter & Schmidhuber, 1997) replaced repeated matmul on the cell state with a Hadamard-product update, opening a “gradient highway.” GRUs (Cho et al., 2014) simplified this to two gates with similar empirical performance.

Both are now mostly historical. Transformers replaced them everywhere. But the gating idea persists in attention masking, gating in mixture-of-experts, and residual gating in modern architectures.

The LSTM cell

State: hidden state $h_t$ and cell state $c_t$.

Gates (each computed from $[h_{t-1},x_t]$):

$$\begin{array}{rl}{f}_t& =\sigma (W_f[h_{t-1},x_t])\text{(forget)}\\ i_t& =\sigma (W_i[h_{t-1},x_t])\text{(input)}\\ o_t& =\sigma (W_o[h_{t-1},x_t])\text{(output)}\\ {\tilde{c}}_t& =\tanh (W_c[h_{t-1},x_t])\text{(candidate)}\end{array}$$

Update:

$$c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t,h_t=o_t\odot \tanh (c_t).$$

The forget gate controls what fraction of the previous cell state survives. The input gate controls what fraction of the new candidate is written. The output gate controls what fraction of the cell is exposed.

Learning objective: compute one LSTM cell update coordinate by coordinate, then read the direct backward multiplier from the same forget gate.

Worked state update

Each gate scales matching coordinates; it does not mix them.

Why Hadamard products fix gradients

The cell-state recurrence is $c_t=f_t\odot c_{t-1}+(\dots )$. Holding the gate values fixed, the direct cell-state path has Jacobian $\partial c_t/\partial c_{t-1}=\mathrm{diag}(f_t)$. If the forget gate stays near 1, gradients pass through nearly unchanged; if it goes near 0, that coordinate and its direct gradient are deliberately forgotten. This direct path cannot amplify a gradient because $0<f_t<1$, although repeated values below 1 can still make it decay.

The GRU cell

Two gates instead of three:

$$\begin{array}{rl}{z}_t& =\sigma (W_z[h_{t-1},x_t])\text{(update)}\\ r_t& =\sigma (W_r[h_{t-1},x_t])\text{(reset)}\\ {\tilde{h}}_t& =\tanh (W_h[r_t\odot h_{t-1},x_t])\\ h_t& =(1-z_t)\odot h_{t-1}+z_t\odot {\tilde{h}}_t.\end{array}$$

Single hidden state, no separate cell. Slightly fewer parameters. Empirically comparable to LSTM on most tasks.

Tradeoffs vs transformers

• Sequential: must process tokens one at a time. Cannot parallelize across the sequence dimension at training time.
• Linear in sequence length at inference (vs $O(n^2)$ for vanilla attention). The advantage that makes RNN-style models attractive again at very long context (Mamba, RWKV, linear attention).
• No explicit attention. Information from token $i$ to token $j$ has to survive $j-i$ gate updates. Long-range dependencies are still hard in practice.

Common pitfalls

• Treating LSTM and GRU as interchangeable. They are close empirically but the cell state in LSTM gives sharper control over long-range memory.
• Using vanilla RNNs in 2025. Almost never the right choice. Either go LSTM/GRU or, more likely, transformer.
• Forgetting truncated BPTT. Backpropagating through 100k tokens is infeasible; truncate at a window (typically 64 to 256 tokens) and cut gradients.

Related

• Backpropagation.
• The attention mechanism.
• Vanishing and exploding gradients.