Long Short-Term Memory (LSTM)

Long short-term memory (LSTM) is a gated recurrent neural network architecture for processing sequences. An LSTM layer carries two vectors from one time step to the next: a hidden state, which is also exposed as the layer's output, and a cell state, which provides a controlled path for retaining information. Learned input, forget, and output gates regulate how the cell state changes and how much of it is revealed. Sepp Hochreiter and Jürgen Schmidhuber introduced the architecture in a 1997 paper in Neural Computation. Their experiments showed that it could learn several synthetic tasks with minimal time lags of more than 1,000 steps, where the recurrent methods tested in the paper did not learn successfully.^[1]

LSTM was designed in response to the difficulty of training ordinary recurrent networks on dependencies separated by many time steps. Its additive cell update gives gradients a more direct route through a sequence than the repeated nonlinear hidden-state update of a simple recurrent network. This design reduces, but does not eliminate, vanishing and exploding gradients. LSTM models still require suitable optimization, initialization, data, and state management. They also retain the sequential dependency of recurrent computation, which limits parallelism across time during training.^[1][3][10]

During the 2010s, researchers applied LSTM to speech recognition, handwriting recognition and generation, machine translation, image captioning, and language modeling.^{[15][19][20][23][24]} The 2017 Transformer paper removed recurrence from its encoder and decoder and emphasized that self-attention permits more parallel computation during training.^[22] LSTM implementations remain available in major machine-learning frameworks for sequence and step-wise recurrent computation.^[10][11]

Motivation

A simple recurrent network can be written as:

h_t = phi(W_x x_t + W_h h_(t-1) + b)

Here x_t is the input at time t, h_t is the hidden state, and phi is a nonlinear activation. Training with backpropagation through time differentiates through repeated applications of this update. The gradient connecting distant time steps contains a product of many Jacobian matrices. Depending on their singular values and the operating points of the nonlinearities, that product can contract toward zero or grow rapidly.

Bengio, Simard, and Frasconi analyzed the resulting trade-off in 1994. A recurrent system that robustly retains information is resistant to changes in its state, but gradient descent then has difficulty assigning credit for that information over a long interval. Their result concerns the increasing difficulty of learning long-term dependencies, not a universal maximum sequence length for every recurrent network.^[2] Pascanu, Mikolov, and Bengio later separated the geometry of the vanishing and exploding cases and described gradient-norm clipping as a response to exploding gradients. Clipping bounds an update when its norm becomes too large, but it does not restore a gradient that has already vanished.^[3]

LSTM changes the state transition rather than relying only on an optimization remedy. Its cell state is updated by addition after elementwise gating. Along the direct cell-state path, the multiplier from one step to the preceding step is the forget-gate value. A value near one can preserve both state and gradient information over that step. The total derivative through a complete LSTM also includes paths through gates and hidden states, so the architecture does not guarantee constant gradients in every trained network.

Development

The 1997 LSTM contained memory cells with fixed self-connections and two multiplicative gates. The input gate controlled writes to a cell, while the output gate controlled access to its activation. Hochreiter and Schmidhuber called the unit's fixed self-connected path a constant error carousel. Their learning procedure truncated selected error paths while retaining the protected path through the cell. The paper reported constant computational complexity per time step and weight, and compared LSTM with real-time recurrent learning, backpropagation through time, recurrent cascade correlation, Elman networks, and neural sequence chunking on artificial tasks.^[1]

The original design did not include a forget gate. Gers, Schmidhuber, and Cummins identified a failure mode on continual streams: without externally marked sequence boundaries and resets, internal cell values could grow and saturate. Their 1999 conference paper and 2000 journal article added an adaptive forget gate that lets a cell reset its own state. The three-gate formulation derived from that work is the usual starting point for modern LSTM implementations.^[4]

Gers, Schraudolph, and Schmidhuber subsequently studied peephole connections from a cell state to its gates. In their timing experiments, peephole LSTM distinguished spike sequences separated by 49 versus 50 steps and generated precisely timed periodic outputs. Those results establish the usefulness of peepholes for the paper's timing tasks, not a general advantage on all sequence problems.^[5]

Bidirectional recurrent networks process a complete input sequence in both temporal directions. Graves and Schmidhuber combined bidirectional processing with LSTM for framewise phoneme classification in 2005. A bidirectional LSTM can use preceding and following context for each output position, but it is unsuitable for a causal output that must be produced before future inputs arrive.^[6]

Connectionist Temporal Classification (CTC), introduced by Graves and colleagues in 2006, supplied an objective for labeling unsegmented sequences. It adds a blank label and sums over possible alignments between an input sequence and a shorter target sequence. CTC is not part of the LSTM cell, although early CTC experiments used recurrent networks and many later systems paired it with LSTM.^[7]

Greff and colleagues compared eight LSTM variants in 5,400 experimental runs across speech, handwriting, and music tasks. They found that the forget gate and output activation were the most important evaluated components, while none of the tested variants improved significantly over the standard architecture across all three data sets. The result is bounded to the variants, tasks, and search procedure in that study.^[8] A separate search by Jozefowicz, Zaremba, and Sutskever evaluated more than 10,000 recurrent architectures. It found cells that beat LSTM and GRU on some tasks but not all of them, and found that adding a bias of one to the LSTM forget gate closed the observed performance gap between LSTM and GRU in their comparison.^[9]

Standard cell

Let x_t be an input vector, h_(t-1) the previous hidden state, and c_(t-1) the previous cell state. A common LSTM without peepholes or projection computes:

Scroll sideways for more →

The asterisk denotes elementwise multiplication. A sigmoid produces values between zero and one, so each gate is a differentiable, coordinate-wise control rather than a hard switch. The candidate and exposed cell state use tanh in this formulation. Frameworks may concatenate the input and previous hidden state into one matrix multiplication or store separate input and recurrent matrices. These arrangements are algebraically equivalent when the same gates and dimensions are used.^[10]

If the input width is I and both the hidden and cell widths are H, a one-layer cell with one bias vector per gate has 4H(I + H) + 4H trainable parameters. An implementation with separate input and recurrent biases stores 8H bias parameters instead. Projection layers, peepholes, coupled gates, and other variants change the count. PyTorch, for example, stores separate input-hidden and hidden-hidden weights and biases, and optionally applies a learned projection to the hidden output.^[10]

The hidden state and cell state serve different roles. c_t is the internal memory carried along the additive path. h_t is the gated output supplied to the next recurrent step and, in a stacked network, to the next layer. A model can return the output at every time step or only a final state. Initial states are commonly zero unless an application supplies or carries state from an earlier segment.

Gradient flow

For the direct update c_t = f_t * c_(t-1) + i_t * g_t, the partial derivative of c_t with respect to c_(t-1), while holding gate values fixed, is f_t. The direct contribution across several steps therefore contains a product of forget gates. This path can remain near one when the relevant gate coordinates stay open, or decay when they stay below one. Other derivative paths pass through the output, candidate, and gates. LSTM provides a mechanism that can preserve error signals; it does not make optimization independent of sequence length.^[1][4]

The input and forget terms also let the network choose between retention and replacement. If f_t is near one and i_t is near zero, a coordinate changes little. If the forget gate closes and the input gate opens, old content is removed while new content enters. These statements describe the equations. They do not imply that a trained gate has a unique human-readable meaning.

Tensor shapes and sequence output

Framework interfaces distinguish sequence length, batch size, input width, hidden width, layer count, and direction count. In PyTorch, an unprojected, one-direction LSTM with batch_first=True accepts an input shaped (batch, sequence, input_size) and returns an output shaped (batch, sequence, hidden_size). Final hidden and cell states add layer and direction dimensions. Bidirectional output has twice the hidden width because the forward and reverse outputs are concatenated.^[10]

TensorFlow's Keras LSTM layer can return only the last output, the full output sequence, or the final hidden and cell states. Its documented defaults include tanh for the activation, sigmoid for the recurrent activation, an orthogonal recurrent initializer, and unit_forget_bias=True, which adds one to the forget-gate bias at initialization. Keras uses a cuDNN-backed path only when a documented set of arguments and input conditions is satisfied.^[11] These are framework defaults, not requirements of the LSTM architecture.

LSTM and GRU

The gated recurrent unit (GRU) was introduced in the recurrent encoder-decoder of Cho and colleagues in 2014. It uses update and reset gates and maintains one recurrent state rather than separate hidden and cell states. In the paper's formulation, the update gate interpolates between the previous state and a candidate state, while the reset gate controls the previous state's contribution to the candidate.^[12]

For equal input and hidden widths, a conventional GRU has three groups of affine transformations where an LSTM has four, so it usually has fewer parameters. That difference does not establish which cell will perform better on a new data set. Chung and colleagues compared GRU and LSTM on polyphonic music modeling and speech-signal modeling. Both gated cells outperformed a simple tanh recurrent unit in most of their experiments, while their relative ranking varied by task and model size.^[13] The larger architecture searches also found no cell that dominated every evaluated problem.^[8][9]

Scroll sideways for more →

Training and state handling

LSTM is normally trained by backpropagation through time. Full backpropagation stores activations across the entire unrolled sequence. Truncated backpropagation limits that graph to a finite window while carrying numerical hidden and cell states into the next window. Detaching the carried state prevents gradients from crossing the boundary. The chosen window controls which dependencies receive direct gradient credit, even though the numerical state itself can persist longer.^[1]

Gradient clipping addresses rare, large gradient norms and is applicable to LSTM because gating does not remove every path that can amplify a derivative.^[3] A clipping threshold is a training hyperparameter, not a property of the cell, so there is no task-independent value that should be prescribed for every model.

Forget-gate initialization changes the initial retention timescale. The empirical architecture search by Jozefowicz and colleagues supports adding one to the forget bias in the settings they tested.^[9] Keras adopts that option by default.^[11] PyTorch exposes the underlying bias tensors but does not document an automatic forget-bias offset in torch.nn.LSTM.^[10] Reporting the framework and version matters when reproducing an initialization recipe.

Dropout also requires attention to placement and masking. Gal and Ghahramani studied a recurrent dropout method that uses the same dropout mask at each time step, and evaluated it on language and speech-recognition tasks.^[14] PyTorch's built-in multilayer dropout argument applies dropout to outputs between recurrent layers, not to the final layer's output and not as a replacement for every recurrent regularization method.^[10] AWD-LSTM later combined several regularizers, including DropConnect on hidden-to-hidden weights, with averaged stochastic gradient descent for language modeling.^[15]

Stateful operation can be useful for streams, but sample identity and reset boundaries must be managed explicitly. Carrying a state from one unrelated sequence into another leaks information across examples. Padding and variable-length batches likewise require masking, packed sequences, or an equivalent mechanism so that padding does not become unintended input. Bidirectional processing requires the full sequence and therefore cannot provide a strictly online result.^[6][10][11]

Variants

LSTM names a family rather than one immutable cell. The following variants change either the recurrence, the topology over which it runs, or the representation stored in memory.

Scroll sideways for more →

Tree-LSTM is not simply a bidirectional chain. Its state is computed from child nodes in a supplied tree, and different formulations handle an unordered set of children or an ordered, fixed-arity tree.^[16] ConvLSTM keeps spatial axes in the hidden and cell states so the recurrence can model a sequence of grids. Its original paper compared fully connected LSTM, ConvLSTM, and an operational radar-echo extrapolation method on precipitation nowcasting data from Hong Kong.^[17]

xLSTM is a later extension and has a separate xLSTM article. The 2024 NeurIPS paper introduced sLSTM, with exponential gating and scalar memory, and mLSTM, with a matrix memory and covariance-style update. The authors designed mLSTM so its recurrence can be parallelized during training. Their reported language-model results are comparisons under the paper's chosen model sizes, training data, and evaluation setup, not evidence that every xLSTM configuration is superior to every transformer or state-space model.^[18]

Documented applications

LSTM's applications are best described through specific systems and evaluations rather than a claim that the cell is the default for an entire present-day field.

Speech and sequence labeling

Graves and Schmidhuber's bidirectional LSTM study evaluated framewise phoneme classification on the TIMIT speech corpus.^[6] Graves, Mohamed, and Hinton later trained deep bidirectional recurrent networks on TIMIT. Their best LSTM result had a 17.7 percent phoneme error rate in that experimental setup.^[19] CTC made it possible to train sequence labelers without a frame-level alignment between acoustic inputs and target symbols.^[7] These historical results helped establish recurrent networks for speech recognition, but they do not describe current production systems that have not published their architectures.

Machine translation

Sutskever, Vinyals, and Le used a multilayer LSTM encoder and decoder for English-to-French translation in 2014. The encoder produced a fixed-dimensional representation of a source sentence, and the decoder generated the target sentence. An ensemble of five LSTMs trained with reversed source sentences obtained 34.8 BLEU on the paper's WMT 2014 test set.^[20] Bahdanau, Cho, and Bengio then allowed a recurrent decoder to form a weighted context from encoder states at each output step, removing the need to place all source information in one fixed vector.^[21]

The attention mechanism in those systems supplemented a recurrent encoder-decoder. The 2017 Transformer removed recurrence from its encoder and decoder and used self-attention plus feed-forward layers. On the two machine-translation tasks in its paper, it improved the authors' reported quality while allowing more parallel training.^[22]

Handwriting and other generation

Graves used recurrent networks to generate online handwriting and character sequences. The handwriting model predicted a mixture distribution over pen movements and a Bernoulli end-of-stroke variable, with a recurrent attention window for conditional generation from text.^[23] This is a concrete example of LSTM modeling both discrete and continuous sequences; it does not imply that generated handwriting is a general test of long-term memory.

Image captioning

The "Show and Tell" system used a convolutional image network to produce an image representation and an LSTM to generate a caption. Vinyals and colleagues evaluated the system on several caption data sets and reported results using BLEU and human judgments.^[24] The work is an early example of a computer vision encoder paired with a recurrent language decoder.

Language representations

AWD-LSTM showed that optimization and regularization choices could substantially improve an LSTM language model on Penn Treebank and WikiText-2.^[15] ELMo subsequently formed contextual word representations from the internal states of a deep bidirectional language model and reported improvements on six natural-language-processing tasks when those representations were added to existing systems.^[25] Because its backward language model uses future context, ELMo is a representation model for complete text spans rather than a causal generator.

Structured and spatiotemporal data

Tree-LSTM applies gated memory updates over a known parse tree or other tree structure.^[16] ConvLSTM applies them to spatial tensors and was introduced for radar-based precipitation nowcasting.^[17] These are architectural adaptations to a supplied topology. They do not establish that LSTM is the best model for every tree, video, weather, or time series problem.

Comparison with transformers

An LSTM step depends on the preceding recurrent state, so a conventional LSTM cannot compute all time positions simultaneously. Its total arithmetic over a sequence grows linearly with sequence length, but the operations along time form a sequential path. Batching, stacked-layer parallelism, and fused kernels improve hardware use without removing that dependency.^[10][22]

Full self-attention compares every position with every other position, giving a quadratic number of pairwise scores in sequence length for a dense attention layer. Its positions can nevertheless be processed in parallel during training. Autoregressive transformer inference commonly stores key and value tensors for prior tokens, while an LSTM carries fixed-width hidden and cell states. These are baseline cost structures. Sparse, local, recurrent, linear-attention, and state-space variants alter them, as do projections, batching, and hardware-specific kernels.^[22]

The fixed-width recurrent state is both a resource advantage and an information bottleneck. It gives conventional LSTM bounded state memory per layer during streaming inference, but the model must compress relevant history into that state. A transformer can retain a separate cached representation for each attended prior token, at a memory cost that grows with context. Neither cost comparison alone determines model quality.^[10][22]

The Transformer paper demonstrated an encoder-decoder without recurrence and reported stronger results than the paper's recurrent and convolutional comparison systems on two machine-translation tasks.^[22] Which architecture is preferable remains empirical and should be evaluated against task-specific data, latency, memory, and hardware constraints.

Implementations

PyTorch provides torch.nn.LSTM for complete sequences and torch.nn.LSTMCell for a single step. The sequence module supports multiple layers, bidirectionality, dropout between layers, packed variable-length inputs, and optional hidden-state projection.^[10] The TensorFlow Keras API provides an LSTM layer and LSTMCell, with controls for sequence output, state return, stateful execution, dropout, and kernel selection.^[11]

Framework outputs should be interpreted from their documented shapes rather than by assuming that a "final state" is the last visible time-step tensor in every configuration. This matters especially for bidirectional models, packed sequences, projections, and stacked layers. Reproducible reporting should include the framework version, input layout, hidden and projection widths, number of layers, directionality, state initialization and reset policy, padding or packing method, dropout placement, and whether the output is taken from every step or only a final state.

Limitations

LSTM reduces a central optimization problem of simple recurrent networks, but several limitations remain:

Scroll sideways for more →

The 1997 results show that LSTM learned the authors' long-lag synthetic tasks.^[1] They are not a proof that any LSTM will retain arbitrary information for thousands of real-world steps. The search studies likewise show that details such as the forget gate and its initialization matter, but they do not identify a universally optimal recurrent cell.^[8][9] Evaluation on the intended data, sequence lengths, latency constraints, and deployment hardware is therefore necessary.

See also

• Recurrent Neural Network
• Vanishing Gradient Problem
• Backpropagation Through Time
• Sequence Model
• Transformer
• Attention Mechanism
• xLSTM

References

1. ^Hochreiter and Schmidhuber, "Long Short-Term Memory," Neural Computation 9(8), 1997
2. ^Bengio, Simard, and Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Transactions on Neural Networks 5(2), 1994
3. ^Pascanu, Mikolov, and Bengio, "On the difficulty of training recurrent neural networks," ICML 2013
4. ^Gers, Schmidhuber, and Cummins, "Learning to Forget: Continual Prediction with LSTM," Neural Computation 12(10), 2000
5. ^Gers, Schraudolph, and Schmidhuber, "Learning Precise Timing with LSTM Recurrent Networks," JMLR 3, 2002
6. ^Graves and Schmidhuber, "Framewise phoneme classification with bidirectional LSTM and other neural network architectures," Neural Networks 18(5-6), 2005
7. ^Graves et al., "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks," ICML 2006
8. ^Greff et al., "LSTM: A Search Space Odyssey," IEEE Transactions on Neural Networks and Learning Systems 28(10), 2017
9. ^Jozefowicz, Zaremba, and Sutskever, "An Empirical Exploration of Recurrent Network Architectures," ICML 2015
10. ^PyTorch documentation, "LSTM"
11. ^TensorFlow documentation, "tf.keras.layers.LSTM"
12. ^Cho et al., "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation," EMNLP 2014
13. ^Chung et al., "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling," 2014
14. ^Gal and Ghahramani, "A Theoretically Grounded Application of Dropout in Recurrent Neural Networks," NeurIPS 2016
15. ^Merity, Keskar, and Socher, "Regularizing and Optimizing LSTM Language Models," ICLR 2018
16. ^Tai, Socher, and Manning, "Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks," ACL 2015
17. ^Shi et al., "Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting," NeurIPS 2015
18. ^Beck et al., "xLSTM: Extended Long Short-Term Memory," NeurIPS 2024
19. ^Graves, Mohamed, and Hinton, "Speech Recognition with Deep Recurrent Neural Networks," ICASSP 2013
20. ^Sutskever, Vinyals, and Le, "Sequence to Sequence Learning with Neural Networks," NeurIPS 2014
21. ^Bahdanau, Cho, and Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate," ICLR 2015
22. ^Vaswani et al., "Attention Is All You Need," NeurIPS 2017
23. ^Graves, "Generating Sequences With Recurrent Neural Networks," 2013
24. ^Vinyals et al., "Show and Tell: A Neural Image Caption Generator," CVPR 2015
25. ^Peters et al., "Deep contextualized word representations," NAACL 2018