LSTM
Last reviewed
Jun 7, 2026
Sources
No citations yet
Review status
Needs citations
Revision
v7 · 7,885 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Jun 7, 2026
Sources
No citations yet
Review status
Needs citations
Revision
v7 · 7,885 words
Add missing citations, update stale details, or suggest a clearer explanation.
See also: Machine learning terms
Long Short-Term Memory (LSTM) is a type of recurrent neural network architecture designed to learn long-range dependencies in sequential data. It was introduced by Sepp sepp hochreiter and Jürgen jurgen schmidhuber in a 1997 paper in Neural Computation[1], and it became the dominant approach to sequence modeling from roughly 2013 to 2018, when it was largely supplanted in language tasks by the transformer[2]. LSTM cells maintain a separate cell state (a long-term memory) and use small learned gates to decide what to store, what to forget, and what to read out at each time step. That structure gives the network an additive, gradient-friendly path through time, which is the main reason LSTMs can train on sequences thousands of steps long without the gradients collapsing the way plain recurrent networks do.
Despite the rise of attention-based models, LSTMs remain in active use for speech recognition front-ends, on-device inference, time series, control policies, and any setting where strict left-to-right semantics and constant per-step compute matter more than raw throughput on a GPU. The architecture also enjoyed a small revival in 2024 with the publication of xLSTM[3], which scaled the original idea to billions of parameters and put it back in conversation with mamba and modern state space model approaches. In March 2025 NXAI released a 7-billion-parameter xLSTM-7B[4], the largest dense LSTM-derived language model trained to date.
The story of LSTM starts with a problem rather than a solution. In April 1991, Sepp Hochreiter, then a master's student at the Technical University of Munich, submitted his diploma thesis Untersuchungen zu dynamischen neuronalen Netzen ("Investigations into Dynamic Neural Networks") under Schmidhuber's advisorship[5]. The thesis contains the first detailed analysis of what is now called the vanishing gradient problem: when backpropagation is unrolled across many steps, the gradient of an early input with respect to a late output decays (or, less often, blows up) exponentially with the number of steps in between. Hochreiter showed this both empirically and analytically. The thesis was written in German and was not widely read at the time, which delayed the field's recognition of the problem by several years.
A standard recurrent neural network updates its hidden state with h_t = tanh(W x_t + U h_{t-1} + b). When trained by backpropagation through time, the gradient of the loss with respect to an early hidden state involves a long product of Jacobians of that recurrence. If those Jacobians have spectral radius below 1, the product shrinks toward zero (vanishing gradient); if above 1, it explodes. The practical consequence is that vanilla RNNs cannot reliably learn dependencies that span more than 10 to 20 steps. Bengio, Simard, and Frasconi gave the first widely cited analysis in IEEE Transactions on Neural Networks in 1994, proving that the same gradient-descent algorithm cannot simultaneously be efficient and capture long-term dependencies[6]. Pascanu, Mikolov, and Bengio later gave the modern treatment, distinguishing the vanishing case (which they attack with a soft regularizer) from the exploding case (for which they propose the now-standard gradient norm clipping)[7]. Gradient clipping fixes the explosion case but not the vanishing case; LSTM attacks the vanishing problem at the source by replacing the multiplicative chain with an additive cell-state path.
Hochreiter and Schmidhuber kept working on the issue identified in the 1991 thesis. Their first description of the new cell appeared in 1995 as TU Munich technical report FKI-207-95 titled "Long Short-Term Memory"[8], and the polished journal version was published two years later in Neural Computation 9(8):1735-1780[1]. The core idea was the constant error carousel: a self-connected linear unit, with weight 1.0 on its self-loop, that lets the gradient flow backward through time without being multiplied by anything that could shrink or explode it. To control what gets written to and read from this carousel, they added two multiplicative "gates": an input gate and an output gate. The 1997 paper proved that the architecture can bridge time lags in excess of 1000 discrete steps on synthetic benchmarks where conventional RNNs fail.
This original 1997 cell had no forget gate. Once a value was written into the cell state, it stayed there until the input gate wrote on top of it. That worked in benchmark tasks with clean sequence boundaries, but it caused problems on continuous streams where the cell state would drift and eventually saturate. Felix Gers, Jürgen Schmidhuber, and Fred Cummins fixed this in "Learning to Forget: Continual Prediction with LSTM" (Neural Computation 12(10):2451-2471, October 2000), adding a third gate that learns when to reset the cell[9]. An earlier conference version appeared in 1999, and the IDSIA technical report carries the number IDSIA-01-99, but the canonical reference is the 2000 journal paper. Almost every modern reference to "the LSTM" actually means this 2000 variant.
The same year, Gers and Schmidhuber added peephole connections in "Recurrent nets that time and count" at IJCNN 2000[10], with a longer JMLR follow-up by Gers, Schraudolph, and Schmidhuber in 2002 that demonstrated the peephole version could learn to count time steps in spike trains spaced 49 or 50 steps apart[11]. Peepholes let the gates inspect the cell state directly when deciding whether to open or close. From there the architecture branched in many directions: bidirectional LSTM (Graves & Schmidhuber, 2005[12]), Connectionist Temporal Classification training (Graves et al., 2006[13]), the simpler GRU (Cho et al., 2014[14]), ConvLSTM for spatiotemporal grids (Shi et al., 2015[15]), and many more. By the mid 2010s LSTM was the default sequence model in deep learning, powering Google Voice Search, Apple's Siri dictation, Google Translate, and most academic NLP papers. The 2017 publication of the Transformer in "Attention Is All You Need" started a fast migration toward attention-based models, but LSTM stayed in the toolbox and, with xLSTM (Beck et al., 2024[3]), came back into research focus.
| year | milestone | reference |
|---|---|---|
| 1991 | Hochreiter's diploma thesis identifies the vanishing gradient problem | TU Munich[5] |
| 1994 | Bengio, Simard, Frasconi prove gradient-based RNN training is fundamentally hard | IEEE TNN[6] |
| 1995 | FKI-207-95 technical report introduces the name "Long Short-Term Memory" | TU Munich[8] |
| 1997 | Original LSTM paper in Neural Computation, input + output gates, CEC | Hochreiter & Schmidhuber[1] |
| 2000 | Forget gate added, "Learning to Forget" published in Neural Computation | Gers, Schmidhuber, Cummins[9] |
| 2000 | Peephole connections introduced at IJCNN | Gers & Schmidhuber[10] |
| 2002 | Peephole LSTM learns precise timing with single-step accuracy | Gers, Schraudolph, Schmidhuber, JMLR[11] |
| 2005 | Bidirectional LSTM with full BPTT, applied to TIMIT phonemes | Graves & Schmidhuber[12] |
| 2006 | Connectionist Temporal Classification training for unsegmented sequences | Graves, Fernández, Gomez, Schmidhuber[13] |
| 2009 | LSTM with CTC wins ICDAR handwriting recognition competition | Graves et al. |
| 2013 | Deep bidirectional LSTM sets TIMIT phoneme record at 17.7% error | Graves, Mohamed, Hinton[16] |
| 2013 | "Generating Sequences with Recurrent Neural Networks" produces realistic cursive | Graves[17] |
| 2014 | Seq2seq with two stacked 4-layer LSTMs reaches 34.8 BLEU on WMT-14 En-Fr | Sutskever, Vinyals, Le[18] |
| 2014 | Distributed LSTM acoustic models scale to Google production speech | Sak, Senior, Beaufays[19] |
| 2014 | GRU introduced as a simpler gated alternative | Cho et al.[14] |
| 2015 | Forget-gate bias = 1 closes the LSTM/GRU gap | Jozefowicz, Zaremba, Sutskever[20] |
| 2015 | Show and Tell wins MSCOCO captioning challenge | Vinyals, Toshev, Bengio, Erhan[21] |
| 2015 | Google Voice Search deploys CTC + LSTM acoustic models in production | Google Research blog[22] |
| 2016 | GNMT replaces phrase-based Google Translate with 8+8 LSTM stack | Wu et al., arXiv:1609.08144[23] |
| 2017 | LSTM Search Space Odyssey: 5,400 runs, eight variants, no clear winner over the standard | Greff et al., IEEE TNNLS[24] |
| 2018 | ELMo bidirectional LSTM language model defines contextual embeddings | Peters et al., NAACL[25] |
| 2018 | OpenAI Five uses a single 4096-unit LSTM per agent to beat Dota 2 professionals | OpenAI[26] |
| 2019 | DeepMind's AlphaStar uses deep LSTM core to reach Grandmaster in StarCraft II | Vinyals et al., Nature[27] |
| 2019 | R2D2 stores recurrent hidden state in replay buffer, quadruples Atari SOTA | Kapturowski et al., ICLR[28] |
| 2024 | xLSTM: exponential gating, scalar (sLSTM) and matrix (mLSTM) variants scale to billions | Beck et al., NeurIPS[3] |
| 2025 | xLSTM-7B released by NXAI, fastest 7B recurrent LLM | Beck et al., arXiv:2503.13427[4] |
Jürgen Schmidhuber has, for more than a decade, publicly argued that he and his collaborators have not received adequate credit for several foundational deep learning contributions, including LSTM. His 2022 "Annotated History of Modern AI and Deep Learning" laid out the case in detail[29]. His critique has been pointed at Geoffrey Hinton, Yann LeCun, and Yoshua Bengio (recipients of the 2018 Turing Award) and at the broader public narrative around the deep learning revolution. The technical claims about the 1991 thesis and 1997 LSTM paper are uncontroversial; what is disputed is the relative weight given to recurrent versus feedforward work in the standard history. This article takes no position on the dispute and attributes each idea to its earliest verifiable publication.
Hochreiter went on to lead the Institute for Machine Learning at Johannes Kepler University Linz, and in 2021 he received the IEEE Computational Intelligence Society's Neural Networks Pioneer Award, in significant part for the LSTM work[30]. Schmidhuber received the same award in 2016. The 1997 Neural Computation paper has been cited tens of thousands of times across Google Scholar and Semantic Scholar.
A modern LSTM cell, as it appears in textbooks and in deep learning libraries, has six computations per time step. Let x_t be the input vector at step t, h_{t-1} the previous hidden state, and c_{t-1} the previous cell state. Let sigma denote the sigmoid function and tanh the hyperbolic tangent. The cell computes:
| step | name | equation | role |
|---|---|---|---|
| 1 | forget gate | f_t = sigma(W_f [h_{t-1}, x_t] + b_f) | how much of the old cell state to keep |
| 2 | input gate | i_t = sigma(W_i [h_{t-1}, x_t] + b_i) | how much of the new candidate to write |
| 3 | candidate | c_tilde_t = tanh(W_c [h_{t-1}, x_t] + b_c) | proposed new content |
| 4 | cell update | c_t = f_t * c_{t-1} + i_t * c_tilde_t | additive memory update |
| 5 | output gate | o_t = sigma(W_o [h_{t-1}, x_t] + b_o) | how much of the cell to expose |
| 6 | hidden output | h_t = o_t * tanh(c_t) | what the rest of the network sees |
The brackets [h_{t-1}, x_t] denote vector concatenation; * denotes elementwise multiplication. Each gate is a small linear layer followed by a sigmoid, so its output is a vector of values between 0 and 1 that act as soft binary masks over the cell-state coordinates. The candidate vector c_tilde_t lives in (-1, 1) thanks to the tanh, and the gates decide how much of it to add and how much of the previous c_{t-1} to keep. The hidden state h_t is a gated, squashed view of the cell state, and it is the only thing other layers (or the next time step) can see.
If the hidden size is H and the input size is X, the four weight matrices each have shape (H, H + X) and the four bias vectors have shape (H). Total parameter count per LSTM cell is 4H(H + X) + 4H, which is roughly four times the parameter count of a vanilla RNN of the same hidden size. With H = 1024 and X = 1024, a single layer holds about 8.4 million parameters. Multilayer LSTMs stack several cells, with the hidden state of layer L feeding into the input of layer L+1.
PyTorch implements an equivalent form with separate input and recurrent weight matrices, which is mostly a notational difference[31]. Its documentation writes the equations as i_t = sigma(W_ii x_t + b_ii + W_hi h_{t-1} + b_hi), and so on for f_t, g_t (the candidate), and o_t, with c_t = f_t * c_{t-1} + i_t * g_t and h_t = o_t * tanh(c_t). Two bias vectors per gate look redundant on paper, since b_ii + b_hi could be folded into a single bias, but keeping them separate matches the cuDNN kernels and avoids extra memory copies during training.
The partial derivative of c_t with respect to c_{t-1} is just f_t. When the forget gate is open (close to 1), the gradient flowing backward through that step is roughly the identity. Compose this across many steps and the product is the elementwise product of forget gates, which can stay near 1 if the network has learned to keep certain coordinates open. This is the modern version of the constant error carousel. The 1997 paper hard-coded the self-loop weight at 1.0; the 2000 paper let the network learn that weight per coordinate per time step. The CEC is the architectural mechanism that lets LSTMs route gradient information across thousands of steps; without it, the chain rule kills the signal long before it reaches the early inputs.
The original 1997 paper proved that an LSTM with appropriate gating could solve the "two-sequence problem" and the "embedded Reber grammar" with time lags greater than 1000 steps, where every alternative architecture they tested (BPTT, RTRL, recurrent cascade-correlation, Elman nets, neural sequence chunking) failed to learn at all[1].
A practical detail that matters a lot is the forget gate bias. With all biases zero, the forget gate sigmoid starts at 0.5, and the cell state decays by half per step on average. Jozefowicz, Zaremba, and Sutskever showed in 2015 that initializing the forget gate bias to 1 (gate sigmoid starts near 0.73) closes most of the gap between LSTM and GRU on a large empirical benchmark of over 10,000 architectures[20]. Many libraries now do this by default; some, including TensorFlow's LSTMCell, expose a forget_bias argument that defaults to 1.0.
The name itself is a precise description of the goal. Standard RNN hidden states are short-term memory: they decay quickly. LSTM keeps short-term memory in h_t and adds a long-term short-term memory in c_t, where "short-term" still refers to learnable activations (as opposed to long-term memory meaning the static synaptic weights). The hyphen in "Long Short-Term Memory" therefore parses as "long [short-term memory]", not "[long short]-term memory."
LSTM is less a single architecture than a family. Greff and colleagues published "LSTM: A Search Space Odyssey" in 2017, comparing eight variants across speech, handwriting, and music tasks with about 5,400 training runs (roughly 15 years of CPU time)[24]. Their conclusion was blunt: none of the standard variants meaningfully beats the modern (forget-gate, no peephole) LSTM on average, the forget gate and the output activation are the most important pieces, and the rest is mostly noise. Still, the family tree is worth knowing.
| variant | year | authors | distinguishing change |
|---|---|---|---|
| Vanilla LSTM | 1997 | Hochreiter & Schmidhuber | Constant error carousel, input and output gates, no forget gate |
| Forget-gate LSTM | 2000 | Gers, Schmidhuber, Cummins | Adds the forget gate, what most people now call "the LSTM" |
| Peephole LSTM | 2000, 2002 | Gers, Schmidhuber, Schraudolph | Gates can inspect c_{t-1} directly, helps with precise timing tasks |
| Bidirectional LSTM | 2005 | Graves & Schmidhuber | Two LSTMs run forward and backward over the same sequence, outputs concatenated |
| Tree LSTM | 2015 | Tai, Socher, Manning | Recurrence over a tree structure rather than a linear chain |
| ConvLSTM | 2015 | Shi et al. | Replaces matrix multiplies with convolutions, designed for video and weather radar |
| Highway LSTM | 2015 | Zhang et al. | Adds highway connections between stacked LSTM layers for deeper stacks[32] |
| Multiplicative LSTM | 2016 | Krause, Lu, Murray, Renals | Input-conditioned recurrence, used in OpenAI's sentiment neuron[33] |
| Quasi-RNN / SRU | 2016, 2017 | Bradbury et al.; Lei et al. | Parallelize across time by removing input-to-hidden recurrence[34] |
| Mogrifier LSTM | 2020 | Melis, Kocišký, Blunsom | Pre-mixes the input and previous hidden state several times before the LSTM update[35] |
| xLSTM (sLSTM, mLSTM) | 2024 | Beck et al. | Exponential gating, scalar and matrix memory, parallelizable variant for billions of parameters[3] |
Peephole connections, introduced by Gers and Schmidhuber at IJCNN 2000[10] and developed further in JMLR 2002[11], let each gate inspect the previous cell state c_{t-1} in addition to the hidden state h_{t-1}. The change is small (one extra term per gate) but matters for tasks that need precise timing, such as counting beats or distinguishing spike sequences spaced 49 versus 50 time steps apart. Peepholes are not the default in modern libraries because they did not improve average performance in the Search Space Odyssey benchmark[24].
Most real systems stack two or more LSTM layers and often run the bottom layer in both directions. Graves and Schmidhuber's 2005 paper on framewise phoneme classification with bidirectional LSTM[12] was the first to show that adding a backward pass meaningfully improves performance on labeled sequence tasks. It also introduced a full-gradient version of LSTM training (the 1997 paper used a truncated gradient). The trick cannot be used for autoregressive generation, since the backward pass would peek at future tokens, but it is the default for tagging, classification, and acoustic modeling. ELMo (Peters et al., 2018[25]) uses a stack of bidirectional LSTMs trained as language models in both directions, with predictions tied at the top.
Tree LSTM (Tai, Socher, Manning, ACL 2015[36]) generalises the chain-structured recurrence to arbitrary tree topologies, with each cell receiving inputs from its child nodes. The two main variants are the Child-Sum Tree LSTM and the N-ary Tree LSTM. Tree LSTMs improved on chain LSTMs for semantic relatedness on SemEval 2014 Task 1 and for sentiment classification on the Stanford Sentiment Treebank[36]. The architecture has been largely displaced by attention-based parsers in production NLP.
The Gated Recurrent Unit, introduced by Cho and colleagues in EMNLP 2014[14], is the most widely used LSTM relative. It collapses the input and forget gates into a single update gate z_t and adds a reset gate r_t controlling how much of the previous hidden state contributes to the candidate. There is no separate cell state. The update is h_t = (1 - z_t) * h_{t-1} + z_t * tanh(W [r_t * h_{t-1}, x_t]). GRU has roughly 25 percent fewer parameters than LSTM at the same hidden size and trains slightly faster; the two are usually within a percentage point on most tasks. See gru for a longer treatment.
| dimension | LSTM | GRU |
|---|---|---|
| gates | input, forget, output | update, reset |
| separate cell state | yes (c_t) | no |
| parameters per cell | 4H(H + X) + 4H | 3H(H + X) + 3H |
| output range | o_t * tanh(c_t) | linear interpolation |
| introduced | 1997, forget gate 2000 | 2014 |
ConvLSTM (Shi et al., NeurIPS 2015[15]) replaces the matrix multiplications inside the gates with convolutions. The cell state and hidden state become 3D tensors (channels by height by width) instead of vectors. This is the natural fit for any task where each time step is itself a spatial grid: weather radar, video, or fluid simulation. The original paper used it for short-term rainfall prediction in collaboration with the Hong Kong Observatory and beat the operational ROVER algorithm on the HKO-7 nowcasting dataset.
The xLSTM paper (Beck et al., 2024[3]) is the most ambitious LSTM revival in two decades. It introduces two new cell types. The sLSTM keeps a scalar cell state per dimension, like the classical LSTM, but replaces the sigmoid input and forget gates with exponential gating combined with a normalizer state for stability. This restores expressive power without sacrificing constant memory at inference. The mLSTM replaces the scalar cell state with a matrix C_t of shape (H, H) and uses a covariance update rule C_t = f_t C_{t-1} + i_t v_t k_t^T, where k_t, v_t, and q_t play the same role as keys, values, and queries in attention. The mLSTM recurrence has no hidden-to-hidden mixing, which makes it fully parallelizable during training via a chunkwise scan, like Mamba and linear attention.
Stacked into residual blocks alternating sLSTM and mLSTM, xLSTM-1.3B competes with Llama-1.3B and Mamba-1.3B at the same training budget. The NXAI release of xLSTM-7B in March 2025 trained 2.3 trillion tokens and reaches competitive performance on standard LLM benchmarks while running inference at constant memory and producing higher token throughput at long sequence lengths than equivalent Transformer baselines[4].
Training an LSTM looks like training any other recurrent network: the sequence is unrolled in time, the loss is summed across supervised steps, and gradients flow back through backpropagation through time. Three details deserve attention.
Gradient clipping. Even with the additive cell-state path, the gates and input-to-hidden weights can produce large gradients on rare events. Clipping the global gradient norm to 1 or 5 is standard; Pascanu, Mikolov, and Bengio (2013) gave the now-standard analysis[7].
Truncated BPTT. Full BPTT over very long sequences is expensive. Truncated BPTT processes the sequence in chunks of, say, 100 to 200 steps, carrying the cell and hidden state forward but only backpropagating gradients within the current chunk. This is what .detach() on the hidden state between batches achieves in PyTorch.
Dropout. Naive dropout on the recurrent connections destroys the long-term memory because the mask changes at every step. Variational dropout (Gal & Ghahramani, 2016[37]) fixes a single mask per sequence; zoneout (Krueger et al., 2017[38]) randomly copies the previous hidden state instead of dropping units to zero. Most production code applies dropout only between stacked LSTM layers, which is what dropout controls in torch.nn.LSTM.
Initialization. Beyond the forget-gate bias trick, orthogonal initialization of the recurrent weight matrix and Glorot/Xavier initialization of the input weight matrix are common defaults. The "identity-RNN" initialization, where the recurrent matrix is the identity and the input weights are scaled by 1/H, is sometimes used to encourage long-memory behavior at the start of training.
Optimizer choice. Adam and RMSProp are the usual choices, with learning rates in the range 0.001 to 0.01. Plain SGD with momentum is harder to tune but sometimes yields the best final perplexity on language modeling. Cyclical learning rate schedules and warmup helped early Transformer NMT systems and have been ported back to LSTM training.
LSTMs powered most of the practical sequence learning systems in the 2010s. A small selection:
| year | system | role of LSTM | reference |
|---|---|---|---|
| 2009 | ICDAR handwriting recognition winner | First RNN to win an international competition, LSTM + CTC | Graves et al. |
| 2013 | Deep RNN for TIMIT phoneme recognition | Deep bidirectional LSTM with CTC achieves 17.7% error on TIMIT | Graves, Mohamed, Hinton, ICASSP 2013[16] |
| 2013 | Generating Sequences with RNNs | Single LSTM stack synthesizes realistic cursive handwriting and Wikipedia text | Graves, arXiv:1308.0850[17] |
| 2014 | Sequence to Sequence Learning | Two stacked 4-layer LSTMs as encoder and decoder, 34.8 BLEU on WMT-14 English-French | Sutskever, Vinyals, Le, NeurIPS 2014[18] |
| 2014 | Sak, Senior, Beaufays acoustic model | Distributed LSTM acoustic models for speech recognition at Google | Sak et al., Interspeech 2014[19] |
| 2015 | Show and Tell image captioning | CNN encoder feeding an LSTM caption decoder, won MSCOCO challenge | Vinyals, Toshev, Bengio, Erhan, CVPR 2015[21] |
| 2015 | Google Voice Search | Production deployment of CTC + LSTM acoustic models on iOS and Android | Google Research blog, 2015-09-24[22] |
| 2015 | Karpathy's char-rnn | Popular blog post showing LSTMs generating Shakespeare and Linux source code | "The Unreasonable Effectiveness of Recurrent Neural Networks"[39] |
| 2016 | Google Neural Machine Translation | 8-layer LSTM encoder, 8-layer LSTM decoder, attention, wordpiece tokens, replaced phrase-based MT | Wu et al., arXiv:1609.08144[23] |
| 2017 | OpenAI Sentiment Neuron | 4096-unit multiplicative LSTM trained on Amazon reviews discovers sentiment as a single feature | Radford, Jozefowicz, Sutskever[33] |
| 2018 | ELMo contextual embeddings | Bidirectional LSTM language model whose hidden states are used as word features | Peters et al., NAACL 2018[25] |
| 2018 | OpenAI Five (Dota 2) | Single 4096-unit LSTM per agent (84% of model parameters), 159M total, trained with PPO | OpenAI, arXiv:1912.06680[26] |
| 2018 | IMPALA distributed RL | Actor-learner architecture with LSTM cores, multi-task DMLab-30 and Atari-57 | Espeholt et al., ICML 2018[40] |
| 2019 | DeepMind AlphaStar (StarCraft II) | Deep LSTM core inside a Transformer-LSTM hybrid policy, Grandmaster level | Vinyals et al., Nature 575[27] |
| 2019 | R2D2 distributed Q-learning | Stores LSTM hidden state in replay buffer, quadruples Atari SOTA | Kapturowski et al., ICLR 2019[28] |
Speech recognition was arguably the first commercially important domain where LSTM beat alternative approaches at scale. Graves, Mohamed, and Hinton's 2013 paper used deep bidirectional LSTM stacks trained with Connectionist Temporal Classification to set a TIMIT phoneme recognition record at 17.7% error, beating the previous best of 20.7%[16]. Sak, Senior, and Beaufays then showed at Interspeech 2014 that LSTM acoustic models could be trained on Google-scale distributed infrastructure and matched or beat existing hybrid DNN-HMM systems[19]. Their distributed training framework split the model across multiple machines, used asynchronous SGD across hundreds of replicas, and trained on tens of thousands of hours of transcribed speech.
Within a year these models were running in Google Voice Search. The Google AI blog post from September 24, 2015, described how the team built better neural network acoustic models using CTC and sequence-discriminative training, replacing the older DNN-HMM hybrid front-end[22]. Similar architectures soon powered Apple Siri dictation and Amazon's Alexa speech front-ends. The Microsoft Switchboard system reported in late 2016 used LSTMs in part of its acoustic model when it reached human parity on the corpus.
Sutskever, Vinyals, and Le's 2014 paper[18] introduced the seq2seq framework: an encoder LSTM compresses an input sequence into a fixed vector and a decoder LSTM expands that vector into the output. The headline trick was reversing the source sentence, which shortens the dependency between the first source word and the first target word and improved BLEU from 30.6 to 34.8 on WMT-14 English-French. The paper used a 4-layer LSTM with 1000 cells per layer.
Bahdanau, Cho, and Bengio added content-based attention in 2015[41], removing the fixed-vector bottleneck and immediately improving translation quality especially on long sentences. With wordpiece encoding and 8+8 LSTM layers, Google Neural Machine Translation (Wu et al., 2016[23]) reduced translation errors by 55 to 85 percent compared to the older phrase-based system, replaced production Google Translate, and was processing more than 18 million sentences daily on the Chinese-to-English pair within months of launch. LSTM-based seq2seq became the engine of production neural machine translation until the Transformer replaced it.
Vinyals, Toshev, Bengio, and Erhan's "Show and Tell" model[21] glued a pretrained convolutional image encoder to an LSTM language decoder, conditioning the LSTM on the visual features at the first step. The system won the 2015 MSCOCO captioning challenge and established the encoder-decoder framework that dominated multimodal generation for several years. The follow-up "Show, Attend, and Tell" (Xu et al., 2015) added attention over CNN feature maps and is widely taught as the first multimodal use of attention.
LSTM cores are standard in deep reinforcement learning whenever the environment is partially observed, which covers most non-trivial tasks. DRQN (Hausknecht & Stone, 2015) extended DQN with an LSTM and beat the original frame-stacking baseline on flickering Atari. OpenAI Five is the most famous LSTM-based RL system: each of the five Dota 2 hero agents is a single 4096-unit LSTM that observes the game state from the API, with the LSTM accounting for 84 percent of the model's 159 million parameters[26]. After 10 months of training on 770 PFlop-day-equivalent compute, OpenAI Five beat the world champion team OG 2-0 at The International 2019. IMPALA (Espeholt et al., ICML 2018[40]) introduced the actor-learner pattern that became standard for scalable RL, with an LSTM core for sequence memory. R2D2 (Kapturowski et al., ICLR 2019[28]) added the missing ingredient of storing recurrent states in the replay buffer and quadrupled the previous state of the art on Atari-57. AlphaStar used a deep LSTM as the backbone of its StarCraft II policy and value networks, reaching Grandmaster level on Battle.net in 2019[27].
Andrej Karpathy's 2015 blog post "The Unreasonable Effectiveness of Recurrent Neural Networks" introduced an entire generation of practitioners to LSTM by showing it generate plausible Shakespeare, Linux kernel source code, and LaTeX algebra from character-level training[39]. LSTM language models held state of the art on Penn Treebank and WikiText benchmarks until Transformer language models took over. ELMo (Peters et al., NAACL 2018[25]) was the last big LSTM-based pretrained model before BERT. Its bidirectional LSTM was trained as a language model in each direction on roughly 1 billion words of text, then the hidden states from all layers were combined as contextual features for downstream tasks.
In time-series forecasting LSTM remains a common baseline alongside Prophet, classical ARIMA, and modern Transformer variants. In computational biology LSTM has been used for protein structure features, secondary structure prediction, and DNA sequence analysis, although Transformers (and now Mamba-style models) have largely taken over there. In robotics, LSTMs are still the default policy core for partially observed control; in finance, they appear in market-microstructure modeling. The architecture's flexibility, modest data requirements, and constant per-step compute keep it competitive in any setting where data is small, sequences are long, or hardware budgets are tight.
The two big problems with LSTM are both about scale.
First, the recurrence is inherently sequential. The cell state at step t depends on the cell state at step t-1, so you cannot parallelize the forward pass over time the way you can for self-attention. On modern GPUs a model that uses 5 percent of the silicon for 100 percent of the time is slower than one that uses 80 percent some of the time, even if the second does more arithmetic. cuDNN's fused LSTM kernel and tricks like quasi-RNN[34] or simple recurrent unit (SRU) recover some of this throughput, but not all of it. The fundamental obstacle is the hidden-to-hidden multiplication, which prevents prefix-scan style parallelization.
Second, LSTMs are hard to scale to extremely long contexts. Even with a forget gate close to 1, the cell state is a fixed-size vector; you cannot cram a million tokens of context into a few thousand floats without losing information. Transformers handle this by keeping a key-value cache that grows with the input, paying quadratic compute for the privilege. State space models and xLSTM-mLSTM aim at the middle ground by using a structured but larger fixed-size memory (a matrix instead of a vector in the mLSTM case).
LSTM hyperparameters (learning rate, gradient clipping, hidden size, dropout) are also tightly coupled, and the same recipe rarely transfers across tasks. The Search Space Odyssey paper concluded that these hyperparameters are mostly independent and gave practical guidelines, but practitioners still report that getting an LSTM to train well takes more babysitting than a Transformer, mostly because Transformer pretraining recipes are now extremely well-documented and LSTM ones are not[24].
Other documented weaknesses include difficulty modeling phenomena that need explicit copy operations across many steps (a problem attention solves by construction), sensitivity to long-tailed distributions, and the "saturation" failure mode where forget gates collapse to 1 or 0 and stop learning. The mogrifier LSTM and several other variants explicitly try to address some of these issues without abandoning the recurrent structure[35].
The Transformer, introduced by Vaswani and colleagues in "Attention Is All You Need" (NeurIPS 2017[2]), removed recurrence entirely and replaced it with self-attention over the whole sequence at once. This was good news on a GPU because every position can be processed in parallel, and bad news on the memory bill because the attention matrix is quadratic in sequence length. For most NLP tasks the trade-off favored Transformers, and from roughly 2018 onward the field shifted accordingly. ELMo (2018) was the last famous LSTM-based pretrained language model; BERT, GPT, and everything that followed used self-attention.
That said, LSTMs and Transformers have different inductive biases and different cost structures, and the choice is not as one-sided as the trend suggests.
| property | LSTM | Transformer |
|---|---|---|
| time per training step | O(T) sequential | O(T^2) parallel |
| time per token at inference | O(1), constant memory | O(T) attention over the cache, O(T) memory |
| state | fixed-size hidden + cell | growing key-value cache |
| parallelism over sequence | poor | excellent |
| typical context length in practice | thousands of steps with care | millions of tokens with hardware tricks |
| inductive bias | strong recency, sequential | none, position must be encoded |
| works well with very small data | yes | usually needs more |
| dominant in speech front-ends (2026) | still common | catching up |
The practical upshot: Transformers win when you have lots of data, lots of compute, and care about throughput. LSTMs win, or at least compete, when you care about constant memory at inference (streaming speech, on-device, embedded), when sequences are very long but mostly recent context matters, when data is small, or when you want a strong online learning algorithm. Speech recognition front-ends, on-device keyboard prediction, real-time translation pipelines, and a long tail of tabular and time-series models still rely on LSTM cells.
After several years where new sequence architectures meant some variant of attention, the late 2023 and 2024 wave of papers brought recurrence back. The headline names are Mamba, RWKV, and xLSTM.
Mamba (Gu and Dao, arXiv:2312.00752, December 2023[42]) is a state space model, not an LSTM, but it inherits the recurrent flavor: linear time in sequence length, constant memory at inference, and, in the selective version, input-dependent gating that closely mirrors what an LSTM does with its forget gate. Mamba-3B was the first sub-quadratic model to clearly match a Transformer of the same size on language modeling, and on long sequences it is significantly faster. Mamba builds on Gu's earlier S4 family of structured state space models and the "selective scan" idea that makes the recurrence data-dependent.
RWKV (Peng et al., Findings of EMNLP 2023[43]) reformulates attention so that it can be computed as a recurrence, giving it Transformer-like training and RNN-like inference. RWKV models have been trained up to 14 billion parameters, the largest dense RNN trained at the time of publication. The architecture is open source under Apache 2.0, with checkpoints from 169M to 14B parameters distributed on Hugging Face.
xLSTM (Beck, Pöppel, Spanring, Auer, Prudnikova, Kopp, Klambauer, Brandstetter, and Hochreiter, arXiv:2405.04517, May 2024[3]) is the closest direct revival of the original idea. The paper introduces two new cell types: sLSTM, which keeps the scalar memory of the original LSTM but adds exponential gating with stabilization, and mLSTM, which replaces the scalar cell state with a matrix and uses a covariance update rule that can be parallelized like attention. Stacked into residual blocks, xLSTM models compete with Llama-style Transformers and Mamba at billion-parameter scale.
NXAI, the company Hochreiter co-founded with Netural X and PIERER Digital Holding in early 2024, has continued to push xLSTM as a basis for European foundation models. The March 2025 release of xLSTM-7B[4] established the architecture as the fastest 7-billion-parameter recurrent language model and the largest LSTM-style network trained to date, with the JAX implementation open-sourced on GitHub at NX-AI/xlstm-jax. NXAI markets the model on its inference efficiency: linear time and constant memory let xLSTM-7B run on smaller hardware than Llama-7B or Mistral-7B at long context lengths, with the company reporting roughly 40 percent lower inference energy on comparable workloads.
Mamba, RWKV, xLSTM, and their relatives (Griffin, Hawk, Retentive Networks, GLA, RetNet) are sometimes called the linear RNN renaissance or post-attention architectures. They share a few common moves: keep a fixed or slowly growing recurrent state, use input-dependent gating to compensate for lost expressive power of attention, and arrange the math so the recurrence can be evaluated as a parallel prefix scan during training. They have already won the long-context, on-device, and streaming markets where Transformers struggle, and have begun to make inroads into general LLM training at the 1 to 10 billion parameter scale.
Every major deep learning framework ships an LSTM implementation backed by a fast GPU kernel. The interfaces are similar enough to swap with a one-line change.
| framework | module | notes |
|---|---|---|
| PyTorch | torch.nn.LSTM, torch.nn.LSTMCell | cuDNN backend, supports stacked, bidirectional, dropout between layers |
| TensorFlow / Keras | tf.keras.layers.LSTM, LSTMCell | XLA and cuDNN paths, the layer auto-selects the fast kernel when conditions are met |
| JAX | flax.linen.OptimizedLSTMCell, haiku.LSTM | functional API, scan over time |
| MXNet | mx.gluon.rnn.LSTM | similar API to PyTorch |
| ONNX | LSTM operator | for model export and cross-framework deployment |
| Apple Core ML | LSTMLayer | converts from PyTorch and TensorFlow for on-device inference |
| TensorFlow Lite | UnidirectionalSequenceLSTM, BidirectionalSequenceLSTM | quantized ops for mobile inference |
import torch
import torch.nn as nn
lstm = nn.LSTM(
input_size=64,
hidden_size=128,
num_layers=2,
batch_first=True,
bidirectional=False,
dropout=0.2,
)
x = torch.randn(32, 100, 64) # (batch, time, features)
outputs, (h_n, c_n) = lstm(x)
# outputs: (32, 100, 128) - hidden state at every step
# h_n, c_n: (2, 32, 128) - final hidden and cell states for each layer
cuDNN's LSTM kernel fuses the four gate matrix multiplies into a single GEMM and applies the elementwise updates in a custom kernel, with sequence parallelism over batch and depth but not time. The kernel handles arbitrary sequence length up to GPU memory limits and supports packed sequences for batches of variable-length inputs. PyTorch's pack_padded_sequence and pad_packed_sequence utilities are the standard way to handle variable-length batches efficiently.
For about five years, from roughly 2013 through 2018, LSTM was synonymous with sequence modeling in deep learning. The original 1997 paper has been cited well over 100,000 times. Andrej Karpathy's 2015 blog post "The Unreasonable Effectiveness of Recurrent Neural Networks"[39] introduced an entire generation of practitioners to LSTM by showing it generating plausible Shakespeare and compilable C code from a character-level model. Christopher Olah's 2015 blog post "Understanding LSTM Networks"[44] remains, more than a decade later, the diagram people reach for when explaining the gates. The cell diagram from that post, with its color-coded gate operations, is reproduced in nearly every introductory deep-learning textbook and lecture slide deck.
The Transformer mostly displaced LSTM in research after 2018, but in production the transition has been slower. With xLSTM, Mamba, and RWKV bringing recurrence back to the frontier, the architecture's second act is still being written.