LSTM

See also: Machine learning terms

Long Short-Term Memory (LSTM) is a recurrent neural network architecture that learns long-range dependencies in sequential data by maintaining a separate, additively updated cell state and using small learned gates to decide what to store, forget, and read out at each time step. It was introduced by Sepp Hochreiter and Jürgen Schmidhuber in a 1997 paper in Neural Computation^[1], and it became the dominant approach to sequence modeling from roughly 2013 to 2018, when it was largely supplanted in language tasks by the transformer^[2]. The gating structure gives the network an additive, gradient-friendly path through time, the constant error carousel, which is the main reason LSTMs can train on sequences thousands of steps long without the gradients collapsing the way plain recurrent networks do. The 1997 paper showed LSTM can "bridge minimal time lags in excess of 1000 discrete-time steps" while remaining "local in space and time" with computational complexity per time step and weight of O(1)^[1]. That paper is now among the most-cited works in artificial intelligence, with roughly 98,800 citations indexed on Semantic Scholar as of 2026^[1].

Despite the rise of attention-based models, LSTMs remain in active use for speech recognition front-ends, on-device inference, time series, control policies, and any setting where strict left-to-right semantics and constant per-step compute matter more than raw throughput on a GPU. The architecture also enjoyed a small revival in 2024 with the publication of xLSTM^[3], which scaled the original idea to billions of parameters and put it back in conversation with mamba and modern state space model approaches. In March 2025 NXAI released a 7-billion-parameter xLSTM-7B^[4], the largest dense LSTM-derived language model trained to date, trained on 2.3 trillion tokens using 128 NVIDIA H100 GPUs^[4].

What problem does LSTM solve?

The story of LSTM starts with a problem rather than a solution. In April 1991, Sepp Hochreiter, then a master's student at the Technical University of Munich, submitted his diploma thesis Untersuchungen zu dynamischen neuronalen Netzen ("Investigations into Dynamic Neural Networks") under Schmidhuber's advisorship^[5]. The thesis contains the first detailed analysis of what is now called the vanishing gradient problem: when backpropagation is unrolled across many steps, the gradient of an early input with respect to a late output decays (or, less often, blows up) exponentially with the number of steps in between. Hochreiter showed this both empirically and analytically. The thesis was written in German and was not widely read at the time, which delayed the field's recognition of the problem by several years.

A standard recurrent neural network updates its hidden state with h_t = tanh(W x_t + U h_{t-1} + b). When trained by backpropagation through time, the gradient of the loss with respect to an early hidden state involves a long product of Jacobians of that recurrence. If those Jacobians have spectral radius below 1, the product shrinks toward zero (vanishing gradient); if above 1, it explodes. The practical consequence is that vanilla RNNs cannot reliably learn dependencies that span more than 10 to 20 steps. Bengio, Simard, and Frasconi gave the first widely cited analysis in IEEE Transactions on Neural Networks in 1994, proving that the same gradient-descent algorithm cannot simultaneously be efficient and capture long-term dependencies^[6]. Pascanu, Mikolov, and Bengio later gave the modern treatment, distinguishing the vanishing case (which they attack with a soft regularizer) from the exploding case (for which they propose the now-standard gradient norm clipping)^[7]. Gradient clipping fixes the explosion case but not the vanishing case; LSTM attacks the vanishing problem at the source by replacing the multiplicative chain with an additive cell-state path.

Authors and history

Hochreiter and Schmidhuber kept working on the issue identified in the 1991 thesis. Their first description of the new cell appeared in 1995 as TU Munich technical report FKI-207-95 titled "Long Short-Term Memory"^[8], and the polished journal version was published two years later in Neural Computation 9(8):1735-1780^[1]. The core idea was the constant error carousel: a self-connected linear unit, with weight 1.0 on its self-loop, that lets the gradient flow backward through time without being multiplied by anything that could shrink or explode it. To control what gets written to and read from this carousel, they added two multiplicative "gates": an input gate and an output gate. The 1997 paper proved that the architecture can bridge time lags in excess of 1000 discrete steps on synthetic benchmarks where conventional RNNs fail.

This original 1997 cell had no forget gate. Once a value was written into the cell state, it stayed there until the input gate wrote on top of it. That worked in benchmark tasks with clean sequence boundaries, but it caused problems on continuous streams where the cell state would drift and eventually saturate. Felix Gers, Jürgen Schmidhuber, and Fred Cummins fixed this in "Learning to Forget: Continual Prediction with LSTM" (Neural Computation 12(10):2451-2471, October 2000), adding a third gate that learns when to reset the cell^[9]. An earlier conference version appeared in 1999, and the IDSIA technical report carries the number IDSIA-01-99, but the canonical reference is the 2000 journal paper. Almost every modern reference to "the LSTM" actually means this 2000 variant.

The same year, Gers and Schmidhuber added peephole connections in "Recurrent nets that time and count" at IJCNN 2000^[10], with a longer JMLR follow-up by Gers, Schraudolph, and Schmidhuber in 2002 that demonstrated the peephole version could learn to count time steps in spike trains spaced 49 or 50 steps apart^[11]. Peepholes let the gates inspect the cell state directly when deciding whether to open or close. From there the architecture branched in many directions: bidirectional LSTM (Graves & Schmidhuber, 2005^[12]), Connectionist Temporal Classification training (Graves et al., 2006^[13]), the simpler GRU (Cho et al., 2014^[14]), ConvLSTM for spatiotemporal grids (Shi et al., 2015^[15]), and many more. By the mid 2010s LSTM was the default sequence model in deep learning, powering Google Voice Search, Apple's Siri dictation, Google Translate, and most academic NLP papers. The 2017 publication of the Transformer in "Attention Is All You Need" started a fast migration toward attention-based models, but LSTM stayed in the toolbox and, with xLSTM (Beck et al., 2024^[3]), came back into research focus.

Timeline of milestones

Scroll sideways for more →

Priority disputes

Jürgen Schmidhuber has, for more than a decade, publicly argued that he and his collaborators have not received adequate credit for several foundational deep learning contributions, including LSTM. His 2022 "Annotated History of Modern AI and Deep Learning" laid out the case in detail^[29]. His critique has been pointed at Geoffrey Hinton, Yann LeCun, and Yoshua Bengio (recipients of the 2018 Turing Award) and at the broader public narrative around the deep learning revolution. The technical claims about the 1991 thesis and 1997 LSTM paper are uncontroversial; what is disputed is the relative weight given to recurrent versus feedforward work in the standard history. This article takes no position on the dispute and attributes each idea to its earliest verifiable publication.

Hochreiter went on to lead the Institute for Machine Learning at Johannes Kepler University Linz, and in 2021 he received the IEEE Computational Intelligence Society's Neural Networks Pioneer Award, in significant part for the LSTM work^[30]. Schmidhuber received the same award in 2016. The 1997 Neural Computation paper has been cited tens of thousands of times across Google Scholar and Semantic Scholar, where it carries roughly 98,800 citations as of 2026^[1].

How does an LSTM cell work?

A modern LSTM cell, as it appears in textbooks and in deep learning libraries, has six computations per time step. Let x_t be the input vector at step t, h_{t-1} the previous hidden state, and c_{t-1} the previous cell state. Let sigma denote the sigmoid function and tanh the hyperbolic tangent. The cell computes:

Scroll sideways for more →

The brackets [h_{t-1}, x_t] denote vector concatenation; * denotes elementwise multiplication. Each gate is a small linear layer followed by a sigmoid, so its output is a vector of values between 0 and 1 that act as soft binary masks over the cell-state coordinates. The candidate vector c_tilde_t lives in (-1, 1) thanks to the tanh, and the gates decide how much of it to add and how much of the previous c_{t-1} to keep. The hidden state h_t is a gated, squashed view of the cell state, and it is the only thing other layers (or the next time step) can see.

If the hidden size is H and the input size is X, the four weight matrices each have shape (H, H + X) and the four bias vectors have shape (H). Total parameter count per LSTM cell is 4H(H + X) + 4H, which is roughly four times the parameter count of a vanilla RNN of the same hidden size. With H = 1024 and X = 1024, a single layer holds about 8.4 million parameters. Multilayer LSTMs stack several cells, with the hidden state of layer L feeding into the input of layer L+1.

PyTorch implements an equivalent form with separate input and recurrent weight matrices, which is mostly a notational difference^[31]. Its documentation writes the equations as i_t = sigma(W_ii x_t + b_ii + W_hi h_{t-1} + b_hi), and so on for f_t, g_t (the candidate), and o_t, with c_t = f_t * c_{t-1} + i_t * g_t and h_t = o_t * tanh(c_t). Two bias vectors per gate look redundant on paper, since b_ii + b_hi could be folded into a single bias, but keeping them separate matches the cuDNN kernels and avoids extra memory copies during training.

Cell state and the constant error carousel

The partial derivative of c_t with respect to c_{t-1} is just f_t. When the forget gate is open (close to 1), the gradient flowing backward through that step is roughly the identity. Compose this across many steps and the product is the elementwise product of forget gates, which can stay near 1 if the network has learned to keep certain coordinates open. This is the modern version of the constant error carousel. The 1997 paper hard-coded the self-loop weight at 1.0; the 2000 paper let the network learn that weight per coordinate per time step. The CEC is the architectural mechanism that lets LSTMs route gradient information across thousands of steps; without it, the chain rule kills the signal long before it reaches the early inputs.

The original 1997 paper proved that an LSTM with appropriate gating could solve the "two-sequence problem" and the "embedded Reber grammar" with time lags greater than 1000 steps, where every alternative architecture they tested (BPTT, RTRL, recurrent cascade-correlation, Elman nets, neural sequence chunking) failed to learn at all^[1]. The authors summarized the design goal in the abstract: LSTM "is local in space and time; its computational complexity per time step and weight is O(1)"^[1], meaning the cost of one update does not grow with sequence length, a property that still distinguishes recurrent models from attention at inference time.

Bias initialization

A practical detail that matters a lot is the forget gate bias. With all biases zero, the forget gate sigmoid starts at 0.5, and the cell state decays by half per step on average. Jozefowicz, Zaremba, and Sutskever showed in 2015 that initializing the forget gate bias to 1 (gate sigmoid starts near 0.73) closes most of the gap between LSTM and GRU on a large empirical benchmark of over 10,000 architectures^[20]. Many libraries now do this by default; some, including TensorFlow's LSTMCell, expose a forget_bias argument that defaults to 1.0.

Why "long short-term"

The name itself is a precise description of the goal. Standard RNN hidden states are short-term memory: they decay quickly. LSTM keeps short-term memory in h_t and adds a long-term short-term memory in c_t, where "short-term" still refers to learnable activations (as opposed to long-term memory meaning the static synaptic weights). The hyphen in "Long Short-Term Memory" therefore parses as "long [short-term memory]", not "[long short]-term memory."

What are the main LSTM variants?

LSTM is less a single architecture than a family. Greff and colleagues published "LSTM: A Search Space Odyssey" in 2017, comparing eight variants across speech, handwriting, and music tasks with 5,400 training runs (roughly 15 years of CPU time)^[24]. Their conclusion was blunt: none of the standard variants meaningfully beats the modern (forget-gate, no peephole) LSTM on average, and they found "the forget gate and the output activation function to be its most critical components," with the rest mostly noise^[24]. Still, the family tree is worth knowing.

Scroll sideways for more →

Peephole LSTM

Peephole connections, introduced by Gers and Schmidhuber at IJCNN 2000^[10] and developed further in JMLR 2002^[11], let each gate inspect the previous cell state c_{t-1} in addition to the hidden state h_{t-1}. The change is small (one extra term per gate) but matters for tasks that need precise timing, such as counting beats or distinguishing spike sequences spaced 49 versus 50 time steps apart. Peepholes are not the default in modern libraries because they did not improve average performance in the Search Space Odyssey benchmark^[24].

Bidirectional LSTM

Most real systems stack two or more LSTM layers and often run the bottom layer in both directions. Graves and Schmidhuber's 2005 paper on framewise phoneme classification with bidirectional LSTM^[12] was the first to show that adding a backward pass meaningfully improves performance on labeled sequence tasks. It also introduced a full-gradient version of LSTM training (the 1997 paper used a truncated gradient). The trick cannot be used for autoregressive generation, since the backward pass would peek at future tokens, but it is the default for tagging, classification, and acoustic modeling. ELMo (Peters et al., 2018^[25]) uses a stack of bidirectional LSTMs trained as language models in both directions, with predictions tied at the top.

Tree LSTM

Tree LSTM (Tai, Socher, Manning, ACL 2015^[36]) generalises the chain-structured recurrence to arbitrary tree topologies, with each cell receiving inputs from its child nodes. The two main variants are the Child-Sum Tree LSTM and the N-ary Tree LSTM. Tree LSTMs improved on chain LSTMs for semantic relatedness on SemEval 2014 Task 1 and for sentiment classification on the Stanford Sentiment Treebank^[36]. The architecture has been largely displaced by attention-based parsers in production NLP.

How does LSTM differ from GRU?

The Gated Recurrent Unit, introduced by Cho and colleagues in EMNLP 2014^[14], is the most widely used LSTM relative. It collapses the input and forget gates into a single update gate z_t and adds a reset gate r_t controlling how much of the previous hidden state contributes to the candidate. There is no separate cell state. The update is h_t = (1 - z_t) * h_{t-1} + z_t * tanh(W [r_t * h_{t-1}, x_t]). GRU has roughly 25 percent fewer parameters than LSTM at the same hidden size and trains slightly faster; the two are usually within a percentage point on most tasks. See gru for a longer treatment.

Scroll sideways for more →

ConvLSTM

ConvLSTM (Shi et al., NeurIPS 2015^[15]) replaces the matrix multiplications inside the gates with convolutions. The cell state and hidden state become 3D tensors (channels by height by width) instead of vectors. This is the natural fit for any task where each time step is itself a spatial grid: weather radar, video, or fluid simulation. The original paper used it for short-term rainfall prediction in collaboration with the Hong Kong Observatory and beat the operational ROVER algorithm on the HKO-7 nowcasting dataset.

xLSTM family

The xLSTM paper (Beck et al., 2024^[3]) is the most ambitious LSTM revival in two decades. It introduces two new cell types. The sLSTM keeps a scalar cell state per dimension, like the classical LSTM, but replaces the sigmoid input and forget gates with exponential gating combined with a normalizer state for stability. This restores expressive power without sacrificing constant memory at inference. The mLSTM replaces the scalar cell state with a matrix C_t of shape (H, H) and uses a covariance update rule C_t = f_t C_{t-1} + i_t v_t k_t^T, where k_t, v_t, and q_t play the same role as keys, values, and queries in attention. The mLSTM recurrence has no hidden-to-hidden mixing, which makes it fully parallelizable during training via a chunkwise scan, like Mamba and linear attention.

Stacked into residual blocks alternating sLSTM and mLSTM, xLSTM-1.3B competes with Llama-1.3B and Mamba-1.3B at the same training budget. The NXAI release of xLSTM-7B in March 2025 trained on 2.3 trillion tokens with a context length of 8192 on 128 NVIDIA H100 GPUs^[4]. It reaches competitive performance on standard LLM benchmarks while running inference at constant memory, and the authors report it achieving roughly 70 percent higher throughput than Codestral Mamba and about 50 percent faster text generation than Mamba baselines on a single H100, independent of context length^[4].

Training

Training an LSTM looks like training any other recurrent network: the sequence is unrolled in time, the loss is summed across supervised steps, and gradients flow back through backpropagation through time. Three details deserve attention.

Gradient clipping. Even with the additive cell-state path, the gates and input-to-hidden weights can produce large gradients on rare events. Clipping the global gradient norm to 1 or 5 is standard; Pascanu, Mikolov, and Bengio (2013) gave the now-standard analysis^[7].

Truncated BPTT. Full BPTT over very long sequences is expensive. Truncated BPTT processes the sequence in chunks of, say, 100 to 200 steps, carrying the cell and hidden state forward but only backpropagating gradients within the current chunk. This is what .detach() on the hidden state between batches achieves in PyTorch.

Dropout. Naive dropout on the recurrent connections destroys the long-term memory because the mask changes at every step. Variational dropout (Gal & Ghahramani, 2016^[37]) fixes a single mask per sequence; zoneout (Krueger et al., 2017^[38]) randomly copies the previous hidden state instead of dropping units to zero. Most production code applies dropout only between stacked LSTM layers, which is what dropout controls in torch.nn.LSTM.

Initialization. Beyond the forget-gate bias trick, orthogonal initialization of the recurrent weight matrix and Glorot/Xavier initialization of the input weight matrix are common defaults. The "identity-RNN" initialization, where the recurrent matrix is the identity and the input weights are scaled by 1/H, is sometimes used to encourage long-memory behavior at the start of training.

Optimizer choice. Adam and RMSProp are the usual choices, with learning rates in the range 0.001 to 0.01. Plain SGD with momentum is harder to tune but sometimes yields the best final perplexity on language modeling. Cyclical learning rate schedules and warmup helped early Transformer NMT systems and have been ported back to LSTM training.

What is LSTM used for?

LSTMs powered most of the practical sequence learning systems in the 2010s. A small selection:

Scroll sideways for more →

Speech recognition

Speech recognition was arguably the first commercially important domain where LSTM beat alternative approaches at scale. Graves, Mohamed, and Hinton's 2013 paper used deep bidirectional LSTM stacks trained with Connectionist Temporal Classification to set a TIMIT phoneme recognition record at 17.7% error, beating the previous best of 20.7%^[16]. Sak, Senior, and Beaufays then showed at Interspeech 2014 that LSTM acoustic models could be trained on Google-scale distributed infrastructure and matched or beat existing hybrid DNN-HMM systems^[19]. Their distributed training framework split the model across multiple machines, used asynchronous SGD across hundreds of replicas, and trained on tens of thousands of hours of transcribed speech.

Within a year these models were running in Google Voice Search. The Google AI blog post from September 24, 2015, described how the team built better neural network acoustic models using CTC and sequence-discriminative training, replacing the older DNN-HMM hybrid front-end^[22]. Similar architectures soon powered Apple Siri dictation and Amazon's Alexa speech front-ends. The Microsoft Switchboard system reported in late 2016 used LSTMs in part of its acoustic model when it reached human parity on the corpus.

Sequence-to-sequence translation

Sutskever, Vinyals, and Le's 2014 paper^[18] introduced the seq2seq framework: an encoder LSTM compresses an input sequence into a fixed vector and a decoder LSTM expands that vector into the output. The headline trick was reversing the source sentence, which shortens the dependency between the first source word and the first target word and improved BLEU from 30.6 to 34.8 on WMT-14 English-French^[18]. The paper used a 4-layer LSTM with 1000 cells per layer.

Bahdanau, Cho, and Bengio added content-based attention in 2015^[41], removing the fixed-vector bottleneck and immediately improving translation quality especially on long sentences. With wordpiece encoding and 8+8 LSTM layers, Google Neural Machine Translation (Wu et al., 2016^[23]) reduced translation errors by 55 to 85 percent compared to the older phrase-based system, replaced production Google Translate, and was processing more than 18 million sentences daily on the Chinese-to-English pair within months of launch. LSTM-based seq2seq became the engine of production neural machine translation until the Transformer replaced it.

Image captioning

Vinyals, Toshev, Bengio, and Erhan's "Show and Tell" model^[21] glued a pretrained convolutional image encoder to an LSTM language decoder, conditioning the LSTM on the visual features at the first step. The system won the 2015 MSCOCO captioning challenge and established the encoder-decoder framework that dominated multimodal generation for several years. The follow-up "Show, Attend, and Tell" (Xu et al., 2015) added attention over CNN feature maps and is widely taught as the first multimodal use of attention.

Reinforcement learning

LSTM cores are standard in deep reinforcement learning whenever the environment is partially observed, which covers most non-trivial tasks. DRQN (Hausknecht & Stone, 2015) extended DQN with an LSTM and beat the original frame-stacking baseline on flickering Atari. OpenAI Five is the most famous LSTM-based RL system: each of the five Dota 2 hero agents is a single 4096-unit LSTM that observes the game state from the API, with the LSTM accounting for 84 percent of the model's 159 million parameters^[26]. After 10 months of training on 770 PFlop-day-equivalent compute, OpenAI Five beat the world champion team OG 2-0 at The International 2019. IMPALA (Espeholt et al., ICML 2018^[40]) introduced the actor-learner pattern that became standard for scalable RL, with an LSTM core for sequence memory. R2D2 (Kapturowski et al., ICLR 2019^[28]) added the missing ingredient of storing recurrent states in the replay buffer and quadrupled the previous state of the art on Atari-57. AlphaStar used a deep LSTM as the backbone of its StarCraft II policy and value networks, reaching Grandmaster level on Battle.net in 2019^[27].

Generative modeling and language

Andrej Karpathy's 2015 blog post "The Unreasonable Effectiveness of Recurrent Neural Networks" introduced an entire generation of practitioners to LSTM by showing it generate plausible Shakespeare, Linux kernel source code, and LaTeX algebra from character-level training^[39]. LSTM language models held state of the art on Penn Treebank and WikiText benchmarks until Transformer language models took over. ELMo (Peters et al., NAACL 2018^[25]) was the last big LSTM-based pretrained model before BERT. Its bidirectional LSTM was trained as a language model in each direction on roughly 1 billion words of text, then the hidden states from all layers were combined as contextual features for downstream tasks.

Other domains

In time-series forecasting LSTM remains a common baseline alongside Prophet, classical ARIMA, and modern Transformer variants. In computational biology LSTM has been used for protein structure features, secondary structure prediction, and DNA sequence analysis, although Transformers (and now Mamba-style models) have largely taken over there. In robotics, LSTMs are still the default policy core for partially observed control; in finance, they appear in market-microstructure modeling. The architecture's flexibility, modest data requirements, and constant per-step compute keep it competitive in any setting where data is small, sequences are long, or hardware budgets are tight.

What are the limitations of LSTM?

The two big problems with LSTM are both about scale.

First, the recurrence is inherently sequential. The cell state at step t depends on the cell state at step t-1, so you cannot parallelize the forward pass over time the way you can for self-attention. On modern GPUs a model that uses 5 percent of the silicon for 100 percent of the time is slower than one that uses 80 percent some of the time, even if the second does more arithmetic. cuDNN's fused LSTM kernel and tricks like quasi-RNN^[34] or simple recurrent unit (SRU) recover some of this throughput, but not all of it. The fundamental obstacle is the hidden-to-hidden multiplication, which prevents prefix-scan style parallelization.

Second, LSTMs are hard to scale to extremely long contexts. Even with a forget gate close to 1, the cell state is a fixed-size vector; you cannot cram a million tokens of context into a few thousand floats without losing information. Transformers handle this by keeping a key-value cache that grows with the input, paying quadratic compute for the privilege. State space models and xLSTM-mLSTM aim at the middle ground by using a structured but larger fixed-size memory (a matrix instead of a vector in the mLSTM case).

LSTM hyperparameters (learning rate, gradient clipping, hidden size, dropout) are also tightly coupled, and the same recipe rarely transfers across tasks. The Search Space Odyssey paper concluded that these hyperparameters are mostly independent and gave practical guidelines, but practitioners still report that getting an LSTM to train well takes more babysitting than a Transformer, mostly because Transformer pretraining recipes are now extremely well-documented and LSTM ones are not^[24].

Other documented weaknesses include difficulty modeling phenomena that need explicit copy operations across many steps (a problem attention solves by construction), sensitivity to long-tailed distributions, and the "saturation" failure mode where forget gates collapse to 1 or 0 and stop learning. The mogrifier LSTM and several other variants explicitly try to address some of these issues without abandoning the recurrent structure^[35].

Why did the Transformer displace LSTM?

The Transformer, introduced by Vaswani and colleagues in "Attention Is All You Need" (NeurIPS 2017^[2]), removed recurrence entirely and replaced it with self-attention over the whole sequence at once. This was good news on a GPU because every position can be processed in parallel, and bad news on the memory bill because the attention matrix is quadratic in sequence length. For most NLP tasks the trade-off favored Transformers, and from roughly 2018 onward the field shifted accordingly. ELMo (2018) was the last famous LSTM-based pretrained language model; BERT, GPT, and everything that followed used self-attention.

That said, LSTMs and Transformers have different inductive biases and different cost structures, and the choice is not as one-sided as the trend suggests.

Scroll sideways for more →

The practical upshot: Transformers win when you have lots of data, lots of compute, and care about throughput. LSTMs win, or at least compete, when you care about constant memory at inference (streaming speech, on-device, embedded), when sequences are very long but mostly recent context matters, when data is small, or when you want a strong online learning algorithm. Speech recognition front-ends, on-device keyboard prediction, real-time translation pipelines, and a long tail of tabular and time-series models still rely on LSTM cells.

Modern revivals

After several years where new sequence architectures meant some variant of attention, the late 2023 and 2024 wave of papers brought recurrence back. The headline names are Mamba, RWKV, and xLSTM.

Mamba and state space models

Mamba (Gu and Dao, arXiv:2312.00752, December 2023^[42]) is a state space model, not an LSTM, but it inherits the recurrent flavor: linear time in sequence length, constant memory at inference, and, in the selective version, input-dependent gating that closely mirrors what an LSTM does with its forget gate. Mamba-3B was the first sub-quadratic model to clearly match a Transformer of the same size on language modeling, and on long sequences it is significantly faster. Mamba builds on Gu's earlier S4 family of structured state space models and the "selective scan" idea that makes the recurrence data-dependent.

RWKV

RWKV (Peng et al., Findings of EMNLP 2023^[43]) reformulates attention so that it can be computed as a recurrence, giving it Transformer-like training and RNN-like inference. RWKV models have been trained up to 14 billion parameters, the largest dense RNN trained at the time of publication. The architecture is open source under Apache 2.0, with checkpoints from 169M to 14B parameters distributed on Hugging Face.

xLSTM

xLSTM (Beck, Pöppel, Spanring, Auer, Prudnikova, Kopp, Klambauer, Brandstetter, and Hochreiter, arXiv:2405.04517, May 2024^[3]) is the closest direct revival of the original idea. The paper introduces two new cell types: sLSTM, which keeps the scalar memory of the original LSTM but adds exponential gating with stabilization, and mLSTM, which replaces the scalar cell state with a matrix and uses a covariance update rule that can be parallelized like attention. Stacked into residual blocks, xLSTM models compete with Llama-style Transformers and Mamba at billion-parameter scale.

NXAI, the company Hochreiter co-founded with Netural X and PIERER Digital Holding in early 2024, has continued to push xLSTM as a basis for European foundation models. The March 2025 release of xLSTM-7B^[4] established the architecture as the fastest 7-billion-parameter recurrent language model and the largest LSTM-style network trained to date, with the JAX implementation open-sourced on GitHub at NX-AI/xlstm-jax. NXAI markets the model on its inference efficiency: linear time and constant memory let xLSTM-7B run on smaller hardware than Llama-7B or Mistral-7B at long context lengths, with the company reporting roughly 40 percent lower inference energy on comparable workloads.

Linear RNN renaissance

Mamba, RWKV, xLSTM, and their relatives (Griffin, Hawk, Retentive Networks, GLA, RetNet) are sometimes called the linear RNN renaissance or post-attention architectures. They share a few common moves: keep a fixed or slowly growing recurrent state, use input-dependent gating to compensate for lost expressive power of attention, and arrange the math so the recurrence can be evaluated as a parallel prefix scan during training. They have already won the long-context, on-device, and streaming markets where Transformers struggle, and have begun to make inroads into general LLM training at the 1 to 10 billion parameter scale.

Implementations

Every major deep learning framework ships an LSTM implementation backed by a fast GPU kernel. The interfaces are similar enough to swap with a one-line change.

Scroll sideways for more →

import torch
import torch.nn as nn

lstm = nn.LSTM(
    input_size=64,
    hidden_size=128,
    num_layers=2,
    batch_first=True,
    bidirectional=False,
    dropout=0.2,
)

x = torch.randn(32, 100, 64)         # (batch, time, features)
outputs, (h_n, c_n) = lstm(x)
# outputs: (32, 100, 128) - hidden state at every step
# h_n, c_n: (2, 32, 128) - final hidden and cell states for each layer

cuDNN's LSTM kernel fuses the four gate matrix multiplies into a single GEMM and applies the elementwise updates in a custom kernel, with sequence parallelism over batch and depth but not time. The kernel handles arbitrary sequence length up to GPU memory limits and supports packed sequences for batches of variable-length inputs. PyTorch's pack_padded_sequence and pad_packed_sequence utilities are the standard way to handle variable-length batches efficiently.

Cultural impact

For about five years, from roughly 2013 through 2018, LSTM was synonymous with sequence modeling in deep learning. The original 1997 paper has been cited well over 100,000 times across indices and carries roughly 98,800 citations on Semantic Scholar as of 2026, making it one of the most-cited papers in the history of artificial intelligence^[1]. Andrej Karpathy's 2015 blog post "The Unreasonable Effectiveness of Recurrent Neural Networks"^[39] introduced an entire generation of practitioners to LSTM by showing it generating plausible Shakespeare and compilable C code from a character-level model. Christopher Olah's 2015 blog post "Understanding LSTM Networks"^[44] remains, more than a decade later, the diagram people reach for when explaining the gates. The cell diagram from that post, with its color-coded gate operations, is reproduced in nearly every introductory deep-learning textbook and lecture slide deck.

The Transformer mostly displaced LSTM in research after 2018, but in production the transition has been slower. With xLSTM, Mamba, and RWKV bringing recurrence back to the frontier, the architecture's second act is still being written.

See also

• recurrent neural network - the broader family LSTM belongs to
• transformer - the attention-based architecture that displaced LSTM in most NLP tasks
• attention mechanism - the core operation behind Transformers
• bahdanau attention - the first content-based attention, added to LSTM seq2seq
• gru - the simpler sibling architecture by Cho et al. 2014
• seq2seq - the encoder-decoder framework first built on LSTM
• backpropagation through time - the training algorithm used for all RNN variants
• vanishing gradient - the problem LSTM was designed to solve
• xlstm - the 2024 modern revival of LSTM
• mamba - the 2023 selective state space competitor
• rwkv - the linear-cost RNN with Transformer training
• state space model - the family Mamba belongs to
• elmo - the last famous LSTM-based pretrained language model
• openai five - the Dota 2 agent built on a single 4096-unit LSTM per hero
• alphastar - the StarCraft II agent with an LSTM policy core
• jurgen schmidhuber - co-inventor of LSTM
• sepp hochreiter - co-inventor of LSTM and head of the xLSTM project

References

1. Hochreiter, S., & Schmidhuber, J., "Long Short-Term Memory", *Neural Computation* 9(8):1735-1780, MIT Press, 1997-11-15. https://direct.mit.edu/neco/article/9/8/1735/6109/Long-Short-Term-Memory. Citation count via Semantic Scholar: https://www.semanticscholar.org/paper/Long-Short-Term-Memory-Hochreiter-Schmidhuber/2e9d221c206e9503ceb452302d68d10e293f2a10. Accessed 2026-06-21. ↩
2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I., "Attention Is All You Need", *NeurIPS*, 2017-06-12. https://arxiv.org/abs/1706.03762. Accessed 2026-05-24. ↩
3. Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S., "xLSTM: Extended Long Short-Term Memory", *NeurIPS 2024*, arXiv:2405.04517, 2024-05-07. https://arxiv.org/abs/2405.04517. Accessed 2026-05-24. ↩
4. Beck, M., Pöppel, K., Lippe, P., Auer, A., et al. (NXAI), "xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference", arXiv:2503.13427, 2025-03-17. https://arxiv.org/abs/2503.13427. Accessed 2026-06-21. ↩
5. Hochreiter, S., *Untersuchungen zu dynamischen neuronalen Netzen*, Diploma thesis, Technische Universität München, 1991-04. https://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf. Accessed 2026-05-24. ↩
6. Bengio, Y., Simard, P., & Frasconi, P., "Learning Long-Term Dependencies with Gradient Descent Is Difficult", *IEEE Transactions on Neural Networks* 5(2):157-166, 1994-03. https://ieeexplore.ieee.org/document/279181. Accessed 2026-05-24. ↩
7. Pascanu, R., Mikolov, T., & Bengio, Y., "On the difficulty of training recurrent neural networks", *Proceedings of ICML*, 2013-02-16. https://arxiv.org/abs/1211.5063. Accessed 2026-05-24. ↩
8. Hochreiter, S., & Schmidhuber, J., "Long Short-Term Memory", Technical Report FKI-207-95, Fakultät für Informatik, Technische Universität München, 1995-08. https://people.idsia.ch/~juergen/lstm/. Accessed 2026-05-24. ↩
9. Gers, F. A., Schmidhuber, J., & Cummins, F., "Learning to Forget: Continual Prediction with LSTM", *Neural Computation* 12(10):2451-2471, MIT Press, 2000-10. https://direct.mit.edu/neco/article/12/10/2451/6415/Learning-to-Forget-Continual-Prediction-with-LSTM. Accessed 2026-05-24. ↩
10. Gers, F. A., & Schmidhuber, J., "Recurrent nets that time and count", *Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks*, IJCNN 2000. https://ieeexplore.ieee.org/document/861302. Accessed 2026-05-24. ↩
11. Gers, F. A., Schraudolph, N. N., & Schmidhuber, J., "Learning Precise Timing with LSTM Recurrent Networks", *Journal of Machine Learning Research* 3:115-143, 2002-08. https://www.jmlr.org/papers/v3/gers02a.html. Accessed 2026-05-24. ↩
12. Graves, A., & Schmidhuber, J., "Framewise phoneme classification with bidirectional LSTM and other neural network architectures", *Neural Networks* 18(5-6):602-610, 2005-07. https://www.cs.toronto.edu/~graves/nn_2005.pdf. Accessed 2026-05-24. ↩
13. Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J., "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks", *Proceedings of ICML*, 2006-06. https://www.cs.toronto.edu/~graves/icml_2006.pdf. Accessed 2026-05-24. ↩
14. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y., "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation", *Proceedings of EMNLP 2014*, 2014-06-03. https://arxiv.org/abs/1406.1078. Accessed 2026-05-24. ↩
15. Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W., & Woo, W., "Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting", *NeurIPS 2015*, 2015-06-13. https://arxiv.org/abs/1506.04214. Accessed 2026-05-24. ↩
16. Graves, A., Mohamed, A., & Hinton, G., "Speech Recognition with Deep Recurrent Neural Networks", *ICASSP 2013*, 2013-03-22. https://arxiv.org/abs/1303.5778. Accessed 2026-05-24. ↩
17. Graves, A., "Generating Sequences With Recurrent Neural Networks", arXiv:1308.0850, 2013-08-04. https://arxiv.org/abs/1308.0850. Accessed 2026-05-24. ↩
18. Sutskever, I., Vinyals, O., & Le, Q. V., "Sequence to Sequence Learning with Neural Networks", *NeurIPS 2014*, 2014-09-10. https://arxiv.org/abs/1409.3215. Accessed 2026-06-21. ↩
19. Sak, H., Senior, A., & Beaufays, F., "Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling", *Interspeech 2014*, 2014-02-05. https://arxiv.org/abs/1402.1128. Accessed 2026-05-24. ↩
20. Jozefowicz, R., Zaremba, W., & Sutskever, I., "An Empirical Exploration of Recurrent Network Architectures", *Proceedings of ICML 2015*, 2015-07-06. https://proceedings.mlr.press/v37/jozefowicz15.pdf. Accessed 2026-05-24. ↩
21. Vinyals, O., Toshev, A., Bengio, S., & Erhan, D., "Show and Tell: A Neural Image Caption Generator", *CVPR 2015*, 2014-11-17. https://arxiv.org/abs/1411.4555. Accessed 2026-05-24. ↩
22. Sak, H., Senior, A., Rao, K., Beaufays, F., & Schalkwyk, J., "Google voice search: faster and more accurate", Google AI Blog, 2015-09-24. https://research.google/blog/google-voice-search-faster-and-more-accurate/. Accessed 2026-05-24. ↩
23. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., et al., "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation", arXiv:1609.08144, 2016-09-26. https://arxiv.org/abs/1609.08144. Accessed 2026-05-24. ↩
24. Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., & Schmidhuber, J., "LSTM: A Search Space Odyssey", *IEEE Transactions on Neural Networks and Learning Systems* 28(10):2222-2232, 2017-07-10. https://arxiv.org/abs/1503.04069. Accessed 2026-06-21. ↩
25. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L., "Deep contextualized word representations", *NAACL 2018*, 2018-02-15. https://arxiv.org/abs/1802.05365. Accessed 2026-05-24. ↩
26. OpenAI, Berner, C., Brockman, G., Chan, B., et al., "Dota 2 with Large Scale Deep Reinforcement Learning", arXiv:1912.06680, 2019-12-13. https://arxiv.org/abs/1912.06680. Accessed 2026-05-24. ↩
27. Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., et al., "Grandmaster level in StarCraft II using multi-agent reinforcement learning", *Nature* 575:350-354, 2019-10-30. https://www.nature.com/articles/s41586-019-1724-z. Accessed 2026-05-24. ↩
28. Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., & Dabney, W., "Recurrent Experience Replay in Distributed Reinforcement Learning", *ICLR 2019*. https://openreview.net/forum?id=r1lyTjAqYX. Accessed 2026-05-24. ↩
29. Schmidhuber, J., "Annotated History of Modern AI and Deep Learning", arXiv:2212.11279, 2022-12-21. https://arxiv.org/abs/2212.11279. Accessed 2026-05-24. ↩
30. Linz Institute of Technology, "AI-Pioneer Prof. Hochreiter Presented with the 'Neural Networks Pioneer Award'", JKU news, 2021. https://www.jku.at/en/linz-institute-of-technology/news-events/news/detail/news/neural-networks-pioneer-award-fuer-ki-pionier-hochreiter/. Accessed 2026-05-24. ↩
31. PyTorch documentation, "torch.nn.LSTM", PyTorch Foundation. https://docs.pytorch.org/docs/stable/generated/torch.nn.LSTM.html. Accessed 2026-05-24. ↩
32. Zhang, Y., Chen, G., Yu, D., Yao, K., Khudanpur, S., & Glass, J., "Highway Long Short-Term Memory RNNs for Distant Speech Recognition", arXiv:1510.08983, 2015-10-30. https://arxiv.org/abs/1510.08983. Accessed 2026-05-24. ↩
33. Radford, A., Jozefowicz, R., & Sutskever, I., "Learning to Generate Reviews and Discovering Sentiment", arXiv:1704.01444, 2017-04-06. https://arxiv.org/abs/1704.01444. Accessed 2026-05-24. ↩
34. Bradbury, J., Merity, S., Xiong, C., & Socher, R., "Quasi-Recurrent Neural Networks", *ICLR 2017*, arXiv:1611.01576, 2016-11-05. https://arxiv.org/abs/1611.01576. Accessed 2026-05-24. ↩
35. Melis, G., Kočiský, T., & Blunsom, P., "Mogrifier LSTM", *ICLR 2020*, arXiv:1909.01792, 2019-09-04. https://arxiv.org/abs/1909.01792. Accessed 2026-05-24. ↩
36. Tai, K. S., Socher, R., & Manning, C. D., "Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks", *ACL 2015*, arXiv:1503.00075, 2015-02-28. https://arxiv.org/abs/1503.00075. Accessed 2026-05-24. ↩
37. Gal, Y., & Ghahramani, Z., "A Theoretically Grounded Application of Dropout in Recurrent Neural Networks", *NeurIPS 2016*, arXiv:1512.05287. https://arxiv.org/abs/1512.05287. Accessed 2026-05-24. ↩
38. Krueger, D., Maharaj, T., Kramár, J., et al., "Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations", *ICLR 2017*, arXiv:1606.01305. https://arxiv.org/abs/1606.01305. Accessed 2026-05-24. ↩
39. Karpathy, A., "The Unreasonable Effectiveness of Recurrent Neural Networks", personal blog, 2015-05-21. https://karpathy.github.io/2015/05/21/rnn-effectiveness/. Accessed 2026-05-24. ↩
40. Espeholt, L., Soyer, H., Munos, R., et al., "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures", *ICML 2018*, arXiv:1802.01561. https://arxiv.org/abs/1802.01561. Accessed 2026-05-24. ↩
41. Bahdanau, D., Cho, K., & Bengio, Y., "Neural Machine Translation by Jointly Learning to Align and Translate", *ICLR 2015*, arXiv:1409.0473, 2014-09-01. https://arxiv.org/abs/1409.0473. Accessed 2026-05-24. ↩
42. Gu, A., & Dao, T., "Mamba: Linear-Time Sequence Modeling with Selective State Spaces", arXiv:2312.00752, 2023-12-01. https://arxiv.org/abs/2312.00752. Accessed 2026-05-24. ↩
43. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., et al., "RWKV: Reinventing RNNs for the Transformer Era", *Findings of EMNLP 2023*, arXiv:2305.13048, 2023-05-22. https://arxiv.org/abs/2305.13048. Accessed 2026-05-24. ↩
44. Olah, C., "Understanding LSTM Networks", colah's blog, 2015-08-27. https://colah.github.io/posts/2015-08-Understanding-LSTMs/. Accessed 2026-05-24. ↩