# Recurrent Neural Network

> Source: https://aiwiki.ai/wiki/recurrent_neural_network
> Updated: 2026-06-20
> Categories: Deep Learning, Machine Learning, Model Architecture, Neural Networks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution to "AI Wiki (aiwiki.ai)".

*See also: [Machine learning terms](/wiki/machine_learning_terms)*

A **recurrent neural network** (**RNN**) is a class of [artificial neural network](/wiki/neural_network) designed to process sequential data by maintaining an internal hidden state that persists information across time steps. Unlike [feedforward neural networks](/wiki/feedforward_neural_network_ffn), which map a fixed-size input to a fixed-size output in a single pass, RNNs contain cyclic connections that allow information to flow from one step of a computation to the next. This makes them naturally suited for tasks where the order of inputs matters, such as language modeling, speech recognition, and time series forecasting. The defining feature of an RNN is that it applies the same set of weights repeatedly at every step of a sequence, reusing a hidden "memory" vector to carry context forward.

RNNs were among the first neural architectures to handle variable-length sequences, and they were among the most widely used architectures in [natural language processing](/wiki/natural_language_processing) and sequence modeling from the late 1980s through the mid-2010s. The most influential RNN variant, Long Short-Term Memory (LSTM), was introduced by Sepp Hochreiter and Jurgen Schmidhuber in 1997, and its founding paper has become the most cited neural network paper of the 20th century, with more than 80,000 citations [6][22]. Although [transformer](/wiki/transformer)-based models have largely supplanted RNNs in many domains since 2017, the core ideas behind recurrent computation remain influential in [deep learning](/wiki/deep_learning), and newer architectures such as state space models are revisiting recurrent principles with modern techniques. Recurrent networks also continue to see use in resource-constrained settings and certain real-time processing tasks.

## ELI5: Explain like I'm 5

Imagine you are reading a story one word at a time. After each word, you update a little summary in your head of what the story is about so far. When you reach the next word, you use both that new word and your running summary to understand what is happening. A recurrent neural network works the same way. It reads a sequence of inputs (like words) one at a time, keeps a "memory" of what it has seen, and uses that memory along with each new input to make predictions. The memory is not perfect, though. If the story is very long, the network might forget details from the beginning, which is why researchers invented improved versions like LSTM and GRU that are better at remembering important things over long stretches. This same idea helps computers understand sentences or patterns in music, and even predict what might come next.

## When were RNNs invented? History and development

The concept of recurrent connections in neural networks dates back to the early days of connectionist research. John Hopfield introduced the Hopfield network in 1982, a form of recurrent network used as an associative memory [1]. While not a sequence model in the modern sense, the Hopfield network demonstrated that recurrent connections could store and retrieve patterns [1].

In 1986, David Rumelhart, [Geoffrey Hinton](/wiki/geoffrey_hinton), and Ronald Williams published their influential work on [backpropagation](/wiki/backpropagation), which laid the groundwork for training multilayer networks. Michael Jordan proposed the "Jordan network" in 1986, where the output layer rather than the hidden layer was fed back as context [2]. Shortly after, in 1990, Jeffrey Elman introduced the "Elman network" (sometimes called the "simple recurrent network"), which added a context layer that fed the previous hidden state back into the network at each time step [3]. Elman opened his paper with the observation that "Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important." [3] This architecture became the prototype for what is now called the "vanilla RNN." Both Elman and Jordan networks established the basic principle of using recurrence to model sequences [3].

The 1990s saw growing awareness of the difficulties in training RNNs on long sequences, particularly the vanishing gradient problem identified by Sepp Hochreiter in his 1991 diploma thesis and later formalized by [Yoshua Bengio](/wiki/yoshua_bengio), Patrice Simard, and Paolo Frasconi in 1994 [5]. This led to the invention of Long Short-Term Memory (LSTM) networks by Hochreiter and Jurgen Schmidhuber in 1997, which became the dominant RNN variant for over a decade [6].

The Gated Recurrent Unit (GRU) was introduced by Kyunghyun Cho and colleagues in 2014 as a simpler alternative to LSTM [10]. Around the same time, the development of sequence-to-sequence models by [Ilya Sutskever](/wiki/ilya_sutskever), Oriol Vinyals, and Quoc Le (2014) [11], along with the [attention](/wiki/attention) mechanism by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio (2014) [12], brought RNN-based architectures to new heights of performance in [machine translation](/wiki/machine_translation) and other tasks.

The publication of "[Attention Is All You Need](/wiki/attention_is_all_you_need)" by Vaswani et al. in 2017 introduced the Transformer, which replaced recurrence entirely with self-attention [14]. The authors proposed "a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." [14] This marked the beginning of a shift away from RNNs in most large-scale NLP applications.

## How do RNNs work?

At each time step _t_, an RNN receives an input vector **x_t** and combines it with the hidden state **h_{t-1}** from the previous time step to produce a new hidden state **h_t**. The hidden state acts as the network's memory: it encodes a compressed summary of all inputs the network has processed so far. An optional output **y_t** can be computed from the hidden state at any time step.

Conceptually, the same set of weights is reused at every time step. This weight sharing is what gives RNNs their ability to generalize across different positions in a sequence regardless of sequence length, and is a defining characteristic of the architecture.

### Mathematical formulation

The simplest (vanilla) RNN computes the hidden state and output as follows:

```
h_t = f(W_hh * h_{t-1} + W_xh * x_t + b_h)
y_t = g(W_hy * h_t + b_y)
```

Where:

| Symbol | Meaning |
|---|---|
| **x_t** | Input vector at time step _t_ |
| **h_t** | Hidden state at time step _t_ |
| **h_{t-1}** | Hidden state from the previous time step |
| **W_xh** | Weight matrix from input to hidden layer |
| **W_hh** | Weight matrix from hidden layer to hidden layer (recurrent weights) |
| **W_hy** | Weight matrix from hidden layer to output |
| **b_h**, **b_y** | Bias vectors |
| **f** | [Activation function](/wiki/activation_function), typically tanh or [ReLU](/wiki/relu) |
| **g** | Output activation (e.g., [softmax](/wiki/softmax) for classification) |

The hidden state **h_0** is usually initialized to a zero vector at the start of a sequence.

### Hidden state as memory

The hidden state h(t) is a vector of fixed dimensionality (chosen as a hyperparameter) that summarizes all information from the input sequence up to time step t. In theory, this allows the RNN to capture arbitrarily long-range dependencies. In practice, vanilla RNNs struggle to retain information over many time steps due to the vanishing gradient problem, which motivated the development of gated architectures like LSTM and GRU.

The hidden state is initialized (typically to a zero vector) at the start of each sequence. As the network processes each element, the hidden state is progressively updated, building a compressed representation of the sequence history.

### Unrolling through time

To understand how an RNN processes a sequence, it helps to "unroll" (or "unfold") the network across time steps. Unrolling replaces the single recurrent cell with a chain of identical cells, one for each time step. Each cell receives the input at its time step and the hidden state from the previous cell, and passes its own hidden state to the next cell. For a sequence of length _T_, the unrolled network has _T_ copies of the same recurrent cell, connected sequentially, with all copies sharing the same weights. When unrolled, an RNN resembles a very deep feedforward network with shared weights at each layer, which is the perspective used during training with [backpropagation](/wiki/backpropagation) through time.

## How are RNNs trained?

### Backpropagation through time (BPTT)

RNNs are trained using **backpropagation through time** (**BPTT**), a direct extension of the standard backpropagation algorithm to sequences. The process works as follows:

1. **Forward pass:** The input sequence is fed through the network one time step at a time, computing hidden states and outputs for each step.
2. **Loss computation:** A [loss function](/wiki/loss_function) (such as cross-entropy for classification or mean squared error for regression) is computed at each output time step, and the individual losses are summed or averaged over the sequence.
3. **Backward pass:** The total loss is backpropagated through the unrolled network, computing gradients with respect to all weights. Because the same parameters appear at every time step, the gradient with respect to any weight is the sum of the gradients contributed by each time step. For example, computing the gradient at time step _t_ = 4 requires backpropagating through three earlier steps and summing the resulting gradient contributions. This summation over time steps is what distinguishes BPTT from ordinary backpropagation.
4. **Weight update:** The accumulated gradients are used to update the weights via an optimizer such as [stochastic gradient descent](/wiki/stochastic_gradient_descent_sgd) or [Adam](/wiki/adam_optimizer).

The computational cost of BPTT is O(T) in both time and memory, where _T_ is the sequence length. For very long sequences, this can become prohibitively expensive.

### Truncated BPTT

To reduce memory and computation costs, practitioners often use **truncated backpropagation through time**. Instead of unrolling the entire sequence, truncated BPTT divides the sequence into shorter segments (for example, 20 or 50 time steps) and backpropagates gradients only within each segment. The hidden state is carried forward from one segment to the next (maintaining continuity), but gradients are not propagated across segment boundaries.

Truncated BPTT introduces a tradeoff: it reduces memory usage and speeds up training, but it limits the network's ability to learn dependencies longer than the truncation window. In practice, this is often an acceptable compromise, especially for tasks where the most relevant context is relatively local. Truncated BPTT was widely used in practice for training language models and other RNN applications on long documents or continuous data streams.

### Teacher forcing

Teacher forcing is a training strategy commonly used with RNNs that generate sequences, such as [language model](/wiki/language_model) decoders and machine translation systems. During training, instead of feeding the model's own output from the previous time step as the next input, teacher forcing supplies the ground-truth token from the training data [4].

This approach speeds up convergence because the model does not have to recover from its own early mistakes during training. However, it introduces a mismatch between training and inference known as **exposure bias**: at inference time, the model must rely on its own predictions, which may differ from the ground-truth tokens it was trained on. Small prediction errors can compound over a long generated sequence, leading to degraded output quality.

Several techniques have been proposed to mitigate exposure bias, including scheduled sampling (gradually transitioning from teacher forcing to model predictions during training) and professor forcing, which uses adversarial training to align the hidden-state dynamics of teacher-forced and free-running modes.

## Why do RNNs struggle with long sequences? The vanishing and exploding gradient problems

The most significant challenge in training vanilla RNNs is the [vanishing gradient problem](/wiki/vanishing_gradient_problem). When gradients are backpropagated through many time steps, they are repeatedly multiplied by the recurrent weight matrix **W_hh** and the derivative of the activation function. If the largest eigenvalue (or singular value) of **W_hh** is less than 1, gradients shrink exponentially with the number of time steps, effectively preventing the network from learning long-range dependencies. This means the network cannot learn dependencies that span many time steps, because the error signal from distant outputs does not reach earlier hidden states. Conversely, if the largest eigenvalue exceeds 1, gradients can grow exponentially, a condition known as the **exploding gradient problem**. Exploding gradients cause numerical instability and can make training diverge entirely.

These problems were formally analyzed by Yoshua Bengio, Patrice Simard, and Paolo Frasconi in 1994 [5]. As they stated in their abstract, "We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases." [5] Their analysis showed that for vanilla RNNs, the influence of an input on the hidden state decays (or explodes) exponentially with the temporal distance, and that the ability to learn long-range dependencies decreases exponentially with the length of the dependency [5]. This analysis motivated the development of gated RNN architectures.

### Solutions to gradient problems

| Technique | Description | Addresses |
|---|---|---|
| Gradient clipping | Rescale gradients when their norm exceeds a threshold | Exploding gradients |
| LSTM / GRU gates | Gating mechanisms that control information flow and maintain stable gradients | Vanishing gradients |
| Orthogonal initialization | Initialize **W_hh** as an orthogonal matrix so eigenvalues start near 1 | Both |
| Skip connections | Add direct connections across multiple time steps | Vanishing gradients |
| Batch normalization / layer normalization | Normalize activations to stabilize training dynamics | Both |
| Gradient regularization | Add a penalty to encourage gradients to remain in a stable range | Both |
| Truncated BPTT | Limit the number of time steps for backpropagation | Exploding gradients (indirectly) |

Gradient clipping, proposed by Thomas Mikolov, is one of the simplest and most widely used techniques. It sets a maximum threshold for the gradient norm; if the gradient exceeds this threshold, it is scaled down proportionally. This prevents the extreme parameter updates caused by exploding gradients but does not solve the vanishing gradient problem. Of the listed techniques, gated architectures (LSTM and GRU) have proven the most effective and widely adopted solution to vanishing gradients.

## Long Short-Term Memory (LSTM)

[LSTM](/wiki/long_short-term_memory_lstm) is a gated RNN architecture introduced by Sepp Hochreiter and Jurgen Schmidhuber in 1997 to address the vanishing gradient problem [6]. The key innovation is a **cell state** (also called the memory cell) that runs through the entire sequence like a conveyor belt, with information added or removed through learned gating mechanisms [6]. In the original paper, the authors wrote that "Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carrousels within special units." [6] This design allows LSTMs to learn dependencies spanning hundreds or even thousands of time steps [6]. The 1997 paper was published in *Neural Computation*, volume 9, issue 8, pages 1735-1780, and is the most cited neural network paper of the 20th century [6][22].

### LSTM architecture

An LSTM cell contains three gates and a cell state. Each gate is implemented as a sigmoid layer followed by element-wise multiplication:

| Component | Function | Equation |
|---|---|---|
| **Forget gate (f_t)** | Decides what information to discard from the cell state | f_t = sigma(W_f * [h_{t-1}, x_t] + b_f) |
| **Input gate (i_t)** | Decides what new information to store in the cell state | i_t = sigma(W_i * [h_{t-1}, x_t] + b_i) |
| **Candidate values** | Creates a vector of new candidate values | C_tilde_t = tanh(W_C * [h_{t-1}, x_t] + b_C) |
| **Cell state update** | Combines old cell state with new candidates | C_t = f_t * C_{t-1} + i_t * C_tilde_t |
| **Output gate (o_t)** | Decides what part of the cell state to output | o_t = sigma(W_o * [h_{t-1}, x_t] + b_o) |
| **Hidden state** | Filtered version of the cell state | h_t = o_t * tanh(C_t) |

The forget gate outputs values between 0 and 1 for each element of the cell state; a value of 1 means "keep this entirely" and 0 means "discard this completely." The input gate and candidate values together determine what new information is written to the cell state. The output gate controls which parts of the cell state are exposed as the hidden state for downstream computation.

Notably, the original 1997 LSTM had only input and output gates; the now-standard forget gate was added by Felix Gers, Jurgen Schmidhuber, and Fred Cummins in 2000, in a paper titled "Learning to Forget: Continual Prediction with LSTM" [23].

Because the cell state update involves only element-wise multiplication and addition (no matrix multiplication by **W_hh**), gradients can flow through the cell state with minimal attenuation. The forget gate allows the network to explicitly "reset" parts of the cell state when they are no longer relevant. This is the core mechanism that mitigates the vanishing gradient problem in LSTMs.

### LSTM variants

Several variants of the original LSTM have been proposed:

- **Peephole connections** (Gers and Schmidhuber, 2000): Allow the gates to look at the cell state directly, not just the hidden state.
- **Coupled forget and input gates**: Replace the separate forget and input gates with a single gate, so the cell forgets exactly as much as it remembers new information (f_t = 1 - i_t).
- **LSTM with projection layers**: Adds a linear projection after the output gate to reduce the dimensionality of the hidden state, reducing the number of parameters.
- **xLSTM** (Hochreiter et al., 2024): An extended LSTM with exponential gating, scalar memory (sLSTM), and matrix memory (mLSTM) variants, designed to compete with transformers at scale [21].

Research by Klaus Greff et al. (2017) systematically evaluated LSTM variants and found that the standard LSTM architecture performed well across tasks, with no single variant consistently outperforming it [15]. The forget gate bias initialization (setting it to 1 so the network defaults to remembering) was found to be one of the most important practical considerations [15].

## Gated Recurrent Unit (GRU)

The Gated Recurrent Unit (GRU) was introduced by Kyunghyun Cho and colleagues in 2014 as a simpler alternative to LSTM [10]. GRUs achieve comparable performance to LSTMs on many tasks while using fewer parameters and less computation. The GRU merges the cell state and hidden state into a single state vector and uses only two gates instead of three [10].

### GRU architecture

A GRU has two gates:

| Component | Function | Equation |
|---|---|---|
| **Update gate (z_t)** | Controls how much of the previous hidden state to retain | z_t = sigma(W_z * [h_{t-1}, x_t] + b_z) |
| **Reset gate (r_t)** | Controls how much of the previous hidden state to forget when computing the candidate | r_t = sigma(W_r * [h_{t-1}, x_t] + b_r) |
| **Candidate hidden state** | Computes a proposed new hidden state | h_tilde_t = tanh(W * [r_t * h_{t-1}, x_t] + b) |
| **Hidden state update** | Interpolates between old and candidate hidden states | h_t = (1 - z_t) * h_{t-1} + z_t * h_tilde_t |

Notably, the GRU does not maintain a separate cell state. All memory is stored directly in the hidden state. The update gate serves a role similar to both the forget and input gates of the LSTM, while the reset gate determines how much past information to incorporate into the candidate computation. When the reset gate is close to 0, the network behaves as if it is reading the first symbol of a sequence, effectively allowing it to drop irrelevant past information. The update gate z_t acts as a direct interpolation between the old state and the candidate state: when z_t is close to 0, the hidden state is largely copied from the previous step; when z_t is close to 1, the hidden state is replaced by the new candidate.

### How does an LSTM differ from a GRU?

| Feature | [LSTM](/wiki/long_short-term_memory_lstm) | GRU |
|---|---|---|
| Number of gates | 3 (forget, input, output) | 2 (update, reset) |
| Separate cell state | Yes | No |
| Parameters per unit | More (roughly 4x hidden size^2) | Fewer (roughly 3x hidden size^2) |
| Training speed | Slower | Faster |
| Long-range dependencies | Generally better on complex sequences | Comparable on shorter or simpler sequences |
| Memory usage | Higher | Lower |

The GRU has fewer parameters than the LSTM, which makes it faster to train and less prone to overfitting on small datasets. Empirical studies have found that neither architecture consistently outperforms the other across all tasks. GRUs tend to do better when data is limited or sequences are shorter, while LSTMs often have an edge on tasks requiring very long-term memory. In practice, the choice between GRU and LSTM is often made based on computational budget and dataset size.

## Comparison of RNN variants

| Feature | Vanilla RNN | [LSTM](/wiki/long_short-term_memory_lstm) | GRU |
|---|---|---|---|
| Year introduced | ~1990 (Elman) | 1997 (Hochreiter & Schmidhuber) | 2014 (Cho et al.) |
| Number of gates | 0 | 3 (forget, input, output) | 2 (update, reset) |
| Separate cell state | No | Yes | No |
| Parameters per unit | Low | High | Medium |
| Long-range dependencies | Poor | Strong | Good |
| Training speed | Fast | Slow | Moderate |
| Risk of vanishing gradients | High | Low | Low |
| Best suited for | Short sequences, simple patterns | Long sequences, complex dependencies | Medium-length sequences, limited compute |
| Common use cases | Toy problems, educational examples | [Machine translation](/wiki/machine_translation), [speech recognition](/wiki/speech_recognition), text generation | Similar to LSTM; preferred when speed matters |

## RNN architectures and variants

### Elman networks and Jordan networks

The earliest practical RNN architectures were simple recurrent networks (SRNs). Jeffrey Elman introduced the **Elman network** in 1990, where the hidden state from the previous time step is copied to a set of "context units" that feed back into the hidden layer [3]. Michael Jordan proposed the **Jordan network** in 1986, which instead feeds the output from the previous time step back to the hidden layer through context units [2].

| Architecture | Recurrent connection | Year |
|---|---|---|
| Jordan network | Output to hidden (via context units) | 1986 |
| Elman network | Hidden to hidden (via context units) | 1990 |

Both architectures were foundational in demonstrating that recurrent networks could learn temporal structure, but they struggled with long sequences due to the vanishing gradient problem.

### Bidirectional RNNs

A bidirectional RNN (BiRNN), introduced by Mike Schuster and Kuldip Paliwal in 1997, processes the input sequence in both the forward and backward directions using two separate hidden states [7]. The forward hidden state captures information from past inputs, while the backward hidden state captures information from future inputs. At each time step, the two hidden states are typically concatenated to form the final representation [7].

Bidirectional RNNs are particularly useful for tasks where the entire input sequence is available before making predictions, such as named entity recognition, part-of-speech tagging, and speech recognition. The motivation for bidirectional processing is that in many tasks, the meaning of a particular element depends on both its past and future context. For example, the correct part-of-speech tag for a word may depend on words that appear both before and after it.

Bidirectional RNNs can use any recurrent cell type (vanilla RNN, LSTM, or GRU) in each direction. Bidirectional LSTMs (BiLSTMs) became the dominant architecture for many NLP tasks in the mid-2010s, including named entity recognition, [sentiment analysis](/wiki/sentiment_analysis), question answering, and syntactic parsing.

A limitation of bidirectional RNNs is that they cannot be used in settings that require causal (left-to-right) generation or real-time/streaming applications, since the backward pass requires access to future inputs.

### Deep (stacked) RNNs

Stacking multiple RNN layers on top of each other creates a **deep RNN** (also called a stacked RNN). The hidden state output of one RNN layer serves as the input sequence for the next layer. Deep RNNs can learn hierarchical representations of sequential data: lower layers might capture local patterns (like phonemes in speech), while higher layers capture more abstract features (like words or phrases).

In practice, deep RNNs with 2 to 4 layers often outperform single-layer RNNs, but performance gains diminish with additional depth. Residual connections, highway connections, and layer normalization are commonly used between layers of deep RNNs to stabilize training and facilitate gradient flow, borrowing ideas from deep feedforward and [convolutional neural network](/wiki/convolutional_neural_network) architectures. Deep RNNs were found to improve performance on tasks such as [speech recognition](/wiki/speech_recognition) and machine translation, though they are more difficult to train than single-layer models.

### Sequence-to-sequence (Seq2Seq) models

The [sequence model](/wiki/sequence_model) known as sequence-to-sequence (Seq2Seq) was introduced independently by Ilya Sutskever, Oriol Vinyals, and Quoc V. Le (2014) [11] and by Kyunghyun Cho et al. (2014) [10]. A Seq2Seq model maps variable-length input sequences to variable-length output sequences and consists of two RNNs:

1. **Encoder:** An RNN that reads the input sequence and compresses it into a fixed-length context vector (typically the final hidden state of the encoder).
2. **Decoder:** An RNN that generates the output sequence one element at a time, conditioned on the context vector and its own previous outputs.

Seq2Seq models achieved breakthrough results in machine translation, text summarization, and conversational AI, and became the foundation for neural machine translation (NMT) systems. They were also applied to dialogue systems and code generation. However, the fixed-length context vector creates an information bottleneck: for long input sequences, the encoder must compress all relevant information into a single vector, which inevitably loses detail.

### Attention mechanisms with RNNs

The [attention](/wiki/attention) mechanism, proposed by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio in 2014, addressed the context vector bottleneck in Seq2Seq models [12]. Instead of relying on a single fixed context vector, the decoder is allowed to attend to all encoder hidden states at each decoding step, computing a weighted sum where the weights reflect the relevance of each encoder state to the current decoding position [12].

Bahdanau attention uses an additive scoring function: the decoder hidden state and each encoder hidden state are passed through separate linear layers, summed, and fed through a tanh activation to produce alignment scores [12]. These scores are then normalized with softmax to produce attention weights.

This innovation dramatically improved translation quality for long sentences and became a standard component of RNN-based Seq2Seq systems [12]. It also provided interpretability, as the attention weights could be visualized to show which input elements the model was focusing on. The success of attention in RNN-based models directly inspired the development of the self-attention mechanism in transformers, which dispensed with recurrence altogether [14].

## What are RNNs used for? Applications

RNNs (and their gated variants) have been applied to a wide range of sequential tasks:

| Application domain | Examples | Typical architecture |
|---|---|---|
| Natural language processing | [Language model](/wiki/language_model) training, sentiment analysis, text classification | LSTM / GRU with attention |
| Machine translation | Translating between languages | Seq2Seq with Bahdanau or Luong attention |
| Speech recognition | Converting audio waveforms to text | Bidirectional LSTM, CTC loss |
| Time series forecasting | Stock prices, weather prediction, energy demand | Stacked LSTM / GRU |
| Music generation | Composing melodies and harmonies | Character-level LSTM |
| Handwriting recognition | Converting handwritten text to digital characters | Bidirectional LSTM with CTC |
| Video analysis | Activity recognition, video captioning | CNN encoder + LSTM decoder |
| Bioinformatics | Protein structure prediction, gene sequence analysis | Bidirectional GRU / LSTM |

### Language modeling

A [language model](/wiki/language_model) assigns probabilities to sequences of words. RNN-based language models process text one word (or character) at a time, using the hidden state to maintain context. At each step, the model predicts the probability distribution over the next word given all previous words.

RNN language models significantly outperformed traditional n-gram models, particularly for capturing longer-range dependencies [9]. Tomas Mikolov's work on RNN-based language models (2010-2012) demonstrated large improvements in [perplexity](/wiki/perplexity) (a standard metric for language models) compared to n-gram baselines [9].

Before the rise of Transformer-based models like GPT and [BERT](/wiki/bert), LSTM-based language models represented the state of the art. [OpenAI](/wiki/openai)'s early language model research used LSTMs before switching to Transformers.

### Machine translation

Neural machine translation using RNN-based seq2seq models with attention became the dominant approach to machine translation from 2014 to 2017. Google's Neural Machine Translation system (GNMT), deployed in 2016, used a deep LSTM encoder-decoder with 8 encoder and 8 decoder layers connected by attention, and reduced translation errors by an average of 60% compared to Google's previous phrase-based production system, as measured by human side-by-side evaluations across several language pairs [24]. This system replaced Google's previous phrase-based statistical machine translation system and achieved near-human-level translation quality on several language pairs.

### Speech recognition

RNNs are well suited for speech recognition because speech is inherently sequential. Deep bidirectional LSTMs became the standard acoustic model in automatic speech recognition (ASR) systems. Connectionist Temporal Classification (CTC), a training criterion designed for sequence labeling with RNNs, was introduced by Alex Graves et al. in 2006 and became widely used in end-to-end speech recognition [8].

Baidu's Deep Speech (2014) and Deep Speech 2 (2015) systems demonstrated that deep RNNs trained on large datasets could achieve competitive speech recognition performance with simplified pipelines. Google's voice search and dictation systems also used LSTM-based models extensively.

### Time series forecasting

RNNs have been applied to forecasting future values in time series data, including stock prices, energy demand, weather patterns, and sensor readings. LSTMs are particularly popular for time series tasks because of their ability to capture both short-term and long-term patterns. However, more recent approaches using Transformers and specialized architectures (such as N-BEATS and Temporal Fusion Transformers) have shown competitive or superior performance on many time series benchmarks.

### Other applications

Additional applications of RNNs include:

- **Text generation:** Character-level and word-level RNNs can generate text that mimics the style of a training corpus. [Andrej Karpathy](/wiki/andrej_karpathy)'s 2015 blog post "The Unreasonable Effectiveness of Recurrent Neural Networks" popularized this application [13].
- **Music generation:** RNNs can learn patterns in musical sequences and produce new compositions. Google's Magenta project used LSTM networks for music generation.
- **Handwriting recognition and generation:** LSTMs have been used for both recognizing handwritten text and generating realistic handwriting.
- **Video analysis:** RNNs combined with [convolutional neural networks](/wiki/convolutional_neural_network) have been used for video captioning, action recognition, and video prediction.
- **[Anomaly detection](/wiki/anomaly_detection):** RNNs can learn the normal patterns in sequential data and flag deviations as anomalies, useful in network security and industrial monitoring.

## Historical importance in NLP

From roughly 2013 to 2018, RNNs (particularly LSTMs and BiLSTMs) were the dominant architecture for most NLP tasks. Key milestones during this period include:

- **[Word embeddings](/wiki/word_embedding):** While not RNN-specific, the development of [Word2Vec](/wiki/word2vec) (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) provided dense input representations that greatly improved RNN performance.
- **Neural machine translation:** The seq2seq model with attention (2014-2015) made neural approaches competitive with and then superior to statistical machine translation.
- **ELMo (2018):** [Embeddings](/wiki/embeddings) from Language Models, developed by Peters et al. at the [Allen Institute for AI](/wiki/ai2), used a bidirectional LSTM to produce contextualized word representations [16]. ELMo demonstrated that pre-trained language model features could dramatically improve downstream NLP tasks, foreshadowing the success of BERT and GPT [16].
- **ULMFiT (2018):** Howard and Ruder proposed Universal Language Model [Fine-tuning](/wiki/fine_tuning), which demonstrated that pre-training an LSTM-based language model on a large corpus and then fine-tuning it on a specific task could achieve strong performance with limited labeled data [17]. This approach was a direct precursor to the fine-tuning paradigm popularized by BERT.

RNNs also played a central role in early neural approaches to parsing, text classification, question answering, and dialogue systems.

## How do RNNs differ from transformers?

The introduction of the transformer architecture by Vaswani et al. in 2017 ("Attention Is All You Need") fundamentally shifted the landscape of sequence modeling [14]. Transformers replaced the recurrent computation of RNNs with self-attention, which computes relationships between all positions in a sequence simultaneously [14].

| Aspect | RNNs | Transformers |
|---|---|---|
| Processing order | Sequential (one step at a time) | Parallel (all positions at once) |
| Training parallelism | Limited by sequential dependency | Fully parallelizable on GPUs/TPUs |
| Long-range dependencies | Difficult (vanishing gradients, even with LSTM) | Handled natively by self-attention |
| Computational complexity per layer | O(T * d^2) | O(T^2 * d) |
| Memory at inference | O(d) constant state | O(T * d) grows with context length (KV cache) |
| Scalability | Hard to scale beyond ~1B parameters | Scales to hundreds of billions of parameters |
| Inductive bias | Strong sequential bias | No inherent sequential bias (uses positional encoding) |

The key advantage of transformers is **parallelization during training**. RNNs must process each token sequentially because computing h_t requires h_{t-1}. Transformers compute attention over all tokens in parallel, enabling efficient use of modern GPU hardware and making it practical to train on far larger datasets with far larger models. This parallelization advantage is the primary reason transformers enabled the jump from hundreds-of-millions-parameter models to trillion-parameter models. [Self-attention](/wiki/self_attention) also connects every position in the sequence directly to every other position, avoiding the long gradient paths that make it difficult for RNNs to learn distant dependencies.

By 2019, Transformer-based models had surpassed RNNs on virtually all major NLP benchmarks. Large language models (LLMs) such as [GPT-2](/wiki/gpt-2), [GPT-3](/wiki/gpt-3), and BERT are all based on the Transformer architecture. In speech recognition and machine translation, Transformers also gradually replaced RNN-based systems.

However, RNNs retain an advantage at inference time for autoregressive generation: they maintain a fixed-size hidden state, giving them O(1) memory per step, whereas transformers must store and attend over a key-value cache that grows linearly with context length.

## The recurrent revival: state space models and beyond

Despite the dominance of transformers, the inference-time efficiency of recurrent models has motivated a resurgence of interest in recurrent-style architectures.

### Structured state space models (S4)

The S4 model (Gu et al., 2021) introduced structured state space models that use linear recurrence with carefully parameterized state matrices [18]. S4 resolved the gradient stability issues of classical RNNs by leveraging HiPPO (High-order Polynomial Projection Operators) initialization, enabling stable modeling of sequences with tens of thousands of steps [18].

### Mamba

[Mamba](/wiki/mamba) (Gu and Dao, 2023) built on S4 by introducing **selective state spaces**, where the state transition matrices B and C are conditioned on the current input token. This gives the model content-aware filtering, analogous to a gating mechanism. The authors identified that "a key weakness of such models is their inability to perform content-based reasoning," and addressed it with input-dependent state transitions [19]. Mamba achieves linear-time complexity O(T), 5x higher inference throughput than comparably sized transformers, and strong performance across language, audio, and genomics [19]. The Mamba-3B model outperformed transformers of the same size on several benchmarks [19].

Mamba-2 (2024) introduced the State Space Duality (SSD) framework, proving mathematically that SSMs and attention are dual representations of the same underlying computation on structured matrices.

### RWKV

[RWKV](/wiki/rwkv) (Receptance Weighted Key Value, 2023) combines the parallelizable training of transformers with the efficient inference of RNNs [20]. It uses a linear attention mechanism and can be formulated as either a transformer (for parallel training) or an RNN (for efficient inference) [20]. RWKV models were scaled as large as 14 billion parameters, which the authors describe as "by far the largest dense RNN ever trained," performing on par with similarly sized transformers [20]. RWKV-7 "Goose" offers linear-time complexity, constant memory at inference, and competitive performance with full-attention models in the 0.7B to 1.5B parameter range.

### xLSTM

Sepp Hochreiter, co-inventor of the original LSTM, returned to the architecture with xLSTM (2024) [21]. xLSTM incorporates exponential gating, state expansion, and normalization techniques, offering two specialized modules: sLSTM (scalar memory) and mLSTM (matrix memory) [21]. While still in the research phase, xLSTM demonstrates that the core LSTM principles remain viable when modernized.

### Linear attention models

Various approaches have also emerged that replace softmax attention with linear approximations, recovering RNN-like recurrence during inference. These developments suggest that the boundary between recurrent and attention-based models is blurring, with hybrid and linear-recurrent architectures seeking the best of both paradigms.

## Notable implementations and frameworks

RNNs are supported by all major [deep learning](/wiki/deep_learning) frameworks:

| Framework | RNN support | Key features |
|---|---|---|
| [PyTorch](/wiki/pytorch) | `torch.nn.RNN`, `torch.nn.LSTM`, `torch.nn.GRU` | Dynamic computation graphs, cuDNN acceleration, packed sequences for variable-length inputs |
| [TensorFlow](/wiki/tensorflow) / [Keras](/wiki/keras) | `tf.keras.layers.LSTM`, `tf.keras.layers.GRU`, `tf.keras.layers.SimpleRNN` | Static and dynamic graphs, TensorFlow Lite for mobile deployment |
| JAX / Flax | Custom implementations via `jax.lax.scan` | Functional transformations, JIT compilation, TPU support |
| ONNX Runtime | LSTM and GRU operators | Cross-framework model deployment and optimization |

Historically, Theano (developed at the University of Montreal) was one of the first frameworks to support efficient RNN training and was used in much of the foundational RNN research, including the original seq2seq and attention papers. Torch (the Lua-based predecessor to PyTorch) was also widely used in early RNN research.

## Practical considerations

When working with RNNs, several practical considerations influence performance:

- **Gradient clipping** is almost always necessary to prevent exploding gradients. A common threshold is to clip the global gradient norm to 1.0 or 5.0.
- **Forget gate bias initialization:** For LSTMs, initializing the forget gate bias to 1.0 (so the gate starts open) significantly improves learning, as recommended by Jozefowicz et al. (2015).
- **[Dropout](/wiki/dropout):** Standard dropout applied to recurrent connections can hurt performance. Variational dropout (Gal and Ghahramani, 2016), which uses the same dropout mask at every time step, is preferred for [regularization](/wiki/regularization) in RNNs.
- **Learning rate schedules:** RNNs often benefit from learning rate warmup and decay schedules. Reducing the learning rate when validation performance plateaus is a common strategy.
- **Sequence length and batching:** Sequences of different lengths in a batch must be padded (with masking) or packed. Efficient batching strategies group sequences of similar length to minimize wasted computation on padding.
- **Hidden state size:** Typical hidden state dimensions range from 128 to 1024. Larger hidden states increase model capacity but also increase computational cost and the risk of overfitting.

## Historical timeline

| Year | Milestone | Researchers |
|---|---|---|
| 1982 | Hopfield network: recurrent network for associative memory | John Hopfield |
| 1986 | Jordan network: first simple recurrent network | Michael Jordan |
| 1989 | Teacher forcing introduced for RNN training | Ronald J. Williams, David Zipser |
| 1990 | Elman network: SRN with hidden-to-hidden recurrence | Jeffrey Elman |
| 1990 | Backpropagation through time formalized | Paul Werbos |
| 1991 | Vanishing gradient problem identified | Sepp Hochreiter (diploma thesis) |
| 1994 | Formal analysis of vanishing gradient problem in RNNs | Yoshua Bengio, Patrice Simard, Paolo Frasconi |
| 1997 | Long Short-Term Memory (LSTM) introduced | Sepp Hochreiter, Jurgen Schmidhuber |
| 1997 | Bidirectional RNNs proposed | Mike Schuster, Kuldip Paliwal |
| 2000 | LSTM forget gate and peephole connections added | Felix Gers, Jurgen Schmidhuber, Fred Cummins |
| 2006 | Connectionist Temporal Classification (CTC) | Alex Graves et al. |
| 2010 | RNN-based language models demonstrate large perplexity gains | Tomas Mikolov et al. |
| 2014 | GRU introduced; Seq2Seq models; Bahdanau attention | Cho et al.; Sutskever et al.; Bahdanau et al. |
| 2016 | Google Neural Machine Translation (GNMT) deployed | Google |
| 2017 | Transformer introduced, beginning RNN decline | Vaswani et al. |
| 2018 | ELMo and ULMFiT demonstrate LSTM-based pretraining | Peters et al.; Howard, Ruder |
| 2021 | S4 structured state space model | Albert Gu et al. |
| 2023 | Mamba: selective state spaces with linear-time inference; RWKV | Albert Gu, Tri Dao; Peng et al. |
| 2024 | xLSTM; Mamba-2; RWKV v6/v7 | Hochreiter et al.; Gu and Dao; Peng et al. |

## See also

- [Graph Machine Learning Models](/wiki/graph_machine_learning_models)

## References

1. Hopfield, J. J. (1982). "Neural networks and physical systems with emergent collective computational abilities." *Proceedings of the National Academy of Sciences*, 79(8), 2554-2558.
2. Jordan, M. I. (1986). "Serial order: A parallel distributed processing approach." *Technical Report 8604*, Institute for Cognitive Science, UC San Diego.
3. Elman, J. L. (1990). "Finding structure in time." *Cognitive Science*, 14(2), 179-211.
4. Williams, R. J., & Zipser, D. (1989). "A learning algorithm for continually running fully recurrent neural networks." *Neural Computation*, 1(2), 270-280.
5. Bengio, Y., Simard, P., & Frasconi, P. (1994). "Learning long-term dependencies with [gradient descent](/wiki/gradient_descent) is difficult." *IEEE Transactions on Neural Networks*, 5(2), 157-166.
6. Hochreiter, S., & Schmidhuber, J. (1997). "Long short-term memory." *Neural Computation*, 9(8), 1735-1780.
7. Schuster, M., & Paliwal, K. K. (1997). "Bidirectional recurrent neural networks." *IEEE Transactions on Signal Processing*, 45(11), 2673-2681.
8. Graves, A., Fernandez, S., Gomez, F., & Schmidhuber, J. (2006). "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks." *Proceedings of ICML*.
9. Mikolov, T., Karafiat, M., Burget, L., Cernocky, J., & Khudanpur, S. (2010). "Recurrent Neural Network Based Language Model." *Proceedings of Interspeech*.
10. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). "Learning phrase representations using RNN encoder-decoder for statistical machine translation." *Proceedings of EMNLP*.
11. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). "Sequence to sequence learning with neural networks." *Advances in Neural Information Processing Systems ([NeurIPS](/wiki/neurips))*, 27.
12. Bahdanau, D., Cho, K., & Bengio, Y. (2015). "Neural machine translation by jointly learning to align and translate." *Proceedings of ICLR 2015* (arXiv:1409.0473).
13. Karpathy, A. (2015). "The Unreasonable Effectiveness of Recurrent Neural Networks." Blog post.
14. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). "Attention is all you need." *Advances in Neural Information Processing Systems*, 30 (arXiv:1706.03762).
15. Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017). "LSTM: A Search Space Odyssey." *IEEE Transactions on Neural Networks and Learning Systems*, 28(10), 2222-2232.
16. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). "Deep Contextualized Word Representations." *Proceedings of NAACL*.
17. Howard, J., & Ruder, S. (2018). "Universal Language Model Fine-tuning for Text Classification." *Proceedings of ACL*.
18. Gu, A., Goel, K., & Re, C. (2021). "Efficiently modeling long sequences with structured state spaces." *Proceedings of ICLR 2022*.
19. Gu, A., & Dao, T. (2023). "Mamba: Linear-time sequence modeling with selective state spaces." *arXiv preprint arXiv:2312.00752*.
20. Peng, B., Alcaide, E., Anthony, Q., et al. (2023). "RWKV: Reinventing RNNs for the Transformer Era." *Findings of EMNLP 2023* (arXiv:2305.13048).
21. Beck, M., Poppel, K., Spanring, M., et al. (2024). "xLSTM: Extended Long Short-Term Memory." *Advances in Neural Information Processing Systems*, 37.
22. Semantic Scholar. "Long Short-Term Memory (Hochreiter & Schmidhuber, 1997)." Citation record listing the paper among the most cited neural network papers, with more than 80,000 citations. https://www.semanticscholar.org/paper/Long-Short-Term-Memory-Hochreiter-Schmidhuber/2e9d221c206e9503ceb452302d68d10e293f2a10
23. Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). "Learning to Forget: Continual Prediction with LSTM." *Neural Computation*, 12(10), 2451-2471.
24. Wu, Y., Schuster, M., Chen, Z., et al. (2016). "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation." *arXiv preprint arXiv:1609.08144*.