# Recurrent Neural Network

> Source: https://aiwiki.ai/wiki/recurrent_neural_network
> Updated: 2026-07-29
> Fact-checked: 2026-07-29
> Categories: Deep Learning, Machine Learning, Model Architecture, Neural Networks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Recurrent Neural Network." aiwiki.ai, 29 Jul 2026. https://aiwiki.ai/wiki/recurrent_neural_network
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

A recurrent neural network (RNN) is a [neural network](https://aiwiki.ai/wiki/neural_network) whose computation includes a state that is passed from one step to the next. For an ordered input sequence, the state at step `t` depends on the current input and an earlier state. Reusing the same transition parameters across steps lets one model operate on sequences of different lengths and makes the model sensitive to order. RNNs are therefore a major family of [sequence models](https://aiwiki.ai/wiki/sequence_model), not a single cell design.[1]

The simplest commonly taught RNN combines the current input with the previous hidden state, applies a nonlinear function, and produces an output. More elaborate recurrent architectures control this update with gates, process a sequence in both directions, or separate an encoder from an autoregressive decoder. Long short-term memory (LSTM) was designed to preserve error flow across longer intervals, while the gated recurrent unit (GRU) introduced reset and update gates that adaptively carry or replace hidden-state information.[9][13]

RNNs played a central role in language modeling, machine translation, speech recognition, and other sequential tasks. Attention-only Transformers later removed the recurrent dependency between positions and enabled more parallel training, but recurrence remains useful when computation must maintain a compact state, operate online, or scale linearly with the number of processed steps. The tradeoff is not simply "old versus new": recurrent, attention-based, convolutional, and state-space designs expose different dependencies, memory costs, and inductive biases.[12][20][21]

## Core computation

### A simple recurrent layer

Let `x_t` be an input vector, `h_t` a hidden-state vector, and `y_t` an output at step `t`. A simple, or vanilla, recurrent layer can be written as:

```
a_t = W_xh x_t + W_hh h_(t-1) + b_h
h_t = phi(a_t)
o_t = W_hy h_t + b_y
y_t = g(o_t)
```

`W_xh`, `W_hh`, and `W_hy` are learned matrices. The same matrices are used at every step. `phi` is an [activation function](https://aiwiki.ai/wiki/activation_function), commonly `tanh` or ReLU in a simple RNN implementation. `g` depends on the task: it can be the identity for real-valued prediction, a sigmoid for independent binary outputs, or [softmax](https://aiwiki.ai/wiki/softmax) for a categorical distribution. Biases and the output projection are sometimes omitted from diagrams, but they are part of many implementations.[1][24]

The initial state `h_0` may be all zeros, a learned parameter, or a state supplied by another component. It is not generally a complete record of the past. It is a finite-dimensional, task-trained summary whose useful information depends on the transition, the objective, the data, and the available state size. Calling it "memory" describes its function, not a guarantee of lossless storage.

A [feedforward neural network](https://aiwiki.ai/wiki/feedforward_neural_network_ffn) maps an input through an acyclic computation. An RNN also becomes acyclic after it is unrolled for a finite sequence: each time step is represented as another copy of the transition, and copies share parameters. The unrolled graph makes temporal dependencies and gradient paths explicit while the compact recurrent notation describes the same computation for arbitrary sequence length.[1]

### Input and output arrangements

Recurrent layers can support several arrangements:

- **Sequence to sequence, aligned:** each input step produces an output step, as in frame-level labeling.
- **Sequence to one:** one final or pooled state is used for classification or regression.
- **One to sequence:** an initial input or state conditions an autoregressive generator.
- **Sequence to sequence, unaligned:** an encoder reads one sequence and a decoder produces another sequence whose length may differ.

These labels describe interfaces, not unique architectures. A sequence classifier may pool all hidden states instead of using only the final one. A decoder may receive its earlier output token, an embedding of that token, and a context from an encoder. A model may also emit several heads with different objectives.

### State, causality, and parameter sharing

In a causal left-to-right RNN, `h_t` can depend only on inputs through step `t`. This supports streaming: after processing a prefix, the model can retain `h_t` and discard earlier activations for inference, subject to any application-specific need to preserve raw inputs. Training is different because earlier activations may be required for gradient computation.

Parameter sharing gives an RNN a form of time-translation invariance: the transition rule does not change merely because an event occurs at a later position. It also creates a repeated dynamical system. Small changes can contract, persist, or expand as the state is transformed many times. This repeated transformation is the source of both the model's temporal expressiveness and several of its optimization difficulties.[1][8]

## Historical development

Recurrent neural computation predates the simple sequence RNN now used in textbooks. Hopfield's 1982 network used symmetric recurrent connections and an energy function to implement content-addressable associative memory. It is historically important, but its convergence toward stored patterns is different from processing an external sequence with one shared transition at successive steps.[2]

Jordan's 1986 technical report described a network with state units that received delayed feedback from output units. Elman's 1990 "Finding Structure in Time" instead copied hidden-unit activations into context units, which were fed back with the next external input. Elman used this architecture to study how internal representations could reflect both the current input and prior internal state. The two feedback patterns are often called Jordan and Elman networks.[3][4]

Backpropagation provides an efficient chain-rule procedure for computing gradients in layered networks. Rumelhart, Hinton, and Williams presented an influential 1986 formulation for learning internal representations. Williams and Zipser developed real-time recurrent learning (RTRL), an online gradient method for recurrent networks, and described teacher forcing. Werbos described backpropagation through time (BPTT), which applies ordinary backpropagation to a recurrent network unfolded across time.[5][6][7]

The repeated Jacobian products in recurrent training made long-term dependencies difficult to learn. Bengio, Simard, and Frasconi analyzed this problem in 1994, showing a tradeoff between robustly storing information and propagating useful gradients with standard gradient descent. Hochreiter and Schmidhuber's 1997 journal paper presented LSTM with protected paths for state and error flow. Schuster and Paliwal introduced bidirectional recurrent neural networks in the same year, allowing each output to use earlier and later context. Gers, Schmidhuber, and Cummins subsequently added the forget gate to LSTM for continual streams in which internal state must be reset selectively.[8][9][10][11]

In 2006, connectionist temporal classification (CTC) provided a differentiable objective for labeling unsegmented sequences without requiring a frame-level target alignment. In 2014, Cho and colleagues introduced the reset- and update-gated recurrent unit now known as the GRU. Deep LSTM encoder-decoder systems demonstrated direct sequence-to-sequence learning, while learned soft alignment allowed a decoder to use different encoder states rather than relying only on one fixed-length vector.[12][13][14][15]

The 2017 Transformer dispensed with recurrence and convolution in its main architecture, using attention to connect sequence positions. Later research revisited state-based sequence computation. S4 used structured state-space dynamics with efficient recurrent and convolutional formulations, Mamba made state-space parameters depend on the input and implemented a selective recurrent scan, and xLSTM proposed new recurrent memory structures and exponential gates.[20][21][22][23]

## Training recurrent networks

### Backpropagation through time

Training begins with a scalar [loss function](https://aiwiki.ai/wiki/loss_function) defined over some or all outputs. BPTT unrolls the recurrent computation, applies backpropagation from later losses through earlier states, and sums the contributions to each shared parameter. If the total loss is:

```
L = sum over t of L_t
```

then a recurrent parameter can affect many `L_t` terms both directly and through every later state reached from the step where it was used. Automatic differentiation performs this bookkeeping, but the conceptual graph is the unrolled recurrence.[1][7]

Full BPTT retains a gradient path across the complete training sequence. Its activation memory and backward work grow with the unrolled length. Truncated BPTT processes a long stream in windows. A state from one window can initialize the next, while the computation graph is detached at a boundary so gradients do not pass farther back. Truncation changes the optimization problem: dependencies longer than the gradient window can affect the forward state, but they receive no direct gradient through that boundary.

A truncation boundary is not necessarily a sequence boundary. Detaching state controls gradient history. Resetting state declares that the next example is independent of the previous one. Confusing them can either leak information across unrelated examples or erase legitimate continuity within one stream.

### Teacher forcing and autoregressive decoding

Some recurrent decoders consume a representation of the previous output. During training, teacher forcing supplies the actual previous target instead of the model's earlier prediction. This gives a supervised input at each step and permits direct token-level optimization. Williams and Zipser used the term in their recurrent-learning work.[6]

At inference time, the true next target is unavailable, so a generator conditions on its own earlier output. An early error can change the later input context and lead to further errors. Scheduled sampling was proposed as a curriculum that gradually substitutes model-generated inputs during training. It reported improvements in the authors' sequence-prediction experiments, but it is one proposed method, not a general guarantee, and teacher-forcing behavior is relevant only when earlier outputs are fed back as inputs.[19]

### Batching, padding, and state handling

Sequences in a batch commonly have different lengths. Padding lets them share a rectangular tensor, but padded positions must be excluded from the loss and, where necessary, from recurrent updates. Otherwise the model can learn from artificial symbols, final states may correspond to padding rather than data, and evaluation can be distorted.

Alternative implementations group examples by length, use packed or ragged representations, or update an active subset of the batch at each step. Each approach must preserve the mapping among examples, states, targets, and lengths. State carried between minibatches must be reordered consistently if examples are shuffled.

Stateful training is appropriate only when each carried state is matched to a continuation of the same ordered stream. For independent records, each record needs a defined initial state. For grouped time series, state should not cross entity boundaries. For evaluation, state-reset rules must match the intended deployment and be reported.

### Objectives and optimization

An RNN can be optimized with [stochastic gradient descent](https://aiwiki.ai/wiki/stochastic_gradient_descent_sgd) or one of its adaptive variants. The objective depends on the task. Categorical sequence prediction commonly uses [cross-entropy](https://aiwiki.ai/wiki/cross-entropy); regression may use a squared, absolute, probabilistic, or domain-specific loss; CTC sums over valid monotonic alignments between input frames and a shorter label sequence.[12]

Learning rate, initialization, batch construction, sequence length, state size, optimizer, precision, and regularization interact. There is no task-independent hidden size, dropout rate, clipping threshold, or truncation length. These values should be selected on training and validation evidence and recorded with the result.

## Gradient propagation and stability

For a general recurrent transition:

```
h_t = F(h_(t-1), x_t; theta)
```

the influence of an earlier state on a later state contains a product of Jacobian matrices:

```
d h_t / d h_k =
    J_t J_(t-1) ... J_(k+1)

where J_i = d h_i / d h_(i-1)
```

In a vanilla tanh RNN, each `J_i` includes both the recurrent weight matrix and derivatives of `tanh` evaluated at that step. The product is therefore time-dependent and direction-dependent. Some components can shrink while others grow. Reducing the explanation to the largest eigenvalue of one weight matrix omits activation saturation, non-normal matrices, changing states, and the orientation of the propagated signal.[8][16]

### Vanishing gradients

If relevant components of the Jacobian product repeatedly contract, gradients arriving at distant steps become very small. Early events then receive little learning signal from later losses. Saturated sigmoid or tanh units can intensify contraction because their derivatives are small. This is the [vanishing gradient problem](https://aiwiki.ai/wiki/vanishing_gradient_problem), but it is not identical to forgetting in the forward pass: a network can preserve a state poorly, propagate a gradient poorly, or do both.[8][16]

Shorter gradient paths, gated additive state updates, careful initialization, residual or skip connections, and normalization can help in particular architectures. Truncation reduces the maximum backward path but also removes direct learning signals beyond the window. No mitigation makes arbitrary long-term memory automatic.

### Exploding gradients

If components of the Jacobian product expand repeatedly, gradient norms can become very large. Updates may be unstable, numerical values can overflow, and training loss can spike. [Gradient clipping](https://aiwiki.ai/wiki/gradient_clipping) caps or rescales a gradient when it exceeds a chosen threshold. Pascanu, Mikolov, and Bengio proposed a norm-clipping strategy and analyzed why exploding gradients can correspond to steep regions of the objective.[16]

Clipping limits update magnitude; it does not recover a gradient component that has already vanished, correct mislabeled data, or ensure stable forward dynamics. A clipping threshold is a hyperparameter whose effect depends on the optimizer, batch, loss scale, and parameterization.

### Forward and backward stability

Forward state behavior and backward gradient behavior should be diagnosed separately. A bounded activation can prevent an activation value from diverging while its derivative still contracts. Conversely, a state trajectory can remain in a reasonable numeric range while gradients are ill-conditioned in some directions.

Useful diagnostics include gradient norms by layer and time, the frequency of clipping, activation and gate distributions, state norms, loss curves by sequence length, and performance stratified by dependency distance. These measurements are more informative than assuming that a gated cell has solved every long-range problem.

## Principal architectures

### Long short-term memory

[Long short-term memory](https://aiwiki.ai/wiki/lstm) separates a cell state `c_t` from the exposed hidden state `h_t`. The original 1997 architecture used a self-connected memory cell with input and output gates. Its fixed self-connection was intended to support constant error flow, while gates controlled writing and reading. The forget gate was not part of that original cell; Gers, Schmidhuber, and Cummins added it later so a network could learn to discard obsolete state during continual prediction.[9][11]

A common modern LSTM formulation is:

```
i_t = sigmoid(W_xi x_t + W_hi h_(t-1) + b_i)
f_t = sigmoid(W_xf x_t + W_hf h_(t-1) + b_f)
g_t = tanh(W_xg x_t + W_hg h_(t-1) + b_g)
o_t = sigmoid(W_xo x_t + W_ho h_(t-1) + b_o)

c_t = f_t * c_(t-1) + i_t * g_t
h_t = o_t * tanh(c_t)
```

`i_t`, `f_t`, and `o_t` are input, forget, and output gates. `g_t` is a candidate update. Multiplication is element-wise. The additive cell update creates a path whose derivative with respect to the previous cell includes the forget gate rather than requiring a new full matrix multiplication at every step. This can preserve gradients more effectively, but a sequence of small forget gates still contracts them, and gates can saturate.[25]

The term LSTM covers multiple variants. Some add peephole connections from the cell to gates, couple input and forget gates, project the hidden state, alter the output activation, or change the number and placement of biases. Greff and colleagues compared eight variants across 5,400 experimental runs on three tasks. None consistently improved on the standard LSTM in those experiments, and the forget gate and output activation appeared particularly important. The finding is bounded to that study and does not prove that one cell is best for all data.[17]

Initialization also matters. Jozefowicz, Zaremba, and Sutskever evaluated more than 10,000 recurrent architectures. In their experiments, initializing the LSTM forget-gate bias to a positive value closed the observed gap with the GRU on some, but not all, tasks. This supports validating architecture and initialization together rather than treating a cell name as a complete specification.[18]

### Gated recurrent unit

Cho and colleagues introduced a simpler gated unit with one hidden state and no separate cell state. Using their original convention:

```
r_t = sigmoid(W_r x_t + U_r h_(t-1))
z_t = sigmoid(W_z x_t + U_z h_(t-1))

h_tilde_t = tanh(W x_t + U (r_t * h_(t-1)))
h_t = z_t * h_(t-1) + (1 - z_t) * h_tilde_t
```

The reset gate `r_t` controls how much of the previous state contributes to the candidate. The update gate `z_t` interpolates between the previous state and the candidate. Under this equation, a value of `z_t` near 1 retains the previous state, while a value near 0 favors the candidate.[13]

Gate notation is not universal. Some descriptions define the complementary update variable, so their symbol near 1 favors the candidate instead. Implementations can also apply the reset gate before or after a recurrent affine transformation. These formulations are not necessarily numerically identical because biases and matrix multiplication interact with the reset. A reproducible report should identify the exact equations or library implementation, not only say "GRU."[26]

A GRU usually has fewer gates and no separate cell vector, which can reduce its parameter count relative to an LSTM with the same input and hidden dimensions. That does not establish a universal speed or quality advantage. Kernel fusion, sequence length, state size, device, batch shape, and optimization can dominate. Comparative studies have found task-dependent outcomes rather than a single winner.[17][18]

### Bidirectional recurrence

A [bidirectional recurrent neural network](https://aiwiki.ai/wiki/bidirectional) runs one recurrence from the start of a sequence and another from the end, then combines their states for each position. Schuster and Paliwal introduced the architecture so outputs could use both past and future context without imposing a fixed look-ahead window.[10]

Bidirectionality is appropriate when the complete sequence is available, such as offline tagging or transcription. It is not causal. A strictly streaming system cannot compute the backward state for a position until future input has arrived. Systems with limited look-ahead can use bounded context, but that is different from a full bidirectional pass.

### Deep and stacked RNNs

Recurrent layers can be stacked so the output sequence of one layer becomes the input sequence of the next. Depth across layers and recurrence across time are distinct axes. Residual connections, [dropout](https://aiwiki.ai/wiki/dropout), and [layer normalization](https://aiwiki.ai/wiki/layer_normalization) may be used, but their placement is part of the model definition. In PyTorch's built-in RNN, LSTM, and GRU modules, the dropout option applies between stacked recurrent layers and not after the final recurrent layer; it therefore has no between-layer effect in a one-layer module.[24][25][26]

Stacking increases representational capacity and the length of vertical gradient paths. It also increases computation at every step. Layer count, hidden dimensions, projections, normalization, residual paths, and dropout placement are part of the architecture and should be reported.

### Reservoir computing

Reservoir methods keep a recurrent dynamical system fixed or largely fixed and train a readout from its states. Jaeger's echo state network is a prominent example: a recurrent reservoir projects an input history into a high-dimensional state, while training is concentrated in the output connection. The approach shifts work away from BPTT, but useful behavior depends on reservoir dynamics, input scaling, state initialization, and regularization.[28]

Reservoir computing is related to RNNs through recurrent state, but its training regime differs from end-to-end learned recurrent networks. It should not be used as evidence that recurrent weights are generally unnecessary.

### Encoder-decoder recurrence and attention

In an RNN [sequence-to-sequence task](https://aiwiki.ai/wiki/sequence-to-sequence_task), an encoder processes an input sequence and a decoder generates an output sequence. Sutskever, Vinyals, and Le used a multilayer LSTM encoder to map a sentence to a fixed-dimensional vector and another LSTM to decode a translation. The model demonstrated that one end-to-end architecture could map variable-length inputs to variable-length outputs.[14]

A single fixed vector can become a bottleneck, especially as the input grows. Bahdanau, Cho, and Bengio introduced learned soft [attention](https://aiwiki.ai/wiki/attention) that computes a weighted context from encoder states for each decoder step. The decoder can then emphasize different source positions while producing different outputs. This attention mechanism was developed within a recurrent encoder-decoder; attention and recurrence are not mutually exclusive.[15]

## Applications and historical role

### Language

RNNs model ordered symbols by updating state as tokens arrive. A recurrent [language model](https://aiwiki.ai/wiki/language_model) estimates a distribution over a next token from earlier context. The same pattern supports sequence labeling, text classification, and generation. Word or subword inputs are typically mapped through a learned [word embedding](https://aiwiki.ai/wiki/word_embedding) or another embedding layer before entering the recurrence.

ELMo and ULMFiT are prominent examples of recurrent transfer-learning systems reported during the period in which Transformer architectures emerged. ELMo formed contextual word representations from a deep bidirectional LSTM language model and improved six NLP tasks in the reported experiments. ULMFiT adapted an AWD-LSTM language model to text classification through staged fine-tuning. These results establish a historical role in [natural language processing](https://aiwiki.ai/wiki/natural_language_processing), not current leadership across every benchmark.[29][30]

Recurrent encoder-decoders also helped establish neural [machine translation](https://aiwiki.ai/wiki/machine_translation). Their fixed-vector limitation motivated attention, and their autoregressive decoders illustrated both teacher forcing and the problem of accumulated generation errors.[14][15][19]

### Speech and unsegmented labeling

Acoustic observations arrive as long frame sequences, while target transcripts contain fewer symbols and usually lack an exact frame-to-symbol alignment. CTC defines a blank symbol and sums the probabilities of all frame-level paths that collapse to the target label sequence. Graves and colleagues trained recurrent networks with this objective and evaluated it on speech data.[12]

CTC assumes a monotonic order between inputs and labels. It does not by itself define the acoustic encoder or a language model. Modern systems may combine CTC with recurrent, convolutional, attention-based, or hybrid encoders. The original work supports RNN use in [speech recognition](https://aiwiki.ai/wiki/speech_recognition), but not a claim that RNNs are required for CTC.

### Time series and control

RNNs have been studied for [time series](https://aiwiki.ai/wiki/time_series), dynamical systems, and control applications.[7][28] A valid application claim still requires a specified forecasting horizon, sampling process, covariates, missing-data treatment, and baseline. Randomly splitting adjacent observations can leak future information and exaggerate performance.

For online control or monitoring, a recurrent state can summarize a stream with fixed state memory per step. That compactness is useful, but an RNN does not automatically model physical constraints, irregular time gaps, interventions, or uncertainty. Those properties require appropriate inputs, objectives, or model structure.

## Comparison with Transformers

[Transformers](https://aiwiki.ai/wiki/transformers) replace the step-to-step hidden-state chain in their main layers with attention and position information. In the original Transformer comparison, a recurrent layer required `O(n)` sequential operations for a sequence of length `n`, while a full [self-attention](https://aiwiki.ai/wiki/self_attention) layer connected positions with `O(1)` sequential depth. This permits more parallel computation during training. Full attention in that formulation uses `O(n^2)` pairwise interactions, whereas a simple recurrent scan has work that grows linearly with sequence length for fixed dimensions.[20]

These asymptotic descriptions do not determine wall-clock performance alone. The recurrent transition must wait for the previous state, but its per-step state can be compact. Attention can process training positions in parallel, but it materializes or computes interactions among positions and autoregressive decoding usually maintains a cache of earlier keys and values. Device utilization, memory bandwidth, kernel implementation, batch size, dimensions, and sequence length all matter.

The maximum computational path between distant positions is also different. A recurrent signal crosses one transition per intervening step. Full attention can connect two positions within one layer. This shorter path was one motivation for attention, but it does not mean an attention model always learns every dependency or that a recurrent model cannot do so.

For streaming inference, a causal RNN updates a fixed-size state and can emit an output immediately. A Transformer can also stream causally, but its standard attention cache grows with the processed context unless it uses a bounded, compressed, recurrent, or otherwise modified memory. Conversely, a bidirectional RNN cannot operate with zero look-ahead, and a recurrent state can discard details that direct attention could revisit.

Architecture choice should follow the data and deployment constraint. Important questions include whether the task is causal, whether the complete sequence is available, how long the relevant dependencies are, whether exact earlier details must remain addressable, how training is parallelized, and what memory is available at inference.

## Related state-space and recurrent developments

A [state-space model](https://aiwiki.ai/wiki/state_space_model) describes how a latent state evolves and generates observations. Neural state-space sequence models connect that tradition to learned deep architectures. Some can be evaluated as a recurrence for streaming and transformed into a convolutional computation for parallel training when their dynamics have suitable structure.

S4 parameterizes a structured linear state-space layer and derives an efficient kernel computation. It is related to an RNN because it carries state, but its structured linear dynamics and convolutional dual are not the same as a vanilla nonlinear recurrent cell.[21]

[Mamba](https://aiwiki.ai/wiki/mamba) makes selected state-space parameters functions of the current input, allowing the model to control what information is propagated or forgotten. Because this input dependence breaks the simple fixed convolution used by earlier state-space models, the authors designed a hardware-aware parallel scan in recurrent mode. Benchmark results from the paper are evidence for its configurations, not a universal replacement claim.[22]

[xLSTM](https://aiwiki.ai/wiki/xlstm) revisits LSTM with exponential gating and two memory designs. The sLSTM variant retains recurrent memory mixing, while mLSTM uses a matrix memory and can be reformulated for parallel computation because it omits hidden-to-hidden memory mixing in that component. xLSTM is a later recurrent architecture, not the equation implemented by a standard LSTM layer.[23]

These designs illustrate a broader continuum. A model can carry a compact state, expose content-addressable interactions, admit a parallel training form, or combine these properties. Labels such as recurrent, attention-based, and state-space identify important structure, but hybrid models can cross the boundaries.

## Implementation

### Framework interfaces

[PyTorch](https://aiwiki.ai/wiki/pytorch) provides separate `RNN`, `LSTM`, and `GRU` modules. Each returns an output sequence and final state. LSTM returns both a hidden state and a cell state; simple RNN and GRU return a hidden state. Options include multiple layers, bidirectionality, dropout between stacked layers, and batch-first input layout. Exact tensor shapes depend on these options and on whether a projection is used.[24][25][26]

[TensorFlow](https://aiwiki.ai/wiki/tensorflow) exposes recurrent cells and a [Keras](https://aiwiki.ai/wiki/keras) `RNN` wrapper that can return either the last output or the full output sequence and can return state. Its stateful mode reuses final states as initial states for corresponding batch entries. That correspondence is a data contract: changing example order without resetting or reassigning state produces incorrect histories.[27]

Frameworks may fuse kernels or use backend-specific algorithms. Mathematical equivalence does not ensure bitwise equality across devices, precisions, or library versions. A deployed recurrent model should record framework and backend versions alongside its weights.

### Input and output shapes

A recurrent input is commonly represented as `batch x time x features` or `time x batch x features`. The hidden state commonly includes layer and direction axes. Shape conventions are frequent sources of silent errors because transposing batch and time can still produce valid matrix dimensions while changing the computation.

For variable lengths, the final tensor position is not necessarily the final real event. A sequence classifier should select states using actual lengths or an explicitly masked pooling rule. For bidirectional layers, the terminal forward and backward states correspond to different ends of the original sequence and must be combined according to the library's documented ordering.

### Initialization and regularization

Initial recurrent weights affect forward dynamics and gradient propagation. Orthogonal or identity-related initializations are sometimes used for recurrent matrices, while gate biases may receive architecture-specific initialization. The Jozefowicz study supports a positive LSTM forget bias in some tested settings, but it does not establish one numeric value for every task.[18]

Regularization can include weight decay, dropout, input noise, early stopping, state penalties, or recurrent-specific methods. The effect depends on placement. Dropout between stacked layers differs from dropping recurrent connections, and a framework's `dropout` argument may not apply to a single-layer recurrence. Documentation and experiments should determine the actual graph.[24][25][26]

### Export and serving

Serving a causal RNN requires an explicit state lifecycle. The system must decide when a session begins, when state is carried, how it is stored, when it expires, and how it is reset. Mixing states among users or streams is both a correctness and privacy failure.

Batching stateful requests requires gathering the correct state for each stream and scattering updated state back afterward. Retries and duplicated messages can apply an event twice unless the surrounding protocol is idempotent. Model export should include state shapes, data types, initial-state behavior, and any length or mask inputs.

Quantized or reduced-precision deployments should be evaluated end to end on sequences that match deployment lengths and state-reset behavior. A one-step operator comparison does not by itself measure the behavior of a transition applied repeatedly.

## Evaluation and reproducibility

An RNN should be evaluated on the task it is intended to solve and on sequence conditions that reflect deployment. Classification needs class-aware metrics and subgroup analysis. Language modeling commonly reports [perplexity](https://aiwiki.ai/wiki/perplexity) derived from token log likelihood, but comparisons require the same tokenization and evaluation corpus. Forecasting needs chronological splits and horizon-specific error. Speech systems need sequence-level measures such as word or character error rate.

Aggregate scores can hide dependence on length. Evaluation should be stratified by sequence length, dependency distance, missingness, and relevant subgroups. For streaming systems, latency, state memory, throughput, and recovery after state reset or packet loss may matter as much as a predictive score.

A reproducible report should state:

- the exact cell equations or framework layer and version;
- input representation, feature scaling, vocabulary, and embedding;
- layer count, hidden and cell dimensions, directions, projections, and output heads;
- initial-state and reset policy;
- padding, packing, masking, and sequence-boundary rules;
- objective, optimizer, learning-rate schedule, gradient-clipping rule, precision, and stopping criterion;
- truncation window and whether state is detached, carried, or reset at each boundary;
- teacher-forcing or generated-input schedule, when applicable;
- regularization and its exact placement;
- random seeds and the number of runs;
- data splits, leakage controls, metrics, and results by sequence length;
- hardware, software, and serving-state lifecycle.

"An LSTM was trained" or "a GRU was faster" omits most of the variables needed to reproduce or interpret a result.

## Limitations

### Sequential dependency

A conventional recurrence cannot compute step `t` until it has computed step `t - 1`. This limits parallelism across time during both a forward pass and autoregressive generation. Batching independent sequences and fusing the cell operations improve hardware use, but they do not remove the dependency chain.[20]

### Finite-state bottleneck

A fixed-width hidden state must compress all information needed from the prefix. Details not preserved in that state cannot be recovered later unless the architecture retains external memory, encoder states, or another addressable representation. Increasing state size raises capacity and cost but does not guarantee that training will learn the desired retention policy.

### Long gradient paths

Dependencies separated by many steps create long products of Jacobians. Gating and additive paths can improve propagation, but gradients can still vanish or explode and useful state can still be overwritten. Clipping addresses excessive magnitude, not missing gradient information.[8][16]

### Truncation bias

Truncated BPTT saves memory and bounds backward work, but its gradient omits dependencies across truncation boundaries. Carrying the forward state does not restore those omitted derivatives. A model may appear to process a long stream while being trained only on local credit assignment.

### Exposure to generated history

An autoregressive decoder trained with teacher forcing sees correct histories during training but its own histories during inference. Errors can compound. Alternative training or decoding methods change the tradeoff, but no single schedule removes every distribution shift.[19]

### Noncausal context

Bidirectional RNNs use future observations. They can improve offline representations but are unsuitable without modification when outputs must be produced before future input arrives. Evaluation that accidentally includes future features or states is leakage, not legitimate bidirectionality.

### Architecture comparisons

Parameter count alone does not determine memory, latency, or accuracy. LSTM, GRU, simple recurrence, attention, and state-space layers have different operations and backend support. A fair comparison must align data, objective, parameter or compute budget, tuning effort, precision, hardware, and stopping rules.

### Interpretability

Gate values and hidden-state coordinates can be inspected, but they are not automatically human-readable explanations. A high forget gate or an attention-like diagnostic does not by itself establish causal reliance. Claims about what a state represents need controlled interventions, probes with appropriate baselines, or other task-specific evidence.

## See also

- [Machine Learning](https://aiwiki.ai/wiki/machine_learning)
- [Deep Learning](https://aiwiki.ai/wiki/deep_learning)
- [Autoregressive Model](https://aiwiki.ai/wiki/autoregressive_model)

## References

1. Goodfellow, I., Bengio, Y., and Courville, A. "Sequence Modeling: Recurrent and Recursive Nets." In *Deep Learning*. MIT Press, 2016. https://www.deeplearningbook.org/contents/rnn.html
2. Hopfield, J. J. "Neural Networks and Physical Systems with Emergent Collective Computational Abilities." *Proceedings of the National Academy of Sciences*, 1982. https://doi.org/10.1073/pnas.79.8.2554
3. Jordan, M. I. "Serial Order: A Parallel Distributed Processing Approach." Technical Report ICS-8604, Institute for Cognitive Science, University of California, San Diego, 1986. https://www.osti.gov/biblio/6910294
4. Elman, J. L. "Finding Structure in Time." *Cognitive Science*, 1990. https://doi.org/10.1207/S15516709COG1402_1
5. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. "Learning Representations by Back-Propagating Errors." *Nature*, 1986. https://doi.org/10.1038/323533a0
6. Williams, R. J., and Zipser, D. "A Learning Algorithm for Continually Running Fully Recurrent Neural Networks." *Neural Computation*, 1989. https://doi.org/10.1162/neco.1989.1.2.270
7. Werbos, P. J. "Backpropagation Through Time: What It Does and How to Do It." *Proceedings of the IEEE*, 1990. https://doi.org/10.1109/5.58337
8. Bengio, Y., Simard, P., and Frasconi, P. "Learning Long-Term Dependencies with Gradient Descent is Difficult." *IEEE Transactions on Neural Networks*, 1994. https://doi.org/10.1109/72.279181
9. Hochreiter, S., and Schmidhuber, J. "Long Short-Term Memory." *Neural Computation*, 1997. https://doi.org/10.1162/neco.1997.9.8.1735
10. Schuster, M., and Paliwal, K. K. "Bidirectional Recurrent Neural Networks." *IEEE Transactions on Signal Processing*, 1997. https://doi.org/10.1109/78.650093
11. Gers, F. A., Schmidhuber, J., and Cummins, F. "Learning to Forget: Continual Prediction with LSTM." *Neural Computation*, 2000. https://doi.org/10.1162/089976600300015015
12. Graves, A., Fernandez, S., Gomez, F., and Schmidhuber, J. "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks." Proceedings of ICML, 2006. https://doi.org/10.1145/1143844.1143891
13. Cho, K., et al. "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation." Proceedings of EMNLP, 2014. https://aclanthology.org/D14-1179/
14. Sutskever, I., Vinyals, O., and Le, Q. V. "Sequence to Sequence Learning with Neural Networks." Advances in Neural Information Processing Systems 27, 2014. https://proceedings.neurips.cc/paper_files/paper/2014/hash/5a18e133cbf9f257297f410bb7eca942-Abstract.html
15. Bahdanau, D., Cho, K., and Bengio, Y. "Neural Machine Translation by Jointly Learning to Align and Translate." ICLR, 2015. https://arxiv.org/abs/1409.0473
16. Pascanu, R., Mikolov, T., and Bengio, Y. "On the Difficulty of Training Recurrent Neural Networks." Proceedings of ICML, 2013. https://proceedings.mlr.press/v28/pascanu13.html
17. Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., and Schmidhuber, J. "LSTM: A Search Space Odyssey." *IEEE Transactions on Neural Networks and Learning Systems*, 2017. https://doi.org/10.1109/TNNLS.2016.2582924
18. Jozefowicz, R., Zaremba, W., and Sutskever, I. "An Empirical Exploration of Recurrent Network Architectures." Proceedings of ICML, 2015. https://proceedings.mlr.press/v37/jozefowicz15.html
19. Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. "Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks." Advances in Neural Information Processing Systems 28, 2015. https://proceedings.neurips.cc/paper/2015/hash/e995f98d56967d946471af29d7bf99f1-Abstract.html
20. Vaswani, A., et al. "Attention Is All You Need." Advances in Neural Information Processing Systems 30, 2017. https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
21. Gu, A., Goel, K., and Re, C. "Efficiently Modeling Long Sequences with Structured State Spaces." ICLR, 2022. https://openreview.net/forum?id=uYLFoz1vlAC
22. Gu, A., and Dao, T. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." Conference on Language Modeling, 2024. https://openreview.net/forum?id=tEYskw1VY2
23. Beck, M., et al. "xLSTM: Extended Long Short-Term Memory." Advances in Neural Information Processing Systems 37, 2024. https://proceedings.neurips.cc/paper_files/paper/2024/hash/c2ce2f2701c10a2b2f2ea0bfa43cfaa3-Abstract-Conference.html
24. PyTorch. "RNN." PyTorch documentation. https://docs.pytorch.org/docs/stable/generated/torch.nn.RNN.html
25. PyTorch. "LSTM." PyTorch documentation. https://docs.pytorch.org/docs/stable/generated/torch.nn.LSTM.html
26. PyTorch. "GRU." PyTorch documentation. https://docs.pytorch.org/docs/stable/generated/torch.nn.GRU.html
27. TensorFlow. "tf.keras.layers.RNN." TensorFlow API documentation. https://www.tensorflow.org/api_docs/python/tf/keras/layers/RNN
28. Jaeger, H. "The 'Echo State' Approach to Analysing and Training Recurrent Neural Networks." German National Research Center for Information Technology, 2001. https://publica.fraunhofer.de/entities/publication/7d4a7eec-a22c-4df0-903d-93f9cd5aca02
29. Peters, M. E., et al. "Deep Contextualized Word Representations." Proceedings of NAACL-HLT, 2018. https://aclanthology.org/N18-1202/
30. Howard, J., and Ruder, S. "Universal Language Model Fine-tuning for Text Classification." Proceedings of ACL, 2018. https://aclanthology.org/P18-1031/