# BiDAF

> Source: https://aiwiki.ai/wiki/bidaf
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: AI Models, Deep Learning, Model Architecture, Natural Language Processing
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "BiDAF." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/bidaf
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**BiDAF** (Bi-Directional Attention Flow) is a [neural network](https://aiwiki.ai/wiki/neural_network) architecture for machine reading comprehension, the task of answering a question about a given context paragraph. It was introduced by Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi and Hannaneh Hajishirzi of the University of Washington and the [Allen Institute for AI](https://aiwiki.ai/wiki/allen_institute_for_ai) in the paper "Bidirectional Attention Flow for Machine Comprehension", posted to arXiv on 5 November 2016 and published as a conference paper at [ICLR](https://aiwiki.ai/wiki/iclr) 2017.[1] The model computes [attention](https://aiwiki.ai/wiki/attention) in two directions, from the context to the question and from the question to the context, and passes the attended vectors for every context word on to later layers instead of compressing the paragraph into a single fixed-size vector.[1] At submission it led the [SQuAD](https://aiwiki.ai/wiki/squad) test leaderboard, with an ensemble scoring 73.3 exact match (EM) and 81.1 F1, and it set new single-model results on the CNN/DailyMail cloze test.[1]

BiDAF became one of the standard reference models of the pre-[BERT](https://aiwiki.ai/wiki/bert) era of [question answering](https://aiwiki.ai/wiki/question_answering). AllenNLP included a reference implementation of it,[2] the SQuAD 2.0 paper used a BiDAF variant as one of its baselines,[3] and adversarial studies such as Jia and Liang's distracting-sentence attack and the "Beat the AI" dataset used it as a model under test.[4][5]

## Background

In machine comprehension (MC) as framed by the BiDAF paper, a system receives a context paragraph and a query and must find the answer in the paragraph.[1] The authors credit large datasets for much of the progress in the field: early datasets such as MCTest were too small to train end-to-end neural models, the CNN/DailyMail and Children's Book Test cloze datasets made deep architectures practical, and SQuAD, released by Rajpurkar et al. in 2016, added more than 100,000 human-written questions over Wikipedia articles.[1]

The paper sorted earlier neural MC models into three groups by how they used attention:[1]

| Group | How attention is used | Examples cited by the paper |
|---|---|---|
| Dynamic attention | Attention weights are updated at each step from the query, the context and the previous attention | Hermann et al. (2015), Chen et al. (2016), Match-LSTM (Wang and Jiang, 2016) |
| Attention computed once | Weights are computed once and fed to the output layer | Kadlec et al. (2016), attention-over-attention (Cui et al., 2016) |
| Multi-hop attention | Attention between query and context is repeated over several layers, as in memory networks | Sordoni et al. (2016), Dhingra et al. (2016), ReasoNet (Shen et al., 2016) |

According to the authors, such mechanisms typically had one or more of three properties: they used attention to summarize the context into a fixed-size vector, the attention at one time step depended on the attention at the previous step, and attention ran in one direction only (the query attending to the context).[1] BiDAF was designed to avoid all three.[1]

## Architecture

BiDAF is a hierarchical, multi-stage model with six layers. The first three are applied to both the context and the query, and the paper compares this staged feature computation to the multi-stage features of [convolutional neural networks](https://aiwiki.ai/wiki/convolutional_neural_network) in computer vision.[1]

| Layer | Function |
|---|---|
| 1. Character embedding | Maps each word to a vector with a character-level CNN, max-pooled over the word |
| 2. Word embedding | Maps each word to a fixed pre-trained [GloVe](https://aiwiki.ai/wiki/glove) vector |
| 3. Contextual embedding | Bidirectional [LSTM](https://aiwiki.ai/wiki/lstm) over the combined embeddings, adding context from surrounding words |
| 4. Attention flow | Couples query and context; produces query-aware feature vectors for each context word |
| 5. Modeling | Two-layer bidirectional LSTM over the query-aware context |
| 6. Output | Task-specific; for SQuAD, predicts the start and end of the answer span |

Source: the BiDAF paper, section 2.[1]

### Embedding layers

The character embedding layer follows Kim (2014): characters are embedded as vectors, passed through a 1D CNN, and the CNN outputs are max-pooled over the whole word to give one fixed-size vector per word.[1] The word embedding layer looks up fixed pre-trained GloVe vectors.[1] The character and word vectors are concatenated and passed through a two-layer highway network, giving a matrix $X \in \mathbb{R}^{d \times T}$ for the $T$ context words and $Q \in \mathbb{R}^{d \times J}$ for the $J$ query words.[1] In the ablation study, removing either embedding hurt accuracy; the authors suggested that word embeddings capture the meaning of whole words while character embeddings help with rare and out-of-vocabulary words.[1]

The contextual embedding layer runs an LSTM in each direction and concatenates their outputs, producing $H \in \mathbb{R}^{2d \times T}$ for the context and $U \in \mathbb{R}^{2d \times J}$ for the query.[1]

### Attention flow layer

The attention flow layer is the model's central component. Both attention directions are derived from one shared similarity matrix $S \in \mathbb{R}^{T \times J}$, in which $S_{tj}$ scores the similarity between context word $t$ and query word $j$:[1]

$$S_{tj} = \alpha(H_{:t}, U_{:j}), \qquad \alpha(h, u) = w_{(S)}^\top [h; u; h \circ u]$$

Here $w_{(S)} \in \mathbb{R}^{6d}$ is a trainable weight vector, $\circ$ is elementwise multiplication and $[;]$ is concatenation.[1] The QANet paper later called this the trilinear similarity function.[6]

**Context-to-query (C2Q) attention** asks which query words matter most for each context word. For context word $t$, the weights over the query are $a_t = \mathrm{softmax}(S_{t:}) \in \mathbb{R}^J$, and the attended query vector is $\tilde{U}_{:t} = \sum_j a_{tj} U_{:j}$.[1]

**Query-to-context (Q2C) attention** asks which context words are most similar to some query word and are therefore likely to matter for the answer. The weights are $b = \mathrm{softmax}(\max_{col}(S)) \in \mathbb{R}^T$, which takes each context word's largest similarity with any query word. The attended context vector $\tilde{h} = \sum_t b_t H_{:t}$ is then tiled $T$ times to form $\tilde{H} \in \mathbb{R}^{2d \times T}$.[1]

The layer's output $G$ combines the contextual embeddings with both attention results for every context word:[1]

$$G_{:t} = \beta(H_{:t}, \tilde{U}_{:t}, \tilde{H}_{:t}), \qquad \beta(h, \tilde{u}, \tilde{h}) = [h; \tilde{u}; h \circ \tilde{u}; h \circ \tilde{h}] \in \mathbb{R}^{8d}$$

Two design choices separate this layer from earlier work. First, nothing is summarized: an attended vector is computed for every time step and flows, together with the earlier representations, into the modeling layer, which the authors argued reduces the information lost through early summarization.[1] Second, the attention is "memory-less": the attention at each time step depends only on the query and the current context word, not on the attention at the previous step.[1] The authors hypothesized that this divides the labor between layers, with the attention layer learning query-context interactions and the modeling layer learning interactions within the query-aware context, and that it keeps errors at earlier time steps from affecting later ones.[1]

### Modeling and output layers

The modeling layer is a two-layer bidirectional LSTM with output size $d$ in each direction, run over $G$ to give $M \in \mathbb{R}^{2d \times T}$. Unlike the contextual embedding layer, it captures interactions among context words conditioned on the query.[1]

The output layer is the only task-specific part; the paper describes the architecture as modular, so the output layer can be swapped while the rest stays the same.[1] For SQuAD, where every answer is a span of the paragraph, the model predicts start and end positions:[1]

$$p^1 = \mathrm{softmax}(w_{(p^1)}^\top [G; M]), \qquad p^2 = \mathrm{softmax}(w_{(p^2)}^\top [G; M^2])$$

where $M^2$ is the output of a further bidirectional LSTM run over $M$, and $w_{(p^1)}, w_{(p^2)} \in \mathbb{R}^{10d}$ are trainable.[1] Training minimizes the negative log probabilities of the true start and end indices averaged over the $N$ training examples:[1]

$$L(\theta) = -\frac{1}{N} \sum_{i=1}^{N} \left[ \log p^1_{y^1_i} + \log p^2_{y^2_i} \right]$$

At test time the model returns the span $(k, l)$ with $k \le l$ that maximizes $p^1_k p^2_l$, found in linear time with dynamic programming.[1]

### Training setup

For SQuAD the authors tokenized text with a PTB-style regular-expression tokenizer and used 100 character-CNN filters of width 5 and a hidden size $d$ of 100, giving about 2.6 million parameters.[1] The model was trained with AdaDelta (minibatch 60, initial learning rate 0.5) for 12 epochs, with a [dropout](https://aiwiki.ai/wiki/dropout) rate of 0.2 on the CNN, all LSTM layers and the final linear layer, and with exponential moving averages of the weights (decay 0.999) used at test time.[1] Training took about 20 hours on a single NVIDIA Titan X [GPU](https://aiwiki.ai/wiki/gpu).[1] The ensemble combined 12 training runs of the same architecture and chose, for each question, the answer with the highest summed confidence.[1]

## Results

### SQuAD

On the hidden SQuAD test set, as recorded on the leaderboard on 6 December 2016, BiDAF's ensemble outperformed all previous entries:[1]

| Model | Single EM | Single F1 | Ensemble EM | Ensemble F1 |
|---|---|---|---|---|
| Logistic regression baseline | 40.4 | 51.0 | - | - |
| Dynamic Chunk Reader | 62.5 | 71.0 | - | - |
| Fine-Grained Gating | 62.5 | 73.3 | - | - |
| Match-LSTM | 64.7 | 73.7 | 67.9 | 77.0 |
| Multi-Perspective Matching | 65.5 | 75.1 | 68.2 | 77.2 |
| Dynamic Coattention Networks | 66.2 | 75.9 | 71.6 | 80.4 |
| R-Net | 68.4 | 77.5 | 72.1 | 79.7 |
| BiDAF | 68.0 | 77.3 | 73.3 | 81.1 |

Source: BiDAF paper, Table 1a. EM is exact match; the [F1 score](https://aiwiki.ai/wiki/f1_score) is a softer metric that gives partial credit.[1]

As the table shows, R-Net had a slightly higher single-model score on that date, while BiDAF's ensemble had the highest EM and F1 overall.[1] On the development set BiDAF scored 67.7 EM and 77.3 F1 as a single model and 72.6 EM and 80.7 F1 as an ensemble.[1][7]

### Ablations

The paper's ablation study on the SQuAD development set (single runs) measured the contribution of each component:[1]

| Variant | EM | F1 |
|---|---|---|
| No character embedding | 65.0 | 75.4 |
| No word embedding | 55.5 | 66.8 |
| No C2Q attention | 57.2 | 67.7 |
| No Q2C attention | 63.6 | 73.7 |
| Dynamic attention | 63.5 | 73.6 |
| BiDAF (single) | 67.7 | 77.3 |

Removing context-to-query attention (replacing the attended query vectors with the average of the query's contextual embeddings) cut EM from 67.7 to 57.2 and F1 from 77.3 to 67.7; the paper describes this as a drop of more than 10 points on both metrics, though by its own table the F1 drop is 9.6 points.[1] The "dynamic attention" row replaces the precomputed attention with attention computed inside the modeling layer's LSTM, as in Bahdanau et al. and Match-LSTM; the simpler static attention beat it by more than 3 points.[1]

An appendix compared other choices for the similarity function $\alpha$. A plain dot product scored 65.5 EM and 75.5 F1, a linear function 59.5 and 69.7, a bilinear function 61.6 and 71.8, and a linear layer after an MLP 66.2 and 76.4 on the development set, all below the function used in the final model. Adding an MLP to the fusion function $\beta$ also gave slightly worse results.[1]

### CNN/DailyMail cloze test

To show the architecture's modularity, the authors adapted only the output layer for the CNN and Daily Mail cloze datasets of Hermann et al. (2015), in which the model must fill in an anonymized entity removed from a summary sentence.[1] Because each answer is a single entity, the model predicts only a start index, masks out non-entity words, and sums the probabilities of all mentions of the correct entity during training. It split articles into 19-word windows around each entity and took about 60 hours to train on eight Titan X GPUs.[1]

| Model | CNN val | CNN test | Daily Mail val | Daily Mail test |
|---|---|---|---|---|
| Attentive Reader (Hermann et al., 2015) | 61.6 | 63.0 | 70.5 | 69.0 |
| AS Reader (Kadlec et al., 2016) | 68.6 | 69.5 | 75.0 | 73.9 |
| Stanford AR (Chen et al., 2016) | 73.8 | 73.6 | 77.6 | 76.6 |
| GA Reader (Dhingra et al., 2016) | 73.0 | 73.8 | 76.7 | 75.7 |
| ReasoNet (Shen et al., 2016) | 72.9 | 74.7 | 77.6 | 76.6 |
| BiDAF (single) | 76.3 | 76.9 | 80.3 | 79.6 |
| Stanford AR ensemble (Chen et al., 2016) | 77.2 | 77.6 | 80.2 | 79.2 |

Accuracy in percent; selected rows from BiDAF paper, Table 3.[1]

BiDAF beat all previous single-run models on both datasets, and on the Daily Mail test set the single model also beat the best ensemble.[1]

### Analysis

The authors examined what the layers learn on the SQuAD development set. Using [cosine similarity](https://aiwiki.ai/wiki/cosine_similarity), they found that question words such as "When", "Where" and "Who" were poorly aligned with likely answers after the word embedding layer, but after the contextual embedding layer "When" matched years, "Where" matched locations and "Who" matched names.[1] A [t-SNE](https://aiwiki.ai/wiki/t_sne) plot showed that the word layer set "May" apart from the other months because of its several meanings, while the contextual layer separated the different uses of the word.[1]

Compared with the feature-based logistic regression baseline of Rajpurkar et al., BiDAF answered more than 86% of the questions the baseline answered correctly and outperformed it in each of the ten most frequent question-word categories.[1] In an error analysis of 50 randomly chosen wrong answers, 50% came from imprecise answer boundaries, 28% from syntactic complications and ambiguities, 14% from paraphrase problems, 4% from needing external knowledge, 2% from needing several sentences, and 2% from tokenization mistakes.[1]

## Implementations

The paper links its code and an interactive demo at allenai.github.io/bi-att-flow.[1] The original implementation, in the allenai/bi-att-flow repository on GitHub, is written in Python with [TensorFlow](https://aiwiki.ai/wiki/tensorflow) (release r0.11, with a separate branch for TensorFlow 1.2) and is licensed under the Apache License 2.0.[7][8] Its README describes a model of about 2.5 million parameters that needs at least 12 GB of GPU memory and converges in about 18,000 steps (roughly 20 hours) on a Pascal-generation Titan X, and it points to a CodaLab worksheet with the weights used for the leaderboard submission.[7]

BiDAF was one of the reference models in AllenNLP, the open-source NLP library from the Allen Institute for AI. The 2018 AllenNLP paper describes it as a reference implementation of the BiDAF machine comprehension model, "which was state of the art for the SQuAD benchmark (Rajpurkar et al., 2016) in early 2017."[2] The model is registered under the name `bidaf` in the allennlp-models package.[9] As of November 2026 the AllenNLP README says the library is in maintenance mode, with bug fixes and answers to questions promised only until 16 December 2022.[10]

The [ONNX](https://aiwiki.ai/wiki/onnx) Model Zoo also published a BiDAF model converted from a CNTK implementation, listed at 68.1 EM on SQuAD v1.1, along with an int8 quantized version.[11] The Model Zoo's GitHub repository has been kept for historical purposes only since 2025, when its model files stopped being downloadable there; both BiDAF files are now hosted on Hugging Face under the onnxmodelzoo organization.[16][17]

## Influence and later work

Several follow-up systems reused BiDAF's attention layer or built directly on the model:

- **Unanswerable questions.** Levy, Seo, Choi and Zettlemoyer (CoNLL 2017) started from BiDAF for zero-shot relation extraction and added a trainable bias to the start and end score vectors, so the model can assign probability to "no answer".[12] The SQuAD 2.0 paper (Rajpurkar, Jia and Liang, ACL 2018) used this BiDAF-No-Answer (BNA) model as a baseline. BNA scored 68.0 EM and 77.3 F1 on SQuAD 1.1 but 59.2 EM and 62.1 F1 on the SQuAD 2.0 test set, where humans reached 89.5 F1.[3]
- **Multi-paragraph reading.** Clark and Gardner's model for multi-paragraph reading comprehension (DocQA) uses the bi-directional attention mechanism from BiDAF to build its query-aware context representation,[13] and adds a self-attention layer after it.[14] On the TriviaQA web data they also reported results for BiDAF itself under different paragraph-selection and training schemes.[13]
- **ELMo.** The [ELMo](https://aiwiki.ai/wiki/elmo) paper (Peters et al., NAACL 2018) used Clark and Gardner's model as its SQuAD baseline, describing it as an improved version of BiDAF that adds self-attention and replaces the LSTMs with gated recurrent units. Adding ELMo raised its test F1 from 81.1% to 85.8%; an 11-member ensemble reached 87.4 F1.[14] The [BERT](https://aiwiki.ai/wiki/bert) paper later listed "BiDAF+ELMo (Single)" at 85.8 test F1 among published SQuAD v1.1 systems.[15]
- **QANet.** Yu et al.'s QANet (ICLR 2018) replaced BiDAF's recurrent layers with convolution and [self-attention](https://aiwiki.ai/wiki/self_attention) but kept its trilinear similarity function, its query-aware input to the model encoder and its start/end output strategy.[6] On the same NVIDIA P100 GPU, QANet trained 4.3 times faster and ran inference 7.0 times faster than the original BiDAF code, and it reached 77.0 development F1, the best score its authors obtained with the BiDAF code, in 3 hours instead of 15.[6]

The BERT paper described the BiDAF approach as an instance of a common pattern for text pairs: encode the two texts separately, then apply bidirectional cross attention. BERT instead uses self-attention over the concatenated pair, which unifies the two stages.[15]

## Robustness and adversarial evaluation

BiDAF featured prominently in early studies of how brittle SQuAD models were. Jia and Liang (EMNLP 2017) added automatically generated distracting sentences to SQuAD paragraphs that did not change the correct answer.[4] Their opening example shows the BiDAF ensemble answering correctly on the original paragraph and then picking an entity from the inserted sentence.[4] On 1,000 sampled development examples, the BiDAF ensemble's F1 fell from 80.0 to 34.2 under the ADDSENT attack, and to 2.7 under ADDANY, which appends ungrammatical word sequences found by querying the model.[4] Across sixteen published models, average F1 dropped from 75% to 36%.[4]

The authors also noted that because BiDAF predicts the start and end positions independently, it sometimes returned very long spans when the adversarial sentence pulled the end pointer but not the start pointer.[4] Training the single BiDAF model on adversarial examples restored much of its accuracy on ADDSENT (70.4 F1, up from 34.8), but a modified attack, ADDSENTMOD, brought it back down to 39.2 F1.[4]

In "Beat the AI" (Bartolo et al., Transactions of the ACL, 2020), crowd workers wrote questions that a model in the annotation loop failed to answer, with BiDAF, BERT-Large and RoBERTa-Large as progressively stronger adversaries. Each model produced a dataset of 10,000 training, 1,000 validation and 1,000 test questions.[5] With the AllenNLP and Transformers implementations the study relied on, BiDAF scored 65.5 EM and 77.5 F1 on the SQuAD 1.1 validation set.[5] The study found that BiDAF generalized poorly to data collected with any model in the loop and could not learn from the data collected against it, while [RoBERTa](https://aiwiki.ai/wiki/roberta) trained on the BiDAF-collected data reached 39.9 F1 on questions it could not answer when trained on SQuAD, only marginally lower than the 41.0 F1 it reached when trained on data collected against RoBERTa itself.[5]

## See also

- [SQuAD](https://aiwiki.ai/wiki/squad)
- [Question answering models](https://aiwiki.ai/wiki/question_answering_models)
- [Attention](https://aiwiki.ai/wiki/attention)
- [ELMo](https://aiwiki.ai/wiki/elmo)
- [Dynabench](https://aiwiki.ai/wiki/dynabench)

## References

1. Seo, Minjoon; Kembhavi, Aniruddha; Farhadi, Ali; Hajishirzi, Hannaneh. "Bidirectional Attention Flow for Machine Comprehension". ICLR 2017 (arXiv:1611.01603), 2016. https://arxiv.org/abs/1611.01603
2. Gardner, Matt; Grus, Joel; Neumann, Mark; Tafjord, Oyvind; Dasigi, Pradeep; Liu, Nelson F.; Peters, Matthew; Schmitz, Michael; Zettlemoyer, Luke. "AllenNLP: A Deep Semantic Natural Language Processing Platform". NLP-OSS workshop at ACL 2018 (arXiv:1803.07640), 2018. https://arxiv.org/abs/1803.07640
3. Rajpurkar, Pranav; Jia, Robin; Liang, Percy. "Know What You Don't Know: Unanswerable Questions for SQuAD". ACL 2018 (arXiv:1806.03822), 2018. https://arxiv.org/abs/1806.03822
4. Jia, Robin; Liang, Percy. "Adversarial Examples for Evaluating Reading Comprehension Systems". EMNLP 2017 (arXiv:1707.07328), 2017. https://arxiv.org/abs/1707.07328
5. Bartolo, Max; Roberts, Alastair; Welbl, Johannes; Riedel, Sebastian; Stenetorp, Pontus. "Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension". Transactions of the Association for Computational Linguistics, Volume 8 (arXiv:2002.00293), 2020. https://arxiv.org/abs/2002.00293
6. Yu, Adams Wei; Dohan, David; Luong, Minh-Thang; Zhao, Rui; Chen, Kai; Norouzi, Mohammad; Le, Quoc V. "QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension". ICLR 2018 (arXiv:1804.09541), 2018. https://arxiv.org/abs/1804.09541
7. Allen Institute for AI. "Bi-directional Attention Flow for Machine Comprehension" (README). GitHub, allenai/bi-att-flow. https://github.com/allenai/bi-att-flow
8. Allen Institute for AI. "LICENSE" (Apache License, Version 2.0). GitHub, allenai/bi-att-flow. https://github.com/allenai/bi-att-flow/blob/master/LICENSE
9. Allen Institute for AI. "allennlp_models/rc/models/bidaf.py". GitHub, allenai/allennlp-models. https://github.com/allenai/allennlp-models/blob/main/allennlp_models/rc/models/bidaf.py
10. Allen Institute for AI. "AllenNLP" (README). GitHub, allenai/allennlp. https://github.com/allenai/allennlp
11. ONNX. "BiDAF". ONNX Model Zoo, GitHub, onnx/models. https://github.com/onnx/models/tree/main/validated/text/machine_comprehension/bidirectional_attention_flow
12. Levy, Omer; Seo, Minjoon; Choi, Eunsol; Zettlemoyer, Luke. "Zero-Shot Relation Extraction via Reading Comprehension". CoNLL 2017 (arXiv:1706.04115), 2017. https://arxiv.org/abs/1706.04115
13. Clark, Christopher; Gardner, Matt. "Simple and Effective Multi-Paragraph Reading Comprehension". ACL 2018 (arXiv:1710.10723), 2018. https://arxiv.org/abs/1710.10723
14. Peters, Matthew E.; Neumann, Mark; Iyyer, Mohit; Gardner, Matt; Clark, Christopher; Lee, Kenton; Zettlemoyer, Luke. "Deep contextualized word representations". NAACL 2018 (arXiv:1802.05365), 2018. https://arxiv.org/abs/1802.05365
15. Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". NAACL 2019 (arXiv:1810.04805), 2019. https://arxiv.org/abs/1810.04805
16. ONNX. "ONNX Model Zoo" (README, deprecation notice). GitHub, onnx/models. https://github.com/onnx/models
17. ONNX Model Zoo. "onnxmodelzoo/bidaf-9". Hugging Face. https://huggingface.co/onnxmodelzoo/bidaf-9

