BERT

BERT, short for Bidirectional Encoder Representations from Transformers, is a pretrained language model introduced by researchers at Google Research in 2018. It uses a stack of bidirectional Transformer encoder blocks to produce a context-dependent representation for every input token. Its paper's central method was to hide selected tokens and train the network to reconstruct them from both left and right context, then adapt the same pretrained parameters to supervised natural language processing tasks with small task-specific output layers.^[1]

BERT did not invent language-model pretraining or the practice of fine-tuning a pretrained network. ELMo had shown the value of contextual representations, while ULMFiT and the original GPT work had already demonstrated transfer from language-model pretraining to supervised tasks. BERT's distinctive contribution was to combine end-to-end fine-tuning with a deeply bidirectional Transformer encoder trained by masked language modeling, and to show that this design worked across a broad set of sentence-level and token-level benchmarks.^[7][8][9]

The original paper was posted as a preprint in October 2018, and Google announced the source-code and checkpoint release on November 2, 2018. The peer-reviewed paper appeared at NAACL 2019 and received the conference's Best Long Paper award. The paper reported new best results at the time on eleven tasks, but those numbers describe specific 2018 evaluation settings rather than present-day rankings.^[1][2][5]

Scope and identity

In a narrow sense, BERT refers to the architecture, pretraining procedure, and Base and Large configurations described by Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. In a broader sense, "BERT model" is sometimes used for later encoder-only systems that retain some combination of bidirectional self-attention, masked-token pretraining, and downstream fine-tuning. This article distinguishes the original model from that larger family because later systems often change the data, objective, tokenizer, position representation, parameter sharing, or intended task.^[1][26]

BERT is an encoder representation model, not a left-to-right text generator. It maps an input sequence to contextual hidden states and can score masked tokens when an MLM head is attached. A fine-tuned BERT can classify sequences, label tokens, compare text pairs, or select answer spans, but the original architecture does not directly define the autoregressive next-token interface used by generative models.^[1][9]

The original English checkpoints were released in cased and uncased forms and in Base and Large sizes. Google subsequently added Chinese and multilingual checkpoints, whole-word-masking variants, and 24 smaller English models. Those releases share a repository but are not all experimental conditions from the NAACL paper. A page such as Bert-base-uncased model denotes one particular checkpoint rather than the complete BERT method.^[3]

Historical context and release

Earlier neural representation methods established several pieces of the approach. ELMo generated contextual token features by combining separately trained forward and backward language models and normally supplied those features to a task-specific architecture. ULMFiT pretrained a recurrent language model and introduced a practical fine-tuning procedure for text classification. The original GPT paper pretrained a left-to-right Transformer and fine-tuned it with modest task-specific changes. BERT built on this transfer learning line while changing how context was available during pretraining.^[7][8][9]

The BERT authors argued that a standard left-to-right language-model objective constrains each representation to preceding context. Simply allowing a deep network to inspect the token it must predict would make the objective trivial. Their solution was the masked language model, which corrupts selected input positions and predicts the original tokens. Because the target is hidden or altered, every encoder layer can use context on both sides without directly copying the target token.^[1]

Google's November 2018 release included TensorFlow code, pretrained cased and uncased checkpoints, and scripts for several paper experiments. The repository later documented compatible third-party implementations, including an early PyTorch port. The distinction between the research paper and the public implementation matters: the repository states that its Python pretraining-data generator is not the exact C++ code used for the paper and that the original code had additional complexity.^[2][3]

One documented application followed in October 2019. Google said it was applying BERT models to Search ranking and featured snippets and that, at launch, the ranking change would help understand one in ten US English searches. This is evidence of a specific deployment announcement, not evidence that the same model, proportion, or serving design remained in use after 2019.^[33]

Architecture

BERT uses a multi-layer bidirectional Transformer encoder derived from the architecture introduced by Vaswani and colleagues. Each block contains multi-head self-attention and a position-wise feed-forward network, together with residual connections and layer normalization. Unlike a causal decoder, the original encoder's attention pattern does not mask future positions: a token representation can incorporate non-padding tokens on both sides of that position.^[1][6]

The paper denotes the number of Transformer blocks by L, hidden width by H, and number of attention heads by A. Its two principal configurations were:^[1]

Scroll sideways for more →

BERT-Base was sized to make comparison with the original GPT model more direct. BERT-Large increased depth and width, and it produced the paper's strongest reported results. The size comparison does not isolate every difference between the two systems: BERT and GPT also differed in objective, attention mask, training corpus, batch size, special-token treatment, and fine-tuning choices.^[1][9]

Self-attention creates a contextual vector for each position by mixing information from other positions. For a sequence of length n, ordinary dense attention computes pairwise interactions across positions, so its attention matrix and core attention computation grow quadratically with n. The original BERT checkpoints were trained with learned positions up to 512 tokens, which sets an architectural and checkpoint boundary for those releases.^[1][6]

The encoder returns one hidden state per input position. What a state means depends on its position, its surrounding text, the model checkpoint, and any downstream fine-tuning. BERT therefore does not assign a single fixed vector to a word type. The representation of a word such as "bank" can differ between financial and river contexts because information is exchanged through the encoder stack.^[1][2]

Input representation and tokenization

The paper constructs each position's initial representation as the sum of a token embedding, a segment embedding, and a learned position embedding:^[1]

$$h_i^0 = e_i^{\mathrm{token}} + e_i^{\mathrm{segment}} + e_i^{\mathrm{position}}$$

Token embeddings identify vocabulary pieces. Segment embeddings distinguish text A from text B in a packed pair. Position embeddings distinguish locations in the sequence. The paper described a WordPiece vocabulary of about 30,000 entries; the precise vocabulary file is checkpoint-specific and must remain paired with its checkpoint.^[1][3]

Every paper-style sequence begins with [CLS]. A [SEP] token separates two packed spans and another terminates the pair. For sequence classification, the original fine-tuning design passes the last-layer hidden state at [CLS] to a classifier. For token tasks, output layers consume individual token states. The paper itself cautions that the pretrained [CLS] state is not automatically a meaningful general-purpose sentence representation without suitable fine-tuning.^[1]

The released tokenization code first performs basic text processing and then applies a greedy longest-match-first lookup against the supplied WordPiece vocabulary. Continuation pieces are marked with ##; if a word cannot be segmented under the vocabulary and implementation limits, the tokenizer emits [UNK]. The uncased English checkpoints lowercase text and strip accent marks, whereas cased checkpoints preserve case and accents.^[3][4]

This code documents tokenization at inference and preprocessing time, not the original vocabulary-learning algorithm. The repository explicitly says that code for learning a new WordPiece vocabulary was not released because that implementation depended on internal C++ libraries. It is therefore unsafe to infer the vocabulary-training score, merge history, or reserved-index policy from tokenization.py alone.^[3][4]

WordPiece can reduce out-of-vocabulary failures by representing an uncommon surface form as several known pieces. It does not guarantee linguistically meaningful morphemes, equal segmentation quality across languages, or robustness to spelling and Unicode variation. Token boundaries also affect downstream labeling because a word-level annotation may need to be aligned with multiple subword states.^[4][21]

Pretraining objectives

The original pre-training procedure optimized masked language modeling and next sentence prediction together. The model was not given human labels for these objectives; targets were constructed automatically from text. The paper calls this unsupervised pretraining, while later literature often uses the more specific term self-supervised learning.^[1]

For masked language modeling, the data generator selected 15 percent of WordPiece positions. Among those selected positions, 80 percent were replaced by [MASK], 10 percent by a random vocabulary token, and 10 percent left unchanged. A vocabulary softmax predicted the original identity at each selected position, and the MLM loss was computed for those selected positions rather than every token.^[1]

The mixed replacement rule reduces but does not eliminate a train-task mismatch. [MASK] appears during pretraining but ordinarily does not appear in downstream inputs. Random and unchanged selections expose the network to selected positions that do not visibly contain [MASK], although the model is not told which unchanged positions are targets. The original appendix found the fine-tuning results in its tested tasks reasonably robust to several masking mixtures, while feature-based use was more sensitive.^[1]

For next sentence prediction, the training input contained spans A and B. In half the examples, B followed A in the source document; in the other half, B was sampled from another part of the corpus and labeled NotNext. The final [CLS] state was used for this binary prediction. "Sentence" in the paper can mean a contiguous text span rather than a grammatical sentence.^[1]

Evidence about NSP is conditional rather than a universal verdict. The BERT ablation reported worse QNLI, MultiNLI, and SQuAD results when NSP was removed under its controlled Base setup. RoBERTa later obtained strong results after removing NSP while also changing data, batching, masking, and training duration. ALBERT replaced NSP with sentence-order prediction, where a true adjacent pair is contrasted with the same pair in reversed order. These experiments show that pretraining recipe and negative-example construction matter; they do not by themselves prove that one sentence objective helps every encoder and dataset.^[1][14][15]

Later work changed the prediction target itself. ELECTRA trained a discriminator to decide whether each token had been replaced by a small generator, giving a learning signal at more positions. SpanBERT masked contiguous spans and added a span-boundary objective. XLNet used permutation-based autoregressive training to obtain bidirectional context without inserting [MASK]. Each addresses a different limitation, so none should be described as a drop-in correction under all compute and task settings.^[16][17][18]

Original training procedure

The paper pretrained English BERT on BooksCorpus, reported as 800 million words, and English Wikipedia, reported as 2.5 billion words. For Wikipedia, the authors retained text passages and excluded lists, tables, and headings. They used document-level text so that the input generator could sample contiguous spans for NSP. These corpus sizes and preprocessing statements describe the authors' 2018 setup, not a reproducible snapshot identifier for either corpus.^[1]

The reported schedule used a batch of 256 sequences for 1,000,000 steps, approximately 40 passes over the combined 3.3-billion-word corpus by the paper's estimate. To reduce the cost of long attention, 90 percent of steps used a maximum sequence length of 128, followed by 10 percent at length 512 so that the longer learned position embeddings received training.^[1]

The optimizer was Adam with learning rate 1e-4, beta1=0.9, beta2=0.999, L2 weight decay 0.01, 10,000 warmup steps, and linear learning-rate decay. The paper used dropout probability 0.1 and the GELU activation. The joint loss was the sum of mean MLM likelihood and mean NSP likelihood. In particular, the documented recipe did include weight decay.^[1]

BERT-Base was trained on four Cloud TPUs, described as 16 TPU chips in total, and BERT-Large on 16 Cloud TPUs, described as 64 chips. The paper says each pretraining run took four days. Those statements report hardware and elapsed time for the original experiment; they are not a dollar-cost estimate and do not imply the same duration on later accelerators or implementations.^[1]

The public repository can generate MLM and NSP examples from arbitrary text, but it is not a byte-for-byte reconstruction of the original training pipeline. Its README identifies the paper implementation as C++ with additional complexity, and it separately notes that the vocabulary-learning code was not released. Reproducing a checkpoint therefore requires more than selecting the headline hyperparameters: corpus version, cleanup, sentence segmentation, vocabulary, randomization, numerical kernels, and code path can all affect the result.^[3]

Fine-tuning and task interfaces

Fine tuning initializes a task model from the pretrained checkpoint and updates all BERT parameters along with a small new output layer. The original paper emphasized a unified interface: most task differences are expressed by how inputs are packed and which hidden states feed the output, rather than by building a different encoder architecture for every benchmark.^[1]

Common interfaces in the paper include:^[1]

Scroll sideways for more →

For GLUE, the paper used batch size 32, three epochs, and development-set selection among learning rates 5e-5, 4e-5, 3e-5, and 2e-5. Its broader appendix suggested batch sizes 16 or 32, learning rates 5e-5, 3e-5, or 2e-5, and two to four epochs. These are an experimental search range, not universally optimal defaults.^[1]

The paper already noted unstable BERT-Large fine-tuning on small GLUE datasets and used multiple random restarts with different data shuffling and classifier initialization. Later experiments found substantial seed-dependent variation on small tasks. Dodge and colleagues attributed comparable portions of tested variance to classifier initialization and data order, while Mosbach and colleagues found that failed runs were better explained by optimization difficulties than by a simple catastrophic-forgetting account and showed that longer, better-tuned training could improve stability.^[1][30][31]

Evaluation must therefore report more than a single favorable run when variance is material. Useful details include the checkpoint, tokenizer, data split, truncation policy, hyperparameter search, number of seeds, selection rule, and both mean and spread across runs. A large benchmark score does not establish calibration, robustness, fairness, or suitability for a different deployment distribution.^[30][31][32]

Original experimental results

The paper evaluated BERT on the GLUE benchmark, SQuAD 1.1 and 2.0, and SWAG, along with other tasks. GLUE combines multiple sentence and sentence-pair datasets; SQuAD 1.1 asks systems to extract an answer span from a supplied passage; SQuAD 2.0 adds questions for which the passage contains no answer; and SWAG asks a model to select a plausible continuation from alternatives.^{[10][11][12][13]}

Selected headline results are listed below. They are historical test results from the BERT paper and repository. "Single" and "ensemble" are not interchangeable, and the strongest SQuAD 1.1 system also used TriviaQA data before SQuAD fine-tuning.^[1][3]

Scroll sideways for more →

The paper's abstract emphasized an official GLUE score of 80.5, MultiNLI accuracy of 86.7 percent, SQuAD 1.1 test F1 of 93.2, and SQuAD 2.0 test F1 of 83.1. The SQuAD 1.1 table makes an important condition explicit: 93.2 was the seven-system ensemble with TriviaQA augmentation, while the corresponding single augmented system reported 91.8 test F1. Reporting only the larger number without the ensemble and augmentation conditions is misleading.^[1]

The results demonstrated transfer across different output structures with limited task-specific architecture. They did not demonstrate that BERT solves language understanding in general, and benchmark improvements can reflect pretraining data, model scale, task-specific selection, or overlap between benchmark regularities and learned representations. Later models and benchmark submissions also make the 2018 leaderboard positions unsuitable as current comparisons.^[1][10][26]

Interpretation and analysis

Probing studies found that BERT hidden states contain information correlated with linguistic annotations. Tenney and colleagues reported a progression under their probing method from local and syntactic information toward semantic-role and coreference information across layers. Clark and colleagues found attention heads that favored delimiters, nearby positions, or broad distributions, as well as some heads aligned with particular syntactic dependencies or coreference relations.^[27][28]

Those findings should be read as measurements under specific probes, not as a complete mechanistic explanation. A later re-analysis found that a simple classical-pipeline account of layer depth did not have conclusive general support once additional factors were considered. The broader BERTology survey likewise concluded that the field had accumulated many observations about representations, objectives, and compression while still lacking a complete account of why BERT works.^[26][29]

Attention visualization has a similar boundary. Some heads exhibit recognizable linguistic patterns, but many do not, and information is distributed across heads and hidden states. More generally, Jain and Wallace showed in several attention-based models that attention distributions can be weakly related to other importance measures and that different distributions can sometimes produce similar predictions. Attention weights can be useful diagnostic evidence, but they are not automatically faithful explanations of a decision.^[28][35]

Fine-tuning also changes the representation being inspected. A probe on the frozen pretrained checkpoint answers a different question from a probe after supervised task training. Conclusions about "what BERT knows" should specify model size, layer, checkpoint, language, input construction, probe capacity, control baseline, and whether parameters were fine-tuned.^[26][27][29]

Limitations and responsible use

The original checkpoints have several architectural limits. Their learned position table supports sequences only up to 512 tokens without modification, and dense self-attention scales quadratically with sequence length. Truncating a longer document can remove decisive context, while splitting it into windows changes cross-window interactions. The model also predicts only selected MLM positions during pretraining, making the learning signal sparser than an objective that supervises every position.^[1][16][18]

MLM introduces artificial corruption, especially [MASK], that is normally absent from downstream inputs. NSP uses random negative spans that can expose topic differences in addition to discourse order. Later work altered or removed these objectives, but results depend on the surrounding recipe; objective comparisons should control data, compute, masking, and model size before attributing a gain to one component.^{[1][14][15][18]}

The original English checkpoints inherit the coverage and historical composition of BooksCorpus and English Wikipedia, use a fixed WordPiece vocabulary, and differ in whether case and accents are normalized. A tokenizer that works acceptably for common English forms can segment specialized terms or other languages poorly. Multilingual BERT covered 104 languages and enabled zero-shot cross-language transfer in the experiments by Pires and colleagues, but that study also found systematic deficiencies and stronger transfer between some language pairs than others.^[1][3][21]

Vanilla BERT is not automatically an efficient semantic-search embedding model. Encoding two texts jointly can model rich cross-text interactions, but it requires a model pass for each pair. The Sentence-BERT study found that naive [CLS] or average pooling from unmodified BERT produced weak sentence embeddings in its similarity evaluations; its siamese and triplet design instead produced reusable vectors that could be compared cheaply for information retrieval and clustering.^[20]

Fine-tuning can be unstable on small data, and development-set selection among many seeds can overstate expected performance if the selection procedure is not reported. Robust evaluation should preserve an untouched test set, repeat training when variance is meaningful, and test inputs that reflect deployment conditions rather than assuming that a benchmark result transfers unchanged.^[1][30][31]

BERT can encode social associations present in its training data. Kurita and colleagues measured gender-related stereotype associations with masked-token probabilities and found a relationship with behavior in a downstream pronoun-resolution case study. The result does not supply a universal scalar "bias score," but it establishes that strong aggregate task performance does not remove the need for subgroup, language, and use-case-specific assessment.^[32]

Because BERT produces statistical representations rather than verified facts or rules, a downstream system can be confidently wrong, sensitive to phrasing, or affected by truncation and domain shift. High-impact applications require task-specific error analysis, human oversight appropriate to the stakes, documentation of the checkpoint and data, and monitoring after distribution changes. These are deployment controls around the model, not properties supplied by pretraining itself.^[26][30][32]

BERT family and related encoders

Later encoders retained portions of BERT while changing a particular design choice. The following table is a map of research directions, not a ranking:^{[14][15][16][17][18][19][20][21][22][23][24][25]}

Scroll sideways for more →

These names do not define a single compatible checkpoint family. Tokenizers, vocabularies, tensor shapes, position schemes, objectives, and licenses can differ, so a downstream model cannot generally exchange checkpoints merely because its name contains "BERT." Reproducing a result requires the exact model identifier and its preprocessing and fine-tuning configuration.^{[3][14][15][24][25]}

Domain-adapted models illustrate another distinction. BioBERT continued pretraining from general-domain BERT on PubMed abstracts and PMC full text, while SciBERT pretrained on a scientific corpus and studied a scientific vocabulary. Their papers reported benefits on selected biomedical or scientific tasks, but those results do not imply that any domain model will improve every task or that a general-domain validation set measures specialized deployment behavior.^[22][23]

ModernBERT is a later bidirectional encoder, not a revision issued by the original BERT authors. Its peer-reviewed 2025 paper describes a 149-million-parameter Base model and a 395-million-parameter Large model, training on two trillion tokens, and native sequence length 8,192. Those figures correct the 139-million figure sometimes repeated for the Base configuration. Performance and efficiency comparisons in that paper depend on its hardware, software, tasks, and batch settings.^[25]

Availability, implementation, and licensing

The original repository released TensorFlow model code, fine-tuning scripts, vocabulary files, configuration files, and checkpoints for BERT-Base and BERT-Large in cased and uncased English forms. It later listed multilingual, Chinese, whole-word-masking, and smaller variants. A checkpoint package includes model weights, a vocabulary, and a configuration; these components must be kept consistent.^[3]

As of the research cutoff of July 28, 2026, GitHub marks google-research/bert as archived by its owner on September 25, 2025 and read-only. That status describes maintenance of this repository. It does not erase the released files, invalidate the 2019 paper, or by itself determine whether independent implementations and derived checkpoints are maintained elsewhere.^[3]

The repository's README says that its code and released models are under the Apache License 2.0, and the repository includes the license text. That statement applies to the materials Google released under those terms; it should not be expanded into a claim that third-party training corpora, downstream datasets, independent implementations, or derived checkpoints all share the same rights or obligations.^[3][34]

For reproducibility, the paper is the authoritative record of the reported experiment, while the repository is the authoritative record of what Google made public. They overlap but are not identical: the public pretraining generator is not the exact paper code, the vocabulary-learning implementation is absent, and later checkpoint additions postdate the original experiments. A careful report should cite both and state which checkpoint, code revision, tokenizer, and dataset preparation it used.^[1][3][4]

References

1. ^Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Proceedings of NAACL-HLT 2019, pages 4171-4186. aclanthology.org/N19-1423
2. ^Devlin, J., and Chang, M.-W. (2018). "Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing." Google Research Blog. research.google/...for-natural-language-processing
3. ^Google Research. "google-research/bert: TensorFlow code and pre-trained models for BERT." GitHub repository and README. github.com/...bert
4. ^Google AI Language Team. "tokenization.py." google-research/bert. github.com/...tokenization.py
5. ^North American Chapter of the Association for Computational Linguistics. (2019). "NAACL 2019 Best Paper Awards." naacl2019.org/...best-papers
6. ^Vaswani, A., et al. (2017). "Attention Is All You Need." Advances in Neural Information Processing Systems 30. proceedings.neurips.cc/...bd053c1c4a845aa-Abstract
7. ^Peters, M. E., et al. (2018). "Deep Contextualized Word Representations." Proceedings of NAACL-HLT 2018, pages 2227-2237. aclanthology.org/N18-1202
8. ^Howard, J., and Ruder, S. (2018). "Universal Language Model Fine-tuning for Text Classification." Proceedings of ACL 2018, pages 328-339. aclanthology.org/P18-1031
9. ^Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). "Improving Language Understanding by Generative Pre-Training." OpenAI. cdn.openai.com/...language_understanding_paper.pdf
10. ^Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2018). "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding." Proceedings of BlackboxNLP 2018. aclanthology.org/W18-5446
11. ^Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). "SQuAD: 100,000+ Questions for Machine Comprehension of Text." Proceedings of EMNLP 2016. aclanthology.org/D16-1264
12. ^Rajpurkar, P., Jia, R., and Liang, P. (2018). "Know What You Don't Know: Unanswerable Questions for SQuAD." Proceedings of ACL 2018. aclanthology.org/P18-2124
13. ^Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. (2018). "SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference." Proceedings of EMNLP 2018. aclanthology.org/D18-1009
14. ^Liu, Y., et al. (2019). "RoBERTa: A Robustly Optimized BERT Pretraining Approach." arXiv:1907.11692. arxiv.org/...1907.11692
15. ^Lan, Z., et al. (2020). "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations." ICLR 2020. openreview.net/forum
16. ^Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. (2020). "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators." ICLR 2020. openreview.net/forum
17. ^Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O. (2020). "SpanBERT: Improving Pre-training by Representing and Predicting Spans." Transactions of the Association for Computational Linguistics 8, pages 64-77. aclanthology.org/2020.tacl-1.5
18. ^Yang, Z., et al. (2019). "XLNet: Generalized Autoregressive Pretraining for Language Understanding." Advances in Neural Information Processing Systems 32. proceedings.neurips.cc/...66733e9ee67cc69-Abstract
19. ^Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." arXiv:1910.01108. arxiv.org/...1910.01108
20. ^Reimers, N., and Gurevych, I. (2019). "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of EMNLP-IJCNLP 2019. aclanthology.org/D19-1410
21. ^Pires, T., Schlinger, E., and Garrette, D. (2019). "How Multilingual is Multilingual BERT?" Proceedings of ACL 2019. aclanthology.org/P19-1493
22. ^Lee, J., et al. (2020). "BioBERT: a pre-trained biomedical language representation model for biomedical text mining." Bioinformatics 36(4), pages 1234-1240. academic.oup.com/...5566506
23. ^Beltagy, I., Lo, K., and Cohan, A. (2019). "SciBERT: A Pretrained Language Model for Scientific Text." Proceedings of EMNLP-IJCNLP 2019. aclanthology.org/D19-1371
24. ^He, P., Liu, X., Gao, J., and Chen, W. (2021). "DeBERTa: Decoding-enhanced BERT with Disentangled Attention." ICLR 2021. openreview.net/forum
25. ^Warner, B., et al. (2025). "Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference." Proceedings of ACL 2025. aclanthology.org/2025.acl-long.127
26. ^Rogers, A., Kovaleva, O., and Rumshisky, A. (2020). "A Primer in BERTology: What We Know About How BERT Works." Transactions of the Association for Computational Linguistics 8, pages 842-866. aclanthology.org/2020.tacl-1.54
27. ^Tenney, I., Das, D., and Pavlick, E. (2019). "BERT Rediscovers the Classical NLP Pipeline." Proceedings of ACL 2019. aclanthology.org/P19-1452
28. ^Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. (2019). "What Does BERT Look at? An Analysis of BERT's Attention." Proceedings of BlackboxNLP 2019. aclanthology.org/W19-4828
29. ^Niu, J., Lu, W., and Penn, G. (2022). "Does BERT Rediscover a Classical NLP Pipeline?" Proceedings of COLING 2022. aclanthology.org/2022.coling-1.278
30. ^Mosbach, M., Andriushchenko, M., and Klakow, D. (2021). "On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines." ICLR 2021. openreview.net/pdf
31. ^Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N. (2020). "Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping." arXiv:2002.06305. arxiv.org/...2002.06305
32. ^Kurita, K., Vyas, N., Pareek, A., Black, A. W., and Tsvetkov, Y. (2019). "Measuring Bias in Contextualized Word Representations." Proceedings of the First Workshop on Gender Bias in Natural Language Processing. aclanthology.org/W19-3823
33. ^Nayak, P. (2019). "Understanding searches better than ever before." Google. blog.google/...search-language-understanding-bert
34. ^Google Research. "Apache License 2.0 for google-research/bert." github.com/...LICENSE
35. ^Jain, S., and Wallace, B. C. (2019). "Attention is not Explanation." Proceedings of NAACL-HLT 2019. aclanthology.org/N19-1357