GLM

GLM, short for General Language Model, is both a pretraining framework for language understanding and generation and the name of a model family developed by researchers associated with Tsinghua University and Zhipu AI. The framework was introduced in a March 2021 preprint and published at ACL 2022 as "GLM: General Language Model Pretraining with Autoregressive Blank Infilling." It trains one Transformer to reconstruct missing spans of text autoregressively, using an attention mask and two-dimensional positional encoding that let the same parameter set process bidirectional context and generate text from left to right.^[1][2]

The name later became an umbrella for a succession of mostly Chinese-English large language models. These include the early GLM-10B checkpoints, the bilingual GLM-130B, the conversational ChatGLM series, and the GLM-4 and GLM-5 generations. The relationship is a research lineage rather than a claim that every release has an identical architecture. Later models changed their normalization, attention mechanisms, positional encoding, context length, post-training methods, and, beginning with GLM-4.5, their use of sparse mixture-of-experts layers.^[6]

Origin and scope

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang first submitted the GLM paper to arXiv on 18 March 2021. The revised paper appeared in the proceedings of the 60th Annual Meeting of the Association for Computational Linguistics in May 2022.^[1][2] Its starting point was a division among the dominant pretraining approaches of the period. BERT-style autoencoding models were designed around masked-token prediction and bidirectional context, GPT-style autoregressive models generated text in one direction, and encoder-decoder models such as T5 handled text-to-text tasks with separate stacks. The authors sought one framework that could address natural language understanding, conditional generation, and unconditional generation without maintaining a separate architecture for each category.^[2]

GLM addressed that problem through autoregressive blank infilling. It retained the idea of corrupting an input by masking text, but replaced independent masked-token prediction with sequential reconstruction of whole spans. The framework could vary the number and length of those spans to create objectives suited to classification-like tasks, sequence-to-sequence generation, or open-ended language modeling.^[1][2]

The term should not be confused with a generalized linear model, which is also commonly abbreviated GLM in statistics. Within this article, GLM refers only to the General Language Model framework and its descendant model family.

Autoregressive blank infilling

For each pretraining example, GLM samples one or more continuous spans from the source text and replaces every sampled span with a single [MASK] token. The missing spans are then placed in a separate generation segment and shuffled into a random order. GLM reconstructs the spans one at a time, while generating the tokens inside each span from left to right.^[2]

The random span order is part of the objective rather than a presentation detail. When several blanks occur in one passage, the model cannot rely on a fixed left-to-right order between them. It learns the conditional probability of each missing span given the corrupted source and any spans already reconstructed. The original paper presented this as a way to model dependencies among missing regions while avoiding the independence assumption used in conventional masked language modeling.^[2]

Each training sequence has two logical parts:

Scroll sideways for more →

This mixed attention mask gives the single Transformer two roles. Part A behaves as a bidirectional representation of the known context. Part B behaves as an autoregressive generator conditioned on that representation and its own preceding output. The framework therefore does not join a separately parameterized encoder to a separately parameterized decoder. It changes which positions can attend to which other positions inside one model.^[2]

For the short-span objective used primarily for language understanding, the paper sampled span lengths from a Poisson distribution with a mean of three and continued sampling until at least 15 percent of the original tokens had been masked. Each missing region was represented by one mask token regardless of its true length.^[2]

Two-dimensional positional encoding

A single mask token creates a positional problem: the source shows where a blank begins but not how many tokens it contains. GLM assigns two position identifiers to each token to keep those concepts separate. The first records the position of the token in the corrupted source. Every generated token belonging to a missing span receives the position of that span's mask token. The second identifier records a token's offset inside its generated span. Part A tokens receive zero for this second coordinate, while Part B positions count upward within each span.^[2]

The model learns embeddings for both coordinates and adds them to the token representation. As a result, it knows where a missing span belongs and how far it has progressed while reconstructing that span, but the corrupted source does not reveal the target length in advance. This differs from approaches that preserve one mask for every missing token or otherwise encode the original positions of the removed tokens.^[2]

The original GLM architecture used one Transformer stack with several implementation changes from the baseline architecture described in the Transformer paper. It rearranged layer normalization and residual connections, used a single linear layer for output prediction, and used GeLU activations. These details describe the 2021 framework experiments. They are not a complete specification for later systems sold or released under the GLM name.^[2]

Pretraining objectives and downstream tasks

The paper used the same reconstruction formulation at three span scales. Only the selection and length of the blanks changed.^[2]

Scroll sideways for more →

During multi-task pretraining, batches mixed the short-span objective with a longer sentence-level or document-level objective. The authors argued that this let parameters learned for bidirectional understanding and autoregressive generation support one another instead of residing in separate models.^[2]

For classification, GLM followed a prompt-based formulation influenced by Pattern-Exploiting Training. A task example was converted into a cloze statement containing a mask, and each class label was mapped to a word or phrase called a verbalizer. The model scored a class by the probability of generating its verbalizer in the blank. For conditional generation, the source became Part A and a mask at the end prompted Part B. Unconditional generation used the long-span objective without a task-specific classifier head.^[2]

Results in the original study

The ACL paper evaluated GLM on SuperGLUE, abstractive summarization, question generation, text infilling, and language modeling. In the authors' experiments, models trained with comparable data and parameter counts outperformed several BERT, T5, and GPT baselines across the studied task groups. On the SuperGLUE development set, for example, GLM-Large averaged 77.0 compared with 72.0 for BERT-Large when both used BooksCorpus and English Wikipedia. A GLM variant trained on a RoBERTa-sized corpus averaged 82.9 compared with 81.5 for the cited RoBERTa-Large baseline.^[2]

The multi-task experiments also exposed a tradeoff rather than a uniform advantage. A 335 million parameter document-level GLM performed worse than an equal-sized GPT baseline on zero-shot language modeling because it divided capacity between infilling and generation. Increasing GLM to 410 million parameters, 1.25 times the paper's GPT-Large baseline, brought the result close to GPT under unidirectional evaluation and above it when GLM used bidirectional context. Removing the second positional coordinate reduced language-modeling accuracy and increased perplexity in the reported ablation.^[2]

These results establish what the authors observed under their data, model sizes, and evaluation procedures. They do not make the 2021 comparisons a current ranking, and they do not by themselves evaluate the later commercial GLM family.

Early checkpoints and scaling

The official THUDM repository released research checkpoints ranging from 110 million to 10 billion parameters. The list included English GLM-Base and GLM-Large models, generation-oriented GLM-Doc variants, an English GLM-10B trained on the Pile, and a Chinese GLM-10B trained on WuDaoCorpora. The repository also supplied code for pretraining, generation, classification, infilling, and conversion to the Hugging Face format.^[3]

The 2024 family report dates the open GLM-10B release to June 2021. Training of GLM-130B began later that year and finished in July 2022 after more than 400 billion English and Chinese tokens. The model and its training details were released in August 2022.^[4][5][6] Scaling required changes beyond the small-model recipe. GLM-130B used DeepNorm, rotary positional encoding, and gated feed-forward units, and its project documented training instability and low-bit inference. Those design and deployment details are covered in the dedicated GLM-130B article.

GLM-130B also used two mask types in its generation interface. [MASK] represented short infilling, while [gMASK] marked the long blank used for left-to-right generation. If a prompt did not contain a mask token, the reference code appended [gMASK] to the end.^[5]

From ChatGLM to GLM-4

Zhipu AI and the Tsinghua team turned the GLM-130B base model into ChatGLM-130B through supervised instruction tuning and work on reinforcement learning from human feedback. ChatGLM-130B went online on 14 March 2023. On the same day, the team released ChatGLM-6B, a 6.2 billion parameter bilingual dialogue model intended to support faster research iteration and local deployment with INT4 quantization.^[6][7]

ChatGLM2-6B followed on 25 June 2023 and ChatGLM3-6B on 27 October. The 2024 report says both successors were pretrained from scratch rather than continued directly from the first 6B checkpoint. Across these releases, the team changed the data, context length, attention implementation, alignment process, and support for functions, code execution, and agent tasks. The name "ChatGLM" therefore denotes a conversational branch of the GLM family, not one checkpoint repeatedly fine-tuned in place.^[6]

GLM-4 became available through Zhipu's API on 16 January 2024. The family report describes GLM-4, GLM-4-Air, and GLM-4-9B as drawing on the preceding three ChatGLM generations. Its pretraining corpus contained about ten trillion tokens, mostly Chinese and English with a smaller amount from 24 other languages. Post-training used multiple stages that included supervised fine-tuning and reinforcement learning from human feedback.^[6]

The open GLM-4-9B models arrived on 5 June 2024. They included a base model, 128K chat variants, an experimental 1M-context chat model, and the visual GLM-4V-9B checkpoint.^[8] The report also described GLM-4 All Tools, which was aligned to choose among a web browser, Python interpreter, text-to-image system, and user-defined functions. Those product variants, their evaluations, and their access terms belong in the GLM-4 article.^[6]

Later family evolution

Later releases retained the GLM name while moving further from the dense architecture used in the 2021 experiments. The table summarizes the language-model lineage and links to the articles that contain model-level specifications and evaluations.

Scroll sideways for more →

GLM-4.5 moved the public flagship line to a sparse architecture. Its flagship used 355 billion total parameters with 32 billion active for each token, while GLM-4.5-Air used 106 billion total and 12 billion active. Z.ai described both as hybrid reasoning models whose single weight set could operate in a thinking mode for reasoning and tools or a non-thinking mode for direct responses.^[9][10] GLM-5 increased both total scale and training data, while the 2026 updates concentrated on sustained tool-driven work and longer contexts.^[12][13][14]

What the family name does and does not imply

Several ideas connect the releases. The family began with a single-stack Transformer trained for both understanding and generation. It continued to emphasize bilingual Chinese-English modeling, scalable pretraining, instruction alignment, and the publication of at least some code or model weights. The later papers also present the releases as one continuous program, explicitly drawing a timeline from the 2021 framework through GLM-130B, the three ChatGLM generations, and GLM-4.^[3][6]

Other properties changed enough that they should not be inferred from the GLM label alone. GLM-130B replaced parts of the original implementation with scale-oriented components. GLM-4 used RMSNorm, SwiGLU, grouped-query attention, and a two-dimensional extension of rotary positional embeddings. Context support grew from 2K in the first ChatGLM generation to 32K in ChatGLM2 and ChatGLM3, then to 128K and experimental 1M variants in GLM-4. GLM-4.5 moved the flagship line from dense models to sparse mixture-of-experts systems.^[6][9]

Post-training also became a larger part of what a GLM release represented. The 2021 paper centered on self-supervised pretraining and task fine-tuning. By the 2024 report, the production family combined large-scale pretraining with supervised instruction tuning, preference alignment, safety work, long-context alignment, and training for tool selection. For this reason, "based on GLM" may refer to descent from the research program and its objectives rather than byte-for-byte retention of the first paper's model configuration.^[2][6]

Availability and licensing

There is no single license for the entire GLM lineage. The original research repository publishes its implementation and checkpoints through the THUDM project, while GLM-130B made its weights, code, training logs, toolkit, and lessons available through a separate repository.^[3][4][5] ChatGLM-6B used Apache 2.0 for repository code but a separate model license for the weights. That license allowed academic research and required registration for free commercial use.^[7]

The GLM-4 family mixed hosted API models with open-weight 9B releases.^[6][8] Beginning with GLM-4.5, the official model cards for the main open-weight releases identify the weights under the MIT license, including GLM-4.5, GLM-5.1, and GLM-5.2.^[10][13][15] Users therefore need to check the specific checkpoint, code repository, and service terms rather than treating "GLM" as a licensing category.

References

1. ^Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang, "GLM: General Language Model Pretraining with Autoregressive Blank Infilling," arXiv, submitted 18 March 2021 and revised 17 March 2022. arxiv.org/...2103.10360
2. ^Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang, "GLM: General Language Model Pretraining with Autoregressive Blank Infilling," Association for Computational Linguistics, Proceedings of ACL 2022, pages 320-335, May 2022. aclanthology.org/2022.acl-long.26
3. ^THUDM, "GLM (General Language Model)," official GitHub repository, accessed 24 July 2026. github.com/...GLM
4. ^Aohan Zeng et al., "GLM-130B: An Open Bilingual Pre-trained Model," arXiv, submitted 6 October 2022; conference paper at ICLR 2023. arxiv.org/...2210.02414
5. ^THUDM and Z.ai, "GLM-130B: An Open Bilingual Pre-Trained Model," official GitHub repository, accessed 24 July 2026. github.com/...GLM-130B
6. ^Team GLM, "ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools," Zhipu AI and Tsinghua University, arXiv, submitted 18 June 2024 and revised 30 July 2024. arxiv.org/...2406.12793
7. ^THUDM and Z.ai, "ChatGLM-6B: An Open Bilingual Dialogue Language Model," official GitHub repository, accessed 24 July 2026. github.com/...ChatGLM-6B
8. ^Z.ai, "GLM-4 Series: Open Multilingual Multimodal Chat LMs," official GitHub repository, model release table dated 5 June 2024, accessed 24 July 2026. github.com/...GLM-4
9. ^Z.ai, "GLM-4.5: Reasoning, Coding, and Agentic Abililties," Z.ai Research, 28 July 2025. z.ai/...glm-4.5
10. ^Z.ai, "GLM-4.5," official Hugging Face model card, accessed 24 July 2026. huggingface.co/...GLM-4.5
11. ^Z.ai, "GLM-4.6: Advanced Agentic, Reasoning and Coding Capabilities," Z.ai Research, 30 September 2025. z.ai/...glm-4.6
12. ^Z.ai, "GLM-5: From Vibe Coding to Agentic Engineering," Z.ai Research, 12 February 2026. z.ai/...glm-5
13. ^Z.ai, "GLM-5.1 & GLM-5," official GitHub repository, accessed 24 July 2026. github.com/...GLM-5 ; Z.ai, "GLM-5.1," official Hugging Face model card, accessed 24 July 2026. huggingface.co/...GLM-5.1
14. ^Z.ai, "GLM-5.2: Built for Long-Horizon Tasks," Z.ai Research, 16 June 2026. z.ai/...glm-5.2
15. ^Z.ai, "GLM-5.2," official Hugging Face model card, accessed 24 July 2026. huggingface.co/...GLM-5.2