Model Architecture

Explore Model Architecture through related topics and the articles other pages reference most.

65 articlesSearch and filter Follow updates by RSS

Explore articles

Browse subtopics: Microsoft

Articles that also belong to these categories. Counts cover all of Model Architecture.

Showing 1-6 of 6 articles

BitNet

BitNet is a family of large language model architectures developed by Microsoft Research Asia that constrain the weights of a transformer to extremely low bit-widths: initially a single bit ({-1, +1}) and…

Large Language Models Microsoft

BitNet b1.58

BitNet b1.58 is a ternary-weight large language model architecture from Microsoft Research in which every weight is constrained to one of three values, -1, 0, or +1

Large Language Models Microsoft

Differential Transformer

The Differential Transformer (often shortened to Diff Transformer or DIFF Transformer) is a decoder-only neural sequence architecture introduced by researchers at Microsoft Research and Tsinghua University in…

Microsoft Transformer Models

LongNet

LongNet is a transformer variant introduced by Microsoft Research in July 2023 that is designed to scale attention to sequences exceeding one billion tokens while preserving performance on shorter inputs.

Microsoft Transformer Models

LongRoPE

LongRoPE is a context-window extension technique for large language models (LLMs) that use rotary position embeddings (RoPE).

Large Language Models Microsoft

YOCO (You Only Cache Once)

YOCO ("You Only Cache Once") is a decoder-decoder neural network architecture for large language models introduced by researchers at Microsoft Research and Tsinghua University in May 2024.

Large Language Models Microsoft