Microsoft

Explore Microsoft through related topics and the articles other pages reference most.

45 articlesSearch and filter Follow updates by RSS

Explore articles

Browse subtopics: Model Architecture

Articles that also belong to these categories. Counts cover all of Microsoft.

Showing 1-6 of 6 articles

BitNet

BitNet is a family of large language model architectures developed by Microsoft Research Asia that constrain the weights of a transformer to extremely low bit-widths: initially a single bit ({-1, +1}) and…

Large Language Models Model Architecture

BitNet b1.58

BitNet b1.58 is a ternary-weight large language model architecture from Microsoft Research in which every weight is constrained to one of three values, -1, 0, or +1

Large Language Models Model Architecture

Differential Transformer

The Differential Transformer (often shortened to Diff Transformer or DIFF Transformer) is a decoder-only neural sequence architecture introduced by researchers at Microsoft Research and Tsinghua University in…

Model Architecture Transformer Models

LongNet

LongNet is a transformer variant introduced by Microsoft Research in July 2023 that is designed to scale attention to sequences exceeding one billion tokens while preserving performance on shorter inputs.

Model Architecture Transformer Models

LongRoPE

LongRoPE is a context-window extension technique for large language models (LLMs) that use rotary position embeddings (RoPE).

Large Language Models Model Architecture

YOCO (You Only Cache Once)

YOCO ("You Only Cache Once") is a decoder-decoder neural network architecture for large language models introduced by researchers at Microsoft Research and Tsinghua University in May 2024.

Large Language Models Model Architecture