AWQ (Activation-aware Weight Quantization)
Activation-aware Weight Quantization (AWQ) is a post-training quantization method for large language models that compresses weights to 4-bit (and optionally 3-bit) integers while keeping near-FP16 task…
Large Language Models
Explore language models, how they work, and the techniques used to build applications with them.
Explore articles
Browse subtopics: Deep Learning
Articles that also belong to these categories. Counts cover all of Large Language Models.
All articles (545)AI Models (184)Open Source AI (165)Chinese AI (92)Natural Language Processing (58)AI Companies (53)Machine Learning (49)Mixture of Experts (46)Multimodal AI (41)Reasoning Models (39)Artificial Intelligence (37)OpenAI (35)AI Benchmarks (33)Anthropic (31)Google DeepMind (26)AI Agents (22)AI Safety (21)Developer Tools (20)Meta AI (20)AI Code Generation (18)Small Language Models (18)Training & Optimization (18)AI Inference (16)Model Architecture (14)Generative AI (13)NVIDIA (13)Prompt Engineering (13)AI Research (12)Deep Learning (12)Google (12)Model Evaluation (12)Data & Datasets (11)AI Tools & Products (9)Conversational AI (9)Microsoft (9)Transformer Models (9)AI Infrastructure (7)Diffusion Models (7)AI Alignment (5)Enterprise AI (5)Research Organizations (5)AI Hardware (4)Healthcare AI (4)Information Retrieval (4)Interpretability (4)People (4)Mathematics (3)AI History (2)AI for Science (2)Algorithms (2)Computer Vision (2)Legal AI (2)Reinforcement Learning (2)Speech & Audio AI (2)AI Policy (1)AI Policy & Regulation (1)AI in Cybersecurity (1)AI in Gaming (1)Autonomous Vehicles (1)ChatGPT (1)China AI (1)Drug Discovery (1)Neural Networks (1)Robotics (1)Software Development (1)Video Generation (1)
Showing 1-12 of 12 articles
Chinchilla scaling laws
The Chinchilla scaling laws are a set of empirical findings published by DeepMind researchers in 2022 showing that, for a fixed compute budget, a large language model trains most efficiently when its number of…
Context window
A context window is the finite token sequence that a language model can process for one invocation.
In-Context Learning
In-context learning (ICL) is the ability of a large language model to learn a new task at inference time by conditioning on a prompt that contains a few input-output examples (demonstrations)
Instruction Tuning
Instruction tuning is the post-pretraining training stage in which a large language model (LLM) is fine-tuned on a curated collection of (instruction
Large Language Model
A large language model (LLM) is an artificial intelligence system built on a transformer neural network with billions to trillions of parameters, trained on massive text corpora to predict the next token in a…
Model merging
Model merging combines the parameters of multiple trained neural networks into a single unified model without any additional training.
Post-training
Post-training is the stage of large language model (LLM) development that comes after pre-training and turns a raw, general-purpose base model into an aligned, instruction-following AI assistant.
QLoRA
QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key vectors in two-dimensional subspaces
Scaling Laws
Neural scaling laws are empirical relationships that describe how a measured outcome changes as a neural network, its training data, or the computation used to train it becomes larger.
Speculative Decoding
Speculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…