AWQ (Activation-aware Weight Quantization)
Activation-aware Weight Quantization (AWQ) is a post-training quantization method for large language models that compresses weights to 4-bit (and optionally 3-bit) integers while keeping near-FP16 task…
Deep Learning
Explore Deep Learning through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Large Language Models
Articles that also belong to these categories. Counts cover all of Deep Learning.
All articles (333)Machine Learning (187)Neural Networks (81)Computer Vision (53)Training & Optimization (41)Natural Language Processing (37)Generative AI (31)Model Architecture (21)Transformer Models (18)Artificial Intelligence (16)Large Language Models (12)People (12)Reinforcement Learning (12)Developer Tools (11)Mathematics (11)AI Inference (9)Interpretability (8)AI Infrastructure (7)Speech & Audio AI (7)AI Hardware (6)AI Companies (5)AI History (5)Google DeepMind (5)Open Source AI (5)AI Research (4)AI for Science (4)Algorithms (4)Model Evaluation (4)Robotics (4)AI Models (3)Google (3)Meta AI (3)OpenAI (3)AI Alignment (2)AI Tools & Products (2)Chinese AI (2)Data & Datasets (2)Healthcare AI (2)MLOps (2)Microsoft (2)Multimodal AI (2)NVIDIA (2)Software Development (2)Statistics (2)AI Benchmarks (1)AI Ethics (1)AI Safety (1)Data Science (1)Diffusion Models (1)Embodied AI (1)Image Generation (1)Information Retrieval (1)Programming Languages (1)Prompt Engineering (1)Video Generation (1)
Showing 1-12 of 12 articles
Chinchilla scaling laws
The Chinchilla scaling laws are a set of empirical findings published by DeepMind researchers in 2022 showing that, for a fixed compute budget, a large language model trains most efficiently when its number of…
Context window
A context window is the finite token sequence that a language model can process for one invocation.
In-Context Learning
In-context learning (ICL) is the ability of a large language model to learn a new task at inference time by conditioning on a prompt that contains a few input-output examples (demonstrations)
Instruction Tuning
Instruction tuning is the post-pretraining training stage in which a large language model (LLM) is fine-tuned on a curated collection of (instruction
Large Language Model
A large language model (LLM) is an artificial intelligence system built on a transformer neural network with billions to trillions of parameters, trained on massive text corpora to predict the next token in a…
Model merging
Model merging combines the parameters of multiple trained neural networks into a single unified model without any additional training.
Post-training
Post-training is the stage of large language model (LLM) development that comes after pre-training and turns a raw, general-purpose base model into an aligned, instruction-following AI assistant.
QLoRA
QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key vectors in two-dimensional subspaces
Scaling Laws
Neural scaling laws are empirical relationships that describe how a measured outcome changes as a neural network, its training data, or the computation used to train it becomes larger.
Speculative Decoding
Speculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…