Context Parallelism
Context Parallelism (CP) is a distributed training strategy that partitions the input sequence dimension of a transformer across multiple accelerators and uses ring-style point-to-point communication to…
AI Infrastructure
Explore AI Infrastructure through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Training & Optimization
Articles that also belong to these categories. Counts cover all of AI Infrastructure.
All articles (281)AI Hardware (98)AI Companies (57)Data Centers (43)Developer Tools (37)Open Source AI (37)NVIDIA (35)Machine Learning (29)AI Inference (26)Google (13)AI Energy (11)Chinese AI (10)MLOps (9)Training & Optimization (9)AI Agents (8)OpenAI (8)Computer Science (7)Deep Learning (7)Enterprise AI (7)Information Retrieval (7)Large Language Models (7)AI Policy & Regulation (6)People (6)Artificial Intelligence (5)Meta AI (5)Microsoft (4)Robotics (4)AI Code Generation (3)AI Research (3)AI Tools & Products (3)Anthropic (3)Generative AI (3)Mixture of Experts (3)Venture Capital (3)AI Models (2)Model Architecture (2)Model Evaluation (2)Programming Languages (2)AI in Gaming (1)Agentic Commerce (1)Data & Datasets (1)Data Science (1)Edge computing (1)Embodied AI (1)Finance AI (1)Hardware (1)Image Generation (1)Mathematics (1)Neural Networks (1)Physical AI (1)Research Organizations (1)Software Development (1)Video Generation (1)
Showing 1-9 of 9 articles
DeepSpeed
DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.
Expert Parallelism
Expert Parallelism (EP) is a model-parallelism strategy specific to Mixture of Experts (MoE) neural networks in which the individual expert sub-networks (typically feed-forward blocks) are sharded across…
Fully Sharded Data Parallel (FSDP)
Fully Sharded Data Parallel (FSDP) is a distributed training technique implemented in PyTorch that shards a model's parameters, gradients, and optimizer states across data-parallel workers, allowing models…
NCCL (NVIDIA Collective Communications Library)
NCCL, the NVIDIA Collective Communications Library, is a library of topology-aware communication primitives for NVIDIA GPU systems.
Pipeline Parallelism
Pipeline parallelism (often abbreviated PP) is a distributed training strategy that splits the layers of a deep neural network across multiple accelerator devices so that each device holds one contiguous block…
Sequence Parallelism
Sequence parallelism (SP) is a family of distributed training techniques for transformer-based neural networks that partitions activations along the sequence (token) dimension across multiple accelerators…
Tensor Core
A Tensor Core is a specialized execution unit inside NVIDIA GPUs that computes a small matrix multiplication and accumulation, D = A x B + C
Tensor Parallelism
Tensor parallelism (TP) is a distributed training technique that splits the individual weight matrices of a neural network layer across multiple devices, so that each device computes a partial result that is…