Data Parallelism
Data parallelism is a distributed training technique in which the same neural network model is replicated across multiple processing units (typically GPUs), each device trains on a different shard of the input…
AI Infrastructure
Explore AI Infrastructure through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Deep Learning
Articles that also belong to these categories. Counts cover all of AI Infrastructure.
All articles (281)AI Hardware (98)AI Companies (57)Data Centers (43)Developer Tools (37)Open Source AI (37)NVIDIA (35)Machine Learning (29)AI Inference (26)Google (13)AI Energy (11)Chinese AI (10)MLOps (9)Training & Optimization (9)AI Agents (8)OpenAI (8)Computer Science (7)Deep Learning (7)Enterprise AI (7)Information Retrieval (7)Large Language Models (7)AI Policy & Regulation (6)People (6)Artificial Intelligence (5)Meta AI (5)Microsoft (4)Robotics (4)AI Code Generation (3)AI Research (3)AI Tools & Products (3)Anthropic (3)Generative AI (3)Mixture of Experts (3)Venture Capital (3)AI Models (2)Model Architecture (2)Model Evaluation (2)Programming Languages (2)AI in Gaming (1)Agentic Commerce (1)Data & Datasets (1)Data Science (1)Edge computing (1)Embodied AI (1)Finance AI (1)Hardware (1)Image Generation (1)Mathematics (1)Neural Networks (1)Physical AI (1)Research Organizations (1)Software Development (1)Video Generation (1)
Showing 1-7 of 7 articles
DeepSpeed
DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.
Fully Sharded Data Parallel (FSDP)
Fully Sharded Data Parallel (FSDP) is a distributed training technique implemented in PyTorch that shards a model's parameters, gradients, and optimizer states across data-parallel workers, allowing models…
GPU computing
GPU computing is the use of a graphics processing unit (GPU) to perform general-purpose computation that was traditionally handled by the central processing unit (CPU).
Model Parallelism
Model parallelism is a distributed training and inference technique that splits a single neural network across multiple processing units so that no individual accelerator has to hold the entire model.
Sequence Parallelism
Sequence parallelism (SP) is a family of distributed training techniques for transformer-based neural networks that partitions activations along the sequence (token) dimension across multiple accelerators…
Tensor Processing Unit (TPU)
A Tensor Processing Unit (TPU) is a family of custom application-specific integrated circuits developed by Google to accelerate machine-learning computation.