Training & Optimization

Explore Training & Optimization through related topics and the articles other pages reference most.

Explore articles

Reset filters
Browse subtopics: AI Infrastructure

Articles that also belong to these categories. Counts cover all of Training & Optimization.

Showing 1-11 of 11 articles

Context Parallelism

Context Parallelism (CP) is a distributed training strategy that partitions the input sequence dimension of a transformer across multiple accelerators and uses ring-style point-to-point communication to…

AI Infrastructure

DeepSpeed

DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.

AI InfrastructureDeep Learning

Olmo-core

Olmo-core is an open-source Python library for modeling and large-scale distributed training with PyTorch. The Allen Institute for AI (Ai2) develops and maintains it under the Apache 2.0 license.

AI InfrastructureDeveloper Tools

Sequence Parallelism

Sequence parallelism (SP) is a family of distributed training techniques for transformer-based neural networks that partitions activations along the sequence (token) dimension across multiple accelerators…

AI InfrastructureDeep Learning

Tensor Core

A Tensor Core is a specialized execution unit inside NVIDIA GPUs that computes a small matrix multiplication and accumulation, D = A x B + C

AI ComputeAI Hardware