Explore AI Infrastructure through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of AI Infrastructure.
Showing 1-29 of 29 articles
AQLM, short for Additive Quantization of Language Models, is a weight-only post-training quantization method that compresses the weights of a large language model to roughly 2 to 3 bits per parameter while…
Amazon SageMaker is Amazon Web Services' fully managed machine learning platform for building, training, and deploying models at scale, first launched at AWS re:Invent on November 29, 2017 and rebranded in…
Attention-FFN disaggregation (AFD), also written attention-feedforward disaggregation or A/F disaggregation, is a large language model serving architecture that runs the attention sublayers and the…
Automatic differentiation (abbreviated AD, also called algorithmic differentiation, autodiff, or autograd) is a family of techniques for computing exact derivatives of a function specified by a computer program
Bittensor is a decentralized machine learning network that uses blockchain-based incentives to pay independent contributors for producing digital commodities such as model inference, training, data, and raw…
Chunked prefill is a scheduling technique for large language model serving that splits the processing of a long input prompt (the prefill) into smaller, fixed-size token chunks and combines each chunk with the…
Cloud TPU is Google Cloud's offering of Tensor Processing Units (TPUs), the family of custom application-specific integrated circuits (ASICs) that Google builds to accelerate machine learning training and…
Contrastive decoding (CD) is a decoding strategy for text generation from a large language model that selects tokens by contrasting two models of different sizes.
Dask is an open-source Python library for parallel and distributed computing that scales the familiar APIs of libraries such as NumPy, pandas, and scikit-learn to process larger-than-memory datasets.
Data parallelism is a distributed training technique in which the same neural network model is replicated across multiple processing units (typically GPUs), each device trains on a different shard of the input…
DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.
GEM (Generative Ads Recommendation Model) is a proprietary foundation model for advertising recommendation developed by Meta Platforms.
GPU computing is the use of a graphics processing unit (GPU) to perform general-purpose computation that was traditionally handled by the central processing unit (CPU).
KV cache quantization is a family of large language model inference optimizations that store the attention key and value (KV) cache in low-bit numeric formats, typically 2 to 4 bits per value
Google Cloud is the public cloud arm of Google and one of the three dominant providers of machine learning infrastructure, alongside Amazon Web Services and Microsoft Azure.
Microscaling (MX) formats are a family of low-precision number formats for machine learning in which a small block of values, normally 32 of them, shares one common scale factor while each value is stored in a…
Model parallelism is a distributed training and inference technique that splits a single neural network across multiple processing units so that no individual accelerator has to hold the entire model.
Prefix caching is an inference optimization for large language model serving that stores and reuses the key-value (KV) cache computed for a shared prompt prefix, so that the prefill for that prefix is computed…
Prompt lookup decoding (PLD), also called n-gram speculative decoding, is an inference acceleration method for large language models that speeds up text generation without changing the model or its outputs.
QuIP (Quantization with Incoherence Processing) is a family of weight-only post-training quantization methods for large language models developed in the RelaxML group at Cornell University
Quantization-aware training (QAT) is a model compression technique in which the effects of quantization are simulated during the training or fine-tuning of a neural network, so that the model learns parameter…
Ray is an open-source distributed computing framework, developed at the University of California, Berkeley's RISELab and commercialized by Anyscale, that lets developers scale Python and artificial…
Sanjay Ghemawat (born 1966) is an American computer scientist and software engineer, best known for co-creating the core distributed-systems infrastructure that powered Google's rise, including the Google File…
Self-speculative decoding is a family of speculative decoding methods that accelerate large language model (LLM) inference by using the target model itself, run in a cheaper reduced-depth mode
SpinQuant is a post-training quantization method for large language models that inserts learned rotation matrices into a transformer network to make its weights, activations, and KV cache easier to represent…
A TPU node is the legacy Google Cloud architecture for accessing Tensor Processing Unit (TPU) hardware, in which a user's virtual machine (VM) runs application code and communicates with a separate
A TPU Pod is a single Google supercomputer built from many Tensor Processing Unit (TPU) chips wired directly to each other by a high-speed Inter-Chip Interconnect (ICI) fabric arranged as a 2D or 3D torus, so…
A TPU worker is a virtual machine (VM) running Linux that has direct access to one or more Tensor Processing Unit (TPU) chips and executes the actual TPU computation on that attached hardware.
A vector database is a database that stores data as high-dimensional vectors (numerical embeddings produced by a machine learning model) and retrieves records by similarity rather than exact match