OctoAI
OctoAI (originally OctoML) was an American artificial intelligence infrastructure company that operated a generative-AI inference platform and, before its pivot
AI Inference
Explore AI Inference through related topics and the articles other pages reference most.
Explore articles
Browse subtopics (41)
Articles that also belong to these categories. Counts cover all of AI Inference.
All articles (97)AI Hardware (30)AI Infrastructure (26)Large Language Models (16)Developer Tools (15)NVIDIA (11)Machine Learning (10)Deep Learning (9)Open Source AI (9)AI Companies (8)Training & Optimization (8)MLOps (6)Model Architecture (4)Reasoning Models (4)Algorithms (3)Chinese AI (3)Reinforcement Learning (3)AI Alignment (2)AI Models (2)Anthropic (2)Artificial Intelligence (2)Data Centers (2)Edge computing (2)Google (2)Information Retrieval (2)Meta AI (2)AI Benchmarks (1)AI Research (1)AI Safety (1)Computer Science (1)Enterprise AI (1)Generative AI (1)Image Generation (1)Microsoft (1)Mixture of Experts (1)Natural Language Processing (1)Prompt Engineering (1)Small Language Models (1)Speech & Audio AI (1)Statistics (1)Transformer Models (1)Video Generation (1)
Showing 61-97 of 97 articles
Offline inference
Offline inference (also called batch inference, static inference, or bulk scoring) is the practice of running a trained machine learning model over a known set of inputs ahead of time and storing the resulting…
On-device AI
On-device AI is the practice of running machine learning models on the phone, laptop, watch, or embedded board a person is actually using, instead of sending the input to a remote data center.
Online inference
Online inference (also called dynamic inference, real-time inference, or on-demand prediction) is the practice of running a trained machine learning model synchronously inside a request path
OpenVINO
OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel for optimizing and deploying deep learning inference across a heterogeneous set of Intel hardware…
Optimum-Quanto
Optimum Quanto, commonly referred to as Quanto, is a PyTorch-based quantization toolkit developed and maintained by Hugging Face that provides linear weight and activation quantization across multiple devices…
PagedAttention
PagedAttention is a KV-cache memory management algorithm for serving large language models that applies the virtual-memory paging technique used by operating systems to the GPU, eliminating memory…
Positron AI
Positron AI is an American semiconductor startup headquartered in Reno, Nevada, that designs and manufactures purpose-built hardware for transformer inference.
Post-processing
In machine learning, post-processing is any operation applied to a model's raw outputs after the prediction step but before the results reach a downstream system or end user.
Product quantization
Product quantization (PQ) is a vector-compression technique for approximate nearest-neighbor (ANN) search that splits each high-dimensional vector into M equal sub-vectors and quantizes each sub-vector with…
Pruning
Pruning is a family of techniques used in machine learning and artificial intelligence to remove parts of a model or search space that are estimated to be unnecessary for accuracy or optimality.
QLoRA
QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…
Qualcomm AI200
Qualcomm AI200 is a rack-scale data-center accelerator for artificial intelligence inference, announced by Qualcomm on 27 October 2025 and slated for commercial availability in 2026 .
Qualcomm AI250
Qualcomm AI250 is a planned data-center artificial intelligence inference accelerator and rack-scale system announced by Qualcomm in late October 2025.
Quantization
Quantization is the representation of values from a large or continuous set by values from a smaller, usually finite set.
RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) is a post-training paradigm for large language models in which the reward signal comes from a deterministic
RadixAttention
RadixAttention is a KV cache management technique introduced in SGLang that uses a radix tree data structure to automatically share and reuse cached key-value tensors across inference requests.
Rebellions REBEL-Quad
REBEL-Quad is a chiplet-based AI inference accelerator developed by Rebellions, a South Korean AI-chip company.
SRAM
SRAM, or static random-access memory, is a semiconductor memory that stores each bit in a latch built from cross-coupled inverters.
Skeleton-of-Thought
Skeleton-of-Thought (SoT) is a prompting technique for large language models that reduces end-to-end generation latency by first eliciting a short outline of the answer (the "skeleton") and then expanding each…
SmoothQuant
SmoothQuant is a training-free, accuracy-preserving post-training quantization (PTQ) method that enables 8-bit weight and 8-bit activation (W8A8) integer inference for large language models without retraining…
Snapdragon AR1
Snapdragon AR1 is a family of Qualcomm system-on-chip platforms built specifically for smart glasses.
SparDA
SparDA (Sparse Decoupled Attention) is an add-on architecture for long-context large language model inference proposed by researchers at NVIDIA in a paper posted to arXiv on 3 June 2026.
Speculative Decoding
Speculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…
Static inference
Static inference is a machine learning serving pattern in which a model generates a batch of predictions offline, ahead of time, and caches them so that applications read the precomputed answer from storage…
StreamingLLM
StreamingLLM is an inference-time technique that allows pretrained transformer language models, originally trained with a finite attention window
Temperature sampling
Temperature sampling is the most common decoding control in large language models: a single hyperparameter, written T
TensorFlow Serving
TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models, exposes them over gRPC and REST APIs, and manages…
Tenstorrent Galaxy Blackhole
Tenstorrent Galaxy Blackhole is an AI inference server built by Tenstorrent, the fabless semiconductor company led by chief executive Jim Keller.
Test-Time Training (TTT)
Test-Time Training (TTT) is a family of machine learning techniques in which a model updates a subset of its own parameters at inference time
Text Generation Inference (TGI)
Text Generation Inference (TGI) is an open-source toolkit developed by Hugging Face for deploying and serving large language models in production.
Top-k sampling
Top-k sampling is a decoding strategy for autoregressive language models that restricts each generation step to the k most probable next tokens.
Xiaomi XRING O100
Xiaomi XRING O100 is a dedicated AI accelerator designed by Xiaomi for local large-model inference on consumer devices.
YaRN
YaRN (Yet another RoPE extensioN) is a compute-efficient method for extending the context window of large language models that use Rotary Position Embeddings (RoPE).
d-Matrix
d-Matrix is a privately held American semiconductor company headquartered in Santa Clara, California that builds accelerators, I/O cards and software for AI inference in data centers.
d-Matrix Corsair
Corsair is the first commercial AI accelerator product from d-Matrix, a Silicon Valley AI inference hardware startup based in Santa Clara, California.
d-Matrix Raptor
Raptor is the second-generation AI inference accelerator from d-Matrix, a Santa Clara semiconductor startup, and the first commercial chip built on the company's 3D stacked digital in-memory compute technology