AI Inference

Explore AI Inference through related topics and the articles other pages reference most.

Explore articles

Browse subtopics (52)

Articles that also belong to these categories. Counts cover all of AI Inference.

Showing 61-120 of 131 articles

Model Compression

Model compression is a family of techniques that reduce the size, memory footprint, and computational cost of machine learning models while preserving as much of their original accuracy as possible.

Modular (company)

Modular (legally Modular Inc, also referred to as Modular AI) is an American AI infrastructure software company that builds tools for running AI models on chips from many vendors without rewriting code for…

AI CompaniesAI Infrastructure

NVIDIA DGX Spark

NVIDIA DGX Spark is a compact, deskside AI development system, marketed by NVIDIA as a "personal AI supercomputer," that puts a Grace Blackwell superchip on a desktop and delivers up to 1 petaFLOP (1,000 TOPS)…

AI ComputeAI Hardware

NVIDIA Groq LPX Rack

NVIDIA Groq 3 LPX is a rack-scale inference accelerator that NVIDIA introduced at GTC 2026, built around 256 Groq Language Processing Units and designed to sit beside Vera Rubin NVL72 racks as a dedicated…

AI ComputeAI Hardware

NVIDIA NIM

NVIDIA NIM (NVIDIA Inference Microservices) is a set of containerized, prebuilt-and-optimized model-serving microservices from NVIDIA that package an AI model, an optimized inference engine, and an…

AI InfrastructureDeveloper Tools

NVIDIA Picasso

NVIDIA Picasso is a cloud-based generative AI foundry from NVIDIA for building, training, and deploying visual generative models that produce images, video, and 3D content from text prompts.

AI HardwareAI Infrastructure

NVIDIA Rubin CPX

NVIDIA Rubin CPX is a class of GPU announced by NVIDIA on September 9, 2025, purpose-built to accelerate the compute-heavy "context" phase of large-model inference.

AI ComputeAI Hardware

Neywa Labs

Neywa Labs is a startup that builds software for running artificial intelligence models locally on personal devices, with a focus on Macs and other Apple silicon hardware.

AI CompaniesDeveloper Tools

NormalFloat 4-bit (NF4)

NormalFloat 4-bit (NF4) is a 4-bit numerical data type for storing the weights of deep neural networks, introduced in the 2023 QLoRA paper by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer…

Training & Optimization

OctoAI

OctoAI (originally OctoML) was an American artificial intelligence infrastructure company that operated a generative-AI inference platform and, before its pivot

AI CompaniesAI Infrastructure

Offline inference

Offline inference (also called batch inference, static inference, or bulk scoring) is the practice of running a trained machine learning model over a known set of inputs ahead of time and storing the resulting…

MLOps

On-device AI

On-device AI is the practice of running machine learning models on the phone, laptop, watch, or embedded board a person is actually using, instead of sending the input to a remote data center.

AI HardwareMachine Learning

Online inference

Online inference (also called dynamic inference, real-time inference, or on-demand prediction) is the practice of running a trained machine learning model synchronously inside a request path

MLOps

OpenVINO

OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel for optimizing and deploying deep learning inference across a heterogeneous set of Intel hardware…

Developer ToolsOpen Source AI

Optimum-Quanto

Optimum Quanto, commonly referred to as Quanto, is a PyTorch-based quantization toolkit developed and maintained by Hugging Face that provides linear weight and activation quantization across multiple devices…

Developer ToolsOpen Source AI

OrcaRouter

OrcaRouter is a hosted API gateway and routing service for large language models and other generative models.

PagedAttention

PagedAttention is a KV-cache memory management algorithm for serving large language models that applies the virtual-memory paging technique used by operating systems to the GPU, eliminating memory…

AI InfrastructureModel Architecture

Positron AI

Positron AI is an American semiconductor startup headquartered in Reno, Nevada, that designs and manufactures purpose-built hardware for transformer inference.

AI CompaniesAI Compute

Post-processing

In machine learning, post-processing is any operation applied to a model's raw outputs after the prediction step but before the results reach a downstream system or end user.

MLOps

PrismML

PrismML (also styled Prism ML; legal name Prism ML, Inc.) is an American artificial intelligence company, based in Pasadena, California

AI CompaniesOpen Source AI

Prompt Caching

Prompt caching is a large language model (LLM) inference optimization that stores the computed key-value (KV) state of a repeated prompt prefix so it can be reused across API calls, cutting both cost and…

Developer ToolsLarge Language Models

Pruning

Pruning is a family of techniques used in machine learning and artificial intelligence to remove parts of a model or search space that are estimated to be unnecessary for accuracy or optimality.

Machine LearningTraining & Optimization

QLoRA

QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…

Deep LearningLarge Language Models

Qualcomm AI200

Qualcomm AI200 is a rack-scale data-center accelerator for artificial intelligence inference, announced by Qualcomm on 27 October 2025 and slated for commercial availability in 2026.

AI ComputeAI Hardware

Qualcomm AI250

Qualcomm AI250 is a planned data-center artificial intelligence inference accelerator and rack-scale system announced by Qualcomm in late October 2025.

AI ComputeAI Hardware

SRAM

SRAM, or static random-access memory, is a semiconductor memory that stores each bit in a latch built from cross-coupled inverters.

AI HardwareAI Infrastructure

Skeleton-of-Thought

Skeleton-of-Thought (SoT) is a prompting technique for large language models that reduces end-to-end generation latency by first eliciting a short outline of the answer (the "skeleton") and then expanding each…

Prompt Engineering

SmoothQuant

SmoothQuant is a training-free, accuracy-preserving post-training quantization (PTQ) method that enables 8-bit weight and 8-bit activation (W8A8) integer inference for large language models without retraining…

Large Language Models

SparDA

SparDA (Sparse Decoupled Attention) is an add-on architecture for long-context large language model inference proposed by researchers at NVIDIA in a paper posted to arXiv on 3 June 2026.

Large Language ModelsModel Architecture

Static inference

Static inference is a machine learning serving pattern in which a model generates a batch of predictions offline, ahead of time, and caches them so that applications read the precomputed answer from storage…

MLOps

StreamingLLM

StreamingLLM is an inference-time technique that allows pretrained transformer language models, originally trained with a finite attention window

Large Language Models

Temperature sampling

Temperature sampling is the most common decoding control in large language models: a single hyperparameter, written T, that divides the model's output logits before the softmax function so that the formula…

Statistics

TensorFlow Serving

TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models, exposes them over gRPC and REST APIs, and manages…

Developer ToolsMLOps