Context caching
Context caching is a large-language-model API feature that stores parts of a request's input (system prompts, instructions, attached documents, or earlier conversation turns) on the provider's infrastructure…
AI Inference
Explore AI Inference through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Developer Tools
Articles that also belong to these categories. Counts cover all of AI Inference.
All articles (97)AI Hardware (30)AI Infrastructure (26)Large Language Models (16)Developer Tools (15)NVIDIA (11)Machine Learning (10)Deep Learning (9)Open Source AI (9)AI Companies (8)Training & Optimization (8)MLOps (6)Model Architecture (4)Reasoning Models (4)Algorithms (3)Chinese AI (3)Reinforcement Learning (3)AI Alignment (2)AI Models (2)Anthropic (2)Artificial Intelligence (2)Data Centers (2)Edge computing (2)Google (2)Information Retrieval (2)Meta AI (2)AI Benchmarks (1)AI Research (1)AI Safety (1)Computer Science (1)Enterprise AI (1)Generative AI (1)Image Generation (1)Microsoft (1)Mixture of Experts (1)Natural Language Processing (1)Prompt Engineering (1)Small Language Models (1)Speech & Audio AI (1)Statistics (1)Transformer Models (1)Video Generation (1)
Showing 1-15 of 15 articles
ExLlamaV2 (EXL2)
EXL2 (ExLlamaV2 format) is an open-source, mixed-bit weight-quantization format for compressing large language models so they run fast on a single consumer-class NVIDIA GPU.
Fireworks AI
Fireworks AI is an artificial intelligence infrastructure company that runs a high-performance inference platform for deploying and serving open large language models (LLMs), image generation models, audio…
FlashInfer
FlashInfer is an open-source GPU kernel library and code-generation system for large language model inference.
Llama API
The Llama API is Meta's first-party hosted cloud service for running Llama models.
Luminal
Luminal (Luminal AI Inc.) is a San Francisco company that builds an open-source AI inference compiler and sells inference capacity and licensed deployments built on it.
NVIDIA BioNeMo Inference Runtime
NVIDIA BioNeMo Inference Runtime (BioIR) is a GPU-accelerated Python library for biomolecular structure prediction inference.
NVIDIA Dynamo
NVIDIA Dynamo is an open-source, low-latency distributed inference serving framework designed to deploy and scale generative AI and reasoning models across large GPU clusters.
NVIDIA NIM
NVIDIA NIM (NVIDIA Inference Microservices) is a set of containerized, prebuilt-and-optimized model-serving microservices from NVIDIA that package an AI model, an optimized inference engine, and an…
NVIDIA Riva
NVIDIA Riva is a GPU-accelerated software development kit and family of containerized inference services for speech and translation AI, built by NVIDIA.
NVIDIA Triton Inference Server
NVIDIA Triton Inference Server is open-source model deployment software that lets teams run trained models from any machine learning or deep learning framework on any processor (GPU, CPU, or other accelerator)…
OpenVINO
OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel for optimizing and deploying deep learning inference across a heterogeneous set of Intel hardware…
Optimum-Quanto
Optimum Quanto, commonly referred to as Quanto, is a PyTorch-based quantization toolkit developed and maintained by Hugging Face that provides linear weight and activation quantization across multiple devices…
TensorFlow Serving
TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models, exposes them over gRPC and REST APIs, and manages…
Text Generation Inference (TGI)
Text Generation Inference (TGI) is an open-source toolkit developed by Hugging Face for deploying and serving large language models in production.