AI Inference

Explore AI Inference through related topics and the articles other pages reference most.

Explore articles

Reset filters
Browse subtopics: Developer Tools

Articles that also belong to these categories. Counts cover all of AI Inference.

Showing 1-24 of 24 articles

Context caching

Context caching is a large-language-model API feature that stores parts of a request's input (system prompts, instructions, attached documents, or earlier conversation turns) on the provider's infrastructure…

Developer ToolsLarge Language Models

Fireworks AI

Fireworks AI is an artificial intelligence infrastructure company that runs a high-performance inference platform for deploying and serving open large language models (LLMs), vision-language models, and…

AI CompaniesDeveloper Tools

FlashMLA

FlashMLA is an open-source library of attention kernels from DeepSeek, built around Multi-head Latent Attention (MLA), the attention variant DeepSeek uses to shrink the KV cache in its large language models.

AI InfrastructureDeveloper Tools

Luminal

Luminal (Luminal AI Inc.) is a San Francisco company that builds an open-source AI inference compiler and sells inference capacity and licensed deployments built on it.

AI CompaniesDeveloper Tools

Modular (company)

Modular (legally Modular Inc, also referred to as Modular AI) is an American AI infrastructure software company that builds tools for running AI models on chips from many vendors without rewriting code for…

AI CompaniesAI Infrastructure

NVIDIA NIM

NVIDIA NIM (NVIDIA Inference Microservices) is a set of containerized, prebuilt-and-optimized model-serving microservices from NVIDIA that package an AI model, an optimized inference engine, and an…

AI InfrastructureDeveloper Tools

Neywa Labs

Neywa Labs is a startup that builds software for running artificial intelligence models locally on personal devices, with a focus on Macs and other Apple silicon hardware.

AI CompaniesDeveloper Tools

OpenVINO

OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel for optimizing and deploying deep learning inference across a heterogeneous set of Intel hardware…

Developer ToolsOpen Source AI

Optimum-Quanto

Optimum Quanto, commonly referred to as Quanto, is a PyTorch-based quantization toolkit developed and maintained by Hugging Face that provides linear weight and activation quantization across multiple devices…

Developer ToolsOpen Source AI

Prompt Caching

Prompt caching is a large language model (LLM) inference optimization that stores the computed key-value (KV) state of a repeated prompt prefix so it can be reused across API calls, cutting both cost and…

Developer ToolsLarge Language Models

TensorFlow Serving

TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models, exposes them over gRPC and REST APIs, and manages…

Developer ToolsMLOps