Developer Tools

Explore Developer Tools through related topics and the articles other pages reference most.

Explore articles

Reset filters
Browse subtopics: AI Inference

Articles that also belong to these categories. Counts cover all of Developer Tools.

Showing 1-24 of 24 articles

Atlas (inference engine)

Atlas is an open-source large language model inference engine written in Rust and CUDA, built for desktop "unified memory" machines such as the NVIDIA DGX Spark (GB10) and AMD Strix Halo (Ryzen AI Max+ 395)…

AI InferenceOpen Source AI

Context caching

Context caching is a large-language-model API feature that stores parts of a request's input (system prompts, instructions, attached documents, or earlier conversation turns) on the provider's infrastructure…

AI InferenceLarge Language Models

ExLlamaV2 (EXL2)

EXL2 (ExLlamaV2 format) is an open-source, mixed-bit weight-quantization format for compressing large language models so they run fast on a single consumer-class NVIDIA GPU.

AI InferenceOpen Source AI

Fireworks AI

Fireworks AI is an artificial intelligence infrastructure company that runs a high-performance inference platform for deploying and serving open large language models (LLMs), vision-language models, and…

AI CompaniesAI Inference

FlashMLA

FlashMLA is an open-source library of attention kernels from DeepSeek, built around Multi-head Latent Attention (MLA), the attention variant DeepSeek uses to shrink the KV cache in its large language models.

AI InferenceAI Infrastructure

Luminal

Luminal (Luminal AI Inc.) is a San Francisco company that builds an open-source AI inference compiler and sells inference capacity and licensed deployments built on it.

AI CompaniesAI Inference

Modular (company)

Modular (legally Modular Inc, also referred to as Modular AI) is an American AI infrastructure software company that builds tools for running AI models on chips from many vendors without rewriting code for…

AI CompaniesAI Inference

NVIDIA Dynamo

NVIDIA Dynamo is an open-source, low-latency distributed inference serving framework designed to deploy and scale generative AI and reasoning models across large GPU clusters.

AI InferenceAI Infrastructure

NVIDIA NIM

NVIDIA NIM (NVIDIA Inference Microservices) is a set of containerized, prebuilt-and-optimized model-serving microservices from NVIDIA that package an AI model, an optimized inference engine, and an…

AI InferenceAI Infrastructure

NVIDIA Riva

NVIDIA Riva is a GPU-accelerated software development kit and family of containerized inference services for speech and translation AI, built by NVIDIA.

AI InferenceEnterprise AI

Neywa Labs

Neywa Labs is a startup that builds software for running artificial intelligence models locally on personal devices, with a focus on Macs and other Apple silicon hardware.

AI CompaniesAI Inference

OpenVINO

OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel for optimizing and deploying deep learning inference across a heterogeneous set of Intel hardware…

AI InferenceOpen Source AI

Optimum-Quanto

Optimum Quanto, commonly referred to as Quanto, is a PyTorch-based quantization toolkit developed and maintained by Hugging Face that provides linear weight and activation quantization across multiple devices…

AI InferenceOpen Source AI

Prompt Caching

Prompt caching is a large language model (LLM) inference optimization that stores the computed key-value (KV) state of a repeated prompt prefix so it can be reused across API calls, cutting both cost and…

AI InferenceLarge Language Models

TensorFlow Serving

TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models, exposes them over gRPC and REST APIs, and manages…

AI InferenceMLOps