Atlas (inference engine)
Atlas is an open-source large language model inference engine written in Rust and CUDA, built for desktop "unified memory" machines such as the NVIDIA DGX Spark (GB10) and AMD Strix Halo (Ryzen AI Max+ 395)…
Explore Developer Tools through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of Developer Tools.
Showing 1-24 of 24 articles
Atlas is an open-source large language model inference engine written in Rust and CUDA, built for desktop "unified memory" machines such as the NVIDIA DGX Spark (GB10) and AMD Strix Halo (Ryzen AI Max+ 395)…
Context caching is a large-language-model API feature that stores parts of a request's input (system prompts, instructions, attached documents, or earlier conversation turns) on the provider's infrastructure…
EXL2 (ExLlamaV2 format) is an open-source, mixed-bit weight-quantization format for compressing large language models so they run fast on a single consumer-class NVIDIA GPU.
Fireworks AI is an artificial intelligence infrastructure company that runs a high-performance inference platform for deploying and serving open large language models (LLMs), vision-language models, and…
FlashInfer is an open-source GPU kernel library and code-generation system for large language model inference.
FlashMLA is an open-source library of attention kernels from DeepSeek, built around Multi-head Latent Attention (MLA), the attention variant DeepSeek uses to shrink the KV cache in its large language models.
The Llama API is Meta's first-party hosted cloud service for running Llama models.
Luminal (Luminal AI Inc.) is a San Francisco company that builds an open-source AI inference compiler and sells inference capacity and licensed deployments built on it.
Magnitude is an open-source inference engine for running open-weight language models on local hardware, developed by Magnitude AI Inc., a San Francisco startup from the Y Combinator Summer 2025 batch.
Modular (legally Modular Inc, also referred to as Modular AI) is an American AI infrastructure software company that builds tools for running AI models on chips from many vendors without rewriting code for…
NVIDIA BioNeMo Inference Runtime (BioIR) is a GPU-accelerated Python library for biomolecular structure prediction inference.
NVIDIA Dynamo is an open-source, low-latency distributed inference serving framework designed to deploy and scale generative AI and reasoning models across large GPU clusters.
NVIDIA NIM (NVIDIA Inference Microservices) is a set of containerized, prebuilt-and-optimized model-serving microservices from NVIDIA that package an AI model, an optimized inference engine, and an…
NVIDIA Riva is a GPU-accelerated software development kit and family of containerized inference services for speech and translation AI, built by NVIDIA.
NVIDIA Triton Inference Server is open-source model deployment software that lets teams run trained models from any machine learning or deep learning framework on any processor (GPU, CPU, or other accelerator)…
Neywa Labs is a startup that builds software for running artificial intelligence models locally on personal devices, with a focus on Macs and other Apple silicon hardware.
OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel for optimizing and deploying deep learning inference across a heterogeneous set of Intel hardware…
Optimum Quanto, commonly referred to as Quanto, is a PyTorch-based quantization toolkit developed and maintained by Hugging Face that provides linear weight and activation quantization across multiple devices…
Project Maya is open-source software for running large mixture-of-experts language models on local hardware.
Prompt caching is a large language model (LLM) inference optimization that stores the computed key-value (KV) state of a repeated prompt prefix so it can be reused across API calls, cutting both cost and…
Splash is an open-source large language model inference engine for Macs with Apple silicon, released by the inference company Inco AI on 18 September 2026 under the Apache-2.0 license.
TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models, exposes them over gRPC and REST APIs, and manages…
Text Generation Inference (TGI) is an open-source toolkit developed by Hugging Face for deploying and serving large language models in production.
Ultrafast is OpenAI's premium speed tier, a processing option that generates output tokens faster than the company's Standard, Fast, Flex and Batch tiers at a correspondingly higher price.