ExLlamaV2 (EXL2)
EXL2 (ExLlamaV2 format) is an open-source, mixed-bit weight-quantization format for compressing large language models so they run fast on a single consumer-class NVIDIA GPU.
AI Inference
Explore AI Inference through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Open Source AI
Articles that also belong to these categories. Counts cover all of AI Inference.
All articles (97)AI Hardware (30)AI Infrastructure (26)Large Language Models (16)Developer Tools (15)NVIDIA (11)Machine Learning (10)Deep Learning (9)Open Source AI (9)AI Companies (8)Training & Optimization (8)MLOps (6)Model Architecture (4)Reasoning Models (4)Algorithms (3)Chinese AI (3)Reinforcement Learning (3)AI Alignment (2)AI Models (2)Anthropic (2)Artificial Intelligence (2)Data Centers (2)Edge computing (2)Google (2)Information Retrieval (2)Meta AI (2)AI Benchmarks (1)AI Research (1)AI Safety (1)Computer Science (1)Enterprise AI (1)Generative AI (1)Image Generation (1)Microsoft (1)Mixture of Experts (1)Natural Language Processing (1)Prompt Engineering (1)Small Language Models (1)Speech & Audio AI (1)Statistics (1)Transformer Models (1)Video Generation (1)
Showing 1-9 of 9 articles
FlashInfer
FlashInfer is an open-source GPU kernel library and code-generation system for large language model inference.
GraphRAG
GraphRAG is a graph-based approach to retrieval-augmented generation developed by Microsoft Research, first described publicly on February 13, 2024 and formalized in the paper "From Local to Global: A Graph…
NVIDIA BioNeMo Inference Runtime
NVIDIA BioNeMo Inference Runtime (BioIR) is a GPU-accelerated Python library for biomolecular structure prediction inference.
NVIDIA Dynamo
NVIDIA Dynamo is an open-source, low-latency distributed inference serving framework designed to deploy and scale generative AI and reasoning models across large GPU clusters.
NVIDIA TensorRT-LLM
NVIDIA TensorRT-LLM is an open-source library developed by nvidia for high-performance inference of large language models on NVIDIA GPUs.
OpenVINO
OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel for optimizing and deploying deep learning inference across a heterogeneous set of Intel hardware…
Optimum-Quanto
Optimum Quanto, commonly referred to as Quanto, is a PyTorch-based quantization toolkit developed and maintained by Hugging Face that provides linear weight and activation quantization across multiple devices…
Text Generation Inference (TGI)
Text Generation Inference (TGI) is an open-source toolkit developed by Hugging Face for deploying and serving large language models in production.