Showing 61-71 of 71 articles
SmoothQuant
SmoothQuant is a training-free, accuracy-preserving post-training quantization (PTQ) method that enables 8-bit weight and 8-bit activation (W8A8) integer...
Large Language Models
Speculative Decoding
See also: transformer, large language model, inference, KV cache, autoregressive model, knowledge distillation Speculative decoding is a lossless inference...
Deep LearningLarge Language Models
Static inference
Static inference is a machine learning serving pattern in which a model generates a batch of predictions offline, ahead of time, and caches them so that...
MLOps
StreamingLLM
StreamingLLM is an inference-time technique that allows pretrained transformer language models, originally trained with a finite attention window, to process...
Large Language Models
Temperature sampling
Temperature sampling is the most common decoding control in large language models: a single hyperparameter, written T, that divides the model's output logits...
Statistics
TensorFlow Serving
TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models,...
Developer ToolsMLOps
Tenstorrent Galaxy Blackhole
Tenstorrent Galaxy Blackhole is an AI inference server built by Tenstorrent, the fabless semiconductor company led by chief executive Jim Keller. The system...
AI Hardware
Test-Time Training (TTT)
Test-Time Training (TTT) is a family of machine learning techniques in which a model updates a subset of its own parameters at inference time, optimizing a...
Machine LearningTraining & Optimization
Text Generation Inference (TGI)
Text Generation Inference (TGI) is an open-source toolkit developed by Hugging Face for deploying and serving large language models in production. Hugging Face...
Developer ToolsOpen Source AI
YaRN
YaRN (Yet another RoPE extensioN) is a compute-efficient method for extending the context window of large language models that use Rotary Position Embeddings...
Deep LearningModel Architecture
d-Matrix Corsair
Corsair is the first commercial AI accelerator product from d-Matrix, a Silicon Valley AI inference hardware startup based in Santa Clara, California....
AI Hardware