Citation and evidence

Self-hosting LLMs

31 min full readUpdated 64 references

This article's verification

Report a problem with this article

Edit article
More

Cite this article

Self-hosting LLMs is the practice of running large language models on infrastructure that an organization operates itself, whether GPU servers it owns, GPU instances it rents or a cluster in its own cloud account, instead of buying access to a model through a provider's per-token API. [1][2] This article covers server-side deployment for teams and products. Running a model on a personal computer is covered in Local AI and How to Run LLMs Locally. Self-hosting depends on models whose weights can be downloaded (open weights) and on inference server software such as vLLM, SGLang and the llama.cpp server, with cluster layers such as KServe, Ray Serve, NVIDIA Dynamo and llm-d above them. [3][4][5][6][7][8][9]

The reasons given in the research literature are privacy and data residency, cost control at high volume, customization, and less dependence on a single provider. [10][1][11] The documented costs are the operations work, the risk of paying for idle GPUs, a capability gap between open-weight models and the best closed models, and the security of endpoints that are often exposed to the internet with weak default settings. [10][12][13][14]

Using an open-weight model is not the same as self-hosting it. Menlo Ventures, which calls these models open-source, estimated that their share of enterprise LLM tokens rose from 11% to 19% between September 2025 and September 2026 while their share of model spending fell from 6% to 4%. It added that much of that volume "runs on inference clouds like Fireworks and Modal", which are third-party endpoints. The report marks both companies as Menlo Ventures portfolio companies. [15]

What counts as self-hosting

Pan and colleagues describe three deployment paradigms for LLMs: cloud, on-premise and hybrid. In the cloud paradigm a provider such as OpenAI, Anthropic or Google runs the model and charges per million input and output tokens. In the on-premise paradigm the organization buys the hardware, pays for electricity and runs an open-weight model itself. A hybrid setup keeps sensitive workloads local and sends other traffic to a cloud API. [1] Rented hardware sits between the two. Knoop and Holtmann put it this way: "Dedicated cloud GPU instances offer improved privacy but with limited guarantees and ongoing costs". [2]

VariantWhat the organization operatesDocumented example
On-premises hardwareGPUs, power, the serving software and the modelThe cost models of Pan et al. and of Knoop and Holtmann assume purchased GPUs [1][2]
Rented GPU instancesThe serving software and the model, on hardware billed by a cloudThe vLLM production stack publishes deployment tutorials for Lambda Labs, AWS EKS and Google Cloud [16]
Managed Kubernetes in the organization's cloud accountContainers and the model inside its own accountNVIDIA validates NIM on the managed Kubernetes services of AWS, Google Cloud, Azure and Oracle [17]
Air-gapped siteEverything, with no outbound internet accessNIM documents pre-staging model files to an offline cache and running without internet access or API keys [17]

Expanded article table

The Gateway API Inference Extension, a Kubernetes project, uses the term in its own description: it "optimizes self-hosting Generative Models on Kubernetes". [18] Xu and colleagues describe self-hosted LLM serving as "emerging as a distinct category of Internet service". [19]

Most of the servers described below expose an OpenAI-compatible API. [3][5][20][7]

The serving stack

A self-hosted deployment has up to three layers: an inference engine that executes the model on the accelerator, an optional packaged or general-purpose server around it, and an orchestration layer that routes requests across replicas and scales them. [20][8] The table lists widely used components. License entries refer to the software's own code and were read from each repository's LICENSE file. They say nothing about the models the software runs. Star counts are from the GitHub API on November 18, 2026.

SoftwareLayerCode licenseGitHub starsNotes
vLLMEngine and serverApache 2.097,178Open source; started at UC Berkeley [3]
SGLangEngine and serverApache 2.039,040Open source [4]
llama.cpp serverEngine and serverMIT135,648Open source; C/C++, runs on CPUs and GPUs [5]
TensorRT-LLMEngineApache 2.0, with exceptions for bundled third-party code15,101For NVIDIA GPUs. Its LICENSE file puts one visual-generation directory under the LTX-2 Community License Agreement, which is not on the OSI list [21][50]
Text Generation InferenceEngine and serverApache 2.010,877Repository archived on 2026-03-21 [22]
Triton Inference ServerGeneral-purpose serverBSD 3-Clause11,153Open source; many frameworks, not only LLMs [23]
NVIDIA NIMPackaged containerProprietary (NVIDIA AI Product Agreement)Not applicableNot open source; packages the open-source vLLM or SGLang engine behind NVIDIA's own layer [20][24][64]
KServeKubernetes platformApache 2.06,245Open source; CNCF incubating project [6][25]
Ray ServeCluster serving libraryApache 2.044,322 (Ray)Open source; part of Ray [26][7]
NVIDIA DynamoOrchestrationApache 2.08,522Open source; sits above the engines [8]
llm-dKubernetes orchestrationApache 2.05,301Open source; CNCF sandbox project [9]

Expanded article table

Inference engines

vLLM was first developed in the Sky Computing Lab at UC Berkeley and introduced with the PagedAttention paper at SOSP 2023. [3][27] Its README lists an OpenAI-compatible server with Anthropic Messages API and gRPC support, continuous batching, prefix caching, many quantization formats, multi-LoRA serving, and support for NVIDIA, AMD and Intel GPUs as well as CPUs and, through plugins, other accelerators. The most recent release as of November 18, 2026 was v0.34.0, published on November 16. [3]

SGLang was introduced in a December 2023 paper by Lianmin Zheng and colleagues. The paper described a runtime built around RadixAttention, a technique for reusing the KV cache across requests, and reported up to 6.4 times the throughput of the systems it was compared with. [28] The project now describes itself as an inference framework for language, vision-language and diffusion models. [4]

The llama.cpp HTTP server is a C/C++ program that serves GGUF models on GPUs and CPUs. Its README lists OpenAI-compatible chat, responses and embeddings routes, parallel decoding for several users, continuous batching (on by default), a Prometheus-compatible /metrics endpoint (disabled unless --metrics is set) and optional API keys. By default the server listens on 127.0.0.1, port 9931. The default port was 8080 until a change merged on October 9, 2026, and the project's Docker images still set port 8080. [5][61]

TensorRT-LLM is NVIDIA's library for LLM inference on its own GPUs. The README describes a PyTorch-based architecture with specialized kernels and a Python API that covers single-GPU through multi-node setups. [21]

Hugging Face's Text Generation Inference (TGI), which the company used in production for Hugging Chat and its inference services, is no longer developed. Its README opens with the notice "text-generation-inference is now in maintenance mode" and recommends vLLM, SGLang, llama.cpp or MLX instead. The last release, v3.3.7, is dated December 19, 2025, and GitHub records the repository as archived (read-only) since March 21, 2026. [22]

NVIDIA's packaged servers

Triton Inference Server is a general-purpose model server that supports TensorRT, PyTorch, ONNX and other frameworks, with HTTP and gRPC protocols based on the KServe protocol. [23] The Triton project maintains separate backends for vLLM and for TensorRT-LLM. [29][30]

NVIDIA NIM is a set of containers that package a model with an inference engine. The documentation for the 2.x line says each image contains one engine, either vLLM or SGLang, behind NVIDIA's own layers for configuration, model download and health checks, whereas the 1.x containers bundled several backends. [20] The engines are open source; NIM itself is proprietary software, and its documentation states that using a NIM means agreeing to the NVIDIA AI Product Agreement. [64] The documentation says the base NIM offering and the NIM Certified Feature Branch are available at no charge under their license terms, and that the NIM Certified Production Branch requires an NVIDIA AI Enterprise subscription. [24]

Cluster orchestration

An engine by itself runs one model replica. Larger deployments add a layer that handles replicas, routing and scaling.

  • KServe is a Kubernetes platform for generative and predictive model inference. Its README lists vLLM and llm-d as LLM backends, an OpenAI-compatible protocol, model caching and request-based autoscaling. [6] The Cloud Native Computing Foundation accepted it at the incubating level on September 29, 2025. [25]
  • Ray Serve LLM builds on Ray Serve for multi-node serving. Its documentation lists an OpenAI-compatible API, multi-model deployment with autoscaling and load balancing, multi-LoRA serving on a shared base model, and vLLM and SGLang as engines. [7]
  • NVIDIA Dynamo describes itself as an "orchestration layer above inference engines" that coordinates SGLang, TensorRT-LLM or vLLM workers and adds disaggregated serving and routing that takes KV-cache contents into account. [8]
  • llm-d is a Kubernetes-based stack that adds cache-aware and load-aware routing, tiered KV-cache offloading and prefill/decode disaggregation above model servers such as vLLM and SGLang. Its README names Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA as founders. Version 1.0.0 was released on October 20, 2026. [9]
  • The vLLM production stack is a Helm-based reference deployment for Kubernetes with a request router, a metrics dashboard and KV-cache offloading. [16]

Throughput concepts

The mechanisms below determine how many simultaneous requests a GPU can serve, and the serving engines expose them as settings. [27][31]

Continuous batching

An LLM produces one token per forward pass, so a request needs as many passes as it has output tokens. Yu and colleagues observed in the Orca paper (OSDI 2022) that servers which batch whole requests perform poorly on this workload: finished requests cannot leave the batch, and new ones must wait for it to end. They proposed iteration-level scheduling, in which the scheduler runs one iteration at a time and can change the batch between iterations, and reported 36.9 times the throughput of NVIDIA FasterTransformer at the same latency on a 175-billion-parameter GPT-3 model. [32] A 2023 Anyscale engineering post described the same approach under the name continuous batching, "also known as dynamic batching, or batching with iteration-level scheduling". [33] vLLM, the llama.cpp server and TGI all list continuous batching among their features. [3][5][22]

KV cache, PagedAttention and prefix caching

During generation the server keeps a key-value cache for every active request. Kwon and colleagues wrote that this memory "is huge and grows and shrinks dynamically", and that managing it badly wastes memory through fragmentation and limits the batch size. Their PagedAttention algorithm borrows the paging technique of operating systems, and the paper reported two to four times the throughput of FasterTransformer and Orca at the same latency. [27] When the cache runs out of space, vLLM preempts some requests and recomputes them later, which its documentation warns can hurt end-to-end latency. [31]

Prefix caching reuses cache blocks between requests that begin with the same tokens. vLLM's documentation calls it "almost a free lunch" because it does not change model outputs. [34] SGLang's RadixAttention is another implementation of the same idea. [28] The saving has a security cost when several users share a server, described in the security section below.

Prefill, the processing of the prompt, and decode, the generation of output, load the GPU differently. vLLM's chunked prefill splits long prompts into pieces and batches them with decoding requests, and one setting trades inter-token latency against time to first token. [31]

Quantization

vLLM's documentation states the trade plainly: "Quantized models take less memory at the cost of lower precision." [35] The formats vLLM lists include FP8, NVFP4, AWQ, GPTQ and GGUF. [3] In one published measurement on NVIDIA consumer GPUs, Knoop and Holtmann found that NVFP4 gave 1.6 times the throughput of unquantized BF16 weights with 41% less energy and a quality loss of 2 to 4% on their benchmarks. [2]

Splitting a model across GPUs

vLLM's guidance is to use one GPU if the model fits, tensor parallelism when it fits on one multi-GPU node, and tensor parallelism combined with pipeline parallelism when it needs several nodes. It recommends pipeline parallelism for GPUs without an NVLink interconnect. [36] The project's security guide says communications between the nodes of a multi-node deployment are "insecure by default" and must be protected by placing the nodes on an isolated network. [37]

Metrics

NVIDIA's benchmarking documentation defines the measures most often used. Time to first token (TTFT) covers queuing, prompt processing and network delay. Inter-token latency (ITL) is the average gap between output tokens. Total tokens per second counts output across all simultaneous requests. As concurrency rises, system throughput increases until the GPU saturates, while each user's own throughput falls. The same page warns that tools define these metrics differently, so results are comparable only when the definitions match. [38]

Sizing a deployment

Memory is the first constraint. The weights must fit in GPU memory, and what remains holds the KV cache, which sets how many tokens can be in flight at once. On NVIDIA and AMD GPUs vLLM by default sizes the KV cache from the memory that is free after startup. It logs the cache size in tokens and an estimate of the maximum number of concurrent requests at the model's full context length; if those numbers are below the requirement, its documentation says to add GPUs or nodes. [31][36] The other levers are a quantized checkpoint, a shorter maximum context and a lower cap on concurrent sequences. [35] AI Wiki's LLM VRAM calculator does this arithmetic for a chosen model size, quantization, context length, concurrency and GPU count.

Throughput has to be measured on the intended workload, because it varies widely with prompt length. Knoop and Holtmann served the same 8-billion-parameter model (Qwen3 8B with NVFP4 weights) with vLLM and measured more than 4,400 tokens per second on two RTX 5090 cards for short-context, high-concurrency traffic, against 411 tokens per second on one card for retrieval-augmented generation with an 8k-token context. [2]

Load-testing tools include GuideLLM, a vLLM project that simulates workloads against OpenAI-compatible servers and reports full distributions of TTFT and ITL, and NVIDIA's AIPerf. [39][38] For hardware comparisons, SemiAnalysis publishes InferenceX, an Apache 2.0 repository of end-to-end inference serving benchmarks with a public dashboard. [40]

Cost compared with per-token APIs

A per-token API is a variable cost: nothing is paid when nothing is sent. Self-hosting on purchased hardware is mostly a fixed cost, so the cost per token depends on how fully the hardware is used. [1][2] The two published comparisons below differ in what they assume about utilization and in which APIs they compare against. AI Wiki's AI cost calculator estimates the API side of the comparison for a given workload.

Pan et al. (2025)

Pan and colleagues modeled the total cost of local deployment as the purchase price of the GPUs plus electricity for eight hours a day, 20 days a month. They set against it the API bill for the number of tokens that hardware could generate in those hours (one third input, two thirds output), so the comparison assumes that the hardware's full output is used during those hours. Across what the paper counts as 54 deployment scenarios, which set nine open-weight models against commercial APIs from OpenAI, Anthropic, xAI and Google, they found that small models (the paper's sub-30B class) often broke even within three months, medium models in 3.8 to 34 months and large models in 3.5 to 69.3 months, stretching to five to nine years against aggressively priced APIs such as Gemini 2.5 Pro. They concluded that on-premise deployment is viable mainly for organizations processing 50 million tokens a month or more, or with strict data-residency requirements. The model leaves out staffing and maintenance, which the authors list as future work. They describe deployment as "a continuous optimization problem rather than a one-time decision" because models, hardware and prices keep changing. [1]

Knoop and Holtmann (2026)

Knoop and Holtmann benchmarked four open-weight models on NVIDIA consumer GPUs with vLLM and reported electricity-only costs of $0.001 to $0.04 per million tokens, which they put at 40 to 200 times cheaper than budget-tier cloud APIs at January 2026 prices, with hardware paying for itself in under four months at 30 million tokens a day. The same paper shows how sensitive the result is to volume: at 1 million tokens a day, an RTX 5090 needs 8,772 days to break even against GPT-5 nano. The authors list the limits of the estimate. It uses manufacturer list prices, counts GPU power only (they put CPU, memory and cooling at a further 20 to 40%), and measures steady synthetic load, not production traffic with its variable request patterns and cold starts. By the authors' estimate, amortizing the hardware over two years adds $0.01 to $0.05 per million tokens, depending on GPU tier and utilization. [2]

Moving API prices

The API side of the comparison does not stand still. Epoch AI found in March 2025 that the price of reaching a fixed level of benchmark performance had fallen by between 9 and 900 times a year, depending on the level, with a median of 50 times. [41] Menlo Ventures estimated that the average price of an open-weight token fell 3.3 times in the year to September 2026, to $0.13 per million, while the blended price of closed-model tokens fell 24% to $0.70. [15] Break-even estimates such as those above are calculated against API prices at one date (January 2026 in Knoop and Holtmann's case). [2]

Utilization

Idle capacity is a documented source of waste. Xiang and colleagues wrote in the Aegaeon paper (SOSP 2025) that serving sporadic, unpredictable requests with dedicated GPU instances "results in substantial resource waste". In their own model marketplace, pooling GPUs across models at token granularity cut the number of GPUs needed to serve tens of models from 1,192 to 213. [12] Vangala and Malik name "challenges in GPU utilization, workload routing, and reliability" as the difficulties of in-house models and report that scale-to-zero automation with routing lowered GPU cost per query by up to 33% against static deployments of the same models. They also note that scale-to-zero brings cold-start latency. [10] Engineers at T-Tech described a related problem: adopting newer models without decommissioning their predecessors enlarges the serving fleet and fragments a finite GPU pool. Their answer was to consolidate traffic from more than 200 internal applications onto one model. [11]

Reasons organizations give

  • Data residency and compliance. Tsymboi and colleagues open their paper with the words "Data-residency constraints force enterprises to self-host LLMs". [11] Pan and colleagues write that on-premise deployments offer full control and compliance for sensitive domains such as healthcare, finance and law. [1] Paloniemi and colleagues, who moved a procurement-bid assessment system from ChatGPT to an on-premise environment, cite the EU's General Data Protection Regulation as an example of the requirements involved. They report that the on-premise version went into use at their industry partner where the ChatGPT version could not, mainly because of cost and partly because of privacy and security. [42]
  • Privacy. Prompts and documents are processed on infrastructure the organization controls. Knoop and Holtmann qualify this for rented hardware, where they see improved privacy "but with limited guarantees". [2]
  • Cost at high, steady volume. See the previous section. [1][2]
  • Customization. Servers such as vLLM and Ray Serve LLM can serve many LoRA adapters on one base model. [3][7] Menlo Ventures reported that the share of AI builders doing post-training on open weights rose from 6% in 2025 to 10% in 2026 and described the motive as lower costs and less dependence on frontier labs. [15] T-Tech reported that a post-trained 32-billion-parameter model beat a model about seven times larger on its in-house benchmarks and now handles 50% of its platform's traffic. [11]
  • Avoiding lock-in. Pan and colleagues list "the difficulty of switching service providers" among the concerns that drive interest in local deployment. [1]
  • Disconnected environments. Some sites cannot make outbound connections at all, a case NIM's documentation addresses with air-gapped deployment. [17]

Lower latency is not automatic. The operator controls the settings that trade time to first token against inter-token latency, and Knoop and Holtmann found consumer GPUs adequate for most of the small-business workloads they tested "except latency-critical long-context RAG, where high-end GPUs remain essential". [31][2]

Drawbacks and risks

Operations work

The organization takes on the running of the service. NVIDIA says NIM is designed for "teams that do not have the bandwidth to track the fast-moving model-serving ecosystem". [20] Menlo Ventures' mid-2025 survey named "the technical complexity of deploying open-source models" among the reasons the enterprise share of those models had stagnated at the time. [43] The software also changes under its users. TGI's repository was archived three months after its last release, and NIM 2.0 replaced the multi-backend container of the 1.x line with a single-engine design, for which NVIDIA publishes a migration guide. [22][20] Neither of the cost models cited above prices this labor. [1][2]

Utilization risk

Dedicated capacity is paid for whether or not requests arrive. The Aegaeon and Vangala and Malik results in the cost section show how much hardware static, per-model deployments can leave idle. [12][10]

Model quality gap

Self-hosting restricts the choice to models with downloadable weights. Epoch AI's November 2024 report found that the best open models had trailed the best closed ones by 5 to 22 months on benchmarks, about one year overall. [44] Its October 2025 update, using a composite capability index, put the average lag at about three months. [13] Menlo Ventures wrote in November 2026 that near-frontier models, among them the open-weight Kimi K3 and DeepSeek V4, "approach frontier performance at a fraction of the price" yet together held just under 5% of LLM API spending, and that buyers "continue to prioritize capability over cost in model selection". [15] Post-training on an organization's own tasks can narrow the gap for a specific workload, as in the T-Tech case above. [11]

Security of exposed endpoints

A self-hosted endpoint is secured by whoever deployed it, and measurement studies have found many that are open to anyone.

  • Hou and colleagues scanned for public LLM services in April 2025 and found 320,102 across 15 frameworks. More than 40% of endpoints used plain HTTP, and Ollama answered more than 35% of unauthenticated API requests. They attribute the exposure to insecure defaults and misconfiguration. [14]
  • Xu and colleagues, in a paper published in the proceedings of the 2026 ACM Internet Measurement Conference, probed Ollama's port 11434 across the IPv4 address space daily from February 2025 to February 2026 and counted 152,137 exposed IP addresses in total over 362 observation days. For five selected vulnerabilities, only 0.43 to 2.90% of hosts running a vulnerable version upgraded in place. Cloud and hosting providers dominated the networks involved, and the authors call the result "a structural exposure surface: persistent, growing, and heavily concentrated". [19]
  • A separate industry scan by SentinelLABS and Censys, published on January 29, 2026, counted 175,108 unique Ollama hosts in 130 countries over 293 days of scanning, with a persistent core of about 23,000 hosts that generated most of the activity. More than 48% of the hosts advertised tool-calling capabilities. This count and the 152,137 addresses of Xu and colleagues come from different scanners and observation windows, so they are two separate measurements. [62]
  • Elzer and colleagues ran a honeypot imitating the Ollama API for 84 days and recorded 290,887 interactions from 2,793 source addresses. Most were automated discovery and model enumeration, but the logs also held remote-code-execution and cryptocurrency-mining payloads, resource-exhaustion attempts and prompt injection. [45]

Default network settings differ between engines. The llama.cpp server binds to 127.0.0.1 unless --host is given. [5] vLLM's --host argument has no default value, and its launcher then binds an empty host address, which it logs as 0.0.0.0, so a vLLM server started without --host listens on port 8000 on every IPv4 network interface of the machine. [60]

Server engines document their own limits. vLLM's security guide says the --api-key option protects only some URL paths and warns: "Do not rely exclusively on --api-key for securing access to vLLM." It recommends a reverse proxy that allowlists endpoints and adds authentication and rate limiting, and a firewall that blocks every incoming connection except to the API port. The same guide says that vLLM "does not provide isolation between tenants that share the same server process" and that prefix caching creates a timing side channel between users of one server (CVE-2025-46570), for which it offers a per-request cache salt. [37] As of November 18, 2026 the vLLM repository listed 126 published security advisories, 99 of them dated 2026 and 36 rated high or critical. [46]

The surrounding infrastructure has been attacked as well. In March 2024 Oligo Security reported the ShadowRay campaign, in which thousands of publicly reachable Ray servers were compromised through a jobs API that lacks authorization. Anyscale, Ray's maintainer, had disputed the underlying CVE on the grounds that the behavior is an intended feature. [47] The Ray 2.60.0 documentation says that Ray 2.59 turns token authentication on by default for local clusters and that Ray 2.61 extends the default to all clusters, including remote and multi-node ones. [63] In August 2025 Wiz Research disclosed a chain of three vulnerabilities in Triton Inference Server that could give a remote, unauthenticated attacker code execution; NVIDIA released a patch. [48]

Model files are a further input to check. Hugging Face warns that loading a pickle file can execute arbitrary code and suggests loading models only from users and organizations the operator trusts. [49] How to Run LLMs Locally covers the verification of downloaded files.

License checks for commercial serving

Two sets of terms apply to a self-hosted deployment, and they are independent. The first covers the serving software. The open-source components in the table above carry Apache 2.0, MIT or BSD 3-Clause licenses, all of which appear on the Open Source Initiative's list of approved open-source licenses. [3][5][23][50] The one exception, a directory of bundled code inside the TensorRT-LLM repository, is noted in the table. [21] The second covers the model weights, whose terms are set by the model's developer. Models are called open-weight in this article because downloadable weights do not by themselves satisfy the OSI's Open Source AI Definition, which also asks for the training code and detailed information about the training data. [51]

Weight licenses differ most on the question of serving a model to third parties. The examples below were read from the license texts in November 2026.

Model familyWeights licenseTerms that bear on a hosted service
gpt-oss, Gemma 4Apache 2.0A permissive, OSI-approved license with no clause specific to hosted services. The gpt-oss repository also carries a one-paragraph usage policy that requires compliance with applicable law [52][53][50]
DeepSeek V3.2MITA permissive, OSI-approved license with no clause specific to hosted services [54][50]
Llama 4Llama 4 Community LicenseAnyone who distributes the model, or a product or service that contains it, must provide a copy of the agreement and display "Built with Llama". An acceptable use policy applies. A licensee whose products or services (or those of its affiliates) had more than 700 million monthly active users in the calendar month before the Llama 4 release date must request a license from Meta, which Meta may grant at its sole discretion [55]
Gemma 3 and earlierGemma Terms of UseOffering the model as a hosted service counts as distribution. The operator must include the use restrictions in its own terms, notify its users of them and give third-party recipients a copy of the agreement [56]
Mistral Large 4Mistral Ecosystem Open-Weight LicenseA licensee that operates a "Model as a Service" business needs a separate commercial license from Mistral before any commercial use. The license defines the term as giving third parties access to the model for inference or post-training, for instance through an API or hosted endpoint, in a way that lets them control the inputs, parameters or training data. End-user products that only embed the model in specific features, internal use, and use through Mistral's own products fall outside the requirement [57]

Expanded article table

The Hugging Face repositories for Llama 4 and Gemma 3 are gated, so access must be requested and granted before the files can be downloaded. [58][59] License texts are revised over time (the Gemma Terms of Use were last modified on April 1, 2026), so the version in force for a given download matters. [56] Open-Weight LLM License Comparison goes through the main model licenses clause by clause, the list of open-weight models records the license of each model AI Wiki covers, and the open-weight model directory lists downloadable models with their weights licenses.

Continue reading

More from See also

More to read

Topics:AI InferenceAI InfrastructureLarge Language ModelsOpen Source AI

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Guanzhong Pan, Vishal Chodnekar, Abinas Roy, Haibo Wang. "A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services". arXiv:2509.18101, August 30, 2025 (version 3 read). arxiv.org/...2509.18101
  2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12Jonathan Knoop, Hendrik Holtmann. "Private LLM Inference on Consumer Blackwell GPUs: A Practical Guide for Cost-Effective Local Deployment in SMEs". arXiv:2601.09527, January 14, 2026. arxiv.org/...2601.09527
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9vLLM project. "vllm-project/vllm" (README and repository metadata). GitHub, accessed November 18, 2026. github.com/...vllm
  4. ^1 ^2 ^3SGLang project. "sgl-project/sglang" (README and repository metadata). GitHub, accessed November 18, 2026. github.com/...sglang
  5. ^1 ^2 ^3 ^4 ^5 ^6 ^7ggml-org. "LLaMA.cpp HTTP Server" (tools/server/README.md in ggml-org/llama.cpp). GitHub, accessed November 18, 2026. github.com/...README.md
  6. ^1 ^2 ^3KServe project. "kserve/kserve" (README and repository metadata). GitHub, accessed November 18, 2026. github.com/...kserve
  7. ^1 ^2 ^3 ^4 ^5Ray project. "Serving LLMs". Ray 2.60.0 documentation, accessed November 18, 2026. docs.ray.io/...llm
  8. ^1 ^2 ^3 ^4NVIDIA. "ai-dynamo/dynamo" (README, LICENSE and repository metadata). GitHub, accessed November 18, 2026. github.com/...dynamo
  9. ^1 ^2 ^3llm-d project. "llm-d/llm-d" (README and repository metadata). GitHub, accessed November 18, 2026. github.com/...llm-d
  10. ^1 ^2 ^3 ^4Bhanu Prakash Vangala, Tanu Malik. "Efficient Multi-Model Orchestration for Self-Hosted Large Language Models". arXiv:2512.22402, December 26, 2025. arxiv.org/...2512.22402
  11. ^1 ^2 ^3 ^4 ^5Olga Tsymboi, Dmitrii Stoianov, Ramil Latypov, Danil Taranets, Daniil Dryabin, Mikhail Gashkov, Viktor Zelenkovskiy, Aleksandr Fida, Gleb Alektorov, Nikita Gulyakov, Arthur Babkin, Aleksandr Medvedev, Pavel Gein, Anatolii Potapov. "From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix". arXiv:2609.01572, September 1, 2026. arxiv.org/...2609.01572
  12. ^1 ^2 ^3Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, Jingren Zhou. "Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market". Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP 2025), pp. 1030-1045, October 2025. doi.org/...3731569.3764815
  13. ^1 ^2Luke Emberson. "Open-weight models lag state-of-the-art by around 3 months on average". Epoch AI Data Insights, October 30, 2025. epoch.ai/...open-weights-vs-closed-weights-models
  14. ^1 ^2Xinyi Hou, Jiahao Han, Yanjie Zhao, Haoyu Wang. "Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Study". arXiv:2505.02502, May 5, 2025. arxiv.org/...2505.02502
  15. ^1 ^2 ^3 ^4Tim Tully, Joff Redfern, Deedy Das, Derek Xiao. "2026: The State of Enterprise AI". Menlo Ventures, November 3, 2026. menlovc.com/...2026-the-state-of-enterprise-ai
  16. ^1 ^2vLLM project. "vllm-project/production-stack" (README). GitHub, accessed November 18, 2026. github.com/...production-stack
  17. ^1 ^2 ^3NVIDIA. "Enterprise-Grade Inference Software Stack". NVIDIA NIM for LLM and VLM documentation, accessed November 18, 2026. docs.nvidia.com/...enterprise-grade
  18. ^Kubernetes SIGs. "kubernetes-sigs/gateway-api-inference-extension" (README). GitHub, accessed November 19, 2026. github.com/...gateway-api-inference-extension
  19. ^1 ^2Zuyao Xu, Xiang Li, Yuqi Qiu, Lu Sun. "Ollama in the Wild: A Longitudinal Measurement of Exposed Ollama LLM Endpoints at Internet Scale". Proceedings of the 2026 ACM Internet Measurement Conference (IMC '26), Karlsruhe, October 12-16, 2026, pp. 514-528, doi:10.1145/3777912.3839823. Preprint: arXiv:2609.07115, September 7, 2026. arxiv.org/...2609.07115
  20. ^1 ^2 ^3 ^4 ^5 ^6NVIDIA. "NVIDIA NIM for LLM and VLM Documentation" (overview). Accessed November 18, 2026. docs.nvidia.com/...overview
  21. ^1 ^2 ^3NVIDIA. "NVIDIA/TensorRT-LLM" (README, LICENSE and repository metadata). GitHub, accessed November 18, 2026. github.com/...TensorRT-LLM
  22. ^1 ^2 ^3 ^4Hugging Face. "huggingface/text-generation-inference" (README and repository metadata). GitHub, accessed November 18, 2026. github.com/...text-generation-inference
  23. ^1 ^2 ^3NVIDIA. "triton-inference-server/server" (README and repository metadata). GitHub, accessed November 18, 2026. github.com/...server
  24. ^1 ^2NVIDIA. "NIM Offerings". NVIDIA NIM for LLM and VLM documentation, accessed November 18, 2026. docs.nvidia.com/...nim-offerings
  25. ^1 ^2Cloud Native Computing Foundation. "KServe". CNCF project page, accessed November 18, 2026. cncf.io/...kserve
  26. ^Ray project. "ray-project/ray" (repository metadata). GitHub, accessed November 18, 2026. github.com/...ray
  27. ^1 ^2 ^3Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica. "Efficient Memory Management for Large Language Model Serving with PagedAttention". SOSP 2023; arXiv:2309.06180, September 12, 2023. arxiv.org/...2309.06180
  28. ^1 ^2Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng. "SGLang: Efficient Execution of Structured Language Model Programs". arXiv:2312.07104, December 12, 2023. arxiv.org/...2312.07104
  29. ^NVIDIA. "triton-inference-server/vllm_backend". GitHub, accessed November 18, 2026. github.com/...vllm_backend
  30. ^NVIDIA. "triton-inference-server/tensorrtllm_backend". GitHub, accessed November 18, 2026. github.com/...tensorrtllm_backend
  31. ^1 ^2 ^3 ^4 ^5vLLM project. "Optimization and Tuning". vLLM documentation (docs/configuration/optimization.md), accessed November 18, 2026. github.com/...optimization.md
  32. ^Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun. "Orca: A Distributed Serving System for Transformer-Based Generative Models". 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), July 2022. usenix.org/...yu
  33. ^Cade Daniel, Chen Shen, Eric Liang, Richard Liaw. "How continuous batching enables 23x throughput in LLM inference while reducing p50 latency". Anyscale Blog, June 22, 2023. anyscale.com/...continuous-batching-llm-inference
  34. ^vLLM project. "Automatic Prefix Caching". vLLM documentation (docs/design/prefix_caching.md), accessed November 19, 2026. github.com/...prefix_caching.md
  35. ^1 ^2vLLM project. "Conserving Memory". vLLM documentation (docs/configuration/conserving_memory.md), accessed November 18, 2026. github.com/...conserving_memory.md
  36. ^1 ^2vLLM project. "Parallelism and Scaling". vLLM documentation (docs/serving/parallelism_scaling.md), accessed November 18, 2026. github.com/...parallelism_scaling.md
  37. ^1 ^2vLLM project. "Security". vLLM documentation (docs/usage/security.md), accessed November 18, 2026. github.com/...security.md
  38. ^1 ^2NVIDIA. "Metrics". NVIDIA NIM Benchmarking documentation, accessed November 18, 2026. docs.nvidia.com/...metrics
  39. ^vLLM project. "vllm-project/guidellm" (README). GitHub, accessed November 19, 2026. github.com/...guidellm
  40. ^SemiAnalysis. "SemiAnalysisAI/InferenceX" (README and repository metadata). GitHub, accessed November 18, 2026. github.com/...InferenceX
  41. ^Ben Cottier, Ben Snodin, David Owen, Tom Adamczewski. "LLM inference prices have fallen rapidly but unequally across tasks". Epoch AI Data Insights, March 12, 2025. epoch.ai/...llm-inference-price-trends
  42. ^Teemu Paloniemi, Manu Setälä, Tommi Mikkonen. "Porting an LLM based Application from ChatGPT to an On-Premise Environment". arXiv:2504.07907, April 10, 2025 (International Conference on Software Reuse, ICSR 2025). arxiv.org/...2504.07907
  43. ^Tim Tully, Joff Redfern, Deedy Das, Derek Xiao. "2025 Mid-Year LLM Market Update: Foundation Model Landscape + Economics". Menlo Ventures, July 31, 2025. menlovc.com/...2025-mid-year-llm-market-update
  44. ^Ben Cottier, Josh You, Natalia Martemianova, David Owen. "How far behind are open models?". Epoch AI, November 4, 2024. epoch.ai/...open-models-report
  45. ^Karina Elzer, Niklas Netterstrøm Johansen, Emmanouil Vasilomanolakis. "OllamaDrama: Designing and Deploying a Honeypot to Measure Attacks on Exposed LLM Infrastructure". arXiv:2609.29757, September 24, 2026. arxiv.org/...2609.29757
  46. ^vLLM project. "Security Advisories" (vllm-project/vllm). GitHub, accessed November 18, 2026. github.com/...advisories
  47. ^Avi Lumelsky, Gal Elbaz, Guy Kaplan. "ShadowRay: First Known Attack Campaign Targeting AI Workloads Actively Exploited In The Wild". Oligo Security, March 26, 2024. oligo.security/...s-actively-exploited-in-the-wild
  48. ^Ronen Shustin, Nir Ohfeld. "Breaking NVIDIA Triton: CVE-2025-23319 - A Vulnerability Chain Leading to AI Server Takeover". Wiz Blog, August 4, 2025. wiz.io/...n-cve-2025-23319-vuln-chain-to-ai-server
  49. ^Hugging Face. "Pickle Scanning". Hugging Face Hub documentation, accessed November 18, 2026. huggingface.co/...security-pickle
  50. ^1 ^2 ^3 ^4Open Source Initiative. "Licenses" (OSI Approved Licenses). Accessed November 19, 2026. opensource.org/licenses
  51. ^Open Source Initiative. "The Open Source AI Definition - 1.0". Accessed November 19, 2026. opensource.org/...open-source-ai-definition
  52. ^OpenAI. "openai/gpt-oss-120b" (model repository, LICENSE and USAGE_POLICY files). Hugging Face, accessed November 19, 2026. huggingface.co/...gpt-oss-120b
  53. ^Google. "Apache License 2.0" (the Gemma 4 license page, linked from the google/gemma-4-31B-it model card on Hugging Face). Gemma documentation, accessed November 19, 2026. ai.google.dev/...gemma_4_license
  54. ^DeepSeek. "deepseek-ai/DeepSeek-V3.2" (model repository, LICENSE file). Hugging Face, accessed November 19, 2026. huggingface.co/...DeepSeek-V3.2
  55. ^Meta. "Llama 4 Community License Agreement". Llama 4 version effective date April 5, 2025; accessed November 18, 2026. llama.com/...license
  56. ^1 ^2Google. "Gemma Terms of Use". Google AI for Developers, last modified April 1, 2026. ai.google.dev/...terms
  57. ^Mistral AI. "Mistral Ecosystem Open-Weight License" (LICENSE file of mistralai/Mistral-Large-4-1T-A52B). Hugging Face, accessed November 18, 2026. huggingface.co/...LICENSE
  58. ^Meta. "meta-llama/Llama-4-Maverick-17B-128E-Instruct" (model repository, access gate). Hugging Face, accessed November 19, 2026. huggingface.co/...Llama-4-Maverick-17B-128E-Instruct
  59. ^Google. "google/gemma-3-27b-it" (model repository, access gate). Hugging Face, accessed November 19, 2026. huggingface.co/...gemma-3-27b-it
  60. ^vLLM project. "vllm/entrypoints/launchers/launcher.py" (server socket setup; the `host` and `port` defaults are defined in vllm/entrypoints/common/cli_args.py). GitHub, accessed November 19, 2026. github.com/...launcher.py
  61. ^Xuan-Son Nguyen (ngxson). "server: use port 9931 by default". Pull request #30159, ggml-org/llama.cpp, GitHub, merged October 9, 2026. github.com/...30159
  62. ^Gabriel Bernadett-Shapiro, Silas Cutler. "Silent Brothers | Ollama Hosts Form Anonymous AI Network Beyond Platform Guardrails". SentinelLABS (SentinelOne), January 29, 2026, updated March 6, 2026. Archived October 10, 2026. web.archive.org/...work-beyond-platform-guardrails
  63. ^Ray project. "Ray token authentication". Ray 2.60.0 documentation, accessed November 19, 2026. docs.ray.io/...token-auth
  64. ^1 ^2NVIDIA. "Legal and Acknowledgements". NVIDIA NIM for LLM and VLM documentation, accessed November 19, 2026. docs.nvidia.com/...legal

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

1 revision · v1 · 6,172 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 19, 2026: 168 claims checked against 78 cited and independent sources; 3 material and 10 minor corrections applied before publication.

Cite this page: AI Wiki. "Self-hosting LLMs." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/self_hosting_llms

Edit article