# How to Run LLMs Locally

> Source: https://aiwiki.ai/wiki/how_to_run_llms_locally
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: AI Inference, Developer Tools, Large Language Models, Open Source AI
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "How to Run LLMs Locally." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/how_to_run_llms_locally
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

This guide explains **how to run LLMs locally**: downloading the weights of a [large language model](https://aiwiki.ai/wiki/large_language_model) (LLM) and performing [inference](https://aiwiki.ai/wiki/inference) on hardware you control, instead of sending prompts to a hosted API. As of November 2026 the usual routes are [Ollama](https://aiwiki.ai/wiki/ollama), [LM Studio](https://aiwiki.ai/wiki/lmstudio), [llama.cpp](https://aiwiki.ai/wiki/llama_cpp), [MLX](https://aiwiki.ai/wiki/mlx) on [Apple silicon](https://aiwiki.ai/wiki/apple_silicon), [Jan](https://aiwiki.ai/wiki/jan_ai) and, for a GPU server, [vLLM](https://aiwiki.ai/wiki/vllm), with [Open WebUI](https://aiwiki.ai/wiki/open_webui) as a browser front end for any of them. [1][2][3][4][5][6][7] The work comes down to four decisions: whether the machine has enough memory, which model and [quantization](https://aiwiki.ai/wiki/quantization) level to download, which runtime to use, and whether to expose the model to other programs through an API. Two checks come before any serious use: that the downloaded files are what they claim to be, and that the model's license allows what you plan to do with it.

[Hugging Face](https://aiwiki.ai/wiki/hugging_face)'s documentation lists four reasons people run models on their own machines: prompts are not sent to a remote server, speed depends on local hardware instead of a provider's servers or connection, the model can be configured freely, and there is no API bill. [8] The costs are covered in the limitations section at the end.

The commands below are copied from each project's official documentation as it stood on November 19, 2026. Command names, default ports and flags change between releases. The llama.cpp README, for example, now starts the server with `llama serve`, while the server's own documentation still describes the `llama-server` binary. [3][9] Check the linked documentation before scripting against any of them. For model recommendations by memory tier, see [Best Local and On-Device LLMs](https://aiwiki.ai/wiki/best_local_llms); for the broader concept see [local AI](https://aiwiki.ai/wiki/local_ai), and for production deployment see [self-hosting LLMs](https://aiwiki.ai/wiki/self_hosting_llms).

## The routes at a glance

| Tool | What it is | Software license | Runs on | Models it loads | Default local address | GitHub stars (2026-11-19) |
|---|---|---|---|---|---|---|
| [Ollama](https://aiwiki.ai/wiki/ollama) | Command-line tool and desktop app with its own model library | [MIT](https://aiwiki.ai/wiki/mit_license) | macOS, Windows, Linux | [GGUF](https://aiwiki.ai/wiki/gguf); imports [safetensors](https://aiwiki.ai/wiki/safetensors) | `127.0.0.1:11434` | 187,575 |
| [LM Studio](https://aiwiki.ai/wiki/lmstudio) | Desktop app, plus the `lms` command-line tool and a headless daemon | Proprietary terms (the `lms` tool is MIT) | Apple silicon Macs; x64 and ARM Windows; x64 and ARM64 Linux | GGUF through llama.cpp; MLX models on Apple silicon | `127.0.0.1`, port 1234 in the documentation's examples | 5,440 (`lms` only) |
| [llama.cpp](https://aiwiki.ai/wiki/llama_cpp) | C/C++ inference engine with a command-line chat and an HTTP server | MIT | Windows, macOS, Linux | GGUF | `127.0.0.1:9931` | 135,648 |
| [MLX](https://aiwiki.ai/wiki/mlx) LM | Python package for text generation and fine-tuning | MIT | Apple silicon Macs | Hugging Face repositories, including pre-converted `mlx-community` models | `localhost:8080` | 7,694 |
| [Jan](https://aiwiki.ai/wiki/jan_ai) | Desktop chat app with a local API server | [Apache 2.0](https://aiwiki.ai/wiki/apache_license) | macOS, Windows, Linux | llama.cpp models; MLX models on Apple silicon | `127.0.0.1:1337` | 45,808 |
| [Open WebUI](https://aiwiki.ai/wiki/open_webui) | Web interface that connects to Ollama or to any OpenAI-compatible server | Open WebUI License (BSD-3-Clause terms plus a branding clause) | macOS, Linux, Windows, through Docker or Python | None of its own; it talks to a backend | UI on port 3000 (Docker) or 8080 (pip) | 159,039 |
| [vLLM](https://aiwiki.ai/wiki/vllm) | Serving engine for GPUs | Apache 2.0 | Linux | Hugging Face model repositories | `localhost:8000` in the documentation; every interface unless `--host` is set | 97,177 |

Sources for the table: each project's documentation, license file and GitHub repository. [1][10][11][12][2][13][14][15][16][17][3][18][19][9][4][20][5][21][22][23][24][6][25][74] The licenses in this table cover the software only. Each model's weights carry their own license, discussed below.

## Step 1: check that the machine has enough memory

Memory decides what can run. The llama.cpp documentation states that models are fully loaded into memory, so a model needs about as much RAM as its file takes on disk. [26] The size of the weights follows from the parameter count and the precision each weight is stored at:

$$\text{weights (bytes)} \approx \frac{N \times b}{8}$$

where $N$ is the number of parameters and $b$ is the average number of bits stored per weight. The llama.cpp documentation gives worked figures for [Llama 3.1](https://aiwiki.ai/wiki/llama_3_1): the 8B model takes 32.1 GB in its original form and 4.9 GB at the Q4_K_M quantization, the 70B model 280.9 GB and 43.1 GB, and the 405B model 1,625.1 GB and 249.1 GB. [26] The same page measures Q4_K_M at an average of 4.8944 bits per weight on the 8B model. [26]

### Context needs memory too

The weights are not the whole requirement. The runtime also keeps a [KV cache](https://aiwiki.ai/wiki/kv_cache) that grows with the [context window](https://aiwiki.ai/wiki/context_window). Ollama's context length page says that a larger context length increases the memory needed to run a model, and that the default depends on the VRAM Ollama finds: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. [27] The project's FAQ gives a different answer, a flat default of 4096 tokens; the help text for `OLLAMA_CONTEXT_LENGTH` in Ollama's source code agrees with the context length page ("default: 4k/32k/256k based on VRAM"). [12][75] The context length page recommends at least 64,000 tokens for web search, agents and coding tools, which is far above the 4k default that applies below 24 GiB. [27] Serving requests in parallel multiplies the allocation: in Ollama's example, a 2K context with 4 parallel requests becomes an 8K context. [12] [KV-cache quantization](https://aiwiki.ai/wiki/kv_cache_quantization), which Ollama offers when Flash Attention is enabled, reduces the cost. Its `q8_0` setting uses about half the memory of the default `f16` cache and `q4_0` about a quarter, with a precision loss that the documentation calls very small for the first and small to medium for the second. [12]

### Where the memory is

With a discrete [GPU](https://aiwiki.ai/wiki/gpu), the aim is to fit the whole model in VRAM. Ollama can spill into system RAM when VRAM runs short, but its quickstart warns that responses may then be slower, and its documentation advises against offloading to the CPU when performance matters. [1][27] The `ollama ps` command shows the split in its `PROCESSOR` column as `100% GPU`, `100% CPU` or a mix such as `48%/52% CPU/GPU`. [12] llama.cpp controls the same split with `-ngl` (`--gpu-layers`), the number of layers to keep in VRAM, which defaults to `auto`; LM Studio's `lms load` takes `--gpu` with a fraction from 0 to 1, `off` or `max`. [9][28] With several GPUs, Ollama puts the model on a single card if it fits and otherwise spreads it across all of them. [12]

Apple silicon Macs have a unified memory architecture in which the CPU and GPU address the same pool, so there is no separate VRAM figure: the model shares one pool with everything else running on the machine. [29] LM Studio recommends 16 GB or more and says 8 GB Macs should stay with smaller models and modest context sizes. [15] Jan's guidance for macOS is 8 GB of RAM for 3B models, 16 GB for 7B and 32 GB for 13B. [5] MLX LM warns that models which are large relative to total RAM can be slow. [4]

A few other machines share memory in the same way. vLLM's documentation names [NVIDIA DGX Spark](https://aiwiki.ai/wiki/nvidia_dgx_spark) (GB10) and [Jetson](https://aiwiki.ai/wiki/nvidia_jetson) as devices where the CPU and operating system share memory with the GPU, and Ollama lists the GB10 among its supported hardware. [30][31]

A model can also run entirely from system memory, which Ollama reports as `100% CPU`. [12] On Intel Macs, Ollama supports the CPU only. [32]

Hardware support differs by tool. Ollama supports [NVIDIA](https://aiwiki.ai/wiki/nvidia) GPUs with compute capability 5.0 or higher and driver 550 or newer (570 or newer for cards with compute capability 5.0 through 6.2), [AMD](https://aiwiki.ai/wiki/amd) cards through [ROCm](https://aiwiki.ai/wiki/rocm) v7, Apple GPUs through Metal, and further cards on Windows and Linux through Vulkan. [31] vLLM requires Linux and, for NVIDIA cards, compute capability 7.5 or higher. [33] On Windows, LM Studio requires AVX2 on x64 processors and recommends at least 16 GB of RAM and 4 GB of dedicated VRAM. [15]

Several tools will estimate the fit before you download or load anything. `lms load --estimate-only <model_key>` prints a memory estimate that accounts for context length and GPU offload, and Jan's model hub marks whether each model fits the machine. [28][34] AI Wiki's [LLM VRAM calculator](https://aiwiki.ai/tools/llm_vram_calculator) does the same arithmetic in the browser, adding KV cache and runtime overhead to the weights for a chosen model size, quantization, context length and number of concurrent requests.

## Step 2: choose a model and a quantization level

### File formats

Which file to download depends on the runtime. llama.cpp requires [GGUF](https://aiwiki.ai/wiki/gguf), a binary format that holds the tensors together with standardized metadata. [19][35] Ollama, LM Studio and Jan all load GGUF files; LM Studio and Jan also run MLX models on Apple silicon, and Ollama can import a directory of safetensors weights. [36][2][21][11] MLX LM takes a Hugging Face repository name, and thousands of pre-converted models sit in the `mlx-community` organization. [4] vLLM downloads standard Hugging Face repositories by default, or [ModelScope](https://aiwiki.ai/wiki/modelscope) ones when `VLLM_USE_MODELSCOPE` is set; its GGUF support moved to a separate plugin and is described by the project as highly experimental. [6][37]

### Reading a quantization name

GGUF files are published at several precisions, named like `Q4_K_M`. The leading number is the nominal bits per weight. `K` marks the "k-quants" family that contributor ikawrakow added to llama.cpp in June 2023, and the last letter (S, M or L) picks a mix. [38] Under that scheme the `Q4_K` type stores weights in 4 bits inside super-blocks, at 4.5 bits per weight once block scales and minimums are counted; `Q4_K_S` uses it for every tensor, while `Q4_K_M` stores half of the `attention.wv` and `feed_forward.w2` tensors at the 6-bit `Q6_K` type and the rest at `Q4_K`. [38] That mix is why a Q4_K_M file averages more than 4.5 bits per weight. Hugging Face's table of GGUF types describes those beginning with `IQ` as deriving their weights with an importance matrix, and `F16` and `BF16` as 16-bit floating-point formats. [35]

llama.cpp's documentation measures the options on Llama 3.1 8B: [26]

| Type | Bits per weight | File size (GiB) |
|---|---|---|
| IQ2_M | 2.9294 | 2.74 |
| Q3_K_M | 3.9960 | 3.74 |
| IQ4_XS | 4.4597 | 4.17 |
| Q4_K_S | 4.6672 | 4.36 |
| Q4_K_M | 4.8944 | 4.58 |
| Q5_K_M | 5.7036 | 5.33 |
| Q6_K | 6.5633 | 6.14 |
| Q8_0 | 8.5008 | 7.95 |
| F16 | 16.0005 | 14.96 |

Q4_K_M is the default in practice. llama.cpp's `-hf` flag downloads the Q4_K_M file from a repository unless another is named, falling back to the first file in the repository when there is none, and Hugging Face's Ollama integration also picks Q4_K_M when that file exists. [9][36] A scan of internet-reachable Ollama hosts that SentinelLABS and Censys published in January 2026 found Q4_K_M on 48% of them. [72]

### What quantization costs

The llama.cpp documentation says quantization may introduce accuracy loss, usually measured as [perplexity](https://aiwiki.ai/wiki/perplexity) or [KL divergence](https://aiwiki.ai/wiki/kl_divergence) against the original, and that an importance matrix can reduce it. [26] When the k-quants were introduced, their author reported that the 6-bit version stayed within 0.1 percent of the 16-bit model's perplexity on the LLaMA models of the time. [38]

Academic evaluations support 4-bit as a working default, with reservations. Jin and colleagues tested quantized models on ten benchmarks in a paper for the Findings of ACL 2024 and found that 4-bit models kept performance comparable to the unquantized ones, and that a larger quantized model can outperform a smaller model. [39] They also found that quantization can slow inference without engineering and hardware support. [39] A NeurIPS 2024 paper by Dutta and colleagues argues that benchmark accuracy understates the change: answers flip from correct to incorrect and the reverse in proportion, so accuracy looks stable while individual outputs differ, and compressed models did significantly worse than the baseline on the free-form MT-Bench test. [40]

In practice this means starting at Q4_K_M, moving up to Q5_K_M, Q6_K or Q8_0 when memory allows, and going below 4 bits only when nothing else fits. For models that take images or audio, the llama.cpp documentation says the separate projector file (`mmproj`) is usually kept at 16-bit or 8-bit precision, because it is small and its quality directly affects the output. [26]

Outside the GGUF world the names differ. MLX LM's own `mlx_lm.convert` quantizes to 4 bits in the README's example, and vLLM accepts formats such as [AWQ](https://aiwiki.ai/wiki/awq), [GPTQ](https://aiwiki.ai/wiki/gptq), bitsandbytes and the FP8 and INT4 schemes of its LLM Compressor library. [4][41]

## Step 3: pick a runtime

### Ollama

Ollama is the shortest path from nothing to a running model. On macOS and Windows it is a downloadable app; on Linux the documented install is one line: [1][42]

```shell
curl -fsSL https://ollama.com/install.sh | sh
```

The quickstart then runs [Gemma 4](https://aiwiki.ai/wiki/gemma_4) E2B, for which it recommends 8 GB of available VRAM or unified memory on a Mac: [1]

```shell
ollama run gemma4:e2b
```

Ollama downloads the model and opens a chat in the terminal; typing `/bye` leaves it. [1] The rest of the day-to-day commands are listed in the CLI reference: [43]

```shell
ollama pull gemma4
ollama ls
ollama ps
ollama stop gemma4
ollama rm gemma4
```

Any GGUF repository on Hugging Face can be run directly, with an optional quantization tag: [36]

```shell
ollama run hf.co/{username}/{repository}:{quantization}
```

To raise the context length for every model, set it when starting the server: [27]

```shell
OLLAMA_CONTEXT_LENGTH=64000 ollama serve
```

Models are stored under `~/.ollama/models` on macOS, `/usr/share/ollama/.ollama/models` on Linux and `C:\Users\%username%\.ollama\models` on Windows, and a loaded model stays in memory for five minutes after its last use unless `keep_alive` or `OLLAMA_KEEP_ALIVE` says otherwise. [12] Ollama also offers cloud models, which require signing in. [1] Its FAQ states that the company does not see prompts or data when models run locally, and that setting `OLLAMA_NO_CLOUD=1` (or `disable_ollama_cloud` in `~/.ollama/server.json`) turns the cloud features off. [12] A Docker image is published as `ollama/ollama`. [44]

### LM Studio

LM Studio is a desktop app with a chat window, a model search that downloads from Hugging Face, and a local server. It runs GGUF models with llama.cpp on all three platforms and MLX models on Apple silicon. [2] The app is proprietary: its terms grant a non-transferable license for personal and internal business use and forbid modification. [13] It has been free for use at work as well as at home since July 8, 2025, when the company dropped the requirement for a separate commercial license. [45] The bundled `lms` command-line tool is MIT-licensed. [14]

Everything in the app can also be driven from the terminal: [46][28][17]

```shell
lms get llama-3.1-8b@q4_k_m
lms load <model_key> --context-length 4096
lms server start
```

`lms ls` lists downloaded models, `lms ps` lists loaded ones and `lms chat` opens a terminal chat. [14] For a machine without a display, LM Studio ships a standalone daemon called llmster, which the documentation recommends over running the desktop app headless: [47]

```shell
curl -fsSL https://lmstudio.ai/install.sh | bash
lms daemon up
```

LM Studio's documentation states that chatting with models, chatting with documents and running the local server need no internet connection once a model has been downloaded. [48]

### llama.cpp

llama.cpp is the engine under several of the other tools: LM Studio and Jan both run local models with it, and Hugging Face's documentation describes Ollama as an application based on it. [2][21][36] Running it directly gives the most control and the fewest layers. The project released version 1.0.0 on November 13, 2026, committing to semantic versioning for its C API, and said at the time that it supported more than 150 model architecture variants. [49]

Pre-built packages install with one command on each platform: [3][18]

```shell
curl -LsSf https://llama.app/install.sh | sh
brew install llama.cpp
winget install llama.cpp
```

The README's quick start downloads a model from Hugging Face and either chats with it or serves it: [3]

```shell
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
```

The `-hf` argument takes `<user>/<model>[:quant]`, and a local file can be passed with `-m` instead. [9] The server's own documentation uses the `llama-server` binary name: [9]

```shell
./llama-server -m models/7B/ggml-model.gguf -c 2048
```

By default the server listens on `127.0.0.1:9931`, and it includes a built-in chat app called Llama Chat. [9] The port number is recent: the default was 8080 until a change merged on October 9, 2026, so builds from before that change use 8080, and the project's Docker images still listen on 8080 inside the container. [73][9] The flags that matter most are `-c` (`--ctx-size`, where the default of 0 takes the value from the model), `-ngl` (`--gpu-layers`), `--host`, `--port` and `--api-key`. [9] Started without a model, `llama-server` enters a router mode that loads and unloads models dynamically and forwards each request to the right one. [9]

llama.cpp also carries the conversion tools. To turn a Hugging Face repository into a quantized GGUF file: [26]

```shell
python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M
```

### MLX on Apple silicon

MLX is an array framework for machine learning on Apple silicon from Apple's machine learning research group, and MLX LM is the Python package that generates text with language models on it. [50][4] Installation and a first prompt take three commands: [4]

```shell
pip install mlx-lm
mlx_lm.generate --prompt "How tall is Mt Everest?"
mlx_lm.chat
```

With no `--model` argument these use `mlx-community/Llama-3.2-3B-Instruct-4bit`; any MLX-compatible repository can be named instead. [4] To convert and quantize a model yourself: [4]

```shell
mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q
```

When a model is large relative to the Mac's RAM, MLX LM tries to keep it fast by wiring its memory, which needs macOS 15 or later, and the documentation suggests raising the system limit with `sudo sysctl iogpu.wired_limit_mb=N`, where N is larger than the model's size in megabytes and smaller than the machine's memory. [4] The package includes an HTTP server (`mlx_lm.server`), which its documentation says is not recommended for production because it implements only basic security checks. [20]

### Jan

Jan is a desktop app from Menlo Research released under the Apache License 2.0, with a request for attribution in its license notice. [5] It needs no commands: on first launch it downloads a default model, after which the chat window works, and further models come from its Hub, which marks whether each one fits the machine. [34] Local models run on llama.cpp or, on Apple silicon, MLX. [21] Jan's documentation notes that as of version 0.8.5 its bundled llama.cpp engine does not load models on Intel Macs. [21] The app can also connect to cloud providers with an API key, and requests routed that way leave the machine. [34][21]

### Open WebUI as a front end

Open WebUI does not run models. It is a web interface that connects to Ollama's API or to any server that implements the OpenAI-compatible API, so it can sit in front of every runtime in this guide. [7][51] The documented Docker command is: [22]

```shell
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data -e WEBUI_SECRET_KEY=your-secret-key --name open-webui --restart always ghcr.io/open-webui/open-webui:main
```

The documentation says to replace `your-secret-key` with the output of `openssl rand -hex 32`. The interface then appears on port 3000, the named volume holds chats, users and settings, and `--add-host` lets the container reach a model server on the host. [22] Without Docker, it installs from PyPI on Python 3.11 or 3.12 and serves on port 8080: [22]

```shell
pip install open-webui
open-webui serve
```

Open WebUI tries to find a local Ollama instance automatically. From inside Docker, a server on the host is reached as `host.docker.internal` in place of `localhost` (for Ollama, `http://host.docker.internal:11434`), and other servers are added under Settings > Connections in the admin section with their URL and API key. [51][7]

Its license is not one of the standard ones. Since version 0.6.6 in April 2025 the Open WebUI License has combined BSD-3-Clause terms with a clause that forbids removing the Open WebUI branding, except for deployments of 50 or fewer users in a rolling 30-day period or with written permission or an enterprise license. [23][52]

### vLLM for a GPU server

vLLM is a high-throughput serving engine, the route to take when a GPU server has to answer many requests at once. It grew out of the PagedAttention work presented at SOSP 2023, an attention algorithm modelled on the paging techniques of operating systems; its authors reported two to four times the throughput of the serving systems they compared against at the same latency. [53] It needs Linux and Python 3.11 to 3.14; the documentation points Windows users to WSL and Apple silicon users to the separate vLLM-Metal project, which uses MLX as its backend. [6][33]

The documented installation for NVIDIA GPUs uses the `uv` package manager: [6]

```shell
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
```

One command then downloads a model and serves it at `http://localhost:8000`: [6]

```shell
vllm serve Qwen/Qwen2.5-1.5B-Instruct
```

There is an official container image as well: [33]

```shell
docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B
```

The server hosts one model at a time. [6] Unlike the desktop tools, vLLM takes the whole GPU by default: on CUDA and ROCm it sizes its KV cache from whatever memory is free after start-up, and `gpu_memory_utilization` defaults to 1.0. The documentation says to lower it when other processes share the GPU or on unified-memory devices. [30] To fit a model into less memory, the project suggests a quantized checkpoint, a shorter maximum context (`max_model_len`), a smaller batch (`max_num_seqs`), or [tensor parallelism](https://aiwiki.ai/wiki/tensor_parallelism) across several GPUs with `--tensor-parallel-size`. [54][37]

## Step 4: expose an OpenAI-compatible endpoint

Every runtime above can serve HTTP routes shaped like the [OpenAI API](https://aiwiki.ai/wiki/openai_api), chiefly `/v1/chat/completions`, so that existing client libraries and tools work once their base URL points at the local machine. [56][16] This convention is described in [OpenAI-compatible API](https://aiwiki.ai/wiki/openai_compatible_api).

| Runtime | How the server starts | Base URL | Authentication |
|---|---|---|---|
| Ollama | With the app, or `ollama serve` | `http://localhost:11434/v1/` | Clients must send a key value, which Ollama ignores |
| LM Studio | Developer tab, or `lms server start` | `http://localhost:1234/v1` | Off by default; API tokens can be required in Server Settings |
| llama.cpp | `llama serve -hf <user>/<model>` | `http://localhost:9931/v1` | Off by default; `--api-key` |
| MLX LM | `mlx_lm.server --model <path_to_model_or_hf_repo>` | `http://localhost:8080/v1` | None documented |
| Jan | Settings > Local API Server > Start Server | `http://127.0.0.1:1337/v1` | Optional API key |
| vLLM | `vllm serve <model>` | `http://localhost:8000/v1` | Off by default; `--api-key` or `VLLM_API_KEY` |

Sources: [56][16][76][57][17][9][20][21][6][55]. LM Studio's port is whatever was last used unless `--port` is given; its documentation's examples assume 1234. [17][16]

A request against Ollama, from its documentation, looks like this: [56]

```shell
curl -X POST http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "gpt-oss:20b",
  "messages": [{ "role": "user", "content": "Say this is a test" }]
}'
```

The official OpenAI client libraries need only a different base URL: [56]

```python
from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',  # required but ignored
)
```

Compatibility is partial everywhere. Ollama says it supports a subset of the OpenAI API, and llama.cpp's documentation makes no strong claim of conformance beyond saying that the endpoint is enough for many applications. [56][9] One gap matters for agents: the OpenAI API has no field for context size, so with Ollama the larger context has to be set on the server or baked into a model with a `Modelfile` line of the form `PARAMETER num_ctx <context size>`. [56] Ollama and llama.cpp also expose a `/v1/messages` route compatible with the [Anthropic API](https://aiwiki.ai/wiki/anthropic_api). [1][9]

### Opening the server to other machines

Ollama, LM Studio, llama.cpp and Jan document `127.0.0.1` as the default bind address, and MLX LM's server starts on localhost. [12][17][9][21][20] vLLM is the exception: its quickstart shows the server at `http://localhost:8000`, but in the project's argument definitions the `host` setting defaults to no value instead of a loopback address, and the launcher then binds an empty host, which it logs as `0.0.0.0`: every network interface. [6][58][74] Passing `--host 127.0.0.1` keeps a vLLM server on the local machine. For the other tools, making a server reachable from the network is a single setting:

- Ollama: set the `OLLAMA_HOST` environment variable, for example `Environment="OLLAMA_HOST=0.0.0.0:11434"` in the systemd unit on Linux or `launchctl setenv OLLAMA_HOST "0.0.0.0:11434"` on macOS. [12]
- LM Studio: `lms server start --bind 0.0.0.0`. [17]
- llama.cpp: `--host` and `--port`. [9]
- Jan: set Server Host to `0.0.0.0` in the server configuration. [21]
- vLLM: `--host` and `--port`; as noted above, it already listens on every interface unless `--host` is given. [6][74]

That setting is also the main security risk of running models locally. Ollama's local server ignores the API key value that clients send, and its FAQ describes exposing the server through a proxy server such as Nginx. [56][12] LM Studio's documentation warns that any bind address other than `127.0.0.1` exposes the server beyond the local machine and recommends enabling authentication. [17] Jan advises against leaving the API key empty when binding to `0.0.0.0`. [21] llama.cpp recommends an API key and a reverse proxy for public deployments, and its security policy lists `llama-server` among the functions not to use in untrusted environments or networks. [9][59] vLLM's `--api-key` protects only the `/v1`, `/v2`, `/inference` and `/cohere` path prefixes, and its documentation says not to rely on it alone. [60]

The warnings reflect what has happened. On January 29, 2026, SentinelOne's SentinelLABS and Censys published the results of 293 days of scanning for Ollama servers reachable from the public internet. They counted 175,108 unique hosts in 130 countries over that period, many of them transient, with a persistent core of about 23,000 hosts that generated most of the activity. The report notes that Ollama binds to `127.0.0.1:11434` by design and that exposing it takes a single configuration change, and it found that over 48% of the hosts advertised [tool calling](https://aiwiki.ai/wiki/function_calling). [72] TechRadar's account of the study said that, according to Pillar Security, attackers were already using such servers to generate spam and malware content at the owners' expense, a practice called LLMjacking. [61]

## Verify what you downloaded

A model file is written by someone else and then loaded by software that has access to your machine. Four checks reduce the risk.

### Use a source you trust

Take the weights from the developer's own repository, or from a publisher you have reason to trust. Hugging Face's security documentation suggests loading models only from users and organizations you trust and relying on signed commits. [62]

### Prefer formats that are not executable

Pickle, the default format for [PyTorch](https://aiwiki.ai/wiki/pytorch) weights (files such as `pytorch_model.bin`), can execute arbitrary code when it is loaded. [62] Safetensors is a format for storing tensors safely, as opposed to pickle, and its documentation describes a safe format as one where a file downloaded at random can be loaded without running arbitrary code. [63] GGUF is a binary format that holds tensors and standardized metadata. [35] Hugging Face scans what is uploaded: files under 2 GB go through the ClamAV malware scanner at each commit, pickle files get an import scan whose results are shown next to the file, and third-party scanners such as JFrog's add their own reports. [64][62][65] Hugging Face says its pickle scan is not foolproof and that checking safety remains the user's responsibility. [62]

### Treat unknown models as untrusted anyway

A file that is not a pickle is not thereby safe. Hugging Face's documentation notes that pickle is not the only exploitable model format. [65] llama.cpp's security policy tells users to be careful with models from unknown developers, to run untrusted models in a sandbox such as a container or virtual machine, to keep the software updated, and to confirm that the hash of a downloaded file matches a known-good value. [59] Some repositories also ship custom code: MLX LM asks for confirmation before running a model whose tokenizer needs `trust_remote_code`, unless `--trust-remote-code` was passed. [4]

### Check the checksums

The Hugging Face command-line tool downloads a repository and can verify the local copy against the checksums recorded on the Hub. The documentation's examples are: [66]

```shell
hf download HuggingFaceH4/zephyr-7b-beta
hf cache verify deepseek-ai/DeepSeek-OCR
hf cache verify deepseek-ai/DeepSeek-OCR --local-dir /path/to/repo
```

On a mismatch, `hf cache verify` prints the affected files and exits with a non-zero status. [66] The tool installs with `curl -LsSf https://hf.co/cli/install.sh | bash`, with `pip install -U "huggingface_hub"`, or with `brew install hf`. [66]

## Read the license before commercial use

The license of the runtime says nothing about the model. Ollama's MIT license or vLLM's Apache license covers that software; the weights you load come with separate terms set by whoever trained them. LM Studio's terms make the point explicitly: third-party models are provided under an agreement between the user and the model's provider, and complying with it is the user's job. [13]

Where to find the terms:

- On Hugging Face, the [model card](https://aiwiki.ai/wiki/model_card) metadata carries a `license` field that is shown on the model page and can be used as a search filter. A custom license appears as `license: other` with a `license_name` and a `license_link`, or as a `LICENSE` file in the repository. [67]
- Some repositories are gated: you must submit an access request before the files download, which the authors approve automatically or by hand, and the request shares your Hugging Face username and email address with them unless they have opted out. [68]
- For a model pulled through Ollama, the `/api/show` endpoint returns the model's license in its `license` field. [69]

Custom licenses need reading in full. The [Llama 3.1 Community License](https://aiwiki.ai/wiki/llama_community_license) is one example: it requires anyone who distributes the model or a derivative to include a copy of the agreement and display "Built with Llama", it incorporates an acceptable use policy, and it withdraws the grant from any licensee whose products had more than 700 million monthly active users in the month before the release date, who must ask Meta for a separate license. [70] The conditions on distribution cover derivative works of the model as well as the original files, so it is worth checking that a community GGUF or MLX conversion of such a model carries the same agreement. [70]

A permissive license on the weights does not by itself make a model open source in the sense defined by the [Open Source Initiative](https://aiwiki.ai/wiki/open_source_initiative). The [Open Source AI Definition](https://aiwiki.ai/wiki/open_source_ai_definition) 1.0 also asks for the training and inference code and for detailed information about the training data, all under OSI-approved terms, which is why this guide says "open-weight". [71] [Open-Weight LLM License Comparison](https://aiwiki.ai/wiki/open_weight_license_comparison) goes through the main model licenses clause by clause, and the [list of open-weight models](https://aiwiki.ai/wiki/list_of_open_weight_models) records the license of each model AI Wiki covers.

## Limitations and trade-offs

- Memory sets a ceiling. A model that does not fit in VRAM still runs from system RAM, but more slowly, and long contexts push memory up further. [1][27]
- Default contexts are small. Ollama starts at 4k tokens on GPUs with less than 24 GiB, well below the 64,000 it recommends for agents and coding tools. [27]
- Quantized models are not identical to the originals. Benchmark scores may hold at 4 bits while individual answers change. [39][40]
- The effect of quantization on speed depends on the software. Jin and colleagues found that quantization can slow inference without engineering support, while llama.cpp's own benchmark of Llama 3.1 8B (on hardware the documentation does not name) generated about 72 [tokens per second](https://aiwiki.ai/wiki/tokens_per_second) at Q4_K_M against about 29 for the 16-bit file. [39][26]
- API compatibility is partial. The local servers implement subsets of the OpenAI API, so a tool written for the hosted service may use a route or field that a given runtime lacks. [56][9]
- The user is the operator. Updates, network exposure and the trustworthiness of downloaded files fall to whoever runs the machine, as the security guidance of llama.cpp, vLLM and Hugging Face says in different words. [59][60][62]
- Not every route suits every job. MLX LM's server is not recommended for production, vLLM requires Linux and serves one model per server, and vLLM's GGUF support is experimental. [20][6][37]

## See also

- [Best Local and On-Device LLMs](https://aiwiki.ai/wiki/best_local_llms)
- [Open-Weight LLM License Comparison](https://aiwiki.ai/wiki/open_weight_license_comparison)
- [List of open-weight models](https://aiwiki.ai/wiki/list_of_open_weight_models)
- [GGUF](https://aiwiki.ai/wiki/gguf)
- [Quantization](https://aiwiki.ai/wiki/quantization)
- [KV cache](https://aiwiki.ai/wiki/kv_cache)
- [Small language model](https://aiwiki.ai/wiki/small_language_model)

## References

1. Ollama. "Quickstart". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/quickstart
2. LM Studio. "Welcome to LM Studio Docs!". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/app
3. ggml-org. "llama.cpp" (README and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. https://github.com/ggml-org/llama.cpp
4. ml-explore. "MLX LM" (README and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. https://github.com/ml-explore/mlx-lm
5. Menlo Research. "Jan - Open-source ChatGPT replacement" (README and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. https://github.com/janhq/jan
6. vLLM project. "Quickstart" (docs/getting_started/quickstart.md). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/docs/getting_started/quickstart.md
7. Open WebUI. "OpenAI-Compatible" (documentation source). open-webui/docs repository, GitHub, accessed November 19, 2026. https://github.com/open-webui/docs/blob/main/docs/getting-started/quick-start/connect-a-provider/starting-with-openai-compatible/index.mdx
8. Hugging Face. "Use AI Models Locally". Hugging Face Hub documentation, accessed November 19, 2026. https://huggingface.co/docs/hub/local-apps
9. ggml-org. "LLaMA.cpp HTTP Server" (tools/server/README.md). llama.cpp repository, GitHub, accessed November 19, 2026. https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
10. Ollama. "ollama/ollama" (repository and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. https://github.com/ollama/ollama
11. Ollama. "Importing a Model". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/import
12. Ollama. "Local FAQ". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/faq
13. LM Studio. "LM Studio App Terms of Service". LM Studio, version of August 23, 2026. https://lmstudio.ai/app-terms
14. LM Studio. "lms - LM Studio's CLI". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/cli
15. LM Studio. "System Requirements". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/app/system-requirements
16. LM Studio. "OpenAI Compatibility Endpoints". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/developer/openai-compat
17. LM Studio. "lms server start". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/cli/serve/server-start
18. ggml-org. "Install pre-built version of llama.cpp" (docs/install.md). llama.cpp repository, GitHub, accessed November 19, 2026. https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md
19. ggml-org. "Obtaining and quantizing models" (docs/models.md). llama.cpp repository, GitHub, accessed November 19, 2026. https://github.com/ggml-org/llama.cpp/blob/master/docs/models.md
20. ml-explore. "HTTP Model Server" (mlx_lm/SERVER.md). mlx-lm repository, GitHub, accessed November 19, 2026. https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md
21. Jan. "Local API Server". Jan documentation, accessed November 19, 2026. https://docs.jan.ai/desktop/api-server
22. Open WebUI. "Quick Start" (documentation source, including the Docker and pip tabs). open-webui/docs repository, GitHub, accessed November 19, 2026. https://github.com/open-webui/docs/tree/main/docs/getting-started/quick-start
23. Open WebUI Inc. "Open WebUI License" (LICENSE file). open-webui/open-webui repository, GitHub, accessed November 19, 2026. https://github.com/open-webui/open-webui/blob/main/LICENSE
24. Open WebUI. "open-webui/open-webui" (README; star count from the GitHub API). GitHub, accessed November 19, 2026. https://github.com/open-webui/open-webui
25. vLLM project. "vllm-project/vllm" (repository and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm
26. ggml-org. "quantize" (tools/quantize/README.md). llama.cpp repository, GitHub, accessed November 19, 2026. https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
27. Ollama. "Context length". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/context-length
28. LM Studio. "lms load". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/cli/local-models/load
29. ml-explore. "Unified Memory" (docs/src/usage/unified_memory.rst). MLX repository, GitHub, accessed November 19, 2026. https://github.com/ml-explore/mlx/blob/main/docs/src/usage/unified_memory.rst
30. vLLM project. "Optimization and Tuning" (docs/configuration/optimization.md). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/docs/configuration/optimization.md
31. Ollama. "Hardware support". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/gpu
32. Ollama. "macOS". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/macos
33. vLLM project. "GPU" installation guide (docs/getting_started/installation/gpu.md and gpu.cuda.inc.md). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/docs/getting_started/installation/gpu.md
34. Jan. "QuickStart". Jan documentation, accessed November 19, 2026. https://docs.jan.ai/desktop/quickstart
35. Hugging Face. "GGUF". Hugging Face Hub documentation, accessed November 19, 2026. https://huggingface.co/docs/hub/gguf
36. Hugging Face. "Use Ollama with any GGUF Model on Hugging Face Hub". Hugging Face Hub documentation, accessed November 19, 2026. https://huggingface.co/docs/hub/ollama
37. vLLM project. "GGUF" (docs/features/quantization/gguf.md). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/docs/features/quantization/gguf.md
38. ikawrakow. "k-quants". Pull request #1684, llama.cpp repository, GitHub, opened June 3, 2023, merged June 5, 2023. https://github.com/ggml-org/llama.cpp/pull/1684
39. Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, Deyi Xiong. "A Comprehensive Evaluation of Quantization Strategies for Large Language Models". Findings of ACL 2024; arXiv:2402.16775, February 26, 2024. https://arxiv.org/abs/2402.16775
40. Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, Ramachandran Ramjee. "Accuracy is Not All You Need". NeurIPS 2024; arXiv:2407.09141, July 12, 2024. https://arxiv.org/abs/2407.09141
41. vLLM project. "Quantization" (docs/features/quantization/README.md). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/docs/features/quantization/README.md
42. Ollama. "Linux". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/linux
43. Ollama. "CLI Reference". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/cli
44. Ollama. "Docker". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/docker
45. Yagil Burowski. "LM Studio is free for use at work". LM Studio Blog, July 8, 2025. https://lmstudio.ai/blog/free-for-work
46. LM Studio. "lms get". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/cli/local-models/get
47. LM Studio. "Run LM Studio as a service (headless)". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/developer/core/headless
48. LM Studio. "Offline Operation". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/app/offline
49. Daniel Bevenius, Georgi Gerganov, merve (ggml-org). "llama.cpp v1.0.0 : A stronger foundation for the local AI ecosystem". Hugging Face blog (team article), November 13, 2026. https://huggingface.co/blog/ggml-org/llama-cpp-v1
50. ml-explore. "MLX" (README). GitHub, accessed November 19, 2026. https://github.com/ml-explore/mlx
51. Open WebUI. "Ollama" (documentation source). open-webui/docs repository, GitHub, accessed November 19, 2026. https://github.com/open-webui/docs/blob/main/docs/getting-started/quick-start/connect-a-provider/starting-with-ollama.mdx
52. Open WebUI. "Open WebUI License" (documentation source). open-webui/docs repository, GitHub, accessed November 19, 2026. https://github.com/open-webui/docs/blob/main/docs/license.mdx
53. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica. "Efficient Memory Management for Large Language Model Serving with PagedAttention". SOSP 2023; arXiv:2309.06180, September 12, 2023. https://arxiv.org/abs/2309.06180
54. vLLM project. "Conserving Memory" (docs/configuration/conserving_memory.md). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md
55. vLLM project. "vLLM CLI Guide" (docs/cli/README.md). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/docs/cli/README.md
56. Ollama. "OpenAI compatibility". Ollama documentation, accessed November 19, 2026. https://docs.ollama.com/api/openai-compatibility
57. LM Studio. "Authentication". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/developer/core/authentication
58. vLLM project. "cli_args.py" (vllm/entrypoints/common/cli_args.py, server argument definitions). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/common/cli_args.py
59. ggml-org. "Security Policy" (SECURITY.md). llama.cpp repository, GitHub, accessed November 19, 2026. https://github.com/ggml-org/llama.cpp/blob/master/SECURITY.md
60. vLLM project. "Security" (docs/usage/security.md). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/docs/usage/security.md
61. Sead Fadilpašić. "Over 175,000 publicly exposed Ollama AI servers discovered worldwide - so fix now". TechRadar, January 30, 2026. https://www.techradar.com/pro/security/over-175-000-publicly-exposed-ollama-ai-servers-discovered-worldwide-so-fix-now
62. Hugging Face. "Pickle Scanning". Hugging Face Hub documentation, accessed November 19, 2026. https://huggingface.co/docs/hub/security-pickle
63. Hugging Face. "safetensors" (README). GitHub, accessed November 19, 2026. https://github.com/safetensors/safetensors
64. Hugging Face. "Malware Scanning". Hugging Face Hub documentation, accessed November 19, 2026. https://huggingface.co/docs/hub/security-malware
65. Hugging Face. "Third-party scanner: JFrog". Hugging Face Hub documentation, accessed November 19, 2026. https://huggingface.co/docs/hub/security-jfrog
66. Hugging Face. "Command Line Interface (CLI)". huggingface_hub documentation, accessed November 19, 2026. https://huggingface.co/docs/huggingface_hub/guides/cli
67. Hugging Face. "Model Cards". Hugging Face Hub documentation, accessed November 19, 2026. https://huggingface.co/docs/hub/model-cards
68. Hugging Face. "Gated models". Hugging Face Hub documentation, accessed November 19, 2026. https://huggingface.co/docs/hub/models-gated
69. Ollama. "Show model details". Ollama API reference, accessed November 19, 2026. https://docs.ollama.com/api-reference/show-model-details
70. Meta. "Llama 3.1 Community License Agreement" (models/llama3_1/LICENSE). meta-llama/llama-models repository, GitHub; Llama 3.1 version release date July 23, 2024. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE
71. Open Source Initiative. "The Open Source AI Definition - 1.0". Accessed November 19, 2026. https://opensource.org/ai/open-source-ai-definition
72. Gabriel Bernadett-Shapiro, Silas Cutler. "Silent Brothers | Ollama Hosts Form Anonymous AI Network Beyond Platform Guardrails". SentinelLABS (SentinelOne), January 29, 2026, updated March 6, 2026. https://www.sentinelone.com/labs/silent-brothers-ollama-hosts-form-anonymous-ai-network-beyond-platform-guardrails/
73. Xuan-Son Nguyen (ngxson). "server: use port 9931 by default". Pull request #30159, llama.cpp repository, GitHub, merged October 9, 2026. https://github.com/ggml-org/llama.cpp/pull/30159
74. vLLM project. "launcher.py" (vllm/entrypoints/launchers/launcher.py, server socket set-up). vLLM repository, GitHub, accessed November 19, 2026. https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/launchers/launcher.py
75. Ollama. "config.go" (envconfig/config.go, server environment settings). ollama/ollama repository, GitHub, accessed November 19, 2026. https://github.com/ollama/ollama/blob/main/envconfig/config.go
76. LM Studio. "LM Studio as a Local LLM API Server". LM Studio documentation, accessed November 19, 2026. https://lmstudio.ai/docs/developer/core/server

