Citation and evidence

How to Run LLMs Locally

33 min full readUpdated 76 references

This article's verification

Report a problem with this article

Edit article
More

Cite this article

This guide explains how to run LLMs locally: downloading the weights of a large language model (LLM) and performing inference on hardware you control, instead of sending prompts to a hosted API. As of November 2026 the usual routes are Ollama, LM Studio, llama.cpp, MLX on Apple silicon, Jan and, for a GPU server, vLLM, with Open WebUI as a browser front end for any of them. [1][2][3][4][5][6][7] The work comes down to four decisions: whether the machine has enough memory, which model and quantization level to download, which runtime to use, and whether to expose the model to other programs through an API. Two checks come before any serious use: that the downloaded files are what they claim to be, and that the model's license allows what you plan to do with it.

Hugging Face's documentation lists four reasons people run models on their own machines: prompts are not sent to a remote server, speed depends on local hardware instead of a provider's servers or connection, the model can be configured freely, and there is no API bill. [8] The costs are covered in the limitations section at the end.

The commands below are copied from each project's official documentation as it stood on November 19, 2026. Command names, default ports and flags change between releases. The llama.cpp README, for example, now starts the server with llama serve, while the server's own documentation still describes the llama-server binary. [3][9] Check the linked documentation before scripting against any of them. For model recommendations by memory tier, see Best Local and On-Device LLMs; for the broader concept see local AI, and for production deployment see self-hosting LLMs.

The routes at a glance

ToolWhat it isSoftware licenseRuns onModels it loadsDefault local addressGitHub stars (2026-11-19)
OllamaCommand-line tool and desktop app with its own model libraryMITmacOS, Windows, LinuxGGUF; imports safetensors127.0.0.1:11434187,575
LM StudioDesktop app, plus the lms command-line tool and a headless daemonProprietary terms (the lms tool is MIT)Apple silicon Macs; x64 and ARM Windows; x64 and ARM64 LinuxGGUF through llama.cpp; MLX models on Apple silicon127.0.0.1, port 1234 in the documentation's examples5,440 (lms only)
llama.cppC/C++ inference engine with a command-line chat and an HTTP serverMITWindows, macOS, LinuxGGUF127.0.0.1:9931135,648
MLX LMPython package for text generation and fine-tuningMITApple silicon MacsHugging Face repositories, including pre-converted mlx-community modelslocalhost:80807,694
JanDesktop chat app with a local API serverApache 2.0macOS, Windows, Linuxllama.cpp models; MLX models on Apple silicon127.0.0.1:133745,808
Open WebUIWeb interface that connects to Ollama or to any OpenAI-compatible serverOpen WebUI License (BSD-3-Clause terms plus a branding clause)macOS, Linux, Windows, through Docker or PythonNone of its own; it talks to a backendUI on port 3000 (Docker) or 8080 (pip)159,039
vLLMServing engine for GPUsApache 2.0LinuxHugging Face model repositorieslocalhost:8000 in the documentation; every interface unless --host is set97,177

Expanded article table

Sources for the table: each project's documentation, license file and GitHub repository. [1][10][11][12][2][13][14][15][16][17][3][18][19][9][4][20][5][21][22][23][24][6][25][74] The licenses in this table cover the software only. Each model's weights carry their own license, discussed below.

Step 1: check that the machine has enough memory

Memory decides what can run. The llama.cpp documentation states that models are fully loaded into memory, so a model needs about as much RAM as its file takes on disk. [26] The size of the weights follows from the parameter count and the precision each weight is stored at:

weights (bytes)≈N×b8\text{weights (bytes)} \approx \frac{N \times b}{8}

where $N$ is the number of parameters and $b$ is the average number of bits stored per weight. The llama.cpp documentation gives worked figures for Llama 3.1: the 8B model takes 32.1 GB in its original form and 4.9 GB at the Q4_K_M quantization, the 70B model 280.9 GB and 43.1 GB, and the 405B model 1,625.1 GB and 249.1 GB. [26] The same page measures Q4_K_M at an average of 4.8944 bits per weight on the 8B model. [26]

Context needs memory too

The weights are not the whole requirement. The runtime also keeps a KV cache that grows with the context window. Ollama's context length page says that a larger context length increases the memory needed to run a model, and that the default depends on the VRAM Ollama finds: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. [27] The project's FAQ gives a different answer, a flat default of 4096 tokens; the help text for OLLAMA_CONTEXT_LENGTH in Ollama's source code agrees with the context length page ("default: 4k/32k/256k based on VRAM"). [12][75] The context length page recommends at least 64,000 tokens for web search, agents and coding tools, which is far above the 4k default that applies below 24 GiB. [27] Serving requests in parallel multiplies the allocation: in Ollama's example, a 2K context with 4 parallel requests becomes an 8K context. [12] KV-cache quantization, which Ollama offers when Flash Attention is enabled, reduces the cost. Its q8_0 setting uses about half the memory of the default f16 cache and q4_0 about a quarter, with a precision loss that the documentation calls very small for the first and small to medium for the second. [12]

Where the memory is

With a discrete GPU, the aim is to fit the whole model in VRAM. Ollama can spill into system RAM when VRAM runs short, but its quickstart warns that responses may then be slower, and its documentation advises against offloading to the CPU when performance matters. [1][27] The ollama ps command shows the split in its PROCESSOR column as 100% GPU, 100% CPU or a mix such as 48%/52% CPU/GPU. [12] llama.cpp controls the same split with -ngl (--gpu-layers), the number of layers to keep in VRAM, which defaults to auto; LM Studio's lms load takes --gpu with a fraction from 0 to 1, off or max. [9][28] With several GPUs, Ollama puts the model on a single card if it fits and otherwise spreads it across all of them. [12]

Apple silicon Macs have a unified memory architecture in which the CPU and GPU address the same pool, so there is no separate VRAM figure: the model shares one pool with everything else running on the machine. [29] LM Studio recommends 16 GB or more and says 8 GB Macs should stay with smaller models and modest context sizes. [15] Jan's guidance for macOS is 8 GB of RAM for 3B models, 16 GB for 7B and 32 GB for 13B. [5] MLX LM warns that models which are large relative to total RAM can be slow. [4]

A few other machines share memory in the same way. vLLM's documentation names NVIDIA DGX Spark (GB10) and Jetson as devices where the CPU and operating system share memory with the GPU, and Ollama lists the GB10 among its supported hardware. [30][31]

A model can also run entirely from system memory, which Ollama reports as 100% CPU. [12] On Intel Macs, Ollama supports the CPU only. [32]

Hardware support differs by tool. Ollama supports NVIDIA GPUs with compute capability 5.0 or higher and driver 550 or newer (570 or newer for cards with compute capability 5.0 through 6.2), AMD cards through ROCm v7, Apple GPUs through Metal, and further cards on Windows and Linux through Vulkan. [31] vLLM requires Linux and, for NVIDIA cards, compute capability 7.5 or higher. [33] On Windows, LM Studio requires AVX2 on x64 processors and recommends at least 16 GB of RAM and 4 GB of dedicated VRAM. [15]

Several tools will estimate the fit before you download or load anything. lms load --estimate-only <model_key> prints a memory estimate that accounts for context length and GPU offload, and Jan's model hub marks whether each model fits the machine. [28][34] AI Wiki's LLM VRAM calculator does the same arithmetic in the browser, adding KV cache and runtime overhead to the weights for a chosen model size, quantization, context length and number of concurrent requests.

Step 2: choose a model and a quantization level

File formats

Which file to download depends on the runtime. llama.cpp requires GGUF, a binary format that holds the tensors together with standardized metadata. [19][35] Ollama, LM Studio and Jan all load GGUF files; LM Studio and Jan also run MLX models on Apple silicon, and Ollama can import a directory of safetensors weights. [36][2][21][11] MLX LM takes a Hugging Face repository name, and thousands of pre-converted models sit in the mlx-community organization. [4] vLLM downloads standard Hugging Face repositories by default, or ModelScope ones when VLLM_USE_MODELSCOPE is set; its GGUF support moved to a separate plugin and is described by the project as highly experimental. [6][37]

Reading a quantization name

GGUF files are published at several precisions, named like Q4_K_M. The leading number is the nominal bits per weight. K marks the "k-quants" family that contributor ikawrakow added to llama.cpp in June 2023, and the last letter (S, M or L) picks a mix. [38] Under that scheme the Q4_K type stores weights in 4 bits inside super-blocks, at 4.5 bits per weight once block scales and minimums are counted; Q4_K_S uses it for every tensor, while Q4_K_M stores half of the attention.wv and feed_forward.w2 tensors at the 6-bit Q6_K type and the rest at Q4_K. [38] That mix is why a Q4_K_M file averages more than 4.5 bits per weight. Hugging Face's table of GGUF types describes those beginning with IQ as deriving their weights with an importance matrix, and F16 and BF16 as 16-bit floating-point formats. [35]

llama.cpp's documentation measures the options on Llama 3.1 8B: [26]

TypeBits per weightFile size (GiB)
IQ2_M2.92942.74
Q3_K_M3.99603.74
IQ4_XS4.45974.17
Q4_K_S4.66724.36
Q4_K_M4.89444.58
Q5_K_M5.70365.33
Q6_K6.56336.14
Q8_08.50087.95
F1616.000514.96

Expanded article table

Q4_K_M is the default in practice. llama.cpp's -hf flag downloads the Q4_K_M file from a repository unless another is named, falling back to the first file in the repository when there is none, and Hugging Face's Ollama integration also picks Q4_K_M when that file exists. [9][36] A scan of internet-reachable Ollama hosts that SentinelLABS and Censys published in January 2026 found Q4_K_M on 48% of them. [72]

What quantization costs

The llama.cpp documentation says quantization may introduce accuracy loss, usually measured as perplexity or KL divergence against the original, and that an importance matrix can reduce it. [26] When the k-quants were introduced, their author reported that the 6-bit version stayed within 0.1 percent of the 16-bit model's perplexity on the LLaMA models of the time. [38]

Academic evaluations support 4-bit as a working default, with reservations. Jin and colleagues tested quantized models on ten benchmarks in a paper for the Findings of ACL 2024 and found that 4-bit models kept performance comparable to the unquantized ones, and that a larger quantized model can outperform a smaller model. [39] They also found that quantization can slow inference without engineering and hardware support. [39] A NeurIPS 2024 paper by Dutta and colleagues argues that benchmark accuracy understates the change: answers flip from correct to incorrect and the reverse in proportion, so accuracy looks stable while individual outputs differ, and compressed models did significantly worse than the baseline on the free-form MT-Bench test. [40]

In practice this means starting at Q4_K_M, moving up to Q5_K_M, Q6_K or Q8_0 when memory allows, and going below 4 bits only when nothing else fits. For models that take images or audio, the llama.cpp documentation says the separate projector file (mmproj) is usually kept at 16-bit or 8-bit precision, because it is small and its quality directly affects the output. [26]

Outside the GGUF world the names differ. MLX LM's own mlx_lm.convert quantizes to 4 bits in the README's example, and vLLM accepts formats such as AWQ, GPTQ, bitsandbytes and the FP8 and INT4 schemes of its LLM Compressor library. [4][41]

Step 3: pick a runtime

Ollama

Ollama is the shortest path from nothing to a running model. On macOS and Windows it is a downloadable app; on Linux the documented install is one line: [1][42]

curl -fsSL https://ollama.com/install.sh | sh

The quickstart then runs Gemma 4 E2B, for which it recommends 8 GB of available VRAM or unified memory on a Mac: [1]

ollama run gemma4:e2b

Ollama downloads the model and opens a chat in the terminal; typing /bye leaves it. [1] The rest of the day-to-day commands are listed in the CLI reference: [43]

ollama pull gemma4
ollama ls
ollama ps
ollama stop gemma4
ollama rm gemma4

Any GGUF repository on Hugging Face can be run directly, with an optional quantization tag: [36]

ollama run hf.co/{username}/{repository}:{quantization}

To raise the context length for every model, set it when starting the server: [27]

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Models are stored under ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows, and a loaded model stays in memory for five minutes after its last use unless keep_alive or OLLAMA_KEEP_ALIVE says otherwise. [12] Ollama also offers cloud models, which require signing in. [1] Its FAQ states that the company does not see prompts or data when models run locally, and that setting OLLAMA_NO_CLOUD=1 (or disable_ollama_cloud in ~/.ollama/server.json) turns the cloud features off. [12] A Docker image is published as ollama/ollama. [44]

LM Studio

LM Studio is a desktop app with a chat window, a model search that downloads from Hugging Face, and a local server. It runs GGUF models with llama.cpp on all three platforms and MLX models on Apple silicon. [2] The app is proprietary: its terms grant a non-transferable license for personal and internal business use and forbid modification. [13] It has been free for use at work as well as at home since July 8, 2025, when the company dropped the requirement for a separate commercial license. [45] The bundled lms command-line tool is MIT-licensed. [14]

Everything in the app can also be driven from the terminal: [46][28][17]

lms get llama-3.1-8b@q4_k_m
lms load <model_key> --context-length 4096
lms server start

lms ls lists downloaded models, lms ps lists loaded ones and lms chat opens a terminal chat. [14] For a machine without a display, LM Studio ships a standalone daemon called llmster, which the documentation recommends over running the desktop app headless: [47]

curl -fsSL https://lmstudio.ai/install.sh | bash
lms daemon up

LM Studio's documentation states that chatting with models, chatting with documents and running the local server need no internet connection once a model has been downloaded. [48]

llama.cpp

llama.cpp is the engine under several of the other tools: LM Studio and Jan both run local models with it, and Hugging Face's documentation describes Ollama as an application based on it. [2][21][36] Running it directly gives the most control and the fewest layers. The project released version 1.0.0 on November 13, 2026, committing to semantic versioning for its C API, and said at the time that it supported more than 150 model architecture variants. [49]

Pre-built packages install with one command on each platform: [3][18]

curl -LsSf https://llama.app/install.sh | sh
brew install llama.cpp
winget install llama.cpp

The README's quick start downloads a model from Hugging Face and either chats with it or serves it: [3]

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The -hf argument takes <user>/<model>[:quant], and a local file can be passed with -m instead. [9] The server's own documentation uses the llama-server binary name: [9]

./llama-server -m models/7B/ggml-model.gguf -c 2048

By default the server listens on 127.0.0.1:9931, and it includes a built-in chat app called Llama Chat. [9] The port number is recent: the default was 8080 until a change merged on October 9, 2026, so builds from before that change use 8080, and the project's Docker images still listen on 8080 inside the container. [73][9] The flags that matter most are -c (--ctx-size, where the default of 0 takes the value from the model), -ngl (--gpu-layers), --host, --port and --api-key. [9] Started without a model, llama-server enters a router mode that loads and unloads models dynamically and forwards each request to the right one. [9]

llama.cpp also carries the conversion tools. To turn a Hugging Face repository into a quantized GGUF file: [26]

python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M

MLX on Apple silicon

MLX is an array framework for machine learning on Apple silicon from Apple's machine learning research group, and MLX LM is the Python package that generates text with language models on it. [50][4] Installation and a first prompt take three commands: [4]

pip install mlx-lm
mlx_lm.generate --prompt "How tall is Mt Everest?"
mlx_lm.chat

With no --model argument these use mlx-community/Llama-3.2-3B-Instruct-4bit; any MLX-compatible repository can be named instead. [4] To convert and quantize a model yourself: [4]

mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q

When a model is large relative to the Mac's RAM, MLX LM tries to keep it fast by wiring its memory, which needs macOS 15 or later, and the documentation suggests raising the system limit with sudo sysctl iogpu.wired_limit_mb=N, where N is larger than the model's size in megabytes and smaller than the machine's memory. [4] The package includes an HTTP server (mlx_lm.server), which its documentation says is not recommended for production because it implements only basic security checks. [20]

Jan

Jan is a desktop app from Menlo Research released under the Apache License 2.0, with a request for attribution in its license notice. [5] It needs no commands: on first launch it downloads a default model, after which the chat window works, and further models come from its Hub, which marks whether each one fits the machine. [34] Local models run on llama.cpp or, on Apple silicon, MLX. [21] Jan's documentation notes that as of version 0.8.5 its bundled llama.cpp engine does not load models on Intel Macs. [21] The app can also connect to cloud providers with an API key, and requests routed that way leave the machine. [34][21]

Open WebUI as a front end

Open WebUI does not run models. It is a web interface that connects to Ollama's API or to any server that implements the OpenAI-compatible API, so it can sit in front of every runtime in this guide. [7][51] The documented Docker command is: [22]

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data -e WEBUI_SECRET_KEY=your-secret-key --name open-webui --restart always ghcr.io/open-webui/open-webui:main

The documentation says to replace your-secret-key with the output of openssl rand -hex 32. The interface then appears on port 3000, the named volume holds chats, users and settings, and --add-host lets the container reach a model server on the host. [22] Without Docker, it installs from PyPI on Python 3.11 or 3.12 and serves on port 8080: [22]

pip install open-webui
open-webui serve

Open WebUI tries to find a local Ollama instance automatically. From inside Docker, a server on the host is reached as host.docker.internal in place of localhost (for Ollama, http://host.docker.internal:11434), and other servers are added under Settings > Connections in the admin section with their URL and API key. [51][7]

Its license is not one of the standard ones. Since version 0.6.6 in April 2025 the Open WebUI License has combined BSD-3-Clause terms with a clause that forbids removing the Open WebUI branding, except for deployments of 50 or fewer users in a rolling 30-day period or with written permission or an enterprise license. [23][52]

vLLM for a GPU server

vLLM is a high-throughput serving engine, the route to take when a GPU server has to answer many requests at once. It grew out of the PagedAttention work presented at SOSP 2023, an attention algorithm modelled on the paging techniques of operating systems; its authors reported two to four times the throughput of the serving systems they compared against at the same latency. [53] It needs Linux and Python 3.11 to 3.14; the documentation points Windows users to WSL and Apple silicon users to the separate vLLM-Metal project, which uses MLX as its backend. [6][33]

The documented installation for NVIDIA GPUs uses the uv package manager: [6]

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

One command then downloads a model and serves it at http://localhost:8000: [6]

vllm serve Qwen/Qwen2.5-1.5B-Instruct

There is an official container image as well: [33]

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B

The server hosts one model at a time. [6] Unlike the desktop tools, vLLM takes the whole GPU by default: on CUDA and ROCm it sizes its KV cache from whatever memory is free after start-up, and gpu_memory_utilization defaults to 1.0. The documentation says to lower it when other processes share the GPU or on unified-memory devices. [30] To fit a model into less memory, the project suggests a quantized checkpoint, a shorter maximum context (max_model_len), a smaller batch (max_num_seqs), or tensor parallelism across several GPUs with --tensor-parallel-size. [54][37]

Step 4: expose an OpenAI-compatible endpoint

Every runtime above can serve HTTP routes shaped like the OpenAI API, chiefly /v1/chat/completions, so that existing client libraries and tools work once their base URL points at the local machine. [56][16] This convention is described in OpenAI-compatible API.

RuntimeHow the server startsBase URLAuthentication
OllamaWith the app, or ollama servehttp://localhost:11434/v1/Clients must send a key value, which Ollama ignores
LM StudioDeveloper tab, or lms server starthttp://localhost:1234/v1Off by default; API tokens can be required in Server Settings
llama.cppllama serve -hf <user>/<model>http://localhost:9931/v1Off by default; --api-key
MLX LMmlx_lm.server --model <path_to_model_or_hf_repo>http://localhost:8080/v1None documented
JanSettings > Local API Server > Start Serverhttp://127.0.0.1:1337/v1Optional API key
vLLMvllm serve <model>http://localhost:8000/v1Off by default; --api-key or VLLM_API_KEY

Expanded article table

Sources: [56][16][76][57][17][9][20][21][6][55]. LM Studio's port is whatever was last used unless --port is given; its documentation's examples assume 1234. [17][16]

A request against Ollama, from its documentation, looks like this: [56]

curl -X POST http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "gpt-oss:20b",
  "messages": [{ "role": "user", "content": "Say this is a test" }]
}'

The official OpenAI client libraries need only a different base URL: [56]

from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',  # required but ignored
)

Compatibility is partial everywhere. Ollama says it supports a subset of the OpenAI API, and llama.cpp's documentation makes no strong claim of conformance beyond saying that the endpoint is enough for many applications. [56][9] One gap matters for agents: the OpenAI API has no field for context size, so with Ollama the larger context has to be set on the server or baked into a model with a Modelfile line of the form PARAMETER num_ctx <context size>. [56] Ollama and llama.cpp also expose a /v1/messages route compatible with the Anthropic API. [1][9]

Opening the server to other machines

Ollama, LM Studio, llama.cpp and Jan document 127.0.0.1 as the default bind address, and MLX LM's server starts on localhost. [12][17][9][21][20] vLLM is the exception: its quickstart shows the server at http://localhost:8000, but in the project's argument definitions the host setting defaults to no value instead of a loopback address, and the launcher then binds an empty host, which it logs as 0.0.0.0: every network interface. [6][58][74] Passing --host 127.0.0.1 keeps a vLLM server on the local machine. For the other tools, making a server reachable from the network is a single setting:

  • Ollama: set the OLLAMA_HOST environment variable, for example Environment="OLLAMA_HOST=0.0.0.0:11434" in the systemd unit on Linux or launchctl setenv OLLAMA_HOST "0.0.0.0:11434" on macOS. [12]
  • LM Studio: lms server start --bind 0.0.0.0. [17]
  • llama.cpp: --host and --port. [9]
  • Jan: set Server Host to 0.0.0.0 in the server configuration. [21]
  • vLLM: --host and --port; as noted above, it already listens on every interface unless --host is given. [6][74]

That setting is also the main security risk of running models locally. Ollama's local server ignores the API key value that clients send, and its FAQ describes exposing the server through a proxy server such as Nginx. [56][12] LM Studio's documentation warns that any bind address other than 127.0.0.1 exposes the server beyond the local machine and recommends enabling authentication. [17] Jan advises against leaving the API key empty when binding to 0.0.0.0. [21] llama.cpp recommends an API key and a reverse proxy for public deployments, and its security policy lists llama-server among the functions not to use in untrusted environments or networks. [9][59] vLLM's --api-key protects only the /v1, /v2, /inference and /cohere path prefixes, and its documentation says not to rely on it alone. [60]

The warnings reflect what has happened. On January 29, 2026, SentinelOne's SentinelLABS and Censys published the results of 293 days of scanning for Ollama servers reachable from the public internet. They counted 175,108 unique hosts in 130 countries over that period, many of them transient, with a persistent core of about 23,000 hosts that generated most of the activity. The report notes that Ollama binds to 127.0.0.1:11434 by design and that exposing it takes a single configuration change, and it found that over 48% of the hosts advertised tool calling. [72] TechRadar's account of the study said that, according to Pillar Security, attackers were already using such servers to generate spam and malware content at the owners' expense, a practice called LLMjacking. [61]

Verify what you downloaded

A model file is written by someone else and then loaded by software that has access to your machine. Four checks reduce the risk.

Use a source you trust

Take the weights from the developer's own repository, or from a publisher you have reason to trust. Hugging Face's security documentation suggests loading models only from users and organizations you trust and relying on signed commits. [62]

Prefer formats that are not executable

Pickle, the default format for PyTorch weights (files such as pytorch_model.bin), can execute arbitrary code when it is loaded. [62] Safetensors is a format for storing tensors safely, as opposed to pickle, and its documentation describes a safe format as one where a file downloaded at random can be loaded without running arbitrary code. [63] GGUF is a binary format that holds tensors and standardized metadata. [35] Hugging Face scans what is uploaded: files under 2 GB go through the ClamAV malware scanner at each commit, pickle files get an import scan whose results are shown next to the file, and third-party scanners such as JFrog's add their own reports. [64][62][65] Hugging Face says its pickle scan is not foolproof and that checking safety remains the user's responsibility. [62]

Treat unknown models as untrusted anyway

A file that is not a pickle is not thereby safe. Hugging Face's documentation notes that pickle is not the only exploitable model format. [65] llama.cpp's security policy tells users to be careful with models from unknown developers, to run untrusted models in a sandbox such as a container or virtual machine, to keep the software updated, and to confirm that the hash of a downloaded file matches a known-good value. [59] Some repositories also ship custom code: MLX LM asks for confirmation before running a model whose tokenizer needs trust_remote_code, unless --trust-remote-code was passed. [4]

Check the checksums

The Hugging Face command-line tool downloads a repository and can verify the local copy against the checksums recorded on the Hub. The documentation's examples are: [66]

hf download HuggingFaceH4/zephyr-7b-beta
hf cache verify deepseek-ai/DeepSeek-OCR
hf cache verify deepseek-ai/DeepSeek-OCR --local-dir /path/to/repo

On a mismatch, hf cache verify prints the affected files and exits with a non-zero status. [66] The tool installs with curl -LsSf https://hf.co/cli/install.sh | bash, with pip install -U "huggingface_hub", or with brew install hf. [66]

Read the license before commercial use

The license of the runtime says nothing about the model. Ollama's MIT license or vLLM's Apache license covers that software; the weights you load come with separate terms set by whoever trained them. LM Studio's terms make the point explicitly: third-party models are provided under an agreement between the user and the model's provider, and complying with it is the user's job. [13]

Where to find the terms:

  • On Hugging Face, the model card metadata carries a license field that is shown on the model page and can be used as a search filter. A custom license appears as license: other with a license_name and a license_link, or as a LICENSE file in the repository. [67]
  • Some repositories are gated: you must submit an access request before the files download, which the authors approve automatically or by hand, and the request shares your Hugging Face username and email address with them unless they have opted out. [68]
  • For a model pulled through Ollama, the /api/show endpoint returns the model's license in its license field. [69]

Custom licenses need reading in full. The Llama 3.1 Community License is one example: it requires anyone who distributes the model or a derivative to include a copy of the agreement and display "Built with Llama", it incorporates an acceptable use policy, and it withdraws the grant from any licensee whose products had more than 700 million monthly active users in the month before the release date, who must ask Meta for a separate license. [70] The conditions on distribution cover derivative works of the model as well as the original files, so it is worth checking that a community GGUF or MLX conversion of such a model carries the same agreement. [70]

A permissive license on the weights does not by itself make a model open source in the sense defined by the Open Source Initiative. The Open Source AI Definition 1.0 also asks for the training and inference code and for detailed information about the training data, all under OSI-approved terms, which is why this guide says "open-weight". [71] Open-Weight LLM License Comparison goes through the main model licenses clause by clause, and the list of open-weight models records the license of each model AI Wiki covers.

Limitations and trade-offs

  • Memory sets a ceiling. A model that does not fit in VRAM still runs from system RAM, but more slowly, and long contexts push memory up further. [1][27]
  • Default contexts are small. Ollama starts at 4k tokens on GPUs with less than 24 GiB, well below the 64,000 it recommends for agents and coding tools. [27]
  • Quantized models are not identical to the originals. Benchmark scores may hold at 4 bits while individual answers change. [39][40]
  • The effect of quantization on speed depends on the software. Jin and colleagues found that quantization can slow inference without engineering support, while llama.cpp's own benchmark of Llama 3.1 8B (on hardware the documentation does not name) generated about 72 tokens per second at Q4_K_M against about 29 for the 16-bit file. [39][26]
  • API compatibility is partial. The local servers implement subsets of the OpenAI API, so a tool written for the hosted service may use a route or field that a given runtime lacks. [56][9]
  • The user is the operator. Updates, network exposure and the trustworthiness of downloaded files fall to whoever runs the machine, as the security guidance of llama.cpp, vLLM and Hugging Face says in different words. [59][60][62]
  • Not every route suits every job. MLX LM's server is not recommended for production, vLLM requires Linux and serves one model per server, and vLLM's GGUF support is experimental. [20][6][37]

Continue reading

More from See also

More to read

Topics:AI InferenceDeveloper ToolsLarge Language ModelsOpen Source AI

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9Ollama. "Quickstart". Ollama documentation, accessed November 19, 2026. docs.ollama.com/quickstart
  2. ^1 ^2 ^3 ^4 ^5LM Studio. "Welcome to LM Studio Docs!". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...app
  3. ^1 ^2 ^3 ^4 ^5ggml-org. "llama.cpp" (README and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. github.com/...llama.cpp
  4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11ml-explore. "MLX LM" (README and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. github.com/...mlx-lm
  5. ^1 ^2 ^3 ^4Menlo Research. "Jan - Open-source ChatGPT replacement" (README and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. github.com/...jan
  6. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11vLLM project. "Quickstart" (docs/getting_started/quickstart.md). vLLM repository, GitHub, accessed November 19, 2026. github.com/...quickstart.md
  7. ^1 ^2 ^3Open WebUI. "OpenAI-Compatible" (documentation source). open-webui/docs repository, GitHub, accessed November 19, 2026. github.com/...starting-with-openai-compatible
  8. ^Hugging Face. "Use AI Models Locally". Hugging Face Hub documentation, accessed November 19, 2026. huggingface.co/...local-apps
  9. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17ggml-org. "LLaMA.cpp HTTP Server" (tools/server/README.md). llama.cpp repository, GitHub, accessed November 19, 2026. github.com/...README.md
  10. ^Ollama. "ollama/ollama" (repository and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. github.com/...ollama
  11. ^1 ^2Ollama. "Importing a Model". Ollama documentation, accessed November 19, 2026. docs.ollama.com/import
  12. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12Ollama. "Local FAQ". Ollama documentation, accessed November 19, 2026. docs.ollama.com/faq
  13. ^1 ^2 ^3LM Studio. "LM Studio App Terms of Service". LM Studio, version of August 23, 2026. lmstudio.ai/app-terms
  14. ^1 ^2 ^3LM Studio. "lms - LM Studio's CLI". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...cli
  15. ^1 ^2 ^3LM Studio. "System Requirements". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...system-requirements
  16. ^1 ^2 ^3 ^4LM Studio. "OpenAI Compatibility Endpoints". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...openai-compat
  17. ^1 ^2 ^3 ^4 ^5 ^6 ^7LM Studio. "lms server start". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...server-start
  18. ^1 ^2ggml-org. "Install pre-built version of llama.cpp" (docs/install.md). llama.cpp repository, GitHub, accessed November 19, 2026. github.com/...install.md
  19. ^1 ^2ggml-org. "Obtaining and quantizing models" (docs/models.md). llama.cpp repository, GitHub, accessed November 19, 2026. github.com/...models.md
  20. ^1 ^2 ^3 ^4 ^5ml-explore. "HTTP Model Server" (mlx_lm/SERVER.md). mlx-lm repository, GitHub, accessed November 19, 2026. github.com/...SERVER.md
  21. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Jan. "Local API Server". Jan documentation, accessed November 19, 2026. docs.jan.ai/...api-server
  22. ^1 ^2 ^3 ^4Open WebUI. "Quick Start" (documentation source, including the Docker and pip tabs). open-webui/docs repository, GitHub, accessed November 19, 2026. github.com/...quick-start
  23. ^1 ^2Open WebUI Inc. "Open WebUI License" (LICENSE file). open-webui/open-webui repository, GitHub, accessed November 19, 2026. github.com/...LICENSE
  24. ^Open WebUI. "open-webui/open-webui" (README; star count from the GitHub API). GitHub, accessed November 19, 2026. github.com/...open-webui
  25. ^vLLM project. "vllm-project/vllm" (repository and LICENSE file; star count from the GitHub API). GitHub, accessed November 19, 2026. github.com/...vllm
  26. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8ggml-org. "quantize" (tools/quantize/README.md). llama.cpp repository, GitHub, accessed November 19, 2026. github.com/...README.md
  27. ^1 ^2 ^3 ^4 ^5 ^6Ollama. "Context length". Ollama documentation, accessed November 19, 2026. docs.ollama.com/context-length
  28. ^1 ^2 ^3LM Studio. "lms load". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...load
  29. ^ml-explore. "Unified Memory" (docs/src/usage/unified_memory.rst). MLX repository, GitHub, accessed November 19, 2026. github.com/...unified_memory.rst
  30. ^1 ^2vLLM project. "Optimization and Tuning" (docs/configuration/optimization.md). vLLM repository, GitHub, accessed November 19, 2026. github.com/...optimization.md
  31. ^1 ^2Ollama. "Hardware support". Ollama documentation, accessed November 19, 2026. docs.ollama.com/gpu
  32. ^Ollama. "macOS". Ollama documentation, accessed November 19, 2026. docs.ollama.com/macos
  33. ^1 ^2 ^3vLLM project. "GPU" installation guide (docs/getting_started/installation/gpu.md and gpu.cuda.inc.md). vLLM repository, GitHub, accessed November 19, 2026. github.com/...gpu.md
  34. ^1 ^2 ^3Jan. "QuickStart". Jan documentation, accessed November 19, 2026. docs.jan.ai/...quickstart
  35. ^1 ^2 ^3Hugging Face. "GGUF". Hugging Face Hub documentation, accessed November 19, 2026. huggingface.co/...gguf
  36. ^1 ^2 ^3 ^4Hugging Face. "Use Ollama with any GGUF Model on Hugging Face Hub". Hugging Face Hub documentation, accessed November 19, 2026. huggingface.co/...ollama
  37. ^1 ^2 ^3vLLM project. "GGUF" (docs/features/quantization/gguf.md). vLLM repository, GitHub, accessed November 19, 2026. github.com/...gguf.md
  38. ^1 ^2 ^3ikawrakow. "k-quants". Pull request #1684, llama.cpp repository, GitHub, opened June 3, 2023, merged June 5, 2023. github.com/...1684
  39. ^1 ^2 ^3 ^4Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, Deyi Xiong. "A Comprehensive Evaluation of Quantization Strategies for Large Language Models". Findings of ACL 2024; arXiv:2402.16775, February 26, 2024. arxiv.org/...2402.16775
  40. ^1 ^2Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, Ramachandran Ramjee. "Accuracy is Not All You Need". NeurIPS 2024; arXiv:2407.09141, July 12, 2024. arxiv.org/...2407.09141
  41. ^vLLM project. "Quantization" (docs/features/quantization/README.md). vLLM repository, GitHub, accessed November 19, 2026. github.com/...README.md
  42. ^Ollama. "Linux". Ollama documentation, accessed November 19, 2026. docs.ollama.com/linux
  43. ^Ollama. "CLI Reference". Ollama documentation, accessed November 19, 2026. docs.ollama.com/cli
  44. ^Ollama. "Docker". Ollama documentation, accessed November 19, 2026. docs.ollama.com/docker
  45. ^Yagil Burowski. "LM Studio is free for use at work". LM Studio Blog, July 8, 2025. lmstudio.ai/...free-for-work
  46. ^LM Studio. "lms get". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...get
  47. ^LM Studio. "Run LM Studio as a service (headless)". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...headless
  48. ^LM Studio. "Offline Operation". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...offline
  49. ^Daniel Bevenius, Georgi Gerganov, merve (ggml-org). "llama.cpp v1.0.0 : A stronger foundation for the local AI ecosystem". Hugging Face blog (team article), November 13, 2026. huggingface.co/...llama-cpp-v1
  50. ^ml-explore. "MLX" (README). GitHub, accessed November 19, 2026. github.com/...mlx
  51. ^1 ^2Open WebUI. "Ollama" (documentation source). open-webui/docs repository, GitHub, accessed November 19, 2026. github.com/...starting-with-ollama.mdx
  52. ^Open WebUI. "Open WebUI License" (documentation source). open-webui/docs repository, GitHub, accessed November 19, 2026. github.com/...license.mdx
  53. ^Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica. "Efficient Memory Management for Large Language Model Serving with PagedAttention". SOSP 2023; arXiv:2309.06180, September 12, 2023. arxiv.org/...2309.06180
  54. ^vLLM project. "Conserving Memory" (docs/configuration/conserving_memory.md). vLLM repository, GitHub, accessed November 19, 2026. github.com/...conserving_memory.md
  55. ^vLLM project. "vLLM CLI Guide" (docs/cli/README.md). vLLM repository, GitHub, accessed November 19, 2026. github.com/...README.md
  56. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Ollama. "OpenAI compatibility". Ollama documentation, accessed November 19, 2026. docs.ollama.com/...openai-compatibility
  57. ^LM Studio. "Authentication". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...authentication
  58. ^vLLM project. "cli_args.py" (vllm/entrypoints/common/cli_args.py, server argument definitions). vLLM repository, GitHub, accessed November 19, 2026. github.com/...cli_args.py
  59. ^1 ^2 ^3ggml-org. "Security Policy" (SECURITY.md). llama.cpp repository, GitHub, accessed November 19, 2026. github.com/...SECURITY.md
  60. ^1 ^2vLLM project. "Security" (docs/usage/security.md). vLLM repository, GitHub, accessed November 19, 2026. github.com/...security.md
  61. ^Sead Fadilpašić. "Over 175,000 publicly exposed Ollama AI servers discovered worldwide - so fix now". TechRadar, January 30, 2026. techradar.com/...s-discovered-worldwide-so-fix-now
  62. ^1 ^2 ^3 ^4 ^5Hugging Face. "Pickle Scanning". Hugging Face Hub documentation, accessed November 19, 2026. huggingface.co/...security-pickle
  63. ^Hugging Face. "safetensors" (README). GitHub, accessed November 19, 2026. github.com/...safetensors
  64. ^Hugging Face. "Malware Scanning". Hugging Face Hub documentation, accessed November 19, 2026. huggingface.co/...security-malware
  65. ^1 ^2Hugging Face. "Third-party scanner: JFrog". Hugging Face Hub documentation, accessed November 19, 2026. huggingface.co/...security-jfrog
  66. ^1 ^2 ^3Hugging Face. "Command Line Interface (CLI)". huggingface_hub documentation, accessed November 19, 2026. huggingface.co/...cli
  67. ^Hugging Face. "Model Cards". Hugging Face Hub documentation, accessed November 19, 2026. huggingface.co/...model-cards
  68. ^Hugging Face. "Gated models". Hugging Face Hub documentation, accessed November 19, 2026. huggingface.co/...models-gated
  69. ^Ollama. "Show model details". Ollama API reference, accessed November 19, 2026. docs.ollama.com/...show-model-details
  70. ^1 ^2Meta. "Llama 3.1 Community License Agreement" (models/llama3_1/LICENSE). meta-llama/llama-models repository, GitHub; Llama 3.1 version release date July 23, 2024. github.com/...LICENSE
  71. ^Open Source Initiative. "The Open Source AI Definition - 1.0". Accessed November 19, 2026. opensource.org/...open-source-ai-definition
  72. ^1 ^2Gabriel Bernadett-Shapiro, Silas Cutler. "Silent Brothers | Ollama Hosts Form Anonymous AI Network Beyond Platform Guardrails". SentinelLABS (SentinelOne), January 29, 2026, updated March 6, 2026. sentinelone.com/...work-beyond-platform-guardrails
  73. ^Xuan-Son Nguyen (ngxson). "server: use port 9931 by default". Pull request #30159, llama.cpp repository, GitHub, merged October 9, 2026. github.com/...30159
  74. ^1 ^2 ^3vLLM project. "launcher.py" (vllm/entrypoints/launchers/launcher.py, server socket set-up). vLLM repository, GitHub, accessed November 19, 2026. github.com/...launcher.py
  75. ^Ollama. "config.go" (envconfig/config.go, server environment settings). ollama/ollama repository, GitHub, accessed November 19, 2026. github.com/...config.go
  76. ^LM Studio. "LM Studio as a Local LLM API Server". LM Studio documentation, accessed November 19, 2026. lmstudio.ai/...server

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

1 revision · v1 · 6,665 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 19, 2026: 212 claims checked against 84 cited and independent sources; 3 material and 11 minor corrections applied before publication.

Cite this page: AI Wiki. "How to Run LLMs Locally." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/how_to_run_llms_locally

Edit article