VuongKB/reference/llama-server-settings.md

---
created: '2026-04-10T01:19:06+00:00'
id: c693060b-7794-4d1c-995d-53b6c951352c
keywords:
- llama-server
- llama.cpp
- HTTP server
- GGUF
- OpenAI API
- Anthropic API
- inference
- completion
- chat completions
- embeddings
- reranking
- speculative decoding
- continuous batching
- KV cache
- sampling
- router mode
- multimodal
- LoRA
- function calling
links: []
modified: '2026-04-10T01:19:06+00:00'
summary: Comprehensive reference for llama-server (LLaMA.cpp HTTP Server) covering
  all command-line parameters (common, sampling, server-specific), API endpoints (native
  and OpenAI/Anthropic-compatible), router mode for multi-model serving, build instructions,
  and usage examples.
tags:
- reference
- llama-cpp
- llama-server
- inference-server
- api
- openai-compatible
title: LLaMA.cpp HTTP Server (llama-server) — Settings & API Reference
---

# LLaMA.cpp HTTP Server (llama-server)

Fast, lightweight, pure C/C++ HTTP server based on [httplib](https://github.com/yhirose/cpp-httplib), [nlohmann::json](https://github.com/nlohmann/json) and **llama.cpp**.

Set of LLM REST APIs and a web UI to interact with llama.cpp.

**Features:**
- LLM inference of F16 and quantized models on GPU and CPU
- [OpenAI API](https://github.com/openai/openai-openapi) compatible chat completions, responses, and embeddings routes
- [Anthropic Messages API](https://docs.anthropic.com/en/api/messages) compatible chat completions
- Reranking endpoint (https://github.com/ggml-org/llama.cpp/pull/9510)
- Parallel decoding with multi-user support
- Continuous batching
- Multimodal ([documentation](../../docs/multimodal.md)) / with OpenAI-compatible API support
- Monitoring endpoints
- Schema-constrained JSON response format
- Prefilling of assistant messages similar to the Claude API
- [Function calling](../../docs/function-calling.md) / tool use for ~any model
- Speculative decoding
- Easy-to-use web UI

For the full list of features, please refer to [server's changelog](https://github.com/ggml-org/llama.cpp/issues/9291)

## Usage

### Common params

| Argument | Explanation |
| -------- | ----------- |
| `-h, --help, --usage` | print usage and exit |
| `--version` | show version and build info |
| `--license` | show source code license and dependencies |
| `-cl, --cache-list` | show list of models in cache |
| `--completion-bash` | print source-able bash completion script for llama.cpp |
| `-t, --threads N` | number of CPU threads to use during generation (default: -1) (env: `LLAMA_ARG_THREADS`) |
| `-tb, --threads-batch N` | number of threads to use during batch and prompt processing (default: same as --threads) |
| `-C, --cpu-mask M` | CPU affinity mask: arbitrarily long hex. Complements cpu-range (default: "") |
| `-Cr, --cpu-range lo-hi` | range of CPUs for affinity. Complements --cpu-mask |
| `--cpu-strict <0\|1>` | use strict CPU placement (default: 0) |
| `--prio N` | set process/thread priority : low(-1), normal(0), medium(1), high(2), realtime(3) (default: 0) |
| `--poll <0...100>` | use polling level to wait for work (0 - no polling, default: 50) |
| `-Cb, --cpu-mask-batch M` | CPU affinity mask: arbitrarily long hex. Complements cpu-range-batch (default: same as --cpu-mask) |
| `-Crb, --cpu-range-batch lo-hi` | ranges of CPUs for affinity. Complements --cpu-mask-batch |
| `--cpu-strict-batch <0\|1>` | use strict CPU placement (default: same as --cpu-strict) |
| `--prio-batch N` | set process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |
| `--poll-batch <0\|1>` | use polling to wait for work (default: same as --poll) |
| `-c, --ctx-size N` | size of the prompt context (default: 0, 0 = loaded from model) (env: `LLAMA_ARG_CTX_SIZE`) |
| `-n, --predict, --n-predict N` | number of tokens to predict (default: -1, -1 = infinity) (env: `LLAMA_ARG_N_PREDICT`) |
| `-b, --batch-size N` | logical maximum batch size (default: 2048) (env: `LLAMA_ARG_BATCH`) |
| `-ub, --ubatch-size N` | physical maximum batch size (default: 512) (env: `LLAMA_ARG_UBATCH`) |
| `--keep N` | number of tokens to keep from the initial prompt (default: 0, -1 = all) |
| `--swa-full` | use full-size SWA cache (default: false) (env: `LLAMA_ARG_SWA_FULL`) |
| `-fa, --flash-attn [on\|off\|auto]` | set Flash Attention use ('on', 'off', or 'auto', default: 'auto') (env: `LLAMA_ARG_FLASH_ATTN`) |
| `--perf, --no-perf` | whether to enable internal libllama performance timings (default: false) (env: `LLAMA_ARG_PERF`) |
| `-e, --escape, --no-escape` | whether to process escapes sequences (\n, \r, \t, \', \", \\) (default: true) |
| `--rope-scaling {none,linear,yarn}` | RoPE frequency scaling method, defaults to linear unless specified by the model (env: `LLAMA_ARG_ROPE_SCALING_TYPE`) |
| `--rope-scale N` | RoPE context scaling factor, expands context by a factor of N (env: `LLAMA_ARG_ROPE_SCALE`) |
| `--rope-freq-base N` | RoPE base frequency, used by NTK-aware scaling (default: loaded from model) (env: `LLAMA_ARG_ROPE_FREQ_BASE`) |
| `--rope-freq-scale N` | RoPE frequency scaling factor, expands context by a factor of 1/N (env: `LLAMA_ARG_ROPE_FREQ_SCALE`) |
| `--yarn-orig-ctx N` | YaRN: original context size of model (default: 0 = model training context size) (env: `LLAMA_ARG_YARN_ORIG_CTX`) |
| `--yarn-ext-factor N` | YaRN: extrapolation mix factor (default: -1.00, 0.0 = full interpolation) (env: `LLAMA_ARG_YARN_EXT_FACTOR`) |
| `--yarn-attn-factor N` | YaRN: scale sqrt(t) or attention magnitude (default: -1.00) (env: `LLAMA_ARG_YARN_ATTN_FACTOR`) |
| `--yarn-beta-slow N` | YaRN: high correction dim or alpha (default: -1.00) (env: `LLAMA_ARG_YARN_BETA_SLOW`) |
| `--yarn-beta-fast N` | YaRN: low correction dim or beta (default: -1.00) (env: `LLAMA_ARG_YARN_BETA_FAST`) |
| `-kvo, --kv-offload, -nkvo, --no-kv-offload` | whether to enable KV cache offloading (default: enabled) (env: `LLAMA_ARG_KV_OFFLOAD`) |
| `--repack, -nr, --no-repack` | whether to enable weight repacking (default: enabled) (env: `LLAMA_ARG_REPACK`) |
| `--no-host` | bypass host buffer allowing extra buffers to be used (env: `LLAMA_ARG_NO_HOST`) |
| `-ctk, --cache-type-k TYPE` | KV cache data type for K. Allowed: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: `LLAMA_ARG_CACHE_TYPE_K`) |
| `-ctv, --cache-type-v TYPE` | KV cache data type for V. Allowed: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: `LLAMA_ARG_CACHE_TYPE_V`) |
| `-dt, --defrag-thold N` | KV cache defragmentation threshold (DEPRECATED) (env: `LLAMA_ARG_DEFRAG_THOLD`) |
| `--mlock` | force system to keep model in RAM rather than swapping or compressing (env: `LLAMA_ARG_MLOCK`) |
| `--mmap, --no-mmap` | whether to memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: enabled) (env: `LLAMA_ARG_MMAP`) |
| `-dio, --direct-io, -ndio, --no-direct-io` | use DirectIO if available (default: disabled) (env: `LLAMA_ARG_DIO`) |
| `--numa TYPE` | attempt optimizations that help on some NUMA systems. Options: distribute, isolate, numactl (env: `LLAMA_ARG_NUMA`) |
| `-dev, --device <dev1,dev2,..>` | comma-separated list of devices to use for offloading (none = don't offload) (env: `LLAMA_ARG_DEVICE`) |
| `--list-devices` | print list of available devices and exit |
| `-ot, --override-tensor <pattern>=<buffer type>,...` | override tensor buffer type (env: `LLAMA_ARG_OVERRIDE_TENSOR`) |
| `-cmoe, --cpu-moe` | keep all Mixture of Experts (MoE) weights in the CPU (env: `LLAMA_ARG_CPU_MOE`) |
| `-ncmoe, --n-cpu-moe N` | keep the MoE weights of the first N layers in the CPU (env: `LLAMA_ARG_N_CPU_MOE`) |
| `-ngl, --gpu-layers, --n-gpu-layers N` | max number of layers to store in VRAM: exact number, 'auto', or 'all' (default: auto) (env: `LLAMA_ARG_N_GPU_LAYERS`) |
| `-sm, --split-mode {none,layer,row}` | how to split model across multiple GPUs (default: layer) (env: `LLAMA_ARG_SPLIT_MODE`) |
| `-ts, --tensor-split N0,N1,N2,...` | fraction of model to offload to each GPU, comma-separated proportions (env: `LLAMA_ARG_TENSOR_SPLIT`) |
| `-mg, --main-gpu INDEX` | GPU to use for model or intermediate results (default: 0) (env: `LLAMA_ARG_MAIN_GPU`) |
| `-fit, --fit [on\|off]` | whether to adjust unset arguments to fit in device memory (default: 'on') (env: `LLAMA_ARG_FIT`) |
| `-fitt, --fit-target MiB0,MiB1,...` | target margin per device for --fit, comma-separated (default: 1024) (env: `LLAMA_ARG_FIT_TARGET`) |
| `-fitc, --fit-ctx N` | minimum ctx size that can be set by --fit option (default: 4096) (env: `LLAMA_ARG_FIT_CTX`) |
| `--check-tensors` | check model tensor data for invalid values (default: false) |
| `--override-kv KEY=TYPE:VALUE,...` | override model metadata by key. Types: int, float, bool, str |
| `--op-offload, --no-op-offload` | whether to offload host tensor operations to device (default: true) |
| `--lora FNAME` | path to LoRA adapter (comma-separated for multiple) |
| `--lora-scaled FNAME:SCALE,...` | path to LoRA adapter with user defined scaling |
| `--control-vector FNAME` | add a control vector (comma-separated for multiple) |
| `--control-vector-scaled FNAME:SCALE,...` | add a control vector with user defined scaling |
| `--control-vector-layer-range START END` | layer range to apply the control vector(s) to |
| `-m, --model FNAME` | model path to load (env: `LLAMA_ARG_MODEL`) |
| `-mu, --model-url MODEL_URL` | model download url (env: `LLAMA_ARG_MODEL_URL`) |
| `-dr, --docker-repo [<repo>/]<model>[:quant]` | Docker Hub model repository (env: `LLAMA_ARG_DOCKER_REPO`) |
| `-hf, -hfr, --hf-repo <user>/<model>[:quant]` | Hugging Face model repository (env: `LLAMA_ARG_HF_REPO`) |
| `-hfd, -hfrd, --hf-repo-draft <user>/<model>[:quant]` | Same as --hf-repo, but for the draft model (env: `LLAMA_ARG_HFD_REPO`) |
| `-hff, --hf-file FILE` | Hugging Face model file, overrides quant in --hf-repo (env: `LLAMA_ARG_HF_FILE`) |
| `-hfv, -hfrv, --hf-repo-v <user>/<model>[:quant]` | Hugging Face model repository for vocoder model (env: `LLAMA_ARG_HF_REPO_V`) |
| `-hffv, --hf-file-v FILE` | Hugging Face model file for vocoder model (env: `LLAMA_ARG_HF_FILE_V`) |
| `-hft, --hf-token TOKEN` | Hugging Face access token (env: `HF_TOKEN`) |
| `--log-disable` | Log disable |
| `--log-file FNAME` | Log to file (env: `LLAMA_LOG_FILE`) |
| `--log-colors [on\|off\|auto]` | Set colored logging (default: 'auto') (env: `LLAMA_LOG_COLORS`) |
| `-v, --verbose, --log-verbose` | Set verbosity level to infinity |
| `--offline` | Offline mode: forces use of cache, prevents network access (env: `LLAMA_OFFLINE`) |
| `-lv, --verbosity, --log-verbosity N` | Set verbosity threshold. 0=generic, 1=error, 2=warning, 3=info, 4=debug (default: 3) (env: `LLAMA_LOG_VERBOSITY`) |
| `--log-prefix` | Enable prefix in log messages (env: `LLAMA_LOG_PREFIX`) |
| `--log-timestamps` | Enable timestamps in log messages (env: `LLAMA_LOG_TIMESTAMPS`) |
| `-ctkd, --cache-type-k-draft TYPE` | KV cache data type for K for the draft model (default: f16) (env: `LLAMA_ARG_CACHE_TYPE_K_DRAFT`) |
| `-ctvd, --cache-type-v-draft TYPE` | KV cache data type for V for the draft model (default: f16) (env: `LLAMA_ARG_CACHE_TYPE_V_DRAFT`) |

### Sampling params

| Argument | Explanation |
| -------- | ----------- |
| `--samplers SAMPLERS` | samplers used for generation, separated by ';' (default: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature) |
| `-s, --seed SEED` | RNG seed (default: -1, use random seed for -1) |
| `--sampler-seq, --sampling-seq SEQUENCE` | simplified sequence for samplers (default: edskypmxt) |
| `--ignore-eos` | ignore end of stream token and continue generating |
| `--temp, --temperature N` | temperature (default: 0.80) |
| `--top-k N` | top-k sampling (default: 40, 0 = disabled) (env: `LLAMA_ARG_TOP_K`) |
| `--top-p N` | top-p sampling (default: 0.95, 1.0 = disabled) |
| `--min-p N` | min-p sampling (default: 0.05, 0.0 = disabled) |
| `--top-nsigma, --top-n-sigma N` | top-n-sigma sampling (default: -1.00, -1.0 = disabled) |
| `--xtc-probability N` | xtc probability (default: 0.00, 0.0 = disabled) |
| `--xtc-threshold N` | xtc threshold (default: 0.10, 1.0 = disabled) |
| `--typical, --typical-p N` | locally typical sampling, parameter p (default: 1.00, 1.0 = disabled) |
| `--repeat-last-n N` | last n tokens to consider for penalize (default: 64, 0 = disabled, -1 = ctx_size) |
| `--repeat-penalty N` | penalize repeat sequence of tokens (default: 1.00, 1.0 = disabled) |
| `--presence-penalty N` | repeat alpha presence penalty (default: 0.00, 0.0 = disabled) |
| `--frequency-penalty N` | repeat alpha frequency penalty (default: 0.00, 0.0 = disabled) |
| `--dry-multiplier N` | DRY sampling multiplier (default: 0.00, 0.0 = disabled) |
| `--dry-base N` | DRY sampling base value (default: 1.75) |
| `--dry-allowed-length N` | allowed length for DRY sampling (default: 2) |
| `--dry-penalty-last-n N` | DRY penalty for the last n tokens (default: -1, 0 = disable, -1 = context size) |
| `--dry-sequence-breaker STRING` | add sequence breaker for DRY sampling; use "none" for no breakers |
| `--adaptive-target N` | adaptive-p: select tokens near this probability (default: -1.00, disabled) |
| `--adaptive-decay N` | adaptive-p: decay rate for target adaptation (default: 0.90) |
| `--dynatemp-range N` | dynamic temperature range (default: 0.00, 0.0 = disabled) |
| `--dynatemp-exp N` | dynamic temperature exponent (default: 1.00) |
| `--mirostat N` | use Mirostat sampling (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0) |
| `--mirostat-lr N` | Mirostat learning rate, parameter eta (default: 0.10) |
| `--mirostat-ent N` | Mirostat target entropy, parameter tau (default: 5.00) |
| `-l, --logit-bias TOKEN_ID(+/-)BIAS` | modifies the likelihood of token appearing in the completion |
| `--grammar GRAMMAR` | BNF-like grammar to constrain generations |
| `--grammar-file FNAME` | file to read grammar from |
| `-j, --json-schema SCHEMA` | JSON schema to constrain generations |
| `-jf, --json-schema-file FILE` | File containing a JSON schema to constrain generations |
| `-bs, --backend-sampling` | enable backend sampling (experimental) (env: `LLAMA_ARG_BACKEND_SAMPLING`) |

### Server-specific params

| Argument | Explanation |
| -------- | ----------- |
| `-lcs, --lookup-cache-static FNAME` | path to static lookup cache for lookup decoding |
| `-lcd, --lookup-cache-dynamic FNAME` | path to dynamic lookup cache for lookup decoding |
| `-ctxcp, --ctx-checkpoints, --swa-checkpoints N` | max context checkpoints per slot (default: 32) (env: `LLAMA_ARG_CTX_CHECKPOINTS`) |
| `-cpent, --checkpoint-every-n-tokens N` | checkpoint every n tokens during prefill (default: 8192) (env: `LLAMA_ARG_CHECKPOINT_EVERY_NT`) |
| `-cram, --cache-ram N` | maximum cache size in MiB (default: 8192, -1 = no limit, 0 = disable) (env: `LLAMA_ARG_CACHE_RAM`) |
| `-kvu, --kv-unified, -no-kvu, --no-kv-unified` | single unified KV buffer shared across all sequences (env: `LLAMA_ARG_KV_UNIFIED`) |
| `--clear-idle, --no-clear-idle` | save and clear idle slots on new task (default: enabled) (env: `LLAMA_ARG_CLEAR_IDLE`) |
| `--context-shift, --no-context-shift` | context shift on infinite text generation (default: disabled) (env: `LLAMA_ARG_CONTEXT_SHIFT`) |
| `-r, --reverse-prompt PROMPT` | halt generation at PROMPT |
| `-sp, --special` | special tokens output enabled (default: false) |
| `--warmup, --no-warmup` | perform warmup with empty run (default: enabled) |
| `--spm-infill` | use Suffix/Prefix/Middle pattern for infill (default: disabled) |
| `--pooling {none,mean,cls,last,rank}` | pooling type for embeddings (env: `LLAMA_ARG_POOLING`) |
| `-np, --parallel N` | number of server slots (default: -1, -1 = auto) (env: `LLAMA_ARG_N_PARALLEL`) |
| `-cb, --cont-batching, -nocb, --no-cont-batching` | continuous batching / dynamic batching (default: enabled) (env: `LLAMA_ARG_CONT_BATCHING`) |
| `-mm, --mmproj FILE` | path to multimodal projector file (env: `LLAMA_ARG_MMPROJ`) |
| `-mmu, --mmproj-url URL` | URL to multimodal projector file (env: `LLAMA_ARG_MMPROJ_URL`) |
| `--mmproj-auto, --no-mmproj, --no-mmproj-auto` | auto-detect multimodal projector (default: enabled) (env: `LLAMA_ARG_MMPROJ_AUTO`) |
| `--mmproj-offload, --no-mmproj-offload` | GPU offloading for multimodal projector (default: enabled) (env: `LLAMA_ARG_MMPROJ_OFFLOAD`) |
| `--image-min-tokens N` | minimum tokens per image for vision models with dynamic resolution (env: `LLAMA_ARG_IMAGE_MIN_TOKENS`) |
| `--image-max-tokens N` | maximum tokens per image for vision models with dynamic resolution (env: `LLAMA_ARG_IMAGE_MAX_TOKENS`) |
| `-otd, --override-tensor-draft <pattern>=<buffer type>,...` | override tensor buffer type for draft model |
| `-cmoed, --cpu-moe-draft` | keep all MoE weights in CPU for draft model (env: `LLAMA_ARG_CPU_MOE_DRAFT`) |
| `-ncmoed, --n-cpu-moe-draft N` | keep MoE weights of first N layers in CPU for draft model (env: `LLAMA_ARG_N_CPU_MOE_DRAFT`) |
| `-a, --alias STRING` | model name aliases, comma-separated (env: `LLAMA_ARG_ALIAS`) |
| `--tags STRING` | model tags, comma-separated (env: `LLAMA_ARG_TAGS`) |
| `--host HOST` | ip address to listen, or UNIX socket if ends with .sock (default: 127.0.0.1) (env: `LLAMA_ARG_HOST`) |
| `--port PORT` | port to listen (default: 8080) (env: `LLAMA_ARG_PORT`) |
| `--reuse-port` | allow multiple sockets to bind to same port (env: `LLAMA_ARG_REUSE_PORT`) |
| `--path PATH` | path to serve static files from (env: `LLAMA_ARG_STATIC_PATH`) |
| `--api-prefix PREFIX` | prefix path the server serves from (env: `LLAMA_ARG_API_PREFIX`) |
| `--webui-config JSON` | JSON for default WebUI settings (env: `LLAMA_ARG_WEBUI_CONFIG`) |
| `--webui-config-file PATH` | JSON file for default WebUI settings (env: `LLAMA_ARG_WEBUI_CONFIG_FILE`) |
| `--webui-mcp-proxy, --no-webui-mcp-proxy` | enable MCP CORS proxy - experimental, do not enable in untrusted environments (env: `LLAMA_ARG_WEBUI_MCP_PROXY`) |
| `--tools TOOL1,TOOL2,...` | enable built-in tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, apply_diff (env: `LLAMA_ARG_TOOLS`) |
| `--webui, --no-webui` | enable the Web UI (default: enabled) (env: `LLAMA_ARG_WEBUI`) |
| `--embedding, --embeddings` | restrict to embedding use case only (env: `LLAMA_ARG_EMBEDDINGS`) |
| `--rerank, --reranking` | enable reranking endpoint (env: `LLAMA_ARG_RERANKING`) |
| `--api-key KEY` | API key(s) for authentication, comma-separated (env: `LLAMA_API_KEY`) |
| `--api-key-file FNAME` | path to file containing API keys |
| `--ssl-key-file FNAME` | PEM-encoded SSL private key (env: `LLAMA_ARG_SSL_KEY_FILE`) |
| `--ssl-cert-file FNAME` | PEM-encoded SSL certificate (env: `LLAMA_ARG_SSL_CERT_FILE`) |
| `--chat-template-kwargs STRING` | additional params for json template parser (env: `LLAMA_CHAT_TEMPLATE_KWARGS`) |
| `-to, --timeout N` | server read/write timeout in seconds (default: 600) (env: `LLAMA_ARG_TIMEOUT`) |
| `--threads-http N` | threads for HTTP request processing (default: -1) (env: `LLAMA_ARG_THREADS_HTTP`) |
| `--cache-prompt, --no-cache-prompt` | prompt caching (default: enabled) (env: `LLAMA_ARG_CACHE_PROMPT`) |
| `--cache-reuse N` | min chunk size for KV shifting cache reuse (default: 0) (env: `LLAMA_ARG_CACHE_REUSE`) |
| `--metrics` | enable prometheus compatible metrics endpoint (env: `LLAMA_ARG_ENDPOINT_METRICS`) |
| `--props` | enable changing global properties via POST /props (env: `LLAMA_ARG_ENDPOINT_PROPS`) |
| `--slots, --no-slots` | expose slots monitoring endpoint (default: enabled) (env: `LLAMA_ARG_ENDPOINT_SLOTS`) |
| `--slot-save-path PATH` | path to save slot kv cache |
| `--media-path PATH` | directory for loading local media files |
| `--models-dir PATH` | directory containing models for router server (env: `LLAMA_ARG_MODELS_DIR`) |
| `--models-preset PATH` | path to INI file containing model presets (env: `LLAMA_ARG_MODELS_PRESET`) |
| `--models-max N` | max models to load simultaneously in router (default: 4, 0 = unlimited) (env: `LLAMA_ARG_MODELS_MAX`) |
| `--models-autoload, --no-models-autoload` | auto-load models in router mode (default: enabled) (env: `LLAMA_ARG_MODELS_AUTOLOAD`) |
| `--jinja, --no-jinja` | use jinja template engine for chat (default: enabled) (env: `LLAMA_ARG_JINJA`) |
| `--reasoning-format FORMAT` | thought tag handling: none, deepseek, deepseek-legacy (default: auto) (env: `LLAMA_ARG_THINK`) |
| `-rea, --reasoning [on\|off\|auto]` | reasoning/thinking in chat (default: auto) (env: `LLAMA_ARG_REASONING`) |
| `--reasoning-budget N` | token budget for thinking: -1 unrestricted, 0 immediate end, N>0 budget (default: -1) (env: `LLAMA_ARG_THINK_BUDGET`) |
| `--reasoning-budget-message MESSAGE` | message injected when reasoning budget exhausted (env: `LLAMA_ARG_THINK_BUDGET_MESSAGE`) |
| `--chat-template JINJA_TEMPLATE` | custom jinja chat template or built-in name (env: `LLAMA_ARG_CHAT_TEMPLATE`) |
| `--chat-template-file JINJA_TEMPLATE_FILE` | custom jinja chat template file (env: `LLAMA_ARG_CHAT_TEMPLATE_FILE`) |
| `--skip-chat-parsing, --no-skip-chat-parsing` | force pure content parser (env: `LLAMA_ARG_SKIP_CHAT_PARSING`) |
| `--prefill-assistant, --no-prefill-assistant` | prefill assistant response (default: enabled) (env: `LLAMA_ARG_PREFILL_ASSISTANT`) |
| `-sps, --slot-prompt-similarity SIMILARITY` | prompt similarity threshold for slot reuse (default: 0.10) |
| `--lora-init-without-apply` | load LoRA adapters without applying |
| `--sleep-idle-seconds SECONDS` | seconds of idleness before server sleeps (default: -1, disabled) |
| `-td, --threads-draft N` | threads for draft model generation |
| `-tbd, --threads-batch-draft N` | threads for draft model batch processing |
| `--draft, --draft-n, --draft-max N` | tokens to draft for speculative decoding (default: 16) (env: `LLAMA_ARG_DRAFT_MAX`) |
| `--draft-min, --draft-n-min N` | minimum draft tokens for speculative decoding (default: 0) (env: `LLAMA_ARG_DRAFT_MIN`) |
| `--draft-p-min P` | minimum speculative decoding probability (default: 0.75) (env: `LLAMA_ARG_DRAFT_P_MIN`) |
| `-cd, --ctx-size-draft N` | prompt context size for draft model (env: `LLAMA_ARG_CTX_SIZE_DRAFT`) |
| `-devd, --device-draft <dev1,dev2,..>` | devices for draft model offloading |
| `-ngld, --gpu-layers-draft N` | draft model GPU layers (default: auto) (env: `LLAMA_ARG_N_GPU_LAYERS_DRAFT`) |
| `-md, --model-draft FNAME` | draft model for speculative decoding (env: `LLAMA_ARG_MODEL_DRAFT`) |
| `--spec-replace TARGET DRAFT` | translate strings between main and draft model |
| `--spec-type [none\|ngram-cache\|ngram-simple\|ngram-map-k\|ngram-map-k4v\|ngram-mod]` | speculative decoding type without draft model (env: `LLAMA_ARG_SPEC_TYPE`) |
| `--spec-ngram-size-n N` | ngram size N for ngram speculative decoding (default: 12) |
| `--spec-ngram-size-m N` | ngram size M for ngram speculative decoding (default: 48) |
| `--spec-ngram-min-hits N` | minimum hits for ngram-map speculative decoding (default: 1) |
| `-mv, --model-vocoder FNAME` | vocoder model for audio generation |
| `--tts-use-guide-tokens` | Use guide tokens to improve TTS word recall |
| `--embd-gemma-default` | use default EmbeddingGemma model |
| `--fim-qwen-1.5b-default` | use default Qwen 2.5 Coder 1.5B |
| `--fim-qwen-3b-default` | use default Qwen 2.5 Coder 3B |
| `--fim-qwen-7b-default` | use default Qwen 2.5 Coder 7B |
| `--fim-qwen-7b-spec` | use Qwen 2.5 Coder 7B + 0.5B draft |
| `--fim-qwen-14b-spec` | use Qwen 2.5 Coder 14B + 0.5B draft |
| `--fim-qwen-30b-default` | use default Qwen 3 Coder 30B A3B Instruct |
| `--gpt-oss-20b-default` | use gpt-oss-20b |
| `--gpt-oss-120b-default` | use gpt-oss-120b |
| `--vision-gemma-4b-default` | use Gemma 3 4B QAT |
| `--vision-gemma-12b-default` | use Gemma 3 12B QAT |

## Environment Variables

If both command line argument and environment variable are set for the same param, the argument takes precedence.

For boolean options like `--mmap` or `--kv-offload`, environment variable handling:
- `LLAMA_ARG_MMAP=true` / `1` / `on` / `enabled` → enabled
- `LLAMA_ARG_MMAP=false` / `0` / `off` / `disabled` → disabled
- Presence of `LLAMA_ARG_NO_MMAP` → disabled

### Docker Compose Example

```yml
services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 4096
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_ENDPOINT_METRICS: 1
      LLAMA_ARG_PORT: 8080
```

## Multimodal Support

Multimodal support is currently an experimental feature. Available in:
- OAI-compatible chat endpoint
- Non-OAI-compatible completions endpoint
- Non-OAI-compatible embeddings endpoint

The multimodal projector is auto-detected when using `-hf`. Use `--no-mmproj` to disable.

## Built-in Tools

Built-in tools enable the LLM to access the local file system from the Web UI. Start with `--tools all` or specify individual tools: `--tools read_file,file_glob_search,...`.

Available tools: `read_file`, `file_glob_search`, `grep_search`, `exec_shell_command`, `write_file`, `edit_file`, `apply_diff`

> ⚠️ Do not enable in untrusted environments.

## Build

### Standard Build

```bash
cmake -B build
cmake --build build --config Release -t llama-server
```

Binary: `./build/bin/llama-server`

### Build with SSL (OpenSSL 3)

```bash
cmake -B build -DLLAMA_OPENSSL=ON
cmake --build build --config Release -t llama-server
```

## Quick Start

### Unix (Linux, macOS)

```bash
./llama-server -m models/7B/ggml-model.gguf -c 2048
```

### Windows

```powershell
llama-server.exe -m models\7B\ggml-model.gguf -c 2048
```

Listens on `127.0.0.1:8080` by default.

### Docker

```bash
docker run -p 8080:8080 -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:server -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080

# With CUDA:
docker run -p 8080:8080 -v /path/to/models:/models --gpus all ghcr.io/ggml-org/llama.cpp:server-cuda -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080 --n-gpu-layers 99
```

## Using with cURL

```sh
curl --request POST \
    --url http://localhost:8080/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'
```

---

## API Endpoints

### GET `/health` — Health Check

Public endpoint (no API key). `/v1/health` also works.

- **503**: `{"error": {"code": 503, "message": "Loading model", "type": "unavailable_error"}}`
- **200**: `{"status": "ok"}`

---

### POST `/completion` — Text Completion

> ⚠️ Not OAI-compatible. Use `/v1/completions` for OAI clients.

**Request options:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `prompt` | string/array/object | required | Prompt as string, token array, mixed, or JSON with multimodal data |
| `temperature` | float | 0.8 | Randomness of generated text |
| `dynatemp_range` | float | 0.0 | Dynamic temperature range |
| `dynatemp_exponent` | float | 1.0 | Dynamic temperature exponent |
| `top_k` | int | 40 | Limit to K most probable tokens |
| `top_p` | float | 0.95 | Nucleus sampling threshold |
| `min_p` | float | 0.05 | Minimum probability relative to most likely token |
| `n_predict` | int | -1 | Max tokens to predict (-1 = infinity) |
| `n_indent` | int | 0 | Min line indentation (for code completion) |
| `n_keep` | int | 0 | Tokens to retain when context exceeded (-1 = all) |
| `n_cmpl` | int | - | Number of completions per prompt |
| `n_cache_reuse` | int | 0 | Min chunk size for KV shifting cache reuse |
| `stream` | bool | false | Stream tokens in real-time via SSE |
| `stop` | array | [] | Stopping strings (not included in output) |
| `typical_p` | float | 1.0 | Locally typical sampling parameter |
| `repeat_penalty` | float | 1.1 | Repetition penalty |
| `repeat_last_n` | int | 64 | Tokens to consider for repeat penalty |
| `presence_penalty` | float | 0.0 | Presence penalty |
| `frequency_penalty` | float | 0.0 | Frequency penalty |
| `dry_multiplier` | float | 0.0 | DRY repetition penalty multiplier |
| `dry_base` | float | 1.75 | DRY penalty base |
| `dry_allowed_length` | int | 2 | DRY allowed repeating length |
| `dry_penalty_last_n` | int | -1 | Tokens to scan for DRY |
| `dry_sequence_breakers` | array | ['\n',':','"','*'] | DRY sequence breakers |
| `xtc_probability` | float | 0.0 | XTC token removal chance |
| `xtc_threshold` | float | 0.1 | XTC minimum probability threshold |
| `mirostat` | int | 0 | Mirostat mode (0=off, 1, 2) |
| `mirostat_tau` | float | 5.0 | Mirostat target entropy |
| `mirostat_eta` | float | 0.1 | Mirostat learning rate |
| `grammar` | string | - | BNF grammar for constrained generation |
| `json_schema` | object | - | JSON schema for constrained generation |
| `seed` | int | -1 | RNG seed (-1 = random) |
| `ignore_eos` | bool | false | Ignore EOS token |
| `logit_bias` | array/object | [] | Modify token likelihoods |
| `n_probs` | int | 0 | Return top N token probabilities |
| `min_keep` | int | 0 | Force minimum tokens from samplers |
| `t_max_predict_ms` | int | 0 | Time limit for prediction phase (ms) |
| `id_slot` | int | -1 | Assign to specific slot |
| `cache_prompt` | bool | true | Re-use KV cache from previous request |
| `return_tokens` | bool | false | Return raw token IDs |
| `samplers` | array | [dry,top_k,typ_p,top_p,min_p,xtc,temperature] | Sampler order |
| `timings_per_token` | bool | false | Include speed info per token |
| `return_progress` | bool | false | Include prompt processing progress in stream mode |
| `post_sampling_probs` | bool | false | Return probabilities after sampling |
| `response_fields` | array | - | List of specific response fields to return |
| `lora` | array | - | Per-request LoRA adapters: `[{"id": 0, "scale": 0.5}]` |

**Prompt input shapes:**
- Single string: `"string"`
- Single token sequence: `[12, 34, 56]`
- Mixed tokens and strings: `[12, 34, "string", 56, 78]`
- JSON object with multimodal: `{"prompt_string": "string", "multimodal_data": ["base64"]}`
- Multiple prompts: `["string1", "string2"]`

**Response format:**

```json
{
  "content": "<generated text>",
  "tokens": [],
  "stop": true,
  "stop_type": "eos|limit|word|none",
  "stopping_word": "",
  "completion_probabilities": [...],
  "generation_settings": {...},
  "model": "<alias>",
  "prompt": "<processed prompt>",
  "tokens_cached": 0,
  "tokens_evaluated": 0,
  "truncated": false,
  "timings": {...}
}
```

---

### POST `/tokenize` — Tokenize Text

**Options:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `content` | string | required | Text to tokenize |
| `add_special` | bool | false | Insert special tokens (BOS) |
| `parse_special` | bool | true | Parse special tokens |
| `with_pieces` | bool | false | Return token pieces with IDs |

**Response:**

Without pieces: `{"tokens": [123, 456, 789]}`

With pieces: `{"tokens": [{"id": 123, "piece": "Hello"}, ...]}`

---

### POST `/detokenize` — Convert Tokens to Text

**Options:**

| Field | Type | Description |
|-------|------|-------------|
| `tokens` | array | Token IDs to detokenize |

---

### POST `/apply-template` — Apply Chat Template

Formats chat messages to a single prompt string without performing inference.

**Options:**

| Field | Type | Description |
|-------|------|-------------|
| `messages` | array | Chat turns (same format as `/v1/chat/completions`) |

**Response:** `{"prompt": "<formatted string>"}`

---

### POST `/embedding` — Generate Embedding

> ⚠️ Not OAI-compatible. Use `/v1/embeddings` for OAI clients.

Supports multimodal embeddings.

**Options:**

| Field | Type | Description |
|-------|------|-------------|
| `content` | string | Text to embed |
| `embd_normalize` | int | -1=none, 0=max abs, 1=taxicab, 2=euclidean, >2=p-norm |

---

### POST `/reranking` — Rerank Documents

Aliases: `/rerank`, `/v1/rerank`, `/v1/reranking`

Requires reranker model with `--embedding --pooling rank`.

**Options:**

| Field | Type | Description |
|-------|------|-------------|
| `query` | string | Query to rank against |
| `documents` | array | Array of document strings |
| `top_n` | int | Top N results |

**Example:**

```shell
curl http://127.0.0.1:8012/v1/rerank \
    -H "Content-Type: application/json" \
    -d '{
        "model": "some-model",
        "query": "What is panda?",
        "top_n": 3,
        "documents": ["hi", "it is a bear", "The giant panda..."]
    }'
```

---

### POST `/infill` — Code Infilling

Takes prefix and suffix, returns predicted completion.

**Options:**

| Field | Type | Description |
|-------|------|-------------|
| `input_prefix` | string | Code prefix |
| `input_suffix` | string | Code suffix |
| `input_extra` | array | Additional context: `[{"filename": "...", "text": "..."}]` |
| `prompt` | string | Added after FIM_MID token |

Also accepts all `/completion` options.

**Repo-level pattern** (if model has `FIM_REPO` and `FIM_FILE_SEP`):

```
<FIM_REP>myproject
<FIM_SEP>{chunk filename}
{chunk text}
...
<FIM_SEP>filename
<FIM_PRE>[prefix]<FIM_SUF>[suffix]<FIM_MID>[prompt]
```

---

### GET `/props` — Server Properties

Returns server global properties including default generation settings, model path, chat template, modalities, and build info.

Enable POST modification with `--props`.

**Response includes:**
- `default_generation_settings` — default params for `/completion`
- `total_slots` — number of parallel slots
- `model_path` — model file path
- `chat_template` — Jinja2 template
- `chat_template_caps` — template capabilities
- `modalities` — supported modalities
- `is_sleeping` — sleep mode status

---

### GET `/slots` — Slot Monitoring

Returns per-slot processing state, metrics, speed, tokens, sampling parameters.

Disable with `--no-slots`.

Query param `?fail_on_no_slot=1` returns 503 if no slots available.

---

### GET `/metrics` — Prometheus Metrics

Enable with `--metrics`.

**Available metrics:**
- `llamacpp:prompt_tokens_total` — prompt tokens processed
- `llamacpp:tokens_predicted_total` — generation tokens processed
- `llamacpp:prompt_tokens_seconds` — avg prompt throughput (tokens/s)
- `llamacpp:predicted_tokens_seconds` — avg generation throughput (tokens/s)
- `llamacpp:kv_cache_usage_ratio` — KV-cache usage (1 = 100%)
- `llamacpp:kv_cache_tokens` — KV-cache tokens
- `llamacpp:requests_processing` — requests currently processing
- `llamacpp:requests_deferred` — requests deferred
- `llamacpp:n_tokens_max` — high watermark context size

---

### POST `/slots/{id_slot}?action=save` — Save Slot Cache

**Options:** `filename` — file name (saved in `--slot-save-path`)

**Response:** `{"id_slot": 0, "filename": "...", "n_saved": N, "n_written": N, "timings": {...}}`

### POST `/slots/{id_slot}?action=restore` — Restore Slot Cache

**Options:** `filename` — file to restore from `--slot-save-path`

### POST `/slots/{id_slot}?action=erase` — Erase Slot Cache

**Response:** `{"id_slot": 0, "n_erased": N}`

---

### GET `/lora-adapters` — List LoRA Adapters

Returns loaded LoRA adapters with id, path, and scale.

### POST `/lora-adapters` — Set LoRA Adapter Scales

Sets global scale for LoRA adapters. Overridden by per-request `lora` field.

```json
[{"id": 0, "scale": 0.2}, {"id": 1, "scale": 0.8}]
```

---

## OpenAI-Compatible API Endpoints

### GET `/v1/models` — Model Info

Returns loaded model information. Always returns single-element list.

`id` defaults to model file path; override with `--alias`.

### POST `/v1/completions` — OAI Completions

Standard OpenAI Completions API. Supports llama.cpp-specific features like `mirostat`.

**Example (Python):**

```python
import openai

client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="sk-no-key-required"
)

completion = client.completions.create(
    model="davinci-002",
    prompt="I believe the meaning of life is",
    max_tokens=8
)
```

### POST `/v1/chat/completions` — OAI Chat Completions

Supports ChatML-formatted messages, streaming, function calling, tool use, reasoning, and multimodal input.

**Response includes `timings` object:**
- `cache_n` — prompt tokens reused from cache
- `prompt_n` — prompt tokens being processed
- `predicted_n` — predicted tokens

Total context tokens = `prompt_n + cache_n + predicted_n`

**Additional options:**
- `response_format` — JSON output with optional schema
- `chat_template_kwargs` — additional template parameters
- `reasoning_format` — reasoning parsing mode
- `generation_prompt` — prefilled generation prompt
- `parse_tool_calls` — parse tool calls
- `parallel_tool_calls` — enable parallel tool calls

**Example:**

```python
client = openai.OpenAI(base_url="http://localhost:8080/v1", api_key="sk-no-key-required")

completion = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": "You are ChatGPT..."},
        {"role": "user", "content": "Write a limerick about python exceptions"}
    ]
)
```

### POST `/v1/responses` — OAI Responses API

Converts Responses request into Chat Completions request internally.

**Example:**

```python
response = client.responses.create(
    model="gpt-4.1",
    instructions="You are ChatGPT...",
    input="Write a limerick about python exceptions"
)
```

### POST `/v1/embeddings` — OAI Embeddings

Requires model with pooling type other than `none`. Embeddings normalized using Euclidean norm.

Supports string or array of strings as input.

### POST `/v1/messages` — Anthropic-Compatible Messages API

Supports streaming via SSE. Tool use requires `--jinja`.

**Options:**

| Field | Type | Description |
|-------|------|-------------|
| `model` | string | Model identifier (required) |
| `messages` | array | Message objects with role and content (required) |
| `max_tokens` | int | Max tokens to generate (default: 4096) |
| `system` | string/array | System prompt |
| `temperature` | float | Sampling temperature 0-1 (default: 1.0) |
| `top_p` | float | Nucleus sampling (default: 1.0) |
| `top_k` | int | Top-k sampling |
| `stop_sequences` | array | Stop sequences |
| `stream` | bool | Enable streaming (default: false) |
| `tools` | array | Tool definitions |
| `tool_choice` | object | `{"type": "auto"}` / `{"type": "any"}` / `{"type": "tool", "name": "..."}` |

### POST `/v1/messages/count_tokens` — Token Counting

Counts tokens without generating. Same params as `/v1/messages` except `max_tokens`.

**Response:** `{"input_tokens": 10}`

---

## Multiple Models (Router Mode)

Launch without specifying a model to enter router mode:

```sh
llama-server
```

### Model Sources

1. **Cached models** (controlled by `LLAMA_CACHE` env var)
2. **Custom directory** (`--models-dir`)
3. **Custom preset** (`--models-preset`)

Add Hugging Face models to cache:
```sh
llama-server -hf <user>/<model>:<tag>
```

### Directory Structure for `--models-dir`

```
models_directory/
├─ llama-3.2-1b-Q4_K_M.gguf          # single file
├─ Qwen3-8B-Q4_K_M.gguf              # single file
├─ gemma-3-4b-it-Q8_0/               # multimodal
│    ├─ gemma-3-4b-it-Q8_0.gguf
│    └─ mmproj-F16.gguf
├─ Kimi-K2-Thinking-UD-IQ1_S/        # multi-shard
│    ├─ ...-00001-of-00006.gguf
│    └─ ...-00006-of-00006.gguf
```

### Model Presets (INI File)

```ini
version = 1

[*]
c = 8192
n-gpu-layers = 8

[ggml-org/MY-MODEL-GGUF:Q8_0]
chat-template = chatml
n-gpu-layers = 123
jinja = true
c = 4096

[custom_model]
model = /path/to/my-awesome-model-Q4_K_M.gguf
```

**Precedence:** CLI args > model-specific preset > global preset `[*]`

**Preset-exclusive options:**
- `load-on-startup` (bool) — auto-load on server start
- `stop-timeout` (int, seconds) — wait before forcing termination (default: 10)

### Routing

POST endpoints use `"model"` field in JSON body. GET endpoints use `model` query parameter.

Auto-load can be disabled with `--no-models-autoload` or per-request `?autoload=true|false`.

### Router API Endpoints

#### GET `/models` — List Available Models

Returns all models with status (loaded, loading, sleeping, unloaded, failed).

#### POST `/models/load` — Load a Model

```json
{"model": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M"}
```

#### POST `/models/unload` — Unload a Model

```json
{"model": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M"}
```

---

## Error Format

Errors follow the OpenAI format:

```json
{
    "error": {
        "code": 401,
        "message": "Invalid API Key",
        "type": "authentication_error"
    }
}
```

**Custom error types:**
- `not_supported_error` (501) — disabled endpoint
- `invalid_request_error` (400) — invalid grammar

---

## Sleeping on Idle

Enable with `--sleep-idle-seconds N`. Unloads model and KV cache after N seconds of inactivity. Any new task triggers reload.

**Exempt endpoints** (don't trigger reload or reset timer): `/health`, `/props`, `/models`

Check status via `GET /props` (`is_sleeping` field).

---

## Web UI

### Custom Default Preferences

```bash
./llama-server -m model.gguf --webui-config '{"pasteLongTextToFileLen": 0, "renderUserContentAsMarkdown": true}'
```

### Legacy Completion UI

```sh
./llama-server -m my_model.gguf -c 8192 --path ./tools/server/public_legacy
```

### Custom Front End

Serve from `--path ./your-directory` and import `/completion.js` for `llamaComplete()`.

```html
<html>
  <body>
    <pre>
      <script type="module">
        import { llama } from '/completion.js'
        for await (const chunk of llama("Your prompt here")) {
          document.write(chunk.data.content)
        }
      </script>
    </pre>
  </body>
</html>
```