VuongKB/reference/llama-server-settings.md

created, id, keywords, links, modified, summary, tags, title

created	id	keywords	links	modified	summary	tags	title
2026-04-10T01:19:06+00:00	c693060b-7794-4d1c-995d-53b6c951352c	llama-server llama.cpp HTTP server GGUF OpenAI API Anthropic API inference completion chat completions embeddings reranking speculative decoding continuous batching KV cache sampling router mode multimodal LoRA function calling		2026-04-10T01:19:06+00:00	Comprehensive reference for llama-server (LLaMA.cpp HTTP Server) covering all command-line parameters (common, sampling, server-specific), API endpoints (native and OpenAI/Anthropic-compatible), router mode for multi-model serving, build instructions, and usage examples.	reference llama-cpp llama-server inference-server api openai-compatible	LLaMA.cpp HTTP Server (llama-server) — Settings & API Reference

LLaMA.cpp HTTP Server (llama-server)

Fast, lightweight, pure C/C++ HTTP server based on httplib, nlohmann::json and llama.cpp.

Set of LLM REST APIs and a web UI to interact with llama.cpp.

Features:

• LLM inference of F16 and quantized models on GPU and CPU
• OpenAI API compatible chat completions, responses, and embeddings routes
• Anthropic Messages API compatible chat completions
• Reranking endpoint (https://github.com/ggml-org/llama.cpp/pull/9510)
• Parallel decoding with multi-user support
• Continuous batching
• Multimodal (documentation) / with OpenAI-compatible API support
• Monitoring endpoints
• Schema-constrained JSON response format
• Prefilling of assistant messages similar to the Claude API
• Function calling / tool use for ~any model
• Speculative decoding
• Easy-to-use web UI

For the full list of features, please refer to server's changelog

Usage

Common params

Argument	Explanation
-h, --help, --usage	print usage and exit
--version	show version and build info
--license	show source code license and dependencies
-cl, --cache-list	show list of models in cache
--completion-bash	print source-able bash completion script for llama.cpp
-t, --threads N	number of CPU threads to use during generation (default: -1) (env: LLAMA_ARG_THREADS)
-tb, --threads-batch N	number of threads to use during batch and prompt processing (default: same as --threads)
-C, --cpu-mask M	CPU affinity mask: arbitrarily long hex. Complements cpu-range (default: "")
-Cr, --cpu-range lo-hi	range of CPUs for affinity. Complements --cpu-mask
--cpu-strict <0|1>	use strict CPU placement (default: 0)
--prio N	set process/thread priority : low(-1), normal(0), medium(1), high(2), realtime(3) (default: 0)
--poll <0...100>	use polling level to wait for work (0 - no polling, default: 50)
-Cb, --cpu-mask-batch M	CPU affinity mask: arbitrarily long hex. Complements cpu-range-batch (default: same as --cpu-mask)
-Crb, --cpu-range-batch lo-hi	ranges of CPUs for affinity. Complements --cpu-mask-batch
--cpu-strict-batch <0|1>	use strict CPU placement (default: same as --cpu-strict)
--prio-batch N	set process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0)
--poll-batch <0|1>	use polling to wait for work (default: same as --poll)
-c, --ctx-size N	size of the prompt context (default: 0, 0 = loaded from model) (env: LLAMA_ARG_CTX_SIZE)
-n, --predict, --n-predict N	number of tokens to predict (default: -1, -1 = infinity) (env: LLAMA_ARG_N_PREDICT)
-b, --batch-size N	logical maximum batch size (default: 2048) (env: LLAMA_ARG_BATCH)
-ub, --ubatch-size N	physical maximum batch size (default: 512) (env: LLAMA_ARG_UBATCH)
--keep N	number of tokens to keep from the initial prompt (default: 0, -1 = all)
--swa-full	use full-size SWA cache (default: false) (env: LLAMA_ARG_SWA_FULL)
-fa, --flash-attn [on|off|auto]	set Flash Attention use ('on', 'off', or 'auto', default: 'auto') (env: LLAMA_ARG_FLASH_ATTN)
--perf, --no-perf	whether to enable internal libllama performance timings (default: false) (env: LLAMA_ARG_PERF)
-e, --escape, --no-escape	whether to process escapes sequences (\n, \r, \t, ', ", \) (default: true)
--rope-scaling {none,linear,yarn}	RoPE frequency scaling method, defaults to linear unless specified by the model (env: LLAMA_ARG_ROPE_SCALING_TYPE)
--rope-scale N	RoPE context scaling factor, expands context by a factor of N (env: LLAMA_ARG_ROPE_SCALE)
--rope-freq-base N	RoPE base frequency, used by NTK-aware scaling (default: loaded from model) (env: LLAMA_ARG_ROPE_FREQ_BASE)
--rope-freq-scale N	RoPE frequency scaling factor, expands context by a factor of 1/N (env: LLAMA_ARG_ROPE_FREQ_SCALE)
--yarn-orig-ctx N	YaRN: original context size of model (default: 0 = model training context size) (env: LLAMA_ARG_YARN_ORIG_CTX)
--yarn-ext-factor N	YaRN: extrapolation mix factor (default: -1.00, 0.0 = full interpolation) (env: LLAMA_ARG_YARN_EXT_FACTOR)
--yarn-attn-factor N	YaRN: scale sqrt(t) or attention magnitude (default: -1.00) (env: LLAMA_ARG_YARN_ATTN_FACTOR)
--yarn-beta-slow N	YaRN: high correction dim or alpha (default: -1.00) (env: LLAMA_ARG_YARN_BETA_SLOW)
--yarn-beta-fast N	YaRN: low correction dim or beta (default: -1.00) (env: LLAMA_ARG_YARN_BETA_FAST)
-kvo, --kv-offload, -nkvo, --no-kv-offload	whether to enable KV cache offloading (default: enabled) (env: LLAMA_ARG_KV_OFFLOAD)
--repack, -nr, --no-repack	whether to enable weight repacking (default: enabled) (env: LLAMA_ARG_REPACK)
--no-host	bypass host buffer allowing extra buffers to be used (env: LLAMA_ARG_NO_HOST)
-ctk, --cache-type-k TYPE	KV cache data type for K. Allowed: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_K)
-ctv, --cache-type-v TYPE	KV cache data type for V. Allowed: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_V)
-dt, --defrag-thold N	KV cache defragmentation threshold (DEPRECATED) (env: LLAMA_ARG_DEFRAG_THOLD)
--mlock	force system to keep model in RAM rather than swapping or compressing (env: LLAMA_ARG_MLOCK)
--mmap, --no-mmap	whether to memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: enabled) (env: LLAMA_ARG_MMAP)
-dio, --direct-io, -ndio, --no-direct-io	use DirectIO if available (default: disabled) (env: LLAMA_ARG_DIO)
--numa TYPE	attempt optimizations that help on some NUMA systems. Options: distribute, isolate, numactl (env: LLAMA_ARG_NUMA)
-dev, --device <dev1,dev2,..>	comma-separated list of devices to use for offloading (none = don't offload) (env: LLAMA_ARG_DEVICE)
--list-devices	print list of available devices and exit
-ot, --override-tensor <pattern>=<buffer type>,...	override tensor buffer type (env: LLAMA_ARG_OVERRIDE_TENSOR)
-cmoe, --cpu-moe	keep all Mixture of Experts (MoE) weights in the CPU (env: LLAMA_ARG_CPU_MOE)
-ncmoe, --n-cpu-moe N	keep the MoE weights of the first N layers in the CPU (env: LLAMA_ARG_N_CPU_MOE)
-ngl, --gpu-layers, --n-gpu-layers N	max number of layers to store in VRAM: exact number, 'auto', or 'all' (default: auto) (env: LLAMA_ARG_N_GPU_LAYERS)
-sm, --split-mode {none,layer,row}	how to split model across multiple GPUs (default: layer) (env: LLAMA_ARG_SPLIT_MODE)
-ts, --tensor-split N0,N1,N2,...	fraction of model to offload to each GPU, comma-separated proportions (env: LLAMA_ARG_TENSOR_SPLIT)
-mg, --main-gpu INDEX	GPU to use for model or intermediate results (default: 0) (env: LLAMA_ARG_MAIN_GPU)
-fit, --fit [on|off]	whether to adjust unset arguments to fit in device memory (default: 'on') (env: LLAMA_ARG_FIT)
-fitt, --fit-target MiB0,MiB1,...	target margin per device for --fit, comma-separated (default: 1024) (env: LLAMA_ARG_FIT_TARGET)
-fitc, --fit-ctx N	minimum ctx size that can be set by --fit option (default: 4096) (env: LLAMA_ARG_FIT_CTX)
--check-tensors	check model tensor data for invalid values (default: false)
--override-kv KEY=TYPE:VALUE,...	override model metadata by key. Types: int, float, bool, str
--op-offload, --no-op-offload	whether to offload host tensor operations to device (default: true)
--lora FNAME	path to LoRA adapter (comma-separated for multiple)
--lora-scaled FNAME:SCALE,...	path to LoRA adapter with user defined scaling
--control-vector FNAME	add a control vector (comma-separated for multiple)
--control-vector-scaled FNAME:SCALE,...	add a control vector with user defined scaling
--control-vector-layer-range START END	layer range to apply the control vector(s) to
-m, --model FNAME	model path to load (env: LLAMA_ARG_MODEL)
-mu, --model-url MODEL_URL	model download url (env: LLAMA_ARG_MODEL_URL)
-dr, --docker-repo [<repo>/]<model>[:quant]	Docker Hub model repository (env: LLAMA_ARG_DOCKER_REPO)
-hf, -hfr, --hf-repo <user>/<model>[:quant]	Hugging Face model repository (env: LLAMA_ARG_HF_REPO)
-hfd, -hfrd, --hf-repo-draft <user>/<model>[:quant]	Same as --hf-repo, but for the draft model (env: LLAMA_ARG_HFD_REPO)
-hff, --hf-file FILE	Hugging Face model file, overrides quant in --hf-repo (env: LLAMA_ARG_HF_FILE)
-hfv, -hfrv, --hf-repo-v <user>/<model>[:quant]	Hugging Face model repository for vocoder model (env: LLAMA_ARG_HF_REPO_V)
-hffv, --hf-file-v FILE	Hugging Face model file for vocoder model (env: LLAMA_ARG_HF_FILE_V)
-hft, --hf-token TOKEN	Hugging Face access token (env: HF_TOKEN)
--log-disable	Log disable
--log-file FNAME	Log to file (env: LLAMA_LOG_FILE)
--log-colors [on|off|auto]	Set colored logging (default: 'auto') (env: LLAMA_LOG_COLORS)
-v, --verbose, --log-verbose	Set verbosity level to infinity
--offline	Offline mode: forces use of cache, prevents network access (env: LLAMA_OFFLINE)
-lv, --verbosity, --log-verbosity N	Set verbosity threshold. 0=generic, 1=error, 2=warning, 3=info, 4=debug (default: 3) (env: LLAMA_LOG_VERBOSITY)
--log-prefix	Enable prefix in log messages (env: LLAMA_LOG_PREFIX)
--log-timestamps	Enable timestamps in log messages (env: LLAMA_LOG_TIMESTAMPS)
-ctkd, --cache-type-k-draft TYPE	KV cache data type for K for the draft model (default: f16) (env: LLAMA_ARG_CACHE_TYPE_K_DRAFT)
-ctvd, --cache-type-v-draft TYPE	KV cache data type for V for the draft model (default: f16) (env: LLAMA_ARG_CACHE_TYPE_V_DRAFT)

Sampling params

Argument	Explanation
--samplers SAMPLERS	samplers used for generation, separated by ';' (default: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature)
-s, --seed SEED	RNG seed (default: -1, use random seed for -1)
--sampler-seq, --sampling-seq SEQUENCE	simplified sequence for samplers (default: edskypmxt)
--ignore-eos	ignore end of stream token and continue generating
--temp, --temperature N	temperature (default: 0.80)
--top-k N	top-k sampling (default: 40, 0 = disabled) (env: LLAMA_ARG_TOP_K)
--top-p N	top-p sampling (default: 0.95, 1.0 = disabled)
--min-p N	min-p sampling (default: 0.05, 0.0 = disabled)
--top-nsigma, --top-n-sigma N	top-n-sigma sampling (default: -1.00, -1.0 = disabled)
--xtc-probability N	xtc probability (default: 0.00, 0.0 = disabled)
--xtc-threshold N	xtc threshold (default: 0.10, 1.0 = disabled)
--typical, --typical-p N	locally typical sampling, parameter p (default: 1.00, 1.0 = disabled)
--repeat-last-n N	last n tokens to consider for penalize (default: 64, 0 = disabled, -1 = ctx_size)
--repeat-penalty N	penalize repeat sequence of tokens (default: 1.00, 1.0 = disabled)
--presence-penalty N	repeat alpha presence penalty (default: 0.00, 0.0 = disabled)
--frequency-penalty N	repeat alpha frequency penalty (default: 0.00, 0.0 = disabled)
--dry-multiplier N	DRY sampling multiplier (default: 0.00, 0.0 = disabled)
--dry-base N	DRY sampling base value (default: 1.75)
--dry-allowed-length N	allowed length for DRY sampling (default: 2)
--dry-penalty-last-n N	DRY penalty for the last n tokens (default: -1, 0 = disable, -1 = context size)
--dry-sequence-breaker STRING	add sequence breaker for DRY sampling; use "none" for no breakers
--adaptive-target N	adaptive-p: select tokens near this probability (default: -1.00, disabled)
--adaptive-decay N	adaptive-p: decay rate for target adaptation (default: 0.90)
--dynatemp-range N	dynamic temperature range (default: 0.00, 0.0 = disabled)
--dynatemp-exp N	dynamic temperature exponent (default: 1.00)
--mirostat N	use Mirostat sampling (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0)
--mirostat-lr N	Mirostat learning rate, parameter eta (default: 0.10)
--mirostat-ent N	Mirostat target entropy, parameter tau (default: 5.00)
-l, --logit-bias TOKEN_ID(+/-)BIAS	modifies the likelihood of token appearing in the completion
--grammar GRAMMAR	BNF-like grammar to constrain generations
--grammar-file FNAME	file to read grammar from
-j, --json-schema SCHEMA	JSON schema to constrain generations
-jf, --json-schema-file FILE	File containing a JSON schema to constrain generations
-bs, --backend-sampling	enable backend sampling (experimental) (env: LLAMA_ARG_BACKEND_SAMPLING)

Server-specific params

Argument	Explanation
-lcs, --lookup-cache-static FNAME	path to static lookup cache for lookup decoding
-lcd, --lookup-cache-dynamic FNAME	path to dynamic lookup cache for lookup decoding
-ctxcp, --ctx-checkpoints, --swa-checkpoints N	max context checkpoints per slot (default: 32) (env: LLAMA_ARG_CTX_CHECKPOINTS)
-cpent, --checkpoint-every-n-tokens N	checkpoint every n tokens during prefill (default: 8192) (env: LLAMA_ARG_CHECKPOINT_EVERY_NT)
-cram, --cache-ram N	maximum cache size in MiB (default: 8192, -1 = no limit, 0 = disable) (env: LLAMA_ARG_CACHE_RAM)
-kvu, --kv-unified, -no-kvu, --no-kv-unified	single unified KV buffer shared across all sequences (env: LLAMA_ARG_KV_UNIFIED)
--clear-idle, --no-clear-idle	save and clear idle slots on new task (default: enabled) (env: LLAMA_ARG_CLEAR_IDLE)
--context-shift, --no-context-shift	context shift on infinite text generation (default: disabled) (env: LLAMA_ARG_CONTEXT_SHIFT)
-r, --reverse-prompt PROMPT	halt generation at PROMPT
-sp, --special	special tokens output enabled (default: false)
--warmup, --no-warmup	perform warmup with empty run (default: enabled)
--spm-infill	use Suffix/Prefix/Middle pattern for infill (default: disabled)
--pooling {none,mean,cls,last,rank}	pooling type for embeddings (env: LLAMA_ARG_POOLING)
-np, --parallel N	number of server slots (default: -1, -1 = auto) (env: LLAMA_ARG_N_PARALLEL)
-cb, --cont-batching, -nocb, --no-cont-batching	continuous batching / dynamic batching (default: enabled) (env: LLAMA_ARG_CONT_BATCHING)
-mm, --mmproj FILE	path to multimodal projector file (env: LLAMA_ARG_MMPROJ)
-mmu, --mmproj-url URL	URL to multimodal projector file (env: LLAMA_ARG_MMPROJ_URL)
--mmproj-auto, --no-mmproj, --no-mmproj-auto	auto-detect multimodal projector (default: enabled) (env: LLAMA_ARG_MMPROJ_AUTO)
--mmproj-offload, --no-mmproj-offload	GPU offloading for multimodal projector (default: enabled) (env: LLAMA_ARG_MMPROJ_OFFLOAD)
--image-min-tokens N	minimum tokens per image for vision models with dynamic resolution (env: LLAMA_ARG_IMAGE_MIN_TOKENS)
--image-max-tokens N	maximum tokens per image for vision models with dynamic resolution (env: LLAMA_ARG_IMAGE_MAX_TOKENS)
-otd, --override-tensor-draft <pattern>=<buffer type>,...	override tensor buffer type for draft model
-cmoed, --cpu-moe-draft	keep all MoE weights in CPU for draft model (env: LLAMA_ARG_CPU_MOE_DRAFT)
-ncmoed, --n-cpu-moe-draft N	keep MoE weights of first N layers in CPU for draft model (env: LLAMA_ARG_N_CPU_MOE_DRAFT)
-a, --alias STRING	model name aliases, comma-separated (env: LLAMA_ARG_ALIAS)
--tags STRING	model tags, comma-separated (env: LLAMA_ARG_TAGS)
--host HOST	ip address to listen, or UNIX socket if ends with .sock (default: 127.0.0.1) (env: LLAMA_ARG_HOST)
--port PORT	port to listen (default: 8080) (env: LLAMA_ARG_PORT)
--reuse-port	allow multiple sockets to bind to same port (env: LLAMA_ARG_REUSE_PORT)
--path PATH	path to serve static files from (env: LLAMA_ARG_STATIC_PATH)
--api-prefix PREFIX	prefix path the server serves from (env: LLAMA_ARG_API_PREFIX)
--webui-config JSON	JSON for default WebUI settings (env: LLAMA_ARG_WEBUI_CONFIG)
--webui-config-file PATH	JSON file for default WebUI settings (env: LLAMA_ARG_WEBUI_CONFIG_FILE)
--webui-mcp-proxy, --no-webui-mcp-proxy	enable MCP CORS proxy - experimental, do not enable in untrusted environments (env: LLAMA_ARG_WEBUI_MCP_PROXY)
--tools TOOL1,TOOL2,...	enable built-in tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, apply_diff (env: LLAMA_ARG_TOOLS)
--webui, --no-webui	enable the Web UI (default: enabled) (env: LLAMA_ARG_WEBUI)
--embedding, --embeddings	restrict to embedding use case only (env: LLAMA_ARG_EMBEDDINGS)
--rerank, --reranking	enable reranking endpoint (env: LLAMA_ARG_RERANKING)
--api-key KEY	API key(s) for authentication, comma-separated (env: LLAMA_API_KEY)
--api-key-file FNAME	path to file containing API keys
--ssl-key-file FNAME	PEM-encoded SSL private key (env: LLAMA_ARG_SSL_KEY_FILE)
--ssl-cert-file FNAME	PEM-encoded SSL certificate (env: LLAMA_ARG_SSL_CERT_FILE)
--chat-template-kwargs STRING	additional params for json template parser (env: LLAMA_CHAT_TEMPLATE_KWARGS)
-to, --timeout N	server read/write timeout in seconds (default: 600) (env: LLAMA_ARG_TIMEOUT)
--threads-http N	threads for HTTP request processing (default: -1) (env: LLAMA_ARG_THREADS_HTTP)
--cache-prompt, --no-cache-prompt	prompt caching (default: enabled) (env: LLAMA_ARG_CACHE_PROMPT)
--cache-reuse N	min chunk size for KV shifting cache reuse (default: 0) (env: LLAMA_ARG_CACHE_REUSE)
--metrics	enable prometheus compatible metrics endpoint (env: LLAMA_ARG_ENDPOINT_METRICS)
--props	enable changing global properties via POST /props (env: LLAMA_ARG_ENDPOINT_PROPS)
--slots, --no-slots	expose slots monitoring endpoint (default: enabled) (env: LLAMA_ARG_ENDPOINT_SLOTS)
--slot-save-path PATH	path to save slot kv cache
--media-path PATH	directory for loading local media files
--models-dir PATH	directory containing models for router server (env: LLAMA_ARG_MODELS_DIR)
--models-preset PATH	path to INI file containing model presets (env: LLAMA_ARG_MODELS_PRESET)
--models-max N	max models to load simultaneously in router (default: 4, 0 = unlimited) (env: LLAMA_ARG_MODELS_MAX)
--models-autoload, --no-models-autoload	auto-load models in router mode (default: enabled) (env: LLAMA_ARG_MODELS_AUTOLOAD)
--jinja, --no-jinja	use jinja template engine for chat (default: enabled) (env: LLAMA_ARG_JINJA)
--reasoning-format FORMAT	thought tag handling: none, deepseek, deepseek-legacy (default: auto) (env: LLAMA_ARG_THINK)
-rea, --reasoning [on|off|auto]	reasoning/thinking in chat (default: auto) (env: LLAMA_ARG_REASONING)
--reasoning-budget N	token budget for thinking: -1 unrestricted, 0 immediate end, N>0 budget (default: -1) (env: LLAMA_ARG_THINK_BUDGET)
--reasoning-budget-message MESSAGE	message injected when reasoning budget exhausted (env: LLAMA_ARG_THINK_BUDGET_MESSAGE)
--chat-template JINJA_TEMPLATE	custom jinja chat template or built-in name (env: LLAMA_ARG_CHAT_TEMPLATE)
--chat-template-file JINJA_TEMPLATE_FILE	custom jinja chat template file (env: LLAMA_ARG_CHAT_TEMPLATE_FILE)
--skip-chat-parsing, --no-skip-chat-parsing	force pure content parser (env: LLAMA_ARG_SKIP_CHAT_PARSING)
--prefill-assistant, --no-prefill-assistant	prefill assistant response (default: enabled) (env: LLAMA_ARG_PREFILL_ASSISTANT)
-sps, --slot-prompt-similarity SIMILARITY	prompt similarity threshold for slot reuse (default: 0.10)
--lora-init-without-apply	load LoRA adapters without applying
--sleep-idle-seconds SECONDS	seconds of idleness before server sleeps (default: -1, disabled)
-td, --threads-draft N	threads for draft model generation
-tbd, --threads-batch-draft N	threads for draft model batch processing
--draft, --draft-n, --draft-max N	tokens to draft for speculative decoding (default: 16) (env: LLAMA_ARG_DRAFT_MAX)
--draft-min, --draft-n-min N	minimum draft tokens for speculative decoding (default: 0) (env: LLAMA_ARG_DRAFT_MIN)
--draft-p-min P	minimum speculative decoding probability (default: 0.75) (env: LLAMA_ARG_DRAFT_P_MIN)
-cd, --ctx-size-draft N	prompt context size for draft model (env: LLAMA_ARG_CTX_SIZE_DRAFT)
-devd, --device-draft <dev1,dev2,..>	devices for draft model offloading
-ngld, --gpu-layers-draft N	draft model GPU layers (default: auto) (env: LLAMA_ARG_N_GPU_LAYERS_DRAFT)
-md, --model-draft FNAME	draft model for speculative decoding (env: LLAMA_ARG_MODEL_DRAFT)
--spec-replace TARGET DRAFT	translate strings between main and draft model
--spec-type [none|ngram-cache|ngram-simple|ngram-map-k|ngram-map-k4v|ngram-mod]	speculative decoding type without draft model (env: LLAMA_ARG_SPEC_TYPE)
--spec-ngram-size-n N	ngram size N for ngram speculative decoding (default: 12)
--spec-ngram-size-m N	ngram size M for ngram speculative decoding (default: 48)
--spec-ngram-min-hits N	minimum hits for ngram-map speculative decoding (default: 1)
-mv, --model-vocoder FNAME	vocoder model for audio generation
--tts-use-guide-tokens	Use guide tokens to improve TTS word recall
--embd-gemma-default	use default EmbeddingGemma model
--fim-qwen-1.5b-default	use default Qwen 2.5 Coder 1.5B
--fim-qwen-3b-default	use default Qwen 2.5 Coder 3B
--fim-qwen-7b-default	use default Qwen 2.5 Coder 7B
--fim-qwen-7b-spec	use Qwen 2.5 Coder 7B + 0.5B draft
--fim-qwen-14b-spec	use Qwen 2.5 Coder 14B + 0.5B draft
--fim-qwen-30b-default	use default Qwen 3 Coder 30B A3B Instruct
--gpt-oss-20b-default	use gpt-oss-20b
--gpt-oss-120b-default	use gpt-oss-120b
--vision-gemma-4b-default	use Gemma 3 4B QAT
--vision-gemma-12b-default	use Gemma 3 12B QAT

Environment Variables

If both command line argument and environment variable are set for the same param, the argument takes precedence.

For boolean options like --mmap or --kv-offload, environment variable handling:

• LLAMA_ARG_MMAP=true / 1 / on / enabled → enabled
• LLAMA_ARG_MMAP=false / 0 / off / disabled → disabled
• Presence of LLAMA_ARG_NO_MMAP → disabled

Docker Compose Example

services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 4096
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_ENDPOINT_METRICS: 1
      LLAMA_ARG_PORT: 8080

Multimodal Support

Multimodal support is currently an experimental feature. Available in:

• OAI-compatible chat endpoint
• Non-OAI-compatible completions endpoint
• Non-OAI-compatible embeddings endpoint

The multimodal projector is auto-detected when using -hf. Use --no-mmproj to disable.

Built-in Tools

Built-in tools enable the LLM to access the local file system from the Web UI. Start with --tools all or specify individual tools: --tools read_file,file_glob_search,....

Available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, apply_diff

⚠️ Do not enable in untrusted environments.

Build

Standard Build

cmake -B build
cmake --build build --config Release -t llama-server

Binary: ./build/bin/llama-server

Build with SSL (OpenSSL 3)

cmake -B build -DLLAMA_OPENSSL=ON
cmake --build build --config Release -t llama-server

Quick Start

Unix (Linux, macOS)

./llama-server -m models/7B/ggml-model.gguf -c 2048

Windows

llama-server.exe -m models\7B\ggml-model.gguf -c 2048

Listens on 127.0.0.1:8080 by default.

Docker

docker run -p 8080:8080 -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:server -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080

# With CUDA:
docker run -p 8080:8080 -v /path/to/models:/models --gpus all ghcr.io/ggml-org/llama.cpp:server-cuda -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080 --n-gpu-layers 99

Using with cURL

curl --request POST \
    --url http://localhost:8080/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'

---

API Endpoints

GET /health — Health Check

Public endpoint (no API key). /v1/health also works.

• 503: {"error": {"code": 503, "message": "Loading model", "type": "unavailable_error"}}
• 200: {"status": "ok"}

---

POST /completion — Text Completion

⚠️ Not OAI-compatible. Use /v1/completions for OAI clients.

Request options:

Field	Type	Default	Description
prompt	string/array/object	required	Prompt as string, token array, mixed, or JSON with multimodal data
temperature	float	0.8	Randomness of generated text
dynatemp_range	float	0.0	Dynamic temperature range
dynatemp_exponent	float	1.0	Dynamic temperature exponent
top_k	int	40	Limit to K most probable tokens
top_p	float	0.95	Nucleus sampling threshold
min_p	float	0.05	Minimum probability relative to most likely token
n_predict	int	-1	Max tokens to predict (-1 = infinity)
n_indent	int	0	Min line indentation (for code completion)
n_keep	int	0	Tokens to retain when context exceeded (-1 = all)
n_cmpl	int	-	Number of completions per prompt
n_cache_reuse	int	0	Min chunk size for KV shifting cache reuse
stream	bool	false	Stream tokens in real-time via SSE
stop	array	[]	Stopping strings (not included in output)
typical_p	float	1.0	Locally typical sampling parameter
repeat_penalty	float	1.1	Repetition penalty
repeat_last_n	int	64	Tokens to consider for repeat penalty
presence_penalty	float	0.0	Presence penalty
frequency_penalty	float	0.0	Frequency penalty
dry_multiplier	float	0.0	DRY repetition penalty multiplier
dry_base	float	1.75	DRY penalty base
dry_allowed_length	int	2	DRY allowed repeating length
dry_penalty_last_n	int	-1	Tokens to scan for DRY
dry_sequence_breakers	array	['\n',':','"','*']	DRY sequence breakers
xtc_probability	float	0.0	XTC token removal chance
xtc_threshold	float	0.1	XTC minimum probability threshold
mirostat	int	0	Mirostat mode (0=off, 1, 2)
mirostat_tau	float	5.0	Mirostat target entropy
mirostat_eta	float	0.1	Mirostat learning rate
grammar	string	-	BNF grammar for constrained generation
json_schema	object	-	JSON schema for constrained generation
seed	int	-1	RNG seed (-1 = random)
ignore_eos	bool	false	Ignore EOS token
logit_bias	array/object	[]	Modify token likelihoods
n_probs	int	0	Return top N token probabilities
min_keep	int	0	Force minimum tokens from samplers
t_max_predict_ms	int	0	Time limit for prediction phase (ms)
id_slot	int	-1	Assign to specific slot
cache_prompt	bool	true	Re-use KV cache from previous request
return_tokens	bool	false	Return raw token IDs
samplers	array	[dry,top_k,typ_p,top_p,min_p,xtc,temperature]	Sampler order
timings_per_token	bool	false	Include speed info per token
return_progress	bool	false	Include prompt processing progress in stream mode
post_sampling_probs	bool	false	Return probabilities after sampling
response_fields	array	-	List of specific response fields to return
lora	array	-	Per-request LoRA adapters: [{"id": 0, "scale": 0.5}]

Prompt input shapes:

• Single string: "string"
• Single token sequence: [12, 34, 56]
• Mixed tokens and strings: [12, 34, "string", 56, 78]
• JSON object with multimodal: {"prompt_string": "string", "multimodal_data": ["base64"]}
• Multiple prompts: ["string1", "string2"]

Response format:

{
  "content": "<generated text>",
  "tokens": [],
  "stop": true,
  "stop_type": "eos|limit|word|none",
  "stopping_word": "",
  "completion_probabilities": [...],
  "generation_settings": {...},
  "model": "<alias>",
  "prompt": "<processed prompt>",
  "tokens_cached": 0,
  "tokens_evaluated": 0,
  "truncated": false,
  "timings": {...}
}

---

POST /tokenize — Tokenize Text

Options:

Field	Type	Default	Description
content	string	required	Text to tokenize
add_special	bool	false	Insert special tokens (BOS)
parse_special	bool	true	Parse special tokens
with_pieces	bool	false	Return token pieces with IDs

Response:

Without pieces: {"tokens": [123, 456, 789]}

With pieces: {"tokens": [{"id": 123, "piece": "Hello"}, ...]}

---

POST /detokenize — Convert Tokens to Text

Options:

Field	Type	Description
tokens	array	Token IDs to detokenize

---

POST /apply-template — Apply Chat Template

Formats chat messages to a single prompt string without performing inference.

Options:

Field	Type	Description
messages	array	Chat turns (same format as /v1/chat/completions)

Response: {"prompt": "<formatted string>"}

---

POST /embedding — Generate Embedding

⚠️ Not OAI-compatible. Use /v1/embeddings for OAI clients.

Supports multimodal embeddings.

Options:

Field	Type	Description
content	string	Text to embed
embd_normalize	int	-1=none, 0=max abs, 1=taxicab, 2=euclidean, >2=p-norm

---

POST /reranking — Rerank Documents

Aliases: /rerank, /v1/rerank, /v1/reranking

Requires reranker model with --embedding --pooling rank.

Options:

Field	Type	Description
query	string	Query to rank against
documents	array	Array of document strings
top_n	int	Top N results

Example:

curl http://127.0.0.1:8012/v1/rerank \
    -H "Content-Type: application/json" \
    -d '{
        "model": "some-model",
        "query": "What is panda?",
        "top_n": 3,
        "documents": ["hi", "it is a bear", "The giant panda..."]
    }'

---

POST /infill — Code Infilling

Takes prefix and suffix, returns predicted completion.

Options:

Field	Type	Description
input_prefix	string	Code prefix
input_suffix	string	Code suffix
input_extra	array	Additional context: [{"filename": "...", "text": "..."}]
prompt	string	Added after FIM_MID token

Also accepts all /completion options.

Repo-level pattern (if model has FIM_REPO and FIM_FILE_SEP):

<FIM_REP>myproject
<FIM_SEP>{chunk filename}
{chunk text}
...
<FIM_SEP>filename
<FIM_PRE>[prefix]<FIM_SUF>[suffix]<FIM_MID>[prompt]

---

GET /props — Server Properties

Returns server global properties including default generation settings, model path, chat template, modalities, and build info.

Enable POST modification with --props.

Response includes:

• default_generation_settings — default params for /completion
• total_slots — number of parallel slots
• model_path — model file path
• chat_template — Jinja2 template
• chat_template_caps — template capabilities
• modalities — supported modalities
• is_sleeping — sleep mode status

---

GET /slots — Slot Monitoring

Returns per-slot processing state, metrics, speed, tokens, sampling parameters.

Disable with --no-slots.

Query param ?fail_on_no_slot=1 returns 503 if no slots available.

---

GET /metrics — Prometheus Metrics

Enable with --metrics.

Available metrics:

• llamacpp:prompt_tokens_total — prompt tokens processed
• llamacpp:tokens_predicted_total — generation tokens processed
• llamacpp:prompt_tokens_seconds — avg prompt throughput (tokens/s)
• llamacpp:predicted_tokens_seconds — avg generation throughput (tokens/s)
• llamacpp:kv_cache_usage_ratio — KV-cache usage (1 = 100%)
• llamacpp:kv_cache_tokens — KV-cache tokens
• llamacpp:requests_processing — requests currently processing
• llamacpp:requests_deferred — requests deferred
• llamacpp:n_tokens_max — high watermark context size

---

POST /slots/{id_slot}?action=save — Save Slot Cache

Options: filename — file name (saved in --slot-save-path)

Response: {"id_slot": 0, "filename": "...", "n_saved": N, "n_written": N, "timings": {...}}

POST /slots/{id_slot}?action=restore — Restore Slot Cache

Options: filename — file to restore from --slot-save-path

POST /slots/{id_slot}?action=erase — Erase Slot Cache

Response: {"id_slot": 0, "n_erased": N}

---

GET /lora-adapters — List LoRA Adapters

Returns loaded LoRA adapters with id, path, and scale.

POST /lora-adapters — Set LoRA Adapter Scales

Sets global scale for LoRA adapters. Overridden by per-request lora field.

[{"id": 0, "scale": 0.2}, {"id": 1, "scale": 0.8}]

---

OpenAI-Compatible API Endpoints

GET /v1/models — Model Info

Returns loaded model information. Always returns single-element list.

id defaults to model file path; override with --alias.

POST /v1/completions — OAI Completions

Standard OpenAI Completions API. Supports llama.cpp-specific features like mirostat.

Example (Python):

import openai

client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="sk-no-key-required"
)

completion = client.completions.create(
    model="davinci-002",
    prompt="I believe the meaning of life is",
    max_tokens=8
)

POST /v1/chat/completions — OAI Chat Completions

Supports ChatML-formatted messages, streaming, function calling, tool use, reasoning, and multimodal input.

Response includes timings object:

• cache_n — prompt tokens reused from cache
• prompt_n — prompt tokens being processed
• predicted_n — predicted tokens

Total context tokens = prompt_n + cache_n + predicted_n

Additional options:

• response_format — JSON output with optional schema
• chat_template_kwargs — additional template parameters
• reasoning_format — reasoning parsing mode
• generation_prompt — prefilled generation prompt
• parse_tool_calls — parse tool calls
• parallel_tool_calls — enable parallel tool calls

Example:

client = openai.OpenAI(base_url="http://localhost:8080/v1", api_key="sk-no-key-required")

completion = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": "You are ChatGPT..."},
        {"role": "user", "content": "Write a limerick about python exceptions"}
    ]
)

POST /v1/responses — OAI Responses API

Converts Responses request into Chat Completions request internally.

Example:

response = client.responses.create(
    model="gpt-4.1",
    instructions="You are ChatGPT...",
    input="Write a limerick about python exceptions"
)

POST /v1/embeddings — OAI Embeddings

Requires model with pooling type other than none. Embeddings normalized using Euclidean norm.

Supports string or array of strings as input.

POST /v1/messages — Anthropic-Compatible Messages API

Supports streaming via SSE. Tool use requires --jinja.

Options:

Field	Type	Description
model	string	Model identifier (required)
messages	array	Message objects with role and content (required)
max_tokens	int	Max tokens to generate (default: 4096)
system	string/array	System prompt
temperature	float	Sampling temperature 0-1 (default: 1.0)
top_p	float	Nucleus sampling (default: 1.0)
top_k	int	Top-k sampling
stop_sequences	array	Stop sequences
stream	bool	Enable streaming (default: false)
tools	array	Tool definitions
tool_choice	object	{"type": "auto"} / {"type": "any"} / {"type": "tool", "name": "..."}

POST /v1/messages/count_tokens — Token Counting

Counts tokens without generating. Same params as /v1/messages except max_tokens.

Response: {"input_tokens": 10}

---

Multiple Models (Router Mode)

Launch without specifying a model to enter router mode:

llama-server

Model Sources

1. Cached models (controlled by LLAMA_CACHE env var)
2. Custom directory (--models-dir)
3. Custom preset (--models-preset)

Add Hugging Face models to cache:

llama-server -hf <user>/<model>:<tag>

Directory Structure for --models-dir

models_directory/
├─ llama-3.2-1b-Q4_K_M.gguf          # single file
├─ Qwen3-8B-Q4_K_M.gguf              # single file
├─ gemma-3-4b-it-Q8_0/               # multimodal
│    ├─ gemma-3-4b-it-Q8_0.gguf
│    └─ mmproj-F16.gguf
├─ Kimi-K2-Thinking-UD-IQ1_S/        # multi-shard
│    ├─ ...-00001-of-00006.gguf
│    └─ ...-00006-of-00006.gguf

Model Presets (INI File)

version = 1

[*]
c = 8192
n-gpu-layers = 8

[ggml-org/MY-MODEL-GGUF:Q8_0]
chat-template = chatml
n-gpu-layers = 123
jinja = true
c = 4096

[custom_model]
model = /path/to/my-awesome-model-Q4_K_M.gguf

Precedence: CLI args > model-specific preset > global preset [*]

Preset-exclusive options:

• load-on-startup (bool) — auto-load on server start
• stop-timeout (int, seconds) — wait before forcing termination (default: 10)

Routing

POST endpoints use "model" field in JSON body. GET endpoints use model query parameter.

Auto-load can be disabled with --no-models-autoload or per-request ?autoload=true|false.

Router API Endpoints

GET /models — List Available Models

Returns all models with status (loaded, loading, sleeping, unloaded, failed).

POST /models/load — Load a Model

{"model": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M"}

POST /models/unload — Unload a Model

{"model": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M"}

---

Error Format

Errors follow the OpenAI format:

{
    "error": {
        "code": 401,
        "message": "Invalid API Key",
        "type": "authentication_error"
    }
}

Custom error types:

• not_supported_error (501) — disabled endpoint
• invalid_request_error (400) — invalid grammar

---

Sleeping on Idle

Enable with --sleep-idle-seconds N. Unloads model and KV cache after N seconds of inactivity. Any new task triggers reload.

Exempt endpoints (don't trigger reload or reset timer): /health, /props, /models

Check status via GET /props (is_sleeping field).

---

Web UI

Custom Default Preferences

./llama-server -m model.gguf --webui-config '{"pasteLongTextToFileLen": 0, "renderUserContentAsMarkdown": true}'

Legacy Completion UI

./llama-server -m my_model.gguf -c 8192 --path ./tools/server/public_legacy

Custom Front End

Serve from --path ./your-directory and import /completion.js for llamaComplete().

<html>
  <body>
    <pre>
      <script type="module">
        import { llama } from '/completion.js'
        for await (const chunk of llama("Your prompt here")) {
          document.write(chunk.data.content)
        }
      </script>
    </pre>
  </body>
</html>