hfd — Hugging Face Downloader

A CLI wrapper around the Hugging Face hf CLI that selectively downloads files from model repositories to local storage. Designed for downloading quantized GGUF files and metadata without pulling entire model repos.

Features

• Selective downloads — metadata (README + JSON) and/or quantized files
• Flexible glob patterns — --include and --exclude for arbitrary filtering
• Repo types — model, dataset, or space repositories
• Revision pinning — branch, tag, or commit hash
• Multiple auth methods — CLI flag, environment variable, or config file
• Dry-run mode — preview what would be downloaded
• Override output path — --output / -o for custom download directories

Setup

1. Install dependencies

pip install -r requirements.txt

This installs huggingface_hub (with CLI) and PyYAML.

2. Configure

cp config.example.yaml config.yaml

Edit config.yaml to set your default download location and optionally your Hugging Face token:

username: lkraven
default_location: /tank/aimodels/llm/
# token: hf_YOUR_TOKEN_HERE

3. Authenticate

The token is resolved in this order (highest priority wins):

Priority	Source	How
1	CLI flag	--token hf_xxx
2	Environment	export HF_TOKEN="hf_xxx"
3	Config file	token: field in config.yaml

Recommended: Use the HF_TOKEN environment variable and keep config.yaml token-free.

Usage

Basic — download metadata only

python hfd.py bartowski/Meta-Llama-3-8B-GGUF

Downloads README.md and all *.json files to /tank/aimodels/llm/bartowski_Meta-Llama-3-8B-GGUF/.

Download a specific quantization

python hfd.py bartowski/Meta-Llama-3-8B-GGUF --quant Q4_K_M

Downloads metadata plus all files matching **/*Q4_K_M.* (searches recursively).

Download everything in a specific folder

Useful for models where quants are organized in folders or split into multiple shards.

python hfd.py username/model --quant-folder UD-Q4_K_XL

Downloads metadata plus every file inside any folder named UD-Q4_K_XL.

Download from a dataset

python hfd.py username/my-dataset --repo-type dataset --include "*.parquet"

Exclude large files

python hfd.py username/big-model --exclude "*.safetensors" "*.bin"

Pin to a specific revision

python hfd.py username/experimental-model --revision v2.1 --quant Q5_K_M

Preview without downloading

python hfd.py bartowski/Meta-Llama-3-8B-GGUF --quant Q4_K_M --dry-run

Custom output directory

python hfd.py username/model -o /tmp/test-download --quant Q4_K_M

Quiet mode (for scripting)

python hfd.py username/model --quant Q4_K_M --quiet

All Options

positional arguments:
  repo_id                Repository ID (e.g. username/repo-name)

options:
  --quant QUANT          Quantization suffix (e.g. Q4_K_M, IQ2_XS)
  --quant-folder FOLDER  Quantization folder name (e.g. UD-Q4_K_XL)
  --repo-type {model,dataset,space}
                         Repository type (default: model)
  --revision REVISION    Git revision: branch, tag, or commit hash
  --include [PATTERNS]   Additional glob patterns to include
  --exclude [PATTERNS]   Glob patterns to exclude
  --token TOKEN          HF API token (overrides config and env)
  --output, -o PATH      Override download directory
  --dry-run              Preview commands without downloading
  --quiet                Suppress progress bars
  --force-download       Re-download existing files
  --max-workers N        Parallel download workers (default: 8)

How It Works

The script makes up to four passes with hf download:

1. Metadata — README.md + *.json (always)
2. Quantized files — **/*<QUANT>.* (if --quant is set)
3. Folder contents — **/<FOLDER>/* (if --quant-folder is set)
4. Custom includes — user-specified --include patterns (if set)

Each pass is a separate hf download call with combined --include patterns. Common flags (--exclude, --quiet, --revision, etc.) are applied to all passes.

File Structure

hfd/
├── hfd.py               # Main script
├── config.example.yaml  # Example configuration (safe to commit)
├── config.yaml          # Your configuration (git-ignored)
├── requirements.txt     # Python dependencies
├── .gitignore           # Prevents config.yaml from being committed
├── hfdocs.context       # `hf` CLI reference output
├── LICENSE              # MIT License
└── README.md            # This file

License

MIT — see LICENSE.