vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(catalog): archive decommissioned + arbo refreshed (502 G post-migration)

Browse Source 
storetank archive fully resolved (919 G -> 0): ~739 G killed (superseded/niche),
177 G migrated into arbo, rest dupes. Rewrite the archive doc as a decommission
record; refresh the arbo catalog to its post-migration 502 G state (+ SDXL/Pony
stack + 9 gen-agnostic utility categories).
This commit is contained in:
vh
2026-06-13 15:40:56 -07:00
parent 308ca6f5d2
commit 5007ec1236
2 changed files with 70 additions and 121 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/arbo-comfyui-model-catalog.md
+35 -50
View File
@@ -1,60 +1,45 @@
# Arbo ComfyUI model catalog (worktank set)
# Arbo ComfyUI model catalog
Snapshot **2026-06-13** of the ComfyUI model tree that arbo (hero/asset
generation) consumes, taken at migration time. **~325 G**, migrated from
`/worktank/comfyui/basedir/models` (irv-ml1 NVMe, was 97% full) →
**`/storetank/arbo/models`** (irv-ml1 SATA-SSD) and overlay-mounted back into
the ComfyUI container at `/basedir/models`.
**Host:** irv-ml1 · **Path:** `/storetank/arbo/models` (SATA SSD; overlay-mounted into
the arbo / comfyui container at `/basedir/models`). **502 G** as of 2026-06-13.
Purpose: decide what to **retain for future arbo use** vs prune. This is the
*live* set ComfyUI serves today — distinct from the older 919 G CivitAI-managed
archive at `/storetank/image-models/comfy` (untouched, separate question).
The single live model tree arbo (hero / asset generation) consumes. On 2026-06-13 it
**absorbed 177 G** of gen-agnostic utilities + the SDXL/Pony stack, migrated from the
now-decommissioned `/storetank/image-models/comfy` archive — see
[`storetank-image-models-archive.md`](storetank-image-models-archive.md) for that record.
## Per-category sizes
| Category | Size | Notes |
| Category | Size | Contents |
|---|---|---|
| `diffusion_models/` | **203 G** | the bulk — flux2 / wan2.2 / qwen-image / z-image / ideogram (GGUF + fp8) |
| `text_encoders/` | **64 G** | qwen3-VL, qwen2.5-VL, gemma4, umt5/t5-xxl, qwen3 (encoders for the above) |
| `checkpoints/` | **31 G** | SDXL/Pony bases + SUPIR upscalers |
| `loras/` | **14 G** | style/character LoRAs |
| `controlnet/` | 5.7 G | flux upscaler + sdxl union-promax |
| `clip_vision/` | 3.6 G | CLIP-ViT-H, clip_vision_h |
| `vae/` | 3.0 G | wan2.2, flux2, flux1, z-image VAEs |
| `ipadapter/` | 1.5 G | ip-adapter sdxl vit-h |
| `RMBG/` `sams/` `upscale_models/` `ultralytics/` | <1.5 G ea | bg-removal, SAM, upscalers, yolo |
| (empty placeholders) | — | unet, embeddings, hypernetworks, gligen, … (`put_*_here` only) |
| `diffusion_models/` | **203 G** | current-gen generators: flux2-klein / wan2.2 / qwen-image / z-image / ideogram (GGUF + fp8) |
| `checkpoints/` | **147 G** | SDXL / Pony / Illustrious bases — cyberrealisticPony_v180Coreshift (12.9 G), ponyRealism V22, novaAnimeXL, juggernaut/dreamshaper Lightning, lustify, hassaku, waiNSFWIllustrious, realDream + SUPIR upscalers |
| `text_encoders/` | **80 G** | qwen3-VL, qwen2.5-VL, gemma, umt5, t5-xxl, clip variants |
| `loras/` | **16 G** | flux2/wan2.2 (gameart, RetroAnimeFlux, flux1_turbo, zit_*) + migrated SDXL/Pony (dmd2_sdxl_4step, ACE++, character-design) |
| `vae/` | 9.6 G | wan2.2 / flux2 / flux1 / z-image / sdxl / wan2.1 VAEs |
| `Aura-SR/` | 9.3 G | AuraSR v1/v2 upscalers |
| `LLM/` + `florence2/` | 8.6 + 3.6 G | Florence-2 PromptGen large/base + CogFlorence captioners |
| `controlnet/` | 8.1 G | flux upscaler + sdxl union-promax |
| `clip_vision/` | 4.4 G | CLIP-ViT-H, clip_vision_h, sigclip |
| `upscale_models/` | 3.8 G | HAT / DAT / RealESRGAN / UltraSharp / Remacri / NMKD / Omni-SR (~50) |
| `grounding-dino/` | 1.6 G | grounding-dino swinb / swint |
| `ipadapter/` | 1.5 G | ip-adapter-plus / _sdxl vit-h |
| `insightface/` | 1.3 G | inswapper_128 + antelopev2 |
| `depthanything/` | 1.3 G | depth-anything v2 (vitl / vits) |
| `facerestore_models/` | 937 M | GFPGAN v1.3/1.4, GPEN-BFR |
| `RMBG/` `clip/` `sams/` `nsfw_detector/` `vitmatte/` `facexlib/` `ultralytics/` … | <1 G ea | bg-removal, EVA02-CLIP-L, SAM-HQ + SAM, nsfw classifier, matte, face-lib, yolo (face/hand/eyes/person) |
## Notable models (size · file)
## Migrated in 2026-06-13 (177 G from the storetank archive)
**diffusion_models/ (203 G):**
- 15.7 G · Qwen-Image-Edit-2509-Q6_K.gguf
- 14.3 G ×4 · wan2.2 i2v/t2v a14b low+high-noise Q8_0
- 13.6 G · Wan2.2-Animate-14B-Q6_K · 13.5 G · Wan2.1_14B_VACE-Q6_K
- 11.8 G · flux1-dev-Q8_0
- 9.3 G ×2 · flux-2-klein 9b + base-9b Q8_0 · 8.9/8.8 G · same in fp8
- 8.6 G ×2 · ideogram4 (+ unconditional) fp8
- 6.7 G · z_image_turbo-Q8_0 · 5.0 G ×2 · Wan2.2-Fun-5B-Control, wan2.2-ti2v-5b
- 4.0 G ×2 · flux-2-klein 4b + base-4b Q8_0 · 3.9 G · anima-base-v1.0 · 3.8 G ×2 · klein-4b fp8
The gen-agnostic utility set (upscalers, Florence-2 captioners, controlnet-union,
grounding-dino / SAM / yolo / depthanything / vitmatte, insightface / facerestore,
ip-adapter, CLIP-vision) **plus** the SDXL/Pony stack (bases + dmd2 / ACE++ /
character-design loras). These work alongside arbo's current FLUX.2 / WAN2.2 / qwen
generators; **comfy-dev** authors the per-model catalog entries + graphs + heroes that
turn them into usable workflows.
**text_encoders/ (64 G):** qwen3vl_8b_fp8 (9.9 G), qwen2.5_vl_7b_fp8 (8.7 G),
gemma4_e4b_it_fp8 (8.4 G), flux2-klein-9b-uncensored-q8 (8.1 G), qwen_3_8b_fp8mixed
(8.1 G), umt5_xxl_fp8 (6.3 G), t5xxl_fp8 (4.6 G), qwen3-4b-zimage-abliterated-q8
(4.0 G), flux2-klein-4b-uncensored-q8 (4.0 G), qwen_3_06b_base (1.1 G).
## Durability
**checkpoints/ (31 G):** cyberrealisticPony_v180Coreshift (12.9 G),
novaAnimeXL_illustrious (6.5 G), lustifySDXLNSFW_apexV8 (6.5 G),
SUPIR-v0Q/v0F_fp16 (2.5 G ea).
**loras/ (14 G):** gameart-game-icon sdxl (2.6 G) + diablo (0.9 G), RetroAnimeFluxV1
(1.0 G), flux1_turbo_alpha, zit_* (nsfw/cinematic/ultrareal), LH_Pixar3D, oil-painting,
hentai_hottie2_klein9b, … (~30 files).
## Retain guidance (for operator)
- **Likely keep (active gen stack):** the flux2-klein family + their text_encoders,
wan2.2 video set, qwen-image-edit, z-image, the SDXL/Pony checkpoints + their LoRAs,
VAEs, controlnet, clip_vision, ipadapter. This is a coherent current toolkit.
- **Prune candidates:** redundant precisions (e.g., both Q8 *and* fp8 of the same
klein model where one suffices), superseded older variants, one-off LoRAs.
- The 919 G `/storetank/image-models/comfy` archive (CivitAI-managed, older) is a
separate, larger reclaim target — abandoned (nothing mounts it). Operator's call.
- `arbo_db` (gallery/history SQLite) — backed up (restic/Backrest), local disk not NFS.
- The model tree itself is **bulk, reproducible-from-source** → not backed up; this
catalog + the migration record are the recovery map.
docs/storetank-image-models-archive.md
+35 -71
View File
@@ -1,88 +1,52 @@
# Storetank image-models archive — curation record + remaining catalog
# Storetank image-models archive — curation / migration / decommission record
**Host:** irv-ml1 · **Path:** `/storetank/image-models/comfy/models`
(this tree is also the native `/opt/ComfyUI/models` symlink target. The **docker /
arbo** ComfyUI uses a *separate, active* tree at `/storetank/arbo/models` —
see [`arbo-comfyui-model-catalog.md`](arbo-comfyui-model-catalog.md).)
**Host:** irv-ml1 · **Former path:** `/storetank/image-models/comfy/models`
(was the native `/opt/ComfyUI/models` symlink target).
**Status: DECOMMISSIONED 2026-06-13** — emptied of all models (919 G → 0). The
active model tree is arbo's `/storetank/arbo/models` — see
[`arbo-comfyui-model-catalog.md`](arbo-comfyui-model-catalog.md).
**Snapshot:** 2026-06-13. This is the older CivitAI-managed model pile, curated
down from **919 G → 238 G** in a kill-the-superseded pass. Two purposes: (1) record
what was removed and why, and (2) catalog what remains, so **comfy-dev** can pick
migration candidates into the active arbo set.
Historical record of how the 919 G CivitAI-managed pile was resolved on 2026-06-13:
**~739 G killed** (superseded / niche), **177 G migrated** into the arbo set, the
remainder dupes arbo already had.
## Curation pass (2026-06-13) — 681 G reclaimed
## 1. Killed — superseded by arbo's current-gen stack (~739 G)
Killed everything superseded by arbo's current-gen stack (FLUX.2-klein, WAN2.2,
qwen-image, z-image). Two principles: generation-locked LoRAs have no value without
their (also-superseded) base models, and arbo already carries its own copies of the
shared encoders/VAEs.
Two principles: generation-locked LoRAs have no value without their (also-superseded)
base models, and arbo already carries its own copies of the shared encoders/VAEs.
| Killed | Size | Why |
|---|---|---|
| **Hunyuan video** (diffusion_models + unet + vae + loras) | 74 G | older video arch; not in arbo, niche |
| **Hunyuan video** (diffusion_models + unet + vae + loras) | 74 G | older video arch; not in arbo |
| **WAN2.1 bases + loras** | 118 G | superseded by arbo's WAN2.2; loras gen-locked |
| **WAN2.1 encoders / VAE** (umt5, xlm-roberta, clip_vision_h, wan VAE) | 44 G | exact dupes of arbo's own copies |
| **WAN2.1 encoders / VAE** (umt5, xlm-roberta, clip_vision_h, wan VAE) | 44 G | dupes of arbo's own copies |
| **FLUX.1 — everything** (dev/schnell/fill + ~20 community merges + loras + flux controlnets / redux / pulid / clip-vision / FLUX.D encoder / Florence-2-Flux) | 445 G | superseded by arbo's FLUX.2-klein; loras gen-locked |
| **orphaned umt5** (root `umt5_xxl_fp8`) | 6.7 G | last WAN remnant |
| **orphaned llava_llama3** (fp16 + fp8) | 23.5 G | HunyuanVideo's text encoder — dead after the Hunyuan kill |
| **TOTAL** | **705 G** | 919 → 214 G |
| **Chroma v10/v11 + SD3.5-large** | 28 G | niche generators, not in arbo |
| **TOTAL** | **~739 G** | |
Kept the shared **t5xxl / clip-L / clip-G / CLIP-ViT-H** encoders — SD3.5 + SDXL
still use them.
## 2. Migrated into arbo (177 G)
## What remains (214 G) — migration catalog for comfy-dev
Everything not superseded moved into `/storetank/arbo/models` (same-filesystem atomic
move, skip-existing so arbo's production copies were never clobbered; 652 files moved,
37 skipped as dupes):
### Generators
- **Gen-agnostic utilities:** Aura-SR v1/v2 + the full `upscale_models/` family
(HAT/DAT/RealESRGAN/UltraSharp/Remacri/NMKD/Omni-SR) · Florence-2 + CogFlorence
captioners (`LLM/` + `florence2/`) · controlnet_union_promax · grounding-dino ·
SAM + SAM-HQ · yolo (`ultralytics/`) · depthanything-v2 · vitmatte · nsfw_detector ·
insightface (inswapper + antelopev2) · facerestore · facexlib · ip-adapter-plus_sdxl ·
CLIP-vision (sigclip, EVA02-CLIP-L)
- **SDXL / Pony stack:** ponyRealism, cyberrealisticPony_v8, lustify, hassaku,
waiNSFWIllustrious, juggernaut / dreamshaper Lightning, SUPIR + loras
(dmd2_sdxl_4step, ACE++, Illustrious/PonyXL character-design)
**`checkpoints/` (122 G) — SDXL / Pony / Illustrious bases.** Self-contained lane,
**not** superseded by FLUX.2 (different style space):
- ponyRealism V22 (MainVAE 6.6 G, Hyper-8S, Hyper-4S) · cyberrealisticPony_v8 ·
uberRealisticPornMerge Pony · hardcoreHentai Pony · babesByStableYogi Pony ·
realDream sdxlPony
- juggernautXL Lightning · dreamshaperXL Lightning · sdxlLightning_4Steps (`SDXL Lightning/`)
- lustifySDXLNSFW (endgame + endgame-DMD2) · hassakuXLPony · waiNSFWIllustrious_v110 ·
batchCOREALISM (`SDXL 1.0/`)
- SUPIR upscalers — v0Q / v0F (`.ckpt` 5.0 G ea + fp16 2.5 G ea)
- controlnetxlCNXL Anytest V4 (2.3 G)
comfy-dev owns the follow-on: per-model catalog entries + graphs + heroes that turn
these files into usable arbo workflows.
**`unet/` (28 G) — other generators:**
- chroma-unlocked v10 + v11 Q8 (9.6 G ea) — Chroma (flux-schnell-derived, Apache)
- sd3.5_large Q8 (8.2 G) — SD3.5 Large
## 3. Final state
### LoRAs (2.1 G) — SDXL/Pony only (FLUX.1/WAN2.1 loras were killed)
- dmd2_sdxl_4step (+ fp16) — SDXL 4-step distill
- ACE++ (ali-vilab): portrait_lora64, subject_lora16, local_lora16 — FLUX-editing loras
- Illustrious + PonyXL character-design concept loras
### Gen-agnostic utility models — the prime migration candidates (work with ANY generator)
- **Upscalers:** Aura-SR v1/v2 (9.3 G) · `upscale_models/` (3.8 G — HAT, DAT,
RealESRGAN, UltraSharp, Remacri, NMKD, Omni-SR, AnimeSharp — ~50 models)
- **Captioning / VLM:** `LLM/` (8.6 G — Florence-2 large/base PromptGen, CogFlorence) ·
`florence2/` (3.6 G — PromptGen v2 + base)
- **ControlNet:** controlnet_union_promax (2.3 G, SDXL union)
- **Detect / segment / matte:** grounding-dino (1.6 G) · sams (721 M — SAM-HQ + SAM ViT-B) ·
ultralytics (146 M — yolo face/hand/eyes/person) · depthanything v2 (1.3 G) ·
vitmatte (198 M) · nsfw_detector (329 M)
- **Face / ID:** insightface (1.3 G — inswapper_128 + antelopev2) · facerestore (937 M) ·
facexlib (156 M)
- **IP-Adapter:** ip-adapter-plus_sdxl_vit-h (809 M)
- **CLIP vision:** CLIP-ViT-H-14 (3.7 G) · sigclip_384 (0.8 G) · EVA02-CLIP-L (0.8 G)
- **VAE:** sdxl base vae (6.5 G) · SD3.5 vae · vae_approx
- **Misc:** interpolation, luts, mediapipe
### Shared text encoders (kept — needed by SD3.5 / SDXL)
- t5xxl fp16 (9.1 G) + Q8 gguf (4.7 G) · SD35L clip_g / clip_l · ViT-L TE variants
- _(llava_llama3 — HunyuanVideo's text encoder — was swept 2026-06-13 after the
Hunyuan kill, 23.5 G; see the curation table above.)_
## Migration guidance (comfy-dev's call, with the operator)
- **High-value, low-risk → migrate into arbo:** the gen-agnostic utilities
(upscalers, Florence-2 captioners, controlnet-union, grounding-dino / SAM / yolo,
depthanything, insightface, ip-adapter, CLIP-vision). They work with arbo's
FLUX.2 / WAN2.2 stack and arbo may lack several.
- **Migrate IF you still gen SDXL/Pony:** the `checkpoints/` SDXL-Pony stack + its
loras — self-contained, not replaced by FLUX.2.
- **Kill candidates:** Chroma + SD3.5-large (niche, not in arbo). _(The orphaned
llava_llama3, 23.5 G, was already swept — see curation table.)_
**Access:** browse the live tree directly at `irv-ml1:/storetank/image-models/comfy/models`.
`/storetank/image-models/comfy/models` (= native `/opt/ComfyUI/models`) holds only
empty category dirs + `put_*_here` placeholders — **638 K, no models**. Decommissioned;
arbo is the single live tree.