From 5007ec1236aea7b3a2105d90e4dfc7f9c067b0e2 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 13 Jun 2026 15:40:56 -0700 Subject: [PATCH] docs(catalog): archive decommissioned + arbo refreshed (502 G post-migration) storetank archive fully resolved (919 G -> 0): ~739 G killed (superseded/niche), 177 G migrated into arbo, rest dupes. Rewrite the archive doc as a decommission record; refresh the arbo catalog to its post-migration 502 G state (+ SDXL/Pony stack + 9 gen-agnostic utility categories). --- docs/arbo-comfyui-model-catalog.md | 85 ++++++++------------ docs/storetank-image-models-archive.md | 106 ++++++++----------------- 2 files changed, 70 insertions(+), 121 deletions(-) diff --git a/docs/arbo-comfyui-model-catalog.md b/docs/arbo-comfyui-model-catalog.md index 52760f3..53f890f 100644 --- a/docs/arbo-comfyui-model-catalog.md +++ b/docs/arbo-comfyui-model-catalog.md @@ -1,60 +1,45 @@ -# Arbo ComfyUI model catalog (worktank set) +# Arbo ComfyUI model catalog -Snapshot **2026-06-13** of the ComfyUI model tree that arbo (hero/asset -generation) consumes, taken at migration time. **~325 G**, migrated from -`/worktank/comfyui/basedir/models` (irv-ml1 NVMe, was 97% full) → -**`/storetank/arbo/models`** (irv-ml1 SATA-SSD) and overlay-mounted back into -the ComfyUI container at `/basedir/models`. +**Host:** irv-ml1 · **Path:** `/storetank/arbo/models` (SATA SSD; overlay-mounted into +the arbo / comfyui container at `/basedir/models`). **502 G** as of 2026-06-13. -Purpose: decide what to **retain for future arbo use** vs prune. This is the -*live* set ComfyUI serves today — distinct from the older 919 G CivitAI-managed -archive at `/storetank/image-models/comfy` (untouched, separate question). +The single live model tree arbo (hero / asset generation) consumes. On 2026-06-13 it +**absorbed 177 G** of gen-agnostic utilities + the SDXL/Pony stack, migrated from the +now-decommissioned `/storetank/image-models/comfy` archive — see +[`storetank-image-models-archive.md`](storetank-image-models-archive.md) for that record. ## Per-category sizes -| Category | Size | Notes | +| Category | Size | Contents | |---|---|---| -| `diffusion_models/` | **203 G** | the bulk — flux2 / wan2.2 / qwen-image / z-image / ideogram (GGUF + fp8) | -| `text_encoders/` | **64 G** | qwen3-VL, qwen2.5-VL, gemma4, umt5/t5-xxl, qwen3 (encoders for the above) | -| `checkpoints/` | **31 G** | SDXL/Pony bases + SUPIR upscalers | -| `loras/` | **14 G** | style/character LoRAs | -| `controlnet/` | 5.7 G | flux upscaler + sdxl union-promax | -| `clip_vision/` | 3.6 G | CLIP-ViT-H, clip_vision_h | -| `vae/` | 3.0 G | wan2.2, flux2, flux1, z-image VAEs | -| `ipadapter/` | 1.5 G | ip-adapter sdxl vit-h | -| `RMBG/` `sams/` `upscale_models/` `ultralytics/` | <1.5 G ea | bg-removal, SAM, upscalers, yolo | -| (empty placeholders) | — | unet, embeddings, hypernetworks, gligen, … (`put_*_here` only) | +| `diffusion_models/` | **203 G** | current-gen generators: flux2-klein / wan2.2 / qwen-image / z-image / ideogram (GGUF + fp8) | +| `checkpoints/` | **147 G** | SDXL / Pony / Illustrious bases — cyberrealisticPony_v180Coreshift (12.9 G), ponyRealism V22, novaAnimeXL, juggernaut/dreamshaper Lightning, lustify, hassaku, waiNSFWIllustrious, realDream + SUPIR upscalers | +| `text_encoders/` | **80 G** | qwen3-VL, qwen2.5-VL, gemma, umt5, t5-xxl, clip variants | +| `loras/` | **16 G** | flux2/wan2.2 (gameart, RetroAnimeFlux, flux1_turbo, zit_*) + migrated SDXL/Pony (dmd2_sdxl_4step, ACE++, character-design) | +| `vae/` | 9.6 G | wan2.2 / flux2 / flux1 / z-image / sdxl / wan2.1 VAEs | +| `Aura-SR/` | 9.3 G | AuraSR v1/v2 upscalers | +| `LLM/` + `florence2/` | 8.6 + 3.6 G | Florence-2 PromptGen large/base + CogFlorence captioners | +| `controlnet/` | 8.1 G | flux upscaler + sdxl union-promax | +| `clip_vision/` | 4.4 G | CLIP-ViT-H, clip_vision_h, sigclip | +| `upscale_models/` | 3.8 G | HAT / DAT / RealESRGAN / UltraSharp / Remacri / NMKD / Omni-SR (~50) | +| `grounding-dino/` | 1.6 G | grounding-dino swinb / swint | +| `ipadapter/` | 1.5 G | ip-adapter-plus / _sdxl vit-h | +| `insightface/` | 1.3 G | inswapper_128 + antelopev2 | +| `depthanything/` | 1.3 G | depth-anything v2 (vitl / vits) | +| `facerestore_models/` | 937 M | GFPGAN v1.3/1.4, GPEN-BFR | +| `RMBG/` `clip/` `sams/` `nsfw_detector/` `vitmatte/` `facexlib/` `ultralytics/` … | <1 G ea | bg-removal, EVA02-CLIP-L, SAM-HQ + SAM, nsfw classifier, matte, face-lib, yolo (face/hand/eyes/person) | -## Notable models (size · file) +## Migrated in 2026-06-13 (177 G from the storetank archive) -**diffusion_models/ (203 G):** -- 15.7 G · Qwen-Image-Edit-2509-Q6_K.gguf -- 14.3 G ×4 · wan2.2 i2v/t2v a14b low+high-noise Q8_0 -- 13.6 G · Wan2.2-Animate-14B-Q6_K · 13.5 G · Wan2.1_14B_VACE-Q6_K -- 11.8 G · flux1-dev-Q8_0 -- 9.3 G ×2 · flux-2-klein 9b + base-9b Q8_0 · 8.9/8.8 G · same in fp8 -- 8.6 G ×2 · ideogram4 (+ unconditional) fp8 -- 6.7 G · z_image_turbo-Q8_0 · 5.0 G ×2 · Wan2.2-Fun-5B-Control, wan2.2-ti2v-5b -- 4.0 G ×2 · flux-2-klein 4b + base-4b Q8_0 · 3.9 G · anima-base-v1.0 · 3.8 G ×2 · klein-4b fp8 +The gen-agnostic utility set (upscalers, Florence-2 captioners, controlnet-union, +grounding-dino / SAM / yolo / depthanything / vitmatte, insightface / facerestore, +ip-adapter, CLIP-vision) **plus** the SDXL/Pony stack (bases + dmd2 / ACE++ / +character-design loras). These work alongside arbo's current FLUX.2 / WAN2.2 / qwen +generators; **comfy-dev** authors the per-model catalog entries + graphs + heroes that +turn them into usable workflows. -**text_encoders/ (64 G):** qwen3vl_8b_fp8 (9.9 G), qwen2.5_vl_7b_fp8 (8.7 G), -gemma4_e4b_it_fp8 (8.4 G), flux2-klein-9b-uncensored-q8 (8.1 G), qwen_3_8b_fp8mixed -(8.1 G), umt5_xxl_fp8 (6.3 G), t5xxl_fp8 (4.6 G), qwen3-4b-zimage-abliterated-q8 -(4.0 G), flux2-klein-4b-uncensored-q8 (4.0 G), qwen_3_06b_base (1.1 G). +## Durability -**checkpoints/ (31 G):** cyberrealisticPony_v180Coreshift (12.9 G), -novaAnimeXL_illustrious (6.5 G), lustifySDXLNSFW_apexV8 (6.5 G), -SUPIR-v0Q/v0F_fp16 (2.5 G ea). - -**loras/ (14 G):** gameart-game-icon sdxl (2.6 G) + diablo (0.9 G), RetroAnimeFluxV1 -(1.0 G), flux1_turbo_alpha, zit_* (nsfw/cinematic/ultrareal), LH_Pixar3D, oil-painting, -hentai_hottie2_klein9b, … (~30 files). - -## Retain guidance (for operator) -- **Likely keep (active gen stack):** the flux2-klein family + their text_encoders, - wan2.2 video set, qwen-image-edit, z-image, the SDXL/Pony checkpoints + their LoRAs, - VAEs, controlnet, clip_vision, ipadapter. This is a coherent current toolkit. -- **Prune candidates:** redundant precisions (e.g., both Q8 *and* fp8 of the same - klein model where one suffices), superseded older variants, one-off LoRAs. -- The 919 G `/storetank/image-models/comfy` archive (CivitAI-managed, older) is a - separate, larger reclaim target — abandoned (nothing mounts it). Operator's call. +- `arbo_db` (gallery/history SQLite) — backed up (restic/Backrest), local disk not NFS. +- The model tree itself is **bulk, reproducible-from-source** → not backed up; this + catalog + the migration record are the recovery map. diff --git a/docs/storetank-image-models-archive.md b/docs/storetank-image-models-archive.md index ee576b9..b906811 100644 --- a/docs/storetank-image-models-archive.md +++ b/docs/storetank-image-models-archive.md @@ -1,88 +1,52 @@ -# Storetank image-models archive — curation record + remaining catalog +# Storetank image-models archive — curation / migration / decommission record -**Host:** irv-ml1 · **Path:** `/storetank/image-models/comfy/models` -(this tree is also the native `/opt/ComfyUI/models` symlink target. The **docker / -arbo** ComfyUI uses a *separate, active* tree at `/storetank/arbo/models` — -see [`arbo-comfyui-model-catalog.md`](arbo-comfyui-model-catalog.md).) +**Host:** irv-ml1 · **Former path:** `/storetank/image-models/comfy/models` +(was the native `/opt/ComfyUI/models` symlink target). +**Status: DECOMMISSIONED 2026-06-13** — emptied of all models (919 G → 0). The +active model tree is arbo's `/storetank/arbo/models` — see +[`arbo-comfyui-model-catalog.md`](arbo-comfyui-model-catalog.md). -**Snapshot:** 2026-06-13. This is the older CivitAI-managed model pile, curated -down from **919 G → 238 G** in a kill-the-superseded pass. Two purposes: (1) record -what was removed and why, and (2) catalog what remains, so **comfy-dev** can pick -migration candidates into the active arbo set. +Historical record of how the 919 G CivitAI-managed pile was resolved on 2026-06-13: +**~739 G killed** (superseded / niche), **177 G migrated** into the arbo set, the +remainder dupes arbo already had. -## Curation pass (2026-06-13) — 681 G reclaimed +## 1. Killed — superseded by arbo's current-gen stack (~739 G) -Killed everything superseded by arbo's current-gen stack (FLUX.2-klein, WAN2.2, -qwen-image, z-image). Two principles: generation-locked LoRAs have no value without -their (also-superseded) base models, and arbo already carries its own copies of the -shared encoders/VAEs. +Two principles: generation-locked LoRAs have no value without their (also-superseded) +base models, and arbo already carries its own copies of the shared encoders/VAEs. | Killed | Size | Why | |---|---|---| -| **Hunyuan video** (diffusion_models + unet + vae + loras) | 74 G | older video arch; not in arbo, niche | +| **Hunyuan video** (diffusion_models + unet + vae + loras) | 74 G | older video arch; not in arbo | | **WAN2.1 bases + loras** | 118 G | superseded by arbo's WAN2.2; loras gen-locked | -| **WAN2.1 encoders / VAE** (umt5, xlm-roberta, clip_vision_h, wan VAE) | 44 G | exact dupes of arbo's own copies | +| **WAN2.1 encoders / VAE** (umt5, xlm-roberta, clip_vision_h, wan VAE) | 44 G | dupes of arbo's own copies | | **FLUX.1 — everything** (dev/schnell/fill + ~20 community merges + loras + flux controlnets / redux / pulid / clip-vision / FLUX.D encoder / Florence-2-Flux) | 445 G | superseded by arbo's FLUX.2-klein; loras gen-locked | | **orphaned umt5** (root `umt5_xxl_fp8`) | 6.7 G | last WAN remnant | | **orphaned llava_llama3** (fp16 + fp8) | 23.5 G | HunyuanVideo's text encoder — dead after the Hunyuan kill | -| **TOTAL** | **705 G** | 919 → 214 G | +| **Chroma v10/v11 + SD3.5-large** | 28 G | niche generators, not in arbo | +| **TOTAL** | **~739 G** | | -Kept the shared **t5xxl / clip-L / clip-G / CLIP-ViT-H** encoders — SD3.5 + SDXL -still use them. +## 2. Migrated into arbo (177 G) -## What remains (214 G) — migration catalog for comfy-dev +Everything not superseded moved into `/storetank/arbo/models` (same-filesystem atomic +move, skip-existing so arbo's production copies were never clobbered; 652 files moved, +37 skipped as dupes): -### Generators +- **Gen-agnostic utilities:** Aura-SR v1/v2 + the full `upscale_models/` family + (HAT/DAT/RealESRGAN/UltraSharp/Remacri/NMKD/Omni-SR) · Florence-2 + CogFlorence + captioners (`LLM/` + `florence2/`) · controlnet_union_promax · grounding-dino · + SAM + SAM-HQ · yolo (`ultralytics/`) · depthanything-v2 · vitmatte · nsfw_detector · + insightface (inswapper + antelopev2) · facerestore · facexlib · ip-adapter-plus_sdxl · + CLIP-vision (sigclip, EVA02-CLIP-L) +- **SDXL / Pony stack:** ponyRealism, cyberrealisticPony_v8, lustify, hassaku, + waiNSFWIllustrious, juggernaut / dreamshaper Lightning, SUPIR + loras + (dmd2_sdxl_4step, ACE++, Illustrious/PonyXL character-design) -**`checkpoints/` (122 G) — SDXL / Pony / Illustrious bases.** Self-contained lane, -**not** superseded by FLUX.2 (different style space): -- ponyRealism V22 (MainVAE 6.6 G, Hyper-8S, Hyper-4S) · cyberrealisticPony_v8 · - uberRealisticPornMerge Pony · hardcoreHentai Pony · babesByStableYogi Pony · - realDream sdxlPony -- juggernautXL Lightning · dreamshaperXL Lightning · sdxlLightning_4Steps (`SDXL Lightning/`) -- lustifySDXLNSFW (endgame + endgame-DMD2) · hassakuXLPony · waiNSFWIllustrious_v110 · - batchCOREALISM (`SDXL 1.0/`) -- SUPIR upscalers — v0Q / v0F (`.ckpt` 5.0 G ea + fp16 2.5 G ea) -- controlnetxlCNXL Anytest V4 (2.3 G) +comfy-dev owns the follow-on: per-model catalog entries + graphs + heroes that turn +these files into usable arbo workflows. -**`unet/` (28 G) — other generators:** -- chroma-unlocked v10 + v11 Q8 (9.6 G ea) — Chroma (flux-schnell-derived, Apache) -- sd3.5_large Q8 (8.2 G) — SD3.5 Large +## 3. Final state -### LoRAs (2.1 G) — SDXL/Pony only (FLUX.1/WAN2.1 loras were killed) -- dmd2_sdxl_4step (+ fp16) — SDXL 4-step distill -- ACE++ (ali-vilab): portrait_lora64, subject_lora16, local_lora16 — FLUX-editing loras -- Illustrious + PonyXL character-design concept loras - -### Gen-agnostic utility models — the prime migration candidates (work with ANY generator) -- **Upscalers:** Aura-SR v1/v2 (9.3 G) · `upscale_models/` (3.8 G — HAT, DAT, - RealESRGAN, UltraSharp, Remacri, NMKD, Omni-SR, AnimeSharp — ~50 models) -- **Captioning / VLM:** `LLM/` (8.6 G — Florence-2 large/base PromptGen, CogFlorence) · - `florence2/` (3.6 G — PromptGen v2 + base) -- **ControlNet:** controlnet_union_promax (2.3 G, SDXL union) -- **Detect / segment / matte:** grounding-dino (1.6 G) · sams (721 M — SAM-HQ + SAM ViT-B) · - ultralytics (146 M — yolo face/hand/eyes/person) · depthanything v2 (1.3 G) · - vitmatte (198 M) · nsfw_detector (329 M) -- **Face / ID:** insightface (1.3 G — inswapper_128 + antelopev2) · facerestore (937 M) · - facexlib (156 M) -- **IP-Adapter:** ip-adapter-plus_sdxl_vit-h (809 M) -- **CLIP vision:** CLIP-ViT-H-14 (3.7 G) · sigclip_384 (0.8 G) · EVA02-CLIP-L (0.8 G) -- **VAE:** sdxl base vae (6.5 G) · SD3.5 vae · vae_approx -- **Misc:** interpolation, luts, mediapipe - -### Shared text encoders (kept — needed by SD3.5 / SDXL) -- t5xxl fp16 (9.1 G) + Q8 gguf (4.7 G) · SD35L clip_g / clip_l · ViT-L TE variants -- _(llava_llama3 — HunyuanVideo's text encoder — was swept 2026-06-13 after the - Hunyuan kill, 23.5 G; see the curation table above.)_ - -## Migration guidance (comfy-dev's call, with the operator) -- **High-value, low-risk → migrate into arbo:** the gen-agnostic utilities - (upscalers, Florence-2 captioners, controlnet-union, grounding-dino / SAM / yolo, - depthanything, insightface, ip-adapter, CLIP-vision). They work with arbo's - FLUX.2 / WAN2.2 stack and arbo may lack several. -- **Migrate IF you still gen SDXL/Pony:** the `checkpoints/` SDXL-Pony stack + its - loras — self-contained, not replaced by FLUX.2. -- **Kill candidates:** Chroma + SD3.5-large (niche, not in arbo). _(The orphaned - llava_llama3, 23.5 G, was already swept — see curation table.)_ - -**Access:** browse the live tree directly at `irv-ml1:/storetank/image-models/comfy/models`. +`/storetank/image-models/comfy/models` (= native `/opt/ComfyUI/models`) holds only +empty category dirs + `put_*_here` placeholders — **638 K, no models**. Decommissioned; +arbo is the single live tree.