From c577d69e2d53000a6c210aefe8475fa82b0ed160 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 26 Aug 2026 13:39:26 -0700 Subject: [PATCH] feat(litellm): expose run-3's merged tune for parallel hand-testing merged-run03 -> ana-ml2:8099, the run-3 ERP/RP SFT merged into stock instruct. Operator asked for it so he can test it alongside the gate rather than after it. NAMED FOR THE ARTIFACT, NOT A TIER. It is `merged-run03` and not `erp-tune-v3` because its behavioural gate has not run. A tier name arriving before the evidence that would justify it is how a name comes to mean something nobody decided -- and with a v2 already in the list, a v3 reads as a successor to anyone holding the shared key. If it passes, `v3` is a name to give it then, as a decision. brokkr-smithy-dev raised this against my own erp-tune-v3 suggestion and was right. The entry carries the preregistrations ABOVE the description, so a reader meets the commitments before the numbers: T6 one-directional (a gain is uninterpretable against a 3.1x fireball tailwind), T3/T4 at ceiling on base so recovery is UNOBSERVABLE rather than merely unpredicted, and any run-2 comparison descriptive and non-attributable with its five confounds named. Also carries the retraction in-line: "bluemoon is the largest loss contributor at 38.6%" came from a words x 1.4 estimator, not a tokenizer. As encoded it is third at 32.9%. The direction survives (1.4% -> 8.0% of total loss) and that is the finding; the superlative does not. Documents why its config.json is the base's copied verbatim: transformers 5.15.1 save_pretrained silently drops text_config.global_head_dim and num_global_key_value_heads, and vLLM then dies in make_layers with a TypeError naming neither the config nor the field. Cost a failed boot to find. A LoRA merge changes weights, not architecture, so the base config is correct by definition. gemma4-26b-a4b-it-base marked CURRENTLY DOWN rather than deleted -- the tuned arm took GPU0 and only one 26B bf16 seat fits on that card. Kept because the seat returns, and deleting a name to re-add it later is how scoped keys get orphaned. Verified: config parses, no duplicate model_name, gateway healthy after reload, completion returns text in `content` with reasoning_content null. --- stacks/litellm/conf/config.yaml | 80 +++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index 895e360..817ab80 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -734,12 +734,92 @@ model_list: # WARNING NOT A COMPOSE STACK - a bare docker run named erp-gate-base, no restart # policy (deliberate: if it dies mid-gate it stays dead and is seen, rather than being # silently recreated underneath a running measurement). + # + # ⚠ CURRENTLY DOWN (2026-08-26). The run-3 gate's tuned arm took GPU0 for window 2, + # and only one 26B bf16 seat fits on that card (~84 GB of 97.9). This entry 503s + # until the seat is brought back. Kept rather than deleted because the seat WILL + # return -- deleting a name and re-adding it later is how scoped keys get orphaned + # against an allowlist that no longer matches. - model_name: gemma4-26b-a4b-it-base litellm_params: model: hosted_vllm/gemma4-26b-a4b-it-base api_base: http://10.250.50.54:8099/v1 api_key: os.environ/VLLM_API_KEY + # merged-run03 -> the in-house ERP/RP SFT, RUN 3, MERGED bf16 (:8099, vLLM, ana-ml2 + # GPU0). Backing artifact /tank/erp-tune/serve/merged-run03. Exposed at the operator's + # request so he can hand-test it in parallel with its own gate. + # + # #################################################################################### + # WARNING ITS BEHAVIOURAL GATE HAS NOT RUN YET. This is not a shippable tune and it is + # not a successor to erp-tune-v2 -- it is an ARTIFACT NAME, deliberately. It is called + # `merged-run03` and not `erp-tune-v3` because a tier name arriving before the evidence + # that would justify it is exactly how a name comes to mean something nobody decided. + # If it passes its gate and the operator wants it served, `v3` is a name to give it + # THEN, as a decision. + # #################################################################################### + # + # Base: /tank/aimodels/gemma4-26b-a4b-it-bf16 -- the official instruct release, + # UNCHANGED from run 2, so run 3 varies the corpus and not the base. LoRA r64/a128 on + # 205 modules, 1 epoch over 9,662 records / 18.60M ctx tokens at max_seq_len 16384. + # Completed 2026-08-26 in 2:26:35, train_loss 3.234. harness eitri-smithy 9d27b4f, + # clean tree at launch, attn backend flex_attention (requested AND resolved). + # + # WHAT RUN 3 CHANGED vs run 2 -- corpus composition, not capability: + # F1 PIPPA root excised + # F2 bot-turn length floor >= 250 words -- MASKED, not deleted + # F3 register cap <= 20 asterisks / 1k words -- MASKED, not deleted + # F4 placeholder leak {{char}}/{{user}} -- MASKED, not stripped + # dedup direction reversed on bluemoon <-> creative-writing (keep the primary + # source, drop the megamix copy); bluemoon 68 -> 126 conversations + # Effective mix as trained, by context: dialogue 45.8% / kvasir 38.0% / fireball 16.2%. + # bluemoon went 1.399% -> 7.960% of total loss, 5.690x. + # + # ⚠ DO NOT QUOTE "bluemoon is the largest loss contributor at 38.6%" -- RETRACTED + # 2026-08-26. That figure came from a words x 1.4 estimator, not a tokenizer. As + # actually encoded the within-dialogue loss split is c2-logs 35.6% / creative-writing + # 31.5% / bluemoon 32.9%, so bluemoon is third. The DIRECTION survives and is the real + # finding: 1.4% -> 8.0% of total loss. + # + # PREREGISTERED, so results are not reinterpreted after the fact: + # * T6 spatial is ONE-DIRECTIONAL this run. fireball rose to 16.2% of context against + # run 2's realized 5.2% (3.1x), so a T6 GAIN is uninterpretable -- the filters and + # the spatial-share rise push the same way. A T6 LOSS is the informative outcome. + # * T3/T4 CANNOT RECOVER. Measured 100.0% on the base arm, i.e. AT CEILING. They are + # must-not-harm instruments this run; "run 3 again failed to recover T3/T4" is not + # a valid reading, there was no recovery available. + # * Any run-2 comparison is DESCRIPTIVE AND NON-ATTRIBUTABLE -- five things changed + # at once (fireball share 3.1x, total tokens 3.4x smaller, kvasir subset, PIPPA + # excised + F2/F3/F4, different step schedule). A difference in either direction + # must NOT be attributed to the filters. + # + # Merge verified against the artifact on disk, not the live model: 205/205 targeted + # weights differ from base, 356/356 vision tensors byte-identical, 50/50 sampled + # untargeted tensors identical, 1013 = 1013 tensor keys. The no-op-merge case is ruled + # out by measurement rather than by absence of an error. + # + # ⚠ ITS config.json IS THE BASE'S, COPIED VERBATIM -- deliberately. transformers 5.15.1 + # `save_pretrained` silently DROPS `text_config.global_head_dim` and + # `text_config.num_global_key_value_heads`, which it does not model; vLLM then reads + # None and dies in make_layers with "TypeError: '>=' not supported between instances of + # 'NoneType' and 'int'", naming neither the config nor the field. A LoRA merge changes + # weights, not architecture, so the base config is correct by definition. The + # save_pretrained output is kept beside it as config.json.save_pretrained-orig. + # + # Serving flags are load-bearing, same trap as erp-tune-v2: --reasoning-parser gemma4 + # PLUS --default-chat-template-kwargs enable_thinking=false, else every plain response + # lands in reasoning_content with a null content field. + # + # 16K context, not 256K. Trained only on sequences <= 16384; served to match. + # + # WARNING NOT A COMPOSE STACK - a bare docker run named erp-gate-tuned, no restart + # policy (deliberate: if it dies mid-gate it stays dead and is seen). + - model_name: merged-run03 + litellm_params: + model: hosted_vllm/merged-run03 + api_base: http://10.250.50.54:8099/v1 + api_key: os.environ/VLLM_API_KEY + general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_url: os.environ/DATABASE_URL