memory: snapshot — R49 1-epoch pilot complete and awaiting adjudication; MeroMero A4B quantized, v2 blocked; althing 3.6.2 rolled
Ships the two pending code changes alongside the memory that describes them: train_voice_lora.py gains --eval-steps/--save-steps (the 3-epoch pilot overfit with per-epoch eval and save_strategy="no", so the minimum was neither visible nor recoverable), and the althing post-office compose is pinned to 3.6.2. Index rewritten: in-flight compressed from 176 lines to the live state, three detail files added for R49 D1-D3, the MeroMero four-failure chain, and the althing rollout. Seven closed pre-08-27 entries archived; the guards held back the rest because they carry open deferred pointers.
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
# `[2026-09-10]` althing 3.6.2 rolled — post office + both heralds, and it was TWO nodes not seven
|
||||
|
||||
forseti shipped 3.6.2 to fix a Claude Code dedupe: the harness drops an inbound peer frame
|
||||
byte-identical to the previous one from the same sender within 30 s, and the herald's poke text was a
|
||||
single constant — so two pokes to a seat inside 30 s lost the second, and that seat waited for rung 1
|
||||
five minutes later. 3.6.2 stamps `From <handle>, poked at HH:MM:SS.` onto every poke so two can never
|
||||
share a body.
|
||||
|
||||
## What I rolled
|
||||
|
||||
post office nh3-docker 3.6.0 -> 3.6.2 built + pushed to gitea, digest-pinned
|
||||
nh3-dev herald 3.6.1 -> 3.6.2
|
||||
nh3-extdev herald 3.1.1 -> 3.6.2
|
||||
|
||||
Every version read back off the RUNNING install, not the tag. Both nodes heartbeating after.
|
||||
|
||||
## ⚠ It was two boxes, not the seven the instruction assumed
|
||||
|
||||
The rollout said "the seven heralds." The post office's own `nodes` table has exactly **two** rows —
|
||||
`nh3-dev` and `nh3-extdev`. The other five have no install, no unit, no heartbeat. Ask the post
|
||||
office, not the fleet inventory.
|
||||
|
||||
## Three traps, all mine to avoid next time
|
||||
|
||||
**My survey said "not-installed" everywhere** because I ran it as `infra-ops` and the heralds run as
|
||||
**`lkraven`**. Same shape as the ana-docker sudo trap but on the USER axis. Had I trusted it I would
|
||||
have concluded the fleet had no heralds.
|
||||
|
||||
**nh3-extdev is worse than "needs the right prefix."** It is a SYSTEM unit (not `--user`), running as
|
||||
`lkraven`, off `/opt/uv-tools/althing-core` — a venv with **no pip and no uv**, so `uv tool install`
|
||||
there is not merely ineffective, it is impossible. Path: build a wheel on nh3-dev, `ensurepip` into
|
||||
the venv, force-reinstall.
|
||||
|
||||
**I took the bus down ~12 minutes, and it was self-inflicted.** The documented backup is
|
||||
stop → checkpoint → verify counts → copy → verify counts. My `PRAGMA wal_checkpoint(TRUNCATE)`
|
||||
DELETED the `-wal`/`-shm` files, and the volume directory is root-owned 755 while the app runs as uid
|
||||
1000 — so WAL mode could not recreate them and the container crash-looped on "attempt to write a
|
||||
readonly database". Fix: `chown 1000:1000` the volume directory. ⚠ The runbook says checkpoint; it
|
||||
does not warn that on this volume that is a one-way trip until ownership is fixed.
|
||||
|
||||
**Then I misread the recovery.** `docker logs --tail 25` without `--timestamps` showed the
|
||||
accumulated crash history from BEFORE the fix and I called it a current failure — nearly rolling back
|
||||
a working deploy. Rollback would not have helped anyway: the fault was in the volume, not the image.
|
||||
|
||||
## The backup lesson that paid for itself
|
||||
|
||||
My first copy took only `post_office.db` and its **sha256 MATCHED the source** — while 7.2 MB of WAL
|
||||
sat uncopied. Restoring it would have silently lost recent mail. The runbook insists on ROW COUNTS
|
||||
rather than hashes for exactly this, and it was right: the hash agreed while the backup was
|
||||
incomplete. Post-deploy counts identical on all six tables (handles 76, messages 1722, recipients
|
||||
1749, nodes 2, sessions 24, notifications 23).
|
||||
|
||||
## The drop-count instrument, corrected
|
||||
|
||||
I reported the defect "reproduces on nh3-dev" with 3 drops in 21 s. **It does not.** Grepping
|
||||
transcripts for the notice WRITES the notice into the transcripts — my grep output and my own quoting
|
||||
of it. Genuine notices carry `"type":"system"`. True count 26 in 15 sessions; my naive count was 109;
|
||||
the session I claimed 3 drops in had **zero**. forseti's own first count had the same contamination.
|
||||
BEFORE baselines captured: nh3-dev **27 in 18**, nh3-extdev **0**.
|
||||
Auto-memory: [[feedback_grep_over_a_log_that_records_your_greps]]
|
||||
@@ -0,0 +1,72 @@
|
||||
# `[2026-09-10]` MeroMero acquisition — the A4B quant landed, the v2 dense has failed FOUR times
|
||||
|
||||
Operator wanted a MeroMero seat. Getting there cost four quant attempts and corrected two of my own
|
||||
wrong hypotheses, so the failure chain is the durable part.
|
||||
|
||||
## The family, because I got it wrong first
|
||||
|
||||
I called the unsuffixed 31B "v1" because v2 is `v2-31B`. The operator caught it. Creation dates from
|
||||
the upstream author settle it:
|
||||
|
||||
2026-04-15 zerofata/G4-MeroMero-26B-A4B 30 layers, kv 8 MoE 128 experts ← the ORIGINAL
|
||||
2026-04-30 zerofata/G4-MeroMero-31B 60 layers, kv 16 DENSE ← a separate line
|
||||
2026-07-31 zerofata/G4-MeroMero-v2-31B 60 layers, kv 16 DENSE
|
||||
|
||||
Not a v1→v2 sequence at one size: an A4B MoE first, then a dense 31B line that later got a v2.
|
||||
⚠ The `-v1-` repo 401s, and I read that as "v1 must be the unsuffixed one" instead of checking dates.
|
||||
|
||||
**The geometry is why it matters.** The A4B is 30 layers / kv 8 — *identical to Pfish-6* (a gemma4
|
||||
26B-A4B) — so it drops into the existing KV budget at 262k. The dense 31B is 60 layers / kv 16, ~4x
|
||||
the KV per token, and did NOT fit 262k on GPU1 alongside the other seats.
|
||||
|
||||
## Published quants: none of them are our scheme
|
||||
|
||||
pekkAi/G4-MeroMero-31B-NVFP4 true v1 dense, W4A4
|
||||
llmfan46/G4-MeroMero-31B-uncensored-heretic-NVFP4 abliterated dense, W4A16_NVFP4 (pulled, 19 GB)
|
||||
pekkAi/G4-MeroMero-26B-A4B-it-...-heretic-NVFP4 abliterated A4B, W4A4 + FP8 KV baked in
|
||||
NVFP4 of v2-31B DOES NOT EXIST, from anyone (0 of 27 v2 repos)
|
||||
|
||||
W4A4 carries our own measured long-context collapse (Granite, 30k ctx), which is why every seat we
|
||||
run is A16. So the operator ruled: pull both ablits bf16 and quantize in-house at W4A16.
|
||||
|
||||
## MTP — checked on the SOURCES, not inferred
|
||||
|
||||
Gemma-4 ships **no MTP head at all**: 0 mtp tensors and no mtp config keys in the A4B bf16, the v2
|
||||
bf16, Pfish-6, or the published quants. The "mtp if applicable" clause is a no-op here; the MTP work
|
||||
in our history is Qwen-family.
|
||||
|
||||
## A4B — SUCCEEDED
|
||||
|
||||
`G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`, 16 G, 7 files, **2m08s**. That time confirms
|
||||
playbook §3.16: weight-only A16 runs a `DataFreePipeline` and never touches the dataset.
|
||||
⚠ Output is missing `preprocessor_config.json` — §4.3 post-step 2 still owed before serving.
|
||||
|
||||
## v2 dense — FOUR failures, each a different layer
|
||||
|
||||
1. **`num_key_value_heads` is None** at `Gemma4TextAttention.__init__`. I hypothesised a transformers
|
||||
version deadlock (config authored by 5.16.1, container on 5.12.1) — **wrong**: all three configs
|
||||
construct fine on 5.12.1. Then I hypothesised the stale `torch_dtype` kwarg reshaping the config —
|
||||
**also wrong**: every kwarg combination returns 16.
|
||||
⚠ **A 2-layer truncation PASSED and hid it.** The failing local is chosen per layer type; only
|
||||
`full_attention` layers take the branch. Testing each layer type individually found it in seconds.
|
||||
Real cause: DogOnKeyboard's config sets `attention_k_eq_v: true` but **omits
|
||||
`num_global_key_value_heads` and `global_head_dim`** — a malformed upload, not a toolchain issue.
|
||||
Patched from zerofata's canonical values (4 / 512) after **shape-verifying** the checkpoint:
|
||||
full-attn k_proj `[2048,5376]` = 4×512, sliding `[4096,5376]` = 16×256, identical to canonical.
|
||||
Original at `config.json.bak-pre-kvfields-20260910`.
|
||||
2. **`initialize model processor ... required when a dataset is provided`** — DogOnKeyboard ships no
|
||||
`processor_config.json` (the A4B does; that is why it succeeded). Copied from canonical, AND
|
||||
dropped the dataset entirely, since §3.16 says A16 ignores it and §3.14 says building it bakes a
|
||||
truncation cap into the shipped tokenizer. Removing it kills both for zero loss.
|
||||
3. **My own bug**: the reference module runs argparse with `required=True` at IMPORT, so blanking
|
||||
`sys.argv` still exited 2. Placeholder args, real argv restored after.
|
||||
4. **`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`** — OPEN. The
|
||||
DogOnKeyboard config carries a `per_layer_config` key that zerofata's lacks (it was the one key in
|
||||
the diff I noted and did not chase). transformers refuses global access to `head_dim` on a
|
||||
heterogeneous config; the fix is likely `allow_global_per_layer_attribute_access=True`, with the
|
||||
warning that a caller assuming homogeneity may then read the wrong value.
|
||||
|
||||
⚠ **My wrapper reported `rc=0` on a failed run** because it read `$?` after an `echo`. A wrapper that
|
||||
reports success on failure is the false-reassurance class; fixed to capture `$?` immediately.
|
||||
|
||||
Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]]
|
||||
@@ -0,0 +1,119 @@
|
||||
# `[2026-09-10]` R49 / BabyBronte — D1–D3 built, carrier settled on dense Qwen3, and the 3-epoch pilot overfit
|
||||
|
||||
Operator opened the day with *"keep the adapter, then look into babybronte and start prepping
|
||||
for a super lightweight small author voice training model regime."* By close the pipeline runs
|
||||
end to end and one adapter exists.
|
||||
|
||||
## Carrier — SETTLED, and it inverted R49's own pin
|
||||
|
||||
Operator: *"use dense qwen3."* Sweep is **`Qwen3-{0.6,1.7,4}B-Base`**, which **overrides R49 H02's
|
||||
stated arms** (the `Qwen3.5` trio). brokkr-smithy amended H02 accordingly.
|
||||
|
||||
The ruling came off a measurement, on pfi-gx10 (GB10), n=10/arm, seq 4096, r=32 on q,k,v,o+MLP,
|
||||
bf16 sdpa, grad-ckpt on, spreads 0.3–2.6%:
|
||||
|
||||
Qwen3-0.6B-Base dense 0.616 B 2,399 tok/s
|
||||
Qwen3-1.7B-Base dense 1.755 B 1,415 tok/s
|
||||
Qwen3-4B-Base dense 4.089 B 717 tok/s
|
||||
Qwen3.5-0.8B-Base hybrid 18 SSM/6 attn 0.765 B 540 tok/s
|
||||
|
||||
**The dense 4.089 B trains 33% FASTER than the hybrid 0.765 B**, on 5.3x the parameters. Cause:
|
||||
Qwen3.5 is 18 linear-attention (Mamba) layers to 6 attention, and no fused kernel is installed
|
||||
(`mamba_ssm`/`causal_conv1d`/`fla`/`kernels` all absent; triton 3.8 present). Grad-ckpt is not the
|
||||
cause (19%, saves 2.6x memory). Batching is not the lever — both families sit at the box roofline
|
||||
at batch 1. Projected per voice: hybrid 0.8B **12 h**, which is LONGER than the 7 h 26B-A4B tune the
|
||||
regime exists to replace. The premise dies on that carrier.
|
||||
|
||||
Registry check closed the "is there something newer" question: **no official Qwen3.6/3.8 below 27B,
|
||||
and neither family publishes a `-Base` checkpoint at any size.** Since the regime requires a
|
||||
non-instruct carrier, dense Qwen3 was not a compromise — it was the only option.
|
||||
|
||||
⚠ Two Qwen3.5 landmines, measured not inferred: the *-Base releases **ship a vision tower** (153
|
||||
`model.visual.*` Linear tensors at 0.8B) plus an MTP head, which `target_modules="all-linear"` would
|
||||
train on text — `AutoModelForCausalLM` drops both, but that RENAMES modules relative to vLLM's
|
||||
serving class, so adapter binding needs a sampled-target-changed check in the serving path. And
|
||||
**cross-document packing is unsafe** because SSM state ignores the attention mask.
|
||||
|
||||
## D1 — corpus
|
||||
|
||||
`gx10:~/r49-corpus/`, instruments `scripts/r49-corpus/{build_corpus,verify_corpus}.py`. Charlotte
|
||||
only (Jane Eyre 1260, Villette 9182, Shirley 30486, The Professor 1028; ids verified against
|
||||
gutenberg.org's own search). **680,291 words · 142 chapters · 950,974 Qwen3 tokens.** 11 acceptance
|
||||
checks pass including lossless round-trip and 0 byte-fallback.
|
||||
|
||||
⚠⚠ **The alphabet INVERTS the Yarros result.** Brontë writes French constantly (Villette, Adèle,
|
||||
Brussels): 75 letters, **23 non-ASCII** — `ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ`. F02 measured Yarros at 0.0002%
|
||||
non-ASCII and derived an ASCII-fold. Under F02's own subset rule the Brontë pool **KEEPS French
|
||||
accents** and **still excludes** Czech/Latvian/Slovak/Hungarian marks. First corpus where deriving
|
||||
the alphabet per work changed the answer rather than confirming it.
|
||||
|
||||
⚠ Typography was inconsistent and it was the TRANSCRIBER, not the author: Shirley = straight quotes
|
||||
+ `--` + zero em-dashes; Jane Eyre/Villette = curly + em-dash; The Professor = curly + `--`.
|
||||
Normalised toward meaning.
|
||||
|
||||
## D2/D3 — entity resolution and rename
|
||||
|
||||
**Gender resolution is now TITLE-FIRST, a change from F02's method rather than a port of it.**
|
||||
Pronoun proximity called **Jane male** — she narrates Jane Eyre, so her name appears mostly in
|
||||
Rochester's dialogue surrounded by his pronouns. Titles have no such blind spot (`Miss Eyre`,
|
||||
`Mrs. Fairfax`, `M. Paul`): **16 entities resolved, zero wrong**, every ambiguous case HELD.
|
||||
**Held means ungendered, not unrenamed** — a held entity is still renamed from the neutral pool,
|
||||
because holding a place leaks it (`Thornfield` × 100 = the `Riders Quadrant` case).
|
||||
|
||||
Pool: French + English per operator, locales `fr_FR, fr_BE, en_GB, en_IE` only (en_US/en_AU carry
|
||||
modern surnames wrong for the 1840s), weighted per work by setting — Brussels novels 60% French,
|
||||
Yorkshire 25%, alphabet-filtered.
|
||||
|
||||
⚠ **F02's collision filter does not carry across corpora, and whole-string filtering is not enough.**
|
||||
The pool drew `Burns` and `Marie` (both Brontë characters), then after fixing that drew
|
||||
**`Pierre-Yves`** while `Pierre` (Mademoiselle St. Pierre) is a Villette character. **Filter by
|
||||
COMPONENT.** Final gate: 0 of 203 source entities survive in any of 24 copy-files.
|
||||
|
||||
## The pilot — and why 3 epochs was wrong
|
||||
|
||||
H02 is **pure continuation by design** (no beat annotation), which is why the pilot was launchable
|
||||
without D4.
|
||||
|
||||
3-epoch run: train loss 3.310 → 2.574 while **held-out loss ROSE every epoch: 3.198 → 3.318 →
|
||||
3.385.** Textbook overfit, and `save_strategy="no"` left nothing to fall back to. Operator: *"kill
|
||||
and retrain 1 epoch."*
|
||||
|
||||
1-epoch rerun, same seed 4919 and byte-identical corpus (sha `3959036cf851bf62`), eval+save every
|
||||
25 of 169 steps so the minimum is LOCATED not assumed:
|
||||
|
||||
step 25 3.2080 step 100 3.1764 step 169 3.1719 ← best, still descending
|
||||
step 50 3.1891 step 125 3.1726
|
||||
step 75 3.1786 step 150 3.1720
|
||||
|
||||
Monotonic descent, no turn. Better on held-out by 0.21 nats while train loss is HIGHER (3.147 vs
|
||||
2.834) — the signature of the earlier run memorising. ⚠ Not strict subsets: cosine LR decays over
|
||||
169 steps here vs 507 there.
|
||||
|
||||
Adapter verified bound: **196/196 lora_B tensors non-zero**. All three arms cut on one harness:
|
||||
`base-unadapted`, `tuned-1ep-seed4919`, `incumbent-style-prompted`.
|
||||
|
||||
## Adjudication is now mine, and the independence is gone
|
||||
|
||||
Operator re-routed R49 entirely to infra-ops (*"leave babybronte to infra — concentrate on r50 and
|
||||
the memory mechanism"*), relayed by brokkr who handed over the instrument and stepped off.
|
||||
|
||||
**I built the corpus, ran the training, and now judge whether it worked** — the exact shape of the
|
||||
three inert gates R49's own journal records. Compensation: brokkr's rule **ratified verbatim and
|
||||
frozen before any adapted text existed**, and I will not amend it after seeing numbers:
|
||||
|
||||
PASS iff adapted delta_cb <= 2.17 on >=3/4 works
|
||||
AND below the base arm by more than the seed-to-seed spread
|
||||
BEAT INCUMBENT iff below the style-prompted gen seat by more than that spread
|
||||
|
||||
⚠ Their controls came back **partly negative and that is what makes them useful**: real held-out
|
||||
Charlotte scores delta_cb 1.65–2.17, Villette ties Thackeray at window level, and **Anne Brontë sits
|
||||
at 2.374**, nearest-Charlotte of the whole panel. So `nearest` cannot separate the sister; only the
|
||||
absolute band can. An adapter landing at 2.2 is not cleanly distinguishable from Anne.
|
||||
|
||||
Incumbent arm targets the CONCRETE model, not the alias: H02's `qwen3.6-35-a3b-heretic` is stale
|
||||
(retired 2026-08-15); `gen` resolves to `hosted_vllm/qwen3.8-27b-uncensored` on ana-ml2:8015,
|
||||
resolved at run start AND end.
|
||||
|
||||
Handoff bundle for adjudication at `/mnt/smithy/handoff/r49/`.
|
||||
|
||||
Related: [[2026-09-10-meromero-acquisition-and-four-quant-failures]]
|
||||
Reference in New Issue
Block a user