vh
20bbb95113
The LossLog callback filtered on `"loss" in logs`. Trainer emits eval under `eval_loss` with no `loss` key, so every eval record was discarded and loss-series.json showed zero eval points while the log a metre away carried `eval_loss: 3.198`. An artefact that omits data which demonstrably exists reads as "no eval was run" rather than "the collector dropped it", which is the failure mode that costs someone a re-run. Collector now accepts either key. Seed 2 gets it from launch; seed 1 is already running with the old code, so recover_eval_series.py parses the eval records back out of its log -- the data was printed, not lost -- and the chain runs that recovery before cutting the generation arms, so the artefact is complete before anything reads it. The two seeds therefore differ in logging code but not in training math: the callback only affects what is recorded, never what is computed, so the weight trajectories remain comparable. Noting it because a difference between the two arms whose spread sets the decision threshold is worth stating even when it is provably inert.
39 lines
1.5 KiB
Python
39 lines
1.5 KiB
Python
"""Recover eval records a run's log emitted but its loss-series artefact dropped.
|
|
|
|
Written because the seed-1 pilot's LossLog callback filtered on `"loss" in logs`,
|
|
which excludes every eval record (Trainer emits `eval_loss` with no `loss` key).
|
|
The data was never lost -- it was printed and not collected -- so it is parsed
|
|
back out of the log rather than re-run. Idempotent: existing eval points are not
|
|
duplicated.
|
|
"""
|
|
import ast, json, re, sys
|
|
from pathlib import Path
|
|
|
|
log, series = Path(sys.argv[1]), Path(sys.argv[2])
|
|
s = json.loads(series.read_text())
|
|
have = {p["step"] for p in s if "eval_loss" in p}
|
|
found = []
|
|
for m in re.finditer(r"\{[^{}]*'eval_loss'[^{}]*\}", log.read_text(errors="replace")):
|
|
try:
|
|
d = ast.literal_eval(m.group(0))
|
|
except Exception:
|
|
continue
|
|
d = {k: (float(v) if isinstance(v, str) and re.fullmatch(r"-?[\d.eE+]+", v) else v)
|
|
for k, v in d.items()}
|
|
found.append(d)
|
|
# Trainer logs eval without a step field; epoch is present, so derive step order.
|
|
added = 0
|
|
for i, d in enumerate(found):
|
|
if d.get("epoch") is None:
|
|
continue
|
|
step = d.get("step")
|
|
if step is None:
|
|
after = [p["step"] for p in s if "loss" in p and p.get("epoch", 0) <= d["epoch"]]
|
|
step = max(after) if after else 0
|
|
if step in have:
|
|
continue
|
|
s.append({"step": step, **d}); have.add(step); added += 1
|
|
s.sort(key=lambda p: (p["step"], "eval_loss" in p))
|
|
series.write_text(json.dumps(s, indent=1))
|
|
print(f" recovered {added} eval records from {log.name} ({len(found)} found in log)")
|