vh
888ba6a714
Host service (runs as llmuser, owns /opt/fluxgym + GPU access) that runs
sd-scripts SDXL LoRA training on demand for arbo — the infra-ops half of the
in-arbo LoRA training Phase 1 ownership split (vh/arbo
docs/contracts/in-arbo-lora-training-phase1.contract.md §4.1/§2).
- Fixed-invocation only (INV-T7): bounded params -> one sd-scripts command
shape; every param range/allowlist/path-containment checked before spawn;
bad request = 422, never a silent downgrade. 14 unit tests green.
- Thin supervisor: never imports torch; subprocesses the fluxgym venv's
accelerate. 1-job-at-a-time (arbo lease is the serializer, 409 is backstop).
Durable job records + boot reconciliation (§4.3).
- API: POST /train, GET /train/{id}[/log], POST /train/{id}/cancel,
GET /gpu-status (per-device VRAM + tts_on_3090 co-OOM signal), GET /healthz.
- Wire-shape (§7 resolved with comfy-dev): shared /worktank/arbo/train handoff
(group arbotrain, setgid 2770); worker binds 0.0.0.0:8203, arbo reaches via
host.docker.internal:host-gateway (reachability proven on 172.20.0.1:8203);
device-aware TTS steering via /gpu-status.
Deployed to irv-ml1 via playbooks/deploy-lora-training-worker.yaml (elway,
idempotent); systemd unit active; /healthz + /gpu-status verified live.
27 lines
1.2 KiB
Desktop File
27 lines
1.2 KiB
Desktop File
[Unit]
|
|
Description=LoRA training worker (arbo in-arbo LoRA training Phase 1, §4.1) — runs sd-scripts on demand
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
# Runs as llmuser: only llmuser can exec the /opt/fluxgym/.venv + owns the GPU-training surface (§2).
|
|
User=llmuser
|
|
Group=llmuser
|
|
WorkingDirectory=/opt/lora-training-worker
|
|
# The worker's OWN venv (fastapi/uvicorn) — it never imports torch; it subprocesses the fluxgym venv.
|
|
ExecStart=/opt/lora-training-worker/.venv/bin/uvicorn worker.app:app --host 0.0.0.0 --port 8203
|
|
Restart=on-failure
|
|
RestartSec=5
|
|
# State + logs live under the service dir; both must be llmuser-writable (the deploy playbook chowns).
|
|
Environment=LORA_WORKER_STATE_DIR=/opt/lora-training-worker/state
|
|
Environment=LORA_WORKER_LOG_DIR=/opt/lora-training-worker/logs
|
|
Environment=LORA_WORKER_HANDOFF_ROOT=/worktank/arbo/train
|
|
# Give the training subprocess a sane PATH (accelerate is invoked by absolute path regardless).
|
|
Environment=PATH=/opt/lora-training-worker/.venv/bin:/usr/local/bin:/usr/bin:/bin
|
|
StandardOutput=append:/opt/lora-training-worker/logs/service.log
|
|
StandardError=append:/opt/lora-training-worker/logs/service.log
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|