diff --git a/docs/fleettools/FLEETTOOLS.md b/docs/fleettools/FLEETTOOLS.md index bcfccf3..59a764a 100644 --- a/docs/fleettools/FLEETTOOLS.md +++ b/docs/fleettools/FLEETTOOLS.md @@ -82,6 +82,12 @@ live contract for its API. Fetch it rather than trusting a transcription. *When:* 3D rendering, STL → image, scene building. *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/blender.md` +- **MIA auto-rig** — Make-It-Animatable v2 on fv-ml1 GPU 3, one-shot: humanoid GLB in, 52-bone + Mixamo-style skeleton + skin weights out (npz, FBX, GLB), ~4.5 s a mesh plus ~13 s model load. + `scripts/mia-run --job DIR -- = ...`. Seeded by default. + *When:* rigging a character mesh for animation. + *Detail:* `/home/lkraven/development/eshpfi-management/stacks/mia/README.md` + ## Working on the fleet itself - **elway** — SSH playbook runner for **CHANGING** things. diff --git a/persistent-memory.md b/persistent-memory.md index 35e6906..dd5e13b 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -150,7 +150,7 @@ _As of 2026-10-01 ~0446 PT._ - **GPU 0:** cyberprev (47.1 GB), gen-small (35.3 GB), voices (10.8 GB), parakeet-nemo (3.6 GB steady). Free 385 MiB, FULL. - **GPU 1:** vllm-coder, erp-seat, meromero-rp, plus intern-decision (cap 14.4 GiB, 32k tokens, peak 15,220 of a 15,437 MiB budget). FULL. -- **GPU 3:** the full-size-seat reserve (Flash-Next is parked). On-demand tenants: Blender, and Scriberr (0 idle, ~5.5 GB per job). When a full-size seat claims GPU 3, Scriberr steps aside to **irv-ml1's A6000**, not back to GPU 1. +- **GPU 3:** the full-size-seat reserve (Flash-Next is parked). On-demand tenants: Blender, Scriberr (0 idle, ~5.5 GB per job), and MIA auto-rig (one-shot `scripts/mia-run`, peak 3.4 GB, added 2026-10-01 for dread-dev). When a full-size seat claims GPU 3, Scriberr steps aside to **irv-ml1's A6000**, not back to GPU 1. ### intern-decision (replaced SemIf on 2026-09-30) @@ -290,6 +290,7 @@ _As of 2026-10-01 ~0446 PT._ ## Recent decisions +- `[2026-10-01]` **MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev).** `scripts/mia-run --job DIR -- name=in.glb ...`, image `local/mia:0.1.0` (10.5 GB), weights in the shared HF cache at pinned revisions (sha256-verified). Acceptance: 3.9–4.7 s a mesh (median of 3), 12.7 s model load, peak 3,394 MiB; seeded runs bit-identical; GPU-vs-CPU distances the same size as sampling noise (positive control: unseeded GPU). Skeleton template is a SUBSTITUTE (gated HF dataset `jasongzy/Mixamo`, terms not accepted; Prime's call). ⚠ fv-ml1 zroot at 85% after the build. → `stacks/mia/README.md` - `[2026-10-01]` **Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed.** They were never 1,788 real snapshots: the prune had deleted everything except one 0555 dir (`vastblue/praxis/references/PraxisPM_Rev0_07.11.26`, 17 files hardlinked into every snapshot), so each old dir was a 22-entry husk and only the newest 48 were complete. Removing them lost no history and freed ~0 bytes (same inodes, link count 1,788). Deletion ran from a generated list of 1,740 literal paths, each verified as a husk first, with 0 errors. The script now runs `chmod -R u+w` before `rm`, logs the error count and the kept count, and fails the unit (exit 3) on a bad prune or exit 1 on a failed rsync. Positive control: a manual run at 0556 pruned the full snapshot `2026-09-29_0602` with 0 errors, leaving 48. Retention stays the designed 48 hourly; the 4.6 GB log rotated at 0530 is kept, compressed to 99 MB, at `~/.config/dev-backup/dev-backup.log.20261001-0530.gz` until someone deletes it. - `[2026-10-01]` **Prime: gen-small keeps its 8 GiB KV pin, and parakeet-nemo keeps CUDA graphs OFF** (~2–8 ms at short clips, accepted). GPU 0's ~1 GB of spare memory is enough with the seat's hard 3,840 MiB cap. - `[2026-10-01]` **⚠ Gitea's `[webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8` silently REJECTS headscale mesh IPs (100.64.0.0/10).** The test API still returns 204 and nothing arrives. The vh/arbo hook therefore targets irv-ml1's LAN `10.6.110.50:9009`; the secret was re-set and a delivery is verified (deploy ran 04:41). A Gitea webhook PATCH without `secret` and `branch_filter` drops both, so always resend them. diff --git a/scripts/mia-run b/scripts/mia-run new file mode 100755 index 0000000..06e5fcf --- /dev/null +++ b/scripts/mia-run @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +# mia-run — one-shot Make-It-Animatable v2 auto-rig on fv-ml1 GPU 3 (dread-dev / Dread Naught; +# Prime's request, 2026-10-01). Same shape as blender-run: a container starts, rigs, exits, and +# hands the GPU back. NOT a seat; nothing stays up between calls. +# +# scripts/mia-run --job DIR [--seed N | --unseeded] -- = [= ...] +# scripts/mia-run --job ~/development/dreadnaught/rig-jobs/j1 -- goblin=in/goblin.glb ogre=in/ogre.glb +# +# Inputs are paths RELATIVE TO DIR and must be inside it: DIR is the only thing shipped to fv-ml1. +# Per mesh, DIR/out/ receives: +# _pred.npz predictions in the input's coordinates, WITH `weights_effective` +# (the finalize.py formula), so no finalize step is needed afterwards +# .fbx .glb _rest.glb MIA's own exports (UI defaults, static rig) +# _apose-hint.fbx .glb _rest.glb same predictions, Blender stage with the A-pose hint +# _run.json stage timings, model-load time, peak GPU memory, seed, provenance +# The model load (~3 x 1 GB checkpoints) is paid once per call, so pass every mesh in one call. +# +# SEEDED BY DEFAULT (seed 0): before every mesh the driver calls MIA's fix_random(seed) and resets +# trimesh's module RNG, which fix_random never reaches (trimesh >= 4 samples surfaces from +# trimesh.util._RANDOM_DEFAULT). Without that, runs of the same mesh differ: joints by 0.1-0.27% of +# height on dread-dev's CPU noise floor. --unseeded restores upstream behaviour. +# +# Files: fv-ml1 does NOT mount /mnt/smithy. DIR is mirrored to +# fv-ml1:/tank/mia/jobs/-/ (rsync --delete, so a rerun never +# inherits leftovers), the driver runs WITH THAT AS ITS WORKING DIRECTORY, and new or changed files +# are copied back into DIR afterwards (nothing is ever deleted locally). Running the SAME DIR twice +# concurrently shares one remote dir, so don't. Remote job dirs are never swept. +# +# Image: local/mia (stacks/mia; IMAGE= in fv-ml1:/opt/docker/compose/mia/.env). Weights come from +# the read-only /tank/aimodels/huggingface mount at pinned revisions; the run never touches the +# network (HF_HUB_OFFLINE=1). The skeleton template is a SUBSTITUTE ("Standard Run.fbx"): the +# official one is in a gated HF dataset whose terms were not accepted (see stacks/mia/README.md). +# +# Budget: GPU 3 (96 GB, borrowed from the vLLM reserve, shared on demand with Blender and Scriberr; +# this ends if a full-size seat moves in). Capped at 32 GB RAM / 16 CPUs so a runaway mesh cannot +# starve the inference seats on the same host. +set -euo pipefail + +HOST=${MIA_SSH_HOST:-infra-ops@10.251.50.54} +ENV_FILE=/opt/docker/compose/mia/.env +JOB="" +DRIVER_OPTS=() +while [ $# -gt 0 ]; do + case "$1" in + --job) JOB=${2:?--job needs a directory}; shift 2 ;; + --seed) DRIVER_OPTS+=(--seed "${2:?--seed needs an integer}"); shift 2 ;; + --unseeded) DRIVER_OPTS+=(--unseeded); shift ;; + --) shift; break ;; + -h|--help) sed -n 2,40p "$0"; exit 0 ;; + *) echo "mia-run: unknown option $1 (meshes go after --)" >&2; exit 2 ;; + esac +done +[ -n "$JOB" ] || { echo "mia-run: --job DIR is required (inputs are read from it, out/ is written into it)" >&2; exit 2; } +[ $# -gt 0 ] || { echo "mia-run: no meshes; pass = after --" >&2; exit 2; } +[ -d "$JOB" ] || { echo "mia-run: --job $JOB is not a directory" >&2; exit 2; } +ABS=$(realpath "$JOB") +BASE=$(basename "$ABS") +[[ $BASE =~ ^[A-Za-z0-9._-]+$ ]] || { echo "mia-run: job dir name '$BASE' must be [A-Za-z0-9._-]" >&2; exit 2; } +for m in "$@"; do + [[ $m == *=* ]] || { echo "mia-run: '$m' is not =" >&2; exit 2; } + f=$(realpath -m "$ABS/${m#*=}") + [[ $f == "$ABS"/* && -f $f ]] || { echo "mia-run: input '${m#*=}' is not a file inside $ABS" >&2; exit 2; } +done + +# Keyed on basename + a hash of the ABSOLUTE local path, as blender-run: two dirs that share a +# basename never share a remote dir. rsync removes only inside that one remote dir; there is no rm +# on a computed path. +NAME=$BASE-$(printf '%s' "$ABS" | sha256sum | cut -c1-8) +ssh -n -o BatchMode=yes "$HOST" "mkdir -p /tank/mia/jobs/$NAME" +rsync -a --delete "$JOB"/ "$HOST:/tank/mia/jobs/$NAME/" + +# Arguments travel as one shell-quoted string: ssh flattens argv into a remote command line. +ARGS=$(printf '%q ' "${DRIVER_OPTS[@]}" "$@") +set +e +ssh -n -o BatchMode=yes "$HOST" "IMG=\$(grep '^IMAGE=' $ENV_FILE | cut -d= -f2) && \ + exec docker run --rm --name mia-run-\$\$ --hostname fv-ml1-mia \ + --runtime nvidia -e NVIDIA_VISIBLE_DEVICES=3 -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \ + --user 1002:1003 --memory 32g --cpus 16 \ + --mount type=bind,src=/tank/aimodels/huggingface,dst=/hf,readonly \ + --mount type=bind,src=/tank/mia/jobs/$NAME,dst=/job \ + -w /job \"\$IMG\" $ARGS" +RC=$? +set -e +rsync -a --update "$HOST:/tank/mia/jobs/$NAME/" "$JOB"/ +exit $RC diff --git a/stacks/mia/.env.example b/stacks/mia/.env.example new file mode 100644 index 0000000..4b87913 --- /dev/null +++ b/stacks/mia/.env.example @@ -0,0 +1,2 @@ +# Image scripts/mia-run starts (built from this dir on fv-ml1 under /opt/docker/src/mia-/). +IMAGE=local/mia:0.1.0 diff --git a/stacks/mia/Dockerfile b/stacks/mia/Dockerfile new file mode 100644 index 0000000..70a86d3 --- /dev/null +++ b/stacks/mia/Dockerfile @@ -0,0 +1,69 @@ +# Make-It-Animatable (MIA) v2: one-shot ML auto-rigger for humanoid meshes (GLB in, 52-bone +# Mixamo-style skeleton + skin weights out). Run by scripts/mia-run on fv-ml1 GPU 3, for dread-dev +# (Dread Naught), Prime's request 2026-10-01. NOT a seat: each run is a `docker run --rm` that +# rigs, writes the job's out/, exits and hands the GPU back. +# +# Code: github.com/jasongzy/Make-It-Animatable, branch v2 @ MIA_COMMIT (MIT), cloned with the +# submodules that commit pins (util/Hunyuan3D_21 @ b691197, util/auto_rig_pro, util/3dgs-render- +# blender-addon). Python: requirements.lock (dread-dev's proven CPU venv, torch family swapped to +# cu129). Upstream app_v2.py is UNPATCHED: dread-dev's CPU patch only upcasts on CPU. +# +# Weights are NOT baked in. /tank/aimodels/huggingface is bind-mounted read-only at /hf, and the two +# pinned snapshots are reached through the symlinks made below: +# repo/output -> jasongzy/Make-It-Animatable @ MIA_WEIGHTS_REV (output/best/v2/*.pth, Apache-2.0) +# repo/data/Standard Run.fbx, repo/data/examples -> the same snapshot (init_blocks() reads +# data/examples/log.csv even headless; without it the driver dies before the first mesh) +# /opt/mia/hy3dgen/tencent/Hunyuan3D-2.1 -> tencent/Hunyuan3D-2.1 @ HY3D_REV (hunyuan3d-vae-v2-1) +# repo/data/Mixamo/bones.fbx -> "Standard Run.fbx": the official template sits in the GATED HF +# dataset jasongzy/Mixamo, whose terms were not accepted on Prime's account. It sets bone roll +# only, never joints or weights (dread-dev, 2026-10-01). Accepting those terms is Prime's call. +FROM python:3.11-slim-bookworm + +ARG MIA_COMMIT=bbd8b158d88879c310ad130f9b25056935d221e9 +ARG MIA_WEIGHTS_REV=ca0daf6cb164f939e77bf32667513fc7558d5f98 +ARG HY3D_REV=0b94677654c57bb9a6b6845cd7b704ccf551d327 + +ENV DEBIAN_FRONTEND=noninteractive \ + PIP_DISABLE_PIP_VERSION_CHECK=1 \ + PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 \ + HF_HOME=/hf \ + HF_HUB_OFFLINE=1 \ + GRADIO_ANALYTICS_ENABLED=False \ + MPLCONFIGDIR=/tmp/mpl \ + HY3DGEN_MODELS=/opt/mia/hy3dgen \ + HOME=/tmp + +# git for the clone; the X11/GL libs are what the bpy 4.3 wheel links against (import fails without). +RUN apt-get update && apt-get install -y --no-install-recommends \ + git ca-certificates libgl1 libegl1 libglib2.0-0 libx11-6 libxrender1 libxxf86vm1 libxfixes3 \ + libxi6 libxkbcommon0 libsm6 libice6 libgomp1 \ + && rm -rf /var/lib/apt/lists/* + +# UV_LINK_MODE=copy: uv's default clone (reflink) fails on the build's overlay fs, "os error 22". +COPY requirements.lock /opt/mia/requirements.lock +RUN pip install -q uv \ + && UV_LINK_MODE=copy uv pip install --system --no-cache --index-strategy unsafe-best-match \ + --extra-index-url https://download.pytorch.org/whl/cu129 \ + --extra-index-url https://download.blender.org/pypi/ \ + --extra-index-url https://miropsota.github.io/torch_packages_builder \ + --find-links https://data.pyg.org/whl/torch-2.8.0+cu129.html \ + -r /opt/mia/requirements.lock \ + && uv pip freeze --system > /opt/mia/freeze.txt + +RUN git clone --single-branch --branch v2 https://github.com/jasongzy/Make-It-Animatable /opt/mia/repo \ + && cd /opt/mia/repo && git checkout -q "$MIA_COMMIT" && git submodule update --init --recursive -q \ + && git -C util/Hunyuan3D_21 rev-parse HEAD > /opt/mia/hunyuan3d_21.commit \ + && rm -rf .git util/*/.git output \ + && W=/hf/hub/models--jasongzy--Make-It-Animatable/snapshots/$MIA_WEIGHTS_REV \ + && ln -s "$W/output" output \ + && ln -s "$W/data/Standard Run.fbx" "data/Standard Run.fbx" \ + && ln -s "$W/data/examples" data/examples \ + && mkdir -p data/Mixamo && ln -s "../Standard Run.fbx" data/Mixamo/bones.fbx \ + && mkdir -p /opt/mia/hy3dgen/tencent \ + && ln -s /hf/hub/models--tencent--Hunyuan3D-2.1/snapshots/$HY3D_REV /opt/mia/hy3dgen/tencent/Hunyuan3D-2.1 \ + && printf 'MIA_COMMIT=%s\nMIA_WEIGHTS_REV=%s\nHY3D_REV=%s\n' "$MIA_COMMIT" "$MIA_WEIGHTS_REV" "$HY3D_REV" > /opt/mia/provenance + +COPY mia_driver.py /opt/mia/mia_driver.py +WORKDIR /job +ENTRYPOINT ["python", "/opt/mia/mia_driver.py"] diff --git a/stacks/mia/README.md b/stacks/mia/README.md new file mode 100644 index 0000000..21054ad --- /dev/null +++ b/stacks/mia/README.md @@ -0,0 +1,98 @@ +# mia — Make-It-Animatable v2 auto-rigger (fv-ml1 GPU 3, one-shot) + +**What:** [Make-It-Animatable](https://github.com/jasongzy/Make-It-Animatable) v2 auto-rigs a +humanoid mesh (GLB in): it predicts a 52-bone Mixamo-style skeleton and skin weights. Dread Naught +(dread-dev) uses it to rig AI-generated characters. On 2026-10-01 it beat Blender bone-heat rigging +on the four game characters (Booth `dreadnaught-rigging`). + +**Shape: on demand only, like blender-run (Prime, 2026-10-01).** Each `scripts/mia-run` call is +one `docker run --rm`: load the models, rig every mesh passed, write the job's `out/`, exit, hand +GPU 3 back. Nothing stays up. There is no compose service; this dir is the image's build context. + +```bash +scripts/mia-run --job DIR -- goblin=in/goblin.glb ogre=in/ogre.glb # seeded (seed 0) +scripts/mia-run --job DIR --seed 7 -- goblin=in/goblin.glb +scripts/mia-run --job DIR --unseeded -- goblin=in/goblin.glb # upstream behaviour +``` + +Inputs are relative to DIR. Per mesh, `DIR/out/` gets `_pred.npz` (with +`weights_effective`), `.fbx`, `.glb`, `_rest.glb`, the three `_apose-hint` +variants, and `_run.json` (timings, peak GPU memory, seed, provenance). The job mirroring +is blender-run's: DIR → `fv-ml1:/tank/mia/jobs/-/`, results copied back. + +## What is pinned + +| piece | pin | where | +|---|---|---| +| code | `jasongzy/Make-It-Animatable` branch v2 @ `bbd8b158` (MIT); submodule `util/Hunyuan3D_21` @ `b6911977` | cloned into the image, `.git` dropped | +| MIA weights | HF `jasongzy/Make-It-Animatable` @ `ca0daf6c` (Apache-2.0): `output/best/v2/{bw_joints,joints_coarse,pose}.pth`, `data/Standard Run.fbx`, `data/examples/` | fv-ml1 HF cache `/tank/aimodels/huggingface`, mounted read-only at `/hf` | +| shape VAE | HF `tencent/Hunyuan3D-2.1` @ `0b946776`: `hunyuan3d-vae-v2-1/{config.yaml,model.fp16.ckpt}` (Tencent Hunyuan community licence) | same | +| Python | `requirements.lock`: dread-dev's CPU venv verbatim, torch family swapped to cu129 (torch 2.8.0, torchvision 0.23.0, torch-cluster 1.6.3, pytorch3d 0.7.8); `bpy` 4.3.0, `trimesh` 5.1.0, `numpy` 1.26.4 | image (`/opt/mia/freeze.txt` records the install) | +| driver | `mia_driver.py` (adapted from dread-dev's `tools/mia/run_mia.py` + `finalize.py`) | image, entrypoint | + +Every weight file's sha256 was checked against the HF API's LFS hash after download (2026-10-01). +`/opt/mia/provenance` in the image and `provenance` in each `_run.json` carry the three pins. + +## Deviations from upstream (all deliberate) + +- **The skeleton template is a substitute.** `data/Mixamo/bones.fbx` → `data/Standard Run.fbx`. + The official file is in the **gated** HF dataset `jasongzy/Mixamo`, whose terms were NOT accepted + on Prime's account; accepting them is his call. The template sets bone roll only, never joints + or weights (dread-dev). Swapping in the official file later changes FBX bone roll, not the npz. +- **Seeded by default.** Before every mesh the driver calls MIA's `fix_random(seed)` (also sets + cudnn deterministic) and resets `trimesh.util._RANDOM_DEFAULT`, which `fix_random` never reaches + (trimesh ≥ 4 samples surface points from its own module RNG). Per-mesh reset also makes a mesh's + result independent of the other meshes in the call. +- **No animation baked** (`animation_file=None`; the UI default is "Standard Run.fbx"). +- **app_v2.py is unpatched.** dread-dev's CPU patch only upcasts to fp32 on CPU; the GPU runs + upstream's fp16 autocast path. + +## Acceptance (2026-10-01) — instruments and raw numbers in `acceptance/` + +Harness: fv-ml1 GPU 3 (RTX PRO 6000 Blackwell Max-Q, otherwise idle at 2 MiB), image +`local/mia:0.1.0`, four meshes (`dreadnaught/art/models/{goblin,villager,captain,ogre}.glb`, +2.7k–3.4k verts), three seeded calls with the mesh order rotated, plus one unseeded call. + +| mesh | GPU pipeline s, median of 3 (spread) | model part s | A-pose-hint export s | CPU pipeline s (dread-dev, median of 3) | +|---|---|---|---|---| +| goblin | **4.53** (4.43–4.57) | 1.88 | 1.27 | 160.8 | +| villager | **3.97** (3.93–3.99) | 1.61 | 1.24 | 116.9 | +| captain | **3.90** (3.79–3.94) | 1.65 | 1.24 | 121.6 | +| ogre | **4.72** (4.45–4.80) | 2.08 | 1.35 | 124.7 | + +- **Model load:** 12.7–13.0 s per call (4 calls), paid once per call. End to end, a 4-mesh call + takes ~50 s including ssh, rsync and container start; a 1-mesh call ~31 s. +- **Peak VRAM:** 3,394 MiB on GPU 3 as nvidia-smi saw it (100 ms polling over all four calls, + includes the CUDA context); torch's peak reserved was 2,712–2,732 MiB per mesh. +- **Repeatable:** the three seeded calls are **bit-identical** for every mesh (max abs diff 0.0 on + joints, tails, weights, weights_effective, pose), despite the rotated mesh order. +- **Matches CPU within the noise.** Distances use dread-dev's `noise_floor.py` metrics (joint + distance / height, weight mass moved, dominant bone changed). GPU-vs-CPU averages 0.83–1.27× the + CPU-vs-CPU distance, per mesh and metric. Positive control: one unseeded GPU call against a + seeded one, i.e. pure sampling noise, lands at 0.80–1.83× — same size, so the instrument sees + sampling noise and the GPU path adds nothing visible on top of it. 24 of 72 GPU-vs-CPU pair + values exceed the CPU-vs-CPU maximum, but that maximum comes from only 3 pairs per mesh, and the + sampling-only control overshoots it the same way. +- **Sensitivity floor:** this comparison cannot resolve a systematic GPU-vs-CPU offset (fp16 vs + fp32) smaller than the sampling noise, ~0.2% of mesh height for joints and ~2% weight mass moved. + A seeded CPU run would isolate it; not done (would cost ~8 min of nh3-dev CPU). + +## Gotchas + +- `init_blocks()` reads `data/examples/log.csv` even headless; the image symlinks `data/examples` + into the HF snapshot. Without it the driver dies before the first mesh. +- uv needs `UV_LINK_MODE=copy` in the build (reflink clone fails on overlayfs, "os error 22"). +- The image is 10.5 GB on fv-ml1's zroot, which sat at 85% after the build. Check free space before + building a new tag, and remove the old tag after. +- GPU 3 is borrowed from the vLLM reserve and shared on demand with Blender and Scriberr. MIA needs + ~3.4 GB, so it coexists with both; if a full-size seat claims the card, this tool has to move. + +## Rebuild + +```bash +# on nh3-dev +ssh infra-ops@10.251.50.54 'sudo -n install -d -o infra-ops -g root /opt/docker/src/mia-' +rsync -a stacks/mia/{Dockerfile,requirements.lock,mia_driver.py} infra-ops@10.251.50.54:/opt/docker/src/mia-/ +ssh infra-ops@10.251.50.54 'cd /opt/docker/src/mia- && docker build -t local/mia: .' +# then point fv-ml1:/opt/docker/compose/mia/.env IMAGE= at the new tag +``` diff --git a/stacks/mia/acceptance/acceptance.json b/stacks/mia/acceptance/acceptance.json new file mode 100644 index 0000000..5bf6c7d --- /dev/null +++ b/stacks/mia/acceptance/acceptance.json @@ -0,0 +1,394 @@ +{ + "goblin": { + "gpu_pipeline_s": [ + 4.569874900858849, + 4.53466919856146, + 4.427527715917677 + ], + "gpu_model_s": [ + 1.9738918957300484, + 1.881500325864181, + 1.8626845581457019 + ], + "apose_hint_s": [ + 1.190177327953279, + 1.4380995847750455, + 1.2671978930011392 + ], + "cpu_pipeline_s": [ + 119.3426258880063, + 708.4532103899983, + 160.76836042600917 + ], + "peak_reserved_mib": [ + 2712.0, + 2732.0, + 2732.0, + 2712.0 + ], + "seeded_maxdiff": { + "joints_head": 0.0, + "joints_tail": 0.0, + "weights": 0.0, + "weights_effective": 0.0, + "pose_to_rest": 0.0 + }, + "cpu_cpu": [ + { + "jh_med": 0.002404872328042984, + "jh_max": 0.009904514066874981, + "jt_max": 0.010788610205054283, + "w_moved_mean": 0.01768580637872219, + "w_moved_p95": 0.06185286119580269, + "dom_changed": 0.02601522842639594 + }, + { + "jh_med": 0.0025397357530891895, + "jh_max": 0.009374394081532955, + "jt_max": 0.010417704470455647, + "w_moved_mean": 0.02369445003569126, + "w_moved_p95": 0.07524719834327698, + "dom_changed": 0.03140862944162436 + }, + { + "jh_med": 0.0021242855582386255, + "jh_max": 0.005268011707812548, + "jt_max": 0.0065661026164889336, + "w_moved_mean": 0.01971343532204628, + "w_moved_p95": 0.06007026135921478, + "dom_changed": 0.022208121827411168 + } + ], + "gpu_cpu": [ + { + "jh_med": 0.002763624768704176, + "jh_max": 0.01039000041782856, + "jt_max": 0.011829185299575329, + "w_moved_mean": 0.025863967835903168, + "w_moved_p95": 0.083327516913414, + "dom_changed": 0.031725888324873094 + }, + { + "jh_med": 0.0024375617504119873, + "jh_max": 0.006176925264298916, + "jt_max": 0.006694309413433075, + "w_moved_mean": 0.023365769535303116, + "w_moved_p95": 0.07489541918039322, + "dom_changed": 0.02728426395939086 + }, + { + "jh_med": 0.0030529131181538105, + "jh_max": 0.00709641445428133, + "jt_max": 0.007498544175177813, + "w_moved_mean": 0.020390696823596954, + "w_moved_p95": 0.06874266266822815, + "dom_changed": 0.02950507614213198 + } + ], + "unseeded_vs_seeded": [ + { + "jh_med": 0.0029595012310892344, + "jh_max": 0.007351752370595932, + "jt_max": 0.010119971819221973, + "w_moved_mean": 0.022764530032873154, + "w_moved_p95": 0.06456947326660156, + "dom_changed": 0.030139593908629442 + } + ], + "same_geometry": true + }, + "villager": { + "gpu_pipeline_s": [ + 3.9700973057188094, + 3.9323174457531422, + 3.9949949418660253 + ], + "gpu_model_s": [ + 1.6040664857719094, + 1.7117067980580032, + 1.6055225860327482 + ], + "apose_hint_s": [ + 1.277365405112505, + 1.0884231911040843, + 1.2362516748253256 + ], + "cpu_pipeline_s": [ + 114.40334213199094, + 116.87012534309179, + 118.42003130697412 + ], + "peak_reserved_mib": [ + 2732.0, + 2712.0, + 2732.0, + 2732.0 + ], + "seeded_maxdiff": { + "joints_head": 0.0, + "joints_tail": 0.0, + "weights": 0.0, + "weights_effective": 0.0, + "pose_to_rest": 0.0 + }, + "cpu_cpu": [ + { + "jh_med": 0.0013478414621204138, + "jh_max": 0.003135952167212963, + "jt_max": 0.004719415679574013, + "w_moved_mean": 0.015827687457203865, + "w_moved_p95": 0.047268014401197433, + "dom_changed": 0.0183884988298228 + }, + { + "jh_med": 0.0016320310533046722, + "jh_max": 0.004904848523437977, + "jt_max": 0.0048230430111289024, + "w_moved_mean": 0.014390517957508564, + "w_moved_p95": 0.04615660756826401, + "dom_changed": 0.01671681711802073 + }, + { + "jh_med": 0.0015891867224127054, + "jh_max": 0.0036788983270525932, + "jt_max": 0.005405386444181204, + "w_moved_mean": 0.01590096764266491, + "w_moved_p95": 0.050222158432006836, + "dom_changed": 0.0183884988298228 + } + ], + "gpu_cpu": [ + { + "jh_med": 0.001549383858218789, + "jh_max": 0.002919533057138324, + "jt_max": 0.007295706775039434, + "w_moved_mean": 0.015061692334711552, + "w_moved_p95": 0.04662124067544937, + "dom_changed": 0.01805416248746239 + }, + { + "jh_med": 0.0012639821507036686, + "jh_max": 0.004058246500790119, + "jt_max": 0.005211320705711842, + "w_moved_mean": 0.016790034249424934, + "w_moved_p95": 0.050929658114910126, + "dom_changed": 0.01905717151454363 + }, + { + "jh_med": 0.0015581324696540833, + "jh_max": 0.0054196459241211414, + "jt_max": 0.005114810075610876, + "w_moved_mean": 0.01441498938947916, + "w_moved_p95": 0.046087272465229034, + "dom_changed": 0.014042126379137413 + } + ], + "unseeded_vs_seeded": [ + { + "jh_med": 0.0020237602293491364, + "jh_max": 0.00419682078063488, + "jt_max": 0.005229325499385595, + "w_moved_mean": 0.015298573300242424, + "w_moved_p95": 0.044845279306173325, + "dom_changed": 0.01905717151454363 + } + ], + "same_geometry": true + }, + "captain": { + "gpu_pipeline_s": [ + 3.902754598064348, + 3.792409762972966, + 3.9422666197642684 + ], + "gpu_model_s": [ + 1.637031597085297, + 1.6480189089197665, + 1.741625044029206 + ], + "apose_hint_s": [ + 1.3538039769046009, + 1.244770233053714, + 1.139452091883868 + ], + "cpu_pipeline_s": [ + 125.6441692209919, + 121.16770973894745, + 121.56508291099453 + ], + "peak_reserved_mib": [ + 2732.0, + 2732.0, + 2712.0, + 2732.0 + ], + "seeded_maxdiff": { + "joints_head": 0.0, + "joints_tail": 0.0, + "weights": 0.0, + "weights_effective": 0.0, + "pose_to_rest": 0.0 + }, + "cpu_cpu": [ + { + "jh_med": 0.0017565949819982052, + "jh_max": 0.00641083437949419, + "jt_max": 0.0061119236052036285, + "w_moved_mean": 0.01863820292055607, + "w_moved_p95": 0.06350292265415192, + "dom_changed": 0.021580102414045354 + }, + { + "jh_med": 0.0020125731825828552, + "jh_max": 0.003998186439275742, + "jt_max": 0.006467505358159542, + "w_moved_mean": 0.017755920067429543, + "w_moved_p95": 0.05389416217803955, + "dom_changed": 0.019385515727871252 + }, + { + "jh_med": 0.0018191682174801826, + "jh_max": 0.0042957765981554985, + "jt_max": 0.0039120386354625225, + "w_moved_mean": 0.02013971656560898, + "w_moved_p95": 0.057384975254535675, + "dom_changed": 0.019385515727871252 + } + ], + "gpu_cpu": [ + { + "jh_med": 0.001694112434051931, + "jh_max": 0.0057935346849262714, + "jt_max": 0.004407331347465515, + "w_moved_mean": 0.01909082941710949, + "w_moved_p95": 0.06022503599524498, + "dom_changed": 0.016093635698610095 + }, + { + "jh_med": 0.0016945323441177607, + "jh_max": 0.005641191732138395, + "jt_max": 0.006225286982953548, + "w_moved_mean": 0.018217289820313454, + "w_moved_p95": 0.0654626339673996, + "dom_changed": 0.019019751280175568 + }, + { + "jh_med": 0.0017832419835031033, + "jh_max": 0.003895824309438467, + "jt_max": 0.00681948009878397, + "w_moved_mean": 0.018107891082763672, + "w_moved_p95": 0.05214930698275566, + "dom_changed": 0.023043160204828092 + } + ], + "unseeded_vs_seeded": [ + { + "jh_med": 0.001529946457594633, + "jh_max": 0.003934196196496487, + "jt_max": 0.005103697534650564, + "w_moved_mean": 0.022645380347967148, + "w_moved_p95": 0.07639797031879425, + "dom_changed": 0.024140453547915143 + } + ], + "same_geometry": true + }, + "ogre": { + "gpu_pipeline_s": [ + 4.71703689894639, + 4.802805577637628, + 4.453627999639139 + ], + "gpu_model_s": [ + 2.076565559953451, + 2.098935986869037, + 1.9066526568494737 + ], + "apose_hint_s": [ + 1.3545660751406103, + 1.3703913709614426, + 1.3345670951530337 + ], + "cpu_pipeline_s": [ + 123.53674800007138, + 124.70388515893137, + 336.82813009101665 + ], + "peak_reserved_mib": [ + 2732.0, + 2732.0, + 2732.0, + 2732.0 + ], + "seeded_maxdiff": { + "joints_head": 0.0, + "joints_tail": 0.0, + "weights": 0.0, + "weights_effective": 0.0, + "pose_to_rest": 0.0 + }, + "cpu_cpu": [ + { + "jh_med": 0.002341177314519882, + "jh_max": 0.004295716993510723, + "jt_max": 0.010711956769227982, + "w_moved_mean": 0.019047463312745094, + "w_moved_p95": 0.05714641511440277, + "dom_changed": 0.0243612596553773 + }, + { + "jh_med": 0.0027327858842909336, + "jh_max": 0.007534473203122616, + "jt_max": 0.011530852876603603, + "w_moved_mean": 0.021840794011950493, + "w_moved_p95": 0.06544578820466995, + "dom_changed": 0.024955436720142603 + }, + { + "jh_med": 0.0020891886670142412, + "jh_max": 0.007218110840767622, + "jt_max": 0.007911363616585732, + "w_moved_mean": 0.021831553429365158, + "w_moved_p95": 0.06872576475143433, + "dom_changed": 0.023767082590612002 + } + ], + "gpu_cpu": [ + { + "jh_med": 0.002296903170645237, + "jh_max": 0.005948834586888552, + "jt_max": 0.007127378601580858, + "w_moved_mean": 0.023190144449472427, + "w_moved_p95": 0.06846748292446136, + "dom_changed": 0.029411764705882353 + }, + { + "jh_med": 0.0035880799405276775, + "jh_max": 0.0077993483282625675, + "jt_max": 0.009203977882862091, + "w_moved_mean": 0.02328675426542759, + "w_moved_p95": 0.07298485934734344, + "dom_changed": 0.030303030303030304 + }, + { + "jh_med": 0.0032017994672060013, + "jh_max": 0.006755590904504061, + "jt_max": 0.008556965738534927, + "w_moved_mean": 0.020125865936279297, + "w_moved_p95": 0.05771474540233612, + "dom_changed": 0.026737967914438502 + } + ], + "unseeded_vs_seeded": [ + { + "jh_med": 0.0030766851268708706, + "jh_max": 0.01164184045046568, + "jt_max": 0.013770738616585732, + "w_moved_mean": 0.023339038714766502, + "w_moved_p95": 0.06661652028560638, + "dom_changed": 0.035056446821152706 + } + ], + "same_geometry": true + } +} \ No newline at end of file diff --git a/stacks/mia/acceptance/compare.py b/stacks/mia/acceptance/compare.py new file mode 100644 index 0000000..cf6f6f6 --- /dev/null +++ b/stacks/mia/acceptance/compare.py @@ -0,0 +1,55 @@ +"""MIA GPU acceptance analysis: timings (median of 3 seeded runs), determinism, and GPU-vs-CPU +agreement measured with dread-dev's noise_floor.py metrics against the CPU run-to-run floor.""" +import itertools, json, statistics as st, sys, numpy as np +A, C = sys.argv[1], sys.argv[2] +MESHES = ["goblin", "villager", "captain", "ogre"] + +def load(p): + d = np.load(p) + if "weights_effective" not in d: + raise SystemExit(f"{p}: no weights_effective") + return d + +def metrics(Ar, Br): + H = float(np.ptp(Ar["verts"][:, 1])) + dj = np.linalg.norm(Ar["joints_head"] - Br["joints_head"], axis=1) / H + dt = np.linalg.norm(Ar["joints_tail"] - Br["joints_tail"], axis=1) / H + WA, WB = Ar["weights_effective"], Br["weights_effective"] + moved = 0.5 * np.abs(WA - WB).sum(1) + return dict(jh_med=float(np.median(dj)), jh_max=float(dj.max()), jt_max=float(dt.max()), + w_moved_mean=float(moved.mean()), w_moved_p95=float(np.percentile(moved, 95)), + dom_changed=float((WA.argmax(1) != WB.argmax(1)).mean())) + +def rng(ms, k): + v = [m[k] for m in ms] + return f"{min(v):.4f}..{max(v):.4f}" + +KEYS = ["jh_med", "jh_max", "jt_max", "w_moved_mean", "w_moved_p95", "dom_changed"] +out = {} +for n in MESHES: + g = {r: load(f"{A}/{r}/out/{n}_pred.npz") for r in ("r1", "r2", "r3", "u1")} + c = [load(f"{C}/{n}_pred.npz"), load(f"{C}/repeats/{n}_rep2_pred.npz"), load(f"{C}/repeats/{n}_rep3_pred.npz")] + runs = {r: json.load(open(f"{A}/{r}/out/{n}_run.json")) for r in ("r1", "r2", "r3", "u1")} + same_geom = all(np.array_equal(g["r1"]["verts"], x["verts"]) and np.array_equal(g["r1"]["faces"], x["faces"]) for x in c) + # determinism across the 3 seeded runs (different mesh orders): exact? + det = {k: max(float(np.abs(g["r1"][k] - g[r][k]).max()) for r in ("r2", "r3")) + for k in ("joints_head", "joints_tail", "weights", "weights_effective", "pose_to_rest")} + cpu_cpu = [metrics(a, b) for a, b in itertools.combinations(c, 2)] # the CPU noise floor (null) + gpu_cpu = [metrics(g["r1"], x) for x in c] # the claim under test + unseeded = [metrics(g["u1"], g[r]) for r in ("r1",)] # positive control: must move + t = [runs[r]["pipeline_total_wall_s"] for r in ("r1", "r2", "r3")] + tm = [runs[r]["model_total_wall_s"] for r in ("r1", "r2", "r3")] + ta = [runs[r]["timings"]["vis_blender_apose_hint"]["wall_s"] for r in ("r1", "r2", "r3")] + pk = [runs[r]["gpu_peak_reserved_mib"] for r in ("r1", "r2", "r3", "u1")] + cpu_t = [json.load(open(p))["pipeline_total_wall_s"] for p in (f"{C}/{n}_run.json", f"{C}/repeats/{n}_rep2_run.json", f"{C}/repeats/{n}_rep3_run.json")] + print(f"\n== {n}: verts {runs['r1']['n_verts']}, same geometry as CPU: {same_geom}") + print(f" GPU pipeline s: {[round(x,2) for x in t]} median {st.median(t):.2f} | model part median {st.median(tm):.2f} | A-pose-hint export median {st.median(ta):.2f} | CPU pipeline median {st.median(cpu_t):.1f}") + print(f" peak torch reserved MiB: {pk}") + print(f" seeded r1/r2/r3 max abs diff: " + ", ".join(f"{k} {v:.2e}" for k, v in det.items())) + for k in KEYS: + print(f" {k:13s} CPU-vs-CPU {rng(cpu_cpu,k)} | GPU-vs-CPU {rng(gpu_cpu,k)} | unseeded-vs-seeded GPU {rng(unseeded,k)}") + out[n] = dict(gpu_pipeline_s=t, gpu_model_s=tm, apose_hint_s=ta, cpu_pipeline_s=cpu_t, peak_reserved_mib=pk, + seeded_maxdiff=det, cpu_cpu=cpu_cpu, gpu_cpu=gpu_cpu, unseeded_vs_seeded=unseeded, same_geometry=same_geom) +init = [json.load(open(f"{A}/{r}/out/goblin_run.json"))["init_wall_s"] for r in ("r1", "r2", "r3", "u1")] +print("\nmodel load (init) s per container:", [round(x, 1) for x in init]) +json.dump(out, open(f"{A}/acceptance.json", "w"), indent=1) diff --git a/stacks/mia/acceptance/run-all.sh b/stacks/mia/acceptance/run-all.sh new file mode 100755 index 0000000..faa69ff --- /dev/null +++ b/stacks/mia/acceptance/run-all.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# MIA GPU acceptance: 3 seeded runs (mesh order rotated) + 1 unseeded run, GPU 3 memory polled throughout. +set -uo pipefail +# Run from a scratch dir holding r1/ r2/ r3/ u1/, each with in/{goblin,villager,captain,ogre}.glb; +# then: python3 compare.py . +A=$(dirname "$(realpath "$0")") +cd ~/development/eshpfi-management +H=infra-ops@10.251.50.54 +POLLPID=$(ssh -n $H 'nohup nvidia-smi -i 3 --query-gpu=timestamp,memory.used --format=csv,noheader,nounits -lms 100 > /tank/mia/vram-accept.csv 2>/dev/null & echo $!') +echo "poller pid $POLLPID" +run() { local r=$1; shift; scripts/mia-run --job "$A/$r" "$@" > "$A/$r.log" 2>&1; local rc=$?; echo "$r rc=$rc"; } +run r1 -- goblin=in/goblin.glb villager=in/villager.glb captain=in/captain.glb ogre=in/ogre.glb +run r2 -- villager=in/villager.glb captain=in/captain.glb ogre=in/ogre.glb goblin=in/goblin.glb +run r3 -- captain=in/captain.glb ogre=in/ogre.glb goblin=in/goblin.glb villager=in/villager.glb +run u1 --unseeded -- goblin=in/goblin.glb villager=in/villager.glb captain=in/captain.glb ogre=in/ogre.glb +ssh -n $H "kill $POLLPID" +rsync -a $H:/tank/mia/vram-accept.csv "$A/" diff --git a/stacks/mia/acceptance/vram-accept.csv b/stacks/mia/acceptance/vram-accept.csv new file mode 100644 index 0000000..8058ea5 --- /dev/null +++ b/stacks/mia/acceptance/vram-accept.csv @@ -0,0 +1,2018 @@ +2026/10/01 09:39:49.511, 2 +2026/10/01 09:39:49.611, 2 +2026/10/01 09:39:49.711, 2 +2026/10/01 09:39:49.811, 2 +2026/10/01 09:39:49.912, 2 +2026/10/01 09:39:50.012, 2 +2026/10/01 09:39:50.112, 2 +2026/10/01 09:39:50.212, 2 +2026/10/01 09:39:50.313, 2 +2026/10/01 09:39:50.413, 2 +2026/10/01 09:39:50.513, 2 +2026/10/01 09:39:50.614, 2 +2026/10/01 09:39:50.714, 2 +2026/10/01 09:39:50.814, 2 +2026/10/01 09:39:50.914, 2 +2026/10/01 09:39:51.015, 2 +2026/10/01 09:39:51.115, 2 +2026/10/01 09:39:51.215, 2 +2026/10/01 09:39:51.316, 2 +2026/10/01 09:39:51.416, 2 +2026/10/01 09:39:51.516, 2 +2026/10/01 09:39:51.617, 2 +2026/10/01 09:39:51.717, 2 +2026/10/01 09:39:51.817, 2 +2026/10/01 09:39:51.918, 2 +2026/10/01 09:39:52.018, 2 +2026/10/01 09:39:52.118, 2 +2026/10/01 09:39:52.219, 2 +2026/10/01 09:39:52.319, 2 +2026/10/01 09:39:52.419, 2 +2026/10/01 09:39:52.519, 2 +2026/10/01 09:39:52.620, 2 +2026/10/01 09:39:52.720, 2 +2026/10/01 09:39:52.820, 2 +2026/10/01 09:39:52.920, 2 +2026/10/01 09:39:53.021, 2 +2026/10/01 09:39:53.121, 2 +2026/10/01 09:39:53.221, 2 +2026/10/01 09:39:53.322, 2 +2026/10/01 09:39:53.422, 2 +2026/10/01 09:39:53.522, 2 +2026/10/01 09:39:53.622, 2 +2026/10/01 09:39:53.723, 2 +2026/10/01 09:39:53.823, 2 +2026/10/01 09:39:53.923, 2 +2026/10/01 09:39:54.024, 2 +2026/10/01 09:39:54.124, 2 +2026/10/01 09:39:54.224, 2 +2026/10/01 09:39:54.324, 2 +2026/10/01 09:39:54.425, 2 +2026/10/01 09:39:54.525, 2 +2026/10/01 09:39:54.625, 2 +2026/10/01 09:39:54.726, 2 +2026/10/01 09:39:54.826, 2 +2026/10/01 09:39:54.926, 2 +2026/10/01 09:39:55.026, 2 +2026/10/01 09:39:55.127, 2 +2026/10/01 09:39:55.227, 2 +2026/10/01 09:39:55.327, 2 +2026/10/01 09:39:55.428, 2 +2026/10/01 09:39:55.528, 2 +2026/10/01 09:39:55.628, 2 +2026/10/01 09:39:55.729, 2 +2026/10/01 09:39:55.829, 2 +2026/10/01 09:39:55.929, 2 +2026/10/01 09:39:56.029, 2 +2026/10/01 09:39:56.130, 2 +2026/10/01 09:39:56.230, 2 +2026/10/01 09:39:56.330, 2 +2026/10/01 09:39:56.431, 2 +2026/10/01 09:39:56.531, 2 +2026/10/01 09:39:56.631, 2 +2026/10/01 09:39:56.731, 2 +2026/10/01 09:39:56.832, 2 +2026/10/01 09:39:56.932, 2 +2026/10/01 09:39:57.032, 2 +2026/10/01 09:39:57.133, 2 +2026/10/01 09:39:57.233, 2 +2026/10/01 09:39:57.333, 2 +2026/10/01 09:39:57.434, 2 +2026/10/01 09:39:57.534, 2 +2026/10/01 09:39:57.634, 2 +2026/10/01 09:39:57.735, 2 +2026/10/01 09:39:57.835, 2 +2026/10/01 09:39:57.935, 2 +2026/10/01 09:39:58.036, 2 +2026/10/01 09:39:58.136, 2 +2026/10/01 09:39:58.236, 2 +2026/10/01 09:39:58.337, 2 +2026/10/01 09:39:58.437, 2 +2026/10/01 09:39:58.537, 2 +2026/10/01 09:39:58.638, 2 +2026/10/01 09:39:58.738, 2 +2026/10/01 09:39:58.838, 2 +2026/10/01 09:39:58.938, 2 +2026/10/01 09:39:59.039, 2 +2026/10/01 09:39:59.139, 2 +2026/10/01 09:39:59.239, 2 +2026/10/01 09:39:59.340, 2 +2026/10/01 09:39:59.440, 2 +2026/10/01 09:39:59.540, 2 +2026/10/01 09:39:59.641, 2 +2026/10/01 09:39:59.741, 2 +2026/10/01 09:39:59.841, 2 +2026/10/01 09:39:59.941, 2 +2026/10/01 09:40:00.042, 2 +2026/10/01 09:40:00.142, 2 +2026/10/01 09:40:00.242, 2 +2026/10/01 09:40:00.342, 2 +2026/10/01 09:40:00.443, 2 +2026/10/01 09:40:00.543, 2 +2026/10/01 09:40:00.643, 2 +2026/10/01 09:40:00.744, 2 +2026/10/01 09:40:00.844, 2 +2026/10/01 09:40:00.944, 2 +2026/10/01 09:40:01.045, 2 +2026/10/01 09:40:01.145, 2 +2026/10/01 09:40:01.245, 2 +2026/10/01 09:40:01.346, 2 +2026/10/01 09:40:01.446, 2 +2026/10/01 09:40:01.546, 2 +2026/10/01 09:40:01.647, 2 +2026/10/01 09:40:01.747, 2 +2026/10/01 09:40:01.847, 2 +2026/10/01 09:40:01.947, 2 +2026/10/01 09:40:02.048, 2 +2026/10/01 09:40:02.148, 2 +2026/10/01 09:40:02.248, 2 +2026/10/01 09:40:02.349, 2 +2026/10/01 09:40:02.449, 2 +2026/10/01 09:40:02.586, 4 +2026/10/01 09:40:02.686, 4 +2026/10/01 09:40:02.787, 4 +2026/10/01 09:40:02.887, 4 +2026/10/01 09:40:02.987, 4 +2026/10/01 09:40:03.088, 4 +2026/10/01 09:40:03.188, 4 +2026/10/01 09:40:03.288, 4 +2026/10/01 09:40:03.389, 4 +2026/10/01 09:40:03.489, 4 +2026/10/01 09:40:03.589, 4 +2026/10/01 09:40:03.689, 4 +2026/10/01 09:40:03.790, 4 +2026/10/01 09:40:03.890, 4 +2026/10/01 09:40:03.990, 4 +2026/10/01 09:40:04.091, 4 +2026/10/01 09:40:04.191, 4 +2026/10/01 09:40:04.291, 4 +2026/10/01 09:40:04.392, 4 +2026/10/01 09:40:04.492, 4 +2026/10/01 09:40:04.592, 4 +2026/10/01 09:40:04.693, 4 +2026/10/01 09:40:04.793, 4 +2026/10/01 09:40:04.893, 4 +2026/10/01 09:40:04.994, 4 +2026/10/01 09:40:05.094, 4 +2026/10/01 09:40:05.194, 4 +2026/10/01 09:40:05.295, 23 +2026/10/01 09:40:05.395, 1002 +2026/10/01 09:40:05.496, 1222 +2026/10/01 09:40:05.596, 1222 +2026/10/01 09:40:05.696, 1222 +2026/10/01 09:40:05.797, 1222 +2026/10/01 09:40:05.897, 1222 +2026/10/01 09:40:05.997, 1222 +2026/10/01 09:40:06.098, 1222 +2026/10/01 09:40:06.198, 1222 +2026/10/01 09:40:06.298, 1222 +2026/10/01 09:40:06.399, 1222 +2026/10/01 09:40:06.499, 1222 +2026/10/01 09:40:06.599, 1222 +2026/10/01 09:40:06.700, 1222 +2026/10/01 09:40:06.800, 1864 +2026/10/01 09:40:06.900, 1864 +2026/10/01 09:40:07.001, 1864 +2026/10/01 09:40:07.101, 1864 +2026/10/01 09:40:07.201, 1864 +2026/10/01 09:40:07.302, 1864 +2026/10/01 09:40:07.402, 1864 +2026/10/01 09:40:07.502, 1864 +2026/10/01 09:40:07.602, 1864 +2026/10/01 09:40:07.703, 1864 +2026/10/01 09:40:07.803, 1864 +2026/10/01 09:40:07.903, 1864 +2026/10/01 09:40:08.004, 1864 +2026/10/01 09:40:08.104, 2024 +2026/10/01 09:40:08.204, 2506 +2026/10/01 09:40:08.305, 2506 +2026/10/01 09:40:08.405, 2506 +2026/10/01 09:40:08.505, 2506 +2026/10/01 09:40:08.605, 2506 +2026/10/01 09:40:08.706, 2506 +2026/10/01 09:40:08.806, 2506 +2026/10/01 09:40:08.906, 2506 +2026/10/01 09:40:09.007, 2598 +2026/10/01 09:40:09.107, 2600 +2026/10/01 09:40:09.207, 2600 +2026/10/01 09:40:09.307, 2632 +2026/10/01 09:40:09.408, 3372 +2026/10/01 09:40:09.508, 3372 +2026/10/01 09:40:09.608, 3372 +2026/10/01 09:40:09.708, 3372 +2026/10/01 09:40:09.809, 3372 +2026/10/01 09:40:09.909, 3372 +2026/10/01 09:40:10.009, 3372 +2026/10/01 09:40:10.110, 3372 +2026/10/01 09:40:10.210, 3372 +2026/10/01 09:40:10.310, 3372 +2026/10/01 09:40:10.411, 3372 +2026/10/01 09:40:10.511, 3372 +2026/10/01 09:40:10.611, 3374 +2026/10/01 09:40:10.711, 3374 +2026/10/01 09:40:10.812, 3374 +2026/10/01 09:40:10.912, 3374 +2026/10/01 09:40:11.012, 3374 +2026/10/01 09:40:11.113, 3374 +2026/10/01 09:40:11.213, 3374 +2026/10/01 09:40:11.313, 3374 +2026/10/01 09:40:11.413, 3374 +2026/10/01 09:40:11.513, 3374 +2026/10/01 09:40:11.614, 3374 +2026/10/01 09:40:11.714, 3374 +2026/10/01 09:40:11.814, 3374 +2026/10/01 09:40:11.914, 3374 +2026/10/01 09:40:12.015, 3374 +2026/10/01 09:40:12.115, 3374 +2026/10/01 09:40:12.215, 3374 +2026/10/01 09:40:12.316, 3374 +2026/10/01 09:40:12.416, 3374 +2026/10/01 09:40:12.516, 3374 +2026/10/01 09:40:12.616, 3374 +2026/10/01 09:40:12.717, 3374 +2026/10/01 09:40:12.817, 3374 +2026/10/01 09:40:12.917, 3374 +2026/10/01 09:40:13.017, 3374 +2026/10/01 09:40:13.118, 2628 +2026/10/01 09:40:13.218, 2628 +2026/10/01 09:40:13.318, 2628 +2026/10/01 09:40:13.419, 2628 +2026/10/01 09:40:13.519, 2628 +2026/10/01 09:40:13.619, 2628 +2026/10/01 09:40:13.720, 2628 +2026/10/01 09:40:13.820, 2628 +2026/10/01 09:40:13.920, 2628 +2026/10/01 09:40:14.021, 2628 +2026/10/01 09:40:14.121, 2628 +2026/10/01 09:40:14.221, 2628 +2026/10/01 09:40:14.322, 2628 +2026/10/01 09:40:14.422, 2628 +2026/10/01 09:40:14.522, 2628 +2026/10/01 09:40:14.622, 2628 +2026/10/01 09:40:14.723, 2628 +2026/10/01 09:40:14.823, 2634 +2026/10/01 09:40:14.924, 2634 +2026/10/01 09:40:15.024, 2634 +2026/10/01 09:40:15.124, 3392 +2026/10/01 09:40:15.225, 3392 +2026/10/01 09:40:15.325, 3392 +2026/10/01 09:40:15.425, 3392 +2026/10/01 09:40:15.526, 3392 +2026/10/01 09:40:15.626, 3392 +2026/10/01 09:40:15.726, 3392 +2026/10/01 09:40:15.826, 3392 +2026/10/01 09:40:15.927, 3394 +2026/10/01 09:40:16.027, 3394 +2026/10/01 09:40:16.127, 3394 +2026/10/01 09:40:16.228, 3394 +2026/10/01 09:40:16.328, 3394 +2026/10/01 09:40:16.428, 3394 +2026/10/01 09:40:16.529, 3394 +2026/10/01 09:40:16.629, 3394 +2026/10/01 09:40:16.729, 3394 +2026/10/01 09:40:16.830, 3394 +2026/10/01 09:40:16.930, 3394 +2026/10/01 09:40:17.030, 3394 +2026/10/01 09:40:17.130, 3394 +2026/10/01 09:40:17.231, 3394 +2026/10/01 09:40:17.331, 3394 +2026/10/01 09:40:17.431, 3394 +2026/10/01 09:40:17.532, 3394 +2026/10/01 09:40:17.632, 3394 +2026/10/01 09:40:17.732, 3394 +2026/10/01 09:40:17.832, 3394 +2026/10/01 09:40:17.933, 3394 +2026/10/01 09:40:18.033, 3394 +2026/10/01 09:40:18.133, 3394 +2026/10/01 09:40:18.234, 3394 +2026/10/01 09:40:18.334, 2628 +2026/10/01 09:40:18.434, 2628 +2026/10/01 09:40:18.535, 2628 +2026/10/01 09:40:18.635, 2628 +2026/10/01 09:40:18.735, 2628 +2026/10/01 09:40:18.835, 2628 +2026/10/01 09:40:18.936, 2628 +2026/10/01 09:40:19.036, 2628 +2026/10/01 09:40:19.136, 2628 +2026/10/01 09:40:19.237, 2628 +2026/10/01 09:40:19.337, 2628 +2026/10/01 09:40:19.437, 2628 +2026/10/01 09:40:19.538, 2628 +2026/10/01 09:40:19.638, 2628 +2026/10/01 09:40:19.738, 2628 +2026/10/01 09:40:19.839, 2628 +2026/10/01 09:40:19.939, 2628 +2026/10/01 09:40:20.039, 2628 +2026/10/01 09:40:20.139, 2634 +2026/10/01 09:40:20.240, 2634 +2026/10/01 09:40:20.340, 3392 +2026/10/01 09:40:20.440, 3392 +2026/10/01 09:40:20.541, 3392 +2026/10/01 09:40:20.641, 3392 +2026/10/01 09:40:20.741, 3392 +2026/10/01 09:40:20.842, 3392 +2026/10/01 09:40:20.942, 3392 +2026/10/01 09:40:21.042, 3392 +2026/10/01 09:40:21.143, 3392 +2026/10/01 09:40:21.243, 3394 +2026/10/01 09:40:21.343, 3394 +2026/10/01 09:40:21.443, 3394 +2026/10/01 09:40:21.544, 3394 +2026/10/01 09:40:21.644, 3394 +2026/10/01 09:40:21.744, 3394 +2026/10/01 09:40:21.845, 3394 +2026/10/01 09:40:21.945, 3394 +2026/10/01 09:40:22.045, 3394 +2026/10/01 09:40:22.146, 3394 +2026/10/01 09:40:22.246, 3394 +2026/10/01 09:40:22.346, 3394 +2026/10/01 09:40:22.447, 3394 +2026/10/01 09:40:22.547, 3394 +2026/10/01 09:40:22.647, 3394 +2026/10/01 09:40:22.748, 3394 +2026/10/01 09:40:22.848, 3394 +2026/10/01 09:40:22.948, 3394 +2026/10/01 09:40:23.049, 3394 +2026/10/01 09:40:23.149, 3394 +2026/10/01 09:40:23.249, 3394 +2026/10/01 09:40:23.349, 3394 +2026/10/01 09:40:23.450, 3394 +2026/10/01 09:40:23.550, 2628 +2026/10/01 09:40:23.650, 2628 +2026/10/01 09:40:23.751, 2628 +2026/10/01 09:40:23.851, 2628 +2026/10/01 09:40:23.951, 2628 +2026/10/01 09:40:24.052, 2628 +2026/10/01 09:40:24.152, 2628 +2026/10/01 09:40:24.252, 2628 +2026/10/01 09:40:24.353, 2628 +2026/10/01 09:40:24.453, 2628 +2026/10/01 09:40:24.553, 2628 +2026/10/01 09:40:24.653, 2628 +2026/10/01 09:40:24.754, 2628 +2026/10/01 09:40:24.854, 2628 +2026/10/01 09:40:24.954, 2628 +2026/10/01 09:40:25.055, 2628 +2026/10/01 09:40:25.155, 2628 +2026/10/01 09:40:25.255, 2628 +2026/10/01 09:40:25.355, 2628 +2026/10/01 09:40:25.456, 2634 +2026/10/01 09:40:25.556, 2634 +2026/10/01 09:40:25.656, 3392 +2026/10/01 09:40:25.757, 3392 +2026/10/01 09:40:25.857, 3392 +2026/10/01 09:40:25.957, 3392 +2026/10/01 09:40:26.057, 3392 +2026/10/01 09:40:26.158, 3392 +2026/10/01 09:40:26.258, 3392 +2026/10/01 09:40:26.358, 3392 +2026/10/01 09:40:26.459, 3392 +2026/10/01 09:40:26.559, 3392 +2026/10/01 09:40:26.659, 3392 +2026/10/01 09:40:26.760, 3392 +2026/10/01 09:40:26.860, 3392 +2026/10/01 09:40:26.960, 3394 +2026/10/01 09:40:27.060, 3394 +2026/10/01 09:40:27.161, 3394 +2026/10/01 09:40:27.261, 3394 +2026/10/01 09:40:27.361, 3394 +2026/10/01 09:40:27.461, 3394 +2026/10/01 09:40:27.561, 3394 +2026/10/01 09:40:27.662, 3394 +2026/10/01 09:40:27.762, 3394 +2026/10/01 09:40:27.862, 3394 +2026/10/01 09:40:27.962, 3394 +2026/10/01 09:40:28.063, 3394 +2026/10/01 09:40:28.163, 3394 +2026/10/01 09:40:28.263, 3394 +2026/10/01 09:40:28.364, 3394 +2026/10/01 09:40:28.464, 3394 +2026/10/01 09:40:28.564, 3394 +2026/10/01 09:40:28.665, 3394 +2026/10/01 09:40:28.765, 3394 +2026/10/01 09:40:28.865, 3394 +2026/10/01 09:40:28.966, 3394 +2026/10/01 09:40:29.066, 3394 +2026/10/01 09:40:29.166, 3394 +2026/10/01 09:40:29.266, 3394 +2026/10/01 09:40:29.367, 3394 +2026/10/01 09:40:29.467, 3394 +2026/10/01 09:40:29.567, 3394 +2026/10/01 09:40:29.668, 2628 +2026/10/01 09:40:29.768, 2628 +2026/10/01 09:40:29.868, 2628 +2026/10/01 09:40:29.969, 2628 +2026/10/01 09:40:30.069, 2628 +2026/10/01 09:40:30.169, 2628 +2026/10/01 09:40:30.269, 2628 +2026/10/01 09:40:30.370, 2628 +2026/10/01 09:40:30.470, 2628 +2026/10/01 09:40:30.570, 2628 +2026/10/01 09:40:30.670, 2628 +2026/10/01 09:40:30.771, 2628 +2026/10/01 09:40:30.871, 2628 +2026/10/01 09:40:30.971, 2628 +2026/10/01 09:40:31.072, 2628 +2026/10/01 09:40:31.172, 2628 +2026/10/01 09:40:31.272, 2628 +2026/10/01 09:40:31.373, 2628 +2026/10/01 09:40:31.473, 2628 +2026/10/01 09:40:31.573, 2628 +2026/10/01 09:40:31.673, 2628 +2026/10/01 09:40:31.774, 2628 +2026/10/01 09:40:31.874, 2628 +2026/10/01 09:40:31.974, 2628 +2026/10/01 09:40:32.075, 2628 +2026/10/01 09:40:32.175, 2 +2026/10/01 09:40:32.275, 2 +2026/10/01 09:40:32.375, 2 +2026/10/01 09:40:32.476, 2 +2026/10/01 09:40:32.576, 2 +2026/10/01 09:40:32.676, 2 +2026/10/01 09:40:32.776, 2 +2026/10/01 09:40:32.877, 2 +2026/10/01 09:40:32.977, 2 +2026/10/01 09:40:33.077, 2 +2026/10/01 09:40:33.177, 2 +2026/10/01 09:40:33.278, 2 +2026/10/01 09:40:33.378, 2 +2026/10/01 09:40:33.478, 2 +2026/10/01 09:40:33.579, 2 +2026/10/01 09:40:33.679, 2 +2026/10/01 09:40:33.779, 2 +2026/10/01 09:40:33.880, 2 +2026/10/01 09:40:33.980, 2 +2026/10/01 09:40:34.080, 2 +2026/10/01 09:40:34.181, 2 +2026/10/01 09:40:34.281, 2 +2026/10/01 09:40:34.381, 2 +2026/10/01 09:40:34.482, 2 +2026/10/01 09:40:34.582, 2 +2026/10/01 09:40:34.682, 2 +2026/10/01 09:40:34.782, 2 +2026/10/01 09:40:34.883, 2 +2026/10/01 09:40:34.983, 2 +2026/10/01 09:40:35.083, 2 +2026/10/01 09:40:35.184, 2 +2026/10/01 09:40:35.284, 2 +2026/10/01 09:40:35.384, 2 +2026/10/01 09:40:35.485, 2 +2026/10/01 09:40:35.585, 2 +2026/10/01 09:40:35.685, 2 +2026/10/01 09:40:35.786, 2 +2026/10/01 09:40:35.886, 2 +2026/10/01 09:40:35.986, 2 +2026/10/01 09:40:36.086, 2 +2026/10/01 09:40:36.187, 2 +2026/10/01 09:40:36.287, 2 +2026/10/01 09:40:36.387, 2 +2026/10/01 09:40:36.488, 2 +2026/10/01 09:40:36.588, 2 +2026/10/01 09:40:36.688, 2 +2026/10/01 09:40:36.789, 2 +2026/10/01 09:40:36.889, 2 +2026/10/01 09:40:36.989, 2 +2026/10/01 09:40:37.090, 2 +2026/10/01 09:40:37.190, 2 +2026/10/01 09:40:37.290, 2 +2026/10/01 09:40:37.390, 2 +2026/10/01 09:40:37.491, 2 +2026/10/01 09:40:37.591, 2 +2026/10/01 09:40:37.691, 2 +2026/10/01 09:40:37.792, 2 +2026/10/01 09:40:37.892, 2 +2026/10/01 09:40:37.992, 2 +2026/10/01 09:40:38.093, 2 +2026/10/01 09:40:38.193, 2 +2026/10/01 09:40:38.293, 2 +2026/10/01 09:40:38.394, 2 +2026/10/01 09:40:38.494, 2 +2026/10/01 09:40:38.594, 2 +2026/10/01 09:40:38.694, 2 +2026/10/01 09:40:38.795, 2 +2026/10/01 09:40:38.895, 2 +2026/10/01 09:40:38.995, 2 +2026/10/01 09:40:39.096, 2 +2026/10/01 09:40:39.196, 2 +2026/10/01 09:40:39.296, 2 +2026/10/01 09:40:39.397, 2 +2026/10/01 09:40:39.497, 2 +2026/10/01 09:40:39.597, 2 +2026/10/01 09:40:39.697, 2 +2026/10/01 09:40:39.798, 2 +2026/10/01 09:40:39.898, 2 +2026/10/01 09:40:39.998, 2 +2026/10/01 09:40:40.099, 2 +2026/10/01 09:40:40.199, 2 +2026/10/01 09:40:40.299, 2 +2026/10/01 09:40:40.399, 2 +2026/10/01 09:40:40.500, 2 +2026/10/01 09:40:40.600, 2 +2026/10/01 09:40:40.700, 2 +2026/10/01 09:40:40.801, 2 +2026/10/01 09:40:40.901, 2 +2026/10/01 09:40:41.001, 2 +2026/10/01 09:40:41.102, 2 +2026/10/01 09:40:41.202, 2 +2026/10/01 09:40:41.302, 2 +2026/10/01 09:40:41.403, 2 +2026/10/01 09:40:41.503, 2 +2026/10/01 09:40:41.603, 2 +2026/10/01 09:40:41.703, 2 +2026/10/01 09:40:41.804, 2 +2026/10/01 09:40:41.904, 2 +2026/10/01 09:40:42.004, 2 +2026/10/01 09:40:42.105, 2 +2026/10/01 09:40:42.205, 2 +2026/10/01 09:40:42.305, 2 +2026/10/01 09:40:42.406, 2 +2026/10/01 09:40:42.506, 2 +2026/10/01 09:40:42.606, 2 +2026/10/01 09:40:42.706, 2 +2026/10/01 09:40:42.807, 2 +2026/10/01 09:40:42.907, 2 +2026/10/01 09:40:43.007, 2 +2026/10/01 09:40:43.108, 2 +2026/10/01 09:40:43.208, 2 +2026/10/01 09:40:43.308, 2 +2026/10/01 09:40:43.409, 2 +2026/10/01 09:40:43.510, 2 +2026/10/01 09:40:43.610, 2 +2026/10/01 09:40:43.710, 2 +2026/10/01 09:40:43.810, 2 +2026/10/01 09:40:43.911, 2 +2026/10/01 09:40:44.011, 2 +2026/10/01 09:40:44.111, 2 +2026/10/01 09:40:44.211, 2 +2026/10/01 09:40:44.312, 2 +2026/10/01 09:40:44.412, 2 +2026/10/01 09:40:44.512, 2 +2026/10/01 09:40:44.613, 2 +2026/10/01 09:40:44.713, 2 +2026/10/01 09:40:44.813, 2 +2026/10/01 09:40:44.914, 2 +2026/10/01 09:40:45.014, 2 +2026/10/01 09:40:45.114, 2 +2026/10/01 09:40:45.215, 2 +2026/10/01 09:40:45.315, 2 +2026/10/01 09:40:45.415, 2 +2026/10/01 09:40:45.515, 2 +2026/10/01 09:40:45.616, 2 +2026/10/01 09:40:45.716, 2 +2026/10/01 09:40:45.816, 2 +2026/10/01 09:40:45.917, 2 +2026/10/01 09:40:46.017, 2 +2026/10/01 09:40:46.117, 2 +2026/10/01 09:40:46.218, 2 +2026/10/01 09:40:46.318, 2 +2026/10/01 09:40:46.418, 2 +2026/10/01 09:40:46.518, 2 +2026/10/01 09:40:46.619, 2 +2026/10/01 09:40:46.719, 2 +2026/10/01 09:40:46.819, 2 +2026/10/01 09:40:46.919, 2 +2026/10/01 09:40:47.020, 2 +2026/10/01 09:40:47.120, 2 +2026/10/01 09:40:47.220, 2 +2026/10/01 09:40:47.321, 2 +2026/10/01 09:40:47.421, 2 +2026/10/01 09:40:47.521, 2 +2026/10/01 09:40:47.622, 2 +2026/10/01 09:40:47.722, 2 +2026/10/01 09:40:47.822, 2 +2026/10/01 09:40:47.923, 2 +2026/10/01 09:40:48.023, 2 +2026/10/01 09:40:48.124, 2 +2026/10/01 09:40:48.224, 2 +2026/10/01 09:40:48.325, 2 +2026/10/01 09:40:48.425, 2 +2026/10/01 09:40:48.525, 2 +2026/10/01 09:40:48.626, 2 +2026/10/01 09:40:48.726, 2 +2026/10/01 09:40:48.827, 2 +2026/10/01 09:40:48.927, 2 +2026/10/01 09:40:49.028, 2 +2026/10/01 09:40:49.128, 2 +2026/10/01 09:40:49.228, 2 +2026/10/01 09:40:49.329, 2 +2026/10/01 09:40:49.429, 2 +2026/10/01 09:40:49.530, 2 +2026/10/01 09:40:49.630, 2 +2026/10/01 09:40:49.731, 2 +2026/10/01 09:40:49.831, 2 +2026/10/01 09:40:49.932, 2 +2026/10/01 09:40:50.032, 2 +2026/10/01 09:40:50.133, 2 +2026/10/01 09:40:50.233, 2 +2026/10/01 09:40:50.334, 2 +2026/10/01 09:40:50.434, 2 +2026/10/01 09:40:50.535, 2 +2026/10/01 09:40:50.635, 2 +2026/10/01 09:40:50.735, 2 +2026/10/01 09:40:50.836, 2 +2026/10/01 09:40:50.936, 2 +2026/10/01 09:40:51.037, 2 +2026/10/01 09:40:51.137, 2 +2026/10/01 09:40:51.238, 2 +2026/10/01 09:40:51.338, 2 +2026/10/01 09:40:51.439, 2 +2026/10/01 09:40:51.539, 2 +2026/10/01 09:40:51.640, 2 +2026/10/01 09:40:51.740, 2 +2026/10/01 09:40:51.841, 2 +2026/10/01 09:40:51.941, 2 +2026/10/01 09:40:52.042, 2 +2026/10/01 09:40:52.142, 2 +2026/10/01 09:40:52.243, 2 +2026/10/01 09:40:52.343, 2 +2026/10/01 09:40:52.444, 2 +2026/10/01 09:40:52.544, 2 +2026/10/01 09:40:52.645, 2 +2026/10/01 09:40:52.745, 2 +2026/10/01 09:40:52.846, 2 +2026/10/01 09:40:52.946, 2 +2026/10/01 09:40:53.047, 2 +2026/10/01 09:40:53.147, 2 +2026/10/01 09:40:53.248, 2 +2026/10/01 09:40:53.348, 2 +2026/10/01 09:40:53.449, 2 +2026/10/01 09:40:53.549, 2 +2026/10/01 09:40:53.650, 2 +2026/10/01 09:40:53.750, 2 +2026/10/01 09:40:53.878, 4 +2026/10/01 09:40:53.978, 4 +2026/10/01 09:40:54.079, 4 +2026/10/01 09:40:54.179, 4 +2026/10/01 09:40:54.280, 4 +2026/10/01 09:40:54.380, 4 +2026/10/01 09:40:54.481, 4 +2026/10/01 09:40:54.581, 4 +2026/10/01 09:40:54.682, 4 +2026/10/01 09:40:54.782, 4 +2026/10/01 09:40:54.882, 4 +2026/10/01 09:40:54.983, 4 +2026/10/01 09:40:55.084, 4 +2026/10/01 09:40:55.184, 4 +2026/10/01 09:40:55.285, 4 +2026/10/01 09:40:55.385, 4 +2026/10/01 09:40:55.485, 4 +2026/10/01 09:40:55.586, 4 +2026/10/01 09:40:55.686, 4 +2026/10/01 09:40:55.787, 4 +2026/10/01 09:40:55.887, 4 +2026/10/01 09:40:55.988, 4 +2026/10/01 09:40:56.088, 4 +2026/10/01 09:40:56.189, 4 +2026/10/01 09:40:56.289, 4 +2026/10/01 09:40:56.390, 4 +2026/10/01 09:40:56.490, 4 +2026/10/01 09:40:56.591, 4 +2026/10/01 09:40:56.691, 4 +2026/10/01 09:40:56.791, 4 +2026/10/01 09:40:56.892, 4 +2026/10/01 09:40:56.992, 4 +2026/10/01 09:40:57.093, 4 +2026/10/01 09:40:57.193, 23 +2026/10/01 09:40:57.294, 1222 +2026/10/01 09:40:57.394, 1222 +2026/10/01 09:40:57.495, 1222 +2026/10/01 09:40:57.595, 1222 +2026/10/01 09:40:57.696, 1222 +2026/10/01 09:40:57.796, 1222 +2026/10/01 09:40:57.897, 1222 +2026/10/01 09:40:57.997, 1222 +2026/10/01 09:40:58.097, 1222 +2026/10/01 09:40:58.198, 1222 +2026/10/01 09:40:58.298, 1222 +2026/10/01 09:40:58.399, 1222 +2026/10/01 09:40:58.499, 1222 +2026/10/01 09:40:58.600, 1724 +2026/10/01 09:40:58.700, 1864 +2026/10/01 09:40:58.801, 1864 +2026/10/01 09:40:58.901, 1864 +2026/10/01 09:40:59.002, 1864 +2026/10/01 09:40:59.102, 1864 +2026/10/01 09:40:59.203, 1864 +2026/10/01 09:40:59.303, 1864 +2026/10/01 09:40:59.404, 1864 +2026/10/01 09:40:59.504, 1864 +2026/10/01 09:40:59.605, 1864 +2026/10/01 09:40:59.705, 1864 +2026/10/01 09:40:59.806, 1864 +2026/10/01 09:40:59.906, 2506 +2026/10/01 09:41:00.006, 2506 +2026/10/01 09:41:00.107, 2506 +2026/10/01 09:41:00.207, 2506 +2026/10/01 09:41:00.308, 2506 +2026/10/01 09:41:00.408, 2506 +2026/10/01 09:41:00.509, 2506 +2026/10/01 09:41:00.609, 2506 +2026/10/01 09:41:00.710, 2600 +2026/10/01 09:41:00.810, 2600 +2026/10/01 09:41:00.910, 2600 +2026/10/01 09:41:01.011, 3370 +2026/10/01 09:41:01.111, 3372 +2026/10/01 09:41:01.212, 3372 +2026/10/01 09:41:01.312, 3372 +2026/10/01 09:41:01.413, 3372 +2026/10/01 09:41:01.513, 3372 +2026/10/01 09:41:01.614, 3372 +2026/10/01 09:41:01.714, 3372 +2026/10/01 09:41:01.815, 3372 +2026/10/01 09:41:01.915, 3372 +2026/10/01 09:41:02.016, 3374 +2026/10/01 09:41:02.116, 3374 +2026/10/01 09:41:02.217, 3374 +2026/10/01 09:41:02.317, 3374 +2026/10/01 09:41:02.418, 3374 +2026/10/01 09:41:02.518, 3374 +2026/10/01 09:41:02.619, 3374 +2026/10/01 09:41:02.719, 3374 +2026/10/01 09:41:02.820, 3374 +2026/10/01 09:41:02.920, 3374 +2026/10/01 09:41:03.021, 3374 +2026/10/01 09:41:03.121, 3374 +2026/10/01 09:41:03.222, 3374 +2026/10/01 09:41:03.322, 3374 +2026/10/01 09:41:03.423, 3374 +2026/10/01 09:41:03.523, 3374 +2026/10/01 09:41:03.623, 3374 +2026/10/01 09:41:03.724, 3374 +2026/10/01 09:41:03.824, 3374 +2026/10/01 09:41:03.925, 3374 +2026/10/01 09:41:04.025, 3374 +2026/10/01 09:41:04.126, 3374 +2026/10/01 09:41:04.226, 2628 +2026/10/01 09:41:04.327, 2628 +2026/10/01 09:41:04.427, 2628 +2026/10/01 09:41:04.528, 2628 +2026/10/01 09:41:04.628, 2628 +2026/10/01 09:41:04.729, 2628 +2026/10/01 09:41:04.829, 2628 +2026/10/01 09:41:04.930, 2628 +2026/10/01 09:41:05.030, 2628 +2026/10/01 09:41:05.131, 2628 +2026/10/01 09:41:05.231, 2628 +2026/10/01 09:41:05.332, 2628 +2026/10/01 09:41:05.432, 2628 +2026/10/01 09:41:05.533, 2628 +2026/10/01 09:41:05.633, 2628 +2026/10/01 09:41:05.734, 2628 +2026/10/01 09:41:05.834, 2634 +2026/10/01 09:41:05.935, 2634 +2026/10/01 09:41:06.035, 3392 +2026/10/01 09:41:06.136, 3392 +2026/10/01 09:41:06.236, 3392 +2026/10/01 09:41:06.337, 3392 +2026/10/01 09:41:06.437, 3392 +2026/10/01 09:41:06.538, 3392 +2026/10/01 09:41:06.638, 3392 +2026/10/01 09:41:06.738, 3392 +2026/10/01 09:41:06.839, 3392 +2026/10/01 09:41:06.939, 3394 +2026/10/01 09:41:07.040, 3394 +2026/10/01 09:41:07.140, 3394 +2026/10/01 09:41:07.241, 3394 +2026/10/01 09:41:07.341, 3394 +2026/10/01 09:41:07.442, 3394 +2026/10/01 09:41:07.542, 3394 +2026/10/01 09:41:07.643, 3394 +2026/10/01 09:41:07.743, 3394 +2026/10/01 09:41:07.844, 3394 +2026/10/01 09:41:07.944, 3394 +2026/10/01 09:41:08.045, 3394 +2026/10/01 09:41:08.145, 3394 +2026/10/01 09:41:08.246, 3394 +2026/10/01 09:41:08.346, 3394 +2026/10/01 09:41:08.447, 3394 +2026/10/01 09:41:08.547, 3394 +2026/10/01 09:41:08.648, 3394 +2026/10/01 09:41:08.748, 3394 +2026/10/01 09:41:08.848, 3394 +2026/10/01 09:41:08.949, 3394 +2026/10/01 09:41:09.049, 3394 +2026/10/01 09:41:09.150, 2628 +2026/10/01 09:41:09.250, 2628 +2026/10/01 09:41:09.351, 2628 +2026/10/01 09:41:09.451, 2628 +2026/10/01 09:41:09.552, 2628 +2026/10/01 09:41:09.652, 2628 +2026/10/01 09:41:09.752, 2628 +2026/10/01 09:41:09.853, 2628 +2026/10/01 09:41:09.953, 2628 +2026/10/01 09:41:10.054, 2628 +2026/10/01 09:41:10.154, 2628 +2026/10/01 09:41:10.255, 2628 +2026/10/01 09:41:10.355, 2628 +2026/10/01 09:41:10.456, 2628 +2026/10/01 09:41:10.556, 2628 +2026/10/01 09:41:10.657, 2628 +2026/10/01 09:41:10.757, 2628 +2026/10/01 09:41:10.857, 2628 +2026/10/01 09:41:10.958, 2628 +2026/10/01 09:41:11.058, 2634 +2026/10/01 09:41:11.159, 2634 +2026/10/01 09:41:11.259, 2634 +2026/10/01 09:41:11.359, 3392 +2026/10/01 09:41:11.460, 3392 +2026/10/01 09:41:11.560, 3392 +2026/10/01 09:41:11.661, 3392 +2026/10/01 09:41:11.761, 3392 +2026/10/01 09:41:11.861, 3392 +2026/10/01 09:41:11.962, 3392 +2026/10/01 09:41:12.062, 3392 +2026/10/01 09:41:12.163, 3392 +2026/10/01 09:41:12.263, 3392 +2026/10/01 09:41:12.364, 3392 +2026/10/01 09:41:12.464, 3394 +2026/10/01 09:41:12.565, 3394 +2026/10/01 09:41:12.665, 3394 +2026/10/01 09:41:12.765, 3394 +2026/10/01 09:41:12.866, 3394 +2026/10/01 09:41:12.966, 3394 +2026/10/01 09:41:13.067, 3394 +2026/10/01 09:41:13.167, 3394 +2026/10/01 09:41:13.267, 3394 +2026/10/01 09:41:13.368, 3394 +2026/10/01 09:41:13.468, 3394 +2026/10/01 09:41:13.569, 3394 +2026/10/01 09:41:13.669, 3394 +2026/10/01 09:41:13.770, 3394 +2026/10/01 09:41:13.870, 3394 +2026/10/01 09:41:13.970, 3394 +2026/10/01 09:41:14.071, 3394 +2026/10/01 09:41:14.171, 3394 +2026/10/01 09:41:14.272, 3394 +2026/10/01 09:41:14.372, 3394 +2026/10/01 09:41:14.473, 3394 +2026/10/01 09:41:14.573, 3394 +2026/10/01 09:41:14.674, 3394 +2026/10/01 09:41:14.774, 3394 +2026/10/01 09:41:14.875, 3394 +2026/10/01 09:41:14.975, 3394 +2026/10/01 09:41:15.076, 3394 +2026/10/01 09:41:15.176, 2628 +2026/10/01 09:41:15.277, 2628 +2026/10/01 09:41:15.377, 2628 +2026/10/01 09:41:15.478, 2628 +2026/10/01 09:41:15.578, 2628 +2026/10/01 09:41:15.679, 2628 +2026/10/01 09:41:15.779, 2628 +2026/10/01 09:41:15.880, 2628 +2026/10/01 09:41:15.980, 2628 +2026/10/01 09:41:16.081, 2628 +2026/10/01 09:41:16.181, 2628 +2026/10/01 09:41:16.281, 2628 +2026/10/01 09:41:16.382, 2628 +2026/10/01 09:41:16.482, 2628 +2026/10/01 09:41:16.583, 2628 +2026/10/01 09:41:16.683, 2628 +2026/10/01 09:41:16.784, 2628 +2026/10/01 09:41:16.884, 2628 +2026/10/01 09:41:16.985, 2628 +2026/10/01 09:41:17.085, 2634 +2026/10/01 09:41:17.185, 2634 +2026/10/01 09:41:17.286, 2634 +2026/10/01 09:41:17.386, 3392 +2026/10/01 09:41:17.487, 3392 +2026/10/01 09:41:17.587, 3392 +2026/10/01 09:41:17.688, 3392 +2026/10/01 09:41:17.788, 3392 +2026/10/01 09:41:17.889, 3392 +2026/10/01 09:41:17.989, 3392 +2026/10/01 09:41:18.090, 3392 +2026/10/01 09:41:18.190, 3392 +2026/10/01 09:41:18.290, 3392 +2026/10/01 09:41:18.391, 3392 +2026/10/01 09:41:18.491, 3394 +2026/10/01 09:41:18.591, 3394 +2026/10/01 09:41:18.692, 3394 +2026/10/01 09:41:18.792, 3394 +2026/10/01 09:41:18.892, 3394 +2026/10/01 09:41:18.993, 3394 +2026/10/01 09:41:19.093, 3394 +2026/10/01 09:41:19.193, 3394 +2026/10/01 09:41:19.294, 3394 +2026/10/01 09:41:19.394, 3394 +2026/10/01 09:41:19.495, 3394 +2026/10/01 09:41:19.595, 3394 +2026/10/01 09:41:19.696, 3394 +2026/10/01 09:41:19.796, 3394 +2026/10/01 09:41:19.897, 3394 +2026/10/01 09:41:19.997, 3394 +2026/10/01 09:41:20.098, 3394 +2026/10/01 09:41:20.198, 3394 +2026/10/01 09:41:20.299, 3394 +2026/10/01 09:41:20.399, 3394 +2026/10/01 09:41:20.499, 3394 +2026/10/01 09:41:20.600, 3394 +2026/10/01 09:41:20.700, 3394 +2026/10/01 09:41:20.800, 3394 +2026/10/01 09:41:20.901, 3394 +2026/10/01 09:41:21.001, 3394 +2026/10/01 09:41:21.102, 2628 +2026/10/01 09:41:21.202, 2628 +2026/10/01 09:41:21.303, 2628 +2026/10/01 09:41:21.403, 2628 +2026/10/01 09:41:21.504, 2628 +2026/10/01 09:41:21.604, 2628 +2026/10/01 09:41:21.705, 2628 +2026/10/01 09:41:21.805, 2628 +2026/10/01 09:41:21.906, 2628 +2026/10/01 09:41:22.006, 2628 +2026/10/01 09:41:22.107, 2628 +2026/10/01 09:41:22.207, 2628 +2026/10/01 09:41:22.308, 2628 +2026/10/01 09:41:22.408, 2628 +2026/10/01 09:41:22.509, 2628 +2026/10/01 09:41:22.609, 2628 +2026/10/01 09:41:22.709, 2628 +2026/10/01 09:41:22.810, 2628 +2026/10/01 09:41:22.910, 2628 +2026/10/01 09:41:23.010, 2628 +2026/10/01 09:41:23.111, 2628 +2026/10/01 09:41:23.211, 2628 +2026/10/01 09:41:23.311, 2628 +2026/10/01 09:41:23.412, 2628 +2026/10/01 09:41:23.512, 2628 +2026/10/01 09:41:23.612, 2 +2026/10/01 09:41:23.742, 2 +2026/10/01 09:41:23.843, 2 +2026/10/01 09:41:23.943, 2 +2026/10/01 09:41:24.044, 2 +2026/10/01 09:41:24.144, 2 +2026/10/01 09:41:24.244, 2 +2026/10/01 09:41:24.345, 2 +2026/10/01 09:41:24.445, 2 +2026/10/01 09:41:24.546, 2 +2026/10/01 09:41:24.646, 2 +2026/10/01 09:41:24.747, 2 +2026/10/01 09:41:24.847, 2 +2026/10/01 09:41:24.947, 2 +2026/10/01 09:41:25.048, 2 +2026/10/01 09:41:25.148, 2 +2026/10/01 09:41:25.249, 2 +2026/10/01 09:41:25.349, 2 +2026/10/01 09:41:25.449, 2 +2026/10/01 09:41:25.550, 2 +2026/10/01 09:41:25.650, 2 +2026/10/01 09:41:25.751, 2 +2026/10/01 09:41:25.851, 2 +2026/10/01 09:41:25.952, 2 +2026/10/01 09:41:26.052, 2 +2026/10/01 09:41:26.152, 2 +2026/10/01 09:41:26.253, 2 +2026/10/01 09:41:26.353, 2 +2026/10/01 09:41:26.454, 2 +2026/10/01 09:41:26.554, 2 +2026/10/01 09:41:26.655, 2 +2026/10/01 09:41:26.755, 2 +2026/10/01 09:41:26.856, 2 +2026/10/01 09:41:26.956, 2 +2026/10/01 09:41:27.056, 2 +2026/10/01 09:41:27.157, 2 +2026/10/01 09:41:27.258, 2 +2026/10/01 09:41:27.358, 2 +2026/10/01 09:41:27.459, 2 +2026/10/01 09:41:27.559, 2 +2026/10/01 09:41:27.660, 2 +2026/10/01 09:41:27.760, 2 +2026/10/01 09:41:27.861, 2 +2026/10/01 09:41:27.961, 2 +2026/10/01 09:41:28.061, 2 +2026/10/01 09:41:28.162, 2 +2026/10/01 09:41:28.262, 2 +2026/10/01 09:41:28.363, 2 +2026/10/01 09:41:28.463, 2 +2026/10/01 09:41:28.564, 2 +2026/10/01 09:41:28.664, 2 +2026/10/01 09:41:28.765, 2 +2026/10/01 09:41:28.865, 2 +2026/10/01 09:41:28.966, 2 +2026/10/01 09:41:29.066, 2 +2026/10/01 09:41:29.167, 2 +2026/10/01 09:41:29.267, 2 +2026/10/01 09:41:29.368, 2 +2026/10/01 09:41:29.468, 2 +2026/10/01 09:41:29.569, 2 +2026/10/01 09:41:29.669, 2 +2026/10/01 09:41:29.769, 2 +2026/10/01 09:41:29.870, 2 +2026/10/01 09:41:29.970, 2 +2026/10/01 09:41:30.071, 2 +2026/10/01 09:41:30.171, 2 +2026/10/01 09:41:30.272, 2 +2026/10/01 09:41:30.372, 2 +2026/10/01 09:41:30.473, 2 +2026/10/01 09:41:30.573, 2 +2026/10/01 09:41:30.674, 2 +2026/10/01 09:41:30.774, 2 +2026/10/01 09:41:30.875, 2 +2026/10/01 09:41:30.975, 2 +2026/10/01 09:41:31.075, 2 +2026/10/01 09:41:31.176, 2 +2026/10/01 09:41:31.276, 2 +2026/10/01 09:41:31.377, 2 +2026/10/01 09:41:31.477, 2 +2026/10/01 09:41:31.578, 2 +2026/10/01 09:41:31.678, 2 +2026/10/01 09:41:31.779, 2 +2026/10/01 09:41:31.879, 2 +2026/10/01 09:41:31.979, 2 +2026/10/01 09:41:32.080, 2 +2026/10/01 09:41:32.180, 2 +2026/10/01 09:41:32.281, 2 +2026/10/01 09:41:32.381, 2 +2026/10/01 09:41:32.482, 2 +2026/10/01 09:41:32.582, 2 +2026/10/01 09:41:32.683, 2 +2026/10/01 09:41:32.783, 2 +2026/10/01 09:41:32.883, 2 +2026/10/01 09:41:32.984, 2 +2026/10/01 09:41:33.084, 2 +2026/10/01 09:41:33.185, 2 +2026/10/01 09:41:33.285, 2 +2026/10/01 09:41:33.386, 2 +2026/10/01 09:41:33.486, 2 +2026/10/01 09:41:33.586, 2 +2026/10/01 09:41:33.687, 2 +2026/10/01 09:41:33.787, 2 +2026/10/01 09:41:33.888, 2 +2026/10/01 09:41:33.988, 2 +2026/10/01 09:41:34.088, 2 +2026/10/01 09:41:34.189, 2 +2026/10/01 09:41:34.289, 2 +2026/10/01 09:41:34.389, 2 +2026/10/01 09:41:34.490, 2 +2026/10/01 09:41:34.590, 2 +2026/10/01 09:41:34.691, 2 +2026/10/01 09:41:34.791, 2 +2026/10/01 09:41:34.892, 2 +2026/10/01 09:41:34.992, 2 +2026/10/01 09:41:35.093, 2 +2026/10/01 09:41:35.193, 2 +2026/10/01 09:41:35.293, 2 +2026/10/01 09:41:35.394, 2 +2026/10/01 09:41:35.494, 2 +2026/10/01 09:41:35.595, 2 +2026/10/01 09:41:35.695, 2 +2026/10/01 09:41:35.796, 2 +2026/10/01 09:41:35.896, 2 +2026/10/01 09:41:35.997, 2 +2026/10/01 09:41:36.097, 2 +2026/10/01 09:41:36.198, 2 +2026/10/01 09:41:36.298, 2 +2026/10/01 09:41:36.399, 2 +2026/10/01 09:41:36.499, 2 +2026/10/01 09:41:36.600, 2 +2026/10/01 09:41:36.700, 2 +2026/10/01 09:41:36.801, 2 +2026/10/01 09:41:36.901, 2 +2026/10/01 09:41:37.001, 2 +2026/10/01 09:41:37.102, 2 +2026/10/01 09:41:37.202, 2 +2026/10/01 09:41:37.303, 2 +2026/10/01 09:41:37.403, 2 +2026/10/01 09:41:37.504, 2 +2026/10/01 09:41:37.604, 2 +2026/10/01 09:41:37.705, 2 +2026/10/01 09:41:37.805, 2 +2026/10/01 09:41:37.905, 2 +2026/10/01 09:41:38.006, 2 +2026/10/01 09:41:38.106, 2 +2026/10/01 09:41:38.207, 2 +2026/10/01 09:41:38.307, 2 +2026/10/01 09:41:38.408, 2 +2026/10/01 09:41:38.508, 2 +2026/10/01 09:41:38.609, 2 +2026/10/01 09:41:38.709, 2 +2026/10/01 09:41:38.810, 2 +2026/10/01 09:41:38.910, 2 +2026/10/01 09:41:39.011, 2 +2026/10/01 09:41:39.111, 2 +2026/10/01 09:41:39.212, 2 +2026/10/01 09:41:39.312, 2 +2026/10/01 09:41:39.413, 2 +2026/10/01 09:41:39.513, 2 +2026/10/01 09:41:39.614, 2 +2026/10/01 09:41:39.714, 2 +2026/10/01 09:41:39.815, 2 +2026/10/01 09:41:39.915, 2 +2026/10/01 09:41:40.015, 2 +2026/10/01 09:41:40.116, 2 +2026/10/01 09:41:40.216, 2 +2026/10/01 09:41:40.317, 2 +2026/10/01 09:41:40.417, 2 +2026/10/01 09:41:40.518, 2 +2026/10/01 09:41:40.618, 2 +2026/10/01 09:41:40.719, 2 +2026/10/01 09:41:40.819, 2 +2026/10/01 09:41:40.920, 2 +2026/10/01 09:41:41.020, 2 +2026/10/01 09:41:41.121, 2 +2026/10/01 09:41:41.221, 2 +2026/10/01 09:41:41.322, 2 +2026/10/01 09:41:41.422, 2 +2026/10/01 09:41:41.523, 2 +2026/10/01 09:41:41.623, 2 +2026/10/01 09:41:41.724, 2 +2026/10/01 09:41:41.824, 2 +2026/10/01 09:41:41.925, 2 +2026/10/01 09:41:42.025, 2 +2026/10/01 09:41:42.126, 2 +2026/10/01 09:41:42.226, 2 +2026/10/01 09:41:42.327, 2 +2026/10/01 09:41:42.427, 2 +2026/10/01 09:41:42.528, 2 +2026/10/01 09:41:42.628, 2 +2026/10/01 09:41:42.729, 2 +2026/10/01 09:41:42.829, 2 +2026/10/01 09:41:42.930, 2 +2026/10/01 09:41:43.030, 2 +2026/10/01 09:41:43.131, 2 +2026/10/01 09:41:43.273, 4 +2026/10/01 09:41:43.373, 4 +2026/10/01 09:41:43.474, 4 +2026/10/01 09:41:43.574, 4 +2026/10/01 09:41:43.675, 4 +2026/10/01 09:41:43.775, 4 +2026/10/01 09:41:43.876, 4 +2026/10/01 09:41:43.976, 4 +2026/10/01 09:41:44.077, 4 +2026/10/01 09:41:44.177, 4 +2026/10/01 09:41:44.278, 4 +2026/10/01 09:41:44.378, 4 +2026/10/01 09:41:44.479, 4 +2026/10/01 09:41:44.579, 4 +2026/10/01 09:41:44.680, 4 +2026/10/01 09:41:44.780, 4 +2026/10/01 09:41:44.881, 4 +2026/10/01 09:41:44.981, 4 +2026/10/01 09:41:45.082, 4 +2026/10/01 09:41:45.182, 4 +2026/10/01 09:41:45.283, 4 +2026/10/01 09:41:45.383, 4 +2026/10/01 09:41:45.484, 4 +2026/10/01 09:41:45.584, 4 +2026/10/01 09:41:45.685, 4 +2026/10/01 09:41:45.785, 4 +2026/10/01 09:41:45.885, 4 +2026/10/01 09:41:45.986, 4 +2026/10/01 09:41:46.086, 23 +2026/10/01 09:41:46.187, 1082 +2026/10/01 09:41:46.288, 1222 +2026/10/01 09:41:46.388, 1222 +2026/10/01 09:41:46.489, 1222 +2026/10/01 09:41:46.589, 1222 +2026/10/01 09:41:46.689, 1222 +2026/10/01 09:41:46.790, 1222 +2026/10/01 09:41:46.890, 1222 +2026/10/01 09:41:46.991, 1222 +2026/10/01 09:41:47.091, 1222 +2026/10/01 09:41:47.192, 1222 +2026/10/01 09:41:47.292, 1222 +2026/10/01 09:41:47.393, 1222 +2026/10/01 09:41:47.493, 1222 +2026/10/01 09:41:47.594, 1222 +2026/10/01 09:41:47.694, 1864 +2026/10/01 09:41:47.795, 1864 +2026/10/01 09:41:47.895, 1864 +2026/10/01 09:41:47.996, 1864 +2026/10/01 09:41:48.096, 1864 +2026/10/01 09:41:48.197, 1864 +2026/10/01 09:41:48.297, 1864 +2026/10/01 09:41:48.398, 1864 +2026/10/01 09:41:48.498, 1864 +2026/10/01 09:41:48.599, 1864 +2026/10/01 09:41:48.699, 1864 +2026/10/01 09:41:48.800, 1864 +2026/10/01 09:41:48.900, 2086 +2026/10/01 09:41:49.001, 2506 +2026/10/01 09:41:49.101, 2506 +2026/10/01 09:41:49.202, 2506 +2026/10/01 09:41:49.302, 2506 +2026/10/01 09:41:49.403, 2506 +2026/10/01 09:41:49.503, 2506 +2026/10/01 09:41:49.604, 2506 +2026/10/01 09:41:49.704, 2506 +2026/10/01 09:41:49.805, 2600 +2026/10/01 09:41:49.905, 2600 +2026/10/01 09:41:50.006, 2600 +2026/10/01 09:41:50.106, 3370 +2026/10/01 09:41:50.207, 3372 +2026/10/01 09:41:50.307, 3372 +2026/10/01 09:41:50.407, 3372 +2026/10/01 09:41:50.508, 3372 +2026/10/01 09:41:50.608, 3372 +2026/10/01 09:41:50.709, 3372 +2026/10/01 09:41:50.809, 3372 +2026/10/01 09:41:50.909, 3372 +2026/10/01 09:41:51.010, 3372 +2026/10/01 09:41:51.110, 3374 +2026/10/01 09:41:51.211, 3374 +2026/10/01 09:41:51.311, 3374 +2026/10/01 09:41:51.412, 3374 +2026/10/01 09:41:51.513, 3374 +2026/10/01 09:41:51.613, 3374 +2026/10/01 09:41:51.714, 3374 +2026/10/01 09:41:51.814, 3374 +2026/10/01 09:41:51.915, 3374 +2026/10/01 09:41:52.015, 3374 +2026/10/01 09:41:52.115, 3374 +2026/10/01 09:41:52.216, 3374 +2026/10/01 09:41:52.316, 3374 +2026/10/01 09:41:52.417, 3374 +2026/10/01 09:41:52.517, 3374 +2026/10/01 09:41:52.618, 3374 +2026/10/01 09:41:52.718, 3374 +2026/10/01 09:41:52.819, 3374 +2026/10/01 09:41:52.919, 3374 +2026/10/01 09:41:53.020, 3374 +2026/10/01 09:41:53.121, 3374 +2026/10/01 09:41:53.221, 3374 +2026/10/01 09:41:53.321, 2628 +2026/10/01 09:41:53.422, 2628 +2026/10/01 09:41:53.522, 2628 +2026/10/01 09:41:53.623, 2628 +2026/10/01 09:41:53.723, 2628 +2026/10/01 09:41:53.824, 2628 +2026/10/01 09:41:53.924, 2628 +2026/10/01 09:41:54.025, 2628 +2026/10/01 09:41:54.125, 2628 +2026/10/01 09:41:54.225, 2628 +2026/10/01 09:41:54.326, 2628 +2026/10/01 09:41:54.426, 2628 +2026/10/01 09:41:54.527, 2628 +2026/10/01 09:41:54.627, 2628 +2026/10/01 09:41:54.728, 2628 +2026/10/01 09:41:54.828, 2628 +2026/10/01 09:41:54.929, 2634 +2026/10/01 09:41:55.029, 2634 +2026/10/01 09:41:55.130, 2634 +2026/10/01 09:41:55.230, 3392 +2026/10/01 09:41:55.330, 3392 +2026/10/01 09:41:55.431, 3392 +2026/10/01 09:41:55.531, 3392 +2026/10/01 09:41:55.631, 3392 +2026/10/01 09:41:55.732, 3392 +2026/10/01 09:41:55.832, 3392 +2026/10/01 09:41:55.933, 3392 +2026/10/01 09:41:56.033, 3392 +2026/10/01 09:41:56.134, 3392 +2026/10/01 09:41:56.234, 3392 +2026/10/01 09:41:56.334, 3394 +2026/10/01 09:41:56.435, 3394 +2026/10/01 09:41:56.535, 3394 +2026/10/01 09:41:56.635, 3394 +2026/10/01 09:41:56.736, 3394 +2026/10/01 09:41:56.836, 3394 +2026/10/01 09:41:56.937, 3394 +2026/10/01 09:41:57.037, 3394 +2026/10/01 09:41:57.138, 3394 +2026/10/01 09:41:57.238, 3394 +2026/10/01 09:41:57.338, 3394 +2026/10/01 09:41:57.439, 3394 +2026/10/01 09:41:57.539, 3394 +2026/10/01 09:41:57.639, 3394 +2026/10/01 09:41:57.740, 3394 +2026/10/01 09:41:57.840, 3394 +2026/10/01 09:41:57.940, 3394 +2026/10/01 09:41:58.041, 3394 +2026/10/01 09:41:58.141, 3394 +2026/10/01 09:41:58.242, 3394 +2026/10/01 09:41:58.342, 3394 +2026/10/01 09:41:58.442, 3394 +2026/10/01 09:41:58.543, 3394 +2026/10/01 09:41:58.643, 3394 +2026/10/01 09:41:58.743, 3394 +2026/10/01 09:41:58.844, 3394 +2026/10/01 09:41:58.944, 2628 +2026/10/01 09:41:59.045, 2628 +2026/10/01 09:41:59.145, 2628 +2026/10/01 09:41:59.245, 2628 +2026/10/01 09:41:59.346, 2628 +2026/10/01 09:41:59.446, 2628 +2026/10/01 09:41:59.546, 2628 +2026/10/01 09:41:59.647, 2628 +2026/10/01 09:41:59.747, 2628 +2026/10/01 09:41:59.848, 2628 +2026/10/01 09:41:59.948, 2628 +2026/10/01 09:42:00.048, 2628 +2026/10/01 09:42:00.149, 2628 +2026/10/01 09:42:00.249, 2628 +2026/10/01 09:42:00.350, 2628 +2026/10/01 09:42:00.450, 2628 +2026/10/01 09:42:00.550, 2628 +2026/10/01 09:42:00.651, 2628 +2026/10/01 09:42:00.751, 2634 +2026/10/01 09:42:00.852, 2634 +2026/10/01 09:42:00.952, 2634 +2026/10/01 09:42:01.053, 3392 +2026/10/01 09:42:01.153, 3392 +2026/10/01 09:42:01.253, 3392 +2026/10/01 09:42:01.354, 3392 +2026/10/01 09:42:01.454, 3392 +2026/10/01 09:42:01.554, 3392 +2026/10/01 09:42:01.655, 3392 +2026/10/01 09:42:01.755, 3392 +2026/10/01 09:42:01.856, 3392 +2026/10/01 09:42:01.956, 3392 +2026/10/01 09:42:02.056, 3392 +2026/10/01 09:42:02.157, 3394 +2026/10/01 09:42:02.257, 3394 +2026/10/01 09:42:02.357, 3394 +2026/10/01 09:42:02.458, 3394 +2026/10/01 09:42:02.558, 3394 +2026/10/01 09:42:02.658, 3394 +2026/10/01 09:42:02.759, 3394 +2026/10/01 09:42:02.859, 3394 +2026/10/01 09:42:02.960, 3394 +2026/10/01 09:42:03.060, 3394 +2026/10/01 09:42:03.160, 3394 +2026/10/01 09:42:03.261, 3394 +2026/10/01 09:42:03.361, 3394 +2026/10/01 09:42:03.462, 3394 +2026/10/01 09:42:03.562, 3394 +2026/10/01 09:42:03.663, 3394 +2026/10/01 09:42:03.763, 3394 +2026/10/01 09:42:03.864, 3394 +2026/10/01 09:42:03.964, 3394 +2026/10/01 09:42:04.065, 3394 +2026/10/01 09:42:04.165, 3394 +2026/10/01 09:42:04.265, 3394 +2026/10/01 09:42:04.366, 3394 +2026/10/01 09:42:04.466, 3394 +2026/10/01 09:42:04.567, 3394 +2026/10/01 09:42:04.667, 2628 +2026/10/01 09:42:04.768, 2628 +2026/10/01 09:42:04.868, 2628 +2026/10/01 09:42:04.969, 2628 +2026/10/01 09:42:05.069, 2628 +2026/10/01 09:42:05.170, 2628 +2026/10/01 09:42:05.270, 2628 +2026/10/01 09:42:05.370, 2628 +2026/10/01 09:42:05.471, 2628 +2026/10/01 09:42:05.571, 2628 +2026/10/01 09:42:05.672, 2628 +2026/10/01 09:42:05.772, 2628 +2026/10/01 09:42:05.873, 2628 +2026/10/01 09:42:05.973, 2628 +2026/10/01 09:42:06.074, 2628 +2026/10/01 09:42:06.174, 2628 +2026/10/01 09:42:06.275, 2628 +2026/10/01 09:42:06.375, 2628 +2026/10/01 09:42:06.476, 2634 +2026/10/01 09:42:06.576, 2634 +2026/10/01 09:42:06.677, 2634 +2026/10/01 09:42:06.777, 3392 +2026/10/01 09:42:06.877, 3392 +2026/10/01 09:42:06.978, 3392 +2026/10/01 09:42:07.078, 3392 +2026/10/01 09:42:07.179, 3392 +2026/10/01 09:42:07.279, 3392 +2026/10/01 09:42:07.380, 3392 +2026/10/01 09:42:07.480, 3392 +2026/10/01 09:42:07.581, 3394 +2026/10/01 09:42:07.681, 3394 +2026/10/01 09:42:07.782, 3394 +2026/10/01 09:42:07.882, 3394 +2026/10/01 09:42:07.983, 3394 +2026/10/01 09:42:08.083, 3394 +2026/10/01 09:42:08.183, 3394 +2026/10/01 09:42:08.284, 3394 +2026/10/01 09:42:08.384, 3394 +2026/10/01 09:42:08.485, 3394 +2026/10/01 09:42:08.585, 3394 +2026/10/01 09:42:08.686, 3394 +2026/10/01 09:42:08.786, 3394 +2026/10/01 09:42:08.887, 3394 +2026/10/01 09:42:08.987, 3394 +2026/10/01 09:42:09.087, 3394 +2026/10/01 09:42:09.188, 3394 +2026/10/01 09:42:09.288, 3394 +2026/10/01 09:42:09.389, 3394 +2026/10/01 09:42:09.489, 3394 +2026/10/01 09:42:09.590, 3394 +2026/10/01 09:42:09.690, 3394 +2026/10/01 09:42:09.791, 3394 +2026/10/01 09:42:09.891, 3394 +2026/10/01 09:42:09.992, 2628 +2026/10/01 09:42:10.092, 2628 +2026/10/01 09:42:10.193, 2628 +2026/10/01 09:42:10.293, 2628 +2026/10/01 09:42:10.393, 2628 +2026/10/01 09:42:10.494, 2628 +2026/10/01 09:42:10.594, 2628 +2026/10/01 09:42:10.695, 2628 +2026/10/01 09:42:10.795, 2628 +2026/10/01 09:42:10.896, 2628 +2026/10/01 09:42:10.996, 2628 +2026/10/01 09:42:11.097, 2628 +2026/10/01 09:42:11.198, 2628 +2026/10/01 09:42:11.298, 2628 +2026/10/01 09:42:11.399, 2628 +2026/10/01 09:42:11.499, 2628 +2026/10/01 09:42:11.599, 2628 +2026/10/01 09:42:11.700, 2628 +2026/10/01 09:42:11.800, 2628 +2026/10/01 09:42:11.901, 2628 +2026/10/01 09:42:12.001, 2628 +2026/10/01 09:42:12.102, 2628 +2026/10/01 09:42:12.202, 2628 +2026/10/01 09:42:12.303, 2 +2026/10/01 09:42:12.444, 2 +2026/10/01 09:42:12.545, 2 +2026/10/01 09:42:12.645, 2 +2026/10/01 09:42:12.745, 2 +2026/10/01 09:42:12.846, 2 +2026/10/01 09:42:12.947, 2 +2026/10/01 09:42:13.047, 2 +2026/10/01 09:42:13.148, 2 +2026/10/01 09:42:13.248, 2 +2026/10/01 09:42:13.349, 2 +2026/10/01 09:42:13.449, 2 +2026/10/01 09:42:13.550, 2 +2026/10/01 09:42:13.650, 2 +2026/10/01 09:42:13.751, 2 +2026/10/01 09:42:13.851, 2 +2026/10/01 09:42:13.951, 2 +2026/10/01 09:42:14.052, 2 +2026/10/01 09:42:14.152, 2 +2026/10/01 09:42:14.253, 2 +2026/10/01 09:42:14.353, 2 +2026/10/01 09:42:14.453, 2 +2026/10/01 09:42:14.554, 2 +2026/10/01 09:42:14.655, 2 +2026/10/01 09:42:14.755, 2 +2026/10/01 09:42:14.856, 2 +2026/10/01 09:42:14.956, 2 +2026/10/01 09:42:15.056, 2 +2026/10/01 09:42:15.157, 2 +2026/10/01 09:42:15.257, 2 +2026/10/01 09:42:15.358, 2 +2026/10/01 09:42:15.458, 2 +2026/10/01 09:42:15.558, 2 +2026/10/01 09:42:15.659, 2 +2026/10/01 09:42:15.759, 2 +2026/10/01 09:42:15.860, 2 +2026/10/01 09:42:15.960, 2 +2026/10/01 09:42:16.061, 2 +2026/10/01 09:42:16.161, 2 +2026/10/01 09:42:16.262, 2 +2026/10/01 09:42:16.362, 2 +2026/10/01 09:42:16.463, 2 +2026/10/01 09:42:16.563, 2 +2026/10/01 09:42:16.663, 2 +2026/10/01 09:42:16.764, 2 +2026/10/01 09:42:16.864, 2 +2026/10/01 09:42:16.965, 2 +2026/10/01 09:42:17.065, 2 +2026/10/01 09:42:17.166, 2 +2026/10/01 09:42:17.266, 2 +2026/10/01 09:42:17.367, 2 +2026/10/01 09:42:17.467, 2 +2026/10/01 09:42:17.568, 2 +2026/10/01 09:42:17.668, 2 +2026/10/01 09:42:17.769, 2 +2026/10/01 09:42:17.869, 2 +2026/10/01 09:42:17.970, 2 +2026/10/01 09:42:18.070, 2 +2026/10/01 09:42:18.171, 2 +2026/10/01 09:42:18.271, 2 +2026/10/01 09:42:18.371, 2 +2026/10/01 09:42:18.472, 2 +2026/10/01 09:42:18.572, 2 +2026/10/01 09:42:18.673, 2 +2026/10/01 09:42:18.773, 2 +2026/10/01 09:42:18.874, 2 +2026/10/01 09:42:18.974, 2 +2026/10/01 09:42:19.075, 2 +2026/10/01 09:42:19.175, 2 +2026/10/01 09:42:19.276, 2 +2026/10/01 09:42:19.376, 2 +2026/10/01 09:42:19.476, 2 +2026/10/01 09:42:19.577, 2 +2026/10/01 09:42:19.677, 2 +2026/10/01 09:42:19.778, 2 +2026/10/01 09:42:19.878, 2 +2026/10/01 09:42:19.979, 2 +2026/10/01 09:42:20.079, 2 +2026/10/01 09:42:20.179, 2 +2026/10/01 09:42:20.280, 2 +2026/10/01 09:42:20.380, 2 +2026/10/01 09:42:20.481, 2 +2026/10/01 09:42:20.581, 2 +2026/10/01 09:42:20.681, 2 +2026/10/01 09:42:20.782, 2 +2026/10/01 09:42:20.882, 2 +2026/10/01 09:42:20.983, 2 +2026/10/01 09:42:21.083, 2 +2026/10/01 09:42:21.184, 2 +2026/10/01 09:42:21.284, 2 +2026/10/01 09:42:21.385, 2 +2026/10/01 09:42:21.485, 2 +2026/10/01 09:42:21.586, 2 +2026/10/01 09:42:21.686, 2 +2026/10/01 09:42:21.786, 2 +2026/10/01 09:42:21.887, 2 +2026/10/01 09:42:21.987, 2 +2026/10/01 09:42:22.088, 2 +2026/10/01 09:42:22.188, 2 +2026/10/01 09:42:22.289, 2 +2026/10/01 09:42:22.389, 2 +2026/10/01 09:42:22.490, 2 +2026/10/01 09:42:22.590, 2 +2026/10/01 09:42:22.691, 2 +2026/10/01 09:42:22.791, 2 +2026/10/01 09:42:22.892, 2 +2026/10/01 09:42:22.992, 2 +2026/10/01 09:42:23.093, 2 +2026/10/01 09:42:23.193, 2 +2026/10/01 09:42:23.294, 2 +2026/10/01 09:42:23.394, 2 +2026/10/01 09:42:23.494, 2 +2026/10/01 09:42:23.595, 2 +2026/10/01 09:42:23.695, 2 +2026/10/01 09:42:23.796, 2 +2026/10/01 09:42:23.896, 2 +2026/10/01 09:42:23.997, 2 +2026/10/01 09:42:24.097, 2 +2026/10/01 09:42:24.198, 2 +2026/10/01 09:42:24.298, 2 +2026/10/01 09:42:24.399, 2 +2026/10/01 09:42:24.499, 2 +2026/10/01 09:42:24.600, 2 +2026/10/01 09:42:24.700, 2 +2026/10/01 09:42:24.801, 2 +2026/10/01 09:42:24.901, 2 +2026/10/01 09:42:25.002, 2 +2026/10/01 09:42:25.102, 2 +2026/10/01 09:42:25.203, 2 +2026/10/01 09:42:25.303, 2 +2026/10/01 09:42:25.403, 2 +2026/10/01 09:42:25.504, 2 +2026/10/01 09:42:25.604, 2 +2026/10/01 09:42:25.705, 2 +2026/10/01 09:42:25.805, 2 +2026/10/01 09:42:25.906, 2 +2026/10/01 09:42:26.007, 2 +2026/10/01 09:42:26.107, 2 +2026/10/01 09:42:26.207, 2 +2026/10/01 09:42:26.308, 2 +2026/10/01 09:42:26.408, 2 +2026/10/01 09:42:26.509, 2 +2026/10/01 09:42:26.609, 2 +2026/10/01 09:42:26.710, 2 +2026/10/01 09:42:26.810, 2 +2026/10/01 09:42:26.910, 2 +2026/10/01 09:42:27.011, 2 +2026/10/01 09:42:27.111, 2 +2026/10/01 09:42:27.212, 2 +2026/10/01 09:42:27.312, 2 +2026/10/01 09:42:27.413, 2 +2026/10/01 09:42:27.513, 2 +2026/10/01 09:42:27.614, 2 +2026/10/01 09:42:27.714, 2 +2026/10/01 09:42:27.815, 2 +2026/10/01 09:42:27.915, 2 +2026/10/01 09:42:28.016, 2 +2026/10/01 09:42:28.116, 2 +2026/10/01 09:42:28.217, 2 +2026/10/01 09:42:28.317, 2 +2026/10/01 09:42:28.418, 2 +2026/10/01 09:42:28.518, 2 +2026/10/01 09:42:28.619, 2 +2026/10/01 09:42:28.719, 2 +2026/10/01 09:42:28.820, 2 +2026/10/01 09:42:28.920, 2 +2026/10/01 09:42:29.020, 2 +2026/10/01 09:42:29.121, 2 +2026/10/01 09:42:29.221, 2 +2026/10/01 09:42:29.322, 2 +2026/10/01 09:42:29.422, 2 +2026/10/01 09:42:29.523, 2 +2026/10/01 09:42:29.623, 2 +2026/10/01 09:42:29.724, 2 +2026/10/01 09:42:29.824, 2 +2026/10/01 09:42:29.925, 2 +2026/10/01 09:42:30.025, 2 +2026/10/01 09:42:30.126, 2 +2026/10/01 09:42:30.226, 2 +2026/10/01 09:42:30.327, 2 +2026/10/01 09:42:30.427, 2 +2026/10/01 09:42:30.527, 2 +2026/10/01 09:42:30.628, 2 +2026/10/01 09:42:30.728, 2 +2026/10/01 09:42:30.829, 2 +2026/10/01 09:42:30.929, 2 +2026/10/01 09:42:31.030, 2 +2026/10/01 09:42:31.130, 2 +2026/10/01 09:42:31.231, 2 +2026/10/01 09:42:31.331, 2 +2026/10/01 09:42:31.432, 2 +2026/10/01 09:42:31.532, 2 +2026/10/01 09:42:31.633, 2 +2026/10/01 09:42:31.733, 2 +2026/10/01 09:42:31.834, 2 +2026/10/01 09:42:31.934, 2 +2026/10/01 09:42:32.034, 2 +2026/10/01 09:42:32.135, 2 +2026/10/01 09:42:32.235, 2 +2026/10/01 09:42:32.336, 2 +2026/10/01 09:42:32.436, 2 +2026/10/01 09:42:32.537, 2 +2026/10/01 09:42:32.637, 2 +2026/10/01 09:42:32.738, 2 +2026/10/01 09:42:32.838, 2 +2026/10/01 09:42:32.938, 2 +2026/10/01 09:42:33.039, 2 +2026/10/01 09:42:33.139, 2 +2026/10/01 09:42:33.240, 2 +2026/10/01 09:42:33.340, 2 +2026/10/01 09:42:33.441, 2 +2026/10/01 09:42:33.541, 2 +2026/10/01 09:42:33.641, 2 +2026/10/01 09:42:33.742, 2 +2026/10/01 09:42:33.842, 2 +2026/10/01 09:42:33.943, 2 +2026/10/01 09:42:34.043, 2 +2026/10/01 09:42:34.144, 2 +2026/10/01 09:42:34.244, 2 +2026/10/01 09:42:34.345, 2 +2026/10/01 09:42:34.445, 2 +2026/10/01 09:42:34.545, 2 +2026/10/01 09:42:34.646, 2 +2026/10/01 09:42:34.746, 2 +2026/10/01 09:42:34.847, 2 +2026/10/01 09:42:34.947, 2 +2026/10/01 09:42:35.047, 2 +2026/10/01 09:42:35.148, 2 +2026/10/01 09:42:35.248, 2 +2026/10/01 09:42:35.349, 2 +2026/10/01 09:42:35.449, 2 +2026/10/01 09:42:35.550, 2 +2026/10/01 09:42:35.650, 2 +2026/10/01 09:42:35.751, 2 +2026/10/01 09:42:35.851, 2 +2026/10/01 09:42:35.952, 2 +2026/10/01 09:42:36.052, 2 +2026/10/01 09:42:36.161, 4 +2026/10/01 09:42:36.261, 4 +2026/10/01 09:42:36.362, 4 +2026/10/01 09:42:36.462, 4 +2026/10/01 09:42:36.563, 4 +2026/10/01 09:42:36.663, 4 +2026/10/01 09:42:36.764, 4 +2026/10/01 09:42:36.864, 4 +2026/10/01 09:42:36.965, 4 +2026/10/01 09:42:37.065, 4 +2026/10/01 09:42:37.166, 4 +2026/10/01 09:42:37.266, 4 +2026/10/01 09:42:37.367, 4 +2026/10/01 09:42:37.467, 4 +2026/10/01 09:42:37.568, 4 +2026/10/01 09:42:37.668, 4 +2026/10/01 09:42:37.769, 4 +2026/10/01 09:42:37.869, 4 +2026/10/01 09:42:37.970, 4 +2026/10/01 09:42:38.070, 4 +2026/10/01 09:42:38.171, 4 +2026/10/01 09:42:38.271, 4 +2026/10/01 09:42:38.372, 4 +2026/10/01 09:42:38.472, 4 +2026/10/01 09:42:38.572, 4 +2026/10/01 09:42:38.673, 4 +2026/10/01 09:42:38.773, 4 +2026/10/01 09:42:38.874, 4 +2026/10/01 09:42:38.974, 4 +2026/10/01 09:42:39.075, 23 +2026/10/01 09:42:39.175, 1222 +2026/10/01 09:42:39.276, 1222 +2026/10/01 09:42:39.376, 1222 +2026/10/01 09:42:39.477, 1222 +2026/10/01 09:42:39.577, 1222 +2026/10/01 09:42:39.678, 1222 +2026/10/01 09:42:39.778, 1222 +2026/10/01 09:42:39.879, 1222 +2026/10/01 09:42:39.979, 1222 +2026/10/01 09:42:40.081, 1222 +2026/10/01 09:42:40.182, 1222 +2026/10/01 09:42:40.282, 1222 +2026/10/01 09:42:40.383, 1222 +2026/10/01 09:42:40.483, 1222 +2026/10/01 09:42:40.584, 1864 +2026/10/01 09:42:40.684, 1864 +2026/10/01 09:42:40.785, 1864 +2026/10/01 09:42:40.885, 1864 +2026/10/01 09:42:40.985, 1864 +2026/10/01 09:42:41.086, 1864 +2026/10/01 09:42:41.186, 1864 +2026/10/01 09:42:41.287, 1864 +2026/10/01 09:42:41.387, 1864 +2026/10/01 09:42:41.488, 1864 +2026/10/01 09:42:41.588, 1864 +2026/10/01 09:42:41.689, 1864 +2026/10/01 09:42:41.789, 1944 +2026/10/01 09:42:41.889, 2506 +2026/10/01 09:42:41.990, 2506 +2026/10/01 09:42:42.090, 2506 +2026/10/01 09:42:42.191, 2506 +2026/10/01 09:42:42.291, 2506 +2026/10/01 09:42:42.391, 2506 +2026/10/01 09:42:42.492, 2506 +2026/10/01 09:42:42.592, 2506 +2026/10/01 09:42:42.693, 2506 +2026/10/01 09:42:42.793, 2600 +2026/10/01 09:42:42.893, 2600 +2026/10/01 09:42:42.994, 2622 +2026/10/01 09:42:43.094, 3372 +2026/10/01 09:42:43.194, 3372 +2026/10/01 09:42:43.295, 3372 +2026/10/01 09:42:43.395, 3372 +2026/10/01 09:42:43.496, 3372 +2026/10/01 09:42:43.596, 3372 +2026/10/01 09:42:43.697, 3372 +2026/10/01 09:42:43.797, 3372 +2026/10/01 09:42:43.898, 3372 +2026/10/01 09:42:43.998, 3372 +2026/10/01 09:42:44.098, 3372 +2026/10/01 09:42:44.199, 3372 +2026/10/01 09:42:44.299, 3374 +2026/10/01 09:42:44.400, 3374 +2026/10/01 09:42:44.500, 3374 +2026/10/01 09:42:44.601, 3374 +2026/10/01 09:42:44.701, 3374 +2026/10/01 09:42:44.801, 3374 +2026/10/01 09:42:44.902, 3374 +2026/10/01 09:42:45.002, 3374 +2026/10/01 09:42:45.103, 3374 +2026/10/01 09:42:45.203, 3374 +2026/10/01 09:42:45.303, 3374 +2026/10/01 09:42:45.404, 3374 +2026/10/01 09:42:45.504, 3374 +2026/10/01 09:42:45.605, 3374 +2026/10/01 09:42:45.705, 3374 +2026/10/01 09:42:45.806, 3374 +2026/10/01 09:42:45.906, 3374 +2026/10/01 09:42:46.007, 3374 +2026/10/01 09:42:46.107, 3374 +2026/10/01 09:42:46.207, 3374 +2026/10/01 09:42:46.308, 3374 +2026/10/01 09:42:46.408, 3374 +2026/10/01 09:42:46.509, 3374 +2026/10/01 09:42:46.609, 3374 +2026/10/01 09:42:46.709, 3374 +2026/10/01 09:42:46.810, 3374 +2026/10/01 09:42:46.910, 2628 +2026/10/01 09:42:47.011, 2628 +2026/10/01 09:42:47.111, 2628 +2026/10/01 09:42:47.212, 2628 +2026/10/01 09:42:47.312, 2628 +2026/10/01 09:42:47.413, 2628 +2026/10/01 09:42:47.513, 2628 +2026/10/01 09:42:47.614, 2628 +2026/10/01 09:42:47.714, 2628 +2026/10/01 09:42:47.815, 2628 +2026/10/01 09:42:47.915, 2628 +2026/10/01 09:42:48.015, 2628 +2026/10/01 09:42:48.116, 2628 +2026/10/01 09:42:48.216, 2628 +2026/10/01 09:42:48.317, 2628 +2026/10/01 09:42:48.417, 2628 +2026/10/01 09:42:48.518, 2628 +2026/10/01 09:42:48.618, 2634 +2026/10/01 09:42:48.718, 2634 +2026/10/01 09:42:48.819, 2634 +2026/10/01 09:42:48.919, 3392 +2026/10/01 09:42:49.020, 3392 +2026/10/01 09:42:49.120, 3392 +2026/10/01 09:42:49.221, 3392 +2026/10/01 09:42:49.321, 3392 +2026/10/01 09:42:49.422, 3392 +2026/10/01 09:42:49.522, 3392 +2026/10/01 09:42:49.623, 3392 +2026/10/01 09:42:49.723, 3394 +2026/10/01 09:42:49.824, 3394 +2026/10/01 09:42:49.924, 3394 +2026/10/01 09:42:50.024, 3394 +2026/10/01 09:42:50.125, 3394 +2026/10/01 09:42:50.225, 3394 +2026/10/01 09:42:50.326, 3394 +2026/10/01 09:42:50.426, 3394 +2026/10/01 09:42:50.527, 3394 +2026/10/01 09:42:50.627, 3394 +2026/10/01 09:42:50.728, 3394 +2026/10/01 09:42:50.828, 3394 +2026/10/01 09:42:50.928, 3394 +2026/10/01 09:42:51.029, 3394 +2026/10/01 09:42:51.129, 3394 +2026/10/01 09:42:51.230, 3394 +2026/10/01 09:42:51.330, 3394 +2026/10/01 09:42:51.431, 3394 +2026/10/01 09:42:51.531, 3394 +2026/10/01 09:42:51.631, 3394 +2026/10/01 09:42:51.732, 3394 +2026/10/01 09:42:51.832, 3394 +2026/10/01 09:42:51.933, 2628 +2026/10/01 09:42:52.033, 2628 +2026/10/01 09:42:52.134, 2628 +2026/10/01 09:42:52.234, 2628 +2026/10/01 09:42:52.335, 2628 +2026/10/01 09:42:52.435, 2628 +2026/10/01 09:42:52.535, 2628 +2026/10/01 09:42:52.636, 2628 +2026/10/01 09:42:52.736, 2628 +2026/10/01 09:42:52.837, 2628 +2026/10/01 09:42:52.937, 2628 +2026/10/01 09:42:53.038, 2628 +2026/10/01 09:42:53.138, 2628 +2026/10/01 09:42:53.239, 2628 +2026/10/01 09:42:53.339, 2628 +2026/10/01 09:42:53.439, 2628 +2026/10/01 09:42:53.540, 2628 +2026/10/01 09:42:53.641, 2628 +2026/10/01 09:42:53.741, 2634 +2026/10/01 09:42:53.842, 2634 +2026/10/01 09:42:53.942, 2634 +2026/10/01 09:42:54.042, 3392 +2026/10/01 09:42:54.143, 3392 +2026/10/01 09:42:54.243, 3392 +2026/10/01 09:42:54.344, 3392 +2026/10/01 09:42:54.444, 3392 +2026/10/01 09:42:54.545, 3392 +2026/10/01 09:42:54.645, 3392 +2026/10/01 09:42:54.746, 3392 +2026/10/01 09:42:54.846, 3394 +2026/10/01 09:42:54.947, 3394 +2026/10/01 09:42:55.047, 3394 +2026/10/01 09:42:55.148, 3394 +2026/10/01 09:42:55.248, 3394 +2026/10/01 09:42:55.349, 3394 +2026/10/01 09:42:55.449, 3394 +2026/10/01 09:42:55.550, 3394 +2026/10/01 09:42:55.650, 3394 +2026/10/01 09:42:55.750, 3394 +2026/10/01 09:42:55.851, 3394 +2026/10/01 09:42:55.951, 3394 +2026/10/01 09:42:56.052, 3394 +2026/10/01 09:42:56.152, 3394 +2026/10/01 09:42:56.253, 3394 +2026/10/01 09:42:56.353, 3394 +2026/10/01 09:42:56.454, 3394 +2026/10/01 09:42:56.554, 3394 +2026/10/01 09:42:56.654, 3394 +2026/10/01 09:42:56.755, 3394 +2026/10/01 09:42:56.855, 3394 +2026/10/01 09:42:56.956, 3394 +2026/10/01 09:42:57.056, 2628 +2026/10/01 09:42:57.157, 2628 +2026/10/01 09:42:57.257, 2628 +2026/10/01 09:42:57.358, 2628 +2026/10/01 09:42:57.458, 2628 +2026/10/01 09:42:57.559, 2628 +2026/10/01 09:42:57.659, 2628 +2026/10/01 09:42:57.760, 2628 +2026/10/01 09:42:57.860, 2628 +2026/10/01 09:42:57.961, 2628 +2026/10/01 09:42:58.061, 2628 +2026/10/01 09:42:58.161, 2628 +2026/10/01 09:42:58.262, 2628 +2026/10/01 09:42:58.362, 2628 +2026/10/01 09:42:58.463, 2628 +2026/10/01 09:42:58.563, 2628 +2026/10/01 09:42:58.664, 2628 +2026/10/01 09:42:58.764, 2628 +2026/10/01 09:42:58.864, 2634 +2026/10/01 09:42:58.965, 2634 +2026/10/01 09:42:59.065, 3392 +2026/10/01 09:42:59.166, 3392 +2026/10/01 09:42:59.266, 3392 +2026/10/01 09:42:59.367, 3392 +2026/10/01 09:42:59.467, 3392 +2026/10/01 09:42:59.568, 3392 +2026/10/01 09:42:59.668, 3392 +2026/10/01 09:42:59.768, 3392 +2026/10/01 09:42:59.869, 3392 +2026/10/01 09:42:59.969, 3392 +2026/10/01 09:43:00.070, 3392 +2026/10/01 09:43:00.170, 3392 +2026/10/01 09:43:00.270, 3394 +2026/10/01 09:43:00.371, 3394 +2026/10/01 09:43:00.471, 3394 +2026/10/01 09:43:00.572, 3394 +2026/10/01 09:43:00.672, 3394 +2026/10/01 09:43:00.773, 3394 +2026/10/01 09:43:00.873, 3394 +2026/10/01 09:43:00.973, 3394 +2026/10/01 09:43:01.074, 3394 +2026/10/01 09:43:01.174, 3394 +2026/10/01 09:43:01.274, 3394 +2026/10/01 09:43:01.375, 3394 +2026/10/01 09:43:01.475, 3394 +2026/10/01 09:43:01.576, 3394 +2026/10/01 09:43:01.676, 3394 +2026/10/01 09:43:01.777, 3394 +2026/10/01 09:43:01.877, 3394 +2026/10/01 09:43:01.978, 3394 +2026/10/01 09:43:02.078, 3394 +2026/10/01 09:43:02.179, 3394 +2026/10/01 09:43:02.279, 3394 +2026/10/01 09:43:02.379, 3394 +2026/10/01 09:43:02.480, 3394 +2026/10/01 09:43:02.580, 3394 +2026/10/01 09:43:02.681, 3394 +2026/10/01 09:43:02.781, 2628 +2026/10/01 09:43:02.882, 2628 +2026/10/01 09:43:02.982, 2628 +2026/10/01 09:43:03.083, 2628 +2026/10/01 09:43:03.183, 2628 +2026/10/01 09:43:03.284, 2628 +2026/10/01 09:43:03.384, 2628 +2026/10/01 09:43:03.484, 2628 +2026/10/01 09:43:03.585, 2628 +2026/10/01 09:43:03.685, 2628 +2026/10/01 09:43:03.786, 2628 +2026/10/01 09:43:03.886, 2628 +2026/10/01 09:43:03.987, 2628 +2026/10/01 09:43:04.087, 2628 +2026/10/01 09:43:04.188, 2628 +2026/10/01 09:43:04.288, 2628 +2026/10/01 09:43:04.389, 2628 +2026/10/01 09:43:04.489, 2628 +2026/10/01 09:43:04.590, 2628 +2026/10/01 09:43:04.690, 2628 +2026/10/01 09:43:04.790, 2628 +2026/10/01 09:43:04.891, 2628 +2026/10/01 09:43:04.991, 2628 +2026/10/01 09:43:05.092, 2628 +2026/10/01 09:43:05.192, 2 +2026/10/01 09:43:05.294, 2 +2026/10/01 09:43:05.394, 2 +2026/10/01 09:43:05.494, 2 +2026/10/01 09:43:05.595, 2 +2026/10/01 09:43:05.695, 2 +2026/10/01 09:43:05.796, 2 +2026/10/01 09:43:05.896, 2 +2026/10/01 09:43:05.997, 2 +2026/10/01 09:43:06.097, 2 +2026/10/01 09:43:06.197, 2 +2026/10/01 09:43:06.298, 2 +2026/10/01 09:43:06.398, 2 +2026/10/01 09:43:06.499, 2 +2026/10/01 09:43:06.599, 2 +2026/10/01 09:43:06.699, 2 +2026/10/01 09:43:06.800, 2 +2026/10/01 09:43:06.900, 2 +2026/10/01 09:43:07.001, 2 +2026/10/01 09:43:07.101, 2 +2026/10/01 09:43:07.202, 2 +2026/10/01 09:43:07.302, 2 +2026/10/01 09:43:07.403, 2 +2026/10/01 09:43:07.503, 2 +2026/10/01 09:43:07.604, 2 +2026/10/01 09:43:07.704, 2 +2026/10/01 09:43:07.805, 2 +2026/10/01 09:43:07.905, 2 +2026/10/01 09:43:08.005, 2 +2026/10/01 09:43:08.106, 2 +2026/10/01 09:43:08.206, 2 +2026/10/01 09:43:08.307, 2 +2026/10/01 09:43:08.407, 2 +2026/10/01 09:43:08.508, 2 +2026/10/01 09:43:08.608, 2 +2026/10/01 09:43:08.709, 2 +2026/10/01 09:43:08.809, 2 +2026/10/01 09:43:08.910, 2 +2026/10/01 09:43:09.010, 2 +2026/10/01 09:43:09.110, 2 +2026/10/01 09:43:09.211, 2 +2026/10/01 09:43:09.311, 2 +2026/10/01 09:43:09.411, 2 +2026/10/01 09:43:09.512, 2 +2026/10/01 09:43:09.612, 2 +2026/10/01 09:43:09.712, 2 +2026/10/01 09:43:09.813, 2 +2026/10/01 09:43:09.913, 2 +2026/10/01 09:43:10.014, 2 +2026/10/01 09:43:10.114, 2 +2026/10/01 09:43:10.214, 2 +2026/10/01 09:43:10.315, 2 +2026/10/01 09:43:10.415, 2 +2026/10/01 09:43:10.516, 2 +2026/10/01 09:43:10.616, 2 +2026/10/01 09:43:10.716, 2 +2026/10/01 09:43:10.817, 2 +2026/10/01 09:43:10.917, 2 +2026/10/01 09:43:11.018, 2 +2026/10/01 09:43:11.118, 2 +2026/10/01 09:43:11.219, 2 +2026/10/01 09:43:11.319, 2 +2026/10/01 09:43:11.419, 2 +2026/10/01 09:43:11.520, 2 +2026/10/01 09:43:11.620, 2 +2026/10/01 09:43:11.721, 2 +2026/10/01 09:43:11.821, 2 +2026/10/01 09:43:11.922, 2 +2026/10/01 09:43:12.022, 2 +2026/10/01 09:43:12.122, 2 +2026/10/01 09:43:12.223, 2 diff --git a/stacks/mia/mia_driver.py b/stacks/mia/mia_driver.py new file mode 100644 index 0000000..cbf4e6a --- /dev/null +++ b/stacks/mia/mia_driver.py @@ -0,0 +1,262 @@ +"""Fleet driver for Make-It-Animatable v2 (app_v2.py): one container run, any number of meshes. + +Usage (inside the local/mia image; scripts/mia-run is the caller-facing wrapper): + python mia_driver.py [--seed N | --unseeded] = [= ...] + +Inputs are paths relative to the working directory (the job dir). Per mesh it writes to out/: + _pred.npz predictions in the INPUT file's coordinates, plus `weights_effective` + .fbx / .glb MIA's FBX (Blender export) and its FBX2glTF preview + _rest.glb Blender glTF export of the rigged rest pose + _apose-hint.* the same predictions, Blender stage re-run with "Input Rest Pose = A-pose" + _run.json stage timings, peak GPU memory, seed, provenance + +Adapted from dread-dev's tools/mia/run_mia.py + finalize.py (dreadnaught repo, 2026-10-01). The +pipeline calls, the npz keys and the artifact set are theirs, unchanged. Differences: + - Paths: repo at /opt/mia/repo; MIA's scratch (the input copy and the work dir it writes beside + it) lives in a temp dir that dies with the container, so only out/ comes back to the caller. + - SEEDED BY DEFAULT. Before EVERY mesh: util.utils.fix_random(seed) (python/numpy/torch seeds, + cudnn deterministic), and trimesh's module RNG reset to default_rng(seed). trimesh >= 4 samples + surface points from trimesh.util._RANDOM_DEFAULT, which fix_random never reaches; unseeded runs + moved joints 0.1-0.27% of height (dread-dev's CPU noise floor). Resetting per mesh also makes a + mesh's result independent of which meshes ran before it in the same process. + --unseeded restores upstream behaviour (fix_random once at model load, trimesh unseeded). + - weights_effective is written straight into the npz (finalize.py's formula: weights > 1e-3 kept, + rows renormalised; what the FBX vertex groups and the GLB WEIGHTS_0 actually carry). + - Each stage is followed by torch.cuda.synchronize() so the per-stage wall times are honest. +""" + +import argparse +import json +import os +import re +import shutil +import sys +import tempfile +import time + +REPO = "/opt/mia/repo" +JOB = os.getcwd() +OUT_DIR = os.path.join(JOB, "out") +NAME_RE = re.compile(r"^[A-Za-z0-9._-]+$") + + +def parse_args(): + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + g = ap.add_mutually_exclusive_group() + g.add_argument("--seed", type=int, default=0, help="seed for every mesh (default 0)") + g.add_argument("--unseeded", action="store_true", help="upstream behaviour: trimesh sampling unseeded") + ap.add_argument("meshes", nargs="+", metavar="NAME=INPUT.glb") + a = ap.parse_args() + jobs = [] + for m in a.meshes: + if "=" not in m: + ap.error(f"{m!r}: expected NAME=INPUT.glb") + name, path = m.split("=", 1) + if not NAME_RE.match(name): + ap.error(f"{name!r}: NAME must be [A-Za-z0-9._-]+") + src = os.path.realpath(os.path.join(JOB, path)) + if not src.startswith(JOB + os.sep): + ap.error(f"{path!r}: inputs must be inside the job dir") + if not os.path.isfile(src): + ap.error(f"{path!r}: no such file in the job dir") + jobs.append((name, src)) + if len({n for n, _ in jobs}) != len(jobs): + ap.error("NAMEs must be unique") + return a, jobs + + +args, JOBS = parse_args() +SEED = None if args.unseeded else args.seed + +os.environ.setdefault("GRADIO_ANALYTICS_ENABLED", "False") +os.environ["HF_HUB_OFFLINE"] = "1" # all weights are on the read-only /hf mount; never touch the network +sys.path.insert(0, REPO) +os.chdir(REPO) + +import numpy as np # noqa: E402 +import torch # noqa: E402 +import trimesh.util # noqa: E402 + +t0 = time.perf_counter() +import app_v2 as A # noqa: E402 +from util.utils import fix_random # noqa: E402 + +A.init_models() +A.init_blocks() # only defines the Gradio component globals the stage functions return as dict keys; no server +t_init = time.perf_counter() - t0 +CUDA = torch.cuda.is_available() +print(f"[mia] init (imports + 3 models + blocks): {t_init:.1f}s, device {'cuda:' + torch.cuda.get_device_name(0) if CUDA else 'cpu'}, seed {SEED}", flush=True) + +STAGES = ["prepare_input", "preprocess", "infer", "vis", "vis_blender", "finish"] +BONE_NAMES = [None] * len(A.BONES_IDX_DICT) +for k, v in A.BONES_IDX_DICT.items(): + BONE_NAMES[v] = k +PARENTS = list(A.KINEMATIC_TREE.parent_indices) +PROVENANCE = dict(line.strip().split("=", 1) for line in open("/opt/mia/provenance") if "=" in line) + + +def sync(): + if CUDA: + torch.cuda.synchronize() + + +def to_np(x): + if isinstance(x, torch.Tensor): + x = x.detach().cpu().numpy() + return np.asarray(x) + + +def run_one(name: str, src: str, scratch: str): + if SEED is not None: + fix_random(SEED) + trimesh.util._RANDOM_DEFAULT = np.random.default_rng(SEED) + if CUDA: + torch.cuda.reset_peak_memory_stats() + inp = os.path.join(scratch, f"{name}.glb") + shutil.copyfile(src, inp) # MIA writes its work dir next to the input file, so run on a copy + work = os.path.join(scratch, name) + + db = A.DB() + gen = A._pipeline( + input_path=inp, + is_gs=False, + opacity_threshold=0.01, + no_fingers=False, # UI default + rest_pose_type="No", # UI default + ignore_pose_parts=[], # UI default + input_normal=True, # UI default (fixed) + bw_fix=True, # UI default: weight post-processing on + bw_vis_bone="LeftArm", # UI default (visualisation only) + restore_global=False, # UI default: outputs in MIA's normalised frame + reset_to_rest=True, # UI default: apply predicted T-pose as the rest pose + animation_file=None, # deviation from UI default ("Standard Run.fbx"): static rig, no animation baked + retarget=True, + inplace=True, + db=db, + ) + timings = {} + snap = {} + for stage in STAGES: + w0 = time.perf_counter() + next(gen) + sync() + timings[stage] = {"wall_s": time.perf_counter() - w0} + print(f"[mia] {name}: {stage} {timings[stage]['wall_s']:.2f}s", flush=True) + if stage == "prepare_input": + snap["verts_input"] = to_np(db.verts)[0].copy() + snap["faces"] = np.asarray(db.faces).copy() + elif stage == "infer": + snap["bw_raw"] = to_np(db.bw)[0].copy() + try: + next(gen) + except StopIteration: + pass + + # --- predictions in the INPUT file's coordinates --- + T = db.global_transform # pytorch3d Transform3d (row-vector): input -> MIA-normalised frame + Tinv = T.inverse() + T_mat = T.get_matrix().transpose(-1, -2)[0].cpu().numpy() # column-vector 4x4 + Tinv_mat = Tinv.get_matrix().transpose(-1, -2)[0].cpu().numpy() + joints_n = np.asarray(db.joints, dtype=np.float32) + tails_n = np.asarray(db.joints_tail, dtype=np.float32) + dev = Tinv.device + joints_in = Tinv.transform_points(torch.from_numpy(joints_n)[None].to(dev))[0].cpu().numpy() + tails_in = Tinv.transform_points(torch.from_numpy(tails_n)[None].to(dev))[0].cpu().numpy() + verts_n = np.asarray(db.verts, dtype=np.float32) + verts_back = Tinv.transform_points(torch.from_numpy(verts_n)[None].to(dev))[0].cpu().numpy() + roundtrip_err = float(np.abs(verts_back - snap["verts_input"]).max()) + pose_n = np.asarray(db.pose, dtype=np.float32) + pose_in = np.einsum("ij,kjl,lm->kim", Tinv_mat, pose_n, T_mat).astype(np.float32) + + W = np.asarray(db.bw, dtype=np.float32) + We = np.where(W > 1e-3, W, 0).astype(np.float32) # finalize.py: MIA's set_weights threshold + We /= np.maximum(We.sum(1, keepdims=True), 1e-12) + + np.savez_compressed( + os.path.join(OUT_DIR, f"{name}_pred.npz"), + verts=snap["verts_input"].astype(np.float32), + faces=snap["faces"].astype(np.int64), + weights=W, + weights_raw=snap["bw_raw"].astype(np.float32), + weights_effective=We, + bone_names=np.array(BONE_NAMES), + parents=np.array(PARENTS, dtype=np.int64), + joints_head=joints_in.astype(np.float32), + joints_tail=tails_in.astype(np.float32), + pose_to_rest=pose_in, + input_to_mia=T_mat.astype(np.float32), + joints_head_mia=joints_n, + joints_tail_mia=tails_n, + verts_mia=verts_n, + ) + + # --- MIA's own artifacts (default settings) --- + copies = { + f"{name}.fbx": db.anim_path, # MIA's primary output (Blender FBX export) + f"{name}.glb": db.anim_vis_path, # MIA's FBX2glTF conversion of the FBX (the app's "GLB preview") + f"{name}_rest.glb": db.rest_vis_path, # Blender glTF export of the rigged rest-pose model + } + missing = [] + for dst, srcp in copies.items(): + if srcp and os.path.isfile(srcp): + shutil.copyfile(srcp, os.path.join(OUT_DIR, dst)) + else: + missing.append(dst) + + # --- variant: same predictions, Blender stage re-run with the UI's "Input Rest Pose = A-pose" hint --- + w0 = time.perf_counter() + db.anim_path = os.path.join(work, f"{name}_apose-hint.fbx") + db.anim_vis_path = os.path.join(work, f"{name}_apose-hint.glb") + db.rest_vis_path = os.path.join(work, f"{name}_apose-hint_rest.glb") + A.vis_blender( + reset_to_rest=True, + remove_fingers=False, + rest_pose_type="A-pose", + ignore_pose_parts=[], + animation_file=None, + retarget=True, + inplace=True, + restore_global=False, + db=db, + ) + timings["vis_blender_apose_hint"] = {"wall_s": time.perf_counter() - w0} + for suffix in ("_apose-hint.fbx", "_apose-hint.glb", "_apose-hint_rest.glb"): + srcp = os.path.join(work, f"{name}{suffix}") + if os.path.isfile(srcp): + shutil.copyfile(srcp, os.path.join(OUT_DIR, f"{name}{suffix}")) + else: + missing.append(f"{name}{suffix}") + + meta = { + "name": name, + "source": os.path.relpath(src, JOB), + "n_verts": int(snap["verts_input"].shape[0]), + "n_faces": int(snap["faces"].shape[0]), + "seed": SEED, + "device": torch.cuda.get_device_name(0) if CUDA else "cpu", + "timings": timings, + "model_total_wall_s": sum(timings[s]["wall_s"] for s in ("preprocess", "infer")), + "pipeline_total_wall_s": sum(timings[s]["wall_s"] for s in STAGES), + "init_wall_s": t_init, + "gpu_peak_allocated_mib": torch.cuda.max_memory_allocated() / 2**20 if CUDA else None, + "gpu_peak_reserved_mib": torch.cuda.max_memory_reserved() / 2**20 if CUDA else None, + "roundtrip_err_input_to_mia_and_back": roundtrip_err, + "missing_artifacts": missing, + "provenance": PROVENANCE, + } + with open(os.path.join(OUT_DIR, f"{name}_run.json"), "w") as f: + json.dump(meta, f, indent=2) + print(f"[mia] {name}: done, pipeline {meta['pipeline_total_wall_s']:.1f}s, peak reserved " + f"{meta['gpu_peak_reserved_mib'] or 0:.0f} MiB, roundtrip err {roundtrip_err:.2e}" + + (f", MISSING {missing}" if missing else ""), flush=True) + return meta + + +if __name__ == "__main__": + os.makedirs(OUT_DIR, exist_ok=True) + failed = 0 + with tempfile.TemporaryDirectory(prefix="mia-") as scratch: + for name, src in JOBS: + meta = run_one(name, src, scratch) + failed += bool(meta["missing_artifacts"]) + sys.exit(1 if failed else 0) diff --git a/stacks/mia/requirements.lock b/stacks/mia/requirements.lock new file mode 100644 index 0000000..f10fe65 --- /dev/null +++ b/stacks/mia/requirements.lock @@ -0,0 +1,113 @@ +# MIA v2 Python lock: dread-dev's CPU venv (nh3-dev, 2026-10-01) verbatim, except the four +# torch-family lines, swapped from +cpu to the cu129 builds upstream's requirements.txt names. +accelerate==1.9.0 +annotated-doc==0.0.5 +annotated-types==0.8.0 +antlr4-python3-runtime==4.9.3 +anyio==4.15.1 +attrs==26.1.0 +bpy==4.3.0 +brotli==1.2.0 +certifi==2026.7.22 +charset-normalizer==3.5.2 +click==8.5.0 +colorlog==6.12.0 +contourpy==1.3.3 +cycler==0.12.1 +cython==3.3.0 +diffusers==0.39.0 +einops==0.8.2 +embreex==4.4.0 +fastapi==0.142.2 +filelock==3.32.3 +fonttools==4.66.1 +fsspec==2026.7.0 +gradio==6.17.3 +gradio-client==2.5.0 +groovy==0.1.2 +h11==0.16.0 +hf-gradio==0.4.1 +hf-xet==1.6.0 +httpcore==1.0.9 +httpx==0.28.1 +huggingface-hub==0.36.2 +idna==3.20 +imageio==2.38.0 +importlib-metadata==9.0.1 +iopath==0.1.10 +jinja2==3.1.6 +jsonschema==4.26.0 +jsonschema-specifications==2025.9.1 +kiwisolver==1.5.1 +lazy-loader==0.6 +lxml==6.1.3 +manifold3d==3.5.4 +mapbox-earcut==2.1.0 +markdown-it-py==4.2.0 +markupsafe==3.0.3 +matplotlib==3.11.2 +mdurl==0.1.2 +mpmath==1.3.0 +networkx==3.6.1 +numpy==1.26.4 +omegaconf==2.3.1 +opencv-contrib-python==4.11.0.86 +opentelemetry-api==1.45.0 +orjson==3.12.0 +packaging==26.3 +pandas==3.0.6 +pillow==12.3.0 +plyfile==1.1.3 +portalocker==4.4.0 +potpourri3d==1.4.0 +psutil==7.2.2 +pycollada==0.9.3 +pydantic==2.13.5 +pydantic-core==2.46.5 +pydub==0.25.1 +pygments==2.21.0 +pymcubes==0.1.6 +pymeshlab==2023.12.post3 +pyparsing==3.3.3 +python-dateutil==2.9.0.post0 +python-multipart==0.0.32 +pytorch3d==0.7.8+pt2.8.0cu129 +pytz==2026.4 +pyyaml==6.0.3 +referencing==0.37.0 +regex==2026.9.29 +requests==2.34.2 +rich==15.0.0 +rpds-py==2026.6.3 +rtree==1.4.1 +safehttpx==0.1.7 +safetensors==0.8.0 +scikit-image==0.26.0 +scipy==1.17.1 +semantic-version==2.10.0 +shapely==2.1.2 +shellingham==1.5.4 +six==1.17.0 +spaces==0.51.3 +starlette==1.7.0 +svg-path==7.1 +sympy==1.14.0 +tifffile==2026.3.3 +timm==1.0.30 +tokenizers==0.20.3 +tomlkit==0.14.0 +torch==2.8.0+cu129 +torch-cluster==1.6.3+pt28cu129 +torchvision==0.23.0+cu129 +tqdm==4.70.1 +transformers==4.46.0 +trimesh==5.1.0 +typer==0.27.2 +typing-extensions==4.16.0 +typing-inspection==0.4.4 +urllib3==2.8.0 +uvicorn==0.54.0 +vhacdx==0.1.0 +xxhash==4.0.1 +zipp==4.1.0 +zstandard==0.25.0