memory: snapshot — Flash-Next seat on one card, and the FV outage that followed
Durable capture so tomorrow's session does not have to reconstruct either half. Built and verified before the power failed: Qwen3.8-Flash-Next serving on a single RTX PRO 6000 with its 51B n-gram table pinned in host RAM and read over CUDA UVA -- 74.36 GiB weights resident, 14.00 GiB KV for 560,654 tokens at the full 262,144 context, 67 GiB host RSS -- plus a gen-large gateway alias verified end to end. The five findings worth carrying: the offload is #54371 (UVA, merged) which supersedes the paused worker-based #53899 and designs out its entire bug family; text_config.ple_embedding_dtype is the load-or-fail discriminator for any community build; --kv-cache-memory makes vLLM SKIP memory profiling and ignore gpu-memory-utilization, which inverts the usual pin-bytes advice and let a 16 GiB pin nearly OOM with no visible failure; MTP is off pending measurement here rather than written off, because the recipe's number is cross-harness and tested k=3 only while the head is one layer run autoregressively; and a container once reported (healthy) with no published port at all, because the healthcheck runs inside the boundary it was trusted to validate. Then the outage. Records it as will-not-self-recover, so no session wastes effort polling a dead site, and carries the three things that change the visit: bypass the UPS rather than using its surge-only bank (both banks share one 12 A inlet -- the surge bank bypasses the inverter, not the current rating), recover power.log before anything else because it is the only load measurement that exists anywhere, and bring seats up one at a time because ten restart:unless-stopped containers loading at once is the largest transient the box can make into whatever just failed. Also records what is still half-done: the stale homepage labels on the 10 containers that died before they could be recreated, which the staged bring-up fixes as a side effect, and the eight drifted stacks plus three untracked host-only stacks that were deliberately left for a deliberate reconciliation.
This commit is contained in:
@@ -1,6 +1,9 @@
|
||||
# FV site dark — 2026-09-13 ~06:56Z
|
||||
|
||||
**Status: UNRESOLVED, needs hands at Fountain Valley.** Written while the site is
|
||||
**Status: UNRESOLVED, WILL NOT SELF-RECOVER, needs hands at Fountain Valley.**
|
||||
Operator 2026-09-13: an overload-tripped UPS does not clear itself and a dead one
|
||||
needs replacing, so there is no recovery to wait for and no point polling the site.
|
||||
Visit planned for 2026-09-14. Do not leave watchers running against FV addresses. Written while the site is
|
||||
down so recovery does not have to be reconstructed from memory.
|
||||
|
||||
## What is down
|
||||
@@ -138,9 +141,58 @@ much longer recovery.
|
||||
4. **Do not restart the MTP campaign.** It is the prime suspect.
|
||||
5. Watch power while seats come up: `nvidia-smi --query-gpu=index,power.draw --format=csv`.
|
||||
|
||||
### ⭐ The staged bring-up also finishes the homepage-label fix, for free
|
||||
|
||||
The 2026-09-13 renumber (commit `3132a16`) repaired 25 compose files on this box but
|
||||
the 10 RUNNING containers were never recreated, so their labels still carried the dead
|
||||
10.250.50.54. Those containers are gone with the power loss.
|
||||
|
||||
⚠ A PLAIN POWER-ON DOES NOT FIX THEM. `restart: unless-stopped` restarts the existing
|
||||
container with its existing labels; labels only attach at container CREATION. But the
|
||||
staged `docker compose up -d <svc>` sequence above **is** a recreate, and the compose
|
||||
files on disk are already corrected — so bringing seats up that way applies the new
|
||||
labels as a side effect and the dashboard comes back correct. Bring them up with
|
||||
`compose up -d`, not by letting Docker restore the old containers.
|
||||
|
||||
Afterwards, confirm with:
|
||||
|
||||
curl -s http://10.0.50.45:5100/api/services | \
|
||||
python3 -c 'import json,sys;d=json.load(sys.stdin);print([s["href"] for g in d for s in (g.get("services") or []) if "10.250.50.54" in (s.get("href") or "")])'
|
||||
|
||||
Expect `[]`. Before the outage that query returned 16 entries.
|
||||
|
||||
**Do not reboot the OPNsense firewall** (standing operator directive; its reboot API
|
||||
403s anyway).
|
||||
|
||||
## ⚠ PROPOSED, NOT RATIFIED — split the power so the management plane survives
|
||||
|
||||
**This is a recommendation awaiting the operator's call, not settled intent.** Written
|
||||
down so tomorrow's rebuild can adopt or reject it deliberately rather than restoring
|
||||
the arrangement that just failed by default.
|
||||
|
||||
UPS <- OPNsense firewall + fv-ml1 BMC only (tens of watts, long runtime)
|
||||
PDU / wall <- GPU chassis (no UPS in series)
|
||||
|
||||
Two reasons:
|
||||
|
||||
1. **It fixes the OOB gap this outage exposed.** The BMC's only route to the fleet is
|
||||
through the firewall, so a power event at the GPU box takes out the management plane
|
||||
with it -- which is precisely why this incident needs a drive rather than a console
|
||||
session. Separate the two and a repeat leaves a live firewall, a live BMC, and
|
||||
remote eyes on a dark chassis.
|
||||
2. **A 1500 VA unit was never going to hold this box.** It has four cards, not the two
|
||||
every record claimed until 2026-09-12.
|
||||
|
||||
⚠ **Do NOT use a UPS's surge-only outlets to get around its rating.** Both outlet banks
|
||||
sit downstream of the same input cord, inlet and internal breaker; for a 120 V 1500 VA
|
||||
unit that is a single NEMA 5-15P rated **12 A at maximum load**, total across all
|
||||
outlets. The surge bank bypasses the inverter, not the current rating. Overloading the
|
||||
inverter trips or kills the unit; overloading the cord is a thermal problem in an
|
||||
unattended rack. Bypass the UPS entirely instead.
|
||||
|
||||
If the GPU box is ever to go on battery, it is a 3000 VA / 2700 W-class unit and
|
||||
probably a 20 A circuit -- but size it from `power.log`, not from a spec sheet.
|
||||
|
||||
## Afterwards
|
||||
|
||||
- ⭐ **The OOB design gap this exposes.** The cutover chose OPNsense-as-subnet-router
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
# `[2026-09-13]` Flash-Next seat stood up on fv-ml1 — then the FV site went dark under two-card load
|
||||
|
||||
Two things happened in one session and they are causally linked: a 176B model was put
|
||||
on one GPU with its embedding table in host RAM, and the attempt to benchmark it on a
|
||||
second GPU took Fountain Valley off the air.
|
||||
|
||||
## What was built and is real
|
||||
|
||||
**`stacks/flash-next-seat/` — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2, `:8022`.**
|
||||
The first seat on the fleet whose weights do not fit its card. 176B total = a 125B main
|
||||
model + a **51B n-gram (PLE) lookup table** that lives in pinned host RAM and is read
|
||||
by the GPU over **CUDA UVA**. Measured, not predicted:
|
||||
|
||||
weights resident 74.36 GiB of 95.6 (load 97-242 s depending on page cache)
|
||||
KV cache 14.00 GiB pinned in bytes -> 560,654 tokens
|
||||
context 262,144 (native max), 2.14x concurrency
|
||||
engine host RSS 67 GiB (47.7 GiB table + runtime)
|
||||
throughput (n=1!) 75.5 / 212.3 / 387.8 tok/s at concurrency 1 / 4 / 8
|
||||
|
||||
Checkpoint `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99`, 126.0 GiB at
|
||||
`/tank/aimodels/qwen38-flash-next-abliterated-nvfp4`. Composition measured from the
|
||||
manifest: **63.3 GiB NVFP4 W4A4 routed experts + 47.7 GiB FP8 PLE table + 14.9 GiB
|
||||
BF16** — so it is NOT an all-4-bit quant, and the single largest item on disk is the
|
||||
FP8 table, 38% of the download.
|
||||
|
||||
**`gen-large` on the LiteLLM gateway**, one alias, verified registered for both the
|
||||
infra-ops admin key and the shared all-agents key. Sampling is the checkpoint's own
|
||||
declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty, min_p and
|
||||
repetition_penalty deliberately unset because the checkpoint declares no canonical
|
||||
value. Commits `f964a47`, `7e62a07`.
|
||||
|
||||
## ⭐⭐ The five things worth carrying forward
|
||||
|
||||
1. **The offload is upstream and merged — via a DIFFERENT PR than the park predicted.**
|
||||
vLLM **#54371** "UVA PLE-offload and Engram tensor parallelism" merged
|
||||
2026-09-09T14:32Z: pinned host memory + direct UVA row reads, no worker process.
|
||||
It **supersedes #53899** (worker-based, now explicitly paused), and that matters
|
||||
because #53899's entire bug family — the TP=1 startup deadlock #53960, the
|
||||
`pidfd_getfd`/yama ptrace gate, the shared-CUDA-event race, silently one-step-stale
|
||||
PLE outputs under CUDA graphs — all came from the separate worker the UVA path does
|
||||
not have. Flag: `--engram-config '{"cpu_offload": true}'`. **In `v0.29.1rc0` and
|
||||
main, NOT in `v0.29.0`** (cut ~6 h before the merge), and not in any `nightly-<sha>`
|
||||
dated 2026-09-09 or earlier — the nightly builds at ~06:16 UTC.
|
||||
2. **⚠ `text_config.ple_embedding_dtype` is the load-or-fail discriminator.** vLLM picks
|
||||
the PLE table's format from that field FIRST. A checkpoint with an FP8 table and no
|
||||
such declaration resolves through ModelOpt's `*.ple.*` exclude to the *unquantized*
|
||||
method and dies on the scale tensor it does ship. Check it before trying any other
|
||||
community build.
|
||||
3. **⚠⚠ `--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore
|
||||
`--gpu-memory-utilization`.** This inverts the usual "pin bytes for
|
||||
reproducibility" advice. 16.00 GiB was tried and a 155,497-token prefill drove GPU 2
|
||||
to 97,074 of 97,887 MiB with the allocator logging *"OOM on device 0 while trying to
|
||||
allocate 488636416 bytes (free: 422117376)"* — 466 MiB wanted, 403 free. The request
|
||||
still completed, so **nothing failed visibly**; that is one step before the shape
|
||||
that crashed mog-sec twice. 14.00 GiB re-probes clean. Do **not** take vLLM's own
|
||||
*"17.46 GiB to fully utilize gpu memory"* line: it is computed from a profile taken
|
||||
at `max-num-batched-tokens` depth and sits 3.5 GiB above what a 150K request
|
||||
survives. `max-num-batched-tokens` is what bounds the activation peak — doubling
|
||||
`max_model_len` left the profiled peak unchanged at 1.65 GiB.
|
||||
4. **⚠ MTP is OFF but NOT written off, and the operator was right to push back.** The
|
||||
seat initially defaulted MTP off citing vLLM's recipe (worse at every concurrency on
|
||||
4xH100: 8-36% less throughput, 32-173% more latency, ~36% acceptance). **That was a
|
||||
cross-harness comparison** — 4xH100/TP=4/Hopper vs 1xBlackwell/TP=1 — and our own
|
||||
rule says those are invalid, not merely noisy. The real gap: the recipe tested
|
||||
**k=3 only**, and `mtp_num_hidden_layers` is **1**, so the draft head is a single
|
||||
module run autoregressively for k>1. If throughput falls monotonically in k while
|
||||
acceptance rises, **k=1 may win and nobody has published that cell.** Campaign built
|
||||
at `services/flash-next-mtp-bench/` with repeats, a boot-to-boot noise floor, and
|
||||
acceptance-greater-than-zero as a positive control. It completed exactly **one**
|
||||
`off_A` rep before the power failed.
|
||||
5. **⚠ "Healthy" and "reachable" are on opposite sides of the container boundary.** The
|
||||
seat once ran `Up 4 minutes (healthy)` with **`PORTS=[]`** — its port bind had lost a
|
||||
race with a dying container, and the healthcheck passed throughout because it runs
|
||||
*inside* the container against `localhost:8000`. Verify `docker port` / the
|
||||
docker-proxy listener, not the healthcheck.
|
||||
|
||||
## ⚠⚠ And then the site went dark — see `docs/runbooks/fv-site-dark-20260913.md`
|
||||
|
||||
Operator authorised running the MTP campaign on idle GPU 3 in parallel with live
|
||||
testing on GPU 2, with the two-card power risk explicitly raised and accepted.
|
||||
|
||||
06:51:08 campaign starts on GPU 3; idle draw GPU0 3.80 / GPU1 3.88 / GPU2 14.27 / GPU3 6.75 W
|
||||
06:54:39 off_A healthy; rep 1 clean
|
||||
06:56:04 off_A rep 2 starts <-- last line ever written
|
||||
06:58:40 EVERY FV address unreachable, BMC included; all other sites 0% loss
|
||||
|
||||
**Leading hypothesis (operator): the 1500 VA Eaton UPS overloaded and died.** It fits
|
||||
better than a breaker trip because a UPS's output rating sits far below the circuit's,
|
||||
making it the first protective device to give — which explains why the site let go at
|
||||
**two** cards loaded rather than four, and why the ~25 W firewall died with it.
|
||||
|
||||
- ⚠ **Will not self-recover.** An overload-tripped UPS needs a human; a dead one needs
|
||||
replacing. Do not poll FV addresses waiting for it.
|
||||
- ⚠ **Do NOT use a UPS's surge-only outlets to exceed its rating.** Both banks are
|
||||
downstream of one input cord/inlet/breaker — for a 120 V 1500 VA unit, a single
|
||||
NEMA 5-15P rated **12 A at maximum load**, total. The surge bank bypasses the
|
||||
inverter, not the current rating. Bypass the UPS entirely instead.
|
||||
- ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` before anything else.**
|
||||
It sampled all four cards every 10 s to the cut and lives on `/tank`, not in a
|
||||
container. **No load figure exists anywhere else** — only idle. Nothing should put a
|
||||
wattage into a UPS purchase until it has been read.
|
||||
- **Blast radius: 19 of 30 gateway aliases**, probed not inferred. ⚠ **There is no
|
||||
local fallback** — every free local model was on fv-ml1; irv-ml1 runs no chat seat at
|
||||
all (TTS/ComfyUI/arbo/clipper on two partly-occupied Ampere cards). The only non-fv
|
||||
chat backends are paid. Any paid coverage must go under a **new opt-in alias name**,
|
||||
never a silent repoint of `summarizer`/`gen`/`classifier`.
|
||||
- ⭐ **The OOB design gap.** OPNsense-as-subnet-router protects box-down/gateway-up and
|
||||
does nothing for a site-wide loss, because the BMC's only route out is through that
|
||||
same gateway. Proposed-not-ratified fix in the runbook: firewall + BMC on the UPS,
|
||||
GPU chassis on raw PDU.
|
||||
|
||||
## Also landed: the renumber the FV cutover missed
|
||||
|
||||
Every fv-ml1 link on the Homepage dashboard was dead — measured **16 entries pointing
|
||||
at 10.250.50.54 and zero at 10.251.50.54**. The sweep script's allowlist was built from
|
||||
files that mention the HOST, and a `homepage.href` mentions only an IP, so every
|
||||
label-only stack fell outside it by construction. Fixed across 24 files plus the host
|
||||
copies; allowlist extended with how to derive it next time. Commit `3132a16`.
|
||||
|
||||
Two bugs fell out: `deploy-stack.sh` rejected any stack name containing a dot (so
|
||||
`qwen3.5-122b`, `qwopus3.5-122b`, `mistral-medium-3.5` could not be deployed **at
|
||||
all**), and scriberr's CORS allowlist contained only the dead IP and the dead
|
||||
`scriberr.ana.internal` — an allowlist with nothing reachable in it.
|
||||
|
||||
⚠ **Still incomplete:** the 10 running containers were never recreated, so the labels
|
||||
are still stale on the host — and a plain power-on will NOT fix them, because labels
|
||||
attach only at creation. The staged `docker compose up -d <svc>` recovery sequence
|
||||
applies them as a side effect. Eight stacks were deliberately NOT pushed from canonical
|
||||
because their host copies have genuine drift; three of those are untracked host-only
|
||||
stacks that should be brought into `stacks/`.
|
||||
+20
-2
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-12 ~19:30 PT (FV cutover EXECUTED — fv-ml1 live at Fountain Valley, renamed/renumbered/serving; BMC online after a VLAN-250 find; four GPUs not two)_
|
||||
_Last updated: 2026-09-13 ~00:10 PT (⚠ FV SITE DARK — UPS suspected overloaded/dead under two-card load; 19 gateway aliases down; visit 2026-09-14. Flash-Next seat + gen-large built and verified before the outage.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -112,7 +112,19 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
## Current state / in-flight
|
||||
_As of 2026-09-12 ~19:30 PT._
|
||||
|
||||
### FV colo — LIVE. Cutover done, one gap open.
|
||||
### ⚠⚠ FV colo — DARK since 2026-09-13 06:56Z. Site visit 2026-09-14.
|
||||
|
||||
Every Fountain Valley address is unreachable, **including the BMC** — the OOB path
|
||||
goes through the same OPNsense gateway, which is also dark. All other sites healthy.
|
||||
Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load.
|
||||
**It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are
|
||||
down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no
|
||||
chat seat). On recovery: recover `power.log` first, mask Docker before the network,
|
||||
then bring seats up one at a time with `compose up -d` — `gen` first, `flash-next`
|
||||
last, and do NOT restart the MTP campaign.
|
||||
→ `docs/runbooks/fv-site-dark-20260913.md`
|
||||
|
||||
### FV colo — the pre-outage state (cutover done, one gap open)
|
||||
- **fv-ml1** (ex ana-ml2) is racked at Fountain Valley, renamed, on `10.251.50.54`, mesh
|
||||
node `100.64.0.7`; **vb-gateway** OPNsense on `10.251.50.1` / `100.64.0.8`; **BMC** on
|
||||
`10.251.250.50`. Public `fv.phasefinal.com` → `172.83.89.66`. `tank` intact, all vLLM
|
||||
@@ -169,6 +181,12 @@ _As of 2026-09-12 ~19:30 PT._
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-13]` ⚠⚠⚠ **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card load test; all other sites healthy. Operator's leading hypothesis: the 1500 VA Eaton UPS overloaded and DIED.** It fits better than a breaker trip because a UPS's output rating sits far below the circuit's, making it the first protective device to give — which explains why the site let go at **two** cards loaded rather than four, and why the ~25 W OPNsense box died with it. ⚠ **Will not self-recover** (tripped needs a human, dead needs replacing) — do NOT poll FV. ⚠ **Do NOT use surge-only outlets to exceed a UPS rating**: both banks share one NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current limit. ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST** — all four cards every 10 s to the cut, on `/tank` not in a container, and the ONLY load measurement that exists. ⚠ **19 of 30 gateway aliases dark and NO local fallback** — every free local model was on fv-ml1, irv-ml1 runs no chat seat at all; the only non-fv chat backends are paid, and any coverage must be a NEW opt-in alias, never a silent repoint. ⭐ **OOB design gap**: OPNsense-as-subnet-router covers box-down/gateway-up and nothing for a site-wide loss, since the BMC's only route out is that gateway. ⚠ Recovery hazard: ten `restart: unless-stopped` vLLM containers will all load at once on power-up — mask Docker first, then `compose up -d` seat by seat (which also finishes the stale-homepage-label fix, since labels attach only at creation). → `docs/runbooks/fv-site-dark-20260913.md`, `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md`
|
||||
- `[2026-09-13]` ⭐⭐ **Qwen3.8-Flash-Next serving on ONE card with its 51B n-gram table in host RAM — the first seat whose weights do not fit its GPU.** `stacks/flash-next-seat/`, fv-ml1 GPU 2 `:8022`, plus a `gen-large` LiteLLM alias. Measured: 74.36 GiB weights resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 context, 67 GiB host RSS, 75.5/212.3/387.8 tok/s at conc 1/4/8 (⚠ n=1). ⭐ The offload is vLLM **#54371 (UVA, merged 2026-09-09)** which **supersedes the paused #53899** — it has no worker process, so #53899's whole bug family (TP=1 deadlock #53960, `pidfd_getfd`/ptrace gate, stale-output-under-graphs) is designed out; in `v0.29.1rc0`, **not** `v0.29.0`. ⚠ **`text_config.ple_embedding_dtype` is the load-or-fail discriminator** for any community build. ⚠⚠ **`--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore `--gpu-memory-utilization`** — 16 GiB nearly OOM'd on a 155K prefill with no visible failure; 14 GiB is the measured-safe value and vLLM's own "17.46 GiB to fully utilize" is 3.5 GiB too high. ⚠ MTP is off **pending measurement here, not written off** — the recipe's number is cross-harness and tested k=3 only, while the head is ONE layer run autoregressively, so k=1 is unpublished and may win (`services/flash-next-mtp-bench/`, one `off_A` rep banked before the outage). ⚠ A container once ran `(healthy)` with `PORTS=[]` — verify `docker port`, not the healthcheck. → `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md`
|
||||
- `[2026-09-13]` **Finished the ana-ml2→fv-ml1 renumber the cutover missed: 16 live Homepage entries pointed at the dead 10.250.50.54 and zero at the live IP.** The sweep allowlist was built from files that mention the HOST and a `homepage.href` mentions only an IP, so every label-only stack fell outside it by construction; 24 files fixed plus host copies, allowlist extended with how to derive it next time. Two bugs fell out: `deploy-stack.sh` **rejected any stack name containing a dot** (so `qwen3.5-122b`/`qwopus3.5-122b`/`mistral-medium-3.5` could not be deployed at all), and scriberr's CORS allowlist held only the dead IP and the dead `scriberr.ana.internal`. ⚠ Incomplete: the 10 running containers were never recreated and a plain power-on will NOT apply labels — the staged `compose up -d` recovery does. Eight stacks deliberately not pushed (real host drift); three of those are untracked host-only stacks. Commits `3132a16`, `969a1b6`, `d79f104`.
|
||||
|
||||
- `[2026-09-13]` **FV→ANA fixed, Beszel18/18 up:** scoped OPNsense hybrid NAT for fv-ml1→ANA; prior NAT/filter rules preserved and rollback-guarded. → `persistent-memory.d/2026-09-13-fv-to-ana-nat.md`
|
||||
|
||||
- `[2026-09-12]` ⭐⭐⭐ **FV CUTOVER EXECUTED — fv-ml1 live at Fountain Valley, renamed, renumbered to 10.251/16, serving inference; BMC online after finding it was tagging 802.1q VLAN 250 into an untagged port; and the box has FOUR RTX PRO 6000 (391 GB VRAM), not the two every doc claimed.** Also: OPNsense write APIs need an `X-CSRFToken` header scraped from a `<script>` block — session cookies alone 403, which is why reboots appeared to work while the operator was power-cycling by hand. → `persistent-memory.d/2026-09-12-fv-cutover-executed.md`
|
||||
|
||||
- `[2026-09-12]` **esh-vm-db Restic fixed:** stale April PG dump caused by TCP auth + masked errors; peer-auth/fail-closed hook and bounded retries installed. Snapshot bc5eeaff and repository check verified. → `persistent-memory.d/2026-09-12-esh-vm-db-restic-repair.md`
|
||||
|
||||
Reference in New Issue
Block a user