esh-pfi-infrastructure/stacks/news-digest/README.md

# news-digest

Twice-daily LLM-curated briefing across Reddit (via JSON API) and
Miniflux's Tech aggregators category. Output is a single static HTML
page styled in the Australis design system with editorial-serif
headlines (Fraunces) — operations-center chrome wrapping news content.

## Why this stack exists

After a few subreddits + HN + Lobste.rs, the Miniflux inbox gets
noisy. This stack:

1. Pulls **top-of-day** posts per subreddit from Reddit's public JSON
   API (gives us scores + upvote ratios — RSS doesn't).
2. Filters by `score >= 50` and `upvote_ratio >= 0.85` (configurable)
   to drop flame-bait and low-effort posts.
3. Pulls non-Reddit recent items from Miniflux's Tech aggregators
   category (HN, Lobste.rs).
4. Sends each source through `granite-4-small` on llama-swap (one
   batched call per source — efficient) for a 2-3 sentence summary +
   single-word tag (news / tutorial / release / discussion / question
   / showcase / drama / meme / other). World + Local headlines also
   get summarized in one batch call per category. (Original default
   was `qwen3.5-35-a3b` but its model file is broken on launch.)
5. Renders an HTML page styled with Australis tokens + Fraunces
   serif headlines.
6. The page is served by a tiny FastAPI app on uvicorn that also
   exposes `/api/{hidden,hide,restore}` for the per-item × button
   (state in `/output/hidden.json`, shared across every device the
   user opens the digest from).

Two editions per day by default (0800 / 2000 local), parametrized via
`DIGEST_CRON_AM` / `DIGEST_CRON_PM` env so each per-user instance can
fire on its own schedule. Plus per-edition archives at
`/edition-YYYY-MM-DD-{am,pm}.html`.

## Architecture

Two containers built from the same Dockerfile, both on `traefik-net`,
sharing a bind-mounted output dir:

```
news-digest-worker  (default ENTRYPOINT — busybox crond)
  ├── busybox crond fires at 0 8,20 * * *
  ├── digest.py:
  │     ├── miniflux /v1/feeds → discover subreddits
  │     ├── reddit JSON top/.json?t=day per sub (gentle 1.5s sleep)
  │     ├── miniflux /v1/entries → tech aggregators
  │     ├── llama-swap /v1/chat/completions → batched per source
  │     └── jinja2 render → /output/index.html (atomic .tmp + rename)
  │                       → /output/edition-2026-04-26-pm.html
  └── style.css / favicon.svg / app.js staged in /output at deploy

news-digest-web     (entrypoint overridden → uvicorn web:app)
  ├── /          → serve /output as static (index.html as default)
  ├── /api/hidden        GET   → JSON array of hidden item ids
  ├── /api/hide          POST  → {id} → adds id to hidden.json
  ├── /api/restore       POST  → {id} → removes id from hidden.json
  ├── /output/hidden.json — durable state (atomic writes + threading lock)
  └── homepage card via container labels (group=News)
```

Hidden state is server-side and global per-user (single-user setup):
hide an article once and it stays hidden in any future edition that
includes the same article. The "Hidden (N)" tray at the bottom of
each page shows items hidden FROM THE CURRENT PAGE; older hidden ids
that aren't present on this page just sit silently in `hidden.json`
and continue to filter future editions.

## Deploy

```bash
scripts/elway ana-docker --playbook playbooks/deploy-news-digest.yaml
```

After first deploy, **fill in MINIFLUX_PASSWORD on the host**:

```bash
ssh ana-docker '
  cd /opt/docker/compose/news-digest
  sed -i "s|^MINIFLUX_PASSWORD=.*|MINIFLUX_PASSWORD=<your-miniflux-admin-password>|" .env
  docker compose up -d
'
```

The container runs the first digest immediately if `/output/index.html`
doesn't exist, so the page populates within a minute or two of bringing
the stack up with real credentials.

Visit <http://10.250.50.70:8181> to read.

## Tuning the noise floor

Defaults in `.env.example`:

| Knob | Default | Effect |
|---|---|---|
| `DIGEST_REDDIT_HOURS` | 12 | Look-back window (matches twice-daily cadence) |
| `DIGEST_MIN_SCORE` | 50 | Reddit minimum upvotes to consider |
| `DIGEST_MIN_RATIO` | 0.85 | Reddit minimum upvote ratio (skips flamebait) |
| `DIGEST_MAX_PER_SUB` | 8 | Cap per subreddit, post-filter |
| `DIGEST_MINIFLUX_HOURS` | 12 | Look-back window for HN/Lobste.rs |
| `DIGEST_MINIFLUX_MAX` | 8 | Cap per non-Reddit feed |

For a busier day, lower `DIGEST_MIN_SCORE`. For a quieter morning
edition, raise it. Edit `.env`, no rebuild needed — the worker reads
env on each cron fire.

## Adding more subreddits

The digest picks up subreddit feeds from Miniflux automatically — any
feed whose URL starts with `https://www.reddit.com/r/<sub>/` gets
queried. To add a sub, just subscribe in Miniflux (UI or API). The
next digest run includes it.

## Updating the LLM model

```bash
ssh ana-docker '
  cd /opt/docker/compose/news-digest
  sed -i "s|^LLAMA_SWAP_MODEL=.*|LLAMA_SWAP_MODEL=<new-model>|" .env
  docker compose up -d
'
```

The model must be loaded in llama-swap's `config.yaml`. Check
`http://10.250.50.54:9292/v1/models` for what's available. Models
with tool/JSON-mode support give better summarization quality;
`granite-4-small` is the current default (small ~4B, fast ~1s/call,
no extended-thinking phase eating the token budget). Avoid
`qwen3.5-35-a3b` (model file broken — process exits on launch) and
`qwen3.6-35-a3b` (defaults to thinking mode, eats budget without output).

## Forcing a fresh digest now

```bash
ssh ana-docker 'docker exec news-digest-worker /usr/local/bin/run-digest.sh'
```

Runs the full pipeline once, ignoring cron. Useful after changing
filtering knobs or adding feeds.

## Customizing the run schedule

Times come from two env vars on the worker, written into the busybox
crontab at container start. Standard 5-field cron syntax.

| Var | Default | Effect |
|---|---|---|
| `DIGEST_CRON_AM` | `0 8 * * *` | morning fire |
| `DIGEST_CRON_PM` | `0 20 * * *` | evening fire |

`busybox crond` honors `$NEWS_DIGEST_TZ` (defaults to
`America/Los_Angeles`), so values are interpreted in the configured TZ.

```bash
# Shift the canonical instance to 7am / 6pm
ssh ana-docker '
  cd /opt/docker/compose/news-digest
  sed -i "s|^DIGEST_CRON_AM=.*|DIGEST_CRON_AM=0 7 * * *|" .env
  sed -i "s|^DIGEST_CRON_PM=.*|DIGEST_CRON_PM=0 18 * * *|" .env
  docker compose up -d --force-recreate news-digest-worker
'
```

Verify the rendered crontab:

```bash
ssh ana-docker 'docker exec news-digest-worker cat /etc/crontabs/root'
```

## Multi-tenant: one instance per teammate

Architecture: **shared miniflux + per-user digest stack**. Miniflux
already supports multi-user natively (each user has their own feeds,
categories, hide-state); we layer a separate news-digest stack per
user on its own port + output dir, scoped to that miniflux user's
credentials.

### Onboarding a new user

```bash
# Defaults — 8am / 8pm local, random password
scripts/add-digest-user.sh alice

# Custom hours
scripts/add-digest-user.sh bob --am "0 6 * * *" --pm "0 17 * * *"

# Weekday-only PM run
scripts/add-digest-user.sh carol --pm "30 18 * * 1-5"

# Pin a known password (still creates the miniflux user if missing)
scripts/add-digest-user.sh dan 'pickyourpassword' --am "0 9 * * *"
```

What the script does:

1. Reads miniflux admin creds from
   `ana-docker:/opt/docker/compose/miniflux/.env`.
2. Allocates the next free `NEWS_DIGEST_PORT` (scans existing
   `news-digest` + `digest-*` `.env` files).
3. Creates the miniflux user via the admin API. Already-exists is
   non-fatal (kept; password not reset).
4. Provisions per-user dirs at
   `/opt/docker/compose/digest-<user>/` and
   `/opt/docker/data/digest-<user>/` (one-time sudo prompt — the
   script falls back to printing the manual command if there's no TTY).
5. Materializes a per-user `.env` (inherits `NEWS_DIGEST_TAG` from the
   canonical stack so all tenants run the same image).
6. Brings the stack up via `docker compose -p digest-<user> up -d`.
7. Runs `seed-headlines.py` against miniflux as the new user (creates
   the World + Local categories with default feeds).
8. Triggers a first digest run so the page isn't blank.

Outputs the digest URL, miniflux login, and rendered cron schedule.

### Per-user file layout

```
ana-docker:
  /opt/docker/compose/digest-<user>/   # compose + .env + build context
    .env                               # auto-generated, contains MINIFLUX_PASSWORD
    compose.yaml
    Dockerfile + digest.py + ...       # build context (image is shared/cached)
  /opt/docker/data/digest-<user>/      # rendered HTML + per-user hidden.json
    index.html
    edition-YYYY-MM-DD-{am,pm}.html
    hidden.json
    .article-cache.json                # extracted article text, 7-day TTL
```

Container names: `digest-<user>-worker` and `digest-<user>-web`.

### Updating an existing user's schedule

Re-running the script with the same username is idempotent:

```bash
# Change alice's evening run to 5:30pm
scripts/add-digest-user.sh alice --pm "30 17 * * *"
```

The miniflux user is kept (password unchanged), the .env is
re-materialized with the new schedule, and the worker container is
recreated. Use the same flow to bump cron times, rotate passwords
(by passing a new one explicitly), or rerun feed seeding.

### Removing a user

```bash
ssh ana-docker '
  cd /opt/docker/compose/digest-alice
  docker compose -p digest-alice down -v
'
# Optional: nuke compose dir + rendered output
ssh -t ana-docker 'sudo rm -rf /opt/docker/compose/digest-alice /opt/docker/data/digest-alice'
# Optional: delete the miniflux user via the admin UI at http://10.250.50.70:8080/
```

The digest is gone immediately; the miniflux account stays around
unless you delete it explicitly (cheap to leave; ~zero resource cost
when no stack is querying its feeds).

## Logs

```bash
ssh ana-docker 'docker logs --tail 100 news-digest-worker'
```

Worker logs each phase (subreddit discovery / fetching / summarizing /
rendering) with timestamps. Per-source LLM filter results show how
many items were kept vs skipped.

## License + attribution

Reddit content surfaced here is owned by its authors and Reddit. The
digest is a derived index pointing at original sources — every item
links back to the Reddit thread (and to the external link if the
post linked out). Same for HN / Lobste.rs.