installer: a download that resumes and retries, and an install log

A field install failed on "firmware download failed", and worked after a
reboot. The download was one bare curl -fL: no retry, no resume, no bound
on a stalled transfer, and nothing on the machine recorded what had gone
wrong.

The download. download_fw makes up to five tries, 5, 15, 30 and 60 seconds
apart. Each try resumes the partial file (curl -C -) and is bounded: 20 s
to connect, and a transfer below 1 KB/s for 30 s ends the try. The file is
written as forgefirm.fw.part and takes its name only when curl finished;
the signature check that follows is what vouches for its content. A full
disk (curl 23) and a release that is not there (HTTP 404) end the tries at
once, because waiting cannot fix them. A partial file the server will not
resume (curl 33 or 36, HTTP 416) starts over. The loop is the installer's
own rather than curl --retry: the factory curl on the bench reference is
7.69.1, whose --retry does not count a resolver failure or a dropped
transfer as retryable, and older factory builds carry older curls. The
owner sees the reason in words with each retry, and the final failure says
that a re-run goes straight to the download, because the archives are kept.

The log. Every run appends to /data/log/forgefirm/install/install.log, in
the log tree's own line format (UTC, program "install"): the installer's
md5 (which revision ran), the factory version and the slots, the owner's
answers, each archive, each download try with curl's exit code, the HTTP
code and the reason, the machine's clock at each try (a wrong clock breaks
TLS), and after a failed try the address, the default route, the resolver
and whether github.com resolves; then the signature and identity checks,
the write, the boot selection, and the reason for any failure through
die(). The log is appended across runs, so the run that failed is still
there after the run that worked. Logging never fails the install.
forgectrl's log export carries the directory (forgectrl 0dae758).

Proven: tests/test_installer.py runs the installer's own functions under
sh against a scripted curl - a clean download, a resolver failure and a
dropped transfer that resume to the full file, the tries running out, 404
and a full disk ending them at once, a stale partial file starting over,
the TLS reason naming the clock, every log line in the tree format, die()
leaving its reason, and an unwritable log not failing the run. The whole
host suite, 409 tests, passes under Linux and the coverage lint is clean.
Bench: the same functions under the factory firmware's own shell (busybox
1.31.1 ash, the factory slot of the bench reference in a chroot) resumed,
retried, ran out of tries and logged exactly as under sh.

Acceptance: logs.tree-tail-export now plants a probe file in the install
directory and requires it back in the export bundle, its line intact and
its MAC and IPv4 address redacted, and requires an install log in the
bundle when the machine has one. The installer itself is not on the image:
the install page fetches it from master, so it is live with this push.

forgetest/forgetest/suite/logs.py

@@ -27,14 +27,46 @@ def _read(path, default=None):
 _LOG_COVERS = [("forgectrl", "src/logs.*"), ("forgectrl", "src/fflog.*"), ("forgectrl", "src/sanitize.*"),
                ("forgectrl", "src/main.c")]
 
+LOGS_ROOT = "/data/log/forgefirm"
+# What the export test plants in the installer's directory: a line to find
+# again, and two addresses (documentation ranges) the sanitizer must take.
+_PROBE_MARK = "forgetest export probe"
+_PROBE_MAC = "02:00:5e:10:20:30"
+_PROBE_IP = "192.0.2.77"
+
 
 @test("logs.tree-tail-export", title="Log tree, tail, and sanitized export", subsystem="logs",
       kind="auto", est_min=1,
       covers=_LOG_COVERS, requires=["forgectrl.auth"],
       description="/logs lists the loggers with their levels and files, /logs/tail returns the "
                   "forgectrl logger's tail, and POST /logs/export streams a sanitized tar.gz "
-                  "bundle that contains neither the panel token nor the camera key.")
+                  "bundle that contains neither the panel token nor the camera key. The bundle "
+                  "also carries the installer's directory of the tree (logs/install/), which no "
+                  "logger feeds: a probe file planted there for the export comes back in the "
+                  "bundle with its addresses redacted, and an install log that the installer "
+                  "left is in the bundle too.")
 def tree_tail_export(ctx):
+    install_dir = os.path.join(LOGS_ROOT, "install")
+    probe = os.path.join(install_dir, "forgetest-probe.txt")
+    made_dir = not os.path.isdir(install_dir)
+    try:
+        os.makedirs(install_dir, exist_ok=True)
+        with open(probe, "w") as f:
+            f.write("%s from %s at %s\n" % (_PROBE_MARK, _PROBE_MAC, _PROBE_IP))
+        _tree_tail_export(ctx, probe)
+    finally:
+        try:
+            os.unlink(probe)
+        except OSError:
+            pass
+        if made_dir:
+            try:
+                os.rmdir(install_dir)
+            except OSError:
+                pass
+
+
+def _tree_tail_export(ctx, probe):
     fc = ctx.forgectrl
     ev = ctx.evidence
     st, body = fc.get("/logs")
@@ -87,6 +119,21 @@ def tree_tail_export(ctx):
         ev[what.replace(" ", "_") + "_leaks"] = leaked
         ctx.check(not leaked, "the sanitized bundle contains the %s: %s", what, leaked)
 
+    # the installer's directory: no logger feeds it, the export carries it
+    by_name = dict(contents)
+    want = "logs/install/" + os.path.basename(probe)
+    got = [name for name in by_name if name.endswith(want)]
+    ev["install_members"] = sorted(name for name in by_name if "/logs/install/" in name)
+    ctx.check(got, "the bundle lacks %s: %s", want, ev["install_members"])
+    if got:
+        text = by_name[got[0]].decode("utf-8", "replace")
+        ctx.check(_PROBE_MARK in text, "the probe file came back without its line: %r", text)
+        for what, value in (("MAC address", _PROBE_MAC), ("IPv4 address", _PROBE_IP)):
+            ctx.check(value not in text, "the install directory left the sanitizer with its %s", what)
+    if os.path.isfile(os.path.join(os.path.dirname(probe), "install.log")):
+        ctx.check(any(name.endswith("logs/install/install.log") for name in by_name),
+                  "the machine has an install log and the bundle does not")
+
 
 # The routing test proves the whole path every logger takes: emitter (or
 # relay) -> /dev/log -> rsyslog rules rendered from the settings -> the
@@ -100,8 +147,7 @@ _ROUTING_COVERS = [("forgectrl", "src/logs.*"), ("forgectrl", "src/fflog.*"), ("
                    ("forgefirm-app", "forgefirm-app/ffmachine.py"), ("forgefirm-app", "forgefirm-app/gfcloud.py"),
                    ("forgefirm-app", "forgefirm-app/gfhome.py")]
 
-LOGS_ROOT = "/data/log/forgefirm"
-LOGGERS = ("forgectrl", "grblhal", "gfcloud", "gfhome", "kernel", "system")
+LOGGERS =("forgectrl", "grblhal", "gfcloud", "gfhome", "kernel", "system")
 _LINE_RE = re.compile(r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d(\.\d+)?[+-]\d\d:\d\d (?P<prog>[A-Za-z0-9_.-]+)\[(?P<pid>[-\d]+)\] "
                       r"(?P<sev>EMERG|ALERT|CRIT|ERR|WARNING|NOTICE|INFO|DEBUG) (?P<msg>.*)$")
 _SEV_RANK = {"off": -1, "error": 3, "warning": 4, "notice": 5, "info": 6, "debug": 7}