Merge pull request #359 from restic/dependabot/github_actions/actions/setup-go-6

Bump actions/setup-go from 5 to 6

.github/ISSUE_TEMPLATE/BUG.md

@@ -3,8 +3,8 @@ name: Bug report
 about: Report a problem with rest-server to help us resolve it and improve
 ---
 
-
 <!--
+
 Welcome! - We kindly ask that you:
 
   1. Fill out the issue template below - not doing so needs a good reason.
@@ -24,6 +24,7 @@ for tracking bugs and feature requests directly relating to the development of
 the software itself, rather than the project.
 
 Thanks for understanding, and for contributing to the project!
+
 -->
 
 
@@ -31,24 +32,26 @@ Output of `rest-server --version` <!-- If using docker, output of `docker images
 ---------------------------------
 
 
-How did you run rest-server exactly?
-------------------------------------
+
+Problem description / Steps to reproduce
+----------------------------------------
 
 <!--
 This section should include at least:
 
+ * A description of the problem you are having with rest-server.
+
  * The complete command line and any environment variables you used to
-   configure rest-server's backend access. Make sure to replace sensitive values!
+   configure rest-server. Make sure to replace sensitive values!
 
  * The output of the commands, what rest-server prints gives may give us much
    information to diagnose the problem!
+
+ * The more time you spend describing an easy way to reproduce the behavior (if
+   this is possible), the easier it is for the project developers to fix it!
 -->
 
 
-What backend/server/service did you use to store the repository?
-----------------------------------------------------------------
-
-
 Expected behavior
 -----------------
 
@@ -56,30 +59,20 @@ Expected behavior
 Describe what you'd like rest-server to do differently.
 -->
 
-
 Actual behavior
 ---------------
 
 <!--
-Please try to concentrate on observations, so only describe what you observed directly.
+In this section, please try to concentrate on observations, so only describe
+what you observed directly.
 -->
 
-
-Steps to reproduce the behavior
--------------------------------
-
-<!--
-The more time you spend describing an easy way to reproduce the behavior (if
-this is possible), the easier it is for the project developers to fix it!
--->
-
-
 Do you have any idea what may have caused this?
 -----------------------------------------------
 
-
-Do you have an idea how to solve the issue?
--------------------------------------------
+<!--
+Did something noteworthy happen on your system, Internet connection, backend services, etc?
+-->
 
 
 Did rest-server help you today? Did it make you happy in any way?

.github/ISSUE_TEMPLATE/FEATURE.md

@@ -1,10 +1,10 @@
 ---
-name: Feature request/enhancement
+name: Feature request
 about: Suggest a new feature or enhancement for rest-server
 ---
 
-
 <!--
+
 Welcome! - We kindly ask that you:
 
   1. Fill out the issue template below - not doing so needs a good reason.
@@ -18,6 +18,7 @@ for tracking bugs and feature requests directly relating to the development of
 the software itself, rather than the project.
 
 Thanks for understanding, and for contributing to the project!
+
 -->
 
 
@@ -30,24 +31,22 @@ later to see what has changed in rest-server when we revisit this issue after so
 time.
 -->
 
-
-What should rest-server do differently?
----------------------------------------
+What should rest-server do differently? Which functionality do you think we should add?
+---------------------------------------------------------------------------------------
 
 <!--
-Please describe the feature you'd like to see added or changed here.
+Please describe the feature you'd like us to add here.
 -->
 
 
-What are you trying to do? What is your use case?
--------------------------------------------------
+What are you trying to do? What problem would this solve?
+---------------------------------------------------------
 
 <!--
 This section should contain a brief description what you're trying to do, which
 would be possible after implementing the new feature.
 -->
 
-
 Did rest-server help you today? Did it make you happy in any way?
 -----------------------------------------------------------------
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,42 +1,41 @@
 <!--
-Thank you very much for contributing code or documentation to rest-server!
-
-Please note that each PR should be preceded by an issue where the suggested
-change can be discussed in general and without focus on specific code. That
-way, work done in the PR will better match what's been agreed in the issue.
-
-Please fill out the following questions to make it easier for us to review
-your changes. You don't have to check all the checkboxes at once, instead
-feel free to add more commits over time.
+Thank you very much for contributing code or documentation to rest-server! Please
+fill out the following questions to make it easier for us to review your
+changes.
 -->
 
-
-What is the purpose of this change? What does it change?
---------------------------------------------------------
+What does this PR change? What problem does it solve?
+-----------------------------------------------------
 
 <!--
-Describe the changes here, as detailed as needed.
+Describe the changes and their purpose here, as detailed as needed.
 -->
 
-
-Was the change discussed in an issue or in the forum before?
-------------------------------------------------------------
+Was the change previously discussed in an issue or on the forum?
+----------------------------------------------------------------
 
 <!--
 Link issues and relevant forum posts here.
 
-If this PR resolves an issue on GitHub, write "Closes #1234" such
-that the issue is closed automatically when this PR is merged.
+If this PR resolves an issue on GitHub, use "Closes #1234" so that the issue
+is closed automatically when this PR is merged.
 -->
 
-
 Checklist
 ---------
 
-- [ ] I have enabled [maintainer edits for this PR](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
-- [ ] I have added tests for all changes in this PR
-- [ ] I have added documentation for the changes (in the manual)
-- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (template [here](https://github.com/restic/rest-server/blob/master/changelog/TEMPLATE))
-- [ ] I have run `gofmt` on the code in all commits
-- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/rest-server/commits/master)
-- [ ] I'm done, this Pull Request is ready for review
+<!--
+You do not need to check all the boxes below all at once. Feel free to take
+your time and add more commits. If you're done and ready for review, please
+check the last box. Enable a checkbox by replacing [ ] with [x].
+
+Please always follow these steps:
+- Enable [maintainer edits](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
+- Run `gofmt` on the code in all commits.
+- Format all commit messages in the same style as [the other commits in the repository](https://github.com/restic/rest-server/blob/master/CONTRIBUTING.md#git-commits).
+-->
+
+- [ ] I have added tests for all code changes.
+- [ ] I have added documentation for relevant changes (in the manual).
+- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (see [template](https://github.com/restic/rest-server/blob/master/changelog/TEMPLATE)).
+- [ ] I'm done! This pull request is ready for review.

.github/dependabot.yml

@@ -0,0 +1,13 @@
+version: 2
+updates:
+  # Dependencies listed in go.mod
+  - package-ecosystem: "gomod"
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"
+
+  # Dependencies listed in .github/workflows/*.yml
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"

.github/workflows/linters.yml

@@ -1,25 +0,0 @@
-name: Style Checkers
-on: [push, pull_request]
-jobs:
-  build:
-    name: Check
-    runs-on: ubuntu-latest
-    steps:
-
-      - name: Set up Go
-        uses: actions/setup-go@v1
-        with:
-          go-version: 1.15.x
-        id: go
-
-      - name: Install golang-ci
-        run: |
-          curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $HOME/bin latest
-
-      - name: Check out code
-        uses: actions/checkout@v1
-
-      - name: Run golang-ci
-        run: |
-          export PATH=$HOME/bin:$PATH
-          golangci-lint run --enable goimports,govet,misspell,stylecheck

.github/workflows/tests.yml

@@ -1,36 +1,119 @@
-name: Build and tests
-on: [push, pull_request]
+name: test
+on:
+  # run tests on push to master, but not when other branches are pushed to
+  push:
+    branches:
+      - master
+
+  # run tests for all pull requests
+  pull_request:
+  merge_group:
+
+permissions:
+  contents: read
+
+env:
+  latest_go: "1.24.x"
+  GO111MODULE: on
+
 jobs:
-  build:
-    name: Build
+  test:
     strategy:
       matrix:
-        go-version:
-          - 1.14.x
-          - 1.15.x
-    runs-on: ubuntu-latest
+        include:
+          - job_name: Linux
+            go: 1.24.x
+            os: ubuntu-latest
+            check_changelog: true
+
+          - job_name: Linux (race)
+            go: 1.24.x
+            os: ubuntu-latest
+            test_opts: "-race"
+
+          - job_name: Linux
+            go: 1.23.x
+            os: ubuntu-latest
+
+    name: ${{ matrix.job_name }} Go ${{ matrix.go }}
+    runs-on: ${{ matrix.os }}
 
     env:
       GOPROXY: https://proxy.golang.org
 
     steps:
-
-      - name: Set up Go ${{ matrix.go-version }}
-        uses: actions/setup-go@v1
-        with:
-          go-version: ${{ matrix.go-version }}
-        id: go
-
       - name: Check out code
-        uses: actions/checkout@v1
+        uses: actions/checkout@v5
 
-      - name: Build
+      - name: Set up Go ${{ matrix.go }}
+        uses: actions/setup-go@v6
+        with:
+          go-version: ${{ matrix.go }}
+
+      - name: Build with build.go
         run: |
           go run build.go --goos linux
           go run build.go --goos windows
           go run build.go --goos darwin
 
-      - name: Run tests
+      - name: Run local Tests
         run: |
-          export PATH=$HOME/bin:$PATH
-          go test ./...
+          go test -cover ${{matrix.test_opts}} ./...
+
+      - name: Check changelog files with calens
+        run: |
+          echo "install calens"
+          go install github.com/restic/calens@latest
+
+          echo "check changelog files"
+          calens
+        if: matrix.check_changelog
+
+  lint:
+    name: lint
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      # allow annotating code in the PR
+      checks: write
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v5
+
+      - name: Set up Go ${{ env.latest_go }}
+        uses: actions/setup-go@v6
+        with:
+          go-version: ${{ env.latest_go }}
+
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@v6
+        with:
+          # Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
+          version: v1.64.8
+          args: --verbose --timeout 5m
+
+        # only run golangci-lint for pull requests, otherwise ALL hints get
+        # reported. We need to slowly address all issues until we can enable
+        # linting the master branch :)
+        if: github.event_name == 'pull_request'
+
+      - name: Check go.mod/go.sum
+        run: |
+          echo "check if go.mod and go.sum are up to date"
+          go mod tidy
+          git diff --exit-code go.mod go.sum
+
+  analyze:
+    name: Analyze results
+    needs: [test, lint]
+    if: always()
+
+    permissions: # no need to access code
+      contents: none
+
+    runs-on: ubuntu-latest
+    steps:
+      - name: Decide whether the needed jobs succeeded or failed
+        uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe
+        with:
+          jobs: ${{ toJSON(needs) }}

.golangci.yml

@@ -0,0 +1,56 @@
+# This is the configuration for golangci-lint for the restic project.
+#
+# A sample config with all settings is here:
+# https://github.com/golangci/golangci-lint/blob/master/.golangci.example.yml
+
+linters:
+  # only enable the linters listed below
+  disable-all: true
+  enable:
+    # make sure all errors returned by functions are handled
+    - errcheck
+
+    # show how code can be simplified
+    - gosimple
+
+    # make sure code is formatted
+    - gofmt
+
+    # examine code and report suspicious constructs, such as Printf calls whose
+    # arguments do not align with the format string
+    - govet
+
+    # make sure names and comments are used according to the conventions
+    - revive
+
+    # detect when assignments to existing variables are not used
+    - ineffassign
+
+    # run static analysis and find errors
+    - staticcheck
+
+    # find unused variables, functions, structs, types, etc.
+    - unused
+
+    # parse and typecheck code
+    - typecheck
+
+    # ensure that http response bodies are closed
+    - bodyclose
+
+    - importas
+
+issues:
+  # don't use the default exclude rules, this hides (among others) ignored
+  # errors from Close() calls
+  exclude-use-default: false
+
+  # list of things to not warn about
+  exclude:
+    # revive: do not warn about missing comments for exported stuff
+    - exported (function|method|var|type|const) .* should have comment or be unexported
+    # revive: ignore constants in all caps
+    - don't use ALL_CAPS in Go names; use CamelCase
+    # revive: lots of packages don't have such a comment
+    - "package-comments: should have a package comment"
+    - "redefines-builtin-id:"

.goreleaser.yml

@@ -1,4 +1,6 @@
 ---
+version: 2
+
 before:
   # Run a few commands to check the state of things. When anything is changed
   # in files commited to the repo, goreleaser will abort before building
@@ -19,37 +21,36 @@ before:
 
 # build a single binary
 builds:
-  -
+  - id: default
     # make sure everything is statically linked by disabling cgo altogether
-    env:
+    env: &build_env
       - CGO_ENABLED=0
 
     # set the package for the main binary
     main: ./cmd/rest-server
 
     flags:
-      # don't include any paths to source files in the resulting binary
+      &build_flags # don't include any paths to source files in the resulting binary
       - -trimpath
 
-    ldflags:
-      # set the version variable in the main package
+    mod_timestamp: "{{ .CommitTimestamp }}"
+
+    ldflags: &build_ldflags # set the version variable in the main package
       - "-s -w -X main.version={{ .Version }}"
 
     # list all operating systems and architectures we build binaries for
     goos:
       - linux
       - darwin
-      - windows
       - freebsd
       - netbsd
       - openbsd
       - dragonfly
-      - plan9
       - solaris
 
     goarch:
       - amd64
-      - 386
+      - "386"
       - arm
       - arm64
       - mips
@@ -58,20 +59,55 @@ builds:
       - ppc64
       - ppc64le
     goarm:
-      - 6
-      - 7
+      - "6"
+      - "7"
+
+  - id: windows-only
+    env: *build_env
+    main: ./cmd/rest-server
+    flags: *build_flags
+    mod_timestamp: "{{ .CommitTimestamp }}"
+    ldflags: *build_ldflags
+    goos:
+      - windows
+    goarch:
+      - amd64
+      - "386"
+      - arm
+      - arm64
 
 # configure the resulting archives to create
 archives:
-  -
+  - id: default
+    builds: [default, windows-only]
+    format: tar.gz
     # package a directory which contains the source file
     wrap_in_directory: true
 
+    builds_info: &archive_file_info
+      owner: root
+      group: root
+      mtime: "{{ .CommitDate }}"
+      mode: 0644
+
     # add these files to all archives
-    files:
-      - LICENSE
-      - README.md
-      - CHANGELOG.md
+    files: &archive_files
+      - src: LICENSE
+        dst: LICENSE
+        info: *archive_file_info
+      - src: README.md
+        dst: README.md
+        info: *archive_file_info
+      - src: CHANGELOG.md
+        dst: CHANGELOG.md
+        info: *archive_file_info
+
+  - id: windows-only
+    builds: [windows-only]
+    formats: [zip]
+    wrap_in_directory: true
+    builds_info: *archive_file_info
+    files: *archive_files
 
 # also build an archive of the source code
 source:
@@ -79,7 +115,7 @@ source:
 
 # build a file containing the SHA256 hashes
 checksum:
-  name_template: 'SHA256SUMS'
+  name_template: "SHA256SUMS"
 
 # sign the checksum file
 signs:
@@ -92,18 +128,117 @@ signs:
       - "--detach-sign"
       - "${artifact}"
 
-# do not generate a changelog file, we're using calens for that
-changelog:
-  skip: true
-
 # configure building the rest-server docker image
 dockers:
   - image_templates:
-      - restic/rest-server:latest
-      - restic/rest-server:{{ .Version }}
+      - restic/rest-server:{{ .Version }}-amd64
     build_flag_templates:
+      - "--platform=linux/amd64"
       - "--pull"
-    extra_files:
+      - "--label=org.opencontainers.image.created={{.Date}}"
+      - "--label=org.opencontainers.image.title={{.ProjectName}}"
+      - "--label=org.opencontainers.image.source=https://github.com/restic/{{ .ProjectName }}"
+      - "--label=org.opencontainers.image.revision={{.FullCommit}}"
+      - "--label=org.opencontainers.image.version={{.Version}}"
+      - "--label=org.opencontainers.image.licenses=BSD-2-Clause"
+    use: buildx
+    dockerfile: "Dockerfile.goreleaser"
+    extra_files: &extra_files
       - docker/create_user
       - docker/delete_user
       - docker/entrypoint.sh
+  - image_templates:
+      - restic/rest-server:{{ .Version }}-i386
+    goarch: "386"
+    build_flag_templates:
+      - "--platform=linux/386"
+      - "--pull"
+      - "--label=org.opencontainers.image.created={{.Date}}"
+      - "--label=org.opencontainers.image.title={{.ProjectName}}"
+      - "--label=org.opencontainers.image.source=https://github.com/restic/{{ .ProjectName }}"
+      - "--label=org.opencontainers.image.revision={{.FullCommit}}"
+      - "--label=org.opencontainers.image.version={{.Version}}"
+      - "--label=org.opencontainers.image.licenses=BSD-2-Clause"
+    use: buildx
+    dockerfile: "Dockerfile.goreleaser"
+    extra_files: *extra_files
+  - image_templates:
+      - restic/rest-server:{{ .Version }}-arm32v6
+    goarch: arm
+    goarm: 6
+    build_flag_templates:
+      - "--platform=linux/arm/v6"
+      - "--pull"
+      - "--label=org.opencontainers.image.created={{.Date}}"
+      - "--label=org.opencontainers.image.title={{.ProjectName}}"
+      - "--label=org.opencontainers.image.source=https://github.com/restic/{{ .ProjectName }}"
+      - "--label=org.opencontainers.image.revision={{.FullCommit}}"
+      - "--label=org.opencontainers.image.version={{.Version}}"
+      - "--label=org.opencontainers.image.licenses=BSD-2-Clause"
+    use: buildx
+    dockerfile: "Dockerfile.goreleaser"
+    extra_files: *extra_files
+  - image_templates:
+      - restic/rest-server:{{ .Version }}-arm32v7
+    goarch: arm
+    goarm: 7
+    build_flag_templates:
+      - "--platform=linux/arm/v7"
+      - "--pull"
+      - "--label=org.opencontainers.image.created={{.Date}}"
+      - "--label=org.opencontainers.image.title={{.ProjectName}}"
+      - "--label=org.opencontainers.image.source=https://github.com/restic/{{ .ProjectName }}"
+      - "--label=org.opencontainers.image.revision={{.FullCommit}}"
+      - "--label=org.opencontainers.image.version={{.Version}}"
+      - "--label=org.opencontainers.image.licenses=BSD-2-Clause"
+    use: buildx
+    dockerfile: "Dockerfile.goreleaser"
+    extra_files: *extra_files
+  - image_templates:
+      - restic/rest-server:{{ .Version }}-arm64v8
+    goarch: arm64
+    build_flag_templates:
+      - "--platform=linux/arm64/v8"
+      - "--pull"
+      - "--label=org.opencontainers.image.created={{.Date}}"
+      - "--label=org.opencontainers.image.title={{.ProjectName}}"
+      - "--label=org.opencontainers.image.source=https://github.com/restic/{{ .ProjectName }}"
+      - "--label=org.opencontainers.image.revision={{.FullCommit}}"
+      - "--label=org.opencontainers.image.version={{.Version}}"
+      - "--label=org.opencontainers.image.licenses=BSD-2-Clause"
+    use: buildx
+    dockerfile: "Dockerfile.goreleaser"
+    extra_files: *extra_files
+  - image_templates:
+      - restic/rest-server:{{ .Version }}-ppc64le
+    goarch: ppc64le
+    build_flag_templates:
+      - "--platform=linux/ppc64le"
+      - "--pull"
+      - "--label=org.opencontainers.image.created={{.Date}}"
+      - "--label=org.opencontainers.image.title={{.ProjectName}}"
+      - "--label=org.opencontainers.image.source=https://github.com/restic/{{ .ProjectName }}"
+      - "--label=org.opencontainers.image.revision={{.FullCommit}}"
+      - "--label=org.opencontainers.image.version={{.Version}}"
+      - "--label=org.opencontainers.image.licenses=BSD-2-Clause"
+    use: buildx
+    dockerfile: "Dockerfile.goreleaser"
+    extra_files: *extra_files
+
+docker_manifests:
+  - name_template: "restic/rest-server:{{ .Version }}"
+    image_templates:
+      - "restic/rest-server:{{ .Version }}-amd64"
+      - "restic/rest-server:{{ .Version }}-i386"
+      - "restic/rest-server:{{ .Version }}-arm32v6"
+      - "restic/rest-server:{{ .Version }}-arm32v7"
+      - "restic/rest-server:{{ .Version }}-arm64v8"
+      - "restic/rest-server:{{ .Version }}-ppc64le"
+  - name_template: "restic/rest-server:latest"
+    image_templates:
+      - "restic/rest-server:{{ .Version }}-amd64"
+      - "restic/rest-server:{{ .Version }}-i386"
+      - "restic/rest-server:{{ .Version }}-arm32v6"
+      - "restic/rest-server:{{ .Version }}-arm32v7"
+      - "restic/rest-server:{{ .Version }}-arm64v8"
+      - "restic/rest-server:{{ .Version }}-ppc64le"

CHANGELOG.md

@@ -1,3 +1,451 @@
+Changelog for rest-server 0.14.0 (2025-05-31)
+============================================
+
+The following sections list the changes in rest-server 0.14.0 relevant
+to users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Sec #318: Fix world-readable permissions on new `.htpasswd` files
+ * Chg #322: Update dependencies and require Go 1.23 or newer
+ * Enh #174: Support proxy-based authentication
+ * Enh #189: Support group accessible repositories
+ * Enh #295: Output status of append-only mode on startup
+ * Enh #315: Hardened tls settings
+ * Enh #321: Add zip archive format for Windows releases
+
+Details
+-------
+
+ * Security #318: Fix world-readable permissions on new `.htpasswd` files
+
+   On startup the rest-server Docker container creates an empty `.htpasswd` file if
+   none exists yet. This file was world-readable by default, which can be a
+   security risk, even though the file only contains hashed passwords.
+
+   This has been fixed such that new `.htpasswd` files are no longer
+   world-readabble.
+
+   The permissions of existing `.htpasswd` files must be manually changed if
+   relevant in your setup.
+
+   https://github.com/restic/rest-server/issues/318
+   https://github.com/restic/rest-server/pull/340
+
+ * Change #322: Update dependencies and require Go 1.23 or newer
+
+   All dependencies have been updated. Rest-server now requires Go 1.23 or newer to
+   build.
+
+   This also disables support for TLS versions older than TLS 1.2. On Windows,
+   rest-server now requires at least Windows 10 or Windows Server 2016. On macOS,
+   rest-server now requires at least macOS 11 Big Sur.
+
+   https://github.com/restic/rest-server/pull/322
+   https://github.com/restic/rest-server/pull/338
+
+ * Enhancement #174: Support proxy-based authentication
+
+   Rest-server now supports authentication via HTTP proxy headers. This feature can
+   be enabled by specifying the username header using the `--proxy-auth-username`
+   option (e.g., `--proxy-auth-username=X-Forwarded-User`).
+
+   When enabled, the server authenticates users based on the specified header and
+   disables Basic Auth. Note that proxy authentication is disabled when `--no-auth`
+   is set.
+
+   https://github.com/restic/rest-server/issues/174
+   https://github.com/restic/rest-server/pull/307
+
+ * Enhancement #189: Support group accessible repositories
+
+   Rest-server now supports making repositories accessible to the filesystem group
+   by setting the `--group-accessible-repos` option. Note that permissions of
+   existing files are not modified. To allow the group to read and write file, use
+   a umask of `007`. To only grant read access use `027`. To make an existing
+   repository group-accessible, use `chmod -R g+rwX /path/to/repo`.
+
+   https://github.com/restic/rest-server/issues/189
+   https://github.com/restic/rest-server/pull/308
+
+ * Enhancement #295: Output status of append-only mode on startup
+
+   Rest-server now displays the status of append-only mode during startup.
+
+   https://github.com/restic/rest-server/pull/295
+
+ * Enhancement #315: Hardened tls settings
+
+   Rest-server now uses a secure TLS cipher suite set by default. The minimum TLS
+   version is now TLS 1.2 and can be further increased using the new
+   `--tls-min-ver` option, allowing users to enforce stricter security
+   requirements.
+
+   https://github.com/restic/rest-server/pull/315
+
+ * Enhancement #321: Add zip archive format for Windows releases
+
+   Windows users can now download rest-server binaries in zip archive format (.zip)
+   in addition to the existing tar.gz archives.
+
+   https://github.com/restic/rest-server/issues/321
+   https://github.com/restic/rest-server/pull/346
+
+
+Changelog for rest-server 0.13.0 (2024-07-26)
+============================================
+
+The following sections list the changes in rest-server 0.13.0 relevant
+to users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Chg #267: Update dependencies and require Go 1.18 or newer
+ * Chg #273: Shut down cleanly on TERM and INT signals
+ * Enh #271: Print listening address after start-up
+ * Enh #272: Support listening on a unix socket
+
+Details
+-------
+
+ * Change #267: Update dependencies and require Go 1.18 or newer
+
+   Most dependencies have been updated. Since some libraries require newer language
+   features, support for Go 1.17 has been dropped, which means that rest-server now
+   requires at least Go 1.18 to build.
+
+   https://github.com/restic/rest-server/pull/267
+
+ * Change #273: Shut down cleanly on TERM and INT signals
+
+   Rest-server now listens for TERM and INT signals and cleanly closes down the
+   http.Server and listener when receiving either of them.
+
+   This is particularly useful when listening on a unix socket, as the server will
+   now remove the socket file when it shuts down.
+
+   https://github.com/restic/rest-server/pull/273
+
+ * Enhancement #271: Print listening address after start-up
+
+   When started with `--listen :0`, rest-server would print `start server on :0`
+
+   The message now also includes the actual address listened on, for example `start
+   server on 0.0.0.0:37333`. This is useful when starting a server with an
+   auto-allocated free port number (port 0).
+
+   https://github.com/restic/rest-server/pull/271
+
+ * Enhancement #272: Support listening on a unix socket
+
+   It is now possible to make rest-server listen on a unix socket by prefixing the
+   socket filename with `unix:` and passing it to the `--listen` option, for
+   example `--listen unix:/tmp/foo`.
+
+   This is useful in combination with remote port forwarding to enable a remote
+   server to backup locally, e.g.:
+
+   ```
+   rest-server --listen unix:/tmp/foo &
+   ssh -R /tmp/foo:/tmp/foo user@host restic -r rest:http+unix:///tmp/foo:/repo backup
+   ```
+
+   https://github.com/restic/rest-server/pull/272
+
+
+Changelog for rest-server 0.12.1 (2023-07-09)
+============================================
+
+The following sections list the changes in rest-server 0.12.1 relevant
+to users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #230: Fix erroneous warnings about unsupported fsync
+ * Fix #238: API: Return empty array when listing empty folders
+ * Enh #217: Log to stdout using the `--log -` option
+
+Details
+-------
+
+ * Bugfix #230: Fix erroneous warnings about unsupported fsync
+
+   Due to a regression in rest-server 0.12.0, it continuously printed `WARNING:
+   fsync is not supported by the data storage. This can lead to data loss, if the
+   system crashes or the storage is unexpectedly disconnected.` for systems that
+   support fsync. We have fixed the warning.
+
+   https://github.com/restic/rest-server/issues/230
+   https://github.com/restic/rest-server/pull/231
+
+ * Bugfix #238: API: Return empty array when listing empty folders
+
+   Rest-server returned `null` when listing an empty folder. This has been changed
+   to returning an empty array in accordance with the REST protocol specification.
+   This change has no impact on restic users.
+
+   https://github.com/restic/rest-server/issues/238
+   https://github.com/restic/rest-server/pull/239
+
+ * Enhancement #217: Log to stdout using the `--log -` option
+
+   Logging to stdout was possible using `--log /dev/stdout`. However, when the rest
+   server is run as a different user, for example, using
+
+   `sudo -u restic rest-server [...] --log /dev/stdout`
+
+   This did not work due to permission issues.
+
+   For logging to stdout, the `--log` option now supports the special filename `-`
+   which also works in these cases.
+
+   https://github.com/restic/rest-server/pull/217
+
+
+Changelog for rest-server 0.12.0 (2023-04-24)
+============================================
+
+The following sections list the changes in rest-server 0.12.0 relevant
+to users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Fix #183: Allow usernames containing underscore and more
+ * Fix #219: Ignore unexpected files in the data/ folder
+ * Fix #1871: Return 500 "Internal server error" if files cannot be read
+ * Chg #207: Return error if command-line arguments are specified
+ * Chg #208: Update dependencies and require Go 1.17 or newer
+ * Enh #133: Cache basic authentication credentials
+ * Enh #187: Allow configurable location for `.htpasswd` file
+
+Details
+-------
+
+ * Bugfix #183: Allow usernames containing underscore and more
+
+   The security fix in rest-server 0.11.0 (#131) disallowed usernames containing
+   and underscore "_". The list of allowed characters has now been changed to
+   include Unicode characters, numbers, "_", "-", "." and "@".
+
+   https://github.com/restic/rest-server/issues/183
+   https://github.com/restic/rest-server/pull/184
+
+ * Bugfix #219: Ignore unexpected files in the data/ folder
+
+   If the data folder of a repository contained files, this would prevent restic
+   from retrieving a list of file data files. This has been fixed. As a workaround
+   remove the files that are directly contained in the data folder (e.g.,
+   `.DS_Store` files).
+
+   https://github.com/restic/rest-server/issues/219
+   https://github.com/restic/rest-server/pull/221
+
+ * Bugfix #1871: Return 500 "Internal server error" if files cannot be read
+
+   When files in a repository cannot be read by rest-server, for example after
+   running `restic prune` directly on the server hosting the repositories in a way
+   that causes filesystem permissions to be wrong, rest-server previously returned
+   404 "Not Found" as status code. This was causing confusing for users.
+
+   The error handling has now been fixed to only return 404 "Not Found" if the file
+   actually does not exist. Otherwise a 500 "Internal server error" is reported to
+   the client and the underlying error is logged at the server side.
+
+   https://github.com/restic/rest-server/issues/1871
+   https://github.com/restic/rest-server/pull/195
+
+ * Change #207: Return error if command-line arguments are specified
+
+   Command line arguments are ignored by rest-server, but there was previously no
+   indication of this when they were supplied anyway.
+
+   To prevent usage errors an error is now printed when command line arguments are
+   supplied, instead of them being silently ignored.
+
+   https://github.com/restic/rest-server/pull/207
+
+ * Change #208: Update dependencies and require Go 1.17 or newer
+
+   Most dependencies have been updated. Since some libraries require newer language
+   features, support for Go 1.15-1.16 has been dropped, which means that
+   rest-server now requires at least Go 1.17 to build.
+
+   https://github.com/restic/rest-server/pull/208
+
+ * Enhancement #133: Cache basic authentication credentials
+
+   To speed up the verification of basic auth credentials, rest-server now caches
+   passwords for a minute in memory. That way the expensive verification of basic
+   auth credentials can be skipped for most requests issued by a single restic run.
+   The password is kept in memory in a hashed form and not as plaintext.
+
+   https://github.com/restic/rest-server/issues/133
+   https://github.com/restic/rest-server/pull/138
+
+ * Enhancement #187: Allow configurable location for `.htpasswd` file
+
+   It is now possible to specify the location of the `.htpasswd` file using the
+   `--htpasswd-file` option.
+
+   https://github.com/restic/rest-server/issues/187
+   https://github.com/restic/rest-server/pull/188
+
+
+Changelog for rest-server 0.11.0 (2022-02-10)
+============================================
+
+The following sections list the changes in rest-server 0.11.0 relevant
+to users. The changes are ordered by importance.
+
+Summary
+-------
+
+ * Sec #131: Prevent loading of usernames containing a slash
+ * Fix #119: Fix Docker configuration for `DISABLE_AUTHENTICATION`
+ * Fix #142: Fix possible data loss due to interrupted network connections
+ * Fix #155: Reply "insufficient storage" on disk full or over-quota
+ * Fix #157: Use platform-specific temporary directory as default data directory
+ * Chg #112: Add subrepo support and refactor server code
+ * Chg #146: Build rest-server at docker container build time
+ * Enh #122: Verify uploaded files
+ * Enh #126: Allow running rest-server via systemd socket activation
+ * Enh #148: Expand use of security features in example systemd unit file
+
+Details
+-------
+
+ * Security #131: Prevent loading of usernames containing a slash
+
+   "/" is valid char in HTTP authorization headers, but is also used in rest-server
+   to map usernames to private repos.
+
+   This commit prevents loading maliciously composed usernames like "/foo/config"
+   by restricting the allowed characters to the unicode character class, numbers,
+   "-", "." and "@".
+
+   This prevents requests to other users files like:
+
+   Curl -v -X DELETE -u foo/config:attack http://localhost:8000/foo/config
+
+   https://github.com/restic/rest-server/issues/131
+   https://github.com/restic/rest-server/pull/132
+   https://github.com/restic/rest-server/pull/137
+
+ * Bugfix #119: Fix Docker configuration for `DISABLE_AUTHENTICATION`
+
+   Rest-server 0.10.0 introduced a regression which caused the
+   `DISABLE_AUTHENTICATION` environment variable to stop working for the Docker
+   container. This has been fixed by automatically setting the option `--no-auth`
+   to disable authentication.
+
+   https://github.com/restic/rest-server/issues/119
+   https://github.com/restic/rest-server/pull/124
+
+ * Bugfix #142: Fix possible data loss due to interrupted network connections
+
+   When rest-server was run without `--append-only` it was possible to lose
+   uploaded files in a specific scenario in which a network connection was
+   interrupted.
+
+   For the data loss to occur a file upload by restic would have to be interrupted
+   such that restic notices the interrupted network connection before the
+   rest-server. Then restic would have to retry the file upload and finish it
+   before the rest-server notices that the initial upload has failed. Then the
+   uploaded file would be accidentally removed by rest-server when trying to
+   cleanup the failed upload.
+
+   This has been fixed by always uploading to a temporary file first which is moved
+   in position only once it was uploaded completely.
+
+   https://github.com/restic/rest-server/pull/142
+
+ * Bugfix #155: Reply "insufficient storage" on disk full or over-quota
+
+   When there was no space left on disk, or any other write-related error occurred,
+   rest-server replied with HTTP status code 400 (Bad request). This is misleading
+   (restic client will dump the status code to the user).
+
+   Rest-server now replies with two different status codes in these situations: *
+   HTTP 507 "Insufficient storage" is the status on disk full or repository
+   over-quota * HTTP 500 "Internal server error" is used for other disk-related
+   errors
+
+   https://github.com/restic/rest-server/issues/155
+   https://github.com/restic/rest-server/pull/160
+
+ * Bugfix #157: Use platform-specific temporary directory as default data directory
+
+   If no data directory is specificed, then rest-server now uses the Go standard
+   library functions to retrieve the standard temporary directory path for the
+   current platform.
+
+   https://github.com/restic/rest-server/issues/157
+   https://github.com/restic/rest-server/pull/158
+
+ * Change #112: Add subrepo support and refactor server code
+
+   Support for multi-level repositories has been added, so now each user can have
+   its own subrepositories. This feature is always enabled.
+
+   Authentication for the Prometheus /metrics endpoint can now be disabled with the
+   new `--prometheus-no-auth` flag.
+
+   We have split out all HTTP handling to a separate `repo` subpackage to cleanly
+   separate the server code from the code that handles a single repository. The new
+   RepoHandler also makes it easier to reuse rest-server as a Go component in any
+   other HTTP server.
+
+   The refactoring makes the code significantly easier to follow and understand,
+   which in turn makes it easier to add new features, audit for security and debug
+   issues.
+
+   https://github.com/restic/rest-server/issues/109
+   https://github.com/restic/rest-server/issues/107
+   https://github.com/restic/rest-server/pull/112
+
+ * Change #146: Build rest-server at docker container build time
+
+   The Dockerfile now includes a build stage such that the latest rest-server is
+   always built and packaged. This is done in a standard golang container to ensure
+   a clean build environment and only the final binary is shipped rather than the
+   whole build environment.
+
+   https://github.com/restic/rest-server/issues/146
+   https://github.com/restic/rest-server/pull/145
+
+ * Enhancement #122: Verify uploaded files
+
+   The rest-server now by default verifies that the hash of content of uploaded
+   files matches their filename. This ensures that transmission errors are detected
+   and forces restic to retry the upload. On low-power devices it can make sense to
+   disable this check by passing the `--no-verify-upload` flag.
+
+   https://github.com/restic/rest-server/issues/122
+   https://github.com/restic/rest-server/pull/130
+
+ * Enhancement #126: Allow running rest-server via systemd socket activation
+
+   We've added the option to have systemd create the listening socket and start the
+   rest-server on demand.
+
+   https://github.com/restic/rest-server/issues/126
+   https://github.com/restic/rest-server/pull/151
+   https://github.com/restic/rest-server/pull/127
+
+ * Enhancement #148: Expand use of security features in example systemd unit file
+
+   The example systemd unit file now enables additional systemd features to
+   mitigate potential security vulnerabilities in rest-server and the various
+   packages and operating system components which it relies upon.
+
+   https://github.com/restic/rest-server/issues/148
+   https://github.com/restic/rest-server/pull/149
+
+
 Changelog for rest-server 0.10.0 (2020-09-13)
 ============================================
 
@@ -7,67 +455,71 @@ to users. The changes are ordered by importance.
 Summary
 -------
 
- * Sec #117: Stricter path sanitization
  * Sec #60: Require auth by default, add --no-auth flag
  * Sec #64: Refuse overwriting config file in append-only mode
+ * Sec #117: Stricter path sanitization
  * Chg #102: Remove vendored dependencies
  * Enh #44: Add changelog file
 
 Details
 -------
 
- * Security #117: Stricter path sanitization
-
-   The framework we're using in rest-server to decode paths to repositories allowed specifying
-   URL-encoded characters in paths, including sensitive characters such as `/` (encoded as
-   `%2F`).
-
-   We've changed this unintended behavior, such that rest-server now rejects such paths. In
-   particular, it is no longer possible to specify sub-repositories for users by encoding the
-   path with `%2F`, such as `http://localhost:8000/foo%2Fbar`, which means that this will
-   unfortunately be a breaking change in that case.
-
-   If using sub-repositories for users is important to you, please let us know in the forum, so we
-   can learn about your use case and implement this properly. As it currently stands, the ability
-   to use sub-repositories was an unintentional feature made possible by the URL decoding
-   framework used, and hence never meant to be supported in the first place. If we wish to have this
-   feature in rest-server, we'd like to have it implemented properly and intentionally.
-
-   https://github.com/restic/restic/issues/117
-
  * Security #60: Require auth by default, add --no-auth flag
 
-   In order to prevent users from accidentally exposing rest-server without authentication,
-   rest-server now defaults to requiring a .htpasswd. If you want to disable authentication, you
-   need to explicitly pass the new --no-auth flag.
+   In order to prevent users from accidentally exposing rest-server without
+   authentication, rest-server now defaults to requiring a .htpasswd. If you want
+   to disable authentication, you need to explicitly pass the new --no-auth flag.
 
-   https://github.com/restic/restic/issues/60
-   https://github.com/restic/restic/pull/61
+   https://github.com/restic/rest-server/issues/60
+   https://github.com/restic/rest-server/pull/61
 
  * Security #64: Refuse overwriting config file in append-only mode
 
-   While working on the `rclone serve restic` command we noticed that is currently possible to
-   overwrite the config file in a repo even if `--append-only` is specified. The first commit adds
-   proper tests, and the second commit fixes the issue.
+   While working on the `rclone serve restic` command we noticed that is currently
+   possible to overwrite the config file in a repo even if `--append-only` is
+   specified. The first commit adds proper tests, and the second commit fixes the
+   issue.
 
-   https://github.com/restic/restic/pull/64
+   https://github.com/restic/rest-server/pull/64
+
+ * Security #117: Stricter path sanitization
+
+   The framework we're using in rest-server to decode paths to repositories allowed
+   specifying URL-encoded characters in paths, including sensitive characters such
+   as `/` (encoded as `%2F`).
+
+   We've changed this unintended behavior, such that rest-server now rejects such
+   paths. In particular, it is no longer possible to specify sub-repositories for
+   users by encoding the path with `%2F`, such as
+   `http://localhost:8000/foo%2Fbar`, which means that this will unfortunately be a
+   breaking change in that case.
+
+   If using sub-repositories for users is important to you, please let us know in
+   the forum, so we can learn about your use case and implement this properly. As
+   it currently stands, the ability to use sub-repositories was an unintentional
+   feature made possible by the URL decoding framework used, and hence never meant
+   to be supported in the first place. If we wish to have this feature in
+   rest-server, we'd like to have it implemented properly and intentionally.
+
+   https://github.com/restic/rest-server/issues/117
 
  * Change #102: Remove vendored dependencies
 
-   We've removed the vendored dependencies (in the subdir `vendor/`) similar to what we did for
-   `restic` itself. When building restic, the Go compiler automatically fetches the
-   dependencies. It will also cryptographically verify that the correct code has been fetched by
-   using the hashes in `go.sum` (see the link to the documentation below).
+   We've removed the vendored dependencies (in the subdir `vendor/`) similar to
+   what we did for `restic` itself. When building restic, the Go compiler
+   automatically fetches the dependencies. It will also cryptographically verify
+   that the correct code has been fetched by using the hashes in `go.sum` (see the
+   link to the documentation below).
 
-   Building the rest-server now requires Go 1.11 or newer, since we're using Go Modules for
-   dependency management. Older Go versions are not supported any more.
+   Building the rest-server now requires Go 1.11 or newer, since we're using Go
+   Modules for dependency management. Older Go versions are not supported any more.
 
-   https://github.com/restic/restic/issues/102
+   https://github.com/restic/rest-server/issues/102
    https://golang.org/cmd/go/#hdr-Module_downloading_and_verification
 
  * Enhancement #44: Add changelog file
 
-   https://github.com/restic/restic/issues/44
-   https://github.com/restic/restic/pull/62
+   https://github.com/restic/rest-server/issues/44
+   https://github.com/restic/rest-server/pull/62
 
 

Dockerfile

@@ -1,3 +1,14 @@
+FROM golang:alpine AS builder
+
+ENV CGO_ENABLED 0
+
+COPY . /build
+WORKDIR /build
+RUN go build -o rest-server ./cmd/rest-server
+
+
+
+
 FROM alpine
 
 ENV DATA_DIRECTORY /data
@@ -8,7 +19,7 @@ RUN apk add --no-cache --update apache2-utils
 COPY docker/create_user /usr/bin/
 COPY docker/delete_user /usr/bin/
 COPY docker/entrypoint.sh /entrypoint.sh
-COPY rest-server /usr/bin
+COPY --from=builder /build/rest-server /usr/bin
 
 VOLUME /data
 EXPOSE 8000

Dockerfile.goreleaser

@@ -0,0 +1,16 @@
+FROM alpine
+
+ENV DATA_DIRECTORY /data
+ENV PASSWORD_FILE /data/.htpasswd
+
+RUN apk add --no-cache --update apache2-utils
+
+COPY docker/create_user /usr/bin/
+COPY docker/delete_user /usr/bin/
+COPY docker/entrypoint.sh /entrypoint.sh
+COPY rest-server /usr/bin
+
+VOLUME /data
+EXPOSE 8000
+
+CMD [ "/entrypoint.sh" ]

README.md

@@ -1,18 +1,17 @@
 # Rest Server
 
 
-[![Status badge for tests](https://github.com/happal/monsoon/workflows/Build%20and%20tests/badge.svg)](https://github.com/happal/monsoon/actions?query=workflow%3A%22Build+and+tests%22)
-[![Status badge for style checkers](https://github.com/happal/monsoon/workflows/Style%20Checkers/badge.svg)](https://github.com/happal/monsoon/actions?query=workflow%3A%22Style+Checkers%22)
+[![Status badge for CI tests](https://github.com/restic/rest-server/workflows/test/badge.svg)](https://github.com/restic/rest-server/actions?query=workflow%3Atest)
 [![Go Report Card](https://goreportcard.com/badge/github.com/restic/rest-server)](https://goreportcard.com/report/github.com/restic/rest-server)
 [![GoDoc](https://godoc.org/github.com/restic/rest-server?status.svg)](https://godoc.org/github.com/restic/rest-server)
 [![License](https://img.shields.io/badge/license-BSD%20%282--Clause%29-003262.svg?maxAge=2592000)](https://github.com/restic/rest-server/blob/master/LICENSE)
 [![Powered by](https://img.shields.io/badge/powered_by-Go-5272b4.svg?maxAge=2592000)](https://golang.org/)
 
-Rest Server is a high performance HTTP server that implements restic's [REST backend API](http://restic.readthedocs.io/en/latest/100_references.html#rest-backend).  It provides secure and efficient way to backup data remotely, using [restic](https://github.com/restic/restic) backup client via the [rest: URL](http://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server).
+Rest Server is a high performance HTTP server that implements restic's [REST backend API](https://restic.readthedocs.io/en/latest/100_references.html#rest-backend).  It provides secure and efficient way to backup data remotely, using [restic](https://github.com/restic/restic) backup client via the [rest: URL](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server).
 
 ## Requirements
 
-Rest Server requires Go 1.11 or higher to build.  The only tested compiler is the official Go compiler.  Building server with `gccgo` may work, but is not supported.
+Rest Server requires Go 1.23 or higher to build.  The only tested compiler is the official Go compiler.
 
 The required version of restic backup client to use with `rest-server` is [v0.7.1](https://github.com/restic/restic/releases/tag/v0.7.1) or higher.
 
@@ -20,69 +19,71 @@ The required version of restic backup client to use with `rest-server` is [v0.7.
 
 For building the `rest-server` binary run `CGO_ENABLED=0 go build -o rest-server ./cmd/rest-server`
 
-## Docker
-
-### Build image
-
-Put the `rest-server` binary in the current directory, then run:
-
-    docker build -t restic/rest-server:latest .
-
-
-### Pull image
-
-    docker pull restic/rest-server
-
 ## Usage
 
-To learn how to use restic backup client with REST backend, please consult [restic manual](http://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server).
+To learn how to use restic backup client with REST backend, please consult [restic manual](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server).
 
-    $ rest-server --help
+```console
+$ rest-server --help
 
-    Run a REST server for use with restic
+Run a REST server for use with restic
 
-    Usage:
-      rest-server [flags]
+Usage:
+  rest-server [flags]
 
-    Flags:
-          --append-only          enable append only mode
-          --cpu-profile string   write CPU profile to file
-          --debug                output debug messages
-      -h, --help                 help for rest-server
-          --listen string        listen address (default ":8000")
-          --log string           log HTTP requests in the combined log format
-          --max-size int         the maximum size of the repository in bytes
-          --no-auth              disable .htpasswd authentication
-          --path string          data directory (default "/tmp/restic")
-          --private-repos        users can only access their private repo
-          --prometheus           enable Prometheus metrics
-          --tls                  turn on TLS support
-          --tls-cert string      TLS certificate path
-          --tls-key string       TLS key path
-      -V, --version              output version and exit
+Flags:
+      --append-only                  enable append only mode
+      --cpu-profile string           write CPU profile to file
+      --debug                        output debug messages
+      --group-accessible-repos       let filesystem group be able to access repo files
+  -h, --help                         help for rest-server
+      --htpasswd-file string         location of .htpasswd file (default: "<data directory>/.htpasswd)"
+      --listen string                listen address (default ":8000")
+      --log filename                 write HTTP requests in the combined log format to the specified filename (use "-" for logging to stdout)
+      --max-size int                 the maximum size of the repository in bytes
+      --no-auth                      disable .htpasswd authentication
+      --no-verify-upload             do not verify the integrity of uploaded data. DO NOT enable unless the rest-server runs on a very low-power device
+      --path string                  data directory (default "/tmp/restic")
+      --private-repos                users can only access their private repo
+      --prometheus                   enable Prometheus metrics
+      --prometheus-no-auth           disable auth for Prometheus /metrics endpoint
+      --proxy-auth-username string   specifies the HTTP header containing the username for proxy-based authentication
+      --tls                          turn on TLS support
+      --tls-cert string              TLS certificate path
+      --tls-key string               TLS key path
+      --tls-min-ver string           TLS min version, one of (1.2|1.3) (default "1.2")
+  -v, --version                      version for rest-server
+```
 
-By default the server persists backup data in `/tmp/restic`.  To start the server with a custom persistence directory and with authentication disabled:
+By default the server persists backup data in the OS temporary directory (`/tmp/restic` on Linux/BSD and others, in `%TEMP%\\restic` in Windows, etc). **If `rest-server` is launched using the default path, all backups will be lost**. To start the server with a custom persistence directory and with authentication disabled:
 
-    rest-server --path /user/home/backup --no-auth
+```sh
+rest-server --path /user/home/backup --no-auth
+```
 
-To authenticate users (for access to the rest-server), the server supports using a `.htpasswd` file to specify users. You can create such a file at the root of the persistence directory by executing the following command (note that you need the `htpasswd` program from Apache's http-tools).  In order to append new user to the file, just omit the `-c` argument.  Only bcrypt and SHA encryption methods are supported, so use -B (very secure) or -s (insecure by today's standards) when adding/changing passwords.
+To authenticate users (for access to the rest-server), the server supports using a `.htpasswd` file to specify users. By default, the server looks for this file at the root of the persistence directory, but this can be changed using the `--htpasswd-file` option. You can create such a file by executing the following command (note that you need the `htpasswd` program from Apache's http-tools).  In order to append new user to the file, just omit the `-c` argument.  Only bcrypt and SHA encryption methods are supported, so use -B (very secure) or -s (insecure by today's standards) when adding/changing passwords.
 
-    htpasswd -B -c .htpasswd username
+```sh
+htpasswd -B -c .htpasswd username
+```
 
 If you want to disable authentication, you must add the `--no-auth` flag. If this flag is not specified and the `.htpasswd` cannot be opened, rest-server will refuse to start.
 
 NOTE: In older versions of rest-server (up to 0.9.7), this flag does not exist and the server disables authentication if `.htpasswd` is missing or cannot be opened.
 
-By default the server uses HTTP protocol.  This is not very secure since with Basic Authentication, user name and passwords will be sent in clear text in every request.  In order to enable TLS support just add the `--tls` argument and add a private and public key at the root of your persistence directory. You may also specify private and public keys by `--tls-cert` and `--tls-key`.
+By default the server uses HTTP protocol.  This is not very secure since with Basic Authentication, user name and passwords will be sent in clear text in every request.  In order to enable TLS support just add the `--tls` argument and add a private and public key at the root of your persistence directory. You may also specify private and public keys by `--tls-cert` and `--tls-key` and set the minimum TLS version to 1.3 using `--tls-min-ver 1.3`.
 
-Signed certificate is required by the restic backend, but if you just want to test the feature you can generate unsigned keys with the following commands:
+Signed certificate is normally required by the restic backend, but if you just want to test the feature you can generate password-less unsigned keys with the following command:
 
-    openssl genrsa -out private_key 2048
-    openssl req -new -x509 -key private_key -out public_key -days 365
+```sh
+openssl req -newkey rsa:2048 -nodes -x509 -keyout private_key -out public_key -days 365 -addext "subjectAltName = IP:127.0.0.1,DNS:yourdomain.com"
+```
+
+Omit the `IP:127.0.0.1` if you don't need your server be accessed via SSH Tunnels. No need to change default values in the openssl dialog, hitting enter every time is sufficient. To access this server via restic use `--cacert public_key`, meaning with a self-signed certificate you have to distribute your `public_key` file to every restic client.
 
 The `--append-only` mode allows creation of new backups but prevents deletion and modification of existing backups. This can be useful when backing up systems that have a potential of being hacked.
 
-To prevent your users from accessing each others' repositories, you may use the `--private-repos` flag which grants access only when a subdirectory with the same name as the user is specified in the repository URL. For example, user "foo" using the repository URLs `rest:https://foo:pass@host:8000/foo` or `rest:https://foo:pass@host:8000/foo/` would be granted access, but the same user using repository URLs `rest:https://foo:pass@host:8000/` or `rest:https://foo:pass@host:8000/foobar/` would be denied access.
+To prevent your users from accessing each others' repositories, you may use the `--private-repos` flag which grants access only when a subdirectory with the same name as the user is specified in the repository URL. For example, user "foo" using the repository URLs `rest:https://foo:pass@host:8000/foo` or `rest:https://foo:pass@host:8000/foo/` would be granted access, but the same user using repository URLs `rest:https://foo:pass@host:8000/` or `rest:https://foo:pass@host:8000/foobar/` would be denied access. Users can also create their own subrepositories, like `/foo/bar/`.
 
 Rest Server uses exactly the same directory structure as local backend, so you should be able to access it both locally and via HTTP, even simultaneously.
 
@@ -92,46 +93,85 @@ There's an example [systemd service file](https://github.com/restic/rest-server/
 
 ### Docker
 
-By default, image uses authentication.  To turn it off, set environment variable `DISABLE_AUTHENTICATION` to any value.
-
-Persistent data volume is located to `/data`.
+Rest Server works well inside a container, images are [published to Docker Hub](https://hub.docker.com/r/restic/rest-server). 
 
 #### Start server
 
+You can run the server with any container runtime, like Docker:
+
+```sh
+    docker pull restic/rest-server:latest
     docker run -p 8000:8000 -v /my/data:/data --name rest_server restic/rest-server
+```
 
-It's suggested to set a container name to more easily manage users (see next section).
+Note that:
 
-You can set environment variable `OPTIONS` to any extra flags you'd like to pass to rest-server.
+- **contrary to the defaults** of `rest-server`, the persistent data volume is located to `/data`.
+- By default, the image uses authentication.  To turn it off, set environment variable `DISABLE_AUTHENTICATION` to any value.
+- By default, the image loads the `.htpasswd` file from the persistent data volume (i.e. from `/data/.htpasswd`). To change the location of this file, set the environment variable `PASSWORD_FILE` to the path of the `.htpasswd` file. Please note that this path must be accessible from inside the container and should be persisted. This is normally done by bind-mounting a path into the container or with another docker volume.
+- It's suggested to set a container name to more easily manage users (`--name` parameter to `docker run`).
+- You can set environment variable `OPTIONS` to any extra flags you'd like to pass to rest-server.
+
+#### Customize the image
+
+The [published image](https://hub.docker.com/r/restic/rest-server) is built from the `Dockerfile` available on this repository, which you may use as a basis for building your own customized images.
+
+```sh
+    git clone https://github.com/restic/rest-server.git 
+    cd rest-server
+    docker build -t restic/rest-server:latest .
+```
 
 #### Manage users
 
 ##### Add user
 
-    docker exec -it rest_server create_user myuser
+```sh
+docker exec -it rest_server create_user myuser
+```
 
 or
 
-    docker exec -it rest_server create_user myuser mypassword
+```sh
+docker exec -it rest_server create_user myuser mypassword
+```
 
 ##### Delete user
 
-    docker exec -it rest_server delete_user myuser
+```sh
+docker exec -it rest_server delete_user myuser
+```
+
+## Proxy Authentication
+
+See above for no authentication (`--no-auth`) and basic authentication.
+
+To delegate authentication to a proxy, use the `--proxy-auth-username` flag. The specified header name, for example `X-Forwarded-User`,
+must be present in the request headers and specifies the username. Basic authentication is disabled when this flag is set.
+
+Warning: rest-server trusts the username in the header. It is the responsibility of the proxy
+to ensure that the username is correct and cannot be forged by an attacker.
 
 
 ## Prometheus support and Grafana dashboard
 
-The server can be started with `--prometheus` to expose [Prometheus](https://prometheus.io/) metrics at `/metrics`.
+The server can be started with `--prometheus` to expose [Prometheus](https://prometheus.io/) metrics at `/metrics`. If authentication is enabled, this endpoint requires authentication for the 'metrics' user, but this can be overridden with the `--prometheus-no-auth` flag.
 
 This repository contains an example full stack Docker Compose setup with a Grafana dashboard in [examples/compose-with-grafana/](examples/compose-with-grafana/).
 
 
+## Group-accessible Repositories
+
+Rest-server supports making repositories accessible to the filesystem group by setting the `--group-accessible-repos` option. Note that permissions of existing files are not modified. To allow the group to read and write file, use a umask of `007`. To only grant read access use `027`. To make an existing repository group-accessible, use `chmod -R g+rwX /path/to/repo`.
+
 ## Why use Rest Server?
 
 Compared to the SFTP backend, the REST backend has better performance, especially so if you can skip additional crypto overhead by using plain HTTP transport (restic already properly encrypts all data it sends, so using HTTPS is mostly about authentication).
 
 But, even if you use HTTPS transport, the REST protocol should be faster and more scalable, due to some inefficiencies of the SFTP protocol (everything needs to be transferred in chunks of 32 KiB at most, each packet needs to be acknowledged by the server).
 
+One important safety feature that Rest Server adds is the optional ability to run in append-only mode. This prevents an attacker from wiping your server backups when access is gained to the server being backed up.
+
 Finally, the Rest Server implementation is really simple and as such could be used on the low-end devices, no problem.  Also, in some cases, for example behind corporate firewalls, HTTP/S might be the only protocol allowed.  Here too REST backend might be the perfect option for your backup needs.
 
 ## Contributors

Release.md

@@ -30,11 +30,11 @@
         git tag -a -s -m "v${VERSION}" "v${VERSION}"
         git push --tags
 
-6. Build the project (use `--skip-publish` for testing, or pass `--config` to
+6. Build the project (use `--snapshot` for testing, or pass `--config` to
    use another config file):
 
         goreleaser \
-          release \
+          release --parallelism 4 \
           --release-notes <(calens --template changelog/CHANGELOG-GitHub.tmpl --version "${VERSION}")
 
 7. Set a new version in `main.go` and commit the result:

VERSION

@@ -1 +1 @@
-0.10.0
+0.14.0

build.go

@@ -3,7 +3,7 @@
 // This program aims to make building Go programs for end users easier by just
 // calling it with `go run`, without having to setup a GOPATH.
 //
-// This program needs Go >= 1.11. It'll use Go modules for compilation. It
+// This program needs Go >= 1.12. It'll use Go modules for compilation. It
 // builds the package configured as Main in the Config struct.
 
 // BSD 2-Clause License
@@ -35,6 +35,7 @@
 // OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
+//go:build ignore_build_go
 // +build ignore_build_go
 
 package main
@@ -57,7 +58,7 @@ var config = Config{
 	Namespace:  "github.com/restic/rest-server",                 // subdir of GOPATH, e.g. "github.com/foo/bar"
 	Main:       "github.com/restic/rest-server/cmd/rest-server", // package name for the main package
 	Tests:      []string{"./..."},                               // tests to run
-	MinVersion: GoVersion{Major: 1, Minor: 11, Patch: 0},        // minimum Go version supported
+	MinVersion: GoVersion{Major: 1, Minor: 23, Patch: 0},        // minimum Go version supported
 }
 
 // Config configures the build.
@@ -122,17 +123,8 @@ func printEnv(env []string) {
 
 // build runs "go build args..." with GOPATH set to gopath.
 func build(cwd string, env map[string]string, args ...string) error {
-	a := []string{"build"}
-
-	// try to remove all absolute paths from resulting binary
-	if goVersion.AtLeast(GoVersion{1, 13, 0}) {
-		// use the new flag introduced by Go 1.13
-		a = append(a, "-trimpath")
-	} else {
-		// otherwise try to trim as many paths as possible
-		a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", cwd))
-		a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", cwd))
-	}
+	// -trimpath removes all absolute paths from the binary.
+	a := []string{"build", "-trimpath"}
 
 	if enablePIE {
 		a = append(a, "-buildmode=pie")
@@ -326,8 +318,8 @@ func (v GoVersion) String() string {
 }
 
 func main() {
-	if !goVersion.AtLeast(GoVersion{1, 11, 0}) {
-		die("Go version (%v) is too old, Go <= 1.11 does not support Go Modules\n", goVersion)
+	if !goVersion.AtLeast(GoVersion{1, 12, 0}) {
+		die("Go version (%v) is too old, restic requires Go >= 1.12\n", goVersion)
 	}
 
 	if !goVersion.AtLeast(config.MinVersion) {
@@ -394,8 +386,12 @@ func main() {
 
 	verbosePrintf("detected Go version %v\n", goVersion)
 
+	preserveSymbols := false
 	for i := range buildTags {
 		buildTags[i] = strings.TrimSpace(buildTags[i])
+		if buildTags[i] == "debug" || buildTags[i] == "profile" {
+			preserveSymbols = true
+		}
 	}
 
 	verbosePrintf("build tags: %s\n", buildTags)
@@ -422,7 +418,11 @@ func main() {
 	if version != "" {
 		constants["main.version"] = version
 	}
-	ldflags := "-s -w " + constants.LDFlags()
+	ldflags := constants.LDFlags()
+	if !preserveSymbols {
+		// Strip debug symbols.
+		ldflags = "-s -w " + ldflags
+	}
 	verbosePrintf("ldflags: %s\n", ldflags)
 
 	var (

changelog/0.11.0_2022-02-10/issue-119

@@ -0,0 +1,9 @@
+Bugfix: Fix Docker configuration for `DISABLE_AUTHENTICATION`
+
+rest-server 0.10.0 introduced a regression which caused the
+`DISABLE_AUTHENTICATION` environment variable to stop working for the Docker
+container. This has been fixed by automatically setting the option `--no-auth`
+to disable authentication.
+
+https://github.com/restic/rest-server/issues/119
+https://github.com/restic/rest-server/pull/124

changelog/0.11.0_2022-02-10/issue-122

@@ -0,0 +1,9 @@
+Enhancement: Verify uploaded files
+
+The rest-server now by default verifies that the hash of content of uploaded
+files matches their filename. This ensures that transmission errors are
+detected and forces restic to retry the upload. On low-power devices it can
+make sense to disable this check by passing the `--no-verify-upload` flag.
+
+https://github.com/restic/rest-server/issues/122
+https://github.com/restic/rest-server/pull/130

changelog/0.11.0_2022-02-10/issue-126

@@ -0,0 +1,7 @@
+Enhancement: Allow running rest-server via systemd socket activation
+
+We've added the option to have systemd create the listening socket and start the rest-server on demand.
+
+https://github.com/restic/rest-server/issues/126
+https://github.com/restic/rest-server/pull/151
+https://github.com/restic/rest-server/pull/127

changelog/0.11.0_2022-02-10/issue-131

@@ -0,0 +1,16 @@
+Security: Prevent loading of usernames containing a slash
+
+"/" is valid char in HTTP authorization headers, but is also used in
+rest-server to map usernames to private repos.
+
+This commit prevents loading maliciously composed usernames like
+"/foo/config" by restricting the allowed characters to the unicode
+character class, numbers, "-", "." and "@".
+
+This prevents requests to other users files like:
+
+curl -v  -X DELETE -u foo/config:attack  http://localhost:8000/foo/config
+
+https://github.com/restic/rest-server/issues/131
+https://github.com/restic/rest-server/pull/132
+https://github.com/restic/rest-server/pull/137

changelog/0.11.0_2022-02-10/issue-146

@@ -0,0 +1,9 @@
+Change: Build rest-server at docker container build time
+
+The Dockerfile now includes a build stage such that the latest rest-server is
+always built and packaged. This is done in a standard golang container to
+ensure a clean build environment and only the final binary is shipped rather
+than the whole build environment.
+
+https://github.com/restic/rest-server/issues/146
+https://github.com/restic/rest-server/pull/145

changelog/0.11.0_2022-02-10/issue-148

@@ -0,0 +1,8 @@
+Enhancement: Expand use of security features in example systemd unit file
+
+The example systemd unit file now enables additional systemd features to
+mitigate potential security vulnerabilities in rest-server and the various
+packages and operating system components which it relies upon.
+
+https://github.com/restic/rest-server/issues/148
+https://github.com/restic/rest-server/pull/149

changelog/0.11.0_2022-02-10/pull-112

@@ -0,0 +1,20 @@
+Change: Add subrepo support and refactor server code
+
+Support for multi-level repositories has been added, so now each user can have
+its own subrepositories. This feature is always enabled.
+
+Authentication for the Prometheus /metrics endpoint can now be disabled with the
+new `--prometheus-no-auth` flag.
+
+We have split out all HTTP handling to a separate `repo` subpackage to cleanly
+separate the server code from the code that handles a single repository. The new
+RepoHandler also makes it easier to reuse rest-server as a Go component in
+any other HTTP server.
+
+The refactoring makes the code significantly easier to follow and understand,
+which in turn makes it easier to add new features, audit for security and debug
+issues.
+
+https://github.com/restic/restic/pull/112
+https://github.com/restic/restic/issues/109
+https://github.com/restic/restic/issues/107

changelog/0.11.0_2022-02-10/pull-142

@@ -0,0 +1,16 @@
+Bugfix: Fix possible data loss due to interrupted network connections
+
+When rest-server was run without `--append-only` it was possible to lose uploaded
+files in a specific scenario in which a network connection was interrupted.
+
+For the data loss to occur a file upload by restic would have to be interrupted
+such that restic notices the interrupted network connection before the
+rest-server. Then restic would have to retry the file upload and finish it
+before the rest-server notices that the initial upload has failed. Then the
+uploaded file would be accidentally removed by rest-server when trying to
+cleanup the failed upload.
+
+This has been fixed by always uploading to a temporary file first which is moved
+in position only once it was uploaded completely.
+
+https://github.com/restic/rest-server/pull/142

changelog/0.11.0_2022-02-10/pull-158

@@ -0,0 +1,8 @@
+Bugfix: Use platform-specific temporary directory as default data directory
+
+If no data directory is specificed, then rest-server now uses the Go standard
+library functions to retrieve the standard temporary directory path for the
+current platform.
+
+https://github.com/restic/rest-server/issues/157
+https://github.com/restic/rest-server/pull/158

changelog/0.11.0_2022-02-10/pull-160

@@ -0,0 +1,13 @@
+Bugfix: Reply "insufficient storage" on disk full or over-quota
+
+When there was no space left on disk, or any other write-related error
+occurred, rest-server replied with HTTP status code 400 (Bad request).
+This is misleading (restic client will dump the status code to the user).
+
+rest-server now replies with two different status codes in these situations:
+* HTTP 507 "Insufficient storage" is the status on disk full or repository
+  over-quota
+* HTTP 500 "Internal server error" is used for other disk-related errors
+
+https://github.com/restic/rest-server/issues/155
+https://github.com/restic/rest-server/pull/160

changelog/0.12.0_2023-04-24/issue-133

@@ -0,0 +1,9 @@
+Enhancement: Cache basic authentication credentials
+
+To speed up the verification of basic auth credentials, rest-server now caches
+passwords for a minute in memory. That way the expensive verification of basic
+auth credentials can be skipped for most requests issued by a single restic
+run. The password is kept in memory in a hashed form and not as plaintext.
+
+https://github.com/restic/rest-server/issues/133
+https://github.com/restic/rest-server/pull/138

changelog/0.12.0_2023-04-24/issue-182

@@ -0,0 +1,8 @@
+Bugfix: Allow usernames containing underscore and more
+
+The security fix in rest-server 0.11.0 (#131) disallowed usernames containing
+and underscore "_". The list of allowed characters has now been changed to
+include Unicode characters, numbers, "_", "-", "." and "@".
+
+https://github.com/restic/restic/issues/183
+https://github.com/restic/restic/pull/184

changelog/0.12.0_2023-04-24/issue-187

@@ -0,0 +1,7 @@
+Enhancement: Allow configurable location for `.htpasswd` file
+
+It is now possible to specify the location of the `.htpasswd`
+file using the `--htpasswd-file` option.
+
+https://github.com/restic/restic/issues/187
+https://github.com/restic/restic/pull/188

changelog/0.12.0_2023-04-24/issue-219

@@ -0,0 +1,9 @@
+Bugfix: Ignore unexpected files in the data/ folder
+
+If the data folder of a repository contained files, this would prevent restic
+from retrieving a list of file data files. This has been fixed. As a workaround
+remove the files that are directly contained in the data folder (e.g.,
+`.DS_Store` files).
+
+https://github.com/restic/rest-server/issues/219
+https://github.com/restic/rest-server/pull/221

changelog/0.12.0_2023-04-24/pull-194

@@ -0,0 +1,13 @@
+Bugfix: Return 500 "Internal server error" if files cannot be read
+
+When files in a repository cannot be read by rest-server, for example after
+running `restic prune` directly on the server hosting the repositories in a
+way that causes filesystem permissions to be wrong, rest-server previously
+returned 404 "Not Found" as status code. This was causing confusing for users.
+
+The error handling has now been fixed to only return 404 "Not Found" if the
+file actually does not exist. Otherwise a 500 "Internal server error" is
+reported to the client and the underlying error is logged at the server side.
+
+https://github.com/restic/restic/issues/1871
+https://github.com/restic/rest-server/pull/195

changelog/0.12.0_2023-04-24/pull-207

@@ -0,0 +1,9 @@
+Change: Return error if command-line arguments are specified
+
+Command line arguments are ignored by rest-server, but there was previously
+no indication of this when they were supplied anyway.
+
+To prevent usage errors an error is now printed when command line arguments
+are supplied, instead of them being silently ignored.
+
+https://github.com/restic/rest-server/pull/207

changelog/0.12.0_2023-04-24/pull-208

@@ -0,0 +1,7 @@
+Change: Update dependencies and require Go 1.17 or newer
+
+Most dependencies have been updated. Since some libraries require newer language
+features, support for Go 1.15-1.16 has been dropped, which means that rest-server
+now requires at least Go 1.17 to build.
+
+https://github.com/restic/rest-server/pull/208

changelog/0.12.1_2023-07-09/issue-230

@@ -0,0 +1,9 @@
+Bugfix: Fix erroneous warnings about unsupported fsync
+
+Due to a regression in rest-server 0.12.0, it continuously printed
+`WARNING: fsync is not supported by the data storage. This can lead to data loss,
+if the system crashes or the storage is unexpectedly disconnected.` for systems
+that support fsync. We have fixed the warning.
+
+https://github.com/restic/rest-server/issues/230
+https://github.com/restic/rest-server/pull/231

changelog/0.12.1_2023-07-09/issue-238

@@ -0,0 +1,8 @@
+Bugfix: API: Return empty array when listing empty folders
+
+Rest-server returned `null` when listing an empty folder. This has been changed
+to returning an empty array in accordance with the REST protocol specification.
+This change has no impact on restic users.
+
+https://github.com/restic/rest-server/issues/238
+https://github.com/restic/rest-server/pull/239

changelog/0.12.1_2023-07-09/pull-217

@@ -0,0 +1,13 @@
+Enhancement: Log to stdout using the `--log -` option
+
+Logging to stdout was possible using `--log /dev/stdout`. However,
+when the rest server is run as a different user, for example, using
+
+	`sudo -u restic rest-server [...] --log /dev/stdout`
+
+this did not work due to permission issues.
+
+For logging to stdout, the `--log` option now supports the special
+filename `-` which also works in these cases.
+
+https://github.com/restic/rest-server/pull/217

changelog/0.13.0_2024-07-26/pull-267

@@ -0,0 +1,7 @@
+Change: Update dependencies and require Go 1.18 or newer
+
+Most dependencies have been updated. Since some libraries require newer language
+features, support for Go 1.17 has been dropped, which means that rest-server
+now requires at least Go 1.18 to build.
+
+https://github.com/restic/rest-server/pull/267

changelog/0.13.0_2024-07-26/pull-271

@@ -0,0 +1,9 @@
+Enhancement: Print listening address after start-up
+
+When started with `--listen :0`, rest-server would print `start server on :0`
+
+The message now also includes the actual address listened on, for example
+`start server on 0.0.0.0:37333`. This is useful when starting a server with
+an auto-allocated free port number (port 0).
+
+https://github.com/restic/rest-server/pull/271

changelog/0.13.0_2024-07-26/pull-272

@@ -0,0 +1,15 @@
+Enhancement: Support listening on a unix socket
+
+It is now possible to make rest-server listen on a unix socket by prefixing
+the socket filename with `unix:` and passing it to the `--listen` option,
+for example `--listen unix:/tmp/foo`.
+
+This is useful in combination with remote port forwarding to enable a remote
+server to backup locally, e.g.:
+
+```
+rest-server --listen unix:/tmp/foo &
+ssh -R /tmp/foo:/tmp/foo user@host restic -r rest:http+unix:///tmp/foo:/repo backup
+```
+
+https://github.com/restic/rest-server/pull/272

changelog/0.13.0_2024-07-26/pull-273

@@ -0,0 +1,9 @@
+Change: Shut down cleanly on TERM and INT signals
+
+Rest-server now listens for TERM and INT signals and cleanly closes down the
+http.Server and listener when receiving either of them.
+
+This is particularly useful when listening on a unix socket, as the server
+will now remove the socket file when it shuts down.
+
+https://github.com/restic/rest-server/pull/273

changelog/0.14.0_2025-05-31/issue-189

@@ -0,0 +1,10 @@
+Enhancement: Support group accessible repositories
+
+Rest-server now supports making repositories accessible to the filesystem group
+by setting the `--group-accessible-repos` option. Note that permissions of
+existing files are not modified. To allow the group to read and write file,
+use a umask of `007`. To only grant read access use `027`. To make an existing
+repository group-accessible, use `chmod -R g+rwX /path/to/repo`.
+
+https://github.com/restic/rest-server/issues/189
+https://github.com/restic/rest-server/pull/308

changelog/0.14.0_2025-05-31/issue-318

@@ -0,0 +1,13 @@
+Security: Fix world-readable permissions on new `.htpasswd` files
+
+On startup the rest-server Docker container creates an empty `.htpasswd` file
+if none exists yet. This file was world-readable by default, which can be
+a security risk, even though the file only contains hashed passwords.
+
+This has been fixed such that new `.htpasswd` files are no longer world-readabble.
+
+The permissions of existing `.htpasswd` files must be manually changed if
+relevant in your setup.
+
+https://github.com/restic/rest-server/issues/318
+https://github.com/restic/rest-server/pull/340

changelog/0.14.0_2025-05-31/issue-321

@@ -0,0 +1,7 @@
+Enhancement: Add zip archive format for Windows releases
+
+Windows users can now download rest-server binaries in zip archive format (.zip)
+in addition to the existing tar.gz archives.
+
+https://github.com/restic/rest-server/issues/321
+https://github.com/restic/rest-server/pull/346

changelog/0.14.0_2025-05-31/pull-295

@@ -0,0 +1,5 @@
+Enhancement: Output status of append-only mode on startup
+
+Rest-server now displays the status of append-only mode during startup.
+
+https://github.com/restic/rest-server/pull/295

changelog/0.14.0_2025-05-31/pull-307

@@ -0,0 +1,12 @@
+Enhancement: Support proxy-based authentication
+
+Rest-server now supports authentication via HTTP proxy headers. This feature can
+be enabled by specifying the username header using the `--proxy-auth-username`
+option (e.g., `--proxy-auth-username=X-Forwarded-User`).
+
+When enabled, the server authenticates users based on the specified header and
+disables Basic Auth. Note that proxy authentication is disabled when `--no-auth`
+is set.
+
+https://github.com/restic/rest-server/issues/174
+https://github.com/restic/rest-server/pull/307

changelog/0.14.0_2025-05-31/pull-315

@@ -0,0 +1,7 @@
+Enhancement: Hardened tls settings
+
+Rest-server now uses a secure TLS cipher suite set by default. The minimum TLS
+version is now TLS 1.2 and can be further increased using the new `--tls-min-ver`
+option, allowing users to enforce stricter security requirements.
+
+https://github.com/restic/rest-server/pull/315

changelog/0.14.0_2025-05-31/pull-322

@@ -0,0 +1,11 @@
+Change: Update dependencies and require Go 1.23 or newer
+
+All dependencies have been updated. Rest-server now requires Go 1.23 or newer
+to build.
+
+This also disables support for TLS versions older than TLS 1.2. On Windows,
+rest-server now requires at least Windows 10 or Windows Server 2016. On macOS,
+rest-server now requires at least macOS 11 Big Sur.
+
+https://github.com/restic/rest-server/pull/322
+https://github.com/restic/rest-server/pull/338

changelog/CHANGELOG.tmpl

@@ -19,10 +19,10 @@ Details
    {{ wrapIndent $par 80 3 }}
 {{ end -}}
 {{ range $id := .Issues }}
-   https://github.com/restic/restic/issues/{{ $id -}}
+   https://github.com/restic/rest-server/issues/{{ $id -}}
 {{ end -}}
 {{ range $id := .PRs }}
-   https://github.com/restic/restic/pull/{{ $id -}}
+   https://github.com/restic/rest-server/pull/{{ $id -}}
 {{ end -}}
 {{ range $url := .OtherURLs }}
    {{ $url -}}

changelog/TEMPLATE

@@ -1,12 +1,20 @@
+# The first line must start with Bugfix:, Enhancement: or Change:,
+# including the colon. Use present tense. Remove lines starting with '#'
+# from this template.
 Bugfix: Fix behavior for foobar (in present tense)
 
+# Describe the problem in the past tense, the new behavior in the present
+# tense. Mention the affected commands, backends, operating systems, etc.
+# Focus on user-facing behavior, not the implementation.
+
 We've fixed the behavior for foobar, a long-standing annoyance for rest-server
 users.
 
-The text in the paragraphs is written in past tense. The last section is a list
-of issue URLs, PR URLs and other URLs. The first issue ID (or the first PR ID,
-in case there aren't any issue links) is used as the primary ID.
+# The last section is a list of issue, PR and forum URLs.
+# The first issue ID determines the filename for the changelog entry:
+# changelog/unreleased/issue-1234. If there are no relevant issue links,
+# use the PR ID and call the file pull-55555.
 
-https://github.com/restic/restic/issues/1234
-https://github.com/restic/restic/pull/55555
-https://forum.restic/.net/foo/bar/baz
+https://github.com/restic/rest-server/issues/1234
+https://github.com/restic/rest-server/pull/55555
+https://forum.restic.net/foo/bar/baz

cmd/rest-server/listener_unix.go

@@ -0,0 +1,58 @@
+//go:build !windows
+// +build !windows
+
+package main
+
+import (
+	"fmt"
+	"log"
+	"net"
+	"strings"
+
+	"github.com/coreos/go-systemd/v22/activation"
+)
+
+// findListener tries to find a listener via systemd socket activation. If that
+// fails, it tries to create a listener on addr.
+func findListener(addr string) (listener net.Listener, err error) {
+	// try systemd socket activation
+	listeners, err := activation.Listeners()
+	if err != nil {
+		panic(err)
+	}
+
+	switch len(listeners) {
+	case 0:
+		// no listeners found, listen manually
+		if strings.HasPrefix(addr, "unix:") { // if we want to listen on a unix socket
+			unixAddr, err := net.ResolveUnixAddr("unix", strings.TrimPrefix(addr, "unix:"))
+			if err != nil {
+				return nil, fmt.Errorf("unable to understand unix address %s: %w", addr, err)
+			}
+			listener, err = net.ListenUnix("unix", unixAddr)
+			if err != nil {
+				return nil, fmt.Errorf("listen on %v failed: %w", addr, err)
+			}
+		} else { // assume tcp
+			listener, err = net.Listen("tcp", addr)
+			if err != nil {
+				return nil, fmt.Errorf("listen on %v failed: %w", addr, err)
+			}
+		}
+
+		log.Printf("start server on %v", listener.Addr())
+		return listener, nil
+
+	case 1:
+		// one listener supplied by systemd, use that one
+		//
+		// for testing, run rest-server with systemd-socket-activate as follows:
+		//
+		//    systemd-socket-activate -l 8080 ./rest-server
+		log.Printf("systemd socket activation mode")
+		return listeners[0], nil
+
+	default:
+		return nil, fmt.Errorf("got %d listeners from systemd, expected one", len(listeners))
+	}
+}

cmd/rest-server/listener_unix_test.go

@@ -0,0 +1,78 @@
+//go:build !windows
+// +build !windows
+
+package main
+
+import (
+	"context"
+	"fmt"
+	"net"
+	"net/http"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+func TestUnixSocket(t *testing.T) {
+	td := t.TempDir()
+
+	// this is the socket we'll listen on and connect to
+	tempSocket := filepath.Join(td, "sock")
+
+	// create some content and parent dirs
+	if err := os.MkdirAll(filepath.Join(td, "data", "repo1"), 0700); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(filepath.Join(td, "data", "repo1", "config"), []byte("foo"), 0700); err != nil {
+		t.Fatal(err)
+	}
+
+	// run the following twice, to test that the server will
+	// cleanup its socket file when quitting, which won't happen
+	// if it doesn't exit gracefully
+	for i := 0; i < 2; i++ {
+		err := testServerWithArgs([]string{
+			"--no-auth",
+			"--path", filepath.Join(td, "data"),
+			"--listen", fmt.Sprintf("unix:%s", tempSocket),
+		}, time.Second, func(ctx context.Context, _ *restServerApp) error {
+			// custom client that will talk HTTP to unix socket
+			client := http.Client{
+				Transport: &http.Transport{
+					DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
+						return net.Dial("unix", tempSocket)
+					},
+				},
+			}
+			for _, test := range []struct {
+				Path       string
+				StatusCode int
+			}{
+				{"/repo1/", http.StatusMethodNotAllowed},
+				{"/repo1/config", http.StatusOK},
+				{"/repo2/config", http.StatusNotFound},
+			} {
+				req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://ignored"+test.Path, nil)
+				if err != nil {
+					return err
+				}
+				resp, err := client.Do(req)
+				if err != nil {
+					return err
+				}
+				err = resp.Body.Close()
+				if err != nil {
+					return err
+				}
+				if resp.StatusCode != test.StatusCode {
+					return fmt.Errorf("expected %d from server, instead got %d (path %s)", test.StatusCode, resp.StatusCode, test.Path)
+				}
+			}
+			return nil
+		})
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+}

cmd/rest-server/listener_windows.go

@@ -0,0 +1,19 @@
+package main
+
+import (
+	"fmt"
+	"log"
+	"net"
+)
+
+// findListener creates a listener.
+func findListener(addr string) (listener net.Listener, err error) {
+	// listen manually
+	listener, err = net.Listen("tcp", addr)
+	if err != nil {
+		return nil, fmt.Errorf("listen on %v failed: %w", addr, err)
+	}
+
+	log.Printf("start server on %v", listener.Addr())
+	return listener, nil
+}

cmd/rest-server/main.go

@@ -1,149 +1,240 @@
 package main
 
 import (
+	"context"
+	"crypto/tls"
 	"errors"
 	"fmt"
 	"log"
+	"net"
 	"net/http"
 	"os"
+	"os/signal"
 	"path/filepath"
 	"runtime"
 	"runtime/pprof"
+	"sync"
+	"syscall"
 
 	restserver "github.com/restic/rest-server"
 	"github.com/spf13/cobra"
 )
 
+type restServerApp struct {
+	CmdRoot    *cobra.Command
+	Server     restserver.Server
+	CPUProfile string
+
+	listenerAddressMu sync.Mutex
+	listenerAddress   net.Addr // set after startup
+}
+
 // cmdRoot is the base command when no other command has been specified.
-var cmdRoot = &cobra.Command{
-	Use:           "rest-server",
-	Short:         "Run a REST server for use with restic",
-	SilenceErrors: true,
-	SilenceUsage:  true,
-	RunE:          runRoot,
-	// Use this instead of other --version code when the Cobra dependency can be updated.
-	//Version:       fmt.Sprintf("rest-server %s compiled with %v on %v/%v\n", version, runtime.Version(), runtime.GOOS, runtime.GOARCH),
+func newRestServerApp() *restServerApp {
+	rv := &restServerApp{
+		CmdRoot: &cobra.Command{
+			Use:           "rest-server",
+			Short:         "Run a REST server for use with restic",
+			SilenceErrors: true,
+			SilenceUsage:  true,
+			Args: func(_ *cobra.Command, args []string) error {
+				if len(args) != 0 {
+					return fmt.Errorf("rest-server expects no arguments - unknown argument: %s", args[0])
+				}
+				return nil
+			},
+			Version: fmt.Sprintf("rest-server %s compiled with %v on %v/%v\n", version, runtime.Version(), runtime.GOOS, runtime.GOARCH),
+		},
+		Server: restserver.Server{
+			Path:      filepath.Join(os.TempDir(), "restic"),
+			Listen:    ":8000",
+			TLSMinVer: "1.2",
+		},
+	}
+	rv.CmdRoot.RunE = rv.runRoot
+	flags := rv.CmdRoot.Flags()
+
+	flags.StringVar(&rv.CPUProfile, "cpu-profile", rv.CPUProfile, "write CPU profile to file")
+	flags.BoolVar(&rv.Server.Debug, "debug", rv.Server.Debug, "output debug messages")
+	flags.StringVar(&rv.Server.Listen, "listen", rv.Server.Listen, "listen address")
+	flags.StringVar(&rv.Server.Log, "log", rv.Server.Log, "write HTTP requests in the combined log format to the specified `filename` (use \"-\" for logging to stdout)")
+	flags.Int64Var(&rv.Server.MaxRepoSize, "max-size", rv.Server.MaxRepoSize, "the maximum size of the repository in bytes")
+	flags.StringVar(&rv.Server.Path, "path", rv.Server.Path, "data directory")
+	flags.BoolVar(&rv.Server.TLS, "tls", rv.Server.TLS, "turn on TLS support")
+	flags.StringVar(&rv.Server.TLSCert, "tls-cert", rv.Server.TLSCert, "TLS certificate path")
+	flags.StringVar(&rv.Server.TLSKey, "tls-key", rv.Server.TLSKey, "TLS key path")
+	flags.StringVar(&rv.Server.TLSMinVer, "tls-min-ver", rv.Server.TLSMinVer, "TLS min version, one of (1.2|1.3)")
+	flags.BoolVar(&rv.Server.NoAuth, "no-auth", rv.Server.NoAuth, "disable authentication")
+	flags.StringVar(&rv.Server.HtpasswdPath, "htpasswd-file", rv.Server.HtpasswdPath, "location of .htpasswd file (default: \"<data directory>/.htpasswd)\"")
+	flags.StringVar(&rv.Server.ProxyAuthUsername, "proxy-auth-username", rv.Server.ProxyAuthUsername, "specifies the HTTP header containing the username for proxy-based authentication")
+	flags.BoolVar(&rv.Server.NoVerifyUpload, "no-verify-upload", rv.Server.NoVerifyUpload,
+		"do not verify the integrity of uploaded data. DO NOT enable unless the rest-server runs on a very low-power device")
+	flags.BoolVar(&rv.Server.AppendOnly, "append-only", rv.Server.AppendOnly, "enable append only mode")
+	flags.BoolVar(&rv.Server.PrivateRepos, "private-repos", rv.Server.PrivateRepos, "users can only access their private repo")
+	flags.BoolVar(&rv.Server.Prometheus, "prometheus", rv.Server.Prometheus, "enable Prometheus metrics")
+	flags.BoolVar(&rv.Server.PrometheusNoAuth, "prometheus-no-auth", rv.Server.PrometheusNoAuth, "disable auth for Prometheus /metrics endpoint")
+	flags.BoolVar(&rv.Server.GroupAccessibleRepos, "group-accessible-repos", rv.Server.GroupAccessibleRepos, "let filesystem group be able to access repo files")
+
+	return rv
 }
 
-var server = restserver.Server{
-	Path:   "/tmp/restic",
-	Listen: ":8000",
-}
+var version = "0.14.0-dev"
 
-var (
-	showVersion bool
-	cpuProfile  string
-)
-
-func init() {
-	flags := cmdRoot.Flags()
-	flags.StringVar(&cpuProfile, "cpu-profile", cpuProfile, "write CPU profile to file")
-	flags.BoolVar(&server.Debug, "debug", server.Debug, "output debug messages")
-	flags.StringVar(&server.Listen, "listen", server.Listen, "listen address")
-	flags.StringVar(&server.Log, "log", server.Log, "log HTTP requests in the combined log format")
-	flags.Int64Var(&server.MaxRepoSize, "max-size", server.MaxRepoSize, "the maximum size of the repository in bytes")
-	flags.StringVar(&server.Path, "path", server.Path, "data directory")
-	flags.BoolVar(&server.TLS, "tls", server.TLS, "turn on TLS support")
-	flags.StringVar(&server.TLSCert, "tls-cert", server.TLSCert, "TLS certificate path")
-	flags.StringVar(&server.TLSKey, "tls-key", server.TLSKey, "TLS key path")
-	flags.BoolVar(&server.NoAuth, "no-auth", server.NoAuth, "disable .htpasswd authentication")
-	flags.BoolVar(&server.AppendOnly, "append-only", server.AppendOnly, "enable append only mode")
-	flags.BoolVar(&server.PrivateRepos, "private-repos", server.PrivateRepos, "users can only access their private repo")
-	flags.BoolVar(&server.Prometheus, "prometheus", server.Prometheus, "enable Prometheus metrics")
-	flags.BoolVarP(&showVersion, "version", "V", showVersion, "output version and exit")
-}
-
-var version = "0.10.0"
-
-func tlsSettings() (bool, string, string, error) {
+func (app *restServerApp) tlsSettings() (bool, string, string, error) {
 	var key, cert string
-	if !server.TLS && (server.TLSKey != "" || server.TLSCert != "") {
+	if !app.Server.TLS && (app.Server.TLSKey != "" || app.Server.TLSCert != "") {
 		return false, "", "", errors.New("requires enabled TLS")
-	} else if !server.TLS {
+	} else if !app.Server.TLS {
 		return false, "", "", nil
 	}
-	if server.TLSKey != "" {
-		key = server.TLSKey
+	if app.Server.TLSKey != "" {
+		key = app.Server.TLSKey
 	} else {
-		key = filepath.Join(server.Path, "private_key")
+		key = filepath.Join(app.Server.Path, "private_key")
 	}
-	if server.TLSCert != "" {
-		cert = server.TLSCert
+	if app.Server.TLSCert != "" {
+		cert = app.Server.TLSCert
 	} else {
-		cert = filepath.Join(server.Path, "public_key")
+		cert = filepath.Join(app.Server.Path, "public_key")
 	}
-	return server.TLS, key, cert, nil
+	return app.Server.TLS, key, cert, nil
 }
 
-func getHandler(server restserver.Server) (http.Handler, error) {
-	mux := restserver.NewHandler(server)
-	if server.NoAuth {
-		log.Println("Authentication disabled")
-		return mux, nil
-	}
-
-	log.Println("Authentication enabled")
-	htpasswdFile, err := restserver.NewHtpasswdFromFile(filepath.Join(server.Path, ".htpasswd"))
-	if err != nil {
-		return nil, fmt.Errorf("cannot load .htpasswd (use --no-auth to disable): %v", err)
-	}
-	return server.AuthHandler(htpasswdFile, mux), nil
+// returns the address that the app is listening on.
+// returns nil if the application hasn't finished starting yet
+func (app *restServerApp) ListenerAddress() net.Addr {
+	app.listenerAddressMu.Lock()
+	defer app.listenerAddressMu.Unlock()
+	return app.listenerAddress
 }
 
-func runRoot(cmd *cobra.Command, args []string) error {
-	if showVersion {
-		fmt.Printf("rest-server %s compiled with %v on %v/%v\n", version, runtime.Version(), runtime.GOOS, runtime.GOARCH)
-		os.Exit(0)
-	}
-
+func (app *restServerApp) runRoot(_ *cobra.Command, _ []string) error {
 	log.SetFlags(0)
 
-	log.Printf("Data directory: %s", server.Path)
+	log.Printf("Data directory: %s", app.Server.Path)
 
-	if cpuProfile != "" {
-		f, err := os.Create(cpuProfile)
+	if app.CPUProfile != "" {
+		f, err := os.Create(app.CPUProfile)
 		if err != nil {
 			return err
 		}
+		defer func() {
+			_ = f.Close()
+		}()
+
 		if err := pprof.StartCPUProfile(f); err != nil {
 			return err
 		}
-		log.Println("CPU profiling enabled")
 		defer pprof.StopCPUProfile()
+
+		log.Println("CPU profiling enabled")
+		defer log.Println("Stopped CPU profiling")
 	}
 
-	handler, err := getHandler(server)
+	if app.Server.NoAuth {
+		log.Println("Authentication disabled")
+	} else {
+		if app.Server.ProxyAuthUsername == "" {
+			log.Println("Authentication enabled")
+		} else {
+			log.Println("Proxy Authentication enabled.")
+		}
+	}
+
+	handler, err := restserver.NewHandler(&app.Server)
 	if err != nil {
 		log.Fatalf("error: %v", err)
 	}
 
-	if server.PrivateRepos {
+	if app.Server.AppendOnly {
+		log.Println("Append only mode enabled")
+	} else {
+		log.Println("Append only mode disabled")
+	}
+
+	if app.Server.PrivateRepos {
 		log.Println("Private repositories enabled")
 	} else {
 		log.Println("Private repositories disabled")
 	}
 
-	enabledTLS, privateKey, publicKey, err := tlsSettings()
+	if app.Server.GroupAccessibleRepos {
+		log.Println("Group accessible repos enabled")
+	} else {
+		log.Println("Group accessible repos disabled")
+	}
+
+	enabledTLS, privateKey, publicKey, err := app.tlsSettings()
 	if err != nil {
 		return err
 	}
-	if !enabledTLS {
-		log.Printf("Starting server on %s\n", server.Listen)
-		err = http.ListenAndServe(server.Listen, handler)
-	} else {
 
-		log.Println("TLS enabled")
-		log.Printf("Private key: %s", privateKey)
-		log.Printf("Public key(certificate): %s", publicKey)
-		log.Printf("Starting server on %s\n", server.Listen)
-		err = http.ListenAndServeTLS(server.Listen, publicKey, privateKey, handler)
+	listener, err := findListener(app.Server.Listen)
+	if err != nil {
+		return fmt.Errorf("unable to listen: %w", err)
 	}
 
-	return err
+	// set listener address, this is useful for tests
+	app.listenerAddressMu.Lock()
+	app.listenerAddress = listener.Addr()
+	app.listenerAddressMu.Unlock()
+
+	tlscfg := &tls.Config{
+		MinVersion: tls.VersionTLS12,
+		CipherSuites: []uint16{
+			tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
+			tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
+			tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
+			tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
+			tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
+			tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
+		},
+	}
+	switch app.Server.TLSMinVer {
+	case "1.2":
+		tlscfg.MinVersion = tls.VersionTLS12
+	case "1.3":
+		tlscfg.MinVersion = tls.VersionTLS13
+	default:
+		return fmt.Errorf("Unsupported TLS min version: %s. Allowed versions are 1.2 or 1.3", app.Server.TLSMinVer)
+	}
+
+	srv := &http.Server{
+		Handler:   handler,
+		TLSConfig: tlscfg,
+	}
+
+	// run server in background
+	go func() {
+		if !enabledTLS {
+			err = srv.Serve(listener)
+		} else {
+			log.Printf("TLS enabled, private key %s, pubkey %v", privateKey, publicKey)
+			err = srv.ServeTLS(listener, publicKey, privateKey)
+		}
+		if err != nil && !errors.Is(err, http.ErrServerClosed) {
+			log.Fatalf("listen and serve returned err: %v", err)
+		}
+	}()
+
+	// wait until done
+	<-app.CmdRoot.Context().Done()
+
+	// gracefully shutdown server
+	if err := srv.Shutdown(context.Background()); err != nil {
+		return fmt.Errorf("server shutdown returned an err: %w", err)
+	}
+
+	log.Println("shutdown cleanly")
+	return nil
 }
 
 func main() {
-	if err := cmdRoot.Execute(); err != nil {
+	// create context to be notified on interrupt or term signal so that we can shutdown cleanly
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+
+	if err := newRestServerApp().CmdRoot.ExecuteContext(ctx); err != nil {
 		log.Fatalf("error: %v", err)
 	}
 }

cmd/rest-server/main_test.go

@@ -1,10 +1,17 @@
 package main
 
 import (
-	"io/ioutil"
+	"context"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/url"
 	"os"
 	"path/filepath"
+	"strings"
+	"sync"
 	"testing"
+	"time"
 
 	restserver "github.com/restic/rest-server"
 )
@@ -27,26 +34,37 @@ func TestTLSSettings(t *testing.T) {
 		expected expected
 	}{
 		{passed{TLS: false}, expected{"", "", false}},
-		{passed{TLS: true}, expected{"/tmp/restic/private_key", "/tmp/restic/public_key", false}},
-		{passed{Path: "/tmp", TLS: true}, expected{"/tmp/private_key", "/tmp/public_key", false}},
-		{passed{Path: "/tmp", TLS: true, TLSKey: "/etc/restic/key", TLSCert: "/etc/restic/cert"}, expected{"/etc/restic/key", "/etc/restic/cert", false}},
-		{passed{Path: "/tmp", TLS: false, TLSKey: "/etc/restic/key", TLSCert: "/etc/restic/cert"}, expected{"", "", true}},
-		{passed{Path: "/tmp", TLS: false, TLSKey: "/etc/restic/key"}, expected{"", "", true}},
-		{passed{Path: "/tmp", TLS: false, TLSCert: "/etc/restic/cert"}, expected{"", "", true}},
+		{passed{TLS: true}, expected{
+			filepath.Join(os.TempDir(), "restic/private_key"),
+			filepath.Join(os.TempDir(), "restic/public_key"),
+			false,
+		}},
+		{passed{
+			Path: os.TempDir(),
+			TLS:  true,
+		}, expected{
+			filepath.Join(os.TempDir(), "private_key"),
+			filepath.Join(os.TempDir(), "public_key"),
+			false,
+		}},
+		{passed{Path: os.TempDir(), TLS: true, TLSKey: "/etc/restic/key", TLSCert: "/etc/restic/cert"}, expected{"/etc/restic/key", "/etc/restic/cert", false}},
+		{passed{Path: os.TempDir(), TLS: false, TLSKey: "/etc/restic/key", TLSCert: "/etc/restic/cert"}, expected{"", "", true}},
+		{passed{Path: os.TempDir(), TLS: false, TLSKey: "/etc/restic/key"}, expected{"", "", true}},
+		{passed{Path: os.TempDir(), TLS: false, TLSCert: "/etc/restic/cert"}, expected{"", "", true}},
 	}
 
 	for _, test := range tests {
-
+		app := newRestServerApp()
 		t.Run("", func(t *testing.T) {
 			// defer func() { restserver.Server = defaultConfig }()
 			if test.passed.Path != "" {
-				server.Path = test.passed.Path
+				app.Server.Path = test.passed.Path
 			}
-			server.TLS = test.passed.TLS
-			server.TLSKey = test.passed.TLSKey
-			server.TLSCert = test.passed.TLSCert
+			app.Server.TLS = test.passed.TLS
+			app.Server.TLSKey = test.passed.TLSKey
+			app.Server.TLSCert = test.passed.TLSCert
 
-			gotTLS, gotKey, gotCert, err := tlsSettings()
+			gotTLS, gotKey, gotCert, err := app.tlsSettings()
 			if err != nil && !test.expected.Error {
 				t.Fatalf("tls_settings returned err (%v)", err)
 			}
@@ -75,35 +93,192 @@ func TestTLSSettings(t *testing.T) {
 }
 
 func TestGetHandler(t *testing.T) {
-	dir, err := ioutil.TempDir("", "rest-server-test")
+	dir, err := os.MkdirTemp("", "rest-server-test")
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer os.Remove(dir)
+	defer func() {
+		err := os.Remove(dir)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	getHandler := restserver.NewHandler
 
 	// With NoAuth = false and no .htpasswd
-	_, err = getHandler(restserver.Server{Path: dir})
+	_, err = getHandler(&restserver.Server{Path: dir})
 	if err == nil {
 		t.Errorf("NoAuth=false: expected error, got nil")
 	}
 
 	// With NoAuth = true and no .htpasswd
-	_, err = getHandler(restserver.Server{NoAuth: true, Path: dir})
+	_, err = getHandler(&restserver.Server{NoAuth: true, Path: dir})
 	if err != nil {
 		t.Errorf("NoAuth=true: expected no error, got %v", err)
 	}
 
-	// Create .htpasswd
-	htpasswd := filepath.Join(dir, ".htpasswd")
-	err = ioutil.WriteFile(htpasswd, []byte(""), 0644)
+	// With NoAuth = false, no .htpasswd and ProxyAuth = X-Remote-User
+	_, err = getHandler(&restserver.Server{Path: dir, ProxyAuthUsername: "X-Remote-User"})
+	if err != nil {
+		t.Errorf("NoAuth=false, ProxyAuthUsername = X-Remote-User: expected no error, got %v", err)
+	}
+
+	// With NoAuth = false and custom .htpasswd
+	htpFile, err := os.CreateTemp(dir, "custom")
 	if err != nil {
 		t.Fatal(err)
 	}
-	defer os.Remove(htpasswd)
+	defer func() {
+		err := os.Remove(htpFile.Name())
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+	_, err = getHandler(&restserver.Server{HtpasswdPath: htpFile.Name()})
+	if err != nil {
+		t.Errorf("NoAuth=false with custom htpasswd: expected no error, got %v", err)
+	}
+
+	// Create .htpasswd
+	htpasswd := filepath.Join(dir, ".htpasswd")
+	err = os.WriteFile(htpasswd, []byte(""), 0644)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() {
+		err := os.Remove(htpasswd)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
 
 	// With NoAuth = false and with .htpasswd
-	_, err = getHandler(restserver.Server{Path: dir})
+	_, err = getHandler(&restserver.Server{Path: dir})
 	if err != nil {
 		t.Errorf("NoAuth=false with .htpasswd: expected no error, got %v", err)
 	}
 }
+
+// helper method to test the app. Starts app with passed arguments,
+// then will call the callback function which can make requests against
+// the application. If the callback function fails due to errors returned
+// by http.Do() (i.e. *url.Error), then it will be retried until successful,
+// or the passed timeout passes.
+func testServerWithArgs(args []string, timeout time.Duration, cb func(context.Context, *restServerApp) error) error {
+	// create the app with passed args
+	app := newRestServerApp()
+	app.CmdRoot.SetArgs(args)
+
+	// create context that will timeout
+	ctx, cancel := context.WithTimeout(context.Background(), timeout)
+	defer cancel()
+
+	// wait group for our client and server tasks
+	jobs := &sync.WaitGroup{}
+	jobs.Add(2)
+
+	// run the server, saving the error
+	var serverErr error
+	go func() {
+		defer jobs.Done()
+		defer cancel() // if the server is stopped, no point keep the client alive
+		serverErr = app.CmdRoot.ExecuteContext(ctx)
+	}()
+
+	// run the client, saving the error
+	var clientErr error
+	go func() {
+		defer jobs.Done()
+		defer cancel() // once the client is done, stop the server
+
+		var urlError *url.Error
+
+		// execute in loop, as we will retry for network errors
+		// (such as the server hasn't started yet)
+		for {
+			clientErr = cb(ctx, app)
+			switch {
+			case clientErr == nil:
+				return // success, we're done
+			case errors.As(clientErr, &urlError):
+				// if a network error (url.Error), then wait and retry
+				// as server may not be ready yet
+				select {
+				case <-time.After(time.Millisecond * 100):
+					continue
+				case <-ctx.Done(): // unless we run out of time first
+					clientErr = context.Canceled
+					return
+				}
+			default:
+				return // other error type, we're done
+			}
+		}
+	}()
+
+	// wait for both to complete
+	jobs.Wait()
+
+	// report back if either failed
+	if clientErr != nil || serverErr != nil {
+		return fmt.Errorf("client or server error, client: %v, server: %v", clientErr, serverErr)
+	}
+
+	return nil
+}
+
+func TestHttpListen(t *testing.T) {
+	td := t.TempDir()
+
+	// create some content and parent dirs
+	if err := os.MkdirAll(filepath.Join(td, "data", "repo1"), 0700); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(filepath.Join(td, "data", "repo1", "config"), []byte("foo"), 0700); err != nil {
+		t.Fatal(err)
+	}
+
+	for _, args := range [][]string{
+		{"--no-auth", "--path", filepath.Join(td, "data"), "--listen", "127.0.0.1:0"},    // test emphemeral port
+		{"--no-auth", "--path", filepath.Join(td, "data"), "--listen", "127.0.0.1:9000"}, // test "normal" port
+		{"--no-auth", "--path", filepath.Join(td, "data"), "--listen", "127.0.0.1:9000"}, // test that server was shutdown cleanly and that we can re-use that port
+	} {
+		err := testServerWithArgs(args, time.Second*10, func(ctx context.Context, app *restServerApp) error {
+			for _, test := range []struct {
+				Path       string
+				StatusCode int
+			}{
+				{"/repo1/", http.StatusMethodNotAllowed},
+				{"/repo1/config", http.StatusOK},
+				{"/repo2/config", http.StatusNotFound},
+			} {
+				listenAddr := app.ListenerAddress()
+				if listenAddr == nil {
+					return &url.Error{} // return this type of err, as we know this will retry
+				}
+				port := strings.Split(listenAddr.String(), ":")[1]
+
+				req, err := http.NewRequestWithContext(ctx, http.MethodGet, fmt.Sprintf("http://localhost:%s%s", port, test.Path), nil)
+				if err != nil {
+					return err
+				}
+				resp, err := http.DefaultClient.Do(req)
+				if err != nil {
+					return err
+				}
+				err = resp.Body.Close()
+				if err != nil {
+					return err
+				}
+				if resp.StatusCode != test.StatusCode {
+					return fmt.Errorf("expected %d from server, instead got %d (path %s)", test.StatusCode, resp.StatusCode, test.Path)
+				}
+			}
+			return nil
+		})
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+}

docker/create_user

@@ -9,8 +9,8 @@ fi
 
 if [ -z "$2" ]; then
     # password from prompt
-    htpasswd -s $PASSWORD_FILE $1
+    htpasswd -B "$PASSWORD_FILE" "$1"
 else
     # read password from command line
-    htpasswd -s -b $PASSWORD_FILE $1 $2
+    htpasswd -B -b "$PASSWORD_FILE" "$1" "$2"
 fi

docker/delete_user

@@ -5,4 +5,4 @@ if [ -z "$1" ]; then
     exit 1
 fi
 
-htpasswd -D $PASSWORD_FILE $1
+htpasswd -D "$PASSWORD_FILE" "$1"

docker/entrypoint.sh

@@ -2,9 +2,11 @@
 
 set -e
 
-if [ -z "$DISABLE_AUTHENTICATION" ]; then
+if [ -n "$DISABLE_AUTHENTICATION" ]; then
+    OPTIONS="--no-auth $OPTIONS"
+else
     if [ ! -f "$PASSWORD_FILE" ]; then
-        touch "$PASSWORD_FILE"
+        ( umask 027 && touch "$PASSWORD_FILE" )
     fi
 
     if [ ! -s "$PASSWORD_FILE" ]; then
@@ -14,4 +16,4 @@ if [ -z "$DISABLE_AUTHENTICATION" ]; then
     fi
 fi
 
-exec rest-server --path "$DATA_DIRECTORY" $OPTIONS
+exec rest-server --path "$DATA_DIRECTORY" --htpasswd-file "$PASSWORD_FILE" $OPTIONS

examples/compose-with-grafana/dashboards/rest-server.json

@@ -1,4 +1,14 @@
 {
+  "__inputs": [
+    {
+      "name": "DS_PROMETHEUS-INFRA",
+      "label": "prometheus-infra",
+      "description": "",
+      "type": "datasource",
+      "pluginId": "prometheus",
+      "pluginName": "Prometheus"
+    }
+  ],
   "__requires": [
     {
       "type": "grafana",
@@ -12,7 +22,7 @@
       "name": "Graph",
       "version": ""
     },
-    {
+	{
       "type": "datasource",
       "id": "prometheus",
       "name": "Prometheus",
@@ -49,7 +59,7 @@
           "bars": false,
           "dashLength": 10,
           "dashes": false,
-          "datasource": "prometheus",
+		  "datasource": "${DS_PROMETHEUS-INFRA}",
           "fill": 1,
           "id": 1,
           "legend": {
@@ -125,7 +135,7 @@
           "bars": false,
           "dashLength": 10,
           "dashes": false,
-          "datasource": "prometheus",
+		  "datasource": "${DS_PROMETHEUS-INFRA}",
           "fill": 1,
           "id": 4,
           "legend": {
@@ -213,7 +223,7 @@
           "bars": false,
           "dashLength": 10,
           "dashes": false,
-          "datasource": "prometheus",
+		  "datasource": "${DS_PROMETHEUS-INFRA}",
           "fill": 1,
           "id": 2,
           "legend": {
@@ -289,7 +299,7 @@
           "bars": false,
           "dashLength": 10,
           "dashes": false,
-          "datasource": "prometheus",
+		  "datasource": "${DS_PROMETHEUS-INFRA}",
           "fill": 1,
           "id": 5,
           "legend": {
@@ -377,7 +387,7 @@
           "bars": false,
           "dashLength": 10,
           "dashes": false,
-          "datasource": "prometheus",
+		  "datasource": "${DS_PROMETHEUS-INFRA}",
           "fill": 1,
           "id": 3,
           "legend": {
@@ -453,7 +463,7 @@
           "bars": false,
           "dashLength": 10,
           "dashes": false,
-          "datasource": "prometheus",
+		  "datasource": "${DS_PROMETHEUS-INFRA}",
           "fill": 1,
           "id": 6,
           "legend": {
@@ -541,7 +551,7 @@
       {
         "allValue": null,
         "current": {},
-        "datasource": "prometheus",
+		"datasource": "${DS_PROMETHEUS-INFRA}",
         "hide": 0,
         "includeAll": false,
         "label": "Instance",

examples/compose-with-grafana/docker-compose.yml

@@ -24,7 +24,7 @@ services:
       - "127.0.0.1:8020:9090"
     volumes:
       - prometheusdata:/prometheus
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./prometheus:/etc/prometheus:ro
     depends_on:
       - restserver
     networks:

examples/systemd/rest-server.service

@@ -2,14 +2,82 @@
 Description=Rest Server
 After=syslog.target
 After=network.target
+Requires=rest-server.socket
+After=rest-server.socket
 
 [Service]
 Type=simple
+# You may prefer to use a different user or group on your system.
 User=www-data
 Group=www-data
-ExecStart=/usr/local/bin/rest-server --path /tmp/restic
+ExecStart=/usr/local/bin/rest-server --path /path/to/backups
 Restart=always
 RestartSec=5
 
+# The following options are available (in systemd v247) to restrict the
+# actions of the rest-server.
+
+# As a whole, the purpose of these are to provide an additional layer of
+# security by mitigating any unknown security vulnerabilities which may exist
+# in rest-server or in the libraries, tools and operating system components
+# which it relies upon.
+
+# IMPORTANT!
+# The following line must be customised to your individual requirements.
+ReadWritePaths=/path/to/backups
+
+# Files in the data repository are only user accessible by default. Default to
+# `UMask=077` for consistency. To make created files group-readable, set to
+# `UMask=007` and pass `--group-accessible-repos` to rest-server via `ExecStart`.
+UMask=077
+
+# If your system doesn't support all of the features below (e.g. because of
+# the use of an older version of systemd), you may wish to comment-out
+# some of the lines below as appropriate.
+CapabilityBoundingSet=
+LockPersonality=true
+MemoryDenyWriteExecute=true
+NoNewPrivileges=yes
+
+# As the listen socket is created by systemd via the rest-server.socket unit, it is
+# no longer necessary for rest-server to have access to the host network namespace.
+PrivateNetwork=yes
+
+PrivateTmp=yes
+PrivateDevices=true
+PrivateUsers=true
+ProtectSystem=strict
+ProtectHome=yes
+ProtectClock=true
+ProtectControlGroups=true
+ProtectKernelLogs=true
+ProtectKernelModules=true
+ProtectKernelTunables=true
+ProtectProc=invisible
+ProtectHostname=true
+RemoveIPC=true
+RestrictNamespaces=true
+RestrictAddressFamilies=none
+RestrictSUIDSGID=true
+RestrictRealtime=true
+# if your service crashes with "code=killed, status=31/SYS", you probably tried to run linux_i386 (32bit) binary on a amd64 host
+SystemCallArchitectures=native
+SystemCallFilter=@system-service
+
+# Additionally, you may wish to use some of the systemd options documented in
+# systemd.resource-control(5) to limit the CPU, memory, file-system I/O and
+# network I/O that the rest-server is permitted to consume according to the
+# individual requirements of your installation.
+#CPUQuota=25%
+#MemoryHigh=bytes
+#MemoryMax=bytes
+#MemorySwapMax=bytes
+#TasksMax=N
+#IOReadBandwidthMax=device bytes
+#IOWriteBandwidthMax=device bytes
+#IOReadIOPSMax=device IOPS, IOWriteIOPSMax=device IOPS
+#IPAccounting=true
+#IPAddressAllow=
+
 [Install]
 WantedBy=multi-user.target

examples/systemd/rest-server.socket

@@ -0,0 +1,5 @@
+[Socket]
+ListenStream = 8000
+
+[Install]
+WantedBy = sockets.target

go.mod

@@ -1,25 +1,28 @@
 module github.com/restic/rest-server
 
-go 1.14
+go 1.23.0
 
 require (
-	github.com/beorn7/perks v0.0.0-20160804104726-4c0e84591b9a // indirect
-	github.com/golang/protobuf v1.0.0 // indirect
-	github.com/gorilla/handlers v1.3.0
-	github.com/inconshreveable/mousetrap v1.0.0 // indirect
-	github.com/matttproud/golang_protobuf_extensions v1.0.0 // indirect
-	github.com/miolini/datacounter v0.0.0-20171104152933-fd4e42a1d5e0
-	github.com/prometheus/client_golang v0.8.0
-	github.com/prometheus/client_model v0.0.0-20171117100541-99fa1f4be8e5 // indirect
-	github.com/prometheus/common v0.0.0-20180110214958-89604d197083 // indirect
-	github.com/prometheus/procfs v0.0.0-20180212145926-282c8707aa21 // indirect
-	github.com/spf13/cobra v0.0.1
-	github.com/spf13/pflag v1.0.0 // indirect
-	goji.io v2.0.2+incompatible
-	golang.org/x/crypto v0.0.0-20180214000028-650f4a345ab4
-	golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a // indirect
+	github.com/coreos/go-systemd/v22 v22.5.0
+	github.com/gorilla/handlers v1.5.2
+	github.com/minio/sha256-simd v1.0.1
+	github.com/miolini/datacounter v1.0.3
+	github.com/prometheus/client_golang v1.22.0
+	github.com/spf13/cobra v1.9.1
+	golang.org/x/crypto v0.38.0
 )
 
-replace goji.io v2.0.0+incompatible => github.com/goji/goji v2.0.0+incompatible
-
-replace github.com/gorilla/handlers v1.3.0 => github.com/gorilla/handlers v1.3.0
+require (
+	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/felixge/httpsnoop v1.0.4 // indirect
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.9 // indirect
+	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/prometheus/client_model v0.6.1 // indirect
+	github.com/prometheus/common v0.62.0 // indirect
+	github.com/prometheus/procfs v0.15.1 // indirect
+	github.com/spf13/pflag v1.0.6 // indirect
+	golang.org/x/sys v0.33.0 // indirect
+	google.golang.org/protobuf v1.36.5 // indirect
+)

go.sum

@@ -1,30 +1,56 @@
-github.com/beorn7/perks v0.0.0-20160804104726-4c0e84591b9a h1:BtpsbiV638WQZwhA98cEZw2BsbnQJrbd0BI7tsy0W1c=
-github.com/beorn7/perks v0.0.0-20160804104726-4c0e84591b9a/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
-github.com/golang/protobuf v1.0.0 h1:lsek0oXi8iFE9L+EXARyHIjU5rlWIhhTkjDz3vHhWWQ=
-github.com/golang/protobuf v1.0.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/gorilla/handlers v1.3.0 h1:tsg9qP3mjt1h4Roxp+M1paRjrVBfPSOpBuVclh6YluI=
-github.com/gorilla/handlers v1.3.0/go.mod h1:Qkdc/uu4tH4g6mTK6auzZ766c4CA0Ng8+o/OAirnOIQ=
-github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
-github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
-github.com/matttproud/golang_protobuf_extensions v1.0.0 h1:YNOwxxSJzSUARoD9KRZLzM9Y858MNGCOACTvCW9TSAc=
-github.com/matttproud/golang_protobuf_extensions v1.0.0/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
-github.com/miolini/datacounter v0.0.0-20171104152933-fd4e42a1d5e0 h1:clkDYGefEWUCwyCrwYn900sOaVGDpinPJgD0W6ebEjs=
-github.com/miolini/datacounter v0.0.0-20171104152933-fd4e42a1d5e0/go.mod h1:P6fDJzlxN+cWYR09KbE9/ta+Y6JofX9tAUhJpWkWPaM=
-github.com/prometheus/client_golang v0.8.0 h1:1921Yw9Gc3iSc4VQh3PIoOqgPCZS7G/4xQNVUp8Mda8=
-github.com/prometheus/client_golang v0.8.0/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
-github.com/prometheus/client_model v0.0.0-20171117100541-99fa1f4be8e5 h1:cLL6NowurKLMfCeQy4tIeph12XNQWgANCNvdyrOYKV4=
-github.com/prometheus/client_model v0.0.0-20171117100541-99fa1f4be8e5/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
-github.com/prometheus/common v0.0.0-20180110214958-89604d197083 h1:BVsJT8+ZbyuL3hypz/HmEiM8h2P6hBQGig4el9/MdjA=
-github.com/prometheus/common v0.0.0-20180110214958-89604d197083/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
-github.com/prometheus/procfs v0.0.0-20180212145926-282c8707aa21 h1:t1goPR/clmILdOPOVkZytGpQMMTHZ0IkF4+AaZZfMJY=
-github.com/prometheus/procfs v0.0.0-20180212145926-282c8707aa21/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
-github.com/spf13/cobra v0.0.1 h1:zZh3X5aZbdnoj+4XkaBxKfhO4ot82icYdhhREIAXIj8=
-github.com/spf13/cobra v0.0.1/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
-github.com/spf13/pflag v1.0.0 h1:oaPbdDe/x0UncahuwiPxW1GYJyilRAdsPnq3e1yaPcI=
-github.com/spf13/pflag v1.0.0/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
-goji.io v2.0.2+incompatible h1:uIssv/elbKRLznFUy3Xj4+2Mz/qKhek/9aZQDUMae7c=
-goji.io v2.0.2+incompatible/go.mod h1:sbqFwrtqZACxLBTQcdgVjFh54yGVCvwq8+w49MVMMIk=
-golang.org/x/crypto v0.0.0-20180214000028-650f4a345ab4 h1:OfaUle5HH9Y0obNU74mlOZ/Igdtwi3eGOKcljJsTnbw=
-golang.org/x/crypto v0.0.0-20180214000028-650f4a345ab4/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
-golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a h1:WXEvlFVvvGxCJLG6REjsT03iWnKLEWinaScsxF2Vm2o=
-golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
+github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
+github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs=
+github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
+github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
+github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
+github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
+github.com/gorilla/handlers v1.5.2 h1:cLTUSsNkgcwhgRqvCNmdbRWG0A3N4F+M2nWKdScwyEE=
+github.com/gorilla/handlers v1.5.2/go.mod h1:dX+xVpaxdSw+q0Qek8SSsl3dfMk3jNddUkMzo0GtH0w=
+github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
+github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
+github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
+github.com/klauspost/cpuid/v2 v2.2.9 h1:66ze0taIn2H33fBvCkXuv9BmCwDfafmiIVpKV9kKGuY=
+github.com/klauspost/cpuid/v2 v2.2.9/go.mod h1:rqkxqrZ1EhYM9G+hXH7YdowN5R5RGN6NK4QwQ3WMXF8=
+github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
+github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
+github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM=
+github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8=
+github.com/miolini/datacounter v1.0.3 h1:tanOZPVblGXQl7/bSZWoEM8l4KK83q24qwQLMrO/HOA=
+github.com/miolini/datacounter v1.0.3/go.mod h1:C45dc2hBumHjDpEU64IqPwR6TDyPVpzOqqRTN7zmBUA=
+github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
+github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
+github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
+github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
+github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
+github.com/prometheus/common v0.62.0 h1:xasJaQlnWAeyHdUBeGjXmutelfJHWMRr+Fg4QszZ2Io=
+github.com/prometheus/common v0.62.0/go.mod h1:vyBcEuLSvWos9B1+CyL7JZ2up+uFzXhkqml0W5zIY1I=
+github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
+github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
+github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
+github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
+github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
+github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
+github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
+github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+golang.org/x/crypto v0.38.0 h1:jt+WWG8IZlBnVbomuhg2Mdq0+BBQaHbtqHEFEigjUV8=
+golang.org/x/crypto v0.38.0/go.mod h1:MvrbAqul58NNYPKnOra203SB9vpuZW0e+RRZV+Ggqjw=
+golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
+golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
+google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
+google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

handlers.go

@@ -1,61 +1,123 @@
 package restserver
 
 import (
-	"encoding/json"
 	"errors"
-	"fmt"
-	"io"
-	"io/ioutil"
 	"log"
 	"net/http"
-	"os"
 	"path"
 	"path/filepath"
 	"strings"
-	"time"
+	"sync"
 
-	"github.com/miolini/datacounter"
-	"github.com/prometheus/client_golang/prometheus"
-	"goji.io/middleware"
-	"goji.io/pat"
+	"github.com/restic/rest-server/quota"
+	"github.com/restic/rest-server/repo"
 )
 
-// Server determines how a Mux's handlers behave.
+// Server encapsulates the rest-server's settings and repo management logic
 type Server struct {
-	Path         string
-	Listen       string
-	Log          string
-	CPUProfile   string
-	TLSKey       string
-	TLSCert      string
-	TLS          bool
-	NoAuth       bool
-	AppendOnly   bool
-	PrivateRepos bool
-	Prometheus   bool
-	Debug        bool
-	MaxRepoSize  int64
+	Path                 string
+	HtpasswdPath         string
+	Listen               string
+	Log                  string
+	CPUProfile           string
+	TLSKey               string
+	TLSCert              string
+	TLSMinVer            string
+	TLS                  bool
+	NoAuth               bool
+	ProxyAuthUsername    string
+	AppendOnly           bool
+	PrivateRepos         bool
+	Prometheus           bool
+	PrometheusNoAuth     bool
+	Debug                bool
+	MaxRepoSize          int64
+	PanicOnError         bool
+	NoVerifyUpload       bool
+	GroupAccessibleRepos bool
 
-	repoSize int64 // must be accessed using sync/atomic
+	htpasswdFile *HtpasswdFile
+	quotaManager *quota.Manager
+	fsyncWarning sync.Once
 }
 
-func (s *Server) isHashed(dir string) bool {
-	return dir == "data"
+// MaxFolderDepth is the maxDepth param passed to splitURLPath.
+// A max depth of 2 mean that we accept folders like: '/', '/foo' and '/foo/bar'
+// TODO: Move to a Server option
+const MaxFolderDepth = 2
+
+// httpDefaultError write a HTTP error with the default description
+func httpDefaultError(w http.ResponseWriter, code int) {
+	http.Error(w, http.StatusText(code), code)
+}
+
+// ServeHTTP makes this server an http.Handler. It handlers the administrative
+// part of the request (figuring out the filesystem location, performing
+// authentication, etc) and then passes it on to repo.Handler for actual
+// REST API processing.
+func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	// First of all, check auth (will always pass if NoAuth is set)
+	username, ok := s.checkAuth(r)
+	if !ok {
+		httpDefaultError(w, http.StatusUnauthorized)
+		return
+	}
+
+	// Perform the path parsing to determine the repo folder and remainder for the
+	// repo handler.
+	folderPath, remainder := splitURLPath(r.URL.Path, MaxFolderDepth)
+	if !folderPathValid(folderPath) {
+		log.Printf("Invalid request path: %s", r.URL.Path)
+		httpDefaultError(w, http.StatusNotFound)
+		return
+	}
+
+	// Check if the current user is allowed to access this path
+	if !s.NoAuth && s.PrivateRepos {
+		if len(folderPath) == 0 || folderPath[0] != username {
+			httpDefaultError(w, http.StatusUnauthorized)
+			return
+		}
+	}
+
+	// Determine filesystem path for this repo
+	fsPath, err := join(s.Path, folderPath...)
+	if err != nil {
+		// We did not expect an error at this stage, because we just checked the path
+		log.Printf("Unexpected join error for path %q", r.URL.Path)
+		httpDefaultError(w, http.StatusNotFound)
+		return
+	}
+
+	// Pass the request to the repo.Handler
+	opt := repo.Options{
+		AppendOnly:      s.AppendOnly,
+		Debug:           s.Debug,
+		QuotaManager:    s.quotaManager, // may be nil
+		PanicOnError:    s.PanicOnError,
+		NoVerifyUpload:  s.NoVerifyUpload,
+		FsyncWarning:    &s.fsyncWarning,
+		GroupAccessible: s.GroupAccessibleRepos,
+	}
+	if s.Prometheus {
+		opt.BlobMetricFunc = makeBlobMetricFunc(username, folderPath)
+	}
+	repoHandler, err := repo.New(fsPath, opt)
+	if err != nil {
+		log.Printf("repo.New error: %v", err)
+		httpDefaultError(w, http.StatusInternalServerError)
+		return
+	}
+	r.URL.Path = remainder // strip folderPath for next handler
+	repoHandler.ServeHTTP(w, r)
 }
 
 func valid(name string) bool {
-	// Based on net/http.Dir
+	// taken from net/http.Dir
 	if strings.Contains(name, "\x00") {
 		return false
 	}
 
-	// Path characters that are disallowed or unsafe under some operating systems
-	// are not allowed here.
-	// The most important one here is '/', since Goji does not decode '%2F' to '/'
-	// during routing, so we can end up with a '/' in the name here.
-	if strings.ContainsAny(name, "/\\:*?\"<>|") {
-		return false
-	}
 	if filepath.Separator != '/' && strings.ContainsRune(name, filepath.Separator) {
 		return false
 	}
@@ -63,15 +125,17 @@ func valid(name string) bool {
 	return true
 }
 
-var validTypes = []string{"data", "index", "keys", "locks", "snapshots", "config"}
-
-func (s *Server) isValidType(name string) bool {
-	for _, tpe := range validTypes {
+func isValidType(name string) bool {
+	for _, tpe := range repo.ObjectTypes {
+		if name == tpe {
+			return true
+		}
+	}
+	for _, tpe := range repo.FileTypes {
 		if name == tpe {
 			return true
 		}
 	}
-
 	return false
 }
 
@@ -95,600 +159,45 @@ func join(base string, names ...string) (string, error) {
 	return filepath.Join(clean...), nil
 }
 
-// getRepo returns the repository location, relative to s.Path.
-func (s *Server) getRepo(r *http.Request) string {
-	if strings.HasPrefix(fmt.Sprintf("%s", middleware.Pattern(r.Context())), "/:repo") {
-		return pat.Param(r, "repo")
+// splitURLPath splits the URL path into a folderPath of the subrepo, and
+// a remainder that can be passed to repo.Handler.
+// Example: /foo/bar/locks/0123... will be split into:
+//
+//	["foo", "bar"] and "/locks/0123..."
+func splitURLPath(urlPath string, maxDepth int) (folderPath []string, remainder string) {
+	if !strings.HasPrefix(urlPath, "/") {
+		// Really should start with "/"
+		return nil, urlPath
 	}
-
-	return "."
+	p := strings.SplitN(urlPath, "/", maxDepth+2)
+	// Skip the empty first one and the remainder in the last one
+	for _, name := range p[1 : len(p)-1] {
+		if isValidType(name) {
+			// We found a part that is a special repo file or dir
+			break
+		}
+		folderPath = append(folderPath, name)
+	}
+	// If the folder path is empty, the whole path is the remainder (do not strip '/')
+	if len(folderPath) == 0 {
+		return nil, urlPath
+	}
+	// Check that the urlPath starts with the reconstructed path, which should
+	// always be the case.
+	fullFolderPath := "/" + strings.Join(folderPath, "/")
+	if !strings.HasPrefix(urlPath, fullFolderPath) {
+		return nil, urlPath
+	}
+	return folderPath, urlPath[len(fullFolderPath):]
 }
 
-// getPath returns the path for a file type in the repo.
-func (s *Server) getPath(r *http.Request, fileType string) (string, error) {
-	if !s.isValidType(fileType) {
-		return "", errors.New("invalid file type")
-	}
-	return join(s.Path, s.getRepo(r), fileType)
-}
-
-// getFilePath returns the path for a file in the repo.
-func (s *Server) getFilePath(r *http.Request, fileType, name string) (string, error) {
-	if !s.isValidType(fileType) {
-		return "", errors.New("invalid file type")
-	}
-
-	if s.isHashed(fileType) {
-		if len(name) < 2 {
-			return "", errors.New("file name is too short")
-		}
-
-		return join(s.Path, s.getRepo(r), fileType, name[:2], name)
-	}
-
-	return join(s.Path, s.getRepo(r), fileType, name)
-}
-
-// getUser returns the username from the request, or an empty string if none.
-func (s *Server) getUser(r *http.Request) string {
-	username, _, ok := r.BasicAuth()
-	if !ok {
-		return ""
-	}
-	return username
-}
-
-// getMetricLabels returns the prometheus labels from the request.
-func (s *Server) getMetricLabels(r *http.Request) prometheus.Labels {
-	labels := prometheus.Labels{
-		"user": s.getUser(r),
-		"repo": s.getRepo(r),
-		"type": pat.Param(r, "type"),
-	}
-	return labels
-}
-
-// isUserPath checks if a request path is accessible by the user when using
-// private repositories.
-func isUserPath(username, path string) bool {
-	prefix := "/" + username
-	if !strings.HasPrefix(path, prefix) {
-		return false
-	}
-	return len(path) == len(prefix) || path[len(prefix)] == '/'
-}
-
-// AuthHandler wraps h with a http.HandlerFunc that performs basic authentication against the user/passwords pairs
-// stored in f and returns the http.HandlerFunc.
-func (s *Server) AuthHandler(f *HtpasswdFile, h http.Handler) http.HandlerFunc {
-	return func(w http.ResponseWriter, r *http.Request) {
-		username, password, ok := r.BasicAuth()
-		if !ok || !f.Validate(username, password) {
-			http.Error(w, http.StatusText(http.StatusUnauthorized), http.StatusUnauthorized)
-			return
-		}
-
-		// resolve all relative elements in the path
-		urlPath := path.Clean(r.URL.Path)
-		if s.PrivateRepos && !isUserPath(username, urlPath) && urlPath != "/metrics" {
-			http.Error(w, http.StatusText(http.StatusUnauthorized), http.StatusUnauthorized)
-			return
-		}
-		h.ServeHTTP(w, r)
-	}
-}
-
-// CheckConfig checks whether a configuration exists.
-func (s *Server) CheckConfig(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("CheckConfig()")
-	}
-	cfg, err := s.getPath(r, "config")
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	st, err := os.Stat(cfg)
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-		return
-	}
-
-	w.Header().Add("Content-Length", fmt.Sprint(st.Size()))
-}
-
-// GetConfig allows for a config to be retrieved.
-func (s *Server) GetConfig(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("GetConfig()")
-	}
-	cfg, err := s.getPath(r, "config")
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	bytes, err := ioutil.ReadFile(cfg)
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-		return
-	}
-
-	_, _ = w.Write(bytes)
-}
-
-// SaveConfig allows for a config to be saved.
-func (s *Server) SaveConfig(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("SaveConfig()")
-	}
-	cfg, err := s.getPath(r, "config")
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	f, err := os.OpenFile(cfg, os.O_CREATE|os.O_WRONLY|os.O_EXCL, 0600)
-	if err != nil && os.IsExist(err) {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusForbidden), http.StatusForbidden)
-		return
-	}
-
-	_, err = io.Copy(f, r.Body)
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	err = f.Close()
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	_ = r.Body.Close()
-}
-
-// DeleteConfig removes a config.
-func (s *Server) DeleteConfig(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("DeleteConfig()")
-	}
-
-	if s.AppendOnly {
-		http.Error(w, http.StatusText(http.StatusForbidden), http.StatusForbidden)
-		return
-	}
-
-	cfg, err := s.getPath(r, "config")
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	if err := os.Remove(cfg); err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		if os.IsNotExist(err) {
-			http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-		} else {
-			http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		}
-		return
-	}
-}
-
-const (
-	mimeTypeAPIV1 = "application/vnd.x.restic.rest.v1"
-	mimeTypeAPIV2 = "application/vnd.x.restic.rest.v2"
-)
-
-// ListBlobs lists all blobs of a given type in an arbitrary order.
-func (s *Server) ListBlobs(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("ListBlobs()")
-	}
-
-	switch r.Header.Get("Accept") {
-	case mimeTypeAPIV2:
-		s.ListBlobsV2(w, r)
-	default:
-		s.ListBlobsV1(w, r)
-	}
-}
-
-// ListBlobsV1 lists all blobs of a given type in an arbitrary order.
-func (s *Server) ListBlobsV1(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("ListBlobsV1()")
-	}
-	fileType := pat.Param(r, "type")
-	path, err := s.getPath(r, fileType)
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	items, err := ioutil.ReadDir(path)
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-		return
-	}
-
-	var names []string
-	for _, i := range items {
-		if s.isHashed(fileType) {
-			subpath := filepath.Join(path, i.Name())
-			var subitems []os.FileInfo
-			subitems, err = ioutil.ReadDir(subpath)
-			if err != nil {
-				if s.Debug {
-					log.Print(err)
-				}
-				http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-				return
-			}
-			for _, f := range subitems {
-				names = append(names, f.Name())
-			}
-		} else {
-			names = append(names, i.Name())
-		}
-	}
-
-	data, err := json.Marshal(names)
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	w.Header().Set("Content-Type", mimeTypeAPIV1)
-	_, _ = w.Write(data)
-}
-
-// Blob represents a single blob, its name and its size.
-type Blob struct {
-	Name string `json:"name"`
-	Size int64  `json:"size"`
-}
-
-// ListBlobsV2 lists all blobs of a given type, together with their sizes, in an arbitrary order.
-func (s *Server) ListBlobsV2(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("ListBlobsV2()")
-	}
-	fileType := pat.Param(r, "type")
-	path, err := s.getPath(r, fileType)
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	items, err := ioutil.ReadDir(path)
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-		return
-	}
-
-	var blobs []Blob
-	for _, i := range items {
-		if s.isHashed(fileType) {
-			subpath := filepath.Join(path, i.Name())
-			var subitems []os.FileInfo
-			subitems, err = ioutil.ReadDir(subpath)
-			if err != nil {
-				if s.Debug {
-					log.Print(err)
-				}
-				http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-				return
-			}
-			for _, f := range subitems {
-				blobs = append(blobs, Blob{Name: f.Name(), Size: f.Size()})
-			}
-		} else {
-			blobs = append(blobs, Blob{Name: i.Name(), Size: i.Size()})
-		}
-	}
-
-	data, err := json.Marshal(blobs)
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	w.Header().Set("Content-Type", mimeTypeAPIV2)
-	_, _ = w.Write(data)
-}
-
-// CheckBlob tests whether a blob exists.
-func (s *Server) CheckBlob(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("CheckBlob()")
-	}
-
-	path, err := s.getFilePath(r, pat.Param(r, "type"), pat.Param(r, "name"))
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	st, err := os.Stat(path)
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-		return
-	}
-
-	w.Header().Add("Content-Length", fmt.Sprint(st.Size()))
-}
-
-// GetBlob retrieves a blob from the repository.
-func (s *Server) GetBlob(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("GetBlob()")
-	}
-
-	path, err := s.getFilePath(r, pat.Param(r, "type"), pat.Param(r, "name"))
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	file, err := os.Open(path)
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-		return
-	}
-
-	wc := datacounter.NewResponseWriterCounter(w)
-	http.ServeContent(wc, r, "", time.Unix(0, 0), file)
-
-	if err = file.Close(); err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	if s.Prometheus {
-		labels := s.getMetricLabels(r)
-		metricBlobReadTotal.With(labels).Inc()
-		metricBlobReadBytesTotal.With(labels).Add(float64(wc.Count()))
-	}
-}
-
-// tallySize counts the size of the contents of path.
-func tallySize(path string) (int64, error) {
-	if path == "" {
-		path = "."
-	}
-	var size int64
-	err := filepath.Walk(path, func(path string, info os.FileInfo, err error) error {
-		if err != nil {
-			return err
-		}
-		size += info.Size()
-		return nil
-	})
-	return size, err
-}
-
-// SaveBlob saves a blob to the repository.
-func (s *Server) SaveBlob(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("SaveBlob()")
-	}
-
-	path, err := s.getFilePath(r, pat.Param(r, "type"), pat.Param(r, "name"))
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	tf, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_EXCL, 0600)
-	if os.IsNotExist(err) {
-		// the error is caused by a missing directory, create it and retry
-		mkdirErr := os.MkdirAll(filepath.Dir(path), 0700)
-		if mkdirErr != nil {
-			log.Print(mkdirErr)
-		} else {
-			// try again
-			tf, err = os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_EXCL, 0600)
-		}
-	}
-	if os.IsExist(err) {
-		http.Error(w, http.StatusText(http.StatusForbidden), http.StatusForbidden)
-		return
-	}
-	if err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	// ensure this blob does not put us over the repo size limit (if there is one)
-	var outFile io.Writer = tf
-	if s.MaxRepoSize != 0 {
-		var errCode int
-		outFile, errCode, err = s.maxSizeWriter(r, tf)
-		if err != nil {
-			if s.Debug {
-				log.Println(err)
-			}
-			if errCode > 0 {
-				http.Error(w, http.StatusText(errCode), errCode)
-			}
-			return
-		}
-	}
-
-	written, err := io.Copy(outFile, r.Body)
-	if err != nil {
-		_ = tf.Close()
-		_ = os.Remove(path)
-		if s.MaxRepoSize > 0 {
-			s.incrementRepoSpaceUsage(-written)
-		}
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
-		return
-	}
-
-	if err := tf.Sync(); err != nil {
-		_ = tf.Close()
-		_ = os.Remove(path)
-		if s.MaxRepoSize > 0 {
-			s.incrementRepoSpaceUsage(-written)
-		}
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	if err := tf.Close(); err != nil {
-		_ = os.Remove(path)
-		if s.MaxRepoSize > 0 {
-			s.incrementRepoSpaceUsage(-written)
-		}
-		if s.Debug {
-			log.Print(err)
-		}
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	if s.Prometheus {
-		labels := s.getMetricLabels(r)
-		metricBlobWriteTotal.With(labels).Inc()
-		metricBlobWriteBytesTotal.With(labels).Add(float64(written))
-	}
-}
-
-// DeleteBlob deletes a blob from the repository.
-func (s *Server) DeleteBlob(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("DeleteBlob()")
-	}
-
-	if s.AppendOnly && pat.Param(r, "type") != "locks" {
-		http.Error(w, http.StatusText(http.StatusForbidden), http.StatusForbidden)
-		return
-	}
-
-	path, err := s.getFilePath(r, pat.Param(r, "type"), pat.Param(r, "name"))
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	var size int64
-	if s.Prometheus || s.MaxRepoSize > 0 {
-		stat, err := os.Stat(path)
-		if err == nil {
-			size = stat.Size()
-		}
-	}
-
-	if err := os.Remove(path); err != nil {
-		if s.Debug {
-			log.Print(err)
-		}
-		if os.IsNotExist(err) {
-			http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
-		} else {
-			http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		}
-		return
-	}
-
-	if s.MaxRepoSize > 0 {
-		s.incrementRepoSpaceUsage(-size)
-	}
-	if s.Prometheus {
-		labels := s.getMetricLabels(r)
-		metricBlobDeleteTotal.With(labels).Inc()
-		metricBlobDeleteBytesTotal.With(labels).Add(float64(size))
-	}
-}
-
-// CreateRepo creates repository directories.
-func (s *Server) CreateRepo(w http.ResponseWriter, r *http.Request) {
-	if s.Debug {
-		log.Println("CreateRepo()")
-	}
-
-	repo, err := join(s.Path, s.getRepo(r))
-	if err != nil {
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	if r.URL.Query().Get("create") != "true" {
-		http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
-		return
-	}
-
-	log.Printf("Creating repository directories in %s\n", repo)
-
-	if err := os.MkdirAll(repo, 0700); err != nil {
-		log.Print(err)
-		http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-		return
-	}
-
-	for _, d := range validTypes {
-		if d == "config" {
-			continue
-		}
-
-		if err := os.MkdirAll(filepath.Join(repo, d), 0700); err != nil {
-			log.Print(err)
-			http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-			return
-		}
-	}
-
-	for i := 0; i < 256; i++ {
-		if err := os.MkdirAll(filepath.Join(repo, "data", fmt.Sprintf("%02x", i)), 0700); err != nil {
-			log.Print(err)
-			http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
-			return
-		}
+// folderPathValid checks if a folderPath returned by splitURLPath is valid and
+// safe.
+func folderPathValid(folderPath []string) bool {
+	for _, name := range folderPath {
+		if name == "" || name == ".." || name == "." || !valid(name) {
+			return false
+		}
 	}
+	return true
 }

handlers_test.go

@@ -4,37 +4,41 @@ import (
 	"bytes"
 	"crypto/rand"
 	"encoding/hex"
+	"fmt"
 	"io"
-	"io/ioutil"
 	"net/http"
 	"net/http/httptest"
 	"os"
+	"path"
 	"path/filepath"
+	"reflect"
 	"strings"
+	"sync"
 	"testing"
+
+	"github.com/minio/sha256-simd"
 )
 
 func TestJoin(t *testing.T) {
 	var tests = []struct {
-		base, name string
-		result     string
+		base   string
+		names  []string
+		result string
 	}{
-		{"/", "foo/bar", "/foo/bar"},
-		{"/srv/server", "foo/bar", "/srv/server/foo/bar"},
-		{"/srv/server", "/foo/bar", "/srv/server/foo/bar"},
-		{"/srv/server", "foo/../bar", "/srv/server/bar"},
-		{"/srv/server", "../bar", "/srv/server/bar"},
-		{"/srv/server", "..", "/srv/server"},
-		{"/srv/server", "../..", "/srv/server"},
-		{"/srv/server", "/repo/data/", "/srv/server/repo/data"},
-		{"/srv/server", "/repo/data/../..", "/srv/server"},
-		{"/srv/server", "/repo/data/../data/../../..", "/srv/server"},
-		{"/srv/server", "/repo/data/../data/../../..", "/srv/server"},
+		{"/", []string{"foo", "bar"}, "/foo/bar"},
+		{"/srv/server", []string{"foo", "bar"}, "/srv/server/foo/bar"},
+		{"/srv/server", []string{"foo", "..", "bar"}, "/srv/server/foo/bar"},
+		{"/srv/server", []string{"..", "bar"}, "/srv/server/bar"},
+		{"/srv/server", []string{".."}, "/srv/server"},
+		{"/srv/server", []string{"..", ".."}, "/srv/server"},
+		{"/srv/server", []string{"repo", "data"}, "/srv/server/repo/data"},
+		{"/srv/server", []string{"repo", "data", "..", ".."}, "/srv/server/repo/data"},
+		{"/srv/server", []string{"repo", "data", "..", "data", "..", "..", ".."}, "/srv/server/repo/data/data"},
 	}
 
 	for _, test := range tests {
 		t.Run("", func(t *testing.T) {
-			got, err := join(filepath.FromSlash(test.base), test.name)
+			got, err := join(filepath.FromSlash(test.base), test.names...)
 			if err != nil {
 				t.Fatal(err)
 			}
@@ -47,27 +51,6 @@ func TestJoin(t *testing.T) {
 	}
 }
 
-func TestIsUserPath(t *testing.T) {
-	var tests = []struct {
-		username string
-		path     string
-		result   bool
-	}{
-		{"foo", "/", false},
-		{"foo", "/foo", true},
-		{"foo", "/foo/", true},
-		{"foo", "/foo/bar", true},
-		{"foo", "/foobar", false},
-	}
-
-	for _, test := range tests {
-		result := isUserPath(test.username, test.path)
-		if result != test.result {
-			t.Errorf("isUserPath(%q, %q) was incorrect, got: %v, want: %v.", test.username, test.path, result, test.result)
-		}
-	}
-}
-
 // declare a few helper functions
 
 // wantFunc tests the HTTP response in res and calls t.Error() if something is incorrect.
@@ -85,6 +68,7 @@ func newRequest(t testing.TB, method, path string, body io.Reader) *http.Request
 // wantCode returns a function which checks that the response has the correct HTTP status code.
 func wantCode(code int) wantFunc {
 	return func(t testing.TB, res *httptest.ResponseRecorder) {
+		t.Helper()
 		if res.Code != code {
 			t.Errorf("wrong response code, want %v, got %v", code, res.Code)
 		}
@@ -94,6 +78,7 @@ func wantCode(code int) wantFunc {
 // wantBody returns a function which checks that the response has the data in the body.
 func wantBody(body string) wantFunc {
 	return func(t testing.TB, res *httptest.ResponseRecorder) {
+		t.Helper()
 		if res.Body == nil {
 			t.Errorf("body is nil, want %q", body)
 			return
@@ -107,6 +92,7 @@ func wantBody(body string) wantFunc {
 
 // checkRequest uses f to process the request and runs the checker functions on the result.
 func checkRequest(t testing.TB, f http.HandlerFunc, req *http.Request, want []wantFunc) {
+	t.Helper()
 	rr := httptest.NewRecorder()
 	f(rr, req)
 
@@ -123,117 +109,209 @@ type TestRequest struct {
 
 // createOverwriteDeleteSeq returns a sequence which will create a new file at
 // path, and then try to overwrite and delete it.
-func createOverwriteDeleteSeq(t testing.TB, path string) []TestRequest {
+func createOverwriteDeleteSeq(t testing.TB, path string, data string) []TestRequest {
 	// add a file, try to overwrite and delete it
 	req := []TestRequest{
 		{
 			req:  newRequest(t, "GET", path, nil),
 			want: []wantFunc{wantCode(http.StatusNotFound)},
 		},
-		{
-			req:  newRequest(t, "POST", path, strings.NewReader("foobar test config")),
+	}
+
+	if !strings.HasSuffix(path, "/config") {
+		req = append(req, TestRequest{
+			// broken upload must fail
+			req:  newRequest(t, "POST", path, strings.NewReader(data+"broken")),
+			want: []wantFunc{wantCode(http.StatusBadRequest)},
+		})
+	}
+
+	req = append(req,
+		TestRequest{
+			req:  newRequest(t, "POST", path, strings.NewReader(data)),
 			want: []wantFunc{wantCode(http.StatusOK)},
 		},
-		{
+		TestRequest{
 			req: newRequest(t, "GET", path, nil),
 			want: []wantFunc{
 				wantCode(http.StatusOK),
-				wantBody("foobar test config"),
+				wantBody(data),
 			},
 		},
-		{
-			req:  newRequest(t, "POST", path, strings.NewReader("other config")),
+		TestRequest{
+			req:  newRequest(t, "POST", path, strings.NewReader(data+"other stuff")),
 			want: []wantFunc{wantCode(http.StatusForbidden)},
 		},
-		{
+		TestRequest{
 			req: newRequest(t, "GET", path, nil),
 			want: []wantFunc{
 				wantCode(http.StatusOK),
-				wantBody("foobar test config"),
+				wantBody(data),
 			},
 		},
-		{
+		TestRequest{
 			req:  newRequest(t, "DELETE", path, nil),
 			want: []wantFunc{wantCode(http.StatusForbidden)},
 		},
-		{
+		TestRequest{
 			req: newRequest(t, "GET", path, nil),
 			want: []wantFunc{
 				wantCode(http.StatusOK),
-				wantBody("foobar test config"),
+				wantBody(data),
 			},
 		},
-	}
+	)
 	return req
 }
 
-// TestResticHandler runs tests on the restic handler code, especially in append-only mode.
-func TestResticHandler(t *testing.T) {
+func createTestHandler(t *testing.T, conf *Server) (http.Handler, string, string, string, func()) {
 	buf := make([]byte, 32)
 	_, err := io.ReadFull(rand.Reader, buf)
 	if err != nil {
 		t.Fatal(err)
 	}
-	randomID := hex.EncodeToString(buf)
+	data := "random data file " + hex.EncodeToString(buf)
+	dataHash := sha256.Sum256([]byte(data))
+	fileID := hex.EncodeToString(dataHash[:])
 
-	var tests = []struct {
-		seq []TestRequest
-	}{
-		{createOverwriteDeleteSeq(t, "/config")},
-		{createOverwriteDeleteSeq(t, "/data/"+randomID)},
-		{
-			// ensure we can add and remove lock files
-			[]TestRequest{
-				{
-					req:  newRequest(t, "GET", "/locks/"+randomID, nil),
-					want: []wantFunc{wantCode(http.StatusNotFound)},
-				},
-				{
-					req:  newRequest(t, "POST", "/locks/"+randomID, strings.NewReader("lock file")),
-					want: []wantFunc{wantCode(http.StatusOK)},
-				},
-				{
-					req: newRequest(t, "GET", "/locks/"+randomID, nil),
-					want: []wantFunc{
-						wantCode(http.StatusOK),
-						wantBody("lock file"),
-					},
-				},
-				{
-					req:  newRequest(t, "POST", "/locks/"+randomID, strings.NewReader("other lock file")),
-					want: []wantFunc{wantCode(http.StatusForbidden)},
-				},
-				{
-					req:  newRequest(t, "DELETE", "/locks/"+randomID, nil),
-					want: []wantFunc{wantCode(http.StatusOK)},
-				},
-				{
-					req:  newRequest(t, "GET", "/locks/"+randomID, nil),
-					want: []wantFunc{wantCode(http.StatusNotFound)},
-				},
-			},
-		},
-	}
-
-	// setup rclone with a local backend in a temporary directory
-	tempdir, err := ioutil.TempDir("", "rclone-restic-test-")
+	// setup the server with a local backend in a temporary directory
+	tempdir, err := os.MkdirTemp("", "rest-server-test-")
 	if err != nil {
 		t.Fatal(err)
 	}
 
 	// make sure the tempdir is properly removed
-	defer func() {
+	cleanup := func() {
 		err := os.RemoveAll(tempdir)
 		if err != nil {
 			t.Fatal(err)
 		}
-	}()
+	}
 
-	// set append-only mode and configure path
-	mux := NewHandler(Server{
-		AppendOnly: true,
-		Path:       tempdir,
+	conf.Path = tempdir
+	mux, err := NewHandler(conf)
+	if err != nil {
+		t.Fatalf("error from NewHandler: %v", err)
+	}
+	return mux, data, fileID, tempdir, cleanup
+}
+
+// TestResticAppendOnlyHandler runs tests on the restic handler code, especially in append-only mode.
+func TestResticAppendOnlyHandler(t *testing.T) {
+	mux, data, fileID, _, cleanup := createTestHandler(t, &Server{
+		AppendOnly:   true,
+		NoAuth:       true,
+		Debug:        true,
+		PanicOnError: true,
 	})
+	defer cleanup()
+
+	var tests = []struct {
+		seq []TestRequest
+	}{
+		{createOverwriteDeleteSeq(t, "/config", data)},
+		{createOverwriteDeleteSeq(t, "/data/"+fileID, data)},
+		{
+			// ensure we can add and remove lock files
+			[]TestRequest{
+				{
+					req:  newRequest(t, "GET", "/locks/"+fileID, nil),
+					want: []wantFunc{wantCode(http.StatusNotFound)},
+				},
+				{
+					req:  newRequest(t, "POST", "/locks/"+fileID, strings.NewReader(data+"broken")),
+					want: []wantFunc{wantCode(http.StatusBadRequest)},
+				},
+				{
+					req:  newRequest(t, "POST", "/locks/"+fileID, strings.NewReader(data)),
+					want: []wantFunc{wantCode(http.StatusOK)},
+				},
+				{
+					req: newRequest(t, "GET", "/locks/"+fileID, nil),
+					want: []wantFunc{
+						wantCode(http.StatusOK),
+						wantBody(data),
+					},
+				},
+				{
+					req:  newRequest(t, "POST", "/locks/"+fileID, strings.NewReader(data+"other data")),
+					want: []wantFunc{wantCode(http.StatusForbidden)},
+				},
+				{
+					req:  newRequest(t, "DELETE", "/locks/"+fileID, nil),
+					want: []wantFunc{wantCode(http.StatusOK)},
+				},
+				{
+					req:  newRequest(t, "GET", "/locks/"+fileID, nil),
+					want: []wantFunc{wantCode(http.StatusNotFound)},
+				},
+			},
+		},
+
+		// Test subrepos
+		{createOverwriteDeleteSeq(t, "/parent1/sub1/config", "foobar")},
+		{createOverwriteDeleteSeq(t, "/parent1/sub1/data/"+fileID, data)},
+		{createOverwriteDeleteSeq(t, "/parent1/config", "foobar")},
+		{createOverwriteDeleteSeq(t, "/parent1/data/"+fileID, data)},
+		{createOverwriteDeleteSeq(t, "/parent2/config", "foobar")},
+		{createOverwriteDeleteSeq(t, "/parent2/data/"+fileID, data)},
+	}
+
+	// create the repos
+	for _, path := range []string{"/", "/parent1/sub1/", "/parent1/", "/parent2/"} {
+		checkRequest(t, mux.ServeHTTP,
+			newRequest(t, "POST", path+"?create=true", nil),
+			[]wantFunc{wantCode(http.StatusOK)})
+	}
+
+	for _, test := range tests {
+		t.Run("", func(t *testing.T) {
+			for i, seq := range test.seq {
+				t.Logf("request %v: %v %v", i, seq.req.Method, seq.req.URL.Path)
+				checkRequest(t, mux.ServeHTTP, seq.req, seq.want)
+			}
+		})
+	}
+}
+
+// createOverwriteDeleteSeq returns a sequence which will create a new file at
+// path, and then deletes it twice.
+func createIdempotentDeleteSeq(t testing.TB, path string, data string) []TestRequest {
+	return []TestRequest{
+		{
+			req:  newRequest(t, "POST", path, strings.NewReader(data)),
+			want: []wantFunc{wantCode(http.StatusOK)},
+		},
+		{
+			req:  newRequest(t, "DELETE", path, nil),
+			want: []wantFunc{wantCode(http.StatusOK)},
+		},
+		{
+			req:  newRequest(t, "GET", path, nil),
+			want: []wantFunc{wantCode(http.StatusNotFound)},
+		},
+		{
+			req:  newRequest(t, "DELETE", path, nil),
+			want: []wantFunc{wantCode(http.StatusOK)},
+		},
+	}
+}
+
+// TestResticHandler runs tests on the restic handler code, especially in append-only mode.
+func TestResticHandler(t *testing.T) {
+	mux, data, fileID, _, cleanup := createTestHandler(t, &Server{
+		NoAuth:       true,
+		Debug:        true,
+		PanicOnError: true,
+	})
+	defer cleanup()
+
+	var tests = []struct {
+		seq []TestRequest
+	}{
+		{createIdempotentDeleteSeq(t, "/config", data)},
+		{createIdempotentDeleteSeq(t, "/data/"+fileID, data)},
+	}
 
 	// create the repo
 	checkRequest(t, mux.ServeHTTP,
@@ -249,3 +327,266 @@ func TestResticHandler(t *testing.T) {
 		})
 	}
 }
+
+// TestResticErrorHandler runs tests on the restic handler error handling.
+func TestResticErrorHandler(t *testing.T) {
+	mux, _, _, tempdir, cleanup := createTestHandler(t, &Server{
+		AppendOnly: true,
+		NoAuth:     true,
+		Debug:      true,
+	})
+	defer cleanup()
+
+	var tests = []struct {
+		seq []TestRequest
+	}{
+		// Test inaccessible file
+		{
+			[]TestRequest{{
+				req:  newRequest(t, "GET", "/config", nil),
+				want: []wantFunc{wantCode(http.StatusInternalServerError)},
+			}},
+		},
+		{
+			[]TestRequest{{
+				req:  newRequest(t, "GET", "/parent4/config", nil),
+				want: []wantFunc{wantCode(http.StatusNotFound)},
+			}},
+		},
+	}
+
+	// create the repo
+	checkRequest(t, mux.ServeHTTP,
+		newRequest(t, "POST", "/?create=true", nil),
+		[]wantFunc{wantCode(http.StatusOK)})
+	// create inaccessible config
+	checkRequest(t, mux.ServeHTTP,
+		newRequest(t, "POST", "/config", strings.NewReader("example")),
+		[]wantFunc{wantCode(http.StatusOK)})
+	err := os.Chmod(path.Join(tempdir, "config"), 0o000)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for _, test := range tests {
+		t.Run("", func(t *testing.T) {
+			for i, seq := range test.seq {
+				t.Logf("request %v: %v %v", i, seq.req.Method, seq.req.URL.Path)
+				checkRequest(t, mux.ServeHTTP, seq.req, seq.want)
+			}
+		})
+	}
+}
+
+func TestEmptyList(t *testing.T) {
+	mux, _, _, _, cleanup := createTestHandler(t, &Server{
+		AppendOnly: true,
+		NoAuth:     true,
+		Debug:      true,
+	})
+	defer cleanup()
+
+	// create the repo
+	checkRequest(t, mux.ServeHTTP,
+		newRequest(t, "POST", "/?create=true", nil),
+		[]wantFunc{wantCode(http.StatusOK)})
+
+	for i := 1; i <= 2; i++ {
+		req := newRequest(t, "GET", "/data/", nil)
+		if i == 2 {
+			req.Header.Set("Accept", "application/vnd.x.restic.rest.v2")
+		}
+
+		checkRequest(t, mux.ServeHTTP, req,
+			[]wantFunc{wantCode(http.StatusOK), wantBody("[]")})
+	}
+}
+
+func TestListWithUnexpectedFiles(t *testing.T) {
+	mux, _, _, tempdir, cleanup := createTestHandler(t, &Server{
+		AppendOnly: true,
+		NoAuth:     true,
+		Debug:      true,
+	})
+	defer cleanup()
+
+	// create the repo
+	checkRequest(t, mux.ServeHTTP,
+		newRequest(t, "POST", "/?create=true", nil),
+		[]wantFunc{wantCode(http.StatusOK)})
+	err := os.WriteFile(path.Join(tempdir, "data", "temp"), []byte{}, 0o666)
+	if err != nil {
+		t.Fatalf("creating unexpected file failed: %v", err)
+	}
+
+	for i := 1; i <= 2; i++ {
+		req := newRequest(t, "GET", "/data/", nil)
+		if i == 2 {
+			req.Header.Set("Accept", "application/vnd.x.restic.rest.v2")
+		}
+
+		checkRequest(t, mux.ServeHTTP, req,
+			[]wantFunc{wantCode(http.StatusOK)})
+	}
+}
+
+func TestSplitURLPath(t *testing.T) {
+	var tests = []struct {
+		// Params
+		urlPath  string
+		maxDepth int
+		// Expected result
+		folderPath []string
+		remainder  string
+	}{
+		{"/", 0, nil, "/"},
+		{"/", 2, nil, "/"},
+		{"/foo/bar/locks/0123", 0, nil, "/foo/bar/locks/0123"},
+		{"/foo/bar/locks/0123", 1, []string{"foo"}, "/bar/locks/0123"},
+		{"/foo/bar/locks/0123", 2, []string{"foo", "bar"}, "/locks/0123"},
+		{"/foo/bar/locks/0123", 3, []string{"foo", "bar"}, "/locks/0123"},
+		{"/foo/bar/zzz/locks/0123", 2, []string{"foo", "bar"}, "/zzz/locks/0123"},
+		{"/foo/bar/zzz/locks/0123", 3, []string{"foo", "bar", "zzz"}, "/locks/0123"},
+		{"/foo/bar/locks/", 2, []string{"foo", "bar"}, "/locks/"},
+		{"/foo/locks/", 2, []string{"foo"}, "/locks/"},
+		{"/foo/data/", 2, []string{"foo"}, "/data/"},
+		{"/foo/index/", 2, []string{"foo"}, "/index/"},
+		{"/foo/keys/", 2, []string{"foo"}, "/keys/"},
+		{"/foo/snapshots/", 2, []string{"foo"}, "/snapshots/"},
+		{"/foo/config", 2, []string{"foo"}, "/config"},
+		{"/foo/", 2, []string{"foo"}, "/"},
+		{"/foo/bar/", 2, []string{"foo", "bar"}, "/"},
+		{"/foo/bar", 2, []string{"foo"}, "/bar"},
+		{"/locks/", 2, nil, "/locks/"},
+		// This function only splits, it does not check the path components!
+		{"/././locks/", 2, []string{".", "."}, "/locks/"},
+		{"/../../locks/", 2, []string{"..", ".."}, "/locks/"},
+		{"///locks/", 2, []string{"", ""}, "/locks/"},
+		{"////locks/", 2, []string{"", ""}, "//locks/"},
+		// Robustness against broken input
+		{"/", -42, nil, "/"},
+		{"foo", 2, nil, "foo"},
+		{"foo/bar", 2, nil, "foo/bar"},
+		{"", 2, nil, ""},
+	}
+
+	for i, test := range tests {
+		t.Run(fmt.Sprintf("test-%d", i), func(t *testing.T) {
+			folderPath, remainder := splitURLPath(test.urlPath, test.maxDepth)
+
+			var fpEqual bool
+			if len(test.folderPath) == 0 && len(folderPath) == 0 {
+				fpEqual = true // this check allows for nil vs empty slice
+			} else {
+				fpEqual = reflect.DeepEqual(test.folderPath, folderPath)
+			}
+			if !fpEqual {
+				t.Errorf("wrong folderPath: want %v, got %v", test.folderPath, folderPath)
+			}
+
+			if test.remainder != remainder {
+				t.Errorf("wrong remainder: want %v, got %v", test.remainder, remainder)
+			}
+		})
+	}
+}
+
+// delayErrorReader blocks until Continue is closed, closes the channel FirstRead and then returns Err.
+type delayErrorReader struct {
+	FirstRead     chan struct{}
+	firstReadOnce sync.Once
+
+	Err error
+
+	Continue chan struct{}
+}
+
+func newDelayedErrorReader(err error) *delayErrorReader {
+	return &delayErrorReader{
+		Err:       err,
+		Continue:  make(chan struct{}),
+		FirstRead: make(chan struct{}),
+	}
+}
+
+func (d *delayErrorReader) Read(_ []byte) (int, error) {
+	d.firstReadOnce.Do(func() {
+		// close the channel to signal that the first read has happened
+		close(d.FirstRead)
+	})
+	<-d.Continue
+	return 0, d.Err
+}
+
+// TestAbortedRequest runs tests with concurrent upload requests for the same file.
+func TestAbortedRequest(t *testing.T) {
+	// the race condition doesn't happen for append-only repositories
+	mux, _, _, _, cleanup := createTestHandler(t, &Server{
+		NoAuth:       true,
+		Debug:        true,
+		PanicOnError: true,
+	})
+	defer cleanup()
+
+	// create the repo
+	checkRequest(t, mux.ServeHTTP,
+		newRequest(t, "POST", "/?create=true", nil),
+		[]wantFunc{wantCode(http.StatusOK)})
+
+	var (
+		id = "b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c"
+		wg sync.WaitGroup
+	)
+
+	// the first request is an upload to a file which blocks while reading the
+	// body and then after some data returns an error
+	rd := newDelayedErrorReader(io.ErrUnexpectedEOF)
+
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+
+		// first, read some string, then read from rd (which blocks and then
+		// returns an error)
+		dataReader := io.MultiReader(strings.NewReader("invalid data from aborted request\n"), rd)
+
+		t.Logf("start first upload")
+		req := newRequest(t, "POST", "/data/"+id, dataReader)
+		rr := httptest.NewRecorder()
+		mux.ServeHTTP(rr, req)
+		t.Logf("first upload done, response %v (%v)", rr.Code, rr.Result().Status)
+	}()
+
+	// wait until the first request starts reading from the body
+	<-rd.FirstRead
+
+	// then while the first request is blocked we send a second request to
+	// delete the file and a third request to upload to the file again, only
+	// then the first request is unblocked.
+
+	t.Logf("delete file")
+	checkRequest(t, mux.ServeHTTP,
+		newRequest(t, "DELETE", "/data/"+id, nil),
+		nil) // don't check anything, restic also ignores errors here
+
+	t.Logf("upload again")
+	checkRequest(t, mux.ServeHTTP,
+		newRequest(t, "POST", "/data/"+id, strings.NewReader("foo\n")),
+		[]wantFunc{wantCode(http.StatusOK)})
+
+	// unblock the reader for the first request now so it can continue
+	close(rd.Continue)
+
+	// wait for the first request to continue
+	wg.Wait()
+
+	// request the file again, it must exist and contain the string from the
+	// second request
+	checkRequest(t, mux.ServeHTTP,
+		newRequest(t, "GET", "/data/"+id, nil),
+		[]wantFunc{
+			wantCode(http.StatusOK),
+			wantBody("foo\n"),
+		},
+	)
+}

htpasswd.go

@@ -26,6 +26,8 @@ THE SOFTWARE.
 
 import (
 	"crypto/sha1"
+	"crypto/sha256"
+	"crypto/subtle"
 	"encoding/base64"
 	"encoding/csv"
 	"log"
@@ -42,15 +44,26 @@ import (
 // CheckInterval represents how often we check for changes in htpasswd file.
 const CheckInterval = 30 * time.Second
 
+// PasswordCacheDuration represents how long authentication credentials are
+// cached in memory after they were successfully verified. This allows avoiding
+// repeatedly verifying the same authentication credentials.
+const PasswordCacheDuration = time.Minute
+
 // Lookup passwords in a htpasswd file.  The entries must have been created with -s for SHA encryption.
 
+type cacheEntry struct {
+	expiry   time.Time
+	verifier []byte
+}
+
 // HtpasswdFile is a map for usernames to passwords.
 type HtpasswdFile struct {
 	mutex    sync.Mutex
 	path     string
 	stat     os.FileInfo
 	throttle chan struct{}
-	Users    map[string]string
+	users    map[string]string
+	cache    map[string]cacheEntry
 }
 
 // NewHtpasswdFromFile reads the users and passwords from a htpasswd file and returns them.  If an error is encountered,
@@ -68,6 +81,7 @@ func NewHtpasswdFromFile(path string) (*HtpasswdFile, error) {
 		path:     path,
 		stat:     stat,
 		throttle: make(chan struct{}),
+		cache:    make(map[string]cacheEntry),
 	}
 
 	if err := h.Reload(); err != nil {
@@ -76,6 +90,7 @@ func NewHtpasswdFromFile(path string) (*HtpasswdFile, error) {
 
 	// Start a goroutine that limits reload checks to once per CheckInterval
 	go h.throttleTimer()
+	go h.expiryTimer()
 
 	go func() {
 		for range c {
@@ -100,6 +115,25 @@ func (h *HtpasswdFile) throttleTimer() {
 	}
 }
 
+func (h *HtpasswdFile) expiryTimer() {
+	for {
+		time.Sleep(5 * time.Second)
+		now := time.Now()
+		h.mutex.Lock()
+		var zeros [sha256.Size]byte
+		// try to wipe expired cache entries
+		for user, entry := range h.cache {
+			if entry.expiry.After(now) {
+				copy(entry.verifier, zeros[:])
+				delete(h.cache, user)
+			}
+		}
+		h.mutex.Unlock()
+	}
+}
+
+var validUsernameRegexp = regexp.MustCompile(`^[\p{L}\d@._-]+$`)
+
 // Reload reloads the htpasswd file. If the reload fails, the Users map is not changed and the error is returned.
 func (h *HtpasswdFile) Reload() error {
 	r, err := os.Open(h.path)
@@ -119,12 +153,23 @@ func (h *HtpasswdFile) Reload() error {
 	}
 	users := make(map[string]string)
 	for _, record := range records {
+		if !validUsernameRegexp.MatchString(record[0]) {
+			log.Printf("Ignoring invalid username %q in htpasswd, consists of characters other than letters, numbers, '_', '-', '.' and '@'", record[0])
+			continue
+		}
 		users[record[0]] = record[1]
 	}
 
 	// Replace the Users map
 	h.mutex.Lock()
-	h.Users = users
+	var zeros [sha256.Size]byte
+	// try to wipe the old cache entries
+	for _, entry := range h.cache {
+		copy(entry.verifier, zeros[:])
+	}
+	h.cache = make(map[string]cacheEntry)
+
+	h.users = users
 	h.mutex.Unlock()
 
 	_ = r.Close()
@@ -166,35 +211,74 @@ func (h *HtpasswdFile) ReloadCheck() error {
 	return nil
 }
 
+var shaRe = regexp.MustCompile(`^{SHA}`)
+var bcrRe = regexp.MustCompile(`^\$2b\$|^\$2a\$|^\$2y\$`)
+
 // Validate returns true if password matches the stored password for user.  If no password for user is stored, or the
 // password is wrong, false is returned.
 func (h *HtpasswdFile) Validate(user string, password string) bool {
 	_ = h.ReloadCheck()
 
+	hash := sha256.New()
+	// hash.Write can never fail
+	_, _ = hash.Write([]byte(user))
+	_, _ = hash.Write([]byte(":"))
+	_, _ = hash.Write([]byte(password))
+
 	h.mutex.Lock()
-	realPassword, exists := h.Users[user]
+	// avoid race conditions with cache replacements
+	cache := h.cache
+	hashedPassword, exists := h.users[user]
+	entry, cacheExists := h.cache[user]
 	h.mutex.Unlock()
 
 	if !exists {
 		return false
 	}
 
-	var shaRe = regexp.MustCompile(`^{SHA}`)
-	var bcrRe = regexp.MustCompile(`^\$2b\$|^\$2a\$|^\$2y\$`)
+	if cacheExists && subtle.ConstantTimeCompare(entry.verifier, hash.Sum(nil)) == 1 {
+		h.mutex.Lock()
+		// repurpose mutex to prevent concurrent cache updates
+		// extend cache entry
+		cache[user] = cacheEntry{
+			verifier: entry.verifier,
+			expiry:   time.Now().Add(PasswordCacheDuration),
+		}
+		h.mutex.Unlock()
+		return true
+	}
 
+	isValid := isMatchingHashAndPassword(hashedPassword, password)
+
+	if !isValid {
+		log.Printf("Invalid htpasswd entry for %s.", user)
+		return false
+	}
+
+	h.mutex.Lock()
+	// repurpose mutex to prevent concurrent cache updates
+	cache[user] = cacheEntry{
+		verifier: hash.Sum(nil),
+		expiry:   time.Now().Add(PasswordCacheDuration),
+	}
+	h.mutex.Unlock()
+
+	return true
+}
+
+func isMatchingHashAndPassword(hashedPassword string, password string) bool {
 	switch {
-	case shaRe.MatchString(realPassword):
+	case shaRe.MatchString(hashedPassword):
 		d := sha1.New()
 		_, _ = d.Write([]byte(password))
-		if realPassword[5:] == base64.StdEncoding.EncodeToString(d.Sum(nil)) {
+		if subtle.ConstantTimeCompare([]byte(hashedPassword[5:]), []byte(base64.StdEncoding.EncodeToString(d.Sum(nil)))) == 1 {
 			return true
 		}
-	case bcrRe.MatchString(realPassword):
-		err := bcrypt.CompareHashAndPassword([]byte(realPassword), []byte(password))
+	case bcrRe.MatchString(hashedPassword):
+		err := bcrypt.CompareHashAndPassword([]byte(hashedPassword), []byte(password))
 		if err == nil {
 			return true
 		}
 	}
-	log.Printf("Invalid htpasswd entry for %s.", user)
 	return false
 }

htpasswd_test.go

@@ -0,0 +1,45 @@
+package restserver
+
+import (
+	"os"
+	"testing"
+)
+
+func TestValidate(t *testing.T) {
+	user := "restic"
+	pwd := "$2y$05$z/OEmNQamd6m6LSegUErh.r/Owk9Xwmc5lxDheIuHY2Z7XiS6FtJm"
+	rawPwd := "test"
+	wrongPwd := "wrong"
+
+	tmpfile, err := os.CreateTemp("", "rest-validate-")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err = tmpfile.Write([]byte(user + ":" + pwd + "\n")); err != nil {
+		t.Fatal(err)
+	}
+	if err = tmpfile.Close(); err != nil {
+		t.Fatal(err)
+	}
+
+	htpass, err := NewHtpasswdFromFile(tmpfile.Name())
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for i := 0; i < 10; i++ {
+		isValid := htpass.Validate(user, rawPwd)
+		if !isValid {
+			t.Fatal("correct password not accepted")
+		}
+
+		isValid = htpass.Validate(user, wrongPwd)
+		if isValid {
+			t.Fatal("wrong password accepted")
+		}
+	}
+
+	if err = os.Remove(tmpfile.Name()); err != nil {
+		t.Fatal(err)
+	}
+}

maxsizewriter.go

@@ -1,80 +0,0 @@
-package restserver
-
-import (
-	"fmt"
-	"io"
-	"net/http"
-	"strconv"
-	"sync/atomic"
-)
-
-// maxSizeWriter limits the number of bytes written
-// to the space that is currently available as given by
-// the server's MaxRepoSize. This type is safe for use
-// by multiple goroutines sharing the same *Server.
-type maxSizeWriter struct {
-	io.Writer
-	server *Server
-}
-
-func (w maxSizeWriter) Write(p []byte) (n int, err error) {
-	if int64(len(p)) > w.server.repoSpaceRemaining() {
-		return 0, fmt.Errorf("repository has reached maximum size (%d bytes)", w.server.MaxRepoSize)
-	}
-	n, err = w.Writer.Write(p)
-	w.server.incrementRepoSpaceUsage(int64(n))
-	return n, err
-}
-
-// maxSizeWriter wraps w in a writer that enforces s.MaxRepoSize.
-// If there is an error, a status code and the error are returned.
-func (s *Server) maxSizeWriter(req *http.Request, w io.Writer) (io.Writer, int, error) {
-	// if we haven't yet computed the size of the repo, do so now
-	currentSize := atomic.LoadInt64(&s.repoSize)
-	if currentSize == 0 {
-		initialSize, err := tallySize(s.Path)
-		if err != nil {
-			return nil, http.StatusInternalServerError, err
-		}
-		atomic.StoreInt64(&s.repoSize, initialSize)
-		currentSize = initialSize
-	}
-
-	// if content-length is set and is trustworthy, we can save some time
-	// and issue a polite error if it declares a size that's too big; since
-	// we expect the vast majority of clients will be honest, so this check
-	// can only help save time
-	if contentLenStr := req.Header.Get("Content-Length"); contentLenStr != "" {
-		contentLen, err := strconv.ParseInt(contentLenStr, 10, 64)
-		if err != nil {
-			return nil, http.StatusLengthRequired, err
-		}
-		if currentSize+contentLen > s.MaxRepoSize {
-			err := fmt.Errorf("incoming blob (%d bytes) would exceed maximum size of repository (%d bytes)",
-				contentLen, s.MaxRepoSize)
-			return nil, http.StatusRequestEntityTooLarge, err
-		}
-	}
-
-	// since we can't always trust content-length, we will wrap the writer
-	// in a custom writer that enforces the size limit during writes
-	return maxSizeWriter{Writer: w, server: s}, 0, nil
-}
-
-// repoSpaceRemaining returns how much space is available in the repo
-// according to s.MaxRepoSize. s.repoSize must already be set.
-// If there is no limit, -1 is returned.
-func (s *Server) repoSpaceRemaining() int64 {
-	if s.MaxRepoSize == 0 {
-		return -1
-	}
-	maxSize := s.MaxRepoSize
-	currentSize := atomic.LoadInt64(&s.repoSize)
-	return maxSize - currentSize
-}
-
-// incrementRepoSpaceUsage increments the current repo size (which
-// must already be initialized).
-func (s *Server) incrementRepoSpaceUsage(by int64) {
-	atomic.AddInt64(&s.repoSize, by)
-}

metrics.go

@@ -1,6 +1,11 @@
 package restserver
 
-import "github.com/prometheus/client_golang/prometheus"
+import (
+	"strings"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/restic/rest-server/repo"
+)
 
 var metricLabelList = []string{"user", "repo", "type"}
 
@@ -52,6 +57,30 @@ var metricBlobDeleteBytesTotal = prometheus.NewCounterVec(
 	metricLabelList,
 )
 
+// makeBlobMetricFunc creates a metrics callback function that increments the
+// Prometheus metrics.
+func makeBlobMetricFunc(username string, folderPath []string) repo.BlobMetricFunc {
+	var f repo.BlobMetricFunc = func(objectType string, operation repo.BlobOperation, nBytes uint64) {
+		labels := prometheus.Labels{
+			"user": username,
+			"repo": strings.Join(folderPath, "/"),
+			"type": objectType,
+		}
+		switch operation {
+		case repo.BlobRead:
+			metricBlobReadTotal.With(labels).Inc()
+			metricBlobReadBytesTotal.With(labels).Add(float64(nBytes))
+		case repo.BlobWrite:
+			metricBlobWriteTotal.With(labels).Inc()
+			metricBlobWriteBytesTotal.With(labels).Add(float64(nBytes))
+		case repo.BlobDelete:
+			metricBlobDeleteTotal.With(labels).Inc()
+			metricBlobDeleteBytesTotal.With(labels).Add(float64(nBytes))
+		}
+	}
+	return f
+}
+
 func init() {
 	// These are always initialized, but only updated if Config.Prometheus is set
 	prometheus.MustRegister(metricBlobWriteTotal)

mux.go

@@ -1,15 +1,16 @@
 package restserver
 
 import (
+	"fmt"
+	"io"
 	"log"
 	"net/http"
 	"os"
-
-	goji "goji.io"
+	"path/filepath"
 
 	"github.com/gorilla/handlers"
 	"github.com/prometheus/client_golang/prometheus/promhttp"
-	"goji.io/pat"
+	"github.com/restic/rest-server/quota"
 )
 
 func (s *Server) debugHandler(next http.Handler) http.Handler {
@@ -21,51 +22,97 @@ func (s *Server) debugHandler(next http.Handler) http.Handler {
 }
 
 func (s *Server) logHandler(next http.Handler) http.Handler {
-	accessLog, err := os.OpenFile(s.Log, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
-	if err != nil {
-		log.Fatalf("error: %v", err)
+	var accessLog io.Writer
+
+	if s.Log == "-" {
+		accessLog = os.Stdout
+	} else {
+		var err error
+		accessLog, err = os.OpenFile(s.Log, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
+		if err != nil {
+			log.Fatalf("error: %v", err)
+		}
 	}
 
 	return handlers.CombinedLoggingHandler(accessLog, next)
 }
 
-// NewHandler returns the master HTTP multiplexer/router.
-func NewHandler(server Server) *goji.Mux {
-	mux := goji.NewMux()
-
-	if server.Debug {
-		mux.Use(server.debugHandler)
+func (s *Server) checkAuth(r *http.Request) (username string, ok bool) {
+	if s.NoAuth {
+		return username, true
 	}
-
-	if server.Log != "" {
-		mux.Use(server.logHandler)
+	if s.ProxyAuthUsername != "" {
+		username = r.Header.Get(s.ProxyAuthUsername)
+		if username == "" {
+			return "", false
+		}
+	} else {
+		var password string
+		username, password, ok = r.BasicAuth()
+		if !ok || !s.htpasswdFile.Validate(username, password) {
+			return "", false
+		}
 	}
-
-	if server.Prometheus {
-		mux.Handle(pat.Get("/metrics"), promhttp.Handler())
-	}
-
-	mux.HandleFunc(pat.Head("/config"), server.CheckConfig)
-	mux.HandleFunc(pat.Head("/:repo/config"), server.CheckConfig)
-	mux.HandleFunc(pat.Get("/config"), server.GetConfig)
-	mux.HandleFunc(pat.Get("/:repo/config"), server.GetConfig)
-	mux.HandleFunc(pat.Post("/config"), server.SaveConfig)
-	mux.HandleFunc(pat.Post("/:repo/config"), server.SaveConfig)
-	mux.HandleFunc(pat.Delete("/config"), server.DeleteConfig)
-	mux.HandleFunc(pat.Delete("/:repo/config"), server.DeleteConfig)
-	mux.HandleFunc(pat.Get("/:type/"), server.ListBlobs)
-	mux.HandleFunc(pat.Get("/:repo/:type/"), server.ListBlobs)
-	mux.HandleFunc(pat.Head("/:type/:name"), server.CheckBlob)
-	mux.HandleFunc(pat.Head("/:repo/:type/:name"), server.CheckBlob)
-	mux.HandleFunc(pat.Get("/:type/:name"), server.GetBlob)
-	mux.HandleFunc(pat.Get("/:repo/:type/:name"), server.GetBlob)
-	mux.HandleFunc(pat.Post("/:type/:name"), server.SaveBlob)
-	mux.HandleFunc(pat.Post("/:repo/:type/:name"), server.SaveBlob)
-	mux.HandleFunc(pat.Delete("/:type/:name"), server.DeleteBlob)
-	mux.HandleFunc(pat.Delete("/:repo/:type/:name"), server.DeleteBlob)
-	mux.HandleFunc(pat.Post("/"), server.CreateRepo)
-	mux.HandleFunc(pat.Post("/:repo"), server.CreateRepo)
-	mux.HandleFunc(pat.Post("/:repo/"), server.CreateRepo)
-
-	return mux
+	return username, true
+}
+
+func (s *Server) wrapMetricsAuth(f http.HandlerFunc) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		username, ok := s.checkAuth(r)
+		if !ok {
+			httpDefaultError(w, http.StatusUnauthorized)
+			return
+		}
+		if s.PrivateRepos && username != "metrics" {
+			httpDefaultError(w, http.StatusUnauthorized)
+			return
+		}
+		f(w, r)
+	}
+}
+
+// NewHandler returns the master HTTP multiplexer/router.
+func NewHandler(server *Server) (http.Handler, error) {
+	if !server.NoAuth && server.ProxyAuthUsername == "" {
+		var err error
+		if server.HtpasswdPath == "" {
+			server.HtpasswdPath = filepath.Join(server.Path, ".htpasswd")
+		}
+		server.htpasswdFile, err = NewHtpasswdFromFile(server.HtpasswdPath)
+		if err != nil {
+			return nil, fmt.Errorf("cannot load %s (use --no-auth to disable): %v", server.HtpasswdPath, err)
+		}
+		log.Printf("Loaded htpasswd file %s", server.HtpasswdPath)
+	}
+
+	const GiB = 1024 * 1024 * 1024
+
+	if server.MaxRepoSize > 0 {
+		log.Printf("Initializing quota (can take a while)...")
+		qm, err := quota.New(server.Path, server.MaxRepoSize)
+		if err != nil {
+			return nil, err
+		}
+		server.quotaManager = qm
+		log.Printf("Quota initialized, currently using %.2f GiB", float64(qm.SpaceUsed())/GiB)
+	}
+
+	mux := http.NewServeMux()
+	if server.Prometheus {
+		if server.PrometheusNoAuth {
+			mux.Handle("/metrics", promhttp.Handler())
+		} else {
+			mux.HandleFunc("/metrics", server.wrapMetricsAuth(promhttp.Handler().ServeHTTP))
+		}
+	}
+	mux.Handle("/", server)
+
+	var handler http.Handler = mux
+	if server.Debug {
+		handler = server.debugHandler(handler)
+	}
+	if server.Log != "" {
+		handler = server.logHandler(handler)
+	}
+	return handler, nil
 }

mux_test.go

@@ -0,0 +1,80 @@
+package restserver
+
+import (
+	"net/http/httptest"
+	"testing"
+)
+
+func TestCheckAuth(t *testing.T) {
+	tests := []struct {
+		name           string
+		server         *Server
+		requestHeaders map[string]string
+		basicAuth      bool
+		basicUser      string
+		basicPassword  string
+		expectedUser   string
+		expectedOk     bool
+	}{
+		{
+			name: "NoAuth enabled",
+			server: &Server{
+				NoAuth: true,
+			},
+			expectedOk: true,
+		},
+		{
+			name: "Proxy Auth successful",
+			server: &Server{
+				ProxyAuthUsername: "X-Remote-User",
+			},
+			requestHeaders: map[string]string{
+				"X-Remote-User": "restic",
+			},
+			expectedUser: "restic",
+			expectedOk:   true,
+		},
+		{
+			name: "Proxy Auth empty header",
+			server: &Server{
+				ProxyAuthUsername: "X-Remote-User",
+			},
+			requestHeaders: map[string]string{
+				"X-Remote-User": "",
+			},
+			expectedOk: false,
+		},
+		{
+			name: "Proxy Auth missing header",
+			server: &Server{
+				ProxyAuthUsername: "X-Remote-User",
+			},
+			expectedOk: false,
+		},
+		{
+			name:   "Proxy Auth send but not enabled",
+			server: &Server{},
+			requestHeaders: map[string]string{
+				"X-Remote-User": "restic",
+			},
+			expectedOk: false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			req := httptest.NewRequest("GET", "/", nil)
+			for header, value := range tt.requestHeaders {
+				req.Header.Set(header, value)
+			}
+			if tt.basicAuth {
+				req.SetBasicAuth(tt.basicUser, tt.basicPassword)
+			}
+
+			username, ok := tt.server.checkAuth(req)
+			if username != tt.expectedUser || ok != tt.expectedOk {
+				t.Errorf("expected (%v, %v), got (%v, %v)", tt.expectedUser, tt.expectedOk, username, ok)
+			}
+		})
+	}
+}

quota/quota.go

@@ -0,0 +1,124 @@
+package quota
+
+import (
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strconv"
+	"sync/atomic"
+)
+
+// New creates a new quota Manager for given path.
+// It will tally the current disk usage before returning.
+func New(path string, maxSize int64) (*Manager, error) {
+	m := &Manager{
+		path:        path,
+		maxRepoSize: maxSize,
+	}
+	if err := m.updateSize(); err != nil {
+		return nil, err
+	}
+	return m, nil
+}
+
+// Manager manages the repo quota for given filesystem root path, including subrepos
+type Manager struct {
+	path        string
+	maxRepoSize int64
+	repoSize    int64 // must be accessed using sync/atomic
+}
+
+// WrapWriter limits the number of bytes written
+// to the space that is currently available as given by
+// the server's MaxRepoSize. This type is safe for use
+// by multiple goroutines sharing the same *Server.
+type maxSizeWriter struct {
+	io.Writer
+	m *Manager
+}
+
+func (w maxSizeWriter) Write(p []byte) (n int, err error) {
+	if int64(len(p)) > w.m.SpaceRemaining() {
+		return 0, fmt.Errorf("repository has reached maximum size (%d bytes)", w.m.maxRepoSize)
+	}
+	n, err = w.Writer.Write(p)
+	w.m.IncUsage(int64(n))
+	return n, err
+}
+
+func (m *Manager) updateSize() error {
+	// if we haven't yet computed the size of the repo, do so now
+	initialSize, err := tallySize(m.path)
+	if err != nil {
+		return err
+	}
+	atomic.StoreInt64(&m.repoSize, initialSize)
+	return nil
+}
+
+// WrapWriter wraps w in a writer that enforces s.MaxRepoSize.
+// If there is an error, a status code and the error are returned.
+func (m *Manager) WrapWriter(req *http.Request, w io.Writer) (io.Writer, int, error) {
+	currentSize := atomic.LoadInt64(&m.repoSize)
+
+	// if content-length is set and is trustworthy, we can save some time
+	// and issue a polite error if it declares a size that's too big; since
+	// we expect the vast majority of clients will be honest, so this check
+	// can only help save time
+	if contentLenStr := req.Header.Get("Content-Length"); contentLenStr != "" {
+		contentLen, err := strconv.ParseInt(contentLenStr, 10, 64)
+		if err != nil {
+			return nil, http.StatusLengthRequired, err
+		}
+		if currentSize+contentLen > m.maxRepoSize {
+			err := fmt.Errorf("incoming blob (%d bytes) would exceed maximum size of repository (%d bytes)",
+				contentLen, m.maxRepoSize)
+			return nil, http.StatusInsufficientStorage, err
+		}
+	}
+
+	// since we can't always trust content-length, we will wrap the writer
+	// in a custom writer that enforces the size limit during writes
+	return maxSizeWriter{Writer: w, m: m}, 0, nil
+}
+
+// SpaceRemaining returns how much space is available in the repo
+// according to s.MaxRepoSize. s.repoSize must already be set.
+// If there is no limit, -1 is returned.
+func (m *Manager) SpaceRemaining() int64 {
+	if m.maxRepoSize == 0 {
+		return -1
+	}
+	maxSize := m.maxRepoSize
+	currentSize := atomic.LoadInt64(&m.repoSize)
+	return maxSize - currentSize
+}
+
+// SpaceUsed returns how much space is used in the repo.
+func (m *Manager) SpaceUsed() int64 {
+	return atomic.LoadInt64(&m.repoSize)
+}
+
+// IncUsage increments the current repo size (which
+// must already be initialized).
+func (m *Manager) IncUsage(by int64) {
+	atomic.AddInt64(&m.repoSize, by)
+}
+
+// tallySize counts the size of the contents of path.
+func tallySize(path string) (int64, error) {
+	if path == "" {
+		path = "."
+	}
+	var size int64
+	err := filepath.Walk(path, func(_ string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+		size += info.Size()
+		return nil
+	})
+	return size, err
+}

repo/repo.go

@@ -0,0 +1,823 @@
+package repo
+
+import (
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log"
+	"math/rand"
+	"net/http"
+	"os"
+	"path/filepath"
+	"regexp"
+	"runtime"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+
+	"github.com/minio/sha256-simd"
+	"github.com/miolini/datacounter"
+	"github.com/restic/rest-server/quota"
+)
+
+// Options are options for the Handler accepted by New
+type Options struct {
+	AppendOnly     bool // if set, delete actions are not allowed
+	Debug          bool
+	NoVerifyUpload bool
+
+	// If set, we will panic when an internal server error happens. This
+	// makes it easier to debug such errors.
+	PanicOnError bool
+
+	BlobMetricFunc BlobMetricFunc
+	QuotaManager   *quota.Manager
+	FsyncWarning   *sync.Once
+
+	// If set makes files group accessible
+	GroupAccessible bool
+
+	// Defaults dir and file mode
+	dirMode  os.FileMode
+	fileMode os.FileMode
+}
+
+// DefaultDirMode is the file mode used for directory creation if not
+// overridden in the Options
+const DefaultDirMode os.FileMode = 0700
+
+// DefaultFileMode is the file mode used for file creation if not
+// overridden in the Options
+const DefaultFileMode os.FileMode = 0600
+
+// GroupAccessibleDirMode is the file mode used for directory creation when
+// group access is enabled
+const GroupAccessibleDirMode os.FileMode = 0770
+
+// GroupAccessibleFileMode is the file mode used for file creation when
+// group access is enabled
+const GroupAccessibleFileMode os.FileMode = 0660
+
+// New creates a new Handler for a single Restic backup repo.
+// path is the full filesystem path to this repo directory.
+// opt is a set of options.
+func New(path string, opt Options) (*Handler, error) {
+	if path == "" {
+		return nil, fmt.Errorf("path is required")
+	}
+
+	opt.dirMode = DefaultDirMode
+	opt.fileMode = DefaultFileMode
+
+	if opt.GroupAccessible {
+		opt.dirMode = GroupAccessibleDirMode
+		opt.fileMode = GroupAccessibleFileMode
+	}
+
+	h := Handler{
+		path: path,
+		opt:  opt,
+	}
+	return &h, nil
+}
+
+// Handler handles all REST API requests for a single Restic backup repo
+// Spec: https://restic.readthedocs.io/en/latest/100_references.html#rest-backend
+type Handler struct {
+	path string // filesystem path of repo
+	opt  Options
+}
+
+// httpDefaultError write a HTTP error with the default description
+func httpDefaultError(w http.ResponseWriter, code int) {
+	http.Error(w, http.StatusText(code), code)
+}
+
+// httpMethodNotAllowed writes a 405 Method Not Allowed HTTP error with
+// the required Allow header listing the methods that are allowed.
+func httpMethodNotAllowed(w http.ResponseWriter, allowed []string) {
+	w.Header().Set("Allow", strings.Join(allowed, ", "))
+	httpDefaultError(w, http.StatusMethodNotAllowed)
+}
+
+// errFileContentDoesntMatchHash is the error raised when the file content hash
+// doesn't match the hash provided in the URL
+var errFileContentDoesntMatchHash = errors.New("file content does not match hash")
+
+// BlobPathRE matches valid blob URI paths with optional object IDs
+var BlobPathRE = regexp.MustCompile(`^/(data|index|keys|locks|snapshots)/([0-9a-f]{64})?$`)
+
+// ObjectTypes are subdirs that are used for object storage
+var ObjectTypes = []string{"data", "index", "keys", "locks", "snapshots"}
+
+// FileTypes are files stored directly under the repo direct that are accessible
+// through a request
+var FileTypes = []string{"config"}
+
+func isHashed(objectType string) bool {
+	return objectType == "data"
+}
+
+// BlobOperation describe the current blob operation in the BlobMetricFunc callback.
+type BlobOperation byte
+
+// Define all valid operations.
+const (
+	BlobRead   = 'R' // A blob has been read
+	BlobWrite  = 'W' // A blob has been written
+	BlobDelete = 'D' // A blob has been deleted
+)
+
+// BlobMetricFunc is the callback signature for blob metrics. Such a callback
+// can be passed in the Options to keep track of various metrics.
+// objectType: one of ObjectTypes
+// operation: one of the BlobOperations above
+// nBytes: the number of bytes affected, or 0 if not relevant
+// TODO: Perhaps add http.Request for the username so that this can be cached?
+type BlobMetricFunc func(objectType string, operation BlobOperation, nBytes uint64)
+
+// ServeHTTP performs strict matching on the repo part of the URL path and
+// dispatches the request to the appropriate handler.
+func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	urlPath := r.URL.Path
+	if urlPath == "/" {
+		// TODO: add HEAD and GET
+		switch r.Method {
+		case "POST":
+			h.createRepo(w, r)
+		default:
+			httpMethodNotAllowed(w, []string{"POST"})
+		}
+		return
+	} else if urlPath == "/config" {
+		switch r.Method {
+		case "HEAD":
+			h.checkConfig(w, r)
+		case "GET":
+			h.getConfig(w, r)
+		case "POST":
+			h.saveConfig(w, r)
+		case "DELETE":
+			h.deleteConfig(w, r)
+		default:
+			httpMethodNotAllowed(w, []string{"HEAD", "GET", "POST", "DELETE"})
+		}
+		return
+	} else if objectType, objectID := h.getObject(urlPath); objectType != "" {
+		if objectID == "" {
+			// TODO: add HEAD
+			switch r.Method {
+			case "GET":
+				h.listBlobs(w, r)
+			default:
+				httpMethodNotAllowed(w, []string{"GET"})
+			}
+
+			return
+		}
+
+		switch r.Method {
+		case "HEAD":
+			h.checkBlob(w, r)
+		case "GET":
+			h.getBlob(w, r)
+		case "POST":
+			h.saveBlob(w, r)
+		case "DELETE":
+			h.deleteBlob(w, r)
+		default:
+			httpMethodNotAllowed(w, []string{"HEAD", "GET", "POST", "DELETE"})
+		}
+
+		return
+	}
+	httpDefaultError(w, http.StatusNotFound)
+}
+
+// getObject parses the URL path and returns the objectType and objectID,
+// if any. The objectID is optional.
+func (h *Handler) getObject(urlPath string) (objectType, objectID string) {
+	m := BlobPathRE.FindStringSubmatch(urlPath)
+	if len(m) == 0 {
+		return "", "" // no match
+	}
+	if len(m) == 2 || m[2] == "" {
+		return m[1], "" // no objectID
+	}
+	return m[1], m[2]
+}
+
+// getSubPath returns the path for a file or subdir in the root of the repo.
+func (h *Handler) getSubPath(name string) string {
+	return filepath.Join(h.path, name)
+}
+
+// getObjectPath returns the path for an object file in the repo.
+// The passed in objectType and objectID must be valid due to earlier validation
+func (h *Handler) getObjectPath(objectType, objectID string) string {
+	// If we hit an error, this is a programming error, because all of these
+	// must have been validated before. We still check them here as a safeguard.
+	if objectType == "" || objectID == "" {
+		panic("invalid objectType or objectID")
+	}
+	if isHashed(objectType) {
+		if len(objectID) < 2 {
+			// Should never happen, because BlobPathRE checked this
+			panic("getObjectPath: objectID shorter than 2 chars")
+		}
+		// Added another dir in between with the first two characters of the hash
+		return filepath.Join(h.path, objectType, objectID[:2], objectID)
+	}
+
+	return filepath.Join(h.path, objectType, objectID)
+}
+
+// sendMetric calls op.BlobMetricFunc if set. See its signature for details.
+func (h *Handler) sendMetric(objectType string, operation BlobOperation, nBytes uint64) {
+	if f := h.opt.BlobMetricFunc; f != nil {
+		f(objectType, operation, nBytes)
+	}
+}
+
+// needSize tells you if we need the file size for metrics of quota accounting
+func (h *Handler) needSize() bool {
+	return h.opt.BlobMetricFunc != nil || h.opt.QuotaManager != nil
+}
+
+// incrementRepoSpaceUsage increments the repo space usage if quota are enabled
+func (h *Handler) incrementRepoSpaceUsage(by int64) {
+	if h.opt.QuotaManager != nil {
+		h.opt.QuotaManager.IncUsage(by)
+	}
+}
+
+// wrapFileWriter wraps the file writer if repo quota are enabled, and returns it
+// as is if not.
+// If an error occurs, it returns both an error and the appropriate HTTP error code.
+func (h *Handler) wrapFileWriter(r *http.Request, w io.Writer) (io.Writer, int, error) {
+	if h.opt.QuotaManager == nil {
+		return w, 0, nil // unmodified
+	}
+	return h.opt.QuotaManager.WrapWriter(r, w)
+}
+
+// checkConfig checks whether a configuration exists.
+func (h *Handler) checkConfig(w http.ResponseWriter, _ *http.Request) {
+	if h.opt.Debug {
+		log.Println("checkConfig()")
+	}
+	cfg := h.getSubPath("config")
+
+	st, err := os.Stat(cfg)
+	if err != nil {
+		h.fileAccessError(w, err)
+		return
+	}
+
+	w.Header().Add("Content-Length", fmt.Sprint(st.Size()))
+}
+
+// getConfig allows for a config to be retrieved.
+func (h *Handler) getConfig(w http.ResponseWriter, _ *http.Request) {
+	if h.opt.Debug {
+		log.Println("getConfig()")
+	}
+	cfg := h.getSubPath("config")
+
+	bytes, err := os.ReadFile(cfg)
+	if err != nil {
+		h.fileAccessError(w, err)
+		return
+	}
+
+	_, _ = w.Write(bytes)
+}
+
+// saveConfig allows for a config to be saved.
+func (h *Handler) saveConfig(w http.ResponseWriter, r *http.Request) {
+	if h.opt.Debug {
+		log.Println("saveConfig()")
+	}
+	cfg := h.getSubPath("config")
+
+	f, err := os.OpenFile(cfg, os.O_CREATE|os.O_WRONLY|os.O_EXCL, h.opt.fileMode)
+	if err != nil && os.IsExist(err) {
+		if h.opt.Debug {
+			log.Print(err)
+		}
+		httpDefaultError(w, http.StatusForbidden)
+		return
+	}
+
+	_, err = io.Copy(f, r.Body)
+	if err != nil {
+		h.internalServerError(w, err)
+		return
+	}
+
+	err = f.Close()
+	if err != nil {
+		h.internalServerError(w, err)
+		return
+	}
+
+	_ = r.Body.Close()
+}
+
+// deleteConfig removes a config.
+func (h *Handler) deleteConfig(w http.ResponseWriter, _ *http.Request) {
+	if h.opt.Debug {
+		log.Println("deleteConfig()")
+	}
+
+	if h.opt.AppendOnly {
+		httpDefaultError(w, http.StatusForbidden)
+		return
+	}
+
+	cfg := h.getSubPath("config")
+
+	if err := os.Remove(cfg); err != nil {
+		// ignore not exist errors to make deleting idempotent, which is
+		// necessary to properly handle request retries
+		if !errors.Is(err, os.ErrNotExist) {
+			h.fileAccessError(w, err)
+		}
+		return
+	}
+}
+
+const (
+	mimeTypeAPIV1 = "application/vnd.x.restic.rest.v1"
+	mimeTypeAPIV2 = "application/vnd.x.restic.rest.v2"
+)
+
+// listBlobs lists all blobs of a given type in an arbitrary order.
+func (h *Handler) listBlobs(w http.ResponseWriter, r *http.Request) {
+	if h.opt.Debug {
+		log.Println("listBlobs()")
+	}
+
+	switch r.Header.Get("Accept") {
+	case mimeTypeAPIV2:
+		h.listBlobsV2(w, r)
+	default:
+		h.listBlobsV1(w, r)
+	}
+}
+
+// listBlobsV1 lists all blobs of a given type in an arbitrary order.
+// TODO: unify listBlobsV1 and listBlobsV2
+func (h *Handler) listBlobsV1(w http.ResponseWriter, r *http.Request) {
+	if h.opt.Debug {
+		log.Println("listBlobsV1()")
+	}
+	objectType, _ := h.getObject(r.URL.Path)
+	if objectType == "" {
+		h.internalServerError(w, fmt.Errorf(
+			"cannot determine object type: %s", r.URL.Path))
+		return
+	}
+	path := h.getSubPath(objectType)
+
+	items, err := os.ReadDir(path)
+	if err != nil {
+		h.fileAccessError(w, err)
+		return
+	}
+
+	names := []string{}
+	for _, i := range items {
+		if isHashed(objectType) {
+			if !i.IsDir() {
+				// ignore files in intermediate directories
+				continue
+			}
+			subpath := filepath.Join(path, i.Name())
+			var subitems []os.DirEntry
+			subitems, err = os.ReadDir(subpath)
+			if err != nil {
+				h.fileAccessError(w, err)
+				return
+			}
+			for _, f := range subitems {
+				names = append(names, f.Name())
+			}
+		} else {
+			names = append(names, i.Name())
+		}
+	}
+
+	data, err := json.Marshal(names)
+	if err != nil {
+		h.internalServerError(w, err)
+		return
+	}
+
+	w.Header().Set("Content-Type", mimeTypeAPIV1)
+	_, _ = w.Write(data)
+}
+
+// Blob represents a single blob, its name and its size.
+type Blob struct {
+	Name string `json:"name"`
+	Size int64  `json:"size"`
+}
+
+// listBlobsV2 lists all blobs of a given type, together with their sizes, in an arbitrary order.
+// TODO: unify listBlobsV1 and listBlobsV2
+func (h *Handler) listBlobsV2(w http.ResponseWriter, r *http.Request) {
+	if h.opt.Debug {
+		log.Println("listBlobsV2()")
+	}
+
+	objectType, _ := h.getObject(r.URL.Path)
+	if objectType == "" {
+		h.internalServerError(w, fmt.Errorf(
+			"cannot determine object type: %s", r.URL.Path))
+		return
+	}
+	path := h.getSubPath(objectType)
+
+	items, err := os.ReadDir(path)
+	if err != nil {
+		h.fileAccessError(w, err)
+		return
+	}
+
+	blobs := []Blob{}
+	for _, i := range items {
+		if isHashed(objectType) {
+			if !i.IsDir() {
+				// ignore files in intermediate directories
+				continue
+			}
+			subpath := filepath.Join(path, i.Name())
+			var subitems []os.DirEntry
+			subitems, err = os.ReadDir(subpath)
+			if err != nil {
+				h.fileAccessError(w, err)
+				return
+			}
+			for _, f := range subitems {
+				fi, err := f.Info()
+				if err != nil {
+					h.fileAccessError(w, err)
+					return
+				}
+				blobs = append(blobs, Blob{Name: f.Name(), Size: fi.Size()})
+			}
+		} else {
+			fi, err := i.Info()
+			if err != nil {
+				h.fileAccessError(w, err)
+				return
+			}
+			blobs = append(blobs, Blob{Name: i.Name(), Size: fi.Size()})
+		}
+	}
+
+	data, err := json.Marshal(blobs)
+	if err != nil {
+		h.internalServerError(w, err)
+		return
+	}
+
+	w.Header().Set("Content-Type", mimeTypeAPIV2)
+	_, _ = w.Write(data)
+}
+
+// checkBlob tests whether a blob exists.
+func (h *Handler) checkBlob(w http.ResponseWriter, r *http.Request) {
+	if h.opt.Debug {
+		log.Println("checkBlob()")
+	}
+
+	objectType, objectID := h.getObject(r.URL.Path)
+	if objectType == "" || objectID == "" {
+		h.internalServerError(w, fmt.Errorf(
+			"cannot determine object type or id: %s", r.URL.Path))
+		return
+	}
+	path := h.getObjectPath(objectType, objectID)
+
+	st, err := os.Stat(path)
+	if err != nil {
+		h.fileAccessError(w, err)
+		return
+	}
+
+	w.Header().Add("Content-Length", fmt.Sprint(st.Size()))
+}
+
+// getBlob retrieves a blob from the repository.
+func (h *Handler) getBlob(w http.ResponseWriter, r *http.Request) {
+	if h.opt.Debug {
+		log.Println("getBlob()")
+	}
+
+	objectType, objectID := h.getObject(r.URL.Path)
+	if objectType == "" || objectID == "" {
+		h.internalServerError(w, fmt.Errorf(
+			"cannot determine object type or id: %s", r.URL.Path))
+		return
+	}
+	path := h.getObjectPath(objectType, objectID)
+
+	file, err := os.Open(path)
+	if err != nil {
+		h.fileAccessError(w, err)
+		return
+	}
+
+	wc := datacounter.NewResponseWriterCounter(w)
+	http.ServeContent(wc, r, "", time.Unix(0, 0), file)
+
+	if err = file.Close(); err != nil {
+		h.internalServerError(w, err)
+		return
+	}
+
+	h.sendMetric(objectType, BlobRead, wc.Count())
+}
+
+// saveBlob saves a blob to the repository.
+func (h *Handler) saveBlob(w http.ResponseWriter, r *http.Request) {
+	if h.opt.Debug {
+		log.Println("saveBlob()")
+	}
+
+	objectType, objectID := h.getObject(r.URL.Path)
+	if objectType == "" || objectID == "" {
+		h.internalServerError(w, fmt.Errorf(
+			"cannot determine object type or id: %s", r.URL.Path))
+		return
+	}
+	path := h.getObjectPath(objectType, objectID)
+
+	_, err := os.Stat(path)
+	if err == nil {
+		httpDefaultError(w, http.StatusForbidden)
+		return
+	}
+	if !os.IsNotExist(err) {
+		h.internalServerError(w, err)
+		return
+	}
+
+	tmpFn := filepath.Join(filepath.Dir(path), objectID+".rest-server-temp")
+	tf, err := tempFile(tmpFn, h.opt.fileMode)
+	if os.IsNotExist(err) {
+		// the error is caused by a missing directory, create it and retry
+		mkdirErr := os.MkdirAll(filepath.Dir(path), h.opt.dirMode)
+		if mkdirErr != nil {
+			log.Print(mkdirErr)
+		} else {
+			// try again
+			tf, err = tempFile(tmpFn, h.opt.fileMode)
+		}
+	}
+	if err != nil {
+		h.internalServerError(w, err)
+		return
+	}
+
+	// ensure this blob does not put us over the quota size limit (if there is one)
+	outFile, errCode, err := h.wrapFileWriter(r, tf)
+	if err != nil {
+		if h.opt.Debug {
+			log.Println(err)
+		}
+		httpDefaultError(w, errCode)
+		return
+	}
+
+	var written int64
+
+	if h.opt.NoVerifyUpload {
+		// just write the file without checking the contents
+		written, err = io.Copy(outFile, r.Body)
+	} else {
+		// calculate hash for current request
+		hasher := sha256.New()
+		written, err = io.Copy(outFile, io.TeeReader(r.Body, hasher))
+
+		// reject if file content doesn't match file name
+		if err == nil && hex.EncodeToString(hasher.Sum(nil)) != objectID {
+			err = errFileContentDoesntMatchHash
+		}
+	}
+
+	if err != nil {
+		_ = tf.Close()
+		_ = os.Remove(tf.Name())
+		h.incrementRepoSpaceUsage(-written)
+		if h.opt.Debug {
+			log.Print(err)
+		}
+		var pathError *os.PathError
+		if errors.As(err, &pathError) && (pathError.Err == syscall.ENOSPC ||
+			pathError.Err == syscall.EDQUOT) {
+			// The error is disk-related (no space left, no quota left),
+			// notify the client using the correct HTTP status
+			httpDefaultError(w, http.StatusInsufficientStorage)
+		} else if errors.Is(err, errFileContentDoesntMatchHash) ||
+			errors.Is(err, io.ErrUnexpectedEOF) ||
+			errors.Is(err, http.ErrMissingBoundary) ||
+			errors.Is(err, http.ErrNotMultipart) {
+			// The error is connection-related, send a client-side HTTP status
+			httpDefaultError(w, http.StatusBadRequest)
+		} else {
+			// Otherwise we have a different internal error, reply with
+			// server-side HTTP status
+			h.internalServerError(w, err)
+		}
+		return
+	}
+
+	syncNotSup, err := syncFile(tf)
+	if err != nil {
+		_ = tf.Close()
+		_ = os.Remove(tf.Name())
+		h.incrementRepoSpaceUsage(-written)
+		h.internalServerError(w, err)
+		return
+	}
+
+	if err := tf.Close(); err != nil {
+		_ = os.Remove(tf.Name())
+		h.incrementRepoSpaceUsage(-written)
+		h.internalServerError(w, err)
+		return
+	}
+
+	if err := os.Rename(tf.Name(), path); err != nil {
+		_ = os.Remove(tf.Name())
+		h.incrementRepoSpaceUsage(-written)
+		h.internalServerError(w, err)
+		return
+	}
+
+	if syncNotSup {
+		h.opt.FsyncWarning.Do(func() {
+			log.Print("WARNING: fsync is not supported by the data storage. This can lead to data loss, if the system crashes or the storage is unexpectedly disconnected.")
+		})
+	} else {
+		if err := syncDir(filepath.Dir(path)); err != nil {
+			// Don't call os.Remove(path) as this is prone to race conditions with parallel upload retries
+			h.internalServerError(w, err)
+			return
+		}
+	}
+
+	h.sendMetric(objectType, BlobWrite, uint64(written))
+}
+
+// tempFile implements a custom version of os.CreateTemp which allows modifying the file permissions
+func tempFile(fn string, perm os.FileMode) (f *os.File, err error) {
+	for i := 0; i < 10; i++ {
+		name := fn + strconv.FormatInt(rand.Int63(), 10)
+		f, err = os.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_EXCL, perm)
+		if os.IsExist(err) {
+			continue
+		}
+		break
+	}
+	return
+}
+
+func syncFile(f *os.File) (bool, error) {
+	err := f.Sync()
+	// Ignore error if filesystem does not support fsync.
+	syncNotSup := err != nil && (errors.Is(err, syscall.ENOTSUP) || isMacENOTTY(err))
+	if syncNotSup {
+		err = nil
+	}
+	return syncNotSup, err
+}
+
+func syncDir(dirname string) error {
+	if runtime.GOOS == "windows" {
+		// syncing a directory is not possible on windows
+		return nil
+	}
+
+	dir, err := os.Open(dirname)
+	if err != nil {
+		return err
+	}
+	err = dir.Sync()
+	// Ignore error if filesystem does not support fsync.
+	if errors.Is(err, syscall.ENOTSUP) || errors.Is(err, syscall.ENOENT) || errors.Is(err, syscall.EINVAL) {
+		err = nil
+	}
+	if err != nil {
+		_ = dir.Close()
+		return err
+	}
+	return dir.Close()
+}
+
+// deleteBlob deletes a blob from the repository.
+func (h *Handler) deleteBlob(w http.ResponseWriter, r *http.Request) {
+	if h.opt.Debug {
+		log.Println("deleteBlob()")
+	}
+
+	objectType, objectID := h.getObject(r.URL.Path)
+	if objectType == "" || objectID == "" {
+		h.internalServerError(w, fmt.Errorf(
+			"cannot determine object type or id: %s", r.URL.Path))
+		return
+	}
+	if h.opt.AppendOnly && objectType != "locks" {
+		httpDefaultError(w, http.StatusForbidden)
+		return
+	}
+
+	path := h.getObjectPath(objectType, objectID)
+
+	var size int64
+	if h.needSize() {
+		stat, err := os.Stat(path)
+		if err == nil {
+			size = stat.Size()
+		}
+	}
+
+	if err := os.Remove(path); err != nil {
+		// ignore not exist errors to make deleting idempotent, which is
+		// necessary to properly handle request retries
+		if !errors.Is(err, os.ErrNotExist) {
+			h.fileAccessError(w, err)
+		}
+		return
+	}
+
+	h.incrementRepoSpaceUsage(-size)
+	h.sendMetric(objectType, BlobDelete, uint64(size))
+}
+
+// createRepo creates repository directories.
+func (h *Handler) createRepo(w http.ResponseWriter, r *http.Request) {
+	if h.opt.Debug {
+		log.Println("createRepo()")
+	}
+
+	if r.URL.Query().Get("create") != "true" {
+		httpDefaultError(w, http.StatusBadRequest)
+		return
+	}
+
+	log.Printf("Creating repository directories in %s\n", h.path)
+
+	if err := os.MkdirAll(h.path, h.opt.dirMode); err != nil {
+		h.internalServerError(w, err)
+		return
+	}
+
+	for _, d := range ObjectTypes {
+		if err := os.Mkdir(filepath.Join(h.path, d), h.opt.dirMode); err != nil && !os.IsExist(err) {
+			h.internalServerError(w, err)
+			return
+		}
+	}
+
+	for i := 0; i < 256; i++ {
+		dirPath := filepath.Join(h.path, "data", fmt.Sprintf("%02x", i))
+		if err := os.Mkdir(dirPath, h.opt.dirMode); err != nil && !os.IsExist(err) {
+			h.internalServerError(w, err)
+			return
+		}
+	}
+}
+
+// internalServerError is called to report an internal server error.
+// The error message will be reported in the server logs. If PanicOnError
+// is set, this will panic instead, which makes debugging easier.
+func (h *Handler) internalServerError(w http.ResponseWriter, err error) {
+	log.Printf("ERROR: %v", err)
+	if h.opt.PanicOnError {
+		panic(fmt.Sprintf("internal server error: %v", err))
+	}
+	httpDefaultError(w, http.StatusInternalServerError)
+}
+
+// internalServerError is called to report an error that occurred while
+// accessing a file. If the does not exist, the corresponding http status code
+// will be returned to the client. All other errors are passed on to
+// internalServerError
+func (h *Handler) fileAccessError(w http.ResponseWriter, err error) {
+	if h.opt.Debug {
+		log.Print(err)
+	}
+	if errors.Is(err, os.ErrNotExist) {
+		httpDefaultError(w, http.StatusNotFound)
+	} else {
+		h.internalServerError(w, err)
+	}
+}

repo/repo_unix.go

@@ -0,0 +1,19 @@
+//go:build !windows
+// +build !windows
+
+package repo
+
+import (
+	"errors"
+	"runtime"
+	"syscall"
+)
+
+// The ExFAT driver on some versions of macOS can return ENOTTY,
+// "inappropriate ioctl for device", for fsync.
+//
+// https://github.com/restic/restic/issues/4016
+// https://github.com/realm/realm-core/issues/5789
+func isMacENOTTY(err error) bool {
+	return runtime.GOOS == "darwin" && errors.Is(err, syscall.ENOTTY)
+}

repo/repo_windows.go

@@ -0,0 +1,4 @@
+package repo
+
+// Windows is not macOS.
+func isMacENOTTY(err error) bool { return false }