Commit `1740aaf153d0`

Vincent Demeester <vincent@sbr.pm>

2026-01-18 18:22:30

feat(backups): add Prometheus monitoring for restic backups

Add prometheus-restic-exporter to aion for backup monitoring: - Fix restic backup path: ~/desktop/documents → /neo/documents (rhea data) - Enable restic exporter on port 9753 with systemd hardening overrides - Configure to run as vincent for SSH access to aix repository Add Prometheus scrape job on sakhalin to collect restic metrics: - Scrape aion:9753 for backup health, snapshot counts, and timestamps - Enable monitoring dashboard via Grafana This provides visibility into backup health and enables alerting on backup failures or stale backups. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

main

1 parent 6842b13

Changed files (2)

systems

aion

extra.nix

sakhalin

extra.nix

@@ -199,7 +199,7 @@ in
       paths = [
         "/neo/pictures/photos/backups" # Immich database dumps only (~100MB, versioned)
         "/home/vincent/desktop/org" # Org files (<1GB)
-        "/home/vincent/desktop/documents" # Personal docs (~113GB)
+        "/neo/documents" # Personal docs rsynced from rhea (~113GB)
         "/var/lib/lidarr" # Lidarr database and config (~4.6GB)
         "/var/lib/audiobookshelf" # Audiobookshelf database and config (~30MB)
       ];
@@ -265,6 +265,16 @@ in
       '';
     };
 
+    # Prometheus exporter for restic backup monitoring
+    prometheus.exporters.restic = {
+      enable = true;
+      port = 9753;
+      user = "vincent"; # Must run as vincent to access SSH keys for aix
+      group = "users";
+      repository = "sftp:vincent@aix.sbr.pm:/data/backup/restic/aion";
+      passwordFile = config.age.secrets."restic-aix-password".path;
+    };
+
     music-playlist-dl = {
       enable = true; # Enable on music migration day
       user = "vincent";
@@ -363,6 +373,20 @@ in
     };
   };
 
+  # Override prometheus-restic-exporter service to disable DynamicUser
+  # This is needed so the service runs as vincent and can access SSH keys
+  systemd.services.prometheus-restic-exporter.serviceConfig = {
+    DynamicUser = lib.mkForce false;
+    User = lib.mkForce "vincent";
+    Group = lib.mkForce "users";
+    ProtectHome = lib.mkForce false; # Disable home protection to allow SSH control sockets
+    RestrictAddressFamilies = lib.mkForce [
+      "AF_UNIX"
+      "AF_INET"
+      "AF_INET6"
+    ]; # Allow all network families for SSH
+  };
+
   networking = {
     useDHCP = lib.mkDefault true;
     firewall = {
@@ -373,6 +397,7 @@ in
         8686 # Lidarr
         9000 # Node exporter
         9709 # Lidarr exportarr (prometheus)
+        9753 # Restic exporter (prometheus)
         9091 # Transmission (music torrents)
         # NFS ports
         111 # rpcbind

@@ -294,6 +294,14 @@ in
             }
           ];
         }
+        {
+          job_name = "restic";
+          static_configs = [
+            {
+              targets = [ "aion.sbr.pm:9753" ];
+            }
+          ];
+        }
       ];
     };

Commit 1740aaf153d0

Commit `1740aaf153d0`