Commit b36ba49b4b53

@@ -0,0 +1,466 @@
+# Git Post-Receive Hooks with Background Task Execution
+
+## Overview
+
+This document describes the git post-receive hook system for kerkouane, which enables background task execution with automatic notifications via ntfy.
+
+The system uses **systemd-run** to execute long-running tasks (like static site generation) in the background, with automatic success/failure notifications sent to ntfy.
+
+## Architecture
+
+### Components
+
+1. **systemd-run** - Native systemd tool for creating transient services
+2. **git-notify@.service** - Systemd service template that sends ntfy notifications
+3. **generate-gitmal.sh** - Helper script for gitmal static site generation
+4. **post-receive hook** - Git hook that triggers background tasks
+
+### Flow
+
+```
+Git Push → post-receive hook → systemd-run (background) → OnSuccess/OnFailure → git-notify@.service → ntfy
+```
+
+### Key Features
+
+- ✅ **Background execution** - Git push returns immediately, work happens asynchronously
+- ✅ **Automatic notifications** - ntfy alerts on success/failure
+- ✅ **Excellent logging** - Full logs via `journalctl -u <unit-name>`
+- ✅ **No additional dependencies** - Uses native systemd (already on NixOS)
+- ✅ **Robust process management** - systemd handles process lifecycle
+
+## Configuration
+
+### NixOS Configuration (systems/kerkouane/extra.nix)
+
+The system is configured in `/home/vincent/src/home/systems/kerkouane/extra.nix`:
+
+```nix
+# Git hook background task execution with notifications
+systemd.services."git-notify@" = {
+  description = "Git build notification for %i";
+  serviceConfig = {
+    Type = "oneshot";
+    ExecStart = pkgs.writeShellScript "git-notify" ''
+      # ... notification script ...
+    '' "%i";
+  };
+};
+```
+
+### Notification Service
+
+The `git-notify@.service` template:
+- Receives the unit name as parameter (%i)
+- Checks the service exit status
+- Parses unit name to extract job type and repository
+- Sends ntfy notification with appropriate emoji and priority
+
+**Notification format:**
+- Success: ✅ Git <job> Success: <repo>
+- Failure: ❌ Git <job> Failed: <repo> (with high priority)
+
+**Unit name format:** `git-<job>-<repo>-<timestamp>`
+- Example: `git-gitmal-myrepo-20260107-143022`
+
+## Usage
+
+### Setting Up a Repository
+
+1. **Navigate to your git repository:**
+   ```bash
+   cd /home/vincent/git/public/myrepo.git
+   ```
+
+2. **Copy the example hook:**
+   ```bash
+   cp /etc/git-hooks/post-receive.example hooks/post-receive
+   chmod +x hooks/post-receive
+   ```
+
+3. **Customize if needed** (the example works for gitmal generation out of the box)
+
+### Example Post-Receive Hook
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_PATH="$(pwd)"
+REPO_NAME=$(basename "$REPO_PATH" .git)
+TIMESTAMP=$(date +%Y%m%d-%H%M%S)
+UNIT_NAME="git-gitmal-${REPO_NAME}-${TIMESTAMP}"
+
+echo "Queuing gitmal generation for $REPO_NAME..."
+
+# Run gitmal generation in background with systemd-run
+sudo systemd-run \
+  --unit="$UNIT_NAME" \
+  --description="Gitmal generation for $REPO_NAME" \
+  --property="OnSuccess=git-notify@%n.service" \
+  --property="OnFailure=git-notify@%n.service" \
+  --property="User=vincent" \
+  --property="Group=users" \
+  --working-directory="$REPO_PATH" \
+  /etc/git-hooks/generate-gitmal.sh "$REPO_PATH"
+
+echo "✓ Gitmal generation queued as: $UNIT_NAME"
+echo "  View status: systemctl status $UNIT_NAME"
+echo "  View logs:   journalctl -u $UNIT_NAME"
+```
+
+### Monitoring Jobs
+
+**Check job status:**
+```bash
+systemctl status git-gitmal-myrepo-20260107-143022
+```
+
+**View job logs:**
+```bash
+journalctl -u git-gitmal-myrepo-20260107-143022
+```
+
+**List all git build jobs:**
+```bash
+systemctl list-units 'git-*'
+```
+
+**Follow logs in real-time:**
+```bash
+journalctl -f -u git-gitmal-myrepo-20260107-143022
+```
+
+## Extending the System
+
+### Creating Custom Build Scripts
+
+You can create additional scripts in `/etc/git-hooks/` for different tasks:
+
+**Example: Build forwarding script**
+
+Add to `systems/kerkouane/extra.nix`:
+
+```nix
+environment.etc."git-hooks/forward-build.sh" = {
+  text = ''
+    #!/usr/bin/env bash
+    set -euo pipefail
+
+    REPO_PATH="$1"
+    TARGET_HOST="$2"
+
+    echo "Forwarding build to $TARGET_HOST..."
+    ssh "$TARGET_HOST" "cd /path/to/repo && git pull && make build"
+    echo "Build forwarding complete"
+  '';
+  mode = "0755";
+};
+```
+
+**Use in post-receive hook:**
+
+```bash
+UNIT_NAME="git-build-${REPO_NAME}-${TIMESTAMP}"
+
+sudo systemd-run \
+  --unit="$UNIT_NAME" \
+  --description="Build forwarding for $REPO_NAME" \
+  --property="OnSuccess=git-notify@%n.service" \
+  --property="OnFailure=git-notify@%n.service" \
+  --property="User=vincent" \
+  --property="Group=users" \
+  /etc/git-hooks/forward-build.sh "$REPO_PATH" "aomi"
+```
+
+### Multiple Jobs in Sequence
+
+To run multiple tasks sequentially, create a wrapper script:
+
+```bash
+environment.etc."git-hooks/multi-task.sh" = {
+  text = ''
+    #!/usr/bin/env bash
+    set -euo pipefail
+
+    REPO_PATH="$1"
+
+    # Task 1: Generate gitmal
+    /etc/git-hooks/generate-gitmal.sh "$REPO_PATH"
+
+    # Task 2: Forward build
+    /etc/git-hooks/forward-build.sh "$REPO_PATH" "aomi"
+
+    echo "All tasks completed"
+  '';
+  mode = "0755";
+};
+```
+
+### Customizing Notifications
+
+Modify the `git-notify@.service` in `kerkouane/extra.nix` to customize:
+
+- **Notification content** - Add more context from job logs
+- **Notification channels** - Send to different ntfy topics
+- **Priority levels** - Adjust based on job type
+- **Additional actions** - Trigger webhooks, update dashboards, etc.
+
+**Example: Different topics per job type**
+
+```bash
+if [ "$JOB_TYPE" = "gitmal" ]; then
+  TOPIC="git-builds"
+elif [ "$JOB_TYPE" = "deploy" ]; then
+  TOPIC="deployments"
+else
+  TOPIC="git-general"
+fi
+
+curl -H "Title: ..." -d "..." "http://localhost:8111/$TOPIC"
+```
+
+## Troubleshooting
+
+### Hook Doesn't Execute
+
+**Check hook permissions:**
+```bash
+ls -l /home/vincent/git/public/myrepo.git/hooks/post-receive
+```
+
+Should be executable: `-rwxr-xr-x`
+
+**Check hook output:**
+When you push, you should see:
+```
+remote: Queuing gitmal generation for myrepo...
+remote: ✓ Gitmal generation queued as: git-gitmal-myrepo-20260107-143022
+```
+
+### Job Fails
+
+**Check systemd logs:**
+```bash
+journalctl -u git-gitmal-myrepo-20260107-143022
+```
+
+**Check service status:**
+```bash
+systemctl status git-gitmal-myrepo-20260107-143022
+```
+
+**Common issues:**
+- Permission errors (check User/Group in systemd-run)
+- Missing directories (check /home/vincent/git/public/)
+- Gitmal errors (check gitmal output in logs)
+
+### Notifications Not Received
+
+**Check ntfy service:**
+```bash
+systemctl status ntfy-sh
+```
+
+**Test notification manually:**
+```bash
+curl -H "Title: Test" -d "Test message" http://localhost:8111/git-builds
+```
+
+**Check notification script:**
+```bash
+# View the notification script
+systemctl cat git-notify@.service
+```
+
+### sudo Password Prompt
+
+If the post-receive hook prompts for sudo password, configure passwordless sudo for systemd-run:
+
+Add to `systems/kerkouane/extra.nix`:
+```nix
+security.sudo.extraRules = [
+  {
+    users = [ "vincent" ];
+    commands = [
+      {
+        command = "${pkgs.systemd}/bin/systemd-run";
+        options = [ "NOPASSWD" ];
+      }
+    ];
+  }
+];
+```
+
+## Performance Considerations
+
+### Resource Management
+
+Systemd provides excellent resource control. Add limits to prevent resource exhaustion:
+
+```bash
+sudo systemd-run \
+  --unit="$UNIT_NAME" \
+  --property="CPUQuota=50%" \
+  --property="MemoryMax=512M" \
+  --property="TasksMax=100" \
+  # ... other properties ...
+```
+
+### Concurrent Jobs
+
+By default, multiple pushes can trigger multiple concurrent jobs. To limit concurrency, use systemd slices or queue systems like **nq** (see research documentation).
+
+### Log Retention
+
+Systemd automatically manages transient unit cleanup. Configure retention:
+
+```nix
+systemd.services.systemd-tmpfiles-clean.serviceConfig = {
+  # Keep transient units for 7 days
+  RuntimeMaxSec = "7d";
+};
+```
+
+## Security Considerations
+
+### Permissions
+
+- Git hooks run as the git repository owner (vincent)
+- systemd-run requires sudo for system service creation
+- Build scripts should not have elevated privileges
+- Use systemd sandboxing for untrusted code
+
+### Notification Content
+
+**Avoid sensitive information in notifications:**
+- ❌ Don't include secrets, tokens, or credentials
+- ❌ Don't include full file paths with sensitive names
+- ✅ Use generic job names and repository names
+- ✅ Link to logs for details (secured by system permissions)
+
+### Network Access
+
+The notification service sends HTTP requests to ntfy:
+- Uses localhost (127.0.0.1:8111) - no external network exposure
+- Can be configured for authentication (ntfy tokens)
+
+## Alternative Approaches Considered
+
+See [comprehensive research documentation](#) for detailed analysis of alternatives:
+
+1. **nq** - Filesystem-based job queue (simple, no daemon)
+2. **task-spooler** - Job queue with email notifications (not in nixpkgs)
+3. **GNU Parallel** - Parallel job execution (different use case)
+4. **Gitea Actions / Woodpecker CI** - Full CI/CD (overkill for simple hooks)
+5. **Simple shell patterns** - nohup/background jobs (no management)
+
+**Why systemd-run was chosen:**
+- Native to NixOS (zero dependencies)
+- Robust process management
+- Excellent logging via journalctl
+- OnSuccess/OnFailure for clean notification integration
+- Production-ready and well-documented
+
+## References
+
+- [systemd-run man page](https://www.freedesktop.org/software/systemd/man/systemd-run.html)
+- [systemd.service man page](https://www.freedesktop.org/software/systemd/man/systemd.service.html)
+- [Git Hooks Documentation](https://git-scm.com/docs/githooks)
+- [ntfy Documentation](https://docs.ntfy.sh/)
+- [Gitmal GitHub](https://github.com/antonmedv/gitmal)
+
+## Examples
+
+### Example 1: Simple Gitmal Generation
+
+```bash
+# In /home/vincent/git/public/myproject.git/hooks/post-receive
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_PATH="$(pwd)"
+REPO_NAME=$(basename "$REPO_PATH" .git)
+TIMESTAMP=$(date +%Y%m%d-%H%M%S)
+UNIT_NAME="git-gitmal-${REPO_NAME}-${TIMESTAMP}"
+
+sudo systemd-run \
+  --unit="$UNIT_NAME" \
+  --property="OnSuccess=git-notify@%n.service" \
+  --property="OnFailure=git-notify@%n.service" \
+  --property="User=vincent" \
+  --property="Group=users" \
+  /etc/git-hooks/generate-gitmal.sh "$REPO_PATH"
+
+echo "Gitmal generation queued"
+```
+
+### Example 2: Custom Build Script
+
+```bash
+# Create custom build script in kerkouane/extra.nix
+environment.etc."git-hooks/build-project.sh" = {
+  text = ''
+    #!/usr/bin/env bash
+    set -euo pipefail
+
+    REPO_PATH="$1"
+    cd "$REPO_PATH"
+
+    # Run tests
+    make test
+
+    # Build project
+    make build
+
+    # Deploy to staging
+    make deploy-staging
+  '';
+  mode = "0755";
+};
+
+# Use in post-receive hook
+sudo systemd-run \
+  --unit="git-build-${REPO_NAME}-${TIMESTAMP}" \
+  --property="OnSuccess=git-notify@%n.service" \
+  --property="OnFailure=git-notify@%n.service" \
+  --property="User=vincent" \
+  /etc/git-hooks/build-project.sh "$REPO_PATH"
+```
+
+### Example 3: Conditional Execution by Branch
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Read stdin from git (old-rev new-rev refname)
+while read oldrev newrev refname; do
+  BRANCH=$(echo "$refname" | sed 's|refs/heads/||')
+
+  if [ "$BRANCH" = "main" ]; then
+    # Production deployment
+    sudo systemd-run \
+      --unit="git-deploy-prod-$(date +%s)" \
+      --property="OnSuccess=git-notify@%n.service" \
+      --property="OnFailure=git-notify@%n.service" \
+      --property="User=vincent" \
+      /etc/git-hooks/deploy-production.sh "$PWD"
+  elif [ "$BRANCH" = "develop" ]; then
+    # Staging deployment
+    sudo systemd-run \
+      --unit="git-deploy-staging-$(date +%s)" \
+      --property="OnSuccess=git-notify@%n.service" \
+      --property="OnFailure=git-notify@%n.service" \
+      --property="User=vincent" \
+      /etc/git-hooks/deploy-staging.sh "$PWD"
+  else
+    echo "Branch $BRANCH - no deployment configured"
+  fi
+done
+```
+
+---
+
+**Last updated:** 2026-01-07
+**Author:** Vincent Demeester (with Claude Code assistance)

@@ -1,4 +1,5 @@
 {
+  config,
   globals,
   lib,
   libx,
@@ -53,22 +54,149 @@ in
     # ../common/services/syncthing.nix
   ];
 
+  # Age secrets
+  age.secrets."ntfy-token" = {
+    file = ../../secrets/sakhalin/ntfy-token.age;
+    mode = "400";
+    owner = "root";
+    group = "root";
+  };
+
   # TODO make it an option ? (otherwise I'll add it for all)
   users.users.vincent.linger = true;
 
   # Allow Caddy to access git repositories in vincent's home
   users.users.caddy.extraGroups = [ "users" ];
 
+  # Allow vincent to run systemd-run without password (for git hooks)
+  security.sudo.extraRules = [
+    {
+      users = [ "vincent" ];
+      commands = [
+        {
+          command = "${pkgs.systemd}/bin/systemd-run";
+          options = [ "NOPASSWD" ];
+        }
+      ];
+    }
+  ];
+
   # Install gitmal for self-hosted git web view
   environment.systemPackages = with pkgs; [
     gitmal
   ];
 
+  # Git hook background task execution with notifications
+  systemd.services."git-notify@" = {
+    description = "Git build notification for %i";
+    serviceConfig = {
+      Type = "oneshot";
+      ExecStart = "${pkgs.writeShellScript "git-notify" ''
+        #!/usr/bin/env bash
+        set -euo pipefail
+
+        UNIT_NAME="$1"
+        RESULT=$(${pkgs.systemd}/bin/systemctl show -p Result --value "$UNIT_NAME")
+        EXIT_CODE=$(${pkgs.systemd}/bin/systemctl show -p ExecMainStatus --value "$UNIT_NAME")
+
+        # Parse unit name to extract job type and repo
+        # Format: git-<job>-<repo>-<timestamp>
+        JOB_TYPE=$(echo "$UNIT_NAME" | cut -d'-' -f2)
+        REPO=$(echo "$UNIT_NAME" | cut -d'-' -f3)
+
+        if [ "$RESULT" = "success" ]; then
+          ${pkgs.curl}/bin/curl -s \
+            -H "Authorization: Bearer $(${pkgs.coreutils}/bin/tr -d '\n' < ${
+              config.age.secrets."ntfy-token".path
+            })" \
+            -H "Title: ✅ Git $JOB_TYPE Success: $REPO" \
+            -H "Tags: white_check_mark,git,$JOB_TYPE" \
+            -H "Priority: default" \
+            -d "Job $UNIT_NAME completed successfully (exit code: $EXIT_CODE)" \
+            "https://ntfy.sbr.pm/git-builds" || true
+        else
+          ${pkgs.curl}/bin/curl -s \
+            -H "Authorization: Bearer $(${pkgs.coreutils}/bin/tr -d '\n' < ${
+              config.age.secrets."ntfy-token".path
+            })" \
+            -H "Title: ❌ Git $JOB_TYPE Failed: $REPO" \
+            -H "Priority: high" \
+            -H "Tags: x,git,$JOB_TYPE,warning" \
+            -d "Job $UNIT_NAME failed (exit code: $EXIT_CODE). Check logs: journalctl -u $UNIT_NAME" \
+            "https://ntfy.sbr.pm/git-builds" || true
+        fi
+      ''} %i";
+    };
+  };
+
+  # Helper script for gitmal generation (called from post-receive hooks)
+  environment.etc."git-hooks/generate-gitmal.sh" = {
+    text = ''
+      #!${pkgs.bash}/bin/bash
+      set -euo pipefail
+
+      REPO_PATH="$1"
+      REPO_NAME=$(basename "$REPO_PATH" .git)
+      OUTPUT_DIR="/home/vincent/git/public/$REPO_NAME"
+
+      echo "Generating gitmal for repository: $REPO_NAME"
+      echo "Repository path: $REPO_PATH"
+      echo "Output directory: $OUTPUT_DIR"
+
+      # Generate static site with gitmal
+      cd "$REPO_PATH"
+      ${pkgs.gitmal}/bin/gitmal --output "$OUTPUT_DIR"
+
+      echo "Gitmal generation complete: $OUTPUT_DIR"
+    '';
+    mode = "0755";
+  };
+
+  # Example post-receive hook template
+  environment.etc."git-hooks/post-receive.example" = {
+    text = ''
+      #!${pkgs.bash}/bin/bash
+      # Example post-receive hook for git repositories
+      # Copy this to your repository's hooks/post-receive and make it executable
+      #
+      # This hook uses systemd-run to execute gitmal generation in the background
+      # with automatic notifications via ntfy when the job completes.
+
+      set -euo pipefail
+
+      REPO_PATH="$(pwd)"
+      REPO_NAME=$(basename "$REPO_PATH" .git)
+      TIMESTAMP=$(date +%Y%m%d-%H%M%S)
+      UNIT_NAME="git-gitmal-''${REPO_NAME}-''${TIMESTAMP}"
+
+      echo "Queuing gitmal generation for $REPO_NAME..."
+
+      # Run gitmal generation in background with systemd-run
+      # OnSuccess/OnFailure will trigger git-notify@.service for notifications
+      sudo ${pkgs.systemd}/bin/systemd-run \
+        --unit="$UNIT_NAME" \
+        --description="Gitmal generation for $REPO_NAME" \
+        --property="OnSuccess=git-notify@''${UNIT_NAME}.service" \
+        --property="OnFailure=git-notify@''${UNIT_NAME}.service" \
+        --property="User=vincent" \
+        --property="Group=users" \
+        --property="Environment=PATH=${pkgs.coreutils}/bin:${pkgs.git}/bin:${pkgs.gitmal}/bin" \
+        --working-directory="$REPO_PATH" \
+        /etc/git-hooks/generate-gitmal.sh "$REPO_PATH"
+
+      echo "✓ Gitmal generation queued as: $UNIT_NAME"
+      echo "  View status: systemctl status $UNIT_NAME"
+      echo "  View logs:   journalctl -u $UNIT_NAME"
+    '';
+    mode = "0755";
+  };
+
   # Setup permissions for git directories (via systemd tmpfiles)
   systemd.tmpfiles.rules = [
     "d /home/vincent 0711 vincent users -" # Allow traversal to git directory
     "d /home/vincent/git 0700 vincent users -" # Private git directory
     "d /home/vincent/git/public 0755 vincent users -" # Public repositories only
+    "d /var/log/git-builds 0755 vincent users -" # Git build logs
   ];
 
   # Disable TPM2 (VPS has no TPM hardware)

@@ -128,6 +128,7 @@ in
     sakhalin
     aion
     rhea
+    kerkouane
   ];
   "secrets/sakhalin/homeassistant-prometheus-token.age".publicKeys = users ++ [ sakhalin ];
   "secrets/demeter/mosquitto-homeassistant-password.age".publicKeys = users ++ [ demeter ];