Commit cfd3f0cfa95c

@@ -0,0 +1,255 @@
+---
+name: Homelab
+description: NixOS homelab infrastructure management for the home repository. USE WHEN working with NixOS systems, home-manager configs, networking, services, or personal infrastructure.
+---
+
+# Homelab Management Skill
+
+## Purpose
+Expert assistance managing personal NixOS infrastructure in the home repository.
+
+### Context Detection
+
+**This skill activates when:**
+- Current working directory is `/home/vincent/src/home` or subdirectories
+- User asks about specific hosts (kyushu, aomi, demeter, rhea, athena, etc.)
+- User mentions home repository infrastructure, services, or configurations
+- Files in `/systems`, `/home/common`, or `/modules` directories are referenced
+- User asks about DNS management, VPN, or homelab services
+
+## Repository Structure
+- **Repository location**: ~/src/home
+- **System configs**: /systems/<hostname>/
+- **Home configs**: /home/common/
+- **Custom packages**: /pkgs/
+- **Globals**: globals.nix (machine definitions, DNS zones, VPN)
+- **Tools**: /tools/ (custom Go tools)
+- **Modules**: /modules/ (custom NixOS modules)
+- **Keyboards**: /keyboards/ (QMK/ZMK firmware)
+- **Imperative**: /imperative/ (non-NixOS host configs)
+
+## Key Machines
+
+### Unstable Systems (nixos-unstable)
+- **kyushu**: Desktop/workstation
+- **aomi**: Laptop/portable
+- **sakhalin**: Development
+- **foobar**: Testing
+
+### Stable Systems (25.05)
+- **athena**: Production server
+- **demeter**: Infrastructure server
+- **aix**: Raspberry Pi 4
+- **aion**: Raspberry Pi 4
+- **rhea**: Raspberry Pi 4
+- **kerkouane**: Raspberry Pi 4
+
+### Desktop Types
+- **sway**: Traditional tiling window manager
+- **niri**: Modern scrollable tiling window manager
+
+## Services and Infrastructure
+
+### DNS Management
+- Managed via `globals.nix` zones
+- Update tool: `tools/update-gandi-dns.sh`
+- Provider: Gandi LiveDNS API
+- Domains: demeester.fr, sbr.pm, openshift-pipelines.org, etc.
+
+### VPN Configuration
+- Wireguard configurations in `/modules/`
+- Client module: `modules/wireguard-client`
+- Server module: `modules/wireguard-server`
+- Network topology in `globals.nix`
+
+### Container Services
+- Docker/Podman setups
+- System-manager for containerized services
+- Service definitions per host
+
+### Secrets Management
+- **agenix** for encrypted secrets
+- Yubikeys for age encryption
+- Host SSH keys for system-level decryption
+- Secrets defined in `secrets.nix`
+
+## Common Operations
+
+### Building and Deploying
+```bash
+# Build and switch current system
+make switch
+
+# Build and activate on next boot
+make boot
+
+# Test build without switching
+make dry-build
+
+# Build specific remote host
+make host/<hostname>/build
+
+# Deploy to remote host (boot) - ALWAYS ASK FIRST
+make host/<hostname>/boot
+
+# Deploy to remote host (switch) - ALWAYS ASK FIRST
+make host/<hostname>/switch
+```
+
+### Package Management
+```bash
+# Build a package
+nix build .#<package-name>
+
+# Install a package
+nix profile install .#<package-name>
+
+# List available packages
+nix flake show
+```
+
+### Maintenance
+```bash
+# Format Nix files
+make fmt
+
+# Clean old generations (>15 days) and build results
+make clean
+
+# Update flake inputs
+nix flake update
+
+# Run pre-commit checks
+make pre-commit
+```
+
+### Keyboard Firmware
+```bash
+# Build Moonlander QMK firmware
+make keyboards/moonlander/build
+
+# Build Eyelash Corne ZMK firmware
+make keyboards/eyelash_corne/build
+
+# Generate keymap visualizations
+make keyboards/draw
+make keyboards/<name>/draw
+```
+
+### DNS Operations
+```bash
+# Show current DNS records
+./tools/show-dns.sh
+
+# Update Gandi DNS (requires GANDIV5_PERSONAL_TOKEN)
+make dns-update-gandi
+```
+
+## Safety Protocols
+
+### Deployment Safety
+- **ALWAYS** ask before deploying to remote hosts
+- **ALWAYS** dry-build before deploying
+- **ALWAYS** confirm target host explicitly
+- Check service dependencies before deployment
+- Never deploy automatically
+
+### Git Operations
+- NEVER update git config without asking
+- NEVER run destructive git commands (force push, hard reset) without explicit request
+- NEVER skip hooks (--no-verify) unless explicitly requested
+- NEVER force push to main/master without warning
+
+### Secrets and Security
+- Do not commit files with secrets (.env, credentials.json)
+- Warn if attempting to commit sensitive files
+- Use agenix for all secret management
+- Verify secrets are encrypted before committing
+
+## Architecture Patterns
+
+### Host Configuration
+Each host has:
+- `boot.nix`: Boot configuration (bootloader, initrd, kernel modules)
+- `hardware.nix`: Hardware-specific settings (mounts, hardware imports)
+- `extra.nix` (optional): Additional host-specific config
+- `home.nix` (optional): Host-specific home-manager config
+
+### Adding a New Host
+1. Create `/systems/<hostname>` directory with `boot.nix` and `hardware.nix`
+2. Add host entry in `flake.nix` nixosConfigurations using `libx.mkHost`
+3. Add machine metadata to `globals.nix` (IPs, SSH keys, syncthing ID)
+4. Update `secrets.nix` with host SSH key if using secrets
+
+### Adding a New Package
+1. Create package directory in `/pkgs/<package-name>` with `default.nix`
+2. Add entry to `/pkgs/default.nix` using `pkgs.callPackage`
+3. Package available via `nix build .#<package-name>`
+
+## Pre-commit Hooks
+
+Configured in `flake.nix`:
+- **Go**: gofmt formatting
+- **Nix**: nixfmt-rfc-style formatting, deadnix linting
+- **Python**: flake8, ruff linting
+- **Shell**: shellcheck
+
+Install hooks: `make install-hooks`
+
+## Testing
+
+### Go Tools
+```bash
+cd tools/<tool-name>
+go test ./...
+```
+
+### NixOS Configurations
+```bash
+# Build test
+nixos-rebuild build --flake .#<hostname>
+
+# Dry run
+nixos-rebuild dry-build --flake .#<hostname>
+```
+
+## Troubleshooting
+
+### Build Failures
+1. Check `nix flake check` for issues
+2. Review recent changes with `git diff`
+3. Verify flake.lock is up to date
+4. Check for syntax errors with `nixfmt`
+
+### Deployment Issues
+1. Verify SSH access to target host
+2. Check network connectivity
+3. Review target host logs
+4. Ensure secrets are properly decrypted
+
+### Service Issues
+1. Check systemd status: `systemctl status <service>`
+2. Review logs: `journalctl -u <service>`
+3. Verify configuration in globals.nix
+4. Check dependencies are running
+
+## Quick Reference
+
+### File Locations
+- Machine definitions: `globals.nix`
+- Flake configuration: `flake.nix`
+- Library functions: `lib/default.nix`
+- Overlays: `overlays/default.nix`
+- Common system modules: `systems/common/`
+- Common home modules: `home/common/`
+
+### Environment Variables
+- `XDG_CONFIG_HOME`: `~/.config` (XDG base directories enabled)
+- Nix config: `~/.config/nix/nix.conf`
+
+### Binary Caches
+- vdemeester.cachix.org
+- chapeau-rouge.cachix.org
+- nixos-raspberrypi.cachix.org
+
+Remember: This is a production infrastructure. Always prioritize safety and ask for confirmation before making changes to remote systems.

@@ -0,0 +1,59 @@
+{
+  "permissions": {
+    "allow": [
+      "WebSearch",
+      "WebFetch(domain:github.com)",
+      "WebFetch(domain:typescript-sdk.jellyfin.org)",
+      "Bash(git add:*)",
+      "Bash(git commit:*)",
+      "Bash(python3 -m flake8:*)",
+      "WebFetch(domain:wiki.nixos.org)",
+      "WebFetch(domain:gist.github.com)",
+      "Skill(GitHub)",
+      "Bash(gh issue view:*)",
+      "Skill(TODOs)",
+      "Bash(claude-hooks-save-session --help)",
+      "Bash(echo \"Exit code: $?\")",
+      "Bash(~/.config/claude/skills/Org/tools/org-manager:*)",
+      "Bash(make:*)",
+      "Bash(nix search:*)",
+      "WebFetch(domain:viric.name)",
+      "WebFetch(domain:riverqueue.com)",
+      "Bash(ssh kerkouane \"mkdir -p /home/vincent/git/public && cd /home/vincent/git/public && git init --bare test-repo.git\")",
+      "Bash(ssh kerkouane.vpn \"mkdir -p /home/vincent/git/public && cd /home/vincent/git/public && git init --bare test-repo.git\")",
+      "Bash(ssh kerkouane.vpn \"cp /etc/git-hooks/post-receive.example /home/vincent/git/public/test-repo.git/hooks/post-receive && chmod +x /home/vincent/git/public/test-repo.git/hooks/post-receive\")",
+      "Bash(ssh:*)",
+      "Bash(nix run github:ryantm/agenix:*)",
+      "Bash(git push:*)",
+      "Bash(git pull:*)",
+      "Bash(done)",
+      "Bash(pkgs = nixosConfigurations.kyushu.pkgs)",
+      "Bash(builtins.functionArgs pkgs.mpv.override)",
+      "Bash(:quit)",
+      "Bash(EOF)",
+      "Bash(gh pr list:*)",
+      "Bash(gh issue list:*)",
+      "Bash(gh pr view:*)",
+      "Bash(git clone:*)",
+      "Bash(cat:*)",
+      "Bash(find:*)",
+      "Bash(nix-instantiate:*)",
+      "Bash(gh search code:*)",
+      "Bash(claude-hooks-save-session)",
+      "Bash(jobs)",
+      "Bash(nix-store:*)",
+      "Bash(/nix/store/anc66y6lsy1sdddwnia27592rnv9bimk-emacs-unstable-30.2/bin/emacs:*)",
+      "Bash(/nix/store/r14qp3lbpw6vz7pl3m454xz2f4zf1pc1-emacs-unstable-30.2/bin/emacs:*)",
+      "Bash(/nix/store/*/bin/emacs)",
+      "Bash(git fetch:*)",
+      "Bash(git rebase:*)",
+      "Bash(grep:*)",
+      "Bash(nix flake check:*)",
+      "Bash(git mv:*)",
+      "Bash(git rm:*)",
+      "Bash(nixos-version:*)"
+    ],
+    "deny": [],
+    "ask": []
+  }
+}

@@ -0,0 +1,710 @@
+# WireGuard iOS Troubleshooting Guide
+
+## Overview
+
+This guide addresses handshake failures when connecting iOS devices to a WireGuard VPN server. iOS WireGuard clients have specific requirements and quirks that differ from other platforms.
+
+## Background
+
+WireGuard uses a simple handshake protocol:
+1. Client sends handshake initiation to server
+2. Server responds with handshake response
+3. Both sides derive session keys
+4. Data packets can flow
+
+When handshakes fail, you'll see:
+- "Handshake did not complete" in iOS WireGuard app
+- No "latest handshake" timestamp on server (`wg show`)
+- Connection timeout without error details
+
+## Common iOS-Specific Causes
+
+### 1. DNS Configuration Issues (MOST COMMON)
+
+**Problem**: iOS handles DNS differently than other platforms. The WireGuard iOS app may refuse to connect or fail silently if DNS is misconfigured.
+
+**Symptoms**:
+- Connection appears to activate but no traffic flows
+- Works on some networks but not others
+- Interface goes "up" but handshake never completes
+
+**Root Causes**:
+- No DNS server specified in configuration
+- Multiple DNS servers listed (iOS sometimes rejects this)
+- DNS server is within VPN tunnel range (10.100.0.x) but unreachable during handshake
+- DNS64/NAT64 networks (some cellular carriers) conflict with WireGuard DNS
+
+**Solution**:
+```ini
+[Interface]
+DNS = 8.8.8.8
+```
+
+**Best Practices**:
+- Use a single, external DNS server (8.8.8.8, 1.1.1.1, or 9.9.9.9)
+- Avoid using multiple DNS servers
+- Don't use VPN-internal DNS during initial connection
+- For privacy, Cloudflare (1.1.1.1) or Quad9 (9.9.9.9) are good alternatives to Google
+
+### 2. MTU Size Mismatches
+
+**Problem**: iOS devices, especially on cellular networks, are sensitive to MTU (Maximum Transmission Unit) size.
+
+**Symptoms**:
+- Handshake completes but no data transfers
+- Works on WiFi but fails on cellular
+- Small packets work but large ones don't (can ping but can't browse)
+
+**Root Causes**:
+- Default WireGuard MTU is 1420 bytes
+- Cellular networks often have lower MTU (1280-1400)
+- WireGuard adds 80 bytes of overhead to IP packets
+- Path MTU discovery doesn't work well with iOS + cellular
+
+**Solution**:
+```ini
+[Interface]
+MTU = 1280
+```
+
+**Testing Different MTU Values**:
+```bash
+# On server after connection established:
+ping -M do -s 1232 10.100.0.5  # Test with 1280 MTU (1232 + 28 ICMP + 20 IP = 1280)
+ping -M do -s 1372 10.100.0.5  # Test with 1420 MTU (1372 + 28 + 20 = 1420)
+```
+
+If lower values work but higher ones fail, reduce MTU in iOS config.
+
+**Recommended Values**:
+- Start with: **1280** (works on most cellular networks)
+- If problems persist: try 1280, 1360, 1400, 1412
+- WiFi-only: can usually use 1420 (default)
+
+### 3. Cellular Network Routing Issues
+
+**Problem**: iOS on cellular data routes traffic differently than WiFi, and some carriers block or NAT UDP traffic aggressively.
+
+**Symptoms**:
+- Works perfectly on home WiFi
+- Fails immediately on cellular data
+- No traffic appears at server firewall
+- Sometimes works, then stops working
+
+**Root Causes**:
+- Carrier-grade NAT (CGNAT) blocks or mangles UDP
+- Carrier firewall blocks UDP port 51820
+- iOS routing table prioritizes cellular routes incorrectly
+- Mobile data restrictions or VPN policies
+
+**Diagnosis**:
+```bash
+# On iOS device, check if UDP packets reach server
+# Have someone run on server while you connect:
+sudo tcpdump -i any udp port 51820
+
+# If you see no packets from your IP, it's a carrier issue
+```
+
+**Solutions**:
+- Test on different cellular network if possible
+- Try alternative WireGuard port (some carriers block 51820)
+- Use TCP-based tunnel (Shadowsocks/V2Ray) if UDP is completely blocked
+- Contact carrier if VPN is required for work
+
+**Workarounds**:
+- Enable VPN only on WiFi (on-demand rules)
+- Use backup VPN provider for cellular
+- Switch to different carrier with better UDP support
+
+### 4. Server NAT/Masquerading Misconfiguration
+
+**Problem**: Server firewall rules don't properly NAT traffic from VPN clients.
+
+**Symptoms**:
+- Handshake completes (shows "latest handshake" on server)
+- Can ping server VPN IP (10.100.0.1) but nothing else
+- No internet access through VPN
+
+**Root Causes**:
+- Missing or incorrect MASQUERADE rule
+- Wrong source subnet in iptables rule
+- Missing IP forwarding (net.ipv4.ip_forward = 0)
+
+**Check Server Config**:
+```bash
+# Verify IP forwarding is enabled
+sysctl net.ipv4.ip_forward
+# Should return: net.ipv4.ip_forward = 1
+
+# Check NAT rule exists
+sudo iptables -t nat -L POSTROUTING -v -n | grep 10.100.0.0
+
+# Should see something like:
+# MASQUERADE  all  --  *  *  10.100.0.0/24  0.0.0.0/0
+```
+
+**Fix in NixOS** (modules/wireguard-server.nix):
+```nix
+networking.firewall.extraCommands = ''
+  iptables -t nat -A POSTROUTING -s 10.100.0.0/24 -j MASQUERADE
+  iptables -A FORWARD -i wg+ -j ACCEPT
+'';
+```
+
+**Common Mistakes**:
+- `-s10.100.0.0/24` (missing space after -s)
+- `-s 10.100.0.0/32` (wrong subnet mask - /32 is single IP, should be /24)
+- Using SNAT instead of MASQUERADE on dynamic IP servers
+
+### 5. Key Mismatches
+
+**Problem**: Public/private keys don't match between client and server config.
+
+**Symptoms**:
+- "Handshake did not complete" immediately
+- No handshake attempts visible on server
+- iOS app shows "Invalid key" or just fails silently
+
+**Root Causes**:
+- Typo when copying public key
+- QR code scanning error
+- Key regeneration on one side but not updated on other
+- PreSharedKey mismatch (if using PSK)
+
+**Verification**:
+
+On iOS device:
+1. Open WireGuard app
+2. Tap on tunnel configuration
+3. Note the **Public key** shown under Interface
+
+On server:
+```bash
+# Check what public key is configured for this peer
+sudo wg show wg0 peers | grep -A5 "1wzFG60hlrAoSYcRKApsH+WK3Zyz8ljdLglb/8JbuW0="
+```
+
+In your NixOS config (globals.nix):
+```nix
+hokkaido = {
+  net = {
+    vpn = {
+      pubkey = "1wzFG60hlrAoSYcRKApsH+WK3Zyz8ljdLglb/8JbuW0=";
+      ips = [ "10.100.0.5" ];
+    };
+  };
+};
+```
+
+**Solutions**:
+- Delete and recreate tunnel on iOS (generates new key pair)
+- Update globals.nix with new public key
+- Rebuild and deploy server config
+- Double-check keys match exactly (case-sensitive, watch for l vs 1, O vs 0)
+
+### 6. Server Firewall Blocking UDP
+
+**Problem**: Server firewall doesn't allow UDP port 51820.
+
+**Symptoms**:
+- Connection times out
+- No errors, just hangs
+- Works from some networks but not others (if server IP is geo-blocked)
+
+**Check Firewall**:
+```bash
+# On NixOS server
+sudo iptables -L INPUT -v -n | grep 51820
+
+# Should see:
+# ACCEPT  udp  --  *  *  0.0.0.0/0  0.0.0.0/0  udp dpt:51820
+```
+
+**Cloud Provider Firewalls**:
+
+If running on DigitalOcean/AWS/GCP, check:
+- **DigitalOcean**: Cloud Firewalls in dashboard
+- **AWS**: Security Groups must allow UDP 51820 inbound
+- **GCP**: VPC Firewall Rules must allow UDP 51820
+
+**Fix in NixOS**:
+```nix
+networking.firewall.allowedUDPPorts = [ 51820 ];
+```
+
+### 7. iOS App-Specific Bugs
+
+**Problem**: Bugs in specific versions of WireGuard iOS app.
+
+**Symptoms**:
+- Works on other devices but not iOS
+- Works on older iOS version but not after update
+- Inconsistent behavior
+
+**Solutions**:
+- Update WireGuard iOS app to latest version
+- Check WireGuard iOS GitHub issues for known bugs
+- Try toggling "On-Demand" activation off and on
+- Force-quit and restart WireGuard app
+- Reboot iOS device (clears network stack caches)
+- Delete and recreate tunnel configuration
+
+## Recommended iOS Client Configuration
+
+For device `hokkaido` (10.100.0.5) connecting to server `kerkouane`:
+
+```ini
+[Interface]
+PrivateKey = <GENERATE_ON_IOS_DEVICE>
+Address = 10.100.0.5/24
+DNS = 8.8.8.8
+MTU = 1280
+
+[Peer]
+PublicKey = +H3fxErP9HoFUrPgU19ra9+GDLQw+VwvLWx3lMct7QI=
+Endpoint = 167.99.17.238:51820
+AllowedIPs = 10.100.0.0/24
+PersistentKeepalive = 25
+```
+
+**Field Explanations**:
+- **PrivateKey**: Generated automatically by iOS app (never share this)
+- **Address**: Your VPN IP from globals.nix
+- **DNS**: Single external DNS server (Google, Cloudflare, or Quad9)
+- **MTU**: Reduced for cellular compatibility
+- **PublicKey**: Server's public key from globals.nix (kerkouane)
+- **Endpoint**: Server's public IP and WireGuard port
+- **AllowedIPs**: VPN subnet to route through tunnel
+- **PersistentKeepalive**: Keep NAT mappings alive (important for mobile)
+
+**For Full Tunnel (Route all traffic through VPN)**:
+```ini
+AllowedIPs = 0.0.0.0/0, ::/0
+```
+
+**For Split Tunnel (Only VPN subnet)**:
+```ini
+AllowedIPs = 10.100.0.0/24
+```
+
+## Step-by-Step Troubleshooting
+
+### Step 1: Verify Server Configuration
+
+```bash
+# SSH to kerkouane
+ssh kerkouane
+
+# Check WireGuard interface is up
+sudo wg show
+
+# Should see:
+# interface: wg0
+#   public key: +H3fxErP9HoFUrPgU19ra9+GDLQw+VwvLWx3lMct7QI=
+#   private key: (hidden)
+#   listening port: 51820
+
+# Check peer is configured
+sudo wg show wg0 peers
+
+# Should include:
+# peer: 1wzFG60hlrAoSYcRKApsH+WK3Zyz8ljdLglb/8JbuW0=
+#   allowed ips: 10.100.0.5/32
+
+# Check firewall allows UDP 51820
+sudo iptables -L INPUT -v -n | grep 51820
+
+# Check NAT rule
+sudo iptables -t nat -L POSTROUTING -v -n | grep 10.100.0
+```
+
+### Step 2: Test Server Connectivity
+
+```bash
+# From any other machine (not iOS yet)
+nc -u -v 167.99.17.238 51820
+
+# Or test with another WireGuard client if available
+```
+
+### Step 3: Configure iOS Client
+
+1. Open WireGuard app on iOS
+2. Tap "Add a tunnel" → "Create from scratch"
+3. Set name: `kerkouane-vpn`
+4. Copy **Public key** shown (you'll need this for server config)
+5. Tap "Add peer"
+6. Enter server configuration
+7. Save tunnel
+
+### Step 4: Update Server with iOS Public Key
+
+```nix
+# In globals.nix, update hokkaido's pubkey if changed
+hokkaido = {
+  net = {
+    vpn = {
+      pubkey = "PASTE_PUBLIC_KEY_FROM_IOS_APP_HERE";
+      ips = [ "10.100.0.5" ];
+    };
+  };
+};
+```
+
+```bash
+# Rebuild server config
+make host/kerkouane/switch
+```
+
+### Step 5: Attempt Connection from iOS
+
+1. Activate tunnel in WireGuard app
+2. Watch for status change
+3. If it shows "Connected", check connection
+
+```bash
+# On iOS, open Safari and try:
+http://10.100.0.1  # Server VPN IP
+http://whoami.sbr.pm  # Public service through VPN
+```
+
+### Step 6: Monitor Server Logs
+
+```bash
+# On server, watch for handshake attempts
+sudo journalctl -u systemd-networkd -f
+
+# Or watch WireGuard interface
+watch -n 1 sudo wg show
+
+# Look for:
+# peer: 1wzFG60hlrAoSYcRKApsH+WK3Zyz8ljdLglb/8JbuW0=
+#   endpoint: <YOUR_IP>:<PORT>
+#   latest handshake: X seconds ago  # This should appear!
+#   transfer: X.XX KiB received, Y.YY KiB sent
+```
+
+### Step 7: Test Different Networks
+
+```bash
+# Try connection on:
+1. Home WiFi
+2. Mobile data (cellular)
+3. Different WiFi network (coffee shop, etc.)
+
+# Note which networks work/fail
+# This helps identify carrier or firewall issues
+```
+
+## Diagnostic Commands
+
+### On Server (kerkouane)
+
+```bash
+# Show full WireGuard status
+sudo wg show all
+
+# Show only latest handshakes
+sudo wg show wg0 latest-handshakes
+
+# Capture WireGuard traffic (run during connection attempt)
+sudo tcpdump -i any -n udp port 51820
+
+# Show connected peers
+sudo wg show wg0 endpoints
+
+# Check if packets are forwarded
+sudo iptables -L FORWARD -v -n | grep wg
+
+# Test routing from server to client (after handshake)
+ping 10.100.0.5
+```
+
+### On iOS Device
+
+```bash
+# No direct CLI on iOS, but you can:
+1. Check WireGuard app logs (tap on tunnel, scroll down)
+2. Use Network utility apps (iNetTools, Ping)
+3. Check "Latest handshake" timestamp in app
+4. Test with Safari: http://10.100.0.1
+```
+
+### On Another Machine (for comparison)
+
+```bash
+# Create test config on Linux/Mac
+sudo wg-quick up wg0
+
+# Or with temporary interface
+sudo ip link add dev wg0 type wireguard
+sudo wg set wg0 private-key <(echo "TEST_PRIVATE_KEY")
+sudo wg set wg0 peer +H3fxErP9HoFUrPgU19ra9+GDLQw+VwvLWx3lMct7QI= \
+  endpoint 167.99.17.238:51820 \
+  allowed-ips 10.100.0.0/24 \
+  persistent-keepalive 25
+sudo ip addr add 10.100.0.99/24 dev wg0
+sudo ip link set wg0 up
+
+# Test handshake
+sudo wg show
+```
+
+## Verification Checklist
+
+After configuration changes, verify:
+
+- [ ] Server shows peer configured: `sudo wg show wg0 peers`
+- [ ] Firewall allows UDP 51820: `sudo iptables -L INPUT -v -n | grep 51820`
+- [ ] NAT rule is correct: `sudo iptables -t nat -L POSTROUTING -v -n`
+- [ ] IP forwarding enabled: `sysctl net.ipv4.ip_forward` returns 1
+- [ ] iOS config has DNS set: Check WireGuard app interface config
+- [ ] iOS config has MTU=1280: Check WireGuard app interface config
+- [ ] iOS public key matches server config: Compare app and globals.nix
+- [ ] Server public key matches iOS peer config: Compare globals.nix and app
+- [ ] Endpoint IP is correct: 167.99.17.238:51820
+- [ ] AllowedIPs is correct: 10.100.0.0/24 (or 0.0.0.0/0 for full tunnel)
+
+## Working Configuration Example
+
+### Server (kerkouane) - modules/wireguard-server.nix
+
+```nix
+{
+  services.wireguard.server = {
+    enable = true;
+    ips = libx.wg-ips globals.machines.kerkouane.net.vpn.ips;
+    peers = libx.generateWireguardPeers globals.machines;
+  };
+
+  boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
+
+  networking.firewall.extraCommands = ''
+    iptables -t nat -A POSTROUTING -s 10.100.0.0/24 -j MASQUERADE
+    iptables -A FORWARD -i wg+ -j ACCEPT
+  '';
+
+  networking.firewall.allowedUDPPorts = [ 51820 ];
+  networking.firewall.trustedInterfaces = [ "wg0" ];
+}
+```
+
+### Client (iOS) - WireGuard App Config
+
+```ini
+[Interface]
+PrivateKey = oK8qaDg9FXYSVwGp9NM8xz8VQm7JwLN7CwTQMb7Hv1E=
+Address = 10.100.0.5/24
+DNS = 8.8.8.8
+MTU = 1280
+
+[Peer]
+PublicKey = +H3fxErP9HoFUrPgU19ra9+GDLQw+VwvLWx3lMct7QI=
+Endpoint = 167.99.17.238:51820
+AllowedIPs = 10.100.0.0/24
+PersistentKeepalive = 25
+```
+
+### Server Config (globals.nix)
+
+```nix
+hokkaido = {
+  net = {
+    vpn = {
+      pubkey = "1wzFG60hlrAoSYcRKApsH+WK3Zyz8ljdLglb/8JbuW0=";
+      ips = [ "10.100.0.5" ];
+    };
+  };
+};
+
+kerkouane = {
+  net = {
+    vpn = {
+      pubkey = "+H3fxErP9HoFUrPgU19ra9+GDLQw+VwvLWx3lMct7QI=";
+      ips = [ "10.100.0.1" ];
+    };
+  };
+};
+```
+
+## Advanced Troubleshooting
+
+### Packet Capture Analysis
+
+```bash
+# On server, capture handshake initiation
+sudo tcpdump -i any -vvv -X udp port 51820
+
+# Look for:
+# - Packets FROM client IP TO server:51820 (initiation)
+# - Packets FROM server:51820 TO client IP (response)
+# - If you see initiation but no response → server issue
+# - If you see no packets at all → network/firewall issue
+```
+
+### Testing with Different Endpoints
+
+```bash
+# Try different port (if carrier blocks 51820)
+# On server:
+networking.wireguard.interfaces.wg0.listenPort = 443;
+networking.firewall.allowedUDPPorts = [ 443 ];
+
+# On iOS:
+Endpoint = 167.99.17.238:443
+```
+
+### Enable Debug Logging (iOS)
+
+1. Export tunnel config from WireGuard app
+2. Edit .conf file, add to [Interface]:
+   ```ini
+   # No debug flag in iOS, but you can:
+   # - Check system Settings → Privacy → Analytics → Analytics Data
+   # - Look for WireGuard crash logs
+   ```
+
+### Test with Minimal Config
+
+```ini
+# Minimal iOS config to isolate issues
+[Interface]
+PrivateKey = <GENERATED>
+Address = 10.100.0.5/32
+
+[Peer]
+PublicKey = +H3fxErP9HoFUrPgU19ra9+GDLQw+VwvLWx3lMct7QI=
+Endpoint = 167.99.17.238:51820
+AllowedIPs = 10.100.0.1/32
+PersistentKeepalive = 25
+```
+
+If this works, gradually add: DNS, full subnet AllowedIPs, MTU changes.
+
+## Performance Tuning
+
+Once working, optimize for performance:
+
+### Increase MTU (if stable)
+
+```ini
+# Test progressively higher MTU values
+MTU = 1280  # Start here
+MTU = 1360  # Try this
+MTU = 1400  # Or this
+MTU = 1420  # Default (best performance if it works)
+```
+
+### Adjust Keepalive
+
+```ini
+# Default
+PersistentKeepalive = 25
+
+# For unstable cellular (more aggressive)
+PersistentKeepalive = 15
+
+# For stable WiFi only (reduce bandwidth)
+PersistentKeepalive = 60
+```
+
+### On-Demand Rules (iOS-specific)
+
+Configure automatic VPN activation:
+
+1. In WireGuard app, tap tunnel
+2. Tap "Edit"
+3. Tap "On-Demand Activation"
+4. Configure rules:
+   - **WiFi**: Connect to VPN when on untrusted WiFi
+   - **Cellular**: Always connect on cellular
+   - **Ethernet**: Connect when on any wired network
+
+Example rules:
+```
+Connect On Demand:
+  ☑ Cellular
+  ☑ WiFi (except: Home_WiFi_SSID)
+  ☐ Ethernet
+```
+
+## Common Error Messages
+
+### "Handshake did not complete after 5 seconds"
+
+**Causes**:
+- Server not reachable (firewall, wrong endpoint)
+- Key mismatch
+- Server not running WireGuard
+
+**Solutions**: Check server status, verify endpoint IP/port, verify keys
+
+### "Unable to resolve endpoint"
+
+**Causes**:
+- DNS resolution failure
+- No internet connection on iOS device
+- Invalid endpoint hostname
+
+**Solutions**: Use IP address instead of hostname, check iOS internet connectivity
+
+### "Invalid key"
+
+**Causes**:
+- Malformed base64 key
+- Wrong key length (WireGuard keys are 32 bytes, 44 chars in base64)
+
+**Solutions**: Regenerate keys, check for copy-paste errors
+
+### Connection works but no internet
+
+**Causes**:
+- NAT/masquerade misconfigured
+- DNS not set or incorrect
+- AllowedIPs too restrictive
+
+**Solutions**: Check server MASQUERADE rule, add DNS to iOS config, verify AllowedIPs
+
+## References
+
+### Documentation
+- [WireGuard Quick Start](https://www.wireguard.com/quickstart/)
+- [WireGuard iOS App](https://apps.apple.com/us/app/wireguard/id1441195209)
+- [NixOS WireGuard Options](https://search.nixos.org/options?query=wireguard)
+
+### Troubleshooting Resources
+- [WireGuard VPN - Handshake error - MikroTik](https://forum.mikrotik.com/viewtopic.php?t=189995)
+- [iOS iPhone not connecting on WireGuard - Super User](https://superuser.com/questions/1859181/why-is-an-iphone-not-connecting-on-an-otherwise-okay-wireguard-network-that-othe)
+- [Wireguard no handshake on iOS - MikroTik](https://forum.mikrotik.com/t/wireguard-no-handshake-on-ios/177335)
+- [Wireguard handshake problem with Apple devices - OpenWrt](https://forum.openwrt.org/t/wireguard-handshake-problem-with-apple-devices/161329)
+- [Wireguard problems on NixOS Discourse](https://discourse.nixos.org/t/wireguard-problems-handshake-did-not-complete/26237)
+
+### Tools
+- [WireGuard Configuration Generator](https://www.wireguardconfig.com/)
+- Network testing: iNetTools (iOS app)
+- Packet capture: tcpdump, Wireshark
+
+## Summary
+
+**Most common iOS WireGuard issues**:
+1. Missing or incorrect DNS configuration
+2. MTU too high for cellular networks
+3. Server NAT/masquerade misconfigured
+4. Carrier blocking UDP or using aggressive NAT
+
+**Quick fixes that solve 90% of issues**:
+- Add `DNS = 8.8.8.8` to iOS config
+- Add `MTU = 1280` to iOS config
+- Fix server NAT rule: `-s 10.100.0.0/24` not `/32`
+- Verify keys match between client and server
+
+**Systematic troubleshooting**:
+1. Verify server configuration and logs
+2. Test from known working client first
+3. Configure iOS with minimal settings
+4. Add features incrementally (DNS, MTU, full routing)
+5. Test on different networks (WiFi vs cellular)
+6. Monitor server during connection attempts
+
+When in doubt, start fresh with minimal config and build up from there.

@@ -19,10 +19,8 @@ in
     overrideFolders = false; # TODO Just in case, will probably set to true later
     guiAddress = libx.syncthingGuiAddress globals.machines."${hostname}";
     settings = {
-      # FIXME this doesn't work, I wish it did.
-      # defaults = {
-      #   ignores = { lines = [ "(?d).DS_Store" "**" ]; };
-      # };
+      # Note: Default ignores are applied per-folder in generateSyncthingFolders
+      # (Syncthing 2.0+ removed the global defaults.ignores feature)
       devices = libx.generateSyncthingDevices hostname globals.machines;
       inherit folders;
     };

@@ -210,6 +210,18 @@ let
   */
   generateSyncthingFolders =
     hostname: machine: machines: folders:
+    let
+      # Default ignore patterns applied to all folders unless overridden
+      defaultIgnores = [
+        "(?d).DS_Store" # macOS metadata files
+        "(?d).localized" # macOS localized folder names
+        "(?d)Thumbs.db" # Windows thumbnails
+        "(?d)desktop.ini" # Windows folder config
+        "*.tmp" # Temporary files
+        "~*" # Backup files
+        ".~lock.*" # LibreOffice lock files
+      ];
+    in
     lib.attrsets.mapAttrs' (
       name: value:
       lib.attrsets.nameValuePair (syncthingFolderPath name value folders) {
@@ -219,6 +231,8 @@ let
           syncthingMachinesWithFolder hostname name machines
         );
         rescanIntervalS = 3600 * 6; # TODO: make it configurable
+        # Apply default ignores if not specified in globals
+        ignores = folders."${name}".ignores or defaultIgnores;
       }
     ) (lib.attrsets.attrByPath [ "syncthing" "folders" ] { } machine);
 

@@ -12,6 +12,9 @@ let
     types
     ;
   cfg = config.services.wireguard.server;
+
+  # Detect if nftables is enabled
+  usingNftables = config.networking.nftables.enable;
 in
 {
   options = {
@@ -42,12 +45,37 @@ in
   config = mkIf cfg.enable {
     environment.systemPackages = [ pkgs.wireguard-tools ];
     boot.kernel.sysctl."net.ipv4.ip_forward" = lib.mkForce 1; # FIXME should probably be mkDefault
-    networking.firewall.extraCommands = ''
-      iptables -t nat -A POSTROUTING -s 10.100.0.0/24 -j MASQUERADE
-      iptables -A FORWARD -i wg+ -j ACCEPT
-    '';
-    networking.firewall.allowedUDPPorts = [ 51820 ];
-    networking.firewall.trustedInterfaces = [ "wg0" ];
+
+    # Firewall configuration - supports both iptables and nftables
+    networking.firewall = {
+      allowedUDPPorts = [ 51820 ];
+      trustedInterfaces = [ "wg0" ];
+
+      # iptables rules (used when nftables is disabled)
+      extraCommands = mkIf (!usingNftables) ''
+        iptables -t nat -A POSTROUTING -s 10.100.0.0/24 -j MASQUERADE
+        iptables -A FORWARD -i wg+ -j ACCEPT
+      '';
+    };
+
+    # nftables rules (used when nftables is enabled)
+    networking.nftables.tables = mkIf usingNftables {
+      wireguard-nat = {
+        family = "ip";
+        content = ''
+          chain postrouting {
+            type nat hook postrouting priority srcnat; policy accept;
+            ip saddr 10.100.0.0/24 masquerade
+          }
+
+          chain forward {
+            type filter hook forward priority filter; policy accept;
+            iifname "wg*" accept
+          }
+        '';
+      };
+    };
+
     networking.wireguard.enable = true;
     networking.wireguard.interfaces = {
       "wg0" = {