Maintain Workflow

Maintain container infrastructure through resource cleanup and security scanning for healthy, secure operations.

Quick Start

Common maintenance tasks:

# Check system health
docker system df          # Disk usage
trivy image myapp:latest  # Security scan

# Quick cleanup
docker container prune    # Remove stopped containers
docker image prune        # Remove dangling images

# Weekly maintenance
docker system prune -a --volumes  # Full cleanup
trivy image --severity CRITICAL,HIGH myapp:latest

Quick commands reference:

# Runtime detection
./DetectRuntime.sh

# Cleanup (Docker/Podman)
docker system df
docker system prune -a --volumes
podman system df
podman system prune -a --volumes

# Security scanning
trivy image myapp:latest
docker scout cves myapp:latest
grype myapp:latest

Resource Cleanup

Assess Current Usage

System-wide overview:

docker system df
podman system df

Detailed view:

docker system df -v
podman system df -v

Pruning Containers

Remove all stopped containers:

docker container prune
podman container prune

Remove specific container:

docker rm container_name
podman rm container_name

Remove all containers (dangerous!):

# Stop all first
docker stop $(docker ps -aq)
docker rm $(docker ps -aq)

podman stop -a
podman rm -a

Filter by age:

# Remove containers stopped more than 24 hours ago
docker container prune --filter "until=24h"
podman container prune --filter "until=24h"

Filter by label:

docker container prune --filter "label=project=myapp"
podman container prune --filter "label=project=myapp"

Pruning Images

Remove dangling images (untagged):

docker image prune
podman image prune

Remove unused images:

docker image prune -a
podman image prune -a

Remove specific image:

docker rmi image_name:tag
podman rmi image_name:tag

Remove all images (dangerous!):

docker rmi $(docker images -q)
podman rmi -a

Remove images created more than 7 days ago:

docker image prune -a --filter "until=168h"
podman image prune -a --filter "until=168h"

Clean specific image tags:

# Remove old versions of specific image
docker images myapp --format "{{.Tag}}" | grep -v "latest\|v1.0" | xargs -I {} docker rmi myapp:{}

Clean by pattern:

# Remove all development images
docker images | grep "\-dev" | awk '{print $3}' | xargs docker rmi

# Remove unnamed images
docker images --filter "dangling=true" -q | xargs docker rmi

Pruning Volumes

Remove unused volumes:

docker volume prune
podman volume prune

Remove specific volume:

docker volume rm volume_name
podman volume rm volume_name

Remove all volumes (dangerous!):

docker volume rm $(docker volume ls -q)
podman volume rm -a

Pruning Networks

Remove unused networks:

docker network prune
podman network prune

Remove specific network:

docker network rm network_name
podman network rm network_name

Build Cache Cleanup

Clear build cache:

docker builder prune
podman system prune --all --volumes

Clear all build cache (more aggressive):

docker builder prune -a

System-wide Cleanup

Prune everything (containers, images, volumes, networks):

docker system prune -a --volumes
podman system prune -a --volumes

With force (skip confirmation):

docker system prune -a --volumes -f
podman system prune -a --volumes -f

Safe Cleanup Strategy

Step-by-step approach:

1. Start conservative:

   # Remove stopped containers
   docker container prune
   
   # Remove dangling images
   docker image prune
   

2. Then volumes:

   # Review volumes first
   docker volume ls
   
   # Remove unused
   docker volume prune
   

3. Finally unused images:

   # This removes images not used by any container
   docker image prune -a
   

4. Build cache (if needed):

   docker builder prune
   

Retention strategy:

# Keep images from last week
docker image prune -a --filter "until=168h"

# Keep containers stopped in last 24h
docker container prune --filter "until=24h"

Automated Cleanup

Docker system prune with cron:

# Add to crontab (weekly cleanup)
0 2 * * 0 docker system prune -a --volumes -f >> /var/log/docker-cleanup.log 2>&1

Podman auto-prune with systemd:

# Podman rootless with systemd timer
systemctl --user enable --now podman-auto-update.timer

Disk Space Management

Check for large log files:

# Docker logs can grow large
ls -lh /var/lib/docker/containers/*/*-json.log

Configure log rotation:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Podman-Specific Cleanup

# Reset entire podman storage (nuclear option)
podman system reset

# Clean up pods
podman pod prune

# Clean up specific user's containers (rootless)
podman system prune --all

Security Scanning

Scanning Tools Overview

Recommended tools:

• Trivy: Comprehensive, recommended for most use cases
• Docker Scout: Built-in Docker, easy integration
• Grype: Anchore’s scanner, good SBOM support
• Snyk: Enterprise features, policy management
• Clair: CoreOS scanner, good for registries

Trivy (Recommended)

Installation:

# Using package manager
# Debian/Ubuntu
sudo apt install trivy

# Fedora/RHEL
sudo dnf install trivy

# macOS
brew install trivy

# Or use container
alias trivy="docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy"

Basic scanning:

trivy image myapp:latest
trivy image nginx:latest

Scan with specific severity:

trivy image --severity HIGH,CRITICAL myapp:latest

Scan for specific vulnerability types:

# OS packages only
trivy image --vuln-type os myapp:latest

# Application dependencies only
trivy image --vuln-type library myapp:latest

# Both
trivy image --vuln-type os,library myapp:latest

Output formats:

# Table (default)
trivy image myapp:latest

# JSON
trivy image -f json myapp:latest

# SARIF (for GitHub)
trivy image -f sarif -o results.sarif myapp:latest

# HTML report
trivy image -f template --template "@contrib/html.tpl" -o report.html myapp:latest

Scan filesystem and configurations:

# Scan project directory
trivy fs /path/to/project
trivy fs --security-checks vuln,secret,config .

# Scan Dockerfile
trivy config Dockerfile

# Scan Kubernetes manifests
trivy config deployment.yaml

Advanced Trivy options:

# Ignore unfixed vulnerabilities
trivy image --ignore-unfixed myapp:latest

# Exit with error on critical vulnerabilities
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan remote image
trivy image ghcr.io/username/myapp:latest

# Use custom cache directory
trivy image --cache-dir /tmp/trivy myapp:latest

Using .trivyignore file:

# .trivyignore
CVE-2023-1234
CVE-2023-5678 # False positive: feature not used

Docker Scout (Built-in)

Enable Docker Scout:

# Login
docker login

# Enable Scout
docker scout enroll

Scan images:

# Quick scan
docker scout quickview myapp:latest

# CVE scan
docker scout cves myapp:latest

# Show only fixable
docker scout cves --only-fixed myapp:latest

# Compare images
docker scout compare myapp:old myapp:new

# Get recommendations
docker scout recommendations myapp:latest

# Export to SARIF
docker scout cves --format sarif myapp:latest > results.sarif

Grype (Anchore)

Installation:

# Using package manager
brew install grype

# Or download binary
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh

# Or use container
alias grype="docker run --rm -v /var/run/docker.sock:/var/run/docker.sock anchore/grype"

Scan images:

# Basic scan
grype myapp:latest

# Scan with filters
grype myapp:latest --only-fixed
grype myapp:latest --fail-on critical

# Output formats
grype -o json myapp:latest
grype -o table myapp:latest
grype -o cyclonedx myapp:latest  # SBOM format

Snyk Container

Installation:

# Install Snyk CLI
npm install -g snyk

# Or use binary
curl -O https://static.snyk.io/cli/latest/snyk-linux
chmod +x snyk-linux
mv snyk-linux /usr/local/bin/snyk

# Authenticate
snyk auth

Scan images:

# Scan image
snyk container test myapp:latest

# Monitor for vulnerabilities
snyk container monitor myapp:latest

# Test and generate report
snyk container test myapp:latest --json-file-output=results.json

Secret Scanning

Trivy secret scan:

trivy fs --security-checks secret .
trivy image --security-checks secret myapp:latest

TruffleHog:

# Install
pip install truffleHog

# Scan filesystem
trufflehog filesystem /path/to/project

# Scan container image
docker save myapp:latest | trufflehog

SBOM Generation

Generate Software Bill of Materials:

# Trivy SBOM
trivy image -f cyclonedx myapp:latest > sbom.json
trivy image -f spdx myapp:latest > sbom.spdx

# Grype SBOM
grype -o cyclonedx myapp:latest > sbom.json

# Syft (dedicated SBOM tool)
syft myapp:latest -o cyclonedx-json > sbom.json

Dockerfile Best Practices Scanning

Hadolint (Dockerfile linter):

# Install
brew install hadolint

# Or use container
docker run --rm -i hadolint/hadolint < Dockerfile

# Scan Dockerfile
hadolint Dockerfile

# Ignore specific rule
hadolint --ignore DL3008 Dockerfile

Docker Bench Security:

# Run security audit
docker run --rm --net host --pid host --userns host --cap-add audit_control \
  -v /var/lib:/var/lib \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /etc:/etc \
  --label docker_bench_security \
  docker/docker-bench-security

Security Best Practices

Base image selection:

# Prefer minimal base images
FROM alpine:3.19
FROM debian:bookworm-slim
FROM gcr.io/distroless/static-debian12

# Avoid
FROM ubuntu:latest  # Too large, "latest" is unpredictable

Multi-stage builds:

# Build stage with more tools
FROM golang:1.21 AS builder
WORKDIR /src
COPY . .
RUN go build -o app

# Runtime stage with minimal image
FROM gcr.io/distroless/static-debian12
COPY --from=builder /src/app /app
ENTRYPOINT ["/app"]

Update dependencies:

# Update OS packages during build
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y --no-install-recommends \
        package1 \
        package2 && \
    rm -rf /var/lib/apt/lists/*

Use specific versions:

# Bad: unpredictable
FROM python:latest

# Good: specific and minimal
FROM python:3.11-slim-bookworm

Run as non-root user:

# Create and use non-root user
RUN adduser -D -u 1000 appuser
USER appuser
CMD ["/app"]

Avoid secrets in images:

# Bad: secrets in image
ENV API_KEY="secret123"
COPY .env /app/.env

# Good: secrets at runtime via environment or secrets management
# No secrets in Dockerfile

Remediation Strategies

Update base images regularly:

docker pull alpine:3.19
docker build --no-cache -t myapp:latest .

Update application dependencies:

# Go
RUN go get -u && go mod tidy

# Node
RUN npm update

# Python
RUN pip install --upgrade -r requirements.txt

Use distroless images:

FROM gcr.io/distroless/python3-debian12
COPY --from=builder /app /app
CMD ["/app/main.py"]

Interpreting Scan Results

Vulnerability severity (CVSS):

• Critical (9.0-10.0): Immediate action required
• High (7.0-8.9): Fix within 1 week
• Medium (4.0-6.9): Fix within 1 month
• Low (0.1-3.9): Fix opportunistically

Handle false positives:

• Vulnerability in unused code path
• Requires conditions that don’t exist
• Already mitigated at runtime
• Document in .trivyignore with reason

CI/CD Integration

GitHub Actions:

name: Container Scan

on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Trivy scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          format: 'sarif'
          output: 'trivy-results.sarif'

      - name: Upload Trivy results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

GitLab CI:

container_scan:
  image: aquasec/trivy:latest
  script:
    - trivy image --format json --output results.json myapp:latest
  artifacts:
    reports:
      container_scanning: results.json

Jenkins:

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'docker build -t myapp:latest .'
            }
        }
        stage('Security Scan') {
            steps {
                sh 'trivy image --exit-code 1 --severity CRITICAL myapp:latest'
            }
        }
    }
}

Monitoring and Health

Resource Usage Monitoring

Check system resources:

# Disk usage overview
docker system df
docker system df -v

# Container resource usage
docker stats
docker stats --no-stream

# Podman stats
podman stats

Container Metrics

Monitor container health:

# View container logs
docker logs container_name
docker logs -f --tail 100 container_name

# Inspect container
docker inspect container_name

# Check container health status
docker ps --filter "health=unhealthy"

Health Checks

Define health checks in Dockerfile:

HEALTHCHECK --interval=30s --timeout=3s \
  CMD curl -f http://localhost:8080/health || exit 1

Check health status:

docker inspect --format='{{.State.Health.Status}}' container_name

Automated Maintenance

Combined cleanup and security scan script:

#!/bin/bash
# weekly-maintenance.sh

# Cleanup
echo "Starting cleanup..."
docker system prune -a --volumes -f

# Security scan
echo "Scanning images..."
for image in $(docker images --format "{{.Repository}}:{{.Tag}}" | grep -v "<none>"); do
  echo "Scanning $image"
  trivy image --severity CRITICAL,HIGH "$image"
done

# Report
echo "Maintenance completed"
docker system df

Schedule with cron:

# Every Sunday at 2 AM
0 2 * * 0 /path/to/weekly-maintenance.sh >> /var/log/docker-maintenance.log 2>&1

Best Practices

Maintenance Schedule

Regular tasks:

• Daily: Monitor disk usage and container health
• Weekly: Prune stopped containers and dangling images
• Monthly: Full system cleanup and security scan
• Quarterly: Review and update base images

Safety Guidelines

Before cleanup:

• Always preview what will be removed
• Never use -f (force) without understanding impact
• Keep tagged images you need
• Backup important volumes before cleanup
• Test in development first

Before security fixes:

• Understand each vulnerability
• Check if it applies to your use case
• Test updates in staging
• Have rollback plan

Security Scanning Workflow

1. Scan during build - Catch issues early
2. Scan before push - Gate deployments
3. Scan in CI/CD - Automated verification
4. Scan running containers - Regular monitoring
5. Monitor for new CVEs - Stay informed

Integration Strategy

Combine cleanup with security:

# Before cleanup, scan what you're keeping
docker images --format "{{.Repository}}:{{.Tag}}" | \
  while read image; do
    trivy image --severity CRITICAL "$image"
  done

# Then cleanup old/vulnerable images
docker image prune -a --filter "until=168h"

Troubleshooting

Cleanup Issues

Issue: Important data deleted

• Prevention: Always use named volumes for important data
• Recovery: Check volume backups if available

Issue: Still using too much space

• Check for large log files
• Configure log rotation
• Clean build cache: docker builder prune -a

Issue: Cannot remove image in use

• Check which containers use it: docker ps -a --filter ancestor=image_name
• Stop and remove containers first

Issue: Permission denied (Docker)

• Use sudo or add user to docker group
• With Podman: run as regular user (rootless)

Security Scan Issues

Issue: Too many false positives

• Use .trivyignore to document
• Check if vulnerability applies to your use case
• Focus on fixable issues first

Issue: Scan takes too long

• Use --ignore-unfixed to skip unfixable issues
• Scan specific vulnerability types only
• Use local cache directory

Issue: Cannot update vulnerable package

• Check if vulnerability affects your code path
• Consider alternative base image
• Document mitigation if no fix available

Issue: Scan fails on large images

• Increase timeout settings
• Use streaming scan mode
• Scan in stages (base image, then layers)

Output Format Examples

After cleanup:

✓ Cleanup completed successfully
  Runtime: docker

  Removed:
    - Containers: 5
    - Images: 12
    - Volumes: 3
    - Networks: 2

  Space reclaimed: 2.5 GB

  Current usage:
    - Images: 5.2 GB
    - Containers: 150 MB
    - Volumes: 800 MB
    - Build cache: 1.1 GB
    - Total: 7.25 GB

Next steps:
  - Schedule regular cleanup
  - Configure log rotation
  - Scan remaining images for vulnerabilities

After security scan:

✓ Security scan completed
  Image: myapp:latest
  Scanner: Trivy

  Vulnerabilities found:
    Critical: 2
    High: 5
    Medium: 12
    Low: 8

  Top issues:
    1. CVE-2023-1234 (Critical) - OpenSSL vulnerability
       Fix: Update to openssl 3.0.10
    2. CVE-2023-5678 (High) - curl buffer overflow
       Fix: Update to curl 8.1.2

  Secrets found: 0
  Misconfigurations: 3

  Recommendations:
    - Update base image from alpine:3.17 to alpine:3.19
    - Remove unnecessary package: wget
    - Add non-root user

  Full report: results.json

Next steps:
  - Update base image
  - Rebuild and rescan
  - Review medium severity issues