Advanced Tekton Workflows

This reference consolidates advanced Tekton workflows that are used less frequently.

Core workflows (used frequently) are in separate workflow files:

Pipelines (workflows/Pipelines.md)
Tasks (workflows/Tasks.md)
Build (workflows/Build.md)
TknCli (workflows/TknCli.md)
Triggers (workflows/Triggers.md)

BestPractices - Tekton Best Practices

Tekton best practices for security, performance, reliability, and maintainability.

When to Use

Optimizing Tekton pipelines
Implementing security best practices
Improving performance
Operating Tekton at scale
Troubleshooting common issues

Security Best Practices

Workspace-Based Authentication (2024 Recommendation)

DO: Use workspace-based authentication

workspaces:
  - name: git-credentials
    secret:
      secretName: git-ssh-key

DON’T: Rely only on built-in credential initialization

Being phased out
All steps get access to all credentials
Credentials copied to disk
Security issues with different UIDs

ServiceAccount and RBAC

Create dedicated ServiceAccounts:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pipeline-sa
secrets:
  - name: docker-credentials
  - name: git-credentials
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pipeline-role
rules:
  - apiGroups: [""]
    resources: ["pods", "services"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pipeline-binding
subjects:
  - kind: ServiceAccount
    name: pipeline-sa
roleRef:
  kind: Role
  name: pipeline-role
  apiGroup: rbac.authorization.k8s.io

Principle of least privilege:

Create separate ServiceAccounts for CI and CD
Grant only necessary permissions
Use Role instead of ClusterRole when possible
Regularly audit permissions

Pod Security

spec:
  steps:
    - name: secure-step
      image: alpine
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL

Secrets Management

DO:

Use Kubernetes Secrets or external secret managers (Vault, AWS Secrets Manager)
Mount secrets as volumes, not environment variables
Use short-lived tokens (OAuth, OIDC)
Rotate secrets regularly
Scan secrets with tools like gitleaks, trufflehog

DON’T:

Commit secrets to Git
Print secrets in logs
Use long-lived credentials
Share secrets across environments

Performance Best Practices

Resource Management

Always set resource requests and limits:

steps:
  - name: build
    image: maven:3.8
    resources:
      requests:
        cpu: 1000m
        memory: 2Gi
      limits:
        cpu: 2000m
        memory: 4Gi

Guidelines:

Set requests based on average usage
Set limits 1.5-2x higher than requests
Monitor actual usage: kubectl top pods
Use task-level resources for simplicity

Configure Pipeline Pruner

Automatic cleanup of old resources:

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
spec:
  pruner:
    disabled: false
    schedule: "0 8 * * *"  # Daily at 8 AM
    keep: 3  # Keep last 3 runs
    resources:
      - pipelinerun
      - taskrun

Manual cleanup:

# Keep last 5 runs
tkn pipelinerun delete --keep 5 -f

# Delete runs older than 60 minutes
tkn pipelinerun delete --keep-since 60 -f

# Delete all failed runs
tkn pipelinerun delete --all --failed -f

Workspace Optimization

Use volumeClaimTemplate for isolation:

workspaces:
  - name: workspace
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

Avoid PVC sharing conflicts:

ReadWriteOnce: Limited to single node
ReadWriteMany: Best for Tekton but less common
Use emptyDir for temporary data
Use subPaths for organization

Caching Strategies

Maven cache:

workspaces:
  - name: maven-cache
    persistentVolumeClaim:
      claimName: maven-cache-pvc

steps:
  - name: build
    image: maven:3.8
    env:
      - name: MAVEN_OPTS
        value: "-Dmaven.repo.local=$(workspaces.maven-cache.path)"

Kaniko cache:

args:
  - --cache=true
  - --cache-repo=myregistry.com/cache
  - --cache-ttl=24h

npm cache:

workspaces:
  - name: npm-cache

steps:
  - name: install
    image: node:18
    script: |
      npm config set cache $(workspaces.npm-cache.path)
      npm install

Reliability Best Practices

Error Handling

Use retries for flaky operations:

tasks:
  - name: flaky-network-test
    retries: 3
    taskRef:
      name: integration-test

Use onError for optional tasks:

tasks:
  - name: optional-scan
    onError: continue
    taskRef:
      name: security-scan

Implement timeout:

spec:
  timeouts:
    pipeline: 2h
    tasks: 1h
    finally: 30m

tasks:
  - name: long-running
    timeout: 45m

Finally Tasks for Cleanup

finally:
  - name: cleanup-workspace
    taskRef:
      name: cleanup
    workspaces:
      - name: workspace
        workspace: shared-data

  - name: send-notification
    taskRef:
      name: notify
    params:
      - name: status
        value: $(tasks.status)
      - name: message
        value: "Pipeline $(context.pipelineRun.name) $(tasks.status)"

Idempotent Tasks

# BAD: Not idempotent
script: |
  echo "data" >> file.txt  # Appends on retry

# GOOD: Idempotent
script: |
  echo "data" > file.txt   # Overwrites on retry

Maintainability Best Practices

Task Reusability

DO: Parameterize everything

params:
  - name: image
  - name: dockerfile
    default: ./Dockerfile
  - name: context
    default: .

DON’T: Hardcode values

# Anti-pattern
args:
  - --destination=myregistry.com/myimage:latest

Use Tekton Hub

Check Hub before creating custom tasks:

tkn hub search git
tkn hub info task git-clone
tkn hub install task git-clone

Reference Hub tasks:

taskRef:
  resolver: hub
  params:
    - name: name
      value: git-clone
    - name: version
      value: "0.10.0"

Migration from Deprecated Features

AVOID: ClusterTasks (deprecated)

# Old
taskRef:
  name: git-clone
  kind: ClusterTask

USE: Resolvers

# New - Cluster resolver
taskRef:
  resolver: cluster
  params:
    - name: name
      value: git-clone
    - name: namespace
      value: tekton-tasks

# New - Hub resolver (preferred)
taskRef:
  resolver: hub
  params:
    - name: name
      value: git-clone

AVOID: PipelineResources (deprecated)

Use Tasks with Results and Workspaces instead

Documentation

Document tasks comprehensively:

metadata:
  name: build-app
  labels:
    app.kubernetes.io/version: "1.0"
  annotations:
    tekton.dev/pipelines.minVersion: "0.50.0"
    tekton.dev/categories: Build Tools
    tekton.dev/tags: build, maven
spec:
  description: |
    Builds Java application using Maven.

    This task compiles source code, runs tests,
    and packages the application as a JAR file.

    ## Parameters
    - maven-goals: Maven goals to execute (default: clean package)
    - settings-path: Path to custom settings.xml

    ## Workspaces
    - source: Source code directory
    - maven-cache: Maven local repository cache (optional)

    ## Results
    - artifact-name: Name of the generated JAR file

Versioning

metadata:
  labels:
    app.kubernetes.io/version: "2.1.0"
    version: "2.1.0"

Anti-Patterns to Avoid

1. Not Setting Resource Limits

Impact: Poor scheduling, resource contention, cluster instability

Solution:

resources:
  requests:
    cpu: 500m
    memory: 1Gi
  limits:
    cpu: 1000m
    memory: 2Gi

2. Using :latest Image Tag

Impact: Non-deterministic builds, cache issues

Solution:

image: maven:3.8.6-openjdk-17  # Specific version

3. Monolithic Tasks

Impact: Poor reusability, difficult debugging

Solution: Break into focused, single-responsibility tasks

4. Ignoring Pipeline Pruner

Impact: etcd bloat, performance degradation

Solution: Configure automatic pruning (see above)

Impact: Pod scheduling failures

Solution:

Use ReadWriteMany
Use volumeClaimTemplate
Avoid parallel tasks with shared workspace

6. Results Over 4KB

Impact: Task failures

Solution: Use workspaces for large data

7. Not Using Tekton Hub

Impact: Reinventing the wheel, maintenance burden

Solution: Check Hub first, contribute improvements

8. Overly Permissive RBAC

Impact: Security risks

Solution: Least privilege principle

9. Secrets in Logs

Impact: Credential exposure

Solution: Never print sensitive data

10. Not Testing Tasks Isolation

Impact: Pipeline failures

Solution: Test TaskRuns independently before using in pipelines

Operating at Scale (Red Hat Lessons)

1. Configure Pipeline Pruner

Delete completed pods to free volumes and improve performance

2. Set Resource Requests/Limits

Scheduler performs poorly without them

3. Understand Workload Bottlenecks

Profile and optimize based on actual constraints

Use volumeClaimTemplate or manage access modes carefully

5. Monitor etcd Performance

Prune old resources, use Tekton Results for long-term storage

6. Plan for Multi-AZ Clusters

PVCs are tied to specific availability zones

7. Use Tekton Resolvers

Replace ClusterTasks with cluster/hub/git resolvers

8. Implement Retry Strategies

Set retries for flaky network operations

9. Version Control Everything

Pin task/pipeline versions for stability

10. Test at Scale

Load test pipelines before production deployment

Monitoring and Observability

Tekton Dashboard

# Install Tekton Dashboard
kubectl apply -f https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml

# Access dashboard
kubectl port-forward -n tekton-pipelines svc/tekton-dashboard 9097:9097

Prometheus Metrics

Tekton exports metrics:

tekton_pipelinerun_duration_seconds
tekton_pipelinerun_count
tekton_taskrun_duration_seconds
tekton_taskrun_count

Logging Best Practices

Structured logging in tasks:

script: |
  echo "[INFO] Starting build"
  echo "[ERROR] Build failed: $ERROR_MSG"
  echo "[SUCCESS] Build completed"

Aggregate logs:

Use Fluentd/Fluent Bit to collect logs
Send to Elasticsearch, Loki, or CloudWatch
Create dashboards in Grafana, Kibana

Alerting

Alert on:

Pipeline failure rate exceeding threshold
Long-running pipelines
Resource quota exhaustion
PVC capacity
Failed webhook deliveries

Checklist

Security Checklist

Use workspace-based authentication
Create dedicated ServiceAccounts
Follow least privilege RBAC
Run containers as non-root
Scan images for vulnerabilities
Rotate credentials regularly
Don’t commit secrets to Git
Use signed commits

Performance Checklist

Set resource requests/limits on all steps
Configure pipeline pruner
Use caching (Kaniko, Maven, npm)
Use volumeClaimTemplate for isolation
Monitor resource usage
Optimize Dockerfile layer caching
Use specific image tags

Reliability Checklist

Set retries for flaky operations
Use onError for optional tasks
Implement timeouts
Use finally tasks for cleanup
Write idempotent tasks
Test tasks in isolation
Monitor pipeline success rate

Maintainability Checklist

Check Tekton Hub before custom tasks
Parameterize all tasks
Document parameters and workspaces
Use resolvers (not ClusterTasks)
Version tasks and pipelines
Use consistent naming conventions
Organize tasks by function

Debug - Debugging Tekton Pipelines

Debugging Tekton pipelines, tasks, and runs by leveraging Kubernetes debugging techniques.

When to Use

PipelineRun or TaskRun failing
Tasks not starting or getting stuck
Investigating pod failures in Tekton
Debugging workspace issues
Troubleshooting trigger problems

Overview

Tekton runs on Kubernetes, so debugging Tekton involves:

Tekton-specific debugging: Using tkn CLI and Tekton resources
Kubernetes debugging: Using kubectl to inspect underlying pods (see Kubernetes/Debug workflow)

This workflow combines both approaches for comprehensive troubleshooting.

Quick Diagnosis

Check PipelineRun/TaskRun Status

# Get pipelinerun status
tkn pipelinerun describe <pipelinerun-name>
tkn pipelinerun describe --last

# Get taskrun status
tkn taskrun describe <taskrun-name>
tkn taskrun describe --last

# List recent runs
tkn pipelinerun list
tkn taskrun list

View Logs

# Stream pipelinerun logs
tkn pipelinerun logs <pipelinerun-name> -f
tkn pipelinerun logs --last -f

# Stream taskrun logs
tkn taskrun logs <taskrun-name> -f
tkn taskrun logs --last -f

# View specific task logs in pipeline
tkn pipelinerun logs <pipelinerun-name> -t <task-name>

Debugging Workflow

1. Identify the Problem

Check PipelineRun status:

# Quick status check
kubectl get pipelinerun <name>

# Detailed status
tkn pipelinerun describe <name>

# Check conditions
kubectl get pipelinerun <name> -o jsonpath='{.status.conditions[*]}'

Common statuses:

Running: Pipeline is executing
Succeeded: Pipeline completed successfully
Failed: Pipeline failed
PipelineRunCancelled: Pipeline was cancelled
PipelineRunTimeout: Pipeline exceeded timeout

2. Find the Failing Task

# List all tasks in pipeline
tkn pipelinerun describe <name>

# Check which tasks failed
kubectl get pipelinerun <name> -o jsonpath='{.status.taskRuns}' | jq

# View task status
tkn taskrun describe <taskrun-name>

3. Examine Task Logs

# View all task logs
tkn pipelinerun logs <name> --all

# View specific failing task
tkn pipelinerun logs <name> -t <failing-task-name>

# View taskrun logs directly
tkn taskrun logs <taskrun-name>

# Get previous logs if pod crashed
kubectl logs <pod-name> -c step-<step-name> --previous

4. Inspect the Pod (Kubernetes Debug)

Find the pod:

# Find pods for pipelinerun
kubectl get pods -l tekton.dev/pipelineRun=<pipelinerun-name>

# Find pods for taskrun
kubectl get pods -l tekton.dev/taskRun=<taskrun-name>

Use Kubernetes Debug workflow:

# Check pod status (see Kubernetes/Debug)
kubectl get pods
kubectl describe pod <pod-name>

# Check pod events
kubectl get events --field-selector involvedObject.name=<pod-name>

# Check resource usage
kubectl top pod <pod-name>

5. Check Task Definition

# View task definition
kubectl get task <task-name> -o yaml

# Check parameters
tkn task describe <task-name>

# Verify workspace requirements
kubectl get task <task-name> -o jsonpath='{.spec.workspaces}'

6. Verify Workspaces

# Check workspace bindings in pipelinerun
kubectl get pipelinerun <name> -o jsonpath='{.spec.workspaces}'

# Check PVC status
kubectl get pvc

# Verify PVC is bound
kubectl get pvc <pvc-name> -o jsonpath='{.status.phase}'

# Check workspace contents (if pod still exists)
kubectl exec <pod-name> -c step-<step-name> -- ls -la /workspace

7. Check Results and Parameters

# View task results
tkn taskrun describe <taskrun-name> | grep -A 10 Results

# Get specific result
kubectl get taskrun <name> -o jsonpath='{.status.taskResults[?(@.name=="<result-name>")].value}'

# Check parameters passed
kubectl get pipelinerun <name> -o jsonpath='{.spec.params}'

Common Tekton Issues

Task Not Starting (Pending)

Symptoms:

TaskRun shows Pending status
Pod not created or stuck in Pending

Debug:

# Check taskrun status
tkn taskrun describe <taskrun-name>

# Check pod scheduling
kubectl describe pod <pod-name>

# Check events
kubectl get events --sort-by=.metadata.creationTimestamp

Common causes:

Insufficient cluster resources (see Kubernetes/Debug - Pending Pods)
PVC not bound
Missing ServiceAccount
Invalid workspace configuration
Resource quota exceeded

Solutions:

# Check node resources
kubectl top nodes

# Check PVC
kubectl get pvc
kubectl describe pvc <pvc-name>

# Check ServiceAccount
kubectl get serviceaccount <sa-name>

# Check resource quotas
kubectl get resourcequota

Task Failing with ImagePullBackOff

Symptoms:

Pod containers can’t pull images
Events show Failed to pull image

Debug:

# Check pod events (see Kubernetes/Debug - ImagePullBackOff)
kubectl describe pod <pod-name> | grep -A 5 "Failed"

# Check task definition
kubectl get task <task-name> -o jsonpath='{.spec.steps[*].image}'

Common causes:

Wrong image name or tag
Private registry authentication missing
Network issues

Solutions:

# Verify image exists
docker pull <image>

# Check imagePullSecrets in ServiceAccount
kubectl get serviceaccount <sa-name> -o yaml

# Verify docker config secret
kubectl get secret <docker-config-secret> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

Task Failing with CrashLoopBackOff

Symptoms:

Step containers keep restarting
Pod shows CrashLoopBackOff

Debug:

# Check previous logs (see Kubernetes/Debug - CrashLoopBackOff)
kubectl logs <pod-name> -c step-<step-name> --previous

# Check pod describe
kubectl describe pod <pod-name>

# Check task script
kubectl get task <task-name> -o jsonpath='{.spec.steps[*].script}'

Common causes:

Script errors or missing dependencies
Incorrect command/args
Missing environment variables
Resource limits too low
Workspace permissions

Solutions:

# Check resource limits
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].resources}'

# Check environment variables
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].env}'

# Exec into workspace to check (if pod running)
kubectl exec <pod-name> -c step-<step-name> -- ls -la /workspace

Workspace Issues

Symptoms:

Tasks can’t access workspaces
Permission denied errors
Data not shared between tasks

Debug:

# Check workspace bindings
kubectl get pipelinerun <name> -o jsonpath='{.spec.workspaces}'
kubectl get taskrun <name> -o jsonpath='{.spec.workspaces}'

# Check PVC access mode
kubectl get pvc <pvc-name> -o jsonpath='{.spec.accessModes}'

# Check if PVC is bound
kubectl get pvc <pvc-name>

# Check volumes in pod
kubectl get pod <pod-name> -o jsonpath='{.spec.volumes}'

# Check workspace mounts
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].volumeMounts}'

Common causes:

PVC not created or not bound
ReadWriteOnce PVC shared across nodes (parallel tasks)
Incorrect workspace name mapping
PVC size too small
Permission issues

Solutions:

# Create missing PVC
kubectl apply -f pvc.yaml

# Use volumeClaimTemplate for isolation
# (see Tekton/Pipelines workflow)

# Check PVC capacity
kubectl get pvc <pvc-name> -o jsonpath='{.status.capacity.storage}'

# Use ReadWriteMany if available
# Or ensure tasks run sequentially

Pipeline Timeout

Symptoms:

PipelineRun shows PipelineRunTimeout
Pipeline stopped before completion

Debug:

# Check timeout configuration
kubectl get pipeline <name> -o jsonpath='{.spec.timeouts}'

# Check pipelinerun duration
kubectl get pipelinerun <name> -o jsonpath='{.status.startTime}'
kubectl get pipelinerun <name> -o jsonpath='{.status.completionTime}'

# Check which task was running
tkn pipelinerun describe <name>

Solutions:

# Increase timeout in pipeline
spec:
  timeouts:
    pipeline: 2h
    tasks: 1h

# Or in pipelinerun
spec:
  timeout: 2h

Results Not Available

Symptoms:

Pipeline can’t access task results
Result referenced but empty

Debug:

# Check if result was emitted
tkn taskrun describe <taskrun-name> | grep -A 10 Results

# Get result value
kubectl get taskrun <name> -o jsonpath='{.status.taskResults}'

# Check result size (must be <4KB)
kubectl logs <pod-name> -c step-<step-name>

Common causes:

Task didn’t write to result path
Result exceeds 4KB limit
Step failed before writing result

Solutions:

# Verify result is written in task
script: |
  echo -n "value" > $(results.result-name.path)

# For large data, use workspace instead
# (see Tekton/Tasks workflow)

Trigger Not Firing

Symptoms:

Webhook received but no PipelineRun created
EventListener not responding

Debug:

# Check EventListener pod
kubectl get pods -l eventlistener=<listener-name>

# Check EventListener logs
kubectl logs -l eventlistener=<listener-name>

# Check EventListener service
kubectl get svc el-<listener-name>

# Test webhook endpoint
kubectl port-forward svc/el-<listener-name> 8080:8080
curl -X POST http://localhost:8080 -d '{}'

Common causes:

EventListener pod not running
Webhook secret mismatch
TriggerBinding extraction error
RBAC permissions missing
Interceptor filtering event

Solutions:

# Check EventListener status
kubectl describe eventlistener <listener-name>

# Check RBAC
kubectl auth can-i create pipelineruns --as=system:serviceaccount:<namespace>:<sa-name>

# Check webhook secret
kubectl get secret <webhook-secret> -o yaml

# Review interceptor filters
kubectl get eventlistener <listener-name> -o jsonpath='{.spec.triggers[*].interceptors}'

Advanced Debugging

Debug Containers (Kubernetes 1.23+)

# Add debug container to running pod
kubectl debug <pod-name> -it --image=busybox --target=step-<step-name>

# Create debug copy of pod
kubectl debug <pod-name> -it --copy-to=<debug-pod-name> --container=debug

Check Tekton Controller Logs

# Get Tekton controller logs
kubectl logs -n tekton-pipelines -l app=tekton-pipelines-controller

# Get webhook logs
kubectl logs -n tekton-pipelines -l app=tekton-pipelines-webhook

# Get triggers controller logs
kubectl logs -n tekton-pipelines -l app.kubernetes.io/component=controller

Inspect etcd for Resource Issues

# Check if resources are being pruned
kubectl get pipelineruns --all-namespaces | wc -l

# Configure pruner if needed
# (see Tekton/BestPractices workflow)

Enable Debug Logging

# Set Tekton controller to debug level
kubectl edit configmap config-logging -n tekton-pipelines

# Add:
# zap-logger-config: |
#   level: debug

Debugging Checklist

Check PipelineRun/TaskRun status with tkn CLI
View logs with tkn pipelinerun logs or tkn taskrun logs
Identify failing task in pipeline
Inspect pod status with kubectl (see Kubernetes/Debug)
Check pod events and describe output
Verify task definition and parameters
Check workspace configuration and PVC status
Validate ServiceAccount and RBAC permissions
Review resource requests and limits
Check for timeout issues
Verify results are within 4KB limit
For triggers: check EventListener logs

Integration with Kubernetes Debug

When debugging Tekton, always remember:

Tekton tasks run as Kubernetes pods
Use Kubernetes/Debug workflow for pod-level issues
Common Kubernetes issues apply: ImagePullBackOff, CrashLoopBackOff, OOMKilled, Pending
Kubernetes debugging tools work: kubectl describe, kubectl logs, kubectl exec, kubectl debug

Debugging flow:

Start with Tekton-specific tools (tkn CLI)
Drill down to Kubernetes pod debugging (kubectl)
Check Tekton controller logs for system issues
Review Tekton configuration (tasks, pipelines, triggers)

Useful Commands Reference

Quick Status

# Last pipelinerun
tkn pipelinerun describe --last

# Failed pipelineruns
tkn pipelinerun list | grep Failed

# Running pipelineruns
kubectl get pipelineruns -o wide | grep Running

Log Streaming

# Follow last pipelinerun
tkn pipelinerun logs --last -f

# All tasks in pipeline
tkn pipelinerun logs <name> --all

# With timestamps
tkn pipelinerun logs <name> -f --timestamps

Pod Inspection

# Find Tekton pods
kubectl get pods -l tekton.dev/pipelineRun

# Describe pod (see Kubernetes/Debug)
kubectl describe pod <pod-name>

# Pod resource usage
kubectl top pod <pod-name>

Cleanup Failed Runs

# Delete failed pipelineruns
tkn pipelinerun delete --all --failed -f

# Keep last N runs
tkn pipelinerun delete --keep 5 -f

Monitoring

Metrics to track:

Pipeline success rate
Pipeline duration
Task failure patterns
Resource usage (CPU, memory)
PVC capacity

Alerting on:

Repeated pipeline failures
Timeout issues
Resource exhaustion
EventListener downtime

Example Debug Session

# 1. Check what's failing
tkn pipelinerun list | head -n 5

# 2. Describe the failed run
tkn pipelinerun describe my-pipeline-run-abc123

# 3. Identify failing task
# Output shows: build-image task failed

# 4. Check task logs
tkn pipelinerun logs my-pipeline-run-abc123 -t build-image

# 5. Find the pod
kubectl get pods -l tekton.dev/taskRun=my-pipeline-run-abc123-build-image

# 6. Use Kubernetes debugging (see Kubernetes/Debug)
kubectl describe pod my-pipeline-run-abc123-build-image-pod-abc123
kubectl logs my-pipeline-run-abc123-build-image-pod-abc123 -c step-build --previous

# 7. Check workspace if needed
kubectl exec my-pipeline-run-abc123-build-image-pod-abc123 -c step-build -- ls -la /workspace/source

# 8. Review task definition
kubectl get task build-image -o yaml

# 9. Check related resources
kubectl get pvc
kubectl get serviceaccount

GitOps - GitOps Integration

Integrating Tekton with GitOps tools (ArgoCD, Flux CD) for complete CI/CD automation.

When to Use

Implementing GitOps workflows
Integrating Tekton with ArgoCD or Flux
Automating deployments via Git
Separating CI and CD concerns
Managing multi-environment deployments

GitOps Architecture

Tekton + ArgoCD Pattern

┌──────────┐    ┌────────────┐    ┌──────────────┐    ┌─────────────┐
│  GitHub  │───▶│   Tekton   │───▶│  GitOps Repo │───▶│   ArgoCD    │
│  (Code)  │    │    (CI)    │    │  (Manifests) │    │    (CD)     │
└──────────┘    └────────────┘    └──────────────┘    └─────────────┘
                     │                                        │
                     │ Build & Push                           │ Sync
                     ▼                                        ▼
               ┌────────────┐                          ┌─────────────┐
               │  Registry  │                          │  Kubernetes │
               └────────────┘                          └─────────────┘

Responsibilities:

Tekton (CI): Build, test, create container images
ArgoCD (CD): Deploy manifests to Kubernetes

Benefits:

Separation of concerns
Git as single source of truth
Declarative deployments
Automated rollback capabilities
Audit trail via Git history

Tekton CI Pipeline for GitOps

Complete CI Pipeline

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: gitops-ci-pipeline
spec:
  params:
    - name: git-url
      description: Application source repository
    - name: git-revision
      description: Git commit/branch
      default: main
    - name: image-name
      description: Container image name
    - name: gitops-repo-url
      description: GitOps repository URL
    - name: gitops-branch
      description: GitOps repository branch
      default: main
    - name: deployment-file
      description: Path to deployment file
      default: k8s/deployment.yaml

  workspaces:
    - name: source-code
    - name: gitops-repo
    - name: docker-credentials
    - name: git-credentials

  tasks:
    # 1. Clone application source
    - name: clone-source
      taskRef:
        resolver: hub
        params:
          - name: name
            value: git-clone
      workspaces:
        - name: output
          workspace: source-code
        - name: ssh-directory
          workspace: git-credentials
      params:
        - name: url
          value: $(params.git-url)
        - name: revision
          value: $(params.git-revision)

    # 2. Extract version from commit
    - name: get-version
      runAfter: [clone-source]
      taskRef:
        name: git-version
      workspaces:
        - name: source
          workspace: source-code

    # 3. Run tests
    - name: test
      runAfter: [clone-source]
      taskRef:
        name: run-tests
      workspaces:
        - name: source
          workspace: source-code

    # 4. Build and push image
    - name: build-push
      runAfter: [test, get-version]
      taskRef:
        resolver: hub
        params:
          - name: name
            value: kaniko
      workspaces:
        - name: source
          workspace: source-code
        - name: dockerconfig
          workspace: docker-credentials
      params:
        - name: IMAGE
          value: $(params.image-name):$(tasks.get-version.results.version)
        - name: EXTRA_ARGS
          value:
            - --cache=true

    # 5. Clone GitOps repository
    - name: clone-gitops
      runAfter: [build-push]
      taskRef:
        resolver: hub
        params:
          - name: name
            value: git-clone
      workspaces:
        - name: output
          workspace: gitops-repo
        - name: ssh-directory
          workspace: git-credentials
      params:
        - name: url
          value: $(params.gitops-repo-url)
        - name: revision
          value: $(params.gitops-branch)

    # 6. Update image tag in GitOps repo
    - name: update-manifest
      runAfter: [clone-gitops]
      taskRef:
        name: yq-update
      workspaces:
        - name: source
          workspace: gitops-repo
      params:
        - name: file
          value: $(params.deployment-file)
        - name: expression
          value: .spec.template.spec.containers[0].image = "$(params.image-name):$(tasks.get-version.results.version)"

    # 7. Commit and push changes
    - name: commit-push
      runAfter: [update-manifest]
      taskRef:
        name: git-cli
      workspaces:
        - name: source
          workspace: gitops-repo
        - name: ssh-directory
          workspace: git-credentials
      params:
        - name: GIT_USER_NAME
          value: "Tekton Pipeline"
        - name: GIT_USER_EMAIL
          value: "tekton@example.com"
        - name: GIT_SCRIPT
          value: |
            git add $(params.deployment-file)
            git commit -m "Update image to $(params.image-name):$(tasks.get-version.results.version)"
            git push origin $(params.gitops-branch)

Update Manifest Task

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: yq-update
spec:
  params:
    - name: file
      description: YAML file to update
    - name: expression
      description: yq expression
  workspaces:
    - name: source
  steps:
    - name: update
      image: mikefarah/yq:latest
      workingDir: $(workspaces.source.path)
      script: |
        #!/bin/sh
        yq eval -i '$(params.expression)' $(params.file)

Git Commit Task

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: git-commit-push
spec:
  params:
    - name: commit-message
    - name: file-path
  workspaces:
    - name: source
    - name: ssh-directory
  steps:
    - name: commit-push
      image: alpine/git:latest
      workingDir: $(workspaces.source.path)
      env:
        - name: GIT_SSH_COMMAND
          value: ssh -i $(workspaces.ssh-directory.path)/id_rsa -o StrictHostKeyChecking=no
      script: |
        #!/bin/sh
        set -e

        git config user.name "Tekton Pipeline"
        git config user.email "tekton@example.com"

        git add $(params.file-path)
        git commit -m "$(params.commit-message)"
        git push origin HEAD

ArgoCD Integration

ArgoCD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/gitops-repo
    targetRevision: main
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Trigger ArgoCD Sync from Tekton

# Task to trigger ArgoCD sync
apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: argocd-sync
spec:
  params:
    - name: app-name
      description: ArgoCD application name
    - name: argocd-server
      description: ArgoCD server URL
      default: argocd-server.argocd.svc:443
  steps:
    - name: sync
      image: argoproj/argocd:latest
      script: |
        #!/bin/sh
        argocd app sync $(params.app-name) \
          --server $(params.argocd-server) \
          --auth-token $ARGOCD_TOKEN
      env:
        - name: ARGOCD_TOKEN
          valueFrom:
            secretKeyRef:
              name: argocd-token
              key: token

Flux CD Integration

Flux ImageUpdateAutomation

Flux automatically updates images:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: myapp
  namespace: flux-system
spec:
  image: myregistry.com/myapp
  interval: 1m
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: myapp
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: myapp
  policy:
    semver:
      range: 1.x.x
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: gitops-repo
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: flux@example.com
        name: Flux
      messageTemplate: "Update image to {{range .Updated.Images}}{{println .}}{{end}}"
    push:
      branch: main
  update:
    path: ./k8s
    strategy: Setters

Tekton with Flux

Simpler Tekton pipeline:

# Tekton only builds and pushes
# Flux automatically detects new image and updates manifests
spec:
  tasks:
    - name: clone
    - name: test
    - name: build-push
  # No manifest update needed - Flux handles it

Multi-Environment GitOps

Directory Structure

gitops-repo/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   └── production/
│       ├── kustomization.yaml
│       └── patches/

Kustomize Update Task

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: kustomize-set-image
spec:
  params:
    - name: image
    - name: overlay
      default: dev
  workspaces:
    - name: source
  steps:
    - name: set-image
      image: k8s.gcr.io/kustomize/kustomize:v5.0.0
      workingDir: $(workspaces.source.path)/overlays/$(params.overlay)
      script: |
        kustomize edit set image app=$(params.image)

Environment-Specific Pipelines

# Dev pipeline - auto-deploy
- name: update-dev
  when:
    - input: $(params.git-branch)
      operator: in
      values: [develop]
  taskRef:
    name: update-gitops
  params:
    - name: overlay
      value: dev

# Staging pipeline - auto-deploy
- name: update-staging
  when:
    - input: $(params.git-branch)
      operator: in
      values: [main]
  taskRef:
    name: update-gitops
  params:
    - name: overlay
      value: staging

# Production - manual approval required
- name: update-production
  when:
    - input: $(params.deploy-to-prod)
      operator: in
      values: ["true"]
  taskRef:
    name: update-gitops
  params:
    - name: overlay
      value: production

Git Credentials Management

SSH Key Secret

# Generate SSH key
ssh-keygen -t ed25519 -C "tekton@example.com" -f tekton_key -N ""

# Create secret
kubectl create secret generic git-ssh-credentials \
  --from-file=id_rsa=tekton_key \
  --from-file=id_rsa.pub=tekton_key.pub \
  --from-file=known_hosts=~/.ssh/known_hosts

# Add public key to GitHub/GitLab as deploy key with write access

Using SSH Credentials

workspaces:
  - name: ssh-directory
    secret:
      secretName: git-ssh-credentials
      items:
        - key: id_rsa
          path: id_rsa
          mode: 0600
        - key: known_hosts
          path: known_hosts

Best Practices

Repository Structure

Separate application and GitOps repositories
Use Kustomize or Helm for multi-environment management
Version GitOps manifests
Protect main branch with pull requests

CI Pipeline

Build and test in Tekton
Push images with semantic versioning
Update GitOps repo via automation
Sign commits for traceability

CD with ArgoCD/Flux

Enable auto-sync for development
Require manual approval for production
Configure sync windows for production
Set up health checks and rollback

Security

Use SSH keys with write-only access to GitOps repo
Rotate credentials regularly
Scan container images before pushing
Use signed commits

Monitoring

Monitor ArgoCD sync status
Alert on sync failures
Track deployment frequency
Measure lead time for changes

Complete GitOps Example

# PipelineRun with all parameters
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: gitops-ci-run-
spec:
  pipelineRef:
    name: gitops-ci-pipeline
  params:
    - name: git-url
      value: https://github.com/org/myapp
    - name: git-revision
      value: main
    - name: image-name
      value: myregistry.com/myapp
    - name: gitops-repo-url
      value: git@github.com:org/gitops-repo.git
    - name: gitops-branch
      value: main
    - name: deployment-file
      value: overlays/production/kustomization.yaml
  workspaces:
    - name: source-code
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
    - name: gitops-repo
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 500Mi
    - name: docker-credentials
      secret:
        secretName: docker-config
    - name: git-credentials
      secret:
        secretName: git-ssh-credentials
  serviceAccountName: gitops-pipeline-sa

Troubleshooting

Manifest Update Fails

# Check workspace contents
kubectl exec <pod-name> -- ls /workspace/gitops-repo

# Verify yq/kustomize syntax
kubectl logs <pod-name> -c step-update

Git Push Fails

# Check SSH credentials
kubectl get secret git-ssh-credentials -o yaml

# Test SSH connection
kubectl run test-git --rm -i --tty --image=alpine/git -- sh
apk add openssh-client
ssh -T git@github.com

ArgoCD Not Syncing

# Check ArgoCD application
kubectl get application myapp -n argocd

# View sync status
argocd app get myapp

# Manual sync
argocd app sync myapp

Wrong Image Tag

# Check task results
kubectl get pipelinerun <name> -o jsonpath='{.status.taskRuns}'

# Verify manifest update
kubectl logs <pod-name> -c step-update

Advanced Tekton Workflows

BestPractices - Tekton Best Practices

When to Use

Security Best Practices

Workspace-Based Authentication (2024 Recommendation)

ServiceAccount and RBAC

Pod Security

Secrets Management

Performance Best Practices

Resource Management

Configure Pipeline Pruner

Workspace Optimization

Caching Strategies

Reliability Best Practices

Error Handling

Finally Tasks for Cleanup

Idempotent Tasks

Maintainability Best Practices

Task Reusability

Use Tekton Hub

Migration from Deprecated Features

Documentation

Versioning

Anti-Patterns to Avoid

1. Not Setting Resource Limits

2. Using :latest Image Tag

3. Monolithic Tasks

4. Ignoring Pipeline Pruner

5. Sharing ReadWriteOnce PVC in Parallel

6. Results Over 4KB

7. Not Using Tekton Hub

8. Overly Permissive RBAC

9. Secrets in Logs

10. Not Testing Tasks Isolation

Operating at Scale (Red Hat Lessons)

1. Configure Pipeline Pruner

2. Set Resource Requests/Limits

3. Understand Workload Bottlenecks

4. Avoid PVC Sharing Conflicts

5. Monitor etcd Performance

6. Plan for Multi-AZ Clusters

7. Use Tekton Resolvers

8. Implement Retry Strategies

9. Version Control Everything

10. Test at Scale

Monitoring and Observability

Tekton Dashboard

Prometheus Metrics

Logging Best Practices

Alerting

Checklist

Security Checklist

Performance Checklist

Reliability Checklist

Maintainability Checklist

Debug - Debugging Tekton Pipelines

When to Use

Overview

Quick Diagnosis

Check PipelineRun/TaskRun Status

View Logs

Debugging Workflow

1. Identify the Problem

2. Find the Failing Task

3. Examine Task Logs

4. Inspect the Pod (Kubernetes Debug)

5. Check Task Definition

6. Verify Workspaces

7. Check Results and Parameters

Common Tekton Issues

Task Not Starting (Pending)

Task Failing with ImagePullBackOff

Task Failing with CrashLoopBackOff

Workspace Issues

Pipeline Timeout

Results Not Available

Trigger Not Firing

Advanced Debugging

Debug Containers (Kubernetes 1.23+)

Check Tekton Controller Logs