The DevOps Team Meeting That Changed Everything:

“We need to provision a DynamoDB table for our new microservice.”

“Okay, create a Terraform module, get it reviewed, apply it, then update your Kubernetes deployment with the table ARN…”

“Can’t we just… declare it in Kubernetes?”

“Well, technically yes, but you’d need to install the ACK controller, configure IRSA, manage the Helm chart, monitor the controller pods, handle upgrades—”

“This is 2026. Why are we still doing this manually?”

Good question.

It was November 2025 at Altimetrik, and we’d just heard about EKS Capabilities—a game-changing feature that moved from GA in December 2025. The promise? AWS manages your Kubernetes controllers for you. No more Helm charts. No more controller pods eating cluster resources. No more IRSA headaches. No more manual upgrades.

We were skeptical. But after implementing it across our multi-region EKS platform serving 10+ engineering teams, we discovered something remarkable: EKS Capabilities doesn’t just eliminate operational overhead—it enables platform engineering at scale.

What we built in 4 weeks:

• ✅ Zero controller pods to manage (AWS runs them)
• ✅ Self-service platform APIs for developers (WebApp, API, BackgroundWorker)
• ✅ AWS resources managed through kubectl (DynamoDB, SQS, S3)
• ✅ 200+ applications deployed using platform templates
• ✅ 87% reduction in infrastructure code per application
• ✅ Developer velocity: 2 days → 30 minutes to deploy new services

This is the complete story of how we transformed our Kubernetes platform using EKS Capabilities with ACK (AWS Controllers for Kubernetes) and KRO (Kubernetes Resource Orchestrator), including the architecture, implementation, RBAC gotchas, and real production learnings.

Table of Contents

1. The Problem: Controller Management Overhead
2. What Are EKS Capabilities?
3. Architecture: EKS Capabilities at Altimetrik
4. Implementation Part 1: Infrastructure with Terraform
5. Implementation Part 2: Enable ACK and KRO
6. Implementation Part 3: Managing AWS Resources with kubectl
7. Implementation Part 4: Building Platform APIs with KRO
8. The RBAC Gotcha (The Part That Cost Us Hours)
9. Real-World Platform Templates
10. Production Results and Impact
11. Lessons Learned

The Problem: Controller Management Overhead

What We Had Before EKS Capabilities

Our Kubernetes platform (October 2024):

Controllers Running in Cluster:
├── AWS Load Balancer Controller (2 replicas)
├── External DNS Controller (2 replicas)
├── cert-manager (3 replicas)
├── ACK SQS Controller (2 replicas)
├── ACK DynamoDB Controller (2 replicas)
├── ACK S3 Controller (2 replicas)
├── ACK RDS Controller (2 replicas)
├── Cluster Autoscaler (2 replicas)
└── Metrics Server (2 replicas)

Total controller pods: 18
Total CPU reserved: 9 vCPUs
Total memory reserved: 18 GB
Cost: ~$280/month just for controllers

The operational burden:

• Helm chart management: 9 different charts to track
• Version upgrades: Quarterly upgrade cycle for all controllers
• IRSA configuration: IAM roles for each controller
• Monitoring: Separate dashboards for each controller
• Debugging: When something breaks, which controller is at fault?
• Resource overhead: Controllers consuming cluster capacity

The breaking point:

One Friday, the ACK DynamoDB controller crashed due to a memory leak. We didn’t notice for 6 hours. During that time, developers created 12 DynamoDB table CRDs that stayed in “Pending” state. When we finally fixed the controller, all 12 tables provisioned simultaneously, hitting AWS API rate limits and causing a cascading failure.

We needed a better way.

graph TB
    subgraph Terraform["Terraform - Infrastructure Provisioning"]
        TF["Terraform<br/>Infrastructure as Code"]
    end
    
    subgraph AWS["AWS Account (us-east-1)"]
        subgraph EKS["EKS Cluster: Eks-Capabilities"]
            subgraph ControlPlane["EKS Control Plane"]
                API["Kubernetes API<br/>EKS v1.34"]
            end
            
            subgraph Capabilities["AWS-Managed Capabilities (Zero Cluster Overhead)"]
                KRO["KRO Controller<br/>Runs on AWS Infrastructure<br/>NOT in your cluster"]
                ACK["ACK Controller<br/>Runs on AWS Infrastructure<br/>NOT in your cluster"]
            end
            
            subgraph RBAC["RBAC Configuration"]
                ClusterRole["ClusterRole<br/>kro-resource-manager"]
                Binding["ClusterRoleBinding<br/>Grants KRO permissions"]
            end
            
            subgraph NodeGroup["Managed Node Group (t3.medium)"]
                Node1["Node 1"]
                Node2["Node 2"]
                
                subgraph Pods["Application Pods ONLY"]
                    Pod1["orders-app<br/>Pod 1"]
                    Pod2["orders-app<br/>Pod 2"]
                end
            end
        end
        
        subgraph AWSServices["AWS Resources Created by ACK"]
            DynamoDB["DynamoDB Table<br/>Eks-Dev-orders"]
            SQS["SQS Queue<br/>Eks-Dev-notifications"]
        end
        
        IAM["IAM Role<br/>Eks-Capabilities-capabilities-role<br/>Assumed by KRO & ACK"]
    end
    
    subgraph Developer["Developer Experience"]
        Dev["👨‍💻 Developer"]
        YAML["webapp.yaml<br/>13 lines<br/>Single manifest"]
    end
    
    TF -->|Provisions| AWS
    TF -->|Creates| EKS
    TF -->|Creates| IAM
    
    Dev -->|kubectl apply| YAML
    YAML -->|Creates| API
    
    API --> KRO
    KRO -->|Creates| Pod1
    KRO -->|Creates| Pod2
    KRO -->|Creates via ACK| ACK
    
    ACK -->|Provisions| DynamoDB
    ACK -->|Provisions| SQS
    
    IAM -.->|Assumed by| KRO
    IAM -.->|Assumed by| ACK
    
    ClusterRole -->|Authorizes| KRO
    Binding -->|Grants| KRO
    
    Pod1 -->|Uses| DynamoDB
    Pod2 -->|Uses| SQS
    
    style Terraform fill:#e6f3ff
    style Capabilities fill:#99ff99
    style NodeGroup fill:#ffcc99
    style AWSServices fill:#ffccff
    style Developer fill:#ffffcc
    style KRO fill:#66ff66
    style ACK fill:#66ff66

What Are EKS Capabilities?

The Paradigm Shift

Traditional approach:

You install ACK controller into your cluster
↓
You manage Helm chart, IRSA, upgrades, monitoring
↓
Controller runs in your cluster (consuming resources)
↓
Controller manages AWS resources

EKS Capabilities approach:

AWS runs controllers in their infrastructure
↓
You enable with single API call
↓
AWS handles scaling, patching, upgrading
↓
Controller manages AWS resources (same outcome)

Think of it like:

• Self-managed = Running your own database on EC2
• EKS Capabilities = Using Amazon RDS

Same functionality. Zero ops burden.

graph TB
    subgraph Before["Before EKS Capabilities"]
        B1["Write Deployment YAML<br/>45 lines"]
        B2["Write Service YAML<br/>20 lines"]
        B3["Write ConfigMap YAML<br/>15 lines"]
        B4["Write Terraform for DynamoDB<br/>30 lines"]
        B5["Write Terraform for SQS<br/>25 lines"]
        B6["Write Terraform for IAM<br/>40 lines"]
        B7["Apply Terraform<br/>terraform apply"]
        B8["Wait for AWS resources<br/>5-10 minutes"]
        B9["Update K8s manifests<br/>with ARNs"]
        B10["kubectl apply<br/>Deploy to cluster"]
        B11["Install & manage<br/>ACK controllers<br/>Helm charts, IRSA"]
        
        B1 --> B2 --> B3 --> B4 --> B5 --> B6
        B6 --> B7 --> B8 --> B9 --> B10
        B11 -.->|"Required for"| B4
        
        BTime["⏱️ Total Time: 2 days<br/>📝 Total Code: 225 lines<br/>🔧 Maintenance: 8 hrs/week"]
    end
    
    subgraph After["After EKS Capabilities"]
        A1["Write WebApp YAML<br/>13 lines<br/>(includes everything!)"]
        A2["kubectl apply<br/>or<br/>git push (ArgoCD)"]
        A3["KRO decomposes<br/>ACK provisions<br/>Automated"]
        A4["Done!<br/>All resources ready"]
        
        A1 --> A2 --> A3 --> A4
        
        ATime["⏱️ Total Time: 30 minutes<br/>📝 Total Code: 13 lines<br/>🔧 Maintenance: 0 hrs/week<br/>✅ AWS manages controllers"]
    end
    
    Before -.->|"Migrate"| After
    
    style Before fill:#ffcccc
    style After fill:#ccffcc
    style BTime fill:#ff9999
    style ATime fill:#99ff99

The Three Capabilities We Use

Cost Model:

• Base: $0.10/hour per capability ($72/month)
• Usage: Based on API calls to AWS services
• Our cost: ~$150/month for ACK + KRO
• Savings vs self-managed: $130/month + zero operational burden

Architecture: EKS Capabilities in our project

The Complete Architecture

Here’s how everything fits together in our production environment:

graph TB
    subgraph Terraform["Terraform - Infrastructure Provisioning"]
        TF["Terraform<br/>Infrastructure as Code"]
    end
    
    subgraph AWS["AWS Account (us-east-1)"]
        subgraph EKS["EKS Cluster: Eks-Capabilities"]
            subgraph ControlPlane["EKS Control Plane"]
                API["Kubernetes API<br/>EKS v1.34"]
            end
            
            subgraph Capabilities["AWS-Managed Capabilities (Zero Cluster Overhead)"]
                KRO["KRO Controller<br/>Runs on AWS Infrastructure<br/>NOT in your cluster"]
                ACK["ACK Controller<br/>Runs on AWS Infrastructure<br/>NOT in your cluster"]
            end
            
            subgraph RBAC["RBAC Configuration"]
                ClusterRole["ClusterRole<br/>kro-resource-manager"]
                Binding["ClusterRoleBinding<br/>Grants KRO permissions"]
            end
            
            subgraph NodeGroup["Managed Node Group (t3.medium)"]
                Node1["Node 1"]
                Node2["Node 2"]
                
                subgraph Pods["Application Pods ONLY"]
                    Pod1["orders-app<br/>Pod 1"]
                    Pod2["orders-app<br/>Pod 2"]
                end
            end
        end
        
        subgraph AWSServices["AWS Resources Created by ACK"]
            DynamoDB["DynamoDB Table<br/>Eks-Dev-orders"]
            SQS["SQS Queue<br/>Eks-Dev-notifications"]
        end
        
        IAM["IAM Role<br/>Eks-Capabilities-capabilities-role<br/>Assumed by KRO & ACK"]
    end
    
    subgraph Developer["Developer Experience"]
        Dev["👨‍💻 Developer"]
        YAML["webapp.yaml<br/>13 lines<br/>Single manifest"]
    end
    
    TF -->|Provisions| AWS
    TF -->|Creates| EKS
    TF -->|Creates| IAM
    
    Dev -->|kubectl apply| YAML
    YAML -->|Creates| API
    
    API --> KRO
    KRO -->|Creates| Pod1
    KRO -->|Creates| Pod2
    KRO -->|Creates via ACK| ACK
    
    ACK -->|Provisions| DynamoDB
    ACK -->|Provisions| SQS
    
    IAM -.->|Assumed by| KRO
    IAM -.->|Assumed by| ACK
    
    ClusterRole -->|Authorizes| KRO
    Binding -->|Grants| KRO
    
    Pod1 -->|Uses| DynamoDB
    Pod2 -->|Uses| SQS
    
    style Terraform fill:#e6f3ff
    style Capabilities fill:#99ff99
    style NodeGroup fill:#ffcc99
    style AWSServices fill:#ffccff
    style Developer fill:#ffffcc
    style KRO fill:#66ff66
    style ACK fill:#66ff66

┌─────────────────────────────────────────────────────┐
│              Terraform (Infrastructure as Code)      │
│                                                      │
│  Provisions:                                         │
│  • AWS Account and VPC                              │
│  • EKS Cluster (version 1.34)                       │
│  • IAM Roles for Capabilities                       │
│  • RBAC ClusterRole                                 │
└─────────────────┬───────────────────────────────────┘
                  │
                  ↓
┌─────────────────────────────────────────────────────┐
│          Amazon EKS Cluster (Eks-Capabilities)      │
│                                                      │
│  ┌────────────────────────────────────────────┐    │
│  │  AWS-Managed Capabilities (Outside Cluster) │    │
│  │  ┌──────────────┐  ┌──────────────┐       │    │
│  │  │ KRO Controller│  │ ACK Controller│       │    │
│  │  │  (AWS-run)   │  │  (AWS-run)   │       │    │
│  │  └──────┬───────┘  └──────┬───────┘       │    │
│  └─────────┼──────────────────┼────────────────┘    │
│            │                  │                     │
│            ↓                  ↓                     │
│  ┌─────────────────────────────────────────┐       │
│  │   Kubernetes Resources Created by KRO   │       │
│  │   • Deployments (orders-app)            │       │
│  │   • Services (ClusterIP)                │       │
│  │   • Pods (2 replicas)                   │       │
│  └─────────────────────────────────────────┘       │
│            │                                        │
│            ↓                                        │
│  ┌─────────────────────────────────────────┐       │
│  │   Managed Node Group (t3.medium)        │       │
│  │   • Min: 2 nodes                        │       │
│  │   • Max: 4 nodes                        │       │
│  │   • Desired: 2 nodes                    │       │
│  └─────────────────────────────────────────┘       │
└─────────────────────────────────────────────────────┘
                  │
                  ↓
┌─────────────────────────────────────────────────────┐
│      AWS Resources Created by ACK Controller        │
│                                                      │
│  ┌─────────────────┐  ┌──────────────────┐         │
│  │ DynamoDB Table  │  │  SQS Queue       │         │
│  │ Eks-Dev-orders  │  │  Eks-Dev-notif   │         │
│  └─────────────────┘  └──────────────────┘         │
└─────────────────────────────────────────────────────┘

Key Points:

• Controllers run outside your cluster (AWS-managed infrastructure)
• Zero controller pods consuming your node resources
• Terraform provisions the EKS cluster and IAM roles
• kubectl manages everything (Kubernetes and AWS resources)
• Developers use platform APIs (WebApp, not Deployment+Service+Queue)

Implementation Part 1: Infrastructure with Terraform

Terraform Module for EKS with Capabilities

# main.tf
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "Eks-Capabilities"
  cluster_version = "1.34"  # Required for Capabilities

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  # Public endpoint for kubectl access
  cluster_endpoint_public_access = true
  
  # Enable cluster creator admin permissions
  enable_cluster_creator_admin_permissions = true

  # Managed node group
  eks_managed_node_groups = {
    main = {
      min_size       = 2
      max_size       = 4
      desired_size   = 2
      instance_types = ["t3.medium"]
      
      # Labels for workload placement
      labels = {
        role = "application"
      }
      
      tags = {
        Environment = "production"
        ManagedBy   = "terraform"
      }
    }
  }

  # Cluster logging
  cluster_enabled_log_types = [
    "api",
    "audit",
    "authenticator",
    "controllerManager",
    "scheduler"
  ]

  tags = {
    Environment = "production"
    Project     = "eks-capabilities"
    ManagedBy   = "terraform"
  }
}

# VPC Module
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "eks-capabilities-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
  enable_dns_hostnames = true
  enable_dns_support   = true

  # Tags for EKS
  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }

  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

IAM Role for EKS Capabilities

# iam-capabilities-role.tf
# This role is assumed by AWS-managed controllers
resource "aws_iam_role" "eks_capabilities" {
  name = "Eks-Capabilities-capabilities-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "capabilities.eks.amazonaws.com"
      }
      Action = ["sts:AssumeRole", "sts:TagSession"]
    }]
  })

  tags = {
    Name        = "EKS Capabilities Role"
    Environment = "production"
  }
}

# Inline policy for ACK controllers
resource "aws_iam_role_policy" "eks_capabilities_policy" {
  name = "eks-capabilities-policy"
  role = aws_iam_role.eks_capabilities.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      # DynamoDB permissions
      {
        Effect = "Allow"
        Action = [
          "dynamodb:CreateTable",
          "dynamodb:DescribeTable",
          "dynamodb:DeleteTable",
          "dynamodb:UpdateTable",
          "dynamodb:TagResource",
          "dynamodb:UntagResource",
          "dynamodb:ListTagsOfResource"
        ]
        Resource = "*"
      },
      # SQS permissions
      {
        Effect = "Allow"
        Action = [
          "sqs:CreateQueue",
          "sqs:DeleteQueue",
          "sqs:GetQueueAttributes",
          "sqs:SetQueueAttributes",
          "sqs:TagQueue",
          "sqs:UntagQueue",
          "sqs:ListQueueTags"
        ]
        Resource = "*"
      },
      # S3 permissions
      {
        Effect = "Allow"
        Action = [
          "s3:CreateBucket",
          "s3:DeleteBucket",
          "s3:GetBucketLocation",
          "s3:ListBucket",
          "s3:PutBucketTagging",
          "s3:GetBucketTagging"
        ]
        Resource = "*"
      }
    ]
  })
}

# Output the role ARN for capability creation
output "eks_capabilities_role_arn" {
  value       = aws_iam_role.eks_capabilities.arn
  description = "ARN of the IAM role for EKS Capabilities"
}

Deploy Infrastructure

terraform init
terraform plan
terraform apply

# Configure kubectl
aws eks update-kubeconfig \
  --region us-east-1 \
  --name Eks-Capabilities

Implementation Part 2: Enable ACK and KRO

Enable Capabilities via AWS CLI

Get your account and role information:

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
CLUSTER_NAME="Eks-Capabilities"
REGION="us-east-1"
ROLE_ARN="arn:aws:iam::${ACCOUNT_ID}:role/Eks-Capabilities-capabilities-role"

echo "Account ID: $ACCOUNT_ID"
echo "Role ARN: $ROLE_ARN"

Enable ACK (AWS Controllers for Kubernetes):

aws eks create-capability \
  --region $REGION \
  --cluster-name $CLUSTER_NAME \
  --capability-name ack \
  --type ACK \
  --role-arn $ROLE_ARN \
  --delete-propagation-policy RETAIN

# Output:
# {
#     "capability": {
#         "name": "ack",
#         "type": "ACK",
#         "status": "CREATING",
#         ...
#     }
# }

Enable KRO (Kubernetes Resource Orchestrator):

aws eks create-capability \
  --region $REGION \
  --cluster-name $CLUSTER_NAME \
  --capability-name kro \
  --type KRO \
  --role-arn $ROLE_ARN \
  --delete-propagation-policy RETAIN

Check status (wait ~2 minutes):

# Check ACK status
aws eks describe-capability \
  --region $REGION \
  --cluster-name $CLUSTER_NAME \
  --capability-name ack

# Check KRO status
aws eks describe-capability \
  --region $REGION \
  --cluster-name $CLUSTER_NAME \
  --capability-name kro

# Both should show: "status": "ACTIVE"

sequenceDiagram
    participant Dev as Developer
    participant Kubectl as kubectl
    participant API as K8s API Server
    participant KRO as KRO Controller<br/>(AWS-managed)
    participant ACK as ACK Controller<br/>(AWS-managed)
    participant K8s as Kubernetes<br/>Cluster
    participant AWS as AWS Services
    
    Note over Dev,AWS: Complete Flow - 13 Lines of YAML → Full Stack
    
    rect rgb(230, 245, 255)
        Note over Dev,API: Phase 1: Developer Applies WebApp
        Dev->>Kubectl: kubectl apply -f webapp.yaml
        Note over Dev: WebApp manifest:<br/>13 lines defining app
        Kubectl->>API: Create WebApp resource
        API->>API: Validate WebApp CRD
        API->>KRO: Notify: New WebApp created
    end
    
    rect rgb(255, 240, 230)
        Note over KRO: Phase 2: KRO Decomposes WebApp
        KRO->>KRO: Read ResourceGraphDefinition
        KRO->>KRO: Extract template variables:<br/>• appName: orders-app<br/>• replicas: 2<br/>• image: nginx:1.27
        KRO->>KRO: Generate child resources:<br/>1. Deployment<br/>2. Service<br/>3. Queue (ACK CRD)
    end
    
    rect rgb(240, 255, 240)
        Note over KRO,K8s: Phase 3: KRO Creates K8s Resources
        KRO->>API: Create Deployment
        API->>K8s: Schedule Deployment
        K8s->>K8s: Create 2 pods
        
        KRO->>API: Create Service
        API->>K8s: Configure Service
        K8s->>K8s: ClusterIP assigned
    end
    
    rect rgb(255, 245, 230)
        Note over KRO,ACK: Phase 4: KRO Creates ACK Resource
        KRO->>API: Create Queue (ACK CRD)
        API->>ACK: Notify: New Queue CRD
        ACK->>ACK: Read Queue spec:<br/>queueName: Eks-Dev-notifications
    end
    
    rect rgb(245, 240, 255)
        Note over ACK,AWS: Phase 5: ACK Provisions AWS Resource
        ACK->>AWS: CreateQueue API call
        AWS->>AWS: Provision SQS queue
        AWS->>ACK: Queue ARN returned
        ACK->>API: Update Queue status:<br/>synced: true<br/>queueURL: https://...
    end
    
    rect rgb(230, 255, 255)
        Note over KRO,Dev: Phase 6: WebApp Status Updated
        API->>KRO: All child resources ready
        KRO->>API: Update WebApp status:<br/>State: ACTIVE<br/>Conditions: Ready=True
        API->>Kubectl: WebApp ready
        Kubectl->>Dev: ✅ Deployment complete!
    end
    
    Note over Dev,AWS: Total time: < 60 seconds<br/>Resources created: 4 (Deployment, Service, Pods, SQS)<br/>Developer effort: 13 lines of YAML

That’s it. No Helm installations. No controller pods. AWS is running these controllers in their managed infrastructure.

Verify Capabilities are Active

# List all capabilities
aws eks list-capabilities \
  --region $REGION \
  --cluster-name $CLUSTER_NAME

# Check CRDs installed by ACK
kubectl get crd | grep -E "(dynamodb|sqs|s3).services.k8s.aws"

# Check CRDs installed by KRO
kubectl get crd | grep kro.run

Expected CRDs:

# ACK CRDs
tables.dynamodb.services.k8s.aws
queues.sqs.services.k8s.aws
buckets.s3.services.k8s.aws

# KRO CRDs
resourcegraphdefinitions.kro.run

Implementation Part 3: Managing AWS Resources with kubectl

Creating DynamoDB Tables with kubectl

File: aws-resources/dynamodb-table.yaml

apiVersion: dynamodb.services.k8s.aws/v1alpha1
kind: Table
metadata:
  name: app-orders-table
  namespace: default
spec:
  tableName: Eks-Dev-orders
  attributeDefinitions:
    - attributeName: orderId
      attributeType: S
    - attributeName: customerId
      attributeType: S
    - attributeName: orderTimestamp
      attributeType: N
  keySchema:
    - attributeName: orderId
      keyType: HASH
    - attributeName: customerId
      keyType: RANGE
  globalSecondaryIndexes:
    - indexName: CustomerIndex
      keySchema:
        - attributeName: customerId
          keyType: HASH
        - attributeName: orderTimestamp
          keyType: RANGE
      projection:
        projectionType: ALL
  billingMode: PAY_PER_REQUEST
  tags:
    - key: Environment
      value: production
    - key: ManagedBy
      value: eks-capabilities
    - key: Team
      value: backend

Apply with kubectl:

kubectl apply -f aws-resources/dynamodb-table.yaml

# Check status
kubectl get table app-orders-table

# Detailed info
kubectl describe table app-orders-table

Within 30-60 seconds:

kubectl get table
NAME                 SYNCED   AGE
app-orders-table     True     45s

Verify in AWS Console: The table Eks-Dev-orders now exists in DynamoDB!

Creating SQS Queues with kubectl

File: aws-resources/sqs-queue.yaml

apiVersion: sqs.services.k8s.aws/v1alpha1
kind: Queue
metadata:
  name: app-notifications-queue
  namespace: default
spec:
  queueName: Eks-Dev-notifications
  attributes:
    VisibilityTimeout: "30"
    MessageRetentionPeriod: "345600"  # 4 days
    ReceiveMessageWaitTimeSeconds: "10"
    DelaySeconds: "0"
  tags:
    Environment: production
    ManagedBy: eks-capabilities
    Team: platform

kubectl apply -f aws-resources/sqs-queue.yaml

# Check status
kubectl get queue app-notifications-queue

# Get queue URL
kubectl get queue app-notifications-queue \
  -o jsonpath='{.status.queueURL}'

The Magic: Kubernetes Reconciliation for AWS Resources

What happens if you delete the queue in AWS Console?

# Delete queue in AWS Console manually
# ACK detects drift within 30 seconds
# ACK recreates the queue automatically

Check the events:

kubectl describe queue app-notifications-queue

# Events:
# Normal  Created  30s   ack-controller  Created SQS queue
# Normal  Synced   15s   ack-controller  Queue configuration synced

This is the power of Kubernetes reconciliation applied to cloud infrastructure.

Implementation Part 4: Building Platform APIs with KRO

The Platform Engineering Vision

As a platform team, we don’t want developers to write:

• Deployment YAML
• Service YAML
• DynamoDB Table YAML
• SQS Queue YAML
• IAM policy YAML
• ConfigMap YAML

We want them to write:

apiVersion: platform.altimetrik.com/v1alpha1
kind: WebApp
metadata:
  name: my-awesome-app
spec:
  image: my-app:v1.0.0
  replicas: 3

And everything else gets created automatically.

That’s what KRO enables.

graph LR
    subgraph DevExperience["Developer Experience - Simple APIs"]
        Dev["👨‍💻 Developer"]
        
        Simple["Single YAML File<br/>13 lines"]
        
        WebApp["WebApp API<br/>Hides complexity"]
        API["API Template<br/>+ Database"]
        Worker["BackgroundWorker<br/>+ Queue"]
    end
    
    subgraph KRO["KRO Orchestration Layer"]
        RGD1["ResourceGraphDefinition<br/>WebApp Template"]
        RGD2["ResourceGraphDefinition<br/>API Template"]
        RGD3["ResourceGraphDefinition<br/>Worker Template"]
    end
    
    subgraph Resources["Resources Created Automatically"]
        subgraph K8sRes["Kubernetes Resources"]
            Deploy["Deployments"]
            Svc["Services"]
            Config["ConfigMaps"]
        end
        
        subgraph AWSRes["AWS Resources (via ACK)"]
            DB["DynamoDB Tables"]
            Queue["SQS Queues"]
            Bucket["S3 Buckets"]
        end
    end
    
    Dev --> Simple
    Simple --> WebApp
    Simple --> API
    Simple --> Worker
    
    WebApp --> RGD1
    API --> RGD2
    Worker --> RGD3
    
    RGD1 --> Deploy
    RGD1 --> Svc
    RGD1 --> Queue
    
    RGD2 --> Deploy
    RGD2 --> Svc
    RGD2 --> DB
    
    RGD3 --> Deploy
    RGD3 --> Queue
    RGD3 --> Config
    
    style DevExperience fill:#ccffcc
    style KRO fill:#e6f3ff
    style K8sRes fill:#ffcc99
    style AWSRes fill:#ffccff
    style WebApp fill:#99ff99
    style API fill:#99ff99
    style Worker fill:#99ff99

Creating a WebApp Platform API

File: platform-apis/webapp-resourcegraph.yaml

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: webapp
spec:
  # Define the custom API schema
  schema:
    apiVersion: v1alpha1
    kind: WebApp
    spec:
      # User-provided fields
      appName: string
      image: string
      replicas: integer
      serviceName: string
      queueName: string
      
      # Optional fields with defaults
      containerPort: 
        type: integer
        default: 80
      servicePort:
        type: integer
        default: 80

  # Define the resources KRO will create
  resources:
    # Resource 1: Kubernetes Deployment
    - id: deployment
      template:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: ${schema.spec.appName}
          labels:
            app: ${schema.spec.appName}
            managed-by: kro
        spec:
          replicas: ${schema.spec.replicas}
          selector:
            matchLabels:
              app: ${schema.spec.appName}
          template:
            metadata:
              labels:
                app: ${schema.spec.appName}
            spec:
              containers:
                - name: app
                  image: ${schema.spec.image}
                  ports:
                    - containerPort: ${schema.spec.containerPort}
                  resources:
                    requests:
                      cpu: 250m
                      memory: 256Mi
                    limits:
                      cpu: 500m
                      memory: 512Mi
                  livenessProbe:
                    httpGet:
                      path: /healthz
                      port: ${schema.spec.containerPort}
                    initialDelaySeconds: 30
                    periodSeconds: 10
                  readinessProbe:
                    httpGet:
                      path: /ready
                      port: ${schema.spec.containerPort}
                    initialDelaySeconds: 5
                    periodSeconds: 5

    # Resource 2: Kubernetes Service
    - id: service
      template:
        apiVersion: v1
        kind: Service
        metadata:
          name: ${schema.spec.serviceName}
          labels:
            app: ${schema.spec.appName}
            managed-by: kro
        spec:
          selector:
            app: ${schema.spec.appName}
          ports:
            - port: ${schema.spec.servicePort}
              targetPort: ${schema.spec.containerPort}
              protocol: TCP
          type: ClusterIP

    # Resource 3: AWS SQS Queue (via ACK)
    - id: queue
      template:
        apiVersion: sqs.services.k8s.aws/v1alpha1
        kind: Queue
        metadata:
          name: ${schema.spec.appName}-queue
          labels:
            app: ${schema.spec.appName}
            managed-by: kro
        spec:
          queueName: ${schema.spec.queueName}
          attributes:
            VisibilityTimeout: "30"
            MessageRetentionPeriod: "345600"
            ReceiveMessageWaitTimeSeconds: "10"
          tags:
            Application: ${schema.spec.appName}
            ManagedBy: kro

Apply the ResourceGraphDefinition:

kubectl apply -f platform-apis/webapp-resourcegraph.yaml

# Verify the new API is registered
kubectl get resourcegraphdefinition webapp

# Check the CRD created
kubectl get crd webapps.platform.altimetrik.com

KRO automatically:

1. Registers WebApp as a Kubernetes resource
2. Creates the CRD
3. Watches for WebApp instances
4. Decomposes them into child resources
5. Manages lifecycle (create, update, delete)

The RBAC Gotcha (The Part That Cost Us Hours)

The Problem We Hit

After creating the ResourceGraphDefinition, we tried to deploy a WebApp:

graph TB
    subgraph Problem["The RBAC Problem"]
        KROIdentity["KRO Controller Identity<br/>arn:aws:sts::ACCOUNT:assumed-role/<br/>Eks-Capabilities-capabilities-role/KRO"]
        
        Policies["EKS Access Policies<br/>✅ AmazonEKSKROPolicy<br/>(Manages WebApp CRDs)<br/><br/>❌ NO permissions for<br/>child resources!"]
        
        Error["Error when creating child resources:<br/>User cannot create resource<br/>'deployments' in API group 'apps'"]
    end
    
    subgraph Solution["The RBAC Solution"]
        ClusterRole["ClusterRole:<br/>kro-resource-manager<br/><br/>Grants permissions for:<br/>• apps/deployments<br/>• core/services<br/>• sqs.services.k8s.aws/queues<br/>• dynamodb.services.k8s.aws/tables"]
        
        Binding["ClusterRoleBinding<br/>Binds ClusterRole to<br/>KRO's STS identity"]
        
        Success["✅ KRO can now create:<br/>• Deployments<br/>• Services<br/>• ACK resources<br/>• ConfigMaps<br/>• Secrets"]
    end
    
    KROIdentity --> Policies
    Policies --> Error
    
    Error -.->|"Fix with"| ClusterRole
    ClusterRole --> Binding
    Binding --> Success
    
    style Problem fill:#ffcccc
    style Solution fill:#ccffcc
    style Error fill:#ff6666
    style Success fill:#66ff66

apiVersion: platform.altimetrik.com/v1alpha1
kind: WebApp
metadata:
  name: orders-app
spec:
  appName: orders-app
  image: nginx:1.27
  replicas: 2
  serviceName: orders-app-svc
  queueName: Eks-Dev-notifications

kubectl apply -f orders-app.yaml

# WebApp created successfully
kubectl get webapp orders-app
NAME          STATE      AGE
orders-app    PENDING    2m

But nothing happened. No Deployment. No Service. No Queue.

kubectl describe webapp orders-app

# Events:
# Warning  ReconcileError  Failed to create deployment: 
#          User "arn:aws:sts::XXXXX:assumed-role/Eks-Capabilities-capabilities-role/KRO" 
#          cannot create resource "deployments" in API group "apps"

The Root Cause

The capabilities IAM role gets EKS access entries with these policies:

• AmazonEKSACKPolicy — manages ACK custom resources
• AmazonEKSKROPolicy — manages KRO’s CRDs (ResourceGraphDefinitions, WebApp instances)

But neither policy grants KRO permission to create the child resources (Deployments, Services, SQS Queues) that the WebApp template defines.

The Solution: RBAC for KRO

File: rbac/kro-clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kro-resource-manager
rules:
  # Deployments
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  
  # Services
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  
  # ConfigMaps (if needed)
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  
  # Secrets (if needed)
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  
  # ACK SQS Queues
  - apiGroups: ["sqs.services.k8s.aws"]
    resources: ["queues"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  
  # ACK DynamoDB Tables
  - apiGroups: ["dynamodb.services.k8s.aws"]
    resources: ["tables"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  
  # ACK S3 Buckets
  - apiGroups: ["s3.services.k8s.aws"]
    resources: ["buckets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kro-resource-manager-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kro-resource-manager
subjects:
  # KRO's identity in Kubernetes
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: "arn:aws:sts::<ACCOUNT_ID>:assumed-role/Eks-Capabilities-capabilities-role/KRO"

Critical detail: KRO’s Kubernetes identity is the STS assumed-role ARN with /KRO appended.

Find the exact identity:

# List EKS access entries
aws eks list-access-entries \
  --cluster-name Eks-Capabilities \
  --region us-east-1

# Describe the capabilities role entry
aws eks describe-access-entry \
  --cluster-name Eks-Capabilities \
  --principal-arn "arn:aws:iam::$ACCOUNT_ID:role/Eks-Capabilities-capabilities-role" \
  --region us-east-1

Apply the RBAC:

# Replace <ACCOUNT_ID> with your actual account ID
sed -i "s/<ACCOUNT_ID>/$ACCOUNT_ID/g" rbac/kro-clusterrole.yaml

kubectl apply -f rbac/kro-clusterrole.yaml

Now retry the WebApp:

kubectl delete webapp orders-app
kubectl apply -f orders-app.yaml

# Within seconds:
kubectl get webapp orders-app
NAME          STATE     AGE
orders-app    ACTIVE    15s

Success! All child resources created.

Real-World Platform Templates

Template 1: WebApp (Full Stack Application)

What it creates:

• Kubernetes Deployment
• Kubernetes Service
• AWS SQS Queue (for async processing)

Developer usage:

apiVersion: platform.altimetrik.com/v1alpha1
kind: WebApp
metadata:
  name: payment-processor
  namespace: production
spec:
  appName: payment-processor
  image: docker.altimetrik.com/payment-processor:v2.1.0
  replicas: 5
  serviceName: payment-svc
  queueName: Eks-Prod-payment-events
  containerPort: 8080
  servicePort: 80

kubectl apply -f payment-processor.yaml

# Everything created in < 60 seconds

Template 2: BackgroundWorker (Queue Consumer)

File: platform-apis/worker-resourcegraph.yaml

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: backgroundworker
spec:
  schema:
    apiVersion: v1alpha1
    kind: BackgroundWorker
    spec:
      appName: string
      image: string
      replicas: integer
      sourceQueue: string  # Existing queue to consume from
      deadLetterQueue: string

  resources:
    # Deployment for worker pods
    - id: deployment
      template:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: ${schema.spec.appName}
        spec:
          replicas: ${schema.spec.replicas}
          selector:
            matchLabels:
              app: ${schema.spec.appName}
              type: worker
          template:
            metadata:
              labels:
                app: ${schema.spec.appName}
                type: worker
            spec:
              containers:
                - name: worker
                  image: ${schema.spec.image}
                  env:
                    - name: SOURCE_QUEUE
                      value: ${schema.spec.sourceQueue}
                    - name: DLQ
                      value: ${schema.spec.deadLetterQueue}
                  resources:
                    requests:
                      cpu: 500m
                      memory: 512Mi
                    limits:
                      cpu: 1000m
                      memory: 1Gi

    # Dead Letter Queue
    - id: dlq
      template:
        apiVersion: sqs.services.k8s.aws/v1alpha1
        kind: Queue
        metadata:
          name: ${schema.spec.appName}-dlq
        spec:
          queueName: ${schema.spec.deadLetterQueue}
          attributes:
            MessageRetentionPeriod: "1209600"  # 14 days

Developer usage:

apiVersion: platform.altimetrik.com/v1alpha1
kind: BackgroundWorker
metadata:
  name: email-sender
spec:
  appName: email-sender
  image: docker.altimetrik.com/email-worker:v1.0.0
  replicas: 3
  sourceQueue: Eks-Prod-email-queue
  deadLetterQueue: Eks-Prod-email-dlq

Template 3: API (HTTP API with Database)

File: platform-apis/api-resourcegraph.yaml

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: api
spec:
  schema:
    apiVersion: v1alpha1
    kind: API
    spec:
      appName: string
      image: string
      replicas: integer
      hostname: string
      databaseTable: string

  resources:
    # Deployment
    - id: deployment
      template:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: ${schema.spec.appName}
        spec:
          replicas: ${schema.spec.replicas}
          selector:
            matchLabels:
              app: ${schema.spec.appName}
          template:
            metadata:
              labels:
                app: ${schema.spec.appName}
            spec:
              containers:
                - name: api
                  image: ${schema.spec.image}
                  ports:
                    - containerPort: 8080
                  env:
                    - name: DATABASE_TABLE
                      value: ${schema.spec.databaseTable}

    # Service
    - id: service
      template:
        apiVersion: v1
        kind: Service
        metadata:
          name: ${schema.spec.appName}-svc
        spec:
          selector:
            app: ${schema.spec.appName}
          ports:
            - port: 80
              targetPort: 8080

    # HTTPRoute (Gateway API)
    - id: route
      template:
        apiVersion: gateway.networking.k8s.io/v1
        kind: HTTPRoute
        metadata:
          name: ${schema.spec.appName}-route
        spec:
          parentRefs:
            - name: production-gateway
              namespace: gateway-system
          hostnames:
            - ${schema.spec.hostname}
          rules:
            - matches:
                - path:
                    type: PathPrefix
                    value: /
              backendRefs:
                - name: ${schema.spec.appName}-svc
                  port: 80

    # DynamoDB Table
    - id: database
      template:
        apiVersion: dynamodb.services.k8s.aws/v1alpha1
        kind: Table
        metadata:
          name: ${schema.spec.appName}-table
        spec:
          tableName: ${schema.spec.databaseTable}
          attributeDefinitions:
            - attributeName: id
              attributeType: S
          keySchema:
            - attributeName: id
              keyType: HASH
          billingMode: PAY_PER_REQUEST

Developer usage (13 lines = complete stack):

apiVersion: platform.altimetrik.com/v1alpha1
kind: API
metadata:
  name: user-api
spec:
  appName: user-api
  image: docker.altimetrik.com/user-api:v1.0.0
  replicas: 5
  hostname: users.altimetrik.com
  databaseTable: Eks-Prod-users

One kubectl apply creates:

• Deployment (5 replicas)
• Service (ClusterIP)
• HTTPRoute (public access via Gateway API)
• DynamoDB Table (AWS resource)

Total YAML: 13 lines Resources created: 4 Time: < 60 seconds

Production Results and Impact

The Transformation at Altimetrik

Before EKS Capabilities (October 2024):

Per Application Deployment:
├── deployment.yaml (45 lines)
├── service.yaml (20 lines)
├── configmap.yaml (15 lines)
├── terraform/dynamodb.tf (30 lines)
├── terraform/sqs.tf (25 lines)
├── terraform/iam.tf (40 lines)
└── helm/ack-controller/values.yaml (50 lines)

Total YAML: 225 lines per application
Controllers to manage: 7 (ACK × 4, cert-manager, ExternalDNS, etc.)
Controller pods: 14 (2 replicas each)
Deployment time: 2 days
Developer dependency: High (need DevOps for AWS resources)

After EKS Capabilities (February 2025):

Per Application Deployment:
└── webapp.yaml (13 lines)

Total YAML: 13 lines per application
Controllers to manage: 0 (AWS-managed)
Controller pods: 0
Deployment time: 30 minutes
Developer dependency: Zero (self-service)

Reduction: 94% less configuration code

Metrics After 3 Months

The Platform APIs We Built

Total: 226 applications deployed using platform templates

graph TB
    subgraph Templates["5 Platform API Templates"]
        T1["WebApp<br/>Deployment + Service + SQS<br/>120 apps using"]
        T2["BackgroundWorker<br/>Deployment + DLQ<br/>45 apps using"]
        T3["API<br/>Deployment + Service + HTTPRoute + DynamoDB<br/>35 apps using"]
        T4["CronJob<br/>CronJob + ConfigMap<br/>18 apps using"]
        T5["StatefulApp<br/>StatefulSet + Service + S3<br/>8 apps using"]
    end
    
    subgraph Teams["10+ Engineering Teams"]
        Team1["Backend Team<br/>45 apps"]
        Team2["Frontend Team<br/>30 apps"]
        Team3["Data Team<br/>25 apps"]
        Team4["Mobile API<br/>20 apps"]
        TeamN["... 6 more teams<br/>106 apps"]
    end
    
    subgraph Results["Platform Results"]
        R1["226 Total Apps<br/>Deployed"]
        R2["94% Less Code<br/>per App"]
        R3["95% Faster<br/>Deployments"]
        R4["Zero Controller<br/>Management"]
    end
    
    T1 --> Team1
    T2 --> Team2
    T3 --> Team3
    T4 --> Team4
    T5 --> TeamN
    
    Team1 --> R1
    Team2 --> R2
    Team3 --> R3
    TeamN --> R4
    
    style Templates fill:#e6f3ff
    style Teams fill:#ccffcc
    style Results fill:#ffffcc

Developer Feedback

“I deployed a complete application stack in 13 lines of YAML. It used to take me 2 days and 7 different files. This is incredible.” - Backend Developer

“No more waiting for DevOps to provision DynamoDB tables. I just declare it in my WebApp and it’s created automatically.” - Full-stack Developer

“The platform team gave us an API (BackgroundWorker) instead of making us Kubernetes experts. Game changer.” - Data Engineer

Cost Impact

Controller infrastructure savings:

• Before: 14 controller pods × 500m CPU × $0.05/vCPU-hour = $252/month
• After: $0 (AWS-managed)
• Capability fees: ACK ($72/mo) + KRO ($72/mo) = $144/month
• Net savings: $108/month

Engineering time savings:

• Controller management: 8 hours/week → 0
• IRSA configuration: 4 hours/week → 0
• Helm upgrades: 6 hours/month → 0
• Value: ~$18,000/year in engineering time

Developer productivity:

• Faster deployments = faster feature delivery
• Self-service = less waiting
• Standardized templates = fewer mistakes

ROI: Overwhelmingly positive

Lessons Learned and Best Practices

Lesson 1: RBAC is Critical

The mistake: Assuming EKS Capabilities policies grant all necessary permissions.

The reality: KRO needs explicit RBAC for child resource management.

The fix:

# Always create RBAC after enabling KRO
kubectl apply -f rbac/kro-clusterrole.yaml

# Verify KRO can create resources
kubectl auth can-i create deployments \
  --as "arn:aws:sts::$ACCOUNT_ID:assumed-role/Eks-Capabilities-capabilities-role/KRO"

Lesson 2: Kubernetes Naming Rules Apply

The gotcha: AWS resource names support mixed case (Eks-Dev-orders), but Kubernetes metadata.name must be lowercase RFC 1123.

The solution:

# In ResourceGraphDefinition
resources:
  - id: queue
    template:
      apiVersion: sqs.services.k8s.aws/v1alpha1
      kind: Queue
      metadata:
        # Kubernetes name (must be lowercase)
        name: ${schema.spec.appName}-queue
      spec:
        # AWS queue name (supports mixed case)
        queueName: ${schema.spec.queueName}

Best practice: Use appName (lowercase) for Kubernetes resources, allow queueName for AWS resources.

Lesson 3: Start Simple, Build Complex

Our progression:

1. Week 1: Simple WebApp (Deployment + Service)
2. Week 2: Add SQS Queue (via ACK)
3. Week 3: Add DynamoDB Table
4. Week 4: Add HTTPRoute (Gateway API integration)
5. Week 5: Build BackgroundWorker template
6. Week 6: Build API template

Don’t try to build everything at once.

Lesson 4: Version Your Platform APIs

# ResourceGraphDefinition versioning
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: webapp-v2  # Version in name
spec:
  schema:
    apiVersion: v2alpha1  # Version in API
    kind: WebApp
    # ...

Why: Allows gradual migration when you improve templates.

# Old apps continue using v1
apiVersion: platform.altimetrik.com/v1alpha1
kind: WebApp

# New apps use v2
apiVersion: platform.altimetrik.com/v2alpha1
kind: WebApp

Lesson 5: Monitor Capability Health

CloudWatch metrics:

# ACK controller metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/EKS \
  --metric-name CapabilityAPICallCount \
  --dimensions Name=CapabilityName,Value=ack \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Sum

Prometheus alerts:

groups:
- name: eks-capabilities
  rules:
  - alert: KROReconciliationFailed
    expr: kro_reconciliation_errors_total > 10
    for: 10m
    annotations:
      summary: "KRO failing to reconcile resources"

  - alert: ACKResourceNotSynced
    expr: ack_resource_synced{synced="false"} > 5
    for: 15m
    annotations:
      summary: "ACK resources not syncing with AWS"

Lesson 6: Platform Documentation is Essential

We created comprehensive docs:

# Platform API Documentation

## WebApp

Creates a complete web application stack.

### Usage:
```yaml
apiVersion: platform.altimetrik.com/v1alpha1
kind: WebApp
metadata:
  name: my-app
spec:
  appName: my-app        # Required: lowercase alphanumeric
  image: my-image:tag    # Required: container image
  replicas: 3            # Required: number of pods
  serviceName: my-svc    # Required: service name
  queueName: My-Queue    # Required: AWS SQS queue name

What Gets Created:

• Kubernetes Deployment (with health checks)
• Kubernetes Service (ClusterIP)
• AWS SQS Queue (in us-east-1)

Examples:

See examples/webapp/ directory

**Developer adoption increased 3x after good documentation.**

### Lesson 7: Capabilities Don't Replace Everything

**What we still self-manage:**
- AWS Load Balancer Controller (for Gateway API)
- External DNS (Route53 integration)
- Karpenter (node autoscaling)

**Why?** These don't have EKS Capability versions yet.

**Best practice:** Use Capabilities where available, self-manage the rest.

### Lesson 8: GitOps + Platform APIs = Magic

**Our ArgoCD integration:**

```yaml
# argocd/applications/orders-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-app
  namespace: argocd
spec:
  project: production
  
  source:
    repoURL: https://bitbucket.org/abcd-company/orders-app.git
    targetRevision: main
    path: k8s
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Developer workflow:

1. Create k8s/webapp.yaml in their Bitbucket repo
2. Commit and push
3. ArgoCD detects change
4. ArgoCD applies WebApp
5. KRO creates all child resources
6. ACK provisions AWS infrastructure
7. Application running in < 2 minutes

Zero kubectl commands. Pure GitOps.

sequenceDiagram
    participant Dev as Developer
    participant BB as Bitbucket Repo
    participant ArgoCD as ArgoCD
    participant API as K8s API
    participant KRO as KRO (AWS-managed)
    participant ACK as ACK (AWS-managed)
    participant AWS as AWS Services
    
    Note over Dev,AWS: GitOps Workflow - Zero kubectl Commands
    
    rect rgb(230, 245, 255)
        Note over Dev,BB: Step 1: Developer Commits
        Dev->>Dev: Create k8s/webapp.yaml<br/>13 lines
        Dev->>BB: git add k8s/webapp.yaml
        Dev->>BB: git commit -m "Add orders app"
        Dev->>BB: git push origin main
    end
    
    rect rgb(255, 240, 230)
        Note over BB,ArgoCD: Step 2: ArgoCD Detects Change
        BB->>ArgoCD: Webhook: New commit
        ArgoCD->>BB: Pull latest manifests
        ArgoCD->>ArgoCD: Compare desired vs current
        Note over ArgoCD: Change detected:<br/>New WebApp resource
    end
    
    rect rgb(240, 255, 240)
        Note over ArgoCD,API: Step 3: ArgoCD Syncs
        ArgoCD->>API: kubectl apply webapp.yaml
        API->>API: Validate WebApp CRD
        API->>KRO: Notify: New WebApp
    end
    
    rect rgb(255, 245, 230)
        Note over KRO,ACK: Step 4: KRO Orchestrates
        KRO->>KRO: Decompose WebApp into:<br/>• Deployment<br/>• Service<br/>• Queue
        KRO->>API: Create Deployment
        KRO->>API: Create Service
        KRO->>API: Create Queue (ACK CRD)
        API->>ACK: Notify: New Queue
    end
    
    rect rgb(245, 240, 255)
        Note over ACK,AWS: Step 5: ACK Provisions AWS
        ACK->>AWS: CreateQueue API
        AWS->>AWS: Provision SQS queue
        AWS->>ACK: Queue created
        ACK->>API: Update status: synced=true
    end
    
    rect rgb(230, 255, 255)
        Note over ArgoCD,Dev: Step 6: Completion
        API->>ArgoCD: All resources healthy
        ArgoCD->>ArgoCD: Sync complete
        ArgoCD->>Dev: Slack notification:<br/>"✅ orders-app deployed"
    end
    
    Note over Dev,AWS: Total time: ~2 minutes<br/>Developer effort: Git commit only<br/>kubectl commands: 0

Conclusion: The Future of Platform Engineering

Four months ago, we:

• Managed 14 controller pods consuming 7 vCPUs and 14 GB RAM
• Spent 8+ hours/week on controller maintenance
• Required 225 lines of YAML per application
• Had 2-day deployment cycles
• Developers depended on DevOps for AWS resources

Today, we:

• Manage zero controllers (AWS does it for us)
• Spend < 1 hour/week on platform maintenance
• Require 13 lines of YAML per application
• Have 30-minute deployment cycles
• Developers self-serve via platform APIs

The transformation metrics:

• 94% reduction in configuration code
• 100% reduction in controller management
• 95% faster deployments
• 46% cost savings on infrastructure
• Unlimited scalability (AWS manages capacity)

The key insights:

1. EKS Capabilities eliminates controller overhead - No more Helm charts, IRSA configuration, or controller monitoring
2. ACK makes AWS Kubernetes-native - DynamoDB and SQS become kubectl get table and kubectl get queue
3. KRO enables platform engineering - Build custom APIs that hide complexity
4. RBAC is the gotcha - KRO needs explicit permissions for child resources
5. GitOps + Capabilities + Platform APIs - The perfect platform stack

EKS Capabilities isn’t just about reducing operational burden—it’s about fundamentally rethinking how we build developer platforms.

Instead of asking developers to become Kubernetes and AWS experts, we give them high-level platform APIs that hide complexity while enforcing best practices. KRO orchestrates Kubernetes resources. ACK provisions AWS infrastructure. ArgoCD automates deployments.

The result? Developers ship features, not YAML.

Resources

Official Documentation:

• EKS Capabilities Documentation
• AWS Controllers for Kubernetes (ACK)
• Kubernetes Resource Orchestrator (KRO)

Reference Implementation:

• EKS Capabilities Example - Original implementation by Asma Elalfy

My Production Infrastructure:

• EKS Platform on GitHub - Production EKS setup at Altimetrik
• Jenkins on EKS - Production Jenkins

Further Reading:

• Platform Engineering Best Practices
• Building Internal Developer Platforms

About the Author: I’m a Senior DevOps and Cloud Engineer with 11+ years of experience building production Kubernetes platforms. Currently at Altimetrik India, I led our migration from NGINX Ingress to Gateway API across 200+ applications serving 10+ engineering teams ahead of the March 2026 NGINX Ingress retirement deadline. This work reduced configuration complexity by 60% while enabling advanced traffic management capabilities through GitOps automation with ArgoCD and Bitbucket. I also manage multi-region Kubernetes clusters on AWS with 99.99% SLA uptime. All infrastructure code is available on my GitHub. Connect with me on LinkedIn.

Questions about Gateway API, migration strategies, or the NGINX Ingress retirement? Drop a comment below or reach out on LinkedIn. I’d love to hear about your networking challenges and migration plans!

The Announcement That Changed Our Roadmap:

It was a Monday morning when I saw the GitHub issue that made my coffee taste bitter:

NGINX Ingress Controller End of Life: March 2026

The NGINX Ingress Controller project will reach end of maintenance in March 2026. After this date, no further updates, security patches, or bug fixes will be released.

Users are encouraged to migrate to alternative ingress solutions or the Kubernetes Gateway API.

My heart sank. We had 200+ applications running on NGINX Ingress across our multi-region EKS clusters at Altimetrik. Serving 10+ engineering teams. Processing millions of requests daily. And now we had 15 months to find an alternative.

But this wasn’t just about avoiding end-of-life. We’d already been hitting NGINX Ingress limitations:

Our Tuesday morning standup, two weeks earlier:

Frontend Team Lead: “We need canary deployments with weighted traffic splitting. Can NGINX Ingress do that?”

Me: “Well… sort of. We’d need to create separate Ingress resources, configure specific annotations, and—”

Team Lead: “What about gRPC routing? Traffic mirroring? Header-based routing?”

Me: “That’s… different annotations. Custom Lua scripts. ConfigMap snippets…”

The NGINX retirement announcement wasn’t a crisis—it was an opportunity. Time to migrate to something better: Kubernetes Gateway API.

graph TB
    subgraph NGINXWay["NGINX Ingress Approach"]
        NApp1["App Team 1<br/>Creates Ingress"]
        NApp2["App Team 2<br/>Creates Ingress"]
        NApp3["App Team 3<br/>Creates Ingress"]
        
        NAnnotations["❌ Annotation Hell<br/>30+ annotations per Ingress"]
        NCanary["❌ Canary = 2 Ingress<br/>Complex management"]
        NLimited["❌ HTTP/HTTPS only<br/>Limited protocols"]
        
        NController["NGINX Ingress<br/>Controller"]
        NLB["Classic LB<br/>Per cluster<br/>$18/mo × 3"]
    end
    
    subgraph GatewayWay["Gateway API Approach"]
        GClass["GatewayClass<br/>(Platform Team)"]
        GGateway["Gateway<br/>(Platform Team)<br/>Shared Infrastructure"]
        
        GApp1["App Team 1<br/>HTTPRoute"]
        GApp2["App Team 2<br/>HTTPRoute"]
        GApp3["App Team 3<br/>HTTPRoute"]
        
        GBenefits["✅ Native Resources<br/>No annotations"]
        GCanary["✅ Canary = 1 Resource<br/>Built-in weights"]
        GMulti["✅ HTTP, gRPC, TCP<br/>Multi-protocol"]
        
        GALB["AWS ALB<br/>Shared<br/>$23/mo total"]
    end
    
    NApp1 --> NController
    NApp2 --> NController
    NApp3 --> NController
    NController --> NLB
    
    NApp1 -.-> NAnnotations
    NApp2 -.-> NCanary
    NApp3 -.-> NLimited
    
    GClass --> GGateway
    GGateway --> GApp1
    GGateway --> GApp2
    GGateway --> GApp3
    
    GApp1 -.-> GBenefits
    GApp2 -.-> GCanary
    GApp3 -.-> GMulti
    
    GGateway --> GALB
    
    style NGINXWay fill:#ffcccc
    style GatewayWay fill:#ccffcc
    style NAnnotations fill:#ff9999
    style NCanary fill:#ff9999
    style NLimited fill:#ff9999
    style GBenefits fill:#99ff99
    style GCanary fill:#99ff99
    style GMulti fill:#99ff99

The Migration Results (4 months later):

• ✅ Migrated 200+ Ingress resources to Gateway API before March 2026 deadline
• ✅ Eliminated 90% of custom annotations
• ✅ Enabled advanced traffic management (canary, mirroring, header routing)
• ✅ Achieved multi-protocol support (HTTP, HTTPS, gRPC, TCP)
• ✅ Reduced configuration complexity by 60%
• ✅ GitOps-driven deployment via ArgoCD and Bitbucket
• ✅ Future-proofed our platform for the next decade

This is the complete story of our migration from NGINX Ingress to Gateway API on AWS EKS at Altimetrik, including all the YAML files, ArgoCD automation, migration strategies, gotchas, and lessons learned.

Table of Contents

1. Why Gateway API? The NGINX Ingress Sunset
2. Understanding Gateway API: Not Just Another Ingress
3. Architecture: How Gateway API Works in EKS
4. Installation: GitOps Approach with ArgoCD
5. Migration Strategy: From Ingress to Gateway
6. Real-World Examples: Before and After
7. Advanced Features We Couldn’t Do Before
8. Production Learnings and Best Practices
9. Should You Migrate?

Why Gateway API? The NGINX Ingress Sunset

The End-of-Life Announcement

March 2026 is the deadline. After that:

• ❌ No security patches
• ❌ No bug fixes
• ❌ No Kubernetes version compatibility updates
• ❌ No community support

For a production platform serving millions of requests, running unmaintained software isn’t an option. We had 15 months to migrate.

But the retirement wasn’t our only motivation.

The Pain Points We Hit with NGINX Ingress

Problem 1: Annotation Hell

Every advanced feature required vendor-specific annotations:

# NGINX Ingress - annotation overload
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    # Basic routing
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    
    # SSL config
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    
    # Rate limiting
    nginx.ingress.kubernetes.io/limit-rps: "100"
    nginx.ingress.kubernetes.io/limit-connections: "10"
    
    # Timeout config
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
    
    # CORS
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    
    # Canary deployment (requires separate Ingress!)
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
    
    # Auth
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    
    # Custom snippets (scary!)
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Custom-Header: value";
      if ($request_uri ~* "^/old-path") {
        return 301 https://$host/new-path;
      }

We had Ingress resources with 30+ annotations. Nobody understood what they all did. Changing one thing broke another.

Problem 2: Limited Traffic Management

Want canary deployments? You need to create two separate Ingress resources:

# Main Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-main
spec:
  rules:
  - host: api.altimetrik.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: api-v1
            port: 80
---
# Canary Ingress (separate resource!)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"  # 10% traffic
spec:
  rules:
  - host: api.altimetrik.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: api-v2  # New version
            port: 80

This is confusing. Two Ingresses for one application. Which one is “real”? What happens if you delete one?

Problem 3: Protocol Limitations

NGINX Ingress is HTTP/HTTPS first. Want gRPC with advanced routing? TCP traffic? WebSocket with custom headers?

Good luck. You’ll need ConfigMaps, snippets, and prayers.

Problem 4: No Role-Based Configuration

At Altimetrik:

• Platform team manages infrastructure (load balancers, TLS certs)
• Application teams manage routing rules

With NGINX Ingress, everything is in one Ingress resource. Platform team changes TLS config? Might accidentally break app routing. App team changes routing? Might break TLS.

No separation of concerns.

The Decision: Migrate to Gateway API

Why Gateway API?

1. Future-proof - Official Kubernetes SIG-Network project (not retiring!)
2. Standardized - No vendor-specific annotations
3. Role-based - Platform and app teams work independently
4. Multi-protocol - HTTP, gRPC, TCP—first-class citizens
5. Advanced routing - Built-in canary, mirroring, header routing
6. Extensible - Policies attach cleanly to resources

Plus, we beat the March 2026 deadline by 2 years.

Understanding Gateway API: Not Just Another Ingress

graph LR
    subgraph PlatformTeam["Platform Team Responsibilities"]
        PT1["GatewayClass<br/>Infrastructure Template"]
        PT2["Gateway<br/>Load Balancer Config<br/>Listeners<br/>TLS Certificates"]
        PT3["AWS Resources<br/>ALB, Security Groups<br/>Target Groups"]
        PT4["Monitoring<br/>Metrics, Alerts<br/>SLOs"]
    end
    
    subgraph AppTeam["Application Team Responsibilities"]
        AT1["HTTPRoute<br/>Routing Rules<br/>Hostnames<br/>Paths"]
        AT2["Backend Configuration<br/>Service References<br/>Weights<br/>Filters"]
        AT3["Traffic Management<br/>Canary Splits<br/>Header Routing<br/>Retries"]
        AT4["Application Services<br/>Deployments<br/>Pods"]
    end
    
    subgraph SharedBoundary["Shared Gateway"]
        Gateway["Gateway Resource<br/>Managed by Platform<br/>Referenced by Apps"]
    end
    
    PT1 --> PT2
    PT2 --> PT3
    PT3 --> PT4
    
    PT2 --> Gateway
    Gateway --> AT1
    
    AT1 --> AT2
    AT2 --> AT3
    AT3 --> AT4
    
    Gateway -.->|"Platform controls<br/>infrastructure"| PT2
    Gateway -.->|"Apps control<br/>routing"| AT1
    
    style PlatformTeam fill:#ffcc99
    style AppTeam fill:#ccffcc
    style SharedBoundary fill:#cce5ff

The Core Concepts

Gateway API introduces 3 main resources:

┌─────────────────────────────────────────────┐
│         GatewayClass                         │
│  (Platform Team - Infrastructure Template)  │
│                                             │
│  Defines: Controller, Parameters            │
└─────────────────┬───────────────────────────┘
                  │
                  ↓
┌─────────────────────────────────────────────┐
│            Gateway                           │
│    (Platform Team - Load Balancer)          │
│                                             │
│  Defines: Listeners, TLS, Addresses         │
└─────────────────┬───────────────────────────┘
                  │
                  ↓
┌─────────────────────────────────────────────┐
│         HTTPRoute / GRPCRoute               │
│      (App Team - Routing Rules)             │
│                                             │
│  Defines: Hosts, Paths, Backends            │
└─────────────────────────────────────────────┘

Think of it like:

• GatewayClass = Type of car (Tesla, BMW, Toyota)
• Gateway = Specific car instance (My Blue Tesla)
• HTTPRoute = Driving directions (Where the car goes)

Key Differences: NGINX Ingress vs Gateway API

Architecture: How Gateway API Works in EKS

graph TB
    subgraph Internet["External Access"]
        Users["👥 Users<br/>Web/Mobile/API Clients"]
    end
    
    subgraph DNS["DNS Layer"]
        R53["Route 53<br/>api.example.com<br/>app.example.com"]
        ACM["AWS Certificate Manager<br/>TLS Certificates"]
    end
    
    subgraph AWS["AWS Load Balancer"]
        ALB["Application Load Balancer<br/>Auto-provisioned by Gateway<br/>Multi-AZ<br/>WAF Enabled"]
    end
    
    subgraph EKS["Amazon EKS Cluster"]
        subgraph GatewaySystem["gateway-system namespace"]
            GClass["GatewayClass<br/>aws-application-load-balancer"]
            Gateway["Gateway<br/>production-gateway<br/>Listeners: HTTP(80), HTTPS(443)"]
            TLS["TLS Certificate<br/>cert-manager"]
        end
        
        subgraph AppNS1["production namespace"]
            Route1["HTTPRoute<br/>api-route<br/>Host: api.example.com"]
            Svc1["Service: api-v1"]
            Pods1["Pods: api-v1<br/>×10"]
        end
        
        subgraph AppNS2["staging namespace"]
            Route2["HTTPRoute<br/>staging-route<br/>Host: staging.example.com"]
            Svc2["Service: staging-app"]
            Pods2["Pods: staging-app<br/>×3"]
        end
        
        subgraph AppNS3["team-a namespace"]
            Route3["HTTPRoute<br/>app-route<br/>Host: app.example.com<br/>Canary: 90/10 split"]
            Svc3a["Service: app-v1"]
            Svc3b["Service: app-v2"]
            Pods3a["Pods: app-v1<br/>×8"]
            Pods3b["Pods: app-v2<br/>×2"]
        end
        
        Controller["AWS Load Balancer<br/>Controller<br/>Watches Gateway API"]
    end
    
    Users --> R53
    R53 --> ACM
    ACM --> ALB
    ALB --> Gateway
    
    GClass -.->|"Defines"| Gateway
    TLS -.->|"Secures"| Gateway
    
    Gateway --> Route1
    Gateway --> Route2
    Gateway --> Route3
    
    Route1 --> Svc1
    Svc1 --> Pods1
    
    Route2 --> Svc2
    Svc2 --> Pods2
    
    Route3 -->|"90%"| Svc3a
    Route3 -->|"10%"| Svc3b
    Svc3a --> Pods3a
    Svc3b --> Pods3b
    
    Controller -.->|"Manages"| ALB
    Controller -.->|"Watches"| Gateway
    
    style Internet fill:#e6f3ff
    style DNS fill:#fff4e6
    style AWS fill:#ffcc99
    style EKS fill:#e6ffe6
    style GatewaySystem fill:#ccffcc
    style AppNS1 fill:#cce5ff
    style AppNS2 fill:#ffccff
    style AppNS3 fill:#ffffcc

Our Production Architecture

┌──────────────────────────────────────────────────────┐
│                    Internet                           │
└────────────────────┬─────────────────────────────────┘
                     │
                     ↓
┌─────────────────────────────────────────────────────┐
│          AWS Application Load Balancer               │
│  (Created automatically by Gateway Controller)       │
│  - Multi-AZ                                          │
│  - WAF Integration                                   │
│  - ACM Certificate                                   │
└────────────────────┬────────────────────────────────┘
                     │
                     ↓
┌─────────────────────────────────────────────────────┐
│              Gateway Resource                        │
│  apiVersion: gateway.networking.k8s.io/v1           │
│  kind: Gateway                                       │
│  - Listeners: HTTP (80), HTTPS (443)                │
│  - TLS: cert-manager integration                    │
└────────────────────┬────────────────────────────────┘
                     │
          ┌──────────┴──────────┐
          ↓                     ↓
┌──────────────────┐   ┌───────────────────┐
│   HTTPRoute      │   │   HTTPRoute       │
│   (Team 1)       │   │   (Team 2)        │
│                  │   │                   │
│  Host: api.com   │   │  Host: web.com    │
│  Path: /v1       │   │  Path: /          │
└────────┬─────────┘   └─────────┬─────────┘
         │                       │
         ↓                       ↓
┌──────────────────┐   ┌───────────────────┐
│  Service: api-v1 │   │  Service: web-app │
│  Pods: 10        │   │  Pods: 5          │
└──────────────────┘   └───────────────────┘

Why This Architecture Wins

1. Platform Team manages Gateway (infrastructure)
2. App Teams manage HTTPRoutes (routing rules)
3. Automatic ALB provisioning by AWS Load Balancer Controller
4. Namespace Isolation - Routes in different namespaces
5. Shared Gateway - Multiple teams use same load balancer
6. GitOps-driven - Everything deployed via ArgoCD from Bitbucket

Installation: GitOps Approach with ArgoCD

At Altimetrik, we follow GitOps principles. Everything is code in Bitbucket, deployed automatically by ArgoCD.

sequenceDiagram
    participant Dev as Developer
    participant BB as Bitbucket Repository
    participant PR as Pull Request
    participant ArgoCD as ArgoCD Controller
    participant K8s as EKS Cluster
    participant Gateway as Gateway Resource
    participant ALB as AWS ALB
    participant Monitor as Monitoring
    
    Note over Dev,Monitor: GitOps-Driven Gateway API Deployment
    
    rect rgb(230, 245, 255)
        Note over Dev,BB: Phase 1: Code Changes
        Dev->>BB: 1. Clone repo locally
        Dev->>Dev: 2. Create HTTPRoute YAML
        Dev->>Dev: 3. Update configuration
        Dev->>BB: 4. Commit changes
        Dev->>BB: 5. Push to feature branch
    end
    
    rect rgb(255, 240, 230)
        Note over BB,PR: Phase 2: Review Process
        BB->>PR: 6. Create Pull Request
        PR->>PR: 7. CI validation<br/>YAML linting<br/>Policy checks
        PR->>PR: 8. Team review
        PR->>BB: 9. Merge to main branch
    end
    
    rect rgb(240, 255, 240)
        Note over ArgoCD,K8s: Phase 3: Automated Deployment
        BB->>ArgoCD: 10. Webhook: New commit detected
        ArgoCD->>BB: 11. Pull latest manifests
        ArgoCD->>ArgoCD: 12. Compare desired state<br/>vs current state
        
        alt Changes Detected
            ArgoCD->>K8s: 13. Apply HTTPRoute
            K8s->>K8s: 14. Validate HTTPRoute
            K8s->>Gateway: 15. Attach to Gateway
            Gateway->>ALB: 16. Update ALB rules
            ALB->>ALB: 17. Configure target groups
            ALB->>ArgoCD: 18. Update complete
        else No Changes
            ArgoCD->>ArgoCD: Already in sync
        end
    end
    
    rect rgb(255, 245, 230)
        Note over ArgoCD,Monitor: Phase 4: Validation
        ArgoCD->>K8s: 19. Check resource health
        K8s->>ArgoCD: Status: Programmed=True
        ArgoCD->>Monitor: 20. Report sync success
        Monitor->>Dev: 21. Slack notification:<br/>"✅ HTTPRoute deployed"
    end
    
    Note over Dev,Monitor: Total time: < 2 minutes from merge to production<br/>Zero manual kubectl commands

Our GitOps Repository Structure

bitbucket/gateway-api-infrastructure/
├── argocd-apps/
│   ├── gateway-api-crds.yaml
│   ├── aws-lb-controller.yaml
│   ├── cert-manager.yaml
│   ├── gateway-class.yaml
│   ├── production-gateway.yaml
│   └── certificates.yaml
├── gateway-api/
│   ├── crds/
│   │   └── gateway-api-crds.yaml
│   ├── aws-lb-controller/
│   │   ├── iam-policy.json
│   │   ├── service-account.yaml
│   │   └── helm-values.yaml
│   ├── cert-manager/
│   │   ├── namespace.yaml
│   │   ├── cert-manager.yaml
│   │   └── cluster-issuer.yaml
│   ├── gateway-system/
│   │   ├── namespace.yaml
│   │   ├── gatewayclass.yaml
│   │   ├── gateway.yaml
│   │   ├── certificates.yaml
│   │   └── reference-grant.yaml
│   └── examples/
│       ├── simple-httproute.yaml
│       ├── canary-httproute.yaml
│       └── grpc-route.yaml
└── README.md

Step 1: Gateway API CRDs (ArgoCD Application)

File: argocd-apps/gateway-api-crds.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gateway-api-crds
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: infrastructure
  
  source:
    repoURL: https://bitbucket.org/altimetrik/gateway-api-infrastructure.git
    targetRevision: main
    path: gateway-api/crds
  
  destination:
    server: https://kubernetes.default.svc
    namespace: gateway-system
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

File: gateway-api/crds/gateway-api-crds.yaml

# This file downloads and applies Gateway API CRDs
apiVersion: v1
kind: ConfigMap
metadata:
  name: gateway-api-installer
  namespace: gateway-system
data:
  install.sh: |
    #!/bin/bash
    kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
---
apiVersion: batch/v1
kind: Job
metadata:
  name: install-gateway-api-crds
  namespace: gateway-system
spec:
  template:
    spec:
      serviceAccountName: gateway-installer
      containers:
      - name: installer
        image: bitnami/kubectl:latest
        command: ["/bin/bash", "/scripts/install.sh"]
        volumeMounts:
        - name: scripts
          mountPath: /scripts
      volumes:
      - name: scripts
        configMap:
          name: gateway-api-installer
          defaultMode: 0755
      restartPolicy: OnFailure

Apply to ArgoCD:

kubectl apply -f argocd-apps/gateway-api-crds.yaml

Step 2: AWS Load Balancer Controller (ArgoCD Application)

File: argocd-apps/aws-lb-controller.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: aws-load-balancer-controller
  namespace: argocd
spec:
  project: infrastructure
  
  source:
    repoURL: https://bitbucket.org/altimetrik/gateway-api-infrastructure.git
    targetRevision: main
    path: gateway-api/aws-lb-controller
  
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false

File: gateway-api/aws-lb-controller/service-account.yaml

# Create service account with IAM role (IRSA)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-load-balancer-controller
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/AmazonEKSLoadBalancerControllerRole

File: gateway-api/aws-lb-controller/helm-values.yaml

# Helm values for AWS LB Controller
clusterName: altimetrik-prod-eks

serviceAccount:
  create: false
  name: aws-load-balancer-controller

# Enable Gateway API support
enableGatewayAPI: true

# High availability
replicaCount: 2

# Resource limits
resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

# Pod anti-affinity
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: aws-load-balancer-controller
        topologyKey: kubernetes.io/hostname

# Monitoring
enableMetrics: true
serviceMonitor:
  enabled: true
  namespace: monitoring

# AWS-specific settings
region: us-east-1
vpcId: vpc-xxxxxxxxx

# Feature gates
featureGates:
  - GatewayAPI=true

File: gateway-api/aws-lb-controller/helm-application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: aws-load-balancer-controller-helm
  namespace: argocd
spec:
  project: infrastructure
  
  source:
    repoURL: https://aws.github.io/eks-charts
    chart: aws-load-balancer-controller
    targetRevision: 1.7.0
    helm:
      valuesObject:
        clusterName: altimetrik-prod-eks
        serviceAccount:
          create: false
          name: aws-load-balancer-controller
        enableGatewayAPI: true
        replicaCount: 2
  
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Step 3: cert-manager (ArgoCD Application)

File: argocd-apps/cert-manager.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
spec:
  project: infrastructure
  
  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    targetRevision: v1.13.0
    helm:
      values: |
        installCRDs: true
        replicaCount: 2
        
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
        
        prometheus:
          enabled: true
          servicemonitor:
            enabled: true
        
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: cert-manager
                topologyKey: kubernetes.io/hostname
  
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

File: gateway-api/cert-manager/cluster-issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: devops@altimetrik.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    # HTTP-01 solver for Gateway API
    - http01:
        gatewayHTTPRoute:
          parentRefs:
          - name: production-gateway
            namespace: gateway-system
            kind: Gateway
    
    # DNS-01 solver for wildcard certificates
    - dns01:
        route53:
          region: us-east-1
          hostedZoneID: Z1234567890ABC

Step 4: GatewayClass and Gateway (ArgoCD Application)

File: argocd-apps/production-gateway.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-gateway
  namespace: argocd
spec:
  project: infrastructure
  
  source:
    repoURL: https://bitbucket.org/altimetrik/gateway-api-infrastructure.git
    targetRevision: main
    path: gateway-api/gateway-system
  
  destination:
    server: https://kubernetes.default.svc
    namespace: gateway-system
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

File: gateway-api/gateway-system/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: gateway-system
  labels:
    name: gateway-system
    managed-by: argocd

File: gateway-api/gateway-system/gatewayclass.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: aws-application-load-balancer
spec:
  controllerName: application-networking.k8s.aws/gateway-api-controller
  description: AWS Application Load Balancer for Gateway API at Altimetrik

File: gateway-api/gateway-system/gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
  namespace: gateway-system
  annotations:
    gateway.aws.application-networking/deploy: "true"
spec:
  gatewayClassName: aws-application-load-balancer
  
  listeners:
  # HTTP listener (redirects to HTTPS)
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All  # Allow HTTPRoutes from any namespace
  
  # HTTPS listener
  - name: https
    protocol: HTTPS
    port: 443
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: production-tls-cert
        namespace: gateway-system
  
  # gRPC listener
  - name: grpc
    protocol: HTTPS
    port: 9000
    allowedRoutes:
      kinds:
      - kind: GRPCRoute
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: production-tls-cert
        namespace: gateway-system

File: gateway-api/gateway-system/certificates.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: production-tls-cert
  namespace: gateway-system
spec:
  secretName: production-tls-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - "*.altimetrik.com"
  - altimetrik.com
  - api.altimetrik.com
  - app.altimetrik.com
  - grpc.altimetrik.com

File: gateway-api/gateway-system/reference-grant.yaml

# Allow HTTPRoutes from any namespace to reference our Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-httproutes-from-all
  namespace: gateway-system
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: "*"  # Allow from any namespace
  - group: gateway.networking.k8s.io
    kind: GRPCRoute
    namespace: "*"
  to:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: production-gateway
  - group: ""
    kind: Secret  # Allow referencing TLS secrets

Step 5: Deploy Everything with ArgoCD

Create ArgoCD App of Apps:

# app-of-apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gateway-api-platform
  namespace: argocd
spec:
  project: infrastructure
  
  source:
    repoURL: https://bitbucket.org/altimetrik/gateway-api-infrastructure.git
    targetRevision: main
    path: argocd-apps
  
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Deploy:

kubectl apply -f app-of-apps.yaml

# Watch ArgoCD deploy everything
argocd app list
argocd app get gateway-api-platform

# Check sync status
argocd app sync gateway-api-platform

ArgoCD automatically deploys in order:

1. Gateway API CRDs
2. cert-manager
3. AWS Load Balancer Controller
4. GatewayClass
5. Production Gateway
6. TLS Certificates

Verify deployment:

# Check Gateway status
kubectl get gateway -n gateway-system

# Should show:
# NAME                 CLASS                          ADDRESS                    PROGRAMMED   AGE
# production-gateway   aws-application-load-balancer  k8s-gateway-...elb.amazonaws.com   True         5m

The GitOps Advantage

Before (Manual kubectl):

• DevOps runs kubectl apply manually
• No audit trail
• Configuration drift
• Hard to replicate across environments

After (ArgoCD + Bitbucket):

• ✅ Git is source of truth
• ✅ Complete audit trail (Git commits)
• ✅ No configuration drift (auto-sync)
• ✅ Easy to replicate (same repo, different branch)
• ✅ Rollback = git revert
• ✅ Review process (Bitbucket PRs)

Migration Strategy: From Ingress to Gateway

gantt
    title NGINX Ingress to Gateway API Migration
    dateFormat  YYYY-MM-DD
    
    section Phase 1: Setup
    Install Gateway API CRDs          :done, setup1, 2024-01-01, 2d
    Install AWS LB Controller         :done, setup2, 2024-01-03, 3d
    Create GatewayClass              :done, setup3, 2024-01-06, 1d
    Create Production Gateway         :done, setup4, 2024-01-07, 2d
    Setup TLS with cert-manager      :done, setup5, 2024-01-09, 3d
    
    section Phase 2: Pilot
    Select 5 Non-Critical Apps       :done, pilot1, 2024-01-15, 2d
    Convert to HTTPRoute             :done, pilot2, 2024-01-17, 5d
    Test in Staging                  :done, pilot3, 2024-01-22, 3d
    Deploy to Production             :done, pilot4, 2024-01-25, 2d
    Document Patterns                :done, pilot5, 2024-01-27, 3d
    
    section Phase 3: Team-by-Team Migration
    Team A Migration (20 apps)       :done, team1, 2024-02-01, 1w
    Team B Migration (25 apps)       :done, team2, 2024-02-08, 1w
    Team C Migration (30 apps)       :done, team3, 2024-02-15, 1w
    Team D Migration (18 apps)       :done, team4, 2024-02-22, 1w
    Team E Migration (22 apps)       :done, team5, 2024-03-01, 1w
    Remaining Teams (85 apps)        :done, team6, 2024-03-08, 4w
    
    section Phase 4: Validation
    Monitor Both Systems             :done, valid1, 2024-02-01, 8w
    Traffic Comparison               :done, valid2, 2024-03-01, 4w
    Performance Testing              :done, valid3, 2024-03-15, 2w
    
    section Phase 5: Cleanup
    Remove NGINX Ingress Resources   :done, clean1, 2024-04-01, 2w
    Decommission NGINX Controller    :done, clean2, 2024-04-15, 1w
    Update Documentation             :done, clean3, 2024-04-22, 1w
    
    section Milestones
    50% Migrated                     :milestone, m1, 2024-02-28, 0d
    90% Migrated                     :milestone, m2, 2024-03-31, 0d
    100% Complete                    :milestone, m3, 2024-04-30, 0d

Our 4-Phase Migration Plan

Phase 1: Setup (Week 1)
├── Deploy Gateway API via ArgoCD
├── Create GatewayClass
├── Create production Gateway
└── Verify TLS certificates working

Phase 2: Pilot (Week 2-3)
├── Migrate 5 non-critical apps
├── Test in staging
├── Document patterns in Bitbucket
└── Fix issues

Phase 3: Gradual Rollout (Week 4-12)
├── Migrate by team (one team per week)
├── Run both Ingress and HTTPRoute in parallel
├── Create migration templates in repo
└── Monitor and validate

Phase 4: Cleanup (Week 13-16)
├── Remove old Ingress resources
├── Decommission NGINX Ingress
└── Update documentation

Application Team Experience: Creating HTTPRoute

Step 1: Developer creates HTTPRoute in their repo

# apps/payment-api/k8s/httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payment-api
  namespace: production
  labels:
    app: payment-api
    team: backend
spec:
  parentRefs:
  - name: production-gateway
    namespace: gateway-system
    kind: Gateway
  
  hostnames:
  - payment.altimetrik.com
  
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: payment-api
      port: 80

Step 2: Commit to Bitbucket

git add k8s/httproute.yaml
git commit -m "feat: migrate to Gateway API"
git push origin main

Step 3: ArgoCD deploys automatically

ArgoCD watches the app repo:

# ArgoCD Application for payment-api
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-api
  namespace: argocd
spec:
  source:
    repoURL: https://bitbucket.org/altimetrik/payment-api.git
    targetRevision: main
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

ArgoCD automatically:

• Detects the new HTTPRoute in Git
• Validates the YAML
• Applies to the cluster
• Reports status in ArgoCD UI

No kubectl commands needed. Pure GitOps.

Real-World Examples: Before and After

graph TB
    subgraph Before["Before - NGINX Ingress (2 Resources)"]
        BUser["User Request"]
        
        BMain["Main Ingress<br/>payment-api<br/>Host: payment.example.com"]
        BCanary["Canary Ingress<br/>payment-api-canary<br/>annotations:<br/>canary: true<br/>canary-weight: 10"]
        
        BV1["Service: payment-v1<br/>90% traffic"]
        BV2["Service: payment-v2<br/>10% traffic"]
        
        BPods1["Pods v1<br/>Stable version"]
        BPods2["Pods v2<br/>Canary version"]
        
        BNote["❌ Two Ingress resources<br/>❌ Confusing management<br/>❌ Risk of misconfiguration"]
    end
    
    subgraph After["After - Gateway API (1 Resource)"]
        AUser["User Request"]
        
        AGateway["Gateway<br/>production-gateway"]
        
        ARoute["HTTPRoute<br/>payment-canary<br/>Host: payment.example.com<br/>Weights:<br/>• payment-v1: 90<br/>• payment-v2: 10"]
        
        AV1["Service: payment-v1<br/>90% traffic"]
        AV2["Service: payment-v2<br/>10% traffic"]
        
        APods1["Pods v1<br/>Stable version"]
        APods2["Pods v2<br/>Canary version"]
        
        ANote["✅ Single HTTPRoute<br/>✅ Clear traffic split<br/>✅ Easy to adjust weights"]
    end
    
    BUser --> BMain
    BUser --> BCanary
    BMain --> BV1
    BCanary --> BV2
    BV1 --> BPods1
    BV2 --> BPods2
    
    AUser --> AGateway
    AGateway --> ARoute
    ARoute -->|"weight: 90"| AV1
    ARoute -->|"weight: 10"| AV2
    AV1 --> APods1
    AV2 --> APods2
    
    Before -.-> BNote
    After -.-> ANote
    
    style Before fill:#ffcccc
    style After fill:#ccffcc
    style BNote fill:#ff9999
    style ANote fill:#99ff99

Example 1: Simple Application (The Easy Migration)

Before (NGINX Ingress):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.altimetrik.com
    secretName: api-tls
  rules:
  - host: api.altimetrik.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

After (Gateway API):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: production
spec:
  parentRefs:
  - name: production-gateway
    namespace: gateway-system
  
  hostnames:
  - api.altimetrik.com
  
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: api-service
      port: 80

Improvements:

• ✅ No annotations needed (TLS handled at Gateway level)
• ✅ Cleaner, more readable
• ✅ Namespace isolation
• ✅ Deployed via ArgoCD from Bitbucket

Example 2: Canary Deployments (The Game Changer)

Before (NGINX Ingress) - TWO resources needed:

# apps/payment-api/k8s/ingress-main.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payment-api
  namespace: production
spec:
  rules:
  - host: payment.altimetrik.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: payment-v1
            port: 80
---
# apps/payment-api/k8s/ingress-canary.yaml (SEPARATE FILE!)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payment-api-canary
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"  # 10% to v2
spec:
  rules:
  - host: payment.altimetrik.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: payment-v2
            port: 80

After (Gateway API) - ONE resource:

# apps/payment-api/k8s/httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payment-canary
  namespace: production
  labels:
    deployment-strategy: canary
spec:
  parentRefs:
  - name: production-gateway
    namespace: gateway-system
  
  hostnames:
  - payment.altimetrik.com
  
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    
    backendRefs:
    # 90% traffic to stable version
    - name: payment-v1
      port: 80
      weight: 90
    
    # 10% traffic to canary version
    - name: payment-v2
      port: 80
      weight: 10

Git commit, ArgoCD deploys. Done.

Adjusting canary percentage?

# Just update the weight in Git
git diff
-      weight: 10
+      weight: 20

git commit -m "Increase canary to 20%"
git push

# ArgoCD syncs automatically in < 30 seconds

Improvements:

• ✅ ONE file instead of two
• ✅ Clear traffic weights
• ✅ GitOps-driven (change in Git = automatic deployment)
• ✅ Easy to adjust percentages

Example 3: Header-Based Routing

Before (NGINX Ingress):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: beta-users
  annotations:
    # Scary Lua script in annotation
    nginx.ingress.kubernetes.io/configuration-snippet: |
      set $target_backend "api-stable";
      if ($http_x_beta_user = "true") {
        set $target_backend "api-beta";
      }
      proxy_pass http://$target_backend;

After (Gateway API):

# apps/api/k8s/httproute-beta.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-beta-routing
  namespace: production
spec:
  parentRefs:
  - name: production-gateway
    namespace: gateway-system
  
  hostnames:
  - api.altimetrik.com
  
  rules:
  # Beta users go to beta backend
  - matches:
    - headers:
      - name: X-Beta-User
        value: "true"
    backendRefs:
    - name: api-beta
      port: 80
  
  # Everyone else goes to stable
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: api-stable
      port: 80

Commit, push, ArgoCD deploys. No Lua scripts.

Example 4: Complete Application Setup in Bitbucket

Here’s how a team structures their repo:

bitbucket/payment-api/
├── k8s/
│   ├── namespace.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── httproute.yaml          # Gateway API route
│   ├── configmap.yaml
│   └── secret.yaml (sealed)
├── argocd/
│   └── application.yaml        # ArgoCD Application manifest
├── Dockerfile
├── src/
└── README.md

File: argocd/application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-api
  namespace: argocd
  labels:
    team: backend
spec:
  project: applications
  
  source:
    repoURL: https://bitbucket.org/altimetrik/payment-api.git
    targetRevision: main
    path: k8s
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 3

Developer workflow:

1. Make changes to k8s/httproute.yaml in Bitbucket
2. Create Pull Request
3. Team reviews
4. Merge to main
5. ArgoCD deploys automatically
6. Monitor in ArgoCD UI

No manual kubectl. No SSH to clusters. Pure GitOps.

Advanced Features We Couldn’t Do Before

graph TB
    subgraph Gateway["Gateway: production-gateway"]
        L1["Listener 1<br/>HTTP (80)"]
        L2["Listener 2<br/>HTTPS (443)"]
        L3["Listener 3<br/>gRPC (9000)"]
        L4["Listener 4<br/>TCP (5432)"]
    end
    
    subgraph Routes["Routes - Different Protocols"]
        HTTP["HTTPRoute<br/>api.example.com<br/>REST API"]
        HTTPS["HTTPRoute<br/>secure.example.com<br/>Web App"]
        GRPC["GRPCRoute<br/>grpc.example.com<br/>UserService<br/>OrderService"]
        TCP["TCPRoute<br/>Database<br/>Connection Pool"]
    end
    
    subgraph Services["Backend Services"]
        RestAPI["REST API Service<br/>Port 8080"]
        WebApp["Web Application<br/>Port 3000"]
        GRPCSvc["gRPC Service<br/>Port 9000"]
        DB["PostgreSQL<br/>Port 5432"]
    end
    
    L1 --> HTTP
    L2 --> HTTPS
    L3 --> GRPC
    L4 --> TCP
    
    HTTP --> RestAPI
    HTTPS --> WebApp
    GRPC --> GRPCSvc
    TCP --> DB
    
    style Gateway fill:#e6f3ff
    style Routes fill:#ccffcc
    style Services fill:#ffcc99

1. Traffic Mirroring (Shadow Traffic)

Test new version with real traffic without impacting users:

# apps/api/k8s/httproute-mirror.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-with-mirror
  namespace: production
spec:
  parentRefs:
  - name: production-gateway
    namespace: gateway-system
  
  hostnames:
  - api.altimetrik.com
  
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    
    # Main backend (serves responses)
    backendRefs:
    - name: api-v1
      port: 80
    
    # Mirror backend (receives copy, responses discarded)
    filters:
    - type: RequestMirror
      requestMirror:
        backendRef:
          name: api-v2-test
          port: 80

graph TB
    subgraph Features["Gateway API Advanced Features"]
        subgraph TrafficSplit["Traffic Splitting"]
            TS1["Weighted Routing<br/>90% stable<br/>10% canary"]
            TS2["A/B Testing<br/>50% variant A<br/>50% variant B"]
        end
        
        subgraph HeaderRouting["Header-Based Routing"]
            HR1["X-Beta-User: true<br/>→ Beta Backend"]
            HR2["X-Mobile-App: android<br/>→ Android API"]
            HR3["X-Region: us-west<br/>→ West Service"]
        end
        
        subgraph QueryParam["Query Parameter Routing"]
            QP1["?version=v2<br/>→ v2 Backend"]
            QP2["?env=staging<br/>→ Staging Backend"]
        end
        
        subgraph Mirroring["Traffic Mirroring"]
            M1["Primary: v1<br/>Mirror: v2<br/>Test with real traffic"]
        end
        
        subgraph HeaderMod["Header Manipulation"]
            HM1["Add: X-Request-ID"]
            HM2["Remove: X-Legacy-Header"]
            HM3["Set: Strict-Transport-Security"]
        end
        
        subgraph Timeouts["Timeouts & Retries"]
            T1["Request Timeout: 30s"]
            T2["Backend Timeout: 25s"]
            T3["Retry on 5xx: 3 attempts"]
        end
    end
    
    subgraph Compare["vs NGINX Ingress"]
        Old1["❌ Annotations"]
        Old2["❌ ConfigMap snippets"]
        Old3["❌ Lua scripts"]
        Old4["❌ Multiple Ingresses"]
    end
    
    Features -.->|"All Native<br/>No Annotations"| Compare
    
    style Features fill:#ccffcc
    style Compare fill:#ffcccc

Use case at Altimetrik: We tested our new payment processing backend with real production traffic for 2 weeks before releasing. Zero user impact, complete confidence in the new version.

Commit to Bitbucket, ArgoCD deploys.

2. gRPC Routing (Finally!)

# apps/user-service/k8s/grpcroute.yaml
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GRPCRoute
metadata:
  name: user-service-grpc
  namespace: production
spec:
  parentRefs:
  - name: production-gateway
    namespace: gateway-system
  
  hostnames:
  - grpc.altimetrik.com
  
  rules:
  # Route by gRPC method
  - matches:
    - method:
        service: com.altimetrik.UserService
        method: GetUser
    backendRefs:
    - name: user-service
      port: 9000
  
  # Route by method type
  - matches:
    - method:
        service: com.altimetrik.UserService
        method: CreateUser
    backendRefs:
    - name: user-service-write
      port: 9000
  
  # Default route for other methods
  - backendRefs:
    - name: user-service
      port: 9000

With NGINX Ingress, this required:

• Custom ConfigMap with gRPC upstreams
• Complex annotations
• Limited routing capabilities

With Gateway API:

• Native gRPC support
• Method-level routing
• Clean, declarative configuration

3. Query Parameter Routing

# apps/feature-flags/k8s/httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: feature-flag-routing
  namespace: production
spec:
  parentRefs:
  - name: production-gateway
    namespace: gateway-system
  
  rules:
  # Route ?beta=true to beta backend
  - matches:
    - queryParams:
      - type: Exact
        name: beta
        value: "true"
    backendRefs:
    - name: api-beta
      port: 80
  
  # Route ?version=v2 to v2 backend
  - matches:
    - queryParams:
      - type: Exact
        name: version
        value: v2
    backendRefs:
    - name: api-v2
      port: 80
  
  # Default route
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: api-stable
      port: 80

Use case: QA team can test new versions by appending ?version=v2 to URL. No special builds needed.

4. Request/Response Header Manipulation

# apps/api/k8s/httproute-headers.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-with-headers
  namespace: production
spec:
  parentRefs:
  - name: production-gateway
    namespace: gateway-system
  
  hostnames:
  - api.altimetrik.com
  
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    
    filters:
    # Add request headers
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: X-Forwarded-For
          value: "${client.ip}"
        - name: X-Request-Start
          value: "${request.time}"
        - name: X-Environment
          value: "production"
        remove:
        - X-Legacy-Header
        - X-Internal-Only
    
    # Add response headers
    - type: ResponseHeaderModifier
      responseHeaderModifier:
        add:
        - name: X-Content-Type-Options
          value: "nosniff"
        - name: X-Frame-Options
          value: "DENY"
        - name: Strict-Transport-Security
          value: "max-age=31536000; includeSubDomains"
        - name: X-Served-By
          value: "gateway-api"
    
    backendRefs:
    - name: api-service
      port: 80

All of this would require complex NGINX configuration snippets before.

Production Learnings and Best Practices

sequenceDiagram
    participant User as User
    participant DNS as Route 53
    participant ALB as AWS ALB
    participant Gateway as Gateway
    participant Route as HTTPRoute
    participant Svc1 as Service v1
    participant Svc2 as Service v2
    participant Pods1 as Pods v1
    participant Pods2 as Pods v2
    
    Note over User,Pods2: Canary Deployment with 90/10 Split
    
    User->>DNS: 1. Resolve api.example.com
    DNS->>User: ALB DNS name
    
    User->>ALB: 2. HTTPS Request
    Note over ALB: TLS Termination<br/>Certificate from ACM
    
    ALB->>Gateway: 3. Route to Gateway listener
    Gateway->>Route: 4. Match HTTPRoute<br/>Host: api.example.com
    
    Route->>Route: 5. Apply traffic weights<br/>90% → v1<br/>10% → v2
    
    alt 90% of requests (stable)
        Route->>Svc1: 6a. Forward to v1 (weight: 90)
        Svc1->>Pods1: 7a. Load balance to v1 pods
        Pods1->>Svc1: 8a. Response
        Svc1->>Route: 9a. Return response
    else 10% of requests (canary)
        Route->>Svc2: 6b. Forward to v2 (weight: 10)
        Svc2->>Pods2: 7b. Load balance to v2 pods
        Pods2->>Svc2: 8b. Response
        Svc2->>Route: 9b. Return response
    end
    
    Route->>Gateway: 10. Response
    Gateway->>ALB: 11. Response
    ALB->>User: 12. HTTPS Response
    
    Note over User,Pods2: Seamless canary deployment<br/>User unaware of backend routing

Lesson 1: GitOps is Non-Negotiable

What we learned: Managing Gateway API resources with kubectl is chaos. GitOps with ArgoCD is the only sane way.

Our approach:

Infrastructure repo (Bitbucket):
├── Gateway API CRDs
├── AWS LB Controller
├── GatewayClass
├── Shared Gateways
└── Platform policies

Application repos (Bitbucket):
├── HTTPRoutes
├── Services
├── Deployments
└── App-specific config

Benefits:

• Complete audit trail
• Easy rollback (git revert)
• Review process (PRs)
• No configuration drift
• Multi-cluster deployment (same repo, different ArgoCD)

Lesson 2: Start with One Gateway, Share Across Teams

What we did:

• Created ONE production Gateway in gateway-system namespace
• All teams create HTTPRoutes in their own namespaces
• HTTPRoutes reference the shared Gateway via ReferenceGrant

Why this works:

• Platform team controls infrastructure
• App teams manage their routing independently
• One AWS ALB = cost savings ($54/mo → $23/mo)
• Simpler TLS certificate management

ArgoCD manages the Gateway:

• Platform team updates Gateway in infrastructure repo
• ArgoCD syncs changes
• All HTTPRoutes automatically benefit

Lesson 3: Use ArgoCD App Projects for Organization

# ArgoCD AppProject for Gateway API
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: gateway-api-infrastructure
  namespace: argocd
spec:
  description: Gateway API platform infrastructure
  
  sourceRepos:
  - https://bitbucket.org/altimetrik/gateway-api-infrastructure.git
  
  destinations:
  - namespace: gateway-system
    server: https://kubernetes.default.svc
  - namespace: kube-system
    server: https://kubernetes.default.svc
  - namespace: cert-manager
    server: https://kubernetes.default.svc
  
  clusterResourceWhitelist:
  - group: 'gateway.networking.k8s.io'
    kind: GatewayClass
  - group: 'gateway.networking.k8s.io'
    kind: Gateway
  - group: 'cert-manager.io'
    kind: ClusterIssuer

Lesson 4: Monitor Gateway and HTTPRoute Status

ArgoCD Health Checks:

ArgoCD automatically monitors Gateway API resources:

• Accepted: True - Route is valid
• ResolvedRefs: True - Backend services exist
• Programmed: True - Rules applied to load balancer

Custom health check in ArgoCD:

# Add to argocd-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.gateway.networking.k8s.io_Gateway: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Programmed" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = "Gateway is programmed"
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for Gateway to be programmed"
    return hs

Lesson 5: Migration - Run Both in Parallel

During migration, run NGINX Ingress AND HTTPRoute simultaneously:

In your Bitbucket repo:

apps/payment-api/k8s/
├── ingress-old.yaml        # Keep during migration
├── httproute-new.yaml      # Test with different hostname
├── service.yaml
└── deployment.yaml

Process:

1. Deploy HTTPRoute with test hostname (payment-test.altimetrik.com)
2. ArgoCD deploys via Bitbucket
3. Test thoroughly
4. Update DNS to point to Gateway ALB
5. Monitor for 1 week
6. Remove ingress-old.yaml from Git
7. ArgoCD auto-prunes the old Ingress

Zero downtime. Zero risk.

Lesson 6: Use ArgoCD Sync Waves for Ordering

# Ensure Gateway exists before HTTPRoutes
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
  namespace: gateway-system
  annotations:
    argocd.argoproj.io/sync-wave: "1"  # Deploy first
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "2"  # Deploy after Gateway

ArgoCD respects sync waves - Gateway deploys before Routes.

Should You Migrate?

graph TB
    Start["Are you using<br/>NGINX Ingress?"] --> NGINXCheck{Using NGINX<br/>Ingress?}
    
    NGINXCheck -->|Yes| Urgent["🚨 URGENT<br/>NGINX Ingress retiring<br/>March 2026<br/><br/>You MUST migrate"]
    NGINXCheck -->|No| OtherIngress{Using other<br/>Ingress<br/>controller?}
    
    OtherIngress -->|Yes| Consider["Consider Gateway API<br/>for standardization"]
    OtherIngress -->|No| NewProject["New project?<br/>Start with Gateway API"]
    
    Urgent --> Timeline{Time until<br/>March 2026?}
    
    Timeline -->|"> 12 months"| Comfortable["✅ Comfortable timeline<br/>Plan 4-6 month migration<br/>Follow our guide"]
    Timeline -->|"6-12 months"| Tight["⚠️ Tight timeline<br/>Start immediately<br/>Dedicate resources"]
    Timeline -->|"< 6 months"| Critical["🔴 Critical!<br/>Emergency migration<br/>All hands on deck"]
    
    Comfortable --> TeamSize{Team size &<br/>complexity?}
    Tight --> TeamSize
    Critical --> Priority["Priority 1:<br/>Migrate NOW"]
    
    TeamSize -->|"< 20 apps"| Small["Small Scale:<br/>2-3 month migration<br/>Single team effort"]
    TeamSize -->|"20-100 apps"| Medium["Medium Scale:<br/>4-6 month migration<br/>Phased approach"]
    TeamSize -->|"100+ apps"| Large["Large Scale:<br/>6-12 month migration<br/>Multiple teams<br/>(Like Altimetrik: 200+ apps)"]
    
    Small --> Features{Need advanced<br/>features?}
    Medium --> Features
    Large --> Features
    
    Features -->|"Canary, gRPC,<br/>Traffic mirroring"| HighValue["✅ High Value<br/>Migrate ASAP<br/>Enable new capabilities"]
    Features -->|"Simple HTTP<br/>routing only"| BasicNeeds["⚠️ Basic Needs<br/>But still must migrate<br/>by March 2026"]
    
    HighValue --> GitOps{Using<br/>GitOps?}
    BasicNeeds --> GitOps
    
    GitOps -->|"Yes<br/>(ArgoCD, Flux)"| EasyPath["✅ Easy Path<br/>Follow Altimetrik approach:<br/>• Bitbucket repos<br/>• ArgoCD automation<br/>• Gradual rollout"]
    GitOps -->|"No"| HarderPath["⚠️ Harder Path<br/>Consider adopting<br/>GitOps first<br/>Then migrate"]
    
    EasyPath --> Action1["ACTION PLAN:<br/>Month 1: Setup Gateway API<br/>Month 2: Pilot (5 apps)<br/>Month 3-4: Team migration<br/>Month 5-6: Cleanup"]
    HarderPath --> Action2["ACTION PLAN:<br/>Week 1-2: Setup GitOps<br/>Then follow standard path"]
    
    Consider --> NewArch["Evaluate for:<br/>• Standardization<br/>• Advanced routing<br/>• Multi-protocol<br/>• Future-proofing"]
    NewProject --> StartRight["✅ Start Right<br/>Use Gateway API<br/>from day one<br/>Don't use legacy Ingress"]
    Priority --> Emergency["EMERGENCY PLAN:<br/>1. Setup Gateway API (Week 1)<br/>2. Migrate critical apps (Week 2-4)<br/>3. Mass migration (Week 5-12)<br/>4. Accept some manual work<br/>5. All done before March 2026"]
    
    Action1 --> Success["✅ Migration Complete<br/>Before March 2026<br/>Future-proof platform"]
    Action2 --> Success
    Emergency --> Success
    NewArch --> Decide["Make informed<br/>decision"]
    StartRight --> Future["Future-proof<br/>from day one"]
    
    style Urgent fill:#ff9999
    style Critical fill:#ff6666
    style Priority fill:#ff6666
    style Timeline fill:#ffcccc
    style Comfortable fill:#ccffcc
    style Tight fill:#ffffcc
    style EasyPath fill:#99ff99
    style Success fill:#99ff99
    style Future fill:#99ff99
    style StartRight fill:#99ff99

You MUST Migrate If:

🚨 You’re using NGINX Ingress (retiring March 2026) 🚨 You need security patches after March 2026 🚨 You want to stay on supported, maintained software

You SHOULD Migrate If:

✅ You need advanced traffic management (canary, mirroring, weighted routing) ✅ You want multi-protocol support (HTTP, gRPC, TCP) ✅ You’re tired of annotation hell ✅ You want role-based infrastructure management ✅ You use GitOps (ArgoCD, Flux) ✅ You plan to adopt service mesh later ✅ You want future-proof networking

You Can Wait If:

⏸️ Simple HTTP routing is all you need ⏸️ You have < 10 applications ⏸️ You’re planning major architecture changes anyway ⏸️ You have time before March 2026

But don’t wait too long. March 2026 will come faster than you think.

Our Migration Results at Altimetrik

Metrics After 4 Months:

Developer Feedback:

“Finally, canary deployments without black magic annotations! And everything through Git—no more kubectl.” - Frontend Team

“gRPC routing just works. No more ConfigMaps and snippets.” - Backend Team

“I can create routes myself via Bitbucket PR without waiting for platform team approval.” - Mobile API Team

“Best part? When NGINX Ingress retires in 2026, we’re already done with migration. No last-minute panic.” - Engineering Manager

Conclusion: Gateway API is the Future (and NGINX is the Past)

Four months ago, we had:

• 200+ Ingress resources with annotation soup
• Ticking clock: NGINX Ingress retiring March 2026
• Complex canary deployments requiring separate resources
• Limited traffic management capabilities
• Protocol limitations
• Manual kubectl deployments

Today, we have:

• 200+ HTTPRoutes with clean, declarative configuration
• Zero concern about March 2026 NGINX Ingress retirement
• Built-in traffic splitting, mirroring, and advanced routing
• Multi-protocol support (HTTP, gRPC, TCP)
• Complete GitOps automation via ArgoCD and Bitbucket
• Team autonomy—developers self-serve
• Future-proof, Kubernetes-native networking

The migration wasn’t easy, but it was necessary. NGINX Ingress is retiring. Gateway API is the official future of Kubernetes networking.

Key takeaways:

1. Start now - Don’t wait until 2026
2. Use GitOps - ArgoCD + Bitbucket makes it manageable
3. Migrate incrementally - Don’t rush, run both in parallel
4. Leverage role separation - Platform team manages Gateway, app teams manage Routes
5. Enjoy the benefits - Advanced routing, multi-protocol, standardization

If you’re running NGINX Ingress, you have until March 2026. Use this time wisely. Migrate to Gateway API on your own schedule, not in a panic.

We did it in 4 months. You can too.

Resources

Official Documentation:

• Gateway API Official Site
• NGINX Ingress Retirement Announcement
• AWS Load Balancer Controller Docs
• Gateway API Implementations

My Production Infrastructure:

• Production Jenkins on EKS - Production Jenkins setup
• EKS Platform on GitHub - Complete EKS platform

Migration Tools:

• ingress2gateway - Official conversion tool
• ArgoCD - GitOps continuous delivery

Further Reading:

• Gateway API vs Ingress
• Gateway API Graduated to GA

About the Author: I’m a Senior DevOps and Cloud Engineer with 11+ years of experience building production Kubernetes platforms. Currently at Altimetrik India, I led our migration from NGINX Ingress to Gateway API across 200+ applications serving 10+ engineering teams ahead of the March 2026 NGINX Ingress retirement deadline. This work reduced configuration complexity by 60% while enabling advanced traffic management capabilities through GitOps automation with ArgoCD and Bitbucket. I also manage multi-region Kubernetes clusters on AWS with 99.99% SLA uptime. All infrastructure code is available on my GitHub. Connect with me on LinkedIn.

Questions about Gateway API, migration strategies, or the NGINX Ingress retirement? Drop a comment below or reach out on LinkedIn. I’d love to hear about your networking challenges and migration plans!

“The production API is down!”

It was 3 AM. Our monitoring alerts were firing. Five engineers jumped on a call. The question everyone asked: “What do the logs say?”

The answer? Nobody knew. Our logs were scattered across 50+ Kubernetes nodes, each pod writing to its own stdout, with no centralized system to search them. We spent 45 minutes just trying to find the relevant log files while customers were experiencing downtime.

That incident became our wake-up call. Within two months, we built a production-grade observability platform using the EFK stack (Elasticsearch, Fluent Bit, Kibana) that now processes millions of log events per day across 20+ Elasticsearch nodes, serving 10+ engineering teams with real-time log visibility.

In this post, I’ll share the complete journey—from architecture decisions to production deployment, including the challenges we faced and the lessons we learned scaling Elasticsearch to handle enterprise-grade log volumes.

Table of Contents

1. Why We Chose EFK Over Other Solutions
2. Architecture Design: Multi-Node Elasticsearch at Scale
3. Production Deployment with ECK Operator
4. Fluent Bit Configuration for Kubernetes
5. Performance Tuning and Optimization
6. Challenges We Faced and How We Solved Them
7. Monitoring the Monitor: Observability for Observability
8. Lessons Learned and Best Practices

Why We Chose EFK Over Other Solutions

Before settling on EFK, we evaluated several logging solutions:

Why EFK won:

• Cost: Self-hosted meant we controlled costs (estimated savings: $75K/year vs CloudWatch)
• Power: Elasticsearch’s full-text search and aggregations were unmatched
• Flexibility: Complete control over retention, indexing, and analysis
• Ecosystem: Mature tooling, extensive community support

The trade-off? We needed to become Elasticsearch experts. But given our scale (projected 50M+ log events/day), it was worth the investment.

Architecture Design: Multi-Node Elasticsearch at Scale

graph TB
    subgraph K8sCluster["Kubernetes Cluster - 50+ Nodes"]
        subgraph Nodes["Worker Nodes"]
            Node1["Node 1<br/>App Pods"]
            Node2["Node 2<br/>App Pods"]
            Node3["Node 3<br/>App Pods"]
            NodeN["Node N<br/>App Pods"]
            
            FB1["Fluent Bit<br/>DaemonSet Pod"]
            FB2["Fluent Bit<br/>DaemonSet Pod"]
            FB3["Fluent Bit<br/>DaemonSet Pod"]
            FBN["Fluent Bit<br/>DaemonSet Pod"]
        end
        
        Node1 -.->|logs| FB1
        Node2 -.->|logs| FB2
        Node3 -.->|logs| FB3
        NodeN -.->|logs| FBN
    end
    
    subgraph ESCluster["Elasticsearch Cluster - 26 Nodes"]
        subgraph Masters["Master Nodes - 3x"]
            M1["Master 1<br/>4GB RAM<br/>50GB Disk"]
            M2["Master 2<br/>4GB RAM<br/>50GB Disk"]
            M3["Master 3<br/>4GB RAM<br/>50GB Disk"]
        end
        
        subgraph Ingest["Ingest Nodes - 3x"]
            I1["Ingest 1<br/>8GB RAM<br/>100GB Disk"]
            I2["Ingest 2<br/>8GB RAM<br/>100GB Disk"]
            I3["Ingest 3<br/>8GB RAM<br/>100GB Disk"]
        end
        
        subgraph DataNodes["Data-ML Nodes - 20x"]
            D1["Data 1<br/>16GB RAM<br/>500GB Disk"]
            D2["Data 2<br/>16GB RAM<br/>500GB Disk"]
            D3["Data 3<br/>16GB RAM<br/>500GB Disk"]
            DN["Data 20<br/>16GB RAM<br/>500GB Disk"]
        end
        
        M1 -.->|cluster<br/>coordination| M2
        M2 -.->|cluster<br/>coordination| M3
        M3 -.->|cluster<br/>coordination| M1
    end
    
    subgraph Visualization["Visualization Layer"]
        K1["Kibana 1<br/>2GB RAM"]
        K2["Kibana 2<br/>2GB RAM"]
        LB["Load Balancer"]
    end
    
    subgraph Users["Users"]
        Engineer["Engineers"]
        SRE["SRE Team"]
        DevOps["DevOps Team"]
    end
    
    FB1 -->|Parse & Enrich| I1
    FB2 -->|Parse & Enrich| I2
    FB3 -->|Parse & Enrich| I3
    FBN -->|Parse & Enrich| I1
    
    I1 -->|Index| D1
    I2 -->|Index| D2
    I3 -->|Index| D3
    I1 -->|Index| DN
    
    D1 -.->|Query| K1
    D2 -.->|Query| K2
    D3 -.->|Query| K1
    DN -.->|Query| K2
    
    K1 --> LB
    K2 --> LB
    
    LB --> Engineer
    LB --> SRE
    LB --> DevOps
    
    style Masters fill:#ffe6e6
    style Ingest fill:#fff4e6
    style DataNodes fill:#e6f3ff
    style Visualization fill:#f0e6ff
    style K8sCluster fill:#e6ffe6

The Evolution: From Single Node to 20+ Node Cluster

Phase 1: Proof of Concept (Week 1-2) We started simple: a single Elasticsearch node running in Kubernetes. It worked for our dev environment but died under production load within hours.

Phase 2: High Availability (Week 3-4) We moved to a 3-node cluster with basic replication. Better, but we hit disk I/O bottlenecks as log volume grew.

Phase 3: Role-Based Architecture (Week 5-8) This is when we designed our current production architecture—and it’s been rock-solid for over a year.

Production Architecture: Role Separation is Key

We separated Elasticsearch nodes into specialized roles, following Elasticsearch best practices for large clusters:

# Our Production Elasticsearch Topology
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: observability
spec:
  version: 8.18.3
  nodeSets:
  
  # MASTER NODES - Cluster coordination only (3 nodes)
  - name: master
    count: 3
    config:
      node.roles: ["master"]
      # Masters don't store data or handle queries
      # Lightweight, dedicated to cluster state management
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            limits:
              memory: 4Gi
              cpu: 2
            requests:
              memory: 4Gi
              cpu: 1
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: gp3
  
  # DATA-ML NODES - Heavy lifting (20+ nodes in our case)
  - name: data-ml
    count: 20
    config:
      node.roles: ["data", "ml", "transform"]
      # These nodes do the real work:
      # - Store all the log data
      # - Execute search queries
      # - Run aggregations
      # - Machine learning jobs
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            limits:
              memory: 16Gi  # Memory-heavy for search performance
              cpu: 4
            requests:
              memory: 16Gi
              cpu: 2
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms8g -Xmx8g"  # 50% of total memory for JVM heap
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 500Gi  # Each data node gets 500GB
        storageClassName: gp3
  
  # INGEST NODES - Pre-processing pipeline (3 nodes)
  - name: ingest
    count: 3
    config:
      node.roles: ["ingest"]
      # Ingest nodes handle document pre-processing:
      # - Parsing JSON
      # - Enriching logs with metadata
      # - Applying ingest pipelines
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            limits:
              memory: 8Gi
              cpu: 2
            requests:
              memory: 8Gi
              cpu: 1
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: gp3

Why This Architecture Works

1. Master Nodes (3 total)

• Purpose: Cluster coordination, index management, node health monitoring
• Why 3?: Quorum-based (2 out of 3 needed for decisions), prevents split-brain
• Resource Profile: Lightweight (4GB RAM), minimal disk
• Key Insight: Masters don’t touch data. Keep them lean and dedicated.

2. Data-ML Nodes (20+ total)

• Purpose: Store indices, execute queries, run ML jobs
• Why 20+?: Horizontal scaling for:
  • Storage capacity (500GB × 20 = 10TB raw)
  • Query throughput (parallel search across shards)
  • Redundancy (2 replicas means 3 copies of each shard)
• Resource Profile: Memory-intensive (16GB RAM), large disk (500GB)
• Key Insight: This is where you scale based on data volume and query load.

3. Ingest Nodes (3 total)

• Purpose: Pre-process logs before indexing (parsing, enrichment, transformation)
• Why separate?: Offloads CPU-intensive parsing from data nodes
• Resource Profile: CPU-focused (2 cores), moderate memory (8GB)
• Key Insight: Prevents ingest spikes from affecting search performance.

The Math: Sizing Your Cluster

Here’s how we calculated our requirements:

Daily Log Volume:

• 50 million log events/day
• Average log size: 1KB
• Total daily data: ~50GB/day

Retention:

• Hot data (7 days): 350GB
• Warm data (23 days): 1.15TB
• Total with replication (2 replicas = 3 copies): ~4.5TB

Shard Strategy:

• Primary shards: 60 (allows scaling to 60 nodes)
• Each shard size target: 30-50GB
• Daily index rollover: fluentbit-YYYY.MM.DD

Result: 20 data nodes × 500GB = 10TB capacity (with comfortable headroom)

Production Deployment with ECK Operator

Why ECK (Elastic Cloud on Kubernetes)?

Managing Elasticsearch manually in Kubernetes is painful. We learned this the hard way with our first cluster:

• Manual pod management
• Complex StatefulSet configurations
• No automated upgrades
• Certificate management nightmares

Enter ECK Operator. It’s like having an Elasticsearch expert embedded in your cluster.

ECK Benefits We Love

1. Automated Lifecycle Management
   • Rolling upgrades with zero downtime
   • Automatic certificate rotation
   • Self-healing (crashed pods automatically replaced)
2. CRD-Based Configuration
   • Elasticsearch, Kibana, and Beats as native Kubernetes resources
   • GitOps-friendly (all config in YAML)
   • Declarative management
3. Production-Ready Defaults
   • TLS everywhere
   • Proper resource limits
   • Volume claim templates
   • Pod disruption budgets

Deployment Steps (The Real Deal)

Step 1: Install CRDs and Operator

# Install Elastic CRDs
kubectl apply -f elastic-customresourcedefinition.yaml

# Deploy ECK Operator
kubectl apply -f eck-operator.yaml

# Wait for operator to be ready
kubectl wait --for=condition=available --timeout=300s \
  deployment/elastic-operator -n elastic-system

Step 2: Deploy Elasticsearch

The beauty of ECK: just apply a single YAML and it handles everything.

kubectl apply -f elasticsearch.yaml

# Watch the magic happen
kubectl get pods -n observability -w

ECK automatically:

• Creates StatefulSets for each nodeSet
• Provisions PersistentVolumeClaims
• Generates TLS certificates
• Configures node discovery
• Starts the cluster with proper bootstrapping

Step 3: Deploy Kibana

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: observability
spec:
  version: 8.18.3
  count: 2  # 2 Kibana instances for HA
  elasticsearchRef:
    name: elasticsearch  # Auto-connects to our ES cluster
  podTemplate:
    spec:
      containers:
      - name: kibana
        resources:
          limits:
            memory: 2Gi
            cpu: 1
          requests:
            memory: 2Gi
            cpu: 0.5

kubectl apply -f kibana.yaml

# Access Kibana locally for testing
kubectl port-forward service/kibana-kb-http 5601:5601 -n observability

# Get the elastic user password (auto-generated by ECK)
kubectl get secret elasticsearch-es-elastic-user \
  -n observability -o jsonpath='{.data.elastic}' | base64 -d

Step 4: Deploy Fluent Bit

Fluent Bit runs as a DaemonSet—one pod per Kubernetes node, collecting logs from all pods on that node.

kubectl apply -f fluent-bit.yaml

# Verify Fluent Bit is running on all nodes
kubectl get daemonset fluent-bit -n observability

Fluent Bit Configuration: The Unsung Hero

Fluent Bit is incredibly lightweight (~450KB Docker image) but powerful. Here’s our production configuration:

sequenceDiagram
    participant App as Application Pod
    participant ContainerRuntime as Container Runtime
    participant FluentBit as Fluent Bit DaemonSet
    participant IngestNode as Ingest Node
    participant DataNode as Data Node
    participant Kibana as Kibana
    participant User as Engineer
    
    Note over App,User: Log Collection & Processing Flow
    
    rect rgb(230, 245, 255)
        Note over App,ContainerRuntime: Phase 1: Log Generation
        App->>ContainerRuntime: Write to stdout/stderr
        ContainerRuntime->>ContainerRuntime: Write to<br/>/var/log/containers/*.log
    end
    
    rect rgb(255, 240, 230)
        Note over FluentBit: Phase 2: Collection & Enrichment
        FluentBit->>FluentBit: Tail log files
        FluentBit->>FluentBit: Parse JSON/multiline
        FluentBit->>FluentBit: Add Kubernetes metadata:<br/>• namespace<br/>• pod name<br/>• labels<br/>• annotations
        FluentBit->>FluentBit: Add cluster metadata
    end
    
    rect rgb(240, 255, 240)
        Note over FluentBit,IngestNode: Phase 3: Buffering & Transmission
        FluentBit->>FluentBit: Buffer logs (5MB limit)
        FluentBit->>IngestNode: Send batch via HTTPS<br/>to Elasticsearch
        
        alt Connection Failed
            FluentBit->>FluentBit: Retry (max 3 attempts)
            FluentBit->>IngestNode: Retry with backoff
        else Connection Success
            IngestNode->>IngestNode: Receive batch
        end
    end
    
    rect rgb(255, 245, 230)
        Note over IngestNode: Phase 4: Pre-Processing
        IngestNode->>IngestNode: Apply ingest pipeline:<br/>• Grok parsing<br/>• Field extraction<br/>• Data enrichment<br/>• Field type conversion
        IngestNode->>IngestNode: Validate document structure
    end
    
    rect rgb(245, 240, 255)
        Note over IngestNode,DataNode: Phase 5: Indexing
        IngestNode->>DataNode: Route to primary shard<br/>based on routing key
        DataNode->>DataNode: Index document:<br/>• Add to inverted index<br/>• Store source<br/>• Update doc values
        DataNode->>DataNode: Replicate to replica shards<br/>(2 replicas = 3 total copies)
        DataNode->>DataNode: Refresh index<br/>(every 30 seconds)
    end
    
    rect rgb(255, 250, 240)
        Note over DataNode,Kibana: Phase 6: Search & Retrieval
        User->>Kibana: Search query:<br/>"ERROR in namespace:production"
        Kibana->>Kibana: Build Elasticsearch DSL query
        Kibana->>DataNode: Execute search across shards
        DataNode->>DataNode: Query all primary shards<br/>in parallel
        DataNode->>DataNode: Aggregate results<br/>Score and rank
        DataNode->>Kibana: Return top results
        Kibana->>Kibana: Format results<br/>Apply visualizations
        Kibana->>User: Display logs with context
    end
    
    Note over App,User: Total Latency: Collection→Search = ~1-2 minutes

Why Fluent Bit Over Logstash or Fluentd?

Winner: Fluent Bit for Kubernetes use cases.

Our Fluent Bit Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: observability
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 5
        Log_Level info
        Parsers_File parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 50MB
        Skip_Long_Lines On
        Refresh_Interval 10

    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_URL https://kubernetes.default.svc:443
        Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix kube.var.log.containers.
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Labels On
        Annotations Off

    [FILTER]
        Name modify
        Match *
        Add cluster_name production-us-east-1
        Add environment production

    [OUTPUT]
        Name es
        Match kube.*
        Host elasticsearch-es-http.observability.svc
        Port 9200
        HTTP_User elastic
        HTTP_Passwd ${ELASTICSEARCH_PASSWORD}
        TLS On
        TLS.Verify Off
        Logstash_Format On
        Logstash_Prefix fluentbit
        Logstash_DateFormat %Y.%m.%d
        Retry_Limit 3
        Buffer_Size 5MB
        Type _doc

Key Configuration Decisions

1. Input: Container Log Collection

Path /var/log/containers/*.log

• Kubernetes writes all container logs here
• Fluent Bit reads from this single location
• Multiline parsing handles stack traces

2. Filter: Kubernetes Metadata Enrichment

Name kubernetes
Merge_Log On
Labels On

This adds critical context to every log:

{
  "kubernetes": {
    "pod_name": "api-server-7d8f9c4b6-x5k2m",
    "namespace_name": "production",
    "container_name": "app",
    "labels": {
      "app": "api-server",
      "version": "v2.1.0"
    }
  }
}

Without this metadata, logs are useless. With it, you can:

• Filter by namespace, pod, container
• Track deployments through labels
• Correlate issues across services

3. Output: Elasticsearch with Logstash Format

Logstash_Format On
Logstash_Prefix fluentbit
Logstash_DateFormat %Y.%m.%d

This creates time-based indices: fluentbit-2024.11.07

Why daily indices?

• Easy retention management (delete old indices)
• Better search performance (smaller indices)
• Rollover at midnight (predictable pattern)

Performance Tuning and Optimization

stateDiagram-v2
    [*] --> Hot: New index created (fluentbit-2024.11.07)
    
    Hot --> Rollover: Conditions met (Max size 50GB or Max age 1 day)
    
    Rollover --> Hot: Create new index (fluentbit-2024.11.08)
    
    Hot --> Warm: Age greater than 7 days
    
    state Warm {
        [*] --> ReadOnly: Set index to read-only
        ReadOnly --> ForcemergeState: Reduce to 1 segment
        ForcemergeState --> Shrink: Shrink to 1 shard
        Shrink --> Compress: Best compression codec
    }
    
    Warm --> Cold: Age greater than 15 days (optional)
    
    state Cold {
        [*] --> Searchable: Move to cheaper storage (S3 via snapshots)
        Searchable --> Frozen: Rarely accessed
    }
    
    Cold --> Delete: Age greater than 30 days
    Warm --> Delete: Age greater than 30 days
    
    Delete --> [*]: Index deleted and storage reclaimed
    
    note right of Hot
        Hot Phase (0-7 days)
        High-performance SSD
        Multiple shards
        Frequently queried
        Cost: High
    end note
    
    note right of Warm
        Warm Phase (7-30 days)
        Standard storage
        Single shard
        Occasional queries
        Cost: Medium
    end note
    
    note right of Delete
        Delete Phase (30+ days)
        Data removed
        Storage freed
        Cost: Zero
    end note

Problem: Slow Queries at Scale

After deploying, we hit performance issues:

• Queries taking 10-30 seconds
• High CPU on data nodes
• Kibana timeouts

Root cause: Poor shard allocation and index settings.

Solution 1: Shard Sizing

Before:

• 5 primary shards per daily index
• Each shard: ~10GB
• Total shards across 30 days: 150 shards

After:

• 60 primary shards per daily index (matches our 20 data nodes × 3)
• Each shard: ~1-2GB
• Better distribution across nodes

Configuration:

PUT _index_template/fluentbit-template
{
  "index_patterns": ["fluentbit-*"],
  "template": {
    "settings": {
      "number_of_shards": 60,
      "number_of_replicas": 2,
      "refresh_interval": "30s",  // Not real-time, but faster indexing
      "codec": "best_compression"
    }
  }
}

Result: Query performance improved by 5x.

Solution 2: Index Lifecycle Management (ILM)

We implemented automated index lifecycle to:

1. Hot phase (0-7 days): Keep on fastest disks, searchable
2. Warm phase (8-30 days): Move to cheaper storage, read-only
3. Delete phase (30+ days): Auto-delete old indices

PUT _ilm/policy/fluentbit-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "readonly": {},
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Impact:

• Storage reduced by 40% (compression + shrinking)
• No more manual index cleanup
• Predictable costs

Solution 3: Heap Size Optimization

The Golden Rule: Set JVM heap to 50% of total pod memory, max 32GB.

Why 50%?

• Elasticsearch uses off-heap memory for lucene caches
• OS needs memory for file system cache
• More heap ≠ better performance (GC overhead)

Why max 32GB?

• Above 32GB, JVM loses “compressed oops” optimization
• 31GB heap performs better than 33GB heap (counterintuitive!)

Our Configuration:

env:
- name: ES_JAVA_OPTS
  value: "-Xms8g -Xmx8g"  # For 16GB pods

Before optimization:

• GC pauses: 2-5 seconds
• Query latency: p99 = 15s

After optimization:

• GC pauses: <500ms
• Query latency: p99 = 2s

Challenges We Faced and How We Solved Them

Challenge 1: Elasticsearch Nodes Crashing Under Load

Symptom: Random pod restarts, OOMKilled errors

Root Cause: Insufficient memory limits + no circuit breakers

Solution:

# Added explicit memory limits
resources:
  limits:
    memory: 16Gi  # Hard limit
  requests:
    memory: 16Gi  # Guaranteed allocation

# Configured circuit breakers
PUT _cluster/settings
{
  "persistent": {
    "indices.breaker.total.limit": "70%",
    "indices.breaker.request.limit": "40%",
    "indices.breaker.fielddata.limit": "40%"
  }
}

Lesson: Always set memory limits = requests for Elasticsearch. Avoid burstable memory.

Challenge 2: Uneven Shard Distribution

Symptom: Some nodes at 90% disk, others at 30%

Root Cause: Elasticsearch’s default shard allocation didn’t account for disk usage well

Solution:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
  }
}

Also enabled shard rebalancing:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.rebalance.enable": "all",
    "cluster.routing.allocation.allow_rebalance": "always"
  }
}

Result: Shards redistributed evenly within 2 hours.

Challenge 3: Fluent Bit Memory Leaks

Symptom: Fluent Bit pods growing from 25MB to 500MB+ over days

Root Cause: Buffer accumulation when Elasticsearch couldn’t keep up

Solution:

[OUTPUT]
    Retry_Limit 3  # Don't retry forever
    Buffer_Size 5MB  # Limit buffer size
    
[INPUT]
    Mem_Buf_Limit 50MB  # Hard limit on input buffer

Also added pod memory limits:

resources:
  limits:
    memory: 200Mi
  requests:
    memory: 100Mi

Result: Stable memory usage, predictable behavior.

Challenge 4: Elasticsearch Upgrade from 7.x to 8.x

The Fear: Would upgrading break everything?

What ECK Did:

1. Rolling upgrade—one node at a time
2. Waited for cluster to turn green after each node
3. Preserved all data
4. Took 4 hours for 26 nodes

Our Steps:

# 1. Take a snapshot first (safety net)
PUT _snapshot/backup_repo/pre-upgrade-snapshot
{
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": false
}

# 2. Update version in YAML
spec:
  version: 8.18.3  # Changed from 7.17.x

# 3. Apply and watch
kubectl apply -f elasticsearch.yaml
kubectl get pods -n observability -w

Result: Zero downtime. ECK handled everything.

Lesson: Trust ECK for upgrades. It’s battle-tested.

Monitoring the Monitor: Observability for Observability

You need to monitor your logging infrastructure—otherwise, how do you know when logs aren’t being collected?

Metrics We Track

1. Elasticsearch Cluster Health

# Prometheus metrics exposed by ECK
kubectl port-forward service/elasticsearch-es-http 9200:9200 -n observability

curl -u elastic:$PASSWORD https://localhost:9200/_cluster/health

Key metrics:

• Cluster status (green/yellow/red)
• Node count
• Active shards
• Unassigned shards (should always be 0)

2. Indexing Rate

GET _cat/indices/fluentbit-*?v&s=index:desc&h=index,pri,rep,docs.count,store.size

We alert if:

• Daily log volume drops >50% (indicates Fluent Bit issues)
• Indexing latency >5s (indicates cluster overload)

3. Fluent Bit Health

# Fluent Bit exposes metrics on port 2020
kubectl port-forward ds/fluent-bit 2020:2020 -n observability

curl http://localhost:2020/api/v1/metrics

Key metrics:

• Input records: logs collected
• Output records: logs sent to Elasticsearch
• Output retries: should be low

4. Kibana Response Time We monitor Kibana’s /api/status endpoint:

curl https://kibana.company.com/api/status

Our Grafana Dashboard

We built a Grafana dashboard that shows:

• Cluster health
• Indexing rate (logs/second)
• Query latency (p50, p95, p99)
• Disk usage per node
• JVM heap usage
• GC frequency

Sample Alert:

- alert: ElasticsearchClusterRed
  expr: elasticsearch_cluster_health_status{color="red"} == 1
  for: 5m
  annotations:
    summary: "Elasticsearch cluster is RED"
    description: "Check unassigned shards and node health"

Lessons Learned and Best Practices

graph LR
    subgraph Input["Log Sources"]
        FB["Fluent Bit<br/>50+ pods"]
    end
    
    subgraph IngestLayer["Ingest Layer"]
        I1["Ingest Node 1"]
        I2["Ingest Node 2"]
        I3["Ingest Node 3"]
        
        Pipeline["Ingest Pipeline<br/>• Parse JSON<br/>• Extract fields<br/>• Enrich metadata<br/>• Type conversion"]
    end
    
    subgraph Coordination["Coordination Layer"]
        M1["Master Node 1"]
        M2["Master Node 2"]
        M3["Master Node 3"]
        
        Tasks["Master Tasks<br/>• Cluster state<br/>• Index management<br/>• Shard allocation<br/>• Node discovery"]
    end
    
    subgraph DataLayer["Data Layer - Hot Tier"]
        D1["Data Node 1<br/>Shards: 1,4,7..."]
        D2["Data Node 2<br/>Shards: 2,5,8..."]
        D3["Data Node 3<br/>Shards: 3,6,9..."]
        DN["Data Node 20<br/>Shards: 60,57,54..."]
        
        Storage["Distributed Storage<br/>• 60 primary shards<br/>• 120 replica shards<br/>• Total: 180 shards<br/>• ~10TB total capacity"]
    end
    
    subgraph WarmTier["Warm Tier (ILM)"]
        W1["Warm Data<br/>7-30 days old"]
        W2["Read-only<br/>Compressed<br/>Shrunk to 1 shard"]
    end
    
    FB -->|"50M events/day<br/>~50GB/day"| I1
    FB --> I2
    FB --> I3
    
    I1 --> Pipeline
    I2 --> Pipeline
    I3 --> Pipeline
    
    Pipeline -->|"Processed<br/>documents"| D1
    Pipeline --> D2
    Pipeline --> D3
    Pipeline --> DN
    
    M1 -.->|"Manages"| D1
    M1 -.->|"Manages"| D2
    M2 -.->|"Manages"| D3
    M3 -.->|"Manages"| DN
    
    M1 --> Tasks
    M2 --> Tasks
    M3 --> Tasks
    
    D1 -->|"After 7 days<br/>(ILM)"| W1
    D2 --> W1
    D3 --> W2
    DN --> W2
    
    W1 -.->|"Delete after<br/>30 days"| Delete["🗑️ Deleted"]
    W2 -.-> Delete
    
    style Input fill:#e6f3ff
    style IngestLayer fill:#fff4e6
    style Coordination fill:#ffe6e6
    style DataLayer fill:#e6ffe6
    style WarmTier fill:#f0e6ff

After a year of running Elasticsearch at scale, here’s what we learned:

1. Start with ECK, Not Manual Deployment

Why: ECK saves months of operational work. Certificate rotation alone is worth it.

2. Separate Node Roles Early

Why: Even at small scale, role separation prevents resource contention. Masters shouldn’t be bogged down by queries.

3. Shard Size Matters More Than Shard Count

Golden rule: Aim for 30-50GB shards. Smaller shards mean more overhead; larger shards mean slow recovery.

4. Index Lifecycle is Non-Negotiable

Why: Without ILM, your disk fills up and you panic-delete indices at 2 AM. Ask me how I know.

5. Circuit Breakers Save Your Cluster

Why: One bad query can OOM your entire cluster. Circuit breakers prevent cascading failures.

6. Monitor Everything

Why: When your logging system fails, you’re blind. We learned this when Fluent Bit stopped working and we didn’t notice for 6 hours.

7. Test Disaster Recovery

Why: We tested our Elasticsearch snapshots once… and discovered they were corrupted. Test your backups regularly.

# Monthly DR drill
1. Take a snapshot
2. Delete a test index
3. Restore from snapshot
4. Verify data integrity

8. Fluent Bit is Better Than Logstash for Kubernetes

Why: Lower resource usage, native Kubernetes integration, simpler config. Unless you need Logstash’s advanced pipelines, use Fluent Bit.

9. Tune for Your Use Case

Our use case: Write-heavy (millions of logs/day), read-occasionally (interactive queries)

Configuration:

• Longer refresh intervals (30s instead of 1s)
• More data nodes than ingest nodes
• Aggressive compression

If you’re read-heavy: Tune differently (more replicas, faster refresh, more cache).

10. Elasticsearch ≠ Database

Why: Don’t treat Elasticsearch like a transactional database. It’s designed for search and analytics, not ACID guarantees.

Lesson: We initially tried using Elasticsearch for critical application data. Bad idea. Use it for logs and metrics, not as your source of truth.

Real-World Impact: The Numbers

After deploying our EFK stack, here’s what changed:

Before EFK:

• Mean time to find relevant logs: 45 minutes
• Engineers manually SSHing into pods
• No centralized log retention
• Zero compliance audit trail
• Log analysis: manual grep through files

After EFK:

• Mean time to find relevant logs: 30 seconds
• Centralized search across all services
• 30-day retention with automated lifecycle
• Complete audit trail for compliance
• Interactive log analysis with Kibana

Cost Comparison:

• Self-hosted EFK: ~$3K/month (EC2 + EBS)
• AWS CloudWatch equivalent: ~$11K/month
• Savings: $96K/year

Operational Metrics:

• 50M+ log events/day
• 99.9% uptime
• <2s query latency (p99)
• Zero manual intervention required (after initial setup)

What’s Next: Future Improvements

We’re continuously improving our observability platform:

Short-term (Next Quarter):

• Machine Learning for Anomaly Detection: Using Elasticsearch ML to detect unusual log patterns
• Advanced Kibana Dashboards: Pre-built dashboards for each team
• Log Sampling: Sample non-critical logs to reduce volume by 50%

Medium-term (Next Year):

• Multi-Cluster Federation: Separate dev/staging/prod clusters with cross-cluster search
• Hot-Warm-Cold Architecture: Move warm data to cheaper storage (S3 via searchable snapshots)
• APM Integration: Add Elastic APM for distributed tracing alongside logs

Conclusion: Observability is a Journey, Not a Destination

Building a production-grade EFK stack wasn’t easy. We hit numerous challenges—OOM kills, uneven shard distribution, Fluent Bit memory leaks, and more. But the result? A logging infrastructure that scales effortlessly and empowers our engineering teams.

The key lessons:

1. Start with ECK for production deployments
2. Separate node roles for clean architecture
3. Tune relentlessly based on your workload
4. Monitor your monitors to avoid blind spots
5. Test disaster recovery before you need it

If you’re building observability at scale, learn from our mistakes. And remember: the goal isn’t perfect logs—it’s fast incident resolution and confident deployments.

Resources

GitHub Repository:

• My EFK Stack Implementation - Complete production-ready deployment with ECK

Official Documentation:

• Elastic Cloud on Kubernetes (ECK)
• Fluent Bit Kubernetes Filter
• Elasticsearch Index Lifecycle Management

Further Reading:

• Elasticsearch Sizing Guide
• Fluent Bit vs Logstash Performance

About the Author: I’m a Senior DevOps and Cloud Engineer with 11+ years of experience building scalable infrastructure. Currently managing multi-region Kubernetes clusters with 20+ Elasticsearch nodes processing millions of logs daily. You can find my complete EFK implementation on GitHub and connect with me on LinkedIn.

Questions? Feedback? Drop a comment below or reach out on LinkedIn. I’d love to hear about your observability journey!

Security breaches don’t happen because of what you protect—they happen because of what you assume is safe. In traditional perimeter-based security models, once an attacker crosses the firewall, they have free reign inside your network. This is why we adopted a zero-trust approach for our Kubernetes infrastructure at scale.

Over the past year, I’ve led the implementation of a comprehensive zero-trust security architecture protecting multi-region Kubernetes clusters serving 10+ engineering teams. In this post, I’ll share the practical lessons, implementation details, and the specific tools we used: Teleport for identity-based access, Kubernetes Network Policies for workload isolation, and Open Policy Agent (OPA) for policy enforcement.

The Zero-Trust Mindset: Never Trust, Always Verify

Zero-trust security operates on a simple principle: assume breach. Every request must be authenticated, authorized, and encrypted—regardless of where it originates. There’s no “inside” or “outside” the network; there’s only verified and unverified.

For Kubernetes environments, this translates to three core pillars:

1. Identity-based access control - Every user and service must prove who they are
2. Least-privilege authorization - Grant only the minimum permissions needed
3. Continuous verification - Monitor and validate every interaction

Let me walk you through how we implemented each pillar in production.

Diagram 1: Zero-Trust Architecture Overview

graph TB
    subgraph External["External Access Layer"]
        Engineer[👨‍💻 Engineer]
        SSO[Okta SSO]
        MFA[2FA/MFA Device]
    end
    
    subgraph Teleport["Identity & Access Layer - Teleport"]
        TeleportProxy[Teleport Proxy<br/>Certificate Authority]
        AuditLog[Audit Log Storage<br/>EFK Stack]
        TeleportProxy --> AuditLog
    end
    
    subgraph AdmissionControl["Admission Control Layer - OPA"]
        Webhook[OPA Gatekeeper<br/>Admission Webhook]
        Policies[Policy Library<br/>Non-root containers<br/>Resource limits<br/>Image tags<br/>Security contexts]
        Webhook --> Policies
    end
    
    subgraph K8sCluster["Kubernetes Cluster"]
        APIServer[Kube API Server]
        
        subgraph Production["Production Namespace"]
            DefaultDeny[Default Deny<br/>Network Policy]
            
            subgraph Frontend["Frontend Zone"]
                FrontendPod[Frontend Pod<br/>Non-privileged]
            end
            
            subgraph Backend["Backend Zone"]
                APIPod[API Pod<br/>Non-privileged]
            end
            
            subgraph Data["Data Zone"]
                DBPod[Database Pod<br/>Non-privileged]
            end
            
            DefaultDeny -.enforces.-> FrontendPod
            DefaultDeny -.enforces.-> APIPod
            DefaultDeny -.enforces.-> DBPod
        end
    end
    
    subgraph NetworkPolicies["Network Policy Enforcement"]
        AllowFE[Allow Policy<br/>Frontend to API]
        AllowBE[Allow Policy<br/>API to Database]
        BlockLateral[Block<br/>Frontend to Database]
    end
    
    Engineer -->|1. Login with SSO| SSO
    SSO -->|2. Authenticate| TeleportProxy
    Engineer -->|3. Provide 2FA| MFA
    MFA -->|4. Verify| TeleportProxy
    TeleportProxy -->|5. Issue Certificate| Engineer
    
    Engineer -->|6. kubectl apply| Webhook
    Webhook -->|7. Validate policies| Policies
    Webhook -->|8. Approved/Rejected| APIServer
    
    Engineer -->|9. kubectl commands| TeleportProxy
    TeleportProxy -->|10. Proxied access| APIServer
    
    APIServer --> Production
    
    FrontendPod -->|Allowed| APIPod
    APIPod -->|Allowed| DBPod
    FrontendPod -.->|Blocked| DBPod
    
    AllowFE -.enforces.-> FrontendPod
    AllowBE -.enforces.-> APIPod
    BlockLateral -.blocks.-> FrontendPod
    
    style Engineer fill:#e1f5ff
    style TeleportProxy fill:#ff9999
    style Webhook fill:#ffcc99
    style DefaultDeny fill:#ffcccc
    style AllowFE fill:#ccffcc
    style AllowBE fill:#ccffcc
    style BlockLateral fill:#ffcccc
    style AuditLog fill:#ffffcc

Diagram 2: Zero-Trust Deployment Flow (Step-by-Step)

This sequence diagram shows what happens from code commit to production deployment.

sequenceDiagram
    participant Dev as 👨‍💻 Developer
    participant Git as Git Repository
    participant CI as CI/CD Pipeline
    participant Teleport as Teleport Proxy
    participant OPA as OPA Gatekeeper
    participant API as Kubernetes API
    participant NP as Network Policy<br/>Controller
    participant Pod as Application Pod
    participant Audit as Audit Log
    
    Note over Dev,Audit: Deployment with Zero-Trust Validation
    
    rect rgb(230, 240, 255)
        Note over Dev,Git: Phase 1: Code & Configuration
        Dev->>Git: Push deployment manifest
        Git->>CI: Trigger pipeline
    end
    
    rect rgb(255, 240, 230)
        Note over CI,OPA: Phase 2: Policy Pre-Validation
        CI->>CI: Run conftest<br/>(OPA policy check)
        alt Policy Violations Found
            CI-->>Dev: ❌ Build Failed<br/>Policy violations detected
            Note over Dev: Fix: Add security context,<br/>resource limits, etc.
        else Policies Pass
            CI->>CI: ✅ Continue deployment
        end
    end
    
    rect rgb(230, 255, 240)
        Note over Dev,Teleport: Phase 3: Authentication
        Dev->>Teleport: tsh login
        Teleport->>Dev: Request SSO + 2FA
        Dev->>Teleport: Provide credentials + MFA
        Teleport->>Audit: Log authentication attempt
        Teleport->>Dev: Issue short-lived certificate<br/>(8 hour expiry)
    end
    
    rect rgb(255, 255, 230)
        Note over Dev,API: Phase 4: Deployment Submission
        Dev->>Teleport: kubectl apply -f deployment.yaml
        Teleport->>Audit: Log command execution
        Teleport->>API: Forward request<br/>(with certificate)
        API->>API: Verify certificate validity
    end
    
    rect rgb(255, 230, 230)
        Note over API,OPA: Phase 5: Admission Control
        API->>OPA: Webhook: Validate resource
        OPA->>OPA: Check against policies:<br/>• Non-root user?<br/>• Resource limits set?<br/>• No 'latest' tag?<br/>• Security context defined?
        
        alt Policy Violation
            OPA-->>API: ❌ Admission Denied<br/>Detailed error message
            API-->>Dev: Deployment rejected with fix
        else All Policies Pass
            OPA->>API: ✅ Admission Approved
        end
    end
    
    rect rgb(240, 230, 255)
        Note over API,Pod: Phase 6: Pod Creation & Network Isolation
        API->>Pod: Create Pod
        Pod->>NP: Register with network policies
        NP->>NP: Apply default deny-all
        NP->>NP: Apply explicit allow rules
        Pod->>Pod: Start container<br/>(non-root, with limits)
    end
    
    rect rgb(230, 255, 255)
        Note over Pod,Audit: Phase 7: Runtime & Monitoring
        Pod->>Pod: Application running
        
        alt Allowed Network Communication
            Pod->>Pod: Communication to<br/>allowed services ✅
        else Blocked Network Communication
            Pod--xPod: Communication blocked<br/>by network policy ❌
            NP->>Audit: Log policy violation
        end
        
        Note over Dev,Audit: All actions logged for compliance
        Teleport->>Audit: Access logs
        OPA->>Audit: Policy decisions
        NP->>Audit: Network events
    end
    
    Note over Dev,Audit: Zero-Trust = Every layer validated, nothing trusted by default

Pillar 1: Identity-Based Infrastructure Access with Teleport

The Problem We Faced

Before Teleport, our access model had several weaknesses:

• SSH keys scattered across engineer laptops
• Shared credentials for production access
• No audit trail of who accessed what and when
• VPN-based perimeter security (once you’re in, you’re in)

This wasn’t just a security risk—it was an operational nightmare. When engineers left, we had to rotate keys. When compliance audits came, we couldn’t provide detailed access logs.

Why Teleport?

Teleport is a unified access plane that provides identity-based access to SSH servers, Kubernetes clusters, databases, and web applications. It replaces static credentials with short-lived certificates and enforces multi-factor authentication (MFA) at every connection.

Here’s what made Teleport perfect for our zero-trust architecture:

• Certificate-based authentication - No more SSH keys to manage
• Built-in 2FA/MFA - Every access requires second factor
• Complete audit logging - Every command, every kubectl exec, recorded
• RBAC integration - Ties directly to our identity provider (Okta)

Implementation Deep Dive

Step 1: Deploying Teleport in High Availability Mode

We deployed Teleport as a StatefulSet in our management cluster with 3 replicas for high availability:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: teleport
  namespace: teleport
spec:
  replicas: 3
  serviceName: teleport
  selector:
    matchLabels:
      app: teleport
  template:
    metadata:
      labels:
        app: teleport
    spec:
      containers:
      - name: teleport
        image: public.ecr.aws/gravitational/teleport:13
        args:
        - --roles=auth,proxy,node
        - --config=/etc/teleport/teleport.yaml
        volumeMounts:
        - name: config
          mountPath: /etc/teleport
        - name: storage
          mountPath: /var/lib/teleport

Step 2: Configuring SSO with MFA

We integrated Teleport with our existing Okta SSO and enforced hardware-based 2FA (YubiKey):

auth_service:
  authentication:
    type: saml
    second_factor: on
    webauthn:
      rp_id: teleport.company.com
    connector_name: okta
  
connectors:
  - kind: saml
    name: okta
    spec:
      acs: https://teleport.company.com/v1/webapi/saml/acs
      entity_descriptor_url: https://company.okta.com/app/...
      attributes_to_roles:
        - name: groups
          value: "DevOps"
          roles: ["devops-admin"]
        - name: groups
          value: "Developers"
          roles: ["developer"]

Step 3: Kubernetes Integration

Connecting Teleport to our EKS clusters was straightforward. We deployed Teleport agents in each cluster:

apiVersion: v1
kind: ConfigMap
metadata:
  name: teleport-agent-config
data:
  teleport.yaml: |
    teleport:
      auth_token: /var/run/secrets/teleport/token
      proxy_server: teleport.company.com:443
    
    kubernetes_service:
      enabled: true
      listen_addr: 0.0.0.0:3027

Now engineers access Kubernetes like this:

# Login with SSO + 2FA
tsh login --proxy=teleport.company.com

# List available clusters
tsh kube ls

# Connect to production cluster
tsh kube login prod-us-east-1

# All kubectl commands are now authenticated and logged
kubectl get pods -n production

The Impact

Before Teleport:

• 45+ static SSH keys to manage
• Zero audit trail for production access
• 2-3 hours to onboard/offboard engineers
• Failed PCI compliance audit due to shared credentials

After Teleport:

• Zero static credentials
• Complete audit log (every command, every session)
• 5 minutes to onboard/offboard engineers
• Passed PCI compliance with zero findings on access control

Pillar 2: Workload Isolation with Network Policies

The Default Kubernetes Problem

By default, Kubernetes has a flat network—every pod can talk to every other pod. This is convenient for development, but it’s a security nightmare in production. If an attacker compromises one pod, they can pivot to any other workload in the cluster.

Network Policies allow you to define rules about which pods can communicate with each other. Think of them as firewalls inside your cluster.

Our Network Policy Strategy

We implemented a defense-in-depth approach with three layers:

Layer 1: Default Deny Everything

First, we created a default deny policy in every namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This blocks all traffic by default. Then we explicitly allow only what’s needed.

Layer 2: Allow Internal Communication

For microservices that need to talk to each other, we use label-based policies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-to-database
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432

Layer 3: External Access Control

We tightly control egress to external services:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-external-apis
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: payment-service
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
    ports:
    - protocol: TCP
      port: 443

This allows DNS resolution and HTTPS to external services while blocking access to internal RFC1918 ranges.

Testing Network Policies

We created a simple test script that every team runs before deploying:

#!/bin/bash
# Test network policy enforcement

echo "Testing pod isolation..."

# This should fail (denied by network policy)
kubectl run -it --rm test-pod --image=busybox --restart=Never -- \
  wget -O- --timeout=2 http://database-service:5432

# This should succeed (allowed by network policy)
kubectl run -it --rm test-pod --image=busybox --restart=Never -- \
  wget -O- --timeout=2 http://api-service:8080/health

Real-World Impact

We caught a security incident where a compromised frontend pod attempted to scan internal services. Network Policies blocked the lateral movement completely. The attacker never got beyond the initial foothold.

Metrics:

• Reduced attack surface by 90%+
• Zero lateral movement in incident response scenarios
• Simplified compliance documentation (clear network boundaries)

Pillar 3: Policy Enforcement with Open Policy Agent (OPA)

Moving Security Left with Admission Control

Teleport secures who can access infrastructure. Network Policies secure how pods communicate. But what about preventing insecure configurations from being deployed in the first place?

This is where OPA Gatekeeper comes in. It’s a Kubernetes admission controller that validates every resource before it’s created.

Our OPA Policy Library

We built a library of custom policies enforcing security best practices:

Policy 1: Require Non-Root Containers

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerRequirements
metadata:
  name: must-run-as-nonroot
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces:
      - production
  parameters:
    runAsNonRoot: true

Policy 2: Block Privileged Containers

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPPrivilegedContainer
metadata:
  name: block-privileged-containers
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    excludedNamespaces:
      - kube-system
      - monitoring

Policy 3: Enforce Resource Limits

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerLimits
metadata:
  name: enforce-resource-limits
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces:
      - production
  parameters:
    cpu: "2000m"
    memory: "4Gi"

Policy 4: Block Latest Image Tags

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sBlockLatestTag
metadata:
  name: block-latest-tag
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment", "StatefulSet"]
  parameters:
    message: "Use specific image tags, not 'latest'"

Integrating OPA into CI/CD

We integrated policy validation into our CI/CD pipeline using conftest:

# .gitlab-ci.yml
test:policy:
  stage: test
  image: openpolicyagent/conftest
  script:
    - conftest test k8s/*.yaml --policy opa-policies/

This gives developers immediate feedback if their manifests violate policies—before they ever reach production.

The Developer Experience

We focused on making policy violations helpful, not frustrating:

$ kubectl apply -f deployment.yaml
Error from server (Forbidden): admission webhook denied the request: 
[must-run-as-nonroot] Container 'app' must run as non-root user
[enforce-resource-limits] Container 'app' must specify resource limits

Fix: Add to your container spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
  resources:
    limits:
      cpu: "1000m"
      memory: "2Gi"

Clear error messages reduced support tickets by 70%.

Bringing It All Together: The Zero-Trust Stack

Here’s how Teleport, Network Policies, and OPA work together:

1. Engineer authenticates → Teleport verifies identity with SSO + 2FA
2. Engineer deploys manifest → OPA validates configuration meets security policies
3. Pod starts running → Network Policies restrict which services it can reach
4. All actions logged → Teleport audit trail + Kubernetes audit logs

Architecture Diagram

The following diagram shows how all three components work together to create a comprehensive zero-trust security model:

![Zero-Trust Architecture Overview - See the full interactive diagram in the Mermaid chart]

Key Flow:

1. Engineer authenticates via Teleport (SSO + 2FA)
2. Receives short-lived certificate (8-hour expiry)
3. Submits deployment manifest
4. OPA validates against security policies
5. If approved, pods are created with network isolation
6. Network Policies enforce communication rules at runtime
7. All actions are logged to audit trail

Deployment Flow: From Code to Production

Here’s what happens step-by-step when deploying with our zero-trust stack:

![Zero-Trust Deployment Flow - See the full interactive diagram in the Mermaid chart]

Notice how there are multiple checkpoints where insecure configurations get caught:

• Pre-deployment: CI/CD policy checks with conftest
• Admission time: OPA Gatekeeper validates before creation
• Runtime: Network Policies enforce communication boundaries

This defense-in-depth approach means security isn’t a single point of failure—it’s layered throughout the entire deployment lifecycle.

Measuring Success: Security Metrics

After implementing our zero-trust architecture, we tracked these metrics:

Access Control:

• 100% of infrastructure access now requires 2FA
• Average session duration decreased from “indefinite” to 8 hours (certificate expiry)
• Zero shared credentials in production

Policy Compliance:

• 95% of policy violations caught in CI/CD (before deployment)
• 99.8% of pods running with security contexts
• Zero privileged containers in production namespaces

Incident Response:

• Mean time to revoke access: 30 minutes → 2 minutes
• Lateral movement attempts: 3 blocked by network policies
• Security audit findings: 15 → 0

Lessons Learned & Best Practices

1. Start with Deny-All, Then Explicitly Allow

Don’t try to secure everything at once. Start with one namespace, implement default-deny network policies, and gradually add allow rules as teams request them. This forces teams to think about what communication is actually necessary.

2. Make Policies Self-Service

Create a policy library with clear documentation. When developers violate a policy, give them the exact fix in the error message. We built a Slack bot that auto-responds with solutions to common policy violations.

3. Audit Logs Are Gold

We pipe all Teleport audit logs to our EFK stack and set up alerts for suspicious patterns:

• kubectl exec commands in production
• Access from unusual geographic locations
• Certificate renewal failures (potential compromise)

4. Test Your Policies in Non-Prod First

We learned this the hard way. Rolling out strict network policies to production without testing caused a 3-hour outage when a critical service couldn’t reach its database. Always test in staging first.

5. Educate Your Teams

Zero-trust changes developer workflows. We ran weekly lunch-and-learn sessions showing teams how to use Teleport, explaining why network policies matter, and demonstrating how to fix OPA violations. Security adoption increased dramatically once teams understood the “why.”

The Road Ahead: What’s Next

Zero-trust is a journey, not a destination. Here’s what we’re working on next:

Short-term (Next Quarter):

• Implement mutual TLS (mTLS) for all service-to-service communication using a service mesh (Istio or Linkerd)
• Add OPA policies for container image provenance (only allow signed images)
• Automated compliance reporting dashboard

Medium-term (Next Year):

• Extend Teleport to database access (PostgreSQL, Redis, etc.)
• Runtime threat detection with Falco
• Zero-trust principles for CI/CD pipelines

Conclusion: Zero-Trust Is Worth the Investment

Implementing zero-trust security in Kubernetes isn’t easy. It requires cultural change, tooling investment, and ongoing education. But the security benefits are undeniable:

• Identity-based access eliminates credential sprawl
• Network policies contain breaches before they spread
• Policy enforcement prevents insecure configurations from reaching production

For organizations running Kubernetes at scale, especially in regulated industries, zero-trust isn’t optional—it’s essential.

If you’re starting your zero-trust journey, focus on one pillar at a time. Get Teleport running first (it has immediate ROI), then layer in network policies, and finally add OPA for policy enforcement.

Resources

Tools Mentioned:

• Teleport - Identity-based infrastructure access
• OPA Gatekeeper - Kubernetes policy enforcement
• Kubernetes Network Policies - Official documentation

Further Reading:

• NIST Zero Trust Architecture (SP 800-207)
• Google BeyondCorp: A New Approach to Enterprise Security
• CNCF Cloud Native Security Whitepaper

About the Author: I’m a Senior DevOps and Cloud Engineer with 11+ years of experience leading infrastructure automation and security for fast-paced engineering teams. Currently implementing zero-trust architectures for multi-region Kubernetes clusters serving 10+ engineering teams. Find me on LinkedIn or GitHub.

Questions? Comments? Drop them below or reach out on LinkedIn. I’d love to hear about your zero-trust journey!

The Challenge That Started It All:

“We need to deploy to production. Which Kubernetes cluster do we use? What’s the YAML format again? Who manages the ingress? How do we get SSL certificates?”

It was my second week at Fidelity Information Services (FIS), and I was shocked. We had 150+ CI/CD pipelines, each team managing their own Kubernetes deployments differently. Some used Helm, others used kubectl apply, a few brave souls hand-edited YAML in production. There was no standardization, no platform engineering, no shared infrastructure.

The cost of this chaos:

• Average time to deploy new service: 2-3 weeks
• Infrastructure inconsistencies causing production incidents: Weekly
• Teams reinventing the wheel: Constantly
• Cloud spend: Spiraling out of control

Fast forward 3 months: We built a production-grade, multi-region EKS platform that:

• Achieved 99.99% SLA (only 52 minutes of unplanned downtime per year)
• Reduced deployment time from weeks to < 2 hours
• Saved 15% in cloud costs (~$4,000/month)
• Served 12+ engineering teams with zero infrastructure management overhead
• Won us the “Star Team Award - DevOps 2023”

This is the story of how we built it, the mistakes we made, and the lessons that will save you months of trial and error.

Table of Contents

1. The Vision: Platform Engineering at Scale
2. Architecture: Multi-Region EKS Design
3. The Foundation: Terraform Modules for Reusability
4. Component 1: GitOps with ArgoCD
5. Component 2: Intelligent Load Balancing (NGINX + ALB)
6. Component 3: Auto-Scaling with Karpenter
7. Component 4: Automated TLS with Cert-Manager
8. Multi-Region Strategy: Active-Active Architecture
9. The Platform Effect: How We Enabled 12 Teams
10. Production Metrics: The 99.99% SLA Story
11. Lessons Learned and Best Practices

The Vision: Platform Engineering at Scale

The Problem We Faced

When I joined FIS’s DevOps team as Senior DevOps Engineer, the infrastructure landscape looked like this:

The Reality (Early 2022):

• 150+ independent CI/CD pipelines in Jenkins
• Each team managing their own Kubernetes deployments
• Multiple EKS clusters (dev, staging, prod) per team
• No standard ingress controller (some NGINX, some ALB, some both)
• Manual certificate management
• Fixed-size node groups (massive waste)
• No disaster recovery strategy
• Zero infrastructure as code

The Mandate from Leadership:

“We’re a financial services company processing billions in transactions. We need enterprise-grade infrastructure with 99.99% SLA. And we need it yesterday.”

Our North Star: Self-Service Platform

We envisioned a platform where:

1. Teams deploy apps in < 2 hours, not weeks
2. Infrastructure is code, versioned and reproducible
3. Security is built-in, not bolted on
4. Costs are optimized automatically
5. High availability is default, not extra effort

The key insight: Stop asking teams to become Kubernetes experts. Build a platform that makes the right thing the easy thing.

Architecture: Multi-Region EKS Design

The High-Level Architecture

We designed for multi-region active-active deployment across US-East-1 and US-West-2:

┌─────────────────────────────────────────────────────────────┐
│                      Route 53                               │
│          (Latency-based routing + Health checks)            │
└────────────┬──────────────────────────────┬─────────────────┘
             │                               │
    ┌────────▼────────┐            ┌────────▼────────┐
    │   US-EAST-1     │            │   US-WEST-2     │
    │  (Primary)      │            │   (DR/Active)   │
    └─────────────────┘            └─────────────────┘
             │                               │
    ┌────────▼────────┐            ┌────────▼────────┐
    │  EKS Cluster    │            │  EKS Cluster    │
    │  • 3 AZs        │            │  • 3 AZs        │
    │  • Karpenter    │            │  • Karpenter    │
    │  • ArgoCD       │            │  • ArgoCD       │
    └─────────────────┘            └─────────────────┘
             │                               │
    ┌────────▼────────┐            ┌────────▼────────┐
    │  ALB + NGINX    │            │  ALB + NGINX    │
    │  (Ingress)      │            │  (Ingress)      │
    └─────────────────┘            └─────────────────┘
             │                               │
    ┌────────▼────────┐            ┌────────▼────────┐
    │  Applications   │            │  Applications   │
    │  (GitOps)       │            │  (GitOps)       │
    └─────────────────┘            └─────────────────┘

graph TB
    subgraph Internet["Internet / Users"]
        Users["👥 Global Users"]
    end
    
    subgraph DNS["DNS Layer"]
        R53["Route 53<br/>Latency-based Routing<br/>Health Checks"]
    end
    
    subgraph USEast["US-EAST-1 Primary Region"]
        subgraph EKSEast["EKS Cluster"]
            CPEast["Control Plane<br/>Multi-AZ"]
            
            subgraph NodesEast["Worker Nodes"]
                Karpenter1["Karpenter<br/>Auto-scaling"]
                Pods1["Application Pods<br/>200+ Services"]
            end
        end
        
        subgraph IngressEast["Ingress Layer"]
            ALBEast["ALB<br/>AWS Integration<br/>WAF + ACM"]
            NGINXEast["NGINX Ingress<br/>Advanced Routing"]
        end
        
        subgraph PlatformEast["Platform Services"]
            ArgoEast["ArgoCD<br/>GitOps"]
            CertEast["Cert-Manager<br/>TLS Automation"]
            DNSEast["External-DNS<br/>DNS Automation"]
        end
        
        subgraph DataEast["Data Layer"]
            RDSEast["RDS Primary<br/>Multi-AZ<br/>Read/Write"]
        end
    end
    
    subgraph USWest["US-WEST-2 Secondary Region"]
        subgraph EKSWest["EKS Cluster"]
            CPWest["Control Plane<br/>Multi-AZ"]
            
            subgraph NodesWest["Worker Nodes"]
                Karpenter2["Karpenter<br/>Auto-scaling"]
                Pods2["Application Pods<br/>200+ Services"]
            end
        end
        
        subgraph IngressWest["Ingress Layer"]
            ALBWest["ALB<br/>AWS Integration<br/>WAF + ACM"]
            NGINXWest["NGINX Ingress<br/>Advanced Routing"]
        end
        
        subgraph PlatformWest["Platform Services"]
            ArgoWest["ArgoCD<br/>GitOps"]
            CertWest["Cert-Manager<br/>TLS Automation"]
            DNSWest["External-DNS<br/>DNS Automation"]
        end
        
        subgraph DataWest["Data Layer"]
            RDSWest["RDS Replica<br/>Multi-AZ<br/>Read-only"]
        end
    end
    
    subgraph Control["GitOps Control"]
        Git["Git Repository<br/>Source of Truth<br/>150+ Pipelines"]
    end
    
    Users --> R53
    R53 -->|"Latency-based<br/>+ Health Check"| ALBEast
    R53 -->|"Latency-based<br/>+ Health Check"| ALBWest
    
    ALBEast --> NGINXEast
    ALBWest --> NGINXWest
    
    NGINXEast --> Pods1
    NGINXWest --> Pods2
    
    Git --> ArgoEast
    Git --> ArgoWest
    
    ArgoEast -.->|"Sync Apps"| Pods1
    ArgoWest -.->|"Sync Apps"| Pods2
    
    Karpenter1 -.->|"Provision Nodes"| NodesEast
    Karpenter2 -.->|"Provision Nodes"| NodesWest
    
    CertEast -.->|"TLS Certs"| NGINXEast
    CertWest -.->|"TLS Certs"| NGINXWest
    
    Pods1 --> RDSEast
    Pods2 --> RDSWest
    
    RDSEast -.->|"Async<br/>Replication"| RDSWest
    
    style USEast fill:#e6f3ff
    style USWest fill:#ffe6e6
    style DNS fill:#fff4e6
    style Control fill:#e6ffe6
    style RDSEast fill:#ccffcc
    style RDSWest fill:#ffcccc

Key Design Decisions

1. Multi-Region from Day One

Why? We’re a financial services company. Downtime = money loss + regulatory issues.

Trade-offs:

• ✅ Survive complete AWS region failure
• ✅ Lower latency for geographically distributed users
• ❌ Higher complexity
• ❌ Higher costs (but worth it)

2. GitOps-First with ArgoCD

Why? Declarative infrastructure, audit trail, easy rollbacks.

3. Hybrid Ingress (ALB + NGINX)

Why? ALB for AWS integration (ACM, WAF), NGINX for advanced routing.

4. Karpenter for Autoscaling

Why? Cluster Autoscaler was too slow. We needed sub-minute scaling.

5. Everything in Terraform Modules

Why? Enable teams to self-serve without becoming infrastructure experts.

The Foundation: Terraform Modules for Reusability

graph TB
    subgraph Root["Root Terraform Configuration"]
        Main["main.tf<br/>Environment Config"]
        Vars["variables.tf"]
        Outputs["outputs.tf"]
    end
    
    subgraph Modules["Reusable Modules"]
        subgraph Core["Core Infrastructure"]
            VPC["vpc/"]
            EKS["eks-cluster/"]
        end
        
        subgraph Platform["Platform Components"]
            ArgoCD["argocd/"]
            Karpenter["karpenter/"]
            Ingress["nginx-ingress/"]
            ALB["alb-controller/"]
        end
        
        subgraph Automation["Automation"]
            Cert["cert-manager/"]
            DNS["external-dns/"]
        end
        
        subgraph Observability["Observability"]
            Prom["prometheus/"]
            Graf["grafana/"]
            Loki["loki/"]
        end
        
        subgraph Security["Security"]
            RBAC["rbac/"]
            Policies["policies/"]
            Vault["vault/"]
        end
    end
    
    subgraph Environments["Environments"]
        Dev["dev/<br/>Development"]
        Stag["staging/<br/>Staging"]
        ProdE["prod-us-east-1/<br/>Production East"]
        ProdW["prod-us-west-2/<br/>Production West"]
    end
    
    Main --> VPC
    Main --> EKS
    
    EKS --> ArgoCD
    EKS --> Karpenter
    EKS --> Ingress
    EKS --> ALB
    EKS --> Cert
    EKS --> DNS
    EKS --> Prom
    EKS --> Graf
    EKS --> RBAC
    EKS --> Policies
    
    Modules -.->|"Used by"| Dev
    Modules -.->|"Used by"| Stag
    Modules -.->|"Used by"| ProdE
    Modules -.->|"Used by"| ProdW
    
    style Root fill:#e6f3ff
    style Modules fill:#fff4e6
    style Environments fill:#e6ffe6
    style Core fill:#ffe6e6
    style Platform fill:#f3e6ff
    style Automation fill:#e6fff3
    style Observability fill:#ffffcc
    style Security fill:#ffcccc

The Game-Changer: Reusable Infrastructure Modules

The biggest impact came from building reusable Terraform modules. Instead of every team writing Terraform from scratch, we created a module library:

terraform-eks-platform/
├── modules/
│   ├── eks-cluster/          # Core EKS cluster
│   ├── argocd/               # GitOps deployment
│   ├── nginx-ingress/        # Internal routing
│   ├── alb-controller/       # AWS Load Balancer
│   ├── karpenter/            # Auto-scaling
│   ├── cert-manager/         # TLS automation
│   ├── external-dns/         # DNS automation
│   ├── observability/        # Prometheus + Grafana
│   └── security/             # Policies + RBAC
├── environments/
│   ├── prod-us-east-1/
│   ├── prod-us-west-2/
│   ├── staging/
│   └── dev/
└── README.md

The EKS Cluster Module

Here’s our battle-tested EKS cluster module:

# modules/eks-cluster/main.tf
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = var.cluster_name
  cluster_version = var.kubernetes_version

  # Cluster endpoint configuration
  cluster_endpoint_public_access  = var.enable_public_access
  cluster_endpoint_private_access = true

  # Cluster subnet configuration
  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnet_ids

  # Enable IRSA (IAM Roles for Service Accounts)
  enable_irsa = true

  # Cluster addons
  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent              = true
      before_compute           = true
      service_account_role_arn = module.vpc_cni_irsa.iam_role_arn
      configuration_values = jsonencode({
        env = {
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET       = "1"
        }
      })
    }
    aws-ebs-csi-driver = {
      most_recent              = true
      service_account_role_arn = module.ebs_csi_irsa.iam_role_arn
    }
  }

  # EKS Managed Node Groups (for system workloads)
  eks_managed_node_groups = {
    system = {
      name = "system-node-group"

      instance_types = ["t3.large"]
      capacity_type  = "ON_DEMAND"

      min_size     = 2
      max_size     = 4
      desired_size = 2

      # Taints for system workloads only
      taints = [{
        key    = "CriticalAddonsOnly"
        value  = "true"
        effect = "NO_SCHEDULE"
      }]

      labels = {
        role = "system"
      }

      tags = merge(
        var.tags,
        {
          "karpenter.sh/discovery" = var.cluster_name
        }
      )
    }
  }

  # Cluster security group rules
  cluster_security_group_additional_rules = {
    egress_nodes_ephemeral_ports_tcp = {
      description                = "To node 1025-65535"
      protocol                   = "tcp"
      from_port                  = 1025
      to_port                    = 65535
      type                       = "egress"
      source_node_security_group = true
    }
  }

  # Node security group rules
  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node all ports/protocols"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
    ingress_cluster_all = {
      description                   = "Cluster to node all ports/protocols"
      protocol                      = "-1"
      from_port                     = 0
      to_port                       = 0
      type                          = "ingress"
      source_cluster_security_group = true
    }
  }

  # Cluster encryption
  cluster_encryption_config = {
    provider_key_arn = var.kms_key_arn
    resources        = ["secrets"]
  }

  # Cluster logging
  cluster_enabled_log_types = [
    "api",
    "audit",
    "authenticator",
    "controllerManager",
    "scheduler"
  ]

  # CloudWatch log group retention
  cloudwatch_log_group_retention_in_days = 90

  tags = var.tags
}

# VPC CNI IRSA
module "vpc_cni_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name_prefix      = "${var.cluster_name}-vpc-cni-"
  attach_vpc_cni_policy = true
  vpc_cni_enable_ipv4   = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:aws-node"]
    }
  }

  tags = var.tags
}

# EBS CSI IRSA
module "ebs_csi_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name_prefix      = "${var.cluster_name}-ebs-csi-"
  attach_ebs_csi_policy = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
    }
  }

  tags = var.tags
}

Using the Module: Simple as This

Teams consume our modules like this:

# environments/prod-us-east-1/main.tf
module "eks_platform" {
  source = "git::https://github.com/your-org/terraform-eks-platform//modules/eks-cluster"

  cluster_name       = "prod-eks-us-east-1"
  kubernetes_version = "1.28"

  vpc_id             = module.vpc.vpc_id
  private_subnet_ids = module.vpc.private_subnets

  # Enable all platform components
  enable_argocd         = true
  enable_karpenter      = true
  enable_nginx_ingress  = true
  enable_alb_controller = true
  enable_cert_manager   = true
  enable_external_dns   = true

  # Domain configuration
  domain_name    = "api.fis.com"
  route53_zone_id = data.aws_route53_zone.main.zone_id

  tags = {
    Environment = "production"
    Team        = "platform"
    CostCenter  = "engineering"
  }
}

Result: Any team can spin up a production-ready EKS cluster in < 30 minutes.

Component 1: GitOps with ArgoCD

Why GitOps?

Before ArgoCD:

• Deployments via kubectl apply (scary!)
• No audit trail
• No easy rollbacks
• Configuration drift

After ArgoCD:

• Git is source of truth
• Every change is tracked
• Automatic sync from Git to cluster
• Self-healing applications

ArgoCD Module

# modules/argocd/main.tf
resource "helm_release" "argocd" {
  name             = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  namespace        = "argocd"
  create_namespace = true
  version          = var.argocd_version

  values = [
    yamlencode({
      global = {
        domain = "argocd.${var.domain_name}"
      }

      server = {
        replicas = 2

        ingress = {
          enabled = true
          ingressClassName = "nginx"
          annotations = {
            "cert-manager.io/cluster-issuer" = "letsencrypt-prod"
            "nginx.ingress.kubernetes.io/backend-protocol" = "HTTPS"
            "nginx.ingress.kubernetes.io/ssl-passthrough" = "true"
          }
          hosts = ["argocd.${var.domain_name}"]
          tls = [{
            secretName = "argocd-tls"
            hosts      = ["argocd.${var.domain_name}"]
          }]
        }

        # High availability
        metrics = {
          enabled = true
          serviceMonitor = {
            enabled = true
          }
        }
      }

      controller = {
        replicas = 2
        metrics = {
          enabled = true
          serviceMonitor = {
            enabled = true
          }
        }
      }

      repoServer = {
        replicas = 2
        metrics = {
          enabled = true
          serviceMonitor = {
            enabled = true
          }
        }
      }

      # Application controller settings
      configs = {
        cm = {
          "timeout.reconciliation" = "180s"
          "application.instanceLabelKey" = "argocd.argoproj.io/instance"
        }
        params = {
          "server.insecure" = "false"
          "application.namespaces" = "*"
        }
      }
    })
  ]

  depends_on = [
    kubernetes_namespace.argocd
  ]
}

# Bootstrap ArgoCD with App of Apps pattern
resource "kubectl_manifest" "argocd_apps" {
  yaml_body = yamlencode({
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"
    metadata = {
      name      = "platform-apps"
      namespace = "argocd"
    }
    spec = {
      project = "default"
      source = {
        repoURL        = var.git_repo_url
        targetRevision = "main"
        path           = "argocd-apps/${var.environment}"
      }
      destination = {
        server    = "https://kubernetes.default.svc"
        namespace = "argocd"
      }
      syncPolicy = {
        automated = {
          prune    = true
          selfHeal = true
        }
        syncOptions = [
          "CreateNamespace=true"
        ]
      }
    }
  })

  depends_on = [helm_release.argocd]
}

The Impact of GitOps

Before ArgoCD:

• Manual deployment: 45 minutes
• Rollback time: 30 minutes (if you remember how)
• Configuration drift: Common
• Audit trail: None

After ArgoCD:

• Automated deployment: < 5 minutes
• Rollback time: < 2 minutes (git revert + sync)
• Configuration drift: Impossible (auto-sync)
• Audit trail: Complete Git history

Real Example:

One night, a developer accidentally deleted a production namespace. With ArgoCD:

1. ArgoCD detected the drift in 30 seconds
2. Auto-sync recreated all resources from Git
3. Application back online in < 3 minutes
4. Zero manual intervention

Without ArgoCD, this would have been a 2-hour incident.

Component 2: Intelligent Load Balancing (NGINX + ALB)

Why Both NGINX and ALB?

The Question Everyone Asks: “Why two ingress controllers?”

The Answer: They serve different purposes.

ALB (AWS Load Balancer Controller):

• ✅ Native AWS integration
• ✅ WAF support for security
• ✅ ACM certificate integration
• ✅ Health checks and target groups
• ❌ Limited routing capabilities
• ❌ No request/response manipulation

NGINX Ingress:

• ✅ Advanced routing (canary, A/B testing)
• ✅ Request/response rewriting
• ✅ Rate limiting, auth
• ✅ WebSocket support
• ❌ No native AWS services integration

Our Solution: Use both!

Internet → ALB → NGINX → Services
         ↓
       WAF, ACM, SSL

ALB Controller Module

# modules/alb-controller/main.tf
resource "helm_release" "aws_load_balancer_controller" {
  name       = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-load-balancer-controller"
  namespace  = "kube-system"
  version    = var.alb_controller_version

  set {
    name  = "clusterName"
    value = var.cluster_name
  }

  set {
    name  = "serviceAccount.create"
    value = "true"
  }

  set {
    name  = "serviceAccount.name"
    value = "aws-load-balancer-controller"
  }

  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = module.alb_controller_irsa.iam_role_arn
  }

  set {
    name  = "region"
    value = var.region
  }

  set {
    name  = "vpcId"
    value = var.vpc_id
  }

  set {
    name  = "enableWaf"
    value = "true"
  }

  set {
    name  = "enableWafv2"
    value = "true"
  }

  set {
    name  = "enableShield"
    value = "false"  # Enable if you have Shield Advanced
  }

  depends_on = [module.alb_controller_irsa]
}

# IRSA for ALB Controller
module "alb_controller_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name_prefix                              = "${var.cluster_name}-alb-controller-"
  attach_load_balancer_controller_policy        = true
  attach_load_balancer_controller_targetgroup_binding_only_policy = false

  oidc_providers = {
    main = {
      provider_arn               = var.oidc_provider_arn
      namespace_service_accounts = ["kube-system:aws-load-balancer-controller"]
    }
  }

  tags = var.tags
}

NGINX Ingress Module

# modules/nginx-ingress/main.tf
resource "helm_release" "nginx_ingress" {
  name       = "ingress-nginx"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  namespace  = "ingress-nginx"
  version    = var.nginx_version

  create_namespace = true

  values = [
    yamlencode({
      controller = {
        replicaCount = 3

        # Use NLB as frontend for NGINX
        service = {
          annotations = {
            "service.beta.kubernetes.io/aws-load-balancer-type" = "nlb"
            "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled" = "true"
            "service.beta.kubernetes.io/aws-load-balancer-backend-protocol" = "tcp"
          }
        }

        # Resource limits
        resources = {
          limits = {
            cpu    = "1000m"
            memory = "1Gi"
          }
          requests = {
            cpu    = "500m"
            memory = "512Mi"
          }
        }

        # Pod anti-affinity for HA
        affinity = {
          podAntiAffinity = {
            requiredDuringSchedulingIgnoredDuringExecution = [{
              labelSelector = {
                matchLabels = {
                  "app.kubernetes.io/name"      = "ingress-nginx"
                  "app.kubernetes.io/component" = "controller"
                }
              }
              topologyKey = "kubernetes.io/hostname"
            }]
          }
        }

        # Metrics for monitoring
        metrics = {
          enabled = true
          serviceMonitor = {
            enabled = true
          }
        }

        # Configuration
        config = {
          use-forwarded-headers = "true"
          compute-full-forwarded-for = "true"
          use-proxy-protocol = "false"
          
          # Performance tuning
          worker-processes = "auto"
          max-worker-connections = "16384"
          
          # Security headers
          hide-headers = "Server,X-Powered-By"
          ssl-protocols = "TLSv1.2 TLSv1.3"
          ssl-ciphers = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256"
          
          # Rate limiting
          limit-req-status-code = "429"
          limit-conn-status-code = "429"
        }

        # Autoscaling
        autoscaling = {
          enabled = true
          minReplicas = 3
          maxReplicas = 10
          targetCPUUtilizationPercentage = 75
        }
      }
    })
  ]
}

Real-World Example: Canary Deployments

With NGINX Ingress, we enabled advanced deployment strategies:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payment-api-canary
  annotations:
    # Send 10% of traffic to canary version
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  rules:
  - host: api.fis.com
    http:
      paths:
      - path: /payments
        pathType: Prefix
        backend:
          service:
            name: payment-api-canary
            port:
              number: 80

Result: We rolled out a critical payment API update to 10% of traffic first, caught a bug, fixed it, then rolled out to 100%. Zero customer impact.

Component 3: Auto-Scaling with Karpenter

sequenceDiagram
    participant User as User
    participant App as Application
    participant K8s as Kubernetes API
    participant Karpenter as Karpenter
    participant AWS as AWS EC2
    participant Pod as New Pod
    
    Note over User,Pod: Traffic Spike Scenario
    
    rect rgb(230, 245, 255)
        Note over User,App: Phase 1: Load Increase
        User->>App: Increased traffic
        App->>App: Current pods at 80% CPU
        App->>K8s: HPA scales deployment
        K8s->>K8s: Create new pod replicas
    end
    
    rect rgb(255, 240, 230)
        Note over K8s,Karpenter: Phase 2: Pending Pods
        K8s->>K8s: Pods in Pending state<br/>(no capacity)
        Karpenter->>K8s: Watch for pending pods
        Note over Karpenter: Detected: 5 pending pods<br/>Need: 8 CPU, 16GB RAM
    end
    
    rect rgb(240, 255, 240)
        Note over Karpenter,AWS: Phase 3: Node Provisioning (< 60s)
        Karpenter->>Karpenter: Calculate optimal instance:<br/>• Spot vs On-Demand<br/>• Instance family (c/m/r)<br/>• Generation (6+)<br/>• Size (xlarge/2xlarge)
        
        Karpenter->>AWS: Launch EC2 instance<br/>Type: m6i.2xlarge (Spot)<br/>Cost: 70% cheaper
        
        AWS->>AWS: Instance launching
        Note over AWS: Boot time: ~30-45 seconds
        
        AWS->>Karpenter: Instance running
        Karpenter->>K8s: Register new node
    end
    
    rect rgb(255, 245, 230)
        Note over K8s,Pod: Phase 4: Pod Scheduling
        K8s->>K8s: Schedule pending pods<br/>to new node
        K8s->>Pod: Start pods on new node
        Pod->>Pod: Containers starting
        Pod->>K8s: Pods Ready
    end
    
    rect rgb(245, 240, 255)
        Note over App,User: Phase 5: Serving Traffic
        K8s->>App: New replicas ready
        App->>User: Handle increased load
        Note over User,Pod: Total time: < 60 seconds<br/>User experience: Seamless
    end
    
    Note over User,Pod: Old Solution (Cluster Autoscaler): 5-10 minutes<br/>New Solution (Karpenter): < 60 seconds

Why Karpenter Changed Everything

Cluster Autoscaler problems:

• Slow to scale (5-10 minutes)
• Required pre-defined node groups
• Couldn’t mix instance types efficiently
• Fragmentation issues

Karpenter advantages:

• Fast scaling (< 60 seconds)
• No pre-defined node groups needed
• Automatically selects best instance types
• Bin-packing optimization

Our Karpenter Module

# modules/karpenter/main.tf
resource "helm_release" "karpenter" {
  name       = "karpenter"
  repository = "oci://public.ecr.aws/karpenter"
  chart      = "karpenter"
  namespace  = "karpenter"
  version    = var.karpenter_version

  create_namespace = true

  values = [
    yamlencode({
      serviceAccount = {
        annotations = {
          "eks.amazonaws.com/role-arn" = module.karpenter_irsa.iam_role_arn
        }
      }

      settings = {
        clusterName     = var.cluster_name
        clusterEndpoint = var.cluster_endpoint
        interruptionQueueName = aws_sqs_queue.karpenter.name
      }

      controller = {
        resources = {
          requests = {
            cpu    = "1"
            memory = "1Gi"
          }
          limits = {
            cpu    = "1"
            memory = "1Gi"
          }
        }

        # High availability
        replicas = 2
        affinity = {
          podAntiAffinity = {
            requiredDuringSchedulingIgnoredDuringExecution = [{
              labelSelector = {
                matchLabels = {
                  "app.kubernetes.io/name" = "karpenter"
                }
              }
              topologyKey = "kubernetes.io/hostname"
            }]
          }
        }
      }
    })
  ]

  depends_on = [
    module.karpenter_irsa
  ]
}

# Karpenter NodePool
resource "kubectl_manifest" "karpenter_node_pool" {
  yaml_body = yamlencode({
    apiVersion = "karpenter.sh/v1beta1"
    kind       = "NodePool"
    metadata = {
      name = "general-purpose"
    }
    spec = {
      template = {
        spec = {
          requirements = [
            {
              key      = "karpenter.sh/capacity-type"
              operator = "In"
              values   = ["spot", "on-demand"]
            },
            {
              key      = "kubernetes.io/arch"
              operator = "In"
              values   = ["amd64"]
            },
            {
              key      = "karpenter.k8s.aws/instance-category"
              operator = "In"
              values   = ["c", "m", "r"]
            },
            {
              key      = "karpenter.k8s.aws/instance-generation"
              operator = "Gt"
              values   = ["5"]  # Gen 6+ instances only
            }
          ]
          
          nodeClassRef = {
            name = "default"
          }

          # Taints for specific workloads
          taints = []

          # Labels
          labels = {
            "workload-type" = "general"
          }
        }
      }

      # Disruption budget
      disruption = {
        consolidationPolicy = "WhenUnderutilized"
        expireAfter         = "720h"  # 30 days
      }

      # Limits
      limits = {
        cpu    = "1000"
        memory = "1000Gi"
      }
    }
  })

  depends_on = [
    helm_release.karpenter,
    kubectl_manifest.karpenter_node_class
  ]
}

# Karpenter EC2NodeClass
resource "kubectl_manifest" "karpenter_node_class" {
  yaml_body = yamlencode({
    apiVersion = "karpenter.k8s.aws/v1beta1"
    kind       = "EC2NodeClass"
    metadata = {
      name = "default"
    }
    spec = {
      amiFamily = "AL2"
      role      = var.karpenter_node_role_name
      
      subnetSelectorTerms = [{
        tags = {
          "karpenter.sh/discovery" = var.cluster_name
        }
      }]

      securityGroupSelectorTerms = [{
        tags = {
          "karpenter.sh/discovery" = var.cluster_name
        }
      }]

      # EBS volume settings
      blockDeviceMappings = [{
        deviceName = "/dev/xvda"
        ebs = {
          volumeSize          = "100Gi"
          volumeType          = "gp3"
          encrypted           = true
          kmsKeyID            = var.kms_key_arn
          deleteOnTermination = true
        }
      }]

      # User data for node bootstrap
      userData = base64encode(<<-EOT
        #!/bin/bash
        /etc/eks/bootstrap.sh ${var.cluster_name}
      EOT
      )

      # Instance profile
      instanceProfile = var.karpenter_instance_profile_name

      # Metadata options
      metadataOptions = {
        httpEndpoint            = "enabled"
        httpProtocolIPv6        = "disabled"
        httpPutResponseHopLimit = 2
        httpTokens              = "required"
      }

      tags = merge(
        var.tags,
        {
          "karpenter.sh/discovery" = var.cluster_name
        }
      )
    }
  })

  depends_on = [helm_release.karpenter]
}

The Karpenter Impact

Before Karpenter (Cluster Autoscaler):

• Pod pending time: 5-10 minutes
• Wasted capacity: ~30% (over-provisioning)
• Cost: High (fixed node groups)

After Karpenter:

• Pod pending time: < 60 seconds
• Wasted capacity: < 5% (bin-packing)
• Cost: 15% reduction (~$4K/month savings)

Real Example:

During a traffic spike (Black Friday sale):

• Before: Took 10 minutes to scale, some requests timed out
• After: Karpenter launched nodes in 45 seconds, zero failed requests

Component 4: Automated TLS with Cert-Manager

Why Cert-Manager is Essential

Manual certificate management:

• 😭 Request cert from CA manually
• 😭 Download cert files
• 😭 Create Kubernetes secrets
• 😭 Remember to renew in 90 days
• 😭 Repeat for every service

Cert-Manager:

• 😎 Automatic Let’s Encrypt integration
• 😎 Auto-renewal before expiration
• 😎 One annotation = automatic HTTPS
• 😎 Sleep better at night

Cert-Manager Module

# modules/cert-manager/main.tf
resource "helm_release" "cert_manager" {
  name       = "cert-manager"
  repository = "https://charts.jetstack.io"
  chart      = "cert-manager"
  namespace  = "cert-manager"
  version    = var.cert_manager_version

  create_namespace = true

  set {
    name  = "installCRDs"
    value = "true"
  }

  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = module.cert_manager_irsa.iam_role_arn
  }

  set {
    name  = "global.leaderElection.namespace"
    value = "cert-manager"
  }

  # High availability
  set {
    name  = "replicaCount"
    value = "2"
  }

  set {
    name  = "webhook.replicaCount"
    value = "2"
  }

  set {
    name  = "cainjector.replicaCount"
    value = "2"
  }

  # Monitoring
  set {
    name  = "prometheus.enabled"
    value = "true"
  }

  depends_on = [module.cert_manager_irsa]
}

# Let's Encrypt ClusterIssuer (Production)
resource "kubectl_manifest" "letsencrypt_prod" {
  yaml_body = yamlencode({
    apiVersion = "cert-manager.io/v1"
    kind       = "ClusterIssuer"
    metadata = {
      name = "letsencrypt-prod"
    }
    spec = {
      acme = {
        server = "https://acme-v02.api.letsencrypt.org/directory"
        email  = var.acme_email
        privateKeySecretRef = {
          name = "letsencrypt-prod"
        }
        solvers = [{
          dns01 = {
            route53 = {
              region       = var.region
              hostedZoneID = var.route53_zone_id
            }
          }
        }]
      }
    }
  })

  depends_on = [helm_release.cert_manager]
}

# Let's Encrypt ClusterIssuer (Staging - for testing)
resource "kubectl_manifest" "letsencrypt_staging" {
  yaml_body = yamlencode({
    apiVersion = "cert-manager.io/v1"
    kind       = "ClusterIssuer"
    metadata = {
      name = "letsencrypt-staging"
    }
    spec = {
      acme = {
        server = "https://acme-staging-v02.api.letsencrypt.org/directory"
        email  = var.acme_email
        privateKeySecretRef = {
          name = "letsencrypt-staging"
        }
        solvers = [{
          dns01 = {
            route53 = {
              region       = var.region
              hostedZoneID = var.route53_zone_id
            }
          }
        }]
      }
    }
  })

  depends_on = [helm_release.cert_manager]
}

How Teams Use It

Application teams just add one annotation to their Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"  # This is all you need!
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.fis.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app
            port:
              number: 80
  tls:
  - hosts:
    - myapp.fis.com
    secretName: myapp-tls  # Cert-Manager creates this automatically

Cert-Manager automatically:

1. Requests certificate from Let’s Encrypt
2. Completes DNS-01 challenge via Route53
3. Creates Kubernetes TLS secret
4. Renews certificate 30 days before expiry

Zero manual work. Ever.

Multi-Region Strategy: Active-Active Architecture

stateDiagram-v2
    [*] --> Normal: Normal Operations
    
    state Normal {
        [*] --> BothHealthy: Both regions healthy
        BothHealthy --> Routing: Route 53 latency-based
        
        state Routing {
            [*] --> East60: 60% → US-EAST-1
            [*] --> West40: 40% → US-WEST-2
        }
    }
    
    Normal --> Incident: Region Failure Detected
    
    state Incident {
        [*] --> HealthCheck: Route 53 health check fails
        HealthCheck --> Alarm: CloudWatch alarm fires
        Alarm --> PagerDuty: Alert on-call engineer
    }
    
    Incident --> AutoFailover: Automatic Failover
    
    state AutoFailover {
        [*] --> DNSUpdate: Route 53 removes failed region
        DNSUpdate --> TrafficShift: 100% → US-WEST-2
        TrafficShift --> Verify: Verify traffic flowing
        
        note right of TrafficShift
            Time: < 2 minutes
            TTL: 60 seconds
            No manual intervention
        end note
    }
    
    AutoFailover --> Decision: Database Status?
    
    Decision --> AppOnly: Application-only failure
    Decision --> DBFailure: Database failure
    
    state AppOnly {
        [*] --> AppRecover: Apps auto-heal in 5 min
        AppRecover --> BackOnline: Region recovers
    }
    
    state DBFailure {
        [*] --> PromoteReplica: Manual: Promote RDS replica
        PromoteReplica --> UpdateConfig: Update connection strings
        UpdateConfig --> RestartApps: Restart affected pods
        
        note right of PromoteReplica
            Time: 15-30 minutes
            Manual step required
            RTO: < 30 minutes
        end note
    }
    
    AppOnly --> [*]: Restored
    DBFailure --> [*]: Restored
    
    note right of AutoFailover
        99.99% SLA maintained:
        • Automated DNS failover
        • Multi-region active-active
        • RTO: < 30 minutes
        • RPO: < 5 minutes
    end note

Why Multi-Region?

The Business Requirement:

“We’re a financial services company. We process billions in transactions. We cannot have a single point of failure.”

Our SLA target: 99.99%

• Allowed downtime: 52.56 minutes/year
• Reality: AWS region outages happen 2-3 times/year
• Solution: Multi-region active-active

The Architecture

                    ┌─────────────────┐
                    │   Route 53      │
                    │  Health Checks  │
                    │  Latency-based  │
                    └────────┬────────┘
                             │
                ┌────────────┴────────────┐
                │                         │
       ┌────────▼────────┐       ┌───────▼─────────┐
       │   US-EAST-1     │       │   US-WEST-2     │
       │   (Primary)     │       │   (Secondary)   │
       └────────┬────────┘       └────────┬────────┘
                │                         │
       ┌────────▼────────┐       ┌───────▼─────────┐
       │  EKS Cluster    │       │  EKS Cluster    │
       │  + Platform     │       │  + Platform     │
       └────────┬────────┘       └────────┬────────┘
                │                         │
       ┌────────▼────────┐       ┌───────▼─────────┐
       │  RDS Primary    │◄──────┤ RDS Read Replica│
       │  (Read/Write)   │       │  (Read-only)    │
       └─────────────────┘       └─────────────────┘

Multi-Region Terraform Setup

# environments/multi-region/main.tf
# Primary Region (US-EAST-1)
module "eks_primary" {
  source = "../../modules/eks-platform"

  providers = {
    aws        = aws.us-east-1
    kubernetes = kubernetes.us-east-1
  }

  cluster_name = "prod-eks-us-east-1"
  region       = "us-east-1"

  # All platform components enabled
  enable_argocd         = true
  enable_karpenter      = true
  enable_nginx_ingress  = true
  enable_alb_controller = true
  enable_cert_manager   = true
  enable_external_dns   = true

  # Multi-region specific
  is_primary_region = true
  peer_region       = "us-west-2"

  tags = local.common_tags
}

# Secondary Region (US-WEST-2)
module "eks_secondary" {
  source = "../../modules/eks-platform"

  providers = {
    aws        = aws.us-west-2
    kubernetes = kubernetes.us-west-2
  }

  cluster_name = "prod-eks-us-west-2"
  region       = "us-west-2"

  # All platform components enabled
  enable_argocd         = true
  enable_karpenter      = true
  enable_nginx_ingress  = true
  enable_alb_controller = true
  enable_cert_manager   = true
  enable_external_dns   = true

  # Multi-region specific
  is_primary_region = false
  peer_region       = "us-east-1"

  tags = local.common_tags
}

# Route53 Health Checks & Routing
module "route53_multiregion" {
  source = "../../modules/route53-multiregion"

  domain_name = "api.fis.com"
  
  # Primary region endpoint
  primary_endpoint = module.eks_primary.ingress_endpoint
  primary_region   = "us-east-1"

  # Secondary region endpoint
  secondary_endpoint = module.eks_secondary.ingress_endpoint
  secondary_region   = "us-west-2"

  # Health check configuration
  health_check_path     = "/healthz"
  health_check_interval = 10
  health_check_timeout  = 5

  tags = local.common_tags
}

Active-Active Traffic Management

# modules/route53-multiregion/main.tf
resource "aws_route53_health_check" "primary" {
  fqdn              = var.primary_endpoint
  port              = 443
  type              = "HTTPS"
  resource_path     = var.health_check_path
  request_interval  = var.health_check_interval
  failure_threshold = 3

  tags = merge(
    var.tags,
    {
      Name = "${var.domain_name}-primary-health-check"
    }
  )
}

resource "aws_route53_health_check" "secondary" {
  fqdn              = var.secondary_endpoint
  port              = 443
  type              = "HTTPS"
  resource_path     = var.health_check_path
  request_interval  = var.health_check_interval
  failure_threshold = 3

  tags = merge(
    var.tags,
    {
      Name = "${var.domain_name}-secondary-health-check"
    }
  )
}

# Latency-based routing with health checks
resource "aws_route53_record" "primary" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = var.domain_name
  type    = "A"

  alias {
    name                   = var.primary_endpoint
    zone_id                = var.primary_zone_id
    evaluate_target_health = true
  }

  set_identifier  = "primary"
  health_check_id = aws_route53_health_check.primary.id
  latency_routing_policy {
    region = var.primary_region
  }
}

resource "aws_route53_record" "secondary" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = var.domain_name
  type    = "A"

  alias {
    name                   = var.secondary_endpoint
    zone_id                = var.secondary_zone_id
    evaluate_target_health = true
  }

  set_identifier  = "secondary"
  health_check_id = aws_route53_health_check.secondary.id
  latency_routing_policy {
    region = var.secondary_region
  }
}

Database Replication

# Primary RDS
resource "aws_db_instance" "primary" {
  provider = aws.us-east-1

  identifier     = "fis-prod-primary"
  engine         = "postgres"
  engine_version = "15.3"
  instance_class = "db.r6g.2xlarge"

  allocated_storage = 1000
  storage_type      = "gp3"
  storage_encrypted = true

  multi_az = true  # Within region HA
  
  backup_retention_period = 7
  backup_window          = "03:00-04:00"

  tags = var.tags
}

# Read Replica in Secondary Region
resource "aws_db_instance" "replica" {
  provider = aws.us-west-2

  replicate_source_db = aws_db_instance.primary.arn

  identifier     = "fis-prod-replica"
  instance_class = "db.r6g.2xlarge"

  # Can be promoted to standalone if primary fails
  skip_final_snapshot = false
  
  tags = var.tags
}

Multi-Region Failover Process

Scenario: US-EAST-1 region failure

Automated (< 5 minutes):

1. Route53 health checks fail in us-east-1
2. Route53 automatically routes traffic to us-west-2
3. Applications continue running in us-west-2

Manual (if database failover needed):

# Promote read replica to primary
aws rds promote-read-replica \
  --db-instance-identifier fis-prod-replica \
  --region us-west-2

# Update application config
kubectl set env deployment/api \
  DATABASE_HOST=fis-prod-replica.us-west-2.rds.amazonaws.com

RTO: < 30 minutes (including database promotion) RPO: < 5 minutes (RDS replication lag)

The Platform Effect: How We Enabled 12 Teams

graph LR
    subgraph DevTeam["Development Team"]
        Dev["👨‍💻 Developer"]
    end
    
    subgraph GitOps["GitOps Workflow"]
        Git["Git Repository"]
        PR["Pull Request"]
        Merge["Merge to Main"]
    end
    
    subgraph ArgoCD["ArgoCD Layer"]
        ArgoDet["ArgoCD<br/>Detects Change"]
        ArgoSync["Sync to Cluster"]
    end
    
    subgraph K8s["Kubernetes Cluster"]
        API["K8s API Server"]
        
        subgraph Apps["Applications"]
            Deploy["Deployment<br/>Created/Updated"]
            Pods["Pods Starting"]
        end
    end
    
    subgraph Karpenter["Karpenter Auto-Scaling"]
        KarpDet["Detect Pending<br/>Pods"]
        KarpProv["Provision<br/>New Nodes"]
    end
    
    subgraph Ingress["Ingress Layer"]
        CertReq["Cert-Manager<br/>Requests TLS"]
        LE["Let's Encrypt<br/>Issues Cert"]
        IngressCfg["Ingress<br/>Configured"]
    end
    
    subgraph DNS["DNS Layer"]
        ExtDNS["External-DNS<br/>Detects Ingress"]
        R53Update["Route 53<br/>Record Created"]
    end
    
    subgraph Users["End Users"]
        User["👥 Users<br/>Access App"]
    end
    
    Dev -->|1. Push Code| Git
    Git -->|2. Create| PR
    PR -->|3. Review & Approve| Merge
    Merge -->|4. Trigger| ArgoDet
    ArgoDet -->|5. Sync| ArgoSync
    ArgoSync -->|6. Apply| API
    API -->|7. Create| Deploy
    Deploy -->|8. Request| Pods
    
    Pods -.->|9. Pending| KarpDet
    KarpDet -.->|10. Provision| KarpProv
    KarpProv -.->|11. Nodes Ready| Pods
    
    Deploy -->|12. Ingress Created| IngressCfg
    IngressCfg -->|13. Request Cert| CertReq
    CertReq -->|14. ACME Challenge| LE
    LE -->|15. Issue Certificate| IngressCfg
    
    IngressCfg -->|16. Detect| ExtDNS
    ExtDNS -->|17. Create Record| R53Update
    
    User -->|18. Access| R53Update
    R53Update -->|19. Route| IngressCfg
    IngressCfg -->|20. Forward| Pods
    
    style DevTeam fill:#e6f3ff
    style GitOps fill:#fff4e6
    style ArgoCD fill:#ffe6f3
    style Karpenter fill:#f3e6ff
    style Ingress fill:#e6fff3
    style DNS fill:#ffffcc

Before: The Chaos

Team Experience (2022):

1. Team needs new EKS cluster
2. DevOps engineer spends 2 weeks configuring everything manually
3. No standardization—each cluster is different
4. Teams struggle with deployments
5. No one knows who to ask for help

DevOps Team:

• Constant interruptions
• Can’t scale support
• Fighting fires daily
• No time for innovation

After: Self-Service Platform

Team Experience (2023):

1. Team fills out simple form or runs Terraform module
2. Platform deploys in < 30 minutes
3. Everything is pre-configured (ArgoCD, ingress, autoscaling, monitoring)
4. Team just deploys apps via Git
5. Clear documentation and support

DevOps Team:

• Proactive infrastructure improvements
• Scaling to 12+ teams easily
• Minimal support burden
• Focus on innovation

The Self-Service Portal

We built a simple portal for teams:

# cluster-request.yaml
apiVersion: platform.fis.com/v1
kind: ClusterRequest
metadata:
  name: payments-team-cluster
spec:
  team: payments
  environment: production
  region: us-east-1
  
  # All platform components auto-enabled
  platformComponents:
    argocd: true
    karpenter: true
    monitoring: true
    logging: true
  
  # Team-specific configuration
  domains:
    - payments-api.fis.com
    - payments-web.fis.com
  
  # Cost allocation
  costCenter: "CC-1234"
  
  # Contact
  owner: "payments-team@fis.com"

Automation does the rest:

1. Terraform creates cluster
2. ArgoCD bootstraps platform components
3. DNS records created
4. TLS certificates provisioned
5. Monitoring dashboards created
6. Team gets access

The Numbers

Platform Impact (18 months):

• Teams onboarded: 12
• Services deployed: 200+
• Manual interventions/month: < 5
• Average deployment time: < 5 minutes
• Platform uptime: 99.99%
• Cost savings: 15% ($4K/month)

Team Velocity:

• Time to production (new service): 2 weeks → 2 hours
• Deployment frequency: Weekly → Multiple per day
• Incident response time: 45 min → < 5 minutes

Production Metrics: The 99.99% SLA Story

graph LR
    subgraph Before["Before Optimization"]
        Fixed["Fixed Node Groups<br/>Always On<br/>$56K/month"]
        Waste["30% Waste<br/>Over-provisioning"]
        Manual["Manual Scaling<br/>Slow Response"]
    end
    
    subgraph Changes["Optimization Changes"]
        Karp["Implemented<br/>Karpenter"]
        Spot["Enabled Spot<br/>Instances"]
        RightSize["Right-sized<br/>Workloads"]
        ILM["Added Index<br/>Lifecycle"]
    end
    
    subgraph After["After Optimization"]
        Dynamic["Dynamic Scaling<br/>On-Demand<br/>$48K/month"]
        Efficient["< 5% Waste<br/>Bin-packing"]
        Auto["Auto-scaling<br/>< 60s Response"]
    end
    
    subgraph Impact["Business Impact"]
        Savings["$8K/month saved<br/>15% reduction"]
        Scale["3x scaling<br/>Same cost"]
        SLA["99.99% SLA<br/>Maintained"]
    end
    
    Fixed --> Karp
    Waste --> Spot
    Manual --> RightSize
    
    Karp --> Dynamic
    Spot --> Efficient
    RightSize --> Auto
    
    Dynamic --> Savings
    Efficient --> Scale
    Auto --> SLA
    
    style Before fill:#ffcccc
    style Changes fill:#ffffcc
    style After fill:#ccffcc
    style Impact fill:#ccffff

Measuring Success

We track these metrics religiously:

Availability Metrics:

• Cluster uptime: 99.99%
• Application availability: 99.95%
• Unplanned downtime (2023): 47 minutes (under 52-minute budget!)

Performance Metrics:

• Average pod start time: < 30 seconds
• Karpenter scale-up time: < 60 seconds
• ArgoCD sync time: < 2 minutes

Cost Metrics:

• Monthly EKS costs: $48K
• Cost per cluster: $4K/month
• Savings vs. initial estimate: 15%

The 99.99% SLA Breakdown

How we achieved it:

1. Multi-Region (75% of reliability)
   • Survives entire region failures
   • Automated failover via Route53
2. Multi-AZ within Region (15%)
   • Survives AZ failures
   • EKS control plane is multi-AZ by default
   • Node groups spread across 3 AZs
3. Auto-Healing (5%)
   • Kubernetes self-healing
   • Karpenter replaces unhealthy nodes
   • ArgoCD self-healing applications
4. Monitoring & Alerting (5%)
   • Catch issues before customers do
   • Automated remediation where possible

Our Only Significant Outage (2023)

Date: August 15, 2023, 2:17 AM Duration: 47 minutes Impact: Payment API in us-east-1 only

What Happened:

1. AWS us-east-1 had partial AZ failure
2. Our primary database (RDS) in affected AZ
3. RDS took 12 minutes to failover to standby AZ
4. Applications recovered automatically

What Saved Us:

• Multi-AZ RDS configuration
• US-WEST-2 cluster unaffected
• Route53 routed 60% of traffic to us-west-2
• Only 40% of users affected

Lessons:

• RDS Multi-AZ failover is slower than expected
• Consider Aurora Global Database for faster cross-region failover
• Route53 health checks worked perfectly

Actions Taken:

• Migrated to Aurora Global Database (sub-second failover)
• Added cross-region read replicas
• Improved monitoring for partial AZ failures

Result: No significant outages since August 2023 (15+ months)

Lessons Learned and Best Practices

After 18 months running production EKS at scale, here’s what we learned:

1. Invest in Reusable Modules Early

The Mistake: Initially, each team wrote their own Terraform.

The Fix: Built a library of reusable modules.

The Impact:

• New cluster deployment: 2 weeks → 30 minutes
• Consistency: 100% (all clusters identical)
• Maintenance: Centralized updates

Best Practice: Start with modules day one. Your future self will thank you.

2. GitOps is Non-Negotiable

The Mistake: Manual kubectl apply deployments early on.

The Fix: Enforced GitOps via ArgoCD.

The Impact:

• Audit trail: Complete
• Rollbacks: < 2 minutes
• Configuration drift: Impossible

Best Practice: If it’s not in Git, it doesn’t exist in production.

3. Multi-Region from Day One

The Mistake: “We’ll add multi-region later”

The Reality: Adding multi-region later is 10x harder.

The Fix: Built multi-region from day one.

Best Practice: Even if you don’t activate both regions initially, architect for multi-region from the start.

4. Karpenter > Cluster Autoscaler

The Data:

• Cluster Autoscaler: 5-10 minute scale-up
• Karpenter: < 60 second scale-up

The Impact:

• Better user experience
• Cost savings (less over-provisioning)
• Simpler configuration

Best Practice: Use Karpenter if you’re on EKS. Period.

5. Monitoring is Infrastructure

The Mistake: Treated monitoring as an afterthought.

The Fix: Baked in Prometheus + Grafana from day one.

The Impact:

• Catch issues before customers
• Data-driven optimization
• Confidence in SLAs

Best Practice: Deploy monitoring before applications.

6. Cost Optimization is Continuous

Our Approach:

• Weekly cost reviews
• Karpenter for right-sizing
• Spot instances where possible
• Reserved Instances for base load

The Impact: 15% cost reduction while scaling 3x.

Best Practice: Treat cost as a first-class metric, not an afterthought.

7. Documentation == Code

Our Standard:

• Every module has README
• Architecture diagrams in Git
• Runbooks as code
• ADRs (Architecture Decision Records)

The Impact:

• Onboarding new engineers: 1 day
• Cross-team knowledge sharing
• Less tribal knowledge

Best Practice: If you wrote code, you wrote documentation.

8. Security by Default

What We Built In:

• Encrypted everything (EBS, secrets, network)
• Least privilege IAM via IRSA
• Network policies by default
• Automatic security updates
• No SSH access to nodes

The Impact:

• Passed SOC 2 audit first try
• Zero security incidents in 18 months

Best Practice: Make secure configuration the default, not opt-in.

9. Test in Production (Safely)

Our Approach:

• Canary deployments
• Feature flags
• Automatic rollbacks
• Chaos engineering (monthly)

The Impact:

• Deploy multiple times per day
• Catch issues with 10% of traffic
• High confidence in deployments

Best Practice: Production is different. Test there, but safely.

10. Platform Teams Scale Differently

The Insight:

• 1 DevOps engineer supported 2 teams (manual)
• 1 platform engineer supports 6+ teams (automated)

The Key: Self-service + automation.

Best Practice: Build platforms, not projects.

Conclusion: From Chaos to Confidence

18 months ago, our infrastructure was chaos:

• 150+ fragmented pipelines
• Inconsistent deployments
• Weekly incidents
• Burnt-out team

Today, we have a platform that:

• ✅ Achieved 99.99% SLA
• ✅ Serves 12+ engineering teams
• ✅ Deploys 200+ services
• ✅ Saved 15% in costs
• ✅ Scaled 3x without growing the DevOps team

The secret? Not any single technology. It’s the combination:

• Terraform modules for consistency
• ArgoCD for GitOps
• Karpenter for intelligent scaling
• Multi-region for resilience
• Platform thinking for scale

Most importantly: We built a platform that makes the right thing the easy thing.

Resources

GitHub Repository:

• My EKS Platform Modules - Complete production-ready EKS platform with reusable Terraform modules

Official Documentation:

• Amazon EKS Best Practices Guide
• Karpenter Documentation
• ArgoCD Documentation
• Cert-Manager Documentation

Terraform Modules:

• terraform-aws-modules/eks
• AWS Load Balancer Controller
• NGINX Ingress Controller

Further Reading:

• EKS Blueprints
• Platform Engineering Best Practices
• GitOps Principles

About the Author: I’m a Senior DevOps and Cloud Engineer with 11+ years of experience. At Fidelity Information Services, I led a 12-member DevOps team in building a production-grade, multi-region EKS platform that achieved 99.99% SLA while serving 12+ engineering teams. This work earned our team the “Star Team Award - DevOps 2023” for driving Kubernetes innovation, infrastructure resilience, and high-impact DevOps team performance. All the Terraform modules and architecture patterns discussed are available on my GitHub. Connect with me on LinkedIn.

Questions? Want to discuss your EKS journey? Drop a comment below or reach out on LinkedIn. I’d love to hear about your platform engineering challenges and share experiences!

Friday, 2:47 PM. The Slack message that changed everything:

“URGENT: Production database in us-east-1 showing corruption errors. Multiple services failing. ETA to restore?”

My heart sank. We were a financial services platform processing millions of transactions daily. Every minute of downtime meant lost revenue, angry customers, and potential regulatory violations. The question everyone was asking: “Can we restore from backup?”

The honest answer? We didn’t know. We had backups, sure—but we’d never actually tested a full disaster recovery scenario. That day, we learned an expensive lesson: Having backups ≠ Having disaster recovery.

This is the story of how we built the Digester Recovery initiative—a comprehensive, automated disaster recovery system using Velero for Kubernetes, AWS Backup for databases, and Terraform for infrastructure-as-code. It’s now battle-tested, fully automated, and has saved us from multiple production incidents.

Table of Contents

1. The Problem: Why Traditional Backup Isn’t Enough
2. The Vision: RTO and RPO Requirements
3. Architecture: Multi-Layer Disaster Recovery
4. Implementation Part 1: Velero for Kubernetes
5. Implementation Part 2: AWS Backup for RDS
6. Implementation Part 3: Terraform Automation
7. Testing: The Critical Part Everyone Skips
8. The Real Test: A Production Disaster Recovery
9. Metrics and Business Impact
10. Lessons Learned and Best Practices

The Problem: Why Traditional Backup Isn’t Enough

What We Had Before

Our backup strategy was typical for many companies:

• Kubernetes: Manual YAML backups in Git (hope we committed everything!)
• Databases: RDS automated backups (enabled but never tested)
• Persistent Volumes: EBS snapshots (sporadic, manual)
• Configuration: Some Terraform, some ClickOps in console

The wake-up call came on that Friday afternoon.

The Incident That Changed Everything

Here’s what happened:

1. A database migration script ran with a bug
2. Corrupted data spread across 3 related tables
3. Application cascade failures—services couldn’t read corrupted data
4. We needed to restore to a point-in-time 2 hours earlier

The scramble:

• “Where are the database backups?” ✅ Found them
• “Can we restore to exactly 2 hours ago?” ❓ Maybe?
• “What about the Kubernetes resources?” ❓ Some in Git, some not
• “How long will restoration take?” ❓ No idea
• “Have we ever tested this?” ❌ Never

We eventually recovered after 4 hours of manual work, but the experience was traumatic. We realized we weren’t just missing backups—we were missing:

1. Automated backup processes across all components
2. Point-in-time recovery (PITR) capability
3. Tested restoration procedures with known RTOs
4. Infrastructure-as-Code for reproducible environments
5. Disaster recovery runbooks and automation

The business impact:

• 4 hours of downtime = $45K in lost transaction fees
• Regulatory reporting delay (financial services = serious)
• Customer trust damage (some moved to competitors)
• Engineering team morale hit

The executive mandate: “This can never happen again. Build a bulletproof DR system.”

The Vision: RTO and RPO Requirements

We sat down with stakeholders and defined our disaster recovery requirements:

RPO (Recovery Point Objective) - How Much Data Can We Lose?

RTO (Recovery Time Objective) - How Fast Can We Recover?

The Critical Question: How Do We Achieve This?

Our solution: Multi-layer disaster recovery with full automation.

Architecture: Multi-Layer Disaster Recovery

We designed a three-layer DR strategy:

Layer 1: Application State (Velero)
    ↓ Backs up Kubernetes resources + PVs
Layer 2: Database State (AWS Backup)
    ↓ Point-in-time recovery for RDS
Layer 3: Infrastructure (Terraform)
    ↓ Entire cluster reproducible as code

graph TB
    subgraph PrimaryRegion["Primary Region - US-EAST-1"]
        subgraph K8s["Kubernetes Cluster - EKS"]
            Apps["Application Pods<br/>StatefulSets<br/>Deployments"]
            PV["Persistent Volumes<br/>EBS"]
            Configs["ConfigMaps<br/>Secrets<br/>CRDs"]
        end
        
        subgraph Data["Data Layer"]
            RDS["RDS PostgreSQL<br/>500GB<br/>Multi-AZ"]
            EBS["EBS Volumes<br/>Attached to Pods"]
        end
        
        subgraph BackupTools["Backup Tools"]
            Velero["Velero<br/>K8s Backup Agent"]
            AWSBackup["AWS Backup<br/>Service"]
        end
    end
    
    subgraph BackupStorage["Backup Storage Layer"]
        S3Primary["S3 Bucket<br/>velero-backups<br/>us-east-1"]
        BackupVault["AWS Backup Vault<br/>us-east-1"]
        TFState["Terraform State<br/>S3 + DynamoDB<br/>Versioned"]
    end
    
    subgraph DRRegion["DR Region - US-WEST-2"]
        S3DR["S3 Bucket<br/>velero-backups<br/>us-west-2<br/>(Replicated)"]
        BackupVaultDR["AWS Backup Vault<br/>us-west-2<br/>(Replicated)"]
        TFStateDR["Terraform State<br/>us-west-2<br/>(Replicated)"]
        
        K8sDR["EKS Cluster<br/>(On-Demand)<br/>Terraform Provisioned"]
        RDSDR["RDS Instance<br/>(On-Demand)<br/>Restored from Backup"]
    end
    
    subgraph IaC["Infrastructure as Code"]
        TF["Terraform Modules<br/>• VPC<br/>• EKS<br/>• RDS<br/>• Velero<br/>• AWS Backup"]
        Git["Git Repository<br/>Version Control"]
    end
    
    Apps -->|Backup Every Hour| Velero
    PV -->|Snapshot| Velero
    Configs -->|Backup| Velero
    
    RDS -->|Continuous PITR| AWSBackup
    EBS -->|Daily Snapshot| AWSBackup
    
    Velero -->|Store Backups| S3Primary
    AWSBackup -->|Store Backups| BackupVault
    
    S3Primary -.->|Cross-Region<br/>Replication| S3DR
    BackupVault -.->|Copy Action| BackupVaultDR
    TFState -.->|Replication| TFStateDR
    
    TF -.->|Provision| K8sDR
    TF -.->|Provision| RDSDR
    
    S3DR -.->|Restore| K8sDR
    BackupVaultDR -.->|Restore| RDSDR
    
    Git -->|Source| TF
    
    style PrimaryRegion fill:#e6f3ff
    style DRRegion fill:#ffe6e6
    style BackupStorage fill:#fff4e6
    style IaC fill:#e6ffe6
    style Velero fill:#ffcccc
    style AWSBackup fill:#ccffcc

Architecture Overview

Backup Targets:

1. Kubernetes Cluster State → Velero → S3
2. RDS Databases → AWS Backup → S3
3. EBS Volumes → AWS Backup → S3 snapshots
4. Terraform State → S3 with versioning
5. Application Configs → Git (GitOps)

Key Design Principles:

• Immutable backups: Write-once, read-many in S3
• Cross-region replication: Primary in us-east-1, DR in us-west-2
• Automated scheduling: No human intervention for routine backups
• Tested regularly: Monthly DR drills
• Infrastructure-as-Code: Everything reproducible via Terraform

How It Works: Continuous Backup in Action

Let me show you how our disaster recovery system operates during normal business hours—continuously protecting data without any manual intervention:

sequenceDiagram
    participant App as Application
    participant K8s as Kubernetes
    participant RDS as RDS Database
    participant Velero as Velero
    participant AWSBackup as AWS Backup
    participant S3 as S3 Primary
    participant S3DR as S3 DR Region
    participant Vault as Backup Vault
    participant VaultDR as DR Vault
    
    Note over App,VaultDR: Normal Operations - Continuous Backup
    
    rect rgb(230, 245, 255)
        Note over App,Velero: Kubernetes Backup (Hourly)
        loop Every Hour
            Velero->>K8s: List all resources
            K8s->>Velero: Return manifests
            Velero->>Velero: Create backup tarball
            Velero->>S3: Upload backup
            S3->>S3: Compress & encrypt
            Note over S3: Backup stored:<br/>velero-backup-20241107-1400
        end
    end
    
    rect rgb(255, 240, 230)
        Note over S3,S3DR: Cross-Region Replication (Real-time)
        S3->>S3DR: Replicate backup
        S3DR->>S3DR: Store replica
        Note over S3DR: DR backup available<br/>within minutes
    end
    
    rect rgb(240, 255, 240)
        Note over RDS,AWSBackup: Database Backup (Continuous)
        loop Every 5 Minutes
            RDS->>RDS: Capture transaction logs
            RDS->>AWSBackup: Send WAL logs
            AWSBackup->>Vault: Store in backup vault
        end
        
        loop Daily at 2 AM
            AWSBackup->>RDS: Trigger full snapshot
            RDS->>AWSBackup: Send snapshot
            AWSBackup->>Vault: Store snapshot
        end
    end
    
    rect rgb(255, 245, 230)
        Note over Vault,VaultDR: DR Replication (Automated)
        Vault->>VaultDR: Copy backup
        VaultDR->>VaultDR: Store DR copy
        Note over VaultDR: Point-in-time<br/>recovery available
    end
    
    Note over App,VaultDR: RPO Achieved: 5 minutes (RDS), 1 hour (K8s)

This diagram shows the continuous backup process that runs 24/7:

Kubernetes Backup Flow (Hourly):

1. Velero queries the Kubernetes API for all resources
2. Creates a compressed tarball of manifests, configurations, and metadata
3. Takes EBS snapshots of persistent volumes
4. Uploads everything to S3 primary bucket
5. S3 automatically replicates to DR region within minutes

Database Backup Flow (Every 5 Minutes):

1. RDS continuously captures transaction logs (WAL files)
2. AWS Backup streams these logs to the backup vault
3. Daily full snapshots supplement the continuous logs
4. Everything is replicated to the DR region vault
5. Result: Can restore to any point in time with 5-minute granularity

Why This Matters: When disaster strikes, we’re never more than 5 minutes behind for databases and 1 hour behind for Kubernetes state. This continuous protection happens automatically—no engineer needs to remember to “take a backup.”

Implementation Part 1: Velero for Kubernetes

What is Velero?

Velero (formerly Ark) is an open-source tool that backs up and restores Kubernetes cluster resources and persistent volumes. Think of it as “Time Machine for Kubernetes.”

What Velero backs up:

• Deployments, Services, ConfigMaps, Secrets
• StatefulSets, DaemonSets, Jobs, CronJobs
• PersistentVolumeClaims and their data
• Custom Resource Definitions (CRDs)
• Namespaces and RBAC policies

Why Velero?

We evaluated several options:

Deploying Velero with Terraform

We automated the entire Velero deployment using Terraform:

# velero.tf
# Create S3 bucket for Velero backups
resource "aws_s3_bucket" "velero_backups" {
  bucket = "velero-backups-${var.environment}-${var.region}"
  
  tags = {
    Name        = "Velero Backups"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Enable versioning for backup protection
resource "aws_s3_bucket_versioning" "velero_backups" {
  bucket = aws_s3_bucket.velero_backups.id
  
  versioning_configuration {
    status = "Enabled"
  }
}

# Enable encryption at rest
resource "aws_s3_bucket_server_side_encryption_configuration" "velero_backups" {
  bucket = aws_s3_bucket.velero_backups.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

# Cross-region replication for DR
resource "aws_s3_bucket_replication_configuration" "velero_replication" {
  bucket = aws_s3_bucket.velero_backups.id
  role   = aws_iam_role.s3_replication.arn

  rule {
    id     = "ReplicateToWest"
    status = "Enabled"

    destination {
      bucket        = aws_s3_bucket.velero_backups_dr.arn
      storage_class = "STANDARD_IA"
    }
  }
}

# IAM role for Velero
resource "aws_iam_role" "velero" {
  name = "velero-${var.environment}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = var.eks_oidc_provider_arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${var.eks_oidc_provider}:sub" = "system:serviceaccount:velero:velero"
        }
      }
    }]
  })
}

# IAM policy for Velero
resource "aws_iam_role_policy" "velero" {
  name = "velero-policy"
  role = aws_iam_role.velero.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ec2:DescribeVolumes",
          "ec2:DescribeSnapshots",
          "ec2:CreateTags",
          "ec2:CreateVolume",
          "ec2:CreateSnapshot",
          "ec2:DeleteSnapshot"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:DeleteObject",
          "s3:PutObject",
          "s3:AbortMultipartUpload",
          "s3:ListMultipartUploadParts"
        ]
        Resource = "${aws_s3_bucket.velero_backups.arn}/*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket"
        ]
        Resource = aws_s3_bucket.velero_backups.arn
      }
    ]
  })
}

# Install Velero using Helm
resource "helm_release" "velero" {
  name       = "velero"
  repository = "https://vmware-tanzu.github.io/helm-charts"
  chart      = "velero"
  namespace  = "velero"
  version    = "5.0.2"

  create_namespace = true

  values = [
    yamlencode({
      initContainers = [{
        name  = "velero-plugin-for-aws"
        image = "velero/velero-plugin-for-aws:v1.8.0"
        volumeMounts = [{
          mountPath = "/target"
          name      = "plugins"
        }]
      }]

      serviceAccount = {
        server = {
          annotations = {
            "eks.amazonaws.com/role-arn" = aws_iam_role.velero.arn
          }
        }
      }

      configuration = {
        provider = "aws"
        
        backupStorageLocation = {
          bucket = aws_s3_bucket.velero_backups.id
          prefix = "backups"
          config = {
            region = var.region
          }
        }

        volumeSnapshotLocation = {
          config = {
            region = var.region
          }
        }
      }

      schedules = {
        # Daily full backup at 2 AM
        daily-backup = {
          schedule = "0 2 * * *"
          template = {
            ttl             = "168h"  # 7 days retention
            includedNamespaces = ["production", "staging"]
            snapshotVolumes = true
          }
        }

        # Hourly backup of critical namespaces
        hourly-critical = {
          schedule = "0 * * * *"
          template = {
            ttl             = "48h"  # 2 days retention
            includedNamespaces = ["production"]
            labelSelector = {
              matchLabels = {
                backup = "critical"
              }
            }
          }
        }
      }

      metrics = {
        enabled = true
        serviceMonitor = {
          enabled = true
        }
      }
    })
  ]

  depends_on = [
    aws_s3_bucket.velero_backups,
    aws_iam_role.velero
  ]
}

Velero Backup Schedules

We implemented three backup schedules based on criticality:

1. Hourly Critical Backups

velero schedule create hourly-critical \
  --schedule="0 * * * *" \
  --include-namespaces production \
  --selector backup=critical \
  --ttl 48h

2. Daily Full Backups

velero schedule create daily-full \
  --schedule="0 2 * * *" \
  --include-namespaces production,staging \
  --snapshot-volumes \
  --ttl 168h

3. Weekly Long-Term Backups

velero schedule create weekly-longterm \
  --schedule="0 3 * * 0" \
  --include-namespaces production \
  --snapshot-volumes \
  --ttl 720h  # 30 days

Selective Backup with Labels

Not all resources need the same backup frequency. We used Kubernetes labels to control backup granularity:

# Label critical deployments for hourly backup
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: production
  labels:
    app: payment-api
    backup: critical  # This gets backed up hourly
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: payment-api
        backup: critical

Testing Velero Backups

The golden rule: Untested backups = No backups.

We created a test procedure:

#!/bin/bash
# velero-test.sh - Monthly DR Drill

echo "=== Velero Disaster Recovery Test ==="

# 1. Create a test namespace with sample resources
kubectl create namespace dr-test
kubectl create deployment nginx --image=nginx -n dr-test
kubectl create configmap test-config --from-literal=key=value -n dr-test

# 2. Take an immediate backup
velero backup create dr-test-backup \
  --include-namespaces dr-test \
  --wait

# 3. Verify backup completed
velero backup describe dr-test-backup

# 4. Delete the namespace (simulate disaster)
kubectl delete namespace dr-test

# 5. Wait for complete deletion
sleep 30

# 6. Restore from backup
velero restore create dr-test-restore \
  --from-backup dr-test-backup \
  --wait

# 7. Verify restoration
kubectl get all -n dr-test

# 8. Cleanup
velero backup delete dr-test-backup
kubectl delete namespace dr-test

echo "=== Test Complete ==="

We run this test monthly. Pass criteria:

• Backup completes in < 5 minutes
• Restore completes in < 10 minutes
• All resources restored correctly
• No data loss

Implementation Part 2: AWS Backup for RDS

Why AWS Backup?

RDS has built-in automated backups, but AWS Backup provides:

• Centralized management: One place for all backup policies
• Cross-region copy: Automatic DR region replication
• Point-in-time recovery (PITR): Restore to any second
• Compliance reporting: Audit-ready backup reports
• Lifecycle policies: Automated transition to cold storage

graph LR
    subgraph Creation["Backup Creation"]
        Event["Trigger Event<br/>• Scheduled backup<br/>• Manual backup<br/>• Pre-deployment backup"]
    end
    
    subgraph Hot["Hot Storage (0-7 days)"]
        HotBackup["Recent Backups<br/>Storage: S3 Standard<br/>Retrieval: Immediate<br/>Cost: $$$$"]
    end
    
    subgraph Warm["Warm Storage (7-30 days)"]
        WarmBackup["Older Backups<br/>Storage: S3 IA<br/>Retrieval: Minutes<br/>Cost: $$$"]
    end
    
    subgraph Cold["Cold Storage (30-90 days)"]
        ColdBackup["Archive Backups<br/>Storage: S3 Glacier<br/>Retrieval: Hours<br/>Cost: $$"]
    end
    
    subgraph Archive["Long-term Archive (90-365 days)"]
        ArchiveBackup["Compliance Backups<br/>Storage: Glacier Deep<br/>Retrieval: 12+ hours<br/>Cost: $"]
    end
    
    Delete["Deletion<br/>After retention period"]
    
    Event --> HotBackup
    HotBackup -->|After 7 days| WarmBackup
    WarmBackup -->|After 30 days| ColdBackup
    ColdBackup -->|After 90 days| ArchiveBackup
    ArchiveBackup -->|After 365 days| Delete
    
    HotBackup -.->|Restore<br/>Time: Seconds| Restore1["Fast Restore"]
    WarmBackup -.->|Restore<br/>Time: Minutes| Restore2["Normal Restore"]
    ColdBackup -.->|Restore<br/>Time: 3-5 hours| Restore3["Slow Restore"]
    ArchiveBackup -.->|Restore<br/>Time: 12+ hours| Restore4["Compliance Restore"]
    
    style Creation fill:#e6f3ff
    style Hot fill:#ffcccc
    style Warm fill:#ffffcc
    style Cold fill:#ccffff
    style Archive fill:#e6ccff
    style Delete fill:#ffcccc

RDS Backup Strategy

Our RDS backup policy:

# aws-backup.tf
# AWS Backup vault
resource "aws_backup_vault" "main" {
  name = "digester-backup-vault-${var.environment}"
  
  tags = {
    Name        = "Digester Backup Vault"
    Environment = var.environment
  }
}

# Backup plan for RDS
resource "aws_backup_plan" "rds_backup" {
  name = "rds-backup-plan-${var.environment}"

  # Rule 1: Continuous backups with 5-minute RPO
  rule {
    rule_name         = "continuous_backup"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 */1 * * ? *)"  # Every hour
    
    start_window      = 60   # Start within 1 hour
    completion_window = 120  # Complete within 2 hours

    lifecycle {
      delete_after = 7  # Keep for 7 days
    }

    enable_continuous_backup = true  # Point-in-time recovery
    
    copy_action {
      destination_vault_arn = aws_backup_vault.dr_region.arn
      
      lifecycle {
        delete_after = 7
      }
    }
  }

  # Rule 2: Daily backups with long retention
  rule {
    rule_name         = "daily_backup"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 2 * * ? *)"  # 2 AM daily

    lifecycle {
      delete_after       = 35  # Keep for 35 days
      cold_storage_after = 30  # Move to cold storage after 30 days
    }

    copy_action {
      destination_vault_arn = aws_backup_vault.dr_region.arn
      
      lifecycle {
        delete_after       = 35
        cold_storage_after = 30
      }
    }
  }

  # Rule 3: Monthly backups for long-term retention
  rule {
    rule_name         = "monthly_backup"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 3 1 * ? *)"  # 1st of month at 3 AM

    lifecycle {
      delete_after       = 365  # Keep for 1 year
      cold_storage_after = 90   # Cold storage after 3 months
    }

    copy_action {
      destination_vault_arn = aws_backup_vault.dr_region.arn
      
      lifecycle {
        delete_after       = 365
        cold_storage_after = 90
      }
    }
  }

  tags = {
    Name        = "RDS Backup Plan"
    Environment = var.environment
  }
}

# Backup selection for RDS instances
resource "aws_backup_selection" "rds_selection" {
  name         = "rds-backup-selection"
  plan_id      = aws_backup_plan.rds_backup.id
  iam_role_arn = aws_iam_role.aws_backup.arn

  selection_tag {
    type  = "STRINGEQUALS"
    key   = "Backup"
    value = "true"
  }

  resources = [
    "arn:aws:rds:${var.region}:${data.aws_caller_identity.current.account_id}:db:*"
  ]
}

# IAM role for AWS Backup
resource "aws_iam_role" "aws_backup" {
  name = "aws-backup-role-${var.environment}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "backup.amazonaws.com"
      }
      Action = "sts:AssumeRole"
    }]
  })
}

# Attach AWS managed policy for RDS backup
resource "aws_iam_role_policy_attachment" "aws_backup_rds" {
  role       = aws_iam_role.aws_backup.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForBackup"
}

resource "aws_iam_role_policy_attachment" "aws_backup_restore" {
  role       = aws_iam_role.aws_backup.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForRestores"
}

Enabling Continuous Backups for 5-Minute RPO

The key to achieving our 5-minute RPO:

# RDS instance with point-in-time recovery
resource "aws_db_instance" "main" {
  identifier     = "digester-db-${var.environment}"
  engine         = "postgres"
  engine_version = "15.3"
  instance_class = "db.r6g.xlarge"

  allocated_storage     = 500
  storage_encrypted     = true
  storage_type          = "gp3"

  # Enable automated backups
  backup_retention_period = 7  # Days
  backup_window          = "03:00-04:00"  # UTC
  
  # Enable point-in-time recovery
  enabled_cloudwatch_logs_exports = ["postgresql", "upgrade"]
  
  # Tagging for AWS Backup selection
  tags = {
    Name        = "Digester Database"
    Environment = var.environment
    Backup      = "true"  # AWS Backup will pick this up
  }
}

How Point-in-Time Recovery Works:

• RDS continuously backs up transaction logs to S3
• You can restore to any second within the retention period
• Example: Restore to exactly 2024-11-07 14:32:45 UTC

Automated Cross-Region Replication

The critical component for regional disaster recovery:

# DR region backup vault
resource "aws_backup_vault" "dr_region" {
  provider = aws.us-west-2  # DR region
  
  name = "digester-dr-vault-${var.environment}"
  
  tags = {
    Name        = "DR Backup Vault"
    Environment = var.environment
    Region      = "us-west-2"
  }
}

# S3 bucket for additional backup storage
resource "aws_s3_bucket" "backup_storage" {
  bucket = "digester-backup-storage-${var.environment}"
  
  versioning {
    enabled = true
  }

  replication_configuration {
    role = aws_iam_role.s3_replication.arn

    rules {
      id     = "backup-replication"
      status = "Enabled"

      destination {
        bucket        = aws_s3_bucket.backup_storage_dr.arn
        storage_class = "GLACIER"
      }
    }
  }
}

Implementation Part 3: Terraform Automation

The Power of Infrastructure-as-Code for DR

Here’s why Terraform is critical for disaster recovery:

Scenario: Entire AWS region goes down (us-east-1)

Without Terraform:

• Manually recreate VPC, subnets, security groups
• Manually provision EKS cluster
• Manually configure IAM roles and policies
• Manually deploy applications
• Time: Days or weeks

With Terraform:

cd terraform
terraform workspace select dr-us-west-2
terraform apply -auto-approve
# Time: 45 minutes

Our Terraform Structure

terraform/
├── environments/
│   ├── production/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── terraform.tfvars
│   │   └── backend.tf
│   └── dr/
│       ├── main.tf
│       ├── variables.tf
│       ├── terraform.tfvars
│       └── backend.tf
├── modules/
│   ├── eks-cluster/
│   ├── rds/
│   ├── velero/
│   ├── aws-backup/
│   ├── vpc/
│   └── monitoring/
└── scripts/
    ├── dr-failover.sh
    ├── dr-test.sh
    └── restore-from-backup.sh

Complete DR Infrastructure Module

# modules/disaster-recovery/main.tf
module "dr_infrastructure" {
  source = "./modules/disaster-recovery"

  environment = var.environment
  region      = var.region
  dr_region   = var.dr_region

  # VPC configuration
  vpc_cidr = var.vpc_cidr
  
  # EKS configuration
  eks_cluster_version = "1.28"
  eks_node_groups = {
    general = {
      desired_size = 3
      max_size     = 10
      min_size     = 3
      instance_types = ["t3.xlarge"]
    }
  }

  # RDS configuration
  rds_instance_class    = "db.r6g.xlarge"
  rds_allocated_storage = 500
  rds_backup_retention  = 7
  
  # Backup configuration
  velero_enabled          = true
  aws_backup_enabled      = true
  cross_region_replication = true
  
  # DR testing
  enable_dr_drills = true
  dr_drill_schedule = "0 10 * * 1"  # Every Monday at 10 AM

  tags = {
    Project     = "Digester"
    ManagedBy   = "Terraform"
    Environment = var.environment
  }
}

Automated DR Failover Script

We created a script to automate failover to DR region:

#!/bin/bash
# dr-failover.sh - Automated failover to DR region

set -e

DR_REGION="us-west-2"
PRIMARY_REGION="us-east-1"
ENVIRONMENT="production"

echo "=== DISASTER RECOVERY FAILOVER ==="
echo "Primary Region: $PRIMARY_REGION"
echo "DR Region: $DR_REGION"
echo ""

read -p "This will initiate failover to DR region. Continue? (yes/no): " confirm
if [ "$confirm" != "yes" ]; then
    echo "Failover cancelled"
    exit 0
fi

echo "Step 1: Provisioning DR infrastructure in $DR_REGION..."
cd terraform/environments/dr
terraform init
terraform workspace select dr-$DR_REGION
terraform apply -auto-approve

echo "Step 2: Restoring RDS from latest backup..."
LATEST_BACKUP=$(aws backup list-recovery-points-by-backup-vault \
    --backup-vault-name digester-backup-vault-$ENVIRONMENT \
    --region $DR_REGION \
    --query 'RecoveryPoints[0].RecoveryPointArn' \
    --output text)

aws backup start-restore-job \
    --recovery-point-arn $LATEST_BACKUP \
    --iam-role-arn $(terraform output -raw backup_restore_role_arn) \
    --region $DR_REGION \
    --metadata '{"DBInstanceIdentifier":"digester-db-dr"}'

echo "Waiting for RDS restore to complete..."
aws rds wait db-instance-available \
    --db-instance-identifier digester-db-dr \
    --region $DR_REGION

echo "Step 3: Restoring Kubernetes resources from Velero..."
velero restore create dr-failover-restore \
    --from-backup daily-backup \
    --wait

echo "Step 4: Updating DNS to point to DR region..."
ROUTE53_ZONE_ID=$(terraform output -raw route53_zone_id)
DR_LB_DNS=$(kubectl get svc ingress-nginx-controller -n ingress-nginx \
    -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

aws route53 change-resource-record-sets \
    --hosted-zone-id $ROUTE53_ZONE_ID \
    --change-batch "{
        \"Changes\": [{
            \"Action\": \"UPSERT\",
            \"ResourceRecordSet\": {
                \"Name\": \"api.digester.com\",
                \"Type\": \"CNAME\",
                \"TTL\": 60,
                \"ResourceRecords\": [{\"Value\": \"$DR_LB_DNS\"}]
            }
        }]
    }"

echo "Step 5: Running smoke tests..."
./scripts/smoke-test.sh $DR_REGION

echo ""
echo "=== FAILOVER COMPLETE ==="
echo "Application is now running in DR region: $DR_REGION"
echo "Monitor dashboard: https://grafana.digester.com"
echo ""

State Management for DR

Critical: Terraform state must survive disasters too.

# backend.tf - S3 backend with cross-region replication
terraform {
  backend "s3" {
    bucket         = "digester-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"
    
    # State versioning enabled on bucket
    # Cross-region replication to us-west-2
  }
}

# S3 bucket for Terraform state
resource "aws_s3_bucket" "terraform_state" {
  bucket = "digester-terraform-state"

  versioning {
    enabled = true
  }

  replication_configuration {
    role = aws_iam_role.s3_replication.arn

    rules {
      id     = "state-replication"
      status = "Enabled"

      destination {
        bucket        = aws_s3_bucket.terraform_state_dr.arn
        storage_class = "STANDARD_IA"
      }
    }
  }

  lifecycle {
    prevent_destroy = true
  }
}

Testing: The Critical Part Everyone Skips

The harsh truth: 60% of companies have never tested their disaster recovery plan. (Gartner 2023)

We don’t want to be in that 60%.

stateDiagram-v2
    [*] --> Schedule
    
    Schedule --> PreCheck
    
    state PreCheck {
        [*] --> VerifyBackups
        VerifyBackups --> CheckInfra
        CheckInfra --> NotifyTeam
    }
    
    PreCheck --> VeleroTest
    
    state VeleroTest {
        [*] --> CreateTestNS
        CreateTestNS --> TakeBackup
        TakeBackup --> DeleteNS
        DeleteNS --> RestoreBackup
        RestoreBackup --> ValidateRestore
    }
    
    VeleroTest --> RDSTest
    
    state RDSTest {
        [*] --> IdentifyRecoveryPoint
        IdentifyRecoveryPoint --> InitiateRestore
        InitiateRestore --> WaitCompletion
        WaitCompletion --> ValidateData
        ValidateData --> CheckIntegrity
    }
    
    RDSTest --> FullFailover
    
    state FullFailover {
        [*] --> ProvisionInfra
        ProvisionInfra --> RestoreDatabase
        RestoreDatabase --> RestoreK8s
        RestoreK8s --> UpdateDNS
        UpdateDNS --> SmokeTests
    }
    
    FullFailover --> Validation
    
    state Validation {
        [*] --> MeasureRTO
        MeasureRTO --> MeasureRPO
        MeasureRPO --> DocumentResults
    }
    
    Validation --> Decision
    
    Decision --> Success
    Decision --> Failure
    
    Success --> GenerateReport
    Failure --> Incident
    
    Incident --> RootCause
    RootCause --> Remediation
    Remediation --> [*]
    
    GenerateReport --> Cleanup
    Cleanup --> [*]
    
    note right of Schedule
        First Monday of Month
        10:00 AM - Start Drill
    end note
    
    note right of VeleroTest
        Target: less than 20 minutes
        Pass criteria: 100% resource recovery
    end note
    
    note right of RDSTest
        Target: less than 30 minutes
        Pass criteria: Zero data corruption
    end note
    
    note right of FullFailover
        Target: less than 2 hours
        Pass criteria: All services functional
    end note

Monthly DR Drills

Every first Monday of the month at 10 AM:

#!/bin/bash
# dr-drill-monthly.sh - Automated DR drill

echo "=== MONTHLY DISASTER RECOVERY DRILL ==="
date

# Test 1: Velero backup and restore
echo "Test 1: Kubernetes backup/restore..."
./scripts/test-velero.sh
if [ $? -eq 0 ]; then
    echo "✅ Velero test passed"
else
    echo "❌ Velero test failed"
    exit 1
fi

# Test 2: RDS point-in-time recovery
echo "Test 2: RDS PITR..."
./scripts/test-rds-pitr.sh
if [ $? -eq 0 ]; then
    echo "✅ RDS test passed"
else
    echo "❌ RDS test failed"
    exit 1
fi

# Test 3: Full DR failover (non-prod)
echo "Test 3: Full DR failover (staging)..."
./scripts/test-dr-failover.sh staging
if [ $? -eq 0 ]; then
    echo "✅ DR failover test passed"
else
    echo "❌ DR failover test failed"
    exit 1
fi

# Generate report
echo "=== DRILL COMPLETE ==="
./scripts/generate-dr-report.sh
echo "Report saved to: dr-reports/$(date +%Y-%m-%d)-drill-report.html"

What We Test Every Month

1. Velero Restore Speed
   • Target: < 20 minutes for full namespace
   • Measured: Actual time from backup to running pods
2. RDS Point-in-Time Recovery
   • Target: < 30 minutes
   • Measured: Time to restore to specific timestamp
3. Cross-Region Failover
   • Target: < 2 hours for complete failover
   • Measured: Total time from initiation to passing smoke tests
4. Data Integrity
   • Target: 100% data accuracy
   • Measured: Checksum comparison before/after

The DR Dashboard

We built a Grafana dashboard showing:

• Last successful backup timestamp
• Backup success rate (target: 99.9%)
• Time since last DR drill
• Estimated RTO/RPO based on latest test
• Backup storage usage and costs

Alerts we configured:

• 🚨 Critical: Backup failed for > 12 hours
• ⚠️ Warning: DR drill overdue by > 7 days
• ⚠️ Warning: Backup completion time increasing trend

The Real Test: A Production Disaster Recovery

sequenceDiagram
    participant Incident as Incident Detected
    participant Eng as Engineer
    participant Terraform as Terraform
    participant VaultDR as DR Backup Vault
    participant S3DR as S3 DR Bucket
    participant EKS as New EKS Cluster
    participant RDS as New RDS Instance
    participant DNS as Route53 DNS
    participant Monitor as Monitoring
    
    Note over Incident,Monitor: DISASTER RECOVERY EXECUTION
    
    rect rgb(255, 230, 230)
        Note over Incident,Eng: Phase 1: Detection (0-5 minutes)
        Incident->>Monitor: Service failures detected
        Monitor->>Monitor: Alert fired
        Monitor->>Eng: Page on-call engineer
        Eng->>Eng: Assess severity
        Note over Eng: Decision: Execute DR failover
    end
    
    rect rgb(255, 245, 230)
        Note over Eng,Terraform: Phase 2: Infrastructure (5-25 minutes)
        Eng->>Terraform: Run dr-failover.sh
        Terraform->>Terraform: terraform workspace select dr
        Terraform->>EKS: Provision EKS cluster
        Note over EKS: Cluster creation: ~15 min
        EKS->>Terraform: Cluster ready
        Terraform->>Terraform: Configure kubectl access
    end
    
    rect rgb(240, 255, 240)
        Note over VaultDR,RDS: Phase 3: Database Restore (25-50 minutes)
        Eng->>VaultDR: Identify recovery point
        VaultDR->>VaultDR: Latest backup:<br/>2024-11-07 11:30:00
        Eng->>VaultDR: start-restore-job
        VaultDR->>RDS: Restore from backup
        RDS->>RDS: Apply transaction logs
        RDS->>RDS: Database available
        Note over RDS: Restore time: ~15 min<br/>RPO: 5 minutes
    end
    
    rect rgb(245, 240, 255)
        Note over S3DR,EKS: Phase 4: Application Restore (50-70 minutes)
        Eng->>S3DR: List available backups
        S3DR->>Eng: Return backup list
        Eng->>S3DR: velero restore create
        S3DR->>EKS: Download backup
        EKS->>EKS: Apply Kubernetes manifests
        EKS->>EKS: Create pods, services, ingress
        EKS->>EKS: Mount persistent volumes
        Note over EKS: All pods running
    end
    
    rect rgb(255, 250, 240)
        Note over DNS,Monitor: Phase 5: Cutover (70-90 minutes)
        Eng->>Monitor: Run smoke tests
        Monitor->>EKS: Test API endpoints
        EKS->>Monitor: Health checks pass
        Monitor->>Eng: All tests passed
        
        Eng->>DNS: Update DNS records
        DNS->>DNS: api.company.com → DR region
        Note over DNS: TTL: 60 seconds
        
        Eng->>Monitor: Monitor traffic shift
        Monitor->>Monitor: Traffic flowing to DR
    end
    
    rect rgb(230, 255, 255)
        Note over Incident,Monitor: Phase 6: Validation (90-120 minutes)
        Monitor->>EKS: Continuous monitoring
        Monitor->>RDS: Check database queries
        Monitor->>Eng: System stable
        Eng->>Eng: Document incident
        Note over Eng: RTO Achieved: 2 hours<br/>System fully recovered
    end
    
    Note over Incident,Monitor: Total Recovery Time: 2 hours

Six months after implementing our DR system, we faced a real disaster.

The Incident: April 15, 2023, 11:47 AM

What happened:

• A software deployment introduced a critical bug
• Bug corrupted primary keys in our transactions table
• Data inconsistency spread to 4 related tables
• Services started failing cascade-style
• Customers couldn’t complete transactions

The old us would have panicked. The new us executed our DR plan.

The Recovery Timeline

11:47 AM - Incident detected

Automated alerts fired:
- Database error rate spike
- Transaction API failure rate > 50%
- Customer complaints flooding in

11:50 AM - Decision made

Engineering lead: "We need to restore to 11:30 AM (before the deployment)"
SRE team: "Executing DR procedure"

11:52 AM - RDS Point-in-Time Recovery initiated

# Restore RDS to 17 minutes ago
aws backup start-restore-job \
    --recovery-point-arn $RECOVERY_POINT \
    --iam-role-arn $RESTORE_ROLE \
    --metadata '{
        "DBInstanceIdentifier":"digester-db-recovery",
        "RestoreTime":"2023-04-15T11:30:00Z"
    }'

12:08 PM - Database restore complete

RDS restored to 11:30 AM state
16 minutes of transactions lost (acceptable RPO)

12:10 PM - Kubernetes rollback

# Rollback deployment
kubectl rollout undo deployment/transaction-api -n production

# Verify pods healthy
kubectl get pods -n production

12:15 PM - Services recovering

- Database connections restored
- Transaction API processing requests
- Error rate dropping

12:23 PM - Full recovery

- All services healthy
- Customer transactions flowing
- Total downtime: 36 minutes

Post-incident:

• Lost transactions: 237 (within our acceptable data loss)
• Customers affected: ~50 (quickly notified and compensated)
• Financial impact: ~$3,500 (vs. potential $45K+ without DR)
• Our DR system worked exactly as designed

What Made the Difference

1. Pre-tested procedures: We knew exactly what to do
2. Automated tooling: No manual steps, no human errors
3. Clear RTO/RPO: Everyone knew what “success” looked like
4. Terraform infrastructure: Confident in reproducibility
5. Monitoring: Detected issue within minutes

CTO’s message to the team:

“The Digester Recovery system just saved us from a disaster. This is why we invest in infrastructure.”

Metrics and Business Impact

Before Digester Recovery

After Digester Recovery

Cost Analysis

Investment:

• Engineering time: 320 hours (2 engineers × 8 weeks)
• AWS Backup costs: ~$600/month
• S3 backup storage: ~$400/month
• Cross-region replication: ~$200/month
• Total monthly cost: ~$1,200

Returns:

• Prevented downtime cost: $45K (first incident alone)
• Reduced MTTR: 4 hours → 36 minutes (85% improvement)
• Avoided regulatory fines: Priceless (financial services = strict rules)
• Team productivity: +20% (no more panic during incidents)
• ROI: Positive within first 2 months

Regulatory Compliance

For financial services, DR isn’t optional—it’s legally required.

What we achieved:

• ✅ PCI DSS Requirement 12.10: Disaster recovery plan
• ✅ SOC 2 Type II: Availability commitments
• ✅ GDPR Article 32: Data protection by design
• ✅ Internal audit: 100% compliance score

Audit finding:

“The Digester Recovery initiative demonstrates industry-leading disaster recovery practices with comprehensive automation, regular testing, and documented procedures.”

Lessons Learned and Best Practices

After two years of running our DR system in production, here’s what we learned:

1. Backups Without Testing = False Security

The mistake we almost made: Implementing backups and assuming they work.

The reality: 34% of restore attempts fail due to corrupted backups, misconfiguration, or incomplete data. (Source: Veeam Data Protection Report)

Our solution: Monthly DR drills, automated testing, metrics tracking.

2. RPO and RTO Must Be Business-Driven

The mistake: IT defines arbitrary numbers (e.g., “let’s aim for 1-hour RTO”).

The better approach: Ask stakeholders:

• “How much data loss is acceptable?”
• “How long can systems be down before major business impact?”
• “What’s the cost of an hour of downtime?”

Our approach:

• Financial transactions → 5-minute RPO (critical)
• Logs and cache → 1-hour RPO (acceptable loss)

3. Automate Everything

Manual DR procedures fail under pressure. When you’re in a real disaster at 3 AM, you don’t want to be reading a 50-page runbook.

What we automated:

• Backup scheduling
• Cross-region replication
• Restore procedures
• DR failover
• Testing and validation
• Reporting and alerting

Result: Any engineer can execute DR recovery with a single command.

4. Multi-Layer Defense

Don’t rely on a single backup system:

• Layer 1: Velero for Kubernetes (application state)
• Layer 2: AWS Backup for RDS (data)
• Layer 3: Terraform for infrastructure (reproducibility)
• Layer 4: Git for configurations (version control)

Why? If one layer fails, others provide redundancy.

5. Cross-Region is Non-Negotiable

Scenario: AWS us-east-1 has a major outage (it’s happened before).

Without cross-region replication: You’re dead in the water.

With cross-region replication: Failover to us-west-2 in < 2 hours.

Our implementation:

• All backups replicated to DR region
• Terraform can provision identical infrastructure in DR region
• DNS failover automated via Route 53

6. State Management is Critical

Your disaster recovery system’s configuration must survive disasters too:

• Terraform state in S3 with versioning + cross-region replication
• Velero backup locations replicated
• Configuration in Git (off-site)

Lesson: Don’t store disaster recovery tools in the region you’re trying to recover from!

7. Document Everything (But Don’t Rely on Documentation)

We maintain detailed DR runbooks, but we don’t rely on humans reading them during incidents.

Better approach:

• Runbooks as code (automation scripts)
• Runbooks as tests (run monthly to verify)
• Runbooks as training (new team members run DR drills)

8. Calculate Your Actual Costs

Many teams underestimate backup costs at scale:

Our costs breakdown:

• S3 storage (primary): ~$300/month
• S3 storage (DR region): ~$150/month
• AWS Backup service: ~$600/month
• Cross-region data transfer: ~$200/month
• EBS snapshots: ~$100/month
• Total: ~$1,350/month

But compare to:

• Cost of 4-hour outage: ~$45K
• Cost of data loss incident: Potentially millions
• Cost of regulatory fines: $$$

ROI is overwhelmingly positive.

9. Cultural Shift: DR is Everyone’s Responsibility

Old mindset: “DR is the ops team’s problem”

New mindset: “Every engineer must understand DR”

How we changed culture:

• Required DR training for all engineers
• Monthly DR drills include developers
• DR metrics in team dashboards
• Post-mortems focus on DR improvements

10. Continuous Improvement

Our DR system today is 10x better than v1:

v1 (Initial):

• Manual Velero backups
• No RDS point-in-time recovery
• No cross-region replication
• No testing

v2 (6 months later):

• Automated scheduling
• AWS Backup integrated
• Cross-region replication
• Monthly tests

v3 (Current):

• Full Terraform automation
• Sub-30-minute RTO
• Chaos engineering integration
• Automated DR drills
• Real-time DR dashboards

The lesson: DR is never “done”—it’s a continuous process.

Conclusion: Sleep Better at Night

Before the Digester Recovery initiative, every on-call rotation was stressful. What if we have a major incident? What if we can’t recover?

After implementing our comprehensive DR system, something changed: We sleep better.

Not because disasters can’t happen—they will. But because we know:

1. Our backups work (we test them monthly)
2. Our restore procedures work (we’ve used them in production)
3. Our RTO/RPO targets are achievable (we measure them)
4. Our team knows what to do (we practice regularly)

The real value of disaster recovery isn’t just in the technology—it’s in the confidence it provides.

When (not if) the next disaster strikes, we’re ready.

Resources and Next Steps

GitHub Repositories:

• Velero Terraform Module - Official Velero
• AWS Backup Terraform Examples

Further Reading:

• Kubernetes Disaster Recovery Best Practices
• AWS Backup Documentation
• Velero Documentation

Tools We Use:

• Velero: Kubernetes backup and restore
• AWS Backup: Centralized backup service
• Terraform: Infrastructure as Code
• Prometheus + Grafana: Monitoring and alerting

Want to Build Your Own DR System?

Start here:

1. Define your RTO and RPO requirements
2. Implement automated backups (start with Velero)
3. Test your backups (seriously, test them)
4. Automate with Terraform
5. Run monthly DR drills
6. Iterate and improve

Remember: The best time to implement disaster recovery was yesterday. The second-best time is today.

About the Author: I’m a Senior DevOps and Cloud Engineer with 10+ years of experience. I led the Digester Recovery initiative at a major financial services company, implementing disaster recovery systems that achieved < 30-minute RTO and 99.7% backup success rates. This work earned our team the “Star Team Award - DevOps 2023” for driving infrastructure resilience and high-impact DevOps performance. Connect with me on LinkedIn or GitHub.

Questions? Experiences to share? Drop a comment below or reach out on LinkedIn. I’d love to hear about your disaster recovery journey—especially if you’ve lived through a real disaster!

The Email That Changed Everything:

From: Finance Team
To: DevOps Team
Subject: URGENT: Cloud Costs Review Meeting - Tomorrow 9 AM

Your AWS bill increased 47% last quarter. We need to discuss 
this immediately.

Q3 AWS Spend: $56,000/month
Q4 Projected: $82,000/month

This is unsustainable.

My stomach dropped. It was October 2022, and I’d been at Fidelity Information Services (FIS) for six months leading our Kubernetes infrastructure transformation. We’d successfully migrated 12 engineering teams to our new EKS platform. Deployments were faster. Reliability was better. Everyone was happy.

Except finance.

The next day’s meeting was brutal. Charts showing exponential cloud cost growth. Questions I couldn’t answer. “Why are we paying $28K/month for EC2 instances that are barely used?” “What are these $8K NAT Gateway charges?” “Can you explain why our EBS costs doubled?”

I couldn’t. We’d focused on velocity and reliability, treating infrastructure costs as someone else’s problem. That day, I learned a hard lesson: In the cloud, every architectural decision is a financial decision.

Fast forward 6 months: We’d cut our monthly AWS bill from $56K to $48K (15% reduction) while tripling our workload capacity. We went from panic to having engineers excited about cost optimization. We built a FinOps culture where cost was as important as performance.

This is the complete story—every optimization, every mistake, every dollar saved. If you’re running Kubernetes in production and your finance team is breathing down your neck, this guide will show you exactly how we did it.

Table of Contents

1. The Cost Crisis: Where $56K Was Going
2. Phase 1: Visibility - You Can’t Optimize What You Can’t See
3. Phase 2: Quick Wins - Low-Hanging Fruit ($2K/month)
4. Phase 3: Karpenter - The Game Changer ($1.5K/month)
5. Phase 4: Spot Instances Strategy ($800/month)
6. Phase 5: Storage Optimization ($400/month)
7. Phase 6: Network Cost Optimization ($300/month)
8. Phase 7: Building a FinOps Culture
9. The Results: Before vs After
10. Lessons Learned and Mistakes to Avoid

The Cost Crisis: Where $56K Was Going

pie title Before Optimization - $56K/month
    "EC2 Instances" : 28000
    "EBS Storage" : 8400
    "NAT Gateways" : 8200
    "Load Balancers" : 5600
    "Data Transfer" : 3360
    "Other" : 2440

pie title After Optimization - $48K/month (15% Savings)
    "EC2 Instances" : 19500
    "EBS Storage" : 8000
    "NAT Gateways" : 3500
    "Load Balancers" : 5300
    "Data Transfer" : 3060
    "Monitoring Tools" : 8640

The Initial Cost Breakdown (October 2022)

When we finally analyzed our AWS bill, here’s what we found:

The Shocking Discoveries:

1. Zombie Instances Everywhere

# What we found
$ kubectl get nodes
NAME                                          STATUS   ROLES    AGE
ip-10-0-1-123.ec2.internal                   Ready    <none>   47d
ip-10-0-1-234.ec2.internal                   Ready    <none>   47d
ip-10-0-1-345.ec2.internal                   Ready    <none>   47d
# ... 32 nodes total

$ kubectl describe nodes | grep -A 5 "Allocated resources"
# Average CPU usage: 23%
# Average Memory usage: 31%

We were paying for 32 nodes running at 30% utilization. That’s like renting a 10-bedroom mansion for a family of three.

2. The EBS Horror Show

# Orphaned volumes
$ aws ec2 describe-volumes \
  --filters "Name=status,Values=available" \
  --query 'Volumes[*].[VolumeId,Size,CreateTime]' \
  --output table

# Result: 147 orphaned volumes, 18 TB total
# Cost: $1,800/month for volumes attached to NOTHING

3. NAT Gateway Money Pit

We had 9 NAT Gateways across 3 regions, 3 AZs each. Cost: $273/month per NAT Gateway + data transfer charges.

Most shocking? 78% of NAT traffic was pods pulling Docker images from public registries. We were paying AWS $6,400/month to download free container images from Docker Hub.

4. Cluster Autoscaler Waste

# Our node groups before optimization
nodeGroups:
- name: general-purpose
  instanceType: m5.2xlarge  # 8 vCPUs, 32GB RAM
  desiredCapacity: 20
  minSize: 15  # Always keep 15 nodes running
  maxSize: 40

Even at 2 AM on Sunday with zero traffic, we had 15 nodes running. Cost: $7,200/month for idle capacity.

The Wake-Up Call Metrics

Let’s be brutally honest about our waste:

Actual Cluster Resource Usage (Oct 2022):

• CPU Reserved: 256 cores
• CPU Actually Used: 59 cores (23% utilization)
• Memory Reserved: 1024 GB
• Memory Actually Used: 318 GB (31% utilization)

The Math:

• We had capacity for 10,000 pods
• We were running 1,200 pods
• We were paying for 8x more infrastructure than we needed

Cost per Pod: $46.67/month (completely unsustainable)

The Executive Mandate:

“Cut costs by 20% within 6 months, or we’re moving to a different cloud provider.”

Gulp. Time to learn FinOps.

gantt
    title 6-Month Cost Optimization Journey
    dateFormat  YYYY-MM-DD
    section Phase 1
    Visibility & Kubecost           :done, p1, 2022-10-15, 2w
    Cost Analysis & Tagging         :done, p2, 2022-10-29, 1w
    
    section Phase 2
    Quick Wins                      :done, q1, 2022-11-05, 2w
    Orphaned Volumes Cleanup        :done, q2, 2022-11-05, 2d
    Right-size Staging              :done, q3, 2022-11-07, 3d
    Auto-shutdown Non-Prod          :done, q4, 2022-11-10, 1w
    Storage Optimization            :done, q5, 2022-11-17, 3d
    
    section Phase 3
    Karpenter Migration             :done, k1, 2022-11-20, 4w
    Testing & Validation            :done, k2, 2022-12-18, 1w
    
    section Phase 4
    Spot Instance Strategy          :done, s1, 2022-12-25, 3w
    Graceful Termination Setup      :done, s2, 2023-01-01, 1w
    Gradual Spot Rollout            :done, s3, 2023-01-08, 2w
    
    section Phase 5
    Network Optimization            :done, n1, 2023-01-22, 3w
    VPC Endpoints                   :done, n2, 2023-01-22, 1w
    NAT Gateway Reduction           :done, n3, 2023-01-29, 1w
    Topology-aware Routing          :done, n4, 2023-02-05, 1w
    
    section Phase 6
    FinOps Culture Building         :done, c1, 2023-02-12, 8w
    Monthly Reviews & Optimization  :active, c2, 2023-04-01, 12w

Phase 1: Visibility - You Can’t Optimize What You Can’t See

Building Cost Observability

The Problem: We had Prometheus monitoring every technical metric, but zero cost visibility. We didn’t know:

• Which teams were spending what
• Which namespaces were most expensive
• Which workloads drove costs
• Where optimization would have the most impact

The Solution: Build a complete cost observability stack.

Step 1: Kubernetes Cost Attribution with Kubecost

We deployed Kubecost to track Kubernetes costs in real-time:

# kubecost-values.yaml
kubecostProductConfigs:
  cloudIntegrationSecret: kubecost-cloud-integration
  
  # AWS billing integration
  athenaProjectID: kubecost-billing
  athenaBucketName: fis-cur-bucket
  athenaRegion: us-east-1
  athenaDatabase: athenacurcfn_fis_billing
  athenaTable: fis_billing_data

# High availability
prometheus:
  server:
    persistentVolume:
      size: 100Gi
      storageClass: gp3
  
ingress:
  enabled: true
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - kubecost.fis.com

Deploy:

helm install kubecost kubecost/cost-analyzer \
  --namespace kubecost \
  --create-namespace \
  --values kubecost-values.yaml

# Access dashboard
kubectl port-forward -n kubecost \
  svc/kubecost-cost-analyzer 9090:9090

Within 24 hours, Kubecost revealed:

Key Insights:

1. ML-training namespace: 8% efficiency, $890 per pod (!!)
2. Staging environment: Using nearly as much as production
3. Most teams: Running 24/7 resources for dev/test

Step 2: Cost Allocation Tags

We implemented comprehensive tagging:

# Terraform - tag everything
locals {
  common_tags = {
    Project     = "kubernetes-platform"
    Team        = var.team_name
    Environment = var.environment
    CostCenter  = var.cost_center
    ManagedBy   = "terraform"
    Application = var.application_name
  }
}

resource "aws_instance" "karpenter_node" {
  # ... instance config ...
  
  tags = merge(
    local.common_tags,
    {
      Name = "karpenter-node-${var.cluster_name}"
      "karpenter.sh/discovery" = var.cluster_name
    }
  )

  volume_tags = merge(
    local.common_tags,
    {
      Name = "karpenter-node-volume"
    }
  )
}

# Kubernetes - label everything
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    team: "platform"
    cost-center: "engineering"
    environment: "production"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
  labels:
    app: api-server
    cost-allocation: "product"
    team: "backend"

Step 3: Cost Dashboards

We built Grafana dashboards showing:

1. Real-time Cost Dashboard
   • Current hourly burn rate
   • Projected monthly cost
   • Cost by namespace/team/environment
   • Efficiency metrics
2. Waste Dashboard
   • Idle resources (< 10% utilization)
   • Overprovisioned resources (> 50% unused)
   • Abandoned resources (no owner tags)
   • Orphaned volumes
3. Optimization Opportunity Dashboard
   • Spot instance candidates
   • Right-sizing recommendations
   • Reserved Instance opportunities

Sample Prometheus queries we used:

# Cost per namespace
sum(
  avg_over_time(
    container_memory_working_set_bytes[1h]
  ) * on(node) group_left() 
  node_ram_hourly_cost
) by (namespace)

# Idle nodes (< 10% CPU utilization)
(
  1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))
) < 0.1

# Pods with no resource limits (cost risk)
count(
  kube_pod_container_resource_limits{resource="memory"} == 0
) by (namespace)

The Impact of Visibility

Before Kubecost:

• Finance meetings: “Why is the bill so high?”
• Us: “¯\(ツ)/¯”

After Kubecost:

• Finance meetings: “Team X is spending $4K/month on staging”
• Us: “Yes, here’s why and here’s our optimization plan”

More importantly: Engineers could see their own costs and started optimizing proactively.

Phase 2: Quick Wins - Low-Hanging Fruit ($2K/month saved)

Once we had visibility, some optimizations were embarrassingly obvious.

Quick Win 1: Delete Orphaned EBS Volumes ($1,800/month)

The Problem: 147 orphaned volumes from deleted nodes.

The Solution:

#!/bin/bash
# cleanup-orphaned-volumes.sh

# Find available (unattached) volumes
ORPHANED_VOLUMES=$(aws ec2 describe-volumes \
  --filters "Name=status,Values=available" \
  --query 'Volumes[*].VolumeId' \
  --output text)

for VOLUME in $ORPHANED_VOLUMES; do
  # Get volume age
  CREATE_TIME=$(aws ec2 describe-volumes \
    --volume-ids $VOLUME \
    --query 'Volumes[0].CreateTime' \
    --output text)
  
  AGE_DAYS=$(( ($(date +%s) - $(date -d "$CREATE_TIME" +%s)) / 86400 ))
  
  # Delete volumes older than 7 days
  if [ $AGE_DAYS -gt 7 ]; then
    echo "Deleting volume $VOLUME (age: $AGE_DAYS days)"
    aws ec2 delete-volume --volume-id $VOLUME
  fi
done

Automated going forward:

# Terraform lifecycle rule
resource "aws_ebs_volume" "karpenter_volume" {
  # ... config ...
  
  lifecycle {
    prevent_destroy = false
  }

  tags = {
    "karpenter.sh/managed" = "true"
    DeleteAfterDays = "7"
  }
}

# Lambda to auto-cleanup
resource "aws_lambda_function" "cleanup_volumes" {
  function_name = "cleanup-orphaned-ebs-volumes"
  runtime       = "python3.9"
  handler       = "lambda_function.lambda_handler"
  
  environment {
    variables = {
      RETENTION_DAYS = "7"
    }
  }
}

# EventBridge rule - run daily
resource "aws_cloudwatch_event_rule" "daily" {
  name                = "daily-ebs-cleanup"
  schedule_expression = "rate(1 day)"
}

Savings: $1,800/month

Quick Win 2: Right-Size Dev/Staging Environments ($300/month)

The Discovery: Staging was using 60% of production resources but handling < 5% of traffic.

# Before - Staging was production copy
staging:
  replicas: 10  # Same as prod
  resources:
    requests:
      cpu: 2000m
      memory: 4Gi  # Same as prod

After - Appropriately sized:

# After - Right-sized for actual load
staging:
  replicas: 2  # 80% reduction
  resources:
    requests:
      cpu: 500m   # 75% reduction
      memory: 1Gi # 75% reduction

The Policy:

# staging-policy.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: staging-limits
  namespace: staging
spec:
  limits:
  - max:
      cpu: "2"
      memory: "4Gi"
    min:
      cpu: "100m"
      memory: "128Mi"
    type: Container
  - max:
      cpu: "8"
      memory: "16Gi"
    type: Pod

Savings: $300/month

Quick Win 3: Shut Down Non-Prod After Hours ($400/month)

The Insight: Dev/test environments don’t need to run 24/7.

# kube-downscaler configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-downscaler
  namespace: kube-downscaler
data:
  config.yaml: |
    # Downscale to 0 replicas outside business hours
    DEFAULT_UPTIME: "Mon-Fri 08:00-18:00 America/New_York"
    DEFAULT_DOWNTIME: "never"
    
    # Namespaces to downscale
    DOWNSCALE_NAMESPACES: "dev-.*,staging,test-.*"
    
    # Grace period
    GRACE_PERIOD: 300

# Install kube-downscaler
helm repo add caiodelgado https://caiodelgadonew.github.io/helm-charts
helm install kube-downscaler caiodelgado/kube-downscaler \
  --namespace kube-downscaler \
  --create-namespace \
  --values downscaler-values.yaml

Annotation for exceptions:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: critical-test-service
  annotations:
    # Opt-out of downscaling
    downscaler/exclude: "true"

Savings: $400/month (64% savings on dev/test infrastructure)

Quick Win 4: Reduce Log Retention ($200/month)

The Problem: CloudWatch Logs with infinite retention.

# Before
resource "aws_cloudwatch_log_group" "eks_cluster" {
  name              = "/aws/eks/${var.cluster_name}/cluster"
  retention_in_days = 0  # Never expire (!!!)
}

After:

# Tiered retention strategy
resource "aws_cloudwatch_log_group" "eks_cluster" {
  name              = "/aws/eks/${var.cluster_name}/cluster"
  retention_in_days = 7  # Control plane logs
}

resource "aws_cloudwatch_log_group" "application_logs" {
  name              = "/aws/eks/${var.cluster_name}/applications"
  retention_in_days = 30  # Application logs
}

resource "aws_cloudwatch_log_group" "audit_logs" {
  name              = "/aws/eks/${var.cluster_name}/audit"
  retention_in_days = 365  # Compliance requirement
}

Plus, moved to S3 for long-term storage:

# Export to S3 after 7 days
resource "aws_s3_bucket" "log_archive" {
  bucket = "fis-eks-logs-archive"

  lifecycle_rule {
    enabled = true

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 90
      storage_class = "GLACIER"
    }

    expiration {
      days = 365
    }
  }
}

Savings: $200/month

Quick Win 5: Optimize EBS Volume Types ($300/month)

The Discovery: Everything was gp2 (old generation).

# Before
aws ec2 describe-volumes \
  --query 'Volumes[*].[VolumeId,VolumeType,Size]' \
  --output table

# All gp2 volumes:
# 100 GB gp2 = $10/month + $0.10/IOPS
# Total: $30,000/month

After - Migrate to gp3:

# gp3 is 20% cheaper + free 3,000 IOPS baseline
# 100 GB gp3 = $8/month (no IOPS charges for baseline)

# Migration script
for VOLUME_ID in $(aws ec2 describe-volumes \
  --filters "Name=volume-type,Values=gp2" \
  --query 'Volumes[*].VolumeId' \
  --output text); do
  
  aws ec2 modify-volume \
    --volume-id $VOLUME_ID \
    --volume-type gp3
done

Terraform going forward:

resource "aws_ebs_volume" "default" {
  availability_zone = var.availability_zone
  size              = var.volume_size
  type              = "gp3"  # Not gp2
  
  # gp3 allows specifying IOPS and throughput
  iops       = 3000   # Free tier
  throughput = 125    # Free tier

  encrypted  = true
  kms_key_id = var.kms_key_arn

  tags = local.common_tags
}

Savings: $300/month (20% reduction in EBS costs)

Total Quick Wins: $2,000/month (3.6% savings)

These optimizations took us < 2 weeks and required zero application changes.

Phase 3: Karpenter - The Game Changer ($1,500/month saved)

graph TB
    subgraph ClusterAutoscaler["Cluster Autoscaler (Before)"]
        CA_Fixed["Fixed Node Groups<br/>m5.2xlarge only"]
        CA_Min["Minimum: 15 nodes<br/>Always running<br/>$7,200/month waste"]
        CA_Slow["Scale-up time:<br/>5-10 minutes"]
        CA_Util["Utilization: 30%<br/>70% waste"]
        CA_Spot["Spot instances: 0%<br/>100% On-Demand"]
        
        CA_Fixed --> CA_Min
        CA_Min --> CA_Slow
        CA_Slow --> CA_Util
        CA_Util --> CA_Spot
    end
    
    subgraph Karpenter["Karpenter (After)"]
        K_Dynamic["Dynamic Selection<br/>Multiple instance types"]
        K_Zero["Minimum: 0 nodes<br/>Scale to zero<br/>$0 idle cost"]
        K_Fast["Scale-up time:<br/>< 60 seconds"]
        K_Util["Utilization: 65%<br/>Bin-packing magic"]
        K_Spot["Spot instances: 70%<br/>30% On-Demand"]
        
        K_Dynamic --> K_Zero
        K_Zero --> K_Fast
        K_Fast --> K_Util
        K_Util --> K_Spot
    end
    
    subgraph Results["Cost Impact"]
        Before["Before: $21,000/mo<br/>Efficiency: 30%"]
        After["After: $19,500/mo<br/>Efficiency: 65%"]
        Savings["Savings: $1,500/mo<br/>7% reduction<br/>116% efficiency gain"]
    end
    
    ClusterAutoscaler -.-> Before
    Karpenter -.-> After
    Before --> Savings
    After --> Savings
    
    style ClusterAutoscaler fill:#ffcccc
    style Karpenter fill:#ccffcc
    style Results fill:#cce5ff
    style Savings fill:#ffffcc

Why Cluster Autoscaler Was Killing Our Budget

Cluster Autoscaler Problems:

1. Fixed Node Groups ```yaml

   We had to predefine node groups

   • name: general-m5-large instanceType: m5.large minSize: 5 # Always keep 5 running

• name: general-m5-xlarge instanceType: m5.xlarge minSize: 3 # Always keep 3 running

Total idle capacity: $2,100/month

2. **Slow Scaling**
- Pod pending → 5-10 minutes to launch node
- Meanwhile, requests queue up or timeout
- Solution? Over-provision (more waste)

3. **Bin-Packing Failures**

Node 1: 2 vCPU, 8 GB RAM Pod A: 1.5 vCPU, 2 GB RAM Pod B: 0.3 vCPU, 1 GB RAM Wasted: 0.2 vCPU, 5 GB RAM (63% memory wasted!)

Node 2: 2 vCPU, 8 GB RAM Pod C: 0.5 vCPU, 6 GB RAM Wasted: 1.5 vCPU, 2 GB RAM (75% CPU wasted!)

### Enter Karpenter

**Karpenter's Magic:**
1. No fixed node groups needed
2. Automatically selects best instance type
3. Consolidates underutilized nodes
4. Launches nodes in < 60 seconds
5. Uses Spot instances intelligently

### Karpenter Implementation

```yaml
# karpenter-nodepool.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      requirements:
        # Allow Karpenter to choose instance type
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        
        # Modern instances only (gen 6+)
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        
        # Allow multiple families
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["c6i", "c6a", "m6i", "m6a", "r6i", "r6a"]
        
        # CPU architecture
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        
        # Instance size range
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["large", "xlarge", "2xlarge", "4xlarge"]
      
      nodeClassRef:
        name: default

  # Disruption settings - key for cost optimization!
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s
    expireAfter: 720h  # 30 days

  # Set limits
  limits:
    cpu: "1000"
    memory: 1000Gi

---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: "KarpenterNodeRole-${CLUSTER_NAME}"
  
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}
  
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}
  
  # Use gp3 volumes
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 3000
        throughput: 125
        encrypted: true
        deleteOnTermination: true
  
  # User data for custom bootstrap
  userData: |
    #!/bin/bash
    /etc/eks/bootstrap.sh ${CLUSTER_NAME}
  
  tags:
    Name: "karpenter-node-${CLUSTER_NAME}"
    Environment: ${ENVIRONMENT}
    ManagedBy: karpenter

Spot Instance Strategy

The Key: Mix Spot and On-Demand intelligently.

# Production workloads - prefer On-Demand
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  template:
    spec:
      nodeSelector:
        karpenter.sh/capacity-type: on-demand
      tolerations:
        - key: spot
          operator: Equal
          value: "false"
          effect: NoSchedule
---
# Batch jobs - Spot is fine
apiVersion: batch/v1
kind: Job
metadata:
  name: data-processing
spec:
  template:
    spec:
      nodeSelector:
        karpenter.sh/capacity-type: spot
      tolerations:
        - key: spot
          operator: Equal
          value: "true"
          effect: NoSchedule

Karpenter automatically:

• Uses Spot for 70% of workloads (saves 70%)
• Falls back to On-Demand if Spot unavailable
• Diversifies across instance types (reduces Spot interruptions)

Consolidation is Magic

Before Karpenter:

Node 1 (m5.xlarge): 10% CPU, 15% Memory
Node 2 (m5.xlarge): 12% CPU, 18% Memory
Node 3 (m5.xlarge): 8% CPU, 12% Memory

Cost: 3 × $140/month = $420/month

After Karpenter Consolidation:

# Karpenter moves all pods to 1 node, terminates others
Node 1 (m5.xlarge): 30% CPU, 45% Memory
Nodes 2 & 3: Terminated

Cost: 1 × $140/month = $140/month
Savings: $280/month per consolidation event

How often does this happen? In our cluster, 5-10 times per day during low-traffic periods.

The Impact of Karpenter

Before Karpenter (Cluster Autoscaler):

• Fixed node groups: 20 m5.2xlarge nodes
• Minimum capacity: 15 nodes always running
• Average utilization: 30%
• Cost: $21,000/month

After Karpenter:

• Dynamic node selection
• Minimum capacity: 0 nodes (scales to zero!)
• Average utilization: 65%
• Cost: $19,500/month

Savings: $1,500/month (7% of total)

Additional benefits:

• Faster scaling: 10 min → 60 seconds
• Better bin-packing: 30% → 65% utilization
• Spot instance usage: 0% → 70%
• Less operational overhead

Phase 4: Spot Instances Strategy ($800/month saved)

flowchart TD
    Start[New Pod Created] --> Classify{Workload Type?}
    
    Classify -->|Stateless API| SpotOK1[Spot-Safe ✅]
    Classify -->|Background Job| SpotOK2[Spot-Safe ✅]
    Classify -->|Batch Processing| SpotOK3[Spot-Safe ✅]
    Classify -->|Stateful Service| SpotPartial[Partial Spot ⚠️]
    Classify -->|Database| NoSpot[No Spot ❌]
    Classify -->|Critical API| SpotPartial2[Partial Spot ⚠️]
    
    SpotOK1 --> Spot100[100% Spot<br/>70% cost savings]
    SpotOK2 --> Spot100
    SpotOK3 --> Spot100
    
    SpotPartial --> Spot50[50% Spot<br/>50% On-Demand<br/>35% savings]
    SpotPartial2 --> Spot30[30% Spot<br/>70% On-Demand<br/>21% savings]
    
    NoSpot --> OnDemand[100% On-Demand<br/>0% savings<br/>Maximum reliability]
    
    Spot100 --> Protection{Protection<br/>Configured?}
    Spot50 --> Protection
    Spot30 --> Protection
    
    Protection -->|Yes| Safe[✅ PDB<br/>✅ Graceful shutdown<br/>✅ Node termination handler<br/>✅ Multi-AZ spread]
    Protection -->|No| Risk[⚠️ Risk of disruption<br/>Configure protection first!]
    
    Safe --> Launch[Launch on Spot<br/>Karpenter handles lifecycle]
    Risk --> Fix[Fix configuration<br/>then retry]
    Fix --> Protection
    
    OnDemand --> Launch2[Launch on On-Demand<br/>Standard provisioning]
    
    Launch --> Monitor[Monitor<br/>Interruption Rate<br/>Target: < 2%]
    Launch2 --> Monitor
    
    Monitor --> Success[Success!<br/>Average savings: 70% on Spot<br/>Zero user impact]
    
    style SpotOK1 fill:#ccffcc
    style SpotOK2 fill:#ccffcc
    style SpotOK3 fill:#ccffcc
    style SpotPartial fill:#ffffcc
    style SpotPartial2 fill:#ffffcc
    style NoSpot fill:#ffcccc
    style Safe fill:#ccffcc
    style Risk fill:#ffcccc
    style Success fill:#ccffff

Understanding Spot Economics

Spot pricing:

• m5.xlarge On-Demand: $0.192/hour = $140/month
• m5.xlarge Spot: $0.057/hour = $42/month
• Savings: 70% 💰

The catch: AWS can reclaim Spot instances with 2 minutes notice.

Spot-Safe Workload Classification

We categorized our workloads:

Graceful Spot Termination Handling

# spot-handler deployment
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aws-node-termination-handler
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: aws-node-termination-handler
  template:
    metadata:
      labels:
        app: aws-node-termination-handler
    spec:
      serviceAccountName: aws-node-termination-handler
      containers:
      - name: aws-node-termination-handler
        image: public.ecr.aws/aws-ec2/aws-node-termination-handler:v1.19.0
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: ENABLE_SPOT_INTERRUPTION_DRAINING
          value: "true"
        - name: ENABLE_SCHEDULED_EVENT_DRAINING
          value: "true"

What it does:

1. Listens for Spot termination notice (2 min warning)
2. Cordons the node (no new pods)
3. Drains existing pods gracefully
4. Pods reschedule to other nodes
5. Zero user-facing impact

Application-Level Spot Handling

# Deployment with proper graceful shutdown
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 10
  
  # Spread across nodes and zones
  template:
    spec:
      # PodDisruptionBudget ensures minimum availability
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: api-server
      
      # Prefer Spot, tolerate On-Demand
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: karpenter.sh/capacity-type
                operator: In
                values: ["spot"]
      
      tolerations:
      - key: spot
        operator: Equal
        value: "true"
        effect: NoSchedule
      
      containers:
      - name: api
        image: api-server:v1.2.3
        
        # Graceful shutdown
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
        
        # Startup/readiness probes
        startupProbe:
          httpGet:
            path: /healthz
            port: 8080
          failureThreshold: 30
          periodSeconds: 10
        
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          periodSeconds: 5
      
      # Allow graceful termination
      terminationGracePeriodSeconds: 30

PodDisruptionBudget for High Availability

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-server-pdb
spec:
  minAvailable: 7  # Keep at least 7 out of 10 replicas
  selector:
    matchLabels:
      app: api-server

This ensures:

• Even if Spot interruptions happen
• At least 7 replicas stay running
• Zero user-facing impact

Spot Instance Diversification

The Strategy: Don’t rely on one instance type.

# Karpenter automatically diversifies
spec:
  requirements:
    - key: karpenter.k8s.aws/instance-family
      operator: In
      values: ["c6i", "c6a", "m6i", "m6a", "r6i", "r6a"]
    
    - key: karpenter.k8s.aws/instance-size
      operator: In
      values: ["xlarge", "2xlarge", "4xlarge"]

Karpenter spreads Spot instances across:

• 6 instance families
• 3 sizes
• 3 availability zones
• = 54 different Spot pools

Why this matters: If one Spot pool has high interruption rates, Karpenter automatically shifts to other pools. Our Spot interruption rate: < 2% (industry average: 5-10%).

Spot Instance Monitoring

# Prometheus alert for Spot interruptions
groups:
- name: spot-instances
  rules:
  - alert: HighSpotInterruptionRate
    expr: |
      sum(rate(spot_interruption_total[1h])) > 5
    for: 15m
    annotations:
      summary: "High Spot interruption rate detected"
      description: " Spot interruptions in last hour"
  
  - alert: SpotNodeCordon
    expr: |
      kube_node_spec_unschedulable{node=~".*spot.*"} == 1
    for: 5m
    annotations:
      summary: "Spot node cordoned"
      description: "Node  has been cordoned"

The Spot Instance Results

Spot Usage Breakdown:

Spot Interruption Impact:

• Total interruptions (6 months): 67 events
• User-facing impact: 0 incidents
• Average pod rescheduling time: 8 seconds

Savings: $800/month (1.4% of total)

Phase 5: Storage Optimization ($400/month saved)

The Storage Audit Revealed Waste

# Total EBS volumes
$ aws ec2 describe-volumes \
  --query 'Volumes[*].[VolumeId,Size,State]' | wc -l
432 volumes

# Total storage
$ aws ec2 describe-volumes \
  --query 'sum(Volumes[*].Size)'
18,436 GB (18 TB!)

# Cost
18,436 GB × $0.08/GB = $1,475/month

Storage Optimization 1: PV Reclaim Policy

The Problem: PersistentVolumes kept forever after pod deletion.

# Before - Default StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain  # Never delete!

After - Smart reclaim policy:

# Production - Retain (safety first)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-prod
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
reclaimPolicy: Retain  # Keep for production
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
# Dev/Test - Delete to save money
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-dev
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
reclaimPolicy: Delete  # Auto-cleanup
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Savings: $150/month (dev/test volume cleanup)

Storage Optimization 2: Right-Size PVCs

The Discovery: Developers requesting huge volumes “just in case.”

# Before - Typical developer request
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Gi  # "Better safe than sorry"
  storageClassName: gp3

# Actual usage: 8 GB
# Waste: 492 GB × $0.08 = $39/month per PVC

Solution - Resource Quotas:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
  namespace: development
spec:
  hard:
    requests.storage: 100Gi  # Total storage limit per namespace
    persistentvolumeclaims: "10"  # Max PVC count
---
apiVersion: v1
kind: LimitRange
metadata:
  name: storage-limits
  namespace: development
spec:
  limits:
  - type: PersistentVolumeClaim
    max:
      storage: 20Gi  # No PVC over 20GB in dev
    min:
      storage: 1Gi

Plus, monitoring & alerts:

# Prometheus alert for oversized PVCs
- alert: PVCOverprovisioned
  expr: |
    (
      kubelet_volume_stats_used_bytes / 
      kubelet_volume_stats_capacity_bytes
    ) < 0.1
  for: 7d
  annotations:
    summary: "PVC  using < 10%"
    description: "Consider downsizing from  GB"

Savings: $120/month (right-sizing volumes)

Storage Optimization 3: EBS Volume Snapshots

The Problem: Taking full snapshots daily = expensive.

# Before
aws ec2 create-snapshot --volume-id vol-xxx --description "daily backup"
# Full 500 GB snapshot = $25/month
# 30 days retention = $750/month

After - Incremental snapshots + lifecycle:

# EBS Snapshot Lifecycle Policy
resource "aws_dlm_lifecycle_policy" "ebs_snapshots" {
  description        = "EBS snapshot lifecycle"
  execution_role_arn = aws_iam_role.dlm_lifecycle.arn
  state              = "ENABLED"

  policy_details {
    resource_types = ["VOLUME"]

    schedule {
      name = "Daily snapshots"

      create_rule {
        interval      = 24
        interval_unit = "HOURS"
        times         = ["03:00"]
      }

      retain_rule {
        count = 7  # Keep 7 daily snapshots
      }

      tags_to_add = {
        SnapshotCreator = "DLM"
        Type            = "automated"
      }

      copy_tags = true
    }

    schedule {
      name = "Weekly snapshots"

      create_rule {
        cron_expression = "cron(0 3 ? * SUN *)"
      }

      retain_rule {
        count = 4  # Keep 4 weekly snapshots
      }
    }

    target_tags = {
      Backup = "true"
    }
  }
}

Incremental snapshot magic:

• First snapshot: 500 GB
• Second snapshot: Only changed blocks (~10 GB)
• Third snapshot: Only changed blocks (~10 GB)

Savings: $130/month (incremental vs full snapshots)

Total Storage Savings: $400/month

Phase 6: Network Cost Optimization ($300/month saved)

graph LR
    subgraph Before["Before Optimization - $11.5K/month"]
        Pod1["Pod"] --> NAT1["NAT Gateway<br/>$0.045/GB"]
        NAT1 --> Internet1["Internet"]
        Internet1 --> S31["S3<br/>Docker Images"]
        Internet1 --> ECR1["ECR<br/>Container Registry"]
        Internet1 --> AWS1["AWS APIs"]
        
        NATCost1["9 NAT Gateways<br/>$2,460/mo<br/>+<br/>143 TB processed<br/>$5,740/mo"]
        CrossAZ1["Cross-AZ Traffic<br/>105 TB × $0.02<br/>$2,100/mo"]
    end
    
    subgraph After["After Optimization - $6.5K/month"]
        Pod2["Pod"] --> Check{Destination?}
        
        Check -->|AWS Service| VPCEndpoint["VPC Endpoint<br/>FREE data transfer"]
        Check -->|Internet| NAT2["NAT Gateway<br/>3 total"]
        Check -->|Same AZ| SameAZ["Same-AZ Pod<br/>$0 transfer"]
        
        VPCEndpoint --> S32["S3 Gateway<br/>Endpoint"]
        VPCEndpoint --> ECR2["ECR Interface<br/>Endpoint"]
        VPCEndpoint --> AWS2["AWS Service<br/>Endpoints"]
        
        NAT2 --> Internet2["Internet<br/>Reduced traffic"]
        
        NATCost2["3 NAT Gateways<br/>$97/mo<br/>+<br/>78 TB processed<br/>$3,400/mo"]
        CrossAZ2["Cross-AZ Traffic<br/>42 TB × $0.02<br/>$840/mo<br/>Topology-aware routing"]
    end
    
    subgraph Savings["Monthly Savings"]
        Save1["NAT Gateway: -$194/mo"]
        Save2["VPC Endpoints: -$2,003/mo"]
        Save3["Cross-AZ: -$106/mo"]
        Total["Total: -$2,303/mo<br/>20% network cost reduction"]
    end
    
    Before -.-> Save1
    Before -.-> Save2
    Before -.-> Save3
    After -.-> Total
    
    style Before fill:#ffcccc
    style After fill:#ccffcc
    style Savings fill:#cce5ff
    style Total fill:#ffffcc

Network costs are the silent budget killer in Kubernetes.

Network Cost Breakdown (Before)

NAT Gateway charges:     $8,200/month
- NAT Gateway hours:     $2,460/month (9 gateways × $0.045/hour)
- Data processing:       $5,740/month (143 TB × $0.045/GB)

Data Transfer:           $3,360/month
- Cross-AZ:             $2,100/month (105 TB × $0.02/GB)
- Internet egress:      $1,260/month (14 TB × $0.09/GB)

Total Network:          $11,560/month

Network Optimization 1: Reduce NAT Gateway Count

The Discovery: 9 NAT Gateways was overkill.

Before:

Region: us-east-1
├── AZ-A: NAT Gateway 1, 2, 3
├── AZ-B: NAT Gateway 4, 5, 6
└── AZ-C: NAT Gateway 7, 8, 9

Cost: 9 × $32.40/month = $291/month

After - One per AZ:

Region: us-east-1
├── AZ-A: NAT Gateway 1
├── AZ-B: NAT Gateway 2
└── AZ-C: NAT Gateway 3

Cost: 3 × $32.40/month = $97/month

Terraform optimization:

# One NAT Gateway per AZ (not per subnet)
resource "aws_nat_gateway" "main" {
  count = length(var.availability_zones)

  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = aws_subnet.public[count.index].id

  tags = merge(
    local.common_tags,
    {
      Name = "nat-gateway-${var.availability_zones[count.index]}"
    }
  )
}

# Route tables - one per AZ
resource "aws_route_table" "private" {
  count  = length(var.availability_zones)
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.main[count.index].id
  }

  tags = merge(
    local.common_tags,
    {
      Name = "private-rt-${var.availability_zones[count.index]}"
      Tier = "Private"
    }
  )
}

Savings: $194/month (NAT Gateway count reduction)

Network Optimization 2: VPC Endpoints for AWS Services

The Problem: API calls to S3, ECR going through NAT Gateway.

# Data flow before VPC endpoints
Pod → NAT Gateway → Internet → S3
     ↑ Costs $0.045/GB processed

# Monthly S3/ECR traffic: 45 TB
# Cost: 45,000 GB × $0.045 = $2,025/month

Solution - VPC Endpoints:

# S3 Gateway Endpoint (FREE!)
resource "aws_vpc_endpoint" "s3" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.${var.region}.s3"
  
  route_table_ids = aws_route_table.private[*].id

  tags = merge(
    local.common_tags,
    {
      Name = "s3-endpoint"
    }
  )
}

# ECR API Endpoint
resource "aws_vpc_endpoint" "ecr_api" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.api"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true

  tags = merge(
    local.common_tags,
    {
      Name = "ecr-api-endpoint"
    }
  )
}

# ECR Docker Endpoint
resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true

  tags = merge(
    local.common_tags,
    {
      Name = "ecr-dkr-endpoint"
    }
  )
}

# Additional endpoints
resource "aws_vpc_endpoint" "ec2" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ec2"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

# CloudWatch Logs
resource "aws_vpc_endpoint" "logs" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.logs"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

Cost breakdown:

• Interface endpoints: 3 × $7.20/month = $21.60/month
• Data processing: $0 (included!)
• Net savings: $2,003/month 🎉

But wait, there’s a catch…

Network Optimization 3: Minimize Cross-AZ Traffic

The Hidden Cost: Cross-AZ data transfer = $0.01/GB in EACH direction.

Example:

Pod in us-east-1a → Database in us-east-1b
Request: 1 MB
Response: 10 MB

Cost: (1 MB + 10 MB) × 2 × $0.01 = $0.22/transfer
If this happens 1M times/day = $220/day = $6,600/month

Solution - Topology-Aware Routing:

# Enable topology-aware hints
apiVersion: v1
kind: Service
metadata:
  name: api-service
  annotations:
    service.kubernetes.io/topology-aware-hints: "auto"
spec:
  type: ClusterIP
  selector:
    app: api-server
  ports:
  - port: 80
    targetPort: 8080

What this does:

• Routes traffic to pods in same AZ when possible
• Falls back to cross-AZ only when necessary
• Reduces cross-AZ traffic by ~60%

Pod Topology Spread:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 9  # 3 per AZ
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: api-server

Savings: $106/month (60% reduction in cross-AZ traffic)

Total Network Savings: $300/month

Phase 7: Building a FinOps Culture

graph TB
    subgraph Vision["FinOps Vision"]
        Goal["Everyone Responsible<br/>for Cloud Costs"]
    end
    
    subgraph Visibility["Visibility Layer"]
        Kubecost["Kubecost<br/>Real-time cost monitoring"]
        Tags["Cost Allocation Tags<br/>Team/Project/Environment"]
        Dashboards["Grafana Dashboards<br/>Per-team visibility"]
    end
    
    subgraph Accountability["Accountability Layer"]
        Teams["12 Engineering Teams"]
        Quotas["Resource Quotas<br/>Budget limits"]
        Alerts["Cost Alerts<br/>Slack notifications"]
    end
    
    subgraph Optimization["Optimization Layer"]
        Council["FinOps Council<br/>Weekly meetings"]
        Reviews["Monthly Cost Reviews<br/>Per team"]
        Gamification["Leaderboard<br/>Cost optimization competition"]
    end
    
    subgraph Automation["Automation Layer"]
        Cleanup["Auto-cleanup<br/>Orphaned resources"]
        Rightsizing["Right-sizing<br/>Recommendations"]
        Shutdown["Auto-shutdown<br/>Non-prod after hours"]
    end
    
    subgraph Results["Results"]
        Culture["Cost-aware Culture<br/>Bottom-up optimization"]
        Sustained["Sustained 15% Savings<br/>$96K/year"]
        Innovation["Savings fund innovation<br/>2 new engineers hired"]
    end
    
    Goal --> Kubecost
    Goal --> Tags
    Goal --> Dashboards
    
    Kubecost --> Teams
    Tags --> Quotas
    Dashboards --> Alerts
    
    Teams --> Council
    Quotas --> Reviews
    Alerts --> Gamification
    
    Council --> Cleanup
    Reviews --> Rightsizing
    Gamification --> Shutdown
    
    Cleanup --> Culture
    Rightsizing --> Sustained
    Shutdown --> Innovation
    
    style Vision fill:#e6f3ff
    style Visibility fill:#fff4e6
    style Accountability fill:#ffe6f3
    style Optimization fill:#f3e6ff
    style Automation fill:#e6fff3
    style Results fill:#ccffcc

Technical optimizations are half the battle. Cultural change is the other half.

The FinOps Principles We Adopted

1. Everyone is Responsible for Cloud Costs
   • Not just DevOps
   • Developers own their application costs
   • Product managers make cost-aware decisions
2. Visibility Drives Accountability
   • Real-time cost dashboards
   • Cost allocated to teams
   • Monthly cost reviews
3. Continuous Optimization
   • Not a one-time project
   • Ongoing monitoring
   • Regular reviews and improvements

The FinOps Team Structure

We didn’t hire new people—we added responsibilities:

FinOps Council (meets weekly):
├── Platform Team Lead (me)
├── Engineering Manager
├── Finance Representative
└── 1 Developer from each team

Responsibilities:
- Review cost trends
- Approve optimization initiatives
- Share learnings across teams
- Set cost budgets

Cost Visibility Tools We Built

1. Per-Team Cost Dashboard

Every team got a Grafana dashboard showing:

• Their monthly spend
• Trend vs last month
• Biggest cost drivers
• Optimization opportunities
• Comparison to other teams

2. Cost Alerts (Slack Integration)

# Prometheus alert
- alert: TeamCostAnomaly
  expr: |
    (
      sum(container_memory_working_set_bytes) by (namespace) 
      * on(namespace) group_left() 
      kube_namespace_cost_per_gb_hour
    ) > 1000  # $1000/month threshold
  annotations:
    summary: " cost spike"
    slack_channel: "#finops-alerts"

Teams get Slack alerts when:

• Cost increases > 20% week-over-week
• Idle resources detected
• Untagged resources found

3. Monthly Cost Reports

Automated email to each team:

Team: Backend Engineering
Month: November 2024

Total Cost: $4,320 (-8% vs October ✅)

Breakdown:
- Compute: $2,800 (65%)
- Storage: $980 (23%)
- Network: $540 (12%)

Top Spenders:
1. api-server: $1,200
2. background-worker: $890
3. cache-cluster: $670

Optimization Opportunities:
⚠️ staging-db using 85% resources in non-business hours
💡 Potential savings: $340/month with auto-shutdown

Spot Usage: 68% (Target: 70%)
Idle Resources: 2 (Down from 8 last month ✅)

Gamification: The Cost Optimization Leaderboard

We made cost optimization fun (yes, really):

Monthly Leaderboard:

🏆 Top Cost Optimizers (November 2024)

🥇 Backend Team: -18% ($780 saved)
   Migrated batch jobs to Spot instances

🥈 Data Science Team: -15% ($520 saved)
   Implemented auto-shutdown for training jobs

🥉 Platform Team: -12% ($410 saved)
   Consolidated idle nodes with Karpenter

Recognition:
- Featured in company newsletter
- $500 team lunch budget
- "Cost Ninja" Slack emoji

The Impact:

• Teams actively look for savings
• Friendly competition
• Cost becomes a feature, not a constraint

The Cost-Aware Development Checklist

We added cost to our Definition of Done:

## Pre-Production Checklist

Performance:
- [ ] Load tested
- [ ] Response time < 200ms p95

Reliability:
- [ ] Health checks configured
- [ ] Graceful shutdown implemented

**Cost Optimization:**
- [ ] Resource requests/limits set appropriately
- [ ] Right-sized for expected load
- [ ] Spot-tolerant (if applicable)
- [ ] Auto-scaling configured
- [ ] Storage optimized (lifecycle policies)
- [ ] Cost estimated in Kubecost

Cost Optimization Training

We ran monthly “FinOps Office Hours”:

• Show Kubecost dashboard
• Walk through team’s costs
• Identify optimization opportunities
• Answer questions
• Share best practices

Topics covered:

• Week 1: Understanding Kubernetes costs
• Week 2: Right-sizing workloads
• Week 3: Spot instances strategies
• Week 4: Storage optimization

The Cultural Shift Results

Before FinOps Culture:

• Developers: “Not my problem”
• Cost conversations: Adversarial
• Optimization: Top-down mandates
• Results: Minimal, temporary

After FinOps Culture:

• Developers: Proactively optimize
• Cost conversations: Collaborative
• Optimization: Bottom-up initiatives
• Results: Sustained 15% savings

Best example: A developer noticed their ML training jobs running 24/7. They implemented auto-shutdown after training completion. Savings: $890/month. Without FinOps culture, this would never have happened.

The Results: Before vs After

graph LR
    Start["Starting Cost<br/>$56,000/mo"] --> Q1["Quick Wins<br/>-$2,000"]
    Q1 --> K["Karpenter<br/>-$1,500"]
    K --> S["Spot Instances<br/>-$800"]
    S --> ST["Storage<br/>-$400"]
    ST --> N["Network<br/>-$300"]
    N --> C["Continuous<br/>Optimization<br/>-$3,000"]
    C --> End["Final Cost<br/>$48,000/mo"]
    
    Start -.->|"Savings: $8,000/mo<br/>15% reduction"| End
    
    Q1 -.->|"2 weeks<br/>$24K annual"| Q1Text["Orphaned volumes<br/>Right-size staging<br/>Auto-shutdown<br/>Log retention<br/>gp3 migration"]
    
    K -.->|"4 weeks<br/>$18K annual"| KText["Dynamic scaling<br/>Bin-packing<br/>Scale to zero<br/>65% utilization"]
    
    S -.->|"3 weeks<br/>$9.6K annual"| SText["70% Spot usage<br/>Graceful termination<br/>Instance diversification<br/>< 2% interruption rate"]
    
    ST -.->|"2 weeks<br/>$4.8K annual"| STText["PV lifecycle<br/>Right-sizing<br/>Incremental snapshots<br/>Auto-cleanup"]
    
    N -.->|"3 weeks<br/>$3.6K annual"| NText["VPC Endpoints<br/>NAT reduction<br/>Topology-aware routing<br/>Cross-AZ optimization"]
    
    C -.->|"Ongoing<br/>$36K annual"| CText["FinOps culture<br/>Monthly reviews<br/>Team optimization<br/>Continuous improvement"]
    
    style Start fill:#ffcccc
    style End fill:#ccffcc
    style Q1 fill:#ffffcc
    style K fill:#ffffcc
    style S fill:#ffffcc
    style ST fill:#ffffcc
    style N fill:#ffffcc
    style C fill:#ffffcc

The Complete Cost Transformation

October 2022 (Before):

Total AWS Spend:           $56,000/month

Breakdown:
- EC2 Instances:           $28,000 (50%)
  └─ Utilization:          30%
- EBS Storage:             $8,400 (15%)
  └─ Orphaned volumes:     147
- NAT Gateways:            $8,200 (15%)
  └─ Count:                9
- Load Balancers:          $5,600 (10%)
- Data Transfer:           $3,360 (6%)
- Other:                   $2,440 (4%)

Efficiency Metrics:
- Cost per Pod:            $46.67
- Spot Instance Usage:     0%
- Average Node CPU:        23%
- Average Node Memory:     31%
- Idle Resources:          ~$17,000/month

April 2023 (After):

Total AWS Spend:           $48,000/month
Savings:                   $8,000/month (15%)

Breakdown:
- EC2 Instances:           $19,500 (41%) ⬇ -$8,500
  └─ Utilization:          65% ⬆
- EBS Storage:             $8,000 (17%) ⬇ -$400
  └─ Orphaned volumes:     0
- NAT Gateways:            $3,500 (7%) ⬇ -$4,700
  └─ Count:                3
- Load Balancers:          $5,300 (11%) ⬇ -$300
- Data Transfer:           $3,060 (6%) ⬇ -$300
- Other:                   $8,640 (18%) ⬆ (Kubecost, monitoring)

Efficiency Metrics:
- Cost per Pod:            $15.38 ⬇ -67%
- Spot Instance Usage:     70% ⬆
- Average Node CPU:        65% ⬆
- Average Node Memory:     68% ⬆
- Idle Resources:          ~$2,400/month ⬇ -86%

Optimization Impact Breakdown

The Scaling Paradox

Here’s the most impressive part:

Workload Growth (Oct 2022 → April 2023):

• Services: 85 → 200 (+135%)
• Pods: 1,200 → 3,600 (+200%)
• Requests/day: 10M → 30M (+200%)

Yet we cut costs by 15%.

How?

• Better utilization (30% → 65%)
• Spot instances (0% → 70%)
• Karpenter optimization
• Cultural shift

ROI on FinOps Investment

Investment:

• Engineering time: 480 hours (3 months, 2 engineers)
• Cost: ~$60,000 (loaded cost)

Returns:

• Monthly savings: $8,000
• Annual savings: $96,000
• ROI: 60% in first year
• Payback period: 7.5 months

Plus intangible benefits:

• Better infrastructure utilization
• Faster deployments (Karpenter)
• More predictable costs
• FinOps culture

The Business Impact

What $8K/month saved meant:

• 2 additional senior engineers hired
• Investment in developer tooling
• Buffer for innovation projects
• Reduced pressure from finance

CEO’s quote:

“The DevOps team didn’t just cut costs—they built a culture of efficiency. That’s even more valuable than the $96K/year savings.”

Lessons Learned and Mistakes to Avoid

After this 6-month FinOps journey, here’s what we learned:

What Worked Well

1. Start with Visibility

Trying to optimize without understanding costs = shooting in the dark.

Best Practice: Deploy Kubecost day one. You can’t fix what you can’t see.

2. Quick Wins Build Momentum

We got $2K/month savings in 2 weeks. That earned us credibility to do bigger changes.

Best Practice: Do the easy stuff first. Build trust before major changes.

3. Karpenter Was Worth the Migration

Migrating from Cluster Autoscaler took 4 weeks. Worth every hour.

Best Practice: If you’re on EKS, use Karpenter. The savings alone justify it.

4. Spot Instances Need Graceful Handling

We had zero user-facing incidents from Spot interruptions because we did it right.

Best Practice: Use AWS Node Termination Handler + PodDisruptionBudgets + proper shutdown.

5. FinOps is Cultural, Not Just Technical

Gamification, leaderboards, and making it fun drove ongoing optimization.

Best Practice: Celebrate cost savings like you celebrate performance improvements.

Mistakes We Made

1. Didn’t Tag Resources from Day One

We spent weeks retroactively tagging resources to understand costs.

Lesson: Tag everything from the start. Make it a hard requirement in Terraform.

2. Underestimated Network Costs

Network was 21% of our bill! We initially focused on compute only.

Lesson: Network optimization has huge ROI. Don’t ignore it.

3. No Cost Budget Alerts Initially

We got our first “$10K spike” surprise before we set up alerting.

Lesson: Set up CloudWatch billing alerts immediately. We use:

• Warning: > 110% of expected monthly cost
• Critical: > 125% of expected monthly cost

4. Deleted Production EBS Volume (Oops)

During cleanup script testing, we accidentally deleted a production volume.

Lesson:

• Always test scripts on dev first
• Add --dry-run flag
• Require --confirm for destructive actions
• Keep good backups (we recovered in 15 minutes)

5. Over-Optimized Early On

We were so aggressive, we caused a production incident when a critical job got evicted from Spot.

Lesson: Gradually increase Spot usage. Start with 30%, then 50%, then 70%.

Best Practices for Kubernetes Cost Optimization

1. Right-Sizing is Ongoing

Don’t set-and-forget resource requests/limits. Review quarterly.

# Review these every 3 months
resources:
  requests:
    cpu: 500m      # Is this still accurate?
    memory: 1Gi    # Usage changed?
  limits:
    cpu: 1000m     # Still needed?
    memory: 2Gi    # Can we lower?

2. Make Developers Cost-Aware

Show them their costs. They’ll optimize.

3. Automate Everything

Manual optimization doesn’t scale.

Our automation:

• Orphaned resource cleanup (daily Lambda)
• Right-sizing recommendations (weekly report)
• Cost anomaly detection (real-time)
• Spot instance management (Karpenter)

4. Use Committed Use Discounts (After Stabilization)

Once you understand baseline usage:

• Savings Plans for consistent compute
• Reserved Instances for predictable workloads
• Additional 20-30% savings on top of our 15%

We didn’t do this in year 1 because workload was too dynamic.

5. Optimize for Your Workload

Don’t blindly copy our strategy. Understand your workload:

• Batch heavy? → Spot instances are your friend
• Real-time APIs? → Karpenter + right-sizing
• Data intensive? → S3 Intelligent-Tiering
• Bursty traffic? → Karpenter consolidation

Common Cost Optimization Myths

Myth 1: “Spot instances are unreliable”

Reality: With proper architecture, Spot is 99% reliable and 70% cheaper.

Myth 2: “Kubernetes is expensive”

Reality: Kubernetes can be cheaper than VMs with proper optimization. Our cost per workload dropped 67%.

Myth 3: “Cost optimization kills performance”

Reality: Our p95 response time improved after optimization (better resource efficiency).

Myth 4: “You need a FinOps team”

Reality: You need a FinOps culture. We did it with existing team + 20% time investment.

Myth 5: “Small optimizations don’t matter”

Reality: 100 × $20/month optimizations = $24K/year savings.

Conclusion: FinOps is a Journey, Not a Destination

Where we started (October 2022):

• AWS bill spiraling out of control
• No cost visibility
• Teams unaware of costs
• Finance breathing down our necks
• Threat of moving to different cloud

Where we are now (6 months later):

• 15% cost reduction ($8K/month, $96K/year)
• Real-time cost visibility
• FinOps culture embedded in teams
• Finance happy (shockingly!)
• Scaling 3x while costs stayed flat

The key lessons:

1. Visibility first - Deploy Kubecost immediately
2. Quick wins build momentum - Start with easy optimizations
3. Karpenter changes the game - Worth the migration effort
4. Spot instances at 70% - With proper architecture
5. Culture matters most - Make cost optimization fun

The ongoing journey:

• We’re now at $44K/month (21% total reduction)
• Targeting $40K/month by year-end
• Savings funding innovation projects
• Teams proactively optimize

The most important insight:

Cost optimization isn’t about cutting corners or sacrificing quality. It’s about engineering excellence. Well-architected systems are both performant AND cost-efficient.

When you optimize for cost, you’re forced to:

• Understand your workloads
• Right-size everything
• Remove waste
• Build resilient architecture

These are the same practices that make systems reliable and fast.

FinOps isn’t a constraint—it’s a forcing function for engineering excellence.

Resources

Cost Visibility Tools:

• Kubecost - Kubernetes cost monitoring
• AWS Cost Explorer - AWS native cost analysis
• Infracost - Terraform cost estimates

Optimization Tools:

• Karpenter - Intelligent Kubernetes autoscaling
• AWS Node Termination Handler - Spot instance management
• kube-downscaler - Auto-shutdown for non-prod

FinOps Resources:

• FinOps Foundation - Best practices and community
• AWS FinOps Blog - AWS-specific optimization
• CNCF FinOps for Kubernetes - Kubernetes cost patterns

Our Infrastructure:

• My EKS Platform GitHub - Production EKS setup with cost optimization

About the Author: I’m a Senior DevOps and Cloud Engineer with 11+ years of experience. At Fidelity Information Services, I led our FinOps transformation, cutting AWS costs from $56K to $48K/month (15% reduction) while scaling workload 3x. This work was part of our platform engineering initiative that earned the “Star Team Award - DevOps 2023.” Connect with me on LinkedIn or check out my GitHub for infrastructure-as-code examples.

Questions about FinOps or Kubernetes cost optimization? Drop a comment below or reach out on LinkedIn. I’d love to hear about your cost optimization journey and challenges!

The Slack Message That Started It All:

Developer: "Hey, how do I deploy to production? I copied 
           the Jenkinsfile from team-backend but it's not working"

Me: "Which Jenkinsfile? We have like 30 different versions"

Developer: "The one that does the Docker build thing"

Me: "That describes 80% of our pipelines 😅"

Developer: "This is ridiculous. Can't we have ONE way to deploy?"

Me: "... hold that thought"

It was November 2021 at Fidelity Information Services (FIS), and our CI/CD infrastructure was a mess. 150+ unique Jenkins pipelines, each team copying and modifying Jenkinsfiles like they were sharing recipes at a potluck. Every pipeline was a snowflake—beautifully unique, impossibly fragile, and melting under the slightest pressure.

The Cost of Chaos:

• Average time to create new pipeline: 2-3 days (copy, modify, debug, repeat)
• Pipeline maintenance: 15+ hours/week (fixing broken builds across teams)
• Deployment failures: ~30% of builds (inconsistent configurations)
• Knowledge silos: Only 3 people understood “the old Java pipeline”
• Developer frustration: Through the roof

Six months later:

• ONE centralized shared library powering all 150+ pipelines
• New pipeline creation time: < 15 minutes
• Pipeline maintenance: < 2 hours/week
• Deployment failures: < 5%
• Any developer can create/modify pipelines
• Saved 200+ engineering hours/month

This is the complete story of how we built production-ready Jenkins infrastructure on Kubernetes and transformed 150 fragmented pipelines into a unified, maintainable CI/CD platform using Jenkins Shared Libraries.

Table of Contents

1. The Pain: 150 Pipelines of Chaos
2. Part 1: Building Production-Ready Jenkins on EKS
3. Part 2: The Shared Library Architecture
4. Part 3: Pipeline Standardization Strategy
5. Part 4: Migration: From Chaos to Consistency
6. Part 5: Results and Impact
7. Lessons Learned

The Pain: 150 Pipelines of Chaos

graph TB
    Title["<b>Before Shared Library - 150 Unique Pipelines</b>"]
    
    subgraph JavaTeam["Java Teams - 48 Pipelines"]
        J1["team-a<br/>Jenkinsfile<br/>73 lines"]
        J2["team-b<br/>Jenkinsfile<br/>85 lines"]
        J3["team-c<br/>Jenkinsfile<br/>67 lines"]
        JN["... 45 more<br/>ALL different"]
    end
    
    subgraph NodeTeam["Node.js Teams - 35 Pipelines"]
        N1["frontend-a<br/>Jenkinsfile<br/>92 lines"]
        N2["frontend-b<br/>Jenkinsfile<br/>78 lines"]
        NN["... 33 more<br/>ALL different"]
    end
    
    subgraph PythonTeam["Python Teams - 22 Pipelines"]
        P1["data-team<br/>Jenkinsfile<br/>56 lines"]
        P2["ml-team<br/>Jenkinsfile<br/>102 lines"]
        PN["... 20 more<br/>ALL different"]
    end
    
    subgraph Problems["The Problems"]
        Prob1["❌ Copy/Paste Culture<br/>No consistency"]
        Prob2["❌ Maintenance Nightmare<br/>15 hrs/week fixing"]
        Prob3["❌ 30% Failure Rate<br/>Inconsistent configs"]
        Prob4["❌ 2-3 Days<br/>To create new pipeline"]
        Prob5["❌ Knowledge Silos<br/>Only 3 people understand"]
    end
    
    Title --> JavaTeam
    Title --> NodeTeam
    Title --> PythonTeam
    
    J1 -.-> Prob1
    N1 -.-> Prob2
    P1 -.-> Prob3
    J2 -.-> Prob4
    N2 -.-> Prob5
    
    style Title fill:#ffcccc,stroke:#cc0000,stroke-width:3px,color:#000
    style JavaTeam fill:#fff5e6
    style NodeTeam fill:#fff5e6
    style PythonTeam fill:#fff5e6
    style Problems fill:#ff9999

graph TB
    subgraph After["After Shared Library - ONE Library, 142 Pipelines"]
        subgraph SharedLib["Jenkins Shared Library"]
            SL["jenkins-shared-library<br/>GitHub Repository"]
            
            Maven["mavenPipeline()<br/>Java/Maven builds"]
            Node["nodePipeline()<br/>Node.js builds"]
            Python["pythonPipeline()<br/>Python builds"]
            Docker["dockerPipeline()<br/>Docker builds"]
            Terraform["terraformPipeline()<br/>Infrastructure"]
            
            SL --> Maven
            SL --> Node
            SL --> Python
            SL --> Docker
            SL --> Terraform
        end
        
        subgraph AllTeams["All Teams Use Same Library"]
            T1["Team A<br/>Jenkinsfile<br/>5 lines"]
            T2["Team B<br/>Jenkinsfile<br/>5 lines"]
            T3["Team C<br/>Jenkinsfile<br/>5 lines"]
            TN["142 pipelines<br/>ALL use shared lib"]
        end
        
        subgraph Benefits["The Benefits"]
            B1["✅ Consistent<br/>Same patterns everywhere"]
            B2["✅ Easy Maintenance<br/>2 hrs/week total"]
            B3["✅ 95% Success Rate<br/>Standardized configs"]
            B4["✅ 15 Minutes<br/>To create new pipeline"]
            B5["✅ Self-Service<br/>Any dev can create"]
        end
    end
    
    Maven --> T1
    Node --> T2
    Python --> T3
    Docker --> TN
    
    T1 -.-> B1
    T2 -.-> B2
    T3 -.-> B3
    TN -.-> B4
    SharedLib -.-> B5
    
    style After fill:#ccffcc
    style SharedLib fill:#99ff99
    style Benefits fill:#ccffff

What We Inherited

When I joined FIS’s DevOps team, here’s what our Jenkins landscape looked like:

Pipeline Inventory (November 2021):

Total Pipelines: 152
├── Java Maven builds: 48 (32 different Jenkinsfiles)
├── Node.js builds: 35 (28 different Jenkinsfiles)
├── Python builds: 22 (19 different Jenkinsfiles)
├── Docker builds: 31 (ALL different)
├── Terraform deployments: 16 (12 different)
└── "Special" pipelines: Random chaos

The Core Problem: Every team had copied a Jenkinsfile from somewhere, modified it for their needs, and then never updated it again. Pipeline maintenance was reactive—fix only when it breaks.

A Day in the Life (Before)

Monday, 9:15 AM:

// team-a's Jenkinsfile (works)
pipeline {
    agent { label 'docker' }
    stages {
        stage('Build') {
            steps {
                sh 'mvn clean package'
            }
        }
    }
}

Monday, 2:30 PM:

// team-b's Jenkinsfile (copied from team-a, modified)
pipeline {
    agent { label 'maven' }  // Different agent!
    stages {
        stage('Build & Test') {  // Combined stages
            steps {
                sh 'mvn clean verify'  // Different Maven goal
                sh 'docker build .'    // Added Docker
            }
        }
    }
}

Tuesday, 10:00 AM:

// team-c's Jenkinsfile (copied from team-b, more modifications)
pipeline {
    agent { label 'java11' }  // Yet another agent!
    stages {
        stage('Checkout') {  // Added explicit checkout
            steps {
                git 'https://github.com/...'
            }
        }
        stage('Build') {
            steps {
                sh 'mvn clean install'  // Different goal again!
                sh 'docker build -t myapp:${BUILD_NUMBER} .'
                sh 'docker push myapp:${BUILD_NUMBER}'
            }
        }
    }
}

Wednesday: All three teams’ builds break because we updated the Java agent image. Now we have to fix 150 pipelines.

The Tipping Point

The Incident: January 2022

We needed to add SonarQube scanning to all pipelines for security compliance. The estimate? 3 weeks to modify 150 pipelines manually.

The Executive Decision:

“This is unsustainable. I don’t care how you do it, but we need ONE way to build and deploy. Make it happen.”

Challenge accepted.

Part 1: Building Production-Ready Jenkins on EKS

Before we could centralize pipelines, we needed rock-solid Jenkins infrastructure.

graph TB
    subgraph Internet["External Access"]
        Users["👨‍💻 Developers<br/>Engineers"]
    end
    
    subgraph DNS["DNS Layer"]
        Route53["Route 53<br/>jenkins-prod.fis.com"]
        ACM["AWS Certificate Manager<br/>TLS Certificate"]
    end
    
    subgraph Ingress["Ingress Layer"]
        NGINX["NGINX Ingress<br/>TLS Termination<br/>Load Balancing"]
    end
    
    subgraph EKS["Amazon EKS Cluster"]
        subgraph Controllers["Jenkins Controllers - StatefulSet"]
            C1["Controller 1<br/>AZ-A<br/>4GB RAM"]
            C2["Controller 2<br/>AZ-B<br/>4GB RAM"]
            C3["Controller 3<br/>AZ-C<br/>4GB RAM"]
            C4["Controller 4<br/>AZ-A<br/>4GB RAM"]
        end
        
        subgraph Storage["Shared Storage"]
            EFS["Amazon EFS<br/>JENKINS_HOME<br/>Encrypted<br/>Multi-AZ"]
        end
        
        subgraph Agents["Dynamic Kubernetes Agents"]
            AgentMaven["Maven Agent<br/>On-demand Pod<br/>Java 17 + Maven 3.9"]
            AgentNode["Node.js Agent<br/>On-demand Pod<br/>Node 18"]
            AgentPython["Python Agent<br/>On-demand Pod<br/>Python 3.11"]
            AgentDocker["Docker Agent<br/>On-demand Pod<br/>Docker-in-Docker"]
        end
        
        subgraph Config["Configuration"]
            JCasC["Jenkins CasC<br/>Configuration as Code<br/>Shared Library config"]
        end
    end
    
    subgraph Monitoring["Monitoring & Backup"]
        Prometheus["Prometheus<br/>Metrics Collection"]
        Grafana["Grafana<br/>Dashboards"]
        Backup["AWS Backup<br/>Daily EFS Snapshots<br/>S3 Configuration Backup"]
    end
    
    Users --> Route53
    Route53 --> ACM
    ACM --> NGINX
    NGINX --> C1
    NGINX --> C2
    NGINX --> C3
    NGINX --> C4
    
    C1 --> EFS
    C2 --> EFS
    C3 --> EFS
    C4 --> EFS
    
    C1 -.->|Spawn| AgentMaven
    C2 -.->|Spawn| AgentNode
    C3 -.->|Spawn| AgentPython
    C4 -.->|Spawn| AgentDocker
    
    JCasC -.->|Configure| C1
    JCasC -.->|Configure| C2
    JCasC -.->|Configure| C3
    JCasC -.->|Configure| C4
    
    C1 --> Prometheus
    C2 --> Prometheus
    Prometheus --> Grafana
    
    EFS --> Backup
    
    style EKS fill:#e6f3ff
    style Controllers fill:#ffcc99
    style Agents fill:#ccffcc
    style Storage fill:#ffccff
    style Monitoring fill:#ffffcc

The Old Jenkins Setup (Don’t Do This)

Jenkins Master: EC2 t3.xlarge
- Single point of failure
- Manual backups (when we remembered)
- Agents: 10 EC2 instances running 24/7
- Cost: $3,200/month
- Scalability: LOL

Problems:

• Jenkins went down? All builds stop
• Need more capacity? Provision EC2 instance (30 minutes)
• Upgrades? Hope and pray
• Backups? “We should really do that…”

The New Architecture: Jenkins on EKS

We designed a production-ready Jenkins platform on Kubernetes:

┌─────────────────────────────────────────────┐
│          Production Jenkins on EKS          │
├─────────────────────────────────────────────┤
│                                             │
│  ┌──────────────────────────────────────┐   │
│  │   Jenkins Controllers (StatefulSet)  │   │
│  │   - 4 replicas across 3 AZs          │   │
│  │   - Anti-affinity rules              │   │
│  │   - Configuration as Code (JCasC)    │   │
│  └──────────────────────────────────────┘   │
│               │                             │
│               ↓                             │
│  ┌──────────────────────────────────────┐   │
│  │   Amazon EFS (Shared Storage)        │   │
│  │   - JENKINS_HOME persistence         │   │
│  │   - Multi-AZ, encrypted              │   │
│  │   - Automated backups                │   │
│  └──────────────────────────────────────┘   │
│               │                             │
│               ↓                             │
│  ┌──────────────────────────────────────┐   │
│  │   Dynamic Kubernetes Agents          │   │
│  │   - On-demand pods                   │   │
│  │   - Language-specific labels         │   │
│  │   - Auto-scaling                     │   │
│  └──────────────────────────────────────┘   │
│               │                             │
│               ↓                             │
│  ┌──────────────────────────────────────┐   │
│  │   NGINX Ingress + TLS                │   │
│  │   - jenkins-prod.example.com         │   │
│  │   - ACM certificate                  │   │
│  └──────────────────────────────────────┘   │
└─────────────────────────────────────────────┘

Step 1: EFS for Jenkins Persistence

Why EFS? JENKINS_HOME needs to be shared across multiple controller replicas.

# efs-volume-mount.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-efs-pv
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-0a1b2c3d4e5f6g7h8  # Your EFS ID
    volumeAttributes:
      encryptionInTransit: "true"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-efs-pvc
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 100Gi

Apply:

kubectl create namespace jenkins
kubectl apply -f efs-volume-mount.yaml

Step 2: Jenkins Helm Values (Production-Ready)

# values.yaml - Production configuration
controller:
  # High availability - 4 replicas
  replicaCount: 4
  
  # Jenkins version
  image:
    tag: "2.452.2-lts"
  
  # Use EFS for JENKINS_HOME
  persistence:
    existingClaim: jenkins-efs-pvc
    storageClass: efs-sc
  
  # Resource requests/limits
  resources:
    requests:
      cpu: "2"
      memory: "4Gi"
    limits:
      cpu: "4"
      memory: "8Gi"
  
  # JVM tuning
  javaOpts: "-Xms2g -Xmx4g -XX:+UseG1GC"
  
  # Configuration as Code (JCasC)
  JCasC:
    enabled: true
    defaultConfig: true
    configScripts:
      welcome-message: |
        jenkins:
          systemMessage: "Production Jenkins - Managed by Platform Team"
      
      # Shared library configuration
      global-libraries: |
        unclassified:
          globalLibraries:
            libraries:
              - name: "jenkins-shared-library"
                retriever:
                  modernSCM:
                    scm:
                      git:
                        remote: "https://github.com/fis-devops/jenkins-shared-library.git"
                        credentialsId: "github-token"
                defaultVersion: "main"
                implicit: true
                allowVersionOverride: true
      
      # Kubernetes cloud configuration
      kubernetes-agent: |
        jenkins:
          clouds:
            - kubernetes:
                name: "kubernetes"
                serverUrl: "https://kubernetes.default"
                namespace: "jenkins"
                jenkinsUrl: "http://jenkins:8080"
                jenkinsTunnel: "jenkins-agent:50000"
                containerCapStr: "100"
                maxRequestsPerHostStr: "32"
                podRetention: "onFailure"
                templates:
                  # Maven agent template
                  - name: "maven"
                    label: "maven"
                    nodeUsageMode: NORMAL
                    containers:
                      - name: "maven"
                        image: "maven:3.9-eclipse-temurin-17"
                        command: "/bin/sh -c"
                        args: "cat"
                        ttyEnabled: true
                        resourceRequestCpu: "1"
                        resourceRequestMemory: "2Gi"
                        resourceLimitCpu: "2"
                        resourceLimitMemory: "4Gi"
                  
                  # Node.js agent template
                  - name: "nodejs"
                    label: "nodejs"
                    nodeUsageMode: NORMAL
                    containers:
                      - name: "nodejs"
                        image: "node:18-alpine"
                        command: "/bin/sh -c"
                        args: "cat"
                        ttyEnabled: true
                        resourceRequestCpu: "500m"
                        resourceRequestMemory: "1Gi"
                        resourceLimitCpu: "1"
                        resourceLimitMemory: "2Gi"
                  
                  # Python agent template
                  - name: "python"
                    label: "python"
                    nodeUsageMode: NORMAL
                    containers:
                      - name: "python"
                        image: "python:3.11-slim"
                        command: "/bin/sh -c"
                        args: "cat"
                        ttyEnabled: true
                        resourceRequestCpu: "500m"
                        resourceRequestMemory: "1Gi"
                        resourceLimitCpu: "1"
                        resourceLimitMemory: "2Gi"
                  
                  # Docker agent template
                  - name: "docker"
                    label: "docker"
                    nodeUsageMode: NORMAL
                    containers:
                      - name: "docker"
                        image: "docker:24-dind"
                        privileged: true
                        command: "/bin/sh -c"
                        args: "cat"
                        ttyEnabled: true
                        resourceRequestCpu: "1"
                        resourceRequestMemory: "2Gi"
                        resourceLimitCpu: "2"
                        resourceLimitMemory: "4Gi"
  
  # Essential plugins
  installPlugins:
    - kubernetes:4029.v5712230ccb_f8
    - workflow-aggregator:596.v8c21c963d92d
    - git:5.2.0
    - configuration-as-code:1810.v9b_c30a_249a_4c
    - blueocean:1.27.9
    - pipeline-stage-view:2.34
    - docker-workflow:572.v950f58993843
    - sonar:2.17.2
    - slack:664.vc9a_90f8b_c24a_
    - prometheus:2.3.2
    - job-dsl:1.87
    - credentials-binding:681.vf91669a_32e45
    - pipeline-utility-steps:2.16.2
    - http_request:1.18
    - timestamper:1.26
    - ws-cleanup:0.45
    - ansicolor:1.0.4
  
  # Security settings
  securityRealm: |-
    local:
      allowsSignup: false
  authorizationStrategy: |-
    loggedInUsersCanDoAnything:
      allowAnonymousRead: false
  
  # Prometheus metrics
  prometheus:
    enabled: true
    serviceMonitorNamespace: monitoring
  
  # Pod anti-affinity for HA
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - jenkins-controller
          topologyKey: kubernetes.io/hostname

# Agent settings
agent:
  enabled: true
  defaultsProviderTemplate: "maven"
  
# Horizontal Pod Autoscaler
autoscaling:
  enabled: true
  minReplicas: 4
  maxReplicas: 8
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80

# Backup configuration
backup:
  enabled: true
  schedule: "H 2 * * *"  # Daily at 2 AM
  destination: s3://fis-jenkins-backups/

# Monitoring
serviceMonitor:
  enabled: true
  namespace: monitoring
  interval: 30s

Step 3: Deploy Jenkins

# Add Jenkins Helm repo
helm repo add jenkinsci https://charts.jenkins.io
helm repo update

# Deploy Jenkins
helm upgrade --install jenkins jenkinsci/jenkins \
  --namespace jenkins \
  --values values.yaml \
  --timeout 10m \
  --wait

# Watch pods come up
kubectl get pods -n jenkins -w

Step 4: NGINX Ingress with TLS

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: jenkins
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - jenkins-prod.example.com
      secretName: jenkins-tls-secret
  rules:
    - host: jenkins-prod.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: jenkins
                port:
                  number: 8080

kubectl apply -f ingress.yaml

Step 5: Access Jenkins

# Get admin password
kubectl exec -it jenkins-0 -n jenkins -- \
  cat /run/secrets/additional/chart-admin-password

# Access: https://jenkins-prod.example.com
# Login with admin user and password

The Infrastructure Results

Before (EC2-based):

• Uptime: 99.5% (manual restarts needed)
• Scale-up time: 30 minutes
• Cost: $3,200/month
• Maintenance: 8 hours/week

After (EKS-based):

• Uptime: 99.95% (self-healing)
• Scale-up time: < 2 minutes (auto-scaling)
• Cost: $2,100/month
• Maintenance: < 2 hours/week

Savings: $1,100/month + 75% less maintenance

Now we had a solid foundation. Time to tackle the pipeline chaos.

Part 2: The Shared Library Architecture

What is a Jenkins Shared Library?

graph LR
    subgraph Repo["jenkins-shared-library Repository"]
        subgraph Vars["vars/ - Pipeline Templates"]
            V1["mavenPipeline.groovy<br/>Java/Maven builds"]
            V2["nodePipeline.groovy<br/>Node.js builds"]
            V3["pythonPipeline.groovy<br/>Python builds"]
            V4["dockerPipeline.groovy<br/>Generic Docker"]
            V5["terraformPipeline.groovy<br/>Infrastructure"]
        end
        
        subgraph Functions["vars/ - Helper Functions"]
            F1["buildDocker.groovy<br/>Docker build"]
            F2["pushDocker.groovy<br/>Docker push"]
            F3["deployK8s.groovy<br/>K8s deployment"]
            F4["runSonarQube.groovy<br/>Code quality"]
            F5["sendSlackNotification.groovy<br/>Slack alerts"]
            F6["scanImage.groovy<br/>Security scan"]
        end
        
        subgraph Src["src/ - Classes"]
            S1["Docker.groovy<br/>Docker helper class"]
            S2["Kubernetes.groovy<br/>K8s helper class"]
            S3["Notifications.groovy<br/>Notification helper"]
            S4["Utils.groovy<br/>Common utilities"]
        end
        
        subgraph Resources["resources/ - Templates"]
            R1["kubernetes/<br/>deployment.yaml<br/>service.yaml"]
            R2["sonar/<br/>sonar-project.properties"]
        end
    end
    
    subgraph Teams["Development Teams"]
        Team1["Team A<br/>Jenkinsfile"]
        Team2["Team B<br/>Jenkinsfile"]
        Team3["Team C<br/>Jenkinsfile"]
    end
    
    V1 --> F1
    V1 --> F2
    V1 --> F3
    V1 --> F4
    V1 --> F5
    
    F1 --> S1
    F3 --> S2
    F5 --> S3
    
    Team1 --> V1
    Team2 --> V2
    Team3 --> V3
    
    style Repo fill:#e6f3ff
    style Vars fill:#ffcc99
    style Functions fill:#ccffcc
    style Src fill:#ffccff
    style Teams fill:#ffffcc

Think of it as a npm package for Jenkins pipelines. Instead of copying code, you import reusable functions.

Before (Copy/Paste):

// Every team's Jenkinsfile
pipeline {
    agent { label 'maven' }
    stages {
        stage('Build') {
            steps {
                sh 'mvn clean package'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
        }
        stage('Docker Build') {
            steps {
                sh 'docker build -t myapp:${BUILD_NUMBER} .'
            }
        }
        stage('Docker Push') {
            steps {
                sh 'docker push myapp:${BUILD_NUMBER}'
            }
        }
        stage('Deploy') {
            steps {
                sh 'kubectl apply -f k8s/'
            }
        }
    }
}

After (Shared Library):

// Every team's Jenkinsfile
@Library('jenkins-shared-library') _

mavenPipeline {
    dockerImage = 'myapp'
    deployEnvironment = 'production'
}

Same functionality. 3 lines instead of 30.

Our Shared Library Structure

jenkins-shared-library/
├── vars/
│   ├── mavenPipeline.groovy       # Maven build pipeline
│   ├── nodePipeline.groovy        # Node.js build pipeline
│   ├── pythonPipeline.groovy      # Python build pipeline
│   ├── dockerPipeline.groovy      # Generic Docker build
│   ├── terraformPipeline.groovy   # Infrastructure deployment
│   ├── helmPipeline.groovy        # Kubernetes Helm charts
│   ├── buildDocker.groovy         # Docker build function
│   ├── pushDocker.groovy          # Docker push function
│   ├── deployK8s.groovy           # Kubernetes deployment
│   ├── runSonarQube.groovy        # Code quality scan
│   ├── runTests.groovy            # Test execution
│   ├── sendSlackNotification.groovy  # Slack integration
│   └── scanImage.groovy           # Container security scan
├── src/
│   └── com/
│       └── fis/
│           └── jenkins/
│               ├── Docker.groovy         # Docker helper class
│               ├── Kubernetes.groovy     # K8s helper class
│               ├── Notifications.groovy  # Notification helper
│               └── Utils.groovy          # Common utilities
├── resources/
│   ├── kubernetes/
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   └── sonar/
│       └── sonar-project.properties
└── README.md

The Maven Pipeline Template

// vars/mavenPipeline.groovy
def call(Map config = [:]) {
    // Default configuration
    def defaults = [
        javaVersion: '17',
        mavenVersion: '3.9',
        dockerImage: '',
        dockerRegistry: 'docker.example.com',
        sonarQubeEnabled: true,
        deployEnvironment: 'staging',
        slackChannel: '#deployments',
        timeoutMinutes: 30
    ]
    
    // Merge user config with defaults
    config = defaults + config
    
    pipeline {
        agent {
            kubernetes {
                label "maven-${UUID.randomUUID().toString()}"
                yaml """
apiVersion: v1
kind: Pod
metadata:
  labels:
    jenkins: agent
spec:
  containers:
  - name: maven
    image: maven:${config.mavenVersion}-eclipse-temurin-${config.javaVersion}
    command:
    - cat
    tty: true
    resources:
      requests:
        memory: "2Gi"
        cpu: "1"
      limits:
        memory: "4Gi"
        cpu: "2"
  - name: docker
    image: docker:24-dind
    command:
    - cat
    tty: true
    privileged: true
    volumeMounts:
    - name: docker-sock
      mountPath: /var/run/docker.sock
  volumes:
  - name: docker-sock
    hostPath:
      path: /var/run/docker.sock
"""
            }
        }
        
        options {
            buildDiscarder(logRotator(numToKeepStr: '10'))
            timestamps()
            timeout(time: config.timeoutMinutes, unit: 'MINUTES')
            ansiColor('xterm')
        }
        
        environment {
            DOCKER_REGISTRY = config.dockerRegistry
            DOCKER_IMAGE = config.dockerImage
            DEPLOY_ENV = config.deployEnvironment
        }
        
        stages {
            stage('Checkout') {
                steps {
                    checkout scm
                    script {
                        env.GIT_COMMIT_SHORT = sh(
                            script: "git rev-parse --short HEAD",
                            returnStdout: true
                        ).trim()
                        env.IMAGE_TAG = "${env.GIT_COMMIT_SHORT}-${env.BUILD_NUMBER}"
                    }
                }
            }
            
            stage('Build') {
                steps {
                    container('maven') {
                        sh """
                            mvn clean package \
                                -DskipTests \
                                -Dmaven.test.skip=true \
                                -B -V
                        """
                    }
                }
            }
            
            stage('Test') {
                steps {
                    container('maven') {
                        sh 'mvn test -B'
                    }
                }
                post {
                    always {
                        junit '**/target/surefire-reports/*.xml'
                    }
                }
            }
            
            stage('SonarQube Analysis') {
                when {
                    expression { config.sonarQubeEnabled }
                }
                steps {
                    container('maven') {
                        script {
                            runSonarQube()
                        }
                    }
                }
            }
            
            stage('Docker Build') {
                when {
                    expression { config.dockerImage != '' }
                }
                steps {
                    container('docker') {
                        script {
                            buildDocker(
                                image: "${config.dockerRegistry}/${config.dockerImage}",
                                tag: env.IMAGE_TAG
                            )
                        }
                    }
                }
            }
            
            stage('Security Scan') {
                when {
                    expression { config.dockerImage != '' }
                }
                steps {
                    container('docker') {
                        script {
                            scanImage(
                                image: "${config.dockerRegistry}/${config.dockerImage}:${env.IMAGE_TAG}"
                            )
                        }
                    }
                }
            }
            
            stage('Docker Push') {
                when {
                    expression { config.dockerImage != '' }
                }
                steps {
                    container('docker') {
                        script {
                            pushDocker(
                                image: "${config.dockerRegistry}/${config.dockerImage}",
                                tag: env.IMAGE_TAG
                            )
                        }
                    }
                }
            }
            
            stage('Deploy') {
                when {
                    branch 'main'
                }
                steps {
                    script {
                        deployK8s(
                            environment: config.deployEnvironment,
                            image: "${config.dockerRegistry}/${config.dockerImage}:${env.IMAGE_TAG}"
                        )
                    }
                }
            }
        }
        
        post {
            success {
                script {
                    sendSlackNotification(
                        channel: config.slackChannel,
                        message: "✅ Build SUCCESS: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                        color: 'good'
                    )
                }
            }
            failure {
                script {
                    sendSlackNotification(
                        channel: config.slackChannel,
                        message: "❌ Build FAILED: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                        color: 'danger'
                    )
                }
            }
            always {
                cleanWs()
            }
        }
    }
}

Reusable Helper Functions

// vars/buildDocker.groovy
def call(Map config = [:]) {
    echo "Building Docker image: ${config.image}:${config.tag}"
    
    sh """
        docker build \
            --tag ${config.image}:${config.tag} \
            --tag ${config.image}:latest \
            --build-arg BUILD_DATE=\$(date -u +'%Y-%m-%dT%H:%M:%SZ') \
            --build-arg VCS_REF=${env.GIT_COMMIT_SHORT} \
            --build-arg VERSION=${config.tag} \
            .
    """
}

// vars/pushDocker.groovy
def call(Map config = [:]) {
    echo "Pushing Docker image: ${config.image}:${config.tag}"
    
    withCredentials([usernamePassword(
        credentialsId: 'docker-registry-credentials',
        usernameVariable: 'DOCKER_USER',
        passwordVariable: 'DOCKER_PASS'
    )]) {
        sh """
            echo \$DOCKER_PASS | docker login ${config.registry} -u \$DOCKER_USER --password-stdin
            docker push ${config.image}:${config.tag}
            docker push ${config.image}:latest
        """
    }
}

// vars/deployK8s.groovy
def call(Map config = [:]) {
    echo "Deploying to ${config.environment}"
    
    withCredentials([file(credentialsId: "kubeconfig-${config.environment}", variable: 'KUBECONFIG')]) {
        sh """
            kubectl set image deployment/${config.deploymentName} \
                ${config.containerName}=${config.image} \
                --namespace=${config.namespace}
            
            kubectl rollout status deployment/${config.deploymentName} \
                --namespace=${config.namespace} \
                --timeout=5m
        """
    }
}

// vars/runSonarQube.groovy
def call(Map config = [:]) {
    withSonarQubeEnv('SonarQube') {
        sh """
            mvn sonar:sonar \
                -Dsonar.projectKey=${env.JOB_NAME} \
                -Dsonar.projectName='${env.JOB_NAME}' \
                -Dsonar.projectVersion=${env.BUILD_NUMBER}
        """
    }
    
    // Wait for quality gate
    timeout(time: 10, unit: 'MINUTES') {
        def qg = waitForQualityGate()
        if (qg.status != 'OK') {
            error "Pipeline aborted due to quality gate failure: ${qg.status}"
        }
    }
}

// vars/sendSlackNotification.groovy
def call(Map config = [:]) {
    slackSend(
        channel: config.channel,
        message: config.message,
        color: config.color,
        botUser: true,
        teamDomain: 'fis-engineering',
        tokenCredentialId: 'slack-token'
    )
}

// vars/scanImage.groovy
def call(Map config = [:]) {
    echo "Scanning image for vulnerabilities: ${config.image}"
    
    sh """
        # Using Trivy for container scanning
        docker run --rm \
            -v /var/run/docker.sock:/var/run/docker.sock \
            aquasec/trivy:latest image \
            --severity HIGH,CRITICAL \
            --exit-code 1 \
            ${config.image}
    """
}

Node.js Pipeline Template

// vars/nodePipeline.groovy
def call(Map config = [:]) {
    def defaults = [
        nodeVersion: '18',
        packageManager: 'npm',  // or 'yarn'
        dockerImage: '',
        dockerRegistry: 'docker.example.com',
        deployEnvironment: 'staging',
        slackChannel: '#deployments',
        runLint: true,
        runTests: true
    ]
    
    config = defaults + config
    
    pipeline {
        agent {
            kubernetes {
                label "nodejs-${UUID.randomUUID().toString()}"
                yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: nodejs
    image: node:${config.nodeVersion}-alpine
    command:
    - cat
    tty: true
  - name: docker
    image: docker:24-dind
    command:
    - cat
    tty: true
    privileged: true
"""
            }
        }
        
        stages {
            stage('Checkout') {
                steps {
                    checkout scm
                }
            }
            
            stage('Install Dependencies') {
                steps {
                    container('nodejs') {
                        script {
                            if (config.packageManager == 'yarn') {
                                sh 'yarn install --frozen-lockfile'
                            } else {
                                sh 'npm ci'
                            }
                        }
                    }
                }
            }
            
            stage('Lint') {
                when {
                    expression { config.runLint }
                }
                steps {
                    container('nodejs') {
                        sh "${config.packageManager} run lint"
                    }
                }
            }
            
            stage('Test') {
                when {
                    expression { config.runTests }
                }
                steps {
                    container('nodejs') {
                        sh "${config.packageManager} run test"
                    }
                }
            }
            
            stage('Build') {
                steps {
                    container('nodejs') {
                        sh "${config.packageManager} run build"
                    }
                }
            }
            
            stage('Docker Build & Push') {
                when {
                    expression { config.dockerImage != '' }
                }
                steps {
                    container('docker') {
                        script {
                            def imageTag = "${env.GIT_COMMIT_SHORT}-${env.BUILD_NUMBER}"
                            buildDocker(
                                image: "${config.dockerRegistry}/${config.dockerImage}",
                                tag: imageTag
                            )
                            pushDocker(
                                image: "${config.dockerRegistry}/${config.dockerImage}",
                                tag: imageTag,
                                registry: config.dockerRegistry
                            )
                        }
                    }
                }
            }
            
            stage('Deploy') {
                when {
                    branch 'main'
                }
                steps {
                    script {
                        deployK8s(
                            environment: config.deployEnvironment,
                            image: "${config.dockerRegistry}/${config.dockerImage}:${env.GIT_COMMIT_SHORT}-${env.BUILD_NUMBER}"
                        )
                    }
                }
            }
        }
        
        post {
            success {
                script {
                    sendSlackNotification(
                        channel: config.slackChannel,
                        message: "✅ Build SUCCESS: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                        color: 'good'
                    )
                }
            }
            failure {
                script {
                    sendSlackNotification(
                        channel: config.slackChannel,
                        message: "❌ Build FAILED: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                        color: 'danger'
                    )
                }
            }
        }
    }
}

Part 3: Pipeline Standardization Strategy

The Migration Philosophy

graph TB
    Title["<b>Pipeline Migration: Before vs After</b>"]
    
    subgraph Before["❌ BEFORE: Custom Pipeline Chaos"]
        B_Header["<b>Traditional Jenkinsfile</b><br/>73 Lines of Code"]
        
        B_Stage1["<b>Stage 1:</b> Checkout<br/>checkout scm"]
        B_Stage2["<b>Stage 2:</b> Build<br/>sh 'mvn clean package'"]
        B_Stage3["<b>Stage 3:</b> Test<br/>sh 'mvn test'<br/>junit '**/*.xml'"]
        B_Stage4["<b>Stage 4:</b> SonarQube<br/>withSonarQubeEnv(...)<br/>sh 'mvn sonar:sonar'"]
        B_Stage5["<b>Stage 5:</b> Docker Build<br/>sh 'docker build -t app .'"]
        B_Stage6["<b>Stage 6:</b> Docker Push<br/>withCredentials(...)<br/>sh 'docker push'"]
        B_Stage7["<b>Stage 7:</b> Deploy<br/>sh 'kubectl apply -f k8s/'"]
        
        B_Header --> B_Stage1
        B_Stage1 --> B_Stage2
        B_Stage2 --> B_Stage3
        B_Stage3 --> B_Stage4
        B_Stage4 --> B_Stage5
        B_Stage5 --> B_Stage6
        B_Stage6 --> B_Stage7
    end
    
    subgraph After["✅ AFTER: Shared Library Simplicity"]
        A_Header["<b>Shared Library Jenkinsfile</b><br/>5 Lines of Code"]
        
        A_Code["@Library('jenkins-shared-library') _<br/><br/>mavenPipeline {<br/>  dockerImage = 'payment-api'<br/>  deployEnvironment = 'production'<br/>}"]
        
        A_Benefits["<b>Automatic Features:</b><br/>✓ All 7 stages automated<br/>✓ Security scanning included<br/>✓ Slack notifications<br/>✓ Error handling<br/>✓ Best practices enforced"]
        
        A_Header --> A_Code
        A_Code --> A_Benefits
    end
    
    subgraph Metrics["📊 Impact Metrics"]
        M1["<b>Code Reduction</b><br/>73 → 5 lines<br/><b>93% less code</b>"]
        M2["<b>Maintenance</b><br/>Per-pipeline → Centralized<br/><b>87% time saved</b>"]
        M3["<b>Consistency</b><br/>Copy/paste → Standardized<br/><b>Zero drift</b>"]
        M4["<b>Quality</b><br/>Manual → Automated<br/><b>25% fewer failures</b>"]
    end
    
    Title --> Before
    Title --> After
    
    Before -.->|Replaced by| After
    
    Before --> Metrics
    After --> Metrics
    
    Metrics --> M1
    Metrics --> M2
    Metrics --> M3
    Metrics --> M4
    
    style Title fill:#2c3e50,stroke:#34495e,stroke-width:3px,color:#ffffff
    style Before fill:#ffe6e6,stroke:#e74c3c,stroke-width:2px
    style After fill:#e6ffe6,stroke:#27ae60,stroke-width:2px
    style Metrics fill:#e6f3ff,stroke:#3498db,stroke-width:2px
    
    style B_Header fill:#ffcccc,stroke:#c0392b,stroke-width:2px
    style A_Header fill:#ccffcc,stroke:#229954,stroke-width:2px
    
    style B_Stage1 fill:#fff5f5
    style B_Stage2 fill:#fff5f5
    style B_Stage3 fill:#fff5f5
    style B_Stage4 fill:#fff5f5
    style B_Stage5 fill:#fff5f5
    style B_Stage6 fill:#fff5f5
    style B_Stage7 fill:#fff5f5
    
    style A_Code fill:#f0fff0
    style A_Benefits fill:#f0fff0
    
    style M1 fill:#d6eaf8,stroke:#2980b9,stroke-width:1px
    style M2 fill:#d6eaf8,stroke:#2980b9,stroke-width:1px
    style M3 fill:#d6eaf8,stroke:#2980b9,stroke-width:1px
    style M4 fill:#d6eaf8,stroke:#2980b9,stroke-width:1px

We couldn’t migrate 150 pipelines overnight. We needed a strategy:

Principle 1: Make it Easy

• Shared library pipelines should be simpler than custom ones
• Default configurations for 80% use cases
• Override anything when needed

Principle 2: Incremental Migration

• Start with new projects
• Migrate high-value pipelines first
• Let low-priority pipelines migrate organically

Principle 3: Don’t Break Existing Pipelines

• Shared library is opt-in, not forced
• Old pipelines continue working
• Migration is developer-driven

Pipeline Categories

We identified 5 main pipeline patterns:

1. Java/Maven Applications (48 pipelines)
   → mavenPipeline

2. Node.js Applications (35 pipelines)
   → nodePipeline

3. Python Applications (22 pipelines)
   → pythonPipeline

4. Docker-only Builds (31 pipelines)
   → dockerPipeline

5. Terraform Infrastructure (16 pipelines)
   → terraformPipeline

Migration Examples

Before - Custom Maven Pipeline (45 lines):

pipeline {
    agent { label 'maven' }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        
        stage('Build') {
            steps {
                sh 'mvn clean package -DskipTests'
            }
        }
        
        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    junit '**/target/surefire-reports/*.xml'
                }
            }
        }
        
        stage('SonarQube') {
            steps {
                withSonarQubeEnv('SonarQube') {
                    sh 'mvn sonar:sonar'
                }
            }
        }
        
        stage('Docker Build') {
            steps {
                sh 'docker build -t payment-api:${BUILD_NUMBER} .'
                sh 'docker tag payment-api:${BUILD_NUMBER} docker.example.com/payment-api:${BUILD_NUMBER}'
            }
        }
        
        stage('Docker Push') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'docker-creds', usernameVariable: 'USER', passwordVariable: 'PASS')]) {
                    sh 'echo $PASS | docker login docker.example.com -u $USER --password-stdin'
                    sh 'docker push docker.example.com/payment-api:${BUILD_NUMBER}'
                }
            }
        }
    }
}

After - Shared Library (5 lines):

@Library('jenkins-shared-library') _

mavenPipeline {
    dockerImage = 'payment-api'
    deployEnvironment = 'production'
}

90% less code. Same functionality. Plus:

• SonarQube scanning
• Security image scanning
• Slack notifications
• Kubernetes deployment
• Error handling
• All maintained centrally

Advanced Configurations

Override default behavior:

@Library('jenkins-shared-library') _

mavenPipeline {
    // Custom configuration
    javaVersion = '11'          // Use Java 11 instead of 17
    mavenVersion = '3.8'        // Specific Maven version
    dockerImage = 'legacy-app'
    deployEnvironment = 'staging'
    sonarQubeEnabled = false    // Skip SonarQube
    slackChannel = '#team-backend'
    timeoutMinutes = 45         // Longer timeout
    
    // Custom Maven goals
    buildCommand = 'mvn clean install -P production'
    
    // Custom Docker build args
    dockerBuildArgs = '--build-arg ENV=prod'
}

Multi-stage deployments:

@Library('jenkins-shared-library') _

mavenPipeline {
    dockerImage = 'api-server'
    
    // Deploy to staging first
    deployEnvironment = 'staging'
    
    // After successful staging deploy, prompt for production
    postDeploy = {
        input(
            message: 'Deploy to production?',
            ok: 'Deploy',
            submitter: 'platform-team'
        )
        
        deployK8s(
            environment: 'production',
            image: "${env.DOCKER_REGISTRY}/${env.DOCKER_IMAGE}:${env.IMAGE_TAG}"
        )
    }
}

Part 4: Migration: From Chaos to Consistency

gantt
    title 6-Month Jenkins Shared Library Migration
    dateFormat  YYYY-MM-DD
    
    section Infrastructure
    Jenkins on EKS Setup          :done, infra1, 2021-11-01, 3w
    EFS Configuration             :done, infra2, 2021-11-08, 1w
    JCasC Implementation          :done, infra3, 2021-11-15, 1w
    Dynamic Agents Setup          :done, infra4, 2021-11-22, 1w
    
    section Shared Library Dev
    Initial Library Structure     :done, lib1, 2021-12-01, 1w
    Maven Pipeline Template       :done, lib2, 2021-12-08, 1w
    Node.js Pipeline Template     :done, lib3, 2021-12-15, 1w
    Python Pipeline Template      :done, lib4, 2021-12-22, 1w
    Helper Functions              :done, lib5, 2021-12-29, 2w
    
    section Phase 1: Pilot
    5 Team Pilot Program          :done, pilot1, 2022-01-10, 3w
    Bug Fixes & Iteration         :done, pilot2, 2022-01-31, 1w
    
    section Phase 2: Early Adopters
    New Projects Mandate          :done, early1, 2022-02-07, 4w
    40 New Pipelines Created      :done, early2, 2022-02-07, 6w
    15 Voluntary Migrations       :done, early3, 2022-02-21, 4w
    
    section Phase 3: Mass Migration
    Migration Campaign Launch     :done, mass1, 2022-03-07, 1w
    Team Workshops & Training     :done, mass2, 2022-03-14, 4w
    Migration Tool Development    :done, mass3, 2022-03-21, 2w
    100+ Pipeline Migrations      :done, mass4, 2022-04-04, 8w
    
    section Results
    142/152 Pipelines Migrated    :milestone, result, 2022-05-30, 0d
    93% Adoption Rate             :milestone, result2, 2022-05-30, 0d

Phase 1: Pilot Program (Month 1)

Selected 5 teams for pilot:

1. Backend API team (Java)
2. Frontend team (Node.js)
3. Data team (Python)
4. DevOps team (Terraform)
5. ML team (Docker-only)

Results:

• Average migration time: 30 minutes per pipeline
• Bugs found in shared library: 12 (fixed quickly)
• Teams’ feedback: 9/10 satisfaction
• Key insight: Slack notifications were the most appreciated feature

Phase 2: Early Adopters (Month 2-3)

Strategy: “New projects must use shared library.”

Results:

• 40 new pipelines created using shared library
• 15 existing pipelines migrated voluntarily
• Common pattern emerged: Teams loved the simplicity

Example migration Pull Request:

# Jenkinsfile (before: 73 lines)
- pipeline {
-     agent { label 'nodejs' }
-     stages {
-         stage('Build') {
-             steps {
-                 sh 'npm install'
-                 sh 'npm run build'
-             }
-         }
-         stage('Test') {
-             steps {
-                 sh 'npm test'
-             }
-         }
-         ...
-     }
- }

# Jenkinsfile (after: 5 lines)
+ @Library('jenkins-shared-library') _
+ 
+ nodePipeline {
+     dockerImage = 'web-app'
+ }

Developer comment:

“This is amazing. I just deleted 70 lines of code and it does MORE than before.”

Phase 3: Mass Migration (Month 4-6)

Strategy: “Carrot, not stick.”

Incentives for migration:

• Free pipeline optimization review
• Guaranteed < 5 minute support response
• Featured in team newsletter
• “Early Adopter” badge in Jenkins
• Team lunch sponsored by platform team

Migration support:

# We created a migration helper script
./scripts/migrate-pipeline.sh \
  --repo "github.com/fis/payment-api" \
  --type "maven" \
  --dry-run

# Output:
✓ Detected: Java Maven project
✓ Found Dockerfile
✓ Found Jenkinsfile (73 lines)

Recommended Jenkinsfile:
────────────────────────────────────
@Library('jenkins-shared-library') _

mavenPipeline {
    dockerImage = 'payment-api'
    javaVersion = '17'
    deployEnvironment = 'production'
}
────────────────────────────────────

Migrate now? (y/n)

Results:

• Migrated pipelines: 142/152 (93%)
• Holdouts: 10 “special snowflake” pipelines
• Migration time: 6 months total
• Average per-pipeline migration: < 1 hour

The Holdouts

10 pipelines didn’t migrate. Why?

1. Legacy mainframe integration (3 pipelines)
   • Too complex, touching mainframe systems
   • Decision: Leave alone, too risky
2. Acquisition pipelines (4 pipelines)
   • From acquired company, different standards
   • Decision: Migrate when team has bandwidth
3. ML training pipelines (2 pipelines)
   • Need GPU support, shared library didn’t support yet
   • Decision: Added GPU support, then migrated
4. “It works, don’t touch it” (1 pipeline)
   • Business-critical, zero tolerance for change
   • Decision: Respected, left alone

The pragmatic approach: Don’t force 100% adoption. 93% is excellent.

Part 5: Results and Impact

Quantitative Results

graph TB
    subgraph Metrics["Key Impact Metrics"]
        subgraph Time["Time Metrics"]
            T1["Pipeline Creation<br/>Before: 2-3 days<br/>After: 15 minutes<br/>🟢 95% faster"]
            T2["Maintenance Time<br/>Before: 15 hrs/week<br/>After: 2 hrs/week<br/>🟢 87% reduction"]
        end
        
        subgraph Quality["Quality Metrics"]
            Q1["Success Rate<br/>Before: 70%<br/>After: 95%<br/>🟢 +25 points"]
            Q2["Failure Resolution<br/>Before: 2 weeks<br/>After: 1 day<br/>🟢 90% faster"]
        end
        
        subgraph Cost["Cost Metrics"]
            C1["Infrastructure<br/>Before: $3,200/mo<br/>After: $2,100/mo<br/>🟢 $1,100 saved"]
            C2["Engineer Time<br/>Before: ~320 hrs/mo<br/>After: ~120 hrs/mo<br/>🟢 $20K value/mo"]
        end
        
        subgraph Adoption["Adoption Metrics"]
            A1["Pipelines Migrated<br/>142 out of 152<br/>🟢 93% adoption"]
            A2["Developer Satisfaction<br/>9/10 rating<br/>🟢 High satisfaction"]
        end
    end
    
    subgraph ROI["Return on Investment"]
        Investment["Investment<br/>480 engineering hours<br/>$60K loaded cost"]
        Returns["Annual Returns<br/>2400 hrs saved/yr<br/>$240K value/yr"]
        Result["ROI: 300%<br/>Payback: 3 months"]
    end
    
    Time --> Investment
    Quality --> Investment
    Cost --> Returns
    Adoption --> Returns
    
    Investment --> Result
    Returns --> Result
    
    style Time fill:#e6f3ff
    style Quality fill:#ccffcc
    style Cost fill:#ffcc99
    style Adoption fill:#ffccff
    style ROI fill:#ffffcc
    style Result fill:#99ff99

Pipeline Creation Time:

Before: 2-3 days (copy, modify, debug, test)
After:  < 15 minutes (configure, test, done)

Reduction: 95%

Pipeline Maintenance:

Before: 15+ hours/week (fixing broken pipelines)
After:  < 2 hours/week (updating shared library)

Reduction: 87%

Deployment Success Rate:

Before: 70% success (30% failure rate)
After:  95% success (5% failure rate)

Improvement: 25 percentage points

Time to Fix Pipeline Issues:

Before: Fix in 150 places (avg 2 weeks)
After:  Fix in 1 place (< 1 day)

Improvement: 90% faster

Qualitative Results

Developer Experience:

Before:

Developer: "How do I add Docker build to my pipeline?"
DevOps:    "Copy this Jenkinsfile from team-backend"
Developer: "It's not working..."
DevOps:    "Did you update line 47?"
Developer: "There is no line 47?"
DevOps:    "You copied the wrong one. Use team-frontend's"
Developer: "😭"

After:

Developer: "How do I add Docker build?"
DevOps:    "Add `dockerImage = 'your-app'` to your pipeline"
Developer: "That's it?"
DevOps:    "That's it."
Developer: "🎉"

Business Impact

Engineering Efficiency:

• Saved 200+ engineering hours/month
• Faster feature delivery (pipelines not a bottleneck)
• Reduced context switching (fewer pipeline issues)

Quality Improvements:

• Consistent security scanning (100% coverage)
• Standardized testing (no skipped tests)
• Automated compliance checks

Cost Savings:

• Reduced Jenkins agent usage: $1,100/month
• Less engineer time on pipeline maintenance: ~$20K/month value
• Faster deployments = faster time to market: Priceless

The Network Effect

What happened after adoption:

1. Teams started contributing to shared library

   Pull Request #47: Add Python pytest support
   Pull Request #52: Add Terraform workspace management
   Pull Request #61: Add Helm chart deployment
   

2. Documentation improved organically
   • Teams added examples
   • FAQ emerged from Slack discussions
   • Best practices documented by users
3. Innovation accelerated
   • New deployment strategies (blue-green, canary)
   • Advanced testing (contract tests, load tests)
   • Security scanning (SAST, DAST, container scanning)

The shared library became the platform for innovation.

sequenceDiagram
    participant Dev as Developer
    participant Git as Git Repository
    participant Jenkins as Jenkins Controller
    participant Lib as Shared Library
    participant Agent as K8s Agent Pod
    participant SonarQube as SonarQube
    participant Docker as Docker Registry
    participant K8s as Kubernetes
    participant Slack as Slack
    
    Note over Dev,Slack: Complete Pipeline Execution Flow
    
    Dev->>Git: 1. Push code + Jenkinsfile
    Note over Git: Jenkinsfile:<br/>mavenPipeline {<br/>  dockerImage = 'app'<br/>}
    
    Git->>Jenkins: 2. Webhook triggers build
    Jenkins->>Jenkins: 3. Parse Jenkinsfile
    Jenkins->>Lib: 4. Load shared library<br/>from GitHub
    
    Lib->>Jenkins: 5. Return mavenPipeline code
    Jenkins->>Agent: 6. Spawn Maven K8s pod
    Note over Agent: Pod with:<br/>- Maven 3.9<br/>- Java 17<br/>- Docker
    
    Jenkins->>Agent: 7. Checkout source code
    Agent->>Agent: 8. Build: mvn clean package
    Agent->>Agent: 9. Test: mvn test
    Agent->>Agent: 10. Generate test reports
    
    Agent->>SonarQube: 11. Run SonarQube scan
    SonarQube->>Agent: 12. Quality gate result
    
    alt Quality Gate Failed
        Agent->>Slack: ❌ Build failed - quality gate
        Agent->>Jenkins: Mark build FAILED
    else Quality Gate Passed
        Agent->>Agent: 13. Docker build
        Agent->>Agent: 14. Security scan (Trivy)
        
        alt Security Issues Found
            Agent->>Slack: ⚠️ Security vulnerabilities found
        else Security OK
            Agent->>Docker: 15. Push Docker image
            Docker->>Agent: Image pushed successfully
            
            Agent->>K8s: 16. Deploy to K8s
            K8s->>K8s: Rolling update
            K8s->>Agent: Deployment successful
            
            Agent->>Slack: ✅ Build SUCCESS<br/>Deployed to production
            Agent->>Jenkins: Mark build SUCCESS
        end
    end
    
    Jenkins->>Agent: 17. Cleanup workspace
    Agent->>Agent: 18. Pod terminates
    
    Note over Dev,Slack: Total time: ~5-8 minutes<br/>All automated via shared library!

Lessons Learned

What Worked Well

1. Start with Infrastructure

Building rock-solid Jenkins on Kubernetes first gave us confidence to tackle pipelines.

Lesson: Don’t build shared libraries on shaky infrastructure.

2. Make it Easier, Not Just Better

Our shared library wasn’t just “better practice”—it was genuinely easier to use.

Lesson: Adoption requires ease, not just righteousness.

3. Defaults Matter

80% of teams used default configurations. We spent 80% of effort on defaults.

Lesson: Optimize for the common case.

4. Incremental Migration

Forcing all 150 teams to migrate at once would have failed.

Lesson: Change management > Technical excellence.

5. Treat Shared Library Like Product

We had:

• Versioning (semantic versioning)
• Documentation
• Changelog
• Support channel (#jenkins-help)
• Regular releases

Lesson: Shared libraries are products, not side projects.

What We’d Do Differently

1. Testing Infrastructure Earlier

We didn’t have great testing for the shared library initially. Led to some production breaks.

Should have done:

// Unit tests for shared library functions
@Test
void testBuildDockerWithDefaultConfig() {
    def result = buildDocker(
        image: 'test-app',
        tag: 'v1.0.0'
    )
    assert result.exitCode == 0
}

// Integration tests
@Test
void testFullMavenPipeline() {
    def job = createTestJob('maven-test')
    def build = job.scheduleBuild2(0).get()
    assert build.result == Result.SUCCESS
}

2. Versioning Strategy

Initially, everyone used main branch. One breaking change = 150 broken pipelines.

Should have done:

// Pin to specific version
@Library('jenkins-shared-library@v2.1.0') _

// Or use version ranges
@Library('jenkins-shared-library@v2.x') _

3. Migration Automation

We manually migrated many pipelines. Should have automated more.

4. Documentation First

We built first, documented later. Should have been reversed.

Lesson: Documentation is part of development, not after.

Common Pitfalls to Avoid

Pitfall 1: Too Much Abstraction

// DON'T: Over-abstract
universalPipeline {
    language = 'java'
    buildTool = 'maven'
    containerization = true
    orchestration = 'kubernetes'
    // ... 50 more config options
}

// DO: Purpose-built pipelines
mavenPipeline {
    dockerImage = 'my-app'
}

Pitfall 2: No Escape Hatch

// DON'T: Force everyone into your abstraction
mavenPipeline {
    // No way to customize
}

// DO: Allow custom stages
mavenPipeline {
    dockerImage = 'my-app'
    
    // Custom pre-build stage
    preBuild = {
        sh 'echo "Custom logic here"'
    }
    
    // Custom post-deploy
    postDeploy = {
        sh 'run-smoke-tests.sh'
    }
}

Pitfall 3: Breaking Changes Without Communication

Always:

• Announce breaking changes 2 weeks ahead
• Provide migration guide
• Support old version for transition period
• Have rollback plan

Pitfall 4: Ignoring Feedback

Early on, developers asked for pythonPipeline. We said “use dockerPipeline.”

Bad idea. They went back to custom pipelines.

Lesson: Listen to users. If many people ask for something, build it.

Best Practices

1. Keep Shared Library Simple

// GOOD: Simple, clear
mavenPipeline {
    dockerImage = 'my-app'
}

// BAD: Too much magic
pipeline {
    agent { 
        magic() 
    }
    stages {
        autoDetect()
    }
}

2. Fail Fast with Clear Errors

def call(Map config = [:]) {
    // Validate required config
    if (!config.dockerImage) {
        error """
        ❌ Missing required parameter: dockerImage
        
        Example:
        mavenPipeline {
            dockerImage = 'your-app-name'
        }
        
        See documentation: https://docs.example.com/jenkins-shared-library
        """
    }
}

3. Version Everything

jenkins-shared-library/
├── CHANGELOG.md
├── VERSION (currently: v3.2.1)
└── docs/
    ├── v3.2.1/
    ├── v3.2.0/
    └── v3.1.0/

4. Monitor Usage

We tracked:

• Which pipeline templates are most used
• Which config options are popular
• Where people struggle (support tickets)

Informed our roadmap.

Conclusion: From Chaos to Platform

Where we started (November 2021):

• 150+ unique, fragmented pipelines
• Copy/paste culture
• High maintenance burden
• Developer frustration
• Inconsistent practices

Where we are now (6 months later):

• ONE shared library powering 142 pipelines
• Self-service pipeline creation (< 15 min)
• 87% reduction in maintenance
• Developer satisfaction: 9/10
• Platform for innovation

The transformation metrics:

• Pipeline creation: 2-3 days → 15 minutes (95% faster)
• Maintenance: 15 hrs/week → 2 hrs/week (87% reduction)
• Success rate: 70% → 95% (25 point improvement)
• Cost: $3,200/mo → $2,100/mo ($1,100 savings)

Most importantly: We transformed CI/CD from a bottleneck into an enabler.

The key lessons:

1. Infrastructure first - Build on solid foundation
2. Make it easy - Simpler beats better
3. Incremental change - Migration takes time
4. Treat as product - Versioning, docs, support
5. Listen to users - Feedback drives adoption

The philosophical shift:

We stopped seeing CI/CD as “pipelines” and started seeing it as a platform. Shared libraries weren’t just code reuse—they were the API for that platform.

When teams can create production-ready pipelines in 15 minutes, they spend more time building features and less time fighting infrastructure.

That’s the win.

Resources

My GitHub Repositories:

• Production Jenkins on EKS - Complete Jenkins deployment on Kubernetes
• EKS Platform Modules - Infrastructure foundation

Official Documentation:

• Jenkins Shared Libraries
• Jenkins on Kubernetes
• Configuration as Code (JCasC)

Tools & Plugins:

• Kubernetes Plugin
• Docker Pipeline Plugin
• SonarQube Scanner
• Slack Notification Plugin

About the Author: I’m a Senior DevOps and Cloud Engineer with 11+ years of experience. At Fidelity Information Services, I led the transformation of our CI/CD infrastructure, consolidating 150+ fragmented pipelines into a unified platform using Jenkins Shared Libraries. This work reduced pipeline maintenance by 87% and earned our team the “Star Team Award - DevOps 2023.” All infrastructure code and examples are available on my GitHub. Connect with me on LinkedIn.

Questions about Jenkins, shared libraries, or CI/CD transformation? Drop a comment below or reach out on LinkedIn. I’d love to hear about your CI/CD challenges and share experiences!