What Certifications Do Not Teach: How to Survive ClickOps in a Terraform World

If you've spent any time studying for AWS certifications, you've been sold a beautifully pristine lie. You are taught that 100% of your infrastructure should be perfectly version-controlled, thoroughly documented, and deployed strictly via Infrastructure as Code (IaC). In the world of multiple-choice exams, if a resource isn't in your Terraform state, it simply doesn't exist.

Then you land a real engineering role, get paged at 2 AM for a P1 outage, and reality hits.

When the application is throwing 500 errors and the company is actively bleeding money, nobody cares about your CI/CD pipeline. An engineer panics, logs straight into the AWS console, manually tweaks a security group or patches a routing table to restore service, and goes back to sleep. They fully intend to backport that change into Terraform on Monday morning.

Spoiler alert: They forget.

Fast forward three weeks. You are tasked with a routine update. You open the repository, make a minor change to a resource tag, and confidently run terraform apply.

In an instant, Terraform looks at the live environment, notices the 2 AM manual hotfix isn't in your code, and aggressively wipes it out. You just accidentally reverted the fix, brought the bug back to life, and took down production. And because your code looked perfect, you have absolutely no idea why it happened.

This is the reality of inheriting a legacy environment. Let's break down how to diagnose this hidden drift, and exactly how to fix it before you destroy your own production environment.

Diagnosing the "ClickOps" Drift

In the cloud-native world, there is a single, unforgiving law you must burn into your brain: Never run apply without reading the plan first.

When you run terraform plan, Terraform performs a three-way reconciliation. It looks at your .tf files, checks your state file, and makes API calls to AWS to see what is actually running in production.

Here is the catch: Terraform is fiercely opinionated. It believes that your code is the ultimate truth. If the AWS console differs from your code, Terraform assumes the AWS console is wrong and will aggressively attempt to overwrite it.

You spot this danger by looking for the ~ update in-place marker in your plan output.

Let’s look at a real-world scenario. Imagine your team uses an AWS Managed Prefix List to manage a shared set of IP ranges across multiple security groups. During a midnight outage, a senior engineer realized a new subnet couldn't communicate with the database. To fix it quickly, they logged into the console and manually added the missing CIDR block to the prefix list.

Weeks later, you run a plan for an unrelated task. You scroll through the output and see something like this:

  ~ resource "aws_ec2_managed_prefix_list" "shared_ips" {
        id             = "pl-12345678"
        name           = "database-access-ips"
        # (4 unchanged attributes hidden)

      - entry {
          - cidr        = "10.0.5.0/24" -> null
          - description = "Emergency fix for new subnet" -> null
        }
    }

Do not ignore that red minus sign.

Terraform is explicitly telling you: "I see a CIDR block in AWS that isn't in my configuration. I am going to delete it to make production match your code."

If you just skim the output and blindly hit yes on the apply, you will successfully deploy your minor update—and instantly sever the database connection for that subnet, causing a massive regression.

Diagnosing drift isn't about looking for errors; it's about looking for changes you didn't write. Once you spot them, you have to reverse-engineer them back into your code before moving forward.

The Reversal Technique

When you spot that red minus sign threatening to delete a manual fix, your goal completely shifts. You are no longer trying to deploy your original update. Your immediate mission is to reverse-engineer reality back into your codebase.

Here is exactly how you safely absorb that 2 AM ClickOps change without breaking production.

Step 1: Isolate the Drift

Do not clear your terminal. Look closely at the terraform plan output from the previous step. Terraform is handing you exactly what you need to write. Copy the exact attributes it is threatening to remove—in our case, the 10.0.5.0/24 CIDR block and its description.

Step 2: Update the .tf File

Open the repository and find the resource block causing the drift. To fix the issue, you simply add the missing configuration directly into your Terraform code, so that your desired state matches the live state.

For our prefix list example, you will update the aws_ec2_managed_prefix_list resource to include the new entry block:

resource "aws_ec2_managed_prefix_list" "shared_ips" {
  name           = "database-access-ips"
  address_family = "IPv4"
  max_entries    = 10

  # ... (other existing entries) ...

  # Backporting the 2 AM manual hotfix
  entry {
    cidr        = "10.0.5.0/24"
    description = "Emergency fix for new subnet"
  }
}

Step 3: The Verification Plan

Save your file and run terraform plan again.

Because your code now explicitly includes the manual hotfix, Terraform sees that the code and the AWS console are in perfect harmony. It drops its demand to overwrite the resource.

You are looking for this holy grail output at the bottom of your terminal:

No changes. Your infrastructure matches the configuration.

Once you see this, the manual change is safely cemented into your Infrastructure as Code. You can now safely proceed with whatever routine update you originally intended to deploy.

Step 4: The Edge Case (Brand New Resources)

This technique works perfectly when someone modifies an existing resource. But what if another engineer built a brand-new S3 bucket or IAM role entirely from scratch in the console?

Updating existing code isn't enough because Terraform doesn't even know the resource exists. In that scenario, you have to write the new resource block in your .tf file from scratch, and then run terraform import to manually map the live AWS resource to your state file. Only then will a plan come back clean.

The Escape Hatch: Using ignore_changes

Sometimes, reconciling state isn't the goal. Sometimes, you actually want Terraform to ignore drift. This is where the lifecycle block comes in. By using the ignore_changes meta-argument, you explicitly tell Terraform: "Even if the live infrastructure differs from my code for these specific attributes, do not attempt to change them back."

It is an incredibly powerful escape hatch, but it is also one of the most abused features in Terraform. Here is how to use it without shooting yourself in the foot.

The Right Way: Accepting Automated Drift

You should use ignore_changes for attributes that are expected to be modified dynamically by AWS or external systems after the resource is created.

A classic example is an AWS Auto Scaling Group (ASG). You might set an initial desired_capacity of 2 in your code to get the environment off the ground. But once the ASG is live, AWS Auto Scaling policies will dynamically increase or decrease that number based on traffic load. If you don't ignore this attribute, your next routine terraform apply will aggressively scale your production fleet back down to 2, instantly degrading performance.

resource "aws_autoscaling_group" "web_fleet" {
  name             = "web-asg"
  max_size         = 10
  min_size         = 2
  desired_capacity = 2 # Set initially, but AWS manages it afterward

  lifecycle {
    ignore_changes = [
      desired_capacity,
      tags["LastUpdated"] # Ignoring a specific tag modified by a CI/CD script
    ]
  }
}

The Wrong Way: Hiding Bad Habits

The dangerous anti-pattern is using ignore_changes just to make a noisy terraform plan go away, rather than fixing an underlying ClickOps culture.

I have seen teams set ignore_changes = [ingress] on an aws_security_group because developers kept manually opening ports in the AWS console and complaining when Terraform closed them the next day.

By doing this, you are permanently blinding your Infrastructure as Code to that attribute. If someone accidentally opens port 22 to 0.0.0.0/0 in the console, terraform plan will show zero changes. You won't know the vulnerability exists unless you stumble upon it in AWS.

You can even use ignore_changes = all to make Terraform provision a resource once and then wash its hands of it completely. But use this sparingly. The escape hatch is meant for predictable system automation, never as a rug to sweep your team's messy manual habits under.

The Engineering Edge

The biggest mindset shift you have to make when transitioning from passing AWS certifications to running actual production environments is how you view your codebase.

Junior engineers blindly trust the code. They assume that if the Terraform files are syntactically valid and the CI/CD pipeline is green, reality will naturally fall in line. If something breaks, they blame whoever touched the console.

Senior engineers verify the state. They know that Infrastructure as Code is not a static, sacred artifact—it is a living system that constantly rubs up against the chaotic, messy reality of human behavior and system outages. In a real production environment, your Terraform configuration isn't the absolute truth; it is just a hypothesis until terraform plan confirms it against the live AWS API.

Your job isn't just to write infrastructure code. Your job is to constantly reconcile that code with reality. Knowing exactly how to safely absorb a 2 AM manual hotfix into your repository, and knowing when to strategically use escape hatches like ignore_changes, is exactly the kind of practical, beyond-the-cert expertise that separates entry-level ticket-takers from lead engineers.

Embrace the messy reality of production, respect the drift, and always read the plan.