Why Is My ECS Deployment Taking 15 Minutes? 3 Real-World Gotchas and the AWS CLI Toolkit

You’ve probably been there.

You push a tiny commit to your main branch. GitHub Actions kicks off. Your unit tests pass in 20 seconds. The Docker container builds and pushes to Amazon ECR in 30 seconds. You’re feeling great about your continuous deployment pipeline.

And then… you hit the ECS deploy step.

The spinner spins. Five minutes pass. Ten minutes pass. You start wondering if AWS is having an outage. At the 15-minute mark, your workflow abruptly dies with a generic timeout error:

Error: Service ecs-flask-dev-service did not reach a steady state.
Process completed with exit code 1.

Here’s the thing: AWS ECS is rarely "stuck." In almost every case, ECS is frantically doing its job behind the scenes—trying to protect your production users from a broken deployment, failing silently, and relaunching tasks in an infinite crash loop.

In this post, we’ll walk through the top 3 gotchas that silently break containerized CI/CD pipelines on AWS ECS Fargate, why they happen, and the exact 4-command AWS CLI toolkit you need to diagnose any ECS deployment issue in under 30 seconds.

The Target Architecture

Before we dive into the failure modes, let’s visualize what a modern, secure ECS CI/CD pipeline looks like:

We’re running a Python Flask application served by Gunicorn on AWS ECS Fargate, with infrastructure provisioned via OpenTofu and deployments orchestrated by GitHub Actions using OpenID Connect (OIDC).

Now, let's break down where things go sideways.

Gotcha #1: The OIDC "Not Authorized" Handshake Trap

If you're still storing long-lived AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in GitHub Secrets, you're exposing your AWS account to unnecessary risk. The gold standard today is OIDC (OpenID Connect): GitHub Actions requests a temporary JSON Web Token (JWT) and exchanges it directly with AWS STS for short-lived credentials.

Sounds clean in theory. But on your first deployment, you often hit this brick wall:

Assuming role with OIDC
Assuming role with OIDC
Error: Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity

Why does this happen?

AWS IAM evaluates the Trust Policy on your deployment role with extreme scrutiny. Three subtle mismatches usually cause this:

1. Default Variable Oversights: When using IaC modules, the github_repo variable might default to a placeholder like my-org/my-repo. When GitHub calls from your-username/your-repo, the sub (subject) claim in the token fails the StringEquals / StringLike check.
2. Case Sensitivity: AWS IAM conditions are case-sensitive. If your GitHub username has mixed casing (e.g. MyOrg) and your policy specifies repo:myorg/..., AWS will reject the request.
3. Outdated Thumbprints: AWS IAM OpenID Connect providers require the root CA thumbprint of GitHub's certificate authority. If your IaC queries GitHub's dynamic leaf certificate rather than the official intermediate CA thumbprints, AWS STS cannot verify the token signature.

The Fix

In your OpenTofu/Terraform OIDC module, use GitHub's official root thumbprints and make your sub condition resilient:

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1",
    "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
  ]
}

resource "aws_iam_role" "github_actions" {
  name = "ecs-github-deploy-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.github.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          "token.actions.githubusercontent.com:sub" = "repo:your-github-username/your-repo:*"
        }
      }
    }]
  })
}

Pro Tip: In your .github/workflows/deploy-app.yml, always specify permissions: id-token: write on the specific job level, not just the root workflow level, to ensure the runner has permission to mint the JWT.

Gotcha #2: The Task Definition Family vs. Service Name Mixup

Once your authentication works, the pipeline needs to fetch the latest Task Definition, swap out the container image tag with the new Git commit SHA, and register a new revision.

That's when you see this:

Run aws ecs describe-task-definition \
  --task-definition ecs-flask-dev-service \
  --query taskDefinition > task-definition.json

aws: [ERROR]: An error occurred (ClientException) when calling DescribeTaskDefinition: 
Unable to describe task definition.

Why does this happen?

It’s easy to accidentally pass the ECS Service Name (ecs-flask-dev-service) to --task-definition.

AWS ECS treats Task Definitions and Services as completely distinct entities. --task-definition expects either the family name (ecs-flask-dev-task) or a full ARN (arn:aws:ecs:...:task-definition/ecs-flask-dev-task:12).

The Fix

Declare explicit environment variables in your workflow:

env:
  ECS_SERVICE: "ecs-flask-dev-service"
  ECS_TASK_DEFINITION: "ecs-flask-dev-task"  # Family name, not service name!

steps:
  - name: Download Current ECS Task Definition
    run: |
      aws ecs describe-task-definition \
        --task-definition ${{ env.ECS_TASK_DEFINITION }} \
        --query taskDefinition > task-definition.json

Gotcha #3: The Internal Port Mismatch Loop (The 15-Minute Timeout)

This is the king of all ECS deployment gotchas.

Your image builds. The new task registers. The deployment starts. And then… you wait. And wait.

(service ecs-flask-dev-service) (task 8beb6986...) (port 5000) is unhealthy in (target-group ecs-flask-dev-tg) due to (reason Health checks failed).
(service ecs-flask-dev-service) has stopped 1 running tasks: (task 8beb6986...).
(service ecs-flask-dev-service) has started 1 tasks: (task c7d9961e...).

Why does this happen?

Look at what happens across your stack during a rolling deployment:

If your Dockerfile has:

CMD ["gunicorn", "--chdir", "src", "--bind", "0.0.0.0:5001", "app:app"]

...while your OpenTofu configuration has:

resource "aws_lb_target_group" "app" {
  port        = 5000
  target_type = "ip"
  ...
}

The container starts up perfectly, boots Gunicorn on 5001, and sits there happily. Meanwhile, the Application Load Balancer probes port 5000, gets Connection Refused, marks the task unhealthy after 2 minutes, and orders ECS to kill it.

ECS obliges, terminates the container, launches another one with the same image, and the loop repeats until your CI/CD job hits its hard timeout.

The Fix

Ensure strict alignment across all 4 touchpoints:

1. Application: gunicorn --bind 0.0.0.0:5000
2. Dockerfile: EXPOSE 5000
3. ECS Task Definition: portMappings = [{ containerPort = 5000 }]
4. ALB Target Group: port = 5000

The 4-Command AWS CLI Troubleshooting Toolkit

When your deployment is running slowly or failing, don't guess in the AWS Web Console. Run these 4 commands from your terminal:

1. View Real-Time Service Events

aws ecs describe-services \
  --cluster ecs-flask-dev-cluster \
  --services ecs-flask-dev-service \
  --region us-east-1 \
  --query 'services[0].events[:6].[createdAt,message]' \
  --output table

• What it tells you: Look for messages like is unhealthy in target-group due to (Health checks failed) or has stopped 1 running tasks.

2. Inspect Target Group Health

# Get Target Group ARN
TG_ARN=$(aws elbv2 describe-target-groups \
  --names ecs-flask-dev-tg \
  --region us-east-1 \
  --query 'TargetGroups[0].TargetGroupArn' \
  --output text)

# Check target health states
aws elbv2 describe-target-health \
  --target-group-arn $TG_ARN \
  --region us-east-1 \
  --output table

• What it tells you: Shows whether your container IPs are initial, healthy, or unhealthy with specific error reasons (Target.FailedHealthChecks vs Target.ResponseCodeMismatch).

3. Stream Live CloudWatch Container Logs

aws logs tail /ecs/ecs-flask-dev --region us-east-1 --since 10m

• What it tells you: See the exact startup output of your app: text [INFO] Starting gunicorn 22.0.0 [INFO] Listening at: http://0.0.0.0:5000

4. Check Rolling Deployment Progress

aws ecs describe-services \
  --cluster ecs-flask-dev-cluster \
  --services ecs-flask-dev-service \
  --region us-east-1 \
  --query 'services[0].deployments[*].{Status:status,Rollout:rolloutState,Desired:desiredCount,Running:runningCount,Pending:pendingCount}' \
  --output table

• What it tells you: Shows whether your deployment is in IN_PROGRESS or COMPLETED state and whether tasks are shifting from old to new revisions.

How to Cut Deployment Times from 5 Minutes to 45 Seconds

Once your containers are healthy, you might still notice that rolling updates take 4–5 minutes per release. You can optimize this dramatically with two configuration tweaks:

1. Tune ALB Target Group Timers

By default, ALBs use conservative timers (30-second health check intervals and 300-second connection draining). For stateless web APIs, you can safely tune these down:

resource "aws_lb_target_group" "app" {
  port        = 5000
  protocol    = "HTTP"
  target_type = "ip"

  health_check {
    path                = "/health"
    interval            = 15  # Check every 15s instead of 30s
    timeout             = 5
    healthy_threshold   = 2   # Mark healthy after 2 successes (30s)
    unhealthy_threshold = 2
  }

  deregistration_delay = 15   # Drain old connections in 15s instead of 300s
}

2. Enable the ECS Deployment Circuit Breaker

If a bad container image ever slips through into production, you don't want your pipeline hanging for 15 minutes. Enable the native circuit breaker:

resource "aws_ecs_service" "app" {
  name            = "ecs-flask-dev-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 2

  deployment_minimum_healthy_percent = 100
  deployment_maximum_percent         = 200

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }
}

With deployment_circuit_breaker enabled, ECS automatically detects task startup failures, cancels the deployment, and rolls back to the previous working image in under 60 seconds.

Takeaways

1. Security First: Eliminate long-lived IAM keys from GitHub Secrets. Use GitHub OIDC with scoped trust policies (repo:<user>/<repo>:*).
2. Zero-Downtime Math: minimum_healthy_percent = 100 and maximum_percent = 200 ensures ECS launches new containers and verifies their health before draining any old traffic.
3. Prevent State Overwrites: When managing ECS in OpenTofu/Terraform, always add lifecycle { ignore_changes = [task_definition, desired_count] } to your service definition so infrastructure runs don't revert new application deployments.
4. Debug with the CLI First: When a deployment hangs, check describe-services events and describe-target-health immediately. The logs never lie.