How to use AWS CLI like a Pro: The Ultimate DevOps Guide to JMESPath for output formatting

You are in the middle of troubleshooting an incident.

You run aws ec2 describe-instances and hit enter.
Your terminal is instantly flooded with a 10,000-line wall of JSON.

To find the one piece of data in this wall of text - say, a list of Instance IDs - you could do something like this.

aws ec2 describe-instances | grep InstanceId | awk -F '"' '{print $4}' | tr -d ','

It works. But this bash script hacking is suboptimal. The moment AWS decides to change the order of their JSON keys, or an unexpected nested array appears, this will not work. So, if you intend to save this for reuse, you may get unexpected results in future.

Also, if you want to print their availability zones alongside the instance IDs , how would you do this with grep and awk? You need to write a whole bash script. But it is inefficient. And may be dangerous in production.

So, let's stop treating AWS API responses like plain text files and start treating them like the queryable databases they are.

Enter JMESPath.

JMESPath is a query language for JSON, and it is built natively into the AWS CLI via the --query parameter. It allows you to parse, filter, and reshape complex JSON payloads safely and programmatically, without ever leaving the AWS CLI or relying on external tools like jq.

The Golden Rule of the AWS CLI: Filters vs. Queries

Before we write a single line of JMESPath, we need to establish the most important rule of the AWS CLI: understanding the boundary between the AWS API (Server-side) and your local machine (Client-side).

• Server-Side (--filters): This is you telling the AWS API, "Only send me data that matches these rules." You use filters to reduce the amount of data traveling over the network. For example, filtering for instances only in us-east-1.
• Client-Side (--query): This is you using JMESPath to shape the data after it arrives on your machine. You use queries to extract exactly the fields you want from the payload AWS sent back.

Think of it like this: --filters is asking the librarian to only bring you books about cloud computing. --query is you taking those books to your desk and extracting only the exact paragraphs you care about into a neat table.

What You Will Learn in This Deep Dive

If you master JMESPath, you will write cleaner automation, troubleshoot faster, and build resilient scripts that don't break when APIs evolve. In this guide, we are going completely down the rabbit hole. We will cover:

1. Navigating the JSON Tree: How to traverse dictionaries and arrays.
2. Shaping the Output: The critical difference between flattening and projecting data.
3. Filtering & Logic: Using conditionals (>, <, ==) and logical operators.
4. Built-In Functions: Sorting, string matching, and counting resources dynamically.
5. Unpacking AWS Tags: How to defeat the final boss of AWS JSON parsing.
6. Quoting Gotchas: Why your Mac scripts fail on Windows, and how to fix them.

Put away awk and grep. Let's get started.

Setting Up Your Sandbox (Optional but Recommended)

Before you start writing complex JMESPath queries, you need to fix your feedback loop.

When you are learning, you will make syntax errors. If your workflow is to run aws ec2 describe-instances --query '...' over and over, you are making a full network round-trip to the AWS API every time you hit enter. Not only is this painfully slow, but in large environments, it can trigger AWS rate limiting (ThrottlingException).

The Trick: Test Locally First

Run your base AWS CLI command once without any queries, and dump the raw JSON output into a local file:

aws ec2 describe-instances > data.json

Now you have a static dataset to play with. To run JMESPath queries against this local file, you can install jp, the official command-line tool for JMESPath.

Installing jp

Since jp is written in Go, it is a lightweight, standalone binary that works everywhere without requiring Python or Node.js.

macOS:

brew install jmespath/jmespath/jp

Linux:

sudo wget https://github.com/jmespath/jp/releases/latest/download/jp-linux-amd64 -O /usr/local/bin/jp
sudo chmod +x /usr/local/bin/jp

Windows:
Download the latest jp.exe from the official GitHub releases page and add it to your System PATH.

Once jp is installed you can test queries with your saved json without touching the AWS API:

jp -f data.json 'Reservations[0].Instances[0].InstanceId'

Level 1: Navigating the JSON Tree

Think of an AWS JSON response as a filesystem. You have folders (Dictionaries/Objects) and ordered lists (Arrays). To find the data you want, you just need to know how to traverse the path.

Dictionaries and Dot Notation

When dealing with a JSON object (data wrapped in curly braces { }), you use dot notation to go a level deeper. It works exactly like calling properties in Python or JavaScript.

Let's say you queried an IAM user and got this payload:

$ aws iam get-user --user-name cloud_app
{
    "User": {
        "Path": "/",
        "UserName": "liyum_app",
        "UserId": "AIDA54WIF7N7EMBNCM6G",
        "Arn": "arn:aws:iam::XXYYZZ:user/cloud_app",
        "CreateDate": "2024-10-07T09:11:03+00:00",
        "Tags": [
            {
                "Key": "SDMK092CD",
                "Value": "route_53"
            }
        ]
    }
}

To extract just the user ID, you step through the keys:

$ aws iam get-user --user-name liyum_app --query 'User.UserId'
"AIDA54WIF7N736SBNCM6G"

Arrays and Indexing

When dealing with arrays - a list of items (data wrapped in square brackets [ ]) - dot notation won't work. You have to tell JMESPath which item in the list you want using zero-based indexing.

Listing all security groups return an array of SecurityGroups.

$ aws ec2 describe-security-groups
{
    "SecurityGroups": [
        {
            "GroupId": "sg-0f36301959195d034",
            .
            .
            .
        },
        {
            "GroupId": "sg-0f36301959195d035",
            .
            .
            .
        }
        {
            "GroupId": "sg-0f36301959195d036",
            .
            .
            .
        },
}

To grab the GroupId of the very first security group in that list, you combine indexing with dot notation:

$ aws ec2 describe-security-groups --query 'SecurityGroups[0].GroupId'
"sg-0f36301959195d034"

The AWS Wrapper Problem: Why --query 'Instances' Fails

This brings us to the biggest stumbling block for beginners. If you run aws ec2 describe-instances, you might assume you can just query the Instances directly. But if you try --query 'Instances', AWS returns null.

Why? Because of The EC2 Wrapper.

Historically, AWS allowed you to launch multiple instances in a single request. To track this, the API wraps your instances inside a Reservations array. Your instances are actually buried two levels deep:

{
  "Reservations": [
    {
      "ReservationId": "r-12345",
      "Instances": [
        { "InstanceId": "i-0abc123" },
        { "InstanceId": "i-0def456" }
      ]
    }
  ]
}

To get the Instance ID of the very first instance inside the very first reservation, you have to traverse the entire path with indexes:

aws ec2 describe-instances --query 'Reservations[0].Instances[0].InstanceId'

But what if you want all instances, not just the first one? You can't just hardcode [0]. You need a way to strip away that Reservations wrapper and pull out all the instances at once.

That is where Flattening and Projections come in.

Level 2: Shaping the Output (Flattening and Projections)

In the last section, we hit a wall. AWS wraps EC2 instances inside a Reservations array. If you want to fetch all your instances, you can't just hardcode an index like Reservations[0]. You need to iterate through every reservation and pull out the instances inside them.

To do this, we use the wildcard * or the flatten operator [].

The Flattening Operators: [*] vs []

Let's look at what happens if you use the standard list projection [*]:

aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId'

If you run this, AWS gives you a deeply nested nightmare: an array containing arrays of Instance IDs. Because [*] preserves the shape of the original JSON, it keeps the instances neatly boxed inside their respective reservations. If you try to pipe this into a bash while loop, it will break immediately.

Here is where the magic happens. Replace the [*] with the flatten operator []:

aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId'

The [] operator says: "Iterate through this array, but crush the walls down." It takes all those separate instance arrays and flattens them into a single, continuous, one-dimensional list of Instance IDs.

Whenever you are dealing with paginated or wrapped AWS resources (like EC2 instances, Route53 record sets, or CloudWatch metrics), the flatten operator [] is your best friend.

Shaping the Data (Projections)

Now that you have a flat list of instances, you probably want more than just the Instance ID. What if you want the ID, the instance type, and the private IP address?

Instead of asking JMESPath for a single string, you ask it for a Projection. You can project your data into two formats: Lists or Hashes.

1. Multi-Select Lists (Perfect for Bash Scripting)

If you wrap your desired keys in square brackets [ ], JMESPath returns a raw array of values for every resource.

aws ec2 describe-instances \
  --query 'Reservations[].Instances[].[InstanceId, InstanceType, PrivateIpAddress]' \
  --output text

Why this rocks. When you combine a multi-select list with --output text, AWS strips away all the JSON brackets and commas, leaving you with clean, tab-separated columns. You can instantly pipe this output directly into an awk command or a bash read loop without any messy parsing.

2. Multi-Select Hashes (Perfect for Human Eyes)

If you wrap your keys in curly braces { }, you can create a brand new, custom JSON object. You can even alias the keys to make them easier to read.

aws ec2 describe-instances \
  --query 'Reservations[].Instances[].{ID: InstanceId, Size: InstanceType, IP: PrivateIpAddress}' \
  --output table

When you combine a multi-select hash with --output table, the AWS CLI uses your custom keys (ID, Size, IP) to generate a beautifully formatted ASCII table right in your terminal. This is the ultimate troubleshooting flex during an incident response call.

Now we can extract exactly what we want and format it perfectly. But what if we only want to see instances that are currently running, or volumes that are larger than 100GB? For that, we need logic.

Level 3: Filtering and Logic

Flattening and projecting data is great, but we rarely want all the data. During an incident, you don't care about the 500 instances that are stopped; you only care about the 3 that are currently running.

To filter arrays, JMESPath uses the ? operator. Think of it as a WHERE clause in SQL.

Conditional Filtering

To filter a list, you open your square brackets, add the ? symbol, and write your condition.

Let's filter our flattened EC2 list to only show running instances:

aws ec2 describe-instances \
  --query 'Reservations[].Instances[?State.Name==`running`].InstanceId'

Note on quoting: Because the entire query is wrapped in single quotes for the bash shell (' '), we use backticks (``) around the wordrunning`. In JMESPath, backticks denote a literal JSON string. We will dive deeper into quoting gotchas later, but for now, just remember to use backticks for strings inside your queries.

Numeric and Logical Operators (&&, ||)

You aren't limited to basic string matching. JMESPath supports standard comparators (>, <, !=) and logical operators (AND &&, OR ||).

Let's look at a classic CloudOps cost-optimization scenario. You want to find all EBS volumes that are unattached (available) AND are larger than 100GB so you can delete them.

aws ec2 describe-volumes \
  --query 'Volumes[?State==`available` && Size > `100`].{ID: VolumeId, Size: Size}' \
  --output table

Instead of pulling down data on thousands of volumes and writing a complex awk script to evaluate the sizes, JMESPath evaluates the logic natively before the text ever prints to your terminal.

Level 4: Built-In Functions (The Secret Weapon)

JMESPath includes a robust set of functions that let you manipulate data dynamically.

Here are the three functions you will use most often in DevOps architectures.

1. Counting with length()

How many Elastic IPs are currently allocated in your account? Usually, an engineer will list the IPs and pipe the output to wc -l. There is a better way. You can pass an array directly into the length() function:

aws ec2 describe-addresses --query 'length(Addresses)'

This returns a single integer. It is lightning-fast and completely eliminates the risk of wc -l accidentally counting blank lines or table headers.

2. Sorting with sort_by()

Let's say you are writing an infrastructure-as-code automation pipeline, and you need to programmatically fetch the exact ImageId of the absolute latest Amazon Linux 2023 AMI.

AWS might return 50 different AMIs. You can use sort_by() to order them by their CreationDate in ascending order. Since sort_by puts the oldest first and the newest last, you can use negative indexing [-1] to grab the very last item in the sorted array:

aws ec2 describe-images \
  --owners amazon \
  --filters "Name=name,Values=al2023-ami-2023.*-x86_64" \
  --query 'sort_by(Images, &CreationDate)[-1].ImageId' \
  --output text

Notice the & before CreationDate. In JMESPath, & represents an expression reference, telling the sort function which key to evaluate against.

3. String Matching (starts_with, contains)

Sometimes you need to find resources based on naming conventions rather than exact matches. If you need to locate all S3 buckets that belong to your production environment (assuming they follow a prod- naming standard), you can evaluate them using starts_with() inside a filter block:

aws s3api list-buckets \
  --query 'Buckets[?starts_with(Name, `prod-`)].Name'

At this point, you can navigate nested JSON, reshape it into tables, filter it with logic, and process it with functions. You are operating at a senior level.

But there is one final boss standing between you and complete AWS CLI mastery, and it breaks almost everyone's scripts: AWS Tags.

Unpacking AWS Tags

You can filter arrays, you can sort AMIs by date, and you can project data into beautiful ASCII tables. But eventually, your manager will ask for a simple report: "Get me a list of all running instances and their names."

You run your query, and suddenly everything falls apart.

Why? Because AWS does not store tags the way a normal developer would. A sane JSON structure for a tagged resource would look like a simple dictionary:

"Tags": {
  "Environment": "Production",
  "Name": "Web-Server-01"
}

If AWS did this, extracting the name would be as easy as querying Tags.Name. But they don't. Because AWS needs to enforce strict schema validation across their API, they format tags as an array of key-value objects:

"Tags": [
  { "Key": "Environment", "Value": "Production" },
  { "Key": "Name", "Value": "Web-Server-01" }
]

To get the name of your server, you can't just use dot notation. You have to search through the array, find the specific object where the Key is equal to "Name", and then extract its Value.

To do this, we need to introduce a new tool: the JMESPath Pipe Operator.

The Pipe Operator (|)

Do not confuse this with a bash pipe. Inside the single quotes of your --query string, the | symbol acts as a JMESPath projection evaluator. It takes the result of the expression on the left and feeds it as the input to the expression on the right.

This is crucial because when we filter the Tags array, the result is still an array—and we need to drill down into it.

The Blueprint: Tags[?Key=='Name'] | [0].Value

This is the exact syntax you need to memorize (or save to your snippets). Let's break down exactly how it defeats the AWS tag structure, step by step.

Step 1: The Filter
Tags[?Key==Name]
First, we look at the Tags array and use the ? operator to filter it. We are telling JMESPath: "Return only the dictionaries inside this array where the 'Key' equals 'Name'."
Because filters always return an array, the output of this step looks like this:
[ { "Key": "Name", "Value": "Web-Server-01" } ]

Step 2: The Pipe
|
We take that filtered array and pipe it to the right side of our query so we can manipulate it further.

Step 3: The Extraction
[0].Value
Because our piped data is an array containing a single object, we use [0] to grab that first object. Now that we have the dictionary itself, we use dot notation .Value to extract the final string.

Putting It All Together

Let's combine this with everything we've learned in the previous sections. We are going to flatten our EC2 instances, filter only the running ones, and create a custom table that projects the Instance ID, the Private IP, and the value of the "Name" tag.

aws ec2 describe-instances \
  --query 'Reservations[].Instances[?State.Name==`running`].{ID: InstanceId, IP: PrivateIpAddress, ServerName: Tags[?Key==`Name`]|[0].Value}' \
  --output table

If you can write and understand that query, you have officially mastered the AWS CLI. You are extracting deeply nested, inconsistently structured data, reshaping it, and formatting it for human readability in a single, native command.

But before you drop these scripts into your team's CI/CD pipelines, there is one last trap you need to avoid: OS-specific quoting rules.

Quoting Gotchas: Why Your Script Fails on Windows

Picture this: You just spent an hour crafting the perfect, elegant JMESPath query on your Mac. You drop it into your team's shared documentation. The next day, a teammate running Windows PowerShell copies and pastes your command, and their terminal explodes with syntax errors.

Welcome to quoting hell.

The AWS CLI uses the underlying operating system's shell to interpret your commands before they are sent to the JMESPath engine. Because Bash (Linux/macOS) and PowerShell (Windows) handle quotes and escape characters completely differently, a query that works flawlessly on one OS will often crash on the other.

Here are the rules of engagement to ensure your scripts actually work for everyone on your team.

The Linux & macOS Rule (Bash/Zsh)

On Unix-based systems, you want to prevent your shell from trying to interpret or expand anything inside your query.

• Outer Query: Wrap the entire --query argument in single quotes (' ').
• Inner Literal Strings: Use backticks () to denote literal strings inside your JMESPath expression.

The Mac/Linux Standard:

aws ec2 describe-instances \
  --query 'Reservations[].Instances[?State.Name==`running`].InstanceId'

The Windows Rule (PowerShell)

If you run that exact command in PowerShell, it fails. Why? Because in PowerShell, the backtick (``) is the escape character (just like the backslash` is in Bash).

When you run running in PowerShell, the shell swallows the backticks before the AWS CLI even sees them. The JMESPath engine ends up receiving the bare word running instead of a quoted string, causing a parsing error.

To fix this for Windows users, you must flip your quote structure:

• Outer Query: Wrap the entire --query argument in double quotes (" ").
• Inner Literal Strings: Use single quotes (' ') for the literal strings inside your expression.

The PowerShell Standard:

aws ec2 describe-instances `
  --query "Reservations[].Instances[?State.Name=='running'].InstanceId"

Note: We also swapped the bash line-continuation backslash \ for the PowerShell line-continuation backtick ``` at the end of the first line.

The Cross-Platform CI/CD Strategy

If you are writing automation scripts for a CI/CD pipeline (like GitHub Actions or GitLab CI), always explicitly define the shell environment (e.g., shell: bash) for your steps. This guarantees that your Linux-style quoting will execute correctly, regardless of the underlying runner's default OS.

When writing runbooks or internal documentation, it is a massive professional courtesy to either standardise your team on Bash (via WSL or Git Bash on Windows) or provide both the Bash and PowerShell syntax for complex queries.

The DevOps Playbook: 4 Real-World Scenarios

Theory is great, but execution is what matters. Here is the CloudQubes playbook—four highly practical, copy-paste ready queries that solve actual engineering problems.

Note: These snippets are formatted for Linux/macOS (Bash). If you are using Windows PowerShell, remember to swap the outer single quotes for double quotes, and the inner backticks for single quotes.

1. Cost Optimization: Hunting Zombie Volumes

When instances are terminated, engineers often forget to check the box to delete the associated storage. Over time, these unattached EBS volumes quietly bleed your cloud budget. This query finds all volumes in the available state (meaning they aren't attached to anything) and outputs their ID, size, and type into a clean table.

aws ec2 describe-volumes \
  --query 'Volumes[?State==`available`].{ID: VolumeId, Size: Size, Type: VolumeType}' \
  --output table

Pro-Tip: You can pipe this directly into an aws ec2 delete-volume loop once you verify the output.

2. Security Audit: Finding Open SSH Ports

Leaving port 22 open to the world (0.0.0.0/0) is a major security violation. Instead of clicking through hundreds of security groups in the AWS console, this query drills into the nested IpPermissions array to find the exact Group IDs violating this rule.

aws ec2 describe-security-groups \
  --query 'SecurityGroups[?IpPermissions[?ToPort==`22` && IpRanges[?CidrIp==`0.0.0.0/0`]]].GroupId' \
  --output text

3. Automation Pipeline: Fetching the Latest AMI

Hardcoding AMI IDs in your CI/CD pipelines or Packer templates is a bad practice because AMIs are regional and constantly updated. This query fetches all Amazon Linux 2023 images owned by AWS, sorts them chronologically by creation date, and grabs the single newest ImageId.

aws ec2 describe-images \
  --owners amazon \
  --filters "Name=name,Values=al2023-ami-2023.*-x86_64" \
  --query 'sort_by(Images, &CreationDate)[-1].ImageId' \
  --output text

4. Operations: The Ultimate EC2 Inventory

This is the holy grail for your daily operations. It flattens the EC2 reservations, filters out stopped or terminated instances, and builds a custom table showing exactly what you need during an incident: the Instance ID, the Private IP, and the value of the "Name" tag.

aws ec2 describe-instances \
  --query 'Reservations[].Instances[?State.Name==`running`].{ID: InstanceId, IP: PrivateIpAddress, ServerName: Tags[?Key==`Name`]|[0].Value}' \
  --output table

Wrapping up

When you first encounter JMESPath, the syntax can feel like a steep learning curve. But think about the alternative: writing brittle, multi-stage bash pipelines that break the second AWS adds a new key to their JSON responses.

By shifting the heavy lifting away from grep and awk and into the native AWS API client, you build tooling that is robust, predictable, and incredibly fast. You stop being someone who just runs commands, and you start acting like an architect who controls the data flow.

The next time you find yourself reaching for a text manipulation tool to parse an AWS response, stop. Build the query natively. Your future self (and your team's on-call engineers) will thank you.

Over to you: What is the most complex data extraction problem you've solved using the AWS CLI? Drop your best JMESPath queries in the comments below.

If you found this deep dive useful, make sure you are subscribed to The Cloud Edge the complimentary newsletter from CloudQubes where we break down real-world cloud engineering techniques every week.