5 PromQL Concepts That Baffled Me at the Start (And How to Dodge Them)

When I was starting out with Prometheus, I was pretty confident. It's just a database Should be like MySQL. So I thought.

And I was so wrong.

The moment I tried to write my first PromQL query, my brain broke a little. I quickly realized Prometheus isn't a relational database filled with neat rows and tables. It's a multi-dimensional time-series engine. My SQL intuition didn't just fail me; it actively led me in the wrong direction, leaving me staring at broken graphs and infuriating "empty query results."

But here's the good news: once I stopped treating PromQL like SQL and understood a few core rules, the math actually became incredibly elegant. And I went on to create extensive monitoring dashboards using PromQL.

Here are the exactly 5 concepts I had to learn the hard way. Let's get these traps out of the way so you can start writing PromQL confidently.

1. Instant Vectors vs. Range Vectors (My First Big Error)#

My Confusion
I remember my first time trying to build a Grafana dashboard. I just wanted to see a simple line chart of how many HTTP requests my app was getting over the last five minutes. So, using what I thought was perfect logic, I typed http_requests_total[5m] into the query bar, hit run, and waited for my beautiful graph. Instead, I was instantly slapped with a red error screen.

The Trap
I had fallen right into the most common beginner trap: failing to grasp the fundamental difference between an Instant Vector and a Range Vector.

• Instant Vector (http_requests_total): This is a snapshot. It gives you a single, current data point for each label combination at the exact millisecond the query evaluates. Because it resolves to a single Y-value for your current X-axis timestamp, you can graph this directly.
• Range Vector (http_requests_total[5m]): This is a bucket of history. By adding the [5m], you are asking Prometheus for every single raw scrape that occurred over that 5-minute window. You cannot graph this directly. Your graphing tool expects one value per timestamp, and you just handed it an entire array of historical data points. It panics and throws an error.

How to Dodge It
Here is the golden rule of thumb I finally adopted: whenever you add time brackets like [5m] to a metric, you are creating a range vector, and range vectors exist almost exclusively to be fed into functions.

You don't graph them; you wrap them.

If you want to visualize that 5-minute window, you need to wrap that range vector in a time-series math function like rate() or increase(). For example: rate(http_requests_total[5m]). That function digests the bucket of historical data, calculates the math, and spits out a shiny new Instant Vector that you can happily plot on your graph.

2. The Illusion of Exact Math: Extrapolation in increase()#

My Confusion
Once I figured out how to use functions, I thought I was in the clear. I needed to count how many HTTP 500 errors my application threw over the last hour. I confidently wrote increase(http_requests_total{status="500"}[1h]). I fully expected to see a clean, whole number - something like 5 or 12.

Instead, PromQL returned 5.319.

I spent the next hour frantically checking logs, convinced my metrics pipeline was completely broken. How in the world can a server throw 0.319 of an error?

The Trap
I was treating Prometheus like a SQL accounting ledger. In a relational database, a COUNT() is an absolute, exact mathematical truth. In Prometheus, functions like increase() and rate() are estimations.

Prometheus relies on scraping targets at intervals (say, every 15 seconds). It is highly unlikely that your 1-hour time window [1h] aligns perfectly down to the millisecond with your first and last metric scrape. To account for this, Prometheus looks at the rate of change between the data points it does have, and then extrapolates that slope out to the exact mathematical boundaries of your 1-hour window.

Because it's projecting a slope rather than counting discrete events, you end up with fractions of an error.

How to Dodge It
You have to let go of the need for perfect, integer-based accounting. Prometheus is designed for alerting, scaling, and observing trends, not for billing or financial audits.

Once I understood why the decimals were there, I learned to accept them. If I need to know roughly how many errors happened, 5.319 tells me exactly what I need to know: the system is failing, and it's failing at a specific rate. If you absolutely need a whole number for a dashboard aesthetic, you can wrap the whole thing in the round() function: round(increase(http_requests_total[1h])). Just remember that underneath, Prometheus is always prioritizing the rate of change over exactness.

3. The "Final Boss": Vector Matching (group_left & group_right)#

My Confusion
I spent an embarrassing amount of time one afternoon just trying to calculate a simple percentage. I had container_memory_usage_bytes and container_spec_memory_limit_bytes. I figured I could just divide one by the other, multiply by 100, and go to lunch.

I wrote container_memory_usage_bytes / container_spec_memory_limit_bytes.

The result? "Empty query result." I stared at the screen. Both metrics existed. Both had data. Why wouldn't it just do the math?

The Trap
I was assuming PromQL handled joins like a SQL database. It doesn't. In PromQL, if you try to perform math between two metrics, it will only calculate a result if the labels on both sides match exactly. If the usage metric has a label called cpu="0" and the limit metric doesn't, PromQL throws its hands up and refuses to do the math.

To fix this, you have to explicitly tell Prometheus which labels to match on using the on() or ignoring() keywords.

But even then, you hit the second trap: the one-to-many relationship. What if you want to join your CPU usage metric with a metric that just holds metadata about the pod's version? The metadata metric has one entry per pod, but the CPU metric has many entries per pod (one for each core). PromQL will fail again because it doesn't know what to do with the extra data.

How to Dodge It
This is where group_left and group_right come in, and understanding them was my biggest "aha!" moment.

Think of group_left as a tractor beam. It tells Prometheus, "Hey, the metric on the left side of this equation has higher cardinality (the 'many' side). Take the labels from the metric on the right (the 'one' side) and pull them over to the left so the math works."

If you write metric_many * on(pod_name) group_left(version) metric_one, you are successfully joining them on the pod_name, and pulling the version label across so it appears in your final result. Once you master this, you can join almost anything.

4. The Phantom Metric: When Data Simply Vanishes#

My Confusion
I’ll never forget the first time one of my services hard-crashed in production. I wasn't too worried at first because I had written what I thought was a bulletproof alert: rate(http_errors_total[5m]) > 0. If there were errors, PagerDuty would wake me up.

Well, it never happened. But, this alarm will not fire if the service burned down entirely.

The Trap
Once again, my SQL brain had betrayed me. In a traditional database, if you ask for a count of errors and there are none (or the system is offline), it returns a 0.

In PromQL, if a pod dies or a service stops sending traffic, the metric doesn't drop to zero. It simply ceases to exist. It vanishes. Because the metric is gone, the condition > 0 doesn't evaluate to false or 0. It evaluates to an empty result. To PromQL, this is no problem. No alerts will fire.

How to Dodge It
You have to actively account for missing data in your queries and alerts.

For critical alerts, you need to learn the absent() function. If I write absent(up{job="my_service"}), the query can trigger an alert specifically when that metric drops off the face of the earth.

If I'm building a dashboard and I just want the graph to drop cleanly to 0 instead of showing a blank gap when traffic stops, I use a neat little trick to artificially inject a zero: rate(http_requests_total[5m]) OR on() vector(0). This tells Prometheus, "If the first query returns nothing, just plot a 0."

5. The Pod Restart Panic: Counter Resets#

My Confusion
Early on, I needed to track the total number of processed background jobs over a 24-hour period. I knew I had a counter metric for this, but I also knew my Kubernetes pods were volatile. They restarted during deployments, scaled up and down, and occasionally crashed. Every time a pod restarted, its local memory wiped out, and its counter dropped back to zero.

I spent half a day trying to write a massively convoluted PromQL query with subqueries and max values just to calculate the "real" total by stitching together the pre-crash and post-crash numbers.

The Trap
I was suffering from a severe lack of trust in the database. Because I didn't fully understand how PromQL functions were built, I assumed a counter dropping to zero would completely destroy the math inside an increase() function. I thought I had to handle the edge cases myself.

How to Dodge It
This was the most relieving discovery of my PromQL journey: functions like rate() and increase() are basically magic.

Prometheus knows that counters are only supposed to go up. So, if it evaluates a time window and sees that a counter's value is suddenly lower than the previous scrape, it immediately flags it as a counter reset. It automatically calculates the difference, stitches the math together, and compensates for the restart seamlessly behind the scenes.

You don't need complex logic. You just write increase(background_jobs_total[24h]) and let Prometheus do the heavy lifting.

Wrapping Up#

Moving from SQL to PromQL is a mental leap. It requires you to stop thinking in static rows and start thinking in streams of time. But once you accept that range vectors need to be wrapped in functions, that decimals are normal, that labels must match exactly, that missing data is invisible, and that counters heal themselves, everything changes.

You stop blindly copying queries from Stack Overflow and start actually speaking the language of your infrastructure.

The best way to solidify this is to break things on purpose. Open your PromQL console today, take one of your most complex dashboard queries, and start peeling it apart layer by layer. Look at the raw range vectors, test the vector matching, and see what happens when you remove the absent() functions.

Keep experimenting, keep breaking things, and I'll see you in the next CloudQubes deep dive!

FAQ#

Why can't I graph a range vector in Prometheus?#

You cannot graph a range vector directly because it returns multiple historical data points per timestamp, whereas graphing tools require a single data point (an instant vector) per timestamp. To visualize a range vector, you must wrap it in a time-series function like rate() or increase() to calculate a single value for the graph to plot.

Why does PromQL increase() return decimals instead of whole numbers?#

PromQL returns decimals for the increase() function because it extrapolates data. Prometheus calculates the rate of change between the actual data points it scraped and projects that slope across the exact time boundaries you requested. This results in a mathematical estimation rather than an exact integer count.

How do you join two metrics in PromQL with different labels?#

To join metrics with differing labels, use the on() or ignoring() keywords to explicitly define which labels Prometheus should match. If you are dealing with a many-to-one relationship, append group_left or group_right to tell Prometheus which side has the higher cardinality and pull the correct labels across the join.

How do I trigger a Prometheus alert when a metric stops reporting?#

Use the absent() function to trigger an alert when a metric stops reporting. In PromQL, when a service dies, the metric doesn't drop to zero—it ceases to exist. Standard conditions like > 0 will return an empty result and fail to fire, making absent() necessary to detect missing data.

What happens to a PromQL counter when a Kubernetes pod restarts?#

When a pod restarts and its memory is wiped, PromQL automatically handles the counter reset. Functions like rate() and increase() are designed to detect when a new scrape value is lower than the previous one. They instantly flag this as a reset and seamlessly compensate for the drop behind the scenes.