Grafana query languages

Grafana draws charts. In the usual setup it does not store the samples. A panel has a data source, a query in that source's language, and a time range from the dashboard picker. Prometheus and Mimir speak PromQL. Loki speaks LogQL. A SQL source speaks SQL. If the panel is empty, the query inspector shows the request after Grafana has substituted $__rate_interval and the other macros. Read that before you rewrite the query. The string on the wire is often not the string on screen.

The metric names below (http_requests_total, http_request_duration_seconds_bucket) are the conventional names from an instrumented service. They are not built into Prometheus. up is built in: it is 1 when a target was scraped and 0 when it was not.

How a range query is shaped

A graph is a range query. Grafana asks for samples from the start of the dashboard range to the end, stepping by about $__interval (the gap between points, from the time range, the panel width, and max data points). A stat or a table is often an instant query: one evaluation at the end. That is why a number and a graph of the "same" query disagree.

An instant vector is one sample per series. A range vector, metric[5m], is the raw samples in that window. rate consumes a range vector and returns an instant vector. You cannot graph the range vector itself.

Selectors

A selector is a metric name plus label matchers. The name is sugar for {__name__="http_requests_total"}. Matchers are = and != for equality, and =~ and !~ for regular expressions.

http_requests_total{job="api", status=~"5.."}

Prometheus regular expressions are RE2, and they are anchored. status=~"5.." matches the whole value, as if you had written ^5..$. Do not add the anchors. status=~"5" matches 5, not 500. A substring search needs .* on both sides.

A matcher on a high-cardinality label (user id, full URL, trace id) pulls a huge number of series. Keep those labels off the metric.

rate and irate

http_requests_total is a counter. It only goes up, except when the process restarts and the counter drops to zero. rate takes a range vector of a counter and returns the per-second average increase over that window. It notices a drop and treats it as a reset, so a restart does not look like a negative rate. It also extrapolates to the ends of the window, so the number is an estimate, not a raw subtraction of the first and last sample.

rate(http_requests_total{job="api"}[5m])

rate needs at least two samples in the window. If you scrape every 15 seconds and the window is 15 seconds, most evaluations see one sample and the series disappears. The practical rule is a window of at least four scrape intervals. Grafana's $__rate_interval encodes that rule:

rate(http_requests_total{job="api"}[$__rate_interval])

The macro is max($__interval + scrape interval, 4 * scrape interval). The scrape interval is the query's Min step if you set one, otherwise the scrape interval configured on the Prometheus data source (15 seconds if you left the default). The panel's Min interval does not change that scrape-interval input. It only stops the step getting finer than you asked. Use $__rate_interval inside rate, increase, and irate. Use $__interval when you mean "the step between points", not when you mean "a window long enough to see two scrapes".

irate uses only the last two samples in the window. It reacts faster and it is noisy. Prometheus' own guidance is to use rate for alerts and recording rules, and to reserve irate for a graph of a very volatile counter where you want the spikes. An alert on irate flaps.

rate is the wrong function for a gauge. CPU temperature, queue depth, and "bytes currently in use" are gauges. Graph the gauge, or smooth it with avg_over_time(queue_depth[$__rate_interval]). deriv is for a gauge you want a per-second slope of. delta is a simple last-minus-first and does not handle counter resets. Counters want rate or increase.

increase(counter[$__rate_interval]) is the count of events over the window, roughly rate times the window in seconds. It extrapolates too, so it can be slightly above the integer number of requests you can count by hand. That is expected. For "how many 500s in this window", increase is the function. For "requests per second", rate is.

sum by

Aggregation happens after rate. Summing the counters first and then taking a rate mixes series that reset at different times, and it is the wrong shape whenever the series do not share a single counter.

sum by (job) (
  rate(http_requests_total[$__rate_interval])
)

by (job) keeps job and drops every other label. without (instance) drops instance and keeps the rest, which is what you want when you are combining replicas but still care about job, route, and status. Forgetting the aggregation leaves one line per series, which on a busy service is one line per pod, and the legend becomes unreadable.

Binary operators match labels on both sides. rate(errors) / rate(requests) divides only the series whose labels are identical. If the error counter has a status label and the request counter does not, the division matches nothing. Aggregate both sides to the labels you care about first:

sum by (job) (rate(http_requests_total{status=~"5.."}[$__rate_interval]))
/
sum by (job) (rate(http_requests_total[$__rate_interval]))

Division by zero produces NaN, and Prometheus drops those points. A job with no traffic disappears instead of showing zero. That is usually what you want on a graph and the wrong answer in an alert that should fire when traffic stops. absent() is the function for "this series vanished".

histogram_quantile and range vectors

A classic Prometheus histogram is a set of counters, one per bucket, labelled le for "less than or equal to". The buckets are cumulative: the le="0.5" series includes every observation that was also in le="0.1". +Inf counts everything. histogram_quantile estimates a quantile by interpolating inside those buckets. The estimate is only as sharp as the bucket boundaries you chose.

histogram_quantile(
  0.95,
  sum by (le, job) (
    rate(http_request_duration_seconds_bucket[$__rate_interval])
  )
)

The order is fixed. rate each bucket, then sum by while keeping le, then histogram_quantile. If the sum drops le, the function has no buckets and the panel is empty. sum by (le) is the whole-service figure. sum by (le, job) is one figure per job. A quantile of 0.95 is the 95th percentile: about 95 percent of observations fell at or below the reported value, within the error of the bucket layout.

Native histograms, available in Prometheus 2.40 and newer, store the histogram in one series and do not use an le label. histogram_quantile(0.95, sum(rate(metric[$__rate_interval]))) is the form for those. If you sum by (le) a native histogram you will not get a percentile. Check whether le is present before you copy a classic query.

avg_over_time and max_over_time take a range vector too. The window sits after the selector: metric{job="api"}[5m]. An offset compares with an earlier time and goes after the window: metric{job="api"}[5m] offset 1d. A subquery such as max_over_time(rate(metric[5m])[1h:]) re-evaluates an instant query. The colon is the step. A recording rule is usually the simpler fix.

$__range is the whole dashboard window as a duration, such as 1h. increase(metric[$__range]) answers "how many events in the range on screen". It is a poor window for a per-second graph: the line becomes one slow average across the entire picker, not a rate you can compare from point to point.

LogQL

Loki stores log streams. A LogQL query starts with a stream selector, which is mandatory. {job="nginx", env="prod"} chooses streams by labels. An empty {} is rejected. Prefer at least one equality matcher. A regex on every label is a scan.

Line filters come next, and they are cheap, because Loki can skip compressed chunks that cannot match. Put them before any parser.

{job="nginx"} |= "error" != "healthcheck" |~ "timeout|refused"

|= keeps lines containing the text. != drops them. |~ and !~ are regular expressions. Filters are case-sensitive. "Error" does not match "error".

Parsers turn a line into labels. | json reads a JSON object. | logfmt reads key=value pairs. | pattern extracts with a template. | regexp is the escape hatch and the slow one. After a parser, label filters can be numeric.

{job="api"} |= "\"status\":" | json | status >= 500

The line filter runs first, so a chunk with no "status": is skipped. status >= 500 works when the field is numeric. A string status wants status =~ "5..". Lines that are not JSON get an __error__ label. If you then build a metric, drop them or they pollute the count:

sum by (status) (
  count_over_time(
    {job="api"} | json | status >= 500 | __error__="" [5m]
  )
)

The range sits at the end of the pipeline, after the filters. rate and count_over_time turn the filtered lines into a number you can graph. count_over_time is a count of lines in the window. rate is that count per second. bytes_over_time and bytes_rate do the same for volume. Aggregations (sum by, avg) work as in PromQL, and the same "aggregate after the range function" rule applies.

unwrap reads a numeric field so you can take a quantile of a value that was buried in the line:

quantile_over_time(
  0.95,
  {job="api"}
    | json
    | unwrap duration_seconds
    | __error__="" [5m]
)

__error__="" here means "unwrap managed to parse a number". Without it, a line with a missing duration is an error sample and the quantile is wrong or empty.

Two cautions. Parsing before filtering makes Loki do the expensive work on every line. And a metric query that groups by a raw URL or a user id recreates the cardinality problem Prometheus already taught you, only the bill arrives as a slow Loki query rather than a series limit. Group by a small label: status, a normalised route, a service name.

Dashboard variables

A variable is a dropdown whose value Grafana pastes into the query before it runs. A query variable on a Prometheus source often uses label_values, which is Grafana syntax, not PromQL. It will not run inside a panel.

label_values(up, job)

That belongs in the variable's query field. The panel then refers to the selection. With multi-value enabled, do not trust the default interpolation. The default joins values in a way that is wrong inside =~. Use the regex format, which escapes metacharacters and joins with |:

up{job=~"${job:regex}"}

Because the matcher is anchored, a single selected value api matches only api. Several values become api|worker. The Include-all option should use a custom all value of .* if you want the matcher to mean "anything" when All is chosen. ${job:raw} does not escape, so a value containing a dot matches more than you think.

$__interval, $__rate_interval, and $__range are built in. They are not variables you create. $__from and $__to are the range edges in milliseconds, which SQL panels use more often than PromQL. An interval variable that you do create is a different thing: a list of steps the user picks. Name it something other than interval if you can, so nobody confuses it with $__interval.

Repeating a panel by a variable clones it once per value. That is right for a handful of jobs and wrong for hundreds of values.

Common mistakes

• rate on a gauge, or a raw counter graphed as if it were already a per-second rate. A counter's raw value mostly tells you how long the process has been up.
• sum(rate(...)) written as rate(sum(...)).
• histogram_quantile without le in the sum by, or with sum by (le) applied to a native histogram.
• A rate window shorter than two scrapes, usually because $__interval was used instead of $__rate_interval. The panel says "No data" and the query is otherwise fine.
• irate in an alert.
• A PromQL regex treated as unanchored. code=~"5" does not mean "starts with 5" and does not mean "contains 5".
• label_values pasted into a panel. It only works in a variable query.
• A multi-value variable expanded with $job instead of ${job:regex} inside =~.
• LogQL that parses every line and only then filters, or that groups a metric by a unique id.
• Stale series. Prometheus stops returning a series that has not been scraped for five minutes by default. An empty panel at a quiet time may be staleness. up == 0 is a failed scrape. A vanished series is the other one.

When a panel is wrong, open the query inspector and run the interpolated query in Explore. Several of the mistakes above are obvious the moment you see the string Grafana sent.