Prometheus Data Source

Query metrics from Prometheus using PromQL in Grafana

Prometheus Data Source

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. Grafana provides native support for querying Prometheus metrics using PromQL (Prometheus Query Language).

Overview

The Prometheus data source allows you to:

• Query time series metrics with PromQL
• Visualize instant vectors and range vectors
• Use metric metadata for query building
• Create alerting rules based on Prometheus queries
• Explore exemplars and tracing integration

Source: public/app/plugins/datasource/prometheus/ and packages/grafana-prometheus/

Configuration

Connection Settings

1. Add Data Source

   Navigate to Configuration > Data Sources > Add data source > Prometheus

2. Configure URL

   Set the Prometheus server URL:

   http://localhost:9090
   

   Use the proxy access mode (default) for server-side queries.

3. Authentication

   Configure authentication if required:

   • Basic authentication
   • TLS client certificate
   • OAuth passthrough
   • Azure authentication
4. Additional Settings

   Configure optional settings:

   • Scrape interval
   • Query timeout
   • HTTP method (GET/POST)

Data Source Options

timeInterval · string · default: ""

Minimum scrape interval (e.g., 30s, 1m). Used to calculate $__interval and $__rate_interval.

queryTimeout · string · default: ""

Maximum time to wait for a query response (e.g., 60s).

httpMethod · string · default: "POST"

HTTP method for queries. POST is recommended for long queries.

disableMetricsLookup · boolean · default: "false"

Disable metric name lookup for improved performance with large metric counts.

prometheusType · string

Prometheus implementation type: Prometheus, Cortex, Mimir, or Thanos.

cacheLevel · string · default: "Low"

Query result cache level: None, Low, Medium, or High.

Query Editor

The Prometheus query editor supports two modes:

Code Mode

Write PromQL queries directly:

rate(http_requests_total{job="api-server"}[5m])

Features:

• Syntax highlighting with Monaco editor
• Auto-completion for metrics, labels, and functions
• Query validation
• Metric browser

Builder Mode

Visual query builder for constructing PromQL queries:

1. Select metric from dropdown
2. Add label filters
3. Choose operations (rate, sum, etc.)
4. Configure range and step

Builder mode generates valid PromQL expressions.

PromQL Examples

Instant Query

Get current value of a metric:

up{job="prometheus"}

Range Query with Rate

Calculate per-second rate over 5 minutes:

rate(http_request_total[5m])

Source: packages/grafana-prometheus/src/components/PromCheatSheet.tsx:13-16

Percentile Calculation

Calculate 95th percentile of request latencies:

histogram_quantile(0.95, sum(rate(prometheus_http_request_duration_seconds_bucket[5m])) by (le))

Source: packages/grafana-prometheus/src/components/PromCheatSheet.tsx:18-21

Aggregation

Sum values and group by label:

sum(rate(http_requests_total[5m])) by (status_code)

Alert Tracking

Sum alerts firing over 24 hours:

sort_desc(sum(sum_over_time(ALERTS{alertstate="firing"}[24h])) by (alertname))

Source: packages/grafana-prometheus/src/components/PromCheatSheet.tsx:23-26

Query Options

Legend Format

Customize series names using label values:

{{instance}} - {{job}}

Output: localhost:9090 - prometheus

Min Step

Defines the graph resolution using duration format:

• 15s - high resolution, may be slow over large time ranges
• 1m - medium resolution
• 5m - low resolution, faster queries

If no step is specified, resolution is calculated automatically based on panel width and time range.

Source: packages/grafana-prometheus/src/components/PromCheatSheet.tsx:28-31

Resolution

Control data point density:

• 1/1 - One data point per pixel
• 1/2 - One data point per 2 pixels (faster)
• 1/10 - Sparse data points

Template Variables

Label Values Query

Populate variable from label values:

label_values(up, instance)

Returns all unique values of the instance label.

Metric Names Query

List all metrics matching a pattern:

metrics(http_.*)

Query Result Variable

Use query results as variable values:

query_result(sum(up) by (job))

Using Variables in Queries

rate(http_requests_total{instance="$instance", job="$job"}[5m])

Exemplars

Exemplars link metrics to trace IDs for deeper analysis:

1. Configure Trace ID

   In data source settings, configure exemplar trace ID destinations:

   jsonData:
     exemplarTraceIdDestinations:
       - name: traceID
         datasourceUid: tempo-uid
   
2. Query with Exemplars

   Enable “Exemplars” in query options. Grafana displays exemplar data points on the graph.

3. Navigate to Traces

   Click on exemplar points to jump to corresponding traces in Tempo or Jaeger.

Advanced Features

Recording Rules

Query pre-computed recording rules for faster dashboards:

job:http_requests_total:rate5m

Recording rules must be defined in Prometheus configuration. They don’t appear in metric autocomplete by default.

Incremental Querying

Enable incremental querying for live dashboards:

jsonData:
  incrementalQuerying: true
  incrementalQueryOverlapWindow: "10m"

Grafana only queries new data since the last refresh, reducing load.

Custom Query Parameters

Add custom parameters to all Prometheus requests:

jsonData:
  customQueryParameters: "timeout=30s&lookback_delta=5m"

Troubleshooting

No data in graph

• Verify time range includes data points
• Check metric exists: up{job="your-job"}
• Confirm Prometheus is scraping the target
• Review Prometheus logs for scrape errors

Query timeout

• Reduce time range
• Increase queryTimeout in data source settings
• Use recording rules for expensive queries
• Add more specific label filters to reduce cardinality

High cardinality warnings

• Avoid labels with unbounded values (IDs, timestamps)
• Use recording rules to pre-aggregate
• Enable disableMetricsLookup for better UI performance
• Configure seriesLimit to prevent overload

Metric autocomplete not working

• Check network connectivity to Prometheus
• Verify /api/v1/label/__name__/values endpoint is accessible
• Try enabling disableMetricsLookup and use manual entry

Best Practices

Use Rate for Counters

Always use rate() or irate() for counter metrics:

rate(http_requests_total[5m])

Avoid Joins

PromQL joins can be expensive. Use recording rules for complex joins.

Label Matchers

Be specific with label matchers to reduce query scope:

{job="api",environment="prod"}

Range Selection

Choose appropriate range intervals:

• [1m] for real-time metrics
• [5m] for general dashboards
• [1h] for long-term trends

Further Reading

• Prometheus Official Documentation
• PromQL Query Language
• Grafana Alerting with Prometheus