Alert Rules

Create and manage alert rules in Grafana’s unified alerting system

Alert rules define the conditions that trigger alerts in Grafana. They combine data queries with expressions to evaluate alert conditions.

Rule Structure

An alert rule consists of several key components:

Complete Alert Rule

interface AlertRule {
  // Identity
  id: number;
  uid: string;        // Unique identifier
  orgID: number;
  
  // Definition
  title: string;
  condition: string;  // RefID of the condition expression
  data: AlertQuery[]; // Queries and expressions
  
  // Scheduling
  intervalSeconds: number;  // How often to evaluate
  for: Duration;            // Pending duration before firing
  keepFiringFor: Duration;  // Keep firing after resolved
  
  // Organization
  namespaceUID: string;  // Folder UID
  ruleGroup: string;     // Group name
  ruleGroupIndex: number;
  
  // Behavior
  noDataState: 'Alerting' | 'NoData' | 'OK' | 'KeepLast';
  execErrState: 'Alerting' | 'Error' | 'OK' | 'KeepLast';
  isPaused: boolean;
  
  // Metadata
  labels: Record<string, string>;
  annotations: Record<string, string>;
  
  // Notifications
  notificationSettings?: NotificationSettings;
  
  // Recording rule specific
  record?: Record;
}

interface AlertQuery {
  refID: string;              // Unique identifier for this query
  queryType: string;          // '' for datasource, 'expression' for expressions
  datasourceUID: string;      // Data source identifier
  model: object;              // Query-specific configuration
  relativeTimeRange: {
    from: number;             // Seconds from now
    to: number;
  };
}

Query and Expression Model

Alert rules use a query + expression model:

1. Data Source Queries: Fetch data from data sources (Prometheus, Loki, etc.)
2. Expressions: Transform and evaluate query results
3. Condition: The expression that determines alert state

Data Source Query Example

{
  "refID": "A",
  "queryType": "",
  "datasourceUID": "prometheus-uid",
  "model": {
    "expr": "up{job=\"grafana\"}",
    "refId": "A"
  },
  "relativeTimeRange": {
    "from": 600,
    "to": 0
  }
}

{
  "refID": "B",
  "queryType": "expression",
  "datasourceUID": "__expr__",
  "model": {
    "type": "reduce",
    "expression": "A",
    "reducer": "last",
    "settings": {
      "mode": "strict"
    }
  }
}

{
  "refID": "C",
  "queryType": "expression",
  "datasourceUID": "__expr__",
  "model": {
    "type": "threshold",
    "expression": "B",
    "conditions": [
      {
        "evaluator": {
          "type": "lt",
          "params": [1]
        }
      }
    ]
  }
}

State Handling

Pending Duration (for)

The for duration specifies how long a condition must be true before the alert fires:

// Alert only fires if condition is true for 5 minutes
for: '5m'

Keep Firing For (keepFiringFor)

Keeps the alert in a firing state even after the condition resolves:

// Keep firing for 10 minutes after resolution
keepFiringFor: '10m'

NoData State

Controls behavior when queries return no data:

Alerting

noDataState: 'Alerting'  // Fire alert when no data

noDataState: 'NoData'    // Create NoData state

noDataState: 'OK'        // Treat as normal

noDataState: 'KeepLast'  // Maintain previous state

Execution Error State

Controls behavior when query execution fails:

execErrState: 'Alerting' | 'Error' | 'OK' | 'KeepLast'

Labels and Annotations

Labels

Labels are used for:

• Routing alerts to contact points
• Grouping related alerts
• Filtering and searching

labels: {
  team: 'backend',
  severity: 'critical',
  service: 'api'
}

Reserved Labels - These labels are automatically managed:

• grafana_folder: Folder title
• __grafana_receiver__: Auto-routing receiver
• __grafana_route_settings_hash__: Notification settings identifier

Annotations

Annotations provide additional context:

annotations: {
  description: 'Service {{ $labels.service }} is down',
  runbook_url: 'https://wiki.example.com/runbooks/service-down',
  summary: 'Critical service outage',
  __dashboardUid__: 'abc123',  // Links to dashboard
  __panelId__: '2'              // Links to specific panel
}

Rule Groups

Rules are organized into groups that share an evaluation interval:

Rule Group Structure

type AlertRuleGroup struct {
    Title      string      // Group name
    FolderUID  string      // Parent folder
    Interval   int64       // Evaluation interval in seconds
    Rules      []AlertRule // Rules in this group
}

// Interval must be:
// 1. Divisible by base interval (default 10s)
// 2. Same for all rules in group
interval: 60  // Valid: 60s is divisible by 10s

Recording Rules

Recording rules pre-compute expensive queries and store results:

Recording Rule Model

type Record struct {
    // Metric name to write
    Metric string
    
    // Which query/expression provides the result
    From string
    
    // Data source to write to
    TargetDatasourceUID string
}

{
  title: 'job:http_requests:rate5m',
  record: {
    metric: 'job:http_requests:rate5m',
    from: 'B',  // RefID of expression
    targetDatasourceUID: 'prometheus-uid'
  },
  data: [
    {
      refID: 'A',
      datasourceUID: 'prometheus-uid',
      model: { expr: 'rate(http_requests_total[5m])' }
    },
    {
      refID: 'B',
      queryType: 'expression',
      model: {
        type: 'reduce',
        expression: 'A',
        reducer: 'sum',
        settings: { mode: 'replaceNonNumeric' }
      }
    }
  ]
}

Recording rules ignore alerting-specific fields:

• noDataState
• execErrState
• condition
• for
• keepFiringFor
• notificationSettings

Notification Settings

Rules can define inline notification routing:

Contact Point Routing

notificationSettings: {
  contactPointRouting: {
    receiver: 'ops-team',
    groupBy: ['alertname', 'severity'],
    groupWait: '30s',
    groupInterval: '5m',
    repeatInterval: '4h',
    muteTimeIntervals: ['business-hours']
  }
}

notificationSettings: {
  policyRouting: {
    policyUID: 'custom-policy-uid'
  }
}

Rule Validation

Rules must pass validation before being saved:

1. UID Validation

   Rule UID must be valid (alphanumeric, hyphens, underscores)

2. Query Validation

   At least one query or expression must be defined

3. Interval Validation

   Interval must be divisible by base interval (default 10s)

4. State Validation

   NoData and ExecErr states must be valid enum values

5. Namespace Validation

   Folder must exist and not be managed by Git Sync

6. Label Validation

   Reserved labels cannot be manually set

Backend API Endpoints

Get Alert Rule

// Location: pkg/services/ngalert/store
func (st *DBstore) GetAlertRuleByUID(
    ctx context.Context,
    query *models.GetAlertRuleByUIDQuery,
) (*models.AlertRule, error)

func (st *DBstore) ListAlertRules(
    ctx context.Context,
    query *models.ListAlertRulesQuery,
) (models.RulesGroup, error)

func (st *DBstore) UpdateRuleGroup(
    ctx context.Context,
    orgID int64,
    namespaceUID string,
    ruleGroup string,
    rules []*models.AlertRule,
) error

Common Patterns

Dashboard Panel Alerts

Link an alert to a dashboard panel:

annotations: {
  __dashboardUid__: 'dashboard-uid',
  __panelId__: '5'
}

Multi-Condition Alerts

Combine multiple conditions:

data: [
  { refID: 'A', /* CPU query */ },
  { refID: 'B', /* Memory query */ },
  { 
    refID: 'C',
    queryType: 'expression',
    model: {
      type: 'math',
      expression: '$A > 80 || $B > 90'
    }
  }
],
condition: 'C'

Related Resources

Overview

Understand the alerting architecture

Contact Points

Configure notification destinations

Notification Policies

Route alerts to contact points

Silences

Temporarily mute notifications