Alert Rules

Create and manage alert rules in Grafana’s unified alerting system

Alert rules define the conditions that trigger alerts in Grafana. They combine data queries with expressions to evaluate alert conditions.

Rule Structure

An alert rule consists of several key components:

Complete Alert Rule

interface AlertRule {
  // Identity
  id: number;
  uid: string;        // Unique identifier
  orgID: number;
  
  // Definition
  title: string;
  condition: string;  // RefID of the condition expression
  data: AlertQuery[]; // Queries and expressions
  
  // Scheduling
  intervalSeconds: number;  // How often to evaluate
  for: Duration;            // Pending duration before firing
  keepFiringFor: Duration;  // Keep firing after resolved
  
  // Organization
  namespaceUID: string;  // Folder UID
  ruleGroup: string;     // Group name
  ruleGroupIndex: number;
  
  // Behavior
  noDataState: 'Alerting' | 'NoData' | 'OK' | 'KeepLast';
  execErrState: 'Alerting' | 'Error' | 'OK' | 'KeepLast';
  isPaused: boolean;
  
  // Metadata
  labels: Record<string, string>;
  annotations: Record<string, string>;
  
  // Notifications
  notificationSettings?: NotificationSettings;
  
  // Recording rule specific
  record?: Record;
}

Query and Expression Model

Alert rules use a query + expression model:

1. Data Source Queries: Fetch data from data sources (Prometheus, Loki, etc.)
2. Expressions: Transform and evaluate query results
3. Condition: The expression that determines alert state

Data Source Query Example

{
  "refID": "A",
  "queryType": "",
  "datasourceUID": "prometheus-uid",
  "model": {
    "expr": "up{job=\"grafana\"}",
    "refId": "A"
  },
  "relativeTimeRange": {
    "from": 600,
    "to": 0
  }
}

State Handling

Pending Duration (for)

The for duration specifies how long a condition must be true before the alert fires:

// Alert only fires if condition is true for 5 minutes
for: '5m'

Keep Firing For (keepFiringFor)

Keeps the alert in a firing state even after the condition resolves:

// Keep firing for 10 minutes after resolution
keepFiringFor: '10m'

NoData State

Controls behavior when queries return no data:

Alerting

noDataState: 'Alerting'  // Fire alert when no data

Execution Error State

Controls behavior when query execution fails:

execErrState: 'Alerting' | 'Error' | 'OK' | 'KeepLast'

Labels and Annotations

Labels

Labels are used for:

• Routing alerts to contact points
• Grouping related alerts
• Filtering and searching

labels: {
  team: 'backend',
  severity: 'critical',
  service: 'api'
}

Reserved Labels - These labels are automatically managed:

• grafana_folder: Folder title
• __grafana_receiver__: Auto-routing receiver
• __grafana_route_settings_hash__: Notification settings identifier

Annotations

Annotations provide additional context:

annotations: {
  description: 'Service {{ $labels.service }} is down',
  runbook_url: 'https://wiki.example.com/runbooks/service-down',
  summary: 'Critical service outage',
  __dashboardUid__: 'abc123',  // Links to dashboard
  __panelId__: '2'              // Links to specific panel
}

Rule Groups

Rules are organized into groups that share an evaluation interval:

Rule Group Structure

type AlertRuleGroup struct {
    Title      string      // Group name
    FolderUID  string      // Parent folder
    Interval   int64       // Evaluation interval in seconds
    Rules      []AlertRule // Rules in this group
}

Recording Rules

Recording rules pre-compute expensive queries and store results:

Recording Rule Model

type Record struct {
    // Metric name to write
    Metric string
    
    // Which query/expression provides the result
    From string
    
    // Data source to write to
    TargetDatasourceUID string
}

Recording rules ignore alerting-specific fields:

• noDataState
• execErrState
• condition
• for
• keepFiringFor
• notificationSettings

Notification Settings

Rules can define inline notification routing:

Contact Point Routing

notificationSettings: {
  contactPointRouting: {
    receiver: 'ops-team',
    groupBy: ['alertname', 'severity'],
    groupWait: '30s',
    groupInterval: '5m',
    repeatInterval: '4h',
    muteTimeIntervals: ['business-hours']
  }
}

Rule Validation

Rules must pass validation before being saved:

1. UID Validation

   Rule UID must be valid (alphanumeric, hyphens, underscores)

2. Query Validation

   At least one query or expression must be defined

3. Interval Validation

   Interval must be divisible by base interval (default 10s)

4. State Validation

   NoData and ExecErr states must be valid enum values

5. Namespace Validation

   Folder must exist and not be managed by Git Sync

6. Label Validation

   Reserved labels cannot be manually set

Backend API Endpoints

Get Alert Rule

// Location: pkg/services/ngalert/store
func (st *DBstore) GetAlertRuleByUID(
    ctx context.Context,
    query *models.GetAlertRuleByUIDQuery,
) (*models.AlertRule, error)

Common Patterns

Dashboard Panel Alerts

Link an alert to a dashboard panel:

annotations: {
  __dashboardUid__: 'dashboard-uid',
  __panelId__: '5'
}

Multi-Condition Alerts

Combine multiple conditions:

data: [
  { refID: 'A', /* CPU query */ },
  { refID: 'B', /* Memory query */ },
  { 
    refID: 'C',
    queryType: 'expression',
    model: {
      type: 'math',
      expression: '$A > 80 || $B > 90'
    }
  }
],
condition: 'C'

Related Resources

Overview

Understand the alerting architecture

Contact Points

Configure notification destinations

Notification Policies

Route alerts to contact points

Silences

Temporarily mute notifications