Grafana Basics: Dashboards, Panels, and Alerts

Grafana visualizes metrics. Connect Prometheus, query data, build dashboards.

What is Grafana?

Prometheus (time-series database)
  ↓ stores metrics
  ↓ cpu_usage, memory_bytes, requests_total
  │
  ↓ (Grafana queries)
  │
Grafana (visualization)
  ├─ Dashboards
  ├─ Panels (graphs, gauges, tables)
  └─ Alerts

Grafana shows "what's happening now" with charts, numbers, and alerts.

Installation

Docker Compose

services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin  # Change this!
    volumes:
      - grafana_data:/var/lib/grafana

volumes:
  grafana_data:

docker-compose up -d grafana

# Access
# http://localhost:3000
# Login: admin / admin

Initial Setup

Open http://localhost:3000
Login: admin / admin
Change password: (recommended)
Add data source: Left menu → Configuration → Data Sources

Data Sources

A data source is where Grafana gets metrics from.

Add Prometheus

Left menu → Configuration → Data Sources
Click "Add data source"
Type: Prometheus
URL: http://prometheus:9090
Click "Save & Test"

Expected: "Data source is working"

Add Other Sources

Source	URL	Use
Prometheus	http://localhost:9090	Time-series metrics
Loki	http://localhost:3100	Logs
InfluxDB	http://localhost:8086	Metrics
PostgreSQL	postgresql://user:pass@host/db	Relational data

Dashboards

A dashboard is a collection of panels.

Create Dashboard

Left menu → Dashboards → New → New Dashboard
Click "+ Add Panel"
Configure panel (see next section)
Click "Save dashboard"
Name: "My Stack Overview"

Pre-built Dashboards

Import from community (much faster than building).

Left menu → Dashboards → Browse
Search "Prometheus" or "Node Exporter"
Click a dashboard
"Import" button
Select your Prometheus data source

Popular imports:

Node Exporter (CPU, memory, disk)
Prometheus (internal metrics)
Docker (container stats)

Panels

A panel is a single visualization.

Panel Types

Type	Use	Example
Graph	Time series (line chart)	CPU usage over time
Stat	Single number	Current memory usage
Gauge	Circular meter	75% capacity
Bar Gauge	Horizontal bar	Service health
Table	Tabular data	Top 10 slowest queries
Heatmap	Dense 2D matrix	Request latency distribution
Pie Chart	Proportional	Error type breakdown
Stat List	Multiple numbers	Services status

Create a Graph Panel

Click "+ Add Panel" (in dashboard)
Panel Type: Graph
Title: "CPU Usage"

Query (Prometheus):

100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Y-Axis Label: "Percent (%)"
Save panel

Create a Stat Panel

"+ Add Panel"
Panel Type: Stat
Title: "Memory Used"

Query:

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

Unit: Bytes (SI)
Thresholds: Green 0-10GB, Yellow 10-15GB, Red 15GB+
Save

Create a Table Panel

"+ Add Panel"
Panel Type: Table
Title: "Top Errors"

Query:

topk(5, sum by (error_type) (rate(errors_total[5m])))

Columns: Customize which to show
Save

Queries (PromQL)

PromQL queries fetch metrics from Prometheus.

Basic Queries

# Current value
node_cpu_cores                    # Latest value

# Rate (change per second)
rate(requests_total[5m])          # Requests/sec (5-min average)

# Sum
sum(memory_bytes)                 # Total memory across all nodes

# Average
avg(response_time_seconds)        # Average response time

# Percentile (latency)
histogram_quantile(0.95, latency_bucket)  # 95th percentile

# Filter by label
requests_total{status="500"}      # Only 500 errors

# Combine filters (AND)
requests_total{status="500", service="api"}

# Multiple filters (OR)
requests_total{status=~"5.."}     # 500-599 errors

Advanced Queries

# Range vector (over time)
increase(errors_total[1h])        # Errors over last hour

# Instant vector (point in time)
memory_bytes offset 5m            # Value 5 mins ago

# Math operations
(used_memory / total_memory) * 100  # Percent

# Aggregation operators
sum by (service) (requests_total)  # Sum per service

# Top/Bottom
topk(3, memory_bytes)             # 3 services with most memory

# Without
count without (instance) (up)     # Count unique services

Variables & Templating

Make dashboards dynamic.

Create Variable

Dashboard → Dashboard Settings (gear icon)
Variables → New Variable
Name: host
Type: Query
Data source: Prometheus
Query: label_values(up, instance)
Save

Use Variable in Panel

In query, reference variable:

node_cpu_usage{instance="$host"}  # $host replaced with selection

Now dashboard has dropdown to select host.

Alerts

Alert when metric crosses threshold.

Create Alert (Classic Condition)

Panel → Edit → Alert tab
Alert name: "High CPU"
Condition: avg() of B > 80
For: 5m (trigger if true for 5 minutes)
Message: "CPU usage above 80%"
Save

Alert Notification Channels

Alerting → Notification channels
New notification channel
Type: Email / Slack / PagerDuty
Configure: Email address, Slack webhook, etc.
Send test notification

Link Alert to Channel

Edit alert (from above)
Send to: Select notification channel
Save

Now when alert fires, notification sent.

Dashboard Layout

Best Practices

Group related panels

[ Server Health ] [ Memory ] [ CPU ]
[ Disk Usage  ] [ Network ]
[ Errors      ] [ Latency ]

Time range at top
- Dashboard settings → Default time range: Last 24h
- User can override with top-right picker
Legend & tooltip
- Show metric names
- Display values on hover
Refresh rate
- Real-time: 5-10 seconds
- Daily stats: 1 hour
- Don't refresh too fast (wastes resources)

Export Dashboard

Dashboard → Share (top-right)
Export → Download JSON
Share JSON file with team
Another instance: Dashboards → Import → Upload JSON

Scripted Dashboards

Generate panels programmatically (advanced).

// dashboard.js
var dashboard = {
  title: "Auto-Generated",
  panels: [
    {
      title: "CPU",
      targets: [{expr: "cpu_usage"}],
      type: "graph"
    },
    {
      title: "Memory",
      targets: [{expr: "memory_usage"}],
      type: "stat"
    }
  ]
};

// Access: http://localhost:3000/dashboard/script/dashboard.js

Common Issues

"No data" in panel:

Check data source is connected (Configuration → Data Sources → Test)
Query is returning values (test in Prometheus Graph tab)
Time range includes data (not querying 1 year old data)

Panel slow to load:

Reduce time range (e.g., Last 24h instead of 90d)
Reduce query complexity
Increase refresh interval (e.g., 30s instead of 5s)

Alerts not firing:

Verify data is flowing (check panel manually)
Check alert condition (sometimes off by 1)
Verify notification channel is configured
Check logs: Alerting → Alert Rules

Memory usage high:

Reduce retention in Prometheus (30 days instead of 1 year)
Reduce scrape frequency (30s instead of 15s)
Disable unused dashboards/panels

Advanced Topics

Annotations

Mark events on dashboard.

// API call
POST /api/annotations
{
  "dashboardId": 1,
  "time": 1614556800,
  "text": "Deployed v1.5",
  "tags": ["deploy"]
}

Graph shows vertical line with event.

Thresholds & Color Zones

Panels change color based on value:

Green: 0-50
Yellow: 50-80
Red: 80+

Set in panel settings.

Stat Panel Examples

Metric: Memory Available
Thresholds: 0, 5GB, 10GB
Colors: Red, Yellow, Green
  Result:
  ┌──────────────────┐
  │     7.2 GB       │ (Yellow)
  └──────────────────┘

Performance Tips

Sample at intervals

# Instead of all data points:
metrics[5m]

# Use rate() which automatically averages:
rate(metrics[5m])

Use recording rules in Prometheus
- Pre-compute expensive queries
- Store as new metric
- Query pre-computed metric in Grafana
Cache dashboard JSON
- Save as JSON, version in git
- Redeploy via API
Monitor Grafana itself
- CPU, memory, query latency
- Self-healing (restart if stuck)