Prometheus & Grafana Complete Guide 2026: PromQL, Alerting, SLOs & Expert Interview Q&A

Prometheus is the de-facto standard for metrics-based monitoring in cloud-native environments. Combined with Grafana for visualisation and AlertManager for notification routing, it forms the observability backbone of almost every production Kubernetes deployment. I have built and operated Prometheus monitoring stacks for production EKS clusters for 4.5 years — writing PromQL queries for SLO dashboards, tuning alert thresholds to eliminate alert fatigue, and architecting Thanos for long-term multi-cluster metrics storage. This guide covers everything from core concepts to advanced SLO monitoring.

What is Prometheus? Architecture and Pull Model

Prometheus is an open-source monitoring system that collects metrics by pulling (scraping) HTTP endpoints exposed by targets. This is the opposite of push-based systems (StatsD, Graphite, Datadog Agent) where applications push metrics to a central collector. The pull model has a key operational advantage: Prometheus can detect whether a target is down (scrape fails) rather than just seeing silence (push stopped). If a service stops pushing metrics, a push-based system cannot distinguish between "service is healthy but not producing metrics" and "service is down."

Prometheus components: the Prometheus server scrapes targets on a configurable interval (default 15s), evaluates alerting and recording rules, and stores time-series data in its local TSDB (Time Series Database). Exporters expose metrics from systems that don't natively speak Prometheus: Node Exporter (Linux OS metrics), cAdvisor (container metrics), Blackbox Exporter (probing HTTP/TCP/DNS endpoints), and hundreds of community exporters. AlertManager receives alerts from Prometheus, deduplicates, groups, silences, and routes them to PagerDuty, OpsGenie, Slack, email, or webhooks. Pushgateway is the exception to the pull model: batch jobs and short-lived jobs push metrics to the gateway, which Prometheus then scrapes.

Prometheus Metric Types Explained

Prometheus has four metric types, each with specific use cases. Using the wrong type is one of the most common mistakes in custom instrumentation:

• Counter — A value that only ever increases (or resets to zero on restart). Ideal for: total HTTP requests, total errors, total bytes sent. Never use a counter for values that go up and down (like current active connections). In PromQL, always use rate() or increase() on counters — the raw counter value is rarely meaningful. Example: http_requests_total{status="500"}
• Gauge — A value that can go up and down. Ideal for: current memory usage, number of active goroutines, current queue depth, temperature. Unlike counters, gauge values are meaningful on their own. Example: node_memory_MemAvailable_bytes
• Histogram — Samples observations into configurable buckets and provides a count and sum. Used for measuring the distribution of values: request latency, response size. Creates three time series per metric: _bucket (one per configured boundary), _count (total observations), _sum (sum of all observed values). Allows calculating percentiles with histogram_quantile(). Example: http_request_duration_seconds_bucket{le="0.1"}
• Summary — Similar to Histogram but calculates quantiles on the client side. Less flexible than Histogram for PromQL aggregation (cannot aggregate quantiles across instances) but more accurate for high-cardinality quantile requirements. Prefer Histogram in almost all cases because server-side quantile calculation allows aggregation.

PromQL — The Query Language

PromQL is a functional query language that operates on time series data. Every interview at SRE level will test your PromQL knowledge. Here are the most important functions and patterns:

Label Matching and Aggregation

Recording Rules for Query Performance

Complex PromQL expressions evaluated on large datasets can be slow — especially when used in Grafana dashboards with multiple panels all running the same expensive query. Recording rules pre-compute expensive expressions at scrape time and store the result as a new time series. Dashboards then query the pre-computed series, which is extremely fast regardless of the underlying data volume.

Alerting Rules and AlertManager

Good alerting is one of the hardest problems in SRE. Too many alerts cause alert fatigue — on-call engineers start ignoring pages. Too few alerts mean outages go undetected. The key principle: alert on symptoms, not causes. Alert on high error rate (what users experience), not on "pod restarted" (a cause that may be self-healing).

AlertManager Configuration

Prometheus on Kubernetes — kube-prometheus-stack

The kube-prometheus-stack Helm chart is the standard way to deploy Prometheus, Grafana, AlertManager, Node Exporter, kube-state-metrics, and the Prometheus Operator on Kubernetes in a single installation. The Prometheus Operator extends Kubernetes with custom resources: ServiceMonitor (auto-discover services to scrape), PodMonitor (scrape individual pods), PrometheusRule (manage alerting and recording rules as Kubernetes resources), and Alertmanager (configure AlertManager declaratively).

SLOs, SLIs, and Error Budget Monitoring

Service Level Objectives (SLOs) are the cornerstone of SRE practice. An SLO is a target reliability commitment: "99.9% of requests will succeed over a 30-day rolling window." A Service Level Indicator (SLI) is the measurement: the actual success rate. The Error Budget is the amount of downtime or errors permitted before breaching the SLO: 99.9% SLO = 0.1% error budget = 43.8 minutes/month of allowed downtime.

Grafana Dashboards and Best Practices

Grafana is the standard visualisation layer for Prometheus. Effective Grafana dashboards follow specific design principles that separate useful operational dashboards from noisy, hard-to-read ones:

• The RED method for service dashboards: Rate (requests per second), Errors (error rate), Duration (latency percentiles). Every service should have a dashboard with these three panels at the top. They answer the first three questions during an incident: how much traffic, how many errors, how slow?
• The USE method for resource dashboards: Utilisation, Saturation, Errors. For every resource (CPU, memory, disk, network), show all three. A node that is 95% CPU utilised with a run queue of 3 is saturated and needs investigation.
• Use template variables for cluster, namespace, and service. A single dashboard template with dropdowns is far more maintainable than dozens of hardcoded dashboards. Define variables as Prometheus label queries: label_values(kube_pod_info, namespace).
• Stat panels for SLO status: A large coloured stat panel showing "99.94% (Target: 99.9%)" with green/red colouring based on threshold is the best way to communicate SLO health at a glance.
• Store dashboards as code: Use Grafonnet (Jsonnet library) or Grafana's provisioning mechanism to store dashboards in Git. Dashboards created manually in the UI are lost when the Grafana pod is restarted without persistence.

Thanos for Long-Term Storage and Multi-Cluster

Prometheus stores data locally with a default 15-day retention. Thanos extends Prometheus for: long-term storage (upload to S3/GCS/Azure Blob), global query (query across multiple Prometheus instances as if they were one), and high availability (deduplicate data from multiple Prometheus replicas). Thanos Sidecar runs alongside each Prometheus instance, uploads TSDB blocks to object storage, and exposes a gRPC endpoint. Thanos Query aggregates results from all Sidecar endpoints. Thanos Compactor downsamples old data for efficient long-term storage. This architecture provides unlimited retention without scaling Prometheus local storage.

Troubleshooting Common Prometheus Issues

• Target shows as DOWN in /targets: Check that the target's /metrics endpoint is accessible from the Prometheus pod's network. Verify the ServiceMonitor label selectors match the Service labels. Check security groups or Network Policies blocking the scrape port. Run kubectl port-forward svc/myapp 8080:8080 and test manually.
• MANY_TO_ONE match error in PromQL: This occurs when joining two time series with different label sets using on() and the right side has more unique label combinations than the left. Use group_left() or group_right() to specify which side to fan out.
• High cardinality causing OOM: Each unique combination of label values creates a separate time series. A label with 10,000 unique values (like user_id or request_id) creates 10,000 time series. This is a cardinality explosion. Never use high-cardinality values as Prometheus labels. Check TSDB head cardinality with prometheus_tsdb_head_series metric. Use the /api/v1/status/tsdb endpoint to find the highest-cardinality metrics.
• Alerts not routing to correct receiver: Use AlertManager's /api/v2/alerts endpoint to see active alerts and their labels. Check the route matching with the amtool command: amtool config routes test --config.file=alertmanager.yml severity=critical team=platform. AlertManager routes are matched top-down; the first match wins unless continue: true is set.

12 Prometheus & Observability Interview Questions with Expert Answers