How to Set Up Kubernetes Monitoring with Alertmanager

There’s a particular kind of dread that comes from finding out about an outage from a customer instead of a pager. Early on, I had Prometheus scraping metrics beautifully but nothing actually notifying anyone when something broke — dashboards full of red graphs that nobody was looking at in real time. Alertmanager is the piece that turns “the data exists” into “someone actually gets told.”

What Alertmanager Does

Prometheus evaluates alerting rules and fires alerts, but Prometheus itself doesn’t handle notification delivery, deduplication, grouping, or silencing. Alertmanager sits downstream of Prometheus and handles:

• Deduplication — the same underlying issue triggering many alerts becomes one notification.
• Grouping — related alerts (e.g., all pods in a crashing Deployment) get bundled into a single notification instead of spamming your Slack channel.
• Routing — different alerts go to different teams/channels based on labels.
• Silencing — temporarily mute known issues (e.g., during planned maintenance).
• Inhibition — suppress lower-priority alerts when a related higher-priority alert is already firing.

Architecture

Prometheus evaluates rules on its own schedule → matching alerts are sent to Alertmanager’s API → Alertmanager groups and deduplicates → routes through a tree of matchers to the correct receiver (Slack, PagerDuty, email, webhook) → applies any active silences or inhibition rules before actually sending.

Prerequisites

• A running Prometheus instance in-cluster (via kube-prometheus-stack or standalone)
• Helm installed
• Notification channel credentials (Slack webhook URL, PagerDuty integration key, etc.)

Step 1: Install via kube-prometheus-stack

If you’re not already running it, this Helm chart bundles Prometheus, Alertmanager, and Grafana together:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# values.yaml
alertmanager:
  alertmanagerSpec:
    replicas: 3
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi

helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  -f values.yaml

replicas: 3 matters — Alertmanager supports clustering for high availability, and running a single replica means a lost pod is a lost alerting pipeline exactly when you need it most.

Verify:

kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager

NAME                                          READY   STATUS    RESTARTS   AGE
alertmanager-kube-prometheus-alertmanager-0   2/2     Running   0          1m
alertmanager-kube-prometheus-alertmanager-1   2/2     Running   0          1m
alertmanager-kube-prometheus-alertmanager-2   2/2     Running   0          1m

Step 2: Configure Alertmanager Routing

With the Prometheus Operator, configuration is managed via the AlertmanagerConfig CRD or a raw Secret. Here’s a full config with routing to Slack for warnings and PagerDuty for critical alerts:

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-kube-prometheus-alertmanager
  namespace: monitoring
type: Opaque
stringData:
  alertmanager.yaml: |
    global:
      resolve_timeout: 5m
      slack_api_url: 'https://hooks.slack.com/services/XXXX/YYYY/ZZZZ'

    route:
      receiver: 'slack-default'
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
      - match:
          severity: critical
        receiver: 'pagerduty-critical'
        continue: true
      - match:
          severity: warning
        receiver: 'slack-warnings'

    receivers:
    - name: 'slack-default'
      slack_configs:
      - channel: '#alerts-general'
        send_resolved: true
        title: '{{ .CommonAnnotations.summary }}'
        text: '{{ .CommonAnnotations.description }}'

    - name: 'slack-warnings'
      slack_configs:
      - channel: '#alerts-warnings'
        send_resolved: true

    - name: 'pagerduty-critical'
      pagerduty_configs:
      - routing_key: '<PAGERDUTY_INTEGRATION_KEY>'
        send_resolved: true

    inhibit_rules:
    - source_match:
        severity: 'critical'
      target_match:
        severity: 'warning'
      equal: ['alertname', 'namespace']

kubectl apply -f alertmanager-config-secret.yaml

The inhibit_rules block is worth understanding closely: if a critical alert is already firing for a given alertname/namespace combination, matching warning-level alerts for the same combination are suppressed — no point paging someone about a warning when the critical version of the same issue already woke them up.

Step 3: Using AlertmanagerConfig CRD (Namespaced, GitOps-friendly)

For teams managing their own alert routing without touching the global Secret, the Prometheus Operator’s AlertmanagerConfig CRD is cleaner:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: team-payments-alerts
  namespace: payments
spec:
  route:
    receiver: 'payments-slack'
    groupBy: ['alertname']
    groupWait: 30s
    repeatInterval: 4h
  receivers:
  - name: 'payments-slack'
    slackConfigs:
    - apiURL:
        name: slack-webhook-secret
        key: url
      channel: '#payments-alerts'
      sendResolved: true

kubectl create secret generic slack-webhook-secret \
  --from-literal=url='https://hooks.slack.com/services/XXXX/YYYY/ZZZZ' \
  -n payments
kubectl apply -f payments-alertmanagerconfig.yaml

This lets each team own their alert routing in their own namespace, version-controlled alongside their application manifests.

Step 4: Define PrometheusRules

Alertmanager only routes alerts that Prometheus fires — you need PrometheusRule objects defining the actual conditions:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-health-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
  - name: pod-health
    rules:
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} is crash looping"
        description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted {{ $value }} times in 15m."

    - alert: PodNotReady
      expr: sum by (namespace, pod) (kube_pod_status_ready{condition="false"}) > 0
      for: 15m
      labels:
        severity: critical
      annotations:
        summary: "Pod {{ $labels.pod }} not ready"
        description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been not-ready for 15 minutes."

    - alert: HighMemoryUsage
      expr: (container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.9
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High memory usage on {{ $labels.pod }}"
        description: "Container {{ $labels.container }} is using {{ $value | humanizePercentage }} of its memory limit."

kubectl apply -f pod-health-alerts.yaml

The labels: release: kube-prometheus on the PrometheusRule is critical — it needs to match the ruleSelector the Helm chart configured for Prometheus to actually pick up your rule.

Step 5: Verify in the Alertmanager UI

kubectl port-forward -n monitoring svc/alertmanager-operated 9093:9093

Open localhost:9093 to see active alerts, silences, and the routing tree.

Silencing Alerts

During planned maintenance, silence alerts via CLI (amtool) or the UI:

amtool silence add alertname="PodCrashLooping" namespace="payments" \
  --duration="2h" \
  --comment="Planned maintenance window" \
  --alertmanager.url=http://localhost:9093

amtool silence query --alertmanager.url=http://localhost:9093

Production High Availability

Run Alertmanager with at least 3 replicas — the Prometheus Operator automatically configures the mesh/gossip protocol between them via --cluster.peer flags, so all replicas share notification state and avoid duplicate pages even if multiple Prometheus instances send the same alert.

kubectl get statefulset -n monitoring alertmanager-kube-prometheus-alertmanager -o yaml | grep replicas

Templating Notifications for Clarity

Default Alertmanager notification templates are functional but generic. Custom templates make pages far more useful when someone’s reading them half-asleep at 3 AM:

templates:
  - '/etc/alertmanager/templates/*.tmpl'

{{ define "slack.custom.text" }}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Namespace:* {{ .Labels.namespace }}
*Summary:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Started:* {{ .StartsAt }}
{{ end }}
{{ end }}

receivers:
- name: 'slack-warnings'
  slack_configs:
  - channel: '#alerts-warnings'
    text: '{{ template "slack.custom.text" . }}'

Consistent templates across every receiver mean whoever’s on call doesn’t have to parse a different format depending on which alert fired — every notification reads the same way, with the same fields, in the same order.

Handling Alert Fatigue at Scale

As a cluster and its alert rules grow, the biggest operational risk isn’t missing alerts — it’s people tuning them out because too many fire. A few practical mitigations beyond routing and inhibition:

• Time-based routing — route low-severity alerts to a muted channel outside business hours, reserving pages for genuinely critical conditions overnight.
• Escalation chains — configure PagerDuty (or equivalent) to escalate to a secondary on-call if the primary doesn’t acknowledge within a set window, rather than relying on Alertmanager’s repeat_interval alone.
• Regular alert audits — periodically review which alerts fired in the last quarter and how often they were actionable versus noise; delete or retune anything with a poor signal-to-noise ratio.

route:
  routes:
  - match:
      severity: warning
    receiver: 'slack-warnings'
    active_time_intervals:
    - business-hours
time_intervals:
- name: business-hours
  time_intervals:
  - weekdays: ['monday:friday']
    times:
    - start_time: '09:00'
      end_time: '18:00'

active_time_intervals restricts a route to only deliver during the specified window — warning-level noise doesn’t need to wake anyone up outside working hours, while critical routes typically skip this restriction entirely.

Common Mistakes

• Running a single Alertmanager replica — turns your alerting pipeline itself into a single point of failure.
• No group_by, causing alert storms — dozens of individual Slack messages instead of one grouped notification during a real incident.
• Missing resolve_timeout tuning, leading to alerts that never clear from dashboards even after the underlying issue resolves.
• Overly broad routing (match_re: severity: .*), sending every alert to every channel regardless of relevance, which trains people to ignore the channel entirely.
• Not setting send_resolved: true, leaving teams unsure whether an issue is still ongoing.
• Forgetting PrometheusRule label selectors don’t match the Prometheus instance’s ruleSelector, so rules silently never get loaded.

Best Practices

• Route by severity label at minimum; add team labels for larger orgs so ownership is explicit.
• Use inhibit_rules to avoid redundant paging for cascading failures.
• Keep repeat_interval reasonable (4-12h) — too short causes alert fatigue, too long means a forgotten firing alert stays silent for a long time.
• Test routing changes with amtool config routes test before applying to production.

Summary

Alertmanager is what turns Prometheus’s raw alerting rules into actionable, deduplicated, correctly-routed notifications — without it, you have data but no signal. Deploy it with multiple replicas for HA, define routing trees by severity and team, back alerts with well-tuned PrometheusRule objects, and use inhibition rules to keep noise down during real incidents. Get this right and the next outage gets caught by your team, not your customers.

References

• Prometheus Alertmanager Documentation
• Prometheus Operator: AlertmanagerConfig
• kube-prometheus-stack Helm Chart
• CNCF Prometheus Project