Offensive Security Researcher & Quantum Cryptography Analyst

How to Set Up Kubernetes Monitoring with Alertmanager

Abdul Wahab Junaid
How to Set Up Kubernetes Monitoring with Alertmanager

There’s a particular kind of dread that comes from finding out about an outage from a customer instead of a pager. Early on, I had Prometheus scraping metrics beautifully but nothing actually notifying anyone when something broke — dashboards full of red graphs that nobody was looking at in real time. Alertmanager is the piece that turns “the data exists” into “someone actually gets told.”

What Alertmanager Does

Prometheus evaluates alerting rules and fires alerts, but Prometheus itself doesn’t handle notification delivery, deduplication, grouping, or silencing. Alertmanager sits downstream of Prometheus and handles:

Architecture

Prometheus evaluates rules on its own schedule → matching alerts are sent to Alertmanager’s API → Alertmanager groups and deduplicates → routes through a tree of matchers to the correct receiver (Slack, PagerDuty, email, webhook) → applies any active silences or inhibition rules before actually sending.

Prerequisites

Step 1: Install via kube-prometheus-stack

If you’re not already running it, this Helm chart bundles Prometheus, Alertmanager, and Grafana together:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# values.yaml
alertmanager:
  alertmanagerSpec:
    replicas: 3
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi

helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  -f values.yaml

replicas: 3 matters — Alertmanager supports clustering for high availability, and running a single replica means a lost pod is a lost alerting pipeline exactly when you need it most.

Verify:

kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager

NAME                                          READY   STATUS    RESTARTS   AGE
alertmanager-kube-prometheus-alertmanager-0   2/2     Running   0          1m
alertmanager-kube-prometheus-alertmanager-1   2/2     Running   0          1m
alertmanager-kube-prometheus-alertmanager-2   2/2     Running   0          1m

Step 2: Configure Alertmanager Routing

With the Prometheus Operator, configuration is managed via the AlertmanagerConfig CRD or a raw Secret. Here’s a full config with routing to Slack for warnings and PagerDuty for critical alerts:

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-kube-prometheus-alertmanager
  namespace: monitoring
type: Opaque
stringData:
  alertmanager.yaml: |
    global:
      resolve_timeout: 5m
      slack_api_url: 'https://hooks.slack.com/services/XXXX/YYYY/ZZZZ'

    route:
      receiver: 'slack-default'
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
      - match:
          severity: critical
        receiver: 'pagerduty-critical'
        continue: true
      - match:
          severity: warning
        receiver: 'slack-warnings'

    receivers:
    - name: 'slack-default'
      slack_configs:
      - channel: '#alerts-general'
        send_resolved: true
        title: '{{ .CommonAnnotations.summary }}'
        text: '{{ .CommonAnnotations.description }}'

    - name: 'slack-warnings'
      slack_configs:
      - channel: '#alerts-warnings'
        send_resolved: true

    - name: 'pagerduty-critical'
      pagerduty_configs:
      - routing_key: '<PAGERDUTY_INTEGRATION_KEY>'
        send_resolved: true

    inhibit_rules:
    - source_match:
        severity: 'critical'
      target_match:
        severity: 'warning'
      equal: ['alertname', 'namespace']

kubectl apply -f alertmanager-config-secret.yaml

The inhibit_rules block is worth understanding closely: if a critical alert is already firing for a given alertname/namespace combination, matching warning-level alerts for the same combination are suppressed — no point paging someone about a warning when the critical version of the same issue already woke them up.

Step 3: Using AlertmanagerConfig CRD (Namespaced, GitOps-friendly)

For teams managing their own alert routing without touching the global Secret, the Prometheus Operator’s AlertmanagerConfig CRD is cleaner:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: team-payments-alerts
  namespace: payments
spec:
  route:
    receiver: 'payments-slack'
    groupBy: ['alertname']
    groupWait: 30s
    repeatInterval: 4h
  receivers:
  - name: 'payments-slack'
    slackConfigs:
    - apiURL:
        name: slack-webhook-secret
        key: url
      channel: '#payments-alerts'
      sendResolved: true

kubectl create secret generic slack-webhook-secret \
  --from-literal=url='https://hooks.slack.com/services/XXXX/YYYY/ZZZZ' \
  -n payments
kubectl apply -f payments-alertmanagerconfig.yaml

This lets each team own their alert routing in their own namespace, version-controlled alongside their application manifests.

Step 4: Define PrometheusRules

Alertmanager only routes alerts that Prometheus fires — you need PrometheusRule objects defining the actual conditions:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-health-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
  - name: pod-health
    rules:
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} is crash looping"
        description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted {{ $value }} times in 15m."

    - alert: PodNotReady
      expr: sum by (namespace, pod) (kube_pod_status_ready{condition="false"}) > 0
      for: 15m
      labels:
        severity: critical
      annotations:
        summary: "Pod {{ $labels.pod }} not ready"
        description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been not-ready for 15 minutes."

    - alert: HighMemoryUsage
      expr: (container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.9
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High memory usage on {{ $labels.pod }}"
        description: "Container {{ $labels.container }} is using {{ $value | humanizePercentage }} of its memory limit."

kubectl apply -f pod-health-alerts.yaml

The labels: release: kube-prometheus on the PrometheusRule is critical — it needs to match the ruleSelector the Helm chart configured for Prometheus to actually pick up your rule.

Step 5: Verify in the Alertmanager UI

kubectl port-forward -n monitoring svc/alertmanager-operated 9093:9093

Open localhost:9093 to see active alerts, silences, and the routing tree.

Silencing Alerts

During planned maintenance, silence alerts via CLI (amtool) or the UI:

amtool silence add alertname="PodCrashLooping" namespace="payments" \
  --duration="2h" \
  --comment="Planned maintenance window" \
  --alertmanager.url=http://localhost:9093

amtool silence query --alertmanager.url=http://localhost:9093

Production High Availability

Run Alertmanager with at least 3 replicas — the Prometheus Operator automatically configures the mesh/gossip protocol between them via --cluster.peer flags, so all replicas share notification state and avoid duplicate pages even if multiple Prometheus instances send the same alert.

kubectl get statefulset -n monitoring alertmanager-kube-prometheus-alertmanager -o yaml | grep replicas

Templating Notifications for Clarity

Default Alertmanager notification templates are functional but generic. Custom templates make pages far more useful when someone’s reading them half-asleep at 3 AM:

templates:
  - '/etc/alertmanager/templates/*.tmpl'

{{ define "slack.custom.text" }}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Namespace:* {{ .Labels.namespace }}
*Summary:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Started:* {{ .StartsAt }}
{{ end }}
{{ end }}

receivers:
- name: 'slack-warnings'
  slack_configs:
  - channel: '#alerts-warnings'
    text: '{{ template "slack.custom.text" . }}'

Consistent templates across every receiver mean whoever’s on call doesn’t have to parse a different format depending on which alert fired — every notification reads the same way, with the same fields, in the same order.

Handling Alert Fatigue at Scale

As a cluster and its alert rules grow, the biggest operational risk isn’t missing alerts — it’s people tuning them out because too many fire. A few practical mitigations beyond routing and inhibition:

route:
  routes:
  - match:
      severity: warning
    receiver: 'slack-warnings'
    active_time_intervals:
    - business-hours
time_intervals:
- name: business-hours
  time_intervals:
  - weekdays: ['monday:friday']
    times:
    - start_time: '09:00'
      end_time: '18:00'

active_time_intervals restricts a route to only deliver during the specified window — warning-level noise doesn’t need to wake anyone up outside working hours, while critical routes typically skip this restriction entirely.

Common Mistakes

Best Practices

Summary

Alertmanager is what turns Prometheus’s raw alerting rules into actionable, deduplicated, correctly-routed notifications — without it, you have data but no signal. Deploy it with multiple replicas for HA, define routing trees by severity and team, back alerts with well-tuned PrometheusRule objects, and use inhibition rules to keep noise down during real incidents. Get this right and the next outage gets caught by your team, not your customers.

References

Exit mobile version